Summary:

After the earthquake on September 19 2020, it looks to me like the only lasting damage to suspensions in vacuum is the ETMY UR magnet being knocked off. 

Suspension ringdown tests:

I did the usual suspension kicking/ringdown test:

• One difference is that I now kick the suspension "N" times where N is the number of PSD averages desired. 
• After kicking the suspension, it is allowed to ring down with the damping disabled, for ~1100 seconds so that we can get spectra with 1mHz resolution.
• We may want to get more e-folding times in, but since the Qs of the modes are a few hundred, I figured this is long enough.
• I think this kind of approach gives better SNR than letting it ringdown 10,000 seconds (for 10 averages with 10 non overlapping segments of 1000 seconds), and I wanted to test this scheme out, seems to work well.
• Attachment #1 shows a summary of the results.
• Attachment #2 has more plots (e.g. transfer function from UL to all other coils), in case anyone is interested in more forensics. The data files are large but if anyone is interested in the times that the suspension was kicked, you can extract it from here.

Conclusions:

1. My cursory scans of the analysis don't throw up any red flags (apart from the known problem of ETMY UR being dislodged) 👌 . 
2. The PRM data is weird 
   • I believe this is because the DC bias voltage to the coils was significantly off from what it normally is when the PRC is aligned.
   • In any case, I am able to lock the PRC, so I think the PRM magnets are fine.
3. The PRC angular FF no longer works turns out this was just a weird interaction with the Oplev loop because the beam was significantly off-centered on the Oplev QPD. Better alignment fixed it, the FF works as it did before.
   • With the PRC locked and the carrier resonant (no ETMs), the old feedforward filters significantly degrade the angular stability to the point that the lock is lost.
   • My best hypothesis is that the earthquake caused a spot shift on PR2/PR3, which changed the TF from seismometer signal to PRC spot motion.
   • Anyways, we can retrain the filter.
   • The fact that the PRC can be locked suggest PR2/PR3 are still suspended and okay.
4. The SRM data is also questionable, because the DC bias voltage wasn't set to the values for an aligned SRC when the data was collected
   • Nevertheless, the time series shows a clean ringdown, so at least all 5 OSEMs are seeing a signal.
   • Fact that the beam comes out at the AS port suggest SR3/SR2 suspensions are fine 👍 

Attachment #2 also includes info about the matrix diagonalization, and the condition numbers of the resulting matrices are as large as ~30 for some suspensions, but I think this isn't a new feature. 

Summary:

The simple interferometer, composed of a single bounce reflection from ITMY and the LO beam deilvered via fiber to the AS table, can be locked - i.e. the phase of the LO beam can be controlled such that the DC light level on the DCPDs after the two beams are interfered can be stabilized. This test allows us to confirm that various parts of the sensing and actuation chain (e.g. PI PZT for homodyne phase control, Trek amplifier etc etc) are working.

I will post more quantitative analysis tomorrow.

Optical configuration:

• LO beam is a pickoff of the main PSL beam from just before it goes into the vacuum. The optical power arriving on each DCPD after the various beamsplitters, coupling loss etc is ~200 uW.
• IFO beam is the single bounce reflection from ITMY. For this test, ETMY, ITMX and ETMY are misaligned. Optical power arriving on each DCPD is ~80uW.
• The two beams are interfered on a 50-50 beamsplitter. The mode-matching efficiency was estimated to be ~50% which isn't stellar, but should be fine for this test.
• So, at half-fringe, we expect the signal on each DCPD to be linearly proportional to the phase difference between the two fields, and so we can use that as an error signal.

Servo topology:

Attachment #2 shows the servo topology.

• For a first attempt to close the feedback loop, we can consider the two blocks labelled "Sensing Chain" and "Actuation chain" to have a flat frequency response. While this isn't true, for a taget loop with ~100 Hz UGF, I think the approximation is reasonable.
• From the peak-to-peak value (160 cts) of the DCPD signals when the homodyne phase is uncontrolled, I estimate a sensing response (at half-fringe) of approximately 0.3 ct/nm, since this corresponds to 532nm of relative phase between the two beams. 
• An inverting summing amplifier is used to map the +/- 2^15 ct DAC range to 0-125V on the PI PZT. Assuming the full stroke of the PZT is 10um per the datasheet, and that this voltage range drives half of the full stroke (this is just a guess since all the old PI PZT circuits were designed to work at 0-250 V), we get an actuation coefficient of 0.075 nm/ct.
• Using these two numbers, we can then design a digital feedback loop that gives an open loop transfer function with ~100 Hz UGF, and sufficient stability margin.
• From the earlier measurements, we have an estimate for the amount of phase fluctuations caused by (i) seismic disturbances and (ii) fiber phase noise. This is the quantity we wish to suppress, and the suppression factor will be 1/(1+L), where L is the open loop gain.
• I didn't do this in any systematic way, but the loop in Attachment #3 seemed like a reasonable shape that would suppress the error signal RMS by ~10x, as shown in Attachment #4. So I decided to try this out.

Other notes:

1. The idea of offloading the DC control voltage to the ITMY suspension seemed to work fine.
2. It also seems like the relative phase between the two beams doesn't drift by so large an amount in short time scales, at least at night/quiet seismic conditions. So it is possible to maintain the lock for several seconds without having to offload the DC signal to the suspensions.
3. I didn't bother adapting the FSS Slow PID script to do this offloading in an automated way, seemed like more trouble than was just doing it by hand. But we may want to automate this in the future.
4. I couldn't make a clean measurement of the loop transfer function using the usual IN1/IN2 method. Introducing a step offset at the error point, the servo is able to track it (I didn't fit the step response time, but it's not as if the loop bandwidth is <1 Hz or something). I have to compare the measured in-loop error signal ASD to the free-running one to get a feel for what the UGF is, I guess, to rule out a weird loop.
5. Update 1100 Oct 6 2020: I have now added measured, in-loop, error point spectra to Attachment #4. Looks like there might be significant sensing noise re-injection.
   • Initially, I forgot to turn the HEPA on the PSL down for the measurement. So I have the two traces to compare. Looks like with the HEPA turned up to full, there is more noise in the 50-200 Hz range.
   • The trace marked "highGain" was taken with an overall loop gain that was 3dB higher than the nominal value - I could see some oscillations start to appear, and in the spectrum, maybe the feature at ~150 Hz is evidence of some gain peaking?

Conclusions:

1. The PI PZT seems to work just fine.
2. Need to look into the loop shape. I guess it's not reasonable to expect a UGF much higher than 100-200 Hz, because of the various delays in the system, but maybe the low frequency suppression can be made better.
3. What are the next steps?? What does this mean for the RF44 sensing scheme?

🤘🤘🤘

We received 20pcs of stuffed demodulator boards from Screaming Circuits today. Some caveats:

1. The AP1053 amplifiers weren't stuffed. Note that this part is no longer in standard production, and lead time for a custom run is ~half a year. I recommend stuffing R2 and using a minicircuits amplifier upstream of this board. We have 6 pcs of AP1053 in hand so we can use those for the first AS WFS, but a second WFS will require some workaround.
2. AD8306ARZ weren't sent to Screaming Circuits. This part is used for the LO and RF signal level detection/monitoring stage, and so aren't crucial to the demodulation operation. @Chub, did we order the correct part now? They are rather pricey so maybe we can just adapt the footprint using some adaptor board?
3. DQS-10-100 hybrid 90 degree splitters were delivered to us after the lot was sent to Screaming Circuits. We have the pieces in hand, so we can just stuff them as necessary.

I removed 1 from the group to stuff some components that weren't sent to Screaming Circuits and test the functionality on the benchtop, the remaining have been stored in a plastic box for now as shown in Attachment #1. The box has been delivered to Chub who will stuff the remaining 19 boards once I've tested the one piece.

LiYuan has kindly done some Total Integrating Sphere (TIS) measurements on ITMU01 and ITMU02. A summary of the measurement is attached. I uploaded the measurements and some analysis script to nodus at /home/export/home/40m_TIS. I created a Wiki page for the measurements and linked to it from the core optics page.

These TIS measurements look very similar to the TIS of the LIGO optics. Further analysis shows that the scatter loss is 10+/-1.7 ppm for ITMU01 and 8.6+/-0.4 ppm for ITMU02.

In this calculation, a gaussian beam the same size bouncing off the 40m ITMs is assumed to scatter from the mirrors. The error is calculated by moving the beam around randomly with STD of 1mm.

In LiYuan's setup, TIS is measured for scattering angles between 1 and 75 degrees. If we go further and assume that the scatter is Lambertian we can extrapolate that the total loss is 10.9+/-1.9 ppm for ITMU01 and 9.2+/-0.5 ppm for ITMU02.

These measurements complete the loss budget nicely since with the 6ppm loss predicted from the phase maps, the total loss in the arm cavities would be 6+10+10=26ppm which is very close to the 28ppm loss that was measured after the arm cavity optics were cleaned.

I have placed 3 new in box, IDP 7 forepumps along the x arm of the interferometer. These are to be used as spares for both the 40m and Clean and Bake.

Tue evening from 4pm~6pm, Koji made a social distant tour for Anchal. We were present around the PSL/AS/ETMX tables.

This looks cool, we should have something similar, can be really useful.

• UPS batteries
• 2x HEPA filters
• VWR chemicals (methanol)

These boxes were moved from the 40m hallway to the inside of the VEA so that we have some space to walk around. You can find some pictures here.

Is it better than Luxor? https://labcit.ligo.caltech.edu/~jharms/luxor.html

Attachment #1: spectra of the phase noise between LO and IFO output fields sensed using the RF44 signal.

• Measurement setup:
  • LO an IFO fields are combined on a beamsplitter, with ~60% mode-matching efficiency.
  • One port of the BS goes to a DCPD.
  • The other port goes to an RF sensing photodiode, PDA10CF. The spec-ed dark noise NEP is ~12 pW/rtHz at 1.6 um, (so let's say 25 pW/rtHz) and transimpedance is 5kohms into a 50 ohm load. We can convert this to an equivalent sensing noise at the error point of this loop, though it's more likely that the electronics (demod, ADC etc) noise downstream dictate the sensing limit, which I measure by blocking light on the photodiode.
• The demodulation is done on one of the newly received D0902745 boards - this was just a more compact setup than many cascaded minicircuit components. We don't have the hardware to package this into a chassis to shield against electronics noise pickup yet, so I'm using a bench supply to power this for now (via a voltage regulation board, D1000217.
• "Dark Noise" = ASD with no light incident on the photodiode. "LO field only" = ASD with only the LO field incident on the photodiode.
• The "Dark noise" trace and "LO field only" traces are converted from cts/rtHz to rad/rtHz by noting that when the Michelson is locked on a dark fringe, the demodulated RF44 quadratures have a pk-pk amplitude of ~160 cts (corresponding to pi radians of phase shift). Since in these conditions the demodulated quadratures do not undergo any fringe wrapping, I converted the spectra by simple multiplication.
• For the "RF44 open loop" trace:
  • The DC offset in the demodulated signal (due to the RF44 signal from the LO field only) is digitally compensated, so that the fringing has (roughly) zero offset.
  • The Michelson was locked on a dark fringe, and the demodulated RF44 quadratures were monitored for ~5 mins. Then arctangent (specifically, arctan2 to get the correct quadrant in the IQ plane) of the two signals was taken to convert the fringing signals to phase noise.

Closing a feedback loop:

• Since it seems like we are sensing a signal (below ~1kHz at least), I tried to close a feedback loop (modelled loop shape shown in Attachment #2, it's just a model because I have to guess what the sensing and actuation gains are, and they're both assumed to be flat, digital delays etc aren't accounted for). I've also added the inferred loop gain by taking the ratio of the in loop and unsuppressed ASDs (though of course I don't account for the flat sensing noise at higher frequencies). At least qualitatively, things line up...
• While I can get the light level on the DCPD to stabilitze somewhat, the loop is not at all stable, and the suppression isn't very good at all.
• Not sure how meaningful any of the spectra with the loop closed are, but FWIW, I've put in the spectra of the demodulated RF44 signals with the loop engaged (RF44 Q is used as the error signal). A clear problem is evident at ~120 Hz, and the forest of lines isn't helping for sure. Also unclear to me why the I and Q signals don't have the same profile at low frequencies.

Conclusions/Questions:

1. What is the reason for the huge forests of lines in the "RF44 open loop" ASD, that are absent in the other two traces? If this were electrical pickup, it should be there in all three traces?
2. Is the shape of the spectrum reasonable? The roll-off above ~5 Hz doesn't seem quite steep enough to be seismic noise from the suspensions. Can it really be that the Michelson dark field has such high phase noise?
3. How can we get this scheme to give us cleaner sensing?
4. The actuation chain was verified to work fine with the single bounce beam from an ITM interfered with the LO field, and using the DC light level as an error signal and locking to the half-fringe point. So the problem is not due to insufficient actuation range. Seems like the error signal is so polluted with these forests of lines that even though there is some suppression of the error signal at low frequencies, the unsuppressed noise is still significant. I can't solve the problem by simply increasing the loop gain...
5. It is not shown here, but with only the LO field incident on the RFPD, I see a drift of the demodulated signals on the ~5 minute timescale - is this just due to fiber length change? If so, this is potentially problematic, as on long time scales, the true zero of the error point of the servo would be changing on the ~5 minute timescale. This would be true even for the final suspended scheme - if the path length between PR2 and the homodyne BS changes by some microns, we would have to correct this at DC?

[JV, GV]

We cleared up some space in the 1Y1 electronics rack to install the 3 new FE machines. I removed the current driver and laser from 1Y1, they are now stored in the E10 cabinet. I will upload some photos to gPhotos soon.

1. I think it's good to have all these FEs in one rack (at least the new ones) - we should then hook it up to an ethernet power source, so that we can remotely power cycle them. I think we have long enough cables to interface to expansion chasses / dolphin switches, but if not, I think it's still a good idea to have these machines in 1Y1 as it is the least sensitive area in terms of immunity to bumping some cable during setup work and disturbing the rest of the IFO.
2. We found that the rails that the Supermicros shipped with the servers seem to be just a little too narrow - we mounted these in the rack, but had considerable difficulty sliding the server units in. Once they are in, they don't slide smoothly. Is there some special trick to installing these? 
3. I spent a few minutes trying to get Debian 8 installed on these machines, so that the rest of the setup work could be done remotely - however, there appear to be some firmware issues and so I'm not gonna dive into this.
   • I couldn't find a disk image for Debian 8.5 which is what the KT wikl recommends, so the OS I tried to install was Debian 8.11.
   • The error that comes up is related to a "stalled CPU" - apparently this is related to some graphics driver issue (there's another forum page that suggests upgrading the BIOS, but I don't think that's the problem here).
   • Anyways, this part of the process is only to install some drivers and do the initial setup - these machines will eventually run a diskless boot from the image on FB, so who knows if there will be some other driver issues/hardware-software incompatibilities there 😱 .
   • We should also make an effort to set these machines up with IPMI, but I think we first need to install an OS and a CLI to setup the IPMI. My cursory browsing of the manual suggests that the initial setup maybe can be done without installing an OS, and then subsequent work, including OS install, can be done remotely. If someone reads more in detail and can provide me a step-by-step, I can follow those instructions (if they aren't available to come into the lab). See here for some brief documentation of how to access the IPMI.

The two ITM spares and two ETM spares are together stored in the optic storage (B110) at Downs. c/o Liyuan and GariLynn

Per Koji's suggestion, I turned the PSL HEPA Variac to 50% just now, so that the power load through the burnt electrical cable is reduced by 75%.

Summary of discussion between Koji and gautam on 14 Oct:

1. Koji questioned the accuracy of the "open loop" ASD shown here. While it may not be entirely accurate to compute the free-running (homodyne) phase noise simply by taking the arctangent of the I and Q signals (because the magnitude of the signal is also changing), gautam claims the estimate is probably still close to the true homodyne phase, especially since the ratio of the "in-loop" and free-running ASDs gives something that closely approximates the magnitude of the supposed OLG of the system.
2. Koji suggested the following tests:
   • Investigate the relative stability of the two RF signal generators involved in this system. Since the 44 MHz electrical LO signal (for demodulation) is generated by a separate IFR from the one used to imprint 11 MHz and 55 MHz phase modulation sidebands on the main PSL beam, we want to investigate what the drift is.
   • Try implementing an analog feedback loop using LB1005 - the idea being we should be able to implement higher bandwidth control, for better suppression of the high frequency noise (which looking at the ASD is not only due to seismic phase modulation of the IFO output field). Maybe some combination of this and the Marconi investigation would suggest why we have these forests of lines in the ASDs of the error signal?
   • Turn off the HEPAs on the PSL enclosure completely as a test, to see if that improves (i) phase noise due to air currents and (ii) mechanical pickup on the fiber producing  phase noise.

I tried all of these last night / overnight, here are my findings.

Analog locking of the homodyne phase:

See Attachment #1. 

• RF44_I was used as the error signal.
• The "C1:OMC-ZETA_IMON_OUT" channel is actually looking at the error signal monitor from the LB1005, and is uncalibrated in this plot.
• The "monitor" port on the demodulator board provides a convenient location for us to route the demodulated signal to an LB1005 box, while simultaneously digitizing both demodulated quadratures.
• Empirically, I found settings that could engage the lock. I also found that I couldn't increase the gain much more without destroying the lock. 
• The time domain signals look much "cleaner" in this analog feedback loop than when I achieved similar stabilization using the digital system. But I will quantify this more when I post some spectra of the in loop error signals.
• I will do some more characterization (loop TF measurement, error point spectrum in lock etc), but in summary, it looks like we still only have ~100 Hz UGF. So something in the loop is limiting the bandwidth. What could it be?
• The main problem is that the LB1005 isn't well suited to remote enabling/disabling of the lock, so this isn't such a great system.

Relative stability of two IFR2023s synchronized to the same FS725 Rb standard:

The electrical LO signal for demodulation of the 44 MHz photocurrent is provided by an IFR2023 signal generator. To maintain a fixed phase relation between this signal, and the phase modulation sidebands imprinted on the interferometer light via a separate IFO2023 signal generator, I synchronize both to the same Rb timing standard (a 10 MHz signal from the FS725 is sent to the rear panel frequency standard input on the IFR). We don't have a direct 44 MHz electrical signal available from the main IFO Marconi at the LSC rack (or anywhere else for that matter). So I decided to do this test at 55 MHz. 

• RF input of the demodulator was driven by 5*11.066209 MHz pickoff from the LSC rack.
• LO input of the demodulator was driven by 5*11.066209 MHz signal from the IFR2023 used for the RF44 demodulation setup.
• The outputs were monitored overnight. The RF44_Q channel had a DC level of nearly 0. So this channel is nearly a linear sensor of the phase noise between LO and RF signals.
• To convert ADC counts to radians, I offset the LO Marconi frequency by 100 Hz, and saw that the two quadratures showed pk-pk variation of ~24000cts. So, at the zero crossing, the conversion is 1/(24000/2) rad/ct ~83urad/ct.
• The result is shown in Attachment #2. The "measurement noise" trace corresponds to the RF. input of the demodulator being terminated to ground with a 50 ohm terminator.
• For comparison, I also overlay the phase noise estimate of an individual IFR from Rana. In his investigation, the claim is that the PLL that locks the IFR to the Rb timing standard has ~1kHz UGF, but if my measurement is correct, the relative stability between the two signal generators synchronized to the same timing standard already. degrades at ~1 Hz. Could be just a cts/rad calibration error I guess.
• In any case, we are far from saturating this limit in the homodyne phase lock.
• There are several sharp lines in this measurement too - but I don't know what exactly the source is. Of course the two marconis are plugged into separate power strips, so that may explain the 60 Hz lines and harmonics, but what about those that aren't a multiple of 60 Hz?

A look at the time domain signal:

With the Michelson locked on the dark fringe, the RF44 I and Q signals in the time domain are shown in Attachment #3 for a 1 minute stretch.

• The RF44 signal level bottoms out at ~40 cts. Okay, so this is the offset.
• However, the maximum value of the RF44 signal amplitude seems to be modulated in time. How can we explain this?

Pushed another update to MCMC simulation. This includes:

• Added new imbalances: ITM transmission, ITM & ETM RoCs.
• Added new static offsets: DHARD, DSOFT, CHARD, CSOFT. All pitch. The RMS is calculated from the data Jon fetched with /input_noises/input_noises.ipynb.
• SRCL noise ASD and RMS are now taken from data in /input_noises.
• RF PD diagnostics were redone: Instead of post-discarding marginal simulations, simulations are now discarded when one or more of the RF PDs demodulated signal does not cross zero when the associated DOFs are scanned by 1um in the offset state.

The DOFs<->RFPD associations I use are:

However, one thing that bothers me is that for some reason ~ 15 out of 160 aLigo simulations are discarded while none for A+. It can also be seen that the A+ simulations are more spread-out which might be related.

The new simulation results are attached.

I'll bring a file binder "40m wiring diagram" to home at the next chance.
There is another one on the shelf in the control room.

(I thought I put it in my bag, but it looks like that I left it somewhere around the fax area)

Alaska M7.5 20:54UTC https://earthquake.usgs.gov/earthquakes/eventpage/us6000c9hg/executive

I looked at the suspensions. The watchdogs have not been tripped.

IMC was locked but continually shaken. (and occasional unlock)

I packaged the HV coil driver into a 2U chassis, hoping for better shielding from pickup. There is still considerable excess noise in measurement vs model around 100 Hz, see Attachment #1. The projected displacement noise from this noise contribution is shown in Attachment #2 - I've also plotted the contribution from the 4.5kohm (planned value for fast path series resistance) for comparison. Attachment #3 has some photos of the measurement setup so if someone sees some red flags, please let me know.

• The noise was measured with the output load connected to a 20ohm load resistor, to simulate an OSEM.
• The input signal was driven with an Acromag, to try and mimic the actual operating conditions as closely as possible (although the fast path input was left unconnected).
• The KEPCO switching HV power supplies were used to power the unit.

I've run out of ideas to try and make the measurement cleaner - the presence of the rather prominent power line harmonics suggests that this is still not perfect, but what more shielding can we implement? I have to make the measurement on the circuit side of the 25 kohm series resistor, so I am using some Pomona minigrabbers to clip onto the leg of the wirewound resistor (see photos in Attachment #3), so that's not great maybe, but what's the alternative?

So if this is truly the noise of the circuit, then while it's an improvement on the current situaiton, it's unsatisfying that such a simple circuit can't match the design expectations. But how do we want to proceed?

I found this H1 alog  entry by Izumi confirming that the calibrated channels CAL-CS_* need the same dewhitening filter.

This encouraged me to download the PRCL and MICH data and using Jon's example notebook. I incorporated these noise spectra into the MCMC simulation. The most recent results are attached.

I am still missing:

• Laser frequency noise
• Laser RIN
• Estimation of the LO phase noise
• Estimation of the BHD breadboard angular noise

Also, now the MCMC repeats a simulation if it doesn't pass the RF PDs test so the number of valid simulations stays the same. I'm still not sure about why the A+ simulations are much more robust to these tests than aLigo simulations.

what is the noise level before the HV stage? i.e. how well is the acromag noise being filtered?

It's not so easy to directly measure this I think, because the filtering is rather aggressive. Attachment #1 shows the measured transfer function (dots) vs the model and Attachment #2 shows the noise. I think this checks out - but I can't definitively rule out some excess noise at 100 Hz from this stage. Because the gain of the HV stage is x31, we'd need a preamp with better than 1nV/rtHz to directly measure the noise I guess. The Acromag noise model in Attachment #2 is based on a measurement I describe here.

Andrew made a battery-powered 0.7 nVrtHz input-referred noise pre-amplifier for gain of 200. That might help you.

RXA: 0.7 nV is OK if you're not interested in low noise measurements. Otherwise, we have the transformer coupled pre-amp from SRS which does 0.15 nV/rHz and the Rai Weiss FET amp which has 0.35 nV for high impedance sources.

[Koji, Rana]

We wanted to track down the excess noise seen in MC_F and other places (see the previous report by Gautam)

Setup1: The IMC was locked and MC_F signal between 500 and 1500Hz was observed. The DTT template was saved as /users/Templates/MC/MCF_noise_201023.xml

- Suspected mech resonance/jitter coupled with clipping or any other imperfections. Poked the various optics and optomechanics on the table. Basically no change. If we tap the laser chassis and the optics close to the laser source, we occasionally unlocked the IMC

- When we touched (lifted) the Innolight controller box from the shelf, for the first time we saw a significant change in the shape of the noise spectrum. The peak around the 700Hz shited towards lower frequency by a few %. Other peaks have no obvious change in the shapes and the heights.

- While observing the MC_F signal on the laptop, we went to the back of the laser controller. Placing a hand close to the fan clearly changes the peak frequency lower. By temporarily disconnecting the fan from the power supply for a short moment, the 700Hz peak could be eliminated. We also tried to see the noise level with the slow thermal servo and diagnosis DB cable disconnected, but we didn't see any significant change of the noise level.

Setup 2: Using the ALS phase tracker, we can observe the relative freq noise of the PSL laser and the ETMY AUX laser without any servo involved. This way we can freely disconnect any cables from the lasers. The measurement template for DTT was saved as /users/Templates/ALS/Y_ALS_FINE_PHASE_OUT_102320.xml

- Noise spectrum before disconnecting the cable (REF0, RMS REF1)

- The Fast PZT input to the PSL was disconnected => This made all the peaks (including the 700Hz) disappeared (REF2, RMS REF3)

- The Fast PZT input was restored as before, then the chain was disconnected at the input of the HV PZT driver (Thorlabs) => Again, this made the peaks disappeared (REF4, RMS REF5)

- The chain was disconnected at the input of the TTFSS box => Again, this made the peaks disappeared (REF6, RMS REF7)

- Disconnected the demod input and the AO cables from the IMC servo board => This made the peaks came back (REF8)

- Disconnected all the input/peripheral cables from the IMC servo board except for the connection to the TTFSS box => Still the excess noise was observed  (REF9)

- In addition to the above, the cable to the FSS box was disconnected but the ground was still touching the MC servo board => This made the peaks disappeared (REF10)

The conclusion is that the noise is injected from the main circuit of the IMC servo board.

Next time we will check if the backplane connection is doing something wrong. Also, we'll test if the presence of the RF signals does something bad to the IMC board via EMI and RFI.

We have reverted the connection and tested if we lock the IMC and Y arm. ==> We saw at least they were locked for a short period. The things are still stabilizing, but left them turned on so they keep trying to lock automatically for the night.

In fact, the problem was the grounding issue (presumably on the IOO racks).
A temporary differential receiver at the TTFSS side was built using an SR560 and a few ponoma cables. This removed the structures ~850Hz.

The MC Servo Output was disconnected from the TTFSS box and monitored with SR785. The 850Hz structure was kept visible no matter what cables, including all the acromag DB cables, were removed. This made me suspicious about the measurement setup. The SR785 was connected to an AC power strip under the SP table and this was too far from the IOO rack.

The SR785 was connected to the AC power strip on 1X2, and now the difference becomes clear. No matter if the acromag cables are connected or not, the connection (particularly ground connection) between the MC servo module and the TTFSS box causes the MC servo output contaminated. (Comparison between Blue and Orange of Attachment #1). During the measurement, the EPICS switch for the fast path was disengaged (=no signal) and the VCO gain (...so called. It's just the MC Servo Gain) was set to be 0dB.

To test if the differential receiving of the MC Servo Output at the PSL helps to reduce this noise, I've built a simple (hacky) differential receiver using an SR560. (Attachment #2)
This kept the noise level same as the disconnected case (Comparison between Green and Orange of Attachment #1, I don't think the difference between them is not significant), while the IMC is locked as before.
Note that we can see that the 36kHz line was significantly reduced. Did we remove this annoying noise?

After talking with Gautam, we decided to leave this configuration while the SE-Diff cable was replaced with a more robust one. (See Attachment #3)

The PSL laser frequency performance was evakluated in the following two ways as we did last week:
1) Use the beat frequency of the free running PSL and the Y-end laser (Attachment #4). The PSL shutter was closed and thus the IMC was not locked.
2) Use the IMC MCF while the IMC was locked. (Attachment #5)

For both cases, the improvement was confirmed.

I also tried to check the reported issue by Gautam on this elog. He used 1Hz BW, but I cheated with 16Hz BW and 10x12.8kHz span PSDs. (Attachment #6)

For the measurement, IN1 GAIN of the IMC Servo was set to be 0dB and the OUT2 was switched to monitor the IN1 noise, while IN1 was terminated by a 50Ohm.

As I mentioned above, the AC power of SR785 was taken from a 1X2 power strip. Is this the reason for the power line forest look less severe compared to the previous case???
Anyway, I tried to use the same differential receiving technique (but with gain of x100) to see if this helps. The differential receiver helped to reduce the structure above 50kHz. The floor noise level was observed to be higher. I didn't pursue this any further, but the forest of the power line looked like a part of the measurement noise. This is indicative that the grounding condition on 1X2 is really not great and we need to review the configuration of the acromag grounding.

Apart from the questionable wiring on the Acromags, one other important difference is in the way the connections were made between the old VME crates to the Eurocrate backplanes, and how we do it now. The thick cables had their sheilds connected to the eurocrate ground (or at least, there was a dedicated ground lug on those cables which we screwed on to the ground terminals on the Eurocrate backplanes). However, in our current configuration, we interface the Acromag ADCs and DACs to the backplane via these adaptor boards. The shields of the DSUB cables are presumably NOT connected to the Eurocrate grounds. This should also be investigated as one potential cause of the grounding issue - while on some of the Eurocrate modules, the P1/P2 connectors may have either the "A" or "C" row of connectors shorted to ground, some may not, and the TTFSS may suffer from such an issue?

Note that we have this problem in all of the slow machines that were upgraded to Acromag (if this turns out to be the issue). 

I wanted to look into the ISS situation. Some weeks ago, I found the PD that was previously used as the in-loop photodiode. I wanted to use this and measure the open-loop RIN at a few places (to see if there's any variation and also to check its functionality). However, I didn't get very far tonight - for a start, the PD height is 3" (while our beam height is 4" everywhere outside the vacuum), and I needed to put together a circuit to supply the 5V bias and +/- 15 V since the transimpedance is done on the head. I was only able to do a low-level functionality test tonight, checked that the DC voltage output varied linearly with the incident power (calibrated against an NF1611 photodiode, data will be put up later). I didn't get to measuring any noise performance - is an incandescent light bulb still shot noise limited at ~10 Hz < f < 10kHz? Some notes:

1. The PD is DC coupled, and has a transimpedance of 1 kohm (inverting AD829 does the transimpedance).
2. Probably a daughter board should be made that supplies the DC power voltages and rotues the output signal to something more convenient like a BNC connector. This daughter board can then also implement a DC coupled path (for monitoring) and AC coupled path (for servoing, fc to be determined).
3. SR560 based ISS was implemented some years ago but I think the improvement was only seen above 100 Hz, and that too was marginal, the stabilized RIN was 10^-6 (monitored on an out-of-loop photodiode I think, but unsure). We'd probably want to aim for at least an order of magnitude better. Unclear at this point why more suppression wasn't possible back then, was it just insufficient loop gain, or was the sensing noise too high? To be investigated.

Unconnected to this work - this problem reared its ugly head again (i noticed it yesterday morning already actually). I don't have the energy to embark on a fix tonight, Koji is going to be in the lab all day tomorrow and so he will fix it.

I'm starting the model restarts from remote. Then later I'll show up in the lab to do more hard resets.
==> It seems that the RFM errors are gone. Here are the steps.

1. Shutdown all the watchdogs
2. login to c1iscex. Shutdown all the realtime models: rtcds kill --all
3. login to c1iscey. Shutdown all the realtime models: rtcds kill --all
4. run scripts/cds/rebootC1LSC.sh on pianosa
5. reboot c1iscex
6. reboot c1isxey
7. Wait until all the machines/models are up by the script
8. restart c1iscex models
9. restart c1iscey models
10. some IPC errors are still visible on the CDS status screen. Lauch c1daf and c1oaf

that little PD in the black mount was never very good. The AD829 is not a good opamp for transimpedance and especially not good for low frequencies. Stefan Ballmer and I were able to get 2e-8 out of these (@100 Hz) many years ago.

I wonder if we have some of Zach's M2ISS photodetectors around, perhaps in QIL or Cryo. I doubt that any of them are in use now. Those had good performance nad BNC output.

Ok I was using the PD in the black mount because Rana recommended it a few weeks ago.

Regarding the M2ISS, I acquired the hardware from QIL some months ago, including a circuit board, and 2 PDs. These had LEMO outputs though (not BNC), and the mounts are not 4". These photodiodes are what I'm using as the airBHD DCPDs right now, and some photos are here - are these the photodiodes you mentioned? Or are there yet more M2ISS photodiodes? I remember Johannes had some custom mounts extruded to make them 4" high, do you mean those? Can I retrieve them his Cryo setup?

BTW, my elog scraping shows only one spectra from Stefan in the ATF elog, and the performance there is more like 1e-7/rtHz @ 100 Hz, and that’s using a dedicated high BW servo circuit, not the SR560. Am I just missing the measurement of 2e-8/rtHz?

1. Two arms / BS / PRM / SRM were aligned. (Attachment 1)
2. IMC was aligned by WFS and the WFS offsets were offloaded.
3. Suspension Status Snapshot (Attachment 2)
4. Oplevs are aligned (Attachment 3)
5. Xarm green was aligned in the daytime. Xarm green refl DC (C1:ALS-X_REFL_DC_OUTPUT) was 620 (aligned) ~1300 (drifted). When unlocked, it was 3750.
6. Yarm green: I saw no flash. We don't have functional PZT alignment since the ASY M2 PZT got broken. I went to the Yend. Something funky is going on with the Yend green. I struggled to have any flash of the cavity. The apertures were not so precise. I finally got TEM00 locked, but the modematching seems exteremely low (like 1/1000?). Basically I saw no power reduction of the refl when the cavity is locked. So at least the cavity was locked but we might need to revisit when we open the chamber
   ==> Gautam thinks it was not like that. So he will check the green alignment tomorrow (Thu).
7. Item checking: I familialized myself with the yend crane operation. Today I learned that there is a power switch on the wall (Attachment 4). The yend has two heavy door storages (Attachments 4/5). The slings to lift the heavy door are in the crane cabinet along with the y arm (Attachment 6). I didn't yet try to find the "hammer" to hit the door if the o-ring stuck too strong, although that's optional.
8. We want to reduce the PSL power. But Gautam wants to use the arm locking with the nominal power, it will be done tomorrow by him.
    
9. The last thing is to check the green trans power. I noticed that the green trans beams are blocked by an HWP for the BHD LO path on the PSL table. (Attachment 7)
   The HWP was moved and the process was recorded in the movie (Attachment 8). The fiber output was monitored by the BHD DC (aka AS110 DC) with the AS path blocked. The fiber output of 22.6mV (offset -2.5mV) was improved tio 29.1mV after the HWP move and the alignment adjustment.
10. Now the green transmissions are visible by the green PDs. Attachment 9 shows the trans and ref of each green beams with and without locking to TEM00. The questionable green TRY was ~0.3. If we compare this with the histrical data (Attachment 10), it is about 1/4 of the value in the past. It's not too crazy but still quite low.

At this point, I'm leaving the lab. All the suspensions (incl SRM) are aligned. PSL/GRX/GRY shutters were left open.

1. Oplev HeNe at ETMY was replaced, see here for my earlier discussion on this.
   • I thought this is a good idea since we want the Oplev as a coarse reference and it'd not be ideal if this HeNe dies during the time the optic is out of the chamber.
   • New HeNe had 2.8mW of power output as measured with Ophir power meter. This is in line with what is expected from these Lumentum heads.
   • I labelled the head with the power output and today's date, re-aligned the Oplev reflection onto its QPD. 
   • After this work, the Y-arm could be locked without the huge angular fluctuations that was visible earlier, 👍 .
2. GTRY anomaly
   • I actually judged that there is no anomaly.
   • The GTRY CDS indicator is actually quite useless - the ADC saturates at ~3500 cts (and not 32768 as you would expect from a 16 bit ADC but that's the well-known whitening filter saturation problem). This should be fixed, but this is a task for later.
   • I measured with a DMM the voltage when the TEM00 is locked to the cavity and GTRY is 0.3 (the nominal value these days), the DC voltage was ~5.6 V. The prompt reflection from the ETM registers ~6.5 V DC. So the mode-matching isn't stellar, but this is again a known issue, and can be fixed later.
3. Other pre-vent checks
   • The Oplevs had drifted significantly, I re-centered ITMs, ETMs and BS after aligning the arm cavities and green beams in the POX/POY lock state. See Attachment #1.
   • I locked the PRMI on carrier and used this config to re-center the PRM Oplev, see Attachment #2.
   • No further action was taken regarding SRM oplev.
   • I checked the ALS noise, see Attachment #3. The X arm ALS has excess noise at ~100 Hz that certainly wasn't there previosuly - sigh. There is nothing I can find about any changes made at EX in the elog.
   • Updated the "DriftMon" values, though I guess we don't even really use this anymore these days?
   • Re-relieved the IMC WFS offsets.
   • Cut the input power to the IMC from 1.007W to 100.1 mW (both numbers measured with Ophir power meter).
   • Replaced the 10% beamsplitter in the MCREFL path on the AS table with a Y1 HR mirror. Note that there is no beam on the IMC WFS in this configuration.
   • Was able to lock the IMC on low power to a TEM00 mode - need to set up the low power autolocker. The IMC autolocker is now set to the low power settings, and I've tested it locks a couple of times. Attachment #5 shows the low power lock in StripTool.
   • Walked around and looked at all the bellows - the jam nuts are up against their stops, and I can't move them with my hands, so I think that's okay.

If everything else looks good, I'll start letting the dry N2 into the main volume after lunch.

BTW, nice video! @ Koji, How difficult was it to edit it into this form? 

1. Jam nuts - checked that none of the nuts move by hand, which according to Steve, is sufficient. I recorded photos of all of them here.
2. Lab particle count - at the SP table, I measured 25,910 / cf @ 0.3 um and 1220 /cf  @ 0.5 um. Steve's guidance is that the latter number should be less than 10,000, so no issues there.
3. Valves - closed all the annuli off, and also closed VM1, VA6 and V1. The procedure calls for VM3 to be opened, presumably so that the RGA remains pumped, but I see no reason why we can't just leave that RGA volume valved off.
4. Started letting N2 into the main volume at ~3pm local time, by opening VV1. Attachment #1 shows the valve configuration adopted for this vent. Went up to 25 torr and then switched over to the "Ultra Zero" instrument grade air cylinders.
5. Aimed for 3-5 torr/min rate of pressure increase locally. The full vent trend can be seen in Attachment #2.
6. Stopped letting air into the main volume when P1a pressure was 700 torr, at which point I disconnected the cylinders from the main volume, and left VV1 open so that the IFO equilibriates to the lab air pressure. I used 4 full cylinders of the instrument grade air, which is par for the course. 
7. Since we anticipate opening the ETMY and output optics chambers, I also vented those annuli. The final state of the valves as I am leaving it for the night is shown in Attachment #3.
8. I re-aligned the IMC mirrors so that I could lock the IMC at low power once again. Indeed, IMC trans of ~1400 cts was realized (see Attachment #4), c.f. ~1500 earlier today, I thought this is fine and didn't optimize further. I think the policy is to not lock the IMC in air unless necessary, so I'm leaving the PSL shutter closed for the night.
9. The ITMs, ETMs and BS were re-aligned such that the Oplev spots are centered. I can see some higher order resonances of the green beams. The AS spot is fringing (the two ND=0.5 filters on the AS CCD camera were removed for better visibility). So this is fine as far as alignment is concerned.

We are now ready to take the doors off. I've already done the basic prep work (loosened bolts, cleaned chamber, carts for tools, fresh ameristat on portable HEPAs etc).

[koji, rana, gautam]

This morning, we did the following;

1. Removed the ETMY chamber heavy door. It is stored on the rack at the east end.
2. Removed OMC chamber heavy door. It is smaller than the other chamber doors, so doesn't sit on the standard size rack we have. So it is laid flat, on a clean sheet of ameristat, on a cart next to the NE corner of the PSL table.
3. After taking some photos of the chamber and making sure the position of the suspension tower was marked by some stops, we removed the ETMY cage and moved it to the cleanroom area. The optic was EQ stopped during the transport, and the OSEMs were removed. The ETMY Oplev HeNe was turned off and the PSL was shuttered to allow us to work without goggles.
4. The broken off magnet was retrieved from inside the OSEM. The shadow sensor voltage recovered a value of ~800cts, which means the LED/PD pair on the UR OSEM seems to work fine, and it was in fact the magnet blocking the PD that was the issue.
5. In the cleanroom area, we removed the optic from the wire loop and placed it on the magnet gluing fixture. The wire is intact (for now), so there is some hope of re-suspending it in the same loop.

The OSEMs remain in the EY vacuum chamber. The next set of steps are:

1. Clean the EP30-2 residue from the broken magnet joint - this will require some scrubbing with an acetone soaked scrub or similar implement.
2. Reglue the broken magnet.

We will most likely work on this tomorrow. At ~1615, I briefly opened the PSL shutter and tweaked the IMC alignment. We will almost certainly change the pointing into the IMC when we remove the old OMC and rebalance that table, so care should be taken when working on that...

To be a bit more clear about what we are going to do in the OMC chamber, I marked-up some photos, see Attachments #1 and #2.

1. OM5 will be rotated to bring the IFO AS beam straight out without any splitting to the OMC.
2. OMMT, OMC, DCPD, DCPD transimpedance amp, and all peripheral optics associated with these components, will be removed. Many of these components are mounted on a breadboard and so removing that breadboard will take care of it. These are marked with pink Xs.

I anticipate that after this work, the only components on the table will be 

1. IM1, to send the PSL beam to the IMC.
2. OMs 5 and 6 to bring the IFO AS beam out onto the AP table (in principle, we could try and eliminate both these optics, if the AS beam happens to exit through one of the viewports cleanly, we will not have any intervening objects in the way once the OMC and peripherals are removed).
3. MMT2 for mode-matching the IMC transmission to the interferometer mode.

Are we in agreement with this plan?

See #15656 for the updated photo

I believe the mirror next to IM1 is for the green beams to be delivered to the PSL table. I think we still want to keep it. Otherwise, the plan looks fine.

Good point - looking back, I also see that I already removed the mirror at the SW corner of the table in 2016. Revised photo in Attachment #1. There is an optic on the east edge of this table whose purpose I'm not sure of, but I'm pretty sure it isn't essential to the main functionality and so can be removed.

I got a call from Calum ~830am today saying some facilities people entered the lab, opened the south entrance door, and tripped the alarm in the process. I came to the lab shortly after and was able to reset the alarm by flipping the switch on the alarm box at the south end entrance to "Alarm OFF". Then, I double checked that the door is closed, and re-enabled the alarm. The particle count at the SP table is not unusually high and the lasers (Oplev HeNe and AUX X) were still on, so doesn't look like any lasting damage was done. The facilities people were apparently wearing laser safety goggles.

[koji, gautam]

1. Glued broken off magnet - curing overnight with lamp to slightly elevate temp for curing.
2. Remvoed material from OMC chamber as per the plan. This is all sitting wrapped up in foil on a cart for tonight, we should figure out a better storage plan eventually.

The IMC isn't resonant for a TEM00 mode at the time of writing - we are waiting for the stack to relax, at which point if the IMC isn't resonant for a TEM00 mode, we will tweak the input pointing into the IMC (we want to use the suspended cavity as the reference, since it is presumably more reliable than the table from which we removed ~50 kgs of weight and shifted the balance.

I am working on the setup of a CDS FE, so please do not attempt any remote login to the IPMI interface of c1bhd until I'm done.

[koji, gautam]

1. Tweaked last steering mirror before PSL beam is launched into vacuum to get the IMC resonant for the TEM00 mode. Less than 1/8 turn in each Pitch and Yaw was required, and we recovered MC2T ~ 1300 cts (c.f. the expected ~1500 cts when the cavity is well aligned, but we didn't touch the IMC mirrors).
2. ETMY magnet re-gluing
   • Regluing was successful. Pickle picker came off smoothly. 
   • We performed the razor blade test on all 6 magnets, the integrity of the joints seems uncompromised.
3. ETMY cleaning
   • Applied F.C. to HR face, HR edge (to remove some residual stains) and AR surface (taking care to go around the magnets).
   • A fresh bottle each of spectroscopic grade acetone and isopropanol was dispensed into clean beakers for the work.
   • Curiously, in the lighting conditions of the cleanroom, both HR and AR faces looked surprisingly clean, even when illuminated with the green flashlight.
   • However, when we took it to the east end and looked at it again with the green flashlight, just before putting it into the chamber, we saw all sorts of stains and markings on the HR side.
   • The central aperture looks fine - the contamination did not happen during the transport so we feel confident not to futz around more with this and since we cleaned it extensively in the cleanroom, we opted not to do another round of cleaning in the chamber.
4. ETMY re-suspension (on the existing wire loop)
   • The back OSEM-holding plates were removed for access reasons.
   • The optic was rested onto the EQ stops. 
   • Patient nudging of the wire around the optic allowed us to eventually get the wire into the V-grooves in the standoffs.
   • The wire was not damaged! (at least, to eye, free-swinging test is going on now, we will see what the eigenfrequencies and Qs are)...
   • We confirmed that the wire is seated in the v-groove, and took some close up photos.
5. ETMY pitch balancing
   • A HeNe was brought into the cleanroom and mounted at 5.5" beam height.
   • Level-ness of the beam (i.e. parallel to the optic table) was done coarsely with a spirit level placed on the HeNe holder, and more fine adjustment was done by checking the beam height at the launch point, and again ~2 meters away.
   • All EQ stops were backed off to ensure the optic was really "free-swinging" - no OSEMs either at this point, since all this was done in the cleanroom.
   • The return beam hit the HeNe's output aperture, so we were satisfied that the pitch balancing was good to ~1mrad.
6. ETMY transport back to EY
   • Since we want to maintain some tension in the wires, the bottom EQ stops were not raised as much as they were when the tower was transported from EY to the cleanroom.
   • As a result, we felt it'd be better for a human (Koji) to carry the cage (rather than cart it, the idea being that there would be fewer vertical impulses due to bumps on the ground etc).
   • The transport went smooth.
7. EY chamber work
   • Optic was placed back in its original position (marked by some L-clamps).
   • In preparation for alignment work, I undid all the CDS changes I made to facilitate the temporary 3-magnet actuation. This necessitated a reboot of c1auxey VME crate.
   • OSEMs were inserted - best effort to half light as usual, and we tried to replicate the rotation in the mounts as closely as possible to what was the case before (using a photo from the last vent as reference).
   • Coarse alignment was done by making the Oplev return beam hit the QPD.
   • Better alignment of the Y-arm cavity axis was done using the green beam. 
   • IMC was locked - the beam alignment target was used to ensure the IR beam was hitting ETMY at the correct height (by moving TT2), and then the ETMY angular alignment was improved to make the return beam go back roughly collinearly.
8. Y-arm POY locking
   • Satisfied with the alignment, we returned to the control room (light doors back on EY chamber) - cavity alignment was tweaked further to make the TRY resonances as high as possible.
   • I could lock the Y-arm length to the PSL frequency using POY as a sensor - TRY was ~0.04-0.05 (since IMCT is 1/10th its usual value, we expect it to be 0.1, so this is not bad).
9. OMC chamber work
   • Tied down the balance weights that were previously placed after the OMC and peripheral optics were removed.
   • Checked the table leveling.
   • Checked that the AS beam is reasonably well centerd on OM5 and OM6. Took some photos.
   • Checked that the IMC could be locked after this work - it could.

So all the primary vent objectives have been achieved 🙌 . The light doors are on the chamber right now. I'm measuring the free-swinging spectra of ETMY overnight. Barring any catastrophic failures and provided all required personnel are available, we will do the final pre-close-up checks, put the heavy doors back on, and pump down starting 10 am Monday, 9 Nov 2020. Some photos here.

Attachment #1 shows the main result - there are 4 peaks. The frequencies are a little different from what I have on file for ETMY and the Qs are a factor of 3-4 lower (except SIDE) than what they are in vacuum, which is not unreasonable I hypothesize. The fits suggest that the peak shape isn't really Lorentzian, the true shape seems to have narrower tails than a Lorentzian, but around the actual peak, the fit is pretty good. More detailed diagnostic plots (e.g. coil-to-coil TFs) are in the compressed Attachment #2. The condition number of the matrix to diagonalize the sensing matrix (i.e. what we multiply the "naive" OSEM 2 Euler basis matrix by) is ~40, which is large, but I wouldn't read too much into it at this point.

I see no red flags here - the PIT peak is a little less prominent than the others, but looking back through the elog, this kind of variation in peak heights doesn't seem unreasonable to me. If anyone wants to look at the data, the suspension was kicked every ~1100seconds from 1288673974, 15 times.

The PMC servo railed and so I re-locked it at ~half range. I've been noticing that the diurnal drift of the PZT control voltage has been larger than usual - not sure if it's entirely correlated with temperature on the PSL table. Anyway the cavity is locked again so all is good.

I was able to boot one of the 3 new Supermicro machines, which I christened c1bhd, in a diskless way (with the boot image hosted on fb, as is the case for all the other realtime FEs in the lab). This is just a first test, but it is reassuring that we can get this custom linux kernel to boot on the new hardware. Some errors about dolphin drivers are thrown at startup but this is to be expected since the server isn't connected to the dolphin network yet. We have the Dolphin adaptor card in hand, but since we have to get another PCIe card (supposedly from LLO according to the BHD spreadsheet), I defer installing this in the server chassis until we have all the necessary hardware on hand.

I also have to figure out the correct BIOS settings for this to really run effectively as a FE (we have to disable all the "un-necessary" system level services) - these machines have BIOS v3.2 as opposed to the older vintages for which there are instructions from K.T. et al.

There may yet be issues with drivers, but this is all the testing that can be done without getting an expansion chassis. After the vent and recovering the IFO, I may try experimenting with the c1ioo chassis, but I'd much prefer if we can do the testing offline on a subnet that doesn't mess with the regular IFO operation (until we need to test the IPC).

Basic IFO alignment checks were done.

1. IMC could be locked - I tweaked the cavity alignment a little to maximize the MC transmission.
2. Y arm can be made resonant for a TEM00 mode of the main beam. I can't run ASS successfully in this low power config. I can see that the axis isn't great, the spot is visually off-center on ITMY, but we should have plenty of actuator range to correct for this with TT1/TT2.
3. X arm shows IR mode flashes in the TRX QPD. The green beam can be made resonant for a TEM00 mode, but that alignment doesn't yield the largest IR resonant peaks in TRX. I suspect it is due to the mis-alignment of the beam axis.
4. AS beam was aligned onto the CCD. I could see clean Michelson fringes by tweaking the BS alignment. On the AP table, I noticed that the beam on the first steering mirror after the AS beam exits the vacuum is a little high. We can easily resolve this by tweaking OM6 pitch a bit, but even if we don't I don't see any major issues as there is plenty of clearance w.r.t. the viewport when the beam exits the vacuum.
5. With the PRM aligned, I can see the REFL beam on the CCD, and it doesn't look clipped.
6. I didn't bother to align the green beams to the arm cavities or re-center the Oplevs - is this necessary? It is a step in the pre-close up checklist, so maybe we should do it... The green transmission does reach the PSL table...

Tomorrow, we should do some visual checks of the chambers / EQ stops on ETMY etc but I don't see any major problems at the moment...

> I didn't bother to align the green beams to the arm cavities or re-center the Oplevs - is this necessary? It is a step in the pre-close up checklist, so maybe we should do it... The green transmission does reach the PSL table...

I don't think so. The beam is reaching the PSL, so we have no motivation to change the green alignment. Regarding the oplev, the green refl should come back to the PDH PD and this gives us additional beam reference. As soon as we find the green resonance after the pumping, we can tweak the green axis so that the spots on the mirrors become reasonable (as well as the green trans CCD on the PSL table).

[koji, rana, gautam]

1100 - EY chamber inspected, no issues were found --> EY heavy door on

1200 - OMC chamber was inspected. OM6 was marginally tweaked to bring the beam down a little in pitch, and also a little northwards in Yaw. --> Heavy door on.

1230 - Pumpdown started. Initially, the annuli volume was pumped down. The procedure calls for doing this with the small turbopumps. However, V7 was left open, and hence, in the process, the TP1 foreline pressure (=P2) hit ~30 torr. This caused TP1 to shutdown. We were able to restart it without issue. This case was not caught by the interlock code, which was running at the time. It should be recitified.

1330 - OMC breadboard clean optics and DCPD hardware were wrapped up and packed into tupperware boxes and stored along the south arm. OMC cavity itself, the OMMT, and the breadboard the OMC was sitting on are wrapped in foil/Ameristat and stored in cabinet S13, lower 2 shelves.

1915 - P1a = 0.5 torr pressure reached. Switched over to pumping the main volume with TP1, backed by TP2 and TP3, which themselves are backed by their respective dry pumps and also the AUX drypump for some extra oomph. All cooling fans available in the area were turned on and directed at the turbo pumps. RV2 was used to throttle the flow suitably.

It was at this point that we hit a snag - RV2 has gotten stuck in a partially open position, see Attachment #1. We can see that the thread doesn't move in response to turning the rotary dial. Fortunately, the valve is partially open, so the main volume continues to be pumped - see Attachment #2 for the full history of today's pumping. We are leaving the main volume pumped in this configuration overnight (TP1 pumping main volume backed by TPs 2 and 3, which are in turn backed by their respective drypumps and also the AUX dry pump). I think there is little to no risk of any damage to the turbo pumps, the interlocks should catch any anomalies. The roughing pumps RP1 and RP3 were turned off and that line was disconnected and capped.

What are our options?

1. The main volume is able to reach the "nominal" pressure of 1e-5 torr, just takes longer.
2. At some point, we may be able to pump the main volume directly with TP2 and TP3 - it is unclear at this point whether it's better to have the conductance limited TP1, which has a higher pumping capacity, or have the smaller TPs 2 and 3 pump through a larger conductance. 
3. Depending on how low the ultimate pressure gets, we may be able to run the usual IFO activities until the replacement pressure gauges arrive in ~1 week, at which point we can vent the pumpspool (leaving the main volume isolated) and either repair this valve or replace it with one of the spares we have.

We need some vacuum experts to comment. Why did this happen? Is this an acceptable failure mode of the valve?

KA Ed:
2230 - P1a = 0.025 torr. The pressure is coming down with log-linear scale. x0.1 per 2.5 hours or so.

Main volume pressure as of 11:30AM 2020/11/10

I've uploaded some more photos here. I believe the problem is a worn out thread where the main rotary handle attaches to the shaft that operates the valve.

This morning, I changed the valve config such that TP2 backs TP1 and that combo continues to pump on the main volume through the partially open RV2. TP3 was reconfigured to pump the annuli - initially, I backed it with the AUX drypump but since the load has decreased now, I am turning the AUX drypump off. At some point, if we want to try it, we can try pumping the main volume via the RGA line using TP2/TP3 and see if that allows us to get to a lower pressure, but for now, I think this is a suitable configuration to continue the IFO work.

There was a suggestion at the meeting that the saturation of the main volume pressure at 1mtorr could be due to a leak - to test, I closed V1 for ~5 hours and saw the pressure increased by 1.5 mtorr, which is in line with our estimates from the past. So I think we can discount that possibility.

Looking back through the elog, 1mtorr is the pressure at which it is deemed safe to send the full power beam into the IMC. After replacing the HR mirror in the MCREFL path with a 10% reflective BS, I just cranked the power back up. IMC is locked. With the increased exposure on the MC2T camera, lots of new scattered light has become visible.