1. Oplev HeNe at ETMY was replaced, see here for my earlier discussion on this.
   • I thought this is a good idea since we want the Oplev as a coarse reference and it'd not be ideal if this HeNe dies during the time the optic is out of the chamber.
   • New HeNe had 2.8mW of power output as measured with Ophir power meter. This is in line with what is expected from these Lumentum heads.
   • I labelled the head with the power output and today's date, re-aligned the Oplev reflection onto its QPD. 
   • After this work, the Y-arm could be locked without the huge angular fluctuations that was visible earlier, 👍 .
2. GTRY anomaly
   • I actually judged that there is no anomaly.
   • The GTRY CDS indicator is actually quite useless - the ADC saturates at ~3500 cts (and not 32768 as you would expect from a 16 bit ADC but that's the well-known whitening filter saturation problem). This should be fixed, but this is a task for later.
   • I measured with a DMM the voltage when the TEM00 is locked to the cavity and GTRY is 0.3 (the nominal value these days), the DC voltage was ~5.6 V. The prompt reflection from the ETM registers ~6.5 V DC. So the mode-matching isn't stellar, but this is again a known issue, and can be fixed later.
3. Other pre-vent checks
   • The Oplevs had drifted significantly, I re-centered ITMs, ETMs and BS after aligning the arm cavities and green beams in the POX/POY lock state. See Attachment #1.
   • I locked the PRMI on carrier and used this config to re-center the PRM Oplev, see Attachment #2.
   • No further action was taken regarding SRM oplev.
   • I checked the ALS noise, see Attachment #3. The X arm ALS has excess noise at ~100 Hz that certainly wasn't there previosuly - sigh. There is nothing I can find about any changes made at EX in the elog.
   • Updated the "DriftMon" values, though I guess we don't even really use this anymore these days?
   • Re-relieved the IMC WFS offsets.
   • Cut the input power to the IMC from 1.007W to 100.1 mW (both numbers measured with Ophir power meter).
   • Replaced the 10% beamsplitter in the MCREFL path on the AS table with a Y1 HR mirror. Note that there is no beam on the IMC WFS in this configuration.
   • Was able to lock the IMC on low power to a TEM00 mode - need to set up the low power autolocker. The IMC autolocker is now set to the low power settings, and I've tested it locks a couple of times. Attachment #5 shows the low power lock in StripTool.
   • Walked around and looked at all the bellows - the jam nuts are up against their stops, and I can't move them with my hands, so I think that's okay.

If everything else looks good, I'll start letting the dry N2 into the main volume after lunch.

BTW, nice video! @ Koji, How difficult was it to edit it into this form? 

I went to 40m yesterday at around 2:30 pm and Koji showed me how to acquire lock in different arms and for different lasers. Finally, we took a preliminary measurement of shaking the ETMX at some discrete frequencies and looking at the beatnote frequency spectrum of X-end laser's fiber-coupled IR and Main laser's IR pick-off.

Basic controls and measurement 101 at 40m

• I learned a few things from Koji about how to align the cavity mirrors for green laser or IR laser.
• I learned how to use ASS and how to align the green end laser to the cavity. I also found out about the window at ETMX chamber where we can directly see the cavity mode, cool stuff.
• Koji also showed me around on how to use diaggui and awggui for taking measurements with any of the channels.

Preliminary measurement for calibration scheme

We verified that we can send discrete frequency excitation signals to ETMX actuators directly and see a corresponding peak in the spectrum of beatnote frequency between fiber-coupled X-end IR laser and main laser IR pickoff.

• I sent excitation signal at 200 Hz, 250 Hz and 270 Hz at C1:SUS-ETMX_LSC_EXC channel using awggui with an amplitude of 100 cts and gain of 2.
• I measured corresponding peaks in the beatnote spectrum using diaggui.
• Page 1 shows the ASD data for the 4 measurements taken with Hanning window and averaging of 10.
• Page 2 shows close up Spectrum data for the 4 measurements taken with flattop window and averaging of 10.
• I converted this frequency signal into displacement by using conversion factor or .

If full interferometer had been locked, we could have used the DARM error signal output to calibrate it against this measurement.

Data

1. Two arms / BS / PRM / SRM were aligned. (Attachment 1)
2. IMC was aligned by WFS and the WFS offsets were offloaded.
3. Suspension Status Snapshot (Attachment 2)
4. Oplevs are aligned (Attachment 3)
5. Xarm green was aligned in the daytime. Xarm green refl DC (C1:ALS-X_REFL_DC_OUTPUT) was 620 (aligned) ~1300 (drifted). When unlocked, it was 3750.
6. Yarm green: I saw no flash. We don't have functional PZT alignment since the ASY M2 PZT got broken. I went to the Yend. Something funky is going on with the Yend green. I struggled to have any flash of the cavity. The apertures were not so precise. I finally got TEM00 locked, but the modematching seems exteremely low (like 1/1000?). Basically I saw no power reduction of the refl when the cavity is locked. So at least the cavity was locked but we might need to revisit when we open the chamber
   ==> Gautam thinks it was not like that. So he will check the green alignment tomorrow (Thu).
7. Item checking: I familialized myself with the yend crane operation. Today I learned that there is a power switch on the wall (Attachment 4). The yend has two heavy door storages (Attachments 4/5). The slings to lift the heavy door are in the crane cabinet along with the y arm (Attachment 6). I didn't yet try to find the "hammer" to hit the door if the o-ring stuck too strong, although that's optional.
8. We want to reduce the PSL power. But Gautam wants to use the arm locking with the nominal power, it will be done tomorrow by him.
    
9. The last thing is to check the green trans power. I noticed that the green trans beams are blocked by an HWP for the BHD LO path on the PSL table. (Attachment 7)
   The HWP was moved and the process was recorded in the movie (Attachment 8). The fiber output was monitored by the BHD DC (aka AS110 DC) with the AS path blocked. The fiber output of 22.6mV (offset -2.5mV) was improved tio 29.1mV after the HWP move and the alignment adjustment.
10. Now the green transmissions are visible by the green PDs. Attachment 9 shows the trans and ref of each green beams with and without locking to TEM00. The questionable green TRY was ~0.3. If we compare this with the histrical data (Attachment 10), it is about 1/4 of the value in the past. It's not too crazy but still quite low.

At this point, I'm leaving the lab. All the suspensions (incl SRM) are aligned. PSL/GRX/GRY shutters were left open.

Ok I was using the PD in the black mount because Rana recommended it a few weeks ago.

Regarding the M2ISS, I acquired the hardware from QIL some months ago, including a circuit board, and 2 PDs. These had LEMO outputs though (not BNC), and the mounts are not 4". These photodiodes are what I'm using as the airBHD DCPDs right now, and some photos are here - are these the photodiodes you mentioned? Or are there yet more M2ISS photodiodes? I remember Johannes had some custom mounts extruded to make them 4" high, do you mean those? Can I retrieve them his Cryo setup?

BTW, my elog scraping shows only one spectra from Stefan in the ATF elog, and the performance there is more like 1e-7/rtHz @ 100 Hz, and that’s using a dedicated high BW servo circuit, not the SR560. Am I just missing the measurement of 2e-8/rtHz?

that little PD in the black mount was never very good. The AD829 is not a good opamp for transimpedance and especially not good for low frequencies. Stefan Ballmer and I were able to get 2e-8 out of these (@100 Hz) many years ago.

I wonder if we have some of Zach's M2ISS photodetectors around, perhaps in QIL or Cryo. I doubt that any of them are in use now. Those had good performance nad BNC output.

I'm starting the model restarts from remote. Then later I'll show up in the lab to do more hard resets.
==> It seems that the RFM errors are gone. Here are the steps.

1. Shutdown all the watchdogs
2. login to c1iscex. Shutdown all the realtime models: rtcds kill --all
3. login to c1iscey. Shutdown all the realtime models: rtcds kill --all
4. run scripts/cds/rebootC1LSC.sh on pianosa
5. reboot c1iscex
6. reboot c1isxey
7. Wait until all the machines/models are up by the script
8. restart c1iscex models
9. restart c1iscey models
10. some IPC errors are still visible on the CDS status screen. Lauch c1daf and c1oaf

I wanted to look into the ISS situation. Some weeks ago, I found the PD that was previously used as the in-loop photodiode. I wanted to use this and measure the open-loop RIN at a few places (to see if there's any variation and also to check its functionality). However, I didn't get very far tonight - for a start, the PD height is 3" (while our beam height is 4" everywhere outside the vacuum), and I needed to put together a circuit to supply the 5V bias and +/- 15 V since the transimpedance is done on the head. I was only able to do a low-level functionality test tonight, checked that the DC voltage output varied linearly with the incident power (calibrated against an NF1611 photodiode, data will be put up later). I didn't get to measuring any noise performance - is an incandescent light bulb still shot noise limited at ~10 Hz < f < 10kHz? Some notes:

1. The PD is DC coupled, and has a transimpedance of 1 kohm (inverting AD829 does the transimpedance).
2. Probably a daughter board should be made that supplies the DC power voltages and rotues the output signal to something more convenient like a BNC connector. This daughter board can then also implement a DC coupled path (for monitoring) and AC coupled path (for servoing, fc to be determined).
3. SR560 based ISS was implemented some years ago but I think the improvement was only seen above 100 Hz, and that too was marginal, the stabilized RIN was 10^-6 (monitored on an out-of-loop photodiode I think, but unsure). We'd probably want to aim for at least an order of magnitude better. Unclear at this point why more suppression wasn't possible back then, was it just insufficient loop gain, or was the sensing noise too high? To be investigated.

Unconnected to this work - this problem reared its ugly head again (i noticed it yesterday morning already actually). I don't have the energy to embark on a fix tonight, Koji is going to be in the lab all day tomorrow and so he will fix it.

Apart from the questionable wiring on the Acromags, one other important difference is in the way the connections were made between the old VME crates to the Eurocrate backplanes, and how we do it now. The thick cables had their sheilds connected to the eurocrate ground (or at least, there was a dedicated ground lug on those cables which we screwed on to the ground terminals on the Eurocrate backplanes). However, in our current configuration, we interface the Acromag ADCs and DACs to the backplane via these adaptor boards. The shields of the DSUB cables are presumably NOT connected to the Eurocrate grounds. This should also be investigated as one potential cause of the grounding issue - while on some of the Eurocrate modules, the P1/P2 connectors may have either the "A" or "C" row of connectors shorted to ground, some may not, and the TTFSS may suffer from such an issue?

Note that we have this problem in all of the slow machines that were upgraded to Acromag (if this turns out to be the issue). 

In fact, the problem was the grounding issue (presumably on the IOO racks).
A temporary differential receiver at the TTFSS side was built using an SR560 and a few ponoma cables. This removed the structures ~850Hz.

The MC Servo Output was disconnected from the TTFSS box and monitored with SR785. The 850Hz structure was kept visible no matter what cables, including all the acromag DB cables, were removed. This made me suspicious about the measurement setup. The SR785 was connected to an AC power strip under the SP table and this was too far from the IOO rack.

The SR785 was connected to the AC power strip on 1X2, and now the difference becomes clear. No matter if the acromag cables are connected or not, the connection (particularly ground connection) between the MC servo module and the TTFSS box causes the MC servo output contaminated. (Comparison between Blue and Orange of Attachment #1). During the measurement, the EPICS switch for the fast path was disengaged (=no signal) and the VCO gain (...so called. It's just the MC Servo Gain) was set to be 0dB.

To test if the differential receiving of the MC Servo Output at the PSL helps to reduce this noise, I've built a simple (hacky) differential receiver using an SR560. (Attachment #2)
This kept the noise level same as the disconnected case (Comparison between Green and Orange of Attachment #1, I don't think the difference between them is not significant), while the IMC is locked as before.
Note that we can see that the 36kHz line was significantly reduced. Did we remove this annoying noise?

After talking with Gautam, we decided to leave this configuration while the SE-Diff cable was replaced with a more robust one. (See Attachment #3)

The PSL laser frequency performance was evakluated in the following two ways as we did last week:
1) Use the beat frequency of the free running PSL and the Y-end laser (Attachment #4). The PSL shutter was closed and thus the IMC was not locked.
2) Use the IMC MCF while the IMC was locked. (Attachment #5)

For both cases, the improvement was confirmed.

I also tried to check the reported issue by Gautam on this elog. He used 1Hz BW, but I cheated with 16Hz BW and 10x12.8kHz span PSDs. (Attachment #6)

For the measurement, IN1 GAIN of the IMC Servo was set to be 0dB and the OUT2 was switched to monitor the IN1 noise, while IN1 was terminated by a 50Ohm.

As I mentioned above, the AC power of SR785 was taken from a 1X2 power strip. Is this the reason for the power line forest look less severe compared to the previous case???
Anyway, I tried to use the same differential receiving technique (but with gain of x100) to see if this helps. The differential receiver helped to reduce the structure above 50kHz. The floor noise level was observed to be higher. I didn't pursue this any further, but the forest of the power line looked like a part of the measurement noise. This is indicative that the grounding condition on 1X2 is really not great and we need to review the configuration of the acromag grounding.

The particle counter on the 40m PSL was removed. The package was made together with the OMC lab particle counter (see the packing list below).

The kit was picked up by Radhika for a python code to read out the numbers.

=== Packing List ===

• MET ONE 227A particle counter
  • used at the 40m. It has the particle reading and the temperature reading.
• Power supply adapter (AC/DC) for 227A
  • Caution: It is not compatible with GT-321.
• MET ONE GT-321
  • I found another type of particle counter in West Bridge.
• Power supply adapter (AC/DC) for GT-321. (Labeled "for GT-321")
  • Caution: It is not compatible with 227A.
• DB9 cable for GT-321
• Air Filter G3111
  • When you run a particle counter attach this filter instead of the dust collecting cup to keep the air in take of the particle counter clean. This should keep the particle level down to zero.
     

[Koji, Rana]

We wanted to track down the excess noise seen in MC_F and other places (see the previous report by Gautam)

Setup1: The IMC was locked and MC_F signal between 500 and 1500Hz was observed. The DTT template was saved as /users/Templates/MC/MCF_noise_201023.xml

- Suspected mech resonance/jitter coupled with clipping or any other imperfections. Poked the various optics and optomechanics on the table. Basically no change. If we tap the laser chassis and the optics close to the laser source, we occasionally unlocked the IMC

- When we touched (lifted) the Innolight controller box from the shelf, for the first time we saw a significant change in the shape of the noise spectrum. The peak around the 700Hz shited towards lower frequency by a few %. Other peaks have no obvious change in the shapes and the heights.

- While observing the MC_F signal on the laptop, we went to the back of the laser controller. Placing a hand close to the fan clearly changes the peak frequency lower. By temporarily disconnecting the fan from the power supply for a short moment, the 700Hz peak could be eliminated. We also tried to see the noise level with the slow thermal servo and diagnosis DB cable disconnected, but we didn't see any significant change of the noise level.

Setup 2: Using the ALS phase tracker, we can observe the relative freq noise of the PSL laser and the ETMY AUX laser without any servo involved. This way we can freely disconnect any cables from the lasers. The measurement template for DTT was saved as /users/Templates/ALS/Y_ALS_FINE_PHASE_OUT_102320.xml

- Noise spectrum before disconnecting the cable (REF0, RMS REF1)

- The Fast PZT input to the PSL was disconnected => This made all the peaks (including the 700Hz) disappeared (REF2, RMS REF3)

- The Fast PZT input was restored as before, then the chain was disconnected at the input of the HV PZT driver (Thorlabs) => Again, this made the peaks disappeared (REF4, RMS REF5)

- The chain was disconnected at the input of the TTFSS box => Again, this made the peaks disappeared (REF6, RMS REF7)

- Disconnected the demod input and the AO cables from the IMC servo board => This made the peaks came back (REF8)

- Disconnected all the input/peripheral cables from the IMC servo board except for the connection to the TTFSS box => Still the excess noise was observed  (REF9)

- In addition to the above, the cable to the FSS box was disconnected but the ground was still touching the MC servo board => This made the peaks disappeared (REF10)

The conclusion is that the noise is injected from the main circuit of the IMC servo board.

Next time we will check if the backplane connection is doing something wrong. Also, we'll test if the presence of the RF signals does something bad to the IMC board via EMI and RFI.

We have reverted the connection and tested if we lock the IMC and Y arm. ==> We saw at least they were locked for a short period. The things are still stabilizing, but left them turned on so they keep trying to lock automatically for the night.

Andrew made a battery-powered 0.7 nVrtHz input-referred noise pre-amplifier for gain of 200. That might help you.

RXA: 0.7 nV is OK if you're not interested in low noise measurements. Otherwise, we have the transformer coupled pre-amp from SRS which does 0.15 nV/rHz and the Rai Weiss FET amp which has 0.35 nV for high impedance sources.

It's not so easy to directly measure this I think, because the filtering is rather aggressive. Attachment #1 shows the measured transfer function (dots) vs the model and Attachment #2 shows the noise. I think this checks out - but I can't definitively rule out some excess noise at 100 Hz from this stage. Because the gain of the HV stage is x31, we'd need a preamp with better than 1nV/rtHz to directly measure the noise I guess. The Acromag noise model in Attachment #2 is based on a measurement I describe here.

what is the noise level before the HV stage? i.e. how well is the acromag noise being filtered?

I found this H1 alog  entry by Izumi confirming that the calibrated channels CAL-CS_* need the same dewhitening filter.

This encouraged me to download the PRCL and MICH data and using Jon's example notebook. I incorporated these noise spectra into the MCMC simulation. The most recent results are attached.

I am still missing:

• Laser frequency noise
• Laser RIN
• Estimation of the LO phase noise
• Estimation of the BHD breadboard angular noise

Also, now the MCMC repeats a simulation if it doesn't pass the RF PDs test so the number of valid simulations stays the same. I'm still not sure about why the A+ simulations are much more robust to these tests than aLigo simulations.

I packaged the HV coil driver into a 2U chassis, hoping for better shielding from pickup. There is still considerable excess noise in measurement vs model around 100 Hz, see Attachment #1. The projected displacement noise from this noise contribution is shown in Attachment #2 - I've also plotted the contribution from the 4.5kohm (planned value for fast path series resistance) for comparison. Attachment #3 has some photos of the measurement setup so if someone sees some red flags, please let me know.

• The noise was measured with the output load connected to a 20ohm load resistor, to simulate an OSEM.
• The input signal was driven with an Acromag, to try and mimic the actual operating conditions as closely as possible (although the fast path input was left unconnected).
• The KEPCO switching HV power supplies were used to power the unit.

I've run out of ideas to try and make the measurement cleaner - the presence of the rather prominent power line harmonics suggests that this is still not perfect, but what more shielding can we implement? I have to make the measurement on the circuit side of the 25 kohm series resistor, so I am using some Pomona minigrabbers to clip onto the leg of the wirewound resistor (see photos in Attachment #3), so that's not great maybe, but what's the alternative?

So if this is truly the noise of the circuit, then while it's an improvement on the current situaiton, it's unsatisfying that such a simple circuit can't match the design expectations. But how do we want to proceed?

I set up an action cam (DJI OSMO Pocket) and brought it back to the 40m. The kit is now placed in the control room cabinet together with the Canon DSLR.

I might have left the USBC chaging cable at home this time. Will bring it back next time.-> The cable was returned to the kit on Oct 23rd.

Alaska M7.5 20:54UTC https://earthquake.usgs.gov/earthquakes/eventpage/us6000c9hg/executive

I looked at the suspensions. The watchdogs have not been tripped.

IMC was locked but continually shaken. (and occasional unlock)

I'll bring a file binder "40m wiring diagram" to home at the next chance.
There is another one on the shelf in the control room.

(I thought I put it in my bag, but it looks like that I left it somewhere around the fax area)

I entered 40m today at around 1:10 pm and left by 1:50 pm. I entered 104 through the machine shop entry. I took top view single picture photos of ITMY, BS, AP, ITMX, ETMX and ETMY tables. The latest photos will be put here on the wiki soon.

Pushed another update to MCMC simulation. This includes:

• Added new imbalances: ITM transmission, ITM & ETM RoCs.
• Added new static offsets: DHARD, DSOFT, CHARD, CSOFT. All pitch. The RMS is calculated from the data Jon fetched with /input_noises/input_noises.ipynb.
• SRCL noise ASD and RMS are now taken from data in /input_noises.
• RF PD diagnostics were redone: Instead of post-discarding marginal simulations, simulations are now discarded when one or more of the RF PDs demodulated signal does not cross zero when the associated DOFs are scanned by 1um in the offset state.

The DOFs<->RFPD associations I use are:

However, one thing that bothers me is that for some reason ~ 15 out of 160 aLigo simulations are discarded while none for A+. It can also be seen that the A+ simulations are more spread-out which might be related.

The new simulation results are attached.

The AC cord from the PSL HEPA variac to the junction box was replaced.
Now the HEPA is running at 70%

Showed up at the 40m at 7pm

Preparation

• Closed the PSL shutter.
• Closed the innolight shutter
• Turned off the HEPA mains switch
• Checked the HEPA fan rating: 115V 4.5A.
• Brought the thickest power cord from the wall stock: the rating is 125V 15A. This should sufficiently hold two HEPAs.

Cable Replacing

• Rechecked the wire connection. The new cord has green/black/white wires. And the colors agree with the color of the wires in the junction box.
• Removed the existing cord.
• Attached the new cord.
• Checked the variac AC plug. The terminals in the plug look normal and the AC plug looked sufficiently rigid.
• Checked the connection again. = OK

Testing

• Turned on the HEPA mains switch
• VairAC turned to 70%
• Checked the air flow - The HEPA fans are sucking the air = OK

Closing the work

• Closed the junction box.
• Cleaned up the roof
• Opend the innolight shutter
• Opened the PSL shutter
• Locked the PMC
• Locked the IMC  - found the transmission was ~80% of the pre-work due to misalignment of the PMC
• Aligned the PMC - this recovered the IMC REFL of ~5.2 when the IMC was unlocked

Leaving the 40m at 9:30pm

Memo: 40m wiring/Mask/Camera/Red Pitaya/Particle Counter

I entered 40m today at around 1:20 pm and left by 1:45 pm. I entered 104 through the machine shop entry. I did the following:

• I took photos and videos of the PSL table with lights on.
• I uncovered the AP table, took photos and video, and covered it back.
• I went to the X End table and took a video without opening the enclosure.
• Apart from flipping light switches, nothing else should have changed.

Summary of discussion between Koji and gautam on 14 Oct:

1. Koji questioned the accuracy of the "open loop" ASD shown here. While it may not be entirely accurate to compute the free-running (homodyne) phase noise simply by taking the arctangent of the I and Q signals (because the magnitude of the signal is also changing), gautam claims the estimate is probably still close to the true homodyne phase, especially since the ratio of the "in-loop" and free-running ASDs gives something that closely approximates the magnitude of the supposed OLG of the system.
2. Koji suggested the following tests:
   • Investigate the relative stability of the two RF signal generators involved in this system. Since the 44 MHz electrical LO signal (for demodulation) is generated by a separate IFR from the one used to imprint 11 MHz and 55 MHz phase modulation sidebands on the main PSL beam, we want to investigate what the drift is.
   • Try implementing an analog feedback loop using LB1005 - the idea being we should be able to implement higher bandwidth control, for better suppression of the high frequency noise (which looking at the ASD is not only due to seismic phase modulation of the IFO output field). Maybe some combination of this and the Marconi investigation would suggest why we have these forests of lines in the ASDs of the error signal?
   • Turn off the HEPAs on the PSL enclosure completely as a test, to see if that improves (i) phase noise due to air currents and (ii) mechanical pickup on the fiber producing  phase noise.

I tried all of these last night / overnight, here are my findings.

Analog locking of the homodyne phase:

See Attachment #1. 

• RF44_I was used as the error signal.
• The "C1:OMC-ZETA_IMON_OUT" channel is actually looking at the error signal monitor from the LB1005, and is uncalibrated in this plot.
• The "monitor" port on the demodulator board provides a convenient location for us to route the demodulated signal to an LB1005 box, while simultaneously digitizing both demodulated quadratures.
• Empirically, I found settings that could engage the lock. I also found that I couldn't increase the gain much more without destroying the lock. 
• The time domain signals look much "cleaner" in this analog feedback loop than when I achieved similar stabilization using the digital system. But I will quantify this more when I post some spectra of the in loop error signals.
• I will do some more characterization (loop TF measurement, error point spectrum in lock etc), but in summary, it looks like we still only have ~100 Hz UGF. So something in the loop is limiting the bandwidth. What could it be?
• The main problem is that the LB1005 isn't well suited to remote enabling/disabling of the lock, so this isn't such a great system.

Relative stability of two IFR2023s synchronized to the same FS725 Rb standard:

The electrical LO signal for demodulation of the 44 MHz photocurrent is provided by an IFR2023 signal generator. To maintain a fixed phase relation between this signal, and the phase modulation sidebands imprinted on the interferometer light via a separate IFO2023 signal generator, I synchronize both to the same Rb timing standard (a 10 MHz signal from the FS725 is sent to the rear panel frequency standard input on the IFR). We don't have a direct 44 MHz electrical signal available from the main IFO Marconi at the LSC rack (or anywhere else for that matter). So I decided to do this test at 55 MHz. 

• RF input of the demodulator was driven by 5*11.066209 MHz pickoff from the LSC rack.
• LO input of the demodulator was driven by 5*11.066209 MHz signal from the IFR2023 used for the RF44 demodulation setup.
• The outputs were monitored overnight. The RF44_Q channel had a DC level of nearly 0. So this channel is nearly a linear sensor of the phase noise between LO and RF signals.
• To convert ADC counts to radians, I offset the LO Marconi frequency by 100 Hz, and saw that the two quadratures showed pk-pk variation of ~24000cts. So, at the zero crossing, the conversion is 1/(24000/2) rad/ct ~83urad/ct.
• The result is shown in Attachment #2. The "measurement noise" trace corresponds to the RF. input of the demodulator being terminated to ground with a 50 ohm terminator.
• For comparison, I also overlay the phase noise estimate of an individual IFR from Rana. In his investigation, the claim is that the PLL that locks the IFR to the Rb timing standard has ~1kHz UGF, but if my measurement is correct, the relative stability between the two signal generators synchronized to the same timing standard already. degrades at ~1 Hz. Could be just a cts/rad calibration error I guess.
• In any case, we are far from saturating this limit in the homodyne phase lock.
• There are several sharp lines in this measurement too - but I don't know what exactly the source is. Of course the two marconis are plugged into separate power strips, so that may explain the 60 Hz lines and harmonics, but what about those that aren't a multiple of 60 Hz?

A look at the time domain signal:

With the Michelson locked on the dark fringe, the RF44 I and Q signals in the time domain are shown in Attachment #3 for a 1 minute stretch.

• The RF44 signal level bottoms out at ~40 cts. Okay, so this is the offset.
• However, the maximum value of the RF44 signal amplitude seems to be modulated in time. How can we explain this?

Per Koji's suggestion, I turned the PSL HEPA Variac to 50% just now, so that the power load through the burnt electrical cable is reduced by 75%.

Koji recommended that I can add whitening filters to suppress ADC noise easily. I added a filter before ADC in ALS loop with 4 zeros at 1.5 Hz and 4 poles at 100 Hz and added a reversed filter in the digital filter of ALS. This did not change the performance of the loop but significantly reduced the contribution of ADC noise above 1 Hz. One can see ALS_controls.yaml for the filter description. Please let me know if this does not make sense or there is something that I have overlooked.

Now, the dominant noise source is DFD noise below 100 Hz and green laser frequency noise above that. For DFD noise, I used data dating back to Kiwamu's paper. The noise contribution from DFD in the model is lower than the latest measured ALS noise budget post on elog. I'll look further into design details and noise of DFD.

Code, data, and schematics

The two ITM spares and two ETM spares are together stored in the optic storage (B110) at Downs. c/o Liyuan and GariLynn

[JV, GV]

We cleared up some space in the 1Y1 electronics rack to install the 3 new FE machines. I removed the current driver and laser from 1Y1, they are now stored in the E10 cabinet. I will upload some photos to gPhotos soon.

1. I think it's good to have all these FEs in one rack (at least the new ones) - we should then hook it up to an ethernet power source, so that we can remotely power cycle them. I think we have long enough cables to interface to expansion chasses / dolphin switches, but if not, I think it's still a good idea to have these machines in 1Y1 as it is the least sensitive area in terms of immunity to bumping some cable during setup work and disturbing the rest of the IFO.
2. We found that the rails that the Supermicros shipped with the servers seem to be just a little too narrow - we mounted these in the rack, but had considerable difficulty sliding the server units in. Once they are in, they don't slide smoothly. Is there some special trick to installing these? 
3. I spent a few minutes trying to get Debian 8 installed on these machines, so that the rest of the setup work could be done remotely - however, there appear to be some firmware issues and so I'm not gonna dive into this.
   • I couldn't find a disk image for Debian 8.5 which is what the KT wikl recommends, so the OS I tried to install was Debian 8.11.
   • The error that comes up is related to a "stalled CPU" - apparently this is related to some graphics driver issue (there's another forum page that suggests upgrading the BIOS, but I don't think that's the problem here).
   • Anyways, this part of the process is only to install some drivers and do the initial setup - these machines will eventually run a diskless boot from the image on FB, so who knows if there will be some other driver issues/hardware-software incompatibilities there 😱 .
   • We should also make an effort to set these machines up with IPMI, but I think we first need to install an OS and a CLI to setup the IPMI. My cursory browsing of the manual suggests that the initial setup maybe can be done without installing an OS, and then subsequent work, including OS install, can be done remotely. If someone reads more in detail and can provide me a step-by-step, I can follow those instructions (if they aren't available to come into the lab). See here for some brief documentation of how to access the IPMI.

Attachment #1: spectra of the phase noise between LO and IFO output fields sensed using the RF44 signal.

• Measurement setup:
  • LO an IFO fields are combined on a beamsplitter, with ~60% mode-matching efficiency.
  • One port of the BS goes to a DCPD.
  • The other port goes to an RF sensing photodiode, PDA10CF. The spec-ed dark noise NEP is ~12 pW/rtHz at 1.6 um, (so let's say 25 pW/rtHz) and transimpedance is 5kohms into a 50 ohm load. We can convert this to an equivalent sensing noise at the error point of this loop, though it's more likely that the electronics (demod, ADC etc) noise downstream dictate the sensing limit, which I measure by blocking light on the photodiode.
• The demodulation is done on one of the newly received D0902745 boards - this was just a more compact setup than many cascaded minicircuit components. We don't have the hardware to package this into a chassis to shield against electronics noise pickup yet, so I'm using a bench supply to power this for now (via a voltage regulation board, D1000217.
• "Dark Noise" = ASD with no light incident on the photodiode. "LO field only" = ASD with only the LO field incident on the photodiode.
• The "Dark noise" trace and "LO field only" traces are converted from cts/rtHz to rad/rtHz by noting that when the Michelson is locked on a dark fringe, the demodulated RF44 quadratures have a pk-pk amplitude of ~160 cts (corresponding to pi radians of phase shift). Since in these conditions the demodulated quadratures do not undergo any fringe wrapping, I converted the spectra by simple multiplication.
• For the "RF44 open loop" trace:
  • The DC offset in the demodulated signal (due to the RF44 signal from the LO field only) is digitally compensated, so that the fringing has (roughly) zero offset.
  • The Michelson was locked on a dark fringe, and the demodulated RF44 quadratures were monitored for ~5 mins. Then arctangent (specifically, arctan2 to get the correct quadrant in the IQ plane) of the two signals was taken to convert the fringing signals to phase noise.

Closing a feedback loop:

• Since it seems like we are sensing a signal (below ~1kHz at least), I tried to close a feedback loop (modelled loop shape shown in Attachment #2, it's just a model because I have to guess what the sensing and actuation gains are, and they're both assumed to be flat, digital delays etc aren't accounted for). I've also added the inferred loop gain by taking the ratio of the in loop and unsuppressed ASDs (though of course I don't account for the flat sensing noise at higher frequencies). At least qualitatively, things line up...
• While I can get the light level on the DCPD to stabilitze somewhat, the loop is not at all stable, and the suppression isn't very good at all.
• Not sure how meaningful any of the spectra with the loop closed are, but FWIW, I've put in the spectra of the demodulated RF44 signals with the loop engaged (RF44 Q is used as the error signal). A clear problem is evident at ~120 Hz, and the forest of lines isn't helping for sure. Also unclear to me why the I and Q signals don't have the same profile at low frequencies.

Conclusions/Questions:

1. What is the reason for the huge forests of lines in the "RF44 open loop" ASD, that are absent in the other two traces? If this were electrical pickup, it should be there in all three traces?
2. Is the shape of the spectrum reasonable? The roll-off above ~5 Hz doesn't seem quite steep enough to be seismic noise from the suspensions. Can it really be that the Michelson dark field has such high phase noise?
3. How can we get this scheme to give us cleaner sensing?
4. The actuation chain was verified to work fine with the single bounce beam from an ITM interfered with the LO field, and using the DC light level as an error signal and locking to the half-fringe point. So the problem is not due to insufficient actuation range. Seems like the error signal is so polluted with these forests of lines that even though there is some suppression of the error signal at low frequencies, the unsuppressed noise is still significant. I can't solve the problem by simply increasing the loop gain...
5. It is not shown here, but with only the LO field incident on the RFPD, I see a drift of the demodulated signals on the ~5 minute timescale - is this just due to fiber length change? If so, this is potentially problematic, as on long time scales, the true zero of the error point of the servo would be changing on the ~5 minute timescale. This would be true even for the final suspended scheme - if the path length between PR2 and the homodyne BS changes by some microns, we would have to correct this at DC?

The only two PZT Phase modulation transfer function measurements I could find are 40m/15206 and 40m/12077. Both these measurements were made to find a good modulation frequency and do not go below 50 kHz. So I don't think these will help us. We'll have to do a frequency transfer function measurement at lower frequencies.
I'm still looking for ALS PDH loop measurements to verify the model. I found this 40m/15059 but it is only near the UGF. The UGF measured here though looks very similar to the model prediction. A bit older measurement in 2017 was this 40m/13238 where I assume by ALS OLTF gautum meant the green laser PDH OLTF. It had similar UGF but the model I have has more phase lag, probably because of a 31.5 kHz pole which comes at U7 through the input low pass coupling through R28, C20 and R29 (See D1400293)

If the green laser is not being used, can I go and take some of these measurements myself?

Is it better than Luxor? https://labcit.ligo.caltech.edu/~jharms/luxor.html

• UPS batteries
• 2x HEPA filters
• VWR chemicals (methanol)

These boxes were moved from the 40m hallway to the inside of the VEA so that we have some space to walk around. You can find some pictures here.

For all the loops where we drive the NPRO PZT, there is some notch/resonance feature due to the PZT mechanical resonance. In the IMC loop this limits the PZT/EOM crossove to be less than 25 kHz. I don't have a model for this, btu it should be included.

If you hunt through the elogs, people have measured the TF of ALS NPRO PZT to phase/frequency. Probably there's also a measured ALS PDH loop somewhere that you could use to verify your model.

This looks cool, we should have something similar, can be really useful.

AUX PDH Loop update

I used D1400293 to get the latest logged details about the universal PDH box used to lock the green laser at X end. The uPDH_X_boost.fil file present there was used to obtain the control model for this box. See attachment one for the code used. Since there is a variable gain stage in the box, I tuned the gain of the filter model F_AUX in ALS_controls.yml to get the maximum phase margin in the PDH lock of the green laser. Unity gain frequency of 8.3 kHz can be achieved in this loop and as Gautam pointed out earlier, it can't be increased much further without changes in the box.

ALS Noise Budget update

The ALS control model remains stable with a reduction in total estimate noise because of the above update. There are few things to change though:

• This model is for a single arm locking where the beatnote signal between green laser and frequency doubled main laser is fed back to ETM at X end. Currently, Gautam is using a different scheme to lock where the feedback is sent to PSL-MC loop and the beat is taken between IR signals.
• In the LSC controls, I couldn't find a place where the digital ALS filter I have been optimizing and Kiwamu used, was placed. From what I gathered, after demodulation of beat note signal, a digital PLL is employed and the error signal is few to the Servo Filters directly. I might be missing some script which specifically switches on a particular set of filter modules in the XARM/YARM path when arms are locked through ALS.
• Another straight forward job for me is to verify the PSL-MC loop parameters with he TTFSS used. I'll do this next.

Tue evening from 4pm~6pm, Koji made a social distant tour for Anchal. We were present around the PSL/AS/ETMX tables.

I have placed 3 new in box, IDP 7 forepumps along the x arm of the interferometer. These are to be used as spares for both the 40m and Clean and Bake.

LiYuan has kindly done some Total Integrating Sphere (TIS) measurements on ITMU01 and ITMU02. A summary of the measurement is attached. I uploaded the measurements and some analysis script to nodus at /home/export/home/40m_TIS. I created a Wiki page for the measurements and linked to it from the core optics page.

These TIS measurements look very similar to the TIS of the LIGO optics. Further analysis shows that the scatter loss is 10+/-1.7 ppm for ITMU01 and 8.6+/-0.4 ppm for ITMU02.

In this calculation, a gaussian beam the same size bouncing off the 40m ITMs is assumed to scatter from the mirrors. The error is calculated by moving the beam around randomly with STD of 1mm.

In LiYuan's setup, TIS is measured for scattering angles between 1 and 75 degrees. If we go further and assume that the scatter is Lambertian we can extrapolate that the total loss is 10.9+/-1.9 ppm for ITMU01 and 9.2+/-0.5 ppm for ITMU02.

These measurements complete the loss budget nicely since with the 6ppm loss predicted from the phase maps, the total loss in the arm cavities would be 6+10+10=26ppm which is very close to the 28ppm loss that was measured after the arm cavity optics were cleaned.

We received 20pcs of stuffed demodulator boards from Screaming Circuits today. Some caveats:

1. The AP1053 amplifiers weren't stuffed. Note that this part is no longer in standard production, and lead time for a custom run is ~half a year. I recommend stuffing R2 and using a minicircuits amplifier upstream of this board. We have 6 pcs of AP1053 in hand so we can use those for the first AS WFS, but a second WFS will require some workaround.
2. AD8306ARZ weren't sent to Screaming Circuits. This part is used for the LO and RF signal level detection/monitoring stage, and so aren't crucial to the demodulation operation. @Chub, did we order the correct part now? They are rather pricey so maybe we can just adapt the footprint using some adaptor board?
3. DQS-10-100 hybrid 90 degree splitters were delivered to us after the lot was sent to Screaming Circuits. We have the pieces in hand, so we can just stuff them as necessary.

I removed 1 from the group to stuff some components that weren't sent to Screaming Circuits and test the functionality on the benchtop, the remaining have been stored in a plastic box for now as shown in Attachment #1. The box has been delivered to Chub who will stuff the remaining 19 boards once I've tested the one piece.

🤘🤘🤘

Summary:

The simple interferometer, composed of a single bounce reflection from ITMY and the LO beam deilvered via fiber to the AS table, can be locked - i.e. the phase of the LO beam can be controlled such that the DC light level on the DCPDs after the two beams are interfered can be stabilized. This test allows us to confirm that various parts of the sensing and actuation chain (e.g. PI PZT for homodyne phase control, Trek amplifier etc etc) are working.

I will post more quantitative analysis tomorrow.

Optical configuration:

• LO beam is a pickoff of the main PSL beam from just before it goes into the vacuum. The optical power arriving on each DCPD after the various beamsplitters, coupling loss etc is ~200 uW.
• IFO beam is the single bounce reflection from ITMY. For this test, ETMY, ITMX and ETMY are misaligned. Optical power arriving on each DCPD is ~80uW.
• The two beams are interfered on a 50-50 beamsplitter. The mode-matching efficiency was estimated to be ~50% which isn't stellar, but should be fine for this test.
• So, at half-fringe, we expect the signal on each DCPD to be linearly proportional to the phase difference between the two fields, and so we can use that as an error signal.

Servo topology:

Attachment #2 shows the servo topology.

• For a first attempt to close the feedback loop, we can consider the two blocks labelled "Sensing Chain" and "Actuation chain" to have a flat frequency response. While this isn't true, for a taget loop with ~100 Hz UGF, I think the approximation is reasonable.
• From the peak-to-peak value (160 cts) of the DCPD signals when the homodyne phase is uncontrolled, I estimate a sensing response (at half-fringe) of approximately 0.3 ct/nm, since this corresponds to 532nm of relative phase between the two beams. 
• An inverting summing amplifier is used to map the +/- 2^15 ct DAC range to 0-125V on the PI PZT. Assuming the full stroke of the PZT is 10um per the datasheet, and that this voltage range drives half of the full stroke (this is just a guess since all the old PI PZT circuits were designed to work at 0-250 V), we get an actuation coefficient of 0.075 nm/ct.
• Using these two numbers, we can then design a digital feedback loop that gives an open loop transfer function with ~100 Hz UGF, and sufficient stability margin.
• From the earlier measurements, we have an estimate for the amount of phase fluctuations caused by (i) seismic disturbances and (ii) fiber phase noise. This is the quantity we wish to suppress, and the suppression factor will be 1/(1+L), where L is the open loop gain.
• I didn't do this in any systematic way, but the loop in Attachment #3 seemed like a reasonable shape that would suppress the error signal RMS by ~10x, as shown in Attachment #4. So I decided to try this out.

Other notes:

1. The idea of offloading the DC control voltage to the ITMY suspension seemed to work fine.
2. It also seems like the relative phase between the two beams doesn't drift by so large an amount in short time scales, at least at night/quiet seismic conditions. So it is possible to maintain the lock for several seconds without having to offload the DC signal to the suspensions.
3. I didn't bother adapting the FSS Slow PID script to do this offloading in an automated way, seemed like more trouble than was just doing it by hand. But we may want to automate this in the future.
4. I couldn't make a clean measurement of the loop transfer function using the usual IN1/IN2 method. Introducing a step offset at the error point, the servo is able to track it (I didn't fit the step response time, but it's not as if the loop bandwidth is <1 Hz or something). I have to compare the measured in-loop error signal ASD to the free-running one to get a feel for what the UGF is, I guess, to rule out a weird loop.
5. Update 1100 Oct 6 2020: I have now added measured, in-loop, error point spectra to Attachment #4. Looks like there might be significant sensing noise re-injection.
   • Initially, I forgot to turn the HEPA on the PSL down for the measurement. So I have the two traces to compare. Looks like with the HEPA turned up to full, there is more noise in the 50-200 Hz range.
   • The trace marked "highGain" was taken with an overall loop gain that was 3dB higher than the nominal value - I could see some oscillations start to appear, and in the spectrum, maybe the feature at ~150 Hz is evidence of some gain peaking?

Conclusions:

1. The PI PZT seems to work just fine.
2. Need to look into the loop shape. I guess it's not reasonable to expect a UGF much higher than 100-200 Hz, because of the various delays in the system, but maybe the low frequency suppression can be made better.
3. What are the next steps?? What does this mean for the RF44 sensing scheme?

Summary:

After the earthquake on September 19 2020, it looks to me like the only lasting damage to suspensions in vacuum is the ETMY UR magnet being knocked off. 

Suspension ringdown tests:

I did the usual suspension kicking/ringdown test:

• One difference is that I now kick the suspension "N" times where N is the number of PSD averages desired. 
• After kicking the suspension, it is allowed to ring down with the damping disabled, for ~1100 seconds so that we can get spectra with 1mHz resolution.
• We may want to get more e-folding times in, but since the Qs of the modes are a few hundred, I figured this is long enough.
• I think this kind of approach gives better SNR than letting it ringdown 10,000 seconds (for 10 averages with 10 non overlapping segments of 1000 seconds), and I wanted to test this scheme out, seems to work well.
• Attachment #1 shows a summary of the results.
• Attachment #2 has more plots (e.g. transfer function from UL to all other coils), in case anyone is interested in more forensics. The data files are large but if anyone is interested in the times that the suspension was kicked, you can extract it from here.

Conclusions:

1. My cursory scans of the analysis don't throw up any red flags (apart from the known problem of ETMY UR being dislodged) 👌 . 
2. The PRM data is weird 
   • I believe this is because the DC bias voltage to the coils was significantly off from what it normally is when the PRC is aligned.
   • In any case, I am able to lock the PRC, so I think the PRM magnets are fine.
3. The PRC angular FF no longer works turns out this was just a weird interaction with the Oplev loop because the beam was significantly off-centered on the Oplev QPD. Better alignment fixed it, the FF works as it did before.
   • With the PRC locked and the carrier resonant (no ETMs), the old feedforward filters significantly degrade the angular stability to the point that the lock is lost.
   • My best hypothesis is that the earthquake caused a spot shift on PR2/PR3, which changed the TF from seismometer signal to PRC spot motion.
   • Anyways, we can retrain the filter.
   • The fact that the PRC can be locked suggest PR2/PR3 are still suspended and okay.
4. The SRM data is also questionable, because the DC bias voltage wasn't set to the values for an aligned SRC when the data was collected
   • Nevertheless, the time series shows a clean ringdown, so at least all 5 OSEMs are seeing a signal.
   • Fact that the beam comes out at the AS port suggest SR3/SR2 suspensions are fine 👍 

Attachment #2 also includes info about the matrix diagonalization, and the condition numbers of the resulting matrices are as large as ~30 for some suspensions, but I think this isn't a new feature. 

Attachment #1 shows that the c1rfm model isn't able to receive any signals from the front end machines at EX and EY. Attachment #2 shows that the problem appears to have started at ~430am today morning - I certainly wasn't doing anything with the IFO at that time.

I don't know what kind of error this is - what does it mean that the receiving model shows errors but the sender shows no errors? It is not a new kind of error, and the solution in the past has been a series of model reboots, but it'd be nice if we could fix such issues because it eats up a lot of time to reboot all the vertex machines. There is no diagnostic information available in all the places I looked. I'll ask the CDS group for help, but I'm not sure if they'll have anything useful since this RFM technology has been retired at the sites (?).

In the meantime, arm cavity locking in the usual way isn't possible since we don't have the trigger signals from the arm cavity transmission. 

Update 1500 4 Oct: soft reboots of models didn't do the trick so I had to resort to hard reboots of all FEs/expansion chassis. Now the signals seem to be okay.

2x500 ml bottles of spectroscopic grade isopropanol were delivered. I marked them with today's date and placed them in the solvent cabinet. In the process, my shoulder bumped the laser interlock switch by the door to the VEA in the drill press area, which turned the PSL NPRO off. I turned it back on just now. The other NPROs are not connected to the interlock and so were unaffected.

Attachment #1 shows that the ITMX, ETMY and beamsplitter Oplev light levels have decayed significantly from their values when installed. In particular, the ETMY and ITMX sum channels are now only 50% of the values when a new HeNe was installed. ELOG search revealed that ITMY and ETMX HeNes were replaced with newly acquired units in March and September of last year respectively. The ITMX oplev was also replaced in March 2019, but the replacement was a unit that was being used to illuminate our tourist attraction glass fiber at EX.

We should replace these before any vent as they are a useful diagnostic for the DC alignement reference.

The optomechanics stock in the lab was in a sad state. We have obtained the following from Thorlabs in the last two months:

1. 6 pcs each of DT12 and DT12B compact translation stages (for lens mode matching).
2. 3pcs each KM100PM + PM3 cube beamsplitter mounts (for polarization control).
3. 1 Post / spacer kit for height adjustment.
4. 3 pcs ea of K6XS + AD11F + F220APC for fiber applications.

I have used some of these for the ari BHD setup. The unused items are stored in the shelves that house the optomechanics ~halfway down the east arm. I'm wondering what's a good setup to document the stock of this stuff so we can always have a healthy stock of optomechanics (at least the non speciality ones like posts, spacers etc). It sucks to realize at 0000hrs that you're missing a 3mm shim or 250mm converging lens or something.

it would be a good idea for us to have an auto-reminder to have us check the flow of all the HEPAs in the lab and elog it once a year so that we can replace filters and pre-filters appropriately.

[JV, GV]

The HEPAs work again. After running the HEPAs for ~1 hour, I checked the particle count on the PSL table - the meter registered 0 for both 0.3 um and 0.5 um. So I decided to turn the NPRO back on, at ~1730 local time. The PMC and IMC were readily locke, so the basic interferometer functionality is returned, and we can now go ahead with (i) vent prep (ii) air BHD tests and (iii) IMC debuggin as was discussed on the call today. The earth is shaking, but nothing serious so far, I will resume alignment of the interferometer later in the evening when hopefully things have calmed down a bit more...

Procedure:

1. Turned off mains switch on the NW corner of the PSL enclosure. Then, disconnected the power cables from mains to Variac and from Variac to HEPAs. Made sure both HEPAs were set to "OFF".
2. With confidence that no AC power was reaching the motors, I removed the pre-filters, and removed the old startup capacitors. These don't have a polarity, but I marked the cable that was connected to the left terminal of the capacitor when the cap is viewed with the label facing you, in the interest of changing as few things as possible.
3. The two new capacitors were measured with the LCR meter - the meter registered ~7.5 uF, as expected. Unsurprisingly, the old capacitors that were removed didn't register any reading on the LCR meter. The terminals weren't shorted, but I don't know what the failure mode for this kind of capacitor is.
4. The two new capacitors were installed. Then, I tested the system by undoing all the changes in bullet #1. We found that the Variac needs to be set to 100% for the motors to startup.
5. The motor speed was found to vary as the Variac dial was turned. FWIW, at the "nominal" setting of 33% on the Variac (when we run the interferometer), I could see both fan blades were turning, but the flow was low enough that you couldn't hear any wind (at least, neither Jordan nor I could).
6. Turned off the mains agian, and cleaned up - restored the insulating rubber sleeve on the capacitor leads, and re-installed the pre-filters on the HEPA blowers. Then we turned both blowers back on. 

Note that the many other issues Koji noted in the preceeding elog (e.g. flaky wiring) have not been addressed.

Flow measurements:

Chub kindly provided us with an electronic anemometer. With the meter held directly against the HEPA filter inside the enclosure, we measured ~700 cfm of airflow on each of the two HEPAs, with the Variac set to 100% and the HEPAs themselves set to "High". With the Variac at 50%, the flow drops to ~160 cfm. At the nominal setting of 33%, the meter didn't register any flow. I don't know what the spec'd flow rate is for this combination of blower + filter, but Jordan says similar units in Downs register ~1500 cfm at the "High" setting. The two protable (similarly sized) HEPA units in the 40m, one at ITMY and one at ETMY, register ~900 cfm and ~1100 cfm respectively, when set to high. So we may want to revisit what the "nominal" HEPA setting should be, in case the filters have become clogged over time. 

Some photos of the HEPA blowers with the pre-filters off and the capacitors switched out may be found here.

I was in the lab from 1630-1830. I have located and visually inspected all the parts required for the magnet regluing / optic cleaning parts of the planned vent, except the fresh batches of scpectroscopic grade solvents. I was in the cleanroom part of the clean and bake lab from 1630-1700.

I removed the forepump to TP2 this morning after the vacuum failure, and tested in the C&B lab. I pumped down on a small volume 10 times, with no issue. The ultimate pressure was ~30 mtorr.

I re-installed the forepump in the afternoon, and restarted TP2, leaving V4 closed. This will run overnight to test, while TP3 backs TP1.

In order to open V1, with TP3 backing TP1, the interlock system had to be reset since it is expecting TP2 as a backing pump. TP2 is running normally, and pumping of the main volume has resumed.

gautam 2030:

1. The monitor (LCD display) at the vacuum rack doesn't work - this has been the case since Monday at least. I usually use my laptop to ssh in so I didn't notice it so it could have been busted from before. But for anyone wishing to use the workstation arrangement at 1X8, this is not great. Today, we borrowed the vertex laptop to ssh in, the vertex laptop has since been returned to its nominal location.
2. The modification to the interlock condition was made by simply commenting out the line requiring V4 to be open for V1 to be opened. I made a copy of the original .yaml file which we can revert to once we go back to the normal config.
3. I also opened VM1 to allow the RGA scans to continue to be meaningful.
4. At the time of writing, all systems seem nominal. See Attachment #2. The vertical line indicates when we started pumping on the main volume again earlier today, with TP3 backing TP1.

​Unclear why the TP2 foreline pump failed in the first place, it has been running fine for several hours now (although TP2 has no load, since V4 isolates it from the main volume). Koji's plots show that the TP2 foreline pressure did not recover even after the interlock tripped and V4 was closed (i.e. the same conditions as TP2 sees right now).