I wanted to look into the ISS situation. Some weeks ago, I found the PD that was previously used as the in-loop photodiode. I wanted to use this and measure the open-loop RIN at a few places (to see if there's any variation and also to check its functionality). However, I didn't get very far tonight - for a start, the PD height is 3" (while our beam height is 4" everywhere outside the vacuum), and I needed to put together a circuit to supply the 5V bias and +/- 15 V since the transimpedance is done on the head. I was only able to do a low-level functionality test tonight, checked that the DC voltage output varied linearly with the incident power (calibrated against an NF1611 photodiode, data will be put up later). I didn't get to measuring any noise performance - is an incandescent light bulb still shot noise limited at ~10 Hz < f < 10kHz? Some notes:

1. The PD is DC coupled, and has a transimpedance of 1 kohm (inverting AD829 does the transimpedance).
2. Probably a daughter board should be made that supplies the DC power voltages and rotues the output signal to something more convenient like a BNC connector. This daughter board can then also implement a DC coupled path (for monitoring) and AC coupled path (for servoing, fc to be determined).
3. SR560 based ISS was implemented some years ago but I think the improvement was only seen above 100 Hz, and that too was marginal, the stabilized RIN was 10^-6 (monitored on an out-of-loop photodiode I think, but unsure). We'd probably want to aim for at least an order of magnitude better. Unclear at this point why more suppression wasn't possible back then, was it just insufficient loop gain, or was the sensing noise too high? To be investigated.

Unconnected to this work - this problem reared its ugly head again (i noticed it yesterday morning already actually). I don't have the energy to embark on a fix tonight, Koji is going to be in the lab all day tomorrow and so he will fix it.

that little PD in the black mount was never very good. The AD829 is not a good opamp for transimpedance and especially not good for low frequencies. Stefan Ballmer and I were able to get 2e-8 out of these (@100 Hz) many years ago.

I wonder if we have some of Zach's M2ISS photodetectors around, perhaps in QIL or Cryo. I doubt that any of them are in use now. Those had good performance nad BNC output.

Ok I was using the PD in the black mount because Rana recommended it a few weeks ago.

Regarding the M2ISS, I acquired the hardware from QIL some months ago, including a circuit board, and 2 PDs. These had LEMO outputs though (not BNC), and the mounts are not 4". These photodiodes are what I'm using as the airBHD DCPDs right now, and some photos are here - are these the photodiodes you mentioned? Or are there yet more M2ISS photodiodes? I remember Johannes had some custom mounts extruded to make them 4" high, do you mean those? Can I retrieve them his Cryo setup?

BTW, my elog scraping shows only one spectra from Stefan in the ATF elog, and the performance there is more like 1e-7/rtHz @ 100 Hz, and that’s using a dedicated high BW servo circuit, not the SR560. Am I just missing the measurement of 2e-8/rtHz?

1. Two arms / BS / PRM / SRM were aligned. (Attachment 1)
2. IMC was aligned by WFS and the WFS offsets were offloaded.
3. Suspension Status Snapshot (Attachment 2)
4. Oplevs are aligned (Attachment 3)
5. Xarm green was aligned in the daytime. Xarm green refl DC (C1:ALS-X_REFL_DC_OUTPUT) was 620 (aligned) ~1300 (drifted). When unlocked, it was 3750.
6. Yarm green: I saw no flash. We don't have functional PZT alignment since the ASY M2 PZT got broken. I went to the Yend. Something funky is going on with the Yend green. I struggled to have any flash of the cavity. The apertures were not so precise. I finally got TEM00 locked, but the modematching seems exteremely low (like 1/1000?). Basically I saw no power reduction of the refl when the cavity is locked. So at least the cavity was locked but we might need to revisit when we open the chamber
   ==> Gautam thinks it was not like that. So he will check the green alignment tomorrow (Thu).
7. Item checking: I familialized myself with the yend crane operation. Today I learned that there is a power switch on the wall (Attachment 4). The yend has two heavy door storages (Attachments 4/5). The slings to lift the heavy door are in the crane cabinet along with the y arm (Attachment 6). I didn't yet try to find the "hammer" to hit the door if the o-ring stuck too strong, although that's optional.
8. We want to reduce the PSL power. But Gautam wants to use the arm locking with the nominal power, it will be done tomorrow by him.
    
9. The last thing is to check the green trans power. I noticed that the green trans beams are blocked by an HWP for the BHD LO path on the PSL table. (Attachment 7)
   The HWP was moved and the process was recorded in the movie (Attachment 8). The fiber output was monitored by the BHD DC (aka AS110 DC) with the AS path blocked. The fiber output of 22.6mV (offset -2.5mV) was improved tio 29.1mV after the HWP move and the alignment adjustment.
10. Now the green transmissions are visible by the green PDs. Attachment 9 shows the trans and ref of each green beams with and without locking to TEM00. The questionable green TRY was ~0.3. If we compare this with the histrical data (Attachment 10), it is about 1/4 of the value in the past. It's not too crazy but still quite low.

At this point, I'm leaving the lab. All the suspensions (incl SRM) are aligned. PSL/GRX/GRY shutters were left open.

1. Oplev HeNe at ETMY was replaced, see here for my earlier discussion on this.
   • I thought this is a good idea since we want the Oplev as a coarse reference and it'd not be ideal if this HeNe dies during the time the optic is out of the chamber.
   • New HeNe had 2.8mW of power output as measured with Ophir power meter. This is in line with what is expected from these Lumentum heads.
   • I labelled the head with the power output and today's date, re-aligned the Oplev reflection onto its QPD. 
   • After this work, the Y-arm could be locked without the huge angular fluctuations that was visible earlier, 👍 .
2. GTRY anomaly
   • I actually judged that there is no anomaly.
   • The GTRY CDS indicator is actually quite useless - the ADC saturates at ~3500 cts (and not 32768 as you would expect from a 16 bit ADC but that's the well-known whitening filter saturation problem). This should be fixed, but this is a task for later.
   • I measured with a DMM the voltage when the TEM00 is locked to the cavity and GTRY is 0.3 (the nominal value these days), the DC voltage was ~5.6 V. The prompt reflection from the ETM registers ~6.5 V DC. So the mode-matching isn't stellar, but this is again a known issue, and can be fixed later.
3. Other pre-vent checks
   • The Oplevs had drifted significantly, I re-centered ITMs, ETMs and BS after aligning the arm cavities and green beams in the POX/POY lock state. See Attachment #1.
   • I locked the PRMI on carrier and used this config to re-center the PRM Oplev, see Attachment #2.
   • No further action was taken regarding SRM oplev.
   • I checked the ALS noise, see Attachment #3. The X arm ALS has excess noise at ~100 Hz that certainly wasn't there previosuly - sigh. There is nothing I can find about any changes made at EX in the elog.
   • Updated the "DriftMon" values, though I guess we don't even really use this anymore these days?
   • Re-relieved the IMC WFS offsets.
   • Cut the input power to the IMC from 1.007W to 100.1 mW (both numbers measured with Ophir power meter).
   • Replaced the 10% beamsplitter in the MCREFL path on the AS table with a Y1 HR mirror. Note that there is no beam on the IMC WFS in this configuration.
   • Was able to lock the IMC on low power to a TEM00 mode - need to set up the low power autolocker. The IMC autolocker is now set to the low power settings, and I've tested it locks a couple of times. Attachment #5 shows the low power lock in StripTool.
   • Walked around and looked at all the bellows - the jam nuts are up against their stops, and I can't move them with my hands, so I think that's okay.

If everything else looks good, I'll start letting the dry N2 into the main volume after lunch.

BTW, nice video! @ Koji, How difficult was it to edit it into this form? 

1. Jam nuts - checked that none of the nuts move by hand, which according to Steve, is sufficient. I recorded photos of all of them here.
2. Lab particle count - at the SP table, I measured 25,910 / cf @ 0.3 um and 1220 /cf  @ 0.5 um. Steve's guidance is that the latter number should be less than 10,000, so no issues there.
3. Valves - closed all the annuli off, and also closed VM1, VA6 and V1. The procedure calls for VM3 to be opened, presumably so that the RGA remains pumped, but I see no reason why we can't just leave that RGA volume valved off.
4. Started letting N2 into the main volume at ~3pm local time, by opening VV1. Attachment #1 shows the valve configuration adopted for this vent. Went up to 25 torr and then switched over to the "Ultra Zero" instrument grade air cylinders.
5. Aimed for 3-5 torr/min rate of pressure increase locally. The full vent trend can be seen in Attachment #2.
6. Stopped letting air into the main volume when P1a pressure was 700 torr, at which point I disconnected the cylinders from the main volume, and left VV1 open so that the IFO equilibriates to the lab air pressure. I used 4 full cylinders of the instrument grade air, which is par for the course. 
7. Since we anticipate opening the ETMY and output optics chambers, I also vented those annuli. The final state of the valves as I am leaving it for the night is shown in Attachment #3.
8. I re-aligned the IMC mirrors so that I could lock the IMC at low power once again. Indeed, IMC trans of ~1400 cts was realized (see Attachment #4), c.f. ~1500 earlier today, I thought this is fine and didn't optimize further. I think the policy is to not lock the IMC in air unless necessary, so I'm leaving the PSL shutter closed for the night.
9. The ITMs, ETMs and BS were re-aligned such that the Oplev spots are centered. I can see some higher order resonances of the green beams. The AS spot is fringing (the two ND=0.5 filters on the AS CCD camera were removed for better visibility). So this is fine as far as alignment is concerned.

We are now ready to take the doors off. I've already done the basic prep work (loosened bolts, cleaned chamber, carts for tools, fresh ameristat on portable HEPAs etc).

[koji, rana, gautam]

This morning, we did the following;

1. Removed the ETMY chamber heavy door. It is stored on the rack at the east end.
2. Removed OMC chamber heavy door. It is smaller than the other chamber doors, so doesn't sit on the standard size rack we have. So it is laid flat, on a clean sheet of ameristat, on a cart next to the NE corner of the PSL table.
3. After taking some photos of the chamber and making sure the position of the suspension tower was marked by some stops, we removed the ETMY cage and moved it to the cleanroom area. The optic was EQ stopped during the transport, and the OSEMs were removed. The ETMY Oplev HeNe was turned off and the PSL was shuttered to allow us to work without goggles.
4. The broken off magnet was retrieved from inside the OSEM. The shadow sensor voltage recovered a value of ~800cts, which means the LED/PD pair on the UR OSEM seems to work fine, and it was in fact the magnet blocking the PD that was the issue.
5. In the cleanroom area, we removed the optic from the wire loop and placed it on the magnet gluing fixture. The wire is intact (for now), so there is some hope of re-suspending it in the same loop.

The OSEMs remain in the EY vacuum chamber. The next set of steps are:

1. Clean the EP30-2 residue from the broken magnet joint - this will require some scrubbing with an acetone soaked scrub or similar implement.
2. Reglue the broken magnet.

We will most likely work on this tomorrow. At ~1615, I briefly opened the PSL shutter and tweaked the IMC alignment. We will almost certainly change the pointing into the IMC when we remove the old OMC and rebalance that table, so care should be taken when working on that...

To be a bit more clear about what we are going to do in the OMC chamber, I marked-up some photos, see Attachments #1 and #2.

1. OM5 will be rotated to bring the IFO AS beam straight out without any splitting to the OMC.
2. OMMT, OMC, DCPD, DCPD transimpedance amp, and all peripheral optics associated with these components, will be removed. Many of these components are mounted on a breadboard and so removing that breadboard will take care of it. These are marked with pink Xs.

I anticipate that after this work, the only components on the table will be 

1. IM1, to send the PSL beam to the IMC.
2. OMs 5 and 6 to bring the IFO AS beam out onto the AP table (in principle, we could try and eliminate both these optics, if the AS beam happens to exit through one of the viewports cleanly, we will not have any intervening objects in the way once the OMC and peripherals are removed).
3. MMT2 for mode-matching the IMC transmission to the interferometer mode.

Are we in agreement with this plan?

See #15656 for the updated photo

I believe the mirror next to IM1 is for the green beams to be delivered to the PSL table. I think we still want to keep it. Otherwise, the plan looks fine.

Good point - looking back, I also see that I already removed the mirror at the SW corner of the table in 2016. Revised photo in Attachment #1. There is an optic on the east edge of this table whose purpose I'm not sure of, but I'm pretty sure it isn't essential to the main functionality and so can be removed.

I got a call from Calum ~830am today saying some facilities people entered the lab, opened the south entrance door, and tripped the alarm in the process. I came to the lab shortly after and was able to reset the alarm by flipping the switch on the alarm box at the south end entrance to "Alarm OFF". Then, I double checked that the door is closed, and re-enabled the alarm. The particle count at the SP table is not unusually high and the lasers (Oplev HeNe and AUX X) were still on, so doesn't look like any lasting damage was done. The facilities people were apparently wearing laser safety goggles.

[koji, gautam]

1. Glued broken off magnet - curing overnight with lamp to slightly elevate temp for curing.
2. Remvoed material from OMC chamber as per the plan. This is all sitting wrapped up in foil on a cart for tonight, we should figure out a better storage plan eventually.

The IMC isn't resonant for a TEM00 mode at the time of writing - we are waiting for the stack to relax, at which point if the IMC isn't resonant for a TEM00 mode, we will tweak the input pointing into the IMC (we want to use the suspended cavity as the reference, since it is presumably more reliable than the table from which we removed ~50 kgs of weight and shifted the balance.

[koji, gautam]

1. Tweaked last steering mirror before PSL beam is launched into vacuum to get the IMC resonant for the TEM00 mode. Less than 1/8 turn in each Pitch and Yaw was required, and we recovered MC2T ~ 1300 cts (c.f. the expected ~1500 cts when the cavity is well aligned, but we didn't touch the IMC mirrors).
2. ETMY magnet re-gluing
   • Regluing was successful. Pickle picker came off smoothly. 
   • We performed the razor blade test on all 6 magnets, the integrity of the joints seems uncompromised.
3. ETMY cleaning
   • Applied F.C. to HR face, HR edge (to remove some residual stains) and AR surface (taking care to go around the magnets).
   • A fresh bottle each of spectroscopic grade acetone and isopropanol was dispensed into clean beakers for the work.
   • Curiously, in the lighting conditions of the cleanroom, both HR and AR faces looked surprisingly clean, even when illuminated with the green flashlight.
   • However, when we took it to the east end and looked at it again with the green flashlight, just before putting it into the chamber, we saw all sorts of stains and markings on the HR side.
   • The central aperture looks fine - the contamination did not happen during the transport so we feel confident not to futz around more with this and since we cleaned it extensively in the cleanroom, we opted not to do another round of cleaning in the chamber.
4. ETMY re-suspension (on the existing wire loop)
   • The back OSEM-holding plates were removed for access reasons.
   • The optic was rested onto the EQ stops. 
   • Patient nudging of the wire around the optic allowed us to eventually get the wire into the V-grooves in the standoffs.
   • The wire was not damaged! (at least, to eye, free-swinging test is going on now, we will see what the eigenfrequencies and Qs are)...
   • We confirmed that the wire is seated in the v-groove, and took some close up photos.
5. ETMY pitch balancing
   • A HeNe was brought into the cleanroom and mounted at 5.5" beam height.
   • Level-ness of the beam (i.e. parallel to the optic table) was done coarsely with a spirit level placed on the HeNe holder, and more fine adjustment was done by checking the beam height at the launch point, and again ~2 meters away.
   • All EQ stops were backed off to ensure the optic was really "free-swinging" - no OSEMs either at this point, since all this was done in the cleanroom.
   • The return beam hit the HeNe's output aperture, so we were satisfied that the pitch balancing was good to ~1mrad.
6. ETMY transport back to EY
   • Since we want to maintain some tension in the wires, the bottom EQ stops were not raised as much as they were when the tower was transported from EY to the cleanroom.
   • As a result, we felt it'd be better for a human (Koji) to carry the cage (rather than cart it, the idea being that there would be fewer vertical impulses due to bumps on the ground etc).
   • The transport went smooth.
7. EY chamber work
   • Optic was placed back in its original position (marked by some L-clamps).
   • In preparation for alignment work, I undid all the CDS changes I made to facilitate the temporary 3-magnet actuation. This necessitated a reboot of c1auxey VME crate.
   • OSEMs were inserted - best effort to half light as usual, and we tried to replicate the rotation in the mounts as closely as possible to what was the case before (using a photo from the last vent as reference).
   • Coarse alignment was done by making the Oplev return beam hit the QPD.
   • Better alignment of the Y-arm cavity axis was done using the green beam. 
   • IMC was locked - the beam alignment target was used to ensure the IR beam was hitting ETMY at the correct height (by moving TT2), and then the ETMY angular alignment was improved to make the return beam go back roughly collinearly.
8. Y-arm POY locking
   • Satisfied with the alignment, we returned to the control room (light doors back on EY chamber) - cavity alignment was tweaked further to make the TRY resonances as high as possible.
   • I could lock the Y-arm length to the PSL frequency using POY as a sensor - TRY was ~0.04-0.05 (since IMCT is 1/10th its usual value, we expect it to be 0.1, so this is not bad).
9. OMC chamber work
   • Tied down the balance weights that were previously placed after the OMC and peripheral optics were removed.
   • Checked the table leveling.
   • Checked that the AS beam is reasonably well centerd on OM5 and OM6. Took some photos.
   • Checked that the IMC could be locked after this work - it could.

So all the primary vent objectives have been achieved 🙌 . The light doors are on the chamber right now. I'm measuring the free-swinging spectra of ETMY overnight. Barring any catastrophic failures and provided all required personnel are available, we will do the final pre-close-up checks, put the heavy doors back on, and pump down starting 10 am Monday, 9 Nov 2020. Some photos here.

Attachment #1 shows the main result - there are 4 peaks. The frequencies are a little different from what I have on file for ETMY and the Qs are a factor of 3-4 lower (except SIDE) than what they are in vacuum, which is not unreasonable I hypothesize. The fits suggest that the peak shape isn't really Lorentzian, the true shape seems to have narrower tails than a Lorentzian, but around the actual peak, the fit is pretty good. More detailed diagnostic plots (e.g. coil-to-coil TFs) are in the compressed Attachment #2. The condition number of the matrix to diagonalize the sensing matrix (i.e. what we multiply the "naive" OSEM 2 Euler basis matrix by) is ~40, which is large, but I wouldn't read too much into it at this point.

I see no red flags here - the PIT peak is a little less prominent than the others, but looking back through the elog, this kind of variation in peak heights doesn't seem unreasonable to me. If anyone wants to look at the data, the suspension was kicked every ~1100seconds from 1288673974, 15 times.

Basic IFO alignment checks were done.

1. IMC could be locked - I tweaked the cavity alignment a little to maximize the MC transmission.
2. Y arm can be made resonant for a TEM00 mode of the main beam. I can't run ASS successfully in this low power config. I can see that the axis isn't great, the spot is visually off-center on ITMY, but we should have plenty of actuator range to correct for this with TT1/TT2.
3. X arm shows IR mode flashes in the TRX QPD. The green beam can be made resonant for a TEM00 mode, but that alignment doesn't yield the largest IR resonant peaks in TRX. I suspect it is due to the mis-alignment of the beam axis.
4. AS beam was aligned onto the CCD. I could see clean Michelson fringes by tweaking the BS alignment. On the AP table, I noticed that the beam on the first steering mirror after the AS beam exits the vacuum is a little high. We can easily resolve this by tweaking OM6 pitch a bit, but even if we don't I don't see any major issues as there is plenty of clearance w.r.t. the viewport when the beam exits the vacuum.
5. With the PRM aligned, I can see the REFL beam on the CCD, and it doesn't look clipped.
6. I didn't bother to align the green beams to the arm cavities or re-center the Oplevs - is this necessary? It is a step in the pre-close up checklist, so maybe we should do it... The green transmission does reach the PSL table...

Tomorrow, we should do some visual checks of the chambers / EQ stops on ETMY etc but I don't see any major problems at the moment...

> I didn't bother to align the green beams to the arm cavities or re-center the Oplevs - is this necessary? It is a step in the pre-close up checklist, so maybe we should do it... The green transmission does reach the PSL table...

I don't think so. The beam is reaching the PSL, so we have no motivation to change the green alignment. Regarding the oplev, the green refl should come back to the PDH PD and this gives us additional beam reference. As soon as we find the green resonance after the pumping, we can tweak the green axis so that the spots on the mirrors become reasonable (as well as the green trans CCD on the PSL table).

[koji, rana, gautam]

1100 - EY chamber inspected, no issues were found --> EY heavy door on

1200 - OMC chamber was inspected. OM6 was marginally tweaked to bring the beam down a little in pitch, and also a little northwards in Yaw. --> Heavy door on.

1230 - Pumpdown started. Initially, the annuli volume was pumped down. The procedure calls for doing this with the small turbopumps. However, V7 was left open, and hence, in the process, the TP1 foreline pressure (=P2) hit ~30 torr. This caused TP1 to shutdown. We were able to restart it without issue. This case was not caught by the interlock code, which was running at the time. It should be recitified.

1330 - OMC breadboard clean optics and DCPD hardware were wrapped up and packed into tupperware boxes and stored along the south arm. OMC cavity itself, the OMMT, and the breadboard the OMC was sitting on are wrapped in foil/Ameristat and stored in cabinet S13, lower 2 shelves.

1915 - P1a = 0.5 torr pressure reached. Switched over to pumping the main volume with TP1, backed by TP2 and TP3, which themselves are backed by their respective dry pumps and also the AUX drypump for some extra oomph. All cooling fans available in the area were turned on and directed at the turbo pumps. RV2 was used to throttle the flow suitably.

It was at this point that we hit a snag - RV2 has gotten stuck in a partially open position, see Attachment #1. We can see that the thread doesn't move in response to turning the rotary dial. Fortunately, the valve is partially open, so the main volume continues to be pumped - see Attachment #2 for the full history of today's pumping. We are leaving the main volume pumped in this configuration overnight (TP1 pumping main volume backed by TPs 2 and 3, which are in turn backed by their respective drypumps and also the AUX dry pump). I think there is little to no risk of any damage to the turbo pumps, the interlocks should catch any anomalies. The roughing pumps RP1 and RP3 were turned off and that line was disconnected and capped.

What are our options?

1. The main volume is able to reach the "nominal" pressure of 1e-5 torr, just takes longer.
2. At some point, we may be able to pump the main volume directly with TP2 and TP3 - it is unclear at this point whether it's better to have the conductance limited TP1, which has a higher pumping capacity, or have the smaller TPs 2 and 3 pump through a larger conductance. 
3. Depending on how low the ultimate pressure gets, we may be able to run the usual IFO activities until the replacement pressure gauges arrive in ~1 week, at which point we can vent the pumpspool (leaving the main volume isolated) and either repair this valve or replace it with one of the spares we have.

We need some vacuum experts to comment. Why did this happen? Is this an acceptable failure mode of the valve?

KA Ed:
2230 - P1a = 0.025 torr. The pressure is coming down with log-linear scale. x0.1 per 2.5 hours or so.

Main volume pressure as of 11:30AM 2020/11/10

For the input matrix diagonalization, it seemed to me that when we had a significant seismic event or a re-alignment of the optic with the bias sliders, the input matrix also changes.

Meaning that our half-light voltage may not correspond to the half point inside the LED beam, but that rather we may be putting the magnet into a partially occluding state. It would be good to check this out by moving the bias to another setting and doing the ringdown there.

Summary:

1. Recovery was complicated by RFM failure on c1iscey - see Attachment #1. This is becoming uncomfortably frequent. As a result, the ETMY suspension wasn't being damped properly. Needed a reboot of c1iscey machine and a restart of the c1rfm model to fix.
2. POX/POY locking was restored. Arm alignment was tuned using the dither alignment system.
3. AS beam was centered on its CCD (I put a total of ND=1.0 filters back on the CCD). Note that the power in the AS beam is ~4x what it was given we have removed the in-vacuum pickoff to the OMC.
4. Green beams were aligned to the arm cavities. See Attachment #2. Both green cameras were adjusted on the PSL table to have the beam be ~centered on them.
5. ALS noise is far too high for locking, needs debugging. See Attachment #3.
6. AS beam was aligned onto the AS55 photodiode. With the PRM aligned, the REFL beam was centerd on the various REFL photodiodes. The PRMI (resonant carrier) could be locked, see Attachment #4.

I want to test out an AS port WFS now that I have all the parts in hand - I guess the Michelson / PRMI will suffice until I make the ALS noise good again, and anyways, there is much assembly work to be done. Overnight, I'm repeating the suspension eigenmode measurement.

The results from the ringdown are attached - in summary:

• The peak positions have shifted <50 mHz from their in-air locations, so that's good I guess
• The fitted Qs of the POS and SIDE eigenmodes are ~500, but those for PIT and YAW are only ~200
• The fitting might be sub-optimal due to spurious sideband lobes around the peaks themselves - I didn't go too deep into investigating this, especially since the damping seems to work okay for now
• There is up to a factor of 5 variation in the response at the eigenfrequencies in the various sensors - this seems rather large
• The condition number of the matrix that would diagonalize the sensing is a scarcely believable 240, but this is unsurprising given the large variation in the response in the different sensors. Unclear what the implications are - I'm not messing with the input matrix for now

Ordered 11/16 from CDW, on PO# S492940, the high voltage Tripp Lite SMART5000XFMRXL  for TP-1.  Should be arriving in about a week.

In prep to try some of these debugging steps, I did the following.

1. ndscope updated from 0.7.9 to 0.11.3 on rossa. I've been testing/assisting the development for a few months now and am happy with it, and like the new features (e.g. PDF export). v0.7.9 is still available on the system so we can revert whenever we want.
2. Arms locked on POX/POY, dither aligned to maximize TRX/TRY, normalization reset.
3. PRMI locked, dither aligned to maximize POPDC.
4. All vertex oplevs re-centered on their QPDs.

While working, I noticed that the annoying tip-tilt drift seems to be worse than it has been in the last few months. The IPPOS QPD is a good diagnostic to monitor stability of TT1/TT2. While trying to trend the data, I noticed that from ~31 Jan (Saturday night/Sunday morning local time), the IP-POS QPD segment data streams seem "frozen", see Attachment #1. This definitely predates the CDS crash on Feb 2. I confirmed that the beam was in fact incident on the IPPOS QPD, and at 1Y2/1Y3 that I was getting voltages going into the c1iscaux Acromag crate. All manner of soft reboots (eth1 network interface, modbusIOC service) didn't fix the problem, so I power cycled the acromag interface crate. This did the trick. I will take this opportunity to raise again the issue that we do not have a useful, reliable diagnsotic for the state of our Acromag systems. The problem seems to not have been with all the ADC cards inside the crate, as other slow ADC channels were reporting sensible numbers.

Anyways, now that the QPD is working again, you can see the drift in Attachment #2. I ran the dither alignment ~4 hours ago, and in the intervening time, the spot, which was previously centered on the AS camera CRT display, has almost drifted completely off (my rough calibration is that the spot has moved 5mm on the AS CCD camera). I was thinking we could try installing the two HAM-A coil drivers to control the TTs, this would allow us to rule out flaky electronics as the culprit, but I realize some custom cabling would be required, so maybe not worth the effort. The phenomenology of the drift make me suspect the electronics - hard for me to imagine that a mechanical creep would stop creeping after 3-4 hours? How would we explain the start of such a mechanical drift? On the other hand, the fact that the drift is almost solely in pitch lends support to the cause being mechanical. This would really hamper the locking efforts, the drift is on short enough timescales that I'd need to repeatedly go back and run the dither alignment between lock attempts - not the end of the world but costs ~5mins per lock attempt.

On to the actual tests: before testing the hardware, I locked the PRMI (no ETMs). In this configuration, I'm surprised to see that there is nearly perfect coherence between the MICH and PRCL error signals between 100Hz-1kHz 🤔 . When the AS55 demodulated signals are whitened prior to digitization (and then de-whitened digitally), the coherence structure changes. The electronics noise (measured with the PSL shutter closed) itself is uncorrelated (as it should be), and below the level of the two aforementioned spectra, so it is some actual signal I'm measuring there with the PRMI locked, and the coherence is on the light fields on the photodiode. So it would seem that I am just injecting a ton of AS55 sensing noise into the PRCL loop via the MICH->PRM LSC output matrix element. Weird. The light level on the AS55 photodiode has increased by ~2x after the September 2020 vent when we removed all the unused output optics and copper OMC. Nevertheless, the level isn't anywhere close to being high enough to saturate the ADC (confirmed by time domain signals in ndscope).

To get some insight into whether the whole RF system is messed up, I first locked the arm cavities with POX and POY as the error signals. Attachment #3 shows the spectra and coherence betweeen these two DoFs (and the dark noise levels for comparison). This is the kind of coherence profile I would expect - at frequencies where the loop gain isn't so high as to squish the cavity length noise (relative to laser frequency fluctuations), the coherence is high. Below 10 Hz, the coherence is lower than between 10-100 Hz because the OLG is high, and presumably, we are close to the sensing noise level. And above ~100 Hz, POX and POY photodiodes aren't sensing any actual relative frequency fluctuations between the arm length and laser frequency, so it's all just electronics noise, which should be incoherent.

The analogous plot for the PRMI lock is shown in Attachment #4. I guess this is telling me that the MICH sensing noise is getting injected into the PRCL error point between 100Hz-1kHz, where the REFL11 photodiode (=PRCL sensor) isn't dark noise limited, and so there is high coherence? I tuned the MICH-->PRM LSC output matrix element to minimize the height of a single frequency line driving the BS+PRM combo at ~313Hz in the PRCL error point. 

All the spectra are in-loop, the loop gain has not been undone to refer this to free-running noise. The OLGs themselves looked fine to me from the usual DTT swept sine measurements, with ~100 Hz UGF.

I'm curious to see if the demod phase for MICH in REFL & AS chamges between thi simple Mcihelson and PRMI. IF there's a change, it could point to a PRCL/f2 mismatch.

But I would still bet on demod chain funniness.

I measured the conversion efficiencies for all the RFPD demod boards except the POP port ones. An RF source was used to drive the PD input on the demod board, one at a time, and the I/F outputs were monitored on a 300 MHz oscilloscope. The efficiency is measured as the percentage ratio V_IF / V_RF. 

I will upload the full report later, but basically, the numbers I measured today are within 10% of what I measured in 2017 when I previously did such a characterization. The orthogonality also seems fine. 

I believe I restored all the connections at 1Y2 correctly, and I can lock POX/POY and PRMI on 1f signals after my work. I will do the noise characterization tomorrow - but I think this test already rules out any funkiness with the demod setup (e.g. non orthogonality of the digitized "I" and "Q" signals). The whitening part of the analog chain remains untested.

Update 2/23 1215: I've broken up the results into the demod boards that do not (Attachment #1) and do (Attachment #2) have a D040179 preamp installed. Actually, the REFL11 AO path also has the preamp installed, but I forgot to capture the time domain data for those channels. The conversion efficiency inferred from the scope was ~5.23 V/V, which is in good agreement with what I measured a few years ago.

• The scope traces were downloaded.
• The resulting X/Y traces are fitted with ellipses to judge the gain imbalance and orthogonality.
• The parameter phi is the rotation of the "bounding box" for the fitted ellipses - if the I and Q channels are exactly orthogonal, this should be either 0 or 90 degrees. There is significant deviation from these numbers for some of the demodulators, do we want to do something about this? Anyways, the REFL11 and AS55 boards, which are used for PRMI locking, report reasonable values. But REFL165 shows an ellipse with significant rotation. This is probably how the CDS phase rotator should be tuned, by fitting an ellipse to the digitized I/Q data and then making the bounding box rotation angle 0 by adjusting the "Measured Diff" parameter.
• The gain imbalance seems okay across the board, better than 1dB.
• The POX and POY traces are a bit weird, looks like there is some non-trivial amount of distortion from the expected pure sinusoid.
• I measured the LO input levels going into each demod board - they all lie in the range 2-3dBm (measured with RF power meter), which is what is to be expected per the design doc. The exception the the 165 MHz LO line, which was 0.4 dBm. So this board probably needs some work. 
• As I mentioned earlier, the conversion efficiencies are consistent with what I measured in 2017. I didn't break out the Eurocards using an extender and directly probe the LO levels at various points, but the fact that the conversion efficiencies have not degraded and the values are consistent with the insertion loss of various components in the chain make me believe the problem lies elsewhere. 

For completeness, I will measure the input terminated I/F output noise levels later today. Note also that my characterization of the optical modulation profile did not reveal anything obviously wrong (to me at least). 

I measured the noise of the I/F outputs of all the LSC demodulators. I made the measurement in two conditions, one with the RF input to the demodulators terminated with 50 ohms to ground, and the other with the RFPD plugged in, but the PSL shutter closed (so the PD dark noise was the input to the demodulator). The LO input was driven at the nominal level for all measurements (2-3 dBm going in to the LO input, measured with the RF power meter, but I don't know what the level reaching the mixer is, because there is a complicated chain of ERA amplifiers and attenuators that determine what the level is). 

As in the previous elog, I have grouped the results into boards that do not (Attachment #1) and do (Attachment #2) have the low noise preamp installed. The top row is for the "Input terminated" measurements, while the bottom is with the RFPD plugged in, but dark. I think not a single board shows the "expected" noise performance for both I and Q channels. In the case where the preamp isn't installed and assuming the mixer is being driven with >17dBm LO, we expect the mixer to demodulate the Johnson noise of 50 ohms, which would be ~1nV/rtHz, and so with the SR785, we shouldn't measure anything in exceess of the instrument noise floor. With the low noise preamp installed, the expected output noise level is ~10nV/rtHz, which should just about be measurable (I didn't use any additional Low Noise front end preamp for these measurements). The AS55_I channel shows noise consistent with what was measured in 2017 after it was repaired, but the Q channel shows ~twice the noise. It seemed odd to me that the Q channels show consistently higher noise levels in general, but I confirmed that the SR785 channel 2 did not show elevated instrument noise at least when terminated with 50 ohms, so seems like a real thing.

While this is clearly not an ideal state of operation, I don't see how this can explain the odd PRMI sensing.

I did the characterization discussed at the meeting today.

1. RF signal at 100 Hz offset from the LO frequency was injected into the PD input on the demod boards.
2. The digitized CDS channels were monitored. I chose to look at the C1:LSC-{PD}_I_OUT and C1:LSC-{PD}_Q_OUT channels. This undoes the effect of the analog whitening, but is before the digital phase rotation.
3. Attachments #1 and Attachments #2 are for the case where the analog whitening is not engaged, white Attachments #3 and Attachments #4 are for when the whitening is engaged, and they look the same (as they should), which rules out any crazy mismatch between the analog filter and the digital dewhitening filter.
4. I have absorbed the flat whitening gain applied to the various PDs in the cts/V calibration indicated on these plots. So the size of the ellipse is proportional to the conversion gain.

I think this test doesn't suggest anything funky in the analog demod/whitening/AA/digitization chain. We can repeat this process after the demod boards are repaired and use the angle of rotation of the ellipse to set the "D" parameter in the CDS phase rotator part, I didn't do it today.

While working at the LSC rack, I lost the input pointing into the IFO (the TT wiring situation is apparently very fragile, and this observation supports the hypothesis that the drifting TTs are symptomatic of some electronics issue). After careful beam walking, it was recovered and the dither alignment system was used to maximize TRX/TRY once again. No lasting damage done. If I can figure out what the pin-mapping is for the TT coils in vacuum, I'm inclined to try installing the two HAM-A coil drivers to control the TTs. Does anyone know where I can find said pin-out? The wiki page links seem broken and there isn't a schematic available there...

Ok it should be possible to back it out from the BOSEM pin out, and the mapping of the in-vacuum quadrupus cable, though careful accounting of mirroring will have to be done... The HAM-A coil driver actually already has a 15 pin output like the iLIGO coil drivers that are currently in use, but the pin mapping is different so we can't just replace the unit. On the bright side, this will clear up 6U of rack space in 1Y2. In fact, we can also consider hooking up the shadow sensor part of the BOSEMs if we plan to install 2 HAM-A coil drivers + 1 Dual satellite amplifier combo (I'm not sure if this number of spares is available in what we ordered from Todd).

After all the work at the LSC rack over the last couple of days, I re-locked the PRMI (ETMs misaligned), and measured the sensing matrix once again. The PRMI was locked using 1f error signals, with AS55_Q as the MICH sensor and REFL11_I as the PRCL sensor. As shown in Attachment #1, the situation has not changed, there is still no separation between the DoFs in the REFL signals. I will measure the MC lock point offset using the error point dither technique today to see if there is something there.

The Marconi frequency was tuned by looking at 

1. The ~3.68 MHz (= 3*f1 - fIMC) peak at the IMC servo error point, TP1A, and
2. The ~25.8 MHz (= 5*f1 - fIMC) peak at the MC REFL PD monitor port. The IMC error point is not a good place to look for this signal because of the post-demodulation low pass filter (indeed, I didn't see any peak above the analyzer noise floor).

The nominal frequency was 11.066209 MHz, and I found that both peaks were simultaneously minimized by adjusting it to 11.066195 MHz, see Attachment #1. This corresponds to a length change of ~20 microns, which I think is totally reasonable. I guess the peaks can't be nulled completely because of imbalance in the positive and negative sidebands. 

Then, I checked for possible offsets at the IMC error point, by injecting a singal to the AO input of the IMC servo board (using the Siglent func gen), at ~300 Hz. I then looked at the peak height at the modulation frequency, and the second harmonic. The former should be minimized when the cavity is exactly on resonance, while the latter is proportional to the modulation depth at the audio frequency. I found that I had to tweak the MC offset voltage slider from the nominal value of 0V to 0.12 V to null the former peak, see Attachment #2. After accounting for the internal voltage division factor of 40, and using my calibration of the IMC error point as 13 kHz/V, this corresponds to a 40 Hz (~50 microns) offset from the true resonant point. Considering the cavity linewidth of ~4 kHz, I think this is a small detuning, and probably changes from lock to lock, or with time of day, temperature etc.

Conclusion: I think neither of these tests suggest that the IMC is to blame for the weirdness in the PRMI sensing, so the mystery continues.

For magnet strength measurement: There is a gaussmeter in the flukes' drawer (2nd from the top). It turns on and reacts to a whiteboard magnet.

I think the PSL HEPA (both 2 units) are not running. The switches were on. And the variac was changed from 60% to 0%~100% a few times but no success.
I have no troubleshooting power anymore today. The main HEPA switch was turned off.

From the last failure, I had ordered 2 extra capacitors (they are placed on top of the PSL enclosure above where the capacitors would normally be installed). If the new capacitors lasted < 6months, may be symptomatic of some deeper problem though, e.g. the HEPA fans themselves need replacing. We don't really have a good diagnostic of when the failure happened I guess as we don't have any channel recording the state of the fans.

[Paco, Anchal]

Came in a little bit after 8 and found the MC unlocked and struggling to lock for the past 3 hours. Looking at the SUS overview, both MC1 and ITMX Watchdogs had tripped so we damped the suspensions and brought them back to a good state. The autolocker was still not able to catch lock, so we cleared the WFS filter history to remove large angular offsets in MC1 and after this the MC caught its lock again.

Looks like two EQs came in at around 4:45 AM (Pacific) suggested by a couple of spikes in the seismic rainbow, and this.

- Last week we found both of the PSL HEPA units were not running.

- I replaced the capacitor of the north unit, but it did not solve the issue. (Note: I reverted the cap back later)
- It was found that the fans ran if the variac was removed from the chain.
- But I'm not certain that we can run the fans in this configuration with no attendance considering fire hazard.

@3AM: UPON LEAVING the lab, I turned off the HEPA. The AC cable was not warm, so it's probably OK, but we should wait for the continuous operation until we replace the scorched AC cable.

The capacitor replacement was not successful. So, the voltages on the fan were checked more carefully. The fan has the three switch states (HIGH/OFF/LOW). If there is no load (SW: OFF), the variac out was as expected. When the load was LOW or HIGH, it looked as if the motor is shorted (i.e. no voltage difference between two wires).

I thought the motors may have been shorted. But if the load resistance was measured with the fluke meter, it showed some resistance

- North Unit: SW LOW 4.6Ohm / HIGH 6.0Ohm
- South Unit: SW LOW 6.0Ohm / HIGH 4.6Ohm (I believe the internal connection is incorrect here)

I believed the motors are alive! Then the fans were switched on with the variac removed... they ran. So I set the switch LOW for the north unit and HIGH for the south unit.

Then I inspected the variac:

• The AC output has some liquid leaking (oil?) (Attachment 1)
• The AC plug on the variac out has a scorch mark (Attachments 2/3)

So, this scorched AC plug/cable connected directly to the AC right now. I'm not 100% confident about the safety of this configuration.
Also I am not sure what was wrong with the system.

• Has the variac failed first? Because of the heat? I believe that the HEPA was running @30% most of the time. Maybe the damage was already there at the failure in Nov 2020?
• Or has the motor stopped at some point and this made the variac failed?
• Was the cable bad and the heat made the variac failed (then the problem is still there).

So, while I'm in the lab today, I'll keep the HEPA running, but upon my taking off, I'll turn it off. We'll discuss what to do in the meeting tomorrow.

Once again, I found the door to the outside in the control room open when I came in ~1215pm. I closed it.

Sorry about that. It must be me. I'll make sure it doesn't happen again. I was careless to not check back, no further explanation.

The C1:IFO-STATE variable is actually a bunch (16 to be precise) of bits, and the byte they form (2 bytes) converted to decimal is what is written to the EPICS channel. It was reported on the call today that the nominal value of the variable when the IMC is locked was "8", while it has become "10" today. In fact, this has nothing to do with the IMC. You can see that the "PMC locked" bit is set in Attachment #1. This is done in the AutoLock.sh PMC autolocker script, which was run a few days ago. Nominally, I just lock the PMC by moving some sliders, and I neglect to set/unset this bit.

Basically, there is no anomalous behavior. This is not to say that the situation cannot be improved. Indeed, we should get rid of the obsolete states (e.g. FSS Locked, MZ locked), and add some other states like "PRMI locked". While there is nothing wrong with setting these bits at the end of execution of some script, a better way would be to configure the EPICS record to automatically set / unset itself based on some diagnostic channels. For example, the "PMC locked" bit should be set if (i) the PMC REFL is < 0.1 AND (ii) PMC TRANS is >0.65 (the exact thresholds are up for debate). Then we are truly recording the state of the IFO and not relying on some script to write to the bit (I haven't thoguht through if there are some edge cases where we need an unreasonable number of diagnostic channels to determine if we are in a certain state or not).

Maybe tighten the tensioner on the door closer so that it closes by itself even in the low velocity case. Or maybe just use the front door like everyone else?

That makes sense. I assumed that IFO-STATE is configured as you have proposed it to be configured. This could be implemented in later.

I was looking at the laser head/amp and somehow decided to open the glue freezer. And it was stuck. I've managed to open it but the upper room was completely frozen.
Some of the batteries were embedded in a block of ice. I think we should throw them out.

Can the person who comes in the morning work on defrosting?

- Coordinate with Yehonathan and move the amps and the wooden crate so that you can move the freezer.

- Remove the contents to somewhere (it's OK to be room temp for a while)

- Unplug the freezer

- Leave the freezer outside with the door open. After a while, the ice will fall without care.

- At the end of the day, move it back to the lab. Continue defrosting the other day if the ice remains.

{Paco, Anchal, Yehonathan}

We emptied the fridge and moved the amplifier equipment on top of the amplifier crate. We unplugged the freezer and moved it out of the lab to defrost (attachment).

There is still a huge chunk of unmelted ice in the fridge. I moved the content of that fridge in the main fridge and put "do not eat" warning signs.

I returned the fridge to the lab and plugged it back in to prevent flooding.

Defrosting will have to continue on Monday.

{Anchal, Paco, Yehonathan}

We took the glue fridge outside.

{Paco, Yehonathan}

We broke the last chunk of ice and cleaned the fridge. We move the fridge back inside and plugged it into the wall. The glues were moved back from the main fridge.

The batteries that were found soaking wet are now somewhat dry and were left on the cabinet drawers for future recycling.

Yikes!  That's ONE filter.  I'll get another from storage.

It's probably too late to say but there are/were two boxes. (just for record)

For past few days, a weird sound of decaying gas leakage comes in the 40m control room from the south west corner of ceiling. Attached is an audio capture. This comes about every 10 min or so. 

I also noticed some sound in the control room. (didn't open the MP3 yet)

I'm afraid that the hard disk in the control room iMac is dying.

I put the box containing the untested OSEMs from KAGRA near the south flow bench on the floor.