Yesterday we noticed that the POS and SIDE eigenmodes were degenerate (with 1mHz spectral resolution). Moreover, the YAW peak had shifted down by ~500 mHz compared to earlier this week, although there was still good separation between PIT and YAW in the Oplev error signals. Ideas were (i) check if EQ stops were not backed out sufficiently, and (ii) look for any fibers/other constraints in the system. Today morning, I inspected the optic again. I felt the EQ stop viton tips were a bit close to the optic, so I backed them out further. Apart from this, I adjusted the LR and SIDE OSEM position in their respective holders to make the sensor voltages closer to half-light. Kicked the optic again just now, let's see if there is any change.

Remaining tasks:

1. Check EY table leveling.
2. Check EY actuation matrix diagonality using this technique.
3. Check that IR resonances are seen (and all the usual pre-pumpdown alignment checks).
4. Take close out pictures.
5. Heavy doors on, pump down.

If everything goes smoothly, I think we should plan for the heavy doors going back on and commencing the pumpdown tomorrow. After discussion with Koji, we came to the conclusion that it isn't necessary to investigate IPANG (high likelihood of it falling off the steering optics during the pumpdown) / AS beam clipping (no strong evidence that this is a problem) for this vent.

Update 1235: Indeed, the eigenmodes are back to their positions from earlier this week. Indeed, the POS and SIDE modes are actually better separated! So, the OSEM/magnet and EQstop/optic interactions are non-negligible in the analysis of the dynamics of the pendulum.

So Cal Earthquake. All suspension watchdogs tripped.

Tried to recover the OSEM damping. 

=> The watchdogs for all suspensions except for ITMX were restored. ITMX seems to be stuck. No further action by me for now.

[Kruthi, Koji]

Koji came to the lab to align the IMC/IFO, but found the mirrors are dancing around. Kruthi told me that there was M7.1 EQ at Ridgecrest. Looks like there are aftershocks of this EQ going on. So we need to wait for an hour to start the alignment work.

ITMX and ETMX are stuck.

- ITM unstuck now
- IMC briefly locked at TEM00

A series of aftershocks came. I could unstick ITMX by turning on the damping during one of the aftershocks.
Between the aftershocks, MC1~3 were aligned to the previous dof values. This allowed the IMC flashing. Once I got the lock of a low order TEM mode, it was easy to recover the alignment to have a weak TEM00.
Now at least temporarily the full alignment of the IMC was recovered.

In fact, ETMX was not stuck until the M7.1 EQ today. After that it got stuck, but during the after shocks, all the OSEMs occasionally showed full swing of the light levels. So I believe the magnets are OK.

We unstuck ETMX by shaking the stack. Most effective was to apply large periodic human sized force to the north STACIS mounts.

At first, we noticed that the face OSEMs showed nearly zero variation.

We tried unsticking it through the usual ways of putting large excitations through AWG into the pit/yaw/side DOFs. This produced only ~0.2 microns of motion as seen by the OSEMs.

After the stack shake, we used the IFO ALIGN sliders to get the oplev beam back on the QPD.

The ETMX sensor trends observed before and after the earthquake are attached.

** plots deleted; SOMEONE, tried to take raster images and turn them into PDF as if this would somehow satisfy our vetor graphics requirement. Boo. lpots must be actual vector graphics PDF

After this activity, the DC bias voltage required on ETMX to restore good X arm cavity alignment has changed by ~1.3 V. Assuming a full actuation range of 30 mrad for +/- 10 V, this implies that the pitch alignment of the stack has changed by ~2 mrad? Or maybe the suspension wires shifted in the standoff grooves by a small amount? This is ~x10 larger than the typical change imparted while working on the table, e.g. during a vent.

Main point is that this kind of range requirement should probably be factored in when thinking about the high-voltage coil driver actuation.

Arnaud and I moved one of the two spare TT suspensions from the south clean cabinet to the bake lab clean room. The main purpose was to inspect the contents of the packaging. According to the label, this suspension was cleaned to Class A standards, so we tried to be clean while handling it (frocks, gloves, masks etc). We found that the foil wrapping contained one suspension cage, with what looked like all the parts in a semi-assembled state. There were no OSEMs or electronics together with the suspension cage. Pictures were taken and uploaded to gPhoto. Arnaud is going to plan his tests, so in the meantime, this unit has been stored in Cabinet #6 in the bake lab cleanroom.

[koji, gautam]

We noticed that the PRM watchdog was tripping frequently. This is a period of enhanced seismic activity. The reason PRM in particular trips often is because the SIDE OSEM has 5x increased transimpedance. We implemented a workaround by modifying the watchdog tripping condition to scale the SD channel RMS by a factor of 0.2 (relative to the UL and LL channels). We restarted the modbus process on c1susaux and tested that the new logic works. Here is the relevant snippet of code:

The db file has a note about this as well so that future debuggers aren't mystified by a factor of 0.2.

All suspension watchdogs were tripped ~90mins ago. I restored the damping. IMC is locked.

ITMX was stuck. I set it free. But notice that the UL Sensor RMS is higher than the other 4? I thought ITMY UL was problematic, but maybe ITMX has also failed, or maybe it's coincidence? Something for IFOtest to figure out I guess. I don't think there is a cable switch between ITMX/ITMY as when I move the ITMX actuators, the ITMX sensors respond and I can also see the optic moving on the camera.

Took me a while to figure out what's going on because we don't have the seis BLRMS - i moved the usual projector striptool traces to the TV screen for better diagnostic ability.

Update 16 July 1515: Even though the RMS is computed from the slow readback channels, for diagnosis, I looked at the spectra of the fast PD monitoring channels (i.e. *_SENSOR_*) for ITMX - looks like the increased UL RMS is coming from enhanced BR-mode coupling and not of any issues with the whitening switching (which seems to work as advertised, see Attachment #3, where the LL traces are meant to be representative of LL, LR, SD and UR channels).

There were several small/medium earthquakes in Ridgecrest and one medium one in Blackhawk CA at about 2000 UTC (i.e. ~ 2 hours ago), one of which caused BS, ITMY, and ETM watchdogs to trip. I restored the damping just now.

Rana and I talked about some (genius) options for the large range DC bias actuation on the SOS, which do not require us to supply high-voltage to the OSEMs from outside the vacuum.

What we came up with (these are pretty vague ideas at the moment):

1. Some kind of thermal actuation.
2. Some kind of electrical actuation where we supply normal (+/- 10 V) from outside the vacuum, and some mechanism inside the chamber integrates (and hence also low-pass filters) the applied voltage to provide a large DC force without injecting a ton of sensor noise.
3. Use the blue piers as a DC actuator to correct for the pitch imbalance --- Kruthi and Milind are going to do some experiments to investigate this possibility later today.

For the thermal option, I remembered that (exactly a year ago to the day!) when we were doing cavity mode scans, once the heaters were turned on, I needed to apply significant correction to the DC bias voltage to bring the cavity alignment back to normal. The mechanism of this wasn't exactly clear to me - furthermore, we don't have a FLIRcam picture of where the heater radiation patter was centered prior to my re-centering of it on the optic earlier this year, so we don't know what exactly we were heating. Nevertheless, I decided to look at the trend data from that night's work - see Attachment #1. This is a minute trend of some ETMY channels from 0000 UTC on 18 July 2018, for 24 hours. Some remarks:

1. We did multiple trials that night, both with the elliptical reflector and the cylindrical setup that Annalisa and Terra implemented. I think the most relevant part of this data is starting at 1500 UTC (i.e. ~8am PDT, which is around when we closed shop and went home). So that's when the heaters were turned off, and the subsequent drift of PIT/YAW are, I claim, due to whatever thermal transients were at play.
2. Just prior to that time, we were running the heater at close to its maximum rated current - so this relaxation is indicative of the range we can get out of this method of actuation.
3. I had wrongly claimed in my discussion with Rana this morning that the change in alignment was mostly in pitch - in fact, the data suggests the change is almost equal in the two DoFs. Oplev and OSEMs report different changes though, by almost a factor of 2....
4. The timescale of the relaxation is ~20 minutes - what part(s) of the suspension take this timescale to heat up/cool down? Unlikely to be the wire/any metal parts because the thermal conductivity is high? 
5. In the optimistic scenario, let's say we get 100 urad of actuation range - over 40m, this corresponds to a beam spot motion of ~8mm, which isn't a whole lot. Since the mechanism of what is causing this misalignment is unclear, we may end up with significantly less actuation range as well.
6. I will repeat the test (i.e. drive the heater and look for drift in the suspension alignment using OSEMs/Oplev) in the afternoon - now I claim the radation pattern is better centered on the optic so maybe we will have a better understanding of what mechanisms are at play.

Also see this elog by Terra.

Attachment #2 shows the results from today's heating. I did 4 steps, which are obvious in the data - I=0.6A, I=0.76A, I=0.9A, and I=1.05A.

In science, one usually tries to implement some kind of interpretation. so as to translate the natural world into meaning.

[Kruthi, Milind]

On Friday, Milind and I performed the pitch adjustment test Rana had asked us to do. Only 1 blue beam in case of ITMX and two in case of ETMY, ETMX and ITMY were accessible. Milind (of mass 72 kg as of 10 May 2019) stood on each of the accessible blue beams of the test mass chambers for one minute and I recorded the corresponding gps time. Before moving to the next beam, we spared more than a minute for relaxation after the standing end time. Following are the recorded gps times. 

PS: For each blue beam relaxation time ~ 1 min after the standing end time

Koji suggested systematic investigation of the ETMX suspension electronics. The tests to be done are:

1. Characterization of PD whitening amplifiers - with the satellite box disconnected, we will look for glitches in the OSEM channels.
2. Characterization of LT1125s in the PD chain of the amplifiers - with the in-vacuum OSEMs disconnected, we will look for glitches due to the on-board transimpedance amplifiers of the satellite box.
3. Characterization using the satellite box tester - this will signal problems with the physical OSEMs.
4. Characterization of the suspension coil driver electronics - this will happen later.

So the ETMX satellite box is unplugged now, starting 530 pm PDT.

The satellite box was reconnected and the suspension was left with watchdog off but OSEM roughly centered. We will watch for glitches over the weekend.

Looking at the sensor and oplev trends over the weekend, there was only one event where the optic seems to have been macroscopically misaligned, at ~11:05:00 UTC on Oct 19 (early Saturday morning PDT). I attach a plot of the 2kHz time series data that has the mean value subtracted and a 0.6-1.2 Hz notch filter applied to remove the pendulum motion for better visualization. The y-axis calibration for the top plot assumes 1 ct ~= 1 um. This "glitch" seems to have a timescale of a few seconds, which is consistent with what we see on the CCD monitors when the cavity is locked - the alignment drifts away over a few seconds.

As usual, this tells us nothing conclusive. Anyways, I am re-enabling the watchdog and pushing on with locking activity and hope the suspension cooperates.

While I was trying to lock the PRMI this evening, I noticed that I couldn't move the REFL beamspot on the CCD field of view by adjusting the slow bias voltages to the PRM. Other suspensions controlled by c1susaux seem to respond okay so at first glance it isn't a problem with the Acromag. Looking at the OSEM sensor input levels, I noticed that UL is much lower than the others - see Attachment #1, seems to have happened ~100 days ago. I plugged the tester box in to check if the problem is with the electronics or if this is an actual shorting of some pins on the physical OSEM as we had in the past. So PRM watchdog is shutdown for now and there is no control of the optic available as the cables are detached. I will replace the connections later in the evening.

Update 10pm:

1. Measured coil inductances with breakout board and LCR meter - all 5 coils returned ~3.28-3.32 mH.
2. Measured coil resistances with breakout board and DMM - all 5 coils returned ~16-17 ohms.
3. Checked OSEM PD capacitance (with no bias voltage) using the LCR meter - each PD returned ~1nF.
4. Checked resistance between LED Cathode and Anode for all 5 LEDs using DMM - each returned Hi-Z.
5. Checked resistance between PD Cathode and Anode for all 5 PDs using DMM - each returned ~430 kohms.
6. Checked that I could change the slow bias voltages and see a response at the expected pins (with the suspension disconnected).

Since I couldn't find anything wrong, I plugged the suspension back in - and voila, the suspect UL PD voltage level came back to a level consistent with the others! See Attachment #2.

Anyway, I had some hours of data with the tester box plugged in - see Attachment #3 for a comparison of the shadow sensor readout with the tester box (all black traces) vs with the suspension plugged in, local damping loops active (coloured traces). The sensing noise re-injection will depend on the specifics of the  local damping loop shapes but I suspect it will limit feedforward subtraction possibilities at low frequencies.

However, I continue to have problems aligning the optic using the slow bias sliders (but the fast ones work just fine) - problem seems to be EPICS related. In Attachment #4, I show that even though I change the soft PITCH bias voltage adjust channel for the PRM, the linked channels which control the actual voltages to the coils take several seconds to show any response, and do so asynchronously. I tried restarting the modbus process on c1susaux, but the problem persists. Perhaps it needs a reboot of the computer and/or the acromag chassis? I note that the same problem exists for the BS and PRM suspensions, but not for ITMX or ITMY (didn't check the IMC optics). Perhaps a particular Acromag DAC unit is faulty / has issues with the internal subnet?

Sigh... hard loch

Looks like a M=4.6 earthquate in Barstow,CA tripped all the suspensions. ITMX got stuck. I restored the local damping on all the suspensions just now, and freed ITMX. Looks like all the suspensions damp okay, so I think we didn't suffer any lasting damage. IMC was re-aligned and is now locked.

In the last couple days, as the IMC ringdowns have been going on, we have noticed that the MC is behaving bad. Misaligning, drifting, etc.

Gautam told me a horror story about him, Koji, and melted wires inside the sat boxes.

I said, "Its getting too hot in there. So let's take the lids off!"

So then we:

1. Removed the lid (only 4 screws were still there)
2. cut off some of the shield - ground wires and insulated them with electrical tape
3. squished the IDC connectors on tightly
4. left it this way to see if MC would get better - certainly the painfully hot heatinks inside the box were now just 110 F or so

After some minutes, we saw no drifting. So maybe my theory of "hot heatsink partially shorting a coil current to GND through partially melted ribbon cable" makes sense? IF this seems better after a month, lets de-lid all the optics.

Let's look at some longer trends and be very careful next to MC2 for the next 3 days! I have put a dangerous mousetrap there to catch anyone who walks near the vacuum chamber.

gautam: the grounding situation per my assessment is that the shield of all the IDC cables are connected to a common metal strip at 1X5 - but in my survey, I didn't see any grounding of this strip to a common ground.

An earthquake around 330 UTC (=730pm yesterday eve) tripped ITMX, ITMY and ETMX watchdogs. ITMX got stuck. I released the stuck optic and re-enabled the local damping loops just now.

ETMX was grossly misaligned.

I re-aligned it and the X arm now locks.

7:00PM with Koji

Both the alignment of the X and Y arms was recovered.

~>z avg 10 C1:LSC-TRX_OUT C1:LSC-TRY_OUT
C1:LSC-TRX_OUT 0.9914034307003021
C1:LSC-TRY_OUT 0.9690877735614777

We are running ass for the X arm to recover the X arm alignment.

Meanwhile, i want to block the Y arm trans PD (Thorlabs). To do it, the PD<->QPD thresholds were changed from 5.0/3.0 to 0.5/0.3.

I returned the triggering threshold to normal values (5/3).

This EQ in Nevada seems to have tripped all watchdogs. ITMX was stuck. It was released, and all the watchdogs were restored. Now the IMC is locked.

This EQ seems to have knocked all suspensions out. ITMX was stuck. It is now released, and the IMC is locked again. It looks like there are some serious aftershocks going on so let's keep an eye on things.

Summary:

I found that there is an issue with the MC1 slow bias voltages. 

Details:

I usually offload the DC part of the output voltage from the WFS servos to the slow bias voltage sliders, so as to preserve maximum actuation range from the fast system. However, today, I found that this servo wasn't working well at all. So I dug a little deeper. Looking at the EPICS database records:

• The user-facing channels are "PIT" and "YAW" bias voltages.
• These are converted to voltages to be sent to individual coils by some calc channels in the EPICS database record. So, for example, the voltage to be sent to the "UL" coil (Upper Left, as viewed from the AR side of the optic), is A+B, where A is the "PIT" voltage and B is the "YAW" voltage. Similar combinations of A and B are used for the other 3 face coils.
• The problem is obvious - if either A or B > 5V, then the requested voltage to be sent to the UL coil is > 10 V, while the Acromag DACs can put out a maximum of 10 V. 
• As it happens, with the IFO currently aligned, MC1 is the only optic which faces this problem. 
• Why has this not been an issue before? In fact, looking at some old data, the "PIT" and "YAW" bias voltages to MC1 were both ~1-2 V in 2018. But I confirmed that something in the region of ~5 V is required from each of the "PIT" and "YAW" channels to bring the MCREFL spot back to the center of the camera, so something has changed the DC alignment of MC1, maybe an earthquake or something? Anyway, with these settings, 2/4 coils are basically saturated, and so we can only move the optic diagonally. 😢 
• Other coils that have  requested output voltages > 5V (so more than half the range of the DAC) include MC2 LL (5.2V), and ETMX LL and LR (5.5 and 5.8 V respectively).
• Either a factor of 0.5 should be included in all the EPICS database records, or else, we should make the "PIT" and "YAW" sliders range only from -5 to +5 V, so that this kind of misleading info isn't wasting time.

We can limit the EPICS values giving some parameters to the channels. cf https://epics.anl.gov/tech-talk/2012/msg00147.php

But this does not solve the MC1 issue. Only we can do right now is to make the output resister half, for example.

This earthquake tripped all suspensions and ITMX got stuck. The watchdogs were restored and the stuck optic was released. The IFO was re-aligned, POX/POY and PRMI on carrier locking all work okay.

I implemented this change today. We only had 100 ohm, 3W resistors in stock (no 200 ohm with adequate power rating). Assuming 10 V is dropped across this resistor, the power dissipation is V^2/R ~ 1 W, so we should have sufficient margin. DCC entry has been updated with new schematic and photo of the component side of the board. Note that the series resistance of the fast actuation path was untouched.

As expected, the requested voltage no longer exceeds the Acromag DAC range, it is now more like 2.5 V. However, I still notice that the MC REFL spot moves somewhat diagonally on the camera image - so maybe the coil gains are seriously imbalanced? Anyway, the WFS control signals can once again be safely offloaded to the slow bias voltages once again, preserving the fast ADC range for other actuation.

The Johnson noise of the series resistor has now increased by a factor of 2, from ~6.4 pA/rtHz to 12.8 pA/rtHz. Assuming a current to force coefficient of 1.6 mN/A per coil, the length noise of the cavity is expected to be 12.8e-12 * 0.064/0.25/(2*pi*100)^2 ~ 8e-18 m/rtHz at 100 Hz. In frequency units, this is 80 uHz/rtHz. I think our IMC noise is at least 10 times higher than this at 100 Hz (in any case, the noise of the coil driver is NOT dominated by the series resistance). Attachment #1 confirms that there isn't any significant MCF noise increase, and I will check with the arm cavity too. Nevertheless, we should, if possible, align the optic better and use as high a series resistance as possible.

The watchdog for MC1 was disabled and the board was pulled out for this work. After it was replaced, the IMC re-locks readily.

Judging by the summary pages, some 18 hours after this change was made and the board re-installed, the MC1 shadow sensors began to report frequent glitches. I can't think of a plausible causal connection, especially given the 18 hour time lag, but also hard to believe there isn't one? As a result, the IMC is no longer able to stay locked for extended periods of time. I did the usual cable squishing, and also took off the lid to see if that helps the situation.

While the reduced series resistance means there is more current flowing through the slow path, 

1. There isn't actually an increase in the net current flowing through the satellite box - this change just re-allocates the current from the fast path to the slow path, but by the time it reaches the satellite box, the current is flowing through the same conductor.
2. afaik, the current buffers on the coil driver aren't overdriven - they are rated for 300 mA. No individual coil is drawing more than 30 mA.
3. the resistors themselves should be running sufficiently below their rated power of 3W (I estimate 2.5 V ^2 / 100 ohms ~ 60 mW).
4. The highest current should be through the UL and LR coils according to the voltage outputs from the Acromag. But the UL coil doesn't show significant glitching, and the LL one does despite drawing negligible DC current.

The attached FLIR camera image re-inforces what we already know, that the thermal environment inside the satellite box is horrible. The absolute temperature calibration may be off, but it was difficult to touch the components with a bare finger, so I'd say its definitely > 70 C.

does the FLIR have an option to export image with a colorbar?

How about just leave the lid open? or more open? I don't know what else can be done in the near term. Maybe swap with the SRM sat box to see if that helps?

Hmm I can't seem to export with the colorbar, might be just my phone though. I tried to add some "cursors" with the temperature at a few spots, but the font color contrast is poor so you have to squint really hard to see the temperatures in the photo I attached.

I'll leave the MC1 box open overnight and see if that improves the situation, and if not, I'll switch in the SRM satellite box tomorrow.

There was no improvement to the situation overnight. So, I did the following today:

1. Ramped bias voltages for SRM and MC1 to 0, shutdown watchdogs.
2. Switched SRM and MC1 satellite boxes. The SRM satellite box lid was opened, while the MC1 lid was left open. The boxes have also been re-labelled lest there be some confusion about which box belongs where.
3. Restored watchdogs and bias voltages. Curiously, the MC1 optic now only requires half the bias voltages it did before to have the correct DC alignment for the optic. The Satellite box is just supposed to be a passive conduit for the drive current, so this is indicative of some PCB traces/cabling being damaged inside what was previously the MC1 satellite box?

IMC is now locked again, I will monitor for glitching/stability.

Update 6pm PDT: as shown in Attachment #1, there is a huge difference in the stability of the lock after the sat box swap. Let's hope it stays this way for a while...

Sigh. Do we have a spare sat box?

This earthquake and friends had tripped all watchdogs. I used the scripted watchdog re-enabler, and released the stuck ITMX (this operation is still requires a human and hasn't been scripted yet). IMC is locked again and all Oplevs report healthy optic alignment.

Summary:

After the earthquake on September 19 2020, it looks to me like the only lasting damage to suspensions in vacuum is the ETMY UR magnet being knocked off. 

Suspension ringdown tests:

I did the usual suspension kicking/ringdown test:

• One difference is that I now kick the suspension "N" times where N is the number of PSD averages desired. 
• After kicking the suspension, it is allowed to ring down with the damping disabled, for ~1100 seconds so that we can get spectra with 1mHz resolution.
• We may want to get more e-folding times in, but since the Qs of the modes are a few hundred, I figured this is long enough.
• I think this kind of approach gives better SNR than letting it ringdown 10,000 seconds (for 10 averages with 10 non overlapping segments of 1000 seconds), and I wanted to test this scheme out, seems to work well.
• Attachment #1 shows a summary of the results.
• Attachment #2 has more plots (e.g. transfer function from UL to all other coils), in case anyone is interested in more forensics. The data files are large but if anyone is interested in the times that the suspension was kicked, you can extract it from here.

Conclusions:

1. My cursory scans of the analysis don't throw up any red flags (apart from the known problem of ETMY UR being dislodged) 👌 . 
2. The PRM data is weird 
   • I believe this is because the DC bias voltage to the coils was significantly off from what it normally is when the PRC is aligned.
   • In any case, I am able to lock the PRC, so I think the PRM magnets are fine.
3. The PRC angular FF no longer works turns out this was just a weird interaction with the Oplev loop because the beam was significantly off-centered on the Oplev QPD. Better alignment fixed it, the FF works as it did before.
   • With the PRC locked and the carrier resonant (no ETMs), the old feedforward filters significantly degrade the angular stability to the point that the lock is lost.
   • My best hypothesis is that the earthquake caused a spot shift on PR2/PR3, which changed the TF from seismometer signal to PRC spot motion.
   • Anyways, we can retrain the filter.
   • The fact that the PRC can be locked suggest PR2/PR3 are still suspended and okay.
4. The SRM data is also questionable, because the DC bias voltage wasn't set to the values for an aligned SRC when the data was collected
   • Nevertheless, the time series shows a clean ringdown, so at least all 5 OSEMs are seeing a signal.
   • Fact that the beam comes out at the AS port suggest SR3/SR2 suspensions are fine 👍 

Attachment #2 also includes info about the matrix diagonalization, and the condition numbers of the resulting matrices are as large as ~30 for some suspensions, but I think this isn't a new feature. 

The MC1 suspension has begun to show evidence of glitches again, from Friday/Saturday. You can look at the suspension Vmon tab a few days ago and see that the excess fuzz in the Vmon was not there before. The extra motion is also clearly evident on the MCREFL spot. I noticed this on Saturday evening as I was trying to recover the IMC locking, but I thought it might be Millikan so I didn't look into it further. Usually this is symptomatic of some Satellite box issues. I am not going to attempt to debug this anymore.

As discussed at the meeting, I decided to effect a satellite box swap for the misbehaving MC1 unit. I looked back at the summary pages Vmon for the SRM channels, and found that in the last month or so, there wasn't any significant evidence of glitchiness. So I decided to effect that swap at ~4pm today. The sequence of steps was:

• SRM and MC1 watchdogs were disabled.
• Unplugged the two satellite boxes from the vacuum flanges.
• For the record: S/N 102 was installed at MC1, and S/N 104 was installed at SRM. Both were de-lidded, supposedly to mitigate the horrible thermal environment a bit. S/N 104 was the one Koji repaired in Aug 2019 (the serial number isn't visible or noted there, but only one box has jumper wires and Koji's photos show the same jumper wires). In June 2020, I found that the repaired box was glitching again, which is when I swapped it for S/N 102.
• After swapping the two units, I re-enabled the local damping on both optics, and was able to re-lock the IMC no issues.

One thing I was reminded of is that the motion of the MC1 optic by controlling the bias sliders is highly cross-coupled in pitch and yaw, it is almost diagonal. If this is true for the fast actuation path too, that's not great. I didn't check it just now.

While I was working on this, I took the opportunity to also check the functionality of the RF path of the IMC WFS. Both WFS heads seem to now respond to angular motion of the IMC mirror - I once again dithered MC2 and looked at the demodulated signals, and see variation at the dither frequency, see Attachment #1. However, the signals seem highly polluted with strong 60 Hz and harmonics, see the zoomed-in time domain trace in Attachment #2. This should be fixed. Also, the WFS loop needs some re-tuning. In the current config, it actually makes the MC2T RIN worse, see Attachment #3 (reference traces are with WFS loop enabled, live traces are with the loop disabled - sorry for the confusing notation, I overwrote the patched version of DTT that I got from Erik that allows the user legend feature, working on getting that back).

I acquired several spare OSEMs (in unknown condition) from Paco. They are stored alongside the shipment from UF.

I wanted to check the functionality of the IMC WFS. I just turned on the WFS servo loops as they were. For the past two hours, they didn't run away. The servo has been left turned on. I don't think there is no reason to keep it turned off.

For whatever reason, the autolocker didn't turn the tickle off for several hours. Seems to work okay now. The linked plot suggests that the coil balancing on MC2 is pretty lousy.

Summary:

1. We will need de-whitening filters for the BHD relay optics in order to meet the displacement noise requirements set out in the DRD. I think these need not be remotely switchable (depends on specifics of LO phase control scheme). SR2, PR2 and PR3 can also have the same config, and probably MC1, MC3 as well.
2. We will need de-whitening filters for the non test mass core IFO optics (PRM, SRM, BS, and probably MC2).
3. I am pretty sure we will not be able to have sufficient DAC range for the latter class of optics if we have to:
   1. Supply the DC bias.
   2. Do the LSC and ASC actuation in the presence of reasonable sensing noise levels.
   3. Engage de-whitening to low-pass-filter the DAC noise at ~200 Hz.

Details:

Attachment #1 shows the DAC noise models for the General Standards 16-bit and 18-bit DACs we are expecting to have.

• The 16-bit model has been validated by me at the 40m a few years ago.
• We have never used the 18-bit flavor at the 40m, and there are all manner of quirks apparently related to zero crossings and such. So the noise may be up to x2 higher (we won't have as much freedom necessarily as the sites to bias the DAC on one side of the zero crossing if we also need to use the same DAC channel to supply the DC bias current for alignment.

Attachment #2 shows the expected actuation range for DC optic alignment, assuming we use the entire DAC range for this purpose.

• Clearly, we need to do other things with the same DAC channels as well, so this is very much an upper bound of what will be possible.
• Let's assume we will not go lower than 100ohms.
• For all new optics we are suspending, we should aim to get the pitch balancing to within 500urad. With a 2x2m=4m optical lever arm, this corresponds to a 2mm spot shift. Should be doable.
• This could turn out to be a serious problem for PRM, BS and SRM if we hope to measure squeezing - the <AUX DOF>-->DARM coupling could be at the level of -40dB, and at 200 Hz, the DAC noise would result in PRCL/MICH/SRCL noise at the level of ~10^-15m/rtHz, which would be 10^-17m/rtHz in DARM. I don't think we can get 20dB of feedforward cancellation at these frequencies. For demonstrating locking using a BHD error signal, maybe this is not a big deal.

Attachment #3 shows the current and proposed (by me, just a rough first pass, not optimized in any way yet) de-whitening filter shapes. These shapes can be tweaked for sure.

• The existing de-whitening filter is way too aggressive. FWIW, the DRD "models" a "4th order Chebyshev low pass filter" which doesn't exist anywhere as far as I know.
• Since the DAC noise is below 1 uV/rtHz at all frequencies of interest, we never need to have >60dB de-whitening anywhere as the input referred noise of any circuit we build will exceed 1 nV/rtHz.
• I propose 3 poles, 3 zeros. In the plot, these poles are located at 30Hz, 50Hz, 2kHz, and the zeros are at 300 Hz, 300 Hz, 800 Hz. 
• The de-whitening is less agressive below 100 Hz, where we still need significant LSC actuation ability. Considering the sensing noise levels at the 40m, I don't know if we can have reasonable LSC and ASC loop shapes and still have the de-whitening.
• Once again, PRM, SRM and BS will be the most challenging.
• For the BHD relay optics, once we have the de-whitening, we won't have the option of turning on a high-frequency (~kHz) dither line because of insufficient DAC range. 

Attachment #4 puts everything into displacement noise units. The electronics noise of the coil driver / de-whitening circuit have not been included so at high frequencies, the projection is better than what will actually be realizable, but still well below the BHD requirement of 3e-17 m/rtHz.

MC1 suspension is glitching again, so this is a good chance to install the new sat box and test it in the field.

As it currently stands the Center of Mass of the Adapter Ring/Optic assembly is 0.0175" out of the plane formed by the suspension wire. See Attachments. The side plate, along with the EQ stops are hidden to show the CoM and the plane.

Note: The changes discussed in the meeting with Calum have not been added and are a work in progress. These changes include:

- Adding a 45 deg chamfer to the both parallel faces of the adapter ring. This along with a modified bracket for the EQ stops will allow for easier adjustment of the screws. 

- Potentially changing material of adapter ring to stainless stell to more accurately emulate the mass of a 3" optic.

- Different adjustment mechanism of the "dumbell" at bottom of adapter ring to something similar to the VOPO suspension (will need to consult Calum further)

[jordan, gautam]

• Ran 60ft long cables from 1X4 to MC1/MC3 chamber flange, via overhead cable tray, and top of PSL enclosure for the last ~20ft. Note that it may be that the overhead cable trays cannot support the weight of the cables for 15 SOSs (total 30 shielded cables with 37 wires as twisted pairs) when we eventually add the optics for the BHD upgrade.
• Installed aLIGO satellite amplifer in 1X4.
• Tapped +/-20 V (which is the available voltage closest to the required +/-18V). For this, the Sorensens were powered down, and the actual taps were made from the fusable blocks powering the Trillium interface box. We made sure to leave an extra slot so that this kind of additional headache is not required for the next person doing such work.
• Once installed, I plugged in the dummy suspension box and verified that the unit performs as expected. 
• Some photos of the installation are here.

After this work, the IMC locked fine, the AS camera has the Michelson fringing, the fast CDS indicators are all green, and the seismometer BLRMS all look good - therefore, I claim no lasting damage was done as a direct result of today's work at 1X4. I will connect up the actual suspension at my leisure later today. Note that the MC1 glitches seem to have gone away, without me doing anything about it. Nevertheless, I think it's about time that we start testing the new hardware. 

Unrelated to this work: while I was testing some characteristics of the MC1 suspension (before we did any work in the VEA, you can see the timestamp in the ndscope), I noticed that the MC1 UL coil channel cannot actually be used to actuate on the optic. The coil driver Vmon channel demonstrates the appropriate response, which means that the problem is either with the Satellite box (it is just a feedthrough, so PCB trace damaged?) or with the OSEM itself (more likely IMO, will know more once I connect the new Satellite Amplifier up). I only show comparison for UL vs UR, but I checked that the other coils seem to be able to actuate the optic. This means we have been running for an indeterminate amount of time with only 3 face actuators on MC1, probably related to me having to do this work. 

Also unrelated to this work - while poking around at 1X5 rear, I noticed that the power connections to the existing Satellite Boxes are (understatedly) flaky, see connections to T1-T4 in Attachment #2..

Note from today's meeting:

1. Can we adjust the thickness of the cylindrical hole for the mirror to move the COM in the plane of the wires. (We should be able to do that)

and

2. Please check how much we can displace the COM by the bottom dumbbell.

There is some non-trivial sign flipping in the sensors/coils in this new setup because it is a hybrid one with the old interfacing electronics (D000210, D010001) and the new Satellite Amplifier (D080276). So I haven't yet gotten the damping working. I am leaving the PSL shutter closed and will keep working on this today/tomorrow. I have made various changes to the c1mcs realtime model and the c1susaux database record where MC1 is concerned. I have backups of the old ones so we can always go back to that if we so desire.

In the meantime, the PSL shutter is closed and there is no light to the IFO.

Update 1700: I've implemented some basic damping and now the IMC is now locked. The WFS loop runs away when I enable it, probably some kind of weird interaction with the (as of now untuned) MC1 local damping loops. I will write up a more detailed report later.

Update 2300: Did the following:

1. Re-calibrated the cts2um filter in all SENSOR filter banks to account for the increased transimpedance and LED drive current. I judged the overall scaling to be x0.25 but this can be calibrated against the bounce peak height for example (it lines up pretty well).
2. Re-measured the input matrix - it was very different from what was loaded. I am measuring this again overnight for some consistency.
3. Re-tuned the damping gains. Now the optic damps well, and the loops seem file to me, both via broadband noise injection TF and by step response metrics.
4. Yet, the WFS servo cannot be enabled. The WFS signal is summed in before the output matrix so I don't know why this would have a different behavior compared to the local damping, or indeed, why this has to be changed. Will need some (WFS) sensing/actuation matrix measurements to know more.

Dropping this for tonight, I'll continue tomorrow. Meanwhile, the OSEM input matrix measurement is being repeated overnight. PSL shutter is closed.

I am setting up a testing rig for the OSEMs we recently obtained. I found the schematic for the OSEM assembly from which the pin assignment can be read.

I connected the OSEM's pin plate to a female DB15 on a breakout board. I find the pin assignment (attachment 1, sorry for the image quality) to be:

There are several things that need to be done for each OSEM.

1. Measuring inductance of the coils. I checked that the measurement wires don't add any measurable inductance.

2. Check that the PDs and LEDs are alive (e.g. check forward voltage drop with fluke)

3. Energize the LED and PD.

4. Check PD DC level. For this, I might need the satellite box amplifier.

5. Check LED spot position on the PD.

6. Re-engrave OSEM S/N if needed.

I still need to figure a sensible scheme for points 3-5.

The WFS servo was recommissioned. The matrix can be tuned a bit more, but for now, I've recovered the old performance and the alignment doesn't seem to be running away, so I defer further tuning for later. The old Satellite box was handed over to Yehonathan for his characterization of the "spare" OSEMs.

This finishes the recovery of the MC1 suspension, I am now satisfied that the local damping loops are performing satisfactorily, that the WFS servo is also stable, and that POX/POY locking is recovered. On MC1, we even have 4 actuatable face OSEMs and the PIT(YAW) bias adjust slider even moves the optic in PIT(YAW), what a luxury. 

I've SDFed all the changes, and have backup of the old realtime model and C1SUSAUX_MC1 database files if we want to go back for whatever reason. The changes required to make this suspension work are different from what will eventually be required for the BHD suspensions (because of the hybrid iLIGO/aLIGO electronics situation), so I will not burden the readers with the tedious details.

Before installation, I performed a bunch of tests on the aLIGO sat amp. All the measurements were made with the dummy suspension box substituting for an actual suspension. Here are the results.

Attachment #1: Transimpedance amplifier noises.

• Measurement setup: J7 of the Satellite Amp goes to J9 on D1900068 front end (even though the connector is actually labelled "J3" on the box we have - maybe a versioning problem?). The outputs then go to a G=100 SR560 in AC coupled mode (the main purpose here was to block the large DC from the SR785, but I tacked on G=100 while I was at it).  
• Top panel shows the raw measured voltages.
• The bottom panel does a bunch of transformations:
  • Undoes the z:p = 3:30 Hz whitening on board the sat amp.
  • Undoes the G=100 gain of the SR560, and the AC coupling poles/zeros of SR560 and SR785.
  • Converts from voltage to current by dividing by the transimpedance gain, 242 kohms. 
• Some model curves are shown for comparing to the measured spectra. It may be possible that we don't need to modify the nominal z:p = 0.4:10 Hz - I don't think the nominal seismic level will saturate the output even with the 0.4:10 Hz whitening, and it gives us even more clearance to the ADC noise (although we don't need it, we are gain limited at those frequencies, this is mostly a suggestion to reduce the workload).
• The neon green curve is measured with the actual MC1 suspension plugged in, local damping enabled. It doesn't line up with the nosie floor of the bench tests, probably because the cts/um conversion factor could be off by some factor? Around 1 kHz, you can also see some broad peaks that are reminiscent of those seen in the MC_F spectrum after the c1psl Acromag upgrade. I hypothesize this is due to some poor grounding. Hopefully, once we get rid of the single-ended sending/receiving components in the suspension electronics chain, these will no longer be an issue.

Attachment #2: LED drive current source noises. I mainly wanted to check a claim by Rich in a meeting some time ago that the LED intensity fluctuations are dominated by inherent LED RIN, and not by RIN on the drive current. 

• Measurement setup: a pair of pomona mini-grabbers was used to clip onto TP3. I found the voltage noise to be sufficiently high that no preamplification was required, and the DC level was <1V, so I just used the SR785 in AC coupled mode. 
• The dummy suspension box was being driven while the measurement was being made (so the current source is loaded).
• One channel (CH6) shows anomalously high nosie. I confirmed this was present even after the box was plugged in for ~1 day, so can't be due to any thermal / equilibriating transients.
• I didn't check for consistency at the monitor testpoint, but that is exposed even with the MC1 suspension plugged in, so we can readily check. Anyways, from the corresponding photodiode curve in Attachment #1, it would seem that this excess RIN in the drive current has no measurable effect on the intensity fluctuations of the LED (the DC value of the paired PD is consistent with the others, ~6V DC). I must say I am surprised by this conclusion. I also checked for coherence between TP3 and the PD output using the SR785, and found none. 🤔 
• Nevertheless, for the remaining channels, it is clear that the drive current is not shot noise limited for <1kHz. This isn't great. One possible reason is that the collector voltage to Q1 is unregulated (my modeling suggests only ~10dB rejection of collector voltage fluctuations at the output). I believe the current source designed by Luis for A+ makes some of these improvements and so maybe Rich was referring to that design, and not the aLIGO Satellite Amplifier flavor we are using. Anyways, this is just academic I think, the performance is the unit is fine for our purposes.

I will update with the MC1 suspension characterization (loop TFs, step responses etc) later.

Yeah, it's really inconsistent. You had 35mA LED drive and the current noise of the noisy channel was 5e-7 A/rtHz at 1Hz. The RIN is 1.4e-5 /rtHz. The approx. received photocurrent is 30uA as we discussed today and this should make the noise around 4e-10 A/rtHz at 1Hz. However, the readout noise level is better than this level. (well below 1e-10 A/rtHz)

BTW, the IMC seemed continuously locked for 5 hours. Good sign.