This earthquake and friends had tripped all watchdogs. I used the scripted watchdog re-enabler, and released the stuck ITMX (this operation is still requires a human and hasn't been scripted yet). IMC is locked again and all Oplevs report healthy optic alignment.

Summary:

After the earthquake on September 19 2020, it looks to me like the only lasting damage to suspensions in vacuum is the ETMY UR magnet being knocked off. 

Suspension ringdown tests:

I did the usual suspension kicking/ringdown test:

• One difference is that I now kick the suspension "N" times where N is the number of PSD averages desired. 
• After kicking the suspension, it is allowed to ring down with the damping disabled, for ~1100 seconds so that we can get spectra with 1mHz resolution.
• We may want to get more e-folding times in, but since the Qs of the modes are a few hundred, I figured this is long enough.
• I think this kind of approach gives better SNR than letting it ringdown 10,000 seconds (for 10 averages with 10 non overlapping segments of 1000 seconds), and I wanted to test this scheme out, seems to work well.
• Attachment #1 shows a summary of the results.
• Attachment #2 has more plots (e.g. transfer function from UL to all other coils), in case anyone is interested in more forensics. The data files are large but if anyone is interested in the times that the suspension was kicked, you can extract it from here.

Conclusions:

1. My cursory scans of the analysis don't throw up any red flags (apart from the known problem of ETMY UR being dislodged) 👌 . 
2. The PRM data is weird 
   • I believe this is because the DC bias voltage to the coils was significantly off from what it normally is when the PRC is aligned.
   • In any case, I am able to lock the PRC, so I think the PRM magnets are fine.
3. The PRC angular FF no longer works turns out this was just a weird interaction with the Oplev loop because the beam was significantly off-centered on the Oplev QPD. Better alignment fixed it, the FF works as it did before.
   • With the PRC locked and the carrier resonant (no ETMs), the old feedforward filters significantly degrade the angular stability to the point that the lock is lost.
   • My best hypothesis is that the earthquake caused a spot shift on PR2/PR3, which changed the TF from seismometer signal to PRC spot motion.
   • Anyways, we can retrain the filter.
   • The fact that the PRC can be locked suggest PR2/PR3 are still suspended and okay.
4. The SRM data is also questionable, because the DC bias voltage wasn't set to the values for an aligned SRC when the data was collected
   • Nevertheless, the time series shows a clean ringdown, so at least all 5 OSEMs are seeing a signal.
   • Fact that the beam comes out at the AS port suggest SR3/SR2 suspensions are fine 👍 

Attachment #2 also includes info about the matrix diagonalization, and the condition numbers of the resulting matrices are as large as ~30 for some suspensions, but I think this isn't a new feature. 

The MC1 suspension has begun to show evidence of glitches again, from Friday/Saturday. You can look at the suspension Vmon tab a few days ago and see that the excess fuzz in the Vmon was not there before. The extra motion is also clearly evident on the MCREFL spot. I noticed this on Saturday evening as I was trying to recover the IMC locking, but I thought it might be Millikan so I didn't look into it further. Usually this is symptomatic of some Satellite box issues. I am not going to attempt to debug this anymore.

As discussed at the meeting, I decided to effect a satellite box swap for the misbehaving MC1 unit. I looked back at the summary pages Vmon for the SRM channels, and found that in the last month or so, there wasn't any significant evidence of glitchiness. So I decided to effect that swap at ~4pm today. The sequence of steps was:

• SRM and MC1 watchdogs were disabled.
• Unplugged the two satellite boxes from the vacuum flanges.
• For the record: S/N 102 was installed at MC1, and S/N 104 was installed at SRM. Both were de-lidded, supposedly to mitigate the horrible thermal environment a bit. S/N 104 was the one Koji repaired in Aug 2019 (the serial number isn't visible or noted there, but only one box has jumper wires and Koji's photos show the same jumper wires). In June 2020, I found that the repaired box was glitching again, which is when I swapped it for S/N 102.
• After swapping the two units, I re-enabled the local damping on both optics, and was able to re-lock the IMC no issues.

One thing I was reminded of is that the motion of the MC1 optic by controlling the bias sliders is highly cross-coupled in pitch and yaw, it is almost diagonal. If this is true for the fast actuation path too, that's not great. I didn't check it just now.

While I was working on this, I took the opportunity to also check the functionality of the RF path of the IMC WFS. Both WFS heads seem to now respond to angular motion of the IMC mirror - I once again dithered MC2 and looked at the demodulated signals, and see variation at the dither frequency, see Attachment #1. However, the signals seem highly polluted with strong 60 Hz and harmonics, see the zoomed-in time domain trace in Attachment #2. This should be fixed. Also, the WFS loop needs some re-tuning. In the current config, it actually makes the MC2T RIN worse, see Attachment #3 (reference traces are with WFS loop enabled, live traces are with the loop disabled - sorry for the confusing notation, I overwrote the patched version of DTT that I got from Erik that allows the user legend feature, working on getting that back).

I acquired several spare OSEMs (in unknown condition) from Paco. They are stored alongside the shipment from UF.

I wanted to check the functionality of the IMC WFS. I just turned on the WFS servo loops as they were. For the past two hours, they didn't run away. The servo has been left turned on. I don't think there is no reason to keep it turned off.

For whatever reason, the autolocker didn't turn the tickle off for several hours. Seems to work okay now. The linked plot suggests that the coil balancing on MC2 is pretty lousy.

Summary:

1. We will need de-whitening filters for the BHD relay optics in order to meet the displacement noise requirements set out in the DRD. I think these need not be remotely switchable (depends on specifics of LO phase control scheme). SR2, PR2 and PR3 can also have the same config, and probably MC1, MC3 as well.
2. We will need de-whitening filters for the non test mass core IFO optics (PRM, SRM, BS, and probably MC2).
3. I am pretty sure we will not be able to have sufficient DAC range for the latter class of optics if we have to:
   1. Supply the DC bias.
   2. Do the LSC and ASC actuation in the presence of reasonable sensing noise levels.
   3. Engage de-whitening to low-pass-filter the DAC noise at ~200 Hz.

Details:

Attachment #1 shows the DAC noise models for the General Standards 16-bit and 18-bit DACs we are expecting to have.

• The 16-bit model has been validated by me at the 40m a few years ago.
• We have never used the 18-bit flavor at the 40m, and there are all manner of quirks apparently related to zero crossings and such. So the noise may be up to x2 higher (we won't have as much freedom necessarily as the sites to bias the DAC on one side of the zero crossing if we also need to use the same DAC channel to supply the DC bias current for alignment.

Attachment #2 shows the expected actuation range for DC optic alignment, assuming we use the entire DAC range for this purpose.

• Clearly, we need to do other things with the same DAC channels as well, so this is very much an upper bound of what will be possible.
• Let's assume we will not go lower than 100ohms.
• For all new optics we are suspending, we should aim to get the pitch balancing to within 500urad. With a 2x2m=4m optical lever arm, this corresponds to a 2mm spot shift. Should be doable.
• This could turn out to be a serious problem for PRM, BS and SRM if we hope to measure squeezing - the <AUX DOF>-->DARM coupling could be at the level of -40dB, and at 200 Hz, the DAC noise would result in PRCL/MICH/SRCL noise at the level of ~10^-15m/rtHz, which would be 10^-17m/rtHz in DARM. I don't think we can get 20dB of feedforward cancellation at these frequencies. For demonstrating locking using a BHD error signal, maybe this is not a big deal.

Attachment #3 shows the current and proposed (by me, just a rough first pass, not optimized in any way yet) de-whitening filter shapes. These shapes can be tweaked for sure.

• The existing de-whitening filter is way too aggressive. FWIW, the DRD "models" a "4th order Chebyshev low pass filter" which doesn't exist anywhere as far as I know.
• Since the DAC noise is below 1 uV/rtHz at all frequencies of interest, we never need to have >60dB de-whitening anywhere as the input referred noise of any circuit we build will exceed 1 nV/rtHz.
• I propose 3 poles, 3 zeros. In the plot, these poles are located at 30Hz, 50Hz, 2kHz, and the zeros are at 300 Hz, 300 Hz, 800 Hz. 
• The de-whitening is less agressive below 100 Hz, where we still need significant LSC actuation ability. Considering the sensing noise levels at the 40m, I don't know if we can have reasonable LSC and ASC loop shapes and still have the de-whitening.
• Once again, PRM, SRM and BS will be the most challenging.
• For the BHD relay optics, once we have the de-whitening, we won't have the option of turning on a high-frequency (~kHz) dither line because of insufficient DAC range. 

Attachment #4 puts everything into displacement noise units. The electronics noise of the coil driver / de-whitening circuit have not been included so at high frequencies, the projection is better than what will actually be realizable, but still well below the BHD requirement of 3e-17 m/rtHz.

MC1 suspension is glitching again, so this is a good chance to install the new sat box and test it in the field.

As it currently stands the Center of Mass of the Adapter Ring/Optic assembly is 0.0175" out of the plane formed by the suspension wire. See Attachments. The side plate, along with the EQ stops are hidden to show the CoM and the plane.

Note: The changes discussed in the meeting with Calum have not been added and are a work in progress. These changes include:

- Adding a 45 deg chamfer to the both parallel faces of the adapter ring. This along with a modified bracket for the EQ stops will allow for easier adjustment of the screws. 

- Potentially changing material of adapter ring to stainless stell to more accurately emulate the mass of a 3" optic.

- Different adjustment mechanism of the "dumbell" at bottom of adapter ring to something similar to the VOPO suspension (will need to consult Calum further)

[jordan, gautam]

• Ran 60ft long cables from 1X4 to MC1/MC3 chamber flange, via overhead cable tray, and top of PSL enclosure for the last ~20ft. Note that it may be that the overhead cable trays cannot support the weight of the cables for 15 SOSs (total 30 shielded cables with 37 wires as twisted pairs) when we eventually add the optics for the BHD upgrade.
• Installed aLIGO satellite amplifer in 1X4.
• Tapped +/-20 V (which is the available voltage closest to the required +/-18V). For this, the Sorensens were powered down, and the actual taps were made from the fusable blocks powering the Trillium interface box. We made sure to leave an extra slot so that this kind of additional headache is not required for the next person doing such work.
• Once installed, I plugged in the dummy suspension box and verified that the unit performs as expected. 
• Some photos of the installation are here.

After this work, the IMC locked fine, the AS camera has the Michelson fringing, the fast CDS indicators are all green, and the seismometer BLRMS all look good - therefore, I claim no lasting damage was done as a direct result of today's work at 1X4. I will connect up the actual suspension at my leisure later today. Note that the MC1 glitches seem to have gone away, without me doing anything about it. Nevertheless, I think it's about time that we start testing the new hardware. 

Unrelated to this work: while I was testing some characteristics of the MC1 suspension (before we did any work in the VEA, you can see the timestamp in the ndscope), I noticed that the MC1 UL coil channel cannot actually be used to actuate on the optic. The coil driver Vmon channel demonstrates the appropriate response, which means that the problem is either with the Satellite box (it is just a feedthrough, so PCB trace damaged?) or with the OSEM itself (more likely IMO, will know more once I connect the new Satellite Amplifier up). I only show comparison for UL vs UR, but I checked that the other coils seem to be able to actuate the optic. This means we have been running for an indeterminate amount of time with only 3 face actuators on MC1, probably related to me having to do this work. 

Also unrelated to this work - while poking around at 1X5 rear, I noticed that the power connections to the existing Satellite Boxes are (understatedly) flaky, see connections to T1-T4 in Attachment #2..

Note from today's meeting:

1. Can we adjust the thickness of the cylindrical hole for the mirror to move the COM in the plane of the wires. (We should be able to do that)

and

2. Please check how much we can displace the COM by the bottom dumbbell.

There is some non-trivial sign flipping in the sensors/coils in this new setup because it is a hybrid one with the old interfacing electronics (D000210, D010001) and the new Satellite Amplifier (D080276). So I haven't yet gotten the damping working. I am leaving the PSL shutter closed and will keep working on this today/tomorrow. I have made various changes to the c1mcs realtime model and the c1susaux database record where MC1 is concerned. I have backups of the old ones so we can always go back to that if we so desire.

In the meantime, the PSL shutter is closed and there is no light to the IFO.

Update 1700: I've implemented some basic damping and now the IMC is now locked. The WFS loop runs away when I enable it, probably some kind of weird interaction with the (as of now untuned) MC1 local damping loops. I will write up a more detailed report later.

Update 2300: Did the following:

1. Re-calibrated the cts2um filter in all SENSOR filter banks to account for the increased transimpedance and LED drive current. I judged the overall scaling to be x0.25 but this can be calibrated against the bounce peak height for example (it lines up pretty well).
2. Re-measured the input matrix - it was very different from what was loaded. I am measuring this again overnight for some consistency.
3. Re-tuned the damping gains. Now the optic damps well, and the loops seem file to me, both via broadband noise injection TF and by step response metrics.
4. Yet, the WFS servo cannot be enabled. The WFS signal is summed in before the output matrix so I don't know why this would have a different behavior compared to the local damping, or indeed, why this has to be changed. Will need some (WFS) sensing/actuation matrix measurements to know more.

Dropping this for tonight, I'll continue tomorrow. Meanwhile, the OSEM input matrix measurement is being repeated overnight. PSL shutter is closed.

I am setting up a testing rig for the OSEMs we recently obtained. I found the schematic for the OSEM assembly from which the pin assignment can be read.

I connected the OSEM's pin plate to a female DB15 on a breakout board. I find the pin assignment (attachment 1, sorry for the image quality) to be:

There are several things that need to be done for each OSEM.

1. Measuring inductance of the coils. I checked that the measurement wires don't add any measurable inductance.

2. Check that the PDs and LEDs are alive (e.g. check forward voltage drop with fluke)

3. Energize the LED and PD.

4. Check PD DC level. For this, I might need the satellite box amplifier.

5. Check LED spot position on the PD.

6. Re-engrave OSEM S/N if needed.

I still need to figure a sensible scheme for points 3-5.

The WFS servo was recommissioned. The matrix can be tuned a bit more, but for now, I've recovered the old performance and the alignment doesn't seem to be running away, so I defer further tuning for later. The old Satellite box was handed over to Yehonathan for his characterization of the "spare" OSEMs.

This finishes the recovery of the MC1 suspension, I am now satisfied that the local damping loops are performing satisfactorily, that the WFS servo is also stable, and that POX/POY locking is recovered. On MC1, we even have 4 actuatable face OSEMs and the PIT(YAW) bias adjust slider even moves the optic in PIT(YAW), what a luxury. 

I've SDFed all the changes, and have backup of the old realtime model and C1SUSAUX_MC1 database files if we want to go back for whatever reason. The changes required to make this suspension work are different from what will eventually be required for the BHD suspensions (because of the hybrid iLIGO/aLIGO electronics situation), so I will not burden the readers with the tedious details.

Before installation, I performed a bunch of tests on the aLIGO sat amp. All the measurements were made with the dummy suspension box substituting for an actual suspension. Here are the results.

Attachment #1: Transimpedance amplifier noises.

• Measurement setup: J7 of the Satellite Amp goes to J9 on D1900068 front end (even though the connector is actually labelled "J3" on the box we have - maybe a versioning problem?). The outputs then go to a G=100 SR560 in AC coupled mode (the main purpose here was to block the large DC from the SR785, but I tacked on G=100 while I was at it).  
• Top panel shows the raw measured voltages.
• The bottom panel does a bunch of transformations:
  • Undoes the z:p = 3:30 Hz whitening on board the sat amp.
  • Undoes the G=100 gain of the SR560, and the AC coupling poles/zeros of SR560 and SR785.
  • Converts from voltage to current by dividing by the transimpedance gain, 242 kohms. 
• Some model curves are shown for comparing to the measured spectra. It may be possible that we don't need to modify the nominal z:p = 0.4:10 Hz - I don't think the nominal seismic level will saturate the output even with the 0.4:10 Hz whitening, and it gives us even more clearance to the ADC noise (although we don't need it, we are gain limited at those frequencies, this is mostly a suggestion to reduce the workload).
• The neon green curve is measured with the actual MC1 suspension plugged in, local damping enabled. It doesn't line up with the nosie floor of the bench tests, probably because the cts/um conversion factor could be off by some factor? Around 1 kHz, you can also see some broad peaks that are reminiscent of those seen in the MC_F spectrum after the c1psl Acromag upgrade. I hypothesize this is due to some poor grounding. Hopefully, once we get rid of the single-ended sending/receiving components in the suspension electronics chain, these will no longer be an issue.

Attachment #2: LED drive current source noises. I mainly wanted to check a claim by Rich in a meeting some time ago that the LED intensity fluctuations are dominated by inherent LED RIN, and not by RIN on the drive current. 

• Measurement setup: a pair of pomona mini-grabbers was used to clip onto TP3. I found the voltage noise to be sufficiently high that no preamplification was required, and the DC level was <1V, so I just used the SR785 in AC coupled mode. 
• The dummy suspension box was being driven while the measurement was being made (so the current source is loaded).
• One channel (CH6) shows anomalously high nosie. I confirmed this was present even after the box was plugged in for ~1 day, so can't be due to any thermal / equilibriating transients.
• I didn't check for consistency at the monitor testpoint, but that is exposed even with the MC1 suspension plugged in, so we can readily check. Anyways, from the corresponding photodiode curve in Attachment #1, it would seem that this excess RIN in the drive current has no measurable effect on the intensity fluctuations of the LED (the DC value of the paired PD is consistent with the others, ~6V DC). I must say I am surprised by this conclusion. I also checked for coherence between TP3 and the PD output using the SR785, and found none. 🤔 
• Nevertheless, for the remaining channels, it is clear that the drive current is not shot noise limited for <1kHz. This isn't great. One possible reason is that the collector voltage to Q1 is unregulated (my modeling suggests only ~10dB rejection of collector voltage fluctuations at the output). I believe the current source designed by Luis for A+ makes some of these improvements and so maybe Rich was referring to that design, and not the aLIGO Satellite Amplifier flavor we are using. Anyways, this is just academic I think, the performance is the unit is fine for our purposes.

I will update with the MC1 suspension characterization (loop TFs, step responses etc) later.

Yeah, it's really inconsistent. You had 35mA LED drive and the current noise of the noisy channel was 5e-7 A/rtHz at 1Hz. The RIN is 1.4e-5 /rtHz. The approx. received photocurrent is 30uA as we discussed today and this should make the noise around 4e-10 A/rtHz at 1Hz. However, the readout noise level is better than this level. (well below 1e-10 A/rtHz)

BTW, the IMC seemed continuously locked for 5 hours. Good sign.

Adjusting the thickness of the cylindrical hole for the mirror on the 2" optic sleeve, from .6875" to .61" thick, moves the CoM to 0.0003" out of plane from the suspension wire. This is with the dumbell at its neutral point.

How close to zero do we need this to be? More fine tuning of that thickness can get it to zero, but this would require much tighter machining tolerance on that hole depth.

Moving the dumbell towards the back of the SOS assembly (noted as negative direction, with origin at the plane formed by the wires), moves the CoM to -0.002" from the plane.

Moving the dumbell towards the front of the SoS assmebly (positive direction wrt the plane formed by the suspension wire), moves the CoM to +0.0022" from the plane.

So the total adjustment range with the dumbell is -0.002"to 0.0022", with the plane formed by the wires as the origin.

See Attachments

We want to move the CoM with the adjustment range so that the residual deviation is adjusted by the bottom dumbbell. 0.0003" is well within the range and good enough.

I briefly talked with Jordan about this. This suspension will have OSEMs right? With 400ohm series resistance for the coil drivers, we will have ~+/-20mrad actuation range. Of course we'd like to use as much of this for interferometry and not static pitch alignment correction (possibly even increase the series resistance to relax the dewhitening requirements). But what is the target adjustability range in mrad with the dumbell/screw config? My target in the linked elog is 500urad (not any systematic optimum, but will allow us to use most of the DAC range for interferometry). Are these numbers in inches commensurate with this 500urad?

On a related note - are there grooves for the wires to sit in on the side of the sleeve? We looked at the solidworks drawing, and noticed that the groove doesn't extend all the way to the top of the clamp. Also, the material of both the clamping piece and the piece onto which the wire is pressed onto is SS. Don't we want them to be Aluminium (or something softer than the wire) so that the wire makes a groove when the clamp is tightened?

Jordan's screenshot actually shows that the vertical distance (Y) is 0.0000". We want to have the vertical distance of CoM from the wire clamping point to be 0.9mm in the nominal SOS design (this might need to be adjusted to have a similar pitch resonant freq for the different inertia of moment). Let's say it is ~mm ish.

The full range of the bottom dumbbell adjustment gives us the CoM adjustment range of +/-0.002” = +/-50um. This corresponds to an alignment range of +/-50mrad. And we want to set it within +/-500urad.
So we need to adjust the dumbbell position with the precision of 1/100 of the full range (precision of 0.5um).

The groove does not extend to the top of the clamp. The groove shallower than the wire diameter cause the hysteresis of the alignment. Also, the material of the pieces should be stainless steel. Al clamp is softer than the wire and will cause the groove to be dug on the material, causing increased bending friction and hysteresis again.

Saying, all of our suspended masses with Al stand-offs are suffering this issue to some extent. That was the reason to buy the ruby standoffs.

Continuing with the OSEM testing. I measure the resistance of the wires from the RLC meter to the coil to be ~ 0.9ohm. I will subtract this number from the subsequent coil resistance measurements.

I took the old MC1 satellite box for powering the PD and LED in the OSEM assemblies. I connected an idc breakout board to J4 and powered the box with a DC supply according to the box's schematics.

After getting a bit confused about some voltage reading from one of the PD readouts Gautam came and basically redid the whole rig. Instead of using breakout boards, he powered the amplifier circuit directly from the DC supply. Then, to connect the OSEM pinboard directly to the J1 connector he made a DB25 ribbon cable where the two connectors are opposite to one another to mimic the situation with the vacuum feedthru. He also connected a DB25 to BNCs breakout cable, specific to the satellite box, to the J3 port to read the individual PDs through a BNC connector. We managed to confirm the normal operation of one OSEM (Normal PD voltage and LED light spot hitting the PD using a camera with no IR filter).

It was getting a bit late. Going to start checking the OSEMs tomorrow.

I can't obtain a consistent view between the existing drawings/photographs and your pin assignment. Please review the pin assignment again to check if yours is correct.

Looking from the back side and the wires are going down, the left bottom pin is "Coil Start" and the upper right adjacent pin is "Coil End". (See attachment)
So in your picture 1 should be the coil start and 4 should be the coil end, but they are not according to your table.

This is my current understanding of the in-vacuum wiring:
1. Facts

• We have the in-air cable pinout. And Gautam recently made a prototype of D2100014 custom cable, and it worked as expected.
• The vacuum feedthrough is a wall with the male pins on the both sides. This mirrors pinout.
• On the in-vacuum cable stand (bracket), the cable has a female connector.

2. From the above facts, the in-vacuum cable is

• DSUB25 female-female cable
• There is no pinout mirroring

Accuglass has the DSUB25 F-F cable off-the-shelf. However, this cable mirrors the pinout (see the datasheet on the pdf in the following link)
https://www.accuglassproducts.com/connector-connector-extension-cable-25-way-female

3. The options are
- ask Accuglass to make a twisted version so that the pinout is not mirrored.
or
- combine Accuglass female-male cable (https://www.accuglassproducts.com/connector-connector-extension-cable-25-way-femalemale) and a gender changer (https://www.accuglassproducts.com/gender-adapter-25d)

4. The length will be routed from the feedthrough to the table via the stacks like a snake to be soft. So, it will require some extra length.

5. Also, the Accuglass cables don't have a flap and holes to fix the connector to a cable post (tower). If we use a conventional 40m-style DSUB25 post (D010194), it will be compatible with their cables. But this will not let us use a DSUB25 male connector to mate. In the future, the suspension will be upgraded and we will need an updated cable post that somehow holds the connectors without fastening the screws...

Yes, my phone camera mirrored the image. Sorry for the confusion.

I see you already uploaded the correct pin assignment.

Continuing with the new rig, I measure the resistance of the cable leading to the coil to be 0.08+(0.52-0.08)+(0.48-0.08)=0.9ohm.

Total: 63 OSEMS. Centered working OSEMS: 42. Will upload a more detailed summary to the wiki soon.

Note: The Olympus camera is eating the AA camera very quickly (need to replace every 1.5 days). I'm guessing this is because of the corrosion in the battery housing.

I finished testing the OSEMs. I put all the OSEMs back in the box. The OSEMS were divided into several bags. I put the OSEM box next to the south flow bench on the floor.

I have uploaded the OSEM catalog to the wiki. I will upload the LED spot images later.

In summary:

Total 64 OSEMS, 31 long, 33 short.

Perfectly centered LED spots, ready for C&B OSEMS: 30, 12 long, 18 short.

Perfectly centered LED spots, need some work (missing pigtails, weird screws) OSEMS: 7, 5 long, 2 short.

Slightly off-centered (subjective) LED spots, ready for C&B OSEMS: 20, 7 long, 13 short.

Slightly off-centered (subjective) LED spots, need some work (missing pigtails, weird screws) OSEMS: 4 long

Defective OSEMS or LED spot way off-center: 3.

I realized I hadn't checked the PRM actuator as thoroughly as I had the others. I used the Oplev as a sensor to check the coil balancing, and I noticed that while all 4 coils show up with the expected 1/f^2 profile at the Oplev error point, the actuator gains seem imbalanced by a factor of ~5. The phase isn't flat because of some filters in the Oplev electronics I guess. The Oplev loops were disabled for the measurement, and the excitations were small enough that the beam stayed reasonably well centered on the QPD throughout. This seems very large to me - the values in the coil output filter gains lead me to expect more like a ~10% mismatch in the actuation strenghts, and similar tests on other optics in the past, e.g. ETMY, have yielded much more balanced results. I'm collecting some free-swinging PRM data now as an additional check. I verified that all the coils seem actuatable at least, by applying a 500 ct step at the offset of the coil output FM, and saw that the optic moved (it was such a test that revealed that MC1 had a busted actuator some time ago). If the eigenmode spectra look as expected, I think we can rule out broken magnets, but I suppose the magnets could still be not well matched in strength?

Herewith, I describe an adventure

1. Balance the OSEM input matrix using the free swinging data (see prev elogs).
2. Balance the coil actuation by changing the digital coil gains. This should be done above 10 Hz using optical levers, or some IMC readout (like the WFS). At the end of this process, you should put a pringle vector into the column of the SUS output matrix that corresponds to one of the SUS OSC/LOCKIN screens. Verily, the pringle excitation should produce no signal in MC_F or da WFS.
3. use the Malik doc on the single suspension to design feed-forward filters for the SUS COIL filter banks. You can get the physical parameters using the design documents on DCC / 40m wiki and then modify them a bit based on the eigenfrequencies in the free swinging data.
4. Model the 2x2 system which includes longitudinal and pitch motion. Consider how accurate the filters must be to maintain a cross-coupling of < 3% in the 0.5-2 Hz band.
5. Is this decoupling forsooth still maintained when you close the SUS damping loops in the model? If not, why so?
6. Make step response measurements of the damping loops and record/plot data. Use physical units of um/urad for the y-axes. How much is the step response cross-coupling?
7. Consider the IMC noise budget: are the low pass filters in the damping loops low-passing enough? How much damping is demasiado (considering the CMRR of the concrete slab for seismic waves)?
8. Can we use Radhika's AAA representation to auto-tune the FF and damping filters? It would be very slick to be able to do this with one button click.

gautam: For those like me who don't know what the AAA representation is: the original algorithm is here, and Lee claims his implementation of it in IIRrational is better, see his slides.

I finished ranking the OSEMS on the OSEM wiki page.

I also moved the OSEM data folder from /home/export/home to /users/public_html and created a soft link instead. I have done the same for the 40m_TIS folder that I uploaded there a while ago.

The PRM got tripped ~5AM this morning. The cause is unclear - the seismometer reports elevated activity ~10 minutes before the ringdown starts (as judged using the OSEMs). But the other optics didn't seem to receive as much of an impulse (I only show the BS sensors here as it sits on the same stack as the PRM). Anyway it certainly wasn't me trying to make life difficult for the morning team.

I was able to restore the damping with reEnableWatchdogs.py. I am now running some suspension tests on the PRM by letting it swing freely so please let that finish. I plan to attempt some locking this evening.

How were the statistics of them? i.e. # of Good OSEMs, # of OK OSEMs, etc...

I just saw the PRM watchdog tripped at ~15:20 local (23:20UTC). I restored the PRM but I saw only the side watchdog tripped.
Again at 15:27

17:55 I found the PRM was oscillating while the watchdogs were not tripped. I turned off the OPLEV servos and this made the PRM calmed down. But I didn't turn on the OPLEVs for the past two trips. How were the OPLEVs turned on???

Ah, I'm sorry, I missed the line that Gautam was running the free-swinging test on the PRM.
The two kicks starting from 23:08:50 and from 23:26:31 were spoiled. Did it make the measurement completely waisted?
 

29 Good OSEMs, of which 1 is questionable (089) with PD voltage of 1.5V, 5 need some work (pigtailing, replace/remove/add screws). We have 4 pigtails. Schematics.

20 OK OSEMs (Slightly off-centered LED spot), of which 3 need some work (pigtailing, replace/remove/add screws).

13 Bad OSEMS (Way off-centered LED spot)

2 Defunct OSEMs

-------

Ed: KA
Good: 23 complete OSEMs +  5 good ones, which need soldering work (there are 4 pigtails and take one from a defunct OSEM).
OK:  Use good 7 OSEMs for the sides. And keep some functional OSEMs as spares.

The procedure is that the optic is kicked to excite it, and allowed to ring down for ~1ksec, with damping turned off. The procedure is repeated 15 times for some averaging. 

Attachment #1 - sensor spectra from yesterday.

Attachment #2 - peaks using the naive diagonalization matrix from yesterday.

Attachment #3 - Data from ~1 year ago. 

The y-axis in all plots is labelled as "cts/rtHz" but these are the DQed channels, which come after a "cts2um" CDS filter - so if that filter is accurate, them the y-axes may be read as um/rtHz.

I wonder if the September 2020 earthquake somehow damaged the PRM suspension, as this experiment would suggest that the problem is not only with the actuation. The data was gathered with the neutral position of the PRM (between kicks) being well aligned for PRMI, and the DC values of all the shadow sensors in this position is close to half-light (~1V, except for side which was more like 4V). Hard to say what exactly is happening since only the PIT DoF has the weird asymmetric peak shape instead of the expected Lorentzian - I would have thought that a damaged wire or broken magnet would affect all 4 DoFs but the F.C. spring experience on ETMY showed that anything is possible.

As I am sitting in the control room, the PRM suspension watchdog tripped again. This time, there is clearly no seismic activity. Yet, the BS suspension also shows a slight disturbance at the same time as the PRM. ITMY shows no perturbation though. My best hypothesis here is that the problem is electrical. In Attachment #1, you can see that all of the Sensors go to -6000 cts (whut?) for ~30 seconds. Zooming in to that segment in Attachment #2, it would appear that the light detected by the LED changed dramatically (went dark?) on all 5 coils. The 4 face coils have the same time constant but the side has a different one, but in any case, this level of light change in half a second is clearly not physical. Then the watchdog trips because this huge apparent motion elicits a kick from the damping loops.

The plots I attach are for the DQed sensor channels, so there is some digital filtering involved. But I confirmed that the signal doesn't go negative if I disable the input to the filter module. So it would seem that the voltage input to the ADC really chanegd polarity, seems unphysical. Could be Satellite Box or whitening electronics I suppose - I think we can exclude bad cabling, as that would just lead to the signals going to 0, whereas it would appear here that they did really change sign (confirmed by looking at the ULPDmon channel, which is digitized by Acromag, which reports -10 V at the time of glitch). But why should the BS care about the PRM electronics going wonky?

In addition to an exorcist, we need functioning electronics!

This optic has been hampering my locking attempts all evening. I switched the PRM and SRM satellite boxes, but then I remembered PRM has the Al foil "hats" to attenuate scattered light. of course the Al foil is conducting and can short the OSEM leads. I put some kapton pieces in between OSEM and foil to try and mitigate this issue but I suppose over time it could have slipped, and is making some intermittent contact, shorting PD anode and cathode (that would explain the PD reporting -10 V instead of some physical value).

If this is the problem we would need a vent to address it. In the daytime I'll measure L and R of the coils to see if the actuator imbalance I reported is also due to the same problem...

[Paco, Anchal]

The triggered code went on at 5:00 am today but a last minute change I made yesterday to increase number of repititions had an error and caused the script to exit putting everything back to normal. So as we came in the morning, we found the mode cleaner locked continuously after one free swing attempt at 5:00 am. I've fixed the script and ran it for 2 hours starting at 8;10 am. Our plan is to get some data atleast to play with when we are here. If the duration is not long enough, we'll try to run this again tomorrow morning. The new script is running on same tmux session 'MCFreeSwingTest' on Rossa

10:13 the script finished and IMC recovered lock.

Thu Mar 11 10:58:27 2021

The test ran succefully with the mode cleaner optics coming back to normal in the end of it. We wrote some scripts to read data and analyze it. More will come in future posts. No other changes were made today to the systems.

Summary

Per Gautam's request, I've checked the coil resistances and inductances.

• PRM/BS/SRM coils were tested.
• All the PRM coils look well-matched in terms of the inductance. Also, I didn't find a significant difference from BS coils.
• Pin 1 of the feedthru connectors is shorted to the vacuum chamber.
• A discovery was that: The SRM DSUB pinouts are mirrored compared to the other suspensions.

Method

A DSUB25 breakout was directly connected to the flange (Attachment 1).
The impedance meter was nulled every time the measurement range and type (R or L) were changed.

Result

Observation

The SRM pinouts seem mirrored compared to the others. In fact, these two connectors are equipped with mirror cables (although they are unshielded ribbons) (Attachment 2).

The SRM sus is located on the ITMY table. There is a long in vacuum DSUB25 cable between the ITMY and BS tables. I suspect that the cable mirrors the pinout and this needs to be corrected by the in-air mirror cables.

I went around the lab and did not find any other suspensions which have the mirror cable.

WIth the BHD configuration, we will move the feedthru for the SRM to the one on the ITMY chamber. So I believe the situation is going to be improved.

For consistency, today, I measured both the BS and PRM actuator balancing using the same technique and don't find as serious an imbalance for the BS as in the PRM case. The Oplev laser source is common for both BS and PRM, but the QPDs are of course distinct.

BTW, I thought the expected resistance of the coil windings of the OSEM is ~13 ohms, while the BS/PRM OSEMs report ~1-2 ohms. Is this okay?

ugh. sounds bad - maybe a short. I suggest measuring the inductance; thats usually a clearer measurement of coil health

I didn't repeat Koji's measurement, but he reports the expected ~3.2mH per coil on all the BS and PRM coils.

After jumping through few hoops, we have one successful result in diagonalizing the input matrix for MC1, MC2 and MC3.

Code:

• Attachment 2 has the code file contained. For now, we can only guarantee it to work on Donatella in conda base environment. Our code is present in scripts/SUS/InMatCalc
• We mostly follow the steps mentioned in 4886 and the matlab codes in scripts/SUS/peakFit.
• Data is first multiplied with currently used inpur matrix to get time series data in DOF (POS, PIT, YAW, SIDE) basis.
• Then, the peak frequencies of each resonance are identified.
• For getting these results, we did not attempt to fit the peaks with lorentzians and took the maxima point of the PSD to get the peak positions. This is only possible if the current input matrix is good enough. We have to adjust some parameters so that our fitting code works always.
• TF estimate between the sensor data w.r.t UL sensor is taken and the values around the peak frequencies of oscillations are averaged to get the sensing matrix.
• This matrix is normalized along DOF axis (columns in our case) and then inverted.
• After inversion, another normaliation is done along DOF axis (now rows).
• Finally we plot the comparison of ASD in DOF basis when using current input matrix and when using our calculated inpur matrix (diagonalizing).
• You can notice in Attachment 1 that after the diagonalization, each DOF shows resonance at only one and its own resonance frequency while earlier there was some mixing shown.
• Absolute value of the calculated DOF might have changed and we need to calibrate them or apply appropriate gain factors in the DOF basis filter chains.

Next steps:

• We'll complete our scripts and make them more general to be used for any optic.
• We'll combine all of them into one single script which can be called by medm.
• In parallel, we'll start onwards from step 2 in 15881.
• Anything else that folks can suggest on our first result. Did we actually do it or are we fooling ourselves?

I have added a .1", 45deg chamfer to the bottom of the adapter ring. This was added for a new placement of the eq stops, since the barrel screws are hard to access/adjust.

This also required a modification to the eq stop bracket, D960008-v2, with 1/4-20 screws angled at 45 deg to line up with the chamfer.

The issue I am running into is there needs to be a screw on the backside of the ring as well, otherwise the ring would fall backwards into the OSEMs in the event of an earthquake. The only two points of contact are these front two angled screws, a third is needed on the opposite side of the CoM for stability. This would require another bracket mounted at the back of the SOS tower, but there is very little open real estate because of the OSEMs.

Instead of this whole chamfer route, is it possible/easier to just swap the screws for the barrel eq stops? Instead of a socket head cap screw, a SS thumb screw such as this, will provide more torque when turning, and remove the need to use a hex wrench to turn.

[Paco, Anchal]

Paco accidentally clicked on C1:SUS-MC1_UL_TO_COIL_SW_1_1 (MC1 POS to UL Coil Switch) and clicked it back on. We didn't see any loss of lock or anything significant on the large monitor on left.

Testing the new calculated input matrix

• Switched off the PSL shutter (C1:PSL-PSL_ShutterRqst)
• Switched off IMC autolocker (C1:IOO-MC_LOCK_ENABLE)
• Uploaded the same input matrix as the current one to check writing function in scripts/SUS/InMatCalc/testingInMat.py . We have created backup text file for current settings in backupMC1InMat.txt .
• Uploaded the new input matrix in normalized form. To normalize, we first made each row vector unit vector and then multiplied by the norm of current input matrix's row vectors (see scripts/SUS/InMatCalc/normalizeNewInputMat.py)
• Switched ON the PSL shutter (C1:PSL-PSL_ShutterRqst)
• Switched ON IMC autolocker (C1:IOO-MC_LOCK_ENABLE)
• Locked was caught immediately. The wavefront sensor of MC1 shows usual movement, nothing crazy.
• So the new input matrix is digestable by the system, but what's the efficacy of it?

< Two inspection people taking pictures of ceiling and portable AC unit passed. They rang the doorbell but someone else let them in. They walked out the back door.>

Testing how good the input matrix for MC1 is:

• We loaded the input matrix butterfly row in C1:SUS-MC1_LOCKIN_INMATRX_1_4 to 8. This matrix is multiplied by C1:SUS-MC1_UL_SEN_IN and so on channels before the calibration to um and application of toher filters.
• We tried to look around on how to load the same filter banks on the signal chain of LOCKIN1 of MC1 but couldn't, so we just manually added gain value of 0.09 in this chain to simulate the calibration factor at the very least.
• We started the oscillator on LOCKIN 1 on MC1 with amplitude 1 and frequency 6 Hz.
• We added butterfly mode actuation output column (UL:1, UR:-1, LL:-1, LR:1), nothing happened to the lock of probably because of low amplitude we put in.
• Now, we plot the ASD of channels like C1:SUS-MC1_SUSPOS_IN1 (for POS, PIT, YAW, SIDE) to see if we see a corresponding peak there. No we don't. See attachment 1.

Restoring the system:

• Added 0 to the LOCKIN1 column in MC1 output matrix.
• Made LOCK1 oscillator 0 Amplitude, 0 Hz.
• Changed back gain on signal chain of LOCKIN1 on MC1.
• Added 0 to C1:SUS-MC1_LOCKIN_INMATRX_1_4 to 8.
• Switched off the PSL shutter (C1:PSL-PSL_ShutterRqst)
• Switched off IMC autolocker (C1:IOO-MC_LOCK_ENABLE)
• Wrote back the old matrix by scripts/SUS/InMatCalc/testingInMat3.py which used the backup we created.
• Switched ON the PSL shutter (C1:PSL-PSL_ShutterRqst)
• Switched ON IMC autolocker (C1:IOO-MC_LOCK_ENABLE)

Here, I present the same input matrices now normalized row by row to have same norm as current matrices rows. These now I plotted better than last time. Other comments same as 15902. Please let us know what you think.

Thu Mar 18 09:11:10 2021 :

Note: The comparison of butterfly dof in the two cases is bit bogus. The reason is that we know what the butterfly vector is in sensing matrix (N_osems x (N_dof +1)) and that is the last column being (1, -1, 1, -1, 0) corresponding to (UL, UR, LR, LL, Side). However, the matrix we multiply with the OSEM data is the inverse of this matrix (which becomes the input matrix) which has dimensions ((N_dof + 1) x N_osem) and has the last row corresponding to the butterfly dof. This row was not stored for old calculation of the input matrix (which is currently in use) and can not be recovered (mathematically not possible) with the existing 5x4 part of that input matrix. We just added (1, -1, 1, -1, 0) row in the bottom of this matrix (as was done in the matlab codes) but that is wrong and hence the butterfly vector looks so bad for the existing input matrix.

Proposal: We should store the last row of generated input matrix somewhere for such calculations. Ideally, another row in the epics channels for the input matrix would be the best place to store them but I guess that would be too destructive to implement. Other options are to store this 5 number information in wiki or just elogs. For this post, the buttefly row for the generated input matrix is present in the attached files (for future references).

[Paco, Anchal]

Since the new generated matrices were created for the measurement made last time, they are of course going to work well for it. We need to test with new independent data to see if it works in general.

• We have run scripts/SUS/InMatCal/freeSwingMC.py for 1 repition and free swinging duration of 1050s on tmux session FreeSwingMC on Rossa. Started at GPS: 1300118787.
• Thu Mar 18 09:24:57 2021 : The script ended successfully. IMC is locked back again. Killing the tmux session.
• Attached are the results of 1-kick test, time series data and the ASD of DOFs for calculated using existing input matrix and our calculated input matrix.
• The existing one was already pretty good except for maybe the side DOF which was improved on our diagonalization.

[Paco]

After Anchal left for his test, I took the time to set up the iMAC station so that Stephen (and others) can remote desktop into it to use Omnigraffle. For this, I enabled the remote login and remote management settings under "Sharing" in "System Settings". These two should allow authenticated ssh-ing and remote-desktopping respectively. The password is the same that's currently stored in the secrets.

Quickly tested using my laptop (OS:linux, RDP client = remmina + VNC protocol) and it worked. Hopefully Stephen can get it to work too.

Good Enough! Let's move on with output matrix tuning. I will talk to you guys about it privately so that the whole doesn't learn our secret, and highly sought after, actuation balancing.

I suspect that changing the DC alignment of the SUS changes the required input/output matrix (since changes in the magnet position w.r.t. the OSEM head change the sensing cross-coupling and the actuation gain), so we want to make sure wo do all this with the mirror at the correct alignment.

[Paco, Anchal]

• We decided to try out the coil actuation balancing after seeing some posts from Gautum about the same on PRM and ETMY.
• We used diaggui to send swept sine excitation signal to C1:SUS-MC3_ULCOIL_EXC and read it back at C1:SUS-MC3_ASCPIT_IN1. Idea was to create transfer function measurements similar to 15880.
• We first tried taking the transfer function with excitation amplitude 0f 1, 10, 50, 200 with damping loops on (swept from 10 to 100 Hz lograthmically in 20 points).
• We found no meaningful measurement and looked like we were just measuring noise.
• We concluded that it is probably because our damping loops are damping all the excitation down.
• So we decided to switch off damping and retry.
• We switched off: C1:SUS-MC3_SUSPOS_SW2 , C1:SUS-MC3_SUSPIT_SW2, C1:SUS-MC3_ASCPIT_SW2, C1:SUS-MC3_ASCYAW_SW2, C1:SUS-MC3_SUSYAW_SW2, and C1:SUS-MC3_SUSSIDE_SW2.
• We repeated teh above measurements going up in amplitudes of excitation as 1, 10, 20. We saw the oscillation going out of UL_COIL but the swept sine couldn't measure any meaningful transfer function to C1:SUS-MC3_ASCPIT_IN1. So we decided to just stop. We are probably doing something wrong.

Trying to go back to same state:

• We switch on: C1:SUS-MC3_SUSPOS_SW2 , C1:SUS-MC3_SUSPIT_SW2, C1:SUS-MC3_ASCPIT_SW2, C1:SUS-MC3_ASCYAW_SW2, C1:SUS-MC3_SUSYAW_SW2, and C1:SUS-MC3_SUSSIDE_SW2.
• But C1:SUS-MC3_ASCYAW_INMON had accumulated about 600 offset and was distrupting the alignment. We switched off C1:SUS-MC3_ASCYAW_SW2 hoping the offset will go away once the optic is just damped with OSEM sensors, but it didn't.
• Even after minutes, the offset in C1:SUS-MC3_ASCYAW_INMON kept on increasing and crossed beyond 2000 counts limit set in C1:IOO-MC3_YAW filter bank.
• We tried to unlock the IMC and lock it back again but the offset still persisted.
• We tried to add bias in YAW DOF by increasing C1:SUS-MC3_YAW_OFFSET, and while it was able to somewhat reduce the WFS C1:SUS-MC3_ASCYAW_INMON offset  but it was misalgning the optic and the lock was lost. So we retracted the bias to 0 and made it zero.
• We tried to track back where the offset is coming from. In C1IOO_WFS_MASTER.adl, we opened the WFS2_YAW filter bank to see if the sensor is indeed reading the increasing offset.
• It is quite weird that C1:IOO-WFS2_YAW_INMON is just oscillating but the output in this WFS2_YAW filter bank is slowly increasing offset.
• We tried to zero the gain and back to 0.1 to see if some holding function is causing it, but that was not the case. The output went back to high negative offset and kept increasing.
• We don't know what else to do. Only this one WFS YAW output is increasing, everything else is at normal level with no increasing offset or peculiar behavior.
• We are leaving C1:SUS-MC3_ASCYAW_SW2 off as it is disrupting the IMC lock.

[Jon walked in, asked him for help]

• Jon suggested to do burt restore on IOO channels.
• We used (selected through burtgooey):
burtwb -f /opt/rtcds/caltech/c1/burt/autoburt/snapshots/2021/Mar/19/08:19/c1iooepics.snap -l /tmp/controls_1210319_113410_0.write.log -o /tmp/controls_1210319_113410_0.nowrite.snap -v <
• No luck, the problem persists.

[Paco, Anchal]

• For MC coil balancing we will use the ASC (i.e. WFS) error signals since there are no OPLEV inputs (are there OPLEVs at all?).

Test MC1

• Using the SUS screen LockIns the plan is to feed excitation(s) through the coil outputs, and look at the ASC(Y/P) error signals.
• A diaggui xml template was saved in /users/Templates/SUS/MC1-actDiag.xml which was based on /users/Templates/SUS/ETMY-actDiag.xml
• Before running the measurement, we of course want to plug our input matrix, so we ran /scripts/SUS/InMatCalc/writeMatrix.py only to find that it tripped the MC1 Watchdog.
  • The SIDE input seems to have the largest rail, but we just followed the procedure of temporarily increasing the WD max! threshold to allow the damping action and then restoring it.
  • This happened because in latest iteration of our code, we followed an advice from the matlab code to ensure the SIDE OSEM -> SIDE DOF matrix element remains positive, but we found out that MC1 SIDE gain (C1:SUS-MC1_SUSSIDE_GAIN) was set to -8000 (instead of a positive value like all other suspensions).
  • So we decided to try our new input matrix with a positive gain value of 8000 at C1:SUS-MC1_SUSSIDE_GAIN and we were able to stablize the optic and acquire lock, but...
  • We saw that WFS YAW dof started accumulating offset and started disturbing the lock (much like last friday). We disabled the ASC Input button (C1:SUS-MC1_ASCYAW_SW2).
  • This made the lock stable and IMC autolocker was able to lock. But the offset kept on increasing (see attachment 1).
  • After sometime, the offset begain to exponential go to some steady state value which was around -3000.
• We wrote back the old matrix values and changed the C1:SUS-MC1_SUSSIDE_GAIN back to -8000. But the ASCYAW offset remained to the same position. We're leaving it disabled again as we don't know how to fix this. Hopefully, it will organically come back to small value later in the day like last time (Gautum just reenabled the ASCYAW input and it worked).

Test MC3

• Defeated by MC1, we moved to MC3.
• Here, the gain value for C1:SUS-MC3_SUSSIDE_GAIN was already positive (+500) so it could directly take our new matrix.
• When we switched off watchdog, loaded the new matrix and switched the watchdog back on.
• The IMC lock was slightly distrupted but remain locked. There was no unusual activity in the WFS sensor values. However, we saw the the SIDE coil output is slowly accumulating offset.
• So we switched off the watchdog before it will trip itself, wrote back the old matrix and reinstated the status quo.
• This suggests we need to carefully look back our latest changes of normalization and have new input matriced which keep the system stable other than working on paper with offline data.

There's an integrator in the MC WFS servos, so you should never be disabling the ASC inputs in the suspensions. Disabling 1 leg in a 6 DOF MIMO system is like a kitchen table with 1 leg removed.

Just diagnose your suspension stuff with the cavity unlocked. You should be able to see the effect by characterizing the damping loops / cross-coupling.