I found MC unlocked this morning. I looked at the 2 day trend of the MC suspensions and found MC2 suspensions have been misaligned.

I used Rana's ezcaservo trick to recover MC lock. This brought the MC_REFL down to 0.7 counts. I did the rest of the alignment by moving the MC2 suspension sliders only. MC_REFL is down to 0.45-0.5 counts and TRANS_SUM is at ~16300.

Also, I found the WFS servo was left turned OFF. I re-enabled them as well.

I tilted the periscope beam and aligned the MC.  Now the spot at the Faraday entrance is near the center of the aperture in up/down space.  The arm powers are only going up to ~0.8, though.  Maybe we should try a little bit of left/right. 

I looked at the IP POS spot with a viewer card, and it looked round, so no obvious egregious clipping in the Faraday.  Someone might take a picture with one of the GigE camera and get us a beam profile there.

We no longer have an MC1 and MC3 camera view.

I can see a bright scatterer that can be seen from the east viewport of the BSC, but I can't tell what it is.  It could be a ghost beam. 

It would be nice to get an image looking into the north viewport of the IOO chamber.  I can't see in there because the BS oplev table is in the way. 

I've asked Manasa and Q to have a look at the MC in the morning.  Rana and I have found it to be slightly uncooperative in relocking after a lockloss.

We have found that just letting the autolocker go doesn't seem to work very well, and sometimes the MC just doesn't want to re-lock.  Closing the PSL shutter or disabling the autolocker for a few minutes (5ish) doesn't do anything, but leaving it closed for a long time (30 ish minutes) helps a lot.  The MC  will relock immediately after a nice long break. 

Friday afternoon the mode cleaner got unlocked. Then some adjustment of the MC1 bias sliders locked it again. The driftmon showed the excursion for pitch and yaw of MC1 becasue it wasn't updated after the change.

Tonight Rana found the MC unlocked and simply touched the sliders to bring the OSEMs back to the driftmon values.

MC1 Yaw remains different from the driftmon. If brught back to htat value, the MC would get unlocked.

More investigation is needed to understand why the MC lock hasn't been stable for the last few days.

 The mode cleaner is still unlocked. I played with the cable at the MC2 satellite to enusre they were all plugged in.

Then I tweaked the the mirrors alignment by the sliders and eventually I could get it locked stably with 1.3 reflection. Then I rebooted C1IOO because the WFS wouldn't engage. After that the cavity wasn't locked anymore. Trying to adjust the mirrors around their position didn't restore the lock.

More work is necessary.

I'll be back on it in a while.

MC is unlocked to measure the free swing of the MC mirrors with the local sensors.

Autolocker is disabled.

MC fell out of lock and was then quite badly misaligned. Mostly in pitch. I realigned it and it locked ok.

Turns out the MC falls often out of lock when the WFS servo comes on. I think the MC2_Trans history is not cleared on lockloss. I cleared it manually and realigned. Seems fine for now.

Actually, do we need to reset the filter history at every lock loss of the MC?

Those DC offsets were necessary to keep the alignment good just until the MC is unlocked.
So if we keep the history, we can maintain the good alignment.

 I suspect the integrators get fed a huge wrong signal on lockloss. Clearing the history on the trans DOFs when the MC was badly aligned gets it nicely aligned again. I switched off the alignment transmission DOFs for now.

I have modified the 'mcwfson' and 'mcwfsoff' scripts to include the Clear History step for the MC2_TRANS_PIT and _YAW filters.  

These scripts can be run, by hand, from LOCKMC screen or from the WFS_MASTER screen.  Use the 'Turn WFS ON/OFF' button. 

The mcautolockmain script will now clear history on all ASC filter banks when the MC unlocks.

I have turned on ASC loops on the MC2_TRANS (= alignment transmission DOFs of the above elog) paths.

Came in at ~ 9 PT this morning to find the IFO "down". The IMC had lost its lock ~ 6 hours before, so at about 03:00 AM. Nothing seemed like the obvious cause; there was no record of increased seismic activity, all suspensions were damped and no watchdog had tripped, and the pressure trends similar to those in recent pressure incidents show nominal behavior (Attachment #1). What happened?

Anyways I simply tried reopening the PSL shutter, and the IMC caught its lock almost immediately. I then locked the arms and everything seems fine for now .

The MC became crazy again.

It seems that there were corresponding steps in the OSEM signals. Look at the one-day trend posted below.

As we've been seeing a bit lately, the MC will be locked happily for several hours, but then it will start misbehaving. 

Today, I measured the spots on the MC mirrors, and found that the MC2 spot was quite far off in yaw (about -3.5 cm).  I recentered the MC2 spot, and then (with the MCWFS on), moved MC1 and 3 until their WFS outputs were close to zero (they had gone up to 100+).  In the ~15 minutes since doing that, the MC refl signal is not oscillating like it was, the transmission is up, and the MC has not unlocked. 

To reiterate, I did not touch any settings of anything, except the alignment of the MC mirrors to center the MC2 spot, and then offload the WFS.  Next time the MC starts acting up, we should measure the spots, and roughly center them, before messing with any other settings.  Note however, that this is a ~10 minute procedure (including the fact that one spot measurement takes a little less than 5 minutes).  This need not be a several hour endeavour. 

Not sure why it was so poorly aligned, since the misalignment "event" happened while we were all away at lunch, but I steered the MC optics until their SUSYAW and SUSPIT values were about the same as they were before they got misaligned.  MC autolocker took over, and things are back to normal.

The MC was not damping earlier this morning around 11:45AM.  The reason was that the INMATRIX on all the three MC1, 2 and 3 were zero.   This was seen earlier when autoburt did not restore this.

This got fixed when I Burt-Restored c1mcsepics.snap

In the afternoon we had to restore c1mcs again to restore the MC_TRANS channels because its INMATRIX was also zero.

Can we do something to make sure this gets done in the autoburt properly?

[Kiwamu, Suresh]

The MC was realigned to readjust the input beam direction in pitch such that the clipping of the beam at the PSL table reduced and the railing of the PZT1 is avoided.

The current spot positions are given below on the last row:

We have quite a lot of decentering in the MC which we must try to remove by parallel transporting the beam in Pitch and Yaw..

At the current settings we might be clipping on the Faraday Isolator as we had estimated that we can allow atmost a 2mm offset in spot positions due to this constraint.

[Rana / Mirko / Kiwamu]

 The watchdogs on the MC suspensions are not working.

Switching off the watchdogs doesn't stop feeding signals to the suspensions.

For tonight, we will leave the controller of the MC suspensions switched off so that the computer won't smash the optics accidentally.

It seems as if someone was looking into this on Wednesday, but as there's no elog, I think probably not.

Tonight we noticed that, in fact, the watchdogs don't work for any of the corner optics (I confirmed that they do work for the ETMs).

Whether the switch for the coil drivers is enabled or disabled, the voltage monitor on the coil drivers responds to the digital signals.

I tried restarting the c1susaux process, hitting reset on the crate, and also keying the crate. The +5V light on the front of the crate is flickering somewhat, but I'm not sure if this is new or not since the power outage. The next step is to track the Xycom signal from the card over to the coil driver to find where the signal is failing to happen.

Since its not working for any of the corner optics, I suspect the crate and not the cards. If that's the issue this will be a painful fix. We are sort of assuming that this is due to the power outage, but in fact, I cannot tell when the last time was that someone rigorously verified the working of these switches. [

I][B]We had better get ready for upgrading our SLOW controls, Jamie.[/B][/I]

~20 minutes ago, maybe right around the time the fb's RAID died (elog 8274) the mode cleaner started behaving weirdly again.  The reflected value is very high, even with the WFS on.  Earlier this evening, I saw that with the WFS off, the MC reflection was high, but the WFS brought it back down to ~0.7 or 0.8.  But now it's ~1.3.  With the WFS off, the reflected value is ~1.1.  I don't really understand.

Also, the PMC has been drifting in alignment in pitch all day, but a lot more later in the day. The PMC trans is 0.800 right now, but it was as high as 0.825 today, and spent most of the day in the high 0.81xxx range today.

I would provide plots, but as mentioned in elog 8274, we can't get data right now.

[Manasa, Annalisa, Jenne]

The MC wasn't locking on TEM00 this morning, and the WFS kept pulling the MC out of alignment.  The MC was realigned, and the WFS spots are back to being roughly centered (all of this only touching the MC sliders), but the WFS keep doing bad things.  They're okay, and improve the alignment slightly at first, but as soon as the FM1 integrator comes on, the MC alignment immediately starts going bad, and within a second or so the MC has unlocked.

The WFS are off right now, and we'll keep investigating after LIGOX.

[Jenne, Manasa, Yuta]

Since we levelled IMC stack, we had to center beam spots on MC mirrors again.
We did this by steering PSL mirror in yaw (about same amount but opposite direction to what we did in elog #8077)
Residual beam tilt compared with a line through MC1 and 3 actuator nodes is ~ 15 mrad, mainly in yaw.

The mode cleaner is super unhappy.  It's rocking around at ~1Hz.

I turned off the WFS and turned them back on after the MC was locked, and it seems a little happier now.  At least it's not falling out of lock ~1/minute.

Regarding http://nodus.ligo.caltech.edu:8080/40m/5867 and http://nodus.ligo.caltech.edu:8080/40m/5869 :

MC_F signal:

The measurements on p. 5867 were done using the ADC attached to the PEM computer. There was a big difference between the MC_F signal recorded directly after the server board and the signal just before the FSS board as recorded by a PEM channel.
To understand how this happens we measured the signal at different places with a spec. analyzer:

1. WIth a locked MC measuring the signal just before the PEM ADCs (meaning after a 60ft BNC cable)
2. Same position, but unlocked and seemingly dark MC
3. Locked MC, signal just before the FSS box
4. MC_F signal that is usually going into the Pentek Generic board and is recorded in C1:IOO

=> The 60ft BNC cable adds a considerable amount of noise, but doesn't fundamentally change the signal. It is weird that the signal is white from approx. 4Hz on.
Due to Jenne's measurement ( http://nodus.ligo.caltech.edu:8080/40m/5848 ) we know the TF from MCL through PD, mixer Pentek and into C1:IOO looks like this:

This is with the double HP from 15Hz on that should be in the Pentek. So one might expect a less white signal going into the FSS board...

 PEM ADCs

The dark noise in the PEM ADCs is actually a factor 10 higher than in the IOO ADCs. Still ok wrt the the seismometers.
We also tried to measure essentially the dark noise of the whole seismometer readout (seismometer box, then ADC). That seemed ok, but is of limited value since the seismometer electronics behave a bit strange when there is no seismometer connected.

I tried unsuccessfully to relock the MC this afternoon.

I came in to find it in a trouble state with a huge amount of noise on C1:PSL-FSS_PCDRIVE visible on the projector monitor. Light was reaching the MC but it was unable to lock.

• I checked the status of the fast machines on the CDS>FE STATUS page. All up.
• Then I checked the slow machine status. c1iscaux and c1psl were both down. I manually reset both machines. The large noise visible on C1:PSL-FSS_PCDRIVE disappeared.
• After the reset, light was no longer reaching the MC, which I take to mean the PMC was not locked. On the PSL>PMC page, I blanked the control signal, reenabled it, and attempted to relock by adjusting the servo gain as Gautam had showed me before. The PMC locks were unstable, with each one lasting only a second or so.
• Next I tried restoring the burt states for c1iscaux and c1psl from a snapshot taken earlier today, before the machine reboots. That did not solve the problem either.

I don't know what had been wrong, but I could lock the PMC as usual.
The IMC got relocked by AutoLocker. I checked the LSC and confirmed at least Y arm could be locked just by turning on the LSC servos.

Great, thanks!

I set up video monitoring of MC1 and MC3

Started the troubleshoot from the MC1 issue. Gautam showed me how to use the fake PD/LED pair to diagnose the satellite box without involving the suspension mechanics.

This revealed that the MC1 has frequent light level glitches which are common for five sensors. This feature does not exist in the test with the MC3 satellite box. I will open and check the MC1 satellite box to find the cause of this common glitches tomorrow. MC1 is currently shutdown and undamped.

BTW, at the MC3 test, i found that J2 of the satellite box (male Dsub) has all the pins too low (or too short?). I brought the box outside and found that the housing of this connector was half broken down. The connector was reassembled and the metal parts of the housing was bent again so that the housing can hold the connector body tightly.

The MC3 satellite box was restored and connected to the cables. As I touched this box, it is still under probation.

About 30mins ago, I saw another glitch on MC1 - this happened while the Watchdog was shutdown.

In order to further narrow down the cause of the glitch, we switched the Coil Driver Board --> Satellite box DB(15?) connectors on the coil drivers between MC1 and MC3 coil driver boards. I also changed the static PIT/YAW bias voltages to MC1 and MC3 such that MC-REFL is now approximately back to the center of the CCD monitor.

Now that all the CDS overview lights are green, I decided to switch back the coil driver outputs to their original state so that the MC optics could be damped and the IMC relocked. I also restored the static PIT/YAW bias values to their original values.

MC1 has been quiet over the last couple of days, lets see how it behaves in the next few days. In all the glitches I have observed, if the IMC is locked and WFS loops are enabled, the loops are able to correct for the DC misalignment caused by the glitch. But the mcwfs off script is currently set up in such a way that the output history is cleared between IMC locks. I made two copies of the mcwfson/mcwfsoff scripts, called mcwfsunhold/mcwfshold respectively. They live in /opt/rtcds/caltech/c1/scripts/MC/WFS. I've also modified the autolocker script to call these modified scripts, such that when the IMC loses lock, the WFS servo outputs are held, while the input is turned off. The hope is that in this configuration, the autolocker can catch a lock even if there is a glitch on MC1.

I haven't tried locking the arms yet, but I think other IFO work discussed at the meeting (like arm loss estimation / cavity scans etc) can proceed.

Seems like this modification didn't really work. There were several large MC1 glitches, and one of them misaligned MC1 so much that the IMC didn't relock for the last ~6 hours. I re-aligned MC1 manually, and now it is locked fine.

that's why the Autolocker clears the outputs; we don't want to be holding the offsets from the last ms of lock when it was all messed up; instead it would be best to have a slow (~mHz) relief script that takes the WFS controls and puts them onto the MC SUS sliders. This would then re-align the MC to the input beam rather than the input to the MC. Which is not the best idea.

I moved the MC1 set of accelerators. Might have bumped things. If things aren't working, look around the MC1 chamber.

Also, I constructed two new XLR cables, but have not tested them yet.

Kevin sent me an email with top secret info on where one of the other accelerometer cubes was hiding (it was with his shaker setup on the south side of the SP table), so I took it and put the 3 MC1 accelerometers in their 3-axis configuration. 

Also, I changed the orientation of both sets of 3 axis accelerometers to reflect a Right Handed configuration, to go along with the new and improved IFO configuration.  Previously (including last night), the MC2 accelerometers were together in a Left Handed configuration.

I have pulled out MC1 coil driver board from its Eurocrate, so IMC is unavailable until further notice. Plans:

1. Thick film --> Thin Film
2. AD797 --> Op27
3. Remove Pots in analog actuation path.
4. Measure noise
5. Route HPF signal (UL DAQ Mon) to front panel. I think we should use the SMA connectors. That way, we have DC and AC voltage monitors available for debugging.

If there are no objections, I will execute Step #5 in the next couple of hours. I'm going to start with Steps 1-4.

This work is now complete. MC1 coil driver board has been reinstalled, local damping of MC1 restored, and IMC has been locked. Detailed report + photos to follow, but measurement of the noise (for one channel) on the electronics workbench shows a broadband noise level of 5nV/rtHz () around 100Hz, which is lower than what was measured here and consistent with what we expect from LISO modeling (with fast input terminated with 50ohm, slow input grounded).

• Marked up schematic + photo post changes uploaded to DCC page.
• There was a capacitor in the DAQ monitor path making a 8kHz corner that I now removed (since the main point of this front panel HPF monitor point is to facilitate easy coil driver noise debugging, and I wanted to be able to use the SR785 out to high frequencies without accounting for an additional low pass). Transfer function from front panel LEMO input to front panel LEMO monitor is shown in Attachment #1.
• Voltage noise measured at DB25 output (with the help of a breakout cable and SR560 G=100) with front panel LEMO input terminated to 50ohm, Bias input grounded, and pin1 of U21A grounded (i.e. watchdog enabled state) is shown in Attachment #2. This measurement was taken on the electronics bench.
• Inside the lab (i.e. coil driver board plugged into eurocrate), the noise measured in the same way looks identical to what was measured in elog13870.
• I tried repeating the measurement by powering the board using an bench power supply and grounding the bias input voltage near 1X6, and the strange noise profile persists. So this supports the hypothesis that some kind of environmental pickup is causing this noise profile. Needs more investigation. 

In any case, if it is indeed true that the optic sees this current noise, the place to make the measurement is probably the Sat. Box. Who knows what the pickup is over the ~15m of cable from 1X6 to the optic.

[rana,gautam]

Summary:

Last night, Rana fact-checked my story about the coil driver noise measurement. Conclusions:

1. There is definitely pickup of strong lines (see Attachment #1. These are hypothesized to come from switching power supplies). Moreover, they breathe. Checkout Rana's twitter page for the video.
2. The lines are almost (but not quite) at integer multiples of 19.5 kHz. The cause of this anharmonicity is to be puzzled out.
3. When the coil driver board is located ~1m away from the SR785 and the bench supply powering it, even though the lines are visible in the spectrum, the low frequency shape does not show the weird broad features I reported here. The measured noise floor level is ~5nV/rtHz, which is consistent with LISO noise + SR560 input noise (see Attachment #2). However, there is still some excess noise at 100 Hz above what the LISO model leads us to expect. 
4. The location of the coil driver board and SR560 relative to the SR785 and the bench power supply I used to power the coil driver board can increase the line heights by ~x50. 
5. The above changes the shape of the low frequency part of the spectrum as well, and it looks more like what is reported in elog13870. The hypothesis is that the high frequency lines are downconverted in the SR560.

Note: All measurements were made with the fast input of the coil driver board terminated with 50ohms and bias input shorted to ground with a crocodile clip cable.

Next steps:

The first goal is to figure out where this pickup is happening, and if it is actually going to the optic. To this end, I will put a passive 100 kHz filter between the coil driver output and the preamp (Busby Box instead of SR560). By getting a clean measurement of the noise floor with the coil driver board in the Eurocrate (with the bias input driven), we can confirm that the optic isn't being buffeted by the excess coil driver noise. If we confirm that the excess noise is not a measurement artefact, we need to think about were the pickup is actually happening and come up with mitigation strategies.

RXA: good section EMI/RFI in Op Amp Applications handbook (2006) by Walt Jung. Also this page: http://www.electronicdesign.com/analog/what-was-noise

We've set a free swing test to trigger at 3:30 am tomorrow for MC1. The script for tests is running on tmux session named 'freeSwingMC1' on rossa. The script will run for about 4.5 hrs and we'll correct the input matrix tomorrow from the results. If anyone wants to work during this time (3:30 am to 8:00 am), you can just kill the script by killing tmux session on rossa. ssh into rossa and type tmux kill-session -t freeSwingMC1.

TL;DR

MC1 LL Sensor showed signs of fluctuating large offsets. We tried to find the issue in the box but couldn't find any. On power cycling, the sensor got back to normal. But in putting back the box, we bumped something and c1susaux slow channels froze. We tried to reboot it, but it didn't work and the channels do not exist anymore.

Today morning we came to find that IMC struggled to lock all night (See attachment 1). We kind of had an indication yesterday evening that MC1 LL Sensor PD had a higher variance than usual and Paco had to reset WFS offsets because they had integrated the noise from this sensor. Something similar happened last night, that a false offset and its fluctuation overwhelmed WFS and MC1 got misaligned making it impossible for IMC to get lock.

In the morning, Paco again reset the WFS offsets but not we were sure that the PD variance from MC1 LL osem was very high. See attachment 2 to see how only 1 OSEM is showing higher noise in comparison to the other 4 OSEMs. This behavior is similar to what we saw earlier in 16138 but for UL sensor. Koji and I fixed it in 16139 and we tested all other channels too.

So, Paco and I, went ahead and took out the MC1 satellite amplifier box S2100029 D1002812, opened the top, and checked all the PD channel testpoints with no input current. We didn't find anything odd. Next we checked the LED dirver circuit testpoints with LED OUT and GND shorted. We got 4.997V on all LED MON testpoints which indicate normal functioning.

We just hooked back everything on the MC1 satellit box and checked the sensor channels again on medm screens. To our surprise, it started functioning normally. So maybe, just a power cycling was required but we still don't know what caused this issue.

BUT when I (Anchal) was plugging back the power cables and D25 connectors on the back side in 1X4 after moving the box back into the rack, we found that the slow channels stopped updating. They just froze!

We got worried for some time as the negative power supply indicator LEDs on the acromag chassis (which is just below the MC1 satellite box) were not ON. We checked the power cables and had to open the side panel of the 1X4 rack to check how the power cables are connected. We found that there is no third wire in the power cables and the acromag chassis only takes in single rail supply. We confirmed this by looking at another acromag chassis on Xend. We pasted a note on the acromag chassis for future reference that it uses only positive rails and negative LED monitors are not usually ON.

Back to solving the frozen acromag issue, we conjectured that maybe the ethernet connection is broken. The DB25 cables for the satellite box are bit short and pull around other cables with it when connected. We checked all the ethernet cabling, it looked fine. On c1susaux computer, we saw that the monitor LED for ethernet port 2 which is connected to acromag chassis is solid ON while the other one (which is probably connection to the switch) is blinking.

We tried doing telnet to the computer, it didn't work. The host refused connection from pianosa workstation. We tried pinging the c1susaux computer, and that worked. So we concluded that most probably, the epics modbus server hosting the slow channels on c1susaux is unable to communicate with acromag chassis and hence the solid LED light on that ethernet port instead of a blinking one. We checked computer restart procedure page for SLOW computers on wiki and found that it said if telnet is not working, we can hard reboot the computer.

We hard reboot the computer by long pressing the power button and then presssing it back on. We did this process 3 times with the same result. The ethernet port 2 LED (Acromag chassis) would blink but the ethernet port 1 LED (connected to switch) would not turn ON. We now can not even ping the machine now, let alone telnet into it. All SUS slow monitor channels are not present now ofcourse. We also tried once pressing the reset button (which the manual said would reboot the machine), but we got the same outcome.

Now, we decided to stop poking around until someone with more experience can help us on this.

Bottomline: We don't know what caused the LL sensor issue and hence it has not been fixed. It can happen again. We lost all C1SUSAUX slow channels which are the OSEM and COIL slow monitor channels for PRM, BS, ITMX, ITMY, MC1, MC2 and MC3.

I noticed 2 periods of frequent IMC locklosses on the StripTool trace, and so checked the MC1 PD readout channels to see if there were any coincident glitches. Turns out there wasnt BUT - the LR and UR signals had changed significantly over the last couple of days, which is when I've been working at 1X5. The fast LR readback was actually showing ~0, but the slow monitor channel had been steady so I suspected some cabling shenanigans.

Turns out, the problem was that the LEMO connector on the front of the MC1 whitening board had gotten jiggled ever so slightly - I re-jiggled it till the LR fast channel registered similar number of counts to the other channels. All looks good for now. For good measure, I checked the 3 day trend for the fast PD readback for all 8 SOS optics (40 channels in all, I didn't look at the ETMs as their whitening boards are at the ends), and everything looks okay... This while situation seems very precarious to me, perhaps we should have a more robust signal routing from the OSEMs to the DAQ that is more immune to cable touching etc...

 Since the MC1 LRSEN channel is not wasn't working, my input matrix diagonalization hasn't worked today wasn't working. So I decided to fix it somehow.

I went to the rack and traced the signal: first at the LEMO monitor on the whitening card, secondly at the 4-pin LEMO cable which goes into the AA chassis.

The signal existed at the input to the AA chassis but not in the screen. So I pressed the jumper wire (used to be AA filter) down for the channel corresponding to the MC1 LRSEN channel.

It now has come back and looks like the other sensors. As you can see from this plot and Joe's entry from a couple weeks ago, this channel has been dead since May 17th.

The ELOG reveals that Kiwamu caught Steve doing some (un-elogged) fooling around there. Burnt Toast -> Steve.

993190663   =      free swinging ringdown restarted again

[Den, Mirko, Suresh]

    We were investigating why there is no correlation between MC1 osem signals and seismic motion.   During this we noticed a recurrence of this old problem of MC1_LR sensor being dead.  I went and pressed down the chip holders where the AA filters used to sit and which now hold the jumper wire.  The board is large and flexible it is quite likely some solder joint is broken on the MC1_LR path on this board.

   The signal came back to life and is okay now. But it can break off again any time.

There appears to be a bad cable connection somewhere on the LR sensor path for the MC1 optic.

The channel C1:SUS-MC1_LRPDMon is reading back 0.664 volts, but the digital sensor channel, C1:SUS-MC1_LRSEN_INMON, is reading about -16.  This should be closer to +1000 or so.

We've temporarily turned off the LRSEN filter module output while this is being looked into.

I briefly went out and checked the cables around the whitening and AA boards for the suspension sensors, but even after wiggling and making sure everything was plugged in solidly.  There was one semi-loose connection, but it wasn't on the MC1 board, but I pushed it all the way in anyways.  The monitor point on the AA board looks correct for the LR channels, although ITMX LR struck me as being very low at about -0.05 Volts.

According to data viewer, the MC1 LR sensor channel went bad roughly two weeks ago, around 00:40 on 5/18 UTC, or 17:40 on 5/17 PDT.

UPDATE:

It appears the AA board (or possibly the SCSI cable connected to it) is the problem in the chain.

[Tomislav Andric, Rana, Yuta]

We put -1 to MC1 OSEM sensor gains and re-tuned MC1 damping.
We also kicked MC1, MC2, MC3 tonight for input matrix diagonalization.

MC1 damping investigations:
 We put -1 to MC1 OSEM sensor gains so that UL/UR/LR/LL/SDSEN_OUT will be positive like other optics.
 OSEM damping filter gains were adjusted.
 We have also checked if having +1 for all UL/UR/LR/LL/SDCOIL_GAIN is correct or not. It has been like this at least for the past year.
 It should be -1 for UR and LL to account for magnets, but if we did put -1 or them, kick in C1:SUS-MC1_PIT_OFFSET mostly gave yaw kick and kick in C1:SUS-MC1_YAW_OFFSET mostly give pitch kick.
 So, we reverted them to be +1.

Input matrix diagonalization:
 We also kicked MC1, MC2, MC3 tonight input matrix diagonalization.
 Kick was done manually at the following times local.
  - MC1 20:08 June 6th, 2022
  - MC2 20:24 June 6th, 2022
  - MC3 20:21 June 6th, 2022
 We will leave watchdogs shutdown to free swing overnight (damping loops are "on").
 This will help get better angular sensor from OSEMs to calibrate WFS signals.

Next:
 - Investigate why MC1 coils gains have +1 for all
 - Calculate input matrix. Make sure SUSPOS/PIT/YAW/SIDE_IN will be in the units of um or urad.

Suggestions:
 - Add filter ramp time of 1sec for all by default
 - Make null stream channel from input matrix for diagnostics