40m QIL Cryo_Lab CTN SUS_Lab TCS_Lab OMC_Lab CRIME_Lab FEA ENG_Labs OptContFac Mariner WBEEShop
  40m Log  Not logged in ELOG logo
Entry  Mon Dec 7 11:25:31 2020, gautam, Update, SUS, MC1 suspension glitchy again 
    Reply  Wed Dec 9 16:22:57 2020, gautam, Update, SUS, Yet another round of Sat. Box. switcharoo WFS2.pngWFS_lineNoise.pngWFSchar.pdf
       Reply  Sun Jan 3 16:26:06 2021, Koji, Update, SUS, IMC WFS check (Yet another round of Sat. Box. switcharoo) Screen_Shot_2021-01-03_at_17.14.57.png
    Reply  Thu May 13 11:55:04 2021, Anchal, Paco, Update, SUS, MC1 suspension misbehaving Screenshot_from_2021-05-13_09-50-24.pngMC1_Glitches_Invest2.pdf
       Reply  Thu May 13 19:38:54 2021, Anchal, Update, SUS, MC1 Satellite Amplifier Debugged MC1_UL_Channel_Fixed.png
          Reply  Mon May 24 19:14:15 2021, Anchal, Paco, Summary, SUS, MC1 Free Swing Test set to trigger 
             Reply  Tue May 25 10:22:16 2021, Anchal, Paco, Summary, SUS, MC1 new input matrix calculated and uploaded SUS_Input_Matrix_Diagonalization.pdf
          Reply  Thu Jun 17 11:45:42 2021, Anchal, Paco, Update, SUS, MC1 Gave trouble again SummaryScreenShot.pngMC1_LL_SENSOR_DEAD.png
             Reply  Thu Jun 17 16:37:23 2021, Anchal, Paco, Update, SUS, c1susaux computer rebooted 
                Reply  Tue Jun 22 11:56:16 2021, Anchal, Paco, Update, SUS, ADC/Slow channels issues CDS_FE_Status.png
                   Reply  Tue Jun 22 16:52:28 2021, Paco, Update, SUS, ADC/Slow channels issues shake_and_damp.png
                      Reply  Wed Jun 23 09:05:02 2021, Anchal, Update, SUS, MC lock acquired back again 
                         Reply  Thu Jun 24 16:40:37 2021, Koji, Update, SUS, MC lock acquired back again P_20210624_163641_1.jpg
Message ID: 16138     Entry time: Thu May 13 11:55:04 2021     In reply to: 15712     Reply to this: 16139
Author: Anchal, Paco 
Type: Update 
Category: SUS 
Subject: MC1 suspension misbehaving 

We came in the morning with the following scene on the zita monitor:

The MC1 watchdog was tripped and seemed like IMC struggled all night with misconfigured WFS offsets. After restoring the MC1 WD, clearing the WFS offsets, and seeing the suspension damp, the MC caught lock. It wasn't long before the MC unlocked, and the MC1 WD tripped again.

We tried few things, not sure what order we tried them in:

  • Letting suspension loops damp without the WFS switched on.
  • Letting suspension loops damp with PSL shutter closed.
  • Restoring old settings of MC suspension.
  • Doing burt restore with command:
    burtwb -f /opt/rtcds/caltech/c1/burt/autoburt/snapshots/2021/May/12/08:19/c1mcsepics.snap -l /tmp/controls_1210513_083437_0.write.log -o /tmp/controls_1210513_083437_0.nowrite.snap -v <

Nothing worked. We kept seeing that ULPD var on MC1 keeps showing kicks every few minutes which jolts the suspension loops. So we decided to record some data with PSL shutter closed and just suspension loops on. Then we switched off the loops and recorded some data with freely swinging optic. Even when optic was freely swinging, we could see impulses in the MC1 OSEM UL PD var which were completely uncorrelated with any seismic activity. Infact, last night was one fo teh calmer nights seismically speaking. See attachment 2 for the time series of OSEM PD variance. Red region is when the coil outputs were disabled.

Inference:

  • We think something is wrong with the UL OSEM of MC1.
  • It seems to show false spikes of motion when there is no such spike present in any other OSEM PD or the seismic data itself.
  • Currently, this is still the case. We sometimes get 10-20 min of "Good behavior" when everything works.
  • But then the impulses start occuring again and overwhelmes the suspension loops and WFS loops.
  • Note, that other optic in IMC behaved perfectly normally throughout this time.
  • In the past, it seems like satellite box has been the culprit for such glitches.
  • We should look into debugging this as ifo is at standstill because of this issue.
  • Earlier, Gautum would post Vmon signals of coil outputs only to show the glitches. We wanted to see if switching off the loops help, so we recorded OSEM PD this time.
  • In hindsight, we should probably look at the OSEM sensor outputs directly too rather than looking at the variance data only. I can do this if people are interested in looking at that too.
  • We've disabled the coil ouputs in MC1 and PSL shutter is off.

Edit Thu May 13 14:47:25 2021 :

Added OSEM Sensor timeseries data on the plots as well. The UL OSEM sensor data is the only channel which is jumping hapazardly (even during free swinging time) and varying by +/- 30. Other sensors only show some noise around a stable position as should be the case for a freely suspended optic.

Attachment 2: MC1_Glitches_Invest2.pdf  614 kB  Uploaded Thu May 13 15:46:22 2021  | Hide | Hide all
MC1_Glitches_Invest2.pdf
ELOG V3.1.3-