40m QIL Cryo_Lab CTN SUS_Lab TCS_Lab OMC_Lab CRIME_Lab FEA ENG_Labs OptContFac Mariner WBEEShop
  40m Log  Not logged in ELOG logo
Entry  Mon Jan 9 09:18:47 2017, Steve, Update, PEM, power glitch MondayMorning.png
    Reply  Tue Jan 10 16:20:11 2017, Steve, Update, CDS, power glitch......Raid is rebuilding rebuilding_in_progress.png
    Reply  Tue Jan 10 21:47:00 2017, rana, Update, CDS, power glitch 
       Reply  Tue Jan 10 22:55:43 2017, gautam, Update, CDS, power glitch - recovery steps 
          Reply  Wed Jan 11 16:35:03 2017, gautam, Update, CDS, power glitch - recovery progress powerGlitchRecoveryJan2017.png
             Reply  Thu Jan 12 17:31:51 2017, gautam, Update, CDS, DC errors DCerror.png
                Reply  Fri Jan 13 21:41:23 2017, Koji, Update, CDS, DC errors 
Message ID: 12702     Entry time: Wed Jan 11 16:35:03 2017     In reply to: 12701     Reply to this: 12708
Author: gautam 
Type: Update 
Category: CDS 
Subject: power glitch - recovery progress 

[lydia, ericq, gautam]

We set about following the instructions linked in the previous elog. A few notes/remarks:

  1. It is important to run the ntpdate commands before restarting the models. Sometimes, multiple restarts of the models were required to turn all the indicator blocks on the MEDM screen green.
  2. There was also an issue of multiple ntpd processes running on the same machine, which obviously caused all sorts of timing havoc. EricQ helped us diagnose and fix these. At the moment, all the lights are green on the CDS status MEDM screen
  3. On the hardware side, apart from the usual suspects of frontends/megatron/optimus/fb needing to be rebooted, I noticed that the ETMX OSEM lights were off on the control room monitors. Investigation pointed to the 2 20V sorensens at the X end outputting 0V, 0A after the power glitch. We turned down both dials, and then gradually ramped them up again. Both Sorensens now read +/-20V, 0.3A, which is in agreement with the label stuck onto them.
  4. Restarted MC autolocker and FSS Slow scripts on megatron. I have not yet looked at the status of the nds2 server on megatron.
  5. 11 MHz Marconi has yet to be restarted - but I am unable to get even the IMC locked at the moment. For some reason, the RMS of the MC1 and MC3 coils are way higher than what I am used to seeing (~5mV rms as compared to the <1mV rms I am used to seeing for a damped optic). I will investigate further. Leaving MC autolocker disabled for now.
ELOG V3.1.3-