40m QIL Cryo_Lab CTN SUS_Lab TCS_Lab OMC_Lab CRIME_Lab FEA ENG_Labs OptContFac Mariner WBEEShop
  40m Log  Not logged in ELOG logo
Entry  Wed Dec 12 11:53:29 2018, aaron, Update, General, Power Outage IMG_0033.HEICIMG_1027.HEICIMG_2605.HEIC
    Reply  Thu Dec 13 01:26:34 2018, gautam, Update, General, Power Outage recovery 
       Reply  Thu Dec 13 12:06:35 2018, gautam, Update, General, Power Outage recovery 
          Reply  Thu Dec 13 20:10:08 2018, Koji, Update, General, Power Outage recovery OMRON_S82K-00712.JPGreset_button_repaired1.JPGreset_button_repaired2.JPGgray_box.JPGgray_box2.JPG
Message ID: 14349     Entry time: Thu Dec 13 01:26:34 2018     In reply to: 14347     Reply to this: 14351
Author: gautam 
Type: Update 
Category: General 
Subject: Power Outage recovery 

[koji, gautam]

After several combinations of soft/hard reboots for FB, FEs and expansion chassis, we managed to recover the nominal RTCDS status post power outage. The final reboots were undertaken by the rebootC1LSC.sh script while we went to Hotel Constance. Upon returning, Koji found all the lights to be green. Some remarks:

  1. It seems that we need to first turn on FB
    • Manually start the open-mx and mx services using
      sudo systemctl start open-mx.service 
      sudo systemctl start mx.service
    • Check that the system time returned by gpstime matches the gpstime reported by internet sources.
    • Manually start the daqd processes using
      sudo systemctl start daqd_*
  2. Then fully power cycle (including all front and rear panel power switches/cables) the FEs and the expansion chassis.
    • This seems to be a necessary step for models run on c1sus (as reported by the CDS MEDM screen) to pick up the correct system time (the FE itself seems to pick up the correct time, not sure what's going on here).
    • This was necessary to clear 0x4000 errors.
  3. Power on the expansion chassis.
  4. Power on the FE.
  5. Start the RTCDS models in the usual way
    • For some reason, there is a 1 second mismatch between the gpstime returned on the MEDM screen for a particular CDS model status, and that in the terminal for the host machine.
    • This in itself doesn't seem to cause any timing errors. But see remark about c1sus above in #2.

The PSL (Edwin) remains in an interlock-triggered state. We are not sure what is causing this, but the laser cannot be powered on until this is resolved.

ELOG V3.1.3-