40m QIL Cryo_Lab CTN SUS_Lab TCS_Lab OMC_Lab CRIME_Lab FEA ENG_Labs OptContFac Mariner WBEEShop
  40m Log  Not logged in ELOG logo
Entry  Mon Feb 15 12:28:03 2016, gautam, Update, General, Something has gone wrong - was there a power outage? Vac_15Feb2016.png
    Reply  Mon Feb 15 13:09:33 2016, Koji, Update, General, Something has gone wrong - was there a power outage? 
       Reply  Tue Feb 16 09:13:39 2016, Steve, Update, General, there was a power outage & EQ 20160215_power_outage.png
          Reply  Tue Feb 16 15:02:19 2016, ericq, Update, General, Power Glitch recovery 
             Reply  Tue Feb 16 23:42:22 2016, gautam, Update, General, Power Glitch recovery - arms recovered 
                Reply  Wed Feb 17 13:45:15 2016, ericq, Update, General, No monit on frontends 
             Reply  Thu Feb 25 11:02:36 2016, Steve, Update, General, Power Glitch again powerGlitch.png
                Reply  Thu Feb 25 11:32:04 2016, gautam, Update, General, Power Glitch again 
          Reply  Mon Feb 22 08:45:46 2016, Steve, Update, SUS, EQ 4.3 grabs ITMX-UL magnet EQ4.3LucerneV.pngITMX-UL.png
             Reply  Mon Feb 22 23:00:21 2016, ericq, Update, SUS, ITMX free, green is flashing 
                Reply  Tue Feb 23 09:51:34 2016, Steve, Update, SUS, ITMX free ITMX_FREED.pngITMX_FREE.png
                   Reply  Tue Feb 23 17:46:09 2016, gautam, Update, SUS, EQ  SUS_Summary_23Feb.png
             Reply  Tue Apr 5 08:55:51 2016, Steve, Update, SUS, local EQ 3.1m eq3.1Valencia.png
                Reply  Wed May 25 08:32:55 2016, Steve, Update, SUS, local EQ 3.5m 3.5Cucam.pnglocal3.5cucam.png
                   Reply  Fri Jun 3 14:24:13 2016, Steve, Update, SUS, local EQ 3.1 m eq3.1mMuscoyCa.png
                      Reply  Fri Jun 10 09:52:51 2016, Steve, Update, SUS, EQ 5.2 mag Borego Spring 5.2mBoregoSpring.pngeqBSs.png
                         Reply  Mon Jun 20 09:36:10 2016, Steve, Update, SUS, EQ 3.1 m Big Bear Lake, CA eq3.1BigBearLakeCA.png
Message ID: 11993     Entry time: Tue Feb 16 15:02:19 2016     In reply to: 11992     Reply to this: 11995   12010
Author: ericq 
Type: Update 
Category: General 
Subject: Power Glitch recovery 

Chiara reports an uptime of >195 days, so its UPS is working fine yes

FB, megatron, optimus booted via front panel button.

Jetstor RAID array (where the frames live) was beeping, since its UPS failed as well. The beep was silenced by clicking on "View Events/Mute Beeper" at 192.168.113.119 in a browser on a martian computer. I've started a data consistency check via the web interface, as well. According to the log, this was last done in July 2015, and took ~19 hrs.

Frontends powered up; models don't start automatically at boot anymore, so I ran rtcds start all on each of them. 

All frontends except c1ioo had a very wrong datetime, so I ran sudo ntpdate -b -s -u pool.ntp.org on all of them, and restarted the models (just updating the time isn't enough). There is an /etc/ntp.conf in the frontend filesystem that points to nodus, which is set up as an NTP server, but I guess this isn't working.

PMC locking was hindered by sticky sliders. I burtrestored the c1psl.snap from Friday, and the PMC locked up fine. (One may be fooled by the unchanged HV mon when moving the offset slider into thinking the HV KEPCO power supplies need to be brought down and up again, but it's just the sliders)

Mode cleaner manually locked and somewhat aligned. Based on my memory of PMC camera/transmission, the pointing changed; the WFS need a round of MC alignment and WFS offset setting, but the current state is fine for operation without all that. 

ELOG V3.1.3-