40m QIL Cryo_Lab CTN SUS_Lab TCS_Lab OMC_Lab CRIME_Lab FEA ENG_Labs OptContFac Mariner WBEEShop
  40m Log  Not logged in ELOG logo
Entry  Mon Aug 23 11:51:26 2021, Koji, Update, General, Campus Wide Power Glitch Reported: Monday, 8/23/21 at 9:30am  
    Reply  Mon Aug 23 19:00:05 2021, Koji, Update, General, Campus Wide Power Glitch Reported: Monday, 8/23/21 at 9:30am  
       Reply  Mon Aug 23 22:51:44 2021, Anchal, Update, General, Time synchronization efforts 
          Reply  Tue Aug 24 09:22:48 2021, Anchal, Update, General, Time synchronization working now 
             Reply  Tue Aug 24 18:11:27 2021, Paco, Update, General, Time synchronization not really working 
                Reply  Tue Aug 24 22:37:40 2021, Anchal, Update, General, Time synchronization not really working 
Message ID: 16288     Entry time: Mon Aug 23 11:51:26 2021     Reply to this: 16290
Author: Koji 
Type: Update 
Category: General 
Subject: Campus Wide Power Glitch Reported: Monday, 8/23/21 at 9:30am  

Campus Wide Power Glitch Reported: Monday, 8/23/21 at 9:30am (more like 9:34am according to nodus log)

nodus: rebooted. ELOG/apache/svn is running. (looks like Anchal worked on it)

chiara: survived the glitch thanks to UPS

fb1: not responding -> @1pm open to login / seemed rebooted only at 9:34am (network path recovered???)

megatron: not responding

optimus: no route to host

c1aux: ping ok, ssh not responding -> needed to use telnet (vme / vxworks)
c1auxex: ssh ok
c1auxey: ping ok, ssh not respoding -> needed to use telnet (vme / vxworks)
c1psl: ping NG, power cycled the switch on 1X2 -> ssh OK now
c1iscaux: ping NG -> rebooted the machine -> ssh recovered

c1iscaux2: does not exist any more
c1susaux: ping NG -> responds after 1X2 switch reboot

c1pem1: telnet ok (vme / vxworks)
c1iool0: does not exist any more

c1vac1: ethernet service restarted locally -> responding
ottavia: doesnot exist?
c1teststand: ping ok, ssh not respoding

3:20PM we started restarting the RTS

ELOG V3.1.3-