40m QIL Cryo_Lab CTN SUS_Lab TCS_Lab OMC_Lab CRIME_Lab FEA ENG_Labs OptContFac Mariner WBEEShop
  40m Log  Not logged in ELOG logo
 List  |  Find  |  Login  |  Help 
Entry  Mon Jul 10 09:49:02 2017, gautam, Update, General, All FEs down CDS_down_10Jul2017.png
    Reply  Mon Jul 10 11:20:20 2017, gautam, Update, General, All FEs down 
       Reply  Mon Jul 10 17:46:26 2017, gautam, Update, General, All FEs down 
          Reply  Mon Jul 10 19:15:21 2017, gautam, Update, General, All FEs down 
             Reply  Mon Jul 10 21:03:48 2017, jamie, Update, General, All FEs down 
                Reply  Mon Jul 10 22:07:35 2017, Koji, Update, General, All FEs down 
                   Reply  Tue Jul 11 15:03:55 2017, gautam, Update, General, All FEs down 
                      Reply  Tue Jul 11 15:12:57 2017, Koji, Update, General, All FEs down 
                         Reply  Wed Jul 12 10:21:07 2017, gautam, Update, General, All FEs down 
                            Reply  Wed Jul 12 14:46:09 2017, gautam, Update, General, All FEs down 
                               Reply  Wed Jul 12 14:52:32 2017, jamie, Update, General, All FEs down 
                                  Reply  Fri Jul 14 17:47:03 2017, gautam, Update, General, Disks from LLO have arrived 
Message ID: 13103     Entry time: Mon Jul 10 09:49:02 2017     Reply to this: 13104
Author: gautam 
Type: Update 
Category: General 
Subject: All FEs down 

Attachment #1: State of CDS overview screen as of 9.30AM today morning when I came in.

Looks like there may have bene a power glitch, although judging by the wall StripTool traces, if there was one, it happened more than 8 hours ago. FB is down atm so can't trend to find out when this happened.

All FEs and FB are unreachable from the control room workstations, but Megatron, Optimus and Chiara are all ssh-able. The latter reports an uptime of 704 days, so all seems okay with its UPS. Slow machines are all responding to ping as well as telnet.

Recovery process to begin now. Hopefully it isn't as complicated as the most recent effort indecision[FAMOUS LAST WORDS]

Attachment 1: CDS_down_10Jul2017.png  23 kB  | Hide | Hide all
CDS_down_10Jul2017.png
ELOG V3.1.3-