40m QIL Cryo_Lab CTN SUS_Lab TCS_Lab OMC_Lab CRIME_Lab FEA ENG_Labs OptContFac Mariner WBEEShop
  40m Log  Not logged in ELOG logo
Entry  Thu Dec 10 08:40:12 2009, Alberto, Update, Computers, Fronte-ends down 
    Reply  Thu Dec 10 08:50:33 2009, Alberto, Update, Computers, Fronte-ends down 
       Reply  Thu Dec 10 10:01:16 2009, Jenne, Update, Computers, Fronte-ends down 
          Reply  Thu Dec 10 10:31:18 2009, Jenne, Update, Computers, Fronte-ends down 
             Reply  Thu Dec 10 13:13:08 2009, Jenne, Update, Computers, fb40m backup restarted 
Message ID: 2382     Entry time: Thu Dec 10 10:01:16 2009     In reply to: 2378     Reply to this: 2383
Author: Jenne 
Type: Update 
Category: Computers 
Subject: Fronte-ends down 

All the front ends are back up.  

Quote:

Quote:

I found all the front-ends, except for C1SUSVME1 and C0DCU1 down this morning. DAQAWG shows up green on the C0DAQ_DETAIL screen but it is on a "bad" satus.

I'll go for a big boot fest.

Since I wanted to understand once for all what's the faulting system when these situations occur, I tried to reboot the computers one by one.

1) I reset the RFM Network by pushing the reset button on the bypass switch on the 1Y7 rack. Then I tried to bring C1SOSVME up by power-cycling and restarting it as in the procedure in the wiki. I repeated a second time but it didn't work. At some point of the restarting process I get the error message "No response from EPICS".
2) I also tried rebooting only C1DCUEPICS but it didn't work: I kept having the same response when restarting C1SOSVME
3) I tried to reboot C0DAQCTRL and C1DCU1 by power cycling their crate; power-cycled and restarted C1SOSVME. Nada. Same response from C1SOSVME.
4) I restarted the framebuilder;  power-cycled and restarted C1SOSVME. Nothing. Same response from C1SOSVME.
5) I restarted the framebuilder, then rebooted C0DAQCTRL and C1DCU, then power-cycled and restarted C1SOSVME. Niente. Same response from C1SOSVME.
 
The following is the so called "Nuclear Option", the only solution that so far has proven to work in these circumstances. Execute the following steps in the order they are listed, waiting for each step to be completed before passing to the next one.
0) Switch off: the frame builder, the C0DAQCTRL and C1DCU crate, C1DCUEPICS
1) turn on the frame builder
2) reset of the RFM Network switch on 1Y7 (although, it's not sure whether this step is really necessary; but it's costless)
3) turn on C1DCUEPICS
4) turn on the C0DAQCTRL and C1DCU crate
 
One other possibility remains to be explored to avoid the Nuclear Option. And that is to just try to reset both RFM Network switches: the one in 1Y7 and the one in 1Y6.

 

ELOG V3.1.3-