40m QIL Cryo_Lab CTN SUS_Lab TCS_Lab OMC_Lab CRIME_Lab FEA ENG_Labs OptContFac Mariner WBEEShop
  40m Log  Not logged in ELOG logo
 List  |  Find  |  Login  |  Help 
Entry  Sun Nov 29 16:56:56 2009, rob, AoG, all down cond., sea of red 
    Reply  Mon Nov 30 10:28:47 2009, Alberto, AoG, all down cond., sea of red 
       Reply  Mon Nov 30 11:29:40 2009, Alberto, AoG, all down cond., sea of red 
Message ID: 2345     Entry time: Mon Nov 30 10:28:47 2009     In reply to: 2344     Reply to this: 2346
Author: Alberto 
Type: AoG 
Category: all down cond. 
Subject: sea of red 

Quote:

Came in, found all front-ends down.

 

Keyed a bunch of crates, no luck:

Requesting coeff update at 0x40f220 w/size of 0x1e44
No response from EPICS

Powered off/restarted c1dcuepics.  Still no luck.

Powered off megatron.  Success!  Ok, maybe it wasn't megatron.  I also did c1susvme1 and c1susvme2 at this time.

 

BURT restored to Nov 26, 8:00am

 

But everything is still red on the C0_DAQ_RFMNETWORK.adl screen, even though the front-ends are running and synced with the LSC.  I think this means the framebuilder or the DAQ controller is the one in trouble--I keyed the crates with DAQCTRL and DAQAWG a couple of times, with no luck, so it's probably fb40m.    I'm leaving it this way--we can deal with it tomorrow.

 I found the red sea when I came in this morning.

I tried several things.

  1. ssh into fb40m: connection refused
  2. telnet fb40m 8087: didn't respond
  3. shutdown fb40m by physically pushing the power button: it worked and the FB came back to life but still with a red light on the MEDM DAQ_DETAIL screen;
  4. powercycled fb40m AND C0DAQCTRL: no improvement
  5. shutdown fb40m, C0DAQCTRL, C1DCUEPICS and pushed the reset button on the RF network crate; then I restarted the computers in this order: fb40m, C1DCUEPICS, C0DAQCTRL: it worked: they came back to life and the lights eventually turned green on the MEDM montior screen

I'm now going to restart the single front -ends and burtgooey them if necessary.

ELOG V3.1.3-