40m QIL Cryo_Lab CTN SUS_Lab TCS_Lab OMC_Lab CRIME_Lab FEA ENG_Labs OptContFac Mariner WBEEShop
  40m Log  Not logged in ELOG logo
 List  |  Find  |  Login  |  Help 
Entry  Sat Jun 23 18:44:15 2012, steve, Update, General, power surge has no effect on the lab 
    Reply  Sat Jun 23 19:57:22 2012, yuta, Summary, Computers, c1ioo is down 
       Reply  Mon Jun 25 10:35:59 2012, Jenne, Summary, Computers, c1ioo is down 
          Reply  Mon Jun 25 11:27:13 2012, Jamie, Summary, Computers, c1ioo is down 
    Reply  Mon Jun 25 08:21:40 2012, steve, Update, General, AC power disturbance on Sat powerdisturbed.png
Message ID: 6867     Entry time: Mon Jun 25 11:27:13 2012     In reply to: 6865
Author: Jamie 
Type: Summary 
Category: Computers 
Subject: c1ioo is down 

Quote:

Quote:

I tried to restart c1ioo becuase I can't live without him.

I couldn't ssh or ping c1ioo, so I did hardware reboot.
c1ioo came back, but now ADC/DAC stats are all red.

c1ioo was OK until 3am when I left the control room last night. I don't know what happened, but StripTool from zita tells me that MC lock went off at around 4pm.

 c1ioo was still all red on the CDS status screen, so I tried a couple of things.

mxstreamrestart (which aliases on the front ends to sudo /etc/init.d/mx_stream restart) didn't help

sudo shutdown -r now didn't change anything either....c1ioo came back with red everywhere and 0x2bad on the IOP

eventually doing as Jamie did for c1sus in elog 6742, rtcds stop all, then rtcds start all fixed everything.  Interestingly, when I tried rtcds start iop, I got the error
Cannot start/stop model 'iop' on host c1ioo, so I just tried rtcds start all, and that worked fine....started with c1x03, then c1ioo, then c1gcv.

There seems to be a problem with how models are coming up on boot since the upgrade.  I think the IOP isn't coming up correctly for some reason, which is then preventing the rest of the models from starting since they depend on the IOP.

The simple way to fix this is to run the following:

ssh c1ioo
rtcds restart all

The "restart" command does the smart thing, by stopping all the models in the correct order (IOP last) and then restarting them also in the correct order (IOP first).

ELOG V3.1.3-