40m QIL Cryo_Lab CTN SUS_Lab TCS_Lab OMC_Lab CRIME_Lab FEA ENG_Labs OptContFac Mariner WBEEShop
  40m Log  Not logged in ELOG logo
Entry  Fri Jun 8 14:40:55 2018, c1lsc, Update, CDS, i am dead 31.png
    Reply  Fri Jun 8 20:15:08 2018, gautam, Update, CDS, Reboot script 31.png
       Reply  Mon Jun 11 18:49:06 2018, gautam, Update, CDS, c1lsc dead again 36.png
       Reply  Wed Dec 4 09:11:42 2019, Yehonathan, Update, CDS, Reboot script medmScreen11.ps
          Reply  Wed Dec 4 12:13:10 2019, gautam, Update, CDS, Reboot script 
Message ID: 13935     Entry time: Fri Jun 8 20:15:08 2018     In reply to: 13934     Reply to this: 13942   15071
Author: gautam 
Type: Update 
Category: CDS 
Subject: Reboot script 

Unfortunately, this has happened (and seems like it will happen) enough times that I set up a script for rebooting the machine in a controlled way, hopefully it will negate the need to repeatedly go into the VEA and hard-reboot the machines. Script lives at /opt/rtcds/caltech/c1/scripts/cds/rebootC1LSC.sh. SVN committed. It worked well for me today. All applicable CDS indicator lights are now green again. Be aware that c1oaf will probably need to be restarted manually in order to make the DC light green. Also, this script won't help you if you try to unload a model on c1lsc and the FE crashes. It relies on c1lsc being ssh-able. The basic logic is:

  1. Ask for confirmation.
  2. Shutdown all vertex optic watchdogs, PSL shutter.
  3. ssh into c1sus and c1ioo, shutdown all models on these machines, soft reboot them.
  4. ssh into c1lsc, soft reboot the machine. No attempt is made to unload the models.
  5. Wait 2 minutes for all machines to come back online.
  6. Restart models on all 3 vertex FEs (IOPs first, then rest).
  7. Prompt user for confirmation to re-enable watchdog status and open PSL shutter.
Attachment 1: 31.png  23 kB  Uploaded Fri Jun 8 21:18:47 2018  | Hide | Hide all
31.png
ELOG V3.1.3-