40m QIL Cryo_Lab CTN SUS_Lab TCS_Lab OMC_Lab CRIME_Lab FEA ENG_Labs OptContFac Mariner WBEEShop
  40m Log  Not logged in ELOG logo
Message ID: 6327     Entry time: Mon Feb 27 19:04:13 2012
Author: jamie 
Type: Update 
Category: CDS 
Subject: spontaneous timing glitch in c1lsc IO chassis? 

For some reason there appears to have been a spontaneous timing glitch in the c1lsc IO chassis that caused all models running on c1lsc to loose timing sync with the framebuilder.  All the models were reporting "0x4000" ("Timing mismatch between DAQ and FE application") in the DAQ status indicator.  Looking in the front end logs and dmesg on the c1lsc front end machine I could see no obvious indication why this would have happened.  The timing seemed to be hooked up fine, and the indicator lights on the various timing cards were nominal.

I restarted all the models on c1lsc, including and most importantly the c1x04 IOP, and things came back fine.  Below is the restart procedure I used.  Note I killed all the control models first, since the IOP can't be restarted if they're still running.  I then restarted the IOP, followed by all the other control models.

controls@c1lsc ~ 0$ for m in lsc ass oaf; do /opt/rtcds/caltech/c1/scripts/killc1${m}; done
controls@c1lsc ~ 0$ /opt/rtcds/caltech/c1/scripts/startc1x04
c1x04epics C1 IOC Server started
 * Stopping IOP awgtpman ...                                                                      [ ok ]
controls@c1lsc ~ 0$ for m in lsc ass oaf; do /opt/rtcds/caltech/c1/scripts/startc1${m}; done
c1lscepics: no process found
ERROR: Module c1lscfe does not exist in /proc/modules
c1lscepics C1 IOC Server started
 * WARNING:  awgtpman_c1lsc has not yet been started.
c1assepics: no process found
ERROR: Module c1assfe does not exist in /proc/modules
c1assepics C1 IOC Server started
 * WARNING:  awgtpman_c1ass has not yet been started.
c1oafepics: no process found
ERROR: Module c1oaffe does not exist in /proc/modules
c1oafepics C1 IOC Server started
 * WARNING:  awgtpman_c1oaf has not yet been started.
controls@c1lsc ~ 0$ 
ELOG V3.1.3-