40m QIL Cryo_Lab CTN SUS_Lab TCS_Lab OMC_Lab CRIME_Lab FEA ENG_Labs OptContFac Mariner WBEEShop
  40m Log  Not logged in ELOG logo
Entry  Mon Jun 23 20:58:38 2014, ericq, Update, CDS, Bootfest 2014! 
    Reply  Tue Jun 24 01:07:56 2014, Jenne, Update, LSC, Bootfest 2014! MMT1_face_23June2014_2315pm.png
       Reply  Tue Jun 24 18:35:53 2014, Jenne, Update, LSC, Still no beam going to IFO 
Message ID: 10088     Entry time: Mon Jun 23 20:58:38 2014     Reply to this: 10090
Author: ericq 
Type: Update 
Category: CDS 
Subject: Bootfest 2014! 

This afternoon, I wanted to start the nominal alignment/adjustment steps for evening time locking, but got sucked into CDS frustrations. 

Primary symptom: TRX and TRY signals were not making it from C1:SUS-ETMX_TR[X,Y]_OUT to C1:LSC-TR[X,Y]_IN1. Various RFM bits were red on the CDS status page. 

 Secondary symptom: ITMX was randomly getting a good sized kick for no apparent reason. I still don't know what was behind this. 

First fix attempt: run sudo ntpdate -b -s -u pool.ntp.org on c1sus and c1lsc front ends, to see if NTP issues were responsible. No result.

Second fix attempt: Restart c1lsc, c1sus and c1rfm models. No change

Next fix attempt: Restart c1lsc and c1sus frontend machines. c1lsc models come back, c1sus models fail to sync / time out/ dmesg has some weird message about ADC channel hopping. At this point, c1ioo, c1iscey and c1iscex all have their models stop working due to sync problems. 

I then ran the above ntp command on all front ends and the FB, and restarted everyone's models (except c1lsc, who stayed working from here on out) which didn't change anything. I command-line rebooted all front ends (except c1lsc) and the FB (which had some dmesg messages about daqd segfaulting, but daqd issues weren't the problem). Still nothing. 

Finally, Koji came along and relieved me from my agony by hard rebooting all of the front ends; pulling out their power cables and seeing the life in their lights fade away... He did this first with the end station machines (c1iscey and c1iscex), and we saw them come back up perfectly happy, and then c1ioo and c1sus followed. At this point, all models came back; green RFM bits abounding, and TR[X,Y] signals propagating through as desired. 

Then, we tried turning the damping/watchdogs back on, which for some strange reason started shaking the hell out of everyone except the ETMs and ITMX. We restarted c1sus and c1mcs, and then damping worked again. Maybe a bad BURT restore was to blame?

At this point, all models were happy, all optics were damped, mode cleaner + WFS locked happily, but no beams were to be seen in the IFO 

The Yarm green would lock fine though, so tip-tilt alignment is probably to blame. I then left the interferometer to Jenne and Koji. 

ELOG V3.1.3-