40m QIL Cryo_Lab CTN SUS_Lab TCS_Lab OMC_Lab CRIME_Lab FEA ENG_Labs OptContFac Mariner WBEEShop
  40m Log  Not logged in ELOG logo
Entry  Thu Mar 8 01:10:22 2018, gautam, Update, General, CDS recovery after work at LSC rack CDS-recovery.png
    Reply  Thu Mar 8 14:41:25 2018, gautam, Update, General, CDS recovery after work at LSC rack CDS-recovery.pngc1sus_timing.pngc1sus_crashed.png
       Reply  Thu Mar 8 18:15:42 2018, gautam, Update, General, CDS recovery after work at LSC rack 
Message ID: 13670     Entry time: Thu Mar 8 14:41:25 2018     In reply to: 13669     Reply to this: 13672
Author: gautam 
Type: Update 
Category: General 
Subject: CDS recovery after work at LSC rack 

As I had found before, restarting the c1oaf model fixed the DC error. There is however still a pesky red indicator light on the "ADC0" in c1oaf. Trying to open up the ADC MEDM screen to investigate this further leads to the blank screen on the bottom right of Attachment #1. Probably has something to do with the fact that the model has an ADC block (because every model needs one?) but no signals are actually being piped to the model directly from the ADC.

Another observation, though I don't have any hypothesis as to why this was happening: on the c1sus machine, the c1sus model would frequently overclock, and then eventually, crash. I observed this behaviour at least 3 times between last night and now. The other models seemed fine though, in fact, IMC stayed locked. Why should this have been the case? It remains to be seen if this was somehow connected to the red DC indicator on c1oaf, though why should this be the case? Isn't the DC just concerned with writing data to frames? Any sort of IPC should be independent? Attachment #2 shows that there's been a definite increase in the maximum time on c1sus clock-cycle since yesterday (it's a 10 day minute trend plot of the model clock cycle timing and also the maximum time). Why? Koji and I did switch off all the Sorensens at the LSC rack for about 30mins, but why should this affect anything at 1X6? There are no red lights in either the c1lsc or c1sus expansion chassis. Curiously, the PRM also seems to be glitchy - as I'm sitting in the control room, I see a spot flashing across vertically on the REFL CRT monitor sporadically. Note that nominally, with PRM misaligned, the REFL CRT should be dark. dmesg on c1sus doesn't shed any light on the issue.

Seems like some high level voodoo indecision.


Edit 330pm: The model just crashed again. dmesg rather unhelpfully just says "ADC timeout". Unclear how to debug further. See Attachment #3.

Quote:

This required multiple hard reboots, but seems like all the RT models are back for now. The only indicator I can't explain is the red DC field on c1oaf. Also, the SUS model seems to be overclocking more frequently than usual, though I can't be sure. The "timing" field of this model's state word is RED, while the other models all seem fine. Not sure what could be going on.

Will debug further tomorrow, when I probably will have to do all this again as I'll need to recompile c1lsc for the ALS electronics test with the new ADC card from the differential AA board.

Attachment 1: CDS-recovery.png  449 kB  | Hide | Hide all
CDS-recovery.png
Attachment 2: c1sus_timing.png  23 kB  | Hide | Hide all
c1sus_timing.png
Attachment 3: c1sus_crashed.png  573 kB  | Hide | Hide all
c1sus_crashed.png
ELOG V3.1.3-