40m QIL Cryo_Lab CTN SUS_Lab TCS_Lab OMC_Lab CRIME_Lab FEA ENG_Labs OptContFac Mariner WBEEShop
  40m Log  Not logged in ELOG logo
Entry  Thu Jul 16 01:04:21 2015, ericq, Update, General, Starting IFO recovery, DAC troubles 
    Reply  Thu Jul 16 11:18:37 2015, jamie, Update, General, Starting IFO recovery, DAC troubles 
       Reply  Thu Jul 16 16:46:18 2015, ericq, Update, General, Starting IFO recovery, DAC troubles 
          Reply  Sat Jul 18 14:55:33 2015, jamie, Update, General, all front ends back up and running cds-good.png
Message ID: 11418     Entry time: Thu Jul 16 01:04:21 2015     Reply to this: 11420
Author: ericq 
Type: Update 
Category: General 
Subject: Starting IFO recovery, DAC troubles 

I've been trying to start recovering IFO functionality, but quickly hit a frustrating roadblock. 

Upon opening the PSL shutter, and deactivating the MC mirror watchdogs, I saw the MC reflected beam moving way more than normal. 

A series of investigations revealed no signals coming out of c1sus's DAC.  crying

The IOP (c1x02) shows two of its DAC-related statewords (DAC and DK) in a fault mode, which means (quoting T1100625):

"As of RCG V2.7, if an error is detected in oneor more DAC modules, the IOP will continue to run but only write zero values to the DAC modules as a protective measure. This can only be cleared by restarting the IOP and all applications running on the affected computer."

The offending card may be DAC1, which has its fourth bit red even with only the IOP running, which corresponds to a "FIFO error". /proc/c1x02/status states, in part:

DAC #0 16-bit fifo_status=2 (OK)
DAC #1 16-bit fifo_status=3 (empty)
DAC #2 16-bit fifo_status=2 (OK)

Squishing cables and restarting the frontend have not helped anything. 

c1lsc, c1isce[x/y] are not suffering from this problem, and appear to be happily using their DACs. c1ioo does not use any DAC channels. 


As a further headache, any time I restart any of the models on the c1sus frontend, the BURT restore is totally bunk. Moreover, using burtgooey to restore a good snapshot to the c1sus model triggers a timing overflow and model crash, maybe not so surprising since the model seems to be averaging ~56usec or so. 

ELOG V3.1.3-