40m QIL Cryo_Lab CTN SUS_Lab CAML OMC_Lab CRIME_Lab FEA ENG_Labs OptContFac Mariner WBEEShop
  40m elog  Not logged in ELOG logo
Entry  Thu Jul 16 01:04:21 2015, ericq, Update, General, Starting IFO recovery, DAC troubles 
    Reply  Thu Jul 16 11:18:37 2015, jamie, Update, General, Starting IFO recovery, DAC troubles 
       Reply  Thu Jul 16 16:46:18 2015, ericq, Update, General, Starting IFO recovery, DAC troubles 
          Reply  Sat Jul 18 14:55:33 2015, jamie, Update, General, all front ends back up and running cds-good.png
Message ID: 11420     Entry time: Thu Jul 16 11:18:37 2015     In reply to: 11418     Reply to this: 11422
Author: jamie 
Type: Update 
Category: General 
Subject: Starting IFO recovery, DAC troubles 
Quote:

I've been trying to start recovering IFO functionality, but quickly hit a frustrating roadblock. 

Upon opening the PSL shutter, and deactivating the MC mirror watchdogs, I saw the MC reflected beam moving way more than normal. 

A series of investigations revealed no signals coming out of c1sus's DAC.  crying

The IOP (c1x02) shows two of its DAC-related statewords (DAC and DK) in a fault mode, which means (quoting T1100625):

"As of RCG V2.7, if an error is detected in oneor more DAC modules, the IOP will continue to run but only write zero values to the DAC modules as a protective measure. This can only be cleared by restarting the IOP and all applications running on the affected computer."

The offending card may be DAC1, which has its fourth bit red even with only the IOP running, which corresponds to a "FIFO error". /proc/c1x02/status states, in part:

DAC #0 16-bit fifo_status=2 (OK)
DAC #1 16-bit fifo_status=3 (empty)
DAC #2 16-bit fifo_status=2 (OK)

Squishing cables and restarting the frontend have not helped anything. 

c1lsc, c1isce[x/y] are not suffering from this problem, and appear to be happily using their DACs. c1ioo does not use any DAC channels. 

We need to update the indicators on the CDS_FE_STATUS screen to expose the new indicators, so that we have better visibility for these issues.

I'm not sure why this DAC is failing. It may indicate an actual problem with the DAC itself.

Quote:

As a further headache, any time I restart any of the models on the c1sus frontend, the BURT restore is totally bunk. Moreover, using burtgooey to restore a good snapshot to the c1sus model triggers a timing overflow and model crash, maybe not so surprising since the model seems to be averaging ~56usec or so. 

This is related to changes to how the front ends load their safe.snaps. I think they're now explictly expecting the file:

targtet/<model>/<model>epics/burt/safe.snap

I'll come over this afternoon and we can get acquainted with the new SDF system that now handles management of the safe.snap files.

ELOG V3.1.3-