40m QIL Cryo_Lab CTN SUS_Lab TCS_Lab OMC_Lab CRIME_Lab FEA ENG_Labs OptContFac Mariner WBEEShop
  40m Log  Not logged in ELOG logo
Entry  Thu Jul 16 01:04:21 2015, ericq, Update, General, Starting IFO recovery, DAC troubles 
    Reply  Thu Jul 16 11:18:37 2015, jamie, Update, General, Starting IFO recovery, DAC troubles 
       Reply  Thu Jul 16 16:46:18 2015, ericq, Update, General, Starting IFO recovery, DAC troubles 
          Reply  Sat Jul 18 14:55:33 2015, jamie, Update, General, all front ends back up and running cds-good.png
Message ID: 11426     Entry time: Sat Jul 18 14:55:33 2015     In reply to: 11422
Author: jamie 
Type: Update 
Category: General 
Subject: all front ends back up and running 

After some surgery yesterday the front ends are all back up and running:

  • Eric found that one of the DAC cards in the c1sus front end was not being properly initialized (with the new RCG code).  Turned out that it was an older version DAC, with a daughter board on top of a PCIe board.  We suspected that there was some compatibility issue with that version of the card, so Eric pulled an unused card from c1ioo to replace the one in c1sus.  That worked and now c1sus is running happily.
  • Eric put the old DAC card into c1ioo, but it didn't like it and was having trouble booting.  I removed the card and c1ioo came up fine on it's own.
  • After all front end were back up and running, all RFM connections were dead.  I tracked this down to the RFM switch being off, because the power cable was not fully seated.  This probably happened when Steve was cleaning up the 1X4/5 racks.  I re-powered the RFM switch and all the RFM connections came back on-line
  • All receivers of Dolphin (DIS) "PCIE" IPC signals from c1ioo where throwing errors.  I tracked this down to the Dolphin cable going to c1ioo being plugged in to the wrong port on the c1ioo dolphin card.  I unplugged it and plugged it into the correct port, which of course caused all front end modules using dolphin to crash.  Once I restarted all those models, everything is back:

ELOG V3.1.3-