40m QIL Cryo_Lab CTN SUS_Lab TCS_Lab OMC_Lab CRIME_Lab FEA ENG_Labs OptContFac Mariner WBEEShop
  40m Log  Not logged in ELOG logo
Entry  Tue May 8 11:41:16 2018, gautam, Update, General, IFO maintenance ITMX_stuck.png
    Reply  Wed May 9 17:30:04 2018, gautam, Update, General, Input beam misaligned InputBeamGone.pngTTpointing.png
       Reply  Wed May 9 19:51:07 2018, gautam, Update, General, Input beam misaligned DACweirdness.pngDACerror.png
          Reply  Sun May 13 15:15:18 2018, gautam, Update, General, CDS crash vertexFEs_crashed.png
             Reply  Sun May 13 17:31:51 2018, gautam, Update, General, CDS crash CDS_overview_20180513.pngAS_1210293643.jpeg
                Reply  Sun May 13 20:48:38 2018, johannes, Update, General, CDS crash 
                Reply  Thu May 17 11:56:37 2018, gautam, Update, General, EPICS process died on c1ioo 
                   Reply  Thu May 24 10:16:29 2018, gautam, Update, General, All models on c1lsc frontend crashed c1lsc_crashed.png
    Reply  Thu May 10 08:45:16 2018, Steve, Update, General, 4.5M eq. Cabazon, CA Cabazon4.5m79m.png4.5Meq.png
       Reply  Thu May 10 11:38:19 2018, gautam, Update, General, ITMY UL ITMY_UL.pdf
       Reply  Wed Aug 22 08:44:09 2018, Steve, Update, General, earth quake yesterday_EQs.png
          Reply  Fri Aug 24 08:04:37 2018, Steve, Update, General, small earth quake small_EQ.png
             Reply  Mon Aug 27 09:14:45 2018, Steve, Update, PEM, small earth quakes  small_EQs_vs_SUSs.png
                Reply  Tue Nov 27 10:50:20 2018, Steve, Update, PEM, earth quake Mexico 5.5M.Mexico.png
                   Reply  Thu Nov 29 08:13:33 2018, Steve, Update, PEM, EQ 3.9m So CA 3.9mSoCA.pngVac_as_today.pngas_is.png
          Reply  Wed Aug 29 09:20:27 2018, Steve, Update, SUS, local 4.4M earth quake 4.4_La_Verne.png3.4_&_4.4M_EQ.png
             Reply  Wed Aug 29 11:46:27 2018, Jon, Update, SUS, local 4.4M earth quake 
             Reply  Thu Sep 20 08:17:14 2018, Steve, Update, SUS, local 3.4M earth quake local_3.4M.png
Message ID: 13828     Entry time: Wed May 9 19:51:07 2018     In reply to: 13827     Reply to this: 13837
Author: gautam 
Type: Update 
Category: General 
Subject: Input beam misaligned 

As suspected - the problem was with the TTs. I tested the TT signal chain by driving a low frequency sine wave using AWG and looking at the signal on an o'scope. But I saw nothing, neither at the AI board monitor point, nor at the actual coil driver mon point. I decided to look at the IOP testpoints for the DAC channels, to see if the signals were going through okay on the digital side. But the IOP channels were flatlined, as viewed on dataviewer (see Attachment #1). This despite the fact that the DAC output monitor screen in the model itself was showing some sensible numbers, again see Attachment #1.

Looking at the CDS overview screen, there were no red flags. But there was a red indicator sneakily hidden away in the IOP model's CDS status screen, the "DAC" field in the state word is red. As Attachment #2 shows, a change in the state word is correlated with the time ASDC went to 0.

Note that there are also no errors on the c1lsc frontend itself, judging by dmesg. I want to do a model restart, but (i) this will likely require reboots of all vertex FEs and (ii) I want to know if any CDS experts want to sniff for clues to what's going on before a model restart wipes out some secret logfiles. I'm a little confused that the rtcds isn't throwing up any errors and causing models to crash if the values are not being written to the registers of the DAC. It may also be that the DAC card itself is dead sad. To re-iterate, all the EPICS readbacks were suggesting that I am injecting a signal right up to the IOP.

Quoting from the runtime diagnostics note:

NOTE: As V2.7, if this error is detected, the IOP will output zero values to all DAC modules, as a protective measure. Only method to clear this is to restart the IOP and all applications on that computer
Attachment 1: DACweirdness.png  1.445 MB  Uploaded Wed May 9 21:04:07 2018  | Hide | Hide all
DACweirdness.png
Attachment 2: DACerror.png  29 kB  Uploaded Wed May 9 21:04:25 2018  | Hide | Hide all
DACerror.png
ELOG V3.1.3-