40m
QIL
Cryo_Lab
CTN
SUS_Lab
TCS_Lab
OMC_Lab
CRIME_Lab
FEA
ENG_Labs
OptContFac
Mariner
WBEEShop
|
40m Log |
Not logged in |
 |
|
Wed Jul 19 08:37:21 2017, Jamie, Update, CDS, Update on front-end/DAQ rebuild
|
Wed Jul 19 14:26:50 2017, Jamie, Update, CDS, Update on front-end/DAQ rebuild
|
Fri Jul 21 18:03:17 2017, Jamie, Update, CDS, Update on front-end/DAQ rebuild
|
Sun Jul 23 22:16:55 2017, Jamie, gautam, Update, CDS, front-end now running with new OS, RCG 
|
Mon Jul 24 10:45:23 2017, gautam, Update, CDS, c1iscex models died
|
Mon Jul 24 10:59:08 2017, Jamie, Update, CDS, c1iscex models died
|
Mon Jul 24 19:28:55 2017, Jamie, Update, CDS, front end MX stream network working, glitches in c1ioo fixed
|
Mon Jul 24 19:57:54 2017, gautam, Update, CDS, IMC locked, Autolocker re-enabled
|
Wed Jul 26 19:13:07 2017, Jamie, Update, CDS, daqd showing same instability as before
|
Fri Jul 28 20:22:41 2017, Jamie, Update, CDS, possible stable daqd configuration with separate DC and FW
|
Mon Jul 31 15:13:24 2017, gautam, Update, CDS, FB ---> FB1
|
Mon Jul 31 18:44:40 2017, Jamie, Update, CDS, CDS system essentially fully recovered
|
Thu Aug 3 19:46:27 2017, Jamie, Update, CDS, new daqd restart procedure
|
Fri Aug 4 09:07:28 2017, rana, Update, CDS, CDS system essentially NOT fully recovered
|
Thu Aug 10 14:25:52 2017, gautam, Update, CDS, Slow EPICS channels -> Frames re-enabled
|
Fri Aug 11 00:10:03 2017, gautam, Update, CDS, Slow EPICS channels -> Frames re-enabled
|
Fri Aug 11 11:14:24 2017, gautam, Update, CDS, Slow EPICS channels -> Frames re-enabled
|
Fri Aug 11 18:53:35 2017, gautam, Update, CDS, Slow EPICS channels -> Frames re-enabled
|
Fri Aug 11 19:34:49 2017, Jamie, Update, CDS, CDS final bits status update
|
|
Message ID: 13197
Entry time: Fri Aug 11 18:53:35 2017
In reply to: 13189
|
Author: |
gautam |
Type: |
Update |
Category: |
CDS |
Subject: |
Slow EPICS channels -> Frames re-enabled |
|
|
Quote: |
Seems like something has failed after I did this - full frames are no longer on Aug 10 being written since ~2.30pm PDT. I found out when I tried to download some of the free-swinging MC1 data.
To clarify, I logged into fb1, and ran sudo systemctl restart daqd_*. The only change I made was to uncomment the line quoted below in the master file.
Looking at the log using systemctl, I see the following (I just tried restarting the daqd processes again):
Aug 11 00:00:31 fb1 daqd_fw[16149]: LDASUnexpected::unexpected: Caught unexpected exception " This is a bug. Please log an LDAS problem report including this message .
Aug 11 00:00:31 fb1 daqd_fw[16149]: daqd_fw: LDASUnexpected.cc:131: static void LDASTools::Error::LDASUnexpected::unexpected(): Assertion `false' failed.
Aug 11 00:00:32 fb1 systemd[1]: daqd_fw.service: main process exited, code=killed, status=6/ABRT
Aug 11 00:00:32 fb1 systemd[1]: Unit daqd_fw.service entered failed state.
Aug 11 00:00:32 fb1 systemd[1]: daqd_fw.service holdoff time over, scheduling restart.
Aug 11 00:00:32 fb1 systemd[1]: Stopping Advanced LIGO RTS daqd frame writer...
Aug 11 00:00:32 fb1 systemd[1]: Starting Advanced LIGO RTS daqd frame writer...
Aug 11 00:00:32 fb1 systemd[1]: daqd_fw.service start request repeated too quickly, refusing to start.
Aug 11 00:00:32 fb1 systemd[1]: Failed to start Advanced LIGO RTS daqd frame writer.
Aug 11 00:00:32 fb1 systemd[1]: Unit daqd_fw.service entered failed state.
Oddly, I am able to access second trends for the same channels from the past which will be useful for the MC1 debugging). Not sure whats going on.
The live data grabbing using cdsutils still seems to be working though - so I've kicked MC1 again, and am grabbing 2 hours of data live on Pianosa.
|
So we tried this again with a fresh build of daqd_fw, and it still fails. The error message is pointing to an underlying bug in the framecpp library ("LDASTools"), which may be tricky to solve. I'm rustling the appropriate bushes... |