40m QIL Cryo_Lab CTN SUS_Lab TCS_Lab OMC_Lab CRIME_Lab FEA ENG_Labs OptContFac Mariner WBEEShop
  40m Log  Not logged in ELOG logo
Entry  Wed Jul 19 08:37:21 2017, Jamie, Update, CDS, Update on front-end/DAQ rebuild  
    Reply  Wed Jul 19 14:26:50 2017, Jamie, Update, CDS, Update on front-end/DAQ rebuild  
    Reply  Fri Jul 21 18:03:17 2017, Jamie, Update, CDS, Update on front-end/DAQ rebuild  
       Reply  Sun Jul 23 22:16:55 2017, Jamie, gautam, Update, CDS, front-end now running with new OS, RCG 2017-07-23-210810_1394x488_scrot.png2017-07-23-211812_387x488_scrot.png
          Reply  Mon Jul 24 10:45:23 2017, gautam, Update, CDS, c1iscex models died c1iscexFailure.png
             Reply  Mon Jul 24 10:59:08 2017, Jamie, Update, CDS, c1iscex models died 
          Reply  Mon Jul 24 19:28:55 2017, Jamie, Update, CDS, front end MX stream network working, glitches in c1ioo fixed 48.png
             Reply  Mon Jul 24 19:57:54 2017, gautam, Update, CDS, IMC locked, Autolocker re-enabled 
             Reply  Wed Jul 26 19:13:07 2017, Jamie, Update, CDS, daqd showing same instability as before 
                Reply  Fri Jul 28 20:22:41 2017, Jamie, Update, CDS, possible stable daqd configuration with separate DC and FW 
                   Reply  Mon Jul 31 15:13:24 2017, gautam, Update, CDS, FB ---> FB1 
                   Reply  Mon Jul 31 18:44:40 2017, Jamie, Update, CDS, CDS system essentially fully recovered 02.png
                      Reply  Thu Aug 3 19:46:27 2017, Jamie, Update, CDS, new daqd restart procedure 
                      Reply  Fri Aug 4 09:07:28 2017, rana, Update, CDS, CDS system essentially NOT fully recovered 
                         Reply  Thu Aug 10 14:25:52 2017, gautam, Update, CDS, Slow EPICS channels -> Frames re-enabled 
                            Reply  Fri Aug 11 00:10:03 2017, gautam, Update, CDS, Slow EPICS channels -> Frames re-enabled 
                               Reply  Fri Aug 11 11:14:24 2017, gautam, Update, CDS, Slow EPICS channels -> Frames re-enabled 
                               Reply  Fri Aug 11 18:53:35 2017, gautam, Update, CDS, Slow EPICS channels -> Frames re-enabled 
                      Reply  Fri Aug 11 19:34:49 2017, Jamie, Update, CDS, CDS final bits status update 
Message ID: 13149     Entry time: Fri Jul 28 20:22:41 2017     In reply to: 13145     Reply to this: 13152   13153
Author: Jamie 
Type: Update 
Category: CDS 
Subject: possible stable daqd configuration with separate DC and FW 

This week Jonathan Hanks and I have been trying to diagnose why the daqd has been unstable in the configuration used by the 40m, with data concentrator (dc) and frame writer (fw) in the same process (referred to generically as 'fb').  Jonathan has been digging into the core dumps and source to try to figure out what's going on, but he hasn't come up with anything concrete yet.

As an alternative, we've started experimenting with a daqd configuration with the dc and fw components running in separate processes, with communication over the local loopback interface.  The separate dc/fw process model more closely matches the configuration at the sites, although the sites put dc and fwprocesses on different physical machines.  Our experimentation thus far seems to indicate that this configuration is stable, although we haven't yet tested it with the full configuration, which is what I'm attempting to do now.

Unfortunately I'm having trouble with the mx_stream communication between the front ends and the dc process.  The dc does not appear to be receiving the streams from the front ends and is producing a '0xbad' status message for each.  I'm investigating.

ELOG V3.1.3-