40m QIL Cryo_Lab CTN SUS_Lab TCS_Lab OMC_Lab CRIME_Lab FEA ENG_Labs OptContFac Mariner WBEEShop
  40m Log  Not logged in ELOG logo
Entry  Wed Jul 19 08:37:21 2017, Jamie, Update, CDS, Update on front-end/DAQ rebuild  
    Reply  Wed Jul 19 14:26:50 2017, Jamie, Update, CDS, Update on front-end/DAQ rebuild  
    Reply  Fri Jul 21 18:03:17 2017, Jamie, Update, CDS, Update on front-end/DAQ rebuild  
       Reply  Sun Jul 23 22:16:55 2017, Jamie, gautam, Update, CDS, front-end now running with new OS, RCG 2017-07-23-210810_1394x488_scrot.png2017-07-23-211812_387x488_scrot.png
          Reply  Mon Jul 24 10:45:23 2017, gautam, Update, CDS, c1iscex models died c1iscexFailure.png
             Reply  Mon Jul 24 10:59:08 2017, Jamie, Update, CDS, c1iscex models died 
          Reply  Mon Jul 24 19:28:55 2017, Jamie, Update, CDS, front end MX stream network working, glitches in c1ioo fixed 48.png
             Reply  Mon Jul 24 19:57:54 2017, gautam, Update, CDS, IMC locked, Autolocker re-enabled 
             Reply  Wed Jul 26 19:13:07 2017, Jamie, Update, CDS, daqd showing same instability as before 
                Reply  Fri Jul 28 20:22:41 2017, Jamie, Update, CDS, possible stable daqd configuration with separate DC and FW 
                   Reply  Mon Jul 31 15:13:24 2017, gautam, Update, CDS, FB ---> FB1 
                   Reply  Mon Jul 31 18:44:40 2017, Jamie, Update, CDS, CDS system essentially fully recovered 02.png
                      Reply  Thu Aug 3 19:46:27 2017, Jamie, Update, CDS, new daqd restart procedure 
                      Reply  Fri Aug 4 09:07:28 2017, rana, Update, CDS, CDS system essentially NOT fully recovered 
                         Reply  Thu Aug 10 14:25:52 2017, gautam, Update, CDS, Slow EPICS channels -> Frames re-enabled 
                            Reply  Fri Aug 11 00:10:03 2017, gautam, Update, CDS, Slow EPICS channels -> Frames re-enabled 
                               Reply  Fri Aug 11 11:14:24 2017, gautam, Update, CDS, Slow EPICS channels -> Frames re-enabled 
                               Reply  Fri Aug 11 18:53:35 2017, gautam, Update, CDS, Slow EPICS channels -> Frames re-enabled 
                      Reply  Fri Aug 11 19:34:49 2017, Jamie, Update, CDS, CDS final bits status update 
Message ID: 13127     Entry time: Wed Jul 19 14:26:50 2017     In reply to: 13125
Author: Jamie 
Type: Update 
Category: CDS 
Subject: Update on front-end/DAQ rebuild  

 

Quote:

After the catastrophic fb disk failure last week we lost essentially the entire front end system (not any of the userapp code, but the front end boot server, operating system, and DAQ).  The fb disk was entirely unrecoverable, so we've been trying to rebuild everything from the bits and pieces lying around, and some disks that Keith Thorne sent from LLO.  We're trying to get the front ends working first, and will work on recovering daqd after.

Luckily, fb1, which was being configured as an fb replacement, is mostly fully configured, including having a copy of the front end diskless root image.  We setup fb1 as the new boot server, and were able to get front ends booting again.  Unfortunately, we've been having trouble running and building models, so something is still amis.  We've been taking a three-pronged approach to getting the front ends running:

  • /diskless/root.fb: This involves booting the front ends from the backup of the diskless root from fb.  Runs gentoo kernel 2.6.34.1.  This should correspond to the environment that all models were built and running against.  But something is missing in the configuration.  The front ends were also mounting /opt from fb, which included the dolphin drivers, and we don't have a copy of that, so models aren't loading or recompiling.
  • /diskless/root.x1boot: Keith sent a disk image of the entire x1boot server from LLO.  It uses gentoo kernel 3.0.8.  This ostensibly includes everything we should need to run the front ends, but it's unfortunately configured with newer versions of some of the software and also isn't loading our existing models or building new ones.  This also seems to be having issues with the dolphin drivers.
  • /diskless/root.jessie: This is an entirely new boot image build from scratch with Debian jessie, using an RTS-patched 3.2 kernel.  This would use the latest versions of everything.  It's mostly working, we just need to rebuild the dolphin driver and source.

It seems that in all cases we need to rebuild the dolphin drivers from source.

To clarify, we're able to boot the x1boot image with the existing 2.6.25 kernel that we have from fb.  The issue with the root.x1boot image is not the kernel version but some of the other support libraries, such as dolphin.

ELOG V3.1.3-