40m QIL Cryo_Lab CTN SUS_Lab TCS_Lab OMC_Lab CRIME_Lab FEA ENG_Labs OptContFac Mariner WBEEShop
  40m Log  Not logged in ELOG logo
Entry  Sun Aug 5 13:28:43 2018, gautam, Update, CDS, c1lsc flaky 
    Reply  Mon Aug 6 00:26:21 2018, gautam, Update, CDS, More CDS woes CDSoverview.png
       Reply  Mon Aug 6 14:38:38 2018, gautam, Update, CDS, More CDS woes CDScrash.pngMC1failures.png
          Reply  Mon Aug 6 19:49:09 2018, gautam, Update, CDS, More CDS woes IMG_7106.JPG
             Reply  Tue Aug 7 11:30:46 2018, gautam, Update, CDS, More CDS woes 
                Reply  Tue Aug 7 22:28:23 2018, gautam, Update, CDS, More CDS woes 
                   Reply  Wed Aug 8 23:03:42 2018, gautam, Update, CDS, c1lsc model started 
                      Reply  Thu Aug 9 12:31:13 2018, gautam, Update, CDS, CDS status update 
                         Reply  Wed Aug 15 21:27:47 2018, gautam, Update, CDS, CDS status update 
                            Reply  Tue Sep 4 10:14:11 2018, gautam, Update, CDS, CDS status update 
                               Reply  Wed Sep 5 10:59:23 2018, wgautam, Update, CDS, CDS status update 
                                  Reply  Thu Sep 6 14:21:26 2018, gautam, Update, CDS, ADC replacement in c1lsc expansion chassis CDSoverview.png
                                     Reply  Fri Sep 7 12:35:14 2018, gautam, Update, CDS, ADC replacement in c1lsc expansion chassis Screenshot_from_2018-09-07_12-34-52.png
                                        Reply  Mon Sep 10 12:44:48 2018, Jon, Update, CDS, ADC replacement in c1lsc expansion chassis 
                                     Reply  Thu Sep 20 11:29:04 2018, gautam, Update, CDS, New PCIe fiber housed PCIeFiberSwap.pngPCIeFiberSwap_FBrebooted.png
                                        Reply  Thu Sep 20 16:19:04 2018, gautam, Update, CDS, New PCIe fiber install postponed to tomorrow 
                                           Reply  Fri Sep 21 16:46:38 2018, gautam, Update, CDS, New PCIe fiber installed and routed PCIeFiber.pngIMG_5878.JPG
Message ID: 14193     Entry time: Wed Sep 5 10:59:23 2018     In reply to: 14192     Reply to this: 14194
Author: wgautam 
Type: Update 
Category: CDS 
Subject: CDS status update 

Rolf came by today morning. For now, we've restarted the FE machine and the expansion chassis (note that the correct order in which to do this is: turn off computer--->turn off expansion chassis--->turn on expansion chassis--->turn on computer). The debugging measures Rolf suggested are (i) to replace the old generation ADC card in the expansion chassis which has a red indicator light always on and (ii) to replace the PCIe fiber (2010 make) running from the c1lsc front-end machine in 1X6 to the expansion chassis in 1Y3, as the manufacturer has suggested that pre-2012 versions of the fiber are prone to failure. We will do these opportunistically and see if there is any improvement in the situation.

Another tip from Rolf: if the c1lsc FE is responsive but the models have crashed, then doing sudo reboot by ssh-ing into c1lsc should suffice* (i.e. it shouldn't take down the models on the other vertex FEs, although if the FE is unresponsive and you hard reboot it, this may still be a problem). I'll modify I've modified the c1lsc reboot script accordingly.

* Seems like this can still lead to the other vertex FEs crashing, so I'm leaving the reboot script as is (so all vertex machines are softly rebooted when c1lsc models crash).

Quote:

c1lsc crashed again. I've contacted Rolf/JHanks for help since I'm out of ideas on what can be done to fix this problem.

ELOG V3.1.3-