40m
QIL
Cryo_Lab
CTN
SUS_Lab
TCS_Lab
OMC_Lab
CRIME_Lab
FEA
ENG_Labs
OptContFac
Mariner
WBEEShop
|
40m Log |
Not logged in |
 |
|
Sun Aug 5 13:28:43 2018, gautam, Update, CDS, c1lsc flaky
|
Mon Aug 6 00:26:21 2018, gautam, Update, CDS, More CDS woes
|
Mon Aug 6 14:38:38 2018, gautam, Update, CDS, More CDS woes 
|
Mon Aug 6 19:49:09 2018, gautam, Update, CDS, More CDS woes
|
Tue Aug 7 11:30:46 2018, gautam, Update, CDS, More CDS woes
|
Tue Aug 7 22:28:23 2018, gautam, Update, CDS, More CDS woes
|
Wed Aug 8 23:03:42 2018, gautam, Update, CDS, c1lsc model started
|
Thu Aug 9 12:31:13 2018, gautam, Update, CDS, CDS status update
|
Wed Aug 15 21:27:47 2018, gautam, Update, CDS, CDS status update
|
Tue Sep 4 10:14:11 2018, gautam, Update, CDS, CDS status update
|
Wed Sep 5 10:59:23 2018, wgautam, Update, CDS, CDS status update
|
Thu Sep 6 14:21:26 2018, gautam, Update, CDS, ADC replacement in c1lsc expansion chassis
|
Fri Sep 7 12:35:14 2018, gautam, Update, CDS, ADC replacement in c1lsc expansion chassis
|
Mon Sep 10 12:44:48 2018, Jon, Update, CDS, ADC replacement in c1lsc expansion chassis
|
Thu Sep 20 11:29:04 2018, gautam, Update, CDS, New PCIe fiber housed 
|
Thu Sep 20 16:19:04 2018, gautam, Update, CDS, New PCIe fiber install postponed to tomorrow
|
Fri Sep 21 16:46:38 2018, gautam, Update, CDS, New PCIe fiber installed and routed 
|
|
Message ID: 14193
Entry time: Wed Sep 5 10:59:23 2018
In reply to: 14192
Reply to this: 14194
|
Author: |
wgautam |
Type: |
Update |
Category: |
CDS |
Subject: |
CDS status update |
|
|
Rolf came by today morning. For now, we've restarted the FE machine and the expansion chassis (note that the correct order in which to do this is: turn off computer--->turn off expansion chassis--->turn on expansion chassis--->turn on computer). The debugging measures Rolf suggested are (i) to replace the old generation ADC card in the expansion chassis which has a red indicator light always on and (ii) to replace the PCIe fiber (2010 make) running from the c1lsc front-end machine in 1X6 to the expansion chassis in 1Y3, as the manufacturer has suggested that pre-2012 versions of the fiber are prone to failure. We will do these opportunistically and see if there is any improvement in the situation.
Another tip from Rolf: if the c1lsc FE is responsive but the models have crashed, then doing sudo reboot by ssh-ing into c1lsc should suffice* (i.e. it shouldn't take down the models on the other vertex FEs, although if the FE is unresponsive and you hard reboot it, this may still be a problem). I'll modify I've modified the c1lsc reboot script accordingly.
* Seems like this can still lead to the other vertex FEs crashing, so I'm leaving the reboot script as is (so all vertex machines are softly rebooted when c1lsc models crash).
Quote: |
c1lsc crashed again. I've contacted Rolf/JHanks for help since I'm out of ideas on what can be done to fix this problem.
|
|