40m QIL Cryo_Lab CTN SUS_Lab TCS_Lab OMC_Lab CRIME_Lab FEA ENG_Labs OptContFac Mariner WBEEShop
  40m Log  Not logged in ELOG logo
Entry  Sun Aug 5 13:28:43 2018, gautam, Update, CDS, c1lsc flaky 
    Reply  Mon Aug 6 00:26:21 2018, gautam, Update, CDS, More CDS woes CDSoverview.png
       Reply  Mon Aug 6 14:38:38 2018, gautam, Update, CDS, More CDS woes CDScrash.pngMC1failures.png
          Reply  Mon Aug 6 19:49:09 2018, gautam, Update, CDS, More CDS woes IMG_7106.JPG
             Reply  Tue Aug 7 11:30:46 2018, gautam, Update, CDS, More CDS woes 
                Reply  Tue Aug 7 22:28:23 2018, gautam, Update, CDS, More CDS woes 
                   Reply  Wed Aug 8 23:03:42 2018, gautam, Update, CDS, c1lsc model started 
                      Reply  Thu Aug 9 12:31:13 2018, gautam, Update, CDS, CDS status update 
                         Reply  Wed Aug 15 21:27:47 2018, gautam, Update, CDS, CDS status update 
                            Reply  Tue Sep 4 10:14:11 2018, gautam, Update, CDS, CDS status update 
                               Reply  Wed Sep 5 10:59:23 2018, wgautam, Update, CDS, CDS status update 
                                  Reply  Thu Sep 6 14:21:26 2018, gautam, Update, CDS, ADC replacement in c1lsc expansion chassis CDSoverview.png
                                     Reply  Fri Sep 7 12:35:14 2018, gautam, Update, CDS, ADC replacement in c1lsc expansion chassis Screenshot_from_2018-09-07_12-34-52.png
                                        Reply  Mon Sep 10 12:44:48 2018, Jon, Update, CDS, ADC replacement in c1lsc expansion chassis 
                                     Reply  Thu Sep 20 11:29:04 2018, gautam, Update, CDS, New PCIe fiber housed PCIeFiberSwap.pngPCIeFiberSwap_FBrebooted.png
                                        Reply  Thu Sep 20 16:19:04 2018, gautam, Update, CDS, New PCIe fiber install postponed to tomorrow 
                                           Reply  Fri Sep 21 16:46:38 2018, gautam, Update, CDS, New PCIe fiber installed and routed PCIeFiber.pngIMG_5878.JPG
Message ID: 14136     Entry time: Mon Aug 6 00:26:21 2018     In reply to: 14133     Reply to this: 14139
Author: gautam 
Type: Update 
Category: CDS 
Subject: More CDS woes 

I spent most of today fighting various CDS errors.

  • I rebooted c1lsc around 3pm, my goal was to try and do some vertex locking and figure out what the implications were of having only ~30% power we used to have at the AS port.
  • Shortly afterwards (~4pm), c1lsc crashed.
  • Using the reboot script, I was able to bring everything back up. But the DC lights on c1sus models were all red, and a 0x4000 error was being reported.
  • This error is indicative of some timing issue, but all the usual tricks (reboot vertex FEs in various order, restart the mx_streams etc) didn't clear this error.
  • I checked the Tempus GPS unit, that didn't report any obvious problems (i.e. front display was showing the correct UTC time).
  • Finally, I decided to shut down all watchdogs, soft reboot all the FEs, soft reboot FB, power cycle all expansion chassis.
  • This seems to have done the trick - I'm leaving c1oaf disabled for now.
  • The remaining red indicators are due to c1dnn and c1oaf being disabled.

Let's see how stable this configuration is. Onto some locking now...

Attachment 1: CDSoverview.png  93 kB  Uploaded Mon Aug 6 01:31:46 2018  | Show | Show all
ELOG V3.1.3-