40m QIL Cryo_Lab CTN SUS_Lab TCS_Lab OMC_Lab CRIME_Lab FEA ENG_Labs OptContFac Mariner WBEEShop
  40m Log  Not logged in ELOG logo
Message ID: 11479     Entry time: Wed Aug 5 10:56:07 2015
Author: ericq 
Type: Update 
Category: CDS 
Subject: Many models crashed  

Last night around 1AM, many of the the frontend models crashed due to an ADC timeout. (But none of the IOPs, and all the c1lsc models were fine.)

 
First, on c1sus (Wed Aug  5 00:56:46 PDT 2015)
[1502036.695639] c1rfm: ADC TIMEOUT 0 46281 9 46153
[1502036.945259] c1pem: ADC TIMEOUT 0 56631 55 56695
[1502036.965969] c1mcs: ADC TIMEOUT 1 56706 2 56770
[1502036.965971] c1sus: ADC TIMEOUT 1 56706 2 56770

Then, simultaneously on c1ioo, c1iscex, and c1iscey. (Wed Aug  5 01:10:53 PDT 2015)

[1509007.391124] c1ioo: ADC TIMEOUT 0 46329 57 46201
[1509007.702792] c1als: ADC TIMEOUT 1 63128 24 63192

[2448096.252002] c1scx: ADC TIMEOUT 0 46293 21 46165
[2448096.258001] c1asx: ADC TIMEOUT 0 46669 13 46541

[1674945.583003] c1scy: ADC TIMEOUT 0 46297 25 46169
[1674945.685002] c1tst: ADC TIMEOUT 0 52993 1 52865

I'm still working on getting things back up and running. Just restarting models wasn't working, so I'm trying some soft reboots...


UPDATE: A soft reboot of all frontends seems to have worked,

Attachment 1: crashes.png  21 kB  | Hide | Hide all
crashes.png
ELOG V3.1.3-