Last night around 1AM, many of the the frontend models crashed due to an ADC timeout. (But none of the IOPs, and all the c1lsc models were fine.)
First, on c1sus (Wed Aug 5 00:56:46 PDT 2015)
[1502036.695639] c1rfm: ADC TIMEOUT 0 46281 9 46153
[1502036.945259] c1pem: ADC TIMEOUT 0 56631 55 56695
[1502036.965969] c1mcs: ADC TIMEOUT 1 56706 2 56770
[1502036.965971] c1sus: ADC TIMEOUT 1 56706 2 56770
Then, simultaneously on c1ioo, c1iscex, and c1iscey. (Wed Aug 5 01:10:53 PDT 2015)
[1509007.391124] c1ioo: ADC TIMEOUT 0 46329 57 46201
[1509007.702792] c1als: ADC TIMEOUT 1 63128 24 63192
[2448096.252002] c1scx: ADC TIMEOUT 0 46293 21 46165
[2448096.258001] c1asx: ADC TIMEOUT 0 46669 13 46541
[1674945.583003] c1scy: ADC TIMEOUT 0 46297 25 46169
[1674945.685002] c1tst: ADC TIMEOUT 0 52993 1 52865
I'm still working on getting things back up and running. Just restarting models wasn't working, so I'm trying some soft reboots...
UPDATE: A soft reboot of all frontends seems to have worked, |