c1iscex is dead again. Red lights, no "breathing" on the FE status screen.
The c1iscex machine itself wasn't dead, the models were just not running. Here are the last messages in dmesg:
[130432.926002] c1spx: ADC TIMEOUT 0 7060 20 7124
[130432.926002] c1scx: ADC TIMEOUT 0 7060 20 7124
[130433.941008] c1x01: timeout 0 1000000
[130433.941008] c1x01: exiting from fe_code()
I'm guessing maybe the timing signal was lost, so the ADC stopped clocking. Since the ADC clock is the everything clock, all the "fe" code (ie. models) aborted. Not sure what would have caused it.
I restarted all the models ("rtcds restart all") and everything came up fine. Obviously we should keep our eyes on things, and note if anything strange was happening if this happens again.