As noted by Steve, the RFM network was down this morning. I noticed that c1susvme1 sync counter was pegged at 16384, so I decided to start with reboots in that viscinity.
After power cycling crates containing c1sosvme, c1susvme1, and c1susvme2 (since the reset buttons didn't work) only c1sosvme and c1susvme2 came back normally. I hooked up a monitor and keyboard to c1susvme1, but saw nothing. I power cycled the c1susvme crate again, and this time I watched it boot properly. I'm not sure why it failed the first time.
The RFM network is now operating normally. I have re-enabled the watchdogs again after having turned them off for the reboots. Steve and I also re-enabled the ITMY coil drivers when I noticed them not damping once the watch dogs were re-enabled. The manual switches had been set to disabled, so we re-enabled them. |