Alberto, Kiwamu, Koji,
this morning we found the RFM network and all the front-ends down.
To fix the problem, we first tried a soft strategy, that is, we tried to restart CODAQCTRL and C1DCUEPICS alone, but it didn't work.
We then went for a big bootfest. We first powered off fb40m, C1DCUEPICS, CODAQCTRL, reset the RFM Network switch. Then we rebooted them in the same order in which we turned them off.
Then we power cycled and restarted all the front-ends.
Finally we restored all the burt snapshots to Monday Dec 7th at 20:00. |