When I came to the 40m, I found most of the FB signals are dead.
The suspensions were not dumped but not too much excited. Use watchdog switches to cut off the coil actuators.
Restarted mxstream from the CDS_FE_STATUS screen. The c1lsc processes got fine. But the FB indicators for c1sus, c1ioo, c1iscex/y are still red.
Sshed into c1sus/ioo, run rtcds restart all . This made them came back under control.
Same treatment for c2iscex and c1iscey. This made c1sus stall again. Also c1iscey did not come back.
At this point I decided to kill all of the rt processes on c1sus/c1ioo/c1iscex/c1iscey to avoid interference between them.
And started to restart from the end machines.
c1iscex did not come back by rtcds restart all.
Run lsmod on c1iscey and found c1x05 persisted stay on the kernel. rmmod did not remove the c1x05 module.
Run software reboot of c1iscey. => c1iscey came back online.
c1iscey did not come back by rtcds restart all.
Run software reboot of c1iscex. => c1iscex came back online.
c1ioo just came back by rtcds restart all.
c1sus did not come back by rtcds restart all.
Run software reboot of c1sus => c1sus came back online.
This series of restarting made the fb connections of some of the c1lsc processes screwed up.
Run the following restarting commands => all of the process are running with FB connection.
rtcds restart c1sup
rtcds restart c1ass
rtcds restart c1lsc
Enable damping loops by reverting the watchdog switches.
All of the FE status are green except for the c1rfm bit 2 (GE FANUC RFM CARD 0).