I checked the situation from my home and the problem was solved.
The main problem was undefined state of the autolocker and the strange undefined switch states, being associated with the bootfest and burtrestore.
- MC UP/DOWN status shows it was up and down. So I ran scripts/MC/mcup and scripts/MC/mcdown. These cleared the MC autolocker status.
- I had a problem handling the FSS. After mcup/mcdown above, I randomly pushed the "enable/disable" buttons and others, and with some reason, it recovered the handling. Actually it acquired the lock autonomously. Kiwamu may have also been working on it at the same time???
- Then, I checked the PSL loop. I disconnected the loop by pushing the "test" button. The DC slider changes the PZT voltage only 0~+24V. This is totally strange and I started pushing the buttons randomly. As soon as I pushed the "BLANK"/"NORMAL" button, the PZT output got back under the control.
- Then I locked the PMC, MZ, and MC as usual.
Alberto: You must be careful as the modulations were restored.
It's been an iffy last few hours here at the 40m. Kiwamu, Koji and I were all sitting at our desks, and the computers / RFM network decided to crash. We brought all of the computers back, but now the RefCav and PMC don't want to lock. I'm a wee bit confused by this. Both Kiwamu and I have given it a shot, and we can each get the ref cav to sit and flash, but we can't catch it. Also, when I bring the PMC slider rail to rail, we see no change in the PMC refl camera. Since c1psl had been finicky coming back the first time, I tried soft rebooting, and then keying the crate again, but the symptoms remained the same. Also, I tried burt restoring to several different times in the last few days, to see if that helped. It didn't. I did notice that MC2 was unhappy, which was a result of the burtrestores setting the MCL filters as if the cavity were locked, so I manually ran mcdown. Also, the MC autolocker script had died, so Kiwamu brought it back to life.
Since we've spent an hour on trying to relock the PSL cavities (the descriptive word I'm going to suggest for us is persistent, not losers), we're giving up in favor of waiting for expert advice in the morning. I suppose there's something obvious that we're missing, but we haven't found it yet......
This is a (sort of) known problem with the EPICS computers: it's generally called the 'sticky slider' problem, but of course it applies to buttons as well. It happens after a reboot, when the MEDM control/readback values don't match the actual applied voltages. The solution (so far) is just to `twiddle' the problematic sliders/button. There's a script somewhere called slider_twiddle that does this, but I don't remember if it has PSL stuff in it. A better solution is probably to have an individual slider twiddle script for each target machine, and add the running of that script to the reboot ritual in the wiki.