I tried to run the seismic StripTool tonight, which seems liek a simple task. But instead I fell through the rabbit hole again...
The seismic.stp couldn't be run from zita/op340m because zita doesn't have EPICS or MEDM and because the op340m version of StripTool cannot load the new file format in which rossa/pianosa save their files.
Once I got it running by sshing in to rossa, I noticed that the BLRMS trends didn't make any sense. Correctly guessed that this was because all of the BP and LP filters were off. Why? Because of bad BURT, that's why.
As it turns out (if you look through the autoburt logs), several of our FE machines haven't been correctly saving snapshots because of some channel count mismatch between old and new SNAP files. So we are not getting the settings restored correctly for these systems when they get booted. Probably, someone has got to suss out the source of the BURT snap messages; it may be that we have to rehash the snap process after any substantial model rebuild.
For c1pemepics, I did a manual restore from the time when Mirko last ELOG'd that BLRMS was trending OK (Nov 23 @ 3 AM).
Seems like we should also get some kind of auto email warning if the BURTs fail in this way. Otherwise, we'll lose a lot of work if it goes on for weeks.
Bottom line: fix the autoburt so that it doesn't fail after model rebuilds. |