Test points were unavailable last night, even after reboots of c1sus and even restarting the daqd process on the frame builder.
Its unclear at this time. My guess is flaky fb and mx_stream codes. At the moment, the daqd often requires several restarts as it segfaults within a minute or two of restarting it.
What we did (aka treating the symptoms):
We rebooted the frame builder machine. I also added the daqd and nds processes to the inittab. Now when these die, they will automatically be restarted.
Steps to add to the inittab on fb
0) If not on fb, ssh -X fb
1) cd /etc/
2) sudo vi inittab or sudo emacs init
3) Add a line like: id:runlevels:action:process
The id is a unqiue 2-4 letter and number identifier for the process
Run levels is the run level of linux that it will start at. 345 will cover the normal cases
action is what to do with the process. Respawn makes it run at startup and also restarts it everytime it dies.
process is the command you want to run
See "man inittab" for more details
In this case we added
daq:345:respawn:/opt/rtcds/caltech/c1/target/fb/daqd -c /opt/rtcds/caltech/c1/target/fb/daqdrc > /opt/rtcds/caltech/c1/target/fb/daqd.log
nds:345:respawn:/opt/rtcds/caltech/c1/target/fb/nds pipe > /opt/rtcds/caltech/c1/target/fb/nds.log
5) Run "sudo /sbin/telinit q". This forces init to rexamine the inittab file
daqd and nds will now automatically restart when they die.
When the frame builder dies, the mx_stream processes on the front ends die as well. These need to be restarted manually at the moment by using "sudo /etc/restart_streams" while on c1sus.
The framebuilder code shouldn't be this flaky.