For some reason this afternoon we've been experiencing a lot of problems with the framebuilder, and with the CDS network in general. The framebuilder has been very unresponsive, although the daqd logs seem to indicate that things are ok. All models will loose contact with fb for very long stretches. Attempts to kill/restart daqd don't seem to fix the problem.
These problems seem to be associated with the general CDS network issues as well. The network seems to become very slow, and the workstations all become very slow. The later I assume is because of the network and that so much of the work we do is on network mounted filesystems (/opt/rtcds, /ligo, etc.).
My current speculation is that daqd on fb is doing something stupid, like trying to read or write a bunch of stuff from /frames, which is also network mounted, and that clogs up the entire network. Some serious network debugging is going to be needed to figure out what's going on, though.
I'm afraid daqd is caught in some bad state now, though. It's not responding to anything, and every attempt to kill it seems to bring it back into the bad state. Hopefully I can get Alex to help me figure out what's going on tomorrow. Maybe it will clear up on it's own tonight...