The framebuilder RAID is back online. The disk had been mounted read-only (see below) so daqd couldn't write frames, which was in turn causing it to segfault immediately, so it was constantly restarting.
The jetstor RAID unit itself has a dead power supply. This is not fatal, since it has three. It has three so it can continue to function if one fails. I have removed the bad supply and gave it to Steve so he can get a suitable replacement.
Some recovery had to be done on fb to get everything back up and running again. I ran into issues trying to do it on the fly, so I eventually just rebooted. It seemed to come back ok, except for something going on with daqd. It was reporting the following error upon restart:
[Tue Mar 12 11:43:54 2013] main profiler warning: 0 empty blocks in the buffer
It was spitting out this message about once a second, until eventually the daqd died. When it restarted it seemed to come back up fine. I'm not exactly clear what those messages were about, but I think it has something to do with not being able to dump it's data buffers to disk. I'm guessing that this was a residual problem from the umounted /frames, which somehow cleared on it's own. Everything seems to be ok now.
Manasa just went inside to recenter the AS beam on the camera after our Yarm spot centering exercises of the evening, and heard a loud beeping. We determined that it is the RAID attached to the framebuilder, which holds all of our frame data that is beeping incessantly. The top center power switch on the back (there are FOUR power switches, and 3 power cables, btw. That's a lot) had a red light next to it, so I power cycled the box. After the box came back up, it started beeping again, with the same front panel message:
H/W monitor power #1 failed.
DO NOT DO THIS. This is what caused all the problems. The unit has three redundant power supplies, for just this reason. It was probably continuing to function fine. The beeping was just to tell you that there was something that needed attention. Rebooting the device does nothing to solve the problem. Rebooting in an attempt to silence beeping is not a solution. Shutting of the RAID unit is basically the equivalent of ripping out a mounted external USB drive. You can damage the filesystem that way. The disk was still functioning properly. As far as I understand it the only problem was the beeping, and there were no other issues. After you hard rebooted the device, fb lost it's mounted disk and then went into emergency mode, which was to remount the disk read-only. It didn't understand what was going on, only that the disk seemed to disappear and the reappear. This was then what caused the problems. It was not the beeping, it was the restarting the RAID that was mounted on fb.
Computers are not like regular pieces of hardware. You can't just yank the power on them. Worse yet is yanking the power on a device that is connected to a computer. DON"T DO THIS UNLESS YOU KNOW WHAT YOU"RE DOING. If the device is a disk drive, then doing this is a sure-fire way to damage data on disk.