Joe got on the phone with Alex, and Alex's magic Alex intuition told him to ask about the RFM switch. The C0DAQ_CTRL's overload light was orange. Alex suggested hitting the reset button on that RFM switch, which we did. That fixed everything -> c0dcu1 came back, as did the frame builder. Rana had pointed out earlier that we could have brought back all of the other front ends, and enabled the damping of the optics even though the FB was still down. It's okay to leave the front ends & watchdogs on, and just reboot the FB, AWG, and DAQ_CTRL computers if that is necessary.
Anyhow, once the FB was back online, we got around to bringing back all of the front ends (as usual, except for the ones which are unplugged because they're in the middle of being upgraded). Everything is back online now.
After all of this craziness, all of the Guralp channels are working happily again. It is still unknown why they starting being digital zero, but they're back again. Maybe I should have rebooted the frame builder in addition to c0dcu1 last night?
This is regards to zero signal being reported by the channels C1:PEM-SEIS_GUR1_X, C1:PEM-SEIS_GUR1_Y, and C1:PEM-SEIS_GUR1_Z.
I briefly swapped Guralp 1 EW and Guralp 2 EW to confirm to myself that it was not on the gurlap end (although the fact that its digital zero is highly indicative a digital realm problem). I then unplugged the 17-32, and then 1-16 channel connections to the 110B. I saw floating noise on the GUR2 channels, but still digital zero on the GUR1 channels, which means its not the BNC break out box.
There was a spare 110B, unconnected in the crate, so to do a quick test of the 110B, I turned off the crate and swapped the 110Bs, after copying the switch configuration of the first 110B to the second one. The original 110B was labeled ADC 1, while the second 110B was labeled ADC 0. The switches were identical except for the ones closest to the Dsub connectors on the front. All those switches in that set were to the right, when looking down at the switches and the Dsub connectors pointing towards yourself.
Unfortunately, the c0duc1 never seemed to come up with the new 110B (ADC 0). So we put the original 110B back. And turned the crate back on.
The fb then didn't seem to come back quite right. We tried rebooting fb40m it, but its still red with status 1. c0daqctrl is green, but c0dcu1 is red, although I'm not positive if thats due to fb40m being in a strange state. Jenne tried a telnet in to port 8087 and shutdown, but that didn't seem to help. At this point, we're going to contact Alex when he gets in around 12:30.