Today's seismometer diagnosis activities are still underway, this is just an update (since I did some reboots):
Problem 1: X and Z channels on both seismometers were flipped. I unplugged an X cable (East/West on the cable labels) and the Z channel (vert) would go to floating ADC zero. Rana suggested that the ADCs sometimes have random channel hopping, and that a reboot of the c0dcu1 computer which handles the PEM ADCU should fix this problem. I keyed the c0dcu1 / c0daqawg crate, those computers came back just fine, and the channels were no longer flipped. This is a good thing. Although now it's actually the Z channels that were / are bad on both seismometers, not the X's.
While rebooting those computers, c1iovme, c1sosvme, c1susvme1 and c1susvme2 crashed. I rebooted them, although for the first few power cycles c1susvme1&2 couldn't mount /cvs/cds/caltech. Eventually they did, and life is good again. Except that the seismometers are still funny.
Some more progress, but still not complete:
Jan and I looked at all of the Gur channels on a 'scope (battery, so as not to be grounded), and 5 of the 6 looked good. We were looking at the BNCs just as they go into the ADC. The one which still looks bad is Gur1Z. The 'scope just doesn't see any signal on that channel.
In addition, the ADC's BNC input #4 (which normally has Gur2Z) looks totally shot. When it's floating, the signal on dataviewer definitely doesn't look floating. I'm probably going to have to move over to another channel, and just give that one up (this ADC already has several channels which have been declared bad, so maybe it's not a surprise that this can happen?)
Since one of the Gur signals looks bad (Gur1Z) and one of the ADC channels looks bad (usual Gur2Z), I switched the Z channels on the ADC board, so the channel being saved as Gur1Z is in fact Gur2Z. This is valid as of ~1:15am until further elog notice.
During my investigations into why Gur1Z is funny, I also looked at the signal on the BNC octopus cable coming straight from the output of the Guralp Breakout Box (this is the cable which goes from "ADC Out" on the back of the box which is a 37 pin D-sub to 9 differential BNCs), and sometimes I saw zero on the 'scope, but sometimes there was a signal which would coincide with jumping tests. Whenever there was a signal however, it was always a way lower amplitude (at least by a factor of 10?) than the other channels.
All of this craziness led to me pulling the Guralp box to investigate.
Upon opening the box, I recalled that the channels go in order: Vert, NS, EW. The Gur1Z channel is that first vert channel, and it's the one which always had a blue input capacitor rather than a surface mount one. Being suspicious of Frank and Alastair, since they seemed unhappy with my capacitor choices, I wondered if they had wiggled the blue cap, and tore something loose. Just in case, and to make things seem more uniform, I replaced the blue cap with a surface mount 1uF cap. (Actually its 0.909uF, replacing the 0.905uF blue cap, according to the black DMM that measures capacitance.) While I was in there, since it had been a problem in the past (elog 2811), I relflowed the solder on some of the resistors, especially near the output op amp.
Anyhow, none of that may have been necessary. All 6 of the Gur channels were examined on a 'scope, using clip doodles to measure the various Test Points on the circuit. I looked at all of the TPs in Gur1Z, and I didn't find that any particular stage was any noisier than the others. Also, all 6 of the Gur channels seemed totally fine in terms of sending a good signal to the output of the box, including Gur1Z which is currently under investigation. All of the channels passed the "output looks ~20x the input" test, and for approximately equal thumping on the ground all 6 channels seemed to have similar amplitude outputs. The Z channels on both channels one and channels two were a little bigger than the X's or Y's, but the 2 Z channels were about the same. This test was done using Guralp2 and the Gur2 cable on both channels 1 and 2, and then checked with Guralp 1, using the Gur2 cable on channels 1. The Vert1 channel always seemed good.
I now am suspicious of one or more of the cables: either the Gur1 cable from the seismometer to the box, or the Vert1 channel of the octopus cable. I'm satisfied that the BNC cables running through the cable tray are okay (although it might not hurt to check that they all successfully send a sine wave...) I opened up the backshell of the Gur1 cable, on the end that connects to the breakout box. Nothing seemed amiss. I still need to Beep the cable to check its connections, and look at the octopus cable.
Recap / Current Status: Breakout Box is reinstalled, both seismometers hooked up. The Z channels on the seismometers are swapped at the ADC input. The dataviewer channels Gur1_X, Gur1_Y, Gur1_Z (which is actually Z of Gur2 seismometer) and Gur2_X, Gur2_Y are all good. Nancy is going to leave the MC in a happy place, and note the time when she's done. Tomorrow I'll check out the cables for the Gur1Z seismometer channel.