The outside particle counts for 0.5 micron are 3 million this morning at 9am. Low clouds, foggy condition with low inversion layer.
This makes the 40m lab 30-50K
I just turned on the HEPA filter at the PSL enclosure.
Please, leave it on high
The Caltech gasoline storage tank is being upgraded.
They are jack hammering and digging with bulldozer 50 yards south of ETMY
After many, many "it'll be there in 2 weeks" from the Guralp people, our seismometer is finally back!
I have it plugged into the Guralp breakout box's Channel 1xyz (so I have unplugged the other Guralp). Both of the Guralp's are currently sitting under the MC1/MC3 chamber.
Before we can have both Guralps up and running, I need to stuff the next 3 channels of the breakout box (back in the fall, I only had Caryn do 1x, 1y, 1z, and now I need 2x, 2y and 2z done with the fancy low-noise resistors), so all the gains match between the 2 sets of channels.
I'm leaving the new Guralp plugged in so we can see how it behaves for the next couple days, until I take out the breakout box for stuffing.
Old Guralp is hooked back up, the new one is sitting next to it, disconnected for now.
For the purpose of testing out the temperature sensors, I stole the PEM-SEIS_MC1X,Y,Z channels.
I unplugged Guralp NS1b, Guralp Vert1b, Guralp EW1b cables from the PEM ADCU(#10,#11,#12) near 1Y7 and put temp sensors in their place (temporarily).
I plugged the Guralp cables back into the PEM ADCU
Guralp NS1b ---> #11
Guralp Vert1b --->#10
Guralp EW1b --->#12
For several of the channels on the PEM ADCU, zeros are occuring at the same time. Does anyone know why that might happen or how to fix it?
To include the plots that I've been working on in some form other than on my computer, here they are:
First is the big surface plot of all the amplitude spectra, taken in 10min intervals on one month of S5 data. The times when the IFO is unlocked are represented by vertical black stripes (white was way too distracting). For the paper, I need to recreate this plot, with traces only at selected times (once or twice a week) so that it's not so overwhelmingly large. But it's pretty cool to look at as-is.
Second is the same information, encoded in a pseudo-BLRMS. (Pseudo on the RMS part - I don't ever actually take the RMS of the spectra, although perhaps I should). I've split the data from the surface plot into bands (The same set of bands that we use for the DMF stuff, since those seem like reasonable seismic bands), and integrated under the spectra for each band, at each time. i.e. one power spectra gives me 5 data points for the BLRMS - one in each band. This lets us see how good the filter is doing at different times.
At the lower frequencies, after ~25 days, the floor starts to pick up. So perhaps that's about the end of how long we can use a given Wiener filter for. Maybe we have to recalculate them about every 3 weeks. That wouldn't be tragic.
I don't really know what the crazy big peak in the 0.1-0.3Hz plot is (it's the big yellow blob in the surface plot). It is there for ~2 days, and it seems awfully symmetric about it's local peak. I have not yet correlated my peaks to high-seismic times in the H1 elog. Clearly that's on the immediate todo list.
Also perhaps on the todo list is to indicate in some way (analagous to the black stripes in the surface plot) times when the data in the band-limited plot is just extrapolated, connecting the dots between 2 valid data points.
A few other thoughts: The time chosen for the training of the filter for these plots is 6:40pm-7:40pm PDT on Sept 9, 2007 (which was a Sunday night). I need to try training the filter on a more seismically-active time, to see if that helps reduce the diurnal oscillations at high frequency. If that doesn't do it, then perhaps having a "weekday filter" and an "offpeak" filter would be a good idea. I'll have to investigate.
I unplugged Guralp EW1b and Guralp Vert1b and plugged in temp sensors temporarily. Guralp NS1b is still plugged in.
Even more plots for the Wiener filtering!
We have a set of spectrograms, which show (in color) the amplitude spectrum, at various times during a one month stretch of time, during S5. Each vertical data-'stripe' is 10min long.
We also have a set of band-limited plots, which take the spectra at each time, and integrate under it, for different frequency bands.
Each set of plots has the following 3 plots: The raw DARM spectrum, a ratio of residual/raw, and the residuals, normalized to the first one (on which the wiener filter was trained).
The residuals are the DARM spectrum, after subtracting the Wiener-filtered seismometer witness data.
From the ratio plots, it looks like the wiener filter is pretty much equally effective at the time on which the filter was trained, as one month later. Static filters may be okey-dokey for a long period of time with for the seismic stuff.
All oplevs servos turned off to protect our suspentions from vibration due to drilling and pounding in CES high bay area.
This activity will be done from 10 am till 3 pm today.
Meanwhile our IFO-air conditions are turned off for maintenance.
Their performance of 6 months is shown on plot.
Guralp Vert1b and Guralp EW1b are plugged back in to PEM ADCU #10 and #12 respectively. Guralp NS1b remains plugged in. So, PEM-SEIS_MC1_X,Y,Z should now corrsp to seismometer as before.
Looks like yesterday was particularly noisy. It's unclear to me why diurnal variation much more visible in MC1_Y, and why the floor wanders.
The first plot shows 5 days. The second plot shows 20 days.
I moved the mobile HEPA filter from ITMX's north door to ITMX-ISCT and covered it up with a merostate tent to accommodate the aluminum foil particle measurement on June 5
It lowered the 40m baseline counts by about a factor of 3 of 0.5 micron and a factor of 2 of 1.0 micron.
The HEPA filter is sweeping the floor and blowing the particles upwards. The MET ONE counter is on the top of the IOOC looking south at ~75 degrees upward.
Both accelerometers have been moved in an attempt to optimize their positions. The MC1 accelerometer was moved from one green bar to the other (I don't know what to call them) at the base of the MC1 and MC3 chambers. That area is pretty tight, as there is an optical table right there, and I did my best to be careful, but if you suspect something has been knocked loose, you might check in that area. The MC2 accelerometer was moved from the horizontal bar down to the metal table on which the MC2 chamber rests.
When I said "MC1/MC2 accelerometers," I meant the entire three-axis accelerometer set at each point.
I spent the week reading up on filter algorithm theory, particularly Wiener filtering. I have also learned how to get data from specific channels at specific times, and I've been getting myself acquainted with Matlab (which I have not previously used). Finally, I started messing around with the positioning of the accelerometers and seismometers in order to try to find the setup that yields the best filtration.
I moved the MC1 set of accelerators. Might have bumped things. If things aren't working, look around the MC1 chamber.
Also, I constructed two new XLR cables, but have not tested them yet.
We now have two 80-foot, female-to-female XLR cables for our pretty new microphones, one yellow and one purple. They have been tested and appropriately labeled.
Also, here is a very helpful pdf for how to properly attach the XLR connectors to a raw quad cable, as well as one for how to put the actual connectors together (ignore the cable instructions on the connector page... the cable depicted is not a quad cable).
[ Jenne, Clara ]
We made new channels for the microphones which came in this week, by editing C1ADCU_PEM.ini (and making an appropriate backup before we modified it) then restarting the framebuilder and the frontend computer C0DCU1. The new channels are:
These are connected to channels 13 and 14 on the PEM ADCU board, just next to the GURALP seismometer channels.
Clara is testing the mics so the max output voltage can be limited to +-2V for the DAQ, then we'll hook them up to our new channels and listen to the IFO (and all the audio frequency noises around it).
I have been working on finding the best spots to put the accelerometer sets in order to best subtract out noise (seismometers next!). Here is a plot of what I've done so far:
All of these were 80-minute samples. The dashed line is unfiltered, solid line filtered. So, Setup #1 looks the best so far, but I didn't leave it there very long, so perhaps it was just a really awesome 80 minutes. I've put the accelerometers back in the Setup #1 position to make sure that it is really better.
And, in case you can't intuitively figure out what configuration the accelerometers are in by such descriptive names, here are some helpful pictures. I didn't know about the digital cameras at first, so these are actually sketches from my notebook, which I helpfully labeled with the setup numbers, color-coded to match the graph above! Also, there are some real-life photographs of the current arrangement (Setup #1' if you forgot).
Doesn't this one look kind of Quentin Blake-esque? (He illustrated for Roald Dahl.)
This is the MC1 set.
Guess which one this is!
So, I'm double-posting, but I figured the last post was long enough as it was, and this is about something different. After double and triple checking the XLR cables, I hooked up the microphone setup (mic---preamp---output) to the oscilloscope to figure out what kind of voltage would register with loud noises. So, I clapped and shouted and forgot to warn the other people in the lab what I was doing (sorry guys) and discovered that, even on the lowest gain setting, my loud noises were generation 2-3 times as much voltage as the ADC can handle (2V). And, since our XLR cables are so freaking long, we probably want to go for a higher gain, which puts us at something like 20 times too much voltage. I doubt this is really necessary, but it's late (early) and I got camera-happy, so I'm going to share anyway:
So, to deal with this issue, I made some nifty voltage dividers. Hopefully they are small enough to fit side-by-side in the ports without needing extra cableage. Anyway, they should prevent the voltage from getting larger than 2V at the output even if the mic setup is producing 50V. Seeing as my screaming as loud as I could about 2mm away from the mic at full gain could only produce 45V, I think this should be pretty safe. I put the ADC in parallel with a 25.5 kOhm resistor, which should have a noise like 10^-8 V/rHz. This is a lot smaller than 1 uV/rHz (the noise in the ADC, if I understood Rana's explanation correctly), so the voltage dividers should pose a noise issue. Now for pictures.
I opened one so you can see its innards.
In case the diagram on the box was too small to decipher...
And finally, I came up with a name scheme for the mics and pre-amps. We now have two Bluebird (bacteriophage) mics named Bonnie and Butch Cassidy. Their preamps are, naturally, Clyde and The Sundance Kid. Sadly, no photos. I know it's disappointing. Also, before anyone gives me crap for putting the labels on the mics upside-down, they are meant to be hung or mounted from high things, and the location (and stiffness) of the cable prevents us from simply standing them up. So they will more than likely be in some kind of upsidedownish position.
I tested the voltage dividers and was getting up to about 3V. I retested the mic w/o the voltage divider in place, and, lo and behold, I was able to generate about 70-75V (previously, I maxed out at 45V). I'm not 100% sure why this was, but it occurs to me that, before, the sounds I was generating were short in duration (loud claps, short yelps). This time, I tried yelling continuously into the microphone. So, probably, I simply wasn't seeing the real peak before on the scope because it was too short to pick up. I have corrected the voltage dividers (by replacing the 25.5 kOhm resistors, which were in parallel with the ADC, with 10 kOhm resistors, taking the voltage ratio to ~60:1) and tested them. I haven't been able to generate more than 1500 mV, so I think they are safe. (It's possible we would have been fine with the old setup, since I think it would be hard to get any noises as loud as I was making, but better safe than sorry, right?)
I'm attaching a diagram of the new-and-improved voltage dividers.
I clamped Bonnie (microphone) to the top of a chamber near the vertex of the arms and placed Clyde (pre-amp) on the table right below (see picture). The cable was laid and Bonnie and Clyde are plugged into port #13 on the ADC. The second cable was plugged into port #14, but it is not connected to anything. I placed the looped up cable on top of the cabinet holding the ADC.
Note: the angle in the photograph is such that we are looking along the y-arm.
I hooked up Bonnie and Clyde last night and tested it today. First I tried some loud noises to make sure I could identify them on the readout. Then, Steve suggested I try to look for some periodic stuff. I set up Butch Cassidy and the Sundance Kid on the cabinets by the MC2 optic. Now for graphs!
I tapped on the microphone a few times. I also yelled a bit, but this is sampling by seconds, so perhaps they got overwhelmed by the tapping.
This time I tried some more isolated yells. I started with a tap so I'd be sure to be able to recognize what happened. Apparently, not so necessary.
Here, it looks like a pretty strong periodic pattern on the second mic (Butch Cassidy). I replaced the lines with dashed ones where the pattern was a little less clear. Possibility interference from something. Mic1 (Bonnie) seems to show a pretty regular beat pattern, which seems reasonable, as it isn't particularly close to any one instrument fan.
So, anyway. I thought those were neat. And that I wanted to share.
After setting up the microphones last week, I modified the Wiener filtering programs so as to include the microphone signals. They didn't seem to do much of anything to reduce the MC_L signal, so I looked at coherences. The microphones don't seem to have much coherence with the MC_L signal at all. I tried moving Bonnie to near the optical table next to the PSL (which isn't in a vacuum, and thus would, presumably, be more affected by acoustic noise), but that didn't seem to make much of a difference. Eventually, I'd like to put a mic in the PSL itself, but I need to work out how to mount it first.
Bonnie's new location.
You can see in bonnie_butch.pdf that none of the mic signals are giving very good coherence, although they all seem to have a peak at 24 Hz. (In fact, everything seems to have a peak there. Must be a resonant frequency of something in the mode cleaner.)
I've also attached plots of the coherences for all six accelerometers and the three Guralp seismometer axes. I plotted the most coherent traces together in the last pdf: the y-axes of the MC2 accelerometer and the two seismometers (the Ranger measures ONLY y) and, interestingly, the z-axis of the MC2 accelerometer. Unsurprisingly, the seismometers are most coherent at the low frequencies, and the MC2-Y accelerometer seems to be coherent at very similar frequencies. The MC2-Z accelerometer, on the other hand, seems to be coherent at the higher frequencies, and is highly complementary to the others. I am not really sure why this would be...
Finally, I was curious about how the noise varies throughout the day, because I didn't want to mistakenly decide that some particular configuration of accelerometers/seismometers/whatever was better than another b/c I picked the wrong time of day to collect the data. So, here is a plot of Wiener filters (using only accelerometer data) taken over 2-hour intervals throughout the entirety of July 6, 2009 (midnight-midnight local).
It's a little bit confusing, and I should probably try to select some representative curves and eliminate the rest to simplify things, but I don't have time to do that before the meeting, so this will have to suffice for now.
In her position overlooking whichever table it is that is next to the PSL, Bonnie drummed up some decent coherence with the PSL-PMC_ERR channel, but not so much with the MC_L. I moved her into the PSL itself, and now there is rather good coherence with the PMC_ERR channel, but still not so great for MC_L.
Bonnie's new home in the PSL.
Bonnie has been strung up on bungees in the PSL so that her position/orientation can be adjusted however we like. She is now hanging pretty low over the table, rather than being attached to the hanging equipment shelf thing. Butch Cassidy has been hung over the AS table.
Moving Bonnie increased the coherence for the PMC_ERR_F signal, but not the MC_L. Butch Cassidy doesn't have much coherence with either.
I noticed that the coherence would drop off very sharply just after 10 kHz - there would be no further spikes or anything of the sort. I used my computer to play a swept sine wave (sweeping from 20Hz to 10kHz) next to Butch Cassidy to see if the same drop-off occurred in the microphone signal itself. Sure enough, the power spectrum showed a sharp drop around 10kHz. Thinking that the issue was that the voltage dividers had too high impedance, I remade one of them with two 280 Ohm and one 10 Ohm resistor, but that didn't make any difference. So, I'm not sure what's happening exactly. I didn't redo the other voltage divider, so Bonnie is currently not operating.
So, I actually took these measurements last week, but I didn't get around to making nice plots and things until now. I figured the time while I wait for the spectrum analyzer to do its thing was a good time.
Having been unable to locate the SR785 and also unsure how to connect it to a computer speaker (and also unable to find a free one), I downloaded a demo of a function generator onto my computer and just used that. (Same thing I used to do the swept sine that created the frequency power response plots I posted last week.) I set the program to a number of different frequencies and had the other end of the cable hooked into the oscilloscope to see a) if I could pick out the frequency and b) see how the magnitude of the microphone output varied with the frequency.
The first set of measurements I took, I didn't realize that I could increase the output power of the function generator. Because the generated sound at the default setting was relatively quiet, the oscilloscope traces were pretty chaotic, so I usually froze the trace so that I could look at it better. I ended up with a lot of weird jumps in the magnitude, but I later realized that there was a lot of beating going on at some frequencies, and the amplitude changes were probably much more drastic for the -20 dB sounds than the 6 dB sounds, since it was closer in amplitude to the surrounding noises. So, I've included that data set in my plots for the sake of completeness, but I'm pretty sure that it is useless.
Once I realized I could increase the power output for the signal generator, I took a set of data with and without the voltage divider at 6 dB. There was a cluster of frequencies that showed significant beating around 1700-3000 Hz in the data WITH the voltage divider, but I did not see any clear beating in the data WITHOUT. In the plots, I simply plotted up the highest and lowest amplitudes I measured for the frequencies with significant beating, since it was obviously hard to tell what the amplitude would have been without any background noise. In the w/o volt. div. set, although I didn't see any obvious beat patterns, the measured amplitudes did jump slightly at the frequencies that showed beats with the voltage divider. So, perhaps I was just not seeing them, but they influenced my amplitude measurements? I'm not sure if it would be possible for the voltage divider itself to cause beat frequencies.
(Note: the amplitudes measured were from zero to peak, as the oscilloscope I was using wouldn't show a big enough vertical range to easily measure the peak-to-peak voltage difference.)
I've attached two plots of my measurements. One has a regular x-scale and includes all the measurements. The second has a logarithmic x-scale and omits the 20 Hz points. I had some troubles being able to pick out the 20 Hz signal on the oscilloscope... I don't know if my computer speakers just don't work well at that frequency or what, but either way, those points seemed highly suspect, and omitting them from the log plot allowed me to spread things out more.
One thing I'm not sure about is the 3000 Hz point. It was one of the ones with a beat frequency (~130 Hz), and the amplitudes were pretty low. The corresponding point from the non-voltage-divider data set is also low. So, I'm not sure what's happening there.
The one thing that I do think is quite clear is that the 1000 Hz drop-off in power when the microphone is connected to the ADC has nothing to do with the voltage divider. Beat issues aside, the shapes are very similar (pay no attention to the absolute scale... obviously, the voltage responses with and without the voltage divider were very different, and I just scaled them to fit in the same plot).
Update: Jenne pointed out that I was not absolutely clear about the voltage scale in my plots. The GREEN and BLUE points are on a mV scale, and the RED points are on a 10mV scale. I should probably redo the plots in Matlab in eventuality, since Excel is hard to use if you want to do anything that is not extremely basic with your plots, but this was my solution for the time being. So, the fact that the RED points, which are the data taken WITHOUT the voltage divider, are lower than the GREEN ones does not in any way indicate that I measured lower voltages when the voltage divider was not used.
Also, a to do list:
- Many of the beat frequencies I picked out were veeeeery slow, indicating that something is going at a frequency that is very close to the arbitrary frequencies I chose to sample, which is a little strange. That, combined with the fact that I saw clear beats with the voltage divider but not without leads me to believe that it may be worth investigating the frequency response of the voltage divider itself.
- Redo the measurements near the anomalous 3000 Hz point with a higher density of sampled frequencies to try to see what the heck is going on there.
I made and tested a female-to-female TRS(audio)-RNC cable. It only has a single channel, so it won't work for stereo speakers or anything, but I should only need one speaker for testing the microphones. The tip of the plug is the signal, the sleeve is ground, and the ring is null.
I've been trying for most of the week to get noise measurements on the output of the Guralp box as well as scross the AD640 chip. The measurements haven't really been making sense, and, being at a loss as to what else I should try, I decided to redo the resistors on the N/S 2 and E/W 2 channels. (I had been comparing the VERT1 and VERT2 channels, as VERT1 has been restuffed and VERT2 has not.) I don't need all three of the second set of channels to do more measurements, so it seemed like a good use of time.
The first thing I noticed was that the VERT2 channel was missing two resistors (R24 and R25). I probably should have noticed this sooner, as they are right by the output points I had been measuring across, but it didn't occur to me that anyone did anything to the VERT2 channel at all. So, probably the measurements on VERT2 are no good.
Note the existence of 100 kOhm resistors on the top channel, and none on the bottom channel (VERT2).
Then, while I was soldering in some 100 Ohm resistors, I happened to notice that the resistors I was using had a different number (1001) on them than the corresponding ones on the already redone channels (1003). I checked the resistance, and the ones on the already redone channels turned out to be 100 kOhm resistors, rather than 100 Ohms. So, I double checked the circuit diagram to make sure that I had read it correctly, and there were a number of resistors that had been relabeled as 100 Ohms and several relabeled as 100 kOhms. On the board, however, they were ALL 100 kOhms. Clearly, one of them is wrong, and I suspect that it is the circuitboard, but I don't know for sure.
The diagram clearly shows that R6 should be a 100k resistor, while R5 and R8 should be 100 Ohm resistors, but they are all the same (100k) on the board. I suspect this may have something to do with larger-than-expected noise measurements. But, it's possible the diagram is wrong, not the board. In any case, I didn't really know what to do, since I wasn't sure which was right, so I just replaced all the resistors I was sure about and removed the 100k and 100 Ohm resistors without replacing them with anything. Incidentally, the box of 100kOhm resistors seems to be missing, so I wouldn't have been able to finish those anyway.
There managed to be just enough 100 kOhm resistors to stuff all the "2" channels (VERT2, N/S2, E/W2) with the fancy low-noise resistors. The first six channels (VERT 1/2, NS 1/2, EW 1/2) are now completely done with the thin-film resistors, taking into account the changes that were made on the circuit diagram. I also replaced the C8 capacitor with the fancy Garrett ones and added capacitors on top of R4 and R13 (after painstakingly making sure that the capacitances are exactly the same for each pair) for the "2" channels. It looks like the capacitors on the "1" channels are the cheaper ones. I will compare the noise measurements later to see if there is any difference - if so, I can replace those as well (although, we're out of the 1 uF capacitors needed for C8).
Speaking of, we are now out of or very low on several types of the Garrett resistors/capacitors: 1 uF, 1kOhm, 100 Ohm, 14.0 Ohm, and 100 kOhm. I left the specifics on Steve's desk so that more can be ordered for the eventual time when the third set of channels needs to be restuffed.
I mapped out the corresponding pins on both ends of the Guralp seismometer cable. Here is the diagram:
The circular 26-pin end of the cable (that plugs into the seismometer) is labeled as above. The other end (the 39-pin end) is not physically numbered, so I just came up with a numbering system. They are both pictured on the non-cable end of the connector. The colored circles indicate the pin pairs.
FROM JENNE, 30JULY2009: the Dsub end is 37 pin, not 39.
I was in the lab last night accelerometerizing and noticed some dents on the tubes that stick out horizontally from the MC2 optical chamber (sorry, I don't know what they're called or what they do). One of them is pretty big... I don't know if this is a problem, but it probably isn't a good thing. Photos below:
This last one is a little hard to see... I was having trouble getting a good angle on it, but it's there. Not quite as significant as the first one though. (The first two pictures are of the same dent.)
The construction people next door seem to be getting pretty excited about pounding things lately. At my desk the floor was shaking like a mini-earthquake, and all of the accelerometers were pretty much railed. Clara has the Guralp box out right now, so the Guralp is unplugged, but the Ranger didn't seem to be railed.
This either (a) is part of the reason the MC is being wonky lately, or (b) has nothing whatsoever to do with it. The MC watchdogs haven't been tripping all the time, so maybe this isn't a primary cause of the wonky-ness.
In looking at a many-days/months trend to see how far back this has been going, it looks like the accelerometers are hitting their rails pretty much all day every day. This may be significantly hindering Clara's Wiener filtering work. I think the gain on the accelerometer's controler panel is already set to 1, but if it's set to 10, we may want to reduce that. Alternatively, we may want to put in attenuators just as the signal is entering the PEM ADCU, to help reduce the amount of rail-hitting that's going on. I don't remember this from a couple of months ago, so this may be a problem that will go away once the construction / landscaping is done next door.
After many issues, I finally have some Guralp box noise. I did not measure every single channel with high resolution at the low frequencies because that would have taken about 3 years, but I could perhaps take some faster measurements for all of them if necessary.
Both Guralps and the Ranger have been placed in our nice new insulated foam box, complete with packing peanuts, in the corner between the x and y arms. The Guralp breakout box has been reinstalled and everything is plugged in in prepartion for the huddle test. However, we're having some issues with ADC channels, which will be worked out tomorrow (hopefully) so that data can be collected over the weekend.
Currently, one Guralp is plugged into the three SEIS-MC1 channels. We made new channels for the second Guralp (GUR-EW, GUR-NS, and GUR-VERT), but had issues with those. So, EW and NS have been plugged into PEM_AUDIO-MIC1 and MIC2 for the time being.
I found that several of the cables are unlabeled so I'm not sure what's plugged in. In the end, I found that the TEMP_2, _3, & _44 channels were working and so I plugged in anything that looked seismic into there.
TEMP_2 is now apparently the X channel of the 2nd Guralp. If someone can figure out which cable belongs to the Y channel, please plug it into TEMP_3 and then we can fix the channel names.
I also removed (gently) all of the accelerometers from MC2's chamber. This didn't break the lock, but I intentionally broke it to make sure it reacquired fine. It did and the MC TRANS QPD showed no significant shift afterwards.
Friday, we were seeing a 2 Hz harmonic series in all of the PEM channels. Today I found that some bad person had put in a 4V (!) signal into one of the channels with a signal generator. The generator was also sneakily stuck way back inside the DCU rack. NO SECRET SIGNAL INJECTIONS!
Since the ADC has a 2Vpk range, this was saturating and putting in harmonics in all the adjacent channels. I disconnected it and turned off the function generator.