You can read through all of our past tests to see what didn't work in tracking things down. As Den mentions, there was actually a lot of evidence that there was some double->single precision action in the filter calculation causing the noise we saw.
However, it turns out that this is NOT the case.
This afternoon I was so confused that I enlisted JZ to help us out. He came over and I tried to replicate the error. When looking at the time series, we noticed that it wasn't random noise; the signals seem to be getting clipped as they crossed zero. Sort of like a stiction problem. JZ left to go replicate the error on an offline system.
This turned out to be the important clue. As we examine the code we find this inside of fm10Gen.c:
this is line is basically trapping the filter history at 1e-20, to prevent some kind of numerical underflow problem (?). Seems reasonable, except that some filters which have higher order low passing in them actually have an overall scale factor which can be small (even as small as 1e-23, as Den pointed out).
So the reason we saw such weird behavior is that the first filter in SUSPOS is an AC coupling filter. This takes the OSEM signal and remove the large mean value. Then the next filter multiplies it by 1e-23 before doing the filtering and you end up with this noise in the filter history.
I looked and this line is commented out in the new BiQuad code, but as far as I can tell this issue has been around in aLIGO, eLIGO, iLIGO, etc. for a long time and could have been causing many cases of excess noise whenever we ended up a tiny gain factor in an IIR filter. At the 40m, there are easily a hundred such cases.
For now, I suppose we can just change this number to 1e-40 or so. I don't know how to calculate what the right number should be. Not sure why this underflow is not an issue for the BiQuad, however.