In order to verify our theory about coherence corruption in linear systems due to the line
if((new_hist < 1e-20) && (new_hist > -1e-20)) new_hist = new_hist<0 ? -1e-20: 1e-20;
in the /opt/rtcds/caltech/c1/core/release/src/include/drv/fm10Gen.c in the iir_filter function I've changed -20 to other numbers and watched at the coherence input and output of the digital filter cheby1("LowPass", 3, 0.1, 0.5)cheby1("LowPass", 6, 1, 1.5). The sampling rate was 2K. The frequency responce of the filter presented in this figure.
The next plot shows psd and coherence of the signal for different numbers in the if-statement line : 1e-20 , 1e-25, 1e-100.
We can see that for present value coherence between input and output signals is small even for low frequencies. The psd of the output signal is also corrupted because at low frequencies it should have the same psd as input signal. For 1e-25 and 1e-100 we can see that coherence is close to 1 at low frequencies so if-statement does not work and we have a first order transition from bad to good filter performance with discontinious jump of coherence.
However, for 1e-25 and 1e-100 data is still corrupted by the round-off error. Lack of coherence for high frequencies can be explained by the fact that dtt tools use single precision for data analysis and output is too small to plot a right coherence. But the coherence is also not precisely 1 for low frequencies. Actually, it is 0.99 for this aggresive filter. We use double precision in the real-time code but still for such kinds of filters round-off error is present. As wrote Daniel Sigg for Cheby filter: "You need a lot more digits than you may naively suspect. In the 8th order example, the output of each SOS is amplified by ~106. This regardless of the fact that the coefficients are all of order 1. If you require a level of 10-3 attenuation in the stop band, you would have lost 9 digits already. Then, add the fact that you have to do of order 104 subtractions to get from 16kHz to 12Hz, loosing another ~2 digits. On top, the high Q section is probably 10 worse than the others and you lost 12 digits. In a real example this may stack up even worse."
Next we need to figure out what effects does round-off error introduce in the performance of the interferometer.