the problems with these circle plots:

1. you have to make the aspect ratio of the PDF correct or else it doesn't look like a circle
2. what we care about is the deviation from circle, so you should plot the difference in a way that let's us see the difference, not just that it sort of looks like a circle. This is similar to how we plot the residual when doing fits.

Both the TF measurement and the noise measurements are useful, but the nosie measurement is much more meaningful. Since we expect the main coupling to be incoherent, what we really want is a noise budget style measurement:

1. Measure the FM noise spectrum with only a single sine wave into the Moku.
2. Same as #1, but with the AM noise added as you already did.
3. Estimate the noise budget contribution by doing PSD subtraction, and then scale that by the excitation magntiude. This will be the contribution of beat amplitude noise to the measured calibration.

Kapton tape is a good insulator, so its a good block for 60 Hz. But it is mostly useless for RF since the capacitance between the mount and the table is

C = epsilon * A / d

For Kapton the dielectric constant is 3.5, the PD mount area is 10 cm x 10 cm, and the film thickness is ~50 um. So C ~ 5 nF.

Z ~ 1 / (2 * pi * 44 MHz) / C

  ~ 0.5 Ohms

this is very exciting! Its the beginning of the task of reducing vast amounts of PEM data into human-useful (bite sized) info. Imagine if we had 60 Hz BLRMS on various channels - we would know exactly when ground loops were happening or disappearing.

For the loop calculation, don't you have to consider the IMC cavity pole? What about the analog filter on the output of the HV driver for the laser PZT?

It would be great if you could calibrate these ASC channels into physical units (e.g. urad or nrad). I am curious to see how the noise spectra compares to the IMC WFS.

Since the data is still on disk, you can probably use the oplev channels to calibrate the WFS. Also, you can calibrate either WFS or oplev by moving the SUS alignment sliders until the arm power goes down by sqrt(2) or 2.

To get data faster with DTT, I ask only for data sampled at 16 Hz. You can either just read the EPICS channels (OUT16) or ask DTT for a BW=16 Hz for the fast channels. No need for high sample rate for step response plots.

• looking at some IMC WFS swept sines, it seems like there is poor margin around 1 Hz
• increasing the overall gain (input) by 2x makes the whole system shake a lot at ~1Hz
• increasing the input gain from 1 to 4 makes the lock break due to oscillations
• I have turned the input slider to 0.5 just now, but it will revert to 1 after the next lock loss for tonights testing
• we can use the observations from now for an hour or so to see if 0.5 is better for the 1 Hz behavior (lets look at the summary pages)

Made a comparison plot between the WFS1 PIT loop and the model. There is good agreement.

• I have had trouble increasing the UGF to > 1 Hz. Usually some instability ~1 Hz.
• Took a swept sine TF of WFS1 P, with the all 6 angular loops closed. The UGFs are all <0.1 Hz or so, so I think it doesn''t affect the loop shapes around 1 Hz.
• The model was not agreeing with the measurement. In addition to the pendulum TF and the WFS1 PIT filters, I had to add the Bounce/Roll mode bandstop filters, and the 28 Hz elliptic low pass which is the post DAC low pass (aka dewhitening filters).
• I expect that some of the wiggles around 1 Hz are due to modeling the pendulum as a viscous damped pendulum, rather than the OSEM damping loop.

Next, I will try to do some loopology to increase the phase margin around 1-3 Hz. HEPA in OFF state for now - please turn on in the morning.

notes 20-Feb-2023

* emacs doesn't run in base conda path (cds). LD_LIBRARY_PATH errors

* works fine if I do 'conda deactivate'. Something weird with our conda env?

* did apt install emacs, xemacs, terminator,

* these should be in the default workstation install

* arrgg!
(cds) ~>dataviewer
bash: dataviewer: command not found

* looks    like apt installs, made    dataviewer executable disappear?

I added a less agressive low pass filter to give some more phase margin, and then increased the overall gain by 4x. There is now some suppression around 1 Hz.

In the attached plot:

1. HEPA ON is the grey traces. Nothing surprising there.
2. The dark PURPLE traces are HEPA OFF, old gain/filter settings.
3. Colorful traces: RLP20 instead of RLP12. input gain slider = 4.0

Its clear from this that the WFS2 PIT servo has a very low gain. There's no UGF bump in the spectra.

I also added a 6 Hz : 3 Hz pole:zero pair to give some phase lead. Turning this on reduces some gain bump in pitch, but makes the WFS1 Yaw loop oscillate.

that is great

I think we would like to set the WFS1 P/Y UGFs to be ~2-3 Hz, and the MC_TRANS loops to have a UGF of ~0.1 Hz.

Could you use your loop gain measurements to set the _GAIN values for those UGFs? I am curious to see if the system is stable with that control.

changed some y-scale limits on the WFS summary pages to zoom in better

this measured Yaw matrix seems very different from the previous one. How can they really both be stable?

WFS error signal spectra w loops ON (G=4) and OFF.

Current output matrix also attached.

does Anyone understand why the broadband noise injection is so bad around 1 Hz? we do not see this issue with swept sine. noise seems good at other frequencies.

Does it have anything to do with the time constant of the resonances?

I have changed the MC SUS output matrices by a few % for some A2L decoupling - if it causes trouble, please feel free to revert.

Anchal came to me and said, "I think those beam offsets are a bunch of stinkin malarkey!", so I decided to investigate.

Instead of Alex's "method" of trusting the actuator calibration, I resolved to have less systematics by adjusting the SUS output matrices ot minimize the A2L and then see what's what vis a vis geometry.

The attached screenshot shows you the measurement setup:

1. copy the DoF vector from DoF column into the LOCKIN1 column.
2. Turn on the OSC/LOCKIN for the optics / DoF in question (in this example its MC2 PITCH)
3. Monitor the peak in the MC_F spectrum
4. Also monitor the mag and phase of the TF of MC_F/LOCKIN_LO
5. use the script stepOutMat.py to step the matrix

Next I'm going to modify the script so that it can handle input arguments for optic/ DOF, etc.

FYI, the LOCKIN screens do have a TRAMP field, but its not on the screens for some reason . Also the screens don't have the optic name on them. :

SUS>caput C1:SUS-MC2_LOCKIN1_OSC_TRAMP 3
Old : C1:SUS-MC2_LOCKIN1_OSC_TRAMP   0
New : C1:SUS-MC2_LOCKIN1_OSC_TRAMP   3

After finishing the tuning of all 3 IMC optics, I have discovered that 27.5 Hz is a bad frequency to tune at: the Mc1/MC3 dewhtiening filters have a 28 Hz cutoff, so they all have slightly different phase shifts at 27-28 Hz due to the different poles due to tolerances in the capacitors (probably).

*Also, I am not able to get a real zero coupling through this method. There always is an orthogonal phase component that can't be cancelled by adjusting gains. On MC3, this is really bad and I don't know why.
 

but what about including the DC reflectivity imbalance of the arms? there would be another BH55 term from that field maybe.

Anchal and I turned on another RL policy (ninedwarfs) with Chris's help.

It looks to be performing great, with good low frequency suppression and low noise injection at higher frequencies.

Going to leave it on overnight. It seems to respond well to lockloss of IMC, me whacking the MC2 chamber, walking near the MC2 chamber, kicking the optics by step in actuators, and turning off the sensors for a few seconds. Pretty robust!

how about the other idea of downloading the I & Q channels and doing the analysis offline? I'm curious if its better or worse. How could the Moku possibly be better?

Another idea is to use the frequency divider and then directly digitize. I believe someone tried that a few years ago, but not sure how good it was.

this afternoon I did some swapping around of linear and RL for IMC WFS.

In the end, I left in in the 'both' state:

• The WFS1,2,MC_TRANS PIT loops are all on but with -40, -40, -20 dB of nominal gain
• the RL is on for PIT

this is so that we have good DC control with integrators and good HF performance too.

Is the A2L coming from the optical lever feedback? If so, we can make a 30 Hz ELP to cut it off by 60 Hz.

As a test, I have replaced the PSL StripTool on the north wall of the control room with a nearly equivalent NDScope display. It is plotting the real time second trend along with min/max values. Let's try this out for awhile, and if its bad we can just close the window.

FYI, the striptool machine, zita, has a mis-configured network setup so that its not getting a useful nameserver. So for now, its running on allegra through ssh -Y. We should put the zita network thing on some sort of issue tracker. Maybe we can have tickets and issues like various companies use for tech support? This could cover all of our main lab work and help us keep track of little problems like this.

it is great to see a sensing matrix without 900 digits of precision!

for choosing what sensor to use, we don't necessarily care about W/m, but more like the equivalent noise of the sensor in m/rtHz, taking into account the real noise floor. In that case, we would possibly rotate the demod phase to maximize the SNR rather than maximize the S or minimize the N.

if systemctl was handling it, how come it died? Or did someone kill it on purpose? I wish we had a tool like Monit or Nagios running to tell us about these things.

Too many cavity poles! It should be just a single pole. Please show us your OLG Bode plot on the same plot as each of the individual pieces of the loop, so we can see what contributes to the TF.

With such a big temperature change, do you still get a reasonable beat note frequency? There's some previous elog of Koji I think that explains how we need to tune the lasers to get the 3 lasers to give 2 beat notes that are below 150 MHz.

I recommend usinng the DC offset method that Koji and I used for measuring the IMC WFS sensing matrix (not the AC method that Anchal used). With a sensing matrix, you should be able to do some partial inversion.

Without any sensing matrix inversion, we would have to rely on a gain hierarchy for getting the loops to work.

With some approximate matrix inversion, the loops are more indepedent of each other. Also if you look at the spectrum of the error signals, it should be clear that the sensing noise is pretty large, and so that sets a natural upper limit to the UGFs. We only want integrator (1/f) loops, but the LPFs cause some extra phase lag.

• with WFS off, I use MC_ALIGN screen to get erasonable power and better cenetering on mc2 QPD
• With good alignment, I unlocked MC and aligned DC beams on WFS
• Re-lock IMC.
• Turn on WFS w OLD matrix.
• Let's see how it goes Mon Oct 9 15:18:20 2023

It still works after a couple of hours. Around 6 PM I did some small alignment changes to bring the control signals closer to zero.

So my conclusion is that it wasn't a matrix issue, but just that the alignment was so off that we were out of the linear range or maybe just off the MC2 QPD. Hopefully it behaves well tonight.

I looked at this code and it is a little untrustworthy. The demod is not being done correctly so there can easily be some aliasing going on.

in this lab we really, really should never use a moving average instead of low pass filtering.

And why import math instead of using numpy??

 This is the fone storree 

we should turn off 3977,3979 and 2351 has no function

Edit, JD:  x8925 is at the Visitor desk, and isn't on the map.

As it turned out, the setting of +26dB for the FSS Fast gain was making the NPRO PZT / Pockels cell crossover too unstable. This was the cause of the large ~30 kHz peak that Jamie was seeing in his spectra (more to follow in the morning).

After recovering the locking, etc. by fiddling with the gains, we went about systematically setting all of the gains/offsets. Jamie's elog will show all of the various spectra with different FSS gains.

For offset setting, this was the procedure:

1. We want the MC servo board to have zero voltage on its MCL and MCF outputs (aka MC_SLOW_MON and MC_FAST_MON) with the Boosts ON, so we switched ON the 40:4000 and the 2 Super Boosts and put the VCO gain into its nominal +25 dB setting. This saturates the outputs and makes them impossible to use as readbacks so you have to be careful. Get the outputs close to zero as you turn on each boost. Finally, you will see that +/- 1mV of input offset (aka MC_REFL_OFFSET) will flip the FAST output between +/- 10V. This is the right setting.
2. With the MC board adjusted to send 0 V into the FSS error point, adjust the FSS input offset (with the Common and Fast gains at the nominal values) so that the FAST output voltage goes to +5.5 V. This is the middle of the range for the high voltage driver which drives the NPRO.

3. Let the MC lock and go through the UP script. When the MCL comes on with the integrator, the FAST voltage will shift from +5.5 V to something else. Use the following line: ezcaservo -r C1:PSL-FSS_FAST -g -1 -s 5.5 C1:SUS-MC2_MCL_OFFSET, to automatically tune the MCL offset.

    

   That's all. I have left the FSS common gain at +10.5 and the Fast gain at +23.5. These seem to be close to the optimum. Jamie will have more tuning tomorrow.
    
   I have also changed the 'mcup' script to set the MCL offset and set the FSS Slow setpoint to shoot for +5.5 V of FSS_FAST.
    
   MC_REFL_OFFSET = +1.176 V
   MC2_MCL_OFFSET = +47.8 counts
   FSS_INOFFSET   = -0.85 V

- we finished the MZ alignment; the contrast is good.

- we did the RFAM tuning using a new technique: a bubble balanced analyzer cube and the StochMon RFPD. This techniques worked well and there's basically no 33 or 166 RFAM. The 133 and 199 are as expected.

- the MC locked right up and then we used the periscope to align to it; the transmission was ~75% of max before periscope tuning. So the beam pointing after the MC should be fine now.

- the Xarm locked up with TRX = 0.97 (no xarm alignment).

If Rob/Yoichi say the alignment is now good, the we absolutely must center the IOO QPDs and IP POS and IP ANG and MC TRANS  today so that we have good references.

-----------------------------------

The first photo is of our nifty new setup to get the beam to the StochMon PD.  The MZ transmitted beam enters the photo from the bottom right corner, and hits the PBS (which we leveled using a bubble level).  The P-polarization light is transmitted through the cube, and the S-polarization is reflected to the left.  The pure S-polarized light hits a Beam Splitter, which we are using as a pickoff to reduce the amount of light which gets to the PD.  Most of the light is dumped on an aluminum dump.  The remaining light hits a steering mirror (Y1 45-S), goes through a lens, and then hits the StochMon PD.  While aligning the MZ to maximize visibility, we look at the small amount of P-polarized light which passes through the PBS on an IR card, and minimize it (since we want to be sending purely S-polarized light through the EOMs and into the MC).

The second photo is of a spectrum analyzer which is directly connected to the RF out of the StochMon PD.  To minimize the 33MHz and 166MHz peaks, we adjust the waveplates before each of the EOMs, and also adjusted the tilt of the EOM holders.

The final photo is of the EOMs themselves with the Olympus camera.

Once we finished all of our MZ aligning, we noticed that the beam input to the MC wasn't perfect, so Rana adjusted the lower periscope mirror to get the pointing a little better.  

The MZ refl is now at 0.300 when locked.  When Rana reduced the modulation depth, the MZ refl was about 0.050 .  Awesome!

Looks like the GUR2_X signal is bad. Jenne says that we need to center it mechanically before the signals will become useful again. Maybe Steve will do this - instructions are in the manuel ?

Issues I notice on first glance:

1. The OSEM Sensor Input matrix and the DOF2COIL Output matrix screens should be their own screens and linked from the overview. Right now they are not. Where is the input matrix?
2. The SIDE GAIN looks like zero on the main screen, but the side OSEM signal seems to be getting through to the SIDE filter bank. . I think the wiring of the SIDE signal through the input matrix is bogus.
3. The OUTPUT matrix seems to be the transpose of the previous OUTPUT matrix and we have lost the wires that connect the inputs and outputs to the matrix. We ought to think about how best to represent things on the OVERVIEW screen; probably only need to have a minimal representation and allow power users to open up the detailed screen.
4. The TIME string is whited out. How will this be done? Does each FE display its local time on its EPICS screens?
5. So far unable to get any channels on DV. How do we look at channels / test points?
6. As far as we can tell, there is no connection between the output of the SUSPOS, etc. filter banks and the OUTPUT MATRIX. So....nothing actually goes to the coil driver. Its hard to imagine that this new SUS could have ever worked. Is there any evidence that the damping actually worked in the past, or was it something like "well, the watchdog values came down to small numbers eventually..." ???
7. We are trying to debug the simulink file, but....the wiki entry on how to do this is out of date (yet updated as recently as August!) some path stuff just probably needs to be edited.

Basically the suspensions are not functioning yet and we can't attempt locking of the MC.

We noticed about a week ago that the NDS2 channel lists were not getting updated on megatron. JZ and I investigated; he was able to fix it all up this afternoon by logging in and snooping around Megatron.

Please try it out and tell me about any problems in getting fresh data.

1. The NDS2 server is what we connect to through our python NDS2 client software to download some data.
2. It has been working for years, but it looks like there was a file corruption of the channel lists that it makes back in 2017.
3. Since the NDS2 server code tries to make incremental changes, it was failing to make a new channel list. Was failing to parse the corrupted file.
4. there was a controls crontab entry to restart the server every morning, but the file name in that tab had a typo, so that wasn't working. I commented it out, since it shouldn't be necessary (lets see how it goes...)
5. the nds2mgr account also has a crontab, but that was failing since it didn't have sudo permission. JZ added nds2mgr to the sudoers list so that should work now.
6. I was able to get new channels as of 4 PM today, so it seems to be working.

* we should remember to rebuild the NDS2 server code for Ubuntu. The thing running on there is for CentOS / SL7, but we moved to Ubuntu recently since the SL7 support is going away.

** the nds2 code & conf files are not backed up anywhere since its not on /cvs/cds. It has 52 GB(!!) of txt channel lists & archives which we don't need to backup

We would like to increase the UGF of the PRC loop so as to allow more suppression of the PRC signal and less pollution of the MICH signal (remember that the PRC/MICH optical gain ratio is huge).

We were already losing phase because of delay in the LSC - SUS digital link. In addition to that, a major source of delay is the analog anti-aliasing (on the LSC error signals before they enter the ADC) and the analog anti-imaging (between the SUS DAC and the coil driver).

 IN addition to these, the other major sources of phase lag in the system are the FM5 filter in the LSC-PRC filter bank, the digital upsampling and downsampling filters, and the DAC sample and hold.

In the near term, we want to modify these analog filters to be more appropriate for our 64 kHz ADC/DAC sample rate. Otherwise, we are getting the double phase lag whammy.

Staring at the schematics for the AA (D000076-01) and the AI (D000186-A), we determined a plan of action.

For the AA, we want to remove the multi-pin AA chip filter from Frequency Devices, Inc. and replace it with a passive LC low pass. Hopefully, these chips are socketed. Rana will design an appropriate LC combo and elog; we should make the change on a Wednesday afternoon so that we have enough soldering help.

For the AI, the filter is a dual bi-quad using discrete components and LT1125 opamps. Not so clear what to do with these. The resistors are all the noisy thick film kind and maybe should be replaced. Koji will find some online design tool for these or do it in LISO. Changing the TF is easy; we can just scale the capacitors. But we also want to make sure that the noise of the AI does not destroy the noise reduction action of the dewhitening board which precedes it.

Jenne should figure out how low the noise needs to be at the input to the coil driver.

P.S. the matlab code which defines these filters

>> [z,p,k] = ellip(4,4,60,2*pi*7570,'s');
>> misc.ai = zpk(z,p,k*10^(4/20)) * zpk([],-2*pi*13e3,2*pi*13e3);
>>
>> % Fudged Anti-Imaging Filter
>> [z,p,k] = ellip(8,0.001,80,2*pi*7570,'s');
>> misc.aa = zpk(z,p,k*10^(0.001/20)) * zpk([],-2*pi*32768,2*pi*32768);

We were having unstable MC locking so we did some physical alignment touch up. Use MC suspension bias to have good MC alignment. Unlock MC and align DC beams to center on WFS. Re-lock and things are now stable.

Someone had been feeding bad info to Eric about MC alignments; we do not center the MC WFS beams with the MC locked.

However, in any case, today the MC2-TRANS servo was not being good so I turned it off. We need the real matrix measurement to bring it back.

Last night after checking cabling and turning on ISS, we tried several times to handoff to REFL_DC but it didn't work at all.

Some issues:

1. The ISS was injecting a lot of very low frequency power fluctuations because of bad AC coupling.
2. The SR560 @ LSC rack was saturating a lot with the x10 gain that Jenne and Rana put in; we turned it back to G = 1.
3. The ISS was also saturating a lot. We turned it off around 4 AM, but still no success.
4. The ALS sequence for finding the Red Resonance takes too long (~2 minutes), so we're trying a faster scheme tonight.

>> mcc -v -m -R -nojvm seisBLRMS.m
Warning: Duplicate directory name: /cvs/cds/caltech/apps/linux/matlab/toolbox/local.
Compiler version: 4.6 (R2007a)
Processing /cvs/cds/caltech/apps/linux/matlab/toolbox/matlab/mcc.enc
Processing /cvs/cds/caltech/apps/linux/matlab/toolbox/signal/mcc.enc
Processing /cvs/cds/caltech/apps/linux/matlab/toolbox/control/mcc.enc
Processing /cvs/cds/caltech/apps/linux/matlab/toolbox/filterdesign/mcc.enc
Processing /cvs/cds/caltech/apps/linux/matlab/toolbox/shared/controllib/mcc.enc
Processing /cvs/cds/caltech/apps/linux/matlab/toolbox/ident/mcc.enc
Warning: an error occurred while parsing class FilterDesignDialog.AbstractEditor:
Undefined function or variable 'DAStudio.Object'.

> In /cvs/cds/caltech/apps/linux/matlab/toolbox/shared/filterdesignlib/@FilterDesignDialog/@CoeffEditor/schema.p>schema at 9
Warning: an error occurred while parsing class FilterDesignDialog.CoeffEditor:
Invalid superclass handle.

Processing /cvs/cds/caltech/apps/linux/matlab/toolbox/fixedpoint/mcc.enc
terminate called after throwing an instance of 'ApplicationRedefinedException*'
Abort (core dumped)
"/cvs/cds/caltech/apps/linux/matlab/bin/mcc" -E "/tmp/fileRnU5Qj_31324": Aborted
??? Error executing mcc, return status = 134.

We restored the good state of the ITMY camera and neatened both the MC2 and ITMY camera.

The MC2 camera was driving a triple T jungle into some random cables and spoiling the image. We removed all T's and the MC2 camera now drives only The Matrix.

The ITMY camera was completely unmounted and T'd. So it was misaligned just by the force of gravity acting on its BNC cable. We swapped the lens for a reasonable sized one and remounted it in its can. We then used orange cable ties to secure the power and BNC cable for the MC2 and ITMY cameras so that tugging on the cables doesn't misalign the cameras. This is part of the camera's SOP.

No more driving 50 Ohm cables and T's with video cables, Steve! If you need a portable video, just use a spigot of the Matrix and then you can control it with a web browser.

I also wiped out the D40's memory card after uploading all of the semi-useful files to the Picasa page.

                 REFL_DC

Unlocked           4.50  V
Locked noWFS       1.30  V
Locked + WFS       0.42  V

 Alex logged in around 10:30 this morning and, at our request, adjusted the configuration of fb40m to have 20 days of lookback.

I wasn't able to get him to elog, but he did email the procedure to us:

1) create a bunch of new "Data???" directories in /frames/full
2) change the setting in /usr/controls/daqdrc file
       set num_dirs=480;

my guess is that the next step is:

3) telnet fb0 8087

    daqd>  shutdown

I checked and we do, in fact, now have 480 directories in /frames/full and are so far using up 11% of our 13TB capacity. Lets try to remember to check up on this so that it doesn't get overfull and crash the framebuilder.

Email from Alex:

Turned out to be the lack of current year information in the IRIG-B signal
received by the Symmetricom GPS card in the frame builder machine caused
this. I have added a constant in daqdrc to bring the seconds forward:

controls@fb /opt/rtcds/caltech/c1/target/fb $ grep symm daqdrc
#set symm_gps_offset=-1;
set symm_gps_offset=31536001;

Hopefully we will be upgrading to the newer timing system at the 40M this
year, so this will not happen again next year.

Doing an 'ls -lrt' in /frames/full/ now shows that the names are correct:

drwxr-xr-x 2 controls controls 249856 Dec 30 19:06 9777
drwxr-xr-x 2 controls controls 192512 Dec 31 16:00 9778
drwxr-xr-x 2 controls controls 151552 Jan  1 08:53 9463
drwxr-xr-x 2 controls controls 266240 Jan  2 12:39 9464
drwxr-xr-x 2 controls controls 258048 Jan  3 16:26 9465
drwxr-xr-x 2 controls controls 253952 Jan  4 20:13 9466
drwxr-xr-x 2 controls controls 151552 Jan  5 13:54 9467
drwxr-xr-x 2 controls controls  12288 Jan  5 15:57 9783

#
# Script to backup via rsync the most recent 40m minute trends and
# any changes to the /cvs/cds filesystem.
#
50 05 * * * /cvs/cds/caltech/scripts/backup/rsync.backup < /dev/null > /cvs/cds/caltech/scripts/\
backup/rsync.backup.log 2>&1

30 06 * * * find /usr/controls/jobs -mtime +7 -exec /bin/rm -f {} \;

 You can read through all of our past tests to see what didn't work in tracking things down. As Den mentions, there was actually a lot of evidence that there was some double->single precision action in the filter calculation causing the noise we saw.

However, it turns out that this is NOT the case.

This afternoon I was so confused that I enlisted JZ to help us out. He came over and I tried to replicate the error. When looking at the time series, we noticed that it wasn't random noise; the signals seem to be getting clipped as they crossed zero. Sort of like a stiction problem. JZ left to go replicate the error on an offline system.

This turned out to be the important clue. As we examine the code we find this inside of fm10Gen.c:

if((new_hist < 1e-20) && (new_hist > -1e-20)) new_hist = new_hist<0 ? -1e-20: 1e-20;

this is line is basically trapping the filter history at 1e-20, to prevent some kind of numerical underflow problem (?). Seems reasonable, except that some filters which have higher order low passing in them actually have an overall scale factor which can be small (even as small as 1e-23, as Den pointed out).

So the reason we saw such weird behavior is that the first filter in SUSPOS is an AC coupling filter. This takes the OSEM signal and remove the large mean value. Then the next filter multiplies it by 1e-23 before doing the filtering and you end up with this noise in the filter history.

I looked and this line is commented out in the new BiQuad code, but as far as I can tell this issue has been around in aLIGO, eLIGO, iLIGO, etc. for a long time and could have been causing many cases of excess noise whenever we ended up a tiny gain factor in an IIR filter. At the 40m, there are easily a hundred such cases.

For now, I suppose we can just change this number to 1e-40 or so. I don't know how to calculate what the right number should be. Not sure why this underflow is not an issue for the BiQuad, however.