Saturation occurs suddenly after running for hours without problem.

I tracked it since the drive was turned on (around 00:36:18) and found that it has been running ok for hours before this happens (attachment 1). Around 3:04:51, the LL channel saturated and then dropped back to normal level. After that, this kept happening and crashed the LL channel (attachment 3).

Attachment 2 is data corresponding to attachment 1.

I checked the behavior of noisemon in the three stations where they are installed. It is consistent that there is always one channels that is saturating and one channel that has excess noise. However, they are not always the same channels.

ETMY, attachment 1, LL saturating, UL excess noise (UR a little more noise)

ITMY, attachment 2, LL saturating, UL excess noise

ITMX, attachment 3, UL saturating, UR excess noise

The first noisemon board was installed at the ETMY station. It was a prototype board that we brought to LLO and installed there since then. I checked the data today (11/14) and its LL channels is not working.

I checked the time series in the attachment 2 and 3. There are some problem causing the circuit to saturate.

I checked the drive signal below 10Hz in attachment 4. The drive signals across all channels are the same.

Noisemon has been installed in LLO ITMY station. Its LL channel starts to saturate at 10/30 00:00:00. I checked the noisemon output at that time (attachment 1).

I checked the October 25 data at L1. The drive signal is zero, according to the bottom plot in attachment 1 and 2. The fastimon channel has a lot of lines. Noisemon still has the 65Hz harmonics.

I also checked the Nov 7 noisemon and fastimon data. We still see the harmonics on Nov 7. Also, one channel is completely dysfunctioning. The fastimon is similar to Oct 25.

Attachment 1: Oct 25 noisemon. We still see the harmonics in the noise.

Attachment 2: Oct 25 fastimon. Two channels are messy.

Attachment 3,4,5: Nov 7 noisemon data. Analyzed in attachment 4. Data in attachment 5.

Attachment 6: Nov 7 fastimon data. Similar to Oct 25 - a lot of lines.

Although L1 has been down since the beginning of October, I checked the spectrum at the output of the nosiemon at L1 ITMY. It looks like the UL channel, as we have seen before, still has the lines,

Noisemon has been installed at L1 for a while. Now we have it on ITMX and ITMY. ITMX was installed first and ITMY was installed on September 11.

Recently, after it was installed at ITMY, we were trying to check the functionality of the circuit - if it measures DAC nosie properly between 20 - 100Hz. When we were doing that, we encountered some strange harmonics of 65Hz in UL channel. It is shown in attachment 6, data from 9/25 ETMY.

We traced back to where it was installed at ETMY, 9/11, and we still see the lines there (attachment 5).

Then we went to check ITMX, since the data was consistent when it was first installed. However, when we use 9/25 data, we see attachment 4. The UL channel measurement is way above the other three channels. Tracing back the dates, we found in ITMX it started 9/10 (attachment 3), comparing to 9/9 (attachment 2) when it was still good.

All the data and scripts are in attachment 1.

Notes (in case you read about noisemon for the first time):

All the plots are

It is suppose to be DAC nosie + Noisemon noise + ADC noise, which DAC noise being dominant between 20 - 100Hz. The purple curves are DAC noise model. The measurements are expected to be close to it between 20Hz and 100Hz.

This is the DAC noise of ITMY PUM coil driver in L1. The data is from September 25, 2019 00:00:00 UTC. I did not subtract the ADC noise and the noisemon noise since I did it for ITMX. The noisemon noise is far below the total noise, and the ADC noise only dominates high frequency. After subtraction, the mismatch above 500Hz will disappear.

There are some lines around 65Hz. I checked the data a few days arounds 9/25. I see them in all the data. I do not understand why yet.

Spectra of the PUM drive signal is attached in 3, noisemon output in 4. I do not see anything at 65Hz in those channels.

ADC noise and Noisemon noise are subtracted in attachment 5.

I checked these data

9/23 00:00, 8:00, 20:00

9/24 8:00,

9/25 0000,0800,2000

9/26 0800,2000

9/27 0000.0800.2000

10/1 0800

The 65Hz harmonics stays there.

I checked time domain for 9/15. It is not saturating. (Attachment 6&7)

Attachment 8 is a check on the output of the noisemon signal when L1 is down. It is from 9/24 1600. Surprisingly, I see the 65Hz harmonics in UL spectrum while I cannot see it when the interferometer is online. I do not see the harmonics in the other three channels.

In attachment 9, I checked ITMX from the same time (9/24 1600, interferometer down). The 65Hz harmonics is there in all channels, and the UL output is higher.

I did a quick estimation of the subtraction result. I subtracted the ADC noise and the noisemon noise. 

Now I subtract it:

The ADC noise and Noisemon noise are converted to DAC volts (divided by the transfer function of the noisemon and coil driver).

Form the results from 10-200Hz, it seems that the calculated noise is DAC noise.

Attachment 2 is the result: subtracted noises vs. model compared to aLIGO noise.

Attachment 1 shows how the subtraction is done: we subtract noisemon noise from the total noise. (Noisemon noise contains ADC noise) Noisemon noise and ADC noise is neglegible at that frequency.

It seems at low frequency what we see there might still be DAC noise, if not other unknown sources.

I don't agree about this. Doesn;t this ignore the noise of the noisemon circuit (analog readout noise + ADC noise) ? I think you must have a model for than noise in order to infer the DAC noise. Or maybe my pringle suggestion has better SNR?

I'm attaching a script to download data from the LIGO sites with python.

I recommend using it in your anaconda3 ENV:

conda install -c conda-forge nds2-client python-nds2-client

and then before running the script you have to initialize your Kerberos token:

kinit miley.cyrus@LIGO.ORG

then you run the script:

python getData.py --ifo=L1 --fs=1024

as usual, run with the -O or -OO flags to silence the debug messages.

The PUM noisemon board has been installed in Livingston ITMX test mass. After the installation, we fetched the coil driver drive signal, noisemon output signal and the coherence between them. 

This is how we calculate the DAC noise spectrum. The unit is V/rtHz.

The data fetching configurations are:

- Start time: 1250467218. Locked for more than 20hrs from there, if you check here: https://ldas-jobs.ligo-la.caltech.edu/~detchar/summary/day/20190822/

- Bin size: 0.1Hz

- Window: Hanning

- Average: 1000

Attachment 1 and 2: plot of the DAC noises in volts compared to the G1401399 model.

Attachment 3 and 4: plot of the DAC noises projected to the displacement of the test mass and incohrerently summed from all the four test masses.

All the data are attached as xml files. They are directly saved from DTT and can be opened in DTT.

To reproduce the plots, run the python code in the zip file. The code runs without any parameters. 

Can't tell what's going on. Pleaese make the plots readable and describe in the elog what precisely is being calculated.

The bin size was 1Hz. I changed it to 0.1Hz. Now we get some real data.

The result is lower when I change the window but still not the same as alog post. A factor of 2 larger at 30Hz.

No. Did not reproduce it. Attachment 3 is the same plot without restrictions on the y axis range.

Trying to see if I can reproduce https://alog.ligo-la.caltech.edu/aLOG/uploads/43363_20190213151300_subtraction_offset.png when I use average of 10.

More avg - more smoothiness but the shape of the noise curve is the same.

The formula used to calculate the noise is

Noise = Sqrt[1 - coherence] * drive

ETMY is about the same as ITMX.

Going to see if # of average makes any difference.

Will compare ITMX with ETMY.

Plotted.

L1 noisemon data for all four channels.

Increase avg to 1000. We are going to calculate the DAC noise from this.

I calculated the noise based on L1 data. My calculation is

(1-coherence) * drive = noise in DAC counts

noise in volts = noise in DAC counts * 20 / 2^16

Is this correct?

Noisemon installed in ITMX at L1. I pulled the 1-coherence data. trying to compare with Valery's measurement: https://alog.ligo-la.caltech.edu/aLOG/index.php?callRep=43240.

When we connect the voltage monitor channel of the noisemon board to a long cable (100ft), the op amp (LT1792) oscillates. Usually putting a 50 ohm resistor at the end will fix it. In this post, I studied how the oscillation happens and why putting a 50 ohm resistor will fix it.

We know 1) op amp has a dominant pole, giving a phase shift of 90 degrees 2) op amp oscillates when the loop gain is unity and the phase shift is 180 degrees. 3) Op amp has some non-zero output resistance.

Based on 3), we can see that when the output is capacitively loaded, there will be another pole in the transfer function due to the RC configuration. Since both R and C are small, it will be at high frequency (as op amp oscillations usually are). Thus, beyond the dominant pole, the phase will keep shifting to 180 degree based on 1). When this happens before the loop gain drops to unity, there will be oscillation based on 2).

Fix: insert a resistor at the output. This fixes the problem since it adds a zero with frequency a bit higher than the parasitic pole. This zero pulls the phase up so that when the loop gain reaches unity, the phase is around 90, at least far from 180, preventing oscillation from happening. The transfer function of this is simulated in LISO. From the plot, we can see the effect of the output resistance pulling the phase up to zero (90 in the case of an opamp because of the dominant pole).

Voltage monitor test data. Theory: 20 * log(1/2 * 1/3) = -15.6dB. ~1dB less attenuation from calculation.

Transfer function. The simulation and the test results differs by a phase of 180 degree (attachment 3). I added 180 degree got attachment 1.

Noisemon transfer function with phase, in dB counts.

I forgot to add the ADC noise. Now that I put the ADC noise there, the noise matches beautifully.

Noise compared with LISO.

In the region we care about noise (20 - 100 Hz), we can see it matches well with LISO calculations.

But why not at all frequencies ??

Analyzed Noisemon + Coil driver TF. The coil driver is in LP_OFF and ACQ_OFF status. The TFMeasured.txt file in the zip is in counts (ADC over DAC) so we need to add 20log₁₀(2) to convert it to volts.

Noisemon + Coil driver TF and noise data. This is raw data measured from the lab. TF data is in counts dB. It needs to be converted to volts dB (+6.02dB). The noise is in ADC counts: 2^16 counts is 40V.

I connected the DAC to ADC direclty (picture 1) and send a sine signal into the DAC. However, I did not get the sine signal back from the ADC. I sent the signal in X1:CRY-DITHER_W_MOD_EXC, channel 9 of DAC and expect the signal from X1:CRY-E_REFLDC_IN1, channel 16 of ADC. However, picture 2 shows what I get: a constant signal around 4400 counts.

1. Transfer functions of 1-4 channels, compare with simulations.

2. Noise of 1-4 channels, compare with simulations.

3. If feasible, nonlinearity.

4. Functionality of fast current, slow current and  voltage monitor channels.

All test results will reply to this post.

We ordered the board from Screaming Circuits and chose to provide the component ourselves. However, the parcel we ordered from Verical was lost by Fedex on its way to Screaming Circuits. The original delivery date was delay from late May to June 13.

Once the board arrives, we will test the board - TF, noise etc. Any others?

We added differential drivers at the outputs of all the monitors. After that the routing becomes impossible at the output connectors.

I replaced the signal ground with an additional signal layer, and reversed the order of the channels on the layout.

Having exhausted all the possible routing tricks, I finally managed to connect the whole board.

Attachment: new PCB schematics with all the changes made.

We (Chris and I) had a conversation with Rich last week and the following work on the noisemon board has been suggested:

1. Name of power nets: +VCC to +15, -VCC to -15. +V to +18; -V to -18, making it clearer what the power is.

2. Fix the off-grid problems of the schematics.

3. Draw the circuits on the schematics in the standard way. (Rich gave me a bunch of snippets that shows the standard way to draw the circuits, like how to draw a sallen key filter)

4. Ground the shells of the D connectors.

5. Add 1 Ohm resistors at the inputs of the power regulators

6. Use polymer tantalum with at least 35V rating. Previously we are not using polymer ones. Rich said the ones (non-polymer ones) we were using burn and explode sometimes.

7. Add "No error checking" for those pins not being used (e.g. unused op amp pins)

8. Disassemble the "repeat()" in the sheet symbol. Making four sheet symbols and connect them directly to the connectors.

9. Change the outputs of the current monitor, noise monitor and the voltage monitor to differential. Previously we had one of the pins of the D connectors pairs grounded. Now we add a differential driver at the end. It doubles the gain and the range.

Rich said my PCB routing was OK, so all the changes can be reflected on the schematics. I have made all the changes on the schematics (I do have the previous version). The current schematics is attached.

However, "#9 change the outputs to differential" requires a lot more space and the current PCB routing does not have enough free space between the components. Thus, this requires routing the whole PCB again, which is what I am working on now.

Some clean up work on the noisemon is done.

1. Added compensate capacitor.

2. Added mounting holes.

3. Added DCC number. https://dcc.ligo.org/LIGO-D1900052

4. Renumbered the components.

5. Added 0 ohm resistor between power ground and signal ground.

6. Added more test points for the voltage monitor and current monitor.

7. Increased schematics font size.

Next I will create the Bill Of Materials. I need to assemble the manufacturer information and put meaningful and consistent descriptions for the components.

Before making a wide deployment, we should also test the latest noisemon circuit for downconversion.

1. Measure the noise output with no DAC signal
2. Measure the noise output with 0.1x the reference DAC signal (low noise ETM drive)
3. Drive a line at high frequency in addition to the reference signal
4. adjust the ampltiudes of the high frequency drive and the reference signal independently and look at how the 20-100 Hz noise changes.

Based on the test results posted, I did the following analysis:

1. Compared measured transfer function to the LISO calculations. Attachment 4 and 6. The transfer functions match well with LISO.

2. Compared measured noise at the output to the LISO calculations. Attachment 1 and 3. The noise is more than LISO calculations by roughly a factor of 2, but I think it is expected - there is coil driver noises (amplified more than 300 times). Also, LISO uses ideal resistors, considering that the noise here is dominated by resistor noise. We also have plots of the noise spectrum with DAC noises injected. In this case, the noise in the passband (20 - 100Hz) is much more, suggesting that the board noise is dominated by the DAC noise.

3. Compared input-referenced measured noise to DAC noise. Attachment 2 and 5. We divided the noise by the transfer function and compare it directly with the DAC noise model. We can see that, in the passband, the board noise is about a factor of 10 less than the DAC noise (channel 2 and 4 has more noise; the signal is polluted by the ADC). 

4. A simple calculation based on the transfer function comparing the ADC noise and the amplified DAC noise. 

DAC noise > 300nV/rtHz. Passband amplification > 50dB > 300. Amplified DAC noise in the passband > 90uV/rtHz, compared to ADC noise 4uV/rtHz.

FYI: 1. I cannot attachment PDF plots directly since it will stuck the elog server. I put some PNG plots, but PDF plots can still be found in the compressed files.

2. Also, channel 2 is more noisey. It comes from ADC not the noisemon.

Duo's noisemon has been in the EE shop/cryo lab for testing.  It is a drop-in replacement for the existing monitor board, including both noisemon and Vmon/Imon/RMSmon circuits for all four channels.

Duo is still working on a log entry summarizing the performance of the new board vs simulation.  This entry shows some measurements of the performance of the new board vs the old board.

Attachments:

1. Transfer functions of the new noisemon board, in coil driver state 1.
2. Noise spectra of the new noisemon board, in coil driver state 1, with and without DAC noise input.
3. Transfer function of an old noisemon board, in coil driver state 1.
4. Noise spectra of an old noisemon board, in coil driver state 1, with (REF traces) and without (live traces) DAC noise input.

Two more oscillations problems are resolved, and there is no more oscillations. In the time series (the inputs are terminated), we see only the 60Hz noise.

- Some big bypass capacitors are used to regulate the power.

- A small capacitor is attached to the negative feedback loop in the second HP filter.

New board/components arrived. I will assemble and test them immediately.

After a few days of struggling (and essential help from Chris), mystery is resolved. Fortunately, the oscillation does not have much to do with my circuit design. It is caused by the RLC resonance formed by 1) the inductance of the parallel wires + 2) capacitance of the signal ground plane and the power ground plane.

As is seen in the picture, I twisted the two grounding wires together (reduce the inductance) and the oscillation is gone.

You can also connect the planes on the board (removing the capacitance) and the oscillation will disappear as well.

It seems there can be multiple reasons for an op amp to oscillate. I wanted to identify the nature of the oscillation.

I want to isolate one stage and see what is going on. I used the extra empty board and assembled the last stage on it. Putting in nothing at all (the input is GND), I get a signal of 5.792MHz, 321mV at the output.

Now that the problem is even more clear, I will keep looking into this.

Photo attached in attachment 1.

The times series output is shown in attachment 2 (Attached picture since I cannot get data from the oscilliscope, which requires floppy disk data transfer). There is an 87mV/rtHz oscillation at about 1.4MHz (op amp oscillation?).

I tested the noise with SR785, both time and frequency domains, in attachment 4 and 5. In time domain, I only see the 60Hz noise, not the 1.4MHz one (maybe because SR785 does not reach that high frequency). In frequency domain, noise in the passband is generally less than 10uVrms/rtHz. With a gain of 125, 10uV/rtHz corresponds to roughly 100nV/rtHz.

Attachment 3 is the transfer function, which is as we wanted, with a gain 2.5 less since this version does not have the last stage.

Internal saturation: what input do we use to test it?

Note: the noise FFT measurement has a lot of time dependence. It fluctuates a lot. Also sometimes (just a few hours before this measurement), I cannot reproduce the noise measurement mysteriously - it gives me much higher noise.

The reason for this problem was found. The gain of the sallen key filters was too high. There is an intrinsic limit of the sallen key filters - they cannot have a gain more than a certain value. Otherwise, they will be unstable. See this TI document  for details.

Here is a full version of the noisemon, with four channels and the power regulator. I did the routing again since the previous routing 1) did not leave enough space for connectors/other components; 2) Altium does not transfer properly from the schematics to the PCB layout when expanding to 4 channels.

1. add photo of stuffed board
2. add time series of output with input terminated
3. check for internal saturations
4. use the software from Craig to download and plot the SR785 data