See trend. This is NOT symptomatic of some frozen slow machine - if I disable the WFS servo inputs, the lock holds just fine.

Turns out that the beam was almost completely missing the WFS2 QPD. WTF 😤. I re-aligned the beam using the steering mirror immediately before the WFS2 QPD, and re-set the dark offsets for good measure. Now the IMC remains stably locked. 

Please - after you work on the interferometer, return it to the state it was in. Locking is hard enough without me having to hunt down randomly misaligned/blocked beams or unplugged cables.

I took this opportunity to do some WFS offset updates.

• First I let the WFS servo settle to some operating point, and then offloaded the DC offsets to the IMC suspensions.
• Then I disabled the WFS servo.
• I hand-tweaked MC1 and MC3 PIT/YAW (while leaving MC2 untouched) to minimize IMC REFL (a more sensitive indicator of the optimal cavity alignment than the transmission).
• Once I felt the IMC REFL was minimized (~1-2% improvement), I set the RF offsets for the WFS while the IMC remained locked. I chose this way of setting the RF offsets as opposed to unlocking the cavity and having the high-power TEM00 mode incident on the WFS QPDs.
• Overnight, I'm going to run the MC2 spot position scanning code (in a tmux session on pianosa, started ~945pm) to see if we can find a place where the transmission is higher, looking at Kruthi's code now to see it makes sense...
• The convergence time of the MC2 spot position loop is pretty slow, so the scan is expected to take a while... Should be done by tomorrow morning though, and I expect no work with the IFO tonight.
• Does this loop have to be so slow? Why can't the gain be higher?

Now with the CM board tested with the signal injected, it turned out that the latch logic was flipped. As the default state locked the digital levels, the buttons other than the mbbo channels were inactive.

By giving 0 to C1:LSC-CM_LATCH_ENABLE, the modification of the digital state is enabled. And with the value of 1, the digital bits on the board is locked.

In order to reflect this, latch.py was modified and now the controls are all activated.

[Koji]

The logic chips 74ALS573 were replaced. And now the gain sliders are working properly.

== Test Status ==

[done] Whitening gain switching test
[done] AA enable/disable switching
[0th order] LO Det Mon channel check
[none] PD I/F board check
[done] QPD I/F board check
[done] CM Board
[none] ALS I/F board

Last week we found that the logic chip for the REFL1 gain switching was not transmitting the input logic. I went to Downs and obtained the chips. After some inspection some other latch chips were suspicious. Therefore U46, U47, and U48 (#1, #3, and #4 from the top) were replaced. After the replacement, the gain measurements were repeated. This time the test for the AO gain was also performed. Now all three slideres show the gain as expected except for the consistent -0.2dB deficit.

Note that the transfer functions for the REFL gains were measured with the input at IN1 or IN2 and the output at TESTA1. The TFs for the AO gain was measured with the excitation at EXC B, the input at TESTB2 and the output at the SERVO output. The gain and phase variantions for the AO gain at low frequency is the effect of AC coupling existing between the excitation and the servo output.

[Update on Oct 14, 2019]

The measured transfer functions show the phase delay determined by the opamps involved. The phase delay well below the pole frequencies can be represented well by a simple time delay (a phase delay linear to the frequency). Attachment 7 shows the time delay estimated by LISO for each gain setting of each gain stage. REFL2 has particularly large phase delay because of the use of OP27s. The delay is even larger when the gain is high presunmably because of the limited GBW.

Yesterday, Koji and I noticed (from the wall StripTool traces) that the vertex seismometer RMS between 0.1-0.3 Hz in the X-direction increased abruptly around 6pm PDT. This morning, when I came in, I noticed that the level had settled back to the normal level. Trending the BLRMS channels over the last 24 hours, I  see that the 0.3-1 Hz band in the Z direction shows some anomalous behaviour almost in the exact same time-band. Hard to believe that any physical noise was so well aligned to the seismometer axes, I'm inclined to think this is indicative of some electronics issues with the Trillium interface unit, which has been known to be flaky in the past.

Summary:

There is ~ 7% variation in the power seen by the MC2 trans QPD, depending on the WFS offsets applied to the MC2 PIT/YAW loops. Some more interpretation is required however, before attributing this to spot-position-dependent loss variation inside the IMC cavity.

Analysis:

Attachment #1: This shows a scatter plot of the MC2 transmission and IMC REFL average values after the WFS loops have converged to the set offset positions. The size of the points are proportional to the normalized variance of the quantity. The purpose of this plot is to show that there is significant variation of the transmission, much more than the variance of an individual datapoint during the course of the averaging (again, the size of the circles is only meant to be indicative, the actual variance in counts is much smaller and wouldn't be visible on this plot scale). For a critically coupled cavity, I would have expected that the TRANS/REFL to be perfectly anti-correlated, but in fact, they are, if anything, correleated. So maybe the WFS loops aren't exactly converging to optimize the inoput pointing for a given offset? 

Attachment #2: Maps of the transmission/reflection as a function of the (YAW, PIT) offset applied. The radial coordinate does not yet mean anything physical - I have to figure out the calibration from offset counts to spot position motion on the optic in mm, to get an idea for how much we scanned the surface of the optic relative to the beam size. The gray circles indicate the datapoints, while the colormaps are scipy-based interpolation. 

Attachment #3: After talking with Koji, I explicitly show the correlation structure between the IMC REFL DCMON and MC2 TRANS. The shaded ellipses indicate the 1, 2 and 3-sigma bounds for the 2D dataset going radially outwards. The correlation coefficient for this dataset is 0.46, which implies moderate positive correlation. 🤔 

Scan algorithm:

The following was implemented in a python scipt:

1. Choose 2 independent random numbers from the uniform distribution in the interval [-0.5, 0.5] (in uncalibrated counts).
2. One of these numebrs is set as the error point offset for the QPD spot-centering PITCH WFS loop, while the other is the YAW offset.
3. Wait for 600 seconds - this long wait is required because the step-response time for these loops is long. 
4. If there is an MC unlock event - wait till the MC relocks, and then another 600 seconds, to give the WFS loops sufficient time to converge.
5. Once the WFS loops have converged, average a few data channels (MC TRANS, REFL, WFS loop error points etc) for 10 seconds, and write these to a file.

I am now setting the offsets to the WFS QPD loop to the place where there was maximum transmission, to see if this is repeatable. In fact it was. Looking at the QPD segment outputs, I noticed that the MC2 transmission spot was rather off-center on the photodiode. So I went to the MC2 in-air optical table and centered the beam till the output on the 4 segments were more balanced, see Attachment #4. Then I re-set the MC2 QPD offsets and re-enabled the WFS servos. The transmission is now a little lower at ~14,500 counts (but still higher than the ~14200 counts we had before), presumably because we have more of the brightest part of the beam falling on the gap between quadrants. For a more reliable measurement, we should use a single-element photodiode for the MC2 transmission.

I simulated this circuit with zero, but haven't gotten the results to match the measurements above.

I think this offset setting thing is not so good. People do this every few years, but putting offsets in servos means that you cannot maintain a stable alignment when there are changes in the laser power, PMC trans, etc. The better thing is to do the centering of the WFS spots with the unlcoked beam after the control offsets have been offloaded to the suspensions.

== Test Status ==

[done] Whitening gain switching test
[done] AA enable/disable switching
[0th order] LO Det Mon channel check
[none] PD I/F board check
[done] QPD I/F board check
[done] CM Board
[none] ALS I/F board

The photos of the latest board can be found as Attachments 3/4

With some input signals, the functionarities of the CM servo switches were tested.

• Latch logic works. But latch alive signal is missing.
• IN1 enable/disable, IN2 enable/disable are properly working
• OUT2 toggle switch for REFL1/REFL2 mon is wokring
• Boost / Super Boosts are working
• EXC A enable/disable, EXC B enable/disable switches are working
• Option 1 and Option 2 now isolate the input when either is enabled (as there is no option board)
• 79Hz-1.6kHz pole zero pair works fine
• OUT1 works fine
• Disable/Enable switch for the fast path works
• Polarity switch works
• AO Gain property changes the gain
• Limitter switch works (Attachments 4/5). The limitter clipps the output at 4~4.5V. The Limitter indicator also works.

After the tests the LSC cables were reconnected (Attachment 6)

In preparation for some locking work tonight, I did the following at the POP in air table with the PRMI locked on carrier:

1. Raised the POP camera by ~5mm. The POP spot is now well centered on the CCD view.
2. Tweaked alignment onto the PDA10CF photodiode that serves as (i) POP22, (ii) POP110, and (iii) POP DC. In lock the POPDC level went from ~800 cts to ~1200 cts.
3. Moved the QPD that witnesses part of the POP beam such that the spot was centered on the photodiode. This may be useful for collecting some FF data or if we want to try feedback to stabilize the PRMI.

TBC...

The boost filters of the CM servo board were tested. Their ZPK models were made.

The transfer functions of the boost filters were measured with the SG output of a SR785 connected to IN1. The IN1 gain was set to be 0dB. The transfer function was taken between the IN1 input and the TEST1A output.
With no boost and normal boost, the input signal amplitude was fixed to 20mVpk. For the other boosts, however, I could expect large gain variation through a single sweep. Therefore automatic SG amplitude tracking was used. The target was to have the output to be 1V with maximum amplitude of 100mV.

Attachment 1 shows the measured transfer functions.

The pole and zero frequencies of the boosts were estimated using LISO. Here the TFs were normalized by the TF of 'no boost' to cancel the delay of the other stages including that of the monitor channel.

ZPK model of Normal Boost:

pole 44.0597566447
zero 4.3927650910k
factor 98.8275377818

ZPK model of Super Boost (State1):

pole 878.5368382789
zero 17.5107366335k
factor 20.0840668188
 

ZPK model of Super Boost (State2):

pole 714.8112014271
pole 1.0147609373k
zero 13.2470941080k
zero 22.2259701828k
factor 404.5411036031
 

ZPK model of Super Boost (State3):

pole 886.3650348470
pole 420.4089305781
pole 887.8490768202
zero 8.3635166134k
zero 15.7953592754k
zero 20.5144907279k
factor 8.2051379423k

Looking at the old latch.st code, looks like this is just a heartbeat signal to indicate the code is alive. I'll implement this. Aesthetically, it'd be also nice to have the hex representation of the "*_SET" channels visible on the MEDM screen.

I installed nds2 on donatello with yum, but still can't import nds2.

I installed nds2 again, this time successfully with

It would be good if you and Shruti can look at how to change the parameters in Zero so as to do a fit to the measured data. Usually, in scipy.optimize we give it a function with some changeable params, so maybe there's a way to pass params to a zero object in that way. I think Ian and Anchal are doing something similar to their FSS Pockel's cell simulator.

After making sure the beams were hitting the 3f photodiodes on the "AP" table, I was able to lock the PRMI with the sidebands resonant inside the RC using 3f error signals. This would be the config we run in when trying to lock some more complicated configuration, such as the PRFPMI (i.e. start with the arms controlled by ALS, held off resonance). Tonight, I will try this (even though obviously I am not ready for the CARM transition step). The 3f lock is pretty robust, I was able to stay locked for minutes at a time and re-acquisition was also pretty quick. See Attachment #1. Not sure how significant it is, but I set the offsets to the 3f paths by averaging the REFL33_I and REFL33_Q signals when the PRMI was locked with the 1f error signals.

As usual, there's a lot of angular motion of the POP spot on the CCD monitor, but the lock seems to be able to ride it out.

Lock-settings (I modified the .snap file accordingly):

REFL33_I --> PRCL, loop gain = -0.019, Trigger on POP22, ON @ 20cts, OFF@0.5cts.

REFL33_Q --> MICH, loop gain = +1.4, Trigger on POP22, ON @ 20cts, OFF@0.5 cts.

This problem has re-surfaced. Is this indicative of some problem with the on-board VGA? Even with 0dB of whitening gain, I see PDH horns that are 10,000 ADC counts in amplitude, whereas the nominal whitening gain for this channel is +18dB. I'll look at it in the daytime, not planning to use REFL55 for any locking tonight.

Summary:

1. ALS control of arms in the CARM/DARM basis seems pretty robust - I was able to hold lock for >40mins tonight. The scripted transition from POX/POY control to ALS control is pretty deterministic now.
2. The PRMI could be locked with the arms detuned from resonance by applying an offset to the CARM loop error point.
3. Much daytime work remains to be done before attempting any sort of reliable locking.

Hardware issues that need addressing:

1. Both EX and EY Trans QPDs need a look. I believe the one at EY is simply blocked (on account of the mode spectroscopy project), while the one at EX shows a weird discontinuity between the Thorlabs PD and the QPD. Could be just a gain/normalization issue I guess. See Attachment #1.
2. While the PRMI stayed locked, I don't think I was using anywhere close to optimal settings. Need to run some sensing lines, measure transfer functions etc, to make the PRMI + arms lock more robust. The PRMI always lost lock when I brought the CARM offset to 0. Could also benefit from some finesse modeling I guess. I could not get a reliable estimate of what the PRG is tonight, because the PRMI didn't stay locked as I approached 0 CARM offset.
3. REFL 55 whitening board needs a checkup.

1. I removed the flip-mount that was installed on the EY in-air table for the mode-spectroscopy project (see Attachment #1). The Transmon QPD at EY sees IR light again.
2. Dark noise checkout - see Attachment #2.
3. Light-level expectations:
   • For the current config, let's say 0.8 W reaches the PRM, and we will have a PRG of 50. 
   • This implies ~5.5 kW circulating power in the arms.
   • This implies ~70mW will get transmitted through the ETM, of which at most half makes it to the QPD. 
   • In the nominal operating condition, we expect more like 6 W circulating in the arm cavity. So something like 30uW is expected to make it out onto the Trans QPDs.
   • But in this condition, we expect to run with the high-gain Thorlabs PD.
   • In reality the number is likely to be somewhat smaller. But we should set the transimpedance gain of this photodiode accordingly. Currently, there are a bunch of ND filters installed on this photodiode, which probably should be removed.
4. Angular control
   • The other purpose these QPDs are expected to serve is to stabilize the angular motion of the cavities when locked with high circulating power.
   • Need to calculate what the sensing noise requirement is.

[Jon, Yehonathan, Gautam, Aaron, Shruti, Koji]

We get together on Wednesday afternoon for cleaning the lab. Particularly, we collected e-wastes: VME crates, VME modules, old slow control cables, and other old/broken electronics. They are piled up in the office area and the cage outside rioght now (Attachments 1/2). We asked Liz to come to pick them up (under the coordination with either Gautam or Koji). Eventually this will free up two office desks.

Also, we made the acromag components organized in plastic boxes. (Attachment 3)

CM Board Slow out (digital length control) path transfer function / pole-zero filter pair (79Hz/1.6kHz) transfer function

The excitation was given from EXC A. The denominator was TESTA2, and the numerator was OUT1.

Attachment 1 shows the measured transfer function with and without PZ filter off and on. The PZ filter provides ~26dB attenuation at  high frequency. The output stage has a single order 100kHz LPF and it is visible in the transfer function.

The transfer function without the PZ filter was modelled by LISO as the following PZK representation. There looked a small step in the TF which caused the additional PZ pair (66~67Hz) but has very minor effect in the mag and phase.

pole 66.2720207366
zero 67.2660731875
pole 93.3044858160k
factor -995.5583556921m

The transfer function of the PZ filter was separately analyzed. The TF with the switch ON was normalized by the one with the switch OFF. Thus it revealed the pure effect of the switch. The PZK model of the stage was estimated to be

pole 79.7312926438
zero 1.6395485993k
factor 996.2196584165m

For the CM board modeling purpose, the transfer function from TESTA2 to TESTB2 was needed. (Attachment 1)

The ZPK model of this part is

pole 76.2369881805
zero 77.4655685092
pole 7.0761486105M
factor -993.0593433578m

The output stage (and AO GAIN stage) of the MC board was modelled. The transfer function was measured with the injection from EXC B. The denominator was TESTB2, and the numerator was SERVO OUT.

This stage is AC coupled by 2x 1st order HPFs. Firstly, this transfer function was measured with AO GAIN set to be 0dB. (Attachment 1)
This TF was used to characterize the cutoffs of the HPF stages, represented as the following ZPK:

zero 1m
zero 1m
pole 6.0502599855
pole 6.0624642854
factor -26.2725046079n

Then the AO GAIN was already measured as seen in [ELOG 14948]. The AO gain TF was then modeled by LISO with the above HPF as the preset. This allows us to characterize the time delay of the AO GAIN part.

Input referred offsets on the IN1/IN2 were tested with different gain settings. The two inputs were plugged by the 50 ohm terminators. The output was monitored at OUT1 (SLOW Length Output). The fast path is AC coupled and has no sensitivity to the offset.

There is the EPICS monitor point for OUT1. With the multimeter it was confirmed that the EPICS monitor (C1:LSC-CM_REFL1_GAIN) has the right value except for the opposite sign because the output stage of OUT1 is inverting. The previous stages have no sign inversion. Therefore, the numbers below does not compensate the sign inversion.

Attachment 1 shows the output offset observed at C1:LSC-CM_REFL1_GAIN. There is some gain variation, but it is around the constant offset of ~26mV. This suggested that the most of the offset is not from the gain stages but from the later stages (like the boost stages). Note that the boost stages were turned off during the measurements.

Attachment 2 shows the input refered offset naively calculated from the above output offset. In dependent from which path was used, the offset with low gain was hugely enhanced.

Since the input referred offset without subtracting the static offset seemed useless, a constant offset of -26mV was subtracted from the calculation (Attachment 2). This shows that the input refered offset can go up to ~+/-20mV when the gain is up to -16dB. Above that, the offset is mV level.

I don't think this level of offset by whichever OP27 or AD829 becomes an issue when the input error signal is the order of a volt.
This suggests that it is more important to properly set the internal offset cancellation as well as to keep the gain setting to be high.
 

The UPS is now incessantly beeping. I cannot handle this constant sound so I shut down all the control room workstations and moved the power strip hosting the 4 CPUs to a wall socket for tonight. Chub and I will replace the UPS batteries tomorrow.

Updated Circuit Diagram and photos: https://dcc.ligo.org/D1500308-v2

- (1) and (6) of the diagram: TFs with various gain slider values for REFL1/REFL2/AO GAIN [ELOG 14948] (gain values and time delay modeling)
- Switching checks, latest photo of the board, Limiter check  [ELOG 14953]
- (2): Boost transfer functions [ELOG 14955]
- (3): Slow (aka Length) CM output path [ELOG 14965]
- (4): Pole-Zero filter TF [ELOG 14965]
- (5): TF from TESTA2 to TESTB2 [ELOG 14966]
- (6): AC coupling TF of the AO GAIN stage [ELOG 14967]
- (7): AC coupling TF of the IN2 stage on IMC servo board [ELOG 15044]

Slow path = (1)*(2 if necessary)*(3)*(4 if necessary)

Fast path = (1)*(2 if necessary)*(4 if necessary)*(5)*(6)

gautam 20191122: Adding the measured AC coupling of the IN2 input of the IMC servo board for completeness.

[Liz, Gautam, Chub, Jordan, Koji]

We removed a significant amount of e-waste from the lab. The garbage was moved to the e-waste station in WB SB and are waiting for disposal.

Batteries + power cables replaced, and computers back on UPS from today ~3pm.

Summary:

There is poor separation of the PRCL and MICH length error signals as sensed in the 3f photodiodes. I don't know why this is so - one possibility is that the MICH-->PRM matrix element in the LSC output matrix needs to be tuned to minimize the MICH -->PRCL coupling.

Details:

Over the last few days, I've been trying to make the 3f locking of the PRMI more reliable. Turns out that while I was able to lock the PRMI on 3f error signals, it was just a fluke. So I set about trying to be more systematic. Here are the steps I followed:

1. Lock the PRMI (i.e. ETMs misaligned) using REFL11 for PRCL, AS55 for MICH.
   • This is the so-called 1f scheme.
   • The servo signs are chosen such that the carrier field is resonant in the PRC.
   • Run the dither alignment to maximize POPDC, minimize ASDC. This is the main purpose of locking in this config.
   • Measure some loop TFs, make sure the servo gains are giving us ~100 Hz UGF on these loops.
2. Change the sign of the servo loops to make the sidebands resonant in the PRC.
   • The error signals are still sourced from the 1f photodiodes.
   • Measure loop TFs, and also the TF between the 1f and 3f error signals. 
   • This allowed me to determine how the servo gains (and signs) that would be appropriate when using the 3f signals in place of the 1f.
   • Determine the offsets in the 3f error signals when the PRMI is locked on 1f error signals. This allows me to set the error point offsets for the PRCL_B and MICH_B paths, which are what is used for the 3f locking.
3. Change the error signals from 1f to 3f. 
   • After much trial and error, I finally managed to get a stable (>10 mins) lock going.
   • Measured some loop TFs.
   • Turned on the notch filters in the control filter bank at the sensMat oscillator frequencies, and ran some sensing lines.

Attachment #1 is the result. I don't know what is the reason for such poor separation of the MICH and PRCL error signals in REFL165. The situation seems very different from when I had the DRMI locked in Nov last year.

After this exercise, I tried for some hours to get the 3f PRMI locking going with the arm cavities held off resonance under ALS control, but had no success. The angular motion of the PRC isn't helping, but I feel this shouldn't be a show stopper.

1. Tuning the MICH-->PRM output matrix element
   • Locked the PRMI with the carrier field resonant in the PRC.
   • REFL11 used to control PRCL, AS55 for MICH.
   • Turned on the sensing notches in the control filter bank. Drove a line in MICH at 311.10 Hz.
   • Tweaked the MICH-->PRM matrix element to minimize the coupling witnessed.
   • As shown in Attachment #1, the minimum coupling was found to be at the value -0.34 (the old value was -0.2655).
   • The minimum was very sharp. A 1% change from the optimum value increased the peak height by > x2. Is this reasonable?
2. Some sensing matrix measurements: After tuning the output matrix element, I locked the PRMI (ETMs misaligned) in four configurations:
   • PRMI locked with carrier resonant. REFL11_I used for PRCL control, AS55_Q used for MICH control.
   • PRMI locked with sidebands resonant. REFL11_I used for PRCL control, AS55_Q used for MICH control.
   • PRMI locked with sidebands resonant. REFL11_I used for PRCL control, REFL165_I used for MICH control (based on sensing matrix measurement and offsets from previous config).
   • PRMI locked with sidebands resonant. REFL33_I used for PRCL control, AS55_Q used for MICH control.
   • The attached GIF shows the evolution of the demodulated sensing lines as we move through configurations.
      
   • The actual PDFs are attached as a zip, Attachment #2.
3. PRMI locking with arms under ALS control
   • The arm cavity lengths were controlled as usual with ALS. This system needs some noise budgeting.
   • I set the CARM offset to -8 (arbitrarily chosen, approximately equal to 20nm, but anyways well above the cavity linewidth).
   • Then I re-aligned the PRM, and attemped to lock the PRMI using the 3f settings determined with no arm cavities --> no success.
   • Tried locking using the 1f error signals instead - in this config, the lock could be established.
   • However, I saw that there was significant light on the AS camera, and I had to put in an offset into the MICH loop to make ASDC go as low as possible.
   • I guess it is possible that the ALS control wasn't precise enough and the leaked light to the dark port was because of differential reflectivity of the arm cavities?
   • Anyways, I ran a sensing matrix measurement with the interferometer in this configuration, and I found that the MICH signal in REFL165 had rotated significantly.
   • I also found that the 3f DC offsets in this configuration were ~5x greater than what was the case for the lock with no arm cavities.

This is as far as I got last night. The first step is to see how reliable the settings determined last night are, today. I don't understand how changing the output matrix element can have brought about such a significant change in the MICH/PRCL separation in all the RF photodiodes.

Some ideas that would help increase the locking duty-cycle in the short term. 

1. Seismometer investigation - something is not quite right with the vertex seismometer. This is the one that is primarily used for feedforward, and can be really helpful.
2. Drifting TTs - it is really annoying to have to re-set the input pointing into the interferometer every ~ hour. See Attachment #1.
3. FSS - this isn't a scientific statement, but there were ~20-30 minute periods last night where the PC drive RMS was displaying sharp spikes repeating every 2-3 seconds, first with increasing and then decreasing height. This is a new feature to me in the long standing PC drive saga but it doesn't tell me exactly what is going on as I don't know in what frequency band the glitch is actually happening. See Attachment #2.
4. ALS noise - while it is possible now to routinely transition the arm length control from the POX/POY to CARM/DARM basis, I see some sharp (<0.1 s) dives in the TRX/TRY levels when the arms are under ALS control. This wasn't present a week ago. Needs to be investigated - I defer this to the daytime tomorrow.

I tried implementing a basic PRMI ASC using the POP QPD as a sensor. The POP22 buildup RMS is reduced by a factor of a few. This is just a first attempt, I think the loop shape can be made much better, but the stability of the lock is already pretty impressive. For some past work, see here.

Koji suggested systematic investigation of the ETMX suspension electronics. The tests to be done are:

1. Characterization of PD whitening amplifiers - with the satellite box disconnected, we will look for glitches in the OSEM channels.
2. Characterization of LT1125s in the PD chain of the amplifiers - with the in-vacuum OSEMs disconnected, we will look for glitches due to the on-board transimpedance amplifiers of the satellite box.
3. Characterization using the satellite box tester - this will signal problems with the physical OSEMs.
4. Characterization of the suspension coil driver electronics - this will happen later.

So the ETMX satellite box is unplugged now, starting 530 pm PDT.

The satellite box was reconnected and the suspension was left with watchdog off but OSEM roughly centered. We will watch for glitches over the weekend.

I've checked the state of the laser interlock switch and everything looked normal.

[Shruti, Rana]

- At the X end, we set up the network analyzer to begin measurement of the AM transfer function by actuation of the laser PZT.

- The lid of the PDH optics setup was removed to make some checks and then replaced.

- From the PDH servo electronics setup the 'GREEN_REFL' and 'TO AUX-X LASER PZT' cables were removed for the measurement and then re-attached after.

- The signal today was too low to make a real measurement of the AM transfer function, but the GPIB scripts and interfacing was tested. 

We also took this opportunity to re-connect the interlock to the Innolight controller (after it was disconnected for diagnosing the mysterious NPRO self-shutdowns). The diode pump current was dialled down to 0, the interlock wires reconnected, and then the diode current was ramped back up to the nominal 2.1 A. The fan to cool the unit remains mounted in a flaky way as we couldn't locate the frame Chub had made for a more secure mounting solution. 

It seems like the pointing of the beam out of the laser head varies somewhat after the startup - I had to adjust the pointing into the PMC a couple of times by ~1 full turn of the Polaris mount screws, but the IMC has been locked (mostly) for the last ~16 hours.

Summary:

There are no unexpected red-flags in the performance of the DFD electronics. The calibration factors for the digital phase tracker system are 71.291 +/- 0.024 deg/MHz for the X delay line and 70.973 +/- 0.024 deg/MHz for the Y delay line, while the noise floor for the frequency noise discrimination is ~0.5 Hz/rtHz above 1 Hz (dominated by ADC noise).

Details:

1. Attachment #1 - This observation is what motivated my investigation.
   •  found that for certain beat frequencies between the PSL + EX lasers, the frequency noise reported by the DFD system was surprisingly low.
   • The measurement condition was: EX laser frequency locked to the arm cavity length by the uPDH servo at EX, arm cavity length locked to PSL frequency via POX locking.
2. To investigate further, I disconnected the output of the NF1611 PDs going to the ZHL-3A amplifiers on the PSL table (after first blocking the PSL light so that the PDs aren't generating any RF output).
   • An RF function generator (IFR2023B) was used to generate an RF signal to mimic the ALS beat signal.
   • I used a power splitter to divide the signal power equally between the two DFD paths.
   • The signal level on the Marconi was set to -5 dBm, to mimic the nominal power level seen by the DFD system.
   • I then performed two tests - (i) to calibrate the Phase Tracker output to deg / MHz and (ii) to measure the frequency noise reported by the DFD system for various signal frequencies.
   • Test (i): sweep the marconi frequency between 10 MHz - 200 MHz, measure the I and Q channels for each phase tracker servo, and figure out the complex argument of the signal using the arctangent. A linear polynomial was fit to the measured datapoints to extract the desired slope.
   • Test (ii): Sample frequencies uniformly distributed between 20 MHz - 80 MHz (nominal range of ALS beat frequencies expected). Reset the phase tracker servo gain, clear the output histories, wait for any transients to die out, and then collect the phase tracker servo output for 1 minute. Compute the FFT to figure out the frequency noise.
   • Attachment #2: Shows the phase tracker calibration, i.e. the results of Test (i). I took this opportunity to update the EPICS calibration fields that convert phase tracker servo output to Hz, the correction was ~7%. These numbers are consistent with what I measured previously - but the updated values weren't registered with SDF so everytime the LSC model was restarted, it reverted to the old values.
   • Attachment #3: Shows the spectra for the various measurements from Test (ii).
   • Attachment #4: Shows the gain of the phase tracker servo as a function of the RF signal frequency. This is a proxy for the signal strength, and the observed trend suggests that the signal power seen after digitization of the demodulated delay line output goes down by ~20% at 80 MHz relative to the level at 20 MHz. Seems reasonable to me, given frequency dependent losses of the intervening electronics / cabling.

Conclusion and next steps:

I still don't know what's responsible for the anomalously low noise levels reported by the ALS-X system sometimes. Next test is to check the EX PDH system, since on the evidence of these tests, the problem seems to be imprinted on the light (though I can't imagine how the noise becomes lower?).

Looking at the sensor and oplev trends over the weekend, there was only one event where the optic seems to have been macroscopically misaligned, at ~11:05:00 UTC on Oct 19 (early Saturday morning PDT). I attach a plot of the 2kHz time series data that has the mean value subtracted and a 0.6-1.2 Hz notch filter applied to remove the pendulum motion for better visualization. The y-axis calibration for the top plot assumes 1 ct ~= 1 um. This "glitch" seems to have a timescale of a few seconds, which is consistent with what we see on the CCD monitors when the cavity is locked - the alignment drifts away over a few seconds.

As usual, this tells us nothing conclusive. Anyways, I am re-enabling the watchdog and pushing on with locking activity and hope the suspension cooperates.

1. Transition of arms from POX/POY to CARM/DARM was much smoother today - a change was made at the EX PDH setup, see here.
2. Reliable settings for 3f locking with arms held off resonance seem to have been found.
3. Took sensing matrix in this condition, measured loop TFs.
4. Reduced CARM offset - reached arm powers ~50 at which point the PRMI lost lock. Reacquisition was quick though.
   • The POP22_I level seemed to decay as I reduced the CARM offset.
   • This would suggest that somehow the PRCL lock point is getting shifted as I reduce the CARM offset.
   • Tonight, I will investigate this more.

Summary:

The EX PDH setup had what I thought was insufficient phase and gain margins. So I lowered the gain a little - the price paid was that the suppression of laser frequency noise of the end laser was reduced. I actually think an intermediate gain setting (G=7) can give us ~35 degrees of phase margin, ~10dB gain margin, and lower residual unsuppressed AUX laser noise - to be confirmed by measurement later. See here for the last activity I did - how did the gain get increased? I can't find anything in the elog.

I made a change to the c1ass model to normalize the PIT and YAW POP QPD outputs by the SUM channel. A saturation block is used to prevent divide-by-zero errors, I set the saturation limits to [1,1e5], since the SUM channel is being recorded as counts right now. Model change is shown in the attached screenshots. I compiled and installed the model. Ran the reboot script to reboot all the vertex FEs to avoid the issue of crashing c1lsc.

During our EX AM/PM setups, I don't think we bumped the PDH gain knob (and I hope that the knob was locked). Possible drift in the PZT response? Good thing Shruti is on the case.

Is there a loop model of green PDH that agrees with the measurement? I'm wondering if something can be done with a compensation network to up the bandwidth or if the phase lag is more like a non-invertible kind.

The closest thing I can think of is here.

Attachment #1 - comparison of the POP QPD PIT and YAW output signal spectra with and without them being normalized by the SUM channel. I guess the shape is different between 30-100 Hz because we have subtracted out the correlated singal due to RIN?

This did not have the effect I desired - I was hoping that by normalizing the signals, I wouldn't need to change the gain of the ASC servo as the buildup in the PRC changed, but I found that the settings that worked well for PRMI locked with the carrier resonant (no arm cavities, see Attachment #2, buildup RIN reduced by a factor of ~4) did not work for the PRMI locked with the sideband resonant. Moreover, Koji raised the point that there will be some point in the transition from arms off resonance to on resonance where the dominant field in the PRC will change from being the circulating PRC carrier to the leaking arm carrier. So the response of the actuator (PRM) to correct for the misalignment may change sign. 

In conclusion, we decided that the best approach to improve the angular stability of the PRC will be to revive the PRC angualr feedforward, which in turn requires the characterization and repair of the apparently faulty vertex seismometer.

I looked into the seismometer situation a bit more today. Here is the story so far - I think more investigation is required:

1. There is an abrupt change in the PEM BLRMS channels around 6pm PDT every day. This has been consistently seen for the last two weeks.
2. The seismometer spectra look normal - see Attachment #1. The reference traces are from some months ago. There is elevated activity between 0.1-0.3 Hz, but this is seen in all the seismometers in all 3 DoFs.
3. I looked at the minute trend of the raw seismometer outputs (before being BLRMSed) for the last 200 days and don't see any abrupt change in characteristics (the data gap is due to the issue in this thread).
4. All the correct BLRMS filters seem to be engaged in the respective filter banks.

Attachment #2 has some spectrograms (they are rather large files). They suggest that the increase in noise in the 0.1-0.3 Hz band in the BS seismometer X channel is real - but there isn't a corresponding increase in the other two seismometers, so the problem could still be electronics related.

I wanted to restart the c1oaf model. As usual, the first time the model was restarted, it came back online with a 0x2bad error. This isn't even listed in the diagnostics manual as one of the recognized error states (unless there is a typo and they mean 0x2bad when they say 0xbad). The fix that has worked for me is to stop and start the model again, but of course, there is some chance of taking all the vertex FEs down in the process. No permutation of mxstream and daqd process restarts have cleared this error. We need some CDS/RCG support to look into this issue and fix it, it is not reasonable to go through reboots of all the vertex FEs every time we want to make a model change.

Summary:

I'd like to revive the PRC angular feedforward system. However, it looks like the coherence between the vertex seismometer channels and the PRC angular motion witness sensor (= POP QPD) is much lower than was found in the past, and hence, the stabilization potential by implementing feedforward seems limited, especially for the Pitch DoF.

Details:

I found that the old filters don't work at all - turning on the FF just increases the angular motion, I can see both the POP and REFL spots moving around a lot more on the CRT monitors.

I first thought I'd look at the frequency-domain weiner filter subtraction to get a lower bound on how much subtraction is possible. I collected ~25 minutes of data with the PRC locked with the carrier resonant (but no arm cavities). Attachment #1 shows the result of the frequency domain subtraction (the dashed lines in the top subplot are RMS). Signal processing details:

• Data was downloaded and downsampled to 64 Hz (from 2kHz for the POP QPD signals and from 128 Hz for the seismometer signals). The 'FIR' option of scipy decimate was used.
• FFT time used was 16 seconds for the multi-coherence calculations

The coherence between target signal (=POP QPD) and the witness channels (=seismometer channels) are much lower now than was found in the past. What could be going on here?

Summary:

The Trillium T240 seismometer needs mass re-centering. Has anyone done this before, and do we have any hardware to do this?

Details:

I went to the Trillium interface box in 1X5. In this elog, Koji says it is D1000749-v2. But looking at the connector footprint on the back panel, it is more consistent with the v1 layout. Anyway I didn't open it to check. Main point is that none of the backplane data I/O ports are used. We are digitizing (using the fast CDS system) the front panel BNC outputs for the three axes. So of the various connectors available on the interface box, we are only using the front panel DB25, the front panel BNCs, and the rear panel power.

The cable connecting this interface box to the actual seismometer is a custom one I believe. It has a 19 pin military circular type hermetic connector on one end, and a DB25 on the other. Power is supplied to the seismometer from the interface box via this cable, so in order to run the test, I had to use a DB25 breakout board to act as a feedthrough and peek at the signals while the seismometer and interface boards were connected. I used Jenne's mapping of the DB25--> 19 pin connector (which also seems consistent with the schematic). Findings:

1. We are supplying the Trillium with 39 V DC between the +PWR and -PWR pins, while the datasheet specifies 9V to 36V DC isolated. Probably this is okay?
2. The analog (AGND) and digital (DGND) ground pins are shorted. Is this okay?
3. I measured the DC voltages between the AGND pin and each of the mass position outputs.
   • These are supposed to indicate when the masses need re-centering.
   • The nominal output ranges for these are +/- 4 V single-ended.
   • I measured the following values (I don't know how the U,V,W basis is mapped onto the cartesian X,Y,Z coordinates):
     U_MP: 0.708 V
     V_MP: -2.151 V
     W_MP: -3.6 V
   • So at the very least, the mass needs centering in the W direction (the manual recommends doing the re-centering procedure when one of these indicators exceeds 3.5 V in absolute value).
4. I also checked the DC voltages of the (X,Y,Z) outputs of the seismometer on the front panel BNCs, and also on the DB25 connector (so directly from the seismometer). These are rated to have a range of 40 Vpp differential between the pins. I measured ~0V on all the three axes - this is a bit confusing as I assumed a de-centered mass would lead to saturation in one of these outouts, but maybe we are measuring velocities and not positions?
5. We probably should consider monitoring these signals long term to inform of such drifts, what is the spare channel situation in the c1sus acromag?
6. Interestingly, today evening, there is no excess noise in the 0.1-0.3 Hz band in the X-axis of the seismometer even though it is past 6pm PDT now, which is usually the time when the excess begins to show up. The z-axis 0.3-1Hz BLRMS channel has flatlined though...

I am holding off on attempting any re-centering, for more experienced people to comment.

[Koji, gautam]

Summary:

We think we got to the bottom of this issue today. The RF signal level going into the demod board is too high. This electronics chain needs some careful gain reallocation.

Details:

I was demonstrating to Koji a strange feature I had noticed in the ALS control, whereby when applying a CARM offset to detune the arms, the two arms seemed to respond differently (based on the transmission levels). This kind of CARM-->DARM coupling seemed strange to me. Anyway, I also noticed that the EPICS indicators on the ALS MEDM screen suggested ADC saturations were going on. I had never really looked at the fast time series of the inputs to the phase tracker servos, but these showed saturating behavior on ndscope traces. I went to the LSC rack and measured these on a scope, indeed, they were ~20V pp.

The output of the BeatMouth PDs are going to a ZHL-3A amplifier - we should consider replacing these with lower gain amplifiers, e.g. the Teledyne AP1053. This is relegated to a daytime task.

Other findings tonight:

While working on the PSL table, I somehow put the IMC FSS into a bad state, reminiscent of this behavior. Seems like this is linked to some flaky connection on the PSL table. One candidate is the unstable attachment of the Pomona box between the NPRO PZT and the FSS output - we should install a short BNC cable between these to avoid the lever arm situation we have right now.

back on new Rossa from Xi computing

1. switched to using Display Port for video; this works. The DVi, HDMI, VGA ports are connected to the motherboard rather than the video card, so they are not active.
2. runs super slow w/ SL 7.6; maybe some service is running after startup?
3. install repos and update according to LLO CDS wiki
4. add controls user and group according to LLO wiki
5. remove gstreamer ugly because it breaks yum update
6. run 'yum update --skip-broken' because GDS doesn't work
7. turn off old selinux stuff
8. modify fstab to get NFS

Next:

1. finish mounting
2. xfce
3. figure out why the LLO install instructions can't install any CDS software (e.g. root, DTT, etc)

Update: Sun Nov 3 18:08:48 2019

1. moved the SL7 fresh install repos back into etc/yum.repos.d/. The LLO instructions has me remove them, but the LLO supplied repos are no good for standard apps. After putting these back was able to install standard apps (terminator, root, diaggui)
2. copied over /etc/fstab lines from pianosa sothat the NFS mounts work correctly
3. added symlinks so that the NFS dirs mount in the right dirs
4. symlink libsasl2.so.3 -> libsasl2.so.2 and now DTT runs and can get data now and in the past
5. install XFCE
6. sitemap / MEDM works
7.  Did "sudo ln -s /usr/lib64/libXm.so.4 /usr/lib64/libXm.so.3" to enable StripTool.

Update: Fri Nov 15 00:00:26 2019:

1. random hanging of machine while doing various window moving or workspace switching
2. turned off power management in XFCE
3. turned off power management on monitor
4. disabled SELINUX
5. firewalld was already off
6. installed most, pdftk, htop, glances, qtgrace, lesstif
7. dataviewer now works and QTgrace is much nicer than XMGrace

In calculating the above numbers, I assumed a DC transimpedance of 10 khhms and an RF Transimpedance of ~800 V/A.

[Elog14480]: per these calculations, with the NewFocus 1611 PDs, we cannot achieve shot noise limited sensing for any power below the rated maximum for linear operation (i.e. 1mW). Moreover, the noise figure of the RF amplifier we use to amplify the sensed beat note before driving the delay-line frequency discriminator is unlikely to be the limiting noise source in the current configuration. Rana suggested that we get two Gain Blocks. These can handle input powers up to ~10dBm while still giving us plenty of power to drive the delay line. This way, we can (i) not compromise on the sacred optical gain, (ii) be well below the 1dB compression point (i.e. avoid nonlinear noise effects) and (iii) achieve a better frequency discriminant. 

Temporary fix: While the gain blocks arrive, I inserted a 10dB (3dB) attenuator between the PSL+EX (PSL+EY) photodiode RF output and the ZHL-3A amplifiers. This way, we are well below the 1dB compression point of said RF amplifiers, and also below the 1dB compression point of the on-board Teledyne AP1053 amplifiers on the demodulator boards we use.

Nest steps: Rana is getting in touch with Rich Abbott to find out if there is any data available on the noise performance of the post-mixer IF amplifier stage in the 0.1 -30 Hz range, where the voltage and current noise of the AD829 OpAmps could be limiting the DFD performance. But in the meantime, the ALS noise seems good again, and there is no evidence of the sort of CARM/DARM coupling that motivated this investigation in the first place. Managed to execute several IR-->ALS transitions tonight in the PRFPMI locking efforts (next elog).

No new Teledyne AP1053s were harmed in this process - I'll send the 5 units back to Rich tomorrow.