Some ideas that would help increase the locking duty-cycle in the short term. 

1. Seismometer investigation - something is not quite right with the vertex seismometer. This is the one that is primarily used for feedforward, and can be really helpful.
2. Drifting TTs - it is really annoying to have to re-set the input pointing into the interferometer every ~ hour. See Attachment #1.
3. FSS - this isn't a scientific statement, but there were ~20-30 minute periods last night where the PC drive RMS was displaying sharp spikes repeating every 2-3 seconds, first with increasing and then decreasing height. This is a new feature to me in the long standing PC drive saga but it doesn't tell me exactly what is going on as I don't know in what frequency band the glitch is actually happening. See Attachment #2.
4. ALS noise - while it is possible now to routinely transition the arm length control from the POX/POY to CARM/DARM basis, I see some sharp (<0.1 s) dives in the TRX/TRY levels when the arms are under ALS control. This wasn't present a week ago. Needs to be investigated - I defer this to the daytime tomorrow.

I tried implementing a basic PRMI ASC using the POP QPD as a sensor. The POP22 buildup RMS is reduced by a factor of a few. This is just a first attempt, I think the loop shape can be made much better, but the stability of the lock is already pretty impressive. For some past work, see here.

Koji suggested systematic investigation of the ETMX suspension electronics. The tests to be done are:

1. Characterization of PD whitening amplifiers - with the satellite box disconnected, we will look for glitches in the OSEM channels.
2. Characterization of LT1125s in the PD chain of the amplifiers - with the in-vacuum OSEMs disconnected, we will look for glitches due to the on-board transimpedance amplifiers of the satellite box.
3. Characterization using the satellite box tester - this will signal problems with the physical OSEMs.
4. Characterization of the suspension coil driver electronics - this will happen later.

So the ETMX satellite box is unplugged now, starting 530 pm PDT.

The satellite box was reconnected and the suspension was left with watchdog off but OSEM roughly centered. We will watch for glitches over the weekend.

I've checked the state of the laser interlock switch and everything looked normal.

[Shruti, Rana]

- At the X end, we set up the network analyzer to begin measurement of the AM transfer function by actuation of the laser PZT.

- The lid of the PDH optics setup was removed to make some checks and then replaced.

- From the PDH servo electronics setup the 'GREEN_REFL' and 'TO AUX-X LASER PZT' cables were removed for the measurement and then re-attached after.

- The signal today was too low to make a real measurement of the AM transfer function, but the GPIB scripts and interfacing was tested. 

We also took this opportunity to re-connect the interlock to the Innolight controller (after it was disconnected for diagnosing the mysterious NPRO self-shutdowns). The diode pump current was dialled down to 0, the interlock wires reconnected, and then the diode current was ramped back up to the nominal 2.1 A. The fan to cool the unit remains mounted in a flaky way as we couldn't locate the frame Chub had made for a more secure mounting solution. 

It seems like the pointing of the beam out of the laser head varies somewhat after the startup - I had to adjust the pointing into the PMC a couple of times by ~1 full turn of the Polaris mount screws, but the IMC has been locked (mostly) for the last ~16 hours.

Summary:

There are no unexpected red-flags in the performance of the DFD electronics. The calibration factors for the digital phase tracker system are 71.291 +/- 0.024 deg/MHz for the X delay line and 70.973 +/- 0.024 deg/MHz for the Y delay line, while the noise floor for the frequency noise discrimination is ~0.5 Hz/rtHz above 1 Hz (dominated by ADC noise).

Details:

1. Attachment #1 - This observation is what motivated my investigation.
   •  found that for certain beat frequencies between the PSL + EX lasers, the frequency noise reported by the DFD system was surprisingly low.
   • The measurement condition was: EX laser frequency locked to the arm cavity length by the uPDH servo at EX, arm cavity length locked to PSL frequency via POX locking.
2. To investigate further, I disconnected the output of the NF1611 PDs going to the ZHL-3A amplifiers on the PSL table (after first blocking the PSL light so that the PDs aren't generating any RF output).
   • An RF function generator (IFR2023B) was used to generate an RF signal to mimic the ALS beat signal.
   • I used a power splitter to divide the signal power equally between the two DFD paths.
   • The signal level on the Marconi was set to -5 dBm, to mimic the nominal power level seen by the DFD system.
   • I then performed two tests - (i) to calibrate the Phase Tracker output to deg / MHz and (ii) to measure the frequency noise reported by the DFD system for various signal frequencies.
   • Test (i): sweep the marconi frequency between 10 MHz - 200 MHz, measure the I and Q channels for each phase tracker servo, and figure out the complex argument of the signal using the arctangent. A linear polynomial was fit to the measured datapoints to extract the desired slope.
   • Test (ii): Sample frequencies uniformly distributed between 20 MHz - 80 MHz (nominal range of ALS beat frequencies expected). Reset the phase tracker servo gain, clear the output histories, wait for any transients to die out, and then collect the phase tracker servo output for 1 minute. Compute the FFT to figure out the frequency noise.
   • Attachment #2: Shows the phase tracker calibration, i.e. the results of Test (i). I took this opportunity to update the EPICS calibration fields that convert phase tracker servo output to Hz, the correction was ~7%. These numbers are consistent with what I measured previously - but the updated values weren't registered with SDF so everytime the LSC model was restarted, it reverted to the old values.
   • Attachment #3: Shows the spectra for the various measurements from Test (ii).
   • Attachment #4: Shows the gain of the phase tracker servo as a function of the RF signal frequency. This is a proxy for the signal strength, and the observed trend suggests that the signal power seen after digitization of the demodulated delay line output goes down by ~20% at 80 MHz relative to the level at 20 MHz. Seems reasonable to me, given frequency dependent losses of the intervening electronics / cabling.

Conclusion and next steps:

I still don't know what's responsible for the anomalously low noise levels reported by the ALS-X system sometimes. Next test is to check the EX PDH system, since on the evidence of these tests, the problem seems to be imprinted on the light (though I can't imagine how the noise becomes lower?).

Looking at the sensor and oplev trends over the weekend, there was only one event where the optic seems to have been macroscopically misaligned, at ~11:05:00 UTC on Oct 19 (early Saturday morning PDT). I attach a plot of the 2kHz time series data that has the mean value subtracted and a 0.6-1.2 Hz notch filter applied to remove the pendulum motion for better visualization. The y-axis calibration for the top plot assumes 1 ct ~= 1 um. This "glitch" seems to have a timescale of a few seconds, which is consistent with what we see on the CCD monitors when the cavity is locked - the alignment drifts away over a few seconds.

As usual, this tells us nothing conclusive. Anyways, I am re-enabling the watchdog and pushing on with locking activity and hope the suspension cooperates.

1. Transition of arms from POX/POY to CARM/DARM was much smoother today - a change was made at the EX PDH setup, see here.
2. Reliable settings for 3f locking with arms held off resonance seem to have been found.
3. Took sensing matrix in this condition, measured loop TFs.
4. Reduced CARM offset - reached arm powers ~50 at which point the PRMI lost lock. Reacquisition was quick though.
   • The POP22_I level seemed to decay as I reduced the CARM offset.
   • This would suggest that somehow the PRCL lock point is getting shifted as I reduce the CARM offset.
   • Tonight, I will investigate this more.

Summary:

The EX PDH setup had what I thought was insufficient phase and gain margins. So I lowered the gain a little - the price paid was that the suppression of laser frequency noise of the end laser was reduced. I actually think an intermediate gain setting (G=7) can give us ~35 degrees of phase margin, ~10dB gain margin, and lower residual unsuppressed AUX laser noise - to be confirmed by measurement later. See here for the last activity I did - how did the gain get increased? I can't find anything in the elog.

I made a change to the c1ass model to normalize the PIT and YAW POP QPD outputs by the SUM channel. A saturation block is used to prevent divide-by-zero errors, I set the saturation limits to [1,1e5], since the SUM channel is being recorded as counts right now. Model change is shown in the attached screenshots. I compiled and installed the model. Ran the reboot script to reboot all the vertex FEs to avoid the issue of crashing c1lsc.

During our EX AM/PM setups, I don't think we bumped the PDH gain knob (and I hope that the knob was locked). Possible drift in the PZT response? Good thing Shruti is on the case.

Is there a loop model of green PDH that agrees with the measurement? I'm wondering if something can be done with a compensation network to up the bandwidth or if the phase lag is more like a non-invertible kind.

The closest thing I can think of is here.

Attachment #1 - comparison of the POP QPD PIT and YAW output signal spectra with and without them being normalized by the SUM channel. I guess the shape is different between 30-100 Hz because we have subtracted out the correlated singal due to RIN?

This did not have the effect I desired - I was hoping that by normalizing the signals, I wouldn't need to change the gain of the ASC servo as the buildup in the PRC changed, but I found that the settings that worked well for PRMI locked with the carrier resonant (no arm cavities, see Attachment #2, buildup RIN reduced by a factor of ~4) did not work for the PRMI locked with the sideband resonant. Moreover, Koji raised the point that there will be some point in the transition from arms off resonance to on resonance where the dominant field in the PRC will change from being the circulating PRC carrier to the leaking arm carrier. So the response of the actuator (PRM) to correct for the misalignment may change sign. 

In conclusion, we decided that the best approach to improve the angular stability of the PRC will be to revive the PRC angualr feedforward, which in turn requires the characterization and repair of the apparently faulty vertex seismometer.

I looked into the seismometer situation a bit more today. Here is the story so far - I think more investigation is required:

1. There is an abrupt change in the PEM BLRMS channels around 6pm PDT every day. This has been consistently seen for the last two weeks.
2. The seismometer spectra look normal - see Attachment #1. The reference traces are from some months ago. There is elevated activity between 0.1-0.3 Hz, but this is seen in all the seismometers in all 3 DoFs.
3. I looked at the minute trend of the raw seismometer outputs (before being BLRMSed) for the last 200 days and don't see any abrupt change in characteristics (the data gap is due to the issue in this thread).
4. All the correct BLRMS filters seem to be engaged in the respective filter banks.

Attachment #2 has some spectrograms (they are rather large files). They suggest that the increase in noise in the 0.1-0.3 Hz band in the BS seismometer X channel is real - but there isn't a corresponding increase in the other two seismometers, so the problem could still be electronics related.

I wanted to restart the c1oaf model. As usual, the first time the model was restarted, it came back online with a 0x2bad error. This isn't even listed in the diagnostics manual as one of the recognized error states (unless there is a typo and they mean 0x2bad when they say 0xbad). The fix that has worked for me is to stop and start the model again, but of course, there is some chance of taking all the vertex FEs down in the process. No permutation of mxstream and daqd process restarts have cleared this error. We need some CDS/RCG support to look into this issue and fix it, it is not reasonable to go through reboots of all the vertex FEs every time we want to make a model change.

Summary:

I'd like to revive the PRC angular feedforward system. However, it looks like the coherence between the vertex seismometer channels and the PRC angular motion witness sensor (= POP QPD) is much lower than was found in the past, and hence, the stabilization potential by implementing feedforward seems limited, especially for the Pitch DoF.

Details:

I found that the old filters don't work at all - turning on the FF just increases the angular motion, I can see both the POP and REFL spots moving around a lot more on the CRT monitors.

I first thought I'd look at the frequency-domain weiner filter subtraction to get a lower bound on how much subtraction is possible. I collected ~25 minutes of data with the PRC locked with the carrier resonant (but no arm cavities). Attachment #1 shows the result of the frequency domain subtraction (the dashed lines in the top subplot are RMS). Signal processing details:

• Data was downloaded and downsampled to 64 Hz (from 2kHz for the POP QPD signals and from 128 Hz for the seismometer signals). The 'FIR' option of scipy decimate was used.
• FFT time used was 16 seconds for the multi-coherence calculations

The coherence between target signal (=POP QPD) and the witness channels (=seismometer channels) are much lower now than was found in the past. What could be going on here?

Summary:

The Trillium T240 seismometer needs mass re-centering. Has anyone done this before, and do we have any hardware to do this?

Details:

I went to the Trillium interface box in 1X5. In this elog, Koji says it is D1000749-v2. But looking at the connector footprint on the back panel, it is more consistent with the v1 layout. Anyway I didn't open it to check. Main point is that none of the backplane data I/O ports are used. We are digitizing (using the fast CDS system) the front panel BNC outputs for the three axes. So of the various connectors available on the interface box, we are only using the front panel DB25, the front panel BNCs, and the rear panel power.

The cable connecting this interface box to the actual seismometer is a custom one I believe. It has a 19 pin military circular type hermetic connector on one end, and a DB25 on the other. Power is supplied to the seismometer from the interface box via this cable, so in order to run the test, I had to use a DB25 breakout board to act as a feedthrough and peek at the signals while the seismometer and interface boards were connected. I used Jenne's mapping of the DB25--> 19 pin connector (which also seems consistent with the schematic). Findings:

1. We are supplying the Trillium with 39 V DC between the +PWR and -PWR pins, while the datasheet specifies 9V to 36V DC isolated. Probably this is okay?
2. The analog (AGND) and digital (DGND) ground pins are shorted. Is this okay?
3. I measured the DC voltages between the AGND pin and each of the mass position outputs.
   • These are supposed to indicate when the masses need re-centering.
   • The nominal output ranges for these are +/- 4 V single-ended.
   • I measured the following values (I don't know how the U,V,W basis is mapped onto the cartesian X,Y,Z coordinates):
     U_MP: 0.708 V
     V_MP: -2.151 V
     W_MP: -3.6 V
   • So at the very least, the mass needs centering in the W direction (the manual recommends doing the re-centering procedure when one of these indicators exceeds 3.5 V in absolute value).
4. I also checked the DC voltages of the (X,Y,Z) outputs of the seismometer on the front panel BNCs, and also on the DB25 connector (so directly from the seismometer). These are rated to have a range of 40 Vpp differential between the pins. I measured ~0V on all the three axes - this is a bit confusing as I assumed a de-centered mass would lead to saturation in one of these outouts, but maybe we are measuring velocities and not positions?
5. We probably should consider monitoring these signals long term to inform of such drifts, what is the spare channel situation in the c1sus acromag?
6. Interestingly, today evening, there is no excess noise in the 0.1-0.3 Hz band in the X-axis of the seismometer even though it is past 6pm PDT now, which is usually the time when the excess begins to show up. The z-axis 0.3-1Hz BLRMS channel has flatlined though...

I am holding off on attempting any re-centering, for more experienced people to comment.

[Koji, gautam]

Summary:

We think we got to the bottom of this issue today. The RF signal level going into the demod board is too high. This electronics chain needs some careful gain reallocation.

Details:

I was demonstrating to Koji a strange feature I had noticed in the ALS control, whereby when applying a CARM offset to detune the arms, the two arms seemed to respond differently (based on the transmission levels). This kind of CARM-->DARM coupling seemed strange to me. Anyway, I also noticed that the EPICS indicators on the ALS MEDM screen suggested ADC saturations were going on. I had never really looked at the fast time series of the inputs to the phase tracker servos, but these showed saturating behavior on ndscope traces. I went to the LSC rack and measured these on a scope, indeed, they were ~20V pp.

The output of the BeatMouth PDs are going to a ZHL-3A amplifier - we should consider replacing these with lower gain amplifiers, e.g. the Teledyne AP1053. This is relegated to a daytime task.

Other findings tonight:

While working on the PSL table, I somehow put the IMC FSS into a bad state, reminiscent of this behavior. Seems like this is linked to some flaky connection on the PSL table. One candidate is the unstable attachment of the Pomona box between the NPRO PZT and the FSS output - we should install a short BNC cable between these to avoid the lever arm situation we have right now.

back on new Rossa from Xi computing

1. switched to using Display Port for video; this works. The DVi, HDMI, VGA ports are connected to the motherboard rather than the video card, so they are not active.
2. runs super slow w/ SL 7.6; maybe some service is running after startup?
3. install repos and update according to LLO CDS wiki
4. add controls user and group according to LLO wiki
5. remove gstreamer ugly because it breaks yum update
6. run 'yum update --skip-broken' because GDS doesn't work
7. turn off old selinux stuff
8. modify fstab to get NFS

Next:

1. finish mounting
2. xfce
3. figure out why the LLO install instructions can't install any CDS software (e.g. root, DTT, etc)

Update: Sun Nov 3 18:08:48 2019

1. moved the SL7 fresh install repos back into etc/yum.repos.d/. The LLO instructions has me remove them, but the LLO supplied repos are no good for standard apps. After putting these back was able to install standard apps (terminator, root, diaggui)
2. copied over /etc/fstab lines from pianosa sothat the NFS mounts work correctly
3. added symlinks so that the NFS dirs mount in the right dirs
4. symlink libsasl2.so.3 -> libsasl2.so.2 and now DTT runs and can get data now and in the past
5. install XFCE
6. sitemap / MEDM works
7.  Did "sudo ln -s /usr/lib64/libXm.so.4 /usr/lib64/libXm.so.3" to enable StripTool.

Update: Fri Nov 15 00:00:26 2019:

1. random hanging of machine while doing various window moving or workspace switching
2. turned off power management in XFCE
3. turned off power management on monitor
4. disabled SELINUX
5. firewalld was already off
6. installed most, pdftk, htop, glances, qtgrace, lesstif
7. dataviewer now works and QTgrace is much nicer than XMGrace

In calculating the above numbers, I assumed a DC transimpedance of 10 khhms and an RF Transimpedance of ~800 V/A.

[Elog14480]: per these calculations, with the NewFocus 1611 PDs, we cannot achieve shot noise limited sensing for any power below the rated maximum for linear operation (i.e. 1mW). Moreover, the noise figure of the RF amplifier we use to amplify the sensed beat note before driving the delay-line frequency discriminator is unlikely to be the limiting noise source in the current configuration. Rana suggested that we get two Gain Blocks. These can handle input powers up to ~10dBm while still giving us plenty of power to drive the delay line. This way, we can (i) not compromise on the sacred optical gain, (ii) be well below the 1dB compression point (i.e. avoid nonlinear noise effects) and (iii) achieve a better frequency discriminant. 

Temporary fix: While the gain blocks arrive, I inserted a 10dB (3dB) attenuator between the PSL+EX (PSL+EY) photodiode RF output and the ZHL-3A amplifiers. This way, we are well below the 1dB compression point of said RF amplifiers, and also below the 1dB compression point of the on-board Teledyne AP1053 amplifiers on the demodulator boards we use.

Nest steps: Rana is getting in touch with Rich Abbott to find out if there is any data available on the noise performance of the post-mixer IF amplifier stage in the 0.1 -30 Hz range, where the voltage and current noise of the AD829 OpAmps could be limiting the DFD performance. But in the meantime, the ALS noise seems good again, and there is no evidence of the sort of CARM/DARM coupling that motivated this investigation in the first place. Managed to execute several IR-->ALS transitions tonight in the PRFPMI locking efforts (next elog).

No new Teledyne AP1053s were harmed in this process - I'll send the 5 units back to Rich tomorrow.

Summary:

1. The two arm lengths can be controlled reliably in the CARM/DARM basis using ALS error signals.
2. With a CARM offset to keep the arm cavitites off resonance, the PRMI can be locked using 3f error signals.
3. On attempting to reduce the CARM offset, I see a drop in the POP22 buildup in the PRC (correlated with the arm powers increasing). Not entirely clear why this is happening.

I ran some sensing measurements at various CARM offsets to check if the PRCL-->REFL33 and MICH-->REFL165 signals were being rotated out of the sensing quadrature as I lowered the CARM offset - there was no evidence of this happening. See Attachment #2. Other possibilities:

• CARM offset dependant offsets in the MICH/PRCL error points?
• Check the RAM due to the EOM? Perhaps the pointing / polarization control into the EOM got degraded.
• Angular stability of the PRC is still pretty poor, getting the angular feedforward back up and running would help the duty cycle enormously.

The IMC went into some crazy state so I'm calling it for the night, need to think about what could be happening and take a closer look at more signals during the CARM offset reduction period for some clues...

I looked at some signals for a 10 second period when the PRMI was locked with at some CARM offset, just before the PRMI lost lock, to see if there are any clues. I don't see any obvious signatures in this set of signals - if anything, the PRM is picking up some pitch offset, this is seen both at the Oplev error point and also in the POP QPD spot position. But why should this be happening as I reduce the CARM offset? The arm transmission is only ~5, so it's hard to imagine that the radiation pressure is somehow torquing the PRM. There are no angular feedback loops actuating on the PRM in this state except the local damping and Oplev loops.

The 1f signals are also changing their mean DC offset values, which may be a signature of a changing offset in the 3f MICH and PRCL error points? The MICH error signal is pretty noisy (maybe I can turn on some LPF to clean this up a bit), but I don't see any DC drift in the PRCL control signal.

I set up a photodiode (PDA10CF) in the IFO REFL beampath and the Agilent NA is sitting on the east side of the PSL enclosure. This was meant to be just a first look, maybe the PDA10CF isn't suitable for this measurement. The measurement condition was - PRM aligned so we have a REFL beam (DC level = 8.4V measured with High-Z). Both ITMs and ETMs were macroscopically misaligned so that there isn't any cavity effects to consider. I collected noise around 11 and 55 MHz, and also a dark measurement, plots to follow. The optics were re-aligned to the nominal config but I left the NA on the east side of the PSL enclosure for now, in anticipation of us maybe wanting to tune something while minimizing a peak.

Attachment #1: Results of a coarse sweep from 5 MHz to 100 MHz. The broadband RIN level is not resolvable above the dark noise of the photodiode, but the peaks at the modulation frequencies (11 MHz, 55 MHz and 29.5 MHz) are clearly visible. Not sure what is the peak at ~44 MHz or 66 MHz. Come to think of it, why is the 29.5 MHz peak so prominent? The IMC cavity pole is ~4kHz so shouldn't the 29.5 MHz be attenuated by 80dB in transmission through the cavity?

Attachment #2: Zoomed in spectra with finer IF bandwidth around the RF modualtion frequencies. From this first measurement, it seems like the RIN/rad level is ~10^5, which I vaguely remember from discussions being the level which is best achieved in practise in the 40m in the past.

Tried a bunch of things tonight.

1. Modified the "ELP300" filter module in the MICH filter bank - this was really a 4th order elliptic low pass with corner at 80 Hz, which was much too low. I tried upping the corner to 500 Hz, and reducing the order, while I was able to enable the filter, there was clearly a gain-peaking feature visible after engaging this module, so the exercise of reducing the high frequency MICH actuation requires more careful (daytime) loop optimization.
2. Tried adding some POPDC to the MICH/PRCL trigger once the PRMI was locked - I thought this would help if the problem was just with POP22 triggering turning off the MICH/PRCL loops, but the problem seems to persist with the mixed matrix trigger as well, once I reach a CARM offset where the arm powers exceed ~10, the PRMI loses lock.
3. One strange feature I don't understand is that with the PRMI locked with the carrier field resonant, when running the dither alignment servo to minimize REFLDC (= more carrier coupled into the PRC), the POPDC level also goes down, but TRX and TRY go up slightly. I confirmed that the beam isn't falling off the POP22 photodiode (Thorlabs PDA10CF), but I don't understand why these two DC powers should fall simultaneously - if I couple more carrier into the PRC, shouldn't the POPDC level also increase?

One possibility is that the arm buildup is exerting some torque on the ITMs, which can also change the PRC cavity axis - as the buildup increases, the dominant component of the circulating field in the PRC comes from the leakage from the overcoupled arm cavity. We used to DC couple the ITM Oplev servos when locking the PRMI. The TRX level of 1 corresponds to ~5W of circulating power in the arm cavity, and the static radiation pressure force due to this circulating power is ~30 nN, rising up to 300nN as the TRX level hits 10. So for 1mm offset of the spot position on the ITM, we'd still only exert 300 pN m of torque. I don't see any transient in the Oplev error signals when locking the arm cavity as usual with POX/POY, but on timescales of several seconds, the Oplev error point shows ~3-5 urad of variation.

I changed the shape of the low pass filter to reduce high frequency sensor noise injection into the MICH control signal. The loop stability isn't adversely affected (lost ~5 degrees of phase margin but still have ~50 degrees), while the control signal RMS is reduced by ~x10. This test was done with the PRMI locked on the carrier, need to confirm that this works with the arms controlled on ALS and PRMI lcoked on sideband.

The POP beam coming out of the vacuum chamber is split by a 50/50 BS and half is diverted to the POP22/POP110/POPDC photodiode (Thorlabs PDA10CF) and the other half goes to the POP QPD. This optical layout is still pretty accurate. I looked at the data of the POPDC and POP QPD SUM channels while the dither alignment was running, to see if I could figure out what's up with the weird correlated dip in REFLDC and POPDC. While the POPDC channel shows some degradation as the REFLDC level goes down (=alignment gets better), the QPD sum channel shows the expected light level increase. So it could yet be some weird clipping somewhere in the beampath - perhaps at the 50/50 BS? I will lock the PRMI (no arms) and check...

While I was trying to lock the PRMI this evening, I noticed that I couldn't move the REFL beamspot on the CCD field of view by adjusting the slow bias voltages to the PRM. Other suspensions controlled by c1susaux seem to respond okay so at first glance it isn't a problem with the Acromag. Looking at the OSEM sensor input levels, I noticed that UL is much lower than the others - see Attachment #1, seems to have happened ~100 days ago. I plugged the tester box in to check if the problem is with the electronics or if this is an actual shorting of some pins on the physical OSEM as we had in the past. So PRM watchdog is shutdown for now and there is no control of the optic available as the cables are detached. I will replace the connections later in the evening.

Update 10pm:

1. Measured coil inductances with breakout board and LCR meter - all 5 coils returned ~3.28-3.32 mH.
2. Measured coil resistances with breakout board and DMM - all 5 coils returned ~16-17 ohms.
3. Checked OSEM PD capacitance (with no bias voltage) using the LCR meter - each PD returned ~1nF.
4. Checked resistance between LED Cathode and Anode for all 5 LEDs using DMM - each returned Hi-Z.
5. Checked resistance between PD Cathode and Anode for all 5 PDs using DMM - each returned ~430 kohms.
6. Checked that I could change the slow bias voltages and see a response at the expected pins (with the suspension disconnected).

Since I couldn't find anything wrong, I plugged the suspension back in - and voila, the suspect UL PD voltage level came back to a level consistent with the others! See Attachment #2.

Anyway, I had some hours of data with the tester box plugged in - see Attachment #3 for a comparison of the shadow sensor readout with the tester box (all black traces) vs with the suspension plugged in, local damping loops active (coloured traces). The sensing noise re-injection will depend on the specifics of the  local damping loop shapes but I suspect it will limit feedforward subtraction possibilities at low frequencies.

However, I continue to have problems aligning the optic using the slow bias sliders (but the fast ones work just fine) - problem seems to be EPICS related. In Attachment #4, I show that even though I change the soft PITCH bias voltage adjust channel for the PRM, the linked channels which control the actual voltages to the coils take several seconds to show any response, and do so asynchronously. I tried restarting the modbus process on c1susaux, but the problem persists. Perhaps it needs a reboot of the computer and/or the acromag chassis? I note that the same problem exists for the BS and PRM suspensions, but not for ITMX or ITMY (didn't check the IMC optics). Perhaps a particular Acromag DAC unit is faulty / has issues with the internal subnet?

Sigh... hard loch

PMC got unlocked at ~4am. I re-locked it. Also tweaked the input pointing into the cavity. The misalignment was mostly in pitch.

There was also a loud buzzing in the control room due to the audio cable being improperly seated in the mixer. I re-seated it.

I reproduced Gautam's sketch of the 1x1 and 1x2 Eurocrates into a pdf image that contains links to the appropriate DCCs in the legend (see attachement).

Thanks. Please update this wiki page too.

https://wiki-40m.ligo.caltech.edu/Electronics/ElectronicsRacks#A1X1

I've installed pyepics on Donatella running

sudo yum install pyepics

Pip and ipython did not seem to be installed yet.

Done.

There are many versions of the POP22 signal path I found on the elog, e.g. this thread. But what I saw at the LSC rack was not quite in agreement with any of those. So here is the latest greatest version.

Since the 2f signals are mainly indicators of power buildups and are used for triggering various PDH loops, I don't know how critical some of these things are, but here are some remarks:

1. There is no Tee + 50 ohm terminator after the minicircuits filters, whose impedance in the stopband are High-Z (I have been told but never personally verified).
2. The RF amplifier used is a Minicircuits ZFL-1000-LN+. This has a gain of 20dB and 1dB compression output power spec of 3dBm. So to be safe, we want to have not more than -20dBm of signal at the input. On a 50-ohm scope (AC coupled), I saw a signal that has ~100mVpp amplitude (there is a mixture of many frequencies so this is not the Vpp of a pure sinusoid). This corresponds to -16dBm. Might be cutting it a bit close even after accounting for cable loss and insertion loss of the bias tee.
3. We use a resistive power splitter to divide the power between the POP22 and POP110 paths, which automatically throws away 50% of the RF power. A better option is the ZAPD-2-252-S+.
4. The Thorlabs PDA10CF photodiode (not this particular one) has been modelled to have a response that can be approximated by a complex pole pair with Q=1 at ~130 MHz. But we are also using this PD for measuring the 110 MHz PD which is a bit close to the band edge?

I did some re-alignment of the POP beam on the IX in air table. Here are the details:

1. Attachment #1 - optical layout.
2. With the PRC locked with the carrier resonant (no arm cavities), there is ~300uW of DC power incident on the Thorlabs PDA10CF, which serves as POP22, POP110 and POPDC photosensor.
   • See this elog for the signal paths.
   • On a scope, this corresponded to ~1.8 V DC of voltage. This is in good agreement with the expected transimpedance gain of 10 kOhms and responsivity of ~0.65 A/W given on the datasheet.
   • This is also in agreement with the ~6000 ADC counts I see in the CDS system (although there are large fluctuations). 
3. These was significant misalignment of the beam on this photodiode at some point:
   • Previously, I had used the CDS system to walk the beam on thde photodiode to try and maximize the power.
   • Today I took a different approach - triggered the MICH and PRCL loops on REFLDC (instead of the usual POPDC / POP22) so I could freely block the beam.
   • I found that there is a fast (f=35mm) lens to make the beam small enough for the PDA10CF. The beam was somewhat mis-centered on this strongly curved optic, and I suspect it was amplifying small misalignments. Anyway it is much better centered now (see Attachment #2) and I have a much stronger POPDC signal (by a factor of ~2-3, see Attachment #3).
   • The ASS dither alignment now shows much more consistent behavior - minimizing REFLDC maximises POPDC, see Attachment #4.
   • I took this opportunity to take some spectra/time-series of the PD output with the interferometer in this configuration. 

Tangentially related to this work - I took the nuclear option and did a hard reboot of the c1susaux Acromag crate on Sunday to fix the EPICS issue - it seems to be gone for now, see Attachment #5.

I created a spreadsheet (Attached) by taking Koji's c1psl sheet from slow_channel_list and filtering out the channels that do not need an Acromag. I added in the QPD channels that are relevant to the PSL from the c1iool0 sheet.

I began mapping the PSL related Eurocrates connectors to their respective VME channels starting with the PMC electronics.

I am confused about the TTFSS interface (D040423): While it is a Eurocrate card, in the schematics it seems to have 50 pin connectors.

I found old wiring schematics that might help with identifying the channels once the connector issue is clarified.

Summary:

I am still unable to achieve arm powers greater than TRX/TRY ~10 while keeping the PRMI locked. A couple of times, I was able to get TRY ~50, but TRX stayed at ~10, or even dropped a little, suggestive of a DARM offset? On the positive side, the ALS system seems to work pretty reliably, and I can keep the arms controlled by ALS for several tens of minutes.

Details:

• Despite my POP beam path improvements, I saw the POP22 level drop as I lowered the CARM offset.
• One strange feature last night was that with the arms held off resonance using ALS, I had to flip the sign and increase the gain by ~x2 of the REFL33_I-->PRCL loop in order to lock the PRMI. This was confirmed by locking on the 1f error signals and measuring the ratio of the response between the 1f and 3f signals while shaking PRCL using DTT swept sine.
• At different CARM offsets, I noted that the DC offset level on the 1f photodiodes (i.e. REFL11 and AS55) were changing significantly.
• I ran a measurement of the sensing matrix with the arm powers hovering around ~10, which is just before I lose the PRMI lock - managed to stay locked for >5 minutes, but the sensing matrix seems to suggest that the REFL33 demod angle needs to be rotated - maybe this is the reason why the PDH horn-to-horn voltage of REFL33 is lower now than it was last week? No idea why that should be, I was around the LSC rack but if the situation is so fragile, seems hopeless.
• MICH sensed by REFL165_Q still seems stable, so that's good...
• So my best hypothesis at the moment is that the PRCL optical gain is falling as I reduce the CARM offset (due to DC offset? or something else?). Needs some detailed modeling for more insight, I'm out of ideas for tests to run while locking as I've gone through the full gamut of OLTF and sensing matrix measurements at various CARM offsets without getting any clues as to what's going on.

I removed the Trillium T240 DAQ interface unit from 1X4 for investigation.

It was returned to the electronics rack and all the connections were re-made. Some details:

1. The board is indeed a D1000749-v2 as Koji said it is. There is just an additional board (labelled D1001872 but for which there is no schematic on the DCC) inside the 1U box that breaks out the D37 connector of the v2 into 3 D15 connectors. I took photos.
2. Armed with the new cable Chub got, and following the manual, I ran the re-centering routine.
   • Now all the mass-monitoring position voltages are <0.3 V DC, as the manual tells me they should be.
   • I noticed that when the seismometer is just plugged in and powered, it takes a few minutes for the mass monitoring voltages to acquire their steady state values.
   • The V indicator reported ~-2V DC, and the W indicator reported -3.9V DC.
   • While running the re-centering routine, I monitored the mass-position indicator voltages (via the backplane D15 connector) on an oscilloscope. See Attachment #1 for the time series. The data was rather noisy, I don't know why this is, so I plot the raw data in light colors and a filtered version in darker colors. Also, there seems to be a gain of x2 in the voltages on the backplane relative to what the T240 manual tells me I should expect, and the values reported when I query the unit via the serial port.
   • We should ideally just install another Acromag ADC in the c1susaux box and acquire these and other available diagnostic information, since the signals are available.
   • We should also probably check the mass position indicator values in a few days to see if they've drifted off again.
   • Looking at the raw time series / spectra of the BS channels, I see no obvious signatures of any change. 
   • I will run a test by locking the PRC and looking for coherence between the seismometer data and angular motion witnessed by the POP QPD, as this was what signalled my investigation in the first place.

Update 445pm: Seems to have done something good - the old feedforward filters reduce the YAW RMS motion by a factor of a few. Pitch performance is not so good, maybe the filter needs re-training, but I see coherence, see Attachment #2 for the frequency domain WF.

Summary:

There seems to be stronger-than-expected coupling between CARM and the 3f sensors. 

Details:

Full analysis tomorrow, but I collected sensing matrix measurements with lines driven in PRCL,MICH and CARM at a couple of CARM offsets. I also wanted to calibrate the CARM offset to physical units so I ran some scans of the CARM offset and collected the data so I can use the arm cavity FSR to calibrate CARM. Koji suggested using REFL165_I for PRCL and REFL165_Q for MICH control - this would allow us to see if the problem was with the 1f sideband only. While the lock could be established, we still couldn't push the arm powers above 10 without breaking the PRMI lock. While changing the CARM offset, we saw a significant shift in the DC offset level of the out-of-loop REFL33_I signal. Need to think about what this means...

Summary:

A coarse calibration of the CARM error point (when on ALS control) is 7.040 +/- 0.030 kHz/ct. This corresponds to approximately 0.95nm/ct. I typically lose the PRMI lock when the CARM offset is ~0.2 cts, which means I am about 1kHz away from the resonance. This is >10 CARM linewidths.

Details:

The calibration was done by sweeping the CARM offset (no PRM) and identifying the arm cavity FSRs by looking for peaks in TRX / TRY. Attachment #1 shows the scan, while Attachment #2 shows a linear fit to the FSRs. In Attachment #2, the frequency axis is taken from the phase tracker servo, which was calibrated by injecting a "known" frequency with the Marconi, and there is good agreement to the expected FSR with 37.79 m long arm cavities. There is much more info in the scan (e.g. modulation depths, mode matching to the arm cavities etc) which I will extract later, but if anyone wants the data (pre-downsampled by me to have a managable filesize), it's attached as a .zip file in Attachment #3.

Here is a comparison of the response of various DoFs in our various RFPD sensors for two different CARM offsets. Even in the case of the smaller CARM offset of ~1kHz, we are several linewidths away from the resonance. Need to do some finesse modeling to make any meaningful statement about this - why is the CARM response in REFL11 apparently smaller for the smaller CARM offset?

If you mistrust my signal processing, the GPS times for which I ran the sensing lines are:

CARM offset = ~30kHz (arm transmission <0.02) --- 1257064777+5min

CARM offset = ~1kHz (arm transmission ~5) --- 1257065566+5min

Koji and I taked about cleaning up some of the flaky cable situation on the PSL table a while ago. The changes were implemented and are documented in Attachment #1. Now the Pomona box between the Thorlabs HV Driver and the NPRO head is sitting on the PSL table (sandwiched between some teflon pieces I found in cabinet S4 along the south arm), and the cables between these two devices are better strain relieved. I turned off the Thorlabs HV supply while working on the PMC table. The IMC could be locked after this work. Probably won't solve the long standing FSS mysteries but probably can't hurt.

Unrelated to this work: I also removed a Bias tee that was just hanging out on top of the FSS electronics, which was used for the modeSpec project.

Gautam and I were talking about some modulation and demodulation and wondered what is the power combining situation for the triple resonant EOM installed 8 years ago. And we noticed that the current setup has additional ~5dB loss associated with the 3-to-1 power combiner. (Figure a)

N-to-1 broadband power combiners have an intrinsic loss of 10 log10(N). You can think about a reciprocal process (power splitting) (Figure b). The 2W input coming to the 2-port power splitter gives us two 1W outputs. The opposite process is power combining as shown in Figure c. This case, the two identical signals are the constructively added in the combiner, but the output is not 20Vpk but 14Vpk. Considering thge linearity, when one of the port is terminated, the output is going to be a half. So we expect 27dBm output for a 30dBm input (Figure d). This fact is frequently oversight particularly when one combines the signals at multiple frequencies (Figrue e). We can avoid this kind of loss by using a frequency-dependent power combiner like a diplexer or a triplexer.

Some details:

- There was a SR560+SR785 (not connected for measurement) placed near the X end which I moved; it is now behind the electronics rack by the X arm beam tube (~15m away).

- Also, for the AM measurement I moved the AG5395A from behind the PSL setup to the X end, where it now is.

- By toggling the XGREEN shutter, I noticed that the cavity was not resonant before I disconnected anything from the setup since the spot shape kept changing, but I proceeded anyway. 

- Because Rana said that it was important for me to mention: the ~5 USD blue-yellow crocs (that I now use) work fine for me.

The AM Measurement:

1. The cables were calibrated with the DC block in the A port (for a A/R measurement)

2. The cable to the PZT was disconnected from the pomona box and connected to the RF out of the NA, the PD output labelled 'GREEN_REFL' was also disconnected and connected to the B port via a DC block. 

3. The ITMX was 'misaligned'. (This allowed the reflected green PD output as seen on the oscilloscope to stabilize.)

4. The PZT is modulated in frequency and the residual amplitude modulation (as observed in the measured reflected green light) is plotted, ref. Attachment 1. The parameters for the plotted data in the attachment were:

# AG4395A Measurement - Timestamp: Nov 07 2019 - 17:04:07
#---------- Measurement Parameters ------------
# Start Frequency (Hz): 10000.0, 10000.0
# Stop Frequency (Hz): 10000000.0, 10000000.0
# Frequency Points: 801, 801
# Measurement Format: LOGM, PHAS
# Measuremed Input: AR, AR
#---------- Analyzer Settings ----------
# Number of Averages: 8
# Auto Bandwidth: On, On
# IF Bandwidth: 300.0, 300.0
# Input Attenuators (R,A,B): 0dB 10dB 20dB 
# Excitation amplitude = -10.0dBm

------------------------------------

Update (19:13 7thNov19):  When the ITMX was intentionally misaligned, Rana and I checked to see if the Oplevs were turned off and they were. But while I was casually checking the Oplevs again, they were on! 

Not sure what to do about this or what caused it. 

Today I realized that pip and other python2,3 packages were installed in the conda base environment, so after running

I could run the python-GPIB scripts to interface with the Agilent.

Although, I did have to add a python2 kernel to jupyter/ipython, which I did in a separate conda environment:

Attachment #1 shows the spectra of our three available seismometers over a period of ~10ksec.

• I don't understand why the z-axis motion reported by the T240 is ~10x lower at 10 mHz compared to the X and Y motions. Is this some electronics noise artefact?
• The difference in the low frequency (<100mHz) shapes of the T240 compared to the Guralps is presumably due to the difference in the internal preamps / readout boxes (?). I haven't checked yet.
• There is almost certainly some issue with the EX Guralp. IIRC this is the one that had cabling issues in the past, and also is the one that was being futzed around for Tctrl, but also could be that its masses need re-centering, since it is EX_X that is showing the anomalous behaviour.
• The coherence structure between the other pairs of sensors is consistent.

Attachment #2 shows the result of applying frequency domain Wiener filter subtraction to the POP QPD (target) with the vertex seismometer signals as witness channels.

• The dataset was PRMI locked with the carrier resonant, ETMs misaligned.
• The dashed lines in these plots correspond to the RMS for the solid line with the same color.
• For both PIT and YAW, I am using BS_X and BS_Y seismometer channels for the MISO filter inputs.
• In particular for PIT, I notice that I am unable to get the same level of performance as in the past, particularly around ~2-3 Hz.
• The BS seismometer health indicators don't signal any obvious problems with the seismometer itself - so something has changed w.r.t. how the ground motion propagates to the PR2/PR3? Or has the seismometer sensing truly degraded? I don't think the dataset I collected was particularly bad compared to the past, and I confirmed similar performance with a separate PRMI lock from a different time period.

this is due to the Equivalence Principle: local accelerations are indistinguishable from spacetime curvature. On a spherical Earth, the local gradient of the metric points in the direction towards the center of the Earth, which is colloquially known as "down".

Here is some disturbance in the spacetime curvature, where the local gradient of the metric seems to have been modulated (in the "downward" as well as in the other two orthogonal Cartesian directions) at ~1 Hz - seems real as far as I can tell, all the suspensions were being shaken about and all the seismometers witnessed it, though the peak is pretty narrow. A broader, less prominent peak also shows up around 0.5 Hz. We couldn't identify any clear source (no LN2 fill-up / obvious CES activity). This event lasted for ~45 mins, and stopped around 2315 local time. Shortly (~5min) after the ~1 Hz peak died down, however, the 3-10 Hz BLRMS channel reports an increase by ~factor of 2. 

Onto trying some locking now that the suspensions have settled down somewhat.