This problem has re-surfaced. Is this indicative of some problem with the on-board VGA? Even with 0dB of whitening gain, I see PDH horns that are 10,000 ADC counts in amplitude, whereas the nominal whitening gain for this channel is +18dB. I'll look at it in the daytime, not planning to use REFL55 for any locking tonight.
Hardware issues that need addressing:
[Jon, Yehonathan, Gautam, Aaron, Shruti, Koji]
We get together on Wednesday afternoon for cleaning the lab. Particularly, we collected e-wastes: VME crates, VME modules, old slow control cables, and other old/broken electronics. They are piled up in the office area and the cage outside rioght now (Attachments 1/2). We asked Liz to come to pick them up (under the coordination with either Gautam or Koji). Eventually this will free up two office desks.
Also, we made the acromag components organized in plastic boxes. (Attachment 3)
CM Board Slow out (digital length control) path transfer function / pole-zero filter pair (79Hz/1.6kHz) transfer function
The excitation was given from EXC A. The denominator was TESTA2, and the numerator was OUT1.
Attachment 1 shows the measured transfer function with and without PZ filter off and on. The PZ filter provides ~26dB attenuation at high frequency. The output stage has a single order 100kHz LPF and it is visible in the transfer function.
The transfer function without the PZ filter was modelled by LISO as the following PZK representation. There looked a small step in the TF which caused the additional PZ pair (66~67Hz) but has very minor effect in the mag and phase.
The transfer function of the PZ filter was separately analyzed. The TF with the switch ON was normalized by the one with the switch OFF. Thus it revealed the pure effect of the switch. The PZK model of the stage was estimated to be
For the CM board modeling purpose, the transfer function from TESTA2 to TESTB2 was needed. (Attachment 1)
The ZPK model of this part is
The output stage (and AO GAIN stage) of the MC board was modelled. The transfer function was measured with the injection from EXC B. The denominator was TESTB2, and the numerator was SERVO OUT.
This stage is AC coupled by 2x 1st order HPFs. Firstly, this transfer function was measured with AO GAIN set to be 0dB. (Attachment 1)
This TF was used to characterize the cutoffs of the HPF stages, represented as the following ZPK:
Then the AO GAIN was already measured as seen in [ELOG 14948]. The AO gain TF was then modeled by LISO with the above HPF as the preset. This allows us to characterize the time delay of the AO GAIN part.
Input referred offsets on the IN1/IN2 were tested with different gain settings. The two inputs were plugged by the 50 ohm terminators. The output was monitored at OUT1 (SLOW Length Output). The fast path is AC coupled and has no sensitivity to the offset.
There is the EPICS monitor point for OUT1. With the multimeter it was confirmed that the EPICS monitor (C1:LSC-CM_REFL1_GAIN) has the right value except for the opposite sign because the output stage of OUT1 is inverting. The previous stages have no sign inversion. Therefore, the numbers below does not compensate the sign inversion.
Attachment 1 shows the output offset observed at C1:LSC-CM_REFL1_GAIN. There is some gain variation, but it is around the constant offset of ~26mV. This suggested that the most of the offset is not from the gain stages but from the later stages (like the boost stages). Note that the boost stages were turned off during the measurements.
Attachment 2 shows the input refered offset naively calculated from the above output offset. In dependent from which path was used, the offset with low gain was hugely enhanced.
Since the input referred offset without subtracting the static offset seemed useless, a constant offset of -26mV was subtracted from the calculation (Attachment 2). This shows that the input refered offset can go up to ~+/-20mV when the gain is up to -16dB. Above that, the offset is mV level.
I don't think this level of offset by whichever OP27 or AD829 becomes an issue when the input error signal is the order of a volt.
This suggests that it is more important to properly set the internal offset cancellation as well as to keep the gain setting to be high.
The UPS is now incessantly beeping. I cannot handle this constant sound so I shut down all the control room workstations and moved the power strip hosting the 4 CPUs to a wall socket for tonight. Chub and I will replace the UPS batteries tomorrow.
Updated Circuit Diagram and photos: https://dcc.ligo.org/D1500308-v2
- (1) and (6) of the diagram: TFs with various gain slider values for REFL1/REFL2/AO GAIN [ELOG 14948] (gain values and time delay modeling)
- Switching checks, latest photo of the board, Limiter check [ELOG 14953]
- (2): Boost transfer functions [ELOG 14955]
- (3): Slow (aka Length) CM output path [ELOG 14965]
- (4): Pole-Zero filter TF [ELOG 14965]
- (5): TF from TESTA2 to TESTB2 [ELOG 14966]
- (6): AC coupling TF of the AO GAIN stage [ELOG 14967]
- (7): AC coupling TF of the IN2 stage on IMC servo board [ELOG 15044]
Slow path = (1)*(2 if necessary)*(3)*(4 if necessary)
Fast path = (1)*(2 if necessary)*(4 if necessary)*(5)*(6)
gautam 20191122: Adding the measured AC coupling of the IN2 input of the IMC servo board for completeness.
[Liz, Gautam, Chub, Jordan, Koji]
We removed a significant amount of e-waste from the lab. The garbage was moved to the e-waste station in WB SB and are waiting for disposal.
Batteries + power cables replaced, and computers back on UPS from today ~3pm.
There is poor separation of the PRCL and MICH length error signals as sensed in the 3f photodiodes. I don't know why this is so - one possibility is that the MICH-->PRM matrix element in the LSC output matrix needs to be tuned to minimize the MICH -->PRCL coupling.
Over the last few days, I've been trying to make the 3f locking of the PRMI more reliable. Turns out that while I was able to lock the PRMI on 3f error signals, it was just a fluke. So I set about trying to be more systematic. Here are the steps I followed:
Attachment #1 is the result. I don't know what is the reason for such poor separation of the MICH and PRCL error signals in REFL165. The situation seems very different from when I had the DRMI locked in Nov last year.
After this exercise, I tried for some hours to get the 3f PRMI locking going with the arm cavities held off resonance under ALS control, but had no success. The angular motion of the PRC isn't helping, but I feel this shouldn't be a show stopper.
This is as far as I got last night. The first step is to see how reliable the settings determined last night are, today. I don't understand how changing the output matrix element can have brought about such a significant change in the MICH/PRCL separation in all the RF photodiodes.
Some ideas that would help increase the locking duty-cycle in the short term.
I tried implementing a basic PRMI ASC using the POP QPD as a sensor. The POP22 buildup RMS is reduced by a factor of a few. This is just a first attempt, I think the loop shape can be made much better, but the stability of the lock is already pretty impressive. For some past work, see here.
Koji suggested systematic investigation of the ETMX suspension electronics. The tests to be done are:
So the ETMX satellite box is unplugged now, starting 530 pm PDT.
The satellite box was reconnected and the suspension was left with watchdog off but OSEM roughly centered. We will watch for glitches over the weekend.
I've checked the state of the laser interlock switch and everything looked normal.
- At the X end, we set up the network analyzer to begin measurement of the AM transfer function by actuation of the laser PZT.
- The lid of the PDH optics setup was removed to make some checks and then replaced.
- From the PDH servo electronics setup the 'GREEN_REFL' and 'TO AUX-X LASER PZT' cables were removed for the measurement and then re-attached after.
- The signal today was too low to make a real measurement of the AM transfer function, but the GPIB scripts and interfacing was tested.
We also took this opportunity to re-connect the interlock to the Innolight controller (after it was disconnected for diagnosing the mysterious NPRO self-shutdowns). The diode pump current was dialled down to 0, the interlock wires reconnected, and then the diode current was ramped back up to the nominal 2.1 A. The fan to cool the unit remains mounted in a flaky way as we couldn't locate the frame Chub had made for a more secure mounting solution.
It seems like the pointing of the beam out of the laser head varies somewhat after the startup - I had to adjust the pointing into the PMC a couple of times by ~1 full turn of the Polaris mount screws, but the IMC has been locked (mostly) for the last ~16 hours.
There are no unexpected red-flags in the performance of the DFD electronics. The calibration factors for the digital phase tracker system are 71.291 +/- 0.024 deg/MHz for the X delay line and 70.973 +/- 0.024 deg/MHz for the Y delay line, while the noise floor for the frequency noise discrimination is ~0.5 Hz/rtHz above 1 Hz (dominated by ADC noise).
Conclusion and next steps:
I still don't know what's responsible for the anomalously low noise levels reported by the ALS-X system sometimes. Next test is to check the EX PDH system, since on the evidence of these tests, the problem seems to be imprinted on the light (though I can't imagine how the noise becomes lower?).
Looking at the sensor and oplev trends over the weekend, there was only one event where the optic seems to have been macroscopically misaligned, at ~11:05:00 UTC on Oct 19 (early Saturday morning PDT). I attach a plot of the 2kHz time series data that has the mean value subtracted and a 0.6-1.2 Hz notch filter applied to remove the pendulum motion for better visualization. The y-axis calibration for the top plot assumes 1 ct ~= 1 um. This "glitch" seems to have a timescale of a few seconds, which is consistent with what we see on the CCD monitors when the cavity is locked - the alignment drifts away over a few seconds.
As usual, this tells us nothing conclusive. Anyways, I am re-enabling the watchdog and pushing on with locking activity and hope the suspension cooperates.
The EX PDH setup had what I thought was insufficient phase and gain margins. So I lowered the gain a little - the price paid was that the suppression of laser frequency noise of the end laser was reduced. I actually think an intermediate gain setting (G=7) can give us ~35 degrees of phase margin, ~10dB gain margin, and lower residual unsuppressed AUX laser noise - to be confirmed by measurement later. See here for the last activity I did - how did the gain get increased? I can't find anything in the elog.
I made a change to the c1ass model to normalize the PIT and YAW POP QPD outputs by the SUM channel. A saturation block is used to prevent divide-by-zero errors, I set the saturation limits to [1,1e5], since the SUM channel is being recorded as counts right now. Model change is shown in the attached screenshots. I compiled and installed the model. Ran the reboot script to reboot all the vertex FEs to avoid the issue of crashing c1lsc.
During our EX AM/PM setups, I don't think we bumped the PDH gain knob (and I hope that the knob was locked). Possible drift in the PZT response? Good thing Shruti is on the case.
Is there a loop model of green PDH that agrees with the measurement? I'm wondering if something can be done with a compensation network to up the bandwidth or if the phase lag is more like a non-invertible kind.
The closest thing I can think of is here.
Attachment #1 - comparison of the POP QPD PIT and YAW output signal spectra with and without them being normalized by the SUM channel. I guess the shape is different between 30-100 Hz because we have subtracted out the correlated singal due to RIN?
This did not have the effect I desired - I was hoping that by normalizing the signals, I wouldn't need to change the gain of the ASC servo as the buildup in the PRC changed, but I found that the settings that worked well for PRMI locked with the carrier resonant (no arm cavities, see Attachment #2, buildup RIN reduced by a factor of ~4) did not work for the PRMI locked with the sideband resonant. Moreover, Koji raised the point that there will be some point in the transition from arms off resonance to on resonance where the dominant field in the PRC will change from being the circulating PRC carrier to the leaking arm carrier. So the response of the actuator (PRM) to correct for the misalignment may change sign.
In conclusion, we decided that the best approach to improve the angular stability of the PRC will be to revive the PRC angualr feedforward, which in turn requires the characterization and repair of the apparently faulty vertex seismometer.
I looked into the seismometer situation a bit more today. Here is the story so far - I think more investigation is required:
Attachment #2 has some spectrograms (they are rather large files). They suggest that the increase in noise in the 0.1-0.3 Hz band in the BS seismometer X channel is real - but there isn't a corresponding increase in the other two seismometers, so the problem could still be electronics related.
Yesterday, Koji and I noticed (from the wall StripTool traces) that the vertex seismometer RMS between 0.1-0.3 Hz in the X-direction increased abruptly around 6pm PDT. This morning, when I came in, I noticed that the level had settled back to the normal level. Trending the BLRMS channels over the last 24 hours, I see that the 0.3-1 Hz band in the Z direction shows some anomalous behaviour almost in the exact same time-band. Hard to believe that any physical noise was so well aligned to the seismometer axes, I'm inclined to think this is indicative of some electronics issues with the Trillium interface unit, which has been known to be flaky in the past.
I wanted to restart the c1oaf model. As usual, the first time the model was restarted, it came back online with a 0x2bad error. This isn't even listed in the diagnostics manual as one of the recognized error states (unless there is a typo and they mean 0x2bad when they say 0xbad). The fix that has worked for me is to stop and start the model again, but of course, there is some chance of taking all the vertex FEs down in the process. No permutation of mxstream and daqd process restarts have cleared this error. We need some CDS/RCG support to look into this issue and fix it, it is not reasonable to go through reboots of all the vertex FEs every time we want to make a model change.
I'd like to revive the PRC angular feedforward system. However, it looks like the coherence between the vertex seismometer channels and the PRC angular motion witness sensor (= POP QPD) is much lower than was found in the past, and hence, the stabilization potential by implementing feedforward seems limited, especially for the Pitch DoF.
I found that the old filters don't work at all - turning on the FF just increases the angular motion, I can see both the POP and REFL spots moving around a lot more on the CRT monitors.
I first thought I'd look at the frequency-domain weiner filter subtraction to get a lower bound on how much subtraction is possible. I collected ~25 minutes of data with the PRC locked with the carrier resonant (but no arm cavities). Attachment #1 shows the result of the frequency domain subtraction (the dashed lines in the top subplot are RMS). Signal processing details:
The coherence between target signal (=POP QPD) and the witness channels (=seismometer channels) are much lower now than was found in the past. What could be going on here?
The Trillium T240 seismometer needs mass re-centering. Has anyone done this before, and do we have any hardware to do this?
I went to the Trillium interface box in 1X5. In this elog, Koji says it is D1000749-v2. But looking at the connector footprint on the back panel, it is more consistent with the v1 layout. Anyway I didn't open it to check. Main point is that none of the backplane data I/O ports are used. We are digitizing (using the fast CDS system) the front panel BNC outputs for the three axes. So of the various connectors available on the interface box, we are only using the front panel DB25, the front panel BNCs, and the rear panel power.
The cable connecting this interface box to the actual seismometer is a custom one I believe. It has a 19 pin military circular type hermetic connector on one end, and a DB25 on the other. Power is supplied to the seismometer from the interface box via this cable, so in order to run the test, I had to use a DB25 breakout board to act as a feedthrough and peek at the signals while the seismometer and interface boards were connected. I used Jenne's mapping of the DB25--> 19 pin connector (which also seems consistent with the schematic). Findings:
I am holding off on attempting any re-centering, for more experienced people to comment.
We think we got to the bottom of this issue today. The RF signal level going into the demod board is too high. This electronics chain needs some careful gain reallocation.
I was demonstrating to Koji a strange feature I had noticed in the ALS control, whereby when applying a CARM offset to detune the arms, the two arms seemed to respond differently (based on the transmission levels). This kind of CARM-->DARM coupling seemed strange to me. Anyway, I also noticed that the EPICS indicators on the ALS MEDM screen suggested ADC saturations were going on. I had never really looked at the fast time series of the inputs to the phase tracker servos, but these showed saturating behavior on ndscope traces. I went to the LSC rack and measured these on a scope, indeed, they were ~20V pp.
The output of the BeatMouth PDs are going to a ZHL-3A amplifier - we should consider replacing these with lower gain amplifiers, e.g. the Teledyne AP1053. This is relegated to a daytime task.
Other findings tonight:
While working on the PSL table, I somehow put the IMC FSS into a bad state, reminiscent of this behavior. Seems like this is linked to some flaky connection on the PSL table. One candidate is the unstable attachment of the Pomona box between the NPRO PZT and the FSS output - we should install a short BNC cable between these to avoid the lever arm situation we have right now.
back on new Rossa from Xi computing
Update: Sun Nov 3 18:08:48 2019
Update: Fri Nov 15 00:00:26 2019:
In calculating the above numbers, I assumed a DC transimpedance of 10 khhms and an RF Transimpedance of ~800 V/A.
[Elog14480]: per these calculations, with the NewFocus 1611 PDs, we cannot achieve shot noise limited sensing for any power below the rated maximum for linear operation (i.e. 1mW). Moreover, the noise figure of the RF amplifier we use to amplify the sensed beat note before driving the delay-line frequency discriminator is unlikely to be the limiting noise source in the current configuration. Rana suggested that we get two Gain Blocks. These can handle input powers up to ~10dBm while still giving us plenty of power to drive the delay line. This way, we can (i) not compromise on the sacred optical gain, (ii) be well below the 1dB compression point (i.e. avoid nonlinear noise effects) and (iii) achieve a better frequency discriminant.
Temporary fix: While the gain blocks arrive, I inserted a 10dB (3dB) attenuator between the PSL+EX (PSL+EY) photodiode RF output and the ZHL-3A amplifiers. This way, we are well below the 1dB compression point of said RF amplifiers, and also below the 1dB compression point of the on-board Teledyne AP1053 amplifiers on the demodulator boards we use.
Nest steps: Rana is getting in touch with Rich Abbott to find out if there is any data available on the noise performance of the post-mixer IF amplifier stage in the 0.1 -30 Hz range, where the voltage and current noise of the AD829 OpAmps could be limiting the DFD performance. But in the meantime, the ALS noise seems good again, and there is no evidence of the sort of CARM/DARM coupling that motivated this investigation in the first place. Managed to execute several IR-->ALS transitions tonight in the PRFPMI locking efforts (next elog).
No new Teledyne AP1053s were harmed in this process - I'll send the 5 units back to Rich tomorrow.
I ran some sensing measurements at various CARM offsets to check if the PRCL-->REFL33 and MICH-->REFL165 signals were being rotated out of the sensing quadrature as I lowered the CARM offset - there was no evidence of this happening. See Attachment #2. Other possibilities:
The IMC went into some crazy state so I'm calling it for the night, need to think about what could be happening and take a closer look at more signals during the CARM offset reduction period for some clues...
I looked at some signals for a 10 second period when the PRMI was locked with at some CARM offset, just before the PRMI lost lock, to see if there are any clues. I don't see any obvious signatures in this set of signals - if anything, the PRM is picking up some pitch offset, this is seen both at the Oplev error point and also in the POP QPD spot position. But why should this be happening as I reduce the CARM offset? The arm transmission is only ~5, so it's hard to imagine that the radiation pressure is somehow torquing the PRM. There are no angular feedback loops actuating on the PRM in this state except the local damping and Oplev loops.
The 1f signals are also changing their mean DC offset values, which may be a signature of a changing offset in the 3f MICH and PRCL error points? The MICH error signal is pretty noisy (maybe I can turn on some LPF to clean this up a bit), but I don't see any DC drift in the PRCL control signal.
I set up a photodiode (PDA10CF) in the IFO REFL beampath and the Agilent NA is sitting on the east side of the PSL enclosure. This was meant to be just a first look, maybe the PDA10CF isn't suitable for this measurement. The measurement condition was - PRM aligned so we have a REFL beam (DC level = 8.4V measured with High-Z). Both ITMs and ETMs were macroscopically misaligned so that there isn't any cavity effects to consider. I collected noise around 11 and 55 MHz, and also a dark measurement, plots to follow. The optics were re-aligned to the nominal config but I left the NA on the east side of the PSL enclosure for now, in anticipation of us maybe wanting to tune something while minimizing a peak.
Attachment #1: Results of a coarse sweep from 5 MHz to 100 MHz. The broadband RIN level is not resolvable above the dark noise of the photodiode, but the peaks at the modulation frequencies (11 MHz, 55 MHz and 29.5 MHz) are clearly visible. Not sure what is the peak at ~44 MHz or 66 MHz. Come to think of it, why is the 29.5 MHz peak so prominent? The IMC cavity pole is ~4kHz so shouldn't the 29.5 MHz be attenuated by 80dB in transmission through the cavity?
Attachment #2: Zoomed in spectra with finer IF bandwidth around the RF modualtion frequencies. From this first measurement, it seems like the RIN/rad level is ~10^5, which I vaguely remember from discussions being the level which is best achieved in practise in the 40m in the past.
Check the RAM due to the EOM? Perhaps the pointing / polarization control into the EOM got degraded.
Tried a bunch of things tonight.
One possibility is that the arm buildup is exerting some torque on the ITMs, which can also change the PRC cavity axis - as the buildup increases, the dominant component of the circulating field in the PRC comes from the leakage from the overcoupled arm cavity. We used to DC couple the ITM Oplev servos when locking the PRMI. The TRX level of 1 corresponds to ~5W of circulating power in the arm cavity, and the static radiation pressure force due to this circulating power is ~30 nN, rising up to 300nN as the TRX level hits 10. So for 1mm offset of the spot position on the ITM, we'd still only exert 300 pN m of torque. I don't see any transient in the Oplev error signals when locking the arm cavity as usual with POX/POY, but on timescales of several seconds, the Oplev error point shows ~3-5 urad of variation.
I changed the shape of the low pass filter to reduce high frequency sensor noise injection into the MICH control signal. The loop stability isn't adversely affected (lost ~5 degrees of phase margin but still have ~50 degrees), while the control signal RMS is reduced by ~x10. This test was done with the PRMI locked on the carrier, need to confirm that this works with the arms controlled on ALS and PRMI lcoked on sideband.
The POP beam coming out of the vacuum chamber is split by a 50/50 BS and half is diverted to the POP22/POP110/POPDC photodiode (Thorlabs PDA10CF) and the other half goes to the POP QPD. This optical layout is still pretty accurate. I looked at the data of the POPDC and POP QPD SUM channels while the dither alignment was running, to see if I could figure out what's up with the weird correlated dip in REFLDC and POPDC. While the POPDC channel shows some degradation as the REFLDC level goes down (=alignment gets better), the QPD sum channel shows the expected light level increase. So it could yet be some weird clipping somewhere in the beampath - perhaps at the 50/50 BS? I will lock the PRMI (no arms) and check...
While I was trying to lock the PRMI this evening, I noticed that I couldn't move the REFL beamspot on the CCD field of view by adjusting the slow bias voltages to the PRM. Other suspensions controlled by c1susaux seem to respond okay so at first glance it isn't a problem with the Acromag. Looking at the OSEM sensor input levels, I noticed that UL is much lower than the others - see Attachment #1, seems to have happened ~100 days ago. I plugged the tester box in to check if the problem is with the electronics or if this is an actual shorting of some pins on the physical OSEM as we had in the past. So PRM watchdog is shutdown for now and there is no control of the optic available as the cables are detached. I will replace the connections later in the evening.
Since I couldn't find anything wrong, I plugged the suspension back in - and voila, the suspect UL PD voltage level came back to a level consistent with the others! See Attachment #2.
Anyway, I had some hours of data with the tester box plugged in - see Attachment #3 for a comparison of the shadow sensor readout with the tester box (all black traces) vs with the suspension plugged in, local damping loops active (coloured traces). The sensing noise re-injection will depend on the specifics of the local damping loop shapes but I suspect it will limit feedforward subtraction possibilities at low frequencies.
However, I continue to have problems aligning the optic using the slow bias sliders (but the fast ones work just fine) - problem seems to be EPICS related. In Attachment #4, I show that even though I change the soft PITCH bias voltage adjust channel for the PRM, the linked channels which control the actual voltages to the coils take several seconds to show any response, and do so asynchronously. I tried restarting the modbus process on c1susaux, but the problem persists. Perhaps it needs a reboot of the computer and/or the acromag chassis? I note that the same problem exists for the BS and PRM suspensions, but not for ITMX or ITMY (didn't check the IMC optics). Perhaps a particular Acromag DAC unit is faulty / has issues with the internal subnet?
Sigh... hard loch
PMC got unlocked at ~4am. I re-locked it. Also tweaked the input pointing into the cavity. The misalignment was mostly in pitch.
There was also a loud buzzing in the control room due to the audio cable being improperly seated in the mixer. I re-seated it.
I reproduced Gautam's sketch of the 1x1 and 1x2 Eurocrates into a pdf image that contains links to the appropriate DCCs in the legend (see attachement).
Thanks. Please update this wiki page too.
I've installed pyepics on Donatella running
sudo yum install pyepics
Pip and ipython did not seem to be installed yet.
There are many versions of the POP22 signal path I found on the elog, e.g. this thread. But what I saw at the LSC rack was not quite in agreement with any of those. So here is the latest greatest version.
Since the 2f signals are mainly indicators of power buildups and are used for triggering various PDH loops, I don't know how critical some of these things are, but here are some remarks:
I did some re-alignment of the POP beam on the IX in air table. Here are the details:
Tangentially related to this work - I took the nuclear option and did a hard reboot of the c1susaux Acromag crate on Sunday to fix the EPICS issue - it seems to be gone for now, see Attachment #5.