I am running some tests on the IMC servo board with an extender card so the IMC will not be locking for a couple of hours.
We've completed almost all of the in-situ testing of the c1psl channels. During this process, we identified several channels which needed to be rewired to different Acromags (BIO sinking v. sourcing). We also elected to change the connector type of a few channels for practical advantages. Those modifications and other issues found during testing are detailed below. Also attached are the updated channel assignments, with a column indicating the in-situ testing status of each channel.
With the Acromag chassis now permanently installed, we tested the C1PSL channels going over the channel list one by one, excluding the IMC channels which Gautam is taking responsibility for (the servo board itself is also in question).
The strategy is to check the response of input channels to specific output channels for expected behaviour whenever is possible.
We marked on the channel list spreadsheet the status of the channels that were tested.
In more detail
PMC Servo Card
Unlocked the PMC by switching C1:PSL-PMC_SW1. Tweaked C1:PSL-PMC_RAMP and observed a change in C1:PSL-PMC_PZT.
We misaligned MC1 to get a measurable signal in WFS channels. NDScoped the corresponding C1:IOO-WFS*_SEG*_I&Q channels and observed a change in those channels in response to switching the attenuation on and off.
The signals were compared to previous values for consistency. Then they were unplugged from the Acromag chassis to confirm their values went to 0 and returned to the same values after being reconnected.
The C1PSL crate has now been installed in a more permanent way in the rack.
After this work, I disabled logging and restarted the modbus service (and copied the current version of the systemd service file to the target directory for backup). The PMC and IMC lock alright. The system is now ready to be tested in-situ. I will separately continue my IMC Servo board tests in the evening.
One thought about how to protect against this kind of silent failure - how about we always run the modbus service with logging enabled, and then send out a warning email and stop the service if the logfile size suddenly blows up (which is characteristic of when the communications process dies)? This should be done in addition to the ping-ing of the individual IPs.
Regarding the burt-restore step that the systemd service runs after starting up the IOC - this is not even that useful, at least in the way it is currently setup (restore the "latest" burt snapshot file). If the maintenance takes >1hour as it often does, the "latest" snapshot for the system under maintenance is just garbage. So either the burt-restore should be for a "known good time" (dangerous because this will require frequent updates of the systemd service every time we find a new safe state) or we should just do it manually (my preference). Then there is no need to install custom packages on the server machine. Anyway, for now, I have not commented this step out.
Jordan is going to take pictures of all the electronics racks and update the relevant wiki pages.
Jon is going to write up the details of todays adventures. But the C1PSL Acromag chassis is sitting on the floor between the IMC beamtube and the 1X1 electronics rack, and is very much a trip hazard. Be careful if youre in that area.
I investigated the problem reported earlier today with the BIO1 channels. By logging the systemd messages generated when the IOC starts, I was immediately able to determine that the problem was not limited to BIO1. The modbus communications were failing for several other units as well.
Because some in-situ rewiring of a handful of channels had recently been done (more on this soon), I initially suspected that one of the Acromags had been damaged in the process. However, removing BIO1 (or other non-communicating modules) did not restore communications with the rest of the modules. To test whether the chassis was the source of the problem at all, we set up a fresh ADC (new out of the package) and directly connected it to the secondary Ethernet interface of c1psl. With only the one new ADC connected, the modbus IOC failed in exactly the same way.
To confirm that the new ADC did in fact work, we connected it to c1auxex in the same configuration. The unit worked fine connected to c1auxex. This established that the source of the problem was the c1psl host. After some extensive debugging, I traced the problem to a pre-execution script (part of the modbus IOC systemd service) which resets the secondary network interface (the one connected to the Acromag chassis) prior to launching the IOC. This was to ensure the secondary interface always had the correct IP address. It appears this reset was somehow creating a race condition that allowed the modbus initializations (first communications with the Acromags) to sometimes start before the network interface had actually come back up.
I still don't understand how this was happening, or why the pre script worked just fine up until yesterday, but eliminating the network interface reset fixes the problem in 100% of the trials we ran. Unfortunately we lost the entire day to debugging this problem, so the final round of testing is still to be completed. We plan to pick it back up tomorrow afternoon.
We are going to replace the old Sun c1ioo with a modernized supermicro. At the opportunity, remove the DAC and BIO cards to use them with the new machines. BTW I also have ~4 32ch BIO cards in my office.
To debug a problem with the new c1psl (later elog), we needed a Supermicro EPICS server that was using the shared EPICS/modbus/asyn binaries rather than a local install. Of those available in the lab (c1iscaux, c1vac, c1susaux being the others), this was the only one which uses the shared install. So I
At which point Jon reset the software end, I restored the slow bias voltage and re-enabled the local damping. The optic seems to have damped okay. The Oplev spot is back in ~center of the QPD and the green beam can be locked to a TEM00 mode (so the alignment is okay - the IR beam is unavailable while c1psl issues are being sorted but I judge that things are back to the nominal state now).
After discussing with Koji, I removed the PZT driver and associated AI card from the Eurocrate at 1X2. The corresponding backplane connectors were also removed from the cross connects. An additional cable going from the DAC to IDC adaptor on 1X2 was removed. Finally, some cables going to the backplane P1 and P2 connectors for slots in which there were no cards were removed.
Finally, there is the IMC WFS whitening boards. These were reconfigured in ~2016 by Koji to have (i) forever whitening, and (ii) fixed gain. So the signals from the P1 connector no longer have any influence on the operation of this board. So I removed these backplane cables as well.
Some pics attached. The only cross connect cabling remaining on the south side of 1X2 is going to the fast BIO adaptor box - I suspect these are the triggered fast whitening switching for the aforementioned WFS whitening board. If so, we could potentially remove those as well, and remove all the cross connects from 1X1 and 1X2.
Update 1720: indeed, as Attachment #2 shows, the RTCDS BIO channels were for the WFS whitening switching so I removed those cables as well. This means all the xconnects can be removed. Also, the DAC and BIO cards in c1ioo are unused.
Do we want to preserve the ability to use the PZT driver in 1X2?
There was some work done on the Acro crate this morning. Unclear if this is independent, but I found that the IMC servo board IN1 slider doesn't respond anymore, even though I had tested it and verified it to be working. Patient debugging showed that BIO1 (and only that acromag unit with the static IP 192.168.114.61) doesn't show up on the subnet in c1psl. Hopefully it's just a loose network cable, if not we will switch out the unit in the afternoon.
Jon is going to make a python script which iteratively pings all devices on the subnet and we will put this info on an MEDM screen to catch this kind of silent failure.
I realigned the input pointing into the PMC this morning. Usually, the way I do this is to minimize any discernible mode structure in the PMC reflection CCD image. Today, I noticed that making the DC reflection go down also makes the DC transmission go down. Possibilities:
Allegra had no network cable and no mouse. We found Allegra'snetwork cable (black) and connected it.
I found a dirty old school mouse and connected it.
I wiped Allegra and now I'm currently installing debian 10 on allegra following Jon's elog.
04/01 update: I forgot to mention that I tried installing cds software by following Jamie's instruction: I added the line in /etc/apt/sources.list.d/lscsoft.list: "deb http://software.ligo.org/lscsoft/debian/ stretch contrib". But this the only thing I managed to do. The next command in the instructions failed.
deb http://software.ligo.org/lscsoft/debian/ stretch contrib"
Revised noise estimates, correcting a couple of factor of 2 and factor of pi errors found in the coil driver noise calculation. Also resolves a strain vs. displacement units confusion using the new pygwinc. Gautam and I have checked these noises against the analytical predictions and believe they are now accurate. Attachments are again:
Attachment 1: Phase quadrature
Attachment 2: Amplitude quadrature
Attachment 3: Comparison to aLIGO design (phase quadrature)
I used existing BNC cables running from the PSL table to the PSL rack and reassigned them to the PSL Shutter and PMC transmission PD channels.
The PSL shutter turned out to be a sinking channel. Jordan reconnected the PSL shutter wires to a sinking BIO Acromag. Channel list is updated.
Both channels have been tested to be working as expected.
gautam add on about EPICS:
P.S - there is a problem we noticed - if the modbus process is started with the local subnet not having a fixed IP address, then all the EPICS channels will not be responsive. The way to fix this is to run the following sequence of commands:
Jordan and I removed another 10 kg of cabling from 1X2. The c1iool0 crate now has all cabling to it disconnected - but it remains in the rack because I can't think of a good way to remove it without disturbing a bunch of cabling to the fast c1iool0 machine. We can remove it the next time the vertex FEs crash. Cross connects have NOT been removed - we will identify which cross connects are not connected to the fast system and trash those.
Updated noise budget curves, now computed using the latest version of pygwinc. This resolves the inconsistency between the gwinc quantum noise curves and Gautam's analytic calculations. As before, the key configuration parameters are listed in the figure titles.
Attachment 1: Phase quadrature
Attachment 2: Amplitude quadrature
Attachment 3: Comparison to aLIGO design (phase quadrature)
The quantum noise curves here are not correct. c.f. amplitude quadrature noise budget.
Had to reboot both end machines and the c1rfm model to get the TRX and TRY signals to the LSC models. Now both arms can be locked using POX/POY respectively.
Channel list with test status
== Test Status ==
[done] Lock PMC and IMC
[done] IMC Servo board test
[done] IMC LO Det Mon channel check
[0th order] WFS quadrant DC mon
[none] WFS I/F monitors
[0th order] WFS attenuators
[none] IOO QPD channels
[done] FSS readbacks
[done] PMC readbacks
Some more detailed elogs about the individual tests will follow.
Basically, I have characterized the IMC Servo board in detail. The summary finding is that the IN2 (=AO gain) slider needs to be investigated.
All other channels need to be verified in a more thorough fashion than my basic checks which were just to guarantee the core interferometer functionality, which is important to me.
[JV, JWR, YD, GV]
$TARGET_DIR = /cvs/cds/caltech/target
It remains to (Jon is taking care of these)
On Monday, we will remove the old c1psl and c1iool0 machines from the electronics rack and install the Acromag crate in a more permanent way. We will also clean up some of the old cabling and cross connects, althoug the situation seems so complicated (some cross connects are also used by the rtcds c1ioo expansion chassis) that I am inclined not to remove any cables.
The area around 1X1/1X2 has a lot of dangling cables and general detritus. Be careful if you are walking around there. We will clean up on monday.
There are several problems evident already.
For now, I've returned the old c1psl connections, the PMC and IMC are both locked. Need to do some debugging on the bench.
And so it begins.
Barring objections, tomorrow (Friday 28 Feb 2020) morning I will commence the switch
There was some UNELOGGED work at EX today. The DFD outputs were also hijacked for loss measurement. Unclear who the culprit was, but there is now a broad noise bump centered around ~180 Hz in the ALS X noise curve, which certainly wasn't there yesterday. Maybe let's keep the few working systems working, it is annoying to have to deal with these auxiliary issues every night. I'll push ahead with locking, hopefully the ALS noise is "good enough".
While my checks of the AO signal path have thrown up some unanswered questions, I don't think there's any evidence in those measurements to suggest the AO crossover can't be realized. Thinking about it today though - I was wondering if it could be that the IN1 gain slider of the CM board is configured optimally. Looking back at some data, when the ALS CARM offset is zeroed, the CM_SLOW digitized data has a peak-to-peak range of ~200 cts. This is paltry. One possibility is that as I am upping the AO path gain, I'm simply injecting the electronics noise of the CM board into the IMC error point. I'd say it is safe to up the IN2 gain by 20dB to -12 dB, in which case the peak-to-peak would be ~2000 cts, still only 10% of the ADC range. It'll be straightforward to re-scale the CARM_B loop gain to account for this. Let's see if this helps.
I'd also like to measure the spectrum of the REFL11_I signal in a few different states. I think I should be able to do this using the OUT2 of the CM servo board. These are the things to try tonight:
in prep for the install tomorrow, we did the following:
Barring objections, tomorrow (Friday 28 Feb 2020) morning I will commence the switch (I still want to work on the IFO tonight).
In 1X1, there is a box labelled "FSS REF" below a KEPCO HV supply. This box had a power cable that wasn't actually connected to any power. I removed said cable.
In the style of the KA characterization of the CM board, the AO path gain EPICS slider (IN2) of the IMC servo board was stepped by 1 dB through the full available range of -32 dB to +31 dB. For each value of the requested gain, I measured the TF from the injected signal (to IN2) to TP1A on the IMC servo board. I used the BNC connector for this test, whereas we use the LEMO connector for the AO path. The source was tee-d off at the SR785 side, with one leg going to IN2 of the IMC servo board, and the other going to CH1A of the SR785. TP1A of the IMC board was connected to CH2A of the SR785.
Attachment #1 - Measured gain vs requested gain.
Attachment #2 - Frequency dependent transfer functions
The motivation here is to try and figure out why I cannot engage the AO path smoothly in the CARM handoff part of lock acquisiton. I plan to use this information to do some loop modeling and project laser frequency noise coupling in various stages of the lock acquisition process.
Today, I did the following tests (and so was touching electronics/cables at/around 1X2):
Results to follow.
After this work, I reverted the EPICS channels to the usual values. The IMC can be locked.
To supplement the material presented during the BHD review, I've put together a projected noise budget for the 40m. Note these are the expected interferometer noises (originating in the IFO itself), not BHD readout noises. The key parameters for each case are listed in the figure title. Also attached is a tarball (attachment 4) containing the ipython notebook, data files, and rolled-back version of pygwinc which were used to generate these figures.
Attachment 1: Phase quadrature readout.
Attachment 2: Comparison to aLIGO design sensitivity (phase quadrature).
Attachment 3: Amplitude quadrature readout.
In order to measure the loss in the arm cavities in reflection, we use the DC method described in T1700117.
It was not trivial to find free channels on the LSC rack. The least intrusive way we found was to disconnect the ALS signals DSUB9 (Attachment 1) and connect a DSUB breakout board instead (Attachment 2).
The names of the channels are ALS_BEATY_FINE_I_IN1_DQ for AS reflection and ALS_BEATY_FINE_Q_IN1_DQ for MC transmission. Actually, the script that downloads the data uses these channels exactly...
We misalign the Y arm (both ITM ad ETM) and start a 30 rep measurement of the X arm loss cavity using /scripts/lossmap_scripts/armLoss/measureArmLoss.py and download the data using dlData.py.
We analyze the data. Raw data is shown in attachment 3. There is some drift in the measurement, probably due to drift of the spot on the mirror. We take the data starting from t=400s when the data seems stable (green vertical line). Attachment 5 shows the histogram of the measurement
X Arm cavity RT loss calculated to be 69.4ppm.
We repeat the same procedure for the Y Arm cavity the day after. Raw data is shown in attachment 5, the histogram in attachment 6.
Y Arm cavity RT loss calculated to be 44.8ppm. The previous measurement of Y Arm was ~ 100ppm...
Loss map measurement is in order.
We did some quick DC balancing of the MC2 coil drivers to reduce the l2a coupling. We updated the gains in the C1:SUS-MC2_UL/UR/LR/LLCOIL to be 1, -0.99, 0.937,-0.933, respectively. The previous values were 1, -1, 1, -1.
The procedures are the following:
Drive UL+LR and change the gain of LR to zero pitch.
Drive UR+LL and change the gain of LL to zero pitch.
Lastly, drive all 4 coils and change UR & LR together to zero yaw.
We used C1:SUS-MC2_LOCKIN1_OSC to create the excitations at 33 Hz w/ 30,000 cts. The angular error signals were derived from IMC WFSs.
While this time we did things by hand, in the future it should be automated as the procedure is sufficiently straightforward.
Seems that the GPS is out of sync on donatella. We could not get any data from diaggui...
The HVAC people replaced a valve and repaired the pneumatic plumbing on the roof air handler. Temperature has been stable during the day since Thursday. If anyone is in the control room during the evening, please make a note of the temperature.
to make the comparisons meaningfully
one needs to correct for the feedback changes
In order to adjust the relative phase for PDH locking, we used the Siglent SDG 1032X function generator which has two outputs whose relative phase can be adjusted.
This Siglent function generator was borrowed from Yehonathan's setup near the PSL table and can be found at the X end disconnected from our setup after our use.
Initially, we used the Siglent at 231.250 kHz and 5 Vpp from each output with zero relative phase to lock the green arm cavity. By moving the phase at intervals of 5deg and looking at the PDH error signals when the cavity was unlocked we concluded that 0deg probably looked like it had the largest linear region (~1.9 V on the yaxis. Refer elog 15218 for more information) as expected.
Then we tried the same for 225.642 kHz, 5 Vpp, and found the optimal demod phase to be -55deg, with linear region of ~3 V (Ref. Attachment 2). A 'bad' frequency 180 kHz was optimized to 10deg and linear region of ~1.5 V.
The error signals at higher frequencies appeared to be quite low (not sure why at the moment) and tuning the phase did not seem to help this much.
For the noise measurement, the IFO arms were locked to IR and green, but even after optimizing the transmission with dither, we couldn't achieve best locking (green transmission was around ~0.2). Further, the IMC went out of lock during the experiment after which Koji helped us by adjusting the gains a locking point of the PMC servo. Attachment 1 contains some noise curves for the 3 frequencies with a reference from an earlier 'good' time.
Check out this elog: ELOG 4354
If this summing box is still used as is, it is probably giving the demod phase adjustment.
Once we adjust the phase we may be able to increase the servo gain for optimal locking. Unless it may be a good idea to increase the gain without optimizing the phase?
Could you please put physical units on the Y-axis and also put labels in the legend which give a physical description of what each trace is?
It would also be good to a separate plot which has the IR locking signal and the green locking signal along with this out of loop noise, all in the same units so that w can see what the ratio is.
We proceeded with the trying to measure the ALS out-of-loop noise of the X arm when the X arm cavity is green-locked using different PDH sideband frequencies.
Before doing the experiment, Koji helped us with getting the arm cavities locked in IR using LSC (length) and ASC (angular).
With the arms locked in IR and green, we repeated the same measurements as before at different sideband frequencies (Refer Attachment 1 - label in Hz). We did not optimize the phase nor did we look at the PDH error signal today which is possibvly why we did not see an improvement in the noise. We will look into this possibly tomorrow.
We measure the IMC transfer function using SR785.
We hook up the AOM driver to the SOURCE OUT, Input PD to CHANNEL ONE and the IMC transmission PD to CHANNEL TWO.
We use the frequency response measurement feature in the SR785. A swept sine from 100KHz to 100Hz is excited with an amplitude of 10mV.
Attachment 1 shows the data with a fit to a low pass filter frequency response.
IMC pole frequency is measured to be 3.795KHz, while the ringdowns predict a pole frequency 3.638KHz, a 4% difference.
The closeness of the results discourages me from calibrating the PDs' transfer functions.
I tend to believe the pole frequency measurement a bit more since it coincides with a linewidth measurement done awhile ago Gautam was telling me about.
I think of trying to try another zero-order ringdown but with much smaller excitation than what used before (500mV) and than move on to the first-order beam.
Also, it seems like the reflection signal in zero-order ringdown (Attachment 2, green trace) has only one time constant similar to the full extinction ringdown. The reason is that due to the fact the IMC is critically coupled there is no DC term in the electric field even when the extinction of light is partial. The intensity of light, therefore, has only one time constant.
Fitting this curve (Attachment 3) gives a time constant of 18us, a bit too small (gives a pole of 4.3KHz). I think a smaller extinction ringdown will give a cleaner result.
Unlikely, the alignment was probably just not good. I restored the alignment and now the arms can be locked to IR frequency.
Even though we were not able to lock the the IR beat (by enabling LSC) during the day possibly because of increased seismic activity
Even though we were not able to lock the the IR beat (by enabling LSC) during the day possibly because of increased seismic activity, we tried to the measure the ALS beat frequency noise by changing the PDH side-band frequency to different values.
I tried choosing values that corresponded to the peaks in the PM/AM as found in elog:15206 but for some reason unknown to us the cavity did not lock between 700-800 kHz.
The three attachments have data for different sideband frequencies:
Attachment 1: 819.472 kHz (peak in PM/AM, measured around noon)
Attachment 2: 225.642 kHz (peak in PM/AM, measured earlier in the morning)
Attachment 3: 693.500 kHz (not a peak in PM/AM)
We don't think these plots mean much and will do the measurement at some quieter time more systematically.
While doing the experiment, the ITMY pitch actuation was changed from -2.302 to -2.3172V because it locked better.
The ITMX, ETMX alignment was also tweaked to try to lock with different sideband frequencies and then restored to the values that were found earlier in the morning.
In the process of setting up some cabling at 1Y2, I must've bumped a cable to the c1lsc expansion chassis. Anyways, the c1lsc models crashed. I ran the reboot script around 530pm PDT. Usual locking behavior was recovered after this. The work at 1Y2 was:
The IN2 to CM board was already connected to I single ended output of the ALS X demodulator. The ~100 Hz UGF digital locking using the CM_SLOW path is straightforward but I didn't have any success with the AO path tonight. I wonder how high BW this lock can be made without injecting a ton of noise into the IMC loop, given that the EX uPDH only has ~ 10 kHz UGF.
Attachment #1 shows the spectra of the ALS signal
Attachment #2 is an OLTF measurement.
[Meenakshi, Gautam, Shruti]
- We initially aligned the arm cavities to get the green lasers locked to them. For the X arm cavity, we tweaked the ITMX and ETMX pitch and yaw and toggled the X green shutter until we saw something like a TEM00 mode on the monitor and a reasonable transmitted power.
- With the LSC servo enabled, the IR light also became resonant with the cavities.
- Then we measured the noise in different configurations. Attachment 1 shows the the ALS OOL (in the IR beat signal) noise with the arms locked inidividually via PDH.
The script for plotting the ALS beat frequency noise is:
I measured the transfer function of the AO path, and think that there are some features indicative of a problem somewhere in the IMC locking loop.
Regardless of the locking scheme used, high bandwidth control of the laser frequency relies on the fact that the laser frequency is slaved to the IMC cavity length with nearly zero error below ~50 kHz (assuming the IMC loop has a UGF > 100 kHz). In my single arm experiments, I didn't know what to make of the ripples that became apparent in the measured OLTF as the AO gain was ramped up.
Tonight, I measured the TF of the "AO path", which modifies the error point of the IMC, thereby changing the laser frequency.
Attachment #1 shows the result of the measurement.
I didn't use POX / POY as a sensor to confirm that this is real frequency noise, I will do so tomorrow. But it may be that realizing a stable crossover is difficult with so many features in the AO path.
Previous thread with a somewhat detailed characterization of the IMC loop electronics.
A few years ago, Koji and I setup a delay line phase shifter, which can be used to impart a (switchable) delay to a signal path. Since we talked about reviving the fast (= high bandwidth) ALS control scheme at the meeting, I reminded myself of the infrastructure available.
For a beat note in the regime 10-100 MHz, we should have plenty of range in this module to add a delay such that we zero one quadrature of the ALS DFD output (for a linear error signal).
I then proceeded to connect the single-ended front panel BNC corresponding to the ALS_X_I DFD channel to the IN2 input of the CM board (this would be what we use for high bandwidth ALS feedback). The conventional ALS system uses the differential output from a rear-panel D-sub, so in principle, both systems could run in parallel. I confirmed that I could see a signal when the IN2 path on the CM board was engaged (monitored using ndscope at the CM_Slow output), and that this signal stabilized when the green laser was locked to the X-arm length, which itself was slaved to the PSL frequency using the usual POX locking scheme. I have not yet routed the LO leg of the ALS_X beat through the delay line phase shifter - see next elog for details.
Update about the ALS MEDM screen slider: the trick was to change the OMSL field of the C1:LSC-BO_1_0 channel to "closed_loop" instead of "supervisory". Once this is done, the DOL value of the same channel can be set to the soft channel C1:ALS-DelayCalc, which sets the 16 bit binary string that controls the delay. Because arbitrary delays are not possible, I think it's more natural for the user to interact with this 16-bit binary string rather than the actual delay itself. So the MEDM screen has been slightly modified from what is shown in Attachment #1.
Over the last couple of days, I've been trying to see if I can measure the phase advance due to the AO path - however, I've been unable to do so for any combination of CM board IN1 gain and MC Servo board IN2 gain I've tried. Yesterday, I tried to understand the loop shapes I was measuring a little more, and already, I think I can't explain some features.
Attachment #1 shows the TF measured (using SR785, and the EXC_A bank of the CM board) when the CM Slow path has been engaged.
Attachment #2 shows error signal spectra for the in-loop PRFPMI DoFs, for a few different conditions.
I believe that a stable crossover is hopeless under these conditions.