No real progress tonight - I made it a bunch of times to the point where CARM was RF only, but I never got to run a measurement to determine what the DARM_B loop gain should be to make the control fully RF.
There was a jump in the main volume pressure at ~6pm PDT yesterday. The cause is unknown, but the pressure doesn't seem to be coming back down (but also isn't increasing alarmingly).
I wanted to look at the RGA scans to see if there were any clues as to what changed, but looks like the daily RGA scans stopped updating on Dec 24 2019. The c0rga machine responsible for running these scans doesn't respond to ssh. Not much to be done until the lockdown is over i guess...
Some short notes, more details tomorrow.
Attachment #1 shows time series of some signals, from the time I ramp of ALS CARM control to a lockloss. With this limited set of signals, I don't see any clear indication of the cause of lockloss, but I was never able to keep the lock going for > a couple of mins.
Attachment #2 shows the CARM OLTF. Compared to last week, I was able to get the UGF a little higher. This particular measurement doesn't show it, but I was also able to engage the regular boost. I did a zeroth order test looking at the CM_SLOW input to make sure that I wasn't increasing the gain so much that the ADC was getting saturated. However, I did notice that the pk-to-pk error signal in this locked, 5kHz UGF state was still ~1000 cts, which seems large?
Attachment #3 shows the DTT measurement of the relative gains of DARM A and B paths. This measurement was taken when the DARM_A gain was 1, and DARM_B gain was 0.015. On the basis of this measurement, DARM_B (=AS55) sees the excitation injected 16dB above the ALS signal, and so the gain of the DARM_B path should be ~0.16 for the same UGF. But I was never able to get the DARM_B gain above 0.02 without breaking the lock (admittedly the lockloss may have been due to something else).
Attachment #4 shows a zoomed in version of Attachment #1 around the time when the lock was lost. Maybe POP_YAW experienced too large an excursion?
Some other misc points:
I think the feedforward filters used for stabilizing MCL with vertex seismometers would benefit from a retraining (last trained in Sep 2015).
I wanted to re-familiarize myself with the seismic feedforward methodology. Getting good stabilization of the PRC angular motion as we have been able to in the past will be a big help for lock acquisition. But remotely, it is easier to work with the IMC length feedforward (IMC is locked more often than the PRC). So I collected 2 hours of data from early Sunday morning and went through the set of steps (partially).
Attachment #1 shows the performance of a first attempt.
Attachment #2 shows a comparison between the filter used in Attachment #1 and the filters currently loaded into the OAF system.
Attachment #3 is the asd after implementing a time domain Wiener filter, while Attachment #4 is an actual measurement from earlier today - it's not quite as good as Attachment #3 would have me expect but that might also be due to the time of the day.
Conclusions and next steps:
On the basis of Attachments #3 and #4, I'd say it's worth it to complete the remaining steps for online implementation: FIR to IIR fitting and conversion to sos coefficients that Foton likes (prefereably all in python). Once I've verified that this works, I'll see if I can get some data for the motion on the POP QPD with the PRMI locked on carrier. That'll be the target signal for the PRC angular FF training. Probably can't hurt to have this implemented for the arms as well.
While this set of steps follows the traditional approach, it'd be interesting if someone wants to try Gabriele's code which I think directly gives a z-domain representation and has been very successful at the sites.
* The y-axes on the spectra are labelled in um/rtHz but I don't actually know if the calibration has been updated anytime recently. As I type this, I'm also reminded that I have to check what the whitening situation is on the Pentek board that digitizes MCL.
The email address in the N2 checking script wasn't right - I now updated it to email the 40m list if the sum of reserve tank pressures fall below 800 PSI. The checker itself is only run every 3 hours (via cron on c1vac).
I reset the remote of this git repo to the 40m version instead of Jon's personal one, to ensure consistency between what's on the vacuum machine and in the git repo. There is now a N2 checker python mailer that will email the 40m list if all the tank pressures are below 600 PSI (>12 hours left for someone to react before the main N2 line pressure drops and the interlocks kick in). For now, the script just runs as a cron job every 3 hours, but perhaps we should integrate it with the interlock process
I'd like to re-measure the transfer function from driving MC2 position to the MC_L_DQ channel (for feedforward purposes). Swept sine would be one option, but I can't get the "Envelope" feature of DTT to work, the excitation amplitude isn't getting scaled as specified in the envelope, and so I'm unable to make the measurement near 1 Hz (which is where the FF is effective). I see some scattered mentions of such an issue in past elogs but no mention of a fix (I also feel like I have gotten the envelope function to work for some other loop measurement templates). So then I thought I'd try broadband noise injection, since that seems to have been the approach followed in the past. Again, the noise injection needs to be shaped around ~1 Hz to avoid knocking the IMC out of lock, but I can't get Foton to do shaped noise injections because it doesn't inherit the sample rate when launched from inside DTT/awggui - this is not a new issue, does anyone know the fix?
Note that we are using the gds2.15 install of foton, but the pre-packaged foton that comes with the SL7 installation doesn't work either.
The envelope feature for swept-sine wasn't working because i specified the frequency grid in the wrong order apparently. Eric von Reis has been notified to include a sorting algorithm in future DTT so that this can be in arbitrary order. fixing that allows me to run a swept sine with enveloped excitation amplitude and hence get the TF I want, but still no shaped noise injections via foton 😢
The problem is that foton does not inherit the model sample rate when launched from DTT/awggui. This is likely some shared/linked/dynamic library issue, the binaries we are running are precompiled presumably for some other OS. I've never gotten this to work since we changed to SL7 (but I did use it successfully in 2017 with the Ubuntu12 install).
do you really mean awggui cannot make shaped noise injections via its foton text box ? That has always worked for me in the past.
If this is broken I'm suspicious there's been some package installs to the shared dirs by someone.
This afternoon, I kept the PRM locked for ~1hour and then measured transfer functions from the PRM angular actuators to the POP QPD spot motion for pitch and yaw between ~1pm and 4pm. After this work, the PRM was misaligned again. I will now work on the feedforward filter design.
Using the data I collected yesterday, the POP angular FF filters have been trained. The offline time-domain performance looks (unbelievably) good, online performance will be verified at the next available opportunity(see update).
The sequence of steps followed is the same as that done for the MCL FF filters. The trace that is missing from Attachment #1 is the measured online subtraction. Some rough notes:
Update Apr 5 1145pm:
I don't have a recent measurement of the optical gain of this config so I can't undo the loop, but in-loop performance doesn't suggest any excess in the 10-100 Hz band. Interestingly, there is considerable improvement below 10 Hz. Maybe some of this is reduced A2L noise because of the better angular stability, but there is also improvement at frequencies where the FF isn't doing anything, so could be some bilinear coupling. The two datasets were collected at approximately the same time in the evening, ~5pm, but on two different days.
I wonder how much noise is getting injected into PRC length at 10-100 Hz due to this. Any change the PRC ERR?
Covid 19 motivated me to revive the summary pages. With Alex Urban's help, the infrastructure was modernized, the wiki is now up to date. I ran a test job for 2020 March 17th, just for the IOO tab, and it works, see here. The LDAS rsync of our frames is still catching up, so once that is up, we can start the old condor jobs and have these updated on a more regular basis.
On Monday, I hooked up an AG4395 to the PMC error point (using the active probe). The idea was to take a spectrum of the PMC error point every time the FSS PC drive RMS channel indicated an excursion from the nominal value. An initial look at the results don't suggest that this technique is particularly informative. I'll have to think more about a workaround, but please share your ideas/thoughts if you have some.
Also, the feature in the spectrum at ~110 kHz makes me suspect some kind of loop instability. I'll measure the IMC loop OLG at the next opportunity.
It appears that the EY green steering PZTs have somehow lost their bipolar actuation range. I will check on them the next time I go to the lab for an N2 switch.
Could be that the power outage busted something in the drive electronics.
I went to EY and saw that the HV power supply was only putting out 50 V and had hit the current limit of 10 mA (nominally, it should be 100 V, drawing ~7mA). This is definitely a problem that has come up after the power shutdown event, as when I re-energized the HV power supply at EY, I had confirmed that it was putting out the nominal values (the supply was not labelled with these nominal numbers so I had to label it). Or maybe I broke it while running the dither alignment tests yesterday, even though I never drove the PZTs above 50 Hz with more than 1000cts (= 300 mV * gain 5 in the HV amplifier = 1.5 V ) amplitude.
The problem was confirmed to be with the M2 PZT (YAW channel) and not the electronics by driving the M2 PZT with the M1 channels. Separately, the M1 PZT could be driven by the M2 channels. I also measured the capacitance of the YAW channels and found it to be nearly twice (~7 uF) of the expected 3 uF - this particular PZT is different from the three others in use by the ASX and ASY system, it is an older vintage, so maybe it just failed? 😔
I don't want to leave 100 V on in this state, so the HV supply at EY was turned off. Good GTRY was recovered by manual alignment of the mirror mounts. If someone has a spare PZT, we can replace it, but for now, we just have to live with manually aligning the green beam often.
I've been thinking about the IMC WFS. I want to repeat the sort of analysis done at LLO where a Finesse model was built and some inferences could be made about, for example, the Gouy phase separation b/w the sensors by comparing the Finesse sensing matrix to a measured sensing matrix. Taking the currently implemented output matrix as a "measurement" (since the IMC WFS stabilize the IMC transmission), I don't get any agreement between it and my Finesse model. Could be that the model needs tweaking, but there are several known issues with the WFS themselves (e.g. imbalanced segment gains).
Building the finesse model:
Some notes about the WFS heads:
Update 215 pm 5/6: adding in some comments from Rana raised during the meeting:
The apparent increase in the ALS noise (witnessed in-loop, e.g. Attachment #2 here) during the CARM offset reduction may have an optomechanical origin.
Update 415pm 5/6: Per the discussion at the meeting, I have now uploaded as Attachment #2 the force-->displacement (i.e. m/N) transfer functions. I now think these are appropriate units. For the ALS case, we could convert the m/N to Hz/N of extra frequency noise imprinted on the AUX laser due to the increased cavity motion. Is W/N really better here, since the mechanism is extra frequency noise on a beatnote, and there isn't really a PDH or DC error signal?
OK so the QPD segments are in the "+" orientation when the 40m IMC WFS heads are mounted at 45 deg. I thought "+" was the natural PIT/YAW basis but I guess in the the LIGO parlance, the "X" orientation was considered more natural.
This is the doc from Keita Kawabe on why the WFS heads should be rotated.
Finally - Attachment #1. This plot uses 16 Hz EPICS data. All y-axes are uncalibrated for now, but TRX/TRY are normalized such that the POX/POY lock yields a transmission of 1. CARM UGF is only ~3 kHz, no boosts were turned on yet.
Attachment #2 and Attachment #3 are phone photos of the camera images of the various ports. After some alignment work, the transmitted arm powers were ~200, i.e. PRG ~10. fwiw, this is the darkest i've ever seen the 40m dark port. c.f. 2016. Of course, the exposure time / ND filter / light levels could all have changed.
This work was possible during the daytime (~6pm PDT), but probably only because it was Sunday. The other rate limiting factor here is the franky terrible IMC duty cycle. TBH, I didn't honestly expect to get so far and ran out of time, but I think the next steps are:
As usual, I would like to request that we don't change the IFO as far as possible until the BHD vent, i found it pretty difficult to get here.
Attachment #4 now shows the measured DARM OLTF when DARM is entirely on AS55_Q control. UGF is ~120 Hz and the phase margin is ~30 deg, seems okay for a first attempt. I'll now need to infer the OLTF over a wider range of frequencies by lining this measurement up with some model, so that I can undo the loop in plotting the DARM ASD.
Attachment #1 is meant to show that having a T=500ppm PR2 optic will not be the dominant contributor to the achievable recycling gain. Nevertheless, I think we should change this optic to start with. Here, I assume:
In relaity, I don't know how good the MM is between the PRC and the arms. All the scans of the arm cavity under ALS control and looking at the IR resonances suggest that the mode-matching into the arm is ~92%, which I think is pretty lousy. Kiwamu and co. claim 99.3% matching into the interferometer, but in all the locks, the REFL mode looks completely crazy, so idk
Yes, \eta_A is the (average) round-trip loss for an arm cavity. I'd estimate this is ~100ppm currently. I edited the original elog to fill in this omission.
The RC mirror specs require some guesswork - the available specs for the Laseroptik mirrors (PR3) are for a 48 degree angle of incidence, and could be as high as 0.5 %. According to the poster, the spec is 2.6% loss inside the recycling cavity but I don't know where I got the number for the AR surface of the G&H PR2, and presumably that includes some guess I made for the MM between the PRC and the arm. Previously, assuming ~1-2% loss inside the RC gave good agreement between model and measurement. Certainly, if we assume similar numbers, a recycling gain of ~11 (200 * T_P=5.637%) is reasonable. But I think we need more data to make a stronger statement.
Is \eta_A the roundtrip loss for an arm?
Thinking about the PRG=10 you saw:
- What's the current PR2/3 AR? 100ppm? 300ppm? The beam double-passes them. So (AR loss)x4 is added.
- Average arm loss is ~150ppm?
Does this explain PRG=10?
I think the boosts that are currently stuffed on the CM board are too aggressive to be usable for locking the interferometer. I propose some changes.
[CM board schematic]
[CM board transfer function measurement]
[Measurement of the AO path TF]. Empirically, I have observed that the CARM OLTF has ~90 degrees phase margin available at the UGF when no boosts are engaged, which is consistent with Koji's measurement. Assuming we want at least 30 degrees phase margin in the final configuration, and assuming a UGF to be ~10 kHz, the current boosts eat up way too much phase at 10 kHz. Attachment #1 shows the current TFs (dashed lines), as the boosts are serially engaged. I have subtracted the 180 degrees coming from the inverting input stage. The horizontal dash-dot line on the lower plot is meant to indicate the frequency at which the boost stages eat up 60 degrees of phase, which tells us if we can meet the 30 degree PM requirement.
In solid lines on Attachment #1, I have plotted the analogous TFs, with the following changes:
These changes will allow possibly two super boosts to be engaged if we can bump up the CARM UGF to ~15 kHz. We sacrifice some DC gain - I have not yet done the noise analysis of the full CARM loop, but it may be that we don't need 120 dB gain at DC to be sensing noise limited. I suppose the pole frequencies can also be halved if we want to keep the same low frequency gain. In any case, in the current form, we can't access all that gain anyways because we can't enable the boosts without the loop going unstable.
The input referred noise gets worse by a factor of 2 as a result of these changes, but the IN1 gain stage noise is maybe already higher? If this sounds like a reasonable plan, I'll implement it the next time I'm in the lab.
This is to facilitate the summary page config fines to be pulled from nodus in a scripted way, without being asked for authentication. If someone knows of a better/more secure way for this to be done, please let me know. The site summary pages seem to pull the config files from a git repo, maybe that's better?
This EQ in Nevada seems to have tripped all watchdogs. ITMX was stuck. It was released, and all the watchdogs were restored. Now the IMC is locked.
The service had failed at 16:09 yesterday. I just restarted it and am now able to fetch data again.
Unrelated to this work: I restarted the httpd service on nodus a couple of times this afternoon while experimenting with the summary pages.
Please try it out and tell me about any problems in getting fresh data.
To test a hypothesis, I have left the PSL shutter closed. I notice significant glitches in the dark electronics offsets on all the 11 MHz photodiode I/Q demodulated input channels, which appear coherent. These are non-negligible in magnitude - for now they are uncalibrated in cts, but for an estimate, the POX11 channel shows a shift of ~20 cts (~200uV at the input to the whitening board), while the PDH fringe is ~200 cts pk2pk. A first look is in Attachment #1. The fact that it's in all the 11 MHz channels makes me suspect something in the RF chain, maybe some amplifier? I'll open the shutter tomorrow.
A big factor in how much IFO locking activities can take place is how cooperative the IMC is.
Since the c1psl upgrade, the IMC duty cycle has definitely deteriorated. I took a measurement of the dark noise at the IMC error point with 1 Hz FFT binwidth, with all electrical connections to the IMC servo board except the Acromag and Eurocrate power disconnected. I was horrified at the prominence of 60 Hz harmonics - see Attachment #1. In the past, this kind of feature has been indicative of some error in the measurement technique - but I confirmed that the lines remain even if I unplug the GPIB box, and all combinations of floating/grounded inputs that I tried. We know for sure that there is some excess noise imprinted on the laser light post upgrade. While these lines almost certainly are not responsible for the PCdrive RMS going bonkers, surely this kind of electrical situation isn't good?
Attachment #2 shows the same information translated to frequency noise units, taking into account the complementary sensitivity function, L/(1+L) - the sum contribution of the 60 Hz peaks to the RMS is ~11.5% of the total over the entire band (c.f. 1.7 % that is expected if the noise at multiples of 60 Hz was approximately equal to the surrounding noise levels). Moreover, the measured RMS is 55 times higher than a LISO model.
How can this be fixed?
Provided the IMC is cooperative, the input pointing isn't drifting, and the RF offsets aren't jumping around too much, the locking sequence is now pretty robust.
Most of the analysis uses data between the GPS times 1274418176 and 1274419654 that are recorded to frames.
Here, I provide some details of the sequence. Obviously, I am presenting one of the quickest transitions to the fully locked state, I don't claim that every attempt is so smooth. But it is pretty cool that the whole thing can be done in ~3 minutes.
See Attachment #1 for the labels.
This particular lock held for ~20 minutes so I could run some loop characterization measurements etc.
I am struggling to explain:
In order to estimate the free-running DARM displacement noise, I measured the DARM OLTF using the usual IN1/IN2 prescription. The measured data was then used to fit some model paramters for a loop model that can be used over a larger frequency range.
In summary, the UGF is ~150 Hz and phase margin is ~30 deg. This loop would probably benefit from some low-pass filter being turned on.
I am able to realize ~8 kHz UGF with ~60 degrees of phase margin on the CARM loop OLTF (combination of analog and digital signal paths).
The response of the PRFPMI length degrees of freedom as measured in the LSC PDs was characterized. Two visualizations are in Attachment #1 and Attachment #2.
This isn't meant to be a serious budget, mainly it was to force myself to write the code for generating this more easily in the future.
The measured RIN of the arm cavity transmission when the PRFPMI is locked is ~10x in RMS relative to the single arm POX/POY lock. It is not yet clear to me where the excess is coming from.
Attachment #1 shows the comparison.
I looked at some DC signals for the buildup of the carrier and sideband fields in various places. The results are shown in Attachments #1 and #2.
I agree, I think the PRC excess angular motion, PIT in particular, is a dominant contributor to the RIN. Attachments #1-#3 support this hypothesis. In these plots, "XARM" should really read "COMM" and "YARM" should really read "DIFF", because the error signals from the two end QPDs are mixed to generate the PIT and YAW error signals for these ASC servos - this is some channel renaming that will have to be done on the ASC model. The fact that the scatter plot between these DoFs has some ellipticity probably means the basis transformation isn't exactly right, because if they were truly orthogonal, we would expect them to be uncorrelated?
I guess what this means is that the stability of the lock could be improved by turning on some POP QPD based feedback control, I'll give it a shot.
- PRC TT misalignment (~3Hz)
Don't can you check the correlation between the POP QPD and the arm RIN
There were many locklosses from the point where the arm powers were somewhat stabilized. Attachments #1 and #2 show two individual locklosses. I think what is happening here is that the BS seismometer X channel is glitching, and creating a transient in the angular feedforward filter that blows the lock. The POP QPD based feedback loop cannot suppress this transient, apparently. For now, I get around this problem by boosting the POP QPD feedback loop a bit, and then turning the feedforward filters off. The fact that the other seismometer channels don't report any transient makes me think the problem is either with the seismometer itself, or the readout electronics. The seismometer masses were recently recentered, so I'm leaning towards the latter.
I didn't explicitly check the data, but I am reasonably certain the same effect is responsible for many PRMI locklosses even with the arms held off resonance (though the tolerance to excursions there is higher). Pity really, the feedforward filters were a big help in the lock acquisition...
The CARM loop now has a UGF of ~12 kHz with a phase margin of ~60 degrees. These values of conventional stability indicators are good. The CARM optical gain that best fits the measurements is 9 MW/m.
I've been working on understanding the loop better, here are the notes.
Attachment #1 shows a block diagram of the loop topology.
Attachment #2 shows the OLGs of the two actuation paths.
Attachment #3 and #4 show the model, overlaid with measurements of the loop OLG and crossover TF respectively.
Attachment #5 shows the evolution of the CARM OLG at a few points in the lock acquisition sequence.
Now the I have a model I believe, I need to think about whether there is any benefit to changing some of these loop shapes. I've already raised the possibility of changing the shape of the boosts on the CM board, with which we could get a bit more suppression in the 100 Hz - 1kHz region (noise budget of laser frequency noise --> DARM required to see if this is necessary).
Attachments #1 and Attachments #2 are in the style of elog15356, but with data from a more recent lock. It'd be nice to calibrate the ASDC channel (and in general all channels) into power units, so we have an estimate of how much sideband power we expect, and the rest can be attributed to carrier leakage to ASDC.
On the basis of Attachments #1, the PRG is ~19, and at times, the arm transmission goes even higher. I'd say we are now in the regime where the uncertainty of the losses in the recycling cavity - maybe beamsplitter clipping? is important in using this info to try and constrain the arm cavity losses. I'm also not sure what to make of the asymmetry between TRX and TRY. Allegedly, the Y arm is supposed to be lossier.
This is very interesting. Do you have the ASDC vs PRG (~ TRXor TRY) plot? That gives you insight on what is the cause of the low recycling gain.
I implemented an ASC servo for the PRC, with the POP QPD as a sensor, and the PRM as the actuator. This has improved the stability of the lock (longer locks are possible), and also reduced the RIN of the arm transmission.
Attachment #1 shows the in-loop error signal suppression, and some out-of-loop monitors (POP22 and POPDC).
Attachment #2 compares the arm transmission RIN with the PRFPMI locked, with and without PRC ASC. The 3 Hz bump is definitely squished, but I think we can do better yet.
Attachments #3-5 are in the style of elog15361. No Oplev signals yet, I'll add them soon.
I guess what this means is that the stability of the lock could be improved by turning on some POP QPD based feedback control, I'll give it a shot
The 40m summary pages have been revived. I've not had to make any manual interventions in the last 5 days, so things seem somewhat stable, but I'm sure there will need to be multiple tweaks made. The primary use of the pages right now are for vacuum, seismic and PSL diagnostics.
For these initial attempts, I was just trying to transition MICH to REFL55Q. I agree, the PRCL situation may be more complicated.
Which 1f signals are you going to use? PRCL has sign flipping at the carrier critical coupling. So if the IFO is close to that condition, 1f PRCL suffers from the sign flipping or large gain variation.
I am inclined to believe that the arm cavity losses are such that the IFO is overcoupled. Some calculations, validated with Finesse modeling also suggest that there isn't a sign change for the CARM error signal when the IFO goes from being undercoupled to overcoupled, but I may have made a mistake here?
Thoughts from others?
This EQ seems to have knocked all suspensions out. ITMX was stuck. It is now released, and the IMC is locked again. It looks like there are some serious aftershocks going on so let's keep an eye on things.
I found that there is an issue with the MC1 slow bias voltages.
I usually offload the DC part of the output voltage from the WFS servos to the slow bias voltage sliders, so as to preserve maximum actuation range from the fast system. However, today, I found that this servo wasn't working well at all. So I dug a little deeper. Looking at the EPICS database records:
Around 5pm local time, the three vertex FEs crashed. AFAIK, no one was in the lab or working on anything CDS related, so this is worrying.
There appears to have been some sort of vacuum failure.
ldas-pcdev1 was down, so the summary pages weren't being generated. I have now switched over to ldas-pcdev6. I suspect some forepump failure, will check up later today unless someone else wants to take care of this.
There was no interlock action, and I don't check the vacuum status every half hour, so there was a period of time last night there was high circulating power in the arm cavities when the main volume pressure was higher than nominal. I have now closed the PSL shutter until the issue is resolved.
It looks like the main vacuum interlock was tripped due to a serial communication error from the TP2 controller. With Rana/Koji's permission, I will open V1 and expose the main volume to TP1 again (#2 in last section).
Recommended course of action:
The pumpspool UPS has its "Replace Battery" indicator light on. Might be a good chance to change the UPS, but at the very least, we should put in fresh batteries (last replacement was in Aug 2017).
I'll say this again - the pumpspool area is noisier than I remember it being, I think one / both of the roughing pumps backing TP2 / TP3 need tip-seal replacements.
BTW, EX is 5C hotter than EY, by virtue of the tarnac outside? In fact, judging by Steve's thermometers, EX reports a 12C swing in 24 hours between 30 C and 18 C (so almost no temperature control) while EY reports a 5C swing between 20 and 25 C. This is borne out by the ETM Oplev data I think...