so far it has run through the weekend with no problems (except that there are huge log files as usual).
I have started to set up monit to run on megatron to watch this process. In principle this would send us alerts when things break and also give a web interface to watch monit. I'm not sure how to do web port forwarding between megatron and nodus, so for now its just on the terminal. e.g.:
monit>sudo monit status
Monit 5.25.1 uptime: 4m
monitoring status Monitored
monitoring mode active
on reboot start
load average [0.15] [0.22] [0.25]
cpu 0.6%us 1.0%sy 0.2%wa
memory usage 1001.4 MB [25.0%]
swap usage 107.2 MB [1.9%]
uptime 40d 17h 55m
boot time Tue, 14 Apr 2020 17:47:49
data collected Mon, 25 May 2020 11:43:03
monitoring status Monitored
monitoring mode active
on reboot start
parent pid 1
effective uid 4666
uptime 3d 1h 22m
cpu total 0.0%
memory 19.4% [776.1 MB]
memory total 19.4% [776.1 MB]
security attribute unconfined
disk read 0 B/s [2.3 GB total]
disk write 0 B/s [17.9 MB total]
data collected Mon, 25 May 2020 11:43:03
A big factor in how much IFO locking activities can take place is how cooperative the IMC is.
Since the c1psl upgrade, the IMC duty cycle has definitely deteriorated. I took a measurement of the dark noise at the IMC error point with 1 Hz FFT binwidth, with all electrical connections to the IMC servo board except the Acromag and Eurocrate power disconnected. I was horrified at the prominence of 60 Hz harmonics - see Attachment #1. In the past, this kind of feature has been indicative of some error in the measurement technique - but I confirmed that the lines remain even if I unplug the GPIB box, and all combinations of floating/grounded inputs that I tried. We know for sure that there is some excess noise imprinted on the laser light post upgrade. While these lines almost certainly are not responsible for the PCdrive RMS going bonkers, surely this kind of electrical situation isn't good?
Attachment #2 shows the same information translated to frequency noise units, taking into account the complementary sensitivity function, L/(1+L) - the sum contribution of the 60 Hz peaks to the RMS is ~11.5% of the total over the entire band (c.f. 1.7 % that is expected if the noise at multiples of 60 Hz was approximately equal to the surrounding noise levels). Moreover, the measured RMS is 55 times higher than a LISO model.
How can this be fixed?
Provided the IMC is cooperative, the input pointing isn't drifting, and the RF offsets aren't jumping around too much, the locking sequence is now pretty robust.
Most of the analysis uses data between the GPS times 1274418176 and 1274419654 that are recorded to frames.
Here, I provide some details of the sequence. Obviously, I am presenting one of the quickest transitions to the fully locked state, I don't claim that every attempt is so smooth. But it is pretty cool that the whole thing can be done in ~3 minutes.
See Attachment #1 for the labels.
This particular lock held for ~20 minutes so I could run some loop characterization measurements etc.
I am struggling to explain:
In order to estimate the free-running DARM displacement noise, I measured the DARM OLTF using the usual IN1/IN2 prescription. The measured data was then used to fit some model paramters for a loop model that can be used over a larger frequency range.
In summary, the UGF is ~150 Hz and phase margin is ~30 deg. This loop would probably benefit from some low-pass filter being turned on.
I am able to realize ~8 kHz UGF with ~60 degrees of phase margin on the CARM loop OLTF (combination of analog and digital signal paths).
The response of the PRFPMI length degrees of freedom as measured in the LSC PDs was characterized. Two visualizations are in Attachment #1 and Attachment #2.
This isn't meant to be a serious budget, mainly it was to force myself to write the code for generating this more easily in the future.
Replaced empty N2 tank, left tank at ~2000 psi, right tank ~2600 psi.
The measured RIN of the arm cavity transmission when the PRFPMI is locked is ~10x in RMS relative to the single arm POX/POY lock. It is not yet clear to me where the excess is coming from.
Attachment #1 shows the comparison.
I looked at some DC signals for the buildup of the carrier and sideband fields in various places. The results are shown in Attachments #1 and #2.
Please see the attached doc.
I think the conclusion is that if the AS1 RoC error is not significantly more than 1%, then with some adjustment of the AS1-AS3 distance (~ 1 cm), we could find a solution that simultaneously makes the AS path mode-matching better than 99% for the t- and s-planes.
The requirement of the LO path is less strict and the current plan using LO1-LO2 actuation should work.
This is very interesting. Do you have the ASDC vs PRG (~ TRXor TRY) plot? That gives you insight on what is the cause of the low recycling gain.
My speculation for the worse RIN is:
- Unoptimized alignment -> Larger linear coupling of the RIN with the misalignment
- PRC TT misalignment (~3Hz)
Don't can you check the correlation between the POP QPD and the arm RIN?
I see. At the 40m, we have the direct transition from ALS to RF. But it's hard to compare them as the storage time is very different.
I agree, I think the PRC excess angular motion, PIT in particular, is a dominant contributor to the RIN. Attachments #1-#3 support this hypothesis. In these plots, "XARM" should really read "COMM" and "YARM" should really read "DIFF", because the error signals from the two end QPDs are mixed to generate the PIT and YAW error signals for these ASC servos - this is some channel renaming that will have to be done on the ASC model. The fact that the scatter plot between these DoFs has some ellipticity probably means the basis transformation isn't exactly right, because if they were truly orthogonal, we would expect them to be uncorrelated?
I guess what this means is that the stability of the lock could be improved by turning on some POP QPD based feedback control, I'll give it a shot.
- PRC TT misalignment (~3Hz)
Don't can you check the correlation between the POP QPD and the arm RIN
how bout corner plot with power signals and oplevs? I think that would show not just linear couplings (like your coherence), but also quadratic couplings (chesire cat grin)
We computed the required actuation range for the telescope design in elog:15357. The result is summarized in the table below. Here we assume we misalign an IFO mirror by 1 urad, and then compute how many urad do we need to move the (AS1, AS4) or (LO1, LO2) mirrors to simultaneously correct for the two gouy phases.
The most demanding ifo mirrors are the ETMs and the BS, for every 1 urad misalignment the telescope needs to move 10-15 urad to correct for that. However, it is unlikely for those mirrors to move more 100 nrad for a locked ifo with ASC engaged. Thus a few urad actuation should be sufficient. For the recycling mirrors, every 1 urad misalignment also requires ~ 1 urad actuation.
As a result, if we could afford 10 urad actuation range for each telescope suspension, then the gouy phase separations we have should be fine.
We looked at the oplev spectra from gps 1274418500 for 512 sec. This should be a period when the ifo was locked in the PRFPMI state according to elog:15348. We just focused on the yaw data for now. Please see the attached plots. The solid traces are for the ASD, and the dotted ones are the cumulative rms. The total rms for each mirror is also shown in the legend.
I am now confused... The ITMs looked somewhat reasonable in that at least the < 1 Hz motion was suppressed. The total rms is ~ 0.1 urad, which was what I would expect naively (~ x100 times worse than aLIGO).
There seems to be no low-freq suppression on the ETMs though... Is there no arm ASC at the moment???
There were many locklosses from the point where the arm powers were somewhat stabilized. Attachments #1 and #2 show two individual locklosses. I think what is happening here is that the BS seismometer X channel is glitching, and creating a transient in the angular feedforward filter that blows the lock. The POP QPD based feedback loop cannot suppress this transient, apparently. For now, I get around this problem by boosting the POP QPD feedback loop a bit, and then turning the feedforward filters off. The fact that the other seismometer channels don't report any transient makes me think the problem is either with the seismometer itself, or the readout electronics. The seismometer masses were recently recentered, so I'm leaning towards the latter.
I didn't explicitly check the data, but I am reasonably certain the same effect is responsible for many PRMI locklosses even with the arms held off resonance (though the tolerance to excursions there is higher). Pity really, the feedforward filters were a big help in the lock acquisition...
The CARM loop now has a UGF of ~12 kHz with a phase margin of ~60 degrees. These values of conventional stability indicators are good. The CARM optical gain that best fits the measurements is 9 MW/m.
I've been working on understanding the loop better, here are the notes.
Attachment #1 shows a block diagram of the loop topology.
Attachment #2 shows the OLGs of the two actuation paths.
Attachment #3 and #4 show the model, overlaid with measurements of the loop OLG and crossover TF respectively.
Attachment #5 shows the evolution of the CARM OLG at a few points in the lock acquisition sequence.
Now the I have a model I believe, I need to think about whether there is any benefit to changing some of these loop shapes. I've already raised the possibility of changing the shape of the boosts on the CM board, with which we could get a bit more suppression in the 100 Hz - 1kHz region (noise budget of laser frequency noise --> DARM required to see if this is necessary).
Attachments #1 and Attachments #2 are in the style of elog15356, but with data from a more recent lock. It'd be nice to calibrate the ASDC channel (and in general all channels) into power units, so we have an estimate of how much sideband power we expect, and the rest can be attributed to carrier leakage to ASDC.
On the basis of Attachments #1, the PRG is ~19, and at times, the arm transmission goes even higher. I'd say we are now in the regime where the uncertainty of the losses in the recycling cavity - maybe beamsplitter clipping? is important in using this info to try and constrain the arm cavity losses. I'm also not sure what to make of the asymmetry between TRX and TRY. Allegedly, the Y arm is supposed to be lossier.
I implemented an ASC servo for the PRC, with the POP QPD as a sensor, and the PRM as the actuator. This has improved the stability of the lock (longer locks are possible), and also reduced the RIN of the arm transmission.
Attachment #1 shows the in-loop error signal suppression, and some out-of-loop monitors (POP22 and POPDC).
Attachment #2 compares the arm transmission RIN with the PRFPMI locked, with and without PRC ASC. The 3 Hz bump is definitely squished, but I think we can do better yet.
Attachments #3-5 are in the style of elog15361. No Oplev signals yet, I'll add them soon.
I guess what this means is that the stability of the lock could be improved by turning on some POP QPD based feedback control, I'll give it a shot
Which 1f signals are you going to use? PRCL has sign flipping at the carrier critical coupling. So if the IFO is close to that condition, 1f PRCL suffers from the sign flipping or large gain variation.
The 40m summary pages have been revived. I've not had to make any manual interventions in the last 5 days, so things seem somewhat stable, but I'm sure there will need to be multiple tweaks made. The primary use of the pages right now are for vacuum, seismic and PSL diagnostics.
For these initial attempts, I was just trying to transition MICH to REFL55Q. I agree, the PRCL situation may be more complicated.
I am inclined to believe that the arm cavity losses are such that the IFO is overcoupled. Some calculations, validated with Finesse modeling also suggest that there isn't a sign change for the CARM error signal when the IFO goes from being undercoupled to overcoupled, but I may have made a mistake here?
Thoughts from others?
This EQ seems to have knocked all suspensions out. ITMX was stuck. It is now released, and the IMC is locked again. It looks like there are some serious aftershocks going on so let's keep an eye on things.
I will be at the 40m, in the Clean and bake lab today from ~9am to ~3pm.
I found that there is an issue with the MC1 slow bias voltages.
I usually offload the DC part of the output voltage from the WFS servos to the slow bias voltage sliders, so as to preserve maximum actuation range from the fast system. However, today, I found that this servo wasn't working well at all. So I dug a little deeper. Looking at the EPICS database records:
We can limit the EPICS values giving some parameters to the channels. cf https://epics.anl.gov/tech-talk/2012/msg00147.php
But this does not solve the MC1 issue. Only we can do right now is to make the output resister half, for example.
For the initial phase of BHD testing, we recently discussed whether the mode-matching telescopes could be built with 100% stock optics. This would allow the optical system to be assembled more quickly and cheaply at a stage when having ultra-low loss and scattering is less important. I've looked into this possibility and conclude that, yes, we do have a good stock optics option. It in fact achieves comprable performance to our optimized custom-curvature design [ELOG 15357]. I think it is certainly sufficient for the initial phase of BHD testing.
It turns out our usual suppliers (e.g., CVI, Edmunds) do not have enough stock options to meet our requirements. This is for two reasons:
However I found that Lambda Research Optics carries 1" and 2" super-polished mirror blanks in an impressive variety of stock curvatures. Even more, they're polished to comprable tolerances as I had specificied for the custom low-scatter optics [DCC E2000296]: irregularity < λ/10 PV, 10-5 scratch-dig, ROC tolerance ±0.5%. They can be coated in-house for 1064 nm to our specifications.
From modeling Lambda's stock curvature options, I find it still possible to achieve mode-matching of 99.9% for the AS beam and 98.6% for the LO beam, if the optics are allowed to move ±1" from their current positions. The sensitivity to the optic positions is slightly increased compared to the custom-curvature design (but by < 1.5x). I have not run the stock designs through Hang's full MC corner-plot analysis which also perturbs the ROCs [ELOG 15339]. However for the early BHD testing, the sensitivity is secondary to the goal of having a quick, cheap implementation.
The following tables show the best telescope designs using stock curvature options. It assumes the optics are free to move ±1" from their current positions. For comparison, the values from the custom-curvature design are also given in parentheses.
The AS relay path is shown in Attachment 1:
The LO relay path is shown in Attachment 2:
I've created a new tab in the BHD procurement spreadsheet ("Stock MM Optics Option") listing the part numbers for the above telescope designs, as well as their fabrication tolerances. The total cost is $2.8k + the cost of the coatings (I'm awaiting a quote from Lambda for the coatings). The good news is that all the curved substrates will receive the same HR/AR coatings, so I believe they can all be done in a single coating run.
We consider the astigmatism effects of the stock options. The conclusions are:
1. For the AS path, the stock should work fine for the phase-one of BHD, if we could tolerate a few percent MM loss. The window for length adjustment to achieve >99% MM for both s and t is only 1 mm for 1% RoC error (compared to ~ 1 cm in the customized case).
2. The LO path seemed tricky. As LO3 & LO4 are both significantly curved (RoC<=0.5 m), the non-zero angle of incidence makes the astigmatism quite sever. For the t-plane the nominal MM can be 0.98, yet for the s-plane, the nominal MM is only 0.72. We could move things around to achieve a MM ~ 0.85, which is probably fine for the phase-one implementation but not long term.
Attachments 1-3 are for the AS path; 4-6 are for the LO path.
1 & 4. Marginalized MM distribution for the AS/LO paths. Here we assumed 5 mm positional error and 1% fractional RoC error. Due to the astigmatism, the nominal s-plane MM is only 0.72 for the LO path.
2 & 5. Scattering plots for the AS/LO paths. We color coded the points as the following: pink: MM>0.99; olive: 0.98<MM<=0.99; grey: MM<=0.98. For the AS path, MM is mostly sensitive to the AS1 RoC and can be adjusted by changing AS1-AS3 distance. For the LO path, the LO3 RoC and LO3-LO4 distance are most critical for the MM.
3 & 6. Assuming +- 1% AS1 (LO3) fractional RoC error, how much can we compensate for it using AS1-AS3 (LO3-LO4) distance. For the AS path, there exists a ~ 1 mm window where the MM for s and t can simultaneously > 99%. For the LO path, the best we can do is to make s and t both ~ 85%.
Can you describe the mode matching in terms of the total MM? Is MM_total = sqrt(MM_vert * MM_horiz)?
MM_total = (MM_vert + MM_horiz) / 2.
The large astigmatic MM loss in the LO case is mainly due to the strong LO4 curvature (R=0.15m) with a 10 deg AOI. I looked again at whether LO1 could be increased from R=5m to the next higher stock value of 7.5m, as this would allow weaker curvatures on LO3 and LO4. However, no, that is not possible---it reduces the LO1-LO2 Gouy phase separation to only 18 deg.
There is, however, a good stock-curvature option if we want to reconsider actuating LO4 instead of LO2 (attachment 1). It achieves 99.2% MM with the OMCs, allowing positions to vary +/-1" from the current design. The LO1-LO4 Gouy phase separation is 72 deg.
Alternatively, we could look at reducing the AOI on LO3 and LO4 (keeping LO1-LO2 actuation).
Around 5pm local time, the three vertex FEs crashed. AFAIK, no one was in the lab or working on anything CDS related, so this is worrying.
Hmm? T1300364 suggests MM_total = Sqrt(MM_Vert * MM_Horiz)
I will be in the Clean and Bake lab today from 9:30am to 4pm.
I don't think we ever discussed why the angular RMS of the ETMs is so much higher than the ITMs. Maybe that's a separate matter because, even assuming the worst case, the actuation range requirement is
(0.82 μrad RMS) x (15 μrad/μrad) x (10 safety factor) = 0.12 mrad
which is still only order 1% of the pitch/yaw pointing range of the Small Optic Suspensions, according to P1600178 (sec. IV. A). Can we check this requirement off the list?
Using the updated AOI's for the LO path: (4.8, 47.9, 2.9, 4.5) deg for (LO1, LO2, LO3, LO4), we obtain the following results.
First two plots are scattering plots for the t and s planes, respectively. Note that here we have changed to 0.5% fractional RoC error and 3 mm positional error. We have also changed the meaning of the colors: pink:MM>0.98; olive 0.95<MM<=0.98, and grey MM<=0.95. It seems that both planes would benefit statistically if we make the LO3-LO4 distance longer by a few mm.
We also consider how much we could compensate for the MM error in the last plot. We have a few mm window to make both planes better than 0.95.
I will be in the Clean and Bake lab from 10am to 4pm today. I will also replace an empty N2 cylinder.
After further astigmatism/tolerance analysis [ELOG 15380, 15387] our conclusion is that the stock-optic telescope designs [ELOG 15379] are sufficient for the first round of BHD testing. However, for the final BHD hardware we should still plan to procure the custom-curvature optics [DCC E2000296]. The optimized custom-curvature designs are much more error-tolerant and have high probability of achieving < 2% mode-matching loss. The stock-curvature designs can only guarantee about 95% mode-matching.
Below are the final distances between optics in the relay paths. The base set of distances is taken from the 2020-05-21 layout. To minimize the changes required to the CAD model, I was able to achieve near-maximum mode-matching by moving only one optic in each relay path. In the AS path, AS3 moves inwards (towards the BHDBS) by 1.06 cm. In the LO path, LO4 moves backwards (away from the BHDBS) by 3.90 cm.
I will be in the Clean and Bake lab today from 11am to 4pm.
There appears to have been some sort of vacuum failure.
ldas-pcdev1 was down, so the summary pages weren't being generated. I have now switched over to ldas-pcdev6. I suspect some forepump failure, will check up later today unless someone else wants to take care of this.
There was no interlock action, and I don't check the vacuum status every half hour, so there was a period of time last night there was high circulating power in the arm cavities when the main volume pressure was higher than nominal. I have now closed the PSL shutter until the issue is resolved.
It looks like the main vacuum interlock was tripped due to a serial communication error from the TP2 controller. With Rana/Koji's permission, I will open V1 and expose the main volume to TP1 again (#2 in last section).
Recommended course of action:
The pumpspool UPS has its "Replace Battery" indicator light on. Might be a good chance to change the UPS, but at the very least, we should put in fresh batteries (last replacement was in Aug 2017).
I'll say this again - the pumpspool area is noisier than I remember it being, I think one / both of the roughing pumps backing TP2 / TP3 need tip-seal replacements.
BTW, EX is 5C hotter than EY, by virtue of the tarnac outside? In fact, judging by Steve's thermometers, EX reports a 12C swing in 24 hours between 30 C and 18 C (so almost no temperature control) while EY reports a 5C swing between 20 and 25 C. This is borne out by the ETM Oplev data I think...
1. I agree that it's likely that it was the temp signal glitch.
Recom #2: I approve to reopen the valves to pump down the main volume. As long as there is no frequent glitch, we can just bring the vacuum back to normal with the current software setup.
2. Recom #1 is also reasonable. You can use simple logic like if we register 10 consecutive samples that exceed the threshold, we can activate the interlock. I feel we should still keep the temp interlock. Switching between pumping mode and the normal operation may cause unexpected omission of the interlocks when it is necessary.
3. We should purchase the UPS battery / replacement rotary TIP seal. Once they are in hand, we can stop the vacuum and execute the replacement. Can one person (who?) accomplish everything with some remote help?
4. The lab temp: you mean, 12degC swing with the AC on!?
I will be in the Clean and Bake lab today from 12pm to 4pm.
Jon and Koji remotely supported Jordan's resetting the TP2 controller.
From the operator's console in front of the vac rack:
Open a terminal window (click the LXTerminal icon on the desktop)
Type "control" + enter to open the vac controls screen
Toggle all the open valves closed (edit by KA: and manually close RV2 by rotating the gate valve handle )
Turn OFF TP2 by clicking the "Off' button. Make sure the status changes and the rotation speed falls to zero (you'll also hear the pump spinning down)
The other pumps (TP1, TP3) can be left running
Once TP2 has stopped spinning, go to the back of the rack and locate the ethernet cable running from the back of the TP2 controller to the IOLAN server (near the top of the rack). Disconnect and reconnect the cable at each end, verifying it is firmly locked in place.
From the front of the rack, power down the TP2 controller (I don't quite remember for the Agilent, but you might have to move the slider on the front from "Remote" to "Local" first)
Wait about 30 seconds, then power it back on. If you had to move the slider to shut it down, revert it back to the "Remote" position.
Go back to the controls screen on the console. If the pump came back up and is communicating serially again, its status will say something other than "NO COMM"
Turn TP2 back on. Verify that it spins up to its nominal speed (66 kRPM)
At this point you can reopen any valves you initially closed (any that were already closed before, leave closed)
TP2 was stopped and at this moment the glitches were gone. Jordan powercycled the TP2 controller and we brought up the TP2 back at the full speed.
However, the glitches came back as before. Obviously we can't go on from here, and we've decided to stop the recovery process here today.
- We left TP1/2/3 running while the valves including RV2 were closed.
- When Jordan is back in the lab next week, we'll try to use TP3 as the backing of TP1 so that we can resume the main volume pumping.
- Currently, TP3 does not have interlocking and that is a risk. Jon is going to implement it.
- Meanwhile, we will try to replace the controller of TP2. We are supposed to have this in the lab. Ask Chub about the location.
- Once we confirm the stability of the diagnostic signals for TP2, we will come back to the nominal pumping scheme.