1. I agree that it's likely that it was the temp signal glitch.
Recom #2: I approve to reopen the valves to pump down the main volume. As long as there is no frequent glitch, we can just bring the vacuum back to normal with the current software setup.

2. Recom #1 is also reasonable. You can use simple logic like if we register 10 consecutive samples that exceed the threshold, we can activate the interlock. I feel we should still keep the temp interlock. Switching between pumping mode and the normal operation may cause unexpected omission of the interlocks when it is necessary.

3. We should purchase the UPS battery / replacement rotary TIP seal. Once they are in hand, we can stop the vacuum and execute the replacement. Can one person (who?) accomplish everything with some remote help?

4. The lab temp: you mean, 12degC swing with the AC on!?

 

Summary:

The pumpspool UPS has its "Replace Battery" indicator light on. Might be a good chance to change the UPS, but at the very least, we should put in fresh batteries (last replacement was in Aug 2017).

I'll say this again - the pumpspool area is noisier than I remember it being, I think one / both of the roughing pumps backing TP2 / TP3 need tip-seal replacements.

BTW, EX is 5C hotter than EY, by virtue of the tarnac outside? In fact, judging by Steve's thermometers, EX reports a 12C swing in 24 hours between 30 C and 18 C (so almost no temperature control) while EY reports a 5C swing between 20 and 25 C. This is borne out by the ETM Oplev data I think...

Summary:

It looks like the main vacuum interlock was tripped due to a serial communication error from the TP2 controller. With Rana/Koji's permission, I will open V1 and expose the main volume to TP1 again (#2 in last section).

Details:

• The vacuum interlock log file at /opt/target/vac.log on c1vac suggests that the interlock was tripped because "TP2 is too warm".
• Looking back at the diagnostics channels, it looks like the TP2 temperature channel registered a rise in temperature of >30 C in <0.2 seconds, see Attachment #1 - seems highly unlikely, probably some kind of glitch in the serial communication? This particular pump is relatively new from Agilent (<2 years installed I think)
• The PSL shutter was automatically closed at ~1150 am today, see Attachment #2. There is some EPICS logic on c1psl (Acromag server) that checks if C1:Vac-P1a_pressure is greater than 3 mTorr (or greater than 500 Torr for in-air locking of the IMC), in which case it closes the shutter, so this seems consistent with expectations.

Recommended course of action:

1. Code in some averaging in the interlock code, so that the interlock isn't triggered on some unphysical glitch like this. As shown in Attachment #3, this has been happening for the past 24 hours (though not before, because the interlock wasn't tripped). Probably need the derivative of the temperature as well, and the derivative should be less than 5 C/s or something physical (in addition to the temperature being high) for the interlock to trip.
2. Re-open V1 to pump down the main volume to nominal pressure so that the interferometer locking activity can resume.
   • One option in the interim is to bypass the TP2 temperature interlock condition.
   • The pressure-based interlocks are probably sufficient to protect the main volume / pumps during the nominal operations - the temperature interlocks are mainly useful during the pumpdown where the TPs have a large load, and so we want to avoid over-stressing them.

There appears to have been some sort of vacuum failure.

ldas-pcdev1 was down, so the summary pages weren't being generated. I have now switched over to ldas-pcdev6. I suspect some forepump failure, will check up later today unless someone else wants to take care of this.

There was no interlock action, and I don't check the vacuum status every half hour, so there was a period of time last night there was high circulating power in the arm cavities when the main volume pressure was higher than nominal. I have now closed the PSL shutter until the issue is resolved.

I will be in the Clean and Bake lab today from 11am to 4pm.

After further astigmatism/tolerance analysis [ELOG 15380, 15387] our conclusion is that the stock-optic telescope designs [ELOG 15379] are sufficient for the first round of BHD testing. However, for the final BHD hardware we should still plan to procure the custom-curvature optics [DCC E2000296]. The optimized custom-curvature designs are much more error-tolerant and have high probability of achieving < 2% mode-matching loss. The stock-curvature designs can only guarantee about 95% mode-matching.

Below are the final distances between optics in the relay paths. The base set of distances is taken from the 2020-05-21 layout. To minimize the changes required to the CAD model, I was able to achieve near-maximum mode-matching by moving only one optic in each relay path. In the AS path, AS3 moves inwards (towards the BHDBS) by 1.06 cm. In the LO path, LO4 moves backwards (away from the BHDBS) by 3.90 cm.

AS Path

LO Path

I will be in the Clean and Bake lab from 10am to 4pm today. I will also replace an empty N2 cylinder.

Using the updated AOI's for the LO path: (4.8, 47.9, 2.9, 4.5) deg for (LO1, LO2, LO3, LO4), we obtain the following results. 

First two plots are scattering plots for the t and s planes, respectively. Note that here we have changed to 0.5% fractional RoC error and 3 mm positional error. We have also changed the meaning of the colors: pink:MM>0.98; olive 0.95<MM<=0.98, and grey MM<=0.95. It seems that both planes would benefit statistically if we make the LO3-LO4 distance longer by a few mm. 

We also consider how much we could compensate for the MM error in the last plot. We have a few mm window to make both planes better than 0.95. 

I don't think we ever discussed why the angular RMS of the ETMs is so much higher than the ITMs. Maybe that's a separate matter because, even assuming the worst case, the actuation range requirement is

(0.82 μrad RMS) x (15 μrad/μrad) x (10 safety factor) = 0.12 mrad

which is still only order 1% of the pitch/yaw pointing range of the Small Optic Suspensions, according to P1600178 (sec. IV. A). Can we check this requirement off the list?

I will be in the Clean and Bake lab today from 9:30am to 4pm.

Hmm? T1300364 suggests MM_total = Sqrt(MM_Vert * MM_Horiz)

Summary:

Around 5pm local time, the three vertex FEs crashed. AFAIK, no one was in the lab or working on anything CDS related, so this is worrying.

Details:

• Reboot script was used to bring all FEs back - only soft reboots were required.
• The IMC and arms can now be locked.
• I think combination of burt + SDF would have reverted all the settings as they should be, but if something appears off, it could be that some EPICS value didn't get reset correctly.

MM_total = (MM_vert + MM_horiz) / 2. 

The large astigmatic MM loss in the LO case is mainly due to the strong LO4 curvature (R=0.15m) with a 10 deg AOI. I looked again at whether LO1 could be increased from R=5m to the next higher stock value of 7.5m, as this would allow weaker curvatures on LO3 and LO4. However, no, that is not possible---it reduces the LO1-LO2 Gouy phase separation to only 18 deg.

There is, however, a good stock-curvature option if we want to reconsider actuating LO4 instead of LO2 (attachment 1). It achieves 99.2% MM with the OMCs, allowing positions to vary +/-1" from the current design. The LO1-LO4 Gouy phase separation is 72 deg.

Alternatively, we could look at reducing the AOI on LO3 and LO4 (keeping LO1-LO2 actuation).

Can you describe the mode matching  in terms of the total MM? Is MM_total = sqrt(MM_vert * MM_horiz)?

We consider the astigmatism effects of the stock options. The conclusions are:

1. For the AS path, the stock should work fine for the phase-one of BHD, if we could tolerate a few percent MM loss. The window for length adjustment to achieve >99% MM for both s and t is only 1 mm for 1% RoC error (compared to ~ 1 cm in the customized case). 

2. The LO path seemed tricky. As LO3 & LO4 are both significantly curved (RoC<=0.5 m), the non-zero angle of incidence makes the astigmatism quite sever. For the t-plane the nominal MM can be 0.98, yet for the s-plane, the nominal MM is only 0.72. We could move things around to achieve a MM ~ 0.85, which is probably fine for the phase-one implementation but not long term. 

Details:

Attachments 1-3 are for the AS path; 4-6 are for the LO path. 

1 & 4. Marginalized MM distribution for the AS/LO paths. Here we assumed 5 mm positional error and 1% fractional RoC error. Due to the astigmatism, the nominal s-plane MM is only 0.72 for the LO path. 

2 & 5. Scattering plots for the AS/LO paths. We color coded the points as the following: pink: MM>0.99; olive: 0.98<MM<=0.99; grey: MM<=0.98. For the AS path, MM is mostly sensitive to the AS1 RoC and can be adjusted by changing AS1-AS3 distance. For the LO path, the LO3 RoC and LO3-LO4 distance are most critical for the MM. 

3 & 6. Assuming +- 1% AS1 (LO3) fractional RoC error, how much can we compensate for it using AS1-AS3 (LO3-LO4) distance. For the AS path, there exists a ~ 1 mm window where the MM for s and t can simultaneously > 99%. For the LO path, the best we can do is to make s and t both ~ 85%. 

Summary

For the initial phase of BHD testing, we recently discussed whether the mode-matching telescopes could be built with 100% stock optics. This would allow the optical system to be assembled more quickly and cheaply at a stage when having ultra-low loss and scattering is less important. I've looked into this possibility and conclude that, yes, we do have a good stock optics option. It in fact achieves comprable performance to our optimized custom-curvature design [ELOG 15357]. I think it is certainly sufficient for the initial phase of BHD testing.

Vendor

It turns out our usual suppliers (e.g., CVI, Edmunds) do not have enough stock options to meet our requirements. This is for two reasons:

• For sufficient LO1-LO2 (AS1-AS4) Gouy phase separation, we require a very particular ROC range for LO1 (AS1) of 5-6 m (2-3 m).
• We also require a 2" diameter for the suspended optics, which is a larger size than most vendors stock for curved reflectors (for example, CVI has no stock 2" options).

However I found that Lambda Research Optics carries 1" and 2" super-polished mirror blanks in an impressive variety of stock curvatures. Even more, they're polished to comprable tolerances as I had specificied for the custom low-scatter optics [DCC E2000296]: irregularity < λ/10 PV, 10-5 scratch-dig, ROC tolerance ±0.5%. They can be coated in-house for 1064 nm to our specifications.

From modeling Lambda's stock curvature options, I find it still possible to achieve mode-matching of 99.9% for the AS beam and 98.6% for the LO beam, if the optics are allowed to move ±1" from their current positions. The sensitivity to the optic positions is slightly increased compared to the custom-curvature design (but by < 1.5x). I have not run the stock designs through Hang's full MC corner-plot analysis which also perturbs the ROCs [ELOG 15339]. However for the early BHD testing, the sensitivity is secondary to the goal of having a quick, cheap implementation.

Stock-Part Telescope Designs

The following tables show the best telescope designs using stock curvature options. It assumes the optics are free to move ±1" from their current positions. For comparison, the values from the custom-curvature design are also given in parentheses.

AS Path

The AS relay path is shown in Attachment 1:

• AS1-AS4 Gouy phase separation: 71°
• Mode-matching to OMC: 99.9%

LO Path

The LO relay path is shown in Attachment 2:

• LO1-LO2 Gouy phase separation: 67°
• Mode-matching to OMC: 98.6%

Ordering Information

I've created a new tab in the BHD procurement spreadsheet ("Stock MM Optics Option") listing the part numbers for the above telescope designs, as well as their fabrication tolerances. The total cost is $2.8k + the cost of the coatings (I'm awaiting a quote from Lambda for the coatings). The good news is that all the curved substrates will receive the same HR/AR coatings, so I believe they can all be done in a single coating run.

I will be at the 40m, in the Clean and bake lab today from ~9am to ~3pm.

We can limit the EPICS values giving some parameters to the channels. cf https://epics.anl.gov/tech-talk/2012/msg00147.php

But this does not solve the MC1 issue. Only we can do right now is to make the output resister half, for example.

Summary:

I found that there is an issue with the MC1 slow bias voltages. 

Details:

I usually offload the DC part of the output voltage from the WFS servos to the slow bias voltage sliders, so as to preserve maximum actuation range from the fast system. However, today, I found that this servo wasn't working well at all. So I dug a little deeper. Looking at the EPICS database records:

• The user-facing channels are "PIT" and "YAW" bias voltages.
• These are converted to voltages to be sent to individual coils by some calc channels in the EPICS database record. So, for example, the voltage to be sent to the "UL" coil (Upper Left, as viewed from the AR side of the optic), is A+B, where A is the "PIT" voltage and B is the "YAW" voltage. Similar combinations of A and B are used for the other 3 face coils.
• The problem is obvious - if either A or B > 5V, then the requested voltage to be sent to the UL coil is > 10 V, while the Acromag DACs can put out a maximum of 10 V. 
• As it happens, with the IFO currently aligned, MC1 is the only optic which faces this problem. 
• Why has this not been an issue before? In fact, looking at some old data, the "PIT" and "YAW" bias voltages to MC1 were both ~1-2 V in 2018. But I confirmed that something in the region of ~5 V is required from each of the "PIT" and "YAW" channels to bring the MCREFL spot back to the center of the camera, so something has changed the DC alignment of MC1, maybe an earthquake or something? Anyway, with these settings, 2/4 coils are basically saturated, and so we can only move the optic diagonally. 😢 
• Other coils that have  requested output voltages > 5V (so more than half the range of the DAC) include MC2 LL (5.2V), and ETMX LL and LR (5.5 and 5.8 V respectively).
• Either a factor of 0.5 should be included in all the EPICS database records, or else, we should make the "PIT" and "YAW" sliders range only from -5 to +5 V, so that this kind of misleading info isn't wasting time.

I will be at the 40m, in the Clean and bake lab today from ~9am to ~3pm.

GariLynn worked on the measurement of E1800089 mirrros.

The result of the data analysis, as well as the data and the codes, have been summarized here:
https://nodus.ligo.caltech.edu:30889/40m_phasemap/#E1800089
 

This EQ seems to have knocked all suspensions out. ITMX was stuck. It is now released, and the IMC is locked again. It looks like there are some serious aftershocks going on so let's keep an eye on things.

Summary:

I am inclined to believe that the arm cavity losses are such that the IFO is overcoupled. Some calculations, validated with Finesse modeling also suggest that there isn't a sign change for the CARM error signal when the IFO goes from being undercoupled to overcoupled, but I may have made a mistake here?

Details:

• We’d like to gain some insight into whether the interferometer is undercoupled, critically coupled, or overcoupled. Factors that determine which of these is true include:
  • Arm cavity losses
  • Recycling cavity losses
• The proxy by which we determine the recycling gain is usually the arm cavity transmission. Assuming T_PRM = 5.637 % according to the wiki, and assuming the arm cavity transmission is normalized to 1 when locked in the POX/POY state, we can say that the PRG is given by G_PRC = TRX × T_PRM, assuming that the (i) the RF sideband fields are perfectly rejected by the arm cavities and (ii) mode-matching efficiency between the input beam and the arm mode is the same as that between the input beam and the CARM mode.
• Apart from this, the other measurement we have available to us is the buildup of the sideband fields, namely POP22 and POP110. We can compare the values in the PRMI lock vs the PRFPMI to make some inference.
• I started off with an analytic calculation of the reflectivity of the compound arm cavity mirror.
  • Attachment #1 suggests we will have an over-coupled IFO for arm cavity losses below ~200 ppm, which is a regime we are almost certainly in now.
• Then, I repeat the analysis for the coupled CARM cavity, with the end mirror as the compound arm mirror and the input mirror as the PRM.
  • I assume 2 % loss in the PRC.
  • Attachment #2 shows that while the carrier field goes through a sign change in amplitude reflectivity (as expected), the sideband fields dont.
  • Per equation 4.2 of Koji's thesis, the error signal for CARM depends on the (signed) IFO reflectivity, and the absolute value of the derivative of the arm cavity reflectivity for the carrier w.r.t. CARM phase.
  • So, we don't expect the REFL11 signal to show a sign change.
  • The situation is more complicated for PRCL in REFL11, because as explicitly evaluated in Eq 4.3 of Koji's thesis, there are two terms that contribute, and their relative magnitudes will dictate the overall sign. 
• For a Finesse validation, I use a simplified 3 mirror coupled cavity to approximate the PRFPMI. I also retained the RF sidebands for diagnostic purposes. The idea was to study these PRG proxies and what their expected behavior is.
  • Attachment #3 shows the PDH error signal in the (arbitrarily defined) REFL11 I quadrature. While the optical gain changes as a function of the arm cavity loss, the actual slope does not change sign. The fact that the zero crossing doesn't happen at exactly 0 CARM offset is because of higher order mode light at the REFL port (in my model, I tried to preserve the flipped folding mirror situation so the mode matching between the arm cavity and PRC in my model is ~96%).
  • In fact, this may explain why a CARM_B offset is required to do the ALS-->IR handoff - the ALS servo wants to keep the arm offset to zero, but at that point, the PDH error signal isn't zero, and so the two loops end up fighting each other?
  • Attachment #4 is a more detailed study of the recycling gain as a function of arm cavity loss, but now including losses in the recycling cavity.

Conclusions:

1. I think the arm cavity losses are in the 60-80 ppm round-trip region. I don't see how we can explain the arm cavity transmission of ~350 otherwise.
2. The fact that REFLDC decreases as the arm transmission increases is because the input beam is getting better matched to the CARM mode, and there is less junk carrier light. 

Thoughts from others?

For these initial attempts, I was just trying to transition MICH to REFL55Q. I agree, the PRCL situation may be more complicated.

Summary:

The 40m summary pages have been revived. I've not had to make any manual interventions in the last 5 days, so things seem somewhat stable, but I'm sure there will need to be multiple tweaks made. The primary use of the pages right now are for vacuum, seismic and PSL diagnostics.

Resources:

• Git repo
• Documentation
• Workflow

Caveats:

• Intermittent failures of cron jobs
  • ​The status page relies on the condor_q command executing successfully on the cluster end. I have seen this fail a few times, so the status page may say the pages are dead whereas they're actually still running.
  • Similarly, the rsync of the pages to nodus where they're hosted can sometimes fail.
  • Usually, these problems are fixed on the next iteration of the respective cron jobs, so check back in ~half hour.
• I haven't really looked into it in detail, but I think our existing C1:IFO-STATE word format is not compatible with what gwsumm wants - I think it expects single bits that are either 0 or 1 to indicate a particular state (e.g. MC locked, POX and POY locked etc). So if we want to take advantage of that infrastructure, we may need to add a few soft EPICS channels that take care of some logic checking (several such bits could also be and-ed together) and then assume either 0 or 1 value. Then we can have the nice duty cycle plots for the IMC (for example).
• I commented out the obsolete channels (e.g. PEM MIC channels). We can add them back later if we so desire.
• For some reason, the jobs that download the data are pretty memory-heavy: I have to request for machines on the cluster with >100 GB (yes 💯GB) ! of memory for the jobs to execute to completion. The frame-data certainly isn't so large, so I wonder what's going on here - is GWPy/GWsumm so heavy? The site summary pages run on a dedicated cluster, so probably the code isn't built for efficiency...
• Weather tab in PEM is still in there but none of those channels mean anything right now.
• The MEDM screenshot stuff is commented out for now too. This should be redone in a better way with some modern screen grab utilities, I'm sure there are plenty of python based ones.
• There seems to be a problem with the condor .dag lockfile / rescue file not being cleared correctly sometimes - I am looking into this.

Woo hoo!

Which 1f signals are you going to use? PRCL has sign flipping at the carrier critical coupling. So if the IFO is close to that condition, 1f PRCL suffers from the sign flipping or large gain variation.

Summary:

I implemented an ASC servo for the PRC, with the POP QPD as a sensor, and the PRM as the actuator. This has improved the stability of the lock (longer locks are possible), and also reduced the RIN of the arm transmission.

Details:

Attachment #1 shows the in-loop error signal suppression, and some out-of-loop monitors (POP22 and POPDC).

• To practise and get some workable servo settings, I locked the PRMI with carrier resonant (no ETMs).
• Then, I compare the beam motion witnessed by the POP QPD with and without the feedback loop enabled.
• I also look at the spectra of the POPDC and POP22 signals, as out-of-loop proxies, to get an estimate of how much noise is being injected out of band.
• In this toy study, both the in-loop and out of loop monitors show good performance.
• However, when repeating the same diagnostics with the PRFPMI locked, I note that while the in-loop suppression looks good, POPDC and POP22 report elevated noise, relative to the PRMI carrier case.
• I don't have a comparison to the PRFPMI locked with the feedback disabled, because of stability reasons. Plus, for the PRMI, the angular feedforward loops were engaged, but for the PRFPMI traces, they were disabled.
• Nevertheless, the arm RIN goes down by ~2.5 in RMS, so this is doing something good.

Attachment #2 compares the arm transmission RIN with the PRFPMI locked, with and without PRC ASC. The 3 Hz bump is definitely squished, but I think we can do better yet. 

Attachments #3-5 are in the style of elog15361. No Oplev signals yet, I'll add them soon.

Attachments #1 and Attachments #2 are in the style of elog15356, but with data from a more recent lock. It'd be nice to calibrate the ASDC channel (and in general all channels) into power units, so we have an estimate of how much sideband power we expect, and the rest can be attributed to carrier leakage to ASDC.

On the basis of Attachments #1, the PRG is ~19, and at times, the arm transmission goes even higher. I'd say we are now in the regime where the uncertainty of the losses in the recycling cavity - maybe beamsplitter clipping? is important in using this info to try and constrain the arm cavity losses. I'm also not sure what to make of the asymmetry between TRX and TRY. Allegedly, the Y arm is supposed to be lossier.

Summary:

The CARM loop now has a UGF of ~12 kHz with a phase margin of ~60 degrees. These values of conventional stability indicators are good. The CARM optical gain that best fits the measurements is 9 MW/m.

I've been working on understanding the loop better, here are the notes.

Details:

Attachment #1 shows a block diagram of the loop topology.

• The "crossover" measurement made at the digital CARM error point, and the OLG measurement at the CM board error point are shown.
• I've tried to include all the pieces in the loop, and yet, I had to introduce a fudge gain in the digital path to get the model to line up with the measurement (see below).

Attachment #2 shows the OLGs of the two actuation paths.

• Aforementioned fudge factor for the digital path is included.
• For the AO path, I assumed a value of the PDH discriminant at the IMC error point to be 13 kHz/V, per my earlier measurement. 
• I trawled the elog for the most up-to-date info about the IMC servo (elog9457, elog13696, elog15044) and CM board, to build up the model. 

Attachment #3 and #4 show the model, overlaid with measurements of the loop OLG and crossover TF respectively.

• No fitting is done yet - the next step would be to add the delay of the CDS system for the digital path, and the analog electronics for the AO path. Though these are likely only small corrections.
• For the crossover TF - I've divided out the digital filters in the CARM_B filter bank, because the injection is made downstream of it (see Attachment #1).
• There is reasonably good agreement between model and measurement.
• I think the biggest source of error is the assumed model for the IMC OLTF.

Attachment #5 shows the evolution of the CARM OLG at a few points in the lock acquisition sequence.

• "Before handoff" corresponds to the state where the primary control is still done by the ALS leg, but the REFL11 signal has begun to enter the picture via the CARM_B path.
• "IN2 ramped" corresponds to the state where the AO path gain (=IN2 gain on the IMC servo board) has been ramped up to its final value (+0 dB), but the overall loop gain (=IN1 gain on the CM board) is still low. So this is preparation for high bandwidth control. Typically, the arm powers will have stabilized in this state, but ALS control is still on.
• "Pre-boost" corresponds to an intermediate state - ALS control is off, but the low frequency boosts have not yet been enabled. I typically first engage some ASC to stabilize things somewhat, and then turn on the boosts.
• "Final" - self explanatory.

Next steps:

Now the I have a model I believe, I need to think about whether there is any benefit to changing some of these loop shapes. I've already raised the possibility of changing the shape of the boosts on the CM board, with which we could get a bit more suppression in the 100 Hz - 1kHz region (noise budget of laser frequency noise --> DARM required to see if this is necessary). 

There were many locklosses from the point where the arm powers were somewhat stabilized. Attachments #1 and #2 show two individual locklosses. I think what is happening here is that the BS seismometer X channel is glitching, and creating a transient in the angular feedforward filter that blows the lock. The POP QPD based feedback loop cannot suppress this transient, apparently. For now, I get around this problem by boosting the POP QPD feedback loop a bit, and then turning the feedforward filters off. The fact that the other seismometer channels don't report any transient makes me think the problem is either with the seismometer itself, or the readout electronics. The seismometer masses were recently recentered, so I'm leaning towards the latter.

I didn't explicitly check the data, but I am reasonably certain the same effect is responsible for many PRMI locklosses even with the arms held off resonance (though the tolerance to excursions there is higher). Pity really, the feedforward filters were a big help in the lock acquisition...

Highlights:

• With better ASC servos implemented, I think the lock stability has been improved. 
• Arm transmission of ~350 was stably maintained (PRG~20, overcoupled). It went as high as 410, so this is now very close to the highest (~425) I've ever managed to get.
• I was trying to get the vertex transitioned to 1f control but it remains out of reach for now.  The noise at ~100 Hz is dominated by MICH-->DARM coupling (as judged by coherence, I haven't yet done the broadband noise injection characterization). I figured I'd try the 1f transition before thinking about feedforward.
• The biggest problems remain flaky electronics (poor IMC duty cycle, jumping RF offsets, newly glitchy seismometer, ...)

Details:

• Faulty seismometer affecting the PRC angular feedforward [ELOG 15365].
• CARM loop modeling [ELOG 15366].
• PRC ASC performance [ELOG 15367].
• Field buildup update [ELOG 15368].

We computed the required actuation range for the telescope design in elog:15357. The result is summarized in the table below. Here we assume we misalign an IFO mirror by 1 urad, and then compute how many urad do we need to move the (AS1, AS4) or (LO1, LO2) mirrors to simultaneously correct for the two gouy phases. 

The most demanding ifo mirrors are the ETMs and the BS, for every 1 urad misalignment the telescope needs to move 10-15 urad to correct for that. However, it is unlikely for those mirrors to move more 100 nrad for a locked ifo with ASC engaged. Thus a few urad actuation should be sufficient. For the recycling mirrors, every 1 urad misalignment also requires ~ 1 urad actuation. 

As a result, if we could afford 10 urad actuation range for each telescope suspension, then the gouy phase separations we have should be fine. 

================================================================

Edits:

We looked at the oplev spectra from gps 1274418500 for 512 sec. This should be a period when the ifo was locked in the PRFPMI state according to elog:15348. We just focused on the yaw data for now. Please see the attached plots. The solid traces are for the ASD, and the dotted ones are the cumulative rms. The total rms for each mirror is also shown in the legend. 

I am now confused... The ITMs looked somewhat reasonable in that at least the < 1 Hz motion was suppressed. The total rms is ~ 0.1 urad, which was what I would expect naively (~ x100 times worse than aLIGO). 

There seems to be no low-freq suppression on the ETMs though... Is there no arm ASC at the moment???

how bout corner plot with power signals and oplevs? I think that would show not just linear couplings (like your coherence), but also quadratic couplings (chesire cat grin)

I agree, I think the PRC excess angular motion, PIT in particular, is a dominant contributor to the RIN. Attachments #1-#3 support this hypothesis. In these plots, "XARM" should really read "COMM" and "YARM" should really read "DIFF", because the error signals from the two end QPDs are mixed to generate the PIT and YAW error signals for these ASC servos - this is some channel renaming that will have to be done on the ASC model. The fact that the scatter plot between these DoFs has some ellipticity probably means the basis transformation isn't exactly right, because if they were truly orthogonal, we would expect them to be uncorrelated?

• In the corner plots, I am plotting the error signals of the ASC servos and the arm transmission. POP feedback is not engaged, but some feedback control to the ETMs based on the QPD signals is engaged.
• In the coherence plot, I show the coherence of the ASC error signals with the POP and TR QPD based error signals, under the same conditions. The coherence is high out to ~20 Hz.

I guess what this means is that the stability of the lock could be improved by turning on some POP QPD based feedback control, I'll give it a shot.

I see. At the 40m, we have the direct transition from ALS to RF. But it's hard to compare them as the storage time is very different.

My speculation for the worse RIN is:

- Unoptimized alignment -> Larger linear coupling of the RIN with the misalignment
- PRC TT misalignment (~3Hz)

Don't can you check the correlation between the POP QPD and the arm RIN?

This is very interesting. Do you have the ASDC vs PRG (~ TRXor TRY) plot? That gives you insight on what is the cause of the low recycling gain.

Please see the attached doc. 

I think the conclusion is that if the AS1 RoC error is not significantly more than 1%, then with some adjustment of the AS1-AS3 distance (~ 1 cm), we could find a solution that simultaneously makes the AS path mode-matching better than 99% for the t- and s-planes. 

The requirement of the LO path is less strict and the current plan using LO1-LO2 actuation should work. 

Summary:

I looked at some DC signals for the buildup of the carrier and sideband fields in various places. The results are shown in Attachments #1 and #2.

Details:

• A previous study may be found here.
• For the carrier field, REFL, POP and TRX/TRY all show the expected behavior. In particular, the REFL/TRX variation is consistent with the study linked in the previous bullet.
• There seems to be some offset between TRX and TRY - I don't yet know if this is real or just some PD gain imbalance issue.
• The 1-sigma variation in TRX/TRY seen here is consistent with the RMS RIN of 0.1 evaluated here.
• For the sideband powers, I guess the phasing of the POP22 and AS110 photodiodes should be adjusted? These are proxies for the buildup of the 11 MHz and 55 MHz sidebands in the vertex region, and so shouldn't depend on the arm offset, and so adjusting the digital demod phases shouldn't affect the LSC triggering for the PRMI locking, I think.
• Based on this data, the recycling gain for the carrier is ~12 +/- 2, so still undercoupled. In fact, at some points, I saw the transmitted power exceed 300, which would be a recycling gain of ~17, which is then nearly the point of critical coupling. REFLDC doesn't hit 0 because of the mode mismatch I guess.

Summary:

The measured RIN of the arm cavity transmission when the PRFPMI is locked is ~10x in RMS relative to the single arm POX/POY lock. It is not yet clear to me where the excess is coming from.

Details:

Attachment #1 shows the comparison.

• For the PRFPMI lock, the ITM Oplev Servos are DC coupled, and the ETM QPD ASC servos are also enabled.
• Admittedly, the PD used in the POX/POY lock case is the Thorlabs PD while when the PRFPMI is locked, it is the QPD.
• I found that there isn't really a big difference in the RIN if we normalize by the IMC transmission or not (this is what the "un-normalized" in the plot legend is referring to).  A scatter plot of TRX vs TRY and TRX/MCtrans vs TRY/MCtrans have nearly identical principal components. 
• To convert to RIN, I divided the ocmputed spectra by the mean value of the data stream. For the POX/POY lock, the arm transmission is normalized to 1, so no further manipulation is required.
• The spectra are truncated to 512 Hz because the IMC sum channel is DQ-ed at 1 kHz, but because of the above bullet point, in principle, I could calculate this out to higher frequencies.

Replaced empty N2 tank, left tank at ~2000 psi, right tank ~2600 psi.

Summary:

This isn't meant to be a serious budget, mainly it was to force myself to write the code for generating this more easily in the future.

Details:

• DARM OLTF model from here was used to undo the loop to convert the in-loop measurement to a free-running estimate.
• The AS55 PD channels were whitened to reduce the effect of ADC noise.
• To measured channel was 'C1:LSC-DARM_IN1_DQ'.
  • Some care needs to be taken when applying the conversion from counts to meters using the sensing element measured here.
  • This is because the sensing matrix measurement was made using the response in the channel 'C1:LSC-AS55_Q_ERR_DQ'.
  • Between 'C1:LSC-DARM_IN1_DQ' and 'C1:LSC-AS55_Q_ERR_DQ' there is a scalar gain of 1e-4, and a z:p = 20:0 filter.
  • These have to be corrected for when undoing the loop, since the measurement point is 'C1:LSC-DARM_IN1_DQ'. 
• The "Dark noise" trace was measured with the PSL shutter closed, but all CDS filters up to 'C1:LSC-DARM_IN1_DQ' enabled as they were when the DARM measurement was taken.
• It would be interesting to see what the budget looks like once the DARM loop gain has been turned down a bit, some low-pass filtering is enabled, and the vertex DoFs are transitioned to 1f control which is hopefully lower noise.

Summary:

The response of the PRFPMI length degrees of freedom as measured in the LSC PDs was characterized. Two visualizations are in Attachment #1 and Attachment #2.

Details:

• The sensing matrix infrastructure in the c1cal model was used.
• The oscillator frequencies are set between 300 - 315 Hz.
• Notch filters at these frequencies were enabled in the CDS filter banks, to prevent actuation at these frequencies (except for CARM, in which case the loop gain is still non-negligible at ~300 Hz, this correction has not yet been applied).
• Mainly, I wanted to know what the DARM sensing response in AS55_Q is. 
  • The measurement yields 2.3e13 cts/m. This is a number that will be used in the noise budget to convert the measured DARM spectrum to units of m/rtHz.
  • We have to multiply this by 10/2^15 V/ct, undo the 6dB whitening gain on the AS55_Q channel, and undo the ~5x gain from V_RF to V_IF (see Attachment #4 of this), to get ~0.69 GV/m from the RFPD.
  • The RF transimpedance of AS55_Q is ~550 ohms, and accounting for the InGaAs responsivity, I get an optical gain of 1.8 MW/m. Need to check how this lines up with expectations from the light levels, but seems reasonable.
  • Note that T_SRM is 10%, we dump 70% of the output field into the unused OMC, and there is a 50/50 BS splitting the light between AS55 and AS110 PDs. Assuming 90% throughput from the rest of the chain, we are only sensing ~1.3 % of the output DARM field.
• Apart from this, I can also infer what the matrix elements / gains need to be for transitioning the PRMI control from 3f to 1f signals. To be done...
• I found these histograms in Attachment #2 to be a cute way of (i) visualizing the variance in the magnitude of the sensing element and (ii) visualizing the separation between the quadratures, which tells us if the (digital) demod phase needs to be modified.
  • The sensing lines were on for 5 minutes (=300 seconds) and the FFT segment length is 5 seconds, so these histograms are binning the 60 different values obtained for the value of the sensing element.
  • The black dashed lines are "kernel density estimates" of the underlying PDFs
  • I haven't done any rigorous statistical analysis on the appropriateness of using this technique for error estimation, so for now, they are just lines...

Summary:

I am able to realize ~8 kHz UGF with ~60 degrees of phase margin on the CARM loop OLTF (combination of analog and digital signal paths).

Details:

• Attachment #1 shows the measured OLTF.
• The measurement is made by using the "EXC A" bank on the CM board, with an SR785. With this technique, the measurement will be poor where the loop gain is high, as the excitation will be squished. Nevertheless, we can estimate the behavior in those regimes by using a model, and fitting it to the regions where the measurement is valid (in this case, above ~1 kHz).
• This measurement was made with IN1 Gain = +4 dB, AO gain = 0 dB, and IMC IN2 gain = 0 dB.
• The regular boost has been enabled, but no super-boosts yet, mainly because I think they consume too much phase close to the UGF. 
• The modeling/fitting of this, including a more thorough characterization of the crossover, will follow...

Summary:

In order to estimate the free-running DARM displacement noise, I measured the DARM OLTF using the usual IN1/IN2 prescription. The measured data was then used to fit some model paramters for a loop model that can be used over a larger frequency range.

Details:

• Attachment #1 shows an overlay of the measured and modelled TFs.
• Attachment #2 shows the various components that went into building up this model. 
  • The digital AA and AI filter coefficients were taken from the RTCDS code.
  • The analog AA and AI filter zpks were taken from here and here respectively.
  • CDS filters taken from the banks enabled. The 20Hz : 0Hz z:p filter in the CARM_B path is also accounted for, as have the violin-mode notches.
  • Pendulum TF is just 1/f^2, the overall scaling is unimportant because it will be fitted (in combination with the overall scaling uncertainty on the DC optical gain), but I used a value of 10 nm/f^2 which should be in the right ballbark.
  • The optical gain includes the DARM pole at ~4.5 kHz for this config.
• With all these components, to make the measurement and fit line up, I added two free parameters - an overall gain, and a delay. 
  • NLSQ minimizer was used to find the best-fit values for these parameters.
  • I'm not sure what to make of the relatively large disagreement between measurement and model below 100 Hz - I'm pretty sure I got all the CDS filters included...
  • Moreover, I don't have a good explanation for why the best-fit delay is 400 us. One RTCDS clock cycle is onyl 60 us, and even with an extra clock cycle for the RFM transfer, I still can't get up to such a high delay...

In summary, the UGF is ~150 Hz and phase margin is ~30 deg. This loop would probably benefit from some low-pass filter being turned on.

Here, I provide some details of the sequence. Obviously, I am presenting one of the quickest transitions to the fully locked state, I don't claim that every attempt is so smooth. But it is pretty cool that the whole thing can be done in ~3 minutes.

See Attachment #1 for the labels.

• A --- Arms are locked on POX/POY, and EX/EY lasers are also locked to their respective arms. The phase tracker outputs are averaged in preparation for transitioning control from POX/POY to ALS. 
• B --- Aforementioned transition has been realized. CARM offset of -4 is applied. Based on this calibration, this is ~ 4 nm.
• C --- PRM is aligned in preparation for 3f vertex locking. Between C and D, the long pause is because I also use this time to DC couple the ITM Oplev servos, which requires some averaging. 
• D --- PRMI is locked. CARM offset reduction begins. Between D and E, I scan CARM through a resonance, and look at the necessary offset requried in the CARM_B (=RF) path. It is a mystery to me why this is required.
• E --- Ramp CARM offset completely to 0. Twiddle CARM_A and DARM_A offsets (=ALS path) to maximize the arm transmitted powers. Between E and F, you can see that the arm powers stabilize somewhat before any RF control is engaged (more on this later), which means we are approximately in the linear regime of the CARM PDH signal, and the switchover can be effected. As I write this, I wonder if there is any benefit to normalizing the REFL_11 error signal (=CM_SLOW) by the arm transmission for a broader capture range?
• F --- CARM_B and DARM_B (=RF) paths engaged. I ramp off the ALS servos between F and G using a 10 second ramptime.
• G --- IFO is now under RF control, ALS control has been turned off completely.
• H --- Rudimentary ASC is enabled. The ITMs are already running with DC coupled Oplev servos, and for the ETMs, I use the Transmon QPDs. The loop shapes/gains for this part haven't been finalized yet, but some improvement in the stability is seen.

This particular lock held for ~20 minutes so I could run some loop characterization measurements etc.

I am struggling to explain:

1. Why POP22 goes to 0 when we zero the CARM offset? The arm length is such that the 2f fields don't experience any abrupt changes in reflectivity from the arm cavity for a wide range of offsets. This signal is the trigger signal for the PRMI LSC control - right now, I get around this problem by mixing in some amount of POP DC once the PRMI is locked. But if the lock is lost, this requires some EPICS button gynmastics to try and salvage the lock... I guess the 1f field components experience a different phase on reflection at various offsets, so maybe I should be looking at sqrt(POP22_I^2 + POP22_Q^2) instead of just POP22_I.
2. Why is an error point offset required in the CARM RF path?

Summary:

Provided the IMC is cooperative, the input pointing isn't drifting, and the RF offsets aren't jumping around too much, the locking sequence is now pretty robust.

Details:

• Lock acquisition sequence [ELOG 15349]
• DARM loop measurement and fitting [ELOG 15350]
• CARM loop measurement and fitting [ELOG 15351]
• Sensing matrix measurement [ELOG 15352]
• Preliminary noise budget [ELOG 15353]
• ASC and arm RIN stabilization [ELOG 15355]
• Buildup of DC and SB fields [ELOG 15356]

Most of the analysis uses data between the GPS times 1274418176 and 1274419654 that are recorded to frames.

A big factor in how much IFO locking activities can take place is how cooperative the IMC is.

Since the c1psl upgrade, the IMC duty cycle has definitely deteriorated. I took a measurement of the dark noise at the IMC error point with 1 Hz FFT binwidth, with all electrical connections to the IMC servo board except the Acromag and Eurocrate power disconnected. I was horrified at the prominence of 60 Hz harmonics - see Attachment #1. In the past, this kind of feature has been indicative of some error in the measurement technique - but I confirmed that the lines remain even if I unplug the GPIB box, and all combinations of floating/grounded inputs that I tried. We know for sure that there is some excess noise imprinted on the laser light post upgrade. While these lines almost certainly are not responsible for the PCdrive RMS going bonkers, surely this kind of electrical situation isn't good?

Attachment #2 shows the same information translated to frequency noise units, taking into account the complementary sensitivity function, L/(1+L) - the sum contribution of the 60 Hz peaks to the RMS is ~11.5% of the total over the entire band (c.f. 1.7 % that is expected if the noise at multiples of 60 Hz was approximately equal to the surrounding noise levels). Moreover, the measured RMS is 55 times higher than a LISO model. 

How can this be fixed?

so far it has run through the weekend with no problems (except that there are huge log files as usual).

I have started to set up monit to run on megatron to watch this process. In principle this would send us alerts when things break and also give a web interface to watch monit. I'm not sure how to do web port forwarding between megatron and nodus, so for now its just on the terminal. e.g.:

monit>sudo monit status
Monit 5.25.1 uptime: 4m

System 'megatron'
  status                       OK
  monitoring status            Monitored
  monitoring mode              active
  on reboot                    start
  load average                 [0.15] [0.22] [0.25]
  cpu                          0.6%us 1.0%sy 0.2%wa
  memory usage                 1001.4 MB [25.0%]
  swap usage                   107.2 MB [1.9%]
  uptime                       40d 17h 55m
  boot time                    Tue, 14 Apr 2020 17:47:49
  data collected               Mon, 25 May 2020 11:43:03

Process 'nds2'
  status                       OK
  monitoring status            Monitored
  monitoring mode              active
  on reboot                    start
  pid                          25007
  parent pid                   1
  uid                          4666
  effective uid                4666
  gid                          4666
  uptime                       3d 1h 22m
  threads                      53
  children                     0
  cpu                          0.0%
  cpu total                    0.0%
  memory                       19.4% [776.1 MB]
  memory total                 19.4% [776.1 MB]
  security attribute           unconfined
  disk read                    0 B/s [2.3 GB total]
  disk write                   0 B/s [17.9 MB total]
  data collected               Mon, 25 May 2020 11:43:03

was dead again this morning - JZ notified

current restart instructions (after ssh to megatron):

cd /home/nds2mgr/nds2-megatron

sudo su nds2mgr

make -f test_restart