40m QIL Cryo_Lab CTN SUS_Lab TCS_Lab OMC_Lab CRIME_Lab FEA ENG_Labs OptContFac Mariner WBEEShop
  40m Log, Page 314 of 339  Not logged in ELOG logo
ID Date Author Typeup Category Subject
  15364   Wed Jun 3 01:34:53 2020 gautamUpdateLSCLock acquisition update portal

Highlights:

  • With better ASC servos implemented, I think the lock stability has been improved. 
  • Arm transmission of ~350 was stably maintained (PRG~20, overcoupled). It went as high as 410, so this is now very close to the highest (~425) I've ever managed to get.
  • I was trying to get the vertex transitioned to 1f control but it remains out of reach for now.  The noise at ~100 Hz is dominated by MICH-->DARM coupling (as judged by coherence, I haven't yet done the broadband noise injection characterization). I figured I'd try the 1f transition before thinking about feedforward.
  • The biggest problems remain flaky electronics (poor IMC duty cycle, jumping RF offsets, newly glitchy seismometer, ...)

Details:

  15365   Wed Jun 3 01:40:13 2020 gautamUpdateElectronicsMore electronics woes

There were many locklosses from the point where the arm powers were somewhat stabilized. Attachments #1 and #2 show two individual locklosses. I think what is happening here is that the BS seismometer X channel is glitching, and creating a transient in the angular feedforward filter that blows the lock. The POP QPD based feedback loop cannot suppress this transient, apparently. For now, I get around this problem by boosting the POP QPD feedback loop a bit, and then turning the feedforward filters off. The fact that the other seismometer channels don't report any transient makes me think the problem is either with the seismometer itself, or the readout electronics. The seismometer masses were recently recentered, so I'm leaning towards the latter.

I didn't explicitly check the data, but I am reasonably certain the same effect is responsible for many PRMI locklosses even with the arms held off resonance (though the tolerance to excursions there is higher). Pity really, the feedforward filters were a big help in the lock acquisition...

Attachment 1: glitchySeis2.png
glitchySeis2.png
Attachment 2: glitchySeis3.png
glitchySeis3.png
  15366   Wed Jun 3 01:46:14 2020 gautamUpdateLSCCARM loop

Summary:

The CARM loop now has a UGF of ~12 kHz with a phase margin of ~60 degrees. These values of conventional stability indicators are good. The CARM optical gain that best fits the measurements is 9 MW/m.

I've been working on understanding the loop better, here are the notes.

Details:

Attachment #1 shows a block diagram of the loop topology.

  • The "crossover" measurement made at the digital CARM error point, and the OLG measurement at the CM board error point are shown.
  • I've tried to include all the pieces in the loop, and yet, I had to introduce a fudge gain in the digital path to get the model to line up with the measurement (see below).

Attachment #2 shows the OLGs of the two actuation paths.

  • Aforementioned fudge factor for the digital path is included.
  • For the AO path, I assumed a value of the PDH discriminant at the IMC error point to be 13 kHz/V, per my earlier measurement. 
  • I trawled the elog for the most up-to-date info about the IMC servo (elog9457, elog13696, elog15044) and CM board, to build up the model. 

Attachment #3 and #4 show the model, overlaid with measurements of the loop OLG and crossover TF respectively.

  • No fitting is done yet - the next step would be to add the delay of the CDS system for the digital path, and the analog electronics for the AO path. Though these are likely only small corrections.
  • For the crossover TF - I've divided out the digital filters in the CARM_B filter bank, because the injection is made downstream of it (see Attachment #1).
  • There is reasonably good agreement between model and measurement.
  • I think the biggest source of error is the assumed model for the IMC OLTF.

Attachment #5 shows the evolution of the CARM OLG at a few points in the lock acquisition sequence.

  • "Before handoff" corresponds to the state where the primary control is still done by the ALS leg, but the REFL11 signal has begun to enter the picture via the CARM_B path.
  • "IN2 ramped" corresponds to the state where the AO path gain (=IN2 gain on the IMC servo board) has been ramped up to its final value (+0 dB), but the overall loop gain (=IN1 gain on the CM board) is still low. So this is preparation for high bandwidth control. Typically, the arm powers will have stabilized in this state, but ALS control is still on.
  • "Pre-boost" corresponds to an intermediate state - ALS control is off, but the low frequency boosts have not yet been enabled. I typically first engage some ASC to stabilize things somewhat, and then turn on the boosts.
  • "Final" - self explanatory.

Next steps:

Now the I have a model I believe, I need to think about whether there is any benefit to changing some of these loop shapes. I've already raised the possibility of changing the shape of the boosts on the CM board, with which we could get a bit more suppression in the 100 Hz - 1kHz region (noise budget of laser frequency noise --> DARM required to see if this is necessary). 

Attachment 1: CM_loop_topology.pdf
CM_loop_topology.pdf
Attachment 2: CARM_TFs.pdf
CARM_TFs.pdf
Attachment 3: CARM_OLTF.pdf
CARM_OLTF.pdf
Attachment 4: CARM_xover.pdf
CARM_xover.pdf
Attachment 5: CARM_OLG_evolution.pdf
CARM_OLG_evolution.pdf
  15367   Wed Jun 3 02:08:00 2020 gautamUpdateLSCPower buildup diagnostics

Attachments #1 and Attachments #2 are in the style of elog15356, but with data from a more recent lock. It'd be nice to calibrate the ASDC channel (and in general all channels) into power units, so we have an estimate of how much sideband power we expect, and the rest can be attributed to carrier leakage to ASDC.

On the basis of Attachments #1, the PRG is ~19, and at times, the arm transmission goes even higher. I'd say we are now in the regime where the uncertainty of the losses in the recycling cavity - maybe beamsplitter clipping? is important in using this info to try and constrain the arm cavity losses. I'm also not sure what to make of the asymmetry between TRX and TRY. Allegedly, the Y arm is supposed to be lossier.

Quote:

This is very interesting. Do you have the ASDC vs PRG (~ TRXor TRY) plot? That gives you insight on what is the cause of the low recycling gain.

Attachment 1: PRFPMIcorner_DC_1275190251_1275190551.pdf
PRFPMIcorner_DC_1275190251_1275190551.pdf
Attachment 2: PRFPMIcorner_SB_1275190251_1275190551.pdf
PRFPMIcorner_SB_1275190251_1275190551.pdf
  15368   Wed Jun 3 02:14:32 2020 gautamUpdateASCPRC ASC improves arm transmission RIN

Summary:

I implemented an ASC servo for the PRC, with the POP QPD as a sensor, and the PRM as the actuator. This has improved the stability of the lock (longer locks are possible), and also reduced the RIN of the arm transmission.

Details:

Attachment #1 shows the in-loop error signal suppression, and some out-of-loop monitors (POP22 and POPDC).

  • To practise and get some workable servo settings, I locked the PRMI with carrier resonant (no ETMs).
  • Then, I compare the beam motion witnessed by the POP QPD with and without the feedback loop enabled.
  • I also look at the spectra of the POPDC and POP22 signals, as out-of-loop proxies, to get an estimate of how much noise is being injected out of band.
  • In this toy study, both the in-loop and out of loop monitors show good performance.
  • However, when repeating the same diagnostics with the PRFPMI locked, I note that while the in-loop suppression looks good, POPDC and POP22 report elevated noise, relative to the PRMI carrier case.
  • I don't have a comparison to the PRFPMI locked with the feedback disabled, because of stability reasons. Plus, for the PRMI, the angular feedforward loops were engaged, but for the PRFPMI traces, they were disabled.
  • Nevertheless, the arm RIN goes down by ~2.5 in RMS, so this is doing something good.

Attachment #2 compares the arm transmission RIN with the PRFPMI locked, with and without PRC ASC. The 3 Hz bump is definitely squished, but I think we can do better yet. 

Attachments #3-5 are in the style of elog15361. No Oplev signals yet, I'll add them soon.

I guess what this means is that the stability of the lock could be improved by turning on some POP QPD based feedback control, I'll give it a shot

Attachment 1: PRC_ASCsignals.pdf
PRC_ASCsignals.pdf
Attachment 2: armRIN_PRC_ASC.pdf
armRIN_PRC_ASC.pdf
Attachment 3: PRFPMIcorner_ASC_PIT_1275190251_1275190551.pdf
PRFPMIcorner_ASC_PIT_1275190251_1275190551.pdf
Attachment 4: PRFPMIcorner_ASC_YAW_1275190251_1275190551.pdf
PRFPMIcorner_ASC_YAW_1275190251_1275190551.pdf
Attachment 5: PRFPMIcorner_ASC_coherence_1275190251_1275190551.pdf
PRFPMIcorner_ASC_coherence_1275190251_1275190551.pdf
  15369   Wed Jun 3 03:29:26 2020 KojiUpdateLSCLock acquisition update portal

Woo hoo!

Which 1f signals are you going to use? PRCL has sign flipping at the carrier critical coupling. So if the IFO is close to that condition, 1f PRCL suffers from the sign flipping or large gain variation.

  15370   Wed Jun 3 11:20:19 2020 gautamUpdateDetCharSummary pages

Summary:

The 40m summary pages have been revived. I've not had to make any manual interventions in the last 5 days, so things seem somewhat stable, but I'm sure there will need to be multiple tweaks made. The primary use of the pages right now are for vacuum, seismic and PSL diagnostics.

Resources:

Caveats:

  • Intermittent failures of cron jobs
    • The status page relies on the condor_q command executing successfully on the cluster end. I have seen this fail a few times, so the status page may say the pages are dead whereas they're actually still running.
    • Similarly, the rsync of the pages to nodus where they're hosted can sometimes fail.
    • Usually, these problems are fixed on the next iteration of the respective cron jobs, so check back in ~half hour.
  • I haven't really looked into it in detail, but I think our existing C1:IFO-STATE word format is not compatible with what gwsumm wants - I think it expects single bits that are either 0 or 1 to indicate a particular state (e.g. MC locked, POX and POY locked etc). So if we want to take advantage of that infrastructure, we may need to add a few soft EPICS channels that take care of some logic checking (several such bits could also be and-ed together) and then assume either 0 or 1 value. Then we can have the nice duty cycle plots for the IMC (for example).
  • I commented out the obsolete channels (e.g. PEM MIC channels). We can add them back later if we so desire.
  • For some reason, the jobs that download the data are pretty memory-heavy: I have to request for machines on the cluster with >100 GB (yes 💯GB) ! of memory for the jobs to execute to completion. The frame-data certainly isn't so large, so I wonder what's going on here - is GWPy/GWsumm so heavy? The site summary pages run on a dedicated cluster, so probably the code isn't built for efficiency...
  • Weather tab in PEM is still in there but none of those channels mean anything right now.
  • The MEDM screenshot stuff is commented out for now too. This should be redone in a better way with some modern screen grab utilities, I'm sure there are plenty of python based ones.
  • There seems to be a problem with the condor .dag lockfile / rescue file not being cleared correctly sometimes - I am looking into this.
  15371   Wed Jun 3 11:40:56 2020 gautamUpdateLSCLock acquisition update portal

For these initial attempts, I was just trying to transition MICH to REFL55Q. I agree, the PRCL situation may be more complicated.

Which 1f signals are you going to use? PRCL has sign flipping at the carrier critical coupling. So if the IFO is close to that condition, 1f PRCL suffers from the sign flipping or large gain variation.

  15372   Wed Jun 3 18:49:47 2020 gautamUpdateLSCPRG and CARM signal sign

Summary:

I am inclined to believe that the arm cavity losses are such that the IFO is overcoupled. Some calculations, validated with Finesse modeling also suggest that there isn't a sign change for the CARM error signal when the IFO goes from being undercoupled to overcoupled, but I may have made a mistake here?

Details:

  • We’d like to gain some insight into whether the interferometer is undercoupled, critically coupled, or overcoupled. Factors that determine which of these is true include:
    • Arm cavity losses
    • Recycling cavity losses
  • The proxy by which we determine the recycling gain is usually the arm cavity transmission. Assuming T_PRM = 5.637 % according to the wiki, and assuming the arm cavity transmission is normalized to 1 when locked in the POX/POY state, we can say that the PRG is given by G_PRC = TRX × T_PRM, assuming that the (i) the RF sideband fields are perfectly rejected by the arm cavities and (ii) mode-matching efficiency between the input beam and the arm mode is the same as that between the input beam and the CARM mode.
  • Apart from this, the other measurement we have available to us is the buildup of the sideband fields, namely POP22 and POP110. We can compare the values in the PRMI lock vs the PRFPMI to make some inference.
  • I started off with an analytic calculation of the reflectivity of the compound arm cavity mirror.
    • Attachment #1 suggests we will have an over-coupled IFO for arm cavity losses below ~200 ppm, which is a regime we are almost certainly in now.
  • Then, I repeat the analysis for the coupled CARM cavity, with the end mirror as the compound arm mirror and the input mirror as the PRM.
    • I assume 2 % loss in the PRC.
    • Attachment #2 shows that while the carrier field goes through a sign change in amplitude reflectivity (as expected), the sideband fields dont.
    • Per equation 4.2 of Koji's thesis, the error signal for CARM depends on the (signed) IFO reflectivity, and the absolute value of the derivative of the arm cavity reflectivity for the carrier w.r.t. CARM phase.
    • So, we don't expect the REFL11 signal to show a sign change.
    • The situation is more complicated for PRCL in REFL11, because as explicitly evaluated in Eq 4.3 of Koji's thesis, there are two terms that contribute, and their relative magnitudes will dictate the overall sign. 
  • For a Finesse validation, I use a simplified 3 mirror coupled cavity to approximate the PRFPMI. I also retained the RF sidebands for diagnostic purposes. The idea was to study these PRG proxies and what their expected behavior is.
    • Attachment #3 shows the PDH error signal in the (arbitrarily defined) REFL11 I quadrature. While the optical gain changes as a function of the arm cavity loss, the actual slope does not change sign. The fact that the zero crossing doesn't happen at exactly 0 CARM offset is because of higher order mode light at the REFL port (in my model, I tried to preserve the flipped folding mirror situation so the mode matching between the arm cavity and PRC in my model is ~96%).
    • In fact, this may explain why a CARM_B offset is required to do the ALS-->IR handoff - the ALS servo wants to keep the arm offset to zero, but at that point, the PDH error signal isn't zero, and so the two loops end up fighting each other?
    • Attachment #4 is a more detailed study of the recycling gain as a function of arm cavity loss, but now including losses in the recycling cavity.

Conclusions:

  1. I think the arm cavity losses are in the 60-80 ppm round-trip region. I don't see how we can explain the arm cavity transmission of ~350 otherwise.
  2. The fact that REFLDC decreases as the arm transmission increases is because the input beam is getting better matched to the CARM mode, and there is less junk carrier light. 

Thoughts from others?

Attachment 1: armCavReflectivities.pdf
armCavReflectivities.pdf
Attachment 2: IFOreflectivities.pdf
IFOreflectivities.pdf
Attachment 3: PDHerrSigs.pdf
PDHerrSigs.pdf
Attachment 4: PRGvsLoss_finesse.pdf
PRGvsLoss_finesse.pdf
  15373   Wed Jun 3 19:19:11 2020 gautamUpdateSUSAll watchdogs tripped

This EQ seems to have knocked all suspensions out. ITMX was stuck. It is now released, and the IMC is locked again. It looks like there are some serious aftershocks going on so let's keep an eye on things.

  15375   Thu Jun 4 08:45:41 2020 JordanUpdateGeneralPresence at 40m

I will be at the 40m, in the Clean and bake lab today from ~9am to ~3pm.

  15376   Thu Jun 4 20:54:40 2020 gautamUpdateSUSMC1 Slow Bias issues

Summary:

I found that there is an issue with the MC1 slow bias voltages. 

Details:

I usually offload the DC part of the output voltage from the WFS servos to the slow bias voltage sliders, so as to preserve maximum actuation range from the fast system. However, today, I found that this servo wasn't working well at all. So I dug a little deeper. Looking at the EPICS database records:

  • The user-facing channels are "PIT" and "YAW" bias voltages.
  • These are converted to voltages to be sent to individual coils by some calc channels in the EPICS database record. So, for example, the voltage to be sent to the "UL" coil (Upper Left, as viewed from the AR side of the optic), is A+B, where A is the "PIT" voltage and B is the "YAW" voltage. Similar combinations of A and B are used for the other 3 face coils.
  • The problem is obvious - if either A or B > 5V, then the requested voltage to be sent to the UL coil is > 10 V, while the Acromag DACs can put out a maximum of 10 V
  • As it happens, with the IFO currently aligned, MC1 is the only optic which faces this problem. 
  • Why has this not been an issue before? In fact, looking at some old data, the "PIT" and "YAW" bias voltages to MC1 were both ~1-2 V in 2018. But I confirmed that something in the region of ~5 V is required from each of the "PIT" and "YAW" channels to bring the MCREFL spot back to the center of the camera, so something has changed the DC alignment of MC1, maybe an earthquake or something? Anyway, with these settings, 2/4 coils are basically saturated, and so we can only move the optic diagonally. 😢 
  • Other coils that have  requested output voltages > 5V (so more than half the range of the DAC) include MC2 LL (5.2V), and ETMX LL and LR (5.5 and 5.8 V respectively).
  • Either a factor of 0.5 should be included in all the EPICS database records, or else, we should make the "PIT" and "YAW" sliders range only from -5 to +5 V, so that this kind of misleading info isn't wasting time.
  15377   Thu Jun 4 21:32:00 2020 KojiUpdateSUSMC1 Slow Bias issues

We can limit the EPICS values giving some parameters to the channels. cf https://epics.anl.gov/tech-talk/2012/msg00147.php

But this does not solve the MC1 issue. Only we can do right now is to make the output resister half, for example.

  15378   Fri Jun 5 08:44:50 2020 JordanUpdateGeneralPresence at 40m

I will be at the 40m, in the Clean and bake lab today from ~9am to ~3pm.

  15379   Sat Jun 6 14:07:30 2020 JonUpdateBHDStock-Part Mode-Matching Telescope Option

Summary

For the initial phase of BHD testing, we recently discussed whether the mode-matching telescopes could be built with 100% stock optics. This would allow the optical system to be assembled more quickly and cheaply at a stage when having ultra-low loss and scattering is less important. I've looked into this possibility and conclude that, yes, we do have a good stock optics option. It in fact achieves comprable performance to our optimized custom-curvature design [ELOG 15357]. I think it is certainly sufficient for the initial phase of BHD testing.

Vendor

It turns out our usual suppliers (e.g., CVI, Edmunds) do not have enough stock options to meet our requirements. This is for two reasons:

  • For sufficient LO1-LO2 (AS1-AS4) Gouy phase separation, we require a very particular ROC range for LO1 (AS1) of 5-6 m (2-3 m).
  • We also require a 2" diameter for the suspended optics, which is a larger size than most vendors stock for curved reflectors (for example, CVI has no stock 2" options).

However I found that Lambda Research Optics carries 1" and 2" super-polished mirror blanks in an impressive variety of stock curvatures. Even more, they're polished to comprable tolerances as I had specificied for the custom low-scatter optics [DCC E2000296]: irregularity < λ/10 PV, 10-5 scratch-dig, ROC tolerance ±0.5%. They can be coated in-house for 1064 nm to our specifications.

From modeling Lambda's stock curvature options, I find it still possible to achieve mode-matching of 99.9% for the AS beam and 98.6% for the LO beam, if the optics are allowed to move ±1" from their current positions. The sensitivity to the optic positions is slightly increased compared to the custom-curvature design (but by < 1.5x). I have not run the stock designs through Hang's full MC corner-plot analysis which also perturbs the ROCs [ELOG 15339]. However for the early BHD testing, the sensitivity is secondary to the goal of having a quick, cheap implementation.

Stock-Part Telescope Designs

The following tables show the best telescope designs using stock curvature options. It assumes the optics are free to move ±1" from their current positions. For comparison, the values from the custom-curvature design are also given in parentheses.

AS Path

The AS relay path is shown in Attachment 1:

  • AS1-AS4 Gouy phase separation: 71°
  • Mode-matching to OMC: 99.9%
Optic ROC (m) Distance from SRM AR (m)
AS1 2.00  (2.80) 0.727  (0.719)
AS2 Flat   (Flat) 1.260  (1.260)
AS3 0.20  (-2.00) 1.864  (1.866)
AS4 0.75  (0.60) 2.578  (2.582)

LO Path

The LO relay path is shown in Attachment 2:

  • LO1-LO2 Gouy phase separation: 67°
  • Mode-matching to OMC: 98.6%
Optic ROC (m) Distance from PR2 AR (m)
LO1 5.00  (6.00) 0.423  (0.403)
LO2 1000 (1000) 2.984  (2.984)
LO3 0.50  (0.75) 4.546  (4.596)
LO4 0.15  (-0.45) 4.912  (4.888)

Ordering Information

I've created a new tab in the BHD procurement spreadsheet ("Stock MM Optics Option") listing the part numbers for the above telescope designs, as well as their fabrication tolerances. The total cost is $2.8k + the cost of the coatings (I'm awaiting a quote from Lambda for the coatings). The good news is that all the curved substrates will receive the same HR/AR coatings, so I believe they can all be done in a single coating run.

Attachment 1: ASpathStock.pdf
ASpathStock.pdf
Attachment 2: LOpathStock.pdf
LOpathStock.pdf
  15380   Mon Jun 8 11:50:02 2020 HangUpdateBHDAstigmatism and scattering plots

We consider the astigmatism effects of the stock options. The conclusions are:

1. For the AS path, the stock should work fine for the phase-one of BHD, if we could tolerate a few percent MM loss. The window for length adjustment to achieve >99% MM for both s and t is only 1 mm for 1% RoC error (compared to ~ 1 cm in the customized case). 

2. The LO path seemed tricky. As LO3 & LO4 are both significantly curved (RoC<=0.5 m), the non-zero angle of incidence makes the astigmatism quite sever. For the t-plane the nominal MM can be 0.98, yet for the s-plane, the nominal MM is only 0.72. We could move things around to achieve a MM ~ 0.85, which is probably fine for the phase-one implementation but not long term. 

Details:

Attachments 1-3 are for the AS path; 4-6 are for the LO path. 

1 & 4. Marginalized MM distribution for the AS/LO paths. Here we assumed 5 mm positional error and 1% fractional RoC error. Due to the astigmatism, the nominal s-plane MM is only 0.72 for the LO path. 

2 & 5. Scattering plots for the AS/LO paths. We color coded the points as the following: pink: MM>0.99; olive: 0.98<MM<=0.99; grey: MM<=0.98. For the AS path, MM is mostly sensitive to the AS1 RoC and can be adjusted by changing AS1-AS3 distance. For the LO path, the LO3 RoC and LO3-LO4 distance are most critical for the MM. 

3 & 6. Assuming +- 1% AS1 (LO3) fractional RoC error, how much can we compensate for it using AS1-AS3 (LO3-LO4) distance. For the AS path, there exists a ~ 1 mm window where the MM for s and t can simultaneously > 99%. For the LO path, the best we can do is to make s and t both ~ 85%. 

Quote:

Summary

For the initial phase of BHD testing, we recently discussed whether the mode-matching telescopes could be built with 100% stock optics. This would allow the optical system to be assembled more quickly and cheaply at a stage when having ultra-low loss and scattering is less important. I've looked into this possibility and conclude that, yes, we do have a good stock optics option. It in fact achieves comprable performance to our optimized custom-curvature design [ELOG 15357]. I think it is certainly sufficient for the initial phase of BHD testing.

Vendor

It turns out our usual suppliers (e.g., CVI, Edmunds) do not have enough stock options to meet our requirements. This is for two reasons:

  • For sufficient LO1-LO2 (AS1-AS4) Gouy phase separation, we require a very particular ROC range for LO1 (AS1) of 5-6 m (2-3 m).
  • We also require a 2" diameter for the suspended optics, which is a larger size than most vendors stock for curved reflectors (for example, CVI has no stock 2" options).

However I found that Lambda Research Optics carries 1" and 2" super-polished mirror blanks in an impressive variety of stock curvatures. Even more, they're polished to comprable tolerances as I had specificied for the custom low-scatter optics [DCC E2000296]: irregularity < λ/10 PV, 10-5 scratch-dig, ROC tolerance ±0.5%. They can be coated in-house for 1064 nm to our specifications.

From modeling Lambda's stock curvature options, I find it still possible to achieve mode-matching of 99.9% for the AS beam and 98.6% for the LO beam, if the optics are allowed to move ±1" from their current positions. The sensitivity to the optic positions is slightly increased compared to the custom-curvature design (but by < 1.5x). I have not run the stock designs through Hang's full MC corner-plot analysis which also perturbs the ROCs [ELOG 15339]. However for the early BHD testing, the sensitivity is secondary to the goal of having a quick, cheap implementation.

Stock-Part Telescope Designs

The following tables show the best telescope designs using stock curvature options. It assumes the optics are free to move ±1" from their current positions. For comparison, the values from the custom-curvature design are also given in parentheses.

AS Path

The AS relay path is shown in Attachment 1:

  • AS1-AS4 Gouy phase separation: 71°
  • Mode-matching to OMC: 99.9%
Optic ROC (m) Distance from SRM AR (m)
AS1 2.00  (2.80) 0.727  (0.719)
AS2 Flat   (Flat) 1.260  (1.260)
AS3 0.20  (-2.00) 1.864  (1.866)
AS4 0.75  (0.60) 2.578  (2.582)

LO Path

The LO relay path is shown in Attachment 2:

  • LO1-LO2 Gouy phase separation: 67°
  • Mode-matching to OMC: 98.6%
Optic ROC (m) Distance from PR2 AR (m)
LO1 5.00  (6.00) 0.423  (0.403)
LO2 1000 (1000) 2.984  (2.984)
LO3 0.50  (0.75) 4.546  (4.596)
LO4 0.15  (-0.45) 4.912  (4.888)

Ordering Information

I've created a new tab in the BHD procurement spreadsheet ("Stock MM Optics Option") listing the part numbers for the above telescope designs, as well as their fabrication tolerances. The total cost is $2.8k + the cost of the coatings (I'm awaiting a quote from Lambda for the coatings). The good news is that all the curved substrates will receive the same HR/AR coatings, so I believe they can all be done in a single coating run.

 

Attachment 1: AS_MM_hist_stock.pdf
AS_MM_hist_stock.pdf
Attachment 2: AS_MM_t_scat_stock.pdf
AS_MM_t_scat_stock.pdf
Attachment 3: AS_MM_adj_stock.pdf
AS_MM_adj_stock.pdf
Attachment 4: LO_MM_hist_stock.pdf
LO_MM_hist_stock.pdf
Attachment 5: LO_MM_s_scat_stock.pdf
LO_MM_s_scat_stock.pdf
Attachment 6: LO_MM_adj_stock.pdf
LO_MM_adj_stock.pdf
  15381   Mon Jun 8 12:49:07 2020 KojiUpdateBHDAstigmatism and scattering plots

Can you describe the mode matching  in terms of the total MM? Is MM_total = sqrt(MM_vert * MM_horiz)?

  15382   Mon Jun 8 17:40:22 2020 JonUpdateBHDAstigmatism and scattering plots

MM_total = (MM_vert + MM_horiz) / 2. 

The large astigmatic MM loss in the LO case is mainly due to the strong LO4 curvature (R=0.15m) with a 10 deg AOI. I looked again at whether LO1 could be increased from R=5m to the next higher stock value of 7.5m, as this would allow weaker curvatures on LO3 and LO4. However, no, that is not possible---it reduces the LO1-LO2 Gouy phase separation to only 18 deg.

There is, however, a good stock-curvature option if we want to reconsider actuating LO4 instead of LO2 (attachment 1). It achieves 99.2% MM with the OMCs, allowing positions to vary +/-1" from the current design. The LO1-LO4 Gouy phase separation is 72 deg.

Optic ROC (m) Distance from PR2 AR (m)
LO1 10 0.378
LO2 1000 2.984
LO3 10 4.571
LO4 7.5 4.926

Alternatively, we could look at reducing the AOI on LO3 and LO4 (keeping LO1-LO2 actuation).

Attachment 1: LOpathStock2.pdf
LOpathStock2.pdf
  15383   Mon Jun 8 18:14:55 2020 gautamUpdateCDSVertex FEs crashed

Summary:

Around 5pm local time, the three vertex FEs crashed. AFAIK, no one was in the lab or working on anything CDS related, so this is worrying.

Details:

  • Reboot script was used to bring all FEs back - only soft reboots were required.
  • The IMC and arms can now be locked.
  • I think combination of burt + SDF would have reverted all the settings as they should be, but if something appears off, it could be that some EPICS value didn't get reset correctly.
Attachment 1: FEcrash_CDSoverview.png
FEcrash_CDSoverview.png
  15384   Mon Jun 8 21:45:47 2020 JonUpdateBHDAstigmatism and scattering plots

Hmm? T1300364 suggests MM_total = Sqrt(MM_Vert * MM_Horiz)

  15385   Tue Jun 9 09:35:02 2020 JordanUpdateGeneralPresence at 40m

I will be in the Clean and Bake lab today from 9:30am to 4pm.

  15386   Tue Jun 9 14:55:43 2020 JonUpdateBHDMM telescope actuation range requirments

I don't think we ever discussed why the angular RMS of the ETMs is so much higher than the ITMs. Maybe that's a separate matter because, even assuming the worst case, the actuation range requirement is

(0.82 μrad RMS) x (15 μrad/μrad) x (10 safety factor) = 0.12 mrad

which is still only order 1% of the pitch/yaw pointing range of the Small Optic Suspensions, according to P1600178 (sec. IV. A). Can we check this requirement off the list?

Quote:

We computed the required actuation range for the telescope design in elog:15357. The result is summarized in the table below. Here we assume we misalign an IFO mirror by 1 urad, and then compute how many urad do we need to move the (AS1, AS4) or (LO1, LO2) mirrors to simultaneously correct for the two gouy phases. 

Actuation requirement in urad per urad misalignment
[urad/urad] ITMX ITMY ETMX ETMY BS PRM PR2 PR3 SR3 SRM
AS1 1.9 2.1 -5.0 -5.5 0.5 0.5 -0.3 0.2 0.1 0.6
AS4 2.9 2.0 -8.8 -5.5 -5.9 -0.7 1.3 -0.7 -0.5 0.7
LO1 -4.0 -3.9 11.0 10.4 1.9 -0.4 -0.2 0.1 0.0 -1.1
LO2 -5.0 -3.7 15.1 10.4 8.7 0.8 1.9 1.1 0.7 -1.3

The most demanding ifo mirrors are the ETMs and the BS, for every 1 urad misalignment the telescope needs to move 10-15 urad to correct for that. However, it is unlikely for those mirrors to move more 100 nrad for a locked ifo with ASC engaged. Thus a few urad actuation should be sufficient. For the recycling mirrors, every 1 urad misalignment also requires ~ 1 urad actuation. 

As a result, if we could afford 10 urad actuation range for each telescope suspension, then the gouy phase separations we have should be fine. 

================================================================

Edits:

We looked at the oplev spectra from gps 1274418500 for 512 sec. This should be a period when the ifo was locked in the PRFPMI state according to elog:15348. We just focused on the yaw data for now. Please see the attached plots. The solid traces are for the ASD, and the dotted ones are the cumulative rms. The total rms for each mirror is also shown in the legend. 

I am now confused... The ITMs looked somewhat reasonable in that at least the < 1 Hz motion was suppressed. The total rms is ~ 0.1 urad, which was what I would expect naively (~ x100 times worse than aLIGO). 

There seems to be no low-freq suppression on the ETMs though... Is there no arm ASC at the moment???

 

  15387   Tue Jun 9 15:02:56 2020 eHangUpdateBHDAstigmatism and scattering plots

Using the updated AOI's for the LO path: (4.8, 47.9, 2.9, 4.5) deg for (LO1, LO2, LO3, LO4), we obtain the following results. 

First two plots are scattering plots for the t and s planes, respectively. Note that here we have changed to 0.5% fractional RoC error and 3 mm positional error. We have also changed the meaning of the colors: pink:MM>0.98; olive 0.95<MM<=0.98, and grey MM<=0.95. It seems that both planes would benefit statistically if we make the LO3-LO4 distance longer by a few mm. 

We also consider how much we could compensate for the MM error in the last plot. We have a few mm window to make both planes better than 0.95. 

Attachment 1: LO_MM_t_scat_stock.pdf
LO_MM_t_scat_stock.pdf
Attachment 2: LO_MM_s_scat_stock.pdf
LO_MM_s_scat_stock.pdf
Attachment 3: LO_MM_adj_stock.pdf
LO_MM_adj_stock.pdf
  15388   Wed Jun 10 14:00:33 2020 JordanUpdateGeneralPresence at 40m

I will be in the Clean and Bake lab from 10am to 4pm today. I will also replace an empty N2 cylinder.

  15389   Thu Jun 11 09:37:38 2020 JonUpdateBHDConclusions on Mode-Matching Telescopes

After further astigmatism/tolerance analysis [ELOG 15380, 15387] our conclusion is that the stock-optic telescope designs [ELOG 15379] are sufficient for the first round of BHD testing. However, for the final BHD hardware we should still plan to procure the custom-curvature optics [DCC E2000296]. The optimized custom-curvature designs are much more error-tolerant and have high probability of achieving < 2% mode-matching loss. The stock-curvature designs can only guarantee about 95% mode-matching.

Below are the final distances between optics in the relay paths. The base set of distances is taken from the 2020-05-21 layout. To minimize the changes required to the CAD model, I was able to achieve near-maximum mode-matching by moving only one optic in each relay path. In the AS path, AS3 moves inwards (towards the BHDBS) by 1.06 cm. In the LO path, LO4 moves backwards (away from the BHDBS) by 3.90 cm.

AS Path

Interval Distance (m) Change (cm)
SRMAR-AS1 0.7192 0
AS1-AS2 0.5405 0
AS2-AS3 0.5955 -1.06
AS3-AS4 0.7058 -1.06
AS4-BHDBS 0.5922 0
BHDBS-OMCIC 0.1527 0

LO Path

Interval Distance (m) Change (cm)
PR2AR-LO1 0.4027 0
LO1-LO2 2.5808 0
LO2-LO3 1.5870 0
LO3-LO4 0.3691 +3.90
LO4-BHDBS 0.2573 +3.90
BHDBS-OMCIC 0.1527 0
  15390   Thu Jun 11 11:14:12 2020 JordanUpdateGeneralPresence at 40m

I will be in the Clean and Bake lab today from 11am to 4pm.

  15391   Thu Jun 11 11:48:43 2020 gautamUpdateVACVac failure

There appears to have been some sort of vacuum failure.

ldas-pcdev1 was down, so the summary pages weren't being generated. I have now switched over to ldas-pcdev6. I suspect some forepump failure, will check up later today unless someone else wants to take care of this.

There was no interlock action, and I don't check the vacuum status every half hour, so there was a period of time last night there was high circulating power in the arm cavities when the main volume pressure was higher than nominal. I have now closed the PSL shutter until the issue is resolved.

Attachment 1: vacFailure.png
vacFailure.png
  15392   Thu Jun 11 16:14:03 2020 gautamUpdateVACVac failure - probable cause is serial comm glitch

Summary:

It looks like the main vacuum interlock was tripped due to a serial communication error from the TP2 controller. With Rana/Koji's permission, I will open V1 and expose the main volume to TP1 again (#2 in last section).

Details:

  • The vacuum interlock log file at /opt/target/vac.log on c1vac suggests that the interlock was tripped because "TP2 is too warm".
  • Looking back at the diagnostics channels, it looks like the TP2 temperature channel registered a rise in temperature of >30 C in <0.2 seconds, see Attachment #1 - seems highly unlikely, probably some kind of glitch in the serial communication? This particular pump is relatively new from Agilent (<2 years installed I think)
  • The PSL shutter was automatically closed at ~1150 am today, see Attachment #2. There is some EPICS logic on c1psl (Acromag server) that checks if C1:Vac-P1a_pressure is greater than 3 mTorr (or greater than 500 Torr for in-air locking of the IMC), in which case it closes the shutter, so this seems consistent with expectations.

Recommended course of action:

  1. Code in some averaging in the interlock code, so that the interlock isn't triggered on some unphysical glitch like this. As shown in Attachment #3, this has been happening for the past 24 hours (though not before, because the interlock wasn't tripped). Probably need the derivative of the temperature as well, and the derivative should be less than 5 C/s or something physical (in addition to the temperature being high) for the interlock to trip.
  2. Re-open V1 to pump down the main volume to nominal pressure so that the interferometer locking activity can resume.
    • One option in the interim is to bypass the TP2 temperature interlock condition.
    • The pressure-based interlocks are probably sufficient to protect the main volume / pumps during the nominal operations - the temperature interlocks are mainly useful during the pumpdown where the TPs have a large load, and so we want to avoid over-stressing them.
Attachment 1: TP2_tempGlitch.png
TP2_tempGlitch.png
Attachment 2: PSL_shutterClosed.png
PSL_shutterClosed.png
Attachment 3: TP2tempGlitches.pdf
TP2tempGlitches.pdf
  15393   Thu Jun 11 17:35:34 2020 gautamUpdateVACPumpspool UPS needs battery replacement

Summary:

The pumpspool UPS has its "Replace Battery" indicator light on. Might be a good chance to change the UPS, but at the very least, we should put in fresh batteries (last replacement was in Aug 2017).

I'll say this again - the pumpspool area is noisier than I remember it being, I think one / both of the roughing pumps backing TP2 / TP3 need tip-seal replacements.

BTW, EX is 5C hotter than EY, by virtue of the tarnac outside? In fact, judging by Steve's thermometers, EX reports a 12C swing in 24 hours between 30 C and 18 C (so almost no temperature control) while EY reports a 5C swing between 20 and 25 C. This is borne out by the ETM Oplev data I think...

  15394   Fri Jun 12 01:23:32 2020 KojiUpdateVACPumpspool UPS needs battery replacement

1. I agree that it's likely that it was the temp signal glitch.
Recom #2: I approve to reopen the valves to pump down the main volume. As long as there is no frequent glitch, we can just bring the vacuum back to normal with the current software setup.

2. Recom #1 is also reasonable. You can use simple logic like if we register 10 consecutive samples that exceed the threshold, we can activate the interlock. I feel we should still keep the temp interlock. Switching between pumping mode and the normal operation may cause unexpected omission of the interlocks when it is necessary.

3. We should purchase the UPS battery / replacement rotary TIP seal. Once they are in hand, we can stop the vacuum and execute the replacement. Can one person (who?) accomplish everything with some remote help?

4. The lab temp: you mean, 12degC swing with the AC on!?

 

  15395   Fri Jun 12 11:40:14 2020 JordanUpdateGeneralPresence at 40m

I will be in the Clean and Bake lab today from 12pm to 4pm.

  15396   Fri Jun 12 17:32:40 2020 KojiUpdateVACPumpspool UPS needs battery replacement

Jon and Koji remotely supported Jordan's resetting the TP2 controller.

Here is the instruction by Jon
From the operator's console in front of the vac rack:
  1. Open a terminal window (click the LXTerminal icon on the desktop)
  2. Type "control" + enter to open the vac controls screen
  3. Toggle all the open valves closed (edit by KA: and manually close RV2 by rotating the gate valve handle )
  4. Turn OFF TP2 by clicking the "Off' button. Make sure the status changes and the rotation speed falls to zero (you'll also hear the pump spinning down) 
  5. The other pumps (TP1, TP3) can be left running
  6. Once TP2 has stopped spinning, go to the back of the rack and locate the ethernet cable running from the back of the TP2 controller to the IOLAN server (near the top of the rack). Disconnect and reconnect the cable at each end, verifying it is firmly locked in place.
  7. From the front of the rack, power down the TP2 controller (I don't quite remember for the Agilent, but you might have to move the slider on the front from "Remote" to "Local" first)
  8. Wait about 30 seconds, then power it back on. If you had to move the slider to shut it down, revert it back to the "Remote" position.
  9. Go back to the controls screen on the console. If the pump came back up and is communicating serially again, its status will say something other than "NO COMM"
  10. Turn TP2 back on. Verify that it spins up to its nominal speed (66 kRPM)
  11. At this point you can reopen any valves you initially closed (any that were already closed before, leave closed)

TP2 was stopped and at this moment the glitches were gone. Jordan powercycled the TP2 controller and we brought up the TP2 back at the full speed.
However, the glitches came back as before. Obviously we can't go on from here, and we've decided to stop the recovery process here today.


- We left TP1/2/3 running while the valves including RV2 were closed.

- When Jordan is back in the lab next week, we'll try to use TP3 as the backing of TP1 so that we can resume the main volume pumping.

- Currently, TP3 does not have interlocking and that is a risk. Jon is going to implement it.

- Meanwhile, we will try to replace the controller of TP2. We are supposed to have this in the lab. Ask Chub about the location.

- Once we confirm the stability of the diagnostic signals for TP2, we will come back to the nominal pumping scheme.

Attachment 1: Screen_Shot_2020-06-12_at_17.22.23.png
Screen_Shot_2020-06-12_at_17.22.23.png
  15397   Fri Jun 12 19:02:52 2020 gautamUpdateVACPumpspool UPS needs battery replacement

I still don't understand why restoring the vacuum is contingent on this functionality working. All the TPs have their own internal logic to shutdown the pump if some damage threshold is exceeded. Plus, we have the pressure-sensor based interlocks to protect the main volume as well as pumps. While the extra redundancy from the readbacks from the controller is useful, clearly it isn't the first line of defense?

The main volume pressure is currently ~10mTorr. If we pump down before this reaches 500mTorr, the procedure is pretty straightforward. Otherwise, we have to do the dance with the manual throttling valve (judging by current rate of increase, unlikely to exceed this over the weekend, but I lose IFO time).

Obviously I don't want to rush this and have some permanent damage, so I'll stay out of this unless otherwise instructed.

  15398   Fri Jun 12 19:23:56 2020 KojiUpdateVACPumpspool UPS needs battery replacement

The vacuum safety policy and design are not clear to me, and I don't know what the first and second defense is. Since we had limited time and bandwidth during the remotely-supported recovery work today, we wanted to work step by step.

The pressure rising rate is 20mtorr/day, and turning on TP3 early next week will resume the main-volume pumping without too much hustle. If you need the IFO time now, contact with Jon and use backing with TP3.

  15399   Fri Jun 12 19:33:31 2020 gautamUpdateVACPumpspool UPS needs battery replacement

Didn't mean to sound whiny. I will wait until the vacuum team tells me it is okay.

Quote:

The vacuum safety policy and design are not clear to me, and I don't know what the first and second defense is. Since we had limited time and bandwidth during the remotely-supported recovery work today, we wanted to work step by step.

The pressure rising rate is 20mtorr/day, and turning on TP3 early next week will resume the main-volume pumping without too much hustle. If you need the IFO time now, contact with Jon and use backing with TP3.

  15400   Tue Jun 16 08:58:11 2020 JordanUpdateGeneralPresence at 40m

I will be at the 40m today at 10am to deliver optics to Downs and to replace the TP2 controller.

  15401   Tue Jun 16 13:05:36 2020 KojiUpdateCOCITM spares and New PR3 mirrors transported to Downs for phasemap measurement

ITMU01 / ITMU02 as well as the five E1800089 mirrors came back to the 40m. Instead, the two ETM spares (ETMU06 / ETMU08) were delivered to GariLynn.
Jordan worked on transportation.

Note that the E1800089 mirrors are together with the ITM container in the precious optics cabinet.

Attachment 1: 40m_Optics.jpg
40m_Optics.jpg
  15402   Tue Jun 16 13:35:03 2020 JonUpdateVACTemporary vac fix / IFO usable again

[Jon, Jordan, Koji]

Today Jordan reconfigured the vac system to allow pumping of the main volume resume, with Jon and Koji remotely advising. All clear to resume normal IFO activities. However, the vac system is operating in a temporary configuration that will have to be reverted as we locate replacement components. Details below.

Procedure

Since serial readback of the TP2 controller seems to be failing, we reconfigured the system with TP3 now backing for TP1. TP2 was valved off (at V4) and shut down until we can replace its controller.

TP3 has its own problems, however. It was valved off in January after its temperature readback began glitching and spuriously triggering the interlocks [ELOG 15140]. However the problem appears to be limited only one readback (rotation speed, current, voltage are fine) and there is enough redundancy in the pump-dependent interlock conditions to safely connect it to the main volume.

We also discovered that sometime since January, the TP3 dry pump has failed. The foreline pressure had risen to 165 torr. Since the TP2 and TP3 dry pumps are not interchangeable (Agilent vs. Varian), we instead valved in the auxiliary dry pump and disconnected the failed dry pump using a KF blank. This is a temporary arrangement until the permanent dry pump can be repaired. Jordan removed it to replace the tip seals and will test it in the bake lab before reinstalling.

With this configuration in place, we proceeded to pump down the main volume without issue (attachment 1). We monitored the pumpdown for about 45 min., until the pressure had reached ~1E-5 torr and TP3 had been transitioned to standby (low-speed) mode.

Summary of topology changes:

  • TP2 valved off and shut down until controller can be replaced
  • TP3 temporarily backing for TP1
  • Auxiliary dry pump temporarily backing for TP3
  • TP3 dry pump has been removed for repairs
Attachment 1: Pumpdown.png
Pumpdown.png
  15403   Tue Jun 16 16:05:26 2020 JordanUpdateGeneralN2 Replacement

I replaced an empty N2 cylinder, there are now two empty tanks in the outside rack.

  15404   Wed Jun 17 16:27:51 2020 gautamUpdateVACQuestions/comments on vacuum

I missed the vacuum discussion on the call today, but I have some questions/comments:

  • Isn’t it true that we didn’t digitally monitor any of the TP diagnostic channels before 2018 December? I don’t have the full history but certainly there wasn’t any failure of the vacuum system connected to pump current/temp/speed from Sep 2015-Dec2018, whereas we have had 2 interruptions in 6 months because of flaky serial communications.
  • According to the manuals, the turbo-pumps have their own internal logic to shut off the pump when either bearing temperature exceeds 60C or current exceeds 1.5A. I agree its good to have some redundancy, but do we really expect that our outer interlock loops will function if the internal ones fail?
  • In what scenario do we expect that all our pressure gauge readbacks fail, but not the TP readbacks? If so, won’t the differential pressure conditions protect the vacuum envelope, and the TPs internal shutoffs will protect the pumps? Except during the pump down phase perhaps, when we want to give a little more headroom to the small TPs to stress them less?

At the very least, I think we should consider making the interlock code have levels (like interrupts on a micro controller). So if the pressure gauges are communicating and are reporting acceptable pressure readings, we should be able to reject unphysical readbacks from the TP controllers.

I still don’t understand why TP2 can’t back TP1, but we just disable all the software interlock conditions contingent on TP2 readbacks. This pump is far newer than TP3, and unless I’ve misunderstood something major about the vacuum infrastructure, I don’t really see why we should trust this flaky serial readbacks for any actionable interlocks, at least without some AND logic (since temperature, current and speed aren’t really independent variables).

I also think we should finally implement the email alert in the event the vacuum interlock is tripped. I can implement this if no one else volunteers.

This might also be a good reminder to get the documentation in order about the new vacuum system.

  15405   Thu Jun 18 09:46:03 2020 JordanUpdateGeneralPresence at 40m

I will be at the 40m today from 9:30am to 4pm.

  15406   Thu Jun 18 11:00:24 2020 JonUpdateVACQuestions/comments on vacuum
Quote:
  • Isn’t it true that we didn’t digitally monitor any of the TP diagnostic channels before 2018 December? I don’t have the full history but certainly there wasn’t any failure of the vacuum system connected to pump current/temp/speed from Sep 2015-Dec2018, whereas we have had 2 interruptions in 6 months because of flaky serial communications.

Looking at images of the old vac screens, the TP2/3 rotation speed and status string were digitally monitored. However I don't know if there were software interlocks predicated on those.

Quote:
  • According to the manuals, the turbo-pumps have their own internal logic to shut off the pump when either bearing temperature exceeds 60C or current exceeds 1.5A. I agree its good to have some redundancy, but do we really expect that our outer interlock loops will function if the internal ones fail?

The temperature and current interlocks are implemented precisely because the pumps can shut themselves off. The concern is not about damaging the pumps (their internal logic protects against that); it's that a pump could automatically shut down and back-vent the IFO to atmosphere. Another interlock (e.g., the pressure differentials) might catch it, but it would depend on the back-vent rate and the scenario has never been tested. The temperature and current interlocks are set to trip just before the pump reaches its internal shut-down threshold.

One way we might be able to reduce our reliance on the flaky serial readbacks is to implement rotation-speed hardware interlocks. The old vac documentation alludes to these, but as far as Chub and I could determine in 2018, they never actually existed. The older turbo controllers, at least, had an analog output proportional to speed which could be used to control a relay to interrupt the V4/5 control signals. I'll look into this for the new controllers. If it could be done, we could likely eliminate the layer of serial-readback interlocks altogether.

 
  • I also think we should finally implement the email alert in the event the vacuum interlock is tripped. I can implement this if no one else volunteers.

That would be awesome if you're willing to volunteer. I agree this would be great to have.

  15407   Thu Jun 18 12:00:36 2020 gautamUpdateVACQuestions/comments on vacuum

I agree there were MEDM fields, but I can't find any record of these channels being recorded till 2018 December, so I don't agree that they were being digitally monitored. You can also look back in the elog (e.g. here and here) and see that the display fields are just blank. I would then assume that no interlocks were dependent on these channels, because otherwise the vacuum interlocks would be perpetually tripped.

Looking at images of the old vac screens, the TP2/3 rotation speed and status string were digitally monitored. However I don't know if there were software interlocks predicated on those.

Sorry but I'm having trouble imagining a scenario how the pressure gauges wouldn't register this before the IFO volume is compromised. Is there some back of the envelope calculations I can do to understand this? Since both the pressure gauges and the TP diagnostic channels are being monitored via EPICS, the refresh rate is similar, so I don't see how we can have a pump temperature / speed / current threshold tripped but NOT have this be registered on all the pressure gauges, seems like a bit of a contrived scenario to me. Our thresholds currently seem to be arbitrary numbers anyway, or are they based on some expected backstreaming rate? Isn't this scenario degenerate with a leak elsewhere in the vacuum envelope that would be caught by the differential pressure interlocks?

The temperature and current interlocks are implemented precisely because the pumps can shut themselves off. The concern is not about damaging the pumps (their internal logic protects against that); it's that a pump could automatically shut down and back-vent the IFO to atmosphere. Another interlock (e.g., the pressure differentials) might catch it, but it would depend on the back-vent rate and the scenario has never been tested. The temperature and current interlocks are set to trip just before the pump reaches its internal shut-down threshold.

For the email alert, can you expose a soft channel that is a flag - if this flag is not 1, then the service will send out an email.

That would be awesome if you're willing to volunteer. I agree this would be great to have.
  15408   Thu Jun 18 14:13:03 2020 JonUpdateVACQuestions/comments on vacuum
I agree there were MEDM fields, but I can't find any record of these channels being recorded till 2018 December, so I don't agree that they were being digitally monitored. You can also look back in the elog (e.g. here and here) and see that the display fields are just blank. I would then assume that no interlocks were dependent on these channels, because otherwise the vacuum interlocks would be perpetually tripped.

Right, I doubt they were ever recorded or used for interlocks. But the readbacks did work at one point in the past. There's a photo of the old vac monitor screen on p. 19 of E1500239 (last updated 2017) which shows the fields once alive.

Sorry but I'm having trouble imagining a scenario how the pressure gauges wouldn't register this before the IFO volume is compromised. Is there some back of the envelope calculations I can do to understand this? Since both the pressure gauges and the TP diagnostic channels are being monitored via EPICS, the refresh rate is similar, so I don't see how we can have a pump temperature / speed / current threshold tripped but NOT have this be registered on all the pressure gauges, seems like a bit of a contrived scenario to me. Our thresholds currently seem to be arbitrary numbers anyway, or are they based on some expected backstreaming rate? Isn't this scenario degenerate with a leak elsewhere in the vacuum envelope that would be caught by the differential pressure interlocks?​

I don't disagree that the pressure gauges would register the change. What I'm not sure about is whether the change would violate any of the existing interlock conditions, triggering a shutdown. Looking at what we have now, the only non-pump-related conditions I see that might catch it are the diffpres conditions:

  • abs(P2 - PTP2) > 1 torr (for a TP2 failure)

  • abs(P3 - PTP3) > 1 torr (for a TP3 failure)

  • abs(P1a - P2) > 1 torr (for either a TP2 or TP3 failure)

For the P1a-P2 differential, the threshold of 1 torr is the smallest value that in practice still allows us to pump down the IFO without having to disable the interlocks (P1a-P2 is the TP1 intake/exhaust differential). The purpose of the P2-PTP2/P3-PTP3 differentials is to prevent V4/5 from opening and suddenly exposing the spinning turbo to high pressure. I'm not aware of a real damage threshold calculation that any one has done; I think < 1 torr is lore passed down by Steve.

If a turbo pump fails, the rate it would backstream is unknown (to me, at least) and likely depends on the failure mode. The scenario I'm concerned about is if the backstream rate is slower than the conduction time through the pumspool and into the main volume. In that case, the pressure gauges will rise more or less together all the way up to atmosphere, likely never crossing the 1 torr differential pressure thresholds.

For the email alert, can you expose a soft channel that is a flag - if this flag is not 1, then the service will send out an email.

There's already a channel C1:Vac-error_status, where if the value is anything other than an empty string, there is an interlock tripped. Does that work?

  15409   Thu Jun 18 15:25:08 2020 JordanUpdateVACTP2 and TP3 Forepump removal

I removed the backing pumps for TP2 and TP3 today to test ultimate pressure and determine if they need a tip seal replacement. This was done with Jon backing me on Zoom. We closed off TP3 and powered down TP3 and the auxilliary pump, in order to remove the forepumps from the exhaust line.

  1. Close V1
  2. Close V5
  3. Turn off TP3
  4. Turn off aux dry pump (manually)
  5. Once the PTP3 foreline pressure has come up to atmosphere, you can disconnect the TP3 dry pump and cap the exhaust line with a KF blank.
  6. Restore the vac configuration in reverse order: dry pump ON, TP3 ON, open V5, open V1

Once pumps were removed I connected a Pirani gauge to the pump directly and pumped down, results as follows:

TP2 Forepump (Agilent IDP 7):

  • Ultimate Pressure: 123 mtorr
  • Hours: 10903

TP3 Forepump (Varian SH 110):

  • Ultimate pressure: ~70 torr
  • Hours: 60300

TP3 forepump defintely needs a new tip seal, and while the pressure on TP2 Forepump was good there was a significant amount of particulate that came out of the exhaust line, so a new tip seal might not be needed but it is recommeded.

  15410   Thu Jun 18 15:46:34 2020 gautamUpdateVACQuestions/comments on vacuum

So why not just have a special mode for the interlock code during pumpdown and venting, and during normal operation we expect the main volume pressure to be <100uTorr so the interlock trips if this condition is violated? These can just be EPICS buttons on the Vac control MEDM screen. Both of these procedures are not "business as usual", and even if we script them in the future, it's likely to have some operator supervising, so I don't think it's unreasonable to have to switch between these modes. I just think the pressure gauges have demonstrated themselves to be much more reliable than these TP serial readbacks (as you say, they worked once upon a time, but that is already evidence of its flakiness?). The Pirani gauges are not ultra-reliable, they have failed in the past, but at least less frequently than this serial comm glitching. In fact, if these readbacks are so flaky, it's not impossible that they don't signal a TP shutdown? I just think the real power of having these multi-channel diagnostics is lost without some AND logic - a turbopump failure is likely to result in an increase in pump current and temperature increase and pump speed decrease, so it's not the individual channel values that should be determining if an interlock is tripped.

I definitely think that protecting the vacuum envelope is a priority - but I don't think it should be at the expense of commissioning time. But if you think these extra interlocks are essential to the safety of the vacuum system, I withdraw my request.

I don't disagree that the pressure gauges would register the change. What I'm not sure about is whether the change would violate any of the existing interlock conditions, triggering a shutdown. Looking at what we have now, the only non-pump-related conditions I see that might catch it are the diffpres conditions:

It would be better to have a flag channel, might be useful for the summary pages too. I will make it if it is too much trouble.

There's already a channel C1:Vac-error_status, where if the value is anything other than an empty string, there is an interlock tripped. Does that work?
  15411   Thu Jun 18 16:56:34 2020 JordanUpdateVACTP2 and TP3 Forepump removal
Quote:

I removed the backing pumps for TP2 and TP3 today to test ultimate pressure and determine if they need a tip seal replacement. This was done with Jon backing me on Zoom. We closed off TP3 and powered down TP3 and the auxilliary pump, in order to remove the forepumps from the exhaust line.

  1. Close V1
  2. Close V5
  3. Turn off TP3
  4. Turn off aux dry pump (manually)
  5. Once the PTP3 foreline pressure has come up to atmosphere, you can disconnect the TP3 dry pump and cap the exhaust line with a KF blank.
  6. Restore the vac configuration in reverse order: dry pump ON, TP3 ON, open V5, open V1

Once pumps were removed I connected a Pirani gauge to the pump directly and pumped down, results as follows:

TP2 Forepump (Agilent IDP 7):

  • Ultimate Pressure: 123 mtorr
  • Hours: 10903

TP3 Forepump (Varian SH 110):

  • Ultimate pressure: ~70 torr
  • Hours: 60300

TP3 forepump defintely needs a new tip seal, and while the pressure on TP2 Forepump was good there was a significant amount of particulate that came out of the exhaust line, so a new tip seal might not be needed but it is recommeded.

I agree with your assessment, Jordan.  If I'm not mistaken the scroll pump for TP2 is new; we had a very early failure with the last new scroll pump (the forepump for TP3) tip seals at just over 5000 hours.  Glad to see my replacement seals held up for over 60K hours. If this is the trend with these pumps, we can simply run them to  around 60000 hours and replace the seals at that time, rather than waiting for failure! - Chub

  15413   Fri Jun 19 07:40:49 2020 JonUpdateVACQuestions/comments on vacuum

I think we should discuss interlock possibilities at a 40m meeting. I'm reluctant to make the system more complicated, but perhaps we can find ways to reduce the reliance on the turbo pump readbacks. I agree they've proven to be the least reliable.

While we may be able to improve the tolerance to certain kinds of hardware malfunctions (and if so, we should), I don't see interlocks triggering on abnormal behavior of critical equipment as the root problem. As I see it, our bigger problem is with all the malfunctioning, mostly end-of-lifetime pieces of vacuum equipment still in use. If we can address the hardware problems, as I'm trying to do with replacements [ELOG 15412], I think that in itself will make the interlocking much less of an issue.

Quote:

So why not just have a special mode for the interlock code during pumpdown and venting, and during normal operation we expect the main volume pressure to be <100uTorr so the interlock trips if this condition is violated? These can just be EPICS buttons on the Vac control MEDM screen. Both of these procedures are not "business as usual", and even if we script them in the future, it's likely to have some operator supervising, so I don't think it's unreasonable to have to switch between these modes. I just think the pressure gauges have demonstrated themselves to be much more reliable than these TP serial readbacks (as you say, they worked once upon a time, but that is already evidence of its flakiness?). The Pirani gauges are not ultra-reliable, they have failed in the past, but at least less frequently than this serial comm glitching. In fact, if these readbacks are so flaky, it's not impossible that they don't signal a TP shutdown? I just think the real power of having these multi-channel diagnostics is lost without some AND logic - a turbopump failure is likely to result in an increase in pump current and temperature increase and pump speed decrease, so it's not the individual channel values that should be determining if an interlock is tripped.

Ok, this can be added pretty easily. Its value will just be toggled between 1 and 0 every time the interlock server raises/clears the existing string channel. Adding the channel will require restarting the whole vac IOC, so I'll do it at a time when Jordan is on hand in case something fails to come back up.

Quote:

It would be better to have a flag channel, might be useful for the summary pages too. I will make it if it is too much trouble.

  15414   Fri Jun 19 08:47:10 2020 JordanUpdateGeneralPresence at 40m

I will be at the 40m today from 9am to 3pm.

  15415   Fri Jun 19 09:57:35 2020 gautamUpdateVACQuestions/comments on vacuum

For this particular email service, ideally the email should be sent out as soon as the interlock is tripped, so this would require a line of code to be added to the main interlock code. Which I guess would require a restart of the interlock service. So let me know when you guys plan to do the dry-pump tip seal replacement operation (when I presume valves will be closed anyways) so that we can do this in a minimally invasive way.

Quote:

Ok, this can be added pretty easily. Its value will just be toggled between 1 and 0 every time the interlock server raises/clears the existing string channel. Adding the channel will require restarting the whole vac IOC, so I'll do it at a time when Jordan is on hand in case something fails to come back up.

ELOG V3.1.3-