40m QIL Cryo_Lab CTN SUS_Lab TCS_Lab OMC_Lab CRIME_Lab FEA ENG_Labs OptContFac Mariner WBEEShop
  40m Log, Page 227 of 339  Not logged in ELOG logo
ID Date Authorup Type Category Subject
  15364   Wed Jun 3 01:34:53 2020 gautamUpdateLSCLock acquisition update portal

Highlights:

  • With better ASC servos implemented, I think the lock stability has been improved. 
  • Arm transmission of ~350 was stably maintained (PRG~20, overcoupled). It went as high as 410, so this is now very close to the highest (~425) I've ever managed to get.
  • I was trying to get the vertex transitioned to 1f control but it remains out of reach for now.  The noise at ~100 Hz is dominated by MICH-->DARM coupling (as judged by coherence, I haven't yet done the broadband noise injection characterization). I figured I'd try the 1f transition before thinking about feedforward.
  • The biggest problems remain flaky electronics (poor IMC duty cycle, jumping RF offsets, newly glitchy seismometer, ...)

Details:

  15365   Wed Jun 3 01:40:13 2020 gautamUpdateElectronicsMore electronics woes

There were many locklosses from the point where the arm powers were somewhat stabilized. Attachments #1 and #2 show two individual locklosses. I think what is happening here is that the BS seismometer X channel is glitching, and creating a transient in the angular feedforward filter that blows the lock. The POP QPD based feedback loop cannot suppress this transient, apparently. For now, I get around this problem by boosting the POP QPD feedback loop a bit, and then turning the feedforward filters off. The fact that the other seismometer channels don't report any transient makes me think the problem is either with the seismometer itself, or the readout electronics. The seismometer masses were recently recentered, so I'm leaning towards the latter.

I didn't explicitly check the data, but I am reasonably certain the same effect is responsible for many PRMI locklosses even with the arms held off resonance (though the tolerance to excursions there is higher). Pity really, the feedforward filters were a big help in the lock acquisition...

Attachment 1: glitchySeis2.png
glitchySeis2.png
Attachment 2: glitchySeis3.png
glitchySeis3.png
  15366   Wed Jun 3 01:46:14 2020 gautamUpdateLSCCARM loop

Summary:

The CARM loop now has a UGF of ~12 kHz with a phase margin of ~60 degrees. These values of conventional stability indicators are good. The CARM optical gain that best fits the measurements is 9 MW/m.

I've been working on understanding the loop better, here are the notes.

Details:

Attachment #1 shows a block diagram of the loop topology.

  • The "crossover" measurement made at the digital CARM error point, and the OLG measurement at the CM board error point are shown.
  • I've tried to include all the pieces in the loop, and yet, I had to introduce a fudge gain in the digital path to get the model to line up with the measurement (see below).

Attachment #2 shows the OLGs of the two actuation paths.

  • Aforementioned fudge factor for the digital path is included.
  • For the AO path, I assumed a value of the PDH discriminant at the IMC error point to be 13 kHz/V, per my earlier measurement. 
  • I trawled the elog for the most up-to-date info about the IMC servo (elog9457, elog13696, elog15044) and CM board, to build up the model. 

Attachment #3 and #4 show the model, overlaid with measurements of the loop OLG and crossover TF respectively.

  • No fitting is done yet - the next step would be to add the delay of the CDS system for the digital path, and the analog electronics for the AO path. Though these are likely only small corrections.
  • For the crossover TF - I've divided out the digital filters in the CARM_B filter bank, because the injection is made downstream of it (see Attachment #1).
  • There is reasonably good agreement between model and measurement.
  • I think the biggest source of error is the assumed model for the IMC OLTF.

Attachment #5 shows the evolution of the CARM OLG at a few points in the lock acquisition sequence.

  • "Before handoff" corresponds to the state where the primary control is still done by the ALS leg, but the REFL11 signal has begun to enter the picture via the CARM_B path.
  • "IN2 ramped" corresponds to the state where the AO path gain (=IN2 gain on the IMC servo board) has been ramped up to its final value (+0 dB), but the overall loop gain (=IN1 gain on the CM board) is still low. So this is preparation for high bandwidth control. Typically, the arm powers will have stabilized in this state, but ALS control is still on.
  • "Pre-boost" corresponds to an intermediate state - ALS control is off, but the low frequency boosts have not yet been enabled. I typically first engage some ASC to stabilize things somewhat, and then turn on the boosts.
  • "Final" - self explanatory.

Next steps:

Now the I have a model I believe, I need to think about whether there is any benefit to changing some of these loop shapes. I've already raised the possibility of changing the shape of the boosts on the CM board, with which we could get a bit more suppression in the 100 Hz - 1kHz region (noise budget of laser frequency noise --> DARM required to see if this is necessary). 

Attachment 1: CM_loop_topology.pdf
CM_loop_topology.pdf
Attachment 2: CARM_TFs.pdf
CARM_TFs.pdf
Attachment 3: CARM_OLTF.pdf
CARM_OLTF.pdf
Attachment 4: CARM_xover.pdf
CARM_xover.pdf
Attachment 5: CARM_OLG_evolution.pdf
CARM_OLG_evolution.pdf
  15367   Wed Jun 3 02:08:00 2020 gautamUpdateLSCPower buildup diagnostics

Attachments #1 and Attachments #2 are in the style of elog15356, but with data from a more recent lock. It'd be nice to calibrate the ASDC channel (and in general all channels) into power units, so we have an estimate of how much sideband power we expect, and the rest can be attributed to carrier leakage to ASDC.

On the basis of Attachments #1, the PRG is ~19, and at times, the arm transmission goes even higher. I'd say we are now in the regime where the uncertainty of the losses in the recycling cavity - maybe beamsplitter clipping? is important in using this info to try and constrain the arm cavity losses. I'm also not sure what to make of the asymmetry between TRX and TRY. Allegedly, the Y arm is supposed to be lossier.

Quote:

This is very interesting. Do you have the ASDC vs PRG (~ TRXor TRY) plot? That gives you insight on what is the cause of the low recycling gain.

Attachment 1: PRFPMIcorner_DC_1275190251_1275190551.pdf
PRFPMIcorner_DC_1275190251_1275190551.pdf
Attachment 2: PRFPMIcorner_SB_1275190251_1275190551.pdf
PRFPMIcorner_SB_1275190251_1275190551.pdf
  15368   Wed Jun 3 02:14:32 2020 gautamUpdateASCPRC ASC improves arm transmission RIN

Summary:

I implemented an ASC servo for the PRC, with the POP QPD as a sensor, and the PRM as the actuator. This has improved the stability of the lock (longer locks are possible), and also reduced the RIN of the arm transmission.

Details:

Attachment #1 shows the in-loop error signal suppression, and some out-of-loop monitors (POP22 and POPDC).

  • To practise and get some workable servo settings, I locked the PRMI with carrier resonant (no ETMs).
  • Then, I compare the beam motion witnessed by the POP QPD with and without the feedback loop enabled.
  • I also look at the spectra of the POPDC and POP22 signals, as out-of-loop proxies, to get an estimate of how much noise is being injected out of band.
  • In this toy study, both the in-loop and out of loop monitors show good performance.
  • However, when repeating the same diagnostics with the PRFPMI locked, I note that while the in-loop suppression looks good, POPDC and POP22 report elevated noise, relative to the PRMI carrier case.
  • I don't have a comparison to the PRFPMI locked with the feedback disabled, because of stability reasons. Plus, for the PRMI, the angular feedforward loops were engaged, but for the PRFPMI traces, they were disabled.
  • Nevertheless, the arm RIN goes down by ~2.5 in RMS, so this is doing something good.

Attachment #2 compares the arm transmission RIN with the PRFPMI locked, with and without PRC ASC. The 3 Hz bump is definitely squished, but I think we can do better yet. 

Attachments #3-5 are in the style of elog15361. No Oplev signals yet, I'll add them soon.

I guess what this means is that the stability of the lock could be improved by turning on some POP QPD based feedback control, I'll give it a shot

Attachment 1: PRC_ASCsignals.pdf
PRC_ASCsignals.pdf
Attachment 2: armRIN_PRC_ASC.pdf
armRIN_PRC_ASC.pdf
Attachment 3: PRFPMIcorner_ASC_PIT_1275190251_1275190551.pdf
PRFPMIcorner_ASC_PIT_1275190251_1275190551.pdf
Attachment 4: PRFPMIcorner_ASC_YAW_1275190251_1275190551.pdf
PRFPMIcorner_ASC_YAW_1275190251_1275190551.pdf
Attachment 5: PRFPMIcorner_ASC_coherence_1275190251_1275190551.pdf
PRFPMIcorner_ASC_coherence_1275190251_1275190551.pdf
  15370   Wed Jun 3 11:20:19 2020 gautamUpdateDetCharSummary pages

Summary:

The 40m summary pages have been revived. I've not had to make any manual interventions in the last 5 days, so things seem somewhat stable, but I'm sure there will need to be multiple tweaks made. The primary use of the pages right now are for vacuum, seismic and PSL diagnostics.

Resources:

Caveats:

  • Intermittent failures of cron jobs
    • The status page relies on the condor_q command executing successfully on the cluster end. I have seen this fail a few times, so the status page may say the pages are dead whereas they're actually still running.
    • Similarly, the rsync of the pages to nodus where they're hosted can sometimes fail.
    • Usually, these problems are fixed on the next iteration of the respective cron jobs, so check back in ~half hour.
  • I haven't really looked into it in detail, but I think our existing C1:IFO-STATE word format is not compatible with what gwsumm wants - I think it expects single bits that are either 0 or 1 to indicate a particular state (e.g. MC locked, POX and POY locked etc). So if we want to take advantage of that infrastructure, we may need to add a few soft EPICS channels that take care of some logic checking (several such bits could also be and-ed together) and then assume either 0 or 1 value. Then we can have the nice duty cycle plots for the IMC (for example).
  • I commented out the obsolete channels (e.g. PEM MIC channels). We can add them back later if we so desire.
  • For some reason, the jobs that download the data are pretty memory-heavy: I have to request for machines on the cluster with >100 GB (yes 💯GB) ! of memory for the jobs to execute to completion. The frame-data certainly isn't so large, so I wonder what's going on here - is GWPy/GWsumm so heavy? The site summary pages run on a dedicated cluster, so probably the code isn't built for efficiency...
  • Weather tab in PEM is still in there but none of those channels mean anything right now.
  • The MEDM screenshot stuff is commented out for now too. This should be redone in a better way with some modern screen grab utilities, I'm sure there are plenty of python based ones.
  • There seems to be a problem with the condor .dag lockfile / rescue file not being cleared correctly sometimes - I am looking into this.
  15371   Wed Jun 3 11:40:56 2020 gautamUpdateLSCLock acquisition update portal

For these initial attempts, I was just trying to transition MICH to REFL55Q. I agree, the PRCL situation may be more complicated.

Which 1f signals are you going to use? PRCL has sign flipping at the carrier critical coupling. So if the IFO is close to that condition, 1f PRCL suffers from the sign flipping or large gain variation.

  15372   Wed Jun 3 18:49:47 2020 gautamUpdateLSCPRG and CARM signal sign

Summary:

I am inclined to believe that the arm cavity losses are such that the IFO is overcoupled. Some calculations, validated with Finesse modeling also suggest that there isn't a sign change for the CARM error signal when the IFO goes from being undercoupled to overcoupled, but I may have made a mistake here?

Details:

  • We’d like to gain some insight into whether the interferometer is undercoupled, critically coupled, or overcoupled. Factors that determine which of these is true include:
    • Arm cavity losses
    • Recycling cavity losses
  • The proxy by which we determine the recycling gain is usually the arm cavity transmission. Assuming T_PRM = 5.637 % according to the wiki, and assuming the arm cavity transmission is normalized to 1 when locked in the POX/POY state, we can say that the PRG is given by G_PRC = TRX × T_PRM, assuming that the (i) the RF sideband fields are perfectly rejected by the arm cavities and (ii) mode-matching efficiency between the input beam and the arm mode is the same as that between the input beam and the CARM mode.
  • Apart from this, the other measurement we have available to us is the buildup of the sideband fields, namely POP22 and POP110. We can compare the values in the PRMI lock vs the PRFPMI to make some inference.
  • I started off with an analytic calculation of the reflectivity of the compound arm cavity mirror.
    • Attachment #1 suggests we will have an over-coupled IFO for arm cavity losses below ~200 ppm, which is a regime we are almost certainly in now.
  • Then, I repeat the analysis for the coupled CARM cavity, with the end mirror as the compound arm mirror and the input mirror as the PRM.
    • I assume 2 % loss in the PRC.
    • Attachment #2 shows that while the carrier field goes through a sign change in amplitude reflectivity (as expected), the sideband fields dont.
    • Per equation 4.2 of Koji's thesis, the error signal for CARM depends on the (signed) IFO reflectivity, and the absolute value of the derivative of the arm cavity reflectivity for the carrier w.r.t. CARM phase.
    • So, we don't expect the REFL11 signal to show a sign change.
    • The situation is more complicated for PRCL in REFL11, because as explicitly evaluated in Eq 4.3 of Koji's thesis, there are two terms that contribute, and their relative magnitudes will dictate the overall sign. 
  • For a Finesse validation, I use a simplified 3 mirror coupled cavity to approximate the PRFPMI. I also retained the RF sidebands for diagnostic purposes. The idea was to study these PRG proxies and what their expected behavior is.
    • Attachment #3 shows the PDH error signal in the (arbitrarily defined) REFL11 I quadrature. While the optical gain changes as a function of the arm cavity loss, the actual slope does not change sign. The fact that the zero crossing doesn't happen at exactly 0 CARM offset is because of higher order mode light at the REFL port (in my model, I tried to preserve the flipped folding mirror situation so the mode matching between the arm cavity and PRC in my model is ~96%).
    • In fact, this may explain why a CARM_B offset is required to do the ALS-->IR handoff - the ALS servo wants to keep the arm offset to zero, but at that point, the PDH error signal isn't zero, and so the two loops end up fighting each other?
    • Attachment #4 is a more detailed study of the recycling gain as a function of arm cavity loss, but now including losses in the recycling cavity.

Conclusions:

  1. I think the arm cavity losses are in the 60-80 ppm round-trip region. I don't see how we can explain the arm cavity transmission of ~350 otherwise.
  2. The fact that REFLDC decreases as the arm transmission increases is because the input beam is getting better matched to the CARM mode, and there is less junk carrier light. 

Thoughts from others?

Attachment 1: armCavReflectivities.pdf
armCavReflectivities.pdf
Attachment 2: IFOreflectivities.pdf
IFOreflectivities.pdf
Attachment 3: PDHerrSigs.pdf
PDHerrSigs.pdf
Attachment 4: PRGvsLoss_finesse.pdf
PRGvsLoss_finesse.pdf
  15373   Wed Jun 3 19:19:11 2020 gautamUpdateSUSAll watchdogs tripped

This EQ seems to have knocked all suspensions out. ITMX was stuck. It is now released, and the IMC is locked again. It looks like there are some serious aftershocks going on so let's keep an eye on things.

  15376   Thu Jun 4 20:54:40 2020 gautamUpdateSUSMC1 Slow Bias issues

Summary:

I found that there is an issue with the MC1 slow bias voltages. 

Details:

I usually offload the DC part of the output voltage from the WFS servos to the slow bias voltage sliders, so as to preserve maximum actuation range from the fast system. However, today, I found that this servo wasn't working well at all. So I dug a little deeper. Looking at the EPICS database records:

  • The user-facing channels are "PIT" and "YAW" bias voltages.
  • These are converted to voltages to be sent to individual coils by some calc channels in the EPICS database record. So, for example, the voltage to be sent to the "UL" coil (Upper Left, as viewed from the AR side of the optic), is A+B, where A is the "PIT" voltage and B is the "YAW" voltage. Similar combinations of A and B are used for the other 3 face coils.
  • The problem is obvious - if either A or B > 5V, then the requested voltage to be sent to the UL coil is > 10 V, while the Acromag DACs can put out a maximum of 10 V
  • As it happens, with the IFO currently aligned, MC1 is the only optic which faces this problem. 
  • Why has this not been an issue before? In fact, looking at some old data, the "PIT" and "YAW" bias voltages to MC1 were both ~1-2 V in 2018. But I confirmed that something in the region of ~5 V is required from each of the "PIT" and "YAW" channels to bring the MCREFL spot back to the center of the camera, so something has changed the DC alignment of MC1, maybe an earthquake or something? Anyway, with these settings, 2/4 coils are basically saturated, and so we can only move the optic diagonally. 😢 
  • Other coils that have  requested output voltages > 5V (so more than half the range of the DAC) include MC2 LL (5.2V), and ETMX LL and LR (5.5 and 5.8 V respectively).
  • Either a factor of 0.5 should be included in all the EPICS database records, or else, we should make the "PIT" and "YAW" sliders range only from -5 to +5 V, so that this kind of misleading info isn't wasting time.
  15383   Mon Jun 8 18:14:55 2020 gautamUpdateCDSVertex FEs crashed

Summary:

Around 5pm local time, the three vertex FEs crashed. AFAIK, no one was in the lab or working on anything CDS related, so this is worrying.

Details:

  • Reboot script was used to bring all FEs back - only soft reboots were required.
  • The IMC and arms can now be locked.
  • I think combination of burt + SDF would have reverted all the settings as they should be, but if something appears off, it could be that some EPICS value didn't get reset correctly.
Attachment 1: FEcrash_CDSoverview.png
FEcrash_CDSoverview.png
  15391   Thu Jun 11 11:48:43 2020 gautamUpdateVACVac failure

There appears to have been some sort of vacuum failure.

ldas-pcdev1 was down, so the summary pages weren't being generated. I have now switched over to ldas-pcdev6. I suspect some forepump failure, will check up later today unless someone else wants to take care of this.

There was no interlock action, and I don't check the vacuum status every half hour, so there was a period of time last night there was high circulating power in the arm cavities when the main volume pressure was higher than nominal. I have now closed the PSL shutter until the issue is resolved.

Attachment 1: vacFailure.png
vacFailure.png
  15392   Thu Jun 11 16:14:03 2020 gautamUpdateVACVac failure - probable cause is serial comm glitch

Summary:

It looks like the main vacuum interlock was tripped due to a serial communication error from the TP2 controller. With Rana/Koji's permission, I will open V1 and expose the main volume to TP1 again (#2 in last section).

Details:

  • The vacuum interlock log file at /opt/target/vac.log on c1vac suggests that the interlock was tripped because "TP2 is too warm".
  • Looking back at the diagnostics channels, it looks like the TP2 temperature channel registered a rise in temperature of >30 C in <0.2 seconds, see Attachment #1 - seems highly unlikely, probably some kind of glitch in the serial communication? This particular pump is relatively new from Agilent (<2 years installed I think)
  • The PSL shutter was automatically closed at ~1150 am today, see Attachment #2. There is some EPICS logic on c1psl (Acromag server) that checks if C1:Vac-P1a_pressure is greater than 3 mTorr (or greater than 500 Torr for in-air locking of the IMC), in which case it closes the shutter, so this seems consistent with expectations.

Recommended course of action:

  1. Code in some averaging in the interlock code, so that the interlock isn't triggered on some unphysical glitch like this. As shown in Attachment #3, this has been happening for the past 24 hours (though not before, because the interlock wasn't tripped). Probably need the derivative of the temperature as well, and the derivative should be less than 5 C/s or something physical (in addition to the temperature being high) for the interlock to trip.
  2. Re-open V1 to pump down the main volume to nominal pressure so that the interferometer locking activity can resume.
    • One option in the interim is to bypass the TP2 temperature interlock condition.
    • The pressure-based interlocks are probably sufficient to protect the main volume / pumps during the nominal operations - the temperature interlocks are mainly useful during the pumpdown where the TPs have a large load, and so we want to avoid over-stressing them.
Attachment 1: TP2_tempGlitch.png
TP2_tempGlitch.png
Attachment 2: PSL_shutterClosed.png
PSL_shutterClosed.png
Attachment 3: TP2tempGlitches.pdf
TP2tempGlitches.pdf
  15393   Thu Jun 11 17:35:34 2020 gautamUpdateVACPumpspool UPS needs battery replacement

Summary:

The pumpspool UPS has its "Replace Battery" indicator light on. Might be a good chance to change the UPS, but at the very least, we should put in fresh batteries (last replacement was in Aug 2017).

I'll say this again - the pumpspool area is noisier than I remember it being, I think one / both of the roughing pumps backing TP2 / TP3 need tip-seal replacements.

BTW, EX is 5C hotter than EY, by virtue of the tarnac outside? In fact, judging by Steve's thermometers, EX reports a 12C swing in 24 hours between 30 C and 18 C (so almost no temperature control) while EY reports a 5C swing between 20 and 25 C. This is borne out by the ETM Oplev data I think...

  15397   Fri Jun 12 19:02:52 2020 gautamUpdateVACPumpspool UPS needs battery replacement

I still don't understand why restoring the vacuum is contingent on this functionality working. All the TPs have their own internal logic to shutdown the pump if some damage threshold is exceeded. Plus, we have the pressure-sensor based interlocks to protect the main volume as well as pumps. While the extra redundancy from the readbacks from the controller is useful, clearly it isn't the first line of defense?

The main volume pressure is currently ~10mTorr. If we pump down before this reaches 500mTorr, the procedure is pretty straightforward. Otherwise, we have to do the dance with the manual throttling valve (judging by current rate of increase, unlikely to exceed this over the weekend, but I lose IFO time).

Obviously I don't want to rush this and have some permanent damage, so I'll stay out of this unless otherwise instructed.

  15399   Fri Jun 12 19:33:31 2020 gautamUpdateVACPumpspool UPS needs battery replacement

Didn't mean to sound whiny. I will wait until the vacuum team tells me it is okay.

Quote:

The vacuum safety policy and design are not clear to me, and I don't know what the first and second defense is. Since we had limited time and bandwidth during the remotely-supported recovery work today, we wanted to work step by step.

The pressure rising rate is 20mtorr/day, and turning on TP3 early next week will resume the main-volume pumping without too much hustle. If you need the IFO time now, contact with Jon and use backing with TP3.

  15404   Wed Jun 17 16:27:51 2020 gautamUpdateVACQuestions/comments on vacuum

I missed the vacuum discussion on the call today, but I have some questions/comments:

  • Isn’t it true that we didn’t digitally monitor any of the TP diagnostic channels before 2018 December? I don’t have the full history but certainly there wasn’t any failure of the vacuum system connected to pump current/temp/speed from Sep 2015-Dec2018, whereas we have had 2 interruptions in 6 months because of flaky serial communications.
  • According to the manuals, the turbo-pumps have their own internal logic to shut off the pump when either bearing temperature exceeds 60C or current exceeds 1.5A. I agree its good to have some redundancy, but do we really expect that our outer interlock loops will function if the internal ones fail?
  • In what scenario do we expect that all our pressure gauge readbacks fail, but not the TP readbacks? If so, won’t the differential pressure conditions protect the vacuum envelope, and the TPs internal shutoffs will protect the pumps? Except during the pump down phase perhaps, when we want to give a little more headroom to the small TPs to stress them less?

At the very least, I think we should consider making the interlock code have levels (like interrupts on a micro controller). So if the pressure gauges are communicating and are reporting acceptable pressure readings, we should be able to reject unphysical readbacks from the TP controllers.

I still don’t understand why TP2 can’t back TP1, but we just disable all the software interlock conditions contingent on TP2 readbacks. This pump is far newer than TP3, and unless I’ve misunderstood something major about the vacuum infrastructure, I don’t really see why we should trust this flaky serial readbacks for any actionable interlocks, at least without some AND logic (since temperature, current and speed aren’t really independent variables).

I also think we should finally implement the email alert in the event the vacuum interlock is tripped. I can implement this if no one else volunteers.

This might also be a good reminder to get the documentation in order about the new vacuum system.

  15407   Thu Jun 18 12:00:36 2020 gautamUpdateVACQuestions/comments on vacuum

I agree there were MEDM fields, but I can't find any record of these channels being recorded till 2018 December, so I don't agree that they were being digitally monitored. You can also look back in the elog (e.g. here and here) and see that the display fields are just blank. I would then assume that no interlocks were dependent on these channels, because otherwise the vacuum interlocks would be perpetually tripped.

Looking at images of the old vac screens, the TP2/3 rotation speed and status string were digitally monitored. However I don't know if there were software interlocks predicated on those.

Sorry but I'm having trouble imagining a scenario how the pressure gauges wouldn't register this before the IFO volume is compromised. Is there some back of the envelope calculations I can do to understand this? Since both the pressure gauges and the TP diagnostic channels are being monitored via EPICS, the refresh rate is similar, so I don't see how we can have a pump temperature / speed / current threshold tripped but NOT have this be registered on all the pressure gauges, seems like a bit of a contrived scenario to me. Our thresholds currently seem to be arbitrary numbers anyway, or are they based on some expected backstreaming rate? Isn't this scenario degenerate with a leak elsewhere in the vacuum envelope that would be caught by the differential pressure interlocks?

The temperature and current interlocks are implemented precisely because the pumps can shut themselves off. The concern is not about damaging the pumps (their internal logic protects against that); it's that a pump could automatically shut down and back-vent the IFO to atmosphere. Another interlock (e.g., the pressure differentials) might catch it, but it would depend on the back-vent rate and the scenario has never been tested. The temperature and current interlocks are set to trip just before the pump reaches its internal shut-down threshold.

For the email alert, can you expose a soft channel that is a flag - if this flag is not 1, then the service will send out an email.

That would be awesome if you're willing to volunteer. I agree this would be great to have.
  15410   Thu Jun 18 15:46:34 2020 gautamUpdateVACQuestions/comments on vacuum

So why not just have a special mode for the interlock code during pumpdown and venting, and during normal operation we expect the main volume pressure to be <100uTorr so the interlock trips if this condition is violated? These can just be EPICS buttons on the Vac control MEDM screen. Both of these procedures are not "business as usual", and even if we script them in the future, it's likely to have some operator supervising, so I don't think it's unreasonable to have to switch between these modes. I just think the pressure gauges have demonstrated themselves to be much more reliable than these TP serial readbacks (as you say, they worked once upon a time, but that is already evidence of its flakiness?). The Pirani gauges are not ultra-reliable, they have failed in the past, but at least less frequently than this serial comm glitching. In fact, if these readbacks are so flaky, it's not impossible that they don't signal a TP shutdown? I just think the real power of having these multi-channel diagnostics is lost without some AND logic - a turbopump failure is likely to result in an increase in pump current and temperature increase and pump speed decrease, so it's not the individual channel values that should be determining if an interlock is tripped.

I definitely think that protecting the vacuum envelope is a priority - but I don't think it should be at the expense of commissioning time. But if you think these extra interlocks are essential to the safety of the vacuum system, I withdraw my request.

I don't disagree that the pressure gauges would register the change. What I'm not sure about is whether the change would violate any of the existing interlock conditions, triggering a shutdown. Looking at what we have now, the only non-pump-related conditions I see that might catch it are the diffpres conditions:

It would be better to have a flag channel, might be useful for the summary pages too. I will make it if it is too much trouble.

There's already a channel C1:Vac-error_status, where if the value is anything other than an empty string, there is an interlock tripped. Does that work?
  15415   Fri Jun 19 09:57:35 2020 gautamUpdateVACQuestions/comments on vacuum

For this particular email service, ideally the email should be sent out as soon as the interlock is tripped, so this would require a line of code to be added to the main interlock code. Which I guess would require a restart of the interlock service. So let me know when you guys plan to do the dry-pump tip seal replacement operation (when I presume valves will be closed anyways) so that we can do this in a minimally invasive way.

Quote:

Ok, this can be added pretty easily. Its value will just be toggled between 1 and 0 every time the interlock server raises/clears the existing string channel. Adding the channel will require restarting the whole vac IOC, so I'll do it at a time when Jordan is on hand in case something fails to come back up.

  15418   Fri Jun 19 16:30:09 2020 gautamUpdateASCSome thoughts about ASC

Summary:

In ELOG 15368, I had claimed that the POP QPD based feedback servo actuating on the PRM stabilized the lock. I now believe this scheme of sensing using the POP QPD and feeding back to the PRM is not a good topology for stabilizing the PRC angular motion.

Details:

  • I was never able to get a measurement of the OLTF of this loop that made sense 
    • the loop was initally commissioned with the PRMI locked on the carrier, and the settings hence inferred to give a ~5 Hz UGF loop were used in the PRFPMI lock.
    • In the PRFPMI configuration, however, the loop gain seemed way too low when I measured using the usual IN1/IN2 method.
    • So it is critical for the lock stability that the angular feedforward works well, which it kind of does now (not that I have changed anything, but the glitches in the seismometer have not resurfaced recently).
    • Hopefully, this becomes less of an issue once we replace the TTs with SOS and OSEM based damping.
  • To get some more insight, I did some finesse modeling
    • Attachment #1 shows the sensing response at the QPDs we have available currently (POP and TR). 
    • I included the telescopes (propagation distances, in-air lenses) to these QPDs as best as I could.
    • A simplified model (3 mirror coupled cavity) is used, so there isn't really a common/differential mode in this picture, but we still get some insight I think.
    • Specifically, once the full lock is realized, the PRC optic motion isn't sensed well with our QPDs, and so it was some fluke that turning on these PRC angular feedback loops worked. 
    • Attachment #2 shows the same info as Attachment #1, but with the pendulum transfer functions (and radiation pressure effects) included. The SOS suspensions are modelled as f0=0.7/0.8 Hz (for P/Y), Q=5, while the tip-tilts have f0~5 Hz, Q~10. The high frequency phase is 0 degrees and not 180 as expected because of the pendulum complex pole pair because of the way the quantity is computed in Finesse.
  • The current scheme I use is:
    • DC couple the ITM oplevs, using their individual Oplev QPDs.
    • Use the TR QPDs, mixed to actuate on the ETMs in a common/differential way.
    • I think the system is under-determined with the sensors we currently have - we wan't to sense the 10 angular modes - PIT and YAW for the PRC, Csoft, Chard, Dsoft and Dhard (using the terminology from Kate's thesis), but we only have 6 sensors of the same field (POP, TRX and TRY QPDs, PIT and YAW from each).
    • So we need more sensors?
  • One thing that can easily be improved I think is to make the ASS system work at high power. 
    • think this should be as simple as scaling the gain for the loops to work for the high power.
    • Then we can counteract the input pointing drift at least.
    • But the ITM Oplev DC coupling would need to be turned OFF and then ON again, I'm not sure if this will introduce some transient that will destroy the lock...

I would also like to bring up the topic of implementing some WFS for the interferometer fields again, there doesn't seem to be any mention of this in the procurement/planning for the BHD. It is not obvious to me yet that we need WFS and not just DC QPDs from a noise point of view, but at least we should discuss this.

Attachment 1: sensingResponse.pdf
sensingResponse.pdf
Attachment 2: sensingResponse_torque.pdf
sensingResponse_torque.pdf
  15419   Fri Jun 19 17:06:50 2020 gautamUpdateLSCWhat should the short-term commissioning goals be?

Summary:

I want some input about what the short-term (next two weeks) commissioning goals should be.

Details:

Before the vacuum fracas, the locking was pretty robust. With some human servoing of the input beam, I could maintain locks for ~1 hour. My primary goals were:

  1. Transition the vertex length DoF control from 3f signals to 1f signals.
  2. Turn on some MICH-->DARM feedforward cancellation, because the noise between ~100 Hz and ~1kHz is dominated by this cross-coupling.

I didn't succeed in either so far.

  1. I find that there is poor separation of the length DoFs in the 1f sensors, which makes this transition hopeless.
    • Why should this be? I can't get any sensing matrix in Finesse to line up with what I measure in-lock.
    • One hypothesis I came up with (but haven't yet tested) is that the offsets from the 3f photodiodes are changing from time to time, which somehow changes the projection of the various DoFs onto the photodiode quadratures. 
    • The attached GIF shows the variation in the measured sensing matrix on two days - while the sensing of MICH/PRCL in the 3f photodiodes have hardly changed, they are significantly different in the 1f photodiodes. Note that the I and Q have changed for REFL11 and REFL55 between the two days because I changed the demod phase.
    • I also thought that maybe the CARM suppression isn't sufficient for REFL11 to be used as a PRCL sensor - but even after engaging a CM board SuperBoost, I was unable to realize the PRCL 3f-->1f transition, even though the CARM-->REFL11 coupling did get smaller in the measured sensing matrix (red line in the GIF). I don't think we can juice up the CARM gain much more without modifying the CM board boosts, see Attachment #1.
  2. I was able to measure the MICH CTRL --> DARM ERR transfer function with somewhat high coherence (~0.98).
    • I then used the infrastructure available in the LSC model to try and implement some cancellation, but didn't really see any effect.
    • Perhaps the TF needs to be measured with higher coherence.
    • It may also be that if I am able to successfully execute the 3f-->1f transition, the coupling gets smaller because the 1f sensing noise is lower?

I guess apart from this, we want to run the ALS scan to try and infer something about the absorption-induced thermal lens. I guess at this point, the costs outweigh the benefits in trying to bring in the SRC as well, since we will be changing the SRC config?

Attachment 1: CARM_superBoost.pdf
CARM_superBoost.pdf
  15420   Fri Jun 19 19:21:25 2020 gautamUpdateGeneralPSL shutter re-opened

The PSL shutter was closed from the vacuum interlock trip. Today, I did the following:

  • Re-aligned input beam to PMC to recover high transmission / low reflection.
  • Re-set the LSC offsets.
  • ETMX watchdog was tripped. Reset it.
  • Opened the PSL shutter, IMC autolocker was able to lock the cavity almost immediately.
  • Tested POX/POY locking, ran the ASS to maximize single arm transmission.

All looks good for now. I will probably get back to PRFPMI locking Monday.

  15423   Mon Jun 22 17:51:50 2020 gautamUpdateCDSc1iscaux was down

The machine needed a hard reboot as it was un-ssh-able. 

The exact time that the machine went down is unknown because the blinkys were not DQ-ed. I've now added these to the EDCU to make these channels actually useful, and we may look back on the reliability (or otherwise) of the Acromag system. To my memory, this is the ~5th time one of the new Acromag servers has needed a hard reboot. While this may be less frequent (?) than the VME machines, perhaps there is some other reason for these dropouts. Maybe something to do with the martian network?

Anyway the machine is back up and running now.

  15427   Wed Jun 24 17:20:16 2020 gautamUpdateLSCWhat should the short-term commissioning goals be?

Per the discussion at the meeting today, the plan of action is:

  1. Lock the PRMI on carrier and measure the sensing matrix, see if the MICH and PRCL signals look sensible in 1f and 3f photodiodes.
  2. Try locking CARM on POP55 (since there is currently no POP55 photodiode, can we use POX/POY as an intermediary?).
  3. For the ASC, can we hijack one of the IMC WFS heads to study what the AS port WFS signals would look like, and maybe close a feedback loop on the ETMs?
    • My guess is no, because currently, the L2A is so poorly tuned on MC2 that the CARM length control messes with the alignment of the IMC significantly.
    • So we need the IMC WFS loops to maintain the pointing.
    • Of course, the MC2 L2A can be tuned to mitigate this problem. 
    • I also believe there is something funky going on with the WFS heads. More to follow on that in a later elog.
    • Apart from these issues, for this scheme to be tested, some mods to the c1ioo model will have to be made so that we can route the servo output to the ETMs (as opposed to the IMC mirrors as is the usual case).

If I missed something, please add here.

Quote:

Summary:

I want some input about what the short-term (next two weeks) commissioning goals should be.

  15428   Wed Jun 24 22:33:44 2020 gautamUpdateSUSEQ tripped all suspensions

This earthquake tripped all suspensions and ITMX got stuck. The watchdogs were restored and the stuck optic was released. The IFO was re-aligned, POX/POY and PRMI on carrier locking all work okay.

  15431   Thu Jun 25 15:11:00 2020 gautamUpdateSUSMC1 coil driver resistance quartered

I implemented this change today. We only had 100 ohm, 3W resistors in stock (no 200 ohm with adequate power rating). Assuming 10 V is dropped across this resistor, the power dissipation is V^2/R ~ 1 W, so we should have sufficient margin. DCC entry has been updated with new schematic and photo of the component side of the board. Note that the series resistance of the fast actuation path was untouched.

As expected, the requested voltage no longer exceeds the Acromag DAC range, it is now more like 2.5 V. However, I still notice that the MC REFL spot moves somewhat diagonally on the camera image - so maybe the coil gains are seriously imbalanced? Anyway, the WFS control signals can once again be safely offloaded to the slow bias voltages once again, preserving the fast ADC range for other actuation.

The Johnson noise of the series resistor has now increased by a factor of 2, from ~6.4 pA/rtHz to 12.8 pA/rtHz. Assuming a current to force coefficient of 1.6 mN/A per coil, the length noise of the cavity is expected to be 12.8e-12 * 0.064/0.25/(2*pi*100)^2 ~ 8e-18 m/rtHz at 100 Hz. In frequency units, this is 80 uHz/rtHz. I think our IMC noise is at least 10 times higher than this at 100 Hz (in any case, the noise of the coil driver is NOT dominated by the series resistance). Attachment #1 confirms that there isn't any significant MCF noise increase, and I will check with the arm cavity too. Nevertheless, we should, if possible, align the optic better and use as high a series resistance as possible.

The watchdog for MC1 was disabled and the board was pulled out for this work. After it was replaced, the IMC re-locks readily.

Quote:

But this does not solve the MC1 issue. Only we can do right now is to make the output resister half, for example.

Attachment 1: MCF.pdf
MCF.pdf
  15433   Fri Jun 26 16:53:38 2020 gautamUpdateElectronicsRFPD characterization

Summary:

While the vacuum system was knocked out, I measured the RF transimpedance (using the AM laser setup, didn't do the shot noise intercept current measurement for now) of all the RFPDs (except PMC REFL). At the very least, the following photodiodes are suspect:

  1. WFS heads - expected transimpedance is 50 kohm unattenuated, and 5 kohm attenuated. I measure values that are x10 lower than this, and the segments are significantly imbalanced. Morover, the attenuators for some quadrants appear to do nothing. This could be a problem with the Acromag system I guess, but the measured transimpedance is nowhere close to the "expected" value. See Attachments #1 and #2. You can also see that the response at 55 MHz is significantly attenuated, so I'm guessing trying to measure the AS port ASC sensing response is going to be difficult.

    Note that I assumed a 1kohm DC transimpedance, which is what I expect from the schematic and also is consistent with the DC voltage I measured, knowing the approximate optical power incident on the photodiode.
  2. POP 22/ POP 110 - this is a Thorlabs PDA10CF diode. It should have a flat gain profile out to ~100 MHz, but I measure some weird features. The other PDA10CF we use, at AS110, shows a more reasonable response. See Attachment #3. I don't know what kind of failure mode this is? Anyway I'll try testing another PDA10CF and if it looks more reasonable, I'll switch out this diode. FWIW, the measured AS110 gain is ~3kohms, whereas the datasheet tells us to expect 5 kohms.

For the remaining photodiodes, I measure a transimpedance that is within ~20% of what is on the wiki page. The notches may benefit from some retuning. While I have the data, I will fit this and post a more complete report on the wiki.

Update July 6 1145am: WFS response plots now have legends mapping quadrants, and I've also added the response of a spare PDA10CF (which is now the new POP22/POP110 photodiode).

Attachment 1: WFS1.pdf
WFS1.pdf
Attachment 2: WFS2.pdf
WFS2.pdf
Attachment 3: buildupMons.pdf
buildupMons.pdf
  15434   Sun Jun 28 15:30:52 2020 gautamUpdateSUSMC1 sat-box de-lidded

Judging by the summary pages, some 18 hours after this change was made and the board re-installed, the MC1 shadow sensors began to report frequent glitches. I can't think of a plausible causal connection, especially given the 18 hour time lag, but also hard to believe there isn't one? As a result, the IMC is no longer able to stay locked for extended periods of time. I did the usual cable squishing, and also took off the lid to see if that helps the situation.

While the reduced series resistance means there is more current flowing through the slow path

  1. There isn't actually an increase in the net current flowing through the satellite box - this change just re-allocates the current from the fast path to the slow path, but by the time it reaches the satellite box, the current is flowing through the same conductor.
  2. afaik, the current buffers on the coil driver aren't overdriven - they are rated for 300 mA. No individual coil is drawing more than 30 mA.
  3. the resistors themselves should be running sufficiently below their rated power of 3W (I estimate 2.5 V ^2 / 100 ohms ~ 60 mW).
  4. The highest current should be through the UL and LR coils according to the voltage outputs from the Acromag. But the UL coil doesn't show significant glitching, and the LL one does despite drawing negligible DC current.

The attached FLIR camera image re-inforces what we already know, that the thermal environment inside the satellite box is horrible. The absolute temperature calibration may be off, but it was difficult to touch the components with a bare finger, so I'd say its definitely > 70 C.

Quote:

I implemented this change today. We only had 100 ohm, 3W resistors in stock (no 200 ohm with adequate power rating). Assuming 10 V is dropped across this resistor, the power dissipation is V^2/R ~ 1 W, so we should have sufficient margin. DCC entry has been updated with new schematic and photo of the component side of the board. Note that the series resistance of the fast actuation path was untouched.

Attachment 1: 20200628T144138.jpg
20200628T144138.jpg
  15436   Sun Jun 28 17:36:35 2020 gautamUpdateSUSMC1 sat-box de-lidded

Hmm I can't seem to export with the colorbar, might be just my phone though. I tried to add some "cursors" with the temperature at a few spots, but the font color contrast is poor so you have to squint really hard to see the temperatures in the photo I attached.

I'll leave the MC1 box open overnight and see if that improves the situation, and if not, I'll switch in the SRM satellite box tomorrow.

Quote:

does the FLIR have an option to export image with a colorbar?

How about just leave the lid open? or more open? I don't know what else can be done in the near term. Maybe swap with the SRM sat box to see if that helps?

  15438   Mon Jun 29 11:55:46 2020 gautamUpdateSUSMC1 sat-box de-lidded

There was no improvement to the situation overnight. So, I did the following today:

  1. Ramped bias voltages for SRM and MC1 to 0, shutdown watchdogs.
  2. Switched SRM and MC1 satellite boxes. The SRM satellite box lid was opened, while the MC1 lid was left open. The boxes have also been re-labelled lest there be some confusion about which box belongs where.
  3. Restored watchdogs and bias voltages. Curiously, the MC1 optic now only requires half the bias voltages it did before to have the correct DC alignment for the optic. The Satellite box is just supposed to be a passive conduit for the drive current, so this is indicative of some PCB traces/cabling being damaged inside what was previously the MC1 satellite box?

IMC is now locked again, I will monitor for glitching/stability.

Update 6pm PDT: as shown in Attachment #1, there is a huge difference in the stability of the lock after the sat box swap. Let's hope it stays this way for a while...

Quote:

I'll leave the MC1 box open overnight and see if that improves the situation, and if not, I'll switch in the SRM satellite box tomorrow.

Attachment 1: SatBoxSwap.jpg
SatBoxSwap.jpg
  15439   Mon Jun 29 15:56:02 2020 gautamUpdateElectronicsRFPD characterization

A more comprehensive report has been uploaded here. I'll zip the data files and add them there too. In summary:

  1. There are several problems with the WFS heads
    • Some attenuators don't seem to work. This could be a problem with the Acromag BIO, or with the relay on the head itself.
    • The measured transimpedance at 29.5 MHz is much lower than expected. We expect ~50 kohms with no attenuation, and 5 kohms without. I measure 100 ohm - 2 kohm with the attenuation disabled, and ~200 ohms with it enabled.
    • Quadrant #3 on both WFS heads behaves differently from the others. There is also evidence of a 200 MHz oscillation for quadrant 3.
    • For some reason, there is a relative minus sign between the TFs measured for the WFS and for the RFPDs. I don't understand where this is coming from - all the OpAmps in the LSC PDs and WFS heads are configured as non-inverting, so why should there be a minus sign? Is this indicative of the polarity of the LEMO output being somehow flipped?
  2. POX 11 photodiode does not have a notch at 22 MHz.
  3. AS55 resonance appears to have shifted closer to 60 MHz, would benefit from a retuning. But the notches appear fine.
  4. PDA10CF photodiode used as the POP22/POP110 readback appears broken in some strange way. As shown in the linked document, a spare PDA10CF in the lab has a much more reasonable response, so I am going to switch out the POP22/POP110 diode with this spare.

I'll upload the data and analysis notebook + liso fit files to the wiki as well shortly. The data, a Jupyter notebook making the plots, and the LISO fit files have been uploaded here.

I didn't do it this time but it'd be nice to also do the noise measurement and get an estimate for the shot-noise intercept current.

Quote:

While I have the data, I will fit this and post a more complete report on the wiki.

  15442   Tue Jun 30 10:59:16 2020 gautamUpdateLSCThree sensing matrices

Summary:

I injected some sensing lines and measured their responses in the various photodiodes, with the interferometer in a few different configurations. The results are summarized in Attachments #1 - #3. Even with the PRMI (no arm cavities) locked on 1f error signals, the MICH and PRCL signals show up in nearly the same quadrature in the REFL port photodiodes, except REFL165. I am now thinking if the output (actuation) matrix has something to do with this - part of the MICH control signal is fed back to the PRM in order to minimize the appearance of the MICH dither in the PRCL error signal, but maybe this matrix element is somehow horribly mistuned?

Details:

Attachment #1:

  • ETMs were misaligned and the PRMI was locked with the carrier resonant in the cavity (i.e. sidebands reflected).
  • The locking scheme was AS55_Q --> MICH and REFL11_I --> PRCL.

Attachment #2:

  • The PRFPMI was locked. The vertex DoFs were still under control using 3f error signals (REFL165_I for PRCL and REFL165_Q for MICH).
  • Still, the MICH/PRCL degeneracy in all photodiodes except REFL165 persists.

Attachment #3:

  • Nearly identical configuration to Attachment #2.
  • The main difference here is that I applied some offsets to the MICH and PRCL error points.
  • The offsets were chosen so that the appearance of a ~300 Hz dither in the length of MICH/PRCL was nulled in the AS110_Q / POP22_I signals respectively.
  • For the latter, the appearance of this peak in the POP110_I signal was also nulled, as it should be if our macroscopic PRC length is set correctly.
  • The offsets that best nulled the peak were 110 cts for PRCL, 25 cts for MICH. The measured sensing response is 1e12 cts/m for PRCL in REFL165_I and 9.2e11 cts/m for MICH in REFL165_Q. So these offsets, in physical units, are: 110 pm for PRCL and 27 pm for MICH. They seem like reasonable numbers to me - the PRC linewidth is ~7.5 nm, so the detuning without any digital offset applied is only 1.5% of the linewidth.
  • Note that I changed the POP22/POP110 demod phases to maximize the signal in the I quadrature. The final numbers were -124 degrees / -10 degrees respectively.
  • Yet another piece of evidence suggesting these were the correct offsets is that the DC value of POX and POY were zero on average after these offsets were applied.
  • However, the MICH/PRCL responses in the 1f REFL port photodiodes remain nearly degenerate.

Some other mysteries that I will investigate further:

  1. While POP22 indicated stable buildup of 11 MHz power in the PRC, I couldn't make any sense of the AS110 signals at the dark port - there was large variation of the signal content in the two quadratures, so unlike the POP signals, I couldn't find a digital demod phase that consistently had all the signal in one of the two quadratures. This is all due to angular fluctuations?
  2. My ASC simulations suggest that the POP QPD is a poor sensor of PRM motion when the PRFPMI is locked. However, I find that turning on a feedback loop with the POP QPD as a sensor and the PRM as the actuator dramatically reduces the low-frequency fluctuations of the arm cavity carrier buildup. 🤔

I blew the long lock last night because I forgot to not clear the ASS offsets when trying to find the right settings for running the ASS system at high power. Will try again tonight...

Quote:

Lock the PRMI on carrier and measure the sensing matrix, see if the MICH and PRCL signals look sensible in 1f and 3f photodiodes.

Attachment 1: PRMI_1f_20200625sensMat.pdf
PRMI_1f_20200625sensMat.pdf
Attachment 2: PRFPMI_20200629sensMat.pdf
PRFPMI_20200629sensMat.pdf
Attachment 3: PRFPMI_20200629sensMat_wOffset.pdf
PRFPMI_20200629sensMat_wOffset.pdf
  15443   Tue Jun 30 22:00:04 2020 gautamUpdateElectronicsGlitchy POX resurfaces

This problem reared its ugly head again. I am inclined to believe the problem is electronic and not on the light, since the POY channels seem immune to this issue (see Attachment #1). I will investigate in the daytime tomorrow. Note that while the POX photodiode head has ~twice the transimpedance than POY (per measurement), the POY signal gets amplified by a ZHL-500-HLN amplifier before heading to the demod electronics (nominal gain is 19dB = x9). There is also some imbalance in the light level at the photodiodes I guess, because overall, the PDH fringe is ~twice as large for the Y arm as the X arm. Basically, the y-axes of the attached plot cannot be directly compared between POX and POY.

Mostly this is an annoyance - right now, the POX signal is only used for locking and dither aligning the X arm cavity, and so once that is done, the locking can proceed (as long as the other channels, e.g. REFL11, aren't glitching as well...)

Attachment 1: glitchyPOX.jpg
glitchyPOX.jpg
  15445   Wed Jul 1 12:50:40 2020 gautamUpdatePEMMC1 accelerometers plugged in

I re-connected the 3 accelerometers located near the MC1/MC3 chamber. It was a bit tedious to get the cabling sorted - I estimate the cable is ~80m long, and the excess length had to be wound around a spool (see Attachment #1), which wasn't really a 1 person job. It's neat-ish for now, but I'm not entirely satisfied. I think we should get shorter cables (~20m), and also mount the pre-amp/power units in a rack instead of leaving it on the floor. The pre-amp settings are x100 for all three channels. The MC2 channels are powered, but are unconnected to the seismometers - it was too tedious to unroll the other spool yesterday. Apart from this, the cable for the "Z" channel had to be re-seated in the strain relief clamp.

I did not enable any of the CDS filters that convert the raw signal into physical units, so for now, these channels are just recording raw counts.

Update 7pm: the spectra in the current config are here - not sure what to make of the MC2_Z channel appearing to show lower noise?

Update July 13 2020 430pm: This afternoon, I hooked up the MC2 accelerometer channels too...

Attachment 1: IMG_8617.JPG
IMG_8617.JPG
Attachment 2: IMG_8616.JPG
IMG_8616.JPG
  15447   Wed Jul 1 18:16:09 2020 gautamUpdateComputersrossa re-re-revival

In an effort to make a second usable workstation, I did the following (remotely) on rossa today (not necessarily in this order, I wasn't maintaining a live log so I forgot):

  1. Fixed /etc/resolv.conf, so that the other martian machines can be found.
  2. Copied over .bashrc file, and the appropriate lines from /etc/fstab from pianosa to rossa.
  3. Ran sudo apt install nfs-common. Then ran sudo mount -a to get /cvs/cds mounted.
  4. Made symlinks for /users and /opt/rtcds , and /ligo. All of these are used by various environment-setting scripts and I chose to preserve the structure, though why we need so many symlinks, I don't know...
  5. Set up the shell variable $NDSSERVER using export NDSSERVER=fb:8088. I'm not sure how, but I believe DTT, awggui etc use this on startup to get the channel list (any
  6. Followed instructions from Erik von Reis at LHO to install the cds workstation packages and dependencies. Worked like a charm 🎃
  7. As a test, I plotted the accelerometer spectra in DTT, see Attachment #1. I also launched foton from inside awggui, and confirmed that the sample rate is inherited and I could designate a filter. But I haven't yet run the noise injection to test it, I'll do that the next time I'm in the lab.
  8. Also checked that medm, StripTool and ndscope, and anaconda python all seem to work 👍🏾.

So, in summary, rossa is now all set up for use during lock acquisition. However, until this machine has undergone a few months of testing, we should freeze the pianosa config and not mess with it.

Note that this version of the "crtools" is rather new. Please, use them and if there is an issue, report the errors! I am going to occassionally try lock acquisition using rossa. 

Quote:

wiped and install Debian 10 on rossa today

still to be done: config it as CDS workstation

please don't try to "fix" it in the meantime

Attachment 1: MCacc.pdf
MCacc.pdf
  15452   Mon Jul 6 00:37:28 2020 gautamUpdateComputersrossa: lib symlink

This is strange - I was definitely able to launch medm when I was working on this machine remotely on Friday. But now, there does seem to be a problem with this shared library being missing.

First of all, I installed mlocate to find where the shared library files are installed. Then I made the symlink, and now sitemap seems to work again.

Weirdly, my changes to /etc/resolv.conf got overwritten somehow. Was this machine rebooted? Uptime suggests it's only been running for ~6 hours at the time of writing of this elog.

sudo apt install mlocate
sudo updatedb
sudo ln -s /usr/lib/x86_64-linux-gnu/libreadline.so.7 /usr/lib/x86_64-linux-gnu/libreadline.so.6
Quote:

when I try 'sitemap' on rossa I get:

medm: error while loading shared libraries: libreadline.so.6: cannot open shared object file: No such file or directory

  15455   Mon Jul 6 12:51:41 2020 gautamUpdateComputersrossa: resolvconf installed

Indeed, this is now fixed by following instructions from here. I rebooted rossa at ~1250 PDT and confirmed that resolv.conf didn't get overwritten. The resolv.conf file also now has the following useful lines at the head:

~>cat /etc/resolv.conf
# Dynamic resolv.conf(5) file for glibc resolver(3) generated by resolvconf(8)
#     DO NOT EDIT THIS FILE BY HAND -- YOUR CHANGES WILL BE OVERWRITTEN
Quote:

yes, I rebooted yesterday to fix the 'steaking white lines' problem in the video/display

maybe we're supposed to edit something besides resolv.conf since that gets over-written on boot for some linux OS

  15457   Mon Jul 6 17:41:19 2020 gautamUpdateLSCAn LSC puzzler

Last Tuesday evening, while attempting the PRFPMI locking, I noticed a strange feature in the LSC signals, which is shown in Attachment #1 (the PDF exported by dataviewer is 14MB so I upload the jpeg instead). As best as I can tell, the REFL33 and POP22 channels show an abrupt jump in the signal levels, while the other channels do not. POP110 shows a slight jump at around the same time, and the large excursion in AS110_Q actually occurs a few seconds later, and is probably some angular excursion of the PRC/BS. I'm struggling to interpret how this can be explained by some interferometric mechanism, but haven't come up with anything yet. The LO for the 3f error signals is the 2f field, but then why doesn't the POP110 channel show a similar jump if there is some abrupt change in the resonant condition? Is such a change even feasible from a cavity length change point of view? Or did the sideband frequency somehow abruptly jump? But if so, why is the jump much more clearly visible in one sideband than the other?

Does anyone have any ideas as to what could be going on here? This may give some clue as to what's up with the weird sensing matrices, but may also be something boring like broken electronics... 

Attachment 1: LSCsignals.jpg
LSCsignals.jpg
  15458   Tue Jul 7 14:06:10 2020 gautamUpdateASCSome more thoughts about ASC

Summary:

I want to be able to run the dither alignment servo with the PRFPMI locked - I've been thinking about what the scheme should be, and I list here some questions I had while thinking about this.

Details:

  1. ITM Oplev DC coupling
    • In the current scheme, I DC couple the ITM Oplev servos after the arms have been aligned to maximize POX/POY transmission.
    • However, looking back at data from when the CARM offset is reduced (e.g. Attachment #1), it looks like the ITMs are being torqued quite a bit during this process (ITMX PIT changes by ~20urad, ITMY YAW by ~10urad in this particular lock attempt). 
    • So the spots are not actually being centered on the test-masses? I guess the spot position on ITMX isn't actually controlled because we have only one actuator (BS) for the XARM beam axis. Is it unexpected that ITMY gets torqued so much? 
    • It is unclear what would happen if the ITM Oplev servos are not DC coupled. I wonder if I'd still be able to reach the high circulating powers and then rely solely on the TR QPDs for the arm cavity angular control.
    • Another possibility is to offload the DC part of the control signal to the optic's slow bias voltage slider, and then turn off the DC coupling. After that, the dither alignment can optimize the cavity alignment.
  2. Dither alignment at high circulating power
    • think that the system should work with the same settings as for the POX/POY locking, with the following changes:
      • Scale the overall loop gain by the arm transmission.
      • Change LSC2ASS matrix element from XARM/YARM ---> DARM.
        Does this sound right?
    • In light of the above, I was thinking that we should introduce a gain scaling field in the c1ass RTCDS model (like we have for the LSC power normalization matrix). Is it worth to go through the somewhat invasive process of model recompilation etc?
    • With the PRFPMI locked, I am wondering if I can simultaneously run the dither alignment loops for all the DoFs. Probably not, especially for MICH, since the actuator is the BS, which is also the actuator for the XARM loop?
Attachment 1: ITM_OL_DCcoupling.png
ITM_OL_DCcoupling.png
  15460   Wed Jul 8 22:50:33 2020 gautamUpdateComputersrossa: more symlinks

I wanted to try using rossa as my locking workstation today. However, a few problems became quickly evident. Basically, any of our scripts that rely on the cdsutils package (there are MANY) will not work on rossa, because of some library error. This machine is running Debian 10, while the cdsutils package is being loaded from a pre-compiled install on the shared drive, so perhaps this isn't surprising?

Digging a little more, I found that actually, a version of cdsutils that actually works with python3 is actually shipped with the standard cds-workstation meta-package. This is great news, and we should try and use this where possible I guess. Deferring further debugging for daytime work.

Anyway, I added a symlink: sudo ln -s /usr/lib/x86_64-linux-gnu/libncurses.so.6 /usr/lib/x86_64-linux-gnu/libncurses.so.5, and installed wmctrl using sudo apt install wmctrl

  15463   Thu Jul 9 16:16:20 2020 gautamUpdateComputersrossa: graphics driver issues?

I noticed these streaky lines again today (but they were not a problem last night). It is annoying if we have to reboot this machine all the time. I wonder if this has something to do with missing drivers. When I ran sudo apt update && sudo apt upgrade, I got several lines like (this isn't the whole stack trace)

W: Possible missing firmware /lib/firmware/nvidia/gp108/acr/ucode_unload.bin for module nouveau
W: Possible missing firmware /lib/firmware/nvidia/gp108/acr/ucode_load.bin for module nouveau
W: Possible missing firmware /lib/firmware/nvidia/gp108/acr/unload_bl.bin for module nouveau
W: Possible missing firmware /lib/firmware/nvidia/gp108/acr/bl.bin for module nouveau

Is this indicative of the graphics drivers being installed incorrectly? I am hesitant to mess with this because I think in the past, it was always trying to update some graphics driver that crashed the whole machine into some weird state where we have to wipe the drive and do a fresh re-install of the OS.

Should we just follow these instructions? The graphics card is apparently Quadro P400, which is one of the supported ones according to the list of supported devices.

Or just swap donatella and rossa monitors and defer the problem for later?

Quote:

yes, I rebooted yesterday to fix the 'steaking white lines' problem in the video/display

  15464   Thu Jul 9 17:12:52 2020 gautamUpdateBHDIn-air BHD

Summary:

We can probably learn something about the interferometer / top level BHD plan with an in-air BHD setup, even if the noise is bad. Here are some thoughts about how we would do it. 

LO delivery:

For this first attempt, we don't really care about the PRC filtering. So possible places to pick off an LO beam are:

LO beam pickoff options
Location Pros Cons
IP POS
  • Filtered by IMC
  • Medium level of invasiveness  
  • We lose the IP POS diagnostic, which is kind of useful nowadays given the drifty TTs.
  • Only few mW LO power available
PSL table IR beam currently going to green doubling setup
  • Least invasive w.r.t. normal IFO operation
  • Plenty of light (~100 mW) available. But how much can we safely couple into fiber?
  • Beam not filtered by IMC (although it is filtered by the PMC)
POX/POY
  • Since this beam is extracted from inside the PRC, probably enjoys the best filtering.
  • Possibly drifts a lot, so tricky to reliably couple into a fiber?
  • Maximally invasive w.r.t. regular IFO operations.

In all cases, I think the easiest option to actually route whatever beam we choose into a fiber, and then bring it over to whatever cavity we choose to use for an OMC. I'm assuming whatever phase control technique we end up using can cancel the fiber phase noise at relevant frequencies.

LO phase control

  • Stress the fiber? This will require us to purchase some custom hardware, and interface it to the CDS system.
  • PZT mirror? We should have sufficient hardware available to drive a PI style PZT mirror.

There is a question about the range, but I think these are the only two realistic options we can implement on a reasonable time scale.

OMC:

Again, there are a few options. Here are some pros and cons that come to my mind.

OMC cavity options
Option Pros Cons
Old copper OMC
  • Probably the simplest option in terms of the peripherals.
  • PZT driver recently verified to work
  • We can get the OMMT and DCPDs out as well.
  • Allows us to not compromise on the RF darm optical gain (not sure if locking will be as easy if we cut the power to the AS55 photodiode by 50-75%).
  • Requires a vent.
  • Probably not the most efficient use of the space on the AP table.
  • Filtering performance isn't quantified.
Spare PMC
  • Doesn't require a vent.
  • Compact footprint.
  • Need to build the cavity.
  • Need to check if the drive electronics from the old copper OMC can easily be interfaced with whatever PZT we use on this cavity.
  • Filtering performance kind of unknown?
Custom cavity with spare mirrors
  • Doesn't require a vent.
  • Probably no more difficult than the spare PMC option?
  • We need at least one actuatable mirror, so we'd need some PZT mounted optic + associated drive electronics.

If we can do a vent (we'd just need a single chamber open), I'd go for the option of getting the copper OMC out and using that. Attachment #1 shows the approximate sizes of the various components (OMMT, OMC cavity, DCPDs), while Attachment #2 shows a rough sketch of where things would go on the AP table, with the rectangles approximately to scale.

CDS:

I'd made a c1omc model sometime ago. Basically, I think we have sufficient ADC/DAC channels in the c1ioo machine for any of the options listed above - but using the copper OMC and associated peripherals would allow the easiest interfacing.

Criticisms/comments/thoughts please.

Attachment 1: OMCchamber.pdf
OMCchamber.pdf
Attachment 2: AP_Table_20180328.pdf
AP_Table_20180328.pdf
  15466   Fri Jul 10 01:25:28 2020 gautamUpdateLSCLocking notes

More tomorrow, but I tried the following tonight:

  1. Dither alignment for PRC / MICH seems to work when the PRFPMI is locked. Unfortunately, the correct settings for the arm cavity dither alignment loops continue to elude me.
  2. I tried some arm ASC loop characterization by stepping the error points of these loops - I saw some weird cross coupling between the loops that needs investigation.
  3. I'm unable to turn an integrator on for the "Common YAW" QPD loop - unclear why this is, but every time I attempt to engage said integrator, the lock is immediately blown. Needs some investigation of the signals.
  4. With the PRC / MICH angular DoFs aligned with the dither alignemnt, and the arm ASC loops hand tuned, I was able to get the darkest dark port I've seen. In terms of ASDC counts, it was ~ 200, which after undoing all the digital gains etc corresponds to ~100 uW of light. I think we can get a rough estimate of the contrast defect by accounting for (i) T_SRM, (ii) OMC pickoff fraction (iii) other losses between the BS dark port and the AP table (iv) 50/50 BS between AS55 and AS110 PD (the ASDC signal is derived from the former) and (v) the throughput of the 55 MHz sideband to the dark port, although there are many uncertainties. 
  15468   Fri Jul 10 15:26:28 2020 gautamUpdateLSCMC2 coils need DC balancing?

I was looking at some signals from last night, see Attachment #1.

  • It looks like as the DC control signal to the MC2 suspension increases, the MC transmission decreases.
    • I confirmed that the IMC REFL level doesn't correspondingly trend up, but didn't include it here for plot compactness, so I think the cavity length isn't being detuned.
    • So the problem is suggestive of some L2A coupling, and the MC2 coil actuators need to be balanced better at DC?
    • You can see from the IMC WFS control signals that the WFS servo is presumably trying to counteract this L2A action, but doesn't succeed, probably because the servo isn't tuned correctly.
    • This is a problem that is distinct from the drifting TT alignment. So it complicates the alignment situation.
    • Ideally, if the dither alignment servos could be made to work for the arm cavities when locked in the PRFPMI config, this wouldn't be so much of a problem, as the TTs would just adjust the beam pointing to match the cavity axes of the arms. But since I haven't managed to get that servo working yet...
  • But why should MC2 need such a large DC control signal ever?
    • In the PRFPMI lock, the CARM servo is supposed to match the laser frequency to the average length of the two arm cavities.
    • The MC2 suspension is used as a frequency actuator in order to realize this matching.
    • But, as you can see, the digital CARM control signal picks up a significant DC offset the deeper we go into the lock.
    • Can't we offload this DC signal to the laser crystal temperature servo? Is there going to be some weird interaction with the existing slow loop? Or is the idea itself flawed?

Attachment #2 shows some ASC metrics. My conclusion here is that running the PRCL and MICH dither alignment servos (former demodulating REFLDC and latter demodulating ASDC to get an error signal) that running the dither alignment servo and hand tuning the arm ASC loop offsets improves the mode matching to the IFO, because:

  1. The arm transmission increases.
  2. POPDC increases.
  3. ASDC decreases.

The REFLDC behavior needs a bit more interpretation I think, because if the IFO is overcoupled (as I claim it is), then better alignment would at some point actually result in REFLDC increasing. 

All the DC signals recorded by the fast system come from the backplane P2 connector of the PD interface boards. According to the schematic, these signals have a voltage gain of 2. The LSC photodiodes themselves have a nominal DC gain of 50 ohms. So, the conversion from power to digital counts is: 0.8 A/W * 50 V/A * 2 * 3276.8 cts/V * whtGain. Inverting, I get 3.8 uW/ct for a whitening gain of 1. This is power measured at the photodiode - optical losses upstream of the photodiode will have to be accounted for separately.

Assuming a modulation depth of 0.2, the 55 MHz sideband power should be ~20 mW. The Schnupp asymmetry is supposed to give us O(1) transmission of this field to the AS port. Then, the SRM will attenuate the field by a factor of 10, so we expect ~2 mW at the AS port. Let's assume 80 % throughput of this field to the AP table, and then there is a 50/50 beamsplitter dividing the light between the AS55 and AS110 photodiodes. So, we expect there to be ~700 uW of power in the TEM00 mode 55 MHz sideband field. This corresponds to 1600 cts according to the above calibration (the ASDC whitening gain is set to 18 dB). The fact that much smaller numbers were seen for ASDC indicates that (i) the schnupp asymmetry is not so perfectly tuned and the actual transmission of the sideband field to the dark port is smaller, or (ii) one or more optical splitting fractions assumed above is wrong. If the former is true, we can still probably infer the contrast defect if we can somehow get an accurate measurement of the sideband transmission to the dark port.

Attachment 1: MC2_balancing.png
MC2_balancing.png
Attachment 2: ASDC.png
ASDC.png
  15469   Sat Jul 11 00:10:22 2020 gautamUpdateComputersrossa: more developmental work

After some consultation with Erik von Reis at LHO, this workstation is progressing towards being usable for most commissioning tasks. DTT, awggui, foton, and MEDM are all now working well. The main limitation now comes from the fact that many of our python scripts are written for python2, and rossa doesn't have many dependencies installed for python2. I see no reason to build these dependencies on rossa for python2, we should not have to work with an unsupported language. But at the same time, I don't want to completely wipe all our python2 scripts, and make them python3, because this would involve a lot of tedious testing that I'm not prepared to undertake at the moment (the problem is compounded by the fact that pianosa does not have many dependencies installed for python3).

So what I have done in the interim is make python3 versions of the most important scripts I need to get the PRFPMI locking working - they are in the scripts directory and have the same names as their python2 counterparts, but have a 3 appended to their names. So when working on rossa, these are the scripts that are called. Eventually, after a lot more testing, we can depracate the old scripts. Currently, where applicable, the MEDM screens allow for either the python2 or python3 version of the script to be called.

Please, for the time being, do not try and install any new packages on rossa unless you are prepared to debug any problems caused and return the machine to a workable state. If you find some issue with a missing package on rossa, (i) make a note of it on the elog, and (ii) if possible, set up your own conda environment for testing and install dependencies to that environment only.

  15471   Sun Jul 12 02:42:01 2020 gautamUpdateLSCLocking (on rossa)

Main goals tonight were:

  1. Check if I can lock the interferometer by working on rossa - indeed, I could! It is much snappier than the ageing pianosa. The viewing angle of the CRT monitors from this corner isn't so good though.
  2. Measure step responses for the arm ASC loops to see if any insight can be gained into these loops. Analysis forthcoming...
Attachment 1: ASCsteps.png
ASCsteps.png
  15472   Sun Jul 12 22:40:35 2020 gautamUpdateElectronicsWFS characterization - old SURF report

After some hunting, I found this old SURF report with the WFS head measurements. The y-axes don't make much sense to me, and I can't find the actual data anywhere (her wiki page doesn't actually exist). So I think it's still unknown if these heads ever had the advertised transimpedance gain, or if the measured transimpedance of ~1kohm was what it always was.

  15475   Mon Jul 13 12:37:05 2020 gautamUpdateComputersrossa: more developmental work

In fact, all these utilities are now available in python3. There may be some bugs (e.g. this), but I've checked basic functionality and things look usable enough for development to proceed. While we can have a python2 env on rossa, I think it's unnecessary.

Quote:

I too, would prefer py3 for everything, but aren't all the cdsutils / gaurdian things still python2?

  15476   Tue Jul 14 00:06:09 2020 gautamUpdateLSCLocking with POX for CARM

I tried using the POX_I error signal for the DC CARM_B path today a couple of times. Got to a point where the AO path could be engaged and the arm powers stabilized somewhat, but I couldn't turn the CARM_A path off without blowing the lock. Now the IMC has entered a temperemental state, so I'm abandoning efforts for tonight, but things to try tomorrow are:

  1. Check that the demod phase is set correctly
  2. With the CARM_B path engaged, measure some CARM OLTFs. Tonight, I was a bit over-optimistic I think, by expecting the scripted transition to take me all the way, but I think I'll have to fiddle around with the gains a bit.
  3. Check for offsets. The AO path should be AC coupled, but maybe the POX signal has some offset?

I have some data from a couple of days ago when the PRFPMI was locked as usual (CARM_B on REFL for both DC and AO paths), and the sensing lines were on, so I can measure the relative strength of the sensing lines in POX/REFL and get an estimate of what the correct digital gain should be.

The motivation here is to see if the sensing matrix looks any different with a modified locking scheme.

ELOG V3.1.3-