For all the loops where we drive the NPRO PZT, there is some notch/resonance feature due to the PZT mechanical resonance. In the IMC loop this limits the PZT/EOM crossove to be less than 25 kHz. I don't have a model for this, btu it should be included.

If you hunt through the elogs, people have measured the TF of ALS NPRO PZT to phase/frequency. Probably there's also a measured ALS PDH loop somewhere that you could use to verify your model.

This looks cool, we should have something similar, can be really useful.

AUX PDH Loop update

I used D1400293 to get the latest logged details about the universal PDH box used to lock the green laser at X end. The uPDH_X_boost.fil file present there was used to obtain the control model for this box. See attachment one for the code used. Since there is a variable gain stage in the box, I tuned the gain of the filter model F_AUX in ALS_controls.yml to get the maximum phase margin in the PDH lock of the green laser. Unity gain frequency of 8.3 kHz can be achieved in this loop and as Gautam pointed out earlier, it can't be increased much further without changes in the box.

ALS Noise Budget update

The ALS control model remains stable with a reduction in total estimate noise because of the above update. There are few things to change though:

• This model is for a single arm locking where the beatnote signal between green laser and frequency doubled main laser is fed back to ETM at X end. Currently, Gautam is using a different scheme to lock where the feedback is sent to PSL-MC loop and the beat is taken between IR signals.
• In the LSC controls, I couldn't find a place where the digital ALS filter I have been optimizing and Kiwamu used, was placed. From what I gathered, after demodulation of beat note signal, a digital PLL is employed and the error signal is few to the Servo Filters directly. I might be missing some script which specifically switches on a particular set of filter modules in the XARM/YARM path when arms are locked through ALS.
• Another straight forward job for me is to verify the PSL-MC loop parameters with he TTFSS used. I'll do this next.

Tue evening from 4pm~6pm, Koji made a social distant tour for Anchal. We were present around the PSL/AS/ETMX tables.

I have placed 3 new in box, IDP 7 forepumps along the x arm of the interferometer. These are to be used as spares for both the 40m and Clean and Bake.

LiYuan has kindly done some Total Integrating Sphere (TIS) measurements on ITMU01 and ITMU02. A summary of the measurement is attached. I uploaded the measurements and some analysis script to nodus at /home/export/home/40m_TIS. I created a Wiki page for the measurements and linked to it from the core optics page.

These TIS measurements look very similar to the TIS of the LIGO optics. Further analysis shows that the scatter loss is 10+/-1.7 ppm for ITMU01 and 8.6+/-0.4 ppm for ITMU02.

In this calculation, a gaussian beam the same size bouncing off the 40m ITMs is assumed to scatter from the mirrors. The error is calculated by moving the beam around randomly with STD of 1mm.

In LiYuan's setup, TIS is measured for scattering angles between 1 and 75 degrees. If we go further and assume that the scatter is Lambertian we can extrapolate that the total loss is 10.9+/-1.9 ppm for ITMU01 and 9.2+/-0.5 ppm for ITMU02.

These measurements complete the loss budget nicely since with the 6ppm loss predicted from the phase maps, the total loss in the arm cavities would be 6+10+10=26ppm which is very close to the 28ppm loss that was measured after the arm cavity optics were cleaned.

We received 20pcs of stuffed demodulator boards from Screaming Circuits today. Some caveats:

1. The AP1053 amplifiers weren't stuffed. Note that this part is no longer in standard production, and lead time for a custom run is ~half a year. I recommend stuffing R2 and using a minicircuits amplifier upstream of this board. We have 6 pcs of AP1053 in hand so we can use those for the first AS WFS, but a second WFS will require some workaround.
2. AD8306ARZ weren't sent to Screaming Circuits. This part is used for the LO and RF signal level detection/monitoring stage, and so aren't crucial to the demodulation operation. @Chub, did we order the correct part now? They are rather pricey so maybe we can just adapt the footprint using some adaptor board?
3. DQS-10-100 hybrid 90 degree splitters were delivered to us after the lot was sent to Screaming Circuits. We have the pieces in hand, so we can just stuff them as necessary.

I removed 1 from the group to stuff some components that weren't sent to Screaming Circuits and test the functionality on the benchtop, the remaining have been stored in a plastic box for now as shown in Attachment #1. The box has been delivered to Chub who will stuff the remaining 19 boards once I've tested the one piece.

🤘🤘🤘

Summary:

The simple interferometer, composed of a single bounce reflection from ITMY and the LO beam deilvered via fiber to the AS table, can be locked - i.e. the phase of the LO beam can be controlled such that the DC light level on the DCPDs after the two beams are interfered can be stabilized. This test allows us to confirm that various parts of the sensing and actuation chain (e.g. PI PZT for homodyne phase control, Trek amplifier etc etc) are working.

I will post more quantitative analysis tomorrow.

Optical configuration:

• LO beam is a pickoff of the main PSL beam from just before it goes into the vacuum. The optical power arriving on each DCPD after the various beamsplitters, coupling loss etc is ~200 uW.
• IFO beam is the single bounce reflection from ITMY. For this test, ETMY, ITMX and ETMY are misaligned. Optical power arriving on each DCPD is ~80uW.
• The two beams are interfered on a 50-50 beamsplitter. The mode-matching efficiency was estimated to be ~50% which isn't stellar, but should be fine for this test.
• So, at half-fringe, we expect the signal on each DCPD to be linearly proportional to the phase difference between the two fields, and so we can use that as an error signal.

Servo topology:

Attachment #2 shows the servo topology.

• For a first attempt to close the feedback loop, we can consider the two blocks labelled "Sensing Chain" and "Actuation chain" to have a flat frequency response. While this isn't true, for a taget loop with ~100 Hz UGF, I think the approximation is reasonable.
• From the peak-to-peak value (160 cts) of the DCPD signals when the homodyne phase is uncontrolled, I estimate a sensing response (at half-fringe) of approximately 0.3 ct/nm, since this corresponds to 532nm of relative phase between the two beams. 
• An inverting summing amplifier is used to map the +/- 2^15 ct DAC range to 0-125V on the PI PZT. Assuming the full stroke of the PZT is 10um per the datasheet, and that this voltage range drives half of the full stroke (this is just a guess since all the old PI PZT circuits were designed to work at 0-250 V), we get an actuation coefficient of 0.075 nm/ct.
• Using these two numbers, we can then design a digital feedback loop that gives an open loop transfer function with ~100 Hz UGF, and sufficient stability margin.
• From the earlier measurements, we have an estimate for the amount of phase fluctuations caused by (i) seismic disturbances and (ii) fiber phase noise. This is the quantity we wish to suppress, and the suppression factor will be 1/(1+L), where L is the open loop gain.
• I didn't do this in any systematic way, but the loop in Attachment #3 seemed like a reasonable shape that would suppress the error signal RMS by ~10x, as shown in Attachment #4. So I decided to try this out.

Other notes:

1. The idea of offloading the DC control voltage to the ITMY suspension seemed to work fine.
2. It also seems like the relative phase between the two beams doesn't drift by so large an amount in short time scales, at least at night/quiet seismic conditions. So it is possible to maintain the lock for several seconds without having to offload the DC signal to the suspensions.
3. I didn't bother adapting the FSS Slow PID script to do this offloading in an automated way, seemed like more trouble than was just doing it by hand. But we may want to automate this in the future.
4. I couldn't make a clean measurement of the loop transfer function using the usual IN1/IN2 method. Introducing a step offset at the error point, the servo is able to track it (I didn't fit the step response time, but it's not as if the loop bandwidth is <1 Hz or something). I have to compare the measured in-loop error signal ASD to the free-running one to get a feel for what the UGF is, I guess, to rule out a weird loop.
5. Update 1100 Oct 6 2020: I have now added measured, in-loop, error point spectra to Attachment #4. Looks like there might be significant sensing noise re-injection.
   • Initially, I forgot to turn the HEPA on the PSL down for the measurement. So I have the two traces to compare. Looks like with the HEPA turned up to full, there is more noise in the 50-200 Hz range.
   • The trace marked "highGain" was taken with an overall loop gain that was 3dB higher than the nominal value - I could see some oscillations start to appear, and in the spectrum, maybe the feature at ~150 Hz is evidence of some gain peaking?

Conclusions:

1. The PI PZT seems to work just fine.
2. Need to look into the loop shape. I guess it's not reasonable to expect a UGF much higher than 100-200 Hz, because of the various delays in the system, but maybe the low frequency suppression can be made better.
3. What are the next steps?? What does this mean for the RF44 sensing scheme?

Summary:

After the earthquake on September 19 2020, it looks to me like the only lasting damage to suspensions in vacuum is the ETMY UR magnet being knocked off. 

Suspension ringdown tests:

I did the usual suspension kicking/ringdown test:

• One difference is that I now kick the suspension "N" times where N is the number of PSD averages desired. 
• After kicking the suspension, it is allowed to ring down with the damping disabled, for ~1100 seconds so that we can get spectra with 1mHz resolution.
• We may want to get more e-folding times in, but since the Qs of the modes are a few hundred, I figured this is long enough.
• I think this kind of approach gives better SNR than letting it ringdown 10,000 seconds (for 10 averages with 10 non overlapping segments of 1000 seconds), and I wanted to test this scheme out, seems to work well.
• Attachment #1 shows a summary of the results.
• Attachment #2 has more plots (e.g. transfer function from UL to all other coils), in case anyone is interested in more forensics. The data files are large but if anyone is interested in the times that the suspension was kicked, you can extract it from here.

Conclusions:

1. My cursory scans of the analysis don't throw up any red flags (apart from the known problem of ETMY UR being dislodged) 👌 . 
2. The PRM data is weird 
   • I believe this is because the DC bias voltage to the coils was significantly off from what it normally is when the PRC is aligned.
   • In any case, I am able to lock the PRC, so I think the PRM magnets are fine.
3. The PRC angular FF no longer works turns out this was just a weird interaction with the Oplev loop because the beam was significantly off-centered on the Oplev QPD. Better alignment fixed it, the FF works as it did before.
   • With the PRC locked and the carrier resonant (no ETMs), the old feedforward filters significantly degrade the angular stability to the point that the lock is lost.
   • My best hypothesis is that the earthquake caused a spot shift on PR2/PR3, which changed the TF from seismometer signal to PRC spot motion.
   • Anyways, we can retrain the filter.
   • The fact that the PRC can be locked suggest PR2/PR3 are still suspended and okay.
4. The SRM data is also questionable, because the DC bias voltage wasn't set to the values for an aligned SRC when the data was collected
   • Nevertheless, the time series shows a clean ringdown, so at least all 5 OSEMs are seeing a signal.
   • Fact that the beam comes out at the AS port suggest SR3/SR2 suspensions are fine 👍 

Attachment #2 also includes info about the matrix diagonalization, and the condition numbers of the resulting matrices are as large as ~30 for some suspensions, but I think this isn't a new feature. 

Attachment #1 shows that the c1rfm model isn't able to receive any signals from the front end machines at EX and EY. Attachment #2 shows that the problem appears to have started at ~430am today morning - I certainly wasn't doing anything with the IFO at that time.

I don't know what kind of error this is - what does it mean that the receiving model shows errors but the sender shows no errors? It is not a new kind of error, and the solution in the past has been a series of model reboots, but it'd be nice if we could fix such issues because it eats up a lot of time to reboot all the vertex machines. There is no diagnostic information available in all the places I looked. I'll ask the CDS group for help, but I'm not sure if they'll have anything useful since this RFM technology has been retired at the sites (?).

In the meantime, arm cavity locking in the usual way isn't possible since we don't have the trigger signals from the arm cavity transmission. 

Update 1500 4 Oct: soft reboots of models didn't do the trick so I had to resort to hard reboots of all FEs/expansion chassis. Now the signals seem to be okay.

2x500 ml bottles of spectroscopic grade isopropanol were delivered. I marked them with today's date and placed them in the solvent cabinet. In the process, my shoulder bumped the laser interlock switch by the door to the VEA in the drill press area, which turned the PSL NPRO off. I turned it back on just now. The other NPROs are not connected to the interlock and so were unaffected.

Attachment #1 shows that the ITMX, ETMY and beamsplitter Oplev light levels have decayed significantly from their values when installed. In particular, the ETMY and ITMX sum channels are now only 50% of the values when a new HeNe was installed. ELOG search revealed that ITMY and ETMX HeNes were replaced with newly acquired units in March and September of last year respectively. The ITMX oplev was also replaced in March 2019, but the replacement was a unit that was being used to illuminate our tourist attraction glass fiber at EX.

We should replace these before any vent as they are a useful diagnostic for the DC alignement reference.

The optomechanics stock in the lab was in a sad state. We have obtained the following from Thorlabs in the last two months:

1. 6 pcs each of DT12 and DT12B compact translation stages (for lens mode matching).
2. 3pcs each KM100PM + PM3 cube beamsplitter mounts (for polarization control).
3. 1 Post / spacer kit for height adjustment.
4. 3 pcs ea of K6XS + AD11F + F220APC for fiber applications.

I have used some of these for the ari BHD setup. The unused items are stored in the shelves that house the optomechanics ~halfway down the east arm. I'm wondering what's a good setup to document the stock of this stuff so we can always have a healthy stock of optomechanics (at least the non speciality ones like posts, spacers etc). It sucks to realize at 0000hrs that you're missing a 3mm shim or 250mm converging lens or something.

it would be a good idea for us to have an auto-reminder to have us check the flow of all the HEPAs in the lab and elog it once a year so that we can replace filters and pre-filters appropriately.

[JV, GV]

The HEPAs work again. After running the HEPAs for ~1 hour, I checked the particle count on the PSL table - the meter registered 0 for both 0.3 um and 0.5 um. So I decided to turn the NPRO back on, at ~1730 local time. The PMC and IMC were readily locke, so the basic interferometer functionality is returned, and we can now go ahead with (i) vent prep (ii) air BHD tests and (iii) IMC debuggin as was discussed on the call today. The earth is shaking, but nothing serious so far, I will resume alignment of the interferometer later in the evening when hopefully things have calmed down a bit more...

Procedure:

1. Turned off mains switch on the NW corner of the PSL enclosure. Then, disconnected the power cables from mains to Variac and from Variac to HEPAs. Made sure both HEPAs were set to "OFF".
2. With confidence that no AC power was reaching the motors, I removed the pre-filters, and removed the old startup capacitors. These don't have a polarity, but I marked the cable that was connected to the left terminal of the capacitor when the cap is viewed with the label facing you, in the interest of changing as few things as possible.
3. The two new capacitors were measured with the LCR meter - the meter registered ~7.5 uF, as expected. Unsurprisingly, the old capacitors that were removed didn't register any reading on the LCR meter. The terminals weren't shorted, but I don't know what the failure mode for this kind of capacitor is.
4. The two new capacitors were installed. Then, I tested the system by undoing all the changes in bullet #1. We found that the Variac needs to be set to 100% for the motors to startup.
5. The motor speed was found to vary as the Variac dial was turned. FWIW, at the "nominal" setting of 33% on the Variac (when we run the interferometer), I could see both fan blades were turning, but the flow was low enough that you couldn't hear any wind (at least, neither Jordan nor I could).
6. Turned off the mains agian, and cleaned up - restored the insulating rubber sleeve on the capacitor leads, and re-installed the pre-filters on the HEPA blowers. Then we turned both blowers back on. 

Note that the many other issues Koji noted in the preceeding elog (e.g. flaky wiring) have not been addressed.

Flow measurements:

Chub kindly provided us with an electronic anemometer. With the meter held directly against the HEPA filter inside the enclosure, we measured ~700 cfm of airflow on each of the two HEPAs, with the Variac set to 100% and the HEPAs themselves set to "High". With the Variac at 50%, the flow drops to ~160 cfm. At the nominal setting of 33%, the meter didn't register any flow. I don't know what the spec'd flow rate is for this combination of blower + filter, but Jordan says similar units in Downs register ~1500 cfm at the "High" setting. The two protable (similarly sized) HEPA units in the 40m, one at ITMY and one at ETMY, register ~900 cfm and ~1100 cfm respectively, when set to high. So we may want to revisit what the "nominal" HEPA setting should be, in case the filters have become clogged over time. 

Some photos of the HEPA blowers with the pre-filters off and the capacitors switched out may be found here.

I was in the lab from 1630-1830. I have located and visually inspected all the parts required for the magnet regluing / optic cleaning parts of the planned vent, except the fresh batches of scpectroscopic grade solvents. I was in the cleanroom part of the clean and bake lab from 1630-1700.

I removed the forepump to TP2 this morning after the vacuum failure, and tested in the C&B lab. I pumped down on a small volume 10 times, with no issue. The ultimate pressure was ~30 mtorr.

I re-installed the forepump in the afternoon, and restarted TP2, leaving V4 closed. This will run overnight to test, while TP3 backs TP1.

In order to open V1, with TP3 backing TP1, the interlock system had to be reset since it is expecting TP2 as a backing pump. TP2 is running normally, and pumping of the main volume has resumed.

gautam 2030:

1. The monitor (LCD display) at the vacuum rack doesn't work - this has been the case since Monday at least. I usually use my laptop to ssh in so I didn't notice it so it could have been busted from before. But for anyone wishing to use the workstation arrangement at 1X8, this is not great. Today, we borrowed the vertex laptop to ssh in, the vertex laptop has since been returned to its nominal location.
2. The modification to the interlock condition was made by simply commenting out the line requiring V4 to be open for V1 to be opened. I made a copy of the original .yaml file which we can revert to once we go back to the normal config.
3. I also opened VM1 to allow the RGA scans to continue to be meaningful.
4. At the time of writing, all systems seem nominal. See Attachment #2. The vertical line indicates when we started pumping on the main volume again earlier today, with TP3 backing TP1.

​Unclear why the TP2 foreline pump failed in the first place, it has been running fine for several hours now (although TP2 has no load, since V4 isolates it from the main volume). Koji's plots show that the TP2 foreline pressure did not recover even after the interlock tripped and V4 was closed (i.e. the same conditions as TP2 sees right now).

Yes, that loop was unstable. I started using the time domain response to check for the stability of loops now. I have been able to improve the filter slightly with more suppression below 20 Hz but still poor phase margin as before. This removes the lower frequency region bump due to seismic noise. The RMS noise improved only slightly with the bump near UGF still the main contributor to the noise.

For inclusion of real spectra, time delays and the anti-aliasing filters, I still need some more information.

Few questions:

• What anti-aliasing filters are used in ALS?
• Is the digital delay fixed to a constant upper limit or is it left to change as per filters? I have already used a 470 us delay (modeled with Pade 4th order approximation).
• I could not find a good place where channel names are listed with corresponding meaning. Where can I find them?
• Is there a channel which keeps record of lock status? In short, how do I find the in-lock times

Additional Info:

The code used to calculate the transfer functions and plot them is in the repo 40m/ALS/noiseBudget

Related Elog post with more details: 40m/15587

Here is the timeline. This suggests TP2 backing RP failure.

1st line: TP2 foreline pressure went up. Accordingly TP2 P, current, voltage, and temp went up. TP2 rotation went down.

2nd line: TP2 temp triggered the interlock. TP2 foreline pressure was still high (10torr) so TP2 struggled and was running at 1 torr.

3rd line: Gautam's operation. TP2 was isolated and stopped.

Between the 1st line and 2nd line, TP2 pressue (=TP1 foreline pressure) went up to 1torr. This made TP1 current increased from 0.55A to 0.68A (not shown in the plot), but TP1 rotation was not affected.

The interlocks tripped at ~630am local time. Jordan reported that TP2 was supposedly running at 52 C (!).

V1 was already closed, but TP2 was still running. With him standing by the rack, I remotely exectued the following sequence:

• VM1 closed (isolates RGA volume).
• VA6 closed (isolates annuli from being pumped).
• V7 opened (TP3 now backs TP1, temporarily, until I'm in the lab to check things out further).
• TP2 turned off.

Jordan confirmed (by hand) that TP2 was indeed hot and this is not just some serial readback issue. I'll do the forensics later.

Dimensions / Specs

- HEPA unit dimentions
- HEPA unit manufacturer
- Motor
- Capacitor

Gautam reported that the PSL HEPA stopped running (ELOG 15592). So I came in today and started troubleshooting.

It looks like that the AC power reaches the motors. However, both motors do not run. It looks like the problem exists in the capacitors, the motors, or both.
Parts specs can be found in the next ELOG.

Attachment 1 is the connection diagram of the HEPA. The AC power is distributed by the breaker panel. The PSL HEPA is assigned to use M22 breaker (Attachment 2). I checked the breaker switch and it was (and is) ON. The power goes to the junction box above the enclosure (Attachment 3). A couple of wires goes to the HEPA switch (right above the enclosure light switch) and the output goes to the variac. The inside of the junction box looked like this (Attachment 4).

By the way, the wires were just twisted and screwed into a metal threaded (but isolated) caps (Attachment 5). Is this legit? Shouldn't we use stronger crimping? Anyway, there was nothing wrong with the caps w.r.t the connection for now.

I could easily trace the power up to the variac. The variac output was just fine (Attachment 6). The cord goes from the variac to the junction box (and then HEPAs) looked scorched. The connection from the plug to HEPAs was still OK, but this should be eventually replaced. Right now the cable was unplugged after the following tests for the safety reason.

The junction box for each HEPA unit was opened to check the voltage. The supply voltage came to the junction boxes and it was just fine. In Attachments 8 & 9, the voltages look low but this is because I just turned the variac only a little.

At the (main) junction box, the resistances of the HEPAs were checked with the Fluke. As the HEPA units are connected to the AC in parallel, the resistances were individually checked as follows.

The coils were not disconnected (... I wonder if the wiring of South HEPA was flipped? But this is not the main issue right now.)
 

By removing the pre-filters, the motors were inspected Attachments 10 & 11. At least the north HEPA motor was warm, indicating there was some current before. A capacitor was connected per motor. When the variac was tuned up a bit, one side of the capacitor could see the voltage. I could not judge which has the issue between the capacitor and the motor.

I got some feedback from Koji who pointed out that the phase tracker is not required here. This situation is similar to the phase locking of two lasers together, which we frequently do, except in that case, we usually we offset the absolute frequencies of the two lasers by some RF frequency, and we demodulate the resulting RF beatnote to use as an error signal. We can usually acquire the lock by simply engaging an integrator (ignoring the fact that if we actuate on the laser PZT, which is a frequency actuator, just a proportional feedback will be sufficient because of the phase->frequency conversion), the idea being that the error signal is frequently going through a zero-crossing (around which the sinusoidal error signal is approximately linear) and we can just "catch" one of these zero crossings, provided we don't run of actuation range.

So the question here becomes, is the RF44 signal a suitable error signal such that we can close a feedback loop in a similar way? To try and get more insight, I tried to work out the situation analytically. I've attached my thinking as a PDF note. I get some pretty messy complicated expressions for the RF44 signal contributions, so it's likely I've made a mistake (though Mathematica did most of the heavy lifting), it'll benefit from a second set of eyes. 

Anyways, I definitely think there is some additional complications than my simple field cartoon from the preceeding elog would imply - the relative phases of the sidebands seem to have an effect, and I still think the lack of the PRC/SRC make the situation different from what Hang/Teng et. al. outlined for the A+ homodyne phase control analysis. Before the HEPA failed, I had tried closing the feedback loop using one quadrature of the demodulated RF44 signal, but had no success with even a simple integrator as the loop (which the experience with the PLL locking says should be sufficient, and pretty easily closed once we see a sinusoidally oscillating demodulated error signal). But maybe I'm overlooking something basic conceptually?

I came to the lab. The control room AC was off -> Now it is on.

Here is the setting of the AC meant for continuous running

This ALS loop is not stable. Its one of those traps that comes from using only the Bode plot to estimate the loop stability. You have to also look at the time domain response - you can look at my feedback lecture for the SURF students for some functions.

This is not a reply to comments given to the last post; Still working on incorporating those suggestions.

Trying out a better filter from scratch

Rana suggested looking first at what needs to be suppressed and then create a filter suited for the noise from scratch. So I discarded all earlier poles and zeros and just kept the resonant gains in the digital filter. With that, I found that all we need is three poles at 1 Hz and a gain of 8.1e5 gives the lowest RMS noise value I could get.

Now there can be some practical reasons unknown to me because of which this filter is not possible, but I just wanted to put it here as I'll add the actual noise spectra into this model now.

Few questions:

• What anti-aliasing filters are used in ALS?
• Is the digital delay fixed to a constant upper limit or is it left to change as per filters? I have already used a 470 us delay (modeled with Pade 4th order approximation).
• I could not find a good place where channel names are listed with corresponding meaning. Where can I find them?
• Is there a channel which keeps a record of lock status? In short, how do I find the in-lock times

The HEPA filters on the PSL enclosure are no longer running. I tried cycling the power switch on the NW corner of the enclosure, and also turned the dial on the Variac down to zero and back up to maximum, no effect. Judging by the indicator LED on it, the power is making it to the Variac - bypassing the Variac and directly connecting the HEPA to the AC power seems to have no effect either.

I can't be sure, but I'm almost certain I heard them running at max speed an hour ago while I was walking around inside the VEA. Probably any damage that can happen has already been done, but I dialled down the Innolight injection current, and closed its shutter till the issue can be resolved.

I removed the forepump (Varian SH-110) for TP3 today to see why it had failed over the weekend. I tested it in the C&B lab and the ultimate pressure was only ~40torr. I checked the tip seals and they were destroyed. The scroll housing also easily pulled off of the motor drive shaft, which is indicative of bad bearings. The excess travel in the bearings likely led to significant increase in tip seal wear. This pump will need to be scrapped, or rebuilt.

I tested the spare Varian SH-110 pump located at the X-end and the ultimate pressure was ~98 mtorr. This pump had tip seals replaced on 11/5/18, and is currently at 55163 operating hours. It has been installed as the TP3 forepump.

Once installed, restarting the pump line occured as follows: V5 Closed, VA6 closed, VASE Closed, VASV closed, VABSSCI closed, VABS closed, VABSSCO closed, VAEV closed, VAEE closed,TP3 was restarted and once at normal operation, valves were opened in same order.

The pressure differential interlock condition for V5 was temporaily changed to 10 torr (by Gautam), so that valves could be opened in a controlled manner. Once, the vacuum system was back to normal state the V5 interlock condition was set back to the nominal 1 torr. Vacuum system is now running normally.

See Attachment #1.

• The original ETMY actuation matrix was the naive one so I simply retuned everything by adding the butterfly mode appropriately.
• The cross coupling between the DOFs has not been characterized, but the local damping, Oplev loops, POX/POY locking, simpel Michelson locking, and ASS dither alignemnt all seem to work so I'm thinking this is good enough for now.
• A copy of the ETMYaux.db file was made, and the slow bias voltage was redistributed among the three available face OSEMs - note that the magnet polarity has to be taken into account.
• The series resistance on the slow path was reduced from 430 ohms, 1W to 100 ohms, 3W, to allow DC alignment of the cavity axis to the beam axis.
• Vertex Oplevs were re-centered on their respective QPDs after locking POX/POY and running the ASS dither alignment.
• The AS spot was a little low on the camera - presumably the SR2/SR3 got macroscopically misaligned. I re-centered the beam on the camera on the AS table (by moving the camera, the beam path was not disturbed).

The beam spot on ETMY looks weird (looks almost like a TEM10 mode), but the one on ITMY seems fine, see Attachment #2. Wonder what's going on there, maybe just a focusing problem?

I think the digital loop in the ALS budget is too optimistic. You have to include all the digital delays and anti-aliasing filters to get the real response.

aslo, I recommend grabbing some of the actual spectra from the in-lock times with nds and using the calibrated spectra as inputs to this mode. Although we don't have good models of the stack, you can sort of infer it by using the calibrated seismometer data and the calibrated MC_F or MC_L channels (for IMC) or XARM/YARM signals for those.

While I stopped by the lab this morning to pick up some things, I took the opportunity to continue the recovery.

• IMC suspensions were sufficiently misaligned that the autolocker couldn't re-acqurie the lock. I manually recovered the alignment and now the IMC is locked again.
• ETMY illuminator was left ON, I turned it off. In the process, I modified the illuminator ON/OFF script to be compatible with python3, but unfortunately, it was written in a way that doesn't permit backward compatibility, so now the illuminators can't be turned ON/OFF via the MEDM screen on pianosa (since the default python is 2.7 on that machine). But it does work on rossa, which I'm using as my primary workstation now (hence the change).
• ITMX watchdog trip threshold was manually reset to the nominal value - the rampdown script was working, but the threshold was ~1400cts (normally ~200 cts) even at 1130am this morning (>12 hours after Koji's work yesterday evening), so I just accelerated the process.
• Suspension realignment - using a mix of green and IR beams and the various cameras/photodiodes as diagnostics, I roughly restored the alignment of all the suspensions, except ETMY. I can see IR resonances in the X arm now.

At some point, we should run the suspension eigenmode routine (kick optics, let them ringdown, measure peak locations and Qs) to confirm that the remaining suspensions are okay, will also help in actuation re-allocation efforts on ETMY. But I didn't do this today.

Leaving the lab at 1150.

Setting the record straight

I found out an error I did in copying some control model values from Kiwamu's matlab code. On fixing those, we get a considerably reduced amount of total noise. However, there was still an unstable region around the unity gain frequency because of a very small phase margin. Attachment 3 shows the noise budget, ALS open-loop transfer function, and AUX PDH open-loop transfer function with ALS disengaged. Attachment 4 is the yaml file containing all required zpk values for the control model used. Note that the noise budget shows out-of-loop residual arm length fluctuations with respect to PSL frequency. The RMS curve on this plot is integrated for the shown frequency region.

Trying to fix the unstable region

Adding two more poles at 100 Hz in the ALS digital filter seems to work in making the ALS loop stable everywhere and additionally provides a steeper roll-off after 100 Hz. Attachment 1 shows the noise budget, ALS open-loop transfer function, and AUX PDH open-loop transfer function with ALS disengaged. Attachment 2 is the yaml file containing all required zpk values for the control model used. Note that the noise budget shows out-of-loop residual arm length fluctuations with respect to PSL frequency. The RMS curve on this plot is integrated for the shown frequency region.

But is it really more stable?

• I tried to think about it from different aspects. One thing is sure that  remains greater than 1 in all of the frequency region plotted for. This is also evident in the common-mode to residual noise transfer function which shows no oscillation peaks and is a clean mirror image of the open-loop transfer function (See Attachment 1, page 2).
• Another way is to look for the phase margin. This is a little controversial way of checking stability. For clarity, the open-loop transfer function I'm plotting does not contain the '-1' feedback in it. So the bad phase value at unity gain frequency is -180 degrees (or 180 degrees) for us. I've taken the difference from the closest side and got 76.2 degrees of phase margin.
• Another way I checked was by plotting a Nyquist plot for the open-loop transfer function. It is said that if the contour does not encircle the point '-1' in the real axis, then the loop would be stable even if the where is the frequency where phase lag becomes -180 degrees at the lowest frequency. For us, is at 1 Hz because of the test mass actuator pole. But I have verified that the Nyquist contour of the open-loop transfer function does not encircle '-1' point. I have not uploaded the Nyquist plot as it is not straight forward to plot. Because of large dc gain, it covers a large region and one needs to zoom in and out to properly follow what the contour is really doing. I didn't get time to make insets for it.

Is this close to reality?

For that, we'll have to take present noise source estimates but Gautum vaguely confirmed that this looked more realistic now 'shape-wise'. If I remember correctly, he mentioned that we currently can achieve 8 pm of residual rms motion in the arm cavity with respect to the PSL frequency. So we might be overestimating our loop's capability or underestimating some noise source. More feedback on this welcome and required.

Additional Info:

The code used to calculate the transfer functions and plot them is in the repo 40m/ALS/noiseBudget

Attachment 5 here shows a block diagram for the control loop model used. Output port 'Res_Disp' is used for referring all the noise sources at the residual arm length fluctuation in the noise budget. The open-loop transfer function for ALS is calculated by -(ALS_DAC->ALS_Out1 / ALS_DAC->ALS_Out2) (removing the -1 negative feedback by putting in the negative sign.) While the AUX PDH open-loop transfer function is calculated by python controls package with simple series cascading of all the loop elements.

Disconcerting because those tip seals were just replaced [15417]. Maybe they were just defective, but if there is a more serious problem with the pump, there is a spare Varian roughing pump (the old TP2 dry pump) sitting at the X-end.

I reset the interlock error to unfreeze the vac controls (leaving V5 closed).

There were two SUSs which didn't look normal.

- ITMX was easily released by the bias slider -> Shake the pitch slider and while all the OSEM values are moving, turn on the damping control (with x10 large watchdog threshold)

- ETMY has UR OSEM 0V output. This means that there is no light. And this didn't change at all with the slider move.
- Went to the Y table and tried to look at the coils. It seems that the UR magnet is detached from the optic and stuck in the OSEM.

We need a vent to fix the suspension, but until then what we can do is to redistribute the POS/PIT/YAW actuations to the three coils.

I supplied a bottle of hand soap. Don't put water in the bottle to dilute it as it makes the soap vulnarable for cotamination.

the EQ was ~14 km south of Caltech and 17 km deep

I'm amazed at how much higher the noise is on the MC2 accelerometer. Is that really how much amplification of the ground motion we're getting? If so, its as if the MC has no vibration isolation from the ground in that band. We should put one set on the ground and make the more direct comparison of the spectra. Also, perhaps do some seismic FF using this sensor - I'm not sure how successful we've been in this band.

Attaching the coherence plot from ldvw.ligo.caltech.edu (apparently it has access to the 40m data, so we can use that as an alternative to dtt or python for remote analysis):

It would be interesting to see if we can use the ML based FF technology from this summer's SURF project by Nadia to increase the coherence by including some slow IMC alignment channels.

I came to the campus and Gautam notified that he just had received the alert from the vac watchdog.

I checked the vac status at c1vac. PTP3 went up to 10 torr-ish and this made the diff pressure for TP3 over 1torr. Then the watchdog kicked in.

To check the TP3 functionality, AUX RP was turned on and the manual valve (MV in the figure) was opened to pump the foreline of TP3. This easily made PTP3 <0.2 torr and TP3 happy (I didn't try to open V5 though).

So the conclusion is that RP for TP3 has failed. Presumably, the tip-seal needs to be replaced.

Right now TP3 was turned off and is ready for the tip-seal replacement. V5 was closed since the watchdog tripped.

the seismometers obviously saturated during the EQ, but the accelerometers captured some of it. It looks like there's different saturation levels on different sensors.

Also, it seems the mounting of the MC2 accelerometers is not so good. There's some ~10-20 Hz resonance its mount that's showing up. Either its the MC2 chamber legs or the accelerometers are clamped poorly to the MC2 baseplate.

Sun Sep 20 00:02:36 2020 edit: fixed indexing error in plots

* also assuming that the sensors are correctly calibrated in the front end to 1 count = 1 um/s^2 (this is what's used in the summ pages)

M4.5 EQ in LA 2020-09-19 06:38:46 (UTC) / -1d 23:38:46 (PDT) https://earthquake.usgs.gov/earthquakes/eventpage/ci38695658/executive

I only checked the watchdogs. All watchdogs were tripped. ITMX and ETMY seemed stuck (or have the OSEM magnet issue). They were left tripped. The watchdogs for the other SUSs were reloaded.

eSummary:

I don't think the proposed scheme for sensing and controlling the homodyne phase will work without some re-thinking of the scheme. I'll try and explain my thinking here and someone can correct me if I've made a fatal flaw in the reasoning somewhere.

Field spectrum cartoon:

Attachment #1 shows a cartoon of the various field components.

• The input field is assumed to be purely phase modulated (at 11 MHz and 55 MHz) creating pairs of sidebands that are in quadrature to the main carrier field.
• The sideband fields are drawn with positive and negative imaginary parts to indicate the relative negative sign between these terms in the Jacobi-Anger expansion.
• For our air BHD setup, the spectrum of the LO beam will also be the same.
• At the antisymmetric (= dark) port of the beamsplitter, the differential mode signal field will always be in the phase quadrature.
• I'm using the simple Michelson as the test setup:
  • The ITMs have real and (nearly) identical reflectivities for all frequency components incident on it.
  • The sideband fields are rotated by 90 degrees due to the i in the Michelson transmission equation.
  • The Schnupp asymmetry preferentially transmits the 55 MHz sideband to the AS port compared to the 11 MHz sideband - note that in the simple Michelson config, I calculate T(11 MHz) = 0.02%, T(55 MHz) = 0.6% (both numbers not accounting for the PRM attenuation).
• I think the cartoon Hang drew up is for the DRFPMI configuration, with the SRC operated in RSE.
  • The main difference relative to the simple Michelson is that the signal field picks up an additional 90 degrees of phase propagating through the SRC.
• For completeness, I also draw the case of the DRFPMI where the SRC is operated at nearly the orthogonal tuning.
  • I think the situation is similar to the simple Michelson

So is there a 90 degree relative shift between the signal quadrature in the simple Michelson vs the DRFPMI? But wait, there are more problems...

Closing a feedback loop using the 44 MHz signal:

We still need to sense the 44 MHz signal with a photodiode, acquire the signal into our CDS system, and close a feedback loop.

• The 44 MHz signal is itself supposed to be generated by the interference between the TEM00 55 MHz sideband from the IFO output with the TEM00 11 MHz sideband from the LO field (let's neglect any mode mismatch, HOMs etc for the moment).
• By splitting this beat signal photocurrent in two, mixing each part with an electrical 44 MHz signal, and digitizing the IF output of said mixers, we should in principle be able to reconstruct the magnitude and phase of the signal.
• The problem is that we know from other measurements that this signal is going to go through multiple fringes, and hence, we don't have a signal that is linear in the quantity we would like to control, namely the homodyne phase (either quadrature signal can be a candidate linear signal around a zero crossing, but when the signals are going through multiple fringes, neither signal stays linear).
• One possible way to get around this problem is to use a phase tracker servo - basically, close a purely digital feedback loop, using one of the demodulated quadratures as an error signal, and changing the demodulation phase digitally such that the signal stays entirely in the orthogonal quadrature. However, such a scheme relies on the signal magnitude remaining constant. If the "error signal" goes to zero for multiple reasons (rotation out of the quadrature being considered, or just that the signal itself goes to zero), then this technique won't work. Of course, the phase tracker doesn't know what the "phase" of the signal is, when it's magnitude is (nearly) zero.
• It is true that we always expect a "background" level of 44 MHz signal, from the 11 MHz and 55 MHz sidebands in the LO beam directly interfering, but this doesn't contain any useful information, and in fact, it'd only contaminate the phase tracker error signal I think.
• So we can't rely on the error staying in one quadrature (like we do for the regular IFO PDH signals, where there is no relative phase propagation between the LO and RF sideband optical fields and so once we set the demodulation phase, we can assume the signal will always stay in that quadrature, and hence we can close a feedback loop), and we can't track the quadrature. What to do? I tried to dynamically change the phase tracker servo gain based on the signal magnitude (calculated in the RTCDS code using sqrt(I**2 + Q^2), but this did not yield good results...

Next steps:

I don't have any bright ideas at the moment - anyone has any suggestions?🤔

Aside:

I wanted to check what kind of signal the photodiode sees when only the LO field is incident on the photodiode. So with the IFO field blocked, I connected the PDA10CF to the Agilent analyzer in "Spectrum" mode, through a DC block. The result is shown in Attachment #2. To calculate the PM/AM ratio, I assumed a modulation depth of 0.2. The RIN was calculated by dividing the spectrum by the DC value of the PDA10CF output, which was ~1V DC. The frequencies are a little bit off from the true modulation frequencies because (i) I didn't sync the AG4395 to a Rb 10 MHz signal, and (ii) the span/BW ratio was set rather coarsely at 3kHz.

I would expect only 44 MHz and 66 MHz peaks, from the interference between the 11 MHz and 55 MHz sideband fields, all other field products are supposed to cancel out (or are in orthogonal quadratures). This is most definitely not what I see - is this level of RIN normal and consistent with past characterization? I've got no history in this particular measurement. 

I had to make a CDS change to the c1lsc model in an effort to get a few more signals into the models. Rather than risk requiring hard reboots (typcially my experience if I try to restart a model), I opted for the more deterministic scripted reboot, at the expense of spending ~20mins to get everything back up and running.

Update 2230: this was more complicated than expected - a nuclear reboot was necessary but now everything is back online and functioning as expected. While all the CDS indicators were green when I wrote this up at ~1800, the c1sus model was having frequent CPU overflows (execution time > 60 us). Not sure why this happened, or why a hard power reboot of everything fixed it, but I'm not delving into this.

The point of all this was that I can now simultaneously digitize 4 channels - 2 DCPDs, and 2 demodulated quadratures of an RF signal.

Assembled is the list of dead pressure gauges. Their locations are also circled in Attachment 1.

For replacements, I recommend we consider the Agilent FRG-700 Pirani Inverted Magnetron Gauge. It uses dual sensing techniques to cover a broad pressure range from 3e-9 torr to atmosphere in a single unit. Although these are more expensive, I think we would net save money by not having to purchase two separate gauges (Pirani + hot/cold cathode) for each location. It would also simplify the digital controls and interlocking to have a streamlined set of pressure readbacks.

For controllers, there are two options with either serial RS232/485 or Ethernet outputs. We probably want the Agilent XGS-600, as it can handle all the gauges in our system (up to 12) in a single controller and no new software development is needed to interface it with the slow controls.

This elog suggests that there is uniformly 1 stage engaged across all channels. I didn't look at the board to see what the jumper situation is, but only 1 stage of whitening is compensated digitally for both _F and _L. The Pomona box attached to the NPRO PZT input is also compensated digitally to convert counts to frequency.

I tried the gain re-allocation between VCO gain and FSS COMM (and also compensated for the cts to Hz conversion in MCF), but it doesn't seem to have the desired effect on the MCF SNR in the 5-50Hz band. Since the IMC stays locked, and I had already made the changes to mcup, I'll keep these gains for now. We can revert to the old settings if the IMC locking duty cycle is affected. Explicitly, the changes made were:

VCO gain: +7dB ---> +13 dB

FSS COMM: +6 ddB ---> +0 dB

The mcdown script wasn't modified, so the lock acquisition gains are the same as they've been. 

Summary:

After more trials, I think the phase tracker part used to provide the error signal for this scheme needs some modification for this servo to work.

Details:

Attachment #1 shows a block diagram of the control scheme.

I was using the "standard" phase tracker part used in our ALS model - but unlike the ALS case, the magnitude of the RF signal is squished to (nearly) zero by the servo. But the phase tracker, which is responsible for keeping the error signal in one (demodulated) quadrature (since our servo is a SISO system) has a UGF that is dependent on the magnitude of the RF signal. So, I think what is happening here is that the "plant" we are trying to control is substantially different in the acquisition phase (where the RF signal magnitude is large) and once the lock is enabled (where the RF signal magnitude becomes comparitively tiny).

I believe this can be fixed by dynamically normalizing the gain of the digital phase tracking loop by the magnitude of the signal = sqrt(I^2 + Q^2). I have made a modified CDS block that I think will do the job but am opting against a model reboot tonight - I will try this in the daytime tomorrow. 

I'm also wondering how to confirm that the loop is doing something good - any ideas for an out-of-loop monitor? I suppose I could use the DCPD - once the homodyne phase loop is successfully engaged, I should be able to drive a line in MICH and check for drift by comparing line heights in the DCPD signal and RF signal. This will requrie some modification of the wiring arrangement at 1Y2 but shouldn't be too difficult...

The HEPAs, on the PSL table and near ITMY, were dialled down  / turned off respectively, at ~8pm at the start of this work. They will be returned to their previous states before I leave the lab tonight.

that's a very curious disconnection

the "Pentek" whitening board that carries the MC channels has jumpers to enable either 1 or 2 stages of 15:150 whitening. Looks lik MC_F has 2 and MC_L has 1.

I guess the MC_F signal is so low because of the high gain on the FSS board.  We could lower the FSS common gain and increase the IMC board's VCO gain to make up for this. Maybe 6 dB would be enough. IF that is risky, we could also up the analog gain on the whitening board.

There was an abrupt change in the MC_F spectrum between August 4 and August 5, judging by the summary pages - the 1 and 3 Hz resonances are no longer visible in the spectrum. Possibly, this indicates some electronics failure on the MC servo board / whitening board, the CDS settings don't seem to have changed. There is no record of any activity in the elog around those dates that would explain such a change. I'll poke around at 1X2 to see if anything looks different.

Update 1740: I found that the MCL / MCF cables were disconnected. So since August 5, these channels were NOT recording any physical quantity. Because their inputs weren't terminated, I guess this isn't a clean measurement of the whitening + AA noise, but particularly for MC_F, I guess we could use more whitening (see Attachment #1). Probably also means that the wandering ~10-30Hz line in the spectrogram is a electronics feature. The connections have now been restored and things look nominal again.

These were delivered to the 40m today and are on Rana's desk

The unit was repaired and returned to the 40m. Now, with a DMM, I measure a DC offset value that is ~1% of the AC signal amplitude. I measured the TF of a simple 1/20 voltage divider and it looks fine. In FFT mode, the high frequency noise floor levels out around 5-7nV/rtHz when the input is terminated in 50 ohms.

I will upload the repair documents to the wiki.

The PMC has been unlocked since September 11 sometime (summary pages are flaky again). I re-locked it just now. I didn't mess with the HEPA settings for now as I'm not using the IFO at the moment, so everything should be running in the configuration reported here. The particulate count numbers (both 0.3um and 0.5um) reported is ~x5-8 of what was reported on Thursday, September 10, after the HEPA filters were turned on. We don't have logging of this information in any automated way so it's hard to correlate things with the conditions in Pasadena. We also don't have a working gauge of the pressure of the vacuum envelope.

The RGA scanning was NOT enabled for whatever reason after the vacuum work. I re-enabled it, and opened VM1 to expose the RGA to the main volume. The unit may still be warming up but this initial scan doesn't look completely crazy compared to the reference trace which is supposedly from a normal time based on my elog scanning (the timestamp is inherited from the c0rga machine whose clock is a bit off).

Update 1500: I checked the particle count on the PSL table and it barely registers on the unit (between 0-20 between multiple trials) so I don't know if we need a better particle coutner or if there is negligible danger of damage to optics due to particulate matter.