It's not so easy to directly measure this I think, because the filtering is rather aggressive. Attachment #1 shows the measured transfer function (dots) vs the model and Attachment #2 shows the noise. I think this checks out - but I can't definitively rule out some excess noise at 100 Hz from this stage. Because the gain of the HV stage is x31, we'd need a preamp with better than 1nV/rtHz to directly measure the noise I guess. The Acromag noise model in Attachment #2 is based on a measurement I describe here.
what is the noise level before the HV stage? i.e. how well is the acromag noise being filtered?
I found this H1 alog entry by Izumi confirming that the calibrated channels CAL-CS_* need the same dewhitening filter.
This encouraged me to download the PRCL and MICH data and using Jon's example notebook. I incorporated these noise spectra into the MCMC simulation. The most recent results are attached.
I am still missing:
Also, now the MCMC repeats a simulation if it doesn't pass the RF PDs test so the number of valid simulations stays the same. I'm still not sure about why the A+ simulations are much more robust to these tests than aLigo simulations.
I packaged the HV coil driver into a 2U chassis, hoping for better shielding from pickup. There is still considerable excess noise in measurement vs model around 100 Hz, see Attachment #1. The projected displacement noise from this noise contribution is shown in Attachment #2 - I've also plotted the contribution from the 4.5kohm (planned value for fast path series resistance) for comparison. Attachment #3 has some photos of the measurement setup so if someone sees some red flags, please let me know.
I've run out of ideas to try and make the measurement cleaner - the presence of the rather prominent power line harmonics suggests that this is still not perfect, but what more shielding can we implement? I have to make the measurement on the circuit side of the 25 kohm series resistor, so I am using some Pomona minigrabbers to clip onto the leg of the wirewound resistor (see photos in Attachment #3), so that's not great maybe, but what's the alternative?
So if this is truly the noise of the circuit, then while it's an improvement on the current situaiton, it's unsatisfying that such a simple circuit can't match the design expectations. But how do we want to proceed?
I set up an action cam (DJI OSMO Pocket) and brought it back to the 40m. The kit is now placed in the control room cabinet together with the Canon DSLR.
I might have left the USBC chaging cable at home this time. Will bring it back next time.-> The cable was returned to the kit on Oct 23rd.
Alaska M7.5 20:54UTC https://earthquake.usgs.gov/earthquakes/eventpage/us6000c9hg/executive
I looked at the suspensions. The watchdogs have not been tripped.
IMC was locked but continually shaken. (and occasional unlock)
I'll bring a file binder "40m wiring diagram" to home at the next chance.
There is another one on the shelf in the control room.
(I thought I put it in my bag, but it looks like that I left it somewhere around the fax area)
I entered 40m today at around 1:10 pm and left by 1:50 pm. I entered 104 through the machine shop entry. I took top view single picture photos of ITMY, BS, AP, ITMX, ETMX and ETMY tables. The latest photos will be put here on the wiki soon.
Pushed another update to MCMC simulation. This includes:
The DOFs<->RFPD associations I use are:
However, one thing that bothers me is that for some reason ~ 15 out of 160 aLigo simulations are discarded while none for A+. It can also be seen that the A+ simulations are more spread-out which might be related.
The new simulation results are attached.
The AC cord from the PSL HEPA variac to the junction box was replaced.
Now the HEPA is running at 70%
Showed up at the 40m at 7pm
Closing the work
Leaving the 40m at 9:30pm
Memo: 40m wiring/Mask/Camera/Red Pitaya/Particle Counter
I entered 40m today at around 1:20 pm and left by 1:45 pm. I entered 104 through the machine shop entry. I did the following:
Summary of discussion between Koji and gautam on 14 Oct:
I tried all of these last night / overnight, here are my findings.
Analog locking of the homodyne phase:
See Attachment #1.
Relative stability of two IFR2023s synchronized to the same FS725 Rb standard:
The electrical LO signal for demodulation of the 44 MHz photocurrent is provided by an IFR2023 signal generator. To maintain a fixed phase relation between this signal, and the phase modulation sidebands imprinted on the interferometer light via a separate IFO2023 signal generator, I synchronize both to the same Rb timing standard (a 10 MHz signal from the FS725 is sent to the rear panel frequency standard input on the IFR). We don't have a direct 44 MHz electrical signal available from the main IFO Marconi at the LSC rack (or anywhere else for that matter). So I decided to do this test at 55 MHz.
A look at the time domain signal:
With the Michelson locked on the dark fringe, the RF44 I and Q signals in the time domain are shown in Attachment #3 for a 1 minute stretch.
Per Koji's suggestion, I turned the PSL HEPA Variac to 50% just now, so that the power load through the burnt electrical cable is reduced by 75%.
Koji recommended that I can add whitening filters to suppress ADC noise easily. I added a filter before ADC in ALS loop with 4 zeros at 1.5 Hz and 4 poles at 100 Hz and added a reversed filter in the digital filter of ALS. This did not change the performance of the loop but significantly reduced the contribution of ADC noise above 1 Hz. One can see ALS_controls.yaml for the filter description. Please let me know if this does not make sense or there is something that I have overlooked.
Now, the dominant noise source is DFD noise below 100 Hz and green laser frequency noise above that. For DFD noise, I used data dating back to Kiwamu's paper. The noise contribution from DFD in the model is lower than the latest measured ALS noise budget post on elog. I'll look further into design details and noise of DFD.
Code, data, and schematics
The two ITM spares and two ETM spares are together stored in the optic storage (B110) at Downs. c/o Liyuan and GariLynn
We cleared up some space in the 1Y1 electronics rack to install the 3 new FE machines. I removed the current driver and laser from 1Y1, they are now stored in the E10 cabinet. I will upload some photos to gPhotos soon.
Attachment #1: spectra of the phase noise between LO and IFO output fields sensed using the RF44 signal.
Closing a feedback loop:
The only two PZT Phase modulation transfer function measurements I could find are 40m/15206 and 40m/12077. Both these measurements were made to find a good modulation frequency and do not go below 50 kHz. So I don't think these will help us. We'll have to do a frequency transfer function measurement at lower frequencies.
I'm still looking for ALS PDH loop measurements to verify the model. I found this 40m/15059 but it is only near the UGF. The UGF measured here though looks very similar to the model prediction. A bit older measurement in 2017 was this 40m/13238 where I assume by ALS OLTF gautum meant the green laser PDH OLTF. It had similar UGF but the model I have has more phase lag, probably because of a 31.5 kHz pole which comes at U7 through the input low pass coupling through R28, C20 and R29 (See D1400293)
If the green laser is not being used, can I go and take some of these measurements myself?
Is it better than Luxor? https://labcit.ligo.caltech.edu/~jharms/luxor.html
These boxes were moved from the 40m hallway to the inside of the VEA so that we have some space to walk around. You can find some pictures here.
For all the loops where we drive the NPRO PZT, there is some notch/resonance feature due to the PZT mechanical resonance. In the IMC loop this limits the PZT/EOM crossove to be less than 25 kHz. I don't have a model for this, btu it should be included.
If you hunt through the elogs, people have measured the TF of ALS NPRO PZT to phase/frequency. Probably there's also a measured ALS PDH loop somewhere that you could use to verify your model.
This looks cool, we should have something similar, can be really useful.
I used D1400293 to get the latest logged details about the universal PDH box used to lock the green laser at X end. The uPDH_X_boost.fil file present there was used to obtain the control model for this box. See attachment one for the code used. Since there is a variable gain stage in the box, I tuned the gain of the filter model F_AUX in ALS_controls.yml to get the maximum phase margin in the PDH lock of the green laser. Unity gain frequency of 8.3 kHz can be achieved in this loop and as Gautam pointed out earlier, it can't be increased much further without changes in the box.
The ALS control model remains stable with a reduction in total estimate noise because of the above update. There are few things to change though:
Tue evening from 4pm~6pm, Koji made a social distant tour for Anchal. We were present around the PSL/AS/ETMX tables.
I have placed 3 new in box, IDP 7 forepumps along the x arm of the interferometer. These are to be used as spares for both the 40m and Clean and Bake.
LiYuan has kindly done some Total Integrating Sphere (TIS) measurements on ITMU01 and ITMU02. A summary of the measurement is attached. I uploaded the measurements and some analysis script to nodus at /home/export/home/40m_TIS. I created a Wiki page for the measurements and linked to it from the core optics page.
These TIS measurements look very similar to the TIS of the LIGO optics. Further analysis shows that the scatter loss is 10+/-1.7 ppm for ITMU01 and 8.6+/-0.4 ppm for ITMU02.
In this calculation, a gaussian beam the same size bouncing off the 40m ITMs is assumed to scatter from the mirrors. The error is calculated by moving the beam around randomly with STD of 1mm.
In LiYuan's setup, TIS is measured for scattering angles between 1 and 75 degrees. If we go further and assume that the scatter is Lambertian we can extrapolate that the total loss is 10.9+/-1.9 ppm for ITMU01 and 9.2+/-0.5 ppm for ITMU02.
These measurements complete the loss budget nicely since with the 6ppm loss predicted from the phase maps, the total loss in the arm cavities would be 6+10+10=26ppm which is very close to the 28ppm loss that was measured after the arm cavity optics were cleaned.
We received 20pcs of stuffed demodulator boards from Screaming Circuits today. Some caveats:
I removed 1 from the group to stuff some components that weren't sent to Screaming Circuits and test the functionality on the benchtop, the remaining have been stored in a plastic box for now as shown in Attachment #1. The box has been delivered to Chub who will stuff the remaining 19 boards once I've tested the one piece.
The simple interferometer, composed of a single bounce reflection from ITMY and the LO beam deilvered via fiber to the AS table, can be locked - i.e. the phase of the LO beam can be controlled such that the DC light level on the DCPDs after the two beams are interfered can be stabilized. This test allows us to confirm that various parts of the sensing and actuation chain (e.g. PI PZT for homodyne phase control, Trek amplifier etc etc) are working.
I will post more quantitative analysis tomorrow.
Attachment #2 shows the servo topology.
After the earthquake on September 19 2020, it looks to me like the only lasting damage to suspensions in vacuum is the ETMY UR magnet being knocked off.
Suspension ringdown tests:
I did the usual suspension kicking/ringdown test:
Attachment #2 also includes info about the matrix diagonalization, and the condition numbers of the resulting matrices are as large as ~30 for some suspensions, but I think this isn't a new feature.
Attachment #1 shows that the c1rfm model isn't able to receive any signals from the front end machines at EX and EY. Attachment #2 shows that the problem appears to have started at ~430am today morning - I certainly wasn't doing anything with the IFO at that time.
I don't know what kind of error this is - what does it mean that the receiving model shows errors but the sender shows no errors? It is not a new kind of error, and the solution in the past has been a series of model reboots, but it'd be nice if we could fix such issues because it eats up a lot of time to reboot all the vertex machines. There is no diagnostic information available in all the places I looked. I'll ask the CDS group for help, but I'm not sure if they'll have anything useful since this RFM technology has been retired at the sites (?).
In the meantime, arm cavity locking in the usual way isn't possible since we don't have the trigger signals from the arm cavity transmission.
Update 1500 4 Oct: soft reboots of models didn't do the trick so I had to resort to hard reboots of all FEs/expansion chassis. Now the signals seem to be okay.
2x500 ml bottles of spectroscopic grade isopropanol were delivered. I marked them with today's date and placed them in the solvent cabinet. In the process, my shoulder bumped the laser interlock switch by the door to the VEA in the drill press area, which turned the PSL NPRO off. I turned it back on just now. The other NPROs are not connected to the interlock and so were unaffected.
Attachment #1 shows that the ITMX, ETMY and beamsplitter Oplev light levels have decayed significantly from their values when installed. In particular, the ETMY and ITMX sum channels are now only 50% of the values when a new HeNe was installed. ELOG search revealed that ITMY and ETMX HeNes were replaced with newly acquired units in March and September of last year respectively. The ITMX oplev was also replaced in March 2019, but the replacement was a unit that was being used to illuminate our tourist attraction glass fiber at EX.
We should replace these before any vent as they are a useful diagnostic for the DC alignement reference.
The optomechanics stock in the lab was in a sad state. We have obtained the following from Thorlabs in the last two months:
I have used some of these for the ari BHD setup. The unused items are stored in the shelves that house the optomechanics ~halfway down the east arm. I'm wondering what's a good setup to document the stock of this stuff so we can always have a healthy stock of optomechanics (at least the non speciality ones like posts, spacers etc). It sucks to realize at 0000hrs that you're missing a 3mm shim or 250mm converging lens or something.
it would be a good idea for us to have an auto-reminder to have us check the flow of all the HEPAs in the lab and elog it once a year so that we can replace filters and pre-filters appropriately.
The HEPAs work again. After running the HEPAs for ~1 hour, I checked the particle count on the PSL table - the meter registered 0 for both 0.3 um and 0.5 um. So I decided to turn the NPRO back on, at ~1730 local time. The PMC and IMC were readily locke, so the basic interferometer functionality is returned, and we can now go ahead with (i) vent prep (ii) air BHD tests and (iii) IMC debuggin as was discussed on the call today. The earth is shaking, but nothing serious so far, I will resume alignment of the interferometer later in the evening when hopefully things have calmed down a bit more...
Note that the many other issues Koji noted in the preceeding elog (e.g. flaky wiring) have not been addressed.
Chub kindly provided us with an electronic anemometer. With the meter held directly against the HEPA filter inside the enclosure, we measured ~700 cfm of airflow on each of the two HEPAs, with the Variac set to 100% and the HEPAs themselves set to "High". With the Variac at 50%, the flow drops to ~160 cfm. At the nominal setting of 33%, the meter didn't register any flow. I don't know what the spec'd flow rate is for this combination of blower + filter, but Jordan says similar units in Downs register ~1500 cfm at the "High" setting. The two protable (similarly sized) HEPA units in the 40m, one at ITMY and one at ETMY, register ~900 cfm and ~1100 cfm respectively, when set to high. So we may want to revisit what the "nominal" HEPA setting should be, in case the filters have become clogged over time.
Some photos of the HEPA blowers with the pre-filters off and the capacitors switched out may be found here.
I was in the lab from 1630-1830. I have located and visually inspected all the parts required for the magnet regluing / optic cleaning parts of the planned vent, except the fresh batches of scpectroscopic grade solvents. I was in the cleanroom part of the clean and bake lab from 1630-1700.
I removed the forepump to TP2 this morning after the vacuum failure, and tested in the C&B lab. I pumped down on a small volume 10 times, with no issue. The ultimate pressure was ~30 mtorr.
I re-installed the forepump in the afternoon, and restarted TP2, leaving V4 closed. This will run overnight to test, while TP3 backs TP1.
In order to open V1, with TP3 backing TP1, the interlock system had to be reset since it is expecting TP2 as a backing pump. TP2 is running normally, and pumping of the main volume has resumed.
Unclear why the TP2 foreline pump failed in the first place, it has been running fine for several hours now (although TP2 has no load, since V4 isolates it from the main volume). Koji's plots show that the TP2 foreline pressure did not recover even after the interlock tripped and V4 was closed (i.e. the same conditions as TP2 sees right now).
Yes, that loop was unstable. I started using the time domain response to check for the stability of loops now. I have been able to improve the filter slightly with more suppression below 20 Hz but still poor phase margin as before. This removes the lower frequency region bump due to seismic noise. The RMS noise improved only slightly with the bump near UGF still the main contributor to the noise.
For inclusion of real spectra, time delays and the anti-aliasing filters, I still need some more information.
The code used to calculate the transfer functions and plot them is in the repo 40m/ALS/noiseBudget
Related Elog post with more details: 40m/15587
## Cavity Pole
Here is the timeline. This suggests TP2 backing RP failure.
1st line: TP2 foreline pressure went up. Accordingly TP2 P, current, voltage, and temp went up. TP2 rotation went down.
2nd line: TP2 temp triggered the interlock. TP2 foreline pressure was still high (10torr) so TP2 struggled and was running at 1 torr.
3rd line: Gautam's operation. TP2 was isolated and stopped.
Between the 1st line and 2nd line, TP2 pressue (=TP1 foreline pressure) went up to 1torr. This made TP1 current increased from 0.55A to 0.68A (not shown in the plot), but TP1 rotation was not affected.
The interlocks tripped at ~630am local time. Jordan reported that TP2 was supposedly running at 52 C (!).
V1 was already closed, but TP2 was still running. With him standing by the rack, I remotely exectued the following sequence:
Jordan confirmed (by hand) that TP2 was indeed hot and this is not just some serial readback issue. I'll do the forensics later.
Dimensions / Specs
- HEPA unit dimentions
- HEPA unit manufacturer
Gautam reported that the PSL HEPA stopped running (ELOG 15592). So I came in today and started troubleshooting.
It looks like that the AC power reaches the motors. However, both motors do not run. It looks like the problem exists in the capacitors, the motors, or both.
Parts specs can be found in the next ELOG.
Attachment 1 is the connection diagram of the HEPA. The AC power is distributed by the breaker panel. The PSL HEPA is assigned to use M22 breaker (Attachment 2). I checked the breaker switch and it was (and is) ON. The power goes to the junction box above the enclosure (Attachment 3). A couple of wires goes to the HEPA switch (right above the enclosure light switch) and the output goes to the variac. The inside of the junction box looked like this (Attachment 4).
By the way, the wires were just twisted and screwed into a metal threaded (but isolated) caps (Attachment 5). Is this legit? Shouldn't we use stronger crimping? Anyway, there was nothing wrong with the caps w.r.t the connection for now.
I could easily trace the power up to the variac. The variac output was just fine (Attachment 6). The cord goes from the variac to the junction box (and then HEPAs) looked scorched. The connection from the plug to HEPAs was still OK, but this should be eventually replaced. Right now the cable was unplugged after the following tests for the safety reason.
The junction box for each HEPA unit was opened to check the voltage. The supply voltage came to the junction boxes and it was just fine. In Attachments 8 & 9, the voltages look low but this is because I just turned the variac only a little.
At the (main) junction box, the resistances of the HEPAs were checked with the Fluke. As the HEPA units are connected to the AC in parallel, the resistances were individually checked as follows.
The coils were not disconnected (... I wonder if the wiring of South HEPA was flipped? But this is not the main issue right now.)
By removing the pre-filters, the motors were inspected Attachments 10 & 11. At least the north HEPA motor was warm, indicating there was some current before. A capacitor was connected per motor. When the variac was tuned up a bit, one side of the capacitor could see the voltage. I could not judge which has the issue between the capacitor and the motor.
I got some feedback from Koji who pointed out that the phase tracker is not required here. This situation is similar to the phase locking of two lasers together, which we frequently do, except in that case, we usually we offset the absolute frequencies of the two lasers by some RF frequency, and we demodulate the resulting RF beatnote to use as an error signal. We can usually acquire the lock by simply engaging an integrator (ignoring the fact that if we actuate on the laser PZT, which is a frequency actuator, just a proportional feedback will be sufficient because of the phase->frequency conversion), the idea being that the error signal is frequently going through a zero-crossing (around which the sinusoidal error signal is approximately linear) and we can just "catch" one of these zero crossings, provided we don't run of actuation range.
So the question here becomes, is the RF44 signal a suitable error signal such that we can close a feedback loop in a similar way? To try and get more insight, I tried to work out the situation analytically. I've attached my thinking as a PDF note. I get some pretty messy complicated expressions for the RF44 signal contributions, so it's likely I've made a mistake (though Mathematica did most of the heavy lifting), it'll benefit from a second set of eyes.
Anyways, I definitely think there is some additional complications than my simple field cartoon from the preceeding elog would imply - the relative phases of the sidebands seem to have an effect, and I still think the lack of the PRC/SRC make the situation different from what Hang/Teng et. al. outlined for the A+ homodyne phase control analysis. Before the HEPA failed, I had tried closing the feedback loop using one quadrature of the demodulated RF44 signal, but had no success with even a simple integrator as the loop (which the experience with the PLL locking says should be sufficient, and pretty easily closed once we see a sinusoidally oscillating demodulated error signal). But maybe I'm overlooking something basic conceptually?
I don't think the proposed scheme for sensing and controlling the homodyne phase will work without some re-thinking of the scheme. I'll try and explain my thinking here and someone can correct me if I've made a fatal flaw in the reasoning somewhere.
I came to the lab. The control room AC was off -> Now it is on.
Here is the setting of the AC meant for continuous running
This ALS loop is not stable. Its one of those traps that comes from using only the Bode plot to estimate the loop stability. You have to also look at the time domain response - you can look at my feedback lecture for the SURF students for some functions.
This is not a reply to comments given to the last post; Still working on incorporating those suggestions.
Rana suggested looking first at what needs to be suppressed and then create a filter suited for the noise from scratch. So I discarded all earlier poles and zeros and just kept the resonant gains in the digital filter. With that, I found that all we need is three poles at 1 Hz and a gain of 8.1e5 gives the lowest RMS noise value I could get.
Now there can be some practical reasons unknown to me because of which this filter is not possible, but I just wanted to put it here as I'll add the actual noise spectra into this model now.
The HEPA filters on the PSL enclosure are no longer running. I tried cycling the power switch on the NW corner of the enclosure, and also turned the dial on the Variac down to zero and back up to maximum, no effect. Judging by the indicator LED on it, the power is making it to the Variac - bypassing the Variac and directly connecting the HEPA to the AC power seems to have no effect either.
I can't be sure, but I'm almost certain I heard them running at max speed an hour ago while I was walking around inside the VEA. Probably any damage that can happen has already been done, but I dialled down the Innolight injection current, and closed its shutter till the issue can be resolved.
I removed the forepump (Varian SH-110) for TP3 today to see why it had failed over the weekend. I tested it in the C&B lab and the ultimate pressure was only ~40torr. I checked the tip seals and they were destroyed. The scroll housing also easily pulled off of the motor drive shaft, which is indicative of bad bearings. The excess travel in the bearings likely led to significant increase in tip seal wear. This pump will need to be scrapped, or rebuilt.
I tested the spare Varian SH-110 pump located at the X-end and the ultimate pressure was ~98 mtorr. This pump had tip seals replaced on 11/5/18, and is currently at 55163 operating hours. It has been installed as the TP3 forepump.
Once installed, restarting the pump line occured as follows: V5 Closed, VA6 closed, VASE Closed, VASV closed, VABSSCI closed, VABS closed, VABSSCO closed, VAEV closed, VAEE closed,TP3 was restarted and once at normal operation, valves were opened in same order.
The pressure differential interlock condition for V5 was temporaily changed to 10 torr (by Gautam), so that valves could be opened in a controlled manner. Once, the vacuum system was back to normal state the V5 interlock condition was set back to the nominal 1 torr. Vacuum system is now running normally.
See Attachment #1.
The beam spot on ETMY looks weird (looks almost like a TEM10 mode), but the one on ITMY seems fine, see Attachment #2. Wonder what's going on there, maybe just a focusing problem?
We need a vent to fix the suspension, but until then what we can do is to redistribute the POS/PIT/YAW actuations to the three coils.