My power at home winked out for a second this morning, but it looks like either nothing happened in the 40m lab or else it rode it out.
MC is locked - lost lock around 11:25 AM and then relocked.
2PM: Arrived at the 40m. Started the work for the coupling of the RF modulated LO beam into a fiber. -> I left the lab at 10:30 PM.
The fiber coupling setup for the phase-modulated beam was made right next to the PSL injection path. (See attachment 1)
- PSL HEPA was running at 33% and is now at 100%
- South End HEPA was not on and is now running
- Yarm Portable HEPA was not running and is now running at max speed: the power was taken beneath the ITMY table. It is better to unplug it when one uses the IFO.
- Yend Portable HEPA was not running and is now running (presumably) at max speed
Particle Levels: (Not sure about the unit. The convention here is to multiply x10 of the reading)
Before running the HEPAs at their maximum
9/10/2020 15:30 / 0.3um 292180 / 0.5um 14420
(cf 9/5/2020 / 0.3um 94990 / 0.5um 6210)
After running the HEPAs at their maximum
The number gradually went down and now became constant at about half of the initial values
9/10/2020 19:30 / 0.3um 124400 / 0.5um 7410
M4.5 EQ in LA 2020-09-19 06:38:46 (UTC) / -1d 23:38:46 (PDT) https://earthquake.usgs.gov/earthquakes/eventpage/ci38695658/executive
I only checked the watchdogs. All watchdogs were tripped. ITMX and ETMY seemed stuck (or have the OSEM magnet issue). They were left tripped. The watchdogs for the other SUSs were reloaded.
the seismometers obviously saturated during the EQ, but the accelerometers captured some of it. It looks like there's different saturation levels on different sensors.
Also, it seems the mounting of the MC2 accelerometers is not so good. There's some ~10-20 Hz resonance its mount that's showing up. Either its the MC2 chamber legs or the accelerometers are clamped poorly to the MC2 baseplate.
Sun Sep 20 00:02:36 2020 edit: fixed indexing error in plots
* also assuming that the sensors are correctly calibrated in the front end to 1 count = 1 um/s^2 (this is what's used in the summ pages)
the EQ was ~14 km south of Caltech and 17 km deep
I'm amazed at how much higher the noise is on the MC2 accelerometer. Is that really how much amplification of the ground motion we're getting? If so, its as if the MC has no vibration isolation from the ground in that band. We should put one set on the ground and make the more direct comparison of the spectra. Also, perhaps do some seismic FF using this sensor - I'm not sure how successful we've been in this band.
Attaching the coherence plot from ldvw.ligo.caltech.edu (apparently it has access to the 40m data, so we can use that as an alternative to dtt or python for remote analysis):
It would be interesting to see if we can use the ML based FF technology from this summer's SURF project by Nadia to increase the coherence by including some slow IMC alignment channels.
I supplied a bottle of hand soap. Don't put water in the bottle to dilute it as it makes the soap vulnarable for cotamination.
There were two SUSs which didn't look normal.
- ITMX was easily released by the bias slider -> Shake the pitch slider and while all the OSEM values are moving, turn on the damping control (with x10 large watchdog threshold)
- ETMY has UR OSEM 0V output. This means that there is no light. And this didn't change at all with the slider move.
- Went to the Y table and tried to look at the coils. It seems that the UR magnet is detached from the optic and stuck in the OSEM.
We need a vent to fix the suspension, but until then what we can do is to redistribute the POS/PIT/YAW actuations to the three coils.
While I stopped by the lab this morning to pick up some things, I took the opportunity to continue the recovery.
At some point, we should run the suspension eigenmode routine (kick optics, let them ringdown, measure peak locations and Qs) to confirm that the remaining suspensions are okay, will also help in actuation re-allocation efforts on ETMY. But I didn't do this today.
Leaving the lab at 1150.
See Attachment #1.
The beam spot on ETMY looks weird (looks almost like a TEM10 mode), but the one on ITMY seems fine, see Attachment #2. Wonder what's going on there, maybe just a focusing problem?
I came to the lab. The control room AC was off -> Now it is on.
Here is the setting of the AC meant for continuous running
Gautam reported that the PSL HEPA stopped running (ELOG 15592). So I came in today and started troubleshooting.
It looks like that the AC power reaches the motors. However, both motors do not run. It looks like the problem exists in the capacitors, the motors, or both.
Parts specs can be found in the next ELOG.
Attachment 1 is the connection diagram of the HEPA. The AC power is distributed by the breaker panel. The PSL HEPA is assigned to use M22 breaker (Attachment 2). I checked the breaker switch and it was (and is) ON. The power goes to the junction box above the enclosure (Attachment 3). A couple of wires goes to the HEPA switch (right above the enclosure light switch) and the output goes to the variac. The inside of the junction box looked like this (Attachment 4).
By the way, the wires were just twisted and screwed into a metal threaded (but isolated) caps (Attachment 5). Is this legit? Shouldn't we use stronger crimping? Anyway, there was nothing wrong with the caps w.r.t the connection for now.
I could easily trace the power up to the variac. The variac output was just fine (Attachment 6). The cord goes from the variac to the junction box (and then HEPAs) looked scorched. The connection from the plug to HEPAs was still OK, but this should be eventually replaced. Right now the cable was unplugged after the following tests for the safety reason.
The junction box for each HEPA unit was opened to check the voltage. The supply voltage came to the junction boxes and it was just fine. In Attachments 8 & 9, the voltages look low but this is because I just turned the variac only a little.
At the (main) junction box, the resistances of the HEPAs were checked with the Fluke. As the HEPA units are connected to the AC in parallel, the resistances were individually checked as follows.
The coils were not disconnected (... I wonder if the wiring of South HEPA was flipped? But this is not the main issue right now.)
By removing the pre-filters, the motors were inspected Attachments 10 & 11. At least the north HEPA motor was warm, indicating there was some current before. A capacitor was connected per motor. When the variac was tuned up a bit, one side of the capacitor could see the voltage. I could not judge which has the issue between the capacitor and the motor.
Dimensions / Specs
- HEPA unit dimentions
- HEPA unit manufacturer
I was in the lab from 1630-1830. I have located and visually inspected all the parts required for the magnet regluing / optic cleaning parts of the planned vent, except the fresh batches of scpectroscopic grade solvents. I was in the cleanroom part of the clean and bake lab from 1630-1700.
The HEPAs work again. After running the HEPAs for ~1 hour, I checked the particle count on the PSL table - the meter registered 0 for both 0.3 um and 0.5 um. So I decided to turn the NPRO back on, at ~1730 local time. The PMC and IMC were readily locke, so the basic interferometer functionality is returned, and we can now go ahead with (i) vent prep (ii) air BHD tests and (iii) IMC debuggin as was discussed on the call today. The earth is shaking, but nothing serious so far, I will resume alignment of the interferometer later in the evening when hopefully things have calmed down a bit more...
Note that the many other issues Koji noted in the preceeding elog (e.g. flaky wiring) have not been addressed.
Chub kindly provided us with an electronic anemometer. With the meter held directly against the HEPA filter inside the enclosure, we measured ~700 cfm of airflow on each of the two HEPAs, with the Variac set to 100% and the HEPAs themselves set to "High". With the Variac at 50%, the flow drops to ~160 cfm. At the nominal setting of 33%, the meter didn't register any flow. I don't know what the spec'd flow rate is for this combination of blower + filter, but Jordan says similar units in Downs register ~1500 cfm at the "High" setting. The two protable (similarly sized) HEPA units in the 40m, one at ITMY and one at ETMY, register ~900 cfm and ~1100 cfm respectively, when set to high. So we may want to revisit what the "nominal" HEPA setting should be, in case the filters have become clogged over time.
Some photos of the HEPA blowers with the pre-filters off and the capacitors switched out may be found here.
it would be a good idea for us to have an auto-reminder to have us check the flow of all the HEPAs in the lab and elog it once a year so that we can replace filters and pre-filters appropriately.
The optomechanics stock in the lab was in a sad state. We have obtained the following from Thorlabs in the last two months:
I have used some of these for the ari BHD setup. The unused items are stored in the shelves that house the optomechanics ~halfway down the east arm. I'm wondering what's a good setup to document the stock of this stuff so we can always have a healthy stock of optomechanics (at least the non speciality ones like posts, spacers etc). It sucks to realize at 0000hrs that you're missing a 3mm shim or 250mm converging lens or something.
2x500 ml bottles of spectroscopic grade isopropanol were delivered. I marked them with today's date and placed them in the solvent cabinet. In the process, my shoulder bumped the laser interlock switch by the door to the VEA in the drill press area, which turned the PSL NPRO off. I turned it back on just now. The other NPROs are not connected to the interlock and so were unaffected.
Tue evening from 4pm~6pm, Koji made a social distant tour for Anchal. We were present around the PSL/AS/ETMX tables.
These boxes were moved from the 40m hallway to the inside of the VEA so that we have some space to walk around. You can find some pictures here.
We cleared up some space in the 1Y1 electronics rack to install the 3 new FE machines. I removed the current driver and laser from 1Y1, they are now stored in the E10 cabinet. I will upload some photos to gPhotos soon.
Per Koji's suggestion, I turned the PSL HEPA Variac to 50% just now, so that the power load through the burnt electrical cable is reduced by 75%.
I entered 40m today at around 1:20 pm and left by 1:45 pm. I entered 104 through the machine shop entry. I did the following:
The AC cord from the PSL HEPA variac to the junction box was replaced.
Now the HEPA is running at 70%
Showed up at the 40m at 7pm
Closing the work
Leaving the 40m at 9:30pm
Memo: 40m wiring/Mask/Camera/Red Pitaya/Particle Counter
I entered 40m today at around 1:10 pm and left by 1:50 pm. I entered 104 through the machine shop entry. I took top view single picture photos of ITMY, BS, AP, ITMX, ETMX and ETMY tables. The latest photos will be put here on the wiki soon.
I set up an action cam (DJI OSMO Pocket) and brought it back to the 40m. The kit is now placed in the control room cabinet together with the Canon DSLR.
I might have left the USBC chaging cable at home this time. Will bring it back next time.-> The cable was returned to the kit on Oct 23rd.
I wanted to look into the ISS situation. Some weeks ago, I found the PD that was previously used as the in-loop photodiode. I wanted to use this and measure the open-loop RIN at a few places (to see if there's any variation and also to check its functionality). However, I didn't get very far tonight - for a start, the PD height is 3" (while our beam height is 4" everywhere outside the vacuum), and I needed to put together a circuit to supply the 5V bias and +/- 15 V since the transimpedance is done on the head. I was only able to do a low-level functionality test tonight, checked that the DC voltage output varied linearly with the incident power (calibrated against an NF1611 photodiode, data will be put up later). I didn't get to measuring any noise performance - is an incandescent light bulb still shot noise limited at ~10 Hz < f < 10kHz? Some notes:
Unconnected to this work - this problem reared its ugly head again (i noticed it yesterday morning already actually). I don't have the energy to embark on a fix tonight, Koji is going to be in the lab all day tomorrow and so he will fix it.
that little PD in the black mount was never very good. The AD829 is not a good opamp for transimpedance and especially not good for low frequencies. Stefan Ballmer and I were able to get 2e-8 out of these (@100 Hz) many years ago.
I wonder if we have some of Zach's M2ISS photodetectors around, perhaps in QIL or Cryo. I doubt that any of them are in use now. Those had good performance nad BNC output.
Ok I was using the PD in the black mount because Rana recommended it a few weeks ago.
Regarding the M2ISS, I acquired the hardware from QIL some months ago, including a circuit board, and 2 PDs. These had LEMO outputs though (not BNC), and the mounts are not 4". These photodiodes are what I'm using as the airBHD DCPDs right now, and some photos are here - are these the photodiodes you mentioned? Or are there yet more M2ISS photodiodes? I remember Johannes had some custom mounts extruded to make them 4" high, do you mean those? Can I retrieve them his Cryo setup?
BTW, my elog scraping shows only one spectra from Stefan in the ATF elog, and the performance there is more like 1e-7/rtHz @ 100 Hz, and that’s using a dedicated high BW servo circuit, not the SR560. Am I just missing the measurement of 2e-8/rtHz?
At this point, I'm leaving the lab. All the suspensions (incl SRM) are aligned. PSL/GRX/GRY shutters were left open.
If everything else looks good, I'll start letting the dry N2 into the main volume after lunch.
Now the green transmissions are visible by the green PDs. Attachment 9 shows the trans and ref of each green beams with and without locking to TEM00. The questionable green TRY was ~0.3. If we compare this with the histrical data (Attachment 10), it is about 1/4 of the value in the past. It's not too crazy but still quite low.
BTW, nice video! @ Koji, How difficult was it to edit it into this form?
We are now ready to take the doors off. I've already done the basic prep work (loosened bolts, cleaned chamber, carts for tools, fresh ameristat on portable HEPAs etc).
[koji, rana, gautam]
This morning, we did the following;
The OSEMs remain in the EY vacuum chamber. The next set of steps are:
We will most likely work on this tomorrow. At ~1615, I briefly opened the PSL shutter and tweaked the IMC alignment. We will almost certainly change the pointing into the IMC when we remove the old OMC and rebalance that table, so care should be taken when working on that...
To be a bit more clear about what we are going to do in the OMC chamber, I marked-up some photos, see Attachments #1 and #2.
I anticipate that after this work, the only components on the table will be
Are we in agreement with this plan?
See #15656 for the updated photo
I believe the mirror next to IM1 is for the green beams to be delivered to the PSL table. I think we still want to keep it. Otherwise, the plan looks fine.
Good point - looking back, I also see that I already removed the mirror at the SW corner of the table in 2016. Revised photo in Attachment #1. There is an optic on the east edge of this table whose purpose I'm not sure of, but I'm pretty sure it isn't essential to the main functionality and so can be removed.
I got a call from Calum ~830am today saying some facilities people entered the lab, opened the south entrance door, and tripped the alarm in the process. I came to the lab shortly after and was able to reset the alarm by flipping the switch on the alarm box at the south end entrance to "Alarm OFF". Then, I double checked that the door is closed, and re-enabled the alarm. The particle count at the SP table is not unusually high and the lasers (Oplev HeNe and AUX X) were still on, so doesn't look like any lasting damage was done. The facilities people were apparently wearing laser safety goggles.
The IMC isn't resonant for a TEM00 mode at the time of writing - we are waiting for the stack to relax, at which point if the IMC isn't resonant for a TEM00 mode, we will tweak the input pointing into the IMC (we want to use the suspended cavity as the reference, since it is presumably more reliable than the table from which we removed ~50 kgs of weight and shifted the balance.
So all the primary vent objectives have been achieved 🙌 . The light doors are on the chamber right now. I'm measuring the free-swinging spectra of ETMY overnight. Barring any catastrophic failures and provided all required personnel are available, we will do the final pre-close-up checks, put the heavy doors back on, and pump down starting 10 am Monday, 9 Nov 2020. Some photos here.
Attachment #1 shows the main result - there are 4 peaks. The frequencies are a little different from what I have on file for ETMY and the Qs are a factor of 3-4 lower (except SIDE) than what they are in vacuum, which is not unreasonable I hypothesize. The fits suggest that the peak shape isn't really Lorentzian, the true shape seems to have narrower tails than a Lorentzian, but around the actual peak, the fit is pretty good. More detailed diagnostic plots (e.g. coil-to-coil TFs) are in the compressed Attachment #2. The condition number of the matrix to diagonalize the sensing matrix (i.e. what we multiply the "naive" OSEM 2 Euler basis matrix by) is ~40, which is large, but I wouldn't read too much into it at this point.
I see no red flags here - the PIT peak is a little less prominent than the others, but looking back through the elog, this kind of variation in peak heights doesn't seem unreasonable to me. If anyone wants to look at the data, the suspension was kicked every ~1100seconds from 1288673974, 15 times.
I'm measuring the free-swinging spectra of ETMY overnight.
Basic IFO alignment checks were done.
Tomorrow, we should do some visual checks of the chambers / EQ stops on ETMY etc but I don't see any major problems at the moment...
Barring any catastrophic failures and provided all required personnel are available, we will do the final pre-close-up checks, put the heavy doors back on, and pump down starting 10 am Monday, 9 Nov 2020.
> I didn't bother to align the green beams to the arm cavities or re-center the Oplevs - is this necessary? It is a step in the pre-close up checklist, so maybe we should do it... The green transmission does reach the PSL table...
I don't think so. The beam is reaching the PSL, so we have no motivation to change the green alignment. Regarding the oplev, the green refl should come back to the PDH PD and this gives us additional beam reference. As soon as we find the green resonance after the pumping, we can tweak the green axis so that the spots on the mirrors become reasonable (as well as the green trans CCD on the PSL table).
1100 - EY chamber inspected, no issues were found --> EY heavy door on
1200 - OMC chamber was inspected. OM6 was marginally tweaked to bring the beam down a little in pitch, and also a little northwards in Yaw. --> Heavy door on.
1230 - Pumpdown started. Initially, the annuli volume was pumped down. The procedure calls for doing this with the small turbopumps. However, V7 was left open, and hence, in the process, the TP1 foreline pressure (=P2) hit ~30 torr. This caused TP1 to shutdown. We were able to restart it without issue. This case was not caught by the interlock code, which was running at the time. It should be recitified.
1330 - OMC breadboard clean optics and DCPD hardware were wrapped up and packed into tupperware boxes and stored along the south arm. OMC cavity itself, the OMMT, and the breadboard the OMC was sitting on are wrapped in foil/Ameristat and stored in cabinet S13, lower 2 shelves.
1915 - P1a = 0.5 torr pressure reached. Switched over to pumping the main volume with TP1, backed by TP2 and TP3, which themselves are backed by their respective dry pumps and also the AUX drypump for some extra oomph. All cooling fans available in the area were turned on and directed at the turbo pumps. RV2 was used to throttle the flow suitably.
It was at this point that we hit a snag - RV2 has gotten stuck in a partially open position, see Attachment #1. We can see that the thread doesn't move in response to turning the rotary dial. Fortunately, the valve is partially open, so the main volume continues to be pumped - see Attachment #2 for the full history of today's pumping. We are leaving the main volume pumped in this configuration overnight (TP1 pumping main volume backed by TPs 2 and 3, which are in turn backed by their respective drypumps and also the AUX dry pump). I think there is little to no risk of any damage to the turbo pumps, the interlocks should catch any anomalies. The roughing pumps RP1 and RP3 were turned off and that line was disconnected and capped.
What are our options?
We need some vacuum experts to comment. Why did this happen? Is this an acceptable failure mode of the valve?
2230 - P1a = 0.025 torr. The pressure is coming down with log-linear scale. x0.1 per 2.5 hours or so.
Main volume pressure as of 11:30AM 2020/11/10
For the input matrix diagonalization, it seemed to me that when we had a significant seismic event or a re-alignment of the optic with the bias sliders, the input matrix also changes.
Meaning that our half-light voltage may not correspond to the half point inside the LED beam, but that rather we may be putting the magnet into a partially occluding state. It would be good to check this out by moving the bias to another setting and doing the ringdown there.
I want to test out an AS port WFS now that I have all the parts in hand - I guess the Michelson / PRMI will suffice until I make the ALS noise good again, and anyways, there is much assembly work to be done. Overnight, I'm repeating the suspension eigenmode measurement.
The results from the ringdown are attached - in summary:
Ordered 11/16 from CDW, on PO# S492940, the high voltage Tripp Lite SMART5000XFMRXL for TP-1. Should be arriving in about a week.
In prep to try some of these debugging steps, I did the following.
While working, I noticed that the annoying tip-tilt drift seems to be worse than it has been in the last few months. The IPPOS QPD is a good diagnostic to monitor stability of TT1/TT2. While trying to trend the data, I noticed that from ~31 Jan (Saturday night/Sunday morning local time), the IP-POS QPD segment data streams seem "frozen", see Attachment #1. This definitely predates the CDS crash on Feb 2. I confirmed that the beam was in fact incident on the IPPOS QPD, and at 1Y2/1Y3 that I was getting voltages going into the c1iscaux Acromag crate. All manner of soft reboots (eth1 network interface, modbusIOC service) didn't fix the problem, so I power cycled the acromag interface crate. This did the trick. I will take this opportunity to raise again the issue that we do not have a useful, reliable diagnsotic for the state of our Acromag systems. The problem seems to not have been with all the ADC cards inside the crate, as other slow ADC channels were reporting sensible numbers.
Anyways, now that the QPD is working again, you can see the drift in Attachment #2. I ran the dither alignment ~4 hours ago, and in the intervening time, the spot, which was previously centered on the AS camera CRT display, has almost drifted completely off (my rough calibration is that the spot has moved 5mm on the AS CCD camera). I was thinking we could try installing the two HAM-A coil drivers to control the TTs, this would allow us to rule out flaky electronics as the culprit, but I realize some custom cabling would be required, so maybe not worth the effort. The phenomenology of the drift make me suspect the electronics - hard for me to imagine that a mechanical creep would stop creeping after 3-4 hours? How would we explain the start of such a mechanical drift? On the other hand, the fact that the drift is almost solely in pitch lends support to the cause being mechanical. This would really hamper the locking efforts, the drift is on short enough timescales that I'd need to repeatedly go back and run the dither alignment between lock attempts - not the end of the world but costs ~5mins per lock attempt.
On to the actual tests: before testing the hardware, I locked the PRMI (no ETMs). In this configuration, I'm surprised to see that there is nearly perfect coherence between the MICH and PRCL error signals between 100Hz-1kHz 🤔 . When the AS55 demodulated signals are whitened prior to digitization (and then de-whitened digitally), the coherence structure changes. The electronics noise (measured with the PSL shutter closed) itself is uncorrelated (as it should be), and below the level of the two aforementioned spectra, so it is some actual signal I'm measuring there with the PRMI locked, and the coherence is on the light fields on the photodiode. So it would seem that I am just injecting a ton of AS55 sensing noise into the PRCL loop via the MICH->PRM LSC output matrix element. Weird. The light level on the AS55 photodiode has increased by ~2x after the September 2020 vent when we removed all the unused output optics and copper OMC. Nevertheless, the level isn't anywhere close to being high enough to saturate the ADC (confirmed by time domain signals in ndscope).
To get some insight into whether the whole RF system is messed up, I first locked the arm cavities with POX and POY as the error signals. Attachment #3 shows the spectra and coherence betweeen these two DoFs (and the dark noise levels for comparison). This is the kind of coherence profile I would expect - at frequencies where the loop gain isn't so high as to squish the cavity length noise (relative to laser frequency fluctuations), the coherence is high. Below 10 Hz, the coherence is lower than between 10-100 Hz because the OLG is high, and presumably, we are close to the sensing noise level. And above ~100 Hz, POX and POY photodiodes aren't sensing any actual relative frequency fluctuations between the arm length and laser frequency, so it's all just electronics noise, which should be incoherent.
The analogous plot for the PRMI lock is shown in Attachment #4. I guess this is telling me that the MICH sensing noise is getting injected into the PRCL error point between 100Hz-1kHz, where the REFL11 photodiode (=PRCL sensor) isn't dark noise limited, and so there is high coherence? I tuned the MICH-->PRM LSC output matrix element to minimize the height of a single frequency line driving the BS+PRM combo at ~313Hz in the PRCL error point.
All the spectra are in-loop, the loop gain has not been undone to refer this to free-running noise. The OLGs themselves looked fine to me from the usual DTT swept sine measurements, with ~100 Hz UGF.
I'm curious to see if the demod phase for MICH in REFL & AS chamges between thi simple Mcihelson and PRMI. IF there's a change, it could point to a PRCL/f2 mismatch.
But I would still bet on demod chain funniness.