Dimensions / Specs
- HEPA unit dimentions
- HEPA unit manufacturer
The interlocks tripped at ~630am local time. Jordan reported that TP2 was supposedly running at 52 C (!).
V1 was already closed, but TP2 was still running. With him standing by the rack, I remotely exectued the following sequence:
Jordan confirmed (by hand) that TP2 was indeed hot and this is not just some serial readback issue. I'll do the forensics later.
Here is the timeline. This suggests TP2 backing RP failure.
1st line: TP2 foreline pressure went up. Accordingly TP2 P, current, voltage, and temp went up. TP2 rotation went down.
2nd line: TP2 temp triggered the interlock. TP2 foreline pressure was still high (10torr) so TP2 struggled and was running at 1 torr.
3rd line: Gautam's operation. TP2 was isolated and stopped.
Between the 1st line and 2nd line, TP2 pressue (=TP1 foreline pressure) went up to 1torr. This made TP1 current increased from 0.55A to 0.68A (not shown in the plot), but TP1 rotation was not affected.
I removed the forepump to TP2 this morning after the vacuum failure, and tested in the C&B lab. I pumped down on a small volume 10 times, with no issue. The ultimate pressure was ~30 mtorr.
I re-installed the forepump in the afternoon, and restarted TP2, leaving V4 closed. This will run overnight to test, while TP3 backs TP1.
In order to open V1, with TP3 backing TP1, the interlock system had to be reset since it is expecting TP2 as a backing pump. TP2 is running normally, and pumping of the main volume has resumed.
Unclear why the TP2 foreline pump failed in the first place, it has been running fine for several hours now (although TP2 has no load, since V4 isolates it from the main volume). Koji's plots show that the TP2 foreline pressure did not recover even after the interlock tripped and V4 was closed (i.e. the same conditions as TP2 sees right now).
I was in the lab from 1630-1830. I have located and visually inspected all the parts required for the magnet regluing / optic cleaning parts of the planned vent, except the fresh batches of scpectroscopic grade solvents. I was in the cleanroom part of the clean and bake lab from 1630-1700.
The HEPAs work again. After running the HEPAs for ~1 hour, I checked the particle count on the PSL table - the meter registered 0 for both 0.3 um and 0.5 um. So I decided to turn the NPRO back on, at ~1730 local time. The PMC and IMC were readily locke, so the basic interferometer functionality is returned, and we can now go ahead with (i) vent prep (ii) air BHD tests and (iii) IMC debuggin as was discussed on the call today. The earth is shaking, but nothing serious so far, I will resume alignment of the interferometer later in the evening when hopefully things have calmed down a bit more...
Note that the many other issues Koji noted in the preceeding elog (e.g. flaky wiring) have not been addressed.
Chub kindly provided us with an electronic anemometer. With the meter held directly against the HEPA filter inside the enclosure, we measured ~700 cfm of airflow on each of the two HEPAs, with the Variac set to 100% and the HEPAs themselves set to "High". With the Variac at 50%, the flow drops to ~160 cfm. At the nominal setting of 33%, the meter didn't register any flow. I don't know what the spec'd flow rate is for this combination of blower + filter, but Jordan says similar units in Downs register ~1500 cfm at the "High" setting. The two protable (similarly sized) HEPA units in the 40m, one at ITMY and one at ETMY, register ~900 cfm and ~1100 cfm respectively, when set to high. So we may want to revisit what the "nominal" HEPA setting should be, in case the filters have become clogged over time.
Some photos of the HEPA blowers with the pre-filters off and the capacitors switched out may be found here.
it would be a good idea for us to have an auto-reminder to have us check the flow of all the HEPAs in the lab and elog it once a year so that we can replace filters and pre-filters appropriately.
The optomechanics stock in the lab was in a sad state. We have obtained the following from Thorlabs in the last two months:
I have used some of these for the ari BHD setup. The unused items are stored in the shelves that house the optomechanics ~halfway down the east arm. I'm wondering what's a good setup to document the stock of this stuff so we can always have a healthy stock of optomechanics (at least the non speciality ones like posts, spacers etc). It sucks to realize at 0000hrs that you're missing a 3mm shim or 250mm converging lens or something.
Attachment #1 shows that the ITMX, ETMY and beamsplitter Oplev light levels have decayed significantly from their values when installed. In particular, the ETMY and ITMX sum channels are now only 50% of the values when a new HeNe was installed. ELOG search revealed that ITMY and ETMX HeNes were replaced with newly acquired units in March and September of last year respectively. The ITMX oplev was also replaced in March 2019, but the replacement was a unit that was being used to illuminate our tourist attraction glass fiber at EX.
We should replace these before any vent as they are a useful diagnostic for the DC alignement reference.
2x500 ml bottles of spectroscopic grade isopropanol were delivered. I marked them with today's date and placed them in the solvent cabinet. In the process, my shoulder bumped the laser interlock switch by the door to the VEA in the drill press area, which turned the PSL NPRO off. I turned it back on just now. The other NPROs are not connected to the interlock and so were unaffected.
Attachment #1 shows that the c1rfm model isn't able to receive any signals from the front end machines at EX and EY. Attachment #2 shows that the problem appears to have started at ~430am today morning - I certainly wasn't doing anything with the IFO at that time.
I don't know what kind of error this is - what does it mean that the receiving model shows errors but the sender shows no errors? It is not a new kind of error, and the solution in the past has been a series of model reboots, but it'd be nice if we could fix such issues because it eats up a lot of time to reboot all the vertex machines. There is no diagnostic information available in all the places I looked. I'll ask the CDS group for help, but I'm not sure if they'll have anything useful since this RFM technology has been retired at the sites (?).
In the meantime, arm cavity locking in the usual way isn't possible since we don't have the trigger signals from the arm cavity transmission.
Update 1500 4 Oct: soft reboots of models didn't do the trick so I had to resort to hard reboots of all FEs/expansion chassis. Now the signals seem to be okay.
After the earthquake on September 19 2020, it looks to me like the only lasting damage to suspensions in vacuum is the ETMY UR magnet being knocked off.
Suspension ringdown tests:
I did the usual suspension kicking/ringdown test:
Attachment #2 also includes info about the matrix diagonalization, and the condition numbers of the resulting matrices are as large as ~30 for some suspensions, but I think this isn't a new feature.
The simple interferometer, composed of a single bounce reflection from ITMY and the LO beam deilvered via fiber to the AS table, can be locked - i.e. the phase of the LO beam can be controlled such that the DC light level on the DCPDs after the two beams are interfered can be stabilized. This test allows us to confirm that various parts of the sensing and actuation chain (e.g. PI PZT for homodyne phase control, Trek amplifier etc etc) are working.
I will post more quantitative analysis tomorrow.
Attachment #2 shows the servo topology.
We received 20pcs of stuffed demodulator boards from Screaming Circuits today. Some caveats:
I removed 1 from the group to stuff some components that weren't sent to Screaming Circuits and test the functionality on the benchtop, the remaining have been stored in a plastic box for now as shown in Attachment #1. The box has been delivered to Chub who will stuff the remaining 19 boards once I've tested the one piece.
LiYuan has kindly done some Total Integrating Sphere (TIS) measurements on ITMU01 and ITMU02. A summary of the measurement is attached. I uploaded the measurements and some analysis script to nodus at /home/export/home/40m_TIS. I created a Wiki page for the measurements and linked to it from the core optics page.
These TIS measurements look very similar to the TIS of the LIGO optics. Further analysis shows that the scatter loss is 10+/-1.7 ppm for ITMU01 and 8.6+/-0.4 ppm for ITMU02.
In this calculation, a gaussian beam the same size bouncing off the 40m ITMs is assumed to scatter from the mirrors. The error is calculated by moving the beam around randomly with STD of 1mm.
In LiYuan's setup, TIS is measured for scattering angles between 1 and 75 degrees. If we go further and assume that the scatter is Lambertian we can extrapolate that the total loss is 10.9+/-1.9 ppm for ITMU01 and 9.2+/-0.5 ppm for ITMU02.
These measurements complete the loss budget nicely since with the 6ppm loss predicted from the phase maps, the total loss in the arm cavities would be 6+10+10=26ppm which is very close to the 28ppm loss that was measured after the arm cavity optics were cleaned.
I have placed 3 new in box, IDP 7 forepumps along the x arm of the interferometer. These are to be used as spares for both the 40m and Clean and Bake.
Tue evening from 4pm~6pm, Koji made a social distant tour for Anchal. We were present around the PSL/AS/ETMX tables.
This looks cool, we should have something similar, can be really useful.
These boxes were moved from the 40m hallway to the inside of the VEA so that we have some space to walk around. You can find some pictures here.
Is it better than Luxor? https://labcit.ligo.caltech.edu/~jharms/luxor.html
Attachment #1: spectra of the phase noise between LO and IFO output fields sensed using the RF44 signal.
Closing a feedback loop:
We cleared up some space in the 1Y1 electronics rack to install the 3 new FE machines. I removed the current driver and laser from 1Y1, they are now stored in the E10 cabinet. I will upload some photos to gPhotos soon.
The two ITM spares and two ETM spares are together stored in the optic storage (B110) at Downs. c/o Liyuan and GariLynn
Per Koji's suggestion, I turned the PSL HEPA Variac to 50% just now, so that the power load through the burnt electrical cable is reduced by 75%.
Summary of discussion between Koji and gautam on 14 Oct:
I tried all of these last night / overnight, here are my findings.
Analog locking of the homodyne phase:
See Attachment #1.
Relative stability of two IFR2023s synchronized to the same FS725 Rb standard:
The electrical LO signal for demodulation of the 44 MHz photocurrent is provided by an IFR2023 signal generator. To maintain a fixed phase relation between this signal, and the phase modulation sidebands imprinted on the interferometer light via a separate IFO2023 signal generator, I synchronize both to the same Rb timing standard (a 10 MHz signal from the FS725 is sent to the rear panel frequency standard input on the IFR). We don't have a direct 44 MHz electrical signal available from the main IFO Marconi at the LSC rack (or anywhere else for that matter). So I decided to do this test at 55 MHz.
A look at the time domain signal:
With the Michelson locked on the dark fringe, the RF44 I and Q signals in the time domain are shown in Attachment #3 for a 1 minute stretch.
Pushed another update to MCMC simulation. This includes:
The DOFs<->RFPD associations I use are:
However, one thing that bothers me is that for some reason ~ 15 out of 160 aLigo simulations are discarded while none for A+. It can also be seen that the A+ simulations are more spread-out which might be related.
The new simulation results are attached.
I'll bring a file binder "40m wiring diagram" to home at the next chance.
There is another one on the shelf in the control room.
(I thought I put it in my bag, but it looks like that I left it somewhere around the fax area)
Alaska M7.5 20:54UTC https://earthquake.usgs.gov/earthquakes/eventpage/us6000c9hg/executive
I looked at the suspensions. The watchdogs have not been tripped.
IMC was locked but continually shaken. (and occasional unlock)
I packaged the HV coil driver into a 2U chassis, hoping for better shielding from pickup. There is still considerable excess noise in measurement vs model around 100 Hz, see Attachment #1. The projected displacement noise from this noise contribution is shown in Attachment #2 - I've also plotted the contribution from the 4.5kohm (planned value for fast path series resistance) for comparison. Attachment #3 has some photos of the measurement setup so if someone sees some red flags, please let me know.
I've run out of ideas to try and make the measurement cleaner - the presence of the rather prominent power line harmonics suggests that this is still not perfect, but what more shielding can we implement? I have to make the measurement on the circuit side of the 25 kohm series resistor, so I am using some Pomona minigrabbers to clip onto the leg of the wirewound resistor (see photos in Attachment #3), so that's not great maybe, but what's the alternative?
So if this is truly the noise of the circuit, then while it's an improvement on the current situaiton, it's unsatisfying that such a simple circuit can't match the design expectations. But how do we want to proceed?
I found this H1 alog entry by Izumi confirming that the calibrated channels CAL-CS_* need the same dewhitening filter.
This encouraged me to download the PRCL and MICH data and using Jon's example notebook. I incorporated these noise spectra into the MCMC simulation. The most recent results are attached.
I am still missing:
Also, now the MCMC repeats a simulation if it doesn't pass the RF PDs test so the number of valid simulations stays the same. I'm still not sure about why the A+ simulations are much more robust to these tests than aLigo simulations.
what is the noise level before the HV stage? i.e. how well is the acromag noise being filtered?
It's not so easy to directly measure this I think, because the filtering is rather aggressive. Attachment #1 shows the measured transfer function (dots) vs the model and Attachment #2 shows the noise. I think this checks out - but I can't definitively rule out some excess noise at 100 Hz from this stage. Because the gain of the HV stage is x31, we'd need a preamp with better than 1nV/rtHz to directly measure the noise I guess. The Acromag noise model in Attachment #2 is based on a measurement I describe here.
Andrew made a battery-powered 0.7 nVrtHz input-referred noise pre-amplifier for gain of 200. That might help you.
we'd need a preamp with better than 1nV/rtHz to directly measure the noise I guess.
RXA: 0.7 nV is OK if you're not interested in low noise measurements. Otherwise, we have the transformer coupled pre-amp from SRS which does 0.15 nV/rHz and the Rai Weiss FET amp which has 0.35 nV for high impedance sources.
We wanted to track down the excess noise seen in MC_F and other places (see the previous report by Gautam)
Setup1: The IMC was locked and MC_F signal between 500 and 1500Hz was observed. The DTT template was saved as /users/Templates/MC/MCF_noise_201023.xml
- Suspected mech resonance/jitter coupled with clipping or any other imperfections. Poked the various optics and optomechanics on the table. Basically no change. If we tap the laser chassis and the optics close to the laser source, we occasionally unlocked the IMC
- When we touched (lifted) the Innolight controller box from the shelf, for the first time we saw a significant change in the shape of the noise spectrum. The peak around the 700Hz shited towards lower frequency by a few %. Other peaks have no obvious change in the shapes and the heights.
- While observing the MC_F signal on the laptop, we went to the back of the laser controller. Placing a hand close to the fan clearly changes the peak frequency lower. By temporarily disconnecting the fan from the power supply for a short moment, the 700Hz peak could be eliminated. We also tried to see the noise level with the slow thermal servo and diagnosis DB cable disconnected, but we didn't see any significant change of the noise level.
Setup 2: Using the ALS phase tracker, we can observe the relative freq noise of the PSL laser and the ETMY AUX laser without any servo involved. This way we can freely disconnect any cables from the lasers. The measurement template for DTT was saved as /users/Templates/ALS/Y_ALS_FINE_PHASE_OUT_102320.xml
- Noise spectrum before disconnecting the cable (REF0, RMS REF1)
- The Fast PZT input to the PSL was disconnected => This made all the peaks (including the 700Hz) disappeared (REF2, RMS REF3)
- The Fast PZT input was restored as before, then the chain was disconnected at the input of the HV PZT driver (Thorlabs) => Again, this made the peaks disappeared (REF4, RMS REF5)
- The chain was disconnected at the input of the TTFSS box => Again, this made the peaks disappeared (REF6, RMS REF7)
- Disconnected the demod input and the AO cables from the IMC servo board => This made the peaks came back (REF8)
- Disconnected all the input/peripheral cables from the IMC servo board except for the connection to the TTFSS box => Still the excess noise was observed (REF9)
- In addition to the above, the cable to the FSS box was disconnected but the ground was still touching the MC servo board => This made the peaks disappeared (REF10)
The conclusion is that the noise is injected from the main circuit of the IMC servo board.
Next time we will check if the backplane connection is doing something wrong. Also, we'll test if the presence of the RF signals does something bad to the IMC board via EMI and RFI.
We have reverted the connection and tested if we lock the IMC and Y arm. ==> We saw at least they were locked for a short period. The things are still stabilizing, but left them turned on so they keep trying to lock automatically for the night.
In fact, the problem was the grounding issue (presumably on the IOO racks).
A temporary differential receiver at the TTFSS side was built using an SR560 and a few ponoma cables. This removed the structures ~850Hz.
The MC Servo Output was disconnected from the TTFSS box and monitored with SR785. The 850Hz structure was kept visible no matter what cables, including all the acromag DB cables, were removed. This made me suspicious about the measurement setup. The SR785 was connected to an AC power strip under the SP table and this was too far from the IOO rack.
The SR785 was connected to the AC power strip on 1X2, and now the difference becomes clear. No matter if the acromag cables are connected or not, the connection (particularly ground connection) between the MC servo module and the TTFSS box causes the MC servo output contaminated. (Comparison between Blue and Orange of Attachment #1). During the measurement, the EPICS switch for the fast path was disengaged (=no signal) and the VCO gain (...so called. It's just the MC Servo Gain) was set to be 0dB.
To test if the differential receiving of the MC Servo Output at the PSL helps to reduce this noise, I've built a simple (hacky) differential receiver using an SR560. (Attachment #2)
This kept the noise level same as the disconnected case (Comparison between Green and Orange of Attachment #1, I don't think the difference between them is not significant), while the IMC is locked as before.
Note that we can see that the 36kHz line was significantly reduced. Did we remove this annoying noise?
After talking with Gautam, we decided to leave this configuration while the SE-Diff cable was replaced with a more robust one. (See Attachment #3)
The PSL laser frequency performance was evakluated in the following two ways as we did last week:
1) Use the beat frequency of the free running PSL and the Y-end laser (Attachment #4). The PSL shutter was closed and thus the IMC was not locked.
2) Use the IMC MCF while the IMC was locked. (Attachment #5)
For both cases, the improvement was confirmed.
I also tried to check the reported issue by Gautam on this elog. He used 1Hz BW, but I cheated with 16Hz BW and 10x12.8kHz span PSDs. (Attachment #6)
For the measurement, IN1 GAIN of the IMC Servo was set to be 0dB and the OUT2 was switched to monitor the IN1 noise, while IN1 was terminated by a 50Ohm.
As I mentioned above, the AC power of SR785 was taken from a 1X2 power strip. Is this the reason for the power line forest look less severe compared to the previous case???
Anyway, I tried to use the same differential receiving technique (but with gain of x100) to see if this helps. The differential receiver helped to reduce the structure above 50kHz. The floor noise level was observed to be higher. I didn't pursue this any further, but the forest of the power line looked like a part of the measurement noise. This is indicative that the grounding condition on 1X2 is really not great and we need to review the configuration of the acromag grounding.
Apart from the questionable wiring on the Acromags, one other important difference is in the way the connections were made between the old VME crates to the Eurocrate backplanes, and how we do it now. The thick cables had their sheilds connected to the eurocrate ground (or at least, there was a dedicated ground lug on those cables which we screwed on to the ground terminals on the Eurocrate backplanes). However, in our current configuration, we interface the Acromag ADCs and DACs to the backplane via these adaptor boards. The shields of the DSUB cables are presumably NOT connected to the Eurocrate grounds. This should also be investigated as one potential cause of the grounding issue - while on some of the Eurocrate modules, the P1/P2 connectors may have either the "A" or "C" row of connectors shorted to ground, some may not, and the TTFSS may suffer from such an issue?
Note that we have this problem in all of the slow machines that were upgraded to Acromag (if this turns out to be the issue).
In fact, the problem was the grounding issue (presumably on the IOO racks).
I wanted to look into the ISS situation. Some weeks ago, I found the PD that was previously used as the in-loop photodiode. I wanted to use this and measure the open-loop RIN at a few places (to see if there's any variation and also to check its functionality). However, I didn't get very far tonight - for a start, the PD height is 3" (while our beam height is 4" everywhere outside the vacuum), and I needed to put together a circuit to supply the 5V bias and +/- 15 V since the transimpedance is done on the head. I was only able to do a low-level functionality test tonight, checked that the DC voltage output varied linearly with the incident power (calibrated against an NF1611 photodiode, data will be put up later). I didn't get to measuring any noise performance - is an incandescent light bulb still shot noise limited at ~10 Hz < f < 10kHz? Some notes:
Unconnected to this work - this problem reared its ugly head again (i noticed it yesterday morning already actually). I don't have the energy to embark on a fix tonight, Koji is going to be in the lab all day tomorrow and so he will fix it.
I'm starting the model restarts from remote. Then later I'll show up in the lab to do more hard resets.
==> It seems that the RFM errors are gone. Here are the steps.
that little PD in the black mount was never very good. The AD829 is not a good opamp for transimpedance and especially not good for low frequencies. Stefan Ballmer and I were able to get 2e-8 out of these (@100 Hz) many years ago.
I wonder if we have some of Zach's M2ISS photodetectors around, perhaps in QIL or Cryo. I doubt that any of them are in use now. Those had good performance nad BNC output.
Ok I was using the PD in the black mount because Rana recommended it a few weeks ago.
Regarding the M2ISS, I acquired the hardware from QIL some months ago, including a circuit board, and 2 PDs. These had LEMO outputs though (not BNC), and the mounts are not 4". These photodiodes are what I'm using as the airBHD DCPDs right now, and some photos are here - are these the photodiodes you mentioned? Or are there yet more M2ISS photodiodes? I remember Johannes had some custom mounts extruded to make them 4" high, do you mean those? Can I retrieve them his Cryo setup?
BTW, my elog scraping shows only one spectra from Stefan in the ATF elog, and the performance there is more like 1e-7/rtHz @ 100 Hz, and that’s using a dedicated high BW servo circuit, not the SR560. Am I just missing the measurement of 2e-8/rtHz?
At this point, I'm leaving the lab. All the suspensions (incl SRM) are aligned. PSL/GRX/GRY shutters were left open.
If everything else looks good, I'll start letting the dry N2 into the main volume after lunch.
Now the green transmissions are visible by the green PDs. Attachment 9 shows the trans and ref of each green beams with and without locking to TEM00. The questionable green TRY was ~0.3. If we compare this with the histrical data (Attachment 10), it is about 1/4 of the value in the past. It's not too crazy but still quite low.
BTW, nice video! @ Koji, How difficult was it to edit it into this form?
We are now ready to take the doors off. I've already done the basic prep work (loosened bolts, cleaned chamber, carts for tools, fresh ameristat on portable HEPAs etc).
[koji, rana, gautam]
This morning, we did the following;
The OSEMs remain in the EY vacuum chamber. The next set of steps are:
We will most likely work on this tomorrow. At ~1615, I briefly opened the PSL shutter and tweaked the IMC alignment. We will almost certainly change the pointing into the IMC when we remove the old OMC and rebalance that table, so care should be taken when working on that...
To be a bit more clear about what we are going to do in the OMC chamber, I marked-up some photos, see Attachments #1 and #2.
I anticipate that after this work, the only components on the table will be
Are we in agreement with this plan?
See #15656 for the updated photo
I believe the mirror next to IM1 is for the green beams to be delivered to the PSL table. I think we still want to keep it. Otherwise, the plan looks fine.
Good point - looking back, I also see that I already removed the mirror at the SW corner of the table in 2016. Revised photo in Attachment #1. There is an optic on the east edge of this table whose purpose I'm not sure of, but I'm pretty sure it isn't essential to the main functionality and so can be removed.
I got a call from Calum ~830am today saying some facilities people entered the lab, opened the south entrance door, and tripped the alarm in the process. I came to the lab shortly after and was able to reset the alarm by flipping the switch on the alarm box at the south end entrance to "Alarm OFF". Then, I double checked that the door is closed, and re-enabled the alarm. The particle count at the SP table is not unusually high and the lasers (Oplev HeNe and AUX X) were still on, so doesn't look like any lasting damage was done. The facilities people were apparently wearing laser safety goggles.
The IMC isn't resonant for a TEM00 mode at the time of writing - we are waiting for the stack to relax, at which point if the IMC isn't resonant for a TEM00 mode, we will tweak the input pointing into the IMC (we want to use the suspended cavity as the reference, since it is presumably more reliable than the table from which we removed ~50 kgs of weight and shifted the balance.