Two ITM spares (ITMU01/ITMU02) and five new PR3 mirrors (E1800089 Rev 7-1~Rev7-5) were transported to Downs for phasemap measurement
This is very interesting. Do you have the ASDC vs PRG (~ TRXor TRY) plot? That gives you insight on what is the cause of the low recycling gain.
My speculation for the worse RIN is:
- Unoptimized alignment -> Larger linear coupling of the RIN with the misalignment
- PRC TT misalignment (~3Hz)
Don't can you check the correlation between the POP QPD and the arm RIN?
I see. At the 40m, we have the direct transition from ALS to RF. But it's hard to compare them as the storage time is very different.
Which 1f signals are you going to use? PRCL has sign flipping at the carrier critical coupling. So if the IFO is close to that condition, 1f PRCL suffers from the sign flipping or large gain variation.
GariLynn worked on the measurement of E1800089 mirrros.
The result of the data analysis, as well as the data and the codes, have been summarized here:
We can limit the EPICS values giving some parameters to the channels. cf https://epics.anl.gov/tech-talk/2012/msg00147.php
But this does not solve the MC1 issue. Only we can do right now is to make the output resister half, for example.
Can you describe the mode matching in terms of the total MM? Is MM_total = sqrt(MM_vert * MM_horiz)?
1. I agree that it's likely that it was the temp signal glitch.
Recom #2: I approve to reopen the valves to pump down the main volume. As long as there is no frequent glitch, we can just bring the vacuum back to normal with the current software setup.
2. Recom #1 is also reasonable. You can use simple logic like if we register 10 consecutive samples that exceed the threshold, we can activate the interlock. I feel we should still keep the temp interlock. Switching between pumping mode and the normal operation may cause unexpected omission of the interlocks when it is necessary.
3. We should purchase the UPS battery / replacement rotary TIP seal. Once they are in hand, we can stop the vacuum and execute the replacement. Can one person (who?) accomplish everything with some remote help?
4. The lab temp: you mean, 12degC swing with the AC on!?
Jon and Koji remotely supported Jordan's resetting the TP2 controller.
From the operator's console in front of the vac rack:
Open a terminal window (click the LXTerminal icon on the desktop)
Type "control" + enter to open the vac controls screen
Toggle all the open valves closed (edit by KA: and manually close RV2 by rotating the gate valve handle )
Turn OFF TP2 by clicking the "Off' button. Make sure the status changes and the rotation speed falls to zero (you'll also hear the pump spinning down)
The other pumps (TP1, TP3) can be left running
Once TP2 has stopped spinning, go to the back of the rack and locate the ethernet cable running from the back of the TP2 controller to the IOLAN server (near the top of the rack). Disconnect and reconnect the cable at each end, verifying it is firmly locked in place.
From the front of the rack, power down the TP2 controller (I don't quite remember for the Agilent, but you might have to move the slider on the front from "Remote" to "Local" first)
Wait about 30 seconds, then power it back on. If you had to move the slider to shut it down, revert it back to the "Remote" position.
Go back to the controls screen on the console. If the pump came back up and is communicating serially again, its status will say something other than "NO COMM"
Turn TP2 back on. Verify that it spins up to its nominal speed (66 kRPM)
At this point you can reopen any valves you initially closed (any that were already closed before, leave closed)
TP2 was stopped and at this moment the glitches were gone. Jordan powercycled the TP2 controller and we brought up the TP2 back at the full speed.
However, the glitches came back as before. Obviously we can't go on from here, and we've decided to stop the recovery process here today.
- We left TP1/2/3 running while the valves including RV2 were closed.
- When Jordan is back in the lab next week, we'll try to use TP3 as the backing of TP1 so that we can resume the main volume pumping.
- Currently, TP3 does not have interlocking and that is a risk. Jon is going to implement it.
- Meanwhile, we will try to replace the controller of TP2. We are supposed to have this in the lab. Ask Chub about the location.
- Once we confirm the stability of the diagnostic signals for TP2, we will come back to the nominal pumping scheme.
The vacuum safety policy and design are not clear to me, and I don't know what the first and second defense is. Since we had limited time and bandwidth during the remotely-supported recovery work today, we wanted to work step by step.
The pressure rising rate is 20mtorr/day, and turning on TP3 early next week will resume the main-volume pumping without too much hustle. If you need the IFO time now, contact with Jon and use backing with TP3.
ITMU01 / ITMU02 as well as the five E1800089 mirrors came back to the 40m. Instead, the two ETM spares (ETMU06 / ETMU08) were delivered to GariLynn.
Jordan worked on transportation.
Note that the E1800089 mirrors are together with the ITM container in the precious optics cabinet.
Sigh. Do we have a spare sat box?
> Can't we offload this DC signal to the laser crystal temperature servo?
No. PSL already follows the MC length. So this offset is coming from the difference between the MC length and the CARM length.
What you can do is to offload the MC length to the CARM DC if this helps.
The usual technique is that keeping the IFO locked with the old set of the signals and the relative gain/TF between the conventional and new signals are measured in-lock so that you can calibrate the new gain/demod-phase setting.
It happened before too. Doesn't it say it has occasional self-testing or something?
I believe we will use two new chassis at most. We'll replace c1ioo from Sun to Supermicro, but we recycle the existing timing system.
Grrr. Let's repair the unit. Let's get a help from Chub & Jordan.
Do you have a second unit in the lab to survive for a while?
When I tested Q3000 for aLIGO, the failure rate was pretty high. Let's get 10pcs.
The new dolphin eventually helps us. But the installation is an invasive change to the existing system and should be done at the installation stage of the 40m BHD.
Teledyne AP1053 etc were transported from Rich's office to the 40m. The box is placed on the shelf at the entrance.
My record tells that there are 7 AP1053 in the box. I did not check the number this time.
2PM: Arrived at the 40m. Started the work for the coupling of the RF modulated LO beam into a fiber. -> I left the lab at 10:30 PM.
The fiber coupling setup for the phase-modulated beam was made right next to the PSL injection path. (See attachment 1)
- Loose fiber coupler: Sorry about that. I could not detect something was loose there, although some of the locks were not tightened.
- S incident instead of P: Sorry about that too. I completely missed that the IMC takes S-pol.
- PSL HEPA was running at 33% and is now at 100%
- South End HEPA was not on and is now running
- Yarm Portable HEPA was not running and is now running at max speed: the power was taken beneath the ITMY table. It is better to unplug it when one uses the IFO.
- Yend Portable HEPA was not running and is now running (presumably) at max speed
Particle Levels: (Not sure about the unit. The convention here is to multiply x10 of the reading)
Before running the HEPAs at their maximum
9/10/2020 15:30 / 0.3um 292180 / 0.5um 14420
(cf 9/5/2020 / 0.3um 94990 / 0.5um 6210)
After running the HEPAs at their maximum
The number gradually went down and now became constant at about half of the initial values
9/10/2020 19:30 / 0.3um 124400 / 0.5um 7410
M4.5 EQ in LA 2020-09-19 06:38:46 (UTC) / -1d 23:38:46 (PDT) https://earthquake.usgs.gov/earthquakes/eventpage/ci38695658/executive
I only checked the watchdogs. All watchdogs were tripped. ITMX and ETMY seemed stuck (or have the OSEM magnet issue). They were left tripped. The watchdogs for the other SUSs were reloaded.
I came to the campus and Gautam notified that he just had received the alert from the vac watchdog.
I checked the vac status at c1vac. PTP3 went up to 10 torr-ish and this made the diff pressure for TP3 over 1torr. Then the watchdog kicked in.
To check the TP3 functionality, AUX RP was turned on and the manual valve (MV in the figure) was opened to pump the foreline of TP3. This easily made PTP3 <0.2 torr and TP3 happy (I didn't try to open V5 though).
So the conclusion is that RP for TP3 has failed. Presumably, the tip-seal needs to be replaced.
Right now TP3 was turned off and is ready for the tip-seal replacement. V5 was closed since the watchdog tripped.
I supplied a bottle of hand soap. Don't put water in the bottle to dilute it as it makes the soap vulnarable for cotamination.
There were two SUSs which didn't look normal.
- ITMX was easily released by the bias slider -> Shake the pitch slider and while all the OSEM values are moving, turn on the damping control (with x10 large watchdog threshold)
- ETMY has UR OSEM 0V output. This means that there is no light. And this didn't change at all with the slider move.
- Went to the Y table and tried to look at the coils. It seems that the UR magnet is detached from the optic and stuck in the OSEM.
We need a vent to fix the suspension, but until then what we can do is to redistribute the POS/PIT/YAW actuations to the three coils.
I came to the lab. The control room AC was off -> Now it is on.
Here is the setting of the AC meant for continuous running
Gautam reported that the PSL HEPA stopped running (ELOG 15592). So I came in today and started troubleshooting.
It looks like that the AC power reaches the motors. However, both motors do not run. It looks like the problem exists in the capacitors, the motors, or both.
Parts specs can be found in the next ELOG.
Attachment 1 is the connection diagram of the HEPA. The AC power is distributed by the breaker panel. The PSL HEPA is assigned to use M22 breaker (Attachment 2). I checked the breaker switch and it was (and is) ON. The power goes to the junction box above the enclosure (Attachment 3). A couple of wires goes to the HEPA switch (right above the enclosure light switch) and the output goes to the variac. The inside of the junction box looked like this (Attachment 4).
By the way, the wires were just twisted and screwed into a metal threaded (but isolated) caps (Attachment 5). Is this legit? Shouldn't we use stronger crimping? Anyway, there was nothing wrong with the caps w.r.t the connection for now.
I could easily trace the power up to the variac. The variac output was just fine (Attachment 6). The cord goes from the variac to the junction box (and then HEPAs) looked scorched. The connection from the plug to HEPAs was still OK, but this should be eventually replaced. Right now the cable was unplugged after the following tests for the safety reason.
The junction box for each HEPA unit was opened to check the voltage. The supply voltage came to the junction boxes and it was just fine. In Attachments 8 & 9, the voltages look low but this is because I just turned the variac only a little.
At the (main) junction box, the resistances of the HEPAs were checked with the Fluke. As the HEPA units are connected to the AC in parallel, the resistances were individually checked as follows.
The coils were not disconnected (... I wonder if the wiring of South HEPA was flipped? But this is not the main issue right now.)
By removing the pre-filters, the motors were inspected Attachments 10 & 11. At least the north HEPA motor was warm, indicating there was some current before. A capacitor was connected per motor. When the variac was tuned up a bit, one side of the capacitor could see the voltage. I could not judge which has the issue between the capacitor and the motor.
Dimensions / Specs
- HEPA unit dimentions
- HEPA unit manufacturer
Here is the timeline. This suggests TP2 backing RP failure.
1st line: TP2 foreline pressure went up. Accordingly TP2 P, current, voltage, and temp went up. TP2 rotation went down.
2nd line: TP2 temp triggered the interlock. TP2 foreline pressure was still high (10torr) so TP2 struggled and was running at 1 torr.
3rd line: Gautam's operation. TP2 was isolated and stopped.
Between the 1st line and 2nd line, TP2 pressue (=TP1 foreline pressure) went up to 1torr. This made TP1 current increased from 0.55A to 0.68A (not shown in the plot), but TP1 rotation was not affected.
Tue evening from 4pm~6pm, Koji made a social distant tour for Anchal. We were present around the PSL/AS/ETMX tables.
Is it better than Luxor? https://labcit.ligo.caltech.edu/~jharms/luxor.html
The two ITM spares and two ETM spares are together stored in the optic storage (B110) at Downs. c/o Liyuan and GariLynn
The AC cord from the PSL HEPA variac to the junction box was replaced.
Now the HEPA is running at 70%
Showed up at the 40m at 7pm
Closing the work
Leaving the 40m at 9:30pm
Memo: 40m wiring/Mask/Camera/Red Pitaya/Particle Counter
I'll bring a file binder "40m wiring diagram" to home at the next chance.
There is another one on the shelf in the control room.
(I thought I put it in my bag, but it looks like that I left it somewhere around the fax area)
Alaska M7.5 20:54UTC https://earthquake.usgs.gov/earthquakes/eventpage/us6000c9hg/executive
I looked at the suspensions. The watchdogs have not been tripped.
IMC was locked but continually shaken. (and occasional unlock)
I set up an action cam (DJI OSMO Pocket) and brought it back to the 40m. The kit is now placed in the control room cabinet together with the Canon DSLR.
I might have left the USBC chaging cable at home this time. Will bring it back next time.-> The cable was returned to the kit on Oct 23rd.
We wanted to track down the excess noise seen in MC_F and other places (see the previous report by Gautam)
Setup1: The IMC was locked and MC_F signal between 500 and 1500Hz was observed. The DTT template was saved as /users/Templates/MC/MCF_noise_201023.xml
- Suspected mech resonance/jitter coupled with clipping or any other imperfections. Poked the various optics and optomechanics on the table. Basically no change. If we tap the laser chassis and the optics close to the laser source, we occasionally unlocked the IMC
- When we touched (lifted) the Innolight controller box from the shelf, for the first time we saw a significant change in the shape of the noise spectrum. The peak around the 700Hz shited towards lower frequency by a few %. Other peaks have no obvious change in the shapes and the heights.
- While observing the MC_F signal on the laptop, we went to the back of the laser controller. Placing a hand close to the fan clearly changes the peak frequency lower. By temporarily disconnecting the fan from the power supply for a short moment, the 700Hz peak could be eliminated. We also tried to see the noise level with the slow thermal servo and diagnosis DB cable disconnected, but we didn't see any significant change of the noise level.
Setup 2: Using the ALS phase tracker, we can observe the relative freq noise of the PSL laser and the ETMY AUX laser without any servo involved. This way we can freely disconnect any cables from the lasers. The measurement template for DTT was saved as /users/Templates/ALS/Y_ALS_FINE_PHASE_OUT_102320.xml
- Noise spectrum before disconnecting the cable (REF0, RMS REF1)
- The Fast PZT input to the PSL was disconnected => This made all the peaks (including the 700Hz) disappeared (REF2, RMS REF3)
- The Fast PZT input was restored as before, then the chain was disconnected at the input of the HV PZT driver (Thorlabs) => Again, this made the peaks disappeared (REF4, RMS REF5)
- The chain was disconnected at the input of the TTFSS box => Again, this made the peaks disappeared (REF6, RMS REF7)
- Disconnected the demod input and the AO cables from the IMC servo board => This made the peaks came back (REF8)
- Disconnected all the input/peripheral cables from the IMC servo board except for the connection to the TTFSS box => Still the excess noise was observed (REF9)
- In addition to the above, the cable to the FSS box was disconnected but the ground was still touching the MC servo board => This made the peaks disappeared (REF10)
The conclusion is that the noise is injected from the main circuit of the IMC servo board.
Next time we will check if the backplane connection is doing something wrong. Also, we'll test if the presence of the RF signals does something bad to the IMC board via EMI and RFI.
We have reverted the connection and tested if we lock the IMC and Y arm. ==> We saw at least they were locked for a short period. The things are still stabilizing, but left them turned on so they keep trying to lock automatically for the night.
The particle counter on the 40m PSL was removed. The package was made together with the OMC lab particle counter (see the packing list below).
The kit was picked up by Radhika for a python code to read out the numbers.
=== Packing List ===
In fact, the problem was the grounding issue (presumably on the IOO racks).
A temporary differential receiver at the TTFSS side was built using an SR560 and a few ponoma cables. This removed the structures ~850Hz.
The MC Servo Output was disconnected from the TTFSS box and monitored with SR785. The 850Hz structure was kept visible no matter what cables, including all the acromag DB cables, were removed. This made me suspicious about the measurement setup. The SR785 was connected to an AC power strip under the SP table and this was too far from the IOO rack.
The SR785 was connected to the AC power strip on 1X2, and now the difference becomes clear. No matter if the acromag cables are connected or not, the connection (particularly ground connection) between the MC servo module and the TTFSS box causes the MC servo output contaminated. (Comparison between Blue and Orange of Attachment #1). During the measurement, the EPICS switch for the fast path was disengaged (=no signal) and the VCO gain (...so called. It's just the MC Servo Gain) was set to be 0dB.
To test if the differential receiving of the MC Servo Output at the PSL helps to reduce this noise, I've built a simple (hacky) differential receiver using an SR560. (Attachment #2)
This kept the noise level same as the disconnected case (Comparison between Green and Orange of Attachment #1, I don't think the difference between them is not significant), while the IMC is locked as before.
Note that we can see that the 36kHz line was significantly reduced. Did we remove this annoying noise?
After talking with Gautam, we decided to leave this configuration while the SE-Diff cable was replaced with a more robust one. (See Attachment #3)
The PSL laser frequency performance was evakluated in the following two ways as we did last week:
1) Use the beat frequency of the free running PSL and the Y-end laser (Attachment #4). The PSL shutter was closed and thus the IMC was not locked.
2) Use the IMC MCF while the IMC was locked. (Attachment #5)
For both cases, the improvement was confirmed.
I also tried to check the reported issue by Gautam on this elog. He used 1Hz BW, but I cheated with 16Hz BW and 10x12.8kHz span PSDs. (Attachment #6)
For the measurement, IN1 GAIN of the IMC Servo was set to be 0dB and the OUT2 was switched to monitor the IN1 noise, while IN1 was terminated by a 50Ohm.
As I mentioned above, the AC power of SR785 was taken from a 1X2 power strip. Is this the reason for the power line forest look less severe compared to the previous case???
Anyway, I tried to use the same differential receiving technique (but with gain of x100) to see if this helps. The differential receiver helped to reduce the structure above 50kHz. The floor noise level was observed to be higher. I didn't pursue this any further, but the forest of the power line looked like a part of the measurement noise. This is indicative that the grounding condition on 1X2 is really not great and we need to review the configuration of the acromag grounding.
I'm starting the model restarts from remote. Then later I'll show up in the lab to do more hard resets.
==> It seems that the RFM errors are gone. Here are the steps.
At this point, I'm leaving the lab. All the suspensions (incl SRM) are aligned. PSL/GRX/GRY shutters were left open.
I believe the mirror next to IM1 is for the green beams to be delivered to the PSL table. I think we still want to keep it. Otherwise, the plan looks fine.
> I didn't bother to align the green beams to the arm cavities or re-center the Oplevs - is this necessary? It is a step in the pre-close up checklist, so maybe we should do it... The green transmission does reach the PSL table...
I don't think so. The beam is reaching the PSL, so we have no motivation to change the green alignment. Regarding the oplev, the green refl should come back to the PDH PD and this gives us additional beam reference. As soon as we find the green resonance after the pumping, we can tweak the green axis so that the spots on the mirrors become reasonable (as well as the green trans CCD on the PSL table).
Main volume pressure as of 11:30AM 2020/11/10
Yes. The datasheet has a recommendation circuit with 10uF caps. Companies are careful to show reproducible, reliably functional circuit examples on datasheets. So, if the caps are there you should try to replicate the design.
Additional bypass capacitors? I use 0.1 uF, 700V DC ceramic capacitors as bypass capacitors close to the leads of the PA95, as is recommended in the datasheet. Can adding a 10uF capacitor in parallel provide better filtering? I'm not sure if one with compatible footprint and voltage rating is readily available, I'll look around.
Basically, they repeated our specs and showed the coating performances for HR/AR for 10deg P and PR/AR for 45deg P. There is no RoC measurement by the vendor.
Nevertheless, their RoC (paper) specs should be compared with our request.