We investigated the low power issue with POX11 photodiode.
Restored arm algiment to get 0.8 max flashing on YARM and 1 max flashing on XARM. I had to move input alignment using TT2-PR3 pair and realign YARM cavity axis using ITMY-ETMY pair.
I would like to advertise this useful tool that I've been using for moving cavity axis or input beam direction. It's a simple code that makes your terminal kind of videogame area. It moves two optics together (either in same direction or opposite direction) by arrow key strokes (left, right for YAW, up, down for PIT). Since it moves two optics together, you actually control the cavity axis or input beam angle or position depening on the mode.
Jordan and I, in order to start pumpig down the RGA Volume, we began by opening V7 and VM. Afterwards, we started RP1 and RP3. After this, the pressure in the line between RP1, RP3, and V6 dropped to 3.4 mTorr. Next, we tried to open V6, although an error message popped up. We haven't been able to erase it since. But we were able to turn on TP2 with V4 closed. The pressure in that line is reporting 1.4 mTorr.
PRP on the sitemap is giving off an incorrect pressure for the line between RP1, RP3, and V6. This is verified by the pressure by the control screen and the physical controller as well.
[Paco, Anchal, Yuta]
We opened the BSC and ITMX chamber in the morning (Friday) to investigate POX11 beam clipping. We immediately found that the POX11 beam was clipping by the recently installed cable posts, so luckily no major realingment had to be done after reinstalling the cable post in a better location.
Because we had the BSC open, we decided to steer the AS1 mirror to align the AS path from ITMY all the way to the vertex chamber. Relatively small AS1 offsets (of ~ 2000 counts each) were added on PIT / YAW to center the beam on ASL (there is slight clipping along PIT, potentially because of the AS2 aperture. We then opened the vertex chamber and located the AS beam with relative ease. We decided to work on this chamber, since major changes propagate heavily downstream (simply changing the IMC pointing).
Anchal removed old optics from the vertex chamber and we installed the steering pair of mirrors for AS path. This changed the balance of the vertex table by a lot. By using the MC REFL camera beam spot we managed to coarsely balance the counterweights and recover the nominal IMC injection pointing. Simply reenabling the IMC autolocker gave us high transmission (~ 970 counts out of the typical 1200 these days).
The final IMC alignment was done by Anchal with delicate PIT motion on the input injection IMC miror to maximize the transmission (to our satisfaction, Anchal's motion was fine enough to keep the IMC locked). The end result was quite satisfying, as we recovered ~ 1200 counts of MC transmission.
Finally, we looked at the arm cavity transmission to see if we were lucky enough to see flashing. After not seeing it, we adjusted TT1 / TT2 to correct for any MMTT1 pitch adjustment needed after the vertex table rebalancing. Suprisingly, we didn't take too long and recovered the nominal arm cavity pointing after a little adjustment. We stopped here, but now the vertex table layout is final, and AS beam still needs to be aligned to the vertex in-air table.
Jordan recieved the new tip seal Friday afternoon and I continued the replacement process in the morning. Finishing up, we proceeded to test the pump in the Clean and Bake room. The pump's pressure lowered to 110 mTorr, and we continue pumping so the seal can recieve a good fitting.
Update: We have confirmed the pump is working great and have reinstalled this back into the vacuum system. Note: The same O-Rings were used.
While the pumpspool is vented, I thought it would be a convenient time to change out the tip seal on the TP3 forepump. This one had not been changed since 2018, so as preventative maintence I had JC remove the pump and begin cleaning/installing the new tip seal.
Unfortunately the tip seal broke, but I have ordered another. We should have this pump ready to go late next week. If one is needed sooner, there is a spare IDP 7 pump we can install as the TP3 forepump.
The DCC has been updated, along with the modified schematic. Links have been attached.
18 (9 pairs) Coil Drivers have been modified. Namely ETMX/ITMX/ITMY/BS/PRM/SRM/MC1/MC2/MC3.
ETMX Coil Driver 1 (UL/LL/UR)now has R=100 // 1.2k ~ 92Ohm for CH1/2/3 S2100624 ETMX Coil Driver 2 (LR/SD)now has R=100 // 1.2k ~ 92Ohm for CH3 S2100631
ITMX Coil Driver 1 (UL/LL/UR)now has R=100 // 1.2k ~ 92Ohm for CH1/2/3 S2100620 IMTX Coil Driver 2 (LR/SD)now has R=100 // 1.2k ~ 92Ohm for CH3 S2100633
ITMY Coil Driver 1 (UL/LL/UR)now has R=100 // 1.2k ~ 92Ohm for CH1/2/3 S2100623 ITMY Coil Driver 2 (LR/SD)now has R=100 // 1.2k ~ 92Ohm for CH3 S2100632
BS Coil Driver 1 (UL/LL/UR)now has R=100 // 1.2k ~ 92Ohm for CH1/2/3 S2100625 BS Coil Driver 2 (LR/SD)now has R=100 // 1.2k ~ 92Ohm for CH3 S2100649
PRM Coil Driver 1 (UL/LL/UR)now has R=100 // 1.2k ~ 92Ohm for CH1/2/3 S2100627 PRM Coil Driver 2 (LR/SD)now has R=100 // 1.2k ~ 92Ohm for CH3 S2100650
SRM Coil Driver 1 (UL/LL/UR)now has R=100 // 1.2k ~ 92Ohm for CH1/2/3 S2100626 SRM Coil Driver 2 (LR/SD)now has R=100 // 1.2k ~ 92Ohm for CH3 S2100648
MC1 Coil Driver 1 (UL/LL/UR)now has R=100 // 1.2k ~ 92Ohm for CH1/2/3 S2100628 MC1 Coil Driver 2 (LR/SD)now has R=100 // 1.2k ~ 92Ohm for CH3 S2100651
MC2 Coil Driver 1 (UL/LL/UR)now has R=100 // 1.2k ~ 92Ohm for CH1/2/3 S2100629 MC2 Coil Driver 2 (LR/SD)now has R=100 // 1.2k ~ 92Ohm for CH3 S2100652
MC3 Coil Driver 1 (UL/LL/UR)now has R=100 // 1.2k ~ 92Ohm for CH1/2/3 S2100630 MC3 Coil Driver 2 (LR/SD)now has R=100 // 1.2k ~ 92Ohm for CH3 S2100653
Will be updating this linking each coil driver to the DCC
We found that one of the Y1-1037-45P marked mirror that we used was actually curved. So we removed it and used a different Y1-1037-45P mirror, adjusted the position of the lens and got the beam to land on POX11 RFPD successfully.
Then in control room, we maximized the POX11_I_ERR PDH signal amplitude by changing C1:LSC-POX11_PHASE_R to 42.95 from -67.7. We kept the C1:LSC-POX11_PHASE_D same at 90. We were getting +/- 200 PDH signal on POX_I_ERR.
Then in our attempt to lock the XARM, when we ran the "Restore XARM (POX)" script, YARM locked!
We are not sure why the YARM locked, we might have gotten lucky today. So we ran ASS on YARM and got the transmission (TRY_OUT) stable at 1. The lock is very robust and retrievable.
Coming back to XARM, we realized that the transmission photodiode used for XARM was the low-gain QPD instead of the thorlabs high gain photodiode. The high-gain photodiode was outputing large negative counts for some reason. We went to the Xend to investigate and found that the high gain photodiode was disconnected for some reason. Does anyone know/remember why we disconnected this photodiode?
We connected the photodiode back and it seems to work normally. We changed the photodiode selection back to high gain photodiode for TRX and on 40 dB attenuation, we see flashing between 1.4 to 1.6. However, we were unable to lock the XARM. We tried changing the gain of the loop, played a little bit with the trigger levels etc but couldn't get it to lock. Next shift team, please try to lock XARM.
Jordan, Tega, JC
Issue has been resolved. Breaker on RP1 was tripped so the RP1 button was reporting ON, but was not actually on which continuously tripped the V6 interlock. Breaker was reset, RP1 and RP3 turned on. The V6 was opened to rough out the RGA volume. Once, pressure was at ~100mtorr, V4 was opened to pump the RGA with TP2. V6 was closed and RP1/3 were turned off.
RGA is pumping down and will take scans next week to determine if a bakeout is needed
[Tega, Yehonathan, Koji, Yuta]
We tried to align AS path this afternoon.
IMC is not aligned now after the work today
Green mirrors/perisocope in IMC chamber were removed since some of them was clipping the AS beam, and this changed the balance of the IMC stack and thus MC1 and MC3 alignment.
Summary of changes:
- Rotated AS2 in roll by 90 deg to have more aperture for the transmission (photo)
- IR beams are now centered on AS1, AS2, AS3 and AS4 (photo, photo)
- Moved ASL towards -X direction for about 1/4 inch
- Installed GRY_SM2 at the nominal position (re-used GR_SM3 from IMC chamber)
- Removed green optics GR_SM4, GR_SM3, GR_PERI2L (GR_PERI2L is now stored at Xend)
- Removed IFI camera mirrors FIV1, FIV2 (they are now stored at Xend) (photo, photo)
- GR_SM4 mount is now reused as GRY_SM1 (Y2-2037-0 is now mounted instead of previously mounted Y2-LW1-2050-UV-45P/AR), and GRY_SM1 is installed at the nominal position (photo)
- Moved weights to balance the stack
OMC chamber (we don't have OMC in this chamber...)
- We swapped AS5 and AS6 so that the nobs comes in -X direction to have more spacing between AS beam and IMC REFL beam (photo)
- Moved weights to balance the stack
What we did:
1. Misaligned ITMX and use ITMY reflected beam to align AS path
2. Centered the IR beam on AS1 using SR2
3. Centered the IR beam on AS2 and ASL using AS1. AS2 was rotated in roll by 90 deg to have more aperture for the transmisson.
4. Centered the IR beam on AS3 using AS2 nobs, centered the IR beam on AS4 by rotating AS3 in yaw.
5. "AS beam" (it turned out that what we are looking was actually not the AS beam!! Some stray light) was in +X direction by 1 inch or so at AS5. Moving AS5 to center the beam would clip IMC REFL beam. So we swapped AS5 and AS6 so that the nobs comes in -X direction to have more spacing between AS beam and IMC REFL beam.
6. Balanced OMC chamber stack again using IMC REFL beam as a referece (bring the IMC REFL beam to the reference red circle on the monitor).
7. Tweaked the alignment of TT1 and TT2 to have Yarm flashing to ~0.9 in TRY.
8. Moved AS5 towards +X by an inch or so to center the "AS beam."
9. Moved ASL towards -X direction for about 1/4 inch and re-centered the beam by AS1 to see if the "AS beam" gets far from IMC REFL at OMC chamber, but the "AS beam" didn't move much.
10. By blocking the beam from ITMY, we found that "AS beam" was not the actual one.
11. Opened IMC chamber and found that AS beam is blocked by the past optics.
12. Removed old green optics and IFI camera mirrors. GR_SM4 mount and GR_SM3 were reused as mentioned above.
13. Tried to balance IMC chamber stack to recover IMC alignment. We used IMC REFL beam as a reference, but it was hard to completely bring the IMC REFL beam to the reference red circle on the monitor. It is now off by a beam diameter or so. No IMC flashing now.
Theoretically, balancing IMC chamber stack would recover all the IFO alignment, but maybe tough. It is maybe easier to align MC1 and MC3 to have IMC locked. Assuming input pointing to IMC is not drifted too much, we should be able to recover Yarm flashing by tweaking TT1 alignment only. However, MC3 SD OSEM is at the edge of the range. We might have to balance the stack more or tweak SD OSEM position.
I feel like there is an instability in my thought process on this. Before my tendency to try to scale and generalize this problem brings me to a full stop I will make small but quick progress.
First thing is to calculate the noise budget for a simple Michelson. The involved optics are:
all sensed with OSEMs and OpLevs only.
Things to fetch:
1. OSEM sensing noise. Where do I get the null stream (AKA butterfly mode)?
2. Oplev noise, look at the SUM channel (or this elog)
3. Actuation TF. Latest elog.
4. Coil driver noise. Going to take the HP supply curve from this elog.
5. Seismic noise + Seismic stack TF. Or maybe just take displacement noise from gwinc.
6. Laser noise. Still need to search.
7. DAC noise. Still need to search.
I'm planning on simulating the BHD readout noise in a manner very similar to the ALS noise model using Simulink. I've made a sketch of the model for the longitudinal DOFs (attached). A model for ASC will be similar but with more measurement devices (OpLevs, QPDs, WFSs).
I'm not pretending to simulate everything in this diagram on the first go, it is just a sketch of the big picture.
[Paco, JC, Anchal]
We balanced the IMC table back again to point that got us 50% of nominal transmission from IMC. Then we tweaked the steering mirror for injection to IMC to get up to 90% of nominal transmission. Finally, we used WFS servo loop to get to the 100% nominal transmission from IMC. However, we found that the WFS loop has been compromised now. It eventually misaligns IMC if left running for a few minutes. This needs to be investigated and fixed.
We aligned the BS, ITMY, and ETMY PIT and YAW to get the flashing on X-arm whilst also keeping the flashing of Y-arm. From attachment 1, it is clear that POXDC photodiode is not receiveing any light, so our next task is to work on POX alignment.
We needed to sort out the POXDC signal so we could work on X-arm alignment. Given that POXDC channel value was approx 6 compared to POYDC value of approx. 180, we decided to open the ITMX chamber to see if we could improve the situation. We worked on the alignment of POX beam but could not improve the DC level which suggests that this was already optimized for. As an aside, we also noticed some stray IR beam from the BS chamber, just above the POX beam which we cold not identify.
Next we moved on to the POP beam alignment, where we noticed that the beam level on LO1 and POP_SM4 was a bit on the high side. Basically, the beam was completely missing the 1" POP_SM4 mirror and was close to the top edge of LO1. So we changed TT2 pitch value from 0.0143 to -0.2357 in order to move the beam position on POP_SM4 mirror. This changed the input alignment, so we compensated using PR2 (0.0 -> 49.0) and PR3 (-5976.560 -> -5689.800). This did not get back the alignment as anticipated, so we moved ITMY pitch from 0.9297 to 0.9107. All of these alignment changes moved the POP beam down by approx 1/5 of an inch from outside the mirro to the edge of POP_SM4 mirror, where about half of the beam is clipped.
We need to repeat these aligment procedures with say 1.5 time the change in TT2 pitch to center the beam on POP_SM4 mirror.
We first aligned the single arm cavity resonance for both arms to get maximum flashing. As we opened the chamber, I found that the POP beam was mostly hitting the POP_SM4 mirror but was clipping about 2 mm on the top edge.
I used TT2-PR3 to lower the injection beam angle and moved pairs of ITMY-ETMY, and ITMX-ETMX to recover as much flashing as I could in the both arms. Then, I moved PR2 in pitch from 49 to 71 to maximize the arm flashing again. After these steps, the POP beam was clearly within the POP_SM4 mirror but still in the upper half of the optic and there was maybe just a mm of clearance from the top edge. I decided to raise POP_SM4 mirror by 0.14" spacer. Now the beam is still in upper half of the mirror but has a good clearance from the edge.
The POP beam is coming outside in the in-air table at as a rising beam in the nominal path near the center of the window. This beam needs to be directed to the POP camera and RFPD on the far-side of the table.
In order to setup POP camera and RFPD on the ITMX table, we decided to first work on the IMC and X/Y-arm alignment.
We zeroed IMC WFS outputs and aligned IMC manually to get IMC transmission of 1200 and reflection of 0.35.
We used the new video game tool that moves the pairs of mirrors - PR3 & ETMY, ITMY & ETMY - in common and differential modes. This brought the Y-arm flashing to 0.8. Note that we used the _OFFSET bias values for PR3 & ETMY alignment instead of the _COMM bias values.
We repeated the same procedure of moving the pairs of mirrors - BS & ETMX, ITMX & ETMX - in common and differential modes but manually this time. This brought the X-arm flashing to ~1.0.
I have made a Simulink diagram to use in the MICH modeling (attachment) for the homodyne angle detection scheme. The model will be used for each optic separately and the noises will be combined in quadrature.
I gathered some more bits of info to fill the Simulink boxes. This is what I have so far:
# Displacement noises from gwinc
# OSEM sensing noise from the null stream
# OpLev noise from SUM channel + Seismic motion
freq = np.logspace(1, 4, 100)
coil_driver_noise = 1*freq/freq # pA/sqrt(Hz), elog 15846
RIN = 1e-2*freq/freq #1/sqrt(Hz), elog 16082
freq_noise = (1e6/freq**2) #Hz/sqrt(Hz), elog 15431
dark_noise = 1e-8 #V/sqrt(Hz) https://wiki-40m.ligo.caltech.edu/Electronics/RFPD/AS55
ADC_noise = 1e-6 #V/sqrt(Hz)
DAC_noise = 1e-6 #V/sqrt(Hz), elog 13003
#POS->BHD from Finesse
#RIN->BHD from Finesse
#Frequency noise->BHD from finesse
#Control filters from MEDM
#Whitening filters from https://wiki-40m.ligo.caltech.edu/Electronics/WhiteningFilters
#Dewhitening filters from elog 12983
DAC_gain = 6.285e-4 #V/cts, elog 16161
coil_driver_gain = 31 # elog 15534
coil_driver_TF = 0.016 #N/A per coil, elog 15846
coil_R = 20e3 #Ohm,, elog 15846
SUS_TF = 1/(0.25*freq**2) #m/N, single pendulum
OSEM_TF = 2*16384*1e3 #cts/m, https://wiki-40m.ligo.caltech.edu/Calibration
ADC_TF = 1638.4 #cts/V
DCPD_responsivity = 0.8 #A/W
DCPD_transimpedance = 1e3 #V/A
We investigated why WFS loop wasn't working. It seemed like WFS1 PIT error signal has a huge offset which would push the loop to misalign all optics' PIT. So we did the following steps:
As I went to correct the ITMX Oplev mirrors, I found that both mirrors were placed in very different positions than the design position. Part of the reason I think was to preserve outside oplev path, and party because a counterweight was in ITMXOL1 position. I had to do following steps to correct this:
During the above work, i must have kicked the cable between the vacuum flange and the satellite amplifier box for ITMX. This disconnected all the OSEMs and Coils. We tried several things to debug this and finally found that nudging the connections on Sat Amp box brought the OSEMs and coils back online. Note that the connector was not partially out or in a state that obviously showed disconnection of the pins. I'm glad we are putting in new electronics soon for the vertex optics as well.
Started work on the relocating the green transmission optics, cameras and PDs. Before removing the any of the optics, we checked and confirmed that the PDs and Cams are indeed connected to the GRN TRX/Y medm channels. Then added labels to the cables before moving them.
Relocated Optics & PDs & Cameras:
Don is working on finalizing the BHD Platform design. All the components on the BHD platform are almost populated and aligned.
Don is still working on the table legs so that we can detach the legs when we need to float the table in the future.
The BHD BS mount will have a third picomotor so that we can steer 3 dof with the mount while the remaining dof needs to be provided by the OMC.
The BHD BS position is going to be adjusted so that the incident and trans beams have sufficient clearance.
The OMC legs (kinematic mounts) need more work so that we can adjust their positions for initial setup while they can be the reference for the reproducible placement of the OMCs.
The OMCs are rigidly held with the legs. For the damping of the 1-kHz body bode, which has a relatively high Q, there will be a dissipative element touching the glass breadboard.
I quickly ran the FEA model to check the resonant freqs of the BHD platform.
The boundary conditions were:
Don has optimized the cut-out size for the OMCs to increase the rigidity of the plate. Also, the ribbed grid is made at the bottom side.
The lowest mode is at 168Hz. Because there is no leg around, it seems reasonable to have this kind of mode as the fundamental mode.
The other mode lined up at 291Hz, 394Hz, 402Hz, ...
The mode freqs will be lower once the platform is loaded. But as the unloaded platform mode, these mode freqs sound pretty good numbers.
Tega and I cleaned up the BS OPLEV Table and took out a couple of mirrors and an extra PD. The PD which was removed is "IP-POS - X/Y Reversed". In addition to this, the cable is zip-tied to the others located on the outside of the table in case this is required later on.
Next, we placed the cameras and mirrors for the green beam into their postions. A beam splitter and 4 mirrors were relocated from PSL table and placed onto the BS Oplev table to complete this. I will upload the picture of the newly updated photo with arrows of the beam routes.
Followed the steps below to complete the ITMX optlev installation. The ITMX optlev return beam now reaches its QPD without being blocked by the input steering mirror.
Although, I centered the ITMX optlev readout, this was not done when the XARM flashing is maximized bcos the IMC chamber was being worked on, so this should be done later when the IR beam is back.
We installed GRX_SM1, GRX_SM2, and finished aligning the GRY_SM1, and GRY_SM2 steering mirrors in the BS and IMC Chambers. GRY_SM1 was slightly misplaced (by ~ 2 inches), so we had to move it slightly. Luckily this didn't grossly misaligned the IMC, and we could recover quickly by touching MC1 & MC3 pitch, and MC1 slight yaw.
Then, Yuta installed GRX_SM1, and GRX_SM2 by repurposing two 45 AOI P-Pol GR transmission mirrors on the flowbench. Because one of the weights on the BSC was in the way of GRX_SM2, it was shifted it before installation. This probably shifted the balancing of the whole table. The GRY beam is still not lock-able to the YARM, so as a proxy for GRY transmission beam we used the slight GRX reflection from the BS, and noted slight clipping through PR3 (in transmission). This should probably be checked with GTRY.
We believe this is the last installation operation of this vent.
We made sure the WFS feedback loop is working, and realigned the arm cavities to be flashing.
Paco and I began aligning the Green Beam in the BS Oplev Table. while aligning the GRN-TRX, the initial beam was entering the table a bit low. To fix this, Paco went into the chamber and correcting the pitch with the steering mirror. The GRN-TRX is now set up, both the PD and Camera. Paco is continuing to work on the GRN-TRY and will update later on today.
In the morning, I will update this post with photos of the new arrangement of the BS OPLEV Table.
Update Wed May 11 16:54:49 2022
GRY is now better mode matched to the YARM and is on the edge of locking, but it more work is needed to improve the alignment. The key difference this time with respect to previous attempts was to scan the two lenses on translation stages along the green injection path. This improved the GTRY level by a factor of 2.5, and I know it can be further improved. Anyways, the locked HOMs are nicely centered on the GTRY PD, so we are likely done with the in-vac GTRY GTRX alignment.
Update Wed May 12 10:59:22 2022
The GTRX PD is now set up and connected. The camera have been set to an angle because the cable to connect it is too thick for the camera to maintain its original position along the side.
Tega and I went in to adjust the POP being in the ITMX Table. The beam entered the table high, so we adjusted the this by adding mirrors (The highlighted in Turqoise are mirrors which adjust the pitch of the beam). All the mirrors are set and we are now in the process of adjusting the PD.
Got POP beam centered on camera and nominally on the two PDs. Attachment #1 shows "carrier" camera.
We finally managed to steer the AS beam from ITMY chamber, through BS and IMC chambers, to the in-air AP table.
We moved the AS5 mirror north to its nominal position and we also moved the ASL lens on BS chamber back to its nominal position. Attached photos are taken after today's alignment work.
I successfully steered out the two output beams from BHD BS to ITMY table today. This required significant changes on the table, but I was able to bring back the table to balance coarsely and then recover YARM flashing with fine tuning of ITMY.
We checked POX and POY RF signal chains for sanity check since Xarm cannot be locked in IR stably as opposed to Yarm.
POX beam seems to be healthy. This issue doesn't prevent us from closing the vacuum tank.
- RF PD has SPB-10.7+ and ZFL-500NL+ attached to the RF output.
- At the demodulation electronics rack, SMA connectors are used everywhere.
- With Yarm flashing at ~1, RF output has ~24 mVpp right after RF PD, ~580mVpp after SPB-10.7+ and ZFL-500NL+, and ~150mVpp at right before the demodulation box.
- There is roughly a factor of 3 loss in the cabling from POY RF PD to the demodulation rack.
- Laser power at POY RF PD was measured to be 16 uW
- RF PD doesn't have amplifiers attached.
- At the demodulation electronics rack, N connector is used.
- With Xarm flashing at ~1, RF output has ~30 mVpp right after RF PD, and ~20mVpp at right before the demodulation box.
- Losses in the cabling from POX RF PD to the demodulation rack is small compared with that for POY.
- Laser power at POX RF PD was measured to be 16 uW
- POX and POY RF PDs are receiving almost the same mount of power
- POY has larger error signal than POX because of RF amplifier, but the cable loss is high
- There might be something in the electronics, but we can close the vacuum tanks
Yehonathan and I attempted to align the LO2 beam today through the BS chamber and ITMX Chamber. We found the LO2 beam was blocked by the POKM1 Mirror. During this attempt, I tapped TT2 with the Laser Card. This caused the mirror to shake and dampen into a new postion. Afterwards, when putting the door back on ITMX, one of the older cables were pulled and the insulation was torn. This caused some major issues and we have been able to regain either of the arms to their original standings.
[Yuta, Anchal, Paco]
As described briefly by JC, there were multiple failure modes going during this work segment.
Indeed, the 64 pin crimp cable from the gold sat amp box broke when work around ITMX chamber was ongoing. We found the right 64 pin head replacement around and moved on to fix the connector in-situ. After a first attempt, we suddenly lost all damping on vertex SUS (driven by these old sat amp electronics) because our c1susaux acromag chassis stopped working. After looking around the 1x5 rack electronics we noted that one of the +- 20 VDC Sorensens were at 11.6 VDC, drawing 6.7 A of current (nominally this supply draws over 5 Amps!) so we realized we had not connected the ITMX sat amp correctly, and the DC rail voltage drop busted the acromag power as well, tripping all the other watchdogs ...
We fixed this by first, unplugging the shorted cable from the rack (at which point the supply went back to 20 VDC, 4.7 A) and then carefully redoing the crimp connector. The second attempt was successful and we restored the c1susaux modbusIOC service (i.e. slow controls).
As we restored the slow controls, and damped most vertex suspensions, we noticed ITMY UL and SD osems were reading 0 counts both on the slow and fast ADCs. We suspected we had pulled some wires around when busy with the ITMX sat amp saga. We found that Side OSEM cLEMO cable was very loose on the whitening board. In fact, we have had no side osem signal on ITMY for some time. We fixed this. Nevertheless the UL channel remained silent... We then did the following tests:
DO NOT TRUST THE SATELLITE BOX TESTER 2.
I heard a rumor about a DAQ problem at the 40m.
To investigate, I tried retrieving data from some channels under C1:SUS-AS1 on the c1sus2 front end. DQ channels worked fine, testpoint channels did not. This pointed to an issue involving the communication with awgtpman. However, AWG excitations did work. So the issue seemed to be specific to the communication between daqd and awgtpman.
daqd logs were complaining of an error in the tpRequest function: error code -3/couldn't create test point handle. (Confusingly, part of the error message was buffered somewhere, and would only print after a subsequent connection to daqd was made.) This message signifies some kind of failure in setting up the RPC connection to awgtpman. A further error string is available from the system to explain the cause of the failure, but daqd does not provide it. So we have to guess...
One of the reasons an RPC connection can fail is if the server name cannot be resolved. Indeed, address lookup for c1sus2 from fb1 was broken:
$ host c1sus2
Host c1sus2 not found: 3(NXDOMAIN)
$ host c1sus2
Host c1sus2 not found: 3(NXDOMAIN)
In /etc/resolv.conf on fb1 there was the following line:
Changing this to search martian got address lookup on fb1 working:
$ host c1sus2
c1sus2.martian has address 192.168.113.87
$ host c1sus2
c1sus2.martian has address 192.168.113.87
But testpoints still could not be retrieved from c1sus2, even after a daqd restart.
In /etc/hosts on fb1 I found the following:
Changing the hardcoded address to the value returned by the nameserver (192.168.113.87) fixed the problem.
It might be even better to remove the hardcoded addresses of front ends from the hosts file, letting DNS function as the sole source of truth. But a full system restart should be performed after such a change, to ensure nothing else is broken by it. I leave that for another time.
[Anchal, Paco, JC]
Thanks Chris for the fix. We are able to access the testpoints now but we started facing another issue this morning, not sure how it is related to what you did.
The steps we have tried to fix this are:
These above steps did not fix the issue. Since we have the testpoints (C1:SUS-ETMX_TRX_OUT & C1:SUS-ETMY_TRY_OUT) for now to monitor the transmission levels, we are going ahead with our upgrade work without resovling this issue. Please let us know if you have any insights.
It looks like the RFM problem started a little after 2am on Saturday morning (attachment 1). It’s subsequent to what I did, but during a time of no apparent activity, either by me or others.
The pattern of errors on c1rfm (attachment 2) looks very much like this one previously reported by Gautam (errors on all IRFM0 ipcs). Maybe the fix described in Koji’s followup will work again (involving hard reboots).
After Xarm and Yarm were aligned by Anchal et al, I aligned AS and REFL path in the AP table.
REFL path was alreasy almost perfectly aligned.
-REFL beam centered on the REFL camera
-Aligned so that REFL55 and REFL33 RFPDs give maximum analog DC outputs when ITMY was misaligned to avoid MICH fringe
-Aligned so that REFL11 give maximum C1:LSC-REFL11_I_ERR (analog DC output on REFL11 RFPD seemed to be not working)
-AS beam centered on the AS camera. AS beam seems to be clipped at right side when you see at the viewport from -Y side.
-Aligned so that AS55 give maximum C1:LSC-ASDC_OUT16 (analog DC output on AS55 RFPD seemed to be not working)
-Aligned so that AS110 give maximum analog DC output
We followed the manual's guide for setting up MTS to sync on external signal. In the xrfdc package, we update the RFdc class to have RunMTS, SysRefEnable, and SysRefDisable functions as prescribed on page 180 of the manual. Then, we attempted to run the new functions in the notebook and read the DAC signal outputs on an oscilloscope. The DACs were not synced. We were also unable to get FIFOlatency readings.
I was finally able to set up a stable suspension model with the help of Yuta and I'm now ready to start doing some MICH noise budgeting with BHD readout. (Tip: turns out that in the zpk function in Matlab you should multiply the poles and zeros by -2*pi to match the zpk TFs in Foton)
I copied all the filters from the suspension MEDM screens into a Matlab. Those filters were concatenated with a single pendulum suspension TF with poles at [0.05e-1+1i, 0.05e-1-1i] and a gain of 4 N/kg.
I multiplied the OLTF with the real gains at the DAC/DAC/OSEMs/Coil Driver and Coils. I ignore whitening/dewhitening for now. The OLTF was calculated with no additional ad-hoc gain.
Attachment 1 shows the calculated open-loop transfer function.
Attachment 2 shows OLTF of ETMY measured last week.
Attachment 3 shows the step and impulse responses of the closed-loop system.
[Anchal, Paco, Yuta]
The SRM Oplev injection and detection paths interfere heavily with the POY11. Due to the limited optical access, I suggest we try steering POYM1 YAW and adapting the RFPD path accordingly.
I centered WFS1 PD so that IMC WFS Servo does not go out of range.
[Anchal, Paco, Yuta, JC]
After agreement from Yuta/Anchal, I moved POYM1 yaw to clear the aforementioned path, and Ian restored the POY11 RFPD path. The demodulation phase might need to be corrected afterwards, before any lockign attempts.
Current OSEM sensor values with all the suspensions aligned are attached.
For 'BS','ITMX','ETMX','ITMY','ETMY','PRM','SRM','LO1','LO2', the ones out of the range [200,800] are marked, and for 'PR2','PR3','SR2','AS1','AS4', the ones out of the range [6000,24000] are marked.
[JC, Tega, Chub]
Today we installed the 200 lbs doors on the end station chambers.
Today, in short we:
Prep for closing and pump down.
[Chub, JC, Jordan, Yuta, Yehonathan, Paco]
Closed in the following order:
After closing the heavy doors, we tried to have GTRY less clipped using PR2, PR3, ITMY and ETMY. During this adventure, we also aligned GRY injection beam by hand. Rotating a waveplate for GRY injection made GRY locking stably at GTRY of ~0.3.
[Vacuum gauge sensors]
Paco informed me that the FRG sensor EPICS channels are not available on dataviewer, so I added them to slow channels ini file (/opt/rtcds/caltech/c1/chans/daq/C0EDCU.ini). I also commented out the old CC1, CC2, CC3 and CC4 gauges. A service restart is required for them to become available but this cannot be done right now because it would adversely affect the progress of the upgrade work. So this would be done at a later date.
git repo - https://git.ligo.org/40m/vac
Finally incorporated the FRGs into the main modbusIOC service and everything seems to be working fine. I have also removed the old sensors (CC1,CC2,CC3,CC4,PTP1,IG1) from the serial client list and their corresponding EPICS channels. Furthermore, the interlock service python script has been updated so that all occurrence of old sensors (turns out to be only CC1) were replaced by their corresponding new FRG sensor (FRG1) and a redundnacy was also enacted for P1a where the interlock condition is replicated with P1a being replaced with FRG1 because they both sense the main volume pressure.