I have restarted c1su2 model with the connections of Run Acquire switch to analog filters on coil drivers. Following steps were taken:
First ssh to c1sus2 and then:
controls@c1sus2:~ 0$ rtcds make c1su2
### building c1su2...
Parsing the model c1su2...
Building EPICS sequencers...
Building front-end Linux kernel module c1su2...
RCG source code directory:
The following files were used for this build:
Successfully compiled c1su2
Compile Warnings, found in c1su2_warnings.log:
WARNING *********** No connection to subsystem output named SUS_DAC1_12
WARNING *********** No connection to subsystem output named SUS_DAC1_13
WARNING *********** No connection to subsystem output named SUS_DAC1_14
WARNING *********** No connection to subsystem output named SUS_DAC1_15
WARNING *********** No connection to subsystem output named SUS_DAC2_7
WARNING *********** No connection to subsystem output named SUS_DAC2_8
WARNING *********** No connection to subsystem output named SUS_DAC2_9
WARNING *********** No connection to subsystem output named SUS_DAC2_10
WARNING *********** No connection to subsystem output named SUS_DAC2_11
WARNING *********** No connection to subsystem output named SUS_DAC2_12
WARNING *********** No connection to subsystem output named SUS_DAC2_13
WARNING *********** No connection to subsystem output named SUS_DAC2_14
WARNING *********** No connection to subsystem output named SUS_DAC2_15
controls@c1sus2:~ 0$ rtcds install c1su2
### installing c1su2...
Installing system=c1su2 site=caltech ifo=C1,c1
Installing start and stop scripts
Updating testpoint.par config file
/opt/rtcds/rtscore/branches/branch-3.4/src/epics/util/updateTestpointPar.pl -par_file=/opt/rtcds/caltech/c1/target/gds/param/archive/testpoint_220315_135808.par -gds_node=26 -site_letter=C -system=c1su2 -host=c1sus2
Installing GDS node 26 configuration file
Installing auto-generated DAQ configuration file
Installing Epics MEDM screens
Running post-build script
/opt/rtcds/userapps/release/cds/common/scripts/generate_KisselButton.py 4 5 C1:SUS-AS1_INMATRIX > /opt/rtcds/caltech/c1/medm/c1su2/C1SUS_AS1_INMATRIX_KB.adl
/opt/rtcds/userapps/release/cds/common/scripts/generate_KisselButton.py 2 4 C1:SUS-AS1_LOCKIN_INMTRX > /opt/rtcds/caltech/c1/medm/c1su2/C1SUS_AS1_LOCKIN_INMTRX_KB.adl
/opt/rtcds/userapps/release/cds/common/scripts/generate_KisselButton.py 5 6 C1:SUS-AS1_TO_COIL --fi > /opt/rtcds/caltech/c1/medm/c1su2/C1SUS_AS1_TO_COIL_KB.adl
/opt/rtcds/userapps/release/cds/common/scripts/generate_KisselButton.py 4 5 C1:SUS-AS4_INMATRIX > /opt/rtcds/caltech/c1/medm/c1su2/C1SUS_AS4_INMATRIX_KB.adl
/opt/rtcds/userapps/release/cds/common/scripts/generate_KisselButton.py 2 4 C1:SUS-AS4_LOCKIN_INMTRX > /opt/rtcds/caltech/c1/medm/c1su2/C1SUS_AS4_LOCKIN_INMTRX_KB.adl
/opt/rtcds/userapps/release/cds/common/scripts/generate_KisselButton.py 5 6 C1:SUS-AS4_TO_COIL --fi > /opt/rtcds/caltech/c1/medm/c1su2/C1SUS_AS4_TO_COIL_KB.adl
/opt/rtcds/userapps/release/cds/common/scripts/generate_KisselButton.py 4 5 C1:SUS-LO1_INMATRIX > /opt/rtcds/caltech/c1/medm/c1su2/C1SUS_LO1_INMATRIX_KB.adl
/opt/rtcds/userapps/release/cds/common/scripts/generate_KisselButton.py 2 4 C1:SUS-LO1_LOCKIN_INMTRX > /opt/rtcds/caltech/c1/medm/c1su2/C1SUS_LO1_LOCKIN_INMTRX_KB.adl
/opt/rtcds/userapps/release/cds/common/scripts/generate_KisselButton.py 5 6 C1:SUS-LO1_TO_COIL --fi > /opt/rtcds/caltech/c1/medm/c1su2/C1SUS_LO1_TO_COIL_KB.adl
/opt/rtcds/userapps/release/cds/common/scripts/generate_KisselButton.py 4 5 C1:SUS-LO2_INMATRIX > /opt/rtcds/caltech/c1/medm/c1su2/C1SUS_LO2_INMATRIX_KB.adl
/opt/rtcds/userapps/release/cds/common/scripts/generate_KisselButton.py 2 4 C1:SUS-LO2_LOCKIN_INMTRX > /opt/rtcds/caltech/c1/medm/c1su2/C1SUS_LO2_LOCKIN_INMTRX_KB.adl
/opt/rtcds/userapps/release/cds/common/scripts/generate_KisselButton.py 5 6 C1:SUS-LO2_TO_COIL --fi > /opt/rtcds/caltech/c1/medm/c1su2/C1SUS_LO2_TO_COIL_KB.adl
/opt/rtcds/userapps/release/cds/common/scripts/generate_KisselButton.py 4 5 C1:SUS-PR2_INMATRIX > /opt/rtcds/caltech/c1/medm/c1su2/C1SUS_PR2_INMATRIX_KB.adl
/opt/rtcds/userapps/release/cds/common/scripts/generate_KisselButton.py 2 4 C1:SUS-PR2_LOCKIN_INMTRX > /opt/rtcds/caltech/c1/medm/c1su2/C1SUS_PR2_LOCKIN_INMTRX_KB.adl
/opt/rtcds/userapps/release/cds/common/scripts/generate_KisselButton.py 5 6 C1:SUS-PR2_TO_COIL --fi > /opt/rtcds/caltech/c1/medm/c1su2/C1SUS_PR2_TO_COIL_KB.adl
/opt/rtcds/userapps/release/cds/common/scripts/generate_KisselButton.py 4 5 C1:SUS-PR3_INMATRIX > /opt/rtcds/caltech/c1/medm/c1su2/C1SUS_PR3_INMATRIX_KB.adl
/opt/rtcds/userapps/release/cds/common/scripts/generate_KisselButton.py 2 4 C1:SUS-PR3_LOCKIN_INMTRX > /opt/rtcds/caltech/c1/medm/c1su2/C1SUS_PR3_LOCKIN_INMTRX_KB.adl
/opt/rtcds/userapps/release/cds/common/scripts/generate_KisselButton.py 5 6 C1:SUS-PR3_TO_COIL --fi > /opt/rtcds/caltech/c1/medm/c1su2/C1SUS_PR3_TO_COIL_KB.adl
/opt/rtcds/userapps/release/cds/common/scripts/generate_KisselButton.py 4 5 C1:SUS-SR2_INMATRIX > /opt/rtcds/caltech/c1/medm/c1su2/C1SUS_SR2_INMATRIX_KB.adl
/opt/rtcds/userapps/release/cds/common/scripts/generate_KisselButton.py 2 4 C1:SUS-SR2_LOCKIN_INMTRX > /opt/rtcds/caltech/c1/medm/c1su2/C1SUS_SR2_LOCKIN_INMTRX_KB.adl
/opt/rtcds/userapps/release/cds/common/scripts/generate_KisselButton.py 5 6 C1:SUS-SR2_TO_COIL --fi > /opt/rtcds/caltech/c1/medm/c1su2/C1SUS_SR2_TO_COIL_KB.adl
Then on rossa, run activateSUS2DQ.py which creates a file C1SU2.ini.NEW. Remove old backup file C1SU2.ini.bak, rename C1SU2.ini to C1SU2.ini.bak and rename C1SU2.ini.NEW to C1SU2.ini:
Then ssh back to c1sus2 and restart the rtcds model:
Then restart daqd services from rossa and burtrestore to latest snap of c1su2epics.snap:
All suspensions are back online and everything is same as before now. Will test later the Run/Acquire switch functionality.
[Anchal, Yehonathan] (on Y Arm)
We first checked if the PZT mirrors M1 and M2 can be controlled. They indeed show no motion even after being connected with a power supply. So green injection path can not be aligned using cds controls right now.
We also noticed that all ETMY slow controls and monitors are offline. That's because the electronics upgrade did not include acromag chassis. This means that the DC bias adjustment is not accessible for ETMY.
[Paco, Ian] (on X Arm)
We installed c1auxey1 computer and the acromag chassis in 1Y4. The computer has been configured properly for nfs mounts to happen and we have initialized a git repo for /cvs/cds/caltech/target/c1auxey1 directory which stores all files for running modbusIOC service on this computer. We connected 18V power source but have not connected the 24V power yet as we need to make a new connector for it. Going on what Koji recommended, we'll connect the 24V power input to 18 V strip as well as the acromags can run on that voltage too.
I started the modbusIOC service on c1auxey1 and added PD variance channels for UR and SD as well. There are unfortunately two issues here:
I'm not sure why but the PIT and YAW offset values of +2725 and -2341 were not sufficient for the reflected OPLEV beam to reach the OPLEV QPD. I had to change the C1:SUS-ETMY_PIT_OFFSET to 5641 and C1:SUS-ETMY_YAW_OFFSET to -4820 to come back to center of the OPLEV QPD. We aligned the Oplevs to center before venting, so hopefully this is our desired ETMY position.
On another note, the issue of ETMY unable to damp was simple. The alignment offsets were on to begin with with values above 1000. This meant that whenever we enabled coil output, ETMY would necessarily get a kick. All I had to do was keep alignment offsets off before starting the damping and slowly increase the alignment offsets to desired position.
- This modification allowed me to align the oplev spot to the center of the QPD. C1:SUS-ETMY_PIT_OFFSET and C1:SUS-ETMY_YAW_OFFSE are +2725 (8%FS) and -2341 (7%FS), respectively.
C1:SUS-ETMY_YAW_OFFSE are +
We did the following alignment steps on Yarm today:
After this, we came back to the control room. One peculiar thing to note is that the C1:LS-Y_REFL_DC_OUT channel is inverted i.e. it is showing negative values for the reflection DC voltage. On this signal, we see a little bit higher-order mode flashing but it is not bright enough to be seen on the face camera of the suspension. We'll continue aligning the cavity using CDS now to get TEM00 mode.
After trying for a bit, we were able to get flash of about 2000 counts which is about 16% of the max value of 12200 counts. We adjusted the ITMY angular position using the IFO_ALIGN screen but used the OPTICAL_ALIGN offset screen to adjust the angular position of ETMY. The pitch and yaw values for ITMY are 1.0360 and -0.260 respectively whereas the pitch and yaw values for ETMY are 5664.0 and -4937.0 respectively.
Another issue, Green Y Shutter can not be controlled with the EPICS controls right now. This needs to be investigated.
Uneloged: yesterday, Paco and Ian tried locking the green laser on Yend to the Y-arm. Nothing noteworthy happened, no luck.
After Ian updated the cts2um filters for OSEM, shouldn't the damping gains be increased back by factor of 10 to previos values? Was the damping gain for SIDE ever changed? we found it at 250.
Can you explain why gain_offset filter was required and why this wasn't done for the side coil?
I updated the gain of the ct2um filter on the OSEMS for ETMY and decreased their gain by a factor of 9 from 0.36 to 0.04.
I added a filter called "gain_offset" to all the coils except for side and added a gain of 0.48.
together these should negate the added gain from the electronics replacement of the ETMY
[Anchal, Paco, Shruti]
[Anchal, JC, Ian, Paco]
We have now fixed all issues with the PD mons of c1susaux2 chassis. The slow channels are now reading same values as the fast channels and there is no arbitrary offset. The binary channels are all working now except for LO2 UL which keeps showing ENABLE OFF. This was an issue earlier on LO1 UR and it magically disappeared and now is on LO2. I think the optical isolators aren't very robust. But anyways, now our watchdog system is fully functional for all BHD suspended optics.
[Paco, Anchal, Ian, JC]
We attempted the alignment of IR beam into the arm cavities. We used PR2 and PR3 (moved manually as well as using cdsutils) and got the YAW aligned pretty good on both X and Y directions. PIT alignment however turned out to be much harder to align. PR2 PR3 didn't have much range, so we zeroed there offset and tried to use TT1, TT2, MMT1, and MMT2 to align the PIT but it would get clipped before reaching BS table if we were to correct for PIT misalignment happening downstream. We concluded that the issue is that one of the PR2, PR3 mirrors have too much PIT offset in equilibrium position. We have requested Koji to change the output resistors in the coil drivers of PR2 and PR3 so that we can correct for the PIT offset in them directly using the coils and reduce load on upstream optics. We have tweaked TT1, TT2, MMT1, and MMT2 positions today, so we do not have the previous reference anymore.
After Koji reduced the output resistors on PR2/PR3 coil drivers, we got much better actuation range. We aligned TT1 TT2 again to get beam centered on PR2 and PR3. Then we used only PR2 and PR3 to do the input beam alignment to Y arm cavity. Using access in ETMY chamber, we aligned the input beam parallel to cavity axis. Slight changes were required in ETMY alignment offsets to get first roundtrip in same spot on ITMY. remaining alignment is finer and needs to be done with a help of a reflection photodiode and cameras in the control room. Immediate next step is to setup POY path for locking the Yarm with IR.
Side note: Because of the large PIT correction required in PR3, we found that our upper OSEMs were hitting totally bright limit and lower OSEMs were hitting totally dark limit on PR3. This also destablized our damping loops. We pushed the upper OSEMs slighlty and pulled back the lower OSEMs slightly to get the PD signal in half shadow region again. This worked and our damping loops are stable again. However, we think we should repeat free swing test in future to diagonalize the input matrix for new OSEM positions.
[Anchal, Paco, JC, Tega]
Today we have aligned the Yarm cavity for IR, with verified 1.5 roundtrips. We also placed following optics in the BS table.
We also cleared the in-air BS table of all previous optics. JC and Tega setup HeNe laser for Oplevs roughly for now. Tega also transported the POY RFPD from ITMY in-air table to the BS in-air table. We aligned the POY path to the table, but we had to move ITMY after that to get 2nd roundtrip in the arm cavity, which misaligned our POY path again. POY path would need to be modified tomorrow.
Nice. Please put this information on the DCC pages of the coil driver units. You'll find links to all the units in this document tree LIGO-E2100447. For each page, click on "Change Metadata" from the left panel and add the change made to the resistor (including the resistor name on PCB, previous and new value), and add a link to your previous elog post which has more details like photos, to "Notes and Changes", and upload an updated version of the circuit schematic by creating an annotation in the previous circuit schematic pdf. Every unit that has a serial number in the lab has a DCC page (if not, we should create one) where we should track all such hard changes.
If someone gets time, let's put in all the cable posts and clean up our cable routing on the tables.
[Anchal, Tega, JC]
We installed cable posts in ITMY, BS, and ITMX chambers for all the new suspensions. Now, there is no point where the OSEM connections are hanging freely.
In BS chamber, we installed one post for LO2 near the north edge of the table and another post for PR3 on the Western edge with the blue cable running around the table on the floor.
In ITMY chamber, we installed the cable post in between AS1 and AS4 with the blue cables running around the table on the floor. This is to ensure the useful part of the table remains empty for future and none of the OSEM cables are taught in air.
I investigated this issue today. At first, it seemed that only new suspension testpoints are inaccessible. I was able to use diaggui for a measurement on MC2. The DAQ network cable between 1X4 and 1Y1 was tied and is very taught now (we should relieve this as soon as possible, best solution is to lay down a longer cable over the bridge). My hypothesis is that the DAQ network might have broken while tying this cable and it probably did not come back since then.
The simplest solution would have been to restart c1su2 models. As I restarted those models though, I found that c1lsc and c1sus models failed. This is very unusual as c1su2 models are independent and share nothing with the other vertex models. So I had to restart all the FE computers eventually. But this did not solve the issue. Worse, now the DAQ isn't working for the vertex machiens as well.
Next step was to try restarting fb1 itself. We switched off all the FE computers, restarted fb1, stopped daqd_* processes, reloaded gpstime module, restarted open-mx, mx, nds and daqd_* process. But the mx.service failed to load with following error message:
● mx.service - LSB: starts MX driver for Myrinet card
Loaded: loaded (/etc/init.d/mx)
Active: failed (Result: exit-code) since Mon 2022-04-25 17:18:02 PDT; 1s ago
Process: 4261 ExecStart=/etc/init.d/mx start (code=exited, status=1/FAILURE)
Apr 25 17:18:02 fb1 mx: Loading mx driver
Apr 25 17:18:02 fb1 mx: insmod: ERROR: could not insert module /opt/mx/sbin/mx_mcp.ko: File exists
Apr 25 17:18:02 fb1 mx: insmod: ERROR: could not insert module /opt/mx/sbin/mx_driver.ko: File exists
Apr 25 17:18:02 fb1 systemd: mx.service: control process exited, code=exited status=1
Apr 25 17:18:02 fb1 systemd: Failed to start LSB: starts MX driver for Myrinet card.
Apr 25 17:18:02 fb1 systemd: Unit mx.service entered failed state.
(Ignore the timestamp above, I ran the command again to capture the error message.)
However, I was able to start all the FE models without any errors and daqd processes are also all running without showing any errors. Everything is green in CDS screen with no error messages. But the only thing still wrong is mx.service which is not running.
From my limited knowledge and experience, mx.service is a one-time script that mounts mx devices in /dev and loads the mx driver. I tried running the script /opt/mx/sbin/mx_start_stop :
This gave the same error. On searching little bit online, "insmod: ERROR; cound not insert module" error comes up when the kernel version of the driver doesnot match the Linux kernel (whatever that means!). Such deep issues should not appear out of nowhere in a previosuly perfectly runnig system. I'll check around more what changed in fb1, network cables etc.
[Anchal, Paco, JC]
We had to open ITMY, ETMY chamber doors to get the cavity aligned again. Once we did that, we regained cavity flashing and were able to align the input injection and cavity alignment to get transmission flashing to 1.0 (C1:LSC-TRY_OUT_DQ). JC later centered both ITMY and ETMY oplevs. The ITMY oplev had become completely out of range.
We also opened ITMX, ETMX chamber doors to get Xarm alignment. Again, it seems that ITMX had moved a lot due to cable post installation.
To be continued
This would be a daily first task in the morning. We'll need to check the status of arm alignment and optimize it back to maximum every morning for the rest of the day's work.
Today, when I came, on openin gthe PSL shutter, IMC was aligned good, both arms were flashing but YARM maximum transmission count was around 0.7 (as opposed to 1 from yesterday) and XARM maximum transmission count was 0.5 (as opposed to 1 from yesterday). I did not change the input alignment to the interferometer. I only used ITMY-ETMY to regain flashing count of 1 on YARM and used BS and tehn ITMX-ETMX to regain flashing count of 0.9 to 1 in XARM.
Even thought the oplevs were centered yesterday, I found the oplev had drifted from the center and the optimal position also is different for all ooptics except EMTY and BS. It is worth nothign that in optimal position both PIT and YAW of ITMY and ITMX are off by 70-90 uradians and ETMX Pit oplev is off by 55 uradians.
I tried out this stack today and found some change of plans.
tl,dr; Jordan is preparing PLS-T238 and TR-1.5 with venting holes and C&B and they would be ready by tomorrow. I have collected all other parts for assembly, still looking for the mirror but I know other lab members know where it is, so no big issue there.
The assemly of this mirror is complete. A slight change here as well, we were supposed to use the former POYM1 (Y1-2037-0) mirror for POP_SM5 but I could not find it. It was stored on the right most edge of the table (see 40m/16450), but it is not there anymore. I found another undocumented mirror on the flow bench on the left edge marked (2010 July: Y1-LW1-2037-UV-0-AR) which means this mirror has a wedge of 1 degree and an AR coating as well. We do not need or care about the wedge or AR coating, so we can use this mirror for POP_SM5. Please let me know if someone was saving this mirror for some other purpose.
I'll finish assembly of POP_SM4 tomorrow and install them in ITMX chamber and resurrect POP path.
Here is more detail of the POP_SM4 mount assembly.
It's a combination of BA2V + PLS-T238 + BA1V + TR-1.5 + LMR1V + Mirror: CM254-750-E03
Between BA1V and PLS-T238, we have to do a washer action to fix the post (8-32) with a 1/4-20 slot. Maybe we can use a 1" post shim from thorlabs/newport.
Otherwise, we should be able to fasten the other joints with silver-plated screws we already have/ordered.
I think TR-1.5 (and a shim) has not been given to Jordan for C&B. I'll take a look at these.
We investigated the low power issue with POX11 photodiode.
Restored arm algiment to get 0.8 max flashing on YARM and 1 max flashing on XARM. I had to move input alignment using TT2-PR3 pair and realign YARM cavity axis using ITMY-ETMY pair.
I would like to advertise this useful tool that I've been using for moving cavity axis or input beam direction. It's a simple code that makes your terminal kind of videogame area. It moves two optics together (either in same direction or opposite direction) by arrow key strokes (left, right for YAW, up, down for PIT). Since it moves two optics together, you actually control the cavity axis or input beam angle or position depening on the mode.
We found that one of the Y1-1037-45P marked mirror that we used was actually curved. So we removed it and used a different Y1-1037-45P mirror, adjusted the position of the lens and got the beam to land on POX11 RFPD successfully.
Then in control room, we maximized the POX11_I_ERR PDH signal amplitude by changing C1:LSC-POX11_PHASE_R to 42.95 from -67.7. We kept the C1:LSC-POX11_PHASE_D same at 90. We were getting +/- 200 PDH signal on POX_I_ERR.
Then in our attempt to lock the XARM, when we ran the "Restore XARM (POX)" script, YARM locked!
We are not sure why the YARM locked, we might have gotten lucky today. So we ran ASS on YARM and got the transmission (TRY_OUT) stable at 1. The lock is very robust and retrievable.
Coming back to XARM, we realized that the transmission photodiode used for XARM was the low-gain QPD instead of the thorlabs high gain photodiode. The high-gain photodiode was outputing large negative counts for some reason. We went to the Xend to investigate and found that the high gain photodiode was disconnected for some reason. Does anyone know/remember why we disconnected this photodiode?
We connected the photodiode back and it seems to work normally. We changed the photodiode selection back to high gain photodiode for TRX and on 40 dB attenuation, we see flashing between 1.4 to 1.6. However, we were unable to lock the XARM. We tried changing the gain of the loop, played a little bit with the trigger levels etc but couldn't get it to lock. Next shift team, please try to lock XARM.
[Paco, Anchal, Yuta]
We opened the BSC and ITMX chamber in the morning (Friday) to investigate POX11 beam clipping. We immediately found that the POX11 beam was clipping by the recently installed cable posts, so luckily no major realingment had to be done after reinstalling the cable post in a better location.
Because we had the BSC open, we decided to steer the AS1 mirror to align the AS path from ITMY all the way to the vertex chamber. Relatively small AS1 offsets (of ~ 2000 counts each) were added on PIT / YAW to center the beam on ASL (there is slight clipping along PIT, potentially because of the AS2 aperture. We then opened the vertex chamber and located the AS beam with relative ease. We decided to work on this chamber, since major changes propagate heavily downstream (simply changing the IMC pointing).
Anchal removed old optics from the vertex chamber and we installed the steering pair of mirrors for AS path. This changed the balance of the vertex table by a lot. By using the MC REFL camera beam spot we managed to coarsely balance the counterweights and recover the nominal IMC injection pointing. Simply reenabling the IMC autolocker gave us high transmission (~ 970 counts out of the typical 1200 these days).
The final IMC alignment was done by Anchal with delicate PIT motion on the input injection IMC miror to maximize the transmission (to our satisfaction, Anchal's motion was fine enough to keep the IMC locked). The end result was quite satisfying, as we recovered ~ 1200 counts of MC transmission.
Finally, we looked at the arm cavity transmission to see if we were lucky enough to see flashing. After not seeing it, we adjusted TT1 / TT2 to correct for any MMTT1 pitch adjustment needed after the vertex table rebalancing. Suprisingly, we didn't take too long and recovered the nominal arm cavity pointing after a little adjustment. We stopped here, but now the vertex table layout is final, and AS beam still needs to be aligned to the vertex in-air table.
I installed POP_SM4 and POP_SM5 in the ITMX chamber in the nominal positions. This must have affected the ITMX Oplev because I could see that one of the ITMX oplev beam was going through POP_SM5. It needs to be changed in order to follow the original plan. However, since POP_SM5 is a 1064 line mirror, it is transparent to the opleve beam, so maybe we can just use the ITMX oplev in the current fashion.
[Paco, JC, Anchal]
We balanced the IMC table back again to point that got us 50% of nominal transmission from IMC. Then we tweaked the steering mirror for injection to IMC to get up to 90% of nominal transmission. Finally, we used WFS servo loop to get to the 100% nominal transmission from IMC. However, we found that the WFS loop has been compromised now. It eventually misaligns IMC if left running for a few minutes. This needs to be investigated and fixed.
We first aligned the single arm cavity resonance for both arms to get maximum flashing. As we opened the chamber, I found that the POP beam was mostly hitting the POP_SM4 mirror but was clipping about 2 mm on the top edge.
I used TT2-PR3 to lower the injection beam angle and moved pairs of ITMY-ETMY, and ITMX-ETMX to recover as much flashing as I could in the both arms. Then, I moved PR2 in pitch from 49 to 71 to maximize the arm flashing again. After these steps, the POP beam was clearly within the POP_SM4 mirror but still in the upper half of the optic and there was maybe just a mm of clearance from the top edge. I decided to raise POP_SM4 mirror by 0.14" spacer. Now the beam is still in upper half of the mirror but has a good clearance from the edge.
The POP beam is coming outside in the in-air table at as a rising beam in the nominal path near the center of the window. This beam needs to be directed to the POP camera and RFPD on the far-side of the table.
We investigated why WFS loop wasn't working. It seemed like WFS1 PIT error signal has a huge offset which would push the loop to misalign all optics' PIT. So we did the following steps:
As I went to correct the ITMX Oplev mirrors, I found that both mirrors were placed in very different positions than the design position. Part of the reason I think was to preserve outside oplev path, and party because a counterweight was in ITMXOL1 position. I had to do following steps to correct this:
During the above work, i must have kicked the cable between the vacuum flange and the satellite amplifier box for ITMX. This disconnected all the OSEMs and Coils. We tried several things to debug this and finally found that nudging the connections on Sat Amp box brought the OSEMs and coils back online. Note that the connector was not partially out or in a state that obviously showed disconnection of the pins. I'm glad we are putting in new electronics soon for the vertex optics as well.
I successfully steered out the two output beams from BHD BS to ITMY table today. This required significant changes on the table, but I was able to bring back the table to balance coarsely and then recover YARM flashing with fine tuning of ITMY.
Thanks Chris for the fix. We are able to access the testpoints now but we started facing another issue this morning, not sure how it is related to what you did.
The steps we have tried to fix this are:
These above steps did not fix the issue. Since we have the testpoints (C1:SUS-ETMX_TRX_OUT & C1:SUS-ETMY_TRY_OUT) for now to monitor the transmission levels, we are going ahead with our upgrade work without resovling this issue. Please let us know if you have any insights.
[Anchal, Paco, Yuta]
I centered WFS1 PD so that IMC WFS Servo does not go out of range.
[Anchal, Paco, Yuta, JC]
I modified the script freeSwing.py to use damping loop output switches to free the optic instead of watchdog or coil output filters. This ensures that the free swing test is being done at the nominal position of the optic. I started tests for LO1, LO2, As2, As4, PR2, PR3, and SR2 in a tmux session names freeSwing on rossa.
Note: LO2 face OSEMs are hardly sensitive to any motion right now due to excessive pitch offset required for LO beam. We should relieve this offset to LO1 and rerun this test later.
tl;dr: Done no harm, no lasting change.
- Use /cvs/cds/caltech/target/c1psl/autoBurt.req as input to test snapshot "/users/anchal/BURTsnaps/controls_1210301_101310_0.snap" on rossa after not succeeding in donatella
- Browse /opt/rtcds/caltech/c1/burt/autoburt/snapshots/TODAY just to know where the snapshots are living. Will store our morning work specific snapshots in local user directories (e.g. /users/anchal/BURTsnaps)
- Switched channels around on video controls; changed C1:VID-MON7 to 16, back to 30, then C1:VID-QUAD2_4 to 16, to 18, then 20, back to 16, to 14 (which identified as PMCT), to 1 (IMC). Anyways, looks like IMC is locked.
[Yehonathan, Paco, Anchal]
- From IOO/LockMC, MC_Servo, FSS --> closed PSL shutter, reopen it and see the lock recovers almost instantly. Try MCRFL shutter, no effect. Toggled PSL shutter one more time, lock recovered.
- From IOO/LockMC, MC_Servo, toggle OPTION (after IP2A), lose and recover lock in similar fashion. MCRFL gets most of the light.
- Looked at IFO_OVERVIEW just to get familiar with the various signals.
A cross-coupling test has been set to trigger at 05:00 am on April 1st, 2021. The script is waiting on tmux session 'cB' on pianosa. /scripts/SUS/OutMatCalc/MC2crossCoupleTest.py is being used here. The script will switch on oscillator in LOCKIN1 of MC2 at 13 Hz and 200 counts and would send it along the POS, PIT and YAW vectors on output matrix one by one, each for 2 minutes. It will take data from C1:IOO-MC_F_DQ, C1:IOO-MC_TRANS_PIT_ERR and C1:IOO-MC_TRANS_YAW_ERR and use it to measure 'sensing matrix' S. Sensing matrix S is defined as the cross-coupling between excited and sensed DOF and we ideally want it to be an identity matrix. The code will use the measured S to create a guess matrix A which on being multiplied by ideal coil output matrix would give us a rotated coil output matrix O. This guess O will be applied and the measurement will be repeated. On each iteration, next, A matrix is defined by:
This recursive algorithm converges A to the inverse of initial S. The above relation is derived by noticing that in steady state . I've taken this idea from a mathematics paper I found on some more complex stuff (c.f. https://doi.org/10.31219/osf.io/yrvck).
At each iteration, all three matrices A, O and S will be stored in a text file for analysis later.
The code has the error-catching capability and would restore the optic to the status quo if an error occurs or watchdogs trip due to earthquakes.
The coil balancing attempt failed. The off-diagonal values in the measured sensing matrices either remained the same or increased.
The attempt in the morning was too slow. By the time we reached, it had reached to iteration 7 only and still nowhere near optimum sensing matrix had reached. We still needed to see if the optimum would eventually reach if more iterations happened.
<Radhika came for shadowing us and learning about 40m>
So we worked a bit on speeding up the data loading process and then ran the code again which now was running much faster. Still within 1 hr or so, we saw it had reached to iteration 7 with no sign of sensing matrix getting any better.
<Paco left for vaccination>
To determine if the method would work in principle, I decided to stop the current run and start with a 0.5 Hz bandwidth run (so about 7 averages with 8s duration data and welch method). This completed 20 iterations before Gautum came. But it was clear now that the method is not converging to a better solution. Need to find a bug in the implementation of the algorithm mentioned in last post or find a better algoritm.
Attachment 1 is the plot of how the sensing matrix's distance from the identity matrix increased over iterations in the last run.
Attachment 2 is the plot for different off-diagonal terms in the sensing matrix. It is clear that POS->PIT,YAW coupling is not being measured properly as it remains constant.
Attachment 3 Gautum told us that there is some naming error in nds and MC_TRANS_PIT/YAW can be read through C1:IOO-MC_TRANS_PIT_ERR and C1:IOO-MC_TRANS_YAW_ERR channels instead. To test if they indeed point to same values, we did a test of exciting YAW degree through LOCKIN1 and seeing if the peaks are visible in the channels. This was also done to give Radhika an opportunity to do something I could confidently mentor about. and to experience using diaggui.
After fixing a few things we felt were wrong in our implementation of the algorithm, we ran the coil balancing for 12 iterations with just 11s per excitation and still taking CSD with 0.1 Hz bandwidth. This time we saw the distance of sensing matrix from identity going down.
Came in a little bit after 8 and found the MC unlocked and struggling to lock for the past 3 hours. Looking at the SUS overview, both MC1 and ITMX Watchdogs had tripped so we damped the suspensions and brought them back to a good state. The autolocker was still not able to catch lock, so we cleared the WFS filter history to remove large angular offsets in MC1 and after this the MC caught its lock again.
Looks like two EQs came in at around 4:45 AM (Pacific) suggested by a couple of spikes in the seismic rainbow, and this.
As mentioned in last post, we earlier made an error in making sure that all time series arrays go in with same sampling rate in CSD calculation. When we fixed that, our recursive method just blew out in all the efforts since then.
We suspect a major issue is how our measured sensing matrix (the cross-coupling matrix between different degrees of freedom on excitation) has significant imaginary parts in it. We discard the imaginary vaues and only use real parts for iterative method, but we think this is not the solution.
Here we present cross-spectral density of different channels representing the three sensed DOFs (normalized by ASD of no excitation data for each involved component) and the sensing matrix (TF estimate) calculated by normalizing the first cross spectral density plots column wise by the diagonal values. These are measured with existing ideal output matrix but with the new input matrix. This is to get an idea of how these elements look when we use them.
Note, that we used only 10 seconds of data in this run and used binwidth of 0.25Hz. When we used binwidth of 0.1 Hz, we found that the peaks were broad and highest at 13.1 Hz instead of 13 Hz which is the excitation frequency used in these measurements.
Today, we finally crossed the last hurdle and got a successful converging coil balancing run.
We ran again this method but with the 'b' parameter as a matrix instead. This provides more gain on some off-diagonal terms than others. This gave us a better convergence with the code reaching to the tolerance level provided (0.01 distance of S matrix from identity) within 16 iterations (~17 mins).
Attachment 1 again shows how the off-diagonal terms go down and how the overall distance of sensing matrix from identity goes down. This is 'Cross coupling budget' of the coils as iterations move forward.
PMC lost lock between 21:00 and 22:00 UTC on April 11th as seen in the summary pages:
That's between 2pm and 3pm on Sunday for us. We're not sure what caused it. We will attempt to lock it back.
Mon Apr 12 08:45:53 2021: we used milind's python script in scripts/PSL/PMC/pmc_autolocker.py. It locked the PMC in about a minute and then IMC catched lock succefully.
However, the PMC transmission PD shows voltage level of about 0.7V. On medm, it is set to turn red below 0.7 and yellow above. In Summary pages in the past, it seems like this value has typically been around 0.74V. Simil;arly, the reflection RFPD DC voltage is around 0.063 V right now while it is supposed to be around 0.04 nominally So the lock is not so healthy.
We tried running this script and the bashscript version too (scripts/PSL/PMC/PMCAutolocker) a couple of times but it was unable to acquire lock.
Then we manually tried to acquire lock by varying the C1:PSL-PMC_RAMP (with gain set to -10 dB) and resetting PZT position by toggling Blank. After a few attempts, we were able to find the lock with transmission PD value around 0.73V and reflection RFPD value around 0.043. PZT control voltage was 30V and shown in red in medm to begin with. So we adjusted the output ramp again to let it come to above 50V (or maybe it just drifted to that value by itself as we could se some slow drift too). At Mon Apr 12 09:50:12 2021 , the PZT voltage was around 58V and shown in green.
We assume this is a good enough point for PMC lock and move on.
We tried two sets of filters on the output matrix POS column in MC2. Both versions failed. Following are some details.
The filters were somewhat successful, how much we can see in attachment 1. The tip about difference between eigenmode basis and cartesian basis was the main thing that helped us take data properly. We still used OSEM data but rotated the output from POS, PIT, YAW to x, theta, phi (cartesian basis where x is also measured as angle projected by suspension length).