I pulled out the YEND acromag chassis to check how a connection is made. I unplugged it and then opened it without removing any front panel connections, and then I proceeded to wire the XEND acromag chassis at the electronics bench. After the work was done, I pushed it back but I couldn't restart modbusIOC, even after running daemon-reload on c1auxey1. The ETMY watchdogs are down for now, as are the vertex and XEND ones. I tried restarting c1auxey1 but this didn't help, so maybe something was wrong when mounting the NFS? I will continue this restoration tomorrow and then replicate this at the XEND.

UPDATE Tue Oct 3 10:13:17 2023

I fixed this issue by ensuring the nfs mount was working. To do this, from c1auxey1 I ran:

and then I restarted the modbusIOC.service. Finally, I damped ETMY.

UPDATE Tue Oct 3 16:17:54 2023

[JC, Paco]

The ETMX acromag chassis wiring is complete (Attachment #1) including the optoisolator bit. JC is completing the front panel arrangement and labels and we should be good to go. I moved a binary output chassis from near the vertex rack to the XEND rack and plan to install it along with the acromag chassis to mirror the YEND rack. Finally, the next step is to correctly deploy the modbusIOC service and paying attention to recent changes, for example those in (40m/17702). Since the wiring is similar to YEND, maybe we can simply copy the db file contents.

[Rana, Vittoria]

At 3 pm, we did some PMC alignment, and the transmission value went from 665 to 680.

Also, we aligned the reflected beam.

Now the unlocked value the reflected photodiode sees is ~440 mV, and when it's locked, it sees ~21 mV. Before doing the alignment, the unlocked value was ~400 mV.

[Rana, Radhika, Murtaza]

WFS Loop Debugging

- We turned on the WFS loops with very small gain (0.01) to see how the error signals behave. There is an existing template to look at the error signals in ndscope (users->Templates->ndscope->IOO->WFS->WFS-overview.yml). We observed C1:IOO-WFS2_IY_DQ stay at a constant offset as we increased the gain to 1.

- The output matrix for WFS (C1IOO_WFS_INMATRIX) was restored to the original value using burtgooey to mitigate the WFSoutMatBalancing.py change 17874.

- (TODO) WFS1 and WFS2 are slightly misaligned as seen on the C1IOO_LOCKMC screen. These need to be aligned when the IFO is unlocked so that the beam is centered on them.

- (TODO) With the PSL shutter turned off, WFS heads should show 0 reading which is not the case. This needs to be corrected for to mitigate the offset readings.

- The electronics upgrade should ideally only affect the suspensions (everything upstream should not need any changes).

- Note: The MC_TRANS error signals look very small in PIT and YAW.

While aligning today I realized the cavAlign step sizes and step factors had not been updated after the upgrade.

Here are the new MEDM command arguments to launch cavAlign. Only factors not equal to 1 are listed. The updates made by me are in red

Interesting notes:

To further homogenize the suspensions, we did the following changes.

 - Turned on DECIMATION in PR2 URCOIL
 - Changed +/- 13 in SDCOIL_GAIN of SR2,LO1,LO2,AS1,AS4 to +/-1 and increased SUSSIDE_GAIN accordingly
 - PRM SDSEN_GAIN was changed from +0.2 to +1 (see 40m/17877)
 - Moved FM6 "gain_offset" of gain(0.48) to FM1 in ETMY *COIL to align with other suspensions. Also added x0.48 to SDCOIL as well, and adjusted SUSSIDE_GAIN accordingly.
 - "Half shorted" binary inputs to coil drivers for PR2,PR3,SR2,LO1,LO2,AS1,AS4 so that they are always in "Acq" mode. FM9 SimDW filters were turned on accordingly.

Before work today:

2023-09-28 17:10:19 UTC (GPS: 1379956237)
channel\optic   MC1     MC2     MC3     BS      ITMX    ITMY    PRM     SRM     ETMX    ETMY    PR2     PR3     SR2     LO1     LO2     AS1     AS4     
ULSEN_GAIN        +1.00   +1.00   +1.00   +1.00   +1.00   +1.00   +1.00   +1.00   +1.00   +1.00   +1.00   +1.00   +1.00   +1.00   +1.00   +1.00   +1.00 
LLSEN_GAIN        +1.00   +1.00   +1.00   +1.00   +1.00   +1.00   +1.00   +1.00   +1.00   +1.00   +1.00   +1.00   +1.00   +1.00   +1.00   +1.00   +1.00 
URSEN_GAIN        +1.00   +1.00   +1.00   +1.00   +1.00   +1.00   +1.00   +1.00   +1.00   +1.00   +1.00   +1.00   +1.00   +1.00   +1.00   +1.00   +1.00 
LRSEN_GAIN        +1.00   +1.00   +1.00   +1.00   +1.00   +1.00   +1.00   +1.00   +1.00   +1.00   +1.00   +1.00   +1.00   +1.00   +1.00   +1.00   +1.00 
SDSEN_GAIN        +1.00   +1.00   +1.00   +1.00   +1.00   +1.00   +0.20   +1.00   +1.00   +1.00   +1.00   +1.00   +1.00   +1.00   +1.00   +1.00   +1.00 
ULSEN_SWSTAT      37923   37923   37923   37923   37923   37923   37923   37923   37923   37923   37923   37923   37923   37923   37923   37923   37923 
LLSEN_SWSTAT      37923   37923   37923   37923   37923   37923   37923   37923   37923   37923   37923   37923   37923   37923   37923   37923   37923 
URSEN_SWSTAT      37923   37923   37923   37923   37923   37923   37923   37923   37923   37923   37923   37923   37923   37923   37923   37923   37923 
LRSEN_SWSTAT      37923   37923   37923   37923   37923   37923   37923   37923   37923   37923   37923   37923   37923   37923   37923   37923   37923 
SDSEN_SWSTAT      37923   37923   37923   37923   37923   37923   37923   37923   37923   37923   37923   37923   37923   37923   37923   37923   37923 
SUSPOS_GAIN     +120.00 +150.00 +100.00 +100.00 +150.00  +50.00  +28.00  +25.00 +150.00  +41.00   +8.00  +10.00  +27.00  +10.00  +10.00  +14.00  +15.00 
SUSPIT_GAIN      +60.00  +10.00  +24.00  +10.00  +14.00   +7.00   +5.00   +1.20  +15.00   +6.00   +2.00   +5.00   +6.00   +4.00   +3.00   +2.50   +3.10 
SUSYAW_GAIN      +60.00  +10.00   +8.00   +3.00  +10.00   +8.00   +4.00   +1.50  +10.00   +6.00   +2.00   +5.00   +6.00   +3.00   +3.00   +3.00   +3.00 
SUSSIDE_GAIN    +100.00 +150.00 +125.00  +10.00  +60.00  +50.00  +50.00   +7.50 +150.00 +300.00  +11.54  +20.00  +10.77   +3.08   +3.85   +6.54   +3.15 
OL_PIT_GAIN       +1.00   +1.00   +1.00   -0.05   +5.00   +3.50   +6.00  +12.68   +1.00   -1.00   +1.00   +1.00   +1.00   +1.00   +1.00   +1.00   +1.00 
OL_YAW_GAIN       +1.00   +1.00   +1.00   +0.10   +5.00   -4.00   -8.00  -15.85   +1.00   -1.00   +1.00   +1.00   +1.00   +1.00   +1.00   +1.00   +1.00 
ULCOIL_GAIN       +1.01   +1.07   +0.94   +1.06   -1.10   +1.06   +0.97   +1.09   -1.01   -1.00   -1.00   -1.00   -1.00   -0.94   -1.05   -0.94   -0.98 
LLCOIL_GAIN       -0.95   -0.98   -0.94   -0.98   +0.90   -1.01   -1.04   -1.00   +0.97   +0.81   +1.00   +1.00   +1.00   +0.98   +0.63   +0.99   +0.97 
URCOIL_GAIN       -0.98   -0.98   -1.04   -1.04   +0.93   -0.99   -1.04   -0.92   +1.03   +0.74   +1.00   +1.00   +1.00   +1.00   +1.34   +1.04   +0.98 
LRCOIL_GAIN       +1.06   +0.97   +1.08   +0.92   -1.07   +0.94   +0.90   +0.99   -0.99   -1.05   -1.00   -1.00   -1.00   -1.07   -0.98   -1.03   -1.07 
SDCOIL_GAIN       +1.00   +1.00   +1.00   +1.00   +1.00   +1.00   +1.00   +1.00   -1.00   -1.00   -1.00   -1.00  -13.00  -13.00  -13.00  +13.00  -13.00 
ULCOIL_SWSTAT     37889   38657   37889   38657   38657   38657   38657   38657   55041   38688   38400   38400   38400   38400   38400   38400   38400 
LLCOIL_SWSTAT     37889   38657   37889   38657   38657   38657   38657   38657   38657   38688   38400   38400   38400   38400   38400   38400   38400 
URCOIL_SWSTAT     37889   38657   37889   38657   38657   38657   38657   38657   38657   38688    5632   38400   38400   38400   38400   38400   38400 
LRCOIL_SWSTAT     37889   38657   37889   38657   38657   38657   38657   38657   38657   38688   38400   38400   38400   38400   38400   38400   38400 
SDCOIL_SWSTAT     37889   38657   37889   38145   38145   38145   38145   38145   38657   38656   38144   38144   38144   38144   38144   38144   38144 

After work today:

2023-09-28 18:26:55 UTC (GPS: 1379960833)
channel\optic   MC1     MC2     MC3     BS      ITMX    ITMY    PRM     SRM     ETMX    ETMY    PR2     PR3     SR2     LO1     LO2     AS1     AS4     
ULSEN_GAIN        +1.00   +1.00   +1.00   +1.00   +1.00   +1.00   +1.00   +1.00   +1.00   +1.00   +1.00   +1.00   +1.00   +1.00   +1.00   +1.00   +1.00 
LLSEN_GAIN        +1.00   +1.00   +1.00   +1.00   +1.00   +1.00   +1.00   +1.00   +1.00   +1.00   +1.00   +1.00   +1.00   +1.00   +1.00   +1.00   +1.00 
URSEN_GAIN        +1.00   +1.00   +1.00   +1.00   +1.00   +1.00   +1.00   +1.00   +1.00   +1.00   +1.00   +1.00   +1.00   +1.00   +1.00   +1.00   +1.00 
LRSEN_GAIN        +1.00   +1.00   +1.00   +1.00   +1.00   +1.00   +1.00   +1.00   +1.00   +1.00   +1.00   +1.00   +1.00   +1.00   +1.00   +1.00   +1.00 
SDSEN_GAIN        +1.00   +1.00   +1.00   +1.00   +1.00   +1.00   +1.00   +1.00   +1.00   +1.00   +1.00   +1.00   +1.00   +1.00   +1.00   +1.00   +1.00 
**SEN_SWSTAT      37923   37923   37923   37923   37923   37923   37923   37923   37923   37923   37923   37923   37923   37923   37923   37923   37923 
(...snip...) 
SUSSIDE_GAIN    +100.00 +150.00 +125.00  +10.00  +60.00  +50.00  +50.00   +7.50 +150.00 +625.00  +11.54  +20.00 +140.00  +40.00  +50.00  +85.00  +40.00 
(...snip...)
SDCOIL_GAIN       +1.00   +1.00   +1.00   +1.00   +1.00   +1.00   +1.00   +1.00   -1.00   -1.00   -1.00   -1.00   -1.00   -1.00   -1.00   +1.00   -1.00 
ULCOIL_SWSTAT     37889   38657   37889   38657   38657   38657   38657   38657   55041   38657   38656   38656   38656   38656   38656   38656   38656 
LLCOIL_SWSTAT     37889   38657   37889   38657   38657   38657   38657   38657   38657   38657   38656   38656   38656   38656   38656   38656   38656 
URCOIL_SWSTAT     37889   38657   37889   38657   38657   38657   38657   38657   38657   38657   38656   38656   38656   38656   38656   38656   38656 
LRCOIL_SWSTAT     37889   38657   37889   38657   38657   38657   38657   38657   38657   38657   38656   38656   38656   38656   38656   38656   38656 
SDCOIL_SWSTAT     37889   38657   37889   38145   38145   38145   38145   38145   38657   38657   38144   38144   38144   38144   38144   38144   38144 

ULCOIL_STAT of ETMY being 55041 is OK. 38657+2**14 = 55041.
Script to produce these tables live in /opt/rtcds/caltech/c1/Git/40m/scripts/SUS/suspension_epics_check.py

Current coil dewhitening filter situations:

Next:
 - Investigate 60 Hz noise in laser frequency and check 28 Hz ELP situation for MC1,MC2,MC3
 - Fix ETMX acromag

PRM/SRM OSEM input calibration

- I realized I had not yet updated the input OSEM cts2um calibration. For PRM and SRM, I changed each cts2um filter gain from 0.36 ---> 0.0417 (factor of 8.64).

- The coil output filters (V2A) had already been updated according to new actuation calibration.

PRM damping/alignment

- As noted in the previous ELOG, PRM SDSEN gain was 0.2 instead of 1. I turned on damping filters for PRM and noticed SD motion was underdamped. I changed PRM SDSEN GAIN from 0.2 ---> 1. Final PRM damping test in Attachment 1.

- With PRM alignment restored (no PIT offset in output matrix), the PRM oplev is aligned and is recording light.

- I left PRM with a misalignment offset (damping loops on; oplev loops off).

SRM damping/alignment

- I performed a damping test of SRM with the optic alignment restored (no PIT offset in output matrix). This way the OSEMs were not saturated. In this configuration, SRM passed a damping test [Attachment 2].

- However in this configuration, no light was hitting the oplev. I tried to manually align but gave up after I saw no improvement. Which raises the question, was the SRM oplev ever (recently) aligned?

- I left SRM with a misalignment offset, but in this state the OSEMs saturate and the damping loops drive unstable motion. So SRM damping loops are off.

[Paco, Yuta]

We checked the polarity of suspension damping loops if they follow the conventions we agreed in 40m/16898.
Suspensions are nicely homogenized nicely , with some exceptions (see Attachment #1).

• ​PRM SDSEN_GAIN is 0.2, but it should be 1.
• LO1, LO2, AS1, AS SDCOIL_GAIN is +/-13, but it should be +/- 1. (Unless there are reasons for these 13)
• Let's make coil dewhitening to be off (in Acq mode) for all by default to homogenize (40m/17875). MC1 and MC3 might require 28Hz ELP for 60Hz noise.
• INMAT should be normalized nicely so that SUSPOS/SIDE_IN will be um and SUSPIT/YAW_IN will be urad. (Are cnts2um in *SEN filters correct?)
• Gain offsets in *COIL filters (e.g. V2A, x0.414) can be adjusted later to have the same actuation efficiencies between suspensions.

Note that *COIL_GAIN are now +--++ or flipped one in the order of UL/LL/UR/LR/SD.

Next:
  - Address the points raised above
  - Make a script to show current EPICs values for all suspensions to check the damping configurations.

[Paco, Yuta]

 We checked whitening and dewhitening situations in all the suspensions, and fixed them for ETMY.

ETMY trans QPD and ETMX trans QPD whitening:
  These QPDs have analog whitening filter of two 40:4s (LIGO-D1400415 and LIGO-D1400414). So, two of 4:40 in FM1 and FM2 of C1:SUS-ETM(Y|X)_QPDx should be always on. FM2s were off, so we turned ON today (see Attachment #1).

Fixing ETMY coil dewhitening BIO switch:
  Binary switching for ETMY coil dewhitening was not working because DB37 cable from Contec 32 BO card was not connected to the Binary Output Interface Chassis (LIGO-D1002593).
  After connecting the DB37 cable with a gender changer (we need a F to F cable), some of the switching worked but not in the correct order. Using a BD37 breakout board, we noticed that the binary switching is doing the switch in the mixed order of coil dewhitening and OSEM whitening. We modified the c1scy model so that the coil dewhitening switches Run/Acq LEDs correctly (Attachment #2 was before, and modified to Attachment #3). OSEM whitening binary switches are now terminated in c1scy model, because OSEM analog whitenings are always on (LIGO-D2100144).
  We also modified c1scx model to match with c1scy, although we don't have the acromag for Xend yet.
  Attachment #4 is the BIO status when ETMX and ETMY are in run mode (coil dewhitening on). ETM(Y|X)_BO_0_0 is for coil dewhitening, and BO_0_1 is for trans QPDs.
  Attachment #5 is the photo of LEDs correctly lit when ETMY is in run mode, after all these modifications.

Summary of Coil Driver situation for all optics:
  See, also, LIGO-D1100687

  By the way, for OSEMs, analog 30:3 whitening are always ON, no matter what the BIO situations are (LIGO-D2100144). So FM1 of C1:SUS-xxxx_xxSEN should be always ON.
  Also, since the recent coil driver upgrade, the order of coil outputs in SUS_SINGLE_COIL is UL/LL/UR/LR/SD, and the signs of C1:SUS-xxxx_xxCOIL_GAIN are +--++ (or flipped one). Note that it used to be +-+-+, as the order was UL/UR/LR/LL/SD (40m/16898).

Next:
  - Check sign convensions on all the suspensions
  - Check 60 Hz noise related dewhitening situation in MC suspensions (40m/17466)
  - For LO1, LO2, AS1, AS4, PR2, PR3, SR2, make them "lower half shorted" so that analog dewhitening will be turned off similarly to other vertex suspensions.

[Paco, JC, Murtaza]

[WIP]

To fix the WFS loops, went through the following steps

With the WFS loop turned off

- We manually aligned the optics MC1, MC2 and MC3 (IOO -> C1IOO_MC_Align) to maximize transmission (MC Trans Sum -> ~13300)

- We manually aligned the QPDS for WFS1 and WFS2 to center the beam by looking at the DC signals (C1IOO_WFS_QPD). The laser was clipping on WFS2

With the WFS loop turned on

- We changed the gains of the WFS filters for all signals (1.0 -> 2.0), this led to faster conversion but clipping on C1:IOO-WFS2_YAW_OUTPUT. The gains were restored to 1.0 and thus left unchanged.

- We increased the reliefMCWFS gain by a factor of 10 by changing the arguments (Execute -> Edit this screen -> Actions -> label/cmd/args -> Arguments) (0.02 -> 0.2) 

- We ran WFSoutMatBalancing.py (17334) to calculate the new output matrix

[JC, Paco, Radhika, Murtaza]

IFO ALIGNED

1. WFS Relief
We tried to change the gain to offload the offsets in the reliefMCWFS script but it The gains might need some tuning to get it to work

2. WFS Error Signals Diverging
The error signal C1:IOO-WFS2_I_PIT_MON was staying at a constant offset from 0 using the existing output matrix (C1IOO_WFS_OUTMATRIX). We tried changing the matrix coefficients that may have caused this behavior but it led to divergence in other signals (C1:IOO-WFS1_I_PIT_MON, C1:IOO-WFS2_I_YAW_MON). THIS NEEDS TO BE FIXED

3. BS was aligned using OPLEV readouts and the damping filters were checked. No funny business for BS anymore.

4. The original IFO Align scripts used the suffix "COMM" for each optic. This was changed to "OFFSET" for all arguments by editing the IFO_ALIGN screen (left click-> Execute -> Edit this screen -> !Align -> Label/Cmd/Args)

5. The OPLEV gains for ITMY were unstable and needed some tuning. New gains: C1:SUS-ITMY_OL_PIT_GAIN (14->3.5) and C1:SUS-ITMY_OL_YAW_GAIN (-8->-4). (The upgrade should not have affected this so this could be revisited later).

6. YARM (transmission ~ 1) and XARM (transmission ~ 0.6) were locked successfully!

ITS A GOOD IDEA TO HAVE PRM AND SRM MISALIGNED WHILE TRYING TO LOCK THE IFO.

[Paco, Murtaza, JC]

Fixed C1SUS2 Crash from this morning

What we did:
  - Attempt to restart only C1SUS2
  - Restart All Machines and Burt Restored to Friday @ 7:19 pm

Summary:

Entering this morning, We were unsure why we were having issues aligning and come to find out that C1SUS2 crashed. Paco attempted to restore by restarted the machine individually and restoring, while this did turn all the machines green on the CDS.MEDM Screen, it did not resolve the issue. So moving forward, please keep in mind that EVEN IF ALL MACHINES SHOW GREEN, OPTICS MAY STILL NOT BE DAMPING. 

Next we continued to restart and reboot the old fashioned way.

I attempted to use the ./restartAllModels.sh script in the "/opt/rtcds/caltech/c1/Git/40m/scripts/cds" directory, but there was an error and the message I got said something along the lines of "Restart medm screen and try again". This was weird and all of the machines were already shutdown. So, to bring them back up, I used the ./startAllModels.sh script. When starting up, i was prompted to provide a burtrestore directory, and I inputted /opt/rtcds/caltech/c1/burt/autoburt/snapshots/2023/Sep/22/20:19.

This worked out and IMC came back to nominal alignment. The primary issue we seem to be coming across now is that C1:IOO-WFS2_PIT_OUTPUT is increasing at a steady rate and this is disrupting our alignment.

PRM CHANNEL 1-4 (BS FEEDTHROUGH 1-3)

[Koji, Radhika, Murtaza]

Connector on the BS Chamber that feeds to PRM UL/LL/UR coils (PRM 1 in Attachment 1) has pin 5 shorted to pin 1 inside the chamber 
- To resolve this, a DB25 connector was recycled from the old coil drivers
- pin 5 was isolated by cutting (green cable on the DB25 connector)
- The connector was attached between the chamber and the cable that runs through to the rack (Attachment 1)
- The connector was labelled (Attachment 2)

The PD outputs were read on the PRM SATAMP (Pins 1-4, Pin 5 (Ground))

No need to apply an external bias to Pin 5!

Can be fixed during the next vent!

Fixed IMC/IFO alignment screens

Pumping configuration changed. Now TP2 is backing TP1 and TP3 is pumping annuli.

[Koji, Radhika, Murtaza]

All upgraded suspensions have reasonable OSEM readings! Ready for damping tests and alignment next week.

We fixed PRM OSEM reading by isolating pin5 of the first DB25 [17871]. This makes the PRM UL unbiased by the PD seems to be receiving some light.

The PRM/BS/ITMX/ITMY SATAMP adapter was removed and the front-end pins were checked for shorting. Indeed, a short was found in the SIDE1-4 ribbon cable inside the sat-amp adapter, from the wires being compressed to one side of the dsub-ribbon adapter at the input joint [Attachment 1]. We reclamped the ribbon and verified there was no shorting and that the pins were properly aligned [Attachment 2]. This means PRM/BS and ITMX/ITMY SIDE signals should no longer be cross coupled.

All OSEM counts looked good after these fixes. Only a few ITMY OSEMS looked low, but Koji checked both PDMON voltages for ITMY, and we confirmed with calibration that the OSEM counts were reasonable.

Today we tried to debug the unreasonable OSEM readings (see previous ELOG)

PRM process

Starting state: PRM face + side values bogus (~0)

1. Somehow through retightening connections, PRM LR+SD counts looked reasonable (and positive). Yay!

2. Toggled on and off the PRM SATAMP; removed Ch1-4 and Ch5-8 PRM inputs

    - Result: BS SD becomes negative when PRM SATAMP is on and Ch 5-8 cable is connected.

2. We disconnected the PRM SATAMP and plugged the PRM inputs into the SRM SATAMP. The SRM SATAMP output was routed to the PRM input on the SATAMP adapter.

    - Result: PRM UL/UR/LL readings still 0.

                 BS SD still negative when SRM SATAMP is turned on and Ch 5-8 from PRM are connected.

                   ---> SRM SATAMP gives same results as PRM SATAMP; PRM SATAMP likely not faulty.

3. Replaced PRM chamber connections with satellite test box for channels 1-4.

    - Result: reasonable PRM UL/UR/LL readings ---> Pin 5 shorting on chamber side is causing issues with Ch1-4.

ITMX - ITMY process

Starting state: ITMX SD counts negative, ITMY SD counts negative (depends on ITMX connection)
To test ITMX (SD negative since change), ITMY (SD negative dependent on ITMX)

 We first switched [ITMX, ITMY]  in the following sequence to get the following results (F = average face values, S = side value)(0 = OFF, 1 = ON)

- [0,0] -> ITMX[F,S] = [300, -5000] ITMY[F,S] = [0, -800]

- [1,0] -> ITMX[F,S] = [~, ~] ITMY[F,S] = [1000, -3000]

- [0,1] -> ITMX[F,S] = [-30, -3000] ITMY[F,S] = [~, ~]

- [1,1] -> ITMX[F,S] = [15000, -10000] ITMY[F,S] = [10000, -3000]

A separate test was done to test ITMX-ITMY coupling on ITMY-side

1. ITMX SAT AMP OFF -> ITMY SIDE GOOD 

2. ITMX SAT AMP ON (Ch 1-4, 5-8 DISCONNECTED) -> ITMY SIDE =/2

3. ITMX SAT AMP ON (Ch 1-4 CONNECTED) -> NO CHANGE FROM 2.

4. ITMX SAT AMP ON (Ch 1-4, 5-8 CONNECTED) -> SAME MAGNITUDE AS 2., FLIPS SIGN

To check if ITMX was faulty from the chamber end for the SIDE DOF, the satellite test box was used for CH 5-8

ITMY SIDE SIGN STILL NEGATIVE 

For the final sanity check for the effect of ITMX on ITMY side sign, we swapped the ITMX and SRM SAT AMPS ({front -> PD OUT 1, 2}, {back -> Ch 1-4, Ch 5-8})

ITMY SIDE SIGN STILL NEGATIVE 

Summary

PRM Ch1-4 shorting issue on chamber side (UL/UR/LL)

BS/ITMY/ITMX SD <0 all seem to be caused by SATAMP adapter or downsteam in ADC2

SUSPECTED FAULTY SAT AMP ADAPTER for ITMX-ITMY SIDE COUPLNG

I recalculated the scale factors between OSEM sensor readings after/before the upgrade. The expected factor is 8.64, although we may want to rethink this if measurements are disagreeing.]

If ITMY can be restored, we can proceed to locking the YARM while PRM/ITMX/BS are worked on.
 

1. BS values seem reasonable

The SD reading is positive and nonzero, even though its still smaller than the face sensor readings by a factor of 2.
 

2. ITMX SD still negative

3. ITMY SD flipped from positive to negative ~3pm 9/21. LL is too small.

4. PRM still not reliable

5. SRM positive but scale factors widely inconsistent, order of magnitude greater than expected (~8.64).

[JC, Paco, Koji]

We had a power outage on Sept 21, 2023 at ~9AM. This is the third power outage this month as far as I remember.

- JC reported the outage was ~2sec. Some UPS supported machines were affected, while some unsupported machines also survived the incident cf c1psl (what!?)

- Some machines were rebooted by itself (cf the RTS hosts).

- megatron and optimus were powered up. Autolockers (optimus) and FSSSlow (megatron) were restored.

c1vac was still on, but the machine didn't come back online.

- The network adapter was reactivated by running the following commands

> cd /sbin
> sudo ifdown eth0
> sudo ifup eth0

However, the acromag seemed freezer, so c1vac was shutdown, the acromag chassis was power cycled, and the c1vac was rebooted. This brought c1vac fully functional again.
Rebooting made TP1 stop (gracefully)

The vacuum pressure of the main volume was high.

- We found that the vacuum pressure was up to 1e-2 torr in the afternoon when we started the recovery. In fact, the main gate valve was close at the power outage last week. See attachments.

- We made sure the valves were properly open/closed and started TP1 again. Once TP1 reached 33.6K RPM we opened the main volume to recover the vacuum pressure.

- The vacuum pressure came back to <1e5 Torr.

[Koji, Murtaza, JC]

Regarding PRM/BS:

• PRM2 cable and BS2 cable were wrongly connected. This was corrected.
   
• This makes the BS face OSEM values reasonable.
  However, the side signal is still close to zero. We confirmed that the sat amp outputs (LED mon/PD mon/PD diff out) looked reasonable for all five BS OSEMs.
  The side signal issue stays downstream of the Vertex ADC adapter.
   
• PRM2 has no issue with the sat amp.
• PRM1: We found that all the LED mon goes down to 0.17V when the vacuum flange is connected.
  It was found that the reference voltage for the LED (TP11 of D080276) went down to low number (like 0.15V) when the in-vac OSEMs were connected.
  I found that this output was not stable. So, I replaced the U4 chip (AD8672), but this didn't help the voltage sagging issue.
• Murtaza and I started checking the short circuits on the flange. We found that Pin 5 (OSEM PD1 Kathode) and Pin 1 (invac cable shield?) only have 5.1 Ohm. Pin 1 is connected to the vacuum chamber.
  • What does it mean? The PD has the reverse bias voltage of 10V applied on the PD Kathode. This bias voltage is shorted to ground via 5 Ohm. To keep the bias line at 10V, we need 2 A.
  • We don't have many options:
    - We can disconnect the internal wire for pin5 from the cable. (Prepare a ribbon cable). This should make the other OSEM PDs properly biased.
    We may be able to use an independent power supply to provide some amount of reverse bias (10V 2A is too much. Probably 1V 0.2A or 2.5V 0.5A?) so that the UL PD somewhat work.

I noticed the same issue today with C1RMS.txt, when trying to update the coil actuation gains for SRM. The filter changes were saved to chans/C1RMS.txt, so next I checked chans/tmp/. There is no chans/tmp/C1RMS.txt, or chans/tmp/C1RMS.diff. The updated filters do not load.

Update from Chris:

C1RMS.txt is a remnant from some model that doesn't exist anymore. It can be removed.

I went ahead and deleted chans/C1RMS.txt and chans/tmp/C1RMS.txt.

[Paco, Radhika]

IMC LOCKED

We used the pre-upgrade C1:SUS-MC1/2/3_SUSPIT/YAW/POS_INMON values as a baseline to restore IMC alignment.

Procedure we followed:

MC1/MC2/MC3 damping restored

I tweaked the coil actuation gains for MC1/MC2/MC3 according to Koji's updated calculations:

With foton and load coefficients working as expected, these coil output filters were successfully added to MC2 and MC3.

Damping tests

Note: While burt restoring C1MCS to a pre-upgrade state, a "NOT OK" flag popped up and the coil balancing gains for MC1/MC2/MC3 were reset to +-1. Koji showed me how to access the original values in c1mcs.snap (using grep) and I restored the coil gains to their values from 9/12/2023.

Next steps

I found the actuator calibration is more complicated. The numbers I reported in the previous elog was not correct.

Here I summarize the numbers of the voltage-to-current conversion.

=== Previous===

=== New ===

e.g. ITMX face coil electronics are x4.6 stronger than the previous coil electronics.

I asked CDS mattermost for help. Chris (Wipf) checked it and reported it is working fine as usual (without fixing anything).

I've reverted the copied C1MCS.txt back in the chans dir (/opt/rtcds/caltech/c1/chans). The filter coefficients were loaded from the GDS screen. The filters were properly updated.

Here is the info from Chris:

Some transient NFS problem, maybe?

One possible clue is that the filter file that the FE actually loads is not chans/<model>.txt,
but chans/tmp/<model>.txt. Before loading, the filter file is copied into the tmp directory,
and a diff of the two files is written to chans/tmp/<model>.diff.
This diff file should normally be an empty file, indicating that the two files match.
But it was not empty at first, so there must have been some issue with the previous load.
After I reloaded, the diff then became empty.

Plan for Sept 20, 2023

For morning people:

• We don't need to replace the long cables. They seem all fine.
• Close the lid of the repaired sat amps. Use a lid with ventilation slits (there is an extra with the empty unit on the same desk).
• Install the sat amps back to the racks. Connect all the cables us. Check if this makes the OSEM values to positive 10~20k counts.
  • If not, check LED mons and PD mons. If they are OK the sat amp is working fine. Track the signal down to the AI chassis to see if the units after the sat amp are working well.
• The SRM2 and ITM2 cables ( connected to the sat amps at the back of the units) cross (i.e. have twisted) at the rack. Please reroute and nicely coil them up.
• Fix the custom cable issues: "The custom cable fastening screw on PRM(1st) was stripped at the flange side. It needs a thread dyeing. The SRM(2st) has the hex nut broken on the sat amp. Need to be fixed."
• Put the proper labels on the long cables at the flanges and the sat amps. The labels should indicate where the connectors are supposed to be connected.
• Clean up the mess and the tools from the lab.

The

After the weekly meeting, we'll continue to work on the suspension control. The lab will be turned to be LASER HAZARD in the afternoon.

[Murtaza, Koji]

ITMX / BS / PRM sat amps were removed from the rack and checked on the workbench. They all work fine with the OSEM simulation box.

With the correct circuit, the LED mon should be 5V, and the PD readout should be 2.6~3.0V (i.e. the differential output has twice the voltage difference of this number).

ITMX Sat Amp fixed:

• The internal wiring for CH1-4 was not connected (or disconnected by mechanical impact) (Attachment 1)
• PD1 channel for CH1 had a metal debris on the transimpedance opamp (Attachment 2)
• The internal board for CH5-6 was connected in the opposite direction. This was because both connectors on the board had the wrong genders.
  This was replaced with a spare. (There was two spares and I consumed one now) (Attachment 3)
   
• Put a ventilated lid instead of the solid lid.

BS Sat Amp:

• All the CHs just worked fine.

PRM Sat Amp:

• Found the bias selector jumpers had not been installed. Fixed. (Attachment 4/5)

[Paco, Murtaza, JC, Koji, Radhika]

MC1/MC2 was working fine.

At this point MC3, SRM, and ITMY are also working fine.

The custom DB25 cables between the sat amps and the flanges are difficult to mate.

• The finger tight was not enough to make all the contacts. Fastening the screws with a screwdriver made MC3 start working fine on CDS.
• The custom cable fastening screw on PRM(1st) was stripped at the flange side. It needs a thread dyeing. The SRM(2st) has the hex nut broken on the sat amp. Need to be fixed.

We checked if the LED mon shows the correct values. When it is connected it shows 5V. If the LED is not connected it shows 0.8V. It goes 0.08V in an unknown state.

• SRM1 all channels were 0V. It turned out that the connection inside the vacuum chamber seemed mirrored. Right now we have temporary mirror ribbon cables to fix this issue. We need two shielded mirror cables for SRM1 and SRM2.
• SRM2 was random (5V, 0.8V, and 0V)
• BS1/2/ITMY1 looked fine.
• ITMY2 was strange.
• ITMX1/2 were completely silent.

We suspected the cable pinouts/ cable mating issue etc, but it turned out that the SRM2 cable was mislabeled and the ITMY2 cable was connected to the SRM sat amp. That's why it was so random. We corrected the connection and SRM and ITMY started working fine on CDS.

We used the OSEM simulation box to test the sat amps. That suggests that the BS/PRM/ITMX problem may be coming from the SAT amps. We need to look into the sat amps.

[Koji, Radhika]

Update to the Foton/Load Coefficients issue:

- "Save" in foton writes/updates filters correctly to chans/C1MCS.txt. However, "Load Coefficients" is currently not reading C1MCS.txt and therefore not loading any changes at all.

    - We saved a backup of C1MCS.txt and replaced it with one from this morning, hoping to see that the new V2A filters vanish from MC1 coil outputs. When we loaded coefficients, the V2A filters remained.

- We then checked if this issue was happening with C1SUS by adding a test filter to ETMY_ULCOIL. Indeed "Load Coefficients" loaded the filter. So it seems to just be a C1MCS issue.

- We restarted C1MCS twice and no change.

MCs OSEM input / coil output gain tuning

Seeing that MC1, MC2, MC3 OSEM readings looked reasonable and consistent, I worked on updating the input OSEM cts2um filter for the 3 suspensions. MC1 OSEM input gains were changed by a factor of 2.15; MC2 and MC3 OSEM input gains were changed by a factor of 8.64 (see previous ELOG for source of these factors).

OLD cts2um gains (units are um/ct):

NEW cts2um gains (units are um/ct):

Next, I moved onto the coil output filters for MC1, MC2, MC3. There was no gain filter already in place for these coil outputs, so I created one called V2A. (This name can be changed. Note: for other optics the coil actuation scaling filters are titled "xN" for scaling N. Eventually we will find an elegant way to set these scalings.)  The coil outputs for MC1, MC2, MC3 were changed by a factor of 1/2.6, or 0.385 (see previous ELOG for source of this factor).

I ran into an issue saving the foton filter coefficients: the filters appear to be saved; however, "Load Coefficients" does not load them onto the medm screen for MC2_URCOIL and all MC3 coils. I've tried toggling the save button and Load Coefficients button, but no luck. I checked and the filters are saved in opt/rtcds/caltech/c1/chans/C1MCS.txt. When changing an existing filter gain, the change is not being applied to the output channel.

MC1 damping

Since all MC1 coil filters saved and loaded successfully but not MC2 or MC3, I was only able to test the damping of MC1. Turning on the damping filters did not supress motion, so I made the following changes:

Attachment 2 shows the damping of MC1 after these changes were made. It looks side the SIDE mode is underdamped and we may want to increase its servo gain. (I flipped the sign of the coil output and confirmed it ringed up, so the sign is correct.)

Next steps

1. Debug why foton is not saving new coil output scaling filters for MC2 UR and all MC3 coils.

2. Assess MC2 and MC3 damping.

MC1 is ready for the damping test

Trouble shooting plan

• Is the LED on? => Check all the LED mon outputs of the sat amp. It should show 5V if the output current is 35mA. If the constant current loop is open (eg no LED / connection failure etc), it rails at the supply voltage.
• Also the CCD videos should show the status of the LEDs (at least for the TMs and the MC mirrors)
• Then is the PD out responding? => Check all the PD mons.
• If the PD mons are normal, but there is no signal it can be the AA problem. Inject test signal to that channel on the AA and see if we can see some number on the CDS.
   
• Is the coil current flowing? => Check if the coil drv mons are responding.
• If not, check if the AI output has the DAC output in that channel.
• If the DAC signal is there, but no current it can be the driver issue, or the coil/cable/flange connection issue.

- Here is the thought how does the factor of 9 come from:

• We are driving OSEM LEDs at 35mA rather than at 25mA. (Honeywell LED SME2470 has quite a linear response for  Irradiance vs. Forward Current.)
• The TIA of the OSEM PD is now 121K instead of the previous 39.2K
• The OSEM output is received by differential AA.

--> Naive estimation is (35/25) x (121k/39.2k) x 2 = 8.64.

- MC1 sat amp has already been replaced with the aLIGO version by Gautam. I wonder where this factor of 2.15 came from (not 2...?).

- Coil driver response:

Previous setup

• BS/PRM/ITMX/ITMY had the differential receiver (cf https://nodus.ligo.caltech.edu:8081/40m/17840)
• All the removed coil drivers have output Rs of 100 Ohm
• The coil resistance is 16~20Ohm.

--> BS/PRM/ITMX/ITMY 2 V_DAC/118 Ohm = 1.7e-2 A/V x V_DAC

--> MC1/MC2/MC3/SRM V_DAC/118 Ohm = 8.5e-3 A/V x V_DAC

--> All the side coils V_DAC/118 Ohm = 8.5e-3 A/V x V_DAC

New setup

• The AIs have the gain of 1.
• The coil driver has a gain of 1.2.
• The output Rs for the face coils are 1.2k//100Ohm = 92Ohm

--> 2 V_DAC * 1.2 / (92+18) Ohm = 2.2e-2 A/V x V_DAC

BS/PRM/ITMX/ITMY face coils will have x1.3 more actuation.

MC1/MC2/MC3/SRM face coils will have x2.6 more actuation.

• The output Rs for the side coils are 1.2k

--> 2 V_DAC * 1.2 / 1200 Ohm = 2.0e-3 A/V x V_DAC

MC1/MC2/MC3/SRM will have less actuation by a factor of 1/4.25.

ITMX 400Ohm
ITMY 400Ohm
BS 100
PRM 100
SRM 100
MC2 427.5/410/411/409.4/410
MC1 434.5/428.4/430.6/432.5/434.0
MC3 432.2/409.4/409.3/410.6/413.8
 

[Koji, Paco, Radhika]

We recorded the 5 OSEM sensor readings for each of the 8 upgraded optics.

1. The correct scale factor seems to be 9x for the sensible OSEM readings. This is consistent with the scale factor calculated here.

2. The expected counts for each sensor is between 10,000-15,000 cts.

3. Several OSEM sensor values have bad readings of ~0 cts, or a few orders of magnitude smaller than expected:

4. Several OSEM sensor values have readings < 0 (there's overlap with the previous group):

5. MC1 has a consistent scale factor of 2.15, quite smaller than expected. Note that its OSEM readings were negative before the upgrade and now positive; hence negative MC1 ratios below.

Here is the full matrix of OSEM ratios after/before upgrade:
 

Next steps
 

We will debug the corresponding circuits tomorrow.

[JC, Paco, Radhika, Koji]

Morning/Afternoon work:

• Long DB cable installation (contd)
  • 14 DB25 cables went through the cable-rack bridge above the ITMX chamber. It's twice the previous # of cables.
  • These cables were connected to the chamber flanges. ITMY Flange1 had been having the 2nd connector malfunction. So the cables were connected to the connector 1 and 3 (as before).
• DSUB cable labeling
  • All the (long) cables connected to the units were labeled appropriately.
• c1lsc fiber reinstallation
  • We found one fiber cable (AlpenIO Inc PCIe 4x10G 100m AIO-PCIe4X-100 (2010)) already routed from 1Y4 to the PSL rack. This is the spare JC told us. We routed the host end to 1Y7.
  • c1lsc is up and running as before. All the models are up an running (burtrestore still needed).
• c1sus channel assignment
  • We have the swap of ADC0/1/2 so that the oplev ADC will have ADC0 (duo tone at CH31)
  • The channel assignments were modified:
    • SUS numbering "n": (0-PRM / 1-BS / 2-ITMX / 3-ITMY / 4-SRM / 5-MC2 / 6-MC1 / 7-MC3)
    • Face OSEMs ADC1 CH (n x 4 + 0~3, UL/LL/UR/LR)
    • Side OSEM ADC2 CH n
    • Oplev Ch ADC0 CH (n x 4 + 0~3)
    • Face OSEMs DAC0 CH (n x 4 + 0~3)
    • Side OSEM DAC1 CH n

• Sus damping control recovery
  • We need to lookin to MC3 UL/UR, PRM SD, SRM UL/LL/UR, ITMX all, ITMY face. See next post.
• Tool / Debris cleaning

c1sus ADC1 AA chassis fix.

• Brought the chassis on the workbench.
• Opened the chassis and "alas!" The internal power cables were not connected to the PCBs. This makes sense why there was no current draw at all.
• The cables were connected.
• The unit was tested with +/-18V power. At least, the diff outputs of the AA boards were all 0V.
• A missing connector screw for the external power connector was fixed

The unit was installed on the 1X5 rack, and the DB9 cables were connected.
Repair mission completed.

I've turned on the Eurocard crate (+/-15V) and the AA units (+/-18V) to check if the oplev channels are working. (It seems to be running well)
Also, the AA units are not too hot so far.

Dolphin Fencing technique

I believe that the dolphin emits some glitches to the other hosts during the host machine shutting down and starting up.
However, if the dolphin is disabled, that FE process will not run.

Therefore, we need some technique:

• When you have a real-time host to be restarted, we can disable the dolphin of that machine.
  e.g. If c1lsc has a problem, run the following command on fb1.
  ./dolphin_ix_port_control.sh --disable 192.168.113.40 2
  This allows us to restart the c1lsc in a safe way.
   
• Restart c1lsc in the above example. Go to c1lsc and run
  sudo reboot; exit
   
• This above brings you back to the previous host you were (suppose it is fb1). Run ping on that restarting machine.
  ping c1lsc
   
• While the c1lsc is shutting down, ping still has the response. Once the restart starts, it makes no response. Then, you can enable dolphin.
  ./dolphin_ix_port_control.sh --enable 192.168.113.40 2
   
• The process comes back automatically. You'll see DK status during the restart. I should disappear once all the models are up.

[Radhika, Paco, Murtaza, Koji]

We made great progress today. It's going well so far.

Morning work (10:30AM~):

• Installed c1sus IO chassis
• Installed all units
• Removed long (previous) custom DB25 cables for MC1

Afternoon work

• Connected all the units to the DC power strip
• Unit powering test was done before the inter-unit DSUBs were connected.
  • We found one AA chassis don't turn on even though internal +/-15V seems supplied. We pulled the unit out.
  • All the other units were fine.
  • Typical current draw of the units: (unit name, positive supply current, negative supply current)
    • AA 0.5A 0.5A (18W)
    • AI 0.3A 0.3A  (11W)
    • BIO 0A 0A (0W)
    • Trillum I/F 0.1A 0.1A (0.4W)
    • Sat amp 0.3A 0.3A (11W)
    • Coil Driver 0.3A 0.2A (9W)
• Cabling between the units
  • Done except for the extracted AA unit and for the BIO units (not needed until we have Acromag).
• Long DB cable installation (sat box removal)
  • Halfway
• c1lsc fiber tracking/replacement (not yet done)
  • We found the spare box with 3 more cables behind the X-arm tube. The replacement has not been done yet.

Evening work

• c1sus powering up and CDS check
  • After the people had left, I tried to start up c1sus. I had used dolphin fencing, but it worked like a charm!
• ADC1 AA chassis repair

To Do on Mon (10AM~)

• Long DB cable installation (contd)
• DSUB cable labeling
• Sus control system recovery
• c1lsc fiber reinstallation and system recovery
• Tool / Debris cleaning

Eventual needs:

• We need good crimping tools.
• Supply shortage of the crimping connectors.
• Fibers should not be routed together with electronics cables. Fibers should be distributed through tubes hanging on the cable racks
• Cable strain relief
• Move the noisy CDS and DC power supplies to the drill press room.

Rack nut policy

Out rack nut / screws are so much contaminated.
It's a mixture of #10-32 (standard) / M5 (wrong) / M6 (wrong).
Even the labeled bottles are contaminated.
Don't believe the installed rack nuts/screws, even if they seemed to work fine. They may be a metric pair.

Golden standard (Attached photo)

• rack nuts marked 1032
• small washers
• 10-32 tapered-round head screws

If you find other hardware, don't mix them with any stocks. Give them to Rack Nut Police (=Koji). He will hide them to some where secret.

[Radhika, Paco, Murtaza, Koji]

Morning work:

• Power Strip assembly
• Power Strip cable crimping
• c1lsc fiber routing (the PCIE fiber was in danger) / c1lsc machine was stopped after dolphin fencing
• We tried to reroute the c1lsc fiber above the racks such that it does not get pitched by the rack doors,
  but it seemed that the fiber was damaged (Attachment 1) and the c1lsc can't talk with the IO chassis anymore. We need to replace the fiber.
  It seems that P/N is PCIEO-4G3-100.0-11 (Samtec)

Afternoon work:

• Removed the sidepanels of 1X3 and 1X4 for easier work
• Removed the long DB25 cables from the old sat boxes.
• DC power strips are installed.
• Finished cleaning up the side cross-connects
• Checked the DC supply conditions.
  • +/-15V Eurocard crate + Guralp requires 0.5A / 0.6A
  • +/-18V appered on the DC power strips correctly
• Cleaned up the floor a bit
• Murtaza noticed that there was some strange intermittent noise around the 1Y0/1 racks.
  It looks like one of the fans for the c1ioo IO chassis are dying.

Evening work:

• Made coil driver short plugs (Attachment 2). This enables the coils while the coil modes are set to be Acquire mode.
• Rack nuts inserted

Tomorrow plan (10:30AM):

• c1sus IO chassis installation
• Unit installation
• Unit powering tests (before connecting them)
• Cabling between the units
• Long DB cable installation (sat box removal)
• c1lsc fiber replacement
• Some above cable removal (fiber, old custom DB25 for MC1, etc)

[Radhika, Paco, Murtaza, Koji]

- Removed all the units that will not be used in the new setup.
- Removed the sidepanel crossconnects
- Removed most of the sidepanel power lines except for the top eurocrate at the top of 1X4 (requires +/-15 pale orange and blue).
- Removed the acromag connections
- Removed the connectors of the long suspension cables

We'll resume the work at 10AM. (We'll have breaks for lunch and the seminar at 3PM)

=== Next steps ===

- Continue to remove the long suspension cables.
- Attach the DC power strip
- Continue to clean up the power lines on the rack side
- Prepare the power lines (Guralp requires +/-15V, the Eurocard crate +/-15V, new power strips (x3) +/-18V)
- Install the units on the racks.

- 4x 1064nm NPROs are OFF. The lab hall is laser safe, although the oplev lasers are on.
  The Laser Warning Signs were turned off by the interlock switch at the PSL enclosure (control room side)

- Watch dogs were turned to "disabled"
- Halted c1sus using dolphin fencing. This worked very well.
The previous report of dolphin fencing not working was due to a typo in my instruction (wrong -disable -> correct --disable).

controls@fb1:~ 0$ ./dolphin_ix_port_control.sh --disable 192.168.113.40 1
--disable
Disabling switch_ip 192.168.113.40, port 1
Complete - csr write addr=0x0001C050, val=0x20820090 (with ret=0)

ssh c1sus

controls@c1sus:~$ rtcds stop --all

OSEM values for 8 vertex optics ~before~ electronics upgrade (averaged over 60 s):

['C1:SUS-BS_ULSEN_IN1', 'C1:SUS-BS_URSEN_IN1', 'C1:SUS-BS_LRSEN_IN1', 'C1:SUS-BS_LLSEN_IN1', 'C1:SUS-BS_SDSEN_IN1', 'C1:SUS-ITMX_ULSEN_IN1', 'C1:SUS-ITMX_URSEN_IN1', 'C1:SUS-ITMX_LRSEN_IN1', 'C1:SUS-ITMX_LLSEN_IN1', 'C1:SUS-ITMX_SDSEN_IN1', 'C1:SUS-ITMY_ULSEN_IN1', 'C1:SUS-ITMY_URSEN_IN1', 'C1:SUS-ITMY_LRSEN_IN1', 'C1:SUS-ITMY_LLSEN_IN1', 'C1:SUS-ITMY_SDSEN_IN1', 'C1:SUS-PRM_ULSEN_IN1', 'C1:SUS-PRM_URSEN_IN1', 'C1:SUS-PRM_LRSEN_IN1', 'C1:SUS-PRM_LLSEN_IN1', 'C1:SUS-PRM_SDSEN_IN1', 'C1:SUS-SRM_ULSEN_IN1', 'C1:SUS-SRM_URSEN_IN1', 'C1:SUS-SRM_LRSEN_IN1', 'C1:SUS-SRM_LLSEN_IN1', 'C1:SUS-SRM_SDSEN_IN1', 'C1:SUS-MC1_ULSEN_IN1', 'C1:SUS-MC1_URSEN_IN1', 'C1:SUS-MC1_LRSEN_IN1', 'C1:SUS-MC1_LLSEN_IN1', 'C1:SUS-MC1_SDSEN_IN1', 'C1:SUS-MC2_ULSEN_IN1', 'C1:SUS-MC2_URSEN_IN1', 'C1:SUS-MC2_LRSEN_IN1', 'C1:SUS-MC2_LLSEN_IN1', 'C1:SUS-MC2_SDSEN_IN1', 'C1:SUS-MC3_ULSEN_IN1', 'C1:SUS-MC3_URSEN_IN1', 'C1:SUS-MC3_LRSEN_IN1', 'C1:SUS-MC3_LLSEN_IN1', 'C1:SUS-MC3_SDSEN_IN1']

[1943.7368693033854, 1597.8396443684896, 1667.4166076660156, 1709.3347656250003, 1760.058290608724, 1013.2686828613281, 1833.5396423339844, 2096.6071411132816, 786.2030975341798, 1152.7826700846356, 923.4767690022786, 447.51257578531903, 678.4539184570312, 971.2538736979167, 720.261508178711, 2458.092639160156, 2245.09764811198, 710.1112263997396, 258.2382120768228, 1225.202982584636, 3048.1043741861986, 3092.901733398437, 77.94405148824053, 181.08351745605466, 5145.644441731771, -6879.474226888021, -6954.500219726562, -3615.385673014323, -5468.913354492189, -3260.200516764323, 1555.9213073730468, 1625.1883911132813, 348.9968526204428, 1011.3114247639974, 1386.0941060384114, 1014.3825622558594, 1218.7048522949217, 1642.9303202311198, 1621.7448221842449, 720.2911783854166]

• [OK] Reflected the sorensen setup (minimal change from the conventional config. (See the attachment)

• Before destroying the current setup, bring the alignment biases for the vertex 8 sus to zero and record all the OSEM values.
  => Radhika did it (next ELOG)
  -> This will give us the ratio of the OSEM error signals to know the gain ratios between before and after. Also this will make it easier to bring the alignment back.

• [OK] I suppose the oplevs are still aligned. We don't need to be too nervous about the oplev spot too much.

• How to compensate the coil force cal? Do we know the ratio from the ETM coil driver swap? (What were the coil output Rs? What are they now? Are the ratios reasonable?)

  • Currently:
    - PRM/BS/ITMX/ITMY DAC output for face coils differential
    - SRM/MC2/MC1/MC3 DAC output for face coils single ended
    - All 8 SD coils single ended
    - Coil Output Rs
      According to D1700218
      PRM unknown to be checked
      BS 100Ohm
      ITMX 100Ohm
      ITMY 100Ohm
      SRM 100Ohm
      MC2 430? unknown to be checked
      MC1 430? unknown to be checked
      MC3 430? unknown to be checked

  • New setup
      DAC output differential
      AI has the gain of 1 / HAM coil driver has a gain of 1.2
      Coil output Rs:
      For all the face coils 1.2k // 100 ~ 92Ohm
      For all the side coils 1.2k

Note on Sorensen:

- Eurocard crate requires +-15V. We can place two 15V Sorensens on 1X4 for Eurocard crate or just leave the current +/-15V supplies.

- The aLIGO units requires +/-18V. We can place two 15V Sorensens on 1X5 or just leave two of the current supplies and set them to +/-18V.

We are ready to do the transition from Wed 1PM.

The items for the upgrade was collected around the vertex area (Attached photo).

- aLIGO-style DC power strip (+/-18V) x3
- DC power cables (orange +/-18V)
- Electronics units for the upgrade.
- DSUB (DB9) cables
- Custom DSUB15-DSUB25 cables
- Custom DSUB25 cables

For the preparation of the electronics upgrade, three 18bit DAC AI chassis were transformed to 16bit version.

The power supply connections were touched, so the units were tested with +/-18V, and they work as expected.

[Radhika, Murtaza, Paco]

Today we decided to take a closer look at the demod phases of the T and L error signals for XARM ASS. By eye we tuned the phases to minimize the signal in Q. Here are the new demod phases:
(THE DEMODULATION PHASE VALUE DO NOT RESTORE BACK TO THE ORIGINAL VALUES WHEN DITHER IS TURNED ON.)

We also noticed that MEDM indicator for dithering on (white --> green LO symbol) for ETM_YAW_L_OSC was tied to the wrong excitation gain channel (C1:ASS-XARM_ITM_YAW_OSC_CLKGAIN instead of C1:ASS-XARM_ETM_YAW_OSC_CLKGAIN). We went ahead and changed this in [insert medm file location]. So now the right green LO symbol appears when the appropriate excitation is turned on.

[Radhika, Murtaza]

This post summarizes XARM ASS efforts from Friday 9/8 and Saturday 9/9.

FRIDAY

On Friday, we continued with our previous output matrix that used the length error signals (ITM/ETM PIT/YAW L) to feed back to ITMX and ETMX (see the previous ELOG). In that state we did not use the transmission error signals and had no feedback going to the BS. We then tried to use the transmission error signals ITM PIT/YAW L as a proxy for BS input pointing and feed them back to the BS. For both PIT and YAW, both signs of feedback resulted in diverging T error signals and a decrease in transmission.

SATURDAY

On Saturday, we used the transmission error signals (ITM/ETM PIT/YAW T) in the sensing matrix to build the output matrix. We got it to a state where we could get the controlled error signals to converge by just feeding back to the ITMX and ETMX (Attachments 1,2). Once we had this working, we tried to feed back a combination of (ETM PIT/YAW L and ITM PIT/YAW T) to correct BS pointing. However, any combinations and signs to the BS dropped transmission and led to diverging error signals.
We then attempted to use the latest working XASS output matrix (before the acromags were pulled out) and see the effect of flipping signs in there (one optic+DOF at a time) We then tried to use the sign logic from the previously working ETM/ITM feedback we got partially working; however the error signals did not converge with any combination.

SUMMARY:

- We are able to successfully feed back to ITMX and ETMX, using either length or transmission error signals. It is when we try to add BS feedback that ASS fails. This can be due to the fact that we need to consider the relative servo gains when treating these loops separately, like Koji mentioned.

- The sensing matrix approach might be the only way to simultaneously optimize feedback for all optics, avoiding the need to tune servo gains. We will revisit this approach on Monday.

     - Koji pointed out that we are reading out the low-passed error signals in order to calculate each step response - we will need to consider our sampling rate and duration of averaging accordingly.

     - It will be harder to iteratively flip signs of each matrix element for this dense matrix, and we will have to be clever about which sign combinations we try for actuation.

I came to the lab to see the recovery work from the power glitch this morning 8:15AM. All CDS seems up. The suspensions are somewhat aligned. Some of them were not damped. The oplevs were off. Radhika is working on the recovery of the FP arms.

I noticed that FSS Slow servo is not working. I always forget what is the right way to turn it on. Here is the summary:

How to turn on FSSSlow (2023 Sept version)

• Go to megatron
• sudo systemctl enable FSSSlow
• sudo systemctl start FSSSlow
• sudo systemctl status FSSSlow

  ● FSSSlow.service - Script to run the PID temperature control servo for the PSL
     Loaded: loaded (/opt/rtcds/caltech/c1/Git/40m/scripts/PSL/FSS/FSSSlow.service; enabled; vendor pre
     Active: active (running) since Fri 2023-09-08 17:10:52 PDT; 1s ago
   Main PID: 2088 (python3)
      Tasks: 6 (limit: 4674)
     CGroup: /system.slice/FSSSlow.service
             └─2088 /usr/bin/python3 /opt/rtcds/caltech/c1/Git/40m/scripts/PSL/FSS/PIDLocker.py PIDConf

  Sep 08 17:10:52 megatron systemd[1]: Started Script to run the PID temperature control servo for the

Tonight I managed to lock CARM and DARM under ALS control only

Configuration

• Arm cavities well aligned, TRY ~ 1.07, TRX ~ 0.98, GTRY ~ 1.2, GTRX ~ 0.77
• HEPA off, WFS 60s offloaded, PD offsets removed, all lights inside the lab were off
• BEATX and BEATY ool residual noise shown in Attachment #1.
• Error points, the A path was the same as what is used for electronic FPMI. For the B path, I describe the tuning below.
  • CARM_A = 0.5 * POX11_I + 0.5 * POY11_I
  • DARM_A = 0.19 * POX11_I - 0.19 * POY11_I
  • CARM_B = -0.7 * ALSX + 0.4 * ALSY  
  • DARM_B = -0.25 * ALSX - 0.17 * ALSY
• Power normalization of the error signals was 0.5 * TRX + 0.5 * TRY in both paths.
• LSC filter banks are the same ones we use for electronic FPMI, and the gains were
  • CARM = 0.011  (UGF ~ 200 Hz) using FM1 to FM5, FM6 and FM8
  • DARM = 0.055 (UGF ~ 150 Hz) using FM1 to FM5, FM6 and FM8
• Control points, I temporarily disabled violin filters around 600 Hz to ease the lock acquisition ... we should really use the VIO TRIG here to avoid having to do this.
  • CARM = -0.734 * MC2
  • DARM = 0.5 * ETMX - 0.5 * ETMY

ALS error signal tuning

To find the error signals for CARM/DARM, I turned on the oscillators (at 307.8 and 313.31 Hz respectively) with 150 counts and enabled FM10 (Notch for sensing matrix) in the CARM and DARM servo banks. I then removed the ALS offsets (C1:LSC-ALSX_OFFSET, C1:LSC-ALSY_OFFSET) and looked at the transfer functions shown in Attachment #2. I optimized the ALS blending until I maximized the CARM and DARM A to B paths and minimized CARM and DARM cross couplings. The signs were chosen to leave a phase of 0.

Handoff

After measuring the OLTFs for eCARM and eDARM (loop closed with the A error point) and tuning the ALS error signals, I gradually blended the A and B paths and checked the OLTFs for CARM and DARM. During this I realized I needed to disable some of the notch violin filters because they sometimes made the DARM loop unstable after >50% blending. In the end the simultaneous CARM_A/DARM_A to CARM_B/DARM_B handoff was successful in 0.5 seconds. Attachment #3 shows the OLTFs under ALS control.

CARM offset

After getting nominally stable ALS control, I tried adding an offset. The LSC CARM offset range was insufficient, so I ended up directly scanning the C1:LSC-ALSX_OFFSET and C1:LSC-ALSY_OFFSET. The first couple of attempts the ramp time was set to 2.0 seconds, and a step of 0.01 was enough to break the lock. I managed to hold the control with as much as C1:LSC_CARM_A_IN1 offset by ~ 500 (rms ~ 200 counts). I roughly estimate this to be ~ 5% of the CARM pole which is 4 kHz in this case so overall 200 Hz which is not that large.

[Radhika, Murtaza]

We recalculated the sensing matrix for XARM ASS by collecting each sensor's step response to an offset in each DOF. This produced the following dense output matrix A (see Attachment 1 for rows/cols):

      [[-0.02047695,  0.        , -0.10262752,  0.        , -0.0157128 , 0.        ],
       [ 0.        ,  0.16908344,  0.        , -0.00929291,  0.        ,-0.35916455],
A =    [-0.28050764,  0.        ,  0.26982002,  0.        , -0.55100297, 0.        ],
       [ 0.        ,  0.85501491,  0.        ,  0.0606197 ,  0.        , 0.27568672],
       [-0.95963335,  0.        ,  0.95742611,  0.        ,  0.83435534, 0.        ],
       [ 0.        , -0.49026554,  0.        , -0.99811768,  0.        , 0.89162641]]

Turning the XASS gain up slowly to ~0.15, we observed that several error signals diverged and transmission started to drop. Debugging this matrix proved difficult since there were many nonzero elements to consider. So we reverted to build the matrix from our intuition, considering the centering and input pointing loops, and using the YASS output matrix as a reference.

The YARM ASS servo gains are all +1. The YASS output matrix has the following length (centering) signal mapping:

We mirrored this in the XASS output matrix. Note that previously the ITM L error signals were not used for XASS. To simplify the process, we decided to just work out the beam centering first and ignore the input pointing coming from the beam splitter (setting BS PIT/YAW matrix elements to 0). We also set all the XARM ASS servo gains to +1. See the output matrix below:

We cleared the outputs and turned on the XARM GAIN slowly (0.1) and immediately noticed the YAW signals in ETM start to diverge (C1:ASS-XARM_ETM_YAW_T_DEMOD_I_OUT16, C1:ASS-XARM_ETM_YAW_L_DEMOD_I_OUT16). We turned down the XARM gain and flipped the sign for the signal going to ETM YAW. (suspect a difference in sign convention).

To check the stability, we sequentially gave offsets in PIT/YAW for ETM and ITM. We saw the signal (C1:ASS-XARM_ETM_PIT_L_DEMOD_I_OUT16) oscillate wildly at a frequency of ~(1/15 Hz). We suspected the ASS loop was driving these oscillations so we turned down the gain going to ETM PIT to 0.25 which worked really well and the transient oscillation of further checks was gone.

We saw similar wild oscillatory signals in ITM PIT (C1:ASS-XARM_ITM_PIT_T_DEMOD_I_OUT16, C1:ASS-XARM_ITM_PIT_L_DEMOD_I_OUT16) on applying offsets so we reduced the gain going to ITM PIT to 0.3. (0.25 and 0.3 are arbitary relatively smaller weights, can be fine tuned).

We checked the stability of this setup as a whole by giving a few offsets to ITMX and ETMX, with a servo gain of 0.15 it did a great job! (0.25 made it diverge once again). See final state for centering in Attachment 1, and error signal suppression in Attachment 2. (Ignore XAUX transmission in grey - we were toggling the shutter.) Note that the Length error signals were successfully suppressed, but the dark green/brown Transmission error signals were not fed back and thus remain nonzero.

WE SHALL INVESTIGATE THE INPUT POINTING NEXT FEEDING BACK TO THE BS and ITMX. We will give an update shortly about whether restoring XARM ASS is feasible by Monday.

COIL BALANCING FOR ETMX
Summary : I ran coil balancing on ETMX using the CoilStrengthBalancing.ipynb script to get a feel for it, no changes were required from the last time it was run by Paco. I realized I was measuring the wrong signal for the POS coupling (C1:SUS-ETMX_SUSPOS_IN1) while trying to minimize the BUT-POS coupling. This was stupid because the shadow sensors and actuation coils in this case are the OSEMs. The LSC error signal would be more appropriate for measuring the POS coupling.

The convention for the actuation vector used for the coils is [UL, UR, LL, LR]. The frequency, excitation counts are given through the LOCKIN1 channel (SUS->ETMX->LOCKIN1->f(Hz), Amp). The excitation vector is set in (SUS->ETMX->Output Filters->LOCKIN1)
Here, the excitation frequency used = 13Hz.
For "decoupling" the degrees of freedom, the script used is given in /opt/rtcds/caltech/c1/Git/40m/scripts/SUS/coilStrengthBalancing/ETMX/CoilStrengthBalancing.ipynb. In here, small steps are taken to "remove" the DOF contribution such that \\new_gain = old_gain +/- e*(decoupling DOF vector)\\. (e = step size)
The decoupling signals are observed in diaggui in the frequency range of (0 - 20Hz) and a bandwidth of 0.5Hz with exponential averaging (10 averages)

1. Minimizing BUT-POS coupling
Here, the LSC error signal (C1:LSC-POX11_I_IN1) is observed to measure the coupling in POS. For this, the arm is kept locked to obtain a decent error signal.
SANITY CHECK: This was tested by exciting POS [1,1,1,1] at 13Hz and measuring at the LSC error signal in diaggui which indeed showed a peak at 13Hz indeed. Damp filters and OPLEV servos were enabled to prevent the loss of lock.
The initial excitation was given to the Butterfly DOF [1, -1, -1, 1] at 10000 counts and 5 steps were taken in both directions of the POS vector to decouple the POS DOF. The initial peak showing up in the LSC error signal was already at a minimum. The excitation was ramped up to 20000 counts, where the peak was still very small. Thus, no change was made here (Attachment 1).

2. Minimizing POS-PIT coupling
Here, the OPLEV signal for PIT (C1:SUS-ETMX_OL_PIT_IN1) is observed to measure the coupling in PIT. The damping filters and OPLEV servos are disabled.
The initial excitation is given to POS [1, 1, 1, 1] at 5000 counts and 5 steps were taken in both directions of the PIT vector to decouple the PIT DOF. The initial peak showing up in the OPLEV signal was already at a minimum. Thus, no change was made here (Attachment 2).

3. Minimizing POS-PIT coupling
Here, the OPLEV signal for YAW (C1:SUS-ETMX_OL_YAW_IN1) is observed to measure the coupling in YAW. The damping filters and OPLEV servos ared disabled.
The initial excitation is given to POS [1, 1, 1, 1] at 5000 counts and 5 steps were taken in both directions of the YAW vector to decouple the YAW DOF. The initial peak showing up in the OPLEV signal was already at a minimum. Thus, no change was made here (Attachment 3).

4. Minimizing PIT-YAW coupling
This was one rather robust and was not susceptible to the decoupling process. Here, the OPLEV signal for both PIT (C1:SUS-ETMX_OL_PIT_IN1) and YAW (C1:SUS-ETMX_OL_YAW_IN1) are used to measure their relative coupling. Either of the DOF can be excited and while the other DOF can be used for the decoupling vector. Here, PIT was excited and the decoupling DOF vector was YAW. The damping filters and OPLEV servos ared disabled.
The initial excitation was given to the PIT DOF [1, 1, -1, -1] at 5000 counts and 5 steps were taken in both directions of the YAW vector to decouple the YAW DOF. The initial peak showing up in the YAW signal was already at a minimum. The excitation was ramped up to 10000 counts, however the YAW peak barely moved. Thus, no change was made here (Attachment 4).

To next.