I successfully steered out the two output beams from BHD BS to ITMY table today. This required significant changes on the table, but I was able to bring back the table to balance coarsely and then recover YARM flashing with fine tuning of ITMY.

• The counterweights were kept at the North end of the table which was in way of one of the output beams of BHD.
• So I saved the level meter positions in my head and removed those counterweights.
• I also needed to remove the cable post for ITMY and SRM that was in the center of the table.
• I installed a new cable post which is just for SRM and is behind AS2. ITMY's cable post is next to it on the other edge of the table. This is to ensure that BHD board can come in later without disturbing existing layout.
• I got 3 Y1-45P and 1 Y1-0 mirror. The Y1-0 mirror was not installed on a mount, so I removed an older optic which was unlabeled and put this on it's mount.
• Note that I noticed that some light (significant enough to be visible on my card) is leaking out of the 45P mirrors. We need to make sure we aren't loosing too much power due to this.
• Both beams are steered through the center of the window, they are separating outside and not clipping on any of the existing optics outside. (See attachment 1, the red beam in the center is the ITMY oplev input beam and the two IR beams are the outputs from BHD BS).
• Also note that I didn't find any LO beam while doing this work. I only used AS beam to align the path.
• I centered the ITMY oplev at the end.

Next steps:

• LO path needs to be tuned up and cleared off again. We need to match the beams on BHD BS as well.
• Setup steering mirrors and photodiodes on the outside table on ITMY.

[Anchal, Yuta]

We checked POX and POY RF signal chains for sanity check since Xarm cannot be locked in IR stably as opposed to Yarm.
POX beam seems to be healthy. This issue doesn't prevent us from closing the vacuum tank.

POY
 - RF PD has SPB-10.7+ and ZFL-500NL+ attached to the RF output.
 - At the demodulation electronics rack, SMA connectors are used everywhere.
 - With Yarm flashing at ~1, RF output has ~24 mVpp right after RF PD, ~580mVpp after SPB-10.7+ and ZFL-500NL+, and ~150mVpp at right before the demodulation box.
 - There is roughly a factor of 3 loss in the cabling from POY RF PD to the demodulation rack.
 - Laser power at POY RF PD was measured to be 16 uW

POX
 - RF PD doesn't have amplifiers attached.
 - At the demodulation electronics rack, N connector is used.
 - With Xarm flashing at ~1, RF output has ~30 mVpp right after RF PD, and ~20mVpp at right before the demodulation box.
 - Losses in the cabling from POX RF PD to the demodulation rack is small compared with that for POY.
 - Laser power at POX RF PD was measured to be 16 uW

Summary
 - POX and POY RF PDs are receiving almost the same mount of power
 - POY has larger error signal than POX because of RF amplifier, but the cable loss is high

Conclusion
 - There might be something in the electronics, but we can close the vacuum tanks

[Yehonathan, JC]

Yehonathan and I attempted to align the LO2 beam today through the BS chamber and ITMX Chamber. We found the LO2 beam was blocked by the POKM1 Mirror. During this attempt, I tapped TT2 with the Laser Card. This caused the mirror to shake and dampen into a new postion. Afterwards, when putting the door back on ITMX, one of the older cables were pulled and the insulation was torn. This caused some major issues and we have been able to regain either of the arms to their original standings.

[Yuta, Anchal, Paco]

As described briefly by JC, there were multiple failure modes going during this work segment.

ITMX SatAmp SAGA

Indeed, the 64 pin crimp cable from the gold sat amp box broke when work around ITMX chamber was ongoing. We found the right 64 pin head replacement around and moved on to fix the connector in-situ. After a first attempt, we suddenly lost all damping on vertex SUS (driven by these old sat amp electronics) because our c1susaux acromag chassis stopped working. After looking around the 1x5 rack electronics we noted that one of the +- 20 VDC Sorensens were at 11.6 VDC, drawing 6.7 A of current (nominally this supply draws over 5 Amps!) so we realized we had not connected the ITMX sat amp correctly, and the DC rail voltage drop busted the acromag power as well, tripping all the other watchdogs ...

We fixed this by first, unplugging the shorted cable from the rack (at which point the supply went back to 20 VDC, 4.7 A) and then carefully redoing the crimp connector. The second attempt was successful and we restored the c1susaux modbusIOC service (i.e. slow controls).

ITMY SatAmp SAGA

As we restored the slow controls, and damped most vertex suspensions, we noticed ITMY UL and SD osems were reading 0 counts both on the slow and fast ADCs. We suspected we had pulled some wires around when busy with the ITMX sat amp saga. We found that Side OSEM cLEMO cable was very loose on the whitening board. In fact, we have had no side osem signal on ITMY for some time. We fixed this. Nevertheless the UL channel remained silent... We then did the following tests:

• Test PD mon outputs on the whitening card. We realized the whitening cards were mislabeled, with ITMX and ITMY flipped . We have labeled them appropriately.
• Tested input DB15 cable with breakout board.
• Went to the ITMY sat amp box and used the satellite box TESTER 2 on J1. It seemed correct.
• We opened the chamber, tested the in-vacuum segments, they all were ok.
• We flipped UR-UL OSEMs and found that the UL OSEM is healthy and works fine on UR channel.
• We tested the in-air cable between satellite box and vacuum flange and it was ok too.
• We suspected that the satellite box tester  is lying, so we replaced the satellite box with the spare old MC1 satellite box, and indeed that solved the issue.

DO NOT TRUST THE SATELLITE BOX TESTER 2.

Current state:

• IMC locking normally.
• All suspensions are damping properly.
• Oplevs are not centered.
• No flashing on either of the arms. We had no luck in ~20 min of attempt with just input injection changed.
• On kicking PR3, we do see some flashing on XARM, which means XARM cavity atleast is somewhat aligned.
• All remaining tasks before pumpdown are still remaining. We just lost the whole day.

I heard a rumor about a DAQ problem at the 40m.

To investigate, I tried retrieving data from some channels under C1:SUS-AS1 on the c1sus2 front end. DQ channels worked fine, testpoint channels did not. This pointed to an issue involving the communication with awgtpman. However, AWG excitations did work. So the issue seemed to be specific to the communication between daqd and awgtpman.

daqd logs were complaining of an error in the tpRequest function: error code -3/couldn't create test point handle. (Confusingly, part of the error message was buffered somewhere, and would only print after a subsequent connection to daqd was made.) This message signifies some kind of failure in setting up the RPC connection to awgtpman. A further error string is available from the system to explain the cause of the failure, but daqd does not provide it. So we have to guess...

One of the reasons an RPC connection can fail is if the server name cannot be resolved. Indeed, address lookup for c1sus2 from fb1 was broken:

$ host c1sus2
Host c1sus2 not found: 3(NXDOMAIN)

In /etc/resolv.conf on fb1 there was the following line:

search martian.113.168.192.in-addr.arpa

Changing this to search martian got address lookup on fb1 working:

$ host c1sus2
c1sus2.martian has address 192.168.113.87

But testpoints still could not be retrieved from c1sus2, even after a daqd restart.

In /etc/hosts on fb1 I found the following:

192.168.113.92  c1sus2

Changing the hardcoded address to the value returned by the nameserver (192.168.113.87) fixed the problem.

It might be even better to remove the hardcoded addresses of front ends from the hosts file, letting DNS function as the sole source of truth. But a full system restart should be performed after such a change, to ensure nothing else is broken by it. I leave that for another time.

[Anchal, Paco, JC]

Thanks Chris for the fix. We are able to access the testpoints now but we started facing another issue this morning, not sure how it is related to what you did.

• The C1:LSC-TRX_OUT and C1:LSC-TRY_OUT channels are stuck to zero value.
• These were the channels we used until last friday to align the interferometer.
• These channels are routed through the c1rfm FE model (Reflected Memory model is the name, I think). These channels carry the IR transmission photodiode monitors at the two ends of the interferometer, where they are first logged into the local FEs as C1:SUS-ETMX_TRX and C1:SUS-ETMY_TRY .
• These channels are then fed to C1:SCX-RFM_TRX -> C1:RFM_TRX -> C1:RFM-LSC_TRX -> C1:LSC-TRX and similar for Y side.
• We are able to see channels in the end FE filtermodule testpoints (C1:SUS-ETMX_TRX_OUT & C1:SUS-ETMY_TRY_OUT)
• However, we are unable to see the same signal in c1rfm filter module testpoints like C1:RFM_TRX_IN1, C1:RFM_TRY_IN1 etc
• There is an IPC error shown in CDS FE status screen for c1rfm in c1sus. But we remember seeing this red for a long time and have been ignoring it so far as everything was working regardless.

The steps we have tried to fix this are:

• Restart all the FE models in c1lsc, c1sus, and c1ioo (without restarting the computers themselves) , and then burt restore.
• Restart all the FE models in c1iscex, and c1iscey (only c1iscey computer was restarted) , and then burt restore.

These above steps did not fix the issue. Since we have  the testpoints (C1:SUS-ETMX_TRX_OUT & C1:SUS-ETMY_TRY_OUT) for now to monitor the transmission levels, we are going ahead with our upgrade work without resovling this issue. Please let us know if you have any insights.

It looks like the RFM problem started a little after 2am on Saturday morning (attachment 1). It’s subsequent to what I did, but during a time of no apparent activity, either by me or others.

The pattern of errors on c1rfm (attachment 2) looks very much like this one previously reported by Gautam (errors on all IRFM0 ipcs). Maybe the fix described in Koji’s followup will work again (involving hard reboots).

After Xarm and Yarm were aligned by Anchal et al, I aligned AS and REFL path in the AP table.
REFL path was alreasy almost perfectly aligned.

REFL path
 -REFL beam centered on the REFL camera
 -Aligned so that REFL55 and REFL33 RFPDs give maximum analog DC outputs when ITMY was misaligned to avoid MICH fringe
 -Aligned so that REFL11 give maximum C1:LSC-REFL11_I_ERR (analog DC output on REFL11 RFPD seemed to be not working)

AS path
 -AS beam centered on the AS camera. AS beam seems to be clipped at right side when you see at the viewport from -Y side.
 -Aligned so that AS55 give maximum C1:LSC-ASDC_OUT16 (analog DC output on AS55 RFPD seemed to be not working)
 -Aligned so that AS110 give maximum analog DC output

We followed the manual's guide for setting up MTS to sync on external signal. In the xrfdc package, we update the RFdc class to have RunMTS, SysRefEnable, and SysRefDisable functions as prescribed on page 180 of the manual. Then, we attempted to run the new functions in the notebook and read the DAC signal outputs on an oscilloscope. The DACs were not synced. We were also unable to get FIFOlatency readings. 

I was finally able to set up a stable suspension model with the help of Yuta and I'm now ready to start doing some MICH noise budgeting with BHD readout. (Tip: turns out that in the zpk function in Matlab you should multiply the poles and zeros by -2*pi to match the zpk TFs in Foton)

I copied all the filters from the suspension MEDM screens into a Matlab. Those filters were concatenated with a single pendulum suspension TF with poles at [0.05e-1+1i, 0.05e-1-1i] and a gain of 4 N/kg.

I multiplied the OLTF with the real gains at the DAC/DAC/OSEMs/Coil Driver and Coils. I ignore whitening/dewhitening for now. The OLTF was calculated with no additional ad-hoc gain.

Attachment 1 shows the calculated open-loop transfer function.

Attachment 2 shows OLTF of ETMY measured last week.

Attachment 3 shows the step and impulse responses of the closed-loop system.

[Anchal, Paco, Yuta]

• We aligned AS path avoiding any clipping to the AP table where we setup a camera with a lens.
  • To do this we had to move AS6 in North direction for ~1cm.
  • The Injection table was imbalanced by this move to drop the IMC transmission to half.
  • We did not balance the table again, we steered the input mirror to reach to 1000 counts (out of 1200 nominal) and then used WFS loop to get to the last bit.
  • The input to the arm cavities did not change much, XARM was still flashing to 0.8 max height and YARM to 0.2. We recovered these easily using the cavity mirror pair.
• We aligned the LO beam to be spatially matched on BHDBS with AS path.
  • The LO beam was steered to roughly overlap with the AS beam outputs on the BHDBS.
  • However, the LO beam size is very large and diverges after LO4.
  • According to 40m/15379, the 0.15m ROC of LO4 right after the beam waist is supposed to collimate the beam to a 522 um waist.
  • We confirmed that LO4 is marked as a 0.15m ROC mirror on its edge and the HR coating is facing the incident beam.
    • Conjecture (AG): The coating was applied to the flat side of the optic instead of the curved side.
    • This would explain why the beam is continuing to diverge after reflecting from LO4, and diverging fast.
  • We need to fix this issue before pumping down otherwise the mode matching would be too poor in BHDBS to have any meaningful results.
• The output of BHDBS was steered out and a GigE camera is set up to see this path.
  • The camera is set to see the transmitted AS beam from BHD BS (and reflected LO beam).
  • But the camera is unable to see any LO beam due to large divergence.
  • The LO beam essentially disappears after ~30 cm from the BHDBS.

[Anchal, Paco, Yuta]

SRM Placement

• SRM was moved from its parked location to the nominal position in the ITMY chamber.
• This imbalanced the table a lot as all SOS towers ended up on the south side of the table.
• I needed additionally three SOS tower side walls to recover the balance of the table.
  • I initially tried to use a level meter on my phone which claimed to have 0.1 degrees of accuracy. But it turned out to be a bad idea.
  • Eventually, I used the spirit bubble level meter we have, along with the OSEM values of ITMY, AS1, and AS4.
• At the end, the table is balanced as it was before, all SOS are damping usually.

SRM Sat Amp Box setup

• SRM Gold Box Sat Amp was found near the BS chamber.
• This box was moved to the ITMY chamber.
• The new flange on the East end was marked earlier for SRM. This flange on the vacuum side was connected with new in-vacuum blue ribbon cables.
• We had previously moved the cable post for SRM (40m/16849) behind AS2. This cable post is connected to the old in-vacuum cable.
• It would have changed the table balance to remove this cable post and connect new in-vacuum cables to it, so we decided to do this in the next vent when we put the BHD board on the table.
• For now, we connected the old in-vacuum cable to the new in-vacuum blue ribbon cables inside.
  • Note, that the old in-vacuum cable has a gender flipping section which also mirrors the pin layout.
  • We installed pin mirroring cables on the outside between the Sat Amp Box and the vacuum flange to revert back the additional mirroring.
  • However it happened, now the Sat Amp Box is working, with all OSEMs and coils alive.
• One peculiarity we found was that the SRM face OSEMs have only about 250-300 um of range, which is roughly 3 times less than the other OSEMs in other SOSs.
• SRM side OSEM however behaves normally.
• After installment, at the free-hanging state, SRM LL OSEM is saturated (too bright) and other face OSEMs are close to total brightness state.
• We'll first put the alignment offsets to get the SRM perpendicular to the beam coming from SR2 and then center the OSEMs in this tiny range.
• The low OSEM range could be due to improper biasing from the Sat Amp Box. Hopefully, with new electronics, this issue would go away in future.

[Paco]

The SRM Oplev injection and detection paths interfere heavily with the POY11. Due to the limited optical access, I suggest we try steering POYM1 YAW and adapting the RFPD path accordingly.

I centered WFS1 PD so that IMC WFS Servo does not go out of range.

[Anchal, Paco, Yuta, JC]

SRM Oplev setup

• We setup SRM oplev path for the aligned position of SRM.
• This was bit hard, because the return beam was following almost the same path as the input beam, and the return beam had become about 1 cm in diameter.
• We replaced one of the in-air steering mirror of SRM op-lev input beam with a 1 inch BS on a non-steeerable mount.
• The returning oplev beam is picked at transmission from this BS.
• Note: we are not sure if this BS is actually coated for IR or Visible. We couldn't find a visible BS in the lab. We should order a 2 in diameter visible BS to be placed in this position.
• Half of the input beam would be used for PRM Oplev input.
• The returning beam was focused with a 100mm focal length lens. Again, this lens is not verified to be for visible wavelength. We think it might have an AR coating for IR. We should get a visible lens for this position also.

PRM Placement

• PRM placed in nominal position + 2 cm, East.
• Currently, PRM SOS tower is blocking BS oplev input beam, this needs to be adjusted.
• Installed PRMOL at nominal position + 2 cm East (to clear path from TT2)
• I balanced the table succesfully, first using spirit bubble level and then OSEM levels of BS, SR2, PR3 and LO2.
  • Note, that we need to adjust OSEM positions in many of these SOS before pumping down.
• Input beam from TT2 is going through center of PRM but the reflction is not coming back from PR2, maybe it is missing PR2 or PR2 alignment needs to be adjusted.

[Paco, Ian]

After agreement from Yuta/Anchal, I moved POYM1 yaw to clear the aforementioned path, and Ian restored the POY11 RFPD path. The demodulation phase might need to be corrected afterwards, before any lockign attempts.

Current OSEM sensor values with all the suspensions aligned are attached.
For 'BS','ITMX','ETMX','ITMY','ETMY','PRM','SRM','LO1','LO2', the ones out of the range [200,800] are marked, and for 'PR2','PR3','SR2','AS1','AS4', the ones out of the range [6000,24000] are marked.

[Anchal, Paco, Yuta]

BS Oplev Path

• The changed position of PRM (40m/16863) meant that BS oplev path is getting clipped by the PRM SOS tower.
• We had to move BSOL ~ 16 cm North and ~ 1.7 cm East.
• This means that the BS Oplev input beam is now coming behind TT2 instead of infront of it.
• We also had to align the beam such that input and returning beam are colinear.
• This meant we, had to change the mount of the upstream beamsplitter in the in-air table so that we can use that for separating the return beam.
• Again, we should order 2 inc visible BS for this path.
• Half of the return beam is making its way all the way back to the laser head. I'm not sure if that can be an issue for our oplev loops.
• We kept the SRM Oplev path same using irises on the table.

PRM Oplev

• Again, due to changed position of PRM and BS Oplev, it became very hard to setup oplev for PRM.
• We found a special position which allows us to catch returning beam through the center of the window.
  • But this returning beam is not prompt reflection from PRM, it is reflection of the HR surface.
  • We are hitting about ~5 mm from the edge of PRMOL mirror (because we cannot move the mirror anymore south to avoid clipping BS and SRM input oplev beams)
  • We put in a 1.1m focal length lens in the input beam to narrow the beam on PRMOL so that it doesn't clip
• We did not put any lens for the return beam. The sensitivity of this oplev might be low due to slighlty bigger beam on the QPD than others (SRM, BS). We can revisit and insert a lens later if required.

Interferometer alignment and PRM alignment

• The work on BS table did not change the table balance much. We got back the alignment pretty much instantly.
• We were able to maximize the arm transmissions.
• Then we used a beam card with hole to check for reflection from PRM and used PRM (mostly pitch correction) to get the return beam back in same way.
• This recovered REFL beam on the camera. We used REFLDC signal to align PRM better and maximized it.
• We centered BS, SRM and PRM oplevs after this point.

LO beam mode correction and spatial overlap

• We tried changing the distance between LO3 and LO4 to get a better output LO beam.
• We also tried to swap the LO4 mirror with the spare mirror but we had the same result.
• Eventually, we decided to move LO3 back to East and LO4 to the west edge of the table. This made the beam sizes comparable.
  • Future exercise: We think that LO1 or LO2 might be significantly off-spec in their ROCs which might cause this issue.
  • We should rerun the calculations with the ROC values of LO1 and LO2 written in the datasheets and figure out the correct LO3-LO4 length required.
  • We can make this change in the next vent if required.
• After the beam sizes were looking approximately similar but more iterations of changing length and realigning are required.

Remaining tasks before pumpdown

• Push/pull too bright/dark OSEMs in the SOSs (40m/16865).
• Finish LO beam mode correction and spatial overlap.
• Center all oplevs, note all beam positions on camera, and note down all DC PD values at proper alignment.

[JC, Tega, Chub]

Today we installed the 200 lbs doors on the end station chambers.

[Paco, Anchal, Yuta]

Today, in short we:

• Recovered alignment of arm cavities, PRC (only ITMX aligned), and then altogether with SRM and PRM aligned to maximize all DCPD levels (AS, POP, REFL, TRX, TRY), but SRC was not flashing and the SRM yaw alignment slider was around its max value, so after recording beam positions on cameras Anchal went into the BS chamber and helped steer the SRC alignment using a combination of SRM, SR2 and AS1. After this every beam was nominally aligned except for LO and AS, which remained to be mode matched.
• Mode matched LO3-LO4 by hand --  -- from the ITMY chamber, the final separation between these two mirrors grew by almost 3 inches with respect to the design (!!!) but the LO and AS beams came out nicely. The canonical path used for the steering was LO path, and then we overlapped the beams with the help of a gige basler camera and a couple of DCPDs (Thorlabs).
• Yuta and Paco started running final checks in preparation for Monday (pumpdown). We aligned the IFO, but noted that using Restore/Misalign sometimes results in hysteresis.. so it is not very reliable for fine alignment modes. Then we optimized DC levels, centered all oplevs, and tweaked Green input alignment on XARM and YARM. The XARM was maximized, but in YARM we could still not get high TEM-00 flashing ...
  • Unfortunately, we discovered a slight clipping of the GTRY beam through PR3 which could mean the current alignment (pointing) is not hitting PR3 center optimally.
• Attached are the screenshot of current aligned state after the work tonight, with oplevs centered, and the OSEM sensor values.

[Paco, Yuta]

Prep for closing and pump down.

• Aligned IFO to maximize DC levels.
  • YARM (flashing peak 0.05 with PRM misaligned), XARM (flashing peak 0.06 with PRM misaligned), PRC (PRY flashing -30 @ POPDC, offset -70 and REFL DC 270), SRC (SRY flashing -30 @ POPDC, offset -70), BHD.
• GTRY clipping
  • We tried moving the alignment of PR3, PR2, ETMY, ITMY to reduce clipping and retain IR flashing. We found it kind of difficult, so we only used the unclipped GTRY temporarily to improve the input YAUX injection after which the YAUX locked. We then restored the clipping in favor of the IR beam alignment.
• PR3 position
  • PR3 seems to be +1 inch away towards East, nominally placed along North-South, and offset in YAW.
• Aligned OPLEVs to center at around Mon May 23 13:20:32 2022
• Snapshot of all cameras in the control room around Mon May 23 13:24:51 2022

[Chub, JC, Jordan, Yuta, Yehonathan, Paco]

Closed in the following order:

• IMC chamber
• OMC chamber
• BS chamber
• ITMY chamber
• ITMX chamber

[Yuta, Paco]

After closing the heavy doors, we tried to have GTRY less clipped using PR2, PR3, ITMY and ETMY. During this adventure, we also aligned GRY injection beam by hand. Rotating a waveplate for GRY injection made GRY locking stably at GTRY of ~0.3.

[Vacuum gauge sensors]

Paco informed me that the FRG sensor EPICS channels are not available on dataviewer, so I added them to slow channels ini file (/opt/rtcds/caltech/c1/chans/daq/C0EDCU.ini). I also commented out the old CC1, CC2, CC3 and CC4 gauges. A service restart is required for them to become available but this cannot be done right now because it would adversely affect the progress of the upgrade work. So this would be done at a later date.

[JC, Jordan, Paco, Chub]

We began with the pumpdown this morning. We started with the annulus volume and proceeded by using the following:

1. Isolate the RGA Volume by closing of valves VM3 and V7.

2. Opened valves VASE, VASV, VABSSCT, VABS, VABSSCO, VAEV, and VAEE, in that order.

3. Open VA6 to allow P3, FRG3, and PAN to equalize.

4. Turn on RP1 and RP3, rough out annulus volume, once <1 torr turn on TP3. Close V6. Open V5 to pump the annulus volume with TP3.

5. Re route pumping from RP1 and RP3 to the main volume by opening V3 and slowly opening RV1.

6. After ~3.5 hours the pressure in the arms was <500mtorr on both FRG1 and P1a. Turn on TP1 and wait to reach full speed 560 Hz

7. Open V1 with RV2 barely open. The pressure diff between P1a and P2/FRG2 needs to be below 1 torr. This took a couple attempts with the manual valve in different positions. The interlocks were tripped for this reason. Repeat step 7 until the manual gate valve was in a position that throttled pumping enough to maintain the <1 torr differential.

8. Slowly open the manual gate valve over the course of ~ 1 hour. Once the manual gate valve fully opened, pressure in the arms was <1mtorr.

9. V7 was closed, leaving only TP2 to back TP1, while TP3 was used to continue pumping the annuli. Left in that configuration overnight (see attached)

We did have to replace gauge PAN becuase it was reading a signal error. In addition, we found the cable is a bit sketchy and has a sharp bend. The signal comes in and out when the cable is fiddled with.

I modified the script freeSwing.py to use damping loop output switches to free the optic instead of watchdog or coil output filters. This ensures that the free swing test is being done at the nominal position of the optic. I started tests for LO1, LO2, As2, As4, PR2, PR3, and SR2 in a tmux session names freeSwing on rossa.

Note: LO2 face OSEMs are hardly sensitive to any motion right now due to excessive pitch offset required for LO beam. We should relieve this offset to LO1 and rerun this test later.

[JC, Yuta]

We did a quick health check of suspesions after the pump down.

Summary:
 - ITMX LRSEN is too bright (~761) and not responding to any optic motions (we knew this before the pump down)
 - ITMY ULCOIL is not working
 - LO1 LLCOIL is not working
 - Damping loops need to be retuned, especially for ETMY (too much damping), SRM, PR3 and AS4 (damping too weak)
 - MC1 sensor outputs are minus instead of plus
 - LO2 OSEMs got stuck during the pump down, but now it is free after some kicks. OSEM sensorr values almost came back (see attached)

What we did:
 1. Kicked optics with C1:SUS-{optic}_{UL,LL,UR,LR,SD}COIL_OFFSET one by one with offsets of +/- 10000 (or 100000), and checked if C1:SUS-{optic}_{UL,LL,UR,LR,SD}SEN_OUT16 move in both directions.

 2. Check if the optic damps nicely.

 3. Attached photo of the note is the result.

Finished building power spectrum analyzer for the RFSoC. There are two things that I would like to address down the road. First is that there is an oscillation between positive and negative voltages at the ADC sampling frequency. This creates an undesirable frequency component at the sampling rate. I have not yet figured out the cause of this positive to negative oscillation and have simply removed half of the samples in order to recover the frequency. Therefore, I would like to figure out the root of this oscillation and remove it. Also, we have a decimation factor of 2 as default by the board which we would like to remove but have been unable to do so.

Example: 8 MHz Square Wave from SRL signal generator.

[Yehonathan, Paco, Yuta, JC]

As we were cleaning up this morning, we heard a high pitch sound that turned into a buzz. After searching for where the sound came from, we noticed the CRT TV went out. We swapped this out with a moniter and used a BNC to VGA adapter to display the cameras.

With some help from the forums, we printed the status of the DAC MTS sync and were able to determined that our board's vivado design does not have MTS enabled on each tile. To fix this, we will need to construct a new Vivado desgin for the board. We were also warned to "make sure to generate correctly a PL_clock and a PL_sysref with your on board clock synthesizers and to capture them in the logic according to the requirements in PG269" of the RF Manual. From this we should be able to sync the DAC and ADC tiles as desired.

[Paco, JC]

Paco and I fixed the ethernet cable which was hanging. We stopped models c1x07 and c1su2, realigned the cable to follow the shelf from top, and returned to turn on the computers.

Note: There was not a long enough ethernet cable, so we used a female to female adapter and attached 2 ethernet cables.

[Paco, Yehonathan]

We fixed the slow control over the green beam shutters.

At the Y arm the wrong BNC was connected to the shutter driver. We connected the correct BNC to the driver and switched the remote mode. The green Y shutter now works but in reverese, meaning that sending 1 to C1:AUX-GREEN_Y_Shutter closes the shutter and vice versa. This needs to be fixed.

At the X end the problem was a bit more complicated. Previously, the shutter was controlled by c1auxey. We figured that c1auxex has a lot of spare bio channels. We found an Acromag BNC front panel (with wires already soldered to the BNCs) lying around in the lab and installed it on the c1auxex Acromag chassie. We then connected the topmost BNC to channel 0 on XT1111A in the chassie. The BNC was connected to the green shutter driver on the X end.

EPIC channel was added to the c1auxex db file while it was commented out on the psl shutters db file. Modbus was restarted on c1auxex and c1psl. c1psl had to be burt restored to regain MC lock. Now the green X shutter works properly.

[JC, Yuta]

We replaced a shutter driver for GRY since it stopped working this morning.
We replaced it with a free driver which was sitting on the ITMY table.
The reverse polarity issue of C1:AUX-GREEN_Y_Shutter was fixed by switching one of the switches of the driver from N.O. to N.C.

Also, "Toggle" button was added to IFO_ALIGN.adl so that we can toggle shutters easily to find TEM00. It runs /home/controls/Git/40m/scripts/ALS/ShutterToggler.py.
 

[JC, Paco, Yuta]

We locked both Y and X arms with POY11 and POX11.
RFM fix (40m/16887) enabled us to use triggering using C1:LSC-TRY/X_OUT.
IR beam is now centered on TMs using ASS (for Yarm, ASS loops cannot be closed fully, so did it manually).

What we did:
 - Aligned both arms so that the beams are roughly centered at TMs using cameras.
 - Yarm lock was easy, but Xarm lock required gain tuning. Somehow, Xarm required x3 higher gain as follows, although the amplitude of POX11_I_ERR seems to be almost the same as POY11_I_ERR. I suspect it is something to do with power normalization matrix (TRX flashing is almost a double of TRY flashing).

C1:LSC-YARM_GAIN = 0.01
C1:LSC-XARM_GAIN = 0.03

 - Run ASS for Yarm. ASS loops cannot be closed fully using default feedback parameters. I guess this is because ITMY ULCOIL is not working (40m/16873). ASS demodulated signals were manually zero-ed by manually aligning ETMY, ITMY and PR3 (and some TT1 and TT2), except for demodulated signals related to ITMY. Beam on ITMY was centered just by using our eyes.
 - Run ASS for Xarm. It seemed to work well.
 - After this, TRX and TRY were as follows and beam positions on TMs were as attached.

C1:LSC-TRX_OUT ~0.95
C1:LSC-TRY_OUT ~ 0.58

(TRX is somehow lower than what we had yesterday... 40m/16886; TRX and TRY photodiode alignment was checked, but seems to be OK.)

 - Centered TMs and BS oplevs.

Next:
 - POX and POY demodulation phases are not fully optimized. Needs re-tuning.
 - Tweak GRX and GRY injection (restore GRY PZTs?)
 - Install ETMXT camera (if it is easy)
 - MICH locking
 - RTS model for BHD needs to be updated

AS path at AP table as re-aligned and confirmed that MICH can be locked with AS55 Q.

What we did:
  - Aligned AS55 and AS110 paths at AP table. AS55 was not receiving enough light. AS110 was not receiving light at all.
  - Changed AS55 I and Q whitening gain from 3dB to 42dB.
  - Zero-ed the RF offsets manually. C1:LSC-AS55_Q_IN1 is having too large offset. When PSL shutter was closed, it reads 13950! Needs investigation.
  - Locked MICH with PRM mis-aligned with configurations attached.

Other Issues:
 - C1:IOO-MC_TRANS_SUM is now stuck at 14009. MC auto locker doesn't work correctly. FIX ME!

[Paco]

Fixed the issue below:

by noting that the C1:IOO-MC_TRANS_SUMFILT_OUT was being held to 14009 counts for some reason. Disabling hold quickly let the IMC autolocker act back.

WFS were also turned ON, and there were a couple other control outputs being held on that loop... Strange!

[Paco, JC]

Paco and I attempted to calculate the input matrices from May 24, 2022, but the OSEM data was all saturated and not very useful. Therefore, we decided to manually investigate the appropriate coil offsets for all BHD SUS. Before, the default offset kick was 30000 counts, but we found that LO1, AS1, AS4, and PR2 cannot take more than 5000 counts. As for LO2, SR2, and PR3 cannot take more than 2000 counts before saturating. Note that all these kick test were taken by kicking OSEM UL on all BHD Optics.

We started the freeSwing.py script on tmux freeSwing session for tomorrow at 1:00 am for only the 5000 count offset SUS.

[Paco, Yuta]

On the topic of high AS55_Q RFPD offset, it seems it stems from a small residual offset on top of the 42 dB whitening filter gain (previously 3 dB). We verified this by looking in the past using dtt and seeing an offset of ~ 100 counts, which are consistent with the hotfix. We reverted the whitening filter gain to +24 dB, in order to accomodate the 10% power difference from AS2. We decided to move forward, and try locking MICH using AS55_Q_ERR. The IQ mixing angle was changed to -167 deg from -122 deg to minimize the signal in AS55_I_ERR. We have also added comb60 filters for AS55. The LSC_MICH filter gain was adjusted to -6 (used to be -13 in the configuration script) to get a MICH_OLTF UGF of 90 Hz (which is the previously measured value as of 2021 July), see Attachment #1 for the MICH OLTF estimate.

We then calibrate MICH using the fringe amplitude, so that  , where  is the amplitude of the error point (C1:LSC-AS55_Q_ERR_DQ) in our case ~ 110 +- 2 counts. The calibrated error point spectral density is shown in Attachment #2. Calibration is done into meters in terms of difference between BS to ITMX length and BS to ITMY length.

[Tomislav Andric, Rana, Yuta]

We put -1 to MC1 OSEM sensor gains and re-tuned MC1 damping.
We also kicked MC1, MC2, MC3 tonight for input matrix diagonalization.

MC1 damping investigations:
 We put -1 to MC1 OSEM sensor gains so that UL/UR/LR/LL/SDSEN_OUT will be positive like other optics.
 OSEM damping filter gains were adjusted.
 We have also checked if having +1 for all UL/UR/LR/LL/SDCOIL_GAIN is correct or not. It has been like this at least for the past year.
 It should be -1 for UR and LL to account for magnets, but if we did put -1 or them, kick in C1:SUS-MC1_PIT_OFFSET mostly gave yaw kick and kick in C1:SUS-MC1_YAW_OFFSET mostly give pitch kick.
 So, we reverted them to be +1.

Input matrix diagonalization:
 We also kicked MC1, MC2, MC3 tonight input matrix diagonalization.
 Kick was done manually at the following times local.
  - MC1 20:08 June 6th, 2022
  - MC2 20:24 June 6th, 2022
  - MC3 20:21 June 6th, 2022
 We will leave watchdogs shutdown to free swing overnight (damping loops are "on").
 This will help get better angular sensor from OSEMs to calibrate WFS signals.

Next:
 - Investigate why MC1 coils gains have +1 for all
 - Calculate input matrix. Make sure SUSPOS/PIT/YAW/SIDE_IN will be in the units of um or urad.

Suggestions:
 - Add filter ramp time of 1sec for all by default
 - Make null stream channel from input matrix for diagnostics

Note that MC1 has a new style sat amp because the old one collapsed. The sign flip might have been the result of the replacement

[Paco, Yuta]

We investigated the ITMY ULCOIL issue (40m/16873).
ULSEN is sensing the optic motion but ULCOIL cannot move the optic.
We confirmed that the coil input is there upto satellite amplifier output.
We also checked that ULCOIL have 3.3 mH and 16 Ohms, which are consistent with other coils.
Mystery remains...
We need to investigate ITMY ULCOIL in the next vent.

What we did:
 - Checked again that C1:SUS-ITMY_ULCOIL_OFFSET does not kick ITMY using OSEM sensor signals and oplev signals. ULSEN moves when ITMY is kicked by other coils.
 - Checked that kick gives voltage changes at coil driver and satellite amplifier output. We unplugged J1 DB25 cable from the feedthru flange and checked the signals sent to coil with oscilloscope.
 - Measured inductance (using BK PRECISION LCR meter) and resistance (using Fluke) of coils for ITMY. Below is the result. UL coil seems to be consistent with other coils. (It seems like BK PRECISION one wil give wrong resistance if the dial is set to the resistance value which is too low compared with the one you want to measure. If you want to measure 16Ω, set the dial to larger than 20Ω, not 2Ω)

Feedthru connector: ITMY1
Pin 3-15 / R = 16.3Ω / L = 3.32 mH (UL)
Pin 7-19 / R = 16.4Ω / L = 3.30 mH (UR)
Pin11-23 / R = 16.2Ω / L = 3.31 mH (LL)

Feedthru connector: ITMY2
Pin 3-15 / N/A
Pin 7-19 / R = 16.3Ω / L = 3.30 mH (SD)
Pin11-23 / R = 16.4Ω / L = 3.33 mH (LR)

Discussions:
 - UL is the only short OSEM in ITMY OSEMs.
 - ITMY have dumbells for magnets.
 - If UL magnet is off, ULSEN would not work. Something not magnetic is working for shadow sensing for UL? Dumbells?
 - ULSEN just sensing some coupling from other OSEMs?

Made plots on i/p noise of redpitaya . Need to reconsider sampling frequency (to improve plot at lower freq)
 

Anchal, Paco and I agreed to follow the polarity conventions below for suspension damping loops.
Some of the polarity/gains were changed to homogenize all the suspensions to the convention.
All the suspensions are homogenized except for MC1 (which have all - in sensor inputs and all - in coil outputs) and AS1 (SDCOIL_GAIN have the same sign as LL).

*SEN_GAIN to be +1
INMATRIX to be the following, and calibrated so that SUSPOS/SIDE_IN will be um and SUSPIT/YAW_IN1 will be urad (calibration to be done later)
+ + + + *
+ + - - *
+ - - + *
* * * * +
SUS*_GAINS to be +
TO_COIL gains to be the following
+1 +1 +1  *
+1 +1 -1  *
+1 -1 +1  *
+1 -1 -1  *
 *  *  * +4
*COIL_GAIN to be the following or flipped one so that SUS*_GAINS will be +
+
-
+
-
+

To do this, the following changes were made

For BS, ITMX, ITMY, PRM, SRM, ETMX, ETMY, MC1, MC2 and MC3 ("old" suspensions), TO_COIL_5_4_GAIN (for side) is changed from +1 to +4 and SUSSIDE_GAINs are divided by 4 accordingly.
For ETMX, the sign of SUSSIDE_GAIN is flipped to +, and SDCOIL_GAIN to be -1 (it was +1).
For MC1, *SEN_GAINs are - (not following the convension; see 40m/16894). The sign of INMATRIX_4_5 (for side) is flipped to +, SUSPOS/PIT/YAW_GAIN are flipped to +, and *COIL_GAIN are flipped to - (not following the convension). IMC WFS output matrix components for MC1 were also flipped. 

Since UL coil actuation is lost, we modified the output matrix of ITMY to use only UR, LR and LL face coils for POS, PIT and YAW actuation. The output matrix was changed to following:

After this change, the damping was still working as good as before. I took PIT to POS/PIT/YAW and YAW to POS/PIT/YAW coupling measurements by exciting C1:SUS-ITMY_ASCPIT[YAW]_EXC and seeing effect at C1:SUS-ITMY_SUS[POS/PIT/YAW]_IN1 when the damping loops were off. Attached are the results. We were able to reduce PIT to YAW and YAW to PIT coupling by 10 dB by this simple change in output matrix. More coil balancing or off-diagonal termsmight help more and should be attempted if required. The coupling to POS did not change much.

Note that attachment 1 shows transfer functions from excitation point to the DOF sensing inputs while attachment two looks at ratio of C1:SUS-ITMY_SUS[POS/PIT]_IN1 to C1:SUS-ITMY_SUSYAW_IN1 which is the actual quantity of interest. I didn't repeat the PIT measurement due to lack of time.

Also note that all such measurements are being recorded in our new measurements git repo. We'll populate this repo with diaggui template+data files as we do measurements.

=== Observation ===

• TP1/2/3 failed. AUX RP did not survive the power outage (of course)

=== Initial Recovery of TP2/TP3 ===

• Stopped TP1/2/3
• Restored RP AUX at the wall breaker module
• Restarted TP2/3

=== Rough pumping down ===

• Found RP1/RP3 left connected (unusual)
• Turned on RP1 and RP3
• Opened V6 and V7 for roughing of the pump spool. Now 0.3 torr for P2/P3/PA* (annuli) - ready to open V4 and V5 to backup TP1 by TP2 and TP3. Waiting for TP2/3 to be accelerated

=== Towards main volume pumping ===

• TP2 accelerated and opened V4 (This caused V6 to close). RP1 and RP3 stopped.
• TP3 hasitated to spin up. Stopped for now.
• TP1 needed to be reset to clear the displacement errors. Once the errors were cleared, TP1 was restarted (acclerating).
• TP1 at normal speed. Opened V1 (main volume pumping started)
   
• Disconnected RP1/RP3 line.
   
• TP3 was restarted and is now stable at 55k RPM (indicated as soft spin / low speed). Pushed "Stand By On" and "then "Stand By OFF". This makes the TP3 back to "NORMAL". Switched to "Stand By On".
• TP2 was also switched to "Stand By On".

Now the main volume and the annuli are pumped down TP1 and TP2/TP3 with RP AUX backing.

The attachment is the pressure glitch for the main volume.

I found the HVACs for the ends were off. They were turned back on.

The IMC alignment was restored and the IMC is nicely locking.

Once the vacuum level recovered P<1e-4 torr, the PSL shutter was able to be opened.

The IMC was still flashing, so the lock to TEM00 was possible.

Once it was locked, the MC2 alignment was tweaked and the autolocker and the WFS kicked in to help the locking/alignment.

The transmission is ~13k and seems reasonable considering the low PMC transmission of the PMC (0.672)

To see if the ULCOIL channel of the ITMY coil driver is working or not, I swapped ITMY coil driver and ITMX coil driver by swapping DB15 cable (see Attachment #2).

With this swap, I confirmed that ITMX can be kicked with C1:SUS-ITMY_ULCOIL_OFFSET, but ITMY cannot be kicked with C1:SUS-ITMX_ULCOIL_OFFSET (see Attachment #1).

This means that the issue is not the in-air electronics.
Mystery remains again...
We need to investigate ITMY ULCOIL in the next vent.

I revereted the swap and confirmed that damping loops work fine again.

what was the result of the inductance measurement? should be ~3.3 mH as measured from the flannge or cable that goes to the flange from sat amp.

ITMY ULCOIL was measured to have ~3.3 mH as measured from the flange. RTFE 40m/16896 .
 

I have at taken photos and added arrows which signify the beam paths for ITMX, ITMY, and Vertex Oplev tables.

The servo gain for MC1 in /opt/rtcds/caltech/c1/Git/40m/scripts/MC/WFS/reliefWFS was flipped to account for COIL_GAIN flip done in 40m/16898.
The reliefWFS script now works fine.

ezcaservo -r 'C1:SUS-MC2_ASCPIT_OUT16' -g ${g} -t ${ts} C1:SUS-MC2_PIT_COMM & 
ezcaservo -r 'C1:SUS-MC2_ASCYAW_OUT16' -g ${g} -t ${ts} C1:SUS-MC2_YAW_COMM & 
ezcaservo -r 'C1:SUS-MC1_ASCPIT_OUT16' -g -${g} -t ${ts} C1:SUS-MC1_PIT_COMM &  
ezcaservo -r 'C1:SUS-MC1_ASCYAW_OUT16' -g -${g} -t ${ts} C1:SUS-MC1_YAW_COMM & 
ezcaservo -r 'C1:SUS-MC3_ASCPIT_OUT16' -g ${g} -t ${ts} C1:SUS-MC3_PIT_COMM & 
ezcaservo -r 'C1:SUS-MC3_ASCYAW_OUT16' -g ${g} -t ${ts} C1:SUS-MC3_YAW_COMM &