[Paco, Anchal, JC]

C1:PSL-PMC_PMCTRANSPD ~ 0.715 this morning, this was increased to ~0.730. There also seems to be an earthquake going on and the MC is flashing.

C1:PSL-PMC_PMCTRANSPD was increased from 0.72 to 0.731

[JC, Paco, Yuta]

It seems like PMC transmission (C1:PSL-PMC_PMCTRANSPD) dropped to ~0.68 from ~0.74 on Dec 27.
We tried to tweak PZT offset for PMC loop and input alignment to PMC, but PMC transmission didn't increased.
PSL laser temperature was also sweeped in the range 30.3 - 31.6 degC, but didn't help. The PSL temperature was reverted to original 30.61(1) degC.
Power measured at PSL output now is 893 mW (measured at our standard place shown in 40m/16672), which used to be 951 mW in June 2022 (40m/16886).
Power measured at PMC input (see attached photo) now is 1.18 W.

Next:
 - What was the previous PMC input power we had?
 - Sweep PSL laser temperature for larger range

PMC aligned (Attachment #1).
Over the past month, PMC transmission is actually slowly growing from 0.68 to 0.70 (Attachment #2), since it suddenly dropped from 0.72 on Dec 27 (40m/17390).

PMC hit a drop around mid-day on Saturday and has been going down since. As of now,

C1:PSL-PMC_PMCTRANSPD - 0.664

I will go in and adjust now.

This is super! And now is the time to replace the internal fan!

We removed the PSL controller internal (broken) fans after it tripped due to overheat.

Background

[Mayank, Radhika]

While aligning Xarm I noticed sudden loss of beam. On Radhikas suggestion we cheked the PSL and found out that the PSL controller was in off state (No lights on front and back panel). We restored the situation by unplugging and replugging the power cord. The PSL worked fine for a few minutes (~ 30 ) and then tripped again.This time the front panel  OFF light was on . See attached image (Attachment #1).

Fix

[Paco, Mayank, Koji-remote]

We disconnected the PSL controller and took this opportunity to investigate the controller's internal cooling mechanism. After disassembling the top panel of the chassis, we saw there are two SUNON - KD1205PHB2 fans meant to run at 12 VDC (1.7 W) connected to the bottom pcb inside the controller. After disconnecting them from this board, we tested them with an externally supplied dc voltage and confirmed they no longer worked. We noted the cooling mechanism is based on a long aluminum heat sink to which most ICs are attached, and the fans are meant to provide airflow towards the rear aperture on the chassis. We followed Koji's suggestion and for now, removed the damaged components (detailed pictures of this operation have been posted in a google photos album elsewhere) to allow heat to flow out more easily.  We reassembled the controller chassis and reinstalled it with the external fan providing the necessary airflow to prevent the unit from tripping again due to overheating. Then we turned on PSL and recovered PMC and IMC locks.

Aftermath

We took a C1:IOO-MC_F_DQ trace after this work to confirm our earlier findings; the trace is attached in Attachment #2. The noise bumps are present as expected. This is still not a desirable configuration so next step would be replacing the external fan, or even better, find the appropriate spare of the internal units and berid the external one.

We Changed The Fans on the PSL

To start, the part Koji ordered is 259-1818-ND from DigiKey. This is a Maglev fan from SUNON that should give us less noise. We have 3 spare replacement fans in case these go bad which are stored in the ___ Cabinet along the Y arm (This will be updated once I find a suitable storage spot for the part.)

Starting the replacement process.

Removing the PSL 

    1. Our first step to doing this was to prepare for removing PSL. We began by doing a 60s MC WFS Relief. This will allows us to turn off WFS and close the PSL Shutter next. This is to prevent a large kick once we place the PSL back in its place. 
    2. Went to the PSL and mushed the off button and turned the key. After this, begin by removing the external fan which is shown in elog 17595. After, continue by unplugging the cables from the back beginning with the power cable. Attachment #1 shows the original positions of the cables connected before removing any. Keep in mind, DO NOT TO TOUCH THE KNOBS. If the inputs are changed, this will throw off the beatnotes of the AUX lasers.
        a. There is a black plug at the bottom with a screw hat is hard to reach. Be very patient taking this off because the position of the cable blocks a screqwdriver from untightening the screw. 
        b. A second person should be on the other side of the table to push the module back into arm's reach. Also to make sure the module does not slide back and fall. 
    3. After removing the module, bring into the control room and place onnto the workbench. Make sure all of the red lights are off and the PSL table is closed properly.

Changing the Fan 
    
    
    1. We removed the top cover of the module and opened it all up. Similar to what is shown in elog 17452.
        a. Keep in mind to wear a grounding wristband when working on this.
    2. After removing the old fans and attempting to install the new ones, the holes did not line up correctly (Shown in Attachment #2). To accomodate for this, we used 6-32 screws which gave us just enough slack to fit in all 4 corners.
    3. Ones the fans were bolted down onto the aluminum plate, I soldered the cables to connecting the fan cable to the cables those of the original PSL fans.
    4. Next I used heat shrinks to cover the bare soldered areas and placed the fans into the module. 
    5. We tested the fans by plugging intp the PSL and turning the key. The fans turned on nicely and we proceeded to put it module back together.

Placing the PSL back in its place. 

    1. We place the PSL back into its original spot and began to connect the cables. Make sure the Power cables is put in LAST. 
    2. After the module was put back, we DID NOT put the external fan back into its place. This is to see if the fans which were installed are good enough to maintain the PSL.
    3. Turn the key and press the on Button.

The noise from the external fan is no longer appearing as shown in attachment #3. The PSL has been on for ~2 hrs now and has not turned off. It seems that the fans are doing their jobs well. 

I found that the MC transmission was low (~13000. Usually it's ~13500) in the morning.
This was due to the low transmission of PMC (~0.675. Usually it's ~0.685), so I restored the PMC alignment using the two steering mirrors before the input mirror.
The transmission of MC and PMC was restored to ~13300 and ~0.687.

[Yuta, Hiroki]

Remote switching of PSL shutter is not working.

When we tried to reset the LSC offsets, we found that the remote switching of the PSL shutter was not working.
We closed the shutter manually by using the toggle switch on the front panel of the shutter driver (N.C. -> N.O. (Attachment 1)) for the momment.
 

[JC Koji]
c1vac was hard rebooted. This fixed the PSL shutter issue.

• Checked the Unibritz Shutter Controller for the PSL shutter. It seems that it responds to the NO/NC switch no matter how the REMOTE/LOCAL switch is. (Normal action)
• Checked the BNC input to the shutter controller. There was no trigger/gate or whatever signal from Acromag.
• c1psl reboot did not help
  • Upon this action, a snapshot was taken.
  • Stopped/restarted modbusIOC.services. Did not work.
  • Rebooted c1psl. Didn't work.
• c1psl was shutdown and c1psl acromag chassis was power-cycled. This didn't help. c1psl was burtrestored.
  • Stopped modbusIOC.services. Shutdown c1psl.
  • Power cycled the Acromag chassis.
  • After a while, c1psl was turned on. The epics channels came back.
  • Confirmed modbusIOC.services was running.
  • Burtrestored the snapshot.
• I remembered that the vacuum pressure could make an interlock action on the PSL shutter.
  Found that c1vac values were all whited out. c1vac was not reachable from Martian.
• Went to c1vac and found it was frozen. Pushed hard reset.
• c1vac came back, and it closed the gate valve V1 (main GV for the main vacuum manifold). We opened V1 from the vac control screen.
• Confirmed the PSL shutter is now operational.

After the exploration of c1psl, the IMC locking was not functioning well. It looked like the MC autolocker issue.

Now I remember that the autolocker was running on docker. I followed the instruction on the wiki page. https://wiki-40m.ligo.caltech.edu/Computers_andScripts/AlwaysRunningScripts

Remember: Docker is running on optimus / systemctl is running on megatron!

Then: I noticed that MC autolocker heartbeat was blinking even with the docker stopped (!?). I confirmed that autolocker is not running on systemctl

I had no hope who is toggling the heartbeat, but "ps -def" on megatron showed me that "AutoLockMC_LowPower.csh" is running! I was afraid that it is a remnant from vent!? But, is that possible?

The process was killed and the docker locker is running well now.

For monitoring the statuses of the PSL HEPA fans, we are planning to attach flow speed sensors (D6F-V03A1) on the HEPA fans.
Before installing the flow sensors, I have finished the followings:

• Tested two flow sensors with alligator clip wires (Attachment 1)
  • No flow: ~ 0.5 V
  • Saturation: ~ 2.7 V
     
• Measured the flow speed of PSL HEPA fans with an alligator-clip-wired flow sensor
  • Medium speed: ~ 0.8 V (~ 0.6 m/s )
  • Maximum speed: ~ 1.4 V (~ 1.8 m/s)
    The flow speed [m/s] was estimated from the linear approximation of the nominal value
     
• Wired flow sensors to install them in PSL and to take the output voltage by an Acromag ADC (Attachment 2)
  • Wired sensors worked as in the case of the alligator-clip-wired sensors.

Next:

• Summarize the details of the used parts and the wiring.
• Install the wired flow sensors in PSL. Monitor their outputs from CDS.

[Paco, Hiroki]  

Installed two flow sensors on the PSL HEPA fans 

To monitor the flow speed of the PSL HEPA fans, we installed two flow sensors:

• Attached two flow sensors on the ceiling of PSL table and connected their outputs to the Acromag ADC:
  • Attachment 1: Connected DB37 to the Acromag ADC
  • Attachment 2: Wiring from the Acromag ADC to the PSL table
  • Attachment 3: Attached flow sensors on the ceiling of the PSL table
  • Attachment 4: Wiring diagram
     
• Changed the flow speed of HEPA fans and monitored the signals from the corresponding sensors with StripTool (Attachment 4):
  • ​Sensor 1:
    • Minimum speed: ~ 0.52 V 
    • Medium speed:   ~ 0.78 V 
    • Maximum speed: ~ 0.83 V 
  • ​Sensor 2:
    • Minimum speed: ~ 0.50 V 
    • Medium speed:   ~ 0.72 V 
    • Maximum speed: ~ 0.83 V 

The results above are non-linear. This might be due to the non-linearity of the flow sensors and the non-linearity of the controllers for HEPA fans.
*Regarding the HEPA fan 1, which the sensor 1 is attached on, there was slight flow even when the speed was minimized. This is why the minimum output of sensor 1 was ~ 0.52 V and not 0.50 V.

Next:

• Calibrate the Acromag ADC by injecting signal from a function generator.
• Calibrate the measued voltage to the flow speed (with linear approximation?) and show them on the CDS screen.

[JC, Koji]

In the morning, JC came in and found that c1sus had crashed. JC tried to save the situation by rebooting the host, inevitably, all the CDS machines needed to be rebooted.
But it was successful! Thanks, JC!

The remaining issue was that the PMC was in a strange state.
- The transmission was low (0.45~0.5V vs 0.67~0.68V nominal)
- The input alignment didn't help much
- The refl spot was not well centered on the CCD

We jiggled the setting for a while and found that the PMC error input offset needs to be significantly increased from the nominal value.
In fact, the offset hit the end of the range (+10V), this made the transmission to be ~0.67, but there may be slight room to improve it.
To investigate this, we need to look at the analog error signal, but I don't have enough energy to look into it today.

After the PMC recovery, the IMC was locking already, and the arms were flashing. I took the alignment of both arms.

[Koji, Hiroki] 

Calibrated flow sensor signals from [volt] to [m/s] and displayed them on CDS screen

We added the EPICS calc records to /cvs/cds/caltech/target/c1psl/psl/pem.db (Attachment 1) and made two channels for the calibrated signals of the flow sensors as follows:

C1:PEM-HEPA_FLOW_PSL1_CAL : Calibrated signal of flow sensor 1 [m/s]
C1:PEM-HEPA_FLOW_PSL2_CAL : Calibrated signal of flow sensor 2 [m/s]

At first, we tried to calibrate the output signal by using the 6-order approximation written in the user's manual (P11, D6F-V03A1) but the resulting outputs became 0 and we failed.
The cause of this failure might be due to the large number of the calculation of the 6-order approximation.
Then we used 5-order approximation that was fitted to the 6-order approximation and succeeded in obtaining the calibrated signals.
The used 5-order approximation is as follows:

    v: output voltage from flow sensor [V]
    A=5.11938
    B=-29.7601
    C=68.6169
    D=-78.4301
    E=46.0640
    F=-10.2767  

Attachment 2 shows the time series data of the raw signals and the calibrated signals when the flow speed was changed in minimum - middle - maximum.
Attachment 3 shows the CDS screen displaying the calibrated signals.

1) Lock it down.
2) Turn it sideways. Use the leveling screws to center the bubble level.
3) Carefully loosen the hanger rod and release slowly the tension to allow
   the mass to recenter.
4) Look through the little viewhole next to the rod to make the white lines
   line up. This means the mass is centered.
5) Look at the output on a scope. It should be freely moving with a ~1 sec.
   period.