I measured the Moku:Go and Moku:Pro delay using a Agilent 4395A network analyzer. I considered the PID controller (0 dB gain); the IIR filter box with a 2nd-order low-pass filter; the FIR filter box with 2 coefficients and 201 coefficients (both low-pass). The current XAUX laser lock is done with a Moku:Go using the IIR filter box, so we would expect ~12 µs of delay.

Basic Idea

The idea behind this study was that in several frequency ranges, there are spikes, or just dominant features, in the ASD of each of the sensors. I’ll model these noise sources as a signal  where  is a Nx1 matrix that describes how the single noise source couples into each sensor. Lets assume that this noise source is the dominant source of noise power in the sensors between  and . Then the band limited covariance matrix will have the following form.

 is the integral of the PSD of the noise source over the frequency band. We can use SVD of  to calculate the result of the PCA.  . Since  is a Nx1 dimensional matrix,  is a 1x1 matrix,  is a Nx1 matrix with a the singular value in the first component and 0’s in the rest, and  is a NxN matrix. The first column of  is equal to  up to a scalar. The result of the columns of  for an orthogonal basis for the space perpendicular to . Therefore, for prominent features in the ASD of the sensors, we can "zoom in" and do a PCA. The first principle component will tell us how the noise source is coupling into the sensor vector space. The rest of the principle components will define subspace orthogonal to the noise source. Therefore a virtual sensor designed in this orthogonal subspace will avoid the noise spike.

Test with simulated noise

The first test I did was to choose some S_n at random and a noise source ASD of  with = 1KHz. I started with the CSD of the unsupressed sensor data, , and calculated the CSD with the new simulated noise: . Attachement 1 shows the ASD of the unsuppressed sensors with this added noise. The second figure is zoomed in on the noise peak. The results of the PCA analysis are summarized by attachment 2. The first figure in attachment 2 shows the ASDs of the virtual sensors formed by the principle components of the PCA. The second figure zooms in on the 1Khz feature. The solid blue curves are the ASDs of the sensors and the dotted red lines are the PCA virtual sensors. This figure shows that the first PCA sensors clearly contains the noise spike but the other N-1 (7) PCA sensors are now flat over this frequency range.

Test with real noise

Attachements 3 and 4 show the same anaylsis done on a real noise feature in the sensors ASDS at ~180Hz and attachments 5 and 6 show the same for a ~640 Hz feature. Both tests show results consistent with the simulated noise test. These are the only tests I've performed thus far so I haven't found a feature yet that doesn't play well with this analysis. 

Conclusions

Virtual sensors could be designed to avoid particular noise spikes this way but a more optimal sensor could avoid multiple noise spikes by transitioning, as a function of frequency, between the othogonal subspaces defined by the PCA. Also, ths technique provides a measure of  up to a scaling. Perhaps this can allow for the origin of these noise features to be identified. 

Summary

The problem in the PMC Frequency Reference Card (35.5MHz) was fixed.
The last opamp stage of the attenuator control (LT1125) had the negative rail, and the EOM drive level was maximally attenuated.
The chip was replaced, and the card was back in the euro crate. The PMC is locking as before.

Diagnosis

Yesterday, I already found that the modulation level was basically zero and had no response to the level slider. Today, the card was extracted from the rack and tested on the workbench. It required +/-24V supplies for the main power and +10V supply for high-power RF amps (SMA-202). Additionally, a function generator to supply a DC voltage is needed to tune the attenuation level manually. (Attachment 1)

Some notes:
- This circuit was previously fixed in 2015 by Yutaro and me: [ELOG 11763]
- Even before that, Jenne and Rana looked at the board for LO fix in 2014 [ELOG 10160]
- The boards DCC number is D980353 however the 35.5MHz version is D000419 and the 40m version has a dedicated DCC number D1400221

Initial look was horrible because some greasy material was covering some parts on the boards (I had the same thing in 2015 too). It turned out that it is a leaked thermal paste from the heat sink at the bottom side. (Attachment 2)

Firstly, the suspicious ERA-5SM was checked. The attenuator control path was suspiciously dead (-5V) but the function generator was attached to the attenuator control pin, the ERA-5SM in the EOM path received the signal and it was working fine(!). The ERA-5SM received 20mVpp and spit out 270mVpp, observed with a 1/10 probe. The LO path one was also alive, receiving 173mVpp_max and yielding 200mVpp.

Then, the high-power amps (SMA-202) were checked. This is an old chip which is not commercially available anymore. So I thought I would be reluctant to replace it was broken. But the amps were just fine. With the above condition, the LO and PC paths' outputs were ~9Vpp and 6.1Vpp.

Now I went into the attenuator control part. And it turned out that the last stage (low pass filter part) was broken. By replacing the LT1125, the attenuator control chain recovered the function. (Attachment 3)

Calibration

The attenuator control voltage was applied to the board, then this control voltage was measured together with the RF output powers (LO and PC). (Attachment 4)

LO power is about 16dBm~17dBm and almost independent with the setting (as expected).
PC power changes from -24dBm to +25dBm.

Attachment 5 is the relationship between the PC RF power and the MODEL Voltage measured in analog.

Restore the setting:

The module was returned to the rack. One thing we should take care of is that the external 10V should be disconnected when the output is not terminated with 50Ohm loads. This is to protect SMA-202 from reflection damage.

Once the board was secured and the 10V supply was connected, the MODET voltage was dependent on the slider setting. The PC output RF power was calibrated against the RFADJ setting. (Attachment 6) It is consistent with the above analog measurements.

The RF level (C1:PSL-PMC_RFADJ) was set to 6.0. This imposes the PC output of +14dBm. C1:PSL-PMC_MODET came back to ~-0.34, which is consistent with the number before the trouble.
I also checked the LO power in situ. The direct output was ~17dBm and the 9dB attenuator made the LO level down to 8dBm. The mixer is ZAD-6, which is 7dBm LO level. So it looks fine.

PDH error signal

The amplitude of the PDH error signal was observed at the IF output of the frequency mixer. The cavity was swept around one of the resonances. It showed a clear PDH error shape with the P-P amplitude of ~130mV. (Attachment 7)

By the way, the PMC error offset slider was swept while the cavity was locked. The error signal indicated

The error signal / Offset slider value
-5.23mV / +10V
-6.11mV / 0V
-7.03mV / -10V
It seems that there is a 1/10⁴ reduction factor between the slider value and the actual applied voltage.

[JC, Yuta]

PMC was unlocked from last night, so we aligned PMC
c1sus2 crashed again during the PMC alignment, so we ran

./opt/rtcds/caltech/c1/Git/40m/scripts/cds/restartAllModels.sh

We burt restored to 2023/Aug/14/16:19 by

./opt/rtcds/caltech/c1/Git/40m/scripts/cds/burtRestoreAndResetSUS.sh /opt/rtcds/caltech/c1/burt/autoburt/snapshots/2023/Aug/14/16:19

Update

In the past 2 weeks I have finalized the pipeline for hyperparameter testing and speeding up the training dramatically. Here is a rundown of the most recent developments:

1. I have pushed a lot of code in the git repo with pipeline for training models using the TF Agent library. Now, all of the models' hyperparameters are specified via config files (config_ppo.yaml), allocated resources on HPC and starting a new run are managed via a bash script (start.sh), and a basic plotting functionality now comes built-in for each model run for easy plotting and evalution of the trained policy.
2. I have increased the speed of the code by making use of more advanced training methods from TF Agents: Compared to PPO_v5.py which uised to take around 25 seconds per iteration, PPO_v6_2.py takes 4 seconds per iteration (given the same training parameters). This allows not only for very fast training, but also much faster debug.
3. I have managed to train a model that effectively solved the environment with constant T_env and no measurement noise. Here is a plot of the policy applied for one episode:
   
   This particular plot comes from the policy of RLCON-60 run, which ran for 12.5 hours, but which seems to have reached "peak" performance in about 2-3 hours judging by the evaluation curve (from neptune.ai):
   
   I know this plot is not great, but basically the plateau has a return between -11064 and -11061 (so effectively constant). The evaluation runs are done 50 interations from each other, which in the case of this run corresponds to around 40 mins of training. Please feel free to check out more details about RLCON-60 at the provided link.
4. I have written an extensive manual in the same git repo with guidance as to how to use the code, and also a mini tutorial on Reinforcement Learning and TF Agents. The manual is not exhaustive and I am still adding to it regularly, but I am always happy to answer questions about the code (I am particularly active in the git issue tracker for the repo).

Moving on, I am ready to start making the environment more complicated: I now need to add things like varying T_env, measurement noise, heating delays, etc. Luckily, these changes can be done independently in PuckEnv.py, allowing us to just use the same code for training once the environment is up and bug-free. For testing purposes of the environments, I also have the env.ipynb, although it is not as streamlined as the rest of the code.

Elog was continuously accessed by net crawlers or something similar. They continuously retrieve random logs and also randomly hits the entries with invalid attachments. The invalid attachments call "convert" to make thumbnails, but fail. Because the crawlers fail to obtain the thumbnails, they try the same URL again, and again.

I decided to setup the firewall (NAT router) to block specified IPs. This was very much successful and made the ELOG lightning fast.

Blocked Spam IP addresses are listed in an IP group in the NAT router setting. It's easy to add more IPs there.

The current setting and some instructions are found on a wiki page: https://wiki-40m.ligo.caltech.edu/FirewallSetting

I went down to the IFO hall and checked the PMC situation. The freq mixer fixed on the rack 1X1 had BNC-T, so the raw error was checked while the PMC was swept around the resonance.
The error signal was ~1mVpp while there is ~1mV offset.

There are a few possibilities: modulation, beam, PD, demodulator, or something else.

I checked if the PD was busted, misaligned, unpowered, etc, but there was no clear sign of what was wrong.
The beam was not well steered on the PD but realigning the PD didn't help the size of the error signal.

Modulation... a possibly high-power RF amp is usually delicate so without consulting with the circuit schematic, I did not want to disconnect the output from a proper load.
Instead, I just checked the modulation level monitor on the PMC Phaseshifter screen. C1:PSL-PMC_MODET said 0~-0.004, which didn't mean anything.
By the way, the LO output had ~1Vpp observed with 50ohm termination on an oscilloscope. That looked OK.

But the history of this channel told us that there was some negative number, which we lost ~3.5 days ago. So it's suspicious.

I checked the circuit schematic of the oscillator box D980353. As I said, the LO was OK. So the oscillator would also be OK. Maybe the EOM path has something.
There is a notorious ERA-5SM in the path. So the first thing to check is this chip. Of course, we have many spares in the blue tower.
 

1. Well, it seems that the PMC error offset issue still remains.

2. The IMC transmon QPD had been aligned for some reference e.g. Max transmission, Center of the MC2 mirror, lowest noise, or else. It's not arbitrary.
    Find the appropriate spot poisiton and then align the QPD for the given spot.

{JC, Yehonathan}

When JC came this morning the IMC was completely misaligned. After tweaking the MC alignment we realized the MC WFS where comletely off the rail. We cleared their history and returned the suspensions to their original position. MC locked immediately.

However there still issues:

1. When WFS where turned on it (now its at 8.194) would slowly misalign the IMC.

2. PMC would get unlocked very quickly everytime the IMC tweaked due the PMC input offset being at the slider's edge at 10V.

First we dealt with the PMC issue. We brought the offset down and tried locking the PMC. When it didn't work we went to the PMC table and connected a triangular wave to EXT DC in the PMC servo. PMC reflection and tansmission where observed using a scope while triggering on the triangular wave.

We changed the input volatge until we could see transmission peaks during the voltage ramp. We aligned the PMC so that the HOMs are minimized. We were then able to lock the PMC at an offset input voltage of 8.2V. Transmission is at 0.67V REFL is at 0.03V.

WFS Servo fix:

We went to the AS table and centered the WFS. We run correct DC and AC WFS offset scripts. When then we locked the IMC and turned on the WFS servo. This time the WFS didn't rail, at least not for the last 1 hour. We also realigned MC2 transmission QPD.

[Koji, Hiroki] 

Calibrated flow sensor signals from [volt] to [m/s] and displayed them on CDS screen

We added the EPICS calc records to /cvs/cds/caltech/target/c1psl/psl/pem.db (Attachment 1) and made two channels for the calibrated signals of the flow sensors as follows:

C1:PEM-HEPA_FLOW_PSL1_CAL : Calibrated signal of flow sensor 1 [m/s]
C1:PEM-HEPA_FLOW_PSL2_CAL : Calibrated signal of flow sensor 2 [m/s]

At first, we tried to calibrate the output signal by using the 6-order approximation written in the user's manual (P11, D6F-V03A1) but the resulting outputs became 0 and we failed.
The cause of this failure might be due to the large number of the calculation of the 6-order approximation.
Then we used 5-order approximation that was fitted to the 6-order approximation and succeeded in obtaining the calibrated signals.
The used 5-order approximation is as follows:

    v: output voltage from flow sensor [V]
    A=5.11938
    B=-29.7601
    C=68.6169
    D=-78.4301
    E=46.0640
    F=-10.2767  

Attachment 2 shows the time series data of the raw signals and the calibrated signals when the flow speed was changed in minimum - middle - maximum.
Attachment 3 shows the CDS screen displaying the calibrated signals.

[JC, Koji]

In the morning, JC came in and found that c1sus had crashed. JC tried to save the situation by rebooting the host, inevitably, all the CDS machines needed to be rebooted.
But it was successful! Thanks, JC!

The remaining issue was that the PMC was in a strange state.
- The transmission was low (0.45~0.5V vs 0.67~0.68V nominal)
- The input alignment didn't help much
- The refl spot was not well centered on the CCD

We jiggled the setting for a while and found that the PMC error input offset needs to be significantly increased from the nominal value.
In fact, the offset hit the end of the range (+10V), this made the transmission to be ~0.67, but there may be slight room to improve it.
To investigate this, we need to look at the analog error signal, but I don't have enough energy to look into it today.

After the PMC recovery, the IMC was locking already, and the arms were flashing. I took the alignment of both arms.

[Hiroki, Paco, Yuta]

As mentioned in elog #17768, we are trying to lock PRFPMI.
To start with a simpler system step by step, we locked FPMI by revising the script for automated locking on Aug. 10th.

Revised automated locking of FPMI:

At first we tried to lock FPMI with the existing script and failed locking because of the changes of some gains.
We revised the script for the automation with the procedure explained in the Supplementary and succeeded in locking FPMI.
The script is easily accessed from the LSC - Actions submenu.
Locking configurations are saved in /opt/rtcds/caltech/c1/Git/40m/scripts/LSC/FPMI-AS55_REFL55.yml. 
The locking procedure by the script is as follows:

• Lock XARM and YARM with CARM and DARM using POX11_I and POY11_I.
• Lock MICH with REFL55_Q.
• Hand off the error signals of CARM and DARM from (POX11_I, POY11_I) to (REFL55_I, AS55_Q).

Attachment 1 shows the screenshot of the locked FPMI with the script above.
Attachment 2 shows the OLTF of MICH. UGF ~ 30 Hz.
 

Supplementary

Gain tuning of CARM and DARM with REFL55_I and AS55_Q:

Firstly, we used the existing script (/opt/rtcds/caltech/c1/Git/40m/scripts/LSC/FPMI-AS55_REFL55.yml) to lock FPMI and succeeded in locking with CARM and DARM by (POX11_I, POY11_I).
However, we failed in locking FPMI with CARM and DARM by (REFL55_I, AS55_Q).
So we measured the OLTF of CARM and DARM by (POX11_I, POY11_I) and that by (REFL55_I, AS55_Q) while FPMI is locked with (POX11_I, POY11_I) and REFL55_Q to see the difference of the OLTFs.

Attachment 3 shows the CARM OLTFs by (POX11_I, POY11_I) (red) and REFL55_I (blue).
UGF of POXY_CARM：~200Hz
UGF of REFL55_CARM: ~220Hz
Gain ratio at 200Hz: (POXY_CARM/REFL55_CARM) = (1.04/1.14) = 0.915

Attachment 4 shows the DARM OLTFs by (POX11_I, POY11_I) (red) and AS55_Q (blue).
UGF of POXY_DARM：~150Hz
UGF of AS55_DARM: ~90Hz
Gain ratio at 150Hz: (POXY_CARM/REFL55_CARM) = (0.9972/0.76)=1.31

In the existing script, CARM and DARM by (POX11_I, POY11_I) are assinged to CARM_A and DARM_A, and CARM and DARM by (REFL55_I, AS55_Q) are assinged to CARM_B and DARM_B.
To match the OLFTs of A and B above, we modified the following gains in the script using the results from the OLTFs measurement above:

CARM_B_GAIN: 1 -> 0.92  
DARM_B_GAIN: 1 -> 1.31

With this modification, we succeeded in locking the FPMI with (REFL55_I, AS55_Q).

Tips on aligning FPMI:

After locking the FPMI, you may find a higher order mode in the AS camera and the ASDC is not fully minimized.
If you misaligned ETMX and ETMY just before locking FPMI, the higher order mode may be produced by the misalignment in pitch for ETMX and ETMY because Misalign -> Restore command in the IFO window does not ensure the perfect restoration.
So you may minimize the ASDC by aligning the pitch of ETMX and ETMY.

[JC, Koji, Yuta, Hiroki]

When we tried to measure the spectrum of C1:LSC-DARM_IN2 with the diaggui, the following error was occured and were not able to measure the spectrum:

"Unable to obtain measurement data" 

We solved this issue by clearing the test points in c1lsc with the following commands:

~> diag
diag> open
diag> tp clear 42 *

"42" corresponds to the DCUID of c1lsc and "*" specifies all the test points.

Details of Clear command: tp clear node# tp#
node# = dcu# of the RTS (see CDS status screen to find the DCU #)
tp# (see CDS status screen to find the TP #)

I just wanted to take the same time series data I took back in 2012 (40m/6874).
ALS noise look much better than 2012, but MICH contrast during both arms hold on IR resonance with ALS looks pretty bad compared with 2012, which indicate unbalance of the arms.

[Radhika, Yuta, Hiroki, Paco]

During YAUX noise measurements, we also locked IR and green lasers in xarm (using Moku:Go lock-in amplifier + digital controller) to look at ALS noise in XARM. I adjusted the controller gain until green transmission looked tightly controlled (less fuzzy). We measured the out-of-loop ALS X beat note fluctuations at 2 gain levels: -12 dB and -14 dB down from the uPDH box fitted response. These are Attachments 1 and 2 respectively.

Note that there was some mirror motion at 1 Hz that is reflected in the spectral densities (coil balancing of ETMX had just been taking place). The -12dB gain adjustment causes frequency noise ~3x higher than the reference above ~70Hz. The -14dB gain adjustment has higher noise from 70-400 Hz, but has slightly suppressed noise above 1 kHz (relative to -12dB gain).

Moku:Go delay measurements will help clarify if this excess noise is a fundamental limitation of the device, or if there is room for improvement by optimizing the controller or further tweaking the gain/offset.

[Hiroki, Paco, Yuta, JC]

Hiroki realized the gain on the AUX feedback loop was slightly high, which was at least partially responsible for the excess frequency noise; after this the lock would still break and result in a mode hop so we tried adjusting the loop gains again this morning. The algorithm that seemed to work was reduce the gain on the analog (UPDH) box until the rail indicators (LEDs) stopped flashing and compensate using the SR560 --

[Paco, yehonathan]

Following Yehonathan's work from yesterday, I found the following coil balancing gains for ETMX:

['UL', 'UR', 'LR', 'LL'] = [-0.993  1.047 -1.007  0.953]

Prompted by the events documented in this previous elog, I used the data I grabbed from the sensors during a previous PRMI lock, described here. I took the time series data from the following channels and compared AS55_Q and REFL11_I to the signals from the fused sensors. 

Sensors measured on the following channels

x1 = "C1:LSC-AS55_I_ERR_DQ" #AS55_I
x2 = "C1:LSC-AS55_Q_ERR_DQ" #AS55_Q
x3 = "C1:LSC-REFL11_I_ERR_DQ" #REFL11_I
x4 = "C1:LSC-REFL11_Q_ERR_DQ" #REFL11_Q
x5 = "C1:LSC-REFL55_I_ERR_DQ" #REFL55_I
x6 = "C1:LSC-REFL55_Q_ERR_DQ" #REFL55_Q
x7 = "C1:LSC-REFL165_I_ERR_DQ" #REFL165_I
x8 = "C1:LSC-REFL165_Q_ERR_DQ" #REFL165_Q

Sensing Matrix

The following sensing matrix was measured during this lock. This is not a new measurement but I'm including it for context since its an important input to the calculations of the fused sensors.

Sensing matrix with the following demodulation phases (counts/m)
{'AS55': 2.0999999046325684, 'REFL55': 76.0199966430664, 'REFL11': 32.638, 'REFL165': 92.23478110797188}
Sensors       MICH @211.1 Hz           PRCL @313.31 Hz           
AS55_I        (+4.44+/-2.93)e+08 [90]    (+1.69+/-0.37)e+10 [0]    
AS55_Q        (+9.89+/-1.00)e+09 [90]    (-3.77+/-0.50)e+09 [0]    
REFL55_I      (-6.12+/-2.24)e+11 [90]    (+3.70+/-0.37)e+13 [0]    
REFL55_Q      (+1.46+/-0.53)e+11 [90]    (-8.89+/-0.86)e+12 [0]    
REFL11_I      (-1.43+/-0.53)e+10 [90]    (+9.13+/-0.86)e+11 [0]    
REFL11_Q      (-7.73+/-3.71)e+08 [90]    (+5.13+/-0.50)e+10 [0]    
REFL165_I     (-3.73+/-0.49)e+09 [90]    (+6.20+/-0.60)e+10 [0]    
REFL165_Q     (-1.38+/-0.10)e+09 [90]    (-2.24+/-0.17)e+10 [0]  

Fused Sensors

The fused senor linear combinations are shown in the bar graphs in attachements 1-3. The the bar height is the weight for each sensor in the linear combination. For attachement 1, I only considered AS55_Q and REFL11_I and formed the fused sensors by choosing the input matrix to be the inverse of the AS55_Q & REFL11_I sensing matrix. This sensing matrix is square so the inverse is unique. The fused sensors in attachements 2 and 3 were calculated from methods described in this previous elog. 

Fused Sensor Signals

Attachment 4 shows the fused sensor signals for the 1 seconds of data for MICH and PRCL. AS55_Q and REFL11_I are plotted for comparison. Those two sensors were used for the PRMI lock so they have no visible low frequency noise. The ASD of the fused sensors are shown in attachement 5. My interpretation of this data is that a higher ASD at low frequencies and a lower ASD at high frequencies are the marker of a good fused sensor. Residual motion of MICH and PRCL during this lock will generate low frequency signals that the good fused sensor will detect. At high frequencies, there is little feedback so the noise measured here is just sensing noise that we want to minimize. 

[Hiroki, Paco, Yuta]

Success in XARM locking with ALSX:

We automated the XARM locking using ALS, the script is easily accessed from the LSC - Actions submenu.
Locking configurations are saved in /opt/rtcds/caltech/c1/Git/40m/scripts/LSC/XARM-ALS.yml . 
The OLTFs for POX11 and ALSX are shown in Attachment 1 (Red: POX11, Blue: ALSX), and the spectra of error signals for POX11 and ALSX are shown in Attachment 2.
Similar to the YARM case, the XARM required FM4 to be off, but also a lower control gain (0.006 instead of the nominal 0.012). To demonstrate the control, we ran an offset lock ramp as with YARM, shown in Attachment 3.

Trials for locking PRFPMI:

We then aligned the PRM and locked PRMI to the carrier while holding the YARM and XARM at an offset using ALS control.  
Then we tried to lock PRMI to the sidebands but we found that the resonant peak of POP110, which was used for the trigger, was negative and failed.
We confirmed that this issue of negative peak is solved by lowering the offset of ALSX and ALSY (Attachment 4 (with 0 offset), Attachment 5 (with slight offset)), so this negative peak might be due to the reflective phase shift of XARM and YARM for sidebands.
However, we were still unable to lock PRMI to the sidebands despite the improved trigger.
We need further investigation...

Upgrading the ETMX coil drivers calls for new coil balancing.

XARM locking using ETMX actuation shows a lot of TRX variation. To make life slightly easier I switch to ITMX actuation. Indeed, the ITMX variation.

1. I drive the ETMX butterfly mode at 13Hz, 10000 cts, and measure the XARM_IN1 signal on diaggui to minimize BUT->POS coupling. There is significant peak at 13Hz.

2. Changing the current coil gains from [-0.8, 1, -1, 1] to [-1,1,-1,1] already reduced the coupling significantly and it seemed that even for 20000 cts the 13 Hz could not be obsereved.

Next, I minimize the POS->PIT coupling

1. I turn off ETMX damping loops.

2. I drive the ETMX POS mode at 13Hz, 1000cts and measure C1:SUS-ETMX_OL_PIT_IN1

3. POS->PIT coupling is minimized using Anchal's technique. When the peak disappears I increase the dither amp to 100000

Next, POS->YAW is minimized in the same way.

Note: when the coil gains change so does the alignment. It is necessary to bring the optic back to the center of the QPD to make fair comparisons.

Since the gains have changed it is necessary to tune the damping loops again.

I measure Q=6 for POS, Q=4 for PIT and Q=3 for YAW.
I decide to reduce the gain of YAW a from 5 to 2.5. Q=5 for PIT now.

I also tune the ETMX OPLEV gains from 1.2 to 1.4.

Final coil balancing gains were:

UL = -1.233 , UR = 0.801, LL =0.767, LR = -1.199

[Paco, yuta]

We found excess noise on the YAUX - PSL ALS beat.

While preparing for a PRFPMI lock, we checked the ALS noise first on YAUX. To our surprise there is significant excess noise above 10 Hz as seen in Attachment #1 (dark blue is today's measurement, cyan is reference from the past). For example, the noise floor at around 100 Hz was 1 Hz, now it is ~ 20 Hz, and the 1 - 1000 Hz rms went up from ~ 200 Hz to > 1 kHz. This may be an issue for CARM offset locking, so we decided to investigate further.

Reviewing the elog, the recent significant change was the breakdown and subsequent replacement of the laser controller. Could the swap made by Koji a few weeks back have introduced excess frequency noise? We also noted the YAUX transmission (C1:ALS-TRY_OUT) has an rms level of ~ 10% which seems excessive but this could be a pathological symptom of our frequency noise or viceversa. The first thing we checked was the OLFT of the YAUX, the shape looks ok and the UGF looks typical, maybe a bit low around 6 kHz. We will repeat this measurement in more detail later.

We took an updated YARM cavity noise budget (see Attachment #2) and confirm that the PSL side looks normal (i.e. MC_F noise spectrum is nominal).

[Paco, Hiroki]  

Installed two flow sensors on the PSL HEPA fans 

To monitor the flow speed of the PSL HEPA fans, we installed two flow sensors:

• Attached two flow sensors on the ceiling of PSL table and connected their outputs to the Acromag ADC:
  • Attachment 1: Connected DB37 to the Acromag ADC
  • Attachment 2: Wiring from the Acromag ADC to the PSL table
  • Attachment 3: Attached flow sensors on the ceiling of the PSL table
  • Attachment 4: Wiring diagram
     
• Changed the flow speed of HEPA fans and monitored the signals from the corresponding sensors with StripTool (Attachment 4):
  • ​Sensor 1:
    • Minimum speed: ~ 0.52 V 
    • Medium speed:   ~ 0.78 V 
    • Maximum speed: ~ 0.83 V 
  • ​Sensor 2:
    • Minimum speed: ~ 0.50 V 
    • Medium speed:   ~ 0.72 V 
    • Maximum speed: ~ 0.83 V 

The results above are non-linear. This might be due to the non-linearity of the flow sensors and the non-linearity of the controllers for HEPA fans.
*Regarding the HEPA fan 1, which the sensor 1 is attached on, there was slight flow even when the speed was minimized. This is why the minimum output of sensor 1 was ~ 0.52 V and not 0.50 V.

Next:

• Calibrate the Acromag ADC by injecting signal from a function generator.
• Calibrate the measued voltage to the flow speed (with linear approximation?) and show them on the CDS screen.

[Paco, Yuta, Hiroki]

We locked PSL frequency using ALSY beat signal and succeeded in sweeping IR resonance for YARM (Attachment 1):

• UGF ~ 150 Hz (Attachment 2)

Automated locking:
Locking configurations are saved in /opt/rtcds/caltech/c1/Git/40m/scripts/LSC/YARM-ALS.yml, and the procedure is almost automated.
When handing off from POY11_I to ALSY, we had to turn FM4 of LSC_YARM off to have more phase margin, as ALSY had excess phase delay compared with POY11_I.
Relative gain including the sign between POY11_I and ALSY need to be tuned in a better way for further automation.
We tried to use 16.4 Hz bounce mode or 29.5 Hz mode to measure the relative gain, but so far not successful.

Noisy ALSY:
The error signal of ALSY is much noisier compared with POY11_I (Attachment 3).
This might be due to the replaced NPRO controller (elog #17685).

Yehonathan, Paco, and Hiroki helped lock the interferomter into PRMI. While locked, the sensing matrix was calaculated with scripts/CAL/SensingMatrix/ReadSensMat.ipynb. Several of the matrix elements flipped sign from the previous matrix I've used: measured in described in this elog. The results are shown in attachement 1. The interferomter was locked with AS55_Q sensing MICH and REFL11_I sensing PRCL. Also, data was recorded from the following channels from 1375574090 - 1375574170 when the lock was stable.

x1 = "C1:LSC-AS55_I_ERR_DQ" #AS55_I
x2 = "C1:LSC-AS55_Q_ERR_DQ" #AS55_Q
x3 = "C1:LSC-REFL11_I_ERR_DQ" #REFL11_I
x4 = "C1:LSC-REFL11_Q_ERR_DQ" #REFL11_Q
x5 = "C1:LSC-REFL55_I_ERR_DQ" #REFL55_I
x6 = "C1:LSC-REFL55_Q_ERR_DQ" #REFL55_Q
x7 = "C1:LSC-REFL165_I_ERR_DQ" #REFL165_I
x8 = "C1:LSC-REFL165_Q_ERR_DQ"

The goal was to try both methods described in this elog about calculating new input matrices. However, there was only time to try the simplest which takes the Moore-Penrose inverss of the sensing matrix and rescaled the first row by the MICH --> AS55_Q sensing matrix element and the second row by the PRCL --> REFL11_I sensing matrix element. The matrix is given below.

[ 0.00024464  0.00237176  0.00955445  0.04220886 -0.0198488   0.0049371 -0.0044138   0.0001007 ]
[ 0.0009515   0.00924028  0.02669179  0.16685194 -0.07720723  0.01935389 -0.01722475  0.0003956 ]

There was only time to plug the matrix in and briefly close the loop and compare the error signals for MICH from the normal sensing matrix with AS55_Q and with the proposed virtual sensor. Lock was not achieved with the new sensors. A screen shot of the error signals are given in attachement 2. The fused error signal is in blue and the AS55_Q signal is in orange. The screen shot shows the error signal for the fused sensor is very different than AS55_Q: large high frequency fluctuations. Based on these results, I am going to use saved time series of the sensors to compute the error signals as functions of time for the data used in the last elog. I'm not sure whether to attribute the poor performance to the large uncertainties in the sensing matrix and the assumption that the sensing matrix is frequency independent or the noises on the sensors which weren't considered in the calculated of this input matrix. 

We moved the Zurich instruments HF2LI to CAML. IMG_7669.JPG

For monitoring the statuses of the PSL HEPA fans, we are planning to attach flow speed sensors (D6F-V03A1) on the HEPA fans.
Before installing the flow sensors, I have finished the followings:

• Tested two flow sensors with alligator clip wires (Attachment 1)
  • No flow: ~ 0.5 V
  • Saturation: ~ 2.7 V
     
• Measured the flow speed of PSL HEPA fans with an alligator-clip-wired flow sensor
  • Medium speed: ~ 0.8 V (~ 0.6 m/s )
  • Maximum speed: ~ 1.4 V (~ 1.8 m/s)
    The flow speed [m/s] was estimated from the linear approximation of the nominal value
     
• Wired flow sensors to install them in PSL and to take the output voltage by an Acromag ADC (Attachment 2)
  • Wired sensors worked as in the case of the alligator-clip-wired sensors.

Next:

• Summarize the details of the used parts and the wiring.
• Install the wired flow sensors in PSL. Monitor their outputs from CDS.

[Paco, Yehonathan, Yuta, Hiroki]  

Achieved stabler lock of sideband-resonant PRMI w/ 3f by tuning feedback damping

We worked on the PRMI and achieved the followings:

• Added an action button to lock sideband-resonant PRMI w/ REFL33 easily
• Tuned the gains for feedback damping to lower the coherences between ASDC and error signals for damping
• Achieved stable lock of sideband-resonant PRMI w/ REFL33
  • Lock stretch for ~7min. : 1375245020 - 1375245310

Details

Action button to lock sideband-resonant PRMI w/ 3f:
To switch the control loop between carrier-resonant PRMI w/ 1f and  sideband-resonant PRMI w/ 3f easily, we added an action button in the LSC window that activates the control loop for sideband-resonant PRMI w/ 3f.
The saved lock conditions are as follows (*whitening gain and demod. angle are not changed by the action button.):

• MICH:
  REFL33_Q (30 dB whitening gain, -19.275 deg demod angle), Power norm. = 0.002*POP110_I, C1:LSC-MICH_GAIN=-1.56, offset=0, boost= off, roll_off=off , Actuator = 0.5*BS-0.307*PRM
• PRCL:
  REFL33_I (30 dB whitening gain, -19.275 deg demod angle),Power norm. = 0.002*POP110_I, C1:LSC-PRCL_GAIN=0.036, offset=0, boost = on, Actuator = 1*PRM
• Trigger:
  POP110_I*1 ⇒ MICH_TRIG ⇒ Threshold upper:150, lower:100
  POP110_I*1 ⇒ PRCL_TRIG ⇒ Threshold upper:150, lower:100

The parameters above are saved in scripts/LSC/PRMIsb-REFL33.yml 

Tuning of feedback damping: 
We locked PRX and monitored the coherences between ASDC and error signals of feedback damping for BS, PR2, PR3, PRM and ITMX.
Then we changed the gains of OSEM loop and OPLEV loop so that the coherences would be small:

• PR3
  C1:SUS-PR3_SUSPIT_GAIN: 30 -> 5
  C1:SUS-PR3_SUSYAW_GAIN: 30 -> 5
• BS
  C1:SUS-BS_SUSPIT_GAIN: 7 -> 10
  C1:SUS-BS_OLPIT_GAIN: -0.1 -> -0.05
• PRM
  ​C1:SUS-PRM_OLPIT_GAIN: 9 -> 6

The resulting spectra and coherences are shown in Attachment 1.

Success in stable locking of sideband-resonant PRMI w/ 3f:
As a result of gain tuning above, we succeeded in locking sideband-resonant PRMI w/ 3f for ~7min.

• Lock stretch: 1375245020 - 1375245310

Moku:Go used for full locking of green laser - OLTF to come

Picking up from where Reuben left off, I used the Moku:Go in multi-instrument mode to replace the signal generator and uPDH box entirely (Moku:Go setup shown in Attachments 1+2). The lock-in amplifier sourced the modulation for the PZT: 210.5 kHz, 1.4 Vpp amplitude (consistent with 7dBm used by the uPDH box) This LO was used to demodulate the REFL signal input. I coarsely tuned the demodulation phase to 90 degrees until the PDH error signal looked reasonable. The PDH error signal was passed to the digital filter box using the same filter as before. After slightly adjusting the gain knob in the filter module (-8 dB), the lock seemed reasonably stable - transmission screenshot in Attachment 3. I got transmission to ~0.8 with the analog loop today, so it was exciting to see this level maintained with the Moku:Go lock (ignoring oscillations from test mass motion). The system remains locked to TEM00 for 5-10 minutes before mode hopping, which is qualitatively comparable to the analog loop as well.

An OLTF of the Moku:Go loop still needs to be taken. Since the loop error point isn't outputted from the Moku (passed direclty between instruments), I'll need to inject an excitation at the control point. When I fed the control signal to input A of the SR560 and tried to lock with the direct output, the lock would repeatedly break. I noticed that the BATT light of the SR560 was on - I'll repeat this with another SR560, but the lock might be breaking due to an offset. Once this is debugged, I can inject an excitation and measure the loop OLTF.

[JC, Paco]

We replaced the HeNe laser for the BS/PRM/SRM Oplev.

After we found a 1103P head replacement in a cabinet along the YARM, JC and I swapped the Oplev laser for BS/PRM/SRM. The head's form factor was the same luckily so we didn't struggle much to recover the QPD signals for BS and PRM. The final alignment aimed to restore ~ 5400 counts in BS (SUM_INMON) and ~ 3100 counts in PRM (SUM_INMON) when aligned.

[Koij Hiroki]

While KA was working on the DAC issue, BS/PRM/SRM Oplev died.

It seems that the BS/PRM/SRM HeNe died and was replaced in 2019 (4yrs + 200 days ago) and 2021 Jan (2yrs + 209 days ago).

We have no energy to work on the HeNe replacement tonight. This needs to be done on Monday.

Tag: OPLEV oplev HeNe died dead

As reported already, we resolved the c1sus DAC0 issue by introducing the differential receivers for the DAC output CHs.

The issue remained in DAC1 (SRM/MC2/MC1/MC3) and DAC2 (All 8 vertex side OSEMs). Their outputs were still received by the single-ended AI boards.
This meant that DAC1 and DAC2 negative side outputs were shorted to ground. This was not great.
(See Attachment 1 for the configuration)

Mitigation

The DAC SCSI-IDC40 adapter units were modified to resolve the issue. This modification was done for the two adapter units among the three.
The modified units were indicated as "DAC Neg out isolated"

These adapter boxes will eventually be removed when we make the transition to the aLIGO electronics. So it's convenient us to touch them.
In the box, the + and - outputs were lined up with a narrow gap. So the modification was a bit cumbersome. The below is the procedure

• First, the solder resist was removed at the corner of the wiring, and copper lines were cut one by one.
• The isolation of the DAC +/-, the isolation of the output sides each other were checked for all the channels.
• Note1: I used a spare PCB found on the workbench for the first modification. I found that one of the left channel (CH9-16) has the output +/- pin connected internally. This would cause + output shorted to the gnd when RevB Ai board is connected. This problem was resolved by cutting one of the IDC40 pin. So don't be surprised if you find one of the pin is missing for the left IDC40 connector for DAC1.
• Note2: One of the + signal line was damaged why the PCB for DAC2 was prepared.

Result

The replacement was done with the watchdogs shutdown. After the replacement, the watchdogs were reloaded and the MC locking were resumed.
I found the apparent behavior of the suspension looked OK but we should keep checking if everything is absolutely fine.

It was surprising that the BS/PRM/SRM OPLEVs lost their spots, but it turned out that it is not related to the electronics work. (See next elog)

[Yuta, Koji, Hiroki]  

Succeeded in locking sideband-resonant PRMI w/ 3f

We worked on the PRMI and achieved the followings:

• Achieved stable lock of carrier-resonant PRMI w/ 1f
  • Lock stretch for ~10min. : 1375235515 - 1375236135
• Succeeded in locking carrier-resonant PRMI w/ 3f
• Succeeded in locking sideband-resonant PRMI w/ 3f

Details

Stable lock of carrier-resonant PRMI w/ 1f:
We tuned the parameters and achieved the stable lock of PRMI w/ 1f:

• MICH:
  AS55_Q (24 dB whitening gain, 2.1 deg demod angle),Power norm. = 0.0008*POPDC sqrt, C1:LSC-MICH_GAIN=0.4,Actuator = 0.5*BS-0.307*PRM
• PRCL:
  REFL11_I (15 dB whitening gain, 32.638 deg demod angle, measued Diff 90),Power norm. = 0.0008*POPDC sqrt,C1:LSC-PRCL_GAIN=-0.0054, Actuator = 1*PRM

The parameters above are added to scripts/LSC/PRMI-AS55_REFL11.yml, so you can import these parameters by pushing the "Actions" button in the LSC window and choosing the PRMI lock.

Time series data of carrier-resonant PRMI lock w/ 1f: 
With the lock conditions above, we locked carrier-resonant PRMI for ~10min.:

• Lock stretch: 1375235515 - 1375236135
• Signal injection for sensing matrix (Notch filters were on):
  • MICH: 211.100 Hz with 1000 counts in amplitude
  • PRCL: 313.310 Hz with 100 counts in amplitude

Rough tuning of demod. angle for REFL33:
We decided to use the REFL33 to lock the carrier-resonant PRMI w/ 3f, so we tuned the demod. angle for REFL33 so that we can obtain MICH and PRCL from REFL33_Q and REFL33_I respectively.
The methods are as follows:

1. Locked the of carrier-resonant PRMI w/ 1f
2. Injected the signal to PRCL @ 313.310 Hz with the amplitude of 100 just the same way as measuring the sensing matrix
3. Observed C1:CAL-SENSMAT_(PRCL,MICH)_REFL33_(I,Q)_AMPMON with ndscope and tuned the demod. angle to minimize the PRCL_REFL33_Q
4. Resulting demod. angle: -20.375 deg (initial: 10 deg)

Gain compensation for locking carrier-resonant PRMI w/ 3f:
To compensate the difference of the gains between 1f lock and 3f lock, we measured the difference of the gains by measuring the transfer function between the error signals in MICH and PRCL:

• MICH: C1:LSC-REFL11_I_ERR / C1:LSC-REFL33_I_ERR (Attachment 1):
  • Signal injection: 211.100 Hz with 1000 counts in amplitude from the function of sensing matrix measurement
  • Gain ~ 3.9, Phase ~ 0 @ 211.100 Hz
• PRCL: C1:LSC-AS55_Q_ERR / C1:LSC-REFL33_Q_ERR (Attachment 2):
  • Signal injection: 313.310 Hz with 100 counts in amplitude from the function of sensing matrix measurement
  • Gain ~ 6.7, Phase ~ 0 deg @313.310 Hz

Fine tuning of demod. angle for REFL33:
We finely tuned the demod. angle for REFL33 as follows:

1. Locked the of carrier-resonant PRMI w/ 1f
2. Injected the signal to PRCL @ 313.310 Hz with the amplitude of 100 just the same way as measuring the sensing matrix
3. Observed the spectrum of C1:LSC-REFL33_(I,Q)_ERR and tuned the demod. angle to minimize the peak @ 313.310 Hz (Attachment 3)
4. Resulting demod. angle: -19.275 deg

Success in locking carrier-resonant PRMI w/ 3f:
Using the result from above and tuning some parameters, we succeeded in locking the carrirer-resonant PRMI w/ 3f.
The conditions are as follows: 

• RFPD InMTRX:
  REFL33_I*6.7 -> PRCL_B, REFL33_Q*3.9 ->MICH_B
• Offset subtraction in DOF Selector:
  PRCL_B: offset=0, MICH_B: offset=37.0 (offset in REFL33 was found when locking MICH with 1f)
• Gains in DOF Selector:
  MICH_A: 0, MICH_B: 1, PRCL_A: 0, PRCL_B: 1
• MICH:
  REFL33_Q (30 dB whitening gain, -19.275 deg demod angle), Power norm. = off, C1:LSC-MICH_GAIN=0.4, boost= off, roll_off= off , Actuator = 0.5*BS-0.307*PRM
• PRCL:
  REFL33_I (30 dB whitening gain, -19.275 deg demod angle), Power norm. = off, C1:LSC-PRCL_GAIN=-0.0054, boost= on ,  Actuator = 1*PRM

Success in locking sideband-resonant PRMI w/ 3f:
We flipped the signs of the error signals and succeeded in locking the sideband-resonant PRMI w/ 3f.
We used POP110 (demod. angle = 178 deg) to check the resonance of the sidebands and to trigger the filters.
Time series data are shown in Attachment 4.
The lock conditions are as follows (also shown in Attachment 5):

• RFPD InMTRX:
  REFL33_I*6.7 -> PRCL_B, REFL33_Q*3.9 ->MICH_B
• Offset subtraction in DOF Selector:
  PRCL_B: offset=0, MICH_B: offset=37.0
• Gains in DOF Selector:
  MICH_A: 0, MICH_B: -1, PRCL_A: 0, PRCL_B: -1
• MICH:
  REFL33_Q ( 30 dB whitening gain, -19.275 deg demod angle), Power norm. = off, C1:LSC-MICH_GAIN=0.4, boost= off, roll_off=off , Actuator = 0.5*BS-0.307*PRM
• PRCL:
  REFL33_I ( 30 dB whitening gain, -19.275 deg demod angle), Power norm. = off, C1:LSC-PRCL_GAIN=-0.0054, boost= on ,  Actuator = 1*PRM
• Trigger:
  POP110_I*2 ⇒ MICH_TRIG ⇒ Threshold upper:200, lower:100
  POP110_I*2 ⇒ PRCL_TRIG ⇒ Threshold upper:200, lower:100
   
• Lock stretch: 1375245020 - 1375245310

We measured the OLTF for PRCL and obtained the UGF of ~ 200Hz, but faied in MICH for low coherence.

The raw frame file is stored in /frames at fb. Before the disk space is flooded, the wiper script deletes the old raw frame files.
I wonder what the state of the wiper script is.

There are wiper scripts (wiper.pl) in /opt/rtcds/caltech/c1/target/fb and /opt/rtcds/caltech/c1/target/daqd, but they no longer seemed to be in use.

I went around the machines and found a service on fb

controls@fb1:~ 0$ sudo systemctl status rts-daq_wiper
● rts-daq_wiper.service - Remove old frame files to reclaim space for the new frame files
   Loaded: loaded (/etc/systemd/system/rts-daq_wiper.service; static; vendor preset: enabled)
   Active: inactive (dead) since Fri 2023-08-04 20:00:27 PDT; 9min ago
  Process: 5355 ExecStart=/opt/rtcds/caltech/c1/scripts/daq_wiper -d /frames -t 2.5 (code=exited, status=0/SUCCESS)

This python version of the wiper script (/opt/rtcds/caltech/c1/scripts/daq_wiper) is running since the last year.
According to the explanation found in the script, the trend data is kept saved, while the raw data is kept deleted to save the specified amount of space (0.25TB it says).
I think this is the configuration we want. So the script was left untouched.

As the preparation of the flow sensor installation, I added two slow channels via c1psl (Acromag IOC).
C1:PEM-HEPA_FLOW_PSL1
C1:PEM-HEPA_FLOW_PSL2

This entry explains how this was done because there is a tricky hidden process that is scarcely explained elsewhere upon adding the slow channels
Tag: How to add slow channels / C0ECDU.ini / Acromag / Advanced LIGO RTS stand-alone EPICS data concentrator

1. Edited the db file

I learned that the spare ADC channels ADC4 CH5 and CH6 are available on c1psl
(See http://nodus.ligo.caltech.edu:8080/40m/[ELOG 17727] )

Created /cvs/cds/caltech/target/c1psl/psl_pem.db

c1psl>cat psl_pem.db
# Greated for Acromag PEM - Koji Arai 2023/8/3
#

grecord(ai,"C1:PEM-HEPA_FLOW_PSL1")
{
        field(DESC,"HEPA Air Flow Speed Monitor Unit 1")
        field(SCAN,".1 second")
    field(DTYP,"asynInt32")
    field(INP,"@asynMask(C1PSL_XT1221_ADC04 5 -16)MODBUS_DATA")
        field(PREC,"3")
        field(LINR,"LINEAR")
        field(EGUF,"10")
        field(EGUL,"-10")
        field(EGU,"Volts")
        field(HOPR,"10")
        field(LOPR,"-10")
        field(ASLO,"1.09225")
}

grecord(ai,"C1:PEM-HEPA_FLOW_PSL2")
{
        field(DESC,"HEPA Air Flow Speed Monitor Unit 2")
        field(SCAN,".1 second")
    field(DTYP,"asynInt32")
    field(INP,"@asynMask(C1PSL_XT1221_ADC04 6 -16)MODBUS_DATA")
        field(PREC,"3")
        field(LINR,"LINEAR")
        field(EGUF,"10")
        field(EGUL,"-10")
        field(EGU,"Volts")
        field(HOPR,"10")
        field(LOPR,"-10")
        field(ASLO,"1.09225")
}

2. Added the db file to cmd file.
Added the following line to /cvs/cds/caltech/target/c1psl/C1_PSL.cmd as

dbLoadDatabase("/cvs/cds/caltech/target/c1psl/psl_pem.db")

3. Restart modbusIOC at c1psl

c1psl>ssh c1psl
controls@c1psl's password:
...
controls@c1psl:~$ sudo systemctl restart modbusIOC.service

4. Added the channels to C0EDCU.ini

Add the following lines to /opt/rtcds/caltech/c1/chans/daq/C0EDCU.ini

Leave the first ~20 channels untouched, as they appears on dataviewer as the default channels.

#HEPA Air Flow Mon channels @c1psl

[C1:PEM-HEPA_FLOW_PSL1]
[C1:PEM-HEPA_FLOW_PSL2]

5. Restart fb

telnet fb 8087
Trying 192.168.113...
Connected to fb.martian.
Escape character is '^]'.
daqd> shutdown

6. Ssh into c1sus and run the following command:

This is a new process. Until this is done, the channels are not recorded.

controls@nodus|daq> ssh c1sus

controls@c1sus:~$ sudo systemctl restart rts-edc_c1sus

[Yuta, Hiroki]

Calibrated the actuators after replacing anti-imaging modules.

As shown in elog #17738, the anti-imaging modules for c1sus DAC0 have been replaced with revised ones.
The new gains were predicted to be increased by a factor of 2, so we calibrated the actuators (BS, ITMX, ITMY and PRM) again.
The results showed that the gains have been increased by a factor of ~2 as predicted.

Summary of calibration:
AS55_Q in MICH : 9.2e8 counts/m 

               Results                                 Previous results
BS        : 50.88e-9 /f^2 m/counts      (26.19e-9 /f^2 m/counts from elog #17679)
ITMX    : 9.50e-9 /f^2 m/counts         (5.07e-9 /f^2 m/counts from elog #17679)
ITMY    : 9.75e-9 /f^2 m/counts         (4.80e-9 /f^2 m/counts from elog #17679)
PRM     : -41.40e-9 /f^2 m/counts      (-20.10e-9 /f^2 m/counts from elog #17522)
 

Methods

Calibration of MICH error signal:
To calibrate the MICH error signal (C1:LSC-AS55_Q_ERR), we followed the same free swinging method as elog #17285 and #16929 but used the update code: scripts/CAL/OpticalGain/getOpticalGain.py.
By fitting the X-Y plot of AS55_Q and  ASDC, we obtained 9.2e8 counts/m counts/m (Attachement 1).

Actuator calibration of BS, ITMX and ITMY:
We locked the MICH and measured the transfer function from C1:LSC-(BS,ITMX,ITMY)_IN2 to C1:LSC-AS55_Q_ERR with excitation from C1:LSC-(BS,ITMX,ITMY)_EXC.
By using the optical gain, measured transfer functions were calibrated to the actuator transfer functions of BS, ITMX and ITMY (Attachment 2):
BS        : 50.88e-9 /f^2 m/counts    
ITMX    : 9.50e-9 /f^2 m/counts     
ITMY    : 9.75e-9 /f^2 m/counts    
Diaggui templates: measurements/LSC/MICH/MICH_(BS,ITMX,ITMY)EXC_Template.xml  
Fitting code: scripts/CAL/MICH/MICHActuatorCalibration.ipynb

Actuator calibration of PRM:
We locked PRCL with PRY misalingning ITMX and measured the transfer function from C1:LSC-(ITMY,PRM)_IN2 to C1:LSC-REFL55_I_ERR with the excitation from C1:LSC-(ITMY,PRM)_EXC (Attachment 3).
Then we obtained the ratio between the transfer functions (Attachment 4) and estimated the actuator transfer function of PRM: -41.40e-9 /f^2 m/counts
Diaggui templates: measurement/LSC/PRM/PRY_(ITMY,PRM)EXC_Template.xml
Fitting code: scripts/CAL/PRMI/PRMActuatorCalibration.ipynb​

Balancing the actuation of MICH in PRMI:
Using the obtained results, we updated the output matrix for PRMI locking by revising scripts/LSC/PRMI-AS55_REFL11.yml: 
C1:LSC-OUTPUT_MTRX MICH-PRM: -0.33 -> -0.307

I went to /cvs/cds/caltech/target and found some folders for apparently old targets.
c0daqawg
c0daqctrl
c0dcu1
c1acro
c1dcuepics
c1losepics
c1psl2
epics-gateway

They were tarballed (.tar.gz) and the folders were removed.

I don't know why we have c1auxey and c1auxey1. And it seems that c1auxey1 is the new one???

We should remove some of them from the autoburt request files.

If there is any unexpected error let me know so that I can revert.

Repeating ADC/DAC noise measurements

I carried out the same measurements of the Moku:Go ADC and DAC noise to compare to the results from Ando Lab. Instead of a flat filter with 50dB of gain, I used the uPDH box fitted filter shape. I recorded spectral densities with an SR785; results are in Attachment 1. These measurements are consistent with those measured in Ando Lab. I included the SR785 noise floor, measured by terminating its input.

DAC noise measurement with single-tone input

Next I tried to measure the Moku's DAC noise using its Waveform Generator and Digital Filter Box in multi-instrument mode. I generated a single-tone digital signal and passed it to an elliptic bandstop filter (fit tightly around the tone). The filtered signal was measured by the SR785. I performed this measurement with 1 kHz and 10 kHz tones [Attachement 2]. While the fundamental peak is suppressed, we still see it and its harmonics (not DAC noise). The floor of these measurements is consistent with the DAC noise reference from the first test, and we can say that the Moku:Go's DAC noise above 100 Hz is below 1 µV/rtHz.

Further ADC noise measurements to come.

After the exploration of c1psl, the IMC locking was not functioning well. It looked like the MC autolocker issue.

Now I remember that the autolocker was running on docker. I followed the instruction on the wiki page. https://wiki-40m.ligo.caltech.edu/Computers_andScripts/AlwaysRunningScripts

Remember: Docker is running on optimus / systemctl is running on megatron!

Then: I noticed that MC autolocker heartbeat was blinking even with the docker stopped (!?). I confirmed that autolocker is not running on systemctl

I had no hope who is toggling the heartbeat, but "ps -def" on megatron showed me that "AutoLockMC_LowPower.csh" is running! I was afraid that it is a remnant from vent!? But, is that possible?

The process was killed and the docker locker is running well now.

[JC Koji]
c1vac was hard rebooted. This fixed the PSL shutter issue.

• Checked the Unibritz Shutter Controller for the PSL shutter. It seems that it responds to the NO/NC switch no matter how the REMOTE/LOCAL switch is. (Normal action)
• Checked the BNC input to the shutter controller. There was no trigger/gate or whatever signal from Acromag.
• c1psl reboot did not help
  • Upon this action, a snapshot was taken.
  • Stopped/restarted modbusIOC.services. Did not work.
  • Rebooted c1psl. Didn't work.
• c1psl was shutdown and c1psl acromag chassis was power-cycled. This didn't help. c1psl was burtrestored.
  • Stopped modbusIOC.services. Shutdown c1psl.
  • Power cycled the Acromag chassis.
  • After a while, c1psl was turned on. The epics channels came back.
  • Confirmed modbusIOC.services was running.
  • Burtrestored the snapshot.
• I remembered that the vacuum pressure could make an interlock action on the PSL shutter.
  Found that c1vac values were all whited out. c1vac was not reachable from Martian.
• Went to c1vac and found it was frozen. Pushed hard reset.
• c1vac came back, and it closed the gate valve V1 (main GV for the main vacuum manifold). We opened V1 from the vac control screen.
• Confirmed the PSL shutter is now operational.

BHD platform is estimated to stop baking by the end of the day. We are following the LIGO Procedure and baking at 150°C By tomorrow morning, I will have the platform out of the Air Bake Oven.

· I have began to prepare the smaller components for the assembly for baking. I am trying to get all of these components cleaned and together by the end of next week.

[Yuta, Hiroki]

In preparation for locking the PRFPMI, we tried to lock the sideband-resonant PRMI (but failed) on Aug. 1st:

• Locked PRY. The resulting UGFs were consistent with the previous lock (elog #17521).
• Locked carrier-resonant PRMI with 1f. The resulting UGFs were consistent with the previous lock (elog #17696)
• Tried to lock sideband-resonant PRMI with 1f (REFL55_I for PRCL and _Q for MICH) but failed. This should be due to the large mixed PRCL signal in REFL55_Q.

Next:

• Precisely tune the demod. angle and minimize the PRCL signal in REFL55_Q so that the MICH signal is dominant over the PRCL signal.
• Try locking sideband-resonant PRMI again.

We tried locking PRMI for the first time since we have solved the strange ITMX/Y issue by replacing the anti-imaging modules (elog #17738).
So we started with locking PRY and carrier-resonant PRMI before locking sideband-resonant PRMI.

Locking PRY:
We misaligned ITMX and locked PRY. Locking conditions are as follows:

• MICH​
  REFL55_I (24 dB whitening gain, 76.02 deg demod. angle), C1:LSC-PRCL_GAIN=-0.03, Actuator=
  1*PRM
  -> UGF ~ 200 Hz
   
• PRCL
  REFL11_I (15 dB whitening gain, 32.638 deg demod angle) C1:LSC-PRCL_GAIN=-0.8, Actuator=
  1*PRM
  -> UGF ~ 120 Hz 

These UGFs are consistent with the previous lock (100 Hz for both, elog #17521) taking into account the increased gain (*2) of the anti-imaging modules (elog #17738) and the difference of the whitening gains.

Locking carrier-resonant PRMI:
We succeeded in locking PRMI after restoring ITMX and aligning PRMI. There was nothing strange in this locking.
The locking conditions and the results are as follows:

• MICH
  AS55_Q (24 dB whitening gain, 2.1 deg demod angle), Power norm. = 1*POPDC, C1:LSC-MICH_GAIN=400, Actuator=0.5*BS-0.33*PRM
  -> UGF ~ 50 Hz (consistent with elog #17696)
   
• PRCL
  REFL11_I (15 dB whitening gain, 32.638 deg demod angle), Power norm. = off, C1:LSC-PRCL_GAIN=-0.005, Actuator = 1*PRM
  -> UGF ~ 220 Hz (consistent with elog #17696)
   
• Resulting POPDC: ~1600 at maximum
• Data for noise budget: gpstime 1374968385 - 1374968500

Trials for locking sideband-resonant PRMI:
We calculated the appropriate gains for MICH and PRCL to realize the same UGFs as carrier-resonant PRMI.
Also, we reduced the whitening gain for REFL55_Q to 12 dB as the error signal were saturated at its peak.
We tried locking changing some parameters based on the following conditions but failed:

• MICH
   REFL55_Q (12 dB whitening gain, 76.02 deg demod angle), Power norm. = off, C1:LSC-MICH_GAIN=0.1,Actuator = 0.5*BS-0.33*PRM
   
• PRCL
  REFL55_I (12 dB whitening gain, 76.02 deg demod angle), Power norm. = off, C1:LSC-PRCL_GAIN=-0.0004, Actuator = 1*PRM 
   
• Trigger
  Varied the threshold using the POP110_I (The power of the upper and lower sidebands. In other words, the beatnote between upper and lower sidebands).

This failure should be due to the large mixed PRCL signal in REFL55_Q.
We need to precisely tune the demod. angle and minimize the PRCL signal in REFL55_Q so that the MICH signal is dominant over the PRCL signal.
 

[Yuta, Hiroki]

Remote switching of PSL shutter is not working.

When we tried to reset the LSC offsets, we found that the remote switching of the PSL shutter was not working.
We closed the shutter manually by using the toggle switch on the front panel of the shutter driver (N.C. -> N.O. (Attachment 1)) for the momment.
 

This post is reprinted from the elog of Ando Lab at the University of Tokyo.
[Author: Satoru Takano]

Recently some people are interested in ADC/DAC noise of Moku:GO. I measured them, and found that ADC noise is much larger than expected (quantization error).
I used "PID Controller" module for the measurement of ADC/DAC noise.

DAC Noise

I set up the module as follows:

• Input channel: open, no input
• Output channel: ch1

I took data by data logger DL-750 and calculated PSDs by myself. Measured DAC noise is shown in Fig.1.

I modeled the noise spectrum by:
​

ADC Noise

I set up the module as follows:

• Input channel: ch1, terminated by 50Ω
• Output channel: ch1
• Filter: Flat, 50dB amplification

I took data by data logger DL-750 and calculated PSDs by myself. Measured ADC noise is shown in Fig.2.

I modeled the noise spectrum by:

Comparison with Theoretical Value

I estimated quantization error of ADC/DAC. Specification of the inputs/outputs are as follows:

• Sampling rate: 125 MHz
• Bit number: 12 Bit
• Voltage range: ±5V, 10Vpp

From these specifications, the estimated quantization noise is . I plotted measured noise and this estimation noise in Fig.3.

Comparing with this value,

• DAC noise: the floor level is almost the same as the estimated quantization noise. Below 1.4kHz an additional 1/f noise exists.
• ADC noise: the floor level at 100kHz is about 10 times larger, and at 100Hz about 100 times larger than the estimation. Below 46Hz another 1/f noise exists.

There is a measured data about ADC noise of Moku:Lab at here. Comparing with this, ADC noise of Moku:GO has a stranger shape than that of Moku:Lab.

Summary

If we are using Moku:GO and worried about sensitivities, we should insert some whitening filter before the input of Moku:GO.

This elog post refers to this previous post which details a PRMI lock where the sensing matrix and OLTFs for MICH and PRCL were measured. In addition, I grabbed the time series data during this lock from AS55, REFL11, REFL55, and REFL165 for I and Q demodulations from GPS time 1373763480 - 1373763560. During the lock, the MICH signal was normalized by POPDC, but I confirmed with Koji this didn't affect the data in the channels I used.

Channels I grabbed from:

x1 = "C1:LSC-AS55_I_ERR_DQ" #AS55_I
x2 = "C1:LSC-AS55_Q_ERR_DQ" #AS55_Q
x3 = "C1:LSC-REFL11_I_ERR_DQ" #REFL11_I
x4 = "C1:LSC-REFL11_Q_ERR_DQ" #REFL11_Q
x5 = "C1:LSC-REFL55_I_ERR_DQ" #REFL55_I
x6 = "C1:LSC-REFL55_Q_ERR_DQ" #REFL55_Q
x7 = "C1:LSC-REFL165_I_ERR_DQ" #REFL165_I
x8 = "C1:LSC-REFL165_Q_ERR_DQ" #REFL165_Q

I calculate (what I'm calling)the covariance matrix: COVAR(X) = [CSD(x_i,  x_j)]_i,j where X is the vector of measurements from each sensor and CSD is the cross spectral density. Each element is the CSD between two sensors and the diagonals are the PSD of each sensor. The ASDs of this data is shown in attachement 1.  

Attachement 4 depicts a block diagram I am using to represnt the feedback system. PC includes details of the control filters, actuators and mechanical plant. After this block, n_d represents displacement noise that enters in Y, the state of MICH and PRCL. S is the sensing matrix and M is the input matrix that fuses the sensors. n_s represnts all other noise on the sensors. Based on this model, the measured data form the sensors had this relation to the input noise sources: X = (I+SPCM₀)^-1(Sn_d+n_s), where M₀ is the input matrix used during the lock. COVAR(X) = COVAR( (I+SPCM₀)^-1(Sn_d+n_s) ) =  (I+SPCM₀)^-1COVAR(Sn_d+n_s)((I+SPCM₀)^-1)^H. These transfer functions take linear combinations of the frequency representations of the signals/noises so the PSDs and CSDs of the new signals get new cross terms based on the mixing. This is captured by maping the covariance matrix between the transfer function and its hermitian conjugate. This was used to remove the suppression by the feedback loop by mapping COVAR(X) with (I+SPCM₀). I define COVAR(X)_unsupressed = (I+SPCM₀)COVAR(X)(!+SPCM₀)^H =  (I+SPCM₀)(I+SPCM₀)^-1COVAR(Sn_d+n_s)((I+SPCM₀)^-1)^H(I+SPCM₀)^H  = COVAR(Sn_d + n_s). The ASDs of the unsuppressed noises in each sensor are shown in attachements 2 and 3. In 2, these ASDs are calibrated to meters by multiplying them by the recipricol of the sensing matrix coefficient mapping MICH to that sensor. In 3, the same is done for PRCL.

The theoretical closed loop performance of new M's can be calculated by mapping the COVAR(X)_unsuppressed by the closed loop transfer function for a signal injected at the sensors, X, to the fused sensors, Z. This TF is (I+MSPC)^-1M. Then, from Z the signals are mapped by (MS)^-1 to simulate the new ability to control and sense Y. As before, these transfer functions are used to map the covariance matrix. I define COVAR(Y) = [(MS)^-1 (I+MSPC)^-1M] COVAR(X)_unsuppressed[(MS)^-1 (I+MSPC)^-1M]^H. I have previouly tried using a transfer function from noise injected at the sensors directly to Y, but I don't think this captures the important dynamics since S and M are rectangular matricies. M maps from a vector space with dimension equal to the number of sensors(8 in this case) to dimension equal to the number of DoFs(2 in this case). Previously, I was using S^-1(I+SPCM)^-1 as the transfer function to map COVAR(X)_unsuppressed --> COVAR(Y) where S^-1 is the Moore-Penrose inverse of S. At low frequencies I think this can be equivalent for a class of M's, but all M dependence goes away for high frequencies when PC --> 0. On the other hand, we expect M to be very important at high frequencies since it determines what sensing noise enters the system. 

Attachements 5 and 6 show the theoretical ASD for MICH and PRCL with several choices of M as well as the open loop noise. These correspond to the square root of the diagonal elements of COVAR(Y). The M₀ curve shows the performance of the lock in which the data was taken. The calculation for this is the same as described above but M₀ was chosen to select AS55_Q for MICH and REFL11_I for PRCL. The MS=G₀ data was calculated by starting with the Moore-Penrose inverse of S, so that MS is diagonal, and scaling the rows to have the same MICH and PRCL gain as M₀. The top row of S^-1 was scaled by the MICH --> AS55_Q sensing matrix element and the bottom row by the PRCL --> REFL11_I sensing matrix element. The last matrix shown, MS=G_0 w/ noise fit is subject to the MS=G_0 constraint just described, but it was also fit to reduce noise entering the system at high frequencies. Above 1KHz, the noise is pretty white in all sensors. The ASDs for 1KHz to just below the cutoff frequency are shown in attachment 8. For these range of frequencies I used M*COVAR(X)_unsuppressed*M^H as a cost function to approximate the open loop sensing noise entering the fused sensors. I used scipy.optimize.minimize() to optimize M for this cost function with repsect to the MS=G_0 constraint that I assumed would be crucial for decent closed loop performance at low frequencie. This optimization was done at every frequency step, but the result was unsuprisingly pretty constant over the chosen frequency range so I averaged the elements of the matrix to make a constant matrix. This matrix is used to calculate the MS=G_0 w/ noise fit data.

Both matrices suggested show improvements in these calculations. The both rely on the sensing matrix, which has large uncertainties and is assumed to be frequency independent which may strongly hinder their performance in the real PRMI LSC system.

The matrices are given below. The order of the channel names above determines the order of the coefficients along each row. 

MS=G_0 matrix:
[ 0.06554358  0.89041025 -0.02285036 -0.08922597  0.0720552   0.00681149 -0.24507124 -0.15858639]
[  4.0546419   55.0763957   -0.46854116  -5.7461138    4.48030295 0.4226362  -15.15734417  -9.809963  ]
MS=G_0 w/ noise fit
[ 0.20865055  0.194148   -1.70063937 -7.07367426  0.04610331 -0.01614077 -0.10858281 -0.06099116]
[  12.91008472   12.00303754 -104.26878095 -437.86044383    2.82103162 -0.99588645   -6.71642619   -3.77348024]

Update

1. I have finished setting up Neptune for easy access to the train/eval metrics of the RL models. Here is a link where you can track my progress. Neptune is an extremely useful tool that allows you to see in real time how the models are bahving: you can see nice plots of losses and returns, as well as my source code, the stdout and stderr, and even the resources plots for the node that my jobs got allocated to. Feel free to play around with it.
2. Currently, there are are only 2 runs (I just started them): one with DQN (RLCON-8), the other with PPO (RLCON-7).
3. As outlined before, I have tried improving the PPO agent by replacing the replay buffer with a reverb server (which is supposed to be the fast way of doing a replay buffer). However, no big speed ups have been achieved. The problem is that PPO is using a special type of layer: a distributional layer. I am unsure as to how this layer works, but certainly it is not a normal "dense" layer. After changing all other components, it seems that the PPO algorithm is very slow not because of the replay as I initially thought, but because of the actual training of the agent at line agent.train(experience).  However, I still have a lead: there is an error signal about tensorflow retracing (see run RLCON-8). I am now looking into that.
4. In the mean time, I will start increasing the size of the Q-net for the DQN and increasing the num_actions to 201 (so a resolution of 0.5% in duty cycle). That is because RLCON-7 seems to not learn too much: the loss is really unstable and the evaluation avg_return is terrible.

Aside on avg_return if you wonder what this number means: basically, at each step of the environment (the code that simulates the temperature of the puck) I am takig the error to be . Then, for an entire episode, the reward is given by the average of these errors (so it gives you average error for all steps in one episode). Then, the reported avg_return is calculated by taking the mean of average errors for 5 (as of now) episodes. Basically, sqrt(|avg_return|) gives you an average deviation away from T_ref: so if avg_return=-25, then you should expect the model to be consistently around 5 degrees away from T_ref.

Mode-matching breadboard has been constructed

I have constructed the mode-matching breadboard for aligning the BHD OMCs.
I also measured the profile of the resulting beam:

Beam waist X: 496 +/- 2 um
Beam waist Y: 504 +/- 3 um
Waist position X (from the front surface of the collimator mount): 224 +/- 1 cm
Waist position Y (from the front surface of the collimator mount): 236 +/- 2 cm
Maximum mode-matching efficiency to OMC: 99.78 +/- 0.07 % (if the waist of the OMC eigen mode is place at 230 cm)
(OMC eigen mode: 489.6 um (horizontal), 490.5 um (veritical))

The details are shown in the following attachments:

• Attachment 1: Current configuration and summary of beam profile
• Attachment 2: Resulting beam profile
• Attachment 3: Photo of mode-matching bread board

[Hiroki, Yehonathan, Yuta, Koji]

Summary

• The c1sus DAC0 issue seemed to be the driving capability of the DAC card.
• The DAC issue was solved by replacing 2x Anti-Imaging modules (D000186 RevB) with 2x Rev C equipped with input amps.
• This may mean that the previous DAC card failure was also the same issue. (i.e. The card could still be OK)
   
• This change may have increased the coil driver responses of the vertex suspensions by a factor of 2
  as the conventional RevB had the single-end inputs, but Rev C has the differential ones.
   
• Enabling the full actuation range caused significant misalignment of the suspensions. But the IFO was realigned and now looks much more normal than before.
  • The ITMX OPLEV servos could be engaged with the conventional gain. No instability is observed anymore.
  • The BS OPLEV gains were too high. |G|=0.4 -> 0.2 to maintain stability.
  • Confirmed Yarm ASS functioning well as before
  • Xarm ASS is partially functioning as before. Some gains were reduced to keep the stability, but still, careful inspection is necessary.

Details

• The monitor outputs of the Anti Anti-Imaging modules (D000186 RevB) were checked while the corresponding coil EXC channels were excited via awggui.
• When the DAC was fully driven, we observed the output was only limited to the negative side of the full scale. The negative side was clipped at -5V. (See Attachment 1). This was the case for all 16 channels for DAC0. We found no issue for DAC1 and DAC2.
• Disconnect the DAC0 SCSI cable from the SCSI-IDC40 adapter unit to isolate the DAC from the analog chain. Could we observe the bipolar signal? Indeed yes, we saw the bipolar signal. (Attachment 2). (We use a SCSI-DSUB9 adapter board.)
• We wondered if this was the failure of the AI modules. Replacing the left D000186 with the same Rev B spare. No change.
• Checked if power cycling the AI module reset it. As this was done with the DAC connected, we lost positive and negative outputs of the DAC (OMG!)
• Rebooting c1sus solved the issue, and indeed the DAC exhibited a bipolar signal even with the Rev B cards.
• We thought we solved the issue! Came back to the control room and worked on a boot fest. After about an hour of struggle, all the RTS are back online. (GJ!)
• But we found that we are back to the original state. (same as Attachment 1)

Some break ....

• Went back to the rack. My thought was that the DAC could work when it was not connected to the AI modules. Once it is connected to the AI modules, the positive side disappears. Probably can't DAC drive all the load?
• Disconnected the 2nd IDC40 cable from the adapter (i.e., 2nd AI isolated). This brings the bipolar signals back! (Attachment 3)
• If the 1st IDC40 was this connected while the 2nd cable was connected, this brings the bipolar signals back to the later half 8chs (Attachment 4)
   
• We have a couple of D000186 Rev C spares. I checked the circuit diagram of Rev C (it is not on DCC but in Jay's circuit diagram archive on 40m BOX). I found that each channel of Rev C has an INA134. This is not a true instrumentation amplifier but a differential amplifier with R=25kOhm. So it was promising.
• Indeed, replacing a RevB with a RevC already solved the issue. Just in case, I've replaced both RevBs for DAC0.

• I wondered what the input impedance of RevB was. The Frequency Devices Filter module has the min input impedance of 10kOhm. However, the Rev B input is single-ended. I believe the negative side is shorted to GND! This probably mean that the gain with Rev C is x2 of the conventional one
   
• IFO alignment / locking / oplev test: as described at the summary.

[JC]
I began to prepare the temporary clean room where we will be putting together the BHD Platform Assembly - D2100085. I wiped down the table with the Acetone and IPA multiple times. After this, I used Mylar sheets to use as the cleanroom walls. I slit the sheet into separate pieces to allow easy access and held these sheets up by using Kapton tape. Next, I connected the HEPA booth to an extension cord which was plugged into the Wall outlet of 1X3. 

I did the same test we did in 40m/17616 to see if DACs are working fine for ITMX and ITMY.
I used awggui to excite all the coil outputs of the optic with 0.1 Hz with an amplitude of 3000 counts, and checked VMons.
Found that ITMX, ITMY, PRM and BS face coil DAC outputs cannot drive positive voltages. In contrast, SRM is fine (see attached).
This is consistent with what we have found in June (40m/17616).
We need to investigate what is causing this DAC failure...

[Maty, JC]
This morning, Maty and I proceeded to work on the BHD Platform work. We rinsed off the BHD platform one more time with water and placed onto the work bench. Here, we wiped down using IPA and removed a ton of the metal shavings from the platform. (Attachment #2 & #3 ) The threads in the platform had a ton of shaving, but we were able to get majority, if not all, off the part. 

Next, we garbed up and prepared the large tub for the BHD Platform. The tub contained 107 L of water, so we added 1000 mL of Liquinox into the tub. The Platform was a bit too wide, so we had to put it in at an angle. We used a beaker to pour the solution over all of the part. Along with this, we used the Sonicating probe to clean the threads. ( Attachment #3 ) We did this for the entire part and then drained the tub. 

I lifted and placed the platform into the blue machine parts washer to rinse off the part. We went thread by thread to make sure all of the Liquinox was rinsed off of the part, this to ~30 minutes. ( Attachment #4 )After rinsing the part off, we brought it over to the clean room flow bench. Here, Maty placed large clothes under to platform. She then went through each hole of the platform with the TopGun to remove the loose excess water. ( Attachement #5 )

We have stopped here and are leaving the BHD Platform to dry overnight. We will insert the BHD Platform into the Large Air Bake Oven TOMORROW! We plan to have it ready by the end of the week.