[JC, Yuta]

PMC was unlocked from last night, so we aligned PMC
c1sus2 crashed again during the PMC alignment, so we ran

./opt/rtcds/caltech/c1/Git/40m/scripts/cds/restartAllModels.sh

We burt restored to 2023/Aug/14/16:19 by

./opt/rtcds/caltech/c1/Git/40m/scripts/cds/burtRestoreAndResetSUS.sh /opt/rtcds/caltech/c1/burt/autoburt/snapshots/2023/Aug/14/16:19

Summary

The problem in the PMC Frequency Reference Card (35.5MHz) was fixed.
The last opamp stage of the attenuator control (LT1125) had the negative rail, and the EOM drive level was maximally attenuated.
The chip was replaced, and the card was back in the euro crate. The PMC is locking as before.

Diagnosis

Yesterday, I already found that the modulation level was basically zero and had no response to the level slider. Today, the card was extracted from the rack and tested on the workbench. It required +/-24V supplies for the main power and +10V supply for high-power RF amps (SMA-202). Additionally, a function generator to supply a DC voltage is needed to tune the attenuation level manually. (Attachment 1)

Some notes:
- This circuit was previously fixed in 2015 by Yutaro and me: [ELOG 11763]
- Even before that, Jenne and Rana looked at the board for LO fix in 2014 [ELOG 10160]
- The boards DCC number is D980353 however the 35.5MHz version is D000419 and the 40m version has a dedicated DCC number D1400221

Initial look was horrible because some greasy material was covering some parts on the boards (I had the same thing in 2015 too). It turned out that it is a leaked thermal paste from the heat sink at the bottom side. (Attachment 2)

Firstly, the suspicious ERA-5SM was checked. The attenuator control path was suspiciously dead (-5V) but the function generator was attached to the attenuator control pin, the ERA-5SM in the EOM path received the signal and it was working fine(!). The ERA-5SM received 20mVpp and spit out 270mVpp, observed with a 1/10 probe. The LO path one was also alive, receiving 173mVpp_max and yielding 200mVpp.

Then, the high-power amps (SMA-202) were checked. This is an old chip which is not commercially available anymore. So I thought I would be reluctant to replace it was broken. But the amps were just fine. With the above condition, the LO and PC paths' outputs were ~9Vpp and 6.1Vpp.

Now I went into the attenuator control part. And it turned out that the last stage (low pass filter part) was broken. By replacing the LT1125, the attenuator control chain recovered the function. (Attachment 3)

Calibration

The attenuator control voltage was applied to the board, then this control voltage was measured together with the RF output powers (LO and PC). (Attachment 4)

LO power is about 16dBm~17dBm and almost independent with the setting (as expected).
PC power changes from -24dBm to +25dBm.

Attachment 5 is the relationship between the PC RF power and the MODEL Voltage measured in analog.

Restore the setting:

The module was returned to the rack. One thing we should take care of is that the external 10V should be disconnected when the output is not terminated with 50Ohm loads. This is to protect SMA-202 from reflection damage.

Once the board was secured and the 10V supply was connected, the MODET voltage was dependent on the slider setting. The PC output RF power was calibrated against the RFADJ setting. (Attachment 6) It is consistent with the above analog measurements.

The RF level (C1:PSL-PMC_RFADJ) was set to 6.0. This imposes the PC output of +14dBm. C1:PSL-PMC_MODET came back to ~-0.34, which is consistent with the number before the trouble.
I also checked the LO power in situ. The direct output was ~17dBm and the 9dB attenuator made the LO level down to 8dBm. The mixer is ZAD-6, which is 7dBm LO level. So it looks fine.

PDH error signal

The amplitude of the PDH error signal was observed at the IF output of the frequency mixer. The cavity was swept around one of the resonances. It showed a clear PDH error shape with the P-P amplitude of ~130mV. (Attachment 7)

By the way, the PMC error offset slider was swept while the cavity was locked. The error signal indicated

The error signal / Offset slider value
-5.23mV / +10V
-6.11mV / 0V
-7.03mV / -10V
It seems that there is a 1/10⁴ reduction factor between the slider value and the actual applied voltage.

Basic Idea

The idea behind this study was that in several frequency ranges, there are spikes, or just dominant features, in the ASD of each of the sensors. I’ll model these noise sources as a signal  where  is a Nx1 matrix that describes how the single noise source couples into each sensor. Lets assume that this noise source is the dominant source of noise power in the sensors between  and . Then the band limited covariance matrix will have the following form.

 is the integral of the PSD of the noise source over the frequency band. We can use SVD of  to calculate the result of the PCA.  . Since  is a Nx1 dimensional matrix,  is a 1x1 matrix,  is a Nx1 matrix with a the singular value in the first component and 0’s in the rest, and  is a NxN matrix. The first column of  is equal to  up to a scalar. The result of the columns of  for an orthogonal basis for the space perpendicular to . Therefore, for prominent features in the ASD of the sensors, we can "zoom in" and do a PCA. The first principle component will tell us how the noise source is coupling into the sensor vector space. The rest of the principle components will define subspace orthogonal to the noise source. Therefore a virtual sensor designed in this orthogonal subspace will avoid the noise spike.

Test with simulated noise

The first test I did was to choose some S_n at random and a noise source ASD of  with = 1KHz. I started with the CSD of the unsupressed sensor data, , and calculated the CSD with the new simulated noise: . Attachement 1 shows the ASD of the unsuppressed sensors with this added noise. The second figure is zoomed in on the noise peak. The results of the PCA analysis are summarized by attachment 2. The first figure in attachment 2 shows the ASDs of the virtual sensors formed by the principle components of the PCA. The second figure zooms in on the 1Khz feature. The solid blue curves are the ASDs of the sensors and the dotted red lines are the PCA virtual sensors. This figure shows that the first PCA sensors clearly contains the noise spike but the other N-1 (7) PCA sensors are now flat over this frequency range.

Test with real noise

Attachements 3 and 4 show the same anaylsis done on a real noise feature in the sensors ASDS at ~180Hz and attachments 5 and 6 show the same for a ~640 Hz feature. Both tests show results consistent with the simulated noise test. These are the only tests I've performed thus far so I haven't found a feature yet that doesn't play well with this analysis. 

Conclusions

Virtual sensors could be designed to avoid particular noise spikes this way but a more optimal sensor could avoid multiple noise spikes by transitioning, as a function of frequency, between the othogonal subspaces defined by the PCA. Also, ths technique provides a measure of  up to a scaling. Perhaps this can allow for the origin of these noise features to be identified. 

== BHD Components ==

BHD BS:
We still have a spare BHD BS @South end optics cabinet. (Attachment 1)

18bit DAC AI:
We have 4x D1000305 aLIGO 18-bit AI Chassis Top Assembly Drawing (4xDB9 Version) @1X3B Rack (Attachment 2)
This version has two DB25M connectors to be connected to DAC. (Attachment 3)

16bit DAC AI kit: To turn the above 18bit AI to D1101521 aLIGO AI 16-Bit DAC Chassis Top Assembly (4xDB9 Dsub config) @Y10 section beneath the tube
- We have 5x rear panels in "Front Panel" box. They are labeled "ADC" rather than "DAC" but work with DACs. (Attachment 4)
- We hacve 4x D0700101 16 bit DAC AI Rear Interface Board in "16bit DAC AI Rear PCB" box. They have already been assembled. (Attachments 5/6)

== Misc discovery ==

Many Eurocard Anti Imaging Boards (Rev C) D000186-C @Y10 section beneath the tube (Attachment 7)
Whaaat! It's the differential version of the iLIGO AI boards... orz

Hidden NPRO set marked broken @Y9 section beneath the tube (Attachments 8-10)
It's the broken NPRO from an end, but we should try to determine which head and controller are broken.
Related ELOG in 2017

[JC, Yuta]

Transitioning from 1f to 3f in both PRMI carrier and sideband is now smooth once you have PRMI nicely aligned (key is to tweak TT1 and TT2).
We measured the sensing matrix for PRMI carrier/sideband locks and measured MICH and PRCL sensitivity with different locking configurations.
PRCL sensitivity does not change between different sensors, but MICH sensitivity gets worse with REFL33_Q.

PRMI sensing matrix during 1f carrier lock:
(whitening gains: AS55 24 dB, REFL55 24 dB, REFL11 15 dB, REFL33 30 dB, REFL165 24 dB)

Sensing matrix with the following demodulation phases (counts/m)
{'AS55': 2.1, 'REFL55': 76.02, 'REFL11': 32.638, 'REFL33': -19.275, 'REFL165': 108.88}
Sensors       MICH @211.1 Hz           PRCL @313.31 Hz           
AS55_I       (+1.77+/-0.44)e+09 [90]    (+3.31+/-0.93)e+09 [0]    
AS55_Q       (+1.45+/-0.10)e+10 [90]    (-0.90+/-1.15)e+09 [0]    
REFL55_I       (-0.06+/-2.51)e+12 [90]    (+1.17+/-0.18)e+13 [0]    
REFL55_Q       (+0.32+/-6.10)e+11 [90]    (-4.99+/-0.38)e+12 [0]    
REFL11_I       (-1.28+/-0.93)e+10 [90]    (+1.16+/-0.07)e+12 [0]    
REFL11_Q       (-0.07+/-1.76)e+09 [90]    (+3.47+/-0.25)e+10 [0]    
REFL33_I       (-3.00+/-1.02)e+09 [90]    (+1.65+/-0.10)e+11 [0]    
REFL33_Q       (+2.06+/-0.64)e+09 [90]    (-8.01+/-0.55)e+09 [0]    
REFL165_I       (-4.16+/-0.51)e+09 [90]    (+8.11+/-0.52)e+10 [0]    
REFL165_Q       (-2.85+/-0.21)e+09 [90]    (-5.59+/-0.56)e+09 [0]    

Ratio AS55_Q/REFL33_Q for MICH was 1.45e10/2.06e9 = 7.0 (it was 3.9 in 40m/17755)
Ratio REFL11_I/REFL33_I for PRCL was 1.16e12/1.65e11 = 7.0 (it was 6.7 in 40m/17755)

PRMI sensing matrix during 3f sideband lock:
(whitening gains: AS55 24 dB, REFL55 24 dB, REFL11 15 dB, REFL33 30 dB, REFL165 24 dB)

Sensing matrix with the following demodulation phases (counts/m)
{'AS55': 2.1, 'REFL55': 76.02, 'REFL11': 32.638, 'REFL33': -19.275, 'REFL165': 108.88}
Sensors       MICH @211.1 Hz           PRCL @313.31 Hz           
AS55_I       (-1.09+/-1.48)e+09 [90]    (-5.73+/-3.01)e+09 [0]    
AS55_Q       (+1.59+/-0.13)e+10 [90]    (-5.50+/-3.46)e+09 [0]    
REFL55_I       (+1.42+/-0.16)e+12 [90]    (-3.70+/-0.17)e+13 [0]    
REFL55_Q       (-1.39+/-0.09)e+12 [90]    (-2.39+/-0.12)e+13 [0]    
REFL11_I       (+1.34+/-0.36)e+10 [90]    (-1.23+/-0.06)e+12 [0]    
REFL11_Q       (+9.64+/-0.78)e+09 [90]    (+1.77+/-0.10)e+11 [0]    
REFL33_I       (+2.89+/-0.54)e+09 [90]    (-1.85+/-0.09)e+11 [0]    
REFL33_Q       (-2.10+/-0.19)e+09 [90]    (+1.77+/-0.10)e+10 [0]    
REFL165_I       (+2.50+/-0.37)e+09 [90]    (-8.24+/-0.38)e+10 [0]    
REFL165_Q       (+3.73+/-0.25)e+09 [90]    (+2.15+/-0.11)e+10 [0]    

Signs flip from carrier lock for REFL33_I and Q, and REFL11_I, but not for AS55_Q.

Jupyter notebook: /Git/40m/scripts/CAL/SensingMatrix/ReadSensMat.ipynb

Locking configurations:
PRMI 1f carrier
 - MICH: 1*AS55_Q
 - PRCL: 1*REFL11_Q

PRMI 1f sideband
 - MICH: 1*AS55_Q
 - PRCL: -1*REFL11_Q

PRMI 3f carrier
 - MICH: 7*REFL33_Q
 - PRCL: 7*REFL33_Q

RPMI 3f sideband
 - MICH: -7*REFL33_Q
 - PRCL: -7*REFL33_Q

Common
 - Trigger on POPDC for carrier, POP110_I for sideband (we can also use POP110_I for both by flipping the sign)
 - C1:LSC-MICH_GAIN = 0.4
 - C1:LSC-PRCL_GAIN = -0.0054
 - No power normalization
 - MICH actuator is 0.5*BS-0.307*PRM
 - PRCL actuator is 1*PRM

PRCL and MICH sensitivity curves:
 See Attachment #1.
 Calibration factors used are as follows.
 C1:CAL-MICH_CINV FM3 is "PRMI_AS55Q" 1/1.45e10=6.9e-11 (from sensing matrix above)
 C1:CAL-MICH_A FM2 is 50.88e-09 (40m/17752)
 C1:CAL-PRCL_CINV FM3 is "PRMI_REFL11I" 1/1.16e12=8.6e-13 (from sensing matrix above)
 C1:CAL-PRCL_A FM2 is 41.40e-09 (40m/17752)

/Git/40m/measurements/LSC/MICH/MICH_Sensitivity_PRMI.xml
/Git/40m/measurements/LSC/PRMI/PRCL_Sensitivity_PRMI.xml

Sensing matrix comparison with past measurements:
 - It is pretty hard to read Gautam's radar plots, but REFL33 and REFL165 both had optical gain of ~10^7 V/m for PRCL and ~10^5 V/m for MICH in PRMI with no arms in 2021 (40m/15883)
 - From his thesis (see Figure 3.21), REFL33 and REFL165 had whitenings gain of 30 dB and 24 dB, respectivly, which are the same as the current gains.
 - Using ADC conversion of 2^16 counts/20 V, ~10^7 V/m for PRCL and ~10^5 V/m for MICH is ~3e11 counts/m for PRCL and ~3e9 counts/m for MICH. This is roughly consistent with the measurement above.
 - This means that REFL33 and REFL165 are probably working as they were in 2021.

Next:
 - Restore PRMI ASS. Alignment takes too much time. (AS WFS?)
 - Further tune REFL33 demodulation phase
 - Tweak suspension damping of ETMX (it is also contributing to ALS out-of-loop noise 40m/17773; coil balancing not enough? 40m/17771; oplev servo tweak necessary?)
 - Investigate ALS out-of-loop noise around 100 Hz (both ALSY 40m/17766 and ALSX 40m/17773)
 - Try PRMI 1f to 3f transition during both arms holded with ALS

After Yuta locked PRMI and measured the sensing matrix, I changed the input matrix to the following and measured the MICH and PRCL OLTFs.

              AS55Q           |      REFL11_I

MICH A:   0.99752413, 0.00247587

PRCL A:  -0.99752413, 0.99752413

This input matrix was calculated by taking the 2x2 submatrix of the AS55Q and REFL11_I components of the sensing matrix. I took the inverse and then scaled the first row by the MICH->AS55_Q sensing matrix element and likewise for the second row with the PRCL->REFL11_I sensing matrix element.

GPS time for data from lock with new input matrix: 1376274300 - 1376274400.

The OLTFs are saved in:

users/deven/PRCL_OLTF_8-16

users/deven/MICH_OLTF_8-16

dB(A) (A-weighted) scale and the Noise Criterion (NC) scale are popularly used in the United States for creating the spectrum. 

dB(A) Explanation: https://www.engineeringtoolbox.com/decibel-d_59.html

NC Explanation: https://www.engineeringtoolbox.com/nc-noise-criterion-d_725.html

(tldr: dBA applies an A-weighted filter to the dB scale to account for relative loudness that humans perceive at different frequencies. NC scale assigns a number, eg NC-55 such that at no frequency does the noise go beyond 55dB)

To capture this coherently, engineers usually use a SPL (Sound Pressure Level) Meter which is in essence, just another microphone. It is calibrated using a SPL Calibration Device which plays sound at a given loudness (usually 94dB, 114dB) and at a set frequency (usually 1kHz). You latch it on your SPL meter and make sure it is reading the output. 

The good thing about an SPL meter is, you don’t need to worry about the internal gains and so forth, it does all of that under the hood and gives you the measurement straight in decibels (A/B/C weighted if required).

The alternative to this suggested was using a UMIK 1 (a microphone that has a low noise floor - needs to be measured, online blogs mention it as ~30dbA). The suggested softwares are REW and Dirac Live (this one is paid).

UMIK 1 can be used in a couple of ways:

1. UMIK 1 + REW has a fair amount of documentation online for setting up. You can calibrate the microphone by pressing some keys which brings you in the green zone which the software likes (hopefully). This video explains it well.
   https://www.youtube.com/watch?v=3mOn6_3j5DU.
   UMIK-1 also comes with a calibration file, however one of the blogs mentions that it does not make a huge difference. This calibration file can be used directly in the REW software. This old graph does not show a huge difference in the bandwidth of interest between the factory settings and the calibrated settings. Additionally, it has a Sens Factor which I don't fully understand. (explained in this blog).
   The two REW features that may get the job done are the SPL Meter and the RTA (Real Time Analyzer). However, the issues with both are as described in this question I posted on their forum. https://www.avnirvana.com/threads/rew-umik-1-measurements-dba-nc.12420/#post-94154 (Update: There was a reply on the post needs to be checked out).

2. The signal can be read directly into something like Audacity and the time series can then be analyzed manually. However, the specification sheet for UMIK 1 does not provide it’s sensitivity so it’s difficult to know what it’s actually reading (have contacted the tech support, they said they’ll get back on Monday posted on a forum as well). A couple of recordings that were done on Audacity were of extremely low level using the UMIK 1 which probably means there’s some gain configurations that need to be figured out to get a good signal.

(Another microphone, the YETI Nano Blue may be used with Audacity, have requested for the specifications sheet. Update: 4.5mV/Pa at 1kHz).

Dismantled the seismometer circuit

At Rana's request, Deven and I have pulled all the wires out of the seismometer at the X end that Adviat set up the heater and sensors for. Right now, the seismometer is uncovered (no can) and the can I had to move next to the server rack that is close to the X end. I am attaching pictures of both the seismometer and the can.

I now realize that I have not turned off the power supply for the heater. If someone can please turn off the power supply that is shown in Advait's elog, please turn it off and reply to this elog. If you are unsure which power supply to turn off, read the first line of the elog that I replied to. Thanks!

Advait's circuits have been removed from the seismometer and stored in a box underneath the desk at the end of the control room (see picture). There is even a label on the box which reads "ANDREI CIRCUITS". At the seismometer there are still a few BNC cables left there: these are the cables that were used to interact with Advait's circuits. They have labels on them so that I can put back the circuits when I come back/when the seismometer can is available again

I've turned off the power supply. I've attached a picture of the rack with the switch that I flipped circled

There was a loss of N2 pressure over this weekend. When I came in to check the cylinder pressures, I was able to hear a hissing sound come from the copper fittings. I attached a photo of where the leak was coming from. I proceeded to tighten the fitting and check with soap and water for any leaks. To do this, I preffered to work with low pressure to make sure nothing would pop off while I was fixing this. Everything is back to normal here, but the vacuum interlocks tripped while I was working on the N2. I got the system back up to its nominal place by following the instructions on the wiki. I've attached a screenshot of what the Vacuum state is now.

[JC, paco]

This morning we noted most optics were tripped, probably as a result of a recent M>5 earthquake in the area (on Sun 08/20). Most optics were restored and damped nicely, except for ITMY.

PMC locked to HOM --> realigned and locked

We aligned PMC to maximize its transmission to ~ 0.670, after this IMC was locked and we engaged the WFS to recover the alignment.

ETMY oplev laser --> replaced aligned and locked

Most suspended optics were restored, but we noticed the OpLev sum on ETMY and ITMY were too low so we checked the lasers on both optics. The ITMY HeNe laser is on, but the one on ETMY is off. JC tested with a new laser head and the controller was determined to be good. Then, we tried resetting the previous one (labeled Oct 25 2020) but didn't have luck, so yet another HeNe laser died. We removed the old one and luckily our spare had the same form factor so it wasn't hard to recover the nominal alignment. After this we verified that the OPLEV loops on ETMY were working.

ITMY local damping --> still "stuck" or worse

The local damping on ITMY is not working properly. This puts it in a weird alignment state which is why we also don't see a large Oplev sum count on the QPD. The shadow sensor (OSEM) signals are all small, the available rms monitors are ~ 0.0, 0.1 mV, and kicking the optic around doesn't produce a corresponding OSEM signal, even when undamped. Therefore, we believe ITMY is either stuck (UR/LR) or worse. We tried the usual "shake" technique but didn't see any sensors being restored.

This is an update for 17794.

The UMIK 1 + REW combination gave satisfactory results for creating the acoustic noise spectrum for various spaces. This combination was corroborated using the NIOSH app on iOS (by OSHA) and the real time readings were usually in the +-2dB range of each other. dBA scale (reference values)  and NC scale (reference values) were used for measurement. Since the microphone is omnidirectional, the data was collected in the upright position. About 300 averages were taken for each reading and for open spaces, the data was collected with minimal activity (a few people walked by while collecting the stairwell data but they tried to be discreet). For closed spaces, readings were taken at 2/3 positions depending on the size of the room.

HIGHLY SUGGEST GETTING A CALIBRATION DEVICE IN THE FUTURE TO MAKE FAITHFUL MEASUREMENTS.

This is the Noise Floor of the UMIK-1 for reference (it was taken by covering it in a multiple layers of a bedsheet). The remaining readings can be found in ANS_Script_Data_Images.zip

The zip file contains the following:
Acoustic Noise Spectrum.pdf - This contains the keywords and spectrums consolidated in one pdf document
ANS_Positions.pdf - This contains images of the mic position while collecting the data
NC_Data_points.txt - This contains the data points used to generate the NC curves (Spline fit)
Spectrum Data - This folder contains the text files with the raw data to create the spectrum as well as some additional information about the recordings (source, date, etc)
Spectrum Images - This folder contains individual images for all the spaces
Final_Analyze_Signal.ipynb - Notebook used to create the spectrum from the text data

Update: Added spectrums for Downs-Lauritsen Rooms (226, 314, Sub Basement Corridor)
Update: Superimposed noise floor on each spectrum

[JC, Koji-remote, paco]

ITMY stuck --> Shaken remotely and restored, ARMS aligned

With Koji's assistance we restored ITMY (it was stuck) and finished aligning both arms. Then JC centered the OpLevs for ETMs, ITMs and BS

ITMY camera blinking --> Replaced camera

JC checked the situation with our ITMYF (face) camera as the image seemed faulty and blinking. The issue this time was not in the power supply as has been before, but rather the CCD itself. After replacing the unit and aligning the ARM cavity, we redrew the marker "guides" on the control room screen for quick reference.

[Ian, Torrey Cullen, Sander Vermeulen]

We are trying to calibrate one of the Wilcoxon accelerometers from the cryo lab to do a seismic study of campus. To calibrate it, we took data on Friday afternoon until about 6 pm for the Wilcoxon in the X, Y, and Z orientations and took cross-spectra with the seismometer down the end of the X arm from the channels C1:PEM-SEIS_EX_X_IN1, C1:PEM-SEIS_EX_Y_IN1, C1:PEM-SEIS_EX_Z_IN1. For the Wilcoxon, we used the channel from [17717] that was not being used. In the image of the panel in [17717] we tried channel 5, with the channel name C1:X01-MADC0_EPICS_CH28 but it was a slow channel. We asked Koji if there was a fast channel we could use, and he lent us channel 4 on that board with the channel name C1:ALS-X_SLOW_SERVO1_IN1. We took data from this channel to do our measurements. nothing was plugged into this channel when we started using it so we left it that way when we were done. 

I have attached our data.

NOTE: As it turns out the seismometer down the x end is not calibrated. We will recalibrate using the seismometer at the vertex

There is a version of this on the McCuller Logbook. It includes some plots. More non-40m related posts will continue there.

Photodiode inventory: [OMC ELOG 615]

[Paco]

Tuning the YAUX laser lowered the excess BEATY noise.

Since as of this post the only change in the YAUX setup was the death and replacement of the NPRO controller, I decided to play with the parameters. I found that increasing the laser power (and compensating for the frequency change by adjusting the temperature) successfully lowers the rms noise of the ALS beat. This is still not as good as it was before (1 Hz/rtHz at 100 Hz), but it is a hint of what may have happened. The initial settings were

ADJ = 0, T=43.6 deg

The final settings were:

ADJ = +6, T=25.8 deg

The maximum power adjustment is +10. Attachment #1 shows a reference (black) before the tuning was made, and after (red and cyan). The cyan trace has the noise eater off, while the red trace had noise eater on. There is no difference as per this measurement, so I left it ON.

We're trying to reduce the demodulated error signals after running the ASS script for the XARM.

After running the ASS script, we initially tried to play around with the with the EXC Gain and brought all of them down to 300. It didn't make a huge difference on the error signals or the transmission signal. We then tried tweaking the XARM_OUT_MTRX by flipping the signs/changing the magnitude but it mostly just made things worse. We then changed that matrix to closely resemble the YARM_OUT_MTRX (structurally). At an XARM GAIN of about 0.02, with the EXC Gains at 300 and the XARM_SEN_MTRX having 1.00 on the diagonal terms, the error signals slowly started converging to 0. However, X_ARM_ETM_PIT_L_DEMOD_I_OUT16 kept oscillating which wasn't good.

We later tried looking at the spectrum for the demodulated signals to see if there were any peaks at frequencies outside of the delmodulating frequencies. Most of them looked consistent with peaks at demodulation frequencies (and modes) and signal input frequencies (60Hz and modes). We compared the spectrum with the YARM where everything was optimized, there were no noticable differences.

Later in the day, both XARM and YARM lost lock a couple of times for reasons unknown. We restored to an earlier point in the day (12:00) suspecting there was misalignment with the input optics.

THE XARM ASS MYSTERY REMAINS.

Dolphin Investigation

- I made a basic description on a wiki page: https://wiki-40m.ligo.caltech.edu/CDS/DolphinSwitch

- Investigation crashed c1lsc/c1sus/c1iscex/c1sus2. Well, it's time to test the dolphin fencing. It seemed successful.

- Rebooted the crashed machines. I accidentally rebooted nodus, but Apache and elog were restarted.

- Burtrestoring to 18:19 snapshots. I suffered from the zero alignment gain issue, but the two arms are aligned and locked.

During the crash, I tried to reboot c1sus2 while the others were running. I actually did not install the script. It seems that it has been there since 2022 Sep.
Here is the instruction:

• Suppose you have one (or multiple) machines are dead (freeze, dolphin glitch, DK, etc).
• From the following list, determine which host you want to restart:
  PORT HOST
1    c1sus
2    c1lsc
3    c1iscex
4    c1iscey
5    c1ioo
6    c1sus2
• ssh into fb1. At the login directory, run the following command with the above port name (replace the "#" with it). If you have multiple hosts, run the command one by one.
  ./dolphin_ix_port_control.sh --disable 192.168.113.40 #
   
• ssh into the problematic machine. Use the following command to reboot it. Now this does not crash other machines!
  sudo reboot
   
• Once the machine starts rebooting, run the dolphin enabling command on fb1.
  ./dolphin_ix_port_control.sh --enable 192.168.113.40 #
  This should be done before the IOP (c1x07 etc) comes up. Otherwise, that IOP fails. It's allright. If the IOP (and other processes fails), just stop them with
  rtcds stop --all
  and enable dolphin with the above command. And then run
  rtcds start --all
   
• Once everything is up, burtrestore appropriate snapshots.

We can improve the process and the location of the script, but this is a good progress I suppose.

OMC Interface Aligner - (It's upside down...)

BHD OFI arrangement

With such a big temperature change, do you still get a reasonable beat note frequency? There's some previous elog of Koji I think that explains how we need to tune the lasers to get the 3 lasers to give 2 beat notes that are below 150 MHz.

Paco took the data which means he already had the beat note.

There is some chance that the beat was recovered after some mode "jumps" but usually the temp gap for the same beat frequency is ~2degC.
So my speculation is that there is a big temp gradient in the crystal now and had to compensate it with the struggle of the crystal TEC.

For the past data see https://nodus.ligo.caltech.edu:8081/40m/3759 or http://nodus.ligo.caltech.edu:8080/40m/12078
http://nodus.ligo.caltech.edu:8080/40m/4439 and so on.

A bit improved the design of OMC Interface Aligner

The idea is...The OMC I/F aligner covers the OMC for aligning the kinematic mounts (3 pairs of a V-groove and a ball) on the OMC. This makes the kinematic mount of two OMCs identical.

However, the OMC kinematic mount can't be adjusted because all the fasteners of the kinematic mounts are hidden by their counterparts.
We can copy the alignment of the OMC to the aligner, but the opposite is not possible.

Took the backup (snapshot) of nodus' /home/export as of Aug 25, 2023

controls@nodus> cd /cvs/cds/caltech/nodus_backup
controls@nodus> rsync -ah --progress --delete /home/export ./export_230825 >rsync.log&

40m BHD OFI Inventory

• OFI HWP 1
  • Motorized Rotary Stage Thorlabs PDR1V Qty 1
  • 0.5inch HWP: QWPO-1064-05-2 IDEX Optical Tech aka CVI Qty 1
  • Stainless SM5 retainer ring POLARIS-SM05RR (Qty 1 + spare 1)
  • Thorlabs KIM001
  • Power Supply KPS201
  • Post D2300286 (86.69mm = 3.413"), Newport Type https://dcc.ligo.org/LIGO-D2300286
  • Fork, Newport Type
• OFI TFP 1/2
  • Thorlabs LMR1V Qty2
  • Post (84.455mm = 3.325), Newport Type Qty2
  • Fork, Newport Type Qty2
  • 1" TFP obtained from LHO
• OFI FR
  • Already in hand
• OFI HWP2
  • D1600166 mount already in hand
  • 1" HWP already in hand
  • Post D2300287 (78.105mm = 3.075"), Newport Type https://dcc.ligo.org/LIGO-D2300287
  • Fork, Newport Type

Ready / Soon Ready
- BHD OMC Cables ready
- OMC#1 / OMC#4 ready
- BHD Platform parts being cleaned
- Assembly area HEPA being built
=> We will be soon ready to assemble BHD Platform and test with the OMC

In progress
- OMC locking setup (Moku)
- Connectors being attached to the BHD Platform actuators (picos & rotation stage)
- BHD Platform OFI parts drawing/procurement

- 40m BHD Electronics (BHD Adapter / DCPD TIA / Actuator driver I/F)

Other vent items
- In-vac ribbon cable holder (JC)

- Connector holder

- Scattered light control

- Pre-vent work
    * ASS recovery / extension

    * ETMX tuning
    * Vertex Eletronics upgrade
    * Fix PZT amps / PZT
    * Acromag

- Vent work items
    * New PR2
    * Alignment
 

The intented AOI for PR2 is 1.5 degrees. I averaged the peak measurements from the Moku:Go spectrum analyzer and from manual python FFT.

The transmissivities for p- and s-polarizations are:

Update to 17809. The free swing test for ETMX took quite some time, here's a brief summary of what was going wrong and a small update on it.

tldr: The scripts haven't been too helpful to obtain the resonant frequencies. Paco (to the rescue) suggested pulling up the raw data from the channels and get a spectrum manually. From the free swing test that was run on Wednesday (1:00am, was ended abruptly), there was enough data to be able to obtain the resonant frequencies for the Position (0.9524Hz) and Pitch (0.7238Hz) DOFs. The resonant frequencies for Yaw (0.8334Hz) and Side (1.0001Hz) were obtained as well (Aug_Res_Freqs.png).
The percent change in resonant frequencies from the last free swing test run by Yehonathan (17714) for all modes was < 2%. Thus, I will skip the diagonalization and moving to the next step of tuning the coils.

Procedure & Possible Changes

The data was collected in batches, for POS, PITCH (SUSfreeswing_ETMX_1377417624_POS_PIT_YAW_SIDE.txt) (1050 seconds, 15 kicks, 10000 counts) first and then for YAW (SUSfreeswing_ETMX_1377619412_YAW.txt) and SIDE (SUSfreeswing_ETMX_1377623258_SIDE.txt) (720 seconds, 5 kicks, 10000 counts) respectively. At some point, the default could be changed for freeSwing.py as the default setting (15 kicks, 1050 seconds) or 4 DOFs takes (4*15*1050/3600 = 17.5 hours) to new setting (5 kicks, 720 seconds) (4*5*720/3600 = 4 hours). getResFreqs.py is still troublesome so the analysis for this particular test was done in python. The notebook (freeSwingtest_Aug23.ipynb) is attached. It uses 2 really nice snippets of code that Paco has written to obtain the channel data and get the spectra (welch) .

Troublemakers
With some hardware changes to the ETMX (suspectedly the acromag), the script assumes some things which were important to run the test. Manual adjustments: Damping turned off, ramp times for DAMP FILTERS and Coil Outputs set to 0 in EPICS. 

The default value for the kick is 30000. However, kicking with this offset in the DOF basis gave clipping in the coil outputs. This was changed by giving it a 10000 offset using options.

Suspicion: In order to run the subsequent scripts (getResFreqs.py and sus_diagonalization.py), it's important to run freeSwing.py by giving the degrees of freedom for which you would want to obtain the resonant frequencies (eg  -k POS PIT YAW SIDE) as there is an internal dependency for it. Not sure how the UR Coil kick (default) is usually processed ahead (need to read into this in detail)

Even with the above changes, getResFreqs.py was having trouble reading data from the channel (ValueError: could not broadcast input array from shape (xxxx) into shape (xxxx)). Unsure why. 

With the same options (5 kicks, 720 seconds, 10000 counts) for the freeswing test conducted at 01:00am (SUSfreeswing_ETMX_1377565258_YAW_SIDE.txt) and 09:00am (SUSfreeswing_ETMX_1377619412_YAW.txt), the time series for YAW looks terrible for the former (?????). The comparisions are attached (freeswing_yaw_5kicks_10000counts_720s_good.png , freeswing_yaw_5kicks_10000counts_720s_bad.png)

Resonant Frequency in Position:  0.9524518193317955 Hz
Resonant Frequency in Pitch:  0.7238633826921645 Hz
Resonant Frequency in Yaw:  0.833423765599566 Hz
Resonant Frequency in Side:  1.0001085187194791 Hz

[Yuta, Hiroki]

*This work was done on Aug. 10th.

Using the calibration function of sitemap/CAL, we measured the MICH displacement spectra in various conditions (Attachment 1).
Optical gains for the calibration were measured before each measurement.
Regarding the actuator transfer function, we used the result from elog #17752.
The measurement conditions are as follows:

• C1:CAL-MICH_W_OUT_DQ (AS55_Q, March 2023)
  Previous result measured in the simple MI locked with AS55_Q.
   
• REFL55_Q dark noise (August 2023)
  Dark noise of REFL55_Q. 
   
• C1:CAL-MICH_W_OUT_DQ (AS55_Q, August 2023)
  Measured in the simple MI locked with AS55_Q (ETMX and ETMY were misaligned).

• C1:CAL-MICH_W_OUT_DQ (REFL55_Q, August 2023)
  Measured in the simple MI locked with REFL55_Q (ETMX and ETMY were misaligned).
  REFL55_Q: 3.02e8 counts/m
  MICH gain: -8
  UGF: ~60Hz
   
• C1:CAL-MICH_W_OUT_DQ (REFL55_Q, POXY, August 2023)
  Measured in the FPMI locked with REFL55_Q, POX11_I and POY11_I.
  REFL55_Q: 2.08e8 counts/m
   
• C1:CAL-MICH_W_OUT_DQ (REFL55_Q, ALSXYdetuned, August 2023)
  Measured in the detuned FPMI locked with REFL55_Q, ALSX and ALSY.
  REFL55_Q: 4.22e8 counts/m
  MICH gain: -8
   
• C1:CAL-MICH_W_OUT_DQ (ASDC,ALSXY, August, 2023)
  We tried to measure the MICH noise of FPMI locked with REFL55_Q, ALSX and ALSY but failed probably due to the ALS noise.
  We succeede in locking MICH with ASDC instead, so we measured the noise with this condition.
  ASDC: 1.19e6 counts/m
  MICH gain: -200

Discussion:

• (AS55_Q, August 2023) is larger than (AS55_Q, March 2023) over a broad frequency range. The effect of *2 gain of replaced anti-imaging modules (elog #17738) is not relevant to this discrepancy because the effect was taken into account as the new actuator transfer function(elog #17752). Therefore, the discrepancy might be due to the incorrect result of the optical gain by this measurement or the previous measurement.
   
• (REFL55_Q, August 2023) is almost the same as (AS55_Q, August 2023) up to ~ 6 Hz, but is noisy with 1/f noise from~  6 Hz to ~ 100 Hz. This 1/f noise cannot be the laser frequency noise because the NPRO laser frequency noise can be estimated as 2e-12/f m/rtHz (assuming NPRO frequency noise: 1e4 Hz/rtHz and Schnupp asymmetry: 3.5 cm) and it is even suppressed by IMC. We should identify the noise source of this 1/f noise.
   
• There is a peak around ~ 200 Hz in (REFL55_Q, August 2023) but this is not the gain peaking because its UGF is ~ 60 Hz.
   
• (REFL55_Q, POXY, August 2023) is almost the same as (REFL55_Q, August 2023) up to ~ 10 Hz but is much noisy around 30 Hz with a wide peak. This peak might be due to the fluctuation of the arm cavities. On another note, the peak around 200 Hz in (REFL55_Q, August 2023) disappeared in (REFL55_Q, POXY, August 2023) for some reason.
   
• (REFL55_Q, ALSXYdetuned, August 2023) is almost the same as (REFL55_Q, August 2023) up to ~ 30 Hz. However, above ~ 30 Hz, it gets smaller than (REFL55_Q, August 2023). This seems very strange for me because (REFL55_Q, ALSXYdetuned, August 2023) had aligned ETMX and ETMY and they can contribute to additional noise even if arm cavities are detuned.

[Paco, Radhika]

ALS control of CARM

Yesterday evening, Paco and I aimed to:

Once e-FPMI was locked (POX + POY --> CARM_A), we fed the ALS beatnote error signals to CARM_B and slowly mixed CARM_A and CARM_B. ALS control of CARM was successful.

The final values used in C1LSC_AUX_ERR_MTRX were (-0.3 ALSX + 0.3 ALSY) --> CARM_B. Note that these signs depend on the sign of each beatnote. The sign of ALSY could be determined by giving an offset, but without an Acromag we had to use trial and error for the sign of ALSX. We observed that using 0.5 magnitude for each signal resulted in too high of a CARM UGF, making the loop unstable. The magnitudes were reduced to 0.3 to give us a comparable UGF to POX/POY control of CARM.

The final ALS CARM OLTF can be found in Attachment 1. Some "wobblyness" was observed in the OLTF. Attachment 2 shows the suppressed in-loop CARM_B error and the out-of-loop CARM_A error. We couldn't identify why CARM_A error has a notch ~325 Hz; this is also present when closing the loop with CARM_A.

We tried to add an LSC CARM offset (would push the PSL frequency away) but could not see the transmission in the arms drop.

Next steps

Increase stability of ALS CARM, turning loop gains

Achieve a CARM offset maintaining lock

Then proceed to lock PRMI sidebands and reduce the CARM offset for PRFPMI

[Radhika, Murtaza]

XARM ASS

We resumed playing around with ASS for XARM. We approached each error signal one at a time to try to determine the sign of actuation and ensure the error was reduced.

Steps:

Next steps:

1. Resume nailing down the "fast" control by using ITM transmission error signal to feedback to ITM

2. Add in "slow" pointing control by feeding back ETM LSC error (centering) to BS.

     *NOTE* We tried to do this before feeding back any ITM error signal, but this immediately caused transmission to drop because ITM had no way to adjust to new input pointing.

[Yuta, Hiroki]

*This work was done on Aug. 10th.

We measured the ALSX noise and ALSY noise on Aug. 10th as shown in Attachment 1.
We used a digagui template to measure the noise and it had the reference of the previous measurement (but the date was not shown).
The measured ALSX this time (red) was noisier compared to the previous result (magenta) in almost all the frequency range.
The measured ALSY (blue) was noisier than the previous result (cyan) above ~ 50 Hz with the flat shape but was better below ~ 10 Hz for some reason.

*This work was done on Aug. 11th.

I put away the beam chopper used in T&R measurement into the shelf in YARM (Attachment 1).

[Koji, Hiroki]

*This work was done on Aug. 11th.

We found a fiber-coupled visible diode laser (iFLEX-1000) in a shelf on YARM (Attachment 1 and 2).
This laser may be used as the replacement of the He-Ne laser for OPLEV.

I recommend usinng the DC offset method that Koji and I used for measuring the IMC WFS sensing matrix (not the AC method that Anchal used). With a sensing matrix, you should be able to do some partial inversion.

Without any sensing matrix inversion, we would have to rely on a gain hierarchy for getting the loops to work.

With some approximate matrix inversion, the loops are more indepedent of each other. Also if you look at the spectrum of the error signals, it should be clear that the sensing noise is pretty large, and so that sets a natural upper limit to the UGFs. We only want integrator (1/f) loops, but the LPFs cause some extra phase lag.

[Radhika, Murtaza]

To create the sensing matrix, we tried the DC offset method by giving offsets in the Pitch and Yaw DOFs for ITMX, ETMX and the BS respectively. The signals we looked at were the demodulated ETMX_L, ETMX_T and ITMX_T. We wrote a quick notebook that does the following things for each DOF:

1. Calculate the mean error signal (over 10s)
2. Give an offset of 3 steps in each DOF corresponding to their step size serially with some buffer time (restoring the offset after each DOF)
3. Calculate the new mean error signal (over 10s)
4. Find the difference in error signals and divide by their respective step sizes to get each sensor's sensitivity to the offset.
5. Invert to obtain the sensing matrix.

Sensing Matrix for Pitch:

=

Sensing Matrix for Yaw:

 = 

*NEED TO TRY THESE OUT*

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

The Output Matrix for the from intuition was set to *Attachment 2* which improved the net average transmission (see Attachment 1), but wasn't really stable after the improvement.

COIL BALANCING FOR ETMX
Summary : I ran coil balancing on ETMX using the CoilStrengthBalancing.ipynb script to get a feel for it, no changes were required from the last time it was run by Paco. I realized I was measuring the wrong signal for the POS coupling (C1:SUS-ETMX_SUSPOS_IN1) while trying to minimize the BUT-POS coupling. This was stupid because the shadow sensors and actuation coils in this case are the OSEMs. The LSC error signal would be more appropriate for measuring the POS coupling.

The convention for the actuation vector used for the coils is [UL, UR, LL, LR]. The frequency, excitation counts are given through the LOCKIN1 channel (SUS->ETMX->LOCKIN1->f(Hz), Amp). The excitation vector is set in (SUS->ETMX->Output Filters->LOCKIN1)
Here, the excitation frequency used = 13Hz.
For "decoupling" the degrees of freedom, the script used is given in /opt/rtcds/caltech/c1/Git/40m/scripts/SUS/coilStrengthBalancing/ETMX/CoilStrengthBalancing.ipynb. In here, small steps are taken to "remove" the DOF contribution such that \\new_gain = old_gain +/- e*(decoupling DOF vector)\\. (e = step size)
The decoupling signals are observed in diaggui in the frequency range of (0 - 20Hz) and a bandwidth of 0.5Hz with exponential averaging (10 averages)

1. Minimizing BUT-POS coupling
Here, the LSC error signal (C1:LSC-POX11_I_IN1) is observed to measure the coupling in POS. For this, the arm is kept locked to obtain a decent error signal.
SANITY CHECK: This was tested by exciting POS [1,1,1,1] at 13Hz and measuring at the LSC error signal in diaggui which indeed showed a peak at 13Hz indeed. Damp filters and OPLEV servos were enabled to prevent the loss of lock.
The initial excitation was given to the Butterfly DOF [1, -1, -1, 1] at 10000 counts and 5 steps were taken in both directions of the POS vector to decouple the POS DOF. The initial peak showing up in the LSC error signal was already at a minimum. The excitation was ramped up to 20000 counts, where the peak was still very small. Thus, no change was made here (Attachment 1).

2. Minimizing POS-PIT coupling
Here, the OPLEV signal for PIT (C1:SUS-ETMX_OL_PIT_IN1) is observed to measure the coupling in PIT. The damping filters and OPLEV servos are disabled.
The initial excitation is given to POS [1, 1, 1, 1] at 5000 counts and 5 steps were taken in both directions of the PIT vector to decouple the PIT DOF. The initial peak showing up in the OPLEV signal was already at a minimum. Thus, no change was made here (Attachment 2).

3. Minimizing POS-PIT coupling
Here, the OPLEV signal for YAW (C1:SUS-ETMX_OL_YAW_IN1) is observed to measure the coupling in YAW. The damping filters and OPLEV servos ared disabled.
The initial excitation is given to POS [1, 1, 1, 1] at 5000 counts and 5 steps were taken in both directions of the YAW vector to decouple the YAW DOF. The initial peak showing up in the OPLEV signal was already at a minimum. Thus, no change was made here (Attachment 3).

4. Minimizing PIT-YAW coupling
This was one rather robust and was not susceptible to the decoupling process. Here, the OPLEV signal for both PIT (C1:SUS-ETMX_OL_PIT_IN1) and YAW (C1:SUS-ETMX_OL_YAW_IN1) are used to measure their relative coupling. Either of the DOF can be excited and while the other DOF can be used for the decoupling vector. Here, PIT was excited and the decoupling DOF vector was YAW. The damping filters and OPLEV servos ared disabled.
The initial excitation was given to the PIT DOF [1, 1, -1, -1] at 5000 counts and 5 steps were taken in both directions of the YAW vector to decouple the YAW DOF. The initial peak showing up in the YAW signal was already at a minimum. The excitation was ramped up to 10000 counts, however the YAW peak barely moved. Thus, no change was made here (Attachment 4).

To next.

[Radhika, Murtaza]

We recalculated the sensing matrix for XARM ASS by collecting each sensor's step response to an offset in each DOF. This produced the following dense output matrix A (see Attachment 1 for rows/cols):

      [[-0.02047695,  0.        , -0.10262752,  0.        , -0.0157128 , 0.        ],
       [ 0.        ,  0.16908344,  0.        , -0.00929291,  0.        ,-0.35916455],
A =    [-0.28050764,  0.        ,  0.26982002,  0.        , -0.55100297, 0.        ],
       [ 0.        ,  0.85501491,  0.        ,  0.0606197 ,  0.        , 0.27568672],
       [-0.95963335,  0.        ,  0.95742611,  0.        ,  0.83435534, 0.        ],
       [ 0.        , -0.49026554,  0.        , -0.99811768,  0.        , 0.89162641]]

Turning the XASS gain up slowly to ~0.15, we observed that several error signals diverged and transmission started to drop. Debugging this matrix proved difficult since there were many nonzero elements to consider. So we reverted to build the matrix from our intuition, considering the centering and input pointing loops, and using the YASS output matrix as a reference.

The YARM ASS servo gains are all +1. The YASS output matrix has the following length (centering) signal mapping:

We mirrored this in the XASS output matrix. Note that previously the ITM L error signals were not used for XASS. To simplify the process, we decided to just work out the beam centering first and ignore the input pointing coming from the beam splitter (setting BS PIT/YAW matrix elements to 0). We also set all the XARM ASS servo gains to +1. See the output matrix below:

We cleared the outputs and turned on the XARM GAIN slowly (0.1) and immediately noticed the YAW signals in ETM start to diverge (C1:ASS-XARM_ETM_YAW_T_DEMOD_I_OUT16, C1:ASS-XARM_ETM_YAW_L_DEMOD_I_OUT16). We turned down the XARM gain and flipped the sign for the signal going to ETM YAW. (suspect a difference in sign convention).

To check the stability, we sequentially gave offsets in PIT/YAW for ETM and ITM. We saw the signal (C1:ASS-XARM_ETM_PIT_L_DEMOD_I_OUT16) oscillate wildly at a frequency of ~(1/15 Hz). We suspected the ASS loop was driving these oscillations so we turned down the gain going to ETM PIT to 0.25 which worked really well and the transient oscillation of further checks was gone.

We saw similar wild oscillatory signals in ITM PIT (C1:ASS-XARM_ITM_PIT_T_DEMOD_I_OUT16, C1:ASS-XARM_ITM_PIT_L_DEMOD_I_OUT16) on applying offsets so we reduced the gain going to ITM PIT to 0.3. (0.25 and 0.3 are arbitary relatively smaller weights, can be fine tuned).

We checked the stability of this setup as a whole by giving a few offsets to ITMX and ETMX, with a servo gain of 0.15 it did a great job! (0.25 made it diverge once again). See final state for centering in Attachment 1, and error signal suppression in Attachment 2. (Ignore XAUX transmission in grey - we were toggling the shutter.) Note that the Length error signals were successfully suppressed, but the dark green/brown Transmission error signals were not fed back and thus remain nonzero.

WE SHALL INVESTIGATE THE INPUT POINTING NEXT FEEDING BACK TO THE BS and ITMX. We will give an update shortly about whether restoring XARM ASS is feasible by Monday.

Tonight I managed to lock CARM and DARM under ALS control only

Configuration

• Arm cavities well aligned, TRY ~ 1.07, TRX ~ 0.98, GTRY ~ 1.2, GTRX ~ 0.77
• HEPA off, WFS 60s offloaded, PD offsets removed, all lights inside the lab were off
• BEATX and BEATY ool residual noise shown in Attachment #1.
• Error points, the A path was the same as what is used for electronic FPMI. For the B path, I describe the tuning below.
  • CARM_A = 0.5 * POX11_I + 0.5 * POY11_I
  • DARM_A = 0.19 * POX11_I - 0.19 * POY11_I
  • CARM_B = -0.7 * ALSX + 0.4 * ALSY  
  • DARM_B = -0.25 * ALSX - 0.17 * ALSY
• Power normalization of the error signals was 0.5 * TRX + 0.5 * TRY in both paths.
• LSC filter banks are the same ones we use for electronic FPMI, and the gains were
  • CARM = 0.011  (UGF ~ 200 Hz) using FM1 to FM5, FM6 and FM8
  • DARM = 0.055 (UGF ~ 150 Hz) using FM1 to FM5, FM6 and FM8
• Control points, I temporarily disabled violin filters around 600 Hz to ease the lock acquisition ... we should really use the VIO TRIG here to avoid having to do this.
  • CARM = -0.734 * MC2
  • DARM = 0.5 * ETMX - 0.5 * ETMY

ALS error signal tuning

To find the error signals for CARM/DARM, I turned on the oscillators (at 307.8 and 313.31 Hz respectively) with 150 counts and enabled FM10 (Notch for sensing matrix) in the CARM and DARM servo banks. I then removed the ALS offsets (C1:LSC-ALSX_OFFSET, C1:LSC-ALSY_OFFSET) and looked at the transfer functions shown in Attachment #2. I optimized the ALS blending until I maximized the CARM and DARM A to B paths and minimized CARM and DARM cross couplings. The signs were chosen to leave a phase of 0.

Handoff

After measuring the OLTFs for eCARM and eDARM (loop closed with the A error point) and tuning the ALS error signals, I gradually blended the A and B paths and checked the OLTFs for CARM and DARM. During this I realized I needed to disable some of the notch violin filters because they sometimes made the DARM loop unstable after >50% blending. In the end the simultaneous CARM_A/DARM_A to CARM_B/DARM_B handoff was successful in 0.5 seconds. Attachment #3 shows the OLTFs under ALS control.

CARM offset

After getting nominally stable ALS control, I tried adding an offset. The LSC CARM offset range was insufficient, so I ended up directly scanning the C1:LSC-ALSX_OFFSET and C1:LSC-ALSY_OFFSET. The first couple of attempts the ramp time was set to 2.0 seconds, and a step of 0.01 was enough to break the lock. I managed to hold the control with as much as C1:LSC_CARM_A_IN1 offset by ~ 500 (rms ~ 200 counts). I roughly estimate this to be ~ 5% of the CARM pole which is 4 kHz in this case so overall 200 Hz which is not that large.

I came to the lab to see the recovery work from the power glitch this morning 8:15AM. All CDS seems up. The suspensions are somewhat aligned. Some of them were not damped. The oplevs were off. Radhika is working on the recovery of the FP arms.

I noticed that FSS Slow servo is not working. I always forget what is the right way to turn it on. Here is the summary:

How to turn on FSSSlow (2023 Sept version)

• Go to megatron
• sudo systemctl enable FSSSlow
• sudo systemctl start FSSSlow
• sudo systemctl status FSSSlow

  ● FSSSlow.service - Script to run the PID temperature control servo for the PSL
     Loaded: loaded (/opt/rtcds/caltech/c1/Git/40m/scripts/PSL/FSS/FSSSlow.service; enabled; vendor pre
     Active: active (running) since Fri 2023-09-08 17:10:52 PDT; 1s ago
   Main PID: 2088 (python3)
      Tasks: 6 (limit: 4674)
     CGroup: /system.slice/FSSSlow.service
             └─2088 /usr/bin/python3 /opt/rtcds/caltech/c1/Git/40m/scripts/PSL/FSS/PIDLocker.py PIDConf

  Sep 08 17:10:52 megatron systemd[1]: Started Script to run the PID temperature control servo for the

[Radhika, Murtaza]

This post summarizes XARM ASS efforts from Friday 9/8 and Saturday 9/9.

FRIDAY

On Friday, we continued with our previous output matrix that used the length error signals (ITM/ETM PIT/YAW L) to feed back to ITMX and ETMX (see the previous ELOG). In that state we did not use the transmission error signals and had no feedback going to the BS. We then tried to use the transmission error signals ITM PIT/YAW L as a proxy for BS input pointing and feed them back to the BS. For both PIT and YAW, both signs of feedback resulted in diverging T error signals and a decrease in transmission.

SATURDAY

On Saturday, we used the transmission error signals (ITM/ETM PIT/YAW T) in the sensing matrix to build the output matrix. We got it to a state where we could get the controlled error signals to converge by just feeding back to the ITMX and ETMX (Attachments 1,2). Once we had this working, we tried to feed back a combination of (ETM PIT/YAW L and ITM PIT/YAW T) to correct BS pointing. However, any combinations and signs to the BS dropped transmission and led to diverging error signals.
We then attempted to use the latest working XASS output matrix (before the acromags were pulled out) and see the effect of flipping signs in there (one optic+DOF at a time) We then tried to use the sign logic from the previously working ETM/ITM feedback we got partially working; however the error signals did not converge with any combination.

SUMMARY:

- We are able to successfully feed back to ITMX and ETMX, using either length or transmission error signals. It is when we try to add BS feedback that ASS fails. This can be due to the fact that we need to consider the relative servo gains when treating these loops separately, like Koji mentioned.

- The sensing matrix approach might be the only way to simultaneously optimize feedback for all optics, avoiding the need to tune servo gains. We will revisit this approach on Monday.

     - Koji pointed out that we are reading out the low-passed error signals in order to calculate each step response - we will need to consider our sampling rate and duration of averaging accordingly.

     - It will be harder to iteratively flip signs of each matrix element for this dense matrix, and we will have to be clever about which sign combinations we try for actuation.

[Radhika, Murtaza, Paco]

Today we decided to take a closer look at the demod phases of the T and L error signals for XARM ASS. By eye we tuned the phases to minimize the signal in Q. Here are the new demod phases:
(THE DEMODULATION PHASE VALUE DO NOT RESTORE BACK TO THE ORIGINAL VALUES WHEN DITHER IS TURNED ON.)

We also noticed that MEDM indicator for dithering on (white --> green LO symbol) for ETM_YAW_L_OSC was tied to the wrong excitation gain channel (C1:ASS-XARM_ITM_YAW_OSC_CLKGAIN instead of C1:ASS-XARM_ETM_YAW_OSC_CLKGAIN). We went ahead and changed this in [insert medm file location]. So now the right green LO symbol appears when the appropriate excitation is turned on.

For the preparation of the electronics upgrade, three 18bit DAC AI chassis were transformed to 16bit version.

The power supply connections were touched, so the units were tested with +/-18V, and they work as expected.

Dolphin Fencing technique

I believe that the dolphin emits some glitches to the other hosts during the host machine shutting down and starting up.
However, if the dolphin is disabled, that FE process will not run.

Therefore, we need some technique:

• When you have a real-time host to be restarted, we can disable the dolphin of that machine.
  e.g. If c1lsc has a problem, run the following command on fb1.
  ./dolphin_ix_port_control.sh --disable 192.168.113.40 2
  This allows us to restart the c1lsc in a safe way.
   
• Restart c1lsc in the above example. Go to c1lsc and run
  sudo reboot; exit
   
• This above brings you back to the previous host you were (suppose it is fb1). Run ping on that restarting machine.
  ping c1lsc
   
• While the c1lsc is shutting down, ping still has the response. Once the restart starts, it makes no response. Then, you can enable dolphin.
  ./dolphin_ix_port_control.sh --enable 192.168.113.40 2
   
• The process comes back automatically. You'll see DK status during the restart. I should disappear once all the models are up.

[JC, Paco, Koji]

We had a power outage on Sept 21, 2023 at ~9AM. This is the third power outage this month as far as I remember.

- JC reported the outage was ~2sec. Some UPS supported machines were affected, while some unsupported machines also survived the incident cf c1psl (what!?)

- Some machines were rebooted by itself (cf the RTS hosts).

- megatron and optimus were powered up. Autolockers (optimus) and FSSSlow (megatron) were restored.

c1vac was still on, but the machine didn't come back online.

- The network adapter was reactivated by running the following commands

> cd /sbin
> sudo ifdown eth0
> sudo ifup eth0

However, the acromag seemed freezer, so c1vac was shutdown, the acromag chassis was power cycled, and the c1vac was rebooted. This brought c1vac fully functional again.
Rebooting made TP1 stop (gracefully)

The vacuum pressure of the main volume was high.

- We found that the vacuum pressure was up to 1e-2 torr in the afternoon when we started the recovery. In fact, the main gate valve was close at the power outage last week. See attachments.

- We made sure the valves were properly open/closed and started TP1 again. Once TP1 reached 33.6K RPM we opened the main volume to recover the vacuum pressure.

- The vacuum pressure came back to <1e5 Torr.

Pumping configuration changed. Now TP2 is backing TP1 and TP3 is pumping annuli.

Fixed IMC/IFO alignment screens

[Paco, Murtaza, JC]

Fixed C1SUS2 Crash from this morning

What we did:
  - Attempt to restart only C1SUS2
  - Restart All Machines and Burt Restored to Friday @ 7:19 pm

Summary:

Entering this morning, We were unsure why we were having issues aligning and come to find out that C1SUS2 crashed. Paco attempted to restore by restarted the machine individually and restoring, while this did turn all the machines green on the CDS.MEDM Screen, it did not resolve the issue. So moving forward, please keep in mind that EVEN IF ALL MACHINES SHOW GREEN, OPTICS MAY STILL NOT BE DAMPING. 

Next we continued to restart and reboot the old fashioned way.

I attempted to use the ./restartAllModels.sh script in the "/opt/rtcds/caltech/c1/Git/40m/scripts/cds" directory, but there was an error and the message I got said something along the lines of "Restart medm screen and try again". This was weird and all of the machines were already shutdown. So, to bring them back up, I used the ./startAllModels.sh script. When starting up, i was prompted to provide a burtrestore directory, and I inputted /opt/rtcds/caltech/c1/burt/autoburt/snapshots/2023/Sep/22/20:19.

This worked out and IMC came back to nominal alignment. The primary issue we seem to be coming across now is that C1:IOO-WFS2_PIT_OUTPUT is increasing at a steady rate and this is disrupting our alignment.

[JC, Paco, Radhika, Murtaza]

1. WFS Relief
We tried to change the gain to offload the offsets in the reliefMCWFS script but it didn't work. The gains might need some tuning to get it to work

2. WFS Error Signals Diverging
The error signal C1:IOO-WFS2_I_PIT_MON was staying at a constant offset from 0 using the existing output matrix (C1IOO_WFS_OUTMATRIX). We tried changing the matrix coefficients that may have caused this behavior but it led to divergence in other signals (C1:IOO-WFS1_I_PIT_MON, C1:IOO-WFS2_I_YAW_MON). THIS NEEDS TO BE FIXED

3. BS was aligned using OPLEV readouts and the damping filters were checked. No funny business for BS anymore.

4. The original IFO Align scripts used the suffix "COMM" for each optic. This was changed to "OFFSET" for all arguments by editing the IFO_ALIGN screen (left click-> Execute -> Edit this screen -> !Align -> Label/Cmd/Args)

5. The OPLEV gains for ITMY were unstable and needed some tuning. New gains: C1:SUS-ITMY_OL_PIT_GAIN (14->3.5) and C1:SUS-ITMY_OL_YAW_GAIN (-8->-4). (The upgrade should not have affected this so this could be revisited later).

6. YARM (transmission ~ 1) and XARM (transmission ~ 0.6) were locked successfully!