40m QIL Cryo_Lab CTN SUS_Lab CAML OMC_Lab CRIME_Lab FEA ENG_Labs OptContFac Mariner WBEEShop
  40m Log, Page 56 of 354  Not logged in ELOG logo
ID Date Author Type Categorydown Subject
  15887   Tue Mar 9 14:37:26 2021 gautamSummarySUSPRM suspension

The PRM got tripped ~5AM this morning. The cause is unclear - the seismometer reports elevated activity ~10 minutes before the ringdown starts (as judged using the OSEMs). But the other optics didn't seem to receive as much of an impulse (I only show the BS sensors here as it sits on the same stack as the PRM). Anyway it certainly wasn't me trying to make life difficult for the morning team.

I was able to restore the damping with reEnableWatchdogs.py. I am now running some suspension tests on the PRM by letting it swing freely so please let that finish. I plan to attempt some locking this evening.

Quote:

[Paco, Anchal]

- Upon arrival, MC is locked, and we can see light in MON5 (PRM) (usually dark).

Attachment 1: PRMtrip.png
PRMtrip.png
  15888   Tue Mar 9 15:19:03 2021 KojiUpdateSUSOSEM testing for SOSs

How were the statistics of them? i.e. # of Good OSEMs, # of OK OSEMs, etc...

  15889   Tue Mar 9 15:22:56 2021 KojiSummarySUSPRM suspension

I just saw the PRM watchdog tripped at ~15:20 local (23:20UTC). I restored the PRM but I saw only the side watchdog tripped.
Again at 15:27

17:55 I found the PRM was oscillating while the watchdogs were not tripped. I turned off the OPLEV servos and this made the PRM calmed down. But I didn't turn on the OPLEVs for the past two trips. How were the OPLEVs turned on???

Ah, I'm sorry, I missed the line that Gautam was running the free-swinging test on the PRM.
The two kicks starting from 23:08:50 and from 23:26:31 were spoiled. Did it make the measurement completely waisted?

 

  15891   Tue Mar 9 18:49:28 2021 YehonathanUpdateSUSOSEM testing for SOSs

29 Good OSEMs, of which 1 is questionable (089) with PD voltage of 1.5V, 5 need some work (pigtailing, replace/remove/add screws). We have 4 pigtails. Schematics.

20 OK OSEMs (Slightly off-centered LED spot), of which 3 need some work (pigtailing, replace/remove/add screws).

13 Bad OSEMS (Way off-centered LED spot)

2 Defunct OSEMs

-------

Ed: KA
Good: 23 complete OSEMs +  5 good ones, which need soldering work (there are 4 pigtails and take one from a defunct OSEM).
OK:  Use good 7 OSEMs for the sides. And keep some functional OSEMs as spares.

 

  15894   Wed Mar 10 11:55:22 2021 gautamUpdateSUSPRM suspension suspect

The procedure is that the optic is kicked to excite it, and allowed to ring down for ~1ksec, with damping turned off. The procedure is repeated 15 times for some averaging. 

Attachment #1 - sensor spectra from yesterday.

Attachment #2 - peaks using the naive diagonalization matrix from yesterday.

Attachment #3 - Data from ~1 year ago. 

The y-axis in all plots is labelled as "cts/rtHz" but these are the DQed channels, which come after a "cts2um" CDS filter - so if that filter is accurate, them the y-axes may be read as um/rtHz.

I wonder if the September 2020 earthquake somehow damaged the PRM suspension, as this experiment would suggest that the problem is not only with the actuation. The data was gathered with the neutral position of the PRM (between kicks) being well aligned for PRMI, and the DC values of all the shadow sensors in this position is close to half-light (~1V, except for side which was more like 4V). Hard to say what exactly is happening since only the PIT DoF has the weird asymmetric peak shape instead of the expected Lorentzian - I would have thought that a damaged wire or broken magnet would affect all 4 DoFs but the F.C. spring experience on ETMY showed that anything is possible.

Attachment 1: PRM_sensorSpectra.pdf
PRM_sensorSpectra.pdf
Attachment 2: PRM_pkFitNaive.pdf
PRM_pkFitNaive.pdf
Attachment 3: PRM_diagComp.pdf
PRM_diagComp.pdf
  15898   Wed Mar 10 17:35:47 2021 gautamUpdateSUSSpooky action at a distance

As I am sitting in the control room, the PRM suspension watchdog tripped again. This time, there is clearly no seismic activity. Yet, the BS suspension also shows a slight disturbance at the same time as the PRM. ITMY shows no perturbation though. My best hypothesis here is that the problem is electrical. In Attachment #1, you can see that all of the Sensors go to -6000 cts (whut?) for ~30 seconds. Zooming in to that segment in Attachment #2, it would appear that the light detected by the LED changed dramatically (went dark?) on all 5 coils. The 4 face coils have the same time constant but the side has a different one, but in any case, this level of light change in half a second is clearly not physical. Then the watchdog trips because this huge apparent motion elicits a kick from the damping loops.

The plots I attach are for the DQed sensor channels, so there is some digital filtering involved. But I confirmed that the signal doesn't go negative if I disable the input to the filter module. So it would seem that the voltage input to the ADC really chanegd polarity, seems unphysical. Could be Satellite Box or whitening electronics I suppose - I think we can exclude bad cabling, as that would just lead to the signals going to 0, whereas it would appear here that they did really change sign (confirmed by looking at the ULPDmon channel, which is digitized by Acromag, which reports -10 V at the time of glitch). But why should the BS care about the PRM electronics going wonky?

In addition to an exorcist, we need functioning electronics!


This optic has been hampering my locking attempts all evening. I switched the PRM and SRM satellite boxes, but then I remembered PRM has the Al foil "hats" to attenuate scattered light. of course the Al foil is conducting and can short the OSEM leads. I put some kapton pieces in between OSEM and foil to try and mitigate this issue but I suppose over time it could have slipped, and is making some intermittent contact, shorting PD anode and cathode (that would explain the PD reporting -10 V instead of some physical value).

If this is the problem we would need a vent to address it. In the daytime I'll measure L and R of the coils to see if the actuator imbalance I reported is also due to the same problem...

Attachment 1: PRMtrip.png
PRMtrip.png
Attachment 2: PRMtrip_zoom.png
PRMtrip_zoom.png
  15902   Thu Mar 11 08:13:24 2021 Paco, AnchalUpdateSUSIMC First Free Swing Test failed due to typo, restarting now

[Paco, Anchal]

The triggered code went on at 5:00 am today but a last minute change I made yesterday to increase number of repititions had an error and caused the script to exit putting everything back to normal. So as we came in the morning, we found the mode cleaner locked continuously after one free swing attempt at 5:00 am. I've fixed the script and ran it for 2 hours starting at 8;10 am. Our plan is to get some data atleast to play with when we are here. If the duration is not long enough, we'll try to run this again tomorrow morning. The new script is running on same tmux session 'MCFreeSwingTest' on Rossa

10:13 the script finished and IMC recovered lock.

Thu Mar 11 10:58:27 2021

The test ran succefully with the mode cleaner optics coming back to normal in the end of it. We wrote some scripts to read data and analyze it. More will come in future posts. No other changes were made today to the systems.

  15907   Fri Mar 12 03:08:23 2021 KojiSummarySUSCoil Rs & Ls for PRM/BS/SRM

Summary

Per Gautam's request, I've checked the coil resistances and inductances.

  • PRM/BS/SRM coils were tested.
  • All the PRM coils look well-matched in terms of the inductance. Also, I didn't find a significant difference from BS coils.
  • Pin 1 of the feedthru connectors is shorted to the vacuum chamber.
  • A discovery was that: The SRM DSUB pinouts are mirrored compared to the other suspensions.

Method

A DSUB25 breakout was directly connected to the flange (Attachment 1).
The impedance meter was nulled every time the measurement range and type (R or L) were changed.

Result

Feedthru connector: PRM1
Pin1 - flange: R = 0.8Ω
Pin11-23 / R = 1.79Ω / L=3.21mH
Pin 7-19 / R = 1.82Ω / L=3.22mH
Pin 3-15 / R = 1.71Ω / L=3.20mH

Feedthru connector: BS1
Pin1 - flange: R = 0.5Ω
Pin11-23 / R = 1.78Ω / L=3.26mH
Pin 7-19 / R = 1.63Ω / L=3.30mH
Pin 3-15 / R = 1.61Ω / L=3.29mH

Feedthru connector: SRM1
Pin1 - flange: R = 0.5Ω

Pin11-24 / R = 18.1Ω / L=3.22mH
Pin 7-20 / R = 18.8Ω / L=3.25mH
Pin 3-16 / R = 20.3Ω / L=3.25mH

Feedthru connector: PRM2
Pin1 - flange: R = 0.6Ω
Pin11-23 / R = 1.82Ω / L=3.20mH
Pin 7-19 / R = 1.53Ω / L=3.20mH
Pin 3-15 / R = N/A

Feedthru connector: BS2
Pin1 - flange: R = 0.6Ω
Pin11-23 / R = 1.46Ω / L=3.27mH
Pin 7-19 / R = 1.54Ω / L=3.24mH
Pin 3-15 / R = N/A

Feedthru connector: SRM2
Pin1 - flange: R = 0.7Ω
Pin11-24 / R = N/A

Pin 7-20 / R = 18.5Ω / L=3.21mH
Pin 3-16 / R = 19.1Ω / L=3.25mH

Observation

The SRM pinouts seem mirrored compared to the others. In fact, these two connectors are equipped with mirror cables (although they are unshielded ribbons) (Attachment 2).

The SRM sus is located on the ITMY table. There is a long in vacuum DSUB25 cable between the ITMY and BS tables. I suspect that the cable mirrors the pinout and this needs to be corrected by the in-air mirror cables.

I went around the lab and did not find any other suspensions which have the mirror cable.

WIth the BHD configuration, we will move the feedthru for the SRM to the one on the ITMY chamber. So I believe the situation is going to be improved.

 

Attachment 1: P_20210311_224651.jpg
P_20210311_224651.jpg
Attachment 2: P_20210311_225359.jpg
P_20210311_225359.jpg
  15913   Fri Mar 12 12:32:54 2021 gautamSummarySUSCoil Rs & Ls for PRM/BS/SRM

For consistency, today, I measured both the BS and PRM actuator balancing using the same technique and don't find as serious an imbalance for the BS as in the PRM case. The Oplev laser source is common for both BS and PRM, but the QPDs are of course distinct.

BTW, I thought the expected resistance of the coil windings of the OSEM is ~13 ohms, while the BS/PRM OSEMs report ~1-2 ohms. Is this okay?

Quote:
 
  • All the PRM coils look well-matched in terms of the inductance. Also, I didn't find a significant difference from BS coils.
Attachment 1: BS_actuator.pdf
BS_actuator.pdf
Attachment 2: PRMact.pdf
PRMact.pdf
  15914   Fri Mar 12 13:01:43 2021 ranaSummarySUSCoil Rs & Ls for PRM/BS/SRM

ugh. sounds bad - maybe a short. I suggest measuring the inductance; thats usually a clearer measurement of coil health

  15915   Fri Mar 12 13:48:53 2021 gautamSummarySUSCoil Rs & Ls for PRM/BS/SRM

I didn't repeat Koji's measurement, but he reports the expected ~3.2mH per coil on all the BS and PRM coils.

Quote:

ugh. sounds bad - maybe a short. I suggest measuring the inductance; thats usually a clearer measurement of coil health

  15926   Tue Mar 16 19:13:09 2021 Paco, AnchalUpdateSUSFirst success in Input Matric Diagonalization

After jumping through few hoops, we have one successful result in diagonalizing the input matrix for MC1, MC2 and MC3.


Code:

  • Attachment 2 has the code file contained. For now, we can only guarantee it to work on Donatella in conda base environment. Our code is present in scripts/SUS/InMatCalc
  • We mostly follow the steps mentioned in 4886 and the matlab codes in scripts/SUS/peakFit.
  • Data is first multiplied with currently used inpur matrix to get time series data in DOF (POS, PIT, YAW, SIDE) basis.
  • Then, the peak frequencies of each resonance are identified.
  • For getting these results, we did not attempt to fit the peaks with lorentzians and took the maxima point of the PSD to get the peak positions. This is only possible if the current input matrix is good enough. We have to adjust some parameters so that our fitting code works always.
  • TF estimate between the sensor data w.r.t UL sensor is taken and the values around the peak frequencies of oscillations are averaged to get the sensing matrix.
  • This matrix is normalized along DOF axis (columns in our case) and then inverted.
  • After inversion, another normaliation is done along DOF axis (now rows).
  • Finally we plot the comparison of ASD in DOF basis when using current input matrix and when using our calculated inpur matrix (diagonalizing).
  • You can notice in Attachment 1 that after the diagonalization, each DOF shows resonance at only one and its own resonance frequency while earlier there was some mixing shown.
  • Absolute value of the calculated DOF might have changed and we need to calibrate them or apply appropriate gain factors in the DOF basis filter chains.

Next steps:

  • We'll complete our scripts and make them more general to be used for any optic.
  • We'll combine all of them into one single script which can be called by medm.
  • In parallel, we'll start onwards from step 2 in 15881.
  • Anything else that folks can suggest on our first result. Did we actually do it or are we fooling ourselves?
Attachment 1: IMC_InputMatrixDiagonalization.pdf
IMC_InputMatrixDiagonalization.pdf IMC_InputMatrixDiagonalization.pdf IMC_InputMatrixDiagonalization.pdf
Attachment 2: InMatCalcScripts.zip
  15929   Wed Mar 17 10:52:48 2021 JordanUpdateSUS3" Ring Adpater for SOS

I have added a .1", 45deg chamfer to the bottom of the adapter ring. This was added for a new placement of the eq stops, since the barrel screws are hard to access/adjust.

This also required a modification to the eq stop bracket, D960008-v2, with 1/4-20 screws angled at 45 deg to line up with the chamfer.

The issue I am running into is there needs to be a screw on the backside of the ring as well, otherwise the ring would fall backwards into the OSEMs in the event of an earthquake. The only two points of contact are these front two angled screws, a third is needed on the opposite side of the CoM for stability. This would require another bracket mounted at the back of the SOS tower, but there is very little open real estate because of the OSEMs.

 

Instead of this whole chamfer route, is it possible/easier to just swap the screws for the barrel eq stops? Instead of a socket head cap screw, a SS thumb screw such as this, will provide more torque when turning, and remove the need to use a hex wrench to turn.

 

Attachment 1: Side_View.png
Side_View.png
Attachment 2: Front_View.png
Front_View.png
Attachment 3: Ring_with_Modifed_Bracket.png
Ring_with_Modifed_Bracket.png
Attachment 4: Back_of_ring.png
Back_of_ring.png
Attachment 5: Front_of_Ring.png
Front_of_Ring.png
  15930   Wed Mar 17 11:57:54 2021 Paco, AnchalUpdateSUSTested New Input Matrix for MC1

[Paco, Anchal]

Paco accidentally clicked on C1:SUS-MC1_UL_TO_COIL_SW_1_1 (MC1 POS to UL Coil Switch) and clicked it back on. We didn't see any loss of lock or anything significant on the large monitor on left.

Testing the new calculated input matrix

  • Switched off the PSL shutter (C1:PSL-PSL_ShutterRqst)
  • Switched off IMC autolocker (C1:IOO-MC_LOCK_ENABLE)
  • Uploaded the same input matrix as the current one to check writing function in scripts/SUS/InMatCalc/testingInMat.py . We have created backup text file for current settings in backupMC1InMat.txt .
  • Uploaded the new input matrix in normalized form. To normalize, we first made each row vector unit vector and then multiplied by the norm of current input matrix's row vectors (see scripts/SUS/InMatCalc/normalizeNewInputMat.py)
  • Switched ON the PSL shutter (C1:PSL-PSL_ShutterRqst)
  • Switched ON IMC autolocker (C1:IOO-MC_LOCK_ENABLE)
  • Locked was caught immediately. The wavefront sensor of MC1 shows usual movement, nothing crazy.
  • So the new input matrix is digestable by the system, but what's the efficacy of it?

< Two inspection people taking pictures of ceiling and portable AC unit passed. They rang the doorbell but someone else let them in. They walked out the back door.>

Testing how good the input matrix for MC1 is:

 

  • We loaded the input matrix butterfly row in C1:SUS-MC1_LOCKIN_INMATRX_1_4 to 8. This matrix is multiplied by C1:SUS-MC1_UL_SEN_IN and so on channels before the calibration to um and application of toher filters.
  • We tried to look around on how to load the same filter banks on the signal chain of LOCKIN1 of MC1 but couldn't, so we just manually added gain value of 0.09 in this chain to simulate the calibration factor at the very least.
  • We started the oscillator on LOCKIN 1 on MC1 with amplitude 1 and frequency 6 Hz.
  • We added butterfly mode actuation output column (UL:1, UR:-1, LL:-1, LR:1), nothing happened to the lock of probably because of low amplitude we put in.
  • Now, we plot the ASD of channels like C1:SUS-MC1_SUSPOS_IN1 (for POS, PIT, YAW, SIDE) to see if we see a corresponding peak there. No we don't. See attachment 1.

Restoring the system:

  • Added 0 to the LOCKIN1 column in MC1 output matrix.
  • Made LOCK1 oscillator 0 Amplitude, 0 Hz.
  • Changed back gain on signal chain of LOCKIN1 on MC1.
  • Added 0 to C1:SUS-MC1_LOCKIN_INMATRX_1_4 to 8.
  • Switched off the PSL shutter (C1:PSL-PSL_ShutterRqst)
  • Switched off IMC autolocker (C1:IOO-MC_LOCK_ENABLE)
  • Wrote back the old matrix by scripts/SUS/InMatCalc/testingInMat3.py which used the backup we created.
  • Switched ON the PSL shutter (C1:PSL-PSL_ShutterRqst)
  • Switched ON IMC autolocker (C1:IOO-MC_LOCK_ENABLE)
Attachment 1: 20210317_MC1_InMATtest.pdf
20210317_MC1_InMATtest.pdf
Attachment 2: MC1_Input_Matrix_Test.tar.gz
  15934   Wed Mar 17 16:30:46 2021 AnchalUpdateSUSNormalized Input Matrices plotted better than SURF students

Here, I present the same input matrices now normalized row by row to have same norm as current matrices rows. These now I plotted better than last time. Other comments same as 15902. Please let us know what you think.


Thu Mar 18 09:11:10 2021 :

Note: The comparison of butterfly dof in the two cases is bit bogus. The reason is that we know what the butterfly vector is in sensing matrix (N_osems x (N_dof +1)) and that is the last column being (1, -1, 1, -1, 0) corresponding to (UL, UR, LR, LL, Side). However, the matrix we multiply with the OSEM data is the inverse of this matrix (which becomes the input matrix) which has dimensions ((N_dof + 1) x N_osem) and has the last row corresponding to the butterfly dof. This row was not stored for old calculation of the input matrix (which is currently in use) and can not be recovered (mathematically not possible) with the existing 5x4 part of that input matrix. We just added (1, -1, 1, -1, 0) row in the bottom of this matrix (as was done in the matlab codes) but that is wrong and hence the butterfly vector looks so bad for the existing input matrix.

Proposal: We should store the last row of generated input matrix somewhere for such calculations. Ideally, another row in the epics channels for the input matrix would be the best place to store them but I guess that would be too destructive to implement. Other options are to store this 5 number information in wiki or just elogs. For this post, the buttefly row for the generated input matrix is present in the attached files (for future references).

Attachment 1: IMC_InputMatrixDiagonalization.pdf
IMC_InputMatrixDiagonalization.pdf IMC_InputMatrixDiagonalization.pdf IMC_InputMatrixDiagonalization.pdf
Attachment 2: NewAndOldMatrices.zip
  15937   Thu Mar 18 09:18:49 2021 Paco, AnchalUpdateSUSTesting of new input matrices with new data

[Paco, Anchal]

Since the new generated matrices were created for the measurement made last time, they are of course going to work well for it. We need to test with new independent data to see if it works in general.


  • We have run scripts/SUS/InMatCal/freeSwingMC.py for 1 repition and free swinging duration of 1050s on tmux session FreeSwingMC on Rossa. Started at GPS: 1300118787.
  • Thu Mar 18 09:24:57 2021 : The script ended successfully. IMC is locked back again. Killing the tmux session.
  • Attached are the results of 1-kick test, time series data and the ASD of DOFs for calculated using existing input matrix and our calculated input matrix.
  • The existing one was already pretty good except for maybe the side DOF which was improved on our diagonalization.

[Paco]

After Anchal left for his test, I took the time to set up the iMAC station so that Stephen (and others) can remote desktop into it to use Omnigraffle. For this, I enabled the remote login and remote management settings under "Sharing" in "System Settings". These two should allow authenticated ssh-ing and remote-desktopping respectively. The password is the same that's currently stored in the secrets.

Quickly tested using my laptop (OS:linux, RDP client = remmina + VNC protocol) and it worked. Hopefully Stephen can get it to work too.

Attachment 1: MC_Optics_Kicked_Time_Series_1.pdf
MC_Optics_Kicked_Time_Series_1.pdf
Attachment 2: TEST_Input_Matrix_Diagonalization.pdf
TEST_Input_Matrix_Diagonalization.pdf TEST_Input_Matrix_Diagonalization.pdf TEST_Input_Matrix_Diagonalization.pdf
  15939   Thu Mar 18 12:46:53 2021 ranaUpdateSUSTesting of new input matrices with new data

Good Enough! Let's move on with output matrix tuning. I will talk to you guys about it privately so that the whole doesn't learn our secret, and highly sought after, actuation balancing.

I suspect that changing the DC alignment of the SUS changes the required input/output matrix (since changes in the magnet position w.r.t. the OSEM head change the sensing cross-coupling and the actuation gain), so we want to make sure wo do all this with the mirror at the correct alignment.

 

  15943   Fri Mar 19 10:49:44 2021 Paco, AnchalUpdateSUSTrying coil actuation balance

[Paco, Anchal]

  • We decided to try out the coil actuation balancing after seeing some posts from Gautum about the same on PRM and ETMY.
  • We used diaggui to send swept sine excitation signal to C1:SUS-MC3_ULCOIL_EXC and read it back at C1:SUS-MC3_ASCPIT_IN1. Idea was to create transfer function measurements similar to 15880.
  • We first tried taking the transfer function with excitation amplitude 0f 1, 10, 50, 200 with damping loops on (swept from 10 to 100 Hz lograthmically in 20 points).
  • We found no meaningful measurement and looked like we were just measuring noise.
  • We concluded that it is probably because our damping loops are damping all the excitation down.
  • So we decided to switch off damping and retry.
  • We switched off: C1:SUS-MC3_SUSPOS_SW2 , C1:SUS-MC3_SUSPIT_SW2, C1:SUS-MC3_ASCPIT_SW2, C1:SUS-MC3_ASCYAW_SW2, C1:SUS-MC3_SUSYAW_SW2, and C1:SUS-MC3_SUSSIDE_SW2.
  • We repeated teh above measurements going up in amplitudes of excitation as 1, 10, 20. We saw the oscillation going out of UL_COIL but the swept sine couldn't measure any meaningful transfer function to C1:SUS-MC3_ASCPIT_IN1. So we decided to just stop. We are probably doing something wrong.

Trying to go back to same state:

  • We switch on: C1:SUS-MC3_SUSPOS_SW2 , C1:SUS-MC3_SUSPIT_SW2, C1:SUS-MC3_ASCPIT_SW2, C1:SUS-MC3_ASCYAW_SW2, C1:SUS-MC3_SUSYAW_SW2, and C1:SUS-MC3_SUSSIDE_SW2.
  • But C1:SUS-MC3_ASCYAW_INMON had accumulated about 600 offset and was distrupting the alignment. We switched off C1:SUS-MC3_ASCYAW_SW2 hoping the offset will go away once the optic is just damped with OSEM sensors, but it didn't.
  • Even after minutes, the offset in C1:SUS-MC3_ASCYAW_INMON kept on increasing and crossed beyond 2000 counts limit set in C1:IOO-MC3_YAW filter bank.
  • We tried to unlock the IMC and lock it back again but the offset still persisted.
  • We tried to add bias in YAW DOF by increasing C1:SUS-MC3_YAW_OFFSET, and while it was able to somewhat reduce the WFS C1:SUS-MC3_ASCYAW_INMON offset  but it was misalgning the optic and the lock was lost. So we retracted the bias to 0 and made it zero.
  • We tried to track back where the offset is coming from. In C1IOO_WFS_MASTER.adl, we opened the WFS2_YAW filter bank to see if the sensor is indeed reading the increasing offset.
  • It is quite weird that C1:IOO-WFS2_YAW_INMON is just oscillating but the output in this WFS2_YAW filter bank is slowly increasing offset.
  • We tried to zero the gain and back to 0.1 to see if some holding function is causing it, but that was not the case. The output went back to high negative offset and kept increasing.
  • We don't know what else to do. Only this one WFS YAW output is increasing, everything else is at normal level with no increasing offset or peculiar behavior.
  • We are leaving C1:SUS-MC3_ASCYAW_SW2 off as it is disrupting the IMC lock.

[Jon walked in, asked him for help]

  • Jon suggested to do burt restore on IOO channels.
  • We used (selected through burtgooey):
    burtwb -f /opt/rtcds/caltech/c1/burt/autoburt/snapshots/2021/Mar/19/08:19/c1iooepics.snap -l /tmp/controls_1210319_113410_0.write.log -o /tmp/controls_1210319_113410_0.nowrite.snap -v <
  • No luck, the problem persists.
  15951   Mon Mar 22 11:57:21 2021 Paco, AnchalUpdateSUSTrying coil actuation balance

[Paco, Anchal]

  • For MC coil balancing we will use the ASC (i.e. WFS) error signals since there are no OPLEV inputs (are there OPLEVs at all?).

Test MC1

  • Using the SUS screen LockIns the plan is to feed excitation(s) through the coil outputs, and look at the ASC(Y/P) error signals.
  • A diaggui xml template was saved in /users/Templates/SUS/MC1-actDiag.xml which was based on /users/Templates/SUS/ETMY-actDiag.xml
  • Before running the measurement, we of course want to plug our input matrix, so we ran /scripts/SUS/InMatCalc/writeMatrix.py only to find that it tripped the MC1 Watchdog.
    • The SIDE input seems to have the largest rail, but we just followed the procedure of temporarily increasing the WD max! threshold to allow the damping action and then restoring it.
    • This happened because in latest iteration of our code, we followed an advice from the matlab code to ensure the SIDE OSEM -> SIDE DOF matrix element remains positive, but we found out that MC1 SIDE gain (C1:SUS-MC1_SUSSIDE_GAIN) was set to -8000 (instead of a positive value like all other suspensions).
    • So we decided to try our new input matrix with a positive gain value of 8000 at C1:SUS-MC1_SUSSIDE_GAIN and we were able to stablize the optic and acquire lock, but...
    • We saw that WFS YAW dof started accumulating offset and started disturbing the lock (much like last friday). We disabled the ASC Input button (C1:SUS-MC1_ASCYAW_SW2).
    • This made the lock stable and IMC autolocker was able to lock. But the offset kept on increasing (see attachment 1).
    • After sometime, the offset begain to exponential go to some steady state value which was around -3000.
  • We wrote back the old matrix values and changed the C1:SUS-MC1_SUSSIDE_GAIN back to -8000. But the ASCYAW offset remained to the same position. We're leaving it disabled again as we don't know how to fix this. Hopefully, it will organically come back to small value later in the day like last time (Gautum just reenabled the ASCYAW input and it worked).

Test MC3

  • Defeated by MC1, we moved to MC3.
  • Here, the gain value for C1:SUS-MC3_SUSSIDE_GAIN was already positive (+500) so it could directly take our new matrix.
  • When we switched off watchdog, loaded the new matrix and switched the watchdog back on.
  • The IMC lock was slightly distrupted but remain locked. There was no unusual activity in the WFS sensor values. However, we saw the the SIDE coil output is slowly accumulating offset.
  • So we switched off the watchdog before it will trip itself, wrote back the old matrix and reinstated the status quo.
  • This suggests we need to carefully look back our latest changes of normalization and have new input matriced which keep the system stable other than working on paper with offline data.
Attachment 1: 210322_MC1_ASCY.pdf
210322_MC1_ASCY.pdf
Attachment 2: NewandOldMatrices.tar.gz
  15952   Mon Mar 22 15:10:00 2021 ranaUpdateSUSTrying coil actuation balance

There's an integrator in the MC WFS servos, so you should never be disabling the ASC inputs in the suspensions. Disabling 1 leg in a 6 DOF MIMO system is like a kitchen table with 1 leg removedcheeky.

Just diagnose your suspension stuff with the cavity unlocked. You should be able to see the effect by characterizing the damping loops / cross-coupling.

  15954   Mon Mar 22 19:07:50 2021 Paco, AnchalUpdateSUSTrying coil actuation balance

We found that following protocol works for changing the input matrices to new matrices:

  • Shut the PSL shutter C1:PSL-PSL_ShutterRqst. Switch off IMC autolocker C1:IOO-MC_LOCK_ENABLE.
  • Switch of the watchdog, C1:SUS-MC1_LATCH_OFF.
  • Update the new matrix. (in case of MC1, we need to change sign of C1:SUS-MC1_SUSSIDE_GAIN for new matrix)
  • Switch on the watchdog back again which enables all the coil outputs. Confirm that the optic is damped with just OSEM sensors.
  • Switch on IMC autolocker C1:IOO-MC_LOCK_ENABLE and open PSL shutter C1:PSL-PSL_ShutterRqst.

We repeated this for MC2 as well and were able to lock. However, we could not do the same for MC3. It was getting unstable as soon as cavity was locked i.e. the WFS were making the lock unstable. However, the unstability was different in different attempts but we didn't try mroe times as we had to go.


Coil actuation balancing:

  • We set LOCKIN1 and LOCKIN2 oscillators at 10.5 Hz anf 13.5 Hz with amplitude of 10 counts.
  • We wrote PIT, YAW and Butterfly actuation vectors (see attached text files used for this) on LOCKIN1 and LOCKIN2 for MC1.
  • We measured C1:SUS-MC1_ASCYAW_IN1 and C1:SUS-MC1_ASCPIT_IN1 and compared it against the case when no excitation was fed.
  • We repeated the above steps for MC2 except that we did not use LOCKIN2. LOCKIN2 was found to already on at oscillator frequency of 0.03Hz with amplitude of 500 counts and was fed to all coils with gain of 1 (so it was effectively moving position DOF at 0.03 Hz.) When we changed it, it became ON back again after we turned on the autolocker, so we guess this must be due to some background script and msut be important so we did not make any changes here. But what is it for?
  • We have gotten some good data for MC1 and MC2 to ponder upon next.
  • MC1 showed no cross coupling at all while MC2 shoed significant cross coupling between PIT and YAW.
  • Both MC1 and MC2 did not show any cross coupling between butterfly actuation and PIT/YAW dof.

On another news, IOO channels died!

  • Infront of us, the medm channels starting with C1:IOO just died. See attachment 8.
  • We are not sure why that happened, but we have reported everything we did up here.
  • This happened around the time we were ready to switch back on the IMC autolocker and open the shutter. But now these channels are dead.
  • All optics were restored with old matrices and settings and are damped in good condition as of now.
  • IMC should lock back as soon as someone can restart the EPICS channels and switch on C1:IOO-MC_LOCK_ENABLE and C1:PSL-PSL_ShutterRqst.
Attachment 1: 20210322_MC1_CoilBalancePIT.pdf
20210322_MC1_CoilBalancePIT.pdf
Attachment 2: 20210322_MC1_CoilBalanceYAW.pdf
20210322_MC1_CoilBalanceYAW.pdf
Attachment 3: 20210322_MC1_CoilBalanceBUTT.pdf
20210322_MC1_CoilBalanceBUTT.pdf
Attachment 4: 20210322_MC2_CoilBalancePIT.pdf
20210322_MC2_CoilBalancePIT.pdf
Attachment 5: 20210322_MC2_CoilBalanceYAW.pdf
20210322_MC2_CoilBalanceYAW.pdf
Attachment 6: 20210322_MC2_CoilBalanceBUTT.pdf
20210322_MC2_CoilBalanceBUTT.pdf
Attachment 7: 20210322_IMC_CoilBalance.tar.gz
Attachment 8: image-e6019a14-9cf3-45f7-8f2c-cc3d7ad1c452.jpg
image-e6019a14-9cf3-45f7-8f2c-cc3d7ad1c452.jpg
  15957   Wed Mar 24 09:23:49 2021 PacoUpdateSUSMC3 new Input Matrix

[Paco]

  • Found IMC locked upon arrival
  • Loaded newest MC3 Input Matrix coefficients using /scripts/SUS/InMatCalc/writeMatrix.py after unlocking the MC, and disabling the watchdog. 
  • Again, the sens signals started increasing after the WD is reenabled with the new Input matrix, so I manually tripped it and restored the old matrix; recovered MC lock.
  • Something is still off with this input matrix that makes the MC3 loop unstable.
  15961   Thu Mar 25 11:46:31 2021 Paco, AnchalUpdateSUSMC2 Coil Balancing updates

Proof-of-principle

  • We excited PIT and YAW dofs using LOCKIN1 in MC2 on Monday.
  • We analyzed this data in a simple analysis explained in Attachment 1 python notebook (also present at /users/anchal/20210323_AnalyszingCoilActuationBalance/)
  • Basically, we tried to estimate the cross coupling in 2x2 matrix from actuated DOF to sensed DOF, inverted it, and applied it to output matrix to undo the cross coupling.
  • Attachments 2 and 3 show how much we performed in undoing the cross coupling.
  • The ratio of 13.5 Hz peaks shows how much coupling is still present.

Going towards 3x3 Coil balancing:

  • In a conversation with Rana yesterday, we understood that we can use MC_F data as POS sensing data out of the loop.
  • So today, we repreated the excitation measurements while exciting POS, PIT and YAW dofs from LOCKIN1 on MC2 and measuring C1:IOO-MC_F, C1:SUS-MC2_ASCPIT_IN1 and C1:SUS-MC2_ASCPIT_IN2.
  • Data from MC_F is converted into units of um using factor 9.57e-8 um/Hz.
  • We changed the excitation amplitude in order to see cross coupling peaks when they were not visible with low excitation.
  • The data was measured while new calculated input matrix was loaded which from our calculations diagonalized the sensing matrix of OSEMs.

Some major changes:

  • We actually found that the C1:SUS-MC2_ASCPIT_IN1 showed a broadband increase in noise today (from Monday) by factor of about 100 in range 0-20 Hz.
  • We were not sure why this changed from our 22nd March measurement.
  • We checked if the gain values in the loops changed in alst 3 days, but they didn't.
  • Then we realized that the WFS1_PIT and WFS2_PIT switched that we turned ON on Tuesday were the only changes that were made in the loop.
  • We turned back OFF C1:IOO-WFS1_PIT_SW1 and C1:IOO-WFS2_PIT_SW1. This actually brought back the noise level of C1:SUS-MC2_ASCPIT_IN1 down to what it was on Monday.

 

Attachment 1: CoilActuationBalancing.ipynb.tar.gz
Attachment 2: MC2_CoilBalancePITnorm_excSamePIT.pdf
MC2_CoilBalancePITnorm_excSamePIT.pdf
Attachment 3: MC2_CoilBalanceYAWnorm_excSameYAW.pdf
MC2_CoilBalanceYAWnorm_excSameYAW.pdf
Attachment 4: 20210325_IMC_CoilBalance.tar.gz
  15966   Thu Mar 25 16:02:15 2021 gautamSummarySUSRepeated measurement of coil Rs & Ls for PRM/BS

Method

Since I am mainly concerned with the actuator part of the OSEM, I chose to do this measurement at the output cables for the coil drivers in 1X4. See schematic for pin-mapping. There are several parts in between my measurement point and the actual coils but I figured it's a good check to figure out if measurements made from this point yield sensible results. The slow bias voltages were ramped off under damping (to avoid un-necessarily kicking the optics when disconnecting cables) and then the suspension watchdogs were shutdown for the duration of the measurement.

I used an LCR meter to measure R and L - as prescribed by Koji, the probe leads were shorted and the readback nulled to return 0. Then for R, I corroborated the values measured with the LCR meter against a Fluke DMM (they turned out to be within +/- 0.5 ohms of the value reported by the BK Precision LCR meter which I think is reasonable).

Result

                   PRM
Pin1-9 (UL)   / R = 30.6Ω / L=3.23mH  
Pin2-10 (LL)  / R = 30.3Ω / L=3.24mH
Pin3-11 (UR) / R = 30.6Ω / L=3.25mH
Pin4-12 (LR) / R = 31.8Ω / L=3.22mH
Pin5-13 (SD) / R = 30.0Ω / L=3.25mH

                       BS
Pin1-9 (UL)   / R = 31.7Ω / L=3.29mH
Pin2-10 (LL)  / R = 29.7Ω / L=3.26mH
Pin3-11 (UR) / R = 29.8Ω / L=3.30mH
Pin4-12 (LR) / R = 29.7Ω / L=3.27mH
Pin5-13 (SD) / R = 29.0Ω / L=3.24mH

Conclusions

On the basis of this measurement, I see no problems with the OSEM actuators - the wire resistances to the flange seem comparable to the nominal OSEM resistance of ~13 ohms, but this isn't outrageous I guess. But I don't know how to reconcile this with Koji's measurement at the flange - I guess I can't definitively rule out the wire resistance being 30 ohms and the OSEMs being ~1 ohm as Koji measured. How to reconcile this with the funky PRM actuator measurement? Possibilities, the way I see it, are:

  1. Magnets on PRM are weird in some way. Note that the free-swinging measurement for the PRM showed some unexpected features.
  2. The imbalance is coming from one of the drive chain - could be a busted current buffer for example.
  3. The measurement technique was wrong.
  15970   Fri Mar 26 11:54:37 2021 Paco, AnchalUpdateSUSMC2 Coil Balancing updates

[Paco, Anchal]

  • Today we spent the morning testing the scripts under ~/c1/scripts/SUS/OutMatCalc/ that automate the procedure (which we have been doing by hand) and catch any "bad" behavior instances that we have identified. In such instances, the script sets up to restore the IMC state smoothly.
  • After some testing and debugging, we managed to get some data for MC2 using ~/c1/scripts/SUS/OutMatCalc/getCrossCouplingData.py
  15971   Sun Mar 28 14:16:25 2021 AnchalSummarySUSMC3 new Input Matrix not providing stable loop

Rana asked us to write out here the new MC3 input matrix we have calculated along with the old one. The new matrix is not working out for us as it can't keep the suspension loops stable.


Matrices:

Old (Current) MC3 Input Matrix (C1:SUS-MC3_INMATRIX_ii_jj)
  UL UR LR LL SD
POS 0.288 0.284 0.212 0.216 -0.406
PIT 2.658 0.041 -3.291 -0.674 -0.721
YAW 0.605 -2.714 0.014 3.332 0.666
SIDE 0.166 0.197 0.105 0.074 1

 

New MC3 Input Matrix (C1:SUS-MC3_INMATRIX_ii_jj)
  UL UR LR LL SIDE
POS 0.144 0.182 0.124 0.086 0.586
PIT 2.328 0.059 -3.399 -1.13 -0.786
YAW 0.552 -2.591 0.263 3.406 0.768
SIDE -0.287 -0.304 -0.282 -0.265 0.871

Note that the new matrix has been made so that the norm of each row is the same as the norm of the corresponding row in the old (current) input matrix.


Peak finding results:

  Guess Values Fittted Values
PIT Resonant Freq. (Hz) 0.771 0.771
YAW Resonant Freq. (Hz) 0.841 0.846
POS Resonant Freq. (Hz) 0.969 0.969
SIDE Resonant Freq. (Hz) 0.978 0.978
PIT Resonance Q 600 345
YAW Resonance Q 230 120
POS Resonance Q 200 436
SIDE Resonance Q 460 282
PIT Resonance Amplitude 500 750
YAW Resonance Amplitude 1500 3872
POS Resonance Amplitude 3800 363
SIDE Resonance Amplitude 170 282

Note: The highest peak on SIDE OSEM sensor free swinging data as well as the DOF basis data created using existing input matrix, comes at 0.978 Hz. Ideally, this should be 1 Hz and in MC1 and MC2, we see the resonance on SIDE DOF to show near 0.99 Hz. If you look closely, there is a small peak present near 1 Hz actually, but it is too small to be the SIDE DOF eigenfrequency. And if it is indeed that, then which of the other 4 peaks is not the DOF we are interested in?

On possiblity is that the POS eigenfrequency which is supposed to be around 0.97 Hz is split off in two peaks due to some sideways vibration and hence these peaks get strongly coupled to SIDE OSEM as well.

P.S. I think something is wrong and out limited experience is not enough to pinpoint it. I can show up more data or plots if required to understand this issue. Let us know what you all think.

Attachment 1: MC3_Input_Matrix_Diagonalization.pdf
MC3_Input_Matrix_Diagonalization.pdf
  15972   Mon Mar 29 10:44:51 2021 Paco, AnchalUpdateSUSMC2 Coil Balancing updates

We ran the coil balancing procedure 4 times while iterating through the output matrix optimization.

Attachment 1, pages 1 to 4 show the progression of cross coupling from current output matrix (which is theoretical ideal) to the latest iteration. We plot the sensed DOF ASD which we used to determine the cross coupling when different excitations are fed using the LOCKIN1 feeding 13Hz oscillation of 200 counts amplitude along the vector defined in output matrix. That means, when we change the output matrix, in subsequent tests, we alos change the exciation direction along with it.

Unfortunately, we don't see a very good optimizations over iterations. While we see some peaks going down in sensed PIT and sensed POS (through MC_F), we rather see an increase in cross coupling in the sensed YAW.


Scripts:

  • For running the tests, we used script in scripts/SUS/OutMatCalc/crossCoupleTest.py and wrote commanding scripts in the /users/anchal/20210329_MC2_TestingNewOutMat .
  • The optimization code is at in scripts/SUS/OutMatCalc/outMatOptimize.py.
  • The code reads sensed DOF data using nds2 and calculated cross spectral density among the sensed DOF at the excitation frequencies.
  • This is normalized by the power spectral density of reference data (no excitation) and power spectral density of position data to create a TF estimate.
  • The real values of the sensor matrix thus created is used to get the inverse matrix.
  • The inverse matrix is first normalized along each row by diagonal elements to get 1 there and then multiplied by previous output matrix to create a new output matrix.
  • I guess, reading the code will be a better way of understanding this algorithm.
Attachment 1: MC2OutMatCrossCouple_Old-to-It3.pdf
MC2OutMatCrossCouple_Old-to-It3.pdf MC2OutMatCrossCouple_Old-to-It3.pdf MC2OutMatCrossCouple_Old-to-It3.pdf MC2OutMatCrossCouple_Old-to-It3.pdf
Attachment 2: 20210329_MC2_CrossCoupleTest.tar.gz
  15973   Mon Mar 29 17:07:17 2021 gautamSummarySUSMC3 new Input Matrix not providing stable loop

I suppose you've tried doing the submatrix approach, where SIDE is excluded for the face DoFs? Does that give a better matrix? To me, it's unreasonable that the side OSEM senses POS motion more than any single face OSEM, as your matrix suggests (indeed the old one does too). If/when we vent, we can try positioning the OSEMs better.

  15974   Mon Mar 29 17:11:54 2021 gautamUpdateSUSMC2 Coil Balancing updates

For this technique to work, (i) the WFS loops must be well tuned and (ii) the beam must be well centered on MC2. I am reasonably certain neither is true. For MC2 coil balancing, you can use a HeNe, there is already one on the table (not powered), and I guess you can use the MC2 trans QPD as a sensor, MC won't need to be locked so you can temporarily hijack that QPD (please don't move anything on the table unless you're confident of recovering everything, it should be possible to do all of this with an additional steering mirror you can install and then remove once your test is done). Then you can do any variant of the techniques available once you have an optical lever, e.g. single coil drive, pringle mode drive etc to do the balancing.

I think Hang had some technique he tried recently as well, maybe that is an improvement.

  15975   Mon Mar 29 17:34:52 2021 ranaUpdateSUSMC2 Coil Balancing updates

I think there's been some mis-communication. There's no updated Hang procedure, but there is the one that Anchal, Paco and I discussed, which is different from what is in the elog.

We'll discuss again, and try to get it right, but no need to make multiple forks yet.

  15982   Wed Mar 31 22:58:32 2021 Anchal, PacoUpdateSUSMC2 Coil Balancing Test

A cross-coupling test has been set to trigger at 05:00 am on April 1st, 2021. The script is waiting on tmux session 'cB' on pianosa. /scripts/SUS/OutMatCalc/MC2crossCoupleTest.py is being used here. The script will switch on oscillator in LOCKIN1 of MC2 at 13 Hz and 200 counts and would send it along the POS, PIT and YAW vectors on output matrix one by one, each for 2 minutes. It will take data from C1:IOO-MC_F_DQ, C1:IOO-MC_TRANS_PIT_ERR and C1:IOO-MC_TRANS_YAW_ERR and use it to measure 'sensing matrix' S. Sensing matrix S is defined as the cross-coupling between excited and sensed DOF and we ideally want it to be an identity matrix. The code will use the measured S to create a guess matrix A which on being multiplied by ideal coil output matrix would give us a rotated coil output matrix O. This guess O will be applied and the measurement will be repeated. On each iteration, next, A matrix is defined by:

A_{k+1} = (1 + \beta) A_k - \beta S_k A_k

This recursive algorithm converges A to the inverse of initial S. The above relation is derived by noticing that in steady state A S = \mathrm{I} \Rightarrow A = A S A \Rightarrow A = A - \beta(A S A - A). I've taken this idea from a mathematics paper I found on some more complex stuff (c.f. https://doi.org/10.31219/osf.io/yrvck).

At each iteration, all three matrices A, O and S will be stored in a text file for analysis later.

The code has the error-catching capability and would restore the optic to the status quo if an error occurs or watchdogs trip due to earthquakes.

  15983   Thu Apr 1 00:50:06 2021 gautamUpdateSUSPRMdiag

I spent some time investigating the PRM this evening, trying out some of the stuff we discussed in the meeting.

  1. I used one of the SUS lockin oscillator to drive the butterfly mode (UL=+1, UR=-1, LL=-1, LR=+1) of the optic, @4.45 Hz, Amplitude=450cts (Oplev loops were engaged during the measurement).
  2. Used the sensing matrix infrastructure to demodulate the Oplev PIT/YAW error signals, using the other lockin channel (so that oscillator is just for demodulation, it doesn't actuate on the suspension).
  3. The digital demod phase was adjusted to put all of the demodulated signal in one ("I") quadrature.
  4. The balancing of the coils was perturbed by adding small amounts of the naive PIT (YAW) DoF to the pringle-mode actuation, while simultaneously looking to minimize the demodulated signal in YAW (PIT). 

Basically, my finding tonight was that I could not improve (make the pringle mode actuation witnessed by the Oplev QPD smaller) by +/- perturbing the butterfly actuation with of 0.05%, 0.5% and 1% of PIT (I didn't try YAW, or other values of PIT, as none of these seemed to do any good). It seems highly unlikely that the existing coil gains (these come after the output matrix) and the actual coil/magnet pairs are so perfectly tuned, so there must be something wrong with my method. I'll try more combos tomorrow. Separately, I verified that the naive PIT (YAW) moves the optic mainly, i.e. to the eye), in PIT(YAW) as judged by the REFL spot on the camera and the readback of the Oplev QPD.

For this work, I made a few changes to filter banks:

  1. Added 2Hz wide BPs centered around 4.45 Hz to the "SIGNAL" FM of the BS and PRM suspension lockins.
  2. Added 100mHz LPFs to the I and Q demodulated outputs.
  3. Made a copy of Kiwamu's perturbcoilbalance_fourosem.py (in scripts/SUS) to add little bit of PIT/YAW to the pringle actuation.

I noticed that the filters/switch states/gains for LOCKIN1 and LOCKIN2 are not consistent within either PRM or BS suspension, or across suspensions. Several filter INs/OUTs were also disabled - something for the SUSdiag team to note, whenever this is scripted, the script should check that the signal is indeed making it end-to-end.

  15984   Thu Apr 1 13:56:49 2021 Anchal, PacoUpdateSUSMC2 Coil Balancing Test Results

The coil balancing attempt failed. The off-diagonal values in the measured sensing matrices either remained the same or increased.


The attempt in the morning was too slow. By the time we reached, it had reached to iteration 7 only and still nowhere near optimum sensing matrix had reached. We still needed to see if the optimum would eventually reach if more iterations happened.

<Radhika came for shadowing us and learning about 40m>

So we worked a bit on speeding up the data loading process and then ran the code again which now was running much faster. Still within 1 hr or so, we saw it had reached to iteration 7 with no sign of sensing matrix getting any better.

<Paco left for vaccination>

To determine if the method would work in principle, I decided to stop the current run and start with a 0.5 Hz bandwidth run (so about 7 averages with 8s duration data and welch method). This completed 20 iterations before Gautum came. But it was clear now that the method is not converging to a better solution. Need to find a bug in the implementation of the algorithm mentioned in last post or find a better algoritm.


Attachment 1 is the plot of how the sensing matrix's distance from the identity matrix increased over iterations in the last run.

Attachment 2 is the plot for different off-diagonal terms in the sensing matrix. It is clear that POS->PIT,YAW coupling is not being measured properly as it remains constant.

Attachment 3 Gautum told us that there is some naming error in nds and MC_TRANS_PIT/YAW can be read through C1:IOO-MC_TRANS_PIT_ERR and C1:IOO-MC_TRANS_YAW_ERR channels instead. To test if they indeed point to same values, we did a test of exciting YAW degree through LOCKIN1 and seeing if the peaks are visible in the channels. This was also done to give Radhika an opportunity to do something I could confidently mentor about. and to experience using diaggui.

Attachment 1: SDistanceFromIdentity.pdf
SDistanceFromIdentity.pdf
Attachment 2: SmatIterations.pdf
SmatIterations.pdf
Attachment 3: TestingExcitationAlongYAW.pdf
TestingExcitationAlongYAW.pdf
  15985   Thu Apr 1 18:01:06 2021 Anchal, PacoUpdateSUSMC2 Coil Balancing Test Results Success??

After fixing a few things we felt were wrong in our implementation of the algorithm, we ran the coil balancing for 12 iterations with just 11s per excitation and still taking CSD with 0.1 Hz bandwidth. This time we saw the distance of sensing matrix from identity going down.


Performance Analysis

  • Attachment 1 shows the trend of distance of Sensing matrix from identity matrix over iterations.
  • Attachment 2 shows the trend of off-diagonal terms in sensing matrix over iterations.
  • Attachment 3 shows the ASD for the different sensed DOF when excited in different DOFs with the new output matrix. This is the better truth of what happened by the end. The true sensing matrix is proportional to the peak heights in this plot. Rows are different sensed DOFs (POS, PIT, YAW) and columns are excited DOFs (POS, PIT, YAW). The black dotted curves are ASD when no excitation was present.

Next step

  • We want to run it for longer, more iterations and more duration to get better averaging. Hopefully, this will do a better job. We'll try running this new code tomorrow at 5:00am.
  • We'll work on using uncertainties of measured data.
  • Use awg to excite all DOF together at different frequencies and make the code faster.
Attachment 1: SDistanceFromIdentity.pdf
SDistanceFromIdentity.pdf
Attachment 2: SmatIterations.pdf
SmatIterations.pdf
Attachment 3: MC2CoilCrossCoupling_opt.png
MC2CoilCrossCoupling_opt.png
  15987   Thu Apr 1 18:48:45 2021 gautamUpdateSUSMC2 Coil Balancing Test Results Success??

In these results, can you also include the new matrix and what the relative imbalances were?

  15988   Thu Apr 1 21:13:54 2021 AnchalUpdateSUSMatrix results, new measurement set to trigger
New Input matrix used for MC2 (C1:SUS-MC2_INMATRIX_ii_jj
  UL UR LR LL SIDE
POS 0.2464 0.2591 0.2676 0.2548 -0.1312
PIT 1.7342 0.7594 -2.494 -1.5192 -0.0905
YAW 1.2672 -2.0309 -0.9625 2.3356 -0.2926
SIDE 0.1243 -0.1512 -0.1691 0.1064 0.9962

New output matrix for MC2 (C1:SUS-MC2_TO_COIL_ii_jj_GAIN)
  POS PIT YAW
UL 1 1.022 0.6554
UR 1 0.9776 -1.2532
LL 1 -0.9775 1.2532
LR 1 -1.0219 -0.6554

Measured Sensing Matrix (Cross Coupling) (Sensed DOF x Excited DOF)
  Excited POS Excited PIT Excited YAW
Sensed POS 1 1.9750e-5 -3.5615e-6
Sensed PIT 0 1 -6.93550e-2
Sensed YAW 0 -2.4429e-4 1

A longer measurement is set to trigger at 5:00 tomorrow on April 2nd, 2021. This measurement will run for 35 iterations with an excitation duration of 120s and bandwidth for CSD measurement set to 0.1 Hz. The script is set to trigger in a tmux session named 'cB' on pianosa.

  15991   Fri Apr 2 14:51:20 2021 AnchalUpdateSUSBug found, need to redo the balancing

Last run gave similar results as the quick run we did earlier. The code has been unable to strike out couplings with POS. We found the bug which is causing this. This was because the sampling rate of MC_F channel is different from the test-point channels used for PIT and YAW. Even though we were aware of it, we made an error in handling it while calculating CSD. Due to this, CSD calculation with POS data was performed by the code with zero padding which made it think that no PIT/YAW <-> POS coupling exist. blushHence our code was only able to fix PIT <-> YAW couplings.

We'll need to do another run with this bug fixed. I'll update this post with details of the new measurement.

 

  15993   Fri Apr 2 15:22:54 2021 gautamUpdateSUSMatrix results, new measurement set to trigger

How should I try to understand why PIT and YAW are so different? 

Quote:
New output matrix for MC2 (C1:SUS-MC2_TO_COIL_ii_jj_GAIN)
  POS PIT YAW
UL 1 1.022 0.6554
UR 1 0.9776 -1.2532
LL 1 -0.9775 1.2532
LR 1 -1.0219 -0.6554
  16001   Tue Apr 6 18:46:36 2021 Anchal, PacoUpdateSUSUpdates on recent efforts

As mentioned in last post, we earlier made an error in making sure that all time series arrays go in with same sampling rate in CSD calculation. When we fixed that, our recursive method just blew out in all the efforts since then.


We suspect a major issue is how our measured sensing matrix (the cross-coupling matrix between different degrees of freedom on excitation) has significant imaginary parts in it. We discard the imaginary vaues and only use real parts for iterative method, but we think this is not the solution.

Here we present cross-spectral density of different channels representing the three sensed DOFs (normalized by ASD of no excitation data for each involved component) and the sensing matrix (TF estimate) calculated by normalizing the first cross spectral density plots column wise by the diagonal values. These are measured with existing ideal output matrix but with the new input matrix. This is to get an idea of how these elements look when we use them.

Note, that we used only 10 seconds of data in this run and used binwidth of 0.25Hz. When we used binwidth of 0.1 Hz, we found that the peaks were broad and highest at 13.1 Hz instead of 13 Hz which is the excitation frequency used in these measurements.


How should we proceed?

  • We feel that we should figure out a way to use the imaginary value of the sensing matrix, either directly or as weights representing noise in that particular data point.
  • Should we increase the excitation amplitude? We are currently using 500 counts of excitation on coil output.
  • Are there any other iterative methods for finding the inverse of the matrix that we should be aware of? Our current method is rudimentary and converges linearly.
  • Should we use the absolute value of the sensing matrix instead? In our experience, that is equivalent to simply taking ratios of the PSD of each channel and does not work as well as the TF estimate method.
Attachment 1: FirstMeasurementPlots.pdf
FirstMeasurementPlots.pdf FirstMeasurementPlots.pdf
  16003   Wed Apr 7 02:50:49 2021 KojiUpdateSUSFlange Inspections

Basically I went around all the chambers and all the DB25 flanges to check the invac cable configurations. Also took more time to check the coil Rs and Ls.

Exceptions are the TTs. To avoid unexpected misalignment of the TTs, I didn't try to disconnect the TT cables from the flanges.

Upon the disconnection of the SOS cables, the following steps are taken to avoid large impact to the SOSs

  • The alignment biases were saved or recorded.
  • Gradually moved the biases to 0
  • Turned off the watchdogs (thus damping)

After the measurement, IMC was lock and aligned. The two arms were locked and aligned with ASS. And the PRM alignment (when "misalign" was disengaged) was checked with the REFL CCD.
So I believe the SOSs are functioning as before, but if you find anything, please let me know.
 

  16004   Wed Apr 7 13:07:03 2021 JordanUpdateSUSCoM on 3"->2" Adapter Ring for SOS

Adding the chamfer around the edge of the optic ring did not change the center of mass relative to the plane from the suspension wires.

The CoM was .0003" away from the plane. Adding the chamfer moved it closer by .0001". See the attached photo.

I've also attached the list of the Moments of Inertia of the SOS Assembly.

Attachment 1: CoM_with_Chamfer.png
CoM_with_Chamfer.png
Attachment 2: Moments_of_Inertia.PNG
Moments_of_Inertia.PNG
  16005   Wed Apr 7 17:38:51 2021 AnchalUpdateSUSTrying to uncouple only PIT and YAW first

To test if our method is working at all, we went for the simpler case of just uncoupling PIT and YAW. This is also because the sensor used for these two degrees of freedom is similar (the MC Trans WFS).

We saw a successful decrease in cross-coupling between PIT and YAW over the first 50 iterations that we tried. Here are some results:


Final output matrix:

Output matrix for uncoupling PIT and YAW from eachother
PIT YAW COILS
1.01858 1.16820 UL
0.98107 -0.79706 UR
-0.98107 0.79706 LL
-1.01858 -1.16820 LR

Plots:

  • Attachment 1 shows distance of sensing matrix from identity as iterations go.
  • Attachment 2 shows the off-diagonal elements of sensing matrix as the iterations increase.
    • It is worth noting that PIT -> YAW coupling was the main element that was reduced successfully while the YAW -> PIT was reducing but much more slowly.
    • Most of the remaining cross coupling in the end was from YAW -> PIT.
  • Attachment 3 shows first 10 oscillations in the time series data during excitation of some of the iterations.
  • Attachment 4 shows the cross spectral density of the sensed data during excitation with each other. This has been normalized by reference PSD data (taken with no excitation) of the sensed DOFs involved in the CSD calculation.
  • Attachment 5 shows the TF estimate made by normalizing CSD data column wise by the diagonal elements. The excitation frequency point in these plots become the Sensing matrix in the calculation.
    • One can notice how the PIT -> YAW element is going down in these plots.
    • Even though we are using only the real value of the sensing matrix, the imaginary values are also going down.

Next, tried uncoupling POS and PIT:

  • Next, we tried to uncouple POS and PIT. We expect them to be more coupled than with YAW.
  • At the time of writing this post, 15 iterations of this attempt have been completed and it is not looking good sad.
  • The distance of the sensing matrix from identity is growing at an accelerated rate.
  • The POS output matrix column seems to be trying to go towards the negative of PIT output matrix column! Why? We don't know.
  • We have seen in the past that once POS transforms into PIT or YAW, it just makes the output matrix worse as no feedback actually goes into the POS column. Eventually, the IMC will cease to remain locked.
  • So, I'm cancelling this attempt for now. Will consider more alternatives later.
Attachment 1: SDistanceFromIdentity.pdf
SDistanceFromIdentity.pdf
Attachment 2: SmatIterations.pdf
SmatIterations.pdf
Attachment 3: TimeSeriesPlots.pdf
TimeSeriesPlots.pdf TimeSeriesPlots.pdf TimeSeriesPlots.pdf TimeSeriesPlots.pdf TimeSeriesPlots.pdf TimeSeriesPlots.pdf TimeSeriesPlots.pdf TimeSeriesPlots.pdf
Attachment 4: CSDPlots.pdf
CSDPlots.pdf CSDPlots.pdf CSDPlots.pdf CSDPlots.pdf CSDPlots.pdf CSDPlots.pdf CSDPlots.pdf CSDPlots.pdf
Attachment 5: SmatrixPlots.pdf
SmatrixPlots.pdf SmatrixPlots.pdf SmatrixPlots.pdf SmatrixPlots.pdf SmatrixPlots.pdf SmatrixPlots.pdf SmatrixPlots.pdf SmatrixPlots.pdf
  16007   Thu Apr 8 17:04:43 2021 Anchal, PacoUpdateSUSFirst Successful Coil Balancing

Today, we finally crossed the last hurdle and got a successful converging coil balancing run. laugh


What was the issue with POS?

  • Position of the MC2 mirror is being sensed using C1:IOO-MC_F_DQ channel which is proportional to the resonant frequency of the locked IMC.
  • However, this sensor is always 180 degrees out of phase of our actuator, the coils.
  • When the coils push the mirror forward, the length of the cavity actually decreases.
  • We added an extra option of providing a sign to the sensors such that -1 will be multiplied to sensed values for sensors which measure in opposite direction to the actuation.
  • This is important, because the feedback is applied to the coil output matrix assuming a particular direction of acctuation.
  • When we gave negative sign for the position sensor, it all started making sense and the algorithm started converging.

First run parameters:

  • We used binwidth of 0.25 Hz and duration of excitation as 41s. This would give welch and csd averaging of 19. We used median averaging to ignore outliers.
  • This iteration was run after PIT and YAW were separetly uncoupled before. We'll post a clean start to end run results in near future.
  • The iteration works in following manner:
    • Define a constant coil matrix C = [[1, 1, 1], [1, 1, -1], [1, -1, 1], [1, -1, -1]] which is ideal coil output matrix.
    • In each iteration, the output matrix Ok is defined as (note @ is the matmul operator):
      Ok = C @ Ak
      where Ak is a 3x3 matrix. A-1 is identity matrix.
    • At the end of each iteration, a sensing matrix is calculated in dimensions sensedDOF x excitedDOF, Sk
    • For next iteration, Ak+1 is calcualted by:
      Ak+1 = Ak - b * (Sk - I)
      where I is the identity matrix.
    • At convergence, the sensing matrix would become same as identity and matrix A will stop updating.
  • For this run, we kept the parameter b to be 0.05. This is similar to the KP parameter in PID loops. It should be between 0 and 1.
  • Since b value was small enough to allow for convergence from the inital point, but later it slowed down the process a lot.
  • Ideally, we should figure out a way to increase this paramter when the coil has been balanced somewhat, to increase the speed of the algorithm.
  • Secondly, we have a code which excites all DOFs at different frequencies directly using excitation channels in coil output matrix using awg.py. But for some reason, the excitation channel for 4th row in the output matrix column only connects intermittantly. Because of this, we can't use this method reliably yet. We can investigate more into it if suggested.

Balancing characteristics:

  • Attachment 1 shows how the distance of sensing matrix falls as iterations increase. We only ran for 50 iterations.
  • Attachment 2 shows how different off-diagonal terms of sensing matrix decreased.
    • Note that POS -> PIT, POS -> YAW and PIT-YAW have settled down to the noise floor.
    • The noise floor can be improved by increasing the excitation amplitude and/or increasing the duration of measurement.
  • Attachment 3 shows the evolution of sensing matrix as iterations move.

Final balanced output matrix:

Final balanced output coil matrix for MC2
POS PIT YAW COILS
1.02956 1.13053 1.19116 UL
1.01210 1.09188 -0.74832 UR
0.98737 -0.85502 0.70485 LR
0.96991 -0.89366 -1.23463 LR
Final Sensing Matrix
  Exc POS Exc PIT Exc YAW
Sens POS 1 -2.96e-2 8.00e-3
Sens PIT 8.58e-4 1 -4.84e-3
Sens YAW 5.97e-4 -1.15e-3 1

Code features and next:

  • Majority of the code is in two files: scripts/SUS/OutMatCalc/MC2crossCoupleTest.py and scripts/SUS/OutMatCalc/crossCoupleTest.py .
  • The code runs from start to end without human involevement and restores the state of channels in any case (error, kyboard interrupt, end of code) using finally statement.
  • Currently, each excitation is done one at a time through LockIn1. As mentioned above, this can be sped up 3 times if we get the awg.py to work reliably.
  • The complete code is in python3 and currently is run through native python3 on allegra (a new debian10 workstation with latest cds-workstation installed).
  • The code can be easily generalized for balancing any optic. Please let us know if we should work on making the generalized optic.
  • We're also working on thinking about increasing b as iterations move forward and the error signal becomes smaller.
  • We can also include the uncertainty in the Sensing matrix measurement to provide a weighted feedback. That way, we can probably increase b more.
Attachment 1: SDistanceFromIdentity.pdf
SDistanceFromIdentity.pdf
Attachment 2: SmatIterations.pdf
SmatIterations.pdf
Attachment 3: SmatrixPlots.pdf
SmatrixPlots.pdf SmatrixPlots.pdf SmatrixPlots.pdf SmatrixPlots.pdf SmatrixPlots.pdf SmatrixPlots.pdf SmatrixPlots.pdf SmatrixPlots.pdf
  16009   Fri Apr 9 13:13:00 2021 Anchal, PacoUpdateSUSFaster coil balancing

We ran again this method but with the 'b' parameter as a matrix instead. This provides more gain on some off-diagonal terms than others. This gave us a better convergence with the code reaching to the tolerance level provided (0.01 distance of S matrix from identity) within 16 iterations (~17 mins).


Attachment 1 again shows how the off-diagonal terms go down and how the overall distance of sensing matrix from identity goes down. This is 'Cross coupling budget' of the coils as iterations move forward.


Jumping to near zero-crossing:

  • Rana mentioned a ezlockin code which first makes 5 step changes in output matrix without using feedback and calculates the changes required to reach zero-crossing in the behavior of the off-diagonal terms during these steps.
  • This is similar to what we did above by hand where we increased the value of b for slowly converging off-diagonal elements.
  • We plan to implement this 'jump' to near zero-crossing method next. Aim is to get a coil balancing code that does the job in ~5 min.
  • We have been throwing away imaginary part of sensing matrix so far. We wanted to get to some owrking solution before we try more complex stuff. We have to figure out global phases in each transfer function estimate to rotate the measured transfer function appropriately.
Attachment 1: SmatIterations.pdf
SmatIterations.pdf
Attachment 2: MC2AllOutmat.txt
1.027604652272846142e+00 1.193175249772460367e+00 1.091939557371080394e+00
1.010054273887021292e+00 1.156057452309880551e+00 -8.392112351146234772e-01
9.895057930131009316e-01 -7.685799469766890768e-01 6.200896409311776880e-01
9.719554146272761930e-01 -8.056977444392685594e-01 -1.311061151554526294e+00
  16010   Fri Apr 9 17:41:12 2021 ranaUpdateSUSFaster coil balancing

convergence is great.

Next we wanna get the F2A filters made since most of the IMC control happens at f < 3 Hz. Once you have the SUS state space model, you should be able to see how this can be done using only the free'swinging eigenfrequencies. Then you should get the closed loop model including the F2A filters and the damping filters to see what the closed loop behavior is like.

 

  16014   Sat Apr 10 10:07:47 2021 ranaUpdateSUSFaster coil balancing

I think I mis-spoke about the balancing channels before. The ~20 Hz balancing could go into either the COIL banks or the SUS output matrix.

I believe its more conceptually clean to do this as gains in the outputmatrix, and leave the coil gains as +/- 1. i.e. we would only use the coil gains to compensate for coil/magnet actuation strength.

Then the high frequency balance goes into the outputmatrix. The F2A and A2L decoupling filters would then be generated having a high frequency gain = 1.

  16017   Mon Apr 12 10:07:35 2021 AnchalUpdateSUSWhat's F2A??

I'm not sure I understand what F2A is? I couldn't find a description of this filter anywhere and don't remember if you have already explained it. Can you describe what is needed to be done again, please? We would keep SUS state space model and seismic transfer functions calculation ready meanwhile.

Quote:

Next we wanna get the F2A filters made since most of the IMC control happens at f < 3 Hz. Once you have the SUS state space model, you should be able to see how this can be done using only the free'swinging eigenfrequencies. Then you should get the closed loop model including the F2A filters and the damping filters to see what the closed loop behavior is like.

 

  16020   Tue Apr 13 09:51:22 2021 ranaUpdateSUSWhat's F2A??

Force to Angle. It just means the filters that are in the POS OUTPUT matrix. I think in the past sometimes they are called F2P or F2A.

These filters account for the frequency dependent coupling of the DOFs around the suspension resonance. Take a look at what Bhavini is doing for the plots.

 

  16031   Wed Apr 14 17:53:38 2021 AnchalUpdateSUSPlan for calculating filter banks for output matrix aka F2A aka F2P

Plan of action

  • Get the transfer functions of the suspension plant from actuated DOF to sensed DOF. We'll verify Bhavini's state-space model and get these transfer functions. Use the model TFs, not measured.
  • For each of POS->POS, PIT->PIT, and YAW->YAW, we'll get the resonant frequency and Q of the resonance from these models. No, forget about the Q.
  • We can correct the resonant frequencies from the measured ones in our free swinging data.
  • Now, we'll repeat the following for each column of output matrix filters (inspired from scripts/SUS/F2Pcalc.py, but not fully understood how/why):
    • Select col (eg. POS)
    • Set f0 to the resonant frequency.
    • Calculate \large f_{UL} = f_0 * \sqrt{G_{UL}} where GUL is the corrected DC gain we got after output matrix optimization earlier. (Not sure how, why?). No, use the SS model.
    • Calculate fUR, fLL, and fLR like above.
    • Set \large Q_{UL} = \sqrt{G_{UL}}   (This just seems like a way of keeping some approximately low Q, ideally we should keep this same to what we got above but that might cause saturation issues like Rana mentioned in the meeting)
    • Then, set the following filter in the output matrix element for UL:
                            G_{UL}\frac{1 + i\frac{f}{f_{UL}Q_{UL}} - \frac{f^2}{f_{UL}^2}}{1 + i\frac{f}{f_{0}} - \frac{f^2}{f_{0}^2}}
      which is in zpk form equivalent to:
                            z: \frac{f_0}{2 Q_{UL}} +/- i f_0 \sqrt{1 - \frac{1}{4Q_{UL}}} \quad, \quad p: \frac{f_0}{2} +/- i f_0 \frac{\sqrt{3}}{2} \quad, \quad k: G_{UL}
    • Repeat the above for UR, LL, LR.
    • Note that this filter function takes values GUL at DC and at high frequencies while it would dip at the resonant frequency for POS with depth and narrowness directly proportional to QUL. No, the DC gain is different from the AC gain.
  • However, the F2P filter plots we found in several places on elog look a bit different. Like here: 40m/4719. One important difference is that the filter magnitude always become 1 after the resonance at higher frequencies. Yes, this is  what we want, since you already did the balancing at high frequencies.
  • A preliminary plot of the above calculation for the 1,1 output matrix filter bank (POS -> UL) is attached in Attachment 1.

Discussion:

  • We can make 12 such filters for the 12 numbers we got for the optimized output matrix. Is that the aim or should we do it only for the POS column as has been done in past?
  • We are not sure how the choice of Q is made in setting the above filter function. We'll think more about it to understand this.
  • We are also not sure how the choice of fUL is made above. It looks like depending on the correction gain, we want to slide the zero positions with respect to the pole positions which are fixed at the resonant frequency as expected. This seems to have some complex explanation.
  • Please let us know if we are planning this right before we dive into these calculations/script writing. Thanks.

Edit Thu Apr 15 08:32:58 2021 :

Comments are from Rana.

Corrected the plot in the attachment. It shows the correct behavior at high frequencies now.

Attachment 1: MC2propF2A_UL.pdf
MC2propF2A_UL.pdf
  16035   Thu Apr 15 11:41:43 2021 AnchalUpdateSUSProposed filters for output matrix aka F2A aka F2P

Here' s aquick update before we leave for lunch. We have managed to calculate some filter that would go on the POS column in MC2 output matrix filter banks aka F2A aka F2P filters. In the afternoon if we can come and work on the IMC, we'll try to load them on the output matrix. We have never done that so it might take some time for us to understand on how to do that. Attached is the bode plot for these proposed filters. Let us know if you have any comments.

Attachment 1: MC2propPOSfb.pdf
MC2propPOSfb.pdf
ELOG V3.1.3-