Here, I present the same input matrices now normalized row by row to have same norm as current matrices rows. These now I plotted better than last time. Other comments same as 15902. Please let us know what you think.
Thu Mar 18 09:11:10 2021 :
Note: The comparison of butterfly dof in the two cases is bit bogus. The reason is that we know what the butterfly vector is in sensing matrix (N_osems x (N_dof +1)) and that is the last column being (1, -1, 1, -1, 0) corresponding to (UL, UR, LR, LL, Side). However, the matrix we multiply with the OSEM data is the inverse of this matrix (which becomes the input matrix) which has dimensions ((N_dof + 1) x N_osem) and has the last row corresponding to the butterfly dof. This row was not stored for old calculation of the input matrix (which is currently in use) and can not be recovered (mathematically not possible) with the existing 5x4 part of that input matrix. We just added (1, -1, 1, -1, 0) row in the bottom of this matrix (as was done in the matlab codes) but that is wrong and hence the butterfly vector looks so bad for the existing input matrix.
Proposal: We should store the last row of generated input matrix somewhere for such calculations. Ideally, another row in the epics channels for the input matrix would be the best place to store them but I guess that would be too destructive to implement. Other options are to store this 5 number information in wiki or just elogs. For this post, the buttefly row for the generated input matrix is present in the attached files (for future references).
Since the new generated matrices were created for the measurement made last time, they are of course going to work well for it. We need to test with new independent data to see if it works in general.
After Anchal left for his test, I took the time to set up the iMAC station so that Stephen (and others) can remote desktop into it to use Omnigraffle. For this, I enabled the remote login and remote management settings under "Sharing" in "System Settings". These two should allow authenticated ssh-ing and remote-desktopping respectively. The password is the same that's currently stored in the secrets.
Quickly tested using my laptop (OS:linux, RDP client = remmina + VNC protocol) and it worked. Hopefully Stephen can get it to work too.
Good Enough! Let's move on with output matrix tuning. I will talk to you guys about it privately so that the whole doesn't learn our secret, and highly sought after, actuation balancing.
I suspect that changing the DC alignment of the SUS changes the required input/output matrix (since changes in the magnet position w.r.t. the OSEM head change the sensing cross-coupling and the actuation gain), so we want to make sure wo do all this with the mirror at the correct alignment.
[Jon walked in, asked him for help]
(selected through burtgooey):
burtwb -f /opt/rtcds/caltech/c1/burt/autoburt/snapshots/2021/Mar/19/08:19/c1iooepics.snap -l /tmp/controls_1210319_113410_0.write.log -o /tmp/controls_1210319_113410_0.nowrite.snap -v <
There's an integrator in the MC WFS servos, so you should never be disabling the ASC inputs in the suspensions. Disabling 1 leg in a 6 DOF MIMO system is like a kitchen table with 1 leg removed.
Just diagnose your suspension stuff with the cavity unlocked. You should be able to see the effect by characterizing the damping loops / cross-coupling.
We found that following protocol works for changing the input matrices to new matrices:
We repeated this for MC2 as well and were able to lock. However, we could not do the same for MC3. It was getting unstable as soon as cavity was locked i.e. the WFS were making the lock unstable. However, the unstability was different in different attempts but we didn't try mroe times as we had to go.
Since I am mainly concerned with the actuator part of the OSEM, I chose to do this measurement at the output cables for the coil drivers in 1X4. See schematic for pin-mapping. There are several parts in between my measurement point and the actual coils but I figured it's a good check to figure out if measurements made from this point yield sensible results. The slow bias voltages were ramped off under damping (to avoid un-necessarily kicking the optics when disconnecting cables) and then the suspension watchdogs were shutdown for the duration of the measurement.
I used an LCR meter to measure R and L - as prescribed by Koji, the probe leads were shorted and the readback nulled to return 0. Then for R, I corroborated the values measured with the LCR meter against a Fluke DMM (they turned out to be within +/- 0.5 ohms of the value reported by the BK Precision LCR meter which I think is reasonable).
Pin1-9 (UL) / R = 30.6Ω / L=3.23mH
Pin2-10 (LL) / R = 30.3Ω / L=3.24mH
Pin3-11 (UR) / R = 30.6Ω / L=3.25mH
Pin4-12 (LR) / R = 31.8Ω / L=3.22mH
Pin5-13 (SD) / R = 30.0Ω / L=3.25mH
Pin1-9 (UL) / R = 31.7Ω / L=3.29mH
Pin2-10 (LL) / R = 29.7Ω / L=3.26mH
Pin3-11 (UR) / R = 29.8Ω / L=3.30mH
Pin4-12 (LR) / R = 29.7Ω / L=3.27mH
Pin5-13 (SD) / R = 29.0Ω / L=3.24mH
On the basis of this measurement, I see no problems with the OSEM actuators - the wire resistances to the flange seem comparable to the nominal OSEM resistance of ~13 ohms, but this isn't outrageous I guess. But I don't know how to reconcile this with Koji's measurement at the flange - I guess I can't definitively rule out the wire resistance being 30 ohms and the OSEMs being ~1 ohm as Koji measured. How to reconcile this with the funky PRM actuator measurement? Possibilities, the way I see it, are:
Rana asked us to write out here the new MC3 input matrix we have calculated along with the old one. The new matrix is not working out for us as it can't keep the suspension loops stable.
Note that the new matrix has been made so that the norm of each row is the same as the norm of the corresponding row in the old (current) input matrix.
Peak finding results:
Note: The highest peak on SIDE OSEM sensor free swinging data as well as the DOF basis data created using existing input matrix, comes at 0.978 Hz. Ideally, this should be 1 Hz and in MC1 and MC2, we see the resonance on SIDE DOF to show near 0.99 Hz. If you look closely, there is a small peak present near 1 Hz actually, but it is too small to be the SIDE DOF eigenfrequency. And if it is indeed that, then which of the other 4 peaks is not the DOF we are interested in?
On possiblity is that the POS eigenfrequency which is supposed to be around 0.97 Hz is split off in two peaks due to some sideways vibration and hence these peaks get strongly coupled to SIDE OSEM as well.
P.S. I think something is wrong and out limited experience is not enough to pinpoint it. I can show up more data or plots if required to understand this issue. Let us know what you all think.
We ran the coil balancing procedure 4 times while iterating through the output matrix optimization.
Attachment 1, pages 1 to 4 show the progression of cross coupling from current output matrix (which is theoretical ideal) to the latest iteration. We plot the sensed DOF ASD which we used to determine the cross coupling when different excitations are fed using the LOCKIN1 feeding 13Hz oscillation of 200 counts amplitude along the vector defined in output matrix. That means, when we change the output matrix, in subsequent tests, we alos change the exciation direction along with it.
Unfortunately, we don't see a very good optimizations over iterations. While we see some peaks going down in sensed PIT and sensed POS (through MC_F), we rather see an increase in cross coupling in the sensed YAW.
I suppose you've tried doing the submatrix approach, where SIDE is excluded for the face DoFs? Does that give a better matrix? To me, it's unreasonable that the side OSEM senses POS motion more than any single face OSEM, as your matrix suggests (indeed the old one does too). If/when we vent, we can try positioning the OSEMs better.
For this technique to work, (i) the WFS loops must be well tuned and (ii) the beam must be well centered on MC2. I am reasonably certain neither is true. For MC2 coil balancing, you can use a HeNe, there is already one on the table (not powered), and I guess you can use the MC2 trans QPD as a sensor, MC won't need to be locked so you can temporarily hijack that QPD (please don't move anything on the table unless you're confident of recovering everything, it should be possible to do all of this with an additional steering mirror you can install and then remove once your test is done). Then you can do any variant of the techniques available once you have an optical lever, e.g. single coil drive, pringle mode drive etc to do the balancing.
I think Hang had some technique he tried recently as well, maybe that is an improvement.
I think there's been some mis-communication. There's no updated Hang procedure, but there is the one that Anchal, Paco and I discussed, which is different from what is in the elog.
We'll discuss again, and try to get it right, but no need to make multiple forks yet.
A cross-coupling test has been set to trigger at 05:00 am on April 1st, 2021. The script is waiting on tmux session 'cB' on pianosa. /scripts/SUS/OutMatCalc/MC2crossCoupleTest.py is being used here. The script will switch on oscillator in LOCKIN1 of MC2 at 13 Hz and 200 counts and would send it along the POS, PIT and YAW vectors on output matrix one by one, each for 2 minutes. It will take data from C1:IOO-MC_F_DQ, C1:IOO-MC_TRANS_PIT_ERR and C1:IOO-MC_TRANS_YAW_ERR and use it to measure 'sensing matrix' S. Sensing matrix S is defined as the cross-coupling between excited and sensed DOF and we ideally want it to be an identity matrix. The code will use the measured S to create a guess matrix A which on being multiplied by ideal coil output matrix would give us a rotated coil output matrix O. This guess O will be applied and the measurement will be repeated. On each iteration, next, A matrix is defined by:
This recursive algorithm converges A to the inverse of initial S. The above relation is derived by noticing that in steady state . I've taken this idea from a mathematics paper I found on some more complex stuff (c.f. https://doi.org/10.31219/osf.io/yrvck).
At each iteration, all three matrices A, O and S will be stored in a text file for analysis later.
The code has the error-catching capability and would restore the optic to the status quo if an error occurs or watchdogs trip due to earthquakes.
I spent some time investigating the PRM this evening, trying out some of the stuff we discussed in the meeting.
Basically, my finding tonight was that I could not improve (make the pringle mode actuation witnessed by the Oplev QPD smaller) by +/- perturbing the butterfly actuation with of 0.05%, 0.5% and 1% of PIT (I didn't try YAW, or other values of PIT, as none of these seemed to do any good). It seems highly unlikely that the existing coil gains (these come after the output matrix) and the actual coil/magnet pairs are so perfectly tuned, so there must be something wrong with my method. I'll try more combos tomorrow. Separately, I verified that the naive PIT (YAW) moves the optic mainly, i.e. to the eye), in PIT(YAW) as judged by the REFL spot on the camera and the readback of the Oplev QPD.
For this work, I made a few changes to filter banks:
I noticed that the filters/switch states/gains for LOCKIN1 and LOCKIN2 are not consistent within either PRM or BS suspension, or across suspensions. Several filter INs/OUTs were also disabled - something for the SUSdiag team to note, whenever this is scripted, the script should check that the signal is indeed making it end-to-end.
The coil balancing attempt failed. The off-diagonal values in the measured sensing matrices either remained the same or increased.
The attempt in the morning was too slow. By the time we reached, it had reached to iteration 7 only and still nowhere near optimum sensing matrix had reached. We still needed to see if the optimum would eventually reach if more iterations happened.
<Radhika came for shadowing us and learning about 40m>
So we worked a bit on speeding up the data loading process and then ran the code again which now was running much faster. Still within 1 hr or so, we saw it had reached to iteration 7 with no sign of sensing matrix getting any better.
<Paco left for vaccination>
To determine if the method would work in principle, I decided to stop the current run and start with a 0.5 Hz bandwidth run (so about 7 averages with 8s duration data and welch method). This completed 20 iterations before Gautum came. But it was clear now that the method is not converging to a better solution. Need to find a bug in the implementation of the algorithm mentioned in last post or find a better algoritm.
Attachment 1 is the plot of how the sensing matrix's distance from the identity matrix increased over iterations in the last run.
Attachment 2 is the plot for different off-diagonal terms in the sensing matrix. It is clear that POS->PIT,YAW coupling is not being measured properly as it remains constant.
Attachment 3 Gautum told us that there is some naming error in nds and MC_TRANS_PIT/YAW can be read through C1:IOO-MC_TRANS_PIT_ERR and C1:IOO-MC_TRANS_YAW_ERR channels instead. To test if they indeed point to same values, we did a test of exciting YAW degree through LOCKIN1 and seeing if the peaks are visible in the channels. This was also done to give Radhika an opportunity to do something I could confidently mentor about. and to experience using diaggui.
After fixing a few things we felt were wrong in our implementation of the algorithm, we ran the coil balancing for 12 iterations with just 11s per excitation and still taking CSD with 0.1 Hz bandwidth. This time we saw the distance of sensing matrix from identity going down.
In these results, can you also include the new matrix and what the relative imbalances were?
A longer measurement is set to trigger at 5:00 tomorrow on April 2nd, 2021. This measurement will run for 35 iterations with an excitation duration of 120s and bandwidth for CSD measurement set to 0.1 Hz. The script is set to trigger in a tmux session named 'cB' on pianosa.
Last run gave similar results as the quick run we did earlier. The code has been unable to strike out couplings with POS. We found the bug which is causing this. This was because the sampling rate of MC_F channel is different from the test-point channels used for PIT and YAW. Even though we were aware of it, we made an error in handling it while calculating CSD. Due to this, CSD calculation with POS data was performed by the code with zero padding which made it think that no PIT/YAW <-> POS coupling exist. Hence our code was only able to fix PIT <-> YAW couplings.
We'll need to do another run with this bug fixed. I'll update this post with details of the new measurement.
How should I try to understand why PIT and YAW are so different?
As mentioned in last post, we earlier made an error in making sure that all time series arrays go in with same sampling rate in CSD calculation. When we fixed that, our recursive method just blew out in all the efforts since then.
We suspect a major issue is how our measured sensing matrix (the cross-coupling matrix between different degrees of freedom on excitation) has significant imaginary parts in it. We discard the imaginary vaues and only use real parts for iterative method, but we think this is not the solution.
Here we present cross-spectral density of different channels representing the three sensed DOFs (normalized by ASD of no excitation data for each involved component) and the sensing matrix (TF estimate) calculated by normalizing the first cross spectral density plots column wise by the diagonal values. These are measured with existing ideal output matrix but with the new input matrix. This is to get an idea of how these elements look when we use them.
Note, that we used only 10 seconds of data in this run and used binwidth of 0.25Hz. When we used binwidth of 0.1 Hz, we found that the peaks were broad and highest at 13.1 Hz instead of 13 Hz which is the excitation frequency used in these measurements.
Basically I went around all the chambers and all the DB25 flanges to check the invac cable configurations. Also took more time to check the coil Rs and Ls.
Exceptions are the TTs. To avoid unexpected misalignment of the TTs, I didn't try to disconnect the TT cables from the flanges.
Upon the disconnection of the SOS cables, the following steps are taken to avoid large impact to the SOSs
After the measurement, IMC was lock and aligned. The two arms were locked and aligned with ASS. And the PRM alignment (when "misalign" was disengaged) was checked with the REFL CCD.
So I believe the SOSs are functioning as before, but if you find anything, please let me know.
Adding the chamfer around the edge of the optic ring did not change the center of mass relative to the plane from the suspension wires.
The CoM was .0003" away from the plane. Adding the chamfer moved it closer by .0001". See the attached photo.
I've also attached the list of the Moments of Inertia of the SOS Assembly.
To test if our method is working at all, we went for the simpler case of just uncoupling PIT and YAW. This is also because the sensor used for these two degrees of freedom is similar (the MC Trans WFS).
We saw a successful decrease in cross-coupling between PIT and YAW over the first 50 iterations that we tried. Here are some results:
Today, we finally crossed the last hurdle and got a successful converging coil balancing run.
We ran again this method but with the 'b' parameter as a matrix instead. This provides more gain on some off-diagonal terms than others. This gave us a better convergence with the code reaching to the tolerance level provided (0.01 distance of S matrix from identity) within 16 iterations (~17 mins).
Attachment 1 again shows how the off-diagonal terms go down and how the overall distance of sensing matrix from identity goes down. This is 'Cross coupling budget' of the coils as iterations move forward.
convergence is great.
Next we wanna get the F2A filters made since most of the IMC control happens at f < 3 Hz. Once you have the SUS state space model, you should be able to see how this can be done using only the free'swinging eigenfrequencies. Then you should get the closed loop model including the F2A filters and the damping filters to see what the closed loop behavior is like.
I think I mis-spoke about the balancing channels before. The ~20 Hz balancing could go into either the COIL banks or the SUS output matrix.
I believe its more conceptually clean to do this as gains in the outputmatrix, and leave the coil gains as +/- 1. i.e. we would only use the coil gains to compensate for coil/magnet actuation strength.
Then the high frequency balance goes into the outputmatrix. The F2A and A2L decoupling filters would then be generated having a high frequency gain = 1.
I'm not sure I understand what F2A is? I couldn't find a description of this filter anywhere and don't remember if you have already explained it. Can you describe what is needed to be done again, please? We would keep SUS state space model and seismic transfer functions calculation ready meanwhile.
Force to Angle. It just means the filters that are in the POS OUTPUT matrix. I think in the past sometimes they are called F2P or F2A.
These filters account for the frequency dependent coupling of the DOFs around the suspension resonance. Take a look at what Bhavini is doing for the plots.
Edit Thu Apr 15 08:32:58 2021 :
Comments are from Rana.
Corrected the plot in the attachment. It shows the correct behavior at high frequencies now.
Here' s aquick update before we leave for lunch. We have managed to calculate some filter that would go on the POS column in MC2 output matrix filter banks aka F2A aka F2P filters. In the afternoon if we can come and work on the IMC, we'll try to load them on the output matrix. We have never done that so it might take some time for us to understand on how to do that. Attached is the bode plot for these proposed filters. Let us know if you have any comments.
We tried two sets of filters on the output matrix POS column in MC2. Both versions failed. Following are some details.
Looks mostly right, but you used the OSEM sensors as readbacks. We are diagonalizing using the cavity sensors. Using the diagonalized input matrix is also good since that will reduce the cross-coupling due to the damping loops.
Its sort of a subtle issue:
The filters were somewhat successful, how much we can see in attachment 1. The tip about difference between eigenmode basis and cartesian basis was the main thing that helped us take data properly. We still used OSEM data but rotated the output from POS, PIT, YAW to x, theta, phi (cartesian basis where x is also measured as angle projected by suspension length).
** The notation here is [UL, UR, LR, LL]
Following up from morning's work, I balanced the coils at DC as well. Attachment 1 is screenshot of striptool in which blue and red traces show ASCYAW and ASCPIT outputs when C1:SUS-MC2_LSC_OFFSET was switched by 500 counts. We see very slight disturbance but no real DC offset shown on PIT and YAW due to position step. This data was taken while nominal F2A filter calculated to balance coils at DC was uploaded
I have uploaded the filters on filter banks 7-10 where FM7 is the nominal filter with Q close to 1 and 8-10 are filters with Q 3, 7 and 10 respectively. The transfer function of these filters can be seen in Attachment 2. Note, that the high frequency gain drops a lot when higher Q filters are used.
These filters are designed such that the total DC gain after the application of coil outputs gains for high frequency balancing (as done in morning 16054) balances the coils at DC.
Since I had access to the complete output matrix that balances the coils to less than 1% cross coupling at high frequencies from 16009, I also did a quick test of DC coil balancing with this kind of high frequency balancing. In this case, I uploaded another set of filters which were made at Q close to 1 and gain such that effective DC gain matrix becomes what I got by balancing in the above case. This set of filter also worked as good as the above filters. This completes the proof that we can also use complete matrix for high frequency coil balancing which can be calculated by a script in 20min and works with DC coil balancing as well. In my opinion, this method is more clear and much faster than toggling values in coil output gains where we have only 4 values to optimize 6 cross-coupling parameters. But don't worry, I'm not wasting time on this and will abandon this effort for now, to be taken up in future.
Corrected the calculation of filters in case of Q different than . There was a bug in the code which I overlooked. I'll correct the filter bank modules tomorrow.
Edit Wed Apr 21 11:06:42 2021 :
I have uploaded the corrected foton filters. Please see attachment 3 for the transfer functions calculated by foton. They match the filters we intended to upload. Only after uploading and closing the foton filter, I realized that the X=7 filter plot (bottom left in attachment 3) does not have dB units on y-axis. It is plotted in linear y-scale (this plot in foton is for phase by default to I guess I forgot to change the scaling when repurposing it for my plot).
We did a step response test with MC2 Suspensoin Damping Gains and optimized them to get <5 oscillations in ringdown.
In the afternoon, we'll complete doing the above steps for MC1 and MC3. Their coil balancing has not been done on DC so, it is bit non-ideal right now. We'll look into scripting this process as well.
the POS column should be all 1 for the AC balancing. Where did those non-1 numbers come from?
Yes, during the AC balancing, POS column was set to all 1. This table shows the final values after all the steps. The first 3 columns are DC balancing results when output matrix was changed. While the last column is for AC balancing. During AC balancing, the output matrix was kept to ideal position as you suggested.
Note: The AC gains were measured by keeping output matrix to ideal values of 1s. When optimizing DC gains, the AC gains were uploaded in coil ouput gains.
The MC / WFS stability seemed off to me. Trending some channels at random, I saw that the MC3 PIT/YAW gains were restored mixed up (PIT was restored to YAW and vice versa) in the last day sometime - I wasn't sure what other settings are off so I did a global burtrestore from the last time I had the interferometer locked since those were settings that at least allow locking (I am not claiming they are optimal).
How are these settings being restored after the suspension optimization? If the burtrestore is randomly mixing up channels, seems like something we should be worried about and look into. I guess it'd also be helpful to make sure we are recording snapshots of all the channels we are changing - I'm not sure if the .req file gets updated automatically / if it really records every EPICS record. It'd be painful to lose some setting because it isn't recorded.
Unconnected to this work - the lights in the BS/PRM chamber were ON, so I turned them OFF. Also unconnected to this work, the summary pages job that updates the "live" plots every half hour seem to be dead again. There is a separate job whose real purpose is to wait for the data from EOD to be transferred to LDAS before filling in the last couple of hours of timeseries data, but seems to me like that is what is covering the entire day now.
The mix up was my fault I think. I restored the channels manually instead of using burt restore. Your message suggests that we can set burt to start noticing channel changes at home point and create a .req file that can be used to restore later. We'll try to learn how to do that. Right now, we only know how to burt restore using the existing snapshots from the autoburt directory, but they touch more things than we work on, I think. Or can we just always burt restore it to morning time? If yes, what snapshot files should we use?
Indeed, you can make your own snapshot by specifying the channels to snap in a .req file. But what I meant was, we should confirm that all the channels that we modify are already in the existing snapshot files in the autoburt dir. If it isn't, we should consider adding it. I think the whole burt system needs some cleaning up - a single day of burt snapshots occupies ~400MB (!) of disk space, but I think we're recording a ton of channels which don't exist anymore. One day...
Your message suggests that we can set burt to start noticing channel changes at home point and create a .req file that can be used to restore later. We'll try to learn how to do that. Right now, we only know how to burt restore using the existing snapshots from the autoburt directory, but they touch more things than we work on, I think. Or can we just always burt restore it to morning time? If yes, what snapshot files should we use?
Today we tested the F2A filters created from the DC gain values listed in 16066.