A longer measurement is set to trigger at 5:00 tomorrow on April 2nd, 2021. This measurement will run for 35 iterations with an excitation duration of 120s and bandwidth for CSD measurement set to 0.1 Hz. The script is set to trigger in a tmux session named 'cB' on pianosa.
Last run gave similar results as the quick run we did earlier. The code has been unable to strike out couplings with POS. We found the bug which is causing this. This was because the sampling rate of MC_F channel is different from the test-point channels used for PIT and YAW. Even though we were aware of it, we made an error in handling it while calculating CSD. Due to this, CSD calculation with POS data was performed by the code with zero padding which made it think that no PIT/YAW <-> POS coupling exist. Hence our code was only able to fix PIT <-> YAW couplings.
We'll need to do another run with this bug fixed. I'll update this post with details of the new measurement.
How should I try to understand why PIT and YAW are so different?
As mentioned in last post, we earlier made an error in making sure that all time series arrays go in with same sampling rate in CSD calculation. When we fixed that, our recursive method just blew out in all the efforts since then.
We suspect a major issue is how our measured sensing matrix (the cross-coupling matrix between different degrees of freedom on excitation) has significant imaginary parts in it. We discard the imaginary vaues and only use real parts for iterative method, but we think this is not the solution.
Here we present cross-spectral density of different channels representing the three sensed DOFs (normalized by ASD of no excitation data for each involved component) and the sensing matrix (TF estimate) calculated by normalizing the first cross spectral density plots column wise by the diagonal values. These are measured with existing ideal output matrix but with the new input matrix. This is to get an idea of how these elements look when we use them.
Note, that we used only 10 seconds of data in this run and used binwidth of 0.25Hz. When we used binwidth of 0.1 Hz, we found that the peaks were broad and highest at 13.1 Hz instead of 13 Hz which is the excitation frequency used in these measurements.
Basically I went around all the chambers and all the DB25 flanges to check the invac cable configurations. Also took more time to check the coil Rs and Ls.
Exceptions are the TTs. To avoid unexpected misalignment of the TTs, I didn't try to disconnect the TT cables from the flanges.
Upon the disconnection of the SOS cables, the following steps are taken to avoid large impact to the SOSs
After the measurement, IMC was lock and aligned. The two arms were locked and aligned with ASS. And the PRM alignment (when "misalign" was disengaged) was checked with the REFL CCD.
So I believe the SOSs are functioning as before, but if you find anything, please let me know.
Adding the chamfer around the edge of the optic ring did not change the center of mass relative to the plane from the suspension wires.
The CoM was .0003" away from the plane. Adding the chamfer moved it closer by .0001". See the attached photo.
I've also attached the list of the Moments of Inertia of the SOS Assembly.
To test if our method is working at all, we went for the simpler case of just uncoupling PIT and YAW. This is also because the sensor used for these two degrees of freedom is similar (the MC Trans WFS).
We saw a successful decrease in cross-coupling between PIT and YAW over the first 50 iterations that we tried. Here are some results:
Today, we finally crossed the last hurdle and got a successful converging coil balancing run.
We ran again this method but with the 'b' parameter as a matrix instead. This provides more gain on some off-diagonal terms than others. This gave us a better convergence with the code reaching to the tolerance level provided (0.01 distance of S matrix from identity) within 16 iterations (~17 mins).
Attachment 1 again shows how the off-diagonal terms go down and how the overall distance of sensing matrix from identity goes down. This is 'Cross coupling budget' of the coils as iterations move forward.
convergence is great.
Next we wanna get the F2A filters made since most of the IMC control happens at f < 3 Hz. Once you have the SUS state space model, you should be able to see how this can be done using only the free'swinging eigenfrequencies. Then you should get the closed loop model including the F2A filters and the damping filters to see what the closed loop behavior is like.
I think I mis-spoke about the balancing channels before. The ~20 Hz balancing could go into either the COIL banks or the SUS output matrix.
I believe its more conceptually clean to do this as gains in the outputmatrix, and leave the coil gains as +/- 1. i.e. we would only use the coil gains to compensate for coil/magnet actuation strength.
Then the high frequency balance goes into the outputmatrix. The F2A and A2L decoupling filters would then be generated having a high frequency gain = 1.
I'm not sure I understand what F2A is? I couldn't find a description of this filter anywhere and don't remember if you have already explained it. Can you describe what is needed to be done again, please? We would keep SUS state space model and seismic transfer functions calculation ready meanwhile.
Force to Angle. It just means the filters that are in the POS OUTPUT matrix. I think in the past sometimes they are called F2P or F2A.
These filters account for the frequency dependent coupling of the DOFs around the suspension resonance. Take a look at what Bhavini is doing for the plots.
Edit Thu Apr 15 08:32:58 2021 :
Comments are from Rana.
Corrected the plot in the attachment. It shows the correct behavior at high frequencies now.
Here' s aquick update before we leave for lunch. We have managed to calculate some filter that would go on the POS column in MC2 output matrix filter banks aka F2A aka F2P filters. In the afternoon if we can come and work on the IMC, we'll try to load them on the output matrix. We have never done that so it might take some time for us to understand on how to do that. Attached is the bode plot for these proposed filters. Let us know if you have any comments.
We tried two sets of filters on the output matrix POS column in MC2. Both versions failed. Following are some details.
Looks mostly right, but you used the OSEM sensors as readbacks. We are diagonalizing using the cavity sensors. Using the diagonalized input matrix is also good since that will reduce the cross-coupling due to the damping loops.
Its sort of a subtle issue:
The filters were somewhat successful, how much we can see in attachment 1. The tip about difference between eigenmode basis and cartesian basis was the main thing that helped us take data properly. We still used OSEM data but rotated the output from POS, PIT, YAW to x, theta, phi (cartesian basis where x is also measured as angle projected by suspension length).
** The notation here is [UL, UR, LR, LL]
Following up from morning's work, I balanced the coils at DC as well. Attachment 1 is screenshot of striptool in which blue and red traces show ASCYAW and ASCPIT outputs when C1:SUS-MC2_LSC_OFFSET was switched by 500 counts. We see very slight disturbance but no real DC offset shown on PIT and YAW due to position step. This data was taken while nominal F2A filter calculated to balance coils at DC was uploaded
I have uploaded the filters on filter banks 7-10 where FM7 is the nominal filter with Q close to 1 and 8-10 are filters with Q 3, 7 and 10 respectively. The transfer function of these filters can be seen in Attachment 2. Note, that the high frequency gain drops a lot when higher Q filters are used.
These filters are designed such that the total DC gain after the application of coil outputs gains for high frequency balancing (as done in morning 16054) balances the coils at DC.
Since I had access to the complete output matrix that balances the coils to less than 1% cross coupling at high frequencies from 16009, I also did a quick test of DC coil balancing with this kind of high frequency balancing. In this case, I uploaded another set of filters which were made at Q close to 1 and gain such that effective DC gain matrix becomes what I got by balancing in the above case. This set of filter also worked as good as the above filters. This completes the proof that we can also use complete matrix for high frequency coil balancing which can be calculated by a script in 20min and works with DC coil balancing as well. In my opinion, this method is more clear and much faster than toggling values in coil output gains where we have only 4 values to optimize 6 cross-coupling parameters. But don't worry, I'm not wasting time on this and will abandon this effort for now, to be taken up in future.
Corrected the calculation of filters in case of Q different than . There was a bug in the code which I overlooked. I'll correct the filter bank modules tomorrow.
Edit Wed Apr 21 11:06:42 2021 :
I have uploaded the corrected foton filters. Please see attachment 3 for the transfer functions calculated by foton. They match the filters we intended to upload. Only after uploading and closing the foton filter, I realized that the X=7 filter plot (bottom left in attachment 3) does not have dB units on y-axis. It is plotted in linear y-scale (this plot in foton is for phase by default to I guess I forgot to change the scaling when repurposing it for my plot).
We did a step response test with MC2 Suspensoin Damping Gains and optimized them to get <5 oscillations in ringdown.
In the afternoon, we'll complete doing the above steps for MC1 and MC3. Their coil balancing has not been done on DC so, it is bit non-ideal right now. We'll look into scripting this process as well.
the POS column should be all 1 for the AC balancing. Where did those non-1 numbers come from?
Yes, during the AC balancing, POS column was set to all 1. This table shows the final values after all the steps. The first 3 columns are DC balancing results when output matrix was changed. While the last column is for AC balancing. During AC balancing, the output matrix was kept to ideal position as you suggested.
Note: The AC gains were measured by keeping output matrix to ideal values of 1s. When optimizing DC gains, the AC gains were uploaded in coil ouput gains.
The MC / WFS stability seemed off to me. Trending some channels at random, I saw that the MC3 PIT/YAW gains were restored mixed up (PIT was restored to YAW and vice versa) in the last day sometime - I wasn't sure what other settings are off so I did a global burtrestore from the last time I had the interferometer locked since those were settings that at least allow locking (I am not claiming they are optimal).
How are these settings being restored after the suspension optimization? If the burtrestore is randomly mixing up channels, seems like something we should be worried about and look into. I guess it'd also be helpful to make sure we are recording snapshots of all the channels we are changing - I'm not sure if the .req file gets updated automatically / if it really records every EPICS record. It'd be painful to lose some setting because it isn't recorded.
Unconnected to this work - the lights in the BS/PRM chamber were ON, so I turned them OFF. Also unconnected to this work, the summary pages job that updates the "live" plots every half hour seem to be dead again. There is a separate job whose real purpose is to wait for the data from EOD to be transferred to LDAS before filling in the last couple of hours of timeseries data, but seems to me like that is what is covering the entire day now.
The mix up was my fault I think. I restored the channels manually instead of using burt restore. Your message suggests that we can set burt to start noticing channel changes at home point and create a .req file that can be used to restore later. We'll try to learn how to do that. Right now, we only know how to burt restore using the existing snapshots from the autoburt directory, but they touch more things than we work on, I think. Or can we just always burt restore it to morning time? If yes, what snapshot files should we use?
Indeed, you can make your own snapshot by specifying the channels to snap in a .req file. But what I meant was, we should confirm that all the channels that we modify are already in the existing snapshot files in the autoburt dir. If it isn't, we should consider adding it. I think the whole burt system needs some cleaning up - a single day of burt snapshots occupies ~400MB (!) of disk space, but I think we're recording a ton of channels which don't exist anymore. One day...
Your message suggests that we can set burt to start noticing channel changes at home point and create a .req file that can be used to restore later. We'll try to learn how to do that. Right now, we only know how to burt restore using the existing snapshots from the autoburt directory, but they touch more things than we work on, I think. Or can we just always burt restore it to morning time? If yes, what snapshot files should we use?
Today we tested the F2A filters created from the DC gain values listed in 16066.
We extended the f2a filter implementation and diagnostics as summarized in 16086 to MC1 and MC3.
Attachment 1 shows the filters with Q=3, 7, 10. We diagnosed using Q=3.
Attachment 2 shows the test summary, exciting with broadband noise on the LSC_EXC and measuring the CSD to estimate the transfer functions.
Attachment 3 shows the filters with Q=3, 7, 10. We diagnosed using Q=3.
Attachment 4 shows the test summary, exciting with broadband noise on the LSC_EXC and measuring the CSD to estimate the transfer functions.
Our main observation (and difference) with respect to MC2 is the filters have relative success for the PIT cross-coupling and not so much for YAW. We already observed this when we tuned the DC output gains to compute the filters.
We ran the f2a filter test for MC1, MC2, and MC3.
The new filters differ from previous versions by a adding non-unity Q factor for the pole pairs as well.
This in terms of zpk is: [ [zr + i zi, zr - i zi], [pr + i pi, pr - i pi], 1] where
We uploaded all these filters using foton, into the three last FM slots on the POS output gain coil.
We ran tests on all suspended optics using the following (nominal) procedure:
C1:IOO-WFS_GAIN to 0.05.
** Excitation = 0.05 - 3.5 Hz uniform noise, 100 amplitude, 100 gain
I have uploaded all the new settings mentioned in 16066 and 16072. The settings were uploaded through a single script present at anchal/20210428_IMC_Tuned_Suspension/uploadNewConfigIMC.py. The settings can be reverted back to old settings through anchal/20210428_IMC_Tuned_Suspension/restoreOldConfigIMC.py. Both these scripts can be run only through python3 in donatella or allegra.
GPSTIME of new settings: 1303690144
New settings include:
We'll wait and watch the performance through summary pages and check back the performance on Monday.
In 16087 we mentioned that we were unable to do a step response test for WFS loop to get an estimate of their UGF. The primary issue there was that we were not putting the step at the right place. It should go into the actuator directly, in this case, on C1:SUS-MC2_PIT_COMM and C1:SUS-MC2_YAW_COMM. These channels directly set an offset in the control loop and we can see how the error signals first jump up and then decay back to zero. The 'half-time' of this decay would be the inverse of the estimated UGF of the loop. For this test, the overall WFS loops gain, C1:IOO-WFS_GAIN was set to full value 1. This test is performed in the changed settings uploaded in 16091.
I did this test twice, once giving a step in PIT and once in YAW.
Attachment 1 is the striptool screenshot for when PIT was given a step up and then step down by 0.01.
Attachment 2 is the striptool screenshot when YAW was given a step up and down by 0.01. Note the difference in x-scale in this plot.
With the input matrix, coil ouput gains and F2A filters loaded as in 16091, I tested the suspension loops' step response to offsets in LSC, ASCPIT and ASCYAW channels, before and after applying the "new damping gains" mentioned in 16066 and 16072. If these look better, we should upload the new (higher) damping gains as well. This was not done in 16091.
Note that in the plots, I have added offsets in the different channels to plot them together, hence the units are "au".
We repeated the same test with IMC unlocked. We had found these gains when IMC was unlocked and their characterization needs to be done with no light in the cavity. attached are the results. Everything else is same as before.
Edit Tue May 4 14:43:48 2021 :
Overall, I would recommend setting the new gains in the suspension loops as well to observe long term effects too.
We have uploaded the new damping gains on all the suspensions of IMC. This completes changing all the configuration to as mentioned in 16066 and 16072. The old setting can be restored by running python3 /users/anchal/20210505_IMC_Tuned_SUS_with_Gains/restoreOldConfigIMC.py from allegra or donatella.
We picked a few parameters from 40m summary page and plotted them to see the effect of new settings. On April 4th, old settings were present. On April 28th (16091), new input matrices and F2A filters were uploaded but suspension gains remained the same. On May 5th (16120), we uploaded new (higher) suspension gains. We chose Sundays on UTC so that it lies on weekends for us. Most probably nobody entered 40m and it was calmer in the institute as well.
We can download data and plot comparisons ourselves and maybe calculate the spectrums of MC_TRANS_PIT/YAW and MC_REFL_DC when IMC was locked. But we want to know if anyone has better ways of characterizing the settings that we should know of before we get into this large data handling which might be time-consuming. From this preliminary 40m summary page plots, maybe it is already clear that we should go back to old settings. Awaiting orders.
No, this is the property of the suspension assembly. The mass says 10kg
Could you do the same for the testmass assembly (only the suspended part)? The units are good, but I expect that the values will be small. I want to keep at least three significant digits.
Here are the mass properties for the only the test mass assembly (optic, 3" ring, and wire block). (Updated with g*mm^2)
We came in the morning with the following scene on the zita monitor:
The MC1 watchdog was tripped and seemed like IMC struggled all night with misconfigured WFS offsets. After restoring the MC1 WD, clearing the WFS offsets, and seeing the suspension damp, the MC caught lock. It wasn't long before the MC unlocked, and the MC1 WD tripped again.
We tried few things, not sure what order we tried them in:
Nothing worked. We kept seeing that ULPD var on MC1 keeps showing kicks every few minutes which jolts the suspension loops. So we decided to record some data with PSL shutter closed and just suspension loops on. Then we switched off the loops and recorded some data with freely swinging optic. Even when optic was freely swinging, we could see impulses in the MC1 OSEM UL PD var which were completely uncorrelated with any seismic activity. Infact, last night was one fo teh calmer nights seismically speaking. See attachment 2 for the time series of OSEM PD variance. Red region is when the coil outputs were disabled.
Edit Thu May 13 14:47:25 2021 :
Added OSEM Sensor timeseries data on the plots as well. The UL OSEM sensor data is the only channel which is jumping hapazardly (even during free swinging time) and varying by +/- 30. Other sensors only show some noise around a stable position as should be the case for a freely suspended optic.
Koji and I did a few tests with an OSEM emulator on the satellite amplifier box used for MC1 which is housed on 1X4. This sat box unit is S2100029 D1002812 that was recently characterized by me 15803. We found that the differential output driver chip AD8672ARZ U2A section for the UL PD was not working properly and had a fluctuating offset at no input current from the PD. This was the cause of the ordeal of the morning. The chip was replaced with a new one from our stock. The preliminary test with the OSEM emulator showed that the channel has the correct DC value.
In further testing of the board, we found that the channel 8 LED driver was not working properly. Although this channel is never used in our current cable convention, it might be used later in the future. In the quest of debugging the issue there, we replaced AD8672ARZ at U1 on channel 8. This did not solve the issue. So we opened the front panel and as we flipped the board, we found that the solder blob shorted the legs of the transistor Q1 2N3904. This was replaced and the test with the LED out and GND shorted indicated that the channel is now properly providing a constant current of 35mA (5V at the monitor out).
After the debugging, the UL channel became the least noisy among the OSEM channels! Mode cleaner was able to lock and maintain it.
We should redo the MC1 input matrix optimization and the coil balancing afterward as we did everything based on the noisy UL OSEM values.
I want to work on the IFO this weekend, so I reverted the IMC suspension settings just now to what I know work (until the new settings are shown quantitatively to be superior). There isn't any instruction here on how to upload the new settings, so after my work, I will just restore from a burt-snapshot from before I changed settings.
In the process, I found something odd in the MC2 coil output filter banks. Attachment #1 shows what it it is today. This weird undetermined state of FM9 isn't great - I guess this flew under the radar because there isn't really any POS actuation on MC2. Where did the gain1 filter I installed go? Some foton filter file corruption? Eventually, we should migrate FM7,FM8-->FM9,FM10 but this isn't on my scope of things to do for today so I am just putting the gain1 filter back so as to have a clean FM9 switched on.
The old setting can be restored by running python3 /users/anchal/20210505_IMC_Tuned_SUS_with_Gains/restoreOldConfigIMC.py from allegra or donatella.
I wrote the values from the c1mcs burt snapshot from ~1400 Saturday May 15, at ~1600 Sunday May 16. I believe this undoes all my changes to the IMC suspension settings.
Calculation for the SOS POS/PIT/YAW resonant frequencies
- Nominal height gap between the CoM and the wire clamping point is 0.9mm (cf T970135)
- To have the similar res freq for the optic with the 3" metal sleeve is 1.0~1.1mm.
As the previous elog does not specify this number for the current configuration, we need to asses this value and the make the adjustment of the CoM height.
For future reference, the new settings can be upoaded from a script in the same directory. Run python /users/anchal/20210505_IMC_Tuned_SUS_with_Gains/uploadNewConfigIMC.py from allegra.
There isn't any instruction here on how to upload the new settings
11 new Satellite Amps were picked up from Downs. 7 more are coming from there. I have one spare unit I made. 1 sat amp has already been used at MC1.
We had 8 HAM-A coil drivers delivered from the assembling company. We also have two coil drivers delivered from Downs (Anchal tested)
We've set a free swing test to trigger at 3:30 am tomorrow for MC1. The script for tests is running on tmux session named 'freeSwingMC1' on rossa. The script will run for about 4.5 hrs and we'll correct the input matrix tomorrow from the results. If anyone wants to work during this time (3:30 am to 8:00 am), you can just kill the script by killing tmux session on rossa. ssh into rossa and type tmux kill-session -t freeSwingMC1.
The test was succesful and brought back the IMC to lock point at the end.
We calculated new input matrix using same code in scripts/SUS/InMatCalc/sus_diagonalization.py . Attachment 1 shows the results.
The calculations are present in scripts/SUS/InMatCalc/MC1.
We uploaded the new MC1 input matrix at:
Unix Time = 1621963200
GPS Time = 1305998418
This was done by running python scripts/SUS/general/20210525_NewMC1Settings/uploadNewConfigIMC.py on allegra. Old IMC settings (before Paco and I started workin on 40m) can be restored by running python scripts/SUS/general/20210525_NewMC1Settings/restoreOldConfigIMC.py on allegra.
Everything looks as stable as before. We'll look into long term trends in a week to see if this helped at all.
The current vertical distance between the CoM and the wire clamping point on the 3" Ring assembly is 0.33mm. That is the CoM is .33 mm below the clamping point of the wire. I took the clamping point to be the top edge of the wire clamp piece. see the below attachments.
I am now modifying the dumbell mechanism at the bottom of the ring to move the CoM to the target distance of 1.1mm.
After changing the material of the Balance Mass from 6061 Al to 304 Steel, and changing the thickness to 0.21" from 0.25". The CoM is now 1.11mm below the clamping point.
Koji expected a mass change of ~ 4g to move the mass to 1.1mm. The 6061 mass weighed ~1.31g and the 304 mass weighs 4.1g.
A potential issue with this is the screw used the adjust the position of these balance masses, threads through both the aluminum ring and this now 304 steel mass. A non silver plated screw could cold weld at the mass, but a silver plated screw will gall in the aluminum threads.