I installed the EPICs base, asyn and modbus modules according to Jon's instructions.
Since the modbus configurations files were already writtten for c1auxey1 (see elog 15292) the only thing I did was to change the IP addresses in ETMYaux.cmd to match the actual assigned IPs.
to match the actual assigned IPs.
I followed the rest of the instructions as written.
The modbus service was activated succesfully.
The only thing left to do is to change ETMYaux.db to reflect to new channels that were added. I believe these are BI channels named C1:SUS-ETMY_xx_ENABLEMon.
to reflect to new channels that were added. I believe these are BI channels named C1:SUS-ETMY_xx_ENABLEMon.
The other day I felt hot at the X end. I wondered if the Xend A/C was off, but the switch right next to the SP table was ON (green light).
I could not confirm if the A/C was actually blowing or not.
I double checked today and the F2A filters in the output matrices of MC1, MC2 and MC3 in the POS column are ON. I do not get what SDF means? Did we need to add these filters elsewhere?
The IMC suspension team should double check their filters are on again. I am not familiar with the settings and I don't think they've been added to the SDF.
With the input matrix, coil ouput gains and F2A filters loaded as in 16091, I tested the suspension loops' step response to offsets in LSC, ASCPIT and ASCYAW channels, before and after applying the "new damping gains" mentioned in 16066 and 16072. If these look better, we should upload the new (higher) damping gains as well. This was not done in 16091.
Note that in the plots, I have added offsets in the different channels to plot them together, hence the units are "au".
We received a stock of DB9 male feed-through connectors. That allowed me to complete the remaining wiring on the c1auxey Acromag chassis. The only thing left to be done is the splicing to the RTS.
The SDF system is supposed to help with restoring the correct settings, complementary to burt. My personal opinion is that there is no need to commit these filters to SDF until we're convinced that they help with the locking / noise performance.
I double checked today and the F2A filters in the output matrices of MC1, MC2 and MC3 in the POS column are ON. I do not get what SDF means? Did we need to add these filters elsewhere
Now that the model is finally compiled I need to make an medm screen for it and put it in the c1sim:/home/controls/docker-cymac/userapps/medm/ directory.
But before doing that I really want to test it using the autogenerated medm screens which are in the virtual cymac in the folder /opt/rtcds/tst/x1/medm/x1sup. In Jon's post he said that I can use the virtual path for sitemap after running $ eval $(./env_cymac)
We finished the installation procedure on the c1auxey1 host machine. There were some adjustments that had to be made for Debian 10. The slow machine wiki page has been updated.
A test database file was made were all the channel names were changed from C1 to C2 in order to not interfere with the existing channels.
We starting testing the channels one by one to check the wiring and the EPICs software. We found some misswirings and fixed them.
Its getting late. I'll continue with the rest of the channels on Monday.
Notice that for all the AI channels the RTN was disconnected while testing.
WFS1 noise injection
C1:LSC-XARM_IN1_DQ / C1:LSC-YARM_IN1_DQ
C1:SUS-ETMX_LSC_OUT_DQ / C1:SUS-ETMY_LSC_OUT_DQ
C1:SUS-MC1_**COIL_OUT / C1:SUS-MC2_**COIL_OUT / C1:SUS-MC3_**COIL_OUT
C1:IOO-WFS1_PIT_ERR / C1:IOO-WFS1_YAW_ERR
** denotes [UL, UR, LL, LR]; the output coils.
When the cymac is started it gives me a list of channels shown below.
$ Initialized TP interface node=8, host=98e93ecffcca
$ Creating X1:DAQ-DC0_X1IOP_STATUS
$ Creating X1:DAQ-DC0_X1IOP_CRC_CPS
$ Creating X1:DAQ-DC0_X1IOP_CRC_SUM
$ Creating X1:DAQ-DC0_X1SUP_STATUS
$ Creating X1:DAQ-DC0_X1SUP_CRC_CPS
$ Creating X1:DAQ-DC0_X1SUP_CRC_SUM
But when I enter it into the Diaggui I get an error:
The following channel could not be found:
My guess is that need to connect to the Diaggui to something that can access those channels. I also need to figure out what those channels are.
We repeated the same test with IMC unlocked. We had found these gains when IMC was unlocked and their characterization needs to be done with no light in the cavity. attached are the results. Everything else is same as before.
Edit Tue May 4 14:43:48 2021 :
Overall, I would recommend setting the new gains in the suspension loops as well to observe long term effects too.
I found a "vice" in the cleanroom (attachment 1). I used it to push dowel pins into the last suspension block using some alcohol as a lubricant.
I then assembled the 7th and last suspension tower (attachment 2).
Things that need to be done:
1. Push Viton tips into vented screws and assemble the earthquake stops.
2. Glue magnets to dumbells.
Rana came and helped us figure us where to inject the noise. Following are the characteristics of the test we did:
Attachment 1 shows a screenshot with awggui and diaggui screens displaying the signal in both angular and longitudinal channels.
Attachment 2 shows the analogous screenshot for MC2.
It seemed like the BIO channels were not working, both the inputs and the outputs. The inputs were working on the windows machine though. That is, when we shorted the BIO channel to the return, or put 0V on it, we could see the LED turn on on the I/O testing screen and when we ramped up the voltage above 3 the LED turned off. This is the expected behavior from a sinking digital input. However, the EPICs caget didn't show any change. All the channels were stuck on Disabled.
We checked the digital outputs by connecting the channels to a fluke. Initially, the fluke showed 13V. We tried to toggle the digital output channels with caput and that didn't work. We checked the outputs with the windows software. For that, we needed to stop the Modbus. To our surprise, the windows software was not able to flip the channels either. We realized that this BIO Acromag unit is probably defective. We replaced it with a different unit and put a warning sticker on the defective unit. Now, the digital outputs were working as expected. When we turned them on the voltage output dropped to 0V. We checked the channels with the EPICs software. We realized that these channels were locked with the closed loop definition. We turned on the channels tied to these output channels (watchdog and toggles) and it worked. The output channels can be flipped with the EPICs software. We checked all the digital output channels and fixed some wiring issues along the way.
The digital input channels were still not working. This is a software issue that we will have to deal with later.
(Yehonathan) Rana noticed that the BNC leads on the chassis front panel didn't have isolation on them so I redid them with shrinking tubes.
With all the PCIe issues now resolved, yesterday I proceeded to build an IOP model for each of new FEs. I assigned them names and DCUIDs consist with the 40m convention, listed below. These models currently exist on only the cloned copy of /opt/rtcds running on the test stand. They will be copied to the main network disk later, once the new systems are fully tested.
The models compile and install successfully. The RCG runtime diagnostics indicate that all is working except for the timing synchronization and DAQD data transmission. This is as expected because neither of these have been set up yet.
The next step is to provide the 65 kHz clock signals from the timing fanout via LC optical fiber. I overlooked the fact that an SPX optical transceiver is required to interface the fiber to the timing slave board. These were not provided with the timing slaves we received. The timing slaves require a particular type of transceiver, 100base-FX/OC-3, which we did not have on hand. (For future reference, there is a handy list of compatible transceivers in E080541, p. 14.) I placed a Digikey order for two Finisar FTLF1217P2BTL, which should arrive within two days.
We redid the WFS noise injection test and have compiled some results on noise contribution in arm cavity noise and IMC frequency noise due to angular noise of IMC.
Attachment 1: Shows the calibrated noise contribution from MC1 ASCPIT OUT to ARM cavity length noise and IMC frequency noise.
After a helpful meeting with Jon, we realized that I have somehow corrupted the sitemap file. So I am going to use the code Chris wrote to regenerate it.
Also, I am going to connect the controller using the IPC parts. The error that I was having before had to do with the IPC parts not being connected properly.
I put the box containing the untested OSEMs from KAGRA near the south flow bench on the floor.
We have uploaded the new damping gains on all the suspensions of IMC. This completes changing all the configuration to as mentioned in 16066 and 16072. The old setting can be restored by running python3 /users/anchal/20210505_IMC_Tuned_SUS_with_Gains/restoreOldConfigIMC.py from allegra or donatella.
Assembled chassis from De Leone placed in the 40 Meter Lab, along the west wall and under the display pedestal table. The leftover parts are in smaller Really Useful boxes, also on the parts pile along the west wall.
I added the IPC parts back to the plant model so that should be done now. It looks like this again here.
I can't seem to find the control model which should look like this. When I open sus_single_control.mdl, it just shows the C1_SUS_SINGLE_PLANT.mdl model. Which should not be the case.
When using mdl2adl I was getting the error:
$ cd /home/controls/mdl2adl
$ ./mdl2adl x1sup.mdl
error: set $site and $ifo environment variables
to set these in the terminal use the following commands:
$ export site=tst
$ export ifo=x1
On most of the systems, there is a script that automatically runs when a terminal is opened that sets these but that hasn't been added here so you must run these commands every time you open the terminal when you are using mdl2adl.
I copied c1scx.mdl to the docker to attach to the plant using the commands:
$ ssh nodus.ligo.caltech.edu
$ cd opt/rtcds/userapps/release/isc/c1/models/simPlant
$ scp c1scx.mdl controls@c1sim:/home/controls/docker-cymac/userapps
We today measured the calibration factors for XARM_OUT and YARM_OUT in nm/cts and replotted our results from 16117 with the correct frequency dependence.
Calibration of XARM_OUT and YARM_OUT
Inferring noise contributions to arm cavities:
Edit Mon May 10 18:31:52 2021
See corrections in 16129.
A few corrections to last analysis:
Today I brought and installed the new optical transceivers (Finisar FTLF1217P2BTL) for the two timing slaves. The timing slaves appear to phase-lock to the clocking signal from the master fanout. A few seconds after each timing slave is powered on, its status LED begins steadily blinking at 1 Hz, just as in the existing 40m systems.
However, some other timing issue remains unresolved. When the IOP model is started (on either FE), the DACKILL watchdog appears to start in a tripped state. Then after a few minutes of running, the TIM and ADC indicators go down as well. This makes me suspect the sample clocks are not really phase-locked. However, the models do start up with no error messages. Will continue to debug...
Did you match the local PC time with the GPS time?
Attached is the control loop diagram when main laser is locked to IMC and a single arm (XARM) is locked to the transmitted light from IMC.
Working with Chris, we decided that it is probably better to use a simple filter module as a controller before we make the model more complicated. I will use the plant model that I have already made (see attachment 1 of this). then attach a single control filter module to that: as seen in attachment 1. because I only want to work with one degree of freedom (position) I will average the four outputs which should give me the position. Then by feeding the same signal to all four inputs I should isolate one degree of freedom while still using the premade plant model.
The model I made that is shown in attachment 2 is the model I made from the plan. And it complies! yay! I think there is a better way to do the average than the way I showed. And since the model is feeding back on itself I think I need to add a delay which Rana noted a while ago. I think it was a UnitDelay (see page 41 of RTS Developer’s Guide). So I will add that if we run into problems but I think there is enough going on that it might already be delayed.
Since our model (x1sup_isolated.mdl) has compiled we can open the medm screens for it. I provide a procedure below which is based on Jon's post.
$ cd docker-cymac
$ eval $(./env_cymac)
$ medm -x /opt/rtcds/tst/x1/medm/x1sup_isolated/X1SUP_ISOLATED_GDS_TP.adl
To see a list of all medm screens use:
$ cd docker-cymac
# cd /opt/rtcds/tst/x1/medm/x1sup_isolated
Some of the other useful ones are:
See attachment 4. This screen shows the POS plant filter module that will be filled by the filter representing the transfer function of a damped harmonic oscillator:
THIS TF HAS BEEN UPDATED SEE NEXT POST
The first one of these screens that are of interest to us (shown in attachment 3) is the X1SUP_ISOLATED_GDS_TP.adl screen, which is the CDS runtime diagnostics screen. This screen tells us "the success/fail state of the model and all its dependencies." I am still figuring out these screens and the best guide is T1100625.
The next step is taking some data and seeing if I can see the position damp over time. To do this I need to:
No, this is the property of the suspension assembly. The mass says 10kg
Could you do the same for the testmass assembly (only the suspended part)? The units are good, but I expect that the values will be small. I want to keep at least three significant digits.
Here are the mass properties for the only the test mass assembly (optic, 3" ring, and wire block). (Updated with g*mm^2)
We came in the morning with the following scene on the zita monitor:
The MC1 watchdog was tripped and seemed like IMC struggled all night with misconfigured WFS offsets. After restoring the MC1 WD, clearing the WFS offsets, and seeing the suspension damp, the MC caught lock. It wasn't long before the MC unlocked, and the MC1 WD tripped again.
We tried few things, not sure what order we tried them in:
Nothing worked. We kept seeing that ULPD var on MC1 keeps showing kicks every few minutes which jolts the suspension loops. So we decided to record some data with PSL shutter closed and just suspension loops on. Then we switched off the loops and recorded some data with freely swinging optic. Even when optic was freely swinging, we could see impulses in the MC1 OSEM UL PD var which were completely uncorrelated with any seismic activity. Infact, last night was one fo teh calmer nights seismically speaking. See attachment 2 for the time series of OSEM PD variance. Red region is when the coil outputs were disabled.
Edit Thu May 13 14:47:25 2021 :
Added OSEM Sensor timeseries data on the plots as well. The UL OSEM sensor data is the only channel which is jumping hapazardly (even during free swinging time) and varying by +/- 30. Other sensors only show some noise around a stable position as should be the case for a freely suspended optic.
Koji and I did a few tests with an OSEM emulator on the satellite amplifier box used for MC1 which is housed on 1X4. This sat box unit is S2100029 D1002812 that was recently characterized by me 15803. We found that the differential output driver chip AD8672ARZ U2A section for the UL PD was not working properly and had a fluctuating offset at no input current from the PD. This was the cause of the ordeal of the morning. The chip was replaced with a new one from our stock. The preliminary test with the OSEM emulator showed that the channel has the correct DC value.
In further testing of the board, we found that the channel 8 LED driver was not working properly. Although this channel is never used in our current cable convention, it might be used later in the future. In the quest of debugging the issue there, we replaced AD8672ARZ at U1 on channel 8. This did not solve the issue. So we opened the front panel and as we flipped the board, we found that the solder blob shorted the legs of the transistor Q1 2N3904. This was replaced and the test with the LED out and GND shorted indicated that the channel is now properly providing a constant current of 35mA (5V at the monitor out).
After the debugging, the UL channel became the least noisy among the OSEM channels! Mode cleaner was able to lock and maintain it.
We should redo the MC1 input matrix optimization and the coil balancing afterward as we did everything based on the noisy UL OSEM values.
I believe I did the identical test with the one in [40m ELOG 15786]. The + input of PA95 was shorted to the ground to exclude the noise from the bias input. The voltage noise at TP6 was measured with +/-300V supply by two HP6209 and two Matsusada R4G360.
With R4G360, the floor level was identical and 60Hz line peaks were less. It looks like R4G360 is cheap, easier and precise to handle, and sufficiently low noise.
The PSL was too hot, so I turned on the south HEPA on the PSL. The north one was on and the south one was off (or so slow as to be inaudible and no vibration, unlike the north one). Lets watch the trend over the weekend and see if the temperature comes down and if the PMC / WFS variations get less. Fri May 14 17:46:26 2021
The NPRO has been off since ~1AM this morning it looks like. Is this intentional? Can I turn it back on (or at least try to)? The interlock signal we are recording doesn't report getting tripped but I think this has been the case in the past too.
After getting the go ahead from Koji, I turned the NPRO back on, following the usual procedure of diode current ramping. PMC and IMC locked. Let's see if this was a one-off or something chronic.
I want to work on the IFO this weekend, so I reverted the IMC suspension settings just now to what I know work (until the new settings are shown quantitatively to be superior). There isn't any instruction here on how to upload the new settings, so after my work, I will just restore from a burt-snapshot from before I changed settings.
In the process, I found something odd in the MC2 coil output filter banks. Attachment #1 shows what it it is today. This weird undetermined state of FM9 isn't great - I guess this flew under the radar because there isn't really any POS actuation on MC2. Where did the gain1 filter I installed go? Some foton filter file corruption? Eventually, we should migrate FM7,FM8-->FM9,FM10 but this isn't on my scope of things to do for today so I am just putting the gain1 filter back so as to have a clean FM9 switched on.
The old setting can be restored by running python3 /users/anchal/20210505_IMC_Tuned_SUS_with_Gains/restoreOldConfigIMC.py from allegra or donatella.
I wrote the values from the c1mcs burt snapshot from ~1400 Saturday May 15, at ~1600 Sunday May 16. I believe this undoes all my changes to the IMC suspension settings.
Looks like the fan lowered the temperature as expected. Need to get a few more days of data to see if its stabilized, or if that's just a fluke.
The vertical line at 00:00 UTC May 18 is about when I turned the fans up/on.
Fluke. Temp fluctuations are as usual, but the overall temperature is still lower. We ought to put some temperature sensors at the X & Y ends to see what's happening there too.
Calculation for the SOS POS/PIT/YAW resonant frequencies
- Nominal height gap between the CoM and the wire clamping point is 0.9mm (cf T970135)
- To have the similar res freq for the optic with the 3" metal sleeve is 1.0~1.1mm.
As the previous elog does not specify this number for the current configuration, we need to asses this value and the make the adjustment of the CoM height.
For future reference, the new settings can be upoaded from a script in the same directory. Run python /users/anchal/20210505_IMC_Tuned_SUS_with_Gains/uploadNewConfigIMC.py from allegra.
There isn't any instruction here on how to upload the new settings
HP HV power supply ( HP6209 ) were returned to Downs
11 new Satellite Amps were picked up from Downs. 7 more are coming from there. I have one spare unit I made. 1 sat amp has already been used at MC1.
We had 8 HAM-A coil drivers delivered from the assembling company. We also have two coil drivers delivered from Downs (Anchal tested)
DC Power Strip Assemblies delivered and stored behind the Y arm tube (Attachment 1)
I also moved the spare 1U Chassis to the same place.
The transfer function given in the previous post was slightly incorrect the units did not make sense the new function is:
I have attached a quick derivation below in attachment 1
We went into 40m to identify where XARM PDH loop control elements are. We didn't touch anything, but this is to note we went in there twice at 10 AM and 11:10 AM.
The plant transfer function of the pendulum in the s domain is:
Using Foton to make a plot of the TF needed and using m=40kg, w0=3Hz, and Q=50 (See attachment 1). It is easiest to enter the above filter using RPoly and saved it as Plant_V1
The new HAM-A coil drivers have a single DB9 connector for all the binary inputs. This requires that the dewhitening switching signals from the fast system be spliced with the coil enable signals from c1auxey. There is a common return for all the binary inputs. To avoid directly connecting the grounds of the two systems, I have looked for a suitable opto-isolator for the c1auxey signals.
I best option I found is the Ocean Controls KTD-258, a 4-channel, DIN-rail-mounted opto-isolator supporting input/output voltages of up to 30 V DC. It is an active device and can be powered using the same 15 V supply as is currently powering both the Acromags and excitation. I ordered one unit to be trialed in c1auxey. If this is found to be good solution, we will order more for the upgrades of c1auxex and c1susaux, as required for compatibility with the new suspension electronics.
- High priority units: 2x 18AI / 1x 16AI / 3x 16AA
All six are reworked and on the electronics workbench. The rest should be ready by the end of the week.