I understand this mst be frustrating for you. But we did not change these settings, knowingly atleast. We have documented all the things we did there. The only thing I can think of which could possibly change any of those channels are the scripts that we ran that are mentioned and the burt restore that we did on all channels (which wasn't really necessary). We promise to be more vigilant of changes that occur when we are present in future.
To my dismay, I found today that somebody had changed the oscillator frequencies for the sensing matrix infrastructure we have. The change happened 2 days and 2 hours ago (I write this at ~1230 on Saturday, 3/6), i.e. ~1030am on Thursday. According to the elog, this is when Anchal and Paco were working on the interferometer, but I can find no mention of these settings being changed. Not cool guys 😒 .
This was relatively easy to track down but I don't know what else may have been messed with. I don't understand how anything that was documented in the elog can lead to this weird doubling of the frequencies.
I have now restored the correct settings. The "sensing matrix" I posted last night is obviously useless.
No we didn't fix the issue. We'll post some screenshots tomorrow. From "sitemap>Shutter>PSL" we meant in Shutter medm window, we clicked on the PSL close button. As pointed later, it switches C1:AUX-PSL_ShutterRqst while the PSL shutter switch on Lock MC medm screen switches C1:PSL-PSL_ShutterRqst. We were not sure if this was intentional, so we didn't change anything.
allegra had fresh Debian 10 installed on it already. I installed cds-workstation packages (with the help of Erik von Reis). I checked that command line caget, caput etc were working. I'll see if medm and other things are working next time we visit the lab.
Here, I present the same input matrices now normalized row by row to have same norm as current matrices rows. These now I plotted better than last time. Other comments same as 15902. Please let us know what you think.
Thu Mar 18 09:11:10 2021 :
Note: The comparison of butterfly dof in the two cases is bit bogus. The reason is that we know what the butterfly vector is in sensing matrix (N_osems x (N_dof +1)) and that is the last column being (1, -1, 1, -1, 0) corresponding to (UL, UR, LR, LL, Side). However, the matrix we multiply with the OSEM data is the inverse of this matrix (which becomes the input matrix) which has dimensions ((N_dof + 1) x N_osem) and has the last row corresponding to the butterfly dof. This row was not stored for old calculation of the input matrix (which is currently in use) and can not be recovered (mathematically not possible) with the existing 5x4 part of that input matrix. We just added (1, -1, 1, -1, 0) row in the bottom of this matrix (as was done in the matlab codes) but that is wrong and hence the butterfly vector looks so bad for the existing input matrix.
Proposal: We should store the last row of generated input matrix somewhere for such calculations. Ideally, another row in the epics channels for the input matrix would be the best place to store them but I guess that would be too destructive to implement. Other options are to store this 5 number information in wiki or just elogs. For this post, the buttefly row for the generated input matrix is present in the attached files (for future references).
Rana asked us to write out here the new MC3 input matrix we have calculated along with the old one. The new matrix is not working out for us as it can't keep the suspension loops stable.
Note that the new matrix has been made so that the norm of each row is the same as the norm of the corresponding row in the old (current) input matrix.
Peak finding results:
Note: The highest peak on SIDE OSEM sensor free swinging data as well as the DOF basis data created using existing input matrix, comes at 0.978 Hz. Ideally, this should be 1 Hz and in MC1 and MC2, we see the resonance on SIDE DOF to show near 0.99 Hz. If you look closely, there is a small peak present near 1 Hz actually, but it is too small to be the SIDE DOF eigenfrequency. And if it is indeed that, then which of the other 4 peaks is not the DOF we are interested in?
On possiblity is that the POS eigenfrequency which is supposed to be around 0.97 Hz is split off in two peaks due to some sideways vibration and hence these peaks get strongly coupled to SIDE OSEM as well.
P.S. I think something is wrong and out limited experience is not enough to pinpoint it. I can show up more data or plots if required to understand this issue. Let us know what you all think.
A longer measurement is set to trigger at 5:00 tomorrow on April 2nd, 2021. This measurement will run for 35 iterations with an excitation duration of 120s and bandwidth for CSD measurement set to 0.1 Hz. The script is set to trigger in a tmux session named 'cB' on pianosa.
Last run gave similar results as the quick run we did earlier. The code has been unable to strike out couplings with POS. We found the bug which is causing this. This was because the sampling rate of MC_F channel is different from the test-point channels used for PIT and YAW. Even though we were aware of it, we made an error in handling it while calculating CSD. Due to this, CSD calculation with POS data was performed by the code with zero padding which made it think that no PIT/YAW <-> POS coupling exist. Hence our code was only able to fix PIT <-> YAW couplings.
We'll need to do another run with this bug fixed. I'll update this post with details of the new measurement.
To test if our method is working at all, we went for the simpler case of just uncoupling PIT and YAW. This is also because the sensor used for these two degrees of freedom is similar (the MC Trans WFS).
We saw a successful decrease in cross-coupling between PIT and YAW over the first 50 iterations that we tried. Here are some results:
I'm not sure I understand what F2A is? I couldn't find a description of this filter anywhere and don't remember if you have already explained it. Can you describe what is needed to be done again, please? We would keep SUS state space model and seismic transfer functions calculation ready meanwhile.
Next we wanna get the F2A filters made since most of the IMC control happens at f < 3 Hz. Once you have the SUS state space model, you should be able to see how this can be done using only the free'swinging eigenfrequencies. Then you should get the closed loop model including the F2A filters and the damping filters to see what the closed loop behavior is like.
Sorry about that. It must be me. I'll make sure it doesn't happen again. I was careless to not check back, no further explanation.
That makes sense. I assumed that IFO-STATE is configured as you have proposed it to be configured. This could be implemented in later.
a better way would be to configure the EPICS record to automatically set / unset itself based on some diagnostic channels. For example, the "PMC locked" bit should be set if (i) the PMC REFL is < 0.1 AND (ii) PMC TRANS is >0.65 (the exact thresholds are up for debate). Then we are truly recording the state of the IFO and not relying on some script to write to the bit (I haven't thoguht through if there are some edge cases where we need an unreasonable number of diagnostic channels to determine if we are in a certain state or not).
Edit Thu Apr 15 08:32:58 2021 :
Comments are from Rana.
Corrected the plot in the attachment. It shows the correct behavior at high frequencies now.
Here' s aquick update before we leave for lunch. We have managed to calculate some filter that would go on the POS column in MC2 output matrix filter banks aka F2A aka F2P filters. In the afternoon if we can come and work on the IMC, we'll try to load them on the output matrix. We have never done that so it might take some time for us to understand on how to do that. Attached is the bode plot for these proposed filters. Let us know if you have any comments.
Following up from morning's work, I balanced the coils at DC as well. Attachment 1 is screenshot of striptool in which blue and red traces show ASCYAW and ASCPIT outputs when C1:SUS-MC2_LSC_OFFSET was switched by 500 counts. We see very slight disturbance but no real DC offset shown on PIT and YAW due to position step. This data was taken while nominal F2A filter calculated to balance coils at DC was uploaded
I have uploaded the filters on filter banks 7-10 where FM7 is the nominal filter with Q close to 1 and 8-10 are filters with Q 3, 7 and 10 respectively. The transfer function of these filters can be seen in Attachment 2. Note, that the high frequency gain drops a lot when higher Q filters are used.
These filters are designed such that the total DC gain after the application of coil outputs gains for high frequency balancing (as done in morning 16054) balances the coils at DC.
Since I had access to the complete output matrix that balances the coils to less than 1% cross coupling at high frequencies from 16009, I also did a quick test of DC coil balancing with this kind of high frequency balancing. In this case, I uploaded another set of filters which were made at Q close to 1 and gain such that effective DC gain matrix becomes what I got by balancing in the above case. This set of filter also worked as good as the above filters. This completes the proof that we can also use complete matrix for high frequency coil balancing which can be calculated by a script in 20min and works with DC coil balancing as well. In my opinion, this method is more clear and much faster than toggling values in coil output gains where we have only 4 values to optimize 6 cross-coupling parameters. But don't worry, I'm not wasting time on this and will abandon this effort for now, to be taken up in future.
Corrected the calculation of filters in case of Q different than . There was a bug in the code which I overlooked. I'll correct the filter bank modules tomorrow.
Edit Wed Apr 21 11:06:42 2021 :
I have uploaded the corrected foton filters. Please see attachment 3 for the transfer functions calculated by foton. They match the filters we intended to upload. Only after uploading and closing the foton filter, I realized that the X=7 filter plot (bottom left in attachment 3) does not have dB units on y-axis. It is plotted in linear y-scale (this plot in foton is for phase by default to I guess I forgot to change the scaling when repurposing it for my plot).
Yes, during the AC balancing, POS column was set to all 1. This table shows the final values after all the steps. The first 3 columns are DC balancing results when output matrix was changed. While the last column is for AC balancing. During AC balancing, the output matrix was kept to ideal position as you suggested.
the POS column should be all 1 for the AC balancing. Where did those non-1 numbers come from?
Koji mentioned that the mode of the laser is different for lower diode currents. So that might be the reason why we got less transmission at the low input power but more afterward.
The mix up was my fault I think. I restored the channels manually instead of using burt restore. Your message suggests that we can set burt to start noticing channel changes at home point and create a .req file that can be used to restore later. We'll try to learn how to do that. Right now, we only know how to burt restore using the existing snapshots from the autoburt directory, but they touch more things than we work on, I think. Or can we just always burt restore it to morning time? If yes, what snapshot files should we use?
I have uploaded all the new settings mentioned in 16066 and 16072. The settings were uploaded through a single script present at anchal/20210428_IMC_Tuned_Suspension/uploadNewConfigIMC.py. The settings can be reverted back to old settings through anchal/20210428_IMC_Tuned_Suspension/restoreOldConfigIMC.py. Both these scripts can be run only through python3 in donatella or allegra.
GPSTIME of new settings: 1303690144
New settings include:
We'll wait and watch the performance through summary pages and check back the performance on Monday.
In 16087 we mentioned that we were unable to do a step response test for WFS loop to get an estimate of their UGF. The primary issue there was that we were not putting the step at the right place. It should go into the actuator directly, in this case, on C1:SUS-MC2_PIT_COMM and C1:SUS-MC2_YAW_COMM. These channels directly set an offset in the control loop and we can see how the error signals first jump up and then decay back to zero. The 'half-time' of this decay would be the inverse of the estimated UGF of the loop. For this test, the overall WFS loops gain, C1:IOO-WFS_GAIN was set to full value 1. This test is performed in the changed settings uploaded in 16091.
I did this test twice, once giving a step in PIT and once in YAW.
Attachment 1 is the striptool screenshot for when PIT was given a step up and then step down by 0.01.
Attachment 2 is the striptool screenshot when YAW was given a step up and down by 0.01. Note the difference in x-scale in this plot.
Tried locking the arms
Did the WFS step response test on IMC in between while waiting for help. See 16094.
Back to trying arm locking
PMC got unlocked
I double checked today and the F2A filters in the output matrices of MC1, MC2 and MC3 in the POS column are ON. I do not get what SDF means? Did we need to add these filters elsewhere?
The IMC suspension team should double check their filters are on again. I am not familiar with the settings and I don't think they've been added to the SDF.
t Both arms were locked simply by using IFO > Configure > ! (YARM) > Restore YARM. I had to use ASS to improve the TRX/TRY to ~0.95.
I measured C1:LSC-XARM_IN1_DQ and C1:LSC-YARM_IN1_DQ while injecting band limited noise in C1:IOO-WFS1_PIT_EXC using uniform noise with amplitude 1000 along with filter defined by string: cheby1("BandPass",4,1,80,100). I calibrated the control arms signals by 2.44 nm/cts calibration factor directly picked up from 13984.
For the duration of this test, all LIMIT switches in the WFS loops were switched OFF.
I do not see any affect on the arm control signal power spectrums with or without the noise injection. Attachement 1 shows the PSD along with PSD of the injection site IN2 signal. I must be doing something wrong, so would like feedback before I go further.
With the input matrix, coil ouput gains and F2A filters loaded as in 16091, I tested the suspension loops' step response to offsets in LSC, ASCPIT and ASCYAW channels, before and after applying the "new damping gains" mentioned in 16066 and 16072. If these look better, we should upload the new (higher) damping gains as well. This was not done in 16091.
Note that in the plots, I have added offsets in the different channels to plot them together, hence the units are "au".
We repeated the same test with IMC unlocked. We had found these gains when IMC was unlocked and their characterization needs to be done with no light in the cavity. attached are the results. Everything else is same as before.
Edit Tue May 4 14:43:48 2021 :
Overall, I would recommend setting the new gains in the suspension loops as well to observe long term effects too.
For past few days, a weird sound of decaying gas leakage comes in the 40m control room from the south west corner of ceiling. Attached is an audio capture. This comes about every 10 min or so.
We have uploaded the new damping gains on all the suspensions of IMC. This completes changing all the configuration to as mentioned in 16066 and 16072. The old setting can be restored by running python3 /users/anchal/20210505_IMC_Tuned_SUS_with_Gains/restoreOldConfigIMC.py from allegra or donatella.
Here's my first attempt at doing angular actuation calibration for IMC mirrors using the method descibed in /users/OLD/kakeru/oplev_calibration/oplev.pdf by Kakeru Takahashi. The key is to see how much is the cavity mode misaligned from the input mode of beam as the mirrors are moved along PIT or YAW.
There two possible kinds of mismatch:
Kakeru's document goes through cases for linear cavities. For IMC, the mode mismatches are bit different. Here's my take on them:
Calibration factor at DC [µrad/cts]
Koji and I did a few tests with an OSEM emulator on the satellite amplifier box used for MC1 which is housed on 1X4. This sat box unit is S2100029 D1002812 that was recently characterized by me 15803. We found that the differential output driver chip AD8672ARZ U2A section for the UL PD was not working properly and had a fluctuating offset at no input current from the PD. This was the cause of the ordeal of the morning. The chip was replaced with a new one from our stock. The preliminary test with the OSEM emulator showed that the channel has the correct DC value.
In further testing of the board, we found that the channel 8 LED driver was not working properly. Although this channel is never used in our current cable convention, it might be used later in the future. In the quest of debugging the issue there, we replaced AD8672ARZ at U1 on channel 8. This did not solve the issue. So we opened the front panel and as we flipped the board, we found that the solder blob shorted the legs of the transistor Q1 2N3904. This was replaced and the test with the LED out and GND shorted indicated that the channel is now properly providing a constant current of 35mA (5V at the monitor out).
After the debugging, the UL channel became the least noisy among the OSEM channels! Mode cleaner was able to lock and maintain it.
We should redo the MC1 input matrix optimization and the coil balancing afterward as we did everything based on the noisy UL OSEM values.
For future reference, the new settings can be upoaded from a script in the same directory. Run python /users/anchal/20210505_IMC_Tuned_SUS_with_Gains/uploadNewConfigIMC.py from allegra.
There isn't any instruction here on how to upload the new settings
I attempted a single arm actuation calibration using IR beatnote (in the directions of soCal idea for DARM calibration)
I fixed the PSL shutter button on Shutters summary page C1IOO_Mech_Shutter.adl. Now PSL switch changes C1:PSL-PSL_ShutterRqst channel. Earlier it was C1:AUX-PSL_ShutterRqst which doesn't do anything.
Attachment 1 shows the closed loop of Xend Green laser Arm PDH lock loop. Free running laser noise gets injected at laser head after the PZT actuation as . The PDH error signal at output of miser is fed to a gain 1 SR560 used as summing junction here. Used in 'A-B mode', the B port is used for sending in excitation where .
We have access to three ports for measurement, marked at output of mixer, at output of SR560, and at PZT out monitor port in uPDH box. From loop algebra, we get following:
, where is the open loop transfer function of the loop.
So measurement of can be done in following two ways (not a complete set):
Working in Xend with mask on has become unbearable. It is very hot there and I would really like if we fix this issue.
Today, the Xend Green laser was just unable to hold lock for longer than 10's of seconds. The longest I could see it hold lock was for about 2 minutes. I couldn't find anything obviously wrong with it. Attached are noise spectrums of error and control points. The control point spectrum shows good matching with typical free running laser noise.
Are the few peaks above 10 kHz in error point spectrum worrysome? I need to think more about it in a cooler place to make sure.
I wanted to take a high frequency spectrum of error point to make sure that higher harmonics of 250 kHz modulation frequency are not leaking into the PDH box after demodulation. However, the lock could not be maintained long enough to take this final measurement. I'll try again tomorrow morning. It is generally cooler in the mornings.
This post is just an update on what's happening. I need to work more to get some meaningful inferences about this loop.
I propose we set up a temperature sensor network as described in attachment 1.
Here there are two types of units:
These sensors can be configured over network by going to their assigned IP addresses. I'm not sure at the moment how to configure the dB files to get them to write on slow EPICS channels.
We will have an unused port on the BASE-GATEWAY (#B) which can be used to run another temperature sensor, maybe at an important rack, PSL table or somewhere else.
In future, if more sensors are required, there are expansion (network switch like) options for BASE-GATEWAY or daisy-chain options for the probes.
Edit Fri Jun 18 16:28:13 2021 :
See this [wiki page](https://wiki-40m.ligo.caltech.edu/Physical_Environment_Monitoring/Thermometers) for updated plan and final quote.
MC was unable to acquire lock because the WFS offsets were cleared to zero at some point and because of that MC was very misaligned to be able to catch back lock. In such cases, one wants the WFS to start accumulating offsets as soon as minimal lock is attained so that the mode cleaner can be automatically aligned. So I did following that worked:
When both arms were locked, we found that ITMY optical lever was very off-center. This seems to have happened after the c1susaux rebooting we did in June 17th. I opened the ITMY table and realigned the OPLev beam to the center when the arm was locked. I repeated this process for BS, PRM and ETMY. I did PRM because I've known that we have been keeping its OpLev off. The reason was clear once I opened the table. The oplev reflection beam was hitting the PD box instead of the PD. After correcting, I was able to swithc on PRM opLev loops and saw normal functioning.
I worked in Yend station, trying to get the ETMY QPD to work properly. When I started, only one (quadrant #3) of the 4 quadrants were seeing any lights. By just changing the beam splitter that reflects some light off to the QPD, I was able to get some amount of light in quadrant #2. However, no amount of steering would show any light in any other quadrants.
The only reason I could think of is that the incoming beam gets partially clipped as it seems to be hitting the beam splitter near the top edge. So for this to work properly, a mirror upstream needs to be adjusted which would change the alignment of TRX photodiode. Without the light on TRX photodiode, there is no lock and there is no light. So one can't steer this beam without lossing lock.
I tried one trick, in which, I changed the YARM lock trigger to POY DC signal. I got it to work to get the lock going even when TRY was covered by a beam finder card. However, this lock was still bit finicky and would loose lock very frequently. It didn't seem worth it to potentially break the YARM locking system for ETMY QPD before running this by anyone and this late in evening. So I reset everything to how it was (except the beam splitter that reflects light to EMTY QPD. That now has equal ligth falling on quadrant #2 and #3.
The settings I temporarily changed were:
All these were reverted back to there previous values manually at the end.
This was a mistake. When arms are locked, PRM is misaligned by setting -800 offset in PIT dof of PRM. The oplev is set to function in normal state not this misalgined configuration. I undid my changes today by switching off the offset, realigning the oplev to center and then restoring the single arm locked state. The PRM OpLevs loops are off now.
PRM because I've known that we have been keeping its OpLev off. The reason was clear once I opened the table. The oplev reflection beam was hitting the PD box instead of the PD. After correcting, I was able to swithc on PRM opLev loops and saw normal functioning.
I did this analysis again by just doing demodulation go 5s time segments of the 60s excitation signal. The major difference is that I was not summing up the sine-cosine multiplied signals, so the error associated was a lot more. If I simply multpy the whole beatnote signal with digital LO created at excitation frequency, divide it up in 12 segments of 5 s each, sum them up individually, then take the mean and standard deviation, I get the answer as:
as opposed to that was calculated using MICH signal earlier by gautum in 13984.
Attachment 1 shows the scatter plot for the complex calibration factors found for the 12 segments.
My aim in the previous post was however to get a time series of the complex calibration factor from which I can take a noise spectral density measurement of the calibration. I'll still look into how I can do that. I'll have to add a low pass filter to integrate the signal. Then the noise spectrum up to the low pass pole frequency would be available. But what would this noise spectrum really mean? I still have to think a bit about it. I'll put another post soon.
We attempted to simulate "oscillator based realtime calibration noise monitoring" in offline analysis with python. This helped us in finding about a factor of sqrt(2) that we were missing earlier in 16171. we measured C1:ALS-BEATX_FINE_PHASE_OUT_HZ_DQ when X-ARM was locked to main laser and Xend green laser was locked to XARM. An excitation signal of amplitude 600 was setn at 619 hz at C1:ITMX_LSC_EXC.
We got a value of . The calibration factor in use is from nm/cts from 13984.
Next steps could be to budget this noise while we setup some way of having this calibration factor generated in realitime using oscillators on a FE model. Calibrating actuation of a single optic in a single arm is easy, so this is a good test setup for getting a noise budget of this calibration method.
[ian, anchal, paco]
After our second attempt of locking PRFPMI tonight, we tried to resotre XARM and YARM locks to IR by clicking on IFO_CONFIGURE>Restore XARM (POX) and IFO_CONFIGURE>Restore YARM (POY) but the arms did not lock. The green lasers were locked to the arms at maximum power, so the relative alignments of each cavity was ok. We were also able to lock PRMI using IFO_CONFIGURE>Restore PRMI carrier.
This was very weird to us. We were pretty sure that the aligment is correct, so we decided to cehck the POX POY signal chain. There was essentially no signal coming at POX11 and there was a -100 offset on it. We could see some PDH signal on POY11 but not enough to catch the locks.
We tried running IFO_CONFIGURE>LSC OFFSETS to cancel out any dark current DC offsets. The changes made by the script are shown in attachment 1.
We went to check the tables and found no light visible on beam finder cards on POX11 or POY11. We found that ITMX was stuck on one of the coils. We unstuck it using the shaking method. The OPLEVs on ITMX after this could not be switched on as the OPLEV servo were railing to limits. But when we ran Restore XARM (POX) again, they started working fine. Something is done by this script that we are not aware of.
We're stopping here. We still can not lock any of the single arms.
Wed Jul 28 11:19:00 2021 Update:
Gautam found that the restoring of POX/POY failed to restore the whitening filter gains in POX11 / POY11. These are meant to be restored to 30 dB and 18 dB for POX11 and POY11 respectively but were set to 0 dB in detriment of any POX/POY triggering/locking. The reason these are lowered is to avoid saturating the speakers during lock acquisition. Yesterday, burt-restore didn't work because we restored the c1lscepics.snap but said gains are actually in c1lscaux.snap. After manually restoring the POX11 and POY11 whitening filter gains, gautam ran the LSCOffsets script. The XARM and YARM were able to quickly lock after we restored these settings.
The root of our issue may be that we didn't run the CARM & DARM watch script (which can be accessed from the ALS/Watch Scripts in medm). Gautam added a line on the Transition_IR_ALS.py script to run the watch script instead.
I redid the measurement of Schnupp asymmetry today and found it to be 3.8 cm 0.9 cm.
I measured a phase difference of 5 1 degrees between the two paths.
The uncertainty in this measurement is much more than gautam's 15956 measurement. I'm not sure yet why, but would look into it.
I used the Valera technique to measure the Schnupp asymmetry to be , see Attachment #1. The data points are points, and the zero crossing is estimated using a linear fit. I repeated the measurement 3 times for each arm to see if I get consistent results - seems like I do. Subtle effects like possible differential detuning of each arm cavity (since the measurement is done one arm at a time) are not included in the error analysis, but I think it's not controversial to say that our Schnupp asymmetry has not changed by a huge amount from past measurements. Jamie set a pretty high bar with his plot which I've tried to live up to.
Late elog. Original time 08/02/2021 21:00.
I locked both arms and ran ASS to reach to optimum alignment. ETMY PIT > 10urad, ITMX P > 10urad and ETMX P < -10urad. Everything else was ok absolute value less than 10urad. I recentered these three.
Than I locked PRMI, ran ASS on PRCL and MICH and checked BS and PRM alignment. They were also less than absolute value 10urad.
I've added the other two temperature sensor modules on Y end (on 1Y4, IP: 192.168.113.241) and in the vertex on (1X2, IP: 192.168.113.242). I've updated the martian host table accordingly. From inside martian network, one can go to the browser and go to the IP address to see the temperature sensor status . These sensors can be set to trigger alarm and send emails/sms etc if temperature goes out of a defined range.
I feel something is off though. The vertex sensor shows temperature of ~28 degrees C, Xend says 20 degrees C and Yend says 26 degrees C. I believe these sensors might need calibration.
Remaining tasks are following:
We made a script under scripts/PEM/temp_logger.py and ran it on megatron. The script uses the requests package to query the latest sensor data from the three sensors every 10 minutes as a json file and outputs accordingly. This is not a permanent solution.
c1teststand subnetwork is now accessible remotely. To log into this network, one needs to do following:
Just to document the IT work I did, doing this connection was bit non-trivial than usual.
I had to add following two lines in the /etc/network/interface file to make the special ip routes persistent even after reboot:
post-up ip route add 192.168.113.200 via 10.0.1.1 dev eno1
post-up ip route add 192.168.113.216 via 10.0.1.1 dev eno1
Late elog: Original time of work Tue Aug 17 20:30 2021
I locked the arms yesterday remotely and tried running runASS.py scripts (generally ran by clicking Run ASS buttons on IFO OVERVIEW screen of ASC screen). We have known for few weeks that this script stopped working for some reason. It would start the dithering and would optimize the alignment but then would fail to freeze the state and save the alignment.
I found the caget('C1:LSC-TRX_OUT') or caget('C1:LSC-TRY_OUT') were not working in any of the workstations. This is weird since caget was able to acquire these fast channel values earlier and we have seen this script to work for about a month without any issue.
Anyways, to fix this, I just changed the channel name to 'C1:LSC-TRY_OUT16' when the script checks in the end if the arm has indeed been aligned. It was only this step that was failing. Now the script is working fine and I tested them on both arms. On the Y arm, I misaligned the arm by adding bias in yaw by changing C1:SUS-ITMY_YAW_OFFSET from -8 to 22. The script was able to align the arm back.
I tried to read a bit and understand the NTP synchronization implementation in FE computers. I'm quite sure that NTP synchronization should be 'yes' if timesyncd are running correctly in the output of timedatectl in these computers. As Koji reported in 15791, this is not the case. I logged into c1lsc, c1sus and c1ioo and saw that RTC has drifted from the software clocks too which does not happen if NTP synchronization was active. This would mean that almost certainly, if the computers are rebooted, the synchronization will be lost and the models will fail to come online.
My current findings are the following (this should be documented in wiki once we setup everything):
The solution to this issue could be as simple as just defining ntpserver in the name server list. But I'm not sure if my understanding of this issue is correct. Comments/suggestions are welcome for future steps.
I added ntpserver as a known host name for address 192.168.113.201 (fb1's address where ntp server is running) in the martian host list in the following files in Chiara:
Note: a host name called ntp was already defined at 192.168.113.11 but I don't know what computer this is.
Then, I restarted the DNS on chiara by doing:
sudo service bind9 restart
Then I logged into c1lsc and c1ioo and ran following:
controls@c1ioo:~ 0$ sudo systemctl restart systemd-timesyncd.service
controls@c1ioo:~ 0$ sudo systemctl status systemd-timesyncd.service -l
● systemd-timesyncd.service - Network Time Synchronization
Loaded: loaded (/lib/systemd/system/systemd-timesyncd.service; enabled)
Active: active (running) since Fri 2021-08-20 07:24:03 UTC; 53s ago
Main PID: 23965 (systemd-timesyn)
Aug 20 07:24:03 c1ioo systemd: Starting Network Time Synchronization...
Aug 20 07:24:03 c1ioo systemd: Started Network Time Synchronization.
Aug 20 07:24:03 c1ioo systemd-timesyncd: Using NTP server 192.168.113.201:123 (ntpserver).
Aug 20 07:24:35 c1ioo systemd-timesyncd: Using NTP server 192.168.113.201:123 (ntpserver).
controls@c1ioo:~ 0$ timedatectl
Local time: Fri 2021-08-20 07:25:28 UTC
Universal time: Fri 2021-08-20 07:25:28 UTC
RTC time: Fri 2021-08-20 07:25:31
Time zone: Etc/UTC (UTC, +0000)
NTP enabled: yes
NTP synchronized: no
RTC in local TZ: no
DST active: n/a
The same output is shown in c1lsc too. The NTP synchronized flag in output of timedatectl command did not change to yes and the RTC is still 3 seconds ahead of the local clock.
Then I went to c1sus to see what was the status output before rstarting the timesyncd service. I got folloing output:
controls@c1sus:~ 0$ sudo systemctl status systemd-timesyncd.service -l
● systemd-timesyncd.service - Network Time Synchronization
Loaded: loaded (/lib/systemd/system/systemd-timesyncd.service; enabled)
Active: active (running) since Tue 2021-08-17 04:38:03 UTC; 3 days ago
Main PID: 243 (systemd-timesyn)
Aug 20 02:02:18 c1sus systemd-timesyncd: Using NTP server 192.168.113.201:123 (ntpserver).
Aug 20 02:36:27 c1sus systemd-timesyncd: Using NTP server 192.168.113.201:123 (ntpserver).
Aug 20 03:10:35 c1sus systemd-timesyncd: Using NTP server 192.168.113.201:123 (ntpserver).
Aug 20 03:44:43 c1sus systemd-timesyncd: Using NTP server 192.168.113.201:123 (ntpserver).
Aug 20 04:18:51 c1sus systemd-timesyncd: Using NTP server 192.168.113.201:123 (ntpserver).
Aug 20 04:53:00 c1sus systemd-timesyncd: Using NTP server 192.168.113.201:123 (ntpserver).
Aug 20 05:27:08 c1sus systemd-timesyncd: Using NTP server 192.168.113.201:123 (ntpserver).
Aug 20 06:01:16 c1sus systemd-timesyncd: Using NTP server 192.168.113.201:123 (ntpserver).
Aug 20 06:35:24 c1sus systemd-timesyncd: Using NTP server 192.168.113.201:123 (ntpserver).
Aug 20 07:09:33 c1sus systemd-timesyncd: Using NTP server 192.168.113.201:123 (ntpserver).
This actually shows that the service was able to find ntpserver correctly at 192.168.113.201 even before I changed the name server file in chiara. So I'm retracting the changes made to name server. They are probably not required.
The configuration files for timesynd.conf are read only even with sudo. I tried changing permissions but that did not work either. Maybe these files are not correctly configured. The man page of timesyncd says to use field 'NTP' to give the ntp servers. Our files are using field 'Servers'. But since we are not getting any error message, I don't think this is the issue here.
I'll look more into this problem.