I was checking on the slow machine channels and found something I could not understand.
On the IOO WFS HEAD screen, there are two sets of 4 switches (magenta rectangles in Attachment 1) labeled 2/4/8/16dB.
But as far as I could confirm with the WFS demod (D980233) and WFS head (D980012) drawings, they are the gain (attenuation) switches for the individual segments.
Their epics variable names are "C1:IOO-WFS1_SEG1_ATTEN", "C1:IOO-WFS1_SEG2_ATTEN", etc...
"C1:IOO-WFS1_SEG1_ATTEN", "C1:IOO-WFS1_SEG2_ATTEN", etc...
I confirmed the switches are alive (effective), and they are not all ON or OFF. I wonder what is the real situation there...
The unfortunate discovery today was that the attenuator switches on the IMC WFS heads are actually assigned to individual segments, and they are active. That means that we have been running the WFS with an uneven gain setting. The attached PDFs show that the signals with the attenuators on and off all at the same time, while the WFS servo output was frozen. A more annoying feature is that when some of the attenuators are on, this does not lower the gain completely. I mean that the attenuated channels show some reduction of the gain, but that is not the level of reduction we see when all attenuators are turned on. This RF could come from some internal RF coupling or some similar effect.
Moreover, the demodulation phases are quite off for most of the segments.
So far, the WFS is running with this uneven attenuation. We take time to characterize the gain and retune the demod phases and input matrices.
its painful, but you and I should probably take these out, bypass the switches and use them with fixed gain; the 'Reed Relay' attenuators are not a good part for this app.
The historical problem is that they tend to self oscillate with full gain because they had 2 MAX4106 in series which couple to each other in the bad way --- need to remove one of them and set the gain of the other one to 10.
The unfortunate discovery today was that the attenuator switches on the IMC WFS heads are actually assigned to individual segments, and they are active. That means that we have been running the WFS with an uneven gain setting.
The WFS servo loop will come on 5 seconds after the MC is locked.
I have uncommented the lines in the mcup script which turn on the WFS servos. But I shifted their location to the part after the MC is locked.
The WFS error signals were recorded in the order
these measurements are made in the linear region, that is the MC is nearly perfectly aligned.
This is the average and std. dev.of 5 measurements taken of the same signals over 10 secs each. The std. dev are under 10%. And hence, I will be using 10 secs for measurements for the WFS signals after perturbations to the mirrors.
I perturbed the Pitch and Yaw of the Three mirrors (in order MC1,2,3), using ezcastep and calculated the coefficients that relate these perturbations to the WFS error signals.
The perturbation made is of -0.01 in each dof , and after measuring the WFS error for it, the system is reverted back to the previous point before making the other perturbation.
I was able to calculate the coefficients since I have assumed a linear relationship..
Following are the coefficients calculated using 10 secs measurements
MC1_P MC1_Y MC2_P MC2_Y MC3_P MC3_Y constant
WFS1_PIT -0.1262 0.3677 -0.4539 -0.6297 -0.1889 -0.1356 0.013664
WFS1_YAW -0.0112 -0.7415 -0.1844 2.4509 -0.0023 -0.3531 -0.016199
WFS2_PIT 0.1251 0.4824 -0.2028 -0.6188 0.0099 -0.1490 0.006890
WFS2_YAW 0.0120 -0.7957 -0.1793 0.9962 -0.0493 0.2672 -0.013695
Also, I measured the same thing for 100s, and to my surprize, even the signs of coeficients are different.
MC1_P MC1_Y MC2_P MC2_Y MC3_P MC3_Y constant
WFS1_PIT -0.1981 0.3065 -0.6084 -0.9349 -0.4002 -0.3538 0.009796
WFS1_YAW 0.0607 -0.6977 0.0592 2.8753 0.3507 0.0373 -0.008194
WFS2_PIT 0.0690 0.4769 -0.2859 -0.7821 -0.1115 -0.2953 0.004150
WFS2_YAW 0.0580 -0.8153 -0.0937 1.1424 0.0650 0.4203 -0.010629
The reason I can understand is that the measurements were not made at the same time, and hence conditions might have changed.
A thing to note in all these coefficients is that they relate the error signals to the 'perturbation' around a certain point (given below). That point is assumed to lie in the linear region.
I just found the singular values and the condition number of the 4*4 matrix relating the WFS error signals and the MC1 and MC2 movements.
Since the MC spots are good, I put the beam back on WFS 1 and WFS 2.
Also, I changed the indicators on the LockMC screen to reflect the change in elog 7289, where we added another on/off switch for the WFS so that the ASS could be on, but the WFS off. For the last month, the WFS could be disabled, but the MC screen's indicators would suggest that we were pushing very significantly on all 3 MC mirrors. Now the MC screen reflects reality a little better.
I also uncommented the WFS lines in the mcup script. Den had commented them out, but didn't elog about it! C'mon Den, please elog stuff!!!! (He confessed out loud the other day, but it still wasn't in the elog).
I'm leaving the WFS loops disabled (even though the MC autolocker tries to turn them on, I have them manually disabled using the extra on/off switch) since they're unstable. I'm in the process of figuring out what's wrong. So far, the WFS improve the MC alignment for a minute or two, and then they totally misalign the MC. This is a work in progress.
We found that the MC REFL image was no longer round and that the MCWFS DC quadrant spots were mostly
in one quadrant. So we re-centered the MCWFS beams in the following way:
1) We unlocked the MZ and adjusted the PZT voltage to keep the beam on the WFS from saturating.
2) Re-aligned the black hole beam dump to center its beam in its aperture.
3) centered the beam on the MCWFS optics and MCWFS QPD displays.
4) Relocked MC.
Below is the image of the IOO Strip tool. You can see that the MC REFL DC is now more flat. The
MC pointing has also been changed (see the MC TRANS HOR & VERT channels). The MC transmitted
light is also now more stable and higher.
We tried to center the QPD, and we found that there were a few hundred mV of dark offset for each
quadrant of QPD. We adjusted them with this scripts:
We found the MC reflection was distorted . And WFC beam went to upward of QPD
We recentered WFC beam and these problems were fixed
in the lab, checkin on the WFS
Sun Jul 5 18:25:50 2020
I redid Gautam's measurements to get a baseline before changing the head, and my results are very different: To me it looks like the WFS2 quadrants are all OK.
I've left the setup as is in case either me or Gautam want to double check. If we're agreed on this response, I'll remove the notches and disable the RF attenuators.
Sun Jul 5 21:42:45 2020
I set up to do the WFS head modifications today, but I was shot down in flames due to a missing AC/DC adapter.
The Prologix GPIB-ethernet dongle needs +8-13 V to run. Some riff raff has removed the adapter and I was thunderstruck to see that it had not been returned.
I did the usual hunt around the lab looking for something with the right specs and connector. I found one that could do +9V and had the right connector, but it didn't light up the adapter so I put it back in black SP table.
I'll order a couple of these (5 ordered for delivery on Wednesday) in case there's a hot demand for the jack / plug combo that this one has. The setup is in the walkway, but I returned the AS table to the usual state and made sure the IMC is locking well.
Clearly this "riff raff" is referring to me. It won't help today I guess but there is one each on the carts holding the SR785 (currently both in the office/electronics bench area), and the only other unit available in the lab is connected to a Prologix box on the Marconi inside the PSL enclosure.
After some hunting, I found this old SURF report with the WFS head measurements. The y-axes don't make much sense to me, and I can't find the actual data anywhere (her wiki page doesn't actually exist). So I think it's still unknown if these heads ever had the advertised transimpedance gain, or if the measured transimpedance of ~1kohm was what it always was.
Trying to figure out what's wrong with the MC WFS:
1) The symptom seems to be that the control signals become very large in the pitch and then the loop breaks when they saturate. Usually this is due to a degenerate matrix or improper inversion. Most likely some of the BURT restore is bad or the analog gain for one of the WFS has been switched when Jamie was doing the "Guardian" debugging.
2) In checking this out, I found that several buttons on the WFS screens were not working (and apparently have never been working). Please try to test things in the future...The filter bank buttons in C1IOO_MC_TRANS_QPD were using relative path names; fixed these to use abs path names. The buttons in the WFS_MASTER for the IOO_PIT banks were using IOO_PITCH instead...
2.5) Recentered beams on WFS heads with MC alignment good and MC unlocked.
3) Main problem in the WFS still not found - disabling this in the autolocker.
Tried a bunch of stuff, but eventually just turned off the TRANS_QPD loops and loops are stable. Needs more debugging.
The low UGFs of the MC WFS servos made the MC insane thesedays:
The servos are too slow and we kept having significant misalignment left uncompensated.
I increased the total gain of the MC WFS from 0.01 to 0.4 (x40) to make the UGFs of the
WFS paths to ~2Hz. This was too much gain for the QPD path so the gains for the QPD paths
were reduced by a factor of 4 (x10 in total).
The script mcwfsup was also modified accordingly.
Tried a bunch of stuff, but eventually just turned off the TRANS_QPD loops and loops are stable. Needs more debugging.
Back around June 18, Jamie was debugging some Guardian code here to replace our MC autolocker. Afterwards our MC WFS stopped working. We never figured out what went wrong, but at the time we turned off the feedback from the MC trans QPD and it stabilized the response at DC.
Today, I noticed that the trans QPD feedback is on. Did anyone do this on purpose?
Its problem causing behavior is slow, but you can catch it if you wait. With the nominal WFS gain of 0.4 the control signal ramps up monotonically at a rate of ~100 counts/minute. Depending upon the static alignment of the MC, this could let it take 10 minutes or a few hours before it rails the MC SUS actuators and breaks the lock. Very sneaky. Don't turn this loop back on without making sure its working and not breaking. I would trend it for you, but the SLOW channels associated with the TRANS QPD servo are not trended --- does anyone know how to get them in the channel list?
I wanted to make sure that the QPD map on the C1IOO_MC_TRANS_QPD.adl screen corresponded to the actual physical quadrants on the photodiode at the MC2 table. We turned MC_WFS_OUT OFF before fiddling around with a red laser pointer to try and map the quadrants.
I initially verified the correspondence between the various quadrants and the text-fields displaying the outputs using PV_Info. I found that there was good agreement in this respect. So for instance, field adjacent to the quadrant marked "1" on the C1IOO_MC_TRANS_QPD.adl screen had the following input channel: IOO_MC_TRANS_SEG1_INMON. The filter banks were empty and there was just an overall gain on -1 on all four channels. The channels leading to the filter-banks were the 'right' ones: quadrant 1 for the top bank, then quadrants 2,3 and 4 down.
Next, a red laser pointer was used to map the quadrants. Here, there was some disagreement between the physical quadrants and the map on the C1IOO_MC_TRANS_QPD.adl screen, which is summarised in the attached image-the whole thing is sort of rotated 180degrees about the centre.
The interpretation of the figure is as follows:
quadrant 1 on screen QPD=bottom right quadrant on QPD
quadrant 2 on screen QPD=top right quadrant on QPD
quadrant 3 on screen QPD=top leftt quadrant on QPD
quadrant 4 on screen QPD=bottom left quadrant on QPD
MC_WFS_OUT was turned back ON.
I just sat down in the control room, and discovered the PMC (and everything else) unlocked. I relocked the PMC, but the MC wasn't coming back. After a moment of looking around, I discovered that the WFS were on, and railing. I ran the "turn WFS off" script, and the MC came back right away, and the WFS came on as they should.
We need to relook at the WFS script, or the MC down script, to make sure that any time the MC is unlocked, no matter why it unlocked, the WFS output is off and the filter histories are cleared.
The only script that can currently take this action is the MC autolocker. If that is disabled first and the PMC unlocks later, the WFS will not be turned off. During the last round of discussions we had about the autolocker script, sometime last Nov, we decided that too much automation is not desirable and that the autolocker should be kept as simple as possible.
I found that the MC WFS had large offset control signals going to the MC SUS. Even though the input switch was off, the integrators were holding the offset.
I have disabled the ASCPIT outputs in the MC SUS. Suresh is going to fix the MC autolocker script to gracefully handle the OFF and ON and then test the script before resuming the WFS testing.
MCL data for OAF may be suspect from this morning.
While going to take some transfer functions of the MC WFS loop, LSC was down. When we tried to restart the FE using 'rtcds restart --all', c1lsc crashed and froze. We manually reset c1lsc, then laboriously determined the correct order of machines to reboot. Here's what works best:
rtcds start c1x04 c1lsc c1ass c1oaf c1cal c1daf
Starting c1dnn crashes the other FE
rtcds restart --all
rtcds restart c1rfm c1sus c1mcs
restarting c1pem crashes the other FE on c1sus
We're seeing a lot of red IPC indicators--perhaps it's an issue with the order we're restarting?
via Polish chat, GV tells us to RTFE
As suggested, I ran the script cds/rebootC1LSC.sh
I got a timeout error when the script tried closing the PSL shutter ('C1:AUX-PSL_ShutterRqst' not found), but Rana and I closed the shutter before leaving last night. c1sus is down, so the script found no route to host c1sus; I'm thinking I need to reset c1sus for the script to run completely. Nonetheless, c1lsc was rebooted, which crashed c1ioo and left the c1lsc FE all red (probably because c1sus wasn't restarted).
I reset c1lsc, c1sus, and c1ioo.
I noticed that the script gives the command 'ssh c1XXX', but we have been getting no route to host using this command. Instead, the machines are currently only reachable as c1XXX.martian. I'm not sure why this is, so I just appended .martian in rebootC1LSC.sh
This time, the script does run. I did get 'no route to host' on c1ioo, so I think I need to reset that machine again. After reset, the script failed to login to c1ioo and c1lsc.
Fri Sep 6 13:09:05 2019
After lunch, I reset the computers again, and try the script again. There is again no route to host for c1ioo. I'm going inside to shutoff the power to c1ioo, since the reset buttom seems to not be working. I still can't login from nodus, so I'm bringing a keyboard and monitor over to plug in directly.
On reset, c1ioo repeatedly reaches the screen in attachment 1, before going black. Holding down shift or ctrl+alt+f1 doesn't get me a command prompt. After waiting/searching the elog for >>3 min, we decided to follow these instructions to cycle the power of c1ioo. The same problem recurred following power up. I found online some instructions that the SunSystems 4600 can hang during reboot if it has become too hot ("reboot during a thermal shutdown"); I did notice that the temperature light was on earlier in this procedure, so perhaps that is the problem. I followed the wiki instructions to shut down the computer again (pressed power button, unplugged 4 power supplies from back of machine), and left it unplugged for 10-30 min (Fri Sep 6 14:46:18 2019 ).
Fri Sep 6 15:03:31 2019
Rana plugged in the power supplies and reset the machine again.
Fri Sep 6 16:30:37 2019
c1ioo is still unreachable! I pressed reset once, and the reset button flashes white. The yellow warning light is still on.
Fri Sep 6 16:54:21 2019
The reset light has stopped flashing, but I still can't access c1ioo. I reset once more, this time watching c1ioo on a monitor directly. I'm still seeing the same boot screen repeatedly. I do see that CPU0 is not clocking, which seems weird.
Following gautam's elog here, I found the Sun Fire X4600 manual for locating faulty CPUs. After the white reset light stopped flashing, I held down the power button to turn off the system. Before shutdown, all of the CPU displayed amber lights; after shutdown, only the leftmost CPU (as viewed from the back, presumably CPU0) displays an amber light. The manual says this is evidence that the CPU or DIMM is faulty. Following the manual, I remove the standby power, then checked out these Instructions for replacing the CPU to remove the CPU; Gautam also has done this before.
Fri Sep 6 20:09:01 2019 Fri Sep 6 20:09:02 2019
I pulled the leftmost CPU module out, following the instructions above. The CPU module matches the physical layout and part number of the Sun Fire X4600 M2 8-DIMM CPU module; pressing the fault reminder light gives amber indicators at the DIMM ejectors, indicating faulty DIMMs (see). The other indicator LEDs did not illuminate.
I located several spare DIMMs in the digital cabinet along Y arm (and a couple with misc computer components in the control room), but didn't find the correct one for this CPU module. The DIMM is Sun PN 371-1764-01; I found it online and ordered eight. Please let me know if this is incorrect.
To protect the CPU module, I've put it in an ESD safe bag with some bubble wrap and a note. It's on the E shop bench.
Assuming you are at pianosa, /etc/resolv.conf is like
# Generated by NetworkManager
But this should be like
as indicated in https://nodus.ligo.caltech.edu:8081/40m/14767
I did this change for now. But this might get overridden by Network Manager.
The MC unlocked ~20 min ago, correlated with 2 consecutive earthquakes in Mexico. The MC came back fine after a few minutes, but the WFS never engaged. I turned them on by hand. I think that Yuta mentioned once that he also had to turn the WFS on by hand. There may be a problem in the unlock/relock catching that needs to be looked at, to make sure the WFS come back on automatically.
Also, someone (Masha and I) should look at the seismic BLRMS. I have suspected for a few days that they're not telling us everything that we want to know. Usually, if there's an earthquake close enough / big enough that it pops the MC out of lock, it is clear from the BLRMS that that's what happened, but right now it doesn't look like much of anything....just kind of flat for hours.
The arrangement of filters in the WFS loop filter banks have been altered, Rana will update with details of the motivation behind these changes. Here is how the screen looks now:
I have updated the C1IOO SDF table, and also the mcwfson script to reflect these changes. The latter has been svn committed.
I installed 4 chassis in the rack 1X2 (characterization on the E-bench was deemed satisfactory, I will upload the analysis later). I ran out of hardware to make power cables so only 2 of them are powered right now (1 32ch AA chassis and 1 WFS head interface). The current limit on the +24V Sorensens was raised to allow for similar margin to the limit with the increased current draw.
While I definitely bumped various cables, I don't seem to have done any lasting damage to the CDS system (the RFM errors remain of course).
Mon Oct 7 14:51:53 2019. I closed the PSL shutter to measure the WFS head responsivity.
I made a thru calibration as in this elog, treating laser, reference PD, and WFS RF output as a three-port device. The DC current supplied to the laser is 20.0 mA in all cases. The Agilent spectrum analyzer supplies a -10 dBm excitation to Jenne laser's AM port, and A/B is measured with 20dB attenuation on each input port. Results are in /users/aaron/WFS/data/191007/. The calibration had 100 averages, all other measurements 32 averages; other parameters found in the yml file, same folder as the data.
I normalized the result by the difference between the dark and bright DC levels of each segment.
Mon Oct 7 17:29:58 2019 opened PSL shutter.
I simulated this circuit with zero, but haven't gotten the results to match the measurements above.
It would be good if you and Shruti can look at how to change the parameters in Zero so as to do a fit to the measured data. Usually, in scipy.optimize we give it a function with some changeable params, so maybe there's a way to pass params to a zero object in that way. I think Ian and Anchal are doing something similar to their FSS Pockel's cell simulator.
The assembly of the head is nearly complete, I thought I'd do some characterization before packaging everything up too nicely. I noticed that the tapped holes in the base are odd-sized. According to the official aLIGO drawing, these are supposed to be 4-40 tapped, but I find that something in between 2-56 and 4-40 is required - so it's a metric hole? Maybe we used some other DCC document to manufacture these parts - does anyone know the exact drawings used? In the meantime, the circuit is placed inside the enclosure with the back panel left open to allow some tuning of the trim caps. The front panel piece for mounting the SMA feedthroughs hasn't been delivered yet so hardware-wise, that's the last missing piece (apart from the aforementioned screws).
Attachment #1 - the circuit as stuffed for the RF frequencies of relevance to the 40m.
Attachment #2 - measured TF from the "Test Input" to Quadrant #1 "RF Hi" output.
Update 11 Dec: For whatever reason, whoever made this box decided to tap 4-40 holes from the bottom (i.e. on the side of the base plate), and didn't thread the holes all the way through, which is why I was unable to get a 4-40 screw in there. To be fair the drawing doesn't specify the depth of the 4-40 holes to be tapped. All the taps we have in the lab have a maximum thread length of 9/16" whereas we need something with at least 0.8" thread length. I'll ask Joe Benson at the physics workshop if he has something I can use, and if not, I'll just drill a counterbore on the bottom side and use the taps we have to go all the way through and hopefully that does the job.
The front panel I designed for the SMA feedthroughs arrived today. Unfortunately, it is impossible for the D-sub shaped holes in this box to accommodate 8 insulated SMA feedthroughs (2 per quadrant for RF low and RF high) - while the actual SMA connector doesn't occupy so much space, the plastic mold around the connector and the nut to hold it are much too bulky. For the AS WFS application, we will only need 4 so that will work, but if someone wants all 8 outputs (plus an optional 9th for the "Test Input"), a custom molded feedthrough will have to be designed.
As for the 170 MHz feature - my open loop modeling in Spice doesn't suggest a lack of phase margin at that frequency so I'm not sure what the cause is there. If this is true, just increasing the gain won't solve the issue (since there is no instability at least by the phase margin metric). Could be a problem with the "Test Input" path I guess. I confirmed it is present in all 4 quadrants.
I think the WFS head performs satisfactorily.
Details and remarks:
If the RF experts see some red flags / think there are more tests that need to be performed, please let me know. Big thanks to Chub for patiently supporting this build effort, I'm pleasantly surprised it worked.
It seems that early this morning MC got very misaligned. Yuta was able to align the Mode Cleaner again by individually adjusting the MC1 MC2, and MC3. Once transmission reach ~12000, we went ahead and turned on WFS. Oddly enough, the transmission began plummeting and MC fell out of lock. After this, Yuta reset the WFS offsets and realigned the WFS QPDs. We then locked MC and turned on WFS once again, but the same issue happened. After fiddeling around with this, we found the if we set C1:IOO-MC2_TRANS_PIT_OUTPUT and C1:IOO-WFS1_YAW_OUTPUT equal to 0, WFS does not cause this issue. Is there a proper to reset WFS, aside from only zeroing the offsets?
This issue is very weird and still unresolved. Without WFS loops, we'll have to realign IMC often and we might loose IMC alignment completely during weekends or long weekends.
I tried following things today but nothign worked:
We need to investigate this as first priority. Maybe some cable is loose, some PD power supply not working etc. Until we fix this, people should align IMC to > 12000 transmission counts whenever they have a spare 5 min. We need to work in place of WFS for sometime.
I could run the WFS servo (6dofs) for more than 15min by flipping the sign for the MC2 Pit and WFS1 Yaw. (See attachments)
This may mean that the sign of the loops / the input matrix / the output matrix, as well as the sensors and actuators, have the problem.
Isn't it the time to measure the sensing/actuation matrices? Maybe Tomislav already has the data?
I have reverted the changes as you may need more careful investigation.
We found that MC1 local damping loop signs were revereted to the state before our standardization on June 7th (40m/16898), but the WFS output matrix was not reverted.
This caused the sign flip in the feedback to MC1, which caused the IMC WFS issue.
This probably happened when we were restarting the models after RTS modeling (40m/16935). We might have used wrong snap files for burt-restoring.
We went back to the snapshot taken at 09:19 June 21, 2022 and now the IMC WFS is working,
We investigated why WFS loop wasn't working. It seemed like WFS1 PIT error signal has a huge offset which would push the loop to misalign all optics' PIT. So we did the following steps:
I'm scripting the WFS sensing matrix measurements. I haven't really scripted DTT before, so I'm trying to find documentation or existing scripts. I came across this elog where Gautam measured a sensing matrix during DRMI lock, and he pointed me to some .xml files used for these measurments.
not need to use DTT. I'm attaching some half-finished notebooks that give the gist.
That's it! Now you have the complex, single frequency TFs. Next you invert the matrix.
"# Get some ASC data - Calculate Sensing Matrix \n",
"### also make the radar plots"
We aligned optics of WFS as it was. Now auto-locker is working to lock MC.
But it still doesn't lock. We notice that the c1lsc machine doesn't work. So we run rebootCILSC.sh.
Now we reset the hardware!
After reset, auto locking didn't work well. Gautum and Aaron reboot slow c1ioo. Then it works, and Gautam returned the MC to a good alignment.
We found the beam is not in the center of the QPD, we (turned off the MC autolocker and MC loop, then) realigned to make beam to get in to the QPD center. Afterwards we start auto locking.
With the WFS on, the maximum MC transmission we observe is 14,700 counts; after the transmission level stabilizes (MC_TRANS pit and yaw brought to 0), the MC transmission is only 14,200 counts. Perhaps the MC_TRANS QPD offsets need adjustment. We relieve the WFS servo of its DC offsets. This is the configuration we'll use for WFS loop measurements this week.
Let's not worry about C1LSC until the c1iscaux upgrade is done.
I'm using the notebooks from rana as a starting point, and making a script to measure and fill the WFS sensing matrix. It lives at /users/aaron/WFS/scripts/WFSsensingMatrix.ipynb for now. Here's what it does; what's been tested is in green, untested is goldenrod, uncoded is fire brick.
To run these on pianosa, I ran (inside the jupyter notebook)
I'm getting an error when starting the nds2 connection
conn = nds2.connection('192.168.113.201', 31200)
Failed to establish a connection[INFO: Request SASL authentication protocol]+
I didn't find anything on the elog about this error, but I'm looking at the nds user manual. The problem was, I didn't have a valid Kerberos ticket; I opened one on Pianosa with my albert.einstein (note all caps ligo.org).
I'm now able to run the scripts Rana mentions, but I haven't been able to grab the channels I want (eg C1:SUS-MC1_ASCPIT_IN1_OUT); it says the channel isn't found. When I check how many of the Caltech channels are available (conn.count_channels('C1*')), there are none. I was connecting to nds.ligo.caltech.edu, but this must be the wrong server (it has all the channels for the sites). fb and fb1 (and the IP they point to, 192.168.113.201) cannot be connected to, giving the error 'Error occurred trying to write to socket.'
I recall that in the cryo lab, we need to use port 8088 to get data from cymac1, and indeed substituting 31200 -> 8088 lets me access the C1 channels (I can count the channels), but no matter what time I request, nds tells me there is no data available (gap). Gautam came by and diagnosed that the gaps I'm seeing in the frames' data are real, fb is down (see elog).
Continuing, I'm going to modify the script to grab live data. I'm using the iterate and next methods. I noticed that the MC2_TRANS pit/yaw channels are not saved to frames, even though WFS1/2 pit/yaw are. Since I expect I'll want to lookback at these channels, I followed the instructions for adding a daq channel, uncommenting the following line in /opt/rtcds/caltech/c1/chans/daq/C1IOO.ini:
I made a backup of the old version of this .ini file, which can be found in /users/aaron/backups/190917_C1IOO.ini. I did not remake the model, as I couldn't find the c1ioo model in /opt/rtcds/caltech/c1/userapps/trunk or from the matlab command prompt. I restarted the fb via telnet, but didn't restart the model or check the svn (got an error?). The _DQ channels are now reachable on dataviewer, so things seem to be working.
I also tried importing cdsutils, so I can control awg in the same script that we read out the sensing matrix, but I'm getting the python3 error when I import cdsutils:
I tried pip upgrading cdsutils, but it's already up-to-date. I get the above error even if I switch to a python 2 kernel; cdsutils is installed in the python2.7 directory, so I don't know why pip is finding it when I'm running a python 3 kernel. I can move on from this for now, but it would be useful to be able to script the excitation along with the measurement.
Tangentially related, Rika wanted to be running some jupyter notebooks while working on donatella. I ran, on donatella:
hm, that didn't work. Also jupyter is installed when you install conda, so I'm not sure how there is a version of conda but not of jupyter. I also see that pip and pip3 are not recognized commands on donatella.
I noticed that some of the functions in the scipy signal processing toolbox were out of date on pianosa. The cheby and welch filters now accept additional kwargs (for eg, before you needed to give IIR filter methods a cutoff frequency normalized to the Nyquist rate, but now you can give it the frequencies and sampling rate separately).
I want to update this package, but I hesitate to break everyone's existing scripts.
Once stop the auto-locker and realigned to make beam to get into QPD again.
After we lock MC, we took TFs from suspension MC1/2/3 PIT/YAW to WFS1/2 PIT/YAW.
Diagnotics test tools
range: 7 Hz to 50 Hz
Column 0: WFS2_PIT 1: WFS2_YAW 2:WFS1_PIT 3: WFS1_YAW 4: TRANCE_PIT 5:TRANCE_YAW
I'm wondering weather the MC1data I saved is correct, becouse I found the channel was changed when I exported MC2 data. So I took MC1 data again.
We got all data for TFs already. Each data is devided to real part and imaginary part. Then we are arranging the datas to obtain TFs.
TF of MC2 is attachiment 1. So tomorrow, I make other TF.
In the data we got yesterday, we can see some filter's effect.
But it is not good coherence above 10Hz, so we mesured again. And this time we save the data as xml file.
And also we chaned the frequency regions broader to watch corner frequency of suspension.
Diagnotics test tools
range: 0.1 Hz to 100 Hz
but at low frequency, the mode maching cavity was unloked cause of too much shaking.
So, we saw single frequency TF, and searched the good amplitude.
First, I tried to get TF @0.1~1 Hz .
0.1 to 1 Hz
points: 61 (I think it's too much becous it takes about an hour)
The TFs and coherence of MC1/PIT to each QPD is below. [above window: coherence, below: TF]
During the mesurement, something happened @0.2-0.3Hz so I stopped it.
We found the coherence of WFS1P and WFS2Y is not good, but others are good.
we guess that it could come from alignment which made Q chainging to small.
Finaly, I also got the .xml data of MC1P 1 Hz to 10 Hz. In this time,
1 to 10 Hz
Now we took single frequency 6 TFs (MC1/2/3 PIT/YAW) @7Hz (Because this frequency has good coherence in all channel).
Aaron wrote the script using dtt to making matrics.
I installed nds2 on donatello with yum, but still can't import nds2.
I installed nds2 again, this time successfully with
conda install -c conda-forge python-nds2-client
[rika, aaron, rana]
We are getting the MC locked in anticipation of making some WFS transfer function measurements.
The PSL screen was all white boxes, so I keyed the PSL crate and burt restored the settings from 11:19am Sep 5 (somewhat earlier than we started rebooting computers). Following this, I ran Milind's unstick.py and then the PSL autolocker script; both worked on the first go, great work Milind!
The modecleaner autolocking script is having substantially more trouble. Rana found that pitch and yaw sliders for all MC optics have been swapped--we think it's because the camera at MC2 has been rotated. Note that for now, sliding pitch gives a change in yaw, and sliding yaw changes pitch.
We noticed that with the WFS servo on, the modecleaner would be well aligned for a while (MC trans ~ 14000), only to lose lock after several minutes. We held the MC2_TRANS_PIT/YAW outputs at 0, so the MC2 QPD does not affect the WFS loop; the beam is well centered on WFS1/2, but not on the MC2 QPD, and with this signal out of the loop MC TRANS recovers to ~15000 counts (consistent with the quiet times over the last 90 days, see attachment 2). Attachment 1 shows the MC lock degrading, followed by some noise where we lost lock, and finally a visible increase in MC trans when we remove the MC2 QPD from the WFS loop.
MC1 Pich 4.4762 Yow 4.4669
MC2 Pich 3.7652 Yow -1.5482
MC3 Pich -0.4159 Yow 1.1477
After automatic locking MC, we stopped automatical locking and took alignment to the center of QPD.
And then again did the automatic locking MC. Finaly Rana move to best alignment.
MC1 Pich 4.4942 Yow 4.6956
MC2 Pich 3.7652 Yow -1.5600
MC3 Pich -0.3789 Yow 1.1477
We used diaggui to measure the response of WFS1/WFS2/MC2 pitch (yaw) to excitations in MC1/MC2/MC3 pitch (yaw). Seeing fluctuations of amplitude ~1 on the MCX_PIT/YAW_OUT channels, we used an amplitude 0.01 excitation at 20 Hz. We will work on scripting some of this tomorrow.
We should also have a plan for the next couple weeks so we are organized; heavily adapted from. Here's what I'm thinking this morning:
It would be good to script some of what we did yesterday. I'm checking out some scripts I'd used for Qryo and armloss measurements to remember the best way to do this.
I noticed yesterday that the PSL_shutterqst box is white, and I've seen timeout requests when eg the reboot script tries to open/close the PSL shutter. It seems like a shutter that should open, so I should find the aux machine to restart it.