I just sat down in the control room, and discovered the PMC (and everything else) unlocked. I relocked the PMC, but the MC wasn't coming back. After a moment of looking around, I discovered that the WFS were on, and railing. I ran the "turn WFS off" script, and the MC came back right away, and the WFS came on as they should.
We need to relook at the WFS script, or the MC down script, to make sure that any time the MC is unlocked, no matter why it unlocked, the WFS output is off and the filter histories are cleared.
The only script that can currently take this action is the MC autolocker. If that is disabled first and the PMC unlocks later, the WFS will not be turned off. During the last round of discussions we had about the autolocker script, sometime last Nov, we decided that too much automation is not desirable and that the autolocker should be kept as simple as possible.
I found that the MC WFS had large offset control signals going to the MC SUS. Even though the input switch was off, the integrators were holding the offset.
I have disabled the ASCPIT outputs in the MC SUS. Suresh is going to fix the MC autolocker script to gracefully handle the OFF and ON and then test the script before resuming the WFS testing.
MCL data for OAF may be suspect from this morning.
While going to take some transfer functions of the MC WFS loop, LSC was down. When we tried to restart the FE using 'rtcds restart --all', c1lsc crashed and froze. We manually reset c1lsc, then laboriously determined the correct order of machines to reboot. Here's what works best:
rtcds start c1x04 c1lsc c1ass c1oaf c1cal c1daf
Starting c1dnn crashes the other FE
rtcds restart --all
rtcds restart c1rfm c1sus c1mcs
restarting c1pem crashes the other FE on c1sus
We're seeing a lot of red IPC indicators--perhaps it's an issue with the order we're restarting?
via Polish chat, GV tells us to RTFE
As suggested, I ran the script cds/rebootC1LSC.sh
I got a timeout error when the script tried closing the PSL shutter ('C1:AUX-PSL_ShutterRqst' not found), but Rana and I closed the shutter before leaving last night. c1sus is down, so the script found no route to host c1sus; I'm thinking I need to reset c1sus for the script to run completely. Nonetheless, c1lsc was rebooted, which crashed c1ioo and left the c1lsc FE all red (probably because c1sus wasn't restarted).
I reset c1lsc, c1sus, and c1ioo.
I noticed that the script gives the command 'ssh c1XXX', but we have been getting no route to host using this command. Instead, the machines are currently only reachable as c1XXX.martian. I'm not sure why this is, so I just appended .martian in rebootC1LSC.sh
This time, the script does run. I did get 'no route to host' on c1ioo, so I think I need to reset that machine again. After reset, the script failed to login to c1ioo and c1lsc.
Fri Sep 6 13:09:05 2019
After lunch, I reset the computers again, and try the script again. There is again no route to host for c1ioo. I'm going inside to shutoff the power to c1ioo, since the reset buttom seems to not be working. I still can't login from nodus, so I'm bringing a keyboard and monitor over to plug in directly.
On reset, c1ioo repeatedly reaches the screen in attachment 1, before going black. Holding down shift or ctrl+alt+f1 doesn't get me a command prompt. After waiting/searching the elog for >>3 min, we decided to follow these instructions to cycle the power of c1ioo. The same problem recurred following power up. I found online some instructions that the SunSystems 4600 can hang during reboot if it has become too hot ("reboot during a thermal shutdown"); I did notice that the temperature light was on earlier in this procedure, so perhaps that is the problem. I followed the wiki instructions to shut down the computer again (pressed power button, unplugged 4 power supplies from back of machine), and left it unplugged for 10-30 min (Fri Sep 6 14:46:18 2019 ).
Fri Sep 6 15:03:31 2019
Rana plugged in the power supplies and reset the machine again.
Fri Sep 6 16:30:37 2019
c1ioo is still unreachable! I pressed reset once, and the reset button flashes white. The yellow warning light is still on.
Fri Sep 6 16:54:21 2019
The reset light has stopped flashing, but I still can't access c1ioo. I reset once more, this time watching c1ioo on a monitor directly. I'm still seeing the same boot screen repeatedly. I do see that CPU0 is not clocking, which seems weird.
Following gautam's elog here, I found the Sun Fire X4600 manual for locating faulty CPUs. After the white reset light stopped flashing, I held down the power button to turn off the system. Before shutdown, all of the CPU displayed amber lights; after shutdown, only the leftmost CPU (as viewed from the back, presumably CPU0) displays an amber light. The manual says this is evidence that the CPU or DIMM is faulty. Following the manual, I remove the standby power, then checked out these Instructions for replacing the CPU to remove the CPU; Gautam also has done this before.
Fri Sep 6 20:09:01 2019 Fri Sep 6 20:09:02 2019
I pulled the leftmost CPU module out, following the instructions above. The CPU module matches the physical layout and part number of the Sun Fire X4600 M2 8-DIMM CPU module; pressing the fault reminder light gives amber indicators at the DIMM ejectors, indicating faulty DIMMs (see). The other indicator LEDs did not illuminate.
I located several spare DIMMs in the digital cabinet along Y arm (and a couple with misc computer components in the control room), but didn't find the correct one for this CPU module. The DIMM is Sun PN 371-1764-01; I found it online and ordered eight. Please let me know if this is incorrect.
To protect the CPU module, I've put it in an ESD safe bag with some bubble wrap and a note. It's on the E shop bench.
Assuming you are at pianosa, /etc/resolv.conf is like
# Generated by NetworkManager
But this should be like
as indicated in https://nodus.ligo.caltech.edu:8081/40m/14767
I did this change for now. But this might get overridden by Network Manager.
The MC unlocked ~20 min ago, correlated with 2 consecutive earthquakes in Mexico. The MC came back fine after a few minutes, but the WFS never engaged. I turned them on by hand. I think that Yuta mentioned once that he also had to turn the WFS on by hand. There may be a problem in the unlock/relock catching that needs to be looked at, to make sure the WFS come back on automatically.
Also, someone (Masha and I) should look at the seismic BLRMS. I have suspected for a few days that they're not telling us everything that we want to know. Usually, if there's an earthquake close enough / big enough that it pops the MC out of lock, it is clear from the BLRMS that that's what happened, but right now it doesn't look like much of anything....just kind of flat for hours.
The arrangement of filters in the WFS loop filter banks have been altered, Rana will update with details of the motivation behind these changes. Here is how the screen looks now:
I have updated the C1IOO SDF table, and also the mcwfson script to reflect these changes. The latter has been svn committed.
I installed 4 chassis in the rack 1X2 (characterization on the E-bench was deemed satisfactory, I will upload the analysis later). I ran out of hardware to make power cables so only 2 of them are powered right now (1 32ch AA chassis and 1 WFS head interface). The current limit on the +24V Sorensens was raised to allow for similar margin to the limit with the increased current draw.
While I definitely bumped various cables, I don't seem to have done any lasting damage to the CDS system (the RFM errors remain of course).
Mon Oct 7 14:51:53 2019. I closed the PSL shutter to measure the WFS head responsivity.
I made a thru calibration as in this elog, treating laser, reference PD, and WFS RF output as a three-port device. The DC current supplied to the laser is 20.0 mA in all cases. The Agilent spectrum analyzer supplies a -10 dBm excitation to Jenne laser's AM port, and A/B is measured with 20dB attenuation on each input port. Results are in /users/aaron/WFS/data/191007/. The calibration had 100 averages, all other measurements 32 averages; other parameters found in the yml file, same folder as the data.
I normalized the result by the difference between the dark and bright DC levels of each segment.
Mon Oct 7 17:29:58 2019 opened PSL shutter.
I simulated this circuit with zero, but haven't gotten the results to match the measurements above.
It would be good if you and Shruti can look at how to change the parameters in Zero so as to do a fit to the measured data. Usually, in scipy.optimize we give it a function with some changeable params, so maybe there's a way to pass params to a zero object in that way. I think Ian and Anchal are doing something similar to their FSS Pockel's cell simulator.
The assembly of the head is nearly complete, I thought I'd do some characterization before packaging everything up too nicely. I noticed that the tapped holes in the base are odd-sized. According to the official aLIGO drawing, these are supposed to be 4-40 tapped, but I find that something in between 2-56 and 4-40 is required - so it's a metric hole? Maybe we used some other DCC document to manufacture these parts - does anyone know the exact drawings used? In the meantime, the circuit is placed inside the enclosure with the back panel left open to allow some tuning of the trim caps. The front panel piece for mounting the SMA feedthroughs hasn't been delivered yet so hardware-wise, that's the last missing piece (apart from the aforementioned screws).
Attachment #1 - the circuit as stuffed for the RF frequencies of relevance to the 40m.
Attachment #2 - measured TF from the "Test Input" to Quadrant #1 "RF Hi" output.
Update 11 Dec: For whatever reason, whoever made this box decided to tap 4-40 holes from the bottom (i.e. on the side of the base plate), and didn't thread the holes all the way through, which is why I was unable to get a 4-40 screw in there. To be fair the drawing doesn't specify the depth of the 4-40 holes to be tapped. All the taps we have in the lab have a maximum thread length of 9/16" whereas we need something with at least 0.8" thread length. I'll ask Joe Benson at the physics workshop if he has something I can use, and if not, I'll just drill a counterbore on the bottom side and use the taps we have to go all the way through and hopefully that does the job.
The front panel I designed for the SMA feedthroughs arrived today. Unfortunately, it is impossible for the D-sub shaped holes in this box to accommodate 8 insulated SMA feedthroughs (2 per quadrant for RF low and RF high) - while the actual SMA connector doesn't occupy so much space, the plastic mold around the connector and the nut to hold it are much too bulky. For the AS WFS application, we will only need 4 so that will work, but if someone wants all 8 outputs (plus an optional 9th for the "Test Input"), a custom molded feedthrough will have to be designed.
As for the 170 MHz feature - my open loop modeling in Spice doesn't suggest a lack of phase margin at that frequency so I'm not sure what the cause is there. If this is true, just increasing the gain won't solve the issue (since there is no instability at least by the phase margin metric). Could be a problem with the "Test Input" path I guess. I confirmed it is present in all 4 quadrants.
I think the WFS head performs satisfactorily.
Details and remarks:
If the RF experts see some red flags / think there are more tests that need to be performed, please let me know. Big thanks to Chub for patiently supporting this build effort, I'm pleasantly surprised it worked.
We investigated why WFS loop wasn't working. It seemed like WFS1 PIT error signal has a huge offset which would push the loop to misalign all optics' PIT. So we did the following steps:
I'm scripting the WFS sensing matrix measurements. I haven't really scripted DTT before, so I'm trying to find documentation or existing scripts. I came across this elog where Gautam measured a sensing matrix during DRMI lock, and he pointed me to some .xml files used for these measurments.
not need to use DTT. I'm attaching some half-finished notebooks that give the gist.
That's it! Now you have the complex, single frequency TFs. Next you invert the matrix.
"# Get some ASC data - Calculate Sensing Matrix \n",
"### also make the radar plots"
We aligned optics of WFS as it was. Now auto-locker is working to lock MC.
But it still doesn't lock. We notice that the c1lsc machine doesn't work. So we run rebootCILSC.sh.
Now we reset the hardware!
After reset, auto locking didn't work well. Gautum and Aaron reboot slow c1ioo. Then it works, and Gautam returned the MC to a good alignment.
We found the beam is not in the center of the QPD, we (turned off the MC autolocker and MC loop, then) realigned to make beam to get in to the QPD center. Afterwards we start auto locking.
With the WFS on, the maximum MC transmission we observe is 14,700 counts; after the transmission level stabilizes (MC_TRANS pit and yaw brought to 0), the MC transmission is only 14,200 counts. Perhaps the MC_TRANS QPD offsets need adjustment. We relieve the WFS servo of its DC offsets. This is the configuration we'll use for WFS loop measurements this week.
Let's not worry about C1LSC until the c1iscaux upgrade is done.
I'm using the notebooks from rana as a starting point, and making a script to measure and fill the WFS sensing matrix. It lives at /users/aaron/WFS/scripts/WFSsensingMatrix.ipynb for now. Here's what it does; what's been tested is in green, untested is goldenrod, uncoded is fire brick.
To run these on pianosa, I ran (inside the jupyter notebook)
I'm getting an error when starting the nds2 connection
conn = nds2.connection('192.168.113.201', 31200)
Failed to establish a connection[INFO: Request SASL authentication protocol]+
I didn't find anything on the elog about this error, but I'm looking at the nds user manual. The problem was, I didn't have a valid Kerberos ticket; I opened one on Pianosa with my albert.einstein (note all caps ligo.org).
I'm now able to run the scripts Rana mentions, but I haven't been able to grab the channels I want (eg C1:SUS-MC1_ASCPIT_IN1_OUT); it says the channel isn't found. When I check how many of the Caltech channels are available (conn.count_channels('C1*')), there are none. I was connecting to nds.ligo.caltech.edu, but this must be the wrong server (it has all the channels for the sites). fb and fb1 (and the IP they point to, 192.168.113.201) cannot be connected to, giving the error 'Error occurred trying to write to socket.'
I recall that in the cryo lab, we need to use port 8088 to get data from cymac1, and indeed substituting 31200 -> 8088 lets me access the C1 channels (I can count the channels), but no matter what time I request, nds tells me there is no data available (gap). Gautam came by and diagnosed that the gaps I'm seeing in the frames' data are real, fb is down (see elog).
Continuing, I'm going to modify the script to grab live data. I'm using the iterate and next methods. I noticed that the MC2_TRANS pit/yaw channels are not saved to frames, even though WFS1/2 pit/yaw are. Since I expect I'll want to lookback at these channels, I followed the instructions for adding a daq channel, uncommenting the following line in /opt/rtcds/caltech/c1/chans/daq/C1IOO.ini:
I made a backup of the old version of this .ini file, which can be found in /users/aaron/backups/190917_C1IOO.ini. I did not remake the model, as I couldn't find the c1ioo model in /opt/rtcds/caltech/c1/userapps/trunk or from the matlab command prompt. I restarted the fb via telnet, but didn't restart the model or check the svn (got an error?). The _DQ channels are now reachable on dataviewer, so things seem to be working.
I also tried importing cdsutils, so I can control awg in the same script that we read out the sensing matrix, but I'm getting the python3 error when I import cdsutils:
I tried pip upgrading cdsutils, but it's already up-to-date. I get the above error even if I switch to a python 2 kernel; cdsutils is installed in the python2.7 directory, so I don't know why pip is finding it when I'm running a python 3 kernel. I can move on from this for now, but it would be useful to be able to script the excitation along with the measurement.
Tangentially related, Rika wanted to be running some jupyter notebooks while working on donatella. I ran, on donatella:
hm, that didn't work. Also jupyter is installed when you install conda, so I'm not sure how there is a version of conda but not of jupyter. I also see that pip and pip3 are not recognized commands on donatella.
I noticed that some of the functions in the scipy signal processing toolbox were out of date on pianosa. The cheby and welch filters now accept additional kwargs (for eg, before you needed to give IIR filter methods a cutoff frequency normalized to the Nyquist rate, but now you can give it the frequencies and sampling rate separately).
I want to update this package, but I hesitate to break everyone's existing scripts.
Once stop the auto-locker and realigned to make beam to get into QPD again.
After we lock MC, we took TFs from suspension MC1/2/3 PIT/YAW to WFS1/2 PIT/YAW.
Diagnotics test tools
range: 7 Hz to 50 Hz
Column 0: WFS2_PIT 1: WFS2_YAW 2:WFS1_PIT 3: WFS1_YAW 4: TRANCE_PIT 5:TRANCE_YAW
I'm wondering weather the MC1data I saved is correct, becouse I found the channel was changed when I exported MC2 data. So I took MC1 data again.
We got all data for TFs already. Each data is devided to real part and imaginary part. Then we are arranging the datas to obtain TFs.
TF of MC2 is attachiment 1. So tomorrow, I make other TF.
In the data we got yesterday, we can see some filter's effect.
But it is not good coherence above 10Hz, so we mesured again. And this time we save the data as xml file.
And also we chaned the frequency regions broader to watch corner frequency of suspension.
Diagnotics test tools
range: 0.1 Hz to 100 Hz
but at low frequency, the mode maching cavity was unloked cause of too much shaking.
So, we saw single frequency TF, and searched the good amplitude.
First, I tried to get TF @0.1~1 Hz .
0.1 to 1 Hz
points: 61 (I think it's too much becous it takes about an hour)
The TFs and coherence of MC1/PIT to each QPD is below. [above window: coherence, below: TF]
During the mesurement, something happened @0.2-0.3Hz so I stopped it.
We found the coherence of WFS1P and WFS2Y is not good, but others are good.
we guess that it could come from alignment which made Q chainging to small.
Finaly, I also got the .xml data of MC1P 1 Hz to 10 Hz. In this time,
1 to 10 Hz
Now we took single frequency 6 TFs (MC1/2/3 PIT/YAW) @7Hz (Because this frequency has good coherence in all channel).
Aaron wrote the script using dtt to making matrics.
I installed nds2 on donatello with yum, but still can't import nds2.
I installed nds2 again, this time successfully with
conda install -c conda-forge python-nds2-client
[rika, aaron, rana]
We are getting the MC locked in anticipation of making some WFS transfer function measurements.
The PSL screen was all white boxes, so I keyed the PSL crate and burt restored the settings from 11:19am Sep 5 (somewhat earlier than we started rebooting computers). Following this, I ran Milind's unstick.py and then the PSL autolocker script; both worked on the first go, great work Milind!
The modecleaner autolocking script is having substantially more trouble. Rana found that pitch and yaw sliders for all MC optics have been swapped--we think it's because the camera at MC2 has been rotated. Note that for now, sliding pitch gives a change in yaw, and sliding yaw changes pitch.
We noticed that with the WFS servo on, the modecleaner would be well aligned for a while (MC trans ~ 14000), only to lose lock after several minutes. We held the MC2_TRANS_PIT/YAW outputs at 0, so the MC2 QPD does not affect the WFS loop; the beam is well centered on WFS1/2, but not on the MC2 QPD, and with this signal out of the loop MC TRANS recovers to ~15000 counts (consistent with the quiet times over the last 90 days, see attachment 2). Attachment 1 shows the MC lock degrading, followed by some noise where we lost lock, and finally a visible increase in MC trans when we remove the MC2 QPD from the WFS loop.
MC1 Pich 4.4762 Yow 4.4669
MC2 Pich 3.7652 Yow -1.5482
MC3 Pich -0.4159 Yow 1.1477
After automatic locking MC, we stopped automatical locking and took alignment to the center of QPD.
And then again did the automatic locking MC. Finaly Rana move to best alignment.
MC1 Pich 4.4942 Yow 4.6956
MC2 Pich 3.7652 Yow -1.5600
MC3 Pich -0.3789 Yow 1.1477
We used diaggui to measure the response of WFS1/WFS2/MC2 pitch (yaw) to excitations in MC1/MC2/MC3 pitch (yaw). Seeing fluctuations of amplitude ~1 on the MCX_PIT/YAW_OUT channels, we used an amplitude 0.01 excitation at 20 Hz. We will work on scripting some of this tomorrow.
We should also have a plan for the next couple weeks so we are organized; heavily adapted from. Here's what I'm thinking this morning:
It would be good to script some of what we did yesterday. I'm checking out some scripts I'd used for Qryo and armloss measurements to remember the best way to do this.
I noticed yesterday that the PSL_shutterqst box is white, and I've seen timeout requests when eg the reboot script tries to open/close the PSL shutter. It seems like a shutter that should open, so I should find the aux machine to restart it.
We identified the Jenne laser and found a long optical fiber that might be able to transport our beam to the AP table.
Now we're searching for documentation on using this laser. Kevin and John measured a TF last year. Koji advised that we needn't worry too much, the current limit is already set correctly and we need only power on the laser.
We moved the breadboard (including a couple PDs, collimating lenses, laser, steering mirrors, etc) over to the AP table, and set it on top of the panel next to the WFS. We mounted the laser on the AP table, and added one lens with f~68 mm after the laser to fit the beam on a single quadrant; the beam was about 1mm diameter (measured by eye) when it entered the QPD. We turned the laser driver on at ~19.4 mA, and directed it to WFS2 via the last two steering mirrors before WFS2.
We monitored the QPD segments' DC level with ndscope on a laptop, and were able to send the beam to each of the four quadrants in turn. We set up the Agilent network analyzer to drive the laser's amplitude modulation and sent the RF signal from the LEMO output on the QPD head directly to the network analyzer. We will take the measurements tomorrow morning.
At Seiji and Gautam's suggestion, we added an additional RF photodiode (NewFocus 1611) to the system so we can calibrate our transfer functions. The configuration is now laser -> BS --> lenses -> QPD and BS --> lenses -> RFPD. We added lenses to get the beams focused on the RFPD and QPD heads, and are again set up for TF measurement.
We took the following data. These parameters were consistent across all measurements:
After taking the data for segment 1, I moved the beam to segment 2. The beam didn't fit on segment 2 without partially illuminating segment 1 (tested by maximizing the signal on segment 2, then blocking the beam. If the beam is entirely on one segment, only that segment should be effected; in this case, we found that segment 1's DC signal also changed when the beam was blocked). We readjusted the telescoping lenses to get the beam a bit smaller, and now the beam fits on segment 2. We know it is entirely on segment 2 because small beam movements do not change the signal on segment 2.
We are trying to take the remaining data, but AGmeasure keeps hanging while sending the data (after taking the measurement, over 10 min). We tried restarting the network analyzer to no avail. I was able to grab the data by cancelling the measurement and running
I've uploaded the spectrum for segment 1 in the meantime. Zero model is on the way.
When I finished up the measurements on WFS2, I removed the cables from the AP table and closed the cover.
EDIT: I forgot to switch the LEMO connector to measure the other segments, so we measured the RF signal from segment 1 even when the beam was on segments 2-4. We'll have to try again tomorrow.
We are at it again. Rika is setting up the TF measurement, I'm looking into scripting the WFS sensing matrix measurement we made earlier in the week so we can return to it next week.
When we mesuring TF of SEG4, the beam leaking to SEG1 about 1%.
We finished mesurement SEG2-4 and get the figure by running PDH_calibrate.ipynb .
edit: We observed during segment 2 measurements that blocking the beam reduced the DC level of segment 1 by less than 1%, but still clearly observable. As you can see in the plots, something is suspicious about the normalization of these TFs. We took segment 1 data a few days before the other segments, so perhaps we weren't getting the full beam on the reference PD during the later measurements? When I make this measurement for WFS1, I will try to fix some of these problems by choosing different telescoping optics, and I will consider whether removing the QPD heads from their table will improve the measurement.
I wanted to make a zero model of this circuit to get a handle on the results. I couldn't import zero on pianosa, and I tried pip installing zero, but was denied due to not finding version 3.0.3 of matplotlib. I finally got it to install using
Oddly, even though I can now import zero when I open a python3 session from the command line, when I open a jupyter notebook and switch to a python3 kernel, the zero module is still unavailable. I think I recall that conda manages the jupyter environment -- is pip managing an entirely separate environment (annoying)?
edit: Yeah, it was something like that. I reminded myself how this works with this article.
I set up the spectrum analyzer to make the WFS head RF transfer function measurement (V/W) on WFS1. I placed the Jenne laser on the AP table, along with the reference PD power supply, laptop, and laser power supply. The Agilent output AM modulates the laser; the reference PD is again NewFocus 1611, with its AC output sent to Agilent's R channel and DC output sent to an oscilloscope;
I closed the PSL shutter while checking for a location to place the breadboard, and opened it while writing this. Headed back to Cryo to pick up the large incandescent bulb we'd borrowed over the summer.
The trend shows a big jolt to the MC1/3 pointing this morning at 8:30.
If not, we will have to put a 'no janitor' sign on all of the 40m doors permanently to prevent mops misaligning our interferometer.
The MC trend for the last 2 days shows that the MC suspensions were kicked again earlier today. Looking back at the suspension channel INMONs along with the MC trans sum shows that the suspensions get kicked everytime MC locks and unlocks. (Attch:1)
So I checked the effect of WFS on the suspensions by disabling and enabling the WFS feedback servo (Attch:2).
Since the IMC is not at it best pointing, whenever the MC autolocker runs and enables the WFS, the suspensions look like they are getting kicked. But really, it's just the WFS doing their job.
Edit, JCD: What this really means is that our DC MC pointing is bad, and we need to move the MC suspensions to offload the WFS. (All of the WFS output numbers for MC1 and 3 were around 100, which is pretty big for those numbers). We should resurrect the WFS offloading scripts so that we can do this more regularly, and not have to do it by hand.
I've measured coherence between seismometer signals and OSEMS of MC 1,2,3
GUR1 was rotated on the angle pi / 4 relative to x arm to match suspos axis of MC1 and MC3. Coherence between MC2 and GUR2_X is high, between MC3 and GUR1_X is ~0.2 but is compensated by GUR1_Y, between MC1 and GUR1_Y below 1Hz is ~0.5 and is not compensated by GUR1_X.
Then I measured coherence between GUR1_Y and MC3 OSEMS in 3 regimes :
Once WFS are ON, signal becomes noisy, FEEDBACK is OK. Then I measured coherence between MC_F and GUR2_X with WFS ON and OFF
Coherence between MC_F and GUR2_X is less when WFS are ON in the range 0.2 - 1 Hz and 4 - 10 Hz. Moreover PSD of MC_F seems to be higher at 10 - 100 Hz. But this may be caused by other reasons.
Then I measured the coherence between IN and OUT signals in WFS_SERVO
One filter bank adds noise to WFS signals and because of that we loose coherence between MC_F and seismic motion.
This effect is due to C1:IOO-WFS1_PIT_LIMIT=2000. When I turned if off, coherence between C1:IOO-WFS_PIT input and output signals restored
The other thing is that WFS actuate on the angular motion, but this couples to the position motion. We need to diagonalize the actuator.
OK. Then we should make this number bigger such that the coherence is still completely maintained.
Is this set in the auto locker? Or manually set?
This effect is due to C1:IOO-WFS1_PIT_LIMIT=2000. When I turned if off, coherence between C1:IOO-WFS_PIT input and output signals restored
OK. Then we should make this number bigger such that the coherence is still completely maintained.
Is this set in the auto locker? Or manually set?
LIMIT is set manually, auto locker does not change it. I've put C1:IOO-WFS1_PIT_LIMIT=4000, it seems to be fine for now.
IMC WFS operating point seemed to get degraded.
- IMC WFS feedback was relieved.
- WFS servo was turned off.
- IMC alignment was tuned carefully
- /opt/rtcds/caltech/c1/scripts/MC/WFS/WFS_FilterBank_offsets was run
/opt/rtcds/caltech/c1/scripts/MC/WFS/WFS_FilterBank_offsets was run
- WFS servo was turned on again
WFS are back on, and working nicely. Den and I had seen a problem (which I had seen before) that when you turn on the integrators in the WFS loops, the MC Refl value gets worse (goes up). Koji reminded me that he had a nice elog (7452) on how to get the MC awesome. Ayaka and I last night stopped after Step 7. Step 8, zeroing the WFS offsets, is the secret important thing that I keep forgetting. I ran the script /opt/rtcds/caltech/c1/scripts/MC/WFS/WFS_FilterBank_offsets, then turned on the WFS loop, and the WFS are working just fine now.
I'm back to wishing that I had control over PZT1. I went back and forth for a while between 1Y4 and the ETMY table to get the IPANG beam centered on the QPD. Initially, the beam was coming out of the vacuum a little high. The digital HV power supply is pitch, and the analog HV power supply is yaw. When I get the beam reasonably well centered on IPANG, with a beautiful, non-clipped, beam, the beam is much too high on IPPOS. The beam barely hits the top of the first out of vac steering mirror, and then is too high on the QPD. It looks like (based on the sum value) that the beam is on the diode, just entirely in the upper 2 quadrants. But if I try to fix this, since IPANG has a longer lever arm, the IPANG beam doesn't come out of the vacuum. I have compromised by getting the beam on IPANG, centered in pitch and ignoring IPPOS pitch. For yaw, since moving PZT2 only makes one of POS or ANG good at a time, I split the difference, so the beam is not really centered in yaw on either, but it's close on both.
AS beam is back on the camera. I forgot that, especially since the beam at REFL is pretty bright, I had moved the AS and REFL cameras out of the beam so we didn't crispy-fry the CCDs. Therefore, the camera spots are no longer a reference of spot position. We can still eyeball the position on, say, the 2" lens, but that's not any kind of accurate. I put the AS camera back to it's normal place, so the AS beam is going toward AS55 and AS110, and a little bit is going to the camera. I have not yet aligned the beams actually on to the PDs.
REFL beam is dumped by a razor dump after the 2" lens. Manasa did some work (elog 7666) to the REFL path, and I'm not 100% sure how it was before, so I leave it to her to please work on during the day tomorrow. I think we need to put back the Y1 that used to be there, but I don't know where the optic is. I put a yaw dither on the PRM with awggui, and saw the REFL beam moving at my 2Hz dither frequency, so this time we actually have a useful beam coming out.
I was trying to lock and look at the ASS for the Yarm, but the transmitted power was oscillating very near 1Hz. Eventually I looked at the mode cleaner, and it was also moving around at a similar frequency. I took spectra of the ETMY SUS damping feedback signals, and they (POS, PIT, YAW) saw this 1Hz motion too (see attached plots...same data, one is a zoom around 1Hz).
As a first place to start, I turned off the WFS, which immediately stopped the MC oscillation. Turning the WFS back on, the oscillation didn't come back. I'm not sure what happened to make the WFS bad, but I perhaps had the ASS dither lines on (I've had them on and off, so I'm not sure), although turning off the dither lines didn't make the WFS any better.
As an aside, the MC refl with the WFS off was ~1.5, rather than the ~0.5 with the WFS on, which means that the PSL beam and the MC axis are not well matched.
The scripts and screens needed to make the MC WFS ouput matrix are once again functional.
I corrected the WFS lockins' phases to ensure that the Q outputs are minimised. Since all the lockins have the same relative phase with respect to the oscillator I found that the same phase works for all of them. About 90 deg in this case.
The scripts used to make the WFS outmatrix measurement live in /cvs/cds/rtcds/caltech/c1/scripts/MC/WFS
1) setupWFSlockins: This script makes sure that all the ASC, WFS_ LKIN and WFS_servo filter banks used in this measurement are set up properly. It also sets the WFS_lockin oscillator to 10 Hz. There are filter modules in the SIG filter bank of the WFS demodulators.
2) senseWFSoutMATRX: This script cycles through the various MC actuators ( MC 1 2 3 : PIT and YAW ) and measures the response of the various ASC sensors (WFS and MC2_TRANS QPD).
3) The data collected by the sensWFSoutMATRX can be analysed with a matlab file called " wfsmatrix3.m " located in a subdirectory under WFS called 'matlab'. I have added some comments in this file to make it easier to follow. The output of this file, at the moment, gives only the " Actuation Vectors " for WFS1P, WFS2P, WFS1Y and WFS2Y. It ignores the MC2TransQPD for now.
4) The lockin outputs are given below ( the 'reduceddata' )
wfs1p wfs2p mc2tp wfs1y wfs2y mc2ty
mc1p 0.2926 -0.4086 0.2926 0.0340 0.0064 0.0001
mc2p -0.2830 -1.3060 -0.2833 0.0628 0.1171 -0.0003
mc3p -0.3283 -0.3455 -0.3288 -0.0456 0.0275 0.0000
mc1y 0.0440 0.0261 0.0429 0.7204 0.9351 -0.0008
mc2y -0.1006 0.0850 -0.1036 -1.5509 -0.3882 0.0165
mc3y 0.0150 -0.0832 0.0144 0.1114 -1.0573 0.0006
5) The actuation vectors are given below
6) This measurement was performed with the WFS servo loops open. I will try to close the loops with this matrix and run the script again to measure the output matrix in closed loop.
7) This a.vectors obtained above are significantly different from that obtained a while ago (elog 5668) before the lockin demod phases (relative to each other) were fixed. This could also be because both are open loop measurements and we might have wandered into the nonlinear regime of the WFS sensors.
The scripts used to make the WFS outmatrix measurement live in /cvs/cds/rtcds/caltech/c1/scripts/MC/WFS
I assume you mean /opt/rtcds/caltech/c1/scripts/MC/WFS.
As I've tried to reitterate many times: we do not use /cvs/cds anymore. Please put all new scripts into the proper location under /opt/rtcds.
Yes the files are in /opt/rtcds/caltech/c1/scripts/MC/WFS.
I just went to wherever the 'scripts' alias takes me, found the 'pwd' and did a cp+paste of the path. I checked to be sure that 'scripts' takes me to /opt/rtcds/caltech/c1/scripts/.
So why does the pwd show /cvs/cds.... instead of /opt/rtcds ?
1) To see if there are significant dark-offsets on the WFS sensors we closed the PSL shutter and found that the offsets are in the 1% range. We decided to ignore them for now.
2) To center the MC_REFL beam on the WFS we opened the PSL shutter, unlocked the MC and then centered the DC_PIT and DC_YAW signals in the C1IOO_WFS_QPD screen.
3) We then looked at the power spectrum of the I and Q signals from WFS1 to see if the spectrum looked okay and found that some of the quadrants looked very different from others. The reason was traced to incorrect Comb60 filters. After correcting these filters we adjusted the R phase angle in the WFS1_SETTINGS screen to suppress the 1Hz natural oscillation signal in the Q channels of all the four quadrants. We repeated this process for WFS2
4) To see if the relative phase of all four quadrants was correct we first drove the MC_length and tried to check the phase of the response on each quadrant. However the response was very weak as the signal was suppressed by the MC servo. Increasing the drive made the PMC lock unstable. So we introduced a 6Hz, 50mVpp signal from an SR785 into the MC_servo (Input2) and with this we were able to excite a significant response in the WFS without affecting the PMC servo. By looking at the time series of the signals from the quadrants we set the R phase angle in WFS_Settings such that all the quadrants showed the same phase response to the MC_length modulation.
Using the larger response were were able to further tweak the R angle to supress the Q channels to about 1% of the I phase signals.
5) I then edited the c1ioo.mdl so that we can use the six lockins just as they are used in MC_ASS. However we can now set elements of the SEN_DMD_MATRX (sensor demod matrix) to select any of the MCL, WFS PIT and YAW channels (or a linear combination of them) for demodulation. The change is shown below. While compiling and model on C1IOO FE machine there were problems which eventually led to the FB crash.
[Larisa and Jenne]
A few weeks ago (on the 28th of January) I had tried to measure the quantum efficiency of one quadrant of the WFS as a function of angle. However, Rana pointed out that I was a spaz, and had forgotten to put a lens in front of the laser. Why I forgot when doing the measurement as a function of angle, but I had remembered while doing it at normal incidence for all of the quadrants, who knows?
Anyhow, Larisa measured the quantum efficiency today. She used WFS2, quadrant 1 (totally oil-free), since that was easier than WFS1. She also used the Jenne Laser (with a lens), since it's more stable and less crappy than the CrystaLasers. We put a 50 Ohm terminator on the RF input of the Jenne Laser, since we weren't doing a swept sine measurement. Again, the Ophir power meter was used to measure the power incident on the diode, and the reflected power, and the difference between them was used as the power absorbed by the diode for the quantum efficiency measurement. A voltmeter was used to measure the output of the diode, and then converted to current as in the quote below.
Still on the to-do list: Replace the WFS2 diode. See if we have one around, otherwise order one. Align beams onto WFS so we can turn on the servo.
QE = (h*c)/(lambda*e) * (I/P)
Where I = (Volts from Pin1 to GND)/2 /500ohms
P = Power from laser - power reflected from diode.
h, c, e are the natural constants, and lambda is 1064nm.
Also, I/P = Responsivity
Larissa is going to put her data and plots into the elog shortly....
Quantum Efficiency Measurement:
I refer to Jamie's LHO elog for the equation governing quantum efficiency of photodiodes: LHO 2 Sept 2009
The information I gathered for each quadrant of each WFS was:  Power of light incident on PD (measured with the Ophir power meter),  Power of light reflected off the PD (since this light doesn't get absorbed, it's not part of the QE), and  the photo current output by the PD (To get this, I measured the voltage out of the DC path that is meant to go to EPICS, and backed out what the current is, based on the schematic, attached).
I found a nifty 25 pin Dsub breakout board, that you can put in like a cable extension, and you can use clip doodles to look at any of the pins on the cable. Since this was a PD activity, and I didn't want to die from the 100V bias, I covered all of the pins I wasn't going to use with electrical tape. After turning down the 100V Kepco that supplies the WFS bias, I stuck the breakout board in the WFS. Since I was able to measure the voltage at the output of the DC path, if you look at the schematic, I needed to divide this by 2 (to undo the 2nd op amp's gain of 2), and then convert to current using the 499 Ohm resistor, R66 in the 1st DC path.
I did all 4 quadrants of WFS1 using a 532nm laser pointer, just to make sure that I had my measurement procedure under control, since silicon PDs are nice and sensitive to green. I got an average QE of ~65% for green, which is not too far off the spec of 70% that Suresh found.
I then did all 8 WFS quadrants using the 1064nm CrystaLaser #2, and got an average QE of ~62% for 1064 (58% if I exclude 2 of the quadrants....see below). Statistics, and whatever else is needed can wait for tomorrow.
Problem with 2 quadrants of WFS2?
While doing all of this, I noticed that quadrants 3 and 4 of WFS2 seem to be different than all the rest. You can see this on the MEDM screens in that all 6 other quadrants, when there is no light, read about -0.2, whereas the 2 funny quadrants read positive values. This might be okay, because they both respond to light, in some kind of proportion to the amount of light on them. I ended up getting QE of ~72% for both of these quadrants, which doesn't make a whole lot of sense since the spec for green is 70%, and silicon is supposed to be less good for infrared than green. Anyhow, we'll have to meditate on this. We should also see if we have a trend, to check how long they have been funny.
Here is the followup on Jenne's February 14th, 2011 update on the quantum efficiency measurements of WFS2.
Attached is a PDF of my calculations, based on measurements ranging between 0-25 degrees in 5 degree increments.
The graph at the bottom plots these angles versus the calculated quantum efficiency at each point and the responsivity. Since quantum efficiency and responsivity only differ by a factor of some natural constants (lamda, e, h, c), I used a graph with two vertical axes, because the points would be plotted at essentially the same location if quantum efficiency (%) and responsivity (Amps/Watts) were graphed on two separate plots.
The calculated values for quantum efficiency based on my measurements (labelled "ExpAverage") were pretty close to what Jenne had calculated in earlier attempts, which was around 60%. Just to test, I compared my quantum efficiency result against the calculation of quantum efficiency using the responsivity value for silicon, 0.5 Amps/Watt, which is labelled as "Spec". Comparison of "ExpAverage" and "Spec" shows that they differ by only about 2%, so I conclude that the theoretical quantum efficiency calculated using a given responsivity agrees with my measurement-based experimental result.
I am (was) able to get the mode cleaner mostly locked, but because WFS2 wasn't centered, the MC would drift, then lose lock. I recentered both the WFS (after unlocking the MC and the MZ), and am now about to commence relocking both of those.
Note to self: WFS get centered based on the direct reflection from MC1. Once the MC is close enough, the WFS are enabled, and they twiddle all 3 MC mirrors to minimize their error signal. Moral of the story: make sure the WFS are centered.
I've taken a bunch of transfer function measurements from the MC ASC PIT and YAW channels to the WFS error signals using the same set of DTT templates Koji used while characterizing the WFS loops a couple of months ago, before the IMC RF changes. Analysis is underway and I will post the results here shortly...
As an aside, Rana had added 10dB and 20dB gains to all of the WFS filter banks yesterday. I tried engaging the 10dB gains on the two MC2_TRANS PD loops, and this did not seem to induce any instability. I stepped both loops and saw that as expected, the 1/e times for both of these loops is about 45 seconds now (compared to ~150 seconds at the nominal gain). These have been running all day today, and the IMC seems well behaved, so I am going to leave these on for now... Jacking up the gain on the MC2_TRANS_QPD loops by 20dB induced instability - same story for the 4 WFS loops with 10dB additional gain...
Thanks to Koji's nice MATLAB script using DttData functions, I was able to quickly analyze the TF data. Essentially, this measurement was a repetition of what was done here. The difference is that the modulation depth has been increased by ~25x compared to that measurement from December 2016. Here are the measured TFs (before accounting for the 1/f^2 normalization) for the various quadrants and the PIT/YAW channels:
The plots above are just to illustrate that the measurement was clean between the band over which the averaging will be done to compute the TF amplitude - i.e. 7-15Hz. The full summary of TF amplitudes, standard deviations, and the sensing matrix in the style of the referenced elog (the actual excel spreadsheet is Attachment #4, minus some of the graphics Koji had on his excel sheet):
Inverting those matrices, we get the matrices that diagonalize the sensor-actuator chain:
I will try implementing these matrices tomorrow and take a look at the step responses of the loops - the idea is that perhaps the system wasn't optimally diagonalized before and perhaps we can now improve the bandwidths of all the loops.
For sensing matrix, better to use single frequency sine response. We don't want to measure around the bounce or above the 28 Hz cutoff filters in the MC SUS.