Some of these channels are not like the others.
The set of 6 accelerometers which were semi-randomly placed underneath the MC2 tank are now back into 2 separate sets of 3 - one set at MC2 and one set at MC1. The channel names once again reflect reality, i.e. MC1_Y is actually under the MC1 tank, and aligned with the y direction. Also, the Guralp under MC1 was moved a little bit to the left, because Sanjit wanted to put the accelerometers where the seismometer had been.
Last night while noise hunting, Rana found that the MC2 trans beam has been left for an unknown length of time just hitting the side of the black enclosure box. Today I put a brand-spankin'-new razor dump on the MC2 table, to dump the beam.
I used the XARM as a reference to measure the frequency noise after the MC. It's huge around 4kHz--hundreds of times larger than the frequency noise the MC servo is actually squashing. This presents a real problem for our noise performance.
An elog search reveals that this noise has been present (although not calibrated till now) for years. We're not sure what's causing it, but suspicion falls on the piezojena input PZTs.
I didn't bother too much about it before because we previously had enough common mode servo oomph to squash it below other DARM noises, and I didn't worry too much about stuff at 4kHz.. Now that we have a weaker FSS and thus much weaker CM servo, we can't squash it, and the most interesting feature of our IFO is at 4kHz.
I'll measure the actual voltage noise going to the PZTs. I remember doing this before and concluding it was ok, but can't find an elog entry. So this time maybe I'll do it right.
"Yes. This master file is used."
Does anyone know if this master file is the real thing that's in use now? Are we really using a file called tpchn_C1_new.par? If anyone sees Alex, please get to the bottom of this.
The c1ass computer, which is now used for the OAF system, has many remnants from the days when it was actually used as an ASS. These PIT and YAW filter banks and other things were taking up a lot of unnecessary space, so I deleted them in the ass.mdl file. These files are all backed up, so we can always revert back to an older version when we want some Alignment Stabilization again someday. I then did a make ass, following the instructions on the 40m Wiki -> Computers and Scripts -> Simulink to Front-End Code page. Rana moved some things around, most notably all of the things (like the ASS screens) which were only in ...../users/alex/.... are now in ....../caltech/cds/advLigo/..... . This required a few restarts of the c1ass machine (after a couple different versions of the simulink diagram....one to make sure we knew how to do it, and then again actually deleting the unused portions).
The big lesson of the night was that there are 2 signal paths for the PEM channels. As is shown in Figure 3 in the mevans document, the PEM channels get the matching filters when they go to the adaptation algorithm, but when they go to the FIR filter, they do not get the matching filters. This is implemented by taking the output of the giant PEM matrix, and having a duplicate of each of the channels "selected for adaptation", one which gets filtered through the PEM_N_ADPT banks, and one which goes straight (in code-land) to the FIR filter. So, it seems like all the filters which we had been including in the input side of the matrix for matching purposes need to be put in the output side. One of the AA32 filters needs to stay in the input side, for actual anti imaging of the PEM channels, then we put the AA32 and AI32 which are for matching the ERR_EMPH and CORR filter banks up in the PEM_N_ADAPT banks. Rana and I made these filters, and they are now turned on appropriately with the OAF down script (so that all the filters are ready and waiting for the OAF to be turned on).
A little success with getting the 3Hz peak reduced, but not a lot beyond that. Tomorrow I'll put the accelerometers back where they used to be to see if they help out at all.
I changed the input channels of the DMF recently so that it now uses 3 Guralp channels in addition to the 3 ACC and 1 Ranger.
op440m:seisblrms>diff seisBLRMS-datachans.txt~ seisBLRMS-datachans.txt
The seisBLRMS channels still have the wrong names of IX and EX, but I have chosen to keep them like this so that we have a long trend. When looking at the historical seisBLRMS trend, we just have to remember that all of the sensors have been around the MC since last summer.
I looked at the data of the day before yesterday (Oct 06) to know how much is the recycling gain.
X arm: (TRX_PRecycled) / (TRX_PRMmisaligned) * T_PRM = 83.1/0.943*0.07 = 6.17
Y arm: (TRX_PRecycled) / (TRX_PRMmisaligned) * T_PRM = 99.2/1.017*0.07 = 6.83
==> G_PR = 6.5 +/- 0.5 (oh...this estimation is so bad...)
From the recycling gain and the arm cavity reflectance, one can get the loss in the recycling cavity.
G_PR = T_PRM / (1-Sqrt(R_PRM * (1-L_PRC)*R_cav))^2
==> loss in the recycling cavity L_PRC: 0.009+/-0.009
(About 1% loss is likely in the recycling cavity)
Measured arm reflectivity R_cav: 0.875 +/- 0.005
Estimated round trip loss L_RT: 157ppm +/- 8ppm
Estimated finesse F: 1213+/-2
Measured arm reflectivity R_cav: 0.869 +/- 0.006
Estimated round trip loss L_RT: 166ppm +/- 8ppm
Estimated finesse F: 1211+/-2
Last night (Oct 07), I ran armLoss script in order to obtain the latest numbers for the arm cavity loss.
Here is the summary
Measured arm reflectivity R_cav: 0.875 +/- 0.005
Estimated round trip loss L_RT: 157ppm +/- 8ppm
Estimated finesse F: 1213+/-2
Data Points: 34
Measured arm reflectivity R_cav: 0.869 +/- 0.006
Estimated round trip loss L_RT: 166ppm +/- 8ppm
Estimated finesse F: 1211+/-2
Data Points: 26
TE=10ppm, LE=L_RT/2, RE=1-TE-LE
TF=0.005, LF=L_RT/2, RF=1-TF-LF
rcav = -rF +(tF^2 rE)/(1-rF rE)
R_cav = rcav^2
F = pi Sqrt(rF rE)/(1-rF rE)
Power cycling c1dcuepics seems to have fixed the EPICs channel problems, and c1lsc, c1asc, and c1iovme are talking again.
I burt restored c1iscepics and c1Iosepics from the snapshot at 6 am this morning.
However, c1susvme1 never came back after the last power cycle of its crate that it shared with c1susvme2. I connected a monitor and keyboard per the reboot instructions. I hit ctrl-x, and it proceeded to boot, however, it displays that there's a media error, PXE-E61, suggests testing the cable, and only offers an option to reboot. From a cursory inspection of the front, the cables seem to look okay. Also, this machine had eventually come back after the first power cycle and I'm pretty sure no cables were moved in between.
I had a go at trying to bring c1susvme1 back online. The first few times I hit the physical reset button, I saw the same error that Joe mentioned, about needing to check some cables. I tried one round of rebooting c1sosvme, c1susvme2 and c1susvme1, with no success. After a few iterations of jiggle cables/reset button/ctrl-x on c1susvme1, it came back. I ran the startup.cmd script, and re-enabled the suspensions, and Mode Cleaner is now locked. So, all systems are back online, and I'm crossing my fingers and toes that they stay that way, at least for a little while.
At around 9:45 the RFM/FB network alarm went off, and I found c1asc, c1lsc, and c1iovme not responding.
I went out to hard restart them, and also c1susvme1 and c1susvme2 after Jenne suggested that.
c1lsc seemed to have a promising come back initially, but not really. I was able to ssh in and run the start command. The green light under c1asc on the RFMNETWORK status page lit, but the reset and CPU usage information is still white, as if its not connected. If I try to load an LSC channel, say like PD5_DC monitor, as a testpoint in DTT it works fine, but the 16 Hz monitor version for EPICs is dead. The fact that we were able to ssh into it means the network is working at least somewhat.
I had to reboot c1asc multiple times (3 times total), waiting a full minute on the last power cycle, before being able to telnet in. Once I was able to get in, I restarted the startup.cmd, which did set the DAQ-STATUS to green for c1asc, but its having the same lack of communication as c1lsc with EPICs.
c1iovme was rebooted, was able to telnet in, and started the startup.cmd. The status light went green, but still no epics updates.
The crate containing c1susvme1 and c1susvme2 was power cycled. We were able to ssh into c1susvme1 and restart it, and it came back fully. Status light, cpu load and channels working. However I c1susvme2 was still having problems, so I power cycled the crate again. This time c1susvme2 came back, status light lit green, and its channels started updating.
At this point, lacking any better ideas, I'm going to do a full reboot, cycling c1dcuepics and proceeding through the restart procedures.
There is some craziness going on with the delay in the PEM path for the OAF. We plot the difference between the C1:PEM-SEIS_GUR1_X and C1:ASS-TOP_PEM_10. These are physically the same channel, plugged into the PEM ADCU, and then the signal is used as a regular PEM channel, and is also sent to the ASS computer and used there for the OAF system. As you can see in the blue trace on the bottom plot, there is a huge amount of delay, and it's very noisy. We also plot the _GUR2_X / ASS-TOP_PEM_2 pair (red), and it has a similar amount of delay, but it is not nearly as fuzzy and noisy. For comparison, we plot the SUS-MC2_MCL (which is identical to IOO-MC_L) and ASS-TOP_ERR_MCL pair (green), and they don't have any big overall delay problems, so it's not totally a problem with the signals getting to the ASS computer.
This problem was present during/after all of the following attempts to fix it:
* The sample rate on the ASS computer is 2048. The PEM channels were being acquired the ADCU at 512. We changed the ADCU sampling rate to 2048 to match.
* We soft rebooted the ASS computer, in case it was a timing problem.
* Doing a "sudo shutdown -r now" while logged in as controls.
We might also try resetting/power cycling c0dcu in the morning. Alex has been emailed to help us try to figure this out.
In other news, the time delay that we measure from the plot gives us 180degrees in ~210Hz. This corresponds to a little more than 2msec of delay, with the C1:ASS version lagging behind the C1:PEM version. (2 samples at 840Hz) Converting to the 2048 sampling rate, we have a delay of 4.8, so 5 front-end cycles. Since Rana measured this morning that the delay indicated by the transfer function is 10 cycles, and this delay shows that the ASS lags the actual seismometer signal by 5 cycles, we should subtract this 5 from the 10 from the transfer function, giving us a final sample-and-hold delay of 5. Coincidentally(?), 5 is the delay that was found in the C1:ASS-TOP screen, after it's one year of dormancy. The point of the delay feature in the code is to help match the delay in the two signal paths: the PEM path and the output path of the filter. Since the output has a lag of 10, and the PEM path has a lag of 5, to make them match, we artificially put in a delay of 5.
Here's a plot of the spectra of the seismometers and MCL. The coherence shows which axes are aligned right now: MC1_X is coherent with GUR_NS which means that its mis-oriented.
I've now swapped the "MC1" cables: so the old "NS" now goes into EW and the old EW now goes into NS. VERT is unchanged.
Also fixed the channel names - the Guralp previously named MC1 is now GUR1 and the other one is GUR2. Also no more EW, NS, & VERT. Its all XYZ.
DAQD restarted with the new channel names.
I spiffed up the order of the cables / sensors plugged into the PEM ADCU. Now all of the seismometers are labeled as Rana left them, and the 2 Guralp's have their sets of 3 channels next to eachother in channel-number-land. None of the accelerometer names/cabling have changed recently. In the table, Cable-label refers to the physical tag tied to the end of the cables plugged into the ADCU...they are meant to be descriptive of what seismometer channels they are hooked up to, and then the names change to something useful for us when they come into the DAQ system. Also, the labels of input channels on the ASS_TOP_PEM screen have been updated accordingly.
I took the pictures of all racks of electronics yesterday, and then uploaded these pictures on the wiki.
You can see them by clicking "pictures" in the wiki page.
I remeasured the OAF time delay using the OAF-TF template from the Templates/ directory.
Troublingly, I found the MC1 dewhitening switches set OFF - please make sure that the MC1 dewhitening is back ON after each OAF tuning so that the interferometer locking is not hosed.
The OAF-TF template had the excitation amplitude set ~20x too high. I reduced it and the coherence was still > 0.95. The phase at 32 Hz was still ~126 deg as Jenne had measured, but since the phase at DC is 180 deg, the overall phase lag is just 180-126 = 54 deg. So the delay should be 54/180 * 32 = 9.7 => 10. Luckily, Jenne is working on an instructional manual for OAF that will make all of this crystal clear.
I drove MC2 in POS and used the resulting response in MC_F to calibrate the IOO-MC_L channel.
Yoichi did an excellent job of calibrating MC_F last year. I have used his calibration of MC_F (220 Hz/count @ DC) to get the MC_L calibration at DC as well as at high frequencies. The hardware dewhitening was OFF for all these measurements.
1. For the DC measurement I excited C1:SUS-MC2_MCL_EXC at 0.0731 Hz. At these frequencies, the MC_L path has much more gain than the MC_F path. So this excitation at the error point makes the length path to drive itself to cancel the digital excitation. Since the overall MC servo gain is huge, this causes the MC_F path to compensate the residual MC_L motion. One can simply take the ratio of MC_L/MC_F to get the calibration of MC_L in Hz.
2. For the AC measurement, I engaged FM9 of the MC2_MCL filter bank. This guy is an elliptic LP with a notch at 660.38 Hz. I then drove MC2_LSC at 660.38 Hz with a sine wave of 500 counts amplitude. The notch makes the gain of the MC_L feedback zero at that frequency. So MC_F has to do all the work. We can simply measure the ratio of MC2_LSC/MC_F to get the AC calibration of MC2_MCL_OUT (aka IOO-MC_L) and MC2_LSC_OUT (aka LSC-MC_L).
MCF/MCL @ 0.0731 Hz = 569.4. So the MC_L calibration at DC is 220 x 569.4 = 125 kHz/count or 6.02 nm/count.
From this we would expect the AC calibration to be (6 nm/count)*(660.38/f_pend)^2 = 13.0 x10^-15 m/count.
The AC measurement gave 1445 counts_peak** of MC_F for the 500 counts (peak) excitation in MC2_LSC. From Yoichi's entry we get that the high frequency calibration of MC_F should be 0.089 Hz/count. So the MC_L calibration at 660 Hz is 0.089*1445/500 = 0.25 Hz / count or 12.3 x 10^-15 m/count. So the AC/DC ratio is close to 1.
Splitting the difference, the new official MC_L calibration is 5.87 nm/counts @ DC with a complex pole pair at 0.972 Hz.
** note: To convert from the peak height observed in DTT with a 50% Overlap Hanning window you must use the following intuitive formula: counts_peak = (counts / rHz) * sqrt(2 * BW). In this case, BW is the number that DTT reports as BW on the screen, NOT the BW that you asked for on the measurement tab.
*** note: when measuring peak heights in a DTT FFT, make sure to unclick the box for 'Bin' under the config tab. Bin suppresses peaks in a plot with a lot of points and is ON by default.
**** note: I have updated the MCF reference in the Templates directory with the Yoichi calibration - spectrum attached. This is probably the most accurate MCF spectrum we have ever put in the elog in the history of the 40m. The implication is that the VCO phase noise is ~5 mHz/rHz. Not bad.
***** note: with the OAF off, I drove a 1.55 Hz sine wave into MC1 and measured the ratio of MC1_MCL_OUT/IOO-MC_L. This gives the DC calibration of MC1_MCL_OUT = 7.98 nm/count.
I propose that anyone who tries to do this kind of thoroughgoing cleaning should make an e-mail to call everyone available to join just for some hours
because every member has a responsibility to keep the lab organized.
And we have the list of things to do: Electronics (now it is halfway) / Cables / Optics / Screws / Tools ...
I spent part of the afternoon cleaning up the area next to the Mode Cleaner where we keep all of our RF stuff: Attenuators, BNC/SMA/LEMO adapters, Mini-Circuits items, and all sorts of other things which are useful while looking at our electronics/RF stuff.
We got another set of "Lyon" drawers, which aided in the organization process....Bob ordered 2, so we now have a 'spare' drawer set if anyone can think of something else to organize (unless this was premeditated for optics or something else?).
As you can see in the picture, (1) it's no longer a total disaster over there, and (2) some of the drawers have sub-divisions to make it faster and easier to find what you're looking for. Please help out by putting things away in their proper place, and adding more labels or dividers to the drawers if there's something else which needs a 'spot'.
As per Matt's instructions in his OAF document (elog 395) in the Tuning section, Sanjit and I took a transfer function measurement from the output of the OAF system, to the input. i.e. we're trying to measure what happens out in the real world when we push on MC1, and how that is fed back to the input of our filter as MC_L. The game plan is to measure this transfer function, and read off the phase at the nyquist frequency, and use this value to calculate the appropriate sample-and-hold delay to be used in the OAF. The downsample rate for the OAF is 32, so that we're running at 64Hz instead of the 2048Hz of the front-end. Thus, our Nyquist frequency is 32Hz.
Phase@Nyquist * ------------------------ = Delay
In the attached figure we do a swept sine from CORR_EXC to ERR_EMPH_OUT to determine the transfer function. Here, we turn off all of the filters in both the CORR and EXC banks, because those are already matched/taken into account in the PEM filter banks.
Using the cursor on DTT, we find that the phase at 29.85Hz is -228.8deg, and at 37.06Hz is -246.0deg. Extrapolating, this means that at 32Hz, we expect about -234deg phase. Using our handy-dandy formula, this means that we should try a delay of 41 or 42 (41.6 is between these two...)
We'll give this a shot!
As Rana pointed out to me last night, I was using continuous phase, which is not good. When using regular phase, I find: (29.85Hz, 131.216deg), (37.06Hz, 113.963deg), so extrapolating gives (32Hz, 126.07deg). Plugging this in to our handy-dandy formula, we get a delay of 22.4, so we should try both 22 and 23.
I'm back to terrorize the PSL table again. The pictures I took yesterday were rubbish--today I'm using a clamp that Steve was nice enough to loan me. I'm starting now, at 10:09 am.
Lock acquisition working well tonight. Was able to engage CM boost (not superboost) with bandwidth of ~10kHz. Also succeeded once in handing off DARM to DC readout.
I set the FSS slow actuator adj to -5.6 at the lunch time. It gave a little help at that time. Now max of the MC Trans is comming back somehow. I hope the MC Trans level is as good as before, if the HEPA is slowed down.
The PSL output looks smaller than the incident. Try to FSS Slow actuator adj of -5.6 (nominal), instead of -3.5.
While I was moving a cart near by the PSL table I pushed the red emergency button that turns off the PSL laser. We had to unlock the button and then power cycle the laser driver to turn the laser back on.
I relocked MZ, FSS, PMC and I'm now waiting for the power to finish ramping up back to the previous value.
Phase@Nyquist * ------------------------
In the revival of the experiement length measurement for the recycling cavities, I turned the auxiliary NPRO back on. The shutter is closed.
I also recollected all the equipment of the experiment after that during the summer it had been scattered around the lab to be used for other purposes (Joe and Zach's cameras and Stephanie and Koji's work with the new EOM).
I have been commissioned to take pictures of the PSL table so that it can be diagrammed. I am starting now (1:42 pm, 10/5/09).
All done (for now). That wasn't so bad, was it?
Working well tonight: the handoff of CARM to RF (REFL2I), successful reduction of CARM offset to zero, and transition control of MCL path to the OUT1 from the common mode board. All that's left in lock acquisition is to try and get the common mode bandwidth up and the boost on.
Late last night after the ETMY settled down from the quake I made some more progress in locking, with the handoff to RF CARM succeeding once. The final reduction of the CARM offset to zero didn't work, however.
quake coming through. I've re-enabled optic damping (except ETMY), and left off the oplevs for now. We can do a resonant-f check over the weekend.
Looks like it was a magnitude 5 near Olancha, where they sell really good fresh jerky. quake
latest news: there's actually been about a dozen earthquakes in Keeler in the last couple hours: http://earthquake.usgs.gov/eqcenter/recenteqsus/Maps/special/California_Nevada_eqs.php
I took the output of the OMC DAC and plugged it directly into an OMC ADC channel to see if I could isolate the OMC DAC weirdness I'd been seeing. It looks like it may have something to do with DTT specifically.
Attachment 1 is a DTT transfer function of a BNC cable and some connectors (plus of course the AI and AA filters in the OMC system). It looks like this on both linux and solaris.
Attachment 2 is a transfer function using sweepTDS (in mDV), which uses TDS tools as the driver for interfacing with testpoints and DAQ channels.
Attachment 3 is a triggered time series, taken with DTT, of the same channels as used in the transfer functions, during a transfer function. I think this shows that the problem lies not with awg or tpman, but with how DTT is computing transfer functions.
I've tried soft reboots of the c1omc, which didn't work. Since the TDS version appears to work, I suspect the problem may actually be with DTT.
Here's the gist of the requirements on the 5x frequency multiplier for the upgrade (see attachemnt - despite the preview down here in the elog, they're 3 pages).
An extended version is available on the wiki.
A more complete document is under work and will soon be available on the wiki as well.
we set the offsets for the MCWFS DC and for the MCWFS demod outputs and then turned off the lights put the MZ at half fringe and then centered the spots on the MCWFS heads.
The MCREFL beam looks symmetric again and the MC REFL power is low.
It got worse again, starting with locking last night, but it has not recovered. Attached is a 3-day trend of SRM cpu load showing the good spell.
The attached shows the 200 day '10-minute' trend of the CPU meters and also the room temperature.
To my eye there is no correlation between the signals. Its clear that c1susvme2 (SRM LOAD) is going up and no evidence that its temperature.
More progress with locking tonight, with initial acquisition and power ramps working. The final handoff to RF CARM still needs work.
I found the wireless router was unplugged from the network--just plugging in the cable solved the problem. For some reason that RJ45 connector doesn't actually latch, so the cable is prone to slipping out of the jack.
Ops. I restored the damping of the suspensions at around 16:30.
The EQ did not change the input beam pointing. All back to normal, except MC2 wachdogs tripped again.
Round 3 for the day of MC2 watchdogs tripping.
I've watchdogged all the suspensions while I mess around with computers. If no one else is using the IFO, we can leave them undamped for a couple of hours to check the resonant frequencies, as long as I don't interrupt data streams with my computer hatcheting.
MZ work has been done. I did not change anything on the circuit.
Recently we observed that the MZ PZT output was sticking at a certain voltage. I found the reason.
Shortly to say "we must return the PZT Ramp offset to 0, after the lock"
I am going to write a MZ auto lock script someday, to do it automatically.
According to the resister values used in the circuit, the PZT HV output voltage is determined by the following formula:
V_PZT = 150 - 12 V_ctrl - 24 Vramp
Here the ramp voltage Vramp moves from -10V to +10V, the feedback control voltage V_ctrl moves from -13V to +13V.
The baseline offset of 150V is provided in the circuit board.
When V_ramp is 0, V_PZT runs from 0 to 300. This is just enough for the full scale of the actual V_PZT range,
that is 0V~280V.
If any Vramp offset is given, V_PZT rails at either side of the edges. This limits the actual range of the PZT out.
This is not nice, but is what happened recently.
I will investigate the MZ board. I will unlock MZ (and MC).
We've also been having problems with timing for c1susvme2. Attached is a one-hour plot of timing data for this cpu, known as SRM. Each spike is an instance of lateness, and a potential cause of lock loss. This has been going on for a quite a while.
Attached is a 3 day trend of SRM CPU timing info. It clearly gets better (though still problematic) at some point, but I don't know why as it doesn't correspond with any work done. I've labeled a reboot, which was done to try to clear out the timing issues. It can also be seen that it gets worse during locking work, but maybe that's a coincidence.
MZ stayed unlocked. Now It was relocked.
Earthquake of magnitude 5.0 shakes ETMY loose.
MC2 lost it's damping later.
Good progress in IFO locking tonight, with the arm powers reaching about half the full resonant maximum.
Still to do is check out some weirdness with the OMC DAC, fix the wireless network, and look at c1susvme2 timing.