It seems that the MC3 problem is intermittent (one-day trend attached). I tried to take advantage of a "clean MC3" night, but the watch script would usually fail at the transition to DC CARM and DARM. It got past this twice and then failed later, during powering up. I need to check the handoff.
Recently the watch script was having difficulty grabbing a lock for more than a few seconds. Rob discovered that the violin notch filters which were activated in the script were causing the instability. We're not sure why yet. The script seems significantly more stable with that step commented out.
We worked on tuning the DD handoff tonight. We checked the DD PD alignments and they looked fine. First I tuned the 3 demod phases to minimize offsets. Then I noticed that the post-handoff MICH xfer function needed an increase in gain to look like the pre-handoff xfer function (which has a UGF of about 25 Hz). I increased the MICH PD9_Q gain from 2 to 7 in the input matrix. But, the handoff to PRC still failed, so tomorrow we will try to find out why.
In the plot, ref0 is before MICH handoff, and ref1 is after MICH handoff. There is also a PRC trace (before PRC handooff).
Rana, Alberto, Pete
We have the DD handoff nominally working. Sometimes, increasing the SRC gain at the end makes MICH get unstable. This could be due to a non-diagonal term in the matrix, or possibly because the DRM locks in a funky mode sometimes.
To get the DD handoff working, first we tuned demod phases in order to zero the offsets in the PD signals handed-off-to. Based on transer function measurements, I set the PRC PD6_I element to 0.1, and set the PD8_I signal to 0, since it didn't seem to be contributing much. We also commented out the MICH gain increase at the end of the DD_handoff script.
It could still be more stable, but it seems to work most of the time.
I played with the DD handoff during the day. The DRM dark port was flickering like a candle flame in Dracula's castle. The demod offsets for the handoff signals looked fine. After MICH handoff, the MICH_CTRL started to get unstable at some low frequency, maybe 3 Hz (I didn't measure). So I increased the MICH gain from 0.1 to 0.17 and it settled down. PRC and SRC went fine. Then the DD_handoff script raised the MICH gain to 0.7, and an instability started to grow in MICH_CTRL (at some higher frequency). I decreased the MICH gain from 0.7 to 0.5, and it settled down and stayed stable.
We were stymied early in the evening by a surreptitiously placed, verbo-visually obfuscated command in the drstep script.
tdsavg 5 C1:LSC-PD4_DC_IN1
was causing grievous woe in the cm_step script. It turned out to fail intermittently at the command line, as did other LSC channels. (But non-LSC channels seem to be OK.) So we power cycled c1lsc (we couldn't ssh).
Then we noticed that computers were out of sync again (several timing fields said 16383 in the C0DAQ_RFMNETWORK screen). We restarted c1iscey, c1iscex, c1lsc, c1susvme1, and c1susvme2. The timing fields went back to 0. But the tdsavg command still intermittently said "ERROR: LDAQ - SendRequest - bad NDS status: 13".
The channel C1:LSC-SRM_OUT16 seems to work with tdsavg every time.
Let us know if you know how to fix this.
Did you try restarting the framebuilder?
What you type is in bold:
op440m> telnet fb40m 8087
After fixing the tp problem, I tried locking again. Grabbing and DD handoff, no problem. Died earlier than last night, handing off CARM to REFL_DC, around arm power of 4 or so. Seems to happen after turning off the moving zero, Rob says it might be touchy in daytime.
Lock acquisition is proceeding smoothly for the most part, but there is a very consistent failure point near the end of the cm_step script.
Near the end of the procedure, while in RF common mode, the sensing for the MCL path of the common mode servo is transitioned from a REFL 166I signal which comes into the LSC whitening board from the demodulator, to another copy of the signal which has passed through the common mode board, and is coming out of the Length output of the common mode board. We do this because the signal which comes through the CM board sees the switchable low-frequency boost filter, and so both paths of the CM servo (AO and MCL) can get that filter switched on at the same time.
The problem is occurring after this transition, which works reliably. However, when the script tries to remove the final CARM offset, and bring the offset to zero, lock is abruptly lost. DARM, CM, and the crossover all look stable, and no excess noise appears while looking at the DARM, CARM, MCF spectra. But lock is always lost right about the same offset.
I've seen this before. At that time, the problem was gone spontaneously the next day.
You could stop just before the offset reaches zero and then try to slowly reduce the offset manually to see where is the threshold.
Well, it hasn't gone away yet. It happened Sat, Mon, and Tues afternoon, as well as Friday. The threshold varies slightly, but is always around ~200-300 cnts. I've tried reducing the offset with the signal coming from the CM board and the signal not going through the CM board, I've also tried jumping the signal to zero (rather than a gradual reduction).
Tonight we'll measure the MC length and set the modulation frequencies, and maybe try some MZ tweaking to do RFAMMon minimization.
I had trouble getting the SRC handoff from SD to DD to work. I tried different gains, flipping the PD7 & 8 demod phases by 180 degrees, and messing with the output matrix to reduce cross-couplings in the state with MICH & PRC on DD and SRC on SD. Eventually I decided to try to make the DRM matrix diagonalization work.
It does, mostly. The handoff is now stable, and the loops all have UGFs around 100Hz. So, tonight anyways, it's possible to run senseDRM and then loadDRMImatrixData.m and run the resulting tdswrite command, and have a working handoff. I had to eliminate a few PDs (PD5 & PD10) to get it to work properly.
It would be nice if this script would measure all the PDs and allow individual setting of loop UGFs and measurement frequencies.
Lock is still being lost, right at the end of the process when trying to reduce the CARM offset to zero.
The IFO is locked, at the operating point (zero CARM offset). The problem with reducing the residual CARM offset in the last stage appears to have been because the common mode gain was getting too high, and so the loop was going unstable at high frequencies.
The cm_step script is currently a confusing mess, with all the debugging statements. I'll clean it up this afternoon and check that it still works.
Last night Rob ran senseDRM and loadDRMImatrixData and came up with the following for the input matrix:
tdswrite C1:LSC-ITMTRX_b2 0.065778 \
C1:LSC-ITMTRX_d2 2.2709 \
C1:LSC-ITMTRX_f2 2.9361 \
C1:LSC-ITMTRX_122 0.42826 \
C1:LSC-ITMTRX_b3 -0.064839 \
C1:LSC-ITMTRX_d3 -0.016913 \
C1:LSC-ITMTRX_f3 -0.021576 \
C1:LSC-ITMTRX_123 -0.0025243 \
C1:LSC-ITMTRX_b5 0.3719 \
C1:LSC-ITMTRX_d5 1.3109 \
C1:LSC-ITMTRX_f5 -0.16412 \
C1:LSC-ITMTRX_125 0.39574 \
C1:LSC-ITMTRX_33 0 \
C1:LSC-ITMTRX_42 0 \
Today, I reran these and got the following, and DD_handoff remained happy:
tdswrite C1:LSC-ITMTRX_b2 -0.10329 \
C1:LSC-ITMTRX_d2 2.0344 \
C1:LSC-ITMTRX_f2 3.2804 \
C1:LSC-ITMTRX_122 0.22516 \
C1:LSC-ITMTRX_b3 -0.076292 \
C1:LSC-ITMTRX_d3 -0.014603 \
C1:LSC-ITMTRX_f3 -0.12101 \
C1:LSC-ITMTRX_123 0.0054128 \
C1:LSC-ITMTRX_b5 0.33521 \
C1:LSC-ITMTRX_d5 1.1425 \
C1:LSC-ITMTRX_f5 -0.32759 \
C1:LSC-ITMTRX_125 0.25877 \
C1:LSC-ITMTRX_33 0 \
C1:LSC-ITMTRX_42 0 \
I wanted to remeasure with the canonical output matrix (-0.7 from MICH to PRM and 0.7 from MICH to SRM), but the DRM freaked out when MICH to PRM went below -0.3.
With the common mode servo bandwidth above 30kHz and the BOOST on (1), I was able to switch on the test mass dewhitening. Finally.
Here's a noise spectrum of the RSE interferometer, in anti-spring mode, with RF readout. I'd say the calibration is "loose."
I used the Buonanno & Chen modification of the KLMTV IFO transfer functions to model the DARM opto-mechanical response. I just guessed at the quadrature, and normalized the optical gain at the frequency of the calibration line used (927Hz, not visible on the plot).
Tonight I tried to lock the interferometer. At the first attempts the arm power didn't go above about 4. The mode cleaner seemed to be not well aligned and it lost lock or got stuck on a wrong mode. I had to run the MC_UP and MC_DOWN scripts to lock it again.
After that the locking proceed more smoothly; at least till a power level in the arms of about 60. Then again the mode cleaner lost lock and I had to run the scripts again. Without the MCWFS servo off the MC reflected power is still rather high (about 1.7); also even when the WFS servo is engaged the reflected power is about 0.5, versus 0.3 that it should be.
Those are both signs of a not very good alignment. Tomorrow I'll have to work on the injection periscope on the PSL table to try to fix that.
Since lately the alignment of the input beam to the interferometer has changed, I went checking the alignment of the beam on the photodiodea. They were all fine except for pd9, that is AS DD 199. Here the DC is totally null. The beam seems to go right on the diode but the scope on the PD's DC output shows no power. This is really strange and bad.
It's railed. This is what halted locking progess on Monday night, as this channel is used for the offloadMCF script, which slowly feeds back a CARM signal to the ETMs to prevent the VCO from saturating.
Attached is a 5 day trend, which shows that the channel went dead a few days ago. All the channels shown are being collected from the same ICS110B (I think), but only some are dead. It looks like they went dead around the time of the "All computers down" from Sunday.
After inspecting PD9 with the viewer and the cards, the beam looks like it is aligned to the photodiode althought there is no signal at the DC output of the photodetector. So I checked the spectrum for PD9_i and Q (see attachments) and it seems that those channels are actually seeing the beam. I'm going to check the alignemtn again and see the efefct on the spectra to make sure that the beam is really hitting the PD.
I aligned PD9. Here are the spectra confirming that.
Attached are the channels being recorded from the ICS110B in 1Y2 (the IOO rack). Channels 12, 13, 16, 17, 22, 24, 25 appear to have gone dead after the computer problems on Sunday.
This has been fixed by one of the two most powerful & useful IFO debugging techniques: rebooting. I keyed the crate in 1Y2.
I found the alignment biases for the PRM and the SRM in a funny state. It seemed like they had been "saved" in badly misaligned position, so the restore scripts on the IFO configure screen were not working. I've manually put them into a better alignment.
After the mini boot fest that Jenne did today, I checked whether that fixed the overflow issues we yesterday prevented the alignemnt of the arms.
I ran the alignment script for the arms getting 0.85 for TRX and 0.75 for TRY: low values.
After I ran the script ,C1SUSVME1 and C1SUSVME2 started having problems with the FE SYNC (counter at 16378). I rebooted those two and fix the sync problem but the transmitted powers didn't improve.
Are we still having problem due to MC misalignment?
I've now also trended the MOPA output power for the last 200 days to check a possible correlation with the FSS reflected power. See attachment.
The trend shows that the laser power has decayed but it seems that the FSS reflected power has done it even faster: 30% drop in the FSS vs 7% for the MOPA in the last 60 days (attachment n.2).
Here we trended also the PMC and the MZ. The drop in the PMC happens at the same rate as the MOPA's.
That let us think that the FSS transmitteed power has gone down because of the reference cavity progressive misalignment to the laser beam.
We need to adjust that alignment sometime.
The drop in the NPRO output power (upper row, 3rd plot: Ch10 C1:PSL_126MOPA_126MON) accompained an increase of "fuzziness" in PMCTRANSPD and both coincided in time with the day we tempoarirly removed the flap from the laser chiller's chiller (July 14 2009).
Spent a lot of time aligning tonight. The BS is not staying put--sometimes after a lock loss it gets badly mis-aligned.
DD handoff is working, after putting beam on REFL diodes and running senseDRM script.
It was fine when I came in earlier today, but I just got back from dinner, and it's not good. I looked in dataviewer, and it seems to have been sliding out for the past couple of hours... Here is a picture:
I swear I am not responsible this time... all I've been doing is working in the control room.
Mode cleaner bounced back on its own about 2 hours ago.
locking work proceeding apace tonight.
diagonalized DRM with setDDphases & senseDRM.
initial locks are fairly quick, aqstep script succeeds reliably.
first part of cm_step (handoff CARM-> MCL) usually works.
tuning up later parts of cm_step (presumably due to optical gain changes resulting from MOPA decline).
got to arm powers ~60.
Tried to lock the interferometer but arm power didn't get over 65.
Tonight, after the weekend, I resumed the work on locking.
When I started the Mode Cleaner was unlocked because the MZ was also unlocked.
I aligned the MZ and the transmitted power reached about 2.5
Initially the interferometer lost lock at arm power of about 3-4. It looked like the alignment wasn't good enough. So I ran the alignment scripts a few times, first the scripts for the single parts and in the end the one for the full IFO.
Then I also locked again the MZ and this time the transmitted power got to about 4.
In the following locking attempts the the arm power reached 65 but then the lock got lost during the handing of CARM to C1:LSC-PD11_I
I'll keep working on that tomorrow night.
While trying to make the OAF work, I found that the XYCOM switches for MC1/3 have been set in the bad way for awhile. This means that the hardware filters were bypassed and that MC1 & MC3 were moving around too much at high frequency and possibly causing trouble with the locking. I have put them back into the default position.
On Friday, Jenne and I were playing around with turning the dewhitening off/on to see if it efffected the OAF stability. At the time, I didn't pay too much attention to what the state was. Looks like it was in the wrong state (hardware bypassed) when we found it. For the OAF work, we generally want it in that bypassed state, but its bad because it makes noise in the interferometer. The bits in question are bits 16-23 on the XYCOM screen.
I have updated the snapshot and set the screen in the appropriate settings. I used a swept sine measurement to verify the filter state. In the attached plot, green corresponds to XYCOM green and red corresponds to red.
It's been a miserable week for lock acquisition, with each night worst than the last. The nadir was around Sunday night, when I couldn't even get a PRM to lock stably, which meant that the auto-alignment scripts could not finish successfully. It now appears that was due to some XYCOM mis-settings.
We've also been having problems with timing for c1susvme2. Attached is a one-hour plot of timing data for this cpu, known as SRM. Each spike is an instance of lateness, and a potential cause of lock loss. This has been going on for a quite a while.
Tonight we also encountered a large peak in the frequency noise around 485 Hz. Changing the MZ lock point (the spot in the PZT range) solved this.
Good progress in IFO locking tonight, with the arm powers reaching about half the full resonant maximum.
Still to do is check out some weirdness with the OMC DAC, fix the wireless network, and look at c1susvme2 timing.
Attached is a 3 day trend of SRM CPU timing info. It clearly gets better (though still problematic) at some point, but I don't know why as it doesn't correspond with any work done. I've labeled a reboot, which was done to try to clear out the timing issues. It can also be seen that it gets worse during locking work, but maybe that's a coincidence.
More progress with locking tonight, with initial acquisition and power ramps working. The final handoff to RF CARM still needs work.
I found the wireless router was unplugged from the network--just plugging in the cable solved the problem. For some reason that RJ45 connector doesn't actually latch, so the cable is prone to slipping out of the jack.
Late last night after the ETMY settled down from the quake I made some more progress in locking, with the handoff to RF CARM succeeding once. The final reduction of the CARM offset to zero didn't work, however.
Working well tonight: the handoff of CARM to RF (REFL2I), successful reduction of CARM offset to zero, and transition control of MCL path to the OUT1 from the common mode board. All that's left in lock acquisition is to try and get the common mode bandwidth up and the boost on.
Lock acquisition working well tonight. Was able to engage CM boost (not superboost) with bandwidth of ~10kHz. Also succeeded once in handing off DARM to DC readout.
Last night, 2+ hour lock, probably broken by me driving too hard (DARM_EXC).
The IFO can now be locked during the daytime. Well, it's locked now.
This is huge. Five hours of lock only interrupted by intentional break from transfer function abuse.