As I (very excitedly) reported in elog 11116, I was able to follow the error signal blending procedure from last night, and get CARM and DARM onto digital non-normalized RF signals. The lock held for about 3 minutes after this transition (elog 11118 has plot of this).
I was then able to script what I did (in the carm_up script), and repeat the transition. Q joined me in the control room, but we have not been able to complete the transition a third time.
Here's the sequence that worked the two times:
After those two attempts, we ran the LSC offset script, since that hadn't been done since early yesterday. We did a quick CARM sweep, and REFL11 seemed to be centered around 0 counts, so we removed the 0.025 count offset from the CARM_B filter bank.
For later attempts, we keep seeing oscillations in the lockloss plots around 50 Hz, as if we're seeing gain peaking at the low side of the phase bubble. We have tried turning off various filters at various levels of RF gain, but none of the combinations seems to be excellent. Turning off the FM6 bounce/roll filter in CARM was particularly bad (immediately lost arm transmitted power), but others weren't good either (eg turning off FM3 boost lost arm powers within a second or so). When we lose arm powers, the RF signals aren't valid, so if you don't turn them off fast enough (and ALS is still on with enough gain), you'll lose the full IFO lock. If you're fast though, you can turn off the CARM and DARM _B outputs and not have to start from scratch.
There seemed to be a very fine line to walk between not enough gain (~50Hz oscillations), and too much gain (200-300Hz oscillations). It has been pretty frustrating later in the evening. We seem to only have about 3dB of gain margin on the low side, when all the boosts are on. Not excellent.
When the RF signals had a moderate amount of gain, but ALS was still holding CARM and DARM, Q checked the phases of REFL11 and AS55 with excitation lines. He rotated AS55 from -55 deg to -30 deg (+25 deg) and REFL11 from 144 deg to 164 deg (+20 deg).
Prior to the all-digital attempts, I tried several times to turn on the AO path, without success. I think that the best that I got was 0dB on the CM board input 1 gain, +14dB on the CM board's AO gain, and -30dB on the MC board's AO gain before the mode cleaner lost lock.
I was hoping that I could get CARM entirely to RF signals, and that would make things more stable and less complicated, and I could try again to turn on the AO path, but we haven't been able to do this tonight.
A few times in the later attempts we tried turning on the UGF servos for CARM or DARM. I'm not sure if the lines kicked things out of lock, or if the UGF servos went a little crazy, or what, but we never survived for more than a few seconds after turning on the excitations.
There is a problem with the optical lever servos. I had thought I'd been seeing it ever since Q re-did the models, and now I'm pretty sure that's what's up. Q is hot on the trail of figuring out what may have changed that shouldn't have. We may want to revert to an old Foton file, and re-copy the old filters into the new filter banks just in case. The watchdog damprestore scripts have been tweaked to clear the oplev filter bank histories before turning on the oplevs, and this seems to solve the symptom of kicking the optic when oplevs are engaged.
Although we haven't been able to make the transition to RF-only a third time, I think we're getting there. Progress has certainly been made in the last 2 days!