There were a bunch of medm processes stalled on megatron (connected with screenshot taking). To see if they were interfering with the other scripts, I killed all of the medm processes, and commented out the line in the crontab that runs the screenshots every 10 mins. Let's see if this improves stability.
With all of the shaking (man-made and divine), it was a hard to debug this problem. Summary of fixes:
At least the DC indicators are telling me that the IMC locking is back to a somewhat stable state. I have not yet checked the frequency noise / RIN.
We found that the caput commands were taking much longer to execute on megatron than on pianosa (for example). Suspecting that this had something to do with the fact that megatron was using EPICS binaries from the shared NFS drive which were compiled for a much older OS, I installed the latest stable release of EPICS on megatron. The new caput commands execute much faster. I also added the local EPICS directory to the head of the $PATH variable used by the MC autolocker and FSS Slow scripts, so that they use the new caput command. But mcup is still slow - maybe my new path definition isn't picked up and it is still using the NFS binaries? To be looked into...
To fix the apparent slowness of execution of the caput commands on megatron, I changed the "ewrite" macro in the mcup and mcdown scripts to use ezcawrite instead of caput. The old lines are simply commented out, and can be reverted to at any point if we so desire. After these changes, we saw that both scripts complete execution much faster.
The goal tonight was to go through the locking scripts to see if I could recover the state from November 2019, when I could have the arm lengths controlled by ALS, and sit at zero CARM offset with the PRMI remaining locked and the arm powers fluctuating between 0-300. The IR-->ALS transitions went smoothly tonight, and the PRMI locking was also fairly robust when the CARM offset was large, but was less good when reduced to 0. Nevertheless, it is good to know that the system can be restored to the state it was late last year. Next step is to figure out how to keep the PRMI locked and get the AO path engaged, this was what I was struggling with before the new year.
You can trend the data for the past few hours and see what the appropriate value. I think these tests should only be done when whoever is running a test is in the lab.
P.S. I was surprised that the IMC didn't lose lock when this step was applied. I manually stepped this voltage between +/- 10 V and didn't see any response in the FSS readbacks. Either the channel doesn't work, or there is a divide by 40 in the physical circuit or something...
A script I was testing errantly set C1:PSL-FSS_INOFFSET => 10 V at about 5:30 pm. I manually reverted the channel value to 0, but I don't know what the value was initially. Someone please check this value if there are problems locking the FSS.
One factor which hampers locking efforts is the apparent drift of the input beam into the IFO. Over timescales of ~1 hour, I have noticed that the spot on the AS camera drifts significantly (~1 spot size) in pitch. The IPPOS QPD bears out this observation, see Attachment #1. The IMC WFS control signals do not show a correlated drift, hence my claim that the TTs are to blame.
I am able to correct this misalignment by moving TT1 in pitch (see Attachment #2, which shows some signals from a ~1 hour PRMI lock, during which time the pointing drifted, and I corrected it by moving TT1 pitch). Assuming the problem is purely TT1 pitch drifting, this corresponds to 3mm / 6m ~500urad of shift in 1 hour - seems very large. The fact that the drift is only present in pitch and doesn't really show up in yaw makes me think the problem is likely mechanical (unless the voltage to the top two coils is drifting relative to the bottom, but no LR drift, which would be very coincidental). At the moment, this is just an annoyance, but it'd be good for this problem to be fixed.
I haven't yet figured out how to make ndscope export these plots to SVG preserving the dark color theme, hence the weird light axes...
Jon brought over a box of parts for constructing the metal PMCs. I have stored it along the Y-arm, on top of the green optics cabinet.
I didn't do an exhaustive inventory check, but the following are the rough contents of the box:
I didn't inspect the optics but since we have so many, I am hoping we can find 3 good quality ones for one cavity at least. We should check that the geometry is suitable for our RF sideband frequencies.
The CARM-->RF transition remains out of reach. No systematic diagnosis scheme comes to mind.
TBC. Mercifully at least the shaker stayed still tonight.
The goal is to try and identify the source of the excess ALS noise as the CARM offset is reduced. The idea is to look at the MC_F spectrum (or the IMC error point) in a few conditions:
#1 vs #2 is like a control experiment, we expect to see the excess noise imprinted on the MC length and hence in MC_F (provided the sensing noise is low enough). #2 vs #3 will be informative of something like backscatter to the PSL increasing the frequency noise. #2/3 vs #4 will help isolate the problem to an individual arm's AUX PDH loop or some optomechanical effect.
I was looking back at some spectra from the last couple of nights but I don't really have an apple-to-apple comparison in the various actuation schemes (some ALS loops were engaged/disengaged), so I'll do a more systematic test tonight. Already, it looks like MC_F is not a good candidate to look for the excess frequency noise, I don't really see a big difference between conditions #1 and #2. According to this, we are looking for an increase at the level of a few 100Hz/rtHz @ ~40 Hz, wheras MC_F is much noisier.
I did some more detailed tests to see if I could isolate where the excess ALS noise at low CARM offset is coming from, by measuring the spectrum of the IMC error point (in loop). The results, shown in Attachment #1 and #2, are inconclusive.
Since MC_F didn't show any signatures of elevated noise, I decided to hook up an SR785 to the A excitation bank TEST1 input of the IMC servo board to monitor the in-loop error signal. I initially took a few measurements spanning 800 Hz in frequency, and to my surprise, I found that there was elevated noise in the frequency band we see an increase in the ALS noise, even when the CARM feedback goes to the ETMs (so the IMC cavity is in principle isolated from the main interferometer). This is Attachment #1. So I re-took a couple of measurements (this time only for the case of CARM feedback to the ETMs), with a 200 Hz frequency span, and found no significant noise elevation. This is Attachment #2. I am led to conclude that the IMC error point level changes over time for reasons other than the CARM offset - it'd be nice to have a spectrogram of the IMC error point and compare excursions relative to the median level over a few 10s of minutes, but we don't have this data stream digitized by the CDS system - maybe I will hijack the MC_L channel temporarily to record this data stream. It seems a waste that we're not able to take full advantage of the measured <10pm RMS noise of the IR ALS system.
I managed to partially stabilize the arm citculating powers - they stay in a region in which the REFL 11 signal is hopefully approximately linear and so I can now measure some loop TFs and tweak the transition appropriately.
The main change I made tonight was to look at the REFL11 signal as I swept the ALS CARM offset through 0. I found that the maximum arm powers coincided with a non-zero REFL11 signal value (i.e. a small CARM offset was required at the input to the CARM_B filter bank). Not so long ago, I had measured the PM/AM ratio for 11 MHz to be ~10^5 - so it's not entirely clear to me where this offset is coming from. Then, I was able to turn on the integrator (z:p = 20:0) in the CARM_B filter bank while maintaining high POP_DC. At this point, I ramped up the IN2 gain on the IMC servo board (= AO path), and was able to further stabilize the power.
Attachment #1 shows this sequence from earlier in the evening. Note that in this state, both ALS and IR control of CARM is in effect. The circulating power is fluctuating wildly - partly this is probably the noisy ALS control path, but there is also the issue of the (lack of) angular control - although looking at the transmon QPDs and the POP QPD signals, they seem pretty stable.
The next step will be to try and turn off the ALS control path. Eventually, I hope to transition DARM control to AS55 as well. But at this point, I can at least begin to make sense of some of the time series signals, and get some insight into how to improve the lock.
No systematic diagnosis scheme comes to mind.
Plots + interpretation tomorrow.
Getting closer... To facilitate this work, I made some convenience scripts that can be run from the CM MEDM screen.
To study the evilution of the AO path TFs a bit more, I've hooked up POY11_Q Mon to IN1 of the CM board. I will revert the usual setup later in the evening.
Update 1730: I've returned the cabling at 1Y2 to the nominal config, and also reverted all EPICS settings that I modified for this test. Y-arm POY locking works. Attachment #1 shows the summary of the results of this test - note that the AO gain was kept fixed at +5dB throughout the test. I have arbitrarily trimmed the length of the frequency vector for some of these traces so that the noisy measurement doesn't impede visual interpretation of the plots so much. At first glance, the performance is as advertised. I basically followed the settings I had here to get started, and then ramped up various gains to check if the measured OLTF evolved in the way that I expected it to. The phase lead due to the AO path is clearly visible.
Some important differences between this test and the REFL11 blending is (i) in the latter case, there will also be a parallel loop, CARM_A, which is effecting some control, and (ii) the optical gain of CARM-->REFL11_I is much higher than L_Y-->POY. So the initial gain settings will have to be different. But I hope to get some insight into what the correct settings should be from this test. I think IMC servo IN2 gain and AO gain slider on the CM board are degenerate in the effect they have, modulo subtle effects like saturation.
One possibility is that the gain allocation I used yesterday was wrong for the dynamic range of the CARM error signal. In some initial trials today, when I set the CM board IN1 gain to -32dB (as in the case of attempting the CARM RF handoff) and compensated for the reduced POY PDH fringe amplitude by increasing the digital gain for the CM_Slow path, I found that there was no phase advance visible even when I ramped up the IMC IN2 gain to +10dB. So, for the CARM handoff too, I might have to start with a higher CM board IN_1 gain, compensate by reducing the CM_Slow digital gain even more, and then try upping the IMC IN2 gain.
P.S. When the excitation input to the CM board was enabled in order to make TF measurements, I saw significant increase in the RMS of the error signal. Probably some kind of ground loop issue.
I found the PMC unlocked this morning. It was re-locked using the usual procedure. I feel like this has been happening more frequently in the last month than before. In the past, the cause seems to have been the PZT voltage drifting too close to one of the rails - however, in this case, it looks like an IMC unlock event is what triggered the PMC lockloss (admittedly the PZT voltage was somewhat close to the rail). It would be good if someone can re-connect the PMC Transmission photodiode, it was a useful diagnostic channel we had working fine before the ringdowns started.
I also tweaked the input pointing into the PMC and ran the WFS DC offset relief script.
Over the last couple of days, I've been trying to see if I can measure the phase advance due to the AO path - however, I've been unable to do so for any combination of CM board IN1 gain and MC Servo board IN2 gain I've tried. Yesterday, I tried to understand the loop shapes I was measuring a little more, and already, I think I can't explain some features.
Attachment #1 shows the TF measured (using SR785, and the EXC_A bank of the CM board) when the CM Slow path has been engaged.
Attachment #2 shows error signal spectra for the in-loop PRFPMI DoFs, for a few different conditions.
I believe that a stable crossover is hopeless under these conditions.
A few years ago, Koji and I setup a delay line phase shifter, which can be used to impart a (switchable) delay to a signal path. Since we talked about reviving the fast (= high bandwidth) ALS control scheme at the meeting, I reminded myself of the infrastructure available.
For a beat note in the regime 10-100 MHz, we should have plenty of range in this module to add a delay such that we zero one quadrature of the ALS DFD output (for a linear error signal).
I then proceeded to connect the single-ended front panel BNC corresponding to the ALS_X_I DFD channel to the IN2 input of the CM board (this would be what we use for high bandwidth ALS feedback). The conventional ALS system uses the differential output from a rear-panel D-sub, so in principle, both systems could run in parallel. I confirmed that I could see a signal when the IN2 path on the CM board was engaged (monitored using ndscope at the CM_Slow output), and that this signal stabilized when the green laser was locked to the X-arm length, which itself was slaved to the PSL frequency using the usual POX locking scheme. I have not yet routed the LO leg of the ALS_X beat through the delay line phase shifter - see next elog for details.
Update about the ALS MEDM screen slider: the trick was to change the OMSL field of the C1:LSC-BO_1_0 channel to "closed_loop" instead of "supervisory". Once this is done, the DOL value of the same channel can be set to the soft channel C1:ALS-DelayCalc, which sets the 16 bit binary string that controls the delay. Because arbitrary delays are not possible, I think it's more natural for the user to interact with this 16-bit binary string rather than the actual delay itself. So the MEDM screen has been slightly modified from what is shown in Attachment #1.
I measured the transfer function of the AO path, and think that there are some features indicative of a problem somewhere in the IMC locking loop.
Regardless of the locking scheme used, high bandwidth control of the laser frequency relies on the fact that the laser frequency is slaved to the IMC cavity length with nearly zero error below ~50 kHz (assuming the IMC loop has a UGF > 100 kHz). In my single arm experiments, I didn't know what to make of the ripples that became apparent in the measured OLTF as the AO gain was ramped up.
Tonight, I measured the TF of the "AO path", which modifies the error point of the IMC, thereby changing the laser frequency.
Attachment #1 shows the result of the measurement.
I didn't use POX / POY as a sensor to confirm that this is real frequency noise, I will do so tomorrow. But it may be that realizing a stable crossover is difficult with so many features in the AO path.
Previous thread with a somewhat detailed characterization of the IMC loop electronics.
In the process of setting up some cabling at 1Y2, I must've bumped a cable to the c1lsc expansion chassis. Anyways, the c1lsc models crashed. I ran the reboot script around 530pm PDT. Usual locking behavior was recovered after this. The work at 1Y2 was:
The IN2 to CM board was already connected to I single ended output of the ALS X demodulator. The ~100 Hz UGF digital locking using the CM_SLOW path is straightforward but I didn't have any success with the AO path tonight. I wonder how high BW this lock can be made without injecting a ton of noise into the IMC loop, given that the EX uPDH only has ~ 10 kHz UGF.
Attachment #1 shows the spectra of the ALS signal
Attachment #2 is an OLTF measurement.
Unlikely, the alignment was probably just not good. I restored the alignment and now the arms can be locked to IR frequency.
Even though we were not able to lock the the IR beat (by enabling LSC) during the day possibly because of increased seismic activity
Today, I did the following tests (and so was touching electronics/cables at/around 1X2):
Results to follow.
After this work, I reverted the EPICS channels to the usual values. The IMC can be locked.
The quantum noise curves here are not correct. c.f. amplitude quadrature noise budget.
In the style of the KA characterization of the CM board, the AO path gain EPICS slider (IN2) of the IMC servo board was stepped by 1 dB through the full available range of -32 dB to +31 dB. For each value of the requested gain, I measured the TF from the injected signal (to IN2) to TP1A on the IMC servo board. I used the BNC connector for this test, whereas we use the LEMO connector for the AO path. The source was tee-d off at the SR785 side, with one leg going to IN2 of the IMC servo board, and the other going to CH1A of the SR785. TP1A of the IMC board was connected to CH2A of the SR785.
Attachment #1 - Measured gain vs requested gain.
Attachment #2 - Frequency dependent transfer functions
The motivation here is to try and figure out why I cannot engage the AO path smoothly in the CARM handoff part of lock acquisiton. I plan to use this information to do some loop modeling and project laser frequency noise coupling in various stages of the lock acquisition process.
In 1X1, there is a box labelled "FSS REF" below a KEPCO HV supply. This box had a power cable that wasn't actually connected to any power. I removed said cable.
in prep for the install tomorrow, we did the following:
Barring objections, tomorrow (Friday 28 Feb 2020) morning I will commence the switch (I still want to work on the IFO tonight).
While my checks of the AO signal path have thrown up some unanswered questions, I don't think there's any evidence in those measurements to suggest the AO crossover can't be realized. Thinking about it today though - I was wondering if it could be that the IN1 gain slider of the CM board is configured optimally. Looking back at some data, when the ALS CARM offset is zeroed, the CM_SLOW digitized data has a peak-to-peak range of ~200 cts. This is paltry. One possibility is that as I am upping the AO path gain, I'm simply injecting the electronics noise of the CM board into the IMC error point. I'd say it is safe to up the IN2 gain by 20dB to -12 dB, in which case the peak-to-peak would be ~2000 cts, still only 10% of the ADC range. It'll be straightforward to re-scale the CARM_B loop gain to account for this. Let's see if this helps.
I'd also like to measure the spectrum of the REFL11_I signal in a few different states. I think I should be able to do this using the OUT2 of the CM servo board. These are the things to try tonight:
There was some UNELOGGED work at EX today. The DFD outputs were also hijacked for loss measurement. Unclear who the culprit was, but there is now a broad noise bump centered around ~180 Hz in the ALS X noise curve, which certainly wasn't there yesterday. Maybe let's keep the few working systems working, it is annoying to have to deal with these auxiliary issues every night. I'll push ahead with locking, hopefully the ALS noise is "good enough".
And so it begins.
Barring objections, tomorrow (Friday 28 Feb 2020) morning I will commence the switch
There are several problems evident already.
For now, I've returned the old c1psl connections, the PMC and IMC are both locked. Need to do some debugging on the bench.
On Monday, we will remove the old c1psl and c1iool0 machines from the electronics rack and install the Acromag crate in a more permanent way. We will also clean up some of the old cabling and cross connects, althoug the situation seems so complicated (some cross connects are also used by the rtcds c1ioo expansion chassis) that I am inclined not to remove any cables.
The area around 1X1/1X2 has a lot of dangling cables and general detritus. Be careful if you are walking around there. We will clean up on monday.
$TARGET_DIR = /cvs/cds/caltech/target
It remains to (Jon is taking care of these)
[JV, JWR, YD, GV]
Channel list with test status
== Test Status ==
[done] Lock PMC and IMC
[done] IMC Servo board test
[done] IMC LO Det Mon channel check
[0th order] WFS quadrant DC mon
[none] WFS I/F monitors
[0th order] WFS attenuators
[none] IOO QPD channels
[done] FSS readbacks
[done] PMC readbacks
Some more detailed elogs about the individual tests will follow.
Basically, I have characterized the IMC Servo board in detail. The summary finding is that the IN2 (=AO gain) slider needs to be investigated.
All other channels need to be verified in a more thorough fashion than my basic checks which were just to guarantee the core interferometer functionality, which is important to me.
Had to reboot both end machines and the c1rfm model to get the TRX and TRY signals to the LSC models. Now both arms can be locked using POX/POY respectively.
Jordan and I removed another 10 kg of cabling from 1X2. The c1iool0 crate now has all cabling to it disconnected - but it remains in the rack because I can't think of a good way to remove it without disturbing a bunch of cabling to the fast c1iool0 machine. We can remove it the next time the vertex FEs crash. Cross connects have NOT been removed - we will identify which cross connects are not connected to the fast system and trash those.
Do we want to preserve the ability to use the PZT driver in 1X2?
I realigned the input pointing into the PMC this morning. Usually, the way I do this is to minimize any discernible mode structure in the PMC reflection CCD image. Today, I noticed that making the DC reflection go down also makes the DC transmission go down. Possibilities:
There was some work done on the Acro crate this morning. Unclear if this is independent, but I found that the IMC servo board IN1 slider doesn't respond anymore, even though I had tested it and verified it to be working. Patient debugging showed that BIO1 (and only that acromag unit with the static IP 192.168.114.61) doesn't show up on the subnet in c1psl. Hopefully it's just a loose network cable, if not we will switch out the unit in the afternoon.
Jon is going to make a python script which iteratively pings all devices on the subnet and we will put this info on an MEDM screen to catch this kind of silent failure.
After discussing with Koji, I removed the PZT driver and associated AI card from the Eurocrate at 1X2. The corresponding backplane connectors were also removed from the cross connects. An additional cable going from the DAC to IDC adaptor on 1X2 was removed. Finally, some cables going to the backplane P1 and P2 connectors for slots in which there were no cards were removed.
Finally, there is the IMC WFS whitening boards. These were reconfigured in ~2016 by Koji to have (i) forever whitening, and (ii) fixed gain. So the signals from the P1 connector no longer have any influence on the operation of this board. So I removed these backplane cables as well.
Some pics attached. The only cross connect cabling remaining on the south side of 1X2 is going to the fast BIO adaptor box - I suspect these are the triggered fast whitening switching for the aforementioned WFS whitening board. If so, we could potentially remove those as well, and remove all the cross connects from 1X1 and 1X2.
Update 1720: indeed, as Attachment #2 shows, the RTCDS BIO channels were for the WFS whitening switching so I removed those cables as well. This means all the xconnects can be removed. Also, the DAC and BIO cards in c1ioo are unused.
To debug a problem with the new c1psl (later elog), we needed a Supermicro EPICS server that was using the shared EPICS/modbus/asyn binaries rather than a local install. Of those available in the lab (c1iscaux, c1vac, c1susaux being the others), this was the only one which uses the shared install. So I
At which point Jon reset the software end, I restored the slow bias voltage and re-enabled the local damping. The optic seems to have damped okay. The Oplev spot is back in ~center of the QPD and the green beam can be locked to a TEM00 mode (so the alignment is okay - the IR beam is unavailable while c1psl issues are being sorted but I judge that things are back to the nominal state now).
Jon is going to write up the details of todays adventures. But the C1PSL Acromag chassis is sitting on the floor between the IMC beamtube and the 1X1 electronics rack, and is very much a trip hazard. Be careful if youre in that area.
The C1PSL crate has now been installed in a more permanent way in the rack.
After this work, I disabled logging and restarted the modbus service (and copied the current version of the systemd service file to the target directory for backup). The PMC and IMC lock alright. The system is now ready to be tested in-situ. I will separately continue my IMC Servo board tests in the evening.
One thought about how to protect against this kind of silent failure - how about we always run the modbus service with logging enabled, and then send out a warning email and stop the service if the logfile size suddenly blows up (which is characteristic of when the communications process dies)? This should be done in addition to the ping-ing of the individual IPs.
Regarding the burt-restore step that the systemd service runs after starting up the IOC - this is not even that useful, at least in the way it is currently setup (restore the "latest" burt snapshot file). If the maintenance takes >1hour as it often does, the "latest" snapshot for the system under maintenance is just garbage. So either the burt-restore should be for a "known good time" (dangerous because this will require frequent updates of the systemd service every time we find a new safe state) or we should just do it manually (my preference). Then there is no need to install custom packages on the server machine. Anyway, for now, I have not commented this step out.
Jordan is going to take pictures of all the electronics racks and update the relevant wiki pages.
I am running some tests on the IMC servo board with an extender card so the IMC will not be locking for a couple of hours.
It seems like the AO path gain stages on the IMC Servo board work just fine. The weird results I reported earlier were likely a measurement error arising from the fact that I did not disconnect the LEMO IN2 cable while measuring using the BNC IN2 connector, which probably made some parasitic path to ground that was screwing the measurement up. Today, I re-did the measurement with the signal injected at the IN2 BNC, and the TF measured being the ratio of TP3 on the board to a split-off of the SR785 source (T-eed off). Attachments #1, #2 shows the result - the gain deficit from the "expected" value is now consistent with that seen on other sliders.
Note that the signal from the CM board in the LSC rack is sent single-ended over a 2-pin LEMO cable (whose return pin is shorted to ground). But it is received differentially on the IMC Servo board. I took this chance to look for evidence of extra power line noise due to potential ground loops by looking at the IMC error point with various auxiliary cables connected to the board - but got distracted by some excess noise (next elog).
Sometime between 1PM and 6PM on Tuesday, excess laser frequency noise shows up in MCF at around 800 Hz, as shown in Attachment #1. Sigh.
While I show the MCF spectrum here, I confirmed that this noise is not injected by the IMC loop (with the PSL shutter closed, and the IMC servo board disconnected from the feedback path to the NPRO, the PMC error and control points still show the elevated noise, see Attachment #2). I don't think the problem is from the PMC loop - see Attachment #3 which is the ALS beat out-of-loop noise with the PMC unlocked (the PSL beam doesn't see the cavity before it gets to the ALS setup, and we only actuate on the cavity length for that loop, so this wasn't even really necessary).
Was there some work on the PSL table on Tuesday afternoon that can explain this?
I did some preliminary debugging of this, and have localized the problem to the output path (after MC slow) on the IMC Servo card. Basically, I monitored the spectrum of the ALS beat frequency fluctuations under a few different conditions:
Toggling C1:IOO-MC_FASTSW, which supposedly isolates the post-MC slow (a.k.a. MCL) part of the servo, I see no difference. I am also reasonably confident this switch itself works, because I can break the IMC lock by toggling it. So pending a more detailed investigation, I am forced to conclude that the problem originates in the part of the IMC servo board after the MCL pickoff. Some cabling was removed at 1X2 on Tuesday between the times when there was no excess and when it showed up, but it's hard to imagine how this could have created this particular problem.
An earthquake around 330 UTC (=730pm yesterday eve) tripped ITMX, ITMY and ETMX watchdogs. ITMX got stuck. I released the stuck optic and re-enabled the local damping loops just now.
Attachment #1 shows the relevant parts of the schematic of the WFS demod board (not whitening board).
Before removing the boards from the eurocrate:
After Koji effected the fix, the boards were re-installed, HV supplies were dialled back up to nominal voltage/currents, and the PMC/IMC were re-locked. The WFS DC channels now no longer saturate even when the IMC is unlocked 👏 👏 . I leave it to Yehonathan / Jon to calibrate these EPICS channels into physical units of mW of power. We should also fix the MEDM screen and remove the un-necessary EPICS channels.
Later in the evening, I took advantage of the non-saturated readbacks to center the beams better on the WFS heads. Then, with the WFS servos disabled, I manually aligned the IMC mirrors till REFLDC was minimized. Then I centered the beam on the MC2 transmission QPD (looking at individual quadrants), and set the WFS1/2 RF offsets and MC2 Trans QPD offsets in this condition.
WFS DC channels are saturating when the IMC is unlocked.
It's been a while since I've attempted any locking, so tonight was mostly getting the various subsystems back together.
Of course the reboot wiped any logs we could have used for clues as to what happened. Next time it'll be good to preserve this info. I suspect the local subnet went down.
P.S. for some reason the system logs are priveleged now - I ran sudo sysctl kernel.dmesg_restrict=0 on c1psl to make it readable by any user. This change won't persist on reboot.
I restarted the IOC but it didn't help.
I am now rebooting c1psl... That seemed to help. PMC screen seem to be working again. I am able to lock the PMC now.