Today I bench-tested most of the Acromag channels in the replacement c1susaux. I connected a DB37 breakout board to each chassis feedthrough connector in turn and tested channels using a multimeter and calibrated voltage source. Today I got through all the digital output channels and analog input channels. Still remaining are the analog output channels, which I will finish tomorrow.
There have been a few wiring issues found so far, which are noted below.
Crossed with LR
Crossed with UR
Crossed with LL
Crossed with UL
Happened again at ~730pm.
The NPRO diag channels don't really tell me what happened in a causal way, but the interlock channel seems suspicious. Why is the nominal value 0.04 V? From the manual, it looks like the TGUARD is an indication of deviations between the set temperature and actual diode laser temperature. Is it normal for it to be putting out 11V?
I'm not going to turn it on again right now while I ponder which of my hands I need to chop off.
I'm restoring it now in the hope we can get some more info on what exactly happened if this is a recurring event.
If thy left hand troubles thee
then let the mirror show the right
for if it troubles enough to cut it off
it would not offend thy sight
I repeated the exercise from yesterday, this time driving the butterfly mode [+1 -1 -1 +1] and adding the tuned PIT and YAW vectors from yesterday to it to minimize appearance in the Oplev error signals.
The measured output matrix is , where rows are the coils in the order [UL,UR,LL,LR] and columns are the DOFs in the order [POS,PIT,YAW,Butterfly]. The conclusions from my previous elog still hold though - the orthogonality between PIT and YAW is poor, so this output matrix cannot be realized by a simple gain scaling of the coil output gains. The "adjustment matrix", i.e. the 4x4 matrix that we must multiply the "ideal" output matrix by to get the measured output matrix has a condition number of 134 (1 is a good condition number, signifies closeness to the identity matrix).
let us have 3 by 4, nevermore
so that the number of columns is no less
and no more
than the number of rows
so that forevermore we live as 4 by 4
so that forevermore we live as 4 by 4
I'm struggling to think
When I got back from lunch just now, I noticed that the PMC TRANS and REFL cameras were showing no spots. I went onto the PSL table, and saw that the NPRO was in fact turned off. I turned it back on.
The laser was definitely ON when I left for lunch around 130pm, and this happend around 140pm. Anjali says no one was in the lab in between. None of the FEs are dead, suggesting there wasn't a labwide power outage, and the EX and EY NPROs were not affected. I had pulled out the diagnostics connector logged by Acromag, I'm restoring it now in the hope we can get some more info on what exactly happened if this is a recurring event. So FSS_RMTEMP isn't working from now on. Sooner we get the PSL Acromag crate together, the better...
I've borrowed the Busby Box for a day or so. Location: QIL lab at Bridge West.
Edit Sat Apr 20 21:16:46 2019 (awade): returned.
Ther isn't a consistent set of OSEM coil gains that explains the best actuation vectors we determined yesterday. Here are the explicit matrices:
There isn't a solution to the matrix equation , i.e. we cannot simply redistribute the actuation vectors we found as gains to the coils and preserve the naive actuation matrix. What this means is that in the OSEM coil basis, the actuation eigenvectors aren't the naive ones we would think for PIT and YAW and POS. Instead, we can put these custom eigenvectors into the output matrix, but I'm struggling to think of what the physical implication is. I.e. what does it mean for the actuation vectors for PIT, YAW and POS to not only be scaled, but also non-orthogonal (but still linearly independent) at ~10 Hz, which is well above the resonant frequencies of the pendulum? The PIT and YAW eigenvectors are the least orthogonal, with the angle between them ~40 degrees rather than the expected 90 degrees.
So we now have matrices that minimize the cross coupling between these DoFs - the idea is to back out the actuation coefficients for the 4 OSEM coils that gives us the most diagonal actuation, at least at AC.
Apr 16, 2019
Borrowed two laser goggles from the 40m. (Returned Apr 29, 2019)
Apr 19, 2019
Borrowed from the 40m:
- Universal camera mount
- 50mm CCD lens
- zoom CCD lens (Returned Apr 29, 2019)
- Olympus SP-570UZ (Returned Apr 29, 2019)
- Special Olympus USB Cable (Returned Apr 29, 2019)
Yehonathan wanted to take some measurements for loss determination. I misaligned the X arm completely and we installed a PD on the AS table so there is no light reaching the AS55 and AS110 PDs. Yehonathan will post the detailed elog.
Rana did a checkout of my story about oddness of the ETMY suspension. Today, we focused on the actuators - the goal was to find the correct coefficients on the 4 face coils that would result in diagonal actuation (i.e. if we actuate on PIT, it only truly moves the PIT DoF, as witnessed by the Oplev, and so on for the other DoFs). Here are the details:
After getting the go ahead from Chub and Jon, I restored the Vacuum state to "Vacuum normal", see Attachment #1. Steps:
controls@c1vac:/opt/target/python/interlocks$ git diff interlock.py
diff --git a/python/interlocks/interlock.py b/python/interlocks/interlock.py
index 28d3366..46a39fc 100755
@@ -52,8 +52,8 @@ class Interlock(object):
self.pumps = 
for pump in interlocks['pumps']:
pm = PumpManager(pump['name'])
- for condition in pump['conditions']:
+ #for condition in pump['conditions']:
+ # pm.register_condition(*condition)
So far the pressure is coming down smoothly, see Attachment #2. I'll keep an eye on it.
PSL shutter was opened at 645pm local time. IMC locked almost immediately.
Update 11pm: The pressure has reached 8.5e-6 torr without hiccup.
EX green stayed locked to XARM length overnight without a problem. The spectrogram doesn't show any alarming time varying features around 2 kHz (or at any other frequency).
I looked into this issue today. Initially, my thinking was that I'd somehow caused clipping in the beampath somewhere which was causing this 2kHz excitation. However, on looking at the spectrum of the in-loop error signal today (Attachment #1), I found no evidence of the peak anymore!
Since the vacuum system is in a non-nominal state, and also because my IR ALS beat setup has been hijacked for the MZ interferometer, I don't have an ALS spectrum, but the next step is to try single arm locking using the ALS error signal. To investigate whether the 2kHz peak is a time-dependent feature, I left the EX green locked to the arm (with the SLOW temperature offloading servo ON), hopefully it stays locked overnight...
These weren't present last week. The peaks are present in the EX PDH error monitor signal, and so are presumably connected with the green locking system. My goal tonight was to see if the arm length control could be done using the ALS error signal as opposed to POX, but I was not successful.
At some point I'd like to reclaim this setup for ALS, but meantime, Anjali can work on characterization/noise budgeting. Since we have some CDS signals, we can even think of temperature control of the NPRO using pythonPID to keep the fringe in the linear regime for an extended period of time.
This happened again, about 30,000 seconds (~2:06pm local time according to the logfile) ago. The cited error was the same -
Hard to believe there was any real power loss, nothing else in the lab seems to have been affected so I am inclined to suspect a buggy UPS communication channel. The PSL shutter was not closed - I believe the condition is for P1a to exceed 3 mtorr (it is at 1 mtorr right now), but perhaps this should be modified to close the PSL shutter in the event of any interlock tripping. Also, probably not a bad idea to send an email alert to the lab mailing list in the event of a vac interlock failure.
For tonight, I only plan to work with the EX ALS system anyways so I'm closing the PSL shutter, I'll work with Chub to restore the vacuum if he deems it okay tomorrow.
It is feeling cold in the office area. According to the digital wall clock near the coffee machine, it is 19C. Rana bumped the thermostat setpoint up by 2F (from 75F to 77F). We need to setup long-term monitoring.
just main points, anajli is going to fill out the details.
To rule out mode-matching as the reason for non-ideal output from the MZ, I suggested using the setup I have on the NW side of the PSL enclosure for the measurement. This uses two identical fiber collimators, and the distance between collimator and recombination BS is approximately the same, so the spatial modes should be pretty well matched.
The spooled fiber we found was not suitable for use as it had a wide key connector and I couldn't find any wide-key FC/PC to narrow-key FC/APC adaptors. So we decided to give the fiber going to the Y end and back (~90m estimated length) a shot. We connected the two fibers at the EY table using a fiber mating sleeve (so the fiber usually bringing the IR pickoff from EY to the PSL table was disconnected from its collimator).
In summary, we cannot explain why the contrast of the MZ is <5%. Spatial mode-overlap is definitely not to blame. Power asymmetry in the two arms of the MZ is one possible explanation, could also be unstable polarization, even though we think the entire fiber chain is PM. Anjali is investigating.
We saw today that the Thorlabs PM beam splitters (borrowed from Andrew until our AFW components arrive) do not treat the two special axes (fast and slow) of the fiber on equal footing. When we coupled light into the fast axis, we saw huge asymmetry between the two split arms of the beamsplitter (3:1 ratio in power instead of the expected 1:1 for a 50/50 BS). Looking at the patch cord with an IR viewer, we also saw light leaking through the core along it. Turns out this part is meant to be used with light coupled to the slow axis only.
had trouble using YUM to update. This turned out to be a config problem with our Martian router, not the new laptop. Since I've changed the WiFi pwd awhile ago for the martian access for the CDS laptops, you'll have to enter that in order to use the laptops.
turned out to be some Access Control nonsense inside of the router. Even loggin in as admin with a cable gave some of the fields the greyed out color (had to hover over the link and then type the URL directly in the browser window). ASIA is now able to connect and use YUM + usual connections. Gautam and I have also moved the router a little to get easier view of its LED lights and not blockk its WiFi signal with the cable tray. We'll get a little shelf so that we can mount it ~1 foot off of the wall.
still, this seems like a bad laptop choice: the Lenovo Ideapad 330 will not have its touchpad supported by SL7 without compiling a new version of the kernel
if this gonna be general for everybody, maybe it oughta be called IFOtest (like the last incarnation that was tried in Livingston) ?
then the SUS tests could just be some smaller set of measurements. Using the same code base, but different params.
The AS spot on the camera was oscillating at ~3 Hz. Looking at the Oplevs, the culprit was the BS PIT DoF. Started about 12 hours ago, not sure what triggered it. I disabled Oplev damping, and waited for the angular motion to settle down a bit, and then re-enabled the servo - damps fine now...
The alignement was disturbed after the replcement of the beam splitter. We tried to get the alignment back . But we are not succeeded yet in getting good interfernce pattern. This is mainly because of poor mode matching of two beams. We will also try with the spooled fiber.
In anticipation of needing to test hundreds of suspension signals after the c1susaux upgrade, I've started developing a Python package to automate these tests: susPython
The core of this package is not any particular test, but a general framework within which any scripted test can be "nested." Built into this framework is extensive signal trapping and exception handling, allowing actuation tests to be performed safely. Namely it protects against crashes of the test scripts that would otherwise leave the suspensions in an arbitrary state (e.g., might destroy alignment).
The package is designed to be used as a standalone from the command line. From within the root directory, it is executed with a single positional argument specifying the suspension to test:
$ python -m suspython ITMY
Currently the package requires Python 2 due to its dependence on the cdsutils package, which does not yet exist for Python 3.
So far I've implemented a cross-consistency test between the DC-bias outputs to the coils and the shadow sensor readbacks. The suspension is actuated in pitch, then in yaw, and the changes in PDMon signals are measured. The expected sign of the change in each coil's PDMon is inferred from the output filter matrix coefficients. I believe this test is sensitive to two types of signal-routing errors: no change in PDMon response (actuator is not connected), incorrect sign in either pitch or yaw response, or in both (two actuators are cross-wired).
The next test I plan to implement is a test of the slow system using the fast system. My idea is to inject a 3-8 Hz excitation into the coil output filter modules (either bandlimited noise or a sine wave), with all coil outputs initially disabled. One coil at a time will be enabled and the change in all VMon signals monitored, to verify the correct coil readback senses the excitation. In this way, a signal injected from the independent and unchanged fast system provides an absolute reference for the slow system.
I'm also aware of ideas for more advanced tests, which go beyond testing the basic signal routing. These too can be added over time within the susPython framework.
Testing is finished.
Will advise when I'm finished, will be by 1 pm for ALS work to begin.
I could probably install the new fan if we have one. Can you do without the laser for a while?
This thread: ELOG 10295
My interpretation of these ELOGs is that we did not have the replacement, and then I brought unknown fan from WB. At the same time, Steve ordered replacement fans which we found in the blue tower yesterday.
The next action is to replace the internal fan, I believe.
My interpretation of these ELOGs is that we did not have the replacement, and then I brought unknown fan from WB. At the same time, Steve ordered replacement fans which we found in the blue tower yesterday.
The next action is to replace the internal fan, I believe.
Steve had showed me some stock of long fibers a while back - they are from Oz Optics, and are 50m long, and are already spooled - so barring objections, we will try the MZ setup with the spooled fiber and see if there is any improvement in the fringing rate of the MZ. Then we can evaluate what additional stabilization of the fiber length is required. Anjali will upload a photo of the spooled fiber.
Attached is my phone recording of what it sounds like right now in the PSL enclosure - not good for frequency noise measurement! The culprit is the little PC fan that is hacked onto the back of the Innolight controller.
Can we get some panel mount FC/APC connectors and put them on a box? Then we could have the whole setup inside of a box that is filled with foam and sits outside the PSL hut.
Updated the noise budget to include the unsuppressed frequency noise from the EX laser. It does not explain the noise between 10-100 Hz, although the 1-3 Hz noise is close.
Actually, I think the curve that should go on the budget is when the X arm length is locked to the PSL frequency, whereas this is when the X arm is just locally damped. I will update it later tonight.
Update 1010pm: I've uploaded the relevant plot as Attachment #2. Predictably, the unsuppressed frequency noise of the EX laser is now higher, because the MC length is a noisier frequency reference than the arm cavity. But still it is a factor of 10 below the measured ALS noise.
Next noises to budget:
I decided to use the more direct method, of disconnecting feedback to the EX laser PZT, and then looking at the cavity flashes.
Attachment #1 shows the cavity swinging through two resonances (data collected via oscilloscope). Traces are for the demodulated PDH error signal (top) and the direct photodiode signal (bottom). The traces don't look very clean - I wonder if some saturation / slew rate effects are at play, because we are operating the PD in the 30 dB setting, where the bandwidth of the PD is spec-ed as 260 kHz, whereas the dominant frequency component of the light on the PD is 430 kHz.
The asymmetric horns corresponding to the sideband resonances were also puzzling. Doing the modeling, Attachment #2, I think this is due to the fact that the demodulation phase is poorly set. The PDH modulation frequency is only ~5x the cavity linewidth, so both the real and imaginary parts of the cavity reflectivity contribute to the error signal. If this calculation is correct, we can benefit (i.e. get a larger PDH discriminant) by changing the demod phase by 60 degrees. However, for 230 kHz, it is impractical to do this by just increasing cable length between the function generator and mixer.
Anyway, assuming that we are at the phi=30 degree situation (since the measurement shows all 3 horns going through roughly the same voltage swing), the PDH discriminant is ~40 uV/Hz. In lock, I estimate that there is ~60 uW of light incident on the PDH reflection photodiode. Using the PD response of 0.2 A/W, transimpedance of 47.5 kohm, and mixer conversion loss of 6dB, the shot-noise limited sensitivity is 0.5 mHz/rtHz. The photodiode dark noise contribution is a little lower - estimated to be 0.2 mHz/rtHz. The loop does not have enough gain to reach these levels.
PDH discriminant (40 uV/Hz, see this elog)
wrong assumption - i checked the gain just now, it is G=10, and is running in the "low-noise" mode, so can only drive 4V. fixed elog, filter.
Note: While working at EX, I saw frequent saturations (red led blinking) on the SR560. Looking a the error mon signal on a scope, it had a pk-to-pk amplitude of ~200mV going into the SR560. Assuming the free-swinging cavity length changes by ~1 um at 1 Hz, the green frequency changes by ~15 MHz, which according to the PDH discriminant calibration of 40 uV/Hz should only make a 60 mV pk to pk signal. So perhaps the cavity length is changing by 4x as much, plausible during daytime with me stomping around the chamber I guess.. My point is that if the SR560 get's saturated (i.e. input > 13000 cts), the DQ-ed spectrum isn't trustworthy anymore. Should hook this up to some proper whitening electronics
G=10 or G=100?
Some time ago, I had done an actuator calibration of ITMX. This suspension hasn't been victim to the recent spate of suspension problems, so I can believe that the results of those measurement are still valid. So I decided to calibrate the in-loop error signal of the EX green PDH loop, which is recorded via an SR560, G=10, by driving a line in ITMY position (thereby modulating the X arm cavity length) while the EX green frequency was locked to the arm cavity length. Knowing the amount I'm modulating the cavity length by (500 cts amplitude sine wave at 33.14159 Hz using awggui, translating to ~17.2 kHz amplitude in green frequency), I demodulated the response in C1:ALS-X_ERR_MON_OUT_DQ channel. At this frequency of ~33 Hz, the servo gain should be large, and so the green laser frequency should track the cavity length nearly perfectly (with transfer function 1/(1+L), where L is the OLG).
The response had amplitude 5.68 +/- 0.10 cts, see Attachment #1. There was a sneaky gain of 0.86 in the filter module, which I saw no reason to keep at this strange value, and so updated to 1, correcting the demodulated response to 6.6 cts. After accounting for this adjustment, the x10 gain of the SR560, and the loop suppression, I put a "cts2Hz" filter in (Attachment #2). I had to guess a value for the OLG at 33 Hz in order to account for the in-loop suppression. So I measured the OLTF using the usual IN1/IN2 method (Attachment #3), and then used a LISO model of the electronics, along with guesses of the cavity pole (18.5 kHz), low-pass filter poles (4x real poles at 70 kHz), PZT actuator gain (1.7 MHz/V) and PDH discriminant (40 uV/Hz, see this elog) to construct a model OLTF. Then I fudged the overall gain to get the model to line up with the measurement between 1-10kHz. Per this model, I should have ~75dB of gain at ~33Hz, so the tracking error to my cavity length modulation should be ~3.05 Hz. Lines up pretty well with the measured value of 4.7 Hz considering the number of guessed parameters. The measured OLG tapers off towards low frequency probably because the increased loop suppression drives one of the measured inputs on the SR785 into the instrument noise floor.
The final calibration number is 7.1 Hz/ct, though the error on this number is large ~30%. Note that these "Hertz" are green frequency changes - so the change to the IR frequency will be half.
Attachment #4 shows the error signal in various conditions, labelled in the legend. Interpretations to follow.
I worked on characterizing the green PDH setup at EX, as part of the ALS noise budgeting process. Summary of my findings:
The main motivation was to get the residual frequency noise of the EX laser when locked to the X arm cavity - but I'll need the V/Hz PDH discriminant to convert the in-loop error signal to frequency units. The idea was to look at the PDH error signal on a scope and match up the horn-to-horn voltage with a model to back out said discriminant, but I'll have to double check my model for errors now given the large mismatch I observe in reflected power.
I tried restarting c1oaf this weekend to see if turning on the MC length FF would affect the ALS noise performance. I burtrestored the filter settings from March 2016. However, I noticed several possible anomalies, which need debugging. I am not turning the model off because of the possibility of having to reboot all the vertex FEs, but this model is totally unusable right now.
To start the noise budgeting, I decided to measure the "DFD noise", which is really the quadrature sum of the following terms:
According to past characterizations of these noises, the ADC noise level, which is expected to be at the level of a few uV/rtHz, is expected to be the dominant noise source.
The measurement was made by disconnecting the NF 1611 free space photodiode from the input to the RF amplifier on the PSL table, and connecting a Marconi (f_carrier = 40 MHz, signal level=-5dBm) instead. The phase tracked output was then monitored, and the resulting digital spectrum is the red curve in Attachment #1. The blue curve is the ASD of fluctuations of the beatnote between the PSL and EX IR beams, as monitored by the DFD system, with the X arm cavity length locked to the PSL frequency via the LSC servo, and the EX green frequency locked to the X arm cavity length by the analog PDH servo.
Assuming the Marconi frquency noise is lower than the ones being budgeted:
Since we haven't been using it, the PID control was not enabled on the doubling oven on the PSL table (it is disabled after every power outage event in the lab). I re-enabled it just now. The setpoint according to the label on the TC200 controller is 36.9 C. The PID paramaters were P=250, I=200, D=40. These are not very good as the overshoot when I turned the control on was 44 C, seems too large. The settling time is also too long, after 10 minutes, the crystal temperature as reported by the TC200 front panel is still oscillating. I can't find anything in the elog about what the nominal PID parameter values were. The X end PID seems much better behaved so I decided to try the same PID gains as is implemented there, P=250, I=60, D=25.
With the Ophir power meter, I measured 60mW of IR light going into the doubling oven, 110uW green light coming out, for a conversion efficiency of 2.7%/W, seems pretty great.
Next, I went to EX and tweaked the steering mirror alignment - I wasn't able to improve the transmission significantly using the PZT sliders on the EPICS screen, and the dither alignment servo isn't working. It required quite a substantial common mode yaw shift of the PZT mirrors to make GTRX ~ 0.5.
I plan to recover the green beat note as well and digitize it using the second available DFD channel (eventually for the Y arm) - then we can simultaneously compare the the green and IR performance (though they will have different noise floors as there is less green light on the green beat PDs, and I think lower transimpedance too).
One of the main draw backs of the measurement was the polarisation was not aligned properly in the setup. So, then the next step was to identify the polarisation at different locations in the beam path and to maximise the polarisation to either S or P component.
So, we introduced HWP at the input beam path after isolator as shown in attachment #1. Also, the polarisation was tested at positions P1, P2, P3, and P4 shown in attachment #1 by placing a polarisation beam splitter at these locations and then by observing the transmitted (P component) and reflected light (S component) using power meter.
The observations at different locations are as the follows
These observations show that the P and S components are almost equal, and this is not a good polarisation arrangement. At this point, we also had to check whether the incoming beam is linearly polarised or not.
To test the same, the PBS was placed at position P1 and the P and S components were observed with power meter as the HWP is rotated.Attachment # 2 shows the results of the same, that is the variation in P and S component as the HWP is rotated.
This result clearly shows that the input beam is linearly polarised. The HWP was then adjusted such that the P component is maximum and coupled to the MZI. With this orientation of HWP, the polarisation observed at different positions P1, P2, P3, and P4 are as follows.
This shows that the polarisation is linearly polarised as well as it is oriented along the P direction (parallel to the optical table).
We have the polarisation maintaining fiber (PM 980) as the delay fiber. The polarisation of the light as it propagates through a PM fiber depends on how well the input beam is coupled to the axis (slow or fast) of the fiber. So, the next task was to couple the light to one of the axes of the fiber.
The alignment key on the fiber is a good indication of the axis of the fiber. In our case, the alignment key lines up with the slow axis of the fiber. We decided to couple the light to the fast axis of the fiber. Since the incoming beam is P polarised, the output fiber coupler was aligned such that the fast axis is parallel to optical table as possible.
A PBS was then introduced after the fiber output collimator . There is a HWP (marked as HWP2 in attachment 1) in front of the input coupler of the fiber as well. This HWP was then rotated and observed the P and S component from the PBS that is now placed after the output coupler with a power meter.The idea was , when the light is coupled to the fast axis of the fiber, we will see the maximum at the P componet at the output
Attachment # 3 shows the observation.
In this way I tried to find the orientation of the HWP2 such that the P component is maximum at the output. But I was not succeeded in this method and observed that the output was fluctuating when the fiber was disturbed. One doubt we had was whether the fiber is PM or not . Thus we checked the fiber end with fiber microscope and confirmed that it is PM fiber.
Thus, we modifed the setup as shown in attachement # 4.The photodetector (PDA55) was monitoring the S component and the output of the detector was observed on an oscilloscope. We rotated the HWP2 such that the S component is almost minimum. At the same time, we were disturbing the fiber and was observing whether the output is fluctuating. The HWP2 angle was tweaked around the minimum of S component and observed the output with disturbing the fiber. This way we found the orientation of HWP2 such that the light is coupled to the fast axis of the fiber and the output was not fluctuating while we disturb the fiber. We tested it by heating the fiber with a heat gun as well and confirmed that the output is not fluctuating and thus the light is coupled to the fast axis of the fiber.
Is this one close to failure as well?
I set up a free-space beat on theNW side of the PSL table between the IR beam from the PSL and from EX, the latter brought to the PSL table via ~40m fiber. Initial measurements suggest very good performance, although further tests are required to be sure. Specifically, the noise below 10 Hz seems much improved.
Attachment #1 shows the optical setup.
Yehonathan came by today so I had to re-align the arms and recover POX/POY locking. This alllowed me to lock the X arm length to the PSL frequency, and lock the EX green laser to the X arm length. GTRX was ~0.36, whereas I know it can be as high as 0.5, so there is definitely room to improve the EX frequency noise suppression.
Attachment #2 shows the ALS out-of-loop noise for the PSL+X combo. The main improvements compared to this time last year are electronic.
Mix the beams in free space. We have the beam coming from EX to the PSL table, so once we mix the two beams, we can use either a fiber or free-space PD to read out the beatnote.
PSL shutter was re-opened at 6pm local time. IMC was locked. As of 10pm, the main volume pressure is already back down to the 8e-6 level.
I can't explain the mechanical switching sound Gautam reported. The relay controlling power to the TP2 forepump is housed in the main AC relay box under the arm tube, not in the Acromag chassis, so it can't be from that. I've cycled through the pumpdown sequence several times and can't reproduce the effect. The Acromag switches for TP2 still work fine.
In any case, I've made modifications to the vacuum interlocks that will help with two of the issues:
C1:AUX-PSL_ShutterRqst --> 0
After finishing this vac work, I began a new pumpdown at ~4:30pm. The pressure fell quickly and has already reached ~1e-5 torr. TP2 current and temp look fine.
However, when opening V4 (the foreline of TP1 pumped by TP2), I heard a loud repeated click track (~5Hz) from the electronics rack. Shortly after, the interlocks shut down all the TPs again, citing "AC power loss". Something is not right, I leave it to Jon and Chub to investigate.
Andrew seems to have an integrated solution of PBS+HWP in a singe mount. Or, I wonder if we should use HWP/QWP before the coupler. I am interested in a general solution for this problem in my OMC setup too.
Bob and Chub concluded that the drypump that serves as TP2's forepump had failed. Steve had told me the whereabouts of a spare Agilent IDP-7. This was meant to be a replacement for the TP3 foreline pump when it failed, but we decided to swap it in while diagnosing the failed drypump (which had 2182 hours continuous running according to the hour counter). Sure enough, the spare pump spun up and the TP2fl pressure dropped at a rate consistent with what is expected. I was then able to spin up TP1, TP2 and TP3.