• The arm powers could be stabilized somewhat once the CM_SLOW path to MC2 was engaged.
• However, I was never able to get the AO path to do anything good.
• Took a bunch of CM board TFs, need to think about what I need to do differently to get this next bit to work.
• An SR785 is sitting next to the LSC rack hooked up to the CM board. I also borrowed the GPIB unit from the AG4395 to grab data from said SR785.
• One thing I noticed that the CARM_B (=CM_SLOW) and DARM_B (=AS55_Q) signals both had a DC offset, so maybe this is indicative of some DC offset in the PRMI 3f signals? Right now, I lock the PRMI without any offsets, and as I reduce the CARM offset, I can see the DC value of REFL11_I and AS55_Q changing significantly. To be investigated in tonight's locking.

[Yehonathan, Gavin]

Measuring POX11_Q_MON and injecting a signal into the ITMX_UL_IN port a signal could not be seen on the function generator. After debugging the source of the issue was two fold:

• By using the quadrant drives for coils (UL, UR etc) a signal has to pass through a switch before reaching the driver. To resolve this the signal input was switched to POS_IN (driving the entire coil at once rather than in quadrants) which has no switch to bypass.
• The averaging on the Stanford SR785 was set too low. By increasing the averages from 10 to 25 the signal became more visible.

Unrelated to these issues the signal input was switched to POY11_Q_MON and ITMY_POS_IN as part of the debugging process. The function generator used was switched from the Stanford to the Siglant SDG 1032X.

An unrelated issue but note worthy was the Lenovo 40m laptop used to measure the IFO state (locked or unlocked) ran out of battery in a very short timespan.

To gauge where the resonance of the test masses are FEA model of a simple 40m test mass was computed to give an esitimate at what frequency the eigenmodes exist. For the first two modes the model gave resonances at 20.366 kHz (butterfly mode) and 28.820 kHz (drumhead mode). Then by measuring with an acquisition time of 1 s at they frequencies on the SR785 and injecting broad band white noise with a mean of 0 V and a stdev of 2 V, small peaks were seen above the noise at 20.260 kHz and 28.846 kHz. By then injecting a sine wave at those frequencies with 9 Vpp, the peak became clearly visible above the noise floor.

The last step is to measure the natural decay of these modes when the excitation is turned off. It is difficult to tell currently if these are indeed eigenmodes or just large cavity injections with an associated stabilisation time (what could appear as a ringdown decay). More investigation is required.

I symlinked the SRmeasure and AGmeasure commands to /usr/bin/ on donatella (as it is done on pianosa) so that these scripts are in $PATH and may be run without having to navigate to the labutils directory.

I tried starting the c1oaf model, but got a DQ error (I want the option of running feedforward during locking even if the filters aren't particularly well tuned yet). Note that this isn't "just a warning light" - some channels are initialized to +/- 1e20, so if you try turning some filters on, you will deliver a massive kick to the optics. Restarting it crashed c1lsc (this is not unexpected behavior - the only way to clear the DQ error is to restart the model, and empirically, the success rate is ~50%). The reboot script brought everything back online smoothly, and the second, time, c1oaf started without any issues.

While looking at the CDS overview screen, I noticed that the c1scy model was reporting frequent RFM errors for the C1:SCY-RFM_ETMY_LSC channel (but none of the others). On the sender model (c1rfm), no errors were being reported. The diag reset button / mxstream restart didn't really work either. See Attachment #1. Just restarting the c1scy model didn't fix the error - I had to reboot the machine and restart the models, and now no errors are being reported.

Attachment #2 shows the current nominal CDS status - the red light on c1lsc is due to some missing c1dnn channels (I'll remove these at the next c1lsc model change because I don't want to un-necessarily reboot the vertex FEs), and the c1omc model is obsolete I guess. c1daf isn't running right now but once I get the new fiber (ordered), I'm gonna restart this model as well.

P.S. The ALS temperature sliders are not SDF-ed. So when the model was restarted, I had to change the sliders back to their old values to get the beat back in the usable range.

In preparation for locking tonight, I re-centered the spots on the Oplev QPDs for the ITMs, BS and PRM after locking and running the dither alignment for the arms and also the PRMI carrier. In the past, DC coupling the ITM Oplevs helped the angular stability a bit, let's see if it still does.

Summary:

I made it to 0 CARM offset, PRMI locked a bunch of times today. However, I could not successfully engage the AO path.

Details:

Much of the procedure is scripted, here is the rough set of steps:

• Transition control of the arms from IR signals to ALS signals.
• DC couple the ITM oplev servos
• Burt-restore the settings for PRMI locking with REFL165I-->PRCL, REFL165Q-->MICH, and then enable the MICH_B / PRCL_B locking servos.
• Add some POPDC to the PRMI triggering (nominally only POP22_I) to let these loops be locked while POP22_I fluctuates wildly when we are near the CARM=0 point.
• Zero the CARM offset.
• Adjust the CARM_A/DARM_A offsets such that CARM_B/DARM_B are fluctuating symmetrically about 0.
• CARM_B gain --> 1.0, to begin the RF blend.
• Prepare to hand the DC control authority to ALS by turning off FM1 in the CARM filter bank, and turning ON an integrator in the CARM_B filter.

As I type this out, I realized that I was incorrectly setting offsets to maximize the arm powers by adjusting CARM/DARM offsets as opposed to CARM_A / DARM_A offsets. Tried another round of locking, but this time, I can't even turn the integrator on to get the arms to click into somewhat stable powers. 

One thing I noticed is that depending on the offsets I put into the 3f locking loops, the mean value of REFL11 and AS55 when the ALS CARM/DARM offsets are zeroed changes quite significantly. What is the correct condition to set these offsets? They are different when locking the PRC without arm cavities, and also seem to change continuously with CARM offset. I am wondering if I have too much offset in one of the vertex locking loops?

[Jordan, gautam]

We did the following:

• Route the fiber from the control room to 1Y2.
• Plug fiber in to FiBox at either end, turned FiBoxes ON.
• Tested the optical connection by driving a 1Vpp 440 Hz sine wave from a function generator - Yehonathan hears it loud and clear in the control room.
• Tested that both CH1 and CH2 work - only CH1 is connected to the speakers in the control room at the moment.
• There is some cross-coupling between the channels - not sure if this is happening in the multi-mode fiber or in the electroncis, but I estimate the isolation to be >30dB.
• Connected CH8 and CH9 of DAC0 in the c1lsc expansion chassis to CH1 and CH2 respectively of the FiBox in 1Y2. 
• Restarted the c1daf model on c1lsc, came up smooth.
• Routed the POY11 error signal through the various matrices in c1daf, and we could 👂 the Y-arm cavity 🔐 😎 
• Channels are muted for now - I'll give this a whirl while doing the PRFPMI locking.

Yehonathan, please center the EX seismometer.

The attached PDF shows the seismometer signals (I'm assuming that they're already calibrated into microns/s) during the lab tour for the art students on 11/1. The big spike which I've zoomed in on shows the time when we were in the control room and we all jumped up at the same time. There were approximately 15 students each with a mass of ~50-70 kg. I estimate that out landing times were all sync'd to within ~0.1 s.

I have re-centered the EX (and EY) seismometers. They are Guralp CMG40-T, and have no special centering procedure except cycling the power a few times. I turned off the power on their interface box, then waited 10s before turning it back on.

The fist atm shows the comparison using data from 8-9 PM Saturday night:

1. there seems to be a factor of 2 calibration diff between the T240 near the BS, and the Guralp seismometers at the end. Which one is right? When was the last time they were cross calibrated?
2. The low coherence between BS_X and EX_X shows the problem. They should be very coherent (> 0.9) for 0.1-1 Hz.

The IBM laptop at EX was running Ubuntu 14, so I allowed it to start upgrading itself to Ubuntu16 as it desired. After it is done, I will upgrade it to 18.04 LTS. We should have them all run LTS.

I noticed recently that Megatron was running Ubuntu 12, so I've started its OS upgrade.

1. Unlocked the IMC + disabled the autolocker from the LockMC screen + closed the PSL shutter (IMC REFL shutter doesn't seem to do anythin)
2. Disabled the "FSS" slow servo on the FSS screen
3. did sudo apt-get update, sudo apt-get upgrade, and then sudo apt-get do-release-upgrade which starts the actual thing
4. According to the internet, the LTS upgrades will go in series rather than up to 18 in one shot, so its now doing 12 -> 14 (Trusty Tapir)

Megatron and IMC autolocking will be down for awhile, so we should use a different 'script' computer this week.

Mon Dec 9 14:52:58 2019

upgrade to Ubuntu 14 complete; now upgrading to 16

I check the seismometers in the last 14 hours (Attached). Seems like the coherenece is restored in the x direction.

{Yehonathan, Rana}

In order to setup a ringdown measurement with perfect extinction we need to align the first order beam from the AOM to the PMC instead of the zeroth order.

We connected a signal generator to the AOM driver and applied some offset voltage. We spot the first order mode and align it to the PMC. The achieved transmitted power is roughly as it was before this procedure.

Along the way few changes has been made in the PSL table:

1. Some dangling BNCs were removed.

2. Laser on the south east side of the PSL table was turned off.

3. DC power supplies were removed (Attachment 1 & 2). The rubber legs on the first one are sticky and leave black residue.

4. The beam block that orginally blocked the AOM high order modes was raised to block the zeroth order mode (Attachment 3).

5. The unterminated BNC T junction (Attachment 4 - before picture). from the PMC mixer to the PMC servo was removed.

However, we are currently unable to lock the PMC on high gain. When the gain is too high the PZT voltage goes straight to max and the lock is lost.

In short:

Steps undertaken:

1. Normally I would unlock the IMC (Disabling the servo between the 'Filter' and 'Polarity' on the Mode Cleaner Servo Screen), but today I did not have to since Rana had kept it unlocked.

2. Misaligned the ITMX. This is to prevent cavity resonances from returning to the laser

3. Turned up the air on the HEPA at the PSL table to 100% during the measurement

4. Cables were connected as before (diagram shown in attachment of elog 15069)

5. The X end laser NPRO was actuated for the TF measurement using a long cable connected to TO AUX_X LASER PZT

Thoughts and observations:

- Reading out the error signal after amplification cannot distinguish between a locked loop or one out of its range. The error signal would be very small in both cases.

- Looking at the beat note on an oscilloscope, there also seemed to be an additional amplitude modulation that I had not noticed earlier. Rana suggested that it may have something to do with the pre-mode cleaner and the AOM being driven at 80 MHz

- Even though the TF was attempted, it seemed too noisy, suggesting that the PLL did not seem to work

- Rana also suggested that it may be a better idea to use the PZT of one of the lasers as the VCO for the PLL feedback instead of the Marconi.

Just realized that the diffracted beam is frequency shifted by 80MHz. It would shift the PZT position in the PMC lock acquisition, wouldn't it?

nvm the PZT can scan over many GHz.

{Jon, Yehonathan}

We burt-restored the PSL and the PMC locked immediately.

The PMC is now locked on the AOM first order mode.

{Yehonathan, Jon}

We are able to lock the PMC on the TEM00 mode of the deflected beam.

However when we turn off the driving voltage to the AOM and back on the lock is not restored. It get stuck on some higher order mode.

There are plethora of modes present when the PZT is scanned, which makes us believe the cavity is misaligned.

To lock again on the TEM00 mode again we disconnect the loop (FP Test point 1), find a TEM00 mode using the DC output adjust and close the loop again.

[Jon, Yehonathan]

We carried out a set of cavity ringdown measurements of the PMC. The 1/e decay time scale is found to be 35.2 +/- 2.4 (systematic) μs. The statistical error is negligible compared to the systematic error, which is taken as the maximum absolute deviation of any measurement from the average value.

To make the measurement, we injected the first order deflection beam of an 80 MHz AOM, then extinguished it quickly by cutting the voltage offset to the AOM driver provided by an RF function generator. A 100 MHz oscilloscope configured to trigger on the falling voltage offset was used to sample the cavity in transmission as sensed by a PDA55. We found the detector noise of the DC-coupled output of the 35.5 MHz REFL PD to be too high for a reflection-side measurement.

Further loss analysis is forthcoming.

Make sure to measure the power drop of the beam downstream of the AOM but before the PMC. Need to plot both together to make sure the chop time is much shorter than the 1/e time.

Megatron is now running Ubuntu 18.04 LTS.

We should probably be able to load all the LSC software on there by adding the appropriate Debian repos.

I have re-enabled the cron jobs in the crontab.

The MC Autolocker and the PSL NPRO Slow/Temperature control are run using 'initctl', so I'll leave that up to Shruti to run/test.

{Yehonathan, Rana, Jon}

To check whether we laser is being shut fast enough for the ringdown measurement we put a PD55 in the path that leads to the beat note setup. The beam is picked off from the back steering mirror after AOM and before the PMC.

@Shruti the PD is now blocking the beam to your setup.

As before, we drive the AOM to deflect the beam. The deflected beam is coupled to the PMC cavity. We lock the PMC and then shut the beam by turning off the output of the function generator that provides voltage to the AOM driver.

We measure the transmitted light of the PMC together with the light that is picked off before the PMC. In Attachment 1, the purple trace is the PMC transmission, the green trace is the peaked-off beam and the yellow trace is the function generator signal.

Rana was pointing out that the PDs, the function generator and the scope were not carefully impedance matched, which could lead to erroneous measurements. He also mentioned that the backscattered beam was too bright which might indicate that the PMC is oscillating. To remedy this we lowered the gain of the PMC lock to ~8.

We repeat the measurement after setting all the components to 50ohm (attachment 2). We then realize that the BNC T junction connected on the function generator is splitting the signal between the 50ohm AOM driver and 1Mohm oscilloscope channel which causes distortions as can be seen. We remove the T junction and get a much cleaner measurement (see next elog).

It seems like either the shutting speed or the PDs are only slightly faster than the PMC. I also check the AOM driver RF output fall time doing the same kind of measurement (attachment 3).

We suspect the PDs' bandwidth is to blame (although they are quoted to have 10MHz bandwidth).

In any case, this is fast enough for the IMC and arm cavities whose lifetime should be much longer than the PMC.

I will post an elog with some numbers tomorrow.

I grab the data we recorded yesterday from the scope and plot it in normalized units (remove noise level and divide by maximum). See attachment.

It can be seen that the measured ringdown time is ~ 17us while the shut-off time is ~12us.

I plan to model the PD+AOM as a lowpass filter with an RC time constant of 12us and undo its filtering action on the PMC trans ringdown measurement to get the actual ringdown time.

Is this acceptable?

I have removed the PD55 + ND filter attached to it (see Attachment) and placed it next to the oscilloscope, after disconnecting its output and power supply. The post is still in place.

I did see the beat after that.

I added to the PSL wiring list the ioo channels and the laser shutter (See attached pdf for an updated list).

The total channel numbers for now:

I counted each mbbo as 1 bo but I am not sure that's correct.

Still need to allocate Acromags.

Updated the channel list (Attached):

1. Removed the MC steering mirror PZT channels

2. Added Sourcing/Sinking column

3. Recounted the mbbos correctly

4. Allocated Acromags:

I think we can start wiring.

1. Some calculations

For a Unity Gain Frequency (UGF) of 1 kHz, assumed PZT response  of 1 MHz/V, Mixer response of 25 mV/ rad, the required gain of the amplifier is

G ~ 0.8

2. Progress

- Measured the mixer response

Measuring mixer response:

- PSL laser temperature was adjusted so that beat frequency was roughly 25 MHz and the amplitude was found to be roughly -30dBm.

- At the RF port instead of the beat signal, a signal of 25 MHz + few kHz at -30 dBm was inputted. The LO was a 25 MHz signal was sent from the Marconi at 7 dBm.

- The mixer output was measured, with setup as in Attachment 1  Figure (A), on an oscilloscope. The slope near the small angle region of the sine curve would be the gain (in V/rad) and was found to be:  rad

- Since from the above calculations it seemed like an amplifer gain of 1 should work for the PLL, I rearranged the set up as in Figure (B) of Attachment 1 to actuate the X end NPRO PZT, I adjusted the PSL temperature (slow control) to try and match the frequency to 25 MHz, but couldn't lock the loop. I was monitoring the error signal after amplification (50 ohm output of the SR 560) which showed oscillations when the beat frequency was near 25 MHz and nothing significant otherwise.

- I used a 20 dB attenuator at the amplifier output and saw the beat note oscillate for longer, but maybe because it was a 50 ohm component in a high impedance channel it did not work either (?). I tried other attenuator combinations with no better luck.

- Is there a better location to add the attenuator? Should I pursue amplifying the beat signal instead?

- Also, it seemed like the beat note drift was higher than earlier. Could it be because the PMC was unlocked?

idk - I'm recently worried about the 'thermal self locking' issue we discussed. I think you should try to measure the linewidth by scanning (with low input power) and also measure the TF directly by modulating the power via the AOM and taking the ratio of input/output with the PDA55s. I'm curious to see if the ringdown is different for low and high powers

This is an ole SURF report on thermal self-locking that may be of use (I haven't read it or checked it for errors, but Royal was pretty good analytically, so its worth looking at)

Final (hopefully) PSL channel list is attached with allocated Acromag channels. Wiring spreadsheet coming soon.

Current Acromag count:

PSL wiring spreadsheet is ready. (But the link was stripped. Koji)

Link to a wiki page  with the link to the wiring spreadsheet (Yehonathan)

I measured PMC ringdowns for several input powers. I change the input power by changing the DC voltage to the AOM.

First, I raise the DC voltage to the AOM from 0V and observe the signal on the picked off PD. I see that at around 0.6V the signal stops rising. The signal on the PD is around 4V at that point so it is not saturated.

Up until now, we provided 1.5V to the AOM, which means it was saturated.

I measured ringdowns at AOM voltages of 0.05, 0.1, 0.3, 0.5, 1 volt by shutting off the DC voltage to the AOM and measuring the signal at the PMC transmission PD and the picked off PD simultaneously for reference.

Attachment 1 shows the reference measurement for different AOM voltages. For low AOM DC voltages, the response of the AOM+PD is slower.

Attachment 2 shows the PMC transmission PD measurements which barely change as a function of AOM voltage but shows the same trend. I believe that if the AOM+PD response was much faster there would be no observable difference between those measurements.

Attachment 3 shows PMC transmissions and references for AOM voltages 0.05V and 1V. It seems like for low AOM voltages we are barely fast enough to measure the PMC ringdown.

I fitted the 0.3V ringdown and reference to a sum of two exponentials (Attachment 4).

The fitting function is explicitly a * nm.exp(-x/b) +c* nm.exp(-x/d) +e

For the PMC transmission I get:

a = 0.21
b = 3.64 (us)
c = 0.69, 
d = 39.62 (us)
e = 2.0e-04

For the reference measurement:

a = 0.34
b = 4.97 (us)
c = 0.58
d= 31.22 (us)
e = 1.11e-03

I am still not able to do deconvolution of the ref from the measurement reliably. I think we should do a network analyzer measurement.

Shruti, the PD is again in your beam path.

I try to measure the linewidth of the PMC by ramping the PMC PZT. 

I do it by connecting a triangular shape signal to FP Test 1 on the PMC servo front panel (I know, it is probably better to connect it to DC EXT. next time.) and turn the servo gain to a minimum.

Attachment 1 shows the PMC transmission PD as the PZT is swept with the EOM connected and when it is disconnected. It shows the PMC over more than 1 free spectral range.

For some reason, I cannot seem to be able to find the 35MHz sidebands which I want to use to calibrate the PZT scan. I made sure that the EOM is driven by a 35MHz signal using the scope. I also made sure that the PMC cannot to lock without the EOM connected.

I am probably doing something silly.

When I was looking at this, the AOM shutdown time was measured to be ~120 ns, and while I wasn't able to do a ringdown measurement with the PMC (it'd just stay locked because at the time i was using the zeroth order beam), the PMC transmission decayed in <200 ns. 

For the IMC servo board, it'd be easiest to copy the wiring scheme for the BIO bits as is configured for the CM board (i.e. copy the grouping of the BIO bits on the individual Acromag units). This will enable us to use the latch code with minimal modifications (it was a pain to debug this the first time around). I don't see any major constraint in the wiring assignment that'd make this difficult.

Turns out the 35MHz sidebands are way too weak to resolve from the resonance when doing a PZT scan.

I connect the IFR2023B function generator on the PSL table to the EOM instead of the FSS box and set it to generate 150MHz at 13dbm.

To observe the resulting weak sideband I place a PDA55 at the peak-off path from the transmission of the PMC where there is much more light than the transmission of the PMC head mirror. Whoever is using this path there is a PD blocking it right now.

I do a PZT scan by connecting a triangular signal to the EXT DC on the PMC servo with and without the EOM (Attachment 1). A weak sideband can clearly be spotted now.

Using the above 150MHz sideband calibration I can find the roundtrip time to be 1.55ns.

I take a high-resolution scan of a resonance peak and fit it to a Lorentzian (Attachment 2) and find a roundtrip loss of 1.3%.

Using the above results the cavity decay time is 119ns.

We should investigate what's going on with the ringdown measurements.

Done.

[Jon, Yehonathan]

We've begun assembling the new c1psl Acromag chassis based on Yehonathan's final pin assignments. So far, parts have been gathered and the chassis itself has been assembled.

Yehonathan is currently wiring up the chassis power and Ethernet feedthroughs, following my wiring diagram from previous assemblies. Once the Acromag units are powered, I will help configure them, assign IPs, etc. We will then turn the wiring over to Chub to complete the Acromag to breakout board wiring.

I began setting up the host server, but immediately hit a problem: We seem to have no more memory cards or solid-state drives, despite having two more SuperMicro servers. I ordered enough RAM cards and drives to finish both machines. They will hopefully arrive tomorrow.

RTFE. Where did the spares go?

I found them, thanks. After c1psl, there are 4 2GB DIMM cards and 1 SSD left. I moved them into the storage bins with all the other Acromag parts.

I've assembled a new SuperMicro rackmount machine to replace c1psl. It is currently set up on the electronics bench.

• OS: Debian 10.2
• Hostname: c1psl1 (will become c1psl after installation)
• IP: 192.168.113.54 (registered in the martian DNS)
• Network drive mount point set up (/cvs/cds), which provides all the EPICS executables.

PSL controls on the sitemap went blank. Rebooted c1psl. PSL screens seem normal again.

{Yehonathan, Jon}

I finished pre-wiring the PSL chassis. I mounted the Acromags on the DIN rails and labeled them. I checked that they are powered up with the right voltage +24V and that the LEDs behave as expected.

[Yehonathan, Jon, Shruti]

Since the PMC would not lock, we initially burt-restored the c1psl machine to the last available shapshot (Dec 10th 2019), but it still would not lock.

Then, it was burt-restored to midnight of Dec 1st, 2019, after which it could be locked.

{Yehonathan, Jon}

I configured the Acromag channels according to the Slow Controls Wiki page.

We started testing the channels. Almost at the beginning we notice that the BIO channels are inverted. High voltage when 0. 0 Voltage when 1. We checked several things:

1. We checked the configuration of the BIOs in the windows machine but nothing pointed to the problem.

2. We isolated one of the BIOs from the DIN rail but the behavior persisted.

3. We checked that the voltages that go into the Acromags are correct.

The next step is to power up an isolated Acromag directly from the power supply. This will tell us if the problem is in the chassis or the EPICs DB.

As per Gautam's request, I list the changes that were made since he left:

1. The AOM driver was connected to a signal generator.

2. The first order beam from the AOM was coupled into the PMC while the zero-order beam is blocked. We might want to keep this configuration if the pointing stability is adequate.

3. c1psl got Burt restored to Dec 1st.

4. Megatron got updated.

Currently, c1susaux seems unresponsive and needs to be rebooted.

{Yehonathan, Jon}

I isolated a BIO Acromag completely from the chassis and powered it up. The inverted behavior persisted.

Turns out this is normal behavior for the XT1111 model.

For digital outputs, one should XT1121. XT1111 should be used for digital inputs.

Slow machines Wiki page was updated along with other pieces of information.

I replaced the XT1111 Acromags with XT1121 and did some rewiring since the XT1121 cannot get the excitation voltage from the DIN rail.

I added an XT1111 Acromag for the single digital input we have in this system.

Summary:

There was no light entering the IFO. I worked on a few things to bring the interferometer to a somewhat usable state. The goal is to get back to PRFPMI locking ASAP.

Details:

Problem: All fast models report a "0x4000" DC error. See Attachment #1.

Solution: I think this is a "known" issue that happened last new year too. The fix was to add a hard-coded 1 second offset to the daqd config files. However, incrementing/decreasing this offset by +/- 1 second did not fix the errors for me today. I'll reach out to JH for more troubleshooting tips.

Update 15 Jan 2020 830am: The problem is now fixed. See here.

Problem: c1susaux and c1auxey were unresponsive.

Solution: Keyed c1auxey. Rebooted c1susaux and as usual, manually started the eth0/eth1 subnets. The Acromag crate did not have to be power-cycled. ITMY got stuck in this process - I released it using the usual bias jiggling. Why did c1susaux fail? When did it fail? Was there some un-elogged cable jiggling in that part of the lab?

Problem: IMC autolocker and FSS slow processes aren't running on megatron after the upgrade.

Solution: Since no one bothered to do this, I setup systemd infrastructure for doing this on megatron. To run these, you do:

and to check their status, use:

The systemd setup is currently done in a naive way (using the bash executable to run a series of commands rather than using the systemd infrastructure itself to setup variables etc) but it works. I confirmed that the autolocker can re-acquire IMC lock, and that the FSS loop only runs when the IMC is locked. I also removed the obsolete messages printed to megatron's console (by editing /etc/motd) on ssh-login, advising the usage of initctl - the updated message reflects the above instructions.

In order to do the IMC locking, I changed the DC voltage to the AOM to +1V DC (it was +0.8 V DC). In this setting, the IMC refl level is ~3.6 V DC. When using the undiffracted AOM beam, we had more like +5.6 V DC (so now we have ~65% of the nominal level) from the IMC REFL PD when the IMC was unlocked. IIRC, the diffraction efficiency of the AOM should be somewhat better, at ~85%. Needs investigation, or better yet, let's just go back to the old configuration of using the undiffracted beam.

There was also an UN-ELOGGED change of the nominal value of the PMC servo gain to 12.8, and no transfer function measurement. There needs to be a proper characterization of this loop done to decide what the new nominal value should be.

I'm going to leave the PSL shutter open and let the IMC stay locked for stability investigations. Tomorrow, I'll check the single-arm locking and the ALS system.

Summary:

Every new year (on Dec 31 or Jan 1), all of the realtime models will report a "0x4000" error. This happens due to an offset to the GPStime driver not being updated. Here is how this can be fixed (slightly modified version of what was done at LASTI).

Steps to fix the DC errors:

1. ssh into FB machine. 
2. Edit the file /opt/rtcds/rtscore/release/src/include/drv/spectracomGPS.c:
   • Look for the code block with a text string that reads something like

     /* 2019 had 365 days and no leap seconds */
                  pHardware->gpsOffset += 31536000;

   • Copy and paste the above string for the appropriate number of years of offset you are adding, and edit the comment string appropriately!.
3. Navigate to /opt/rtcds/rtscore/release/src/drv/symmetricom. Run the following commands:

   sudo make
   sudo make install

4. Stop all the daqd processes and reload symmetricom:

   sudo systemctl daqd_* stop
   sudo modprobe -r symmetricom
   sudo modprobe symmetricom

5. Re-start the daqd processes:

   sudo service daqd_* start

Independent of this, there is a 1 second offset between the gpstimes reported by /proc/gps and gpstime. However, this doesn't seem to drift. We had effected a static offset to correct for this in the daqd config files, and it looks like these do not need to be updated on a yearly basis. All the daqd indicators are now green, see Attachment #1.

Single arm locking using POX and POY has been restored. After running the dither alignment servos, the TRX/TRY levels are ~0.7. This is consistent with the IMC transmission being ~11000 counts with the AOM 1st order diffracted beam (c.f. 15000 counts with the undiffracted beam).

I don't think this is an accurate statement. XT1111 modules have sinking digital outputs, while XT1121 modules have sourcing digital outputs. Depending on the requirement, the appropriate units should be used. I believe the XT1111 is the appropriate choice for most of our circuits.