I measured the RF response of the fiber-coupled NewFocus 1611, calibrating out the cable delay. The laser current was set to 20.0 mA, and the RF power going into the splitter was -10 dBm. The DC voltage was 1.87 V, and Gautam and I measured the power from the fiber at 344uW.

Something still looks very wrong -- the PD is supposed to be flat out to 1GHz, and physical units pending, need food.

The 1GHz PD has a bit more flat response, but the laser and the driving network have more frequency dependence as you saw.

I think the metric of interest here is the consistency of the AC transimpedance of the proposed new "Reference PD" (= fiber coupled NF1611) vs the old reference (free space NF1611), since everything will be calibrated against that.

Summary:

The fiber-coupled PD seems to have a factor of ~1.5 difference in responsivity compared to the free-space PD. There are some differences in the two ways I made the measurement that I don't yet understand.

Details

I measured relative responsivities of the fiber and free coupled NewFocus 1611 PDs (scaled by the Jenne AM transfer function).

I made the measurement in two ways, see attachment three. In attachment one, I show the response for separately measuring the two PDs relative to a pickoff of the source (two-port thru calibration). In attachment two I measure the relative responses directly, without picking off a reference (three-port calibration). I scaled the transfer functions by their DC voltages; both PDs have transimpedances of 700 V/A.

However, there are some clear differences in the response (overall factor of 0.5dB offset that may be explained by a miscalibrated DC level; apparent periodicity in attachment 1) that I don't yet understand.The free path of the non-fiber PD is ~5-6 inches, which accounts for the ~45 degrees of phase advance of the fiber relative to free coupled PD signal. (12.7cm / (c / 300 MHz) * 360 degrees ~ 45 degrees)

[Jon, Yehonathan, Gautam, Aaron, Shruti, Koji]

We get together on Wednesday afternoon for cleaning the lab. Particularly, we collected e-wastes: VME crates, VME modules, old slow control cables, and other old/broken electronics. They are piled up in the office area and the cage outside rioght now (Attachments 1/2). We asked Liz to come to pick them up (under the coordination with either Gautam or Koji). Eventually this will free up two office desks.

Also, we made the acromag components organized in plastic boxes. (Attachment 3)

The UPS is now incessantly beeping. I cannot handle this constant sound so I shut down all the control room workstations and moved the power strip hosting the 4 CPUs to a wall socket for tonight. Chub and I will replace the UPS batteries tomorrow.

[Liz, Gautam, Chub, Jordan, Koji]

We removed a significant amount of e-waste from the lab. The garbage was moved to the e-waste station in WB SB and are waiting for disposal.

Batteries + power cables replaced, and computers back on UPS from today ~3pm.

Some ideas that would help increase the locking duty-cycle in the short term. 

1. Seismometer investigation - something is not quite right with the vertex seismometer. This is the one that is primarily used for feedforward, and can be really helpful.
2. Drifting TTs - it is really annoying to have to re-set the input pointing into the interferometer every ~ hour. See Attachment #1.
3. FSS - this isn't a scientific statement, but there were ~20-30 minute periods last night where the PC drive RMS was displaying sharp spikes repeating every 2-3 seconds, first with increasing and then decreasing height. This is a new feature to me in the long standing PC drive saga but it doesn't tell me exactly what is going on as I don't know in what frequency band the glitch is actually happening. See Attachment #2.
4. ALS noise - while it is possible now to routinely transition the arm length control from the POX/POY to CARM/DARM basis, I see some sharp (<0.1 s) dives in the TRX/TRY levels when the arms are under ALS control. This wasn't present a week ago. Needs to be investigated - I defer this to the daytime tomorrow.

I worked on the setup up for the phase modulation measurement of the X end NPRO PZT. A previous similar measurement can be found here (12077). The setup was assembled based on the schematic in Attachment1.

Mixer used: Level 7, Mini circuits ZP-3+
LPF: up to 1.9MHz

Cables exiting the PSL table:
1. LO (Marconi -> Mixer)
2. RF (PSL+X beat note -> Mixer) The cable for this was taken from the Beat Mouth (otherwise connected to the oscilloscope)
3. Ext modulator (SR560 -> Marconi)

The long cable labled 'X Green Beat' was used to connect to the PZT (from the network analyzer).

Observations: The beat note kept floating between 0 and ~100 MHz

The PLL part of the circuit was tested coarsely with the spectrum analyzer function of the Agilent, where the loop was seen to stabilize when the carrier frequency of the Marconi was close to the instantaneous beat frequency.

Were some cables from the ALS beat setup modified? I can't see the beat on the scope, and this elog doesn't say anything about cable connection rearrangement. At ~2311, I am reverting the setup to as it should be.

I symlinked the SRmeasure and AGmeasure commands to /usr/bin/ on donatella (as it is done on pianosa) so that these scripts are in $PATH and may be run without having to navigate to the labutils directory.

In short:

Steps undertaken:

1. Normally I would unlock the IMC (Disabling the servo between the 'Filter' and 'Polarity' on the Mode Cleaner Servo Screen), but today I did not have to since Rana had kept it unlocked.

2. Misaligned the ITMX. This is to prevent cavity resonances from returning to the laser

3. Turned up the air on the HEPA at the PSL table to 100% during the measurement

4. Cables were connected as before (diagram shown in attachment of elog 15069)

5. The X end laser NPRO was actuated for the TF measurement using a long cable connected to TO AUX_X LASER PZT

Thoughts and observations:

- Reading out the error signal after amplification cannot distinguish between a locked loop or one out of its range. The error signal would be very small in both cases.

- Looking at the beat note on an oscilloscope, there also seemed to be an additional amplitude modulation that I had not noticed earlier. Rana suggested that it may have something to do with the pre-mode cleaner and the AOM being driven at 80 MHz

- Even though the TF was attempted, it seemed too noisy, suggesting that the PLL did not seem to work

- Rana also suggested that it may be a better idea to use the PZT of one of the lasers as the VCO for the PLL feedback instead of the Marconi.

1. Some calculations

For a Unity Gain Frequency (UGF) of 1 kHz, assumed PZT response  of 1 MHz/V, Mixer response of 25 mV/ rad, the required gain of the amplifier is

G ~ 0.8

2. Progress

- Measured the mixer response

Measuring mixer response:

- PSL laser temperature was adjusted so that beat frequency was roughly 25 MHz and the amplitude was found to be roughly -30dBm.

- At the RF port instead of the beat signal, a signal of 25 MHz + few kHz at -30 dBm was inputted. The LO was a 25 MHz signal was sent from the Marconi at 7 dBm.

- The mixer output was measured, with setup as in Attachment 1  Figure (A), on an oscilloscope. The slope near the small angle region of the sine curve would be the gain (in V/rad) and was found to be:  rad

- Since from the above calculations it seemed like an amplifer gain of 1 should work for the PLL, I rearranged the set up as in Figure (B) of Attachment 1 to actuate the X end NPRO PZT, I adjusted the PSL temperature (slow control) to try and match the frequency to 25 MHz, but couldn't lock the loop. I was monitoring the error signal after amplification (50 ohm output of the SR 560) which showed oscillations when the beat frequency was near 25 MHz and nothing significant otherwise.

- I used a 20 dB attenuator at the amplifier output and saw the beat note oscillate for longer, but maybe because it was a 50 ohm component in a high impedance channel it did not work either (?). I tried other attenuator combinations with no better luck.

- Is there a better location to add the attenuator? Should I pursue amplifying the beat signal instead?

- Also, it seemed like the beat note drift was higher than earlier. Could it be because the PMC was unlocked?

Summary:

There was no light entering the IFO. I worked on a few things to bring the interferometer to a somewhat usable state. The goal is to get back to PRFPMI locking ASAP.

Details:

Problem: All fast models report a "0x4000" DC error. See Attachment #1.

Solution: I think this is a "known" issue that happened last new year too. The fix was to add a hard-coded 1 second offset to the daqd config files. However, incrementing/decreasing this offset by +/- 1 second did not fix the errors for me today. I'll reach out to JH for more troubleshooting tips.

Update 15 Jan 2020 830am: The problem is now fixed. See here.

Problem: c1susaux and c1auxey were unresponsive.

Solution: Keyed c1auxey. Rebooted c1susaux and as usual, manually started the eth0/eth1 subnets. The Acromag crate did not have to be power-cycled. ITMY got stuck in this process - I released it using the usual bias jiggling. Why did c1susaux fail? When did it fail? Was there some un-elogged cable jiggling in that part of the lab?

Problem: IMC autolocker and FSS slow processes aren't running on megatron after the upgrade.

Solution: Since no one bothered to do this, I setup systemd infrastructure for doing this on megatron. To run these, you do:

and to check their status, use:

The systemd setup is currently done in a naive way (using the bash executable to run a series of commands rather than using the systemd infrastructure itself to setup variables etc) but it works. I confirmed that the autolocker can re-acquire IMC lock, and that the FSS loop only runs when the IMC is locked. I also removed the obsolete messages printed to megatron's console (by editing /etc/motd) on ssh-login, advising the usage of initctl - the updated message reflects the above instructions.

In order to do the IMC locking, I changed the DC voltage to the AOM to +1V DC (it was +0.8 V DC). In this setting, the IMC refl level is ~3.6 V DC. When using the undiffracted AOM beam, we had more like +5.6 V DC (so now we have ~65% of the nominal level) from the IMC REFL PD when the IMC was unlocked. IIRC, the diffraction efficiency of the AOM should be somewhat better, at ~85%. Needs investigation, or better yet, let's just go back to the old configuration of using the undiffracted beam.

There was also an UN-ELOGGED change of the nominal value of the PMC servo gain to 12.8, and no transfer function measurement. There needs to be a proper characterization of this loop done to decide what the new nominal value should be.

I'm going to leave the PSL shutter open and let the IMC stay locked for stability investigations. Tomorrow, I'll check the single-arm locking and the ALS system.

Single arm locking using POX and POY has been restored. After running the dither alignment servos, the TRX/TRY levels are ~0.7. This is consistent with the IMC transmission being ~11000 counts with the AOM 1st order diffracted beam (c.f. 15000 counts with the undiffracted beam).

Per Yehonathan's request, I removed one PDA10CF from a pickoff of REFL on the AS table (it was being used for the mode spectroscopy project). I placed a razor beam dump where the PD used to be, so that when the PRM is aligned, this pickoff is dumped. This is so that team ringdowns can use a fast PD.

With Gautam's help today the PLL managed to be be locked for a few brief moments. Turns out the signal power of the beat was an issue.

What was changed prior to/ during the experiment:

1. The PSL shutter was closed so not light goes into the input mode cleaner.

2. HEPA turned up (will be turned back down to ~30%)

3. AOM driver offset voltage decreased from 1V to ~100 mV (this will be reverted to 1V by the end of today). This increases the beat signal by deflecting the zeroth order beam to create the beat.

4. Output of servo SR 560 sent to the PZT of the X NPRO laser (the cable was disconnected from the pomona box at the X end)

5. The SR560, mixer, LPF and cables required for connections were moved into the PSL enclosure.

6. The error and control signals were hooked up to the oscilloscope where the beat outputs were visible (the setup has been reverted back to the original).

Elog 14687 has a detailed description of the conditions that provide a stable lock. I was told that the PI controller (LB1005) may be a better servo than the SR560, but today it was not used.

1) Parameters during the more successful attempts:

LPF: 5 MHz, Mixer: ZP-3+

Gain set at SR560: varied, but generally 200

Filter at SR560: 1 Hz low pass (single pole? at least by the label)

2) The LO had to be very close (<2 MHz) to the beat frequency in order to achieve a lock for ~30s

gautam edits:

• the error signal for the PLL was being sourced from the 20dB coupled port on the BeatMouth™.
• additionally, most of the power in the PSL beam coupled into the fiber was being deflected into the first order beam by team ringdown.
• The Vpp of the mixer output (when using the coupled beat and low PSL beam power) was a paltry 5-10 mVpp .
• I suggested using the direct NF1611 output for this measurement instead of the coupled output (alternatively, use an amp). it's probably also better to use the LB1005 for locking the PLL, long term, this can be set up to be controlled remotely, and a slow PID servo can be used to extend the duration of the lock by servoing either the marconi carrier freq or the EX temp ctrl.

Setup Update:

- No more SR 560, upgraded to LB1005 P-I controller.  Because: Elog 14687. Schematic of new setup shown in Attachment 1.

- For this, the Marconi was moved to the other (east) side of the PSL table and a power supply was also placed in the enclosure.

I think that the RF power at the mixer in this new configuration is 0 dBm (since the spectrum analyzer read ~ -20 dBm)

Progress Today:

- Turned up the HEPA to 100%, closed the PSL shutter, misaligned the ITMX, connected the LB1005 to the PZT. [The PZT has been reconnected to the X arm PDH servo, HEPA back to 20-30%]

- Tried to look for the PSL+X beat, but it was not there. Gautam identified the flipmount in the path which sorted it out (eventually), but there was no elog about it.

- After much trial, the loop seemed to lock with PI corner 1 kHz, gain ~2.9 (as read on knob), LFGL set to 90 dB. The beat note looked quite stable on the oscilloscope, but the error signal had an rms of ~100 mV (Rana pointed out that it could be the laser noise) and the lock lasted for ~1 min each time.

The parameters were similar to that in elog 14687. Why do we require such a high PI corner frequency and LFGL?

The 24 V Sorenson (2nd from bottom) in the small rack west of 1x2 was repurposed to 12V 600 mA, and was run to a terminal block on the north side of 1X1. Cables were routed underneath 1X1 and 1X2 to the terminal blocks. 12V was then routed to the PSL table and banana clip terminals were added.

Over the past few days, I have been trying to make measurements of the phase modulation transfer function by modulating the X end laser PZT via PLL.

The setup was modified every time during the experiment in the same manner as mentioned in elog 15148.

I could not make the PLL lock for long enough to take a proper TF measurement, resulting in TFs that look like Attachment 1. The next step would be to use the method of a delay line frequency discriminator instead of the PLL.

Comments about locking with LB1005 PI controller:

1. I do not understand why the high PI corner frequency of 1kHz or 3kHz was required to lock.
2. The rms level of the error signal when locked was ~100 mV, which is 25% of the total mixer range (~400 mVpp). Decreasing the gain only caused the loop to go out of lock and did not decrease this noise in the error signal.
3. The setup was also partly inside the PSL enclosure, with the HEPA turned to 100%, which is probably a noisy environment for this measurement. Closing and opening the shutters or any disturbance near the enclosure resulted in movement of the beat note up to 5 MHz.
4. It may have been a better idea to actuate the PSL laser instead of the X NPRO because of its larger range, but would this solve the issue with the noise?

 Today I began working on a TF measurement based on the delay line frequency discriminator setup in elog 4254 using a single mixer (without the 'I' and 'Q' readout).

For this, I re-organised the setup for the PLL measurement of the transfer function (elog 15148), increasing the HEPA for the initial changes while the PSL door was open, and then reverting it back to ~30%:

• I removed the 20dB coupler and connected the splitter directly after the amplifer to split the beat note signal into two coaxial cables one of which was ~1.5m longer than the other
• The recombined signals were combined in a mixer outside the PSL enclosure. I also replaced the 1.9 MHz LPF with a 5 MHz LPF.
• I used an SR 560 to amplify the signal after the LPF.

With the above setup the power that was seen at each channel of the delay line was <1dBm, which is not ideal for the any of the available mixers.

After the group meeting, I changed the amplifer to ZHL-3A (that is near the beat mouth) instead of a ZFL-500HLN because it had a higher gain (~28dB as opposed to ~19dB of the latter). The power seen at each of the delay line channels is over 5.5 dBm. This is consistent with the estimation 0 dBm beat -> -20 dBm after 20dB coupler -> 8 dBm after amplifier -> 5 dBm after splitter with insertion loss of 3 dB.

Is this sufficient enough for the mixer to work? In Attachment 1: A shows the mixer output (point B in Attachment 2) when the IMC is locked, in B the IMC is unlocked at the middle of the spectrum, and each of the dips show the DC voltage being sent to the PSL temperature servo being decreased by 0.01 V.

Gautam pointed me to the location of a few other RF amplifiers (ZHL-32A+, ZHL-1A) which don't possess a higher gain but can be used without disrupting the ALS related work (I was told).

For shorter duration changes that I made later, I opened and closed the PSL enclosure doors without changing the HEPA.

Attachment 2 shows the current setup as is, but I might add a PSL servo tomorrow to stabilise its frequency corresponding to a null mixer output without driving anything else.

yes, its fine to use this with a level 3 or level 7 mixer; let's see some PM transfer functions !

I could not find any level 3 mixers, but by adjusting the beat frequency the power in each of the delay line channels rose to almost 6.5 dBm.

In short: Delay line seems to work

Things I did earlier today:

1. Played with the slow servo on the FSS screen, but then reset the parameters to what was there before (Later found out that this was to lock the PSL freq to the IMC when the IMC power is significant.)
2. Connected the AG 4395A to the X PZT
3. Closed the PSL shutter

Transfer function measurement: (Refer Attachment 1)

Everything about the setup remained as I had left it earlier: described in elog 15174

except

• SR560 gain set to 10, DC coupled
• DC block at channel A of Agilent (The measurement was A/R)

I did not use a slow servo, but took individual sweeps adjusting the PSL temperature each time to bring the error voltage between +/-25 mV. The beat frequency was over 100 MHz.

For the plot posted in Attachment 1, the measurement paramters are the following. Will do further measurements/analysis tomorrow.

# AG4395A Measurement - Timestamp: Jan 30 2020 - 21:58:00
# Parameter File: TFAG4395Atemplate.yml
#---------- Measurement Parameters ------------
# Start Frequency (Hz): 50000.0, 50000.0
# Stop Frequency (Hz): 1000000.0, 1000000.0
# Frequency Points: 801, 801
# Measurement Format: LOGM, PHAS
# Measuremed Input: AR, AR
#---------- Analyzer Settings ----------
# Number of Averages: 1
# Auto Bandwidth: Off, Off
# IF Bandwidth: 1000.0, 1000.0
# Input Attenuators (R,A,B): 0dB 0dB 0dB 
# Excitation amplitude = -20.0dBm

Jon brought over a box of parts for constructing the metal PMCs. I have stored it along the Y-arm, on top of the green optics cabinet.

I didn't do an exhaustive inventory check, but the following are the rough contents of the box:

• 41 deg AoI flat mirrors, R=99% @ 1064nm --- 11 pcs
• 6.8 deg AoI curved mirrors --- 5 pcs
• PZTs --- 3pcs
• Metal PMC body --- 2 pcs
• "Baked PZT endcaps" --- 3 pcs
• Ball bearings, clamps, misc hardware

I didn't inspect the optics but since we have so many, I am hoping we can find 3 good quality ones for one cavity at least. We should check that the geometry is suitable for our RF sideband frequencies.  

I changed the office area thermostate near Steve's desk from 68F to 73F today. Please do not change it.

If anyone from facilities comes to adjust something, please put the details in the elog on the same day so that we can know to undo that change rather than chase down other drifts in the system.

Not sure what's wrong, but the workstation desk is freezing cold again and the room temp is 18degC (64degF).

I took a few AM TF measurements at the X end for which I:

• Misaligned the ITMX (then re-aligned it)
• Opened the X green shutter during the measurements and closed it at the end
• Moved the Agilent from the PSL area to the X end, the delay line and mixer still remains near the PSL area (will move it soon)
• Took a bunch of TFs

I will post the data soon.

I took the metal PMC box and examined its content and find the following items:

There seem to be enough parts to build 2 PMCs + spares.

I find several problems in the metal PMCs:

PMC1 has a broken screw in one of its flat mirror mounts (Attachment 16). We need to get it out in the machine shop.

PMC2 one of the flat mirrors has a scratch on the AR coating and its ORing is failing (Attachment 17). Mirror and ORing need to be replaced.

I measure the physical dimensions of the PMC with the help of https://dcc.ligo.org/LIGO-E1400332. The roundtrip is found to be 24cm which gives an FSR of 1.25GHz.

I use Evan Hall's Python script for calculating the mode spectrum as a function of the cavity length of the metal PMC and overlay the RF sidebands (Green dashed lines) on it (Attachment 18) to check for any HOM coincidence. The width of the lines is the mode splitting due to the cavity astigmatism.

It seems like the only issue might come from a 10th order modes (green ribbon) which are hopefully small enough in reality.

On February 5, 2020, the Dell engineering workstation located in the 40M lab, was replaced with a newer Engineering workstation, per a request from Koji . The new workstation should perform a good deal better over the older unit. It has more cores, more memory and a better video card. Since this unit is being used by the 40M group, the Comsol s/w pkg. was also installed on the unit.

During the computer swap, Koji had a problem with a print job and it was discovered the bottom tray of the HP5550 printer was broken. The broken tray was replaced from another unit that was being disposed of.

Ran HDMI to the new tv mounted on the north wall of control room.

The HVAC people replaced a valve and repaired the pneumatic plumbing on the roof air handler.  Temperature has been stable during the day since Thursday.  If anyone is in the control room during the evening, please make a note of the temperature.

Chub

Came this morning to find the PMC was unlocked since 6AM. Laser is still on, but PMC REFL PD DC shows dead white constant 0V on PMC screen. All the controls on the PMC screen show constant 0V actually except for the PMC_ERR_OUTPUT which is a fast channel.

Is PSL Acromag already failing?

I restarted the IOC but it didn't help.

I am now rebooting c1psl... That seemed to help. PMC screen seem to be working again. I am able to lock the PMC now.

IMC was locking easily once some switches on the MC servo screen were put to normal states.

TTs were grossly misaligned. Onces they where aligned, arm cavities were locking easily. Dither align for the X arm is very slow though...

Of course the reboot wiped any logs we could have used for clues as to what happened. Next time it'll be good to preserve this info. I suspect the local subnet went down.

P.S. for some reason the system logs are priveleged now - I ran sudo sysctl kernel.dmesg_restrict=0 on c1psl to make it readable by any user. This change won't persist on reboot.

Yesterday evening I took nearly all of the masks, gloves, gowns, alcohol wipes, hats, and shoe covers. These were the ones in the cleanroom cabinets at the east end of the Y-arm, as well as the many boxes under the yarm near those cabinets.

This photo album shows the stuff, plus some other random photos I took around the same time (6-7 PM) of the state of parts of the lab.

[Larry (on site), Koji & Gautam (remote)]

Network recovery (Larry/KA)

• Asked Larry to get into the lab. 

• 14:30 Larry went to the lab office area. He restarted (power cycled) the edge-switch (on the rack next to the printer). This recovered the ssh-access to nodus. 

• Also Larry turned on the CAD WS. Koji confirmed the remote access to the CAD WS.

Nodus recovery (KA)

• Apr 12, 22:43 nodus was restarted.

• Apache (dokuwiki, svn, etc) recovered along with the systemctl command on wiki

• ELOG recovered by running the script

Control Machines / RT FE / Acromag server Status

• Judging by uptime, basically only the machines that are on UPS (all control room workstations + chiara) survived the power outage. All RT FEs are down. Apart from c1susaux, the acromag servers are back up (but the modbus processes have NOT been restarted yet). Vacuum machine is not visible on the network (could just be a networking issue and the local subnet to valves/pumps is connected, but no way to tell remotely).

• KA imagines that FB took some finite time to come up. However, the RT machines required FB to download the OS. That made the RTs down. If so, what we need is to power cycle them.

• Acromag: unknown state

​The power was lost at Apr 12 22:39:42, according to the vacuum pressure log. The power loss was for a few min.

[Koji / Gautam (Remote)]

Lab status

• Gray Panel: The lab AC was off. Turned on all three (N/S, CTRL RM, E/W)
• The control room AC was running.

Work stations

• Control Room: All the control machines were running. We knew that nodus/chiara/fb were running
• 1X6/7:
  • JETSTOR was making beeping sound. “Power #1 failed””power #2 failed”
  • Optimus & megatron were off -> turned on -> up and running now
• 1X1/2:
  • ​Power cycled the netgear at the top of the IOO rack (maybe not necessary)
  • Turned on c1ioo -> up and running now
• 1X4/5: ​Rebooted c1sus / c1lsc -> up and running now
• 1X9: Rebooted c1iscex -> up and running now
• 1Y4: Rebooted c1iscex -> up and running now

Vacuum status

• Looked like everything was running as if it did not see the power glitch
• TP1 normal: Set speed 33.6k rpm / Actual speed 33.6k rpm 
• TP2 normal: 66k rpm / PTP2 16.0 mtorr
• TP3 normal: 31k rpm / PTP3 45.4mtorr
• P1 LOW / P2 1.7mtorr / CC2 1.1e-6 / P3 7.6e-2 / P4 LO
• Annuli: 2.7~3torr
• CC1 9.6e-6 / SUPER BEE 0.9mtorr

C1VAC recovery

• c1vac was alive, but was isolated from the martian network
• Checked the network I/F status with /sbin/ifconfig -a
  • eth0 had no IP
  • eth1 had the vac subnet IP (192.168.114.9)
• Ran sudo /sbin/ifdown eth0 then  sudo /sbin/ifup eth0
• The I/F eth0 started running and c1vac became visible from martian
• Later checked the vacuum screen: The pressure values and valve statuses looked normal.
  The interlock state was “running”. The system state was “unrecognized”.

End RTS recovery 

• The end slow machines (auxex and auxey) were already running
• Restarting end RT models:
  • c1iscey -> rtcds start --all
  • c1iscex -> rtcds start --all
• Confirmed that the models can dump the SUSs

Vertex RTS recovery

• We wanted to use the reboot script. (/opt/rtcds/caltech/c1/scripts/cds/rebootC1LSC.sh)
• c1susaux​​
  • To be safe, we wanted to bring c1susaux first.
  • c1susaux does not make the network I/Fs up automatically upon reboot.
    -> Connect an LCD display / keyboard / mouse to c1susaux
    -> Ran sudo /sbin/ifup eth0 and sudo /sbin/ifup eth1
  • Now c1susaux is visible from martian.
  • Login c1susaux and ran:  
    sudo systemctl start modbusIOC.service 
    -> c1susaux epics is up and running now
  • ...Meanwhile c1susaux lost its eth1 somehow. This made the slow values of 8 vertex sus all zero
    -> Ran sudo /sbin/ifdown eth1 and sudo /sbin/ifup eth1 again on c1susaux ->  this resolved the issue
• c1psl
  • Login c1psl and ran:  
    sudo systemctl start modbusIOC.service 
    -> c1psl epics is up and running now
• Prepared for the rebooting script
  • Ran /opt/rtcds/caltech/c1/scripts/cds/rebootC1LSC.sh
  • Rebooting was done successfully. All the suspensions looked free and healthy.
  • Burtrestored c1susaux (used Apr 12 21:19 snapshot)

Hardware

• PSL laser / Xend AUX laser / Yend AUX laser were off -> turned on
• The PMC was immediately automatically locked.
• The main marconi was off -> forgot to turn on
• The end temp controllers for the SHG crystals were on but not enabled -> now enabled

RTS recovery ~ part 2

• FB: FB status of all the RTS models were still red
• Timing: c1x01/2/3/5 were 1 sec behind of FB and c1x04 was 2 sec behind
• -> Remedy:  https://nodus.ligo.caltech.edu:8081/40m/14349
  • Software rebooting of FB
  • Manually start the open-mx and mx services using
  • sudo systemctl start open-mx.service 
  • sudo systemctl start mx.service
  • Check that the system time returned by gpstime matches the gpstime reported by internet sources. e.g. http://leapsecond.com/java/gpsclock.htm
  • Manually start the daqd processes using
    sudo systemctl start daqd_*
• This made all the FB(FE) indicators green!
• Ran the reboot script again -> All green!

IMC recovery

• The IMC status was checked
• No autolocker, but it could be manually locked. i.e. MC1/2/3 were not so much misalignment
• Autolocker/Slow FSS recovery along with https://nodus.ligo.caltech.edu:8081/40m/15121
  • sudo systemctl start MCautolocker.service
  • sudo systemctl start FSSSlow.service
• Both of them failed to run
• Note by Gautam: The problem with the systemctl commands failing was that the NFS mount points weren’t mounted. Which in turn was because of the familiar /etc/resolv.conf problem. I added chiara to the namespace in this file, and then manually mounted the NFS mount points. This fixed the problem.
  Now the IMC is locked and the autolocker is left running.

Burt restore

• Used Apr 12 21:19 snapshot
• c1psl
• c1alsepics/c1assepics/c1asxepics/c1asyepics
• c1aux/c1auxex/c1auxey/
• c1iscaux/c1susaux
• This made REFL and AS beams back to the CCDs. As has small fringes.
• Y arm has small IR flashes as well as green flashes.

JETSTOR recovery

• JETSTOR was beeping. 
• Shutdown megatron
• Followed the instruction https://nodus.ligo.caltech.edu:8081/40m/13107
• This stopped beeping. Waiting for JETSTOR to come up -> In a minute, JETSTOR display became normal and all disks showed green.
• Bring megatron back up again

N2 bottle

• The left N2 bottle was empty. The right one had 1500PSI.
• Replaced the left bottle with the spare one in the room.
• Now the left one 2680PSI and the right one 1400PSI.

Closing

• Closed PSL/AUX laser shutters
• Turned off the lights in the lab, CTRL room, and the office.

​Remaining Issues

• [done] MCAutoLocker / FSSSlow scripts are not running
• The PRM alignment slider has no effect (although the PRM is aligned…) -> SLOW DAQ frozen???
• JETSTOR is not mounted on megatron [gautam mounted Jetstor on megatron on 4/18 at 2pm]

• Empty N2 replaced. 
• Logged back into zita and started the StripTool traces (even though we keep the TV off nowadays).
• c1susaux acro-crate power cycled to re-enable PRM suspension control (all other vertex optics also now respond to slow bias voltage sliders being moved).
• c1iscaux needed a hard reboot as it wasn’t seen on martian. I power cycled the crate for good measure.
• Marconi turned back on with correct frequency/amplitude.
• c0rga is now seen again on martian network. I re-enabled the RGA scanning so that it takes a scan every morning at 4am. 
• The forepumps for TP2/TP3 are noisier than I remember. The former has ~10,000 hrs on the clock. How often does the tip seal replacement need to happen?
• HV supplies for ASX/ASY PZTs re-energized.
• IFO re-aligned for locking.
• c1oaf and c1daf models restarted. c1oaf required the usual start/stop/start sequence to make the DAQ errors go away, and luckily the FE didn’t crash when the model was unloaded.
• POX/POY/PRMI 1f carrier/green locking all was smooth.
• For some reason, the PRC angular FF filters i trained no longer do anything good (but MCL is still good). collected 20mins of PRMI 1f locked data for investigations.

Two ITM spares (ITMU01/ITMU02) and five new PR3 mirrors (E1800089 Rev 7-1~Rev7-5) were transported to Downs for phasemap measurement

I was in the lab for Clean and Bake activities and I replaced an empty N2 tank. Left tank is at 2600 psi right tank at ~1300 psi.

Replaced empty N2 tank, left tank at ~2000 psi, right tank ~2600 psi.

I will be at the 40m, in the Clean and bake lab today from ~9am to ~3pm.

I will be at the 40m, in the Clean and bake lab today from ~9am to ~3pm.

I will be in the Clean and Bake lab today from 9:30am to 4pm.

I will be in the Clean and Bake lab from 10am to 4pm today. I will also replace an empty N2 cylinder.

I will be in the Clean and Bake lab today from 11am to 4pm.

I will be in the Clean and Bake lab today from 12pm to 4pm.

I will be at the 40m today at 10am to deliver optics to Downs and to replace the TP2 controller.

I replaced an empty N2 cylinder, there are now two empty tanks in the outside rack.

I will be at the 40m today from 9:30am to 4pm.