I will be at the 40m today at 10am to deliver optics to Downs and to replace the TP2 controller.

I replaced an empty N2 cylinder, there are now two empty tanks in the outside rack.

I will be at the 40m today from 9:30am to 4pm.

I will be at the 40m today from 9am to 3pm.

The four 4x25DSUB and single 8x25DSUB feedthrough flanges have arrived and will be picked up from the dock and brought to the 40M lab.

The PSL shutter was closed from the vacuum interlock trip. Today, I did the following:

• Re-aligned input beam to PMC to recover high transmission / low reflection.
• Re-set the LSC offsets.
• ETMX watchdog was tripped. Reset it.
• Opened the PSL shutter, IMC autolocker was able to lock the cavity almost immediately.
• Tested POX/POY locking, ran the ASS to maximize single arm transmission.

All looks good for now. I will probably get back to PRFPMI locking Monday.

I will be at the 40m today from 11am to 4pm.

I will be in the Clean and Bake lab today from 10am to 4pm.

I will be in the Clean and Bake lab from 11pm to 4pm

I will be in the Clean and Bake lab today from 11am to 4pm.

I will be in the Clean and Bake lab today from 11:30am to 4pm

I will be in the clean and bake lab today from 9am to 4pm.

I will be in the Clean and Bake lab from 9am to 4pm today.

As part of an ongoing effort to improve airflow in workspaces/bathrooms on campus, I have installed an air scrubber unit in each of the bathrooms at the 40m lab.

I will be in the Clean and Bake lab today from 8:30am to 4pm

I will be in the clean and bake lab today from 9am to 3pm.

I will be in the Clean and Bake lab today from 9am to 3pm

I will be in the Clean and Bake lab today from 9am to 4pm

I will be in the Clean and Bake Lab today from 9am to 4pm.

The emergency lamps above the exit sign on the NW entrance to the control room are on. I tried opening and closing the door, but it remains on. Probably nothing to worry about, but noting here anyway.

It happened before too. Doesn't it say it has occasional self-testing or something?

True - it is now not on anymore.

The (masked) tech accessed all areas in the lab (office area, control room, VEA) between ~230pm-3pm. The laser safety goggles he used have been kept aside for appropriate sanitaiton.

1. I found that an EPICS channel wasn't reset to the correct value by burtrestore after the FE bootfest yesterday.
   • This cost me the whole of last night, found it finally tonight. 
   • I'll try and modify the locking scripts to better capture such errors, but ideally, we should just use Guardian or something since it's made for this purpose already.
   • Anyways, tonight I was able to re-acquire the PRFPMI lock in a completely scripted way.
2. Locking CARM on POX remains out of reach.
   • I think this has to do with the fact that the zero-crossing of the CARM and REFL error signals are dependent on the 3f PRCL/MICH error point offsets.
   • So even if the DC gain is right, the fact that we use POX for the digital AO path and REFL for the analog AO path is leading to some conflict I think.
   • Ran out of energy tonight, I'll try again tomorrow.

The DQ channels of the ETM coils were active tonight, so I'll make the coil driver actuation budget over the next couple of days.

I will be in the Clean and Bake lab today from 9am to 4pm.

There was a power outage ~30 mins ago that knocked out CDS, PSL etc. The lights in the office area also flickered briefly. Working on recovery now. The elog was also down (since nodus presumably rebooted), I restarted the service just now. Vacuum status seems okay, even though the status string reads "Unrecognized".

The recovery was complete at 1830 local time. Curiously, the EX NPRO and the doubling oven temp controllers stayed on, usually they are taken out as well. Also, all the slow machines and associated Acromag crates survived. I guess the interruption was so fleeting that some devices survived.

The control room workstation, zita, which is responsible for the IFO status StripTool display on the large TV screen, has some display driver issues I think - it crashed twice when I tried to change the default display arrangement (large TV + small monitor). It also wants to update to Ubuntu 18.04 LTS, but I decided not to for the time being (it is running Ubuntu 16.04 LTS). Anyways, after a couple of power cycles, the wall StripTools are up once again.

A technician came to the lab today at ~4pm. He entered the VEA (with booties and googles), and also the clean and bake lab. The whole procedure lasted ~10 minutes. I did not follow him around, but was available in the control room throughout the process. I think the whole episode went without incident.

BTW, this guy didn't ring the doorbell, I just happened to be here when he came by. I don't know if this is usual practise - are we happy with the technicians entering the VEA and/or clean and bake labs without supervision? AFAIK, this wasn't scheduled.

My power at home winked out for a second this morning, but it looks like either nothing happened in the 40m lab or else it rode it out.

MC is locked - lost lock around 11:25 AM and then relocked.

2PM: Arrived at the 40m. Started the work for the coupling of the RF modulated LO beam into a fiber. -> I left the lab at 10:30 PM.

The fiber coupling setup for the phase-modulated beam was made right next to the PSL injection path. (See attachment 1)

• For the alignment of the beam, the main PSL path, including the alignment of the 2" PO mirror, has not been touched.
• There are two PO beams with the optical power of 0.8mW (left) and 1.6mW (right). Both had been blocked but the right one was designed to be used for PSL POS and ANG. For the fiber coupling, the right beam was used.
• The alignment/mode-matching work has been done with a short (2m?) fiber patch cable from Thorlabs. The fiber is the same as the one used for LO delivery.
• I tried to have a mode-matching telescope in the LO path. I ended up having no lens for the best result. The resulting transmitted power is 1.21mW out of 1.64mW incident (~74%). These powers were measured with the Ophir power meter. (Note that Thorlabs' fiber power meter indicated 1.0mW transmission.)

Some notes

• After the PSL activity, the IMC locking was checked to see if I messed up the PSL alignment. It locks fine and looks fine.
  • The input shutter (left closed after Jon's vacuum work?) was opened.
  • The alignment was not optimal and had some pitch misalignment (e.g. TEM03).
  • After some MC SUS alignment, the automatic locking of TEM00 was recovered. Mainly MC3 pitch was moved (+0.17).
  • I've consulted with Gautam and he thinks this is with the level of regular drift. The AS beam was visible.
• The IMC and MI were moving so much, but this seemed just the usual Saturday night Millikan shake.
• During the activity, the PSL HEPA was turned up to 100 and it was reverted to 33 after the work.
• I have been wearing a mask and gloves throughout the work there.

15:30
- PSL HEPA was running at 33% and is now at 100%
- South End HEPA was not on and is now running
- Yarm Portable HEPA was not running and is now running at max speed: the power was taken beneath the ITMY table. It is better to unplug it when one uses the IFO.
- Yend Portable HEPA was not running and is now running (presumably) at max speed

Particle Levels: (Not sure about the unit. The convention here is to multiply x10 of the reading)

Before running the HEPAs at their maximum
9/10/2020 15:30 / 0.3um 292180 / 0.5um 14420
(cf 9/5/2020 / 0.3um 94990 / 0.5um 6210)
==>
After running the HEPAs at their maximum
The number gradually went down and now became constant at about half of the initial values
9/10/2020 19:30 / 0.3um 124400 / 0.5um 7410

M4.5 EQ in LA 2020-09-19 06:38:46 (UTC) / -1d 23:38:46 (PDT) https://earthquake.usgs.gov/earthquakes/eventpage/ci38695658/executive

I only checked the watchdogs. All watchdogs were tripped. ITMX and ETMY seemed stuck (or have the OSEM magnet issue). They were left tripped. The watchdogs for the other SUSs were reloaded.

the seismometers obviously saturated during the EQ, but the accelerometers captured some of it. It looks like there's different saturation levels on different sensors.

Also, it seems the mounting of the MC2 accelerometers is not so good. There's some ~10-20 Hz resonance its mount that's showing up. Either its the MC2 chamber legs or the accelerometers are clamped poorly to the MC2 baseplate.

Sun Sep 20 00:02:36 2020 edit: fixed indexing error in plots

* also assuming that the sensors are correctly calibrated in the front end to 1 count = 1 um/s^2 (this is what's used in the summ pages)

the EQ was ~14 km south of Caltech and 17 km deep

I'm amazed at how much higher the noise is on the MC2 accelerometer. Is that really how much amplification of the ground motion we're getting? If so, its as if the MC has no vibration isolation from the ground in that band. We should put one set on the ground and make the more direct comparison of the spectra. Also, perhaps do some seismic FF using this sensor - I'm not sure how successful we've been in this band.

Attaching the coherence plot from ldvw.ligo.caltech.edu (apparently it has access to the 40m data, so we can use that as an alternative to dtt or python for remote analysis):

It would be interesting to see if we can use the ML based FF technology from this summer's SURF project by Nadia to increase the coherence by including some slow IMC alignment channels.

I supplied a bottle of hand soap. Don't put water in the bottle to dilute it as it makes the soap vulnarable for cotamination.

There were two SUSs which didn't look normal.

- ITMX was easily released by the bias slider -> Shake the pitch slider and while all the OSEM values are moving, turn on the damping control (with x10 large watchdog threshold)

- ETMY has UR OSEM 0V output. This means that there is no light. And this didn't change at all with the slider move.
- Went to the Y table and tried to look at the coils. It seems that the UR magnet is detached from the optic and stuck in the OSEM.

We need a vent to fix the suspension, but until then what we can do is to redistribute the POS/PIT/YAW actuations to the three coils.

While I stopped by the lab this morning to pick up some things, I took the opportunity to continue the recovery.

• IMC suspensions were sufficiently misaligned that the autolocker couldn't re-acqurie the lock. I manually recovered the alignment and now the IMC is locked again.
• ETMY illuminator was left ON, I turned it off. In the process, I modified the illuminator ON/OFF script to be compatible with python3, but unfortunately, it was written in a way that doesn't permit backward compatibility, so now the illuminators can't be turned ON/OFF via the MEDM screen on pianosa (since the default python is 2.7 on that machine). But it does work on rossa, which I'm using as my primary workstation now (hence the change).
• ITMX watchdog trip threshold was manually reset to the nominal value - the rampdown script was working, but the threshold was ~1400cts (normally ~200 cts) even at 1130am this morning (>12 hours after Koji's work yesterday evening), so I just accelerated the process.
• Suspension realignment - using a mix of green and IR beams and the various cameras/photodiodes as diagnostics, I roughly restored the alignment of all the suspensions, except ETMY. I can see IR resonances in the X arm now.

At some point, we should run the suspension eigenmode routine (kick optics, let them ringdown, measure peak locations and Qs) to confirm that the remaining suspensions are okay, will also help in actuation re-allocation efforts on ETMY. But I didn't do this today.

Leaving the lab at 1150.

See Attachment #1.

• The original ETMY actuation matrix was the naive one so I simply retuned everything by adding the butterfly mode appropriately.
• The cross coupling between the DOFs has not been characterized, but the local damping, Oplev loops, POX/POY locking, simpel Michelson locking, and ASS dither alignemnt all seem to work so I'm thinking this is good enough for now.
• A copy of the ETMYaux.db file was made, and the slow bias voltage was redistributed among the three available face OSEMs - note that the magnet polarity has to be taken into account.
• The series resistance on the slow path was reduced from 430 ohms, 1W to 100 ohms, 3W, to allow DC alignment of the cavity axis to the beam axis.
• Vertex Oplevs were re-centered on their respective QPDs after locking POX/POY and running the ASS dither alignment.
• The AS spot was a little low on the camera - presumably the SR2/SR3 got macroscopically misaligned. I re-centered the beam on the camera on the AS table (by moving the camera, the beam path was not disturbed).

The beam spot on ETMY looks weird (looks almost like a TEM10 mode), but the one on ITMY seems fine, see Attachment #2. Wonder what's going on there, maybe just a focusing problem?

I came to the lab. The control room AC was off -> Now it is on.

Here is the setting of the AC meant for continuous running

Gautam reported that the PSL HEPA stopped running (ELOG 15592). So I came in today and started troubleshooting.

It looks like that the AC power reaches the motors. However, both motors do not run. It looks like the problem exists in the capacitors, the motors, or both.
Parts specs can be found in the next ELOG.

Attachment 1 is the connection diagram of the HEPA. The AC power is distributed by the breaker panel. The PSL HEPA is assigned to use M22 breaker (Attachment 2). I checked the breaker switch and it was (and is) ON. The power goes to the junction box above the enclosure (Attachment 3). A couple of wires goes to the HEPA switch (right above the enclosure light switch) and the output goes to the variac. The inside of the junction box looked like this (Attachment 4).

By the way, the wires were just twisted and screwed into a metal threaded (but isolated) caps (Attachment 5). Is this legit? Shouldn't we use stronger crimping? Anyway, there was nothing wrong with the caps w.r.t the connection for now.

I could easily trace the power up to the variac. The variac output was just fine (Attachment 6). The cord goes from the variac to the junction box (and then HEPAs) looked scorched. The connection from the plug to HEPAs was still OK, but this should be eventually replaced. Right now the cable was unplugged after the following tests for the safety reason.

The junction box for each HEPA unit was opened to check the voltage. The supply voltage came to the junction boxes and it was just fine. In Attachments 8 & 9, the voltages look low but this is because I just turned the variac only a little.

At the (main) junction box, the resistances of the HEPAs were checked with the Fluke. As the HEPA units are connected to the AC in parallel, the resistances were individually checked as follows.

The coils were not disconnected (... I wonder if the wiring of South HEPA was flipped? But this is not the main issue right now.)
 

By removing the pre-filters, the motors were inspected Attachments 10 & 11. At least the north HEPA motor was warm, indicating there was some current before. A capacitor was connected per motor. When the variac was tuned up a bit, one side of the capacitor could see the voltage. I could not judge which has the issue between the capacitor and the motor.

Dimensions / Specs

- HEPA unit dimentions
- HEPA unit manufacturer
- Motor
- Capacitor

I was in the lab from 1630-1830. I have located and visually inspected all the parts required for the magnet regluing / optic cleaning parts of the planned vent, except the fresh batches of scpectroscopic grade solvents. I was in the cleanroom part of the clean and bake lab from 1630-1700.

[JV, GV]

The HEPAs work again. After running the HEPAs for ~1 hour, I checked the particle count on the PSL table - the meter registered 0 for both 0.3 um and 0.5 um. So I decided to turn the NPRO back on, at ~1730 local time. The PMC and IMC were readily locke, so the basic interferometer functionality is returned, and we can now go ahead with (i) vent prep (ii) air BHD tests and (iii) IMC debuggin as was discussed on the call today. The earth is shaking, but nothing serious so far, I will resume alignment of the interferometer later in the evening when hopefully things have calmed down a bit more...

Procedure:

1. Turned off mains switch on the NW corner of the PSL enclosure. Then, disconnected the power cables from mains to Variac and from Variac to HEPAs. Made sure both HEPAs were set to "OFF".
2. With confidence that no AC power was reaching the motors, I removed the pre-filters, and removed the old startup capacitors. These don't have a polarity, but I marked the cable that was connected to the left terminal of the capacitor when the cap is viewed with the label facing you, in the interest of changing as few things as possible.
3. The two new capacitors were measured with the LCR meter - the meter registered ~7.5 uF, as expected. Unsurprisingly, the old capacitors that were removed didn't register any reading on the LCR meter. The terminals weren't shorted, but I don't know what the failure mode for this kind of capacitor is.
4. The two new capacitors were installed. Then, I tested the system by undoing all the changes in bullet #1. We found that the Variac needs to be set to 100% for the motors to startup.
5. The motor speed was found to vary as the Variac dial was turned. FWIW, at the "nominal" setting of 33% on the Variac (when we run the interferometer), I could see both fan blades were turning, but the flow was low enough that you couldn't hear any wind (at least, neither Jordan nor I could).
6. Turned off the mains agian, and cleaned up - restored the insulating rubber sleeve on the capacitor leads, and re-installed the pre-filters on the HEPA blowers. Then we turned both blowers back on. 

Note that the many other issues Koji noted in the preceeding elog (e.g. flaky wiring) have not been addressed.

Flow measurements:

Chub kindly provided us with an electronic anemometer. With the meter held directly against the HEPA filter inside the enclosure, we measured ~700 cfm of airflow on each of the two HEPAs, with the Variac set to 100% and the HEPAs themselves set to "High". With the Variac at 50%, the flow drops to ~160 cfm. At the nominal setting of 33%, the meter didn't register any flow. I don't know what the spec'd flow rate is for this combination of blower + filter, but Jordan says similar units in Downs register ~1500 cfm at the "High" setting. The two protable (similarly sized) HEPA units in the 40m, one at ITMY and one at ETMY, register ~900 cfm and ~1100 cfm respectively, when set to high. So we may want to revisit what the "nominal" HEPA setting should be, in case the filters have become clogged over time. 

Some photos of the HEPA blowers with the pre-filters off and the capacitors switched out may be found here.

it would be a good idea for us to have an auto-reminder to have us check the flow of all the HEPAs in the lab and elog it once a year so that we can replace filters and pre-filters appropriately.

The optomechanics stock in the lab was in a sad state. We have obtained the following from Thorlabs in the last two months:

1. 6 pcs each of DT12 and DT12B compact translation stages (for lens mode matching).
2. 3pcs each KM100PM + PM3 cube beamsplitter mounts (for polarization control).
3. 1 Post / spacer kit for height adjustment.
4. 3 pcs ea of K6XS + AD11F + F220APC for fiber applications.

I have used some of these for the ari BHD setup. The unused items are stored in the shelves that house the optomechanics ~halfway down the east arm. I'm wondering what's a good setup to document the stock of this stuff so we can always have a healthy stock of optomechanics (at least the non speciality ones like posts, spacers etc). It sucks to realize at 0000hrs that you're missing a 3mm shim or 250mm converging lens or something.

2x500 ml bottles of spectroscopic grade isopropanol were delivered. I marked them with today's date and placed them in the solvent cabinet. In the process, my shoulder bumped the laser interlock switch by the door to the VEA in the drill press area, which turned the PSL NPRO off. I turned it back on just now. The other NPROs are not connected to the interlock and so were unaffected.

Tue evening from 4pm~6pm, Koji made a social distant tour for Anchal. We were present around the PSL/AS/ETMX tables.

• UPS batteries
• 2x HEPA filters
• VWR chemicals (methanol)

These boxes were moved from the 40m hallway to the inside of the VEA so that we have some space to walk around. You can find some pictures here.

[JV, GV]

We cleared up some space in the 1Y1 electronics rack to install the 3 new FE machines. I removed the current driver and laser from 1Y1, they are now stored in the E10 cabinet. I will upload some photos to gPhotos soon.

1. I think it's good to have all these FEs in one rack (at least the new ones) - we should then hook it up to an ethernet power source, so that we can remotely power cycle them. I think we have long enough cables to interface to expansion chasses / dolphin switches, but if not, I think it's still a good idea to have these machines in 1Y1 as it is the least sensitive area in terms of immunity to bumping some cable during setup work and disturbing the rest of the IFO.
2. We found that the rails that the Supermicros shipped with the servers seem to be just a little too narrow - we mounted these in the rack, but had considerable difficulty sliding the server units in. Once they are in, they don't slide smoothly. Is there some special trick to installing these? 
3. I spent a few minutes trying to get Debian 8 installed on these machines, so that the rest of the setup work could be done remotely - however, there appear to be some firmware issues and so I'm not gonna dive into this.
   • I couldn't find a disk image for Debian 8.5 which is what the KT wikl recommends, so the OS I tried to install was Debian 8.11.
   • The error that comes up is related to a "stalled CPU" - apparently this is related to some graphics driver issue (there's another forum page that suggests upgrading the BIOS, but I don't think that's the problem here).
   • Anyways, this part of the process is only to install some drivers and do the initial setup - these machines will eventually run a diskless boot from the image on FB, so who knows if there will be some other driver issues/hardware-software incompatibilities there 😱 .
   • We should also make an effort to set these machines up with IPMI, but I think we first need to install an OS and a CLI to setup the IPMI. My cursory browsing of the manual suggests that the initial setup maybe can be done without installing an OS, and then subsequent work, including OS install, can be done remotely. If someone reads more in detail and can provide me a step-by-step, I can follow those instructions (if they aren't available to come into the lab). See here for some brief documentation of how to access the IPMI.

Per Koji's suggestion, I turned the PSL HEPA Variac to 50% just now, so that the power load through the burnt electrical cable is reduced by 75%.

I entered 40m today at around 1:20 pm and left by 1:45 pm. I entered 104 through the machine shop entry. I did the following:

• I took photos and videos of the PSL table with lights on.
• I uncovered the AP table, took photos and video, and covered it back.
• I went to the X End table and took a video without opening the enclosure.
• Apart from flipping light switches, nothing else should have changed.