I'll keep developing the camera server on a parallel track using the "new_..." directory naming convention. One thing I forgot to note is that the new pylon/pypylon packages require Python 3, so will not work with any of the 2.7 scripts. All of the environment I need to set up is exclusively Python 3. I won't change anything in the Python 2.7 environment in current use.

Also, I found the source of the bit resolution issue: Joe B's code loads a set of initialization parameters from a config file. One of them is "Frame Type = Mono8" which sets the dynamic range of the stream. I'll look into how this should be changed. 

I've started setting up the last new rackmount SuperMicro as a dedicated server for the GigE cameras. The new machine is currently sitting on the end of the electronics test bench. It is assigned the hostname c1cam at IP 192.168.113.116 on the martian network. I've installed Debian 10, which will be officially supported until July 2024.

I've added the /cvs/cds NFS mount and plan to house all the client/server code on this network disk. Any dependencies that must be built from source will be put on the network disk as well. Any dependencies that can be gotten through the package manager, however, will be installed locally but in an automated way using a reqs file.

We should ask Chub to reorder several more SuperMicro rackmount machines, SSD drives, and DRAM cards. Gautam has the list of parts from Johannes' last order.

I borrowed an old-looking Variac variable transformer from the power supplies cabinet along the y-arm. It is currently in the TCS lab.

I added the list of new c1iscaux channels to /opt/rtcds/caltech/c1/chans/daq/C0EDCU.ini and restarted the framebuilder. Koji had thought some of these channels might have previously existed under slightly different names. However, after looking through C0EDCU.ini and the other _SLOW.ini files, I did not find any candidates for removal. As far as I can tell, all of these channels are being recorded for the first time.

Following the death of rossa, which was hosting the only working environment for the GigE camera software, I've set up a new dedicated rackmount camera server: c1cam (details here). The Python server script is now configured as a persistent systemd service, which automatically starts on boot and respawns after a crash. The server depends on a set of EPICS channels being available to control the camera settings, so c1cam is also running a softIOC service hosting these channels. At the moment only the ETMX camera is set up, but we can now easily add more cameras.

Usage

Instructions for connecting to a live video feed are posted here. Any machine on the martian network can stream the feed(s). The only requirement is that the client machine have GStreamer 0.10 installed (all the control room workstations satisfy this).

Code Locations

As much as possible, the code and dependencies are hosted on the /cvs/cds network drive instead of installed locally. The client/server code and the Pylon5, PyPylon, and PyEpics dependencies are all installed at /cvs/cds/rtcds/caltech/c1/scripts/GigE. The configuration files for the soft IOC are located at /cvs/cds/caltech/target/c1cam.

Upgrade Goals

The 40m GigE camera code is a slightly-updated version of the 10+ year-old camera code in use at the sites. Consequently every one of its dependencies is now deprecated. Ultimately, we'd like to upgrade to the following:

• Python 2.7 --> 3.7
• Basler Pylon 5.0.12 --> 5.2.0
• PyPylon 1.1.1 --> 1.4.0
• GStreamer 0.10 --> 1.2

This is a long-term project, however, as many of these APIs are very different between Python 2 and 3.

[Jon, Yehonathan]

We carried out a set of cavity ringdown measurements of the PMC. The 1/e decay time scale is found to be 35.2 +/- 2.4 (systematic) μs. The statistical error is negligible compared to the systematic error, which is taken as the maximum absolute deviation of any measurement from the average value.

To make the measurement, we injected the first order deflection beam of an 80 MHz AOM, then extinguished it quickly by cutting the voltage offset to the AOM driver provided by an RF function generator. A 100 MHz oscilloscope configured to trigger on the falling voltage offset was used to sample the cavity in transmission as sensed by a PDA55. We found the detector noise of the DC-coupled output of the 35.5 MHz REFL PD to be too high for a reflection-side measurement.

Further loss analysis is forthcoming.

[Jon, Yehonathan]

We've begun assembling the new c1psl Acromag chassis based on Yehonathan's final pin assignments. So far, parts have been gathered and the chassis itself has been assembled.

Yehonathan is currently wiring up the chassis power and Ethernet feedthroughs, following my wiring diagram from previous assemblies. Once the Acromag units are powered, I will help configure them, assign IPs, etc. We will then turn the wiring over to Chub to complete the Acromag to breakout board wiring.

I began setting up the host server, but immediately hit a problem: We seem to have no more memory cards or solid-state drives, despite having two more SuperMicro servers. I ordered enough RAM cards and drives to finish both machines. They will hopefully arrive tomorrow.

I've assembled a new SuperMicro rackmount machine to replace c1psl. It is currently set up on the electronics bench.

• OS: Debian 10.2
• Hostname: c1psl1 (will become c1psl after installation)
• IP: 192.168.113.54 (registered in the martian DNS)
• Network drive mount point set up (/cvs/cds), which provides all the EPICS executables.

Summary

I have completed the new EPICS channel database for the c1psl and c1ioo channels (now combined into the new c1psl Acromag machine). I've tested a small subset of channels on the electronics bench to confirm that the addressing and analog channel calibrations are correct in a general sense. At this point, we are handing the chassis off to Chub to complete the wiring of the Acromag terminals to Dsub feedthroughs. At the 40m meeting today, we identified Feb. 17-22 as a potential window for installation in the interferometer (Gautam is out of town then). Below are some implementaton details for future reference.

Analog channel calibration for Acromag

For analog input (ai) channels, the Acromag outputs raw values ranging from +/-30,000 counts, but the EPICS IOC interprets the data type as ranging from +/-2^15 = 32,768. Similarly, for analog output (ao) channels, the Acromag expects a drive signal in the range +/-30,000 counts. To achieve proper scaling, Johannes had previously changed the EGUF and EGUL fields from +/-10 V to +/-10.923 V. However, changing the engineering fields makes it much harder for a human to read off the real physical I/O range of the channel.

A better way to achieve the correct scaling is to simply set the field  ASLO=1.09225 (65,536 / 60,001) in addition to the normal EGUF and EGUL field values (+/-10 V). Setting this field forces a rescaling of the number of raw counts that works as so (assuming a 16-bit bipolar ADC or DAC, as are the Acromags):

In the above mapping, OVAL is the value of the channel in engineering units (e.g., V) and RVAL is its raw value in counts. It is not the case that either the ASLO/AOFF or EGUF/EGUL fields are used, but not both. The ASLO/AOFF parameters are always applied (but their default values are ASLO=1 and AOFF=0, so have no effect unless changed). The EGUF and EGUL parameters are then additionally applied if the field LINR="LINEAR" is set.

This conversion allows the engineering fields to remain unchanged from the real physical range. The ASLO value is the same for both analog input and output channels. I have implemented this on all the new c1psl and c1ioo channels and confirmed it to work using a calibrated input voltage source.

Gautam and I debugged a communications problem with TP3 that was causing its python service to fail. We traced the problem back to the querying of the pump controller for its operational parameters (speed, voltage, temp). Some small percentage of the time (~5%, indeterministically), the pump controller is returning an invalid response which causes the service to shut itself down and signal a NO COMM error.

As a temporary fix, I wrapped the failing query in an exception handler to continue past this particular error. However, we suspect the microprocessor in the TP3 controller may be beginning to fail. There is a spare controller sitting right next to it in the vacuum rack. We will ask Chub to install the replacement in the near future.

gautam: this pump is responsible for pumping the annular volume under normal operations. while this problem is being resolved, the annular volume is valved off (as it has been since July 2019 anyway which is when this problem first manifested).

To facilitate wiring the c1psl chassis and scripting loopback tests, I've compiled a distilled spreadsheet with the Acromag-to-breakout board wiring, broken down by connector. This information is extractable from the master spreadsheet, but not easily. There were also a few apparent typos which are fixed here.

The wiring assignments at the time of writing are attached below. Here is the link to the latest spreadsheet.

I've started a spreadsheet for the BHD optics specifications and populated it with my best initial guesses. There are a few open questions we still need to resolve, mostly related to mode-matching:

• PR2 replacement: What transmission do we need for a ~100 mW pickoff? Also, do we want to keep the current curvature of -700 m?
• LO mode-matching telescope: What are the curvatures of the two mirrors?
• Lenses: We have six of them in the current layout. What FLs do we need?

The spreadsheet is editable by anyone. If you can contribute any information, please do!

After some discussion with Gautam, I decided to build more spare channels into the new c1psl machine. This is anticipation of adding new laser and ISS channels in the near future, to avoid having to disconnect the installed chassis and pull it out of the rack. The spare channels will be wired to DB37M feedthroughs on the front side of the chassis, with enough wire length to be able to pull the breakout boards out of the front to reconfigure their wiring as needed (e.g., split off channels onto a separate connector).

To have enough overhead, this will require installing 1 additional ADC unit (XT1221) and 1 additional DAC (XT1541). We have enough spare BIO channels among the existing units (both sinking and sourcing). This will give us:

• 13 spare ADC channels
• 14 spare DAC channels
• 16 spare sinking BIO channels
• 12 spare sourcing BIO channels

The updated c1psl chassis wiring assignments are attached. It adds 4 new DB37M connectors for the spare channels (highlighted in yellow) and fixes one typo Jordan found while wiring today. The most current spreadsheet is available here.

Last night Yehonathan and I located the two steel PMCs in the QIL, with help from Anchal. They are currently sitting on my desk in Bridge, inside a box that also contains optics and other OMC parts. I will bring them over to the 40m the next time I come.

A script I was testing errantly set C1:PSL-FSS_INOFFSET => 10 V at about 5:30 pm. I manually reverted the channel value to 0, but I don't know what the value was initially. Someone please check this value if there are problems locking the FSS.

I tested the c1psl AO channels on the electronics bench on Friday. While I found all the wiring to be correct, some of the channels exhibited excess noise with all appearances of a grounding problem.

Today Jordan, Gautam, and I investigated this further. It is indeed a grounding problem, but actually with the Acromag ADCs. The Acromag DAC outputs are single-ended (return is grounded), so (for the purpose of a loopback test) I would expect to leave the ADC inputs ungrounded. This is the configuration I tested Friday. Today we also tested driving the ADC with a floating source. The ADC noise behavior is exactly the same, whether the source end is grounded or not.

However, grounding the minus pin of the ADC channel eliminates the noise. We don't understand why this seems to be required irrespective of the driving source, so there something we're missing about the ADC design. As it turns out, this same fix was made to the AI channels of the previously-upgraded Acromag machines. I know Chub and I had to do this for the AI channels of c1vac, but at the time we thought the source grounding was causing the issue. However, today Jordan and I looked inside c1iscaux, which Chub wired, and confirmed that its AI channels are wired in the same way.

So in any case, Jordan is grounding the c1psl AI channels in the same way as c1iscaux. Once this is done, we'll continue with the bench testing tomorrow.

gautam: here are my notes about this issue when i was doing the c1iscaux testing. As I note there, "previously-upgraded Acromag machines" in the plural may be a bit of a stretch - I have no idea what the grounding situation is in c1susaux / c1auxex for example.

Today I engineered the last piece of the new c1psl system: the multi-bit binary output (mbbo) channels that control the MC servo board gains. These 6-bit channels have to be split across two 4-bit Acromag registers. To enforce synchronous switching, I adapted the latch.py script developed by Gautam to address this problem in c1iscaux. Analogously to the c1iscaux implementation, I scripted the code to automatically run as a systemd service which is launched by the main modbusIOC service. I tested this all using the DB37 LED test board and confirmed it to work.

This now completes the electronics bench testing.

There are still several DB37 connectors to be wired, which carry only spare channels for future use and are not interfaced with the EPICS IOC. Jordan and I discussed this today and he or Chub will complete it shortly. To allow time for the spare channel wiring to be completed (as well as for more locking progress before interruption), Gautam and I think Monday/Tuesday next week would be the earliest possible window to install the new system.

To supplement the material presented during the BHD review, I've put together a projected noise budget for the 40m. Note these are the expected interferometer noises (originating in the IFO itself), not BHD readout noises. The key parameters for each case are listed in the figure title. Also attached is a tarball (attachment 4) containing the ipython notebook, data files, and rolled-back version of pygwinc which were used to generate these figures.

Attachment 1: Phase quadrature readout.

Attachment 2: Comparison to aLIGO design sensitivity (phase quadrature).

Attachment 3: Amplitude quadrature readout.

Updated noise budget curves, now computed using the latest version of pygwinc. This resolves the inconsistency between the gwinc quantum noise curves and Gautam's analytic calculations. As before, the key configuration parameters are listed in the figure titles.

Attachment 1: Phase quadrature

Attachment 2: Amplitude quadrature

Attachment 3: Comparison to aLIGO design (phase quadrature)

Revised noise estimates, correcting a couple of factor of 2 and factor of pi errors found in the coil driver noise calculation. Also resolves a strain vs. displacement units confusion using the new pygwinc. Gautam and I have checked these noises against the analytical predictions and believe they are now accurate. Attachments are again:

Attachment 1: Phase quadrature

Attachment 2: Amplitude quadrature

Attachment 3: Comparison to aLIGO design (phase quadrature)

I investigated the problem reported earlier today with the BIO1 channels. By logging the systemd messages generated when the IOC starts, I was immediately able to determine that the problem was not limited to BIO1. The modbus communications were failing for several other units as well.

Because some in-situ rewiring of a handful of channels had recently been done (more on this soon), I initially suspected that one of the Acromags had been damaged in the process. However, removing BIO1 (or other non-communicating modules) did not restore communications with the rest of the modules. To test whether the chassis was the source of the problem at all, we set up a fresh ADC (new out of the package) and directly connected it to the secondary Ethernet interface of c1psl. With only the one new ADC connected, the modbus IOC failed in exactly the same way.

To confirm that the new ADC did in fact work, we connected it to c1auxex in the same configuration. The unit worked fine connected to c1auxex. This established that the source of the problem was the c1psl host. After some extensive debugging, I traced the problem to a pre-execution script (part of the modbus IOC systemd service) which resets the secondary network interface (the one connected to the Acromag chassis) prior to launching the IOC. This was to ensure the secondary interface always had the correct IP address. It appears this reset was somehow creating a race condition that allowed the modbus initializations (first communications with the Acromags) to sometimes start before the network interface had actually come back up.

I still don't understand how this was happening, or why the pre script worked just fine up until yesterday, but eliminating the network interface reset fixes the problem in 100% of the trials we ran. Unfortunately we lost the entire day to debugging this problem, so the final round of testing is still to be completed. We plan to pick it back up tomorrow afternoon.

We've completed almost all of the in-situ testing of the c1psl channels. During this process, we identified several channels which needed to be rewired to different Acromags (BIO sinking v. sourcing). We also elected to change the connector type of a few channels for practical advantages. Those modifications and other issues found during testing are detailed below. Also attached are the updated channel assignments, with a column indicating the in-situ testing status of each channel.

Post-installation modifications

• All four channels connected to the sourcing BIO module were found to in fact require sinking I/O. They were reassigned to sinking BIO modules. Affected channels:
  • C1:PSL-FSS_FASTSWEEP
  • C1:PSL-FSS_SW1
  • C1:PSL-FSS_SW2
  • C1:PSL-PSL_Shutter
• Added a new AI channel:
  • C1:PSL-FSS_MIXERM
• Removed an unneccessary AI channel:
  • C1:PSL-FSS_LODET
• Moved two AI channels from BNC connectors to a new Dsub connector (labelled DB25M-2 in the spreadsheet).
  • C1:PSL-FSS_RCTEMP
  • C1:PSL-FSS_RMTEMP_VOLTS

Issues identified during testing

• Digital calibration. The following channels work, but we need to verify their EPICS calibration parameters (EGUF/EGUL):
  • C1:PSL-FSS_FASTGAIN
  • C1:PSL-FSS_FAST
  • C1:PSL-PMC_RFADJ
  • C1:PSL-PMC_MODET
• IMC servo board. The Acromag channels themselves were found to work, but the linearity of the mbbo gain stages are in question (i.e., a potential problem with the board). GV is currently testing the servo board.
• PSL QPD board apears to be dead. We connected a scope directly to the test points on the board and measured a high level of noise and no signal (for all four of the QPD channels). I understand this QPD has not been used in some time, so it may not have been noticed before.
• WFS DC channels are saturating when the IMC is unlocked. The acceptance range of the Acromag ADC is only +/-10 V, but we measured sensor voltages as high as ~14 V. It appears that the old ADCs were somehow accepting a range of 0 to +20 V instead of -10 to +10 V. However, the Acromags do not support the input range 0-20 V. Since SNR is not critical for these channels (they're used only for initial alignment), I propose we simply install a voltage divider inside the chassis, just before the Acromag, for each of these signals.

Summary

Today I finished implementing loopback monitors of the up/down state of the slow controls machines. They are visible on a new MEDM screen accessible from Sitemap > CDS > Slow Machine Status (pictured in attachment 1). Each monitor is a single EPICS binary channel hosted by the slow machine, which toggles its state at 1 Hz (an alive "blinker"). For each machine, the monitor is defined by a separate database file named c1[machine]_state.db located in the target directory.

This is implemented for all upgraded machines, which includes every slow machine except for c1auxey. This is the next and final one slated for replacement.

Implementation

The blinkers are currently implemented as soft channels, but I'd like to ultimately convert them to hard channels using two sinking/sourcing BIO units. This will require new wiring inside each Acromag chassis, however. For now, as soft channels, the monitors are sensitive to a failure of the host machine or a failure of the EPICS IOC. As hard channels, they will additionally be sensitive to a failure of the secondary network interface, as has been known to happen.

Each slow machine's IOC had to be restarted this afternoon to pick up the new channels. The IOCs were restarted according to the following procedure.

c1auxex

• Disabled OPLEV servos on ETMX
• Zeroed slow biases
• Disabled watchdog
• Restarted IOC
• Reverted 1-3

​c1vac

• Closed V1, VM1
• Restarted IOC
• Returned valves to original state

c1psl

• Disabled IMC autolocker
• Closed PSL shutter
• Restarted IOC
• Reverted 1-2

c1iscaux

• ​Restarted IOC

c1susaux

• Disabled IMC autolocker
• Closed shutter
• Disabled OPLEV servos on: MC1, MC2, MC3, BS, ITMX, ITMX, PRM, SRM
• Zeroed slow biases
• Disabled watchdogs
• Restarted IOC
• Reverted 1-5

The intial recovery of c1susaux did not succeed. Most visibly, the alignment state of the IFO was not restored. After some debugging, we found that the restart of the modbus service was partially failing at the final burt-restore stage. The latest snapshot file /opt/rtcds/caltech/c1/burt/autoburt/latest/c1susaux.snap was not found. I manually restored a known good snapshot from earlier in the day (15:19) and we were able to relock the IMC and XARM. GV and I were just talking earlier today about eliminating these burt-restores from the systemd files. I think we should.

Since there has been a proliferation of BHD Google docs recently, I've linked them all from the BHD wiki page. Let's continue adding any new docs to this central list.

I used Yehonathan's wiring assignments to lay the rest of groundwork for the final slow controls machine upgrade, c1auxey. Actions completed:

• Created an internal wiring diagram for assembling the Acromag chassis (log in with LIGO.ORG credentials to view/edit)
• Created a new target directory on the network drive:

The "1" will be dropped after the new system is permanently installed.

• Populated the target directory with files:
  • modbusIOC.service - wraps the EPICS IOC as a systemd service
  • ETMYaux.env - defines the EPICS environment variables
  • ETMYaux.cmd - command file to set up the EPICS IOC
  • ETMYaux.sh - enables DAC outputs to the suspension (executed lastly)
• Created the EPICS channel databases:
  • ETMYaux.db - migration of the existing database
  • c1auxey_state.db - contains logic for loopback monitoring of the IOC "alive" state (visible from Sitemap > CDS > Slow Controls Status)

Hardware-wise, this system will require:

• 2 Acromag XT-1221 units (ADC)
• 1 Acromag XT-1541 unit (DAC)
• 1 Acromag XT-1111 unit (sinking BIO)

I know that we do have these quantities left on hand. The next steps are to set up the Supermicro host and begin assembling the Acromag chassis. Both of these activities require an in-person presence, so I think this is as far as we can advance this project for now.

Yehonathan's list does include C1:AUX-GREEN_Y_Shutter and I copied its definition from /cvs/cds/caltech/target/c1aux/ShutterInterlock.db into the new ETMYaux.db file.

I noticed ShutterInterlock.db still contains about a dozen channels. Some of them appear to be ghosts (like the C1:AUX-PSL_Shutter[...] set, which has since become C1:PSL-PSL_Shutter[...] hosted on c1psl) but others like C1:AUX-GREEN_X_Shutter appear to still be in active use.

I wanted to pass along a complication pointed out by K. Thorne re: our plan to use Gen1 (old) Dolphin IPC cards in the new real-time machines: c1bhd, c1sus2. The implication is that we may be forced to install a very old OS (e.g., Debian 8) for compatibility with the IPC card driver, which could lead to other complications like an incompatibility with the modern network interface.

I have a query out to Dolphin asking:

1. Have they done any testing of these old drivers on Linux kernel 4.x (e.g., Debian 9/10)?
2. Is there any way to buy modern IPC cards for the two new machines and interface them with our existing Gen1 network?

I'll add more info if I hear back from them.

Answers from Dolphin:

1. No, and kernel 4.x (modern Linux) definitely will not work with the Gen1 cards.
2. No, cards using different PCIe chipsets cannot be mixed.

Since upgrading every front end is out of the question, our only option is to install an old OS (Linux kernel < 3.x) on the two new machines. Based on Keith's advice, I think we should go with Debian 8. (Link to Keith's Debian 8 instructions.)

In the past year, pygwinc has expanded to support not just fundamental noise calculations (e.g., quantum, thermal) but also any number of user-defined noises. These custom noise definitions can do anything, from evaluating an empirical model (e.g., electronics, suspension) to loading real noise measurements (e.g., laser AM/PM noise). Here is an example of the framework applied to H1.

Starting with the BHD review-era noises, I have set up the 40m pygwinc fork with a working noise budget which we can easily expand. Specific actions:

• Updated the 40m fork to the latest pygwinc version (while preserving the commit history).
• Added a directory ./CIT40m containing the 40m-specific noise budget files (created by GV).
• Added an ipython notebook CIT40m.ipynb at the root level showing how to generate a noise budget.
• Integrated our DAC and seismic noise estimators into pygwinc.
• Marked the old 40m NB repo as obsolete (last commit > 2 yrs ago). Many of these noise estimates are probably stale, but I will work with GV to identify which ones can be migrated.

I set up our fork in this way to keep the 40m separate from the main pygwinc code (i.e., not added to as a built-in IFO type). With the 40m code all contained within one root-level directory (with a 40m-specific name), we should now always be able to upgrade to the latest pygwinc without creating intractable merge conflicts.

Summary

I've generated specifications for the new BHD optics. This includes the suspended relay mirrors as well as the breadboard optics (but not the OMCs).

To design the mode-matching telescopes, I updated the BHD mode-matching scripts to reflect Koji's draft layout (Dec. 2019) and used A La Mode to optimize ROCs and positions. Of the relay optics, only a few have an AOI small enough for curvature (astigmatism) and most of those do not have much room to move. This reduced the optimization considerably.

These ROCs should be viewed as a first approximation. Many of the distances I had to eyeball from Koji's drawings. I also used the Gaussian PRC/SRC modes from the current IFO, even though the recycling cavities will both slightly change. I set up a running list of items like these that we still need to resolve in the BHD README.

Optics Specifications

• 2" mirrors (E2000226-v1)
• 2" 50:50 beamsplitters (E2000227-v1)
• 2" lenses (E2000228-v1)
• PR2 replacement (E2000229-v1)

At a glance, all the specifications can be seen in the optics summary spreadsheet.

LO Telescope Design

The LO beam originates from the PR2 transmission (POP), near ITMX. It is relayed to the BHD beamsplitter (and mode-matched to the OMCs) via the following optical sequence:

• LM1 (ROC = +10 m, AOI ≈ 3°)
• LM2 (Flat, AOI ≈ 45°)
• MMT1 (Flat, AOI ≈ 5°)
• MMT2 (ROC = +3.5 m, AOI ≈ 5°)

The resulting beam profile is shown in Attachment 1.

AS Telescope Design

The AS beam is relayed from the SRM to the BHD beamsplitter (and mode-matched to the OMCs) via the following sequence:

• AS1 (ROC = +1.5 m, AOI ≈ 3°)
• AS2 (Flat, AOI ≈ 45°)
• Lens (FL = -125 mm)

A lens is used because there is not enough room on the BHD breadboard for a pair of (low-AOI) telescope mirrors, like there is in the LO path. The resulting beam profile is shown in Attachment 2.

Nice, and we should also permanently install the camera server (c1cam) which is still sitting on the electronics bench. It is running an adapted version of the Python 2/Debian 8 site code. Maybe if COVID continues long enough I'll get around to making the Python 3 version we've long discussed.

Hang and I have reanalyzed the BHD telescope designs, with the goal of identifying sufficiently non-degenerate locations for ASC actuation. Given the limited room to reposition optics and the requirement to remain insensitive to small positioning errors, we conclude it is not possible put sufficient Gouy phase separation between the AS1/AS2 and LO1/LO2 locations. However, we can make the current layout work if we instead actuate AS1/AS4 and LO1/LO4. This would require actuating one optic on the breadboard for each relay path. If possible, we believe this offers the simplest solution (i.e., least modification to the current layout).

LO Telescope Design (Attachment 1)

Radius of curvatures:

• LO1: +10 m
• LO2: flat
• LO3: +15 m
• LO4: flat

AS Telescope Design (Attachment 2)

Radius of curvatures:

• AS1: +3 m
• AS2: flat
• AS3: -1 m
• AS4: flat

Summary

For the initial phase of BHD testing, we recently discussed whether the mode-matching telescopes could be built with 100% stock optics. This would allow the optical system to be assembled more quickly and cheaply at a stage when having ultra-low loss and scattering is less important. I've looked into this possibility and conclude that, yes, we do have a good stock optics option. It in fact achieves comprable performance to our optimized custom-curvature design [ELOG 15357]. I think it is certainly sufficient for the initial phase of BHD testing.

Vendor

It turns out our usual suppliers (e.g., CVI, Edmunds) do not have enough stock options to meet our requirements. This is for two reasons:

• For sufficient LO1-LO2 (AS1-AS4) Gouy phase separation, we require a very particular ROC range for LO1 (AS1) of 5-6 m (2-3 m).
• We also require a 2" diameter for the suspended optics, which is a larger size than most vendors stock for curved reflectors (for example, CVI has no stock 2" options).

However I found that Lambda Research Optics carries 1" and 2" super-polished mirror blanks in an impressive variety of stock curvatures. Even more, they're polished to comprable tolerances as I had specificied for the custom low-scatter optics [DCC E2000296]: irregularity < λ/10 PV, 10-5 scratch-dig, ROC tolerance ±0.5%. They can be coated in-house for 1064 nm to our specifications.

From modeling Lambda's stock curvature options, I find it still possible to achieve mode-matching of 99.9% for the AS beam and 98.6% for the LO beam, if the optics are allowed to move ±1" from their current positions. The sensitivity to the optic positions is slightly increased compared to the custom-curvature design (but by < 1.5x). I have not run the stock designs through Hang's full MC corner-plot analysis which also perturbs the ROCs [ELOG 15339]. However for the early BHD testing, the sensitivity is secondary to the goal of having a quick, cheap implementation.

Stock-Part Telescope Designs

The following tables show the best telescope designs using stock curvature options. It assumes the optics are free to move ±1" from their current positions. For comparison, the values from the custom-curvature design are also given in parentheses.

AS Path

The AS relay path is shown in Attachment 1:

• AS1-AS4 Gouy phase separation: 71°
• Mode-matching to OMC: 99.9%

LO Path

The LO relay path is shown in Attachment 2:

• LO1-LO2 Gouy phase separation: 67°
• Mode-matching to OMC: 98.6%

Ordering Information

I've created a new tab in the BHD procurement spreadsheet ("Stock MM Optics Option") listing the part numbers for the above telescope designs, as well as their fabrication tolerances. The total cost is $2.8k + the cost of the coatings (I'm awaiting a quote from Lambda for the coatings). The good news is that all the curved substrates will receive the same HR/AR coatings, so I believe they can all be done in a single coating run.

MM_total = (MM_vert + MM_horiz) / 2. 

The large astigmatic MM loss in the LO case is mainly due to the strong LO4 curvature (R=0.15m) with a 10 deg AOI. I looked again at whether LO1 could be increased from R=5m to the next higher stock value of 7.5m, as this would allow weaker curvatures on LO3 and LO4. However, no, that is not possible---it reduces the LO1-LO2 Gouy phase separation to only 18 deg.

There is, however, a good stock-curvature option if we want to reconsider actuating LO4 instead of LO2 (attachment 1). It achieves 99.2% MM with the OMCs, allowing positions to vary +/-1" from the current design. The LO1-LO4 Gouy phase separation is 72 deg.

Alternatively, we could look at reducing the AOI on LO3 and LO4 (keeping LO1-LO2 actuation).

Hmm? T1300364 suggests MM_total = Sqrt(MM_Vert * MM_Horiz)

I don't think we ever discussed why the angular RMS of the ETMs is so much higher than the ITMs. Maybe that's a separate matter because, even assuming the worst case, the actuation range requirement is

(0.82 μrad RMS) x (15 μrad/μrad) x (10 safety factor) = 0.12 mrad

which is still only order 1% of the pitch/yaw pointing range of the Small Optic Suspensions, according to P1600178 (sec. IV. A). Can we check this requirement off the list?

After further astigmatism/tolerance analysis [ELOG 15380, 15387] our conclusion is that the stock-optic telescope designs [ELOG 15379] are sufficient for the first round of BHD testing. However, for the final BHD hardware we should still plan to procure the custom-curvature optics [DCC E2000296]. The optimized custom-curvature designs are much more error-tolerant and have high probability of achieving < 2% mode-matching loss. The stock-curvature designs can only guarantee about 95% mode-matching.

Below are the final distances between optics in the relay paths. The base set of distances is taken from the 2020-05-21 layout. To minimize the changes required to the CAD model, I was able to achieve near-maximum mode-matching by moving only one optic in each relay path. In the AS path, AS3 moves inwards (towards the BHDBS) by 1.06 cm. In the LO path, LO4 moves backwards (away from the BHDBS) by 3.90 cm.

AS Path

LO Path

[Jon, Jordan, Koji]

Today Jordan reconfigured the vac system to allow pumping of the main volume resume, with Jon and Koji remotely advising. All clear to resume normal IFO activities. However, the vac system is operating in a temporary configuration that will have to be reverted as we locate replacement components. Details below.

Procedure

Since serial readback of the TP2 controller seems to be failing, we reconfigured the system with TP3 now backing for TP1. TP2 was valved off (at V4) and shut down until we can replace its controller.

TP3 has its own problems, however. It was valved off in January after its temperature readback began glitching and spuriously triggering the interlocks [ELOG 15140]. However the problem appears to be limited only one readback (rotation speed, current, voltage are fine) and there is enough redundancy in the pump-dependent interlock conditions to safely connect it to the main volume.

We also discovered that sometime since January, the TP3 dry pump has failed. The foreline pressure had risen to 165 torr. Since the TP2 and TP3 dry pumps are not interchangeable (Agilent vs. Varian), we instead valved in the auxiliary dry pump and disconnected the failed dry pump using a KF blank. This is a temporary arrangement until the permanent dry pump can be repaired. Jordan removed it to replace the tip seals and will test it in the bake lab before reinstalling.

With this configuration in place, we proceeded to pump down the main volume without issue (attachment 1). We monitored the pumpdown for about 45 min., until the pressure had reached ~1E-5 torr and TP3 had been transitioned to standby (low-speed) mode.

Summary of topology changes:

• TP2 valved off and shut down until controller can be replaced
• TP3 temporarily backing for TP1
• Auxiliary dry pump temporarily backing for TP3
• TP3 dry pump has been removed for repairs

Looking at images of the old vac screens, the TP2/3 rotation speed and status string were digitally monitored. However I don't know if there were software interlocks predicated on those.

The temperature and current interlocks are implemented precisely because the pumps can shut themselves off. The concern is not about damaging the pumps (their internal logic protects against that); it's that a pump could automatically shut down and back-vent the IFO to atmosphere. Another interlock (e.g., the pressure differentials) might catch it, but it would depend on the back-vent rate and the scenario has never been tested. The temperature and current interlocks are set to trip just before the pump reaches its internal shut-down threshold.

One way we might be able to reduce our reliance on the flaky serial readbacks is to implement rotation-speed hardware interlocks. The old vac documentation alludes to these, but as far as Chub and I could determine in 2018, they never actually existed. The older turbo controllers, at least, had an analog output proportional to speed which could be used to control a relay to interrupt the V4/5 control signals. I'll look into this for the new controllers. If it could be done, we could likely eliminate the layer of serial-readback interlocks altogether.

That would be awesome if you're willing to volunteer. I agree this would be great to have.

Right, I doubt they were ever recorded or used for interlocks. But the readbacks did work at one point in the past. There's a photo of the old vac monitor screen on p. 19 of E1500239 (last updated 2017) which shows the fields once alive.

I don't disagree that the pressure gauges would register the change. What I'm not sure about is whether the change would violate any of the existing interlock conditions, triggering a shutdown. Looking at what we have now, the only non-pump-related conditions I see that might catch it are the diffpres conditions:

• abs(P2 - PTP2) > 1 torr (for a TP2 failure)

• abs(P3 - PTP3) > 1 torr (for a TP3 failure)

• abs(P1a - P2) > 1 torr (for either a TP2 or TP3 failure)

For the P1a-P2 differential, the threshold of 1 torr is the smallest value that in practice still allows us to pump down the IFO without having to disable the interlocks (P1a-P2 is the TP1 intake/exhaust differential). The purpose of the P2-PTP2/P3-PTP3 differentials is to prevent V4/5 from opening and suddenly exposing the spinning turbo to high pressure. I'm not aware of a real damage threshold calculation that any one has done; I think < 1 torr is lore passed down by Steve.

If a turbo pump fails, the rate it would backstream is unknown (to me, at least) and likely depends on the failure mode. The scenario I'm concerned about is if the backstream rate is slower than the conduction time through the pumspool and into the main volume. In that case, the pressure gauges will rise more or less together all the way up to atmosphere, likely never crossing the 1 torr differential pressure thresholds.

There's already a channel C1:Vac-error_status, where if the value is anything other than an empty string, there is an interlock tripped. Does that work?

Replacement Hardware Purchase List

I've created a purchase list of hardware needed to restore the aging vacuum system. This wasn't planned as part of the BHD upgrade, but I've added it to the BHD procurement list since hardware replacements have become necessary.

The list proposes replacing the aging TP3 Varian turbo pump with the newer Agilent model which has already replaced TP2. It seems I was mistaken in believing we already had a second Agilent pump on hand. A thorough search of the lab has not turned it up, and Steve himself has told me he doesn't remember ordering a second one. Fortunately Steve did leave us a detailed Agilent parts list [ELOG 14322].

It also proposes replacing the glitching TP2 Agilent controller with a new one. The existing one can be sent back for repair and then retained as a spare. Considering that one of these controllers is already malfunctioning after < 2 years, I think it's a very good idea to have a spare on hand.

Known Hardware Issues

Below is our current list of vacuum hardware issues. Items that this purchase list will address (limited to only the most urgent) are highlighted in yellow.

• Replace the UPS
  • Need a 240V socket for TP1 (currently TP1 is not protected from power loss)
  • Need RS232/485 comms with the interlock server (current UPS: serial readbacks have failed, battery is failing)
• Remove/replace the failed pressure gauges (~5)
• Add more cold cathode sensors to the main volume for sensor redundancy (currently the main-volume interlocks rely on only 1 working sensor)
• Replace TP3 (controller is failing)
• Replace TP2 controller (serial interface has failed)
• Remove RP2
  • Dead and also not needed. We already have to throttle the pumpdown rate with only two roughing pumps
• Remove/refurbish the cryopump
  • Contamination risk to have it sitting connectable to the main volume

I think we should discuss interlock possibilities at a 40m meeting. I'm reluctant to make the system more complicated, but perhaps we can find ways to reduce the reliance on the turbo pump readbacks. I agree they've proven to be the least reliable.

While we may be able to improve the tolerance to certain kinds of hardware malfunctions (and if so, we should), I don't see interlocks triggering on abnormal behavior of critical equipment as the root problem. As I see it, our bigger problem is with all the malfunctioning, mostly end-of-lifetime pieces of vacuum equipment still in use. If we can address the hardware problems, as I'm trying to do with replacements [ELOG 15412], I think that in itself will make the interlocking much less of an issue.

Ok, this can be added pretty easily. Its value will just be toggled between 1 and 0 every time the interlock server raises/clears the existing string channel. Adding the channel will require restarting the whole vac IOC, so I'll do it at a time when Jordan is on hand in case something fails to come back up.

The vac system is going down at 11 am today for planned maintenance:

• Re-install the repaired TP2 and TP3 dry pumps [ELOG 15417]
• Incorporate an auto-mailer and flag channel into the controls code for signaling tripped interlocks [ELOG 15413]

We will advise when the work is completed.

This work is finally complete. The dry pump replacement was finished quickly but the controls updates required some substantial debugging.

For one, the mailer code I had been given to install would not run against Python 3.4 on c1vac, the version run by the vac controls since about a year ago. There were some missing dependencies that proved difficult to install (related to Debian Jessie becoming unsupported). I ultimately solved the problem by migrating the whole system to Python 3.5. Getting the Python keyring working within systemd (for email account authentication) also took some time.

Edit: The new interlock flag channel is named C1:Vac-interlock_flag.

Along the way, I discovered why the interlocks had been failing to auto-close the PSL shutter: The interlock was pointed to the channel C1:AUX-PSL_ShutterRqst. During the recent c1psl upgrade, we renamed this channel C1:PSL-PSL_ShutterRqst. This has been fixed.

The main volume is being pumped down, for now still in a TP3-backed configuration. As of 8:30 pm the pressure had fallen back to the upper 1E-6 range. The interlock protection is fully restored. Any time an interlock is triggered in the future, the system will send an immediate notification to 40m mailing list. 👍

​I looked into how the new UPS devices suggested by Chub would communicate with the vac interlocks. There are several possible ways, listed in order of preference:

• Python interlock service directly queries the UPS via a USB link using the (unofficial) tripplite package. Direct communication would be ideal because it avoids introducing a dependency on third-party software outside the monitoring/control capability of the interlock manager. However the documentation warns this package does not work for all models...
• Configure Tripp Lite's proprietary software (PowerAlert Local) to send SYSLOG event messages (UDP packets) to a socket monitored by the Python interlock manager.
• Configure the proprietary software to execute a custom script upon an event occurring. The script would, e.g., set an EPICS flag channel which the interlock manager is continually monitoring.

I recommend we proceed with ordering the Tripp Lite 36HW20 for TP1 and Tripp Lite 1AYA6 for TP2 and TP3 (and other 120V electronics). As far as I can tell, the only difference between the two 120V options is that the 6FXN4 model is TAA-compliant.

As suggested last week, Hang and I have reviewed the A+ BHD status (DRD, CDD, and reviewers' comments) and compiled a list of key unanswered questions which could be addressed through Finesse analysis.

In anticipation of others helping with this modeling effort, we've tried to break questions into self-contained projects and estimated their level of difficulty. As you'll see, they range from beginner to Finesse guru.

Here is the procedure for setting up the three new BHD front-ends (c1bhd, c1sus2, c1ioo - replacement). This plan is based on technical advice from Rolf Bork and Keith Thorne.

The overall topology for each machine is shown here. As all our existing front-ends use (obsolete) Dolphin PCIe Gen1 cards for IPC, we have elected to re-use Dolphin Gen1 cards removed from the sites. Different PCIe generations of Dolphin cards cannot be mixed, so the only alternative would be to upgrade every 40m machine. However the drivers for these Gen1 Dolphin cards were last updated in 2016. Consequently, they do not support the latest Linux kernel (4.x) which forces us to install a near-obsolete OS for compatibility (Debian 8).

Hardware

• Servers: SuperMicro 5018R-M [RECEIVED]
• IPC cards: Dolphin DXH510-A0 (PCIe x4 Gen1) [LLO will provide; I've asked Keith Thorne to ship them]
• I/O cards: One Stop Systems OSS-PCIe-HIB25-x4-H-H (PCIe x4 Gen2) [Rich Abbott will provide; part of the I/O chassis hardware we are receiving from aLIGO]
• I/O cables: One Stop Systems OSS-PCIe-CBL-x4-3M (PCIe x4 3m) [RECEIVED]
• IPC cables: Data Storage Cables C5252-40ME-O (fiber optic 40m) [RECEIVED]

Software

• OS: Debian 8.11 (Linux kernel 3.16)
• IPC card driver: Dolphin DX 4.4.5 [works only with Linux kernel 2.6 to 3.x]
• I/O card driver: None required, per the manual

Install Procedure

1. Follow Keith Thorne's procedure for setting up Debian 8 front-ends
2. Apply the real-time kernel patches developed for Debian 9, but modified for kernel 3.16 [these are UNTESTED against Debian 8; Keith thinks they may work, but they weren't discovered until after the Debian 9 upgrade]
3. Install the PCIe expansion cards and Dolphin DX driver (driver installation procedure)

Chub has placed the order for two new UPS units (115V for TP2/3 and a 220V version for TP1).

They will arrive within the next two weeks.

This year we've struggled with vacuum controls unreliability (e.g., spurious interlock triggers) caused by decaying hardware. Here are details of the vacuum refurbishment plan I described on the 40m call this week.

☑ Refurbish TP2 and TP3 dry pumps. Completed [ELOG 15417].

☑ Automated notifications of interlock-trigger events. Email to 40m list and a new interlock flag channel. Completed [ELOG 15424].

☐ Replace failing UPS.

• Two new Tripp Lite units on order, 110V and 230V [ELOG 15465].
• Jordan will install them in the vacuum rack once received.
• Once installed, Jon will come test the new units, set up communications, and integrate them into the interlock system following this plan [ELOG 15446].
• Jon will move the pumps and other equipment to the new UPS units only after completing the above step.

☐ Remove interlock dependencies on TP2/TP3 serial readbacks. Due to persistent glitching [ELOG 15140, ELOG 15392].

Unlike TP2 and TP3, the TP1 readbacks are real analog signals routed to Acromags. As these have caused us no issues at all, the plan is to eliminate dependence on the TP2/3 digital readbacks in favor of the analog controller outputs. All the digital readback channels will continue to exist, but the interlock system will no longer depend on them. This will require adding 2 new sinking BI channels each for TP2 and TP3 (for a total of 4 new channels). We have 8 open Acromag XT1111 channels in the c1vac system [ELOG 14493], so the new channels can be accommodated. The below table summarizes the proposed changes.

To carry out the next steps of the vac refurbishment plan [ELOG 15499], I've ordered parts necessary for interfacing the UPS units and the analog TP2/3 controller outputs with c1vac. The purchase list is appended to the main BHD list and is located here. Some parts we already had in the boxes of Acromag materials. Jordan is gathering what we do already have and staging it on the vacuum controls console table - please don't move them or put them away.