40m QIL Cryo_Lab CTN SUS_Lab TCS_Lab OMC_Lab CRIME_Lab FEA ENG_Labs OptContFac Mariner WBEEShop
  40m Log, Page 82 of 341  Not logged in ELOG logo
ID Date Authorup Type Category Subject
  14856   Fri Aug 23 19:10:02 2019 JonUpdateCamerasGigE camera server is online

Following the death of rossa, which was hosting the only working environment for the GigE camera software, I've set up a new dedicated rackmount camera server: c1cam (details here). The Python server script is now configured as a persistent systemd service, which automatically starts on boot and respawns after a crash. The server depends on a set of EPICS channels being available to control the camera settings, so c1cam is also running a softIOC service hosting these channels. At the moment only the ETMX camera is set up, but we can now easily add more cameras.

Usage

Instructions for connecting to a live video feed are posted here. Any machine on the martian network can stream the feed(s). The only requirement is that the client machine have GStreamer 0.10 installed (all the control room workstations satisfy this).

Code Locations

As much as possible, the code and dependencies are hosted on the /cvs/cds network drive instead of installed locally. The client/server code and the Pylon5, PyPylon, and PyEpics dependencies are all installed at /cvs/cds/rtcds/caltech/c1/scripts/GigE. The configuration files for the soft IOC are located at /cvs/cds/caltech/target/c1cam.

Upgrade Goals

The 40m GigE camera code is a slightly-updated version of the 10+ year-old camera code in use at the sites. Consequently every one of its dependencies is now deprecated. Ultimately, we'd like to upgrade to the following:

  • Python 2.7 --> 3.7
  • Basler Pylon 5.0.12 --> 5.2.0
  • PyPylon 1.1.1 --> 1.4.0
  • GStreamer 0.10 --> 1.2

This is a long-term project, however, as many of these APIs are very different between Python 2 and 3.

  15093   Wed Dec 11 15:01:57 2019 JonSummaryPSLPMC cavity ringdown measurement

[Jon, Yehonathan]

We carried out a set of cavity ringdown measurements of the PMC. The 1/e decay time scale is found to be 35.2 +/- 2.4 (systematic) μs. The statistical error is negligible compared to the systematic error, which is taken as the maximum absolute deviation of any measurement from the average value.

To make the measurement, we injected the first order deflection beam of an 80 MHz AOM, then extinguished it quickly by cutting the voltage offset to the AOM driver provided by an RF function generator. A 100 MHz oscilloscope configured to trigger on the falling voltage offset was used to sample the cavity in transmission as sensed by a PDA55. We found the detector noise of the DC-coupled output of the 35.5 MHz REFL PD to be too high for a reflection-side measurement.

Further loss analysis is forthcoming.

Attachment 1: IMG_0101.jpg
IMG_0101.jpg
  15111   Mon Jan 6 15:36:55 2020 JonUpdatePSLAssembly underway for c1psl upgrade

[Jon, Yehonathan]

We've begun assembling the new c1psl Acromag chassis based on Yehonathan's final pin assignments. So far, parts have been gathered and the chassis itself has been assembled.

Yehonathan is currently wiring up the chassis power and Ethernet feedthroughs, following my wiring diagram from previous assemblies. Once the Acromag units are powered, I will help configure them, assign IPs, etc. We will then turn the wiring over to Chub to complete the Acromag to breakout board wiring.

I began setting up the host server, but immediately hit a problem: We seem to have no more memory cards or solid-state drives, despite having two more SuperMicro servers. I ordered enough RAM cards and drives to finish both machines. They will hopefully arrive tomorrow.

  15114   Tue Jan 7 18:51:51 2020 JonUpdatePSLNew c1psl server assembled

I've assembled a new SuperMicro rackmount machine to replace c1psl. It is currently set up on the electronics bench.

  • OS: Debian 10.2
  • Hostname: c1psl1 (will become c1psl after installation)
  • IP: 192.168.113.54 (registered in the martian DNS)
  • Network drive mount point set up (/cvs/cds), which provides all the EPICS executables.
  15125   Wed Jan 15 14:10:28 2020 JonConfigurationPSLNew EPICS database for C1PSL + C1IOO

Summary

I have completed the new EPICS channel database for the c1psl and c1ioo channels (now combined into the new c1psl Acromag machine). I've tested a small subset of channels on the electronics bench to confirm that the addressing and analog channel calibrations are correct in a general sense. At this point, we are handing the chassis off to Chub to complete the wiring of the Acromag terminals to Dsub feedthroughs. At the 40m meeting today, we identified Feb. 17-22 as a potential window for installation in the interferometer (Gautam is out of town then). Below are some implementaton details for future reference.

Analog channel calibration for Acromag

For analog input (ai) channels, the Acromag outputs raw values ranging from +/-30,000 counts, but the EPICS IOC interprets the data type as ranging from +/-2^15 = 32,768. Similarly, for analog output (ao) channels, the Acromag expects a drive signal in the range +/-30,000 counts. To achieve proper scaling, Johannes had previously changed the EGUF and EGUL fields from +/-10 V to +/-10.923 V. However, changing the engineering fields makes it much harder for a human to read off the real physical I/O range of the channel.

A better way to achieve the correct scaling is to simply set the field  ASLO=1.09225 (65,536 / 60,001) in addition to the normal EGUF and EGUL field values (+/-10 V). Setting this field forces a rescaling of the number of raw counts that works as so (assuming a 16-bit bipolar ADC or DAC, as are the Acromags):

OVAL = (RVAL * ASLO + AOFF + 2^15) * (EGUF - EGUL) / 2^16 + EGUL

In the above mapping, OVAL is the value of the channel in engineering units (e.g., V) and RVAL is its raw value in counts. It is not the case that either the ASLO/AOFF or EGUF/EGUL fields are used, but not both. The ASLO/AOFF parameters are always applied (but their default values are ASLO=1 and AOFF=0, so have no effect unless changed). The EGUF and EGUL parameters are then additionally applied if the field LINR="LINEAR" is set.

This conversion allows the engineering fields to remain unchanged from the real physical range. The ASLO value is the same for both analog input and output channels. I have implemented this on all the new c1psl and c1ioo channels and confirmed it to work using a calibrated input voltage source.

  15140   Wed Jan 22 16:19:47 2020 JonUpdateVACTP3 controller errors

Gautam and I debugged a communications problem with TP3 that was causing its python service to fail. We traced the problem back to the querying of the pump controller for its operational parameters (speed, voltage, temp). Some small percentage of the time (~5%, indeterministically), the pump controller is returning an invalid response which causes the service to shut itself down and signal a NO COMM error.

As a temporary fix, I wrapped the failing query in an exception handler to continue past this particular error. However, we suspect the microprocessor in the TP3 controller may be beginning to fail. There is a spare controller sitting right next to it in the vacuum rack. We will ask Chub to install the replacement in the near future.

gautam: this pump is responsible for pumping the annular volume under normal operations. while this problem is being resolved, the annular volume is valved off (as it has been since July 2019 anyway which is when this problem first manifested).

  15150   Thu Jan 23 23:07:04 2020 JonConfigurationPSLc1psl breakout board wiring

To facilitate wiring the c1psl chassis and scripting loopback tests, I've compiled a distilled spreadsheet with the Acromag-to-breakout board wiring, broken down by connector. This information is extractable from the master spreadsheet, but not easily. There were also a few apparent typos which are fixed here.

The wiring assignments at the time of writing are attached below. Here is the link to the latest spreadsheet.

Attachment 1: c1psl_feedthrough_wiring.pdf
c1psl_feedthrough_wiring.pdf c1psl_feedthrough_wiring.pdf c1psl_feedthrough_wiring.pdf c1psl_feedthrough_wiring.pdf
  15151   Fri Jan 24 13:56:21 2020 JonUpdateBHDBHD optics specifications

I've started a spreadsheet for the BHD optics specifications and populated it with my best initial guesses. There are a few open questions we still need to resolve, mostly related to mode-matching:

  • PR2 replacement: What transmission do we need for a ~100 mW pickoff? Also, do we want to keep the current curvature of -700 m?
  • LO mode-matching telescope: What are the curvatures of the two mirrors?
  • Lenses: We have six of them in the current layout. What FLs do we need?

The spreadsheet is editable by anyone. If you can contribute any information, please do!

  15168   Tue Jan 28 19:12:30 2020 JonConfigurationPSLSpare channels added to c1psl chassis

After some discussion with Gautam, I decided to build more spare channels into the new c1psl machine. This is anticipation of adding new laser and ISS channels in the near future, to avoid having to disconnect the installed chassis and pull it out of the rack. The spare channels will be wired to DB37M feedthroughs on the front side of the chassis, with enough wire length to be able to pull the breakout boards out of the front to reconfigure their wiring as needed (e.g., split off channels onto a separate connector).

To have enough overhead, this will require installing 1 additional ADC unit (XT1221) and 1 additional DAC (XT1541). We have enough spare BIO channels among the existing units (both sinking and sourcing). This will give us:

  • 13 spare ADC channels
  • 14 spare DAC channels
  • 16 spare sinking BIO channels
  • 12 spare sourcing BIO channels

The updated c1psl chassis wiring assignments are attached. It adds 4 new DB37M connectors for the spare channels (highlighted in yellow) and fixes one typo Jordan found while wiring today. The most current spreadsheet is available here.

Attachment 1: c1psl_feedthrough_wiring_v2.pdf
c1psl_feedthrough_wiring_v2.pdf c1psl_feedthrough_wiring_v2.pdf c1psl_feedthrough_wiring_v2.pdf c1psl_feedthrough_wiring_v2.pdf c1psl_feedthrough_wiring_v2.pdf
  15176   Thu Jan 30 12:52:10 2020 JonUpdateBHDMetal OMCs procured

Last night Yehonathan and I located the two steel PMCs in the QIL, with help from Anchal. They are currently sitting on my desk in Bridge, inside a box that also contains optics and other OMC parts. I will bring them over to the 40m the next time I come.

  15178   Thu Jan 30 17:31:28 2020 JonUpdatePSLErrant FSS_INOFFSET change

A script I was testing errantly set C1:PSL-FSS_INOFFSET => 10 V at about 5:30 pm. I manually reverted the channel value to 0, but I don't know what the value was initially. Someone please check this value if there are problems locking the FSS.

  15184   Mon Feb 3 15:22:39 2020 JonUpdatePSLc1psl progress/Acromag ADC grounding

I tested the c1psl AO channels on the electronics bench on Friday. While I found all the wiring to be correct, some of the channels exhibited excess noise with all appearances of a grounding problem.

Today Jordan, Gautam, and I investigated this further. It is indeed a grounding problem, but actually with the Acromag ADCs. The Acromag DAC outputs are single-ended (return is grounded), so (for the purpose of a loopback test) I would expect to leave the ADC inputs ungrounded. This is the configuration I tested Friday. Today we also tested driving the ADC with a floating source. The ADC noise behavior is exactly the same, whether the source end is grounded or not.

However, grounding the minus pin of the ADC channel eliminates the noise. We don't understand why this seems to be required irrespective of the driving source, so there something we're missing about the ADC design. As it turns out, this same fix was made to the AI channels of the previously-upgraded Acromag machines. I know Chub and I had to do this for the AI channels of c1vac, but at the time we thought the source grounding was causing the issue. However, today Jordan and I looked inside c1iscaux, which Chub wired, and confirmed that its AI channels are wired in the same way.

So in any case, Jordan is grounding the c1psl AI channels in the same way as c1iscaux. Once this is done, we'll continue with the bench testing tomorrow.

gautam: here are my notes about this issue when i was doing the c1iscaux testing. As I note there, "previously-upgraded Acromag machines" in the plural may be a bit of a stretch - I have no idea what the grounding situation is in c1susaux / c1auxex for example.

  15194   Thu Feb 6 21:54:13 2020 JonUpdatePSLc1psl bench testing complete

Today I engineered the last piece of the new c1psl system: the multi-bit binary output (mbbo) channels that control the MC servo board gains. These 6-bit channels have to be split across two 4-bit Acromag registers. To enforce synchronous switching, I adapted the latch.py script developed by Gautam to address this problem in c1iscaux. Analogously to the c1iscaux implementation, I scripted the code to automatically run as a systemd service which is launched by the main modbusIOC service. I tested this all using the DB37 LED test board and confirmed it to work.

This now completes the electronics bench testing.

There are still several DB37 connectors to be wired, which carry only spare channels for future use and are not interfaced with the EPICS IOC. Jordan and I discussed this today and he or Chub will complete it shortly. To allow time for the spare channel wiring to be completed (as well as for more locking progress before interruption), Gautam and I think Monday/Tuesday next week would be the earliest possible window to install the new system.

  15226   Wed Feb 26 21:43:48 2020 JonSummaryBHDProjected IFO noise budget, post-BHD upgrade

To supplement the material presented during the BHD review, I've put together a projected noise budget for the 40m. Note these are the expected interferometer noises (originating in the IFO itself), not BHD readout noises. The key parameters for each case are listed in the figure title. Also attached is a tarball (attachment 4) containing the ipython notebook, data files, and rolled-back version of pygwinc which were used to generate these figures.

Attachment 1: Phase quadrature readout.

Attachment 2: Comparison to aLIGO design sensitivity (phase quadrature).

Attachment 3: Amplitude quadrature readout.

Attachment 1: 40m_phase_quad.pdf
40m_phase_quad.pdf
Attachment 2: 40m_aligo_comp.pdf
40m_aligo_comp.pdf
Attachment 3: 40m_ampl_quad.pdf
40m_ampl_quad.pdf
Attachment 4: noise_budget.tar
  15241   Mon Mar 2 23:49:03 2020 JonSummaryBHDProjected IFO noise budget, post-BHD upgrade

Updated noise budget curves, now computed using the latest version of pygwinc. This resolves the inconsistency between the gwinc quantum noise curves and Gautam's analytic calculations. As before, the key configuration parameters are listed in the figure titles.

Attachment 1: Phase quadrature

Attachment 2: Amplitude quadrature

Attachment 3: Comparison to aLIGO design (phase quadrature)

Quote:

The quantum noise curves here are not correct. c.f. amplitude quadrature noise budget.

Attachment 1: 40m_phase_quad.pdf
40m_phase_quad.pdf
Attachment 2: 40m_ampl_quad.pdf
40m_ampl_quad.pdf
Attachment 3: 40m_aligo_comp.pdf
40m_aligo_comp.pdf
  15244   Tue Mar 3 18:11:05 2020 JonSummaryBHDProjected IFO noise budget, post-BHD upgrade

Revised noise estimates, correcting a couple of factor of 2 and factor of pi errors found in the coil driver noise calculation. Also resolves a strain vs. displacement units confusion using the new pygwinc. Gautam and I have checked these noises against the analytical predictions and believe they are now accurate. Attachments are again:

Attachment 1: Phase quadrature

Attachment 2: Amplitude quadrature

Attachment 3: Comparison to aLIGO design (phase quadrature)

Attachment 1: 40m_phase_quad.pdf
40m_phase_quad.pdf
Attachment 2: 40m_ampl_quad.pdf
40m_ampl_quad.pdf
Attachment 3: 40m_aligo_comp.pdf
40m_aligo_comp.pdf
  15253   Wed Mar 4 22:38:31 2020 JonUpdatePSLc1psl communications problem resolved

I investigated the problem reported earlier today with the BIO1 channels. By logging the systemd messages generated when the IOC starts, I was immediately able to determine that the problem was not limited to BIO1. The modbus communications were failing for several other units as well.

Because some in-situ rewiring of a handful of channels had recently been done (more on this soon), I initially suspected that one of the Acromags had been damaged in the process. However, removing BIO1 (or other non-communicating modules) did not restore communications with the rest of the modules. To test whether the chassis was the source of the problem at all, we set up a fresh ADC (new out of the package) and directly connected it to the secondary Ethernet interface of c1psl. With only the one new ADC connected, the modbus IOC failed in exactly the same way.

To confirm that the new ADC did in fact work, we connected it to c1auxex in the same configuration. The unit worked fine connected to c1auxex. This established that the source of the problem was the c1psl host. After some extensive debugging, I traced the problem to a pre-execution script (part of the modbus IOC systemd service) which resets the secondary network interface (the one connected to the Acromag chassis) prior to launching the IOC. This was to ensure the secondary interface always had the correct IP address. It appears this reset was somehow creating a race condition that allowed the modbus initializations (first communications with the Acromags) to sometimes start before the network interface had actually come back up.

I still don't understand how this was happening, or why the pre script worked just fine up until yesterday, but eliminating the network interface reset fixes the problem in 100% of the trials we ran. Unfortunately we lost the entire day to debugging this problem, so the final round of testing is still to be completed. We plan to pick it back up tomorrow afternoon.

  15256   Thu Mar 5 19:45:23 2020 JonSummaryPSLC1PSL in-situ test results

We've completed almost all of the in-situ testing of the c1psl channels. During this process, we identified several channels which needed to be rewired to different Acromags (BIO sinking v. sourcing). We also elected to change the connector type of a few channels for practical advantages. Those modifications and other issues found during testing are detailed below. Also attached are the updated channel assignments, with a column indicating the in-situ testing status of each channel.

Post-installation modifications

  • All four channels connected to the sourcing BIO module were found to in fact require sinking I/O. They were reassigned to sinking BIO modules. Affected channels:
    • C1:PSL-FSS_FASTSWEEP
    • C1:PSL-FSS_SW1
    • C1:PSL-FSS_SW2
    • C1:PSL-PSL_Shutter
  • Added a new AI channel:
    • C1:PSL-FSS_MIXERM
  • Removed an unneccessary AI channel:
    • C1:PSL-FSS_LODET
  • Moved two AI channels from BNC connectors to a new Dsub connector (labelled DB25M-2 in the spreadsheet).
    • C1:PSL-FSS_RCTEMP
    • C1:PSL-FSS_RMTEMP_VOLTS

Issues identified during testing

  • Digital calibration. The following channels work, but we need to verify their EPICS calibration parameters (EGUF/EGUL):
    • C1:PSL-FSS_FASTGAIN
    • C1:PSL-FSS_FAST
    • C1:PSL-PMC_RFADJ
    • C1:PSL-PMC_MODET
  • IMC servo board. The Acromag channels themselves were found to work, but the linearity of the mbbo gain stages are in question (i.e., a potential problem with the board). GV is currently testing the servo board.
  • PSL QPD board apears to be dead. We connected a scope directly to the test points on the board and measured a high level of noise and no signal (for all four of the QPD channels). I understand this QPD has not been used in some time, so it may not have been noticed before.
  • WFS DC channels are saturating when the IMC is unlocked. The acceptance range of the Acromag ADC is only +/-10 V, but we measured sensor voltages as high as ~14 V. It appears that the old ADCs were somehow accepting a range of 0 to +20 V instead of -10 to +10 V. However, the Acromags do not support the input range 0-20 V. Since SNR is not critical for these channels (they're used only for initial alignment), I propose we simply install a voltage divider inside the chassis, just before the Acromag, for each of these signals.
Attachment 1: c1psl_feedthrough_wiring_-_By_Connector_(3).pdf
c1psl_feedthrough_wiring_-_By_Connector_(3).pdf c1psl_feedthrough_wiring_-_By_Connector_(3).pdf c1psl_feedthrough_wiring_-_By_Connector_(3).pdf c1psl_feedthrough_wiring_-_By_Connector_(3).pdf
  15276   Fri Mar 13 20:00:50 2020 JonUpdateComputersLoopback monitoring for slow machines

Summary

Today I finished implementing loopback monitors of the up/down state of the slow controls machines. They are visible on a new MEDM screen accessible from Sitemap > CDS > Slow Machine Status (pictured in attachment 1). Each monitor is a single EPICS binary channel hosted by the slow machine, which toggles its state at 1 Hz (an alive "blinker"). For each machine, the monitor is defined by a separate database file named c1[machine]_state.db located in the target directory.

This is implemented for all upgraded machines, which includes every slow machine except for c1auxey. This is the next and final one slated for replacement.

Implementation

The blinkers are currently implemented as soft channels, but I'd like to ultimately convert them to hard channels using two sinking/sourcing BIO units. This will require new wiring inside each Acromag chassis, however. For now, as soft channels, the monitors are sensitive to a failure of the host machine or a failure of the EPICS IOC. As hard channels, they will additionally be sensitive to a failure of the secondary network interface, as has been known to happen.

Each slow machine's IOC had to be restarted this afternoon to pick up the new channels. The IOCs were restarted according to the following procedure.

c1auxex

  • Disabled OPLEV servos on ETMX
  • Zeroed slow biases
  • Disabled watchdog
  • Restarted IOC
  • Reverted 1-3

​c1vac

  • Closed V1, VM1
  • Restarted IOC
  • Returned valves to original state

c1psl

  • Disabled IMC autolocker
  • Closed PSL shutter
  • Restarted IOC
  • Reverted 1-2

c1iscaux

  • ​Restarted IOC

c1susaux

  • Disabled IMC autolocker
  • Closed shutter
  • Disabled OPLEV servos on: MC1, MC2, MC3, BS, ITMX, ITMX, PRM, SRM
  • Zeroed slow biases
  • Disabled watchdogs
  • Restarted IOC
  • Reverted 1-5

The intial recovery of c1susaux did not succeed. Most visibly, the alignment state of the IFO was not restored. After some debugging, we found that the restart of the modbus service was partially failing at the final burt-restore stage. The latest snapshot file /opt/rtcds/caltech/c1/burt/autoburt/latest/c1susaux.snap was not found. I manually restored a known good snapshot from earlier in the day (15:19) and we were able to relock the IMC and XARM. GV and I were just talking earlier today about eliminating these burt-restores from the systemd files. I think we should.

Attachment 1: Screen_Shot_2020-03-13_at_7.59.55_PM.png
Screen_Shot_2020-03-13_at_7.59.55_PM.png
  15284   Thu Mar 26 17:41:18 2020 JonOmnistructureBHDBHD docs compilation

Since there has been a proliferation of BHD Google docs recently, I've linked them all from the BHD wiki page. Let's continue adding any new docs to this central list.

  15292   Thu Apr 2 16:31:33 2020 JonUpdateCDSC1AUXEY wiring + channel list
Quote:

I have made a wiring + channel list that need to be included in the new C!AUXEY Acromag.

I used Yehonathan's wiring assignments to lay the rest of groundwork for the final slow controls machine upgrade, c1auxey. Actions completed:

  • Created an internal wiring diagram for assembling the Acromag chassis (log in with LIGO.ORG credentials to view/edit)
  • Created a new target directory on the network drive:
/cvs/cds/caltech/target/c1auxey1

The "1" will be dropped after the new system is permanently installed.

  • Populated the target directory with files:
    • modbusIOC.service - wraps the EPICS IOC as a systemd service
    • ETMYaux.env - defines the EPICS environment variables
    • ETMYaux.cmd - command file to set up the EPICS IOC
    • ETMYaux.sh - enables DAC outputs to the suspension (executed lastly)
  • Created the EPICS channel databases:
    • ETMYaux.db - migration of the existing database
    • c1auxey_state.db - contains logic for loopback monitoring of the IOC "alive" state (visible from Sitemap > CDS > Slow Controls Status)

Hardware-wise, this system will require:

  • 2 Acromag XT-1221 units (ADC)
  • 1 Acromag XT-1541 unit (DAC)
  • 1 Acromag XT-1111 unit (sinking BIO)

I know that we do have these quantities left on hand. The next steps are to set up the Supermicro host and begin assembling the Acromag chassis. Both of these activities require an in-person presence, so I think this is as far as we can advance this project for now.

  15294   Fri Apr 3 12:09:53 2020 JonUpdateCDSC1AUXEY wiring + channel list
Quote:

We want to migrate the end shutter controls from c1aux to the end acromags. Could you include them to the list if not yet?

This will let us remove c1aux from the rack, I believe.

Yehonathan's list does include C1:AUX-GREEN_Y_Shutter and I copied its definition from /cvs/cds/caltech/target/c1aux/ShutterInterlock.db into the new ETMYaux.db file.

I noticed ShutterInterlock.db still contains about a dozen channels. Some of them appear to be ghosts (like the C1:AUX-PSL_Shutter[...] set, which has since become C1:PSL-PSL_Shutter[...] hosted on c1psl) but others like C1:AUX-GREEN_X_Shutter appear to still be in active use.

  15295   Fri Apr 3 13:40:07 2020 JonUpdateBHDBHD front-end complication

I wanted to pass along a complication pointed out by K. Thorne re: our plan to use Gen1 (old) Dolphin IPC cards in the new real-time machines: c1bhd, c1sus2. The implication is that we may be forced to install a very old OS (e.g., Debian 8) for compatibility with the IPC card driver, which could lead to other complications like an incompatibility with the modern network interface.

Hardware is easy - you will also need a DX switch and the cables

As for the driver - the last update (version 4.4.5) was in 2016.  The notes on it say valid for Linux kernel 2.6 to 3.x.  This implies that it will not work with Linux kernel 4.x and greater

So - Gentoo with 3.0 kernel OK, SL7 (kernel 3.10)  - OK,   Debian 8 (kernel 3.16) - OK   

But Debian 9 (kernel 4.9),Debian 10 (kernel 4.19) - NOT OK

We have Gentoo with kernel 3.0  boot server, etc. [used in L1,H1 production right now, but not much longer] The hard part here will be making sure we have network drivers for the SuperMicro 5018-MR.

CDS was never able to get real-time builds to work well on Linux kernels from 3.2 on up until we got to Debian 9. This is not to say that the tricks and stripped-down RCG we found worked for real-time on Debian 9 and 10 won’t work on, say, Debian 8.  But we have not tried.

I have a query out to Dolphin asking:

  1. Have they done any testing of these old drivers on Linux kernel 4.x (e.g., Debian 9/10)?
  2. Is there any way to buy modern IPC cards for the two new machines and interface them with our existing Gen1 network?

I'll add more info if I hear back from them.

  15299   Tue Apr 7 10:56:39 2020 JonUpdateBHDBHD front-end complication
Quote:

I have a query out to Dolphin asking:

  1. Have they done any testing of these old drivers on Linux kernel 4.x (e.g., Debian 9/10)?
  2. Is there any way to buy modern IPC cards for the two new machines and interface them with our existing Gen1 network?

Answers from Dolphin:

  1. No, and kernel 4.x (modern Linux) definitely will not work with the Gen1 cards.
  2. No, cards using different PCIe chipsets cannot be mixed.

Since upgrading every front end is out of the question, our only option is to install an old OS (Linux kernel < 3.x) on the two new machines. Based on Keith's advice, I think we should go with Debian 8. (Link to Keith's Debian 8 instructions.)

  15300   Tue Apr 7 15:30:40 2020 JonSummaryNoiseBudget40m noise budget migrated to pygwinc

In the past year, pygwinc has expanded to support not just fundamental noise calculations (e.g., quantum, thermal) but also any number of user-defined noises. These custom noise definitions can do anything, from evaluating an empirical model (e.g., electronics, suspension) to loading real noise measurements (e.g., laser AM/PM noise). Here is an example of the framework applied to H1.

Starting with the BHD review-era noises, I have set up the 40m pygwinc fork with a working noise budget which we can easily expand. Specific actions:

  • Updated the 40m fork to the latest pygwinc version (while preserving the commit history).
  • Added a directory ./CIT40m containing the 40m-specific noise budget files (created by GV).
  • Added an ipython notebook CIT40m.ipynb at the root level showing how to generate a noise budget.
  • Integrated our DAC and seismic noise estimators into pygwinc.
  • Marked the old 40m NB repo as obsolete (last commit > 2 yrs ago). Many of these noise estimates are probably stale, but I will work with GV to identify which ones can be migrated.

I set up our fork in this way to keep the 40m separate from the main pygwinc code (i.e., not added to as a built-in IFO type). With the 40m code all contained within one root-level directory (with a 40m-specific name), we should now always be able to upgrade to the latest pygwinc without creating intractable merge conflicts.

  15305   Thu Apr 16 21:13:20 2020 JonUpdateBHDBHD optics specifications

Summary

I've generated specifications for the new BHD optics. This includes the suspended relay mirrors as well as the breadboard optics (but not the OMCs).

To design the mode-matching telescopes, I updated the BHD mode-matching scripts to reflect Koji's draft layout (Dec. 2019) and used A La Mode to optimize ROCs and positions. Of the relay optics, only a few have an AOI small enough for curvature (astigmatism) and most of those do not have much room to move. This reduced the optimization considerably.

These ROCs should be viewed as a first approximation. Many of the distances I had to eyeball from Koji's drawings. I also used the Gaussian PRC/SRC modes from the current IFO, even though the recycling cavities will both slightly change. I set up a running list of items like these that we still need to resolve in the BHD README.

Optics Specifications

At a glance, all the specifications can be seen in the optics summary spreadsheet.

LO Telescope Design

The LO beam originates from the PR2 transmission (POP), near ITMX. It is relayed to the BHD beamsplitter (and mode-matched to the OMCs) via the following optical sequence:

  • LM1 (ROC = +10 m, AOI 3°)
  • LM2 (Flat, AOI  45°)
  • MMT1 (Flat, AOI  5°)
  • MMT2 (ROC = +3.5 m, AOI  5°)

The resulting beam profile is shown in Attachment 1.

AS Telescope Design

The AS beam is relayed from the SRM to the BHD beamsplitter (and mode-matched to the OMCs) via the following sequence:

  • AS1 (ROC = +1.5 m, AOI  3°)
  • AS2 (Flat, AOI  45°)
  • Lens (FL = -125 mm)

A lens is used because there is not enough room on the BHD breadboard for a pair of (low-AOI) telescope mirrors, like there is in the LO path. The resulting beam profile is shown in Attachment 2.

Attachment 1: LO_Beam_Calc-v1.pdf
LO_Beam_Calc-v1.pdf
Attachment 2: AS_Beam_Calc-v1.pdf
AS_Beam_Calc-v1.pdf
  15311   Thu Apr 23 09:52:02 2020 JonUpdateCamerasGigE w better NIR sensitivvity

Nice, and we should also permanently install the camera server (c1cam) which is still sitting on the electronics bench. It is running an adapted version of the Python 2/Debian 8 site code. Maybe if COVID continues long enough I'll get around to making the Python 3 version we've long discussed.

Quote:

There's this elog from Stephen about better 1064 sensitivity from Basler. We should consider getting one if he finds that its actual SNR is as good as we would expect from the QE improvement.

  15334   Fri May 15 09:18:04 2020 JonUpdateBHDBHD telescope designs accounting for ASC

Hang and I have reanalyzed the BHD telescope designs, with the goal of identifying sufficiently non-degenerate locations for ASC actuation. Given the limited room to reposition optics and the requirement to remain insensitive to small positioning errors, we conclude it is not possible put sufficient Gouy phase separation between the AS1/AS2 and LO1/LO2 locations. However, we can make the current layout work if we instead actuate AS1/AS4 and LO1/LO4. This would require actuating one optic on the breadboard for each relay path. If possible, we believe this offers the simplest solution (i.e., least modification to the current layout).

LO Telescope Design (Attachment 1)

Radius of curvatures:

  • LO1: +10 m
  • LO2: flat
  • LO3: +15 m
  • LO4: flat

AS Telescope Design (Attachment 2)

Radius of curvatures:

  • AS1: +3 m
  • AS2: flat
  • AS3: -1 m
  • AS4: flat
Attachment 1: LOpath.pdf
LOpath.pdf
Attachment 2: ASpath.pdf
ASpath.pdf
  15379   Sat Jun 6 14:07:30 2020 JonUpdateBHDStock-Part Mode-Matching Telescope Option

Summary

For the initial phase of BHD testing, we recently discussed whether the mode-matching telescopes could be built with 100% stock optics. This would allow the optical system to be assembled more quickly and cheaply at a stage when having ultra-low loss and scattering is less important. I've looked into this possibility and conclude that, yes, we do have a good stock optics option. It in fact achieves comprable performance to our optimized custom-curvature design [ELOG 15357]. I think it is certainly sufficient for the initial phase of BHD testing.

Vendor

It turns out our usual suppliers (e.g., CVI, Edmunds) do not have enough stock options to meet our requirements. This is for two reasons:

  • For sufficient LO1-LO2 (AS1-AS4) Gouy phase separation, we require a very particular ROC range for LO1 (AS1) of 5-6 m (2-3 m).
  • We also require a 2" diameter for the suspended optics, which is a larger size than most vendors stock for curved reflectors (for example, CVI has no stock 2" options).

However I found that Lambda Research Optics carries 1" and 2" super-polished mirror blanks in an impressive variety of stock curvatures. Even more, they're polished to comprable tolerances as I had specificied for the custom low-scatter optics [DCC E2000296]: irregularity < λ/10 PV, 10-5 scratch-dig, ROC tolerance ±0.5%. They can be coated in-house for 1064 nm to our specifications.

From modeling Lambda's stock curvature options, I find it still possible to achieve mode-matching of 99.9% for the AS beam and 98.6% for the LO beam, if the optics are allowed to move ±1" from their current positions. The sensitivity to the optic positions is slightly increased compared to the custom-curvature design (but by < 1.5x). I have not run the stock designs through Hang's full MC corner-plot analysis which also perturbs the ROCs [ELOG 15339]. However for the early BHD testing, the sensitivity is secondary to the goal of having a quick, cheap implementation.

Stock-Part Telescope Designs

The following tables show the best telescope designs using stock curvature options. It assumes the optics are free to move ±1" from their current positions. For comparison, the values from the custom-curvature design are also given in parentheses.

AS Path

The AS relay path is shown in Attachment 1:

  • AS1-AS4 Gouy phase separation: 71°
  • Mode-matching to OMC: 99.9%
Optic ROC (m) Distance from SRM AR (m)
AS1 2.00  (2.80) 0.727  (0.719)
AS2 Flat   (Flat) 1.260  (1.260)
AS3 0.20  (-2.00) 1.864  (1.866)
AS4 0.75  (0.60) 2.578  (2.582)

LO Path

The LO relay path is shown in Attachment 2:

  • LO1-LO2 Gouy phase separation: 67°
  • Mode-matching to OMC: 98.6%
Optic ROC (m) Distance from PR2 AR (m)
LO1 5.00  (6.00) 0.423  (0.403)
LO2 1000 (1000) 2.984  (2.984)
LO3 0.50  (0.75) 4.546  (4.596)
LO4 0.15  (-0.45) 4.912  (4.888)

Ordering Information

I've created a new tab in the BHD procurement spreadsheet ("Stock MM Optics Option") listing the part numbers for the above telescope designs, as well as their fabrication tolerances. The total cost is $2.8k + the cost of the coatings (I'm awaiting a quote from Lambda for the coatings). The good news is that all the curved substrates will receive the same HR/AR coatings, so I believe they can all be done in a single coating run.

Attachment 1: ASpathStock.pdf
ASpathStock.pdf
Attachment 2: LOpathStock.pdf
LOpathStock.pdf
  15382   Mon Jun 8 17:40:22 2020 JonUpdateBHDAstigmatism and scattering plots

MM_total = (MM_vert + MM_horiz) / 2. 

The large astigmatic MM loss in the LO case is mainly due to the strong LO4 curvature (R=0.15m) with a 10 deg AOI. I looked again at whether LO1 could be increased from R=5m to the next higher stock value of 7.5m, as this would allow weaker curvatures on LO3 and LO4. However, no, that is not possible---it reduces the LO1-LO2 Gouy phase separation to only 18 deg.

There is, however, a good stock-curvature option if we want to reconsider actuating LO4 instead of LO2 (attachment 1). It achieves 99.2% MM with the OMCs, allowing positions to vary +/-1" from the current design. The LO1-LO4 Gouy phase separation is 72 deg.

Optic ROC (m) Distance from PR2 AR (m)
LO1 10 0.378
LO2 1000 2.984
LO3 10 4.571
LO4 7.5 4.926

Alternatively, we could look at reducing the AOI on LO3 and LO4 (keeping LO1-LO2 actuation).

Attachment 1: LOpathStock2.pdf
LOpathStock2.pdf
  15384   Mon Jun 8 21:45:47 2020 JonUpdateBHDAstigmatism and scattering plots

Hmm? T1300364 suggests MM_total = Sqrt(MM_Vert * MM_Horiz)

  15386   Tue Jun 9 14:55:43 2020 JonUpdateBHDMM telescope actuation range requirments

I don't think we ever discussed why the angular RMS of the ETMs is so much higher than the ITMs. Maybe that's a separate matter because, even assuming the worst case, the actuation range requirement is

(0.82 μrad RMS) x (15 μrad/μrad) x (10 safety factor) = 0.12 mrad

which is still only order 1% of the pitch/yaw pointing range of the Small Optic Suspensions, according to P1600178 (sec. IV. A). Can we check this requirement off the list?

Quote:

We computed the required actuation range for the telescope design in elog:15357. The result is summarized in the table below. Here we assume we misalign an IFO mirror by 1 urad, and then compute how many urad do we need to move the (AS1, AS4) or (LO1, LO2) mirrors to simultaneously correct for the two gouy phases. 

Actuation requirement in urad per urad misalignment
[urad/urad] ITMX ITMY ETMX ETMY BS PRM PR2 PR3 SR3 SRM
AS1 1.9 2.1 -5.0 -5.5 0.5 0.5 -0.3 0.2 0.1 0.6
AS4 2.9 2.0 -8.8 -5.5 -5.9 -0.7 1.3 -0.7 -0.5 0.7
LO1 -4.0 -3.9 11.0 10.4 1.9 -0.4 -0.2 0.1 0.0 -1.1
LO2 -5.0 -3.7 15.1 10.4 8.7 0.8 1.9 1.1 0.7 -1.3

The most demanding ifo mirrors are the ETMs and the BS, for every 1 urad misalignment the telescope needs to move 10-15 urad to correct for that. However, it is unlikely for those mirrors to move more 100 nrad for a locked ifo with ASC engaged. Thus a few urad actuation should be sufficient. For the recycling mirrors, every 1 urad misalignment also requires ~ 1 urad actuation. 

As a result, if we could afford 10 urad actuation range for each telescope suspension, then the gouy phase separations we have should be fine. 

================================================================

Edits:

We looked at the oplev spectra from gps 1274418500 for 512 sec. This should be a period when the ifo was locked in the PRFPMI state according to elog:15348. We just focused on the yaw data for now. Please see the attached plots. The solid traces are for the ASD, and the dotted ones are the cumulative rms. The total rms for each mirror is also shown in the legend. 

I am now confused... The ITMs looked somewhat reasonable in that at least the < 1 Hz motion was suppressed. The total rms is ~ 0.1 urad, which was what I would expect naively (~ x100 times worse than aLIGO). 

There seems to be no low-freq suppression on the ETMs though... Is there no arm ASC at the moment???

 

  15389   Thu Jun 11 09:37:38 2020 JonUpdateBHDConclusions on Mode-Matching Telescopes

After further astigmatism/tolerance analysis [ELOG 15380, 15387] our conclusion is that the stock-optic telescope designs [ELOG 15379] are sufficient for the first round of BHD testing. However, for the final BHD hardware we should still plan to procure the custom-curvature optics [DCC E2000296]. The optimized custom-curvature designs are much more error-tolerant and have high probability of achieving < 2% mode-matching loss. The stock-curvature designs can only guarantee about 95% mode-matching.

Below are the final distances between optics in the relay paths. The base set of distances is taken from the 2020-05-21 layout. To minimize the changes required to the CAD model, I was able to achieve near-maximum mode-matching by moving only one optic in each relay path. In the AS path, AS3 moves inwards (towards the BHDBS) by 1.06 cm. In the LO path, LO4 moves backwards (away from the BHDBS) by 3.90 cm.

AS Path

Interval Distance (m) Change (cm)
SRMAR-AS1 0.7192 0
AS1-AS2 0.5405 0
AS2-AS3 0.5955 -1.06
AS3-AS4 0.7058 -1.06
AS4-BHDBS 0.5922 0
BHDBS-OMCIC 0.1527 0

LO Path

Interval Distance (m) Change (cm)
PR2AR-LO1 0.4027 0
LO1-LO2 2.5808 0
LO2-LO3 1.5870 0
LO3-LO4 0.3691 +3.90
LO4-BHDBS 0.2573 +3.90
BHDBS-OMCIC 0.1527 0
  15402   Tue Jun 16 13:35:03 2020 JonUpdateVACTemporary vac fix / IFO usable again

[Jon, Jordan, Koji]

Today Jordan reconfigured the vac system to allow pumping of the main volume resume, with Jon and Koji remotely advising. All clear to resume normal IFO activities. However, the vac system is operating in a temporary configuration that will have to be reverted as we locate replacement components. Details below.

Procedure

Since serial readback of the TP2 controller seems to be failing, we reconfigured the system with TP3 now backing for TP1. TP2 was valved off (at V4) and shut down until we can replace its controller.

TP3 has its own problems, however. It was valved off in January after its temperature readback began glitching and spuriously triggering the interlocks [ELOG 15140]. However the problem appears to be limited only one readback (rotation speed, current, voltage are fine) and there is enough redundancy in the pump-dependent interlock conditions to safely connect it to the main volume.

We also discovered that sometime since January, the TP3 dry pump has failed. The foreline pressure had risen to 165 torr. Since the TP2 and TP3 dry pumps are not interchangeable (Agilent vs. Varian), we instead valved in the auxiliary dry pump and disconnected the failed dry pump using a KF blank. This is a temporary arrangement until the permanent dry pump can be repaired. Jordan removed it to replace the tip seals and will test it in the bake lab before reinstalling.

With this configuration in place, we proceeded to pump down the main volume without issue (attachment 1). We monitored the pumpdown for about 45 min., until the pressure had reached ~1E-5 torr and TP3 had been transitioned to standby (low-speed) mode.

Summary of topology changes:

  • TP2 valved off and shut down until controller can be replaced
  • TP3 temporarily backing for TP1
  • Auxiliary dry pump temporarily backing for TP3
  • TP3 dry pump has been removed for repairs
Attachment 1: Pumpdown.png
Pumpdown.png
  15406   Thu Jun 18 11:00:24 2020 JonUpdateVACQuestions/comments on vacuum
Quote:
  • Isn’t it true that we didn’t digitally monitor any of the TP diagnostic channels before 2018 December? I don’t have the full history but certainly there wasn’t any failure of the vacuum system connected to pump current/temp/speed from Sep 2015-Dec2018, whereas we have had 2 interruptions in 6 months because of flaky serial communications.

Looking at images of the old vac screens, the TP2/3 rotation speed and status string were digitally monitored. However I don't know if there were software interlocks predicated on those.

Quote:
  • According to the manuals, the turbo-pumps have their own internal logic to shut off the pump when either bearing temperature exceeds 60C or current exceeds 1.5A. I agree its good to have some redundancy, but do we really expect that our outer interlock loops will function if the internal ones fail?

The temperature and current interlocks are implemented precisely because the pumps can shut themselves off. The concern is not about damaging the pumps (their internal logic protects against that); it's that a pump could automatically shut down and back-vent the IFO to atmosphere. Another interlock (e.g., the pressure differentials) might catch it, but it would depend on the back-vent rate and the scenario has never been tested. The temperature and current interlocks are set to trip just before the pump reaches its internal shut-down threshold.

One way we might be able to reduce our reliance on the flaky serial readbacks is to implement rotation-speed hardware interlocks. The old vac documentation alludes to these, but as far as Chub and I could determine in 2018, they never actually existed. The older turbo controllers, at least, had an analog output proportional to speed which could be used to control a relay to interrupt the V4/5 control signals. I'll look into this for the new controllers. If it could be done, we could likely eliminate the layer of serial-readback interlocks altogether.

 
  • I also think we should finally implement the email alert in the event the vacuum interlock is tripped. I can implement this if no one else volunteers.

That would be awesome if you're willing to volunteer. I agree this would be great to have.

  15408   Thu Jun 18 14:13:03 2020 JonUpdateVACQuestions/comments on vacuum
I agree there were MEDM fields, but I can't find any record of these channels being recorded till 2018 December, so I don't agree that they were being digitally monitored. You can also look back in the elog (e.g. here and here) and see that the display fields are just blank. I would then assume that no interlocks were dependent on these channels, because otherwise the vacuum interlocks would be perpetually tripped.

Right, I doubt they were ever recorded or used for interlocks. But the readbacks did work at one point in the past. There's a photo of the old vac monitor screen on p. 19 of E1500239 (last updated 2017) which shows the fields once alive.

Sorry but I'm having trouble imagining a scenario how the pressure gauges wouldn't register this before the IFO volume is compromised. Is there some back of the envelope calculations I can do to understand this? Since both the pressure gauges and the TP diagnostic channels are being monitored via EPICS, the refresh rate is similar, so I don't see how we can have a pump temperature / speed / current threshold tripped but NOT have this be registered on all the pressure gauges, seems like a bit of a contrived scenario to me. Our thresholds currently seem to be arbitrary numbers anyway, or are they based on some expected backstreaming rate? Isn't this scenario degenerate with a leak elsewhere in the vacuum envelope that would be caught by the differential pressure interlocks?​

I don't disagree that the pressure gauges would register the change. What I'm not sure about is whether the change would violate any of the existing interlock conditions, triggering a shutdown. Looking at what we have now, the only non-pump-related conditions I see that might catch it are the diffpres conditions:

  • abs(P2 - PTP2) > 1 torr (for a TP2 failure)

  • abs(P3 - PTP3) > 1 torr (for a TP3 failure)

  • abs(P1a - P2) > 1 torr (for either a TP2 or TP3 failure)

For the P1a-P2 differential, the threshold of 1 torr is the smallest value that in practice still allows us to pump down the IFO without having to disable the interlocks (P1a-P2 is the TP1 intake/exhaust differential). The purpose of the P2-PTP2/P3-PTP3 differentials is to prevent V4/5 from opening and suddenly exposing the spinning turbo to high pressure. I'm not aware of a real damage threshold calculation that any one has done; I think < 1 torr is lore passed down by Steve.

If a turbo pump fails, the rate it would backstream is unknown (to me, at least) and likely depends on the failure mode. The scenario I'm concerned about is if the backstream rate is slower than the conduction time through the pumspool and into the main volume. In that case, the pressure gauges will rise more or less together all the way up to atmosphere, likely never crossing the 1 torr differential pressure thresholds.

For the email alert, can you expose a soft channel that is a flag - if this flag is not 1, then the service will send out an email.

There's already a channel C1:Vac-error_status, where if the value is anything other than an empty string, there is an interlock tripped. Does that work?

  15412   Thu Jun 18 22:33:57 2020 JonOmnistructureVACVac hardware purchase list

Replacement Hardware Purchase List

I've created a purchase list of hardware needed to restore the aging vacuum system. This wasn't planned as part of the BHD upgrade, but I've added it to the BHD procurement list since hardware replacements have become necessary.

The list proposes replacing the aging TP3 Varian turbo pump with the newer Agilent model which has already replaced TP2. It seems I was mistaken in believing we already had a second Agilent pump on hand. A thorough search of the lab has not turned it up, and Steve himself has told me he doesn't remember ordering a second one. Fortunately Steve did leave us a detailed Agilent parts list [ELOG 14322].

It also proposes replacing the glitching TP2 Agilent controller with a new one. The existing one can be sent back for repair and then retained as a spare. Considering that one of these controllers is already malfunctioning after < 2 years, I think it's a very good idea to have a spare on hand.

Known Hardware Issues

Below is our current list of vacuum hardware issues. Items that this purchase list will address (limited to only the most urgent) are highlighted in yellow.

  • Replace the UPS
    • Need a 240V socket for TP1 (currently TP1 is not protected from power loss)
    • Need RS232/485 comms with the interlock server (current UPS: serial readbacks have failed, battery is failing)
  • Remove/replace the failed pressure gauges (~5)
  • Add more cold cathode sensors to the main volume for sensor redundancy (currently the main-volume interlocks rely on only 1 working sensor)
  • Replace TP3 (controller is failing)
  • Replace TP2 controller (serial interface has failed)
  • Remove RP2
    • Dead and also not needed. We already have to throttle the pumpdown rate with only two roughing pumps
  • Remove/refurbish the cryopump
    • Contamination risk to have it sitting connectable to the main volume
  15413   Fri Jun 19 07:40:49 2020 JonUpdateVACQuestions/comments on vacuum

I think we should discuss interlock possibilities at a 40m meeting. I'm reluctant to make the system more complicated, but perhaps we can find ways to reduce the reliance on the turbo pump readbacks. I agree they've proven to be the least reliable.

While we may be able to improve the tolerance to certain kinds of hardware malfunctions (and if so, we should), I don't see interlocks triggering on abnormal behavior of critical equipment as the root problem. As I see it, our bigger problem is with all the malfunctioning, mostly end-of-lifetime pieces of vacuum equipment still in use. If we can address the hardware problems, as I'm trying to do with replacements [ELOG 15412], I think that in itself will make the interlocking much less of an issue.

Quote:

So why not just have a special mode for the interlock code during pumpdown and venting, and during normal operation we expect the main volume pressure to be <100uTorr so the interlock trips if this condition is violated? These can just be EPICS buttons on the Vac control MEDM screen. Both of these procedures are not "business as usual", and even if we script them in the future, it's likely to have some operator supervising, so I don't think it's unreasonable to have to switch between these modes. I just think the pressure gauges have demonstrated themselves to be much more reliable than these TP serial readbacks (as you say, they worked once upon a time, but that is already evidence of its flakiness?). The Pirani gauges are not ultra-reliable, they have failed in the past, but at least less frequently than this serial comm glitching. In fact, if these readbacks are so flaky, it's not impossible that they don't signal a TP shutdown? I just think the real power of having these multi-channel diagnostics is lost without some AND logic - a turbopump failure is likely to result in an increase in pump current and temperature increase and pump speed decrease, so it's not the individual channel values that should be determining if an interlock is tripped.

Ok, this can be added pretty easily. Its value will just be toggled between 1 and 0 every time the interlock server raises/clears the existing string channel. Adding the channel will require restarting the whole vac IOC, so I'll do it at a time when Jordan is on hand in case something fails to come back up.

Quote:

It would be better to have a flag channel, might be useful for the summary pages too. I will make it if it is too much trouble.

  15421   Mon Jun 22 10:43:25 2020 JonConfigurationVACVac maintenance at 11 am

The vac system is going down at 11 am today for planned maintenance:

  • Re-install the repaired TP2 and TP3 dry pumps [ELOG 15417]
  • Incorporate an auto-mailer and flag channel into the controls code for signaling tripped interlocks [ELOG 15413]

We will advise when the work is completed.

  15424   Mon Jun 22 20:06:06 2020 JonConfigurationVACVac maintenance complete

This work is finally complete. The dry pump replacement was finished quickly but the controls updates required some substantial debugging.

For one, the mailer code I had been given to install would not run against Python 3.4 on c1vac, the version run by the vac controls since about a year ago. There were some missing dependencies that proved difficult to install (related to Debian Jessie becoming unsupported). I ultimately solved the problem by migrating the whole system to Python 3.5. Getting the Python keyring working within systemd (for email account authentication) also took some time.

Edit: The new interlock flag channel is named C1:Vac-interlock_flag.

Along the way, I discovered why the interlocks had been failing to auto-close the PSL shutter: The interlock was pointed to the channel C1:AUX-PSL_ShutterRqst. During the recent c1psl upgrade, we renamed this channel C1:PSL-PSL_ShutterRqst. This has been fixed.

The main volume is being pumped down, for now still in a TP3-backed configuration. As of 8:30 pm the pressure had fallen back to the upper 1E-6 range. The interlock protection is fully restored. Any time an interlock is triggered in the future, the system will send an immediate notification to 40m mailing list. 👍

Quote:

The vac system is going down at 11 am today for planned maintenance:

  • Re-install the repaired TP2 and TP3 dry pumps [ELOG 15417]
  • Incorporate an auto-mailer and flag channel into the controls code for signaling tripped interlocks [ELOG 15413]
Attachment 1: Pumpdown-6-22-20.png
Pumpdown-6-22-20.png
  15446   Wed Jul 1 18:03:04 2020 JonConfigurationVACUPS replacements

​I looked into how the new UPS devices suggested by Chub would communicate with the vac interlocks. There are several possible ways, listed in order of preference:

  • Python interlock service directly queries the UPS via a USB link using the (unofficial) tripplite package. Direct communication would be ideal because it avoids introducing a dependency on third-party software outside the monitoring/control capability of the interlock manager. However the documentation warns this package does not work for all models...
  • Configure Tripp Lite's proprietary software (PowerAlert Local) to send SYSLOG event messages (UDP packets) to a socket monitored by the Python interlock manager.
  • Configure the proprietary software to execute a custom script upon an event occurring. The script would, e.g., set an EPICS flag channel which the interlock manager is continually monitoring.

I recommend we proceed with ordering the Tripp Lite 36HW20 for TP1 and Tripp Lite 1AYA6 for TP2 and TP3 (and other 120V electronics). As far as I can tell, the only difference between the two 120V options is that the 6FXN4 model is TAA-compliant.

  15456   Mon Jul 6 15:10:40 2020 JonSummaryBHD40m --> A+ BHD design analysis

As suggested last week, Hang and I have reviewed the A+ BHD status (DRD, CDD, and reviewers' comments) and compiled a list of key unanswered questions which could be addressed through Finesse analysis.

In anticipation of others helping with this modeling effort, we've tried to break questions into self-contained projects and estimated their level of difficulty. As you'll see, they range from beginner to Finesse guru.

  15462   Thu Jul 9 16:02:33 2020 JonHowToCDSProcedure for setting up BHD front-ends

Here is the procedure for setting up the three new BHD front-ends (c1bhd, c1sus2, c1ioo - replacement). This plan is based on technical advice from Rolf Bork and Keith Thorne.

The overall topology for each machine is shown here. As all our existing front-ends use (obsolete) Dolphin PCIe Gen1 cards for IPC, we have elected to re-use Dolphin Gen1 cards removed from the sites. Different PCIe generations of Dolphin cards cannot be mixed, so the only alternative would be to upgrade every 40m machine. However the drivers for these Gen1 Dolphin cards were last updated in 2016. Consequently, they do not support the latest Linux kernel (4.x) which forces us to install a near-obsolete OS for compatibility (Debian 8).

Hardware

Software

  • OS: Debian 8.11 (Linux kernel 3.16)
  • IPC card driver: Dolphin DX 4.4.5 [works only with Linux kernel 2.6 to 3.x]
  • I/O card driver: None required, per the manual

Install Procedure

  1. Follow Keith Thorne's procedure for setting up Debian 8 front-ends
  2. Apply the real-time kernel patches developed for Debian 9, but modified for kernel 3.16 [these are UNTESTED against Debian 8; Keith thinks they may work, but they weren't discovered until after the Debian 9 upgrade]
  3. Install the PCIe expansion cards and Dolphin DX driver (driver installation procedure)
  15465   Thu Jul 9 18:00:35 2020 JonConfigurationVACUPS replacements

Chub has placed the order for two new UPS units (115V for TP2/3 and a 220V version for TP1).

They will arrive within the next two weeks.

Quote:

​I looked into how the new UPS devices suggested by Chub would communicate with the vac interlocks. There are several possible ways, listed in order of preference:

  • Python interlock service directly queries the UPS via a USB link using the (unofficial) tripplite package. Direct communication would be ideal because it avoids introducing a dependency on third-party software outside the monitoring/control capability of the interlock manager. However the documentation warns this package does not work for all models...
  • Configure Tripp Lite's proprietary software (PowerAlert Local) to send SYSLOG event messages (UDP packets) to a socket monitored by the Python interlock manager.
  • Configure the proprietary software to execute a custom script upon an event occurring. The script would, e.g., set an EPICS flag channel which the interlock manager is continually monitoring.

I recommend we proceed with ordering the Tripp Lite 36HW20 for TP1 and Tripp Lite 1AYA6 for TP2 and TP3 (and other 120V electronics). As far as I can tell, the only difference between the two 120V options is that the 6FXN4 model is TAA-compliant.

  15499   Thu Jul 23 15:58:24 2020 JonSummaryVACVacuum controls refurbishment plan

This year we've struggled with vacuum controls unreliability (e.g., spurious interlock triggers) caused by decaying hardware. Here are details of the vacuum refurbishment plan I described on the 40m call this week.

 Refurbish TP2 and TP3 dry pumps. Completed [ELOG 15417].

 Automated notifications of interlock-trigger events. Email to 40m list and a new interlock flag channel. Completed [ELOG 15424].

Replace failing UPS.

  • Two new Tripp Lite units on order, 110V and 230V [ELOG 15465].
  • Jordan will install them in the vacuum rack once received.
  • Once installed, Jon will come test the new units, set up communications, and integrate them into the interlock system following this plan [ELOG 15446].
  • Jon will move the pumps and other equipment to the new UPS units only after completing the above step.

Remove interlock dependencies on TP2/TP3 serial readbacks. Due to persistent glitching [ELOG 15140, ELOG 15392].

Unlike TP2 and TP3, the TP1 readbacks are real analog signals routed to Acromags. As these have caused us no issues at all, the plan is to eliminate dependence on the TP2/3 digital readbacks in favor of the analog controller outputs. All the digital readback channels will continue to exist, but the interlock system will no longer depend on them. This will require adding 2 new sinking BI channels each for TP2 and TP3 (for a total of 4 new channels). We have 8 open Acromag XT1111 channels in the c1vac system [ELOG 14493], so the new channels can be accommodated. The below table summarizes the proposed changes.

Channel Type Status Description Interlock
C1:Vac-TP1_current AI exists Current draw (A) keep
C1:Vac-TP1_fail BI exists Critical fault has occurred keep
C1:Vac-TP1_norm BI exists Rotation speed is within +/-10% of set point new
C1:Vac-TP2_rot soft exists Rotation speed (krpm) remove
C1:Vac-TP2_temp soft exists Temperature (C) remove
C1:Vac-TP2_current soft exists Current draw (A) remove
C1:Vac-TP2_fail BI new Critical fault has occurred new
C1:Vac-TP2_norm BI new Rotation speed is >80% of set point new
C1:Vac-TP3_rot soft exists Rotation speed (krpm) remove
C1:Vac-TP3_temp soft exists Temperature (C) remove
C1:Vac-TP3_current soft exists Current draw (A) remove
C1:Vac-TP3_fail BI new Critical fault has occurred new
C1:Vac-TP3_norm BI new Rotation speed is >80% of set point new
  15501   Mon Jul 27 15:48:36 2020 JonSummaryVACVacuum parts ordered

To carry out the next steps of the vac refurbishment plan [ELOG 15499], I've ordered parts necessary for interfacing the UPS units and the analog TP2/3 controller outputs with c1vac. The purchase list is appended to the main BHD list and is located here. Some parts we already had in the boxes of Acromag materials. Jordan is gathering what we do already have and staging it on the vacuum controls console table - please don't move them or put them away.

Quote:

Replace failing UPS.

Remove interlock dependencies on TP2/TP3 serial readbacks. Due to persistent glitching [ELOG 15140, ELOG 15392].

  15502   Tue Jul 28 12:22:40 2020 JonUpdateVACVac interlock test today 1:30 pm

This afternoon Jordan is going to carry out a test of the V4 and V5 hardware interlocks. To inform the interlock improvement plan [15499], we need to characterize exactly how these work (they pre-date the 2018 upgrade). I have provided him a sequence of steps for each test and will also be backing him up on Zoom.

We will close V1 as a precaution but there should be no other impact to the IFO. The tests are expected to take <1 hour. We will advise when they are completed.

  15504   Tue Jul 28 14:11:14 2020 JonUpdateVACVac interlock test today 1:30 pm

This test has been completed. The IFO configuration has been reverted to nominal.

For future reference: yes, both the V4 and V5 hardware interlocks were found to still be connected and work. A TTL signal from the analog output port of each pump controller (TP2 and TP3) is connected to an auxiliary relay inside the main valve relay box. These serve the purpose of interupting the (Acromag) control signal to the primary V4/5 relay. This interrupt is triggered by each pump's R1 setpoint signal, which is programmed to go low when the rotation speed falls below 80% of the low-speed setting.

Quote:

This afternoon Jordan is going to carry out a test of the V4 and V5 hardware interlocks. To inform the interlock improvement plan [15499], we need to characterize exactly how these work (they pre-date the 2018 upgrade). I have provided him a sequence of steps for each test and will also be backing him up on Zoom.

We will close V1 as a precaution but there should be no other impact to the IFO. The tests are expected to take <1 hour. We will advise when they are completed.

  15525   Fri Aug 14 10:03:37 2020 JonUpdateCDSTiming distribution slot availability

That's great news we won't have to worry about a new timing fanout for the two new machines, c1bhd and c1sus2. And there's no plan to change Dolphin IPC drivers. The plan is only to install the same (older) version of the driver on the two new machines and plug into free slots in the existing switch.

Quote:

The new dolphin eventually helps us. But the installation is an invasive change to the existing system and should be done at the installation stage of the 40m BHD.

  15526   Fri Aug 14 10:10:56 2020 JonConfigurationVACVacuum repairs today

The vac system is going down now for planned repairs [ELOG 15499]. It will likely take most of the day. Will advise when it's back up.

ELOG V3.1.3-