40m QIL Cryo_Lab CTN SUS_Lab TCS_Lab OMC_Lab CRIME_Lab FEA ENG_Labs OptContFac Mariner WBEEShop
  40m Log, Page 80 of 339  Not logged in ELOG logo
ID Date Authorup Type Category Subject
  13987   Tue Jun 19 18:56:55 2018 JonUpdateGeneralAUX beam alignment issues

Not much progress today with the AUX cavity scans. I've determined there still are some alignment issues.

At the start of today a large AUX/PSL beat note was visible on the AS110 sensor, at a similar power as where we left off last night (-60 dBm). Proceeding from there, I attempted to reproduce Johannes' measurement of the cavity transmission resonances. I misaligned the X-arm, locked the Y-arm cavity, and scanned the AUX RF offset approximately 8 MHz in 2 kHz steps. This should have swept through two FSRs, but nothing was visible.

Further inspection revealed that none of the PSL light was making it back to through the AUX fiber to the PSL table. I take this to mean that the beam seen earlier on AS110 was the ITMY reflection, and that the AUX injection axis was no longer reaching ETMY. I also found that the AUX beam size just after the 90/10 beasmsplitter looks anomolously large. Maybe a lens was recently changed? In any case, the mode-matching looks like it is going to need to be readjusted.

  13994   Thu Jun 21 09:33:02 2018 JonUpdate AUX Mode Scans of YARM, PRC cavities

[Jon, Keerthana, Sandrina]

Yesterday we carried out preliminary proof-of-concept measurements using the new AS-port-injected AUX laser to resolve cavity mode resonances.

At the time we started, I found the beat note levels consistent with what Johannes had reported the night before post-realignment. Hence we did not change the AUX alignment.

Test 1: YARM Mode Scan

  • IFO locked in YARM configuration on carrier.
  • Confirmed the presence of a -80 dBm beat note on the temporary YEND broadband PD (i.e., at the cavity transmission).
  • Slowly canned the RF offset of the AUX laser from 50 MHz (nominal) to 60 MHz in 10 kHz steps.
  • Attachment 1 shows the measured scan in "max hold" mode. The bottom panel is the transmission spectrum and the top panel is the reflection, with the AUX/PSL carrier-carrier beat note visible to the far left. In addition to the FSR, it looks to me like the scan resolves at least two HOMs.

Test 2: PRC Mode Scan

  • IFO locked in PRMI configuration on carrier.
  • Moved the temporary 150 MHz PDA10CF from the YEND to an unused pickoff of the REFL33 beam (i.e., the PRC transmission of the AUX beam). There was an existing 50-50 beamsplitter just before REFL33 whose reflected beam was directed onto a beam dump. The PD is now placed in that location. The modification to the AS table is shown in Attachment 2.
  • We made a similar slow scan of the AUX RF offset over ~35 MHz in 10 kHz steps.
  • We resolve the 22 MHz FSR, but it is apparent that an incoherent "max-hold" analyzer measurement is inadequate. The problem is that in max-hold mode, because the 11 MHz-spaced PSL sidebands also beat with the AUX subcarrier, we measure a messy superposition of the PSLcarrier-AUXcarrier beat AND all of the PSLsideband-AUXcarrier beats. The next step is to use the AOM to make a coherent measurement at only the frequency of PSL/AUX carrier-carrier beat.

The SURFs have the data from last night's scans and will be separately posting plots of these measurements. We'll continue today with mode scans using AM sidebands rather than the AUX RF offset.

Attachment 1: YARM_AUX_RF-offset_scan.pdf
YARM_AUX_RF-offset_scan.pdf
Attachment 2: temp_broadband_refl33.pdf
temp_broadband_refl33.pdf
  14010   Sat Jun 23 13:08:41 2018 JonUpdateAUXFirst Coherent AUX Scan of PRC Using AM Sidebands

[Jon, Keerthana, Sandrine]

Thu.-Fri. we continued with PRC scans using the AUX laser, but now the "scanned" parameter is the frequency of AM sidebands, rather than the frequency of the AUX carrier itself. The switch to AM (or PM) allows us to coherently measure the cavity transfer as a function of modulation frequency.

In order to make a sentinel measurement, I installed a broadband PDA255 at an unused pickoff behind the first AUX steering mirror on the AS table. The sentinel PD measures the AM actually imprinted on the light going into the IFO, making our measurement independent of the AOM response. This technique removes not only the (non-flat) AOM transfer function, but also any non-linearities from, e.g., overdriving the AOM. The below photo shows the new PD (center) on the AS table.

With the sentinel PD installed, we proceeded as follows.

  • Locked IFO in PRMI on carrier.
  • Locked AUX PLL to PSL.
  • Tuned the frequency of the AUX laser (via the RF offset) to bring the carrier onto resonance with the PRC.
  • Swept the AOM modulation frequency 0-60 MHz while measuring the AUX reflection and injection signals.

The below photo shows the measured transfer function [AUX Reflection / AUX Injection]. The measurement coherence is high to ~55 MHz (the AOM bandwidth is 60 MHz). We clearly resolve two FSRs, visible as Lorentzian dips at which more AUX power couples into the cavity. The SURFs have these data and will be separately posting figures for the measurements.

With the basic system working, we attempted to produce HOMs, first by partially occluding the injected AUX beam with a razor blade, then by placing a thin two-prong fork in the beam path. We also experimented with using a razor blade on the output to partially occlude the reflection beam just before the sensor. We were able to observe an apparent secondary dip indicative of an HOM a few times, as shown below, but could not repeat this deterministically. Besides not having fine control over the occlusion of the beams, there is also large few-Hz angular noise shaking the AS beam position. I suspect from moment to moment the HOM content is varying considerably due to the movement of the AS beam relative to the occluding object. I'm now thinking about more systematic ways to approach this.

 

  14020   Tue Jun 26 17:20:33 2018 JonConfigurationCamerasLLO Python Camera Software is Working

Thanks to a discussion yesterday with Joe Betzweiser, I was able to identify and fix the remaining problem with the LLO GigE camera software. It is working now, currently only on rossa, but can be set up on all the machines. I've started a wiki page with documentation and usage instructions here:

https://wiki-40m.ligo.caltech.edu/Electronics/GigE_Cameras

This page is also linked from the main 40m wiki page under "Electronics."

This software has the ability to both stream live camera feeds and to record feeds as .avi files. It is described more on the wiki page.

  14033   Fri Jun 29 18:16:32 2018 JonConfigurationPSLChanges to AUX Optical Layout on PSL Table

In order to use the 0th-order deflection beam from the AOM for cavity mode scans, I've coaligned this beam to the existing mode-matching/launch optics set up for the 1st-order beam.

Instead of being dumped, the 0th-order beam is now steered by two 45-degree mirrors into the existing beam path. The second mirror is on a flip mount so that we can quickly switch between 0th-order/1st-order injections. None of the existing optics were touched, so the 1st-order beam alignment should still be undisturbed.

Currently the 0th-order beam is being injected into the IFO. After attenuating so as to not exceed 100 mW incident on the fiber, approximately 50 mW of power reaches the AS table. That coupling efficiency is similar to what we have with the 1st-order beam. With the Y-arm cavity locked and the AUX PLL locked at RF offset = 47.60 MHz (an Y-arm FSR), I observed a -50 dBm beat note at Y-end transmission.

Attachment 1: PSL_AUX_SETUP_CHANGE.pdf
PSL_AUX_SETUP_CHANGE.pdf
  14035   Tue Jul 3 11:59:10 2018 JonUpdateAUXAUX Carrier Scan of Y-Arm Cavity

I made the first successful AUX laser scan of a 40m cavity last night.

Attachment #1 shows the measured Y-end transmission signal w.r.t. the Agilent drive signal, which was used to sweep the AUX carrier frequency. This is a distinct approach from before, where the carrier was locked at a fixed offset from the PSL carrier and the frequency of AM sidebands was swept instead. This AUX carrier-only technique appears to be advantageous.

This 6-15 MHz scan resolves three FSR peaks (TEM00 resonances) and at least six other higher-order modes. The raw data are also enclosed (attachment #2). I'll leave it as an excercise for the SURFs to compute the Y-arm cavity Gouy phase.

Attachment 1: yarm_carrier_trans.pdf
yarm_carrier_trans.pdf
Attachment 2: AG4395A_02-07-2018_185504.txt
# AG4395A Measurement - Timestamp: Jul 02 2018 - 18:55:04
#---------- Measurement Parameters ------------
# Start Frequency (Hz): 6000000.0, 6000000.0
# Stop Frequency (Hz): 15000000.0, 15000000.0
# Frequency Points: 801, 801
# Measurement Format: LOGM, PHAS
# Measuremed Input: AR, AR
#---------- Analyzer Settings ----------
# Number of Averages: 16
# Auto Bandwidth: Off, Off
... 807 more lines ...
  14036   Wed Jul 4 19:11:49 2018 JonUpdateAUXMore Testing of AUX-Laser Mode Scanning

More progress on the AUX-laser cavity scans.

Changes to the Setup

  • For scans, the Agilent is now being used as a standalone source of the LO signal provided to the AUX PLL (instead of the Marconi), which sets the RF offset. We discovered that when the sweep is "held" in network analyzer mode, it does not turn off the RF drive signal, but rather continues outputting a constant signal at the hold frequency. This eliminates the need to use the more complicated double-deomdulation previously in use. The procedure is to start and immediately hold the sweep, then lock the PLL, then restart the sweep. The PLL is able to reliably remain locked for frequency steps of up to ~30 kHz. The SURFs are preparing schematics of both the double- and single-demodulation techniques.
  • Both the Marconi and Agilent are now phase-locked to the 10 MHz time reference provided by the rabidium clock. This did noticeably shift the measured resonance frequencies.
  • I raised the PI controller gain setting to 4.5, which seems to better suppress the extra noise being injected.
  • I've procured a set of surgical needles for occluding the beam to produce HOMs. However, I have not needed to use them so far, as the TEM00 purity of the AUX beam appears to already be low. The below scans show only the intrinisic mode content.

New Results

  • YARM scan at 70 uW injection power (Attachment #1). The previously reported YARM scan was measured with 9 mW of injected AUX power, 100x larger than the power available from the SQZ laser at the sites. This scan repeats the measurement with the AUX power attenuated to uW. It still resolves the FSR and at least three HOMs.
  • PRC scan (Attachment #2) at 9 mW injection power. It appears to resolve the FSR and at least three HOMs. Angular injection noise was found to cause large fluctuations in the measured signal power. This dominates the error bars shown below, but affects only the overall signal amplitude (not the peak frequency locations). The SQZ angular alignment loops should mitigate this issue at the sites.

Both data sets are attached.

Attachment 1: yarm_trans_70uW.pdf
yarm_trans_70uW.pdf
Attachment 2: prc_trans_9mW.pdf
prc_trans_9mW.pdf
Attachment 3: yarm_carrier_trans_70uW.tar.gz
Attachment 4: prc_carrier_trans_9mW.tar.gz
  14044   Sun Jul 8 12:20:12 2018 JonSummaryAUXGouy Phase Measurements from AUX-Laser Scans

This note reports analysis of cavity scans made by directly sweeping the AUX laser carrier frequency (no sidebands). The measurement is made by sweeping the RF offset of the AUX-PSL phase-locked loop and demodulating the cavity reflection/transmission signal at the offset frequency.

Y-Arm Scan

Due to the simplicity of its expected response, the Y-arm cavity was scanned first as a test of the AUX hardware and the sensitivity of the technique. Attachment 1 shows the measured cavity transmission with respect to RF drive signal.

The AUX laser launch setup is capable of injecting up to 9.3 mW into the AS port. This high-power measurement is shown by the black trace. The same measurement is repeated for a realistic SQZ injection power, 70 uW, indicated by the red curve. At low power, the technique still clearly resolves the FSR and six HOM resonances. From the identified mode resonance frequencies the following cavity parameters are directly extracted.

YARM Gautam's Finesse Model Actual
FSR 3.966 MHz 3.967 MHz
Gouy phase 54.2 deg 52.0 deg

PRC Scan

An analogous scan was performed for the PRC, with the IFO locked on PSL carrier in PRMI. Attachment 2 shows the measurement of PRC transmission with respect to drive signal.

The scan resolves HOM resonances to at least ~13th order, whose frequencies yield the following cavity parameters.

PRC Gautam's Finesse Model Actual
FSR 22.30 MHz 22.20 MHz
Gouy phase 13.4 deg 15.4 deg

SRC Scan

Ideally (and at the sites) the SRC mode resonances will be measured in SRMI configuration. Because every other cavity is misaligned, this configuration provides an easily-interpretable spectrum whose resonances can all be attributed to the SRC.

Due to time constraints at the 40m, the IFO could not be restored to lockability in SRMI. It has been more than two years since this configuration was last run. For this reason the scan was made instead with the IFO locked in DRMI, as shown in Attachment 3. The quantity measured is the AUX reflection with respect to drive signal.

This result requires far more interpretation because resonances of both the SRC and PRC are superposed. However, the resonances of the PRC are known a priori from the independent PRMI scan. The SRC mode resonances identified below do not conincide with any of the first five PRC mode resonances.

Based on the identified mode resonance frequencies, the SRC parameters are measured as follows.

SRC Gautam's Finesse Model Actual
FSR 27.65 MHz 27.97 MHz
Gouy phase 10.9 deg 8.8 deg

Lessons Learned

From experience with the 40m, the main challenges to repeating this measurement at the sites will be the following.

  • Pointing jitter of the input AUX beam. This causes the PSL-AUX beam overlap to vary at transmission (or reflection), causing variation in the amplitude of the AUX-PSL beat note. As far as we can tell, the frequency of the resonances (the only object of this measurement) is not changing in time, only the relative amplitudes of the diferent mode peaks. I believe the SQZ alignment loops will mitigate this problem at the sites.
  • Stabilization of the network analyzer time base. We found the intrinsic frequency stability of the network analyzer (Agilent 4395A) to be unacceptably large. We solved this problem by phase-locking the Agilent to an external reference, a 10-MHz signal provided by an atomic clock.
Attachment 1: yarm_aux_carrier_trans.pdf
yarm_aux_carrier_trans.pdf
Attachment 2: prmi_aux_carrier_trans.pdf
prmi_aux_carrier_trans.pdf
Attachment 3: drmi_aux_carrier_trans.pdf
drmi_aux_carrier_trans.pdf
  14091   Fri Jul 20 18:30:47 2018 JonConfigurationAUXRecommend to install AUX PZT driver

I recently realized that the PLL is only using about 20% of the available actuation range of the AUX PZT. The +/-10 V control signal from the LB1005 is being directly inputted into the fast AUX PZT channel, which has an input range of +/-50 V.

I recommend to install a PZT driver (amplifier) between the controller and laser to use the full available actuator range. For cavity scans, this will increase the available sweep range from +/-50 MHz to +/-250MHz. This has a unique advantage even if slow temperature feedback is also implemented. To sample faster than the timescale of most of the angular noise,  scans generally need to be made with a total sweep time <1 sec. This is faster than the PLL offset can be offloaded via the slow temperature control, so the only way to scan more than 100 MHz in one measurement is with a larger dynamic range.

  14171   Mon Aug 20 15:16:39 2018 JonUpdateCDSRebooted c1lsc, slow machines

When I came in this morning no light was reaching the MC. One fast machine was dead, c1lsc, and a number of the slow machines: c1susaux, c1iool0, c1auxex, c1auxey, c1iscaux. Gautam walked me through reseting the slow machines manually and the fast machines via the reboot script. The computers are all back online and the MC is again able to lock.

  14179   Thu Aug 23 15:26:54 2018 JonUpdateIMCMC/PMC trouble

I tried unsuccessfully to relock the MC this afternoon.

I came in to find it in a trouble state with a huge amount of noise on C1:PSL-FSS_PCDRIVE visible on the projector monitor. Light was reaching the MC but it was unable to lock.

  • I checked the status of the fast machines on the CDS>FE STATUS page. All up.
  • Then I checked the slow machine status. c1iscaux and c1psl were both down. I manually reset both machines. The large noise visible on C1:PSL-FSS_PCDRIVE disappeared.
  • After the reset, light was no longer reaching the MC, which I take to mean the PMC was not locked. On the PSL>PMC page, I blanked the control signal, reenabled it, and attempted to relock by adjusting the servo gain as Gautam had showed me before. The PMC locks were unstable, with each one lasting only a second or so.
  • Next I tried restoring the burt states for c1iscaux and c1psl from a snapshot taken earlier today, before the machine reboots. That did not solve the problem either.
  14187   Tue Aug 28 18:39:41 2018 JonUpdateCDSC1LSC, C1AUX reboots

I found c1lsc unresponsive again today. Following the procedure in elog #13935, I ran the rebootC1LSC.sh script to perform a soft reboot of c1lsc and restart the epics processes on c1lsc, c1sus, and c1ioo. It worked. I also manually restarted one unresponsive slow machine, c1aux.

After the restarts, the CDS overview page shows the first three models on c1lsc are online (image attached). The above elog references c1oaf having to be restarted manually, so I attempted to do that. I connect via ssh to c1lsc and ran the script startc1oaf. This failed as well, however.

In this state I was able to lock the MICH configuration, which is sufficient for my purposes for now, but I was not able to lock either of the arm cavities. Are some of the still-dead models necessary to lock in resonant configurations?

Attachment 1: CDS_FE_STATUS.png
CDS_FE_STATUS.png
  14190   Wed Aug 29 11:46:27 2018 JonUpdateSUSlocal 4.4M earth quake

I freed ITMX and coarsely realigned the IFO using the OPLEVs. All the alignments were a bit off from overnight.

The IFO is still only able to lock in MICH mode currently, which was the situation before the earthquake. This morning I additionally tried restoring the burt state of the four machines that had been rebooted in the last week (c1iscaux, c1aux, c1psl, c1lsc) but that did not solve it.

Quote:

All suspension tripped. Their damping restored. The MC is locked.

ITMX-UL & side magnets are stuck.

 

 

  14196   Mon Sep 10 12:44:48 2018 JonUpdateCDSADC replacement in c1lsc expansion chassis

Gautam and I restarted the models on c1lsc, c1ioo, and c1sus. The LSC system is functioning again. We found that only restarting c1lsc as Rolf had recommended did actually kill the models running on the other two machines. We simply reverted the rebootC1LSC.sh script to its previous form, since that does work. I'll keep using that as required until the ongoing investigations find the source of the problem.

Quote:

Looks like the ADC was not to blame, same symptoms persist.

Quote:

The PCIe fiber replacement is a more involved project (Steve is acquiring some protective tubing to route it from the FE in 1X6 to the expansion chassis in 1Y3), but hopefully the problem was the ADC card with red indicator light, and replacing it has solved the issue.

 

  14278   Tue Nov 6 19:41:46 2018 JonOmnistructure c1vac1/2 replacement

This afternoon I started setting up the Supermicro 5017A-EP that will replace c1vac1/2. Following Johannes's procedure in 13681 I installed Debian 8.11 (jessie). There is a more recent stable release, 9.5, now available since the first acromag machine was assembled, but I stuck to version 8 for consistency. We already know that version to work. The setup is sitting on the left side of the electronics bench for now.

  14282   Wed Nov 7 19:17:18 2018 JonOmnistructure modbusIOC is running on c1vac replacement

Today I finished setting up the server that will replace the c1vac1/2 machines. I put it on the martian network at the unassigned IP 192.168.113.72. I assigned it the hostname c1vac and added it to the DNS lookup tables on chiara.

I created a new targets directory on the network drive for the new machine: /cvs/cds/caltech/target/c1vac. After setting EPICS environment environment variables according to 13681 and copying over (and modifiying) the files from /cvs/cds/caltech/target/c1auxex as templates, I was able to start a modbusIOC server on the new machine. I was able to read and write (soft) channel values to the EPICS IOC from other machines on the martian network.

I scripted it as a systemd-managed process which automatically starts on boot and restarts after failure, just as it is set up on c1auxex.

  14287   Fri Nov 9 22:24:22 2018 JonOmnistructure Wiring of Vacuum Acromag Chassis Complete

Wiring of the power, Ethernet, and indicator lights for the vacuum Acromag chassis is complete. Even though this crate will only use +24V DC, I wired the +/-15V connector and indicator lights as well to conform to the LIGO standard. There was no wiring diagram available, so I had to reverse-engineer the wiring from the partially complete c1susaux crate. Attached is a diagram for future use. The crate is ready to begin software developing on Monday. 

Attachment 1: IMG_2987.jpg
IMG_2987.jpg
Attachment 2: IMG_2985.jpg
IMG_2985.jpg
Attachment 3: IMG_2986.jpg
IMG_2986.jpg
  14296   Wed Nov 14 21:34:44 2018 JonOmnistructure Vacuum Acromags installed and tested

All 7 Acromag units are now installed in the vacuum chassis. They are connected to 24V DC power and Ethernet.

I have merged and migrated the two EPICS databases from c1vac1 and c1vac2 onto the new machine, with appropriate modifications to address the Acromags rather than VME crate.

I have tested all the digital output channels with a voltmeter, and some of the inputs. Still more channels to be tested.

I’ll follow up with a wiring diagram for channel assignments.

Attachment 1: IMG_3003.jpg
IMG_3003.jpg
  14308   Mon Nov 19 22:45:23 2018 JonOmnistructure Vacuum System Subnetwork

I've set up a closed subnetwork for interfacing the vacuum hardware (Acromags and serial devices) with the new controls machine (c1vac; 192.168.113.72). The controls machine has two Ethernet interfaces, one which faces outward into the martian network and another which faces the internal subnetwork, 192.168.114.xxx. The second network interface was configured via the following procedure.

1. Add the following lines to /etc/network/interfaces:

allow-hotplug eth1
iface eth1 inet static
address 192.168.114.9
netmask 255.255.255.0

2. Restart the networking services:

$sudo /etc/init.d/networking restart

3. Enable DNS lookup on the martian network by adding the following lines to /etc/resolv.conf:

search martian
nameserver 192.168.113.104

4. Enable IP forwarding from eth1 to eth0:

$sudo echo 1 > /proc/sys/net/ipv4/ip_forward

5. Configure IP tables to allow outgoing connections, while keeping the LAN invisible from outside the gateway (c1vac):

$sudo iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE
$sudo iptables -A FORWARD -i eth0 -o eth1 -m state --state RELATED,ESTABLISHED -j ACCEPT
$sudo iptables -A FORWARD -i eth1 -o eth0 -j ACCEPT

6. Finally, because the EPICS 3.14 server binds to all network interfaces, client applications running on c1vac now see two instances of the EPICS server---one at the outward-facing address and one at the LAN address. To resolve this ambiguity, two additional enviroment variables must be set that specify to local clients which server address to use. Add the following lines to /home/controls/.bashrc:

EPICS_CA_AUTO_ADDR_LIST=NO
EPICS_CA_ADDR_LIST=192.168.113.72

A list of IP addresses so far assigned on the subnetwork follows.

Device IP Address
Acromag XT1111a 192.168.114.1
Acromag XT1111b 192.168.114.2
Acromag XT1111c 192.168.114.3
Acromag XT1111d 192.168.114.4
Acromag XT1111e 192.168.114.5
Acromag XT1121a 192.168.114.6
Acromag XT1121b 192.168.114.7
Perle IOLAN SDS16 192.168.114.8
c1vac 192.168.114.9
  14309   Mon Nov 19 23:38:41 2018 JonOmnistructure Vacuum Acromag Channel Assignments

I've completed bench testing of all seven vacuum Acromags installed in a custom rackmount chassis. The system contains five XT1111 modules (sinking digital I/O) used for readbacks of the state of the valves, TP1, CP1, and the RPs. It also contains two XT1121 modules (sourcing digital I/O) used to pass 24V DC control signals to the AC relays actuating the valves and RPs. The list of Acromag channel assignments is attached.

I tested each input channel using a manual flip-switch wired between signal pin and return, verifying the EPICS channel readout to change appropriately when the switch is flipped open vs. closed. I tested each output channel using a voltmeter placed between signal pin and return, toggling the EPICS channel on/off state and verifying the output voltage to change appropriately. These tests confirm the Acromag units all work, and that all the EPICS channels are correctly addressed.

Attachment 1: Binary_IO_Channel_Assignments.pdf
Binary_IO_Channel_Assignments.pdf
  14315   Sun Nov 25 17:41:43 2018 JonOmnistructure Vacuum Controls Upgrade - Status and Plans

New hardware has been installed in the vacuum controls rack. It is shown in the below post-install photo.

  • Supermicro server (c1vac) which will be replacing c1vac1 and c1vac2.
  • 16-port Ethernet switch providing a closed local network for all vacuum devices.
  • 16-port IOLAN terminal server for multiplexing/Ethernetizing all RS-232 serial devices.

Below is a high-level summary of where things stand, and what remains to be done.

Completed:

 Set up of replacement controls server (c1vac).

  • Supermicro 1U rackmount server, running Debian 8.5.
  • Hosting an EPICS modbus IOC, scripted to start/restart automatically as a system service.
  • First Ethernet interface put on the martian network at 192.168.113.72.
  • Second Ethernet interface configured to host a LAN at 192.168.114.xxx for communications with all vacuum electronics. It connects to a 16-port Ethernet switch installed in the vacuum electronics rack.
  • Server installed in vacuum electronics rack (see photo).

 Set up of Acromag terminals.

  • 6U rackmount chassis frame assembled; 15V DC, 24V DC, and Ethernet wired.
  • Acromags installed in chassis and configured for the LAN (5 XT1111 units, 2 XT1121 units).

 EPICS database migration.

  • All vacuum channels moved to the modbus IOC, with the database updated to address the new Acromags. [The new channels are running concurrently at "C1:Vac2-...." to avoid conflict with the existing system.]
  • Each hard channel was individually tested on the electronics bench to confirm correct addressing and Acromag operation.

 Set up of 16-port IOLAN terminal server (for multiplexing/Ethernetizing the serial devices).

  • Configured for operation on the LAN. Each serial device port is assigned a unique IP address, making the terminal server transparent to client TCP applications.
  • Most of the pressure gauges are now communicating with the controls server via TCP.

Ongoing this week:

  • [Jon] Continue migrating serial devices to ports on the terminal server. Still left are the turbo pumps, N2 gauge, and RGA.
  • [Jon] Continue developing Python code for communicating with gauges and pumps via TCP sockets. A beta version of gauge readout code is running now.
  • [Chub] Install feedthrough panels on the Acromag chassis. Connect the wiring from feedthrough panels to the assigned Acromag slots.
  • [Chub/Jon] Test all the hard EPICS channels on the electronics bench, prior to installing the crate in the vacuum rack.
  • [Chub/Jon] Install the crate in the vacuum rack; connect valve/pump readbacks and actuators; test each hard EPICS channel in situ.
  • [Jon] Once all the signal connections have been made, in situ testing of the Python interlock code can begin.
Attachment 1: rack_photo.jpg
rack_photo.jpg
  14320   Mon Nov 26 21:58:08 2018 JonOmnistructure Serial Vacuum Signals

All the serial vacuum signals are now interfaced to the new digital controls system. A set of persistent Python scripts will query each device at regular intervals (up to ~10 Hz) and push the readings to soft channels hosted by the modbus IOC. Similar scripts will push on/off state commands to the serial turbo pumps.

IP Addresses/Comm Settings

Each serial device is assigned an IP address on the local subnet as follows. Its serial communication parameters as configured in the terminal server are also listed.

Device IP Address Baud Rate Data Bits Stop Bits Parity
MKS937a vacuum gauge controller 192.168.114.11 9600 8 1 even
MKS937b vacuum gauge controller 192.168.114.12 9600 8 1 even
GP307 vacuum gauge controller 192.168.114.13 9600 8 1 even
GP316a vacuum gauge controller 192.168.114.14 9600 8 1 even
GP316b vacuum gauge controller 192.168.114.15 9600 8 1 even
N2 pressure line gauge 192.168.114.16 9600 7 1 odd
TP2/3 192.168.114.17/18 9600 8 1 none

Hardware Modifications

  • Each of the five vacuum gauge controllers has an RJ45 adapter installed directly on its DB9/DB25 output port. Because the RJ45 cable now plugs directly into the terminal server, instead of passing through some additional adapters as it formerly did, it was necessary to reverse the wiring of the controller TXD and RXD pins to Ethernet pins. The DB9/25-to-RJ45 adapters on the back of the controllers are now wired as follows.
    • For the MKS controllers: DB2 (RXD) --> Eth4;  DB3 (TXD) --> Eth5;  DB5 (RTN) --> Eth6
    • For the Granville-Phillips controllers: DB2 (TXD) --> Eth5;  DB3 (RXD) --> Eth4;  DB7 (RTN) --> Eth6
  • I traced a communications error with the GP307 gauge controller all the way back to what I would have suspected least, the controller itself. The comm card inside each controller has a set of mechanical relay switches which set the communications parameters (baud rate, parity, etc.). Knowing that this controller was not part of the original installation, but was swapped in to replace the original in 2009, I pulled the controller from the rack and checked the internal switch settings. Sure enough, the switch settings (pictured below) were wrong. In the background of the photo is the unit removed in 2009, which has the correct settings. After setting the correct communications parameters, the controller immediately began communicating with the server. Did these readouts (PRP, PTP1) never work since 2009? I don't see how they could.
Attachment 1: GP307_relays.jpeg
GP307_relays.jpeg
  14325   Fri Nov 30 15:53:52 2018 JonOmnistructureGeneralN2 pressure gauge fix

I've made a repair to the N2 pressure monitor. I don't believe the polarity of the analog signal into the controller actually was reversed. I found the data sheet (attached) for the transducer model we have installed. Its voltage should read ~0 at 0 PSI and 100mV at 100 PSI. As wired, the input voltage reads +80 mV as it should.

The controller calibrates the sensor voltage to PSI (i.e., applies a scale and offset) based on two settable reference points which appeared to be incorrect. I changed them to:

  1. 0 mV = 0 PSI (neglecting the small dark bias)
  2. 100 mV = 100 PSI

After the change, the pressure reads 80 PSI. Let's see if the time history now shows a sensible trend. 

Quote:

[koji, gautam, jon, steve]

  • We suspect analog voltage from N2 pressure gauge is connected to interfacing Omega controller with the 'wrong' polarity (i.e pressure is rising over ~4 days and then rapidly falling instead of the other way around). This should be fixed.

 

Attachment 1: N2_transducer_datasheet.pdf
N2_transducer_datasheet.pdf
  14327   Sun Dec 2 16:08:44 2018 JonOmnistructureUpgradeFeedthroughs for Vacuum Acromag Chassis

Below is an inventory of the signal feedthroughs that need to be installed on the vacuum Acromag crate this week.

Type Qty Connects to # Chs Signals
DB-37 female 1 Main AC relay box 18 Valve/roughing pump control
DB-9 female 5** Satellite AC relay boxes 3-4/box Valve control
DB-25 male 1 Turbo pump 1 controller 5 Pump status readbacks
DB-9 male 30 Valve position indicators 2/valve Valve position readbacks
DB-9 male 3 Roughing pump controllers 1/pump Pump status readbacks
DB-9 male 1 Cryo pump controller 2 Pump status readbacks

**The original documentation lists five satellite boxes (one for each test mass chamber and one for the beamsplitter chamber), but Chub reports not all of them are in use. We may remove the ones not used.

  14330   Tue Dec 4 10:38:12 2018 JonOmnistructureUpgradeUpdated Feedthrough List for Vacuum Acromag Chassis

Based on new input from Chub, attached is the revised list of signal cable feedthroughs needed on the vacuum system Acromag crate. I believe this list is now complete.

Attachment 1: acromag_chassis_feedthroughs.pdf
acromag_chassis_feedthroughs.pdf
  14333   Thu Dec 6 17:33:33 2018 JonOmnistructureGeneralN2 line disconnected

I believe I finally have the N2 gauge working correctly. The wiring is unchanged from its original state and the controller has been recalibrated.

After letting the line pressure drop to 0 PSI as indicated by the analog gauge in the drill-press room, I recorded the number of counts read by the Omega controller. Then I pressurized the line to 80 PSI, again indicated by the analog gauge, and recorded the Omega counts again. I entered these two reference points into the controller (automatically determines the gain and offset from these), then confirmed the readings to agree with the anaog gauge as I varied the line pressure.

The two reference points are:

0 PSI  :  13 counts
80 PSI : 972 counts

 

Quote:

[jon, gautam]

In the latest installment in this puzzler: turns out that maybe the trend of the "N2 pressure" channel increasing over the ~3 day timescale it takes a cylinder of N2 to run out is real, and is a feature of the way our two N2 cylinder lines/regulators are setup (for the automatic switching between cylinders when one runs out). In order to test this hypothesis, we'd like to have the line pressure be 0 initially, and then just have 1 cylinder hooked up. 

 

  14348   Wed Dec 12 18:27:07 2018 JonOmnistructureUpgradeAnalog signals, A/D Acromag added to vacuum system

There turned out to be a few analog signals for the vacuum system after all. The TP2/3 foreline pressure gauges were never part of the digital system, but we wanted to add them, as some of the interlock conditions should be predicated on their readings. Each gauge connects to an old Granville-Phillips 375 controller which only has an analog output. Interfacing these signals with the new system required installing an Acromag XT1221 8-channel A/D unit. Taking advantage of the extra channels, I also moved the N2 delivery line pressure transducer to the XT1221, eliminating the need for its separate Omega DPiS32 controller. When the new high-pressure transducers are added to the two N2 tanks, their signals can also be connected.

The XT1221 is mounted on the DIN rail inside the chassis and I have wired a DB-9 feedthrough for each of its three input signals. It is assigned the IP 192.168.114.27 on the vacuum subnet. Testing the channels in situ revealed a subtley in calibrating them to physical units. It was first encountered by Johannes in a series of older posts, but I repeat it here in one place.

An analog-input EPICS channel can be calibrated from raw ADC counts to physical units (e.g., sensor voltage) in two ways:

  1. Via LINR="LINEAR" by setting the engineering-units fields EGUF="[V_max_adc]", EGUL="[V_min_adc]"
  2. Via LINR="NO CONVERSION" by manually setting the gain ASLO="[V/count]" and offset AOFF="[V_offset]"

From the documentation, under the engineering-units method EPICS internally computes:

where EGUF="eng units full scale", EGUL="eng units low", and "full scale A/D counts" is the full range of ADC counts. EPICS automatically infers the range of ADC counts based on the data type returned by the ADC. For a 16-bit ADC like the XT1221, this number is 2^16 = 65,536.

The problem is that, for unknown reasons, the XT1221 rescales its values post-digitization to lie within the range +/-30,000 counts. This corresponds to an actual "full scale A/D counts" = 60,001. If a multiplicative correction factor of 65,536/60,000 is absorbed into the values of EGUF and EGUL, then the first term in the above summation can be corrected. However, the second term (the offset) has no dependence on "full scale A/D counts" and should NOT absorb a correction factor. Thus adjusting the EGUF and EGUL values from, e.g., 10V to 10.92V is only correct when EGUL=0V. Otherwise there is a bias introduced from the offset term also being rescaled.

The generally correct way to handle this correction is to use the manual "NO CONVERSION" method. It constructs calibrated values by simply applying a specified gain and offset to the raw ADC counts:

calibrated val = (measured A/D counts)  x ASLO + AOFF

The gain ASLO="[(V_max_adc - V_min_adc) / 60,001]" and the offset AOFF="0". I have tested this on the three vacuum channels and confirmed it works. Note that if the XT1221 input voltage range is restricted from its widest +/-10V setting, the number of counts is not necessarily 60,001. Page 42 of the manual gives the correct counts for each voltage setting.

  14372   Thu Dec 20 08:38:27 2018 JonUpdateVACPumpdown tomorrow

Linked is the pumpdown procedure, contained in the old 40m documentation. The relevant procedure is "All Off --> Vacuum Normal" on page 11.

Quote:

I just spoke to Jon who asked me to make this elog - we will be ready to test one or more parts of the pumpdown procedure tomorrow (12/20), so we should proceed as planned to put the heavy doors back on EY and OMC chambers at 9am tomorrow morning. Jon will circulate a more detailed procedure about the pumpdown steps later today evening.

 

  14375   Thu Dec 20 21:29:41 2018 JonOmnistructureUpgradeVacuum Controls Switchover Completed

[Jon, Chub, Koji, Gautam]

Summary

Today we carried out the first pumpdown with the new vacuum controls system in place. It performed well. The only problem encountered was with software interlocks spuriously closing valves as the Pirani gauges crossed 1E-4 torr. At that point their readback changes from a number to "L OE-04, " which the system interpreted as a gauge failure instead of "<1E-4." This posed no danger and was fixed on the spot. The main volume was pumped to ~10 torr using roughing pumps 1 and 3. We were limited only by time, as we didn't get started pumping the main volume until after 1pm. The three turbo pumps were also run and tested in parallel, but were isolated to the pumpspool volume. At the end of the day, we closed every pneumatic valve and shut down all five pumps. The main volume is sealed off at ~10 torr, and the pumpspool volume is at ~1e-6 torr. We are leaving the system parked in this state for the holidays. 

Main Volume Pumpdown Procedure

In pumping down the main volume, we carried out the following procedure.

  1. Initially: All valves closed (including manual valves RV1 and VV1); all pumps OFF.
  2. Manually connected roughing pump line to pumpspool via KF joint.
  3. Turned ON RP1 and RP2.
  4. Waited until roughing pump line pressure (PRP) < 0.5 torr.
  5. Opened V3.
  6. Waited until roughing pump line pressure (PRP) < 0.5 torr.
  7. Manually opened RV1 throttling valve to main volume until pumpdown rate reached ~3 torr/min (~3 hours on roughing pumps).
  8. Waited until main volume pressure (P1a/P1b) < 0.5 torr.

We didn't quite reach the end of step 8 by the time we had to stop. The next step would be to valve out the roughing pumps and to valve in the turbo pumps.

Hardware & Channel Assignments

All of the new hardware is now permanently installed in the vacuum rack. This includes the SuperMicro rack server (c1vac), the IOLAN serial device server, a vacuum subnet switch, and the Acromag chassis. Every valve/pump signal cable that formerly connected to the VME bus through terminal blocks has been refitted with a D-sub connector and screwed directly onto feedthroughs on the Acromag chassis.

The attached pdf contains the master list of assigned Acromag channels and their wiring.

Attachment 1: 40m_vacuum_acromag_channels.pdf
40m_vacuum_acromag_channels.pdf 40m_vacuum_acromag_channels.pdf 40m_vacuum_acromag_channels.pdf
  14383   Fri Jan 4 10:25:19 2019 JonOmnistructureVACN2 line valved off

Yes, for TP2 and TP3. They both have a small vent valve that opens automatically on shutdown.

Quote:

Independent question: Are all the turbo forelines vented automatically? We manually did it for the main roughing line.

 

 

  14384   Fri Jan 4 11:06:16 2019 JonOmnistructureUpgradeVac System Punchlist

The base Acromag vacuum system is running and performing nicely. Here is a list of remaining questions and to-do items we still need to address.

Safety Issues

  • Interlock for HV supplies. The vac system hosts a binary EPICS channel that is the interlock signal for the in-vacuum HV supplies. The channel value is OFF when the main volume pressure is in the arcing range, 3 mtorr - 500 torr, and ON otherwise. Is there something outside the vacuum system monitoring this channel and toggling the HV supplies?
  • Exposed 30-amp supply terminals. The 30-amp output terminals on the back of the Sorensen in the vac rack are exposed. We need a cover for those.
  • Interlock for AC power loss. The current vac system is protected only from transient power glitches, not an extended loss. The digital system should sense an outage and put the IFO into a safe state (pumps spun down and critical valves closed) before the UPS battery is fully drained. However, it presently has no way of sensing when power has been lost---the system just continues running normally on UPS power until the battery dies, at which point there is a sudden, uncontrolled shutdown. Is it possible for the digital system to communicate directly with the UPS to poll its activation state?

Infrastructure Improvements

  • Install the new N2 tank regulator and high-pressure transducers (we have the parts; on desk across from electronics bench). Run the transducer signal wires to the Acromag chassis in the vacuum rack.
  • Replace the kludged connectors to the Hornet and SuperBee serial outputs with permanent ones (we need to order the parts).
  • Wire the position indicator readback on the manual TP1 valve to the Acromag chassis.
  • Add cable tension relief to the back of the vac rack.
  • Add the TP1 analog readback signals (rotation speed and current) to the digital system.  Digital temperature, current, voltage, and rotation speed signals have already been added for TP2 and TP3.
  • Set up a local vacuum controls terminal on the desk by the vac rack.
  • Remove gauges from the EPICS database/MEDM screens that are no longer installed or functional. Potential candidates for removal: PAN, PTP1, IG1, CC2, CC3, CC4.
  • Although it appeared on the MEDM screen, the RGA was never interfaced to the old vac system. Should it be connected to c1vac now?
  14387   Mon Jan 7 11:54:12 2019 JonConfigurationComputer Scripts / ProgramsVac system shutdown

I'm making a controlled shutdown of the vac controls to add new ADC channels. Will advise when it's back up.

  14388   Mon Jan 7 19:21:45 2019 JonConfigurationComputer Scripts / ProgramsVac system shutdown

ADC work finished for the day. The vac controls are back up, with all valves CLOSED and all pumps OFF.

Quote:

I'm making a controlled shutdown of the vac controls to add new ADC channels. Will advise when it's back up.

 

  14390   Tue Jan 8 19:13:39 2019 JonUpdateUpgradeReady for pumpdown tomorrow

Everything is set for a second pumpdown tomorrow. We'll plan to start pumping after the 1pm meeting. Since the main volume is already at 12 torr, the roughing phase won't take nearly as long this time.

I've added new channels for the TP1 analog readings (current and speed) and for the two N2 tank pressure readings. Chub finished installing the new regulator and has run the transducer signal cable to the vacuum rack. In the morning he will terminate the cable and make the final connection to the Acromag.

Gautam and I updated the framebuilder config file, adding the newly-added channels to the list of those to be logged. We also set up a git repo containing all of the Python interlock/interfacing code: https://git.ligo.org/40m/vacpython. The idea is to use the issue tracker to systematically document any changes to the interlock code.

  14393   Wed Jan 9 20:01:25 2019 JonUpdateVACSecond pumpdown completed

[Jon, Koji, Chub, Gautam]

Summary

The second pumpdown with the new vacuum system was completed successfully today. A time history is attached below.

We started with the main volume still at 12 torr from the Dec. pumpdown. Roughing from 12 to 0.5 torr took approximately two hours, at which point we valved out RP1 and RP3 and valved in TP1 backed by TP2 and TP3. We additionally used the AUX dry pump connected to the backing lines of TP2 and TP3, which we found to boost the overall pump rate by a factor of ~3. The manual hand-crank valve directly in front of TP1 was used to throttle the pump rate, to avoid tripping software interlocks. If the crank valve is opened too quickly, the pressure differential between the main volume (TP1 intake) and TP1 exhaust becomes >1 torr, tripping the V1 valve-close interlock. Once the main volume pressure reached 1e-2 torr, the crank valve could be opened fully.

We allowed the pumpdown to continue until reaching 9e-4 torr in the main volume. At this point we valved off the main volume, valved off TP2 and TP3, and then shut down all turbo pumps/dry pumps. We will continue pumping tomorrow under the supervision of an operator. If the system continues to perform problem-free, we will likely leave the turbos pumping on the main volume and annuli after tomorrow.

New Vac Control Station

We installed a local controls terminal for the vacuum system on the desk in front of the vacuum rack (pictured below). This console is connected directly to c1vac and can be used to monitor/control the system even during a network outage or power failure. The entire pumpdown was run from this station today.

To open a controls MEDM screen, open a terminal and execute the alias

$control

Similarly, to open a monitor-only MEDM screen, execute the alias

$monitor
Attachment 1: Screenshot_from_2019-01-09_20-00-39.png
Screenshot_from_2019-01-09_20-00-39.png
Attachment 2: IMG_3088.jpg
IMG_3088.jpg
  14396   Thu Jan 10 19:59:08 2019 JonUpdateVACVac System Running Normally on Turbo Pumps

[Jon, Gautam, Chub]

Summary

We continued the pumpdown of the IFO today. The main volume pressure has reached 1.9e-5 torr and is continuing to fall. The system has performed without issue all day, so we'll leave the turbos continuously running from here on in the normal pumping configuration. Both TP2 and TP3 are currently backing for TP1. Once the main volume reaches operating pressure, we can transition TP3 to pump the annuli. They have already been roughed to ~0.1 torr. At that point the speed of all three turbo pumps can also be reduced. I've finished final edits/cleanup of the interlock code and MEDM screens.

Python Code

All the python code running on c1vac is archived to the git repo: 

https://git.ligo.org/40m/vacpython

This includes both the interlock code and the serial device clients for interfacing with gauges and pumps.

MEDM Monitor/Control

We're still using the same base MEDM monitor/control screens, but they have been much improved. Improvements:

  • Valves now light up in red when they are open. This makes it much easier to see at a glance what is valved in/out.
  • Every pump in the system (except CP1) is now digitally controlled from the MEDM control screen. No more need to physically push any buttons in the vaccum rack. 👍
  • The turbo pumps now show additional diagnostic readouts: speed (TP1/2/3), temperature (TP2/3), current draw (TP1/2/3), and voltage (TP2/3).
  • The foreline pressure gauge readouts for TP2/3 have been added to the digital system.
  • The two new main volume gauges, Hornet and SuperBee, have been added to the digital system as well.
  • New transducers have been added to read back the two N2 tank pressures.
  • The interlock code generates a log file of all its actions. A field in the MEDM screens specifies the location of the log file.
  • A tripped interlock (appearing as a message in the "Error message" field) must be manually cleared via the "Clear error message" button on the control screen before the system will accept any more manual valve input.

Note: The apparent glitches in the pressure and TP diagnostic channels are due to the interlock system being taken down to implement some of these changes.

Attachment 1: Screen_Shot_2019-01-10_at_7.58.24_PM.png
Screen_Shot_2019-01-10_at_7.58.24_PM.png
Attachment 2: CCs.png
CCs.png
Attachment 3: TPs.png
TPs.png
  14410   Sun Jan 20 23:41:00 2019 JonOmnistructureVACNotes on vac serial comm, adapter wiring

I've attached my handwritten notes covering all the serial communications in the vac system, and the relevant wiring for all the adapters, etc. I'll work with Chub to produce a final documentation, but in the meantime this may be a useful reference.

Attachment 1: Jon_wiring_notes.tar.gz
  14453   Thu Feb 14 18:16:24 2019 JonUpdateVACVacromag failure

I sent Gautam instructions to first try stopping the modbus service, power cycling the Acromag chassis, then restarting the service. I've seen the Acromags go into an unresponsive state after a strong electrical transient or shorted signal wires, and the unit has to be power cycled to be reset.

If this doesn't resolve it, I'll come in tomorrow to help with the Acromag replacement. We have plenty of spares.

Quote:

[chub, gautam]

Sumary:

One of the XT1111 units (XT1111a) in the new vacuum system has malfunctioned. So all valves are closed, PSL shutter is also closed, until this is resolved.

 

  14456   Fri Feb 15 11:58:45 2019 JonUpdateVACVac system is back up

The problem encountered with the vac controls was indeed resolved via the recommendation I posted yesterday. The Acromags had gone into a protective state (likely caused by an electrical transient in one of the signals) that could only be cleared by power cycling the units. After resetting the system, the main volume pressure dropped quickly and is now < 2e-5 torr, so normal operations can resume. For future reference, below is the procedure to safely reset these units from a trouble state.

Vacromag Reset Procedure

  • TP2 and TP3 can be left running, but isolate them by closing valves V4 and V5.
  • TP1 can also be left running, but manually flip the operation mode on the front of the controller from REMOTE to LOCAL. This prevents the pump from receiving a "stop" command when its control Acromag shuts down.
  • Close all the pneumatic valves in the system (they'll otherwise close automatically when their control Acromags shut down).
  • On c1vac, stop the modbusIOC service. Sometimes this takes ~1 min to actually terminate.
  • Turn off the Acromags by flipping the "24 V" on the back of the chassis.
  • Wait ~10 sec, then turn them back on.
  • Start the modbusIOC service. It may take up to ~1 min for all the readings on the MEDM screen to initialize.
  • Ensure that the rotation speed of TP1,2,3 are still all nominal.
  • If pumps are OK, open V4, V5, and V7, then open V1. This restores the system to the "Maximum pumping speed" state.
  • Flip the TP1 controller operation state back to REMOTE.
  14461   Fri Feb 15 20:07:02 2019 JonUpdateVACUpdated vacuum punch list

While working on the vac controls today, I also took care of some of the remaining to-do items. Below is a summary of what was done, and what still remains.

Completed today

  • TP2/3 overcurrent interlock raised from 1 to 1.2 A. This was tripping during normal operation as the pump accelerates from low-speed (standby) to normal-speed mode.
  • Interlock conditions on VABSSCO/VABSSCI removed. Per discussion with Steve, these are not vent valves, but rather isolation valves between the BS/IOO/OMC annuli. The interlocks were preventing the valves from opening, and hence the IOO and OMC annuli from being pumped.
  • Channel exposed for interlocking in-vacuum high-voltage drivers. The channel name is C1:Vac-interlock_high_voltage. The vac interlock service sets this channel's value to 0 when the main volume pressure is in the range 3 mtorr-500 torr, and to 1 otherwise.
  • Annuli pumping integrated into the set of recognized states. "Vacuum normal" now refers to TP1 and TP2 pumping on the main volume AND TP3 pumping on all the annuli. The system is currently running in this state.
  • TP1 lowered to the nominal speed setting recommended by Steve: 33.6 krpm (560 Hz).

Still remaining

  • Implement a "blinker" input-output signal loop between two Acromags to detect hardware failures like the one today.
  • Add an AC power monitor to sense extended power losses and automatically put the system into safe shutdown.
  • Migrate the RGA to c1vac. Still some issues getting the serial comm working.
  • Troubleshoot the SuperBee (backup) main volume Parani gauge. It has not communicated with c1vac since a serial adapter was replaced two weeks ago. Chub thinks the gauge was possibly damaged by arcing during the replacement.
  • Scripting for more automated pumpdowns.
  • Generate a bootable backup hard drive for c1vac, which could be swapped in on a short time scale after a failure.
  14487   Wed Mar 20 12:31:30 2019 JonUpdateVACDoing vac controls work

I'm rebooting the IOLAN server to load new serial ports. The interlocks might trip when the pressure gauge readbacks cut out.

  14488   Wed Mar 20 19:26:25 2019 JonUpdateVACProtection against AC power loss

Today I implemented protection of the vac system against extended power losses. Previously, the vac controls system (both old and new) could not communicate with the APC Smart-UPS 2200 providing backup power. This was not an issue for short glitches, but for extended outages the system had no way of knowing it was running on dwindling reserve power. An intelligent system should sense the outage and put the IFO into a controlled shutdown, before the batteries are fully drained.

What enabled this was a workaround Gautam and I found for communicating with the UPS serially. Although the UPS has a serial port, neither the connector pinout nor the low-level command protocol are released by APC. The only official way to communicate with the UPS is through their high-level PowerChute software. However, we did find "unofficial" documentation of APC's protocol. Using this information, I was able to interface the the UPS to the IOLAN serial device server. This allowed the UPS status to be queried using the same Python/TCP sockets model as all the other serial devices (gauges, pumps, etc.). I created a new service called "serial_UPS.service" to persistently run this Python process like the others. I added a new EPICS channel "C1:Vac-UPS_status" which is updated by this process.

With all this in place, I added new logic to the interlock.py code which closes all valves and stops all pumps in the event of a power failure. To be conservative, this interlock is also tripped when the communications link with the UPS is disconnected (i.e., when the power state becomes unknown). I tested the new conditions against both communication failure (by disconnecting the serial cable) and power failure (by pressing the "Test" button on the UPS front panel). This protects TP2 and TP3. However, I discovered that TP1---the pump that might be most damaged by a sudden power failure---is not on the UPS. It's plugged directly into a 240V outlet along the wall. This is because the current UPS doesn't have any 240V sockets. I'd recommend we get one that can handle all the turbo pumps.

For future reference:

Pin 1: RxD

Pin 2: TxD

Pin 5: GND

Standard: RS-232

Baud rate: 2400

Data bits: 8

Parity: none

Stop bits: 1

Handshaking: none

 

 

Attachment 1: IMG_3146.jpg
IMG_3146.jpg
  14489   Wed Mar 20 20:07:22 2019 JonUpdateVACDoing vac controls work

Work is completed and the vac system is back in its nominal state.

Quote:

I'm rebooting the IOLAN server to load new serial ports. The interlocks might trip when the pressure gauge readbacks cut out.

 

  14490   Thu Mar 21 12:46:22 2019 JonUpdateVACMore vac controls upgrades

The vac controls system is going down for migration from Python 2.7 to 3.4. Will advise when it is back up.

  14491   Thu Mar 21 17:22:52 2019 JonUpdateVACMore vac controls upgrades

I've converted all the vac control system code to run on Python 3.4, the latest version available through the Debian package manager. Note that these codes now REQUIRE Python 3.x. We decided there was no need to preserve Python 2.x compatibility. I'm leaving the vac system returned to its nominal state ("vacuum normal + RGA").

Quote:

The vac controls system is going down for migration from Python 2.7 to 3.4. Will advise when it is back up.

 

  14493   Thu Mar 21 18:36:59 2019 JonOmnistructureUpgradeVacuum Controls Switchover Completed

Updated vac channel list is attached. There are several new ADC channels.

Quote:

Hardware & Channel Assignments

All of the new hardware is now permanently installed in the vacuum rack. This includes the SuperMicro rack server (c1vac), the IOLAN serial device server, a vacuum subnet switch, and the Acromag chassis. Every valve/pump signal cable that formerly connected to the VME bus through terminal blocks has been refitted with a D-sub connector and screwed directly onto feedthroughs on the Acromag chassis.

The attached pdf contains the master list of assigned Acromag channels and their wiring.

Attachment 1: 40m_Vacuum_Acromag_Channels_20190321.pdf
40m_Vacuum_Acromag_Channels_20190321.pdf 40m_Vacuum_Acromag_Channels_20190321.pdf 40m_Vacuum_Acromag_Channels_20190321.pdf
  14495   Mon Mar 25 10:21:05 2019 JonUpdateUpgradec1susaux upgrade plan

Now that the Acromag upgrade of c1vac is complete, the next system to be upgraded will be c1susaux. We chose c1susaux because it is one of the highest-priority systems awaiting upgrade, and because Johannes has already partially assembled its Acromag replacement (see photos below). I've assessed the partially-assembled Acromag chassis and the mostly-set-up host computer and propose we do the following to complete the system.

Documentation

As I go, I'm writing step-by-step documentation here so that others can follow this procedure for future systems. The goal is to create a standard procedure that can be followed for all the remaining upgrades.

Acromag Chassis Status

The bulk of the remaining work is the wiring and testing of the rackmount chassis housing the Acromag units. This system consists of 17 units: 10 ADCs, 4 DACs, and 3 digitial I/O modules. Johannes has already created a full list of channel wiring assignments. He has installed DB37-to-breakout board feedthroughs for all the signal cable connections. It looks like about 40% of the wiring from the breakout boards to Acromag terminals is already done.

The Acromag units have to be initially configured using the Windows laptop connected by USB. Last week I wasn't immediately able to check their configuration because I couldn't power on the units. Although the DC power wiring is complete, when I connected a 24V power supply to the chassis connector and flipped on the switch, the voltage dropped to ~10V irrespective of adjusting the current limit. The 24V indicator lights on the chassis front and back illuminated dimly, but the Acromag lights did not turn on. I suspect there is a short to ground somewhere, but I didn't have time to investigate further. I'll check again this week unless someone else looks at it first.

Host Computer Status

The host computer has already been mostly configured by Johannes. So far I've only set up IP forwarding rules between the martian-facing and Acromag-facing ethernet interfaces (the Acromags are on a subnet inaccessible from the outside). This is documented in the link above. I also plan to set up local installations of modbus and EPICS, as explained below. The new EPICS command file (launches the IOC) and database files (define the channels) have already been created by Johannes. I think all that remains is to set up the IOC as a persistent system service.

Host computer OS

Recommendation from Keith Thorne:

For CDS lab-wide, Jamie Rollins and Ryan Blair have been maintaining Debian 8 and 9 repos with some of these.  
They have somewhat older EPICS versions and may not include all the modules we have for SL7.
One worry is whether they will keep up Debian 9 maintained, as Debian 10 is already out.

I would likely choose Debian 9 instead of Ubuntu 18.04.02, as not sure of Ubuntu repos for EPICS libraries.

Based on this, I propose we use Debian 9 for our Acromag systems. I don't see a strong reason to switch to SL7, especially since c1vac and c1susaux are already set-up using Debian 8. Although Debian 8 is one version out of date, I think it's better to get a well-documented and tested procedure in place before we upgrade the working c1vac and c1susaux computers. When we start building the next system, let's install Debian 9 (or 10, if it's available), get it working with EPICS/modbus, then loop back to c1vac and c1susaux for the OS upgrade.

Local vs. central modbus/EPICS installation

The current convention is for all machines to share a common installation which is hosted on the /cvs/cds network drive. This seems appealing because only a single central EPICS distribution needs to be maintained. However, from experience attempting this on c1vac, I'm convinced this is a bad design for the new Acromag systems.

The problem is that any network outage, even routine maintenance or brief glitches, wreaks havoc on Acromags set up this way. When the network is interrupted, the modbus executable disappears mid-execution, crashing the process and hanging the OS (I think related to the deadlocked NFS mount), so that the only way to recover is to manually power-cycle. Still worse, this can happen silently (channel values freeze), meaning that, e.g., watchdog protections might fail.

To avoid this, I'm planning to install a local EPICS distribution from source on c1susaux, just as I did for c1vac. This only takes a few minutes to do, and I will include the steps in the documented procedure. Building from source also better protects against OS-dependent buginess.

Main TODO items

  • Debug issue with Acromag DC power wiring
  • Complete wiring from chassis feedthroughs to Acromag terminals, following this wiring diagram
  • Check/set the configuration of each Acromag unit using the software on the Windows laptop
  • Set the analog channel calibrations in the EPICS database file
  • Test each channel ex situ. Chub and I discussed an idea to use two DB-37F breakout boards, with the wiring between the board terminals manually set. One DAC channel would be calibrated and driven to test other ADC channels. A similar approach could be used for the digital input/output channels.
Attachment 1: IMG_3136.jpg
IMG_3136.jpg
Attachment 2: IMG_3138.jpg
IMG_3138.jpg
Attachment 3: IMG_3137.jpg
IMG_3137.jpg
  14497   Tue Mar 26 18:35:06 2019 JonUpdateUpgradeModbus IOC is running on c1susaux2

Thanks to new info from Johannes, I was able to finish setting up the modbus IOC on c1susaux2. It turns out the 17 Acromags draw ~1.9 A, which is way more than I had expected. Hence the reason I had suspected a short. Adding a second DC supply in parallel solves the problem. There is no issue with the wiring.

With the Acromags powered on, I carried out the following:

  • Confirmed c1susaux2 can communicate with each Acromag at its assigned IP address
  • Modified the EPICS .cmd file to point to the local modbus installation (not the remote executable on /cvs/cds)
  • Debugged several IOC initialization errors. All were caused by minor typos in the database files.
  • Scripted the modbus IOC to launch as a systemd service (will add implementation details to the documentation page)

The modbusIOC is now running as a peristent system service, which is automatically launched on boot and relaunched after a crash. I'm able to access a random selection of channels using caget.

What's left now is to finish the Acromag-to-feedthrough wiring, then test/calibrate each channel.

  14500   Fri Mar 29 11:43:15 2019 JonUpdateUpgradeFound c1susaux database bug

I found the current bias output channels, C1:SUS-<OPTIC>_<DOF>BiasAdj, were all pointed at C1:SUS-<OPTIC>_ULBiasSet for every degree of freedom. This same issue appeared in all eight database files (one per optic), so it looks like a copy-and-paste error. I fixed them to all reference the correct degree of freedom.

  14505   Mon Apr 1 12:01:52 2019 JonUpdateCDS 

I brought c1susaux back online this morning for suspension-channel test scripting. It had been dead for some time. I followed the procedure outlined in #12542. ITMY became stuck during this process, which Gautam tells me always happens since the last vacuum access, but ITMX is not stuck.

ELOG V3.1.3-