The fire alarm came on around 15:05  for about 2-3 minutes. We all  left the lab and counted heads.  I called Paul Mackel x2646 (cell 626/ 890- 3259) at Fire Protection Services. He said that this alarm test was planned and we should of got an email notice. Perhaps I missed that notes.

I cleaned up the south Electronics bench today.

The other two, as well as several of the desks are in some chaotic state of degradation. Please clean up your areas and put away projects which do not need to remain staged for several months. Try to eliminate "that's not mine" and "I don't know who's that is" from your vocabulary.  Fight back against entropy!

(Steve, Chris)

The pumpdown had stalled because of some ancient vacuum interlock code that prevented opening the valve V1 between the turbo pump and the main volume.

This interlock [0] compares the channels C1:Vac-P1_pressure and C1:Vac-PTP1_pressure, neither of which is functioning at the moment. The P1 channel apparently stopped reading sometime during the vent, and contained a value of ~700 torr, while the PTP1 channel contained 0. So the interlock code saw this huge apparent pressure difference and refused to move the valve.

To bypass this check, we used caput to enter a pressure of 0 for P1.

[0] /cvs/cds/caltech/state/from_luna/VacInterlock.st

I had Rich show me his approach to a chassis for the Acromag modules. The document tree for his design can be found on the DCC. Note that he's using the high densitymodel ES series, which is available as a bare board variant with pluggable screw terminals:

He can fit up to 4 of these in a 2U chassis and has outsourced the wiring from front panel Dsubs to the board connectors to an external company. At the 40m (and in West Bridge) we currently only have the rail mounted XT series

At first glance the specs are very similar. Both A/D and D/A flavors have 16-bit precision in both cases. The high density ES series with Rich's layout can achieve 128 A/D per 2U, 64 D/A per 2U, or 384 DIO per 2U. Into a 4U chassis of the type we have currently we can fit ~32 XT modules (assuming two rows), which results in very similar numbers, except for the DAC, of which we could fit more.

XT1221-000 (8 diff. channel 16-bit ADC)                          $495.00      $61.88/ch
XT1541-000 (8 channel 16-bit DAC and 4 discrete I/O )    $525.00      $65.63/ch
XT1120-000 (16 channel DIO)                                         $320.00     $20.00/ch

ES2162-0010 (32 diff. channel 16-bit ADC)                     $2050.00    $64.06/ch
ES2172-0010 (16 channel 16-bit DAC)                           $1400.00    $87.50/ch
ES2113-0010 (96 channel DIO)                                      $1100.00    $11.46/ch

It's cheaper to stick with the current XT models, but they need the bulkier 4U chassis. The good news is that actually all these models have 16 bit precision, which wasn't clear to me before. Lydia and I will work out what connectors we want on the boxes, and how many modules/channels we need where. Rich also got me in touch with Keith Thorne, who handles the analog I/O Acromag at LLO, and I will ask him for advice. From his documents on the DCC it seems that he is using yet another series: EN. The 968EN-4008 for example is a rail-mounted ADC with pluggable connections, but looses quite clearly in price per channel.

For a generic multipurpose DAQ interface box the ES series is the best approach in my opinion, because it offers a more compact design. We could for example fit 1 ADC, 2 DAC, 1 DIO in a 2U chassis for 32/32/96 channels. The combined price tag for this scenario would be ~$6k.

Current Acromag chassis status:

I found out that Acromag offers DIN rail mounting kits for the open boards, so we can actually fit both XT series and ES/EN series in the same boxes, depending on the signal needs. The primary design driver will be the ES footprint, but if we find we don't need that many channels in some of the units, it's interchangable. For the wiring to the front panel - for which we will have a standard front panel express design, but may order modified ones for the custom needs of the 40m, I will contract the same company that Rich used for the wiring in his DIO box (Panel mount connectors terminating in loose wires/pre-routed plugs for Acromag units). We will either run a single DIN rail along the length of the chassis, or have two in parallel across.

Lydia and I took close looks at the breakout arrangements on the rack sides, and determined that because of the many cross-connects between non-DAQ ports it is not possible to redo and debug this in a reasonable amount of time without essentially shutting down the interferometer. So instead, we will connect the chassis directly to the slots that were previously leading to the slow machines. They come in two different flavors: The ADC modules have 64 pins, while the DIO and DAC ones have 50. There are a couple things we can do:

• For ADC: Put favorite 64+ pin connector on front panel. I would advocate for the 68 pin VHDIC (SCSI-5). This standard ist widely used, has a sturdy connector, and usually off-the-shelf cables have twisted pair leads.
• For DAC+DIO: Either use favorite 50 pin connector (there are 50-pin DSUB connectors, and also 50-pin IDC connectors with backshell), or also send the signals through VHDIC connectors, tolerating a few unused lanes. I would prefer the second option, after all it all goes to some 64 pin VME-crate backplane connector in the end, so if we ever get rid of the rack-side breakouts the wiring will much more uniform.
• For good measure, we will add a few (16 maybe) BNC connectors to the front panel.
• A standardized front panel could have a variety of different connectors by default: DSUBs, BNCs, etc., to be used when needed with some initial default wiring.
• Note that THEORETICALLY we could even connect all backplane EUROCARD ports to the Acromag chassis and do the cross-connect wiring entirely inside, although that would make the inside extremely messy.

Based on Rich's design I will get started on a parts list and wiring diagrams to send out to the cable company.

Very, very cool!  

While walking down to the X end to reset c1iscex I heard what I would call a "rythmic squnching" sound coming from under the turbo pump.  I would have said the sound was coming from a roughing pump, but none of them are on (as far as I can tell).

Steve maybe look into this??

Ifo pressure was 5.5 mTorr this Monday morning. The PSL shutter was still open. TP2 controller failed. Interlock closed V1, V4 and VM1

Turbo pump 2 is the fore pump of the Maglev. The pressure here was 3.9 Torr so The Maglev got warm ~38C but it was still rotating at 560 Hz normal with closed V1

Pressure plots are not available because of computer problems.

What I did:

Looked at pressures of Hornet and Super Bee  Instru Tech. Inc

Closed all annuloses and VA6,  disconnected V4 and VA6 and turned on external fan to cool Maglev

Opened V7 to pump the Maglev fore line with TP3

V1 opened manually when foreline pressure dropped to <2mTorr at P2 and the body temp of the Maglev cooled down to  25-27 C

VM1 opened at 1e-5 Torr

Valve configuration: vacuum normal with annuloses not pumped

Ifo pressure 8.5e-6 Torr -IT at 10am,  P2 foreline pressure 64 mTorr, TP3 controller 0.17A   22C  50Krpm

note: all valves open manually, interlock can only close them

PS: please call me next time you see the vacuum is not Vacuum Normal

I have recommissioned the Zojirushi coffee pot lid. You may, once again, align the dots in order to make the carafe pourable.

Details:

The Zojirushi lid is a two part mechanism:

1. The top part of the lid must be removed for cleaning.
2. When replacing the lid the two components must be aligned to < 3 mrad precision so that the "teeth" are able to land in the groove.
3. There is a 4-fold degeneracy in this process. To break the degeneracy, align the dot on top with the spout gap (visible from the bottom view).
4. After proper alignment and mating, the two parts should snap together and the relative alignment wiggle available should be < 2 mrad.
5. After screwing the two-piece lid onto the carafe, ensure that the 2 dots are separated by < 170 deg in the closed position.

I have moved the USB flash drives from the electronics bench back into the middle drawer of the cabinet next to the AC which is west of the fridge. Drawer re-enlabeled.

Larry Wallace hooked up a new switch (Brocade FWS 648G) today which is our 40m lab interface to the outside world internet. Its faster.

He then, just now, switched over the cables which were going to the old one into the new one, including NODUS and the NAT Router. CDS machines can still connect to the outside world.

In the next week or two, he'll install a new NAT for us so that we can have high speed comm from CDS to the world.

[steve,gautam]

We laid out a 45m long BNC cable from the LSC rack to the IOO rack via overhead cable trays. There is ~5m excess length on either side, which have been coiled up and cable-tied for now. The ends are labelled "TO LSC RACK" and "TO IOO RACK" on the appropriate ends. This is to facilitate hooking up the output of the DFD for making a PM measurement of the AUX X laser. There is already a long cable that runs from the IOO rack to the X end.

Acromag Wireup Update

Channels no longer in use

• From the VME analog output (VMIVME 4116) to the QPD Whitening board (no DCC number on the front), 3 channels are no longer in use
• From the anti-image filter (D000186) to the ADC (VMIVME 3113A) 5 channels are no longer in use (these are the only channels from the anti-image filter, so this filter is no longer in use at all?)
• From the universal dewhitening filter (D000183) to a binary I/O adapter (channels 1-16), 4 channels are no longer in use. These are the only channels from the dewhitening filter
• From a second universal dewhitening filter (D000183) to another the binary I/O adapter (channels 1-16), one channel is no longer in use (this was the only channel from this dewhitening filter).
• From the opti-lever (D010033) to the VME ADC (VMIVME 3113A), 7 channels are no longer in use (this was all of the channels from the opti lever)
• From the SUS PD Whitening/Interface board (D000210) to a binary I/O adapter (channels 1-16), 5 channels are no longer in use. 
• Note that none of the binary I/O adapter channels are in use.

Exactly: you'll have to list explicitly what functions those channels had so that we know what we're losing before we make the switch.

I got the the SuperMicro 1U server box from Larry W on Monday and set it up in the CryoLab for initial testing.

The specs: https://www.supermicro.com/products/system/1U/5015/SYS-5015A-EHF-D525.cfm

The processor is an Intel D525 dual core atom processor with 1.8 GHz (i386 architecture, no 64-bit support). The unit has a 250GB SSD and 4GB RAM.

I installed Debian Jessie on it without any problems and compiled the most recent stable versions of EPICS base (3.15.5), asyn drivers (4-32), and modbus module (2-10-1). EPICS and asyn each took about 10 minutes, and modbus about 1 minute.

I copied the database files and port driver definitions for the cryolab from cryoaux, whose modbus services I suspended, and initialized the EPICS modbus IOC on the SuperMicro machine instead. It's working flawlessly so far, but admittedly the box is not under heavy load in the cryolab, as the framebuilder there is logging only the 16 analog channels.

I have recently worked out some kinks in the port driver and channel definitions, most importantly:

• mosbus IOC initialization is performed automatically by systemd on reboot
• If the IOC crashes or a system reboot is required the Acromag units freeze in their last current state. When the IOC is started a single read operation of all A/D registers is performed and the result taken as the initial value of the corresponding channel, causing no discontinuity in generated voltage EVER (except of course for the rare case when the Acromags themselves have to be restarted)

Aaron and I set 12/4 as a tentative date when we will be ready to attempt a swap. Until then the cabling needs to be finished and a channel database file needs to be prepared.

[Aaron, Johannes]

We configured the AtomServer for the Martian network today. Hostname is c1auxex2, IP is 192.168.113.49. Remote access over SSH is enabled.

There will be 6 acromag units served by c1auxex2.

Some hardware to assemble the Acromag box and adapter PCBs are still missing, and the wiring and channel definitions have to be finalized. The port driver initialization instructions and channel definitions are currently locally stored in /home/controls/modbusIOC/ but will eventually be migrated to a shared location, but we need to decide how exactly we want to set up this infrastructure.

• Should the new machines have the same hostnames as the ones they're replacing? For the transition we simply named it c1auxex2.
• Because the communication of the server machine with the DAQ modules is happening over TCP/IP and not some VME backplane bus we could consolidate machines, particularly in the vertex area.
• It would be good to use the fact that these SuperMicro servers have 2+ ethernet ports to separate CDS EPICS traffic from the modbus traffic. That would also keep the 30+ IPs for the Acromag thingies off the Martian host tables.

I wired up the power distribution, and ethernet cables in the Acromag chassis today. For the time being it's all kind of loose in there but tomorrow the last parts should arrive from McMaster to put everything in its place. I had to unplug some of the wiring that Aaron had already done but labeled everything before I did so. I finalized the IP configuration via USB for all the units, which are now powered through the chassis and active on the network.

I started transcribing the database file ETMXaux.db that is loaded by c1auxex in the format required by the Acromags and made sure that the new c1auxex2 properly functions as a server, which it does.

ToDo-list:

• Need to calibrate the +/- 10V swing of the analog channels via the USB utility, but that requires wiring the channels to the connectors and should probably be done once the unit sits in the rack
• Need to wire power from the Sorensens into the chassis. There are +/- 5V, +/- 15V and +/- 20V present. The Acromags need only +12V-32V, for which I plan to use the +20V, and an excitation voltage for the binary channels, for which I'm going to wire the +5V. Should do this through the fuse rails on the side.
• The current slow binary channels are sinking outputs, same as the XT1111 16-channel module we have. The additional 4 binary outputs of the XT1541 are sourcing, and I'm currently not sure if we can use them with the sos driver and whitening vme boards that get their binary control signals from the slow system.
• Confirm switching of binary channels (haven't used model XT1111 before, but I assume the definitions are identical to XT1121)
• Setup remaining essential EPICS channels and confirm that dimensions are the same (as in both give the same voltage for the same requested value)
• Disconnect DIN cables, attach adapter boards + DSUB cables
• Testing

Getting the chassis ready took a little longer than anticipated, mostly because I had not looked into the channel list myself before and forgot about Lydia's post which mentions that some of the switching controls have to be moved from the fast to the slow DAQ. We would need a total of 5+5+4+8=22 binary outputs. With the existing Acromag units we have 16 sinking outputs and 8 sourcing outputs. I looked through all the Eurocrate modules and confirmed that they all use the same switch topology which has sourcing inputs.

While one can use a pull-down resistor to control a sourcing input with a sourcing output,

pulling down the MAX333A input (datasheet says logic low is <0.8V) requires something like 100 Ohms for the pull down resistor, which would require ~150mA of current PER CHANNEL, which is unreasonable. Instead, I asked Steve to buy a second XT1111 and modified the chassis to accomodate more Acromag units.

I have now finished wiring the chassis (except for 8 remaining bypass controls to the whitening board which need the second XT1111), calibrated all channels in use, confirmed all pin locations via the existing breakout boards and DCC drawings for the eurocrate modules, and today Steve and I added more fuses to the DIN rail power distribution for +20V and +15V.

There was not enough contingent free space in the XEND rack to mount the chassis, so for now I placed it next to it.

c1auxex2 is currently hosting all original physical c1auxex channels (not yet calc records) under their original name with an _XT added at the end to avoid duplicate channel names. c1auxex is still in control of ETMX. All EPICS channels hosted by c1auxex2 are in dimensions of Volts. The plan for tomorrow is to take c1auxex off the grid, rename the c1auxex2 hosted channels and transfer ETMX controls to it, provided we can find enough 37pin DSub cables (8). I made 5 adapter boards for the 5 Eurocrate modules that need to talk to the slow DAQ through their backplane connector.

The new slow machine c1auxex2 is ready to deploy. Unfortunately we don't have enough 37pin DSub cables to connect all channels. In fact, we need a total of 8, and I found only three male-male cables and one gender changer. I asked Steve to buy more.

Over the past week I have transferred all EPICS records - soft channels and physical ones - from c1auxex to c1auxex2, making changes where needed. Today I started the in-situ testing

1. Unplugged ETMX's satellite box
2. Unplugged the eurocrate backplane DIN cables from the SOS Driver and QPD Whitening filter modules (the ones that receive ao channels)
3. Measured output voltages on the relevant pins for comparison after the swap
4. Turned off c1auxex by key, removed ethernet cable
5. Started the modbus ioc on c1auxex2
6. Slow machine indicator channels came online, ETMX Watchdog was responsive (but didn't have anything to do due to missing inputs) and reporting. PIT/YAW sliders function as expected
7. Restoring the previous settings gives output voltages close to the previous values, in fact the exact values requested (due to fresh calibration)
8. Last step is to go live with c1auxex2 and confirm the remaining channels work as expected.

I copied the relevant files to start the modbus server to /cvs/cds/caltech/target/c1auxex2, although kept local copies in /home/controls/modbusIOC/ from which they're still run.

I wonder what's the best practice for this. Probably to store the database files centrally and load them over the network on server start?

I expanded the previous panels to 6U height for the new DAQ chassis we're buying for the upgrade. I figure it's best if we stick to the modular design, so I'm showing a panel for 8 BNC connectors as an example. The front panel has 12 slots, the back has 10 plus power connectors, switches, and the ethernet plug.

I moved the power switch to the rear because it's a waste of space to put it in the front, and it's not like we're power cycling this thing all the time. Note that the unit only requires +24V (for general operation, +20V also does the trick, as is the situation for ETMX) and +15V (excitation field for the binary I/O modules). While these could fit into a single CONEC power connector, it's probably for the better if we don't make a version that supplies a large positive voltage where negative is expected, so I put in two CONEC plugs for +/- 15 and +/- 24.

I want to order 5-6 of these as soon as possible, so if anyone wants anything changed or sees a problem, please do tell!

This afternoon I started setting up the Supermicro 5017A-EP that will replace c1vac1/2. Following Johannes's procedure in 13681 I installed Debian 8.11 (jessie). There is a more recent stable release, 9.5, now available since the first acromag machine was assembled, but I stuck to version 8 for consistency. We already know that version to work. The setup is sitting on the left side of the electronics bench for now.

Today I finished setting up the server that will replace the c1vac1/2 machines. I put it on the martian network at the unassigned IP 192.168.113.72. I assigned it the hostname c1vac and added it to the DNS lookup tables on chiara.

I created a new targets directory on the network drive for the new machine: /cvs/cds/caltech/target/c1vac. After setting EPICS environment environment variables according to 13681 and copying over (and modifiying) the files from /cvs/cds/caltech/target/c1auxex as templates, I was able to start a modbusIOC server on the new machine. I was able to read and write (soft) channel values to the EPICS IOC from other machines on the martian network.

I scripted it as a systemd-managed process which automatically starts on boot and restarts after failure, just as it is set up on c1auxex.

Wiring of the power, Ethernet, and indicator lights for the vacuum Acromag chassis is complete. Even though this crate will only use +24V DC, I wired the +/-15V connector and indicator lights as well to conform to the LIGO standard. There was no wiring diagram available, so I had to reverse-engineer the wiring from the partially complete c1susaux crate. Attached is a diagram for future use. The crate is ready to begin software developing on Monday. 

All 7 Acromag units are now installed in the vacuum chassis. They are connected to 24V DC power and Ethernet.

I have merged and migrated the two EPICS databases from c1vac1 and c1vac2 onto the new machine, with appropriate modifications to address the Acromags rather than VME crate.

I have tested all the digital output channels with a voltmeter, and some of the inputs. Still more channels to be tested.

I’ll follow up with a wiring diagram for channel assignments.

I've set up a closed subnetwork for interfacing the vacuum hardware (Acromags and serial devices) with the new controls machine (c1vac; 192.168.113.72). The controls machine has two Ethernet interfaces, one which faces outward into the martian network and another which faces the internal subnetwork, 192.168.114.xxx. The second network interface was configured via the following procedure.

1. Add the following lines to /etc/network/interfaces:

allow-hotplug eth1
iface eth1 inet static
address 192.168.114.9
netmask 255.255.255.0

2. Restart the networking services:

3. Enable DNS lookup on the martian network by adding the following lines to /etc/resolv.conf:

search martian
nameserver 192.168.113.104

4. Enable IP forwarding from eth1 to eth0:

5. Configure IP tables to allow outgoing connections, while keeping the LAN invisible from outside the gateway (c1vac):

6. Finally, because the EPICS 3.14 server binds to all network interfaces, client applications running on c1vac now see two instances of the EPICS server---one at the outward-facing address and one at the LAN address. To resolve this ambiguity, two additional enviroment variables must be set that specify to local clients which server address to use. Add the following lines to /home/controls/.bashrc:

EPICS_CA_AUTO_ADDR_LIST=NO
EPICS_CA_ADDR_LIST=192.168.113.72

A list of IP addresses so far assigned on the subnetwork follows.

I've completed bench testing of all seven vacuum Acromags installed in a custom rackmount chassis. The system contains five XT1111 modules (sinking digital I/O) used for readbacks of the state of the valves, TP1, CP1, and the RPs. It also contains two XT1121 modules (sourcing digital I/O) used to pass 24V DC control signals to the AC relays actuating the valves and RPs. The list of Acromag channel assignments is attached.

I tested each input channel using a manual flip-switch wired between signal pin and return, verifying the EPICS channel readout to change appropriately when the switch is flipped open vs. closed. I tested each output channel using a voltmeter placed between signal pin and return, toggling the EPICS channel on/off state and verifying the output voltage to change appropriately. These tests confirm the Acromag units all work, and that all the EPICS channels are correctly addressed.

New hardware has been installed in the vacuum controls rack. It is shown in the below post-install photo.

• Supermicro server (c1vac) which will be replacing c1vac1 and c1vac2.
• 16-port Ethernet switch providing a closed local network for all vacuum devices.
• 16-port IOLAN terminal server for multiplexing/Ethernetizing all RS-232 serial devices.

Below is a high-level summary of where things stand, and what remains to be done.

Completed:

✔ Set up of replacement controls server (c1vac).

• Supermicro 1U rackmount server, running Debian 8.5.
• Hosting an EPICS modbus IOC, scripted to start/restart automatically as a system service.
• First Ethernet interface put on the martian network at 192.168.113.72.
• Second Ethernet interface configured to host a LAN at 192.168.114.xxx for communications with all vacuum electronics. It connects to a 16-port Ethernet switch installed in the vacuum electronics rack.
• Server installed in vacuum electronics rack (see photo).

✔ Set up of Acromag terminals.

• 6U rackmount chassis frame assembled; 15V DC, 24V DC, and Ethernet wired.
• Acromags installed in chassis and configured for the LAN (5 XT1111 units, 2 XT1121 units).

✔ EPICS database migration.

• All vacuum channels moved to the modbus IOC, with the database updated to address the new Acromags. [The new channels are running concurrently at "C1:Vac2-...." to avoid conflict with the existing system.]
• Each hard channel was individually tested on the electronics bench to confirm correct addressing and Acromag operation.

✔ Set up of 16-port IOLAN terminal server (for multiplexing/Ethernetizing the serial devices).

• Configured for operation on the LAN. Each serial device port is assigned a unique IP address, making the terminal server transparent to client TCP applications.
• Most of the pressure gauges are now communicating with the controls server via TCP.

Ongoing this week:

• [Jon] Continue migrating serial devices to ports on the terminal server. Still left are the turbo pumps, N2 gauge, and RGA.
• [Jon] Continue developing Python code for communicating with gauges and pumps via TCP sockets. A beta version of gauge readout code is running now.
• [Chub] Install feedthrough panels on the Acromag chassis. Connect the wiring from feedthrough panels to the assigned Acromag slots.
• [Chub/Jon] Test all the hard EPICS channels on the electronics bench, prior to installing the crate in the vacuum rack.
• [Chub/Jon] Install the crate in the vacuum rack; connect valve/pump readbacks and actuators; test each hard EPICS channel in situ.
• [Jon] Once all the signal connections have been made, in situ testing of the Python interlock code can begin.

All the serial vacuum signals are now interfaced to the new digital controls system. A set of persistent Python scripts will query each device at regular intervals (up to ~10 Hz) and push the readings to soft channels hosted by the modbus IOC. Similar scripts will push on/off state commands to the serial turbo pumps.

IP Addresses/Comm Settings

Each serial device is assigned an IP address on the local subnet as follows. Its serial communication parameters as configured in the terminal server are also listed.

Hardware Modifications

• Each of the five vacuum gauge controllers has an RJ45 adapter installed directly on its DB9/DB25 output port. Because the RJ45 cable now plugs directly into the terminal server, instead of passing through some additional adapters as it formerly did, it was necessary to reverse the wiring of the controller TXD and RXD pins to Ethernet pins. The DB9/25-to-RJ45 adapters on the back of the controllers are now wired as follows.
  • For the MKS controllers: DB2 (RXD) --> Eth4;  DB3 (TXD) --> Eth5;  DB5 (RTN) --> Eth6
  • For the Granville-Phillips controllers: DB2 (TXD) --> Eth5;  DB3 (RXD) --> Eth4;  DB7 (RTN) --> Eth6
• I traced a communications error with the GP307 gauge controller all the way back to what I would have suspected least, the controller itself. The comm card inside each controller has a set of mechanical relay switches which set the communications parameters (baud rate, parity, etc.). Knowing that this controller was not part of the original installation, but was swapped in to replace the original in 2009, I pulled the controller from the rack and checked the internal switch settings. Sure enough, the switch settings (pictured below) were wrong. In the background of the photo is the unit removed in 2009, which has the correct settings. After setting the correct communications parameters, the controller immediately began communicating with the server. Did these readouts (PRP, PTP1) never work since 2009? I don't see how they could.

I've made a repair to the N2 pressure monitor. I don't believe the polarity of the analog signal into the controller actually was reversed. I found the data sheet (attached) for the transducer model we have installed. Its voltage should read ~0 at 0 PSI and 100mV at 100 PSI. As wired, the input voltage reads +80 mV as it should.

The controller calibrates the sensor voltage to PSI (i.e., applies a scale and offset) based on two settable reference points which appeared to be incorrect. I changed them to:

1. 0 mV = 0 PSI (neglecting the small dark bias)
2. 100 mV = 100 PSI

After the change, the pressure reads 80 PSI. Let's see if the time history now shows a sensible trend. 

Below is an inventory of the signal feedthroughs that need to be installed on the vacuum Acromag crate this week.

**The original documentation lists five satellite boxes (one for each test mass chamber and one for the beamsplitter chamber), but Chub reports not all of them are in use. We may remove the ones not used.

Based on new input from Chub, attached is the revised list of signal cable feedthroughs needed on the vacuum system Acromag crate. I believe this list is now complete.

[jon, gautam]

In the latest installment in this puzzler: turns out that maybe the trend of the "N2 pressure" channel increasing over the ~3 day timescale it takes a cylinder of N2 to run out is real, and is a feature of the way our two N2 cylinder lines/regulators are setup (for the automatic switching between cylinders when one runs out). In order to test this hypothesis, we'd like to have the line pressure be 0 initially, and then just have 1 cylinder hooked up. When we went into the drill-press area, we heard a hiss, turns out that one of the cylinders is leaking (to be fair, this was labelled, but i thought it isn't great to have a higher N2 concentration in an enclosed space). Since we don't need any actuation ability, I valved off the leaky cylinder, and disconnected the other properly functioning one. Attachment #1 shows the current state.

I believe I finally have the N2 gauge working correctly. The wiring is unchanged from its original state and the controller has been recalibrated.

After letting the line pressure drop to 0 PSI as indicated by the analog gauge in the drill-press room, I recorded the number of counts read by the Omega controller. Then I pressurized the line to 80 PSI, again indicated by the analog gauge, and recorded the Omega counts again. I entered these two reference points into the controller (automatically determines the gain and offset from these), then confirmed the readings to agree with the anaog gauge as I varied the line pressure.

The two reference points are:

There turned out to be a few analog signals for the vacuum system after all. The TP2/3 foreline pressure gauges were never part of the digital system, but we wanted to add them, as some of the interlock conditions should be predicated on their readings. Each gauge connects to an old Granville-Phillips 375 controller which only has an analog output. Interfacing these signals with the new system required installing an Acromag XT1221 8-channel A/D unit. Taking advantage of the extra channels, I also moved the N2 delivery line pressure transducer to the XT1221, eliminating the need for its separate Omega DPiS32 controller. When the new high-pressure transducers are added to the two N2 tanks, their signals can also be connected.

The XT1221 is mounted on the DIN rail inside the chassis and I have wired a DB-9 feedthrough for each of its three input signals. It is assigned the IP 192.168.114.27 on the vacuum subnet. Testing the channels in situ revealed a subtley in calibrating them to physical units. It was first encountered by Johannes in a series of older posts, but I repeat it here in one place.

An analog-input EPICS channel can be calibrated from raw ADC counts to physical units (e.g., sensor voltage) in two ways:

1. Via LINR="LINEAR" by setting the engineering-units fields EGUF="[V_max_adc]", EGUL="[V_min_adc]"
2. Via LINR="NO CONVERSION" by manually setting the gain ASLO="[V/count]" and offset AOFF="[V_offset]"

From the documentation, under the engineering-units method EPICS internally computes:

where EGUF="eng units full scale", EGUL="eng units low", and "full scale A/D counts" is the full range of ADC counts. EPICS automatically infers the range of ADC counts based on the data type returned by the ADC. For a 16-bit ADC like the XT1221, this number is 2^16 = 65,536.

The problem is that, for unknown reasons, the XT1221 rescales its values post-digitization to lie within the range +/-30,000 counts. This corresponds to an actual "full scale A/D counts" = 60,001. If a multiplicative correction factor of 65,536/60,000 is absorbed into the values of EGUF and EGUL, then the first term in the above summation can be corrected. However, the second term (the offset) has no dependence on "full scale A/D counts" and should NOT absorb a correction factor. Thus adjusting the EGUF and EGUL values from, e.g., 10V to 10.92V is only correct when EGUL=0V. Otherwise there is a bias introduced from the offset term also being rescaled.

The generally correct way to handle this correction is to use the manual "NO CONVERSION" method. It constructs calibrated values by simply applying a specified gain and offset to the raw ADC counts:

calibrated val = (measured A/D counts)  x ASLO + AOFF

The gain ASLO="[(V_max_adc - V_min_adc) / 60,001]" and the offset AOFF="0". I have tested this on the three vacuum channels and confirmed it works. Note that if the XT1221 input voltage range is restricted from its widest +/-10V setting, the number of counts is not necessarily 60,001. Page 42 of the manual gives the correct counts for each voltage setting.

[Jon, Chub, Koji, Gautam]

Summary

Today we carried out the first pumpdown with the new vacuum controls system in place. It performed well. The only problem encountered was with software interlocks spuriously closing valves as the Pirani gauges crossed 1E-4 torr. At that point their readback changes from a number to "L OE-04, " which the system interpreted as a gauge failure instead of "<1E-4." This posed no danger and was fixed on the spot. The main volume was pumped to ~10 torr using roughing pumps 1 and 3. We were limited only by time, as we didn't get started pumping the main volume until after 1pm. The three turbo pumps were also run and tested in parallel, but were isolated to the pumpspool volume. At the end of the day, we closed every pneumatic valve and shut down all five pumps. The main volume is sealed off at ~10 torr, and the pumpspool volume is at ~1e-6 torr. We are leaving the system parked in this state for the holidays. 

Main Volume Pumpdown Procedure

In pumping down the main volume, we carried out the following procedure.

1. Initially: All valves closed (including manual valves RV1 and VV1); all pumps OFF.
2. Manually connected roughing pump line to pumpspool via KF joint.
3. Turned ON RP1 and RP2.
4. Waited until roughing pump line pressure (PRP) < 0.5 torr.
5. Opened V3.
6. Waited until roughing pump line pressure (PRP) < 0.5 torr.
7. Manually opened RV1 throttling valve to main volume until pumpdown rate reached ~3 torr/min (~3 hours on roughing pumps).
8. Waited until main volume pressure (P1a/P1b) < 0.5 torr.

We didn't quite reach the end of step 8 by the time we had to stop. The next step would be to valve out the roughing pumps and to valve in the turbo pumps.

Hardware & Channel Assignments

All of the new hardware is now permanently installed in the vacuum rack. This includes the SuperMicro rack server (c1vac), the IOLAN serial device server, a vacuum subnet switch, and the Acromag chassis. Every valve/pump signal cable that formerly connected to the VME bus through terminal blocks has been refitted with a D-sub connector and screwed directly onto feedthroughs on the Acromag chassis.

The attached pdf contains the master list of assigned Acromag channels and their wiring.

Per the discussion yesterday, I valved off the N2 line in the drill press room at 11 am PST today morning so as to avoid any accidental software induced gate-valve actuation during the holidays. The line pressure is steadily dropping...

Attachment #1 shows that while the main volume pressure was stable overnight, the the pumpspool pressure has been steadily rising. I think this is to be expected as the turbo pumps aren't running and the valves can't preserve the <1mtorr pressure over long timescales?

Attachment #2 shows the current VacOverview MEDM screen status.

Independent question: Are all the turbo forelines vented automatically? We manually did it for the main roughing line.

Larry W came by the 40m, and reported that there was a campus-wide power glitch (he was here to check if our networking infrastructure was affected). I thought I'd check the status of the vacuum.

• Attachment #1 is a screenshot of the Vac overview MEDM screen. Clearly something has gone wrong with the modbus process(es). Only the PTP2 and PTP3 gauges seem to be communicative.
• Attachment #2 shows the minute trend of the pressure gauges for a 12 day period - it looks like there is some issue with the frame builder clock, perhaps this issue resurfaced? But checking the system time on FB doesn't suggest anything is wrong.. I double checked with dataviewer as well that the trends don't exist... But checking the status of the individual daqd processes indeed showed that the dates were off by 1 year, so I just restarted all of them and now the time seems correct. How can we fix this problem more permanently? Also, the P1b readout looks suspicious - why are there periods where it seems like we are reading values better than the LSB of the device?

I decided to check the systemctl process status on c1vac:

So something did happen today that required restart of the modbus processes. But clearly not everything has come back up gracefully. A few lines of dmesg (there are many more segfaults):

I don't know enough about the new system so I'm leaving this for Jon to debug. Attachment #3 shows that the analog readout of the P1 pressure gauge suggests that the IFO is still under vacuum, so no random valve openings were effected (as expected, since we valved off the N2 line for this very purpose).

Yes, for TP2 and TP3. They both have a small vent valve that opens automatically on shutdown.

The base Acromag vacuum system is running and performing nicely. Here is a list of remaining questions and to-do items we still need to address.

Safety Issues

• Interlock for HV supplies. The vac system hosts a binary EPICS channel that is the interlock signal for the in-vacuum HV supplies. The channel value is OFF when the main volume pressure is in the arcing range, 3 mtorr - 500 torr, and ON otherwise. Is there something outside the vacuum system monitoring this channel and toggling the HV supplies?
• Exposed 30-amp supply terminals. The 30-amp output terminals on the back of the Sorensen in the vac rack are exposed. We need a cover for those.
• Interlock for AC power loss. The current vac system is protected only from transient power glitches, not an extended loss. The digital system should sense an outage and put the IFO into a safe state (pumps spun down and critical valves closed) before the UPS battery is fully drained. However, it presently has no way of sensing when power has been lost---the system just continues running normally on UPS power until the battery dies, at which point there is a sudden, uncontrolled shutdown. Is it possible for the digital system to communicate directly with the UPS to poll its activation state?

Infrastructure Improvements

• Install the new N2 tank regulator and high-pressure transducers (we have the parts; on desk across from electronics bench). Run the transducer signal wires to the Acromag chassis in the vacuum rack.
• Replace the kludged connectors to the Hornet and SuperBee serial outputs with permanent ones (we need to order the parts).
• Wire the position indicator readback on the manual TP1 valve to the Acromag chassis.
• Add cable tension relief to the back of the vac rack.
• Add the TP1 analog readback signals (rotation speed and current) to the digital system.  Digital temperature, current, voltage, and rotation speed signals have already been added for TP2 and TP3.
• Set up a local vacuum controls terminal on the desk by the vac rack.
• Remove gauges from the EPICS database/MEDM screens that are no longer installed or functional. Potential candidates for removal: PAN, PTP1, IG1, CC2, CC3, CC4.
• Although it appeared on the MEDM screen, the RGA was never interfaced to the old vac system. Should it be connected to c1vac now?

I've attached my handwritten notes covering all the serial communications in the vac system, and the relevant wiring for all the adapters, etc. I'll work with Chub to produce a final documentation, but in the meantime this may be a useful reference.

Updated vac channel list is attached. There are several new ADC channels.

Now that the replacement susaux machine is installed and fully tested, I renamed it from c1susaux2 to c1susaux and updated the DNS lookup tables on chiara accordingly.

For future reference:

• The updated list of c1susaux channel wiring (includes the "coil enable" --> "coil test" digital outputs change)
• Step-by-step instructions on how to set up an Acromag system from scratch

This evening I added a second ADC module to the prototype Acromag chassis. This chassis can now read out all the PSL diagnostic channels.

I configured the second ADC identically to the first ("ADC0"), and assigned it IP address 192.168.113.122. I confirmed it is visible on the martian network.

There was an existing but unused DB-15 feedthrough which I used for ADC1 channels 1-7. The eighth channel I left unwired, but there are slots available in the neighboring DB-25 feedthough, if that channel is needed in the future. The channel wiring assignments are as follows.

I tested all seven of these channels by applying a calibrated voltage source and measuring the response via the Windows Acromag software. All work and are correctly calibrated to better than 0.1%.

I've started setting up the last new rackmount SuperMicro as a dedicated server for the GigE cameras. The new machine is currently sitting on the end of the electronics test bench. It is assigned the hostname c1cam at IP 192.168.113.116 on the martian network. I've installed Debian 10, which will be officially supported until July 2024.

I've added the /cvs/cds NFS mount and plan to house all the client/server code on this network disk. Any dependencies that must be built from source will be put on the network disk as well. Any dependencies that can be gotten through the package manager, however, will be installed locally but in an automated way using a reqs file.

We should ask Chub to reorder several more SuperMicro rackmount machines, SSD drives, and DRAM cards. Gautam has the list of parts from Johannes' last order.

Recently, accordian to Gautam, the NDS2 server has been dying on Megatron ~daily or weekly. The prescription is to restart the server.

1. I could find no instructions (that work) in the elog or wiki. We must remove the misleading entries from the wiki and update it with whatever works as of today.
2. There is a line (which has been commented out) in the Megatron crontab which is close to the right command, but it has the wrong path.
3. Running the command from the CRON (/home/nds2mgr/nds2-megatron/test_restart), gives several errrors.
4. when I run the init.d command which is in the script, it seems to run fine
5. the server then takes several minutes to get itself together; i.e. just because it is running doesn't mean that you can get data. I recommend waiting 5-10 minutes.

Also, megatron is running Ubuntu 12 !! Let's decide on a day to upgrade it to a Debian 18ish....word from Rolf is that Scientific Linux is fading out everywhere, so Debian is the new operating system for all conformists.

Since there has been a proliferation of BHD Google docs recently, I've linked them all from the BHD wiki page. Let's continue adding any new docs to this central list.

Replacement Hardware Purchase List

I've created a purchase list of hardware needed to restore the aging vacuum system. This wasn't planned as part of the BHD upgrade, but I've added it to the BHD procurement list since hardware replacements have become necessary.

The list proposes replacing the aging TP3 Varian turbo pump with the newer Agilent model which has already replaced TP2. It seems I was mistaken in believing we already had a second Agilent pump on hand. A thorough search of the lab has not turned it up, and Steve himself has told me he doesn't remember ordering a second one. Fortunately Steve did leave us a detailed Agilent parts list [ELOG 14322].

It also proposes replacing the glitching TP2 Agilent controller with a new one. The existing one can be sent back for repair and then retained as a spare. Considering that one of these controllers is already malfunctioning after < 2 years, I think it's a very good idea to have a spare on hand.

Known Hardware Issues

Below is our current list of vacuum hardware issues. Items that this purchase list will address (limited to only the most urgent) are highlighted in yellow.

• Replace the UPS
  • Need a 240V socket for TP1 (currently TP1 is not protected from power loss)
  • Need RS232/485 comms with the interlock server (current UPS: serial readbacks have failed, battery is failing)
• Remove/replace the failed pressure gauges (~5)
• Add more cold cathode sensors to the main volume for sensor redundancy (currently the main-volume interlocks rely on only 1 working sensor)
• Replace TP3 (controller is failing)
• Replace TP2 controller (serial interface has failed)
• Remove RP2
  • Dead and also not needed. We already have to throttle the pumpdown rate with only two roughing pumps
• Remove/refurbish the cryopump
  • Contamination risk to have it sitting connectable to the main volume