I've checked out cdsutils-274 to /opt/rtcds/cdsutils, and updated the /ligo/apps/ligoapps-user-env.sh to have the newer machines use it by default. This was to gain access to the cdsutils.Step methods for use in the smooth ASS handoffs script.
I installed awgstream-2.16.14 in /ligo/apps/ubuntu12. As with all the ubuntu12 "packages", you need to source the ubuntu12 ligoapps environment script:
controls@pianosa|~ > . /ligo/apps/ubuntu12/ligoapps-user-env.sh
controls@pianosa|~ > which awgstream
I tested it on the SRM LSC filter bank. In one terminal I opened the following camonitor on C1:SUS-SRM_LSC_OUTMON. In another terminal I ran the following:
controls@pianosa|~ > seq 0 .1 16384 | awgstream C1:SUS-SRM_LSC_EXC 16384 -
Channel = C1:SUS-SRM_LSC_EXC
File = -
Scale = 1.000000
Start = 1092790384.000000
The camonitor output was:
controls@pianosa|~ > camonitor C1:SUS-SRM_LSC_OUTMON
C1:SUS-SRM_LSC_OUTMON 2014-08-22 17:44:50.997418 0
C1:SUS-SRM_LSC_OUTMON 2014-08-22 17:52:49.155525 218.8
C1:SUS-SRM_LSC_OUTMON 2014-08-22 17:52:49.393404 628.4
C1:SUS-SRM_LSC_OUTMON 2014-08-22 17:52:49.629822 935.6
C1:SUS-SRM_LSC_OUTMON 2014-08-22 17:52:58.210810 15066.8
C1:SUS-SRM_LSC_OUTMON 2014-08-22 17:52:58.489501 15476.4
C1:SUS-SRM_LSC_OUTMON 2014-08-22 17:52:58.747095 15886
C1:SUS-SRM_LSC_OUTMON 2014-08-22 17:52:59.011415 0
In other words, it seems to work.
I just installed cdsutils r351 at /ligo/apps/linux-x86_64/cdsutils. It should be available on all workstations.
It includes a bunch of bug fixes and feature improvements, including the step stuff that Rana was complaining about.
Cdsutils r361 installed, for the "avg" updates. aLOG
It's super cold in the control room and EE bench area tonight. I'm wondering if, similar to what happened on Dec 29th (http://nodus.ligo.caltech.edu:8080/40m/10846) the campus steam is off? Or just our heater is broken? The thermostat is cranked up to 80 over by the bathrooms (this is usually ~74F), but we're still cold.
It's 69F in the control room right now (usually mid-high 70s).
EDIT, JCD, 4am: It's 64.3F in the desk area, 67.8F in the control room. It also smells in the control room like some heater has been off for a while, and is turning back on - that burned dust smell that happens after you haven't turned on the heater all summer.
EDIT again: The burn-y smell is getting stronger I think. Security is sending someone over to come check it out.
After some searching, including help from 4 security guys (I think they don't have a lot to do at 4:30am :), we found that Ottavia is super warm, and smelled burn-y. She has been powered down and unplugged. Security guys may call Steve's desk to follow up later today.
Fire alarm went off several minutes ago. Talked to security and they said there was no fire. It beeped twice again just now. No one has been working on the IFO today.
The fire alarm came on around 15:05 for about 2-3 minutes. We all left the lab and counted heads. I called Paul Mackel x2646 (cell 626/ 890- 3259) at Fire Protection Services. He said that this alarm test was planned and we should of got an email notice. Perhaps I missed that notes.
I cleaned up the south Electronics bench today.
The other two, as well as several of the desks are in some chaotic state of degradation. Please clean up your areas and put away projects which do not need to remain staged for several months. Try to eliminate "that's not mine" and "I don't know who's that is" from your vocabulary. Fight back against entropy!
The pumpdown had stalled because of some ancient vacuum interlock code that prevented opening the valve V1 between the turbo pump and the main volume.
This interlock  compares the channels C1:Vac-P1_pressure and C1:Vac-PTP1_pressure, neither of which is functioning at the moment. The P1 channel apparently stopped reading sometime during the vent, and contained a value of ~700 torr, while the PTP1 channel contained 0. So the interlock code saw this huge apparent pressure difference and refused to move the valve.
To bypass this check, we used caput to enter a pressure of 0 for P1.
I had Rich show me his approach to a chassis for the Acromag modules. The document tree for his design can be found on the DCC. Note that he's using the high densitymodel ES series, which is available as a bare board variant with pluggable screw terminals:
He can fit up to 4 of these in a 2U chassis and has outsourced the wiring from front panel Dsubs to the board connectors to an external company. At the 40m (and in West Bridge) we currently only have the rail mounted XT series
At first glance the specs are very similar. Both A/D and D/A flavors have 16-bit precision in both cases. The high density ES series with Rich's layout can achieve 128 A/D per 2U, 64 D/A per 2U, or 384 DIO per 2U. Into a 4U chassis of the type we have currently we can fit ~32 XT modules (assuming two rows), which results in very similar numbers, except for the DAC, of which we could fit more.
XT1221-000 (8 diff. channel 16-bit ADC) $495.00 $61.88/ch
XT1541-000 (8 channel 16-bit DAC and 4 discrete I/O ) $525.00 $65.63/ch
XT1120-000 (16 channel DIO) $320.00 $20.00/ch
ES2162-0010 (32 diff. channel 16-bit ADC) $2050.00 $64.06/ch
ES2172-0010 (16 channel 16-bit DAC) $1400.00 $87.50/ch
ES2113-0010 (96 channel DIO) $1100.00 $11.46/ch
It's cheaper to stick with the current XT models, but they need the bulkier 4U chassis. The good news is that actually all these models have 16 bit precision, which wasn't clear to me before. Lydia and I will work out what connectors we want on the boxes, and how many modules/channels we need where. Rich also got me in touch with Keith Thorne, who handles the analog I/O Acromag at LLO, and I will ask him for advice. From his documents on the DCC it seems that he is using yet another series: EN. The 968EN-4008 for example is a rail-mounted ADC with pluggable connections, but looses quite clearly in price per channel.
For a generic multipurpose DAQ interface box the ES series is the best approach in my opinion, because it offers a more compact design. We could for example fit 1 ADC, 2 DAC, 1 DIO in a 2U chassis for 32/32/96 channels. The combined price tag for this scenario would be ~$6k.
Current Acromag chassis status:
I found out that Acromag offers DIN rail mounting kits for the open boards, so we can actually fit both XT series and ES/EN series in the same boxes, depending on the signal needs. The primary design driver will be the ES footprint, but if we find we don't need that many channels in some of the units, it's interchangable. For the wiring to the front panel - for which we will have a standard front panel express design, but may order modified ones for the custom needs of the 40m, I will contract the same company that Rich used for the wiring in his DIO box (Panel mount connectors terminating in loose wires/pre-routed plugs for Acromag units). We will either run a single DIN rail along the length of the chassis, or have two in parallel across.
Lydia and I took close looks at the breakout arrangements on the rack sides, and determined that because of the many cross-connects between non-DAQ ports it is not possible to redo and debug this in a reasonable amount of time without essentially shutting down the interferometer. So instead, we will connect the chassis directly to the slots that were previously leading to the slow machines. They come in two different flavors: The ADC modules have 64 pins, while the DIO and DAC ones have 50. There are a couple things we can do:
Based on Rich's design I will get started on a parts list and wiring diagrams to send out to the cable company.
Very, very cool!
While walking down to the X end to reset c1iscex I heard what I would call a "rythmic squnching" sound coming from under the turbo pump. I would have said the sound was coming from a roughing pump, but none of them are on (as far as I can tell).
Steve maybe look into this??
Ifo pressure was 5.5 mTorr this Monday morning. The PSL shutter was still open. TP2 controller failed. Interlock closed V1, V4 and VM1
Turbo pump 2 is the fore pump of the Maglev. The pressure here was 3.9 Torr so The Maglev got warm ~38C but it was still rotating at 560 Hz normal with closed V1
Pressure plots are not available because of computer problems.
What I did:
Looked at pressures of Hornet and Super Bee Instru Tech. Inc
Closed all annuloses and VA6, disconnected V4 and VA6 and turned on external fan to cool Maglev
Opened V7 to pump the Maglev fore line with TP3
V1 opened manually when foreline pressure dropped to <2mTorr at P2 and the body temp of the Maglev cooled down to 25-27 C
VM1 opened at 1e-5 Torr
Valve configuration: vacuum normal with annuloses not pumped
Ifo pressure 8.5e-6 Torr -IT at 10am, P2 foreline pressure 64 mTorr, TP3 controller 0.17A 22C 50Krpm
note: all valves open manually, interlock can only close them
PS: please call me next time you see the vacuum is not Vacuum Normal
The Zojirushi lid is a two part mechanism:
I have moved the USB flash drives from the electronics bench back into the middle drawer of the cabinet next to the AC which is west of the fridge. Drawer re-enlabeled.
Larry Wallace hooked up a new switch (Brocade FWS 648G) today which is our 40m lab interface to the outside world internet. Its faster.
He then, just now, switched over the cables which were going to the old one into the new one, including NODUS and the NAT Router. CDS machines can still connect to the outside world.
In the next week or two, he'll install a new NAT for us so that we can have high speed comm from CDS to the world.
We laid out a 45m long BNC cable from the LSC rack to the IOO rack via overhead cable trays. There is ~5m excess length on either side, which have been coiled up and cable-tied for now. The ends are labelled "TO LSC RACK" and "TO IOO RACK" on the appropriate ends. This is to facilitate hooking up the output of the DFD for making a PM measurement of the AUX X laser. There is already a long cable that runs from the IOO rack to the X end.
Exactly: you'll have to list explicitly what functions those channels had so that we know what we're losing before we make the switch.
I got the the SuperMicro 1U server box from Larry W on Monday and set it up in the CryoLab for initial testing.
The specs: https://www.supermicro.com/products/system/1U/5015/SYS-5015A-EHF-D525.cfm
The processor is an Intel D525 dual core atom processor with 1.8 GHz (i386 architecture, no 64-bit support). The unit has a 250GB SSD and 4GB RAM.
I installed Debian Jessie on it without any problems and compiled the most recent stable versions of EPICS base (3.15.5), asyn drivers (4-32), and modbus module (2-10-1). EPICS and asyn each took about 10 minutes, and modbus about 1 minute.
I copied the database files and port driver definitions for the cryolab from cryoaux, whose modbus services I suspended, and initialized the EPICS modbus IOC on the SuperMicro machine instead. It's working flawlessly so far, but admittedly the box is not under heavy load in the cryolab, as the framebuilder there is logging only the 16 analog channels.
I have recently worked out some kinks in the port driver and channel definitions, most importantly:
Aaron and I set 12/4 as a tentative date when we will be ready to attempt a swap. Until then the cabling needs to be finished and a channel database file needs to be prepared.
We configured the AtomServer for the Martian network today. Hostname is c1auxex2, IP is 192.168.113.49. Remote access over SSH is enabled.
There will be 6 acromag units served by c1auxex2.
Some hardware to assemble the Acromag box and adapter PCBs are still missing, and the wiring and channel definitions have to be finalized. The port driver initialization instructions and channel definitions are currently locally stored in /home/controls/modbusIOC/ but will eventually be migrated to a shared location, but we need to decide how exactly we want to set up this infrastructure.
I wired up the power distribution, and ethernet cables in the Acromag chassis today. For the time being it's all kind of loose in there but tomorrow the last parts should arrive from McMaster to put everything in its place. I had to unplug some of the wiring that Aaron had already done but labeled everything before I did so. I finalized the IP configuration via USB for all the units, which are now powered through the chassis and active on the network.
I started transcribing the database file ETMXaux.db that is loaded by c1auxex in the format required by the Acromags and made sure that the new c1auxex2 properly functions as a server, which it does.
Getting the chassis ready took a little longer than anticipated, mostly because I had not looked into the channel list myself before and forgot about Lydia's post which mentions that some of the switching controls have to be moved from the fast to the slow DAQ. We would need a total of 5+5+4+8=22 binary outputs. With the existing Acromag units we have 16 sinking outputs and 8 sourcing outputs. I looked through all the Eurocrate modules and confirmed that they all use the same switch topology which has sourcing inputs.
While one can use a pull-down resistor to control a sourcing input with a sourcing output,
pulling down the MAX333A input (datasheet says logic low is <0.8V) requires something like 100 Ohms for the pull down resistor, which would require ~150mA of current PER CHANNEL, which is unreasonable. Instead, I asked Steve to buy a second XT1111 and modified the chassis to accomodate more Acromag units.
I have now finished wiring the chassis (except for 8 remaining bypass controls to the whitening board which need the second XT1111), calibrated all channels in use, confirmed all pin locations via the existing breakout boards and DCC drawings for the eurocrate modules, and today Steve and I added more fuses to the DIN rail power distribution for +20V and +15V.
There was not enough contingent free space in the XEND rack to mount the chassis, so for now I placed it next to it.
c1auxex2 is currently hosting all original physical c1auxex channels (not yet calc records) under their original name with an _XT added at the end to avoid duplicate channel names. c1auxex is still in control of ETMX. All EPICS channels hosted by c1auxex2 are in dimensions of Volts. The plan for tomorrow is to take c1auxex off the grid, rename the c1auxex2 hosted channels and transfer ETMX controls to it, provided we can find enough 37pin DSub cables (8). I made 5 adapter boards for the 5 Eurocrate modules that need to talk to the slow DAQ through their backplane connector.
The new slow machine c1auxex2 is ready to deploy. Unfortunately we don't have enough 37pin DSub cables to connect all channels. In fact, we need a total of 8, and I found only three male-male cables and one gender changer. I asked Steve to buy more.
Over the past week I have transferred all EPICS records - soft channels and physical ones - from c1auxex to c1auxex2, making changes where needed. Today I started the in-situ testing
I copied the relevant files to start the modbus server to /cvs/cds/caltech/target/c1auxex2, although kept local copies in /home/controls/modbusIOC/ from which they're still run.
I wonder what's the best practice for this. Probably to store the database files centrally and load them over the network on server start?
I expanded the previous panels to 6U height for the new DAQ chassis we're buying for the upgrade. I figure it's best if we stick to the modular design, so I'm showing a panel for 8 BNC connectors as an example. The front panel has 12 slots, the back has 10 plus power connectors, switches, and the ethernet plug.
I moved the power switch to the rear because it's a waste of space to put it in the front, and it's not like we're power cycling this thing all the time. Note that the unit only requires +24V (for general operation, +20V also does the trick, as is the situation for ETMX) and +15V (excitation field for the binary I/O modules). While these could fit into a single CONEC power connector, it's probably for the better if we don't make a version that supplies a large positive voltage where negative is expected, so I put in two CONEC plugs for +/- 15 and +/- 24.
I want to order 5-6 of these as soon as possible, so if anyone wants anything changed or sees a problem, please do tell!
This afternoon I started setting up the Supermicro 5017A-EP that will replace c1vac1/2. Following Johannes's procedure in 13681 I installed Debian 8.11 (jessie). There is a more recent stable release, 9.5, now available since the first acromag machine was assembled, but I stuck to version 8 for consistency. We already know that version to work. The setup is sitting on the left side of the electronics bench for now.
Today I finished setting up the server that will replace the c1vac1/2 machines. I put it on the martian network at the unassigned IP 192.168.113.72. I assigned it the hostname c1vac and added it to the DNS lookup tables on chiara.
I created a new targets directory on the network drive for the new machine: /cvs/cds/caltech/target/c1vac. After setting EPICS environment environment variables according to 13681 and copying over (and modifiying) the files from /cvs/cds/caltech/target/c1auxex as templates, I was able to start a modbusIOC server on the new machine. I was able to read and write (soft) channel values to the EPICS IOC from other machines on the martian network.
I scripted it as a systemd-managed process which automatically starts on boot and restarts after failure, just as it is set up on c1auxex.
Wiring of the power, Ethernet, and indicator lights for the vacuum Acromag chassis is complete. Even though this crate will only use +24V DC, I wired the +/-15V connector and indicator lights as well to conform to the LIGO standard. There was no wiring diagram available, so I had to reverse-engineer the wiring from the partially complete c1susaux crate. Attached is a diagram for future use. The crate is ready to begin software developing on Monday.
All 7 Acromag units are now installed in the vacuum chassis. They are connected to 24V DC power and Ethernet.
I have merged and migrated the two EPICS databases from c1vac1 and c1vac2 onto the new machine, with appropriate modifications to address the Acromags rather than VME crate.
I have tested all the digital output channels with a voltmeter, and some of the inputs. Still more channels to be tested.
I’ll follow up with a wiring diagram for channel assignments.
I've set up a closed subnetwork for interfacing the vacuum hardware (Acromags and serial devices) with the new controls machine (c1vac; 192.168.113.72). The controls machine has two Ethernet interfaces, one which faces outward into the martian network and another which faces the internal subnetwork, 192.168.114.xxx. The second network interface was configured via the following procedure.
1. Add the following lines to /etc/network/interfaces:
iface eth1 inet static
2. Restart the networking services:
$sudo /etc/init.d/networking restart
3. Enable DNS lookup on the martian network by adding the following lines to /etc/resolv.conf:
4. Enable IP forwarding from eth1 to eth0:
$sudo echo 1 > /proc/sys/net/ipv4/ip_forward
5. Configure IP tables to allow outgoing connections, while keeping the LAN invisible from outside the gateway (c1vac):
$sudo iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE
$sudo iptables -A FORWARD -i eth0 -o eth1 -m state --state RELATED,ESTABLISHED -j ACCEPT
$sudo iptables -A FORWARD -i eth1 -o eth0 -j ACCEPT
6. Finally, because the EPICS 3.14 server binds to all network interfaces, client applications running on c1vac now see two instances of the EPICS server---one at the outward-facing address and one at the LAN address. To resolve this ambiguity, two additional enviroment variables must be set that specify to local clients which server address to use. Add the following lines to /home/controls/.bashrc:
A list of IP addresses so far assigned on the subnetwork follows.
I've completed bench testing of all seven vacuum Acromags installed in a custom rackmount chassis. The system contains five XT1111 modules (sinking digital I/O) used for readbacks of the state of the valves, TP1, CP1, and the RPs. It also contains two XT1121 modules (sourcing digital I/O) used to pass 24V DC control signals to the AC relays actuating the valves and RPs. The list of Acromag channel assignments is attached.
I tested each input channel using a manual flip-switch wired between signal pin and return, verifying the EPICS channel readout to change appropriately when the switch is flipped open vs. closed. I tested each output channel using a voltmeter placed between signal pin and return, toggling the EPICS channel on/off state and verifying the output voltage to change appropriately. These tests confirm the Acromag units all work, and that all the EPICS channels are correctly addressed.
New hardware has been installed in the vacuum controls rack. It is shown in the below post-install photo.
Below is a high-level summary of where things stand, and what remains to be done.
✔ Set up of replacement controls server (c1vac).
✔ Set up of Acromag terminals.
✔ EPICS database migration.
✔ Set up of 16-port IOLAN terminal server (for multiplexing/Ethernetizing the serial devices).
All the serial vacuum signals are now interfaced to the new digital controls system. A set of persistent Python scripts will query each device at regular intervals (up to ~10 Hz) and push the readings to soft channels hosted by the modbus IOC. Similar scripts will push on/off state commands to the serial turbo pumps.
Each serial device is assigned an IP address on the local subnet as follows. Its serial communication parameters as configured in the terminal server are also listed.
I've made a repair to the N2 pressure monitor. I don't believe the polarity of the analog signal into the controller actually was reversed. I found the data sheet (attached) for the transducer model we have installed. Its voltage should read ~0 at 0 PSI and 100mV at 100 PSI. As wired, the input voltage reads +80 mV as it should.
The controller calibrates the sensor voltage to PSI (i.e., applies a scale and offset) based on two settable reference points which appeared to be incorrect. I changed them to:
After the change, the pressure reads 80 PSI. Let's see if the time history now shows a sensible trend.
[koji, gautam, jon, steve]
Below is an inventory of the signal feedthroughs that need to be installed on the vacuum Acromag crate this week.
**The original documentation lists five satellite boxes (one for each test mass chamber and one for the beamsplitter chamber), but Chub reports not all of them are in use. We may remove the ones not used.
Based on new input from Chub, attached is the revised list of signal cable feedthroughs needed on the vacuum system Acromag crate. I believe this list is now complete.
In the latest installment in this puzzler: turns out that maybe the trend of the "N2 pressure" channel increasing over the ~3 day timescale it takes a cylinder of N2 to run out is real, and is a feature of the way our two N2 cylinder lines/regulators are setup (for the automatic switching between cylinders when one runs out). In order to test this hypothesis, we'd like to have the line pressure be 0 initially, and then just have 1 cylinder hooked up. When we went into the drill-press area, we heard a hiss, turns out that one of the cylinders is leaking (to be fair, this was labelled, but i thought it isn't great to have a higher N2 concentration in an enclosed space). Since we don't need any actuation ability, I valved off the leaky cylinder, and disconnected the other properly functioning one. Attachment #1 shows the current state.
I believe I finally have the N2 gauge working correctly. The wiring is unchanged from its original state and the controller has been recalibrated.
After letting the line pressure drop to 0 PSI as indicated by the analog gauge in the drill-press room, I recorded the number of counts read by the Omega controller. Then I pressurized the line to 80 PSI, again indicated by the analog gauge, and recorded the Omega counts again. I entered these two reference points into the controller (automatically determines the gain and offset from these), then confirmed the readings to agree with the anaog gauge as I varied the line pressure.
The two reference points are:
In the latest installment in this puzzler: turns out that maybe the trend of the "N2 pressure" channel increasing over the ~3 day timescale it takes a cylinder of N2 to run out is real, and is a feature of the way our two N2 cylinder lines/regulators are setup (for the automatic switching between cylinders when one runs out). In order to test this hypothesis, we'd like to have the line pressure be 0 initially, and then just have 1 cylinder hooked up.
There turned out to be a few analog signals for the vacuum system after all. The TP2/3 foreline pressure gauges were never part of the digital system, but we wanted to add them, as some of the interlock conditions should be predicated on their readings. Each gauge connects to an old Granville-Phillips 375 controller which only has an analog output. Interfacing these signals with the new system required installing an Acromag XT1221 8-channel A/D unit. Taking advantage of the extra channels, I also moved the N2 delivery line pressure transducer to the XT1221, eliminating the need for its separate Omega DPiS32 controller. When the new high-pressure transducers are added to the two N2 tanks, their signals can also be connected.
The XT1221 is mounted on the DIN rail inside the chassis and I have wired a DB-9 feedthrough for each of its three input signals. It is assigned the IP 192.168.114.27 on the vacuum subnet. Testing the channels in situ revealed a subtley in calibrating them to physical units. It was first encountered by Johannes in a series of older posts, but I repeat it here in one place.
An analog-input EPICS channel can be calibrated from raw ADC counts to physical units (e.g., sensor voltage) in two ways:
From the documentation, under the engineering-units method EPICS internally computes:
where EGUF="eng units full scale", EGUL="eng units low", and "full scale A/D counts" is the full range of ADC counts. EPICS automatically infers the range of ADC counts based on the data type returned by the ADC. For a 16-bit ADC like the XT1221, this number is 2^16 = 65,536.
The problem is that, for unknown reasons, the XT1221 rescales its values post-digitization to lie within the range +/-30,000 counts. This corresponds to an actual "full scale A/D counts" = 60,001. If a multiplicative correction factor of 65,536/60,000 is absorbed into the values of EGUF and EGUL, then the first term in the above summation can be corrected. However, the second term (the offset) has no dependence on "full scale A/D counts" and should NOT absorb a correction factor. Thus adjusting the EGUF and EGUL values from, e.g., 10V to 10.92V is only correct when EGUL=0V. Otherwise there is a bias introduced from the offset term also being rescaled.
The generally correct way to handle this correction is to use the manual "NO CONVERSION" method. It constructs calibrated values by simply applying a specified gain and offset to the raw ADC counts:
calibrated val = (measured A/D counts) x ASLO + AOFF
The gain ASLO="[(V_max_adc - V_min_adc) / 60,001]" and the offset AOFF="0". I have tested this on the three vacuum channels and confirmed it works. Note that if the XT1221 input voltage range is restricted from its widest +/-10V setting, the number of counts is not necessarily 60,001. Page 42 of the manual gives the correct counts for each voltage setting.
[Jon, Chub, Koji, Gautam]
Today we carried out the first pumpdown with the new vacuum controls system in place. It performed well. The only problem encountered was with software interlocks spuriously closing valves as the Pirani gauges crossed 1E-4 torr. At that point their readback changes from a number to "L OE-04, " which the system interpreted as a gauge failure instead of "<1E-4." This posed no danger and was fixed on the spot. The main volume was pumped to ~10 torr using roughing pumps 1 and 3. We were limited only by time, as we didn't get started pumping the main volume until after 1pm. The three turbo pumps were also run and tested in parallel, but were isolated to the pumpspool volume. At the end of the day, we closed every pneumatic valve and shut down all five pumps. The main volume is sealed off at ~10 torr, and the pumpspool volume is at ~1e-6 torr. We are leaving the system parked in this state for the holidays.
In pumping down the main volume, we carried out the following procedure.
We didn't quite reach the end of step 8 by the time we had to stop. The next step would be to valve out the roughing pumps and to valve in the turbo pumps.
All of the new hardware is now permanently installed in the vacuum rack. This includes the SuperMicro rack server (c1vac), the IOLAN serial device server, a vacuum subnet switch, and the Acromag chassis. Every valve/pump signal cable that formerly connected to the VME bus through terminal blocks has been refitted with a D-sub connector and screwed directly onto feedthroughs on the Acromag chassis.
The attached pdf contains the master list of assigned Acromag channels and their wiring.
Per the discussion yesterday, I valved off the N2 line in the drill press room at 11 am PST today morning so as to avoid any accidental software induced gate-valve actuation during the holidays. The line pressure is steadily dropping...
Attachment #1 shows that while the main volume pressure was stable overnight, the the pumpspool pressure has been steadily rising. I think this is to be expected as the turbo pumps aren't running and the valves can't preserve the <1mtorr pressure over long timescales?
Attachment #2 shows the current VacOverview MEDM screen status.
Independent question: Are all the turbo forelines vented automatically? We manually did it for the main roughing line.
Larry W came by the 40m, and reported that there was a campus-wide power glitch (he was here to check if our networking infrastructure was affected). I thought I'd check the status of the vacuum.
I decided to check the systemctl process status on c1vac:
controls@c1vac:~$ sudo systemctl status modbusIOC.service
● modbusIOC.service - ModbusIOC Service via procServ
Loaded: loaded (/etc/systemd/system/modbusIOC.service; enabled)
Active: active (running) since Thu 2019-01-03 14:53:49 PST; 11min ago
Main PID: 16533 (procServ)
├─16533 /usr/bin/procServ -f -L /opt/target/modbusIOC.log -p /run/...
├─16534 /opt/epics/modules/modbus/bin/linux-x86_64/modbusApp /opt/...
Jan 03 14:53:49 c1vac systemd: Started ModbusIOC Service via procServ.
Warning: Unit file changed on disk, 'systemctl daemon-reload' recommended.
So something did happen today that required restart of the modbus processes. But clearly not everything has come back up gracefully. A few lines of dmesg (there are many more segfaults):
[1706033.718061] python: segfault at 8 ip 000000000049b37d sp 00007fbae2b5fa10 error 4 in python2.7[400000+31d000]
[1706252.225984] python: segfault at 8 ip 000000000049b37d sp 00007fd3fa365a10 error 4 in python2.7[400000+31d000]
[1720961.451787] systemd-udevd: starting version 215
[1782064.269844] audit: type=1702 audit(1546540443.159:38): op=linkat ppid=21820 pid=22823 auid=4294967295 uid=1000 gid=1000 euid=1000 suid=1000 fsuid=1000 egid=1000 sgid=1000 fsgid=1000 tty=pts0 ses=4294967295 comm="git" exe="/usr/bin/git" res=0
[1782064.269866] audit: type=1302 audit(1546540443.159:39): item=0 name="/cvs/cds/caltech/target/c1vac/.git/objects/85/tmp_obj_uAXhPg" inode=173019272 dev=00:21 mode=0100444 ouid=1001 ogid=1001 rdev=00:00 nametype=NORMAL
[1782064.365240] audit: type=1702 audit(1546540443.255:40): op=linkat ppid=21820 pid=22823 auid=4294967295 uid=1000 gid=1000 euid=1000 suid=1000 fsuid=1000 egid=1000 sgid=1000 fsgid=1000 tty=pts0 ses=4294967295 comm="git" exe="/usr/bin/git" res=0
[1782064.365271] audit: type=1302 audit(1546540443.255:41): item=0 name="/cvs/cds/caltech/target/c1vac/.git/objects/58/tmp_obj_KekHsn" inode=173019274 dev=00:21 mode=0100444 ouid=1001 ogid=1001 rdev=00:00 nametype=NORMAL
[1782064.460620] audit: type=1702 audit(1546540443.347:42): op=linkat ppid=21820 pid=22823 auid=4294967295 uid=1000 gid=1000 euid=1000 suid=1000 fsuid=1000 egid=1000 sgid=1000 fsgid=1000 tty=pts0 ses=4294967295 comm="git" exe="/usr/bin/git" res=0
[1782064.460652] audit: type=1302 audit(1546540443.347:43): item=0 name="/cvs/cds/caltech/target/c1vac/.git/objects/cb/tmp_obj_q62Pdr" inode=173019276 dev=00:21 mode=0100444 ouid=1001 ogid=1001 rdev=00:00 nametype=NORMAL
[1782064.545449] audit: type=1702 audit(1546540443.435:44): op=linkat ppid=21820 pid=22823 auid=4294967295 uid=1000 gid=1000 euid=1000 suid=1000 fsuid=1000 egid=1000 sgid=1000 fsgid=1000 tty=pts0 ses=4294967295 comm="git" exe="/usr/bin/git" res=0
[1782064.545480] audit: type=1302 audit(1546540443.435:45): item=0 name="/cvs/cds/caltech/target/c1vac/.git/objects/e3/tmp_obj_gPI4qy" inode=173019277 dev=00:21 mode=0100444 ouid=1001 ogid=1001 rdev=00:00 nametype=NORMAL
[1782064.640756] audit: type=1702 audit(1546540443.527:46): op=linkat ppid=21820 pid=22823 auid=4294967295 uid=1000 gid=1000 euid=1000 suid=1000 fsuid=1000 egid=1000 sgid=1000 fsgid=1000 tty=pts0 ses=4294967295 comm="git" exe="/usr/bin/git" res=0
[1783440.878997] systemd: Unit serial_TP3.service entered failed state.
[1784682.147280] systemd: Unit serial_TP2.service entered failed state.
[1786407.752386] systemd: Unit serial_MKS937b.service entered failed state.
[1792371.508317] systemd: serial_GP316a.service failed to run 'start' task: No such file or directory
[1795550.281623] systemd: Unit serial_GP316b.service entered failed state.
[1796216.213269] systemd: Unit serial_TP3.service entered failed state.
[1796518.976841] systemd: Unit serial_GP307.service entered failed state.
[1796670.328649] systemd: serial_Hornet.service failed to run 'start' task: No such file or directory
[1797723.446084] systemd: Unit serial_MKS937b.service entered failed state.
I don't know enough about the new system so I'm leaving this for Jon to debug. Attachment #3 shows that the analog readout of the P1 pressure gauge suggests that the IFO is still under vacuum, so no random valve openings were effected (as expected, since we valved off the N2 line for this very purpose).
Yes, for TP2 and TP3. They both have a small vent valve that opens automatically on shutdown.
The base Acromag vacuum system is running and performing nicely. Here is a list of remaining questions and to-do items we still need to address.
I've attached my handwritten notes covering all the serial communications in the vac system, and the relevant wiring for all the adapters, etc. I'll work with Chub to produce a final documentation, but in the meantime this may be a useful reference.
Updated vac channel list is attached. There are several new ADC channels.
Hardware & Channel Assignments