Apparently all of the ION pump valves (VIPEE, VIPEV, VIPSV, VIPSE) opened, which vented the main volume up to 62 mTorr. All of the annulus valves (VAVSE, VAVSV, VAVBS, VAVEV, VAVEE) also appeared to be open. One of the roughing pumps was also turned on. Other stuff we didn't notice? Bad.
Several of the suspensions were kicked pretty hard (600+ mV on some sensors) as a result of this quick vent wind. All of the suspensions are damped now, so it doesn't look like we suffered any damage to suspensions.
CLOSE CALL on the vacuum system:
Jamie and I disabled V1, VM2 and VM3 gate valves by disconnecting their 120V solenoid actuator before the swap of the VME crate.
The vacuum controller unexpectedly lost control over the swap as Jamie described it. We were lucky not to do any damage! The ion pumps were cold and clean. We have not used them for years so their outgassing possibly accumulated to reach ~10-50 Torr
I disconnected_ immobilized and labelled the following 6 valves: the 4 large ion pump gate valves and VC1, VC2 of the cryo pump. Note: the valves on the cryo pump stayed closed. It is crucial that a warm cry pump is kept closed!
This will not allow the same thing to happen again and protect the IFO from warm cryo contamination.
The down side of this that the computer can not identify vacuum states any longer.
This vacuum system badly needs an upgrade. I will make a list.
While I was doing the oil change of the roughing pumps I accidentally touched the 24 V adjustment knob on the power supply.
All valve closed to default condition. I realized that the current indicator was red at 0.2A and the voltage fluctuated from 3-13V
Increased current limiter to 0.4A and set voltage to 24V I think this was the reason for the caos of valve switching during the VME swap.
I asked Jamie to read these docs so we can do some troubleshooting while at atmosphere. We have to have the all valve closed DEFAULT condition working-tested.
I'd like to reconnect all valves after this test so the state identification software could work, including interlocks.
Posted the following documents on 40m wiki.