Steve and I managed to access the fuse in the vacuum VME crate, but replacing it did not bring it back up.  We decided to replace the entire crate.

We manually checked that the most important valves, VC1, VM1 and V1, were all closed.  We disconnected their power so that they would automatically close, and we wouldn't have to worry about them accidentally opening when we rebooted the system.

We noted where all the cables were, disconnected everything, and removed the crate.  We noted that one of the values switched when we disconnected one of the IPC cables from a VME card.  We'll note which one it was in a followup post.  We thought that was a little strange, since the VME crate was completely unpowered.

Anyway, we removed the crate, swapped in a spare, replaced all the cards and connections, double checked everything, then powered up the crate.  That's when minor chaos ensued.

When the system came back online after about 20 seconds, we heard a whole bunch of valves switching.  Luckily we were able to get the medm screens back up so that we could see what was going on.

Apparently all of the ION pump valves (VIPEE, VIPEV, VIPSV, VIPSE) opened, which vented the main volume up to 62 mTorr.  All of the annulus valves (VAVSE, VAVSV, VAVBS, VAVEV, VAVEE) also appeared to be open.  One of the roughing pumps was also turned on.  Other stuff we didn't notice?  Bad.

We ran around and manually unplugged all of the ION pump valves, since I couldn't immediately pull up the vacuum control screen.  Once that was done and we could see that the main volume was closed off we went back to figure out what was going on.

We got the medm vacuum control screen back (/cvs/cds/caltech/medm/c0/ve/VacControl_BAK.adl.  really??)  There was a lot of inconsistency between the readback states of the valves and the switch settings.  Toggling the switches seemed to bring things back in line.  At this point it seemed that we had control of the system again.  The epics readings were consistent with what we were seeing in the vacuum rack.

We went through and closed everything that should have been closed.  The line pressure between the big turbo pump TP1 and the rest of the pumps was up at atmosphere, 700 Torr.  We connected the roughing pumps and pumped down the lines so that we could turn the turbos back on.  Once TP2 and TP3 were up to speed, we turned on TP1 and opened V1 to start pumping the main volume back town.   The main volume is at 7e-4 Torr right now.

So there are a couple of problems with the vacuum system.

• Why the hell did valves open when we rebooted the VME crate?  That's very bad.  That should never happen.  If the system is set to come up to an unsafe state that needs to be fixed ASAP.  The ION pump valves should never have opened.  Nor the annulus valves.
• Why were the switches and the readbacks showing different states?
• Apparently there is no control of the turbo pumps through MEDM.  This should be fixed.

I connect belledona, the laptop at the vacuum station to the wired network, so that it's connection would be less flaky.