40m QIL Cryo_Lab CTN SUS_Lab TCS_Lab OMC_Lab CRIME_Lab FEA ENG_Labs OptContFac Mariner WBEEShop
  40m Log  Not logged in ELOG logo
Entry  Mon Oct 15 08:12:45 2012, Steve, Update, VAC, blank vac monitor blankvac.png
    Reply  Mon Oct 15 13:07:51 2012, jamie, Update, VAC, vacuum VME crate broken, replaced, minor vacuum mayhem ensues 
       Reply  Mon Oct 15 14:51:16 2012, Jenne, Update, SUS, SUS were kicked hard as a result 
          Reply  Mon Oct 15 17:08:01 2012, Steve, Update, VAC, vacuum is back with some disabled valves VMEcrateout.pngIMG_1725.JPGwarning.pngshortcomings.jpg
             Reply  Wed Oct 24 09:15:12 2012, Steve, Update, VAC, vacuum rack 24V ps was current limited currentlimited24V.jpg
                Reply  Mon Nov 5 10:34:48 2012, Steve, Update, VAC, existing vacuum documents 
                Reply  Fri Aug 16 09:35:18 2013, Steve, Update, VAC, Vacuum Normal state recognition is back VacuumNormal.pngvacValvesDisabled.jpg
Message ID: 7547     Entry time: Mon Oct 15 13:07:51 2012     In reply to: 7545     Reply to this: 7548
Author: jamie 
Type: Update 
Category: VAC 
Subject: vacuum VME crate broken, replaced, minor vacuum mayhem ensues 

Steve and I managed to access the fuse in the vacuum VME crate, but replacing it did not bring it back up.  We decided to replace the entire crate.

We manually checked that the most important valves, VC1, VM1 and V1, were all closed.  We disconnected their power so that they would automatically close, and we wouldn't have to worry about them accidentally opening when we rebooted the system.

We noted where all the cables were, disconnected everything, and removed the crate.  We noted that one of the values switched when we disconnected one of the IPC cables from a VME card.  We'll note which one it was in a followup post.  We thought that was a little strange, since the VME crate was completely unpowered.

Anyway, we removed the crate, swapped in a spare, replaced all the cards and connections, double checked everything, then powered up the crate.  That's when minor chaos ensued.

When the system came back online after about 20 seconds, we heard a whole bunch of valves switching.  Luckily we were able to get the medm screens back up so that we could see what was going on.

Apparently all of the ION pump valves (VIPEE, VIPEV, VIPSV, VIPSE) opened, which vented the main volume up to 62 mTorr.  All of the annulus valves (VAVSE, VAVSV, VAVBS, VAVEV, VAVEE) also appeared to be open.  One of the roughing pumps was also turned on.  Other stuff we didn't notice?  Bad.

We ran around and manually unplugged all of the ION pump valves, since I couldn't immediately pull up the vacuum control screen.  Once that was done and we could see that the main volume was closed off we went back to figure out what was going on.

We got the medm vacuum control screen back (/cvs/cds/caltech/medm/c0/ve/VacControl_BAK.adl.  really??)  There was a lot of inconsistency between the readback states of the valves and the switch settings.  Toggling the switches seemed to bring things back in line.  At this point it seemed that we had control of the system again.  The epics readings were consistent with what we were seeing in the vacuum rack.

We went through and closed everything that should have been closed.  The line pressure between the big turbo pump TP1 and the rest of the pumps was up at atmosphere, 700 Torr.  We connected the roughing pumps and pumped down the lines so that we could turn the turbos back on.  Once TP2 and TP3 were up to speed, we turned on TP1 and opened V1 to start pumping the main volume back town.   The main volume is at 7e-4 Torr right now.

 

So there are a couple of problems with the vacuum system.

  • Why the hell did valves open when we rebooted the VME crate?  That's very bad.  That should never happen.  If the system is set to come up to an unsafe state that needs to be fixed ASAP.  The ION pump valves should never have opened.  Nor the annulus valves.
  • Why were the switches and the readbacks showing different states?
  • Apparently there is no control of the turbo pumps through MEDM.  This should be fixed.

I connect belledona, the laptop at the vacuum station to the wired network, so that it's connection would be less flaky.

 

ELOG V3.1.3-