40m QIL Cryo_Lab CTN SUS_Lab TCS_Lab OMC_Lab CRIME_Lab FEA ENG_Labs OptContFac Mariner WBEEShop
  40m Log  Not logged in ELOG logo
Entry  Tue Apr 2 18:40:01 2019, gautam, Update, VAC, Vac failure vacFail.png
    Reply  Wed Apr 3 09:07:46 2019, gautam, Update, VAC, Vac failure vacFailOvernight.png
       Reply  Wed Apr 3 10:42:36 2019, gautam, Update, VAC, TP2 forepump replaced 
          Reply  Wed Apr 3 16:17:17 2019, Jon, Update, VAC, TP2 forepump replaced IMG_3180.jpg
             Reply  Wed Apr 3 18:35:54 2019, gautam, Update, VAC, PSL shutter re-opened 
                Reply  Fri Apr 5 01:10:18 2019, gautam, Update, VAC, TP3 forepump is also noisy 
             Reply  Tue Apr 16 22:06:51 2019, gautam, Update, VAC, Vac interlock tripped again Screenshot_from_2019-04-16_22-05-47.pngScreenshot_from_2019-04-16_22-06-02.png
                Reply  Wed Apr 17 18:12:06 2019, gautam, Update, VAC, Vac interlock tripped again Screenshot_from_2019-04-17_18-11-45.pngScreenshot_from_2019-04-17_18-21-30.png
Message ID: 14514     Entry time: Wed Apr 3 16:17:17 2019     In reply to: 14512     Reply to this: 14515   14546
Author: Jon 
Type: Update 
Category: VAC 
Subject: TP2 forepump replaced 

I can't explain the mechanical switching sound Gautam reported. The relay controlling power to the TP2 forepump is housed in the main AC relay box under the arm tube, not in the Acromag chassis, so it can't be from that. I've cycled through the pumpdown sequence several times and can't reproduce the effect. The Acromag switches for TP2 still work fine.

In any case, I've made modifications to the vacuum interlocks that will help with two of the issues:

  1. For the "AC power loss" over-triggering: New logic added requiring the UPS to be out of the "on line power, battery OK" state for ~5 seconds before tripping the interlock. This will prevent electrical transients from triggering an emergency shutdown, as seems to be the case here (the UPS briefly isolates the load to battery during such events).
  2. PSL interlocking: New logic added which directly sets C1:AUX-PSL_ShutterRqst --> 0 (closes the PSL shutter) when the main volume pressure is 3 mtorr-500 torr. Previously there was a channel exposed for this interlock (C1:Vac-interlock_high_voltage), but c1aux was not actually monitoring it. Following the convention of every vac interlock, after the PSL shutter has been closed, it has to be manually reopened. Once the pressure is out of this range, the vac system will stop blocking the shutter from reopening, but it will not perform the reopen action itself. gautam: a separate interlock logic needs to be implemented on c1aux (the shutter machine) that only permits the shutter to be opened if the Vac pressure range is okay. The SUS watchdog style AND logic in the EPICS database file should work just fine.

After finishing this vac work, I began a new pumpdown at ~4:30pm. The pressure fell quickly and has already reached ~1e-5 torr. TP2 current and temp look fine.

Quote:

However, when opening V4 (the foreline of TP1 pumped by TP2), I heard a loud repeated click track (~5Hz) from the electronics rack. Shortly after, the interlocks shut down all the TPs again, citing "AC power loss". Something is not right, I leave it to Jon and Chub to investigate.

Attachment 1: IMG_3180.jpg  4.971 MB  Uploaded Wed Apr 3 17:58:20 2019  | Hide | Hide all
IMG_3180.jpg
ELOG V3.1.3-