40m QIL Cryo_Lab CTN SUS_Lab TCS_Lab OMC_Lab CRIME_Lab FEA ENG_Labs OptContFac Mariner WBEEShop
  40m Log  Not logged in ELOG logo
Entry  Thu Jun 11 11:48:43 2020, gautam, Update, VAC, Vac failure vacFailure.png
    Reply  Thu Jun 11 16:14:03 2020, gautam, Update, VAC, Vac failure - probable cause is serial comm glitch TP2_tempGlitch.pngPSL_shutterClosed.pngTP2tempGlitches.pdf
       Reply  Thu Jun 11 17:35:34 2020, gautam, Update, VAC, Pumpspool UPS needs battery replacement 
          Reply  Fri Jun 12 01:23:32 2020, Koji, Update, VAC, Pumpspool UPS needs battery replacement 
             Reply  Fri Jun 12 17:32:40 2020, Koji, Update, VAC, Pumpspool UPS needs battery replacement Screen_Shot_2020-06-12_at_17.22.23.png
                Reply  Fri Jun 12 19:02:52 2020, gautam, Update, VAC, Pumpspool UPS needs battery replacement 
                   Reply  Fri Jun 12 19:23:56 2020, Koji, Update, VAC, Pumpspool UPS needs battery replacement 
                      Reply  Fri Jun 12 19:33:31 2020, gautam, Update, VAC, Pumpspool UPS needs battery replacement 
                         Reply  Tue Jun 16 13:35:03 2020, Jon, Update, VAC, Temporary vac fix / IFO usable again Pumpdown.png
                            Reply  Wed Jun 17 16:27:51 2020, gautam, Update, VAC, Questions/comments on vacuum 
                               Reply  Thu Jun 18 11:00:24 2020, Jon, Update, VAC, Questions/comments on vacuum 
                                  Reply  Thu Jun 18 12:00:36 2020, gautam, Update, VAC, Questions/comments on vacuum 
                                     Reply  Thu Jun 18 14:13:03 2020, Jon, Update, VAC, Questions/comments on vacuum 
                                        Reply  Thu Jun 18 15:46:34 2020, gautam, Update, VAC, Questions/comments on vacuum 
                                           Reply  Fri Jun 19 07:40:49 2020, Jon, Update, VAC, Questions/comments on vacuum 
                                              Reply  Fri Jun 19 09:57:35 2020, gautam, Update, VAC, Questions/comments on vacuum 
Message ID: 15392     Entry time: Thu Jun 11 16:14:03 2020     In reply to: 15391     Reply to this: 15393
Author: gautam 
Type: Update 
Category: VAC 
Subject: Vac failure - probable cause is serial comm glitch 

Summary:

It looks like the main vacuum interlock was tripped due to a serial communication error from the TP2 controller. With Rana/Koji's permission, I will open V1 and expose the main volume to TP1 again (#2 in last section).

Details:

  • The vacuum interlock log file at /opt/target/vac.log on c1vac suggests that the interlock was tripped because "TP2 is too warm".
  • Looking back at the diagnostics channels, it looks like the TP2 temperature channel registered a rise in temperature of >30 C in <0.2 seconds, see Attachment #1 - seems highly unlikely, probably some kind of glitch in the serial communication? This particular pump is relatively new from Agilent (<2 years installed I think)
  • The PSL shutter was automatically closed at ~1150 am today, see Attachment #2. There is some EPICS logic on c1psl (Acromag server) that checks if C1:Vac-P1a_pressure is greater than 3 mTorr (or greater than 500 Torr for in-air locking of the IMC), in which case it closes the shutter, so this seems consistent with expectations.

Recommended course of action:

  1. Code in some averaging in the interlock code, so that the interlock isn't triggered on some unphysical glitch like this. As shown in Attachment #3, this has been happening for the past 24 hours (though not before, because the interlock wasn't tripped). Probably need the derivative of the temperature as well, and the derivative should be less than 5 C/s or something physical (in addition to the temperature being high) for the interlock to trip.
  2. Re-open V1 to pump down the main volume to nominal pressure so that the interferometer locking activity can resume.
    • One option in the interim is to bypass the TP2 temperature interlock condition.
    • The pressure-based interlocks are probably sufficient to protect the main volume / pumps during the nominal operations - the temperature interlocks are mainly useful during the pumpdown where the TPs have a large load, and so we want to avoid over-stressing them.
Attachment 1: TP2_tempGlitch.png  161 kB  Uploaded Thu Jun 11 17:32:50 2020  | Hide | Hide all
TP2_tempGlitch.png
Attachment 2: PSL_shutterClosed.png  78 kB  Uploaded Thu Jun 11 17:35:02 2020  | Hide | Hide all
PSL_shutterClosed.png
Attachment 3: TP2tempGlitches.pdf  2.153 MB  Uploaded Thu Jun 11 17:47:33 2020  | Hide | Hide all
TP2tempGlitches.pdf
ELOG V3.1.3-