40m QIL Cryo_Lab CTN SUS_Lab TCS_Lab OMC_Lab CRIME_Lab FEA ENG_Labs OptContFac Mariner WBEEShop
  40m Log  Not logged in ELOG logo
Entry  Mon Apr 19 09:17:51 2021, Jordan, Update, VAC, Empty N2 Tanks N2_Pressure.PNG
    Reply  Wed Apr 21 12:56:00 2021, Jordan, Update, VAC, Empty N2 Tanks Screenshot_2021-04-21_12-53-26.png
       Reply  Wed Apr 28 11:31:40 2021, Jon, Update, VAC, Empty N2 Tanks 
Message ID: 16090     Entry time: Wed Apr 28 11:31:40 2021     In reply to: 16064
Author: Jon 
Type: Update 
Category: VAC 
Subject: Empty N2 Tanks 

I checked out what happened on c1vac. There are actually two independent monitoring codes running:

  1. The interlock service, which monitors the line directly connected to the valves.
  2. A seaparate convenience mailer, running as a cronjob, that monitors the tanks.

The interlocks did not trip because the low-pressure delivery line, downstream of the dual-tank regulator, never fell below the minimum pressure to operate the valves (65 PSI). This would have eventually occurred, had Jordan been slower to replace the tanks. So I see no problem with the interlocks.

On the other hand, the N2 mailer should have sent an email at 2021-04-18 15:00, which was the first time C1:Vac-N2T1_pressure dropped below the 600 PSI threshold. N2check.log shows these pressures were recorded at this time, but does not log that an email was sent. Why did this fail? Not sure, but I found two problems which I did fix:

  • One was that the code was checking the sensor on the low-pressure side (C1:Vac-N2_pressure; nominally 75 PSI) against the same 600 PSI threshold as the tanks. This channel should either be removed or a separate threshold (65 PSI) defined just for it. I just removed it from the list because monitoring of this channel is redundant with the interlock service. This does not explain the failure to send an email.
  • The second issue was that the pyN2check.sh script appeared to be calling Python 3 to run a Python 2 script. At least that was the situation when I tested it, and this was causing it to fail partway through. This might well explain the problem with no email. I explicitly set python --> python2 in the shell script.

The code then ran fine for me when I retested it. I don't see any further issues.

Quote:

Installed T2 today, and leaked checked the entire line. No issues found. It could have been a bad valve on the tank itself. Monitored T2 pressure for ~2 hours to see if there was any change. All seems ok.

Quote:

When I came into the lab this morning, I noticed that both N2 tanks were empty. I had swapped one on Friday (4-16-21) before I left the lab. Looking at the logs, the right tank (T2) sprung a leak shortly shortly after install. I leak checked the tank coupling after install but did not see a leak. There could a leak further down the line, possibly at the pressure transducer.

The left tank (T1) emptied normally over the weekend, and I quickly swapped the left tank for a full one, and is curently at ~2700 psi. It was my understanding that if both tanks emptied, V1 would close automatically and a mailer would be sent out to the 40m group. I did not receive an email over the weekend, and I checked the Vac status just now and V1 was still open.

I will keep an eye on the tank pressure throughout the day, and will try to leak check the T2 line this afternoon, but someone should check the vacuum interlocks and verify.

 

ELOG V3.1.3-