TP3 dry pump  replaced at 540 mT as TP3 50K_rpm 0.3A with annulos load. It's top seal life time was 11,252 hrs

We run out of N2 for the vacuum system. The pressure peaked at 1.3 mTorr with MC locked. V1 did not closed because the N2 pressure sensor failed.

We are back to vac normal. I will be here tomorrow to check on things.

 ITMX damping restored.

[Koji, Steve]

Summary

The N₂ pressure reading (C1:VAC-N2PRES) is now up-to-date after rebooting c1vac1.
The vaccum system is "Vacuum normal". We now have a space pressure transducer.

Introduction

Our vacuum valves are manipulated with 60~75 PSI of nitrogen. All the valves are configured to be closed in the case of low N₂ supply pressure.
In order to avoid this safety shutdown accidentally triggered, we have two N₂ cylinders to sustain the vacuum valves. When one cylinder goes to low
the mechanical valve switches over to the other cylinder.

We have the monitor channel for this (combined) cylinder pressure. One shoulbe be able to see periodical pressure variation when the auto cylinder
switch is operating. However, the nirogen pressure reading got stuck at 66 PSI on Dec.16, 2014 (See attached 60-day plot of N2 supply pressure). 

What we did

This morning we tracked down the cause of the trouble. We first closed the valves on EPICS and started to vary the N2 pressure.

Our first guess was the pressure transducer (Omega #236PC100GW) that was already 15 yrs old. We even has a sensor spare for replacement.
But it turned out that the direct voltage reading (to be 1mV/PSI) is changing correctly. The second guess was Omega Controller-Monitor
#DPiS32-C24 that is reading the voltage from the tranceducer. The display on this small black unit was changing corresponding to the
pressure change.

So our thought was
1) RS232C of the monitor unit is not working correctly
or
2) c1vac1 is not communicating with the monitor unit.

We wondered what could cause c1vac1 not communicating with the monitor unit, but we were afraid that some function got stuck
during either the nodus upgrade or chiara rebooting (or something else). So we decided to reboot c1vac1

In order to avoid any glitch in the main vacuum pressure, Steve disconnected some of the controller connectors for the closed valves.
We did this treatment before and it was successful.

Then c1vac1 was rebooted just by telnet and type reboot in the terminal.
Once the target is back in action, we noticed that the monitor value started to move.

Steve reverted the cables to the valves and operated the valves to recover "Vacuum Normal" state. Everything is now nicely settled.

We run out of N2 for the vacuum system 6 hrs ago. The pressure rose to 1.2 mTorr with V1 closed. The interlock worked! See Nirogen presure reading fixed at http://nodus.ligo.caltech.edu:8080/40m/10968

The vacuum interlock: Nitrogen pressure transducer is reading the pneumatic pressure continously at the pump spool and c1vac1 processing it. When it drops below 60 PSI it closes V1 gate valve and V4 & V5.  Gate valve V1 needs minimum 60 PSI to close. It is critical that V1 is closed before you run out of Nitrogen so the IFO pressure is contained.

IFO vacuum is back to Vac Normal. The MC is locked.

cc4 = 2E-6 Torr with VM1 open.

Daily N2 consumption measured to be 530PSI as 3 days on 3-27-2015 but note: it does vary !

I have seen it as high as 900 psi  The long term average ~750 psi

Based on Jenne's chiara disk usage monitoring script, I made a script that checks the N2 pressure, which will send an email to myself, Jenne, Rana, Koji, and Steve, should the pressure fall below 60psi. I also updated the chiara disk checking script to work on the new Nodus setup. I tested the two, only emailing myself, and they appear to work as expected. 

The scripts are committed to the svn. Nodus' crontab now includes these two scripts, as well as the crontab backup script. (It occurs to me that the crontab backup script could be a little smarter, only backing it up if a change is made, but the archive is only a few MB, so it's probably not so important...)

Batteries replaced after 3.5 years with Amstron AP-1250F2,  8x 12V 6Ah

The script checking the N2 pressure is not working.  I signed into the foteee account to look at some of the picasa photos, and there are thousands of emails (one every 10 minutes for the past month!) with error messages.  Q, can you please make it stop (having errors)?

The error looks like it's mad about a "caget" command.  I don't have time to investigate further though.

Ugh, this turns out to be because cron doesn't source the controls bashrc that defines where to find caget and all that jazz that many commands depend on. This is probably also why the AutoMX cron job isn't working either. 

Also, cron automatically emails everything from stderr to the email address that is configured for the user, which is why the n2 script blew up the foteee account and why the AutoMX script was blowing up my email yesterday. This can be avoided by doing something like this in the crontab:

0 8 * * * /bin/somecommand >> somefile.log 2>&1

(The >> part means that the standard output is appended to some log file, while the 2>&1 means send the standard error stream to the same place as stdout)

I made this change for the n2 script, so the foteee email account should be safe from this script. I haven't figured out the right way to set up cron to have all the right $PATH and other environment stuff, such as epics may need, so the script is still not working. 

This watch script gives little time to replace N2 cylinder. When the regulated supply drops below 60 psi the cylinder pressure is 60 psi too.

It is more of a statement that V1 is closed and act accordingly. It's only practical if you are in the lab.

Rana pointed it out correctly that we need this message 24 hrs before it happens. This requires monitoring  of total supplies , not the regulated one.

So we need pressure transducers on each nitrogen cylinder, before the regulator. The sum of the two N2 cylinder when they are full 4000 - 4500 psi

The first email should be send out at 1000 psi as sum of the two cylinders. This means that you have  1 day to replace nitrogen cylinder.

 Most of the time the daily consumption is 750 +-50 psi   

However sometimes  this variation goes up   ~750 +-150 psi

We want to have a VAC page in the summaries, so Steve - please put a list of important channel names for the vacuum system into the elog so that we can start monitoring for trouble.

Also, anyone that has any ideas can feel free to just add a comment to the summary pages DisQus comment section with the 40m shared account or make your own account.

We have 2 transduser PX303-3KG5V   http://www.omega.com/pressure/pdf/PX303.pdf They  will be installed on the out put of the N2 cylinders to read the supply pressure.

I will order one DC power supply  http://www.omega.com/pptst/PSU93_FPW15.html      PSU-93

One full cylinder pressure is ~ 2400 PSI max so two of them will give us ~9Vdc

The email reminder should be send at 1000 PSI  =  1.8 V

As it was requested by the Bos.

It would be nice to read from the epic screen C1:Vac-state_mon.......Current State: Vacuum Normal, valve configuration

CC1s  are not reading any longer. It is an attempt to clean them over the weekend at 85C

These brand new gauges "10421002" sn 11823-vertical, sn 11837 horizontal replaced 11 years old 421 http://nsei.missouri.edu/manuals/hps-mks/421%20Cold%20Cathode%20Ionization%20Guage.pdf  on 09-06-2012 http://nodus.ligo.caltech.edu:8080/40m/7441

Baking both CC1 at 85 C for 60 hrs did not help.

The temperature is increased to 125 C and it is being repeated.

Vacuum Operation Guide is up loaded into the 40m-wiki. This is an old master copy. Not exact in terms of real action, but it is still a good guide of logic.

Rana has promissed to watch the N2 supply and change cylinder when it is empty. I will be Hanford next week.

Koji and Steve succeded rebooting C1vac1, C1vac2 and pressure reading is working now

More tomorrow .........

The serial connections to the vacuum gauges were recovered by rebooting c1vac1 and c1vac2.

Steve claimed that the vacuum screen had showed "NO COMM" at the vacuum pressure values.
The epics connection to c1vac was fine. We could logged in to c1vac1 with telnet too although c1vac2 had no response.

After some inspection, we decided to reboot the slow machines. Steve manually XXXed YYY valves (to be described)
to prepare for any possible unwanted switching. Initially Koji thought only c1vac2 can be rebooted. But it was wrong.
If the reset button is pushed, all of the modules on the same crate is reset. So everything was reset. After ~3min we still
don't have the connection to c1vac1 restored. We decided to another reboot. This time I pushed c1vac1 reset button.
After waiting about two minutes, the ADCs started to show green lights and the switch box started scanning.
We recovered the telnet connection to c1vac1 and epics functions. c1vac2 is still note responding to telnet, and
the values associated with c1vac2 are still blank.

Steve restored the valves and everything was back to normal.

Koji and Steve,

One computer expert and one vacuum expert required.

Atm 1,  problem condition: gauges are not reading for a week, error message "NO COMM" and all computer LEDs are green

Atm 2, prepare to safe reboot:

            a, close V1, disconnect it's power cable and turn off Maglev, wait till rotation stops

            b, close PSL shutter ( take adrenaline if needed )

            c, close V4, V5, VA6 valves and disconnect their cables. "Moving" error message indicating this condition.

               V1 is not showing "Moving" because its power cable disconnected only! It will show it if its position indicator cable is disconnected too. There is no need for that.

               These valves closed and disabled will not allow accidental venting of main volume.

            d, push reset, reseting c1vac2 will reset c1vac1 also, wait ~ 6 minutes

"Vacuum Normal" valve configuration was restored after succesful reboot as follows:

             a, reconnect cable and open V4 and V5 at P2 & P3 <1e-1 Torr

             b, observe that P2  < 1e-3 Torr and retsart Maglev

             c, wait till Maglev reaches full speed of 560 Hz and reconnect-open V1

             d, reconnect-open VA6 at P3 <1e-3 Torr

NOTE: VM1 valve was locked in open position and it was not responding before and after reboot

          Error message on Atm2 is indicating this locked condition: "opening VM1 will vent IFO"

          This is a fauls message. The valve is frozen in open position. We need a softwear expert help.

1. Yesterday we did put the vacuum system into safe to reboot mode  for the ethernet card swap.
2. Atm1, IFO pressure
3. Atm2, as prepared valve configuration where " moving " means closed and disconnected

TP2 dry fore pump sn PLE10082 was replaced at pressure 717 mTorr,  TP2 50K rpm 0.33A @ 112,677 hrs

Top seal life was 8,160 hrs

Model  SH110, Sn LP1007L556 was installed. It's fore line pressure after 30 minutes of running 38 mTorr, TP2 turbo at 50K rpm 0.18A

CC4 cold cathode gauge jump triggered interlock to close VM1 valve to protect the RGA.

The IFO pressure is 1e-5 Torr

Vac normal was recovered by opening VM1

The cold cathode gauge is back to normal. cc4 is the last gauge is "functioning"

MKS is not responding. The spare controller and gauges are back for repair.

The IFO pressure is estimated ~1E-6 Torr with modified Vac normal valve configuration.

CC4 is in a jumping mode between 2e-5 and 1e-6 Torr

 Pressure based interlock kicks in to close VM1   at 2e-5 Torr to protect the RGA.

I did open VM1 repeatedly in the last few mornings but as cc4 jumps VM1 closes.

As VM1  closed to RGA scans are not seeing the IFO. I will look at some scans on Monday.

Mean while I opened VM2 to lower the pressure for the RGA. This change will be read by "Current Status: Undefined State"

So do not panic, the IFO pressure is normal.

I need someone's help to raise the interlock threshold to 5e-5 Torr

I'm buying a new cold cathode gauge on Monday.

note: cc1 is out of order!

         just read P1  the pressure is < 7e-4 Torr  This gauge is very reliable and it is at the low end of it's range.

RGA background scan.

The IFO is closed off from RGA with VM1

CC4 ( and CC1)  is still flaky and it's interlock closes VM1

At 10:02AM, the N2 Pressure fell below 60 PSI. The watch script saw this happen, but I did not recieve the email it is supposed to send 

C1:Vac-P1_pressure reads 7e-4, which is the same as it has for the past ~2 days, so the V1 interlock worked fine. 

I've put some fresh N2 into the system, and Bob will pop in over the weekend to check it. I'll stay on top of it until Steve gets back. 

After consulting ELOGs and the 40m wiki, I reasoned it was ok to open the V1 to reconnect the turbo pump to the main IFO volume and VM1 to reconnect the RGA, and have now done so. 

Vacuum normal configuration with VM1-closed, VM2-open valve positions. Power load normal 24V 0.2A

Maglev rotation 560 Hz at room temp body temp.

"NO COMM" error message on medm screen. Gauge controller pressures are read able.

All vac comp LEDs are green. We have to reboot on Monday to enable communication.

Steve pointed out that in the aftermath of the Nitrogen running out a couple of times last week, the RGA had shut itself off thinking that there was a leak and so it was not performing the scheduled scans once a day. So the data files from the scheduled scans were empty in the /opt/rtcds/caltech/c1/scripts/RGA/logs directory. The wiki page for getting it up and running again is up-to-date, but the script RGAset.py did not exist on the c0rga machine, which the RGA is communicating with via serial port. I copied over the script RGAset.py from rossa to c0rga and ran the script on that machine - but the error flags it returned were not all 0 (indicating some error according to the manual) - so I edited the script to send just the initialize command ('IN0') and commented out the other commands, after which I got error flags which were all 0. After this, I ran a manual scan using 'RGAlogger.py', and it appears that the RGA is now able to take scans again - I'm attaching a plot of the scan results. We've saved this scan as a reference to compare against after a few days. 

[ericq, Gautam, Steve]

Following roughly the same procedure as ELOG 11354, c1vac1 and c1vac2 were rebooted. The symptoms were identical to the situation in that ELOG; c1vac1 could be pinged and telneted to, but c1vac2 was totally unresponsive. 

The only change in the linked procedure was that we did not shut down the maglev. Since I unwittingly had it running for days without V4 open while Steve was away, we now know that it can handle shorter periods of time than that...

Upon reboot, many channels were readable again, unfortunately the channels for TP2 and TP3 are still blank. We were able to return to "Vacuum normal state," but because of unknowned communication problems with VM1's interlock, we can't open VM1 for the RGA. Instead we opened VM2 to expose the RGA to the main IFO volumn, but this isn't part of the "Normal" state definite, so things currently read "Undefined state".

Precondition:

1, Pressure gauges had no communication ( NO COMM ) with c1vac2

2, Lost N2 supply on Oct 9 This triggered a normal all valve closed condition. At this point you replace N2 cylinders and manually swich valves to recreate VAC NORMAL configuration in the correct sequential order.

   The very last thing you do is open V1 gate valve.

   a, check TP2 that is the forepump of the Maglev. Foreline pressure to drypump  ~ 10- 100 mTorr, rotation speed 50 Krpm

   b, open V4 if P2 <1Torr

   c, check Maglev rotating at 560 Hz

   d, open V1 if P1 <500 mTorr

   e, check TP3 foreline, rotation speed and open V5 if P3 <1 Torr with VA6 closed

   f, open VA6 if PAN <1 Torr

  g, open annulos valves one by one , like VASE if PASE <1 Torr and so on...........Now the Current State: should read Vac Normal

3, Maglev run for 7 days with V4 closed. This encreased its foreline pressure to estimated few Torrs  and its body temp rose ~30C on the outside.

   So it was sweating and it may be back streamed.

The present RGA data is indicating that it had to be very mild.

The RGA will have better sensitivity with VM1 open and VM2 closed.

The PSL output shutter stayed open during these period is pointing  to IFO pressure stayed P1 <3 mTorr

PROBLEM: P1 and P2 plot should show nothing where there is no communication.http://nodus.ligo.caltech.edu:8080/40m/151016_182003/oct16Fpm2015.png

                   How do we check if pressure based software interlocks are working in this no communication condition?

Convectron gauge check:

Brand new 10 years old convectron gauge at atm was swapped into the place of existing gauges to see if they read close to 760 Torrs

They did reasonable well at the low end. I tried to imitate calibration and bring down the high end with little success.

( P1 and P2  were reading 7e-4 to ~660 Torr  The correction of the upper end pushed up the lower end too. I will correct this later

P3 and P4 high ends are way off )

The point is that they work.  Convectron gauges will be replaced and calibrated at the next vent.

Interlocks were not triggered during this test. I was expected to close the PSL shutter when P1 was reading 760 Torr

This hide some problem or  not understanding.

It was good to see CC1 and CC4 working at the moment

Somehow I succeded opening VM1 and closed VM2 = vacuum normal

I just hope it stays open overnight to get comparable RGA scan.

Clean comparable scan at vacuum normal. There was no backstreaming.

CC1 and CC2 are working again. Why did they start working  again ?

Q adjusted the Dataviewer so it is not chopping of data any more. Thanks.

Cold cathode gauge reading of 10 years.

The noise floor of the Rga scan is glitching less today

Glitches are gone. Rga scan is good

Rana stated yesterday that there will be a vacuum control update in the close future. Witnesses : Rich, Chris and Dave

Can you give me this in writing?

TP3 drypump replaced after 10,344 hrs at 750 mTorr foreline pressure.

The foreline pressure is 13 mTorr after 8 hrs of running, TP3: 50K rpm, 0.14 Amp with all annuloses pumped.

The annulos pressures are 0.3 - 5 mtorr

************************

Frame builder just crashed again

Amstron batteries replaced after 11 months with SP-12-5.5HR,  2 years warranty from replaceUPSbattery.com

Note: the replace battery LED did not go out ( well pasted 24 hrs ) till the self test bottom was hold down for 2-3 sec

Our last RGA scan is from February 14, 2016  We had a power outage on the 15th

Gautom has not succeded  reseting it. The old c0rga computer looks dead. Q may resurrect it, if he can?

The c0rga computer was off, I turned it on via front panel button. After running RGAset.py, RGAlogger.py seems to run. However, there are error messages in the output of the plotrgascan MATLAB script; evidiently there are some negative/bogus values in the output. 

I'll look into it more tomorrow.

It looks like the hardware reset did the trick. Previously, I had just tried ssh-ing into c0rga and rebooting it. At the time, however, Steve and I noticed that the various LEDs on the RGA unit weren't on, as they are supposed to be in the nominal operating state. Today, Steve reported that all LEDs except the RS232 one were on today, so I just tried following the steps in this elog again, looks like things are back up and running. I'm attaching a plot of the scan generated using plotrgascan MATLAB script, it looks comparable to the plot in elog 11697, which if I remember right, was acceptable.

Unless there is some reason we want to keep this c0rga machine, I will recommission one of the spare Raspberry Pis lying around to interface with the RGA scanner when I get the time...

This is a cold scan.

3-4 hrs ago we run out of nitrogen. We are back to Vacuum Normal

We have good RGA scan now. There was no scan for 3 months.

The drill room floor will be retiled Thursday, June 16. Temporary nitrogen line set up will allow emptying the hole area.

Ifo room entry will be through control room.

The retiling work has finished, Steve and I restored the N2 supply configuration to its normal state. The sequence of steps followed was:

1. Went to the X end and closed the following valves, roughly in this order: VAEE, VAEV, VABS, VABSCI, VASV, VASE, V4, V1.
2. Checked the RPM on the various turbo pump controllers to make sure they were in their nominal states
3. Disconnect the electrical connections to V1, V4, V5 and VA6 - just to make sure some spurious signal doesn't unintentionally open any of these valves while we are mucking around with the N2 supply
4. Close the valves on the N2 cylinders in the drill room. Disconnect the temporary nitrogen line (at this point, the N2 pressure to the IFO valves goes down from ~7-PSI to 0), reconnect the old supply chain, taking care that we aren't unintentionally loosening any of the Swagelock connections while unscrewing stuff
5. Replaced one of the N2 cylinders that was running low.
6. Reopen the cylinders, restore N2 pressure to IFO valves to ~70PSI.
7. Do steps 1-3 in reverse: i.e. reconnect power to all valves, open them in the reverse order we closed them while monitoring the state of the various turbo pumps. 
8. Acknowledged the error message on the C0VAC medm screen

Note: the valve isolating the RGA automatically shutoff during this work, possibly because it detected a pressure above its threshold - after checking the appropriate pressure gauges, we reopened this valve as well. 

The attached screenshot suggests that everything went as planned and that the vacuum system is back to normal...