The ion pumps were vented just before pumpdown and their gate valves were opened.

This is an effort to minimize a possible leak through their gates.

Is there a volunteer who goes home late and would close off the roughing? tonight

 Pumpdown completed. IR shutter opened at P1 1 mTorr  The block is still in the beam path.

Remember to protect MCR pd before crack up the PSL power.

The ion pump gate valves were just closed by cc1 triggered interlock

The cry pump was "regenerated" during the vent and it's outgassing rate minimized.

CC3 cold cathode gauge was replaced.

Valve configuration for week end:

1, VA6 disconnected to avoid accidental venting the IFO through the annulos

2, VC2 disconnected to insure that the cryo stays closed

3, RGA is not running, It's pressure limit 1e-5 Torr

 Valve configuration:       Vacuum Normal    is reached in really 4 days if we do not count overnight rest of roughing.

 VA6 and VC2 are reconnected. I'm turning on the RGA next

 All 4 ion pumps were vented with air and pumped down to ~ 1e-4 Torr

 Ion pumps gate valve control cables are connected and their pumps are disconnected.

 The black relay box was tested repeatedly and it stopped misbehaiving.

We were at atmosphere for 13 days.  Chamber BS, ITMX, ITMY and ETMY were opened.

Al foil "cups" were placed on the back side OSEMs of PRM.

 All normal.

Pd 76 and 77 are compared at 30 days of pumping. We spend 13-14 days at atmosphere  before each.

 Pump down 76 was with leaky ion pump gate valve. The ion pumps are not in use for years so they accumulated some higher pressure  PLUS the valve switching caos at computer reboot most likely

increased the ion pumps pressures to about 10-20 torr

I think one of the ion pump gate valve was not sealing well. This leak was holding back pump down speed at pd76

 The 40m experienced a building-wide power failure for ~30 seconds at ~7:38 pm today.

Thought that might be important...

 Out gassing plus leak rate   0.15  mTorr / hour

 The pressure rose to 2.5 mTorr in 17 hours

 V1 was opened at 1:56pm

 VM2 opened at 2:10 so the RGA region is back to 1e-5 torr

 We just received the loaner controller that will be swapped in it tomorrow morning.

The vacuum pressure will rise somewhat during this action.

 The loaner controller is swapped in. It has  520 Hz rotation speed.  This speed use to be 680 Hz with our old one.

 The  Maglev controller was serviced at Osaka Vacuum. It was swapped in this morning.

 TP2 dry pump replaced at fore pump pressure 1 Torr,  TP2 50K_rpm 0.34A

 Top seal life 6,362 hrs

 New seal performance at 1 hr  36 mTorr, 

 Maglev at 560 Hz, cc1 6e-6 Torr

 Rga scan at pump down 77, vacuum normal valve configuration, maglev rotation 560 Hz and day 111

This morning valve condition: V1, VM1, V4 and V5 valves were closed. IFO pressure rose to 1.3 mTorr

It was caused by low N2 pressure.  Our vacuum valves are moved-controlled by 60-70 PSI of nitrogen.

When this supply drops below 50-60 PSI the interlock closes V1 valve. This is the minimum pressure required to move the large valves.

It is our responsibility to check the N2 cylinder pressure supply.

The vacuum valve configuration is back to VAC. NORMAL,  CC1  4.8E-6 Torr

PS: Bob says that the second cylinder was full this morning, but the auto-switch over did not happen.

 The TESCOM automatic changeover regulator  [ model ACS 012-1011 ] manifold is most likely  clogged. The new one will arrive 8-8-2014

This means that the IFO pressure may go up to a few mTorr when we change cylinder or V1 valve triggered because there is no nitrogen supply.

Interlock closed valve V1, V4, V5 and VM1 when the nitrogen supply run out. The IFO pressure rose to P1 1 mTorr

In order to recover Vacuum Normal valve configuration I did the following:

Replaced both nitrogen cylinders. Confirmed pneumatic nitrogen pressure 70 PSI.   Opened valves V4 and V5

At P2 < 1 mTorr, Maglev rotation 560 Hz , V1 was opened.

VM1 was opened when CC1 pressure dropped below < 1e-5 torr

Please  take a look at the N2 cylinders pressure on Friday to insure that there is enough for the week end.

The daily consumption is 600-700 PSI

 Jam nuts checked. Oplev servos turned off. Particle count checked. Vertex crane functionality checked.

 The vent is completed. ITMX was kicked up accidentally. Valve configuration: chamber open, RGA is pumped through VM2  Maglev

[Steve, EricQ, Jenne]

ITMY and BS heavy doors are off, light doors are on.  Q is aligning the IFO.

 The RGA time stamp was correct last at 20140527

  Rga stopped scanning at 20140530

Q checked the earth quake stops of SRM and we put the ITMY & BS doors on. 

We  stopped pumping just short of 3 hours at 320 Torr.  Pumping speed was 2.7 Torr / min with partially closed RV1 and butterfly valve/

RP1&3 roughing pump hose is disconnected. Butterfly valve removed. The vac envelope is closed. 

 This is our second stop. I will be back this afternoon.     IFO  P1 3.5 Torr

 Pump down reached "vacuum normal" state. IFO _P1 pressure 1e-4 torr in 8 hrs actual pumping time

 PSL shutter is opened.

 IFO_P1 pressure 1.6e-5 torr after 6 days at atm

PS: PSL sliding door 11 was left open overnight. The PSL particle count will reach room counts in 20 seconds at low speed of HEPA

We have two cold cathode gauges at the pump spool and one  signal cable to controller. CC1  in horizontal position and CC1 in vertical position.  

CC1 h started not reading so I moved cable over to CC1 v

PSL Innolight laser and the 3 units of IFO air conditions turned on.

The vacuum system reaction to losing power: V1 closed and Maglev shut down. Maglev is running on 220VAC so it is not connected to VAC-UPS.  V1 interlock was triggered by Maglev "failure" message.

Maglev was reset and started. After Chiara was turned on manually I could bring up the vac control screen through Nodus and opened V1

"Vacuum Normal" valve configuration was recovered instantly.

It is arriving Thursday

 Pump  spool valves V5, V4, V3 sweating a lot. VM3 and VC2 not so much.

They are VAT valves F28-62887-03, 11, 14 and so on ~15-16 years old.

 I'm speculating that some plastic is aging-braking down at the atmospheric-pneumatic side of valves.
The vacuum side is not effected, according to vacuum pressure readings.

May be some condensation from the small turbos? No

I'm looking for an identical valve to examine, but I can not find one.

We are using industrial grade 99.96% Nitrogen to actuate these valves.

Valves are not effected are  dry: VA6, V6, V7 and all annuloses.

They are VAT valves F28-62887-03, 11, 14 and so on ~15-16 years old.

VAT's answer:

Yes, our engineers are aware of this issue.  They say:

The pneumatic actuator needs lubricant as the O-ring (Viton) slides in the cylinder. Without grease the O-ring would be abraded and leaking after only a relatively few cycles.  The lubricant used in our pneumatic actuators is an emulsion of oil and Teflon flakes.   Vibration, many cycles and sometimes high temperature lead to the separation of the oil and Teflon.   That is apparently the issue you are seeing.

VAT is and has been testing and qualifying new lubricants, and this is one of the factors we are always looking to improve.  The formula we used 15 years ago in these valves seems to have performed reasonable  well.  Our formula today should perform even better.

We realize this explanation does not help you with these existing valves, but 15 years of service is not too bad is it? 

Steve -NOTE:bonnet seal is metal so there is no way this oil can get into our vacuum ( only if the bellow leaks )

 IFO vacuum, air condition and PMC HV are still down. PSL out put beam is blocked on the table.

 We are pumping again. This is a temporary configuration. The annuloses are at atmosphere. The reset reboot of c1Vac1 and 2 opened everything except the valves that were disconnected.

TP2 lost it's vent solenoid power supply and dry pump during the power outage.

They were replaced but the new small turbo controller is not set up as the old TP2 was so it does not allow V4 to open. 

Tomorrow I will swap back the old controller,  pump down the annuloses and close off the ion pumps.

I removed the beam block from the PSL table and opened the shutter. CC4 has the real pressure 2e-5 Torr  

CC1 is not real.

 Tp2 is controlled by old controller. Annuloses pumped down. Valve configuration: "vacuum normal "

  Ion pumps closed at  <1e-4 mT

 Our first RGA scan since May 27, 2014 elog10585

 The Rga is still warming up. It was turned on 3 days ago as we recovered from the second power outage.

 EricQ and Steve,

Steve preset the vacuum for safe-reboot mode with C1vac1 and C1vac2 running normal: closed valves as shown, stopped Maglev & disconnected valves V1 plus valves with moving labels.

(The position indicator of the valves changes to " moving " when its cable disconnected )

Eric shut down Chiara, installed APC's UPS Pro 1000 and restarted it.

All went well. Nothing unexpected happened. So we can conclude that the vacuum system with running C1vac1 and C1vac2 is not effected by Chiara's losing AC power.

 TP3 dry pump  replaced at 540 mT as TP3 50K_rpm 0.3A with annulos load. It's top seal life time was 11,252 hrs

We run out of N2 for the vacuum system. The pressure peaked at 1.3 mTorr with MC locked. V1 did not closed because the N2 pressure sensor failed.

We are back to vac normal. I will be here tomorrow to check on things.

 ITMX damping restored.

[Koji, Steve]

Summary

The N₂ pressure reading (C1:VAC-N2PRES) is now up-to-date after rebooting c1vac1.
The vaccum system is "Vacuum normal". We now have a space pressure transducer.

Introduction

Our vacuum valves are manipulated with 60~75 PSI of nitrogen. All the valves are configured to be closed in the case of low N₂ supply pressure.
In order to avoid this safety shutdown accidentally triggered, we have two N₂ cylinders to sustain the vacuum valves. When one cylinder goes to low
the mechanical valve switches over to the other cylinder.

We have the monitor channel for this (combined) cylinder pressure. One shoulbe be able to see periodical pressure variation when the auto cylinder
switch is operating. However, the nirogen pressure reading got stuck at 66 PSI on Dec.16, 2014 (See attached 60-day plot of N2 supply pressure). 

What we did

This morning we tracked down the cause of the trouble. We first closed the valves on EPICS and started to vary the N2 pressure.

Our first guess was the pressure transducer (Omega #236PC100GW) that was already 15 yrs old. We even has a sensor spare for replacement.
But it turned out that the direct voltage reading (to be 1mV/PSI) is changing correctly. The second guess was Omega Controller-Monitor
#DPiS32-C24 that is reading the voltage from the tranceducer. The display on this small black unit was changing corresponding to the
pressure change.

So our thought was
1) RS232C of the monitor unit is not working correctly
or
2) c1vac1 is not communicating with the monitor unit.

We wondered what could cause c1vac1 not communicating with the monitor unit, but we were afraid that some function got stuck
during either the nodus upgrade or chiara rebooting (or something else). So we decided to reboot c1vac1

In order to avoid any glitch in the main vacuum pressure, Steve disconnected some of the controller connectors for the closed valves.
We did this treatment before and it was successful.

Then c1vac1 was rebooted just by telnet and type reboot in the terminal.
Once the target is back in action, we noticed that the monitor value started to move.

Steve reverted the cables to the valves and operated the valves to recover "Vacuum Normal" state. Everything is now nicely settled.

We run out of N2 for the vacuum system 6 hrs ago. The pressure rose to 1.2 mTorr with V1 closed. The interlock worked! See Nirogen presure reading fixed at http://nodus.ligo.caltech.edu:8080/40m/10968

The vacuum interlock: Nitrogen pressure transducer is reading the pneumatic pressure continously at the pump spool and c1vac1 processing it. When it drops below 60 PSI it closes V1 gate valve and V4 & V5.  Gate valve V1 needs minimum 60 PSI to close. It is critical that V1 is closed before you run out of Nitrogen so the IFO pressure is contained.

IFO vacuum is back to Vac Normal. The MC is locked.

cc4 = 2E-6 Torr with VM1 open.

Daily N2 consumption measured to be 530PSI as 3 days on 3-27-2015 but note: it does vary !

I have seen it as high as 900 psi  The long term average ~750 psi

Based on Jenne's chiara disk usage monitoring script, I made a script that checks the N2 pressure, which will send an email to myself, Jenne, Rana, Koji, and Steve, should the pressure fall below 60psi. I also updated the chiara disk checking script to work on the new Nodus setup. I tested the two, only emailing myself, and they appear to work as expected. 

The scripts are committed to the svn. Nodus' crontab now includes these two scripts, as well as the crontab backup script. (It occurs to me that the crontab backup script could be a little smarter, only backing it up if a change is made, but the archive is only a few MB, so it's probably not so important...)