The Osaka TG390MCAB maglev turbo pump's controller TC010M has passed the 40,000 hrs of operation. This triggered the " alarm" LED  warning light to come on. 

It is normal maintenance.  Maglev TP-1 is running perfectly.  Osaka will send us a loaner-controller that we can use while they do the std maintenance.

I'm thinking of ~ February to do this.

[Jenne, Steve]

Steve noticed that the RGA was not logging data and that not all the correct connection lights were on, and he wasn't able to run the "RGAset.py" script (in ...../scripts/RGA/) that sets up the proper parameters. 

I looked, and the computer was not mounting the file system.  I did a remote shutdown, then Steve went in and pushed the power button to turn the machine back on.  After it booted up, it was able to talk to the file system, so I started ..../scripts/RGA/RGAset.py .  The first 2 times I ran the script, it reported errors, but the 3rd time, it reported no communication errors.  So, now that the computer can again talk to the file system, it should be able to run the cronjob, which is set to take data at 4am every day.  Steve will check in the morning to confirm that the data is there.  (The last data that's logged is 22Dec2013, 4am, which is right around when Koji reported and then fixed the file system).

 We are venting tomorrow. This give us an opportunity to fix sick vacuum controller computer. Jamie volunteered.

Remember to rough down cryo and ion pump volumes. Their pressure can be at 1 Torr range after years of accumulated outgassing.  Without total valve controls it is dangerous to have these air pockets. Some of their  gate valves can be leaking and that would explane the slower pumpdown speed. 

There is BLANK VacControl_BAK.adl screen only. 

I can move a valve by disconnecting it's  solenoid power if it's position is normally open.

I will close V1 and check computer cable connections and move on with manual - hand disconnect ea valve to be moved into the right position for vent. Valve positions will be confirmed by looking manual indicators on valves.

 The 40m vacuum envelope vent is completed with instrument grade air.

Valve configuration: chamber open, RGA is pumped through VM3 by TP3,

[Steve Koji]

We pushed the reset button of c1vac1 and c1vac2 and the vacuum screen is back.

First, we pushed the reset button of c1vac1 and pushed the one on c1vac2.
This did not bring c1vac2 up. We pushed the reset of c1vac2 again and now everything of the vacuum screen is back.

 1, Reset works

 2, Default values are lost. They actually reversed into open and turn on everything when power recycle c1vac1, c1vac2 or 24V dc power supply. This can vent the IFO in an event of power failure!

      It's may be the time to go back to an isolated, vacuum controller only computer.

The present valve configuration at Atm1

a, all annuloses are vented

b, valve cables disconnected at: VC1 and 4 IP gate valves

c, the RGA is off and it is pumped by the Maglev through VM2

d, cryo pump is being roughed with TP3  It's outgassing rate was 25 mTorr / min

e, Roughing hose is disconnected

 Vacuum checks before pump down:

1, check intermittent contact issue in black relay box

2, reconnect cable to ion pumps and vent them to atm before pump down

3, repeat power shutdown-reboot at atmosphere

 The ion pumps were vented just before pumpdown and their gate valves were opened.

This is an effort to minimize a possible leak through their gates.

Is there a volunteer who goes home late and would close off the roughing? tonight

 Pumpdown completed. IR shutter opened at P1 1 mTorr  The block is still in the beam path.

Remember to protect MCR pd before crack up the PSL power.

The ion pump gate valves were just closed by cc1 triggered interlock

The cry pump was "regenerated" during the vent and it's outgassing rate minimized.

CC3 cold cathode gauge was replaced.

Valve configuration for week end:

1, VA6 disconnected to avoid accidental venting the IFO through the annulos

2, VC2 disconnected to insure that the cryo stays closed

3, RGA is not running, It's pressure limit 1e-5 Torr

 Valve configuration:       Vacuum Normal    is reached in really 4 days if we do not count overnight rest of roughing.

 VA6 and VC2 are reconnected. I'm turning on the RGA next

 All 4 ion pumps were vented with air and pumped down to ~ 1e-4 Torr

 Ion pumps gate valve control cables are connected and their pumps are disconnected.

 The black relay box was tested repeatedly and it stopped misbehaiving.

We were at atmosphere for 13 days.  Chamber BS, ITMX, ITMY and ETMY were opened.

Al foil "cups" were placed on the back side OSEMs of PRM.

 All normal.

Pd 76 and 77 are compared at 30 days of pumping. We spend 13-14 days at atmosphere  before each.

 Pump down 76 was with leaky ion pump gate valve. The ion pumps are not in use for years so they accumulated some higher pressure  PLUS the valve switching caos at computer reboot most likely

increased the ion pumps pressures to about 10-20 torr

I think one of the ion pump gate valve was not sealing well. This leak was holding back pump down speed at pd76

 The 40m experienced a building-wide power failure for ~30 seconds at ~7:38 pm today.

Thought that might be important...

 Out gassing plus leak rate   0.15  mTorr / hour

 The pressure rose to 2.5 mTorr in 17 hours

 V1 was opened at 1:56pm

 VM2 opened at 2:10 so the RGA region is back to 1e-5 torr

 We just received the loaner controller that will be swapped in it tomorrow morning.

The vacuum pressure will rise somewhat during this action.

 The loaner controller is swapped in. It has  520 Hz rotation speed.  This speed use to be 680 Hz with our old one.

 The  Maglev controller was serviced at Osaka Vacuum. It was swapped in this morning.

 TP2 dry pump replaced at fore pump pressure 1 Torr,  TP2 50K_rpm 0.34A

 Top seal life 6,362 hrs

 New seal performance at 1 hr  36 mTorr, 

 Maglev at 560 Hz, cc1 6e-6 Torr

 Rga scan at pump down 77, vacuum normal valve configuration, maglev rotation 560 Hz and day 111

This morning valve condition: V1, VM1, V4 and V5 valves were closed. IFO pressure rose to 1.3 mTorr

It was caused by low N2 pressure.  Our vacuum valves are moved-controlled by 60-70 PSI of nitrogen.

When this supply drops below 50-60 PSI the interlock closes V1 valve. This is the minimum pressure required to move the large valves.

It is our responsibility to check the N2 cylinder pressure supply.

The vacuum valve configuration is back to VAC. NORMAL,  CC1  4.8E-6 Torr

PS: Bob says that the second cylinder was full this morning, but the auto-switch over did not happen.

 The TESCOM automatic changeover regulator  [ model ACS 012-1011 ] manifold is most likely  clogged. The new one will arrive 8-8-2014

This means that the IFO pressure may go up to a few mTorr when we change cylinder or V1 valve triggered because there is no nitrogen supply.

Interlock closed valve V1, V4, V5 and VM1 when the nitrogen supply run out. The IFO pressure rose to P1 1 mTorr

In order to recover Vacuum Normal valve configuration I did the following:

Replaced both nitrogen cylinders. Confirmed pneumatic nitrogen pressure 70 PSI.   Opened valves V4 and V5

At P2 < 1 mTorr, Maglev rotation 560 Hz , V1 was opened.

VM1 was opened when CC1 pressure dropped below < 1e-5 torr

Please  take a look at the N2 cylinders pressure on Friday to insure that there is enough for the week end.

The daily consumption is 600-700 PSI

 Jam nuts checked. Oplev servos turned off. Particle count checked. Vertex crane functionality checked.

 The vent is completed. ITMX was kicked up accidentally. Valve configuration: chamber open, RGA is pumped through VM2  Maglev

[Steve, EricQ, Jenne]

ITMY and BS heavy doors are off, light doors are on.  Q is aligning the IFO.

 The RGA time stamp was correct last at 20140527

  Rga stopped scanning at 20140530

Q checked the earth quake stops of SRM and we put the ITMY & BS doors on. 

We  stopped pumping just short of 3 hours at 320 Torr.  Pumping speed was 2.7 Torr / min with partially closed RV1 and butterfly valve/

RP1&3 roughing pump hose is disconnected. Butterfly valve removed. The vac envelope is closed. 

 This is our second stop. I will be back this afternoon.     IFO  P1 3.5 Torr

 Pump down reached "vacuum normal" state. IFO _P1 pressure 1e-4 torr in 8 hrs actual pumping time

 PSL shutter is opened.

 IFO_P1 pressure 1.6e-5 torr after 6 days at atm

PS: PSL sliding door 11 was left open overnight. The PSL particle count will reach room counts in 20 seconds at low speed of HEPA

We have two cold cathode gauges at the pump spool and one  signal cable to controller. CC1  in horizontal position and CC1 in vertical position.  

CC1 h started not reading so I moved cable over to CC1 v

PSL Innolight laser and the 3 units of IFO air conditions turned on.

The vacuum system reaction to losing power: V1 closed and Maglev shut down. Maglev is running on 220VAC so it is not connected to VAC-UPS.  V1 interlock was triggered by Maglev "failure" message.

Maglev was reset and started. After Chiara was turned on manually I could bring up the vac control screen through Nodus and opened V1

"Vacuum Normal" valve configuration was recovered instantly.

It is arriving Thursday

 Pump  spool valves V5, V4, V3 sweating a lot. VM3 and VC2 not so much.

They are VAT valves F28-62887-03, 11, 14 and so on ~15-16 years old.

 I'm speculating that some plastic is aging-braking down at the atmospheric-pneumatic side of valves.
The vacuum side is not effected, according to vacuum pressure readings.

May be some condensation from the small turbos? No

I'm looking for an identical valve to examine, but I can not find one.

We are using industrial grade 99.96% Nitrogen to actuate these valves.

Valves are not effected are  dry: VA6, V6, V7 and all annuloses.

They are VAT valves F28-62887-03, 11, 14 and so on ~15-16 years old.

VAT's answer:

Yes, our engineers are aware of this issue.  They say:

The pneumatic actuator needs lubricant as the O-ring (Viton) slides in the cylinder. Without grease the O-ring would be abraded and leaking after only a relatively few cycles.  The lubricant used in our pneumatic actuators is an emulsion of oil and Teflon flakes.   Vibration, many cycles and sometimes high temperature lead to the separation of the oil and Teflon.   That is apparently the issue you are seeing.

VAT is and has been testing and qualifying new lubricants, and this is one of the factors we are always looking to improve.  The formula we used 15 years ago in these valves seems to have performed reasonable  well.  Our formula today should perform even better.

We realize this explanation does not help you with these existing valves, but 15 years of service is not too bad is it? 

Steve -NOTE:bonnet seal is metal so there is no way this oil can get into our vacuum ( only if the bellow leaks )

 IFO vacuum, air condition and PMC HV are still down. PSL out put beam is blocked on the table.

 We are pumping again. This is a temporary configuration. The annuloses are at atmosphere. The reset reboot of c1Vac1 and 2 opened everything except the valves that were disconnected.

TP2 lost it's vent solenoid power supply and dry pump during the power outage.

They were replaced but the new small turbo controller is not set up as the old TP2 was so it does not allow V4 to open. 

Tomorrow I will swap back the old controller,  pump down the annuloses and close off the ion pumps.

I removed the beam block from the PSL table and opened the shutter. CC4 has the real pressure 2e-5 Torr  

CC1 is not real.

 Tp2 is controlled by old controller. Annuloses pumped down. Valve configuration: "vacuum normal "

  Ion pumps closed at  <1e-4 mT