We tried not to open chambers above 10,000 particles of 0.5 micron cf/min

New items going in:                       2 rasor beam traps, 5 badly oxidized old silver plated  setscrew with spring loaded tips......to be replaced in the future, viton tips for eq screws....some are lose, gold plated allen wrench installed at ITMX bottom, reglued magnet on ITMX

Bad hardware things found:          nylon ball "locking elements" on OSEM locking set screws with screwdriver  slot, lose  1064 nm filter on OSEM pd

 This seal is good for daily use- operation only. The IFO has to be sealed  with light metal doors every night so ants and other bugs can not find their way in.

Our janitor Kevin is mopping the hole IFO room floor area with 5%  ant killing solution in water in order to discourage bugs getting close to our openings of the vented chamber.

You may be sensitive to this chemical too.  Do not open chamber till after lunch.

Roger.

Matt and Koji:

We closed the light doors of the chambers.

Cold cathode gauge CC4  is reading normal.

CC1 is glitching, it is probably dirty.

CC2 is fluctuating too much and it is cutting out for 6-7 minutes. It must be insulated by deposits and there is no emission current.

I think the same goes for P1

They will have to be replaced at the next vent

The COVAC_MONITOR.adl was changed. Ion pump labeled as disconnected means: ion pump controller power turned off and ion pump gate valve control connection disconnected.

RP2 roughing pump labeled disconnected means:  hardware  disconnected.

The actual operational, valve control screen has not been changed yet.

We run out of N2 for the vacuum system 6 hrs ago. The pressure rose to 1.2 mTorr with V1 closed. The interlock worked! See Nirogen presure reading fixed at http://nodus.ligo.caltech.edu:8080/40m/10968

The vacuum interlock: Nitrogen pressure transducer is reading the pneumatic pressure continously at the pump spool and c1vac1 processing it. When it drops below 60 PSI it closes V1 gate valve and V4 & V5.  Gate valve V1 needs minimum 60 PSI to close. It is critical that V1 is closed before you run out of Nitrogen so the IFO pressure is contained.

IFO vacuum is back to Vac Normal. The MC is locked.

cc4 = 2E-6 Torr with VM1 open.

Daily N2 consumption measured to be 530PSI as 3 days on 3-27-2015 but note: it does vary !

I have seen it as high as 900 psi  The long term average ~750 psi

We run out of N2 for the vacuum system. The pressure peaked at 1.3 mTorr with MC locked. V1 did not closed because the N2 pressure sensor failed.

We are back to vac normal. I will be here tomorrow to check on things.

Amstron batteries replaced after 11 months with SP-12-5.5HR,  2 years warranty from replaceUPSbattery.com

Note: the replace battery LED did not go out ( well pasted 24 hrs ) till the self test bottom was hold down for 2-3 sec

APC Smart -UPS 2200   model: SUA2200RM2U   batteries were replaced by compatible RBC43, 8x  12V5A

Batteries replaced after 3.5 years with Amstron AP-1250F2,  8x 12V 6Ah

Ben and I found this vacuum valve relay box intermittently shorthing yesterday.

It effects V4, V5, VA6 and VM1........   Please do not touch this box under the beam pipe next to the vac rack!

The function of this box to send 120VAC to the vacuum valve to move.

Steve and I managed to access the fuse in the vacuum VME crate, but replacing it did not bring it back up.  We decided to replace the entire crate.

We manually checked that the most important valves, VC1, VM1 and V1, were all closed.  We disconnected their power so that they would automatically close, and we wouldn't have to worry about them accidentally opening when we rebooted the system.

We noted where all the cables were, disconnected everything, and removed the crate.  We noted that one of the values switched when we disconnected one of the IPC cables from a VME card.  We'll note which one it was in a followup post.  We thought that was a little strange, since the VME crate was completely unpowered.

Anyway, we removed the crate, swapped in a spare, replaced all the cards and connections, double checked everything, then powered up the crate.  That's when minor chaos ensued.

When the system came back online after about 20 seconds, we heard a whole bunch of valves switching.  Luckily we were able to get the medm screens back up so that we could see what was going on.

Apparently all of the ION pump valves (VIPEE, VIPEV, VIPSV, VIPSE) opened, which vented the main volume up to 62 mTorr.  All of the annulus valves (VAVSE, VAVSV, VAVBS, VAVEV, VAVEE) also appeared to be open.  One of the roughing pumps was also turned on.  Other stuff we didn't notice?  Bad.

We ran around and manually unplugged all of the ION pump valves, since I couldn't immediately pull up the vacuum control screen.  Once that was done and we could see that the main volume was closed off we went back to figure out what was going on.

We got the medm vacuum control screen back (/cvs/cds/caltech/medm/c0/ve/VacControl_BAK.adl.  really??)  There was a lot of inconsistency between the readback states of the valves and the switch settings.  Toggling the switches seemed to bring things back in line.  At this point it seemed that we had control of the system again.  The epics readings were consistent with what we were seeing in the vacuum rack.

We went through and closed everything that should have been closed.  The line pressure between the big turbo pump TP1 and the rest of the pumps was up at atmosphere, 700 Torr.  We connected the roughing pumps and pumped down the lines so that we could turn the turbos back on.  Once TP2 and TP3 were up to speed, we turned on TP1 and opened V1 to start pumping the main volume back town.   The main volume is at 7e-4 Torr right now.

So there are a couple of problems with the vacuum system.

• Why the hell did valves open when we rebooted the VME crate?  That's very bad.  That should never happen.  If the system is set to come up to an unsafe state that needs to be fixed ASAP.  The ION pump valves should never have opened.  Nor the annulus valves.
• Why were the switches and the readbacks showing different states?
• Apparently there is no control of the turbo pumps through MEDM.  This should be fixed.

I connect belledona, the laptop at the vacuum station to the wired network, so that it's connection would be less flaky.

[ericq, Gautam, Steve]

Following roughly the same procedure as ELOG 11354, c1vac1 and c1vac2 were rebooted. The symptoms were identical to the situation in that ELOG; c1vac1 could be pinged and telneted to, but c1vac2 was totally unresponsive. 

The only change in the linked procedure was that we did not shut down the maglev. Since I unwittingly had it running for days without V4 open while Steve was away, we now know that it can handle shorter periods of time than that...

Upon reboot, many channels were readable again, unfortunately the channels for TP2 and TP3 are still blank. We were able to return to "Vacuum normal state," but because of unknowned communication problems with VM1's interlock, we can't open VM1 for the RGA. Instead we opened VM2 to expose the RGA to the main IFO volumn, but this isn't part of the "Normal" state definite, so things currently read "Undefined state".

[ Gautam and Steve ]

c1vac1 and c1vac2 were rebooted and the gauges are communicating now. V1, VA6, V5 and V4 were closed and disconnected to avoid unexpected valve switching. All went smoothly.

The new ITcc gauge is at 1e-5 Torr as CC1   This is the gauge that should be logged in slow channel.

TP2 fore line dry pump was replaced this morning after 382 day of operation.

TP3 dry pump is very noisy, but it's pressure still 47 mTorr

 Vac- rack is powered back up.  UPS first, than all other power switches from top to bottom of the rack, except Maglev

Manually started one by one TP2 and TP3  to accelerate to 50 KRPM

Brought up vac.control screen on lap-top at /cvs/cds/caltech/medm/c0/ve/VacControl_BAK.adl

V5 and VM3 were opened so TP3 can pump on the RGA

V4 was opened so TP2 can pump on the Maglev-TP1. The Maglev power was  turned on and started acceleration.

The vac control screen positions indicators were checked for true position and annulies vent valves were opened.

RGA manual on/off switch was turned at the top of the RGA-head. Ubuntu copmuter was started at cc4 1.1e-6 Torr

The RGA communication was started with:  ssh c0rga from control room

The rga-script was started ./RGAset.py     This script turns on the filament, rs-232 and scan parameters etc

Vac -configuration: IFO-P1 at atm, RGA is pumped and running in background mode, all annulos at atm

There is BLANK VacControl_BAK.adl screen only. 

I can move a valve by disconnecting it's  solenoid power if it's position is normally open.

I will close V1 and check computer cable connections and move on with manual - hand disconnect ea valve to be moved into the right position for vent. Valve positions will be confirmed by looking manual indicators on valves.

Rana stated yesterday that there will be a vacuum control update in the close future. Witnesses : Rich, Chris and Dave

Can you give me this in writing?

Convectron gauge check:

Brand new 10 years old convectron gauge at atm was swapped into the place of existing gauges to see if they read close to 760 Torrs

They did reasonable well at the low end. I tried to imitate calibration and bring down the high end with little success.

( P1 and P2  were reading 7e-4 to ~660 Torr  The correction of the upper end pushed up the lower end too. I will correct this later

P3 and P4 high ends are way off )

The point is that they work.  Convectron gauges will be replaced and calibrated at the next vent.

Interlocks were not triggered during this test. I was expected to close the PSL shutter when P1 was reading 760 Torr

This hide some problem or  not understanding.

It was good to see CC1 and CC4 working at the moment

Somehow I succeded opening VM1 and closed VM2 = vacuum normal

I just hope it stays open overnight to get comparable RGA scan.

In Steve's absence, I've tried to keep an eye on the health of the vacuum system. From Attachment #1, the pressure of the main volume seems stable, no red flags there. I also don't here any anomalously loud sounds near the vacuum pumps. I've changed the N2 cylinders that keep V1 open twice, on Wednesday and Sunday of last week. So in summary, the vacuum system looks fine based on all the metrics I know of.

(Steve, Chris)

The pumpdown had stalled because of some ancient vacuum interlock code that prevented opening the valve V1 between the turbo pump and the main volume.

This interlock [0] compares the channels C1:Vac-P1_pressure and C1:Vac-PTP1_pressure, neither of which is functioning at the moment. The P1 channel apparently stopped reading sometime during the vent, and contained a value of ~700 torr, while the PTP1 channel contained 0. So the interlock code saw this huge apparent pressure difference and refused to move the valve.

To bypass this check, we used caput to enter a pressure of 0 for P1.

[0] /cvs/cds/caltech/state/from_luna/VacInterlock.st

Morning condition: vacuum rack power is still off, no MEDM screen reading.....meaning unknown vacuum pressure.We closed PSL shutter immediately.

Joe restored c1iscepis and Jenne powered up the vac-rack UPS. Now the rest of the vac-rack power were restored from starting at the top to bottom.

P1 was reading 15 mTorr.  We restarted pumps and  set vacuum valve positions. V1 opening required Rob's recipe of elog # 1863 to defeat interlock that

has a non communicating gauge: PTP1

CC1 pressure just reached 1e-6 Torr at VAC NORMAL configuration.

State condition: Vac Normal,          CC1 ~8e-6 Torr

The IFO pressure peaked at  8.3 mTorr after 2days of not pumping.

CLOSE CALL on the vacuum system:

Jamie and I disabled V1, VM2 and VM3 gate valves by disconnecting their 120V solenoid actuator before the swap of the VME crate.

The vacuum controller unexpectedly lost control over the swap as Jamie described it. We were lucky not to do any damage! The ion pumps were cold and clean. We have not used them for years so their outgassing possibly  accumulated to reach ~10-50 Torr

I disconnected_ immobilized and labelled the following 6 valves:  the 4 large ion pump gate valves and VC1,  VC2  of the cryo pump. Note: the valves on the cryo pump stayed closed. It is crucial that a warm cry pump is kept closed!

This will not allow the same thing to happen again and protect the IFO from warm cryo contamination.

The down side of this that the computer can not identify vacuum states any longer.

This vacuum system badly needs an upgrade. I will make a list.

 ITMX damping restored.

 Out gassing plus leak rate   0.15  mTorr / hour

 The pressure rose to 2.5 mTorr in 17 hours

 V1 was opened at 1:56pm

 VM2 opened at 2:10 so the RGA region is back to 1e-5 torr

 Valve configuration:       Vacuum Normal    is reached in really 4 days if we do not count overnight rest of roughing.

 VA6 and VC2 are reconnected. I'm turning on the RGA next

 All 4 ion pumps were vented with air and pumped down to ~ 1e-4 Torr

 Ion pumps gate valve control cables are connected and their pumps are disconnected.

 The black relay box was tested repeatedly and it stopped misbehaiving.

We were at atmosphere for 13 days.  Chamber BS, ITMX, ITMY and ETMY were opened.

Al foil "cups" were placed on the back side OSEMs of PRM.

RP1 and RP3 roughing pump manual of Leybold D30A oily rotory pump

Fore pump of TP2 & TP3 Varian SH-100 Dry Scroll

TP2 and TP3 small turbo drag pump  Varian 969-9361 

TP2 and TP3 turbo controller Varian 969-9505

TP1 magnetically suspended turbo pump  Osaka TG390MCAB, sn360 and controller TC010M  and note : this  pump  running on 208VAC single phaseIt is not on the UPS !  

                                                             Osaka Maglev Manual        and Osaka Controller Communication Wiring                                                                                                                                

VC1 cryo pump CTI-Cryogenics Cryo Torr 8 sn 8g23925  SAFETY note: compressor single phase 208VAC and the head driver 3 phase 208VAC Compressor and driver have each separate power cord!

Installed at 40m wiki  also

 While I was doing the oil change of the roughing pumps I accidentally touched the 24 V adjustment knob on the power supply.

All valve closed to default condition. I realized that the current indicator was red at 0.2A  and the voltage fluctuated from 3-13V

Increased current limiter to 0.4A and set voltage to 24V     I think this was the reason for the caos of valve switching during the VME swap.

We did the following:

1, closed V1, VM1,  annuloses: VASE, VASV, VABS, VAEV,  VAEE and VA6

2, stop rotation of Maglev-TP1, waited to decellerate and turned off power to it

3, closed V4,  stoped rotation of TP2, waited to decellerate and turned power off

4, opened VM3 to RGA that is still running

I will come in tomorrow 9-10am to restart pumping.

The V1 gate valve specs installed at 40m wiki page. VAT model number 10846-UE44-0007 Our main volume pumping goes through this 8" id gate valve V1 to Maglev turbo  or Cryo pump to  VC1

The ion pumps have 6" id gate valves:VAT 10844-UE44-AAY1,  Pneumatic actuator with position indicator and double acting solenoid valve 115V 60Hz  Purchased 1999 Dec 22

UHV gate valves 2.5" id. VAT  10836-UE44    Pneumatic actuator with position indicator and double acting solenoid valve 115V 60 hz, IFO to RGA  VM1 &  RGA to Maglev  VM2

mini UHV gate valve 1.5" id.   VAT  01032-UE01      2016 cataloge page 14,   manual - no position indicator, VM4  next to manual adjustable fine leak valve to RGA

UHV angle valve 1.5" id, model VAT 28432-GE41, Viton plate seal, pneumatic actuator with position indicator & solenoid valve 115V & single acting closing spring  MEDM screen: VM3,VC2, V3,V4,V5,V6,VA6,V7 & annuloses Each chamber annulos has 2 valves.

UHV angle valve 1.5" id, model VAT 57132-GE05   go page 208,   Metal tip seal, manual actuating only with position indicator,   MEDM screen: roughing RV1 and venting VV1 hand wheel needed to close to torque spec

UHV angle valve  1.5" id. model VAT 28432-GE01           Viton plate seal, manual operation only at IT gauges Hornet & Super Bee and  ion pumps roughing  ports. These are not labeled.

The Cryo pump interlock wiring was added too

Note: all moving valve plate seals are single.

This condition is created when we run out of nitrogen that holds the valves in set positions. The Maglev and other turbos are still running in this configuration so one has to pay attention that the foreline valves are open.

Reset  error message on vacuum screen and opened V4 & V5 at 3pm today.

We started a vacuum work in this morning. And still it's going on.

Although the last night the green team replaced a steering mirror by an 80% reflector on the PLS table, the beam axis to the MC looks fine.

The MC refl beam successfully goes into the MCrefl PD, and we can see the MC flashing as usual.

We started measuring the distance of the optics inside the vacuum chamber, found the distance from MC3 to MMT1(curved mirror) is ~13cm shorter than the design.

We moved the positions of the flat mirror after the Faraday and the MMT1, but could not track the beam very well because we did not completely lock the MC.

Now we are trying to get the lock of the MC by steering the MC mirrors.

P.S.

Kevin suceeded in locking it !!

 Hey guys,

I just finished redoing the calc based on the measurements that happened last week.  Using the average of the Vert and Horz measurements in Kevin's elog 2986, I find that we need to make the MMT telescope ~8cm longer.  So, can you please place the flat mirror after the Faraday in the same place as the drawing, but move the MMT1 79mm farther away from that flat mirror?  Looking at the table layouts that Koji has on the wiki, this should still (barely) fit.

New distances:

d2a = 884.0mm (no change) ------  MC3 to Flat after Faraday

d2b = 1123.2mm (move MMT1 farther toward center of BS table)  -------- Flat after Faraday (SM1) to MMT1

d3 = 1955.0mm (result of moving MMT1)  ---------  MMT1 to MMT2

d4a = 1007.9mm (no chnage)        ----------- MMT2 to SM2

d4b = 495.6mm (no change) ------------ SM2 to PRM

I just got off the phone with Alberto and Kiwamu, and I'm going to try to recalculate things based on their measurements of the distances between MC3 and SM1.  It sounds like the CAD drawings we have aren't totally correct.    I know that when we opened doors just before Christmas we measured the distances between the BS table and the ITM tables, but I don't think we measured the distance between the IOO table and the BS table.  Hopefully we can fit everything in our chambers.....

As preparation for the upcoming planned power outage we turned turbos, RGA off and closed valves.

IFO chamber is not pumped now. Small leaks and out gassing will push the pressure up slowly. At 3 mTorr of P1 the PSL output shutter

will be closed by the interlock.

It is OK to use light in the IFO up to this point.

The vacuum system is coming up to atm. The vent was started with slow N2 flow. The PSL output is blocked 2 places on the table.

 IFO_P1 pressure 1.6e-5 torr after 6 days at atm

PS: PSL sliding door 11 was left open overnight. The PSL particle count will reach room counts in 20 seconds at low speed of HEPA

  VENT OBJECTIVES:

1, Fix ETMX sus "jump issue"

2, First Contact clean the arms

3, Install new spare cold cathode and convectron gauges: InstruTech-Hornet

4, Install 50 mm apeture beam baffles

5, Check and clean optical quality viewport from inside

The following bullets were executed:

• Closed shutter of PSL-IR and green shutters at the ends 
• Checked jam nuts are protecting bellows 

• Check crane functionality & cleanliness last week

• Particle count must be under 10,000 counts / cf min for 0.5 micron 
• Checked all metal window covers are on. 
• Check 5 cylinders (24 cft size) of instrument grade air, called Alfa Gas 1 in stock.
• Took pictures of medm screens: sus-sum, aligned oplevs pos, alignment values and vac configuration
• Turned Oplev servos off
• Closed V1 and VM1, opened VM2
• Opened VV1 manual valve to Nitrogen cylinder and let the P1 to rise to 25 Torr
• Switched over to Instrument Grade Air Cylinder and continoed the vent with 8-10 PSI reading at the pressure regulator

We are venting the 40m IFO

RGA background scan

Vacuum Status: Chamber Open

All chamber annuloses are vented.  Vac Monitor screen is not communicating with gauges. The valve position indicator are working.

RGA is pumped by Maglev through VM2

I just disconnected the 6th instrument grade air cylinder from the vacuum envelope at 720 Torr. Now it will reach equilibrium through a filter as it sucks in lab air.

This is the sure way not to over pressurize the chamber.

IFO is at atmosphere. The MC can be locked in air now.

The doors will be coming off tomorrow 8am sharp.

Do we want to install the ITM baffles?

What about the found OSEM filters?