Vacuum Operation Guide is up loaded into the 40m-wiki. This is an old master copy. Not exact in terms of real action, but it is still a good guide of logic.

Rana has promissed to watch the N2 supply and change cylinder when it is empty. I will be Hanford next week.

Koji and Steve succeded rebooting C1vac1, C1vac2 and pressure reading is working now

More tomorrow .........

The serial connections to the vacuum gauges were recovered by rebooting c1vac1 and c1vac2.

Steve claimed that the vacuum screen had showed "NO COMM" at the vacuum pressure values.
The epics connection to c1vac was fine. We could logged in to c1vac1 with telnet too although c1vac2 had no response.

After some inspection, we decided to reboot the slow machines. Steve manually XXXed YYY valves (to be described)
to prepare for any possible unwanted switching. Initially Koji thought only c1vac2 can be rebooted. But it was wrong.
If the reset button is pushed, all of the modules on the same crate is reset. So everything was reset. After ~3min we still
don't have the connection to c1vac1 restored. We decided to another reboot. This time I pushed c1vac1 reset button.
After waiting about two minutes, the ADCs started to show green lights and the switch box started scanning.
We recovered the telnet connection to c1vac1 and epics functions. c1vac2 is still note responding to telnet, and
the values associated with c1vac2 are still blank.

Steve restored the valves and everything was back to normal.

Koji and Steve,

One computer expert and one vacuum expert required.

Atm 1,  problem condition: gauges are not reading for a week, error message "NO COMM" and all computer LEDs are green

Atm 2, prepare to safe reboot:

            a, close V1, disconnect it's power cable and turn off Maglev, wait till rotation stops

            b, close PSL shutter ( take adrenaline if needed )

            c, close V4, V5, VA6 valves and disconnect their cables. "Moving" error message indicating this condition.

               V1 is not showing "Moving" because its power cable disconnected only! It will show it if its position indicator cable is disconnected too. There is no need for that.

               These valves closed and disabled will not allow accidental venting of main volume.

            d, push reset, reseting c1vac2 will reset c1vac1 also, wait ~ 6 minutes

"Vacuum Normal" valve configuration was restored after succesful reboot as follows:

             a, reconnect cable and open V4 and V5 at P2 & P3 <1e-1 Torr

             b, observe that P2  < 1e-3 Torr and retsart Maglev

             c, wait till Maglev reaches full speed of 560 Hz and reconnect-open V1

             d, reconnect-open VA6 at P3 <1e-3 Torr

NOTE: VM1 valve was locked in open position and it was not responding before and after reboot

          Error message on Atm2 is indicating this locked condition: "opening VM1 will vent IFO"

          This is a fauls message. The valve is frozen in open position. We need a softwear expert help.

1. Yesterday we did put the vacuum system into safe to reboot mode  for the ethernet card swap.
2. Atm1, IFO pressure
3. Atm2, as prepared valve configuration where " moving " means closed and disconnected

TP2 dry fore pump sn PLE10082 was replaced at pressure 717 mTorr,  TP2 50K rpm 0.33A @ 112,677 hrs

Top seal life was 8,160 hrs

Model  SH110, Sn LP1007L556 was installed. It's fore line pressure after 30 minutes of running 38 mTorr, TP2 turbo at 50K rpm 0.18A

CC4 cold cathode gauge jump triggered interlock to close VM1 valve to protect the RGA.

The IFO pressure is 1e-5 Torr

Vac normal was recovered by opening VM1

The cold cathode gauge is back to normal. cc4 is the last gauge is "functioning"

MKS is not responding. The spare controller and gauges are back for repair.

The IFO pressure is estimated ~1E-6 Torr with modified Vac normal valve configuration.

CC4 is in a jumping mode between 2e-5 and 1e-6 Torr

 Pressure based interlock kicks in to close VM1   at 2e-5 Torr to protect the RGA.

I did open VM1 repeatedly in the last few mornings but as cc4 jumps VM1 closes.

As VM1  closed to RGA scans are not seeing the IFO. I will look at some scans on Monday.

Mean while I opened VM2 to lower the pressure for the RGA. This change will be read by "Current Status: Undefined State"

So do not panic, the IFO pressure is normal.

I need someone's help to raise the interlock threshold to 5e-5 Torr

I'm buying a new cold cathode gauge on Monday.

note: cc1 is out of order!

         just read P1  the pressure is < 7e-4 Torr  This gauge is very reliable and it is at the low end of it's range.

RGA background scan.

The IFO is closed off from RGA with VM1

CC4 ( and CC1)  is still flaky and it's interlock closes VM1

At 10:02AM, the N2 Pressure fell below 60 PSI. The watch script saw this happen, but I did not recieve the email it is supposed to send 

C1:Vac-P1_pressure reads 7e-4, which is the same as it has for the past ~2 days, so the V1 interlock worked fine. 

I've put some fresh N2 into the system, and Bob will pop in over the weekend to check it. I'll stay on top of it until Steve gets back. 

After consulting ELOGs and the 40m wiki, I reasoned it was ok to open the V1 to reconnect the turbo pump to the main IFO volume and VM1 to reconnect the RGA, and have now done so. 

Vacuum normal configuration with VM1-closed, VM2-open valve positions. Power load normal 24V 0.2A

Maglev rotation 560 Hz at room temp body temp.

"NO COMM" error message on medm screen. Gauge controller pressures are read able.

All vac comp LEDs are green. We have to reboot on Monday to enable communication.

Steve pointed out that in the aftermath of the Nitrogen running out a couple of times last week, the RGA had shut itself off thinking that there was a leak and so it was not performing the scheduled scans once a day. So the data files from the scheduled scans were empty in the /opt/rtcds/caltech/c1/scripts/RGA/logs directory. The wiki page for getting it up and running again is up-to-date, but the script RGAset.py did not exist on the c0rga machine, which the RGA is communicating with via serial port. I copied over the script RGAset.py from rossa to c0rga and ran the script on that machine - but the error flags it returned were not all 0 (indicating some error according to the manual) - so I edited the script to send just the initialize command ('IN0') and commented out the other commands, after which I got error flags which were all 0. After this, I ran a manual scan using 'RGAlogger.py', and it appears that the RGA is now able to take scans again - I'm attaching a plot of the scan results. We've saved this scan as a reference to compare against after a few days. 

[ericq, Gautam, Steve]

Following roughly the same procedure as ELOG 11354, c1vac1 and c1vac2 were rebooted. The symptoms were identical to the situation in that ELOG; c1vac1 could be pinged and telneted to, but c1vac2 was totally unresponsive. 

The only change in the linked procedure was that we did not shut down the maglev. Since I unwittingly had it running for days without V4 open while Steve was away, we now know that it can handle shorter periods of time than that...

Upon reboot, many channels were readable again, unfortunately the channels for TP2 and TP3 are still blank. We were able to return to "Vacuum normal state," but because of unknowned communication problems with VM1's interlock, we can't open VM1 for the RGA. Instead we opened VM2 to expose the RGA to the main IFO volumn, but this isn't part of the "Normal" state definite, so things currently read "Undefined state".

Precondition:

1, Pressure gauges had no communication ( NO COMM ) with c1vac2

2, Lost N2 supply on Oct 9 This triggered a normal all valve closed condition. At this point you replace N2 cylinders and manually swich valves to recreate VAC NORMAL configuration in the correct sequential order.

   The very last thing you do is open V1 gate valve.

   a, check TP2 that is the forepump of the Maglev. Foreline pressure to drypump  ~ 10- 100 mTorr, rotation speed 50 Krpm

   b, open V4 if P2 <1Torr

   c, check Maglev rotating at 560 Hz

   d, open V1 if P1 <500 mTorr

   e, check TP3 foreline, rotation speed and open V5 if P3 <1 Torr with VA6 closed

   f, open VA6 if PAN <1 Torr

  g, open annulos valves one by one , like VASE if PASE <1 Torr and so on...........Now the Current State: should read Vac Normal

3, Maglev run for 7 days with V4 closed. This encreased its foreline pressure to estimated few Torrs  and its body temp rose ~30C on the outside.

   So it was sweating and it may be back streamed.

The present RGA data is indicating that it had to be very mild.

The RGA will have better sensitivity with VM1 open and VM2 closed.

The PSL output shutter stayed open during these period is pointing  to IFO pressure stayed P1 <3 mTorr

PROBLEM: P1 and P2 plot should show nothing where there is no communication.http://nodus.ligo.caltech.edu:8080/40m/151016_182003/oct16Fpm2015.png

                   How do we check if pressure based software interlocks are working in this no communication condition?

Convectron gauge check:

Brand new 10 years old convectron gauge at atm was swapped into the place of existing gauges to see if they read close to 760 Torrs

They did reasonable well at the low end. I tried to imitate calibration and bring down the high end with little success.

( P1 and P2  were reading 7e-4 to ~660 Torr  The correction of the upper end pushed up the lower end too. I will correct this later

P3 and P4 high ends are way off )

The point is that they work.  Convectron gauges will be replaced and calibrated at the next vent.

Interlocks were not triggered during this test. I was expected to close the PSL shutter when P1 was reading 760 Torr

This hide some problem or  not understanding.

It was good to see CC1 and CC4 working at the moment

Somehow I succeded opening VM1 and closed VM2 = vacuum normal

I just hope it stays open overnight to get comparable RGA scan.

Clean comparable scan at vacuum normal. There was no backstreaming.

CC1 and CC2 are working again. Why did they start working  again ?

Q adjusted the Dataviewer so it is not chopping of data any more. Thanks.

Cold cathode gauge reading of 10 years.

The noise floor of the Rga scan is glitching less today

Glitches are gone. Rga scan is good

Rana stated yesterday that there will be a vacuum control update in the close future. Witnesses : Rich, Chris and Dave

Can you give me this in writing?

TP3 drypump replaced after 10,344 hrs at 750 mTorr foreline pressure.

The foreline pressure is 13 mTorr after 8 hrs of running, TP3: 50K rpm, 0.14 Amp with all annuloses pumped.

The annulos pressures are 0.3 - 5 mtorr

************************

Frame builder just crashed again

Amstron batteries replaced after 11 months with SP-12-5.5HR,  2 years warranty from replaceUPSbattery.com

Note: the replace battery LED did not go out ( well pasted 24 hrs ) till the self test bottom was hold down for 2-3 sec

Our last RGA scan is from February 14, 2016  We had a power outage on the 15th

Gautom has not succeded  reseting it. The old c0rga computer looks dead. Q may resurrect it, if he can?

The c0rga computer was off, I turned it on via front panel button. After running RGAset.py, RGAlogger.py seems to run. However, there are error messages in the output of the plotrgascan MATLAB script; evidiently there are some negative/bogus values in the output. 

I'll look into it more tomorrow.

It looks like the hardware reset did the trick. Previously, I had just tried ssh-ing into c0rga and rebooting it. At the time, however, Steve and I noticed that the various LEDs on the RGA unit weren't on, as they are supposed to be in the nominal operating state. Today, Steve reported that all LEDs except the RS232 one were on today, so I just tried following the steps in this elog again, looks like things are back up and running. I'm attaching a plot of the scan generated using plotrgascan MATLAB script, it looks comparable to the plot in elog 11697, which if I remember right, was acceptable.

Unless there is some reason we want to keep this c0rga machine, I will recommission one of the spare Raspberry Pis lying around to interface with the RGA scanner when I get the time...

This is a cold scan.

3-4 hrs ago we run out of nitrogen. We are back to Vacuum Normal

We have good RGA scan now. There was no scan for 3 months.

The drill room floor will be retiled Thursday, June 16. Temporary nitrogen line set up will allow emptying the hole area.

Ifo room entry will be through control room.

The retiling work has finished, Steve and I restored the N2 supply configuration to its normal state. The sequence of steps followed was:

1. Went to the X end and closed the following valves, roughly in this order: VAEE, VAEV, VABS, VABSCI, VASV, VASE, V4, V1.
2. Checked the RPM on the various turbo pump controllers to make sure they were in their nominal states
3. Disconnect the electrical connections to V1, V4, V5 and VA6 - just to make sure some spurious signal doesn't unintentionally open any of these valves while we are mucking around with the N2 supply
4. Close the valves on the N2 cylinders in the drill room. Disconnect the temporary nitrogen line (at this point, the N2 pressure to the IFO valves goes down from ~7-PSI to 0), reconnect the old supply chain, taking care that we aren't unintentionally loosening any of the Swagelock connections while unscrewing stuff
5. Replaced one of the N2 cylinders that was running low.
6. Reopen the cylinders, restore N2 pressure to IFO valves to ~70PSI.
7. Do steps 1-3 in reverse: i.e. reconnect power to all valves, open them in the reverse order we closed them while monitoring the state of the various turbo pumps. 
8. Acknowledged the error message on the C0VAC medm screen

Note: the valve isolating the RGA automatically shutoff during this work, possibly because it detected a pressure above its threshold - after checking the appropriate pressure gauges, we reopened this valve as well. 

The attached screenshot suggests that everything went as planned and that the vacuum system is back to normal...

  VENT OBJECTIVES:

1, Fix ETMX sus "jump issue"

2, First Contact clean the arms

3, Install new spare cold cathode and convectron gauges: InstruTech-Hornet

4, Install 50 mm apeture beam baffles

5, Check and clean optical quality viewport from inside

The following bullets were executed:

• Closed shutter of PSL-IR and green shutters at the ends 
• Checked jam nuts are protecting bellows 

• Check crane functionality & cleanliness last week

• Particle count must be under 10,000 counts / cf min for 0.5 micron 
• Checked all metal window covers are on. 
• Check 5 cylinders (24 cft size) of instrument grade air, called Alfa Gas 1 in stock.
• Took pictures of medm screens: sus-sum, aligned oplevs pos, alignment values and vac configuration
• Turned Oplev servos off
• Closed V1 and VM1, opened VM2
• Opened VV1 manual valve to Nitrogen cylinder and let the P1 to rise to 25 Torr
• Switched over to Instrument Grade Air Cylinder and continoed the vent with 8-10 PSI reading at the pressure regulator

We are venting the 40m IFO

I just disconnected the 6th instrument grade air cylinder from the vacuum envelope at 720 Torr. Now it will reach equilibrium through a filter as it sucks in lab air.

This is the sure way not to over pressurize the chamber.

The last RGA scan of this pumpdown 78

Pressure plot of 640 days long pd 78

CC1 cold cathode gauge was jump started with an accidental pressure glitch, that you can see on P1 plot

Vacuum Status: Chamber Open

All chamber annuloses are vented.  Vac Monitor screen is not communicating with gauges. The valve position indicator are working.

RGA is pumped by Maglev through VM2

Gautam, Koji

We replaced the right N2 bottle as it was empty.

RGA background scan

I've visited the purge clean air system at LHO Yarm mid-station with John Worden.

The system is described C981637. There is a schematic in C981637-06-V (Vol.6).pdf although the schematic has some differences (or uncorrected mistakes).

This system is intended to provide positive pressure when a soft cover is attached to a chamber door. When the door is open, the purging does not help to keep the chamber clean because the flow is too slow. This protection has to be done with overhead HEPA filters (22x5000cfm). It may be possible that this purge air helps the tube not to allow dusts to come in. But before using this, the chambers and the tubes have to be cleaned, according to John.

- Here at the site, the purge air system is started up a day before the vent. This system is used for the vent air, the purge air, and turbo foreline filling.

- Air intake (attachment 1): At the site, the air is intaken from the VEA. We want to incorporate somewhat clean air instead of dirty, dusty, outside air.

- Initial filter (attachment 2): a high volume filter before the compressors.

- The compressors (attachment 3, 4) are 5x 6 horse power air compressor each goes up to 160 psi. They are turned on and off depending on the demand of the air. Which is turned on is revolved by the controller to equalize the compressor usage hours.

- The compressed air goes through the air cooler (heat exchanger) to remove the heat by the compressor work.

- This air goes through prefilters and accumulated in the air receiver (100psi) (attachment 5). This receiver tank has an automated vent valve for periodical water drainage at the bottom.

- The accumulated air is discharged to twin drier towers (attachment 6, blue). The tower is operated by the controller (attachment 7) alternately with a period of 4min (or 10min by setting). When one of the towers is working, a humid air comes from the bottom and the dry air is discharged from the top. A part of the dry air goes into the other tower from the top to the bottom and dries the tower. There is a vent at the bottom to discharge water periodically.

- The dried air goes through 4 types of filters. After the last filter, all of the plumbing should be made of stainless steel to keep cleanliness.

- The air goes to the pressure reducing regulator (attachment 8, gray). The final flow speed at the chamber side is 50cfm max, according to John.

- The lower pressure air goes through the final filter (attachment 8, blue). As the pressure is low, this filter is big in order to keep the volume of the air flow.

- The purge air is supplied to the chamber side with KF50 (attachment 9). There is a vent valve (attachment 10) for safety and also to run a dry air for at least a day before the use to clean up the supply line. The purge line is disconnected when no in use.

- The entire system (attachment 11) and size comparison (attachment 12).

Thanks for checking this out Koji

The builder in 1996 was Process System International, Inc ( Westborough, MA ) It does not exist any longer or I just could not find them. Flow diagramm at Atm1

Should I be keep looking for a company who could quote us for building a similar smaller unit with 10 - 15 cfm flowrate?

Note: my intension with the two mobile-overhead HEPA filter was the same as John Worden's " clean air overpressured tent " at chamber entrance.

Atm2, Our unit has 650 cfm, velocity 90 fpm at resistance 0.5"    It may be enough to give a little overpressure if we seal it well to the chamber

We use to use them to minimize dirt getting inside the chanbers.

We have no number for the CFM without calculation. We can't assume a random number like 10-15

 We are pumping down. The annuloses are below 10 mTorr

The vacuum gauges are not communicating with the medm so there is no plot available.

The main volume pumping is stopped at P1 = 220 Torr  ( New SuperBee 174 Torr ) for overnight.

note: SuperBee was reading 791 Torr at atm and it was not set to 760 !

(Steve, Chris)

The pumpdown had stalled because of some ancient vacuum interlock code that prevented opening the valve V1 between the turbo pump and the main volume.

This interlock [0] compares the channels C1:Vac-P1_pressure and C1:Vac-PTP1_pressure, neither of which is functioning at the moment. The P1 channel apparently stopped reading sometime during the vent, and contained a value of ~700 torr, while the PTP1 channel contained 0. So the interlock code saw this huge apparent pressure difference and refused to move the valve.

To bypass this check, we used caput to enter a pressure of 0 for P1.

[0] /cvs/cds/caltech/state/from_luna/VacInterlock.st

P1 IFO pressure is 1 mTorr, valve configuration: vacuum normal, annulosses are pumped, RGA is off, not pumped.

THANKS to Chris !

The shutters can be opened with high power.

No communication error message still exist.

I will reboot c1vac1 and c1vac2 to get gauge communication with medm tomorrow.

The pumpdown started at 9-12-2016

The IFO is at 5e-5 Torr vacuum normal after 73 day at atm.

ITMX needed to be freed and ETMY-UL is still misbehaving occasionally.

New pirani   and cold cathode gauges added at this vent. They were baked at 100 C for 6 hrs under vacuum.

Go to the Vac Rack to read IFO pressure from the gauge itself when Vac. Monitor is blank as it is now !

IFO pressure 3.7e-5 Torr at new cold cathode InstruTech - Hornet

New items in vacuum:

1, ETMX sus tower with new baked sus wire, EP30-2 epoxied magnets, same at different locations also........ ..........  and 2 ruby wirestandoffs.

2, First Contact cleaned arm test masses only. This technic was a 1st time use in our vacuum system.

3, 50 mm ID green glass baffles at the ends

4, witness mirrors at ETMX and ITMY (old oplev mirrors) We observed a very dusty system: sides of optics, towers and tables were wiped. Hepa tents used at Y arm and BS

5, new pirani, cc gauge and 1.5" right angle valve

[ Gautam and Steve ]

c1vac1 and c1vac2 were rebooted and the gauges are communicating now. V1, VA6, V5 and V4 were closed and disconnected to avoid unexpected valve switching. All went smoothly.

The new ITcc gauge is at 1e-5 Torr as CC1   This is the gauge that should be logged in slow channel.

TP2 fore line dry pump was replaced this morning after 382 day of operation.

TP3 dry pump is very noisy, but it's pressure still 47 mTorr

The last good rga scan at vent 78  day 38

The RGA is removed for repaire. It's volume at atmophere and sealed.. P4 reading of 38 Torr is not correct.