Our last RGA scan is from February 14, 2016  We had a power outage on the 15th

Gautom has not succeded  reseting it. The old c0rga computer looks dead. Q may resurrect it, if he can?

The c0rga computer was off, I turned it on via front panel button. After running RGAset.py, RGAlogger.py seems to run. However, there are error messages in the output of the plotrgascan MATLAB script; evidiently there are some negative/bogus values in the output. 

I'll look into it more tomorrow.

It looks like the hardware reset did the trick. Previously, I had just tried ssh-ing into c0rga and rebooting it. At the time, however, Steve and I noticed that the various LEDs on the RGA unit weren't on, as they are supposed to be in the nominal operating state. Today, Steve reported that all LEDs except the RS232 one were on today, so I just tried following the steps in this elog again, looks like things are back up and running. I'm attaching a plot of the scan generated using plotrgascan MATLAB script, it looks comparable to the plot in elog 11697, which if I remember right, was acceptable.

Unless there is some reason we want to keep this c0rga machine, I will recommission one of the spare Raspberry Pis lying around to interface with the RGA scanner when I get the time...

This is a cold scan.

3-4 hrs ago we run out of nitrogen. We are back to Vacuum Normal

We have good RGA scan now. There was no scan for 3 months.

The drill room floor will be retiled Thursday, June 16. Temporary nitrogen line set up will allow emptying the hole area.

Ifo room entry will be through control room.

The retiling work has finished, Steve and I restored the N2 supply configuration to its normal state. The sequence of steps followed was:

1. Went to the X end and closed the following valves, roughly in this order: VAEE, VAEV, VABS, VABSCI, VASV, VASE, V4, V1.
2. Checked the RPM on the various turbo pump controllers to make sure they were in their nominal states
3. Disconnect the electrical connections to V1, V4, V5 and VA6 - just to make sure some spurious signal doesn't unintentionally open any of these valves while we are mucking around with the N2 supply
4. Close the valves on the N2 cylinders in the drill room. Disconnect the temporary nitrogen line (at this point, the N2 pressure to the IFO valves goes down from ~7-PSI to 0), reconnect the old supply chain, taking care that we aren't unintentionally loosening any of the Swagelock connections while unscrewing stuff
5. Replaced one of the N2 cylinders that was running low.
6. Reopen the cylinders, restore N2 pressure to IFO valves to ~70PSI.
7. Do steps 1-3 in reverse: i.e. reconnect power to all valves, open them in the reverse order we closed them while monitoring the state of the various turbo pumps. 
8. Acknowledged the error message on the C0VAC medm screen

Note: the valve isolating the RGA automatically shutoff during this work, possibly because it detected a pressure above its threshold - after checking the appropriate pressure gauges, we reopened this valve as well. 

The attached screenshot suggests that everything went as planned and that the vacuum system is back to normal...

  VENT OBJECTIVES:

1, Fix ETMX sus "jump issue"

2, First Contact clean the arms

3, Install new spare cold cathode and convectron gauges: InstruTech-Hornet

4, Install 50 mm apeture beam baffles

5, Check and clean optical quality viewport from inside

The following bullets were executed:

• Closed shutter of PSL-IR and green shutters at the ends 
• Checked jam nuts are protecting bellows 

• Check crane functionality & cleanliness last week

• Particle count must be under 10,000 counts / cf min for 0.5 micron 
• Checked all metal window covers are on. 
• Check 5 cylinders (24 cft size) of instrument grade air, called Alfa Gas 1 in stock.
• Took pictures of medm screens: sus-sum, aligned oplevs pos, alignment values and vac configuration
• Turned Oplev servos off
• Closed V1 and VM1, opened VM2
• Opened VV1 manual valve to Nitrogen cylinder and let the P1 to rise to 25 Torr
• Switched over to Instrument Grade Air Cylinder and continoed the vent with 8-10 PSI reading at the pressure regulator

We are venting the 40m IFO

I just disconnected the 6th instrument grade air cylinder from the vacuum envelope at 720 Torr. Now it will reach equilibrium through a filter as it sucks in lab air.

This is the sure way not to over pressurize the chamber.

The last RGA scan of this pumpdown 78

Pressure plot of 640 days long pd 78

CC1 cold cathode gauge was jump started with an accidental pressure glitch, that you can see on P1 plot

Vacuum Status: Chamber Open

All chamber annuloses are vented.  Vac Monitor screen is not communicating with gauges. The valve position indicator are working.

RGA is pumped by Maglev through VM2

Gautam, Koji

We replaced the right N2 bottle as it was empty.

RGA background scan

I've visited the purge clean air system at LHO Yarm mid-station with John Worden.

The system is described C981637. There is a schematic in C981637-06-V (Vol.6).pdf although the schematic has some differences (or uncorrected mistakes).

This system is intended to provide positive pressure when a soft cover is attached to a chamber door. When the door is open, the purging does not help to keep the chamber clean because the flow is too slow. This protection has to be done with overhead HEPA filters (22x5000cfm). It may be possible that this purge air helps the tube not to allow dusts to come in. But before using this, the chambers and the tubes have to be cleaned, according to John.

- Here at the site, the purge air system is started up a day before the vent. This system is used for the vent air, the purge air, and turbo foreline filling.

- Air intake (attachment 1): At the site, the air is intaken from the VEA. We want to incorporate somewhat clean air instead of dirty, dusty, outside air.

- Initial filter (attachment 2): a high volume filter before the compressors.

- The compressors (attachment 3, 4) are 5x 6 horse power air compressor each goes up to 160 psi. They are turned on and off depending on the demand of the air. Which is turned on is revolved by the controller to equalize the compressor usage hours.

- The compressed air goes through the air cooler (heat exchanger) to remove the heat by the compressor work.

- This air goes through prefilters and accumulated in the air receiver (100psi) (attachment 5). This receiver tank has an automated vent valve for periodical water drainage at the bottom.

- The accumulated air is discharged to twin drier towers (attachment 6, blue). The tower is operated by the controller (attachment 7) alternately with a period of 4min (or 10min by setting). When one of the towers is working, a humid air comes from the bottom and the dry air is discharged from the top. A part of the dry air goes into the other tower from the top to the bottom and dries the tower. There is a vent at the bottom to discharge water periodically.

- The dried air goes through 4 types of filters. After the last filter, all of the plumbing should be made of stainless steel to keep cleanliness.

- The air goes to the pressure reducing regulator (attachment 8, gray). The final flow speed at the chamber side is 50cfm max, according to John.

- The lower pressure air goes through the final filter (attachment 8, blue). As the pressure is low, this filter is big in order to keep the volume of the air flow.

- The purge air is supplied to the chamber side with KF50 (attachment 9). There is a vent valve (attachment 10) for safety and also to run a dry air for at least a day before the use to clean up the supply line. The purge line is disconnected when no in use.

- The entire system (attachment 11) and size comparison (attachment 12).

Thanks for checking this out Koji

The builder in 1996 was Process System International, Inc ( Westborough, MA ) It does not exist any longer or I just could not find them. Flow diagramm at Atm1

Should I be keep looking for a company who could quote us for building a similar smaller unit with 10 - 15 cfm flowrate?

Note: my intension with the two mobile-overhead HEPA filter was the same as John Worden's " clean air overpressured tent " at chamber entrance.

Atm2, Our unit has 650 cfm, velocity 90 fpm at resistance 0.5"    It may be enough to give a little overpressure if we seal it well to the chamber

We use to use them to minimize dirt getting inside the chanbers.

We have no number for the CFM without calculation. We can't assume a random number like 10-15

 We are pumping down. The annuloses are below 10 mTorr

The vacuum gauges are not communicating with the medm so there is no plot available.

The main volume pumping is stopped at P1 = 220 Torr  ( New SuperBee 174 Torr ) for overnight.

note: SuperBee was reading 791 Torr at atm and it was not set to 760 !

(Steve, Chris)

The pumpdown had stalled because of some ancient vacuum interlock code that prevented opening the valve V1 between the turbo pump and the main volume.

This interlock [0] compares the channels C1:Vac-P1_pressure and C1:Vac-PTP1_pressure, neither of which is functioning at the moment. The P1 channel apparently stopped reading sometime during the vent, and contained a value of ~700 torr, while the PTP1 channel contained 0. So the interlock code saw this huge apparent pressure difference and refused to move the valve.

To bypass this check, we used caput to enter a pressure of 0 for P1.

[0] /cvs/cds/caltech/state/from_luna/VacInterlock.st

P1 IFO pressure is 1 mTorr, valve configuration: vacuum normal, annulosses are pumped, RGA is off, not pumped.

THANKS to Chris !

The shutters can be opened with high power.

No communication error message still exist.

I will reboot c1vac1 and c1vac2 to get gauge communication with medm tomorrow.

The pumpdown started at 9-12-2016

The IFO is at 5e-5 Torr vacuum normal after 73 day at atm.

ITMX needed to be freed and ETMY-UL is still misbehaving occasionally.

New pirani   and cold cathode gauges added at this vent. They were baked at 100 C for 6 hrs under vacuum.

Go to the Vac Rack to read IFO pressure from the gauge itself when Vac. Monitor is blank as it is now !

IFO pressure 3.7e-5 Torr at new cold cathode InstruTech - Hornet

New items in vacuum:

1, ETMX sus tower with new baked sus wire, EP30-2 epoxied magnets, same at different locations also........ ..........  and 2 ruby wirestandoffs.

2, First Contact cleaned arm test masses only. This technic was a 1st time use in our vacuum system.

3, 50 mm ID green glass baffles at the ends

4, witness mirrors at ETMX and ITMY (old oplev mirrors) We observed a very dusty system: sides of optics, towers and tables were wiped. Hepa tents used at Y arm and BS

5, new pirani, cc gauge and 1.5" right angle valve

[ Gautam and Steve ]

c1vac1 and c1vac2 were rebooted and the gauges are communicating now. V1, VA6, V5 and V4 were closed and disconnected to avoid unexpected valve switching. All went smoothly.

The new ITcc gauge is at 1e-5 Torr as CC1   This is the gauge that should be logged in slow channel.

TP2 fore line dry pump was replaced this morning after 382 day of operation.

TP3 dry pump is very noisy, but it's pressure still 47 mTorr

The last good rga scan at vent 78  day 38

The RGA is removed for repaire. It's volume at atmophere and sealed.. P4 reading of 38 Torr is not correct.

I have completed the following non-Steve portions of the pre-vent checklist [wiki-40m.ligo.caltech.edu]

• Center all oplevs, transom QPDs (and IPPOS + IPANG if they are set up)
• Align the arm cavities for IR and align the green lasers to the arms.
• Update the SUS Driftmon values
• Reconcile all SDF differences
• Reduce input power to no more than 100mW (measured at the PSL shutter) by adjusting wave plate+PBS setup on the PSL table BEFORE the PMC. (Using the WP + PBS that already exist after the laser.)
• Replace 10% BS before MC REFL PD with Y1 mirror and lock MC at low power.
• Close shutter of PSL-IR and green shutters at the ends

All shutters are closed. Ready for Steve to check nuts and begin venting!

IFO is at atmosphere. The MC can be locked in air now.

The doors will be coming off tomorrow 8am sharp.

Do we want to install the ITM baffles?

What about the found OSEM filters?

Pumping again after 7 days at atmosphere.

BS, ITMY and OMC chambers were open only.

Checked: jam nuts, viewport covers and beam shutters.

Oplev servo turned off and medm screens shots taken.

New item in vacuum:  green shade 14 glass beam block at IR-input [ from the PSL ] viewport to block green reflection-scatter.

Reminder: viewport is not AR coated for green!

IFO pressure 1.7E-4 Torr on new not logged cold cathode gauge. P1 <7E-4 Torr

Valve configuration: vac.normal with anunulossess closed off.

TP3 was turned off with a failing drypump. It will be replaced tomorrow.

Dry pump of TP3 replaced after 9.5 months of operation.[ 45 mTorr d3 ]

The annulosses are pumped.

Valve configuration: vac normal, IFO pressure 4.5E-5 Torr [1.6E-5 Torr d3 ] on new ITcc gauge, RGA is not installed yet.

Note how fast the pressure is dropping when the vent is short.

All time stamps are blank on the MEDM screens.

Quad rods and  ionizer kit: consisting of repeller cage, anode grid, focus plate and  filament were replaced....... under repair # RGA200/12 ECA 100416-12967

The electronic ECU is not connected. It is beeing pumped at IFO ITcc 9.7E-6 Torr vacNormal

The Rga was turned on yesterday.

TP3 drypump replaced at 655 mTorr, no load, tp3 0.3A 

This seal lasted only for 33 days at  123,840 hrs

The replacement is performing well: TP3 foreline pressure is 55 mTorr, no load, tp3 0.15A at 15 min  [ 13.1 mTorr at d5 ]

Valve configuration: Vacuum Normal, ITcc 8.5E-6 Torr

The vacuum envelope ~1 C warmer today than 7 days ago. Bad peaks are coming down as normal.

Valve configuration: vacuum normal

RGA head temp: 43.5 C

Vac envelope temp: 23 C

Valve configuration: vacuum normal

Vacuum envelope: 23C

Rga head: 44C

I think this cron job is running on NODUS (our gateway) instead of our scripts machine:

*/1 * * * * /opt/rtcds/caltech/c1/scripts/Admin/n2Check.sh >> /opt/rtcds/caltech/c1/scripts/Admin/n2Check.log 2>&1

moreover this script has a 90MB log file full of not finding its channel

I wish this script was in python instead of BASH and I wish it would run on megatron instead of nodus (why can't megatron send us email too?) and I wish that this log file would get wiped out once in awhile. Currently its been spitting out errors since at least a month ago:

Tue Jan 31 14:10:02 PST 2017 : N2 Pressure:

Channel connect timed out: 'C1:Vac-N2pres' not found.

(standard_in) 1: syntax error

Valve configuration: vacuum normal

Vacuum envelope: 23.5 C

RGA head: 46.6 C

Gautam and Steve,

The medm monitor & vac control screens were totally blank since ~ May 24, 2017    Experienced vacuum knowledge is required for this job.

IDENTIFY valve configuration:

                        How to confirm valve configuration when all vac mons are blank?  Each valve has a manual-mechanical position indicator. Look at pressure readings and turbo pump controllers. VAC NORMAL configuration was confirmed based on these information.

Preparation: disconnect valves ( disconnect meaning: valve closes and stays paralized ) in this sequence VC2, VC1 power, VA6, V5, V4 & V1 power,      at ifo pressure 7.3E-6 Torr-it  ( it  = InstruTech cold cathode gauge )

                            This gauge is independent from all other rack  mounted   instrumentation and it is still not logged.

                            Switching to this valve configuration with disconnected valves will insure NOT  venting of the vacuum envelope by accidental glitching voltage drop or computer malfunction.

RESET  v1Vac1 .........in 2-3 minutes........ ( v1Vac1 - 2 )  the vac control screen started reading pressures & position

                    Connected cables to valves (meaning: valve will open if it was open before it was disconnected and it will be control able from computer ) in the following order: V4, V1 power, V5, VA6, VC2 & VC1 power,      at ifo 2E-5 Torr-it.....

                     ....vac configuration is reading VAC NORMAL,

                     ifo 7.4E-6 Torr-it

We have to hook up the it-cold cathode gauge to be monitored - logged !  this should be the substitute for the out of order CC1 pressure gauge.

While walking down to the X end to reset c1iscex I heard what I would call a "rythmic squnching" sound coming from under the turbo pump.  I would have said the sound was coming from a roughing pump, but none of them are on (as far as I can tell).

Steve maybe look into this??

Ifo pressure was 5.5 mTorr this Monday morning. The PSL shutter was still open. TP2 controller failed. Interlock closed V1, V4 and VM1

Turbo pump 2 is the fore pump of the Maglev. The pressure here was 3.9 Torr so The Maglev got warm ~38C but it was still rotating at 560 Hz normal with closed V1

Pressure plots are not available because of computer problems.

What I did:

Looked at pressures of Hornet and Super Bee  Instru Tech. Inc

Closed all annuloses and VA6,  disconnected V4 and VA6 and turned on external fan to cool Maglev

Opened V7 to pump the Maglev fore line with TP3

V1 opened manually when foreline pressure dropped to <2mTorr at P2 and the body temp of the Maglev cooled down to  25-27 C

VM1 opened at 1e-5 Torr

Valve configuration: vacuum normal with annuloses not pumped

Ifo pressure 8.5e-6 Torr -IT at 10am,  P2 foreline pressure 64 mTorr, TP3 controller 0.17A   22C  50Krpm

note: all valves open manually, interlock can only close them

PS: please call me next time you see the vacuum is not Vacuum Normal

The RGA did not shut down at the turbo pump controller failing.

Gautam and Steve,

Spare Varian turbo-V 70 controller, Model 969-9505, sn 21612 was swapped in. It is running the turbo fine @ 50Krpm but it does not allow it's V4 valve to be opened............

It turns out that TP2 @ 75Krpm will allow V4 to open and close. This must be a software issue.

So Vacuum Normal is operational if TP2 is running 75,000 rpm

We want to run at 50,000 rpm on the long term.

Note: the RS232 Dsub connector on the back of this controller is mounted 180 degrees opposite than TP3  and old failed TP2 controller

PS: controller is shipping out for repair 7-28-2017

IFO pressure 2 Torr,    PSL  shutter closed.  I'm pumping down with 2 roughing pumps with ion pump gate valves open and annulosses at atm.

The vacuum envelope was vented to 17 Torr while I was replacing the USP battery stack. More about this later.....

Do not plan on using the interferrometer tonight.  I will complete the pumpdown tomorrow morning.

Pumpdown stopped for over night  at ~  1 Torr

The roughing line disconnected.  Valves condition indicator  "moving " means that it is closed and it's cable disconnected so it can not move.

The RGA is off and VM1 is stuck.

pd80b has reached Vac Normal. IFO pressure 0.5 mTorr

We need our vauum channels back in dataviewer.

Steve and I found the previous draft of the 40m Vacuum Document. Someone in 2015 had browsed into the Docs history and then saved the old 2013 version as the current one.

We restored the version from 2014 which has all of Steve's edits. I have put that version (which is now the working copy) into the DCC:  https://dcc.ligo.org/E1500239.

The latest version is in our Google Docs place as usual. Steve is going to have a draft ready for us to ready be Tuesday, so please take a look then and we can discuss what needs doing at next Wednesday's 40m meeting.

Finally we got the cold cathode gauge working. IFO pressure 7e-6 Torr-it, vacuum normal valve configuration with all 4 ion pump gate valves closed at ~ 9e-6 Torr.  The cryo pump was also pumped yesterday to remove the accumulated outgassing build up.

The accidental vent was my mistake; made when I was replacing the battery pack of the UPS. The installed pack measured 51 V without any load. The "replace battery" warning light did not go out after the batteries were replaced. 

I then mistakenly and repeatedly pushed the "test" button to reset this, but I did not wait long enough for the batterries to get fully charged. The test put the full load on the new batteries and their charging condition got worse. I made the mistake when trying to put the load from the battery to online and pushed "O" so the power was cut and the computer rebooted to the all off condition. On top of this, I disconnected the wrong V1 cable to close V1.  As the computer rebooted it's interlock closed V1 at 17 Torr.

Never hit O on the Vacuum UPS !

Note: the " all off " configuration should be all valves closed ! This should be fixed now.

In case of  emergency  you can close V1 with disconnecting it's actuating power as shown on Atm3 if you have peumatic pressure 60 PSI 

[gautam, steve]

In the aftermath of the accidental vent, it looks like the RGA was shutdown.

We followed the instructions in this elog to restart the RGA.

Seems to be working now, Steve says we just need to wait for it to warm up before we can collect a reliable scan.

[steve, gautam]

At Rolf/Rich Abbott's request, we performed a check of the UPS today.

Steve believed that the UPS was functioning as it should, and the recent accidental vent was because the UPS batteries were insufficiently charged when the test was performed. Today, we decided to try testing the UPS.

We first closed V1, VM1 and VA6 using the MEDM screen. We prepared to pull power on all these valves by loosening the power connections (but not detaching them). [During this process, I lost the screw holding the power cord fixed to the gate valve V1 - we are looking for a replacement right now but it seems to be an odd size. It is cable tied for now.]

The battery charge indicator LEDs on the UPS indicated that the batteries were fully charged.

Next, we hit the "Test" button on the UPS - it has to be held down for ~3 seconds for the test to be actually initiated, seems to be a safety feature of the UPS. Once the test is underway, the LED indicators on the UPS will indicate that the loading is on the UPS batteries. The test itself lasts for ~5seconds, after which the UPS automatically reverts to the nominal configuration of supplying power from the main line (no additional user input is required).

In this test, one of the five battery charge indicator LEDs went off (5 ON LEDs indicate full charge).

So on the basis of this test, it would seem that the UPS is functioning as expected. It remains to be investigated if the various hardware/software interlocks in place will initiate the right sequence of valve closures when required.