Vacuum normal configuration with VM1-closed, VM2-open valve positions. Power load normal 24V 0.2A
Maglev rotation 560 Hz at room temp body temp.
"NO COMM" error message on medm screen. Gauge controller pressures are read able.
All vac comp LEDs are green. We have to reboot on Monday to enable communication.
Steve pointed out that in the aftermath of the Nitrogen running out a couple of times last week, the RGA had shut itself off thinking that there was a leak and so it was not performing the scheduled scans once a day. So the data files from the scheduled scans were empty in the /opt/rtcds/caltech/c1/scripts/RGA/logs directory. The wiki page for getting it up and running again is up-to-date, but the script RGAset.py did not exist on the c0rga machine, which the RGA is communicating with via serial port. I copied over the script RGAset.py from rossa to c0rga and ran the script on that machine - but the error flags it returned were not all 0 (indicating some error according to the manual) - so I edited the script to send just the initialize command ('IN0') and commented out the other commands, after which I got error flags which were all 0. After this, I ran a manual scan using 'RGAlogger.py', and it appears that the RGA is now able to take scans again - I'm attaching a plot of the scan results. We've saved this scan as a reference to compare against after a few days.
[ericq, Gautam, Steve]
Following roughly the same procedure as ELOG 11354, c1vac1 and c1vac2 were rebooted. The symptoms were identical to the situation in that ELOG; c1vac1 could be pinged and telneted to, but c1vac2 was totally unresponsive.
The only change in the linked procedure was that we did not shut down the maglev. Since I unwittingly had it running for days without V4 open while Steve was away, we now know that it can handle shorter periods of time than that...
Upon reboot, many channels were readable again, unfortunately the channels for TP2 and TP3 are still blank. We were able to return to "Vacuum normal state," but because of unknowned communication problems with VM1's interlock, we can't open VM1 for the RGA. Instead we opened VM2 to expose the RGA to the main IFO volumn, but this isn't part of the "Normal" state definite, so things currently read "Undefined state".
1, Pressure gauges had no communication ( NO COMM ) with c1vac2
2, Lost N2 supply on Oct 9 This triggered a normal all valve closed condition. At this point you replace N2 cylinders and manually swich valves to recreate VAC NORMAL configuration in the correct sequential order.
The very last thing you do is open V1 gate valve.
a, check TP2 that is the forepump of the Maglev. Foreline pressure to drypump ~ 10- 100 mTorr, rotation speed 50 Krpm
b, open V4 if P2 <1Torr
c, check Maglev rotating at 560 Hz
d, open V1 if P1 <500 mTorr
e, check TP3 foreline, rotation speed and open V5 if P3 <1 Torr with VA6 closed
f, open VA6 if PAN <1 Torr
g, open annulos valves one by one , like VASE if PASE <1 Torr and so on...........Now the Current State: should read Vac Normal
3, Maglev run for 7 days with V4 closed. This encreased its foreline pressure to estimated few Torrs and its body temp rose ~30C on the outside.
So it was sweating and it may be back streamed.
The present RGA data is indicating that it had to be very mild.
The RGA will have better sensitivity with VM1 open and VM2 closed.
The PSL output shutter stayed open during these period is pointing to IFO pressure stayed P1 <3 mTorr
PROBLEM: P1 and P2 plot should show nothing where there is no communication.http://nodus.ligo.caltech.edu:8080/40m/151016_182003/oct16Fpm2015.png
How do we check if pressure based software interlocks are working in this no communication condition?
Convectron gauge check:
Brand new 10 years old convectron gauge at atm was swapped into the place of existing gauges to see if they read close to 760 Torrs
They did reasonable well at the low end. I tried to imitate calibration and bring down the high end with little success.
( P1 and P2 were reading 7e-4 to ~660 Torr The correction of the upper end pushed up the lower end too. I will correct this later
P3 and P4 high ends are way off )
The point is that they work. Convectron gauges will be replaced and calibrated at the next vent.
Interlocks were not triggered during this test. I was expected to close the PSL shutter when P1 was reading 760 Torr
This hide some problem or not understanding.
It was good to see CC1 and CC4 working at the moment
Somehow I succeded opening VM1 and closed VM2 = vacuum normal
I just hope it stays open overnight to get comparable RGA scan.
Clean comparable scan at vacuum normal. There was no backstreaming.
CC1 and CC2 are working again. Why did they start working again ?
Q adjusted the Dataviewer so it is not chopping of data any more. Thanks.
Cold cathode gauge reading of 10 years.
The noise floor of the Rga scan is glitching less today
Glitches are gone. Rga scan is good
Rana stated yesterday that there will be a vacuum control update in the close future. Witnesses : Rich, Chris and Dave
Can you give me this in writing?
TP3 drypump replaced after 10,344 hrs at 750 mTorr foreline pressure.
The foreline pressure is 13 mTorr after 8 hrs of running, TP3: 50K rpm, 0.14 Amp with all annuloses pumped.
The annulos pressures are 0.3 - 5 mtorr
Frame builder just crashed again
Amstron batteries replaced after 11 months with SP-12-5.5HR, 2 years warranty from replaceUPSbattery.com
Batteries replaced after 3.5 years with Amstron AP-1250F2, 8x 12V 6Ah
APC Smart -UPS 2200 model: SUA2200RM2U batteries were replaced by compatible RBC43, 8x 12V5A
Note: the replace battery LED did not go out ( well pasted 24 hrs ) till the self test bottom was hold down for 2-3 sec
Our last RGA scan is from February 14, 2016 We had a power outage on the 15th
Gautom has not succeded reseting it. The old c0rga computer looks dead. Q may resurrect it, if he can?
The c0rga computer was off, I turned it on via front panel button. After running RGAset.py, RGAlogger.py seems to run. However, there are error messages in the output of the plotrgascan MATLAB script; evidiently there are some negative/bogus values in the output.
I'll look into it more tomorrow.
It looks like the hardware reset did the trick. Previously, I had just tried ssh-ing into c0rga and rebooting it. At the time, however, Steve and I noticed that the various LEDs on the RGA unit weren't on, as they are supposed to be in the nominal operating state. Today, Steve reported that all LEDs except the RS232 one were on today, so I just tried following the steps in this elog again, looks like things are back up and running. I'm attaching a plot of the scan generated using plotrgascan MATLAB script, it looks comparable to the plot in elog 11697, which if I remember right, was acceptable.
Unless there is some reason we want to keep this c0rga machine, I will recommission one of the spare Raspberry Pis lying around to interface with the RGA scanner when I get the time...
This is a cold scan.
3-4 hrs ago we run out of nitrogen. We are back to Vacuum Normal
We have good RGA scan now. There was no scan for 3 months.
The drill room floor will be retiled Thursday, June 16. Temporary nitrogen line set up will allow emptying the hole area.
Ifo room entry will be through control room.
The retiling work has finished, Steve and I restored the N2 supply configuration to its normal state. The sequence of steps followed was:
Note: the valve isolating the RGA automatically shutoff during this work, possibly because it detected a pressure above its threshold - after checking the appropriate pressure gauges, we reopened this valve as well.
The attached screenshot suggests that everything went as planned and that the vacuum system is back to normal...
1, Fix ETMX sus "jump issue"
2, First Contact clean the arms
3, Install new spare cold cathode and convectron gauges: InstruTech-Hornet
4, Install 50 mm apeture beam baffles
5, Check and clean optical quality viewport from inside
The following bullets were executed:
Check crane functionality & cleanliness last week
We are venting the 40m IFO
Steve has ordered some teflon parts to take the place of the metal parts in his acetone-soaking jig. They should arrive tomorrow.
So, we will be begin the venting process tomorrow. Doors to come off on Tuesday.
I just disconnected the 6th instrument grade air cylinder from the vacuum envelope at 720 Torr. Now it will reach equilibrium through a filter as it sucks in lab air.
This is the sure way not to over pressurize the chamber.
The last RGA scan of this pumpdown 78
Pressure plot of 640 days long pd 78
CC1 cold cathode gauge was jump started with an accidental pressure glitch, that you can see on P1 plot
Vacuum Status: Chamber Open
All chamber annuloses are vented. Vac Monitor screen is not communicating with gauges. The valve position indicator are working.
RGA is pumped by Maglev through VM2
We replaced the right N2 bottle as it was empty.
RGA background scan
I've visited the purge clean air system at LHO Yarm mid-station with John Worden.
The system is described C981637. There is a schematic in C981637-06-V (Vol.6).pdf although the schematic has some differences (or uncorrected mistakes).
This system is intended to provide positive pressure when a soft cover is attached to a chamber door. When the door is open, the purging does not help to keep the chamber clean because the flow is too slow. This protection has to be done with overhead HEPA filters (22x5000cfm). It may be possible that this purge air helps the tube not to allow dusts to come in. But before using this, the chambers and the tubes have to be cleaned, according to John.
- Here at the site, the purge air system is started up a day before the vent. This system is used for the vent air, the purge air, and turbo foreline filling.
- Air intake (attachment 1): At the site, the air is intaken from the VEA. We want to incorporate somewhat clean air instead of dirty, dusty, outside air.
- Initial filter (attachment 2): a high volume filter before the compressors.
- The compressors (attachment 3, 4) are 5x 6 horse power air compressor each goes up to 160 psi. They are turned on and off depending on the demand of the air. Which is turned on is revolved by the controller to equalize the compressor usage hours.
- The compressed air goes through the air cooler (heat exchanger) to remove the heat by the compressor work.
- This air goes through prefilters and accumulated in the air receiver (100psi) (attachment 5). This receiver tank has an automated vent valve for periodical water drainage at the bottom.
- The accumulated air is discharged to twin drier towers (attachment 6, blue). The tower is operated by the controller (attachment 7) alternately with a period of 4min (or 10min by setting). When one of the towers is working, a humid air comes from the bottom and the dry air is discharged from the top. A part of the dry air goes into the other tower from the top to the bottom and dries the tower. There is a vent at the bottom to discharge water periodically.
- The dried air goes through 4 types of filters. After the last filter, all of the plumbing should be made of stainless steel to keep cleanliness.
- The air goes to the pressure reducing regulator (attachment 8, gray). The final flow speed at the chamber side is 50cfm max, according to John.
- The lower pressure air goes through the final filter (attachment 8, blue). As the pressure is low, this filter is big in order to keep the volume of the air flow.
- The purge air is supplied to the chamber side with KF50 (attachment 9). There is a vent valve (attachment 10) for safety and also to run a dry air for at least a day before the use to clean up the supply line. The purge line is disconnected when no in use.
- The entire system (attachment 11) and size comparison (attachment 12).
Thanks for checking this out Koji
The builder in 1996 was Process System International, Inc ( Westborough, MA ) It does not exist any longer or I just could not find them. Flow diagramm at Atm1
Should I be keep looking for a company who could quote us for building a similar smaller unit with 10 - 15 cfm flowrate?
Note: my intension with the two mobile-overhead HEPA filter was the same as John Worden's " clean air overpressured tent " at chamber entrance.
Atm2, Our unit has 650 cfm, velocity 90 fpm at resistance 0.5" It may be enough to give a little overpressure if we seal it well to the chamber
We use to use them to minimize dirt getting inside the chanbers.
We have no number for the CFM without calculation. We can't assume a random number like 10-15
We are pumping down. The annuloses are below 10 mTorr
The vacuum gauges are not communicating with the medm so there is no plot available.
The main volume pumping is stopped at P1 = 220 Torr ( New SuperBee 174 Torr ) for overnight.
note: SuperBee was reading 791 Torr at atm and it was not set to 760 !
The pumpdown had stalled because of some ancient vacuum interlock code that prevented opening the valve V1 between the turbo pump and the main volume.
This interlock  compares the channels C1:Vac-P1_pressure and C1:Vac-PTP1_pressure, neither of which is functioning at the moment. The P1 channel apparently stopped reading sometime during the vent, and contained a value of ~700 torr, while the PTP1 channel contained 0. So the interlock code saw this huge apparent pressure difference and refused to move the valve.
To bypass this check, we used caput to enter a pressure of 0 for P1.
P1 IFO pressure is 1 mTorr, valve configuration: vacuum normal, annulosses are pumped, RGA is off, not pumped.
THANKS to Chris !
The shutters can be opened with high power.
No communication error message still exist.
I will reboot c1vac1 and c1vac2 to get gauge communication with medm tomorrow.
The pumpdown started at 9-12-2016
The IFO is at 5e-5 Torr vacuum normal after 73 day at atm.
ITMX needed to be freed and ETMY-UL is still misbehaving occasionally.
New pirani and cold cathode gauges added at this vent. They were baked at 100 C for 6 hrs under vacuum.
Go to the Vac Rack to read IFO pressure from the gauge itself when Vac. Monitor is blank as it is now !
P1 IFO pressure is 1 mTorr, valve configuration: vacuum normal, annulosses are pumped, RGA is not pumped.
IFO pressure 3.7e-5 Torr at new cold cathode InstruTech - Hornet
New items in vacuum:
1, ETMX sus tower with new baked sus wire, EP30-2 epoxied magnets, same at different locations also........ .......... and 2 ruby wirestandoffs.
2, First Contact cleaned arm test masses only. This technic was a 1st time use in our vacuum system.
3, 50 mm ID green glass baffles at the ends
4, witness mirrors at ETMX and ITMY (old oplev mirrors) We observed a very dusty system: sides of optics, towers and tables were wiped. Hepa tents used at Y arm and BS
5, new pirani, cc gauge and 1.5" right angle valve
[ Gautam and Steve ]
c1vac1 and c1vac2 were rebooted and the gauges are communicating now. V1, VA6, V5 and V4 were closed and disconnected to avoid unexpected valve switching. All went smoothly.
The new ITcc gauge is at 1e-5 Torr as CC1 This is the gauge that should be logged in slow channel.
TP2 fore line dry pump was replaced this morning after 382 day of operation.
TP3 dry pump is very noisy, but it's pressure still 47 mTorr
The last good rga scan at vent 78 day 38
The RGA is removed for repaire. It's volume at atmophere and sealed.. P4 reading of 38 Torr is not correct.
I have completed the following non-Steve portions of the pre-vent checklist [wiki-40m.ligo.caltech.edu]
All shutters are closed. Ready for Steve to check nuts and begin venting!
IFO is at atmosphere. The MC can be locked in air now.
The doors will be coming off tomorrow 8am sharp.
Do we want to install the ITM baffles?
What about the found OSEM filters?
Pumping again after 7 days at atmosphere.
BS, ITMY and OMC chambers were open only.
Checked: jam nuts, viewport covers and beam shutters.
Oplev servo turned off and medm screens shots taken.
New item in vacuum: green shade 14 glass beam block at IR-input [ from the PSL ] viewport to block green reflection-scatter.
Reminder: viewport is not AR coated for green!
IFO pressure 1.7E-4 Torr on new not logged cold cathode gauge. P1 <7E-4 Torr
Valve configuration: vac.normal with anunulossess closed off.
TP3 was turned off with a failing drypump. It will be replaced tomorrow.
Dry pump of TP3 replaced after 9.5 months of operation.[ 45 mTorr d3 ]
The annulosses are pumped.
Valve configuration: vac normal, IFO pressure 4.5E-5 Torr [1.6E-5 Torr d3 ] on new ITcc gauge, RGA is not installed yet.
Note how fast the pressure is dropping when the vent is short.
All time stamps are blank on the MEDM screens.
Quad rods and ionizer kit: consisting of repeller cage, anode grid, focus plate and filament were replaced....... under repair # RGA200/12 ECA 100416-12967
The electronic ECU is not connected. It is beeing pumped at IFO ITcc 9.7E-6 Torr vacNormal
The Rga was turned on yesterday.
TP3 drypump replaced at 655 mTorr, no load, tp3 0.3A
This seal lasted only for 33 days at 123,840 hrs
The replacement is performing well: TP3 foreline pressure is 55 mTorr, no load, tp3 0.15A at 15 min [ 13.1 mTorr at d5 ]
Valve configuration: Vacuum Normal, ITcc 8.5E-6 Torr
The vacuum envelope ~1 C warmer today than 7 days ago. Bad peaks are coming down as normal.
Valve configuration: vacuum normal
RGA head temp: 43.5 C
Vac envelope temp: 23 C
Vacuum envelope: 23C
Rga head: 44C
I think this cron job is running on NODUS (our gateway) instead of our scripts machine:
*/1 * * * * /opt/rtcds/caltech/c1/scripts/Admin/n2Check.sh >> /opt/rtcds/caltech/c1/scripts/Admin/n2Check.log 2>&1
Based on Jenne's chiara disk usage monitoring script, I made a script that checks the N2 pressure, which will send an email to myself, Jenne, Rana, Koji, and Steve, should the pressure fall below 60psi. I also updated the chiara disk checking script to work on the new Nodus setup. I tested the two, only emailing myself, and they appear to work as expected.
The scripts are committed to the svn. Nodus' crontab now includes these two scripts, as well as the crontab backup script. (It occurs to me that the crontab backup script could be a little smarter, only backing it up if a change is made, but the archive is only a few MB, so it's probably not so important...)
moreover this script has a 90MB log file full of not finding its channel
I wish this script was in python instead of BASH and I wish it would run on megatron instead of nodus (why can't megatron send us email too?) and I wish that this log file would get wiped out once in awhile. Currently its been spitting out errors since at least a month ago:
Tue Jan 31 14:10:02 PST 2017 : N2 Pressure:
Channel connect timed out: 'C1:Vac-N2pres' not found.
(standard_in) 1: syntax error