[ Gautam and Steve ]
c1vac1 and c1vac2 were rebooted and the gauges are communicating now. V1, VA6, V5 and V4 were closed and disconnected to avoid unexpected valve switching. All went smoothly.
The new ITcc gauge is at 1e-5 Torr as CC1 This is the gauge that should be logged in slow channel.
TP2 fore line dry pump was replaced this morning after 382 day of operation.
TP3 dry pump is very noisy, but it's pressure still 47 mTorr
[ericq, Gautam, Steve]
Following roughly the same procedure as ELOG 11354, c1vac1 and c1vac2 were rebooted. The symptoms were identical to the situation in that ELOG; c1vac1 could be pinged and telneted to, but c1vac2 was totally unresponsive.
The only change in the linked procedure was that we did not shut down the maglev. Since I unwittingly had it running for days without V4 open while Steve was away, we now know that it can handle shorter periods of time than that...
Upon reboot, many channels were readable again, unfortunately the channels for TP2 and TP3 are still blank. We were able to return to "Vacuum normal state," but because of unknowned communication problems with VM1's interlock, we can't open VM1 for the RGA. Instead we opened VM2 to expose the RGA to the main IFO volumn, but this isn't part of the "Normal" state definite, so things currently read "Undefined state".
The last good rga scan at vent 78 day 38
RGA background scan
Vacuum Status: Chamber Open
All chamber annuloses are vented. Vac Monitor screen is not communicating with gauges. The valve position indicator are working.
RGA is pumped by Maglev through VM2
The RGA is removed for repaire. It's volume at atmophere and sealed.. P4 reading of 38 Torr is not correct.
I have completed the following non-Steve portions of the pre-vent checklist [wiki-40m.ligo.caltech.edu]
All shutters are closed. Ready for Steve to check nuts and begin venting!
IFO is at atmosphere. The MC can be locked in air now.
The doors will be coming off tomorrow 8am sharp.
Do we want to install the ITM baffles?
What about the found OSEM filters?
Pumping again after 7 days at atmosphere.
BS, ITMY and OMC chambers were open only.
Checked: jam nuts, viewport covers and beam shutters.
Oplev servo turned off and medm screens shots taken.
New item in vacuum: green shade 14 glass beam block at IR-input [ from the PSL ] viewport to block green reflection-scatter.
Reminder: viewport is not AR coated for green!
IFO pressure 1.7E-4 Torr on new not logged cold cathode gauge. P1 <7E-4 Torr
Valve configuration: vac.normal with anunulossess closed off.
TP3 was turned off with a failing drypump. It will be replaced tomorrow.
Dry pump of TP3 replaced after 9.5 months of operation.[ 45 mTorr d3 ]
The annulosses are pumped.
Valve configuration: vac normal, IFO pressure 4.5E-5 Torr [1.6E-5 Torr d3 ] on new ITcc gauge, RGA is not installed yet.
Note how fast the pressure is dropping when the vent is short.
All time stamps are blank on the MEDM screens.
Quad rods and ionizer kit: consisting of repeller cage, anode grid, focus plate and filament were replaced....... under repair # RGA200/12 ECA 100416-12967
The electronic ECU is not connected. It is beeing pumped at IFO ITcc 9.7E-6 Torr vacNormal
The Rga was turned on yesterday.
TP3 drypump replaced at 655 mTorr, no load, tp3 0.3A
This seal lasted only for 33 days at 123,840 hrs
The replacement is performing well: TP3 foreline pressure is 55 mTorr, no load, tp3 0.15A at 15 min [ 13.1 mTorr at d5 ]
Valve configuration: Vacuum Normal, ITcc 8.5E-6 Torr
The vacuum envelope ~1 C warmer today than 7 days ago. Bad peaks are coming down as normal.
Valve configuration: vacuum normal
RGA head temp: 43.5 C
Vac envelope temp: 23 C
Vacuum envelope: 23C
Rga head: 44C
I think this cron job is running on NODUS (our gateway) instead of our scripts machine:
*/1 * * * * /opt/rtcds/caltech/c1/scripts/Admin/n2Check.sh >> /opt/rtcds/caltech/c1/scripts/Admin/n2Check.log 2>&1
Based on Jenne's chiara disk usage monitoring script, I made a script that checks the N2 pressure, which will send an email to myself, Jenne, Rana, Koji, and Steve, should the pressure fall below 60psi. I also updated the chiara disk checking script to work on the new Nodus setup. I tested the two, only emailing myself, and they appear to work as expected.
The scripts are committed to the svn. Nodus' crontab now includes these two scripts, as well as the crontab backup script. (It occurs to me that the crontab backup script could be a little smarter, only backing it up if a change is made, but the archive is only a few MB, so it's probably not so important...)
moreover this script has a 90MB log file full of not finding its channel
I wish this script was in python instead of BASH and I wish it would run on megatron instead of nodus (why can't megatron send us email too?) and I wish that this log file would get wiped out once in awhile. Currently its been spitting out errors since at least a month ago:
Tue Jan 31 14:10:02 PST 2017 : N2 Pressure:
Channel connect timed out: 'C1:Vac-N2pres' not found.
(standard_in) 1: syntax error
Vacuum envelope: 23.5 C
RGA head: 46.6 C
Gautam and Steve,
The medm monitor & vac control screens were totally blank since ~ May 24, 2017 Experienced vacuum knowledge is required for this job.
IDENTIFY valve configuration:
How to confirm valve configuration when all vac mons are blank? Each valve has a manual-mechanical position indicator. Look at pressure readings and turbo pump controllers. VAC NORMAL configuration was confirmed based on these information.
Preparation: disconnect valves ( disconnect meaning: valve closes and stays paralized ) in this sequence VC2, VC1 power, VA6, V5, V4 & V1 power, at ifo pressure 7.3E-6 Torr-it ( it = InstruTech cold cathode gauge )
This gauge is independent from all other rack mounted instrumentation and it is still not logged.
Switching to this valve configuration with disconnected valves will insure NOT venting of the vacuum envelope by accidental glitching voltage drop or computer malfunction.
RESET v1Vac1 .........in 2-3 minutes........ ( v1Vac1 - 2 ) the vac control screen started reading pressures & position
Connected cables to valves (meaning: valve will open if it was open before it was disconnected and it will be control able from computer ) in the following order: V4, V1 power, V5, VA6, VC2 & VC1 power, at ifo 2E-5 Torr-it.....
....vac configuration is reading VAC NORMAL,
ifo 7.4E-6 Torr-it
We have to hook up the it-cold cathode gauge to be monitored - logged ! this should be the substitute for the out of order CC1 pressure gauge.
While walking down to the X end to reset c1iscex I heard what I would call a "rythmic squnching" sound coming from under the turbo pump. I would have said the sound was coming from a roughing pump, but none of them are on (as far as I can tell).
Steve maybe look into this??
Ifo pressure was 5.5 mTorr this Monday morning. The PSL shutter was still open. TP2 controller failed. Interlock closed V1, V4 and VM1
Turbo pump 2 is the fore pump of the Maglev. The pressure here was 3.9 Torr so The Maglev got warm ~38C but it was still rotating at 560 Hz normal with closed V1
Pressure plots are not available because of computer problems.
What I did:
Looked at pressures of Hornet and Super Bee Instru Tech. Inc
Closed all annuloses and VA6, disconnected V4 and VA6 and turned on external fan to cool Maglev
Opened V7 to pump the Maglev fore line with TP3
V1 opened manually when foreline pressure dropped to <2mTorr at P2 and the body temp of the Maglev cooled down to 25-27 C
VM1 opened at 1e-5 Torr
Valve configuration: vacuum normal with annuloses not pumped
Ifo pressure 8.5e-6 Torr -IT at 10am, P2 foreline pressure 64 mTorr, TP3 controller 0.17A 22C 50Krpm
note: all valves open manually, interlock can only close them
PS: please call me next time you see the vacuum is not Vacuum Normal
The RGA did not shut down at the turbo pump controller failing.
Ifo pressure was 5.5 mTorr this morning. The PSL shutter was still open. TP2 controller failed. Interlock closed V1, V4 and VM1
Turbo pump 2 is the fore pump of the Maglev. The pressure here was 3.9 Torr so The Magelv got warm ~38C but it was still rotating at 560 Hz normal with closed V1
Spare Varian turbo-V 70 controller, Model 969-9505, sn 21612 was swapped in. It is running the turbo fine @ 50Krpm but it does not allow it's V4 valve to be opened............
It turns out that TP2 @ 75Krpm will allow V4 to open and close. This must be a software issue.
So Vacuum Normal is operational if TP2 is running 75,000 rpm
We want to run at 50,000 rpm on the long term.
Note: the RS232 Dsub connector on the back of this controller is mounted 180 degrees opposite than TP3 and old failed TP2 controller
PS: controller is shipping out for repair 7-28-2017
IFO pressure 2 Torr, PSL shutter closed. I'm pumping down with 2 roughing pumps with ion pump gate valves open and annulosses at atm.
The vacuum envelope was vented to 17 Torr while I was replacing the USP battery stack. More about this later.....
Do not plan on using the interferrometer tonight. I will complete the pumpdown tomorrow morning.
Pumpdown stopped for over night at ~ 1 Torr
The roughing line disconnected. Valves condition indicator "moving " means that it is closed and it's cable disconnected so it can not move.
The RGA is off and VM1 is stuck.
pd80b has reached Vac Normal. IFO pressure 0.5 mTorr
We need our vauum channels back in dataviewer.
Steve and I found the previous draft of the 40m Vacuum Document. Someone in 2015 had browsed into the Docs history and then saved the old 2013 version as the current one.
We restored the version from 2014 which has all of Steve's edits. I have put that version (which is now the working copy) into the DCC: https://dcc.ligo.org/E1500239.
The latest version is in our Google Docs place as usual. Steve is going to have a draft ready for us to ready be Tuesday, so please take a look then and we can discuss what needs doing at next Wednesday's 40m meeting.
Finally we got the cold cathode gauge working. IFO pressure 7e-6 Torr-it, vacuum normal valve configuration with all 4 ion pump gate valves closed at ~ 9e-6 Torr. The cryo pump was also pumped yesterday to remove the accumulated outgassing build up.
The accidental vent was my mistake; made when I was replacing the battery pack of the UPS. The installed pack measured 51 V without any load. The "replace battery" warning light did not go out after the batteries were replaced.
I then mistakenly and repeatedly pushed the "test" button to reset this, but I did not wait long enough for the batterries to get fully charged. The test put the full load on the new batteries and their charging condition got worse. I made the mistake when trying to put the load from the battery to online and pushed "O" so the power was cut and the computer rebooted to the all off condition. On top of this, I disconnected the wrong V1 cable to close V1. As the computer rebooted it's interlock closed V1 at 17 Torr.
Never hit O on the Vacuum UPS !
Note: the " all off " configuration should be all valves closed ! This should be fixed now.
In case of emergency you can close V1 with disconnecting it's actuating power as shown on Atm3 if you have peumatic pressure 60 PSI
In the aftermath of the accidental vent, it looks like the RGA was shutdown.
We followed the instructions in this elog to restart the RGA.
Seems to be working now, Steve says we just need to wait for it to warm up before we can collect a reliable scan.
We have good RGA scan now. There was no scan for 3 months.
At Rolf/Rich Abbott's request, we performed a check of the UPS today.
Steve believed that the UPS was functioning as it should, and the recent accidental vent was because the UPS batteries were insufficiently charged when the test was performed. Today, we decided to try testing the UPS.
We first closed V1, VM1 and VA6 using the MEDM screen. We prepared to pull power on all these valves by loosening the power connections (but not detaching them). [During this process, I lost the screw holding the power cord fixed to the gate valve V1 - we are looking for a replacement right now but it seems to be an odd size. It is cable tied for now.]
The battery charge indicator LEDs on the UPS indicated that the batteries were fully charged.
Next, we hit the "Test" button on the UPS - it has to be held down for ~3 seconds for the test to be actually initiated, seems to be a safety feature of the UPS. Once the test is underway, the LED indicators on the UPS will indicate that the loading is on the UPS batteries. The test itself lasts for ~5seconds, after which the UPS automatically reverts to the nominal configuration of supplying power from the main line (no additional user input is required).
In this test, one of the five battery charge indicator LEDs went off (5 ON LEDs indicate full charge).
So on the basis of this test, it would seem that the UPS is functioning as expected. It remains to be investigated if the various hardware/software interlocks in place will initiate the right sequence of valve closures when required.
The V1 gate valve specs installed at 40m wiki page. VAT model number 10846-UE44-0007 Our main volume pumping goes through this 8" id gate valve V1 to Maglev turbo or Cryo pump to VC1
The ion pumps have 6" id gate valves:VAT 10844-UE44-AAY1, Pneumatic actuator with position indicator and double acting solenoid valve 115V 60Hz Purchased 1999 Dec 22
UHV gate valves 2.5" id. VAT 10836-UE44 Pneumatic actuator with position indicator and double acting solenoid valve 115V 60 hz, IFO to RGA VM1 & RGA to Maglev VM2
mini UHV gate valve 1.5" id. VAT 01032-UE01 2016 cataloge page 14, manual - no position indicator, VM4 next to manual adjustable fine leak valve to RGA
UHV angle valve 1.5" id, model VAT 28432-GE41, Viton plate seal, pneumatic actuator with position indicator & solenoid valve 115V & single acting closing spring MEDM screen: VM3,VC2, V3,V4,V5,V6,VA6,V7 & annuloses Each chamber annulos has 2 valves.
UHV angle valve 1.5" id, model VAT 57132-GE05 go page 208, Metal tip seal, manual actuating only with position indicator, MEDM screen: roughing RV1 and venting VV1 hand wheel needed to close to torque spec
UHV angle valve 1.5" id. model VAT 28432-GE01 Viton plate seal, manual operation only at IT gauges Hornet & Super Bee and ion pumps roughing ports. These are not labeled.
The Cryo pump interlock wiring was added too
Note: all moving valve plate seals are single.
The RGA was turned on 7 days ago. It's 46 C now. The X-arm room tem ~20 C
IFO pressure 6.5e-6 Torr at IT-Hornet gauge. Valve configuration vacuum normal.
RP1 and RP3 roughing pump manual of Leybold D30A oily rotory pump
Fore pump of TP2 & TP3 Varian SH-100 Dry Scroll
TP2 and TP3 small turbo drag pump Varian 969-9361
TP2 and TP3 turbo controller Varian 969-9505
TP1 magnetically suspended turbo pump Osaka TG390MCAB, sn360 and controller TC010M and note : this pump running on 208VAC single phaseIt is not on the UPS !
Osaka Maglev Manual and Osaka Controller Communication Wiring
VC1 cryo pump CTI-Cryogenics Cryo Torr 8 sn 8g23925 SAFETY note: compressor single phase 208VAC and the head driver 3 phase 208VAC Compressor and driver have each separate power cord!
Installed at 40m wiki also
Valve configuration: Vacuum normal
Note: Tp2 running at 75Krpm 0.25A 26C has a load high pitch sound today. It's fore line pressure 78 mTorr. Room temp 20C
IFO pressure 1.2e-5 Torr at 9:30am
Atm. 1, This was the vacuum condition this morning.
IFO P1 9.7 mTorr, V1 open, V4 was in closed position , ~37 C warm Maglev at normal 560Hz rotation speed with foreline pressure 3.9 Torr because V4 closed 2 days ago when TP2 failed .....see Atm.3
The error messege at TP2 controller was: fault overtemp.
I did the following to restored IFO pumping: stopped pumping of the annulose with TP3 and valves were configured so TP3 can be the forepump of the Maglev.
closed VM1 to protect the RGA, close PSL shutter .....see Gautam entry
aux fan on to cool down Maglev-TP1, room temp 20 C,
aux drypump turned on and opend to TP3 foreline to gain pumping speed,
closed PAN to isolate annulos pumping,
opened V7 to pump Maglev forline with TP3 running at 50Krpm, It took 10 minutes to reach P2 1mTorr from 3.9 Torr
aux drypump closed off at P2 1 mTorr, TP3 foreline pressure 362 mTorr.......see Atm.2
As we are running now:
IFO pressure 7e-6 Torr at Hornet cold cathode gauge at 15:50 We have no IFO CC1 logging now. Annuloses are in 3-5 mTorr range are not pumped.
TP3 as foreline pump of TP1 at 50 Krpm, 0.24 A, 24 C, it's drypump forline pressure 324 mTorr
V4 valve cable is disconnected.
I need help with wiring up the logging of the Hornet cold cathode gauge.
Our new Agilent Technology TwisTorr 84FS AG rack controller ( English Manual pages 195-297 ) RS232/485, product number X3508-64001, sn IT1737C383
This controller, turbo and it's drypump needs to be set up into our existing vacuum system. The intake valve of this turbo (V4) has to have a hardwired interlock that closes V4 when rotation speed is less than 20% of preset RPM speed.
The unit has an analoge 10Vdc output that is proportional to rotation speed. This can be used with a comperator to direct the interlock or there may be set software option in the controller to close the valve if the turbo slows down more than 20%
The last Upgrade of the 40m Vacuum System 1/2/2000 discribes our vauum system LIGO-T000054-00-R
Here the LabView / Metrabus controls were replaced by VME processor and an Epic interface
We do not have schematics of the hardware wiring.
We need help with this.
PSL shutter closed at 6e-6 Torr-it
The foreline pressure of the drypump is 850 mTorr at 8,446 hrs of seal life
V1 will be closed for ~20 minutes for drypump replacement..........
9:30am dry pump replaced, PSL shutter opened at 7.7E-6 Torr-it
Valve configuration: vacuum normal as TP3 is the forepump of the Maglev & annuloses are not pumped.
The TP3 foreline pressure was 4.8 Torr, 50K rpm 0.54A and 31C........Maglev rotation normal 560 Hz....... IFO pressure 7.2e- 6 Torrit was not effected
V1 closed ......replaced drypump.........V1 opened
IFO 6.9e-6 Torrit at 19:55, TP3fl 18 mT, 50Krpm 0.15A 24C
VM1 is still closed
Annuloses are not pumped for 30 days, since TP2 failed. IFO pressure 7e-6 Torr it, Rga 2.6e-6 Torr
Valve configuration: Vacuum Normal as TP3 is the forepump of Maglev, annuloses are not puped at 1.1 Torr
TP3 50K rpm, 0.15A 24C, foreline pressure 16.1 mTorr
There was a power outage.
The IFO pressure is 12.8 mTorr-it and it is not pumped. V1 is still closed. TP1 is not running. The Rga is not powered.
The PSL output shutter is still closed. 2W Innolight turned on and manual beam block placed in its beampath.
3 AC units turned on at room temp 84F
IFO pumped down from 44 mTorr to 9.6e-6 Torr with Maglev backed with only TP3
Aux drypump was helping our std drypump during this 1 hour period. TP3 reached 32 C and slowed down 47K rpm
The peak foreline pressure at P2 was ~3 Torr
Hornet cold cathode gauge setting: research mode, air,
2830 HV 1e-4A at 9.6e-6 Torr,
[ 3110 HV 8e-5A at 7.4e-6 Torr one day later ]
Annuloses are at 2 Torr, not pumped
Valve configuration: vacuum normal, RGA is still off
PSL shutter is opened automatically. Manual block removed.
End IR lasers and doublers are turned on.
NOTE: Maglev " rotation X " on vacuum medm screen is not working! " C1:Vac-TP1_rot " channel was removed. Use " NORMAL X " for rotation monitoring.
*We removed this (i.e. rotation) field from the MEDM screen to avoid confusion.
pd80b rga scan at 175 day. IFO pressure 7.3e-6 Torr-it
Condition: vacuum normal, annuloses not pumped. Rga turned on yesterday.
The rga was not on since last poweroutage Jan 2, 2018 It is warming up and outgassing Atm2
Hornet cold cathode gauge analoge output [ DSub9 pin 3 and 7 ] are wired to go ETMX Acromag. It was reading 4.9V at 7.8e-6 Torr [ 3,110 V 8.35e-5A ] at the end of a 24ft BNC cable. Now it has to be hook up to an Acromag channel.
This will replace the not functioning C1: Vac-CC1_pressure
gautam: the motivation behind hooking this gauge up to our DAQ system is that non-vacuum-system-experts have a quick diagnostic to make sure everything is in order. This gauge is physically placed adjacent to V1, and so if something goes wrong with our vacuum pumps, we would see the effect here immediately. we did note that occassionally, the reading fluctuated by ~1V on the DMM used to check the voltage output at the end of the BNC cable, so we still need to run some long-term stability analysis once this channel is hooked up to the Acromag. For future reference, in order to make this gauge work, we need to check that
The forline pressure of TP3 was 399 mTorr
It was replaced this morning at TP3 controller 134,638hrs with the "failed TP2 station" drypump. The foreline pressure now at TP3 is 100 mTorr at 6 hrs of operation.[ at day 3 63 mT ]
IFO pressure at CC Hornet 7.9e - 6 Torr
Valve configuration: vacuum normal as TP3 is the forepump of the Maglev & the annuloses are not pumped
I wired the six available BNC connectors on the front panel of the new XEND slow DAQ to physical Acromag channels. There were two unused ADC channels and eight DAC channels, of which I connected four. The following entries were added to /cvs/cds/caltech/target/c1auxex2/ETMXAUX2.db /caltech/target/c1auxex2/ETMXaux2.db
C1:Vac-CC1_HORNET_PRESSURE_VOLT is converted to the additional soft channel C1:Vac-CC1_HORNET_PRESSURE in units of torr using the conversion stated in the manual. A quick check showed that the resulting number and the displayed pressure on the vacuum gauge itself agree to ~1e-8 torr. Gautam added the new EPICS calc channel to the C0EDCU and restarted FB, now the data is being recorded.
Three of the output channels do not have a purpose yet, so their epics records were created but remain inactive for the time being.
We have the IFO pressure logged again! Thanks Johannes and Gautam
This InstruTech cold cathode ionization vacuum gauge " Hornet " was installed 2016 Sep 14
Here is the CC1 gauge history of 10 years from 2015 Dec 1
The next thing to do is put this channel C1:Vac-CC1_HORNET_PRESSURE on the 40m Vacuum System Monitor [ COVAC_MONITOR.adl ]
gautam 1pm: Vac MEDM screen monitor has been edited to change the readback channel for the CC1 pressure field - see Attachment #2. Seems to work okay.
In Steve's absence, I've tried to keep an eye on the health of the vacuum system. From Attachment #1, the pressure of the main volume seems stable, no red flags there. I also don't here any anomalously loud sounds near the vacuum pumps. I've changed the N2 cylinders that keep V1 open twice, on Wednesday and Sunday of last week. So in summary, the vacuum system looks fine based on all the metrics I know of.
Pumpdown 80 at 511 days and pd80b at 218 days
Valve configuration: special vacuum normal, annuloses are not pumped at 3 Torr, IFO pressure 7.4e-6 Torr at vac envelope temp 22 +- 1C degrees
CC1 old MKS cold cathode gauge randomly turns on- off. This makes software interlock close VM1 to protect RGA So the closed off RGA region pressure goes up and the result is distorted RGA scan.
CC1 MKS gauge is disconnected and VM1 opened. This reminds me that we should connect our interlocks to CC1 Hornet Pressure gauge.
Valve configuration: special vacuum normal, annuloses are not pumped at 3 Torr, IFO pressure 7.4e-6 Torr at vac envelope temp 22 +- 1C degrres