[ericq, Gautam, Steve]
Following roughly the same procedure as ELOG 11354, c1vac1 and c1vac2 were rebooted. The symptoms were identical to the situation in that ELOG; c1vac1 could be pinged and telneted to, but c1vac2 was totally unresponsive.
The only change in the linked procedure was that we did not shut down the maglev. Since I unwittingly had it running for days without V4 open while Steve was away, we now know that it can handle shorter periods of time than that...
Upon reboot, many channels were readable again, unfortunately the channels for TP2 and TP3 are still blank. We were able to return to "Vacuum normal state," but because of unknowned communication problems with VM1's interlock, we can't open VM1 for the RGA. Instead we opened VM2 to expose the RGA to the main IFO volumn, but this isn't part of the "Normal" state definite, so things currently read "Undefined state".
[ Gautam and Steve ]
c1vac1 and c1vac2 were rebooted and the gauges are communicating now. V1, VA6, V5 and V4 were closed and disconnected to avoid unexpected valve switching. All went smoothly.
The new ITcc gauge is at 1e-5 Torr as CC1 This is the gauge that should be logged in slow channel.
TP2 fore line dry pump was replaced this morning after 382 day of operation.
TP3 dry pump is very noisy, but it's pressure still 47 mTorr
There is a planned power outage tomorrow, Saturday from 7am till midnight.
I vented all annulies and switched to ALL OFF configuration. The small region of the RGA is still under vacuum.
The vac-rack: gauges, c1vac1 and UPS turned off.
Vac- rack is powered back up. UPS first, than all other power switches from top to bottom of the rack, except Maglev
Manually started one by one TP2 and TP3 to accelerate to 50 KRPM
Brought up vac.control screen on lap-top at /cvs/cds/caltech/medm/c0/ve/VacControl_BAK.adl
V5 and VM3 were opened so TP3 can pump on the RGA
V4 was opened so TP2 can pump on the Maglev-TP1. The Maglev power was turned on and started acceleration.
The vac control screen positions indicators were checked for true position and annulies vent valves were opened.
RGA manual on/off switch was turned at the top of the RGA-head. Ubuntu copmuter was started at cc4 1.1e-6 Torr
The RGA communication was started with: ssh c0rga from control room
The rga-script was started ./RGAset.py This script turns on the filament, rs-232 and scan parameters etc
Vac -configuration: IFO-P1 at atm, RGA is pumped and running in background mode, all annulos at atm
There is BLANK VacControl_BAK.adl screen only.
I can move a valve by disconnecting it's solenoid power if it's position is normally open.
I will close V1 and check computer cable connections and move on with manual - hand disconnect ea valve to be moved into the right position for vent. Valve positions will be confirmed by looking manual indicators on valves.
Rana stated yesterday that there will be a vacuum control update in the close future. Witnesses : Rich, Chris and Dave
Can you give me this in writing?
Convectron gauge check:
Brand new 10 years old convectron gauge at atm was swapped into the place of existing gauges to see if they read close to 760 Torrs
They did reasonable well at the low end. I tried to imitate calibration and bring down the high end with little success.
( P1 and P2 were reading 7e-4 to ~660 Torr The correction of the upper end pushed up the lower end too. I will correct this later
P3 and P4 high ends are way off )
The point is that they work. Convectron gauges will be replaced and calibrated at the next vent.
Interlocks were not triggered during this test. I was expected to close the PSL shutter when P1 was reading 760 Torr
This hide some problem or not understanding.
It was good to see CC1 and CC4 working at the moment
Somehow I succeded opening VM1 and closed VM2 = vacuum normal
I just hope it stays open overnight to get comparable RGA scan.
In Steve's absence, I've tried to keep an eye on the health of the vacuum system. From Attachment #1, the pressure of the main volume seems stable, no red flags there. I also don't here any anomalously loud sounds near the vacuum pumps. I've changed the N2 cylinders that keep V1 open twice, on Wednesday and Sunday of last week. So in summary, the vacuum system looks fine based on all the metrics I know of.
The pumpdown had stalled because of some ancient vacuum interlock code that prevented opening the valve V1 between the turbo pump and the main volume.
This interlock  compares the channels C1:Vac-P1_pressure and C1:Vac-PTP1_pressure, neither of which is functioning at the moment. The P1 channel apparently stopped reading sometime during the vent, and contained a value of ~700 torr, while the PTP1 channel contained 0. So the interlock code saw this huge apparent pressure difference and refused to move the valve.
To bypass this check, we used caput to enter a pressure of 0 for P1.
Morning condition: vacuum rack power is still off, no MEDM screen reading.....meaning unknown vacuum pressure.We closed PSL shutter immediately.
Joe restored c1iscepis and Jenne powered up the vac-rack UPS. Now the rest of the vac-rack power were restored from starting at the top to bottom.
P1 was reading 15 mTorr. We restarted pumps and set vacuum valve positions. V1 opening required Rob's recipe of elog # 1863 to defeat interlock that
has a non communicating gauge: PTP1
CC1 pressure just reached 1e-6 Torr at VAC NORMAL configuration.
State condition: Vac Normal, CC1 ~8e-6 Torr
The IFO pressure peaked at 8.3 mTorr after 2days of not pumping.
Apparently all of the ION pump valves (VIPEE, VIPEV, VIPSV, VIPSE) opened, which vented the main volume up to 62 mTorr. All of the annulus valves (VAVSE, VAVSV, VAVBS, VAVEV, VAVEE) also appeared to be open. One of the roughing pumps was also turned on. Other stuff we didn't notice? Bad.
Several of the suspensions were kicked pretty hard (600+ mV on some sensors) as a result of this quick vent wind. All of the suspensions are damped now, so it doesn't look like we suffered any damage to suspensions.
CLOSE CALL on the vacuum system:
Jamie and I disabled V1, VM2 and VM3 gate valves by disconnecting their 120V solenoid actuator before the swap of the VME crate.
The vacuum controller unexpectedly lost control over the swap as Jamie described it. We were lucky not to do any damage! The ion pumps were cold and clean. We have not used them for years so their outgassing possibly accumulated to reach ~10-50 Torr
I disconnected_ immobilized and labelled the following 6 valves: the 4 large ion pump gate valves and VC1, VC2 of the cryo pump. Note: the valves on the cryo pump stayed closed. It is crucial that a warm cry pump is kept closed!
This will not allow the same thing to happen again and protect the IFO from warm cryo contamination.
The down side of this that the computer can not identify vacuum states any longer.
This vacuum system badly needs an upgrade. I will make a list.
We run out of N2 for the vacuum system. The pressure peaked at 1.3 mTorr with MC locked. V1 did not closed because the N2 pressure sensor failed.
We are back to vac normal. I will be here tomorrow to check on things.
ITMX damping restored.
Out gassing plus leak rate 0.15 mTorr / hour
The pressure rose to 2.5 mTorr in 17 hours
V1 was opened at 1:56pm
VM2 opened at 2:10 so the RGA region is back to 1e-5 torr
Pumpdown completed. IR shutter opened at P1 1 mTorr The block is still in the beam path.
Remember to protect MCR pd before crack up the PSL power.
The ion pump gate valves were just closed by cc1 triggered interlock
The cry pump was "regenerated" during the vent and it's outgassing rate minimized.
CC3 cold cathode gauge was replaced.
Valve configuration for week end:
1, VA6 disconnected to avoid accidental venting the IFO through the annulos
2, VC2 disconnected to insure that the cryo stays closed
3, RGA is not running, It's pressure limit 1e-5 Torr
Valve configuration: Vacuum Normal is reached in really 4 days if we do not count overnight rest of roughing.
VA6 and VC2 are reconnected. I'm turning on the RGA next
All 4 ion pumps were vented with air and pumped down to ~ 1e-4 Torr
Ion pumps gate valve control cables are connected and their pumps are disconnected.
The black relay box was tested repeatedly and it stopped misbehaiving.
We were at atmosphere for 13 days. Chamber BS, ITMX, ITMY and ETMY were opened.
Al foil "cups" were placed on the back side OSEMs of PRM.
RP1 and RP3 roughing pump manual of Leybold D30A oily rotory pump
Fore pump of TP2 & TP3 Varian SH-100 Dry Scroll
TP2 and TP3 small turbo drag pump Varian 969-9361
TP2 and TP3 turbo controller Varian 969-9505
TP1 magnetically suspended turbo pump Osaka TG390MCAB, sn360 and controller TC010M and note : this pump running on 208VAC single phaseIt is not on the UPS !
Osaka Maglev Manual and Osaka Controller Communication Wiring
VC1 cryo pump CTI-Cryogenics Cryo Torr 8 sn 8g23925 SAFETY note: compressor single phase 208VAC and the head driver 3 phase 208VAC Compressor and driver have each separate power cord!
Installed at 40m wiki also
The V1 gate valve specs installed at 40m wiki page. VAT model number 10846-UE44-0007 Our main volume pumping goes through this 8" id gate valve V1 to Maglev turbo or Cryo pump to VC1
The ion pumps have 6" id gate valves:VAT 10844-UE44-AAY1, Pneumatic actuator with position indicator and double acting solenoid valve 115V 60Hz Purchased 1999 Dec 22
UHV gate valves 2.5" id. VAT 10836-UE44 Pneumatic actuator with position indicator and double acting solenoid valve 115V 60 hz, IFO to RGA VM1 & RGA to Maglev VM2
mini UHV gate valve 1.5" id. VAT 01032-UE01 2016 cataloge page 14, manual - no position indicator, VM4 next to manual adjustable fine leak valve to RGA
UHV angle valve 1.5" id, model VAT 28432-GE41, Viton plate seal, pneumatic actuator with position indicator & solenoid valve 115V & single acting closing spring MEDM screen: VM3,VC2, V3,V4,V5,V6,VA6,V7 & annuloses Each chamber annulos has 2 valves.
UHV angle valve 1.5" id, model VAT 57132-GE05 go page 208, Metal tip seal, manual actuating only with position indicator, MEDM screen: roughing RV1 and venting VV1 hand wheel needed to close to torque spec
UHV angle valve 1.5" id. model VAT 28432-GE01 Viton plate seal, manual operation only at IT gauges Hornet & Super Bee and ion pumps roughing ports. These are not labeled.
The Cryo pump interlock wiring was added too
Note: all moving valve plate seals are single.
While I was doing the oil change of the roughing pumps I accidentally touched the 24 V adjustment knob on the power supply.
All valve closed to default condition. I realized that the current indicator was red at 0.2A and the voltage fluctuated from 3-13V
Increased current limiter to 0.4A and set voltage to 24V I think this was the reason for the caos of valve switching during the VME swap.
We did the following:
1, closed V1, VM1, annuloses: VASE, VASV, VABS, VAEV, VAEE and VA6
2, stop rotation of Maglev-TP1, waited to decellerate and turned off power to it
3, closed V4, stoped rotation of TP2, waited to decellerate and turned power off
4, opened VM3 to RGA that is still running
I will come in tomorrow 9-10am to restart pumping.
This condition is created when we run out of nitrogen that holds the valves in set positions. The Maglev and other turbos are still running in this configuration so one has to pay attention that the foreline valves are open.
Reset error message on vacuum screen and opened V4 & V5 at 3pm today.
We started a vacuum work in this morning. And still it's going on.
Although the last night the green team replaced a steering mirror by an 80% reflector on the PLS table, the beam axis to the MC looks fine.
The MC refl beam successfully goes into the MCrefl PD, and we can see the MC flashing as usual.
We started measuring the distance of the optics inside the vacuum chamber, found the distance from MC3 to MMT1(curved mirror) is ~13cm shorter than the design.
We moved the positions of the flat mirror after the Faraday and the MMT1, but could not track the beam very well because we did not completely lock the MC.
Now we are trying to get the lock of the MC by steering the MC mirrors.
Kevin suceeded in locking it !!
I just finished redoing the calc based on the measurements that happened last week. Using the average of the Vert and Horz measurements in Kevin's elog 2986, I find that we need to make the MMT telescope ~8cm longer. So, can you please place the flat mirror after the Faraday in the same place as the drawing, but move the MMT1 79mm farther away from that flat mirror? Looking at the table layouts that Koji has on the wiki, this should still (barely) fit.
d2a = 884.0mm (no change) ------ MC3 to Flat after Faraday
d2b = 1123.2mm (move MMT1 farther toward center of BS table) -------- Flat after Faraday (SM1) to MMT1
d3 = 1955.0mm (result of moving MMT1) --------- MMT1 to MMT2
d4a = 1007.9mm (no chnage) ----------- MMT2 to SM2
d4b = 495.6mm (no change) ------------ SM2 to PRM
I just got off the phone with Alberto and Kiwamu, and I'm going to try to recalculate things based on their measurements of the distances between MC3 and SM1. It sounds like the CAD drawings we have aren't totally correct. I know that when we opened doors just before Christmas we measured the distances between the BS table and the ITM tables, but I don't think we measured the distance between the IOO table and the BS table. Hopefully we can fit everything in our chambers.....
As preparation for the upcoming planned power outage we turned turbos, RGA off and closed valves.
IFO chamber is not pumped now. Small leaks and out gassing will push the pressure up slowly. At 3 mTorr of P1 the PSL output shutter
will be closed by the interlock.
It is OK to use light in the IFO up to this point.
The vacuum system is coming up to atm. The vent was started with slow N2 flow. The PSL output is blocked 2 places on the table.
Pump down reached "vacuum normal" state. IFO _P1 pressure 1e-4 torr
PSL shutter is opened.
IFO_P1 pressure 1.6e-5 torr after 6 days at atm
PS: PSL sliding door 11 was left open overnight. The PSL particle count will reach room counts in 20 seconds at low speed of HEPA
1, Fix ETMX sus "jump issue"
2, First Contact clean the arms
3, Install new spare cold cathode and convectron gauges: InstruTech-Hornet
4, Install 50 mm apeture beam baffles
5, Check and clean optical quality viewport from inside
The following bullets were executed:
Check crane functionality & cleanliness last week
We are venting the 40m IFO
Steve has ordered some teflon parts to take the place of the metal parts in his acetone-soaking jig. They should arrive tomorrow.
So, we will be begin the venting process tomorrow. Doors to come off on Tuesday.
RGA background scan
Vacuum Status: Chamber Open
All chamber annuloses are vented. Vac Monitor screen is not communicating with gauges. The valve position indicator are working.
RGA is pumped by Maglev through VM2
I just disconnected the 6th instrument grade air cylinder from the vacuum envelope at 720 Torr. Now it will reach equilibrium through a filter as it sucks in lab air.
This is the sure way not to over pressurize the chamber.
IFO is at atmosphere. The MC can be locked in air now.
The doors will be coming off tomorrow 8am sharp.
Do we want to install the ITM baffles?
What about the found OSEM filters?
I have completed the following non-Steve portions of the pre-vent checklist [wiki-40m.ligo.caltech.edu]
All shutters are closed. Ready for Steve to check nuts and begin venting!
- checked all jam nuts
- checked all viewports are covered
- turned oplev servos off
- took pictures of medm screens: sus summs, aligned oplev centering, IFO& MC alignment biases and vac configuration
- checked particle counts
- checked crane operational safety
- closed V1, VM1, annuloses
- opened VV1 and vented with Airgas brand, Industrial Grade Nitrogen [ 99.99% ] to 25 Torr
- switched over to Airgas brand compressed air, Alphagas " AI UZ300 " with Total Hydro Carbon 0.1 PPM
[Steve, EricQ, Jenne]
ITMY and BS heavy doors are off, light doors are on. Q is aligning the IFO.
ITMY in vac table needs leveling.
PSL shutter closed, manual block in place, HV turned off. P1 is at 200 Torr now. Jenne is taking over here.
Valves closed, 500 torr. Steve will finish off Monday morning, then we'll take off doors and get to work.
We are almost at atm. P1 750 Torr, We are slowly reaching equilibrium.
The access connector was removed, BS and ITMY chambers were opened yesterday.
New one piece aluminum shell access connector installed. Jenne lowered the PSL out put into MC to ~100mW
The vent has been finished.
Now the pressure inside the chamber is 760 torr, and it's getting equilibrium with the atmospheric pressure.
Therefore we are ready and can open the door of the chamber tomorrow.
The vent has been finished successfully in this morning.
The vent was finished successfully this morning.
Thanks to Kiwamu, Alberto and Koji
Koji, Jamie and I talked together and I decided to VENT TOMORROW MORNING. Main purpose of this vent is to see if PRM is flipped or not.
June 28 (Thu)
Prepare for the vent tonight
June 29 (Fri)
Start vent in the morning
Look into PRC in the evening. If PRM was flipped, we will correct them. We'll use REFL to align the PRM. If PRM was not flipped, look into PR2,PR3 and other related optics.
June 30 (Sat)
July 1 (Sun)
Thinking time. I can work if needed.
July 2 (Mon)
If we need something else to do, do it.
If not, start pumping.
July 4th is the Independence Day. So, I need IFO working before July 4th.
We will just open the BS chamber.
- PRM flipping
- PR2, PR3 flipping
- PRC suspensions
- Cipping check in PRC
We will just open the BS chamber.
- PRM flipping
- PR2, PR3 flipping
- PRC suspensions
- Cipping check in PRC
What do you mean by PR2, PR3 flipping? They are (supposed to be) flat mirrors, so obviously they should be installed correctly, but they won't change the mode matching in a huge way if they're backwards, right?
For the PRM, I recommend checking (a) the arrow inscribed on the thinner side of the optic and (b) that the arrow *actually* points to the HR side. I'm pretty sure I installed all the optics with the arrow pointing away from the OSEMs, but I never did a thorough check that the arrow always actually pointed to the HR coated side. I don't remember any optics where I said "hmmm, that's funny, the arrow is pointing backwards", but nor did I write down that I had checked.
Also, hopefully the PRM is correct. If however it's not, that means that all of the magnets are glued onto the HR side, and we'll have to redo all of the magnet gluing. The guiderods should be fine, but all 6 magnets would need redoing. If we were very, very careful and didn't break any of the magnets off of the dumbbells, it's a 24 hour turnaround due to drying time. Since inevitably we break magnets away from dumbbells, conservatively we should think about a 48 hour turnaround.
We see some ghost beam spots at POP. This may come from the back of PR2 and PR3. Also, they may change mode matching because of thermal lensing, mirror deformation, and other unexpected reasons. I thought we should check every mirrors in PRC, if PRM is not flipped.
We are going to check PRM just because we spent so much time for the PRC problem, and still don't have the solution or evidence.
PRM flipping is kind of the only idea for the root of all evil -- terrible beam shape, low PR gain, unstable PRMI lock.
So, I want to check with my eye during the stay.
I don't think we have to redo magnet gluing. It's okay to leave them on HR side.
The 40m vacuum envelope is at atm It took 4.5 hours at ~3 Torr/min speed. Atm1
What I did:
Turned PZT voltages off: 84.1V at OMC-rack location
- 99V at 1Y3 location as PZT_pitch,
56.6V KEPCO at 1Y3 location as PZT_yaw
Took screen shuts of SUS_SUMMARY, yesterday, Atm3
Turned off OPLEV _SERVOs, today , Atm2
Closed PSL_SHUTTER and green shutters at the ends
Set vacuum valves for vent position and opened VM2 for the RGA
VENT NOW and FIX ALIGNMENT!
P1 at 750 Torr, Vent valve VV1 is left open to atm. It will reach equilibrium in an hour. The PSL power is not turned down yet. The beam path is blocked at two places inside the enclosure.
The low power 20 mW input MC can be locked tonight.
Next day's - Thursday notes:
We reached 760 Torr of atm in 8 hrs at 1.7 Torr/ min with 5 cylinders of instrument grade air.
Used 500 PSI of nitrogen to get to 20 Torr at first. This flow and the first cylinder of N2 has to be slower next time.
1, oplevs were centered on locked caveties 2 days ago
2, just before vent we rechecked arms pointing, they wre still locking
3,eloged strain gages and turned HVs off at PZT-steering at 1Y3 rack and OMC-PZT ps at 1X2 aux rack. They were actually off.
4,,checked jam nuts 6 places x 3
5, turned oplev servos off
6, eloged SUS summery page
7, checked particle counts at ITMY: 1 micron 300, 0.5 micron 3K, 0.3 micron 30K particles / cft-min. All three cranes were wiped 2 days ago.
8, blocked laser into IFO: main 1064 in 1.25W were blocked by manual 2 blocks on PSL table. The PSL output shutter was left open to be triggered by interlock at P1 3 mTorr. It did!
Aux NPRO-AP and green light input at both ends were blocked.
Kiwamu and Steve have started venting the 40m vacuum envelope.
centered oplevs at resonating cavities,
ITM references were set by green pointer from the ends by Koji,
closed PSL shutter and placed manual block into beam path,
checked jamnuts in locked positions on bellows,
turned HV off at PZT-Jena "steering mirror" power supply and OMC HV ps
checked particle counts,
switched oplev servos off,
set up N2 cylinder to start vent from 1e-6 Torr to 25 Torr,
have ~ 6 cylinders of instrument grade compressed air to bring envelope from 25 Torr to 760 Torr
All three cranes were wiped off today.
Kiwamu has completed the vent.
The vent is completed. ITMX was kicked up accidentally. Valve configuration: chamber open, RGA is pumped through VM2 Maglev