40m QIL Cryo_Lab CTN SUS_Lab TCS_Lab OMC_Lab CRIME_Lab FEA ENG_Labs OptContFac Mariner WBEEShop
  40m Log, Page 331 of 344  Not logged in ELOG logo
ID Date Author Type Categoryup Subject
  14088   Thu Jul 19 13:35:30 2018 SteveSummaryVACannuloses pumped

Roughing down the annuloses required closing V1 for 13 minutes

IFO is 2.2e-5 Torr

  14098   Mon Jul 23 09:58:52 2018 SteveSummaryVACRGA scan at day 6

 

 

  14137   Mon Aug 6 09:34:02 2018 SteveUpdateVACRGA scan at day 20

 

 

  14167   Thu Aug 16 07:50:28 2018 SteveUpdateVACpumpdown 81 at day 30

 

 

  14168   Thu Aug 16 14:48:14 2018 SteveUpdateVACwhy do we need a root pump?

Basic Pump Throughput Concepts

What is Pump Throughput?

The manufacturer of a vacuum pump supplies a chart for each pump showing pumping speed (volume in unit time) vs pressure. The example, for a fictitious pump, shows the pumping speed is substantially constant over a large pressure range.

By multiplying pumping speed by pressure at which that pumping speed occurs, we get a measure called pump throughput. We can tabulate those results, as shown in the table below, or plot them as a graph of pressure vs pump throughput. As is clear from the chart,  pump throughput (which might also be called mass flow) decreases proportionally with PRESSURE, at least over the pressure range where pumping speed is constant.

 

Pumping Speed Pressure Pressure x Pumping Speed
100 L/sec 10 torr 1000 torr.liter/sec
100 L/sec 1 torr 100 torr.liter/sec
100 L/sec 0.1 torr 10 torr.liter/sec
100 L/sec 0.01 torr

1 torr.liter/sec

The roughing pump speed actually will reach 0 l/s  at it's ultimate pressure performance.

Our roughing pump  pumping speed will slowly drop  as chamber pressure drops. Below 10 Torr this decrease is accelerated and bottoms out. This where the Root pump can help. See NASA evaluation of dry rough pumps...What is a root pump 

We have been operating succsessfully with a narrow margin. The danger is that the Maglev forline peaks at 4 Torr. This puts load on the small turbo TP2, TP3 &  large TP1

The temperature of these TP2 & 3  70 l/s drag turbos go up to 38 C and their  rotation speed slow to 45K rpm from 50K rpm because of the large volume 33,000 liters

Either high temp or low rotation speed of drag turbo or long time of overloading  can shut down the small turbo pumps......meaning: stop pumping, wait till they cool down

The manual gate valve installed helped to lower peak temp to 32C It just took too long.

We have been running with 2 external fans [one on TP1 & one on TP3]  for cooling and one aux drypump to help lowering the foreline pressure of TP2 & 3

The vacuum control upgrade should include adding root pump into the zero pumping speed range.

 

Atm1,   Pump speed chart:   TP1  turbo -red, root pump -blue and mechanical pump green. Note green color here representing an oily rotory pump. Our small drypumps [SH-100] typically run above 100 mTorr

                                           They are the forepump of TP2 & 3     Our pumpdown procedure: Oily Leybold rotory pumps ( with safety orifice 350 mT to atm ) rough to 500 mTorr

                                                                                                 Here we switch over to TP2 & 3 running at 50k RPM with drypumps SH-100 plus Aux Triscroll

                                                                                                 TP1- Maglev rotating full speed when V1 is opened at full volume at 500 mTorr 

                         History: the original design of the early 1990s had no dry scroll pumps. Oil free dry scrools replaced the oily forepumps of TP2 & TP3 in ~2002  at the cost of degrading the forline pressure somewhat.

                                     We had 2 temperature related Maglev failers in 2005 Aug 8 and 2006 April 5  Osaka advised us to use AUX fan to cool TP1  This helped. 

Atm2,   Wanted Root pump - Leybold EcoDry 65 plus  

Atm3,   Typical 8 hrs pumpdown from 2007 with TP2 & 3 

Atm4,   Last pumpdown zoomed in from 400 mT to 1mT with throttled gate  valve took 9 hrs  The foreline pressure of TP1 peaked at 290 mT, TP3 temperature peaked at 32C

            This technic is workable, but 9 hrs is too long.

Atm5,   The lowest pressure achived in  the 40m Vacuum Envelope 5e-7 Torr with pumps Maglev  ~300 l/s,  Cryo 1500 l/s  and 3 ion pumps of 500 l/s      [ in April 2002 at pumpdown 53 day 7 ] with annuloses at ~ 10 mTorr

Atm6,  Osaka TG390MCAB Throughput with screen ~300 L/s at 12 cfm backing pump

  14183   Fri Aug 24 10:51:23 2018 SteveUpdateVACpumpdown 81 at day 38

 

 

  14189   Wed Aug 29 09:56:00 2018 SteveUpdateVACMaglev controller needs service

TP-1 Osaka maglev controller  [  model TCO10M,  ser V3F04J07 ]  needs maintenance. Alarm led  on indicating  that we need Lv2 service.

The turbo and the controller are in good working order.

*****************************

Hi Steve,

Our maintenance level 2 service price is $...... It consists of a complete disassembly of the controller for internal cleaning of all ICB’s, replacement of all main board capacitors, replacement of all internal cooling units, ROM battery replacement, re-assembly, and mandatory final testing to make sure it meets our factory specifications. Turnaround time is approximately 3 weeks.

  RMA 5686 has been assigned to Caltech’s returning TC010M controller. Attached please find our RMA forms. Complete and return them to us via email, along with your PO, prior to shipping the cont

Best regards,

Pedro Gutierrez

Osaka Vacuum USA, Inc.

510-770-0100 x 109

*************************************************

our TP-1 TG390MCAB is 9 years old. What is the life expectancy of this turbo?

                        The Osaka maglev turbopumps are designed with a 100,000 hours(or ~ 10 operating years) life span but as you know most of our end-users are

                        running their Osaka maglev turbopumps in excess of 10+, 15+ years continuously.     The 100,000 hours design value is based upon the AL material being rotated at

                        the given speed.   But the design fudge factor have somehow elongated the practical life span.  

We should have the cost of new maglev & controller in next year budget. I  put the quote into the wiki.

 

                         

 

  14207   Fri Sep 21 16:51:43 2018 gautamUpdateVACc1vac1 is unresponsive

Steve pointed out that some of the vacuum MEDM screen fields were reporting "NO COMM". Koji confirmed that this is a c1vac1 problem, likely the same as reported here and can be fixed using the same procedure.

However, Steve is worried that the interlock won't kick in in case of a vacuum emergency, so we are leaving the PSL shutter closed over the weekend. The problem will be revisited on Monday.

  14215   Mon Sep 24 15:06:10 2018 gautamUpdateVACc1vac1 reboot + TP1 controller replacement

[steve, gautam]

Following the procedure in this elog, we effected a reset of the vacuum slow machines. Usually, I just turn the key on these crates to do a power cycle, but Steve pointed out that for the vacuum machines, we should only push the "reset" button.

While TP1 was spun down, we took the opportunity to replace the TP1 controller with a spare unit the company has sent us for use while our unit is sent to them for maintenance. The procedure was in principle simple (I only list the additional ones, for the various valve closures, see the slow machine reset procedure elog):

  • Turn power off using switch on rear.
  • Remove 4 connecting cables on the back.
  • Switch controllers.
  • Reconnect 4 cables on the back panel.
  • Turn power back on using switch on rear.

However, we were foiled by a Philips screw on the DB37 connector labelled "MAG BRG", which had all its head worn out. We had to make a cut in this screw using a saw blade, and use a "-" screwdriver to get this troublesome screw out. Steve suspects this is a metric gauge screw, and will request the company to send us a new one, we will replace it when re-installing the maintaiend controller. 

Attachments #1 and #2 show the Vacuum MEDM screen before and after the reboot respectively - evidently, the fields that were reading "NO COMM" now read numbers. Attachment #3 shows the main volume pressure during this work.

Quote:

The problem will be revisited on Monday.

  14217   Wed Sep 26 10:07:16 2018 SteveUpdateVACwhy reboot c1vac1

Precondition: c1vac1 & c1vac2 all LED warning lights green [ atm3 ], the only error message is in the gauge readings NO COMM, dataviewer will plot zero [ atm1 ], valves are operational

When our vacuum gauges read " NO COMM " than our INTERLOCKS  do  NOT communicate either.

So V1 gate valve and PSL output shutter can not be triggered to close if the the IFO pressure goes up.                        

   [ only CC1_HORNET_PRESSURE reading is working in this condition because it goes to a different compuer ] 

Quote:

[steve, gautam]

Following the procedure in this elog, we effected a reset of the vacuum slow machines. Usually, I just turn the key on these crates to do a power cycle, but Steve pointed out that for the vacuum machines, we should only push the "reset" button.

While TP1 was spun down, we took the opportunity to replace the TP1 controller with a spare unit the company has sent us for use while our unit is sent to them for maintenance. The procedure was in principle simple (I only list the additional ones, for the various valve closures, see the slow machine reset procedure elog):

  • Turn power off using switch on rear.
  • Remove 4 connecting cables on the back.
  • Switch controllers.
  • Reconnect 4 cables on the back panel.
  • Turn power back on using switch on rear.

However, we were foiled by a Philips screw on the DB37 connector labelled "MAG BRG", which had all its head worn out. We had to make a cut in this screw using a saw blade, and use a "-" screwdriver to get this troublesome screw out. Steve suspects this is a metric gauge screw, and will request the company to send us a new one, we will replace it when re-installing the maintaiend controller. 

Attachments #1 and #2 show the Vacuum MEDM screen before and after the reboot respectively - evidently, the fields that were reading "NO COMM" now read numbers. Attachment #3 shows the main volume pressure during this work.

Quote:

The problem will be revisited on Monday.

 

  14229   Thu Oct 4 08:25:50 2018 SteveUpdateVACrga scan pd81 at day 78

 

 

  14244   Fri Oct 12 08:27:05 2018 SteveUpdateVACdrypump

Gautam and Steve,

Our TP3 drypump seal is at 360 mT [0.25A load on small turbo]  after one year.  We tried to swap in old spare drypump with new tip  seal. It was blowing it's fuse, so we could not do it.

Noisy aux drypump turned on and opened to TP3 foreline [ two drypumps are in  the foreline now ]  The pressure is 48 mT and 0.17A load on small turbo.

  14247   Fri Oct 12 17:37:03 2018 SteveUpdateVACpressure gauge choices

We want to measure the pressure gradient in the 40m IFO

Our old MKS cold cathodes are out of order. The existing working gauge at the pumpspool is InstruTech CCM501

The plan is to purchase 3 new gauges for ETMY, BS and MC2 location.

Basic cold cathode     or    Bayard-Alpert Pirani

    

 

  14256   Mon Oct 15 13:59:42 2018 SteveUpdateVACdrypump replaced

Steve & Bob,

Bob removed the head cover from the housing to inspect the condition of the the tip seal. The tip seal was fine but the viton cover seal had a bad hump. This misaligned the tip seal and it did not allow it to rotate.

It was repositioned an carefully tithened. It worked. It's starting current transiant measured 28 A and operational mode 3.5 A

This load is normal with an old pump. See the brand new DIP7 drypump as spare was 25 A at start and  3.1 A in operational mode. It is amazing how much punishment a slow blow ceramic 10A  fuse can take [ 0215010.HXP ]

In the future one should measure the current pick up [ transient <100ms ] after the the seal change with Fluke 330 Series Current Clamp

 

It was swapped in and the foreline pressure dropped to 24 mTorr after 4 hours. It is very good. TP3 rotational drive current  0.15 A at 50K rpm   24C

Quote:

Gautam and Steve,

Our TP3 drypump seal is at 360 mT [0.25A load on small turbo]  after one year.  We tried to swap in old spare drypump with new tip  seal. It was blowing it's fuse, so we could not do it.

Noisy aux drypump turned on and opened to TP3 foreline [ two drypumps are in  the foreline now ]  The pressure is 48 mT and 0.17A load on small turbo.

 

  14262   Mon Oct 22 15:19:05 2018 SteveUpdateVACMaglev controller serviced

Gautam & Steve,

Our controller is back with Osaka maintenace completed. We swapped it in this morning.

Quote:

TP-1 Osaka maglev controller  [  model TCO10M,  ser V3F04J07 ]  needs maintenance. Alarm led  on indicating  that we need Lv2 service.

The turbo and the controller are in good working order.

*****************************

Hi Steve,

Our maintenance level 2 service price is $...... It consists of a complete disassembly of the controller for internal cleaning of all ICB’s, replacement of all main board capacitors, replacement of all internal cooling units, ROM battery replacement, re-assembly, and mandatory final testing to make sure it meets our factory specifications. Turnaround time is approximately 3 weeks.

  RMA 5686 has been assigned to Caltech’s returning TC010M controller. Attached please find our RMA forms. Complete and return them to us via email, along with your PO, prior to shipping the cont

Best regards,

Pedro Gutierrez

Osaka Vacuum USA, Inc.

510-770-0100 x 109

*************************************************

our TP-1 TG390MCAB is 9 years old. What is the life expectancy of this turbo?

                        The Osaka maglev turbopumps are designed with a 100,000 hours(or ~ 10 operating years) life span but as you know most of our end-users are

                        running their Osaka maglev turbopumps in excess of 10+, 15+ years continuously.     The 100,000 hours design value is based upon the AL material being rotated at

                        the given speed.   But the design fudge factor have somehow elongated the practical life span.  

We should have the cost of new maglev & controller in next year budget. I  put the quote into the wiki.

 

                         

 

 

  14264   Wed Oct 31 17:54:25 2018 gautamUpdateVACCC1 hornet power connection restored

Steve reported to me that the CC1 Hornet gauge was not reporting the IFO pressure after some cable tracing at EX. I found that the power to the unit had been accidentally disconnected. I re-connected the power and manually turned on the HV on the CC gauge (perhaps this can be automated in the new vacuum paradigm). IFO pressure of 8e-6 torr is being reported now.

  14279   Tue Nov 6 23:19:06 2018 gautamUpdateVACc1vac1 FAIL lights on (briefly)

Jon and I stuck a extender card into the eurocrate at 1X8 earlier today (~5pm PT), to see if the box was getting +24V DC from the Sorensen or not. Upon sticking the card in, the FAIL LEDs on all the VME cards came on. We immediately removed the extender card. Without any intervention from us, after ~1 minute, the FAIL LEDs went off again. Judging by the main volume pressure (Attachment #1) and the Vacuum MEDM screen (Attachment #2), this did not create any issues and the c1vac1 computer is still responsive.

But Steve can perhaps run a check in the AM to confirm that this activity didn't break anything.

Is there a reason why extender cards shouldn't be stuck into eurocrates?

  14281   Wed Nov 7 08:32:32 2018 SteveUpdateVACc1vac1 FAIL lights on (briefly)...checked

The vacuum and MC are OK

Quote:

Jon and I stuck a extender card into the eurocrate at 1X8 earlier today (~5pm PT), to see if the box was getting +24V DC from the Sorensen or not. Upon sticking the card in, the FAIL LEDs on all the VME cards came on. We immediately removed the extender card. Without any intervention from us, after ~1 minute, the FAIL LEDs went off again. Judging by the main volume pressure (Attachment #1) and the Vacuum MEDM screen (Attachment #2), this did not create any issues and the c1vac1 computer is still responsive.

But Steve can perhaps run a check in the AM to confirm that this activity didn't break anything.

Is there a reason why extender cards shouldn't be stuck into eurocrates?

 

  14291   Tue Nov 13 16:15:01 2018 SteveUpdateVACrga scan pd81 at day 119

 

 

  14299   Fri Nov 16 10:26:12 2018 SteveUpdateVAC single viton O-rings

The 40m vacuum envelope has one large single O-ring on the OOC west side. All other doors have double O-ring with annuloses.

There are 3  spacers to protect o-ring. They should not be removed!

 

The Cryo-pump static seal to VC1 also viton. All gate valves and right angle valve plates have single viton o-ring seal.

Small single viton o-rings on all optical quality viewports.

Helium will permiate through these fast. Leak checking time is limited to 5-10 minutes.

All other seals are copper gaskits. We have 2 manual right angle with METAL-dynamic seal [ VATRING ] as  VV1 & RV1

 

  14301   Fri Nov 16 15:09:31 2018 SteveConfigurationVACnot venting cryo and ion pumps

Notes on the ion pumps and cryo pump:

  • Our 4 ion pumps were closed off for a lomg time. I estmated their pressure to be around ~1 Torr. After talking with Koji we decided not to vent them.

  • It'd be still useful to wire their position sensors. But make sure we do not actuate the valves. 

  • The cryo pump was regenerated to 1e-4 Torr about 2 years ago. It's pressure can be ~ 2 Torr with charcoal powder. It is a dirty system at room temperature.

  • Do not actuate VC1 and VC2, and keep its manual valve closed.

  • IF someone feels we should vent them for some reason, let us know here in the elog before Monday morning.

 

Quote:

Wiring of the power, Ethernet, and indicator lights for the vacuum Acromag chassis is complete. Even though this crate will only use +24V DC, I wired the +/-15V connector and indicator lights as well to conform to the LIGO standard. There was no wiring diagram available, so I had to reverse-engineer the wiring from the partially complete c1susaux crate. Attached is a diagram for future use. The crate is ready to begin software developing on Monday. 

 

  14305   Mon Nov 19 14:59:48 2018 ChubUpdateVACVent 81

Vent 80 is nearly complete; the instrument is almost to atmosphere.  All four ion pump gate valves have been disconnected, though the position sensors are still connected,and all annulus valves are open.  The controllers of TP1 and TP3 have been disconnected from AC power. VC1 and VC2 have been disconnected and must remained closed. Currently, the RGA is being vented through the needle valve and the RGA had been shut off at the beginning of the vent preparations.  VM1 and VM3 could not be actuated.  The condition status is still listed as Unidentified because of the disconnected valves. 

  14306   Mon Nov 19 17:09:00 2018 SteveUpdateVACVent 81

Gautam, Aaron, Chub and Steve,

Quote:

Vent 80 is nearly complete; the instrument is almost to atmosphere.  All four ion pump gate valves have been disconnected, though the position sensors are still connected,and all annulus valves are open.  The controllers of TP1 and TP3 have been disconnected from AC power. VC1 and VC2 have been disconnected and must remained closed. Currently, the RGA is being vented through the needle valve and the RGA had been shut off at the beginning of the vent preparations.  VM1 and VM3 could not be actuated.  The condition status is still listed as Unidentified because of the disconnected valves. 

The vent 81 is completed.

4 ion pumps and cryo pump are at ~ 1-4 Torr (estimated as we have no gauges there), all other parts of the vacuum envelope are at atm. P2 & P3 gauges are out of order.

V1 and VM1 are in a locked state. We suspect this is because of some interlock logic.

TP1 and TP3 controllers are turned off.

Valve conditions as  shown: ready to be opened or closed or moved or rewired. To re-iterate: VC1, VC2, and the Ion Pump valves shouldn't be re-connected during the vac upgrade.

Thanks for all of your help.

  14307   Mon Nov 19 22:01:50 2018 gautamUpdateVACLoose nut on valve

As I was turning off the lights in the VEA, I heard a rattling sound from near the PSL enclosure. I followed it to a valve - I couldn't see a label on this valve in my brief effort to find one, but it is on the south-west corner of the IMC table, so maybe VABSSCI or VABSSCO? The power cable is somehow spliced with an attachment that looks to be bringing gas in/out of the valve (See Attachment #1), and the nut on the bottom was loose, the whole power cable + mettal attachment was responsible for the rattling. I finger-tightened the nut and the sound went away.

  14310   Tue Nov 20 13:13:01 2018 gautamUpdateVACIMC alignment is okay

I checked the IMC alignment following the vent, for which the manual beam block placed on the PSL table was removed. The alignment is okay, after minor touchup, the MC Trans was ~1200 cts which is roughly what it was pre-vent. I've closed the PSL shutter again.

  14318   Mon Nov 26 15:58:48 2018 SteveUpdateVACVent 81

Gautam, Aaron, Chub & Steve,

ETMY heavy door replaced by light one.

We did the following:  measured 950 particles/cf min of 0.5 micron at SP table, wiped crane and it's cable, wiped chamber,

                                placed heavy door on clean merostate covered stand, dry wiped o-rings and isopropanol wiped Aluminum light cover

                              

Quote:

Gautam, Aaron, Chub and Steve,

Quote:

Vent 80 is nearly complete; the instrument is almost to atmosphere.  All four ion pump gate valves have been disconnected, though the position sensors are still connected,and all annulus valves are open.  The controllers of TP1 and TP3 have been disconnected from AC power. VC1 and VC2 have been disconnected and must remained closed. Currently, the RGA is being vented through the needle valve and the RGA had been shut off at the beginning of the vent preparations.  VM1 and VM3 could not be actuated.  The condition status is still listed as Unidentified because of the disconnected valves. 

The vent 81 is completed.

4 ion pumps and cryo pump are at ~ 1-4 Torr (estimated as we have no gauges there), all other parts of the vacuum envelope are at atm. P2 & P3 gauges are out of order.

V1 and VM1 are in a locked state. We suspect this is because of some interlock logic.

TP1 and TP3 controllers are turned off.

Valve conditions as  shown: ready to be opened or closed or moved or rewired. To re-iterate: VC1, VC2, and the Ion Pump valves shouldn't be re-connected during the vac upgrade.

Thanks for all of your help.

 

  14322   Tue Nov 27 17:06:51 2018 SteveConfigurationVACAgilent 84FS turbo installed as TP2

Chub & Steve,

We swapped in our  replacement of Varian V70D "bear-can" turbo as factory clean.

The new Agilent TwisTorr 84 FS  turbo pump [ model x3502-64002,  sn IT17346059 ]  with intake screen, fan, vent valve. The controller  [ model 3508-64001, sn IT1737C383 ] and a larger drypump IDP-7,  [ model x3807-64010, sn MY17170019 ] was installed.

Next things to do:

  1. implement hardware interlock to close V4 at 80% pumping speed slowdown of "standby" rotation speed, estimated to be ~ 40,000 RPM ( when Standby 50K RPM  )
  2. set up isolation valve in the foreline of TP2, with delayed start of the IDP-7 and/or use relay to power drypump.  This turbo controller can not switch off or start of the dry pump. [ Agilent isolation valve #X3202-60055, with position indicator, pneumatic actuation, 115V solenoid ]..........as a second thought, we do not need isolation valve if we go with the relay option. The IDP-7 has built in delay of 10-15 sec
  3. test performance of new turbo
  14367   Wed Dec 19 14:19:15 2018 KojiSummaryVACPlan for pumpoing down test

We still need elaborated test procedure posted

12/29 Wed

  • Jon continues to work on valve actuator tests.
  • Chub continues to work on wiring / fixing wiring.
  • At the end of the day Jon is going to send out a notification email of "GO"/"NO GO" for pumping.

 

12/30 Thu

  • 9AM: Start closing two doors unless Jon gives us NO GO sign.
  • 10AM: Start pumping down
    • Test roughing pump capability via new control system
    • (Independently) Test turbo rotating procedure. This time we will not open the gate valve between the TP1 and the main volume. This is because we want to take care of the backing turbo loads while we gradually open the gate valve. This will take more hours to be done and we will not be able to finish this test by the end of Thu.
    • At the end of the procedure, we isolate the main volume, stop all the pumps, and vent the roghing pumps to save them from the oil backstream.

gautam: Koji and I were just staring at the vacuum screen, and realized that the drypumps, which are the backing pumps for TP2 and TP3, are not reflected on the MEDM screen. This should be rectified.

Steve also mentioned that the new small turbo controller does not directly interface with the drypump. So we need some system to delay the starting of the turbo itself, once the drypump has been engaged. Does this system exist?

  14370   Wed Dec 19 21:14:50 2018 gautamUpdateVACPumpdown tomorrow

I just spoke to Jon who asked me to make this elog - we will be ready to test one or more parts of the pumpdown procedure tomorrow (12/20), so we should proceed as planned to put the heavy doors back on EY and OMC chambers at 9am tomorrow morning. Jon will circulate a more detailed procedure about the pumpdown steps later today evening.

  14372   Thu Dec 20 08:38:27 2018 JonUpdateVACPumpdown tomorrow

Linked is the pumpdown procedure, contained in the old 40m documentation. The relevant procedure is "All Off --> Vacuum Normal" on page 11.

Quote:

I just spoke to Jon who asked me to make this elog - we will be ready to test one or more parts of the pumpdown procedure tomorrow (12/20), so we should proceed as planned to put the heavy doors back on EY and OMC chambers at 9am tomorrow morning. Jon will circulate a more detailed procedure about the pumpdown steps later today evening.

 

  14373   Thu Dec 20 10:28:43 2018 gautamUpdateVACHeavy doors back on for pumpdown 82

[Chub, Koji, Gautam]

We replaced the EY and IOO chamber heavy doors by 10:10 am PST. Torquing was done first oen round at 25 ft-lb, next at 45 ft-lb (we trust the calibration on the torque wrench, but how reliable is this? And how important are these numbers in ensuring a smooth pumpdown?). All went smooth. The interior of the IOO chamber was found to be dirty when Koji ran a wipe along some surfaces.

For this pumpdown, we aren't so concerned with having the IFO in an operating state as we will certainly vent it again early next year. So we didn't follow the full close-up checklist.

Jon and Chub and Koji are working on starting the pumpdown now... In order to not have to wear laser safety goggles while we closed doors and pumped down, I turned off all the 1064nm lasers in the lab.

  14377   Fri Dec 21 11:13:13 2018 gautamOmnistructureVACN2 line valved off

Per the discussion yesterday, I valved off the N2 line in the drill press room at 11 am PST today morning so as to avoid any accidental software induced gate-valve actuation during the holidays. The line pressure is steadily dropping...

Attachment #1 shows that while the main volume pressure was stable overnight, the the pumpspool pressure has been steadily rising. I think this is to be expected as the turbo pumps aren't running and the valves can't preserve the <1mtorr pressure over long timescales?

Attachment #2 shows the current VacOverview MEDM screen status.

  14379   Fri Dec 21 12:57:10 2018 KojiOmnistructureVACN2 line valved off

Independent question: Are all the turbo forelines vented automatically? We manually did it for the main roughing line.

 

  14380   Thu Jan 3 15:08:37 2019 gautamOmnistructureVACVac status unknown

Larry W came by the 40m, and reported that there was a campus-wide power glitch (he was here to check if our networking infrastructure was affected). I thought I'd check the status of the vacuum.

  • Attachment #1 is a screenshot of the Vac overview MEDM screen. Clearly something has gone wrong with the modbus process(es). Only the PTP2 and PTP3 gauges seem to be communicative.
  • Attachment #2 shows the minute trend of the pressure gauges for a 12 day period - it looks like there is some issue with the frame builder clock, perhaps this issue resurfaced? But checking the system time on FB doesn't suggest anything is wrong.. I double checked with dataviewer as well that the trends don't exist... But checking the status of the individual daqd processes indeed showed that the dates were off by 1 year, so I just restarted all of them and now the time seems correct. How can we fix this problem more permanently? Also, the P1b readout looks suspicious - why are there periods where it seems like we are reading values better than the LSB of the device?

I decided to check the systemctl process status on c1vac:

controls@c1vac:~$ sudo systemctl status modbusIOC.service
● modbusIOC.service - ModbusIOC Service via procServ
   Loaded: loaded (/etc/systemd/system/modbusIOC.service; enabled)
   Active: active (running) since Thu 2019-01-03 14:53:49 PST; 11min ago
 Main PID: 16533 (procServ)
   CGroup: /system.slice/modbusIOC.service
           ├─16533 /usr/bin/procServ -f -L /opt/target/modbusIOC.log -p /run/...
           ├─16534 /opt/epics/modules/modbus/bin/linux-x86_64/modbusApp /opt/...
           └─16582 caRepeater

Jan 03 14:53:49 c1vac systemd[1]: Started ModbusIOC Service via procServ.

Warning: Unit file changed on disk, 'systemctl daemon-reload' recommended.

So something did happen today that required restart of the modbus processes. But clearly not everything has come back up gracefully. A few lines of dmesg (there are many more segfaults):

[1706033.718061] python[23971]: segfault at 8 ip 000000000049b37d sp 00007fbae2b5fa10 error 4 in python2.7[400000+31d000]
[1706252.225984] python[24183]: segfault at 8 ip 000000000049b37d sp 00007fd3fa365a10 error 4 in python2.7[400000+31d000]
[1720961.451787] systemd-udevd[4076]: starting version 215
[1782064.269844] audit: type=1702 audit(1546540443.159:38): op=linkat ppid=21820 pid=22823 auid=4294967295 uid=1000 gid=1000 euid=1000 suid=1000 fsuid=1000 egid=1000 sgid=1000 fsgid=1000 tty=pts0 ses=4294967295 comm="git" exe="/usr/bin/git" res=0
[1782064.269866] audit: type=1302 audit(1546540443.159:39): item=0 name="/cvs/cds/caltech/target/c1vac/.git/objects/85/tmp_obj_uAXhPg" inode=173019272 dev=00:21 mode=0100444 ouid=1001 ogid=1001 rdev=00:00 nametype=NORMAL
[1782064.365240] audit: type=1702 audit(1546540443.255:40): op=linkat ppid=21820 pid=22823 auid=4294967295 uid=1000 gid=1000 euid=1000 suid=1000 fsuid=1000 egid=1000 sgid=1000 fsgid=1000 tty=pts0 ses=4294967295 comm="git" exe="/usr/bin/git" res=0
[1782064.365271] audit: type=1302 audit(1546540443.255:41): item=0 name="/cvs/cds/caltech/target/c1vac/.git/objects/58/tmp_obj_KekHsn" inode=173019274 dev=00:21 mode=0100444 ouid=1001 ogid=1001 rdev=00:00 nametype=NORMAL
[1782064.460620] audit: type=1702 audit(1546540443.347:42): op=linkat ppid=21820 pid=22823 auid=4294967295 uid=1000 gid=1000 euid=1000 suid=1000 fsuid=1000 egid=1000 sgid=1000 fsgid=1000 tty=pts0 ses=4294967295 comm="git" exe="/usr/bin/git" res=0
[1782064.460652] audit: type=1302 audit(1546540443.347:43): item=0 name="/cvs/cds/caltech/target/c1vac/.git/objects/cb/tmp_obj_q62Pdr" inode=173019276 dev=00:21 mode=0100444 ouid=1001 ogid=1001 rdev=00:00 nametype=NORMAL
[1782064.545449] audit: type=1702 audit(1546540443.435:44): op=linkat ppid=21820 pid=22823 auid=4294967295 uid=1000 gid=1000 euid=1000 suid=1000 fsuid=1000 egid=1000 sgid=1000 fsgid=1000 tty=pts0 ses=4294967295 comm="git" exe="/usr/bin/git" res=0
[1782064.545480] audit: type=1302 audit(1546540443.435:45): item=0 name="/cvs/cds/caltech/target/c1vac/.git/objects/e3/tmp_obj_gPI4qy" inode=173019277 dev=00:21 mode=0100444 ouid=1001 ogid=1001 rdev=00:00 nametype=NORMAL
[1782064.640756] audit: type=1702 audit(1546540443.527:46): op=linkat ppid=21820 pid=22823 auid=4294967295 uid=1000 gid=1000 euid=1000 suid=1000 fsuid=1000 egid=1000 sgid=1000 fsgid=1000 tty=pts0 ses=4294967295 comm="git" exe="/usr/bin/git" res=0
[1783440.878997] systemd[1]: Unit serial_TP3.service entered failed state.
[1784682.147280] systemd[1]: Unit serial_TP2.service entered failed state.
[1786407.752386] systemd[1]: Unit serial_MKS937b.service entered failed state.
[1792371.508317] systemd[1]: serial_GP316a.service failed to run 'start' task: No such file or directory
[1795550.281623] systemd[1]: Unit serial_GP316b.service entered failed state.
[1796216.213269] systemd[1]: Unit serial_TP3.service entered failed state.
[1796518.976841] systemd[1]: Unit serial_GP307.service entered failed state.
[1796670.328649] systemd[1]: serial_Hornet.service failed to run 'start' task: No such file or directory
[1797723.446084] systemd[1]: Unit serial_MKS937b.service entered failed state.

 

I don't know enough about the new system so I'm leaving this for Jon to debug. Attachment #3 shows that the analog readout of the P1 pressure gauge suggests that the IFO is still under vacuum, so no random valve openings were effected (as expected, since we valved off the N2 line for this very purpose).

  14383   Fri Jan 4 10:25:19 2019 JonOmnistructureVACN2 line valved off

Yes, for TP2 and TP3. They both have a small vent valve that opens automatically on shutdown.

Quote:

Independent question: Are all the turbo forelines vented automatically? We manually did it for the main roughing line.

 

 

  14391   Wed Jan 9 11:07:09 2019 gautamUpdateVACNew Vac channel logging

Looks like I didn't restart all the daqd processes last night, so the data was not in fact being recorded to frames. I just restarted everything, and looks like the data for the last 3 minutes are being recorded yes. Is it reasonable that the TP1 current channel is reporting 0.75A of current draw now, when the pump is off? Also the temperature readback of TP3 seems a lot jumpier than that of TP2, probably has to do with the old controller having fewer ADC bits or something, but perhaps the SMOO needs to be adjusted.

Quote:
 

Gautam and I updated the framebuilder config file, adding the newly-added channels to the list of those to be logged.

  14393   Wed Jan 9 20:01:25 2019 JonUpdateVACSecond pumpdown completed

[Jon, Koji, Chub, Gautam]

Summary

The second pumpdown with the new vacuum system was completed successfully today. A time history is attached below.

We started with the main volume still at 12 torr from the Dec. pumpdown. Roughing from 12 to 0.5 torr took approximately two hours, at which point we valved out RP1 and RP3 and valved in TP1 backed by TP2 and TP3. We additionally used the AUX dry pump connected to the backing lines of TP2 and TP3, which we found to boost the overall pump rate by a factor of ~3. The manual hand-crank valve directly in front of TP1 was used to throttle the pump rate, to avoid tripping software interlocks. If the crank valve is opened too quickly, the pressure differential between the main volume (TP1 intake) and TP1 exhaust becomes >1 torr, tripping the V1 valve-close interlock. Once the main volume pressure reached 1e-2 torr, the crank valve could be opened fully.

We allowed the pumpdown to continue until reaching 9e-4 torr in the main volume. At this point we valved off the main volume, valved off TP2 and TP3, and then shut down all turbo pumps/dry pumps. We will continue pumping tomorrow under the supervision of an operator. If the system continues to perform problem-free, we will likely leave the turbos pumping on the main volume and annuli after tomorrow.

New Vac Control Station

We installed a local controls terminal for the vacuum system on the desk in front of the vacuum rack (pictured below). This console is connected directly to c1vac and can be used to monitor/control the system even during a network outage or power failure. The entire pumpdown was run from this station today.

To open a controls MEDM screen, open a terminal and execute the alias

$control

Similarly, to open a monitor-only MEDM screen, execute the alias

$monitor
  14394   Thu Jan 10 10:23:46 2019 gautamUpdateVACovernight leak rate

Overnight, the pressure increased from 247 uTorr to 264 uTorr over a period of 30000 seconds. Assuming an IFO volume of 33,000 liters, this corresponds to an average leak rate of ~20 uTorr L / s. It'd be interesting to see how this compares with the spec'd leak rates of the Viton O-ring seals and valves/ outgassing rates. The two channels in the screenshot are monitoring the same pressure from the same sensor, top pane is a digital readout while the bottom is a calibrated analog readout that is subsequently digitized into the CDS system.

Quote:
 

We allowed the pumpdown to continue until reaching 9e-4 torr in the main volume. At this point we valved off the main volume, valved off TP2 and TP3, and then shut down all turbo pumps/dry pumps. We will continue pumping tomorrow under the supervision of an operator. If the system continues to perform problem-free, we will likely leave the turbos pumping on the main volume and annuli after tomorrow.

  14395   Thu Jan 10 11:32:40 2019 ChubUpdateVACManual valve interfaced with CDS

Connected the manual gate valve status indicator to the Acromag box this morning.  Labeled the temporary cable (a 50' 9p DSUB, will order a proper sized cable shortly) and the panel RV2.  

  14396   Thu Jan 10 19:59:08 2019 JonUpdateVACVac System Running Normally on Turbo Pumps

[Jon, Gautam, Chub]

Summary

We continued the pumpdown of the IFO today. The main volume pressure has reached 1.9e-5 torr and is continuing to fall. The system has performed without issue all day, so we'll leave the turbos continuously running from here on in the normal pumping configuration. Both TP2 and TP3 are currently backing for TP1. Once the main volume reaches operating pressure, we can transition TP3 to pump the annuli. They have already been roughed to ~0.1 torr. At that point the speed of all three turbo pumps can also be reduced. I've finished final edits/cleanup of the interlock code and MEDM screens.

Python Code

All the python code running on c1vac is archived to the git repo: 

https://git.ligo.org/40m/vacpython

This includes both the interlock code and the serial device clients for interfacing with gauges and pumps.

MEDM Monitor/Control

We're still using the same base MEDM monitor/control screens, but they have been much improved. Improvements:

  • Valves now light up in red when they are open. This makes it much easier to see at a glance what is valved in/out.
  • Every pump in the system (except CP1) is now digitally controlled from the MEDM control screen. No more need to physically push any buttons in the vaccum rack. 👍
  • The turbo pumps now show additional diagnostic readouts: speed (TP1/2/3), temperature (TP2/3), current draw (TP1/2/3), and voltage (TP2/3).
  • The foreline pressure gauge readouts for TP2/3 have been added to the digital system.
  • The two new main volume gauges, Hornet and SuperBee, have been added to the digital system as well.
  • New transducers have been added to read back the two N2 tank pressures.
  • The interlock code generates a log file of all its actions. A field in the MEDM screens specifies the location of the log file.
  • A tripped interlock (appearing as a message in the "Error message" field) must be manually cleared via the "Clear error message" button on the control screen before the system will accept any more manual valve input.

Note: The apparent glitches in the pressure and TP diagnostic channels are due to the interlock system being taken down to implement some of these changes.

  14398   Mon Jan 14 10:06:53 2019 gautamUpdateVACVent 82 complete

[chub, gautam]

  • IFO pressure was ~2e-4 torr when we started, on account of the interlock code closing all valves because the N2 line pressure dropped below threshold (<65 psi)
  • Chub fixed the problem on the regulator in the drill-press area where the N2 tanks are, the N2 line is now at ~75 psi so that we have the ability to actuate valves if we so desire
  • We decided that there is no need to vent the pumpspool this time - avoiding an unnecessary turbo landing, so the pumpspool is completely valved off from the main volume and TPs 1-3 are left running
  • Went through the pre-vent checklist:
    • Chub measured particle count, deemed it to be okay (I think we should re-locate the particle counter to near 1X8 because that is where the air enters the IFO anyways, and that way, we can hook it up to the serial device server and have a computerized record of this number as we had in the past, instead of writing it down in a notebook)
    • Checked that the PSL was manually blocked from entering the IFO
    • Walked through the lab, visually inspected Jam Nuts and window covers, all was deemed okay
  • Moved 2 tanks of N2 into the lab on account of the rain
  • Started the vent at ~930am PST
    • There were a couple of short bursty increases in the pressure as we figured out the right valve settings but on average, things are rising at approx the same rate as we had in vent 81...
    • There was a rattling noise coming from the drypump that is the forepump for TP2 (Agilent) - turned out to be the plastic shell/casing on the drypump, moreover, the TP2 diagnostics (temperature, current etc) are all normal.
    • The CC1 gauge (Hornet) is supposed to have an auto-shutoff of its High Voltage when the pressure exceeds 10 mTorr, but it was reporting pressures in the 1 mTorr range even when the adjacent Pirani was at 25 torr. To avoid risk of damage, we manually turned the HV off. There needs to be a python script that can be executed to transition control between the remote and local control modes for the hornet, we had to Power Cycle the gauge because it wouldn't give us local control over the HV.
    • Transitioned from N2 to dry air at P1a=25 torr. We had some trouble finding the correct regulator (left-handed thread) for the dry air cylinders, it was stored in a cabinet labelled green optics no
    • Disconnected dry air from VV1 intake once P1b reached 700 torr, to let lab air flow into the IFO and avoid overpressuring.
    • VA* and VAV* valves were opened so as to vent the annuli as we anticipate multiple chamber openings for this vent.

As of 8pm local time, the IFO seems to have equilibriated to atmospheric pressure (I don't hear the hiss of in-rushing air near 1X8 and P1a reports 760 torr). The pumpspool looks healthy and there are no signs in the TP diagnostics channels that anything bad happened to the pumps. Chub is working on getting the N2 setup more robust, we plan to take the EY door off at 9am tomorrow morning with Bob's help.

* I took this opportunity to follow instructions on pg 29 of the manual and set the calibration for the SuperBee pirani gauge to 760 torr so that it is in better agreement with our existing P1a Pirani gauge. The correction was ~8% (820-->760).

  14406   Fri Jan 18 17:44:14 2019 gautamUpdateVACPumping on RGA volume

Steve came by the lab today, and looked at the status of the upgraded vacuum system. He recommended pumping on the RGA volume, since it has not been pumped on for ~3 months on account of the vacuum upgrade. The procedure (so we may script this operation in the future) was:

  1. Start with the pumpspool completely isolated from the main IFO volume.
  2. Open V5, pump down the section between V5 and VM3. Keep an eye on PTP3.
  3. Open VM3, keep an eye on P4. It was reporting ~10 mtorr, went to "LO".
  4. Close VM3 and V5, transition pumping of the RGA volume to TP1 which is backed by TP2 (we had to open V4 as all valves were closed due to an N2 pressure drop event).
  5. Open VM2.
  6. Watch CC4.

CC4 pressure has been steadily falling. Steve recommends leaving things in this state over the weekend. He recommends also turning the RGA unit on so that the temperature rises and there is a bakeout of the RGA. The temperature may be read off manually using a probe attached to it.

  14410   Sun Jan 20 23:41:00 2019 JonOmnistructureVACNotes on vac serial comm, adapter wiring

I've attached my handwritten notes covering all the serial communications in the vac system, and the relevant wiring for all the adapters, etc. I'll work with Chub to produce a final documentation, but in the meantime this may be a useful reference.

  14412   Tue Jan 22 20:45:21 2019 gautamUpdateVACNew N2 setup

The N2 ran out this weekend (again no reminder email, but I haven't found the time to setup the Python mailer yet). So all the valves Steve and I had opened, closed (rightly so, that's what the interlocks are supposed to do). Chub will post an elog about the new N2 valve setup in the Drill-press room, but we now have sufficient line pressure in the N2 line again. So Chub and I re-opened the valves to keep pumping on the RGA.

  14419   Fri Jan 25 16:14:51 2019 gautamUpdateVACVacuum interlock code, N2 warning

I reset the remote of this git repo to the 40m version instead of Jon's personal one, to ensure consistency between what's on the vacuum machine and in the git repo. There is now a N2 checker python mailer that will email the 40m list if all the tank pressures are below 600 PSI (>12 hours left for someone to react before the main N2 line pressure drops and the interlocks kick in). For now, the script just runs as a cron job every 3 hours, but perhaps we should integrate it with the interlock process?

Quote:

All the python code running on c1vac is archived to the git repo: 

https://git.ligo.org/40m/vacpython

  14429   Sat Feb 2 21:53:24 2019 KojiUpdateVACovernight leak rate

The pressure of the main volume increased from ~1mtorr to 50mtorr for the past 24 hours (86ksec). This rate is about x1000 of the reported number on Jan 10. Do we suspect vacuum leak?

Quote:

Overnight, the pressure increased from 247 uTorr to 264 uTorr over a period of 30000 seconds. Assuming an IFO volume of 33,000 liters, this corresponds to an average leak rate of ~20 uTorr L / s.

 

  14430   Sun Feb 3 15:15:21 2019 gautamUpdateVACovernight leak rate

I looked into this a bit today. Did a walkthrough of the lab, didn't hear any obvious hissing (makes sense, that presumably would signal a much larger leak rate).

Attachment #1: Data from the 30 ksec we had the main vol valved off on Jan 10, but from the gauges we have running right now (the CC gauges have not had their HV enabled yet so we don't have that readback).

Attachment #2: Data from ~150 ksec from Friday night till now.

Interpretation: The number quoted from Jan 10 is from the cold-cathode gauge (~20 utorr increase). In the same period, the Pirani gauge reports a increase of ~5 mtorr (=250x the number reported by the cold-cathode gauge). So which gauge do we trust in this regime more? Additionally, the rate at which the annuli pressures are increasing seem consistent between Jan 10 and now, at ~100 mtorr every 30 ksec.

I don't think this is conclusive, but at least the leak rates between Jan 10 and now don't seem that different for the annuli pressures. Moreover, for the Jan 10 pumpdown, we had the IFO at low pressure for several days over the chirstmas break, which presumably gave time for some outgassing which was cleaned up by the TPs on Jan 10, whereas for this current pumpdown, we don't have that luxury.

Do we want to do a systematic leak check before resuming the pumpdown on Monday? The main differences in vacuum I can think of are

  1. Two pieces of Kapton tape are now in the EY chamber.
  2. Possible resiudue from cleaning solvents in IY and EY chambers are still outgassing.

This entry by Steve says that the "expected" outgassing rate is 3-5 mtorr per day, which doesn't match either the current observation or that from Jan 10.

  14431   Sun Feb 3 20:52:34 2019 KojiUpdateVACovernight leak rate

We can pump down (or vent) annuli. If this is the leak between the main volume and the annuli, we will be able to see the effect on the leak rate. If this is the leak of an  outer o-ring, again pumping down (or venting) of the annuli should temporarily decrease (or increase) the leak rate..., I guess. If the leak rate is not dependent on the pressure of the annuli, we can conclude that it is internal outgassing.

  14432   Mon Feb 4 12:23:24 2019 gautamUpdateVACpumpdown 83 - leak tests

[koji, gautam]

As planned, we valved off the main volume and the annuli from the turbo-pumps at ~730 PM PST. At this time, the main volume pressure was 30 uTorr. It started rising at a rate of ~200 uTorr/hr, which translates to ~5 mtorr/day, which is in the ballpark of what Steve said is "normal". However, the calibration of the Hornet gauge seems to be piecewise-linear (see Attachment #1), so we will have to observe overnight to get a better handle on this number.

We decided to vent the IY and EY chamber annular volumes, and check if this made anu dramatic changes in the main volume pressure increase rate, presumably signalling a leak from the outside. However, we saw no such increase - so right now, the working hypothesis is still that the main volume pressure increase is being driven by outgassing of something from the vacuum.

Let's leave things in this state overnight - V1 and V5 closed so that neither the main volume nor the annuli are being pumped, and get some baseline numbers for what the outgassing rate is.

  14434   Tue Feb 5 10:11:30 2019 gautamUpdateVACleak tests complete, pumpdown 83 resumed

I guess we forgot to close V5, so we were indeed pumping on the ITMY and ETMY annuli, but the other three were isolated suggest a leak rate of ~200-300 mtorr/day, see Attachment #1 (consistent with my earlier post).

As for the main volume - according to CC1, the pressure saturates at ~250 uTorr and is stable, while the Pirani P1a reports ~100x that pressure. I guess the cold-cathode gauge is supposed to be more accurate at low pressures, but how well do we believe the calibration on either gauge? Either ways, based on last night's test (see Attachment #2), we can set an upper limit of 12 mtorr/day. This is 2-3x the number Steve said is normal, but perhaps this is down to the fact that the outgassing from the main volume is higher immediately after a vent and in-chamber work. It is also 5x lower rate of pressure increase than what was observed on Feb 2.

I am resuming the pumping down with the turbo-pumps, let's see how long we take to get down to the nominal operating pressure of 8e-6 torr, it ususally takes ~ 1 week. V1, VASV, VASE and VABS were opened at 1030am PST. Per Chub's request (see #14435), I ran RP1 and RP3 for ~30 seconds, he will check if the oil level has changed.

Quote:
 

Let's leave things in this state overnight - V1 and V5 closed so that neither the main volume nor the annuli are being pumped, and get some baseline numbers for what the outgassing rate is.

ELOG V3.1.3-