40m QIL Cryo_Lab CTN SUS_Lab TCS_Lab OMC_Lab CRIME_Lab FEA ENG_Labs OptContFac Mariner WBEEShop
  40m Log, Page 328 of 344  Not logged in ELOG logo
New entries since:Wed Dec 31 16:00:00 1969
ID Date Author Type Categoryup Subject
  10533   Wed Sep 24 16:02:58 2014 JenneUpdateVACvent is completed

[Steve, EricQ, Jenne]

ITMY and BS heavy doors are off, light doors are on.  Q is aligning the IFO.

  10536   Thu Sep 25 08:21:15 2014 SteveUpdateVACvent day 1

Quote:

[Steve, EricQ, Jenne]

ITMY and BS heavy doors are off, light doors are on.  Q is aligning the IFO.

 

  10547   Mon Sep 29 09:11:25 2014 SteveUpdateVACvent day 5 RGA scan

 

 

  10548   Mon Sep 29 10:29:25 2014 SteveUpdateVACRGA is not running

 

 The RGA time stamp was correct last at 20140527

 

  Rga stopped scanning at 20140530

  10553   Tue Sep 30 16:18:57 2014 SteveUpdateVACpump down #78 has started

 

Q checked the earth quake stops of SRM and we put the ITMY & BS doors on. 

  10555   Tue Sep 30 18:02:53 2014 SteveUpdateVACpump down #78 stops at 320 Torr

 

We  stopped pumping just short of 3 hours at 320 Torr.  Pumping speed was 2.7 Torr / min with partially closed RV1 and butterfly valve/

RP1&3 roughing pump hose is disconnected. Butterfly valve removed. The vac envelope is closed. 

  10556   Wed Oct 1 10:21:19 2014 SteveUpdateVACpump down #78 stops at 3.5 Torr

Quote:

 

We  stopped pumping just short of 3 hours at 320 Torr.  Pumping speed was 2.7 Torr / min with partially closed RV1 and butterfly valve/

RP1&3 roughing pump hose is disconnected. Butterfly valve removed. The vac envelope is closed. 

 This is our second stop. I will be back this afternoon.     IFO  P1 3.5 Torr

  

  10557   Wed Oct 1 16:26:53 2014 SteveUpdateVACpump down #78 completed

 

 Pump down reached "vacuum normal" state. IFO _P1 pressure 1e-4 torr in 8 hrs actual pumping time

 PSL shutter is opened.

  10559   Thu Oct 2 09:23:23 2014 SteveUpdateVACvent 77

Quote:

 

 Pump down reached "vacuum normal" state. IFO _P1 pressure 1e-4 torr

PSL shutter is opened.

 IFO_P1 pressure 1.6e-5 torr after 6 days at atm

 

PS: PSL sliding door 11 was left open overnight. The PSL particle count will reach room counts in 20 seconds at low speed of HEPA

  10563   Fri Oct 3 10:10:37 2014 SteveUpdateVACcold cathode gauge reading switched

 

We have two cold cathode gauges at the pump spool and one  signal cable to controller. CC1  in horizontal position and CC1 in vertical position.  

CC1 h started not reading so I moved cable over to CC1 v

  10568   Mon Oct 6 10:23:43 2014 SteveUpdateVACUnexpected power shutdown

Quote:

We had an unexpected power shutdown for 5 sec at ~ 9:15 AM.

Chiara had to be powered up and am in the process of getting everything else back up again.

Steve checked the vacuum and everything looks fine with the vacuum system.

PSL Innolight laser and the 3 units of IFO air conditions turned on.

The vacuum system reaction to losing power: V1 closed and Maglev shut down. Maglev is running on 220VAC so it is not connected to VAC-UPS.  V1 interlock was triggered by Maglev "failure" message.

Maglev was reset and started. After Chiara was turned on manually I could bring up the vac control screen through Nodus and opened V1

"Vacuum Normal" valve configuration was recovered instantly.

 

Chiara needs UPS 

It is arriving Thursday

  10579   Tue Oct 7 16:55:16 2014 SteveUpdateVACUnexpected sweaty valves

 Pump  spool valves V5, V4, V3 sweating a lot. VM3 and VC2 not so much.

They are VAT valves F28-62887-03, 11, 14 and so on ~15-16 years old.

 I'm speculating that some plastic is aging-braking down at the atmospheric-pneumatic side of valves.
The vacuum side is not effected, according to vacuum pressure readings.

May be some condensation from the small turbos? No

I'm looking for an identical valve to examine, but I can not find one.

We are using industrial grade 99.96% Nitrogen to actuate these valves.

Valves are not effected are  dry: VA6, V6, V7 and all annuloses.

 

  10584   Wed Oct 8 08:46:57 2014 SteveUpdateVACVAT valves actuator lubricant

Quote:

 Pump  spool valves V5, V4, V3 sweating a lot. VM3 and VC2 not so much.

They are VAT valves F28-62887-03, 11, 14 and so on ~15-16 years old.

 I'm speculating that some plastic is aging-braking down at the atmospheric-pneumatic side of valves.
The vacuum side is not effected, according to vacuum pressure readings.

May be some condensation from the small turbos? No

I'm looking for an identical valve to examine, but I can not find one.

We are using industrial grade 99.96% Nitrogen to actuate these valves.

Valves are not effected are  dry: VA6, V6, V7 and all annuloses.

 

VAT's answer:

Yes, our engineers are aware of this issue.  They say:

The pneumatic actuator needs lubricant as the O-ring (Viton) slides in the cylinder. Without grease the O-ring would be abraded and leaking after only a relatively few cycles.  The lubricant used in our pneumatic actuators is an emulsion of oil and Teflon flakes.   Vibration, many cycles and sometimes high temperature lead to the separation of the oil and Teflon.   That is apparently the issue you are seeing.

VAT is and has been testing and qualifying new lubricants, and this is one of the factors we are always looking to improve.  The formula we used 15 years ago in these valves seems to have performed reasonable  well.  Our formula today should perform even better.

We realize this explanation does not help you with these existing valves, but 15 years of service is not too bad is it? 

Steve -NOTE:bonnet seal is metal so there is no way this oil can get into our vacuum ( only if the bellow leaks )

  10587   Thu Oct 9 11:56:35 2014 SteveUpdateVACPower outage II & recovery

Quote:

Post 30-40min unexpected power outage this morning, Steve checked the status of the vacuum and I powered up Chiara.

I brought back the FE machines and keyed all the crates to bring back the slow machines but for the vac computers.

c1vac1 is not responding as of now. All other computers have come back and are alive.

 

 IFO vacuum, air condition and PMC HV are still down. PSL out put beam is blocked on the table.

  10590   Thu Oct 9 17:33:28 2014 SteveUpdateVACPower outage II & recovery

Quote:

Quote:

Post 30-40min unexpected power outage this morning, Steve checked the status of the vacuum and I powered up Chiara.

I brought back the FE machines and keyed all the crates to bring back the slow machines but for the vac computers.

c1vac1 is not responding as of now. All other computers have come back and are alive.

 

 IFO vacuum, air condition and PMC HV are still down. PSL out put beam is blocked on the table.

 We are pumping again. This is a temporary configuration. The annuloses are at atmosphere. The reset reboot of c1Vac1 and 2 opened everything except the valves that were disconnected.

TP2 lost it's vent solenoid power supply and dry pump during the power outage.

They were replaced but the new small turbo controller is not set up as the old TP2 was so it does not allow V4 to open. 

Tomorrow I will swap back the old controller,  pump down the annuloses and close off the ion pumps.

I removed the beam block from the PSL table and opened the shutter. CC4 has the real pressure 2e-5 Torr  

CC1 is not real.

  10597   Fri Oct 10 14:41:04 2014 SteveUpdateVACPower outage II & recovery

Quote:

Quote:

Quote:

Post 30-40min unexpected power outage this morning, Steve checked the status of the vacuum and I powered up Chiara.

I brought back the FE machines and keyed all the crates to bring back the slow machines but for the vac computers.

c1vac1 is not responding as of now. All other computers have come back and are alive.

 

 IFO vacuum, air condition and PMC HV are still down. PSL out put beam is blocked on the table.

 We are pumping again. This is a temporary configuration. The annuloses are at atmosphere. The reset reboot of c1Vac1 and 2 opened everything except the valves that were disconnected.

TP2 lost it's vent solenoid power supply and dry pump during the power outage.

They were replaced but the new small turbo controller is not set up as the old TP2 was so it does not allow V4 to open. 

Tomorrow I will swap back the old controller,  pump down the annuloses and close off the ion pumps.

I removed the beam block from the PSL table and opened the shutter. CC4 has the real pressure 2e-5 Torr  

CC1 is not real.

 Tp2 is controlled by old controller. Annuloses pumped down. Valve configuration: "vacuum normal "

  Ion pumps closed at  <1e-4 mT

  10599   Mon Oct 13 14:44:52 2014 SteveUpdateVACRGA scan pd78 -day 13

 Our first RGA scan since May 27, 2014 elog10585

 The Rga is still warming up. It was turned on 3 days ago as we recovered from the second power outage.

 

  10741   Mon Dec 1 17:13:57 2014 SteveUpdateVACRGA scan pd78 -day 63

Quote:

 Our first RGA scan since May 27, 2014 elog10585

 The Rga is still warming up. It was turned on 3 days ago as we recovered from the second power outage.

 

 

  10837   Tue Dec 23 14:33:24 2014 SteveUpdateVACChiara gets UPS

Quote:

Quote:

We had an unexpected power shutdown for 5 sec at ~ 9:15 AM.

Chiara had to be powered up and am in the process of getting everything else back up again.

Steve checked the vacuum and everything looks fine with the vacuum system.

PSL Innolight laser and the 3 units of IFO air conditions turned on.

The vacuum system reaction to losing power: V1 closed and Maglev shut down. Maglev is running on 220VAC so it is not connected to VAC-UPS.  V1 interlock was triggered by Maglev "failure" message.

Maglev was reset and started. After Chiara was turned on manually I could bring up the vac control screen through Nodus and opened V1

"Vacuum Normal" valve configuration was recovered instantly.

 

Chiara needs UPS 

It is arriving Thursday

 EricQ and Steve,

Steve preset the vacuum for safe-reboot mode with C1vac1 and C1vac2 running normal: closed valves as shown, stopped Maglev & disconnected valves V1 plus valves with moving labels.

(The position indicator of the valves changes to " moving " when its cable disconnected )

Eric shut down Chiara, installed APC's UPS Pro 1000 and restarted it.

All went well. Nothing unexpected happened. So we can conclude that the vacuum system with running C1vac1 and C1vac2 is not effected by Chiara's losing AC power.

  10838   Tue Dec 23 15:37:32 2014 SteveUpdateVACTP3 drypump replaced

Quote:

Quote:

 

 TP2's fore line - dry pump replaced at performance level 600 mTorr after 10,377 hrs of continuous operation.

Where are the foreline pressure gauges? These values are not on the vac.medm screen.

The new tip seal dry pump lowered the small turbo foreline pressure 10x

TP2fl after 2 day of pumping 65mTorr

 TP2 dry pump replaced at fore pump pressure 1 Torr,  TP2 50K_rpm 0.34A

 Top seal life 6,362 hrs

 New seal performance at 1 hr  36 mTorr, 

 Maglev at 560 Hz, cc1 6e-6 Torr

 

 TP3 dry pump  replaced at 540 mT as TP3 50K_rpm 0.3A with annulos load. It's top seal life time was 11,252 hrs

 

 

  10843   Fri Dec 26 17:45:21 2014 SteveUpdateVACvac pressure rose to 1.3 mTorr

We run out of N2 for the vacuum system. The pressure peaked at 1.3 mTorr with MC locked. V1 did not closed because the N2 pressure sensor failed.

We are back to vac normal. I will be here tomorrow to check on things.

  10845   Sun Dec 28 07:29:16 2014 SteveUpdateVACvacuum is normal

Quote:

We run out of N2 for the vacuum system. The pressure peaked at 1.3 mTorr with MC locked. V1 did not closed because the N2 pressure sensor failed.

We are back to vac normal. I will be here tomorrow to check on things.

 ITMX damping restored.

  10968   Tue Feb 3 15:20:13 2015 Steve, KojiUpdateVACPneumatic pressure is being read again

[Koji, Steve]

Summary

The N2 pressure reading (C1:VAC-N2PRES) is now up-to-date after rebooting c1vac1.
The vaccum system is "Vacuum normal". We now have a space pressure transducer.

Introduction

Our vacuum valves are manipulated with 60~75 PSI of nitrogen. All the valves are configured to be closed in the case of low N2 supply pressure.
In order to avoid this safety shutdown accidentally triggered, we have two N2 cylinders to sustain the vacuum valves. When one cylinder goes to low
the mechanical valve switches over to the other cylinder.

We have the monitor channel for this (combined) cylinder pressure. One shoulbe be able to see periodical pressure variation when the auto cylinder
switch is operating. However, the nirogen pressure reading got stuck at 66 PSI on Dec.16, 2014 (See attached 60-day plot of N2 supply pressure). 

What we did

This morning we tracked down the cause of the trouble. We first closed the valves on EPICS and started to vary the N2 pressure.

Our first guess was the pressure transducer (Omega #236PC100GW) that was already 15 yrs old. We even has a sensor spare for replacement.
But it turned out that the direct voltage reading (to be 1mV/PSI) is changing correctly. The second guess was Omega Controller-Monitor
#DPiS32-C24 that is reading the voltage from the tranceducer. The display on this small black unit was changing corresponding to the
pressure change.

So our thought was
1) RS232C of the monitor unit is not working correctly
or
2) c1vac1 is not communicating with the monitor unit.

We wondered what could cause c1vac1 not communicating with the monitor unit, but we were afraid that some function got stuck
during either the nodus upgrade or chiara rebooting (or something else). So we decided to reboot c1vac1

In order to avoid any glitch in the main vacuum pressure, Steve disconnected some of the controller connectors for the closed valves.
We did this treatment before and it was successful.

Then c1vac1 was rebooted just by telnet and type reboot in the terminal.
Once the target is back in action, we noticed that the monitor value started to move.

Steve reverted the cables to the valves and operated the valves to recover "Vacuum Normal" state. Everything is now nicely settled.

  11156   Sun Mar 22 18:42:40 2015 SteveUpdateVACvac pressure rose to 1.2mTorr

 

Quote:

We run out of N2 for the vacuum system. The pressure peaked at 1.3 mTorr with MC locked. V1 did not closed because the N2 pressure sensor failed.

We are back to vac normal. I will be here tomorrow to check on things.

We run out of N2 for the vacuum system 6 hrs ago. The pressure rose to 1.2 mTorr with V1 closed. The interlock worked! See Nirogen presure reading fixed at http://nodus.ligo.caltech.edu:8080/40m/10968

 

The vacuum interlock: Nitrogen pressure transducer is reading the pneumatic pressure continously at the pump spool and c1vac1 processing it. When it drops below 60 PSI it closes V1 gate valve and V4 & V5.  Gate valve V1 needs minimum 60 PSI to close. It is critical that V1 is closed before you run out of Nitrogen so the IFO pressure is contained.

 

IFO vacuum is back to Vac Normal. The MC is locked.

cc4 = 2E-6 Torr with VM1 open.

 

Daily N2 consumption measured to be 530PSI as 3 days on 3-27-2015 but note: it does vary !

I have seen it as high as 900 psi  The long term average ~750 psi

  11159   Mon Mar 23 10:36:55 2015 ericqUpdateVACPressure watch script

Based on Jenne's chiara disk usage monitoring script, I made a script that checks the N2 pressure, which will send an email to myself, Jenne, Rana, Koji, and Steve, should the pressure fall below 60psi. I also updated the chiara disk checking script to work on the new Nodus setup. I tested the two, only emailing myself, and they appear to work as expected. 

The scripts are committed to the svn. Nodus' crontab now includes these two scripts, as well as the crontab backup script. (It occurs to me that the crontab backup script could be a little smarter, only backing it up if a change is made, but the archive is only a few MB, so it's probably not so important...)

  11178   Fri Mar 27 10:38:40 2015 steveUpdateVACvac rack UPS battery replaced

Batteries replaced after 3.5 years with Amstron AP-1250F2,  8x 12V 6Ah

Quote:

APC Smart -UPS 2200   model: SUA2200RM2U   batteries were replaced by compatible RBC43, 8x  12V5A

 

  11184   Tue Mar 31 09:03:32 2015 SteveUpdateVACRGA scan pd78 day 182

 

 

  11243   Fri Apr 24 17:30:32 2015 JenneUpdateVACPressure watch script broken
Quote:

I made a script that checks the N2 pressure, which will send an email to myself, Jenne, Rana, Koji, and Steve, should the pressure fall below 60psi.

The script checking the N2 pressure is not working.  I signed into the foteee account to look at some of the picasa photos, and there are thousands of emails (one every 10 minutes for the past month!) with error messages.  Q, can you please make it stop (having errors)?

The error looks like it's mad about a "caget" command.  I don't have time to investigate further though.

  11249   Sat Apr 25 18:50:47 2015 ericqUpdateVACPressure watch script broken

Ugh, this turns out to be because cron doesn't source the controls bashrc that defines where to find caget and all that jazz that many commands depend on. This is probably also why the AutoMX cron job isn't working either. 

Also, cron automatically emails everything from stderr to the email address that is configured for the user, which is why the n2 script blew up the foteee account and why the AutoMX script was blowing up my email yesterday. This can be avoided by doing something like this in the crontab:

0 8 * * * /bin/somecommand >> somefile.log 2>&1

(The >> part means that the standard output is appended to some log file, while the 2>&1 means send the standard error stream to the same place as stdout)

I made this change for the n2 script, so the foteee email account should be safe from this script. I haven't figured out the right way to set up cron to have all the right $PATH and other environment stuff, such as epics may need, so the script is still not working. 

  11260   Mon Apr 27 14:37:55 2015 SteveUpdateVACN2 pneumatic pressure watch

 

Quote:

Based on Jenne's chiara disk usage monitoring script, I made a script that checks the N2 pressure, which will send an email to myself, Jenne, Rana, Koji, and Steve, should the pressure fall below 60psi. I also updated the chiara disk checking script to work on the new Nodus setup. I tested the two, only emailing myself, and they appear to work as expected. 

The scripts are committed to the svn. Nodus' crontab now includes these two scripts, as well as the crontab backup script. (It occurs to me that the crontab backup script could be a little smarter, only backing it up if a change is made, but the archive is only a few MB, so it's probably not so important...)

This watch script gives little time to replace N2 cylinder. When the regulated supply drops below 60 psi the cylinder pressure is 60 psi too.

It is more of a statement that V1 is closed and act accordingly. It's only practical if you are in the lab.

Rana pointed it out correctly that we need this message 24 hrs before it happens. This requires monitoring  of total supplies , not the regulated one.

So we need pressure transducers on each nitrogen cylinder, before the regulator. The sum of the two N2 cylinder when they are full 4000 - 4500 psi

The first email should be send out at 1000 psi as sum of the two cylinders. This means that you have  1 day to replace nitrogen cylinder.

 Most of the time the daily consumption is 750 +-50 psi   

However sometimes  this variation goes up   ~750 +-150 psi

  11261   Mon Apr 27 21:42:07 2015 ranaUpdateVACSummary pages

We want to have a VAC page in the summaries, so Steve - please put a list of important channel names for the vacuum system into the elog so that we can start monitoring for trouble.

Also, anyone that has any ideas can feel free to just add a comment to the summary pages DisQus comment section with the 40m shared account or make your own account.

  11262   Tue Apr 28 09:49:26 2015 SteveUpdateVACVac Summery Channels

 

 

       Channel        

                    Function                                          Interlock action           
            

C1:Vac-P1_pressure   

 IFO vac envelope pressure           at 3 mT close V1 and PSL shutter
C1:Vac-P2_pressure  Maglev foreline pressure                      at 6 Torr close V1 
C1:Vac-P3_pressure  annuloses    
C1:Vac-CC1_pressure  IFO pressure   at 1e-5 Torr close VM1
C1:Vac-CC4_pressure  RGA  pressure    
 C1:Vac-N2pres  valve's drive  pneumatic 60-80PSI    

 

at 55 PSI close V1, at 45 PSI close all 
 It  does not exist yet 2 N2 cylinder sum pressure  

 

  11264   Thu Apr 30 16:30:25 2015 SteveUpdateVACN2 pneumatic pressure watch set up

We have 2 transduser PX303-3KG5V   http://www.omega.com/pressure/pdf/PX303.pdf They  will be installed on the out put of the N2 cylinders to read the supply pressure.

I will order one DC power supply  http://www.omega.com/pptst/PSU93_FPW15.html      PSU-93

One full cylinder pressure is ~ 2400 PSI max so two of them will give us ~9Vdc

The email reminder should be send at 1000 PSI  =  1.8 V

 

 

 

 

  11274   Tue May 5 16:02:57 2015 SteveUpdateVACVac Summery Channels with discription

As it was requested by the Bos.

It would be nice to read from the epic screen C1:Vac-state_mon.......Current State: Vacuum Normal, valve configuration

Quote:

       Channel        

                    Function                                              Description                                             Interlock         
             

C1:Vac-P1_pressure   

 Main volume of 40m interferro meter        P=Pirani gauge, Pressure range: ATM-760  to 1e-4 Torr at 3 mT close V1 and PSL shutter
C1:Vac-P2_pressure  Maglev foreline pressure

Maglev is the main pump of our vacuum system below 500 mTorr

It's long term pressure has to be <500 mTorr                  

 at 6 Torr close V1 
C1:Vac-P3_pressure  annuloses

 Each chamber has it's own annulos. These small volumes are indipendent from main volume.     Their  pressure  ranges are <5 mTorr at vac. normal valve configuration.

                                          
C1:Vac-CC1_pressure  IFO main volume

CC1=cold cathode gauge (low emmision), Pressure range: 1e-4 to 1e-10 Torr,

In vac- normal configuration CC1= 2e-6 Torr

at 1e-5 Torr close VM1
C1:Vac-CC4_pressure  RGA  pressure In vac-normal configuration CC1=CC4  
 C1:Vac-N2pres  valve's drive pneumatic    

The N2 supply is regulated to 60-80 PSI out put at the auto cylinder changer.

 at 55 PSI close V1, at 45 PSI close all 
 It  does not exist yet 2 N2 cylinder sum pressure

Each cylinder pressure will be measured before the regulator and summed for warning message to be send

at 1000 PSI

 

  11276   Fri May 8 14:30:09 2015 SteveUpdateVACCC1 cold cathode gauges are baked now

CC1s  are not reading any longer. It is an attempt to clean them over the weekend at 85C

These brand new gauges "10421002" sn 11823-vertical, sn 11837 horizontal replaced 11 years old 421 http://nsei.missouri.edu/manuals/hps-mks/421%20Cold%20Cathode%20Ionization%20Guage.pdf  on 09-06-2012 http://nodus.ligo.caltech.edu:8080/40m/7441

Quote:

 

We have two cold cathode gauges at the pump spool and one  signal cable to controller. CC1  in horizontal position and CC1 in vertical position.  

CC1 h started not reading so I moved cable over to CC1 v

 

  11287   Tue May 12 14:57:52 2015 SteveUpdateVACCC1 cold cathode gauges are baked now

Baking both CC1 at 85 C for 60 hrs did not help.

The temperature is increased to 125 C and it is being repeated.

Quote:

CC1s  are not reading any longer. It is an attempt to clean them over the weekend at 85C

These brand new gauges "10421002" sn 11823-vertical, sn 11837 horizontal replaced 11 years old 421 http://nsei.missouri.edu/manuals/hps-mks/421%20Cold%20Cathode%20Ionization%20Guage.pdf  on 09-06-2012 http://nodus.ligo.caltech.edu:8080/40m/7441

Quote:

 

We have two cold cathode gauges at the pump spool and one  signal cable to controller. CC1  in horizontal position and CC1 in vertical position.  

CC1 h started not reading so I moved cable over to CC1 v

 

 

  11292   Fri May 15 16:18:28 2015 SteveUpdateVACVac Operation Guide

Vacuum Operation Guide is up loaded into the 40m-wiki. This is an old master copy. Not exact in terms of real action, but it is still a good guide of logic.

Rana has promissed to watch the N2 supply and change cylinder when it is empty. I will be Hanford next week.

  11352   Wed Jun 10 15:54:14 2015 SteveUpdateVACVacuum comp. rebooted

Koji and Steve succeded rebooting C1vac1, C1vac2 and pressure reading is working now

More tomorrow .........

 

  11353   Thu Jun 11 19:40:59 2015 KojiUpdateVACVacuum comp. rebooted

The serial connections to the vacuum gauges were recovered by rebooting c1vac1 and c1vac2.

Steve claimed that the vacuum screen had showed "NO COMM" at the vacuum pressure values.
The epics connection to c1vac was fine. We could logged in to c1vac1 with telnet too although c1vac2 had no response.

After some inspection, we decided to reboot the slow machines. Steve manually XXXed YYY valves (to be described)
to prepare for any possible unwanted switching. Initially Koji thought only c1vac2 can be rebooted. But it was wrong.
If the reset button is pushed, all of the modules on the same crate is reset. So everything was reset. After ~3min we still
don't have the connection to c1vac1 restored. We decided to another reboot. This time I pushed c1vac1 reset button.
After waiting about two minutes, the ADCs started to show green lights and the switch box started scanning.
We recovered the telnet connection to c1vac1 and epics functions. c1vac2 is still note responding to telnet, and
the values associated with c1vac2 are still blank.

Steve restored the valves and everything was back to normal.

  11354   Fri Jun 12 08:40:17 2015 SteveUpdateVACVacuum comp. rebooted

Koji and Steve,

One computer expert and one vacuum expert required.

Quote:

The serial connections to the vacuum gauges were recovered by rebooting c1vac1 and c1vac2.

Steve claimed that the vacuum screen had showed "NO COMM" at the vacuum pressure values.
The epics connection to c1vac was fine. We could logged in to c1vac1 with telnet too although c1vac2 had no response.

After some inspection, we decided to reboot the slow machines. Steve manually XXXed YYY valves (to be described)
to prepare for any possible unwanted switching. Initially Koji thought only c1vac2 can be rebooted. But it was wrong.
If the reset button is pushed, all of the modules on the same crate is reset. So everything was reset. After ~3min we still
don't have the connection to c1vac1 restored. We decided to another reboot. This time I pushed c1vac1 reset button.
After waiting about two minutes, the ADCs started to show green lights and the switch box started scanning.
We recovered the telnet connection to c1vac1 and epics functions. c1vac2 is still note responding to telnet, and
the values associated with c1vac2 are still blank.

Steve restored the valves and everything was back to normal.

Atm 1,  problem condition: gauges are not reading for a week, error message "NO COMM" and all computer LEDs are green

Atm 2, prepare to safe reboot:

            a, close V1, disconnect it's power cable and turn off Maglev, wait till rotation stops

            b, close PSL shutter ( take adrenaline if needed )

            c, close V4, V5, VA6 valves and disconnect their cables. "Moving" error message indicating this condition.

               V1 is not showing "Moving" because its power cable disconnected only! It will show it if its position indicator cable is disconnected too. There is no need for that.

               These valves closed and disabled will not allow accidental venting of main volume.

            d, push reset, reseting c1vac2 will reset c1vac1 also, wait ~ 6 minutes

"Vacuum Normal" valve configuration was restored after succesful reboot as follows:

             a, reconnect cable and open V4 and V5 at P2 & P3 <1e-1 Torr

             b, observe that P2  < 1e-3 Torr and retsart Maglev

             c, wait till Maglev reaches full speed of 560 Hz and reconnect-open V1

             d, reconnect-open VA6 at P3 <1e-3 Torr

NOTE: VM1 valve was locked in open position and it was not responding before and after reboot

          Error message on Atm2 is indicating this locked condition: "opening VM1 will vent IFO"

          This is a fauls message. The valve is frozen in open position. We need a softwear expert help.

 

 

  11387   Wed Jul 1 10:01:25 2015 SteveUpdateVACRGA scan pd78 day 275

 

 

  11483   Thu Aug 6 09:16:57 2015 SteveUpdateVACChiara gets new Ethernet card
  1. Yesterday we did put the vacuum system into safe to reboot mode  for the ethernet card swap.
  2. Atm1, IFO pressure
  3. Atm2, as prepared valve configuration where " moving " means closed and disconnected

 

  11593   Mon Sep 14 10:41:03 2015 SteveUpdateVACTP2 dry pump replaced

TP2 dry fore pump sn PLE10082 was replaced at pressure 717 mTorr,  TP2 50K rpm 0.33A @ 112,677 hrs

Top seal life was 8,160 hrs

Model  SH110, Sn LP1007L556 was installed. It's fore line pressure after 30 minutes of running 38 mTorr, TP2 turbo at 50K rpm 0.18A

 

 

  11625   Mon Sep 21 11:12:14 2015 SteveUpdateVACcold cathode is flaky

CC4 cold cathode gauge jump triggered interlock to close VM1 valve to protect the RGA.

The IFO pressure is 1e-5 Torr

Vac normal was recovered by opening VM1

 

  11633   Tue Sep 22 08:58:38 2015 SteveUpdateVACcold cathode is flaky

The cold cathode gauge is back to normal. cc4 is the last gauge is "functioning"

MKS is not responding. The spare controller and gauges are back for repair.

 

  11644   Fri Sep 25 17:00:38 2015 SteveUpdateVACcold cathode is flaky

The IFO pressure is estimated ~1E-6 Torr with modified Vac normal valve configuration.

CC4 is in a jumping mode between 2e-5 and 1e-6 Torr

 Pressure based interlock kicks in to close VM1   at 2e-5 Torr to protect the RGA.

I did open VM1 repeatedly in the last few mornings but as cc4 jumps VM1 closes.

As VM1  closed to RGA scans are not seeing the IFO. I will look at some scans on Monday.

Mean while I opened VM2 to lower the pressure for the RGA. This change will be read by "Current Status: Undefined State"

So do not panic, the IFO pressure is normal.

I need someone's help to raise the interlock threshold to 5e-5 Torr

I'm buying a new cold cathode gauge on Monday.

 

note: cc1 is out of order!

         just read P1  the pressure is < 7e-4 Torr  This gauge is very reliable and it is at the low end of it's range.

 

  11666   Mon Oct 5 10:11:35 2015 SteveUpdateVACRGA scan pd78 day 386

RGA background scan.

The IFO is closed off from RGA with VM1

CC4 ( and CC1)  is still flaky and it's interlock closes VM1

 

  11678   Fri Oct 9 11:41:09 2015 ericqUpdateVACN2 Pressure fell

At 10:02AM, the N2 Pressure fell below 60 PSI. The watch script saw this happen, but I did not recieve the email it is supposed to send frown

C1:Vac-P1_pressure reads 7e-4, which is the same as it has for the past ~2 days, so the V1 interlock worked fine. 

I've put some fresh N2 into the system, and Bob will pop in over the weekend to check it. I'll stay on top of it until Steve gets back. 

After consulting ELOGs and the 40m wiki, I reasoned it was ok to open the V1 to reconnect the turbo pump to the main IFO volume and VM1 to reconnect the RGA, and have now done so. 

  11695   Fri Oct 16 18:27:52 2015 SteveUpdateVAC vacuum normal condition

Vacuum normal configuration with VM1-closed, VM2-open valve positions. Power load normal 24V 0.2A

Maglev rotation 560 Hz at room temp body temp.

"NO COMM" error message on medm screen. Gauge controller pressures are read able.

All vac comp LEDs are green. We have to reboot on Monday to enable communication.

 

  11697   Mon Oct 19 11:20:34 2015 gautamUpdateVACRGA scan reset

Steve pointed out that in the aftermath of the Nitrogen running out a couple of times last week, the RGA had shut itself off thinking that there was a leak and so it was not performing the scheduled scans once a day. So the data files from the scheduled scans were empty in the /opt/rtcds/caltech/c1/scripts/RGA/logs directory. The wiki page for getting it up and running again is up-to-date, but the script RGAset.py did not exist on the c0rga machine, which the RGA is communicating with via serial port. I copied over the script RGAset.py from rossa to c0rga and ran the script on that machine - but the error flags it returned were not all 0 (indicating some error according to the manual) - so I edited the script to send just the initialize command ('IN0') and commented out the other commands, after which I got error flags which were all 0. After this, I ran a manual scan using 'RGAlogger.py', and it appears that the RGA is now able to take scans again - I'm attaching a plot of the scan results. We've saved this scan as a reference to compare against after a few days. 

ELOG V3.1.3-