40m QIL Cryo_Lab CTN SUS_Lab TCS_Lab OMC_Lab CRIME_Lab FEA ENG_Labs OptContFac Mariner WBEEShop
  40m Log, Page 329 of 339  Not logged in ELOG logo
ID Date Author Type Categoryup Subject
  16316   Wed Sep 8 18:00:01 2021 KojiUpdateVACcronjobs & N2 pressure alert

In the weekly meeting, Jordan pointed out that we didn't receive the alert for the low N2 pressure.

To check the situation, I went around the machines and summarized the cronjob situation.
[40m wiki: cronjob summary]
Note that this list does not include the vacuum watchdog and mailer as it is not on cronjob.

Now, I found that there are two N2 scripts running:

1. /opt/rtcds/caltech/c1/scripts/Admin/n2Check.sh on megatron and is running every minute (!)
2. /opt/rtcds/caltech/c1/scripts/Admin/N2check/pyN2check.sh on c1vac and is running every 3 hours.

Then, the N2 log file was checked: /opt/rtcds/caltech/c1/scripts/Admin/n2Check.log

Wed Sep 1 12:38:01 PDT 2021 : N2 Pressure: 76.3621
Wed Sep 1 12:38:01 PDT 2021 : T1 Pressure: 112.4
Wed Sep 1 12:38:01 PDT 2021 : T2 Pressure: 349.2
Wed Sep 1 12:39:02 PDT 2021 : N2 Pressure: 76.0241
Wed Sep 1 12:39:02 PDT 2021 : N2 pressure has fallen to 76.0241 PSI !

Tank pressures are 94.6 and 98.6 PSI!

This email was sent from Nodus.  The script is at /opt/rtcds/caltech/c1/scripts/Admin/n2Check.sh

Wed Sep 1 12:40:02 PDT 2021 : N2 Pressure: 75.5322
Wed Sep 1 12:40:02 PDT 2021 : N2 pressure has fallen to 75.5322 PSI !

Tank pressures are 93.6 and 97.6 PSI!

This email was sent from Nodus.  The script is at /opt/rtcds/caltech/c1/scripts/Admin/n2Check.sh

...

The error started at 11:39 and lasted until 13:01 every minute. So this was coming from the script on megatron. We were supposed to have ~20 alerting emails (but did none).
So what's happened to the mails? I tested the script with my mail address and the test mail came to me. Then I sent the test mail to 40m mailing list. It did not reach.
-> Decided to put the mail address (specified in /etc/mailname , I believe) to the whitelist so that the mailing list can accept it.
I did run the test again and it was successful. So I suppose the system can now send us the alert again.
And alerting every minute is excessive. I changed the check frequency to every ten minutes.

What's happened to the python version running on c1vac?
1) The script is running, spitting out some error in the cron report (email on c1vac). But it seems working.
2) This script checks the pressures of the bottles rather than the N2 pressure downstream. So it's complementary.
3) During the incident on Sept 1, the checker did not trip as the pressure drop happened between the cronjob runs and the script didn't notice it.
4) On top of them, the alert was set to send the mails only to an our former grad student. I changed it to deliver to the 40m mailing list. As the "From" address is set to be some ligox...@gmail.com, which is a member of the mailing list (why?), we are supposed to receive the alert. (And we do for other vacuum alert from this address).

 

 

 

 

  16404   Thu Oct 14 18:30:23 2021 KojiSummaryVACFlange/Cable Stand Configuration

Flange Configuration for BHD

We will need total 5 new cable stands. So Qty.6 is the number to be ordered.


Looking at the accuglass drawing, the in-vaccum cables are standard dsub 25pin cables only with two standard fixing threads.

https://www.accuglassproducts.com/sites/default/files/PDF/Partpdf/110070_3.pdf

For SOSs, the standard 40m style cable bracket works fine. https://dcc.ligo.org/D010194-x0

However, for the OMCs, we need to make the thread holes available so that we can mate DB25 male cables to these cables.
One possibility is to improvise this cable bracket to suspend the cables using clean Cu wires or something. I think we can deal with this issue in situ.


Ha! The male side has the 4-40 standoff (jack) screws. So we can hold the male side on the bracket using the standoff (jack) screws and plug the female cables. OK! The issue solved!

https://www.accuglassproducts.com/sites/default/files/PDF/Partpdf/110029_3.pdf

Attachment 1: 40m_flange_layout_20211014.pdf
40m_flange_layout_20211014.pdf
  16410   Mon Oct 18 10:02:17 2021 KojiUpdateVACVent Started / Completed

[Chub, Jordan, Anchal, Koji]

- Checked the main volume is isolated.
- TP1 and TP2 were made isolated from other volumes. Stopped TP1. Closed V4 to isolate TP1 from TP2.
- TP3 was made isolated. TP3 was stopped.
- We wanted to vent annuli, but it was not allowed as VA6 was open. We closed VA6 and vented the annuli with VAVEE.
- We wanted to vent the volume between VA6, V5, VM3, V7 together with TP1. So V7 was opened. This did not change the TP1 pressure (P2 = 1.7mmTorr) .
- We wanted to connect the TP1 volume with the main volume. But this was not allowed as TP1 was not rotating. We will vent TP1 through TP2 once the vent of the main volume is done.

- Satrted venting the main volume@Oct 18, 2021 9:45AM PDT

- We started from 10mTorr/min, and increased the vent speed to 200mTorr/min, 700mTorr/min, and now it is 1Torr/min @ 20Torr
- 280Torr @11:50AM
- 1atm  @~2PM


We wanted to vent TP1. We rerun the TP2 and tried to slowly introduce the air via TP2. But the interlock prevents the action.

Right now the magenta volume in the attachment is still ~1mTorr. Do we want to open the gate valves manually? Or stop the interlock process so that we can bypass it?

Attachment 1: Screen_Shot_2021-10-18_at_14.52.34.png
Screen_Shot_2021-10-18_at_14.52.34.png
Attachment 2: Screenshot_2021-10-18_15-08-59.png
Screenshot_2021-10-18_15-08-59.png
  16412   Tue Oct 19 10:59:09 2021 KojiUpdateVACVent Started / Completed

[Chub, Jordan, Yehonathan, Anchal, Koji]

North door of the BS chamber opened

 

  16413   Tue Oct 19 11:30:39 2021 KojiUpdateVACHow to vent TP1

I learned that TP1 was vented through the RGA room in the past. This can be done by opening VM2 and a manual valve ("needle valve")
I checked the setup and realized that this will vent RGA. But it is OK as long as we turns of the RGA during vent and bake it once TP1 is back.

Additional note:

- It'd be nice to take a scan for the current background level before the work.
- Turn RGA EM and filament off, let it cool down overnight. 
- Vent with clean N2 or clean air. (Normal operating temp ~80C is to minimize accumulation of H-C contaminations.)
- There is a manual switch and indicators on the top of the RGA amp. It has auto protection to turn filament off if the pressure increase over ~1e-5.

Attachment 1: Screen_Shot_2021-10-18_at_14.52.34.png
Screen_Shot_2021-10-18_at_14.52.34.png
  16418   Wed Oct 20 15:58:27 2021 KojiUpdateVACHow to vent TP1

Probably the hard disk of c0rga is dead. I'll follow up in this elog later today.

Looking at the log in /opt/rtcds/caltech/c1/scripts/RGA/logs , it seemed that the last RGA scan was Sept 2, 2021, the day when we had the disk full issue of chiara.
I could not login to c0rga from control machines.
I was not aware of the presence for c0rga until today, but I could locate it in the X arm.
The machine was not responding and it was rebooted, but could not restart. It made some knocking sound. I am afraid that the HDD failed.

I think we can
- prepare a replacement linux machine for the python scripts
or
- integrate it with c1vac

  16490   Mon Dec 6 14:26:52 2021 KojiUpdateVACPumping down the RGA section

Jordan reported that the RGA section needs to be pumped down to allow the analyzer to run at sufficiently low pressure (P<1e-4 torr).
The RGA section was pumped down with the TP2/TP3. The procedure is as listed below.
If the pressure go up to P>1e-4 torr, we need to keep the pump running until the scan is ready.

----
### Monitor / Control screen setup ###
1. On c1vac: cd /cvs/cds/caltech/target/c1vac/medm
2. medm -x C0VAC_MONITOR.adl&
3. RGA section (P4) 3.6e-1 torr / P3/P2 still atm.
4. medm -x C0VAC_CONTROL.adl

### TP2/TP3 backing ###
5. Turn on AUX RP with the circuit breaker hanging on the AC.
6. Open manual valve for TP2/3/ backing / PTP2/3 ~ 8torr

### TP2/TP3 starting ###
7. Turned on TP2/TP3 with the Standby OFF

### Pump down the pump spool ###
8. Connect manual RP line (Quick Connect)
9. Turned on RP1/RP3 -> quickly reached 0.4 torr
10. Open V6 for pump spool pumping -> Immediately go down to sufficiently low pressure for TP2/TP3.
(10.5 I had to close V6 at this point)
11. Open V5 to start pumping pump spool with TP3 (TP2 still stand by) -> P3 immediately goes down below 1e-4 torr. This automatically closed V6 because of the low pressure of P3 (interlocking)

### Pump down the RGA section ###
12. Open VM3 to pump down RGA section -> P4 goes down to <1e-4 torr
13. P2 is still 2e-3. So decided to open V4 to use TP2 (now it's ready) too. -> Saturated at 1.7e-3

### Shutting down ###
14. Close VM3
15. Close V4/V5 to isolate TP2/TP3
16. Stop TP2/TP3 -> Slowing down
17. Stop RP1/RP3
18. Close the manual valves for TP2/3/ backing
19. Stop AUX RP with the circuit breaker hanging on the AC.

  16493   Tue Dec 7 13:12:50 2021 KojiUpdateVACPumping down the RGA section

So that Jordan can run the RGA scan this afternoon, I ran TP3 and started pumping down the RGA section.

Procedure:
- Same 1~4
- Same 5
- 6 Opened only the backing path for TP3
- 7 Turned on TP3 only

- TP3 reached the nominal full speed @75kRPM

- 11 Opened V5 to pump the pump spool -> Immediately reached P3<1e-4
- 12 Opened VM3 to pump the RGA section -> Immediately reached P4<1e-4

The pumps are kept running. I'll come back later to shut down the pumps.
=> Jordan wants to heat the filament (?) and to run the scan tomorrow.
So we decided to keep TP3 running overnight. I switched TP3 to the stand-by mode (= lower rotation speed @50kRPM)

 

  16494   Wed Dec 8 10:14:43 2021 JordanUpdateVACPumping down the RGA section

After an overnight pumpdown/RGA warm up, I took a 100 amu scan of the RGA volume and subsequent pumping line. Attached is a screenshot along with the .txt file. Given the high argon peak (40) and the N2/O2 ratio, it looks like there is a decent sized air leak somehwere in the volume.

Are we interested in the hydrocarbon leak rates of this volume? That will require another scan with one of the calibrated leaks opened.

Edit: Added a Torr v AMU plot to see the partial pressures

Quote:

So that Jordan can run the RGA scan this afternoon, I ran TP3 and started pumping down the RGA section.

Procedure:
- Same 1~4
- Same 5
- 6 Opened only the backing path for TP3
- 7 Turned on TP3 only

- TP3 reached the nominal full speed @75kRPM

- 11 Opened V5 to pump the pump spool -> Immediately reached P3<1e-4
- 12 Opened VM3 to pump the RGA section -> Immediately reached P4<1e-4

The pumps are kept running. I'll come back later to shut down the pumps.
=> Jordan wants to heat the filament (?) and to run the scan tomorrow.
So we decided to keep TP3 running overnight. I switched TP3 to the stand-by mode (= lower rotation speed @50kRPM)

 

 

Attachment 1: 40m_RGAVolume_12_8_21.PNG
40m_RGAVolume_12_8_21.PNG
Attachment 2: 40m_RGAVolume_Torr_12_8_21.PNG
40m_RGAVolume_Torr_12_8_21.PNG
  16501   Fri Dec 10 19:22:01 2021 KojiUpdateVACPumping down the RGA section

The scan result was ~x10 higher than the previously reported scan on 2020/9/15 (https://nodus.ligo.caltech.edu:8081/40m/15570), which was sort of high from the reference taken on 2018/7/18.

This just could mean that the vacuum level at the RGA was x10 high.
We'll just go ahead with the vacuum repair and come back to the RGA once we return to "vacuum normal".

Meanwhile, I asked Jordan to turn off the RGA to make it cool down. I shut off RGA section and turned TP2 off.

  16508   Wed Dec 15 15:06:08 2021 JordanUpdateVACVacuum Feedthru Install

Jordan, Chub

We installed the 4x DB25 feedthru flange on the North-West port of ITMX chamber this afternoon. It is ready to go.

  16529   Tue Dec 21 16:35:39 2021 KojiUpdateVACITMX NW feedthru (LO1-1) connector pin bent

I've received a report that a pin of an ITMX NW feedthru connector was bent. (Attachment 1)
The connector is #1 (upper left) and planned to be used for LO1-1.

This is Pin25 and used for the PD K of OSEM #1. This means that Coil Driver #1 (3 OSEMs) uses this pin, but Coil Driver #2 (2 OSEMs) does not.

Anyways, I tried to fix it by bending it back. WIth some tools, it was straightened enough for plugging the cable connector. (Attachment 2)

It seemed that the pins were exceptionally soft compared to the ones used for usual DSUBs, probably because of the vacuum compatibility.
So it's better to approach the pins in parallel to the surface and not apply mating pressure until you are sure that all the 25pins are inserted in the counterpart holes.

Attachment 1: PXL_20211222_002019620.jpg
PXL_20211222_002019620.jpg
Attachment 2: PXL_20211222_003014068.jpg
PXL_20211222_003014068.jpg
  16634   Mon Jan 31 10:39:19 2022 JordanUpdateVACTP1 and Manual Gate Valve Removal

Jordan, Chub

Today, Chub and I removed TP1 and the failed manual gate valve off of the pumping spool.

First, P2 needed to be vented in order to remove TP1. TP1 has a purge valve on the side of the pump which we slowly opened bringing the P2 volume up to atmosphere. Although, this was not vented using the dry air/N2, using this purge valve eliminated the need to vent the RGA volume.

Then we disconnected TP1 foreline, removed TP1+8" flange reducer, then the gate valve. All of the removed hardware looked good, so no need to replace bolts/nuts, only needs new gaskets. TP1 and the failed valve are sitting on a cart wrapped in foil next to the pumping station.

Attachment 1: 20220131_100637.jpg
20220131_100637.jpg
Attachment 2: 20220131_102807.jpg
20220131_102807.jpg
Attachment 3: 20220131_102818.jpg
20220131_102818.jpg
Attachment 4: 20220131_100647.jpg
20220131_100647.jpg
  16643   Thu Feb 3 10:25:59 2022 JordanUpdateVACTP1 and Manual Gate Valve Install

Jordan, Chub

Chub and I installed the new manual gate valve (Nor-Cal GVM-6002-CF-K79) and reinstalled TP1. The new gate valave was placed with the sealing side towards the main 40m volume, then TP1 was installed on top and the foreline reattched to TP1.

This valve has a hard stop in the actuator to prevent over torquing.

 

Attachment 1: 20220203_101455.jpg
20220203_101455.jpg
Attachment 2: 20220203_094831.jpg
20220203_094831.jpg
Attachment 3: 20220203_094823.jpg
20220203_094823.jpg
  16682   Sat Feb 26 01:01:40 2022 TegaUpdateVACOngoing work to get the FRG gauges readouts to EPICs channels

I will make a detailed elog later today giving a detailed outline of the connection from the Agilent gauge controller to the vacuum subnet and the work I have been doing over the past two days to get data from the unit to EPICs channels. I just want to mention that I have plugged the XGS-600 gauge controller into the serial server on the vacuum subnet. I check the vacuum medm screen and I can confirm that the other sensors did not experience and issues are a result of this. I also currently have two of the FRG-700 connected to the controller but I have powered the unit down after the checks.

  16683   Sat Feb 26 15:45:14 2022 TegaUpdateVACOngoing work to get the FRG gauges readouts to EPICs channels

I have attached a flow diagram of my understanding of how the gauges are connected to the network.

Earlier today, I connected the XGS-600 gauge controller to the IOLAN Serial Device Server on port 192.168.114.22 .

The plan is a follows:

1. Update the serial device yaml file to include this new ip entry for the XGS-600 gauge controller

2. Create a serial gauge class "serial_gauge_xgs.py" for the XGS-600 gauge controller that inherits from the serial gauge parent class for EPICS communication with a serial device via TCP sockets.

  • Might be better to use the current channels of the devices that are being replaced initially, i.e.
  • C1:Vac-FRG1_pressure C1:Vac-CC1_pressure
    C1:Vac-FRG2_pressure C1:Vac-CCMC_pressure
    C1:Vac-FRG3_pressure C1:Vac-PTP1_pressure
    C1:Vac-FRG4_pressure C1:Vac-CC4_pressure
    C1:Vac-FRG5_pressure C1:Vac-IG1_pressure

3. Modify the launcher file to include the XGS gauge controller. Following the same pattern used  to start the service for the other serial gauges, we can start the communication between the XGS-600 gauge controller and the IOLAN serial server and write data to EPICS channels using

controls@c1vac> python launcher.py XGS600

If we are able to establish communication between the XGS-600 gauge controller and write it gause data to EPICS channels, go on to steps 4.

4. Create a serial service file "serial_XGS600.service" and place it in the service folder

5. Add the new EPICS channels to the database file

6. Add the "serial_XGS600.service" to line 10 and 11 of modbusIOC.service

7. Later on, when we are ready, we can restart the updated modbusIOC service

 

For vacuum signal flow and Acromag channel assignments see [1]  and [2] respectively. For the 16 port IOLAN SDS (Serial Device Server) ethernet connections, see [3]. 

[1] https://wiki-40m.ligo.caltech.edu/Vacuum-Upgrade-2018?action=AttachFile&do=view&target=40m_Vacuum_System_Signal_Flow.pdf

[2] https://wiki-40m.ligo.caltech.edu/Vacuum-Upgrade-2018?action=AttachFile&do=view&target=AcromagChannelAssignment.pdf

[3] https://git.ligo.org/40m/vac/-/blob/master/python/serial/serial_devices.yaml

Attachment 1: Vac-gauges-flow-diagram.png
Vac-gauges-flow-diagram.png
  16688   Mon Feb 28 19:15:10 2022 TegaUpdateVACOngoing work to get the FRG gauges readouts to EPICs channels

I decided to create an independent service for the XGS data readout so we can get this to work first before trying to integrate into current system. After starting the service, I noticed that the EPICS channel were not updating as expected. So I started to debug the problem and managed to track it down to an ip socket connect() error, i.e. we get a connection error for the ip address assigned to the LAN port to which the XGS box was connected. After trying a few things and searching the internet, I think the error indicates that this particular LAN port is not yet configured. I reached this conclusion after noting that only a select number of LAN ports connected without issues and these are the ports that already had devices connected. So it must be the case that the LAN ports were somehow configured. The next step is to look at the IOLAN manual to figure out how to configure the ip port for the XGS controller. Fingers crossed.

  16691   Tue Mar 1 20:38:49 2022 TegaUpdateVACOngoing work to get the FRG gauges readouts to EPICs channels

During my investigation, I inadvertently overwrote the serial port configuration for the connected devices. So I am now working to get it all back. I have attached screenshots of the config settings that brought back communication that is not garbled. There is no physical connection to port 6, which I guess was initially used for the UPS serial communication but not anymore. Also, ports 9 and 10 are connected to Hornet and SuperBee, both of which have not been communicating for a while and are to be replaced, so there is no way to confirm communication with them. Otherwise, the remaining devices seem to be communicating as before.

I still could not establish communication with the XGS-600 controller using the serial port settings given in the manual, which also happen to work via Serial to USB adapter, so I will revisit the problem later. My immediate plan is to do a Serial Ethernet, then Ethernet to Serial, and then Serial to USB connection to see if the USB code still works. If it does then at least I know the problem is not coming from the Serial to Ethernet adapters. Then I guess I will replace the controller with my laptop and see what signal comes through when I send a message to the controller via the IOLAN serial device server. Hopefully, I can discover what's wrong by this point.

 

Note to self: Before doing anything, do a sanity check by comparing the settings on the IOLAN SDS and the config settings that worked for the Serial to USB communication and post an elog for this for reference.

Attachment 1: Working_Serial_Port_List_1.png
Working_Serial_Port_List_1.png
Attachment 2: Working_Serial_Port_List_2.png
Working_Serial_Port_List_2.png
Attachment 3: Working_Config_Ports#1-5.png
Working_Config_Ports#1-5.png
Attachment 4: Working_Config_Ports#7-8.png
Working_Config_Ports#7-8.png
  16692   Wed Mar 2 11:50:39 2022 TegaUpdateVACOngoing work to get the FRG gauges readouts to EPICs channels

Here is the IOLAN SDS TCP socket setting and the USBserial setting for comparison.

I have also included the python script and output from the USBserial test from earlier.

Attachment 1: XGS600_IOLAN_settings_1.png
XGS600_IOLAN_settings_1.png
Attachment 2: XGS600_IOLAN_settings_2.png
XGS600_IOLAN_settings_2.png
Attachment 3: XGS600_USBserial_settings.png
XGS600_USBserial_settings.png
Attachment 4: XGS600_comm_test.py
#!/usr/bin/env python

#Created 2/24/22 by Tega Edo
'''Script to read/write to the XGS-600 Gauge Controller'''

import serial
import sys,os,math,time

ser = serial.Serial('/dev/cu.usbserial-1410') # open serial port 

... 74 more lines ...
Attachment 5: XGS600_comm_test_result.txt
----- Multiple Sensor Read Commands -----

Sent to XGS-600 -> #0001\r : Read XGS contents
response : >FE4CFE4CFE4C

Sent to XGS-600 -> #0003\r : Read Setpoint States
response : >0000

Sent to XGS-600 -> #0005\r : Read software revision
response : >0206,0200,0200,0200
... 69 more lines ...
  16693   Wed Mar 2 12:40:08 2022 TegaUpdateVACOngoing work to get the FRG gauges readouts to EPICs channels

Connector Test:

A quick test to rule out any issue with the Ethernet to Serial adapter was done using the setup shown in Attachment 1. The results rule out any connector problem.

 

IOLAN COMM test (as per Koji's suggestion):

The next step is to swap the controller with a laptop set up to receive serial commands using the same settings as the XGS600 controller. Basically, run a slightly modified version of python script where we go into listening mode. Then send commands to the TCP socket on the IOLAN SDS unit using c1vac and check what data makes its way to the laptop USBserial terminal. After working on this for a bit, I realized that we do not need to do anything on the c1vac machine. We only need to start the service as it would work normally. So I wrote a small python code for a basic XGS-600 controller emulator, see Attachment 4. The outputs from the laptop and c1vac terminals are Attachments 5 and 6 respectively. 

These results show that we can communicate via the assigned IP address "192.168.114.22" and the commands that are sent from c1vac reaches the laptop in the correct format. Furthermore, the serial_XSG service, a part modbusIOC_XGS service, which usually exits with an error seems fine now after successfully communicating with the laptop. I don't know why it did not die after the tests. I also found a bug in my code as a result of the test, where the status field for the fourth gauge didn't get written to. 

 

Pressure reading issue:

I noticed that the pressure reading was not giving the atmospheric value of ~760 Torrs as expected. Looking through my previous readouts, it seems the unit showed this atm value of ~761 Torrs when the first gauge was attached. However, a closer look at the issue revealed a transient behavior, i.e. when the unit is turned on the reading dips to atm value but eventually rises up to 1000 Torrs. I don't think this is a calibration problem bcos the value of 1000 Torrs is the maximum value for the gauge range. I also found out that when the XGS-controller has been running for a while, a power cycle does not have this transient behavior. So maybe a faulty capacitor somewhere? I have attached a short video clip that shows what happens when the XGS-controller unit is turned on.

Attachment 1: IMG_20220302_123529382.jpg
IMG_20220302_123529382.jpg
Attachment 2: XGS600_Serial2Ethernet2Serial2USB_comm_test_result.txt
$ python3 XGS600_comm_test.py  

----- Multiple Sensor Read Commands -----

Sent to XGS-600 -> #0001\r : Read XGS contents
response : >FE4CFE4CFE4C

Sent to XGS-600 -> #0003\r : Read Setpoint States
response : >0000

... 73 more lines ...
Attachment 3: VID-20220302-WA0001.mp4
Attachment 4: comm_test_c1vac_to_laptop_via_iolansds.py
#!/usr/bin/env python

#Created 3/2/22 by Tega Edo
'''Script to emulate XGS-600 controller using laptop USBserial port'''

import serial
import sys,os,math,time

ser = serial.Serial('/dev/cu.usbserial-1410') # open serial port 

... 19 more lines ...
Attachment 5: laptop_terminal.txt
(base) tega.edo@Tegas-MBP serial % python3 comm_test_c1vac_to_laptop_via_iolansds.py

----- Listen for USBserial command and asynchronously send data in XGS600 format -----

Command received from c1vac [1] : 

Data sent to c1vac [1] : >1.000E+00,NOCBL    ,NOCBL    ,NOCBL    ,2.00E+00,NOCBL\r
Command received from c1vac [2] : 

Data sent to c1vac [2] : >2.000E+00,NOCBL    ,NOCBL    ,NOCBL    ,3.00E+00,NOCBL\r
... 54 more lines ...
Attachment 6: c1vac_terminal.txt
controls@c1vac:/opt/target/python/serial$ caget C1:Vac-FRG1_status && caget C1:Vac-FRG2_status && caget C1:Vac-FRG3_status && caget C1:Vac-FRG4_status && caget C1:Vac-FRG5_status
C1:Vac-FRG1_status             1.530E+02
C1:Vac-FRG2_status             OFF
C1:Vac-FRG3_status             OFF
C1:Vac-FRG4_status             NO COMM
C1:Vac-FRG5_status             1.55E+02
controls@c1vac:/opt/target/python/serial$ caget C1:Vac-FRG1_status && caget C1:Vac-FRG2_status && caget C1:Vac-FRG3_status && caget C1:Vac-FRG4_status && caget C1:Vac-FRG5_status
C1:Vac-FRG1_status             1.630E+02
C1:Vac-FRG2_status             OFF
C1:Vac-FRG3_status             OFF
... 70 more lines ...
  16701   Fri Mar 4 18:12:44 2022 KojiUpdateVACRGA pumping down

1. Jordan reported that the newly installed Pirani gauge for P2 shows 850Torr while PTP2 show 680 Torr. Because of this, the vacuum interlock fails when we try to open V4.

2. Went to c1vac. Copied the interlock setting file interlock_conditions.yaml to interlock_conditions_220304.yaml
3. Deleted diffpressure line and pump_underspeed line for V4
4. Restarted the interlock service

controls@c1vac:/opt/target/python/interlocks$ sudo systemctl status interlock.service  
controls@c1vac:/opt/target/python/interlocks$ sudo systemctl restart interlock.service
controls@c1vac:/opt/target/python/interlocks$ sudo systemctl status interlock.service

5. The above 2~4 was unnecessary. Start over.


Let RP1/3 pump down TP1 section through the pump spool. Then let TP2 pump down TP1 and RGA.

1. Open V7. This made P2 a bit lower (P2 is alive) and P3.
2. Connected the main RP tube to the RP port.
2. Started RP1/3. PRP quickly reaches 0.4Torr.
3. Opened V6 this made P3 and O2 below 1Torr.
4. Close V6. Shutdown RP1/3. Disconnect the RP tube.
5. Turn on auxRP at the wall powe
6. Turn on TP2. Wait for the starting up.
7. Open V4. Once the pressure is below Pirani range, open VM3.
8. Keep it running over the weekend.

9. Once TP2 reached the nominal speed, the "StandBy" button was clicked to lower the rotation speed (for longer life of TP2)

Attachment 1: Screenshot_2022-03-04_19-38-11.png
Screenshot_2022-03-04_19-38-11.png
  16704   Sun Mar 6 18:14:45 2022 TegaUpdateVACOngoing work to get the FRG gauges readouts to EPICs channels

Following repeated failure to establish communication between c1vac and the XGS600 controller via Perle IOLAN serial device server, I decided to monitor the signal voltage of the communication channels (pin#2, pin#3 and pin#5) using an oscilloscope. The result of this investigation is presented in the attached pdf document. In summary, it seems I have used a crossed wired RS232 serial cable instead of a normal RS232 serial cable, so the c1vac read request command is being relayed on the wrong comm channel (pin#2 instead of pin#3). I will swap out the cable to see if this resolves the problem.  

Attachment 1: iolan_xgs_comm_investigation.pdf
iolan_xgs_comm_investigation.pdf iolan_xgs_comm_investigation.pdf iolan_xgs_comm_investigation.pdf iolan_xgs_comm_investigation.pdf
  16706   Mon Mar 7 13:53:40 2022 TegaUpdateVACOngoing work to get the FRG gauges readouts to EPICs channels

So it appears that my deduction from the pictures of needing a cable swap was correct, however, it turns out that the installed cable was actually the normal RS232 and what we need instead is the RS232 null cable. After the swap was done, the communication between c1vac and the XGS600 controller became active. Although, the data makes it all to the to c1vac without any issues, the scope view of it shows that it is mainly utilizing the upper half of the voltage range which is just over 50% of the available range. I don't know what to make of this.

 

I guess, the only remaining issue now is the incorrect atmospheric pressure reading of 1000 Torrs. 

 

Quote:

Following repeated failure to establish communication between c1vac and the XGS600 controller via Perle IOLAN serial device server, I decided to monitor the signal voltage of the communication channels (pin#2, pin#3 and pin#5) using an oscilloscope. The result of this investigation is presented in the attached pdf document. In summary, it seems I have used a crossed wired RS232 serial cable instead of a normal RS232 serial cable, so the c1vac read request command is being relayed on the wrong comm channel (pin#2 instead of pin#3). I will swap out the cable to see if this resolves the problem.  

 

Attachment 1: iolan_xgs_comm_live.pdf
iolan_xgs_comm_live.pdf
  16707   Mon Mar 7 14:52:34 2022 KojiUpdateVACOngoing work to get the FRG gauges readouts to EPICs channels

Great trouble shoot!

> I guess, the only remaining issue now is the incorrect atmospheric pressure reading of 1000 Torrs. 

This is just a calibration issue. The controller should have the calibration function.
(The other Pirani showing 850Torr was also a calibration issue although I didn't bother to correct it. I think the pirani's typically has large distribution of the calibration values and requires individual calibration)

  16713   Tue Mar 8 12:08:47 2022 TegaUpdateVACOngoing work to get the FRG gauges readouts to EPICs channels

OMG, it worked! It was indeed a calibration issue and all I had to do was press the "OK" button after selecting the "CAL" tab beside the pressure reading. Wow.

Quote:

Great trouble shoot!

> I guess, the only remaining issue now is the incorrect atmospheric pressure reading of 1000 Torrs. 

This is just a calibration issue. The controller should have the calibration function.
(The other Pirani showing 850Torr was also a calibration issue although I didn't bother to correct it. I think the pirani's typically has large distribution of the calibration values and requires individual calibration)

 

Attachment 1: XGS600_calibration.pdf
XGS600_calibration.pdf
  16717   Wed Mar 9 10:23:28 2022 JordanUpdateVACLeak Testing of New Manual Gate Valve - Attempt #1

Jordan, Chub, Paco

Chub and I went into the lab this morning to leak check the new gate valve after pumping over the weekend. This would be done through the RGA and spraying helium around the newly installed flanges. The RGA is set to monitor the partial pressure of helium versus time and we visually watch for any spikes in pressure to indicate an air leak.

So, I mistakenly thought the gate valve was opened and both sides were being pumped on, this was not the case. The vavle was closed so there was a pressure differential whenI turned the handle it tripped the interlocks and closed V4. I closed VM3 to the RGA volume to prevent the filament from being damaged.

Then the medm screen for the vac controls started flashing rapidly, I closed the window and reopened the controls to find all the panels were white, but we could still see the read only vac screen. So Paco restarted c1vac and the controls were restored. V7 closed, but was then reopened.

We now need to restart pumping with the odd pressure differentials between V4 and VM3.

Plan

- TP2 turned off along with the AUX pump

- VM3 opened to vent RGA volume (RGA turned off, filament cooled)

- Open the manual vent screw on TP1 to bring the RGA+TP1 volume back to atmosphere, now no pressure diff. on V4

- Open V4, restart AUX pump to rough out the volume

- Connect RP1/3 line to the pump scroll, turn on RP1/3 RGA+TP1 volume went to mtorr

- Slowly open the manual gate valve

- Connect AUX pump to TP2 and rough out

- Restart TP2, once at speed open V4, close V6 and turn off RP1/3

Now we are pumping on the RGA/TP1 volume with TP2, leak check attempt #2 will happen tomorrow morning

  16721   Thu Mar 10 09:39:59 2022 JordanUpdateVACLeak Testing of New Manual Gate Valve - Attempt #2

This morning Chub and I leak checked the manual gate valve with the RGA and helium. There was no change in the helium partial pressure while spraying helium around the flanges, all looks good.

 

I also took a 100 AMU analog scan, after the filament had warmed up overnight and the plot was quite noisy even with the lowest scan speed. I recommend this unit go back to SRS for a filament replacement/recalibration. I am worried yesterday's "vent" of the RGA volume may have burned the filament. See the comparison of yesterday's analog scan to today's below.

Attachment 1: 3-09-2022.PNG
3-09-2022.PNG
Attachment 2: 3-10-2022.PNG
3-10-2022.PNG
  16747   Tue Mar 29 15:07:11 2022 TegaSummaryVACOpening of MC and ETM chambers

[Anchal, Chub, Ian, Paco, Tega]

Today, we opened the MC chamber, ETMX chamber and ETMY chamber.

  16769   Mon Apr 11 11:00:30 2022 JancarloUpdateVACC1VAC Reboot and Nitrogen tanks

[Paco, JC, Ian, Jordan, Chub]

Checking in the morning, I walked over to the Nitrogen tanks to check the levels. Noticed one tank was empty, so I swapped it out. Chub came over to check the levels and to take note of how many tanks were left available for usage (None). Chub continued to put in a work order for a set of full Nitrogen tanks. We should be set on Nitrogen until Thursday this week (4/14/22).

As for C1VAC, this morning, Paco and I attempted to open the PSL shutter, but the interlock system was tripped so we didn't get any light into the IFO. We traced the issue down to C1VAC being unresponsive. We discussed this may have interlocked as a result of the Nitrogen tanks running out, but we do not believe this was the issue since we would have recieved an email. We tried troubleshooting as much as possible avoiding a reboot, but were unable to solve the issue. In response, we ran the idea of a reboot across Jordan and Ian, where everyone was in agreement, and fixed the system. Restarting c1vac seems to have closed V4, but this didn't cause any issues with the current state of the vacuum system.

After opening the PSL shutter again, we see the laser down the IFO, so we resume alignment work

  16772   Tue Apr 12 09:05:21 2022 JordanUpdateVACNew Pressure Gauge Install/Pump Spool Vent

Today, Tega and I would like to vent the pump spool an dinstall the new FRG-400 Agilent Pressure Gauges (per elog 15703). The attached picture shows the volume needed to be vented highlighted in red, and the gauges that need to be replaced/removed (purple dot next to the name).

The vent plan is as follows:

Open RV2

Open VM3

Open V7

Open V4

Shut down TP2

Install new gauges

Will add to post with updates post vent.

Attachment 1: Screenshot_2022-04-12_08-42-33.png
Screenshot_2022-04-12_08-42-33.png
Attachment 2: 81CB0936-1B19-4722-8A32-C3DC1D1FBC21.heic
Attachment 3: 2262ECEF-6200-4E95-8E32-C83CB9EB4F17.heic
Attachment 4: B5712B34-ECF6-43BE-BCB2-38CD775CF653.heic
Attachment 5: BEE14A07-976A-45C2-82A0-1774D377941E.heic
Attachment 6: 84557D6E-6AAE-47CF-A3AD-1DF329FEB550.heic
  16773   Wed Apr 13 12:50:07 2022 TegaUpdateVACNew Pressure Gauge Install/Pump Spool Vent

[Jordan, JC, Tega]

We have installed all the FRGs and updated the VAC medm screens to display their sensor readings. The replacement map is CC# -> FRG#, where # in [1..4] and PRP1 -> FRG5. We now need to clean up the C1VAC python code so that it is not overloaded with non-function gauges (CC1,CC2,CC3,CC4,PRP1). Also, need to remove the connection cables for the old replaced gauges.

Attachment 1: VAC_FRG_UPGRADE.png
VAC_FRG_UPGRADE.png
  16777   Thu Apr 14 09:04:30 2022 JordanUpdateVACRGA Volume RGA Scans

Prior to venting the RGA volume on Tuesday (4/12/2022) I took an RGA scan of the volume to be vented (RGA+TP1 volume+Manual Gate Valve) to see if there was a difference after replacing the manual gate valve. Attached is the plot from 4/12/22, and an overlay plot to complare 4/12/22 to 12/10/2021, when the same volume was scanned with the old (defective) manual gate valve.

There is a significant drop in the ratio O2 compared the the nitrogen peak and reduced Argon (AMU 40) which indicates there is no longer a large air leak.

12/10/21 N2/O2 ratio ~ 4 (Air 78%N2 / 21%O2)

4/12/22 N2/O2 ratio ~ 10      

There is one significant (above noise level) peak above AMU 46, which is at AMU 58. This could possibly be acetone (AMU 43 and 58) but overall the new RGA Volume scans look significantly better after the manual gate valve replacement. Well done!

Attachment 1: 40mRGA_Overlay.pdf
40mRGA_Overlay.pdf
Attachment 2: RGAVolume_4_12_22.PNG
RGAVolume_4_12_22.PNG
  16794   Thu Apr 21 11:31:35 2022 JCUpdateVACGauges P3/P4

[Jordan, JC]

It was brought to attention during yesterday's meeting that the pressures in the vacuum system were not equivalent althought the valve were open. So this morning, Jordan and I reviewed the pressure gauges P3 and P4. We attempted to recalibrate, but the gauges were unresponsive. Following this, we proceeded to connect new gauges on the outside to test for a calibration. The two gauges successfully calibrated at atmosperic pressure. We then proceeded to remove the old gauges and install the new ones. 

Attachment 1: IMG_0560.jpeg
IMG_0560.jpeg
Attachment 2: IMG_0561.jpeg
IMG_0561.jpeg
  16796   Thu Apr 21 16:36:56 2022 TegaUpdateVACcleanup work for vacuum git repo

git repo - https://git.ligo.org/40m/vac

Finally incorporated the FRGs into the main modbusIOC service and everything seems to be working fine. I have also removed the old sensors (CC1,CC2,CC3,CC4,PTP1,IG1) from the serial client list and their corresponding EPICS channels. Furthermore, the interlock service python script has been updated so that all occurrence of old sensors (turns out to be only CC1) were replaced by their corresponding new FRG sensor (FRG1) and a redundnacy was also enacted for P1a where the interlock condition is replicated with P1a being replaced with FRG1 because they both sense the main volume pressure.

  16806   Fri Apr 22 14:14:33 2022 JordanUpdateVACTP3 Forepump tip seal replacement

Jordan, JC

While the pumpspool is vented, I thought it would be a convenient time to change out the tip seal on the TP3 forepump. This one had not been changed since 2018, so as preventative maintence I had JC remove the pump and begin cleaning/installing the new tip seal.

Unfortunately the tip seal broke, but I have ordered another. We should have this pump ready to go late next week. If one is needed sooner, there is a spare IDP 7 pump we can install as the TP3 forepump.

Attachment 1: IMG_0572.jpeg
IMG_0572.jpeg
Attachment 2: IMG_0573.jpeg
IMG_0573.jpeg
  16820   Fri Apr 29 08:34:40 2022 JCUpdateVACRGA Pump Down

Jordan and I, in order to start pumpig down the RGA Volume, we began by opening V7 and VM. Afterwards, we started RP1 and RP3. After this, the pressure in the line between RP1, RP3, and V6 dropped to 3.4 mTorr. Next, we tried to open V6, although an error message popped up. We haven't been able to erase it since. But we were able to turn on TP2 with V4 closed. The pressure in that line is reporting 1.4 mTorr.

 

PRP on the sitemap is giving off an incorrect pressure for the line between RP1, RP3, and V6. This is verified by the pressure by the control screen and the physical controller as well. 

Attachment 1: Screen_Shot_2022-04-29_at_8.46.53_AM.png
Screen_Shot_2022-04-29_at_8.46.53_AM.png
  16822   Mon May 2 10:00:34 2022 JordanUpdateVACTP3 Forepump tip seal replacement

[JC, Jordan]

Jordan recieved the new tip seal Friday afternoon and I continued the replacement process in the morning. Finishing up, we proceeded to test the pump in the Clean and Bake room. The pump's pressure lowered to 110 mTorr, and we continue pumping so the seal can recieve a good fitting.

Update: We have confirmed the pump is working great and have reinstalled this back into the vacuum system.                                                                                                                   Note: The same O-Rings were used.

Quote:

Jordan, JC

While the pumpspool is vented, I thought it would be a convenient time to change out the tip seal on the TP3 forepump. This one had not been changed since 2018, so as preventative maintence I had JC remove the pump and begin cleaning/installing the new tip seal.

Unfortunately the tip seal broke, but I have ordered another. We should have this pump ready to go late next week. If one is needed sooner, there is a spare IDP 7 pump we can install as the TP3 forepump.

 

  16825   Tue May 3 13:18:47 2022 JCUpdateVACRGA Pump Down

Jordan, Tega, JC

Issue has been resolved. Breaker on RP1 was tripped so the RP1 button was reporting ON, but was not actually on which continuously tripped the V6 interlock. Breaker was reset, RP1 and RP3 turned on. The V6 was opened to rough out the RGA volume. Once, pressure was at ~100mtorr, V4 was opened to pump the RGA with TP2. V6 was closed and RP1/3 were turned off.

RGA is pumping down and will take scans next week to determine if a bakeout is needed

Quote:

Jordan and I, in order to start pumpig down the RGA Volume, we began by opening V7 and VM. Afterwards, we started RP1 and RP3. After this, the pressure in the line between RP1, RP3, and V6 dropped to 3.4 mTorr. Next, we tried to open V6, although an error message popped up. We haven't been able to erase it since. But we were able to turn on TP2 with V4 closed. The pressure in that line is reporting 1.4 mTorr.

 

PRP on the sitemap is giving off an incorrect pressure for the line between RP1, RP3, and V6. This is verified by the pressure by the control screen and the physical controller as well. 

 

  16867   Fri May 20 12:42:23 2022 TegaUpdateVACDoor installation on the end stations

[JC, Tega, Chub]

Today we installed the 200 lbs doors on the end station chambers.

  16870   Tue May 24 10:37:09 2022 TegaUpdateVACadded FRG channels to slow channel ini file

[Vacuum gauge sensors]

Paco informed me that the FRG sensor EPICS channels are not available on dataviewer, so I added them to slow channels ini file (/opt/rtcds/caltech/c1/chans/daq/C0EDCU.ini). I also commented out the old CC1, CC2, CC3 and CC4 gauges. A service restart is required for them to become available but this cannot be done right now because it would adversely affect the progress of the upgrade work. So this would be done at a later date.

Quote:

git repo - https://git.ligo.org/40m/vac

Finally incorporated the FRGs into the main modbusIOC service and everything seems to be working fine. I have also removed the old sensors (CC1,CC2,CC3,CC4,PTP1,IG1) from the serial client list and their corresponding EPICS channels. Furthermore, the interlock service python script has been updated so that all occurrence of old sensors (turns out to be only CC1) were replaced by their corresponding new FRG sensor (FRG1) and a redundnacy was also enacted for P1a where the interlock condition is replicated with P1a being replaced with FRG1 because they both sense the main volume pressure.

 

  16871   Tue May 24 11:04:53 2022 JCUpdateVACBeginning Pumpdown

[JC, Jordan, Paco, Chub]

We began with the pumpdown this morning. We started with the annulus volume and proceeded by using the following:

1. Isolate the RGA Volume by closing of valves VM3 and V7.

2. Opened valves VASE, VASV, VABSSCT, VABS, VABSSCO, VAEV, and VAEE, in that order.

3. Open VA6 to allow P3, FRG3, and PAN to equalize.

4. Turn on RP1 and RP3, rough out annulus volume, once <1 torr turn on TP3. Close V6. Open V5 to pump the annulus volume with TP3.

5. Re route pumping from RP1 and RP3 to the main volume by opening V3 and slowly opening RV1.

6. After ~3.5 hours the pressure in the arms was <500mtorr on both FRG1 and P1a. Turn on TP1 and wait to reach full speed 560 Hz

7. Open V1 with RV2 barely open. The pressure diff between P1a and P2/FRG2 needs to be below 1 torr. This took a couple attempts with the manual valve in different positions. The interlocks were tripped for this reason. Repeat step 7 until the manual gate valve was in a position that throttled pumping enough to maintain the <1 torr differential.

8. Slowly open the manual gate valve over the course of ~ 1 hour. Once the manual gate valve fully opened, pressure in the arms was <1mtorr.

9. V7 was closed, leaving only TP2 to back TP1, while TP3 was used to continue pumping the annuli. Left in that configuration overnight (see attached)

 

We did have to replace gauge PAN becuase it was reading a signal error. In addition, we found the cable is a bit sketchy and has a sharp bend. The signal comes in and out when the cable is fiddled with.

Attachment 1: PUMPDOWN-2022-05-24_16-57-59.png
PUMPDOWN-2022-05-24_16-57-59.png
Attachment 2: C1VAC_Screenshot_2022-05-24_16-59-27.png
C1VAC_Screenshot_2022-05-24_16-59-27.png
  16900   Wed Jun 8 16:03:12 2022 KojiUpdateVACPower Outage 220608: Vacuum Recovery

=== Observation ===

  • TP1/2/3 failed. AUX RP did not survive the power outage (of course)

=== Initial Recovery of TP2/TP3 ===

  • Stopped TP1/2/3
  • Restored RP AUX at the wall breaker module
  • Restarted TP2/3

=== Rough pumping down ===

  • Found RP1/RP3 left connected (unusual)
  • Turned on RP1 and RP3
  • Opened V6 and V7 for roughing of the pump spool. Now 0.3 torr for P2/P3/PA* (annuli) - ready to open V4 and V5 to backup TP1 by TP2 and TP3. Waiting for TP2/3 to be accelerated

=== Towards main volume pumping ===

  • TP2 accelerated and opened V4 (This caused V6 to close). RP1 and RP3 stopped.
  • TP3 hasitated to spin up. Stopped for now.
  • TP1 needed to be reset to clear the displacement errors. Once the errors were cleared, TP1 was restarted (acclerating).
  • TP1 at normal speed. Opened V1 (main volume pumping started)
     
  • Disconnected RP1/RP3 line.
     
  • TP3 was restarted and is now stable at 55k RPM (indicated as soft spin / low speed). Pushed "Stand By On" and "then "Stand By OFF". This makes the TP3 back to "NORMAL". Switched to "Stand By On".
  • TP2 was also switched to "Stand By On".

Now the main volume and the annuli are pumped down TP1 and TP2/TP3 with RP AUX backing.

The attachment is the pressure glitch for the main volume.

 

Attachment 1: Screenshot_2022-06-08_16-43-13.png
Screenshot_2022-06-08_16-43-13.png
  16916   Wed Jun 15 07:26:35 2022 JCUpdateVACBeginning Pumpdown

[Jordan, JC]

Jordan and I went in to retore the Vacuum System back to it's original state before the power loss on June 8, 2022. The process went smoothly as we first closed V7 and opened VM3 (in that order).

The RP1/3 line did not have the KF blank installed. That was added and the RP flex line was capped off.

Quote:

[JC, Jordan, Paco, Chub]

We began with the pumpdown this morning. We started with the annulus volume and proceeded by using the following:

1. Isolate the RGA Volume by closing of valves VM3 and V7.

2. Opened valves VASE, VASV, VABSSCT, VABS, VABSSCO, VAEV, and VAEE, in that order.

3. Open VA6 to allow P3, FRG3, and PAN to equalize.

4. Turn on RP1 and RP3, rough out annulus volume, once <1 torr turn on TP3. Close V6. Open V5 to pump the annulus volume with TP3.

5. Re route pumping from RP1 and RP3 to the main volume by opening V3 and slowly opening RV1.

6. After ~3.5 hours the pressure in the arms was <500mtorr on both FRG1 and P1a. Turn on TP1 and wait to reach full speed 560 Hz

7. Open V1 with RV2 barely open. The pressure diff between P1a and P2/FRG2 needs to be below 1 torr. This took a couple attempts with the manual valve in different positions. The interlocks were tripped for this reason. Repeat step 7 until the manual gate valve was in a position that throttled pumping enough to maintain the <1 torr differential.

8. Slowly open the manual gate valve over the course of ~ 1 hour. Once the manual gate valve fully opened, pressure in the arms was <1mtorr.

9. V7 was closed, leaving only TP2 to back TP1, while TP3 was used to continue pumping the annuli. Left in that configuration overnight (see attached)

 

We did have to replace gauge PAN becuase it was reading a signal error. In addition, we found the cable is a bit sketchy and has a sharp bend. The signal comes in and out when the cable is fiddled with.

 

  16980   Fri Jul 8 14:03:33 2022 JCHowToVACVacuum Preparation for Power Shutdown

[Koji, JC]

Koji and I have prepared the vacuum system for the power outage on Saturday.

  1. Closed V1 to isolate the main volume.
  2. Closed of VASE, VASV, VABSSCI,VABS, VABSSCO, VAEV, and VAEE.
  3. Closed V6, then close VM3 to isolate RGA
  4. Turn off TP1 (You must check the RPMs on the TP1 Turbo Controller Module)
  5. Close V5
  6. Turn off TP3 (There is no way to check the RPMs, so be patient)
  7. Close V4 (System State changes to 'All pneumatic valves are closed)
  8. Turn off TP2 (There is no way to check the RPMs, so be patient)
  9. Close Vacuum Valves (on TP2 and TP3) which connect to the AUX Pump.
  10. Turn of AUX Pump with the breaker switch wall plug.

From here, we shutdown electronics.

  1. Run /sbin/shutdown -h now on c1vac to shut the host down.
  2. Manually turn off power to electronic modules on the rack.
    • GP316a
    • GP316b
    • Vacuum Acromags
    • PTP3
    • PTP2
    • TP1
    • TP2 (Unplugged)
    • TP3 (Unplugged)

 

Attachment 1: Screen_Shot_2022-07-12_at_7.02.14_AM.png
Screen_Shot_2022-07-12_at_7.02.14_AM.png
  16985   Mon Jul 11 15:26:12 2022 JCHowToVACStartup after Power Outage

[Koji, Jc]

Koji and I began starting that Vacuum system up.

  1. Reverse order step 2 of shutting down electronics. Anthing after, turn on manually.
  2. If C1vac does not come back, then restart by holding the reset button.
  3. Open VA6
  4. Open VASE, VASV,VABSSCI, VABS, VABSSCO, VAEV, and VAEE
  5. Open V7
  6. Check P3 and P2, if they are at high pressure, approx. 1 Torr range, then you must use the roughing pumps.
  7. Connect Rotary pump tube. (Manually)
  8. Turn on AUX Pump
  9. Manually open TP2 and TP3 valves.
  10. Turn on TP2 and TP3, when the pumps finish startup, turn off Standby to bring to nominal speed.
  11. Turn on RP1 and RP3
  12. Open V6
  13. Once P3 reaches <<1 Torr, close V6 to isolate the Roughing pumps.
  14. When TP2 and TP3 are at nominal speed, open V5 and V4.
  15. Now TP1 is well backed, turn on TP1.
  16. When TP1 is at nominal speed, Open V1.
  16986   Mon Jul 11 17:25:43 2022 TegaUpdateVACfixed obsolete reference bug in serial_XGS600 service

Koji noticed that the FRG sensors were not updating due to reference to an obsolete modbusIOC_XGS service, which was used temporarily to test the operation of the serial XGS sensor readout to EPICS. The information in this service was later moved into modbusIOC.service but the dependence on the modbusIOC_XGS.service was not removed from the serial_XGS600.service. This did not present any issue before the shutdown, probably bcos the obsolete service was already loaded but after the restart of c1vac, the obsolete service file modbusIOC_XGS.service was no longer available. This resulted in  serial_XGS600.service throwing a failure to load error for the missing obsolete modbusIOC_XGS service. The fix involved replacing two references to 'modbusIOC_XGS' with 'modbusIOC' in  /opt/target/services/serial_XGS600.service.

I also noticed that the date logged in the commit message was Oct 2010 and that I could not do a push from c1vac due to an error in resolving git.ligo.org. I was able to push the commit from my laptop git repo but was unable to do a pull on c1vac to keep it synced with the remote repo.

  16987   Mon Jul 11 17:41:52 2022 KojiHowToVACStartup after Power Outage

- Once the FRG gauge readings are back (see next elog by Tega), I could open V1 to pump down the main vacuum manifold.
- TP2/TP3 were brought back to stand-by mode (slower spinning)
- V7 was closed to separate the annuli side and TP1

During the vacuum recovery, I saw TPs were automatically turned on as soon as the backing pumps were engaged. I could not figure out what caused this automation.

Also, I saw some gate valve states changed while I was not touching them. e.g. V7 was close / VM3 was open / etc
I really had no idea what/who was handling these.

As of ~18:00 local, the main volume pressure is ~2e-5 torr and ready to open the PSL shutter.

Attachment 1: Screen_Shot_2022-07-11_at_18.13.00.png
Screen_Shot_2022-07-11_at_18.13.00.png
  4274   Fri Feb 11 16:43:09 2011 steveUpdateVIDEOMC1 & 3 video monitor

I set up video monitoring of MC1 and MC3

Attachment 1: P1070415.JPG
P1070415.JPG
  4314   Thu Feb 17 13:20:06 2011 SureshUpdateVIDEOSome more labels

[Larisa, Kiwamu, Steve and Suresh]

 

  We continued the labeling of video cables. All exiting cables which are going to be used used in the new scheme have been labeled.

We also labeled the cables running from the video mux to the TV monitors in the computer room. Some of these will be removed or reallocated.

We will continue next Wednesday (after the meeting) and will lay cables that are most urgently required. 

 i

  4346   Wed Feb 23 16:56:17 2011 Larisa ThorneUpdateVIDEOCable laying...continued

Having finished labeling the existing cables to match their new names, we (Steve, Kiwamu and Larisa) moved on to start laying new cables and labeling them according to the list.

 

Newly laid cables include: ETMXT (235'), ETMX (235'), POP (110') and MC2 (105').  All were checked by connecting a camera to a monitor and checking the clarity of the resulting image. Note that these cables were only laid, so they are not plugged in.

 

The MC2 cable needs to be ~10' longer; it won't reach to where it's supposed to. It is currently still in its place. 

The three other cables were all a lot longer than necessary.

ELOG V3.1.3-