 Thu Dec 3 10:33:00 2020, gautam, Update, VAC, TrippLite UPS delivered 
|
The latest greatest UPS has been delivered. I will move it to near the vacuum rack in its packaging for storage. It weighs >100lbs so care will have
to be taken when installing - can the rack even support this? |
 Thu Dec 3 14:53:58 2020, Jon, Update, VAC, Replacing pressure gauges
|
Update to the gauge replacement plan (15692), based on Jordan's walk-through today. He
confirmed:
All of the gauges being replaced are mounted via 2.75" ConFlat flange. The new FRGs have the same footprint, so no adapters |
|
Wed Dec 9 20:14:49 2020, gautam, Update, VAC, UPS failure
|
Summary:
The (120V) UPS at the vacuum rack is faulty.
The drypump backing TP2 is faulty.
Current status of vacuum system:
The
old UPS is now powering the rack again. Sometime ago, I noticed the "replace battery" indicator light on this unit was on. But it is no longer |
|
Thu Dec 10 11:07:24 2020, Chub, Update, VAC, UPS fault
|
| Is that a fault code that you can decipher in the manual, or just a light telling you nothing but your UPS is dead? |
|
Thu Dec 10 11:17:50 2020, Chub, Update, VAC, UPS fault
|
I can't find anything in the manual that describes the nature of the FAULT message. In fact, it's not mentioned at all. If the unit
detects a fault at its output, I would expect a bit more information. This unit does a programmable level of input error protection, too, usually
set at 100%. Still, there is no indication in the manual whether an input issue would be described as a fault; that usually means a short or lifted |
|
Thu Dec 10 13:05:52 2020, Jon, Update, VAC, UPS failure
|
I've investigated the vacuum controls failure that occurred last night. Here's what I believe happened.
From looking at the system logs, it's clear that there was a sudden loss of power to the control computer (c1vac). Also, the system
was actually down for several hours. The syslog shows normal EPICS channel writes (pressure readback updates, etc., and many of them per minute) which |
|
Thu Dec 10 14:29:26 2020, gautam, Update, VAC, UPS failure
|
I don't buy this story - P2 only briefly burped around GPStime 1291608000 which is around 8pm local time, which is when I was recovering the system.
Today. Jordan talked to Jon Feicht - apparently there is some kind of valve in the TP2 forepump, which only opens ~15-20 seconds after turning
the pump on. So the loud sound I was hearing yesterday was just some transient phenomenon. So today morning at ~9am, we turned on TP2. Once again, PTP2 |
|
Wed Jan 6 15:28:04 2021, gautam, Update, VAC, Vac rack UPS batteries replaced
|
[chub, gautam]
the replacement was done this afternoon. The red "Replace Battery" indicator is no longer on. |
|
Mon Apr 19 09:17:51 2021, Jordan, Update, VAC, Empty N2 Tanks
|
When I came into the lab this morning, I noticed that both N2 tanks were empty. I had swapped one on Friday (4-16-21) before I left the lab. Looking
at the logs, the right tank (T2) sprung a leak shortly shortly after install. I leak checked the tank coupling after install but did not see a leak. There
could a leak further down the line, possibly at the pressure transducer. |
|
Wed Apr 21 12:56:00 2021, Jordan, Update, VAC, Empty N2 Tanks
|
Installed T2 today, and leaked checked the entire line. No issues found. It could have been a bad valve on the tank itself. Monitored T2 pressure for
~2 hours to see if there was any change. All seems ok.
Quote:
|
|
Wed Apr 28 11:31:40 2021, Jon, Update, VAC, Empty N2 Tanks
|
I checked out what happened on c1vac. There are actually two independent monitoring codes running:
The interlock service, which monitors the line directly connected to the valves.
A seaparate convenience mailer, running
as a cronjob, that monitors the tanks.
The interlocks did not trip because the low-pressure delivery line, downstream of the dual-tank |
|
Wed Sep 1 14:16:21 2021, Jordan, Update, VAC, Empty N2 Tanks
|
The right N2 tank had a bad/loose valve and did not fully open. This morning the left tank was just about empty and the right tank showed 2000+ psi on
the gauge. Once the changeover happened the copper line emptied but the valve to the N2 tank was not fully opened. I noticed the gauges were both reading
zero at ~1pm just before the meeting. I swapped the left tank, but not in time. The vacuum interlocks tripped at 1:04 pm today when the N2 pressure to |
|
Wed Sep 8 18:00:01 2021, Koji, Update, VAC, cronjobs & N2 pressure alert
|
In the weekly meeting, Jordan pointed out that we didn't receive the alert for the low N2 pressure.
To check the situation, I went around the machines and summarized the cronjob situation.
[40m wiki: cronjob summary] |
|
Thu Oct 14 18:30:23 2021, Koji, Summary, VAC, Flange/Cable Stand Configuration
|
Flange Configuration for BHD
We will need total 5 new cable stands. So Qty.6 is the number to be ordered.
Looking at the accuglass drawing, the in-vaccum cables are standard dsub 25pin cables only with two standard fixing threads. |
|
Mon Oct 18 10:02:17 2021, Koji, Update, VAC, Vent Started / Completed
|
[Chub, Jordan, Anchal, Koji]
- Checked the main volume is isolated.
- TP1 and TP2 were made isolated from other volumes. Stopped TP1. Closed V4 to isolate TP1 from TP2. |
|
Tue Oct 19 10:59:09 2021, Koji, Update, VAC, Vent Started / Completed
|
[Chub, Jordan, Yehonathan, Anchal, Koji]
North door of the BS chamber opened
|
|
Tue Oct 19 11:30:39 2021, Koji, Update, VAC, How to vent TP1
|
I learned that TP1 was vented through the RGA room in the past. This can be done by opening VM2 and a manual valve ("needle valve")
I checked the setup and realized that this will vent RGA. But it is OK as long as we turns of the RGA during vent and bake it once
TP1 is back. |
|
Wed Oct 20 15:58:27 2021, Koji, Update, VAC, How to vent TP1
|
Probably the hard disk of c0rga is dead. I'll follow up in this elog later today.
Looking at the log in /opt/rtcds/caltech/c1/scripts/RGA/logs , it seemed that the last RGA scan was Sept 2, 2021, the day when we
had the disk full issue of chiara. |
|
Mon Dec 6 14:26:52 2021, Koji, Update, VAC, Pumping down the RGA section
|
Jordan reported that the RGA section needs to be pumped down to allow the analyzer to run at sufficiently low pressure (P<1e-4 torr).
The RGA section was pumped down with the TP2/TP3. The procedure is as listed below.
If the pressure go up to P>1e-4 torr, we need to keep the pump running until the scan is ready. |
|
Tue Dec 7 13:12:50 2021, Koji, Update, VAC, Pumping down the RGA section
|
So that Jordan can run the RGA scan this afternoon, I ran TP3 and started pumping down the RGA section.
Procedure:
- Same 1~4 |
|
Wed Dec 8 10:14:43 2021, Jordan, Update, VAC, Pumping down the RGA section
|
After an overnight pumpdown/RGA warm up, I took a 100 amu scan of the RGA volume and subsequent pumping line. Attached is a screenshot along with the
.txt file. Given the high argon peak (40) and the N2/O2 ratio, it looks like there is a decent sized air leak somehwere in the volume.
Are we interested in the hydrocarbon leak rates of this volume? That will require another scan with one of the calibrated leaks opened. |
|
Fri Dec 10 19:22:01 2021, Koji, Update, VAC, Pumping down the RGA section
|
The scan result was ~x10 higher than the previously reported scan on 2020/9/15 (https://nodus.ligo.caltech.edu:8081/40m/15570),
which was sort of high from the reference taken on 2018/7/18.
This just could mean that the vacuum level at the RGA was x10 high. |
|
Wed Dec 15 15:06:08 2021, Jordan, Update, VAC, Vacuum Feedthru Install
|
Jordan, Chub
We installed the 4x DB25 feedthru flange on the North-West port of ITMX chamber this afternoon. It is ready to go. |
|
Tue Dec 21 16:35:39 2021, Koji, Update, VAC, ITMX NW feedthru (LO1-1) connector pin bent
|
I've received a report that a pin of an ITMX NW feedthru connector was bent. (Attachment 1)
The connector is #1 (upper left) and planned to be used for LO1-1.
This is Pin25 and used for the PD K of OSEM #1. This means that Coil Driver #1 (3 OSEMs) uses this pin, but Coil Driver #2 (2 OSEMs) does not. |
|
Mon Jan 31 10:39:19 2022, Jordan, Update, VAC, TP1 and Manual Gate Valve Removal
|
Jordan, Chub
Today, Chub and I removed TP1 and the failed manual gate valve off of the pumping spool.
First, P2 needed to be vented in order to remove TP1. TP1 has a purge valve on the side of the pump which we slowly opened bringing the P2 volume |
|
Thu Feb 3 10:25:59 2022, Jordan, Update, VAC, TP1 and Manual Gate Valve Install
|
Jordan, Chub
Chub and I installed the new manual gate valve (Nor-Cal GVM-6002-CF-K79) and reinstalled TP1. The new gate valave was placed with the sealing
side towards the main 40m volume, then TP1 was installed on top and the foreline reattched to TP1. |
|
Sat Feb 26 01:01:40 2022, Tega, Update, VAC, Ongoing work to get the FRG gauges readouts to EPICs channels
|
I will make a detailed elog later today giving a detailed outline of the connection from the Agilent gauge controller to the vacuum subnet and the
work I have been doing over the past two days to get data from the unit to EPICs channels. I just want to mention that I have plugged the XGS-600 gauge
controller into the serial server on the vacuum subnet. I check the vacuum medm screen and I can confirm that the other sensors did not experience and |
|
Sat Feb 26 15:45:14 2022, Tega, Update, VAC, Ongoing work to get the FRG gauges readouts to EPICs channels
|
I have attached a flow diagram of my understanding of how the gauges are connected to the network.
Earlier today, I connected the XGS-600 gauge controller to the IOLAN Serial Device Server on port 192.168.114.22 .
The plan is a follows: |
|
Mon Feb 28 19:15:10 2022, Tega, Update, VAC, Ongoing work to get the FRG gauges readouts to EPICs channels
|
I decided to create an independent service for the XGS data readout so we can get this to work first before trying to integrate into current system.
After starting the service, I noticed that the EPICS channel were not updating as expected. So I started to debug the problem and managed to track it down
to an ip socket connect() error, i.e. we get a connection error for the ip address assigned to the LAN port to which the XGS box was connected. After trying |
|
Tue Mar 1 20:38:49 2022, Tega, Update, VAC, Ongoing work to get the FRG gauges readouts to EPICs channels
|
During my investigation, I inadvertently overwrote the serial port configuration for the connected devices. So I am now working to get it all back. I
have attached screenshots of the config settings that brought back communication that is not garbled. There is no physical connection to port 6, which
I guess was initially used for the UPS serial communication but not anymore. Also, ports 9 and 10 are connected to Hornet and SuperBee, both of which |
|
Wed Mar 2 11:50:39 2022, Tega, Update, VAC, Ongoing work to get the FRG gauges readouts to EPICs channels
|
Here is the IOLAN SDS TCP socket setting and the USBserial setting for comparison.
I have also included the python script and output from the USBserial test from earlier. |
|
Wed Mar 2 12:40:08 2022, Tega, Update, VAC, Ongoing work to get the FRG gauges readouts to EPICs channels 6x
|
Connector Test:
A quick test to rule out any issue with the Ethernet to Serial adapter was done using the setup shown in Attachment 1. The results rule out any
connector problem. |
|
Fri Mar 4 18:12:44 2022, Koji, Update, VAC, RGA pumping down
|
1. Jordan reported that the newly installed Pirani gauge for P2 shows 850Torr while PTP2 show 680 Torr. Because of this, the vacuum interlock fails when
we try to open V4.
2. Went to c1vac. Copied the interlock setting file interlock_conditions.yaml to interlock_conditions_220304.yaml |
|
Sun Mar 6 18:14:45 2022, Tega, Update, VAC, Ongoing work to get the FRG gauges readouts to EPICs channels
|
Following repeated failure to establish communication between c1vac and the XGS600 controller via Perle IOLAN serial device server, I decided to monitor the
signal voltage of the communication channels (pin#2, pin#3 and pin#5) using an oscilloscope. The result of this investigation is presented
in the attached pdf document. In summary, it seems I have used a crossed wired RS232 serial cable instead of a normal RS232 serial cable, so the c1vac |
|
Mon Mar 7 13:53:40 2022, Tega, Update, VAC, Ongoing work to get the FRG gauges readouts to EPICs channels
|
So it appears that my deduction from the pictures of needing a cable swap was correct, however, it turns out that the installed cable was actually
the normal RS232 and what we need instead is the RS232 null cable. After the swap was done, the communication between c1vac and the XGS600 controller became
active. Although, the data makes it all to the to c1vac without any issues, the scope view of it shows that it is mainly utilizing the upper half of |
|
Mon Mar 7 14:52:34 2022, Koji, Update, VAC, Ongoing work to get the FRG gauges readouts to EPICs channels
|
Great trouble shoot!
> I guess, the only remaining issue now is the incorrect atmospheric pressure reading of 1000 Torrs.
This is just a calibration issue. The controller should have the calibration function. |
|
Tue Mar 8 12:08:47 2022, Tega, Update, VAC, Ongoing work to get the FRG gauges readouts to EPICs channels
|
OMG, it worked! It was indeed a calibration issue and all I had to do was press the "OK" button after selecting the "CAL" tab beside
the pressure reading. Wow.
|
|
Wed Mar 9 10:23:28 2022, Jordan, Update, VAC, Leak Testing of New Manual Gate Valve - Attempt #1
|
Jordan, Chub, Paco
Chub and I went into the lab this morning to leak check the new gate valve after pumping over the weekend. This would be done through the RGA
and spraying helium around the newly installed flanges. The RGA is set to monitor the partial pressure of helium versus time and we visually watch for |
|
Thu Mar 10 09:39:59 2022, Jordan, Update, VAC, Leak Testing of New Manual Gate Valve - Attempt #2
|
This morning Chub and I leak checked the manual gate valve with the RGA and helium. There was no change in the helium partial pressure while spraying
helium around the flanges, all looks good.
|
|
Tue Mar 29 15:07:11 2022, Tega, Summary, VAC, Opening of MC and ETM chambers
|
[Anchal, Chub, Ian, Paco, Tega]
Today, we opened the MC chamber, ETMX chamber and ETMY chamber. |
|
Mon Apr 11 11:00:30 2022, Jancarlo, Update, VAC, C1VAC Reboot and Nitrogen tanks
|
[Paco, JC, Ian, Jordan, Chub]
Checking in the morning, I walked over to the Nitrogen tanks to check the levels. Noticed one tank was empty, so I swapped it out. Chub came
over to check the levels and to take note of how many tanks were left available for usage (None). Chub continued to put in a work order for a set of full |
|
Tue Apr 12 09:05:21 2022, Jordan, Update, VAC, New Pressure Gauge Install/Pump Spool Vent 6x
|
Today, Tega and I would like to vent the pump spool an dinstall the new FRG-400 Agilent Pressure Gauges (per elog 15703). The attached picture shows
the volume needed to be vented highlighted in red, and the gauges that need to be replaced/removed (purple dot next to the name).
The vent plan is as follows: |
|
Wed Apr 13 12:50:07 2022, Tega, Update, VAC, New Pressure Gauge Install/Pump Spool Vent
|
[Jordan, JC, Tega]
We have installed all the FRGs and updated the VAC medm screens to display their sensor readings. The replacement map is CC# -> FRG#, where
# in [1..4] and PRP1 -> FRG5. We now need to clean up the C1VAC python code so that it is not overloaded with non-function gauges (CC1,CC2,CC3,CC4,PRP1). |
|
Thu Apr 14 09:04:30 2022, Jordan, Update, VAC, RGA Volume RGA Scans
|
Prior to venting the RGA volume on Tuesday (4/12/2022) I took an RGA scan of the volume to be vented (RGA+TP1 volume+Manual Gate Valve) to see if there
was a difference after replacing the manual gate valve. Attached is the plot from 4/12/22, and an overlay plot to complare 4/12/22 to 12/10/2021, when
the same volume was scanned with the old (defective) manual gate valve. |
|
Thu Apr 21 11:31:35 2022, JC, Update, VAC, Gauges P3/P4
|
[Jordan, JC]
It was brought to attention during yesterday's meeting that the pressures in the vacuum system were not equivalent althought the valve were
open. So this morning, Jordan and I reviewed the pressure gauges P3 and P4. We attempted to recalibrate, but the gauges were unresponsive. Following this, |
|
Thu Apr 21 16:36:56 2022, Tega, Update, VAC, cleanup work for vacuum git repo
|
git repo - https://git.ligo.org/40m/vac
Finally incorporated the FRGs into the main modbusIOC service and everything seems to be working fine. I have also removed the old sensors (CC1,CC2,CC3,CC4,PTP1,IG1)
from the serial client list and their corresponding EPICS channels. Furthermore, the interlock service python script has been updated so that all occurrence |
|
Fri Apr 22 14:14:33 2022, Jordan, Update, VAC, TP3 Forepump tip seal replacement
|
Jordan, JC
While the pumpspool is vented, I thought it would be a convenient time to change out the tip seal on the TP3 forepump. This one had not been
changed since 2018, so as preventative maintence I had JC remove the pump and begin cleaning/installing the new tip seal. |
|
Fri Apr 29 08:34:40 2022, JC, Update, VAC, RGA Pump Down
|
Jordan and I, in order to start pumpig down the RGA Volume, we began by opening V7 and VM. Afterwards, we started RP1 and RP3. After this, the pressure
in the line between RP1, RP3, and V6 dropped to 3.4 mTorr. Next, we tried to open V6, although an error message popped up. We haven't been able
to erase it since. But we were able to turn on TP2 with V4 closed. The pressure in that line is reporting 1.4 mTorr. |
|
Mon May 2 10:00:34 2022, Jordan, Update, VAC, TP3 Forepump tip seal replacement
|
[JC, Jordan]
Jordan recieved the new tip seal Friday afternoon and I continued the replacement process in the morning. Finishing up, we proceeded to test
the pump in the Clean and Bake room. The pump's pressure lowered to 110 mTorr, and we continue pumping so the seal can recieve a good fitting. |
|
Tue May 3 13:18:47 2022, JC, Update, VAC, RGA Pump Down
|
Jordan, Tega, JC
Issue has been resolved. Breaker on RP1 was tripped so the RP1 button was reporting ON, but was not actually on which continuously tripped the
V6 interlock. Breaker was reset, RP1 and RP3 turned on. The V6 was opened to rough out the RGA volume. Once, pressure was at ~100mtorr, V4 was opened to |