 Thu Apr 30 07:29:01 2020, Chub, Update, VAC, N2 delivered.
|
Hi All,
The new nitrogen cylinders were delivered to the rack at the west entrance. We only get one Airgas delivery per week during the stay-at-home
order, but so far they've not let us down. |
Thu Jun 11 11:48:43 2020, gautam, Update, VAC, Vac failure 
|
There appears to have been some sort of vacuum failure.
ldas-pcdev1 was down, so the summary pages weren't being generated. I have now switched over to ldas-pcdev6. I suspect some forepump failure,
will check up later today unless someone else wants to take care of this. |
 Thu Jun 11 16:14:03 2020, gautam, Update, VAC, Vac failure - probable cause is serial comm glitch
|
Summary:
It looks like the main vacuum interlock was tripped due to a serial communication error from the TP2 controller. With Rana/Koji's
permission, I will open V1 and expose the main volume to TP1 again (#2 in last section). |
|
Thu Jun 11 17:35:34 2020, gautam, Update, VAC, Pumpspool UPS needs battery replacement
|
Summary:
The pumpspool UPS has its "Replace Battery"
indicator light on. Might be a good chance to change the UPS, but at the very least, we should put in fresh batteries (last replacement was in Aug |
|
Fri Jun 12 01:23:32 2020, Koji, Update, VAC, Pumpspool UPS needs battery replacement
|
1. I agree that it's likely that it was the temp signal glitch.
Recom #2: I approve to reopen the valves to pump down the main volume. As long as there is no frequent glitch, we can just bring the vacuum back
to normal with the current software setup. |
|
Fri Jun 12 17:32:40 2020, Koji, Update, VAC, Pumpspool UPS needs battery replacement
|
Jon and Koji remotely supported Jordan's resetting the TP2 controller.
Here is the instruction by Jon
From
the operator's console in front of the vac rack:
Open a terminal window (click the LXTerminal icon |
|
Fri Jun 12 19:02:52 2020, gautam, Update, VAC, Pumpspool UPS needs battery replacement
|
I still don't understand why restoring the vacuum is contingent on this functionality working. All the TPs have their own internal logic to shutdown
the pump if some damage threshold is exceeded. Plus, we have the pressure-sensor based interlocks to protect the main volume as well as pumps. While the
extra redundancy from the readbacks from the controller is useful, clearly it isn't the first line of defense? |
|
Fri Jun 12 19:23:56 2020, Koji, Update, VAC, Pumpspool UPS needs battery replacement
|
The vacuum safety policy and design are not clear to me, and I don't know what the first and second defense is. Since we had limited time and bandwidth
during the remotely-supported recovery work today, we wanted to work step by step.
The pressure rising rate is 20mtorr/day, and turning on TP3 early next week will resume the main-volume pumping without too much hustle. If you |
|
Fri Jun 12 19:33:31 2020, gautam, Update, VAC, Pumpspool UPS needs battery replacement
|
Didn't mean to sound whiny. I will wait until the vacuum team tells me it is okay.
Quote:
The vacuum safety policy and design are not clear to me, and I don't know what the first and second defense |
|
Tue Jun 16 13:35:03 2020, Jon, Update, VAC, Temporary vac fix / IFO usable again
|
[Jon, Jordan, Koji]
Today Jordan reconfigured the vac system to allow pumping of the main volume resume, with Jon and Koji remotely advising. All clear to
resume normal IFO activities. However, the vac system is operating in a temporary configuration that will have to be reverted as we locate replacement |
|
Wed Jun 17 16:27:51 2020, gautam, Update, VAC, Questions/comments on vacuum
|
I missed the vacuum discussion on the call today, but I have some questions/comments:
Isn’t it true that we didn’t digitally monitor any of the TP diagnostic channels before 2018 December? I don’t
have the full history but certainly there wasn’t any failure of the vacuum system connected to pump current/temp/speed from Sep 2015-Dec2018, whereas |
|
Thu Jun 18 11:00:24 2020, Jon, Update, VAC, Questions/comments on vacuum
|
Quote:
Isn’t it true that we didn’t digitally monitor any of the TP diagnostic channels
before 2018 December? I don’t have the full history but certainly there wasn’t any failure of the vacuum system connected to pump current/temp/speed |
|
Thu Jun 18 12:00:36 2020, gautam, Update, VAC, Questions/comments on vacuum
|
I agree there were MEDM fields, but I can't find any record of these channels being recorded till 2018 December, so I don't agree that they were
being digitally monitored. You can also look back in the elog (e.g. here and here)
and see that the display fields are just blank. I would then assume that no interlocks were dependent on these channels, because otherwise the vacuum interlocks |
|
Thu Jun 18 14:13:03 2020, Jon, Update, VAC, Questions/comments on vacuum
|
I agree there were MEDM fields, but I can't find any record of these channels
being recorded till 2018 December, so I don't agree that they were being digitally monitored. You can also look back in the elog (e.g. here
and here) and see that the display fields are just blank. I |
|
Thu Jun 18 15:25:08 2020, Jordan, Update, VAC, TP2 and TP3 Forepump removal
|
I removed the backing pumps for TP2 and TP3 today to test ultimate pressure and determine if they need a tip seal replacement. This was done with Jon
backing me on Zoom. We closed off TP3 and powered down TP3 and the auxilliary pump, in order to remove the forepumps from the exhaust line.
Close V1
Close V5
Turn off TP3
Turn off aux dry pump (manually)
Once the PTP3 foreline pressure |
|
Thu Jun 18 15:46:34 2020, gautam, Update, VAC, Questions/comments on vacuum
|
So why not just have a special mode for the interlock code during pumpdown and venting, and during normal operation we expect the main volume pressure
to be <100uTorr so the interlock trips if this condition is violated? These can just be EPICS buttons on the Vac control MEDM screen. Both of these
procedures are not "business as usual", and even if we script them in the future, it's likely to have some operator supervising, so I don't |
|
Thu Jun 18 16:56:34 2020, Jordan, Update, VAC, TP2 and TP3 Forepump removal
|
Quote:
I removed the backing pumps for TP2 and TP3 today to test ultimate pressure and determine if they need a tip seal
replacement. This was done with Jon backing me on Zoom. We closed off TP3 and powered down TP3 and the auxilliary pump, in order to remove the forepumps |
|
Thu Jun 18 22:33:57 2020, Jon, Omnistructure, VAC, Vac hardware purchase list
|
Replacement Hardware Purchase List
I've created a purchase
list of hardware needed to restore the aging vacuum system. This wasn't planned as part of the BHD upgrade, but I've added it to
the BHD procurement list since hardware replacements have become necessary. |
|
Fri Jun 19 07:40:49 2020, Jon, Update, VAC, Questions/comments on vacuum
|
I think we should discuss interlock possibilities at a 40m meeting. I'm reluctant to make the system more complicated, but perhaps we can
find ways to reduce the reliance on the turbo pump readbacks. I agree they've proven to be the least reliable.
While we may be able to improve the tolerance to certain kinds of hardware malfunctions (and if so, we should), I don't see interlocks triggering |
|
Fri Jun 19 09:57:35 2020, gautam, Update, VAC, Questions/comments on vacuum
|
For this particular email service, ideally the email should be sent out as soon as the interlock is tripped, so this would require a line of code to
be added to the main interlock code. Which I guess would require a restart of the interlock service. So let me know when you guys plan to do the dry-pump
tip seal replacement operation (when I presume valves will be closed anyways) so that we can do this in a minimally invasive way. |
|
Fri Jun 19 14:03:50 2020, Jordan, Update, VAC, Forepump Tip Seal Replacement
|
Tip Seals were replaced on the forepumps for TP2 and TP3, and both are ready to be installed back onto the forelines.
TP2 Forepump Ultimate Pressure: 180 mtorr
TP3 Forepump Ultimate Pressure: 120 mtorr |
|
Mon Jun 22 10:43:25 2020, Jon, Configuration, VAC, Vac maintenance at 11 am
|
The vac system is going down at 11 am today for planned maintenance:
Re-install the repaired TP2 and TP3 dry pumps [ELOG 15417]
Incorporate
an auto-mailer and flag channel into the controls code for signaling tripped interlocks [ELOG 15413]
We |
|
Mon Jun 22 20:06:06 2020, Jon, Configuration, VAC, Vac maintenance complete
|
This work is finally complete. The dry pump replacement was finished quickly but the controls updates required some substantial debugging.
For one, the mailer code I had been given to install would not run against Python 3.4 on c1vac, the version run by the vac controls since about
a year ago. There were some missing dependencies that proved difficult to install (related to Debian Jessie becoming unsupported). I ultimately solved |
|
Tue Jun 23 17:54:56 2020, rana, Configuration, VAC, Vac maintenance complete
|
I propose we go for all CAPS for all channel names. The lower case names is just a holdover from Steve/Alan from the 90's. All other systems are
all CAPS.
It avoids us having to force them all to UPPER in the scripts and channel lists. |
|
Wed Jul 1 18:03:04 2020, Jon, Configuration, VAC, UPS replacements
|
I looked into how the new UPS devices suggested by Chub would communicate with the vac interlocks. There are several possible ways, listed in
order of preference:
Python interlock service directly queries the UPS via a USB link using the (unofficial) tripplite |
|
Thu Jul 9 18:00:35 2020, Jon, Configuration, VAC, UPS replacements
|
Chub has placed the order for two new UPS units (115V for TP2/3 and a 220V version for TP1).
Tripp Lite SMX1000RT2U: https://www.grainger.com/product/TRIPP-LITE-UPS-System-36HW20
Tripp
Lite Smart1000LCD: https://www.grainger.com/product/TRIPP-LITE-Smart-UPS-1AYA6
They |
|
Thu Jul 23 15:58:24 2020, Jon, Summary, VAC, Vacuum controls refurbishment plan
|
This year we've struggled with vacuum controls unreliability (e.g., spurious interlock triggers) caused by decaying hardware. Here are details
of the vacuum refurbishment plan I described on the 40m call this week.
☑ Refurbish TP2 and TP3 dry pumps. Completed [ELOG |
|
Fri Jul 24 15:40:59 2020, Jordan, Update, VAC, Installation of two new UPS units
|
I installed the Tripp Lite SMX1000RT2U and Tripp Lite Smart1000LCD at the
bottom of the 1x8 electronics rack. These are plugged in to power, and are ready for testing. All other cables (serial, usb, etc.) have been left on the
table next to the 1x8 rack. |
|
Mon Jul 27 15:48:36 2020, Jon, Summary, VAC, Vacuum parts ordered
|
To carry out the next steps of the vac refurbishment plan [ELOG 15499], I've ordered
parts necessary for interfacing the UPS units and the analog TP2/3 controller outputs with c1vac. The purchase list is appended to the main
BHD list and is located here. Some parts |
|
Tue Jul 28 12:22:40 2020, Jon, Update, VAC, Vac interlock test today 1:30 pm
|
This afternoon Jordan is going to carry out a test of the V4 and V5 hardware interlocks. To inform the interlock improvement plan [15499],
we need to characterize exactly how these work (they pre-date the 2018 upgrade). I have provided him a sequence of steps for each test and will also
be backing him up on Zoom. |
|
Tue Jul 28 14:11:14 2020, Jon, Update, VAC, Vac interlock test today 1:30 pm
|
This test has been completed. The IFO configuration has been reverted to nominal.
For future reference: yes, both the V4 and V5 hardware interlocks were found to still be connected and work.
A TTL signal from the analog output port of each pump controller (TP2 and TP3) is connected to an auxiliary relay inside the main valve relay |
|
Fri Aug 14 10:10:56 2020, Jon, Configuration, VAC, Vacuum repairs today
|
The vac system is going down now for planned repairs [ELOG 15499]. It will likely take
most of the day. Will advise when it's back up. |
|
Sat Aug 15 02:02:13 2020, Jon, Configuration, VAC, Vacuum repairs today
|
Vacuum work is completed. The TP2 and TP3 interlocks have been overhauled as proposed in ELOG
15499 and seem to be performing reliably. We're now back in the nominal system state, with TP2 again backing for TP1 and TP3 pumping
the annuli. I'll post the full implementation details in the morning. |
|
Sat Aug 15 15:12:22 2020, Jon, Configuration, VAC, Overhaul of small turbo pump interlocks
|
Summary
Yesterday I completed the switchover of small turbo pump interlocks as proposed in ELOG
15499. This overhaul altogether eliminates the dependency on RS232 readbacks, which had become unreliable (glitchy) in both controllers.
In their place, the V4(5) valve-close interlocks are now predicated on an analog controller output whose voltage goes high when the rotation |
|
Mon Aug 24 08:13:56 2020, Jon, Update, VAC, UPS installation
|
I'm in the lab this morning to interface the two new UPS units with the digital controls system. Will be out by lunchtime. The disruptions to the
vac system should be very brief this time. |
|
Mon Aug 24 11:25:07 2020, Jon, Update, VAC, UPS installation
|
I'm leaving the lab shortly. We're not ready to switch over the vac equipment to the new UPS units yet.
The 120V UPS is now running and interfaced to c1vac via a USB cable. The unofficial tripplite
python package is able to detect and connect to the unit, but then read queries fail with "OS Error: No data received." The firmware has |
|
Wed Aug 26 15:48:31 2020, gautam, Update, VAC, Control screen left open on vacuum workstation
|
I found that the control MEDM screen was left open on the c1vac workstation. This should be closed every time you leave the workstation, to avoid
accidental button pressing and such.
The network outage meant that the EPICS data from the pressure gauges wasn't recorded until I reset everything ~noon. So there isn't |
|
Fri Sep 4 15:26:55 2020, Jon, Update, VAC, Vac system UPS installation
|
| The vac controls are going down now to pull and test software changes. Will advise when the work is completed. |
|
Fri Sep 4 21:12:51 2020, Jon, Update, VAC, Vac system UPS installation
|
The vac work is completed. All of the vacuum equipment is now running on the new 120V UPS, except for TP1. The 230V TP1 is still running off wall power,
as it always has. After talking with Tripp Lite support today, I believe there is a problem with the 230V UPS. I will post a more detailed note in the
morning. |
|
Sat Sep 5 12:01:10 2020, Jon, Update, VAC, Vac system UPS installation
|
Summary
Yesterday's UPS switchover was mostly a success. The new Tripp
Lite 120V UPS is fully installed and is communicating with the slow controls system. The interlocks are configured to trigger a
controlled shutdown upon an extended power outage (> ~30 s), and they have been tested. All of the 120V pumpspool equipment (the full c1vac/LAN/Acromag |
|
Wed Sep 16 12:03:07 2020, Jon, Update, VAC, Replacing pressure gauges
|
Assembled is the list of dead pressure gauges. Their locations are also circled in Attachment 1.
Gauge
Type
Location
CC1
Cold cathode
Main volume
CC3
Cold |
|
Sat Sep 19 18:07:35 2020, Koji, Update, VAC, TP3 RP failure
|
I came to the campus and Gautam notified that he just had received the alert from the vac watchdog.
I checked the vac status at c1vac. PTP3 went up to 10 torr-ish and this made the diff pressure for TP3 over 1torr. Then the watchdog kicked in.
To check the TP3 functionality, AUX RP was turned on and the manual valve (MV in the figure) was opened to pump the foreline of TP3. This easily |
|
Sat Sep 19 19:37:16 2020, not Koji, Update, VAC, TP3 RP failure
|
Disconcerting because those tip seals were just replaced [15417]. Maybe they were just defective,
but if there is a more serious problem with the pump, there is a spare Varian roughing pump (the old TP2 dry pump) sitting at the X-end.
I reset the interlock error to unfreeze the vac controls (leaving V5 closed). |
|
Mon Sep 21 15:57:08 2020, Jordan, Update, VAC, TP3 Forepump Replacement and Vac reset
|
I removed the forepump (Varian SH-110) for TP3 today to see why it had failed over the weekend. I tested it in the C&B lab and the ultimate pressure
was only ~40torr. I checked the tip seals and they were destroyed. The scroll housing also easily pulled off of the motor drive shaft, which is indicative
of bad bearings. The excess travel in the bearings likely led to significant increase in tip seal wear. This pump will need to be scrapped, or rebuilt. |
|
Wed Sep 23 08:57:18 2020, gautam, Update, VAC, TP2 running HOT
|
The interlocks tripped at ~630am local time. Jordan reported that TP2 was supposedly running at 52 C (!).
V1 was already closed, but TP2 was still running. With him standing by the rack, I remotely exectued the following sequence:
VM1 closed (isolates RGA volume).
VA6 closed (isolates annuli from being pumped).
V7 opened (TP3 now backs TP1, |
|
Wed Sep 23 10:06:52 2020, Koji, Update, VAC, TP2 running HOT
|
Here is the timeline. This suggests TP2 backing RP failure.
1st line: TP2 foreline pressure went up. Accordingly TP2 P, current, voltage, and temp went up. TP2 rotation went down.
2nd line: TP2 temp triggered the interlock. TP2 foreline pressure was still high (10torr) so TP2 struggled and was running at 1 torr. |
|
Wed Sep 23 15:06:54 2020, Jordan, Update, VAC, TP2 Forepump Re-install
|
I removed the forepump to TP2 this morning after the vacuum failure, and tested in the C&B lab. I pumped down on a small volume 10 times, with no
issue. The ultimate pressure was ~30 mtorr.
I re-installed the forepump in the afternoon, and restarted TP2, leaving V4 closed. This will run overnight to test, while TP3 backs TP1. |
|
Tue Oct 6 14:35:16 2020, Jordan, Update, VAC, Spare forepumps
|
| I have placed 3 new in box, IDP 7 forepumps along the x arm of the interferometer. These are to be used as spares for both the 40m and Clean and Bake. |
|
Tue Nov 10 11:59:37 2020, gautam, Update, VAC, Stuck RV2
|
I've uploaded some more photos here. I believe the problem is a worn out thread where the
main rotary handle attaches to the shaft that operates the valve.
This morning, I changed the valve config such that TP2 backs TP1 and that combo continues to pump on the main volume through the partially open |
|
Wed Nov 18 17:51:50 2020, gautam, Update, VAC, Agilent pressure gauge controller delivered
|
| It is stored along with the cables that arrived a few weeks ago, awaiting the gauges which are now expected next week sometime. |