40m QIL Cryo_Lab CTN SUS_Lab TCS_Lab OMC_Lab CRIME_Lab FEA ENG_Labs OptContFac Mariner WBEEShop
  40m Log, Page 327 of 337  Not logged in ELOG logo
ID Date Author Type Categoryup Subject
  16404   Thu Oct 14 18:30:23 2021 KojiSummaryVACFlange/Cable Stand Configuration

Flange Configuration for BHD

We will need total 5 new cable stands. So Qty.6 is the number to be ordered.


Looking at the accuglass drawing, the in-vaccum cables are standard dsub 25pin cables only with two standard fixing threads.

https://www.accuglassproducts.com/sites/default/files/PDF/Partpdf/110070_3.pdf

For SOSs, the standard 40m style cable bracket works fine. https://dcc.ligo.org/D010194-x0

However, for the OMCs, we need to make the thread holes available so that we can mate DB25 male cables to these cables.
One possibility is to improvise this cable bracket to suspend the cables using clean Cu wires or something. I think we can deal with this issue in situ.


Ha! The male side has the 4-40 standoff (jack) screws. So we can hold the male side on the bracket using the standoff (jack) screws and plug the female cables. OK! The issue solved!

https://www.accuglassproducts.com/sites/default/files/PDF/Partpdf/110029_3.pdf

  16410   Mon Oct 18 10:02:17 2021 KojiUpdateVACVent Started / Completed

[Chub, Jordan, Anchal, Koji]

- Checked the main volume is isolated.
- TP1 and TP2 were made isolated from other volumes. Stopped TP1. Closed V4 to isolate TP1 from TP2.
- TP3 was made isolated. TP3 was stopped.
- We wanted to vent annuli, but it was not allowed as VA6 was open. We closed VA6 and vented the annuli with VAVEE.
- We wanted to vent the volume between VA6, V5, VM3, V7 together with TP1. So V7 was opened. This did not change the TP1 pressure (P2 = 1.7mmTorr) .
- We wanted to connect the TP1 volume with the main volume. But this was not allowed as TP1 was not rotating. We will vent TP1 through TP2 once the vent of the main volume is done.

- Satrted venting the main volume@Oct 18, 2021 9:45AM PDT

- We started from 10mTorr/min, and increased the vent speed to 200mTorr/min, 700mTorr/min, and now it is 1Torr/min @ 20Torr
- 280Torr @11:50AM
- 1atm  @~2PM


We wanted to vent TP1. We rerun the TP2 and tried to slowly introduce the air via TP2. But the interlock prevents the action.

Right now the magenta volume in the attachment is still ~1mTorr. Do we want to open the gate valves manually? Or stop the interlock process so that we can bypass it?

  16412   Tue Oct 19 10:59:09 2021 KojiUpdateVACVent Started / Completed

[Chub, Jordan, Yehonathan, Anchal, Koji]

North door of the BS chamber opened

 

  16413   Tue Oct 19 11:30:39 2021 KojiUpdateVACHow to vent TP1

I learned that TP1 was vented through the RGA room in the past. This can be done by opening VM2 and a manual valve ("needle valve")
I checked the setup and realized that this will vent RGA. But it is OK as long as we turns of the RGA during vent and bake it once TP1 is back.

Additional note:

- It'd be nice to take a scan for the current background level before the work.
- Turn RGA EM and filament off, let it cool down overnight. 
- Vent with clean N2 or clean air. (Normal operating temp ~80C is to minimize accumulation of H-C contaminations.)
- There is a manual switch and indicators on the top of the RGA amp. It has auto protection to turn filament off if the pressure increase over ~1e-5.

  16418   Wed Oct 20 15:58:27 2021 KojiUpdateVACHow to vent TP1

Probably the hard disk of c0rga is dead. I'll follow up in this elog later today.

Looking at the log in /opt/rtcds/caltech/c1/scripts/RGA/logs , it seemed that the last RGA scan was Sept 2, 2021, the day when we had the disk full issue of chiara.
I could not login to c0rga from control machines.
I was not aware of the presence for c0rga until today, but I could locate it in the X arm.
The machine was not responding and it was rebooted, but could not restart. It made some knocking sound. I am afraid that the HDD failed.

I think we can
- prepare a replacement linux machine for the python scripts
or
- integrate it with c1vac

  16490   Mon Dec 6 14:26:52 2021 KojiUpdateVACPumping down the RGA section

Jordan reported that the RGA section needs to be pumped down to allow the analyzer to run at sufficiently low pressure (P<1e-4 torr).
The RGA section was pumped down with the TP2/TP3. The procedure is as listed below.
If the pressure go up to P>1e-4 torr, we need to keep the pump running until the scan is ready.

----
### Monitor / Control screen setup ###
1. On c1vac: cd /cvs/cds/caltech/target/c1vac/medm
2. medm -x C0VAC_MONITOR.adl&
3. RGA section (P4) 3.6e-1 torr / P3/P2 still atm.
4. medm -x C0VAC_CONTROL.adl

### TP2/TP3 backing ###
5. Turn on AUX RP with the circuit breaker hanging on the AC.
6. Open manual valve for TP2/3/ backing / PTP2/3 ~ 8torr

### TP2/TP3 starting ###
7. Turned on TP2/TP3 with the Standby OFF

### Pump down the pump spool ###
8. Connect manual RP line (Quick Connect)
9. Turned on RP1/RP3 -> quickly reached 0.4 torr
10. Open V6 for pump spool pumping -> Immediately go down to sufficiently low pressure for TP2/TP3.
(10.5 I had to close V6 at this point)
11. Open V5 to start pumping pump spool with TP3 (TP2 still stand by) -> P3 immediately goes down below 1e-4 torr. This automatically closed V6 because of the low pressure of P3 (interlocking)

### Pump down the RGA section ###
12. Open VM3 to pump down RGA section -> P4 goes down to <1e-4 torr
13. P2 is still 2e-3. So decided to open V4 to use TP2 (now it's ready) too. -> Saturated at 1.7e-3

### Shutting down ###
14. Close VM3
15. Close V4/V5 to isolate TP2/TP3
16. Stop TP2/TP3 -> Slowing down
17. Stop RP1/RP3
18. Close the manual valves for TP2/3/ backing
19. Stop AUX RP with the circuit breaker hanging on the AC.

  16493   Tue Dec 7 13:12:50 2021 KojiUpdateVACPumping down the RGA section

So that Jordan can run the RGA scan this afternoon, I ran TP3 and started pumping down the RGA section.

Procedure:
- Same 1~4
- Same 5
- 6 Opened only the backing path for TP3
- 7 Turned on TP3 only

- TP3 reached the nominal full speed @75kRPM

- 11 Opened V5 to pump the pump spool -> Immediately reached P3<1e-4
- 12 Opened VM3 to pump the RGA section -> Immediately reached P4<1e-4

The pumps are kept running. I'll come back later to shut down the pumps.
=> Jordan wants to heat the filament (?) and to run the scan tomorrow.
So we decided to keep TP3 running overnight. I switched TP3 to the stand-by mode (= lower rotation speed @50kRPM)

 

  16494   Wed Dec 8 10:14:43 2021 JordanUpdateVACPumping down the RGA section

After an overnight pumpdown/RGA warm up, I took a 100 amu scan of the RGA volume and subsequent pumping line. Attached is a screenshot along with the .txt file. Given the high argon peak (40) and the N2/O2 ratio, it looks like there is a decent sized air leak somehwere in the volume.

Are we interested in the hydrocarbon leak rates of this volume? That will require another scan with one of the calibrated leaks opened.

Edit: Added a Torr v AMU plot to see the partial pressures

Quote:

So that Jordan can run the RGA scan this afternoon, I ran TP3 and started pumping down the RGA section.

Procedure:
- Same 1~4
- Same 5
- 6 Opened only the backing path for TP3
- 7 Turned on TP3 only

- TP3 reached the nominal full speed @75kRPM

- 11 Opened V5 to pump the pump spool -> Immediately reached P3<1e-4
- 12 Opened VM3 to pump the RGA section -> Immediately reached P4<1e-4

The pumps are kept running. I'll come back later to shut down the pumps.
=> Jordan wants to heat the filament (?) and to run the scan tomorrow.
So we decided to keep TP3 running overnight. I switched TP3 to the stand-by mode (= lower rotation speed @50kRPM)

 

 

  16501   Fri Dec 10 19:22:01 2021 KojiUpdateVACPumping down the RGA section

The scan result was ~x10 higher than the previously reported scan on 2020/9/15 (https://nodus.ligo.caltech.edu:8081/40m/15570), which was sort of high from the reference taken on 2018/7/18.

This just could mean that the vacuum level at the RGA was x10 high.
We'll just go ahead with the vacuum repair and come back to the RGA once we return to "vacuum normal".

Meanwhile, I asked Jordan to turn off the RGA to make it cool down. I shut off RGA section and turned TP2 off.

  16508   Wed Dec 15 15:06:08 2021 JordanUpdateVACVacuum Feedthru Install

Jordan, Chub

We installed the 4x DB25 feedthru flange on the North-West port of ITMX chamber this afternoon. It is ready to go.

  16529   Tue Dec 21 16:35:39 2021 KojiUpdateVACITMX NW feedthru (LO1-1) connector pin bent

I've received a report that a pin of an ITMX NW feedthru connector was bent. (Attachment 1)
The connector is #1 (upper left) and planned to be used for LO1-1.

This is Pin25 and used for the PD K of OSEM #1. This means that Coil Driver #1 (3 OSEMs) uses this pin, but Coil Driver #2 (2 OSEMs) does not.

Anyways, I tried to fix it by bending it back. WIth some tools, it was straightened enough for plugging the cable connector. (Attachment 2)

It seemed that the pins were exceptionally soft compared to the ones used for usual DSUBs, probably because of the vacuum compatibility.
So it's better to approach the pins in parallel to the surface and not apply mating pressure until you are sure that all the 25pins are inserted in the counterpart holes.

  16634   Mon Jan 31 10:39:19 2022 JordanUpdateVACTP1 and Manual Gate Valve Removal

Jordan, Chub

Today, Chub and I removed TP1 and the failed manual gate valve off of the pumping spool.

First, P2 needed to be vented in order to remove TP1. TP1 has a purge valve on the side of the pump which we slowly opened bringing the P2 volume up to atmosphere. Although, this was not vented using the dry air/N2, using this purge valve eliminated the need to vent the RGA volume.

Then we disconnected TP1 foreline, removed TP1+8" flange reducer, then the gate valve. All of the removed hardware looked good, so no need to replace bolts/nuts, only needs new gaskets. TP1 and the failed valve are sitting on a cart wrapped in foil next to the pumping station.

  16643   Thu Feb 3 10:25:59 2022 JordanUpdateVACTP1 and Manual Gate Valve Install

Jordan, Chub

Chub and I installed the new manual gate valve (Nor-Cal GVM-6002-CF-K79) and reinstalled TP1. The new gate valave was placed with the sealing side towards the main 40m volume, then TP1 was installed on top and the foreline reattched to TP1.

This valve has a hard stop in the actuator to prevent over torquing.

 

  16682   Sat Feb 26 01:01:40 2022 TegaUpdateVACOngoing work to get the FRG gauges readouts to EPICs channels

I will make a detailed elog later today giving a detailed outline of the connection from the Agilent gauge controller to the vacuum subnet and the work I have been doing over the past two days to get data from the unit to EPICs channels. I just want to mention that I have plugged the XGS-600 gauge controller into the serial server on the vacuum subnet. I check the vacuum medm screen and I can confirm that the other sensors did not experience and issues are a result of this. I also currently have two of the FRG-700 connected to the controller but I have powered the unit down after the checks.

  16683   Sat Feb 26 15:45:14 2022 TegaUpdateVACOngoing work to get the FRG gauges readouts to EPICs channels

I have attached a flow diagram of my understanding of how the gauges are connected to the network.

Earlier today, I connected the XGS-600 gauge controller to the IOLAN Serial Device Server on port 192.168.114.22 .

The plan is a follows:

1. Update the serial device yaml file to include this new ip entry for the XGS-600 gauge controller

2. Create a serial gauge class "serial_gauge_xgs.py" for the XGS-600 gauge controller that inherits from the serial gauge parent class for EPICS communication with a serial device via TCP sockets.

  • Might be better to use the current channels of the devices that are being replaced initially, i.e.
  • C1:Vac-FRG1_pressure C1:Vac-CC1_pressure
    C1:Vac-FRG2_pressure C1:Vac-CCMC_pressure
    C1:Vac-FRG3_pressure C1:Vac-PTP1_pressure
    C1:Vac-FRG4_pressure C1:Vac-CC4_pressure
    C1:Vac-FRG5_pressure C1:Vac-IG1_pressure

3. Modify the launcher file to include the XGS gauge controller. Following the same pattern used  to start the service for the other serial gauges, we can start the communication between the XGS-600 gauge controller and the IOLAN serial server and write data to EPICS channels using

controls@c1vac> python launcher.py XGS600

If we are able to establish communication between the XGS-600 gauge controller and write it gause data to EPICS channels, go on to steps 4.

4. Create a serial service file "serial_XGS600.service" and place it in the service folder

5. Add the new EPICS channels to the database file

6. Add the "serial_XGS600.service" to line 10 and 11 of modbusIOC.service

7. Later on, when we are ready, we can restart the updated modbusIOC service

 

For vacuum signal flow and Acromag channel assignments see [1]  and [2] respectively. For the 16 port IOLAN SDS (Serial Device Server) ethernet connections, see [3]. 

[1] https://wiki-40m.ligo.caltech.edu/Vacuum-Upgrade-2018?action=AttachFile&do=view&target=40m_Vacuum_System_Signal_Flow.pdf

[2] https://wiki-40m.ligo.caltech.edu/Vacuum-Upgrade-2018?action=AttachFile&do=view&target=AcromagChannelAssignment.pdf

[3] https://git.ligo.org/40m/vac/-/blob/master/python/serial/serial_devices.yaml

  16688   Mon Feb 28 19:15:10 2022 TegaUpdateVACOngoing work to get the FRG gauges readouts to EPICs channels

I decided to create an independent service for the XGS data readout so we can get this to work first before trying to integrate into current system. After starting the service, I noticed that the EPICS channel were not updating as expected. So I started to debug the problem and managed to track it down to an ip socket connect() error, i.e. we get a connection error for the ip address assigned to the LAN port to which the XGS box was connected. After trying a few things and searching the internet, I think the error indicates that this particular LAN port is not yet configured. I reached this conclusion after noting that only a select number of LAN ports connected without issues and these are the ports that already had devices connected. So it must be the case that the LAN ports were somehow configured. The next step is to look at the IOLAN manual to figure out how to configure the ip port for the XGS controller. Fingers crossed.

  16691   Tue Mar 1 20:38:49 2022 TegaUpdateVACOngoing work to get the FRG gauges readouts to EPICs channels

During my investigation, I inadvertently overwrote the serial port configuration for the connected devices. So I am now working to get it all back. I have attached screenshots of the config settings that brought back communication that is not garbled. There is no physical connection to port 6, which I guess was initially used for the UPS serial communication but not anymore. Also, ports 9 and 10 are connected to Hornet and SuperBee, both of which have not been communicating for a while and are to be replaced, so there is no way to confirm communication with them. Otherwise, the remaining devices seem to be communicating as before.

I still could not establish communication with the XGS-600 controller using the serial port settings given in the manual, which also happen to work via Serial to USB adapter, so I will revisit the problem later. My immediate plan is to do a Serial Ethernet, then Ethernet to Serial, and then Serial to USB connection to see if the USB code still works. If it does then at least I know the problem is not coming from the Serial to Ethernet adapters. Then I guess I will replace the controller with my laptop and see what signal comes through when I send a message to the controller via the IOLAN serial device server. Hopefully, I can discover what's wrong by this point.

 

Note to self: Before doing anything, do a sanity check by comparing the settings on the IOLAN SDS and the config settings that worked for the Serial to USB communication and post an elog for this for reference.

  16692   Wed Mar 2 11:50:39 2022 TegaUpdateVACOngoing work to get the FRG gauges readouts to EPICs channels

Here is the IOLAN SDS TCP socket setting and the USBserial setting for comparison.

I have also included the python script and output from the USBserial test from earlier.

  16693   Wed Mar 2 12:40:08 2022 TegaUpdateVACOngoing work to get the FRG gauges readouts to EPICs channels

Connector Test:

A quick test to rule out any issue with the Ethernet to Serial adapter was done using the setup shown in Attachment 1. The results rule out any connector problem.

 

IOLAN COMM test (as per Koji's suggestion):

The next step is to swap the controller with a laptop set up to receive serial commands using the same settings as the XGS600 controller. Basically, run a slightly modified version of python script where we go into listening mode. Then send commands to the TCP socket on the IOLAN SDS unit using c1vac and check what data makes its way to the laptop USBserial terminal. After working on this for a bit, I realized that we do not need to do anything on the c1vac machine. We only need to start the service as it would work normally. So I wrote a small python code for a basic XGS-600 controller emulator, see Attachment 4. The outputs from the laptop and c1vac terminals are Attachments 5 and 6 respectively. 

These results show that we can communicate via the assigned IP address "192.168.114.22" and the commands that are sent from c1vac reaches the laptop in the correct format. Furthermore, the serial_XSG service, a part modbusIOC_XGS service, which usually exits with an error seems fine now after successfully communicating with the laptop. I don't know why it did not die after the tests. I also found a bug in my code as a result of the test, where the status field for the fourth gauge didn't get written to. 

 

Pressure reading issue:

I noticed that the pressure reading was not giving the atmospheric value of ~760 Torrs as expected. Looking through my previous readouts, it seems the unit showed this atm value of ~761 Torrs when the first gauge was attached. However, a closer look at the issue revealed a transient behavior, i.e. when the unit is turned on the reading dips to atm value but eventually rises up to 1000 Torrs. I don't think this is a calibration problem bcos the value of 1000 Torrs is the maximum value for the gauge range. I also found out that when the XGS-controller has been running for a while, a power cycle does not have this transient behavior. So maybe a faulty capacitor somewhere? I have attached a short video clip that shows what happens when the XGS-controller unit is turned on.

  16701   Fri Mar 4 18:12:44 2022 KojiUpdateVACRGA pumping down

1. Jordan reported that the newly installed Pirani gauge for P2 shows 850Torr while PTP2 show 680 Torr. Because of this, the vacuum interlock fails when we try to open V4.

2. Went to c1vac. Copied the interlock setting file interlock_conditions.yaml to interlock_conditions_220304.yaml
3. Deleted diffpressure line and pump_underspeed line for V4
4. Restarted the interlock service

controls@c1vac:/opt/target/python/interlocks$ sudo systemctl status interlock.service  
controls@c1vac:/opt/target/python/interlocks$ sudo systemctl restart interlock.service
controls@c1vac:/opt/target/python/interlocks$ sudo systemctl status interlock.service

5. The above 2~4 was unnecessary. Start over.


Let RP1/3 pump down TP1 section through the pump spool. Then let TP2 pump down TP1 and RGA.

1. Open V7. This made P2 a bit lower (P2 is alive) and P3.
2. Connected the main RP tube to the RP port.
2. Started RP1/3. PRP quickly reaches 0.4Torr.
3. Opened V6 this made P3 and O2 below 1Torr.
4. Close V6. Shutdown RP1/3. Disconnect the RP tube.
5. Turn on auxRP at the wall powe
6. Turn on TP2. Wait for the starting up.
7. Open V4. Once the pressure is below Pirani range, open VM3.
8. Keep it running over the weekend.

9. Once TP2 reached the nominal speed, the "StandBy" button was clicked to lower the rotation speed (for longer life of TP2)

  16704   Sun Mar 6 18:14:45 2022 TegaUpdateVACOngoing work to get the FRG gauges readouts to EPICs channels

Following repeated failure to establish communication between c1vac and the XGS600 controller via Perle IOLAN serial device server, I decided to monitor the signal voltage of the communication channels (pin#2, pin#3 and pin#5) using an oscilloscope. The result of this investigation is presented in the attached pdf document. In summary, it seems I have used a crossed wired RS232 serial cable instead of a normal RS232 serial cable, so the c1vac read request command is being relayed on the wrong comm channel (pin#2 instead of pin#3). I will swap out the cable to see if this resolves the problem.  

  16706   Mon Mar 7 13:53:40 2022 TegaUpdateVACOngoing work to get the FRG gauges readouts to EPICs channels

So it appears that my deduction from the pictures of needing a cable swap was correct, however, it turns out that the installed cable was actually the normal RS232 and what we need instead is the RS232 null cable. After the swap was done, the communication between c1vac and the XGS600 controller became active. Although, the data makes it all to the to c1vac without any issues, the scope view of it shows that it is mainly utilizing the upper half of the voltage range which is just over 50% of the available range. I don't know what to make of this.

 

I guess, the only remaining issue now is the incorrect atmospheric pressure reading of 1000 Torrs. 

 

Quote:

Following repeated failure to establish communication between c1vac and the XGS600 controller via Perle IOLAN serial device server, I decided to monitor the signal voltage of the communication channels (pin#2, pin#3 and pin#5) using an oscilloscope. The result of this investigation is presented in the attached pdf document. In summary, it seems I have used a crossed wired RS232 serial cable instead of a normal RS232 serial cable, so the c1vac read request command is being relayed on the wrong comm channel (pin#2 instead of pin#3). I will swap out the cable to see if this resolves the problem.  

 

  16707   Mon Mar 7 14:52:34 2022 KojiUpdateVACOngoing work to get the FRG gauges readouts to EPICs channels

Great trouble shoot!

> I guess, the only remaining issue now is the incorrect atmospheric pressure reading of 1000 Torrs. 

This is just a calibration issue. The controller should have the calibration function.
(The other Pirani showing 850Torr was also a calibration issue although I didn't bother to correct it. I think the pirani's typically has large distribution of the calibration values and requires individual calibration)

  16713   Tue Mar 8 12:08:47 2022 TegaUpdateVACOngoing work to get the FRG gauges readouts to EPICs channels

OMG, it worked! It was indeed a calibration issue and all I had to do was press the "OK" button after selecting the "CAL" tab beside the pressure reading. Wow.

Quote:

Great trouble shoot!

> I guess, the only remaining issue now is the incorrect atmospheric pressure reading of 1000 Torrs. 

This is just a calibration issue. The controller should have the calibration function.
(The other Pirani showing 850Torr was also a calibration issue although I didn't bother to correct it. I think the pirani's typically has large distribution of the calibration values and requires individual calibration)

 

  16717   Wed Mar 9 10:23:28 2022 JordanUpdateVACLeak Testing of New Manual Gate Valve - Attempt #1

Jordan, Chub, Paco

Chub and I went into the lab this morning to leak check the new gate valve after pumping over the weekend. This would be done through the RGA and spraying helium around the newly installed flanges. The RGA is set to monitor the partial pressure of helium versus time and we visually watch for any spikes in pressure to indicate an air leak.

So, I mistakenly thought the gate valve was opened and both sides were being pumped on, this was not the case. The vavle was closed so there was a pressure differential whenI turned the handle it tripped the interlocks and closed V4. I closed VM3 to the RGA volume to prevent the filament from being damaged.

Then the medm screen for the vac controls started flashing rapidly, I closed the window and reopened the controls to find all the panels were white, but we could still see the read only vac screen. So Paco restarted c1vac and the controls were restored. V7 closed, but was then reopened.

We now need to restart pumping with the odd pressure differentials between V4 and VM3.

Plan

- TP2 turned off along with the AUX pump

- VM3 opened to vent RGA volume (RGA turned off, filament cooled)

- Open the manual vent screw on TP1 to bring the RGA+TP1 volume back to atmosphere, now no pressure diff. on V4

- Open V4, restart AUX pump to rough out the volume

- Connect RP1/3 line to the pump scroll, turn on RP1/3 RGA+TP1 volume went to mtorr

- Slowly open the manual gate valve

- Connect AUX pump to TP2 and rough out

- Restart TP2, once at speed open V4, close V6 and turn off RP1/3

Now we are pumping on the RGA/TP1 volume with TP2, leak check attempt #2 will happen tomorrow morning

  16721   Thu Mar 10 09:39:59 2022 JordanUpdateVACLeak Testing of New Manual Gate Valve - Attempt #2

This morning Chub and I leak checked the manual gate valve with the RGA and helium. There was no change in the helium partial pressure while spraying helium around the flanges, all looks good.

 

I also took a 100 AMU analog scan, after the filament had warmed up overnight and the plot was quite noisy even with the lowest scan speed. I recommend this unit go back to SRS for a filament replacement/recalibration. I am worried yesterday's "vent" of the RGA volume may have burned the filament. See the comparison of yesterday's analog scan to today's below.

  16747   Tue Mar 29 15:07:11 2022 TegaSummaryVACOpening of MC and ETM chambers

[Anchal, Chub, Ian, Paco, Tega]

Today, we opened the MC chamber, ETMX chamber and ETMY chamber.

  16769   Mon Apr 11 11:00:30 2022 JancarloUpdateVACC1VAC Reboot and Nitrogen tanks

[Paco, JC, Ian, Jordan, Chub]

Checking in the morning, I walked over to the Nitrogen tanks to check the levels. Noticed one tank was empty, so I swapped it out. Chub came over to check the levels and to take note of how many tanks were left available for usage (None). Chub continued to put in a work order for a set of full Nitrogen tanks. We should be set on Nitrogen until Thursday this week (4/14/22).

As for C1VAC, this morning, Paco and I attempted to open the PSL shutter, but the interlock system was tripped so we didn't get any light into the IFO. We traced the issue down to C1VAC being unresponsive. We discussed this may have interlocked as a result of the Nitrogen tanks running out, but we do not believe this was the issue since we would have recieved an email. We tried troubleshooting as much as possible avoiding a reboot, but were unable to solve the issue. In response, we ran the idea of a reboot across Jordan and Ian, where everyone was in agreement, and fixed the system. Restarting c1vac seems to have closed V4, but this didn't cause any issues with the current state of the vacuum system.

After opening the PSL shutter again, we see the laser down the IFO, so we resume alignment work

  16772   Tue Apr 12 09:05:21 2022 JordanUpdateVACNew Pressure Gauge Install/Pump Spool Vent

Today, Tega and I would like to vent the pump spool an dinstall the new FRG-400 Agilent Pressure Gauges (per elog 15703). The attached picture shows the volume needed to be vented highlighted in red, and the gauges that need to be replaced/removed (purple dot next to the name).

The vent plan is as follows:

Open RV2

Open VM3

Open V7

Open V4

Shut down TP2

Install new gauges

Will add to post with updates post vent.

  16773   Wed Apr 13 12:50:07 2022 TegaUpdateVACNew Pressure Gauge Install/Pump Spool Vent

[Jordan, JC, Tega]

We have installed all the FRGs and updated the VAC medm screens to display their sensor readings. The replacement map is CC# -> FRG#, where # in [1..4] and PRP1 -> FRG5. We now need to clean up the C1VAC python code so that it is not overloaded with non-function gauges (CC1,CC2,CC3,CC4,PRP1). Also, need to remove the connection cables for the old replaced gauges.

  16777   Thu Apr 14 09:04:30 2022 JordanUpdateVACRGA Volume RGA Scans

Prior to venting the RGA volume on Tuesday (4/12/2022) I took an RGA scan of the volume to be vented (RGA+TP1 volume+Manual Gate Valve) to see if there was a difference after replacing the manual gate valve. Attached is the plot from 4/12/22, and an overlay plot to complare 4/12/22 to 12/10/2021, when the same volume was scanned with the old (defective) manual gate valve.

There is a significant drop in the ratio O2 compared the the nitrogen peak and reduced Argon (AMU 40) which indicates there is no longer a large air leak.

12/10/21 N2/O2 ratio ~ 4 (Air 78%N2 / 21%O2)

4/12/22 N2/O2 ratio ~ 10      

There is one significant (above noise level) peak above AMU 46, which is at AMU 58. This could possibly be acetone (AMU 43 and 58) but overall the new RGA Volume scans look significantly better after the manual gate valve replacement. Well done!

  16794   Thu Apr 21 11:31:35 2022 JCUpdateVACGauges P3/P4

[Jordan, JC]

It was brought to attention during yesterday's meeting that the pressures in the vacuum system were not equivalent althought the valve were open. So this morning, Jordan and I reviewed the pressure gauges P3 and P4. We attempted to recalibrate, but the gauges were unresponsive. Following this, we proceeded to connect new gauges on the outside to test for a calibration. The two gauges successfully calibrated at atmosperic pressure. We then proceeded to remove the old gauges and install the new ones. 

  16796   Thu Apr 21 16:36:56 2022 TegaUpdateVACcleanup work for vacuum git repo

git repo - https://git.ligo.org/40m/vac

Finally incorporated the FRGs into the main modbusIOC service and everything seems to be working fine. I have also removed the old sensors (CC1,CC2,CC3,CC4,PTP1,IG1) from the serial client list and their corresponding EPICS channels. Furthermore, the interlock service python script has been updated so that all occurrence of old sensors (turns out to be only CC1) were replaced by their corresponding new FRG sensor (FRG1) and a redundnacy was also enacted for P1a where the interlock condition is replicated with P1a being replaced with FRG1 because they both sense the main volume pressure.

  16806   Fri Apr 22 14:14:33 2022 JordanUpdateVACTP3 Forepump tip seal replacement

Jordan, JC

While the pumpspool is vented, I thought it would be a convenient time to change out the tip seal on the TP3 forepump. This one had not been changed since 2018, so as preventative maintence I had JC remove the pump and begin cleaning/installing the new tip seal.

Unfortunately the tip seal broke, but I have ordered another. We should have this pump ready to go late next week. If one is needed sooner, there is a spare IDP 7 pump we can install as the TP3 forepump.

  16820   Fri Apr 29 08:34:40 2022 JCUpdateVACRGA Pump Down

Jordan and I, in order to start pumpig down the RGA Volume, we began by opening V7 and VM. Afterwards, we started RP1 and RP3. After this, the pressure in the line between RP1, RP3, and V6 dropped to 3.4 mTorr. Next, we tried to open V6, although an error message popped up. We haven't been able to erase it since. But we were able to turn on TP2 with V4 closed. The pressure in that line is reporting 1.4 mTorr.

 

PRP on the sitemap is giving off an incorrect pressure for the line between RP1, RP3, and V6. This is verified by the pressure by the control screen and the physical controller as well. 

  16822   Mon May 2 10:00:34 2022 JordanUpdateVACTP3 Forepump tip seal replacement

[JC, Jordan]

Jordan recieved the new tip seal Friday afternoon and I continued the replacement process in the morning. Finishing up, we proceeded to test the pump in the Clean and Bake room. The pump's pressure lowered to 110 mTorr, and we continue pumping so the seal can recieve a good fitting.

Update: We have confirmed the pump is working great and have reinstalled this back into the vacuum system.                                                                                                                   Note: The same O-Rings were used.

Quote:

Jordan, JC

While the pumpspool is vented, I thought it would be a convenient time to change out the tip seal on the TP3 forepump. This one had not been changed since 2018, so as preventative maintence I had JC remove the pump and begin cleaning/installing the new tip seal.

Unfortunately the tip seal broke, but I have ordered another. We should have this pump ready to go late next week. If one is needed sooner, there is a spare IDP 7 pump we can install as the TP3 forepump.

 

  16825   Tue May 3 13:18:47 2022 JCUpdateVACRGA Pump Down

Jordan, Tega, JC

Issue has been resolved. Breaker on RP1 was tripped so the RP1 button was reporting ON, but was not actually on which continuously tripped the V6 interlock. Breaker was reset, RP1 and RP3 turned on. The V6 was opened to rough out the RGA volume. Once, pressure was at ~100mtorr, V4 was opened to pump the RGA with TP2. V6 was closed and RP1/3 were turned off.

RGA is pumping down and will take scans next week to determine if a bakeout is needed

Quote:

Jordan and I, in order to start pumpig down the RGA Volume, we began by opening V7 and VM. Afterwards, we started RP1 and RP3. After this, the pressure in the line between RP1, RP3, and V6 dropped to 3.4 mTorr. Next, we tried to open V6, although an error message popped up. We haven't been able to erase it since. But we were able to turn on TP2 with V4 closed. The pressure in that line is reporting 1.4 mTorr.

 

PRP on the sitemap is giving off an incorrect pressure for the line between RP1, RP3, and V6. This is verified by the pressure by the control screen and the physical controller as well. 

 

  16867   Fri May 20 12:42:23 2022 TegaUpdateVACDoor installation on the end stations

[JC, Tega, Chub]

Today we installed the 200 lbs doors on the end station chambers.

  16870   Tue May 24 10:37:09 2022 TegaUpdateVACadded FRG channels to slow channel ini file

[Vacuum gauge sensors]

Paco informed me that the FRG sensor EPICS channels are not available on dataviewer, so I added them to slow channels ini file (/opt/rtcds/caltech/c1/chans/daq/C0EDCU.ini). I also commented out the old CC1, CC2, CC3 and CC4 gauges. A service restart is required for them to become available but this cannot be done right now because it would adversely affect the progress of the upgrade work. So this would be done at a later date.

Quote:

git repo - https://git.ligo.org/40m/vac

Finally incorporated the FRGs into the main modbusIOC service and everything seems to be working fine. I have also removed the old sensors (CC1,CC2,CC3,CC4,PTP1,IG1) from the serial client list and their corresponding EPICS channels. Furthermore, the interlock service python script has been updated so that all occurrence of old sensors (turns out to be only CC1) were replaced by their corresponding new FRG sensor (FRG1) and a redundnacy was also enacted for P1a where the interlock condition is replicated with P1a being replaced with FRG1 because they both sense the main volume pressure.

 

  16871   Tue May 24 11:04:53 2022 JCUpdateVACBeginning Pumpdown

[JC, Jordan, Paco, Chub]

We began with the pumpdown this morning. We started with the annulus volume and proceeded by using the following:

1. Isolate the RGA Volume by closing of valves VM3 and V7.

2. Opened valves VASE, VASV, VABSSCT, VABS, VABSSCO, VAEV, and VAEE, in that order.

3. Open VA6 to allow P3, FRG3, and PAN to equalize.

4. Turn on RP1 and RP3, rough out annulus volume, once <1 torr turn on TP3. Close V6. Open V5 to pump the annulus volume with TP3.

5. Re route pumping from RP1 and RP3 to the main volume by opening V3 and slowly opening RV1.

6. After ~3.5 hours the pressure in the arms was <500mtorr on both FRG1 and P1a. Turn on TP1 and wait to reach full speed 560 Hz

7. Open V1 with RV2 barely open. The pressure diff between P1a and P2/FRG2 needs to be below 1 torr. This took a couple attempts with the manual valve in different positions. The interlocks were tripped for this reason. Repeat step 7 until the manual gate valve was in a position that throttled pumping enough to maintain the <1 torr differential.

8. Slowly open the manual gate valve over the course of ~ 1 hour. Once the manual gate valve fully opened, pressure in the arms was <1mtorr.

9. V7 was closed, leaving only TP2 to back TP1, while TP3 was used to continue pumping the annuli. Left in that configuration overnight (see attached)

 

We did have to replace gauge PAN becuase it was reading a signal error. In addition, we found the cable is a bit sketchy and has a sharp bend. The signal comes in and out when the cable is fiddled with.

  16900   Wed Jun 8 16:03:12 2022 KojiUpdateVACPower Outage 220608: Vacuum Recovery

=== Observation ===

  • TP1/2/3 failed. AUX RP did not survive the power outage (of course)

=== Initial Recovery of TP2/TP3 ===

  • Stopped TP1/2/3
  • Restored RP AUX at the wall breaker module
  • Restarted TP2/3

=== Rough pumping down ===

  • Found RP1/RP3 left connected (unusual)
  • Turned on RP1 and RP3
  • Opened V6 and V7 for roughing of the pump spool. Now 0.3 torr for P2/P3/PA* (annuli) - ready to open V4 and V5 to backup TP1 by TP2 and TP3. Waiting for TP2/3 to be accelerated

=== Towards main volume pumping ===

  • TP2 accelerated and opened V4 (This caused V6 to close). RP1 and RP3 stopped.
  • TP3 hasitated to spin up. Stopped for now.
  • TP1 needed to be reset to clear the displacement errors. Once the errors were cleared, TP1 was restarted (acclerating).
  • TP1 at normal speed. Opened V1 (main volume pumping started)
     
  • Disconnected RP1/RP3 line.
     
  • TP3 was restarted and is now stable at 55k RPM (indicated as soft spin / low speed). Pushed "Stand By On" and "then "Stand By OFF". This makes the TP3 back to "NORMAL". Switched to "Stand By On".
  • TP2 was also switched to "Stand By On".

Now the main volume and the annuli are pumped down TP1 and TP2/TP3 with RP AUX backing.

The attachment is the pressure glitch for the main volume.

 

  16916   Wed Jun 15 07:26:35 2022 JCUpdateVACBeginning Pumpdown

[Jordan, JC]

Jordan and I went in to retore the Vacuum System back to it's original state before the power loss on June 8, 2022. The process went smoothly as we first closed V7 and opened VM3 (in that order).

The RP1/3 line did not have the KF blank installed. That was added and the RP flex line was capped off.

Quote:

[JC, Jordan, Paco, Chub]

We began with the pumpdown this morning. We started with the annulus volume and proceeded by using the following:

1. Isolate the RGA Volume by closing of valves VM3 and V7.

2. Opened valves VASE, VASV, VABSSCT, VABS, VABSSCO, VAEV, and VAEE, in that order.

3. Open VA6 to allow P3, FRG3, and PAN to equalize.

4. Turn on RP1 and RP3, rough out annulus volume, once <1 torr turn on TP3. Close V6. Open V5 to pump the annulus volume with TP3.

5. Re route pumping from RP1 and RP3 to the main volume by opening V3 and slowly opening RV1.

6. After ~3.5 hours the pressure in the arms was <500mtorr on both FRG1 and P1a. Turn on TP1 and wait to reach full speed 560 Hz

7. Open V1 with RV2 barely open. The pressure diff between P1a and P2/FRG2 needs to be below 1 torr. This took a couple attempts with the manual valve in different positions. The interlocks were tripped for this reason. Repeat step 7 until the manual gate valve was in a position that throttled pumping enough to maintain the <1 torr differential.

8. Slowly open the manual gate valve over the course of ~ 1 hour. Once the manual gate valve fully opened, pressure in the arms was <1mtorr.

9. V7 was closed, leaving only TP2 to back TP1, while TP3 was used to continue pumping the annuli. Left in that configuration overnight (see attached)

 

We did have to replace gauge PAN becuase it was reading a signal error. In addition, we found the cable is a bit sketchy and has a sharp bend. The signal comes in and out when the cable is fiddled with.

 

  4274   Fri Feb 11 16:43:09 2011 steveUpdateVIDEOMC1 & 3 video monitor

I set up video monitoring of MC1 and MC3

  4314   Thu Feb 17 13:20:06 2011 SureshUpdateVIDEOSome more labels

[Larisa, Kiwamu, Steve and Suresh]

 

  We continued the labeling of video cables. All exiting cables which are going to be used used in the new scheme have been labeled.

We also labeled the cables running from the video mux to the TV monitors in the computer room. Some of these will be removed or reallocated.

We will continue next Wednesday (after the meeting) and will lay cables that are most urgently required. 

 i

  4346   Wed Feb 23 16:56:17 2011 Larisa ThorneUpdateVIDEOCable laying...continued

Having finished labeling the existing cables to match their new names, we (Steve, Kiwamu and Larisa) moved on to start laying new cables and labeling them according to the list.

 

Newly laid cables include: ETMXT (235'), ETMX (235'), POP (110') and MC2 (105').  All were checked by connecting a camera to a monitor and checking the clarity of the resulting image. Note that these cables were only laid, so they are not plugged in.

 

The MC2 cable needs to be ~10' longer; it won't reach to where it's supposed to. It is currently still in its place. 

The three other cables were all a lot longer than necessary.

  4390   Wed Mar 9 16:07:42 2011 kiwamuUpdateVIDEOcable session

[Koji, Steve, Suresh, Kiwamu]

The following video cables have been newly laid down :

  - MC1F/MC3F (65 ft.)

  - PMCR (100 ft.)

  - PSL spare (100 ft.)

  - PSL1  (100 ft.)

  - PSL2  (100 ft.)

 

  4462   Wed Mar 30 17:01:08 2011 Larisa ThorneUpdateVIDEOCable laying...continued

[Steve, Suresh, Kiwamu, Larisa]

 

Only the PRM/BS cable was laid today.

In one of the previous updates on cable laying, it was noted that the MC2 cable needed an additional 10' and the MC2T needed an additional 15' to reach their destinations.  We cut and put BNC ends on 10' and 15' cables and connected them to the original cables in order to make them long enough.

 

This concludes the laying of new cables. Suresh is currently working on the QUADs...

  4474   Thu Mar 31 08:31:44 2011 SureshUpdateVIDEOCable laying...continued

The video work has crossed a milestone.    

Kiwamu and Steve have shifted the three quads from the control room to the Video MUX rack (1Y1) and have wired them to the MUX.

The monitors in the control room have been repositioned and renumbered.  They are now connected directly to the MUX. 

Please see the new cable list for the input and output channels on the MUX.

As of today, all cables according the new plan are in place.  Their status   indicated on the wiki page above is not verified .  Please ignore that column for now, we will be updating that soon.

I shifted the MC1F/MC3F camera and the MC2F cameras onto the new cables.  Also connected the monitors at the BS chamber and end of the X arm to their respective cables.  I have removed the RG58B BNC (black) cables running from MC2 to BS and from ETMXF to the top of the Flow Bench. 

Some of the old video cables are still in place but are not used.   We might consider removing them to clear up the clutter. 

Some of the video cables in use are orange and if the lab's  cable color code is to be enforced these will have to be replaced with blue ones..

Some of the cables in use running from the MUX to the monitor in the control room are the white 50 Ohm variety.  There are also black RG59 Cables running the same way ( we have surplus cables in that path)  and we have to use those instead of the white ones. 

There are a number of tasks remaining:

a)  The inputs from the various existing cameras have to be verified. 

b) There are quite a few cameras which are yet to be installed.

c) The Outputs may not not be connected to their monitors.  That the monitors may still be connected to an old cable which is not connected to the MUX.  The new cable should be lying around close by.  So if you see a blank monitor please connect it to its new cable. 

d) The status column on the wiki page has to be updated.

e) Some of the currently in place may need to be replaced and some need to be removed.  We need to discuss our priorities and come up with a plan for that.

After checking everything we can certify that the video cabling system is complete.

I would like Joon Ho to take care of this verification+documenting process and declaring that the job is complete. 

 

Steve attached these two pictures.

  4498   Thu Apr 7 13:12:23 2011 KojiHowToVIDEOVideo switching tip

Long time ago, I looked at the manual of the video switcher.
http://media.extron.com/download/files/userman/Plus_Ultra_MAV_C.pdf
Here is the summary. This will be the basic of the more sophisticated switching program which may have GUI.

In principle, you can manually control the matrix via telnet. At the console machines, you can connect to the matrix using telnet

telnet 192.168.113.92

This opens TCP/IP port 23 of the specified machine. You will receive some messages.
Then type some command like:
--------------------

  • 1*2!       (connect input#1 to output#2)
  • 1,           (save the current setting into preset1)
  • 1.           (restore the setting from preset1)

--------------------

Basicaly that's all. There are many other features but I don't think we need them.

We can create a simple program with any of the language as any of the language has the capability of the TCP/IP connection.
e.g. C, Perl, Python. Tcl/Tk
Any of them are fine.

Now what we have to think about is how to implement the interface in the epics screen (or whatever).
It needs some investigation how the people is thinking as the ideal interface.
But, first of all, you should make the above three operations available as a simple UNIX command like:

videoswitch -i 192.168.113.92 1 2
videoswitch -i 192.168.113.92 -store 1
videoswitch -i 192.168.113.92 -recall 1
(There is no such command yet. These are showing what it should be!)

This can be done by a single day work and our life will be much better.

  4529   Fri Apr 15 02:30:24 2011 KojiHowToVIDEOVideo switching tip

I have made a small python script to handle the video matrix.

It is too far from the perfection, but I release it as it is already useful in some extent.

The script is in the /cvs/cds/rtcds/caltech/c1/scripts/general directory.

usage:

videoswitch.py in_ch_name out_ch_name

in_ch_name is one of the followings

MC2F, IFOPO, OMCR, FI, AS_Spare, ITMYF, ITMXF, ETMYF, ETMXF,
PMCR, RCR, RCT, PSL_Spare, PMCT, ETMXT, MC2T, POP, IMCR, REFL,
MC1F, SRMF, AS, ETMYT, PRM, OMCT, Quad1, Quad2, Quad3

out_ch_name is one of the followings

Mon1, Mon2, Mon3, Mon4, Mon5, Mon6, Mon7,
ETMY, MC1, PSL1, PSL2, ETMX, MC2, CRT9,CRT10,Projector,
Quad1_1, Quad1_2, Quad1_3, Quad1_4,
Quad2_1, Quad2_2, Quad2_3, Quad2_4,

Quad3_1, Quad3_2, Quad3_3, Quad3_4

ELOG V3.1.3-