 Wed Feb 9 11:56:24 2022, Anchal, Update, General, Bringing back CDS 
|
[Anchal, Paco]
Bringing back CDS took a lot of work yesterday. I'm gonna try to summarize the main points here.
mx_start_stop
For some reason, fb1 was not able to mount mx devices automatically on system boot. This was |
|
Wed Feb 9 13:55:05 2022, Koji, Update, General, Bringing back CDS
|
Great recovery work and cleaning of the rebooting process.
I'm just curious: Did you observe that the c1sus2 cards have different numbering order than the previous along with the power outage/cycling? |
|
Wed Feb 9 16:43:35 2022, Paco, Update, General, Scheduled power outage recovery - Locking mode cleaner(s)
|
[Paco, Anchal]
We went in and measured the power after the power splitting HWP at the PSL table. Almost right before the PSL shutter (which was closed),
when the PMC was locked we saw ~ 598 mW (!!)
Checking back on ESP300, it seems the channel was not enabled even though the right angle was punched |
|
Thu Feb 10 15:41:00 2022, Anchal, Update, General, Scheduled power outage recovery - Locking mode cleaner(s)
|
I found out that the ESP300 service needs to be run in root mode for it to be able to connect to the USB port of HWP motor controller. While doing this
change, I noticed that the channels hosted by c1psl might have a duplication conflict with some other channel hosting computer, because a lot of them show
the Warning: "Identical process variable names on multiple servers" which is not good. Someone should look into this conflict. |
|
Thu Feb 10 17:57:48 2022, Anchal, Update, General, Scheduled power outage recovery - Locking mode cleaner(s)
|
Something is wrong with the Video MUX. The system did not turn back on with full functionality. Even though we see the screens as they were before the
power shutdown, we have lost control on switching any of the videos. I went to check the wiki
page about Video MUX which told be we should be able to see the configuration screen on this link, but the page |
|
Thu Feb 10 19:03:23 2022, Koji, Update, General, Scheduled power outage recovery - Locking mode cleaner(s)
|
I came back to the 40m and started the investigation.
If I ping 192.168.113.92, it responds. But telnet (port 23) was rejected. I somehow tried ssh and it responds! I even could login to the host
using usual password. Here is the prompt. |
|
Thu Feb 10 19:46:37 2022, Koji, Update, General, Scheduled power outage recovery - Locking mode cleaner(s)
|
== Assign new IP address to c1sus2 ==
cf: [40m ELOG 16398] [40m
ELOG 16396] |
|
Thu Feb 10 21:10:43 2022, Koji, Update, General, Video Mux setting reset
|
Now the video matrix is responding correctly and the web interface shows up. (Attachment 1)
Also the video buttons respond as usual. I pushed Locking Template button to bring the setting back to nominal. (Attachment 2) |
 Thu Feb 10 21:51:02 2022, Koji, Update, CDS, [Solved] Huge random numbers flowing into ETMX/ETMY ASC PIT/YAW
|
Huge random numbers are flowing into ETMX/ETMY ASC PIT/YAW. Because of this, I could not damp the ETMX/ETMY suspension at the beginning during the recovery
from rebooting. (Attachment 1)
By turning off the output of the ASC filters, the mirrors were successfully damped. |
|
Fri Feb 11 10:56:38 2022, Anchal, Update, CDS, [Solved] Huge random numbers flowing into ETMX/ETMY ASC PIT/YAW
|
Yeah, this is a known issue actually. We go to ASC screen and manually swich off all the outputs after every reboot. We haven't been able to find
a way to set default so that when the model comes online, these outputs remain switched off. We should find a way for this.
|
|
Fri Feb 11 11:17:00 2022, Anchal, Update, General, Scheduled power outage recovery
|
I found that two computers are not powering up in the control room, Ottavia and Allegra. Allegra was important for us as it had the current version of
LIGO CDS workstation installed on, providing us with options to use latest packages written by LIGO CDS team. I think the power issue should be resolvable
if someone opens it and knows what thye are doing. Do we have any way of getting fuse repairs on such computers? Both these computers are Dell XPS 420. |
|
Fri Feb 11 12:22:19 2022, rana, Update, CDS, [Solved] Huge random numbers flowing into ETMX/ETMY ASC PIT/YAW
|
you can hand edit the autoBurt file which the FE uses to set the values after boot up. Just
make a python script that amends all of the OFF or ZERO that are needed to make things safe. This would be the autoBurt snap used on boot up only, and
not the hourly snaps. |
|
Fri Feb 11 16:09:11 2022, Anchal, Update, General, Scheduled power outage recovery - Input power increased
|
We increased the input power to IMC by replacing the 98% transmission BS by a 10% transmission BS on the detection table (reverse of what mentioned in
40m/16408 see attachment 8-9). We then realigned the BS so that MC RFPD is centered. Then we realigned two steering mirrors to get the beam centered on |
|
Fri Feb 11 17:07:19 2022, Anchal, Update, CDS, [Solved] Huge random numbers flowing into ETMX/ETMY ASC PIT/YAW
|
The autoBurt file for FE already has the C1:ASC-ETMX_PIT_SW2 (and other channels for ETMY, ITMX, ITMY, BS and for YAW) present, and I checked the last
snapshot file from Feb 7th, 2022, which has 0 for these channels. So I'm not sure why when FE boots up, it does not follow the switch configuration.
Fr safety, I changed all the gains of these filter modules, named like C1:ASC-XXXX_YYY_GAIN (where XXXX is ETMX, ETMY, ITMX, ITMY, or BS , and YYY is PIT |
|
Mon Feb 14 18:31:50 2022, Paco, Update, General, Scheduled power outage recovery - IMC recovery progress
|
[Paco, Anchal, Tega]
We have been realigning the IMC as of last Friday (02/11). Today we made some significant progress (still at high input power), but the IMC autolocker
is unable to engage a stable mode lock. We have made some changes to reach this point, including re-centering of the MC1 REFL beam on the ccd, centering |
|
Mon Feb 14 21:03:25 2022, Koji, Update, General, Scheduled power outage recovery
|
I opened the boxes. Allegra has obvious vent of at least 4 caps. And the power supply did not respond even a paper clip test was performed. https://www.silverstonetek.com/downloads/QA/PSU/PSU-Paper%20Clip-EN.pdf
(Paper Clip Test)
=> The mother board and the PSU are dead. |
|
Tue Feb 15 19:32:50 2022, Koji, Update, General, Scheduled power outage recovery - IMC recovery progress
|
Reduced the IMC power to 100mW
Setup: The power meter was placed right before the final aperture (Attachment 1)
Before the adjustment: Initial position of the HWP was 37.29deg and the input power was 987mW (Attachments 2/3) |
|
Tue Feb 15 19:40:02 2022, Koji, Update, General, IMC locking
|
IMC is locking now. There was nothing wrong: just a careful alignment + proper gain adj
=== Primary Alignment ===
- I used WFS error signals as the indicator of the PDH error signals. Checked C1:IOO-WFS1_(I/Q)n_ERR and ended up C1:IOO-WFS1_I4_ERR |
|
Wed Feb 16 15:19:41 2022, Anchal, Update, General, Reconfigured MC reflection path for low power
|
I reconfigured the MC reflection path for low power. This meant the following changes:
Replaced the 10% reflection BS by 98% reflection beam splitter
Realigned the BS angle to get maximum on C1:IOO-MC_RFPD_DCMON
when cavity is unlocked.
Then realigned the steering mirrors for WFS1 and WFS2.
I tried to align the light for MC reflection CCD |
|
Tue Feb 22 18:47:51 2022, Ian MacMillan, Update, SUS, ETMY SUS Electronics Replacement
|
[Ian, Koji]
In preparation for the replacement of the suspension electronics that control the ETMY, I took measurments of the system excluding the CDS System.
I took transfer functions from the input to the coil drivers to the output of the OSEMs for each sensor: UL, UR, LL, LR, and SIDE. These graphs are |
|
Wed Feb 23 15:08:57 2022, Anchal, Update, General, Removed extra beamsplitter in MC WFS path
|
| As discussed in the meeting, I removed the extra beam splitter that dumps most of the beam going towards WFS photodiodes. This beam splitter needs to be placed back in position before increasing the input power to IMC at nominal level. This is to get sufficient light on the WFS photodiodes so that we can keep IMC locked for more than 3 days. Currently IMC is unlocked |
|
Thu Feb 24 14:32:57 2022, Anchal, Update, General, MC RFPD DCMON channel got stuck to 0
|
I found a peculiar issue today. The C1:IOO-MC_RFPD_DCMON remains constantly 0. I wonder if the RFPF output is being read properly. I opened the table
and used an oscilloscope to confirm that the DC output from the MC REFL photodiode is coming consistently but our EPICs channel is not reading it. I tried
restarting the modbusIOC service but that did not affect anything. I power cycled the acromag chassis while keeping the modbusIOC service off, and then |
|
Thu Feb 24 18:05:58 2022, Yehonathan, Update, BHD, Re-susspension of AS1
|
{Yehonathan, Anchal, Paco}
Yesterday, Anchal and Paco removed AS1 from the vacuum chamber and moved it into the cleanroom. The suspension wires were cut and the AS1 optic
was put on the table. |
|
Thu Feb 24 19:26:32 2022, Anchal, Update, General, IMC Locking
|
I think I have aligned the cavity, including MC1 such that we are seeing flashing of fundamental mode and significant transmission sum value as well.However,
I'm unable to catch lock following Koji's method in 40m/16673. Autolocker could not
catch lock either. Maybe I am doing something wrong, I'll pickup again tomorrow, hopefully the cavity won't drift too much in this time. |
|
Fri Feb 25 14:00:08 2022, Ian MacMillan, Update, SUS, ETMY SUS Electronics Replacement
|
[Koji, Ian]
We looked at a few power supplies and we found one that was marked "CHECK IF THIS WORKS" in yellow. We found that the power supply
worked but the indicator light didn't work. I tried a two other lights from other power supplies that did not work but they did not work. Koji ordered |
|
Fri Feb 25 14:48:53 2022, Ian MacMillan, Update, SUS, ETMY SUS Electronics Replacement
|
I moved the network-enabled power strip from above the power supplies on rack 1y4 to below. Nothing was powered through the strip when I unplugged everything
and I connected everything to the same port after. |
|
Sat Feb 26 01:01:40 2022, Tega, Update, VAC, Ongoing work to get the FRG gauges readouts to EPICs channels
|
I will make a detailed elog later today giving a detailed outline of the connection from the Agilent gauge controller to the vacuum subnet and the
work I have been doing over the past two days to get data from the unit to EPICs channels. I just want to mention that I have plugged the XGS-600 gauge
controller into the serial server on the vacuum subnet. I check the vacuum medm screen and I can confirm that the other sensors did not experience and |
|
Sat Feb 26 15:45:14 2022, Tega, Update, VAC, Ongoing work to get the FRG gauges readouts to EPICs channels
|
I have attached a flow diagram of my understanding of how the gauges are connected to the network.
Earlier today, I connected the XGS-600 gauge controller to the IOLAN Serial Device Server on port 192.168.114.22 .
The plan is a follows: |
|
Sat Feb 26 23:48:14 2022, Koji, Update, SUS, ETMY SUS Electronics Replacement
|
[Ian, Koji] - Activity on 25th (Fri)
We continued working on the ETMY electronics replacement.
- The units were fixed on the rack along with the rack plan. |
|
Sun Feb 27 00:37:00 2022, Koji, Update, General, IMC Locking Recovery
|
Summary:
- IMC was locked.
- Some alignment change in the output optics. |
|
Sun Feb 27 01:12:46 2022, Koji, Update, General, IMC manual alignment procedure
|
We expect that the MC sus are susceptible to the temperature change and the alignment drifts away with time.
Here is the proper alignment procedure.
0) Assume there is no TEM00 flash or locking, but the IMC is still flashing with higher-order modes. |
|
Mon Feb 28 15:51:07 2022, Ian MacMillan, Update, SUS, ETMY 1Y4 Electronics Replacement
|
[Paco, Ian]
paco helped me wire the ETMY 1Y4 rack. We wired the following (copied from Koji's email):
Use DB9-DB9 to complete the wiring between
16bit DAC AI Chassis - End DAC Adapter (4 cables)
End DAC Adapter |
|
Mon Feb 28 19:15:10 2022, Tega, Update, VAC, Ongoing work to get the FRG gauges readouts to EPICs channels
|
I decided to create an independent service for the XGS data readout so we can get this to work first before trying to integrate into current system.
After starting the service, I noticed that the EPICS channel were not updating as expected. So I started to debug the problem and managed to track it down
to an ip socket connect() error, i.e. we get a connection error for the ip address assigned to the LAN port to which the XGS box was connected. After trying |
|
Tue Mar 1 16:01:14 2022, Paco, Update, Electronics, RFSoC 2x2 board -- setup for remote work & BALUN saga
|
[Tommy, Paco]
Since last week I've worked with tommy on getting the RFSoC 2x2 board to get some TFs from simple minicircuits type filters. The first thing
I did was set up the board (which is in the office area) for remote access. I hooked up the TCP/IP port to a wall ethernet socket (LIGO-04) and the caltech |
|
Tue Mar 1 19:26:24 2022, Koji, Update, SUS, ETMY SUS Electronics Replacement
|
The replacement key switches and Ne Indicators came in. They were replaced and work fine now.
The power supply units were tested with the X end HeNe display. It turned out that one unit has the supply module for 1350V 4.9mA while the other
two do 1700V 4.9mA. |
|
Tue Mar 1 20:38:49 2022, Tega, Update, VAC, Ongoing work to get the FRG gauges readouts to EPICs channels
|
During my investigation, I inadvertently overwrote the serial port configuration for the connected devices. So I am now working to get it all back. I
have attached screenshots of the config settings that brought back communication that is not garbled. There is no physical connection to port 6, which
I guess was initially used for the UPS serial communication but not anymore. Also, ports 9 and 10 are connected to Hornet and SuperBee, both of which |
|
Wed Mar 2 11:50:39 2022, Tega, Update, VAC, Ongoing work to get the FRG gauges readouts to EPICs channels
|
Here is the IOLAN SDS TCP socket setting and the USBserial setting for comparison.
I have also included the python script and output from the USBserial test from earlier. |
|
Wed Mar 2 12:40:08 2022, Tega, Update, VAC, Ongoing work to get the FRG gauges readouts to EPICs channels 6x
|
Connector Test:
A quick test to rule out any issue with the Ethernet to Serial adapter was done using the setup shown in Attachment 1. The results rule out any
connector problem. |
|
Wed Mar 2 14:02:43 2022, Yehonathan, Update, BHD, Re-susspension of AS1
|
Yesterday, I rebuilt the OpLev setup in the cleanroom in order to suspend AS1. It took me a while to find all the necessary parts but I found them in
the end.
The HeNe laser was placed on the optical table and turned on. The beam was aimed to bounce off a folding mirror to the SOS tower. |
|
Thu Mar 3 04:11:36 2022, Koji, Update, SUS, ETMY 1Y4 Electronics Replacement
|
For the Y-end electronics replacement, we want to remove unused power supplies. In fact, we already removed the +/-5V supplies from the stack. I was
checking what supply voltages are used by the Eurocard modules. I found that D990399 QPD whitening board
had the possible use of +/-5V. |
|
Thu Mar 3 04:24:23 2022, Koji, Update, SUS, ETMY 1Y4 Electronics Replacement
|
The DC power strip at Y-end was connected to the bottom two Sorensen power supplies. They are configured to provide +/-18V.
|
|
Thu Mar 3 17:09:46 2022, Paco, Update, BHD, Re-susspension of AS1
|
[Anchal, Paco]
Wire clamp spare was installed, furthermore AS1 was reinstalled on adapter, attached wire clamps, and cleaned using ionized air gun. Finally,
we suspended it on the SOS tower and left it resting on the bottom earthquake stops; ready for balancing. |
|
Thu Mar 3 17:21:11 2022, Ian MacMillan, Update, SUS, ETMY 1Y4 Electronics Replacement
|
[Koji, Ian]
1) We attached the 30 coil driving cables to the vacuum feed through to the sat amp [40m
wiki] they run along the cable tray then up and down into the rack. |
|
Fri Mar 4 18:12:44 2022, Koji, Update, VAC, RGA pumping down
|
1. Jordan reported that the newly installed Pirani gauge for P2 shows 850Torr while PTP2 show 680 Torr. Because of this, the vacuum interlock fails when
we try to open V4.
2. Went to c1vac. Copied the interlock setting file interlock_conditions.yaml to interlock_conditions_220304.yaml |
|
Sat Mar 5 02:03:46 2022, Koji, Update, SUS, ETMY 1Y4 Electronics Replacement 6x
|
Oplev saga
Summary
- The new coil driver had not enough alignment range to bring the oplev beam back to the QPD center |
|
Sun Mar 6 18:14:45 2022, Tega, Update, VAC, Ongoing work to get the FRG gauges readouts to EPICs channels
|
Following repeated failure to establish communication between c1vac and the XGS600 controller via Perle IOLAN serial device server, I decided to monitor the
signal voltage of the communication channels (pin#2, pin#3 and pin#5) using an oscilloscope. The result of this investigation is presented
in the attached pdf document. In summary, it seems I have used a crossed wired RS232 serial cable instead of a normal RS232 serial cable, so the c1vac |
|
Mon Mar 7 10:06:32 2022, Yehonathan, Update, IOO, IMC unlocked again, completely misaligned
|
Came this morning and saw that the IMC is unlocked.
Went into MC Lock screen and see that the watchdog is down and the PSL shutter is closed. I tried to open the shutter but nothing happened -
no REFL signal or beam on the MC REFL camera . |
|
Mon Mar 7 13:53:40 2022, Tega, Update, VAC, Ongoing work to get the FRG gauges readouts to EPICs channels
|
So it appears that my deduction from the pictures of needing a cable swap was correct, however, it turns out that the installed cable was actually
the normal RS232 and what we need instead is the RS232 null cable. After the swap was done, the communication between c1vac and the XGS600 controller became
active. Although, the data makes it all to the to c1vac without any issues, the scope view of it shows that it is mainly utilizing the upper half of |
|
Mon Mar 7 14:52:34 2022, Koji, Update, VAC, Ongoing work to get the FRG gauges readouts to EPICs channels
|
Great trouble shoot!
> I guess, the only remaining issue now is the incorrect atmospheric pressure reading of 1000 Torrs.
This is just a calibration issue. The controller should have the calibration function. |
|
Mon Mar 7 14:55:33 2022, Koji, Update, IOO, IMC unlocked again, completely misaligned
|
Hmm, the bias values were reset at 2022-03-03-20:01UTC which is 2022-03-03-12:01 PST with no apparent disruption of the data acquisition (= no resetting
of the RTS). Not sure how this could happen.
|