40m QIL Cryo_Lab CTN SUS_Lab TCS_Lab OMC_Lab CRIME_Lab FEA ENG_Labs OptContFac Mariner WBEEShop
  40m Log, Page 38 of 323  Not logged in ELOG logo
ID Date Author Type Category Subject
  14391   Wed Jan 9 11:07:09 2019 gautamUpdateVACNew Vac channel logging

Looks like I didn't restart all the daqd processes last night, so the data was not in fact being recorded to frames. I just restarted everything, and looks like the data for the last 3 minutes are being recorded yes. Is it reasonable that the TP1 current channel is reporting 0.75A of current draw now, when the pump is off? Also the temperature readback of TP3 seems a lot jumpier than that of TP2, probably has to do with the old controller having fewer ADC bits or something, but perhaps the SMOO needs to be adjusted.

Quote:
 

Gautam and I updated the framebuilder config file, adding the newly-added channels to the list of those to be logged.

Attachment 1: Screenshot_from_2019-01-09_11-08-28.png
Screenshot_from_2019-01-09_11-08-28.png
  14390   Tue Jan 8 19:13:39 2019 JonUpdateUpgradeReady for pumpdown tomorrow

Everything is set for a second pumpdown tomorrow. We'll plan to start pumping after the 1pm meeting. Since the main volume is already at 12 torr, the roughing phase won't take nearly as long this time.

I've added new channels for the TP1 analog readings (current and speed) and for the two N2 tank pressure readings. Chub finished installing the new regulator and has run the transducer signal cable to the vacuum rack. In the morning he will terminate the cable and make the final connection to the Acromag.

Gautam and I updated the framebuilder config file, adding the newly-added channels to the list of those to be logged. We also set up a git repo containing all of the Python interlock/interfacing code: https://git.ligo.org/40m/vacpython. The idea is to use the issue tracker to systematically document any changes to the interlock code.

  14389   Tue Jan 8 10:27:27 2019 gautamUpdateGeneralNear-term in-chamber work

Here is a list of tasks I think we should prioritize for the next two weeks. The idea is to get back to the previous state of being able to do single arm, PRMI-on-carrier and DRMI locking, before making further changes.

Once the new folding mirrors arrive, I'd like to modify the SRC length to allow locking in the signal-recycled config as opposed to RSE. Still need to do the detailed layout, but I think the in-vacuum layout will work. In that case, I'd like to also move the OMC and OMMT to the IY table, and also move the in-air AS photodiodes to the IY in-air optical table. This is why I've omitted the OMC alignment from this near-term list, but if we want to not move the OMC, then we probably should add alignment of the AS beam to the OMC to this list.

List of in-chamber tasks for 1/2019

Chamber Task(s)
EY
  • Clean ETMY optic and suspension
  • Put ETMY suspension back in place, recover Y-arm cavity alignment
  • Remove any residual hardware from unused heater setup
  • Restore parabolic heater setup, center radiation pattern as best as possible on ETMY
  • Check beam position on IPANG steering mirror
IY
  • Clean ITMY optic and suspension cage
  • Restore ITMY suspension, recover Y arm cavity alignment.
  • Check position of AS beam on OM1/OM2
BS/PRM (if we decide to open it)
  • Replace BS/PRM Oplev HeNe, bring the beam in and out of vacuum with beam well centerd on in-vacuum mirrors (can take this opportunity to fix the in-air layout as well to minimize un-necessary steering mirrors)
  • Check position of AS beam on OM3/OM4, adjust if necessary
  • Check position of IPPOS and IPANG beams on their respective steering optics
OMC (if we decide to open it)
  • Check position of AS beam on OM5/OM6
  • Ensure AS beam exits the vacuum cleanly
  14388   Mon Jan 7 19:21:45 2019 JonConfigurationComputer Scripts / ProgramsVac system shutdown

ADC work finished for the day. The vac controls are back up, with all valves CLOSED and all pumps OFF.

Quote:

I'm making a controlled shutdown of the vac controls to add new ADC channels. Will advise when it's back up.

 

  14387   Mon Jan 7 11:54:12 2019 JonConfigurationComputer Scripts / ProgramsVac system shutdown

I'm making a controlled shutdown of the vac controls to add new ADC channels. Will advise when it's back up.

  14386   Fri Jan 4 17:43:24 2019 gautamUpdateCDSTiming issues

[J Hanks (remote), koji, gautam]

Summary:

The problem stems from the way GPS timing signals are handled by the FEs and FB. We effected a partial fix:

  • Now, old frame data is no longer being overwritten
  • For the channels that are indeed being recorded now, the correct time stamp is being applied so they can be found in /frames by looking for the appropriate gpstime.

Details:

  • The usual FE/FB power cycling did not fix the problem.
  • The gps time used by FB and associated RT processes may be found by using  cat /proc/gps (i.e. this is different from the system time found by using date, or gpstime).
  • This was off by 2 years.
  • The way this worked up till now was by adding a fixed offset to this time.
    • This offset can be found as a line saying set symm_gps_offset=31622382 in daqdrc.fw (for example)
    • There were similar lines in daqdrc.rcv and daqdrc.dc - however, they were not all the same offset! We couldn't figure out why.
    • All these files live in /opt/rtcds/caltech/c1/target/daqd/.

Changes effected:

  1. First, we tried changing the offset in the daqdrc.fw file only.
    • Incremented it by 24*60*60*365 = number of seconds in a year with no leap seconds/days.
    • This did not fix the problem.
  2. So J Hanks decided to rebuild the Spectracom driver - these commands may not be comprehensive, but I think I got everything).
    • The relevant file is spectracomGPS.c (made a copy of /usr/src/symmetricom-3.3~rc1, called symmetricom-3.3~rc1-patched, this file is in /usr/src/symmetricom-3.3~rc1-patched/include/drv)
    • Added the following lines:
      /* 2018 had no leap seconds or leap days, so adjust for that */
             pHardware->gpsOffset += 31536000;
    • re-built and installed the modified symmetricom driver.
    • Checked that cat /proc/gps now yields the correct time.
    • Reset the gps time offsets in daqdrc.fw, daqdrc.rcv and daqdrc.dc to 0
    • With these steps, the frames were being written to /frames with the correct timestamp.
  3. Next, we checked the timing on the FEs
    • Basically, J Hanks rebuilt the version of the symmetricom driver that is used by the rtcds models to mimic the changes made for FB.
    • This did the trick for c1lsc and c1ioo - cat /proc/gps now returns the correct time on those FEs.
    • However, c1sus remains problematic (it initially reported a GPS time from 2 years ago, and even after the re-installed driver, is 4 days behind) - he suspects that this is because c1sus is the only FE with a Symmetricom/Spectracom card installed in the I/O chassis. So c1sus reports a gpstime that is ~4 days behind the "correct" gpstime.
    • He is going to check with Rolf/other CDS experts to figure out if it's okay for us to simply remove the card and run the models, or if we need to take other steps.
    • As part of this work, the c1x02 IOP model was recompiled, re-installed and re-started.

The realtime models were not restarted (although all the vertex FEs are running) - we can regroup next week and decide what is the correct course of action.

Quote:
 
  • Attachment #2 shows the minute trend of the pressure gauges for a 12 day period - it looks like there is some issue with the frame builder clock, perhaps this issue resurfaced? But checking the system time on FB doesn't suggest anything is wrong.. I double checked with dataviewer as well that the trends don't exist... But checking the status of the individual daqd processes indeed showed that the dates were off by 1 year, so I just restarted all of them and now the time seems correct. How can we fix this problem more permanently? Also, the P1b readout looks suspicious - why are there periods where it seems like we are reading values better than the LSB of the device?
  14385   Fri Jan 4 15:18:15 2019 KojiUpdateGeneralChiara disk clean up and internally mounted

[Koji Gautam]

Took the opprtunity of the power glitch to take care of the disk situation of chiara.

- Unmounted /cvs/cds from nodus. This did not affect the services on nodus as they don't use /cvs/cds

- Go to chiara, shut it down, and physically checked the labels of the drives.

root = 0.5TB
/cvs/cds = 4TB HGST
backup of /cvs/cds= 6TB HGST

- These three disks are internally mounted and connected with SATA. Previously, 6TB was on USB.

- There were two other drives (2TB and 3TB) but they seemed logically or physically broken. These two disks were removed from chiara. (they came back online after reformatting on mac. So they seem still physically alive).

controls@chiara|~> df
df: `/var/lib/lightdm/.gvfs': Permission denied
Filesystem      1K-blocks       Used  Available Use% Mounted on
/dev/sda1       461229088   10690932  427109088   3% /
udev             15915020          4   15915016   1% /dev
tmpfs             3185412        848    3184564   1% /run
none                 5120          0       5120   0% /run/lock
none             15927044        144   15926900   1% /run/shm
/dev/sdb1      5814346836 1783407788 3737912972  33% /media/40mBackup
/dev/sdc1      3845709644 1884187232 1766171536  52% /home/cds
controls@chiara|~> lsblk
NAME   MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
sda      8:0    0 465.8G  0 disk
├─sda1   8:1    0 446.9G  0 part /
├─sda2   8:2    0     1K  0 part
└─sda5   8:5    0  18.9G  0 part [SWAP]
sdb      8:16   0   5.5T  0 disk
└─sdb1   8:17   0   5.5T  0 part /media/40mBackup
sdc      8:32   0   3.7T  0 disk
└─sdc1   8:33   0   3.7T  0 part /home/cds
sr0     11:0    1  1024M  0 rom

- Rebooted the machine and just came back without any error. This time the control room machines were not shutdown, but they just recovered the NFS once chiara got back.

Attachment 1: P_20190104_143336.jpg
P_20190104_143336.jpg
Attachment 2: P_20190104_143357.jpg
P_20190104_143357.jpg
  14384   Fri Jan 4 11:06:16 2019 JonOmnistructureUpgradeVac System Punchlist

The base Acromag vacuum system is running and performing nicely. Here is a list of remaining questions and to-do items we still need to address.

Safety Issues

  • Interlock for HV supplies. The vac system hosts a binary EPICS channel that is the interlock signal for the in-vacuum HV supplies. The channel value is OFF when the main volume pressure is in the arcing range, 3 mtorr - 500 torr, and ON otherwise. Is there something outside the vacuum system monitoring this channel and toggling the HV supplies?
  • Exposed 30-amp supply terminals. The 30-amp output terminals on the back of the Sorensen in the vac rack are exposed. We need a cover for those.
  • Interlock for AC power loss. The current vac system is protected only from transient power glitches, not an extended loss. The digital system should sense an outage and put the IFO into a safe state (pumps spun down and critical valves closed) before the UPS battery is fully drained. However, it presently has no way of sensing when power has been lost---the system just continues running normally on UPS power until the battery dies, at which point there is a sudden, uncontrolled shutdown. Is it possible for the digital system to communicate directly with the UPS to poll its activation state?

Infrastructure Improvements

  • Install the new N2 tank regulator and high-pressure transducers (we have the parts; on desk across from electronics bench). Run the transducer signal wires to the Acromag chassis in the vacuum rack.
  • Replace the kludged connectors to the Hornet and SuperBee serial outputs with permanent ones (we need to order the parts).
  • Wire the position indicator readback on the manual TP1 valve to the Acromag chassis.
  • Add cable tension relief to the back of the vac rack.
  • Add the TP1 analog readback signals (rotation speed and current) to the digital system.  Digital temperature, current, voltage, and rotation speed signals have already been added for TP2 and TP3.
  • Set up a local vacuum controls terminal on the desk by the vac rack.
  • Remove gauges from the EPICS database/MEDM screens that are no longer installed or functional. Potential candidates for removal: PAN, PTP1, IG1, CC2, CC3, CC4.
  • Although it appeared on the MEDM screen, the RGA was never interfaced to the old vac system. Should it be connected to c1vac now?
  14383   Fri Jan 4 10:25:19 2019 JonOmnistructureVACN2 line valved off

Yes, for TP2 and TP3. They both have a small vent valve that opens automatically on shutdown.

Quote:

Independent question: Are all the turbo forelines vented automatically? We manually did it for the main roughing line.

 

 

  14382   Thu Jan 3 21:17:49 2019 ranaConfigurationComputersWorkstation Upgrade: Donatella -> Scientific Linux 7.2

donatella was one of our last workstations running ubuntu12. we installed SL7 on there today

  1. had to use a DVD; wouldn't boot from USB stick
  2. made sure to use userID=1001 and groupID=1001 at the initial install part
  3. went to the Keith Thorne LLO wiki on SL7
  4. The 'yum update' command failed due to a gstreamer conflict. I did "yum remove gstreamer1-plugins-ugly-free-1.10.4-3.el7.x86_64" and then it continued a bit more.
  5. Then there are ~20 errors related to gds-crtools that look like this:Error: Package: gds-crtools-2.18.12-1.el7.x86_64 (lscsoft-production) Requires: libMatrix.so.6.14()(64bit)

  6. I re-ran the yum install .... command using the --skip-broken command and that seemed to complete, although I guess the GDS stuff will not work.
  7. Installed: terminator, inconsolata-fonts, 
  8. Installed XFCE desktop as per K Thorne:  yum groupinstall "Xfce" -y
  9.  
Attachment 1: IMG_20190103_205158.jpg
IMG_20190103_205158.jpg
  14380   Thu Jan 3 15:08:37 2019 gautamOmnistructureVACVac status unknown

Larry W came by the 40m, and reported that there was a campus-wide power glitch (he was here to check if our networking infrastructure was affected). I thought I'd check the status of the vacuum.

  • Attachment #1 is a screenshot of the Vac overview MEDM screen. Clearly something has gone wrong with the modbus process(es). Only the PTP2 and PTP3 gauges seem to be communicative.
  • Attachment #2 shows the minute trend of the pressure gauges for a 12 day period - it looks like there is some issue with the frame builder clock, perhaps this issue resurfaced? But checking the system time on FB doesn't suggest anything is wrong.. I double checked with dataviewer as well that the trends don't exist... But checking the status of the individual daqd processes indeed showed that the dates were off by 1 year, so I just restarted all of them and now the time seems correct. How can we fix this problem more permanently? Also, the P1b readout looks suspicious - why are there periods where it seems like we are reading values better than the LSB of the device?

I decided to check the systemctl process status on c1vac:

controls@c1vac:~$ sudo systemctl status modbusIOC.service
● modbusIOC.service - ModbusIOC Service via procServ
   Loaded: loaded (/etc/systemd/system/modbusIOC.service; enabled)
   Active: active (running) since Thu 2019-01-03 14:53:49 PST; 11min ago
 Main PID: 16533 (procServ)
   CGroup: /system.slice/modbusIOC.service
           ├─16533 /usr/bin/procServ -f -L /opt/target/modbusIOC.log -p /run/...
           ├─16534 /opt/epics/modules/modbus/bin/linux-x86_64/modbusApp /opt/...
           └─16582 caRepeater

Jan 03 14:53:49 c1vac systemd[1]: Started ModbusIOC Service via procServ.

Warning: Unit file changed on disk, 'systemctl daemon-reload' recommended.

So something did happen today that required restart of the modbus processes. But clearly not everything has come back up gracefully. A few lines of dmesg (there are many more segfaults):

[1706033.718061] python[23971]: segfault at 8 ip 000000000049b37d sp 00007fbae2b5fa10 error 4 in python2.7[400000+31d000]
[1706252.225984] python[24183]: segfault at 8 ip 000000000049b37d sp 00007fd3fa365a10 error 4 in python2.7[400000+31d000]
[1720961.451787] systemd-udevd[4076]: starting version 215
[1782064.269844] audit: type=1702 audit(1546540443.159:38): op=linkat ppid=21820 pid=22823 auid=4294967295 uid=1000 gid=1000 euid=1000 suid=1000 fsuid=1000 egid=1000 sgid=1000 fsgid=1000 tty=pts0 ses=4294967295 comm="git" exe="/usr/bin/git" res=0
[1782064.269866] audit: type=1302 audit(1546540443.159:39): item=0 name="/cvs/cds/caltech/target/c1vac/.git/objects/85/tmp_obj_uAXhPg" inode=173019272 dev=00:21 mode=0100444 ouid=1001 ogid=1001 rdev=00:00 nametype=NORMAL
[1782064.365240] audit: type=1702 audit(1546540443.255:40): op=linkat ppid=21820 pid=22823 auid=4294967295 uid=1000 gid=1000 euid=1000 suid=1000 fsuid=1000 egid=1000 sgid=1000 fsgid=1000 tty=pts0 ses=4294967295 comm="git" exe="/usr/bin/git" res=0
[1782064.365271] audit: type=1302 audit(1546540443.255:41): item=0 name="/cvs/cds/caltech/target/c1vac/.git/objects/58/tmp_obj_KekHsn" inode=173019274 dev=00:21 mode=0100444 ouid=1001 ogid=1001 rdev=00:00 nametype=NORMAL
[1782064.460620] audit: type=1702 audit(1546540443.347:42): op=linkat ppid=21820 pid=22823 auid=4294967295 uid=1000 gid=1000 euid=1000 suid=1000 fsuid=1000 egid=1000 sgid=1000 fsgid=1000 tty=pts0 ses=4294967295 comm="git" exe="/usr/bin/git" res=0
[1782064.460652] audit: type=1302 audit(1546540443.347:43): item=0 name="/cvs/cds/caltech/target/c1vac/.git/objects/cb/tmp_obj_q62Pdr" inode=173019276 dev=00:21 mode=0100444 ouid=1001 ogid=1001 rdev=00:00 nametype=NORMAL
[1782064.545449] audit: type=1702 audit(1546540443.435:44): op=linkat ppid=21820 pid=22823 auid=4294967295 uid=1000 gid=1000 euid=1000 suid=1000 fsuid=1000 egid=1000 sgid=1000 fsgid=1000 tty=pts0 ses=4294967295 comm="git" exe="/usr/bin/git" res=0
[1782064.545480] audit: type=1302 audit(1546540443.435:45): item=0 name="/cvs/cds/caltech/target/c1vac/.git/objects/e3/tmp_obj_gPI4qy" inode=173019277 dev=00:21 mode=0100444 ouid=1001 ogid=1001 rdev=00:00 nametype=NORMAL
[1782064.640756] audit: type=1702 audit(1546540443.527:46): op=linkat ppid=21820 pid=22823 auid=4294967295 uid=1000 gid=1000 euid=1000 suid=1000 fsuid=1000 egid=1000 sgid=1000 fsgid=1000 tty=pts0 ses=4294967295 comm="git" exe="/usr/bin/git" res=0
[1783440.878997] systemd[1]: Unit serial_TP3.service entered failed state.
[1784682.147280] systemd[1]: Unit serial_TP2.service entered failed state.
[1786407.752386] systemd[1]: Unit serial_MKS937b.service entered failed state.
[1792371.508317] systemd[1]: serial_GP316a.service failed to run 'start' task: No such file or directory
[1795550.281623] systemd[1]: Unit serial_GP316b.service entered failed state.
[1796216.213269] systemd[1]: Unit serial_TP3.service entered failed state.
[1796518.976841] systemd[1]: Unit serial_GP307.service entered failed state.
[1796670.328649] systemd[1]: serial_Hornet.service failed to run 'start' task: No such file or directory
[1797723.446084] systemd[1]: Unit serial_MKS937b.service entered failed state.

 

I don't know enough about the new system so I'm leaving this for Jon to debug. Attachment #3 shows that the analog readout of the P1 pressure gauge suggests that the IFO is still under vacuum, so no random valve openings were effected (as expected, since we valved off the N2 line for this very purpose).

Attachment 1: Screenshot_from_2019-01-03_15-19-51.png
Screenshot_from_2019-01-03_15-19-51.png
Attachment 2: Screenshot_from_2019-01-03_15-14-14.png
Screenshot_from_2019-01-03_15-14-14.png
Attachment 3: 997B13A9-CAAF-409C-A6C2-00414D30A141.jpeg
997B13A9-CAAF-409C-A6C2-00414D30A141.jpeg
  14379   Fri Dec 21 12:57:10 2018 KojiOmnistructureVACN2 line valved off

Independent question: Are all the turbo forelines vented automatically? We manually did it for the main roughing line.

 

  14377   Fri Dec 21 11:13:13 2018 gautamOmnistructureVACN2 line valved off

Per the discussion yesterday, I valved off the N2 line in the drill press room at 11 am PST today morning so as to avoid any accidental software induced gate-valve actuation during the holidays. The line pressure is steadily dropping...

Attachment #1 shows that while the main volume pressure was stable overnight, the the pumpspool pressure has been steadily rising. I think this is to be expected as the turbo pumps aren't running and the valves can't preserve the <1mtorr pressure over long timescales?

Attachment #2 shows the current VacOverview MEDM screen status.

Attachment 1: VacGauges.png
VacGauges.png
Attachment 2: Screenshot_from_2018-12-21_13-02-06.png
Screenshot_from_2018-12-21_13-02-06.png
  14376   Fri Dec 21 11:11:51 2018 gautamUpdateCDSLogging of new Vacuum channels

The N2 pressure channel name was also wrong in C0EDCU.ini, so I updated it this morning to the correct name and units:

[C1:Vac-N2_pressure]
units=psi

Now it too is being recorded to frames.

  14375   Thu Dec 20 21:29:41 2018 JonOmnistructureUpgradeVacuum Controls Switchover Completed

[Jon, Chub, Koji, Gautam]

Summary

Today we carried out the first pumpdown with the new vacuum controls system in place. It performed well. The only problem encountered was with software interlocks spuriously closing valves as the Pirani gauges crossed 1E-4 torr. At that point their readback changes from a number to "L OE-04, " which the system interpreted as a gauge failure instead of "<1E-4." This posed no danger and was fixed on the spot. The main volume was pumped to ~10 torr using roughing pumps 1 and 3. We were limited only by time, as we didn't get started pumping the main volume until after 1pm. The three turbo pumps were also run and tested in parallel, but were isolated to the pumpspool volume. At the end of the day, we closed every pneumatic valve and shut down all five pumps. The main volume is sealed off at ~10 torr, and the pumpspool volume is at ~1e-6 torr. We are leaving the system parked in this state for the holidays. 

Main Volume Pumpdown Procedure

In pumping down the main volume, we carried out the following procedure.

  1. Initially: All valves closed (including manual valves RV1 and VV1); all pumps OFF.
  2. Manually connected roughing pump line to pumpspool via KF joint.
  3. Turned ON RP1 and RP2.
  4. Waited until roughing pump line pressure (PRP) < 0.5 torr.
  5. Opened V3.
  6. Waited until roughing pump line pressure (PRP) < 0.5 torr.
  7. Manually opened RV1 throttling valve to main volume until pumpdown rate reached ~3 torr/min (~3 hours on roughing pumps).
  8. Waited until main volume pressure (P1a/P1b) < 0.5 torr.

We didn't quite reach the end of step 8 by the time we had to stop. The next step would be to valve out the roughing pumps and to valve in the turbo pumps.

Hardware & Channel Assignments

All of the new hardware is now permanently installed in the vacuum rack. This includes the SuperMicro rack server (c1vac), the IOLAN serial device server, a vacuum subnet switch, and the Acromag chassis. Every valve/pump signal cable that formerly connected to the VME bus through terminal blocks has been refitted with a D-sub connector and screwed directly onto feedthroughs on the Acromag chassis.

The attached pdf contains the master list of assigned Acromag channels and their wiring.

Attachment 1: 40m_vacuum_acromag_channels.pdf
40m_vacuum_acromag_channels.pdf 40m_vacuum_acromag_channels.pdf 40m_vacuum_acromag_channels.pdf
  14374   Thu Dec 20 17:17:41 2018 gautamUpdateCDSLogging of new Vacuum channels

Added the following channels to C0EDCU.ini:

[C1:Vac-P1b_pressure]
units=torr
[C1:Vac-PRP_pressure]
units=torr
[C1:Vac-PTP2_pressure]
units=torr
[C1:Vac-PTP3_pressure]
units=torr
[C1:Vac-TP2_rot]
units=kRPM
[C1:Vac-TP3_rot]
units=kRPM

Also modified the old P1 channel to

[C1:Vac-P1a_pressure]
units=torr

Unfortunately, we realized too late that we don't have these channels in the frames, so we don't have the data from this test pumpdown logged, but we will have future stuff. I say we should also log diagnostics from the pumps, such as temperature, current etc. After making the changes, I restarted the daqd processes.


Things to add to ASA wiki page once the wiki comes back online:

  1. What is the safe way to clean the cryo pump if we want to use it again?
  2. What are safe conditions to turn the RGA on?
  14373   Thu Dec 20 10:28:43 2018 gautamUpdateVACHeavy doors back on for pumpdown 82

[Chub, Koji, Gautam]

We replaced the EY and IOO chamber heavy doors by 10:10 am PST. Torquing was done first oen round at 25 ft-lb, next at 45 ft-lb (we trust the calibration on the torque wrench, but how reliable is this? And how important are these numbers in ensuring a smooth pumpdown?). All went smooth. The interior of the IOO chamber was found to be dirty when Koji ran a wipe along some surfaces.

For this pumpdown, we aren't so concerned with having the IFO in an operating state as we will certainly vent it again early next year. So we didn't follow the full close-up checklist.

Jon and Chub and Koji are working on starting the pumpdown now... In order to not have to wear laser safety goggles while we closed doors and pumped down, I turned off all the 1064nm lasers in the lab.

  14372   Thu Dec 20 08:38:27 2018 JonUpdateVACPumpdown tomorrow

Linked is the pumpdown procedure, contained in the old 40m documentation. The relevant procedure is "All Off --> Vacuum Normal" on page 11.

Quote:

I just spoke to Jon who asked me to make this elog - we will be ready to test one or more parts of the pumpdown procedure tomorrow (12/20), so we should proceed as planned to put the heavy doors back on EY and OMC chambers at 9am tomorrow morning. Jon will circulate a more detailed procedure about the pumpdown steps later today evening.

 

  14371   Wed Dec 19 22:11:28 2018 KojiUpdateGeneralHow to align the copper OMC

The OMC input optics layout is attached

Checked the spot position on OMMT-FM1. It was off from the center. This was causing the spot on OMMT1 off-center. This was fixed by the steering mirror for the AUX laser.

The beam alignment onto the OMC was tweaked with OMC-SM1 and OMC-SM2. This was the painful part. We had to make a sensor card that could get in to the narrow space of the OMC. (Attachment 2 right)

Attachment 2 left shows the naming convention of the OMC mirrors.

For the alignment, we gave 5Vpp trig waves at 3.1Hz to the input of the PZT amp so that the cavity is kept scanned continuously. Firstly check the rough spot positions for OMC-CM1 and OMC-CM2. If you carefully use the card, you can check if the beam is returning to OMC-IC. This return beam should have roughtly same hight as the incident beam. This can be adjusted by either of the steering mirrors.

Once the beam is going around the mirrors multiple times, the spot alignment can be checked at OMC-CM1. Bring a card right in front of CM1. If the card is lifter slightly above the incident spot, this automatically allows for the outgoing beam to go through. Depending on the pitch alignment, the next roundtrip (1RT) will be seen on the card. As you lift the card up more, you will be able to see more round trip beams (e.g. 2RT, 3RT, in the figure). If the yaw alignment is perfect, these spots would be lined up vertically. So you can try to align the horizontal direction with the steering mirrors. Then the vertical alignment can be done with the pitch knobs.

At this point you should be able to see some super high-order transmission at the OMC trans. For today, we stopped here as we already ran out of the knob ranges at multiple knobs. This is because the beam height in the mode matching telescope was not right, and the steering mirrors had to work more than their range.

Attachment 1: 110804_40m_OMC_layout.pdf
110804_40m_OMC_layout.pdf
Attachment 2: OMC_alignment.pdf
OMC_alignment.pdf
  14370   Wed Dec 19 21:14:50 2018 gautamUpdateVACPumpdown tomorrow

I just spoke to Jon who asked me to make this elog - we will be ready to test one or more parts of the pumpdown procedure tomorrow (12/20), so we should proceed as planned to put the heavy doors back on EY and OMC chambers at 9am tomorrow morning. Jon will circulate a more detailed procedure about the pumpdown steps later today evening.

  14369   Wed Dec 19 19:51:19 2018 gautamUpdateGeneralPumpdown prep

[Koji, gautam]

Summary:

We are ready to put the heavy doors back on the chambers and do some test pumpdowns tomorrow morning if Jon gives us the go-ahead. Also, Koji made the OMC resonate some of the AUX beam light we send into ityes

Details:

  1. EY work:
    • IMC was locked, and we attempted to locate the beam with an IR card inside the chamber.
    • Koji found that the beam was too high, we were over-shooting the entire black-glass baffle on the EY table.
    • So I moved the TTs to try and center the beam through the aperture of aforementioned baffle.
    • Once this was done, we found that the beam was misaligned in yaw by ~1-inch in transmission on the EY optics table (there was an iris in place marking the cavity transmission axis). This explains why I couldn't find any TRY flashes while moving the TTs around.
    • We hypothesize that without the 2 degree ETM wedge in place, there isn't a compatible axis for the ITM transmission to also make it through the EY baffle and transmission iris. Over ~1m, the 2 degree wedge makes roughly 1.4 inch translation in yaw, so this seems to be a plausible hypothesis.
    • The ETMY suspension was moved from the mini-cleanroom setup back into the EY vacuum chamber. Two clamps (finger tightened only) hold it in place on the NE edge of the optical table. We decided that this is a better resting palce for the cage over the holidays than an in-air cleanroom.
  2. OMC chamber work:
    • While we were in clean garb, we decided to also investigate the OMC situation a bit.
    • It quickly became apparent that it was hopeless for me to work in chamber in the tightly confined IOO chamber. So Koji went in to have a look.
    • Koji will post the detailed alignment procedure - but after some alignment of the AUX laser input beam axis using in air steering mirrors and Koji's expert tweaking of the pointing into the OMC, we observed some resonances of the OMC.
    • Attachment #1 shows the full-range triangle ramp applied to the OMC length PZT (top row) and the OMC REFL signal (bottom row), measured using a PDA520 (chosen for its large active area) connected to a scope (AC-coupled, 1Mohm impedance, averaged to make the dips more prominent).
    • The OMC transmission was also (barely) visible on an IR card.
    • So the OMC length PZT seems capable of sweeping the length of the cavity. Based on the size of the dips we saw, the MM into the cavity is sub 1-percent.
    • The transmission PDs didn't output any measurable signal - but I'm not sure that the satellite box / readout electronics have been carefully characterized on the electroncis bench, so that will have to be done first.
    • We replaced the copper cover of the OMC (finger tightened for now) in case we do any test pumpdowns tomorrow. HV supply has been turned off, and the AUX laser has been reverted to standby mode.
Attachment 1: OMCscan.pdf
OMCscan.pdf
  14368   Wed Dec 19 15:15:56 2018 gautamUpdateIOOTT1/TT2 stepping

I removed the ND filter from the ETMYT camera to facilitate searching for a TRY beam. This should be replaced before we go back to high power.

  14367   Wed Dec 19 14:19:15 2018 KojiSummaryVACPlan for pumpoing down test

We still need elaborated test procedure posted

12/29 Wed

  • Jon continues to work on valve actuator tests.
  • Chub continues to work on wiring / fixing wiring.
  • At the end of the day Jon is going to send out a notification email of "GO"/"NO GO" for pumping.

 

12/30 Thu

  • 9AM: Start closing two doors unless Jon gives us NO GO sign.
  • 10AM: Start pumping down
    • Test roughing pump capability via new control system
    • (Independently) Test turbo rotating procedure. This time we will not open the gate valve between the TP1 and the main volume. This is because we want to take care of the backing turbo loads while we gradually open the gate valve. This will take more hours to be done and we will not be able to finish this test by the end of Thu.
    • At the end of the procedure, we isolate the main volume, stop all the pumps, and vent the roghing pumps to save them from the oil backstream.

gautam: Koji and I were just staring at the vacuum screen, and realized that the drypumps, which are the backing pumps for TP2 and TP3, are not reflected on the MEDM screen. This should be rectified.

Steve also mentioned that the new small turbo controller does not directly interface with the drypump. So we need some system to delay the starting of the turbo itself, once the drypump has been engaged. Does this system exist?

Attachment 1: Screenshot_from_2018-12-19_14-49-34.png
Screenshot_from_2018-12-19_14-49-34.png
  14366   Wed Dec 19 00:12:46 2018 gautamUpdateOMC40m OMC DCC node

I made a node to collect drawings/schematics for the 40m OMC, added the length drive for now. We should collect other stuff (TT drivers, AA/AI, mechanical drawings etc) there as well for easy reference.

Some numbers FTR:

  • OMC length PZT capacitance was measured to be 209 nF.
  • Series resistance in HV path of OMC lenght PZT driver is 10 kohms, so this creates a LP with corner 1/2/pi/10kohm/200nF ~80 Hz.
  • Per Rob's thesis, the length PZT has DC actuation coefficient of 8.3 nm/V, ∼ 2 µm range. 
  14365   Tue Dec 18 18:13:32 2018 aaronUpdateOMCOMC L HV Piezo driver tests (again)

I tested the OMC-L HV driver box again, and made the following observations:

  • Drove the HV diff pins (2,7) with a 5V triangle wave
    • Observed that with a ~0.4V offset on the drive, the HV output (measured directly with a 10x probe) has a 0-(almost)200V triangle wave (for 200V HV in), saturating near 200V and near 0V somewhat before reaching the full range of the triangle
    • The HV mon gives the same answer as measuring the HV output directly, and is reduced 100x compared to the HV output.
    • At 1Hz and above, the rolloff of the low pass still attenuates the drive a bit, and we don't reach the full range.
  • Drove the HV dither pins (1,6) with a 100mV to 10V triangle wave, around 15kHz
    • Even at 10V, the dithering is near the noise of the mon channel, so while I could see a slight peak changing on the FFT near the dither frequency, I couldn't directly observe this on a scope using the mon channel
    • However, measuring the HV directly I do see the dither applied on top of the HV signal. The amplitude of the dither is the same on the HV output as on the dither drive.

[gautam, aaron]

We searched for blips while nominally scanning the OMC length.

We sent a 0.1Hz, 10Vpp triangle wave to the OMC piezo drive diff channels, so the piezo length is seeing a slow triangle wave from 0-200V.

Then, we applied a ~15kHz dither to the OMC length. This dither is added directly onto the HV signal, so the amplitude of the dither at the OMC is the same as the amplitude of the dither into the HV driver.

We monitored the OMC REFL signal (where we saw no blips yesterday) and mixed this with the 15kHz dither signal to get an error signal. Gautam found a pomona box with a low pass filter, so we also low passsed to get rid of some unidentified high frequency noise we were seeing (possibly a ground loop at the function generator? it was present with the box off, but gone with the AC line unplugged). [So we made our own lock-in amplifier.] Photo attached.

We tested the transfer function of the LP, and finding it at 100kHz rather than the advertised 10kHz, we opened the box, removed a resistor to change the 3dB back to 10kHz, and confirmed this by measuring the TF.

We didn't see flashes of error signal in the mixed reflection either, so we suspect that either the PZT is not actuating on the OMC or the alignment is bad. Based on what appears to be the shimmering of far-misaligned fringes on the AS camera, Aaron's suspicion from aligning the cavity with the card, and the lack of flashes, we suspect the alignment. To avoid being stymied by a malfunctioning PZT, we can scan the laser frequency next time rather than the PZT length.

Attachment 1: IMG_4576_copy.jpg
IMG_4576_copy.jpg
  14364   Tue Dec 18 11:42:40 2018 ChubUpdateGeneralAcromag box wired

The Auxiliary DAQ Chassis, or Acromag box, is now wired and ready for testing.  I will be sorting the cables at the vacuum rack to make connection to the box easier.

 

  14363   Mon Dec 17 20:45:40 2018 aaronUpdateOMCAligning the OMC

[gautam, aaron]

We did work in the OMC chamber today to get the OMC aligned. Aaron was in the clean suit while Gautam steered in-air optics. We modified the aux input steering optics and the final two OMC steering optics (between OMMT and OMC), but did not modify any of the AS path optics.

I had already aligned AUX approximately into the AS port from the AP table. With the OMC N door open, we aligned the aux beam first to OM6, then to OMPO, then OM5. OM5 was the last optic in the OMC chamber that we could align to.

From there, Gautam found the aux beam clipping on a few optics on its way to SR4 using the IR viewer. Once we were approximately hitting SR4, we got a return beam in the OMC chamber, which we were able to coalign with the input aux beam.

We had already done the alignment of SR5 into the OMC during the last vent, so we immediately had a refl off of the OMC, which we aligned onto a PD520 from the PSL table (larger aperture than the previous PD, which anyway needed a macroscopic adjustment to catch the refl beam).

Next, we removed the OMC cover, wrapped it in foil, and placed it in the makeshift clean room near the Y end. The screws remain in a foil bucket in the OMC chamber. With the cover off, Aaron moved the OMC input steering mirrors to align the beam in the OMC. We measured ~2.4mW in the OMC refl beam, which means about 240uW is transmitted into the OMC. Aaron thinks the beam overlaps itself after one round trip in the cavity, but that the entire plane may be too low in pitch, so more alignment may be needed here.

With the beam approximately aligned into the OMC, we energized the OMC-L piezo driver with 200V, and applied a ~0.03Hz triangle wave on the OMC diff input (pins 2-7). We monitor the REFL PD, piezo mon, function generator signal, and one of the trans PDs. We noticed that the PZT mon shows the driver saturating before the function generator reaches its full +-10V, which is something to investigate.

We saw what could have been regular dips in the REFL PD signal, but realized that with an unkown level of mode matching, it will be hard to tell whether the light becomes resonant with the DC signal. Gautam has suggested coaligning the aux and PSL beams, then observing the PDH signal from the PSL beam as the OMC sweeps through resonance, while turning aux back on anytime we try to make adjustments to the alignment of the OMC (so I can see the beam in the cavity).

I'll think through the plan in some more detail and we will try to have the OMC locked tomorrow.


gautam:

  1. All references to SR4 etc actually refer to OM4 etc.
  2. For this experiment, we are using the prompt reflection of the AUX beam from the HR surface of the SRM, so as to get maximum light into the cavity.
  3. For 2.4 mW incident on OMC1 (actually we measured the REFL beam on the AS table), we expect ~24 uW inside the cavity, which isn't a lot but still was visible on the card.
  4. After this work, I checked the IMC alignment - it was still easily able to lock to a TEM00 mode, but the transmission was ~half (i.e. ~600cts) of what I am used to it being in low power mode (~1300 cts). I didn't align the cavity to the input beam, as I think in this case, the right thing to do is to align the input beam to the IMC cavity axis.
Attachment 1: IMG_5922.JPG
IMG_5922.JPG
Attachment 2: IMG_5921.JPG
IMG_5921.JPG
  14362   Sat Dec 15 20:04:03 2018 gautamUpdateIOOTT1/TT2 stepping

I'm running a script that moves TT1 and TT2 randomly in some restricted P/Y space to try and find an alignment that gets some light onto the TRY PD. Test started at gpstime 1228967990, should be done in a few hours. The IMC has to remain locked for the duration of this test. I will close the PSL shutter once the test is done. Not sure if the light level transmitted through the ITM, which I estimate to be ~30uW, will be enough to show up on the TRY PD, but worth a shot I figure.

Test was completed and PSL shutter was closed at 1228977122.

  14361   Sat Dec 15 18:29:53 2018 KojiUpdateGeneralChiara new USB 4TB DIsk

The local backup was done at 18:18 after 11h18m of running.

2018-12-15 07:00:01,699 INFO       Updating backup image of /cvs/cds
2018-12-15 18:17:56,378 INFO       Backup rsync job ran successfully, transferred 5717707 files.

 

  14360   Fri Dec 14 22:19:22 2018 KojiUpdateGeneralChiara new USB 4TB DIsk

Edit: It was not 4TB disk but 6TB disk in fact. (We actually ordered 4TB disk...)

I think the problem of the backup disk was the flaky power supply for the external drive.
I swapped the drive to a new HGST 4TB one, but it was neither recognized nor spun up with the external power supply we had. So I decided to put both the new and old drives in the PC chassis to power them up with the internal power supply. I tested the old disk via a USB-SATA cable. However, this disk was not recognized. I noticed that the disk was not HGST 4TB but Seagate 3TB. Is it possible? I thought it was 4TB... Did I miss something?

Once the new 4TB was connected to the USB-SATA, it was very smooth to get it mounted. Now the disk is mounted as /media/40mBackup as before. /etc/fstab was also modified with the new UUID. All the command logs are found here below.

Let's see how the morning backup goes. It would take a while to copy everything on the new disk. So it was actually very nice to set this disk up by Friday midnight.


controls@chiara|~> lsblk

NAME   MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
sda      8:0    0 465.8G  0 disk 
---- sda1   8:1    0 446.9G  0 part /
---- sda2   8:2    0     1K  0 part 
---- sda5   8:5    0  18.9G  0 part [SWAP]
sdb      8:16   0   1.8T  0 disk 
---- sdb1   8:17   0   1.8T  0 part 
sdc      8:32   0   3.7T  0 disk 
---- sdc1   8:33   0   3.7T  0 part /home/cds
sr0     11:0    1  1024M  0 rom  
sdd      8:64   0   5.5T  0 disk 

controls@chiara|~> sudo mkfs -t ext4 /dev/sdd1

mke2fs 1.42 (29-Nov-2011)
Filesystem label=
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
Stride=0 blocks, Stripe width=0 blocks
183144448 inodes, 1465130385 blocks
73256519 blocks (5.00%) reserved for the super user
First data block=0
Maximum filesystem blocks=4294967296
44713 block groups
32768 blocks per group, 32768 fragments per group
4096 inodes per group
Superblock backups stored on blocks: 
    32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208, 
    4096000, 7962624, 11239424, 20480000, 23887872, 71663616, 78675968, 
    102400000, 214990848, 512000000, 550731776, 644972544
Allocating group tables: done                            
Writing inode tables: done                            
Creating journal (32768 blocks): done
Writing superblocks and filesystem accounting information: done      

controls@chiara|~> blkid

/dev/sda1: UUID="972db769-4020-4b74-b943-9b868c26043a" TYPE="ext4" 
/dev/sda5: UUID="a3f5d977-72d7-47c9-a059-38633d16413e" TYPE="swap" 
/dev/sdc1: UUID="92dc7073-bf4d-4c58-8052-63129ff5755b" TYPE="ext4" 
/dev/sdd1: UUID="1843f813-872b-44ff-9a4e-38b77976e8dc" TYPE="ext4" 

controls@chiara|~> sudo emacs -nw /etc/fstab
controls@chiara|~> cat /etc/fstab

# /etc/fstab: static file system information.
#
# Use 'blkid -o value -s UUID' to print the universally unique identifier
# for a device; this may be used with UUID= as a more robust way to name
# devices that works even if disks are added and removed. See fstab(5).
#
# <file system> <mount point>   <type>  <options>       <dump>  <pass>
proc            /proc           proc    nodev,noexec,nosuid 0       0
# / was on /dev/sda1 during installation
UUID=972db769-4020-4b74-b943-9b868c26043a /               ext4    errors=remount-ro 0       1
# swap was on /dev/sda5 during installation
UUID=a3f5d977-72d7-47c9-a059-38633d16413e none            swap    sw              0       0
#UUID="90a5c98a-22fb-4685-9c17-77ed07a5e000"    /media/40mBackup       ext4      defaults,relatime,commit=60       0         0
UUID="1843f813-872b-44ff-9a4e-38b77976e8dc"    /media/40mBackup       ext4      defaults,relatime,commit=60       0         0

#fb:/frames      /frames nfs     ro,bg


UUID=92dc7073-bf4d-4c58-8052-63129ff5755b   /home/cds    ext4    defaults,relatime,commit=60    0   0

controls@chiara|~> sudo mount -a
controls@chiara|~> df

Filesystem      1K-blocks       Used  Available Use% Mounted on
/dev/sda1       461229088   10694700  427105320   3% /
udev             15915020         12   15915008   1% /dev
tmpfs             3185412        868    3184544   1% /run
none                 5120          0       5120   0% /run/lock
none             15927044        484   15926560   1% /run/shm
/dev/sdc1      3845709644 1809568856 1840789912  50% /home/cds
/dev/sdd1      5814346836     190408 5521130352   1% /media/40mBackup
  14359   Fri Dec 14 14:25:36 2018 KojiUpdateCDSchiara backup

fsck of chiara backup disk (UUID="90a5c98a-22fb-4685-9c17-77ed07a5e000") was done. But this required many files to be fixed. So the backed-up files are not reliable now.
On the top of that, the disk became not recognized from the machine.

I went to the disk and disconnected the USB and then the power supply, which was/is connected to the UPS.
Then they are reconnected again. This made the disk came back as /media/90a5c98a-22fb-4685-9c17-77ed07a5e000. (*)
After unmounting this disk, I ran "sudo mount -a" to follow the way of mounting as fstab does.
Now I am running the backup script manually so that we can pretend to maintain a snapshot of the day at least.

(*) This is the same situation we found at the recovery from the power shutdown. So my hypothesis is that on Oct 16 at 7 AM during the backup there was a USB failure or disk failure or something which unmounted the disk. This caused some files got damaged. Also this caused the disk mounted as /media/90a5c98a-22fb-4685-9c17-77ed07a5e000. So since then, we did not have the backup.
Update (20:00): The disk connection failed again. I think this disk is no longer reliable.

 

Attachment 1: fsck_log.log
sudo fsck -yV UUID="90a5c98a-22fb-4685-9c17-77ed07a5e000"           [238/276]
[sudo] password for controls:
fsck from util-linux 2.20.1
[/sbin/fsck.ext4 (1) -- /media/40mBackup] fsck.ext4 -y /dev/sde1
e2fsck 1.42 (29-Nov-2011)
/dev/sde1 contains a file system with errors, check forced.
Pass 1: Checking inodes, blocks, and sizes
Error reading block 527433852 (Attempt to read block from filesystem resulted in
 short read) while getting next inode from scan.  Ignore error? yes

... 283 more lines ...
  14358   Fri Dec 14 13:06:12 2018 aaronUpdateOMCAligning the OMC

I replaced the 2'' AUX-AS combining BS with a freshly mounted 2'' HR mirror for 1064. The mirror is labelled 'Y1-2037-45-P', and had a comment on its case: 'V'. I aligned the AUX beam from the new HR mirror into the next iris, so AUX passes through irises both before and after the new optic. Now, AS does not go out to the AS PDs.

I mounted the old BS on the SP table in a random orientation.

I also dumped the beam transmitted through one of the AUX steering mirrors before the new HR mirror.

  14357   Fri Dec 14 13:02:24 2018 aaronUpdateOMCAligning the OMC

I replaced the 2'' AUX-AS combining BS with a 2'' HR mirror for 1064. I aligned the AUX beam from the new HR mirror into the next iris, so AUX passes through irises both before and after the new optic. Now, AS does not go out to the AS PDs.

I mounted the old BS on the SP table in a random orientation.

I also dumped the beam transmitted through one of the AUX steering mirrors before the new HR mirror.

  14356   Thu Dec 13 22:56:28 2018 gautamUpdateCDSFrames

[koji, gautam]

We looked into the /frames situation a bit tonight. Here is a summary:

  1. We have already lost some second trend data since the new FB has been running from ~August 2017.
  2. The minute trend data is still safe from that period, we believe.
  3. The Jetstor has ~2TB of trend data in the /frames/trend folder.
    • This is a combination of "second", "minute_raw" and "minute".
    • It is not clear to us what the distinction is between "minute_raw" and "minute", except that the latter seems to go back farther in time than the former.
    • Even so, the minute trend folder from October 2011 is empty - how did we manage to get the long term trend data?? From the folder volumes, it appears that the oldest available trend data is from ~July 24 2015.

Plan of action:

  1. The wiper script needs to be tweaked a bit to allow more storage for the minute trends (which we presumably want to keep for long term).
  2. We need to clear up some space on FB1 to transfer the old trend data from Jetstor to FB1.
  3. We need to revive the data backup via LDAS. Also summary pages.

BTW - the last chiara (shared drive) backup was October 16 6 am. dmesg showed a bunch of errors, Koji is now running fsck in a tmux session on chiara, let's see if that repairs the errors. We missed the opportunity to swap in the 4TB backup disk, so we will do this at the next opportunity.

  14355   Thu Dec 13 22:36:42 2018 aaronUpdateOMCAligning the OMC

I turned on AUX, and aligned the aux beam to be centered on the first optic the AS beam sees on the AP table. I then turned off the AUX laser.

  14354   Thu Dec 13 22:24:21 2018 aaronUpdateOMCOMC channels

I completed testing of the AI board mentioned above. In addition to the blown fuse, there were two problems:

  • A was a large drop of solder splattered on some of the ch 1 ICs, which is why we couldn't maintain any voltage. I removed the solder.
  • The +12V wire from the power board to the AI board was loose, so I removed and replaced that crimp connection

After this, I tested the TF of all channels. For the most part, I found the expected 3rd order ~7500Hz cheby with notches at ~16kHz and 32kHz. However, some of the channels had shallower or deeper notches. By ~32kHz, I was below the resolution on the spectrum analyzer. Perhaps I just have nonideal settings? I'll attach a few representative examples.

I reinstalled the chassis at 1X2, but haven't connected power.

 

  14353   Thu Dec 13 20:10:08 2018 KojiUpdateGeneralPower Outage recovery

[Gautam, Aaron, Koji]

The PSL interlock system was fixed and now the 40m lab is laser hazard as usual.


- The schematic diagram of the interlock system D1200192
- We have opened the interlock box. Immediately we found that the DC switching supply (OMRON S82K-00712) is not functioning anymore.  (Attachment #1)
- We could not remove the module as the power supply was attached on the DIN rail. We decided to leave the broken supply there (it is still AC powered with no DC output).

- Instead, we brought a DC supply adapter from somewhere and chopped the head so that we can hook it up on the crimping-type quick connects. In Attachment #1, the gray is +12V, and the orange and black lines are GND.

- Upon the inspection, the wires of the "door interlock reset button" fell off and the momentary switch (GRAYHILL 30-05-01-502-03) got broken. So it was replaced with another momentary swicth, which is way smaller than the original unfortunately. (Attachments 2 and 3)

- Once the DC supply adapter was pluged to an AC tap, we heard the sounds of the relays working, and we recovered the laser hazard lamps, PSL door alerm lamps. Also it was confirmed that the PSL innolight is operatable now. 

- BTW, there is the big switch box on the wall close to the PSL enclosure. Some of the green lamps were gone. We found that we have plenty of spare lamps and relays inside of the box. So we replaced the bulbs and know the A.C. lights are functioning. (Attachments 4 & 5)

Attachment 1: OMRON_S82K-00712.JPG
OMRON_S82K-00712.JPG
Attachment 2: reset_button_repaired1.JPG
reset_button_repaired1.JPG
Attachment 3: reset_button_repaired2.JPG
reset_button_repaired2.JPG
Attachment 4: gray_box.JPG
gray_box.JPG
Attachment 5: gray_box2.JPG
gray_box2.JPG
  14352   Thu Dec 13 18:12:47 2018 gautamUpdateIOOND filter on AS camera changed

In order to see the AS beam a bit more clearly in our low-power config, I swapped out the ND=1.0 filter on the AS camera for ND=0.5.

  14351   Thu Dec 13 12:06:35 2018 gautamUpdateGeneralPower Outage recovery

I did a walkaround and checked the status of all the interlock switches I could find based on the SOP and interlock wiring diagram, but the PSL remains interlocked. I don't want to futz around with AC power lines so I will wait for Koji before debugging further. All the "Danger" signs at the VEA entry points aren't on, suggesting to me that the problem lies pretty far upstream in the wiring, possibly at the AC line input? The Red lights around the PSL enclosure, which are supposed to signal if the enclosure doors are not properly closed, also do not turn on, supporting this hypothesis...

I confirmed that there is nothing wrong with the laser itself - i manually shorted the interlock pins on the rear of the controller and the laser turned on fine, but I am not comfortable operating in this hacky way so I have restored the interlock connections until we decide the next course of action...

Quote:
 

The PSL (Edwin) remains in an interlock-triggered state. We are not sure what is causing this, but the laser cannot be powered on until this is resolved.

  14350   Thu Dec 13 10:03:07 2018 ChubUpdateGeneralOMC chamber

Bob, Aaron, and I removed the door from the OMC chamber this morning.  Everything went well.

  14349   Thu Dec 13 01:26:34 2018 gautamUpdateGeneralPower Outage recovery

[koji, gautam]

After several combinations of soft/hard reboots for FB, FEs and expansion chassis, we managed to recover the nominal RTCDS status post power outage. The final reboots were undertaken by the rebootC1LSC.sh script while we went to Hotel Constance. Upon returning, Koji found all the lights to be green. Some remarks:

  1. It seems that we need to first turn on FB
    • Manually start the open-mx and mx services using
      sudo systemctl start open-mx.service 
      sudo systemctl start mx.service
    • Check that the system time returned by gpstime matches the gpstime reported by internet sources.
    • Manually start the daqd processes using
      sudo systemctl start daqd_*
  2. Then fully power cycle (including all front and rear panel power switches/cables) the FEs and the expansion chassis.
    • This seems to be a necessary step for models run on c1sus (as reported by the CDS MEDM screen) to pick up the correct system time (the FE itself seems to pick up the correct time, not sure what's going on here).
    • This was necessary to clear 0x4000 errors.
  3. Power on the expansion chassis.
  4. Power on the FE.
  5. Start the RTCDS models in the usual way
    • For some reason, there is a 1 second mismatch between the gpstime returned on the MEDM screen for a particular CDS model status, and that in the terminal for the host machine.
    • This in itself doesn't seem to cause any timing errors. But see remark about c1sus above in #2.

The PSL (Edwin) remains in an interlock-triggered state. We are not sure what is causing this, but the laser cannot be powered on until this is resolved.

  14348   Wed Dec 12 18:27:07 2018 JonOmnistructureUpgradeAnalog signals, A/D Acromag added to vacuum system

There turned out to be a few analog signals for the vacuum system after all. The TP2/3 foreline pressure gauges were never part of the digital system, but we wanted to add them, as some of the interlock conditions should be predicated on their readings. Each gauge connects to an old Granville-Phillips 375 controller which only has an analog output. Interfacing these signals with the new system required installing an Acromag XT1221 8-channel A/D unit. Taking advantage of the extra channels, I also moved the N2 delivery line pressure transducer to the XT1221, eliminating the need for its separate Omega DPiS32 controller. When the new high-pressure transducers are added to the two N2 tanks, their signals can also be connected.

The XT1221 is mounted on the DIN rail inside the chassis and I have wired a DB-9 feedthrough for each of its three input signals. It is assigned the IP 192.168.114.27 on the vacuum subnet. Testing the channels in situ revealed a subtley in calibrating them to physical units. It was first encountered by Johannes in a series of older posts, but I repeat it here in one place.

An analog-input EPICS channel can be calibrated from raw ADC counts to physical units (e.g., sensor voltage) in two ways:

  1. Via LINR="LINEAR" by setting the engineering-units fields EGUF="[V_max_adc]", EGUL="[V_min_adc]"
  2. Via LINR="NO CONVERSION" by manually setting the gain ASLO="[V/count]" and offset AOFF="[V_offset]"

From the documentation, under the engineering-units method EPICS internally computes:

where EGUF="eng units full scale", EGUL="eng units low", and "full scale A/D counts" is the full range of ADC counts. EPICS automatically infers the range of ADC counts based on the data type returned by the ADC. For a 16-bit ADC like the XT1221, this number is 2^16 = 65,536.

The problem is that, for unknown reasons, the XT1221 rescales its values post-digitization to lie within the range +/-30,000 counts. This corresponds to an actual "full scale A/D counts" = 60,001. If a multiplicative correction factor of 65,536/60,000 is absorbed into the values of EGUF and EGUL, then the first term in the above summation can be corrected. However, the second term (the offset) has no dependence on "full scale A/D counts" and should NOT absorb a correction factor. Thus adjusting the EGUF and EGUL values from, e.g., 10V to 10.92V is only correct when EGUL=0V. Otherwise there is a bias introduced from the offset term also being rescaled.

The generally correct way to handle this correction is to use the manual "NO CONVERSION" method. It constructs calibrated values by simply applying a specified gain and offset to the raw ADC counts:

calibrated val = (measured A/D counts)  x ASLO + AOFF

The gain ASLO="[(V_max_adc - V_min_adc) / 60,001]" and the offset AOFF="0". I have tested this on the three vacuum channels and confirmed it works. Note that if the XT1221 input voltage range is restricted from its widest +/-10V setting, the number of counts is not necessarily 60,001. Page 42 of the manual gives the correct counts for each voltage setting.

  14347   Wed Dec 12 11:53:29 2018 aaronUpdateGeneralPower Outage

At 11:13 am there was a ~2-3 second interruption of all power at the 40m.

I checked that nobody was in any of the lab areas at the time of the outage.

I walked along both arms of the 40m and looked for any indicator lights or unusual activity. I took photos of the power supplies that I encountered, attached. I tried to be somewhat complete, but didn't have a list of things in mind to check, so I may have missed something. 

I noticed an electrical buzzing that seemed to emanate from one of the AC adapters on the vacuum rack. I've attached a photo of which one, the buzzing changes when I touch the case of the adapter. I did not modify anything on the vacuum rack. There is also 

Most of the cds channels are still down. I am going through the wiki for procedures on what to log when the power goes off, and will follow the procedures here to get some useful channels.

Attachment 1: IMG_0033.HEIC
Attachment 2: IMG_1027.HEIC
Attachment 3: IMG_2605.HEIC
  14346   Tue Dec 11 22:50:07 2018 aaronUpdateOMCAligning the OMC

I did the following:

  • Noticed that the OMC rack's power has +-18V, but I had tested the HV driver with +-15V. Maybe fine, something to watch.
  • Checked that nothing but the OMC driver board was in use on the OMC's Sorensen (the QPD whitening board in the OMC rack is not in use, and anyway is labeled +-15V), then turned down the rack voltage from 18 to 15V. Photos attached of AUX_OMC_S Sorensen bank.
  • I hadn't used the alternative dither before. I started by driving the alternative dither with a 10Vpp sine wave at 1-10 Hz. I have both the DC and AC driver mons on a scope.
    • Initially, I only give it 10V at the HV. I don't see much, nor at 30V, while driving with 0-10V sine waves between 0.1-100Hz.
    • In my last log, I hadn't been using the alternative dither.
  • Instead, I switch over to the main piezo drive, which is sent over DB9. Now I see the following on the AC/DC piezo mon channels:
    • Increasing the HV input (increasing in steps from 10-50V) yields 1V at the DC piezo mon for 50V at the HV input.
    • Driving under a few 100s of Hz results in no change to the AC dither mon. Driving <1Hz results in a small (~10% for a 10Vpp drive) at the HV. I didn't take a full transfer function, but it is the thing to do with cds.
    • Changing the drive amplitude changes the AC mon amplitude proportionally
    • At a few kHz, the 10Vpp drive saturates the AC mon.
    • Photos are in order:
      • 1Hz drive, visible on the DC mon channel in green
      • 1kHz drive 10Vpp, visible on the AC mon channel in violet
      • 1kHz drive 5Vpp
      • 5kHz drive 10Vpp, saturates the AC mon channel
Attachment 1: 8323029A-970E-4BEA-833E-77E709300446.jpeg
8323029A-970E-4BEA-833E-77E709300446.jpeg
Attachment 2: C52735BB-0C56-41A1-B731-678CDDCEC921.jpeg
C52735BB-0C56-41A1-B731-678CDDCEC921.jpeg
Attachment 3: 3F8A0B3B-5C7C-4876-B3A7-332F560D554D.jpeg
3F8A0B3B-5C7C-4876-B3A7-332F560D554D.jpeg
Attachment 4: 3DC88B6A-4213-4ABD-A890-7EC317D9EED0.jpeg
3DC88B6A-4213-4ABD-A890-7EC317D9EED0.jpeg
Attachment 5: C0C4F9C0-9574-4A17-9414-B99D6E27025F.jpeg
C0C4F9C0-9574-4A17-9414-B99D6E27025F.jpeg
Attachment 6: A191E1DE-552F-42A5-BED7-246001248BBD.jpeg
A191E1DE-552F-42A5-BED7-246001248BBD.jpeg
  14345   Tue Dec 11 18:20:59 2018 gautamUpdateOptical LeversBS/PRM HeNe is dead

I found that the BS/PRM OL SUM channels were reading close to 0. So I went to the optical table, and found that there was no beam from the HeNe. I tried power-cycling the controller, there was no effect. From the trend data, it looks like there was a slow decay over ~400000 seconds (~ 5 days) and then an abrupt shutoff. This is not ideal, because we would have liked to use the Oplevs as a DC alignment reference during the ventnoI plan to use the AS camera to recover some sort of good Michelson alignment, and then if we want to, we can switch out the HeNe.

*How can I export PDF from NDscope?

Attachment 1: BSOL_dead.png
BSOL_dead.png
  14344   Tue Dec 11 14:33:29 2018 gautamUpdateCDSNDScope

NDscope is now running on pianosa. To be really useful, we need the templates, so I've made /users/Templates/NDScope_templates where these will be stored. Perhaps someone can write a parser to convert dataviewer .xml to something ndscope can understand. To get it installed, I had to run:

sudo yum install ndscope
sudo yum install python34-gpstime
sudo yum install python34-dateutil
sudo yum install python34-requests

 I also changed the pythonpath variable to include the python3.4 site-packages library in .bashrc

Quote:

https://alog.ligo-wa.caltech.edu/aLOG/index.php?callRep=44971

Let's install Jamie's new Data Viewer

Attachment 1: ndscope.png
ndscope.png
  14343   Tue Dec 11 14:24:18 2018 aaronUpdateOMCAligning the OMC

I set up a function generator to drive OMC-L, and have the two DCPD mons and the OMC REFL PD sent to an oscilloscope. I need to select a cds channel over which to read the REFL signal.

The two DCPD mon channels have very different behaviors on the PD mons at the sat box (see attachment). PD1 has an obvious periodicity, PD2 has less noise overall and looks more white. I don't yet understand this, and whether it is caused by real light, something at the PDs, or something at the sat box.

I've again gone through the operations that will happen with the OMC chamber vented. Here's how it'll go, with some of the open questions that I'm discussing with Gautam or whoever is around the 40m:

  1. Function generator is driving OMC-L. Right now there is one 150V Kepco supply in use, located on the ground just to the right of the OMC rack. I only have plans to power it on while scanning OMC-L, and until the OMC is fully in use the standard practice will be to use this HV with two people in the lab and shut it off after the immediate activities.
    1. To do: Is a second drive necessary for the TT drivers? I don't think it is during this vent, because we will want to align into the OMC with the TTs in a 'neutral' state. I recall that the way the TT drivers are set up, 0V from the dac to the driver is the 'centered' position for all TTs. Unless we want to compensate for some known shift of the chambers during pumpdown, I think this is the TT position we should use while aligning the OMMT into the OMC.
    2. To do: make sure I'm driving the right pins with the function generator. Update: Seems I was driving the right channels, here's the pinout.
  2. We will use the reflection of aux from the SRM to align into the OMC.
    1. Gautam pointed out that I hadn't accounted for the recombination BS for the aux beam being 90-10. This means there's actually something like 300uW of aux onto the OMC, rather than ~3mW. This should still be enough to see on a card, so it is fine.
    2. However, the aux beam is aligned to be colinear with the AS beam when the SRM is misaligned. So the question is whether the wedge on the SRM makes the SRM-reflected aux beam not colinear with the AS beam

 

---------

Talked with Gautam for a good while about the above plan. In trying to figure out why the DCPD sat box appears to have a different TF for the two PDs (seems to be some loose cabling problem at the mons, because wiggling the cables changed this), we determined that the AA chassis also wasn't behaving as expected--driving the expected channels (28-31) with a sine wave yields some signal at the 100Hz driving frequency, but all save ch31 were noisy. We also still saw the 100Hz when the chassis was unplugged. I will continue pursuing this, but in the meantime I'm making an IDE40 to DB37 connector so I can drive the ADC channels directly with the DAC channels I've defined (need to match pinouts for D080303 to D080302). I also will make a new SCSI to DB37 adapter that is more robust than mentioned here. I also need to replace the cable carrying HV to the OMC-L driver, so that it doesn't have a wire-to-wire solder joint.

We moved a razor blade on the AP table so it is no longer blocking the aux beam. We checked the alignment of aux into the AS port. AUX and AS are not colinear anywhere on the AP table, and despite confirming that the main AS beam is still being reflected off of the OMC input mirror, the returning AUX beam does not reach the AP table (and probably is not reaching the OMC). AUX needs to be realigned such that it is colinear with the AS beam. It would be good if in this configuration, the SRM is held close to its position when the interferometer is locked, but the TTs should provide us some (~2.5mrad) actuation. Gautam will do this alignment and I will calculate whether the TTs will be able to compensate for any misalignment of the SRM.

Here is the new plan and minimal things to do for the door opening tomorrow:

  1. Function generator is driving OMC-L.
    1. The PZT mon channel is sent to the oscilloscope.
    2. To do: confirm again that the triangle wave I send in results in the expected triangle wave going to the OMC, using this mon channel.
  2. The OMC REFL signal is being sent to the AP PD. See photo.
    1. Need to align into this PD, but this alignment can be done in air on the AP table.
  3. Monitor the DCPD signals using the TPs from the sat box going to the oscilloscope.
    1. There may be further problems with the sat box, but for the initial alignment into the OMC only the REFL signal is necessary.
    2. Not minimally necessary, but the sat box needs a new case. It has a front, back, and bottom, but no main case, so the board is exposed.
  4. I will move the OMMT-to-OMC steering mirrors while watching the scope for flashes in the REFL signal.

That is the first, minimal sequence of steps, which I plan to complete tomorrow. After aligned into the OMC, the alignment into the DCPDs shouldn't need modification. Barring work needed to align from OMC to DCPDs, I think most other work with the OMC can be done in-air.

  14342   Tue Dec 11 13:48:04 2018 aaronUpdateOMCOMC channels

Koji gave me some tips on testing this board that I wanted to write down, notes probably a bit intermingled with my thoughts. Thanks Koji, also for the DCC and equipment logging!

  • Test the power and AI boards separately with an external supply, ramping the voltage up slowly for each.
  • If it seems the AI board is actually drawing too much current, may need to check its TPs for where a problem might be
    • If it's really extensive may use an IR camera to see what elements are getting too hot
    • Testing in segments will prevent breaking more components
  • Check the regulator that I've replaced
  • The 1 Ohm resistors may have been acting as extra 1A fuses. i need to make sure the resistors I've used to replace them are rated for >1W, if this is the case.
  • Can check the resistance between +-12V and Gnd inputs on the AI board, if there is a short drawing too much current it may show up there.
  • The 7812 may be an appropriate regulator, but the input voltage may need to be somewhat higher than with the low drop regulator that was used before.
  • I want to double check the diagram on the DCC
  14341   Tue Dec 11 13:42:44 2018 KojiUpdateOMCOMC channels

FYI:

D050368 Anti-Imaging Chassis
https://dcc.ligo.org/LIGO-D050368

https://labcit.ligo.caltech.edu/~tetzel/files/equip

D050368 Adl SUS/SEI Anti-Image filter board 
S/N 100-102 Assembled by screaming circuits. Begin testing 4/3/06 
S/N xxx Mohana returned it to the shop. No S/N or traveler. Put in shop inventory 4/24/06 
S/N 103 Rev 01. Returned from Screaming circuits 7/10/06. complete except for C28, C29 
S/N 104-106 Rev 01. Returned from Screaming circuits 7/10/06. complete except for C28, C29 Needs DRV-135’s installed 
S/N 107-111 Rev 02 (32768 Hz) Back from assembly 7/14/06 
S/N 112-113 Rev 03 (65536 Hz) assembled into chassis and waiting for test 1/29/07 
S/N 114 Rev 03 (65536 Hz) assembled and ready for test 020507 


D050512 RBS Interface Chassis Power Supply Board (Just an entry. There is no file)

https://dcc.ligo.org/LIGO-D050512

RBS Interface Chassis Power Board D050512-00

https://labcit.ligo.caltech.edu/~rolf/jayfiles/drawings/D050512-00.pdf
 

 

  14340   Mon Dec 10 19:47:06 2018 aaronUpdateOMCOMC channels

Taking another look at the datasheet, I don't think LM7812 is an appropriate replacement and I think the LM2940CT-12 is supposed to supply 1A, so it's possible the problem actually is on the power board, not on the dewhitening board. The board takes +/- 15V, not +/- 24...

Quote:
 
  1. I identified that the resistors were 1Ohm, and replaced them (though I couldn't find 1Ohm surface mount resistors). I also replaced the voltage regulator in case it was broken. I couldn't find the exact model, so I replaced the LM2940CT-12 with an LM7812, which I think is the newer 12V regulator.

 

ELOG V3.1.3-