40m QIL Cryo_Lab CTN SUS_Lab TCS_Lab OMC_Lab CRIME_Lab FEA ENG_Labs OptContFac Mariner WBEEShop
  40m Log, Page 32 of 335  Not logged in ELOG logo
ID Date Author Type Category Subject
  15307   Sat Apr 18 14:57:44 2020 YehonathanUpdateLoss MeasurementArm transfer function measurement

Ok, now I understand my foolishness. It should definitely be 1/sqrt(f^2+fp^2) .

Quote:
However, the data seem to fit well to 1/sqrt(f^2+fp^2) - electric field response - but not to 1/(f^2+fp^2) - intensity response. (Attachment 3).
  15306   Sat Apr 18 13:32:31 2020 ranaUpdateCamerasGigE w better NIR sensitivvity

There's this elog from Stephen about better 1064 sensitivity from Basler. We should consider getting one if he finds that its actual SNR is as good as we would expect from the QE improvement.

Might allow for better scatter measurements - not that we need more signal, but it could allow us to use shorter exposure times and reduce blurring due to the wobbly beams.

  15305   Thu Apr 16 21:13:20 2020 JonUpdateBHDBHD optics specifications

Summary

I've generated specifications for the new BHD optics. This includes the suspended relay mirrors as well as the breadboard optics (but not the OMCs).

To design the mode-matching telescopes, I updated the BHD mode-matching scripts to reflect Koji's draft layout (Dec. 2019) and used A La Mode to optimize ROCs and positions. Of the relay optics, only a few have an AOI small enough for curvature (astigmatism) and most of those do not have much room to move. This reduced the optimization considerably.

These ROCs should be viewed as a first approximation. Many of the distances I had to eyeball from Koji's drawings. I also used the Gaussian PRC/SRC modes from the current IFO, even though the recycling cavities will both slightly change. I set up a running list of items like these that we still need to resolve in the BHD README.

Optics Specifications

At a glance, all the specifications can be seen in the optics summary spreadsheet.

LO Telescope Design

The LO beam originates from the PR2 transmission (POP), near ITMX. It is relayed to the BHD beamsplitter (and mode-matched to the OMCs) via the following optical sequence:

  • LM1 (ROC = +10 m, AOI 3°)
  • LM2 (Flat, AOI  45°)
  • MMT1 (Flat, AOI  5°)
  • MMT2 (ROC = +3.5 m, AOI  5°)

The resulting beam profile is shown in Attachment 1.

AS Telescope Design

The AS beam is relayed from the SRM to the BHD beamsplitter (and mode-matched to the OMCs) via the following sequence:

  • AS1 (ROC = +1.5 m, AOI  3°)
  • AS2 (Flat, AOI  45°)
  • Lens (FL = -125 mm)

A lens is used because there is not enough room on the BHD breadboard for a pair of (low-AOI) telescope mirrors, like there is in the LO path. The resulting beam profile is shown in Attachment 2.

Attachment 1: LO_Beam_Calc-v1.pdf
LO_Beam_Calc-v1.pdf
Attachment 2: AS_Beam_Calc-v1.pdf
AS_Beam_Calc-v1.pdf
  15304   Wed Apr 15 15:15:17 2020 ChubUpdateVACnitrogen cylinders delivered

Four nitrogen cylinders replaced the empties in the rack at the west entrance.  Additionally, Airgas will now deliver only once a week.  Let me know via email or text when the there are four empties in the rack and I'll order the next round.

  15303   Tue Apr 14 23:50:06 2020 KojiUpdateGeneral40m power glitch recovery

[Koji / Gautam (Remote)]

Lab status

  • Gray Panel: The lab AC was off. Turned on all three (N/S, CTRL RM, E/W)
  • The control room AC was running.

Work stations

  • Control Room: All the control machines were running. We knew that nodus/chiara/fb were running
  • 1X6/7:
    • JETSTOR was making beeping sound. “Power #1 failed””power #2 failed”
    • Optimus & megatron were off -> turned on -> up and running now
  • 1X1/2:
    • Power cycled the netgear at the top of the IOO rack (maybe not necessary)
    • Turned on c1ioo -> up and running now
  • 1X4/5: Rebooted c1sus / c1lsc -> up and running now
  • 1X9: Rebooted c1iscex -> up and running now
  • 1Y4: Rebooted c1iscex -> up and running now

Vacuum status

  • Looked like everything was running as if it did not see the power glitch
  • TP1 normal: Set speed 33.6k rpm / Actual speed 33.6k rpm 
  • TP2 normal: 66k rpm / PTP2 16.0 mtorr
  • TP3 normal: 31k rpm / PTP3 45.4mtorr
  • P1 LOW / P2 1.7mtorr / CC2 1.1e-6 / P3 7.6e-2 / P4 LO
  • Annuli: 2.7~3torr
  • CC1 9.6e-6 / SUPER BEE 0.9mtorr

C1VAC recovery

  • c1vac was alive, but was isolated from the martian network
  • Checked the network I/F status with /sbin/ifconfig -a
    • eth0 had no IP
    • eth1 had the vac subnet IP (192.168.114.9)
  • Ran sudo /sbin/ifdown eth0 then  sudo /sbin/ifup eth0
  • The I/F eth0 started running and c1vac became visible from martian
  • Later checked the vacuum screen: The pressure values and valve statuses looked normal.
    The interlock state was “running”. The system state was “unrecognized”.

End RTS recovery 

  • The end slow machines (auxex and auxey) were already running
  • Restarting end RT models:
    • c1iscey -> rtcds start --all
    • c1iscex -> rtcds start --all
  • Confirmed that the models can dump the SUSs

Vertex RTS recovery

  • We wanted to use the reboot script. (/opt/rtcds/caltech/c1/scripts/cds/rebootC1LSC.sh)
  • c1susaux​​
    • To be safe, we wanted to bring c1susaux first.
    • c1susaux does not make the network I/Fs up automatically upon reboot.
      -> Connect an LCD display / keyboard / mouse to c1susaux
      -> Ran sudo /sbin/ifup eth0 and sudo /sbin/ifup eth1
    • Now c1susaux is visible from martian.
    • Login c1susaux and ran:  
      sudo systemctl start modbusIOC.service 
      -> c1susaux epics is up and running now
    • ...Meanwhile c1susaux lost its eth1 somehow. This made the slow values of 8 vertex sus all zero
      -> Ran sudo /sbin/ifdown eth1 and sudo /sbin/ifup eth1 again on c1susaux ->  this resolved the issue
  • c1psl
    • Login c1psl and ran:  
      sudo systemctl start modbusIOC.service 
      -> c1psl epics is up and running now
  • Prepared for the rebooting script
    • Ran /opt/rtcds/caltech/c1/scripts/cds/rebootC1LSC.sh
    • Rebooting was done successfully. All the suspensions looked free and healthy.
    • Burtrestored c1susaux (used Apr 12 21:19 snapshot)

Hardware

  • PSL laser / Xend AUX laser / Yend AUX laser were off -> turned on
  • The PMC was immediately automatically locked.
  • The main marconi was off -> forgot to turn on
  • The end temp controllers for the SHG crystals were on but not enabled -> now enabled

RTS recovery ~ part 2

  • FB: FB status of all the RTS models were still red
  • Timing: c1x01/2/3/5 were 1 sec behind of FB and c1x04 was 2 sec behind
  • -> Remedy:  https://nodus.ligo.caltech.edu:8081/40m/14349
    • Software rebooting of FB
    • Manually start the open-mx and mx services using
    • sudo systemctl start open-mx.service 
    • sudo systemctl start mx.service
    • Check that the system time returned by gpstime matches the gpstime reported by internet sources. e.g. http://leapsecond.com/java/gpsclock.htm
    • Manually start the daqd processes using
      sudo systemctl start daqd_*
  • This made all the FB(FE) indicators green!
  • Ran the reboot script again -> All green!

IMC recovery

  • The IMC status was checked
  • No autolocker, but it could be manually locked. i.e. MC1/2/3 were not so much misalignment
  • Autolocker/Slow FSS recovery along with https://nodus.ligo.caltech.edu:8081/40m/15121
    • sudo systemctl start MCautolocker.service
    • sudo systemctl start FSSSlow.service
  • Both of them failed to run
  • Note by Gautam: The problem with the systemctl commands failing was that the NFS mount points weren’t mounted. Which in turn was because of the familiar /etc/resolv.conf problem. I added chiara to the namespace in this file, and then manually mounted the NFS mount points. This fixed the problem.
    Now the IMC is locked and the autolocker is left running.

Burt restore

  • Used Apr 12 21:19 snapshot
  • c1psl
  • c1alsepics/c1assepics/c1asxepics/c1asyepics
  • c1aux/c1auxex/c1auxey/
  • c1iscaux/c1susaux
  • This made REFL and AS beams back to the CCDs. As has small fringes.
  • Y arm has small IR flashes as well as green flashes.

JETSTOR recovery

  • JETSTOR was beeping. 
  • Shutdown megatron
  • Followed the instruction https://nodus.ligo.caltech.edu:8081/40m/13107
  • This stopped beeping. Waiting for JETSTOR to come up -> In a minute, JETSTOR display became normal and all disks showed green.
  • Bring megatron back up again

N2 bottle

  • The left N2 bottle was empty. The right one had 1500PSI.
  • Replaced the left bottle with the spare one in the room.
  • Now the left one 2680PSI and the right one 1400PSI.

Closing

  • Closed PSL/AUX laser shutters
  • Turned off the lights in the lab, CTRL room, and the office.

Remaining Issues

  • [done] MCAutoLocker / FSSSlow scripts are not running
  • The PRM alignment slider has no effect (although the PRM is aligned…) -> SLOW DAQ frozen???
  • JETSTOR is not mounted on megatron [gautam mounted Jetstor on megatron on 4/18 at 2pm]
  15302   Mon Apr 13 16:51:49 2020 ranaSummaryDAQNODUS: rsyncd daemon / service set up

I just now modified the /etc/rsyncd.conf file as per Dan Kozak's instructions. The old conf file is still there with the file name appended with today's date.

I then enabled the rsync daemon to run on boot using 'enable'. I'll ask Dan to start the file transfers again and see if this works.

controls@nodus|etc> sudo systemctl start rsyncd.service
controls@nodus|etc> sudo systemctl enable rsyncd.service
Created symlink from /etc/systemd/system/multi-user.target.wants/rsyncd.service to /usr/lib/systemd/system/rsyncd.service.
controls@nodus|etc> sudo systemctl status rsyncd.service
● rsyncd.service - fast remote file copy program daemon
   Loaded: loaded (/usr/lib/systemd/system/rsyncd.service; enabled; vendor preset: disabled)
   Active: active (running) since Mon 2020-04-13 16:49:12 PDT; 1min 28s ago
 Main PID: 4950 (rsync)
   CGroup: /system.slice/rsyncd.service
           └─4950 /usr/bin/rsync --daemon --no-detach

Apr 13 16:49:12 nodus.martian.113.168.192.in-addr.arpa systemd[1]: Started fast remote file copy program daemon.
Apr 13 16:49:12 nodus.martian.113.168.192.in-addr.arpa systemd[1]: Starting fast remote file copy program daemon...

  15301   Mon Apr 13 15:28:07 2020 KojiUpdateGeneralPower Event and recovery

[Larry (on site), Koji & Gautam (remote)]

Network recovery (Larry/KA)

  • Asked Larry to get into the lab. 

  • 14:30 Larry went to the lab office area. He restarted (power cycled) the edge-switch (on the rack next to the printer). This recovered the ssh-access to nodus. 

  • Also Larry turned on the CAD WS. Koji confirmed the remote access to the CAD WS.

Nodus recovery (KA)

  • Apr 12, 22:43 nodus was restarted.

  • Apache (dokuwiki, svn, etc) recovered along with the systemctl command on wiki

  • ELOG recovered by running the script

Control Machines / RT FE / Acromag server Status

  • Judging by uptime, basically only the machines that are on UPS (all control room workstations + chiara) survived the power outage. All RT FEs are down. Apart from c1susaux, the acromag servers are back up (but the modbus processes have NOT been restarted yet). Vacuum machine is not visible on the network (could just be a networking issue and the local subnet to valves/pumps is connected, but no way to tell remotely).

  • KA imagines that FB took some finite time to come up. However, the RT machines required FB to download the OS. That made the RTs down. If so, what we need is to power cycle them.

  • Acromag: unknown state

The power was lost at Apr 12 22:39:42, according to the vacuum pressure log. The power loss was for a few min.

  15300   Tue Apr 7 15:30:40 2020 JonSummaryNoiseBudget40m noise budget migrated to pygwinc

In the past year, pygwinc has expanded to support not just fundamental noise calculations (e.g., quantum, thermal) but also any number of user-defined noises. These custom noise definitions can do anything, from evaluating an empirical model (e.g., electronics, suspension) to loading real noise measurements (e.g., laser AM/PM noise). Here is an example of the framework applied to H1.

Starting with the BHD review-era noises, I have set up the 40m pygwinc fork with a working noise budget which we can easily expand. Specific actions:

  • Updated the 40m fork to the latest pygwinc version (while preserving the commit history).
  • Added a directory ./CIT40m containing the 40m-specific noise budget files (created by GV).
  • Added an ipython notebook CIT40m.ipynb at the root level showing how to generate a noise budget.
  • Integrated our DAC and seismic noise estimators into pygwinc.
  • Marked the old 40m NB repo as obsolete (last commit > 2 yrs ago). Many of these noise estimates are probably stale, but I will work with GV to identify which ones can be migrated.

I set up our fork in this way to keep the 40m separate from the main pygwinc code (i.e., not added to as a built-in IFO type). With the 40m code all contained within one root-level directory (with a 40m-specific name), we should now always be able to upgrade to the latest pygwinc without creating intractable merge conflicts.

  15299   Tue Apr 7 10:56:39 2020 JonUpdateBHDBHD front-end complication
Quote:

I have a query out to Dolphin asking:

  1. Have they done any testing of these old drivers on Linux kernel 4.x (e.g., Debian 9/10)?
  2. Is there any way to buy modern IPC cards for the two new machines and interface them with our existing Gen1 network?

Answers from Dolphin:

  1. No, and kernel 4.x (modern Linux) definitely will not work with the Gen1 cards.
  2. No, cards using different PCIe chipsets cannot be mixed.

Since upgrading every front end is out of the question, our only option is to install an old OS (Linux kernel < 3.x) on the two new machines. Based on Keith's advice, I think we should go with Debian 8. (Link to Keith's Debian 8 instructions.)

  15298   Mon Apr 6 16:46:40 2020 gautamUpdateASCPOP angular FF filters trained and tested

I don't have a recent measurement of the optical gain of this config so I can't undo the loop, but in-loop performance doesn't suggest any excess in the 10-100 Hz band. Interestingly, there is considerable improvement below 10 Hz. Maybe some of this is reduced A2L noise because of the better angular stability, but there is also improvement at frequencies where the FF isn't doing anything, so could be some bilinear coupling. The two datasets were collected at approximately the same time in the evening, ~5pm, but on two different days.

Quote:

I wonder how much noise is getting injected into PRC length at 10-100 Hz due to this. Any change the PRC ERR?

Attachment 1: PRCL_comparison.pdf
PRCL_comparison.pdf
  15297   Mon Apr 6 12:26:07 2020 ranaUpdateASCPOP angular FF filters trained and tested

that's pretty great performance. maybe you can also upload some code so that we can do it later too - or maybe in the 40m GIT

I wonder how much noise is getting injected into PRC length at 10-100 Hz due to this. Any change the PRC ERR?

  15296   Fri Apr 3 17:15:53 2020 gautamUpdateASCPOP angular FF filters trained and tested

Summary:
Using the data I collected yesterday, the POP angular FF filters have been trained. The offline time-domain performance looks (unbelievably) good, online performance will be verified at the next available opportunity(see update).

Details:

The sequence of steps followed is the same as that done for the MCL FF filters. The trace that is missing from Attachment #1 is the measured online subtraction. Some rough notes:

  • The "target" channels for the subtraction are the POP QPD PIT/YAW signals, normalized by the QPD sum. For the time that the PRMI was locked yesterday, the QPD readouts suggested that the beam was well centered on the QPD, but the POP QPD (OT-301) doesn't give me access to individual quadrant signals so I couldn't actually verify this.
  • I used 64s impulse time on the FIR filter for training. Maybe this is too long, but anyways, the calculation only takes a few seconds even with 64^2 taps.
  • I found that the Levinson matrix algorithm sometimes failed for this particular dataset. I didn't bother looking too much into why this is happening, the brute force matrix inversion took ~4 times longer but still was only ~5 seconds to calculate the optimal filter for 20 mins of training data sampled at 64 Hz.
  • The actuator TF was measured with >0.9 coherence between 0.3 Hz - 10 Hz and fitted, and the fit was used for subsequent analysis. Fit is shown in Attachment #2.
  • FIR to IIR fitting took considerable tweaking, but I think I got good enough fits, see Attachments #3, #4. In fact, there may be some benifit to making the shape smoother outside the subtraction band but I couldn't get IIRrational to cooperate. Need to confirm that this isn't re-injecting noise.

Update Apr 5 1145pm:

  • Attachment #1 has now been updated to show the online performance. The comparison between the "test" and "validation" datasets aren't really apple-to-apple because they were collected at different times, but I think there's enough evidence here to say that the feedforward is helping.
  • Attachment #5 shows that the POP DC (= PRC intracavity buildup) RMS has been stabilized by more than x2. This signal wasn't part of the training process, and I guess it's good that the intracavity power is more stable with the feedforward on. Median averaging was used for the spectral densities, there were still some abrupt glitches during the time this dataset was collected.
  • The next step is to do the PRFPMI locking with all of these recently retuned feedforward loops engaged and see if that helps things.
Quote:

This afternoon, I kept the PRM locked for ~1hour and then measured transfer functions from the PRM angular actuators to the POP QPD spot motion for pitch and yaw between ~1pm and 4pm. After this work, the PRM was misaligned again. I will now work on the feedforward filter design.

Attachment 1: FIRvIIR.pdf
FIRvIIR.pdf
Attachment 2: PRM_act_calib.pdf
PRM_act_calib.pdf
Attachment 3: IIR_fit_to_FIR_PIT.pdf
IIR_fit_to_FIR_PIT.pdf
Attachment 4: IIR_fit_to_FIR_YAW.pdf
IIR_fit_to_FIR_YAW.pdf
Attachment 5: POP_DC_comparison.pdf
POP_DC_comparison.pdf
  15295   Fri Apr 3 13:40:07 2020 JonUpdateBHDBHD front-end complication

I wanted to pass along a complication pointed out by K. Thorne re: our plan to use Gen1 (old) Dolphin IPC cards in the new real-time machines: c1bhd, c1sus2. The implication is that we may be forced to install a very old OS (e.g., Debian 8) for compatibility with the IPC card driver, which could lead to other complications like an incompatibility with the modern network interface.

Hardware is easy - you will also need a DX switch and the cables

As for the driver - the last update (version 4.4.5) was in 2016.  The notes on it say valid for Linux kernel 2.6 to 3.x.  This implies that it will not work with Linux kernel 4.x and greater

So - Gentoo with 3.0 kernel OK, SL7 (kernel 3.10)  - OK,   Debian 8 (kernel 3.16) - OK   

But Debian 9 (kernel 4.9),Debian 10 (kernel 4.19) - NOT OK

We have Gentoo with kernel 3.0  boot server, etc. [used in L1,H1 production right now, but not much longer] The hard part here will be making sure we have network drivers for the SuperMicro 5018-MR.

CDS was never able to get real-time builds to work well on Linux kernels from 3.2 on up until we got to Debian 9. This is not to say that the tricks and stripped-down RCG we found worked for real-time on Debian 9 and 10 won’t work on, say, Debian 8.  But we have not tried.

I have a query out to Dolphin asking:

  1. Have they done any testing of these old drivers on Linux kernel 4.x (e.g., Debian 9/10)?
  2. Is there any way to buy modern IPC cards for the two new machines and interface them with our existing Gen1 network?

I'll add more info if I hear back from them.

  15294   Fri Apr 3 12:09:53 2020 JonUpdateCDSC1AUXEY wiring + channel list
Quote:

We want to migrate the end shutter controls from c1aux to the end acromags. Could you include them to the list if not yet?

This will let us remove c1aux from the rack, I believe.

Yehonathan's list does include C1:AUX-GREEN_Y_Shutter and I copied its definition from /cvs/cds/caltech/target/c1aux/ShutterInterlock.db into the new ETMYaux.db file.

I noticed ShutterInterlock.db still contains about a dozen channels. Some of them appear to be ghosts (like the C1:AUX-PSL_Shutter[...] set, which has since become C1:PSL-PSL_Shutter[...] hosted on c1psl) but others like C1:AUX-GREEN_X_Shutter appear to still be in active use.

  15293   Thu Apr 2 22:19:18 2020 KojiUpdateCDSC1AUXEY wiring + channel list

We want to migrate the end shutter controls from c1aux to the end acromags. Could you include them to the list if not yet?

This will let us remove c1aux from the rack, I believe.

 

  15292   Thu Apr 2 16:31:33 2020 JonUpdateCDSC1AUXEY wiring + channel list
Quote:

I have made a wiring + channel list that need to be included in the new C!AUXEY Acromag.

I used Yehonathan's wiring assignments to lay the rest of groundwork for the final slow controls machine upgrade, c1auxey. Actions completed:

  • Created an internal wiring diagram for assembling the Acromag chassis (log in with LIGO.ORG credentials to view/edit)
  • Created a new target directory on the network drive:
/cvs/cds/caltech/target/c1auxey1

The "1" will be dropped after the new system is permanently installed.

  • Populated the target directory with files:
    • modbusIOC.service - wraps the EPICS IOC as a systemd service
    • ETMYaux.env - defines the EPICS environment variables
    • ETMYaux.cmd - command file to set up the EPICS IOC
    • ETMYaux.sh - enables DAC outputs to the suspension (executed lastly)
  • Created the EPICS channel databases:
    • ETMYaux.db - migration of the existing database
    • c1auxey_state.db - contains logic for loopback monitoring of the IOC "alive" state (visible from Sitemap > CDS > Slow Controls Status)

Hardware-wise, this system will require:

  • 2 Acromag XT-1221 units (ADC)
  • 1 Acromag XT-1541 unit (DAC)
  • 1 Acromag XT-1111 unit (sinking BIO)

I know that we do have these quantities left on hand. The next steps are to set up the Supermicro host and begin assembling the Acromag chassis. Both of these activities require an in-person presence, so I think this is as far as we can advance this project for now.

  15291   Thu Apr 2 15:53:01 2020 gautamUpdateASCPRMI 1f locked for collecting feedforward data

This afternoon, I kept the PRM locked for ~1hour and then measured transfer functions from the PRM angular actuators to the POP QPD spot motion for pitch and yaw between ~1pm and 4pm. After this work, the PRM was misaligned again. I will now work on the feedforward filter design.

  15290   Wed Apr 1 00:51:41 2020 gautamUpdateWienerSlightly improved MCL FF

Summary:

Retraining the MCL filters resulted in a slight improvement in the performance. Compared to no FF, the RMS in the 0.5-5 Hz range is reduced by approximately a factor of 3

Details:

Attachment #1 shows my re-measurement of the MC2 position drive to MCL transfer function.

  • The measurement was made using DTT swept sine, with the amplitude enveloped appropriately to avoid knocking the IMC out of lock.
  • Coherence was >0.97 for all datapoints.
  • Fitting was done using Lee's IIRrational, with the weighting being the coherence. I think there are some features of the fitting I don't fully understand, but I wanted to try and do everything in python and for this simple fit, it came out nicely I think. 

Attachment #2 shows the IIR fits to the FIR filters calculated here

  • Again, IIRrational was used. 
  • In the frequency band where subtraction is possible, the fit is good.
  • But there is definitely room for improvement in the way this is done, for now, I did quite a bit "by eye" and tweaked the order of the filter and the minimum number of excess poles relative to zeros to get the AC coupling, but it'd be nice to make all of this iterative and quantitative (e.g. by minimizing a cost function).
  • One nice feature of IIRrational is that it directly gives me a formatted string I can paste into foton. The order of these fits were 22, so I split them into two 19+3 order filters to be compatible with the realtime system before loading the coefficients (the overall gain was allocated to a single filter arbitrarily, with the other filter in the pair set to have unity gain in the zpk representation).

Attachment #3 shows several MCL spectra.

  • Blue trace is the unsubtracted test dataset.
  • Red is the performance of the calculated FIR filter, but the filtering is done offline.
  • Gold is the performance of the IIR fit to the FIR filter, as shown in Attachment #2, applied offline to the test dataset.
  • Green is the calculated ASD of MCL from a ~1 hour stretch from earlier tonight, when I left the feedforward loop on. So this is an actual measurement of the online performacne of the filter.
  • Grey is the performance of the old filter loaded in the CDS system - the filtering is done using scipy, and the sos coefficients from the C1OAF.txt file.

Conclusions + next steps

  1. Retraining the filters has resulted in a slight improvement, especially at ~3 Hz.
  2. More tests need to be done to confirm that noise isn't being reinjected in the frequency bands where subtraction isn't possible (e.g. using arm cavities as OOL sensors).
  3. The online filter isn't quite as good as what we would expect from calculations (green trace is noisier than gold). Need to think about why this is.
  4. Why can't we get more subtraction at 1 Hz?
  5. Now that I have the infrastructure ready, I will attempt to revive the PRC angular FF loops, which was the whole point of this exercise. 
Attachment 1: MC2_act_calib.pdf
MC2_act_calib.pdf
Attachment 2: IIR_fit_to_FIR.pdf
IIR_fit_to_FIR.pdf
Attachment 3: FIRvIIR.pdf
FIRvIIR.pdf
  15289   Tue Mar 31 23:54:57 2020 gautamUpdateCDSFoton for shaped noise injections

The problem is that foton does not inherit the model sample rate when launched from DTT/awggui. This is likely some shared/linked/dynamic library issue, the binaries we are running are precompiled presumably for some other OS. I've never gotten this to work since we changed to SL7 (but I did use it successfully in 2017 with the Ubuntu12 install).

Quote:

do you really mean awggui cannot make shaped noise injections via its foton text box ? That has always worked for me in the past.

If this is broken I'm suspicious there's been some package installs to the shared dirs by someone.

  15288   Tue Mar 31 23:35:50 2020 ranaUpdateCDSFoton for shaped noise injections

do you really mean awggui cannot make shaped noise injections via its foton text box ? That has always worked for me in the past.

If this is broken I'm suspicious there's been some package installs to the shared dirs by someone.

  15287   Tue Mar 31 09:39:41 2020 gautamUpdateCDSFoton for shaped noise injections

I'd like to re-measure the transfer function from driving MC2 position to the MC_L_DQ channel (for feedforward purposes). Swept sine would be one option, but I can't get the "Envelope" feature of DTT to work, the excitation amplitude isn't getting scaled as specified in the envelope, and so I'm unable to make the measurement near 1 Hz (which is where the FF is effective). I see some scattered mentions of such an issue in past elogs but no mention of a fix (I also feel like I have gotten the envelope function to work for some other loop measurement templates). So then I thought I'd try broadband noise injection, since that seems to have been the approach followed in the past. Again, the noise injection needs to be shaped around ~1 Hz to avoid knocking the IMC out of lock, but I can't get Foton to do shaped noise injections because it doesn't inherit the sample rate when launched from inside DTT/awggui - this is not a new issue, does anyone know the fix?

Note that we are using the gds2.15 install of foton, but the pre-packaged foton that comes with the SL7 installation doesn't work either.

Update:

The envelope feature for swept-sine wasn't working because i specified the frequency grid in the wrong order apparently. Eric von Reis has been notified to include a sorting algorithm in future DTT so that this can be in arbitrary order. fixing that allows me to run a swept sine with enveloped excitation amplitude and hence get the TF I want, but still no shaped noise injections via foton 😢 

  15286   Mon Mar 30 19:02:49 2020 ranaUpdateGeneraldonated cleanroom supplies to Hospitals

Yesterday evening I took nearly all of the masks, gloves, gowns, alcohol wipes, hats, and shoe covers. These were the ones in the cleanroom cabinets at the east end of the Y-arm, as well as the many boxes under the yarm near those cabinets.

This photo album shows the stuff, plus some other random photos I took around the same time (6-7 PM) of the state of parts of the lab.

  15285   Thu Mar 26 22:31:34 2020 YehonathanUpdateCDSC1AUXEY wiring + channel list

I have made a wiring + channel list that need to be included in the new C!AUXEY Acromag.

It was mostly copied from C1AUXEX

I ignored the IPANG channels since it is going to be removed from the table.

  15284   Thu Mar 26 17:41:18 2020 JonOmnistructureBHDBHD docs compilation

Since there has been a proliferation of BHD Google docs recently, I've linked them all from the BHD wiki page. Let's continue adding any new docs to this central list.

  15283   Wed Mar 25 15:15:55 2020 gautamUpdateVACVacuum interlock code, N2 warning update

The email address in the N2 checking script wasn't right - I now updated it to email the 40m list if the sum of reserve tank pressures fall below 800 PSI. The checker itself is only run every 3 hours (via cron on c1vac).

Quote:

I reset the remote of this git repo to the 40m version instead of Jon's personal one, to ensure consistency between what's on the vacuum machine and in the git repo. There is now a N2 checker python mailer that will email the 40m list if all the tank pressures are below 600 PSI (>12 hours left for someone to react before the main N2 line pressure drops and the interlocks kick in). For now, the script just runs as a cron job every 3 hours, but perhaps we should integrate it with the interlock process

  15282   Tue Mar 24 19:41:57 2020 gautamUpdateWienerSeismic feedforward for MCL

Summary:

I think the feedforward filters used for stabilizing MCL with vertex seismometers would benefit from a retraining (last trained in Sep 2015). 

Details:

I wanted to re-familiarize myself with the seismic feedforward methodology. Getting good stabilization of the PRC angular motion as we have been able to in the past will be a big help for lock acquisition. But remotely, it is easier to work with the IMC length feedforward (IMC is locked more often than the PRC). So I collected 2 hours of data from early Sunday morning and went through the set of steps (partially).

Attachment #1 shows the performance of a first attempt.

  • 1 hour of data was used as a training set, and another hour to validate the trained filter.
  • All the data was downsampled to 64 Hz.
  • The number of FIR filter taps was 32 seconds * 64 Hz. 
  • Going through some old elogs, there were a number of suggestions from various people about how the training should be done
    • There was a suggestion that pre-filtering the target signal by the (inverse) actuator TF (i.e. TF from MC2 drive to MCL) is beneficial, presumably because it gives the Wiener filter fitting fewer parameters to fit.
    • There was also suggestions that some frequency-dependent weighting of the target signal should be done (e.g. by bandpassing MCL between 0.1 Hz - 10 Hz) to emphasize subtraction in this band.
    • For this particular example, in my limited paramter space exploration, I found that neither of these measures had particularly significant impact.
  • In any case, the time-domain FIR filtering seems to approach the theoretical best possible performance (based on coherence information). 
  • I have not yet checked what the theoretical limit on subtraction will be based on the seismometer noise ASD.

Attachment #2 shows a comparison between the filter used in Attachment #1 and the filters currently loaded into the OAF system. 

  • In the band where significant subtraction is possible, there is some difference in the shape of the filter.
  • Why should this have changed? I guess there are multiple possibilities - seismometer recentering, signal chain changes, ...

Attachment #3 is the asd after implementing a time domain Wiener filter, while Attachment #4 is an actual measurement from earlier today - it's not quite as good as Attachment #3 would have me expect but that might also be due to the time of the day. 

Conclusions and next steps:

On the basis of Attachments #3 and #4, I'd say it's worth it to complete the remaining steps for online implementation: FIR to IIR fitting and conversion to sos coefficients that Foton likes (prefereably all in python). Once I've verified that this works, I'll see if I can get some data for the motion on the POP QPD with the PRMI locked on carrier. That'll be the target signal for the PRC angular FF training. Probably can't hurt to have this implemented for the arms as well.

While this set of steps follows the traditional approach, it'd be interesting if someone wants to try Gabriele's code which I think directly gives a z-domain representation and has been very successful at the sites.

* The y-axes on the spectra are labelled in um/rtHz but I don't actually know if the calibration has been updated anytime recently. As I type this, I'm also reminded that I have to check what the whitening situation is on the Pentek board that digitizes MCL.

Attachment 1: IMCseisFF.pdf
IMCseisFF.pdf
Attachment 2: filterComp.pdf
filterComp.pdf
Attachment 3: oldFilter_v_proposed.pdf
oldFilter_v_proposed.pdf
Attachment 4: MCL_ff_performance.pdf
MCL_ff_performance.pdf
  15281   Thu Mar 19 03:33:28 2020 gautamUpdateLSCMore locking updates

Some short notes, more details tomorrow.

  1. I was able to make it to CARM on RF only ~10 times tonight.
  2. Highest stable circulating power was ~200 (recycling gain ~10) but the control scheme is still not finalized in terms of offsets etc.
  3. DARM to RF transition was never fully engaged - I got to a point where the ALS gain was reduced to <half its nominal value, but IMC always lost lock.
  4. CARM loop UGF of ~5 kHz was realized. I was also able to turn on a regular boost. But couldn't push the gain up much more than this. Should probably modify the boosts on this board, their corner frequencies are pretty high.
  5. The increased FSS flakiness post c1psl upgrade is definitely hurting this effort, there are periods of ~20-30mins when the IMC just wont lock.

Attachment #1 shows time series of some signals, from the time I ramp of ALS CARM control to a lockloss. With this limited set of signals, I don't see any clear indication of the cause of lockloss, but I was never able to keep the lock going for > a couple of mins.

Attachment #2 shows the CARM OLTF. Compared to last week, I was able to get the UGF a little higher. This particular measurement doesn't show it, but I was also able to engage the regular boost. I did a zeroth order test looking at the CM_SLOW input to make sure that I wasn't increasing the gain so much that the ADC was getting saturated. However, I did notice that the pk-to-pk error signal in this locked, 5kHz UGF state was still ~1000 cts, which seems large?

Attachment #3 shows the DTT measurement of the relative gains of DARM A and B paths. This measurement was taken when the DARM_A gain was 1, and DARM_B gain was 0.015. On the basis of this measurement, DARM_B (=AS55) sees the excitation injected 16dB above the ALS signal, and so the gain of the DARM_B path should be ~0.16 for the same UGF. But I was never able to get the DARM_B gain above 0.02 without breaking the lock (admittedly the lockloss may have been due to something else).

Attachment #4 shows a zoomed in version of Attachment #1 around the time when the lock was lost. Maybe POP_YAW experienced too large an excursion?

Some other misc points:

  • It was much quicker to acquire the PRMI lock with CARM held off resonance using the 1f signals rather than 3f - so I did that and then once the lock is acquired, transfer control to 3f signals (using CDS ramptime) before zeroing the CARM offset.
  • The whole process is pretty speedy - it takes <5mins to get to the CARM on RF only stage provided the PRMI lock doesn't take too long (the transition from POX/POY to ALS sequence takes <1min).
  • I am wondering what the correct way to set the offsets for the 3f error signals is? 
  • The arm buildup is strongly dependent on the DC alignment of the PRMI - the best buildups I got were when I tweaked the BS alignment after the CARM offset was zeroed.
Attachment 1: PRFPMI_lock.png
PRFPMI_lock.png
Attachment 2: CARMTF.pdf
CARMTF.pdf
Attachment 3: DARM_AvB.pdf
DARM_AvB.pdf
Attachment 4: lockLoss.png
lockLoss.png
  15280   Wed Mar 18 22:10:41 2020 KojiUpdateVACMain vol pressure jump

I was in the lab at the time. But did not notice anything (like turbo sound etc). I was around ETMX/Y (1X9, 1Y4) rack and SUS rack (1X4/5), but did not go into the Vac region.

  15279   Wed Mar 18 21:43:26 2020 gautamUpdateVACMain vol pressure jump

There was a jump in the main volume pressure at ~6pm PDT yesterday. The cause is unknown, but the pressure doesn't seem to be coming back down (but also isn't increasing alarmingly).

I wanted to look at the RGA scans to see if there were any clues as to what changed, but looks like the daily RGA scans stopped updating on Dec 24 2019. The c0rga machine responsible for running these scans doesn't respond to ssh. Not much to be done until the lockdown is over i guess...

Attachment 1: VacPresJump.png
VacPresJump.png
  15278   Tue Mar 17 01:22:03 2020 gautamUpdateLSCLocking updates

Summary:

No real progress tonight - I made it a bunch of times to the point where CARM was RF only, but I never got to run a measurement to determine what the DARM_B loop gain should be to make the control fully RF.

Details:

  • Touched up PMC alignment.
  • There were very few BNC cables available at the rack near SW corner of the PSL table - the short BNC cables are NOT meant to be daisy chained to make long cables to run along the arm, I removed all those.
  • Restored SR785 at LSC rack for CARM TF measurements.
  • I was able to get the CARM UGF ~5 kHz, but everytime I was trying to run a DTT swept sine to measure the ratio of DARM_B_IN1 / DARM_A_IN1, the lock was lost - not sure if this is because of the excitation injected or something else.
  • I'll probably give this another shot Wednesday eve.
  15277   Mon Mar 16 15:23:03 2020 YehonathanUpdateLoss MeasurementArm transfer function measurement

I measured the cross-calibration of the two PDs on the PSL table.

I used the existing flip mounted BS that routes the beam into a PDA255, the same as in the IMC transmission.

I placed a PDA520, the same as the one measuring TRY_OUT on the ETMY table,  on the transmission of the BS (Attachment 1).

I used the SR785 to measure the frequency response of PDA520 with reference to PDA255 (Attachment 2). Indeed, calibration is quite significant.

I calibrated the Y arm frequency response measurement.

However, the data seem to fit well to 1/sqrt(f^2+fp^2) - electric field response - but not to 1/(f^2+fp^2) - intensity response. (Attachment 3).

Also, the extracted fp is 3.8KHz (Finesse ~ 500) in the good fit -> too small.

When I did this measurement for the IMC in the past I fitted the response to 1/sqrt(f^2+fp^2) by mistake but I didn't notice it because I got a pole frequency that was consistent with ringdown measurements.

I also cross calibrated the PDs participating in the IMC measurement but found that the calibration amounted for distortions no bigger than 1db.

Attachment 1: Cross_calibration_setup.jpg
Cross_calibration_setup.jpg
Attachment 2: PDA520overPDA255_Response.pdf
PDA520overPDA255_Response.pdf
Attachment 3: YArmFrequencyResponse.pdf
YArmFrequencyResponse.pdf
  15276   Fri Mar 13 20:00:50 2020 JonUpdateComputersLoopback monitoring for slow machines

Summary

Today I finished implementing loopback monitors of the up/down state of the slow controls machines. They are visible on a new MEDM screen accessible from Sitemap > CDS > Slow Machine Status (pictured in attachment 1). Each monitor is a single EPICS binary channel hosted by the slow machine, which toggles its state at 1 Hz (an alive "blinker"). For each machine, the monitor is defined by a separate database file named c1[machine]_state.db located in the target directory.

This is implemented for all upgraded machines, which includes every slow machine except for c1auxey. This is the next and final one slated for replacement.

Implementation

The blinkers are currently implemented as soft channels, but I'd like to ultimately convert them to hard channels using two sinking/sourcing BIO units. This will require new wiring inside each Acromag chassis, however. For now, as soft channels, the monitors are sensitive to a failure of the host machine or a failure of the EPICS IOC. As hard channels, they will additionally be sensitive to a failure of the secondary network interface, as has been known to happen.

Each slow machine's IOC had to be restarted this afternoon to pick up the new channels. The IOCs were restarted according to the following procedure.

c1auxex

  • Disabled OPLEV servos on ETMX
  • Zeroed slow biases
  • Disabled watchdog
  • Restarted IOC
  • Reverted 1-3

​c1vac

  • Closed V1, VM1
  • Restarted IOC
  • Returned valves to original state

c1psl

  • Disabled IMC autolocker
  • Closed PSL shutter
  • Restarted IOC
  • Reverted 1-2

c1iscaux

  • ​Restarted IOC

c1susaux

  • Disabled IMC autolocker
  • Closed shutter
  • Disabled OPLEV servos on: MC1, MC2, MC3, BS, ITMX, ITMX, PRM, SRM
  • Zeroed slow biases
  • Disabled watchdogs
  • Restarted IOC
  • Reverted 1-5

The intial recovery of c1susaux did not succeed. Most visibly, the alignment state of the IFO was not restored. After some debugging, we found that the restart of the modbus service was partially failing at the final burt-restore stage. The latest snapshot file /opt/rtcds/caltech/c1/burt/autoburt/latest/c1susaux.snap was not found. I manually restored a known good snapshot from earlier in the day (15:19) and we were able to relock the IMC and XARM. GV and I were just talking earlier today about eliminating these burt-restores from the systemd files. I think we should.

Attachment 1: Screen_Shot_2020-03-13_at_7.59.55_PM.png
Screen_Shot_2020-03-13_at_7.59.55_PM.png
  15275   Fri Mar 13 14:28:39 2020 gautamUpdatePSLAdded tees to PMC Trans and REFL

I want to monitor the PMC TRANS and REFL levels on the PSL table - previously there were some cables going to the oscilloscope on the shelf but someone had removed these. I re-installed them just now. While there, I disconnected the drive to the AOM - there must've been some DC signal going to it because when I removed the cable, the PMC and IMC transmission were recovered to their nominal levels.

  15274   Fri Mar 13 12:48:47 2020 Larry WallaceUpdateelogCert Renewal

Updated the cert in /etc/httpd/ssl. The new cert is good until March 12, 2022.

  15273   Fri Mar 13 00:32:38 2020 gautamUpdateLSCSome progress

Finally, some RF only CARM, see Attachment #1. During this time, DARM was also on a blend of IR and ALS control, but I couldn't turn the ALS path off in ~4-5 attempts tonight (mostly me pressing a wrong button). Attachment #2 shows the CARM OLTF, with ~2kHz UGF - for now, I didn't bother turning any boosts on. PRCL and MICH are still on 3f signals.

The recycling gain is ~7-8 (so losses >200ppm), but there may be some offset in some loop. I'll look at REFL DC tomorrow.

Can we please make an effort to keep the IFO in this state for the next week or two
- it really helped tonight I didn't have to spend 2 hours fixing some random stuff and could focus on the task at hand.

Attachment 1: RFonly_CARM.png
RFonly_CARM.png
Attachment 2: CARMTF.pdf
CARMTF.pdf
  15272   Thu Mar 12 16:13:16 2020 gautamUpdatePSLTemperature sensors connected to Acromag

[jordan, gautam]

the long DB25 cable to connect the Acromag chassis to the temperature sensor interface box arrived. We laid it out today. This cable does the following:

  1. Supply the box with +/- 24 V DC from the chassis. The pin arrangement here is rather unfortunate (the +/-24 V DC and GND pins are in close proximity), so if you're not careful, you'll create a short as you plug this cable in (as we found out today). So the best practise is to power down the crate before connecting/disconnecting this cable. Jordan will label this accordingly tomorrow.
  2. Pipe the "TAMB" and "TCAV" signals, corresponding to temperature sensors mounted to the PSL table-top and reference cavity exterior respectively, to the Acromags. We found during some initial testing that the "TAMB" signal was reaching the DB25 connector on the Acromag chassis, but wasn't going to any ADC channel - this was rectified.

Both signals now show up in the EPICS channels, but are noisy - I suspect this is because the return pin of the Acromag is not shorted to ground (this is a problem I've seen on the bench before). We will rectify this tomorrow as well.

We took this opportunity to remvoe the bench supply and temporary Acromag crate (formerly known as c1psl2) from under the PSL table. While trying to find some space to store the bench supply, we came across a damaged Oscilloscope in the second "Electronics" cabinet along the Y-arm, see Attachment #1. 

After this work, I found that the IMC autolocker was reliably failing to run the mcup script at the stage where the FSS gains are ramped up to their final values. I was, however, able to smoothly transition to the low-noise locked state if I was manipulating the EPICS sliders by hand. So I added an extra 2 seconds of sleep time between the increasing of the VCO gain to the final value and the ramping of the FSS gains in the mcup script (where previously there was none). Now the autolocker performs reliably.

Attachment 1: P_20200317_153736_vHDR_On_2.jpg
P_20200317_153736_vHDR_On_2.jpg
  15271   Thu Mar 12 12:44:34 2020 gautamUpdateGeneralPMC got unlcoked

Of course the reboot wiped any logs we could have used for clues as to what happened. Next time it'll be good to preserve this info. I suspect the local subnet went down.

P.S. for some reason the system logs are priveleged now - I ran sudo sysctl kernel.dmesg_restrict=0 on c1psl to make it readable by any user. This change won't persist on reboot.

Quote:

I restarted the IOC but it didn't help.

I am now rebooting c1psl... That seemed to help. PMC screen seem to be working again. I am able to lock the PMC now.

  15270   Thu Mar 12 11:10:49 2020 YehonathanUpdateGeneralPMC got unlcoked

Came this morning to find the PMC was unlocked since 6AM. Laser is still on, but PMC REFL PD DC shows dead white constant 0V on PMC screen. All the controls on the PMC screen show constant 0V actually except for the PMC_ERR_OUTPUT which is a fast channel.

 

Is PSL Acromag already failing?

 

I restarted the IOC but it didn't help.

I am now rebooting c1psl... That seemed to help. PMC screen seem to be working again. I am able to lock the PMC now.

IMC was locking easily once some switches on the MC servo screen were put to normal states.

TTs were grossly misaligned. Onces they where aligned, arm cavities were locking easily. Dither align for the X arm is very slow though...

  15269   Thu Mar 12 10:43:50 2020 ranaUpdateLoss MeasurementArm transfer function measurement

                               when doing the AM sweeps of cavities

make sure to cross-calibrate the detectors

                       else you'll make of science much frivolities

            much like the U.S. elections electors

  15268   Thu Mar 12 01:33:21 2020 gautamUpdateLSCResuming locking activities

It's been a while since I've attempted any locking, so tonight was mostly getting the various subsystems back together.

  • Reconnected SR785 at 1Y3 to CM board for TF measurements.
  • POX/POY locking work fine
  • Locked PRMI (no ETMs) with carrier resonant, fixed PRM and BS alignment.
  • ALS-X noise is still elevated - disconnected it from the switchable delay line and now I'm directly piping the 3dB coupled part of the beat to the LO input of the demod board. But the high freq contribution to the RMS is still ~x2-x3 of what it was in November 2019. But the noise should only depend on the other (delayed) part of the beat (the discriminant is set thus).
  • With arms POX/POY locked, checked that there was no elevated coherence between POX_I/POY_I signals between 800 Hz - 1.2 kHz, which is where I see the excess noise in the laser frequency control signal (see Attachment #1). So this suggests that the IMC locking loop suppresses the noise to a level that the arm cavities don't witness it. It's probably still worth checking this out and fixing it, but it wasn't a show stopper.
  • Transition from POX/POY lock to ALS lock was smooth - I forgot to use the POX/POY photodiodes as OOL sensors to measure the noise in this config to see if it was elevated, but anyway, I was able to push on.
  • PRMI 3f locking worked okay.
  • Main thing I wanted to check today is to try the AO transition with a bit more IN1 gain on the CM board - hypothesis being the high frequency part of the CARM signal is buried in the noise if we run with -32dB of IN1 gain. 
    • Set IN1 gain to -10dB instead.
    • In this config, I checked that with the CARM offset at 0 (CARM still under purely ALS control), the CM_SLOW path was registering ~4000 cts pk-pk, which is healthily within the ADC's range.
    • I was able to engage the CARM_B path and semi-stabilize the arm powers (after compensating for the increased IN1 gain by decreasing the CARM_B gain) and turn on the integrator.
    • However, before I could try any AO path action, the IMC loses lock - too tired to try more tonight, I'll try again tomrrow.
Attachment 1: ExcessFreqNoise_LSC.pdf
ExcessFreqNoise_LSC.pdf
  15267   Wed Mar 11 21:03:57 2020 KojiUpdateBHDSOS packages from Syracuse

I opened the packages send from Syracuse.

- The components are not vacuum clean. We need C&B.
- Some large parts are there, but many parts are missing to build complete SOSs.

- No OSEMs.
- Left and right panels for 6 towers
- 3 base blocks
- 1 suspension block
- 8 OSEM plates. (1 SOS needs 2 plates)

- The parts looks like old versions. The side panels needs insert pins to hold the OSEMs in place. We need to check what needs to be inserted there.

- An unrelated tower was also included.

Attachment 1: P_20200311_203449_vHDR_On.jpg
P_20200311_203449_vHDR_On.jpg
  15266   Wed Mar 11 18:12:53 2020 gautamSummaryPSLWFS Demod board modifications

[koji, gautam]

Attachment #1 shows the relevant parts of the schematic of the WFS demod board (not whitening board). 

  • The basic problem was that the switchable gain channels were not accounted for in the Acromag channel list 😒.
  • What this meant was that the DC gain was set to the default x100 (since the two DG211s that provide the switchable x10 and x1 gain options had their control logic pins pulled up to +5V because these pins weren't connected to any sinking BIO channel).
  • Rather than set up new connections to Acromags inside the chassis (though we have plenty of spares), Koji and I decided to make these fixed to x1 gain.
  • The actual fix was implemented as shown in the annotated schematic. There are some pictures 📷 of the PCB in the DCC entry linked above.
  • Amusingly, this board will now require a sourcing BIO unit if we want to still have the capability of switching gains.

Before removing the boards from the eurocrate: 

  • I dialled down the Kepco HV supplies
  • disconnected all the cabling to these boards after noting down cable numbers etc.

After Koji effected the fix, the boards were re-installed, HV supplies were dialled back up to nominal voltage/currents, and the PMC/IMC were re-locked. The WFS DC channels now no longer saturate even when the IMC is unlocked 👏 👏 . I leave it to Yehonathan / Jon to calibrate these EPICS channels into physical units of mW of power. We should also fix the MEDM screen and remove the un-necessary EPICS channels.

Later in the evening, I took advantage of the non-saturated readbacks to center the beams better on the WFS heads. Then, with the WFS servos disabled, I manually aligned the IMC mirrors till REFLDC was minimized. Then I centered the beam on the MC2 transmission QPD (looking at individual quadrants), and set the WFS1/2 RF offsets and MC2 Trans QPD offsets in this condition.

Quote:

WFS DC channels are saturating when the IMC is unlocked.

Attachment 1: D980233-B_Mar2020Mods.pdf
D980233-B_Mar2020Mods.pdf
  15265   Wed Mar 11 16:46:25 2020 HangUpdateIOOMC2 coil balancing

My old scheme was flawed as I used pitch as the readback. The pitch signal could not distinguish the cross-coupling due to coil imbalance and that due to the natural suspension L2P. A new scheme based on yaw alone has been developed and will be integrated into ifo_test. For now we revert the C1:SUS-MC2_UL/UR/LR/LLCOIL gains back to 1, -1, 1, -1. 

Quote:

[Yehonathan, Hang]

We did some quick DC balancing of the MC2 coil drivers to reduce the l2a coupling. We updated the gains in the C1:SUS-MC2_UL/UR/LR/LLCOIL to be 1, -0.99, 0.937,-0.933, respectively. The previous values were 1, -1, 1, -1.

The procedures are the following:

Lock IMC.

Drive UL+LR and change the gain of LR to zero pitch.

Drive UR+LL and change the gain of LL to zero pitch.

Lastly, drive all 4 coils and change UR & LR together to zero yaw. 

We used C1:SUS-MC2_LOCKIN1_OSC to create the excitations at 33 Hz w/ 30,000 cts. The angular error signals were derived from IMC WFSs.

While this time we did things by hand, in the future it should be automated as the procedure is sufficiently straightforward. 

  15264   Tue Mar 10 19:59:09 2020 YehonathanUpdateLoss MeasurementArm transfer function measurement

I want to measure the transfer function of the arm cavities to extract the pole frequencies and get more insight into what is going on with the DC loss measurements.

The idea is to modulate the light using the PSL AOM. Measure the light transmitted from the arm cavities and use the light transferred from the IMC as a reference.

I tried to start measuring the X arm but the transmission PD is connected to the QPD whitening filter board with a 4 pin Lemo for which I couldn't find an adapter.

  • I switch to the Y arm where the transmission PD - Thorlabs PDA520 (250KHz Bandwidth) - is BNC all the way.
  • I lay an 82ft BNC cable from the Y Arm 1Y4 to 1Y1 where the BNC from the IMC Trans PD and an SR785 are found. 
  • I lock the Arm cavities.
  • I connect the AOM cable to the source, the TRY PD (Teed off from the QPD whitening filter) to CH1 and IMC_TRANS to CH2 and measure the transfer function using a swept sine with an offset of 300mV and amplitude of 100mV.
  • I fit it to a low pass filter function - see attachment 1 - but it seems like the fit rails off at 10KHz. 

Could this be because of the PDA520 limited BWs? I tried playing with the PD gain/bandwidth switch but it seems like it was already set to high bandwidth/low gain.

In any case, the extracted pole frequency ~ 2.9kHz implies a finesse > 600 (assuming FSR = 3.9MHz) which is way above the maximal finesse (~ 450) for the arm cavities.

I disconnected the source from the AOM. But left the other two BNCs connected to the SR785. Also, TRY PD is still teed off. Long BNC cable is still on the ground.

Attachment 1: YArmFrequencyResponse.pdf
YArmFrequencyResponse.pdf
  15263   Tue Mar 10 19:58:16 2020 yehonathanUpdateSUS 

I returned the triggering threshold to normal values (5/3).

Meanwhile, i want to block the Y arm trans PD (Thorlabs). To do it, the PD<->QPD thresholds were changed from 5.0/3.0 to 0.5/0.3.

  15262   Tue Mar 10 14:30:16 2020 yehonathanUpdateSUS 

ETMX was grossly misaligned.

I re-aligned it and the X arm now locks.

7:00PM with Koji

Both the alignment of the X and Y arms was recovered.

~>z avg 10 C1:LSC-TRX_OUT C1:LSC-TRY_OUT
C1:LSC-TRX_OUT 0.9914034307003021
C1:LSC-TRY_OUT 0.9690877735614777

We are running ass for the X arm to recover the X arm alignment.

Meanwhile, i want to block the Y arm trans PD (Thorlabs). To do it, the PD<->QPD thresholds were changed from 5.0/3.0 to 0.5/0.3.

Attachment 1: Screenshot_from_2020-03-10_19-02-31.png
Screenshot_from_2020-03-10_19-02-31.png
  15261   Sat Mar 7 15:18:30 2020 gautamUpdateSUSEQ tripped some suspensions

An earthquake around 330 UTC (=730pm yesterday eve) tripped ITMX, ITMY and ETMX watchdogs. ITMX got stuck. I released the stuck optic and re-enabled the local damping loops just now.

Attachment 1: EQ_6Mar.png
EQ_6Mar.png
  15260   Fri Mar 6 16:33:11 2020 gautamUpdateIOOExcess laser frequency noise

I did some preliminary debugging of this, and have localized the problem to the output path (after MC slow) on the IMC Servo card. Basically, I monitored the spectrum of the ALS beat frequency fluctuations under a few different conditions: 

  • With the BNC to the NPRO PZT disconnected, there is no noise. So the laser and the FSS phase correction EOM (which the ALS beat pickoff sees) are not responsible.
  • With the input to the Koji summing box disconnected, still no excess - so the summing box + Thorlabs HV driver are not responsible.
  • With the TTFSS output connected to the summing box, but with the input switch to the TTFSS box disabled (isolating the on-PSL table parts of the FSS system), still no excess.
  • With the input to the TTFSS enabled, and the BNC output of the IMC Servo card disconnected at 1X2, still no excess.
  • Finally, when I connect the BNC cable, the excess starts to show up.

Toggling C1:IOO-MC_FASTSW, which supposedly isolates the post-MC slow (a.k.a. MCL) part of the servo, I see no difference. I am also reasonably confident this switch itself works, because I can break the IMC lock by toggling it. So pending a more detailed investigation, I am forced to conclude that the problem originates in the part of the IMC servo board after the MCL pickoff. Some cabling was removed at 1X2 on Tuesday between the times when there was no excess and when it showed up, but it's hard to imagine how this could have created this particular problem.

  15259   Fri Mar 6 01:19:19 2020 gautamUpdateIOOExcess laser frequency noise

Summary:

Sometime between 1PM and 6PM on Tuesday, excess laser frequency noise shows up in MCF at around 800 Hz, as shown in Attachment #1. Sigh.

Details:

While I show the MCF spectrum here, I confirmed that this noise is not injected by the IMC loop (with the PSL shutter closed, and the IMC servo board disconnected from the feedback path to the NPRO, the PMC error and control points still show the elevated noise, see Attachment #2). I don't think the problem is from the PMC loop - see Attachment #3 which is the ALS beat out-of-loop noise with the PMC unlocked (the PSL beam doesn't see the cavity before it gets to the ALS setup, and we only actuate on the cavity length for that loop, so this wasn't even really necessary).

Was there some work on the PSL table on Tuesday afternoon that can explain this

Attachment 1: MCF.pdf
MCF.pdf
Attachment 2: ExcessFreqNoise.pdf
ExcessFreqNoise.pdf
Attachment 3: ALSnoise.pdf
ALSnoise.pdf
  15258   Fri Mar 6 01:12:10 2020 gautamUpdateElectronicsIMC Servo IN2 path looks just fine

It seems like the AO path gain stages on the IMC Servo board work just fine. The weird results I reported earlier were likely a measurement error arising from the fact that I did not disconnect the LEMO IN2 cable while measuring using the BNC IN2 connector, which probably made some parasitic path to ground that was screwing the measurement up. Today, I re-did the measurement with the signal injected at the IN2 BNC, and the TF measured being the ratio of TP3 on the board to a split-off of the SR785 source (T-eed off). Attachments #1, #2 shows the result - the gain deficit from the "expected" value is now consistent with that seen on other sliders.

Note that the signal from the CM board in the LSC rack is sent single-ended over a 2-pin LEMO cable (whose return pin is shorted to ground). But it is received differentially on the IMC Servo board. I took this chance to look for evidence of extra power line noise due to potential ground loops by looking at the IMC error point with various auxiliary cables connected to the board - but got distracted by some excess noise (next elog).

Attachment 1: AO_inputTFs_5Mar.pdf
AO_inputTFs_5Mar.pdf
Attachment 2: sliderCal_5Mar.pdf
sliderCal_5Mar.pdf
ELOG V3.1.3-