40m QIL Cryo_Lab CTN SUS_Lab TCS_Lab OMC_Lab CRIME_Lab FEA ENG_Labs OptContFac Mariner WBEEShop
  40m Log, Page 3 of 344  Not logged in ELOG logo
New entries since:Wed Dec 31 16:00:00 1969
ID Datedown Author Type Category Subject
  17230   Mon Nov 7 08:36:10 2022 JCUpdateSUSIMC out of alignment -- f2A failure

[Paco, JC]

We came in this morning and noted the IMC was grossly misaligned, with MC3 still damped but with >= 100 rms motion in all coil monitors no (a lot but not enough to trip the WD)... Turning off the WFS didn't do much so it was obviously an issue with the recent f2A output filters, so we turned all off (though only MC3 had this excess motion). After this we aligned IMC, engaged the lock and turned WFS back on.

There was no elog about f2A beyond this test scheduled to run Friday, I guess the filters were meant to stay on long term?

  17229   Fri Nov 4 15:43:05 2022 JCUpdateLab OrganizationNew Toolbox Organization

The new tool box has came in! I have spent serious time organizing these tools and making it look pretty, so please take care of it! A few things I would like to note.

  • The tool box has a power strip, so this is now where we will be charging the power tools. There is also a selected cabinet for the fully charged batteries, bits, and the power tools themselves. 
  • The cart next to the tool box had to turned 90 degrees in order to fit the tool box in. While moving this, I noticed this cart is piled with random stuff, let’s tackle this on Wednesdays clean up.
  • The tools from this tool box are specifically assigned to the VERTEX area (and the X-End until we get another toolbox.) Please return all the tools in their right tool box and cabinet after usage. 
  • The heavier duty equipment such as pipe wrenches, hammers, and power tools will remained stored here in the bottom cabinets. 

Hope you all like it!

  17228   Thu Nov 3 20:07:01 2022 AnchalSummaryBHDAS1 coil balancing required

[Anchal, Koji]

The LO phase lock that was achieved lasts for a short time because as soon as a considerable POS offset is required on AS1, the POS to PIT coupling causes the AS-LO overlap to go away. To fix this, we need to balance the coil outputs of AS1 atleast and add the f2a filters too. To follow similar method as used for IMC optics, we need a sensor for true PIT and YAW motion of AS1. Today, we looked into the possiblity of installing a QPD at BHD output path to use it for AS1, AS4, LO1, LO2, SR2, PR2 and PR3 coil strength tuning. We found a QPD which is mentioned in this elog. We found QPD interface boards setup for old MCT and MC Refl QPDs (dating before 2008). We also found the old IP-POS QPD cable between 1Y2 and BS Oplev table. We took out this cable from BS oplev end upto ITMY opleve table, put on a new DB25 connector on the ribbon cable, and connected it to the QPD on ITMY table. There is still following work to be done:

  • Move back the BHD port camera a few inches and the lends with it.
  • Put a beamsplitter in the beam going to this camera and align it to fall on the new QPD.
  • Connect the the other end of cable to QPD interface board on 1Y2.
  • Take the lemo outputs or IDC outputs from the QPD interface board to spare ADC inputs (maybe on LSC I/O chassis or SUS2 I/O chassis).
  • Make changes in RTS model to read this QPD input.
  • Enjoy balancing the coils on the 7 new suspensions.
  17227   Thu Nov 3 17:36:00 2022 AnchalUpdateSUSF2A filters on MC1, MC2, and MC3 set to test at 1 am
Quote:

 I'll setup some test of switching between different Q filters in future.

The f2A filters are set to test on IMC optics. The script used is testF2AFilters.py. The script is running on rossa in a tmux session named f2aTest. It will trigger at 1 am, Nov 4th 2022. First the script will turn off all F2A filters on IMC optics, wait for an hour, then it will try out the three F2A filter sets with different Q values, one at a time, for one hour each. So the test should last for roughly 4 hours. The gpstime stamps will be written in a logfile that can be used later to readback noise performance of IMC with different filter. The script has a try-except failsafe to revert things to original state if something fails. To stop the script from triggering or stop it during runtime, do following on rossa:

tmux a -t f2aTest
ctrl+C
exit

 

  17226   Thu Nov 3 17:01:47 2022 ChrisUpdateCDSSLwebview fixed

The simulink webview generation cronjob on megatron was not working, apparently due to a matlab license problem. I switched it to matlab 2019a (the current CDS standard), and added a script to generate medm screens from the webview. This can help with keeping hand-made screens up to date, or be a substitute for hand-made screens if the model is simple.

  17225   Thu Nov 3 16:22:11 2022 AnchalUpdateSUSMC3 coil strengths balanced

Balanced MC3 coil strengths using the same method.

Final coil strengths found:

C1:SUS-MC3_ULCOIL_GAIN: 0.942
C1:SUS-MC3_URCOIL_GAIN: -1.038
C1:SUS-MC3_LRCOIL_GAIN: 1.075
C1:SUS-MC3_LLCOIL_GAIN: -0.945

 

  17224   Thu Nov 3 16:00:58 2022 alexSummaryGeneralSensoray updates

I am currently working on getting the driver reinstalled on Donatella for the sensoray. An issue keeps arising that will not allow me to run "make" successfully in the unzipped driver folder. Will continue to remedy this.

This is why there is no light showing up on the device while plugged in. The computer does see the device, but does not show its model due to the inability for it to communicate without the driver.

 

-Alex

  17223   Thu Nov 3 14:42:57 2022 AnchalUpdateSUSMC2 coil strengths balanced

Balanced MC2 coil strengths using the same method.

Final coil strengths found:

C1:SUS-MC2_ULCOIL_GAIN: 1.074
C1:SUS-MC2_URCOIL_GAIN: -0.979
C1:SUS-MC2_LRCOIL_GAIN: 0.97
C1:SUS-MC2_LLCOIL_GAIN: -0.976

  17222   Thu Nov 3 14:00:29 2022 AnchalUpdateSUSMC1 coil strengths balanced

I balanced the face coil strengths of MC1 using following steps:

  • At all points, keep sum(abs(coil_gains)) = 4
  • After reading coil gains, remove the signs. Do the operations as below, and before writing put back the signs.
  • Butterfly to POS decoupling:
    • Drive butterfly mode at 13 Hz using LOCKIN2 on MC1 and look at C1:IOO-MC_F_DQ for position fluctuations
    • Subtract 0.05 times the BUT vector to coil strengths to see the effect on C1:IOO-MC_F_DQ using diaggui exponential averaging of 5, BW=1.
    • Use Newton-Raphson from here to reach to no POS actuation when driving butterfly mode.
  • POS to PIT decouping:
    • Drive LOCKIN2 in POS mode at 13 Hz and look for PIT signal at C1:IOO-MC_TRANS_P_DQ using diaggui exponential averaging of 5, BW=1.
    • Subtract 0.05 times the PIT vector to coil strengths
    • Use Newton-Raphson from here to reach to no PIT actuation when driving POS.
  • POS to YAW decoupling:
    • Drive LOCKIN2 in POS mode at 13 Hz and look for YAW signal at C1:IOO-MC_TRANS_Y_DQ using diaggui exponential averaging of 5, BW=1.
    • Subtract 0.05 times the YAW vector to coil strengths
    • Use Newton-Raphson from here to reach to no YAW actuation when driving POS.

By the end, I was able to see no actuation on POS when butterfly is driven with 30000 counts amplitude at 13 Hz. I was able to see no PIT or YAW actuation when POS is driven with 10000 counts at 13 Hz.

Final coil strengths found:

C1:SUS-MC1_ULCOIL_GAIN: -1.008
C1:SUS-MC1_URCOIL_GAIN: -0.98  
C1:SUS-MC1_LRCOIL_GAIN: -1.06
C1:SUS-MC1_LLCOIL_GAIN: -0.952

I used this notebook while doing the above work. It has a couple of functions that could be useful in future while doing similar balancing.

 

  17221   Wed Nov 2 16:34:56 2022 PacoSummaryCalibrationSingle arm calibration run

[Anchal, Paco]

We added a notch filter on ETMY (the actuation point of the YARM control loop) to inject our calibration line at 575.170 Hz. The excitation is injected using the DARM Oscillator, with an exc. gain of ~ 500 (this gets us a decent > 10 SNR line in the ALS Y beat). With the arm cavity locked to the PSL (~150 Hz control bandwidth), and the aux laser locked to the cavity (~10 kHz control bandwidth) the goal of this run is to calibrate our actuator strength and more importantly to budget its uncertainty. For this we have looked at the ALS beat stability using Allan statistics; we noticed the ALS beatnote frequency fluctuations start to become dominated by 1/f (or divergent noise due to systematic drifts in the YARM loop) after 10 seconds (see Attachment #1) (we have managed to see 30 seconds stability with the HEPAs off and without locking to IMC).

Our prediction is that our demodulated calibration lines will display the least residual rms noise when averaging down to around this time. This is the only reason one would use allan statistics; to quantify the separation between statistical and systematic effects in a frequency measurement. To be continued...


gpstimes:

  • 1351464997 to 1351467139
  • 1351467259 to 1351468221
  • 1351468318 to ...
  17220   Tue Nov 1 17:59:23 2022 AnchalUpdateSUSAdded F2A filters on MC1, MC2, and MC3

I've cleared all old attempts on F2A filters on MC1, MC2, and MC3, and added the default F2A filter described in the last post. I added 3 such sets of filters, with Q=3, 7, and 10. I have turned on Q=3 filter for all IMC optics right now. I'll setup some test of switching between different Q filters in future.

  17219   Tue Nov 1 17:17:27 2022 AnchalUpdateSUSModified F2A filter design and trial on MC1

After a quick discussion with Yuta, we figured that the introduction of a finite Q that Peter Fritschel does in this DCC doc T010140 for the poles pair, he should have done the same for the zeros pair as well otherwise there will be a notch at around 1 Hz. So I simply modified the filter design to have same Q for both zero pair and pole pair and got following transfer functions:

For upper coils:

T(s) = \frac{s^2 + s\omega_0\sqrt{1 + h/D}/Q + \omega_0^2(1+h/D))}{s^2 + s\omega_0/Q + \omega_0^2}

for lower coils:

T(s) = \frac{s^2 + s\omega_0\sqrt{1 - h/D}/Q + \omega_0^2(1-h/D))}{s^2 + s\omega_0/Q + \omega_0^2}

Attachment 1 shows the new filter design. I tested this filter set on MC1 and the optic kept on going as if nothing changed. That is atleast a good sign. Now next step would be test test if this actually helped in reducing the POS->PIT coupling on MC1, maybe using WFS signals.

The filters were added using this createF2Afilters.py script.

 

  17218   Tue Nov 1 15:41:18 2022 AnchalUpdateSUSF2A filter design and trial on MC1

Following discussion in this elog thread (40/6004), I used the design of F2A (force to angle(pitch)) decoupling filter as mentioned in this DCC doc T010140. This document is very useful as it talks about the overall control loops of a suspension, including sensor signal conditioning, damping filter shapes, force to pitch decoupling, pitch to position decoupling, and coil strength balancing. In future, if people are working on suspension characterization and damping, this document is a good resource to read.


Force to Angle (Pitch) decoupling filter

The document address this problem with first principle calculation using the geometry of single suspensions. As a first pass, I decided to use the design value of these geometric paramters to create a filter design for upper coils and one for lower coils. The parameters are listed in table 2. I used following:

  Description Value used
L Vertical dis-tance  from  the  suspension  point  to  the  wire  take-off  point 247.1 mm
h Pitch distance (distance above the center-of-mass of the wire take-off point) 0.9 mm
l L + h 248 mm
D Vertical distance from a magnet to the center of the optic 24.7 mm
Q Q value used in poles of the filter (doc says to use 1000) 3, 5, 10
\omega_0 Position resonant angular frequency given by \sqrt{g/l} 6.288 rad/s

Using above parameters, we can define the F2A filter for upper coils as:

T(s) = \frac{s^2 + \omega_0^2(1 + h/D)}{s^2 + s \omega_0 /Q + \omega_0^2}

and for lower coil:

T(s) = \frac{s^2 + \omega_0^2(1 - h/D)}{s^2 + s \omega_0 /Q + \omega_0^2}

I used the design values as listed in the table above and got the filters as shown in attachment 1 for Q=3 case. I think the Q is higher than what other f2a filters I have seen for example at ETMY, the filters are as shown in attachment 2.

I tried turning on MC1 f2a filters but the watchdog tripped in about 4 minutes. This was the case when WFS were turned off. Another trial also lead to the same result. I tried this on MC2 and MC3 as well, all of them tripped after som time. See attachment 3 to see MC1 tripping on these filters.

I'll now try to use a lower Q filter.

  17217   Mon Oct 31 21:04:43 2022 KojiUpdateASCWFS inspection

Inspected the lab to see what we can do about the IFO WFS:

  • WFS heads
    • 1 functional WFS head (tuned at 11/55MHz) @AS Table [40m ELOG 15736]
    • 1 WFS head case (empty) @Section Y10 below the tube, plastic box
    • 2 WFS PCBs, components stuffed, tuning freq unknown @Section Y10 below the tube, plastic box
  • Deomdulators
    • no 4ch IQ demod unit (some component possibly spare)
    • Build 4 iLIGO demods?
  • Whitening / AA
    • No permanent unit: Maybe we can borrow something from the BHD
  • ADC CHs
    • c1ioo seems to have just 8 more spare channels.
    • Borrow a card from bhd? This will require an AI. But their location would be close to the final positions.
  17216   Wed Oct 26 16:04:12 2022 PacoSummaryBHDLO phase control with RF + audio sidebands

[Yuta, Paco]

Today we again locked the LO phase with BH55 + Audio dithering under a zero-offset MICH


Configuration

We worked with MICH locked using AS55_Q with an offset = 0. Our BH55_Q_ERR is the same as in the previous elog (in this thread).We reduced the MICH offset from 50 to 0 slowly and kept an eye on the BH55 error signals. We realized that at zero offset, most of the error signal was in BH55_I_ERR (why?) so we rotated it back to BH55_Q_ERR (146 deg --> 56 deg). We then looked at the audio demod angle, and optimized it to allocated the error signal in the I quadrature (-15 deg --> 40 deg).


Lock

We close a loop with the above configuration to lock the LO phase using only filters FM5, FM8 and then optionally boost with FM2. The measured UGF ~ 20 Hz similar to the configuration with an offset present; and it seems there is some residual noise at ~ 20 Hz (observed in the residual error signal time trace with ndscope).


Next tasks:

  • Noise budget residual error in this configuration
  • Investigate negative count offset in DCPDs
  • Investigate why does the rotation angle change from single bounce to MICH?
  17215   Wed Oct 26 10:28:05 2022 PacoUpdateIOOHEPA ON / WFS ON

Turned HEPA ON this morning at 10:28 local (pacific time) or gpstime = 1350802758. WFS ON right after that. IMC was locked and nominally aligned at this time.

  17214   Tue Oct 25 22:24:46 2022 KojiUpdateIOOHEPA OFF / WFS OFF

Turned HEPA OFF / Lab Lights OFF / WFS Input Gain switched off for the IMC WFS signal tests.
The IMC is still well aligned.

Clean data from the following time:
2022/10/26 5:24:00 UTC (10/25 22:24 PDT)
GPS 1350797056

 

  17213   Tue Oct 25 22:01:53 2022 ranaConfigurationPEMAuto Z on trillium interface board

thanks, this seems to have recentered well.

It looks like it started to act funny at 0400 UTC on 10/24, so thats 9 PM on Sunday in the 40m. What was happening then?

 

  17212   Tue Oct 25 17:27:11 2022 PacoSummaryBHDLO phase control with RF + audio sidebands

[Yuta, Paco]

Today we locked LO phase with BH55 + Audio dithering


Configuration

We worked with MICH locked using AS55_Q with an offset = 50. Our BH55_Q_ERR is the same as in the previous elog (in this thread). We enabled audio dithering of AS1 to produce 280.54 Hz sidebands (exc gain = 15000). We used ELP80 (elliptic, 4th order lowpass with the second resonant notch at 280.54 Hz) at the BH55_Q_AS1_DEMOD_I output. This allowed us to generate an error signal to feedback into AS1 POS. Attachment #1 shows a screen capture of this configuration.


Lock

We close a loop with the above configuration to lock the LO phase using only filters FM5, FM8 and then optionally boost with FM2. The compromise we had to make because of our phase margin was to achieve UGF ~ 20 Hz (in contrast with ~ 70 Hz used in single bounce). Attachment #2 shows the measured OLTFs for LO_PHASE control using this scheme; the red was the final measured loop, while the blue was our initial reference before increasing the servo gain.

 

  17211   Tue Oct 25 14:29:56 2022 PacoSummarySEIEarthquake tripped SUS

[Yuta, Paco, JC]

This eq  potentially tripped ETMY, PR2, PR3, AS1, AS4, SR2, LO1, LO2 suspensions during today's WB meeting. We restored them into normal local damping.


We aligned the arm cavities just to verify things were ok and then moved on to BHD comissioning. No problems spotted so far.

  17210   Tue Oct 25 13:55:37 2022 KojiConfigurationPEMAuto Z on trillium interface board

This nicely brought the sensing signal back to ~zero. See attachment


Some basic info:

  • BS Seismometer is T240 (Trillium)
  • The interface unit is at 1X5 Slot 26. D1002694
  • aLIGO Trillium 240 Interface Quick Start Guide T1000742
  17209   Tue Oct 25 09:57:34 2022 PacoConfigurationPEMAuto Z on trillium interface board

I pressed the Auto-Z(ero) button for ~ 3 seconds at ~9:55 local (pacific) time on the trillium interface on 1X5.

  17208   Tue Oct 25 08:25:00 2022 JCUpdateBHDBHD fringe aligned with reduced LO and AS beam clipping

I aligned today using this scheme. I couldn't seem to get C1:IOO-MC_TRANS_SUM above 13400 by using WFS or manually aligning. The original state before was the following:
                  Pitch       Yaw     
C1:SUS-MC1: 
    -0.4672     -0.7714
C1:SUS-MC2:      4.0446     -1.3558
C1:SUS-MC3:     -2.0006      1.6001

 

  17207   Tue Oct 25 05:26:24 2022 ranaSummaryIOOMC SUS tuning

in looking closer at the IMC WFS performance I notice 2 issues:

  1. the watchdog thresholds are set to 200 mV, but this only made sense back when the OSEM calibration was 2 V/mm. Because of the increased analog gain(?) the RMS value of the watchdog sensors is now ~35 mV for MC1, so it will trip its watchdog and unlock the IMC with a 10x smaller seismic impulse than before. I recommend changing the watchdog thresholds appropriately changing the OSEM sensor signals to something so that its the same for all SUS.
  2. For the MC SUS, the F2P or F2A decoupling filters are not on. SO the POS damping loop is injecting a lot of pitch noise into the mirrors. We could perhaps lower the ~1 Hz angular motion by commissioning those filters for the MC optics. Does anyone know why we have several filters in those filter banks? I think we could contain it all in 1, although its fine to make a few different ones with different Q's for testing the performance.
  17206   Mon Oct 24 18:01:00 2022 PacoSummaryBHDBHD actuation measurements

[Yuta, Paco]

Today we calibrated the actuation on BHD suspended optics: LO1, LO2, AS1, AS4.
Actuation transfer functions for these optics look good.


ITMY actuation

For a reference we locked LO-ITMY single bounce using the LSC MICH loop. The error point was BH55_Q, the whitening filter gain was 45 dB, IQ demod rotation angle = 151.061 deg, the servo gain was -10, and the actuation point was ITMY. The measured UGF for this loop was ~ 150 Hz when FM2, 3, 4, 5 and 8 were all enabled. Note FM8 is an elliptic low pass (600 Hz cutoff).


LO1, LO2, AS1, AS4 actuation

We then lock the LO phase by feeding back BH55_Q_ERR to the actuation points under test with exactly the same filters but a servo gain of 0.6 but otherwise we are using the same servo filters FM2, 3, 4, 5 and 8 for this controls. The measured UGFs were all near ~ 70 Hz.

Here we had to be careful not to excite mechanical (?) resonances similar to the previously observed "violin" modes in LO1. In particular, we first noticed unsupressed 816 Hz noise in AS1 was being reinjected by the loop sometimes tripping the local damping loops, so we added bandstop filters at the AS1_LSC output filter bank. The resulting loop was then allowed to increase the gain and turn on FM2 and FM3 (boosts). This was also the case in AS4, where 268 Hz and second + third harmonics appeared to be excited by our feedback control. Finally, AS4 also displayed some mechanical excitation at 96.7 Hz, which seemed too low to be a "violin" mode, and its "Q" factor was not as high. We added a bandstop for this as well.

Attachment #1 shows LO_PHASE OLTFs when actuating in the different optics. By taking the actuation ratios (Attachment #2) with respect to our ITMY actuation reference and which had previously been calibrated to be 4.74e-9 / f^2 m / cts, we now have estimated our BHD suspension actuation calibrations to be:

  • LO1 = 3.14e-8 / f^2 m / cts
  • LO2 = 2.52e-8 / f^2 m / cts
  • AS1 = 3.14e-8 / f^2 m / cts
  • AS4 = 2.38e-8 / f^2 m / cts

This magnitudes are consistent with the expected coil driver ranges (about a factor of 10 difference).

  17205   Sat Oct 22 21:36:28 2022 ranaSummaryBHDBH55 phase locking efforts

give us an animated GIF of this cool new tool! - I'm curious what happens if you look at 2 DoF of the same suspension. Also would be cool to apply a bandpass filter before plotting XY, so that you could look for correlations at higher frequencies, not just seismic noise

Quote:
 

Using xyplot tool, we tried to see the relationship between C1:HPC-BHDC_DIFF_OUT and C1:HPC-BH55_Q_DEMOD_I_OUT. The two signals, according to our theory, should be 90 degrees out of phase and should form an ellipse on XY plot. But what we saw was basically no correlation between the two.

 
  17204   Fri Oct 21 16:15:10 2022 yutaSummaryBHDLO phase locking with BH55 audio dither trials

[Paco, Yuta]

We are still struggling with locking LO phase in MICH or ITM single bounce with BH55 with audio dither.
Without audio dither, BH55 can be used to lock.

What works:
 - LO phase locking with ITMX single bounce, using BH55_Q
  - BH55_Q configuration: 45 dB whitening gain, with whitening filter on.
  - C1:LSC-BH55_PHASE_R=147.621 deg gives most signal in BH55_Q.
  - LO phase can be locked using BH55_Q, C1:HPC-LO_PHASE_GAIN=-0.5 (bright fringe for A, dark for B), feeding back to LO1 gives UGF of ~80Hz (funny structure in ~20 Hz region; see Attachment #1)

 - LO phase locking with ITMX single bounce, using BHDC_DIFF
  - BHDC B/A = 1.57 (gain balanced with C1:HPC-IN_MTRX)
  - LO phase can be locked using BHDC_DIFF, C1:HPC-LO_PHASE_GAIN=-0.4 (mid-fringe lock), feeding back to LO1 gives UGF of ~50 Hz (see Attachment #2).

 - LO phase locking with MICH locked with AS55_Q, using BH55_Q
  - AS55_Q configuration: 24 dB whitening gain, with whitening filter off
  - C1:LSC-AS55_PHASE_R=-150 deg gives most signal in AS55_Q
  - MICH can be locked using AS55_Q, C1:LSC-MICH_GAIN=-10, C1:LSC-MICH_OFFSET=30 (slightly off from AS dark fringe), feeding back to 0.5*BS gives UGF of ~100Hz (see Attachment #3)
  - LO phase can be locked using BH55_Q, C1:HPC-LO_PHASE_GAIN=-0.8 (bright fringe for A, dark for B), feeding back to LO1 gives UGF of ~45Hz (see Attachment #4)

 - LO phase locking with MICH locked with AS55_Q, using BHDC_DIFF
  - LO phase can be locked using BHDC_DIFF, C1:HPC-LO_PHASE_GAIN=1 (mid-fringe lock), feeding back to LO1. Not a very stable lock.

What does not work:
 - LO phase locking using BH55_Q demodulated at LO1 (or AS1) dither frequency, neither in ITMX sigle bounce or MICH locked with/without offset using AS55_Q
  - C1:HPC-AS1_POS_OSC_FREQ=142.7 Hz, C1:HPC-AS1_POS_OSC_CLKGAIN=3000, C1:HPC-BH55_Q_AS1_DEMOD_PHASE=-15 deg, BLP30 is used.
  - Attachment #5 shows error signals when LO phase is locked with BH55_Q. BHDC_DIFF and BH55_Q_AS1_DEMOD_I having some coherence is a good indication, but we cannot lock LO phase with BH55_Q_AS1_DEMOD_I yet.
  - Also, injection at 13.14 Hz with an amplitude of 300 for AS1 can be seen in both BH55_Q and BH55_Q_AS1_DEMOD_I (26 Hz peak for BHDC_DIFF, as it is quadratic, as expected), which means that BH55_Q_AS1_DEMOD_I is seeing something.

Next:
 - Check actuation TFs for LO1, LO2, AS1 too see if there are any funny structures at ~ 20 Hz.
 - LO phase locking might require at least ~50 Hz of UGF. Use higher audio dither frequency so that we can increase the control bandwidth.
 - Check analog filtering situation for BHDC_A and BHDC_B signals (they go minus when fringes are moving fast)

  17203   Fri Oct 21 10:37:36 2022 AnchalSummaryBHDBH55 phase locking efforts

After the amplifier was modified with a capacitor, we continued trying to approach locking LO phase to in quadrature with AS beam. Following is a short summary of the efforts:

  • To establish some ground, we tested locking MICH using BH55_Q instead of AS55_Q. After amplification, BH55_Q is almost the same level in signal as AS55_Q and a robust lock was possible.
  • Then we locked the LO phase using BH55_Q (single RF sideband locking), which locks the homodyne phase angle to 90 degrees. We were able to successfully do this by turning on extra boost at FM2 and FM3 along with FM4 and FM5 that were used to catch lock.
  • We also tried locking in a single ITMY bounce configuration. This is a Mach-Zehnder interferometer with PR2 acting as the first beam splitter and BHDBS as the recombination beamsplitter. Note that we failed earlier at this attempt due to the busted demodulation board. This lock worked as well with single RF demodulation using BH55_Q.
  • The UGF achieved in the above configurations was ~15 Hz.
  • In between and after the above steps, we tried using audio dither + RF sideband, and double demodulation to lock the LO phase but it did not work:
    • We could see a good Audio dither signal at 142.7 Hz on the BH55_Q signal. SNR above 20 was seen.
    • However, on demodulating this signal and transferring all signal to C1:HPC-BH55_Q_DEMOD_I_OUT, we were unable to lock the LO phase.
    • Using xyplot tool, we tried to see the relationship between C1:HPC-BHDC_DIFF_OUT and C1:HPC-BH55_Q_DEMOD_I_OUT. The two signals, according to our theory, should be 90 degrees out of phase and should form an ellipse on XY plot. But what we saw was basically no correlation between the two.
    • Later, I tried one more thing. The comb60 filter on BH55 is not required when using audio dither with it, so I switched it off.
      • I turned off comb60 filters on both BH55_I and BH55_Q filter modules.
      • I set the audio dither to 120 Hz this time to utilize the entire 120 Hz region between 60 Hz and 180 Hz power line peaks.
      • I changed the demodulation low pass filter to 60 Hz Butterworth filter. I tried using 2nd order to lose less phase due to this filter.
      • These steps did not fetch me any different results than before, but I did not get a good time to investigate this further as we moved into CDS upgrade activities.
  17202   Thu Oct 20 19:47:24 2022 TegaUpdateCDSRelocate front-ends to Rack 1X7

We mounted chiara, all front-end machines and switches in rack 1x7; reconnected power, dolphin, onestop and timing cables; and restarted the front-ends. Attachments 1 & 2 show the front and rear of rack 1X7. We are going to continue the clean up work tomorrow.

 

The ifo is not back up as can be seen in attachment 3. We think the timing issue mentioned earlier is the culprit, but all FEs seem to agree to within a second, so I am not sure. I restarted the models with iop errors other than the timing error, i.e. c1lsc, c1sus, c1ioo and c1iscex. This eliminated most of the errors but the timing error. However, the overflow field on c1lsc is non-zero and the number keeps increasing - indicating a problem with c1lsc? The new status is shown in attachement 4. My understanfin is the a red `TIM` flag in the CDS stateword is not a functional problem, so I guess we are almost there. I did a burt restore on rossa using a snapshot we took earlier today before the shutdown, reset the SUS watchdogs and started the docker services on optimus. Now the IMC is locked.

 

We are still getting shared memory glitch on c1ioo, see attachment 5.


Note: We left nodus, megatron, optimus and fb1 in rack 1X6 for now.

 

  17201   Thu Oct 20 14:13:42 2022 yutaSummaryPSLPMC and IMC sideband frequencies measured

I measured the sideband frequencies for PMC and IMC lock (to use it for Mariner PMC and IMC design).

PMC: 35.498912(2) MHz
IMC: 29.485038(2) MHz

Details:
 - Mini-Circuits UFC-6000 was used. The spec sheet says the frequency accuracy in 1-40 MHz is +/- 2 Hz.
 - "29.5 MHz OUT" port of 40m Frequency Generation Unit (LIGO-T1000461) was connected to UFC-6000 to measure IMC sideband frequency.
 - "LO TO SERVO" port of Crystal Frequency Ref (LIGO-D980353) was connected to UFC-6000 to measure PMC sideband frequency.
 - It seems like IMC sideband frequency changed from 29.485 MHz to 29.491 MHz back in 2011 (40m/4621). We are back to 29.485 MHz. I'm not sure what happened after this.

  17200   Wed Oct 19 11:09:20 2022 RadhikaUpdateBHDBH55 RF output amplified

[Anchal, Radhika]

We selected a 102K (1 nF) ceramic capacitor and a 100 uF electrolytic capacitor for the RF amplifier power pins. I soldered the connections and reinstalled the amplifier [Attachments 1, 2].

Quote:

1) please remember to follow the loading and power up instructions to avoid destroying our low noise RF amplifiers. Its not as easy as powering up any usual device.

2) also, please use the correct decoupling capacitors at the RF amp power pins. Its going to have problems if its powered from a distant supply over a long cable.

 

  17199   Wed Oct 19 09:48:49 2022 AnchalUpdateOptimal ControlIMC open loop noise monitor

Turning WFS loops back on at:

PDT: 2022-10-19 09:48:16.956979 PDT
UTC: 2022-10-19 16:48:16.956979 UTC
GPS: 1350233314.956979

  17198   Tue Oct 18 20:43:38 2022 AnchalUpdateOptimal ControlIMC open loop noise monitor

WFS loops were running for past 2 hours when I made the overall gain slider zero at:

PDT: 2022-10-18 20:42:53.505256 PDT
UTC: 2022-10-19 03:42:53.505256 UTC
GPS: 1350186191.505256

The output values are fixed to a good alignment. IMC transmission is about 14100 counts right now. I'll turn on the loop tomorrow morning. Data from tonight can be used for monitoing open loop noise.

  17197   Tue Oct 18 15:24:48 2022 TegaUpdateCDSRack 1X7 work proposal

We have decided to split the remaining CDS work on rack 1x7 into two phases, both of which end with us bringing the 40m systems back up.

 

Phase 1 (Clear rack 1X7 of all mounted pieces of equipment)

  • Move nodus to 1X6 bottom slot
  • Move optimus to 1X6 (replaces old fat FB which can be moved to storage)
  • Move DAQ and network switches to the top of 1X7 rack
  • Move the UPS to under 1X6
  • Clear 1X7 power rail of any connections

 

Phase 2 (Replace the mounting rails and mount all pieces of equipment)

  • Mount DAQ, network and Dolphin switches at the top rear of 1x7 rack
  • Mount 6 new front-ends
  • Mount KVM switch
  • Move nodus back to 1X7
  • Move optimus back to 1X7

 

  17196   Mon Oct 17 22:27:25 2022 ranaUpdateBHDBH55 RF output amplified

1) please remember to follow the loading and power up instructions to avoid destroying our low noise RF amplifiers. Its not as easy as powering up any usual device.

2) also, please use the correct decoupling capacitors at the RF amp power pins. Its going to have problems if its powered from a distant supply over a long cable.

  17195   Mon Oct 17 20:04:16 2022 AnchalUpdateBHDBH55 RF output amplified

[Radhika, Anchal]

We have added an RF amplifier to the output of BH55. See the MICH signal on BH55 outputs as compared to AS55 output on the attached screenshot.

Quote:

Next steps:

- Amplify the BH55 RF signal before demodulation to increase the SNR. In order to power an RF amplifier, we need to use a breakout board to divert some power from the DB15 cable currently powering BH55.

 


Details:

  • Radhika first tried to use ZFL-500-HLN+ amplifier taken out from the amplifier storage along X-arm.
  • She used a DB15 breakout board to source the amplifier power from PD interface cable.
  • However, she reported no signal at the output.
  • We found that BH55 RFPD was not properly fixed tot eh optical table. We bolted it down properly and aligned the beam to the photodiode.
  • We still did not see any RF output.
  • I took over from Radhika on this issue. I tested the transfer function of the amplifier using moku:lab. I found that it was not amplifying at all.
  • I brought in a beanchtop PS and tested the amplifier by powering it directly. It drew 100 mA of current but showed no amplififcation in transfer function. The transfer function was constant at -40 dB with or without the amplifier powered.
  • I took out another RF amplifier from the same storage. This time a ZFL-1000-LN. I tested it with both benchtop PS and PD interface power source, it was wokring with 20 dB amplification.
  • I completed the installation and cable management. See photos attached.
  • I also took the opportunity to center the ITMY oplev.

Please throw away malfunctioning parts or label them malfunctioning before storing them with other parts. If we have to test each and every part before installation, it will waste too much of our time.

 

  17194   Mon Oct 17 17:42:35 2022 JCHowToOPLEV TablesIMC Reflected beam sketch

I sketched up a quick drawing with estimated length for the IMC reflected beam. This includes the distances and focal length. Distances are from optic to optic. 

  17193   Mon Oct 17 11:57:27 2022 ChrisOmnistructureCDSTiming system monitoring

I started a GPS timing monitor script, /opt/rtcds/caltech/c1/Git/40m/scripts/cds/tempusTimingMon/tempusTimingMon.py, which runs inside a docker container on optimus. It accesses the GPS receiver (EndRun Tempus LX) status via pysnmp, and creates the following epics channels with pcaspy:

  • C1:DAQ-GPS_TFOM “The Time Figure of Merit (TFOM) value ranges from 3 to 9 and indicates the current estimate of the worst case time error. It is a logarithmic scale, with each increment indicating a tenfold increase in the worst case time error boundaries.” (nominally 3, corresponding to 100 nsec)
  • C1:DAQ-GPS_STATE “Current GPS signal processor state. One of 0, 1 or 2, with 0 being the acquisition state and 2 the fully locked on state.”
  • C1:DAQ-GPS_SATS “Current number of GPS satellites being tracked.”
  • C1:DAQ-GPS_DAC “Current 16 bit, Voltage Controlled TCXO DAC value. Typical range is 20000 to 40000, where more positive numbers have the effect of raising the TCXO frequency.”
  • C1:DAQ-GPS_SNR “The current average carrier to noise ratio of all tracked satellites, in units of dB. Values less than 35 indicate weak signal conditions.”

Attached is a 1-day trend of the above channels, along with the IRIG-B timing offsets from the IOPs. No big timing excursions or slippages were seen yet, although the c1sus IOP (FEC-20) timing seems to be hopping around by one IOP sample time (15 µsec).

  17192   Sat Oct 15 17:22:56 2022 ChrisUpdateOptimal ControlIMC alignment controller testing

[Anchal, Radhika, Jamie, Chris]

We conducted a test of three alternative controllers for the IMC pitch DOFs on Friday. These were loaded into a new RTS model c1sbr, which runs on the c1ioo front end as a user-space program at 256 Hz. It communicates with the c1ioo controller via shared memory IPCs to exchange error and control signals.

The IMC maintained lock during the handoffs, and we were able to take one minute of data for each (circa GPS 1349807926, 1349808426, 1349808751; spectra attached), which we can review to assess the performance vs the baseline. (On the first trial, lock was lost at the end when the script tried to switch back to the baseline controller, because we did not take care to clear the integrators. On subsequent trials we did that part by hand.)

The method of setting up this test was convoluted, but now that we see it working, we can start putting in the merge requests to get the changes better integrated into the system. First, modifications were required to the realtime code generator, to get controllers running at the new sample rate of 256 Hz. (This was done in a separate filesystem image on fb1, /diskless/root.buster256, which is only loaded by c1ioo, so as to isolate the changes from the other front end machines.) The generated code then needed hand-edits to insert additional header files and linker options, so that the alternative controllers could be loaded from .so shared libraries. Also, the kernel parameters had to be set as described here, to allow the user-space controller to have a CPU core all to itself. Finally, isolating the core was done following the recipe in this script (skipping the parts related to docker, since we didn’t use it).

  17191   Fri Oct 14 17:04:28 2022 RadhikaUpdateBHDBH55 Q abnormality + fix

[Yuta, Anchal, Radhika]

Yesterday we attempted to lock MICH and BHD using the BH55_Q_ERR signal. We adjusted the demodulation phase to send the bulk of the error signal to the Q quadrature. With the LO beam misaligned, we first locked MICH with AS55_Q_ERR. We tried handing over the feedback signal to BH55_Q_ERR, which in theory should have been equivalent to AS55_Q_ERR. But this would not reduce the error and would instead break the MICH lock. Qualitatively the BH55_Q signal looked noisier than AS55_Q.

We used the Moku:Lab to send a 55 MHz signal into the demod board, replacing the BH55 RF input [Attachment 1]. The frequency was chosen to be 10 Hz  away from the demodulation frequency (5x Marconi source frequency). However, a 10Hz peak was not visible from the spectra - instead, we observed a 60 Hz peak. Tweaking the frequency offset a few times, we realized that there must be a ~50Hz offset between the Moku:Lab and the Marconi.

We generated an X-Y plot of BH55_Q vs. AS55_DC with the MICH fringe: this did not follow a circle or ellipse, but seemed to incoherently jump around. Meanwhile the X-Y plot BH55_I vs. AS55_DC looked like a coherent ellipse. This indicated that something might have been wrong with the demod board producing the I and Q quadrature signals.

We fed the BH55 RF signal into an unused demod board (previously AS165) [Attachment 2] and updated the channel routing accordingly. This step recovered elliptical I and Q signals with Moku input signal, and their relative gain was adjusted to produce a circle X-Y plot [Attachment 3]. C1:LSC-BH55_Q_GAIN was adjusted to 155.05/102.90=1.5068, and measured diff C1:LSC-BH55_PHASE_D was adjusted to 94.42 deg.

Now BH55_Q_ERR was able to be used to lock the MICH DOF. However, BH55 still appears to be noisy in both I and Q quadratures, causing the loop to feedback a lot of noise.

Next steps:

- Amplify the BH55 RF signal before demodulation to increase the SNR. In order to power an RF amplifier, we need to use a breakout board to divert some power from the DB15 cable currently powering BH55.

  17190   Thu Oct 13 23:52:45 2022 KojiUpdateIMCIMC ASC: summary pages and notes

The output matrices have been calculated on Aug 4, 2022 by me. [40m ELOG 17060]

Regarding the noise see [40m ELOG 17061]

With regard to the current IMC WFS design, a SURF student in 2014 (Andres Medina) and Nick Smith made the revision.
The telescope design was described in the elogs [40m ELOG 10410] [40m ELOG 10427] and also T1400670.

  17189   Thu Oct 13 23:25:22 2022 ranaUpdateIMCIMC ASC: summary pages and notes

Tega has kindly made a summary page for the IMC WFS. Its in a tab on the usual summary pages.

One thing I notice is that the feedback to MC2 YAW seems to have very little noise. What's up with that?

The output matrix (attached) shows that the WFS have very little feedback to MC2 in YAW, but normal feedback in PIT. Has anyone recalculated this output matrix in the past ~1-2 years?

I'm going to read Prof. Izumi's paper (https://arxiv.org/abs/2002.02703) to get some insight.

The output matrix doesn't seem to have any special thing to make this happen. Any ideas on what this could be?

  17188   Thu Oct 13 19:06:42 2022 KojiUpdateLab OrganizationLab Cleanup 10/12/2022

I have moved the following electronics / components to "Section Y10 beneath the tube"

  • IQ Demod Spares D0902745 (Components)
  • Sat Box / HAM-A 40m parts D08276/D110117
  • 16bit DAC AI Rear PCB D070101-v3
  • D1900163 HV COIL DRIVER
  • Ribbon Cables for Chassis (Cardboard box)
  • Chassis DC Breaker Switches (Cardboard box)
  • Triplett HDMI displays (x3) / Good for portable CCD monitors / PC monitors. Battery powered!
  • ISC Whitening BI Config Boards D1001631/D1900166
  • AA/AI Untested D070081
  • WFS Interface / Soft Start D1101865/D1101816
  • Internal Wiring Kit Cable Spools
  • ISC Whitening / Interface D1001530 / D1002054
  • aLIGO WFS Head D1101614
  • Internal Wiring Kit (Large Plastic Box)
  • Front/Rear Panels (Large Plastic Box)
  • HV Coil Driver Test Kit / Spare PA95s (Large Plastic Box)

 

  17187   Thu Oct 13 14:46:34 2022 JCUpdateLab OrganizationLab Cleanup 10/12/2022

During Wednesday’s lab clean up, we made a ton of progress in organization. Our main focus was to tackle CDS debris from the ongoing upgrade. We proceeded with the following tasks.

  • Loop the OneStop Cables and mark ‘good’ or ‘bad’.
  • Clean materials from the PD testing table.
  • Removed the SuperMicro boxes from the lab
  • Vacuum area organization.
  • Remove old plastic containers from the laboratory.
  • Relocate Koji’s electronics underneath Y-Arm
  • Arrange cabling to the TestStand and create clearance to the Machine Shop/Laboratory exit.

Attachment #2 shows that all the CDS equipment has been relocated behind Section X3 of the X-Arm.

  17186   Wed Oct 12 18:18:16 2022 ranaUpdateASSModel Changes

Although we've usually used this empirical way to run the alignment, I'd prefer if we had an analytic / numerical model for it.

Radhika, coud you look into writing down the equations for how the various dithers show up in the various error signals into a Overleaf doc? I'm thinking about this currently for the IMC, so we can zoom about it next week once you've had a chance to think about it for a few days. It would be helpful to have this worked out for the 40m alignment for future debugging. Co-located dither and actuation is not likely the best way to do things from a noise and loop x-coupling point of view.

 

  17185   Wed Oct 12 14:01:23 2022 RadhikaUpdateASSModel Changes

[Anchal, Radhika]

We proceeded with the TODO items from [17166].

We tried to update the YARM ASS output matrix to appropriately feed back the ETM and ITM T error signals (input beam pointing) to actuate on PR2 and PR3. Using the existing matrix (used for actuating on TT1 and TT2) led to diverging error signals and big drops in transmission. We iteratively tried flipping signs on the matrix elements, but exhausting all combinations of parity was not efficent, since angular sign conventions could be arbitrary across optics. 

We decided to go ahead with Yuta's suggestion of dithering on PR2 and PR3 for input beam pointing, instead of ETMY and ITMY. This would simplify the output matrix greatly since dithering and actuation would now be applied to the same optics. Anchal made the necessary model changes. We tried a diagonal identity submatrix (for input pointing) to map each error signal to the corresponding DOF. With the length (L) control loops disengaged, this configuration decreased all T error signals and increased YARM transmision. We then re-engaged the L loops: the final result is that YARM transmission reached just below 1 [Attachment 1].

  17184   Tue Oct 11 16:52:42 2022 AnchalUpdateIOORenaming WFS channels to match LIGO site conventions

[Tega, Anchal]

c1mcs and c1ioo models have been updated to add new acquisition of data.


IOO:WFS channels

We found from https://ldvw.ligo.caltech.edu/ldvw/view that following channels with "WFS" in them are acquired at the sites:

  • :IOO-WFS1_IP
  • :IOO-WFS1_IY
  • :IOO-WFS2_IP
  • :IOO-WFS2_IY

These are most probably error signals of WFS1 and WFS2. At 40m, we have following channel names instead:

  • C1:IOO-WFS1_I_PIT_OUT
  • C1:IOO-WFS1_I_YAW_OUT
  • C1:IOO-WFS2_I_PIT_OUT
  • C1:IOO-WFS2_I_YAW_OUT

And similar for Q outputs as well. We also have chosen quadrature signals (signals after sensing matrix) at:

  • C1:IOO-WFS1_PIT_OUT
  • C1:IOO-WFS1_YAW_OUT
  • C1:IOO-WFS2_PIT_OUT
  • C1:IOO-WFS2_YAW_OUT

We added following testpoints and are acquiruing them at 1024 Hz:

  • C1:IOO-WFS1_IP  (same as C1:IOO-WFS1_I_PIT_OUT)
  • C1:IOO-WFS1_IY  (same as C1:IOO-WFS1_I_YAW_OUT)
  • C1:IOO-WFS2_IP  (same as C1:IOO-WFS2_I_PIT_OUT)
  • C1:IOO-WFS2_IY  (same as C1:IOO-WFS2_I_YAW_OUT)

IOO-MC_TRANS

For the transmission QPD at MC2, we found following acquired channels at the site:

  • :IOO-MC_TRANS_DC
  • :IOO-MC_TRANS_P
  • :IOO-MC_TRANS_Y

We created testpoints in c1mcs.mdl to add these channel names and acquire them. Following channels are now available at 1024 Hz:

  • C1:IOO-MC_TRANS_DC
  • C1:IOO-MC_TRANS_P
  • C1:IOO-MC_TRANS_Y

We started acquiring following channels for the 6 error signals at 1024 Hz:

  • C1:IOO-WFS1_PIT_IN1
  • C1:IOO-WFS1_YAW_IN1
  • C1:IOO-WFS2_PIT_IN1
  • C1:IOO-WFS2_YAW_IN1
  • C1:IOO-MC2_TRANS_PIT_IN1
  • C1:IOO-MC2_TRANS_YAW_IN1

We started acquiring following 6 control signals at 1024 Hz as well:

  • C1:IOO-MC1_PIT_OUT
  • C1:IOO-MC1_YAW_OUT
  • C1:IOO-MC2_PIT_OUT
  • C1:IOO-MC2_YAW_OUT
  • C1:IOO-MC3_PIT_OUT
  • C1:IOO-MC3_YAW_OUT

RXA: useful to know that you have to append "_DQ" to all of the channel names above if you want to find them with nds2-client.


Other changes:

In order to get C1:IOO-MC_TRANS_[DC/P/Y], we had to get rid of same named EPICS output channels in the model. These were been acquired at 16 Hz before this way. We then updated medm screens and autolocker config file. For slow outputs of these channels, we are using C1:IOO-MC_TRANS_[PIT/YAW/SUMFILT]_OUTPUT now. We had to restart daqd service for changes to take effect. This can be done by sshing into fb1 and running:

sudo systemctl restart rts-daqd

Now there is a convinient button present in FE overview status medm screen to restart DAQD service by a simple click.

  17183   Tue Oct 11 11:27:58 2022 KojiUpdateCDSCDS Upgrade remaining issues

Perhaps 1PPS around the PSL was used for the Rb standard to be locked to the GPS 1PPS.

If we need to drive multiple devices, we should use a fanout circuit to avoid distorting the 1PPS. -CW

  17182   Tue Oct 11 10:58:36 2022 ChrisUpdateCDSCDS Upgrade remaining issues

The original fb1 now boots from its new drive, which is installed in a fixed drive bay and connected via SATA. There are no spare SATA power cables inside the chassis, so we’re temporarily powering it from an external power supply borrowed from a USB to SATA kit (see attachment).

The easiest way to eliminate the external supply would be to use a 4-pin Molex to SATA adapter, since the chassis has plenty of 4-pin Molex connectors available. Unfortunately those adapters sometimes start fires. The ones with crimped connectors are thought to be safer than those with molded connectors. However, the safest course will probably be to just migrate the OS partition onto the 2 TB device from the hardware RAID array, once we’re happy with it.

Historic data from the past frames should now be available, as should NDS2 access.

Starting to look into the timing issue:

  • The GPS receiver (EndRun Tempus LX) seems to have low signal from its antenna: it’s sometimes losing lock with the satellites. We should find a way to log the signal strength and locked state of this receiver in EPICS and frames (perhaps using its SNMP interface).
  • There was a BNC tee attached to the 1PPS output of the receiver. One cable was feeding the timing system, and another one went somewhere into the PSL (if I traced it correctly?). I removed the tee and connected the timing system directly to 1PPS.
  • I updated the firmware on the Tempus LX (but don’t expect this to make any difference)
  17181   Mon Oct 10 10:14:05 2022 ChrisUpdateCDSCDS Upgrade remaining issues

List of remaining tasks to iron out various wrinkles with the upgrade:

The situation with the timing system is that we have not touched it at all in the upgrade, but have added a new diagnostic: Spectracom timing boards in each FE, to compare vs the ADC clock. So I expect that what we’re seeing now is not new, but may not have been noticed previously.

What we’re seeing is:

  • Timing status in the IOP statewords is red
  • The offset between FE timing and Spectracom is large: ~1000 µsec or greater. At the sites, this is typically much smaller, like 10 µsec
  • The offset is not stable (see first attachment). There are both excursions where it wanders off by tens of µsec and then comes back, as well as slips where it runs away by hundreds of µsec before settling down at a different level.

Possible issues stemming from unstable timing:

  • One of the timing excursions apparently glitched the ADC clock on c1ioo and c1sus2. Those IOPs are currently running with cycle time >15 µsec (see second attachment). This may mean the inputs on certain ADCs are delayed by an extra sample time, relative to the other ADCs
  • If the offset drifts too far, a discrepancy in the timestamps can glitch the IPCs and daqd
ELOG V3.1.3-