40m QIL Cryo_Lab CTN SUS_Lab TCS_Lab OMC_Lab CRIME_Lab FEA ENG_Labs OptContFac Mariner WBEEShop
  40m Log, Page 217 of 339  Not logged in ELOG logo
ID Date Authorup Type Category Subject
  14275   Tue Nov 6 15:23:48 2018 gautamUpdateIOOIMC problematic

The IMC has been misbehaving for the last 5 hours. Why? I turned the WFS servos off. afaik, aaron was the last person to work on the IFO, so i'm not taking any further debugging steps so as to not disturb his setup.

Attachment 1: MCwonky.png
MCwonky.png
  14279   Tue Nov 6 23:19:06 2018 gautamUpdateVACc1vac1 FAIL lights on (briefly)

Jon and I stuck a extender card into the eurocrate at 1X8 earlier today (~5pm PT), to see if the box was getting +24V DC from the Sorensen or not. Upon sticking the card in, the FAIL LEDs on all the VME cards came on. We immediately removed the extender card. Without any intervention from us, after ~1 minute, the FAIL LEDs went off again. Judging by the main volume pressure (Attachment #1) and the Vacuum MEDM screen (Attachment #2), this did not create any issues and the c1vac1 computer is still responsive.

But Steve can perhaps run a check in the AM to confirm that this activity didn't break anything.

Is there a reason why extender cards shouldn't be stuck into eurocrates?

Attachment 1: Screenshot_from_2018-11-06_23-18-23.png
Screenshot_from_2018-11-06_23-18-23.png
Attachment 2: Screenshot_from_2018-11-06_23-19-26.png
Screenshot_from_2018-11-06_23-19-26.png
  14283   Wed Nov 7 19:20:53 2018 gautamUpdateComputersPaola Battery Error

The VEA vertex laptop, paola, has a flashing orange indicator which I take to mean some kind of battery issue. When the laptop is disconnected from its AC power adaptor, it immediately shuts down. So this machine is kind of useless for its intended purpose of being a portable computer we can work at optical tables withno. The actual battery diagnostics (using upower) don't report any errors. 

  14284   Wed Nov 7 19:42:01 2018 gautamUpdateGeneralIFO checkup and DRMI locking prep

Earlier today, I rebooted a few unresponsive VME crates (susaux, auxey).

The IMC has been unhappy for a couple of days - the glitches in the MC suspensions are more frequent. I reset the dark offsets, minimized MCREFL by hand, and then re-centered the beam on the MC2 Trans QPD. In this config, the IMC has been relatively stable today, although judging by the control room StripTool WFS control signal traces, the suspension glitches are still happening. Since we have to fix the attenuator issue anyways soon, we can do a touch-up on IMC WFS.

I removed the DC PD used for loss measurements. I found that the AS beam path was disturbed - there is a need to change the alignment, this just makes it more work to get back to IFO locking as I have to check alignment onto the AS55 and AS110 PDs.

Single arm locking worked with minimal effort - although the X arm dither alignment doesn't do the intended job of maximizing the transmission. Needs a checkup.

PRMI locking (carrier resonant) was also pretty easy. Stability of the lock is good, locks hold for ~20 minutes at a time and only broke because I was mucking around. However, when the carrier is resonant, I notice a smeared scatter pattern on the ITMX camera that I don't remember from before. I wonder if the FF idea can be tested in the simpler PRMI config.

After recovering these two simpler IFO configurations, I improved the cavity alignment by hand and with the ASS servos that work. Then I re-centered all the Oplev beams onto their respective QPDs and saved the alignment offsets. I briefly attemped DRMI locking, but had little success, I'm going to try a little later in the evening, so I'm leaving the IFO with the DRMI flashing about, LSC mode off.

  14285   Wed Nov 7 23:07:11 2018 gautamUpdateLSCDRMI locking recovered

I had some success today. I hope that the tweaks I made will allow working with the DRMI during the day as well, though it looks like the main limiting factor in lock duty cycle is angular stability of the PRC.

  • Since there has been some change in the light levels / in vacuum optical paths, I decided to be a bit more systematic.
  • Initial guess of locking gains / demod phases was what I had last year.
  • Then I misaligned SRM, and locked PRMI, for the sideband resonant in the PRC (but still no arm cavities, and using 1f Refl error signals).
  • Measured loop TFs, adjusted gains, re-enabled boosts.
  • Brought the SRM back into the picture. Decided to trigger SRCL loop on AS110I rather than the existing POP22I (because why should 2f1 signal buildup carry information about SRCL?). New settings saved to the configure script. Reduced MICH gain to account for the SRC cavity gain.
  • Re-measured loop TFs, re-adjusted gains. More analysis about the state of the loops tomorrow, but all loops have UGF ~100-120 Hz.
  • Ran some sensing lines - need to check my sensing matrix making script, and once I get the matrix elements, I can correct the error signal demod phasing as necessary.

[Attachment #1]: Repeatable and reliable DRMI locks tonight, stability is mainly limited by angular glitches - I'm not sure yet if these are due to a suspect Oplev servo on the PRM, or if they're because of the tip-tilt PR2/PR3/SR2/SR3.

[Attachment #2]: A pass at measuring the TF from SRCL error point to MICH error point via control noise re-injection. I was trying to measure down to 40 Hz, but lost the lock, and am calling it for the night.

[Attachment #3]: Coherence between PRM oplev error point and beam spot motion on POP QPD.

Note that the MICH actuation is not necessarily optimally de-coupled by actuating on the PRM and SRM yet (i.e. the latter two elements of the LSC output matrix are not precisely tuned yet).

What is the correct way to make feedforward filters for this application? Swept-sine transfer function measurement? Or drive broadband noise at the SRCL error point and then do time-domain Wiener filter construction using SRCL error as the witness and MICH error as the target? Or some other technique? Does this even count as "feedforward" since the sensor is not truly "outside" the loop?

Attachment 1: Screenshot_from_2018-11-07_23-05-58.png
Screenshot_from_2018-11-07_23-05-58.png
Attachment 2: SRCL2MICH_crosscpl.pdf
SRCL2MICH_crosscpl.pdf
Attachment 3: PRCangularCoh_rot.pdf
PRCangularCoh_rot.pdf
  14286   Fri Nov 9 15:00:56 2018 gautamUpdateIOONo IFO beam as TT1 UL hijacked for REFL55 check

This problem resurfaced. I'm doing the debugging.

6:30pm - "Solved" using the same procedure of stepping through the whitening gains with a small (10 DAC cts pk) signal applied. Simply stepping through the gains with input grounded doesn't seem to do the trick.

Attachment 1: REFL55_wht_chk.png
REFL55_wht_chk.png
  14288   Sat Nov 10 17:32:33 2018 gautamUpdateLSCNulling MICH->PRCL and MICH->SRCL

With the DRMI locked, I drove a line in MICH using the sensing matrix infrastructure. Then I looked at the error points of MICH, PRCL and SRCL. Initially, the sensing line oscillator output matrix for MICH was set to drive only the BS. Subsequently, I changed the --> PRM and --> SRM matrix elements until the line height in the PRCL and SRCL error signals was minimized (i.e. the change to PRCL and SRCL due to the BS moving, which is a geometric effect, is cancelled by applying the opposite actuation to the PRM/SRM respectively. Then I transferred these to the LSC output matrix (old numbers in brackets).

MICH--> PRM = -0.335 (-0.2655)

MICH--> SRM = -0.35 (+0.25)

I then measured the loop TFs - all 3 loops had UGFs around 100 Hz, coinciding with the peaks of the phase bubbles. I also ran some sensing lines and did a sensing matrix measurement, Attachment #1 - looks similar to what I have obtained in the past, although the relative angles between the DoFs makes no sense to me. I guess the AS55 demod phase can be tuned up a bit.

The demodulation was done offline - I mixed the time series of the actuator and sensor signals with a "local oscillator" cosine wave - but instead of using the entire 5 minute time series and low-passing the mixer output, I divvied up the data into 5 second chunks, windowed with a Tukey window, and have plotted the mean value of the resulting mixer output.

Unrelated to this work: I re-aligned the PMC on the PSL table, mostly in Pitch.

Attachment 1: sensMat_2018-11-10.pdf
sensMat_2018-11-10.pdf
  14292   Tue Nov 13 18:09:24 2018 gautamUpdateLSCInvestigation of SRCL-->MICH coupling

Summary:

I've been looking into the cross-coupling from the SRCL loop control point to the Michelson error point.

[Attachment #1] - Swept sine measurement of transfer function from SRCL_OUT_DQ to MICH_IN1_DQ. Details below.

[Attachment #2] - Attempt to measure time variation of coupling from SRCL control point to MICH error point. Details below.

[Attachment #3] - Histogram of the data in Attachment #2.

[Attachment #4] - Spectrogram of the duration in which data in #2 and #3 were collected, to investigate the occurrance of fast glitches.

Hypothesis: (so that people can correct me where I'm wrong - 40m tests are on DRMI so "MICH" in this discussion would be "DARM" when considering the sites)

  • SRM motion creates noise in MICH.
  • The SRM motion may be naively decomposed into two contributions -
    • Category #1: "sensing noise induced" motion, which comes about because of the SRCL control loop moving the SRM due to shot noise (or any other sensing noise) of the SRCL PDH photodiode, and
    • Category #2: all other SRM motion.
  • We'd like to cancel the former contribution from DARM.
  • The idea is to measure the transfer function from SRCL control point to the MICH error point. Knowing this, we can design a filter so that the SRCL control signal is filtered and summed in at the MICH error point to null the SRCL coupling to MICH.
  • Caveats/questions:
    • Introducing this extra loop actually increases the coupling of the "all other" category of SRM motion to MICH. But the hypothesis is that the MICH noise at low frequencies, which is where this increased coupling is expected to matter, will be dominated by seismic/other noise contributions, and so we are not actually degrading the MICH sensitivity.
    • Knowing the nosie-budget for MICH and SRCL, can we AC couple the feedforward loop such that we are only doing stuff at frequencies where Category #1 is the dominant SRCL noise?

Measurement details and next steps:

Attachment #1

  • This measurement was done using DTT swept sine.
  • Plotted TF is from SRCL_OUT to MICH_IN, so the SRCL loop shape shouldn't matter.
  • I expect the pendulum TF of the SRM to describe this shape - I've overlaid a 1/f^2 shape, it's not quite a fit, and I think the phase profile is due to a delay, but I didn't fit this.
  • I had to average at each datapoint for ~10 seconds to get coherence >0.9.
  • The whole measurement takes a few minutes.

Attachments #2 and #3

  • With the DRMI locked, I drove a sine wave at 83.13 Hz at the SRCL error point using awggui.
  • I ramped up the amplitude till I could see this line with an SNR of ~10 in the MICH error signal.
  • Then I downloaded ~10mins of data, demodulated it digitally, and low-passed the mixer output.
  • I had to use a pretty low corner frequency (0.1 Hz, second order butterworth) on the LPF, as otherwise, the data was too noisy.
  • Even so, the observed variation seems too large - can the coupling really change by x100?
  • The scatter is huge - part of the problem is that there are numerous glitches while the DRMI is locked.
  • As discussed at the meeting today, I'll try another approach of doing multiple swept-sines and using Craig's TFplotter utility to see what scatter that yields.

Attachments #2

  • Spectrogram generated with 1 second time strides, for the duration in which the 83 Hz line was driven.
  • There are a couple of large fast glitches visible.
Attachment 1: TF_sweptSineMeas.pdf
TF_sweptSineMeas.pdf
Attachment 2: digitalDemod.pdf
digitalDemod.pdf
Attachment 3: digitalDemod_hist.pdf
digitalDemod_hist.pdf
Attachment 4: DRMI_LSCspectrogram.pdf
DRMI_LSCspectrogram.pdf
  14293   Tue Nov 13 21:53:19 2018 gautamUpdateCDSRFM errors

This problem resurfaced, which I noticed when I couldn't get the single arm locks going.

The fix was NOT restarting the c1rfm model, which just brought the misery of all vertex FEs crashing and the usual dance to get everything back.

Restarting the sender models (i.e. c1scx and c1scy) seems to have done the trick though.

Attachment 1: RFMerrors.png
RFMerrors.png
  14298   Fri Nov 16 00:47:43 2018 gautamUpdateLSCMore DRMI characterization

Summary:

  • More DRMI characterization was done.
  • I was working on trying to improve the stability of the DRMI locks as this is necessary for any serious characterization.
  • Today I revived the PRC angular feedforward - this was a gamechanger, the DRMI locks were much more stable. It's probably worth spending some time improving the POP LSC/ASC sensing optics/electronics looking towards the full IFO locking.
  • Quantitatively, the angular fluctuations as witnessed by the POP QPD is lowered by ~2x with the FF on compared to offyes [Attachment #1, references are with FF off, live traces are with FF on].
  • The first DRMI lock I got is already running 15 mins, looking stable.
    • Update: Out of the ~1 hour i've tried DRMI locking tonight, >50 mins locked!
  • I think the filters can be retrained and this performance improved, something to work on while we are vented.
  • Ran sensing lines, measured loop TFs, analysis tomorrow, but I think the phasing of the 1f PDs is now okay.
    • MICH in AS55 Q, demod phase = -92deg, +6dB wht gain.
    • PRCL in REFL11 I, demod phase = +18 deg, +18dB wht gain.
    • SRCL in REFL55 I, demod phase = -175 deg, +18dB wht gain.
  • Also repeated the line in SRCL-->witness in MICH test.
    • At least 10 minutes of data available, but I'm still collecting since the lock is holding.
    • This time I drove the line at ~124 Hz with awggui, since this is more a regime where we are sensing noise dominated.

Prep for this work:

  • Reboots of c1psl, c1iool0, c1susaux.
  • Removed AS port PD loss measurement PD.
  • Initial alignment procedure as usual: single arms --> PRMI locked on carrier --> DRMI

I was trying to get some pics of the optics as a zeroth-level reference for the pre-vent loss with the single arms locked, but since our SL7 upgrade, the sensoray won't work anymore no. I'll try fixing this during the daytime.

Attachment 1: PRCff.pdf
PRCff.pdf
Attachment 2: DRMI_withPRCff.png
DRMI_withPRCff.png
  14303   Sun Nov 18 00:59:33 2018 gautamUpdateGeneralVent prep

I've begun prepping the IFO for the vent, and completed most of the IFO related items on the checklist. The power into the MC has been cut, but the low-power autolocker has not been checked. I will finish up tomorrow and post the go ahead. PSL shutter is closed for tonight.

  14304   Sun Nov 18 17:09:02 2018 gautamUpdateGeneralVent prep

Vent prep

Following the checklist, I did these:

  • Both arms were locked to IR, TRY and TRX maximized using ASS.
  • GTRY and GTRX were also maximized.
  • ITM/ETM Oplevs centered with TRX/TRY maximized, PRM/SRM/BS Opelvs were centered once the DRMI was locked and aligned.
  • Attachment #1 summarizes the above 3 bullets.
  • Sensoray was made to work with Donatella (Raspberry Pi video server idea is good but although the sensoray drivers look to have installed correctly, when the Sensoray unit is plugged into the RPi USB port, the red light doesn't come on, so I opted to not spend too much time on it for the moment).
  • Photos of all ports in various locked configurations are saved in /users/sensoray/SensorayCaptures/Nov2018Vent
  • PSL power into the IMC was cut from 1.07 W (measured after G&H mirror) to 97 mW. I opted to install a new HWP+PBS after the PMC to cut the power, so we don't have to fiddle around so much with the PMC locking settings [Attachment #3, this was the only real candidate location as the IMC wants s-polarized light].
  • 2" R=10% BS in the IMC REFL path was replaced with a 2" Y1 HR mirror, so there is no MCREFL till we turn the power back up.
  • IMC was locked.
  • Low power MC autolocker works [Attachment #2]. The reduction in MCREFL is because of me manually aligning the cavity, WFS servos are disabled in low power mode since there is no light incident on the WFS heads.
  • Updated the SUS driftmon values (though I'm not really sure how useful this is).
  • PSL shutter will remain closed, but I have not yet installed a manual beam block in the beam path on the PSL table.

@Steve & Chub, we are ready to vent tomorrow (Monday Nov 19). 

Attachment 1: VentPrepNov2018.png
VentPrepNov2018.png
Attachment 2: MCautolocker_lowPower.png
MCautolocker_lowPower.png
Attachment 3: IMG_7163.JPG
IMG_7163.JPG
  14307   Mon Nov 19 22:01:50 2018 gautamUpdateVACLoose nut on valve

As I was turning off the lights in the VEA, I heard a rattling sound from near the PSL enclosure. I followed it to a valve - I couldn't see a label on this valve in my brief effort to find one, but it is on the south-west corner of the IMC table, so maybe VABSSCI or VABSSCO? The power cable is somehow spliced with an attachment that looks to be bringing gas in/out of the valve (See Attachment #1), and the nut on the bottom was loose, the whole power cable + mettal attachment was responsible for the rattling. I finger-tightened the nut and the sound went away.

Attachment 1: IMG_7171.JPG
IMG_7171.JPG
  14310   Tue Nov 20 13:13:01 2018 gautamUpdateVACIMC alignment is okay

I checked the IMC alignment following the vent, for which the manual beam block placed on the PSL table was removed. The alignment is okay, after minor touchup, the MC Trans was ~1200 cts which is roughly what it was pre-vent. I've closed the PSL shutter again.

  14313   Wed Nov 21 09:59:26 2018 gautamUpdateLSCLSC feedforward block diagram

Attachment #1 is a block diagram depicting the pathway by which the vertex DOF control signals can couple into DARM (adapted from a similar diagram in Gabriele's Virgo note on the subject). I've also indicated some points where noise can couple into either loop. In general, there are sensing noises that couple in at the error point of the loop, and actuation noises that couple in at the control point. In this linear picture, each block represents a (possibly time varying) transfer function. So we can write out the node-to-node transfer functions and evaluate the various couplings.

The motivation is to see if we can first simulate with some realistic noise and time-varying couplings (and then possibly test on the realtime system) the effectiveness of the filter denoted by "FF" in canceling out the shot noise from the auxiliary loop being re-injected into the DARM loop via the DARM sensor. Does this look correct?

Attachment 1: IMG_7173.JPG
IMG_7173.JPG
  14314   Wed Nov 21 16:48:11 2018 gautamUpdateCOCEY mini cleanroom setup

With Chub's help, I've setup a mini cleanroom at EY - Attachment #1. The HEPA unit is running on high now. All surfaces were wiped with isopropanol, we can wipe everything down again on Monday and replace the foil.

Attachment 1: IMG_7174.JPG
IMG_7174.JPG
  14319   Mon Nov 26 17:16:27 2018 gautamUpdateSUSEY chamber work

[steve, rana, gautam]

  • PSL and EY 1064nm laser (physical) shutters on the head were closed so that we and sundance crew could work without laser safety goggles. EY oplev laser was also turned off.
  • Cylindrical heater setup removed:
    • heater wiring meant the heater itself couldn't be easily removed from the chamber
    • two lenses and Al foil cylinder removed from chamber, now placed on the mini-cleanroom table.
  • Parabolic heater is untouched for now. We can re-insert it once the test mass is back in, so that we can be better informed about the clipping situation.
  • ETMY removed from chamber.
    • EQ stops were engaged.
    • Pictures were taken
    • OSEMs were removed from cage, placed in foil holders.
    • Cage clamps were removed after checking that marker clamps were in place.
    • Optic was moved first to NW corner of table, then out of the vacuum onto the mini-cleanroom desk Chub and I had setup last week.
    • Hoepfully there isn't an earthquake. EY has been marked as off-limits to avoid accidental bumping / catasrophic wire/magnet/optic breaking.
    • We sealed up the mini cleanroom with tape. F.C. cleaning tomorrow or at another opportune moment.
    • Light door was put back on for the evening.

Rana pointed out that the OSEM cabling, because of lack of a plastic shielding, is grounded directly to the table on which it is resting. A glass baking dish at the base of the seismic stack prevents electrical shorting to the chamber. However, there are some LEMO/BNC cables as well on the east side of the stack, whose BNC ends are just lying on the base of the stack. We should use this opportunity to think about whether anything needs to be done / what the influence of this kind of grounding is (if any) on actuator noise.

Steve also pointed out that we should replace the rubber pads which the vacuum chamber is resting on (Attachment #1, not from this vent, but just to indicate what's what). These serve the purpose of relieving small amounts of strain the chamber may experience relative to the beam tube, thus helping preserve the vacuum joints b/w chamber and tube. But after (~20?) years of being under compression, Steve thinks that the rubber no longer has any elasticity, and so should be replaced.

Attachment 1: IMG_5251.JPG
IMG_5251.JPG
  14324   Thu Nov 29 17:46:43 2018 gautamUpdateGeneralSome to-dos

[koji, gautam, jon, steve]

  • We suspect analog voltage from N2 pressure gauge is connected to interfacing Omega controller with the 'wrong' polarity (i.e pressure is rising over ~4 days and then rapidly falling instead of the other way around). This should be fixed.
  • N2 check script logic doesn't seem robust. Indeed, it has not been sending out warning emails (threshold is set to 60 psi, it has certainly gone below this threshold even with the "wrong" polarity pressure gauge hookup). Probably the 40m list is rejecting the email because controls isn't a part of the 40m group.
  • Old frames have to be re-integrated from JETSTOR to the new FB in order to have long timescale lookback.
  • N2 cylinder pressure gauges (at the cylinder end) need a power supply - @ Steve, has this been purchased? If not, perhaps @ Chub can order it.
  • MEDM vacuum screen should be updated to have gate valves be a different color to the spring-loaded valves. Manual valve between TP1 and V1 should also be added.
  • P2, P3 and P4 aren't returning sensible values (they should all be reading ~760torr as is P1). @ Steve, any idea if these gauges are broken?
  • Hornet gauges (CC and Pirani) should be hooked up to the new vacuum system.
  • add slow channels of   foreline pressures of TP2 & 3   and    C1:Vac-IG1_status_pressure
  14326   Fri Nov 30 19:37:47 2018 gautamUpdateLSCLSC feedforward block diagram

I wanted to set up an RTCDS model to understand this problem better. Attachment #1 is the simulink diagram of the signal flow. The idea will be to put in the appropriate filter shapes into the various filter blocks denoting the DARM and auxiliary DoF plants, controllers and actuators, and then use awggui / diaggui to inject some noises and see if in this idealized model I can achieve good subtraction. Then we can build up to applying a time varying cross coupling between DARM and the vertex DoF, and see how good the adaptive FF works. Still need to setup some MEDM screens to make working with the test system easier.

I figured c1omc would be the least invasive model to set this upon without risking losing any of our IR/green alignment references. Compile and install went smooth, see Attachment #2. The c1omc model was clocking 4us before, now it's using 7us.

Attachment #3 shows the top level of the OMC model, while Attachment #4 shows the MEDM screen.

* Note to self: when closing a loop inside the realtime model, there has to be a delay block somewhere in the loop, else a compilation error is thrown.

Attachment 1: LSC_FF_tester.png
LSC_FF_tester.png
Attachment 2: Screenshot_from_2018-11-30_19-41-07.png
Screenshot_from_2018-11-30_19-41-07.png
Attachment 3: Screenshot_from_2018-12-10_15-31-23.png
Screenshot_from_2018-12-10_15-31-23.png
Attachment 4: SimLSC.png
SimLSC.png
  14328   Sun Dec 2 17:26:58 2018 gautamUpdateIMCIMC ringdown fitting

Recently we wondered at the meeting what the IMC round trip loss was. I had done several ringdowns in the winter of 2017, but because the incident light on the cavity wasn't being extinguished completely (the AOM 0th order beam is used), the full Isogaio et. al. analysis could not be applied (there were FSS induced features in the reflection ringdown signal). Nevertheless, I fitted the transmission ringdowns. They looked like clean exponentials, and judging by the reflection signals (see previous elogs in this thread), the first ~20us of data is a clean exponential, so I figured we may get some rough value of the loss by just fitting the transmission data. 

The fitted storage time is 60.8 \pm 2.7 \mu s.However, this number isn't commensurate with the 40m IMC spec of a critically coupled cavity with 2000ppm transmissivity for the input and output couplers.

Attachment #1: Expected storage time for a lossless cavity, with round-trip length ~27m. MC2 is assumed to be perfectly reflecting. The IMC length is known to better than 100 Hz uncertainty because the marconi RF modulation signal is set accordingly. For the 40m spec, I would expect storage times of ~40 usec, but I measure almost 30% longer, at ~60 usec.

Attachment #2: Fits and residuals from the 10 datasets I had collected. This isn't a super informative plot because there are 10 datasets and fits, but to eye, the fits are good, and the diagonal elements of the covariance matrix output by scipy's curve_fit back this up. The function used to fit the t > 0 portions of these signals (because the light was extinguished at t=0 by actuating on the AOM) is \text{Transmission} = Ae^{-\frac{2t}{\tau_{\mathrm{storage}}}}, where A and tau are the fitted parameters. In the residuals, the same artefacts visible in the reflection signal are seen.

Attachment #3: Scatter plot of the data. Width of circles are proportional to fit error on individual measurements (i just scaled the marker size arbitrarily to be able to visually see the difference in uncertainty, the width doesn't exactly indicate the error), while the dahsed lines are the global mean and +/- 1 sigma levels.

Attachment #4: Cavity pole measurement. Using this, I get an estimate of the loss that is a much more believable 300 \pm 20\, \mathrm{ppm}.

Attachment 1: tauTheoretical.pdf
tauTheoretical.pdf
Attachment 2: ringdownFit.pdf
ringdownFit.pdf
Attachment 3: ringdownScatter.pdf
ringdownScatter.pdf
Attachment 4: cavPole.pdf
cavPole.pdf
  14331   Tue Dec 4 18:24:05 2018 gautamOmnistructureGeneralN2 line disconnected

[jon, gautam]

In the latest installment in this puzzler: turns out that maybe the trend of the "N2 pressure" channel increasing over the ~3 day timescale it takes a cylinder of N2 to run out is real, and is a feature of the way our two N2 cylinder lines/regulators are setup (for the automatic switching between cylinders when one runs out). In order to test this hypothesis, we'd like to have the line pressure be 0 initially, and then just have 1 cylinder hooked up. When we went into the drill-press area, we heard a hiss, turns out that one of the cylinders is leaking (to be fair, this was labelled, but i thought it isn't great to have a higher N2 concentration in an enclosed space). Since we don't need any actuation ability, I valved off the leaky cylinder, and disconnected the other properly functioning one. Attachment #1 shows the current state.

Attachment 1: IMG_7195.JPG
IMG_7195.JPG
  14334   Fri Dec 7 12:51:06 2018 gautamUpdateIMCIMC ringdown fitting

I started putting together some code to implement some ideas we discussed at the Tuesday meeting here. Pipeline isn't setup yet, but i think it's commented okay so if people want to play around with it, the code lives on the 40m gitlab

Model parameters:

  • T+ --- average transmission of MC1 and MC3.
  • T- --- difference in transmission between MC1 and MC3 (this basis is used rather than T1 and T3, because the assumption is that since they were coated in the same coating run, the difference in transmission should be small, even if there is considerable uncertainty in the actual average transmission number.
  • T2 --- MC2 transmission.
  • Lrt --- Round trip loss in the cavity.
  • "sigma" --- a nuisance parameter quantifying the error in the time domain ringdown data.

Simulation:

  • Using these model parameters, calculate some simulated time-domain ringdowns. Optionally, add some noise (assumed Gaussian).
  • Try and back out the true values of the model parameters using emcee - priors were assumed to be uniformly distributed, with a +/- 20% uncertainty around the central value.
  • For a first test, see if there is any improvement in the parameter estimation uncertainty using only transmission ringdown vs both transmission and reflection.

Initial results and conclusions:

  • Attachment #1 - Simulated time series used for this study. The "fit" trace is computed using the median values from the monte-carlo.
  • Attachment #2 - Corner plots showing the distribution of the estimated parameter values, using only transmission ringdown. The "true" values are indicated using the thick blue lines.
  • Attachment #3 - Corner plots showing the distribution of the estimated parameter values, using both transmission and reflection ringdowns.
  • The overall approach seems to work okay. There seems to be only marginal improvement in the uncertainty in estimated parameters using both ringdown signals, at least in the simulation.
  • However, everything seems pretty sensitive to the way the likelihood and priors are coded up - need to explore this a bit more.

Next steps:

  • Add more simulated measurements, see if we can constrain these parameters more tightly. 
  • Use linear error analysis to see if that tells us which measurements we should do, without having to go through the emcee.

There still seems to be some data quality issues with the ringdown data I have, so I don't think we really gain anything from running this analysis on the data I have already collected - but in the future, we can do the ringdown with complete extinguishing of the input light, and repeat the analysis.

As for whether we should clean the IMC mirrors - I'm going to see how much power comes out at the REFL port (with PRM aligned) this afternoon, and compare to the input power. This technique suffers from uncertainty in the Faraday insertion loss, isolation and IMC parameters, but I am hoping we can at least set a bound on what the IMC loss is.

Attachment 1: time_reflAndTrans.pdf
time_reflAndTrans.pdf
Attachment 2: corner_transOnly.pdf
corner_transOnly.pdf
Attachment 3: corner_reflAndTrans.pdf
corner_reflAndTrans.pdf
  14335   Fri Dec 7 17:04:18 2018 gautamUpdateIOOIMC transmission
  • Power just before PSL shutter on PSL table = 97 +/- 1 mW. Systematic error unknown.
  • Power from IFO REFL on AP table = 40 +/- 1 mW. Systematic error unknown.

Both were measured using the FieldMate power meter. I was hesitant to use the Ophir power meter as there is a label on it that warns against exceeding 100 mW. I can't find anything in the elog/wiki about the measured inesrtion loss / isolation of the input faraday, but this seems like a pretty low amount of light to get back from PRM. The IMC visibility using the MC_REFL DC values is ~87%. Assuming perfect transmission of the 87% of the 97mW that's coupled into the IMC, and assuming a further 5% loss between the Faraday rejected port and the AP table, the Faraday insertion loss would be ~30%. Realistically, the IMC transmission is lower. There is also some part of the light picked off for IPPOS. Judging by the shape of the REFL spot on the camera, it doesn't look clipped to me.

Either way, seems like we are only getting ~half of the 1W we send in on the back of PRM. So maybe it's worth it to investigate the situation in the IOO chamber during this vent.


c1pslc1susaux,c1iool0,caux  crates were keyed. Also, the physical shutter on the PSL NPRO, which was closed last Monday for the Sundance crew filming, was opened and the PMC was locked. PMC remains locked, but there is no light going into the IMC.

  14339   Mon Dec 10 15:53:16 2018 gautamUpdateLSCSwept-sine measurement with DTT

Disclaimer: This is almost certainly some user error on my part.

I've been trying to get this running for a couple of days, but am struggling to understand some behavior I've been seeing with DTT.

Test:

I wanted to measure some transfer functions in the simulated model I set up.

  • To start with, I put a pendulum (f0 = 1Hz, Q=5) TF into one of the filter modules
  • Isolated it from the other interconnections (by turning off the MEDM ON/OFF switches).
  • Set up a DTT swept-sine measurement
    • EXC channel was C1:OMC-TST_AUX_A_EXC
    • Monitored channels were C1:OMC-TST_AUX_A_IN2 and C1:OMC-TST_AUX_A_OUT.
    • Transfer function being measured was C1:OMC-TST_AUX_A_OUT/C1:OMC-TST_AUX_A_IN2.
    • Coherence between the excitation and output were also monitored.
  • Sweep parameters:
    • Measurement band was 0.1 - 900 Hz
    • Logarithmic, downward.
    • Excitation amplitude = 1ct, waveform = "Sine"

Unexplained behavior:

  • The transfer function measurement fails with a "Synchronization error", at ~15 Hz.
    • I don't know what is special about this frequency, but it fails repeatedly at the same point in the measurement.
  • Coherence is not 1 always
    • Why should the coherence deviate from 1 since everything is simulated? I think numerical noise would manifest when the gain of the filter is small (i.e. high frequencies for the pendulum), but the measurement and coherence seem fine down to a few tens of Hz.

To see if this is just a feature in the simulated model, I tried measuring the "plant" filter in the C1:LSC-PRCL filter bank (which is also just a pendulum TF), and run into the same error. I also tried running the DTT template on donatella (Ubuntu12) and pianosa (SL7), and get the same error, so this must be something I'm doing wrong with the way the measurement is being run / setup. I couldn't find any mention of similar problems in the SimPlant elogs I looked through, does anyone have an idea as to what's going on here?

* I can't get the "import" feature of DTT to work - I go through the GUI prompts to import an ASCII txt file exported from FOTON but nothing selectable shows up in DTT once the import dialog closes (which I presume means that the import was successful). Are we using an outdated version of DTT (GDS-2.15.1)?  But Attachment #1 shows the measured part of the pendulum TF, and is consistent with what is expected until the measurement terminates with a synchronization error.


the import problem is fixed - when importing, you have to give names to the two channels that define the TF you're importing (these can be arbitrary since the ASCII file doesn't have any channel name information). once i did that, the import works. you can see that while the measurement ran, the foton TF matches the DTT measured counterpart.


11 Dec 2pm: After discussing with Jamie and Gabriele, I also tried changing the # of points, start frequency etc, but run into the same error (though admittedly I only tried 4 combinations of these, so not exhaustive).

Attachment 1: SimTF.pdf
SimTF.pdf
  14344   Tue Dec 11 14:33:29 2018 gautamUpdateCDSNDScope

NDscope is now running on pianosa. To be really useful, we need the templates, so I've made /users/Templates/NDScope_templates where these will be stored. Perhaps someone can write a parser to convert dataviewer .xml to something ndscope can understand. To get it installed, I had to run:

sudo yum install ndscope
sudo yum install python34-gpstime
sudo yum install python34-dateutil
sudo yum install python34-requests

 I also changed the pythonpath variable to include the python3.4 site-packages library in .bashrc

Quote:

https://alog.ligo-wa.caltech.edu/aLOG/index.php?callRep=44971

Let's install Jamie's new Data Viewer

Attachment 1: ndscope.png
ndscope.png
  14345   Tue Dec 11 18:20:59 2018 gautamUpdateOptical LeversBS/PRM HeNe is dead

I found that the BS/PRM OL SUM channels were reading close to 0. So I went to the optical table, and found that there was no beam from the HeNe. I tried power-cycling the controller, there was no effect. From the trend data, it looks like there was a slow decay over ~400000 seconds (~ 5 days) and then an abrupt shutoff. This is not ideal, because we would have liked to use the Oplevs as a DC alignment reference during the ventnoI plan to use the AS camera to recover some sort of good Michelson alignment, and then if we want to, we can switch out the HeNe.

*How can I export PDF from NDscope?

Attachment 1: BSOL_dead.png
BSOL_dead.png
  14349   Thu Dec 13 01:26:34 2018 gautamUpdateGeneralPower Outage recovery

[koji, gautam]

After several combinations of soft/hard reboots for FB, FEs and expansion chassis, we managed to recover the nominal RTCDS status post power outage. The final reboots were undertaken by the rebootC1LSC.sh script while we went to Hotel Constance. Upon returning, Koji found all the lights to be green. Some remarks:

  1. It seems that we need to first turn on FB
    • Manually start the open-mx and mx services using
      sudo systemctl start open-mx.service 
      sudo systemctl start mx.service
    • Check that the system time returned by gpstime matches the gpstime reported by internet sources.
    • Manually start the daqd processes using
      sudo systemctl start daqd_*
  2. Then fully power cycle (including all front and rear panel power switches/cables) the FEs and the expansion chassis.
    • This seems to be a necessary step for models run on c1sus (as reported by the CDS MEDM screen) to pick up the correct system time (the FE itself seems to pick up the correct time, not sure what's going on here).
    • This was necessary to clear 0x4000 errors.
  3. Power on the expansion chassis.
  4. Power on the FE.
  5. Start the RTCDS models in the usual way
    • For some reason, there is a 1 second mismatch between the gpstime returned on the MEDM screen for a particular CDS model status, and that in the terminal for the host machine.
    • This in itself doesn't seem to cause any timing errors. But see remark about c1sus above in #2.

The PSL (Edwin) remains in an interlock-triggered state. We are not sure what is causing this, but the laser cannot be powered on until this is resolved.

  14351   Thu Dec 13 12:06:35 2018 gautamUpdateGeneralPower Outage recovery

I did a walkaround and checked the status of all the interlock switches I could find based on the SOP and interlock wiring diagram, but the PSL remains interlocked. I don't want to futz around with AC power lines so I will wait for Koji before debugging further. All the "Danger" signs at the VEA entry points aren't on, suggesting to me that the problem lies pretty far upstream in the wiring, possibly at the AC line input? The Red lights around the PSL enclosure, which are supposed to signal if the enclosure doors are not properly closed, also do not turn on, supporting this hypothesis...

I confirmed that there is nothing wrong with the laser itself - i manually shorted the interlock pins on the rear of the controller and the laser turned on fine, but I am not comfortable operating in this hacky way so I have restored the interlock connections until we decide the next course of action...

Quote:
 

The PSL (Edwin) remains in an interlock-triggered state. We are not sure what is causing this, but the laser cannot be powered on until this is resolved.

  14352   Thu Dec 13 18:12:47 2018 gautamUpdateIOOND filter on AS camera changed

In order to see the AS beam a bit more clearly in our low-power config, I swapped out the ND=1.0 filter on the AS camera for ND=0.5.

  14356   Thu Dec 13 22:56:28 2018 gautamUpdateCDSFrames

[koji, gautam]

We looked into the /frames situation a bit tonight. Here is a summary:

  1. We have already lost some second trend data since the new FB has been running from ~August 2017.
  2. The minute trend data is still safe from that period, we believe.
  3. The Jetstor has ~2TB of trend data in the /frames/trend folder.
    • This is a combination of "second", "minute_raw" and "minute".
    • It is not clear to us what the distinction is between "minute_raw" and "minute", except that the latter seems to go back farther in time than the former.
    • Even so, the minute trend folder from October 2011 is empty - how did we manage to get the long term trend data?? From the folder volumes, it appears that the oldest available trend data is from ~July 24 2015.

Plan of action:

  1. The wiper script needs to be tweaked a bit to allow more storage for the minute trends (which we presumably want to keep for long term).
  2. We need to clear up some space on FB1 to transfer the old trend data from Jetstor to FB1.
  3. We need to revive the data backup via LDAS. Also summary pages.

BTW - the last chiara (shared drive) backup was October 16 6 am. dmesg showed a bunch of errors, Koji is now running fsck in a tmux session on chiara, let's see if that repairs the errors. We missed the opportunity to swap in the 4TB backup disk, so we will do this at the next opportunity.

  14362   Sat Dec 15 20:04:03 2018 gautamUpdateIOOTT1/TT2 stepping

I'm running a script that moves TT1 and TT2 randomly in some restricted P/Y space to try and find an alignment that gets some light onto the TRY PD. Test started at gpstime 1228967990, should be done in a few hours. The IMC has to remain locked for the duration of this test. I will close the PSL shutter once the test is done. Not sure if the light level transmitted through the ITM, which I estimate to be ~30uW, will be enough to show up on the TRY PD, but worth a shot I figure.

Test was completed and PSL shutter was closed at 1228977122.

  14366   Wed Dec 19 00:12:46 2018 gautamUpdateOMC40m OMC DCC node

I made a node to collect drawings/schematics for the 40m OMC, added the length drive for now. We should collect other stuff (TT drivers, AA/AI, mechanical drawings etc) there as well for easy reference.

Some numbers FTR:

  • OMC length PZT capacitance was measured to be 209 nF.
  • Series resistance in HV path of OMC lenght PZT driver is 10 kohms, so this creates a LP with corner 1/2/pi/10kohm/200nF ~80 Hz.
  • Per Rob's thesis, the length PZT has DC actuation coefficient of 8.3 nm/V, ∼ 2 µm range. 
  14368   Wed Dec 19 15:15:56 2018 gautamUpdateIOOTT1/TT2 stepping

I removed the ND filter from the ETMYT camera to facilitate searching for a TRY beam. This should be replaced before we go back to high power.

  14369   Wed Dec 19 19:51:19 2018 gautamUpdateGeneralPumpdown prep

[Koji, gautam]

Summary:

We are ready to put the heavy doors back on the chambers and do some test pumpdowns tomorrow morning if Jon gives us the go-ahead. Also, Koji made the OMC resonate some of the AUX beam light we send into ityes

Details:

  1. EY work:
    • IMC was locked, and we attempted to locate the beam with an IR card inside the chamber.
    • Koji found that the beam was too high, we were over-shooting the entire black-glass baffle on the EY table.
    • So I moved the TTs to try and center the beam through the aperture of aforementioned baffle.
    • Once this was done, we found that the beam was misaligned in yaw by ~1-inch in transmission on the EY optics table (there was an iris in place marking the cavity transmission axis). This explains why I couldn't find any TRY flashes while moving the TTs around.
    • We hypothesize that without the 2 degree ETM wedge in place, there isn't a compatible axis for the ITM transmission to also make it through the EY baffle and transmission iris. Over ~1m, the 2 degree wedge makes roughly 1.4 inch translation in yaw, so this seems to be a plausible hypothesis.
    • The ETMY suspension was moved from the mini-cleanroom setup back into the EY vacuum chamber. Two clamps (finger tightened only) hold it in place on the NE edge of the optical table. We decided that this is a better resting palce for the cage over the holidays than an in-air cleanroom.
  2. OMC chamber work:
    • While we were in clean garb, we decided to also investigate the OMC situation a bit.
    • It quickly became apparent that it was hopeless for me to work in chamber in the tightly confined IOO chamber. So Koji went in to have a look.
    • Koji will post the detailed alignment procedure - but after some alignment of the AUX laser input beam axis using in air steering mirrors and Koji's expert tweaking of the pointing into the OMC, we observed some resonances of the OMC.
    • Attachment #1 shows the full-range triangle ramp applied to the OMC length PZT (top row) and the OMC REFL signal (bottom row), measured using a PDA520 (chosen for its large active area) connected to a scope (AC-coupled, 1Mohm impedance, averaged to make the dips more prominent).
    • The OMC transmission was also (barely) visible on an IR card.
    • So the OMC length PZT seems capable of sweeping the length of the cavity. Based on the size of the dips we saw, the MM into the cavity is sub 1-percent.
    • The transmission PDs didn't output any measurable signal - but I'm not sure that the satellite box / readout electronics have been carefully characterized on the electroncis bench, so that will have to be done first.
    • We replaced the copper cover of the OMC (finger tightened for now) in case we do any test pumpdowns tomorrow. HV supply has been turned off, and the AUX laser has been reverted to standby mode.
Attachment 1: OMCscan.pdf
OMCscan.pdf
  14370   Wed Dec 19 21:14:50 2018 gautamUpdateVACPumpdown tomorrow

I just spoke to Jon who asked me to make this elog - we will be ready to test one or more parts of the pumpdown procedure tomorrow (12/20), so we should proceed as planned to put the heavy doors back on EY and OMC chambers at 9am tomorrow morning. Jon will circulate a more detailed procedure about the pumpdown steps later today evening.

  14373   Thu Dec 20 10:28:43 2018 gautamUpdateVACHeavy doors back on for pumpdown 82

[Chub, Koji, Gautam]

We replaced the EY and IOO chamber heavy doors by 10:10 am PST. Torquing was done first oen round at 25 ft-lb, next at 45 ft-lb (we trust the calibration on the torque wrench, but how reliable is this? And how important are these numbers in ensuring a smooth pumpdown?). All went smooth. The interior of the IOO chamber was found to be dirty when Koji ran a wipe along some surfaces.

For this pumpdown, we aren't so concerned with having the IFO in an operating state as we will certainly vent it again early next year. So we didn't follow the full close-up checklist.

Jon and Chub and Koji are working on starting the pumpdown now... In order to not have to wear laser safety goggles while we closed doors and pumped down, I turned off all the 1064nm lasers in the lab.

  14374   Thu Dec 20 17:17:41 2018 gautamUpdateCDSLogging of new Vacuum channels

Added the following channels to C0EDCU.ini:

[C1:Vac-P1b_pressure]
units=torr
[C1:Vac-PRP_pressure]
units=torr
[C1:Vac-PTP2_pressure]
units=torr
[C1:Vac-PTP3_pressure]
units=torr
[C1:Vac-TP2_rot]
units=kRPM
[C1:Vac-TP3_rot]
units=kRPM

Also modified the old P1 channel to

[C1:Vac-P1a_pressure]
units=torr

Unfortunately, we realized too late that we don't have these channels in the frames, so we don't have the data from this test pumpdown logged, but we will have future stuff. I say we should also log diagnostics from the pumps, such as temperature, current etc. After making the changes, I restarted the daqd processes.


Things to add to ASA wiki page once the wiki comes back online:

  1. What is the safe way to clean the cryo pump if we want to use it again?
  2. What are safe conditions to turn the RGA on?
  14376   Fri Dec 21 11:11:51 2018 gautamUpdateCDSLogging of new Vacuum channels

The N2 pressure channel name was also wrong in C0EDCU.ini, so I updated it this morning to the correct name and units:

[C1:Vac-N2_pressure]
units=psi

Now it too is being recorded to frames.

  14377   Fri Dec 21 11:13:13 2018 gautamOmnistructureVACN2 line valved off

Per the discussion yesterday, I valved off the N2 line in the drill press room at 11 am PST today morning so as to avoid any accidental software induced gate-valve actuation during the holidays. The line pressure is steadily dropping...

Attachment #1 shows that while the main volume pressure was stable overnight, the the pumpspool pressure has been steadily rising. I think this is to be expected as the turbo pumps aren't running and the valves can't preserve the <1mtorr pressure over long timescales?

Attachment #2 shows the current VacOverview MEDM screen status.

Attachment 1: VacGauges.png
VacGauges.png
Attachment 2: Screenshot_from_2018-12-21_13-02-06.png
Screenshot_from_2018-12-21_13-02-06.png
  14380   Thu Jan 3 15:08:37 2019 gautamOmnistructureVACVac status unknown

Larry W came by the 40m, and reported that there was a campus-wide power glitch (he was here to check if our networking infrastructure was affected). I thought I'd check the status of the vacuum.

  • Attachment #1 is a screenshot of the Vac overview MEDM screen. Clearly something has gone wrong with the modbus process(es). Only the PTP2 and PTP3 gauges seem to be communicative.
  • Attachment #2 shows the minute trend of the pressure gauges for a 12 day period - it looks like there is some issue with the frame builder clock, perhaps this issue resurfaced? But checking the system time on FB doesn't suggest anything is wrong.. I double checked with dataviewer as well that the trends don't exist... But checking the status of the individual daqd processes indeed showed that the dates were off by 1 year, so I just restarted all of them and now the time seems correct. How can we fix this problem more permanently? Also, the P1b readout looks suspicious - why are there periods where it seems like we are reading values better than the LSB of the device?

I decided to check the systemctl process status on c1vac:

controls@c1vac:~$ sudo systemctl status modbusIOC.service
● modbusIOC.service - ModbusIOC Service via procServ
   Loaded: loaded (/etc/systemd/system/modbusIOC.service; enabled)
   Active: active (running) since Thu 2019-01-03 14:53:49 PST; 11min ago
 Main PID: 16533 (procServ)
   CGroup: /system.slice/modbusIOC.service
           ├─16533 /usr/bin/procServ -f -L /opt/target/modbusIOC.log -p /run/...
           ├─16534 /opt/epics/modules/modbus/bin/linux-x86_64/modbusApp /opt/...
           └─16582 caRepeater

Jan 03 14:53:49 c1vac systemd[1]: Started ModbusIOC Service via procServ.

Warning: Unit file changed on disk, 'systemctl daemon-reload' recommended.

So something did happen today that required restart of the modbus processes. But clearly not everything has come back up gracefully. A few lines of dmesg (there are many more segfaults):

[1706033.718061] python[23971]: segfault at 8 ip 000000000049b37d sp 00007fbae2b5fa10 error 4 in python2.7[400000+31d000]
[1706252.225984] python[24183]: segfault at 8 ip 000000000049b37d sp 00007fd3fa365a10 error 4 in python2.7[400000+31d000]
[1720961.451787] systemd-udevd[4076]: starting version 215
[1782064.269844] audit: type=1702 audit(1546540443.159:38): op=linkat ppid=21820 pid=22823 auid=4294967295 uid=1000 gid=1000 euid=1000 suid=1000 fsuid=1000 egid=1000 sgid=1000 fsgid=1000 tty=pts0 ses=4294967295 comm="git" exe="/usr/bin/git" res=0
[1782064.269866] audit: type=1302 audit(1546540443.159:39): item=0 name="/cvs/cds/caltech/target/c1vac/.git/objects/85/tmp_obj_uAXhPg" inode=173019272 dev=00:21 mode=0100444 ouid=1001 ogid=1001 rdev=00:00 nametype=NORMAL
[1782064.365240] audit: type=1702 audit(1546540443.255:40): op=linkat ppid=21820 pid=22823 auid=4294967295 uid=1000 gid=1000 euid=1000 suid=1000 fsuid=1000 egid=1000 sgid=1000 fsgid=1000 tty=pts0 ses=4294967295 comm="git" exe="/usr/bin/git" res=0
[1782064.365271] audit: type=1302 audit(1546540443.255:41): item=0 name="/cvs/cds/caltech/target/c1vac/.git/objects/58/tmp_obj_KekHsn" inode=173019274 dev=00:21 mode=0100444 ouid=1001 ogid=1001 rdev=00:00 nametype=NORMAL
[1782064.460620] audit: type=1702 audit(1546540443.347:42): op=linkat ppid=21820 pid=22823 auid=4294967295 uid=1000 gid=1000 euid=1000 suid=1000 fsuid=1000 egid=1000 sgid=1000 fsgid=1000 tty=pts0 ses=4294967295 comm="git" exe="/usr/bin/git" res=0
[1782064.460652] audit: type=1302 audit(1546540443.347:43): item=0 name="/cvs/cds/caltech/target/c1vac/.git/objects/cb/tmp_obj_q62Pdr" inode=173019276 dev=00:21 mode=0100444 ouid=1001 ogid=1001 rdev=00:00 nametype=NORMAL
[1782064.545449] audit: type=1702 audit(1546540443.435:44): op=linkat ppid=21820 pid=22823 auid=4294967295 uid=1000 gid=1000 euid=1000 suid=1000 fsuid=1000 egid=1000 sgid=1000 fsgid=1000 tty=pts0 ses=4294967295 comm="git" exe="/usr/bin/git" res=0
[1782064.545480] audit: type=1302 audit(1546540443.435:45): item=0 name="/cvs/cds/caltech/target/c1vac/.git/objects/e3/tmp_obj_gPI4qy" inode=173019277 dev=00:21 mode=0100444 ouid=1001 ogid=1001 rdev=00:00 nametype=NORMAL
[1782064.640756] audit: type=1702 audit(1546540443.527:46): op=linkat ppid=21820 pid=22823 auid=4294967295 uid=1000 gid=1000 euid=1000 suid=1000 fsuid=1000 egid=1000 sgid=1000 fsgid=1000 tty=pts0 ses=4294967295 comm="git" exe="/usr/bin/git" res=0
[1783440.878997] systemd[1]: Unit serial_TP3.service entered failed state.
[1784682.147280] systemd[1]: Unit serial_TP2.service entered failed state.
[1786407.752386] systemd[1]: Unit serial_MKS937b.service entered failed state.
[1792371.508317] systemd[1]: serial_GP316a.service failed to run 'start' task: No such file or directory
[1795550.281623] systemd[1]: Unit serial_GP316b.service entered failed state.
[1796216.213269] systemd[1]: Unit serial_TP3.service entered failed state.
[1796518.976841] systemd[1]: Unit serial_GP307.service entered failed state.
[1796670.328649] systemd[1]: serial_Hornet.service failed to run 'start' task: No such file or directory
[1797723.446084] systemd[1]: Unit serial_MKS937b.service entered failed state.

 

I don't know enough about the new system so I'm leaving this for Jon to debug. Attachment #3 shows that the analog readout of the P1 pressure gauge suggests that the IFO is still under vacuum, so no random valve openings were effected (as expected, since we valved off the N2 line for this very purpose).

Attachment 1: Screenshot_from_2019-01-03_15-19-51.png
Screenshot_from_2019-01-03_15-19-51.png
Attachment 2: Screenshot_from_2019-01-03_15-14-14.png
Screenshot_from_2019-01-03_15-14-14.png
Attachment 3: 997B13A9-CAAF-409C-A6C2-00414D30A141.jpeg
997B13A9-CAAF-409C-A6C2-00414D30A141.jpeg
  14386   Fri Jan 4 17:43:24 2019 gautamUpdateCDSTiming issues

[J Hanks (remote), koji, gautam]

Summary:

The problem stems from the way GPS timing signals are handled by the FEs and FB. We effected a partial fix:

  • Now, old frame data is no longer being overwritten
  • For the channels that are indeed being recorded now, the correct time stamp is being applied so they can be found in /frames by looking for the appropriate gpstime.

Details:

  • The usual FE/FB power cycling did not fix the problem.
  • The gps time used by FB and associated RT processes may be found by using  cat /proc/gps (i.e. this is different from the system time found by using date, or gpstime).
  • This was off by 2 years.
  • The way this worked up till now was by adding a fixed offset to this time.
    • This offset can be found as a line saying set symm_gps_offset=31622382 in daqdrc.fw (for example)
    • There were similar lines in daqdrc.rcv and daqdrc.dc - however, they were not all the same offset! We couldn't figure out why.
    • All these files live in /opt/rtcds/caltech/c1/target/daqd/.

Changes effected:

  1. First, we tried changing the offset in the daqdrc.fw file only.
    • Incremented it by 24*60*60*365 = number of seconds in a year with no leap seconds/days.
    • This did not fix the problem.
  2. So J Hanks decided to rebuild the Spectracom driver - these commands may not be comprehensive, but I think I got everything).
    • The relevant file is spectracomGPS.c (made a copy of /usr/src/symmetricom-3.3~rc1, called symmetricom-3.3~rc1-patched, this file is in /usr/src/symmetricom-3.3~rc1-patched/include/drv)
    • Added the following lines:
      /* 2018 had no leap seconds or leap days, so adjust for that */
             pHardware->gpsOffset += 31536000;
    • re-built and installed the modified symmetricom driver.
    • Checked that cat /proc/gps now yields the correct time.
    • Reset the gps time offsets in daqdrc.fw, daqdrc.rcv and daqdrc.dc to 0
    • With these steps, the frames were being written to /frames with the correct timestamp.
  3. Next, we checked the timing on the FEs
    • Basically, J Hanks rebuilt the version of the symmetricom driver that is used by the rtcds models to mimic the changes made for FB.
    • This did the trick for c1lsc and c1ioo - cat /proc/gps now returns the correct time on those FEs.
    • However, c1sus remains problematic (it initially reported a GPS time from 2 years ago, and even after the re-installed driver, is 4 days behind) - he suspects that this is because c1sus is the only FE with a Symmetricom/Spectracom card installed in the I/O chassis. So c1sus reports a gpstime that is ~4 days behind the "correct" gpstime.
    • He is going to check with Rolf/other CDS experts to figure out if it's okay for us to simply remove the card and run the models, or if we need to take other steps.
    • As part of this work, the c1x02 IOP model was recompiled, re-installed and re-started.

The realtime models were not restarted (although all the vertex FEs are running) - we can regroup next week and decide what is the correct course of action.

Quote:
 
  • Attachment #2 shows the minute trend of the pressure gauges for a 12 day period - it looks like there is some issue with the frame builder clock, perhaps this issue resurfaced? But checking the system time on FB doesn't suggest anything is wrong.. I double checked with dataviewer as well that the trends don't exist... But checking the status of the individual daqd processes indeed showed that the dates were off by 1 year, so I just restarted all of them and now the time seems correct. How can we fix this problem more permanently? Also, the P1b readout looks suspicious - why are there periods where it seems like we are reading values better than the LSB of the device?
  14389   Tue Jan 8 10:27:27 2019 gautamUpdateGeneralNear-term in-chamber work

Here is a list of tasks I think we should prioritize for the next two weeks. The idea is to get back to the previous state of being able to do single arm, PRMI-on-carrier and DRMI locking, before making further changes.

Once the new folding mirrors arrive, I'd like to modify the SRC length to allow locking in the signal-recycled config as opposed to RSE. Still need to do the detailed layout, but I think the in-vacuum layout will work. In that case, I'd like to also move the OMC and OMMT to the IY table, and also move the in-air AS photodiodes to the IY in-air optical table. This is why I've omitted the OMC alignment from this near-term list, but if we want to not move the OMC, then we probably should add alignment of the AS beam to the OMC to this list.

List of in-chamber tasks for 1/2019

Chamber Task(s)
EY
  • Clean ETMY optic and suspension
  • Put ETMY suspension back in place, recover Y-arm cavity alignment
  • Remove any residual hardware from unused heater setup
  • Restore parabolic heater setup, center radiation pattern as best as possible on ETMY
  • Check beam position on IPANG steering mirror
IY
  • Clean ITMY optic and suspension cage
  • Restore ITMY suspension, recover Y arm cavity alignment.
  • Check position of AS beam on OM1/OM2
BS/PRM (if we decide to open it)
  • Replace BS/PRM Oplev HeNe, bring the beam in and out of vacuum with beam well centerd on in-vacuum mirrors (can take this opportunity to fix the in-air layout as well to minimize un-necessary steering mirrors)
  • Check position of AS beam on OM3/OM4, adjust if necessary
  • Check position of IPPOS and IPANG beams on their respective steering optics
OMC (if we decide to open it)
  • Check position of AS beam on OM5/OM6
  • Ensure AS beam exits the vacuum cleanly
  14391   Wed Jan 9 11:07:09 2019 gautamUpdateVACNew Vac channel logging

Looks like I didn't restart all the daqd processes last night, so the data was not in fact being recorded to frames. I just restarted everything, and looks like the data for the last 3 minutes are being recorded yes. Is it reasonable that the TP1 current channel is reporting 0.75A of current draw now, when the pump is off? Also the temperature readback of TP3 seems a lot jumpier than that of TP2, probably has to do with the old controller having fewer ADC bits or something, but perhaps the SMOO needs to be adjusted.

Quote:
 

Gautam and I updated the framebuilder config file, adding the newly-added channels to the list of those to be logged.

Attachment 1: Screenshot_from_2019-01-09_11-08-28.png
Screenshot_from_2019-01-09_11-08-28.png
  14392   Wed Jan 9 11:33:35 2019 gautamUpdateCDSTiming issues still persist

Summary:

The gps time mismatch between /proc/gps and gpstime seems to be resolved. However, the 0x4000 DC errors still persist. It is not clear to me why.

Details:

On the phone with J Hanks on Friday, he reminded me that c1sus seems to be the only machine with an IRIG-B timing card installed. I can't find the elog but I remembered that Jamie, ericq and I had done this work in 2016 (?), and I also remembered Jamie saying it wasn't working exactly as expected. Since the DAQ was working fine before this card was installed, and since there are no problems with the recording of channels from the other four FE machines without this card installed, I decided to simply pull out the card from the expansion chassis. The card has been stored in the CDS/FE cabinet along the Y arm for now. There was also a cable that interfaces to the card which brings over the 1pps from the GPS unit, which has also been stored in the CDS/FE cabinet.

This seems to have resolved the mismatch between the gpstime reported by cat /proc/gps and the gpstime commands - Attachment #1 (the <1 second mismatch is presumably due to the deadtime between commands). However, the 0x4000 DC errors still persist. I'll try the full power cycle of FEs and FB which has fixed this kind of error in the past, but apart from that, I'm out of ideas.

Update 1215:

Following the instructions in this elog did not fix the problem. The problem seems to be with the daqd_fw service, which reports the following:

controls@fb1:~ 0$ sudo systemctl status daqd_fw.service 
● daqd_fw.service - Advanced LIGO RTS daqd frame writer
   Loaded: loaded (/etc/systemd/system/daqd_fw.service; enabled)
   Active: failed (Result: start-limit) since Wed 2019-01-09 12:17:12 PST; 2min 0s ago
  Process: 2120 ExecStart=/usr/bin/daqd_fw -c /opt/rtcds/caltech/c1/target/daqd/daqdrc.fw (code=killed, signal=ABRT)
 Main PID: 2120 (code=killed, signal=ABRT)

Jan 09 12:17:12 fb1 systemd[1]: Unit daqd_fw.service entered failed state.
Jan 09 12:17:12 fb1 systemd[1]: daqd_fw.service holdoff time over, scheduling restart.
Jan 09 12:17:12 fb1 systemd[1]: Stopping Advanced LIGO RTS daqd frame writer...
Jan 09 12:17:12 fb1 systemd[1]: Starting Advanced LIGO RTS daqd frame writer...
Jan 09 12:17:12 fb1 systemd[1]: daqd_fw.service start request repeated too quickly, refusing to start.
Jan 09 12:17:12 fb1 systemd[1]: Failed to start Advanced LIGO RTS daqd frame writer.
Jan 09 12:17:12 fb1 systemd[1]: Unit daqd_fw.service entered failed state.
                                                     

Update 1530:

The frame-writer error was tracked down to a C0EDCU issue. Jon told me that the Hornet CC1 pressure gauge channel was renamed to . C1:Vac-CC1_pressure, and I made the change in the C0EDCU file. However, it returns a value of 9990000000.0, which the frame writer is not happy about... Keeping the old channel name makes the frame-writer run again (although the actual data is bunk).

Update 1755:

J Hanks suggested adding a 1 second offset to the daqdrc config files. This has now fixed the 0x4000 errors, and we are back to the "nominal" RTCDS status screen now - Attachment #2.

Attachment 1: gpstimeSync.png
gpstimeSync.png
Attachment 2: Screenshot_from_2019-01-09_17-56-58.png
Screenshot_from_2019-01-09_17-56-58.png
  14394   Thu Jan 10 10:23:46 2019 gautamUpdateVACovernight leak rate

Overnight, the pressure increased from 247 uTorr to 264 uTorr over a period of 30000 seconds. Assuming an IFO volume of 33,000 liters, this corresponds to an average leak rate of ~20 uTorr L / s. It'd be interesting to see how this compares with the spec'd leak rates of the Viton O-ring seals and valves/ outgassing rates. The two channels in the screenshot are monitoring the same pressure from the same sensor, top pane is a digital readout while the bottom is a calibrated analog readout that is subsequently digitized into the CDS system.

Quote:
 

We allowed the pumpdown to continue until reaching 9e-4 torr in the main volume. At this point we valved off the main volume, valved off TP2 and TP3, and then shut down all turbo pumps/dry pumps. We will continue pumping tomorrow under the supervision of an operator. If the system continues to perform problem-free, we will likely leave the turbos pumping on the main volume and annuli after tomorrow.

Attachment 1: OvernightLeak.png
OvernightLeak.png
  14397   Fri Jan 11 16:38:57 2019 gautamUpdateGeneralSome alignment checks

The pumpdown seems to be progressing smoothly, so I think we are going to stick with the plan decided on Wednesday, and vent the IFO on Monday at 8am. I decided to do some checks of the IFO alignment.

I turned on the PSL again (so goggles are advisable again inside the VEA until this work is done), re-locked the PMC, and opened the PSL shutter into the vacuum (still low power 100 mW beam going into vacuum). The IMC alignment required minor tweaking, but I recovered ~1300 cts transmission which is what it was --> so we didn't macroscopically change the input pointing into the IMC while working on the IOO table.

Centering the ITMY oplev spot, there is a spot on the AS camera roughly centered on the control room monitor, so the TT pointing must also be pretty close.

Then I centered the ITMY oplev spot to check how well-aligned or otherwise the Michelson was - the BS has no Oplev so there was considerable angular motion of the Michelson spot, but it looked like on average, it was swinging around through a well aligned place. I saved the slow bias voltages for the ITMs and BS in this config.

Then I re-aligned ETMX and checked the green transmission - it was okay, ~0.3, and I was able to increase it to ~0.4 using the EX green PZT mirrors. So far so good.

Finally, I tried to lock the X-arm on IR - after zeroing the offsets on the transmission QPD, there seems to be a few flashes as the cavity swings through resonances, but no discernible PDH error signal. Moreover the input pointing of the IR into the X arm is controlled by the BS which is swinging around all over the place right now, so perhaps locking is hopeless, but the overall alignment of the IFO seems not too bad. Once ETMY is cleaned and put back in place, perhaps the Y arm can be locked.

I shuttered the PSL and inserted a manual beam block, and also turned off the EX laser so that we can vent on Monday without laser goggles.

*Not directly related to this work: we still have to implement the vacuum interlock condition that closes the PSL shutter in the event of a vacuum failure. It's probably fine now while the PSL power is attenuated, but once we have the high power beam going in, it'd be a good to revert to the old standard.

Attachment 1: pd82.png
pd82.png
Attachment 2: LSC_X.png
LSC_X.png
  14398   Mon Jan 14 10:06:53 2019 gautamUpdateVACVent 82 complete

[chub, gautam]

  • IFO pressure was ~2e-4 torr when we started, on account of the interlock code closing all valves because the N2 line pressure dropped below threshold (<65 psi)
  • Chub fixed the problem on the regulator in the drill-press area where the N2 tanks are, the N2 line is now at ~75 psi so that we have the ability to actuate valves if we so desire
  • We decided that there is no need to vent the pumpspool this time - avoiding an unnecessary turbo landing, so the pumpspool is completely valved off from the main volume and TPs 1-3 are left running
  • Went through the pre-vent checklist:
    • Chub measured particle count, deemed it to be okay (I think we should re-locate the particle counter to near 1X8 because that is where the air enters the IFO anyways, and that way, we can hook it up to the serial device server and have a computerized record of this number as we had in the past, instead of writing it down in a notebook)
    • Checked that the PSL was manually blocked from entering the IFO
    • Walked through the lab, visually inspected Jam Nuts and window covers, all was deemed okay
  • Moved 2 tanks of N2 into the lab on account of the rain
  • Started the vent at ~930am PST
    • There were a couple of short bursty increases in the pressure as we figured out the right valve settings but on average, things are rising at approx the same rate as we had in vent 81...
    • There was a rattling noise coming from the drypump that is the forepump for TP2 (Agilent) - turned out to be the plastic shell/casing on the drypump, moreover, the TP2 diagnostics (temperature, current etc) are all normal.
    • The CC1 gauge (Hornet) is supposed to have an auto-shutoff of its High Voltage when the pressure exceeds 10 mTorr, but it was reporting pressures in the 1 mTorr range even when the adjacent Pirani was at 25 torr. To avoid risk of damage, we manually turned the HV off. There needs to be a python script that can be executed to transition control between the remote and local control modes for the hornet, we had to Power Cycle the gauge because it wouldn't give us local control over the HV.
    • Transitioned from N2 to dry air at P1a=25 torr. We had some trouble finding the correct regulator (left-handed thread) for the dry air cylinders, it was stored in a cabinet labelled green optics no
    • Disconnected dry air from VV1 intake once P1b reached 700 torr, to let lab air flow into the IFO and avoid overpressuring.
    • VA* and VAV* valves were opened so as to vent the annuli as we anticipate multiple chamber openings for this vent.

As of 8pm local time, the IFO seems to have equilibriated to atmospheric pressure (I don't hear the hiss of in-rushing air near 1X8 and P1a reports 760 torr). The pumpspool looks healthy and there are no signs in the TP diagnostics channels that anything bad happened to the pumps. Chub is working on getting the N2 setup more robust, we plan to take the EY door off at 9am tomorrow morning with Bob's help.

* I took this opportunity to follow instructions on pg 29 of the manual and set the calibration for the SuperBee pirani gauge to 760 torr so that it is in better agreement with our existing P1a Pirani gauge. The correction was ~8% (820-->760).

Attachment 1: Vent82Summary.png
Vent82Summary.png
  14399   Tue Jan 15 10:52:38 2019 gautamUpdateSUSEY door opened

[chub, bob, gautam]

We took the heavy door off the EY chamber at ~930am.

Chamber work:

  • ETMY suspension cage was returned to its nominal position.
  • Unused hardware from the annular heater setup was removed.
  • The unused heater had its leads snipped close to the heater crimp point, and the exposed part of the bare wires was covered with Kapton tape (we should remove the source leads as well in air to avoid any accidental shorting)

Waiting for the table to level off now. Plan for later today / tomorrow is as follows:

  1. Lock the Y arm, recover good cavity alignment.
  2. Position parabolic heater such that clipping issue is resolved.
  3. Move optic to edge of table for FC cleaning
  4. Clean optic
  5. Return suspension cage to nominal position.
  14400   Tue Jan 15 15:27:36 2019 gautamUpdateGeneralLasers and other stuff turned back on

VEA is now a laser hazard area as usual, several 1064nm lasers in the lab have been turned back on. Apart from this

  • the IFR was reset to the nominal modulation settings of +13dBm output at 11.066209 MHz (this has to be done manually following each power failure).
  • The temeprature control unit for the EY doubling oven PID control was turned back on.
  • The EY Oplev HeNe was turned back on.
  • EY green PZT HV Kepco was turned back on.
  14401   Tue Jan 15 15:49:47 2019 gautamUpdateSUSEY door opened

While restoring OSEMs on ETMY, I noticed that the open voltages for the UR and LL OSEMs had significantly (>30%) changed from their values from ~2 years ago. The fact that it only occurred in 2 coils seemed to rule out gradual wear and tear, so I looked up the trends from Nov 25 - Nov 28 (Sundance visited on Nov 26 which is when we removed the cage). Not surprisingly, these are the exact two OSEMs that show a decrease in sensor voltage when the OSEMs were pulled out. I suspect that when I placed them in their little Al foil boats, I shorted out some contacts on the rear (this is reminiscent of the problem we had on PRM in 2016). I hope the problem is with the current buffer IC in the satellite box and not the physical diode, I'll test with the tester box and evaluate the problem further.


Chamber work by Chub and gautam:

  1. Table leveling was checked with a clean spirit level
    • Leveling was substantially off in two orthogonal directions, along the beam axis as well as perpendicular to it.
    • We moved almost all the weights available on the table.
    • Managed to get the leveling correct to within 1 tick on the level.
    • We are not too worried about this for now, the final leveling will be after heater repositioning, ETMY cleaning etc.
  2. ETMY OSEM re-insertion
    • OSEMs were re-inserted till their mean voltage was ~ half the open values.
    • Local damping seems to work just fine.
Attachment 1: EY_OSEMs.png
EY_OSEMs.png
ELOG V3.1.3-