40m QIL Cryo_Lab CTN SUS_Lab TCS_Lab OMC_Lab CRIME_Lab FEA ENG_Labs OptContFac Mariner WBEEShop
  40m Log, Page 270 of 341  Not logged in ELOG logo
ID Date Author Typeup Category Subject
  12656   Fri Dec 2 10:57:08 2016 steveUpdatePEM door seal replaced

The south end door leaky weather seals replaced.

The aim is here to get some overpressure inside / outside so the lab partical count would not depend on outside condition.

 

Attachment 1: 1_year_PEM.png
1_year_PEM.png
  12657   Fri Dec 2 11:56:42 2016 gautamUpdateLSCMC1 LEMO jiggled

I noticed 2 periods of frequent IMC locklosses on the StripTool trace, and so checked the MC1 PD readout channels to see if there were any coincident glitches. Turns out there wasnt yes BUT - the LR and UR signals had changed significantly over the last couple of days, which is when I've been working at 1X5. The fast LR readback was actually showing ~0, but the slow monitor channel had been steady so I suspected some cabling shenanigans.

Turns out, the problem was that the LEMO connector on the front of the MC1 whitening board had gotten jiggled ever so slightly - I re-jiggled it till the LR fast channel registered similar number of counts to the other channels. All looks good for now. For good measure, I checked the 3 day trend for the fast PD readback for all 8 SOS optics (40 channels in all, I didn't look at the ETMs as their whitening boards are at the ends), and everything looks okay... This while situation seems very precarious to me, perhaps we should have a more robust signal routing from the OSEMs to the DAQ that is more immune to cable touching etc...

  12659   Fri Dec 2 16:21:12 2016 gautamUpdateGeneralrepaired projector, new mixer arrived and installed

The most recent power outage took out our projector and mixer. The projector was sent for repair while we ordered a new mixer. Both arrived today. Steve is working on re-installing the projector right now, and I installed the mixer which was verified to be working with our DAFI system (although the 60Hz issue still remains to be sorted out). The current channel configuration is:

Ch1: 3.5mm stereo output from pianosa

Ch2: DAFI (L)

Ch3: DAFI (R)

I've set some random gains for now, but we will have audio again when locking laugh

  12660   Fri Dec 2 16:40:29 2016 gautamUpdateIMC24V fuse pulled out

I've pulled out the 24V fuse block which supplies power to the AOM RF driver. The way things are set up on the PSL table, this same voltage source powers the RF amplifiers which amplify the green beatnote signals before sending them to the LSC rack. So I turned off the green beat PDs before pulling out the fuse. I then disconnected the input to the RF driver (it was plugged into a DS345 function generator on the PSL table) and terminated it with a 50 ohm terminator. I want to figure out a smart way of triggering the AOM drive and recording a ringdown on the scope, after which I will re-connect the RF driver to the DS345. The RF driver, as well as the green beat amplifiers and green beat PDs, remain unpowered for now...

  12661   Fri Dec 2 18:02:37 2016 KojiUpdateIOOIMC WFS Demod board measurement & analysis

ELOG of the Wednesday work.

It turned out that the IMC WFS demod boards have the PCB board that has a different pattern for each of 8ch.
In addition, AD831 has a quite narrow leg pitch with legs that are not easily accessible.
Because of these, we (Koji and Rana) decided to leave the demodulator chip untouched.

I have plugged in the board with the WFS2-Q1 channel modified in order to check the significance of the modification.

WFS performance before the modification

Attachment 1 shows the PSD of WFS2-I1_OUT calibrated to be referred to the demodulator output. (i.e. Measured PSDs (cnt/rtHz) were divided by 8.9*2^16/20)
There are three curves: One is the output with the MC locked (WFS servos not engaged). The second is the PSD with the PSL beam blocked (i.e. dark noise). The third is the electronics noise with the RF input terminated and the nominal LO supplied.

This tells us that the measured PSD was dominated by the demodulator noise in the dark condition. And the WFS signal was also dominated by the demod noise below 0.1Hz and above 20Hz. There are annoying features at 0.7, 1.4, 2.1, ... Hz. They basically impose these noise peaks on the stabilized mirror motion.

WFS performance after the modification

Attachment 2 shows the PSD of WFS2-Q1_OUT calibrated to be referred to the demodulator output. (i.e. Measured PSDs (cnt/rtHz) were divided by 21.4*2^16/20)
There are three same curves as the other plot. In addition to these, the PSD of WFS2-I1_OUT with the MC locked is also shown as a red curve for comparison.

This figure tells us that the measured PSD below 20Hz was dominated by the demodulator noise in the dark condition. And the WFS signal is no longer dominated by the electronics noise. However, there still are the peaks at the harmonics of 0.7, 1.4, 2.1, ... Hz. I need further inspection of the FWS demod and whtening boards to track down the cause of these peaks.

Attachment 1: WFS_demod_noise_orig.pdf
WFS_demod_noise_orig.pdf
Attachment 2: WFS_demod_noise_mod.pdf
WFS_demod_noise_mod.pdf
  12662   Sat Dec 3 13:27:35 2016 KojiUpdateIOOIMC WFS Demod board measurement & analysis

ELOG of the work on Thursday

Gautam suggested looking at the preamplifier noise by shorting the input to the first stage. I thought it was a great idea.

To my surprise, the noise of the 2nd stage was really high compared to the model. I proceeded to investigate what was wrong.

It turned out that the resistors used in this sallen-key LPF were thick film resistors. I swapped them with thin film resistors and this gave the huge improvement of the preamplifier noise in the low frequency band.

Attachment 1 shows the summary of the results. Previously the input referred noise of the preamp was the curve in red. We the resistors replaced, it became the curve in magenta, which is pretty close to the expected noise level by LISO model above 3Hz (dashed curves). Unfortunately, the output of the unit with the demodulator connected showed no improvement (blue vs green), because the output is still limited by the demodulator noise. There were harmonic noise peaks at n x 10Hz before the resistor replacement. I wonder if this modification also removed the harmonic noise seen in the CDS signals. I will check this next week.

Attachment 2 shows the current schematic diagram of the demodulator board. The Q of the sallen key filter was adjusted by the gain to have 0.7 (butter worth). We can adjust the Q by the ratio of the capacitance. We can short 3.83K and remove 6.65K next to it. And use 22nF and 47nF for the capacitors at the positive input and the feedback, respectively. This reduces the number of the resistors.

Attachment 1: WFS_demod_noise.pdf
WFS_demod_noise.pdf
Attachment 2: demod.pdf
demod.pdf
  12663   Mon Dec 5 01:58:16 2016 gautamUpdateIMCIMC ringdowns

Over the weekend, I worked a bit on getting these ringdowns going. I will post a more detailed elog tomorrow but here is a quick summary of the changes I made hardware-wise in case anyone sees something unfamiliar in the lab...

  • PDA10CF PD installed on PSL table in the beam path that was previously used for the 3f cancellation trials
  • PDA255 installed on MC2 trans table, long BNC cable running from there to vertex via overhead cable tray
  • PDA255 installed on AS table in front of one of the (currently unused) WFS

I spent a while in preparation for these trials (details tomorrow) like optimizing AOM alignment/diffracted power ratio, checking AOM and PMC switching times etc, but once the hardware is laid out, it is easy to do a bunch of ringdowns in quick succession with an ethernet scope. Tonight I did about 12 ringdowns - but stupidly, for the first 10, I was only saving 1 channel from the oscilloscope instead of the 3 we want to apply the MIT method.

Here is a representative plot of the ringdown - at the moment, I don't have an explanation for the funky oscillations in the reflected PD signal, need to think on this.. More details + analysis to follow...


Dec 5 2016, 130pm:

Actually the plot I meant to put up is this one, which has the time window acquired slightly longer. The feature I am referring to is the 100kHz oscillation in the REFL signal. Any ideas as to what could be causing this?

Attachment 1: IMCringdown.pdf
IMCringdown.pdf
Attachment 2: IMCringdown_2.pdf
IMCringdown_2.pdf
  12664   Mon Dec 5 15:05:37 2016 gautamUpdateLSCMC1 glitches are back

For no apparent reason, the MC1 glitches are back. Nothing has been touched near the PD whitening chassis today, and the trend suggests the glitching started about 3 hours ago.. I had disabled the MC1 watchdog for a while to avoid the damping loop kicking the suspension around when these glitches occur, but have re-enabled it now. IMC is holding lock for some minutes... I was hoping to do another round of ringdowns tonight, but if this persists, its going to be difficult...

  12665   Mon Dec 5 15:55:25 2016 gautamUpdateIMCIMC ringdowns

As promised, here is the more detailed elog.


Part 1: AOM alignment and diffraction efficiency optimization

I started out by plugging in the input to the AOM driver back to the DS345 on the PSL table, after which I re-inserted the 24V fuse that was removed. I first wanted to optimize the AOM alignment and see how well we could cut the input power by driving the AOM. In order to investigate this, I closed the PMC, unlocked the PSL shutter, and dialed the PSL power down to ~100mW using the waveplate in front of the laser. Power before touching anything just before the AOM was 1.36W as measured with the Coherent power meter. 

The photodiode (PDA255) for this experiment was placed downstream of the 1%(?) transmissive optic that steers the beam into the PMC (this PD would also be used in Part 2, but has since been removed)...

Then I tuned the AOM alignment till I maximized the DC power on this newly installed PD. It would have been nicer to have the AOM installed on the mount such that the alignment screws were more easily accessible, but I opted against doing any major re-organization for the time being. Even after optimizing the AOM alignment, the diffraction efficiency was only ~15%, for 1V to the AOM driver input. So I decided to play with the AOM driver a bit.

Note that the AOM driver is powered by 24V DC, even though the spec sheet says it wants 28V. Also, the "ALC" input is left unconnected, which should be fine for our purposes. I opted to not mess with this for the time being - rather, I decided to tweak the RF adjust potentiometer on the front of the unit, which the spec sheet says can adjust the RF power between 1W and 2W. By iteratively tuning this pot and the AOM alignment, I was able to achieve a diffraction efficiency of ~87% (spec sheet tells us to expect 80%), in a switching time of ~130ns (spec sheet tells us to expect 200ns, but this is presumably a function of the beam size in the AOM). These numbers seemed reasonable to me, so I decided to push on. Note that I did not do a thorough check of the linearity of the AOM driver after touching the RF adjust potentiometer as Koji did - this would be relevant if we want to use the AOM as an ISS servo actuator, but for the ringdown, all that matters is the diffraction efficiency and switching time, which seemed satisfactory. 

At this point, I turned the PSL power back up (measured 1.36W just before the AOM). Before this, I estimated the PD would have ~10mW power incident on it, and I wanted it to be more like 1mW, so I I put an ND 1.0 filter on to avoid saturation.


Part 2: PMC "ringdown"

As mentioned in my earlier elog, we want the PMC to cut the light to the IMC in less than 1us. While I was at it, I decided to see if I could do a ringdown measurement for the PMC. For this, I placed two more PDs in addition to the one mentioned in Part 1. One monitored the transmitted intensity (PDA10CF, installed in the old 3f cancellation trial beam path, ~1mW incident on it when PMC is locked and well aligned). I also split off half the light to the PMC REFL CCD (2mW, so after splitting, PMC CCD gets 1mW through some ND filters, and my newly installed PD (PDA255) receives ~1mW). Unfortunately, the PMC ringdown attempts were not successful - the PMC remains locked even if we cut the incident light by 85%. I guess this isn't entirely surprising, given that we aren't completely extinguishing the input light - this document deals with this issue.... But the PMC transmitted intensity does fall in <200ns (see plot in earlier elog), which is what is critical for the IMC ringdown anyways. So I moved on.


Part 3: IMC ringdown

The PDA10CF installed in part 2 was left where it was. The reflected and transmitted light monitors were PDA255. The former was installed in front of the WFS2 QPD on the AS table (needed an ND1.0 filter to avoid damage if the IMC unlocks not as part of the ringdown, in which case ~6mW of power would be incident on this PD), while the latter was installed on the MC2 transmission table. We may have to remove the former, but I don't see any reason to remove the latter PD. I also ran a long cable from the MC2 trans table to the vertex area, which is where I am monitoring the various signals.

  

The triggering arrangement is shown below.

  

To actually do the ringdown, here is the set of steps I followed.

  1. Make sure settings on scope (X & Y scales, triggering) are optimized for data capture. All channels are set to 50ohm input impedance. The trigger comes from the "TTL" output of the DS345, whose "signal" output drives the AOM driver. Set the trigger to external, the mode should be "normal" and not "auto" (this keeps the data on the screen until the next trigger, allowing us to download the data via ethernet.
  2. The DS345 is set to output a low frequency (0.005Hz) square wave, with 1Vpp amplitude, 0.5V offset (so the AOM driver input is driven between 0V and 1V DC, which is what we want). This gives us ~100 seconds to re-lock the IMC, and download the data, all while chilling in the control room
  3. The autolocker was excellent yesterday, re-acquiring the IMC lock in ~30secs almost every time. But in the few instances it didn't work, turn the autolocker off (but make sure the MC2 tickle is on, it helps) and manually lock the IMC by twiddling the gain slider (basically manually do what the autolock script does). As mentioned above, you have ~100 secs to do this, if not just wait for 200secs and the next trigger...
  4. In the meantime, download the data (script details to follow). I've made a little wrapper script (/users/gautam/2016_12_IMCloss/grabChans.sh) which uses Tobin's original python script, which unfortunately only grabs data one channel at a time. The shell script just calls the function thrice, and needs two command line arguments, namely the base name for the files to which the data will be written, and an IP address for the scope...

It is possible to do ~15 ringdowns in an hour, provided the seismic activity is low and the IMC is in a good mood. Unfortunately, I messed up my data acquisiton yesterday, so I only have data from 2 ringdowns, which I will work on fitting and extracting a loss number from. The ringing in the REFL signal is also a mystery to me. I will try using another PDA255 and see if this persists. But anyways, I think we can exclude the later part of the REFL signal, and fit the early exponential decay, in the worst case. The ringdown signal plots have been uploaded to my previous elog. Also, the triggering arrangement can be optimized further, for example by using the binary output from one of our FEs to trigger the actual waveform instead of leaving it in this low frequency oscillation, but given our recent experience with the Binary Output cards, I thought this is unnecessary for the time being...

Data analysis to follow.


I have left all the PDs I put in for this measurement. If anyone needs to remove the one in front of WFS2, go ahead, but I think we can leave the one on the MC2 trans table there...

Attachment 2: AOMswitching.pdf
AOMswitching.pdf
Attachment 6: electricalLayout.pdf
electricalLayout.pdf
  12666   Mon Dec 5 19:29:52 2016 gautamUpdateIMCIMC ringdowns

The MC1 suspension troubles vanished as they came - but the IMC was remaining locked stably so I decided to do another round of ringdowns, and investigate this feature in the reflected light a bit more closely. Over 9 ringdowns, as seen in the below figure, the feature doesn't quite remain the same, but qualitatively the behaviour is similar.

Steve helped me find another PDA255 and so I will try switching out this detector and do another set of ringdowns later tonight. It just occurred to me that I should check the spectrum of the PD output out to high frequencies, but I doubt I will see anything interesting as the waveform looks clean (without oscillations) just before the trigger...

Attachment 1: REFLanomaly.pdf
REFLanomaly.pdf
  12667   Tue Dec 6 00:43:41 2016 gautamUpdateIMCmore IMC ringdowns

In an effor to see if I could narrow down the cause of the 100kHz ringing seen in the reflected PD signal, I tried a few things.

  1. Changed the PD - there was a PDA 255 sitting on the PSL table by the RefCav. Since it wasn't being used, I swapped the PD I was using with this. Unfortunately, this did not solve the problem.
  2. Used a different channel on the oscilloscope - ringing persisted
  3. Changed BNC cable running from PD to oscilloscope - ringing persisted
  4. Checked the spectrum of the PD under dark and steady illumination conditions for any features at 100kHz, saw nothing (as expected) 

I was working under the hypothesis that the ringing was due to some impedance mismatch between the PD output and the oscilloscope, and 4 above supports this. However, most documents I can find online, for example this one, recommend connecting the PD output via 50ohm BNC to a scope with input impedance 50ohms to avoid ringing, which is what I have done. But perhaps I am missing something.

Moreover, the ringdown in reflection actually supplies two of the five variables needed to apply the MIT method of loss estimation. I suppose we could fit the parameter "m4" from the ringdown in transmission, and then use this fitted value on the ringdown in reflection to see where the reflected power settles (i.e. the parameter "m3" as per the MIT paper). I will try analyzing the data on this basis.

I also measured the power levels at each of the PDs, these should allow us to calibrate the PD voltage outputs to power in Watts. All readings were taken with the Ophir power meter, with the filter removed, and the IMC locked.

PD Power level
REFL 0.47 mW (measured before 1.0 ND filter)
Trans 203 uW
Incident 1.06 mW

 

  12668   Tue Dec 6 13:37:02 2016 KojiUpdateIOOIMC WFS Demod board measurement & analysis

I have implemented the modification to the demod boards (Attachment 1).
Now, I am looking at the noise in the whitening board. Attachment 2 shows the comparison of the error signal with the input of the whitening filter shorted and with the 50ohm terminator on the demodulator board. The message is that the whitening filter dominates the noise below 3Hz.

I am looking at the schematic of the whitening board D990196-B. It has an VGA AD602 at the input. I could not find the gain setting for this chip.
If the gain input is fixed at 0V, AD602 has the gain of 10dB. The later stages are the filters. I presume they have the thick film resistors.
Then they may also cause the noise. Not sure which is the case yet.

Also it seems that 0.7Hz noise is still present. We can say that this is coming from the demod board but not on the work bench but in the eurocard crate.

Attachment 1: demod.pdf
demod.pdf
Attachment 2: WFS_error_noise.pdf
WFS_error_noise.pdf
  12669   Tue Dec 6 16:47:40 2016 KojiUpdateIOOIMC WFS whitening filter investigation

The whitening board saids it is Rev B, but the actual component values are more like Rev. C.

The input stage (AD602) has an input resistor of 909 Ohm.
This is causing a big attenuation of the signal (x1/10) because the input impedance of AD602 is not high. And this screws up the logarithm of the gain.
I don't think this is a right approach.

Attachment 1: D990196-C.pdf
D990196-C.pdf
  12670   Tue Dec 6 17:54:08 2016 KojiUpdateIOOIMC WFS whitening filter investigation

The input resistor 909Ohm of AD602 was shorted. I've confirmed that the gain (= attenuation by voltage division) was increased by a factor of 10.
This modification was done for WFS2-I1 and WFS2-Q1. Also the thick film resistors for the WFS2-I1 channel was all replaced with thin film resistors.

Attachment 1 shows the comparison of the noise levels. The curves were all calibrated referred to the response of the original whitening filter configuration.
(i.e. measurement done after the gain change was compensated by the factor of 10.)

Now the AF chain is not limited by the noise in the whitening filter board. (Brown)
In fact, this noise level was completely identical between I1 and Q1. Therefore, I don't think we need this resistor replacement for the whitening filter board.

We can observe the improvement of the overall noise level below 10Hz. (Comparison between green and red/blue)
As the signal level goes up, the noise above 100Hz was also improved.

Now we need to take care of the n x 0.7Hz feature which is in the demod board...
 

Attachment 1: 34.png
34.png
  12671   Tue Dec 6 22:41:49 2016 KojiUpdateIOOIMC WFS whitening filter investigation

I have implemented the same modification (shorting the input resistor of AD602) to the two whitening boards.

  12672   Wed Dec 7 11:52:48 2016 ericqUpdateIMCPartial IMC ringdowns

The transients are likely due to doppler interference due to the input laser frequency sloshing due to errant control signals after the IMC unlock. I performed a few "partial" ringdowns by reducing the power by about 80% while keeping the IMC servo locked. (Function generator at 0.5Vpp square wave, 0.25V offet. Turned IMC boosts off to increase the stable range of the servo).

I need to work out how to extract the loss from this, I think having a partial ringdown may change the calculations somewhat; the time constants in the trans and refl signals are not identical.

Thanks to Gautams nice setup, it was very easy to take these measurements. Thanks! Code and data attached.

Attachment 2: IMCpartial.zip
  12673   Thu Dec 8 07:56:05 2016 SteveUpdatePEMEQ6.5m Northen CA

No damage. ITMY is glitching, so it has not been damped.

 

Attachment 1: eq6.5FerndaleCA.png
eq6.5FerndaleCA.png
Attachment 2: 16d_glitching-_trend.png
16d_glitching-_trend.png
Attachment 3: EQ6.5_&4.7mFerndaleCa.png
EQ6.5_&4.7mFerndaleCa.png
  12674   Thu Dec 8 10:13:43 2016 SteveUpdateLSCglitching ITMY_UL has a history

 

 

Attachment 1: glitching__ITMY-UL_2007.png
glitching__ITMY-UL_2007.png
  12675   Thu Dec 8 19:01:21 2016 ranaUpdateIMCPartial IMC ringdowns

Mach Zucker on howto do Ringdowns:  https://dcc.ligo.org/LIGO-T900007

  12676   Tue Dec 13 17:26:42 2016 KojiUpdateIOOIMC WFS whitening filter investigation

Rana pointed out that this modification (removal of 900Ohm) leave the input impedance as low as 100Ohm.
As OP284 can drive up to 10mA, the input can span only +/-1V with some nonlinearity.

Rather than reinstalling the 900Ohms, Rana will investigate the old-days fix for the whitening filter that may involve the removal of AD602s.
Until the solution is supplied, the IMC WFS project is suspended.

  12677   Wed Dec 14 19:16:57 2016 LydiaUpdateCDSAcromag Binary I/O testing

I looked into converting the QPD whitening switches for the X end to Acromag.

  • To test this out and be able to freely toggle filters without messing anything up, I added a temporary dummy cdsFiltCtrl module (ACROMAG_BIO_TEST) to the c1scx model.
  • The filters can be toggled from the automatically generated medm screen medm/c1scx/C1SCX_ACROMAG_BIO_TEST.adl
  • The control output of the dummy filter bank is sent to a channel named C1:SCX-ACROMAG_SWCTRL.
  • I was able to read in the appropriate bits from there and send them to the appropriate acromag channel using a calcout channel.
    • I couldn't get individual bo channels to work. This Acromag module is configured to write to 4 channels at a time, so I set that up with an analog output channel. The calcout channel shifts each relevant bit from C1:SCX-ACROMAG_SWCTRL to the right place for writing to the Acromag. 
  • I connected the Acromag XT1111 Binary I/O unit to a temporary power supply and verified that toggling the filters on and off changed the output appropriately. This is a sinking output model so the output pin is connected to the return if the switch is on. 

The plan from here:

  • Determine how to configure these outputs to be compatible with the QPD whitening board.
  • Modify the SUS PD whitening board to always use the analog filter and remove digital option in models.
  • Test DACs 
  • Verify that the QPD whitening gain switches aren't doing anything
  • Assemble new Acromag box for X end and connect to QPD whitening, SUS PD whitening and SOS driver boards
  12678   Thu Dec 15 03:46:19 2016 ranaUpdateIOOIMC WFS whitening filter investigation

https://dcc.ligo.org/LIGO-D1400414

As it turns out, its not so old as I thought. Jenne and I reworked these in 2014-2015. The QPD whitening is the same as the IMC WFS whitening so we can just repeat those fixes here for the IMC.

Quote:

Rana pointed out that this modification (removal of 900Ohm) leave the input impedance as low as 100Ohm.
As OP284 can drive up to 10mA, the input can span only +/-1V with some nonlinearity.

Rather than reinstalling the 900Ohms, Rana will investigate the old-days fix for the whitening filter that may involve the removal of AD602s.
Until the solution is supplied, the IMC WFS project is suspended.

 

  12681   Thu Dec 22 09:37:20 2016 SteveUpdateVACRGA scan at day 63

Valve configuration: vacuum normal

RGA head temp: 43.5 C

Vac envelope temp: 23 C

 

Attachment 1: pd80-d63.png
pd80-d63.png
Attachment 2: pd80-580Hz-d63.png
pd80-580Hz-d63.png
  12687   Thu Dec 29 10:24:56 2016 SteveUpdatePEMEQ5.7mHawthorn NV

Sus damping restored.

 

Attachment 1: eq5.7HawthorneNV.png
eq5.7HawthorneNV.png
  12691   Thu Dec 29 21:48:32 2016 ranaUpdateIOOMC AutoLocker hung because c1iool0 asleep again

MC unlocked, Autolocker waiting for c1iool0 EPICS channels to respond. c1iool0 was responding to ping, but not to telnet. Keyed the crate and its coming back now.

There's many mentions of c1iool0 in the recent past, so it seems like its demise must be imminent. Good thing we have an Acromag team on top of things!

Also, the beam on WFS2 is too high and the autolocker is tickling the Input switch on the servo board too much: this is redundant / conflicting with the MC2 tickler.

  12692   Fri Dec 30 10:27:46 2016 ranaUpdateDetCharsummary pages dead again

Dead again. No outputs for the past month. We really need a cron job to check this out rather than wait for someone to look at the web page.

  12693   Thu Jan 5 21:43:16 2017 ranaUpdateDetCharsummary pages dead again

Max tells us that soem conf files were bad and that he did something and now some pages are being made. But the PEM and MEDM pages are bank. Also the ASC tab looks bogus to me.

  12695   Sun Jan 8 12:47:06 2017 ranaUpdateGeneralOptical Layout in DCC

Manasa pointed me to the CAD drawings in the 40m SVN and I've now uploaded them to the 40m DCC Tree so that EricG and SteveV can convert them into SolidWorks.

  12696   Mon Jan 9 09:18:47 2017 SteveUpdatePEMpower glitch

There was a power glitch last night around 1:15am

The vacuum was not effected.

PSL laser turned on, PMC locked, PSL shutter opened and MC locked.

IR lasers at the ends turned on.

East arm air cond turned on.

The computers are all done.

The last power glitch was at Nov 3, 2016

 

 

Attachment 1: MondayMorning.png
MondayMorning.png
  12697   Mon Jan 9 16:12:30 2017 SteveUpdateGeneralOptical Layout in DCC

Caltech Facilities promissed to email the 40m facility drawings in Cad format.

I organized the old of optical , vacuum and facility layout drawings on paper in the old cabinet. 

Quote:

Manasa pointed me to the CAD drawings in the 40m SVN and I've now uploaded them to the 40m DCC Tree so that EricG and SteveV can convert them into SolidWorks.

 

Attachment 1: drawings_on_paper.jpg
drawings_on_paper.jpg
  12698   Tue Jan 10 14:24:09 2017 SteveUpdateVACRGA scan at day 82

Valve configuration: vacuum normal

Vacuum envelope: 23C

Rga head: 44C

 

Attachment 1: pd80VNd82.png
pd80VNd82.png
  12699   Tue Jan 10 16:20:11 2017 SteveUpdateCDSpower glitch......Raid is rebuilding

Jamie started the fm40m Raid rebuilding. It has been beeping since the power outage.

Summary pages have no reading since power glitch.

 

Attachment 1: rebuilding_in_progress.png
rebuilding_in_progress.png
  12700   Tue Jan 10 21:47:00 2017 ranaUpdateCDSpower glitch

Does "done" mean they are OK or they are somehow damaged? Do you mean the workstations or the front end machines?

The computers are all done.

megatron and optimus are not responding to ping commands or ssh -- please power them up if they are off; we need them to get data remotely

  12701   Tue Jan 10 22:55:43 2017 gautamUpdateCDSpower glitch - recovery steps

Here is a link to an elog with the steps I had to follow the last time there was a similar power glitch.

The RAID array restart was also done not too long ago, we should also do a data consistency check as detailed here, if not already..

If someone hasn't found the time to do this, I can take care of it tomorrow afternoon after I am back.

Quote:

Does "done" mean they are OK or they are somehow damaged? Do you mean the workstations or the front end machines?

The computers are all done.

megatron and optimus are not responding to ping commands or ssh -- please power them up if they are off; we need them to get data remotely

 

  12702   Wed Jan 11 16:35:03 2017 gautamUpdateCDSpower glitch - recovery progress

[lydia, ericq, gautam]

We set about following the instructions linked in the previous elog. A few notes/remarks:

  1. It is important to run the ntpdate commands before restarting the models. Sometimes, multiple restarts of the models were required to turn all the indicator blocks on the MEDM screen green.
  2. There was also an issue of multiple ntpd processes running on the same machine, which obviously caused all sorts of timing havoc. EricQ helped us diagnose and fix these. At the moment, all the lights are green on the CDS status MEDM screen
  3. On the hardware side, apart from the usual suspects of frontends/megatron/optimus/fb needing to be rebooted, I noticed that the ETMX OSEM lights were off on the control room monitors. Investigation pointed to the 2 20V sorensens at the X end outputting 0V, 0A after the power glitch. We turned down both dials, and then gradually ramped them up again. Both Sorensens now read +/-20V, 0.3A, which is in agreement with the label stuck onto them.
  4. Restarted MC autolocker and FSS Slow scripts on megatron. I have not yet looked at the status of the nds2 server on megatron.
  5. 11 MHz Marconi has yet to be restarted - but I am unable to get even the IMC locked at the moment. For some reason, the RMS of the MC1 and MC3 coils are way higher than what I am used to seeing (~5mV rms as compared to the <1mV rms I am used to seeing for a damped optic). I will investigate further. Leaving MC autolocker disabled for now.
  12703   Wed Jan 11 19:20:23 2017 Max IsiUpdateSummary PagesDecember outage

The summary pages were not successfully generated for a long period of time at the end of 2016 due to syntax errors in the PEM and Weather configuration files.

These errors caused the INI parser to crash and brought down the whole gwsumm system. It seems that changes in the configuration of the Condor daemon at the CIT clusters may have made our infrastructure less robust against these kinds of problems (which would explain why there wasn't a better error message/alert), but this requires further investigation.

In any case, the solution was as simple as correcting the typos in the config side (on the nodus side) and restarting the cron jobs (on the cluster side, by doing `condor_rm 40m && condor_submit DetectorChar/condor/gw_daily_summary.sub`). Producing pages for the missing days will take some time (how to do so for a particular day is explained in the wiki https://wiki-40m.ligo.caltech.edu/DailySummaryHelp).

RXA: later, Max sent us this secret note:

However, I realize it might not be clear from the page which are the key steps. These are just running:

1) ./DetectorChar/bin/gw_daily_summary --day YYYYMMDD --file-tag some_custom_tag To create pages for day YYYYMMDD (the file-tag option is not strictly necessary but will prevent conflict with other instances of the code running simultaneously).

2) sync those days back to nodus by doing, eg: ./DetectorChar/bin/pushnodus 20160701 20160702

This must all be done from the cluster using the 40m shared account.
  12704   Thu Jan 12 02:45:53 2017 JohannesUpdateGeneralNext armloss steps

As stated in elog 12618, using an oscilloscope to average the reflected powers and thus circumventing all filtering yielded much better results than before:

XARM: 21 +/- 35 ppm
YARM: 69 +/- 45 ppm

We can probably decrease the measurement uncertainty further by using a larger photodiode that is more suited for DC measurements. It will be placed in the AS pathtemporarily. If we get below 10 ppm systematic errors will begin to matter. To get those under control I will have to re-determine the visibility in the arm cavities and the modulation indices. The numbers to match from an estimate via the power recycing gain are <= 50 ppm arm average from elog 12586. Once the measurement scheme is up and running, we can proceed to generate ETM lossmaps. ITM will still be tricky but let's see what we can do.

Following Yutaro's approach, we can move the beams on the optcs in a deterministic way by several mm on the ETMs. Moving the beam is achieved by introducing offsets into the ASS auto alignment. As an example, the Yaw dither for ETMY is shown:

Each of the 8 test mass rotational degrees of freedom is driven by a particular frequency, and 2 signals are digitally demodulated in the real-time system: The arm transmission ("T") and the LSC arm length feedback signal to the ETM (L). The T signal feeds back to the input pointing, aka Tip Tilts and BS. This maximizes the transmission for a given test mass orientation. The L feedback controls the beam position on the mirrors in the arms. It minimizes the coupling of the dither to the length feedback, which is achieved when the beam goes through the axis of the rotational motion. This is where we introduce the offset:

The signal C1:ASS-YARM_ETM_YAW_L_DEMOD_I_OFFSET (for this example) moves the locking point of the dither-to-length coupling and thus moves the beam around on the ETM. This is true for the PIT and YAW of all test masses except ITMX. In the current configuration the TTs optimize the alignment into the YARM, and for the X we only have the BS, which is why the beam spot on ITMX cannot be independently controlled as-is. We could, however, for the sake of this measurement, temporarily temporarily give TT authority to the XARM feedback to control the ITMX beam position. I imagine something like dither-aligning with ASS the normal way, and then run a customized script in which the XARM is treated as the YARM, feecback to the BS is cut, and the YAW signals are inverted due to the reflection on BS.

Knowing the angle of the offset gives us a way to calculate the beam spot displacement with the cavity geometry. For best results I want to make sure our OpLev calibration is still good (laser power decay, although last time this was done was only about a year ago), which would be analogous to elog 11831.

As for ITM beam position, this scheme only works partially, because it would require the beam to steer further off its axis than in the ETM case. This is problematic because of the spacing between tip tilts and ITMs. I summarize:

  1. Place larger DCPD in AS path
  2. Confirm mode-matching and mod-indices
  3. Assess loss in center with zero offsets
  4. Uncertainty low enough? If not get better.
  5. Calibrate OpLevs
  6. Introduce calibrated offsets in dither alignment
  7. Wander beam on test masses, recording arm losses
  8. ???
  9. Profit
Attachment 1: ass_illustration.pdf
ass_illustration.pdf
  12708   Thu Jan 12 17:31:51 2017 gautamUpdateCDSDC errors

The IFO is more or less back to an operational state. Some details:

  1. The IMC mirror excess motion alluded to in the previous elog was due to some timing issues on c1sus. The "DAC" and "DK" blocks in the c1x02 diag word were red instead of green. Restarting all the models on c1sus fixed the problem
  2. When c1ioo was restarted, all of Koji's changes (digital) to the MC WFS servo where lost as they were not committed to the SDF. Eric suggested that I could just restore them from burt snapshots, which is what I did. I used the c1iooepics.snap file from 12:19PM PST on 26 December 2016, which was a time when the WFS servo was working well as per this elog by Koji. I have also committed all the changes to the SDF. IMC alignment has been stable for the last 4 hours.
  3. Johannes aligned and locked the arms today. There was a large DC offset on POX11, which was zeroed out by closing the PSL shutter and running LSC offsets. Both arms lock and stay aligned now.
  4. The doubling oven controller at the Y end was switched off. Johannes turned it on.
  5. Eric and I started a data consistency check on the RAID array yesterday, it has completed today and indicated no issues
  6. NDS2 is now running again on megatron so channel access from outside should(???) be possible again.

One error persists - the "DC" indicator (data concentrator?) on the CDS medm screen for the various models spontaneously go red and return to green often. Is this a known issue with an easy fix?

  12709   Thu Jan 12 23:22:34 2017 ranaUpdateSummary PagesDecember outage

Pages still not working: PEM and MEDM blank.

  • Committed existing MEDM grabbing scripts to SVN. Ran the cron job on megatron by hand. It grabs PNG files, but somehow its not getting into the summary pages.
  • Changed the MEDM grabbing scripts to use '/usr/bin/env'.
  • GW summary log files were numbering in the many thousands, so I moved everything over 320 days old into the OLD/ sub-directory using 'find . -type f -mtime +320 -exec mv {} OLD/ \;' (the semi-colon is needed)
  • Did apt-get upgrade on Megatron.
  • pinged Max
  • Stared at GWsumm docs to see if there's a clue about what (if anything) is wrong with the .ini file.
  12710   Fri Jan 13 08:54:32 2017 JohannesUpdateGeneralDC PD installed

I installed a DC PD (Thorlabs PDA 520) in the beam path to AS55. I placed a 2" 90/10 BS on a flip mount that picks of enough light for the PD to spit out ~8V when the port is bright. Single arm continuous signal will be ~2V. While most of the light still continues towards AS55, the displacement from the BS moves the beam off AS55, so I used the flip mount in case anyone needs to use AS55. The current configuration is UP.

When we're done with loss investigations the flip mount should be removed from the bench.

I hooked the PD up to an ethernet-enabled scope and started scripting the loss map measurement (scope can receive commands via http so we can automate the data acquisition). The scope that was present at the bench and had been used for the MC ringdown measurements had a 'scrambled' screen that I couldn't fix so I had to retrieve another scope ("scope1"). I'll try to find out what's wrong with it but we may have to send it in for repair.

 

  12711   Fri Jan 13 10:53:03 2017 SteveUpdatePEMdoors are fixed

Control room to outside door was realigned.

                                                   It is self closing now.

Control room to IFO door lock optimized to soft closing.

All other doors lubricated by Alex of the key shop.

  12712   Fri Jan 13 14:18:28 2017 SteveUpdatePEMair condition fixed

The old control room AC  has been stick in heating mode for about 2 months. It's thermostate and fan belt  was finally replaced. It was calibrated and set to 71 F ( just behind 1X6 on west wall ) around 1pm.

Out belt; sad inside 

at 4 pm Rana cried

It must be too tight.

Attachment 1: PEM_120d.png
PEM_120d.png
  12713   Fri Jan 13 14:33:00 2017 MAX (not Rana)UpdateSummary PagesDecember outage

PEM config file was also lacking a section named "summary", which is necessary for all parent tabs; this has now been solved. I have deactivated the MEDM pages because Praful's screencap script seemed to be broken (I should have logged this, I apologize).

Quote:

Pages still not working: PEM and MEDM blank.

  • Committed existing MEDM grabbing scripts to SVN. Ran the cron job on megatron by hand. It grabs PNG files, but somehow its not getting into the summary pages.
  • Changed the MEDM grabbing scripts to use '/usr/bin/env'.
  • GW summary log files were numbering in the many thousands, so I moved everything over 320 days old into the OLD/ sub-directory using 'find . -type f -mtime +320 -exec mv {} OLD/ \;' (the semi-colon is needed)
  • Did apt-get upgrade on Megatron.
  • pinged Max
  • Stared at GWsumm docs to see if there's a clue about what (if anything) is wrong with the .ini file.

 

  12715   Fri Jan 13 21:41:23 2017 KojiUpdateCDSDC errors

I think I fixed the DC error issue

1. I added the leap second (leapsecond ?) entry for 2016/12/31, 23:60:00 UTC to daqdrc


[OLD]
set gps_leaps = 820108813 914803214 1119744016;
[NEW]
set gps_leaps = 820108813 914803214 1119744016 1167264018;

2. Restarted FB and all realtime models

Now I don't see any RED light.

  12716   Fri Jan 13 23:39:46 2017 gautamUpdateGeneralETMX suspension electronics problems?

[Koji,gautam]

After Koji's leap second fix, we were playing around with the X arm locking. In particular, we were playing around with the limit value on the X arm LSC filter bank - the nominal value is 4000, we wanted to see if we could increase this without kicking the optic while acquiring arm lock. We initially increased it to 8000, and then turned it off altogether. Then we rapidly turned the output of the servo ON/OFF, and looked at the arm transmission to see if it came back to the level before unlocking, as an indication of whether the optic was kicked.

These trials suggested a value of 8000 for the limiter was OK, so we left the LSC mode on with the limiter set to 8000. But just as we were about to leave for the night, I noticed on the wall Striptool that the X arm was unlocked. Investigating, we found that the green wasn't even locking to a HOM. Further investigation of the Oplev spot showed that ETMX had received a large kick (both pitch and law errors were ~200urad). ITMX was unaffected.

We initially tried lowering the LSC limit value back to 4000, then used first the Oplev spot and then the green to align the arm. But turning on LSC misaligned the arm after acquiring lock. So we decided to leave LSC off, thinking that the notorious ETMX suspension problems have resurfaced. As a diagnostic, we figured we'd leave the watchdog tripped, and use the Oplev to see if the optic was getting kicked. But the act of turning the watchdog off kicked the optic again (WHY?!).

Looking at the ETMX sus screen, turning off all the damping and LSC (but watchdog on) still leaves a non-zero offset in the "Vmon" field, between 0.02-0.05V depending on the coil. Turning the watchdog OFF takes all these to 0.009V, although I can see the LR value fluctuating between 0.004V and 0.009V. I went to the Xend and squished all the cables on the Sat. Box, but the problem persisted.

At this time, I can't think of any explanation, so I am giving up for the night. To avoid unnecessarily kicking the optic, I am going to unplug the suspension from the Sat. Box and leave one of our tester boxes plugged in, lets see if that sheds any light on the situation...


Notes:

  1. The +/-20V sorensens at this end were "tripped" for a few days after the power glitch until they were reset and turned back on yesterday. But this should not affect Vmon, as these Sorensens only supply the DC voltage for the coil bias, which is a slow machine channel?
  2. The X arm was staying locked and well aligned for hours on end earlier this afternon - in fact it was locked for about 2 hours 6-8 hours ago, I can still see the trace on the wall StripTool....
  12718   Sat Jan 14 12:12:03 2017 ranaUpdateDAQminute trends missing

Did we turn off minute trend writing in one of the recent FrameBuilder debug sessions? Seems we only have second trends in 2016. Maybe this explains why its so slow to get minute trends? Dataviewer has to rebuild it from second trend.

controls@nodus|frames > l
total 64
drwx------   2 root     root     16384 Jun  8  2009 lost+found/
drwxr-xr-x   2 controls controls  4096 Jul 14  2015 tmp/
-rw-r--r--   1 controls controls     0 Jul 14  2015 test-file
drwxr-xr-x   5 controls controls  4096 Apr  7  2016 trend/
drwxr-xr-x   4 root     root      4096 Apr 11  2016 archive/
drwxr-xr-x 789 controls controls 36864 Jan 13 19:34 full/
controls@nodus|frames > cd trend
controls@nodus|trend > l
total 3340
drwxr-xr-x 258 controls controls 3342336 Jul  6  2015 minute_raw/
drwxr-xr-x 387 controls controls   36864 Nov  5  2015 minute/
drwxr-xr-x 969 controls controls   36864 Jan 13 19:49 second/

  12719   Sat Jan 14 12:36:57 2017 ericqUpdateDAQminute trends missing

Yes, writing minute trends causes hourly FB crashes in the current state of things. The "raw" minute trending is turned on, but I think that these are unknown to nds.

  12722   Mon Jan 16 18:54:01 2017 ranaUpdateSUSBS: whitening re-engaged

Found that the BS whitening was off. Gautam says that "it has always been that way" and "there's nothing in the elog about this" and "I have no special relationship with Putin".

I looked at DV and DTT while turning the OSEM whitening back on. As expected, the sensor noise improved by 10x above 10 Hz. The time series shows no problems - its just less fuzzy now.

All OSEM spectra after the switch show on upper panel of plot. Lower panel shows comparison of BS UL before/after. To rotate the DTT PDF landscape output I typed this:

pdftk BS-white.pdf cat 1N output BSwhite.pdf

"if you see something, do something"

Attachment 1: BSwhite.pdf
BSwhite.pdf
  12725   Mon Jan 16 23:25:07 2017 gautamUpdateSUSMC1 SUS electronics investigation

[rana,gautam]

Summary:

  • MC1 glitchy behaviour is back
  • Found a broken LEMO cable, left unplugged for the night -> to be repaired tomorrow
  • Further diagnosis to follow

During the course of Rana's inspection of the general state of the IFO, he commented that there seemed to be several seismic-related IMC lock losses in the time that he had been observing it. This issue looked suspiciously like the the MC1 glitches I had noticed sometime late last year, especially since each time the IMC would unlock, we could see significant amounts of motion on MC REFL. To diagnose, we did the following:

  1. Closed PSL shutter
  2. Ramped down the gains of the MC1 damping loops by factor of 1000 in ~4 secs using z step
  3. Shut down the watchdog for MC1
  4. Observed dataviewer traces for glitches

Sure enough, there were several glitches that occurred in all 5 sensor channels. These glitches varied in size from a few counts (the smaller ones) to 60-70counts for the bigger ones. In the past, squishing the LEMO connector on the front of the PD whitening board (D000210) had apparently made the glitching go away. So tonight, for starters, we squished everything else - Sat. Box connectors, the breakout board from Sat. Box to whitening board on the back of 1X6, and the DB connector on the front of the whitening board. This had no effect - the glitching remained consistent.

Next, Rana pulled out two of the three 4pin LEMOs, and left only those coresponding to UL/LL plugged in - but the glitching persisted in these two channels. We then pulled out the board. It was installed in 1998, but has a sticker on it that says "fixed in 2003". Not sure what the fix was. Visual inspection of the circuit didn't show anything obviously faulty, but it did look like the two MAX333A quad switches (these control whether the whitening is bypassed or not) had been replaced at some point. There are other undesirable features, such as the use of thick film resistors, but nothing that would explain the glitchy behaviour.

Next, we re-inserted the whitening board back into its original slot in the Eurocrate, but switched the cables (both D sub and LEMO, but only on the whitening board end) between the boards for MC1 and MC3 (i.e. MC1 cables were routed through the whitening board that was originally used for MC3, and vice-versa). But the glitches remained consistent on the MC1 channels. So it looks like the board is not a likely culprit.

Finally, we went in and squished all the cables from the PD whitening board to the ADC (via an AA filter board). For some of the LEMO cables from the whitening board, the LEMO backshells were not properly tightened. Rana fixed these before putting them back in. Some of the connectors were also not pushed in tightly enough, Rana heard the click when he pushed them in. The cables from the adaptor board to the ADC itself looked fine, it was screwed on at both ends, and all these connections looked snug enough. In the interest of completeness, Rana also pushed in the backplane connectors on the Eurocrate (these supply the signals from the BIO cards to switch the whitening ON/OFF). The one corresponding to MC1 was indeed a little loose.

Coming back to the control room, we saw that the MC1 LR sensor was dead. After some investigation, Rana found that on the AA filter board end, one of the 4pin LEMOs from the whitening board had one of its wires come unstuck from where it was soldered (this presumably happened while we were squishing cables tonight, as the LR channel was fine before that). Also, there was no heat shrink used on any of the solder joints. Could this explain the glitchy behaviour? Perhaps, but the glitches remained in the 3 channels that were connected. Anyways, I will repair this cable tomorrow, and we can see if this has fixed the problem or not..


Some misc points:

  1. Regarding the adaptor boards that take the PD signals from the satellite box and route it to the whitening board, there are some clamps that hold the IDE connectors in place for MC1, MC2 and MC3 boards, but not for the others (see attached picture). Steve, can we install clamps for all of the boards? [taken care of, see here]
  2. The whitening boards are not screwed in place into the Eurocrate. This should be rectified.

PSL shutter is closed, MC1 watchdog is shutdown for the night.

Attachment 1: 20170116_231625.png
20170116_231625.png
Attachment 2: IMG_7175.JPG
IMG_7175.JPG
Attachment 3: IMG_7174.JPG
IMG_7174.JPG
  12726   Tue Jan 17 20:39:30 2017 ranaUpdateComputer Scripts / Programsnodus web apache simlinks too soft

I tried to follow these instructions today to make the Simulink Webview accessible:

controls@nodus|public_html > ln -sfn /users/public_html/FE /export/home/

Quote:

The story is: we currently don't expose the whole /users/public_html folder. Instead, we are symlinking the folders from public_html to /export/home/ on nodus, which is where apache looks for things

So, I fixed the links on the Core Optics page by running:

controls@nodus|~ > ln -sfn /users/public_html/40m_phasemap /export/home/

But...I got a "403 Forbidden" message. What is the secret handshake to get this to work? And why have we added this extra step of security?

ELOG V3.1.3-