BOOM SHAKA-LAKA

M4.5 EQ in LA 2020-09-19 06:38:46 (UTC) / -1d 23:38:46 (PDT) https://earthquake.usgs.gov/earthquakes/eventpage/ci38695658/executive

I only checked the watchdogs. All watchdogs were tripped. ITMX and ETMY seemed stuck (or have the OSEM magnet issue). They were left tripped. The watchdogs for the other SUSs were reloaded.

the seismometers obviously saturated during the EQ, but the accelerometers captured some of it. It looks like there's different saturation levels on different sensors.

Also, it seems the mounting of the MC2 accelerometers is not so good. There's some ~10-20 Hz resonance its mount that's showing up. Either its the MC2 chamber legs or the accelerometers are clamped poorly to the MC2 baseplate.

Sun Sep 20 00:02:36 2020 edit: fixed indexing error in plots

* also assuming that the sensors are correctly calibrated in the front end to 1 count = 1 um/s^2 (this is what's used in the summ pages)

the EQ was ~14 km south of Caltech and 17 km deep

I'm amazed at how much higher the noise is on the MC2 accelerometer. Is that really how much amplification of the ground motion we're getting? If so, its as if the MC has no vibration isolation from the ground in that band. We should put one set on the ground and make the more direct comparison of the spectra. Also, perhaps do some seismic FF using this sensor - I'm not sure how successful we've been in this band.

Attaching the coherence plot from ldvw.ligo.caltech.edu (apparently it has access to the 40m data, so we can use that as an alternative to dtt or python for remote analysis):

It would be interesting to see if we can use the ML based FF technology from this summer's SURF project by Nadia to increase the coherence by including some slow IMC alignment channels.

All suspension watchdogs were tripped ~90mins ago. I restored the damping. IMC is locked.

ITMX was stuck. I set it free. But notice that the UL Sensor RMS is higher than the other 4? I thought ITMY UL was problematic, but maybe ITMX has also failed, or maybe it's coincidence? Something for IFOtest to figure out I guess. I don't think there is a cable switch between ITMX/ITMY as when I move the ITMX actuators, the ITMX sensors respond and I can also see the optic moving on the camera.

Took me a while to figure out what's going on because we don't have the seis BLRMS - i moved the usual projector striptool traces to the TV screen for better diagnostic ability.

Update 16 July 1515: Even though the RMS is computed from the slow readback channels, for diagnosis, I looked at the spectra of the fast PD monitoring channels (i.e. *_SENSOR_*) for ITMX - looks like the increased UL RMS is coming from enhanced BR-mode coupling and not of any issues with the whitening switching (which seems to work as advertised, see Attachment #3, where the LL traces are meant to be representative of LL, LR, SD and UR channels).

Earth quake M5.3    2018-04-05 19:29:16UTC          Santa Cruz Island, CA

Nothing tripped. No obvious damage.

There seems to be something funny going on with MATLAB's license authentication on the control room workstations. Earlier today, I was able to start MATLAB on pianosa, but now attempting to run /cvs/cds/caltech/apps/linux64/matlab/bin/matlab -desktop results in the message: 

License checkout failed.
License Manager Error -15
MATLAB is unable to connect to the license server. 
Check that the license manager has been started, and that the MATLAB client machine can communicate
with the license server.

Troubleshoot this issue by visiting: 
http://www.mathworks.com/support/lme/R2013a/15

Diagnostic Information:
Feature: MATLAB 
License path: /home/controls/.matlab/R2013a_licenses:/cvs/cds/caltech/apps/linux64/matlab/licenses/license.dat:/cv
s/cds/caltech/apps/linux64/matlab/licenses/network.lic 
Licensing error: -15,570. System Error: 115

Tried to open MATLAB on Donatella and found the error:

MATLAB is selecting SOFTWARE OPENGL rendering.

License checkout failed.
License Manager Error -9
This error may occur when: 
-The hostid of this computer does not match the hostid in the license file. 
-A Designated Computer installation is in use by another user. 
If no other user is currently running MATLAB, you may need to activate.

Troubleshoot this issue by visiting: 
http://www.mathworks.com/support/lme/R2015b/9

Diagnostic Information:
Feature: MATLAB 
License path: /home/controls/.matlab/R2015b_licenses/license_donatella_865865_R2015b.lic:/cvs/cds/caltech/apps/lin
ux64/matlab15b/licenses/license.dat:/cvs/cds/caltech/apps/linux64/matlab15b/licenses/license_chiara_
865865_R2015b.lic:/cvs/cds/caltech/apps/linux64/matlab15b/licenses/license_pianosa_865865_R2015b.lic 
Licensing error: -9,57.

So I used  my caltech credentials to get an activation key for the computer. I could not find the option for a campus license so I used the individual single machine license.

Now it can be run by going to the location:

and running

On opening MATLAB, there were a whole bunch of other errors including a low-level graphics error when we tried to plot something.

When I came to the 40m this afternoon, the MC was unlocked. Here is the trend of MC_F for last 2 hours

C1:PSL-PMC_PMCTRANSPD = 0.800

Should I just disable the auto locker or try to realign it?

I measured the dark noise and regular intensity noise in MC trans tonight to get some estimate for the free running spectrum that the Chas ISS must overcome. PDF is attached.

XML file is /users/rana/dtt/MC-trans_130325.xml

The RIN normalization was done by using the mean of the SUM time series. The dark noise was measured by misaligning MC2 and making the same measurement with the bright normalization.

1. Found the low pass filters OFF in a couple of the MC SUS damping loops. This allows injection of OSEM noise all the way up to ~100 Hz. I deleted unused ones and turned on the 'Cheby' for all SUS DOFs: cheby1("LowPass",2,0.1,3)cheby1("LowPass",6,1,12)gain(1.13501)
2. Tuned the Bounce/Roll filters to catch the bounce and roll modes for all 3 MC SUS (based on the Mech Res Wiki page). These are now engaged on ALL MC SUS DOFs. They have been deleted from the ASCPIT/YAW filter banks and moved into the WFS screen into the MC1,2,3 filter banks there to be more in line with our knowledge of multi loop instability from notches in individual loops:ellip("BandStop",4,1,40,15.9,17.2)ellip("BandStop",4,1,40,23.5,24.7)gain(1.25893)

MC loop is near unstable. Common gain reduced. Needs more loop tuning.

We've often seen that the MC gets into a state where it keeps losing lock and the EOM drive shows a large RMS. We've usually been looking at the noise spectrum to diagnose this.

Tonight we finally just measured the OLG. The attached plot shows the loop gain measured with the 4395 on the MC servo board

Although the phase margin is a healthy 45 degrees, its close to instability at 1 MHz. For this plot, I reduced the gain by 3 dB and now the margin is ~7 dB. So usually its pretty close to unstable and at least its always making a noise peak.

That whole TF above a few hundred kHz is weird. We should tune out whatever makes it so flat and also remove the resonance that makes the 1 MHz peak; maybe its from some post mixer low pass?

Anyone interested in helping in the investigation ought to measure the TF of the MC demod board, the MC servo board, and the FSS box.

Silver lining: if we fix this loop shape, we might be able to have a much more stable IMC and IFO.

[Kiwamu, Suresh]

We attempted to minimise the A2L coupling in the MC mirrors (centering the beam spot on the actuation node on each optic).  While it was easy to minimise the coupling in the pitch for all the three optics and yaw of MC2, the yaw alignment of MC1 and MC3 proved to be difficult.  For one the adjustment required was quite large, so much so that PSL alignment into the MC is often lost during this adujstment.  We had to align the PSL coupling several times in order to proceed.   And the MC settles into a new position when the MC-PSL servo loop was disturbed by random denizens in the lab.  Requiring us to start over again.

Kiwamu wrote a script to measure the MC(optic)(Pitch/yaw) -> Lockin(#1 to #6) matrix.  Inverting this matrix gave us the linear combination of the offsets to put on the MC# PIT/YAW  inorder to minimise a specific Lockin output.  However the cross couplings were not completely eliminated.  This made it very hard to predict what a given set of offsets were going to do to the Lockin outputs.

Net result:  the spots are centered in vertical direction (pitch) but not in the horizontal (yaw)

Day time tasks have started so I am quitting now.

MC unlocked over the weekend and also got severely mis-aligned. It all started around midnight on Saturday.

At first I thought that this was due to the MCS CPU meter being railed at 60 us, so I deleted a bunch of filters in MC1,2,3 that are unused and left over from Den's quantization noise investigations. This reduced the CPU load somewhat, but didn't make any real improvements. Turning on the ASC filter banks in the MC SUS still mis-aligned the MC.

With the MC WFS and MC ASS turned off, there is still some digital junk coming in and misaligning things. Plot attached.

Similar stuff coming in on ITMX, but not ITMY.

Tried restarting various FEs, but there was no effect. Also tried rebooting c1lsc, c1ioo, & c1sus. Finally did 'shutdown -r now' on all 5 computers on the CDS overview screen and simultaneously (almost) pressed the reset button on the RFM switch above the old c1pem crate. Everything came back OK except for c1oaf (I had to manually button his BURT button) and now the ASC inputs on all the SUS are zero when they should be and MC is well locked and aligned.

Rob and I used to do this trick when he thought that a cosmic ray had corrupted a bit in the RFM network.

In the c1ioo model there were filter modules at the output of the WFS output matrix, right before going to the MC SUS ASCs but right after the dither line inputs, that were not exposed in the C1IOO_WFS_OVERVIEW screen (bad!).  I switched the order of these modules and the dither sums, so these output filters are now before the dither inputs.  This will allow us to turn off all the WFS feedback while still allowing the dither lines.

I updated the medm screens as well (see attached images).

• 1) turned of the autolocker running the MC-down script
  2) brought the reflected beam spot back on the MC-reflection camera and on the reflection photodiode (REFL-PD)
  3) turned on the LSC servo
  4) tweaked the periscope's mirrors until the cavity got locked on a TEM00 mode
  5) tweaked the periscope aiming at ~0.3V from the REFL-PD and ~3V on the transmission photodiode (TRANS-PD).
  

Trying to track the MC positions back for a few days, it seems that the data hasn't been recorded properly for a while. Something happened yesterday after my boot fest and then the record got restored. Attached here are the readbacks showing the event for MC1.

Is anything wrong with the data record?

MC unlocked, Autolocker waiting for c1iool0 EPICS channels to respond. c1iool0 was responding to ping, but not to telnet. Keyed the crate and its coming back now.

There's many mentions of c1iool0 in the recent past, so it seems like its demise must be imminent. Good thing we have an Acromag team on top of things!

Also, the beam on WFS2 is too high and the autolocker is tickling the Input switch on the servo board too much: this is redundant / conflicting with the MC2 tickler.

c1iool0 was down again. Rather than key the crate, this time I just pushed the reset button on the front and it came back.

As move towards the wonderfulness of AcroMag, we also have to buy a computer  to handle all of these IOCs. Let's install the new c1iool0 over by the SUS computer.

Someone had left a typo in the MC autolocker script recently while trying to set the lock threshold to 0.09. As a result, the autlocker wouldn't run.

I repaired it, made a few readability improvements, and checked in the new version to the SVN. If you make script changes, check them in. If you think its too minor of a change for a SVN checkin, don't do it at all.

/opt/rtcds/caltech/c1/scripts/MC/AutoLockMC.csh  was modified last night.

1. Autolocker sometimes forget to turn off the MC2Tickle. I added the following lines to make sure to turn it off.

    echo autolockMCmain: MC locked, nothing for me to do >> ${lfnam}
    echo just in case turn off MC2 tickle >> ${lfnam}
    ${SCRIPTS}/MC/MC2tickleOFF

2. During the lock acquisition, Autolocker frequently stuck on a weak mode. So the following lines were added
so that the Autolocker toggles the servo switch while waiting for the lock.

    echo autolockMCmain: Mon=$mclockstatus, Waiting for MC to lock .. >> ${lfnam}
    # Turn off MC Servo Input button
    ezcawrite C1:IOO-MC_SW1 1
    date >> ${lfnam}
    sleep 0.5;
    # Turn on MC Servo Input button
    ezcawrite C1:IOO-MC_SW1 0
    sleep 0.5;

 Just a quick update on the status of the auto locker:

The auto locker now runs and respawns automatically on megatron, through ubuntu's "upstart" init system, instead of cron. 

• The autolocker script itself is:/opt/rtcds/caltech/c1/scripts/MC/AutoLockMC.csh
• The startup script called by the upstart system is: /opt/rtcds/caltech/c1/scripts/MC/AutoLockMC.init
• The upstart configuration is: /etc/init/MCautolocker.conf
• The auto locker is started by executing this command on megatron: sudo initctl start MCautolocker

This was pretty simple to set up, once I figured out how to provide the necessary environment variables in AutoLockMC.init 

[Koji Rana]

MC Autolocker was updated. (i.e. mcup and mcdown were updated)

mcup:

• Turn on the MCL input and output switches
• Change the MCL gain from 0 to -300 with nominal ramp time of 5sec
• Turn on FM2, FM5, MF7 after a sleep of 5sec. Note: FM1 FM8 FM9 are always on.
• Set the offset of 42 counts
• Turn on the offset

# Turn on MCL servo loop
echo mcup: Turning on MCL servo loop...
date
ezcaswitch C1:SUS-MC2_MCL INPUT OUTPUT ON
ezcawrite C1:SUS-MC2_MCL_GAIN -300
sleep 5
ezcaswitch C1:SUS-MC2_MCL FM2 FM5 FM7 ON
# Offset to take off the ADC offset of MC_F
ezcawrite C1:SUS-MC2_MCL_OFFSET 42
ezcaswitch C1:SUS-MC2_MCL OFFSET ON


This offset of 42 count is applied in order to compensate the ADC offset of MC_F channel.
The MCL servo squishes the MC_F signal. i.e. The DC component of MC_F goes to zero.
However, if the ADC of MC_F has an offset, the actual analog MC_F signal, which is fed to FSS BOX,
still keep some offset. This analog offset causes deviation from the operating point of the FSS (i.e. 5V).

mcdown:

• Basically the revese process of mcup.
• This script keeps FM1 FM8 FM9 turned on.

# Turn off MCL servo loop
echo mcdown: Turning off MCL servo loop...
date
ezcawrite C1:SUS-MC2_MCL_GAIN 0
ezcaswitch C1:SUS-MC2_MCL INPUT OUTPUT OFFSET FMALL OFF FM1 FM8 FM9 ON
# Remove Offset to take off the ADC offset of MC_F
ezcawrite C1:SUS-MC2_MCL_OFFSET 0

The MC autolocker hasn't been so snappy recently, and has been especially fussy today. Previously, the mcup script was triggered immediately once the transmission was above a certain threshold. However, this could waste time if it was just an errant flash. Hence, I've added a 0.5 second delay and a second threshold check before mcup is triggered. 

After breaking the lock 5ish times, it does seem to come back quicker.

 Somehow the caget/caput commands are really slow. I'm not sure if this is new behavior or not, but after changing values, it takes ~1-2 seconds to move on to the next command.

On op340m, once I ran autolockMCmain40m in the foreground, I could see that there were a lot of error messages that weren't being caught in the log file. On Solaris, the TDS and EZCA binaries work well, but the caget/pu stuff is missing a lot of the new flags and the new CDSUTILS python scripts are not there. So I decided to stop running on there and move over to running the MC autolocker on megatron. I also changes the mcup and mcdown scripts to BASH from CSH. There are still some PERL commands in there from Rob/Matt/Sam, so I also added the proper commands so that the pick up the scripts/general/perlmodules/ directory.

Yes, yes, I know. You would all feel warm in your tummy if we did everything in python because its the best language ever, but how about we lock the interferometer first and then we can rewrite all the last decades worth of scripts?

Looked at the trend from the last time the MCWFS were one (2 weeks ago!) and aligned the MC suspension back to that point. After that the WFS come on fine and MC looks good. I fixed the nameserver/DNS/fstab stuff on megatron, asia, and belladonna.

I've left AutoLockMC.csh (too painful to convert to BASH) running on megatron via an open terminal on pianosa. If it looks OK for the weekend, Q should move the crontab line from op340m over to megatron and then update the Wiki appropriately.

CDSUTILS is also gone from the path on all the workstations, so we need Jamie to tell us by ELOG how to set it up, or else we have to use ezcaread / ezcawrite forever.

Today I checked out the SR560 around the lab. I confirmed that the one with the label "channel A noisy" is effectively mulfuctioning.

It was coonected to the lock-in amplifier set up for the drum mode peak readout.

I repleaced that with a working one.

The error spectra I took so far are not that informative, I'm afraid. The first three posted here refer to Wed 17 in the afternoon, where things were quiet, the LSC control was off and the MC was reliably locked. The last two plots refer to Wed night, while Q and I were doing some locking work; in particular, these were taken just after one of the locklosses described in elog 10814. Sadly, they aren't much different from the "quiet" ones.

I can add some considerations though: Q and I saw some weird effects during that night, using a live reading of such spectra, which couldn't be saved though; such effects were quite fast both in appearance and disapperance, therefore difficult to save using the snapshot measurement, which is the only one that can save the data as of now; moreover, these effects were certainly seen during the locklosses, but sometimes also in normal circumstances. What we saw was a broad peak in the range 5e4-1e5 Hz with peak value ~1e-5 V/rtHz, just after the main peak shown in the attached spectra.

che tristezza  

What we want is to have the high and low noise spectra on the same plot. The high noise one should be triggered by a high PC DRIVE signal.

There was an abrupt change in the MC_F spectrum between August 4 and August 5, judging by the summary pages - the 1 and 3 Hz resonances are no longer visible in the spectrum. Possibly, this indicates some electronics failure on the MC servo board / whitening board, the CDS settings don't seem to have changed. There is no record of any activity in the elog around those dates that would explain such a change. I'll poke around at 1X2 to see if anything looks different.

Update 1740: I found that the MCL / MCF cables were disconnected. So since August 5, these channels were NOT recording any physical quantity. Because their inputs weren't terminated, I guess this isn't a clean measurement of the whitening + AA noise, but particularly for MC_F, I guess we could use more whitening (see Attachment #1). Probably also means that the wandering ~10-30Hz line in the spectrogram is a electronics feature. The connections have now been restored and things look nominal again.

that's a very curious disconnection

the "Pentek" whitening board that carries the MC channels has jumpers to enable either 1 or 2 stages of 15:150 whitening. Looks lik MC_F has 2 and MC_L has 1.

I guess the MC_F signal is so low because of the high gain on the FSS board.  We could lower the FSS common gain and increase the IMC board's VCO gain to make up for this. Maybe 6 dB would be enough. IF that is risky, we could also up the analog gain on the whitening board.

This elog suggests that there is uniformly 1 stage engaged across all channels. I didn't look at the board to see what the jumper situation is, but only 1 stage of whitening is compensated digitally for both _F and _L. The Pomona box attached to the NPRO PZT input is also compensated digitally to convert counts to frequency.

I tried the gain re-allocation between VCO gain and FSS COMM (and also compensated for the cts to Hz conversion in MCF), but it doesn't seem to have the desired effect on the MCF SNR in the 5-50Hz band. Since the IMC stays locked, and I had already made the changes to mcup, I'll keep these gains for now. We can revert to the old settings if the IMC locking duty cycle is affected. Explicitly, the changes made were:

VCO gain: +7dB ---> +13 dB

FSS COMM: +6 ddB ---> +0 dB

The mcdown script wasn't modified, so the lock acquisition gains are the same as they've been. 

I've been looking into the data of Jan 06 and Jan 15 taken during daytime, as the night before we left the PRC aligned in order to allow the IFO to flash; the purpose is to find out if some flashes from the IFO could propagate back to the IMC and cause it to lose lock; I will show here a sample plot, all of the others are in the archive attached.

My impression is that these locklosses of the IMC are not caused by these flashes: the signals MC_[F/L] seem quite stable until the lock loss, and I don't see any correlation with what happens to REFLDC that could cause the lockloss (apart from its drop as a consequence of the lockloss itself); in addition, in most occasions I noticed that the FSS started to go crazy just before the lock loss, and that suggests me that the lockloss source is internal to the IMC.

I can't see anything strange happen to MC_TRANS either as long as the IMC is locked, no fluctuations or weird behaviour. I also plotted the MC_REFL_SUM channel. but it is too slow to be useful for this kind of "hunt".

The input offset on the MC length servo board changes the lock point of the length loop (by how much? need to calibrate this slider into meters & Hz).

The SUM signal on the MC WFS is ~few 1000. This is several times larger than the pit/yaw signals. This is bad. it means that the TEM00 mode on the WFS (or what the WFS interperets as a TEM00) is larger than the TEM01/10 that its supposed to measure.

So if the beam moves on the WFS head it will convert this large common mode signal into a differential one.

We moved the MC Servo offset around from -3 to +3 V today and saw that it does affect the transmitted light level, but we need to think more to see how to put the offset at the real center of the resonance. This is complicated by the fact that the MCWFS loops seem to have some several minutes time constant so things are essentially always drifting.

1. Characterize and juice up the WFS loops.
2. Figure out how to set the MC length loop offset. Is this bad offset changing the zero point of the MC WFS loops?
3. If so, it may be a source of excess jitter noise in the interferometer.

I changed the McREFL SMOO to make it easier to use this noisy channel to diagnose small alignment changes:

caput C1:IOO-MC_RFPD_DCMON.SMOO 0.1

This morning, after Steve pointed out that the readout RFAMPD_DC was zero, I thought of realigning the beam on the photodiode. Maybe I touched the lens or the beam splitter that send the beam on the diode when I installed an other beam splitter to make the measurement of the calibration between two ThorLabs PDA255 photodiodes.

After aligning the beam on the RFAMPD, the voltage of the DC readout was lower than it used to be (C1:IOO-RFAMPD_DC ~ 0.4 now vs. 4 as it was on November 4th).

I maximized the DC readout but the problem seems to be that the beam spot is not a round TEM00. In particular the spot looks like that of a TEM10 mode.

Since we're looking at the MC transmitted beam, is it possible that the MC is locked on the wrong mode?

Check out the attached picture.

I found that the MC autolocker was OFF. Kiwamu says he turned it off because its slow. Suresh says that he has some feelings that maybe something is wrong. I'll let them describe what they know about the MC in an elog.

I checked the trend of the MC and PMC transmissions for the past 30 days:

Looks like the alignment has been drifitng. PMC was corrected recently by Koji, but the alignment of the input beam to the MC or the MC itself has to be fixed. Has someone been twiddling the MC SUS alignment biases??

[Valera, Suresh]

The first time I noticed that the MC was not locking was after I had finished switching the RF source installation.  Before this change the RF modulation frequency (for MC) was 29.485 MHz as read from the Marconi RF Source.  We replaced this with a Wenzel crystal source at 29.491 MHz.  This may have changed the loop gain. 

Today, I changed the MC alignment to optimise the MC lock.  Valera pointed out that this is not a desirable solution since it would shift the beam pointing for all components downstream.  However, since we are not sure what was the last stable configuration, we decided to stay with the current settings for now and see the trends of several parameters which would tell us if something is drifting and causing the autolocker to fail.

The MC Auto locker is now working okay.  However to obtain lock initially we reduced the loop gain by decreasing the VCO gain.  We then increased the gain after the autolocker had locked the MC.

I measured the mode cleaner open loop gain with the HP3563A. 

The UGF is 64kHz, phase margin is 28 deg.

I measured the mode cleaner open loop gain.  It's around 60kHz with 29 degs of phase margin.

Since Suresh has installed the RF source box and changed the cable configuration somewhat,

the demodulation phase for the MC locking became off by about 10 degree.

I changed the length of some cables and obtained a good demodulation phase by the same technique as Suresh and Koji did before (see here for detail).

I maximized the Q signal. The lock of the MC looks healthy.