40m QIL Cryo_Lab CTN SUS_Lab TCS_Lab OMC_Lab CRIME_Lab FEA ENG_Labs OptContFac Mariner WBEEShop
 40m Log, Page 242 of 344 Not logged in
ID Date Author Type Category Subject
10992   Tue Feb 10 02:40:54 2015 JenneUpdateLSCSome locking thoughts on PRMI

[EricQ, Jenne]

We wanted to make the PRMI lock more stable tonight, which would hopefully allow us to hold lock much longer.  Some success, but nothing out-of-this-world.

We realized late last week that we haven't been using the whitening for the ASDC and POPDC signals, which are combined to make the MICH error signal.  ASDC whitening is on, and seems great.  POPDC whitening (even if turned on after lock is acquired) seems to make the PRMI lock more fussy.  I need to look at this tomorrow, to see if we're saturating anything when the whitening is engaged for POPDC.

One of the annoying things about losing the PRMI lock (when CARM and DARM have kept ALS lock) is that the UGF servos wander off, so you can't just reacquire the lock.  I have added triggering to the UGF servo input, so that if the cavity is unlocked (really, untriggered), the UGF servo input gets a zero, and so isn't integrating up to infinity.  It might need a brief "wait" in there, since any flashes allow signal through, and those can add up over time if it takes a while for the PRMI to relock.  UGF screens reflect this new change.

10994   Tue Feb 10 03:09:02 2015 ericqUpdateLSCSome locking thoughts on PRMI

Unfortunately, we only had one good CARM offset reduction to powers of about 25, but then my QPD loop blew it. We spent the vast majority of the night dealing with headaches and annoyances.

Things that were a pain:

• If TRX is showing large excursions after finding resonance, there is no hope. These translate into large impulses while reducing the CARM offset, which the PRMI has no chance of handling. The first time aligning the green beat did not help this. For some reason, the second time did, though the beatnote amplitude wasn't increased noticibly.
• NOTICE: We should re-align the X green beatnote every night, after a solid ASS run, before any serious locking work.
• Afterwards, phase tracker UGFs (which depend on beatnote amplitude, and thereby frequency) should be frequently checked.
• We suffered some amount from ETMX wandering. Not only for realigning between lock attempts, but on one occasion, with CARM held off, GTRX wandered to half its nominal value, leading to a huge effective DARM offset, which made it impossible to lock MICH with any reasonble power in the arms. Other times, simply turning off POX/POY locking, after setting up the beatnotes, was enough to significantly change the alignment.
• IMC was mildly tempermental, at its worst refusing to lock for ~20min. One suspicion I have is that when the PMC PZT is nearing its rail, things go bad. The PZT voltage was above 200 when this was happening, after relocking the PMC to ~150, it seems ok. I thing I've also had this problem at PZT voltages of ~50. Something to look out for.

Other stuff:

• We are excited for the prospect of the FOL system, as chasing the FSS temperature around is no fun.
• UGF servo triggering greatly helps the PRMI reacquire if it briefly flashes out, since the multipliers don't run away. This exacerbated the ALS excursion problem.
• Using POPDC whitening made it very tough to hold the PRMI. Maybe because we didn't reset the dark offset...?
10995   Tue Feb 10 13:48:58 2015 manasaUpdateLSCProbable cause for headaches last night

I found the PSL enclosure open (about a feet wide) on the north side this morning. I am assuming that whoever did the X beatnote alignment last night forgot to close the door to the enclosure before locking attempts

 Quote: Unfortunately, we only had one good CARM offset reduction to powers of about 25, but then my QPD loop blew it. We spent the vast majority of the night dealing with headaches and annoyances.  Things that were a pain: If TRX is showing large excursions after finding resonance, there is no hope. These translate into large impulses while reducing the CARM offset, which the PRMI has no chance of handling. The first time aligning the green beat did not help this. For some reason, the second time did, though the beatnote amplitude wasn't increased noticibly.  NOTICE: We should re-align the X green beatnote every night, after a solid ASS run, before any serious locking work.  Afterwards, phase tracker UGFs (which depend on beatnote amplitude, and thereby frequency) should be frequently checked.  We suffered some amount from ETMX wandering. Not only for realigning between lock attempts, but on one occasion, with CARM held off, GTRX wandered to half its nominal value, leading to a huge effective DARM offset, which made it impossible to lock MICH with any reasonble power in the arms. Other times, simply turning off POX/POY locking, after setting up the beatnotes, was enough to significantly change the alignment.  IMC was mildly tempermental, at its worst refusing to lock for ~20min. One suspicion I have is that when the PMC PZT is nearing its rail, things go bad. The PZT voltage was above 200 when this was happening, after relocking the PMC to ~150, it seems ok. I thing I've also had this problem at PZT voltages of ~50. Something to look out for.  Other stuff: We are excited for the prospect of the FOL system, as chasing the FSS temperature around is no fun.  UGF servo triggering greatly helps the PRMI reacquire if it briefly flashes out, since the multipliers don't run away. This exacerbated the ALS excursion problem.  Using POPDC whitening made it very tough to hold the PRMI. Maybe because we didn't reset the dark offset...?

10996   Tue Feb 10 16:01:21 2015 manasaUpdateGeneralAUX X fiber coupled 72%

Plan C finally worked. We have 1.454mW of AUX X light at the PSL table (2mW incident on the fiber coupler).

Attached is the layout of the telescope.

What I did:

I stuck in Lens 1 (f=200mm) and measured the beam width after the focus of the lens at several points. I fit the data and calculated the beam waist and its position after this lens.

I used the calculated waist and matched it with an appropriate lens and target (fiber coupler) distance. I calculated the maximum coupling efficiency to be 77% for Lens 2 with f=50mm and the fiber coupler placed at 20cm from the waist of Lens1. I was able to obtain 72% coupling after putting the telescope together.

I locked the arms, ran ASS and brought back GTRX to its usual optimum value of ~0.5 counts after closing. We also have the X arm beatnote on the spectrum analyzer.

Notes:

There are still a couple of things to fix. The rejected beam from the beam sampler has to be dumped using a razor blade.

10997   Tue Feb 10 18:37:17 2015 ericqUpdateASCQPD ASC ready to go

I've remeasured the QPD ASC sensing coefficients, and figured out what I did weird with the actuation coefficients. I've rearranged the controller filters to be able to turn on the boost in a triggered way, and written Up/Down scripts that I've tested numerous times, and Jenne has used as well; they are exposed on the ASC screen.

All four loops (2 arms * pit,yaw), have their gains set for 8Hz UGF, and have 60 degrees of phase margin. The loop shape is the same as the previous ELOG post. Here is the current on/off performance. The PDH signals (not shown, but in attached xml) show no extra noise, and the low frequency RIN goes down a bit, whic is good. The oplevs error signals are a bit noisier, but I suppose that's unavoidable. The Y-arm performs a bit better than the X-arm.

The up/down scripts don't touch the filters' trigger settings at all, just handles switching the input and output and clearing history. FM1 contains the boost which is intended to have a longer trigger delay than the filters themselves.

10998   Wed Feb 11 00:07:54 2015 ranaUpdateLSCLock Loss plot

Here is a lock loss from around 11 PM tonight. Might be due to poor PRC signals.  $\oint {\frac{\partial PRCL}{\partial x}}$

This is with arm powers of ~6-10. You can see that with such a large MICH offset, POP22 signal has gone done to zero. Perhaps this is why the optical gain for PRCL has also dropped by a factor of 30 .

This seems untenable . We must try this whole thing with less MICH offset so that we can have a reasonable PRCL signal.

10999   Wed Feb 11 02:42:05 2015 JenneUpdateLSCPRC error signal RF spectra

Since we're having trouble keeping the PRC locked as we reduce the CARM offset, and we saw that the POP22 power is significantly lower in the 25% MICH offset case than without a MICH offset, Rana suggested having a look at the RF spectra of the REFL33 photodiode, to see what's going on.

The Agilent is hooked up to the RF monitor on the REFL33 demod board.  The REFL33 PD has a notch at 11MHz and another at 55MHz, and a peak at 33MHz.

We took a set of spectra with MICH at 25% offset, and another set with MICH at 15% offset.  Each of these sets has 4 traces, each at a different CARM offset.  Out at high CARM offset, the arm power vs. CARM offset is pretty much independent of MICH offset, so the CARM offsets are roughly the same between the 2 MICH offset plots.

What we see is that for MICH offset of 25%, the REFL33 signal is getting smaller with smaller CARM offset!!  This means, as Rana mentioned earlier this evening, that there's no way we can hold the PRC locked if we reduce the CARM offset any more.

However, for the MICH offset 15% case, the REFL 33 signal is getting bigger, which indicates that we should be able to hold the PRC.  We are still losing PRC lock, but perhaps it's back to mundane things like actuator saturation, etc.

The moral of the story is that the 3f locking seems to not be as good with large MICH offsets.  We need a quick Mist simulation to reproduce the plots below, to make sure this all jives with what we expect from simulation.

For the plots, the blue trace has the true frequency, and each successive trace is offset in frequency by a factor of 1MHz from the last, just so that it's easier to see the individual peak heights.

Here is the plot with MICH at 25% offset:

And here is the plot with MICH at 15% offset:

Note that the analyzer was in "spectrum" mode, so the peak heights are the true rms values.  These spectra are from the monitor point, which is 1/10th the value that is actually used.  So, these peak heights (mVrms level) times 10 is what we're sending into the mixer.  These are pretty reasonable levels, and it's likely that we aren't saturating things in the PD head with these levels.

The peaks at 100MHz, 130MHz and 170MHz that do not change height with CARM offset or MICH offset, we assume are some electronics noise, and not a true optical signal.

Also, a note to Q, the new netgpib scripts didn't write data in a format that was back-compatible with the old netgpib stuff, so Rana reverted a bunch of things in that directory back to the most recent version that was working with his plotting scripts.  sorry.

11000   Wed Feb 11 03:41:12 2015 KojiUpdateLSCPRC error signal RF spectra

As the measurements have been done under feedback control, the lower RF peak height does not necessary mean
the lower optical gain although it may be the case this time.

These non-33MHz signals are embarassingly high!
We also need to check how these non-primary RF signals may cause spourious contributions in the error signals,
including the other PDs.

11001   Wed Feb 11 04:08:53 2015 JenneUpdateLSCNew Locking Paradigm?

[Rana, Jenne]

While meditating over what to do about the fact that we can't seem to hold PRMI lock while reducing the CARM offset, we have started to nucleate a different idea for locking

We aren't sure if perhaps there is some obvious flaw (other than it may be tricky to implement) that we're not thinking about, so we invite comments.  I'll make a cartoon and post it tomorrow, but the idea goes like this.....

Can we use ALS to hold both CARM and DARM by actuating on the ETMs, and sit at (nominally) zero offset for all degrees of freedom?  PRMI would need to be stably held with 3f signals throughout this process.

1) Once we're close to zero offset, we should see some PDH signal in REFL11.  With appropriate triggering (REFLDC goes low, and REFL11I crosses zero), catch the zero crossing of REFL11I, and feed it back to MC2. We may want to use REFL11 normalized by the sum of the arm transmissions to some power (1, 0.5, or somewhere in between may maximize the linear range even more, according to Kiwamu).  The idea (very similar to the philosophy of CESAR) is that we're using ALS to start the stabilization, so that we can catch the REFL11 zero crossing.

2) Now, the problem with doing the above is that actuating on the mode cleaner length will change the laser frequency.  But, we know how much we are actuating, so we can feed forward the control signal from the REFL11 carm loop to the ALS carm loop.  The goal is to change the laser frequency to lock it to the arms, without affecting the ALS lock.  This is the part where we assume we might be sleepy, and missing out on some obvious reason why this won't work.

3) Once we have CARM doubly locked (ALS pushing on ETMs, REFL11 pushing on MC/laser frequency), we can turn off the ALS system. Once we have the linear REFL11 error signal, we know that we have enough digital gain and bandwidth to hold CARM locked, and we should be able to eek out a slightly higher UGF since there won't be as many digital hops for the error signal to transverse.

4) The next step is to turn on the high bandwidth common mode servo.  If ALS is still on at this point, it will get drowned out by the high gain CM servo, so it will be effectively off.

5) Somewhere in here we need to transition DARM to AS55Q.  Probably that can happen after we've turned on the digital REFL11 path, but it can also probably wait until after the CM board is on.

The potential show-stoppers:

Are we double counting frequency cancellation or something somewhere?  Is it actually possible to change the laser frequency without affecting the ALS system?

Can we hold PRMI lock on 3f even at zero CARM offset?  Anecdotally from a few trials in the last hour or so, it seems like coming in from negative carm offset is more successful - we get to slightly higher arm powers before the PRMI loses lock.  We should check if we think this will work in principle and we're just saturating something somewhere, or if 3f can't hold us to zero carm offset no matter what.

A note on technique:  We should be able to get the transfer function between MC2 actuation and ALS frequency by either a direct measurement, or Wiener filtering.  We need this in order to get the frequency subtraction to work in the correct units.

11002   Wed Feb 11 16:49:41 2015 ranaUpdateelogELOGD restarted

No elog response from outside and no elogd process on nodus, so I restarted it using 'start-elog.csh'.

11003   Wed Feb 11 17:31:11 2015 ericqUpdateLSCRFPD spectra

For future reference, I've taken spectra of our various RFPDs while the PRMI was sideband locked on REFL33, using a 20dB RF coupler at the RF input of the demodulator boards. The 20dB coupling loss has been added back in on the plots. Data files are attached in a zip.

Exceptions:

• The REFL165 trace was taken at the input of the amplifier that immediately preceeds the demod board.
• The 'POPBB' trace was taken with the coupler at the input of the bias tee, that leads to an amplifier, then splitter, then the 110 and 22 demod boards.

I also completely removed the cabling for REFLDC -> CM board, since it doesn't look like we plan on using it anytime in the immediate future.

11004   Wed Feb 11 18:07:42 2015 ericqUpdateLSCRFPD spectra

After some discussion with Koji, I've asked Steve to order some SBP-30+ bandpass filters as a quick and cheap way to help out REFL33. (Also some SBP-60+ for 55MHz, since we only have 1*fmod and 2*fmod bandpasses here in the lab).

11006   Wed Feb 11 18:44:53 2015 manasaUpdateGeneralOptical fiber module

I have moved the optical fiber module for FOL to the PSL table. It is setup on the optical table right now for testing.

Once tests are done, the box will move to the rack inside the PSL enclosure.

While doing any beat note alignment, please watch out for the loose fibers at the north side of the PSL enclosure until they are sheilded securely (probably tomorrow morning).

11007   Wed Feb 11 22:13:44 2015 JenneUpdateLSCNew Locking Paradigm - LSC model changes

In order to try out the new locking scheme tonight, I have modified the LSC model.  Screens have not yet been made.

It's a bit of a special case, so you must use the appropriate filter banks:

CARM filter bank should be used for ALS lock.  MC filter bank should be used for the REFL1f signal.

The output of the MC filter bank is fed to a new filter bank (C1:LSC-MC_CTRL_FF).  The output of this new filter bank is summed with the error point of the CARM filter bank (after the CARM triggered switch).

The MC triggering situation is now a little more sophisticated than it was.  The old trigger is still there (which will be used for something like indicating when the REFL DC has dipped).  That trigger is now AND-ed with a new zero crossing trigger, to make the final trigger decision.  For the zero crossing triggering, there is a small matrix (C1:LSC-ZERO_CROSS_MTRX) to choose what REFL 1f signal you'd like to use (in order, REFL11I, REFL11Q, REFL55I, REFL55Q).  The absolute value of this is compared to a threshold, which is set with the epics value C1:LSC-ZERO_CROSS_THRESH.  So, if the absolute value of your chosen RF signal is lower than the threshold, this outputs a 1, which is AND-ed by the usual schmidt trigger.

At this moment, the input and output switches of the new filter bank are off, and the gain is set to zero.  Also, the zero crossing selection matrix is all zeros, and the threshold is set to 1e9, so it is always triggered, which means that effectively MC filter bank just has it's usual, old triggering situation.

11008   Thu Feb 12 01:00:18 2015 ranaUpdateLSCRFPD spectra

The nonlinearity in the LSC detection chain (cf T050268) comes from the photodetector and not the demod board. The demod board has low pass or band pass filters which Suresh installed a long time ago (we should check out what's in REFL33 demod board).

Inside the photodetector the nonlinearity comes about because of photodiode bias modulation (aka the Grote effect) and slew rate limited distortion in the MAX4107 preamp.

11009   Thu Feb 12 01:43:09 2015 ranaUpdateLSCNew Locking Paradigm - LSC model changes

With the Y Arm locked, we checked that we indeed can get loop decoupling using this technique.

The guess filter that we plugged in is a complex pole pair at 1 Hz. We guessed that the DC gain should be ~4.5 nm count. We then converted this number into Hz and then into deg(?) using some of Jenne's secret numbers. Then after measuring, we had to increase this number by 14.3 dB to an overall filter module gain of +9.3.

The RED trace is the usual 'open loop gain' measurement we make, but this time just on the LSC-MC path (which is the POY11_I -> ETMY path).

The BLUE trace is the TF between the ALS-Y phase tracker output and the FF cancellation signal. We want this to be equal ideally.

The GREEN trace is after the summing point of the ALS and the FF. So this would go to zero when the cancellation is perfect.

So, not bad for a first try. Looks like its good at DC and worse near the red loop UGF. It doesn't change much if I turn off the ALS loop (which I was running with ~10-15x lower than nominal gain just to keep it out of the picture). We need Jenne to think about the loop algebra a little more and give us our next filter shape iteration and then we should be good.

11010   Thu Feb 12 03:43:54 2015 ericqUpdateLSC3F PRMI at zero ALS CARM

I have been able to recover the ability to sit at zero CARM offset while the PRMI is locked on RELF33 and CARM/DARM are on ALS, effectively indefinitely. However, I feel like the transmon QPDs are not behaving ideally, because the reported arm powers freqently go negative as the interferometer is "buzzing" through resonance, so I'm not sure how useful they'll be as normalizing signals for REFL11. I tried tweaking the DARM offset to help the buildup, since ALS is only roughly centered on zero for both CARM and DARM, but didn't have much luck.

Example:

Turning off the whitening on the QPD segments seems to make everything saturate, so some thinking with daytime brain is in order.

How I got there:

It turns out triggering is more important than the phase margin story I had been telling myself. Also, I lost a lot of time to needing demod angle change in REFL33. Maybe I somehow caused this when I was all up on the LSC rack today?

We have previously put TRX and TRY triggering elements into the PRCL and MICH rows, to guard against temporary POP22 dips, because if arm powers are greater than 1, power recylcing is happening, so we should keep the loops engaged. However, since TRX and TRY are going negative when we buzz back and forth through the resonsnace, the trigger row sums to a negative value, and the PRMI loops give up.

Instead, we can used the fortuitously unwhitened POPDC, which can serve the same function, and does not have the tendancy to go negative. Once I enabled this, I was able to just sit there as the IFO angrily buzzed at me.

Here are my PRMI settings

REFL33 - Rotation 140.2 Degrees, -89.794 measured diff

PRCL = 1 x REFL33 I; G = -0.03; Acquire FMs 4,5; Trigger FMs 2, 9; Limit: 15k ; Acutate 1 x PRM

MICH = 1 x REFL33 Q, G= 3.0, Acquire FMs 4,5,8; Trigger FM 2, 3; Limit: 30k; Actuate -0.2625 x PRM + 0.5 x BS

Triggers = 1 x POP22 I + 0.1 * POPDC, 50 up 5 down

Just for kicks, here's a video of the buzzing as experienced in the control room

11011   Thu Feb 12 11:14:29 2015 JenneUpdateLSCNew Locking Paradigm - Loop-gebra

I have calculated the response of this new 2.5 loop system.

The first attachment is my block diagram of the system.  In the bottom left corner are the one-hop responses from each green-colored point to the next.  I use the same matrix formalism that we use for Optickle, which Rana described in the loop-ology context in http://nodus.ligo.caltech.edu:8080/40m/10899

In the bottom right corner is the closed loop response of the whole system.

Also attached is a zipped version of the mathematica notebook used to do the calculation.

EDIT, JCD, 17Feb2015:  Updated loop diagram and calculation:  http://131.215.115.52:8080/40m/11043

11012   Thu Feb 12 11:59:58 2015 KojiUpdateLSCNew Locking Paradigm - Loop-gebra

The goals are:

- When the REFL path is dead (e.g. S_REFL = 0), the system goes back to the ordinary ALS loop. => True (Good)

- When the REFL path is working, the system becomes insensityve to the ALS loop
(i.e. The ALS loop is inactivated without turning off the loop.) => True when (...) = 0

Are they correct?

Then I just repeat the same question as yesterday:

S is a constant, and Ps are cavity poles. So,  approximately to say, (...) = 0 is realized by making D = 1/G_REFL.
In fact, if we tap the D-path before the G_REFL, we remove this G_REFL from (...). (=simpler)
But then, this means that the method is rather cancellation between the error signals than
cancellation between the actuation. Is this intuitively reasonable? Or my goal above is wrong?

11013   Thu Feb 12 12:16:04 2015 manasaUpdateGeneralFiber shielding

[Steve, Manasa]

The fibers around the PSL table were shielded to avoid any tampering.

11014   Thu Feb 12 12:23:21 2015 manasaUpdateGeneralWaking up the PDFR measurement system

[EricG, Manasa]

We woke up the PDFR measurement setup that has been sleeping since summer. We ran a check for the laser module and the multiplexer module. We tried setting things up for measuring frequency response of AS55.
We could not repeat Nichin's measurements because the gpib scripts are outdated and need to be revised.

PDFR diode laser was shutdown after this job.

11015   Thu Feb 12 15:21:37 2015 ericqUpdateComputer Scripts / Programsnetgpib updates

I've fixed the gpib scripts for the SR785 and AG4395A to output data in the same format as expected by older scripts when called by them. In addition, there are now some easier modes of operation through the measurement scripts SRmeasure and AGmeasure. These are on the \$PATH for the main control room machines, and live in scripts/general/netgpib

Case 1: I manually set up a measurement on the analyzer, and just want to download / plot the data.

Make sure you have a yellow prologix box plugged in, and can ping the address it is labeled with. (i.e. 'vanna'). Then, in the directory you want to save the data, run:

SRmeasure -i vanna -f mydata --getdata --plot

This saves mydata_(datetime).txt and mydata_(datetime).pdf in the current directory.

In all cases, AGmeasure has the identical syntax. If the GPIB address is something other than 10, specifiy it with -a, but this is rarely the case.

Case 2: I want to remotely specify a measurement

Rather than a series of command line arguments, which may get lost to the mists of time, I've set the scripts up to use parameter files that serve as arguments to the scripts.

Get the templates for spectrum and TF measurements in your current directory by running

SRmeasure --template

Set the parameters with your text editor of choice, such as frequency span, filename output, whether to create a plot or not, then run the measurement:

SRmeasure SR785template.yml

Case 3: I want to compare my data with previous measurements

In the template parameter files, there is an option 'plotRefs', that will automatically plot the data from files whose filenames start with the same string as the current measurement.

If, in the "#" commented out header of the data file, there is a line that contains "memo:" or "timestamp:", it will include the text that follows in the plot legend.

There are also methods to remotely trigger an already configured measurement, or remotely reset an unresponsive instrument. Options can be perused by looking at the help in SRmeasure -h

I've tested, debugged, and used them for a bit, but wrinkles may remain. They've been svn40m committed, and I also set up a separate git repository for them at github.com/e-q/netgpibdata

11016   Thu Feb 12 19:18:49 2015 JenneUpdateLSCNew Locking Paradigm - Loop-gebra

EDIT, JCD, 17Feb2015:  Updated loop diagram and calculation: http://131.215.115.52:8080/40m/11043

Okay, Koji and I talked (after he talked to Rana), and I re-looked at the original cartoon from when Rana and I were thinking about this the other day.

The original idea was to be able to actuate on the MC frequency (using REFL as the sensor), without affecting the ALS loop.  Since actuating on the MC will move the PSL frequency around, we need to tell the ALS error signal how much the PSL moved in order to subtract away this effect. (In reality, it doesn't matter if we're actuating on the MC or the ETMs, but it's easier for me to think about this way around).  This means that we want to be able to actuate from point 10 in the diagram, and not feel anything at point 4 in the diagram (diagram from http://131.215.115.52:8080/40m/11011)

This is the same as saying that we wanted the green trace in http://131.215.115.52:8080/40m/11009 to be zero.

So.  What is the total TF from 10 to 4?

${\rm TF}_{\rm (10 \ to \ 4)} = \frac{D_{\rm cpl} + {\color{DarkRed} A_{\rm refl}} {\color{DarkGreen} P_{\rm als}}}{1-{\color{DarkRed} A_{\rm refl} G_{\rm refl} S_{\rm refl} P_{\rm refl}} - {\color{DarkGreen} A_{\rm als} G_{\rm als} S_{\rm als}} ({\color{DarkGreen} P_{\rm als}} + D_{\rm cpl} {\color{DarkRed} G_{\rm refl} P_{\rm refl} S_{\rm refl}})}$

So, to set this equal to zero (ALS is immune to any REFL loop actuation), we need $D_{\rm cpl} = - {\color{DarkRed} A_{\rm refl}} {\color{DarkGreen} P_{\rm als}}$.

Next up, we want to see what this means for the closed loop gain of the whole system.  For simplicity, let's let $H_* = A_* G_* S_* P_*$, where * can be either REFL or ALS.

Recall that the closed loop gain of the system (from point 1 to point 2)  is

${\rm TF}_{\rm (1 \ to \ 2)} = \frac{1}{1-{\color{DarkRed} A_{\rm refl} G_{\rm refl} S_{\rm refl} P_{\rm refl}} - {\color{DarkGreen} A_{\rm als} G_{\rm als} S_{\rm als}} ({\color{DarkGreen} P_{\rm als}} + D_{\rm cpl} {\color{DarkRed} G_{\rm refl} P_{\rm refl} S_{\rm refl}})}$ , so if we let  $D_{\rm cpl} = - {\color{DarkRed} A_{\rm refl}} {\color{DarkGreen} P_{\rm als}}$ and simplify, we get

${\rm TF}_{\rm (1 \ to \ 2)} = \frac{1}{1-{\color{DarkRed} H_{\rm refl}} - {\color{DarkGreen} H_{\rm als}} + {\color{DarkRed} H_{\rm refl}}{\color{DarkGreen} H_{\rm als}}}$

This seems a little scary, in that maybe we have to be careful about keeping the system stable.  Hmmmm.  Note to self:  more brain energy here.

Also, this means that I cannot explain why the filter wasn't working last night, with the guess of a complex pole pair at 1Hz for the MC actuator.  The  ALS plant has a cavity pole at ~80kHz, so for our purposes is totally flat.  The only other thing that comes to mind is the delays that exist because the ALS signals have to hop from computer to computer.  But, as Rana points out, this isn't really all that much phase delay below 100Hz where we want the cancellation to be awesome.

I propose that we just measure and vectfit the transfer function that we need, since that seems less time consuming than iteratively tweaking and checking.

Also, I just now looked at the wiki, and the MC2 suspension resonance for pos is at 0.97Hz, although I don't suspect that that will have changed anything significantly above a few Hz.  Maybe it makes the cancellation right near 1Hz a little worse, but not well above the resonance.

11017   Thu Feb 12 22:28:16 2015 JenneUpdateLSCNew Locking Paradigm - LSC model changes, screens modified

I have modified the LSC trigger matrix screen, as well as the LSC overview screen, to reflect the modifications to the model from yesterday.

Also, I decided that we probably won't ever want to trigger the zero crossing on the Q phase signals of REFL.  Instead, we may want to try it out with the single arms, so the zero crossing selection matrix is now REFL11I, REFL55I, POX11I, POY11I, in that order.

11018   Thu Feb 12 23:40:12 2015 ranaUpdateIOOBounce / Roll stopband filters added to MC ASC filter banks

The filters were already in the damping loops but missing the MC WFS path. I checked that these accurately cover the peaks at 16.5 Hz and 23.90 and 24.06 Hz.

11019   Thu Feb 12 23:47:45 2015 KojiUpdateLSC3f modulation cancellation

- I built another beat setup on the PSL table at the South East side of the table.
- The main beam is not touched, no RF signal is touched, but recognize that I was present at the PSL table.
- The beat note is found. The 3rd order sideband was not seen so far.
- A PLL will be built tomorrow. The amplifier box Manasa made will be inspected tomorrow.

- One of the two beams from the picked-off beam from the main beam line was introduced to the beat setup.
(The other beam is used of for the beam pointing monitors)
There is another laser at that corner and the output from this beam is introduced into the beat setup.
The combined beam is introduced to PDA10CF (~150MHz BW).

- The matching of the beam there is poor. But without much effort I found the beat note.
The PSL laser had 31.33 deg Xtal temp. When the beat was found, the aux laser had the Xtal temp of 40.88.

- I could observe the sidebands easily, with a narrower BW of the RF analizer I could see the sidebands up to the 2nd order.
The 3rd order was not seen at all.

- The beat note had the amplitude of about -30dBm. One possibility is to amplify the signal. I wanted to use a spare channel
of the ALS/FOLL amplifier box. But it gave me rather attenuation than any amplification.
I'll look at the box tomorrow.

- Also the matching of two beams are not great. The PD also has clipping I guess. These will also be improved tomorrow

- Then the beat note will be locked at a certain frequency using PLL so that we can reduce the measurement BW more.

11020   Fri Feb 13 03:28:34 2015 ranaUpdateLSCNew Locking Paradigm - Loop-gebra

## Not so fast!

In the drawing, the FF path should actually be summed in after the Phase Tracker (i.e. after S_ALS). This means that the slow response of the phase tracker needs to be taken into account in the FF cancellation filter. i.e. D = -A_REFL * P_ALS * S_ALS. Since the Phase Tracker is a 1/f loop with a 1 kHz UGF, at 100 Hz, we can only get a cancellation factor of ~10.

So, tonight we added a 666:55 boost filter into the phase tracker filter bank. I think this might even make the ALS locking loops less laggy. The boost is made to give us better tracking below ~200 Hz where we want better phase performance in the ALS and more cancellation of the ALS-Fool. If it seems to work out well we can keep it. If it makes ALS more buggy, we can just shut it off.

Its time to take this loop cartoon into OmniGraffle.

11021   Fri Feb 13 03:44:56 2015 JenneUpdateLSCHeld using ALS for a while at "0" CARM offset with PRMI

[Jenne, Rana]

We wanted to jump right in and see if we were ready to try the new "ALS fool" loop decoupling scheme, so we spent some time with CARM and DARM at "0" offset, held on ALS, with PRMI locked on REFL33I&Q (no offsets).  Spoiler alert:  we weren't ready for the jump.

The REFL11 and AS55 PDs had 0dB analog whitening, which means that we weren't well-matching our noise levels between the PD noise and the ADC noise.  The photodiodes have something of the order nanovolt level noise, while the ADC has something of the order microvolt level noise.  So, we expect to need an analog gain of 1000 somewhere, to make these match up.  Anyhow, we have set both REFL11 and AS55 to 18dB gain.

On a related note, it seems not so great for the POX and POY ADC channels to be constantly saturated when we have some recycling gain, so we turned their analog gains down from 45dB to 0dB.  After we finished with full IFO locking, they were returned to their nominal 45dB levels.

We also checked the REFL33 demod phase at a variety of CARM offsets, and we see that perhaps it changes by one or two degrees for optimal rotation, but it's not changing drastically.  So, we can set the REFL33 demod phase at large CARM offset, and trust it at small CARM offset.

We then had a look at the transmon QPD inputs (before the dewhitening) for each quadrant.  They are super-duper saturating, which is not so excellent.
QPDsaturation_12Feb2015.pdf

We think that we want to undo the permanently-on whitening situation.  We want to make the second stage of whitening back to being switchable.  This means taking out the little u-shaped wires that are pulling the logic input of the switches to ground.  We think that we should be okay with one always on, and one switchable.  After the modification, we must check to make sure that the switching behaves as expected.  Also, I need to figure out what the current situation is for the end QPDs, and make sure that the DCC document tree matches reality.  In particular, the Yend DCC leaf doesn't include the gain changes, and the Xend leaf which does show those changes has the wrong value for the gain resistor.

After this, we started re-looking at the single arm cancellation, as Rana elogged about separately.

ALSfool_12Feb2015.pdf

11022   Fri Feb 13 14:27:30 2015 ericqUpdateElectronicsSecond QPD Whitening Switch enabled

I have re-enabled the second whitening stage switching on each quadrant of each end's QPD whitening board, to try and avoid saturations at full power. Looking at the spectra while single arm locked, I confirmed that the FM2 whitening switch works as expected. FM1 should be left on, as it is still hard-wired to whiten.

The oscillations in the Y QPD still exist. Jenne is updating the schematics on the DCC.

11023   Fri Feb 13 14:59:13 2015 ericqUpdateElectronicsSecond QPD Whitening Switch enabled

Went to zero CARM offset on ALS; transmission QPDs are still saturating :(

Maybe we need to switch off all whitening.

11024   Fri Feb 13 17:07:51 2015 JenneUpdateElectronicsSecond QPD Whitening Switch enabled

I first updated the DCC branches for the Xend and Yend to reflect the as-built situation from December 2014, and then I updated the drawings after Q's modifications today.

11025   Fri Feb 13 18:56:44 2015 ranaUpdateElectronicsSecond QPD Whitening Switch enabled

Depends on the plots of the whitening I guess; if its low freq sat, then we lower the light level with ND filters. If its happening above 10 Hz, then we switch off the whitening.

 Quote: Went to zero CARM offset on ALS; transmission QPDs are still saturating :( Maybe we need to switch off all whitening.

11026   Fri Feb 13 19:41:13 2015 ericqUpdateGeneralRF amplifier for ALS

As Koji found one of the spare channels of the ALS/FOL RF amplifier box nonfunctional yesterday, I pulled it out to fix it. I found that one of the sma cables did not conduct.

It was replaced with a new cable from Koji. Also, I rearranged the ports to be consistent across the box, and re-labeled with the gains I observed.

It has been reinstalled, and the Y frequency counter that is using one of the channels shows a steady beat freq.

I cannot test the amplitude of the green X beat at this time, as Koji is on the PSL table with the PSL shutter closed, and is using the control room spectrum analyzer. However, the dataviewer trace for the fine_phase_out_Hz looks like free swinging cavity motion, so its probably ok.

11027   Sat Feb 14 00:42:02 2015 KojiUpdateGeneralRF amplifier for ALS

The RF analyzer was returned to the control room. There are two beat notes from X/Y confirmed.

I locked the arms and aligned them with ASS.

When the end greens are locked at TEM00, X/Y beat amplitudes were ~33dBm and ~17dBm. respectively.
I don't judge if they are OK or not, as I don't recall the nominal values.

11028   Sat Feb 14 00:48:13 2015 KojiUpdateLSC3f modulation cancellation

[SUCCESS] The 3f sideband cancellation seemed worked nicely.

- Beat effeciency improved: ~30% contrast (no need for amplification)

- PLL locked

- 3f modulation sideband was seen

- The attenuation of the 55MHz modulation and the delay time between the modulation source was adjusted to
have maximum reduction of the 3f sidebands as much as allowed in the setup. This adjustment has been done
at the frequency generation box at 1X2 rack.

- The measurement and receipe for the sideband cancellation come later.

- This means that I jiggled the modulation setup at 1X2 rack. Now the modulation setup was reverted to the original,
but just be careful to any change of the sensing behavior.

- The RF analyzer was returned to the control room.

- The HEPA speed was reduced from 100% (during the action on the table) to 40%.

11030   Sat Feb 14 20:20:24 2015 JenneUpdateLSCALS fool cartoon

The ALS fool scheme is now diagrammed up in OmniGraffle, including its new official icon.  The mathematica notebook has not yet been updated.

EDIT, JCD, 17Feb2015:  Updated cartoon and calculation: http://131.215.115.52:8080/40m/11043

11037   Mon Feb 16 02:49:57 2015 JenneUpdateLSCALS fool measured decoupling TF

I have measured very, very carefully the transfer function from pushing on MC2 to the Yarm ALS beatnote.  In the newest loop diagram in http://nodus.ligo.caltech.edu:8080/40m/11030, this is pushing at point 10 and sensing at point 4.

Since it's a bunch of different transfer functions (to get the high coherence that we need for good cancellation to be possible), I attach my Matlab figure that includes only the useful data points.  I put a coherence cutoff of 0.99, so that (assuming the fit were perfect, which it won't be), we would be able to get a maximum cancellation of a factor of 100.

This plot also includes the vectfit to the data, which you can see is pretty good, although I need to separately plot the residuals (since the magnitude data is so small, the residuals for the mag don't show up in the auto plot that vectfit gives).

If you recall from http://nodus.ligo.caltech.edu:8080/40m/11020, we are expecting this transfer function to consist of the suspension actuator (pendulum with complex pole pair around 1Hz), the ALS plant (single pole at 80kHz) and the ALS sensor shape (the phase tracker is an integrator, with a boost consisting of a zero at 666Hz and a pole at 55Hz).  That expected transfer function does not multiply up to give me this wonky shape.  Brain power is needed here.

Just in case you were wondering if this depends on the actuator used (ETM vs MC2), or IFO configuration (single arm vs. PRFPMI), it doesn't.  The only discrepancy between these transfer functions is the expected sign flip between the MC2 and ETMY actuators.  So, they're all pretty consistent.

Before locking the PRFPMI, I copied the boost filter (666:55) from the YARM ALS over to Xarm ALS, so now both arms have the same boost.

YARM_actTF_compareActuators.pdf

Things to do for ALSfool:

• Put fitted TF into the MC_CTRL_FF filter bank, and try to measure the expected cancellation, a la http://nodus.ligo.caltech.edu:8080/40m/11009
• Quick test with single arm, ALS locked using full loop (high gain, all boosts), since the previous versions were with ALS very loosely locked.
• Does this measured transfer function actually give us good cancellation?
• Think.  Why should the transfer function look like this??
• Try it on the full PRFPMI
11038   Mon Feb 16 03:10:42 2015 KojiUpdateLSCALS fool measured decoupling TF

Wonkey shape: Looks like a loop supression. Your http://nodus.ligo.caltech.edu:8080/40m/11016 also suggests it too, doesn't it?

11039   Mon Feb 16 15:08:26 2015 JenneUpdateLSCALS fool measured decoupling TF

Dang it, yes. You're right.  I should have caught that.

Since Dcpl and Srefl are both zero during the measurement (since it was an ALS lock), this is actually

$\frac{{\color{DarkRed} A_{\rm refl}} {\color{DarkGreen} P_{\rm als} S_{\rm als}}}{1 - {\color{DarkGreen} A_{\rm als} G_{\rm als} S_{\rm als} P_{\rm als}}}$

So, I need to remove the effect of the ALS closed loop, to get the actual quantity I was looking for.

11041   Tue Feb 17 00:24:47 2015 rana, jenneUpdateLSCALS Fool filter updated for more cancellation

## Today we measured the TFs again and then updated the filter in the POY -> ALS FF path so as to get 10x better cancellation.

The cancellation went from ~10 dB to ~30 dB. This seems good enough. The new filter 'Comp1' is just constructed by eye. We then had to tune the filter module gain to a few %. Seems good enough for now, but we should really try to understand what it is and why it is the way that it is. In the above plot, the ORANGE trace is the old cancellation and the GREEN one is the new one. The filter TF is attached below - its not special, we made it by presing buttons in FOTON until the TF matched the measured TF of ALSY/LSC-MC_CTRL_FF_OUT.

11042   Tue Feb 17 04:04:32 2015 JenneUpdateLSCALS fool math

I re-did the Mathematica notebook according to the most current diagram (note to daytime self: attach .nb file!!!), and found that the denominator has changed, such that plugging in the new D=-A_refl*P_als*S_als gives the same

full-system closed loop gain of    $\frac{1}{1-H_{\rm als} - H_{\rm refl} + H_{\rm als}H_{\rm refl}}$

where $H_{*} = A_* G_* S_* P_*$  is the open loop gain, and the * indicates either the REFL or ALS portions of the system.

I have also plotted some things with Matlab, although I'm a little confused, and my daytime self needs to spend some more time thinking about this.

In the actuators (both for REFL and ALS), I include a pendulum, the digital anti-imaging filters that let us go from the 16kHz model to the 64kHz IOP and the analog anti-imaging filters after the DAC.  Note to self:  still need to include violin filters here.

For the servo gains, I copy the filters that we are using from Foton, and give them the same overall gain multiplier that is in the filter bank.  For the ALS going through the CARM filter bank, this is FMs 1, 2, 3, 5, 6 with a gain of 15.  For the RF (actually, POY here) going through the MC filter bank, this is FMs 4, 5, 7 with a gain of 0.08.

For the plants of each system, since this is still single arm lock, I just include a cavity pole (80kHz for ALS, 18kHz for REFL).

In the sensors (both for REFL and ALS), I include the analog anti-aliasing as well as the digital anti-aliasing to allow us to go from the 64kHz IOP to the 16kHz front end system.  For the ALS I also include in the sensor the closed loop response of the phase tracker loop (H/(1-H), where H is the open loop gain of the phase tracker).  For both sensors, I also include a semi-arbitrary number to make the full single-loop open loop gain have a UGF of 200Hz.  In the ALS sensor, I also include a minus sign to make the full open loop gain have the correct phase.

Here I plot the open loop gains of the individual single loops, as well as the open loop gain of the full system (Hals + Hrefl - Hals*Hrefl).  I feel like I must be missing a minus sign in my ALS loop, but I don't know where, and my nighttime brain doesn't want to just throw in minus signs without knowing why.  That will affect how the final ALSfool (blue trace) looks, so maybe it's not really as crazy as it looks right now.

Also, I was trying to explain to myself why we are getting the shape that we are in our measurements of the cancellation (http://nodus.ligo.caltech.edu:8080/40m/11041).  But, I can't.  Below are the plots of the transfer functions from either point 9 or 10 (before or after the G_refl) to point 5, which is the ALS error point.  The measurement in elog 11041 should correspond to the blue trace here.  For these traces, the decoupling is set to just (-A_refl), although there aren't any noticeable changes in the shape if I just set D=0.  If we start with the assumption that D=0, the shape and magnitude are basically identical to this plot, and then as we make D=-A_refl P_als S_als, the transfer functions both go to zero.

So.  Why is it that with no decoupling, the transfer function from 10 to 5 is tiny?  Why do the shapes plotted below look nothing at all like the measured cancellation shape?  Daytime brain needs to think some more.

11043   Tue Feb 17 16:36:08 2015 JenneUpdateLSCALS fool math

Here is an updated cartoon, with the ALS sensor explicitly shown as the beatbox times the closed loop response of the phase tracker servo.

The most important transfer functions are written on the diagram.  Others can be extracted from the attached Mathematica file (which corresponds to this diagram).

11044   Tue Feb 17 16:44:04 2015 KojiUpdateLSCDelay line installed again

For tonight's experiment, I re-installed the delay line cable and changed the attenuation to 10dB for the 55MHz modulation.

I quickly locked the PLL and checked that the modulation is the ratio of the field strength between the worst (19ns) and best
case (28ns) is 31dB, that is ~35 times reduction.

11045   Tue Feb 17 19:49:51 2015 KojiUpdateLSCDelay line un-installed again

The modulation setting was reverted.
Demod phase for REFL11/33/55/165 and AS55 were reverted to the previous numbers too.

11046   Wed Feb 18 01:58:51 2015 ericqUpdateLSCALS Fool single arm performance

I'm playing around with the lastest ALS fool feedforward on the Yarm, and I like what I'm seeing.

First, I verified that I could reproduce the TF shapes in ELOG 11041, which I was able to do with a gain of +9.3 and FMs 5 and 6 in the FF module.

Then, I locked the arm on ALS with full bandwidth, and on POY with the settings currently used the MC module, and took their spectra as references. (I put an excitation on the arm at 443Hz to line them up to the same arbitrary units.)

Then, with ALS at its usual 100Hz UGF and boosts on, the Fool path on, and the MC FM set to trigger on/off at 0.8/0.5, I slowly brought ALS towards zero offset and saw it pop right into resonance.  I then manually triggered the PDH boosts.

Here are some spectra, showing that, with the Fool path on:

• POY unsurprisingly picks up the high frequency noise of the ALS. (Could be mitigated by judicious lowpassing?)
• The in-fool-loop POY noise is WAY more supressed at low frequencies, so the loops are definitely working together. RMS is about 2x smaller too.

Once the PDH loop is running, the ALS loop can be switched out at the CARM FM output without much of an effect beyond a small kick.

However, looking at the loop shapes, maybe I got lucky here. I took the usual injection TFs at the MC FM, the CARM FM, and at ETMY, to get the overall OLG; all of them have >0.9 coherence pretty much everywhere except the first two points.

As desired, the PDH loop looks pretty normal.

I have no intuition about how the fooled CARM loop should look, since this is even more complicated than a two-loop system.

I don't currently know what is causing the odd feature in the overall at ~50Hz, and it spooked me out when I saw the multiple UGF crossings. The only thing I could think of happening there is the pole in the ALS phase tracker boost. I turned it off, and remeasured; the feature persists...

To wrap it up, here's something I think is pretty cool. Here's what happens when I give ETMY a 1000 count position step impulse (no ramp). [Here, CARM is on ALS with G=12, but only FM5 on]

Although the arm was controlled with IR before the kick, ALS maintained control. As soon as ALS brought the arm back towards resonance, the PDH loop picked it right back up.

Coooool.

Some random notes:

• We should really DQ the output of the feedforward FM. I'll try to remember to do this tonight
• I was having problems with the zero crossing triggering, so I didn't end up using it, desipte trying to. Maybe we should implement the Schmitt-y style of "Am I below the threshold? Wait. Am I closer to zero now? Ok, Go!"
• The 1Hz pole in the FF FM rings, unsurprisingly, when the MC FM triggers on briefly, which can be a pain. I made a FM4 which is a Q=1 (instead of 7) pole pair for less ringy. This probably hurts the cancellation somewhat, but I was impatient.
• Alternatively, we could try to figure out how to force history clear when the MC filter is triggered off.

DTT data is attached, in case it's useful to anyone!

11047   Wed Feb 18 17:51:40 2015 ericqUpdateLSCALS Fool impulse response

Koji raised a good question about the step response I wrote about. Namely, if the UGF of the ALS servo is around 100Hz, we would expect it to settle with a characteristic time on the order of tens of milliseconds, not seconds, as was seen in the plot I posted.

I claim that the reason for the slow response was the fact that the feedforward FM stayed on after the kick, despite the MC filter bank being triggered off. Since it has filters that look like a oscillator at 1Hz, the ringdown or exponential decay of this filter's output in response to the large impulsive output of the PDH control signal just before being triggered down would slowly push the ALS error signal around through the feedforward path.

Given this reasoning, this should be helped by adding output triggering to the FF filter that uses the MC trigger matrix row, as I wanted to do anyways. I have now added this into the LSC model (as well as DQ at 2kHz for the MC_CTRL_FF_OUT), and the impulse response is indeed much quicker.

In the following plot, I hit ETMY with a five sample, 5000 amplitude, impulse (rather than a step, as I did yesterday). The system comes back to PDH lock after ~40ms.

11048   Wed Feb 18 19:06:40 2015 KojiUpdateLSCALS Fool impulse response

11049   Thu Feb 19 04:09:21 2015 ericqUpdateElectronicsSecond QPD Whitening Switch enabled

X end QPD has recieved 0.2+0.4 absorptive ND filters. Y end QPD got one at 0.6. This appears to have mitigated the saturations for now; the unwhitened signals no longer go negative. The digital gains have been reset.

11050   Thu Feb 19 04:16:45 2015 JenneUpdateLSCALS fool attempt with PRFPMI

[Jenne, EricQ]

We tried several times tonight to engage the Fool path with the PRFPMI.  No success.

First, we locked the arms on ALS, in CARM/DARM mode, and measured the cancellation ability, to make sure that the filter shape and gain was set correctly.  For the PRFPMI, it was okay using the same shape as the single arm case, but the gain was +20.0.  There might be a bit more cancellation to be gained if we adjust the shape at the ~1dB level, but we're already able to get 20dB of cancellation, so we decided that would be enough to give things a try.  To get this much cancellation, we set the phase tracker loops to both have 2kHz UGFs, almost exactly.  We should implement a UGF servo, or the amplitude method version of that as Koji suggested ages ago, so that the phase tracker is always at the same place.

I don't think that REFL 11 is seeing as much CARM as I expect.  We ended up switching over to linearized REFL55 for our attempts.  When we're close to zero CARM offset, the arms are constantly flashing through resonance, and we get the loud buzzing.  REFL11 doesn't seem to see any of this, even though we should be close enough to see some PDH action.  REFL55 does change as we get closer to resonance, so I think it's seeing some real CARM stuff.

We tried engaging the Fool, but I don't think it did anything too useful. We need to make an estimate of what we expect our gain of the REFL loop to be - or at least the sign.

The PRMI is still not stable enough.  It keeps falling out of lock when we get to high-ish arm powers.  Not good.  More brain power tomorrow on the modulation cancellation issue.

Perhaps if things are stable at moderate arm powers, we can use an excitation to line up the ALS vs. REFL error signals, and then watch the noises of them change as we move around in CARM offset.  This should tell us when the linearized REFL signal is quiet enough that it's worth triggering and trying to transfer over.

The last lockloss tonight, there was something funny going on, that we can't explain.  Even though both arms were locked on the CARM/DARM combined ALS signals, beatx doesn't see the giant oscillation that causes carm to lose lock until much later.  Fool was trying to do something, but that should affect both als individal signals in the same way.  Mystery.

11051   Thu Feb 19 15:45:43 2015 ericqUpdateCDSBad CDS behavior

At about 10AM, the C1LSC frontend stopped reporting any EPICS information. The arms were locked at the time, and remained so for some hours, until I noticed the totally whited-out MEDM screens. The machine would respond to pings, but did not respond to ssh, so we had to manually reboot.

Soon thereafter, we had a global 15min EPICS freeze, and have been in a weird state ever since. Epics has come back (and frozen again), but the fast frontends are still wonky, even when EPICS is not frozen. Intermittantly, the status blinkers and GPS time EPICS values will freeze for multiple seconds at a time, sporadically updating. Looking at a StripTool trace of an IOPs GPS time value shows a line with smooth portions for about 30 seconds, about 2 minutes apart. Between this is totally jagged step function behavior. C1LSC needed to be power cycled again; trying to restart the models is tough, because the EPICS slowdown makes it hard to hit the BURT button, as is needed for the model to start without crashing.

The DAQ network switch, and martian switch inside were power cycled, to little effect. I'm not sure how to diagnose network issues with the frontends. Using iperf, I am able to show hundreds of Mbit/s bandwidth betweem the control room machines and the frontends, but their EPICS is still totally wonky.

What can we do???

ELOG V3.1.3-