I looked at 2 of the locklosses from last night, (1:19am and 1:27am), and saw that for both, the DARM loop started to oscillate just before we lost lock.  In the trials tonight, we were more watchful of gain peaking.

Here is the 1:19am lockloss

And here is the 1:27am lockloss

 So you can see what we were doing, and what the effect was, here is a few minutes of data just before the 1:27am lockloss. The times I note below are rough, but should give you an idea of what happened at which time.

0 sec:  Arms are held on resonance with ALS.

10 sec:  CARM offset of 3nm added.

20 sec:  PRM restored, one flash, then PRMI acquires lock.

30 sec:  CARM offset reduced to 2nm, transmitted powers are about 0.1

50 sec:  Transition CARM to 1/sqrt(trans) signals.  Note that we are using the high gain Thorlabs PD here, so the noise is better than last Thursday.

60-110 sec:  CARM offset reduction to about 1nm.

110 sec:  CARM's LSC low frequency boost engaged.

120 sec:  DARM transitioned to AS55Q.

170 sec:  DARM's LSC low frequency boost engaged.

Last night, EricQ and I were concerned that we might need some CARM UGF servoing, so I added a UGF servo block, copied from the aLIGO LSC model, to our LSC model.  The block is inline with the CARM servo, after the output triggering, just before the output matrix.  Q put together some screens, which are accessible from the main LSC screen. 

The model is compiled and running.  We didn't get very far in testing it though before Koji pointed out that it is a slow solution, and not a fast one like we were searching for.  We were hoping to deal with the momentary power buildup, and thus optical gain change, as the arms flash close to resonance.  The UGF servo will not work nearly that fast though.  We may want it for slow UGF servo-ing, but it's not the solution to what Q and I were thinking about yesterday.  Regular ol' dynamic normalization is closer to the right answer for this.

In tonight's activities, Koji and I found that we probably want a CESAR block for DARM as well as CARM, so that we can independently normalize AS55Q. 

To solve the DARM oscillation issue from last night (that I discovered this evening when I finally looked at the time series data), we may want to implement a DARM UGF servo.  For tonight, as we reduced the CARM offset and started seeing gain peaking in the DARM spectra, I hand-reduced the DARM gain.

[Jenne, Koji]

This evening, Koji and I followed the procedure from last night (elog 9817), with the exceptions that as we saw gain peaking in the DARM spectrum, we lowered the DARM servo gain.  We also left the DARM FM4 boost off, and added (TRX+TRY) power normalization to AS55 (although we still had to hand-reduce the gain).    These changes enabled us to reduce the CARM offset much further.  We were able to get the transmitted powers to hold steady at about 1 count while on the IR signals, which is a new record for us.  (We had in the past held the arms with ALS at several counts, but the power fluctuations were huge, and now they are nice and small).

After that, we started looking at whether we could transition CARM over to REFL11I.  We tried a few times, but never made it all the way.

Here are some times for data-foraging tomorrow:

8:27pm, nice transition, CARM offset reduction to 0.6 before lockloss.

9:19pm, turned on power normalization for AS55Q, then reduced CARM offset to 0.5

9:40pm, Lockloss after reducing CARM offset to -0.24, arm transmitted powers around 0.9.

gps 1081748419: First trial trying to transition CARM to REFL11I normalized by (TRX+TRY).

gps 1081749965:  Tried to transition CARM to (REFL11 + REFL33)/(TRX+TRY).  Got about 1/3 of the way through the transition (in terms of matrix element value steps) before lockloss.

11:56pm, Tried to add in REFL11I to CARM error signal (without reducing 1/sqrt(trans) matrix elements).  We increased the REFL11 matrix element until we saw gain peaking, and then tried reducing the 1/sqrt(trans) contribution, and lost lock.  We were only at an offset of 0.3, so we probably weren't close enough to the resonance yet.  We were able to add in REFL11 information, but this was probably not too hard, since there wasn't much actual information in it.

Thoughts:

* It's a little weird that once we are on IR signals, the 0 CARM offset point that we find with ALS is not the true CARM offset point.  Although, this may be because we're just going to an averaged no CARM offset place with ALS, but since ALS is noisy, we won't ever really be holding on the zero offset point.  Anyhow, when we were using the 1/sqrt(trans) signals for CARM, and the CARM digital offset was -0.24, the ALSX and ALSY outputs were both about 0.5 in magnitude.

* We're getting there! 

I've been trying to get c1ioo on the Dolphin network, but have not yet been successful.

Background: if we can put the c1ioo machine on the fast Dolphin IPC network, we can essentially eliminate latencies between the c1als model and the c1lsc model, which are currently connected via a rube goldberg-esq c1lsc->dolphin->c1sus->rfm->c1ioo configuration.

Rolf gave us a Dolpin host adapter card, and we purchased a Dolphin fiber cable to run from the 1X2 rack to the 1X4 rack where the Dolphin switch is.

Yesterday I installed the dolphin card into c1ioo.  Unfortunately, c1ioo, which is Sun Fire X4600, and therefore different than the rest of the front end machines, doesn't seem to be recognizing the card.  The /etc/dolphin_present.sh script, which is supposed to detect the presence of the card by grep'ing for the string 'Stargen' in the lspci output, returns null.

I've tried moving the card to different PCIe slots, as well as swapping it out with another Dolphin host adapter that we have.  Neither worked.

I looked at the Dolphin host adapter installed in c1lsc and it's quite different, presumably a newer or older model.  Not sure if that has anything to do with anything.

I'm contacting Rolf to see if he has any other ideas.

 I have made the same modification to the Yarm trans PD whitening board as was done for the xend, to increase our SNR.  I put in a 2.1kOhm thin film resistor in the Rgain place.

When I was pulling the board, the ribbon cable that goes to the ADC had its connector break.  I redid the ribbon connector before putting the board back. 

I see signals coming into the digital system for both the high gain and low gain Y transmission PDs, so I think we're back.  I will re-do the normalization after Jamie is finished working on the computers for the day.

So it turns out that the card that Rolf had given me was not a Dolphin host adapter after all.  He did have an actual host adapter board on hand, though, and kindly let us take it.  And this one works!

I installed the new board in c1ioo, and it recognized it.  Upon boot, the dolphin configuration scripts managed to automatically recognize the card, load the necessary kernel modules, and configure it.  I'll describe below how I got everything working.

However, at some point mx_stream stopped working on c1ioo.  I have no idea why, and it shouldn't be related to any of this dolphin stuff at all.  But given that mx_stream stopped working at the same time the dolphin stuff started working, I didn't take any chances and completely backed out all the dolphin stuff on c1ioo, including removing the dolphin host adapter from the chassis all together.  Unfortunately that didn't fix any of the mx_stream issues, so mx_stream continues to not work on c1ioo.  I'll follow up in a separate post about that.  In the meantime, here's what I did to get dolphin working on c1ioo:

c1ioo Dolphin configuration

To get the new host recognized on the Dolphin network, I had to make a couple of changes to the dolphin manager setup on fb.  I referenced the following page:

https://cdswiki.ligo-la.caltech.edu/foswiki/bin/view/CDS/DolphinHowTo

Below are the two patches I made to the dolphin ("dis") config files on fb:

--- /etc/dis/dishosts.conf.bak    2014-04-17 09:31:08.000000000 -0700
+++ /etc/dis/dishosts.conf    2014-04-17 09:28:27.000000000 -0700
@@ -26,6 +26,8 @@
 ADAPTER:  c1sus_a0 8 0 4
 HOSTNAME: c1lsc
 ADAPTER:  c1lsc_a0 12 0 4
+HOSTNAME: c1ioo
+ADAPTER:  c1ioo_a0 16 0 4
 
 # Here we define a socket adapter in single mode.
 #SOCKETADAPTER: sockad_0 SINGLE 0

--- /etc/dis/networkmanager.conf.bak    2014-04-17 09:30:40.000000000 -0700
+++ /etc/dis/networkmanager.conf    2014-04-17 09:30:48.000000000 -0700
@@ -39,7 +39,7 @@
 # Number of nodes in X Dimension. If you are using a single ring, please 
 # specify number of nodes in ring.
 
--dimensionX 2;
+-dimensionX 3;
 
 # Number of nodes in Y Dimension. 

I then had to restart the DIS network manager to see these changes take affect:

$ sudo /etc/init.d/dis_networkmgr restart

I then rebooted c1ioo one more time, after which c1ioo showed up in the dxadmin GUI.

At this point I tried adding a dolphin IPC connection between c1als and c1lsc to see if it worked.  Unfortunately everything crashed every time I tried to run the models (including models on other machines!).  The problem was that I had forgotten to tell the c1ioo IOP (c1x03) to use PCIe RFM (i.e. Dolphin).  This is done by adding the following flag to the cdsParamters block in the IOP:

pciRfm=1

Once this was added, and the IOP was rebuilt/installed/restarted and came back up fine.  The c1als model with the dolphin output also came up fine.

However, at this point I ran into the c1ioo mx_stream problem and started backing everything out.

While trying to get dolphin working on c1ioo, the c1ioo mx_stream processes mysteriously stopped working.  The mx_stream process itself just won't start now.  I have no idea why, or what could have happened to cause this change.  I was working on PCIe dolphin stuff, but have since backed out everything that I had done, and still the c1ioo mx_stream process will not start.

mx_stream relies on the open-mx kernel module, but that appears to be fine:

controls@c1ioo ~ 0$ /opt/open-mx/bin/omx_info  
Open-MX version 1.3.901
 build: root@fb:/root/open-mx-1.3.901 Wed Feb 23 11:13:17 PST 2011

Found 1 boards (32 max) supporting 32 endpoints each:
 c1ioo:0 (board #0 name eth1 addr 00:14:4f:40:64:25)
   managed by driver 'e1000'
   attached to numa node 0

Peer table is ready, mapper is 00:30:48:d6:11:17
================================================
  0) 00:14:4f:40:64:25 c1ioo:0
  1) 00:30:48:d6:11:17 c1iscey:0
  2) 00:25:90:0d:75:bb c1sus:0
  3) 00:30:48:be:11:5d c1iscex:0
  4) 00:30:48:bf:69:4f c1lsc:0
controls@c1ioo ~ 0$ 

However, if trying to start mx_stream now fails:

controls@c1ioo ~ 0$ /opt/rtcds/caltech/c1/target/fb/mx_stream -s c1x03 c1ioo c1als -d fb:0
c1x03
mmapped address is 0x7f885f576000
mapped at 0x7f885f576000
send len = 263596
OMX: Failed to find peer index of board 00:00:00:00:00:00 (Peer Not Found in the Table)
mx_connect failed
controls@c1ioo ~ 1$ 

I'm not quite sure how to interpret this error message.  The "00:00:00:00:00:00" has the form of a 48-bit MAC address that would be used for a hardware identifier, ala the second column of the OMC "peer table" above, although of course all zeros is not an actual address.  So there's some disconnect between mx_stream and the actually omx configuration stuff that's running underneath.

Again, I have no idea what happened.  I spoke to Rolf and he's going to try to help sort this out tomorrow.

Jamie tells me that the 2 big consequences of this are (a) we are not archiving any data that is collected on the ioo machine, and (b) that we will not have access to test points on the IOO or ALS models.

To make sure that this is not a show-stopper for locking, I have locked the arms using ALS.  The signals seem to still be getting from the ALS model to the LSC model, and I'm able to acquire ALS lock, so we should be able to work tonight.  All of the data that I have been looking at lately has been coming off of the LSC machine, so we should even be okay in terms of look-back for lockloss studies, etc.

Jenne made some suggestions for some plots that would be useful on our CARM offset reduction adventures, so I made some with my MIST model. 

First, here's a plot showing the transfer function of CARM to TRX, with logarithmically spaced offsets out to 3nm. While not a control signal, it shows us where the optical plant resonance stuff is happening. The peaks in this TF correspond to peaks in REFL11, REFL55, AS11, etc., as in the close-to-resonance TFs in ELOG 9785. 

[more to come, had a MATLAB issue]

10 days of on-off glitching?   It is a kick. The LSC is off, so it must be the ALS

Some time series data from Wednesday night. 

Here is the first time we've gotten the arm transmissions to about 1 count, while holding CARM and DARM on IR signals, so the transmission, as well as POPDC, were relatively quiet.

As we were attempting to transition CARM to REFL11I, at least 2 of the times, we were hitting a CARM oscillation:

 To fix open-mx connection to c1ioo, had to restart the mx mapper on fb machine. Command is /opt/mx/sbin/mx_start_mapper, to be run as root. Once this was done, omx_info on c1ioo computer showed fb:0 in the table and mx_stream started back up on its own. 

Thanks so much Rolf (and Keith)!

Last night, as well as tonight, the ALS seems not quite as robust as it was earlier in the week.

I have just taken noise spectra, and ALS is definitely more noisy than usual. 

These plots are with the arms held in CARM and DARM mode, with servo gains of 8. I was seeing the beginnings of gain peaking at a gain of 10, so I turned it back to 8.  Our ALS in-loop RMS is usually something like a few hundred Hz, but I'm seeing over 1kHz, so I have a factor of 4 or 5 too much noise.  Why?!?!?

Came in, and the PMC and MC needed aligning. 

PMC Trans .74->.83

I wasn't able to move the MC1&3 the right way in pitch without having a terrible MC2 position... using the move MC2 scripts to bring it back threw off the other spots.

 The  Maglev controller was serviced at Osaka Vacuum. It was swapped in this morning.

I have noticed that ALS noise has been at 1KHz rms since LSC arm lock servos have been used to lock arms using ALS error signals. May be this has not been given much attention.

But looking more closely at the ALS noise (better dtt resolution for noise power spectrum) , there seems to be too much noise suppression at <1Hz and not much happening at around 10Hz.

Attachment 1 (data files at /users/manasa/data/140421/)

So I made a bunch of transfer function measurements for ALS and phase tracker servo. Koji will be using these and redesigning the servo filters so that we can get more suppression at 10Hz.

Other than this I also found that the Y arm showed more high frequency noise as compared to the X arm. (Edit by manasa: Thinking back now, this could be related to the onset of 60Hz noise at the Y end elog 9838. But still has to be looked at after fixing TRY)

Attachment 2

Tip: Once the arms are ALS locked, enabling the SLOW_SERVO helps hold the lock stably. 

P.S. I realigned the Y green to the arm and brought GTRY to 0.93

To do:

Find out what makes Y arm in-loop noise at high frequency higher than X arm.

This evening, I was not able to successfully transition CARM from ALS to 1/sqrt(trans) signals.  The TRY time series looked odd, so I took a spectra, and we have huge 60Hz noise in TRY. 

I found a lock stretch from around 6:30pm that did not show the 60Hz noise, and then there was a lock stretch around 8pm that did have the noise.  So, something happened at the Yend between 6:30 and 8pm tonight.

Asking around, this was the time frame in which Manasa was down at the Yend to realign the green beam, and to check cabling for the PZT_OUT and ERR_MON signals to the ADC.

Looking at the spectra, Rana noted that we have even as well as odd harmonics of the 60Hz line, which is unusual.

To try to diagnose the problem, Rana and I tried to make sure no cables' connectors were touching, and that no equipment was plugged in that shouldn't be.  We noticed that none of: the shutter, the Thorlabs TRY PD, or the QPD TRY are isolated from the table.  To see if perhaps the shutter was the problem, I turned off the power to the Yend green shutter, and unplugged the cable.  The cable is laying on the table, with the connector sitting on a piece of plastic to isolate it.  Removing the shutter from the system did not change anything. 

We don't see the 60Hz noise in the Xarm, so it's not on the laser light itself.  Also, we don't see the 60Hz lines in the Yarm feedback signal, so we're not putting the lines onto the mirror, and thus onto the Yarm's light. 

Manasa, can you please take a look, and see if you can figure out what is going on?  We need TRY so that we can transition to 1/sqrt(trans) signals for CARM.  Thanks!!

[Jenne, Q]

The frame builder (or something) is unhappy again.  I know that we've seen this before, but I can't find the elog entry that relates to this particular problem.

Every few minutes, the fb status lights on the CDS_STATUS screen go white, and then come back green.  It's annoying when it happens every hour or so (which is unfortunately typical), but it's pretty debilitating when it stops dataviewer and dtt every few minutes.  Just from the way the lights change, it looks like perhaps the daqd process is restarting itself periodically? 

In an effort to familiarize myself with the analog CM servo, I've begun to replicate Koji and Den's work from the ELOG post that this is a reply to.

I hooked POX11Q into the IN1 of the CM board. (POX is rotated by ~86 degrees in the CDS, meaning analog Q is almost perfect.)

While there, I took out the too-long delay cables Jenne introduced for REFLs 11 and 55. (Also note: when we do cable-based analog phasing in the future, we should do it on the LO side, instead of the PD input side.) I also heard a dangerously crinkly sound from the short SMA cable for REFL11, so I replaced it with a beefy looking new one I found on the SP table.

I messed with the gain and offset in the CM_SLOW input filter to get it to look just like POX11_I_ERR, and was able to lock the arm on it without an issue. I then put the SR560 between the CM and MC (30k pole, but also AC coupled, because I figure the digital loop should be doing the work down there, and don't want to kick the AO with an offset), and was able to turn on the AO path with a gain of 8dB on the CM board and 10dB on the MC board, as detailed in Koji's procedures.

I wasn't able to increase the AO gain to 9dB without breaking lock, but maybe this is ok, because by judging by the LSC filter gains, POX11 might be about 3 times bigger than POY, so maybe 8dB AO gain on POX ~ =18dB AO gain on POY? I was able to put the CM servo offset at 0, but turning on boosts promptly kicked the MC out of lock.

I'm stopping for the night; but tomorrow I'll bust out a spectrum analyzer to see if I actually have won some bandwidth with the CM servo, and check out the situation with the offsets and boosts.

 I went to the Y end to look at the TRY 60Hz noise situation this morning. While looking at TRY noise on dtt, I found that just lifting the cable away from the cable bunch that runs out of the table suppressed the noise drastically. 

Attachment 1

I removed the unwanted bnc connector in the path of the already long TRY cable running from the PD to the 1Y4 rack and isolated it from the bunch. TRY became less noisy.

But the noise was back again earlier in the evening and it looks like the noise is very much related to the TRY cable. TRY cable might have moved from its sweet spot while I was around checking cable connections yesterday.

I couldn't find a spare to replace it right away today (We need a BNC to 4 pin lemo).

 The detectors and electronics on this table are not properly isolated. To reduce the 60 Hz and ground loops, photodiodes and shutter must be isolated by using plastic spacers as we usually do elsewhere - this table just seems to have a few oversights.

Steve can start assembling all of the pieces to do this in the morning and then we can start the swapping after the meeting.

The high gain Transmon cable should be a regular BNC. There's no need for 4-pin LEMO in this usage, so the best move is to modify the board and replace the 4-pin LEMO connector with an isolated panel mount BNC female.

The AC adapter for this diode (and all of the detectors on the table) should get their power from a power strip which gets plugged into the rack with the whitening boards. The SHG oven, the Uniblitz shutter, and any cameras can get their power from another power strip if needed/wanted.

[Steve, Manasa]

To find noise source

1. Swapped the power cable of the PD and checked that it is connected to the right power source.

2. Changed the aluminium base of the post holding the diode so that the diode is floating

3. Grounded the table and the rack

4. Routed the cable on the other side of the beam tube to isolate it from other cables.

After all the above, we still found that shaking the cable was making TRY noisy.

I pulled out the PD whitening board to replace the 4 pin lemo connector with a bnc connector so that we can swap the cable with a new one. So there is no TRY right now.

The MON outputs of the Y end QPD whitening board were hot earlier today while pulling it out of the crate. After swapping the 4 pin lemo connector with an isolated panel mount bnc connector, I stuck the board back into the crate and this immediately kicked the ETMY suspension. Jenne and I went to the Y end to look at what was going on. We removed the board from the crate after smelling something burning. The MON output ports of the whitening board were super hot this time. There is no sign of any components melting on the board (comparing the board with its pictures that were taken earlier) and a tester board stuck into the crate lights up just fine.

So the back panel is still ok. We need to troubleshoot or replace the whitening board.

Edit, JCD:  The attached photos are from right after I replaced the "Rgain" resistor, elog 9823.  What they show is that it looks like some of the melting / burning may have already been happening before I pulled the board, and I just never noticed :(  In particular, look at the resistors on the main board above the blue "G" sticker.  There isn't a difference that I can tell between this photo from last week, and today's situation. 

 I re-plugged in the Yend shutter, and turned it on.

I tried some locking anyway tonight, even though we don't have TRY. 

The biggest conclusion is that I miss the auto-resonance-finding.  I've been roughly scanning the Y-ALS offset to find the POY zero crossing when I see the resonance on the test mass face cameras. 

The next-biggest conclusion, is that I can hold the PRFPMI close to resonance, using ALS for CARM and DARM.  I was trying to transition DARM to AS55, but I couldn't get the last bit of the way.  That is, I couldn't turn off the ALS control.  So, I think that AS55 wasn't actually holding DARM, until maybe the last moment or so.

Anyhow, here are some time series.  My average TRX value is around 40 counts, and POPDC is maybe 250 counts (just PRMI, POPDC is about 75 counts).  Obviously this is noisy as hell, but I'm not using any IR signals for the arms.  Near the end of this first time series, I am trying to switch to AS55 for DARM.

Zooming in, my real lockloss is due to PRCL oscillating at ~350 Hz:

However, I also saw ~25Hz peaks in CARM and DARM on the spectra starting to show up, and I see a ~25 Hz oscillation in DARM a few moments after the PRCL lockloss.  (Plot #2 is a zoom of the ~1.1 second mark on Plot #3.)

The locking parameters:

CARM:

Input:  Using the new CESAR matrix, -1*ALSX, +1*ALSY.  Beatnotes both move up in freq if temp sliders move up.

Servo: gain = 6, FMs 1, 2, 3, 5, 6, 7, 9 on.  Offset = 0 counts. 

Output = -1*MC2

DARM:

Input:  +1*ALSX, +1*ALSY

Servo:  gain = 4.  FMs 1, 2, 3, 5, 6, 7, 9 on.  Offset = 0 counts.

Output = -1*ETMX, +1*ETMY

PRCL:

Input:  +1*REFL33_I, Norm = +0.01*POPDC, sqrt engaged.

Servo:  acquisition easier with -0.04 or -0.06, less gain peaking at -0.02   FMs 4, 5 on; 2, 3, 6, 9 triggered with 0.5 sec delay.  Servo trigger = POPDC, up 100, down 10.  FM trigger = POPDC, up 300, down 20.

Output = +1*PRM

PRCL ASC off, PRM oplev on.

MICH:

Input:  +1*REFL33_Q, Norm = +0.01*POPDC, sqrt engaged.

Servo:  gain = 2, FMs 4, 5 on; 2, 3 triggered with 0.2 sec delay.  Servo trigger = POPDC, up 100, down 10.  FM trigger = POPDC, up 300, down 20.

Output = +0.5*BS, -0.2625*PRM

PDs:

REFL33 analog gain set to 30 dB for both I&Q.

AS55 set to 0 dB for both I&Q.  AS55 had DC normalization of 80 counts (which was the measured number for PRFPMI when TRX was about 0.1 count this evening)

This seems the ever best stability at the zero offset PRFPMI.

Can you look at REFLDC in this data stream too? How was it promising?

Here is 1 second of data, with REFLDC, POPDC and TRX:

Here is a zoom of the first 3 big peaks in TRX.  The weird jumps at the beginning of each TRX peak are due to the triggered switching between the Thorlabs trans PD and the QPD trans PD.  Clearly we need to work on their relative normalizations.  There are also little jumps after each peak as the triggering sends the signal back to the Thorlabs PD.

Here is a zoom of the single big peak about halfway through the 1 second of data:

And here is a zoom of the tail of that peak.  It looks to me like we want to start thinking about using REFL DC when our transmitted powers are around 2 counts.  We could do as soon as 1 count, but 2 is a little farther into the dip.

 maybe the tantalum caps on the daughter board power supply lines are blown? If so, replace with 35V+ ceramic.

I added ~1m of cable to the LO side of the REFL11 Demodulator, which brought its PRCL demod phase to about 8 degrees. According to my simulations, PRCL and CARM have the same angle (but opposite sign) at resonance. There seems to be a severe lack of SMA cables in the lab, so I didn't tune it to be any closer. Cos(8 degrees)=.99, so I think it should be fine to use it for the CARM servo, since none of the other signals are going to be nearly as big. I plugged analog REFL 11 I back into the CARM servo IN1. 

As for IN2, I threw together a temporary setup for using REFLDC as a complementary signal. I T'd off the REFLDC signal (which is the DC signal out of REFL55), and sent it into an SR560 to subtract an offset. The offset comes from a 1Hz-passive-pomona-box-low-passed C1:IOO-TT4_LR output, since there are 8 DAC channels set up for the nonexistent tip tilts 3 and 4 actively running. The output of the SR560 is sent to the CARM servo IN2. 

I adjusted the offset by turning on only IN2 in the CARM servo MEDM screen, and looking at the CM_SLOW signal in data viewer. I adjusted gains and such to get it to look just like REFLDC with the PRC locked. There was good coherence and no appreciable phase difference from DC out to some kHz, albeit a dip in coherence to about .8-.9 from ~40 to 300Hz, for some reason. (This included turning on the unWhite FM in the REFLDC filter bank)

If this signal turns out to be useful, it will be relatively straightforward to put together a little box that does the offset subtraction nicely, but this should do for our immediate needs.

Lastly, I hung up this plot in the control room to give us information about the DC values of different signals as the CARM offset changes. This is helpful to see what our CARM offset is, based on the transmission we se, when different signals start to have length dependence, where they start/stop being linear, etc. The TRX curve is scaled to a maximum of 600, REFLDC is normalized to input power = 1, and all the rest are arbitrarily scaled to fit on the plot. I've assumed 75ppm loss on all mirrors in my simulation (PRM, BS, 2xITM, 2xETM), mostly to get some realistic profile of REFLDC. 

The main problem was a panel fixing bolt that caused the short circuits between power supply layers.
This burned the PCB and secondarily caused permanent short circuit between +15V/-15V/+5V layers.

Diagnosis

- The resistances between +15V, +5V, and -15V were low. The resistance between +15V and -15V is 13 Ohm.
  The one between +5V and -15V is 7Ohm. And the one between +15 and +5 is 19Ohm. So the situation is

                o -15V
                |
+15V o-(13 Ohm)-+-(9 Ohm)-o +5V

Even after removing all of the active components from the board, they remained the same.

- The tantalum caps were removed from the board and it was confirmed that they are not the cause of the issue.

- The panel was removed from the module for the component migration to a spare board (to be described in the other entry).
I found that the screw hole and the screw have burnt marks. The screw need an insulation tube to avoid short circuit.
The other screw was also bare. The spare board has the screws with the insulation tubes.

[ericq, Jenne, Zach]

We spent some time tonight trying to push our CARM locking further, to little avail. DARM/CARM loop oscillations kept sneaking up on us. We measured some MC2 motion -> REFL11 Transfer Functions to see if we could see CARM plant features; plots will come in the near future...

I went to WB and found the last spare module of D990399 revB. We need to thank Frank for his foresight.

The original (=broken) board had various modifications from this revB.
I had to check the schemaric diagram and the difference between the boards and migrate some of the SMD components from left to right.

Here is the deciphered features of the QPD whitening board:
- The input stage is a VGA amp (AD602). It has the internal input impedance of 100 Ohm. The series resister
  of 909 Ohm gives us 1/10 voltage division! It is more tricky as the QPD (D990272) has the output impedances of 50Ohm
  (for the both side of the differential out) and on resistance of MAX333A. So it could have been deviated by ~10% from the nominal.

- Variable gain control: The input has 1/10 voltage division. The gain is fixed at the unity. In total the gain of the variable control stage is 1/10.
  This gives us the gain range of +42dB/-22dB for +10V/-10V. The actual range is limited to be -10~30dB.

- Whitening stages. Each channel has two sets of the whitening path and the bypass path.
  They could be switched by binary control inputs but I permanently enabled the whitening by pulling the MAX333 control inputs to the ground.
  The whitening zero and pole are at 4.02Hz and 40.6Hz.

  Each bypass path has an additional cap of 220pF in parallel to 35.7kOhm (R101 and R103 for CH1), resulting in the pole at 20.2kHz.
  Each whitening paths had a 5.6nF cap (C53 and C64). This cap was replaced with 350pF, resulting in the move of the pole freq from 800Hz to 12.7kHz.

- There are two anti-aliasing stages which were designed for 2kHz sampling rate. They are identical sallen key 2nd-order LPFs with fc=766Hz and Q=0.74 (~ butterworth).
  As all of these caps were removed, they are just voltage followers now.

- The final stage (AD620) has the gain resister of 16.5k. The gain is 1+(49.4k/16.5k) = 3.99.

- The 4pin lemo connector (J8) was removed from the board. We instead installed an isolated BNC connector on the panel for the thorlabs PD serving as the high gain PD.

- There is a daughter board for the high gain PD. This seems to be the butterworth low pass filter with fc=~30kHz.
  The differential output of the daughter board is connected to pin 17 and 18 of J10 (S5 Out and Rtn).

- The input of the daughter board is differential (AD620). Therefore the LEMO connectros next to the BNC were wrapped with Kapton tapes for isolation.

Board test at the workbench.

- The test required two dual power supply as the unit requires +/-5V and +/-15V.

- The four channels were tested with the signal injection. 1kHz input yielded 20mVpp across the AD602 input. The output of the 1st whitening stage was
  60mVpp. This makes sense as the gain of the AD620 is -10dB (1/10 and 10dB). The output of the 2nd whitening stage was 600mVpp.
  Finally the output of the output stage was confirmed to be 2400mVpp. This was confirmed for four channels.

- The daughter board output was also checked. The gain is the unity and flat upto ~10kHz.

Board installation

- Jenne installed the module. This time there was no smoke.

Gain mystery

- It was not sure how the whitening gains have been given.

- The corresponding database entry was found in /cvs/cds/caltech/target/c1auxey/ETMYaux.db as

grecord(ao,"C1:ASC-QPDY_S1WhiteGain")
grecord(ao,"C1:ASC-QPDY_S2WhiteGain")
grecord(ao,"C1:ASC-QPDY_S3WhiteGain")
grecord(ao,"C1:ASC-QPDY_S4WhiteGain")

- The gains for S2-S4 were set to be 30. However, C1:ASC-QPDY_S1WhiteGain was set to be 8.62068.
And it was not writable.

- After some investigation, it was found that the database was wrong. The DAC channel was changed from S100 to S0.
The corrected entry is shown here.

grecord(ao,"C1:ASC-QPDY_S1WhiteGain")
{
        field(DESC,"Whitening gain for QPDY Seg 1")
        field(DTYP,"VMIVME-4116")
        field(OUT,"#C0 S0 @")
        field(PREC,"1")
        field(EGUF,"42")
        field(EGUL,"-22")
        field(EGU,"dB")
        field(LINR,"LINEAR")
        field(DRVH,"30")
        field(DRVL,"-10")
        field(HOPR,"30")
        field(LOPR,"-10")
}

- Once c1auxey was rebooted, the S1 whitening gain became writable. Now all of the channels were set to be +30dB (max).

 Probably things would have worked better if you would have gotten your hair done at the same place as me.

To make the mcwfson/off scripts work from rossa (and not just Jamie's pet machine) I swapped the sh-bang line at the top of the script to use 'env bash' instead of 'env csh' in the case of mcwfsoff and 'env tcsh' in the case of mcwfson.

The script was failing to work due to $OSTYPE being defined for pianosa csh/tcsh, but not on rossa.

During debugging I also bypassed the ezcawrapper for ezcaswitch so that now when ezcaswitch is called, it directly runs the binary and not the script which calls the binary with numerous retries. In the future, all new scripts will be rewritten to use cdsutils, but until then beware of ezcaswitch failures.

WFS scripts checked into the SVN.

This was all in an effort to get Koji to allow me to upgrade pianosa to ubuntu 12 so that I can have ipython notebook on there.

Objections to upgrading pianosa? (chiara and megatron are already running ubuntu 12)

Inspired by a comment by Koji the other day, I spent some time yesterday and today working on locking a (very lossy) power recycled Y-arm. ITMX was misaligned, to save myself the headache of dealing with ITMY getting a sign flip and ITMX staying the same when the arm resonates. 

My main goal was to achieve high bandwidth control with the analog CARM servo. 

TL,DR: Transisitoned 90% to REFLDC through CM_SLOW at TRY = 2.1 twice. Couldn't make it all the way over. 

PRCL settings:

• Input: REFL165 I.
• Actuate on PRM +1
• Control: G=-.32 (~100Hz UGF); Acq on FM 4,5; Trig 1,2,3,6,9 (I modified the +10dB in FM1 to a 1kHz ELP)
• Trig: POP 110 I: 1.5 up, 0 down (max was around 4 counts, very weak PRC!)

The PRC was very stable in this configuration, which doesn't surprise me due to its simplicity. I was honestly a little surprised there was enough light to lock on 3f. REFL33 didn't work. 

My efforts to bring the Y-arm into lock were very similar to the CARM procedure we've been using recently. (Which is the motivation for this exercise)

At first I was actuating on ETMY, and got to the point where I wanted to start bringing in the CARM servo slow output, then realized that I didn't want to actuate both on the ETM and MC AO. (Maybe this would be doable, but in the end, not what I'm interested in learning about in terms of overlap with CARM locking)

From then on, I only actuated YARM on MC2. (Heads up, my lock-losses will show up in the trends of the MC2 Trans addition to the WFS.)

Transitioning the arm to SqrtInv TRY control was just as straightforward as it has been for CARM. However, engaging the LSCLock FM (FM4), would sometimes work beautifully, and sometimes kick the hell out of MC2. Keeping an eye on the error signal spectrum and UGF gave no indication which outcome would happen. Once FM4 could be engaged, the transmitted power was very stable. Without FM4, reducing the offset didn't get very far without losing lock. 

I tried a few times to bring in CM_slow (set to just IN2, i.e. offset adjusted REFLDC), at arbitrary arm powers, with little success. I didn't know how much arm power to expect at resonance, and thus didn't really know where on the line width I was.

I knew I was mostly outside of the linear regime of the PDH signals, since, even though I had good coherence between, say, REFL11 I and SqrtInvTry, with an ETMY excitation on; when I would turn TRY normalization on/off, I would see the sign of the TF change. 

I then realized that I could actively keep an eye on the trend of POY11, to see when I got to the PDH "hump", which is where REFLDC starts being usable, and SqrtInv is reaching its limit. 

This brought me to a YARM offset of .115, with a steady TRY of about 2.1. I adjusted the analog offset of the REFLDC input to the CARM board, and the digital gain of the CMSLOW input filter to get 1:1 correspondence between CMSLOW and the SQRTINVY channels. Their spectra were neigh identical, with CMSLOW having slightly more high frequency noise. 

I started stepping SQRTINV down by .1, and upping CMSLOW by .1. This shifted the offset around, so I opted for taking away gain before bringing it back, because I didn't want to get so close to resonance that SQRTINV would freak out. I got to .1*SQRTINVY + .9*CMSLOW, and lost lock. TRY was getting noisier as I made the transition. 

I'm not sure what exactly was the reason for failure. I'm going to go back over some of the data to try to get an idea.. Maybe I should've loosened up some of the gain/boosts during the transition. 

So, no great success story yet, but this configuration is a lot simpler than the full PRFPMI, and I feel that I should soon be able to get it fully controlled, and figure out a systematic way to make the digital to analog transition for this PRFP cavity, and thus have a much more informed basis for doing the same for CARM control. 

The measured open loop TF of the ALS Phase Tracker loop for each arm was characterized by an empirical model on LISO.

The model for the open loop TF has pole 1m instead of the one at DC as LISO has a difficulty to model it.
Digital time delay and the sampling effect seem to be well represented by a zero at ~8kHz and delay of  ~60us.
(cf 16kHz sampling => 61us)

The XARM phase tracker has the UGF of 1.5kHz. This is too low because
1) The phase rotation at 100Hz is visible in the plot.
2) We don't much care about the closed loop bump in the phase tracker as long as the phase tracker keeps its continuity.

So I suggest to increase the gain so that we have the UGF of 3kHz. (phase margin: 24deg)

The red curve in the plot is the closed loop response calculated by CLTF =  - OLTF / (1-OLTF).

The model results are used in the ALS servo models.

The measured openloop TF of the ALS servo for each was characterized by a ZPK model.

The openloop TF can be modeled by:

1) Filter TF obtained from foton
2) Actuator response with appropriate assumption
3) Phase tracker closed loop TF
4) Delay caused by the digital control
5) anything else

For 1) ZPK models of the servo filter was obtained from foton. It turned out that the TF of FM5 doesn't match with the ZPK model in foton.
Therefore the TF was exported and fitted with LISO. This seems to be related to the pole frequency (3kHz) which is too close to Nyquist frequency (8kHz).

FM(:,1)  = zero1(f,5).*pole1(f,0.001)*5000;
FM(:,2)  = zero1(f,1).*pole1(f,0.001)*1000;
FM(:,3)  = zero2(f,4.5,1.4619).*pole1(f,0.001).*pole1(f,0.001)*20.2501*1e6;
FM(:,4)  = zero2(f,35,2).*pole2(f,3,3).*zero1(f,3000).*pole1(f,1).*pole2(f,3000,1/sqrt(2)).*pole1(f,700).*zero1(f,10).*zero1(f,350).*136e1;
FM(:,5)  = zero1(f,1).*pole1(f,4.010e3).*pole2(f,17.3211e3,1.242).*zero2(f,18.865e3,100e3);
FM(:,6)  = zero2(f,3.2,0.966775).*pole2(f,3.2,30.572);
FM(:,7)  = zero2(f,16.5,2.48494).*pole2(f,16.5,78.5807).*zero2(f,24.0,2.22483).*pole2(f,24.0,7.03551);
FM(:,8)  = 1;
FM(:,9)  = zero2(f,7.50359,1.07194).*pole2(f,1.43429,0.717146)*27.5653;
FM(:,10) = 1;
dc_gain = 14;

FM1/2/3/5/6/7/9 are used for the control.

For 2), a resonant freq of 0.97 with Q of 5 was assumed.

The model for 3) was obtained by the previous entry.

Now the measured TF was divided by the known part of the model 1) ~ 3) and empirically fitted in LISO.

### XARM ###
pole 392.5021429051 698.1992431753m
zero 42.3128869460k 31.0954443799m
pole 589.2716424428 2.8325268375
factor 8.3430140244
delay 34.7536691023p

### YARM ###
pole 416.2463334253 743.2196174175m
zero 97.9161062704M 114.6703921876m
pole 626.0463515310 2.7671041771

factor 9.0045911761
delay 34.0945727358p

These compensation TF have weird TF. Probably the frequency response of the delay and the analog AA/AI filters without the high frequency data
led the LISO make up this. I'm requesting Masayuki to provide the AA/AI data to make the estimation more reasonable.
For the servo modeling, this is sufficient and we'll go a head.

The results of the OLTF modeling are attached.

As we now have the loop model, we can characterize the error signals.

We have the following data:

1) Free-running ALS error signals (i.e. phase tracker output) calibrated in Hz (for 532nm) (blue)
2) Controlled ALS error signals calibrated in Hz (for 532nm) (red)
3) ALS error signals measured with X and Y arm locked with the IR PDH. (black)
    This is likely to represent the sensing noise of the beatnote detection

from 2) we can derive the similar quantity as 1)
4) Estimated free-running ALS error signals from the controlled signals (green)

Remarks:

- From 1) and 4) we can see that the phase tracker is not perfectly linear. It seems that fast fringing of the phase tracker is causing upconversion.

- From 2) and 3) the servo loops don't have enough gain between 3Hz and 20Hz. On the other hand they have too much gain bekow 3Hz.

Based on the evaluation of the error signals, the new servo was designed.

Concept:
- Don't touch the locking filters. (i.e. FM5)
- Sacrifice some phase at 150Hz to increase the gain between 3-20Hz.
- As resonant gains costs the phase without increasing the LF gains, replace them with a poles for the integrators.

FM(:,1) = zero2(f,.5,.7).*pole2(f,0.001,.7)*(0.5/0.001)^2;
FM(:,2) = zero2(f,5,2).*pole2(f,3,3).*pole1(f,1).*zero1(f,5)*5*(5/3)^2;
FM(:,3) = zero2(f,25,.7).*pole2(f,3.2,10)*(25/3.2)^2; % Zero crossing
FM(:,4) = zero2(f,35,2).*pole2(f,3,3).*zero1(f,3000).*pole1(f,1).*pole2(f,3000,1/sqrt(2)).*pole1(f,700).*zero1(f,10).*zero1(f,350).*136e1;
FM(:,5) = zero1(f,1).*pole1(f,4010).*pole2(f,17.3211e3,1.242).*zero2(f,18.865e3,100e3);
FM(:,6) = zero2(f,5,2).*pole2(f,10,2).*pole2(f,16.5,30).*zero2(f,30,2);
FM(:,7) = 1;
FM(:,8) = 1;
FM(:,9) = 1;
FM(:,10) = 1;
dc_gain = 14;

FM1/2/3/5/6 are expected to be used for the control.

FM1: Boost below 0.5Hz. This does not cost the phase margin.
FM2: Increase the gain below 5Hz. This hardly cost the phase margin.
FM3: Boost below 25Hz. This is the main phase cost at UGF. This has a complex pole pair at 3Hz with Q=10 to supress the stack motion.
FM6: zero-pole-pole-zero combination to boost the gain between 5 to 30Hz. This eats the phase margin a little.

Note that the phase tracker gain for the X arm was increased by factor of 2.5.

Comparison of the new and old servo OLTF
The new servo has the same UGF, slightly less phase margin, and more gain between 1.5 and 25Hz.

The expected error signals derived from the estimated free running error signals of the ALS.

1) Previously estimated free-running noise (blue)
2) Previous in-loop ALS error signal (red)
3) Estimated error signal with the new servo (green)
4) Out-of-loop noise of the ALS with the arm controlled with the IR PDH (black)

Now the error signal (green) is expected to be very white.
The suppressed noise between 3 to 20Hz are below the sensing noise level.
There seems a little excess at 24.5Hz and 28Hz. If it is limiting the RMS, we need to take care of them.

Here are the MATLAB scripts and LISO codes for all of these servo analyses

 I was unhappy with the discontinuities between the Thorlabs and QPD versions of our transmitted light powers.  I realized that in the olden days, we just used the Thorlabs PD, and we set the no-light offset in the LSC version of the TR[x,y] filter banks.  However, now that we have brought the QPDs back, we are setting the dark offsets in the end suspension models, so that the signal chosen by the trigger already has its offset taken care of before we send it to the LSC model. 

Anyhow, having the offsets script try to put a value in the C1:LSC-TR[x,y]_OFFSET was giving us an extra offset and then when we did the normalizations, the numbers came out all wrong.  So.  I have removed the C1:LSC-TR[x,y] filter banks from the offset list, since they were made redundant. 

I have redone the normalizations for both arms (after running the ASS scripts).  I checked by watching the _OUT16 versions of the Thorlabs and QPD diodes before the triggering happens, and as I put offsets into the LSC servos to change the transmitted power, the diodes both change in the same way.  So, we'll have to see if this holds true for more than just values 0-1 the next time we lock, but hopefully it won't need changing for a while.

 Last long extension cord removed from 1Y1 to ITMX-ISCT

The AC power strip at ITMX-ISCT is coming from wall L#26

[Jenne, Koji]

The IFO is being uncooperative tonight, and I have an early morning meeting, so I'm calling it a night. 

Koji's filter module changes have been propagated from the Xarm to the Yarm, to CARM and to DARM.  (Actually, Q overwrote the changes to Xarm on Sunday accidentally, so first he reverted those for us, and then we propagated the changes). 

Today, with careful measuring, we find that for X and Y arms individually locked with the ALS, we want the gains to be +17 for the Yarm, and -17 for the Xarm (with the beatnote up-is-up convention).  This puts the UGFs at 150 Hz. 

We then switched over to CARM and DARM locking.  We guessed that the gains should be a factor of 2 lower since we're pushing on both ETMs for DARM, and the MC2 actuator is roughly the same strength as the sum of the ETMs.  In the end, after measuring the CARM and DARM loops, we find that the gains should be +7.5 for CARM, and +8.0 for DARM to set the UGFs at 150 Hz.  The servo is a little bit delicate, so having too low of gain is not okay. 

For some reason, we seem to be utilizing more actuator range with the new setup, so the limiters in the filter banks have been set to 11,000 (previously were 8,000), and the ALS watch script (ALSdown.py) threshold has been increased to 10,000 (previously 7,000). 

When finding the IR resonances with the new scheme, we are having trouble holding lock throughout the scan.  I have set the tramp for the coarse part of the scan to be 0.05 seconds (previously 0.01 seconds), which is an increase of a factor of 5 in the ramp time.  This helps, but may still not be enough, since we don't always hold lock until both IR resonances are found.

Probably the most annoying thing from tonight is the fact that ETMY keeps drifting off, particularly in yaw, when locked.  I don't have an explanation of why this is happening, but you can watch it happen sometimes, and the lock will be lost shortly thereafter.  Definitely when we lose lock and the ETM gets kicked, it is far enough away in yaw alignment that I have to completely redo the Yarm alignment.  This happens whether or not the ETMY oplevs are on.

To summarize, 3 scripts have been modified:

(1) ALSdown - threshold increased  (Modification from last week - turns off the slow temp servos for the end lasers, clears histories)

(2) ALSfindIRresonance - increase ramp time

(3) Lock_ALS_CARMandDARM - final gain values set to 7.5 for CARM and 8 for DARM, no filters come on until gains all the way up, turns on new set of Koji filters. (Modification from last week - turns on the slow temperature servos for the end lasers)

Be aware that this may affect POP QPD and POP RF Thorlabs PD

 TP2 dry pump replaced at fore pump pressure 1 Torr,  TP2 50K_rpm 0.34A

 Top seal life 6,362 hrs

 New seal performance at 1 hr  36 mTorr, 

 Maglev at 560 Hz, cc1 6e-6 Torr

c1sus and c1iscey were reset. The PMC needed to be locked. MC locked instantly.

The FSS ion pump power supply was turned on.

c1sus and c1isey were not talking to fb. The usual mxstream restart did not help.

Restarted fb

>>ssh fb

>>telnet fb 8087
shutdown

All lights on the FE status screen are green now.

Note that Steve did an mxstreamrestart earlier today because the same machines c1sus and c1isey were not talking to fb.