That's a very smooth DARM transition - its good news that the dALS signals don't have a huge offset w.r.t the real error signal.

It would be interesting to see if the MICH can be locked and stay locked which CARM is ramped in. We would want to hold it with the Q phase of the CARM PD once its on.

May not be a milestone, but its cool anyway. 

+2 points.

Will also be cool to see how soon the CM servo can be switched on in the acquisition sequence. Maybe ALS_COMM -> CM board, gets mixed with TRXY for low frequencies in the intermediate stage before final RF?

I'm not sure why, but the WFS were turned off when I came in this morning.  The MC was not staying locked, and even during brief locks, the FSS FAST out was railed at 10. 

Aligning the MC mirrors to maximize the transmission, and then engaging the WFS seems to have made things better.

I have compressed the IFO Configure screen.  All PRMI things (sideband lock and carrier lock) are in the PRMI button, all arm things (both RF and ALS) are in the respective arm buttons.

I have also made a new set of scripts for CARM and DARM lock acquisition with ALS. 

I hope that each button's purpose is clear, but take a second to look at them before you next use the IFO Configure screen.

As payment for borrowing 2 of our seismometers, Zach has made us a new Trillium cable, to go from the granite station to the readout box, which we can put into 1X7, where the PEM ADC is.  To put the T-240 in side the can, and seal it, we need a little jumper cable from the seismometer to the granite, but for now, we can just pass this cable underneath the can.

This afternoon, I was toying around with reducing either the CARM or DARM offsets (so, put in a CARM offset, leave DARM zero, lock the PRMI, then reduce CARM offset to zero.  Or, put in a DARM offset, leaving CARM offset zero, lock the PRMI, then reduce the DARM offset to zero).

When looking at the data, I see that the MICH error signal gets fuzzier when the arms get close to resonance. (Note here that because I forgot to zero the carm offset before finding the resonances, -3 is my zero point for this plot and the next.)

Here is a zoom of the last piece of this time series, but with both TRX and TRY plotted (along with POPDC, CARM_ERR and DARM_ERR), where you can see that I had a momentary power buildup of > 100 transmission counts, which is about 20% of our final expected power.

Here is a different time series, showing a reduction of the DARM offset, and you can see that as the offset approaches zero, the MICH error signal gets noticeably more fuzzy.  Somewhere near the 240 second mark, I lose PRMI lock.

[Jenne, EricQ]

I told Koji that I wanted to play with the common mode servo this evening, and he pointed out that we only get the signals after the digital demod phase angle in the digital system (obviously).  So, if I want to use either REFL11 or REFL55 for my CARM signal, I want to do something in analog-land so that my digital demod phase is close to 0 or 90. 

While we had the PRFPMI locked (with CARM offset of 2 or 3 nm), we set the demod phases of REFL11 and REFL55 to minimize a CARM line in the Q-phase.  This gave us -34 degrees for REFL11, and -75 degrees for REFL55. 

We calculated that about 1 degree of phase shift is about 1/(2 * pi * freq), or about 1.4e-8 seconds of delay for 11MHz.  We took the speed of light in the cables to be about 2/3*c, so 1.4e-8 * 2e8 = 2.9 meters per degree for 11MHz.  Since REFL11 was 34 degrees from 0, we estimate that we need to add about 98 meters of cable to the REFL11 signal path.  The same calculation for 55 MHz, but with a 15 degree shift required, gives 8.8 meters of cable to be added to the REFL55 signal path. 

I connected up some long BNC cables, and inserted them between the heliax breakout board on the LSC rack, and the respective PD inputs of the REFL11 and REFL55 demod boards.  I used (45 meters + 45 meters + a little bit) for REFL11, and used about 9 meters for REFL55. 

When we relocked the PRFPMI, and redid the phasing, we were very close to zero for both REFL11 and REFL55!  REFL11's digital demod phase is now +1 degree, and REFL55's digital demod phase is -5 degrees.

We changed the input of the CM servo board from POY (which Den and Koji had been using in December - see elog 9500) to REFL11 I MON. 

Q locked the FPMI (separate reply elog), and then we tried engaging the CM analog servo.  We were not successful. 

These settings were mostly copied from elog 9500, so they are almost surely not correct. 

CM servo screen:  In1 gain = 31dB, switch on, offset = -2.7V, boost off, super boosts off, option=disable, 79:1.6k switch disabled, polarity minus, option disable, AO gain=8dB, limiter enable.

For the slow path, CM_SLOW -> MC LSC servo had a +1 in the input matrix. 

CM filters in the AUX_ERR screen:  FM1 (unwhite) on, all others off, gain = 2.6. 

MC servo filters:  FM7, FM10 on, all others off (no triggered filter modules).  Gain = 0 initially.

MC servo board AO path disabled initially, G=-32dB initially.

Once Q had the FPMI locked, I tried increasing just the CM analog gain (by enabling the AO path on the MC board, and increasing the gain).  Doing this, I lost lock at -3 dB. 

I then tried again, this time alternating increasing the analog gain, and increasing the MC LSC servo gain.  I got up to 3e-3 for the MC digital gain, and -7 dB for the analog gain before we lost lock again.

We have determined that we should probably try just locking one of the arms with POX or POY, as Den and Koji did, to get a feel for how the system works.

For future reference:

As we were poking around with the common mode servo in an FPMI configuration, we locked CARM/DARM with ALS as in recent ELOGs.

MICH was locked on ASDC: ASDC -> MICH = 10.0 in the DCPD DoF Matrix (I couldn't easily get AS55Q working, ASDC worked quickly and good enough)

MICH gain +25, FM4 FM5 On, FM2 switched on once locked. Offset was manually adjusted to get closer to dark fringe.

Actuated on BS: MICH->BS = 0.5 in Output Matrix.

I have never used such a long cable for RF phase adjustment. The speed of the signal is 2e8 m/s and the frequency is ~10e6 Hz.
This means that the wavelength is only about 20m. How could you end up with ~100meters?
The convenient way to remember the cable delay is "1m, 1MHz, 2deg". This gives us ~1.5m for 11MHz and 34deg.

In fact, 1 degree of phase shift is not 1/(2 pi freq) second of delay, but f/360.

For such a precise phase adjustment, it is better to calibrate the delay with the network analyzer.

I guess this is normal. DARM has (almost) the same effect of MICH on the corner interferometer signals, just increased in amplitude by the arm cavity amplification. When the arm is out of resonance, DARM is almost completely depressed and it is not affecting MICH at all. On the other hand, when the cavities are exactly at resonance, DARM signal is amplified w.r.t. MICH by the cavity gain (2F/pi).

Since DARM is still controlled with ALS, it is probably noisy. The closer to resonance you move the cavities, the more ALS noise in DARM will affect MICH.

The 2W Innolight was off for 4 hours.

Yesterday, MC2 alignment was slipping all day. Even when the WFS was off (i.e. there wa sno actuation), I had continual misalignment caused by MC2

I was afraid that the MC2 mirror is on a bistable position somehow. So I gave -5000 offset on the MC2 LSC. We'll see how it makes the MC happier.

[Jenne, EricQ]

This evening, as part of locking activities, we threw together some handy scripts.

The first one, "Lock_ALS_CARM_and_DARM.py" (no judging of my naming style!!), lives in .../scripts/ALS/ . 

It acquires ALS lock in CARM and DARM mode, so we don't have to do it by hand anymore.

The first thing that it does is ask you to acknowledge that your beatnotes are in place, and they follow our new (newer than the last elog about conventions) beatnote convention.  You are reminded in the terminal window what that convention is:  When the temperature sliders for either arm is INCREASED, the beatnote frequency should INCREASE. 

After you acknowledge that the beatnotes are good, it sets the CARM and DARM servo gains to zero, enables the outputs, sets the input matrix elements, clears the phase tracker histories, and starts ramping up the gains (with +1,+1 for DARM, the darm servo gain is +positive.  with -1*ALSX,+1*ALSY for CARM, the carm servo gain is -negative).  At a gain of 3, it engages the integrators and the resonant gains.  At the final gain of 6, it engages the boosts.

We have used this script ~10 times tonight, and it's been great every time.

The next two scripts are for making the transition from ALS to IR signals.  They both live in ..../scripts/PRFPMI/

"Transition_CARM_ALS_to_TransSqrtInv.py" (again - no judging!) slowly blends the input matrix elements to swap CARM control from the ALS signals to the 1/sqrt(trans) signals.  It takes a few steps, and asks for a keyboard input between steps.  This is because if our 1/sqrt(trans) offsets aren't perfect, we can start to lose transmission power.  To mitigate this, we decrease the offset in the CARM servo filter bank to get more power back.  This script requires an input, which is what you want the final sqrtinv matrix elements to be.  It will fail without this.  For a CARM offset, both of the final sqrtinv matrix elements will have the same sign.

"Transition_DARM_ALS_to_AS55.py" (I can telepathically hear you judging me right now.)  does the same blending, except to swap DARM control from ALS signals to AS55Q.  For the same reason of imperfect offset-setting, it takes several steps, to allow you to adjust the CARM offset if needed. Although, after typing this, I realized that perhaps we should have been tweaking the DARM offset.  Either way, this transition required much less tweaking of offsets than the CARM transition did.  Again, the script requires an input, which is your final desired AS55Q->DARM matrix element value.

*  Both of these scripts should be run at a digital CARM offset of about 2 counts, although with the offset tweaking during the CARM transition, I usually end at about 1.5 counts. 

*  To determine the final gain value for the CARM sqrtinv matrix elements, we have been using a spare filter bank (ex. XARM), and having the input to that be the sum of the sqrtinv channels.  We then put in a CARM line, and look at the transfer function between the temporary filter bank's input, and the CARM_IN1. 

*  To determine the final gain value for the DARM AS55 matrix element, we have been doing a similar thing, looking at the transfer function between DARM_IN1 and AS55Q with a DARM line on.  We have been putting this DC gain into the static PD normalization (4th block from the left on the big LSC screen), although with the new script, it will be easier to just put that value into the matrix element.

[Jenne, EricQ]

Tonight, we transitioned CARM and DARM to IR signals, took loop transfer functions, and determined that we could engage the LSC boosts (FM4 in the CARM and DARM servos, which are the same as the XARM and YARM servos). 

Q is preparing spectra to post, and I will dig out time series.  Look for these tomorrow, if they aren't posted tonight.

For the time series data fetching, I have taken notes on what we were doing when, so that I can actually find the data.

11:09pm:  CARM's LSC boost on for the first time

11:14pm:  DARM transferred to AS55Q

11:21pm:  DARM's LSC boost on for the first time

(lockloss)

11:53pm:  CARM transition

12:02am:  DARM transition done, both LSC boosts on

12:04am:  lockloss after reducing CARM digital offset to 0.4

12:45am: PRMI + 2 arms flashing, with no CARM or DARM offsets (arms still on ALS) because we forgot to put in the CARM offset before restoring PRM alignment.  PRMI may have been actually locked, or we may just have been flashing....need to look through the data to see what our recycling looked like.

(lockloss)

1:05am:  pretty smooth transition completed (both CARM and DARM), but we lost lock while reducing the CARM offset.

1:19am: lockloss - why?? We were just sitting at a CARM offset of about 1.3nm (1.3 counts), holding on IR signals.  We were not touching any IFO things while looking at some plots, and just lost lock.  Want to see if we can understand why.

1:27am:  another nice smooth transition for both CARM and DARM to IR signals, but almost immediate lockloss when reducing the CARM offset.

Using the new ALS lock acquisition scripts (elog 9816) and our transition scripts, getting back to PRFPMI lock is pretty smooth and procedural.

* Align arms using ASS (ifo configure screen, restore xarm and yarm, run both arms' ass scripts).

* Align PRMI, no arms (ifo configure screen, restore prmi sideband)

* Find ALS beatnotes, with arm lasers on opposite sides of the PSL.  For both, when increasing the value of the temperature slider, the beatnote should increase in frequency.  (ifo configure screen, restore CARM and DARM als)

* Run ...../scripts/ALS/Lock_ALS_CARM_and_DARM.py

* Run "Find resonance" scripts from ALS screen for each arm.

* Put in a 3 count offset to CARM loop.

* Restore PRM alignment.  (PRMI should acquire lock immediately, although PRM may need some small alignment tweaking).  Enable PRCL and MICH outputs, PRM and BS actuation outputs.

* Reduce CARM offset to 2 counts. 

* Set offsets of 1/sqrt(TRX) and 1/sqrt(TRY) filter banks in the AUXERR section of the LSC screen.  The outputs of both should equal 2 counts (to match the 2 count offset in the CARM loop). 

* Run .../scripts/PRFPMI/Transition_CARM_ALS_to_TransSqrtInv.py , making sure to reduce the CARM digital offset if needed, to keep the arm transmissions at about 0.1 counts.

* Engage FM4 of the CARM filter bank, which is the LSC boost.

* Run .../scripts/PRFPMI/Transition_DARM_ALS_to_AS55.py , making sure to reduce the CARM (or should be DARM?) digital offset if needed, to keep the arm transmissions at about 0.1 counts.

* Engage FM4 of the DARM filter bank, which is the LSC boost.

Notes for going forward:

When we have small-ish digital CARM offsets, such that both of our arm transmitted powers are about 0.1 or higher, we see clear coherence between our CARM_IN1 (the 1/sqrt(trans) signals) and a normalized REFL11_I (again using a spare filter bank like XARM to get REFL11 normalized by (TRX+TRY) ).  We have not yet tried transitioning the CARM digital error signal to this normalized REFL11.

Even though we see that the IFO is much less noisy (as measured by significantly reduced RIN in TRX and TRY as visible by eye on Dataveiwer), we are still losing lock when we reduce the CARM offset.  I have noted above several times, for when we had locklosses, so that I can see if I see anything elucidating in the time series data.

 As Jenne mentioned, we took OLTF transfer functions, and determined that we had more than enough phase margin to switch on the LSC boosts in FM4. This improved the error signal noise spectra quite a lot, and noticeably reduced the TRX/TRY fluctuations, and actuation output. 

Here's the CARM OLTF (FM4 boost on in red, boost off in black)

Here's what happened to the CARM and DARM spectra when we turned on the boosts. (ALS only in black, initial IR signal transitions in mid-color, boosted IR signals in bright color)

I looked at 2 of the locklosses from last night, (1:19am and 1:27am), and saw that for both, the DARM loop started to oscillate just before we lost lock.  In the trials tonight, we were more watchful of gain peaking.

Here is the 1:19am lockloss

And here is the 1:27am lockloss

 So you can see what we were doing, and what the effect was, here is a few minutes of data just before the 1:27am lockloss. The times I note below are rough, but should give you an idea of what happened at which time.

0 sec:  Arms are held on resonance with ALS.

10 sec:  CARM offset of 3nm added.

20 sec:  PRM restored, one flash, then PRMI acquires lock.

30 sec:  CARM offset reduced to 2nm, transmitted powers are about 0.1

50 sec:  Transition CARM to 1/sqrt(trans) signals.  Note that we are using the high gain Thorlabs PD here, so the noise is better than last Thursday.

60-110 sec:  CARM offset reduction to about 1nm.

110 sec:  CARM's LSC low frequency boost engaged.

120 sec:  DARM transitioned to AS55Q.

170 sec:  DARM's LSC low frequency boost engaged.

Last night, EricQ and I were concerned that we might need some CARM UGF servoing, so I added a UGF servo block, copied from the aLIGO LSC model, to our LSC model.  The block is inline with the CARM servo, after the output triggering, just before the output matrix.  Q put together some screens, which are accessible from the main LSC screen. 

The model is compiled and running.  We didn't get very far in testing it though before Koji pointed out that it is a slow solution, and not a fast one like we were searching for.  We were hoping to deal with the momentary power buildup, and thus optical gain change, as the arms flash close to resonance.  The UGF servo will not work nearly that fast though.  We may want it for slow UGF servo-ing, but it's not the solution to what Q and I were thinking about yesterday.  Regular ol' dynamic normalization is closer to the right answer for this.

In tonight's activities, Koji and I found that we probably want a CESAR block for DARM as well as CARM, so that we can independently normalize AS55Q. 

To solve the DARM oscillation issue from last night (that I discovered this evening when I finally looked at the time series data), we may want to implement a DARM UGF servo.  For tonight, as we reduced the CARM offset and started seeing gain peaking in the DARM spectra, I hand-reduced the DARM gain.

[Jenne, Koji]

This evening, Koji and I followed the procedure from last night (elog 9817), with the exceptions that as we saw gain peaking in the DARM spectrum, we lowered the DARM servo gain.  We also left the DARM FM4 boost off, and added (TRX+TRY) power normalization to AS55 (although we still had to hand-reduce the gain).    These changes enabled us to reduce the CARM offset much further.  We were able to get the transmitted powers to hold steady at about 1 count while on the IR signals, which is a new record for us.  (We had in the past held the arms with ALS at several counts, but the power fluctuations were huge, and now they are nice and small).

After that, we started looking at whether we could transition CARM over to REFL11I.  We tried a few times, but never made it all the way.

Here are some times for data-foraging tomorrow:

8:27pm, nice transition, CARM offset reduction to 0.6 before lockloss.

9:19pm, turned on power normalization for AS55Q, then reduced CARM offset to 0.5

9:40pm, Lockloss after reducing CARM offset to -0.24, arm transmitted powers around 0.9.

gps 1081748419: First trial trying to transition CARM to REFL11I normalized by (TRX+TRY).

gps 1081749965:  Tried to transition CARM to (REFL11 + REFL33)/(TRX+TRY).  Got about 1/3 of the way through the transition (in terms of matrix element value steps) before lockloss.

11:56pm, Tried to add in REFL11I to CARM error signal (without reducing 1/sqrt(trans) matrix elements).  We increased the REFL11 matrix element until we saw gain peaking, and then tried reducing the 1/sqrt(trans) contribution, and lost lock.  We were only at an offset of 0.3, so we probably weren't close enough to the resonance yet.  We were able to add in REFL11 information, but this was probably not too hard, since there wasn't much actual information in it.

Thoughts:

* It's a little weird that once we are on IR signals, the 0 CARM offset point that we find with ALS is not the true CARM offset point.  Although, this may be because we're just going to an averaged no CARM offset place with ALS, but since ALS is noisy, we won't ever really be holding on the zero offset point.  Anyhow, when we were using the 1/sqrt(trans) signals for CARM, and the CARM digital offset was -0.24, the ALSX and ALSY outputs were both about 0.5 in magnitude.

* We're getting there! 

I've been trying to get c1ioo on the Dolphin network, but have not yet been successful.

Background: if we can put the c1ioo machine on the fast Dolphin IPC network, we can essentially eliminate latencies between the c1als model and the c1lsc model, which are currently connected via a rube goldberg-esq c1lsc->dolphin->c1sus->rfm->c1ioo configuration.

Rolf gave us a Dolpin host adapter card, and we purchased a Dolphin fiber cable to run from the 1X2 rack to the 1X4 rack where the Dolphin switch is.

Yesterday I installed the dolphin card into c1ioo.  Unfortunately, c1ioo, which is Sun Fire X4600, and therefore different than the rest of the front end machines, doesn't seem to be recognizing the card.  The /etc/dolphin_present.sh script, which is supposed to detect the presence of the card by grep'ing for the string 'Stargen' in the lspci output, returns null.

I've tried moving the card to different PCIe slots, as well as swapping it out with another Dolphin host adapter that we have.  Neither worked.

I looked at the Dolphin host adapter installed in c1lsc and it's quite different, presumably a newer or older model.  Not sure if that has anything to do with anything.

I'm contacting Rolf to see if he has any other ideas.

 I have made the same modification to the Yarm trans PD whitening board as was done for the xend, to increase our SNR.  I put in a 2.1kOhm thin film resistor in the Rgain place.

When I was pulling the board, the ribbon cable that goes to the ADC had its connector break.  I redid the ribbon connector before putting the board back. 

I see signals coming into the digital system for both the high gain and low gain Y transmission PDs, so I think we're back.  I will re-do the normalization after Jamie is finished working on the computers for the day.

So it turns out that the card that Rolf had given me was not a Dolphin host adapter after all.  He did have an actual host adapter board on hand, though, and kindly let us take it.  And this one works!

I installed the new board in c1ioo, and it recognized it.  Upon boot, the dolphin configuration scripts managed to automatically recognize the card, load the necessary kernel modules, and configure it.  I'll describe below how I got everything working.

However, at some point mx_stream stopped working on c1ioo.  I have no idea why, and it shouldn't be related to any of this dolphin stuff at all.  But given that mx_stream stopped working at the same time the dolphin stuff started working, I didn't take any chances and completely backed out all the dolphin stuff on c1ioo, including removing the dolphin host adapter from the chassis all together.  Unfortunately that didn't fix any of the mx_stream issues, so mx_stream continues to not work on c1ioo.  I'll follow up in a separate post about that.  In the meantime, here's what I did to get dolphin working on c1ioo:

c1ioo Dolphin configuration

To get the new host recognized on the Dolphin network, I had to make a couple of changes to the dolphin manager setup on fb.  I referenced the following page:

https://cdswiki.ligo-la.caltech.edu/foswiki/bin/view/CDS/DolphinHowTo

Below are the two patches I made to the dolphin ("dis") config files on fb:

--- /etc/dis/dishosts.conf.bak    2014-04-17 09:31:08.000000000 -0700
+++ /etc/dis/dishosts.conf    2014-04-17 09:28:27.000000000 -0700
@@ -26,6 +26,8 @@
 ADAPTER:  c1sus_a0 8 0 4
 HOSTNAME: c1lsc
 ADAPTER:  c1lsc_a0 12 0 4
+HOSTNAME: c1ioo
+ADAPTER:  c1ioo_a0 16 0 4
 
 # Here we define a socket adapter in single mode.
 #SOCKETADAPTER: sockad_0 SINGLE 0

--- /etc/dis/networkmanager.conf.bak    2014-04-17 09:30:40.000000000 -0700
+++ /etc/dis/networkmanager.conf    2014-04-17 09:30:48.000000000 -0700
@@ -39,7 +39,7 @@
 # Number of nodes in X Dimension. If you are using a single ring, please 
 # specify number of nodes in ring.
 
--dimensionX 2;
+-dimensionX 3;
 
 # Number of nodes in Y Dimension. 

I then had to restart the DIS network manager to see these changes take affect:

$ sudo /etc/init.d/dis_networkmgr restart

I then rebooted c1ioo one more time, after which c1ioo showed up in the dxadmin GUI.

At this point I tried adding a dolphin IPC connection between c1als and c1lsc to see if it worked.  Unfortunately everything crashed every time I tried to run the models (including models on other machines!).  The problem was that I had forgotten to tell the c1ioo IOP (c1x03) to use PCIe RFM (i.e. Dolphin).  This is done by adding the following flag to the cdsParamters block in the IOP:

pciRfm=1

Once this was added, and the IOP was rebuilt/installed/restarted and came back up fine.  The c1als model with the dolphin output also came up fine.

However, at this point I ran into the c1ioo mx_stream problem and started backing everything out.

While trying to get dolphin working on c1ioo, the c1ioo mx_stream processes mysteriously stopped working.  The mx_stream process itself just won't start now.  I have no idea why, or what could have happened to cause this change.  I was working on PCIe dolphin stuff, but have since backed out everything that I had done, and still the c1ioo mx_stream process will not start.

mx_stream relies on the open-mx kernel module, but that appears to be fine:

controls@c1ioo ~ 0$ /opt/open-mx/bin/omx_info  
Open-MX version 1.3.901
 build: root@fb:/root/open-mx-1.3.901 Wed Feb 23 11:13:17 PST 2011

Found 1 boards (32 max) supporting 32 endpoints each:
 c1ioo:0 (board #0 name eth1 addr 00:14:4f:40:64:25)
   managed by driver 'e1000'
   attached to numa node 0

Peer table is ready, mapper is 00:30:48:d6:11:17
================================================
  0) 00:14:4f:40:64:25 c1ioo:0
  1) 00:30:48:d6:11:17 c1iscey:0
  2) 00:25:90:0d:75:bb c1sus:0
  3) 00:30:48:be:11:5d c1iscex:0
  4) 00:30:48:bf:69:4f c1lsc:0
controls@c1ioo ~ 0$ 

However, if trying to start mx_stream now fails:

controls@c1ioo ~ 0$ /opt/rtcds/caltech/c1/target/fb/mx_stream -s c1x03 c1ioo c1als -d fb:0
c1x03
mmapped address is 0x7f885f576000
mapped at 0x7f885f576000
send len = 263596
OMX: Failed to find peer index of board 00:00:00:00:00:00 (Peer Not Found in the Table)
mx_connect failed
controls@c1ioo ~ 1$ 

I'm not quite sure how to interpret this error message.  The "00:00:00:00:00:00" has the form of a 48-bit MAC address that would be used for a hardware identifier, ala the second column of the OMC "peer table" above, although of course all zeros is not an actual address.  So there's some disconnect between mx_stream and the actually omx configuration stuff that's running underneath.

Again, I have no idea what happened.  I spoke to Rolf and he's going to try to help sort this out tomorrow.

Jamie tells me that the 2 big consequences of this are (a) we are not archiving any data that is collected on the ioo machine, and (b) that we will not have access to test points on the IOO or ALS models.

To make sure that this is not a show-stopper for locking, I have locked the arms using ALS.  The signals seem to still be getting from the ALS model to the LSC model, and I'm able to acquire ALS lock, so we should be able to work tonight.  All of the data that I have been looking at lately has been coming off of the LSC machine, so we should even be okay in terms of look-back for lockloss studies, etc.

Jenne made some suggestions for some plots that would be useful on our CARM offset reduction adventures, so I made some with my MIST model. 

First, here's a plot showing the transfer function of CARM to TRX, with logarithmically spaced offsets out to 3nm. While not a control signal, it shows us where the optical plant resonance stuff is happening. The peaks in this TF correspond to peaks in REFL11, REFL55, AS11, etc., as in the close-to-resonance TFs in ELOG 9785. 

[more to come, had a MATLAB issue]

10 days of on-off glitching?   It is a kick. The LSC is off, so it must be the ALS

Some time series data from Wednesday night. 

Here is the first time we've gotten the arm transmissions to about 1 count, while holding CARM and DARM on IR signals, so the transmission, as well as POPDC, were relatively quiet.

As we were attempting to transition CARM to REFL11I, at least 2 of the times, we were hitting a CARM oscillation:

 To fix open-mx connection to c1ioo, had to restart the mx mapper on fb machine. Command is /opt/mx/sbin/mx_start_mapper, to be run as root. Once this was done, omx_info on c1ioo computer showed fb:0 in the table and mx_stream started back up on its own. 

Thanks so much Rolf (and Keith)!

Last night, as well as tonight, the ALS seems not quite as robust as it was earlier in the week.

I have just taken noise spectra, and ALS is definitely more noisy than usual. 

These plots are with the arms held in CARM and DARM mode, with servo gains of 8. I was seeing the beginnings of gain peaking at a gain of 10, so I turned it back to 8.  Our ALS in-loop RMS is usually something like a few hundred Hz, but I'm seeing over 1kHz, so I have a factor of 4 or 5 too much noise.  Why?!?!?

Came in, and the PMC and MC needed aligning. 

PMC Trans .74->.83

I wasn't able to move the MC1&3 the right way in pitch without having a terrible MC2 position... using the move MC2 scripts to bring it back threw off the other spots.

 The  Maglev controller was serviced at Osaka Vacuum. It was swapped in this morning.

Not working again. I tried the commands in Koji's elog as well as the ones from my notes, but the AG 4395 hooked up to the yellow box named crocetta doesn't work. It gets to the stage of opening the data files and hangs. I tried it with many variations on pianosa and rossa. Also tried power cycling the analyzer and the Prologix and the bridge.

While hooked up to the MC error point I set the modulation frequency from 137 to 133 Hz to minimize the 3 MHz peak as usual.

I have noticed that ALS noise has been at 1KHz rms since LSC arm lock servos have been used to lock arms using ALS error signals. May be this has not been given much attention.

But looking more closely at the ALS noise (better dtt resolution for noise power spectrum) , there seems to be too much noise suppression at <1Hz and not much happening at around 10Hz.

Attachment 1 (data files at /users/manasa/data/140421/)

So I made a bunch of transfer function measurements for ALS and phase tracker servo. Koji will be using these and redesigning the servo filters so that we can get more suppression at 10Hz.

Other than this I also found that the Y arm showed more high frequency noise as compared to the X arm. (Edit by manasa: Thinking back now, this could be related to the onset of 60Hz noise at the Y end elog 9838. But still has to be looked at after fixing TRY)

Attachment 2

Tip: Once the arms are ALS locked, enabling the SLOW_SERVO helps hold the lock stably. 

P.S. I realigned the Y green to the arm and brought GTRY to 0.93

To do:

Find out what makes Y arm in-loop noise at high frequency higher than X arm.

To check the basolute frequency stability of the old monochrome HP 8591E RF Spectrum analyzer that we're using for the ALS beat readout, I hooked its 10 MHz reference output (from its rear panel) into the A channel of the SRS SR620 frequency counter. The SR620 is locked to the FS 720 Rubidium clock via the 10 MHz connections in their rear panels.

So, we can assume that this is a good absolute readout. It reads 9.999860.7 +/- 0.3 Hz. So its 139.1-139.4 Hz lower than 10 MHz. The +/- 0.3 is just a slow drift that I see over the course of 10 minutes.

So, let's say that the analyzer is low by 10 ppm, so the arm length estimates are short by ~0.4 mm. A negligible correction, so there's no need to use atomic clocks to measure our arm lengths.

This evening, I was not able to successfully transition CARM from ALS to 1/sqrt(trans) signals.  The TRY time series looked odd, so I took a spectra, and we have huge 60Hz noise in TRY. 

I found a lock stretch from around 6:30pm that did not show the 60Hz noise, and then there was a lock stretch around 8pm that did have the noise.  So, something happened at the Yend between 6:30 and 8pm tonight.

Asking around, this was the time frame in which Manasa was down at the Yend to realign the green beam, and to check cabling for the PZT_OUT and ERR_MON signals to the ADC.

Looking at the spectra, Rana noted that we have even as well as odd harmonics of the 60Hz line, which is unusual.

To try to diagnose the problem, Rana and I tried to make sure no cables' connectors were touching, and that no equipment was plugged in that shouldn't be.  We noticed that none of: the shutter, the Thorlabs TRY PD, or the QPD TRY are isolated from the table.  To see if perhaps the shutter was the problem, I turned off the power to the Yend green shutter, and unplugged the cable.  The cable is laying on the table, with the connector sitting on a piece of plastic to isolate it.  Removing the shutter from the system did not change anything. 

We don't see the 60Hz noise in the Xarm, so it's not on the laser light itself.  Also, we don't see the 60Hz lines in the Yarm feedback signal, so we're not putting the lines onto the mirror, and thus onto the Yarm's light. 

Manasa, can you please take a look, and see if you can figure out what is going on?  We need TRY so that we can transition to 1/sqrt(trans) signals for CARM.  Thanks!!

[Jenne, Q]

The frame builder (or something) is unhappy again.  I know that we've seen this before, but I can't find the elog entry that relates to this particular problem.

Every few minutes, the fb status lights on the CDS_STATUS screen go white, and then come back green.  It's annoying when it happens every hour or so (which is unfortunately typical), but it's pretty debilitating when it stops dataviewer and dtt every few minutes.  Just from the way the lights change, it looks like perhaps the daqd process is restarting itself periodically? 

In an effort to familiarize myself with the analog CM servo, I've begun to replicate Koji and Den's work from the ELOG post that this is a reply to.

I hooked POX11Q into the IN1 of the CM board. (POX is rotated by ~86 degrees in the CDS, meaning analog Q is almost perfect.)

While there, I took out the too-long delay cables Jenne introduced for REFLs 11 and 55. (Also note: when we do cable-based analog phasing in the future, we should do it on the LO side, instead of the PD input side.) I also heard a dangerously crinkly sound from the short SMA cable for REFL11, so I replaced it with a beefy looking new one I found on the SP table.

I messed with the gain and offset in the CM_SLOW input filter to get it to look just like POX11_I_ERR, and was able to lock the arm on it without an issue. I then put the SR560 between the CM and MC (30k pole, but also AC coupled, because I figure the digital loop should be doing the work down there, and don't want to kick the AO with an offset), and was able to turn on the AO path with a gain of 8dB on the CM board and 10dB on the MC board, as detailed in Koji's procedures.

I wasn't able to increase the AO gain to 9dB without breaking lock, but maybe this is ok, because by judging by the LSC filter gains, POX11 might be about 3 times bigger than POY, so maybe 8dB AO gain on POX ~ =18dB AO gain on POY? I was able to put the CM servo offset at 0, but turning on boosts promptly kicked the MC out of lock.

I'm stopping for the night; but tomorrow I'll bust out a spectrum analyzer to see if I actually have won some bandwidth with the CM servo, and check out the situation with the offsets and boosts.

 I went to the Y end to look at the TRY 60Hz noise situation this morning. While looking at TRY noise on dtt, I found that just lifting the cable away from the cable bunch that runs out of the table suppressed the noise drastically. 

Attachment 1

I removed the unwanted bnc connector in the path of the already long TRY cable running from the PD to the 1Y4 rack and isolated it from the bunch. TRY became less noisy.

But the noise was back again earlier in the evening and it looks like the noise is very much related to the TRY cable. TRY cable might have moved from its sweet spot while I was around checking cable connections yesterday.

I couldn't find a spare to replace it right away today (We need a BNC to 4 pin lemo).

 The detectors and electronics on this table are not properly isolated. To reduce the 60 Hz and ground loops, photodiodes and shutter must be isolated by using plastic spacers as we usually do elsewhere - this table just seems to have a few oversights.

Steve can start assembling all of the pieces to do this in the morning and then we can start the swapping after the meeting.

The high gain Transmon cable should be a regular BNC. There's no need for 4-pin LEMO in this usage, so the best move is to modify the board and replace the 4-pin LEMO connector with an isolated panel mount BNC female.

The AC adapter for this diode (and all of the detectors on the table) should get their power from a power strip which gets plugged into the rack with the whitening boards. The SHG oven, the Uniblitz shutter, and any cameras can get their power from another power strip if needed/wanted.

[Steve, Manasa]

To find noise source

1. Swapped the power cable of the PD and checked that it is connected to the right power source.

2. Changed the aluminium base of the post holding the diode so that the diode is floating

3. Grounded the table and the rack

4. Routed the cable on the other side of the beam tube to isolate it from other cables.

After all the above, we still found that shaking the cable was making TRY noisy.

I pulled out the PD whitening board to replace the 4 pin lemo connector with a bnc connector so that we can swap the cable with a new one. So there is no TRY right now.

The MON outputs of the Y end QPD whitening board were hot earlier today while pulling it out of the crate. After swapping the 4 pin lemo connector with an isolated panel mount bnc connector, I stuck the board back into the crate and this immediately kicked the ETMY suspension. Jenne and I went to the Y end to look at what was going on. We removed the board from the crate after smelling something burning. The MON output ports of the whitening board were super hot this time. There is no sign of any components melting on the board (comparing the board with its pictures that were taken earlier) and a tester board stuck into the crate lights up just fine.

So the back panel is still ok. We need to troubleshoot or replace the whitening board.

Edit, JCD:  The attached photos are from right after I replaced the "Rgain" resistor, elog 9823.  What they show is that it looks like some of the melting / burning may have already been happening before I pulled the board, and I just never noticed :(  In particular, look at the resistors on the main board above the blue "G" sticker.  There isn't a difference that I can tell between this photo from last week, and today's situation. 

 I re-plugged in the Yend shutter, and turned it on.

I tried some locking anyway tonight, even though we don't have TRY. 

The biggest conclusion is that I miss the auto-resonance-finding.  I've been roughly scanning the Y-ALS offset to find the POY zero crossing when I see the resonance on the test mass face cameras. 

The next-biggest conclusion, is that I can hold the PRFPMI close to resonance, using ALS for CARM and DARM.  I was trying to transition DARM to AS55, but I couldn't get the last bit of the way.  That is, I couldn't turn off the ALS control.  So, I think that AS55 wasn't actually holding DARM, until maybe the last moment or so.

Anyhow, here are some time series.  My average TRX value is around 40 counts, and POPDC is maybe 250 counts (just PRMI, POPDC is about 75 counts).  Obviously this is noisy as hell, but I'm not using any IR signals for the arms.  Near the end of this first time series, I am trying to switch to AS55 for DARM.

Zooming in, my real lockloss is due to PRCL oscillating at ~350 Hz:

However, I also saw ~25Hz peaks in CARM and DARM on the spectra starting to show up, and I see a ~25 Hz oscillation in DARM a few moments after the PRCL lockloss.  (Plot #2 is a zoom of the ~1.1 second mark on Plot #3.)

The locking parameters:

CARM:

Input:  Using the new CESAR matrix, -1*ALSX, +1*ALSY.  Beatnotes both move up in freq if temp sliders move up.

Servo: gain = 6, FMs 1, 2, 3, 5, 6, 7, 9 on.  Offset = 0 counts. 

Output = -1*MC2

DARM:

Input:  +1*ALSX, +1*ALSY

Servo:  gain = 4.  FMs 1, 2, 3, 5, 6, 7, 9 on.  Offset = 0 counts.

Output = -1*ETMX, +1*ETMY

PRCL:

Input:  +1*REFL33_I, Norm = +0.01*POPDC, sqrt engaged.

Servo:  acquisition easier with -0.04 or -0.06, less gain peaking at -0.02   FMs 4, 5 on; 2, 3, 6, 9 triggered with 0.5 sec delay.  Servo trigger = POPDC, up 100, down 10.  FM trigger = POPDC, up 300, down 20.

Output = +1*PRM

PRCL ASC off, PRM oplev on.

MICH:

Input:  +1*REFL33_Q, Norm = +0.01*POPDC, sqrt engaged.

Servo:  gain = 2, FMs 4, 5 on; 2, 3 triggered with 0.2 sec delay.  Servo trigger = POPDC, up 100, down 10.  FM trigger = POPDC, up 300, down 20.

Output = +0.5*BS, -0.2625*PRM

PDs:

REFL33 analog gain set to 30 dB for both I&Q.

AS55 set to 0 dB for both I&Q.  AS55 had DC normalization of 80 counts (which was the measured number for PRFPMI when TRX was about 0.1 count this evening)

This seems the ever best stability at the zero offset PRFPMI.

Can you look at REFLDC in this data stream too? How was it promising?

Here is 1 second of data, with REFLDC, POPDC and TRX:

Here is a zoom of the first 3 big peaks in TRX.  The weird jumps at the beginning of each TRX peak are due to the triggered switching between the Thorlabs trans PD and the QPD trans PD.  Clearly we need to work on their relative normalizations.  There are also little jumps after each peak as the triggering sends the signal back to the Thorlabs PD.

Here is a zoom of the single big peak about halfway through the 1 second of data:

And here is a zoom of the tail of that peak.  It looks to me like we want to start thinking about using REFL DC when our transmitted powers are around 2 counts.  We could do as soon as 1 count, but 2 is a little farther into the dip.

 maybe the tantalum caps on the daughter board power supply lines are blown? If so, replace with 35V+ ceramic.

I added ~1m of cable to the LO side of the REFL11 Demodulator, which brought its PRCL demod phase to about 8 degrees. According to my simulations, PRCL and CARM have the same angle (but opposite sign) at resonance. There seems to be a severe lack of SMA cables in the lab, so I didn't tune it to be any closer. Cos(8 degrees)=.99, so I think it should be fine to use it for the CARM servo, since none of the other signals are going to be nearly as big. I plugged analog REFL 11 I back into the CARM servo IN1. 

As for IN2, I threw together a temporary setup for using REFLDC as a complementary signal. I T'd off the REFLDC signal (which is the DC signal out of REFL55), and sent it into an SR560 to subtract an offset. The offset comes from a 1Hz-passive-pomona-box-low-passed C1:IOO-TT4_LR output, since there are 8 DAC channels set up for the nonexistent tip tilts 3 and 4 actively running. The output of the SR560 is sent to the CARM servo IN2. 

I adjusted the offset by turning on only IN2 in the CARM servo MEDM screen, and looking at the CM_SLOW signal in data viewer. I adjusted gains and such to get it to look just like REFLDC with the PRC locked. There was good coherence and no appreciable phase difference from DC out to some kHz, albeit a dip in coherence to about .8-.9 from ~40 to 300Hz, for some reason. (This included turning on the unWhite FM in the REFLDC filter bank)

If this signal turns out to be useful, it will be relatively straightforward to put together a little box that does the offset subtraction nicely, but this should do for our immediate needs.

Lastly, I hung up this plot in the control room to give us information about the DC values of different signals as the CARM offset changes. This is helpful to see what our CARM offset is, based on the transmission we se, when different signals start to have length dependence, where they start/stop being linear, etc. The TRX curve is scaled to a maximum of 600, REFLDC is normalized to input power = 1, and all the rest are arbitrarily scaled to fit on the plot. I've assumed 75ppm loss on all mirrors in my simulation (PRM, BS, 2xITM, 2xETM), mostly to get some realistic profile of REFLDC. 

Started working tonight. Don't know if anyone did anything to the Martian network or not, so its a mystery...

I also modified the script and SVN'd it. It now correctly takes your plot wants and adjust the linear/log of the axes accordingly

./SPAG4395A.py -i crocetta -A -v 4 --att=0 --start=1kHz --end=10MHz --bw=1kHz --plot --semiy

and also saves the plot as out.pdf

In the attached image I show the MC board TP1A output. The two peaks around 3.7 Mhz are the sideband beats we speak of. The lower one is proportional to the MC length/frequency mistmatch.

The main problem was a panel fixing bolt that caused the short circuits between power supply layers.
This burned the PCB and secondarily caused permanent short circuit between +15V/-15V/+5V layers.

Diagnosis

- The resistances between +15V, +5V, and -15V were low. The resistance between +15V and -15V is 13 Ohm.
  The one between +5V and -15V is 7Ohm. And the one between +15 and +5 is 19Ohm. So the situation is

                o -15V
                |
+15V o-(13 Ohm)-+-(9 Ohm)-o +5V

Even after removing all of the active components from the board, they remained the same.

- The tantalum caps were removed from the board and it was confirmed that they are not the cause of the issue.

- The panel was removed from the module for the component migration to a spare board (to be described in the other entry).
I found that the screw hole and the screw have burnt marks. The screw need an insulation tube to avoid short circuit.
The other screw was also bare. The spare board has the screws with the insulation tubes.

[ericq, Jenne, Zach]

We spent some time tonight trying to push our CARM locking further, to little avail. DARM/CARM loop oscillations kept sneaking up on us. We measured some MC2 motion -> REFL11 Transfer Functions to see if we could see CARM plant features; plots will come in the near future...

I went to WB and found the last spare module of D990399 revB. We need to thank Frank for his foresight.

The original (=broken) board had various modifications from this revB.
I had to check the schemaric diagram and the difference between the boards and migrate some of the SMD components from left to right.

Here is the deciphered features of the QPD whitening board:
- The input stage is a VGA amp (AD602). It has the internal input impedance of 100 Ohm. The series resister
  of 909 Ohm gives us 1/10 voltage division! It is more tricky as the QPD (D990272) has the output impedances of 50Ohm
  (for the both side of the differential out) and on resistance of MAX333A. So it could have been deviated by ~10% from the nominal.

- Variable gain control: The input has 1/10 voltage division. The gain is fixed at the unity. In total the gain of the variable control stage is 1/10.
  This gives us the gain range of +42dB/-22dB for +10V/-10V. The actual range is limited to be -10~30dB.

- Whitening stages. Each channel has two sets of the whitening path and the bypass path.
  They could be switched by binary control inputs but I permanently enabled the whitening by pulling the MAX333 control inputs to the ground.
  The whitening zero and pole are at 4.02Hz and 40.6Hz.

  Each bypass path has an additional cap of 220pF in parallel to 35.7kOhm (R101 and R103 for CH1), resulting in the pole at 20.2kHz.
  Each whitening paths had a 5.6nF cap (C53 and C64). This cap was replaced with 350pF, resulting in the move of the pole freq from 800Hz to 12.7kHz.

- There are two anti-aliasing stages which were designed for 2kHz sampling rate. They are identical sallen key 2nd-order LPFs with fc=766Hz and Q=0.74 (~ butterworth).
  As all of these caps were removed, they are just voltage followers now.

- The final stage (AD620) has the gain resister of 16.5k. The gain is 1+(49.4k/16.5k) = 3.99.

- The 4pin lemo connector (J8) was removed from the board. We instead installed an isolated BNC connector on the panel for the thorlabs PD serving as the high gain PD.

- There is a daughter board for the high gain PD. This seems to be the butterworth low pass filter with fc=~30kHz.
  The differential output of the daughter board is connected to pin 17 and 18 of J10 (S5 Out and Rtn).

- The input of the daughter board is differential (AD620). Therefore the LEMO connectros next to the BNC were wrapped with Kapton tapes for isolation.

Board test at the workbench.

- The test required two dual power supply as the unit requires +/-5V and +/-15V.

- The four channels were tested with the signal injection. 1kHz input yielded 20mVpp across the AD602 input. The output of the 1st whitening stage was
  60mVpp. This makes sense as the gain of the AD620 is -10dB (1/10 and 10dB). The output of the 2nd whitening stage was 600mVpp.
  Finally the output of the output stage was confirmed to be 2400mVpp. This was confirmed for four channels.

- The daughter board output was also checked. The gain is the unity and flat upto ~10kHz.

Board installation

- Jenne installed the module. This time there was no smoke.

Gain mystery

- It was not sure how the whitening gains have been given.

- The corresponding database entry was found in /cvs/cds/caltech/target/c1auxey/ETMYaux.db as

grecord(ao,"C1:ASC-QPDY_S1WhiteGain")
grecord(ao,"C1:ASC-QPDY_S2WhiteGain")
grecord(ao,"C1:ASC-QPDY_S3WhiteGain")
grecord(ao,"C1:ASC-QPDY_S4WhiteGain")

- The gains for S2-S4 were set to be 30. However, C1:ASC-QPDY_S1WhiteGain was set to be 8.62068.
And it was not writable.

- After some investigation, it was found that the database was wrong. The DAC channel was changed from S100 to S0.
The corrected entry is shown here.

grecord(ao,"C1:ASC-QPDY_S1WhiteGain")
{
        field(DESC,"Whitening gain for QPDY Seg 1")
        field(DTYP,"VMIVME-4116")
        field(OUT,"#C0 S0 @")
        field(PREC,"1")
        field(EGUF,"42")
        field(EGUL,"-22")
        field(EGU,"dB")
        field(LINR,"LINEAR")
        field(DRVH,"30")
        field(DRVL,"-10")
        field(HOPR,"30")
        field(LOPR,"-10")
}

- Once c1auxey was rebooted, the S1 whitening gain became writable. Now all of the channels were set to be +30dB (max).