I added ~1m of cable to the LO side of the REFL11 Demodulator, which brought its PRCL demod phase to about 8 degrees. According to my simulations, PRCL and CARM have the same angle (but opposite sign) at resonance. There seems to be a severe lack of SMA cables in the lab, so I didn't tune it to be any closer. Cos(8 degrees)=.99, so I think it should be fine to use it for the CARM servo, since none of the other signals are going to be nearly as big. I plugged analog REFL 11 I back into the CARM servo IN1. 

As for IN2, I threw together a temporary setup for using REFLDC as a complementary signal. I T'd off the REFLDC signal (which is the DC signal out of REFL55), and sent it into an SR560 to subtract an offset. The offset comes from a 1Hz-passive-pomona-box-low-passed C1:IOO-TT4_LR output, since there are 8 DAC channels set up for the nonexistent tip tilts 3 and 4 actively running. The output of the SR560 is sent to the CARM servo IN2. 

I adjusted the offset by turning on only IN2 in the CARM servo MEDM screen, and looking at the CM_SLOW signal in data viewer. I adjusted gains and such to get it to look just like REFLDC with the PRC locked. There was good coherence and no appreciable phase difference from DC out to some kHz, albeit a dip in coherence to about .8-.9 from ~40 to 300Hz, for some reason. (This included turning on the unWhite FM in the REFLDC filter bank)

If this signal turns out to be useful, it will be relatively straightforward to put together a little box that does the offset subtraction nicely, but this should do for our immediate needs.

Lastly, I hung up this plot in the control room to give us information about the DC values of different signals as the CARM offset changes. This is helpful to see what our CARM offset is, based on the transmission we se, when different signals start to have length dependence, where they start/stop being linear, etc. The TRX curve is scaled to a maximum of 600, REFLDC is normalized to input power = 1, and all the rest are arbitrarily scaled to fit on the plot. I've assumed 75ppm loss on all mirrors in my simulation (PRM, BS, 2xITM, 2xETM), mostly to get some realistic profile of REFLDC. 

[ericq, Jenne, Zach]

We spent some time tonight trying to push our CARM locking further, to little avail. DARM/CARM loop oscillations kept sneaking up on us. We measured some MC2 motion -> REFL11 Transfer Functions to see if we could see CARM plant features; plots will come in the near future...

I should've included this in my Thursday night ELOG... That evening, I aligned the mode cleaner with reasonable MC1/3 spot positions, and the MC2 spots very close to centered, and recentered the WFS and MC2 Trans QPDs. The mode cleaner held up very well over the course of that evening, even when actuating CARM on MC2 with WFS engaged (which previously wasn't very stable when the WFS weren't well aligned).

Inspired by a comment by Koji the other day, I spent some time yesterday and today working on locking a (very lossy) power recycled Y-arm. ITMX was misaligned, to save myself the headache of dealing with ITMY getting a sign flip and ITMX staying the same when the arm resonates. 

My main goal was to achieve high bandwidth control with the analog CARM servo. 

TL,DR: Transisitoned 90% to REFLDC through CM_SLOW at TRY = 2.1 twice. Couldn't make it all the way over. 

PRCL settings:

• Input: REFL165 I.
• Actuate on PRM +1
• Control: G=-.32 (~100Hz UGF); Acq on FM 4,5; Trig 1,2,3,6,9 (I modified the +10dB in FM1 to a 1kHz ELP)
• Trig: POP 110 I: 1.5 up, 0 down (max was around 4 counts, very weak PRC!)

The PRC was very stable in this configuration, which doesn't surprise me due to its simplicity. I was honestly a little surprised there was enough light to lock on 3f. REFL33 didn't work. 

My efforts to bring the Y-arm into lock were very similar to the CARM procedure we've been using recently. (Which is the motivation for this exercise)

At first I was actuating on ETMY, and got to the point where I wanted to start bringing in the CARM servo slow output, then realized that I didn't want to actuate both on the ETM and MC AO. (Maybe this would be doable, but in the end, not what I'm interested in learning about in terms of overlap with CARM locking)

From then on, I only actuated YARM on MC2. (Heads up, my lock-losses will show up in the trends of the MC2 Trans addition to the WFS.)

Transitioning the arm to SqrtInv TRY control was just as straightforward as it has been for CARM. However, engaging the LSCLock FM (FM4), would sometimes work beautifully, and sometimes kick the hell out of MC2. Keeping an eye on the error signal spectrum and UGF gave no indication which outcome would happen. Once FM4 could be engaged, the transmitted power was very stable. Without FM4, reducing the offset didn't get very far without losing lock. 

I tried a few times to bring in CM_slow (set to just IN2, i.e. offset adjusted REFLDC), at arbitrary arm powers, with little success. I didn't know how much arm power to expect at resonance, and thus didn't really know where on the line width I was.

I knew I was mostly outside of the linear regime of the PDH signals, since, even though I had good coherence between, say, REFL11 I and SqrtInvTry, with an ETMY excitation on; when I would turn TRY normalization on/off, I would see the sign of the TF change. 

I then realized that I could actively keep an eye on the trend of POY11, to see when I got to the PDH "hump", which is where REFLDC starts being usable, and SqrtInv is reaching its limit. 

This brought me to a YARM offset of .115, with a steady TRY of about 2.1. I adjusted the analog offset of the REFLDC input to the CARM board, and the digital gain of the CMSLOW input filter to get 1:1 correspondence between CMSLOW and the SQRTINVY channels. Their spectra were neigh identical, with CMSLOW having slightly more high frequency noise. 

I started stepping SQRTINV down by .1, and upping CMSLOW by .1. This shifted the offset around, so I opted for taking away gain before bringing it back, because I didn't want to get so close to resonance that SQRTINV would freak out. I got to .1*SQRTINVY + .9*CMSLOW, and lost lock. TRY was getting noisier as I made the transition. 

I'm not sure what exactly was the reason for failure. I'm going to go back over some of the data to try to get an idea.. Maybe I should've loosened up some of the gain/boosts during the transition. 

So, no great success story yet, but this configuration is a lot simpler than the full PRFPMI, and I feel that I should soon be able to get it fully controlled, and figure out a systematic way to make the digital to analog transition for this PRFP cavity, and thus have a much more informed basis for doing the same for CARM control. 

Just to be clear, the normal POY signals are not currently present, so the restore POY script will not result in the arm locking. POY11_I is turned off, POY11_Q is the output of the CM board, which can be used to lock the arm, as we did tonight. 

The POY digital demos angle went -56 -> 90, to get all of POY11_Q_IN1 to POY11_I_ERR

Miscellaneous things:

• Right now, the cable from CM board ->MC board is a BNC. There appeared to be a differential 2-pin lemo hanging around for this purpose, but it didn't seem to be transmitting the signal. However, we will want something better than a BNC to keep this signal clean. 
• I took SR785 TFs of the CM board from IN to the slow and fast outs. They looked reasonable, and will be posted in time. 
• We enabled the 79:1.6k filter in the CM screen (though it is unclear if these are the actual values...), and put in its inverse in the digital path. I.e. we only want this shape in the AO path, to give it 1/f shape in the vicinity of the crossover. This is only necessary in the uncoupled cavity case. 

 (Edited this post; Forgot to account for the FMs other than 4 and 5... it now agrees better!)

I did some quick MATLAB simulation of the relevant loops to try and understand what was going on. I put the digital UGF around 200Hz, and then brought in the AO path with both signs. 

In these plots, blue is digital only, green is AO+digital with the crossover happening at the UGF, and red is the AO gain set to five times of what it was in the green curve. 

Based on the phase curves in the loop measurements, I would be inclined to say the pink -AO case corresponds to the opposite sign plot, and the +AO case to the same sign plot. 

This correspondence also holds for the appearance of the peaks in the noise curves, the Opposite sign case has a dip in loop gain at ~50Hz (pink curve, -AO), same sign around ~30Hz (brown curve, +AO). 

However, both of these look like they become unstable at some point in the transition! This agrees with our experience last night...

I'll fiddle around and try to come up with some compensating digital filter that will make the Opposite sign scenario work. 

The MATLAB code I used to make these plots is attached. 

The PMC has been unlocked for ~23 hours. FSS slow was at ~-1.5 V. I zeroed it, and relocked the PMC, transmission is ~0.81V. MC with WFS came back fine.

 [Rana, ericq]

Today, we got a ~2kHz bandwidth lock of the YARM with the AO path. We weren't able to turn any boosts on, due to POY noise. 

Rana and Koji have written scripts (/scripts/PRFPMI/cm_step and cm_down) that work very reliably. 

Here is an OLTF. (Violin filter was off, the crap around 600Hz goes away with them on)

My MATLAB modeling was useful is predicting the features of the loop shape, and the dependence on AO gain/crossover. Still, I need to check it out, because there is nonzero discrepancy between reality and my model (this may be hiding in the non flat MC AO response, i.e. the bump at ~35kHz. Alternatively, the crossover frequency is a free parameter...)

In any case, we have confidence that the CM board is mostly working predictably. We presume that our current obstacle is the very noisy nature of POY, and thus it's not worth spending more time in this configuration. 

Upcoming plans:

• Use the CM board to control the Y arm coupled with the PRM. ("PRY"?)
• Determine the game plane for high BW control of CARM. 

Next steps:

• Check CM board boosts turn on politely (Transients, TFs)
• Use fast spectrum analyzer to check MC loop gain out to a few MHz. (The bump in the tens of kHz should be fixed / moved higher)
• Think about noise performance of, say, REFLDC, ASDC, RF AS signals, etc. in the PRY case, figure out which one to use first. 
• We may want to first focus on directly locking the arm on an RF signal, figure out gains etc. and then figure out how to do DC->RF handoff nicely, or if high bandwidth DC signal control is even feasible. 

RXA: we should also use AS45 instead of POY11. It has better SNR and I think our whole problem is too little light on POY.

 I took a look at the MC OLTF and AO path TFs with the fast agilent analyzer. 

I played with the relative gain of the EOM and PZT, but couldn't really change the MC OLTF shape much without making the PC Drive RMS angry. 

However, it turns out we have plenty of phase headroom to up the MC UGF from ~100kHz to ~180, with about 40 degrees of phase margin and ~7dB of gain margin. As I write this, PC drive RMS is around 1.1, and FSS Fast at 5.6, so I think the extra gain is fine for now. 

This pushes up and smoothens out the gain peaking in the AO path; see this figure:

(why does ELOG hate my python plots?! argggg)

Rana's rule of thumb was "We need at least +3dB MC loop gain at our CM servo UGF," so it looks like high tens of kHz bandwidth may be doable from the AO standpoint.

RXA: No, no, no, no, no, noooo. Rana said we need a gain of 3-10 at the CM UGF, not +3 dB.

 I've checked that the 2pin lemo connector that was run some time ago from the LSC rack to the MC board does indeed transmit signals. To try and evaluate its suitability I did the following:

• Generated a 5mVpp 1.3kHz signal with an SR785 and fed that into CM board In1, all boosts off, 0dB AO gain. 
• Both BNC and LEMO connected to CM servo out
• One of BNC or LEMO connected to IN2 of MC servo, input gain of 30dB but disabled, OUT2 switched to AO and fed to Agilent spectrum analyzer. 
• Terminated MC IN2 for comparison. 

No real difference was seen between the two cases. The signal peak was the same height, width. 60Hz and harmonics were of the same amplitude. Here are the spectra out to 200k, they are very similar. 

Mode cleaner was locked during this whole thing. This may interfere with the measurement, but is similar to the use case for the AO path. If ground loop / spurious noise issues keep occurring, it will be worthwhile to examine the noise of the CM and MC servo paths, inputs and outputs more carefully. 

On pianosa: The ezca.Ezca class somehow initializes with its prefix set to "C1:", even though the docstring says the default is None. This makes existing scripts act wonky, because they're looking for channels like "C1:C1:FO-BLAH".

In ligo/apps/linux-x86_64, I ran ln -sfn cdsutils-old cdsutils to get the old version back for now, so I don't have to edit all of our up/down scripts.

Also, Chiara can't find the epics package when I try to load Ezca. It exists in '/usr/lib/pymodules/python2.6/epics/__init__.pyc' on pianosa, but there is no corresponding 2.7 folder on chiara.

 Still no success tonight

• We took CARM OLTFs at various CARM offsets and could clearly see the peak in the optical TF (in once case ~2.5kHz), which gave us an indication of our offset (~200pm)
• REFLDC effectively sees the same plant TF as the transmission signals plus a zero at ~110 Hz, at all offsets under 1nm, from my simulations; this pushes up the optical resonance and causes a loop instability when we try to handoff. 
• We need to make the CARM OLTF steeper to suppress this instability, but also to make a good crossover with the AO path, which otherwise has too similar of a slope around the UGF, as we saw with our one arm test. 
• We're thinking of trying to turn the AO path on with REFLDC while keeping the arms on SQRTINV signals. This may be tricky, but if we can get the loop bandwidth above the optical peak, it'll be a lot easier to deal with, and transfer digital control to REFLDC as well. 

 I have redone the SPSR785 (spectrum measurement) and TFSR785 (TF measurement) commands in scripts/general/netgpibdata. This was mostly motivated by my frustration with typing out either a ton of command line arguments, or rooting around in the script itself; I'd rather just have a static file where I define the measurement, and can keep track of easier. 

They currently take one argument: a parameter file where all the measurement details are specified. (i.e. IP address, frequencies, etc.) There are a few template files in the same directory that they use as default. (Such as TFSR785template.yml)

If you call the functions with the option '--template', it will copy a template file into your working directory for you to modify as you wish. "SPSR785 -h" gives you some information as well (currently minimal, but I'll be adding more)

In the parameter file, you can also ask for the data to be plotted (and saved as pdf) when the measurement is finished. In SPSR785, and soon TFSR785, you can specify a directory where the script will look for reference traces to plot along with the results, presuming they were taken with the same measurement parameters and have the same filename stem. 

I've tested both on Pianosa, and they seem to work as expected. 

Todo:

• Add support for modifying some parameters at the command line 
• Extend to the Agilent analyzer
• Maybe the analyzer settings written to the output file should be verified by GPIB query, instead of writing out the intended settings. (I've never seen them go wrong, though)
• Make sure that the analyzer has PSD units off when taking a TF. (Thought I could use resetSR785 for this, but there's some funkiness happening with that script currently.)
• Possibly unify into one script that sees what kind of analyzer you're requesting, and then passes of to the device/measurement type specific script, so we don't have to remember many commands. 

Comments, criticism, and requests are very welcome. 

(P.S. all the random measurement files and plots that were in netgpibdata are now in netgpibdata/junk. I feel like this isn't really a good place to be keeping data. Old versions of the scripts I changed are in netgpibdata/oldScripts)

 It's taken a lot of trial and error, but I've found a path through MATLAB loops that seems like it may be stable at all points.

CAVEAT: This doesn't give any indication as to why we weren't able to turn up the AO gain more last night, as far as I can tell, so it's not all good news. 

However, it's still ok to at least have a plan that works in simulation... 

Based on the location of the optical resonance peak in the CARM plant, we estimate our CARM offset to be 200pm. I haven't simulated TFs there exactly, but do have 100pm and 300pm TFs. This procedure works in MATLAB starting at either, though 100pm is a little nicer than 300. MATLAB data and code is attached in a zip. 

The steps below correspond to the attached figures: Bode plots and step response of the Loop at each step. 

0. [Not Plotted] DCTrans sensing, MCL actuation on CARM. FMs1,2,3,5,8; UGF = 120. (DARM not considered at all)

1. AO path just turned on. Crossover with MCL path ~ 3.5kHz. 
2. AO gain increased. Crossover ~ 500 Hz. There are now multiple UGFs! Handling all of these in a stable manner is tricky. 
3. AO gain increased. Crossover = 150Hz. [No simulations with a higher crossover survived the next steps]
4. Compensation filter applied to MCL path; 1 real Zero at 105Hz and a pole at 1k. From a TF point of view, this is sort of like switching to REFLDC, but the SNR at low frequencies is probably better in TR signals at this point. 
5. CARM offset reduced to 30pm. (This smoothens out the optical plant resonance.)
6. Overall gain increased by factor of 3. There is now just one UGF at a few kHz, above the optical resonance. From here, gain can be further increased, boosts can go on, offset can go way down. In reality, we should switch to a single error signal once we're back to one UGF, and go from there. 

#4 Seems like the most sticky part. While both sides of this look stable as far as I can tell. I feel that flipping from the red phase curve to the teal might not actually be ok, since they are on either side of the bad phase of 0 degrees. It isn't immediately evident to me how to easily model the transitions between steps, rather than just the stability of of each step in the steady state. 

Den and I spent some time with the interferometer last night with hopes of bringing in the AO path, but were stymied by the (re)occurrence of the anomalously high low frequency motion of the Xarm, as seen by fluctuations of TRX from .9 to .2 while "held" on resonance.

Jenne reported that they weren't seeing it earlier in the evening, and then it started again when I showed up. Holding the arms on IR, we could see a fair amount of excess low frequency noise in the BEATX_FINE_PHASE_OUT_HZ channel, as compared to BEATY, bringing its RMS to 5 times that of the Y arm. From the shape of the excess noise (broad slope from DC to tens of Hz), Rana suspected air currents and/or scattering effects being the culprit.

Den poked around a bit on the PSL table, which didn't really change much. He then went down to the X end table to inspect the table, and while he was there, I noticed the noise go down to being in line with the Yarm. I joined him at the end, and we found the beat phase noise in the frequency region of concern to be hugely sensitive to tapping on the enclosure, air current, etc. There is also a ton of green light everywhere, and multiple spots of green light around the green refl PD.

At that point however, the quiescent noise was acceptable (TRX fluctuations of <.2), so we went back to the control room to try to lock. Unfortunately, after a few attempts, the noise was back. At this point, we went home. The layout of the end table likely needs some attention to try and minimize our susceptibility to excess scatter effects. 

So, I really should have done this as soon as Manasa measured the arm lengths... I've updated my MIST model with the real arm lengths, but still am using assumed identical losses of 75ppm on each mirror. (I've tried measuring the arm losses for real, but got numbers in the hundreds of ppms, so I need to reexamine things...) 

Here's a simulation of the fields in a perfectly locked PRC when CARM is swept (Normalized to input power = 1). 

More importantly, here's the latest simulation of MICH vs. PRCL demodulation angle separation in the 3F signals. It seems that we may be getting burned by using REFL33 for the PRC lock. REFL165, on the other hand looks much more robust. We should try this out. 

(Some of my previous simulations incorrectly implemented MICH excitations; I only moved the ITMS, not the ETMS along with them, so some other stuff slipped in... )

 Here's the angles of MICH and PRCL from the my earlier plot by themselves; this shows that the individual demod angles in REFL165 aren't changing much either. 

I fixed a bug in the SUS_SINGLE screen, where the total YAW output was incorrectly displayed (TO_COIL_3_1 instead of TO_COIL_1_3). I noticed this by seeing that the yaw bias slider had no effect on the number that claimed to be the yaw sum. The first time I did this, I accidently changed the screen size a bit which smushed things together, but that's fixed now.

I committed it to the svn, along with some uncommitted changed to the oplev servo screen.

PMC has been unlocked for ~4hrs, not sure why. It's servo gain was down at -10dB...

Relocked with transmission of .76V, MC locks fine with WFS, transmission of 15.5k.

I came in and PMC transmission was at 0.5V, and ETMX was swinging around a lot, (LSC mode was on). 

Turning off oplevs let ETMX calm down. I realigned the PMC to 0.82V. 

MC wouldn't relock, it looked misaligned in pitch and yaw on MC camera.

I've touched the alignment, and gotten the reflection below 0.5, but it unlocks periodically, spot positions aren't great, and turning on WFS throws it out of alignment. ughhhhh

I'll put more detail in the morning, but I was able to get the PRM/ITMY/ETMY coupled cavities locked with 32kHz bandwidth using the AO path. (However, this is a pretty low-finesse situation, since the BS is dumping so much power out of the PRC. Full buildup is only 3 or 4 times the single arm power)

Since our ALS is better than it was a month ago when I last played with this, I was able to hop straight from ALS to REFL11 I on resonance, with the PRY locked on 3f.

Here are some quick OLTF plots I took along the way.

I'm using this configuration to validate my loop modelling for the full double arm case. Right off the bat, this tells me that the "minus" polarity on the CM servo is the correct one. I didn't use REFLDC at all tonight, but I figure I can check it out by doing the transition backwards, so to speak.

Sorry, the X arm is completely misaligned. This is the configuration I first tried in ELOG 9859, that is: a PRM->ITMY recycling cavity and ITMY->ETMY arm cavity. ITMX is completely misaligned, so the BS is dumping much of the recycling cavity light out, which is why I wrote "low finesse." This is the first time I've used REFL11 to control any of our cavities, though.

ELOG is back up and running after last Friday's disk-crash-a-thon. SVN is still a work in progress. Jenne and I are now restarting computers and such.

In all of the fstabs, we're using chiara's IP instead of name, so that if the nameserver part isn't working, we can still get the NFS mounts.

On control room computers, we mount the NFS through /etc/fstab having lines like:

192.168.113.104:/home/cds /cvs/cds nfs rw,bg 0 0
fb:/frames /frames nfs ro,bg 0 0

Then, things like /cvs/cds/foo are locally symlinked to /opt/foo

For the diskless machines, we edited the files in /diskless/root. On FB, /diskless/root/etc/fstab becomes

master:/diskless/root                   /         nfs     sync,hard,intr,rw,nolock,rsize=8192,wsize=8192    0 0
master:/usr                             /usr      nfs     sync,hard,intr,ro,nolock,rsize=8192,wsize=8192    0 0
master:/home                            /home     nfs     sync,hard,intr,rw,nolock,rsize=8192,wsize=8192    0 0
none                                    /proc     proc    defaults          0 0
none                                    /var/log        tmpfs   size=100m,rw    0 0
none                                    /var/lib/init.d tmpfs   size=100m,rw    0 0
none                                    /dev/pts        devpts  rw,nosuid,noexec,relatime,gid=5,mode=620        0 0
none                                    /sys            sysfs   defaults        0 0
master:/opt                             /opt      nfs    async,hard,intr,rw,nolock  0 0
192.168.113.104:/home/cds/rtcds         /opt/rtcds      nfs     nolock  0 0
192.168.113.104:/home/cds/rtapps        /opt/rtapps     nfs     nolock  0 0

("master" is defined in /diskless/root/etc/hosts to be 192.168.113.202, which is fb's IP)

and /diskless/root/etc/resolv.conf becomes:

search martian

nameserver 192.168.113.104 #Chiara

I'm kind of confused by the current auto locker situation. Somebody renamed the script from autolockMCmain40m to AutoLockMC.csh and did not update the crontab with the new filename. 

Furthermore it doesn't seem like  scripto_cron,a script which keeps the auto locker alive, runs ok on ottavia. When I run this on the command line, it claims that the process is already running, and returns some bunk PID that doesn't correspond to any running process. The script has a line stating setenv OSTYPE solaris , so maybe there's something funky going on with it's pgrep-ing or other command parsing.   

Lastly, if I try running AutoLockMC.csh directly on ottavia, I get a bunch of complaints about pmath and pezcabit not being found. 

 FB is acting strange. When ssh-ing in, certain commands cause an inescapable hang, which can't be ctrl-c'd out of. Telling it to reboot does nothing. This kind of situation was seen by me before, when we were getting all the front ends back, I eventually hard rebooted it, hoping it was a one time thing. Guess it's not. 

Looking at the dmesg output, daqd seems to be segfault-ing all over the place. This may be related... Here are some examples:

451314.730502] daqd[17339]: segfault at 7ff589ae3b30 ip 00007ff589ae3b30 sp 00007ff49931dfb8 error 15 in libmyriexpress.so[7ff589ae3000+1000]

[530516.313238] daqd[18442] general protection ip:7f3f2ce73a6c sp:7f3e29949d50 error:0

[530516.313250] daqd[18420] general protection ip:7f3f2ce73a6c sp:7f3e2a19fd50 error:0 in libc-2.10.1.so[7f3f2ce3f000+14c000]

[530516.313262]  in libc-2.10.1.so[7f3f2ce3f000+14c000]

[530516.327083] daqd[18412]: segfault at 3b04c9cd0 ip 00007f3f2ce73a6c sp 00007f3e2a4a7d50 error 4 in libc-2.10.1.so[7f3f2ce3f000+14c000]

[537695.364481] daqd[18489]: segfault at 12dbbcae0 ip 00007fa35a3b8a0a sp 00007fa298381af0 error 6 in libmyriexpress.so[7fa35a399000+28000]

[577316.821618] daqd[18758]: segfault at 7f5c4d3e9b30 ip 00007f5c4d3e9b30 sp 00007f5b5cc23fb8 error 15 in libmyriexpress.so[7f5c4d3e9000+1000]

I'm not inclined to go reboot it right now, but not sure how to address these problems...

Some time ago, Rana changed the PS1 prompt codes on the control room computers. However, the exit codes of commands weren't being displayed, and there was some lingering color changing after the line. Hence, I changed it to look like this:

PS1='\[\033[0;35m\]\u'
PS1="$PS1\[\033[0;30m\]@"
PS1="$PS1\[\033[0;33m\]\h"
PS1="$PS1\[\033[0;97m\]|"
PS1="$PS1\[\033[0;92m\]\W"                                                 
PS1="$PS1\[\033[0;31m\] \${?##0}"
PS1="$PS1\[\033[0;97m\]>\[\033[0m\] "

The \${?##0} means: display the exit code if it is not zero (which means success). Thus, it only displays the exit code when its something other than what is expected.

This is still happening. Specifically: on all of the control room computers, calls to caget display the result immediately, but then hang for five seconds (consistently five). We had also seen a situation where calls hang indefinitely on ottavia/pianosa, but a reboot "fixes" this.

Some observations:

• Front end machines and the FB have proper caget/caput response times.
• Control room machines have some odd ping behavior when targeting frontends/FB; namely the ping times themselves are ok, but each ping line takes quite some time to show up, which made us think that there is odd network routing issue happening with some network switch. 
• Front ends and FB get epics from /opt/rtapps, whereas control room machines get epics from /ligo/apps, which has different contents. (Is this for Gentoo vs. Ubuntu? I don't really get why this is the case...). This means different environment setting scripts to be called, so maybe the control room machines are misconfigured in some way for the new name server?

I poked around the network settings on all of these machines, but everything seemed reasonable. Nothing was changed. Rossa and Pianosa have their network settings done through some Ubuntu GUI, but I don't know where the settings are written. I had expected their settings to be in /etc/network/interfaces; maybe we should change this to be consistent with other machines, and easier to administrate via the terminal. 

Despite all this, ezcaread is fine.

I set up a raspberry pi on the martian network, to be hooked up to a frequency counter for tracking ALS beatnotes. 

The instructions at https://wiki-40m.ligo.caltech.edu/Martian_Host_Table are outdated, the name server configuration is now at /etc/bind/zones/martian.db, I need to remember to update the wiki soon. 

In any case, the raspberry pi is called "domenica," is found at 192.168.113.107, and has the standard controls user, with /cvs/cds mounted in the same way as the control room machines. 

Once I'm comfortable with the configuration of the pi, I'm going to take an image of the SD card that serves as its hard drive, so that we can just image new cards for future raspberry pis on the martian network if we ever want them. 

I think I've fixed the caget/caput issue. Rana's observation that pinging the IP directly was faster than pinging the hostname set me on a path of googling which informed making the following changes to the DNS setup on chiara (specifically, informed by this thread: http://www.dslreports.com/forum/r11836974-BIND-slow-to-reply-over-LAN-Solved)

/etc/bind/named.conf.local has these lines:

The first zone command links hostnames like c1lsc to an IP like 192.168.113.62, but apparently in the second, we need to do the inverse. So, for each line in martian.db like

c1lsc           A       192.168.113.62

I added a line in rev.113.168.192.in-addr.arpa like so:

62 IN PTR c1lsc.martian

This seems kind of silly, but now if you do the host command from a workstation, it can find the hostname associated with an IP. 

[At this point, note that we have a bunch of duplicate entries in https://wiki-40m.ligo.caltech.edu/Martian_Host_Table  with these scipe## hostnames. What are these for?]

Manasa let me know that the ELOG was down, and that she used the normal restart procedure, but then all entries were gone. 

This is because the ELOG has been moved on nodus, so going to the old place and running the restart script starts up the old dysfunctional ELOG installation. 

The proper place to restart the ELOG is now nodus:/export/home/elog/start-elog.csh

I'm updating the relevant 40m wiki page now. 

This afternoon, I wanted to start the nominal alignment/adjustment steps for evening time locking, but got sucked into CDS frustrations. 

Primary symptom: TRX and TRY signals were not making it from C1:SUS-ETMX_TR[X,Y]_OUT to C1:LSC-TR[X,Y]_IN1. Various RFM bits were red on the CDS status page. 

 Secondary symptom: ITMX was randomly getting a good sized kick for no apparent reason. I still don't know what was behind this. 

First fix attempt: run sudo ntpdate -b -s -u pool.ntp.org on c1sus and c1lsc front ends, to see if NTP issues were responsible. No result.

Second fix attempt: Restart c1lsc, c1sus and c1rfm models. No change

Next fix attempt: Restart c1lsc and c1sus frontend machines. c1lsc models come back, c1sus models fail to sync / time out/ dmesg has some weird message about ADC channel hopping. At this point, c1ioo, c1iscey and c1iscex all have their models stop working due to sync problems. 

I then ran the above ntp command on all front ends and the FB, and restarted everyone's models (except c1lsc, who stayed working from here on out) which didn't change anything. I command-line rebooted all front ends (except c1lsc) and the FB (which had some dmesg messages about daqd segfaulting, but daqd issues weren't the problem). Still nothing. 

Finally, Koji came along and relieved me from my agony by hard rebooting all of the front ends; pulling out their power cables and seeing the life in their lights fade away... He did this first with the end station machines (c1iscey and c1iscex), and we saw them come back up perfectly happy, and then c1ioo and c1sus followed. At this point, all models came back; green RFM bits abounding, and TR[X,Y] signals propagating through as desired. 

Then, we tried turning the damping/watchdogs back on, which for some strange reason started shaking the hell out of everyone except the ETMs and ITMX. We restarted c1sus and c1mcs, and then damping worked again. Maybe a bad BURT restore was to blame?

At this point, all models were happy, all optics were damped, mode cleaner + WFS locked happily, but no beams were to be seen in the IFO 

The Yarm green would lock fine though, so tip-tilt alignment is probably to blame. I then left the interferometer to Jenne and Koji. 

 We had fiddled with the scripts/general/scripto_cron script to try and get the MC auto locker working on ottavia, but in doing so broke op340m's reliance on it to run it's cron jobs, like FSSSlowServo.

I've reverted scripto_cron to its original state, and the FSS slow servo starts up again.  

However, scripts like this that we want to always have on seem to be a better fit, to me, for the init system, like we do with daqd and nds on the FB. op340m's inittab looks different than what I'm used to, so I'm not making any changes; this is just a thought. 

MC autolocker is still being ran from an Ottavia GUI terminal; I'll try to get it consistently running on megatron, as suggested in  ELOG10039, now that caget/caput issues seem to be sorted. 

Addendum: I've changed the MC auto locker script to have megatron as its host. Haven't yet gotten it to run automatically; it's running in a detached tmux terminal. I'll finish it up tomorrow. 

IR Alignment of the IFO is recovered! Green alignment could use some attention. 

Arms and PRC hold well, powers are within normal ranges. (TR[X,Y] >.9, POP110I > 400)

Details:

• Starting from where Koji had brought things (i.e. beams on REFL, AS), I gradually stepped the PRM back to where I had a PRC lock right before the bootfest, using SUSPIT_IN and SUSYAW IN, while stepping the TT pointing (arbitrarily shifting TT1 and TT2) to keep the REFL camera spot centered. 
• Green was locked fairly well to the Yarm (GTRY = 0.8), so I knew it supported a mode. I misaligned ITMX to keep the Xarm out of the way. 
• Adjusting TT1 and TT2, I was able to get some light on the POP camera, while keeping REFL spot visible with small adjustments to PRM
• I set up the PRM to set up some PRM-ITMY retroreflection, and saw tiny flashes in TRY. 
• From here, I wandered around with the tip-tilts to try and get better arm pointing, while tweaking PRM for the REFL spot and retroflection, and BS for the AS spot. I got this to TRY flashes of ~0.3

Then, Jenne took over, worked some alignment magic, and did the hard part of getting the Yarm locked and ASS'd. 

• I then took back over and got the Xarm locked and ASS'd.
• I saved relevant mirror positions, and set up the PRMI, got it locked, saved PRM position. 
• Jiggled the BS/PRM oplev HeNe power connector, it turns on now, oplevs are working. 
• Similar to TT1, TT2 gains for PIT and YAW are now 300. Strictly speaking, they don't have to be, but PIT would be very nearly railed, and I changed YAW so that all four gains would be consistent.

Caveats:

Green no longer locks to 00 on the Y-arm, and the X-arm green transmission isn't the best despite PZT fiddling (~.6). Also, when green is locked to the Xarm, I see a distinct circular spot on the GTRY camera, with Y green and PSL green shutters closed. 

While Jenne used Yarm ASS successfully, when I run it now, it slowly pushes things out of alignment, and two of the traces (YARM_ETM_YAW_L_DEMOD_I_OUTPUT, and the same for ITM) have a reasonable constant offset that doesn't move away. 

 Just a quick update on the status of the auto locker:

The auto locker now runs and respawns automatically on megatron, through ubuntu's "upstart" init system, instead of cron. 

• The autolocker script itself is:/opt/rtcds/caltech/c1/scripts/MC/AutoLockMC.csh
• The startup script called by the upstart system is: /opt/rtcds/caltech/c1/scripts/MC/AutoLockMC.init
• The upstart configuration is: /etc/init/MCautolocker.conf
• The auto locker is started by executing this command on megatron: sudo initctl start MCautolocker

This was pretty simple to set up, once I figured out how to provide the necessary environment variables in AutoLockMC.init 

Koji has provided a 2TB USB3 external hard drive that will get daily backups of chiara:/home/cds, the idea being that if the internal HD at /dev/sdb1 fails, we can physically open the external up, and swap the hard drive into chiara. 

We're running an rsync job on it right now, which should be done in a few hours. (USB3 is fast!)

I've also written a backup script at scripts/backup/rsync_chiara.backup which keeps its books in scripts/backup/rsync_chiara.backup.log 

I'm adding a entry to the root crontab on chiara to execute the script every day at 7am. 

 I had some syntax errors in the script that prevented the script from doing the right thing. The backup is now up to date, and the cronjob should work. 

The autolocker claimed it was running and blinking, but not doing anything (i.e. lock bit was not updating and no switches or sliders being touched)

After stopping and starting it a number of times, it began working again, through no real changes of my own. I'm a little mystified as to what the problem was... keep an eye out.

Using these numbers for both arms (Modulation takes away .2*.25 = 5% power, mode matching takes away 7% after that), I get the following from my data from March:

Xarm loss is 561.19 +/- 14.57 ppm

Yarm loss is 130.67 +/- 18.97 ppm

Obviously, the Xarm number looks very fishy, but its behavior was qualitatively very different when I took the data. ASDC would change from ~0.298 to ~0.306 when the Yarm was locked vs. misaligned, whereas the xarm numbers were .240 to .275. 

In any case, I'll do the measurement again tomorrow, being careful with offsets and alignment; it won't take too long. 

 ITMY oplev was nearly clipping in yaw, causing wonky behavior (POY lock popping in and out frequently). I recentered it and the arm is locking fine now. 

 [Koji, ericq]

We were working on getting back into the locking groove tonight.

The POP2F and REFL3F demod angles needed some tuning to lock the PRC reliably. The green alignments were mostly fine, the X end PZT ASS works reasonably well. Suspensions, especially the ITMs, seemed to be drifting a fair deal; today was fairly hot out, I guess.

We only got to the point of attempting the SqrtInv handoff once (which failed because I forgot to check the filter bank offsets). This was because the Mode Cleaner refused to stay locked longer than ~5-10 minutes at a time. We adjusted the MC and FSS servo offsets by the usual means, but this didn't make a difference.

We discussed and decided that the time is right to roll up our sleeves and dig into the MC loop, and try to figure out why these intermittent times of unreliability keep cropping up. We will check out the servo board, and see if we can find the missing phase than Evan observed, as well as characterize the FSS/PZT crossover, and investigate what kind of conditions we may create that cause the PC to saturate.

 Here's a fun fact: since the great computer failure of June2014, the PRM Oplev gains have been ZERO.

arrrrggggh

I've restored the gains to their old values, and measured the loop TFs.

A heads up to anyone using SVN with computers on the Martian network:

When we moved the svn repository on nodus to /export, we set it up such that the internet-facing svn URL was unchanged. However, it turns out that the martian network machines (i.e. Stuff mounted on the NFS share) were still pointing to the old svn files in /cvs/cds/caltech/svn, and thus not seeing new revisions made in /export/home/svn. If your martian network svn'd files got weird, this is why. 

I'm relocating the root svn URLs on the martian machines' checkouts to point to the nodus https address as I find them, to make them robust against future local movement of the svn files. 

Peoples' user files should be fine, this looks like it'll only really affect things such as scripts and medm screens, etc. 

 Last night, I poked around to try and see if I could reproduce the sketchy MC behavior by exciting MC2 in a way that may be similar to what we do when using it as a CARM actuator. 

The short of it is that at frequencies under 1k, the MC lock didn't mind MC2 position excitations up to 8000 counts. However around 4-5k, a 1000 count excitation would induce a good deal of low frequency (2-5Hz) activity in the MC trans power, causing it to fluctuate by thousands of counts before unlocking. If I turned the excitation off before the unlock, it would eventually settle back down, but not immediately. 

I was able to reproduce this a handful of times before it decided to stop locking altogether, perhaps because of its random mood swings, or perhaps because this kind of disturbance is related to the mood swings...

Koji and Evan have both brought up a good point that we may not be backing up the svn and ELOG properly.

I have modified the rsync.backup script that nodus' cron runs every night that backs up /cvs/cds to what I presume are the tape backups at ldas-cit.ligo.caltech.edu.

Specifically, I added two rsync commands that grab the svn and elog directories from /export/home and copy them to their old locations in /cvs/cds/caltech. This way, the old locations are updated, and the tape backups stay current.

[ericq, Jenne]

We both happened to come by today to fix things up.

When I arrived, the PMC was locked to a 01 mode, which I fixed. The PMC transmission is still worryingly low. MC locked happily. 

ETMX was getting odd kicks, the kind where a DC shift would occur suddenly, and then go away a few moments later. I turned off all dynamic coil outputs, and looked at the MON output of the SOS driver with a scope to try and see if the DAC or dewhitening was glitching, but didn't see anything... Meanwhile, Jenne fiddled with the TTs until we got beams on POP and REFL. (EDIT, JCD:  Useful strategies were to put an excitation onto TT2, and move TT1 until the scattered beam in the chamber was moving at the excitation frequency,  Find the edges of TT2 by finding where the scattered light stops seeing the excitation, and center the beam on TT2.  By then, I think I saw the beam on the PRM face camera.  Then, put a temporary camera looking at the face of PR2.  Using TT2 to center here got us the beam on the POP camera.)

We then walked PRM and the TTs around to keep those two camera beams and get the PRM oplev beam back on its QPD. At this point, ITMX was misaligned (by us), and ITMY aligned to get some recycled flashes into the Y-arm. Y-arm was locked to green, and we poked TTs to get better IR flashes. Misaligning PRM, we had Y-Arm flashes of ~0.7. From there, the michelson and then X-arm were roughly aligned. Both arms were seeing flashes of about 0.7, and the MICH fringes on the AS port look nice.

Frustratingly, the SUS->LSC communication for TRY and TRX isn't working, and could not be fixed by any combination of model or front-end restarting... Thus we haven't been able to actually lock the arms and run ASS. THIS IS VERY FRUSTRATING. 

Additionally, at the point where we were getting light back into the Yarm, the ITMX that were seen on Friday were happening again, tripping the watchdog. Also, something in the Yarm cavity is getting intermittently pushed around, as can be seen by the green lock suddenly wandering off. All of these suspension shenanigans seem to be independent of oplev damping. 

It troubles me that this whole situation is fairly similar to the last time we lost the input pointing (ELOG 10088)

In any case, we feel that we have gotten the IFO alignment to a lockable state.

TRX and TRY communication were recovered by doing a simultaneous reboot of all of the frontends.

Working with the interferometer has been extremely frustrating today. Having transmission values let us lock and ASS, but that has been less helpful than you would hope.

Saving the ASS offsets has repeatedly resulted in an overall bad change in alignment, moving the TTs and other things off randomly.

ITMX continues to be kicked. ITMY intermittently wanders away. It has not been possible to maintain IFO alignment for a reasonable length of time.

Also, the wall IOO striptool shows the MC2 Trans QPD Yaw having large step-function features. The MC is having an ok duty cycle, but this just may mean that the WFS are able to absorb what is happening to the MC suspensions.

The suspensions are really misbehaving. We need to get to the bottom of this, or else we are going to keep losing time to alignment.

I've checked out cdsutils-274 to /opt/rtcds/cdsutils, and updated the /ligo/apps/ligoapps-user-env.sh to have the newer machines use it by default. This was to gain access to the cdsutils.Step methods for use in the smooth ASS handoffs script. 

 I noticed some weird behavior on the ETMY oplev that led me to check them all out. 

The short of it is that the ETMY oplev has a pretty small angular range, compared to the displays and other oplevs. I measured how much angular motion each oplev can sense before the beam no longer hits all four quadrants (thus losing the ability to sense).  This could account for some of the additional angular motion of the mirrors... maybe. 

Also, some of the QPD quadrants had offsets as big as 400 counts, thus distorting the zero point. Anyways, here are the angular ranges of each QPD, assuming the current urad/cnt calibrations are valid. 

EMTY

• P: +- 25urad
• Y +- 30urad

ITMY

• P:+-160urad
• Y:+-172urad

BS

• P:+-43urad
• Y:+-40urad

ITMX

(Note: ITMX's oplev pitch and yaw is almost 30 degrees off of the alignment sliders' pitch/yaw coordinates. Steve tells me this is due to the tight nature of getting the oplev beam to the mirror without clipping.)

• P:+-110urad
• Y:+-80urad

ETMX

• P:+-45urad
• Y:+-85urad

PRM

• P:+-50urad
• Y:+-45urad

SRM

• P:+-80urad
• Y:+-80urad

I wrote a script to zero all of the QPD quadrants' offsets (it lives in /scripts/OL) and have used it successfully. The oplev laser must  be off before using it. 

I have measured the current boosted MC CLG below 100kHz with an SR785. Swept sine only could get me down to 10kHz, but I was able to get down to 5kHz with a noise-injection measurement. 

I am attaching the SR785 outputs, which are in dB and Degrees. Additionally I pruned the areas of bad coherence out of these, and merged them to provide data files for the CLG and OLG in Real,Imaginary format.