Our online subtraction filters for PRC angle and MC length were trained on the raw ADC signals, so I've inverted the filters that Rana installed in the PEM filter banks in the OAF signal conditioning filter banks (C1:OAF-WIT_STS1X, etc.)

It's not perfect, since the inversion would be unstable, and thus needs a low pass. I used an ELP at 800Hz. The error in the inversion is then something like half a degree at 5Hz, which is the highest frequency we really ever subtract at. It should be ok.

We looked at the DRMI noise spectrum and chose new excitation frequencies such that the lines are lower in frequency than before (~300 Hz instead of 800 Hz) and also not in some noisy region.

New filters is saved and loaded for all LSC DOFs.

Frequencies are:

• CARM: 309.21 Hz
• DARM: 307.88 Hz
• MICH: 311.1 Hz
• PRCL: 313.31 Hz
• SRCL: 315.17 Hz

POP110 and POP22 demod angles were adjusted for DRMI lock. 

Last week, I never achieved a fully 1F lock, REFL165 was used for SRCL. Tonight, we created input matrix settings for pure 1F locking, and did some signal mixing to reduce the PRCL to SRCL coupling. The PRCL to MICH coupling was already low, since AS55 is fairly insensitive to PRCL. 

Similarly, for the 3F signals, some signal mixing of REFL33I and REFL165Q was used to reduce the PRCL to MICH coupling. The PRCL to SRCL coupling in REFL165 isn't too bad, so no compensation was done. Interestingly, in this setting, the 3F MICH and SRCL signals agree with the 1F signals on their zero crossing, so no offsets are needed. REFL33 I does need an offset, however, to match the REFL11I PRCL zero crossing. 

The DRMI acquires faster with SRCL set to 165I. Once acquired, the 1F/3F can be made smoothly, and both settings are very stable. The sensing matrix in each setting is consistent with each other. (The PRCL and SRCL lines in AS55 change, but really I shouldn't even plot them, since they're not very coherent). 

For some reason, these show a sign flip relative to last week's measurements. The relative angles are consistent, though. 

Next up is finding the right coefficient for the SRM in the MICH output matrix, when actuating on the BS. 

The ETMY enclosure feedthrough - north is installed. The sealing material is hard to work with.

The upper empty blocks will be replaced by something soft to make changing cables easy.

When making the Wiener filter OFF/ON comparisons, we want to use the median PSD estimates, not the mean (which is what pwelch gives you).

cf. Sujan's note and Evan's follow-up

The median will be less sensitive to the transients / gltiches and will show more improvement I think.

TP2 dry fore pump sn PLE10082 was replaced at pressure 717 mTorr,  TP2 50K rpm 0.33A @ 112,677 hrs

Top seal life was 8,160 hrs

Model  SH110, Sn LP1007L556 was installed. It's fore line pressure after 30 minutes of running 38 mTorr, TP2 turbo at 50K rpm 0.18A

Just a heads up while I'm out for a bit: the delay line is currently installed in the 55MHz modulation path. 

I'll be back later, and will revert the setup.

I turned on the MCF FF in the OAF today (we need to fix the labeling of the 'ON' buttons on the RHS of the screen). The performance is still good; before / after attached.

Not only is the 1 Hz performance in the MC still good, but the X & Y arm noise reduction is ~1 order of magnitude. Good to know that the filters aren't changing much with time.

Can we just leave this on all the time now? Seems to be OK and there's no visible increase in the arm noise with this on.

Also did some updates to the summary pages and added a CDS FEC tab for CPU times.

Please take a look at the summary pages and bring a list of demands to the Wednesday meeting.

With the adjustable delay line box installed in the 55MHz modulation path, I've measured the PRMI sensing matrix as a function of delay / relative phase between the 11MHz and 55MHz modulations. The relative frequency difference of 44MHz tells us that this should be cyclical after ~23nsec of delay, but losses in the delay cable change this; see Koji's elogs about the modulation cancellation setup for details. 

TL;DR: Nothing really changes, other than REFL33 optical gain. MICH/PRCL angles remain degenerate.

The results aren't so surprising. The demod angles for the 55MHz diodes don't even change, since the same 55MHz signal is used for the modulator and demodulators, so delaying it before the split should go unnoticed. Most of these measurements were made during the same lock stretch, PRCL on REFL11 I and MICH on AS55Q.

The only signals we would expect to change much are ones that have significant contriubtions from field products influenced by both modulations. None of the 1F PDs are like this, nor is REFL165. REFL33 is the odd man out, where the +44MHz field produced as a -11MHz sideband on the +55MHz sideband beats with the +11MHz sideband (and the same with the signs flipped). I made a simulation for the 40m poster at the March 2015 LVC meeting, but I don't think it ever made it to the ELOG. 

So:

Here are the results for the 0ns and 4ns cases, as an illustration of what changes (REFL33), and what doesn't (everything else). Again, these are calibrated to Volts out of the analog demod boards per meter of DoF motion. 

So, since REFL33 is the only one really changing, let's just look at it by itself:

Qualitatively, the change in magnitude looks similar to the simulation result. The demod angles fall by some roughly linear amount. The angle difference is even more stationary than predicted there, though. 

ETMY optical table enclosure feedthrough- south is in. Now it is time to see how air tightness increases performance.

1. POP110 RF amps are powered from the cross connect. But that +15V block has GND connections that are not connected to the ground.
    i.e. The ground potential is given by the signal ground. (Attachment 1)

    This is caused by the misuse of the DIN connector  blocks. The hod side uses an isolated block assuming a fuse is inserted.
    However, the ground sides also have the isolated blocks

2. One of the POP110 RF cable has a suspicious shiled. The rigidity of the cable is low, suggesting the broken shield. (Attachment 2)

I added the 0.1 uF and 47 nF caps that I mentioned so that we can now bypass the AA filters for these channels. (mistakenly installed 47 instead of 0.47 nF on the first round and we got 350 Hz poles instead of 35 kHz)

Gautam and I checked out the AA sit and it seems that the XYCOM-220 cable which ought to allow switching of the AA filter is not connected on the XYCOM side, so the LSC AA filters are always ON. In order to bypass them, we'll need to just short the bypass control pins or just set the +5V on the board to GND, by lifting the EMI3 filter and shorting C6.

I have so far only made the changes on s/n 115 (used for AS55, REFL55, and REFL165), other 2 boards to follow soon.

Before making the AA change, we want to measure the HF spectrum the ADC for each of our main signals in the PRFPMI state. In lieu of that, we'll measure the spectrum at the I/Q mon ports of the demod boards via SR785 and then use matlab to propagate the signals to the ADC to make our estimate of how much anti-aliasing we need.

Changes relative to D990694-B:

1. R215, R216, R217, R218, R219: 4.75k -> 9.53k.  This change was made long to make the DC gain of channels 4-8 be unity, the same as channels 1-3.
2. 0.1 uF NPO cap in parallel with R127, R128, R129, R130, R131, R132, R133, R134.
3. R127, R128, R129, R130, R131, R132, R133, R134 all 100k (was already like this) to keep LT1128 from floating up when input cables are disconnected.
4. C158, C159, C160, C161, C162, C163, C164, C165, C166, C167, C168, C169, C170, C171, C172, C173, all were empty, now are 0.47 nF NPO.

I also looked at the noise in a few different configurations to see what we ought to do next.

BLACK: AS55I_IN1 with 0 dB whitening gain and whitening filter OFF, so its all just ADC noise

RED: same but with +45 dB whitening gain and WF ON, so above 10 Hz this is now the noise of the PD / demod chain

BLUE: RED w/ the anti-WF applied

PURPLE: in-loop POX11_I spectrum with x-arm locked

The conversion from counts to volts 0.0006, so the black trace is ~5 uV/rHz as expected. Its clear that we would be sort of OK for most of our channels if we just had 1 stage of whitening. I think we ought to convert the input stage into a 100:20 stage and also change the other whitenings into a 100:20 instead of 150:15. Then we'll have less gain at 15 Hz, but more at 100 Hz.

We really need to buy some surface mount capacitors, Steve - we ought to have at least 100 of all the ones in that little gray cabinet.

I doubt we'll see any effect until we carefully seal the holes. If there's 1 hole in your boat it still sinks.

No damage. The BS sensor UR 0.220 V has been low for some times.

Dataviewer does not work for long term trend.

ETMY - Guralp (B-MIT) was covered with copper lined can yesterday afternoon. It's long cable is connected to ADC interface box input 1

The vertex Trillium was covered just ~2 days before Ignacio left.

ETMX - Guralp (A-Caltech) is not covered. The long 40m cable is disconnected at the the south end.

[Steve, gautam]

We fixed the problematic DIN connectors on 1Y2, by swapping out the 3 DIN connector blocks that were of the wrong type (see attached image for the difference between the types appropriate for "Live" and "Ground").

Before doing anything, Eric turned the Wenzel multiplier off. We have not turned this back on.

Then we turned off the power supply unit at the base of 1Y2, removed the connectors from the rail, swapped out the connectors, reinstalled them on the rail, and turned the power supply back on. After swapping these out, we verified with a multimeter that between each pair of "Live" and "Ground" blocks, there was ~15V. We could now use the third unused pair of blocks to power the delay line phase shifter box, though for the moment, it remains powered by the bench power supply. 

1. The delay-line box is now hooked up to the cross connect +15V supply.

2. The broken RF cable was fixed.

It is actually the POP22 cable.
Therefore, we might see significant change of the signal size for POP22.
Be aware.

RG405 + SMA connector rule

- Don't bend the cable at the connector.

- Always use a cap on the connector. It is a part of the impedance matching.

- Use transparent shrink tube for strain relieving and isolation. This allow us to check the condition of the shield without removing the cover.

Steve and I turned on the box this morning so that the IMC would lock again.

For future reference, remember that one should turn off the Marconi output before turning off the RF distribution box. Don't drive the input of unpowered RF amps.

Steve and I turned on the box this morning so that the IMC would lock again.

For future reference, remember that one should turn off the Marconi output before turning off the RF distribution box. Don't drive the input of unpowered RF amps.

Once again, the transmitted X green beam was showing enormous intensity fluctuations (50x higher than normal). Last month, I reduced the AUX X laser current from 2.0A to 1.9A, which I thought had fixed it somehow.

However, when I sent to the end to check it out today, I found the SR560 which is there to amplify the green PDH error signal before being sent to the AA board was overloading. Not so surprising, since the error signal was similarly noisy as the transmitted light. 

I turned the SR560 gain down, and, after relocking, the transmitted light was stable. I've turned the AUX X laser current back up to 2.0A, it's previous nominal value, and the green transmitted light is still stable. 

I'm a little mystified that the 560 could intefere with the loop, since it is not in the feedback path. Could it be that when it is overloading, it sends garbage backwards out of the inputs? But even then, its input is not connected to the real error point, but the buffered monitor port. Could it be interfering via the power line?

Before, I had hesitated adding gain to the PDH board's monitor point for DAQ purposes, because the motivation of the port is to provide a 1:1 version of the real error signal, and I didn't want to add gain to the AA board, because we normally don't have gain in those boards, and I didn't want to surprise future people. The SR560 was meant to be temporary, but as often happens, it was forgotten. Now, I think I will add gain to the error monitor buffer stage of the PDH boards. 

To get around the problems between the pomona LPF and low CM board input impedance, I've placed the LPF at the CM board fast output. This won't work as a permanent solution, since we only want to lowpass the ALS signal, but it should be fine for a single arm test. 

However, I kept getting blown out of lock when turning up the AO gain, but well before I really expect any real action from the fast path. Looking at the OLTF, I was seeing some large spike at ~36kHz nearing 0dB loop gain with unstable phase. This prompted me to look at the ALS error signal out to higher bandwidth with the SR785; before I only ever looked at it through the digital system. 

So, with the X arm locked via POX11 I, and ITMY misaligned to use AS55 as an out of loop sensor, I measured the spectrum of the I ouput of the ALS X demod board (which was set to be near a zero crossing via the delay line), and the Q Mon of the AS55 demod board. 

Both ALS and AS55 show a sharp line at around 36.5kHz, so something is really happening in the IFO at this frequency. Koji might have seen an indication of this back in March.

What's going on here? And what would be different about PRFPMI that wouldn't have made this a problem for locking?

 I looked at REFL11 and REFL55 during PRMI lock - the line is there.

In fact, it is even visible in REFL11 I from a single bounce off of the PRM (ITMs misaligned).

This led me to look at the IMC error point (via the OUT2 on the servo board, no compensation for the input gain). Also there!

One the Wiki (https://wiki-40m.ligo.caltech.edu/40mHomePage), we have a Mech Resonance page for mechanical frequencies and a PEM page where we want to list the sources of all of our environmental lines. So please put in an entry when you find out what's at this frequency. This reminds me that I need to upload my MC2 COMSOL eigenmode analysis.

I modified the perl script which does our hourly autoburt so that it can run on megatron instead of op340m (old Solaris machine). Nothing major, just some path stuff. That was the last function of op340m that I know of, so after a week of watching this we ought to be able to power it off and send it to e-waste.

Seems to work so far. It complains about some models that aren't running but mostly it reports successful snapshot taking based on the .req files.

Unfortunately, it seems that its only doing the new target directory, so its missing all of our old VME machines which still use the /cvs/cds/caltech/target area.

But I think Gautam and Jamie and Aidan have volunteered to start our slow controls upgrade by moving the EX slow controls to Acromag and into the new target area. We ought to modify the CRON to point at the old directory for now, but its a temporary fix hopefully.

CC4 cold cathode gauge jump triggered interlock to close VM1 valve to protect the RGA.

The IFO pressure is 1e-5 Torr

Vac normal was recovered by opening VM1

The MC autolocker and FSSslow scripts weren't running on Megtron. These were started by running the following commands on megatron:

The new autoburt cronjob was failing because the .cron file was not executable (fixed by chmod +x burtnew.cron), and the new perl script didn't use the full path for ifconfig. Similarly, the simulink webview updating script was failing because the full path for matlab wasn't being given. Both of these fixes have been tested and commited to SVN. 

In general, cron scripts can be a real pain, since the cron process doesn't run our .bashrc, and so doesn't know about updates to $PATH, or other environment vairables that get updated through /ligo/cdscfg/workstationrc.sh, which is called by .bashrc. So something that manually works fine in the terminal may not play out as expected when run by cron.

I've been putting together a new machine that Rolf got for us as a replacement for fb.

I've installed and configured the OS, and compiled daqd and the necessary supporting software.  I want to try acquiring data with it.  This will require removing the current/old fb from the DAQ network, and adding the new machine.  It should be able to be done relatively non-invasively, such that none of the front end configuration needs to be adjusted, and the old fb can be put back in place easily.

If the test is successfully, then I'll push ahead with the rest of the replacement (such as either moving or copying the /frames RAID to the new machine).

I will do this work in the early AM tomorrow, September 22, 2015.

Given the RF component power supply grounding, POP110, POP22 and REFL165 all changed somewhat. They have all been rephased for the DRMI, as they were before. 

I tweaked the 3F DRMI settings, and chose to phase REFL165I to PRCL, instead of SRCL as before, to try and minimize the PRCL->MICH coupling instead of the SRCL->MICH coupling. 

With these settings, I once locked the DRMI for ~5 seconds with the arms held off on ALS, during which I could see some indications of neccesary demod angle changes. Haven't yet gotten longer, but we're getting there...

The cold cathode gauge is back to normal. cc4 is the last gauge is "functioning"

MKS is not responding. The spare controller and gauges are back for repair.

I've moved the OAF MC2 signal path to go directly from c1oaf to c1mcs, so that the LSC being ON/OFF doesn't interfere with the MC length seismic feedforward. Since the FB is currently down, I can't do a full test, but looking at monitor points in StripTool indicates it's working as intended. 

I also cleaned up some LSC medm stuff; exposing the existing SRCL UGF servo, and removing a misleading arrow. This reminds me that I need to get calibration lockins back up and running...

Today I've been trying to get the new frame builder, tentatively 'fb1', to work.  It's not fully working yet, so I'm about to revert the system back to using 'fb'.  The switch-over process is annoying, since our one myrinet card has to be moved between the hosts.

A brief update on the process so far:

I'm being a little bold with this system by trying to build daqd against more system libraries, instead of the manually installed stuff usually nominally required.  Here's some of the relevant info about th fb1 system:

• Debian 7 (wheezy)
• lscsoft ldas-tools-framecpp-dev 2.4.1-1+deb7u0
• lscsoft gds-dev 2.17.2-2+deb7u0
• lscsoft libmetaio-dev 8.4.0-1+deb7u0
• lscsoft libframe-dev 8.20-1+deb7u0
• /opt/rtapps/epics-1.4.12.2_long
• /opt/mx-1.2.16
• advLigoRTS trunk

I finally managed to get daqd to build against the advLigoRTS trunk (post 2.9 branch).  I'll post detailed build log once I work out all the kinks.  It runs ok, including writing out full frames, as well as second and minute trends and raw minute trends, but there are a couple of show-stopper problems:

• daqd segfaults if the C1EDCU.ini is specified.  If I comment out that one file from the 'master' channel ini file list then it runs without segfaulting.
• Something is going on with the mx_streams from the front ends:
  • They appear to look ok from the daqd side, but the FEC-<ID>_FB_NET_STATUS indicators remain red.  The "DAQ" bit in the STATE_WORD is also red.  Again, this is even though data seems to be flowing.
  • The mx_stream processes on the front ends are dying (and restarting via monit) about every 2 minutes.  It's unclear what exactly is happening, but they all dia around the same time, so it possibly initiated from a daqd problem.  Around the time of the mx_stream failures, we see this in the daqd log:

[Tue Sep 22 17:24:07 2015] GPS MISS dcu 91 (TST); dcu_gps=1127003062 gps=1127003063

Aborted 1 send requests due to remote peer Aborted 1 send requests due to remote peer 00:25:90:0d:75:bb (c1sus:0) disconnected
mx_wait failed in rcvr eid=004, reqn=11; wait did not complete; status code is Remote endpoint is closed
00:30:48:d6:11:17 (c1iscey:0) disconnected
mx_wait failed in rcvr eid=002, reqn=235; wait did not complete; status code is Remote endpoint is closed
disconnected from the sender on endpoint 002
mx_wait failed in rcvr eid=005, reqn=253; wait did not complete; status code is Bad session (missing mx_connect?)
disconnected from the sender on endpoint 005
disconnected from the sender on endpoint 004
[Tue Sep 22 17:24:13 2015] GPS MISS dcu 39 (PEM); dcu_gps=1127003062 gps=1127003069

• Occaissionally the daqd process dies when the front end mx_streams processes die.

I'll keep investigating, hopefully with some feedback from Keith and Rolf tomorrow.

[ericq, Gautam]

We can reliably lock the DRMI with the arms held off on ALS. 

I have not been able to hold it at zero CARM offset; but this is probably just a matter of setting up the right loop shapes with enough phase margin to handle the CARM fluctuations ( or figuring out high bandwidth ALS...)

Right now, it's the most stable at CARM offsets larger (in magnitude) than -1. Positive CARM offsets don't work well for some reason. 

The key to getting this to work was to futz around, starting from the misaligned arms DRMI settings, until brief locks were seen (triggering all 3 DRMI DoFs on POP22, since the correct AS110 sign was amiguous). I could tell from how the control signals responded to gain changes that REFL165Q, which was being used as the MICH error signal, was seeing significant cross coupling from both PRCL and SRCL, suggesting the demod angle of REFL165 had to be adjusted. I randomly tweaked the REFL165 demod angle until a 20 second lock was achieved, with excitations running. Then, I downloaded that data and analyzed the sensing matrix. This showed me that the REFL33 demod angle was ok, and the PRCL-from-SRCL subtraction factor determined with the arms misaligned was still valid. The main difference was indeed the SRCL angle in REFL165.

With the REFL165 demod angle properly adjusted, the DRMI would briefly lock, but the DRMI had become somewhat misaligned at this point, and the SRC could be seen to mode hop. Interestingly, the higer order modes had an opposite sign in AS110, with respect to the TM00. At that point, I went back to PRMI on carrier to dither-align the BS and PRM. 

With alignment set, the DRMI would lock on TM00 readily, still only triggering on POP22. I set the AS110 angle, and moved SRCL triggering over to that, which sped up acquisition even more. The input matrix and FM gains from no-arms DRMI still work for acquistion; UGF servos were used to adjust overall gains a bit. 

At CARM offsets larger in magnitude than -1, the DRMI lock seems indefinite. I just broke it to see how fast it would acquire; 3 seconds. 

Lastly, here is the sensing matrix at CARM offset of -4, measured over five minutes. REFL11 is the only degenerate looking PD. Thus, I feel like controlling the DRMI of the DRFPMI should be more managable than I had feared.

(I didn't include/excite CARM or DARM, because I'm not sure it would really mean anything at such a large CARM offset)

Looking good. How many meters of CARM is '-1 counts'?

Nice!!

I noticed that Chiara's backup HD (which has a capacity of 1.8TB, vs the main drives 2TB) was near to getting full, meaning that we would soon be without a local backup. 

I freed up ~200GB of space by compressing the autoburt snapshots from 2012, 2013, 2014. Nothing is deleted, I've just compressed text files into archives, so we can still dig out the data whenever we want.

Just a quick note for now: I've repopulated C1CAL with a limited set of lockin oscillators/demodulators, informed by the aLIGO common LSC model. Screens are updated too. 

Rather than trying to do the whole magnitude phase decompostion, it just does the demodulation of the RFPD signals online; everything beyond that is up to the user to do offline. 

Briefly testing with PRMI, it seems to work as expected. There is some beating evident from the fact that the MICH and PRCL oscillation frequencies are only 2Hz apart; the demod low pass is currently at an arbitrary 1Hz, so it doesn't filter the beat much. 

Screens, models, etc. all svn'd.

Our last effort to change the existing Al-6061 wire standoffs was at April 2012

We requested sapphire and/or ruby materials with smaller R at the bottom of the  groove. Groove polishing was asked for.

Insaco Inc. quote 84740 as " best effort " NO POLISHING.  The groove cut to be with eximer laser.

Jeff Lewis as 9-12-2012:  the LIGO sapphire prisms grooves were NOT POLISHED  but Resonatics used the corner of a rasor blade to scrape off the

ablated material wich was redeposited in and around the grooves.
 

We have talked about the drift of ETMX sus on the Wednesday meeting.

It has stopped moving on Jan 8, 2015 and it has been reasanable stable since than.

The IFO pressure is estimated ~1E-6 Torr with modified Vac normal valve configuration.

CC4 is in a jumping mode between 2e-5 and 1e-6 Torr

 Pressure based interlock kicks in to close VM1   at 2e-5 Torr to protect the RGA.

I did open VM1 repeatedly in the last few mornings but as cc4 jumps VM1 closes.

As VM1  closed to RGA scans are not seeing the IFO. I will look at some scans on Monday.

Mean while I opened VM2 to lower the pressure for the RGA. This change will be read by "Current Status: Undefined State"

So do not panic, the IFO pressure is normal.

I need someone's help to raise the interlock threshold to 5e-5 Torr

I'm buying a new cold cathode gauge on Monday.

note: cc1 is out of order!

         just read P1  the pressure is < 7e-4 Torr  This gauge is very reliable and it is at the low end of it's range.

Brief update about the fb replacement status.

The new hardware for fb is in the rack, temporarily sitting on top of megatron, and on the CDS network with the name 'fb1'.  I've installed an OS on it and have re-built daqd.

Earlier this week I swapped it into the network and tried to get it to acquire data from the front ends.  I was ultimately unsuccessfully.  The problem seemed to be the mx_stream communication from the front ends to the new host.

The swap is sort of a pain because we only have one Myrinet fiber network adapter card that has to be moved between machines, which of course requires shutting down both machines and opening up their chassis.  I instructed Steve to order us a new Myrinet card for the new machine, which will allow us to swap daqd machines by just moving the fiber connection.  Once that's in place (early next week) I'll go back to trying to figure out what the issue is with the mx_streams.

If all else fails I'll take the repulsive last resort of either swapping or cloning the disk from the old fb.

I don't see any evidence of it getting more stable. It seems there was a big step in January, but the problem we were talking about - the suspension shifting when it gets a big kick - can't be proven to be gone or not by just looking at the trends. The real issue is whether or not it slips when we put in a large step in the LSC.

Earlier today, the front panels for the 1U chassis I obtained to house the Wenzel dividers + RF amplifiers arrived, which meant that finally I had everything needed to complete the assembly. Pictures of the finished arrangement attached. 

Summary of the arrangement:

• Two identical channels (RF amplifier + /64 divider + /256 divider), one for each arm
• The front panels are anodized, and isolated SMA feedthroughs are used 
• Given the large number of units to be supplied with DC power (2 amplifiers + 4 dividers), I chose to use two D1000217 power regulators (the default configuration takes +-18V as input, and outputs regulated +-15V, which was fine for the dividers, but the ZKL-1R5 requires +12V, so I changed the resistor R2 in the schematic from a 10.7K to 8.451K so as to accommodate this).
• The amplifiers and dividers are mounted on a steel plate, which is itself mounted on the chassis via insulating posts. 

Testing:

• I first verified the power regulator circuitry without hooking up the amplifiers/dividers - with a multimeter, I verified I was getting +15V and +12V as expected.
• I then connected the amplifiers and dividers, and decided to first check the behaviour of each channel using the Fluke 6061 RF function generator and an oscilloscope. One of the channels (X-arm in the current configuration) worked fine - I got a 0-2.5V square wave as the output for input signals as low as -38dBm at 130MHz (consistent with out earlier observations).
• The Y-arm channel however did not give me any output. In order to debug the problem, I decided to check the output after the amplifier first. The amplifier does not seem to be working for this channel - I get the same amplitude at the output as at the input. I verified that the correct DC power voltage of +12V was being supplied with a multimeter, but I am not sure how to debug this further. The amplifier is basically straight out of the box, and as far as I can tell, I have not done anything to damage it, as this was the first time I am connecting it to anything, and I repeated the same steps on the Y-arm as the X-arm, which seems to work alright.
• The rest of the Y-arm signal chain was verified to be working by bypassing the amplifier stage (the attached photographs show the box in this configuration. There seems to be no issues with the divider part of the signal chain. 

Once I figure out the problem with this amplifier/replace it, the box is ready to be installed. 

Fast ALS control continues to elude me. 

I fixed my LPF to take the input impedance of the CM board input into account; this unfortunately results in about -12dB DC gain of the ALS signal due to voltage-divider-y things, but by my estimation, this still puts the DFD noise above the input-referred voltage noise of the input AD829 on the CM board, so it'll do for now. The 120Hz pole shows up as expected when comparing the usual digital channels and the CM_SLOW output, and is digitally compensated with a zero at 120Hz (with a digital pole at 5k so nothing blows up). 

However, there seems to be some zero in the analog path somewhere that spoils the loop shape for the AO path. Here's a measurement of the X arm OLG from 10-100kHz, when the digital control is happening with ~100Hz UGF via ALS X I -> CM IN2 -> CM_SLOW -> LSC_CARM -> ETMX, and there is some AO action via ALS X I -> CM IN2 -> IMC IN2

The peak is recognizable as the gain peaking in the IMC servo (and changes predictably with changes to the IMC crossover and loop gains), which is expected. However, one can see that the magnitude is roughly flat before the peak, and the phase is around 0. With the 1/f LPF, we should see some downward slope and phase starting around -90. 

Thus, there must be some zero in the fast or common path, maybe at a few kHz where the digital loop wouldn't really see its effect. I'm not sure what it could be at this point in time.

One thought I had is that I never really checked the TF of DFD response to frequency modulation of the RF beat. I used an SR785 to drive the external FM input of a Fluke 1061A synthesizer, and saw it to be totally flat from 1-100kHz with carriers from 30-100MHz, so that should be fine. (For a little while I was confused by what seemed to be some heavy high-passing going on, but it turns out that the Fluke just can't push much low frequency FM; the manual says -3dB at 20Hz.)

Use LISO - see what it tells you. I would think that you should make a differential RC filter to get the right behavior. (e.g. 1K on each leg and 1 uF between them)

Each leg of the diff input of the board has a 4k input impedance.

But surely the AO input to the MC servo should also make sense independently.

The new 2x2 fiber couplers arrived today so Eric gave me an overview on the changes to be made to the existing configuration of the FOL fiber box. I removed the box from the table after ensuring that the PDs were powered OFF and removing and capping all fiber leads on the front panel. Here is a summary of the changes made.

• On-Off positions for the rocker switches corrected - these switches for the power to the PDs were installed such that the "1" position was OFF. I flipped both the switches such that the "1" position now corresponds to ON (see Attachment #1).
• All the couplers/beam combiners/splitters were initially removed. 
• I then re-configured the layout as per the schematic (Attachment #2). I only needed to use one of the 4 new 2x2 couplers ordered. I think the 1x2 couplers are appropriate for mixing the PSL and AUX beams, as if we use a 2x2 coupler, half of the mixed light goes nowhere? Indeed, if we had one more such coupler, we could do away with the 2x2 coupler I am now using to divide the PSL light into two. 
• The spec-sheets on the inside of the top cover were updated to reflect the new hardware (Attachment #3).
• The old hardware from the box that was not used, along with their spec-sheets, are stored temporarily in a Thorlabs lab snacks box (all the fibers have been capped).
• The finished layout is shown in Attachment #4.

I then ran a quick check to see what the power levels were at the input to the PDs, using the fiber coupled power meter. However, I found that there was no light in the fiber marked "PSL light in" (the power meter read out "Sig. Low"). The X arm Aux light had an input power of 1.12 mW, which after the various coupling losses etc went down to 63 uW just before the PD. The corresponding figures for the Y arm are 200 uW and 2.2 uW. I am not too sure of how the AUX light is coupled into fibers so I am not trying to tweak the alignment to see if I get more power. 

LISO confirms that I did my algebra right in picking the component values, and shows no extra zeros. 

I also took some TFs with the SR785 and confirmed that both CM board inputs behave the same, and that including the LPF on the input gives the expected 1/f shape at the slow and fast outputs.

Eric pointed out that the 1x2 couplers that were used in the previous arrangement and which I recycled, were in fact NOT appropriate - they are not 50-50 couplers but 90-10 couplers, which explains the measured power levels I quoted here.

I switched out these for a pair of the newly arrived 2x2 couplers, and have also replaced the datasheets on the inside of the top cover. I then redid the power level measurements, and got some sensible values this time (see Attachment #1 for revised layout and measured power levels, numbers in red are powers for PSL light, numbers in green are for AUX laser light, and all numbers are in mW). I did find that the 90-10 splitter in the PSL+Y path was not working (though the one in the PSL+X path seems to be working fine), and hence, have not quoted power levels at the output of these splitters. For now, I guess we can bypass the splitters and take the PSL+AUX light from the 2x2 couplers directly to the PDs.

I got Steve to get us a new Myrinet fiber network adapter card for fb1:

• Myrinet 10G-PCIE-8B-S

I just finished installing the card in fb1, and it came up fine.  We happened to have a spare fiber, and a spare fiber jack in the DAQ switch, so I went ahead and plugged it in in parallel to the old fb:

controls@fb1:~/rtbuild/trunk 130$ /opt/mx/bin/mx_info
MX Version: 1.2.16
MX Build: controls@fb1:/opt/src/mx-1.2.16 Fri Sep 18 18:32:59 PDT 2015
1 Myrinet board installed.
The MX driver is configured to support a maximum of:
    8 endpoints per NIC, 1024 NICs on the network, 32 NICs per host
===================================================================
Instance #0:  364.4 MHz LANai, PCI-E x8, 2 MB SRAM, on NUMA node 0
    Status:         Running, P0: Link Up
    Network:        Ethernet 10G

    MAC Address:    00:60:dd:43:74:62
    Product code:   10G-PCIE-8B-S
    Part number:    09-04228
    Serial number:  485052
    Mapper:         00:60:dd:46:ea:ec, version = 0x00000000, configured
    Mapped hosts:   7

                                                        ROUTE COUNT
INDEX    MAC ADDRESS     HOST NAME                        P0
-----    -----------     ---------                        ---
   0) 00:60:dd:43:74:62 fb1:0                             1,0
   1) 00:25:90:0d:75:bb c1sus:0                           1,0
   2) 00:30:48:be:11:5d c1iscex:0                         1,0
   3) 00:30:48:d6:11:17 c1iscey:0                         1,0
   4) 00:30:48:bf:69:4f c1lsc:0                           1,0
   5) 00:14:4f:40:64:25 c1ioo:0                           1,0
   6) 00:60:dd:46:ea:ec fb:0                              1,0

We can now work on fb1 while fb continues to run and collect data from the front ends.

I'm still not getting the mx_stream connections to the new fb1 daq to work.  I'm leaving everything running as is on fb for the moment.

Gautam and Steve,

The decommissioned server from LDAS is retired to the 40m   with 32 cores and 128GB of memory in rack 1X7   http://docs.oracle.com/cd/E19121-01/sf.x4600/

Summary

I've not really been able to make additional progress with the new 'fb1' DAQ.  It's still flaky as hell.  Therefore we're still using old 'fb'.

Issues

mx_stream

The mx_stream processes on the front ends initially run fine, connecting to the daqd and transferring data, with both DAQ-..._STATUS and FE-..._FB_NET_STATUS indicators green.  Then after about two minutes all the mx_stream processes on all the front ends die.  Monit eventually restarts them all, at which point they come up green for a while until the crash again ~2 minutes later.  This is essentially the same situation as reported previously.

In the daqd logs when the mx_streams die:

Aborted 2 send requests due to remote peer 00:30:48:be:11:5d (c1iscex:0) disconnected
Aborted 2 send requests due to remote peer 00:14:4f:40:64:25 (c1ioo:0) disconnected
Aborted 2 send requests due to remote peer 00:30:48:d6:11:17 (c1iscey:0) disconnected
Aborted 2 send requests due to remote peer 00:25:90:0d:75:bb (c1sus:0) disconnected
Aborted 1 send requests due to remote peer 00:30:48:bf:69:4f (c1lsc:0) disconnected
mx_wait failed in rcvr eid=000, reqn=176; wait did not complete; status code is Remote endpoint is closed
disconnected from the sender on endpoint 000
mx_wait failed in rcvr eid=000, reqn=177; wait did not complete; status code is Connectivity is broken between the source and the destination
disconnected from the sender on endpoint 000
mx_wait failed in rcvr eid=000, reqn=178; wait did not complete; status code is Connectivity is broken between the source and the destination
disconnected from the sender on endpoint 000
mx_wait failed in rcvr eid=000, reqn=179; wait did not complete; status code is Connectivity is broken between the source and the destination
disconnected from the sender on endpoint 000
mx_wait failed in rcvr eid=000, reqn=180; wait did not complete; status code is Connectivity is broken between the source and the destination
disconnected from the sender on endpoint 000
[Thu Oct  1 19:00:09 2015] GPS MISS dcu 39 (PEM); dcu_gps=1127786407 gps=1127786425

[Thu Oct  1 19:00:09 2015] GPS MISS dcu 39 (PEM); dcu_gps=1127786408 gps=1127786426

[Thu Oct  1 19:00:09 2015] GPS MISS dcu 39 (PEM); dcu_gps=1127786408 gps=1127786426

In the mx_stream logs:

controls@c1iscey ~ 0$ /opt/rtcds/caltech/c1/target/fb/mx_stream -r 0 -W 0 -w 0 -s 'c1x05 c1scy c1tst' -d fb1:0
mmapped address is 0x7f0df23a6000
mmapped address is 0x7f0dee3a6000
mmapped address is 0x7f0dea3a6000
send len = 263596
Connection Made
isendxxx failed with status Remote Endpoint Unreachable
disconnected from the sender

daqd

While the mx_stream processes are running daqd seems to write out data just fine.  At least for the full frames.  I manually verified that there is indeed data in the frames that are written.

Eventually, though, daqd itself crashes with the same error that we've been seeing:

main profiler warning: 0 empty blocks in the buffer

I'm not exactly sure what the crashes are coincident with, but it looks like they are also coincident with the writing out of the minute and/or second trend files.  It's unclear how it's related to the mx_stream crashes, if at all.  The mx_stream crashes happen every couple of minutes, whereas the daqd itself crashes much less frequently.

The new daqd can't handle EDCU files.  If an EDCU file is specified (e.g. C0EDCU.ini in our case), the daqd will segfault very soon after startup.  This was an issue with the current daqd on fb, but was "fixed" by moving where the EDCU file was specified in the master file.

Conclusion

There are a number of differences between the fb1 and fb configurations:

• newer OS (Debian 7 vs. ancient gentoo)
• newer advLigoRTS (trunk vs. 2.9.4)
• newer framecpp library installed from LSCSoft Debian repo (2.4.1-1+deb7u0 vs. 1.19.32-p1)

It's possible those differences could account for the problems (/opt/rtapps/epics incompatible with this Debian install, for instance).  Somehow I doubt it.  I wonder if all the weird network issues we've been seeing are somehow involved.  If the NFS mount of chiara is problematic for some reason that would affect everything that mounts it, which includes all the front ends and fb/fb1.

There are two things to try:

• Fix the weird network problem.  Try remove EVERYTHING from the network except for chiara, fb/fb1, and the front ends and see if that helps.
• Rebuild fb1 with Ubuntu and daqd as prescribed by Keith Thorne.