Looks good.

Once the control cable (bakplane cable) is identified, we can install the module to the LSC analog rack.

We should be able to test the CM servo with either POX or POY and only one correspoding arm without modifying the servo TF.
Just for this test, we don't need to use MCL.

I had a look on x,y arms stabilization using ALS. Input green beam was misaligned and I was loosing 00 every few minutes. I vent on the floor and realigned green beams.

YARM alignemt was smooth - transmission increased from 0.4 to 0.85 with PSL shutter off.

XARM was tough. Steering mirrors did not have any derivatives when transmission power was 0.5. I walked the beam with piezos but got only 0.55. It seems that the input beam is mismatched to the cavity. When the transmission was 1 last time? Does anyone have a model of the xend table to compute mode matching?

Input green alignent was improved and I could keep arms stabilized for periods of ~30min - 1 hour. Still not forever.

I noticed that ALS_XARM and ALS_YARM servos have limiters of 6000 and control signal had high frequency components that were not rolled off as shown on the plot "ETMY_DRIVE". I have added a low pass filter that reduced RMS by factor of 5 and took 7 degrees of phase at UGF=150 Hz. Now margin is 33 degrees.

Then I excited ETMY longitudinally at 100 Hz and measured first and second harmonics of the YARM RIN. I got total DC offset of 0.3 nm. This means significant length coupling to RIN. First of all, "scan arm" script does not tune the offset very precise. I guess it looks at DC power, checks when cavity passes through symmetrical points of the resonance and takes the average. It is also useful to look at POX/POY and confirm that average is 0. Plot "ALS_RIN" shows comparison of YARM power fluctuations when it is locked using IR and stabilized using ALS. By manually correcting the offset I could reduce length coupling into RIN, coherence was ~0.1.

Cavity RMS motion also couples length to RIN. Plot "ALS_IR" shows YARM error signal. I also looked at POY signal (LSC-YARM_IN1) as an OOL sensor. At low frequencies POY sees only IMC length fluctuations converted to frequency. I have engaged MCL path and ALS error and LSC error signals overlaped. Cavity RMS motion is measured to be 200 pm.

 These seem like pretty terrible loop shapes. Can you give us a plot with the breakdown of several of the TFs and some .m file?

We should be able to estimate the noise coming out of the MC using the single arm and then make a guess for the CM loop gain requirement. There's no reason to keep the old Boost shapes; those were used in the old MC configuration which had a RefCav. In addition to minimizing the EOM range, we should also minimize the AO signal as Koji has pointed out. In practice, I've seen that using ~300 Hz of offset makes no harm with 4 kHz MC pole.

Last time we designed MCL loop with UGF ~ 30 Hz and I think, it was hard to lock the arm because of large frequency noise injected to IFO.

This time I made a low bandwidth MCL loop with UGF=8 Hz. MCL error RMS is suppressed by factor of 10 and arms lock fine.

Attached plots show MCL OL, MCL error suppression and frequency noise injection to arms.

It is interesting that spectrum of arms increases below 1 Hz meaning that IMC sensing noise dominates in this range.

I did not include the loop into the IMC autolocker. I think it is necessary to turn it on only during day time activity and when beatnote is moving too much during arm stabilization.

Attached is matlab code that I used

 % IMC OL
G = zpk(-2*pi*8964, 2*pi*[-10; -10; -10; -1000; -274000], db2mag(242.5)) * ...
    tf([1 0.8*1.55e+05 3.1806e+10], 1);

% CARM PATH
CARM = G/(1+G);

% Common mode boosts
BOOST = zpk(-2*pi*4000, -2*pi*40, 1);
BOOST1 = zpk(-2*pi*20000, -2*pi*1000, 1);
BOOST2 = zpk(-2*pi*20000, -2*pi*1000, 1);
BOOST3 = zpk(-2*pi*4500, -2*pi*300, 1);

% Coupled cavity pole
CCPole = zpk([], -2*pi*100, 2*pi*100);

% Servo gain
Gain = db2mag(43);

% CARM OL with boosts
H = CARM * CCPole * BOOST * Gain;
H1 = H * BOOST1;
H2 = H1 * BOOST2;
H3 = H2 * BOOST3;

% Plot
% bode(H, H1, H2, H3, 2*pi*logspace(3, 5, 10000));
% bode(1/(1+H), 1/(1+H1), 1/(1+H2), 1/(1+H3), 2*pi*logspace(3, 5, 10000));

X,Y arms were stabilized using ALS and moved 5 nm from the resonance, PRMI was locked on sideband using REFL165 I&Q. POP angular servo was running; PRMI RIN was good (~2-3%)

During slow offset reduction I was sweeping MICH, PRCL and POP servos for instabilities due to possible optical gain variations, loops were fine.

I could reduce offset down to ~200 pm and then lost lock due to 60 Hz oscillations as shown on the attached plot "arm_offset"

Arms were stabilized with RMS comparable to the offset and power in arms was fluctuating from 3 to 45.

60 Hz line most probably comes from MICH. RMS is dominated by the power lines and is ~ 1 nm as seen on the plot "PRMI_CAL". I think this is too much but we need to do simulations.

Now the module is inserted at the 2nd crate from the top of 1Y2 (LSC analog rack). It is next to the DCPD whitening module.

I found the backplane cable for the Common Mode servo module.
I traced a cable form XY220 at the right most module on the crate where iscaux2 is living.
This cable was connected to the upper backplane connector.

Switching of the module is tested. All the switches and gain control are doing their job.

It was found that the offset and slow readback are not responding.
I checked the schematic of the CM servo module (D040180).
It seems that there is another cable for the offset and read back voltages.

When PRMI is locked on REFL 165 I&Q signals MICH rms is dominated by the 60 Hz line and harmonics. It comes from demodulation board.

To increase SNR ZFL-100 LN amplifier (+23.5dB) was installed in LSC analog rack. MICH 60 Hz and harmonics are improved as shown on the plot "mich_err"

I have also added a few resg at low frequencies. MICH rms is not 3*10^-10. In Optickle I simulated power dependence in PRC and ARMs on MICH motion. Plot is attached.

 I think we need to stabilize MICH even more, down to ~3*10^-11 . We can think about increasing RF amplifier gain, modulation index and power on BB PD.

CARM offset reduction was a little better today due to improved MICH RMS. Power in arms increases up to 15 and than starts to oscillate up to 70 and then PRMI looses lock.

Tomorrow we need to discuss where to put RF amplifier. Current design has several drawbacks:

• DC power for the amplifier is wired from a custom (not rack based) +15V power supply that was already inside the lsc rack and used for other ZFL-100LN
• BNC cables are used because I could not find any long SMA cables
• we would like gain of ~40 dB instead of 23.5 dB

I found another backplane cable for the CM servo module. It is plugged to the module now.

I can see that C1:LSC-CM_SLOW_MON is responding to C1:LSC-CM_REFL_OFFSET.
But C1:LSC-CM_SUM_MON and C1:LSC-CM_FAST_MON are not replated to the given offset.
I probably need to check the cross connects.

Koji, Den

Some results and conclusions from tonight:

PRC macroscopic length is detuned. We measured REFL phases in carrier and sideband configurations - they are different by ~45 degrees for both 11 and 55 MHz sidebands. Additional measurement with phase locked lasers is required.

We got stable lock of PRMI+2arms with CARM offset of ~200 pm. We think this is the point when we should transition to 1/sqrt(TR) signals. We plan to rewire LSC model and also test CM servo with 1 arm during the day.

POP ASC OL shape changes when we reduce CARM offset probably due to normalization by sum inside the PD. Servo gets almost useless when PRMI power fluctuates by a factor of few.

SMA cables were made and installed for the REFL165 RF amplifier in lsc rack.

 I looked at the BBPD design so that we could make a POP22/110. It looks like it will be easy (I hope).

 The first attachment shows the schematic with the RF notch modified to handle 55 MHz. As long as the capacitor in this notch can be kept to below 20 pF, it doesn't degrade the noise so much,

The second attachment shows the TF and input referred noise. We ought to be able to get 20 pA/rHz at the input to the first RF amplifier.

The LISO files are in the svn under liso/examples/aLIGO_BBPD/,

Later, if we have to notch more than just 55 MHz, we can add a notch between the 2 RF amplifiers as Koji has done for the REFL165.

I noticed that we have not been saving the 1/sqrt(TRX) and 1/sqrt(TRY) data, so I modified the c1lsc model and added them to the DAQ channels block.  I restarted the c1lsc model, and the _DQ channels are now archived.

Den just plugged an output from the common mode board into an LSC whitening board (the spare channel that used to be called "PD_XXX_I" in the LSC model).  I have modified the LSC model to reflect the new name of the new signal ("CM_SLOW"), and have added this channel to the LSC input matrix.  Koji is, I believe, adding this channel to the LSC screen in the auxiliary error signals section.  I am also adding the _OUT of the filter bank to the DAQ channels block.

I have looked at the photo of the Xend QPD from elog 9367, as well as the schematic for the board (D990272). 

Things that will need swapping out:

• Thick film resistors in the signal path need to be changed to thin film.
• MAX333 needs to be replaced with MAX333A.  The 333 has "ON Resistance" of 140-175 ohms, whereas the 333A has "ON Resistance" of 20-35 ohms.
• AD797 needs to be replaced by OPA140.  The 797 is a low voltage noise op-amp, but for a diode we want low current noise.  AD797 has 2pA/rtHz at 1kHz, whereas the OP140 has 0.8fA/rtHz at 1kHz (see Zach's elog 8125 re: OPA140).

I have ordered from digikey via techmart 10 each of the MAX333A's and the OPA140's.  (4 per QPD times 2 QPDs plus 2 spares = 10).  Both of these new chips have the same footprint and pinout as the part that they are replacing, so it'll be a fairly easy task.

Next up, I need to make a LISO model for the circuit for one of the quadrants, to see what shape it'll turn out to be.  Part of this will include deciding what resistors and capacitors to put in the OPA140 gain stage. 

Right now, the AD797s say on the schematic that the gain options are different by a factor of 5, but the actual QPD has a different resistor than is on the schematic, and there is also a capacitor in parallel with each resistor, so I need to just pull those out, and pick some values that make sense to me. 

Rana and I have discussed ignoring the 2nd and 3rd gain switching options on each quadrant, as that is getting to be more fine control than we need. 

Other things on the board:

• The 50 ohm resistors to ground for the "QPD_rtn" have all shorted.  Rana says this is good, so leave it as-is.
• The positive input to the AD797's all have a 100 ohm resistor to ground, rather than just being connected to ground.  Why is this??

For now, I will probably just work on the QPD head, and not the whitening board.  For now, we can run with 1 stage of whitening, and if we need lower noise, we can revisit the whitening board (including replacing the thick film resistors with thin film).

When thinking about what gains I want on my gain stages, I want to have my full arm power (~700 TRX units) be ~20,000 counts from the ADC.  So, I want my single arm power (1 TRX unit) to be ~30 counts from the ADC.  This is not such a big number, so this may also require more thinking. 

As a CM slow path test I locked free swinging yarm by actuating on MC length with bandwidth of 200 Hz. Crossover with AO is not stable so far.

I used xarm as an ool frequency noise sensor. MC2 violin mode is at 645 Hz, I have added a notch filter to LSC-MC2 bank.

 Since we use the TransMon QPD for triggering the high/low gain switching we need to run with the whitening OFF during lock acquisition and the turn it on after we have the arms locked with ALS. This should be put into the up/down scripts.

Somehow the POP22 and POP110 demod phases weren't correct anymore.  I guess Den saw this after he changed the setup for the REFL165 PD at the LSC rack, but didn't elog it. 

I went out to the LSC rack, and found that the power supply that is supplying the amplifiers for both POP22/110 and REFL165 was set to ~16V each channel.  I put it back to 15V for each channel.  I don't know what Den intended for the 165 amplifier (more volts is more gain), but the POP22/110 amplifier usually runs with 15V. 

I also reset the POP22 and POP110 demod phases.  Since I'm not able to lock PRMI on sideband this morning (why?!?!), I locked on the carrier, and moved the phases around until POP22 and POP110 were both maximally negative.  The phases are/were:

This is a ~60 degree change for both PDs. 

I am not sure if Den ever checked the demod phase of REFL165 after he put in the new SMA cable (there's no mention of it in the elog!), so I'm going to check that to see if it helps get PRMI locking back.  I know that Den had also been using REFL11 for PRMI locking, but the parameters he used for that aren't in the log either.

40m crew and visitor Holger Muller from Berkeley.

This is not a test. Life can be dangerous in the 40m Control  Room.

I checked out the REFL165 demod phase, and it looks like it was okay.  it was -20.9 degrees.  I turned on my sensing matrix oscillators, and maximized the PRCL signal in the REFL165_I_ERR channel, and got a pretty good maximization at +155 degrees.  I used this to lock the PRMI on sidebands, with MICH gain of +0.3, and PRCL gain of +0.1 . 

Since this is working, I'm leaving the REFL 165 phase here, at +155 degrees, although this is almost exactly 180 degrees from what Den left it at, so I'm not sure why I was not able to lock with a demod phase of -20.9.  (I tried all 4 permutations of signs, with gain values of the same magnitude (0.3 for MICH, 0.1 for PRCL), and wasn't able to lock.  I'll try to figure this out tomorrow, but it was time for the meeting, then the IFO has been busy doing more important things the rest of the afternoon.

Plan for checking:  Lock with demod phase of 155, measure TF to one of the other REFL diodes (11, 33 or 55), lock on that other REFL diode.  Then, change the REFL 165 demod phase back to -20.9, and measure the transfer function again.  Hopefully the answer is just that I was doing something dumb, and it works easily.  This test/measurement should only take a few minutes, but it'll make me happier knowing that things still work as they should.

Koji, Den

Procedure:

• lock yarm on IR, wire POY to CM input
• transition arm to CM length path by actuating on IMC
• increase AO gain for a stable crossover
• engage CM boosts

Result:

• arm can be kept on resonance and even acquired on MC2
• stable length / AO crossover is achieved
• high bandwidth loop can not be engaged because POY signal is too noisy and EOM is running out of range

We spent some time tuning CM slow servo such that fast path would be stable in the AO gain range -32db -> 29dB (UGF=20kHz) when all boosts are turned off and common gain is 25dB. Current filters that we use for locking are not good enough - AO can not be engaged due to oscillations around 1kHz. This is clearly seen from slow path closed loop transfer function. I will attach servo shapes tomorrow.

Attached plot "EOM" shows EOM rms voltage while changing AO gain from -10dB to 4dB. For UGF of 20kHz we need AO gain of 29dB.

It seems we can start using CM servo for CARM offset but the sensor should be at least factor of 30 better than POY. Add another factor of 10 if we would like to use BOOST 2 and BOOST 3.

I worked on the mitigation of c1mcs time-over issue this afternoon.

The timing for the c1mcs is successfully reduced from >60us to 45us.

The previous models are svned in redoubt as follows:

MCS rev. 6696
RFM rev. 6697
IOO rev. 6698

What I changed was:

- Remove connection from ALS (on c1ioo) to MCS (on c1sus). This should be all done in LSC. (# of RFM IPC in MCS -1)

- MC2 trans QPD filters are moved from IOO to MCS to reduce the RFM channels in MCS.
  Previously the signals for the 4 segments are sent. Now the processed siganls (pit/yaw/sum) are sent. (# of RFM IPC in IOO -1, MCS -1)

- WFS MC3 feedback channels are moved from MCS to RFM to distribute the RFM channels (# of RFM IPC in MCS -2, in RFM +2)

model    prev. timing[us] current timing[us]  diff in time[us]  diff in ch#
c1mcs         >60                45                -15              -4
c1rfm         47                 53                + 6              +2       
c1ioo         47                 36                -11              -1

Revisions of the new models:
MCS rev. 6702
RFM rev. 6701
IOO rev. 6700

Koji, Den

CM Servo with POY11 successfully engaged. UGF: ~15kHz.

Tonight we decided to repeat one arm locking using high-bandwidth CM servo. We low-passed AO signal to avoid saturations of the EOM. We tried different configurations that compromise between noise and loop phase margin and ended up with a pole at 30kHz. SR560 is used as a low-pass filter.

Another problem that we faced was big (~2.6V) electronic offset at the input of 40:4000 BOOST. Once engaged, cavity would be kicked out of lock. We calibrated this offset to be almost half linewidth of the cavity (~300pm). To avoid lock loss during engaging the boost we increased common mode gain to maximum (31 dB).

Measured OL is attached. UGF is 15kHz, phase margin is 60 degrees. We have also simulated evolution of loop shape during bringing AO path. Plot is attached.

The final procedure is

• set common gain up to 31dB, AO gain to 8dB, MC IN2 gain 10dB, CM offset 0.7V
• lock arm with CM slow path with bandwidth of 200 Hz
• enable AO path, gradually increase slow and fast gains by 12 dB
• enable boost

up/down scripts are to be made

(Offset Edit on Dec 20 10:38PM)

Configuration:
POY11QMon -> CM Servo In1 -> CM Servo -->Out1 -> ADC -> CM Slow FM -> LSC MC Servo FM -> ETMY(x1.0) -> DAC -> ETMY
                                       |
                                       -->Servo Out -> SR560 (DC, 1st order 30kHz LPF) -> MC In2

Lock acquisition path 1


Initial condition:

CM Slow FM:

• Gain 2.6

CM Servo setting:

• In1 Gain 31dB, SW:OFF, Offset -1.880, Boost Off, Super Boost Off, AO +8dB

MC Servo setting:

• In2 10dB, SW:OFF

Acquisition:

• Engage LSC
• LSC MC servo gain +0.1, FM7/FM10 (Trigger FM2 with 3sec delay)
• Turn on MC

Transition:

• Enable AO path (CM servo In1 SW:ON, MC servo In2 SW:ON)
• LSC MC gain +0.1 -> +0.2
• AO path gain 8dB->14dB
• LSC MC gain +0.2 -> +0.35
• AO path gain 14dB->18dB
• CM servo offset -1.88 -2.7 -> 0.8 0.0 (gradually)
• Enable CM servo Boost

Lock acquisition path 2

Initial condition:

CM Slow FM:

• Gain 2.6

CM Servo setting:

• In1 Gain 31dB, SW:OFF, Offset -1.880, Boost Off, Super Boost Off, AO +20dB

MC Servo setting:

• In2 10dB, SW:OFF

Acquisition:

• Engage LSC
• LSC Yarm G=+0.25 FM4/5 (Trigger FM3/6/7/8)

Transition:

• Enable MC servo In2 (SW:ON)
• Set LSC MC gain +0.2 FM7/10
• Enable LSC MC (On)
• Enable CM servo In1 (SW:ON)
• Disable LSC Yarm (OFF)
• Change CM servo offset -1.88 -> +0.700 -2.70 -> 0.0
• Enable CM servo Boost
• Turn on LSC CM FM2 (optional)

Transition to ETMY LSC to MCL

• After all of the transition: LSC output matrix ETMY (+1.00)
• LSC output matrix MC2 (-1.00)
• LSC output matrix ETMY (0.00)

This too huge offset difference with/without "BOOST" switch should be checked.

 The lamp lasted for 4,622 hours.

This time I purchased just the bare lamp itself . The housing doubles the price. The disadvantage of this technic that the lamp housing window can not be cleaned  perfectly. Atm2 picture is exaggerating this spot.

However, It does not degrade the image quality.

Koji, Steve

 It was a bad experience again with our vacuum system.  The valves went crazy as we rebooted the computer. This was required for the swap in of a good 24V power supply.

The IFO was vented to 27 Torr through the annuloses, VA6, V7, Maglev,VM2 and VM1 (VC2 was open too)

I just opened the PSL shutter after a 4 hours pumpdown.

Condition: annuloses are not pumped, the IFO and the RGA are pumped as Atm2 shows

I will be here tomorrow morning to switch over to vacuum normal. 

More details later

I checked the offset situation in the CM servo boost circuit. 

- The offset voltage on the CM servo screen is a raw DAC output. This number is diluted by the voltage divider before the amplifier.
  So, the actual offset of the boost circuit was much smaller. (~20mV)

- There is a offset trimmer on the board. This was adjusted so that the boost does not generate an output offset.

- So the default offset is 0V.

- When the arm was locked with (digital) POY11, the CM servo offset is necessary to be -2.7 (now).
  This means that analog POY11Q and digital POY11 has different offset for the best resonance transmission.
  That is believable if POY11I is contributing to the digital POY11 signal.

I'm leaving the 40m now. IFO is aligned. Everything look good.

- The main volume P1=5e-4, CC1=1.4e-5 is still pumped by TP1 and TP2

- RGA P4<0e-4, CC4 2.1e-7, is pumped by TP3

- The annuluses are isolated.

- RP1/2/3 are off.

The recovery- pumpdown reached valve configuration  vacuum normal at 20 hours, cc1 7.7e-6 Torr

Lesson learned: turn all pumps off, close all valves before you reboot ! like you would prepare for AC power shut down.

 The date is correct on this monitor.

Last stored RGA scan data Dec 20, 2013

IFO pressure at CC1 5.8e-6 Torr

Valve configuration: Vacuum Normal, confirmable only by manual checking of position indicators and pressure gauge controllers  readouts

We probably need to restart the machine, but I didn't want to touch c1vac1 and c1vac2. 

 The trend shows a big jolt to the MC1/3 pointing this morning at 8:30.

Was anyone working anywhere near there today? There is no elog.

If not, we will have to put a 'no janitor' sign on all of the 40m doors permanently to prevent mops misaligning our interferometer.

When I got in this morning at 9-something (9:45 maybe?), Steve was taking dust photos on the AS table, of the MC Refl path.  Other than that, I don't have any information. 

Also, Tuesday is our traditional janitor day, so I'm hesitant to put our blame there.  (I think we've kept Tuesdays, even though we're on a less-often schedule....Steve will have to correct me if I'm wrong on this).

 The PRCL once again doesn't want to lock on sidebands for me.  I can lock on the carrier just fine (using the IFO Config settings, along with some hand-alignment of the PRM). 

However, I can't convince it to lock on sidebands.  Using the configs that I used on Dec 18th (elog 9491), I'm not getting it.  I've done the arm ASS alignment, and I've run LSCoffsets, both of which seemed to do their things appropriately. 

I'm going to attribute this today to not being in the groove yet, and I'll look at it again in the morning.

 I was taking pictures at the AP table at the morning and ETMX optical table after noon. There was no activity on the IOO chamber.

 Look at the last 2 hours of  Rana's trend plot. MC1 and MC2 sensor voltage started increasing.

I think it was a drift action.

 NOT drift. The sudden steps are certainly the result of being kicked. The slow drift at the end of the day might be a slow strain relaxation.

It pays to be careful and not put too much weight or impulsive forces on the chambers or tables.

The MC trend for the last 2 days shows that the MC suspensions were kicked again earlier today.  Looking back at the suspension channel INMONs along with the MC trans sum shows that the suspensions get kicked everytime MC locks and unlocks. (Attch:1)

So I checked the effect of WFS on the suspensions by disabling and enabling the WFS feedback servo (Attch:2).

Since the IMC is not at it best pointing, whenever the  MC autolocker runs and enables the WFS, the suspensions look like they are getting kicked.  But really, it's just the WFS doing their job. 

Edit, JCD:  What this really means is that our DC MC pointing is bad, and we need to move the MC suspensions to offload the WFS.  (All of the WFS output numbers for MC1 and 3 were around 100, which is pretty big for those numbers).  We should resurrect the WFS offloading scripts so that we can do this more regularly, and not have to do it by hand.

 Aligned MC to offload the WFS

1. Turned OFF the WFS feedback servo.

2. Aligned the MC suspensions by moving the pit and yaw sliders. MC trans sum brought from ~11000 counts to ~15000 counts. MC RFPD DCMON reads 0.45 counts.

3. Turned ON the WFS servo. The WFS output now reads in the order of 0 to +/-15.

4. Measured the MC spot positions. The spot positions look like they moved for the better compared to what they were yesterday.

[Rana, Jenne]

We turned off the WFS servos, and looked at the MC REFL DC, and saw that it was still good, so we said that since the MC spots are pretty good, that we'll keep this alignment for now. 

Rana put the beam back on the center of the IOO QPDs on the PSL table.

We switched a steering mirror in the WFS path that was the wrong handed-ness to be the correct handed-ness, then put the beam on the centers of the WFS.  We turned on the WFS, and everything seems good.

While we were out on the table, we also changed the anodized aluminum dump for a razor dump, to catch the reflection from the 2inch lens that is the first thing the MC refl path sees out of vac.

There were no major drifts in the WFS error signals while we were gone for dinner, so the MC seems okay for now.

After locking the arms (after the MC alignment work), Manasa and I aligned IP POS, IP ANG, and both end transmission QPDs. 

We noticed that IP ANG is clipping in yaw as it comes onto the end table.  It looks to me like it's clipping on the edge of the plastic box's aperture, but I can't guarantee that it's not also clipping elsewhere. 

The daqd process is segfaulting and restarting itself every 30 seconds or so.  It's pretty frustrating. 

Just for kicks, I tried an mxstream restart, clearing the testpoints, and restarting the daqd process, but none of things changed anything.  

Manasa found an elog from a year ago (elog 7105 and preceding), but I'm not sure that it's a similar / related problem.  Jamie, please help us!

Here is a screen dump from the "dtail":

Every 1.0s: dmesg | tail -50                                                                                                                         Tue Jan  7 22:43:23 2014

[   33.498691]  [<ffffffff8104a063>] kthread+0x7a/0x82
[   33.498695]  [<ffffffff81003654>] kernel_thread_helper+0x4/0x10
[   33.498698]  [<ffffffff81049fe9>] ? kthread+0x0/0x82
[   33.498701]  [<ffffffff81003650>] ? kernel_thread_helper+0x0/0x10
[   33.498703] ---[ end trace 6236defa99b3e091 ]---
[   33.498705] mx INFO: Board 0: allocated MSI IRQ 67
[   33.498713] mx INFO: CPU0: PAT = 0x7010600070106
[   33.498715] mx INFO: CPU0: new PAT = 0x1010600070106
[   33.498718] mx INFO: Board 0: Using PAT index 6
[   33.499101] eth0: no IPv6 routers present
[   33.531013] mx INFO: Board 0: device 8, rev 0, 1 ports and 2096896 bytes of SRAM available
[   33.531017] mx INFO: Board 0: Bridge is 10de:005d
[   33.531228] mx INFO: Board 0: MAC address = 00:60:dd:46:ea:ec
[   33.535971] mx INFO: Loaded mcp of len 235448
[   34.489244] mx INFO: Starting usermode mapper at /opt/mx/sbin/mx_start_mapper
[   39.148855] mx INFO: mx0: Link0 is UP
[   39.588511] mx INFO: myri0: Will use skbuf frags (4096 bytes, order=0)
[   39.589299] mx INFO: 1 Myrinet board found and initialized
[  287.706367] daqd used greatest stack depth: 3368 bytes left
[86605.907520] daqd[18407]: segfault at 38b08e4c0 ip 00007f11b3942a6c sp 00007f10b1917d50 error 4
[86605.907530] daqd[18424]: segfault at 38b544f90 ip 00007f11b3942a6c sp 00007f10b12c6d30 error 4 in libc-2.10.1.so[7f11b390e000+14c000] in libc-2.10.1.so[7f11b390e000+14c00
0]
[86605.907544]
[86605.919454] daqd[21319] general protection ip:7f11b3942a6c sp:7f10b1814d30 error:0
[86605.919462] daqd[18442] general protection ip:7f11b3942a6c sp:7f10b0bf4d30 error:0
[86605.919615] daqd[18443]: segfault at 38aee3db0 ip 00007f11b3942a6c sp 00007f10b0b73d50 error 4 in libc-2.10.1.so[7f11b390e000+14c000]
[86605.919694] daqd[18412]: segfault at 38aff35d0 ip 00007f11b3942a6c sp 00007f10b1752d30 error 4
[86605.919701] daqd[18417]: segfault at 38b544f70 ip 00007f11b3942a6c sp 00007f10b154dd50 error 4 in libc-2.10.1.so[7f11b390e000+14c000]
[86605.919708] daqd[18445]: segfault at 38aff35b0 ip 00007f11b3942a6c sp 00007f10b0ab1d50 error 4
[86605.919733] daqd[18429]: segfault at 38b42ae90 ip 00007f11b3942a6c sp 00007f10b10c1d50 error 4 in libc-2.10.1.so[7f11b390e000+14c000]
[86605.919741] daqd[18440]: segfault at 38b08e480 ip 00007f11b3942a6c sp 00007f10b0cb6d30 error 4 in libc-2.10.1.so[7f11b390e000+14c000]
[86605.958551]  in libc-2.10.1.so[7f11b390e000+14c000] in libc-2.10.1.so[7f11b390e000+14c000]
[86605.958557]
[86605.958577]  in libc-2.10.1.so[7f11b390e000+14c000]
[86605.958586]  in libc-2.10.1.so[7f11b390e000+14c000]
[86605.959639] daqd used greatest stack depth: 3160 bytes left
[98139.100888] show_signal_msg: 13 callbacks suppressed
[98139.100895] daqd[23753]: segfault at 39c7363b0 ip 00007f5bf253ba6c sp 00007f5b69b48d30 error 4 in libc-2.10.1.so[7f5bf2507000+14c000]
[98687.815120] daqd used greatest stack depth: 2984 bytes left
[208995.594227] daqd[10386] general protection ip:7f3b7c930a6c sp:7f3a79f09d50 error:0 in libc-2.10.1.so[7f3b7c8fc000+14c000]
[353015.067479] daqd used greatest stack depth: 2880 bytes left
[367406.863618] daqd[13078]: segfault at 41 ip 0000000000000041 sp 00007fb1f0ba2cf8 error 14 in daqd[400000+7c000]
[367406.863833] daqd[13104] general protection ip:7fb2f3018a6c sp:7fb1f01c8d30 error:0
[367406.863877] daqd[13086] general protection ip:7fb2f3018a6c sp:7fb1f089ad30 error:0
[367406.877408] daqd[13080]: segfault at 41 ip 0000000000000041 sp 00007fb1f0ae0ca8 error 14 in daqd[400000+7c000]
[367406.877435]  in libc-2.10.1.so[7fb2f2fe4000+14c000]
[367406.877442] daqd[13100]: segfault at 39ba287b0 ip 00007fb2f3018a6c sp 00007fb1f034cd30 error 4 in libc-2.10.1.so[7fb2f2fe4000+14c000]
[367406.878372]  in libc-2.10.1.so[7fb2f2fe4000+14c000]
[399802.887523] daqd[18295] general protection ip:7fb056a71a6c sp:7faf96125f10 error:0 in libc-2.10.1.so[7fb056a3d000+14c000]
[410595.969327] daqd[22057]: segfault at 3a91f27b0 ip 00007f48e96eea6c sp 00007f47e6c26d50 error 4 in libc-2.10.1.so[7f48e96ba000+14c000]
[410595.988926] daqd[22068]: segfault at 3a91f2790 ip 00007f48e96eea6c sp 00007f47e681bd30 error 4 in libc-2.10.1.so[7f48e96ba000+14c000]

The problem is not exactly the same as what's described in 7105, but the symptoms are so similar I assumed they must have a similar source.

And sure enough, /frames is completely full:

controls@fb /opt/rtcds/caltech/c1/target/fb 0$ df -h /frames/
Filesystem            Size  Used Avail Use% Mounted on
/dev/sda1              13T   13T     0 100% /frames
controls@fb /opt/rtcds/caltech/c1/target/fb 0$

So the problem in both cases was that it couldn't write out the frames.  Unfortunately daqd is apparently too stupid to give us a reasonable error message about what's going on.

So why is /frames full?  Apparently the wiper script is either not running, or is failing to do it's job.  My guess is that this is a side effect of the linux1 raid failure we had over xmas.

 The last 4 hour trend for WFS error signals show some amount of drift. We should still look at the long term trend to solve the issue.

It actually looks like the wiper script has been running fine.  There is a log from Tuesday morning:

controls@fb ~ 0$ cat /opt/rtcds/caltech/c1/target/fb/wiper.log

Tue Jan  7 06:00:02 PST 2014

Directory disk usage:
/frames/trend/minute_raw 385289132k
/frames/trend/second 100891124k
/frames/full 12269554048k
/frames/trend/minute 1906772k
Combined 12757641076k or 12458633m or 12166Gb

/frames size 13460088620k at 94.78%
/frames is below keep value of 95.00%
Will not delete any files
df reported usage 97.72%
controls@fb ~ 0$

So now I'm wondering if something else has been filling up the frames today.  Has anything changed today that might cause more data than usual to be written to frames?

I'm manually running the wiper script now to clear up some /frames.  Hopefully that will solve the problem temporarily.

It seems that the most important short-term task we have right now is to figure out what our PRC length is, and what our tolerance from nominal is.  Gabriele and EricQ are going to work on that tomorrow.  If our PRC is of a length that we can't do anything useful for full IFO locking, we need to open up and fix it sooner rather than later.

While we're in there, we need to also put a baffle on the back side of the PRM cage, to protect the OSEMs from stray light.  Den and I discovered before Christmas that turning off the OSEM and OpLev damping to the PRM (while using the POP QPD for ASC) significantly reduced the power fluctuations in the PRC.  We still had arm power fluctuations, but I believe those are likely because our ALS system can't hold an arm precisely at full resonance.  So, putting a black glass baffle with ~2 inch aperture right up against the OSEMs should help a lot.  This week, I'll ask Steve to make me a quickie to-scale cardboard version of the baffles that he has had cut, so I can try securing it to the dirty suspension cage that we have out.  I will also check to make sure I have seen with my own eyes the baffles that I need, and copper wire to tie it to the cage.

Other, lower-priority things that we should do eventually:

* Steve, please find another razor beam dump for the WFS reflections - Rana and I used one of the ones that was there for reflection off the 2 inch lens in the MC refl path (replacing the aluminum dump that has been there for ages).  We also need to label all of our razor dumps with their purpose, with a label on top, so we remember not to remove dumps that are actually in use.

* At some point, we should change the one remaining steering mirror in the main PSL path that is aluminum, to a steel Polaris ("Polanski" or "Polish") mount.  For now, we should just make sure we have one handy.  Hopefully this will help reduce the PMC transmission drift that we see.

* Steve, in the morning sometime this week, can you please do a test of the drift of the IOO QPDs?  We'd like to see a trend that is maybe 30 or 60 minutes long of the QPD signals.  First 10 minutes, all lights in IFO room off.  Then, 10 minutes with the lights in the PSL on.  Then, the rest of the time the PSL lights off.  We want to see if these are hot enough to be causing a big temperature change in the PSL box, which may then be causing some optics to drift.

* QPD code in the simulink models (trans QPDs, but also OpLevs, and anywhere else we do normalization) needs to have anti-divide-by-zero protection.  I'll take care of this, it should be a quick copy of what we have elsewhere in the simulink code.

* Note to self for the future, instead of doing a dither alignment for the ASS for the arms, we can use the IP POS and IP ANG, as well as end transmission QPD signals.  However, for now, the ASS is working just fine.

* We want to go back to the idea of putting a lens into the in-vac IP ANG path, to avoid the clipping that Manasa and I were seeing tonight.  We want something of order 2inch diameter, 1meter focal length.  The material doesn't matter, but we do want it AR coated for 1064nm on both sides.  We also need to make sure that we could use a fixed 2 inch in-vac mirror mount, or something, to hold this lens.  If that won't work, we need to come up with another plan.  Manasa is working on thinking about precisely what lens we want to buy for a nice guoy phase telescope for IPANG, so we'll buy a lens after she puts her conclusions in the elog.

* An idea for the MC spots plot that Rana had was to plot the beam tilt and translation, rather than the raw spot positions on the mirrors.  The point of this would be to make it easier to see what the output beam from the MC looks like.  For MC pointing, we should also think about what our actual tolerances are.  The biggest thing is that we need to get through the Faraday without being too close to any edge, and also the REFL beam needs to come back through without clipping.  For now, we're just visually checking that the POP beam and the REFL beam both look unclipped since we don't have access to good camera views of either side of the Faraday.

The wiper script is done and deleted a whole bunch of stuff to clean up some space:

controls@fb ~ 0$ /opt/rtcds/caltech/c1/target/fb/wiper.pl --delete

Tue Jan  7 23:09:21 PST 2014

Directory disk usage:
/frames/trend/minute_raw 385927520k
/frames/trend/second 125729084k
/frames/full 12552144324k
/frames/trend/minute 2311404k
Combined 13066112332k or 12759875m or 12460Gb

/frames size 13460088620k at 97.07%
/frames above keep value of 95.00%
Frame area size is 12401156668k
/frames/full size 12552144324k keep 11781098835k
/frames/trend/second size 125729084k keep 24802313k
/frames/trend/minute size 2311404k keep 620057k
Deleting some full frames to free 771045488k
- /frames/full/10685/C-R-1068567600-16.gwf
- /frames/full/10685/C-R-1068567616-16.gwf
...
controls@fb ~ 0$ df -h /frames
Filesystem            Size  Used Avail Use% Mounted on
/dev/sda1              13T   12T  826G  94% /frames
controls@fb ~ 0$

So it cleaned up 826G of space.  It looks like the fb is stabilized for the moment.  On site folks should confirm...

asdfasdfsadf sadf asdf

dadq is logging the following error messages to it's log related to the fact that it can't connect to c1vac1 and c1vac2:

CAC: Unable to connect because "Connection timed out"
CA.Client.Exception...............................................
    Warning: "Virtual circuit disconnect"
    Context: "c1vac2.martian:5064"
    Source File: ../cac.cpp line 1127
    Current Time: Tue Jan 07 2014 23:50:53.355609430
..................................................................
CAC: Unable to connect because "Connection timed out"
CA.Client.Exception...............................................
    Warning: "Virtual circuit disconnect"
    Context: "c1vac1.martian:5064"
    Source File: ../cac.cpp line 1127
    Current Time: Tue Jan 07 2014 23:50:53.356568469
..................................................................

 Not sure if this is related to the full /frames issue that we've been seeing.

Steve may actually be onto something with the clamps that he had made a year and a half ago.  These clamps hold the glass, and then clamp to the base of the suspension cage.  Not the table, but the base of the suspension cage. The drawings are in elog 6344.  I'm not sure that the 1/4-20 holes in the clamp things are exactly where we'll want them, but we should be able to just dog it down to the base of the suspension.  I need to check this, but it may be even easier than tieing the glass to the cage.

Also, something to think about is that the earthquake stop screws extend backwards farther than the OSEMs.  I'm not sure anymore if we have shorter 1/4-20 earthquake stops around (if we do, they should be in the cleanroom shelves), but if we can't swap those out, they'll limit how close we can get to the OSEMs. 

Here's an overhead photo from 6 Sept 2012: