I was in the lab, near the south end of the ITMX oplev table, looking for something, and I bumped the POP ASC QPD's power supply.  I thought that it was fine, but did not adequately check it.  When EricQ asked me just now about why the PRC is so wobbly today, I checked, and the power for the QPD wasn't properly connected (it's kind of a crappy connector, that if you nudge, contacts or loses contact).  Anyhow, I restored power to the QPD, and the PRC looks a little more stable now.  My fault for not checking more carefully, and my apologies to Q and Gabriele for their frustrations this afternoon.

 I analyzed the data we took yesterday, both using AS11 and AS55. For each value of the phase I estimated the Q/P ratio using a demodulation code. Then I used a linear regression fit to estimate the zero crossing point.

Here are the plots of the data points with the fits:

The measurements a re more noisy in the PRMI configuration, as expected since we had a lot of angular motion. Also, the AS11 data is more noisy. However, the estimated phase differences between PRMI and MICH configurations are:

• AS11 = -10.9 +- 1.0 degrees
• AS55 = -21.1 +- 0.4 degrees

The simulation already described in slogs 9539 and 9541 provides the calibration in terms of PRC length. Here are the curves

The corresponding length errors are

• AS11 = 1.44 +- 0.13 cm
• AS55 = 0.59 +- 0.01 cm

The two results are not consistent one with the other and they are both not consistent with the previous estimate of 4 cm based on the 55MHz sideband peak splitting.

I don't know the reason for this incongruence. I checked the simulation, repeating it with Optickle and I got the same results. So I'm confident that the simulation is not completely wrong.

I also tried to understand which parameters of the IFO can affect the result. The following ones have no impact

• Beam matching
• ITM curvatures
• Schnupp asymmetry
• Distance PR-BS
• ITM and PRM misalignments

The only parameters that could affect the curves are offsets in MICH and PRCL locking point. We should check if this is happening. A first quick look (with EricQ) seems to indicate that we indeed have an offset in PRCL. However, tonight the PRMI is not locking stably on the sidebands. 

If possibile, we will repeat the measurement later on tonight, checking first the PRCL offset.

It looks like there was a 4.4 magnitude earthquake near Fontana, CA around 1:30am today.  This tripped all of the suspension watchdogs, which Q has just now re-enabled. 

 [EricQ, Gabriele]

We could carry out the measurement of PRC length. The AS110 photodiode was plugged into REFL11. So REFL11 is giving us the AS11 signal. Here is the procedure.

1. Lock MICH.
2. Add a line in MICH (amplitude 20000 counts)
3. Tune AS11 demod phase to have the line in I.
4. Change the demod phase by steps of 1 degree around the rough optimum, taking one minute of data at each point
5. Lock PRMI on sidebands
6. Add a line in MICH (amplitude 500 counts)
7. Tune AS11 demod phase to have the line in I.
8. Change the demod phase by steps of 1 degree around the rough optimum, taking one minute of data at each point

We repeated the same measurement also using AS55, with the same procedure.

Roughly, the phase difference for AS11 was 11 degrees and for AS55 it was 23 degrees. A more detailed analysis and a calibration in terms of PRC length will follow.

[Gabriele, EricQ] 

Finally, we managed to lock PRMI on sidebands:

• MICH locked on AS55_Q with gain -10. Demod phase of AS55=18
• PRCL locked on REFL55_I with gain -0.04. Demod phase of REFL55=88
• Triggering on POP110_I at 50/10
• Filter "1:5" of MICH engaged, this improved a lot the stability

GariLyn is using our green light on the west side of the PSL table. The green PDA36As were moved and the HEPA turned up to 60V

 I do like to move IOO-QPD_POS temporarily to see that the feedback has anything to do with with the pointing.

[Gabriele, EricQ]

We wanted to try the PRC length measurement,but we ended up spending all the afternoon to lock the PRMI on sidebands. Here are some results

• Lock of PRM on carrier is easy with MICH on AS55_Q and PRCL on REFL55_I. The optimal gains to avoid correction gain peaking are MICH=-5 and PRCL=0.012. The alignment today was must more stable over time than on Friday
• We wanted to move MICH on REFL55_Q. After a few trials we discovered that REFL55_Q is not seeing any MICH signal at all. This is quite strange and we don't understand this.
• Locking PRMI on carrier using REFL165_I and Q was difficult. We thought that was due to a RF amplifier installed in December. We tried to remove it, but since this did not help, we put it back
• We could lock PRMI on carrier and sidebands using REFL11 signals. The optimal demodulation phase is 155. Lock on carrier was achieved with gains MICH=5, PRCL=0.7. Lock on sidebands with MICH gain=-10 and PRCL gain=-0.3. The MICH correction is very high and exciting a tone at few hundreds Hz. Maybe a violin mode of the ITMs?
• Finally we could tune the REFL165 phase to 118.5 and lock on carrier. The lock was not very stable. Gain are: lock on carrier MICH=-0.5, PRCL=0.05. The error signal has a very big offset: to get a dark fringe we had to add a MICH offset of 500 counts. We also had to engage the CLP400 filter to avoid saturating ITMs corrections with MICH.

 The Osaka TG390MCAB maglev turbo pump's controller TC010M has passed the 40,000 hrs of operation. This triggered the " alarm" LED  warning light to come on. 

It is normal maintenance.  Maglev TP-1 is running perfectly.  Osaka will send us a loaner-controller that we can use while they do the std maintenance.

I'm thinking of ~ February to do this.

 Three good days of IOO pointing: Friday, Sat and Sun    What was changed?  May be the clamping on Friday?

IOO vertical changes recovering as tempeture. IP is clipping at plastic enclosure of ETMY

NOTE: ANTS at the PSL optical table.  I will mop with chemicals tomorrow if we see more.

 Then there must be something else slowly drifting. It was very clear that the good alignment of the IFO was every time lost after few minutes...

 IOO pointing is drifting in pitch. I'd like to use a QPD instead of the paper target to see if the Innolite output is stable. The idea is to move temporarily IOO-QPD_POS to  this location

 Its very doubtful that the MC yaw drift matters for the IFO. That's just a qualitative correlation; the numbers don't hang together.

I looked at IOO QPDs again. QPD_POS was clamped by one screw. Dog clamp was added on the unclamped side.

QPD_ANG chassis has no isolation to optical table..._POS has.  QPD_ANG  base was tightened also.

Both QPDs moved a little bit but I did not centered them.  The spot sizes are 2-3 mm  They should be smaller.

How ever, we still can not explane the pitch movement of the IOO beam

Razor beam dumps were labeled at the AP table.

The 40m roof was cleaned from leafs this morning.

[ericq, Gabriele, Manasa]

 We wanted to perform the PRC length measurement today with an AS11 signal, but such a signal didn't exist. So, we have temporarily connected the AS110 PD signal (which is some Thorlabs PD, and not a resonant one) into the REFL11 demod board. 

We then proceeded with the goal of locking the PRC with REFL165. A few parameters that were changed along the way as we aligned and locked things:

• the XARM gain was increased from 0.4 to 0.5 to help it acquire lock
• the MICH gain was decreased from -10 to -5 since there was some gain peaking in its servo output
• the REFL165 demodulation phase was changed from 155 to 122, to place a PRCL excitation entirely within I (we did this while locked on the carrier)

Sadly, in the end, we couldn't lock the PRC on a sideband in a stable manner. The alignment would drift faster than we could optimize the alignment and gains for the PRC. I.e. we would lock the PRC on the carrier, align PRM (and maybe touch ITMX) to maximize POPDC, switch to sideband locking, try to lock, and things would start looking misaligned. Switching back to carrier locking, the beam spots on REFL (for example) would have moved.

Manasa noted the MC_TRANS_Y has been substantially drifting along with small drift in MC_TRANS_P as well. So we need to fix the source of the mode cleaner beam drifting if we want to make this measurement. 

Can you please explain this? I don't understand what exactly is the issue or 'great idea'.

I think we should be OK with just a single lens in the vacuum. But what we need is the ray tracing analysis to show what the effect will be on the IPANG readout.

  IOO QPDs tested in dark, lighted and open PSL enclosure. The created temperature change 0.03 C has  effect on monitoring  in pitch.

 Atm1,  all lights off 10 min, PSL enclosure lights on  10 min, all lights off 15 min, open  door # 11 at north east corner of enclosure ( HEPA filters are running at 30V ) for 10 min, closed-dark enclosure 15 min

              dark 10, lighted 10, dark 15, open-dark 10 and closed-dark 15 minutes

Atm2, Pitch drift of 24 hours does not recover

 [Gabriele, EricQ]

Actually it is difficult to see any laser frequency line in the dark fringe signal, since the Schnupp asymmetry is small. It is much better to use a differential MICH excitation which gives a better signal at the dark port.

We repeated the simulation explained before. We can use both the AS55 or the AS11 signals, bout the first one has a limited linear range and the expected 4cm value is very close to saturation.

For the IPANG telescope design, we are in the 'beyond the Rayleigh range' regime. So using a single lens to make the beam small is not a great idea. I have put down a solution where we use a pair of lenses; one of which will be mounted in-vacuum in the ETMY chamber and the other on the endtable.
This way we will also allow have some freedom to configure the layout out-of vacuum in case the need arises. The layout will look something like in the cartoon:

I also made a choice of using longer focal length lenses (CVI 2" lenses f =1 m). Below is the beam path summary for IPANG telescope. I have used the waist diameter at the ITM for propagation. The endtable is roughly at 41.2m. The QPD will be placed in front of the waist (w0=47um).
 

 [ericq,Gabriele]

So, we want an relatively quick measurement of the PRC length error (with sign!) at the order of .5 centimeter or so. Rana suggested the "demodulation phase method," i.e. lock the simple Michelson, measure what demodulation phase brings the 1F signal entirely within the phase quadrature, then lock the PRMI and measure the demodulation phase again. This tells you something about the length of the PRC. 

Gabriele and I worked through a simulation using MIST to determine how to actually do this. We simulated the case of injecting a line at 1kHz in the laser frequency via the laser's PZT and looking at the transfer function of the 1kHz signal to the I and Q at the 1F AS demodulated signal when locked. (Michelson locked on the dark fringe, PRC locked on 11MHz sideband) With the I and Q in hand, we can measure some demodulation phase angle that would bring everything into I. 

When the PRC length is in the ideal location, the demodulation phases in the two cases are the just about the same. Sweeping the length of the PRC around the ideal length gives us a monotonic function in the difference in the demodulation phases:

So, with this simulation, we should be able to calibrate a measured difference in demod phase into the length error of the cavity! We will proceed and report...

Steve may actually be onto something with the clamps that he had made a year and a half ago.  These clamps hold the glass, and then clamp to the base of the suspension cage.  Not the table, but the base of the suspension cage. The drawings are in elog 6344.  I'm not sure that the 1/4-20 holes in the clamp things are exactly where we'll want them, but we should be able to just dog it down to the base of the suspension.  I need to check this, but it may be even easier than tieing the glass to the cage.

Also, something to think about is that the earthquake stop screws extend backwards farther than the OSEMs.  I'm not sure anymore if we have shorter 1/4-20 earthquake stops around (if we do, they should be in the cleanroom shelves), but if we can't swap those out, they'll limit how close we can get to the OSEMs. 

Here's an overhead photo from 6 Sept 2012:

I ran a simulation of a double cavity with a PRC length mismatched w.r.t. the modulation frequency. I summarized the results in the attached PDF. I think it would be important to have a cross check of the results.

In brief:

A mismatch between PRC length and modulation frequency do have an effect on error signals

Multiple zeros appear in REFL_3f/PRCL that can be removed by careful tuning of the demodulation phase (however, the shape of the signal makes difficult to understand which phase is good…)

No visible effect on REFL_1f/CARM

But a large PRCL signal appears in REFL_1f_I, which is used to control CARM. This is not good.

A mismatch of the order of 0.5 cm has a small effect.

dadq is logging the following error messages to it's log related to the fact that it can't connect to c1vac1 and c1vac2:

CAC: Unable to connect because "Connection timed out"
CA.Client.Exception...............................................
    Warning: "Virtual circuit disconnect"
    Context: "c1vac2.martian:5064"
    Source File: ../cac.cpp line 1127
    Current Time: Tue Jan 07 2014 23:50:53.355609430
..................................................................
CAC: Unable to connect because "Connection timed out"
CA.Client.Exception...............................................
    Warning: "Virtual circuit disconnect"
    Context: "c1vac1.martian:5064"
    Source File: ../cac.cpp line 1127
    Current Time: Tue Jan 07 2014 23:50:53.356568469
..................................................................

 Not sure if this is related to the full /frames issue that we've been seeing.

The wiper script is done and deleted a whole bunch of stuff to clean up some space:

controls@fb ~ 0$ /opt/rtcds/caltech/c1/target/fb/wiper.pl --delete

Tue Jan  7 23:09:21 PST 2014

Directory disk usage:
/frames/trend/minute_raw 385927520k
/frames/trend/second 125729084k
/frames/full 12552144324k
/frames/trend/minute 2311404k
Combined 13066112332k or 12759875m or 12460Gb

/frames size 13460088620k at 97.07%
/frames above keep value of 95.00%
Frame area size is 12401156668k
/frames/full size 12552144324k keep 11781098835k
/frames/trend/second size 125729084k keep 24802313k
/frames/trend/minute size 2311404k keep 620057k
Deleting some full frames to free 771045488k
- /frames/full/10685/C-R-1068567600-16.gwf
- /frames/full/10685/C-R-1068567616-16.gwf
...
controls@fb ~ 0$ df -h /frames
Filesystem            Size  Used Avail Use% Mounted on
/dev/sda1              13T   12T  826G  94% /frames
controls@fb ~ 0$

So it cleaned up 826G of space.  It looks like the fb is stabilized for the moment.  On site folks should confirm...

asdfasdfsadf sadf asdf

It seems that the most important short-term task we have right now is to figure out what our PRC length is, and what our tolerance from nominal is.  Gabriele and EricQ are going to work on that tomorrow.  If our PRC is of a length that we can't do anything useful for full IFO locking, we need to open up and fix it sooner rather than later.

While we're in there, we need to also put a baffle on the back side of the PRM cage, to protect the OSEMs from stray light.  Den and I discovered before Christmas that turning off the OSEM and OpLev damping to the PRM (while using the POP QPD for ASC) significantly reduced the power fluctuations in the PRC.  We still had arm power fluctuations, but I believe those are likely because our ALS system can't hold an arm precisely at full resonance.  So, putting a black glass baffle with ~2 inch aperture right up against the OSEMs should help a lot.  This week, I'll ask Steve to make me a quickie to-scale cardboard version of the baffles that he has had cut, so I can try securing it to the dirty suspension cage that we have out.  I will also check to make sure I have seen with my own eyes the baffles that I need, and copper wire to tie it to the cage.

Other, lower-priority things that we should do eventually:

* Steve, please find another razor beam dump for the WFS reflections - Rana and I used one of the ones that was there for reflection off the 2 inch lens in the MC refl path (replacing the aluminum dump that has been there for ages).  We also need to label all of our razor dumps with their purpose, with a label on top, so we remember not to remove dumps that are actually in use.

* At some point, we should change the one remaining steering mirror in the main PSL path that is aluminum, to a steel Polaris ("Polanski" or "Polish") mount.  For now, we should just make sure we have one handy.  Hopefully this will help reduce the PMC transmission drift that we see.

* Steve, in the morning sometime this week, can you please do a test of the drift of the IOO QPDs?  We'd like to see a trend that is maybe 30 or 60 minutes long of the QPD signals.  First 10 minutes, all lights in IFO room off.  Then, 10 minutes with the lights in the PSL on.  Then, the rest of the time the PSL lights off.  We want to see if these are hot enough to be causing a big temperature change in the PSL box, which may then be causing some optics to drift.

* QPD code in the simulink models (trans QPDs, but also OpLevs, and anywhere else we do normalization) needs to have anti-divide-by-zero protection.  I'll take care of this, it should be a quick copy of what we have elsewhere in the simulink code.

* Note to self for the future, instead of doing a dither alignment for the ASS for the arms, we can use the IP POS and IP ANG, as well as end transmission QPD signals.  However, for now, the ASS is working just fine.

* We want to go back to the idea of putting a lens into the in-vac IP ANG path, to avoid the clipping that Manasa and I were seeing tonight.  We want something of order 2inch diameter, 1meter focal length.  The material doesn't matter, but we do want it AR coated for 1064nm on both sides.  We also need to make sure that we could use a fixed 2 inch in-vac mirror mount, or something, to hold this lens.  If that won't work, we need to come up with another plan.  Manasa is working on thinking about precisely what lens we want to buy for a nice guoy phase telescope for IPANG, so we'll buy a lens after she puts her conclusions in the elog.

* An idea for the MC spots plot that Rana had was to plot the beam tilt and translation, rather than the raw spot positions on the mirrors.  The point of this would be to make it easier to see what the output beam from the MC looks like.  For MC pointing, we should also think about what our actual tolerances are.  The biggest thing is that we need to get through the Faraday without being too close to any edge, and also the REFL beam needs to come back through without clipping.  For now, we're just visually checking that the POP beam and the REFL beam both look unclipped since we don't have access to good camera views of either side of the Faraday.

It actually looks like the wiper script has been running fine.  There is a log from Tuesday morning:

controls@fb ~ 0$ cat /opt/rtcds/caltech/c1/target/fb/wiper.log

Tue Jan  7 06:00:02 PST 2014

Directory disk usage:
/frames/trend/minute_raw 385289132k
/frames/trend/second 100891124k
/frames/full 12269554048k
/frames/trend/minute 1906772k
Combined 12757641076k or 12458633m or 12166Gb

/frames size 13460088620k at 94.78%
/frames is below keep value of 95.00%
Will not delete any files
df reported usage 97.72%
controls@fb ~ 0$

So now I'm wondering if something else has been filling up the frames today.  Has anything changed today that might cause more data than usual to be written to frames?

I'm manually running the wiper script now to clear up some /frames.  Hopefully that will solve the problem temporarily.

 The last 4 hour trend for WFS error signals show some amount of drift. We should still look at the long term trend to solve the issue.

The problem is not exactly the same as what's described in 7105, but the symptoms are so similar I assumed they must have a similar source.

And sure enough, /frames is completely full:

controls@fb /opt/rtcds/caltech/c1/target/fb 0$ df -h /frames/
Filesystem            Size  Used Avail Use% Mounted on
/dev/sda1              13T   13T     0 100% /frames
controls@fb /opt/rtcds/caltech/c1/target/fb 0$

So the problem in both cases was that it couldn't write out the frames.  Unfortunately daqd is apparently too stupid to give us a reasonable error message about what's going on.

So why is /frames full?  Apparently the wiper script is either not running, or is failing to do it's job.  My guess is that this is a side effect of the linux1 raid failure we had over xmas.

The daqd process is segfaulting and restarting itself every 30 seconds or so.  It's pretty frustrating. 

Just for kicks, I tried an mxstream restart, clearing the testpoints, and restarting the daqd process, but none of things changed anything.  

Manasa found an elog from a year ago (elog 7105 and preceding), but I'm not sure that it's a similar / related problem.  Jamie, please help us!

Here is a screen dump from the "dtail":

Every 1.0s: dmesg | tail -50                                                                                                                         Tue Jan  7 22:43:23 2014

[   33.498691]  [<ffffffff8104a063>] kthread+0x7a/0x82
[   33.498695]  [<ffffffff81003654>] kernel_thread_helper+0x4/0x10
[   33.498698]  [<ffffffff81049fe9>] ? kthread+0x0/0x82
[   33.498701]  [<ffffffff81003650>] ? kernel_thread_helper+0x0/0x10
[   33.498703] ---[ end trace 6236defa99b3e091 ]---
[   33.498705] mx INFO: Board 0: allocated MSI IRQ 67
[   33.498713] mx INFO: CPU0: PAT = 0x7010600070106
[   33.498715] mx INFO: CPU0: new PAT = 0x1010600070106
[   33.498718] mx INFO: Board 0: Using PAT index 6
[   33.499101] eth0: no IPv6 routers present
[   33.531013] mx INFO: Board 0: device 8, rev 0, 1 ports and 2096896 bytes of SRAM available
[   33.531017] mx INFO: Board 0: Bridge is 10de:005d
[   33.531228] mx INFO: Board 0: MAC address = 00:60:dd:46:ea:ec
[   33.535971] mx INFO: Loaded mcp of len 235448
[   34.489244] mx INFO: Starting usermode mapper at /opt/mx/sbin/mx_start_mapper
[   39.148855] mx INFO: mx0: Link0 is UP
[   39.588511] mx INFO: myri0: Will use skbuf frags (4096 bytes, order=0)
[   39.589299] mx INFO: 1 Myrinet board found and initialized
[  287.706367] daqd used greatest stack depth: 3368 bytes left
[86605.907520] daqd[18407]: segfault at 38b08e4c0 ip 00007f11b3942a6c sp 00007f10b1917d50 error 4
[86605.907530] daqd[18424]: segfault at 38b544f90 ip 00007f11b3942a6c sp 00007f10b12c6d30 error 4 in libc-2.10.1.so[7f11b390e000+14c000] in libc-2.10.1.so[7f11b390e000+14c00
0]
[86605.907544]
[86605.919454] daqd[21319] general protection ip:7f11b3942a6c sp:7f10b1814d30 error:0
[86605.919462] daqd[18442] general protection ip:7f11b3942a6c sp:7f10b0bf4d30 error:0
[86605.919615] daqd[18443]: segfault at 38aee3db0 ip 00007f11b3942a6c sp 00007f10b0b73d50 error 4 in libc-2.10.1.so[7f11b390e000+14c000]
[86605.919694] daqd[18412]: segfault at 38aff35d0 ip 00007f11b3942a6c sp 00007f10b1752d30 error 4
[86605.919701] daqd[18417]: segfault at 38b544f70 ip 00007f11b3942a6c sp 00007f10b154dd50 error 4 in libc-2.10.1.so[7f11b390e000+14c000]
[86605.919708] daqd[18445]: segfault at 38aff35b0 ip 00007f11b3942a6c sp 00007f10b0ab1d50 error 4
[86605.919733] daqd[18429]: segfault at 38b42ae90 ip 00007f11b3942a6c sp 00007f10b10c1d50 error 4 in libc-2.10.1.so[7f11b390e000+14c000]
[86605.919741] daqd[18440]: segfault at 38b08e480 ip 00007f11b3942a6c sp 00007f10b0cb6d30 error 4 in libc-2.10.1.so[7f11b390e000+14c000]
[86605.958551]  in libc-2.10.1.so[7f11b390e000+14c000] in libc-2.10.1.so[7f11b390e000+14c000]
[86605.958557]
[86605.958577]  in libc-2.10.1.so[7f11b390e000+14c000]
[86605.958586]  in libc-2.10.1.so[7f11b390e000+14c000]
[86605.959639] daqd used greatest stack depth: 3160 bytes left
[98139.100888] show_signal_msg: 13 callbacks suppressed
[98139.100895] daqd[23753]: segfault at 39c7363b0 ip 00007f5bf253ba6c sp 00007f5b69b48d30 error 4 in libc-2.10.1.so[7f5bf2507000+14c000]
[98687.815120] daqd used greatest stack depth: 2984 bytes left
[208995.594227] daqd[10386] general protection ip:7f3b7c930a6c sp:7f3a79f09d50 error:0 in libc-2.10.1.so[7f3b7c8fc000+14c000]
[353015.067479] daqd used greatest stack depth: 2880 bytes left
[367406.863618] daqd[13078]: segfault at 41 ip 0000000000000041 sp 00007fb1f0ba2cf8 error 14 in daqd[400000+7c000]
[367406.863833] daqd[13104] general protection ip:7fb2f3018a6c sp:7fb1f01c8d30 error:0
[367406.863877] daqd[13086] general protection ip:7fb2f3018a6c sp:7fb1f089ad30 error:0
[367406.877408] daqd[13080]: segfault at 41 ip 0000000000000041 sp 00007fb1f0ae0ca8 error 14 in daqd[400000+7c000]
[367406.877435]  in libc-2.10.1.so[7fb2f2fe4000+14c000]
[367406.877442] daqd[13100]: segfault at 39ba287b0 ip 00007fb2f3018a6c sp 00007fb1f034cd30 error 4 in libc-2.10.1.so[7fb2f2fe4000+14c000]
[367406.878372]  in libc-2.10.1.so[7fb2f2fe4000+14c000]
[399802.887523] daqd[18295] general protection ip:7fb056a71a6c sp:7faf96125f10 error:0 in libc-2.10.1.so[7fb056a3d000+14c000]
[410595.969327] daqd[22057]: segfault at 3a91f27b0 ip 00007f48e96eea6c sp 00007f47e6c26d50 error 4 in libc-2.10.1.so[7f48e96ba000+14c000]
[410595.988926] daqd[22068]: segfault at 3a91f2790 ip 00007f48e96eea6c sp 00007f47e681bd30 error 4 in libc-2.10.1.so[7f48e96ba000+14c000]

After locking the arms (after the MC alignment work), Manasa and I aligned IP POS, IP ANG, and both end transmission QPDs. 

We noticed that IP ANG is clipping in yaw as it comes onto the end table.  It looks to me like it's clipping on the edge of the plastic box's aperture, but I can't guarantee that it's not also clipping elsewhere. 

[Rana, Jenne]

We turned off the WFS servos, and looked at the MC REFL DC, and saw that it was still good, so we said that since the MC spots are pretty good, that we'll keep this alignment for now. 

Rana put the beam back on the center of the IOO QPDs on the PSL table.

We switched a steering mirror in the WFS path that was the wrong handed-ness to be the correct handed-ness, then put the beam on the centers of the WFS.  We turned on the WFS, and everything seems good.

While we were out on the table, we also changed the anodized aluminum dump for a razor dump, to catch the reflection from the 2inch lens that is the first thing the MC refl path sees out of vac.

There were no major drifts in the WFS error signals while we were gone for dinner, so the MC seems okay for now.

 Aligned MC to offload the WFS

1. Turned OFF the WFS feedback servo.

2. Aligned the MC suspensions by moving the pit and yaw sliders. MC trans sum brought from ~11000 counts to ~15000 counts. MC RFPD DCMON reads 0.45 counts.

3. Turned ON the WFS servo. The WFS output now reads in the order of 0 to +/-15.

4. Measured the MC spot positions. The spot positions look like they moved for the better compared to what they were yesterday.

The MC trend for the last 2 days shows that the MC suspensions were kicked again earlier today.  Looking back at the suspension channel INMONs along with the MC trans sum shows that the suspensions get kicked everytime MC locks and unlocks. (Attch:1)

So I checked the effect of WFS on the suspensions by disabling and enabling the WFS feedback servo (Attch:2).

Since the IMC is not at it best pointing, whenever the  MC autolocker runs and enables the WFS, the suspensions look like they are getting kicked.  But really, it's just the WFS doing their job. 

Edit, JCD:  What this really means is that our DC MC pointing is bad, and we need to move the MC suspensions to offload the WFS.  (All of the WFS output numbers for MC1 and 3 were around 100, which is pretty big for those numbers).  We should resurrect the WFS offloading scripts so that we can do this more regularly, and not have to do it by hand.

 NOT drift. The sudden steps are certainly the result of being kicked. The slow drift at the end of the day might be a slow strain relaxation.

It pays to be careful and not put too much weight or impulsive forces on the chambers or tables.

 I was taking pictures at the AP table at the morning and ETMX optical table after noon. There was no activity on the IOO chamber.

 Look at the last 2 hours of  Rana's trend plot. MC1 and MC2 sensor voltage started increasing.

I think it was a drift action.

 The PRCL once again doesn't want to lock on sidebands for me.  I can lock on the carrier just fine (using the IFO Config settings, along with some hand-alignment of the PRM). 

However, I can't convince it to lock on sidebands.  Using the configs that I used on Dec 18th (elog 9491), I'm not getting it.  I've done the arm ASS alignment, and I've run LSCoffsets, both of which seemed to do their things appropriately. 

I'm going to attribute this today to not being in the groove yet, and I'll look at it again in the morning.

When I got in this morning at 9-something (9:45 maybe?), Steve was taking dust photos on the AS table, of the MC Refl path.  Other than that, I don't have any information. 

Also, Tuesday is our traditional janitor day, so I'm hesitant to put our blame there.  (I think we've kept Tuesdays, even though we're on a less-often schedule....Steve will have to correct me if I'm wrong on this).

 The trend shows a big jolt to the MC1/3 pointing this morning at 8:30.

Was anyone working anywhere near there today? There is no elog.

If not, we will have to put a 'no janitor' sign on all of the 40m doors permanently to prevent mops misaligning our interferometer.

Since this configuration change, the daily backup was speeded up by factor of more than two.
It was really limited by the bandwidth of the RAID array.

/cvs/cds/rtcds/caltech/c1/scripts/backup/rsync.backup.cumlog:

...
rsync.backup start: 2013-12-20-05:00:00, end: 2013-12-20-07:04:28, errcode 0
...
rsync.backup start: 2014-01-05-05:00:00, end: 2014-01-05-05:55:04, errcode 0

(The daily backup starts from 5:00)

I'm not sure which pointing Rana wanted to fix today, but here's a measurement of the MC spots.  They actually look pretty good.  There is some room for improvement, but not a lot, so I'm leaving it alone for now, while I play with other things in the IFO.

spot positions in mm (MC1,2,3 pit MC1,2,3 yaw):
[0.63368182839757914, 1.3004245778952557, 0.33621668795755993, -1.5585578137597658, -3.1344594013487286, -1.0533063060089816]

 I went to the PSL table to re-align the input pointing to the IMC. After trying to optimize the pointing into the PMC and not succeeding I also then touched the wrong mirror and messed up our IOO QPD reference pointing.

The IMC is locking again, but I'll have to fix the pointing on Monday.

 This is a 10-minute trend of the last 60 days of the pointing of the PSL beam.

The main fluctuation seems to be at the ~30 day time scale (not 24 hour) and its all in the vertical direction; the horizontal drift is ~10x less (as long as we believe there is no calibration error).

So what's causing all of this vertical shift? And why is there not just as much horizontal??

 Events of the power supply swap:

1, Tested 24V DC ps from Todd

2, Closed V1, VM1 and all annulos valves to create safety net for the reboot. Turbo pumps left on running.

3, Turned computer off

4, Swap power supplies and turned it on

5, Turning the power on of c1vac2 created caos switching of valves. This resulted in a air vent as shown below.

6, VM1 was jammed and it was unable to close. The IOO beam shutter closed and the IFO was venting with air for a few minutes. Maglev did an emergency shut down. TP2's V4 and TP3' V5 closed. RP1 and RP3                           roughing         pumps turned on, their hose was not connected as usual. The RGA shut down to protect itself.

7, Closed annulos valves, stoped the vent at P1 27 torr as the vacuum control was  manually recovered

8, The Maglev and the annuloses were roughed out 500 mtorr . The Maglev was restarted.

9, The IFO pump down followed std procedure from 27 torr. VM1 was moving again as the pressure differential was removed from it.

 Remember: next time at atm .....rough down the cryo volume from 27 torr !

We probably need to restart the machine, but I didn't want to touch c1vac1 and c1vac2. 

 The date is correct on this monitor.

Last stored RGA scan data Dec 20, 2013

IFO pressure at CC1 5.8e-6 Torr

Valve configuration: Vacuum Normal, confirmable only by manual checking of position indicators and pressure gauge controllers  readouts

Well done Koji!  I'm very impressed with the sysadmin skillz.

IFO restart after the recovery of linux1

Machine recovery in the following order
- Start fb
- Start the following machines: mafalda, megatron, op340m
- Start c1ioo, c1lsc, c1sus, c1iscex, c1iscey

CDS recovery / burtrestore

- Confirm all of the RT systems are running in "green". If not, restart corresponding model.
- c1iscaux, ciscaux2 didn't have response (white boxes). Went to the LCS digital rack and power cycled these targets
- burtrestore: The snapshots at Dec 19 05:07 were used. For c1iscaux and c1iscaux2 the snapshots at Dec 22 05:07 were used.

fast machines
c1alsepics.snap
c1assepics.snap
c1asxepics.snap
c1calepics.snap
c1iooepics.snap
c1lscepics.snap
c1lspepics.snap
c1mcsepics.snap
c1oafepics.snap
c1pemepics.snap
c1rfmepics.snap
c1scxepics.snap
c1scyepics.snap
c1spxepics.snap
c1supepics.snap
c1susepics.snap
c1tstepics.snap

slow machines
c1auxex.snap
c1auxey.snap
c1aux.snap
c1iool0.snap
c1iscaux2.snap
c1iscaux.snap
c1psl.snap
c1susaux.snap

IFO recovery

- Reload watchdogs => restore sus damping
- MC misaligned but TEM00 was locked
- Gave a small touch on MC2 yaw => IMC almost aligned
- Autolocker wasn't running => Manually launched rather than wait for an hour for cron to launch it
- PMC was largely misaligned. => Aligned on the PSL table (PSLTRANS 0.640->0.753)
- MC WFS ON
- IFO X/Y arm locked and aligned with ASS.
- PRMI mode: manually aligned PRM. The PRMIsb momentally locked.

Dec 22 between 6AM and 7AM, physical or logical failure has occure on the 4th disk in the RAID array on linux1.
This caused the RAID disk fell into the readonly mode. All of the hosts dependent on linux1 via NFS were affected by the incident.

Today the system has been recovered. The failed filesystem was restored by copying all of the files (1.3TB total) on the RAID to a 2TB SATA disk.
The depending hosts were restarted and we recovered elog/wiki access as well as the interferometer control system.

Recovery process

o Recover the access to linux1

- Connect an LCD display on the host. The keyboard is already connected and on the machine.
- One can login to linux1 from one of the virtual consoles, which can be switched by Alt+1/2/3 ...etc
- The device file of the RAID is /dev/sda1
- The boot didn't go straightforward as mounting of the disks accoding to /dev/fstab doesn't go well.
- The 40m root password was used to login with the filesystem recovery mode.
- Use the following command to make the editing of /etc/fstab available

# mount -o rw, remount /

- In order to make the normal reboot successfull, the line for the RAID in /etc/fstab needed to be commented out.

o Connect the external disk on linux1

- Brought a spare 2TB SATA disk from rossa.
- Connect the disk via an USB-SATA enclosure (dev/sdd1)
- Mount the 2TB disk on /tmpdisk
- Run the following command for the duplication

# rsync -aHuv --progress /home/ /tmpdisk/ >/rsync_KA_20131229_0230.log

- Because of the slow SCSI I/F, the copy rate was limited to ~6MB/s. The copy started on 27th and finished 31st.

o Restart linux1

- It was found that linux1 couldn't boot if the USB drive is connected.
- The machine has two SATA ports. These two are used for another RAID array that is not actually used. (/oldhome)
- linux1 was pulled out from the shelf in order to remove the two SATA disks.
- The 2TB disk was installed on the SATA port0.
- Restart linux1 but didn't start as the new disk is recognized as the boot disk.
- The BIOS setting was changed so that the 80GB PATA disk is recognized as the boot disk.
- The boot process fell into the filesystem recovery mode again. /etc/fstab was modified as follows.

- Another reboot make the operating system launched as usual.

o What's happen to the RAID?

- Hot removal of the disk #4.
- Hot plug of the disk #4.
- Disk #4 started to get rebuilt -> ~3hours rebuilding done
- This made the system marked as "clean". Now the raid (/dev/sdb1) can be mounted as usual.

o Nodus

- Root password of nodus is not known.
- Connect an LCD monitor and a Sun keyboard on nodus.
- Type Stop-A. This leads the nodus transition to the monitor mode.
- Type sync.
- This leads the system rebooted.

The recovery- pumpdown reached valve configuration  vacuum normal at 20 hours, cc1 7.7e-6 Torr

Lesson learned: turn all pumps off, close all valves before you reboot ! like you would prepare for AC power shut down.

The previous LSC whitening filters for the DCPDs were in an unknown state (although the transfer functions were actually measured and fit a while ago)
They had no DC gain control and some of the channels had modifications.

To make the setup clear, the filter module was replaced with the spare module without any modification.

The channels are now respoding to the whitening gain switches. So far there is no screen for the new  whitening gains yet.

Also I found that POX11 DC cable has not been connected. Now it is connected.