Three good days of IOO pointing: Friday, Sat and Sun    What was changed?  May be the clamping on Friday?

IOO vertical changes recovering as tempeture. IP is clipping at plastic enclosure of ETMY

NOTE: ANTS at the PSL optical table.  I will mop with chemicals tomorrow if we see more.

 Then there must be something else slowly drifting. It was very clear that the good alignment of the IFO was every time lost after few minutes...

 IOO pointing is drifting in pitch. I'd like to use a QPD instead of the paper target to see if the Innolite output is stable. The idea is to move temporarily IOO-QPD_POS to  this location

 Its very doubtful that the MC yaw drift matters for the IFO. That's just a qualitative correlation; the numbers don't hang together.

I looked at IOO QPDs again. QPD_POS was clamped by one screw. Dog clamp was added on the unclamped side.

QPD_ANG chassis has no isolation to optical table..._POS has.  QPD_ANG  base was tightened also.

Both QPDs moved a little bit but I did not centered them.  The spot sizes are 2-3 mm  They should be smaller.

How ever, we still can not explane the pitch movement of the IOO beam

Razor beam dumps were labeled at the AP table.

The 40m roof was cleaned from leafs this morning.

[ericq, Gabriele, Manasa]

 We wanted to perform the PRC length measurement today with an AS11 signal, but such a signal didn't exist. So, we have temporarily connected the AS110 PD signal (which is some Thorlabs PD, and not a resonant one) into the REFL11 demod board. 

We then proceeded with the goal of locking the PRC with REFL165. A few parameters that were changed along the way as we aligned and locked things:

• the XARM gain was increased from 0.4 to 0.5 to help it acquire lock
• the MICH gain was decreased from -10 to -5 since there was some gain peaking in its servo output
• the REFL165 demodulation phase was changed from 155 to 122, to place a PRCL excitation entirely within I (we did this while locked on the carrier)

Sadly, in the end, we couldn't lock the PRC on a sideband in a stable manner. The alignment would drift faster than we could optimize the alignment and gains for the PRC. I.e. we would lock the PRC on the carrier, align PRM (and maybe touch ITMX) to maximize POPDC, switch to sideband locking, try to lock, and things would start looking misaligned. Switching back to carrier locking, the beam spots on REFL (for example) would have moved.

Manasa noted the MC_TRANS_Y has been substantially drifting along with small drift in MC_TRANS_P as well. So we need to fix the source of the mode cleaner beam drifting if we want to make this measurement. 

Can you please explain this? I don't understand what exactly is the issue or 'great idea'.

I think we should be OK with just a single lens in the vacuum. But what we need is the ray tracing analysis to show what the effect will be on the IPANG readout.

  IOO QPDs tested in dark, lighted and open PSL enclosure. The created temperature change 0.03 C has  effect on monitoring  in pitch.

 Atm1,  all lights off 10 min, PSL enclosure lights on  10 min, all lights off 15 min, open  door # 11 at north east corner of enclosure ( HEPA filters are running at 30V ) for 10 min, closed-dark enclosure 15 min

              dark 10, lighted 10, dark 15, open-dark 10 and closed-dark 15 minutes

Atm2, Pitch drift of 24 hours does not recover

 [Gabriele, EricQ]

Actually it is difficult to see any laser frequency line in the dark fringe signal, since the Schnupp asymmetry is small. It is much better to use a differential MICH excitation which gives a better signal at the dark port.

We repeated the simulation explained before. We can use both the AS55 or the AS11 signals, bout the first one has a limited linear range and the expected 4cm value is very close to saturation.

For the IPANG telescope design, we are in the 'beyond the Rayleigh range' regime. So using a single lens to make the beam small is not a great idea. I have put down a solution where we use a pair of lenses; one of which will be mounted in-vacuum in the ETMY chamber and the other on the endtable.
This way we will also allow have some freedom to configure the layout out-of vacuum in case the need arises. The layout will look something like in the cartoon:

I also made a choice of using longer focal length lenses (CVI 2" lenses f =1 m). Below is the beam path summary for IPANG telescope. I have used the waist diameter at the ITM for propagation. The endtable is roughly at 41.2m. The QPD will be placed in front of the waist (w0=47um).
 

 [ericq,Gabriele]

So, we want an relatively quick measurement of the PRC length error (with sign!) at the order of .5 centimeter or so. Rana suggested the "demodulation phase method," i.e. lock the simple Michelson, measure what demodulation phase brings the 1F signal entirely within the phase quadrature, then lock the PRMI and measure the demodulation phase again. This tells you something about the length of the PRC. 

Gabriele and I worked through a simulation using MIST to determine how to actually do this. We simulated the case of injecting a line at 1kHz in the laser frequency via the laser's PZT and looking at the transfer function of the 1kHz signal to the I and Q at the 1F AS demodulated signal when locked. (Michelson locked on the dark fringe, PRC locked on 11MHz sideband) With the I and Q in hand, we can measure some demodulation phase angle that would bring everything into I. 

When the PRC length is in the ideal location, the demodulation phases in the two cases are the just about the same. Sweeping the length of the PRC around the ideal length gives us a monotonic function in the difference in the demodulation phases:

So, with this simulation, we should be able to calibrate a measured difference in demod phase into the length error of the cavity! We will proceed and report...

Steve may actually be onto something with the clamps that he had made a year and a half ago.  These clamps hold the glass, and then clamp to the base of the suspension cage.  Not the table, but the base of the suspension cage. The drawings are in elog 6344.  I'm not sure that the 1/4-20 holes in the clamp things are exactly where we'll want them, but we should be able to just dog it down to the base of the suspension.  I need to check this, but it may be even easier than tieing the glass to the cage.

Also, something to think about is that the earthquake stop screws extend backwards farther than the OSEMs.  I'm not sure anymore if we have shorter 1/4-20 earthquake stops around (if we do, they should be in the cleanroom shelves), but if we can't swap those out, they'll limit how close we can get to the OSEMs. 

Here's an overhead photo from 6 Sept 2012:

I ran a simulation of a double cavity with a PRC length mismatched w.r.t. the modulation frequency. I summarized the results in the attached PDF. I think it would be important to have a cross check of the results.

In brief:

A mismatch between PRC length and modulation frequency do have an effect on error signals

Multiple zeros appear in REFL_3f/PRCL that can be removed by careful tuning of the demodulation phase (however, the shape of the signal makes difficult to understand which phase is good…)

No visible effect on REFL_1f/CARM

But a large PRCL signal appears in REFL_1f_I, which is used to control CARM. This is not good.

A mismatch of the order of 0.5 cm has a small effect.

dadq is logging the following error messages to it's log related to the fact that it can't connect to c1vac1 and c1vac2:

CAC: Unable to connect because "Connection timed out"
CA.Client.Exception...............................................
    Warning: "Virtual circuit disconnect"
    Context: "c1vac2.martian:5064"
    Source File: ../cac.cpp line 1127
    Current Time: Tue Jan 07 2014 23:50:53.355609430
..................................................................
CAC: Unable to connect because "Connection timed out"
CA.Client.Exception...............................................
    Warning: "Virtual circuit disconnect"
    Context: "c1vac1.martian:5064"
    Source File: ../cac.cpp line 1127
    Current Time: Tue Jan 07 2014 23:50:53.356568469
..................................................................

 Not sure if this is related to the full /frames issue that we've been seeing.

The wiper script is done and deleted a whole bunch of stuff to clean up some space:

controls@fb ~ 0$ /opt/rtcds/caltech/c1/target/fb/wiper.pl --delete

Tue Jan  7 23:09:21 PST 2014

Directory disk usage:
/frames/trend/minute_raw 385927520k
/frames/trend/second 125729084k
/frames/full 12552144324k
/frames/trend/minute 2311404k
Combined 13066112332k or 12759875m or 12460Gb

/frames size 13460088620k at 97.07%
/frames above keep value of 95.00%
Frame area size is 12401156668k
/frames/full size 12552144324k keep 11781098835k
/frames/trend/second size 125729084k keep 24802313k
/frames/trend/minute size 2311404k keep 620057k
Deleting some full frames to free 771045488k
- /frames/full/10685/C-R-1068567600-16.gwf
- /frames/full/10685/C-R-1068567616-16.gwf
...
controls@fb ~ 0$ df -h /frames
Filesystem            Size  Used Avail Use% Mounted on
/dev/sda1              13T   12T  826G  94% /frames
controls@fb ~ 0$

So it cleaned up 826G of space.  It looks like the fb is stabilized for the moment.  On site folks should confirm...

asdfasdfsadf sadf asdf

It seems that the most important short-term task we have right now is to figure out what our PRC length is, and what our tolerance from nominal is.  Gabriele and EricQ are going to work on that tomorrow.  If our PRC is of a length that we can't do anything useful for full IFO locking, we need to open up and fix it sooner rather than later.

While we're in there, we need to also put a baffle on the back side of the PRM cage, to protect the OSEMs from stray light.  Den and I discovered before Christmas that turning off the OSEM and OpLev damping to the PRM (while using the POP QPD for ASC) significantly reduced the power fluctuations in the PRC.  We still had arm power fluctuations, but I believe those are likely because our ALS system can't hold an arm precisely at full resonance.  So, putting a black glass baffle with ~2 inch aperture right up against the OSEMs should help a lot.  This week, I'll ask Steve to make me a quickie to-scale cardboard version of the baffles that he has had cut, so I can try securing it to the dirty suspension cage that we have out.  I will also check to make sure I have seen with my own eyes the baffles that I need, and copper wire to tie it to the cage.

Other, lower-priority things that we should do eventually:

* Steve, please find another razor beam dump for the WFS reflections - Rana and I used one of the ones that was there for reflection off the 2 inch lens in the MC refl path (replacing the aluminum dump that has been there for ages).  We also need to label all of our razor dumps with their purpose, with a label on top, so we remember not to remove dumps that are actually in use.

* At some point, we should change the one remaining steering mirror in the main PSL path that is aluminum, to a steel Polaris ("Polanski" or "Polish") mount.  For now, we should just make sure we have one handy.  Hopefully this will help reduce the PMC transmission drift that we see.

* Steve, in the morning sometime this week, can you please do a test of the drift of the IOO QPDs?  We'd like to see a trend that is maybe 30 or 60 minutes long of the QPD signals.  First 10 minutes, all lights in IFO room off.  Then, 10 minutes with the lights in the PSL on.  Then, the rest of the time the PSL lights off.  We want to see if these are hot enough to be causing a big temperature change in the PSL box, which may then be causing some optics to drift.

* QPD code in the simulink models (trans QPDs, but also OpLevs, and anywhere else we do normalization) needs to have anti-divide-by-zero protection.  I'll take care of this, it should be a quick copy of what we have elsewhere in the simulink code.

* Note to self for the future, instead of doing a dither alignment for the ASS for the arms, we can use the IP POS and IP ANG, as well as end transmission QPD signals.  However, for now, the ASS is working just fine.

* We want to go back to the idea of putting a lens into the in-vac IP ANG path, to avoid the clipping that Manasa and I were seeing tonight.  We want something of order 2inch diameter, 1meter focal length.  The material doesn't matter, but we do want it AR coated for 1064nm on both sides.  We also need to make sure that we could use a fixed 2 inch in-vac mirror mount, or something, to hold this lens.  If that won't work, we need to come up with another plan.  Manasa is working on thinking about precisely what lens we want to buy for a nice guoy phase telescope for IPANG, so we'll buy a lens after she puts her conclusions in the elog.

* An idea for the MC spots plot that Rana had was to plot the beam tilt and translation, rather than the raw spot positions on the mirrors.  The point of this would be to make it easier to see what the output beam from the MC looks like.  For MC pointing, we should also think about what our actual tolerances are.  The biggest thing is that we need to get through the Faraday without being too close to any edge, and also the REFL beam needs to come back through without clipping.  For now, we're just visually checking that the POP beam and the REFL beam both look unclipped since we don't have access to good camera views of either side of the Faraday.

It actually looks like the wiper script has been running fine.  There is a log from Tuesday morning:

controls@fb ~ 0$ cat /opt/rtcds/caltech/c1/target/fb/wiper.log

Tue Jan  7 06:00:02 PST 2014

Directory disk usage:
/frames/trend/minute_raw 385289132k
/frames/trend/second 100891124k
/frames/full 12269554048k
/frames/trend/minute 1906772k
Combined 12757641076k or 12458633m or 12166Gb

/frames size 13460088620k at 94.78%
/frames is below keep value of 95.00%
Will not delete any files
df reported usage 97.72%
controls@fb ~ 0$

So now I'm wondering if something else has been filling up the frames today.  Has anything changed today that might cause more data than usual to be written to frames?

I'm manually running the wiper script now to clear up some /frames.  Hopefully that will solve the problem temporarily.

 The last 4 hour trend for WFS error signals show some amount of drift. We should still look at the long term trend to solve the issue.

The problem is not exactly the same as what's described in 7105, but the symptoms are so similar I assumed they must have a similar source.

And sure enough, /frames is completely full:

controls@fb /opt/rtcds/caltech/c1/target/fb 0$ df -h /frames/
Filesystem            Size  Used Avail Use% Mounted on
/dev/sda1              13T   13T     0 100% /frames
controls@fb /opt/rtcds/caltech/c1/target/fb 0$

So the problem in both cases was that it couldn't write out the frames.  Unfortunately daqd is apparently too stupid to give us a reasonable error message about what's going on.

So why is /frames full?  Apparently the wiper script is either not running, or is failing to do it's job.  My guess is that this is a side effect of the linux1 raid failure we had over xmas.

The daqd process is segfaulting and restarting itself every 30 seconds or so.  It's pretty frustrating. 

Just for kicks, I tried an mxstream restart, clearing the testpoints, and restarting the daqd process, but none of things changed anything.  

Manasa found an elog from a year ago (elog 7105 and preceding), but I'm not sure that it's a similar / related problem.  Jamie, please help us!

Here is a screen dump from the "dtail":

Every 1.0s: dmesg | tail -50                                                                                                                         Tue Jan  7 22:43:23 2014

[   33.498691]  [<ffffffff8104a063>] kthread+0x7a/0x82
[   33.498695]  [<ffffffff81003654>] kernel_thread_helper+0x4/0x10
[   33.498698]  [<ffffffff81049fe9>] ? kthread+0x0/0x82
[   33.498701]  [<ffffffff81003650>] ? kernel_thread_helper+0x0/0x10
[   33.498703] ---[ end trace 6236defa99b3e091 ]---
[   33.498705] mx INFO: Board 0: allocated MSI IRQ 67
[   33.498713] mx INFO: CPU0: PAT = 0x7010600070106
[   33.498715] mx INFO: CPU0: new PAT = 0x1010600070106
[   33.498718] mx INFO: Board 0: Using PAT index 6
[   33.499101] eth0: no IPv6 routers present
[   33.531013] mx INFO: Board 0: device 8, rev 0, 1 ports and 2096896 bytes of SRAM available
[   33.531017] mx INFO: Board 0: Bridge is 10de:005d
[   33.531228] mx INFO: Board 0: MAC address = 00:60:dd:46:ea:ec
[   33.535971] mx INFO: Loaded mcp of len 235448
[   34.489244] mx INFO: Starting usermode mapper at /opt/mx/sbin/mx_start_mapper
[   39.148855] mx INFO: mx0: Link0 is UP
[   39.588511] mx INFO: myri0: Will use skbuf frags (4096 bytes, order=0)
[   39.589299] mx INFO: 1 Myrinet board found and initialized
[  287.706367] daqd used greatest stack depth: 3368 bytes left
[86605.907520] daqd[18407]: segfault at 38b08e4c0 ip 00007f11b3942a6c sp 00007f10b1917d50 error 4
[86605.907530] daqd[18424]: segfault at 38b544f90 ip 00007f11b3942a6c sp 00007f10b12c6d30 error 4 in libc-2.10.1.so[7f11b390e000+14c000] in libc-2.10.1.so[7f11b390e000+14c00
0]
[86605.907544]
[86605.919454] daqd[21319] general protection ip:7f11b3942a6c sp:7f10b1814d30 error:0
[86605.919462] daqd[18442] general protection ip:7f11b3942a6c sp:7f10b0bf4d30 error:0
[86605.919615] daqd[18443]: segfault at 38aee3db0 ip 00007f11b3942a6c sp 00007f10b0b73d50 error 4 in libc-2.10.1.so[7f11b390e000+14c000]
[86605.919694] daqd[18412]: segfault at 38aff35d0 ip 00007f11b3942a6c sp 00007f10b1752d30 error 4
[86605.919701] daqd[18417]: segfault at 38b544f70 ip 00007f11b3942a6c sp 00007f10b154dd50 error 4 in libc-2.10.1.so[7f11b390e000+14c000]
[86605.919708] daqd[18445]: segfault at 38aff35b0 ip 00007f11b3942a6c sp 00007f10b0ab1d50 error 4
[86605.919733] daqd[18429]: segfault at 38b42ae90 ip 00007f11b3942a6c sp 00007f10b10c1d50 error 4 in libc-2.10.1.so[7f11b390e000+14c000]
[86605.919741] daqd[18440]: segfault at 38b08e480 ip 00007f11b3942a6c sp 00007f10b0cb6d30 error 4 in libc-2.10.1.so[7f11b390e000+14c000]
[86605.958551]  in libc-2.10.1.so[7f11b390e000+14c000] in libc-2.10.1.so[7f11b390e000+14c000]
[86605.958557]
[86605.958577]  in libc-2.10.1.so[7f11b390e000+14c000]
[86605.958586]  in libc-2.10.1.so[7f11b390e000+14c000]
[86605.959639] daqd used greatest stack depth: 3160 bytes left
[98139.100888] show_signal_msg: 13 callbacks suppressed
[98139.100895] daqd[23753]: segfault at 39c7363b0 ip 00007f5bf253ba6c sp 00007f5b69b48d30 error 4 in libc-2.10.1.so[7f5bf2507000+14c000]
[98687.815120] daqd used greatest stack depth: 2984 bytes left
[208995.594227] daqd[10386] general protection ip:7f3b7c930a6c sp:7f3a79f09d50 error:0 in libc-2.10.1.so[7f3b7c8fc000+14c000]
[353015.067479] daqd used greatest stack depth: 2880 bytes left
[367406.863618] daqd[13078]: segfault at 41 ip 0000000000000041 sp 00007fb1f0ba2cf8 error 14 in daqd[400000+7c000]
[367406.863833] daqd[13104] general protection ip:7fb2f3018a6c sp:7fb1f01c8d30 error:0
[367406.863877] daqd[13086] general protection ip:7fb2f3018a6c sp:7fb1f089ad30 error:0
[367406.877408] daqd[13080]: segfault at 41 ip 0000000000000041 sp 00007fb1f0ae0ca8 error 14 in daqd[400000+7c000]
[367406.877435]  in libc-2.10.1.so[7fb2f2fe4000+14c000]
[367406.877442] daqd[13100]: segfault at 39ba287b0 ip 00007fb2f3018a6c sp 00007fb1f034cd30 error 4 in libc-2.10.1.so[7fb2f2fe4000+14c000]
[367406.878372]  in libc-2.10.1.so[7fb2f2fe4000+14c000]
[399802.887523] daqd[18295] general protection ip:7fb056a71a6c sp:7faf96125f10 error:0 in libc-2.10.1.so[7fb056a3d000+14c000]
[410595.969327] daqd[22057]: segfault at 3a91f27b0 ip 00007f48e96eea6c sp 00007f47e6c26d50 error 4 in libc-2.10.1.so[7f48e96ba000+14c000]
[410595.988926] daqd[22068]: segfault at 3a91f2790 ip 00007f48e96eea6c sp 00007f47e681bd30 error 4 in libc-2.10.1.so[7f48e96ba000+14c000]

After locking the arms (after the MC alignment work), Manasa and I aligned IP POS, IP ANG, and both end transmission QPDs. 

We noticed that IP ANG is clipping in yaw as it comes onto the end table.  It looks to me like it's clipping on the edge of the plastic box's aperture, but I can't guarantee that it's not also clipping elsewhere. 

[Rana, Jenne]

We turned off the WFS servos, and looked at the MC REFL DC, and saw that it was still good, so we said that since the MC spots are pretty good, that we'll keep this alignment for now. 

Rana put the beam back on the center of the IOO QPDs on the PSL table.

We switched a steering mirror in the WFS path that was the wrong handed-ness to be the correct handed-ness, then put the beam on the centers of the WFS.  We turned on the WFS, and everything seems good.

While we were out on the table, we also changed the anodized aluminum dump for a razor dump, to catch the reflection from the 2inch lens that is the first thing the MC refl path sees out of vac.

There were no major drifts in the WFS error signals while we were gone for dinner, so the MC seems okay for now.

 Aligned MC to offload the WFS

1. Turned OFF the WFS feedback servo.

2. Aligned the MC suspensions by moving the pit and yaw sliders. MC trans sum brought from ~11000 counts to ~15000 counts. MC RFPD DCMON reads 0.45 counts.

3. Turned ON the WFS servo. The WFS output now reads in the order of 0 to +/-15.

4. Measured the MC spot positions. The spot positions look like they moved for the better compared to what they were yesterday.

The MC trend for the last 2 days shows that the MC suspensions were kicked again earlier today.  Looking back at the suspension channel INMONs along with the MC trans sum shows that the suspensions get kicked everytime MC locks and unlocks. (Attch:1)

So I checked the effect of WFS on the suspensions by disabling and enabling the WFS feedback servo (Attch:2).

Since the IMC is not at it best pointing, whenever the  MC autolocker runs and enables the WFS, the suspensions look like they are getting kicked.  But really, it's just the WFS doing their job. 

Edit, JCD:  What this really means is that our DC MC pointing is bad, and we need to move the MC suspensions to offload the WFS.  (All of the WFS output numbers for MC1 and 3 were around 100, which is pretty big for those numbers).  We should resurrect the WFS offloading scripts so that we can do this more regularly, and not have to do it by hand.

 NOT drift. The sudden steps are certainly the result of being kicked. The slow drift at the end of the day might be a slow strain relaxation.

It pays to be careful and not put too much weight or impulsive forces on the chambers or tables.

 I was taking pictures at the AP table at the morning and ETMX optical table after noon. There was no activity on the IOO chamber.

 Look at the last 2 hours of  Rana's trend plot. MC1 and MC2 sensor voltage started increasing.

I think it was a drift action.

 The PRCL once again doesn't want to lock on sidebands for me.  I can lock on the carrier just fine (using the IFO Config settings, along with some hand-alignment of the PRM). 

However, I can't convince it to lock on sidebands.  Using the configs that I used on Dec 18th (elog 9491), I'm not getting it.  I've done the arm ASS alignment, and I've run LSCoffsets, both of which seemed to do their things appropriately. 

I'm going to attribute this today to not being in the groove yet, and I'll look at it again in the morning.

When I got in this morning at 9-something (9:45 maybe?), Steve was taking dust photos on the AS table, of the MC Refl path.  Other than that, I don't have any information. 

Also, Tuesday is our traditional janitor day, so I'm hesitant to put our blame there.  (I think we've kept Tuesdays, even though we're on a less-often schedule....Steve will have to correct me if I'm wrong on this).

 The trend shows a big jolt to the MC1/3 pointing this morning at 8:30.

Was anyone working anywhere near there today? There is no elog.

If not, we will have to put a 'no janitor' sign on all of the 40m doors permanently to prevent mops misaligning our interferometer.

Since this configuration change, the daily backup was speeded up by factor of more than two.
It was really limited by the bandwidth of the RAID array.

/cvs/cds/rtcds/caltech/c1/scripts/backup/rsync.backup.cumlog:

...
rsync.backup start: 2013-12-20-05:00:00, end: 2013-12-20-07:04:28, errcode 0
...
rsync.backup start: 2014-01-05-05:00:00, end: 2014-01-05-05:55:04, errcode 0

(The daily backup starts from 5:00)

I'm not sure which pointing Rana wanted to fix today, but here's a measurement of the MC spots.  They actually look pretty good.  There is some room for improvement, but not a lot, so I'm leaving it alone for now, while I play with other things in the IFO.

spot positions in mm (MC1,2,3 pit MC1,2,3 yaw):
[0.63368182839757914, 1.3004245778952557, 0.33621668795755993, -1.5585578137597658, -3.1344594013487286, -1.0533063060089816]

 I went to the PSL table to re-align the input pointing to the IMC. After trying to optimize the pointing into the PMC and not succeeding I also then touched the wrong mirror and messed up our IOO QPD reference pointing.

The IMC is locking again, but I'll have to fix the pointing on Monday.

 This is a 10-minute trend of the last 60 days of the pointing of the PSL beam.

The main fluctuation seems to be at the ~30 day time scale (not 24 hour) and its all in the vertical direction; the horizontal drift is ~10x less (as long as we believe there is no calibration error).

So what's causing all of this vertical shift? And why is there not just as much horizontal??

 Events of the power supply swap:

1, Tested 24V DC ps from Todd

2, Closed V1, VM1 and all annulos valves to create safety net for the reboot. Turbo pumps left on running.

3, Turned computer off

4, Swap power supplies and turned it on

5, Turning the power on of c1vac2 created caos switching of valves. This resulted in a air vent as shown below.

6, VM1 was jammed and it was unable to close. The IOO beam shutter closed and the IFO was venting with air for a few minutes. Maglev did an emergency shut down. TP2's V4 and TP3' V5 closed. RP1 and RP3                           roughing         pumps turned on, their hose was not connected as usual. The RGA shut down to protect itself.

7, Closed annulos valves, stoped the vent at P1 27 torr as the vacuum control was  manually recovered

8, The Maglev and the annuloses were roughed out 500 mtorr . The Maglev was restarted.

9, The IFO pump down followed std procedure from 27 torr. VM1 was moving again as the pressure differential was removed from it.

 Remember: next time at atm .....rough down the cryo volume from 27 torr !

We probably need to restart the machine, but I didn't want to touch c1vac1 and c1vac2. 

 The date is correct on this monitor.

Last stored RGA scan data Dec 20, 2013

IFO pressure at CC1 5.8e-6 Torr

Valve configuration: Vacuum Normal, confirmable only by manual checking of position indicators and pressure gauge controllers  readouts

Well done Koji!  I'm very impressed with the sysadmin skillz.

IFO restart after the recovery of linux1

Machine recovery in the following order
- Start fb
- Start the following machines: mafalda, megatron, op340m
- Start c1ioo, c1lsc, c1sus, c1iscex, c1iscey

CDS recovery / burtrestore

- Confirm all of the RT systems are running in "green". If not, restart corresponding model.
- c1iscaux, ciscaux2 didn't have response (white boxes). Went to the LCS digital rack and power cycled these targets
- burtrestore: The snapshots at Dec 19 05:07 were used. For c1iscaux and c1iscaux2 the snapshots at Dec 22 05:07 were used.

fast machines
c1alsepics.snap
c1assepics.snap
c1asxepics.snap
c1calepics.snap
c1iooepics.snap
c1lscepics.snap
c1lspepics.snap
c1mcsepics.snap
c1oafepics.snap
c1pemepics.snap
c1rfmepics.snap
c1scxepics.snap
c1scyepics.snap
c1spxepics.snap
c1supepics.snap
c1susepics.snap
c1tstepics.snap

slow machines
c1auxex.snap
c1auxey.snap
c1aux.snap
c1iool0.snap
c1iscaux2.snap
c1iscaux.snap
c1psl.snap
c1susaux.snap

IFO recovery

- Reload watchdogs => restore sus damping
- MC misaligned but TEM00 was locked
- Gave a small touch on MC2 yaw => IMC almost aligned
- Autolocker wasn't running => Manually launched rather than wait for an hour for cron to launch it
- PMC was largely misaligned. => Aligned on the PSL table (PSLTRANS 0.640->0.753)
- MC WFS ON
- IFO X/Y arm locked and aligned with ASS.
- PRMI mode: manually aligned PRM. The PRMIsb momentally locked.

Dec 22 between 6AM and 7AM, physical or logical failure has occure on the 4th disk in the RAID array on linux1.
This caused the RAID disk fell into the readonly mode. All of the hosts dependent on linux1 via NFS were affected by the incident.

Today the system has been recovered. The failed filesystem was restored by copying all of the files (1.3TB total) on the RAID to a 2TB SATA disk.
The depending hosts were restarted and we recovered elog/wiki access as well as the interferometer control system.

Recovery process

o Recover the access to linux1

- Connect an LCD display on the host. The keyboard is already connected and on the machine.
- One can login to linux1 from one of the virtual consoles, which can be switched by Alt+1/2/3 ...etc
- The device file of the RAID is /dev/sda1
- The boot didn't go straightforward as mounting of the disks accoding to /dev/fstab doesn't go well.
- The 40m root password was used to login with the filesystem recovery mode.
- Use the following command to make the editing of /etc/fstab available

# mount -o rw, remount /

- In order to make the normal reboot successfull, the line for the RAID in /etc/fstab needed to be commented out.

o Connect the external disk on linux1

- Brought a spare 2TB SATA disk from rossa.
- Connect the disk via an USB-SATA enclosure (dev/sdd1)
- Mount the 2TB disk on /tmpdisk
- Run the following command for the duplication

# rsync -aHuv --progress /home/ /tmpdisk/ >/rsync_KA_20131229_0230.log

- Because of the slow SCSI I/F, the copy rate was limited to ~6MB/s. The copy started on 27th and finished 31st.

o Restart linux1

- It was found that linux1 couldn't boot if the USB drive is connected.
- The machine has two SATA ports. These two are used for another RAID array that is not actually used. (/oldhome)
- linux1 was pulled out from the shelf in order to remove the two SATA disks.
- The 2TB disk was installed on the SATA port0.
- Restart linux1 but didn't start as the new disk is recognized as the boot disk.
- The BIOS setting was changed so that the 80GB PATA disk is recognized as the boot disk.
- The boot process fell into the filesystem recovery mode again. /etc/fstab was modified as follows.

- Another reboot make the operating system launched as usual.

o What's happen to the RAID?

- Hot removal of the disk #4.
- Hot plug of the disk #4.
- Disk #4 started to get rebuilt -> ~3hours rebuilding done
- This made the system marked as "clean". Now the raid (/dev/sdb1) can be mounted as usual.

o Nodus

- Root password of nodus is not known.
- Connect an LCD monitor and a Sun keyboard on nodus.
- Type Stop-A. This leads the nodus transition to the monitor mode.
- Type sync.
- This leads the system rebooted.

The recovery- pumpdown reached valve configuration  vacuum normal at 20 hours, cc1 7.7e-6 Torr

Lesson learned: turn all pumps off, close all valves before you reboot ! like you would prepare for AC power shut down.

The previous LSC whitening filters for the DCPDs were in an unknown state (although the transfer functions were actually measured and fit a while ago)
They had no DC gain control and some of the channels had modifications.

To make the setup clear, the filter module was replaced with the spare module without any modification.

The channels are now respoding to the whitening gain switches. So far there is no screen for the new  whitening gains yet.

Also I found that POX11 DC cable has not been connected. Now it is connected.

I'm leaving the 40m now. IFO is aligned. Everything look good.

- The main volume P1=5e-4, CC1=1.4e-5 is still pumped by TP1 and TP2

- RGA P4<0e-4, CC4 2.1e-7, is pumped by TP3

- The annuluses are isolated.

- RP1/2/3 are off.

I checked the offset situation in the CM servo boost circuit. 

- The offset voltage on the CM servo screen is a raw DAC output. This number is diluted by the voltage divider before the amplifier.
  So, the actual offset of the boost circuit was much smaller. (~20mV)

- There is a offset trimmer on the board. This was adjusted so that the boost does not generate an output offset.

- So the default offset is 0V.

- When the arm was locked with (digital) POY11, the CM servo offset is necessary to be -2.7 (now).
  This means that analog POY11Q and digital POY11 has different offset for the best resonance transmission.
  That is believable if POY11I is contributing to the digital POY11 signal.

Koji, Steve

 It was a bad experience again with our vacuum system.  The valves went crazy as we rebooted the computer. This was required for the swap in of a good 24V power supply.

The IFO was vented to 27 Torr through the annuloses, VA6, V7, Maglev,VM2 and VM1 (VC2 was open too)

I just opened the PSL shutter after a 4 hours pumpdown.

Condition: annuloses are not pumped, the IFO and the RGA are pumped as Atm2 shows

I will be here tomorrow morning to switch over to vacuum normal. 

More details later

The bug is still there but the incorrect bits are now overridden.

 The lamp lasted for 4,622 hours.

This time I purchased just the bare lamp itself . The housing doubles the price. The disadvantage of this technic that the lamp housing window can not be cleaned  perfectly. Atm2 picture is exaggerating this spot.

However, It does not degrade the image quality.

I submitted a bug report for this:

https://bugzilla.ligo-wa.caltech.edu/bugzilla3/show_bug.cgi?id=553

However, given how old our RCG version is (2.5 vs. 2.8 current deployed at the sites) I don't think we're going to see any traction on this.  Even if this is still a bug in 2.8, they'll only fix it in 2.8.  There's no way they're going to make a bug fix release for 2.5.

We need to upgrade.

Just wanted to point out that the correct "modern" path to this stuff is:

/opt/rtcds/caltech/c1/scripts/general/netgpibdata

This is, of course, the same directory, but under the correct "/opt/rtcds", instead of the old, incorrect "/cvs/cds".

This too huge offset difference with/without "BOOST" switch should be checked.

up/down scripts are to be made

(Offset Edit on Dec 20 10:38PM)

Configuration:
POY11QMon -> CM Servo In1 -> CM Servo -->Out1 -> ADC -> CM Slow FM -> LSC MC Servo FM -> ETMY(x1.0) -> DAC -> ETMY
                                       |
                                       -->Servo Out -> SR560 (DC, 1st order 30kHz LPF) -> MC In2

Lock acquisition path 1


Initial condition:

CM Slow FM:

• Gain 2.6

CM Servo setting:

• In1 Gain 31dB, SW:OFF, Offset -1.880, Boost Off, Super Boost Off, AO +8dB

MC Servo setting:

• In2 10dB, SW:OFF

Acquisition:

• Engage LSC
• LSC MC servo gain +0.1, FM7/FM10 (Trigger FM2 with 3sec delay)
• Turn on MC

Transition:

• Enable AO path (CM servo In1 SW:ON, MC servo In2 SW:ON)
• LSC MC gain +0.1 -> +0.2
• AO path gain 8dB->14dB
• LSC MC gain +0.2 -> +0.35
• AO path gain 14dB->18dB
• CM servo offset -1.88 -2.7 -> 0.8 0.0 (gradually)
• Enable CM servo Boost

Lock acquisition path 2

Initial condition:

CM Slow FM:

• Gain 2.6

CM Servo setting:

• In1 Gain 31dB, SW:OFF, Offset -1.880, Boost Off, Super Boost Off, AO +20dB

MC Servo setting:

• In2 10dB, SW:OFF

Acquisition:

• Engage LSC
• LSC Yarm G=+0.25 FM4/5 (Trigger FM3/6/7/8)

Transition:

• Enable MC servo In2 (SW:ON)
• Set LSC MC gain +0.2 FM7/10
• Enable LSC MC (On)
• Enable CM servo In1 (SW:ON)
• Disable LSC Yarm (OFF)
• Change CM servo offset -1.88 -> +0.700 -2.70 -> 0.0
• Enable CM servo Boost
• Turn on LSC CM FM2 (optional)

Transition to ETMY LSC to MCL

• After all of the transition: LSC output matrix ETMY (+1.00)
• LSC output matrix MC2 (-1.00)
• LSC output matrix ETMY (0.00)