The arm's CCD cameras were reset as picture shows last week.

The height adjustment only works at ITMX. Short post holders are ordered to make this feature  available on the rest.

The 75 ohms video and power supply cables are stress relieved. Solid cover can be attached now without  miss aligning cameras.

This is the two arms locked, for an hour.  No integrator in either loop, but from this it looks like ETMY may have a bigger length2angle problem than ETMX.  I'll put some true integrators in the loops and do this again.

 There appear to be at least two independent problems: the coil balancing for ETMY is bad, and something about ITMX is broken (maybe a coil driver). 

The Y-arm becomes significantly misaligned during long locks, causing the arm power to drop.  This misalignment tracks directly with the DC drive on ETMY.  Power returns to the maximum after breaking and re-establishing lock.

ITMX alignment wanders around sporadically, as indicated by the oplevs and the X-arm transmitted power.  Power returns to previous value (not max) after breaking and re-establishing lock.

Both loops have integrators.

I tried to get some minute trend data today, but was unable to get it from inside or outside the control room using our matlab or python tools.

It seems the NDS2 interface will not work anywhere since it needs our minute trends to be written as frames; in the last version that Jamie left us, our minute trend frame files are not being written since they lead to periodic daqd crashes.

From inside the control room, we can get the minute trend (only with DataViewer). I've attached 30 days of BS_X just to show its real.

We can get the numerical data from the Grace plot window using the menu option Data->Export->ASCII.

You must select all of the 'Write Sets' to get all of the traces in the plot window. The resulting ascii file is not in a great format, but its not terrible.

X,Y arms were stabilized using ALS and moved 5 nm from the resonance, PRMI was locked on sideband using REFL165 I&Q. POP angular servo was running; PRMI RIN was good (~2-3%)

During slow offset reduction I was sweeping MICH, PRCL and POP servos for instabilities due to possible optical gain variations, loops were fine.

I could reduce offset down to ~200 pm and then lost lock due to 60 Hz oscillations as shown on the attached plot "arm_offset"

Arms were stabilized with RMS comparable to the offset and power in arms was fluctuating from 3 to 45.

60 Hz line most probably comes from MICH. RMS is dominated by the power lines and is ~ 1 nm as seen on the plot "PRMI_CAL". I think this is too much but we need to do simulations.

I removed the video monitoring can and replaced it with Olympus SP-570UZ camera. It has no IR blocker. The OSEM light are dominant because I can not zoom in more.

I left the camera in place so you can try it. Leave the LEXAN plate on the glass window so no accident can happen. The illuminator is on and you can turn it off-on with the manual switch, close to the camera. Camera manual is on my desk.

The attached is the ETMY image with the single arm locked. This was the best I could do. Here is the recipe

• Turn on SP570UZ
• Switch to "M" mode (Manual aperture and exposure)
• Set the aperture to be the widest (smallest F number) and the exposure to be maximum (15 second).
• Switch to AF mode by the lens side switch
• Use the lens dial to adjust the zoom until the OSEMs fill the central 1/3 box (i.e. 1/9 area of the field of view). If you zoom more, you can't focus the spot later.
• Use menu button to switch to ISO1600 (You are now capable to see the beam spot)
• Switch to MF mode by the lens side switch
• Use the lens dial to adjust the focus to have the sharpest image of the spot. This can be achieved at the focal distance of ~1m
• Use menu button to switch back to ISO64
• Push the shutter (I didn't use it, but you should be able to use 2sec timer)

Your technique works Koji

Today I've been trying to get the new frame builder, tentatively 'fb1', to work.  It's not fully working yet, so I'm about to revert the system back to using 'fb'.  The switch-over process is annoying, since our one myrinet card has to be moved between the hosts.

A brief update on the process so far:

I'm being a little bold with this system by trying to build daqd against more system libraries, instead of the manually installed stuff usually nominally required.  Here's some of the relevant info about th fb1 system:

• Debian 7 (wheezy)
• lscsoft ldas-tools-framecpp-dev 2.4.1-1+deb7u0
• lscsoft gds-dev 2.17.2-2+deb7u0
• lscsoft libmetaio-dev 8.4.0-1+deb7u0
• lscsoft libframe-dev 8.20-1+deb7u0
• /opt/rtapps/epics-1.4.12.2_long
• /opt/mx-1.2.16
• advLigoRTS trunk

I finally managed to get daqd to build against the advLigoRTS trunk (post 2.9 branch).  I'll post detailed build log once I work out all the kinks.  It runs ok, including writing out full frames, as well as second and minute trends and raw minute trends, but there are a couple of show-stopper problems:

• daqd segfaults if the C1EDCU.ini is specified.  If I comment out that one file from the 'master' channel ini file list then it runs without segfaulting.
• Something is going on with the mx_streams from the front ends:
  • They appear to look ok from the daqd side, but the FEC-<ID>_FB_NET_STATUS indicators remain red.  The "DAQ" bit in the STATE_WORD is also red.  Again, this is even though data seems to be flowing.
  • The mx_stream processes on the front ends are dying (and restarting via monit) about every 2 minutes.  It's unclear what exactly is happening, but they all dia around the same time, so it possibly initiated from a daqd problem.  Around the time of the mx_stream failures, we see this in the daqd log:

[Tue Sep 22 17:24:07 2015] GPS MISS dcu 91 (TST); dcu_gps=1127003062 gps=1127003063

Aborted 1 send requests due to remote peer Aborted 1 send requests due to remote peer 00:25:90:0d:75:bb (c1sus:0) disconnected
mx_wait failed in rcvr eid=004, reqn=11; wait did not complete; status code is Remote endpoint is closed
00:30:48:d6:11:17 (c1iscey:0) disconnected
mx_wait failed in rcvr eid=002, reqn=235; wait did not complete; status code is Remote endpoint is closed
disconnected from the sender on endpoint 002
mx_wait failed in rcvr eid=005, reqn=253; wait did not complete; status code is Bad session (missing mx_connect?)
disconnected from the sender on endpoint 005
disconnected from the sender on endpoint 004
[Tue Sep 22 17:24:13 2015] GPS MISS dcu 39 (PEM); dcu_gps=1127003062 gps=1127003069

• Occaissionally the daqd process dies when the front end mx_streams processes die.

I'll keep investigating, hopefully with some feedback from Keith and Rolf tomorrow.

I got Steve to get us a new Myrinet fiber network adapter card for fb1:

• Myrinet 10G-PCIE-8B-S

I just finished installing the card in fb1, and it came up fine.  We happened to have a spare fiber, and a spare fiber jack in the DAQ switch, so I went ahead and plugged it in in parallel to the old fb:

controls@fb1:~/rtbuild/trunk 130$ /opt/mx/bin/mx_info
MX Version: 1.2.16
MX Build: controls@fb1:/opt/src/mx-1.2.16 Fri Sep 18 18:32:59 PDT 2015
1 Myrinet board installed.
The MX driver is configured to support a maximum of:
    8 endpoints per NIC, 1024 NICs on the network, 32 NICs per host
===================================================================
Instance #0:  364.4 MHz LANai, PCI-E x8, 2 MB SRAM, on NUMA node 0
    Status:         Running, P0: Link Up
    Network:        Ethernet 10G

    MAC Address:    00:60:dd:43:74:62
    Product code:   10G-PCIE-8B-S
    Part number:    09-04228
    Serial number:  485052
    Mapper:         00:60:dd:46:ea:ec, version = 0x00000000, configured
    Mapped hosts:   7

                                                        ROUTE COUNT
INDEX    MAC ADDRESS     HOST NAME                        P0
-----    -----------     ---------                        ---
   0) 00:60:dd:43:74:62 fb1:0                             1,0
   1) 00:25:90:0d:75:bb c1sus:0                           1,0
   2) 00:30:48:be:11:5d c1iscex:0                         1,0
   3) 00:30:48:d6:11:17 c1iscey:0                         1,0
   4) 00:30:48:bf:69:4f c1lsc:0                           1,0
   5) 00:14:4f:40:64:25 c1ioo:0                           1,0
   6) 00:60:dd:46:ea:ec fb:0                              1,0

We can now work on fb1 while fb continues to run and collect data from the front ends.

I'm still not getting the mx_stream connections to the new fb1 daq to work.  I'm leaving everything running as is on fb for the moment.

It worked. The first backup of the year is now from Wednesday, 01/10/18 at 15:19. Ten days of automatic backups are missing. Up until 2204 the year folders had been pre-emptively created so why was 2018 missing?

gautam: this is a bit suspect still - the snapshot file for c1auxex at least seemed to be too light on channels recorded. this was before any c1auxex switching. to be investigated.

I was looking into setting up autoburt for the new c1auxex2 and found that it stopped making automatic backups for all machines after the beginning of the new year. There is no 2018 folder (it was the only one missing) in /opt/rtcds/caltech/c1/burt/autoburt/snapshots and the /latest/ link in /opt/rtcds/caltech/c1/burt/autoburt/ leads to the last backup of 2017 on 12/31/17 at 23:19.

The autoburt log file shows that the back script last ran today 01/10/18 at 14:19, as it should have, but doesn't show any errors and ends with "You are at the 40m".

I'm not familiar with the autoburt scheduling using cronjobs. If I'm not mistaken the relevant cronjob file is /cvs/cds/rtcds/caltech/c1/scripts/autoburt/autoburt.cron which executes /cvs/cds/rtcds/caltech/c1/scripts/autoburt/autoburt.pl

I've never used perl but there's the following statement when establishing the directory for the new backup:

I manually created the /opt/rtcds/caltech/c1/burt/autoburt/snapshots/2018/ directory. Maybe this fixes the hickup? Gotta wait about 30 minutes.

Tobin & Keith pointed out in the LLO ilog that there was a code bug in the autoburt.pl script for autoburts.

I edited the autoburt.pl script so that it will work from now until 2099 (by which time we may no longer be using this version of perl):

nodus:autoburt>diff autoburt.pl~ autoburt.pl
234c234
<     $thisyear = "200".$timestamp[5];
---
>     $thisyear = "20".$timestamp[5];

The autoburt has not been working ever since 11PM on New Year's eve.

I ran it by hand and it seems to run fine. I noticed along the way that it was running on op340m (our old Sun Blade 150 machine). The autoburt.pl was pointing at /cvs/cds/bin/perl

which is Perl v5.0. I changed it to use '/usr/bin/env' and now points at '/usr/bin/perl' which is perl 5.8. It runs fine with the new perl:

op340m:scripts>time perl /cvs/cds/scripts/autoburt.pl >> /cvs/cds/caltech/logs/autoburtlog.log
5.37u 6.29s 2:13.41 8.7%

Also ran correctly, via cron, at 9AM.

I'm kind of confused by the current auto locker situation. Somebody renamed the script from autolockMCmain40m to AutoLockMC.csh and did not update the crontab with the new filename. 

Furthermore it doesn't seem like  scripto_cron,a script which keeps the auto locker alive, runs ok on ottavia. When I run this on the command line, it claims that the process is already running, and returns some bunk PID that doesn't correspond to any running process. The script has a line stating setenv OSTYPE solaris , so maybe there's something funky going on with it's pgrep-ing or other command parsing.   

Lastly, if I try running AutoLockMC.csh directly on ottavia, I get a bunch of complaints about pmath and pezcabit not being found. 

I renamed the Autolocker and described it in the elog from this weekend. To run it, you have to run it from the scripts/MC/ directory (as always). I restarted the autolocker on Ottavia with nohup.

> nohup ./AutoLockMC.csh &

To figure out how to get cron to run it, we will have to debug the difference between linux and solaris pgrep, so that's in progress.

I am NOT convinced that the autolocker script is running or doing anything. The MC was unlocked as of this morning and the autolocker wasn't doing anything to any of the MC servo buttons and sliders. The autolocker control shows that it is 'alive' but it still blinks 'MC down' even after I locked the MC manually. I am suspecting that this might have to do something with the caget and caput in autolock script itself rather than the CRON. I will look into this problem later today.

Moral of the story: The autolocker is not doing its job.

 Moral is wrong.

AutoLocker was working fine last night and we observed it to run and do the appropriate mcdown and mcup stuff. Probably something is breaking with it after several hours.

If you have suspicions about the script, best to look through the logs during the first few hours when we had it running yesterday.

The MC autolocker is once again running on Ottavia, with the nohup command.  

It was hanging for a long time (at least minutes) on the cavput and the caputs in the MC2 tickle on and off scripts.  I claim that there isn't a good reason to not just use ezcawrite, or whatever the latest and greatest fully functioning function is, so I've changed the cavput to a series of ezcawrites, and all of the caputs are also ezcawrites.  Now I don't see any hanging, and the MC locks itself.

This does not solve the scripto_cron issue, so if Ottavia is rebooted, or the autolocker is otherwise killed, it will not start itself up. 

  We've noticed that the caget/caput and cdsutils take a long time to return the command prompt. The following two ping commands indicate that it may be related to routing or our new BIND9 DNS setup on chiara.

controls@ottavia|~ > ping -c 5 -D c1sus

PING c1sus.martian (192.168.113.85) 56(84) bytes of data.

[1403221338.383403] 64 bytes from 192.168.113.85: icmp_req=1 ttl=64 time=0.114 ms

[1403221343.386211] 64 bytes from 192.168.113.85: icmp_req=2 ttl=64 time=0.059 ms

[1403221348.387666] 64 bytes from 192.168.113.85: icmp_req=3 ttl=64 time=0.060 ms

[1403221353.389736] 64 bytes from 192.168.113.85: icmp_req=4 ttl=64 time=0.059 ms

[1403221354.390713] 64 bytes from 192.168.113.85: icmp_req=5 ttl=64 time=0.060 ms

--- c1sus.martian ping statistics ---

5 packets transmitted, 5 received, 0% packet loss, time 20007ms

rtt min/avg/max/mdev = 0.059/0.070/0.114/0.023 ms

controls@ottavia|~ > ping -c 5 -D 192.168.113.85

PING 192.168.113.85 (192.168.113.85) 56(84) bytes of data.

[1403221463.737857] 64 bytes from 192.168.113.85: icmp_req=1 ttl=64 time=0.078 ms

[1403221464.737353] 64 bytes from 192.168.113.85: icmp_req=2 ttl=64 time=0.059 ms

[1403221465.737318] 64 bytes from 192.168.113.85: icmp_req=3 ttl=64 time=0.050 ms

[1403221466.737316] 64 bytes from 192.168.113.85: icmp_req=4 ttl=64 time=0.052 ms

[1403221467.737321] 64 bytes from 192.168.113.85: icmp_req=5 ttl=64 time=0.050 ms

--- 192.168.113.85 ping statistics ---

5 packets transmitted, 5 received, 0% packet loss, time 3999ms

rtt min/avg/max/mdev = 0.050/0.057/0.078/0.014 ms

if (`pgrep lockMZ | wc -l` > 1) then
   echo lockMZ is already running! 
   exit
endif

if                                                                                                                       
  pgrep `basename $0` | grep -v $$ > /dev/null                                                                           
then                                                                                                                     
  echo Another copy of this program is already running.  Exiting!                                                        
  exit 1                                                                                                                 
fi

Background:
 Now we can use pyNDS(see elog #3722), so I wrote a python script that adjusts Q-value automatically.

What I did:
1. Wrote a python script. /cvs/cds/caltech/users/yuta/scripts/QAdjuster.py

2. Ran this script for every DOF, every MC suspension.

Result:
 Following lines and attached are example output when I ran QAdjuster.py for MC3 POS.

Currently, C1:SUS-MC3_SUSPOS_GAIN is 5

Connecting.... authenticate failed: Unspecified error
Connecting.... done
Current GPS time is 971149410

I kicked C1:SUS-MC3_SUSPOS_OFFSET

Waiting for the ringdown...... 0\

Current GPS time is 971149431

omega_0= 6.326764,
Q= 11.254252,
A= -53.491850,
delta= 5.007821,
ofset= 4349.808434

Q-value was 11.3
Set C1:SUS-MC3_SUSPOS_GAIN = 10.2311380794

 Current Q-values automatically set are as follows.

Next work:
 - Modify the script so that it adjusts all the channels automatically. Now, it is one by one, trial by trial.
 - Modify the script so that it automatically turns the offset switch on. Now, it must be turned on beforehand.
 - Write a how-to

We were too much annoyed by frequent stall of mxstream. We'll update the RCG when time comes (not too much future).

But for now, we need an automatic mxstream resetting.

I found there is such a script already.

/opt/rtcds/caltech/c1/scripts/cds/autoMX

So this script was registered to crontab on megatron.
It is invoked every 5 minutes.

# Auto MXstream reset when it fails
0,5,10,15,20,25,30,35,40,45,50,55 * * * * /opt/rtcds/caltech/c1/scripts/cds/autoMX >> /opt/rtcds/caltech/c1/scripts/cds/autoMX.log

Made a front and back panel and slot panels for DSub and IDC breakouts. I want to send this out soon, are there any comments? Preferences for color schemes?

[johannes, gautam]

1. Johannes aligned the single bounce off the ITM into the AUX fiber on the AS table, and also the AUX beam into the fiber on the PSL table.
   • Mode matching isn't spectaular anywhere in this chain.
   • But we have 2.6mW of light going into the SRM with the AOM deflection into the 1st order beam (which is what we send into the IFO) maximum.
2. We set up some remote capabilities for the PLL and Marconi frequency (=PLL setpoint) control.
3. Motivation was to try and lock DRMI, and look for some resonance of the AUX beam in the SRC.
   • We soon realized this was a way too lofty goal.
   • So we decided to try the simpler system of PRMI locked on carrier.
   • We were successfully able to sweep the Marconi setpoint in up to 20kHz steps (although we can only move the setpoint in one direction, not sure I know why now).
   • Then we decided to look for resonances of the AUX beam in the arm cavity.
   • Still no cigar
     
4. Plus points:
   • PLL can be reliably locked remotely.
   • Marconi freq. can be swept deterministically remotely.
5. Tomorrow:
   • Fix polarization issues. There is some low freq drift (~5min period) of the power incident on the fiber on the PSL table which we don't understand.
   • Verify MM into IFO and also into fiber at PSL table.
   • Do mode spectroscopy.

I was wondering why the PMC modulation sidebands are showing up on the control room analyzer with ~6dB difference in amplitude. Then I realized that it is reasonable for the cabling to have 6dB higher loss at 80 MHz compared to 20 MHz.

I brought the NPRO from the Crackle experiment over to the 40m Lab and set it up on the PSL table to replace the slowly dying AUX laser. I also brought along a Faraday isolator, broadband EOM, and an ISOMET AOM with driver electronics from the optics storage in the Crackle Lab.

This laser is a much newer model, made in 2008, and still has all its mojo, but we should probably keep up the practice of turning it off when it's not going to be used for a while. I measured 320 mW leaving the laser, and 299mW of that going through the Faraday isolator, whose Brewster-angle polarizer I had to clean because they were a little dusty. While the laser output is going strong, the controller displays a power output of only 10 mW, which makes me think that the power monitoring PD is busted. This is a completely different failure mode from what we've seen with the other NPROs that we can hopefully get repaired at some point, particularly because the laser is newer, but for now it's installed on the PSL table. This likely means that the noise eater isn't working on this unit either, for different reasons, but at least we have plenty of optical power.

The setup is very similar to before, with the addition of a Faraday isolator and a broadband EOM, in case we decide to get more bandwidth in the PLL. I changed the Crystal Technologies 3200-113 200 MHz AOM for an ISOMET 80 MHz AOM with RF driver from the Crackle lab's optics storage and sized the AUX beam to a diameter of 200 micron. I couldn't locate an appropriate heat sink for the driver, which is still in factory condiction, but since the PSL AOM also runs on 80MHz I used that one instead. The two AOMs saturate at different RF powers, so care must be taken to not drive the AUX AOM too high. At 600 mV input to the driver the deflection into the first order was maximal at 73 % of the input power, with the second order beam and the first order on the other side cleary visible.

In order to speed things up I didn't spend too much time on mode-matching, but the advantage of the fiber setup is that we can always improve later if need be without affecting things downstream. I coupled the first order beam into the fiber to the AS table with 58% efficiency, and restored the beat with the PSL laser on the NewFocus 1611. The contrast there is only about 20%, netting a -20 dBm beat note. This is only a marginal improvement from before, so the PLL will work as usual, but if we get the visibility up a little in the future we won't need to amplify the PD signal for the PLL anymore.

Some more things I wanted to do but didn't get to today are

• Measure intensity noise of aux laser
• Measure rise and fall times of new AOM
• Get PLL back up and running
• Place 90/10 beamsplitter in AS path and couple IFO output into fiber (= couple fiber output into IFO)

I'll resume this work tomorrow. I turned the aux laser and the AOM driver input off. For the PSL beat the AOM drive is not needed, and the power in the optical fiber should not exceed 100 mW, so the offset voltage to the AOM RF driver has to remain below 300 mV.

> While the laser output is going strong, the controller displays a power output of only 10 mW, which makes me think that the power monitoring PD is busted.

NPRO internal power monitor often shows smaller value than the actual due to a broken PD or misalignment. I don't think we need to fix it.

STEVE: Aux Lightwave M126-1064-200, sn259 [July 2009] 1.76A, ADJ 9,  9mW on it's display should not mislead you. It's output  320mW

We have the appropriate heatsink - I'd like to minimize interference with the main beam wherever possible.

If damage to the fiber is a concern, I think it's better to use a PBS + waveplate to attenuate the power going into the fiber. When the AOM switching is hooked up to CDS, it's easy to imagine a wrong button being pressed or a wrong value being typed in.

It would probably also be good to have a pickoff monitor for the NPRO DC power so that we can confirm its health (in the short run, we can hijack a PSL Acromag channel for this purpose, as we now do for FSS_RMTEMP). I don't know that we need an EOM for the PLL, as in order to get that going, we probably need some fast electronics for the EOM path, like an FSS box. 

STEVE: I ordered the right heatsink for the acousto after Koji pointed out that the vertical fins are 20% more efficient. Why? Because hot air rises. It will be here in 3-4 days.

Today we had some trouble launching an excitation on C1:IOO-MC_LSC_EXC from awggui. The error read:

awgSetChannel: failed getIndexAWG C1:SUS-MC2_LSC_EXC ret=-3

What solved this was the following :

1. launch the dtt command line interface
2. Anchal remembers a slot number 37008
3. We issue >> awg free 37008
4. Slot freed, launch a new instance of awggui

AWG is not working. This needs to be fixed.

I could set the channel and the parameters in the AWGGUI screen, but it never inject signals to the realtime system.

Actually awg works fine without any problems when the excitation channels belong to the c1lsc machine.

It seems that the awg doesn't inject signals on the channels of the c1sus machine, for example C1:SUS-BS_LSC_EXC and so on.

[root@c1omc controls]# /opt/gds/awgtpman -2 &
[1] 16618
[root@c1omc controls]# mmapped address is 0x55577000
32 kHz system
Spawn testpoint manager
no test point service registered
Test point manager startup failed; -1

[1]+  Exit 1                  /opt/gds/awgtpman -2

This turned out to be fallout from the /cvs/cds transition.  Remounting and restarting fixed it.

LockAcq is back on track, with the full script working well.  Measurements in progress.

The RGA region is beeing monitored during the vent. This will tell us how clean the RGA itself is.

The RGA back ground at day 29 of this vent.

this afternoon I did some swapping around of linear and RL for IMC WFS.

In the end, I left in in the 'both' state:

• The WFS1,2,MC_TRANS PIT loops are all on but with -40, -40, -20 dB of nominal gain
• the RL is on for PIT

this is so that we have good DC control with integrators and good HF performance too.

I want to try to do the measurement with the network analyzer used as local oscillator, instead of the Marconis that I'm using now. Tha could give me better noise rejection. It would also give me information about the phase.

Also I wouldn't dislike abandoning the GPIB interfaces to acquire data.

I spent this afternoon trying to debug fb1, with very little to show for it.  We're back to running from fb.

The first thing I did was to recompile EPICS from source, so that all the libraries needed by daqd were compiled for the system at hand.  I compiled epics-3.14-12-2_long from source, and installed it at /opt/rtapps/epics on local disk, not on the /opt/rtapps network mount.  I then recompiled daqd against that, and the framecpp, gds, etc from the LSCSoft packages.  So everything has been compiled for this version of the OS.  The compilation goes smoothly.

There are two things that I see while running this new daqd on fb1:

instability with mx_streams

The mx stream connection between the front ends and the daqd is flaky.  Everything will run fine for a while, the spontaneously one or all of the mx_stream processes on the front ends will die.  It appears more likely that all mx_stream processes will die at the same time.  It's unclear if this is some sort of chain reaction thing, or if something in daqd or in the network itself is causing them all to die at the same time.  It is independent of whether or not we're using multiple mx "end points" (i.e. a different one for each front end and separate receiver threads in the daqd) or just a single one (all front ends connecting to a single mx receiver thread in daqd).

Frequently daqd will recover from this.  The monit processes on the front ends restart the mx_stream processes and all will be recovered.  However occaissionally, possibly if the mx_streams do not recover fast enough (which seems to be related to how frequently the receiver threads in daqd can clear themselves), daqd will start to choke and will start spitting out the "empty blocks" messages that are harbirnger of doom:

Aborted 2 send requests due to remote peer 00:30:48:be:11:5d (c1iscex:0) disconnected
00:30:48:d6:11:17 (c1iscey:0) disconnected
mx_wait failed in rcvr eid=005, reqn=182; wait did not complete; status code is Remote endpoint is closed
disconnected from the sender on endpoint 005
mx_wait failed in rcvr eid=001, reqn=24; wait did not complete; status code is Remote endpoint is closed
disconnected from the sender on endpoint 001
[Wed Dec  9 18:40:14 2015] main profiler warning: 1 empty blocks in the buffer
[Wed Dec  9 18:40:15 2015] main profiler warning: 0 empty blocks in the buffer
[Wed Dec  9 18:40:16 2015] main profiler warning: 0 empty blocks in the buffer

My suspicion is that this time of failure is tied to the mx stream failures, so we should be looking at the mx connections and network to solve this problem.

frame writing troubles

There's possibly a separate issue associated with writing the second or minute trend files to disk.  With fair regularity daqd will die soon after it starts to write out the trend frames, producing the similar "empty blocks" messages.

I made a simple script to backup crontab (/opt/rtcds/caltech/c1/scripts/crontab/backupCrontab).

#!/bin/bash

crontab -l > /opt/rtcds/caltech/c1/scripts/crontab/crontab_$(hostname).$(date '+%Y%m%d%H%M%S')

I put this script into op340m crontab.

00 8 * * * /opt/rtcds/caltech/c1/scripts/crontab/backupCrontab

It took me 30 minutes to write and check this one line script. I hate shell scripts.

The backup of /cvs/cds (which runs as a cron job on fb40m; see /cvs/cds/caltech/scripts/backup/000README.txt) 
has been down since fb40m was rebooted on March 3.
I was unable to start it because of conflicting ssh keys in /home/controls/.ssh .
With help from Dan Kozak, we got it to work with both sets of keys
( id_rsa, which allows one to ssh between computers in our 113 network without typing a password,
 and backup2PB which allows the cron job to push the backup files to the archive in Powell-Booth).

It still goes down every time one reboots fb40m, and I don't have a solution.
A simple solution is for the script to send an email whenever it can't connect via ssh keys
(requiring a restart of ssh-agent with a passphrase), but email doesn't seem to work on fb40m.
I'll see if I can get help on how to have sendmail run on fb40m.

Ever since July 22, the backup script that runs on fb40m has failed to ssh to ldas-cit.ligo.caltech.edu to back up our trend frames and /cvs/cds.

This was a new failure mode which the scripts didn't catch, so I only noticed it when fb40m was rebooted a couple of days ago.

Alex fixed the problem (RAID array was configured with the wrong IP address, conflicting with the outside world), and I modified the script ( /cvs/cds/caltech/scripts/backup/rsync.backup ) to handle the new directory structure Alex made.

Now the backup is current and the automated script should keep it so, at least until the next time fb40m is rebooted...