The backup of /cvs/cds (which runs as a cron job on fb40m; see /cvs/cds/caltech/scripts/backup/000README.txt) 
has been down since fb40m was rebooted on March 3.
I was unable to start it because of conflicting ssh keys in /home/controls/.ssh .
With help from Dan Kozak, we got it to work with both sets of keys
( id_rsa, which allows one to ssh between computers in our 113 network without typing a password,
 and backup2PB which allows the cron job to push the backup files to the archive in Powell-Booth).

It still goes down every time one reboots fb40m, and I don't have a solution.
A simple solution is for the script to send an email whenever it can't connect via ssh keys
(requiring a restart of ssh-agent with a passphrase), but email doesn't seem to work on fb40m.
I'll see if I can get help on how to have sendmail run on fb40m.

After the boot-fest, the nightly backup to Powell-Booth failed, and an automatic email got sent to me. I restarted the ssh agent, following the instructions in /cvs/cds/caltech/scripts/backup/000README.txt .

CES Mezzanine is beeing rebuilt to accommodate our new neighbor: the 20ft high water slide...& .jacuzzi

All our ac power transformers are up there. Yesterday we labelled the power switch of 480VAC on the mezz

that we need to keep to run the 3 cranes in the lab.

After many, many "it'll be there in 2 weeks" from the Guralp people, our seismometer is finally back!

I have it plugged into the Guralp breakout box's Channel 1xyz (so I have unplugged the other Guralp).  Both of the Guralp's are currently sitting under the MC1/MC3 chamber.

Before we can have both Guralps up and running, I need to stuff the next 3 channels of the breakout box (back in the fall, I only had Caryn do 1x, 1y, 1z, and now I need 2x, 2y and 2z done with the fancy low-noise resistors), so all the gains match between the 2 sets of channels.

I'm leaving the new Guralp plugged in so we can see how it behaves for the next couple days, until I take out the breakout box for stuffing.

In response to Steve's elog entry, and for 40m posterity, I provide the Paschen Curve.

Restarted backup since fb40m was rebooted.

2009 May 18 03:39:36 UTC

Earthquake Details

 Wilcoxon 731A seismic accelerometers and Guralp CMG-40T-old seismometer at magnitude 5 and 4 erthquakes

Rana suggested using OSEM sensing voltages as guide lines to look seismic activity.

As you see todays drilling and tumping activity was nothing compared to the EQ of mag 5 and 4

Optic level servos are turned back on.

What Steve means is that there is some drilling going on in the CES shop to accomodate the new water flume group. We want to

make sure that the mirrors don't move enough to break the magnets. On the dataviewer we should look to make sure that the

sensor channels stay between 0-2 V.     -Rana

Once again, this morning I found the wireless router disconnected from the LAN cable. No martian WiFi was available.

I wonder who is been doing that and for what reason.

I have increased the temperature setpoint in the office area by ~0.75 deg F. Figure attached. Also a few days ago I increased the setpoint of the AC in the control room. Looks like the Laser is able to handle the changes in office area temperature so far, but lets see how it fares over the weekend.

Before heading back to the 40m to check on the computer situation, I thought I'd check the web screenshots page that Kakeru worked on, and it looks like none of the screens have been updated since June 1st.  I don't know what the story is on that one, or how to fix it, but it'd be handy if it were fixed.

 Apparently I broke this when I added op540m to the webstatus page.  It's fixed now.

in the rack next to the printer.  It sounds like a fan is hitting something.

Ever since July 22, the backup script that runs on fb40m has failed to ssh to ldas-cit.ligo.caltech.edu to back up our trend frames and /cvs/cds.

This was a new failure mode which the scripts didn't catch, so I only noticed it when fb40m was rebooted a couple of days ago.

Alex fixed the problem (RAID array was configured with the wrong IP address, conflicting with the outside world), and I modified the script ( /cvs/cds/caltech/scripts/backup/rsync.backup ) to handle the new directory structure Alex made.

Now the backup is current and the automated script should keep it so, at least until the next time fb40m is rebooted...

Steve, Rana, Ben, Jenne, Alberto, Rob

UPS in the vacuum rack failed this afternoon, cutting off power to the vacuum control system.  After plugging all the stuff that had been plugged into the UPS into the wall, everything came back up.  It appears that V1 closed appropriately, TP1 spun down gracefully on its own battery, and the pressure did not rise much above 3e-6 torr. 

The UPS fizzed and smelled burnt.  Rana will order a new, bigger, better, faster one.

We've had a devil of a time getting V1 to open, due to the Interlock code. 

The short story is that if C1:Vac-PTP1_pressure > 1.0, the interlock code won't let you push the button to open V1 (but it won't close V1). 

PTP1 is broken, so the interlock was frustrating us.   It's been broken for a while, but this hasn't bitten us till now.

We tried swapping out the controller for PTP1 with one of Bob's from the Bake lab, but it didn't work. 

It said "NO COMM" in the C1:Vac-PTP1_status, so I figured it wouldn't update if we just used tdswrite to change C1:Vac-PTP1_pressure to 0.0.  This actually worked, and V1 is now open.  This is a temporary fix.

There were no injuries...Now we need to get some new chairs.

This morning I found that all the front end computers down. A failure of the RFM network drove all the computers down.

I was about to restart them all, but it wasn't necessary. After I power cycled and restarted C1SOSVME all the other computers and RFM network came back to their green status on the MEDM screen. After that I just had to reset and then restart C1SUSVME1/2.

I (think I) restarted DMF.  It's on Mafalda, running in matlab (not the complied version which Rana was having trouble with back in the day).  To start Matlab, I did "nohup matlab", ran mdv_config, then started seisBLRMS.m running.  Since I used nohup, I then closed the terminal window, and am crossing my fingers in hopes that it continues to work.  I would have used Screen, but that doesn't seem to work on Mafalda.

 Just kidding. That plan didn't work.  The new plan: I started a terminal window on Op540, which is ssh-ed into Mafalda, and started up matlab to run seisBLRMS.  That window is still open. 

Because Unix was being finicky, I had to open an xterm window (xterm -bg green -fg black), and then ssh to mafalda and run matlab there.  The symptoms which led to this were that even though in a regular terminal window on Op540, ssh-ed to mafalda, I could access tconvert, I could not make gps.m work in matlab.  When Rana ssh-ed from Allegra to Op540 to Mafalda and ran matlab, he could get gps.m to work.  So it seems like it was  a Unix terminal crazy thing. Anyhow, starting an xterm window on Op540m and ssh-ing to mafalda from there seemed to work.

Hopefully this having a terminal window open and running DMF will be a temporary solution, and we can get the compiled version to work again soon.