40m QIL Cryo_Lab CTN SUS_Lab TCS_Lab OMC_Lab CRIME_Lab FEA ENG_Labs OptContFac Mariner WBEEShop
  40m Log  Not logged in ELOG logo
Message ID: 3734     Entry time: Mon Oct 18 11:22:13 2010
Author: Jenne 
Type: Update 
Category: Computers 
Subject: Shame on people not elogging! FrameFiles backups not working. 

On the one hand, SHAME ON ALL PEOPLE WHO DON'T ELOG THINGS, such as the moving of scripts directories (it was a pain to figure out that that's part of why the backup scripts are broken).  On the other hand, the moving of the scripts directories brought to light a critical problem in the backup scheme. None of the frame files have been backed up since Joe replaced fb40m with fb, on ~23 Sept (I think).

What went down:

The frame builder was replaced, and no backup script was started up on the new machine.  Sadface.  Crontab doesn't work yet on the new machine, and also the 'ssh-add' commands give an error:

controls@fb /cvs/cds/rtcds/caltech/c1/scripts/backup $ ssh-add ~/.ssh/id_rsa
No such file or directory
controls@fb /cvs/cds/rtcds/caltech/c1/scripts/backup $ ssh-add ~/.ssh/backup2PB 
No such file or directory

Thus, I know that the backup was never running on the new fb.  However, the check-er script runs on nodus, and looks at the logfile, and since there was no script running, it wasn't adding to the log file, so the last log was an "Okay, everything worked" entry.  So, the check-er script kept sending me daily emails saying that everything was okie-dokey. 

Since all of the scripts were moved (Joe said this happened on Friday, although there's no elog about it), the check-er script, and all of the rest of the backup scripts point to the wrong places (the old scripts/backup directory), so I didn't receive any emails about the backup either way (usually it at least sends a "Hey, I'm broken" email).  This clued me in that we need to check things out, and I discovered that it's all gone to hell.

Since I can't add the ssh clients to the new fb, I can't actually log in to the backup computers over in Powell-Booth to check when the last legitimately successful backup was. But I suspect it was just before the fb was replaced.

So, we need to get Crontab up and running on the new Frame Builder machine so that we can run cron jobs, and we also need to figure out this backup hullabaloo.  I think I'll email / call Dan Kozak over in downs, who was talking about upgrading our backup scheme anyway.


ELOG V3.1.3-