40m QIL Cryo_Lab CTN SUS_Lab TCS_Lab OMC_Lab CRIME_Lab FEA ENG_Labs OptContFac Mariner WBEEShop
  40m Log  Not logged in ELOG logo
Entry  Mon Oct 20 15:02:00 2008, Yoichi, Configuration, Computers, /cvs/cds restored 
    Reply  Mon Oct 20 16:18:00 2008, Alan, Configuration, Computers, /cvs/cds restored 
Message ID: 1060     Entry time: Mon Oct 20 16:18:00 2008     In reply to: 1059
Author: Alan 
Type: Configuration 
Category: Computers 
Subject: /cvs/cds restored 

I moved missing files in /cvs/cds restored by Alan and Stuart to the original locations.
I confirmed autoburt runs, and dtt, which had also been having trouble running, runs ok now.

I found an interesting piece of evidence on allegra, our new 64bit linux machine.
In the Trash of controls Desktop on that machine, there is /cvs/cds/vw/ directory.
I remember that when I last time emptied the trash bin on the machine (yesterday), it took somewhat long time.
Too bad that I did not pay attention to what was actually in the Trash, but now I have a feeling that in the Trash were
missing /cvs/cds/* directories.
While emptying the Trash, I encountered several errors saying permission denied or something like that, and skipped those files.
Sometimes, when you move something from NFS mounted directories to the Trash, you get this kind of errors.
So my guess is that someone accidentally (or intentionally) moved /cvs/cds/* except for "caltech" to the Trash of allegra.
And I completely removed them carelessly.

In the meantime, I have re-started the nightly backup for /frames/minute-trends
but NOT YET for /cvs/cds ,
since I fear that we'll find another problem and will need to go back to the June 27 backup.
Let's wait a few days for the dust to settle,
and if everyone feels confident that /cvs/cds is ok,
I'll restart the backup of that.

How I restored the files, for the record:

Stuart mounted /archive/backup onto an accessible computer (garrak.ligo.caltech.edu ) and I logged on to controls@nodus and ran this command:

/cvs/cds/caltech/scripts/backup/rsync --rsync-path=/usr/bin/rsync --rsh=/usr/bin/ssh --compress --verbose --archive --hard-links --exclude=caltech/ ajw@garrak.ligo.caltech.edu:/backup/40m/cvs /cvs/cds/recover_20081020

I had to type in my GC password, and it ran for ~20 minutes (would have been much longer had I asked for /cvs/cds/caltech as well!).

you can view the backups by logging on to garrak.ligo.caltech.edu with your GC account:
ELOG V3.1.3-