40m QIL Cryo_Lab CTN SUS_Lab TCS_Lab OMC_Lab CRIME_Lab FEA ENG_Labs OptContFac Mariner WBEEShop
  40m Log  Not logged in ELOG logo
Entry  Sun Aug 28 22:49:40 2011, Suresh, Update, CDS, fb down 
    Reply  Mon Aug 29 00:49:00 2011, kiwamu, Update, CDS, Re : fb down 
       Reply  Mon Aug 29 12:05:32 2011, jamie, Update, CDS, Re : fb down 
          Reply  Mon Aug 29 18:16:10 2011, jamie, Update, CDS, Re : fb down 
             Reply  Tue Aug 30 11:28:56 2011, jamie, Update, CDS, framebuilder back up 
                Reply  Tue Aug 30 14:33:52 2011, jamie, Update, CDS, all front-ends back up and running 
Message ID: 5317     Entry time: Mon Aug 29 12:05:32 2011     In reply to: 5316     Reply to this: 5319
Author: jamie 
Type: Update 
Category: CDS 
Subject: Re : fb down 

fb was requiring manual fsck on it's disks because it was sensing filesystem errors.  The errors had to do with the filesystem timestamps being in the future.  It turned out that fb's system date was set to something in 2005.  I'm not sure what caused the date to be so off (motherboard battery problem?)  But I did determine after I got the system booting that the NTP client on fb was misconfigured and was therefore incapable of setting the system date.  It seems that it was configured to query a non-existent ntp server.  Why the hell it would have been set like this I have no idea.

In any event, I did a manual check on /dev/sdb1, which is the root disk, and postponed a check on /dev/sda1 (the RAID mounted at /frames) until I had the system booting.  /dev/sda1 is being checked now, since there are filesystems errors that need to be corrected, but it will probably take a couple of hours to complete.  Once the filesystems are clean I'll reboot fb and try to get everything up and running again.

ELOG V3.1.3-