The framebuilder is back online now, minus it's symmetricom GPS card.  The card seems to have failed entirely, and was not able to be made to work at downs either.  It has been entirely removed from fb.

As a fall back, the system has been made to work off of the system NTP-based time synchronization.  The latest symmetricom driver, which is part of the RCG 2.4 branch, will fall back to using local time if the GPS synchronization fails.  The new driver was compiled from our local checkout of the 2.4 source in the new to-be-used-in-the-future rtscore directory:

controls@fb ~ 0$ diff {/opt/rtcds/rtscore/branches/branch-2.4/src/drv/symmetricom,/lib/modules/2.6.34.1/kernel/drivers/symmetricom}/symmetricom.ko
controls@fb ~ 0$ 

The driver was reloaded.   daqd was also linked against the last running stable version and restarted:

controls@fb ~ 0$ ls -al $(readlink -f /opt/rtcds/caltech/c1/target/fb/daqd)
-rwxr-xr-x 1 controls controls 6592694 Dec 15 21:09 /opt/rtcds/caltech/c1/target/fb/daqd.20120104
controls@fb ~ 0$ 

We'll have to keep an eye on the system, to see that it continues to record data properly, and that the fb and the front-ends remain in sync.

The question now is what do we do moving forward.  CDS is not supporting the symmetricom cards anymore, and have moved to using Spectracom GPS/IRIG-B cards.  However, Downs has neither at the moment.  Even if we get a new Spectracom card, it might not work in this older Sun hardware, in which case we might need to consider upgrading the framebuilder to a new machine (one supported by CDS).