After running into more problems with the upgrade, I eventually decided to abort todays upgrade attempt, and revert back to where we were this morning (RTS 2.1). I'll try to follow this with a fuller report explaining what problems I encountered when attempting the upgrade.
However, when Alex and I were trying to figure out what was going wrong in the upgrade, it appears that the fb symmetricom card lost the ability to sync with the GPS receiver. When the symmeticom module is loaded, dmesg shows the following:
[ 285.591880] Symmetricom GPS card on bus 6; device 0
[ 285.591887] PIC BASE 2 address = fc1ff800
[ 285.591924] Remapped 0x17e2800
[ 285.591932] Current time 947125171s 94264us 800ns
[ 285.591940] Current time 947125171s 94272us 600ns
[ 285.591947] Current time 947125171s 94280us 200ns
[ 285.591955] Current time 947125171s 94287us 700ns
[ 285.591963] Current time 947125171s 94295us 800ns
[ 285.591970] Current time 947125171s 94303us 300ns
[ 285.591978] Current time 947125171s 94310us 800ns
[ 285.591985] Current time 947125171s 94318us 300ns
[ 285.591993] Current time 947125171s 94325us 800ns
[ 285.592001] Current time 947125171s 94333us 900ns
[ 285.592005] Flywheeling, unlocked...
Because of this, the daqd doesn't get the proper timing signal, and consequently is out of sync with the timing from the models.
It's completely unclear what caused this to happen. The card seemed to be working all day today, then Alex and I were trying to debug some other(maybe?) timing issues and the symmetricom card all of a sudden stopped syncing to the GPS. We tried rebooting the frame builder and even tried pulling all the power to the machine, but it never came back up. We checked the GPS signal itself and to the extend that we know what that signal is supposed to look like it looked ok.
I speculate that this is also the cause of the problems were were seeing earlier in the week. Maybe the symmetricom card has just been acting flaky, and something we did pushed it over the edge.
Anyway, we will try to replace it tomorrow, but Alex is skeptical that we have a replacement of this same card. There may be a newer Spectracom card we can use, but there may be problems using it on the old sun hardware that the fb is currently running on. We'll see.
In the mean time, the daqd is running rogue, off of it's own timing. Surprisingly all of the models are currently showing 0x0 status, which means no problems. It doesn't seem to be recording any data, though. Hopefully we'll get it all sorted out tomorrow.