This morning we attempted to replace the c1sus front end machine with a spare that had been given a second CPU, and therefore 6 additional cores (for a total of 12). The idea was to give c1sus more cores so that we could split up c1rfm into two separate models that would not be running on the hairy edge of their cycle time allocation. Well, after struggling to get it working we eventually aborted and put the old machine back in.
The problem was that the c1sus model was running erratically, frequently jumping up to 100 usec of a 60 usec clock allocation. We eventually tracked the problem down to the fact that the CPUs in the new machine are of an inferior and slower model, than what's in the old c1sus machine. The CPU were running about 30% slower, which was enough to bump c1sus, which nominally runs at ~51 usec, over it's limit.
This is of course stupid, and I take the blame. I skimped on the CPUs when I bought the spare machines in an attempt to keep the cost down, and didn't forgot that I had done that when we started discussing using one of the spares as a c1sus replacement.
I think we can salvage things, though, by just purchasing a better CPU, one that matches what's currently in c1sus. I'll get Steve on it:
In any event, the old c1sus is back in place, and everything is back as it was.