Rolf and Alex came over this morning to see if they could help debug some issues we have been seeing with IPC transmission between the c1sus and c1lsc machines.

c1oaf, which runs on c1lsc, sees a lot of transmission errors on it's dolphin receivers from c1rfm, which runs on c1sus.  Their speculation is that c1rfm is trying to process too many channels, and it's not able to read off all the RFM channels and retransmit them over dolphin to c1lsc before the end of cycle.  To test this they turned off all RFM reads on c1rfm and the dolphin receiver errors on c1lsc all went away.  We ran into other problems before I had a chance to pester them about what the take-away is here.  We might just need to reduce the load on c1rfm, maybe by introducing a c1rfm2?

We then tried to debug an issue in the c1sus machine where models would occasionally run slow for a cycle, or run slow when a different model on the machine was loaded or unloaded.  The suspect was BIOS settings.  Unfortunately, we ran into trouble when we tried to tweak the BIOS setting on c1sus.  We found that all the serial/COM ports were on, which is usually a big no-no for the RTS (the interrupts cause many cycle delays).  However, turning off the COM ports prevented the machine from booting at all.  This was a big mystery.  The machine seemed to be acting flaky in general as well, since the boot (pre-kernel) would hang in various places after different reboots.  Alex went to grab us a spare machine that we're going to try swapping out this afternoon.