At the moment, c1sus and c1mcs on the c1sus machine seem to be dead in the water. At this point, it is unclear to me why.
Apparently during the 40m meeting, Alex was able to get test points working for the c1mcs model. He said he "had to slow down mx_stream startup on c1sus". When we returned at 2pm, things were running fine.
We began updating all the matrix values on the medm screens. Somewhere towards the end of this the c1sus model seemed to have crashed, leaving only c1x02 and c1mcs running. There were no obvious error messages I saw in dmesg and the target/c1sus/logs/log.txt file (although that seems to empty these days). We quickly saved to burt snap shots, one of c1sus and one of c1mcs and saved them to /opt/rtcds/catlech/c1/target/snapshots directory temporarily. We then ran the killc1sus script on c1sus, and then after confirming the code was removed, ran the startup script, startc1sus. The code seemed to come back partly. It was syncing up and finding the ADC/DAC boards, but not doing any real computations. The cycle time was reporting reasonably, but the usr time (representing computation done for the model) was 0. There were no updating monitor channels on the medm screens and filters would not turn on.
At this point I tried bringing down all 3 models, and restarting c1x02, then c1sus and c1mcs. At this point, both c1sus and c1mcs came back partly, doing no real calculations. c1x02 appears to be working normally (or at least the two filter banks in that model are showing changing channels from ADCs properly). I then tried rebooting the c1sus machine. It came back in the same state, working c1x02, non-calculating c1sus and c1mcs.