Den noticed this, and will write more later, I just wanted to sum up what Alex said / did while he was here a few minutes ago....
Errors are probably really happening.... c1oaf computer status 4-bit thing: GRGG. The Red bit is indicating receiving errors. Probably the oaf model is doing a sample-and-hold thing, sampling every time (~1 or 2 times per sec) it gets a successful receive, and then holding that value until it gets another successful receive.
Den is adding EPICS channels to record the ERR out of the PCIE dolphin memory CDS_PART, so that we can see what the error is, not just that one happened.
Alex restarted oaf model: sudo rmmod c1oaf.ko, sudo insmod c1oaf.ko . Clicked "diag reset" on oaf cds screen several times, nothing changed. Restarted c1oaf again, same rmmod, insmod commands.
Den, Alex and I went into the IFO room, and looked at the LSC computer, SUS computer, SUS I/O chassis, LSC I/O chassis and the dolphin switch that is on the back of the rack, behind the SUS IO chassis. All were blinking happily, none showed symptoms of errors.
Alex restarted the IOP process: sudo rmmod c1x04, sudo insmod c1x04. Chans on dataviewer still bad, so this didn't help, i.e. it wasn't just a synchronization problem. oaf status: RRGG. lsc status: RGGG. ass status: RGGG.
sudo insmod c1lsc.ko, sudo insmod c1ass.ko, sudo insmod c1oaf.ko . oaf status: GRGG. lsc status: GGGG. ass status: GGGG. This means probably lsc needs to send something to oaf, so that works now that lsc is restarted, although oaf still not receiving happily.
Alex left to go talk to Rolf again, because he's still confused.
Comment, while writing elog later: c1rfm status is RRRG, c1sus status is RRGG, c1oaf status is GRGG, both c1scy and c1scx are RGRG. All others are GGGG.