There has been another one of "those" events and all of the front end machines are down.
We poked around and Rob determined that the FEs can't get the EPICS data from EPICS. The
dcuepics machine is hooked up and running and all of the epics binaries are running. We also
tried resetting its RFM switch as well as power cycling the box using the "poweroff" command.
Not a sausage.
Rob points out that although the Signal Detect lights are on on the cards, the 'Own Data' light
is not on on the dcuepics' card although it is on for some of the cards on the other boxes.
We have placed messages with the Russian. If anyone sees him, don't let him go without fixing things.
Also, make sure to follow him around with notepad and possibly a camera to record what it is that
he does. If he's muttering, maybe try to use a sensitive hidden sound recorder. |