I popped by the 40m, and was dismayed to find that all of the front end computers are red (only framebuilder, DAQcontroler, PEMdcu, and c1susvmw1 are green....all the rest are RED).
I keyed the crates, and did the telnet.....startup.cmd business on them, and on c1asc I also pushed the little reset button on the physical computer and tried the telnet....startup.cmd stuff again. Utter failure.
I have to pick someone up from the airport, but I'll be back in an hour or two to see what more I can do.
I think the problem was caused by a failure of the RFM network: the RFM MEDM screen showed frozen values even when I was power recycling any of the FE computers. So I tried the following things:
After Alberto's bootfest which was more successful than mine, I tried powercycling the AWG crate one more time. No success. Just as Alberto had gotten, I got the DAQ screen's AWG lights to flash green, then go back to red. At Alberto's suggestion, I also gave the physical reset button another try. Another round of flash-green-back-red ensued.
When I was in a few hours ago while everything was hosed, all the other computer's 'lights' on the DAQ screen were solid red, but the two AWG lights were flashing between green and red, even though I was power cycling the other computers, not touching the AWG at the time. Those are the lights which are now solid red, except for a quick flash of green right after a reboot.
I poked around in the history of the curren and old elogs, and haven't found anything referring to this crazy blinking between good and bad-ness for the AWG computers. I don't know if this happens when the tpman goes funky (which is referred to a lot in the annals of the elog in the same entries as the AWG needing rebooting) and no one mentions it, or if this is a new problem. Alberto and I have decided to get Alex/someone involved in this, because we've exhausted our ideas.