It turned out that the UPS installed on Nov 22 failed (cf https://nodus.ligo.caltech.edu:8081/40m/16479 ). As a fact, it was alive just for 2 weeks!
The APC UPS unit indicated F06. According to the manual (https://www.apc.com/shop/us/en/products/APC-Power-Saving-Back-UPS-Pro-1000VA/P-BR1000G), F06 means "Relay Welding" and can not be fixed by a user. Resetting the UPS eliminated the error, but I didn't want to have the same issue while no one is in the lab, I moved the megatron power source from the UPS to the power strip on 1Y7. So, megatron is currently vulnerable to a power glitch.
After the power cords were restored, megatron eventually recovered ssh terminals. I manually ran autoburt.cron at 16:50 so that the latest snapshot is taken. |