[Jenne, Mirko, Kiwamu, Koji, and Jamie by phone]
We just got off the phone with Jamie. In addition to all the stuff that Kiwamu mentioned, Mirko reverted the c1oaf model and C-code to stuff that was working successfully on Friday (using "svn export, rev # 1134) since that's what we were working on when all hell broke loose.
We did a few rounds of "sudo shutdown -h now" on c1lsc and c1sus machines, and pulled the power cords out.
We also switched the c1ioo and c1lsc 1PPS fibers in the fanout chassis, to see if that would fix the problem. Nope. c1ioo is still fine, and c1lsc is still not fine.
Still getting "No Sync".
We're going to call in Alex in the morning, if we can't figure it out soon. |