Today, this problem happened again (check history for details). I have done the following investigative steps:

• Situation: The South PMC PZT Ramp was changing by the EPICS voltage but the PZT displacement value was stuck. Also, the South Laser Slow voltage channel also goes back to some value even after changing it.
• I took out the South PMC box from the rack. During this:
  • I had to disconnect the 9V supply used by acromag DACs for setting gain values and switch values.
  • I also had to disconnect the ethernet cable too.
  • I also disconnected RFPD input to the box, HV input, and PMC PZT.
• The first thing I noticed is that the PMC PZT not following the ramp problem is fixed.
• However, the slow output of the laser was going back to a set value (-6.1046V) without any reason.
• I attached the SPMC RFPD RF (Downconverted) signal back to FPTest1 port the PMC just got locked! This happened even though the engage button was switched off.
  • And again, the PMC PZT ramp stopped causing any changes to the PZT displacement value. This is common when the PMC is locked in a normal manner.
• I detached the SPMC RFPD, the changes were happening normally. So the presence of a strong signal might be the issue.
• But, when I connected the RFPD back, the problem was not there anymore. At this point, I found the following reproducible issue:
  • ​Engage button is off.
  • PZT displacement is following the ramp normally.
  • Slowly change the ramp and at the sweet spot, the PMC gets locked. Remember the engage is still off.
  • Now, changing the ramp doesn't change the PZT displacement anymore.
  • Switching on the engage ON doesn't change anything.
  • Switching it back OFF unlocks the PMC and the PZT displacement starts responding to ramp voltage again.
• That's weird right. Since this was reproducible, I did this a few times and found that the problem doesn't necessarily happen at the locking point. It can happen anywhere. And in that case, the 5th step of switching on the engage does show a difference in locked mode. And again, switching it back OFF resolves the issue.
• At this point, I checked back the slow voltage and it had changed to -6.1086. Now apparently this was the new point it liked to stay on regardless of any changes made to the EPICS channel controlling it.
• I checked if the signal for 'PMC Engage' to the U5A AD602. This signal works as expected. The gain setting signal also works as expected. The Ramp setting signal also works as expected. So there is no evidence that the acromag card or the EPICS channel is causing this issue (at least in the PMC problem).
• It could be that the AD602 switches ON itself due to some reason and doesn't switch off until it explicitly sees a low-high change on its pin 4.
• Also, it could be that the U9 PA85 is the one causing trouble. But I doubt that as the behavior matches more with engaging the lock upstream at U5A AD602.
• I restarted the lab by restarting the channels and scripts (this is done in a preset protocol by restartAll command).
• This resolved the issue with the slow voltage of the laser. But the PMC problem persists. In fact, it is not locking now.

I'll take another inspection with a fresh mind next time. This problem needs to be resolved as we can't leave some unexplained phenomenon to keep happening in the lab.