Ok, hmm,  still doesn't make sense. The first problem is that you don't have 50Ohm termination for the mixer signal when measuring with the scope, but if you connect it to the pdh box you have 50 Ohms. But that can't explain the plot alone.

Hmm, the amplifier might be a problem, it's a low-noise one but i didn't check the RF noise after installation. So remove the amplifier and redo the measurements. Be careful as the size of the error signal changes quite a lot and you have to turn the gain of the box back to 8 or so. So you have to re-measure everything.