Turns out the reason that the BEATY signal wasn't working is that one of the two RF amplifiers (both of which are model ZHL-32A), isn't amplifying. Voltage at the pins is fine, so maybe its just broken. When the ZHL-3As that Rana ordered arrive, I'll install those. 

Switching the working amplifier between the two channels, and using a Marconi driving -20dBm (the Y green beatnote amplitude), the phase tracker output RMSs are 70Hz and 150Hz for X and Y, respectively, which isn't too exciting. There is enough whitening gain and filtering that I don't think ADC noise is an issue (The magnitude of the phase tracker Q is ~10kcounts after +6dB whitening gain). 

The RMS in both channels mostly comes from a whole mess of 60Hz harmonics. I'll see what I can do by taking better care of the delay line cables, but it is kind of weird that this would be worse now, given that there was little care given to them before either.

Also, for now, so I don't have to lug the marconi around everywhere, I'm currently driving both channels of the demod board with a spare 55MHz LO output of the LSC LO distribution box, which ends up being a factor of 5 smaller phase tracker error signal, but the noise level is about the same as with the marconi.