Thank you for the tips, Koji. Frank also mentioned to me today that it would be easier to do the measurement with the loop closed (i.e. measure output / input while driving somewhere else in the loop). I'm not sure how we will be able to do this for the SWEEP -> OUT measurement, though, and we still need to understand why turning the boost on while locked kicks us out of lock.
That's the natural desire, but it is often not straightforward.
It depends on the feature of your LF boost, but in general a LF boost yields a huge gain at DC. Every circuits has some small mount of DC offset and it is amplified by the boost.
e.g. Suppose 1mV input-equivalent offset with DC gain of 10^4. This causes the output of 10V.
What you may be able to do is put an offset to the FFT analyzer output such that the output offset of the box is nulled. If you are lucky, this would let you measure the boost TF.
Even if you have some offset in the servo filter, usually it is not the matter as the servo try to squeeze the offset by the feedback.
Usually small input-equivalent offset of the servo filter turned into huge offset at the output of the filter module, if the loop is not closed.
Once the loop is closed, the feedback signal is adjusted such that the error signal cancels this offset.
Of course, the actual situation depends on how much offset you do actually have. So the above mentions are just general comments.