How about to measure the AR reflectivity at larger (but small) angles the extrapolate the function to smaller angle,
or estimate an upper limit?

The spot separation is

D = 2 d Tan(\phi) Cos(\theta), where \phi = ArcSin(Sin(\theta) * n)

D = 2 d Tan(\phi) Cos(\theta), where \phi = ArcSin(Sin(\theta) / n)         (<== correction by Manasa's entry)

\theta is the angle of incidence. For a small \theta, D is propotional to \theta.

So If you double the incident angle, the beam separation will be doubled,
while the reflectivity is an even function of the incident angle (i.e. the lowest order is quadratic).

I am not sure until how much larger angle you can use the quadratic function rather than a quartic function.
But thinking about the difficulty you have, it might be worth to try.