A key step was turning off the whitening filters. With the previous setting (G = 15 dB, white on), the error signals (post anti-whitening) had amplitudes of ~500 counts. This means that they can go as high as (150/15)^2 * 500 = 50000 counts on the ADC.
The purpose of the whitening filter is to match the noise / range of the signal to the ADC. What we would like to do is use the minimum gain so as to make the RFPD electronics noise + shot noise be ~equal to the ADC noise. i.e.
sqrt(V_PD^2 + v_shot^2) * G_white = V_ADC
The RFPD noise is ~3 nV before the internal preamp. The MAX4107 has a gain of 10. There is a factor of 1/2 from the voltage division of the RFPD's 50 Ohm series resistor and the input impedance of the mixer. There is also a power splitter between the PD output and the mixer which gives us a 3 dB loss. The mixer has a conversion loss of ~5-6 dB depending upon the LO level.
V_PD = 3e-9 * (10 * 1/2 * 1/sqrt(2) * 1/2) = 5e-9 V/rHz (this is already bad; the signal coming out of the mixer needs to be amplified by x10 before going out to the whitening board).
In any case, its clear that we need something like 60 dB of gain for the PD noise to match the ADC noise. This is why increasing the whitening gain improves the error signal's SNR, reduces the hash driving the optics, and improves the locking. We should run with 45 dB gain and the switch on whitening after the lock.
Even better would be to modify the LT1128 input stage of the card to have the single stage of fixed whitening as we did for iLIGO. Then we can have triple whitening in lock.