We have been trying to figure out why we see (more) excess low frequency noise in the current configuration. The ~100x optical gain imbalance could have been an unrelated issue, but it seemed like a reasonable place to start today.
We began by bypassing the PDs altogether and just injecting part of the LO signal directly into the RF input of the mixers (where the RFPD signal comes in). This should create a DC signal at the low-passed IF output, which can be maximized by tuning the LO phases with the dials on the boxes. Using equal input RF levels, the maximized outputs of the respective mixers were different by a factor of ~100 (220 mV to 2 mV). We double-checked that the LO levels were equal, and since they were we deduced that something was wrong with the primary box's demod electronics. Checking each of the parts individually, we found that the ZX05 mixer was busted. We replaced it and the optical gain is now 100x higher. This explains the discrepancy we have been noticing from the theoretically achievable level for months.
I reduced the gain of the servo to accommodate the optical gain, but we still needed >10dB of attenuation between the PD and the mixer to keep the loop from oscillating. This needs investigation.
When we plugged everything back in and took a spectrum, we found that the low-frequency noise was back to the level (a bit lower, in fact) where it was before we reduced the optical power from 30 mW to 3 mW. It is a bit worse at higher frequencies, but this is the best we have ever done in the current configuration. Plot:
It is a bit disheartening that there isn't a real improvement, but we must consider this: