I added one more amplifier stage (ZFL-500 LN) after the detector. Since noise figure of ZFL-500LN (2.9 dB) is lower than that of ZHL-3A (5 dB), ZFL-500LN is the first amplifier stage after the photo detector and it is followed by ZHL-3A.

Attachment # 1 shows the beat note spectrum measured from the spectrum analyser. There was a 30 dB attenuator in the path during the measurement. So, the output RF power from the MZI (with two stages of amplification) is now about 3 dBm and the SNR of 37 dB is preserved even after two stages of amplification.

So, now the RF power to the RF port of the mixer is 3 dBm. I have attached the video of signal from the PLL loop at different gain (G=1, G=2,G=5) values in SR 560. The time domain trace seems to very noise. I suspect this is because of the inherent large noise in 2-micron laser diode with a broad line width of 2 MHz.

I then attempted to do the closed loop transfer function in the present PLL configuration by injecting the signal from SR 785. Attachment 2 shows the closed and open loop transfer functions at different gain values in SR 560 when the actuation slope is 10 kHz/V. Attachment 3 shows the closed and open loop transfer functions at different values of actuation slope when the gain is 5. The magnitude and phase traces are not very smooth as we observed when we did the similar measurement with an arbitrary function generator (AFG) as the RF source. In this case, when MZI output is fed in as the RF source, the RF power is fluctuating.

I also tried to do the frequency noise measurement. Attchement # 4 is the FM noise at different gain values when the actuation slope is 10 kHz/V. Attachement 5 is the FM noise at different actuation slope values when the gain is 5. This time, depending on the gain value and the actuation slope value, a short frequency span was considered in SR 785 for the frequency noise measurement. The frequency span is considered based on the value of unity gain frequencies  that are approximated from the open loop transfer functions measured from attachment # 2 and #3