After reproting phase noise issues with the VCO, here, I have changed the VCO for a high power RF amplifier - a mini circuits ZHL--5W==sma (ZHL-5W-1). It is being driven by thre Moku to provide the 80 MHz signal. Please note the following:
The new setup is shown in attachment 1. Attachment 2 show the phase noise of the LO arm. This is now a much more acceptable level, as compared to the previous VCO. I am outputting -19.2 dBm, and measuring 5.3 dBm on input 2. This means I am driving the AOM at 24.8 dBm, or 0.302 W.
With the greater drive on the AOM, as compared to previous incorrect useage of the VCO, the signal from the IFO arm is much more robust. Attachment 3 shows the phase time series of the IFO arm. You will notice the large jumps, which spoil the spectrum, not shown here. In attachment 4 I zoom into the largest of these jumps. You can see it is comprised of several linear transitions, which occur over a short time period.
Aidan has suggested that locking the laser is the best way to avoide these non-stationary noise sources. He suggests that these jumps could be mode hops of the laser.