I searched for the limiting noise source at high frequency of the beat measurement. I have not founded anything conclusive yet, but the key
seems to be the power on ACAV path.
From previous entries, the evidence suggested that I should look into the PLL readout technique for beat measurement, so I tried:
1) changing the beam spot size on the beat RFPD, to see if it was the result from scattering on the RFPD. the beat still looks the same even the beam is clipped.
2) adjusting + removed the 1/4, 1/2 wave plates that turn the circularly polarized beams from to cavities to linearly polarized beam,
3) using ND filters to reduce power from each of the beam, and both of them,
4) checkig the noise level of the RFPD and the mixer out, gain 10. They are lower than the beat.
So It is probably not the PLL problem. Then I revisited the idea of the noise in ACAV/ RCAV loop.
I used a 0.3 ND filter to reduce the power in RCAV path, after I adjusted the gain, the beat still remained the same.
However, when I used the filter to reduce the power into ACAV, beat was getting higher, and could not be brought down with the higher gain.
So I increased the power to ACAV from 1 mW to 2 mW, and maintained the power to RCAV at 1mW and measured the beat.
The flat high frequency noise goes down by about a factor of 2. This is surprising, I checked the error noise in ACAV loop and made sure that it was not the limiting part. I will think about it.
Note that peaks around 170 Hz are from the periscopes, and peaks around 600Hz are from the mirror mount on the periscopes.
I plan to replace them with better one after RCAV's chamber is fixed tomorrow.