I did some theoretical calculations using the datasheet value of second harmonic SFDR from MAX4107 and the transfer function I measured from Test IN ports of RFPDs (using 100 kOhm series resistance).

Calculation:

• Attached notebook has all the calculations. In all places, "Omega" refers to the modulation frequency.
• I first calculated power at 0, 1-Omega and 2-Omega from the resonance frequency in the light incident on the cavity using Bessel functions.
• I used reflection function from the cavity (using transmittance of 5 ppm, cavity length of 1.45" and assuming lossless cavity):
  
• Using this, I calculated reflected power at  0, 1-Omega and 2-Omega from the resonance frequency.
• Then I used -40 dBc SFDR for second harmonic generation mentioned in MAX4107 datasheet when used with a gain of 10. I used following formula to calculate this:
  
• Finally, I also calculated ratio of 1-Omega signal (actual beat between Sideband and Carrier) and the 1-Omega Distortion signal (generated) due to the non-linearity of MAX4107.

Inferences:

• These are purely theoretical calculations but I can put in the second harmonic SFDR by measurement tomorrow.
• The suppression of 2nd harmonics by the notch in the RFPD circuits is very bad. It only suppresses the second harmonic by 25-30 dB.
• This is not enough as this 2-Omega signal would mix a lot with 1-Omega and would also go forward into the TTFSS box after downconversion.
• The ratio of Signal at 1-Omega to the distortion at 1-Omega is about just 8 at 100 Hz. This doesn't reach 100 even till 1 kHz.
• And below 10 Hz, the distortion dominates the 1-Omega signal from RF out.
• This looks like a big limitation to our FSS loop. The notch filters in the RFPDs need to be doing much better job then they are doing.

Edited Wed Nov 20 14:43:07 2019: Corrected an error in code.