I calculated loss in the cavity by using the cavity's pole and transmission. For our cavity with 38kHz pole, T=300ppm, loss on each mirror is ~13ppm. For AlGaAs coating with 10 ppm loss per mirror (absorption+scatter), T = 120 ppm for Finesse = 22,000 is a good choice for us.
We want to choose a Transmission that is about 10 times larger than the mirror loss. If the transmission is comparable or smaller than the loss, most of the light will loss in the scatter/ absorption and there will be no light coming out.
I calculated to find the loss in the cavity following the instruction from [Siegman, Laser page 436],
the cavities' poles were measured in PSL:425. RCAV = 38kHz, ACAV = 54kHz which corresponds to round trip loss of 24ppm and 160ppm. For now I assume that round trip loss = absorption x 2 + scattered light x2. I used 5 ppm for absorption on each mirror, since it seems to be a conventional number. This gives scatter loss on each mirror = 7 ppm (RCAV) and 75 ppm (ACAV). The number from ACAV is quite bad, and I think it might be due to the fume from the old package, see Frank's comment. I'll use internal loss from RCAV (12 ppm for each mirrro) which gives the transmission of 120 ppm.
==Transmission vs coating layers (thickness) and finesse==
I calculated the transmission of mirror with various thickness [ half wave cap, quarter,(N pairs of quarter/quarter) Subsrtate] (where blue represents SiO2, and green represent Ta2O5) .
table 1: coating structure vs T and finesse for SiO2/Ta2O5.
Note, the thickness for AlAs/GaAs might change, but the relation between T and Finesse will be the same. G. Cole told us that scatter loss is ~ 3-4 ppm, and absorption is ~6 ppm. So the round trip loss is 20 ppm. Here the second table gives the values of finesse and T, with 20ppm loss.
table2: T vs finesse for 20ppm loss roundtrip (AlGaAs).
For SiO2,Ta2O5 coating, I understand that the actual value of transmission can be adjust by changing the thickness of the last few layers, and it should not change the total coating thickness that much. So the above table is still a good quick reference for thickness vs tranmission. If we want T = 120 ppm, the thickness should be around 4.56 micron. This increases the Brownian thermal noise by a factor of sqrt(4.56/4.25) = 1.03, not a big advantage here. For T = 120ppm, the finesse is 22,000.
Photothermal noise will become larger when the cavity's finesse is increased, but it will not be a problem. Photothermal noise is proportional to sqrt of finesse, PSL:1014. With the finesse of 22,000, it will be comparable to the coating thermal noise at DC upto 25 Hz, PSL:1037. For AlAs/GaAs, the coating will be optimized for photo thermal noise, so intensity noise will not be a problem. So I think with T = 120 ppm for finesse = 22000 will be an appropriate choice for us.