Found at least two (minor) problems:

1. So far i measured the zero-crossing of the mixer signal and both peak values and modeled the signal using a sine.
   For calibration i used the slope of the sine near the zero-crossing.
   As the frequency range for pi in shift is about 60MHz the function is almost linear for the range we measure over 24h (~2MHz).
   Today i measured the mixer signal in small steps and compared both techniques. The result surprised me a little bit:
   

 

it is not a lot of difference in slope but it is surprisingly linear over a range of 40MHz!

2. the channel which contains the calibrated VCO monitor signal uses a wrong sign.
   The fit is right, but i remembered that i exchanged the BNC to 2-pole LEMO connector some time ago because it had the wrong sign.
   However, the frequency tuning curve i measured with the old (wrong) cable.
   I used the right sign on my computer but implemented the wrong one in EPICS, so that data was/is wrong. Will change that tomorrow.
   This can't explain the problems i had last week as i used Matlab to convert the monitor signal into frequency, not that channel.
   Anyway, this is how the data i took last night (19h) looks like using the right slope (see item 1):
   
   so i think once the EPICS channel is fixed we can trust that channel