I turned out that all the noise I was seeing in the QPD spectrum was due to ambient light. I covered the QPD with a box and switched off all the light. As shown in the following plot the noise is lower.

Considering that in the final setup we'll have a beam spot radius of 0.5mm, the sensitivity to beam motion on the QPD will be 23e6 V/m. The following plot shows the resulting beam motion sensitivity, if limited by electroninc noise:

It's at a level of 6e-15 m/rHz at all frequencies above 120 Hz.