To mitigate the issue of ambient light pulluting the QPD signal, I mounted the prototype into a custum built box. This helps a lot. My plan is to add a short piece of black pipe in the front, to further shield from incident light.

The new box also provides a clean way to mount the QPD.