There is no case in which you can run the MAX4107 or LMH with less gain than the minimum gain of the datasheet. If you look at the time series of the PD with low gain its probably oscillating like crazy.

From the datasheet's Bode plots, you can see how the phase margin is for low gains: bad news.