I've set up network with a CNN encoder (front end) feeding into a single LSTM cell followed by the output layer (see attachment #1). The network requires significantly more memory than the previous ones. It takes around 30s for one epoch of training. Attached are the predicted yaw motion and the fft of the same. The FFT looks rather curious. I still haven't done any tuning and these are only the preliminary results.

Well, what about the previous conv nets?

What I did:

1. Extensive tuning - of learning rate, batch size, dropout ratio, input size using a grid search
2. Trained each network for 75 epochs and obtained weights, predicted motion and corresponding FFT, error etc.

What I observed:

1. Loss curves look okay, validation loss isn't going up, so I don't think overfitting is the issue
2. Training for over (even) 75 epochs seems to be pointless.

What I think is going wrong:

1. Input size- relatively large input size: 350 x 350. Here, the input image size seems to be 128 x 128.
2. Inadequate pre-processing.
   1. I have not applied any filters/blurs etc. to the frames.
   2. I have also not tried dimensionality reduction techniques such as PCA

What I will try now:

1. Collect new data: with smaller amplitudes and different frequencies
2. Tune the LSTM network for the data I have
3. Try new CNN architectures with more aggressive max pooling and fewer parameters
4. Ensembling the models (see this and this). Right now, I have multiple models trained either with same architecture and different hyperparameters or with different architectures. As a first pass, I intend to average the predictions of all the models and see if that improves performance.