40m QIL Cryo_Lab CTN SUS_Lab TCS_Lab OMC_Lab CRIME_Lab FEA ENG_Labs OptContFac Mariner WBEEShop
  40m Log  Not logged in ELOG logo
Entry  Thu May 23 15:37:30 2019, Milind, Update, Cameras, Simulation enhancements and performance of contour detection 6x
    Reply  Sat May 25 20:29:08 2019, Milind, Update, Cameras, Simulation enhancements and performance of contour detection residue_normalised_x.pdfresidue_normalised_x.pdfresidue_normalised_x.pdfpredicted_motion_x.pdfnormalised_comparison_y.pdf
       Reply  Wed Jun 12 22:02:04 2019, Milind, Update, Cameras, Simulation enhancements  simulated_motion0.mp4simulated_motion0.mp4simulated_motion0.mp4
          Reply  Mon Jun 17 14:36:13 2019, Milind, Update, Cameras, Convolutional neural networks for beam tracking 
             Reply  Tue Jun 18 22:54:59 2019, Milind, Update, Cameras, Convolutional neural networks for beam tracking 
                Reply  Tue Jun 25 00:25:47 2019, Milind, Update, Cameras, Convolutional neural networks for beam tracking 8x
                   Reply  Tue Jun 25 22:14:10 2019, Milind, Update, Cameras, Convolutional neural networks for beam tracking 
                      Reply  Thu Jun 27 20:48:22 2019, Milind, Update, Cameras, Convolutional neural networks for beam tracking readme.txtframe0.pdfLearning_curves.pngMotion.png
                         Reply  Thu Jul 4 18:19:08 2019, Milind, Update, Cameras, Convolutional neural networks for beam tracking Motion.pdfError.pdfLearning_curves.pdf
                            Reply  Mon Jul 8 17:52:30 2019, Milind, Update, Cameras, Convolutional neural networks for beam tracking Motion.pdf
                               Reply  Tue Jul 9 22:13:26 2019, Milind, Update, Cameras, Convolutional neural networks for beam tracking predicted_motion_first.pdfpcdev5_time.png
                                  Reply  Wed Jul 10 22:32:38 2019, Milind, Update, Cameras, Convolutional neural networks for beam tracking 
                               Reply  Mon Jul 15 14:09:07 2019, Milind, Update, Cameras, CNN LSTM for beam tracking cnn-lstm.pngfft_yaw.pdfyaw_motion.pdf
                                  Reply  Fri Jul 19 16:47:06 2019, Milind, Update, Cameras, CNNs for beam tracking || Analysis of results 7x
                                     Reply  Sat Jul 20 12:16:39 2019, gautam, Update, Cameras, CNNs for beam tracking || Analysis of results 
                                        Reply  Sat Jul 20 14:43:45 2019, Milind, Update, Cameras, CNNs for beam tracking || Analysis of results frame0.pdfsubplot_yaw_test.pdfintensity_histogram.mp4network2.pdf
                                           Reply  Wed Jul 24 20:05:47 2019, Milind, Update, Cameras, CNNs for beam tracking || Tales of desperation saturation_percentage.pdf
                   Reply  Thu Jul 25 00:26:47 2019, Milind, Update, Cameras, Convolutional neural networks for beam tracking centroid.pdfsubplot_yaw_test.pdf
          Reply  Mon Jun 17 22:19:04 2019, Milind, Update, Computer Scripts / Programs, PMC autolocker 
             Reply  Mon Jul 1 20:18:01 2019, Milind, Update, Computer Scripts / Programs, PMC autolocker 
                Reply  Tue Jul 2 12:30:44 2019, Milind, Update, Computer Scripts / Programs, PMC autolocker 
                   Reply  Sun Jul 7 17:54:34 2019, Milind, Update, Computer Scripts / Programs, PMC autolocker 
          Reply  Tue Jun 25 23:52:37 2019, Milind, Update, Cameras, Simulation enhancements  
             Reply  Mon Jul 1 20:11:34 2019, Milind, Update, Cameras, Simulation enhancements  
Message ID: 14726     Entry time: Thu Jul 4 18:19:08 2019     In reply to: 14706     Reply to this: 14734
Author: Milind 
Type: Update 
Category: Cameras 
Subject: Convolutional neural networks for beam tracking 

The quoted elog has figures which indicate that the network did not learn (train or generalize) on the used data. This is a scary thing as (in my experience) it indicates that something is fundamentally wrong with either the data or model and learning will not happen despite how hyperparameters are tuned. To check this, I ran the training experiment for nearly 25 hyperparameter settings (results here)with the old data and was able to successfully overfit the data. Why is this progress? Well, we know that we are on the right track and the task is to reduce overfitting. Whether, that will happen through more hyperparameter tuning, data collection or augmentation remains to be seen. See attachments for more details. 

Why is the fit so perfect at the start and bad later? Well, that's because the first 90% of the test data is  the training data I overfit to and the latter the validation data that the network has not generalized well to.

Quote:

And finally, a network is trained!

Result summary (TLDR :-P) : No memory was used. Model trained. Results were garbage. Will tune hyperparameters now. Code pushed to github.

 

More details of the experiment:

Aim:

  1. To train a network to check that training occurs and get a feel for what the learning might be like.
  2. To set up the necessary framework to perform mulitple experiments and record results in a manner facilitating comparison.
  3. To track beam spot motion.

What I did:

  1. Set up a network that learns a framewise mapping as described in here.
  2. Training data: 0.9 x 1791 frames. Validation data: 0.1 x 1791 frames. Test data (only prediction): all the 1791 frames
  3. Hyperparameters: Attachment #1
  4. Did no tuning of hyperparameters.
  5. Compiled and fit the models and saved the results.

 

What I saw

  1. Attachment #2: data fed to the network after pre-processing - median blur + crop
  2. Attachment #3: learning curves.
  3. Attachment #4: true and predicted motion. Nothing great.

What I think is going wrong-

  1. No hyperparameter tuning. This was only a first pass but is being reported as it will form the basis of all future experiments.
  2. Too little data.
  3. Maybe wrong architecture.

Well, what now?

  1. Tune hyperparmeters (try to get the network to overfit on the data and then test on that. We'll then know for sure that all we probably need is more data?)
  2. Currently the network has around 200k parameters. Maybe reduce that.
  3. Set up a network that takes as input (one example corresponding to one forward pass)  a bunch of frames and predicts a vector of position values that can be used as continuous data).
Quote:

I got to speak to Gabriele about the project today and he suggested that if I am using Rana's memory based approach, then I had better be careful to ensure that the network does not falsely learn to predict a sinusoid at all points in time and that if I use the frame wise approach I try to somehow incorporate the fact that certain magnitudes and frequencies of motion are simply not physically possible. Something that Rana and Gautam emphasized as well.

Quote:
 
  1. Network training for beam spot tracking: I will begin training the convolutional network with the data pre-processed as described above. I will also simultaneously prepare data acquired from the GigE and train networks on that. Note: I planned to experiment with framewize predictions and hence did some of the work described above. However, I will restrict the number of experiments on that and perform more of those that use 3D convolution. Rana also pointed out that it would be interesting to have the network output uncertainity in the predictions. I am not sure how this can be done, but I will look into it.
Attachment 1: Motion.pdf  31 kB  Uploaded Thu Jul 4 23:13:05 2019  | Hide | Hide all
Motion.pdf
Attachment 2: Error.pdf  27 kB  Uploaded Thu Jul 4 23:13:19 2019  | Hide | Hide all
Error.pdf
Attachment 3: Learning_curves.pdf  13 kB  Uploaded Thu Jul 4 23:13:41 2019  | Hide | Hide all
Learning_curves.pdf
ELOG V3.1.3-