40m
QIL
Cryo_Lab
CTN
SUS_Lab
TCS_Lab
OMC_Lab
CRIME_Lab
FEA
ENG_Labs
OptContFac
Mariner
WBEEShop
|
40m Log |
Not logged in |
 |
|
Thu May 23 15:37:30 2019, Milind, Update, Cameras, Simulation enhancements and performance of contour detection 6x
|
Sat May 25 20:29:08 2019, Milind, Update, Cameras, Simulation enhancements and performance of contour detection    
|
Wed Jun 12 22:02:04 2019, Milind, Update, Cameras, Simulation enhancements  
|
Mon Jun 17 14:36:13 2019, Milind, Update, Cameras, Convolutional neural networks for beam tracking
|
Tue Jun 18 22:54:59 2019, Milind, Update, Cameras, Convolutional neural networks for beam tracking
|
Tue Jun 25 00:25:47 2019, Milind, Update, Cameras, Convolutional neural networks for beam tracking 8x
|
Tue Jun 25 22:14:10 2019, Milind, Update, Cameras, Convolutional neural networks for beam tracking
|
Thu Jun 27 20:48:22 2019, Milind, Update, Cameras, Convolutional neural networks for beam tracking   
|
Thu Jul 4 18:19:08 2019, Milind, Update, Cameras, Convolutional neural networks for beam tracking  
|
Mon Jul 8 17:52:30 2019, Milind, Update, Cameras, Convolutional neural networks for beam tracking
|
Tue Jul 9 22:13:26 2019, Milind, Update, Cameras, Convolutional neural networks for beam tracking 
|
Wed Jul 10 22:32:38 2019, Milind, Update, Cameras, Convolutional neural networks for beam tracking
|
Mon Jul 15 14:09:07 2019, Milind, Update, Cameras, CNN LSTM for beam tracking  
|
Fri Jul 19 16:47:06 2019, Milind, Update, Cameras, CNNs for beam tracking || Analysis of results 7x
|
Sat Jul 20 12:16:39 2019, gautam, Update, Cameras, CNNs for beam tracking || Analysis of results
|
Sat Jul 20 14:43:45 2019, Milind, Update, Cameras, CNNs for beam tracking || Analysis of results   
|
Wed Jul 24 20:05:47 2019, Milind, Update, Cameras, CNNs for beam tracking || Tales of desperation
|
Thu Jul 25 00:26:47 2019, Milind, Update, Cameras, Convolutional neural networks for beam tracking 
|
Mon Jun 17 22:19:04 2019, Milind, Update, Computer Scripts / Programs, PMC autolocker
|
Mon Jul 1 20:18:01 2019, Milind, Update, Computer Scripts / Programs, PMC autolocker
|
Tue Jul 2 12:30:44 2019, Milind, Update, Computer Scripts / Programs, PMC autolocker
|
Sun Jul 7 17:54:34 2019, Milind, Update, Computer Scripts / Programs, PMC autolocker
|
Tue Jun 25 23:52:37 2019, Milind, Update, Cameras, Simulation enhancements
|
Mon Jul 1 20:11:34 2019, Milind, Update, Cameras, Simulation enhancements
|
|
Message ID: 14746
Entry time: Wed Jul 10 22:32:38 2019
In reply to: 14741
|
Author: |
Milind |
Type: |
Update |
Category: |
Cameras |
Subject: |
Convolutional neural networks for beam tracking |
|
|
I trained a bunch (around 25 or so - to tune hyperparameters) of networks today. They were all CNNs. They all produced garbage. I also looked at lstm networks with CNN encoders (see this very useful link) and gave some thought to what kind of architecture we want to use and how to go about programming it (in Keras, will use tensorflow if I feel like I need more control). I will code it up tomorrow after some thought and discussion. I am not sure if abandoning CNNs is the right thing to do or if I should continue probing this with more architectures and tuning attempts. Any thoughts?
Right now, after speaking to Stuart (ldas_admin) I've decided on coding up the LSTM thing and then running that on one machine while probing the CNN thing on another.
Update on 10 July, 2019: I'm attaching all the results of training here in case anyone is interested in the future.
Quote: |
I received access today. After some incredible hassle, I was able to set up my repository and code on the remote system. Following this, Gautam wrote to Gabriele to ask him about which GPUs to use and if there was a previously set up environment I could directly use. Gabriele suggested that I use pcdev2 / pcdev3 / pcdev11 as they have good gpus. He also said that I could use source ~gabriele.vajente/virtualenv/bin/activate to use a virtualenv with tensorflow, numpy etc. preinstalled. However, I could not get that working, Therefore I created my own virtual environment with the necessary tensorflow, keras, scipy, numpy etc. libraries and suitable versions. On ssh-ing into the cluster, it can be activated using source /home/millind.vaddiraju/beamtrack/bin/activate. How do I know everything works? Well, I trained a network on it! With the new data. Attached (see attachment #1) is the prediction data for completely new test data. Yeah, its not great, but I got to observe the time it takes for the network to train for 50 epochs-
- On pcdev5 CPU: one epoch took ~1500s which is roughly 25 minutes (see Attachment #2). Gautam suggested that I try to train my networks on Optimus. I think this evidence should be sufficient to decide against that idea.
- On my GTX 1060: one epoch took ~30s. Which is 25 minutes (for 50 epochs) to train a network.
- On pcdev11 GPU (Titan X I think): each epoch took ~16s which is a far more reasonable time.
Therefore, I will carry out all training only on this machine from now.
Note to self:
Steps to repeat what you did are:
- ssh in to the cluster using ssh albert.einstein@ssh.ligo.org as described here.
- activate virtualenv as descirbed above
- navigate to code and run it.
Quote: |
I attempted to train a bunch of networks on the new data to test if the code was alright but realised quickly that, training on my local machine is not feasilble at all as training for 10 epochs took roughly 6 minutes. Therefore, I have placed a request for access to the cluster and am waiting for a reply. I will now set up a bunch of experiments to tune hyperparameters for this data and see what the results are.
|
|
|