40m QIL Cryo_Lab CTN SUS_Lab TCS_Lab OMC_Lab CRIME_Lab FEA ENG_Labs OptContFac Mariner WBEEShop
  40m Log, Page 132 of 339  Not logged in ELOG logo
ID Date Authorup Type Category Subject
  14669   Thu Jun 13 15:08:31 2019 MilindUpdateElectronicsVCO pickup by Rich

Rich dropped by at around 3:00 PM today and picked up the VCO in Attachment #1 and left the note in Attachment #2 on Gautam's desk with the promise of bringing it back soon.

Attachment 1: WhatsApp_Image_2019-06-13_at_15.06.57.jpeg
WhatsApp_Image_2019-06-13_at_15.06.57.jpeg
Attachment 2: WhatsApp_Image_2019-06-13_at_15.06.57(1).jpeg
WhatsApp_Image_2019-06-13_at_15.06.57(1).jpeg
  14671   Thu Jun 13 21:29:52 2019 MilindUpdateCamerasSteps to interact with GigE

As directed by Gautam, I have set up one script- interact.py (at /opt/rtcds/caltech/c1/scripts/GigE/SnapPy_pypylon/interact.py) to perform the following two tasks:

  1. View the GigE feed for a fixed period of time.
  2. Record the GigE feed for a fixed amount of time.

 

Steps to view GigE feed for a fixed amount of time:

  1. Run the following commands in the terminal to navigate to the concerned directory and then view the feed
    1. cd /opt/rtcds/caltech/c1/scripts/GigE/SnapPy_pypylon
    2. python interact.py --path_to_config <path_config> --mode 0 --view_time <viewing_time>, where
      1. path_config: full path to configuration file, defaults to /opt/rtcds/caltech/c1/scripts/GigE/SnapPy_pypylon/C1-CAM-ETMX.ini if --path_to_config is not used
      2. viewing_time: time in seconds for which the feed is to be displayed. The server is closed  after this time and the window freezes and can be manually closed.
    3. Exiting the feed in between: The script terminates automatically after the specified time. To terminate the feed in between, close the window manually using the x icon the top right. This makes sure that the server is correctly closed. If closed using the Ctrl-C command in the terminal, the server is left running and any attempt to unwittingly set up another results in an error (see Attachment #1). In this case, the server and client processes needs to be identified manually and killed. I have used the following steps
      1. ps -eaf | grep server, then identify the PID for the python camera_server.py process
      2. kill PID
      3. similarly for the client file

Steps to record the GigE feed for a fixed amount of time:

I tried to look for elegant solutions that wouldn't require editing the code that Jon wrote and stumbled upon this useful bit of information but ended up deciding that it was just easier to change the camera_client_movie.py (/opt/rtcds/caltech/c1/scripts/GigE/SnapPy_pypylon/camera_client_movie.py). It can still be run as previously described, where video recording is terminated by using Ctrl-C. Steps to record for a fixed period of time are

  1. cd /opt/rtcds/caltech/c1/scripts/GigE/SnapPy_pypylon
  2. python interact.py --path_to_config <path_config> --mode 1 --save_time <recording_time> --file_name filename, where\
    1. path_config: full path to configuration file, defaults to /opt/rtcds/caltech/c1/scripts/GigE/SnapPy_pypylon/C1-CAM-ETMX.ini if --path_to_config is not used
    2. recording_time: time in seconds for which the feed is to be saved. No video is displayed during this time.
    3. filename: full path to the file where the video is to be saved. Overwrites any existing files.

I'll make aliases for these to make the whole process more user friendly. I'm halting this for now and will discuss what else needs to be done once Gautam gets back.


Regarding the autolocker: I spoke to Aaron today and as he is in tomorrow, I'll ask him about the burt files and the ideal configuration.

I'm also starting with GANs now.

 

 

Attachment 1: terminal_error_server.pdf
terminal_error_server.pdf
  14678   Mon Jun 17 14:36:13 2019 MilindUpdateCamerasConvolutional neural networks for beam tracking

Begun setting up an environment (as mentioned before, on my local machine) and scripts to run experiments with Convolutional networks for beam tracking. All code has been pushed to this folder in the GigEcamera repository. I am presently looking for pre-processing techniques for the video which go beyond the usual "Crop the images! Normalize pixel values! Convert to Grayscale!".

Quote:

Networks for beam tracking:

  1. I will use the architectures suggested in this work with a few modifications. I will use MSE loss function, Adam optimizer and my local GPU for training.

 

  14680   Mon Jun 17 22:19:04 2019 MilindUpdateComputer Scripts / ProgramsPMC autolocker

As Rana asked me to in the last meeting, I dug through the elogs to determine what had become of the previous autolockers. I stumbled upon this elog by Rana from before Gautam cleaned up the medm screen. Out of curiosity, I ran the autolocker script using the instructions in Rana's elog. I did this a total of 5 times and could lock the PMC 3 times fairly quickly. I attempted to decipher the details of the code but did not make much headway owing to my unfamiliarity with the language. From what I could make out from the medm screen while the autolocker was running, it appeared to be the same method as that in this elog. I will take a look at it again tomorrow. However, I intend to spend most of tomorrow working on preprocessing the data, developing the CNN script and then the simulation. 

Quote:
 
  1.  I shall also begin working on a script to autolock the PMC based on what Rana showed me on Monday. I will also take a look at the the contents of this elog and try to pick up from there. I hope to make significant progress by the next lab meeting.

  14682   Tue Jun 18 22:54:59 2019 MilindUpdateCamerasConvolutional neural networks for beam tracking

Worked further on this. I skimmed through a few resources to look for details of what pre-processing can be done. Here (am planning to convert all these resources, particularly those I come across for GANs into either a README on the repo or a Wiki soon) are some of the useful things I found during today's reading. The work I skimmed through today mostly pointed to the use of a median filter for pre-processing, if any is to be done. I am presently using the Sequential() API in Keras to set up the neural network. I will train it tomorrow.

Quote:

Begun setting up an environment (as mentioned before, on my local machine) and scripts to run experiments with Convolutional networks for beam tracking. All code has been pushed to this folder in the GigEcamera repository. I am presently looking for pre-processing techniques for the video which go beyond the usual "Crop the images! Normalize pixel values! Convert to Grayscale!".

 

  14694   Tue Jun 25 00:25:47 2019 MilindUpdateCamerasConvolutional neural networks for beam tracking

In the previous meeting, Koji pointed out (once again) that I should determine if the displacement values and frames are synchronized before training a network. Pooja did the following last time. Koji also suggested that I first predict the motion (a series of x and y coordintates) and then slide resulting plots around until I get the best match for the original motion. This is however not possible with a neural network based approach as the network learns exactly what you show it and therefore it will learn any mismatch between the labels and the frames and predict exactly that. Therefore I came up with what Koji described as "hacky" method to achieve the same using the opencv work described previously in this elog (the only addition being the application of a mask to block out the OSEMs and work only with the beam spot) .

Hacky technique to sync frames and labels:

  1. I ran the OpenCV algorithm on the data to obtain a plot for predicted motion depicted in Attachment #2. As is evident, the predicted motion is only an approximate of the actual motion and also displays a shift . However, a plot of the fourier transform of the signal (see Attachment #1) shows that the components present are the same. However, the predominant frequency component is 0.22 Hz rather than 0.2 Hz as stated by Pooja in her elog. I wonder if this is of any consequence. Therefore, this predicted motion can be slid around until it overlaps with the applied sinusoidal dither signal "well".
  2. Defining "well": I computed an error signal as the differnece between the predicted signal and the actual motion with each signal being normalized by subtracting the mean and then dividing the resulting signal by the maximum value (see Attachment #3). The lower the power of the resulting signal, the better the synchronization of the predicted and actual signal. Note: To achieve this overlap of signals, datapoints are removed from either the start or end of the signals and this effectively reduces the number of data points available for training by 36 pionts (see Attachment #4, positive and negative shifts merely indicate if the predicted signal is being moved right or left).
  3. Attachments #5,#6 show the resutls of shifting the data by 36 samples. it is evident that there is far greater overlap of the prediction and the actual values.
  4. Well, what now? I will use the mapping between labels and frames obtained by the above steps to train a neural network.

[Koji, Milind - 21/06/2019]

  1. Well, the above is fine, but why is contour detection really necessary? Why not take a weighted sum of all the pixel values (in a rectangular region obtained, say, after blocking out the OSEMs) to see what the centroid motion is? Black areas (0 pixel intensity values) will not contribute to this sum anyway. Perhaps that can be used for the sliding instead of the above (fallible!) approach, specially for cases in which the beam "spot"  is just a collection of random speckles?
    1. Something like this was done by Pooja where she computed the sum of pixel intensities in a rectangular region containing the beam spot. However, she did this for very noisy data and observed intensity variation at a frequency double that of the applied signal.
    2. Results of applying a median filter and doing the same are presented in Attachment #7. Clearly, they can't be used for this sliding task.
    3. Results of computing the weighted sum of all the coordinates (with pixel intensities as the weights are presented in Attachment #8. Clearly, for this data and for this task, the contour approach seems to be a better method. Further, these resutls just serve to prove Rana's point that such simple, unsophisticated, naive approaches will not produce desired results and therefore, shall be presented in this very context in the report that is due.
  2. The contour detection technique does not work if the beam spot is just a cokllection of speckles. In that case Koji suggested that we use a bounding convex hull instead of a contour. Alternately, for a bunch of speckles I can perform dilation to reduce it to the same problem.
  3. Using gpstime for time stamping: To determine the absolute time which a frame is grabbed. However, the time between the time being recorded and grabbing of frame needs to be determined for this which should be doable using linux/python commands.
Quote:

Worked further on this. I skimmed through a few resources to look for details of what pre-processing can be done. Here (am planning to convert all these resources, particularly those I come across for GANs into either a README on the repo or a Wiki soon) are some of the useful things I found during today's reading. The work I skimmed through today mostly pointed to the use of a median filter for pre-processing, if any is to be done. I am presently using the Sequential() API in Keras to set up the neural network. I will train it tomorrow.

 


Upcoming work (in the order of priority):

  1. Data acquisition: With the mode cleaner being locked and Kruthi having focused on to the beam spot, I will obtain data for training both GANs and the convolutional networks. I really hope that some of the work done above can be extended to the new data. Rana suggested that I automate this by writing a script which I will do after a discussion with Gautam tomorrow.
  2. Network training for beam spot tracking: I will begin training the convolutional network with the data pre-processed as described above. I will also simultaneously prepare data acquired from the GigE and train networks on that. Note: I planned to experiment with framewize predictions and hence did some of the work described above. However, I will restrict the number of experiments on that and perform more of those that use 3D convolution. Rana also pointed out that it would be interesting to have the network output uncertainity in the predictions. I am not sure how this can be done, but I will look into it.
  3. Simulation:
    1. Putting the physics in: Previously, I worked on adding point scatterers. I shall add the effect of surface roughness and incorporate the BRDF next. Just as Gautam did, Rana also reccommended that I go through Hiro Yamamoto's work to improve my understanding of this.
    2. GANs: I will put together a readme (which I will turn into a wiki later) for all the material that I am using to develop my ideas about GAN training. Currently, my understanding of GANs is that they take as input noise vectors which are fed to the generative networks which then produce the fakes. This clearly isn't the only way to do it as GANs are used for several applications such as image generation from text. I am referring to these papers to set up the necessary architecture.
  4. PMC autolocker: I will convert the existing autolocker script to python. Rana also suggested that it would be interesting to see what the best settings of the hyperparameters would be to lock the PMC the fastest. I will write a script to do that and plot a 3D surface plot of the average time taken to lock the PMC as a function of the PZT scan speed and the Servo gain to determine the optimal setting of these "hyperparameters".
  5. Cleaning up/ formalizing code: Rana pointed out that any code that messes with channel values must return them to the original settings once the script is finished running. I have overlooked this and will add code to do this to all the files I have created thus far. Further, while most of my code is well documented and frequently pushed to Github, I will make sure to push any code that I might have missed to github.
  6. Talk to Jon!: Gautam suggested that I speak to Jon about the machine requirements for setting up a dedicated machine for running the camera server and about connecting the GigE to a monitor now that we have a feed. Koji also suggested that I talk to him about somehow figuring out the hardware to ensure that the GigE clock is the same as the rest of the system.

 

Attachment 1: Spectra.pdf
Spectra.pdf
Attachment 2: normalised_comparison_y.pdf
normalised_comparison_y.pdf
Attachment 3: residue_normalised_y.pdf
residue_normalised_y.pdf
Attachment 4: error_power_sliding.pdf
error_power_sliding.pdf
Attachment 5: normalised_comparison_y.pdf
normalised_comparison_y.pdf
Attachment 6: residue_normalised_y.pdf
residue_normalised_y.pdf
Attachment 7: intensum.pdf
intensum.pdf
Attachment 8: centroid.pdf
centroid.pdf
  14697   Tue Jun 25 22:14:10 2019 MilindUpdateCamerasConvolutional neural networks for beam tracking

I discussed this with Gautam and he asked me to come up with a list of signals that I would need for my use and then design the data acquisition task at a high level before proceeding. I'm working on that right now. We came up with a very elementary sketch of what the script will do-

  1. Check the MC is locked.
  2. Choose an exposure value.
  3. Choose a frequency and amplitude value for the applied sinusoidal dither (check warning by Gabriele below).
  4. Apply sinusoidal dither to optic.
  5. Timestamping: Record gpstime, instantaneous channel values and a frame. These frames can later be put together in a sequence and a network can be trained on this. (NEED TO COME UP WITH SOMETHING CLEVERER THAN THIS!)

Tomorrow I will try and prepare a dummy script for this before the meeting at noon. Gautam asked me to familiarize myself with the awg, cdsutils (I have already used ezca before) to write the script. This will also help me do the following two tasks-

  1. IFO test scripts that Rana asked me to work on a while ago
  2. The PMC autolocker scripts that Rana asked me work on
Quote:
 

Upcoming work (in the order of priority):

  1. Data acquisition: With the mode cleaner being locked and Kruthi having focused on to the beam spot, I will obtain data for training both GANs and the convolutional networks. I really hope that some of the work done above can be extended to the new data. Rana suggested that I automate this by writing a script which I will do after a discussion with Gautam tomorrow.

 

I got to speak to Gabriele about the project today and he suggested that if I am using Rana's memory based approach, then I had better be careful to ensure that the network does not falsely learn to predict a sinusoid at all points in time and that if I use the frame wise approach I try to somehow incorporate the fact that certain magnitudes and frequencies of motion are simply not physically possible. Something that Rana and Gautam emphasized as well.

I am pushing the code that I wrote for

  1. Kruthi's exposure variation - ccd calibration experiment
  2. modified camera_client_movie.py code (currently at /opt/rtcds/caltech/c1/scripts/GigE/SnapPy_pypylon)
  3. interact.py (to interact with the GigE in viewing or recording mode) (currently at /opt/rtcds/caltech/c1/scripts/GigE/SnapPy_pypylon)

to the GigEcamera repository.

 

Gautam also asked me to look at Jigyasa's report and elog 13443 to come up with the specs of a machine that would accomodate a dedicated camera server.

 

Quote:
 
  1. Network training for beam spot tracking: I will begin training the convolutional network with the data pre-processed as described above. I will also simultaneously prepare data acquired from the GigE and train networks on that. Note: I planned to experiment with framewize predictions and hence did some of the work described above. However, I will restrict the number of experiments on that and perform more of those that use 3D convolution. Rana also pointed out that it would be interesting to have the network output uncertainity in the predictions. I am not sure how this can be done, but I will look into it.
  2. Cleaning up/ formalizing code: Rana pointed out that any code that messes with channel values must return them to the original settings once the script is finished running. I have overlooked this and will add code to do this to all the files I have created thus far. Further, while most of my code is well documented and frequently pushed to Github, I will make sure to push any code that I might have missed to github.
  3. Talk to Jon!: Gautam suggested that I speak to Jon about the machine requirements for setting up a dedicated machine for running the camera server and about connecting the GigE to a monitor now that we have a feed. Koji also suggested that I talk to him about somehow figuring out the hardware to ensure that the GigE clock is the same as the rest of the system.

 

 

  14698   Tue Jun 25 23:52:37 2019 MilindUpdateCamerasSimulation enhancements

Yesterday, Rana asked me to look at Hiro Yamamoto's docs on the DCC to improve the simulation. I'm performing a first pass (=> Just skimming through to see if they're relevant, I will go through them more carefully soon!) and putting up stuff here for future reference. @Kruthi's help much appreciated!

  14700   Wed Jun 26 11:11:40 2019 MilindUpdateIOOPMC and IMC locked again, some MEDM maintenance

After helping Aaron key the crate and do a burt restore, I realized that it would probably be best to record the steps that Koji showed me to do a burt restore as a reference for (anyone) the future

Commands (in terminal):

  1. burttoday: changes to the directory with snapshots for the day (/opt/rtcds/caltech/c1/burt/autoburt/today)
  2. burtgooey: opens a new window with several buttons of which "Restore" needs to be selected. This opens up a second window as shown in Attachment #1. Click on Snapshot files and navigate to the snapshot you wish to restore (these are present at /opt/rtcds/caltech/c1/burt/autoburt/snapshots) and select that. A green "OK" button indicates if the Restore can be performed without a hitch. Hit "Restore" to perform the burtrestore.

 

Also Gautam explained today that the sticky slider problem is a hardware issue. That it basically means that the signal (voltage output for instance) that you request from the medm screen is not what the hardware delivers. Twice now, we have got around that with a burtrestore. My understanding of a burt restore is that it is a restoration of values from a certain time to the EPICS channels. Therefore, I don't understand why a restoration at the software level should fix how the hardware responds? Why does this happen?

Attachment 1: burtgooey.pdf
burtgooey.pdf
  14706   Thu Jun 27 20:48:22 2019 MilindUpdateCamerasConvolutional neural networks for beam tracking

And finally, a network is trained!

Result summary (TLDR :-P) : No memory was used. Model trained. Results were garbage. Will tune hyperparameters now. Code pushed to github.

 

More details of the experiment:

Aim:

  1. To train a network to check that training occurs and get a feel for what the learning might be like.
  2. To set up the necessary framework to perform mulitple experiments and record results in a manner facilitating comparison.
  3. To track beam spot motion.

What I did:

  1. Set up a network that learns a framewise mapping as described in here.
  2. Training data: 0.9 x 1791 frames. Validation data: 0.1 x 1791 frames. Test data (only prediction): all the 1791 frames
  3. Hyperparameters: Attachment #1
  4. Did no tuning of hyperparameters.
  5. Compiled and fit the models and saved the results.

 

What I saw

  1. Attachment #2: data fed to the network after pre-processing - median blur + crop
  2. Attachment #3: learning curves.
  3. Attachment #4: true and predicted motion. Nothing great.

What I think is going wrong-

  1. No hyperparameter tuning. This was only a first pass but is being reported as it will form the basis of all future experiments.
  2. Too little data.
  3. Maybe wrong architecture.

Well, what now?

  1. Tune hyperparmeters (try to get the network to overfit on the data and then test on that. We'll then know for sure that all we probably need is more data?)
  2. Currently the network has around 200k parameters. Maybe reduce that.
  3. Set up a network that takes as input (one example corresponding to one forward pass)  a bunch of frames and predicts a vector of position values that can be used as continuous data).
Quote:

I got to speak to Gabriele about the project today and he suggested that if I am using Rana's memory based approach, then I had better be careful to ensure that the network does not falsely learn to predict a sinusoid at all points in time and that if I use the frame wise approach I try to somehow incorporate the fact that certain magnitudes and frequencies of motion are simply not physically possible. Something that Rana and Gautam emphasized as well.

 
Quote:
 
  1. Network training for beam spot tracking: I will begin training the convolutional network with the data pre-processed as described above. I will also simultaneously prepare data acquired from the GigE and train networks on that. Note: I planned to experiment with framewize predictions and hence did some of the work described above. However, I will restrict the number of experiments on that and perform more of those that use 3D convolution. Rana also pointed out that it would be interesting to have the network output uncertainity in the predictions. I am not sure how this can be done, but I will look into it.

 

 

Attachment 1: readme.txt
Experiment file: train.py
batch_size: 32
dropout_probability: 0.8
eta: 0.0001
filter_size: 19
filter_type: median
initializer: xavier
num_epochs: 50
activation_function: relu
dense_layer_units: 64
... 10 more lines ...
Attachment 2: frame0.pdf
frame0.pdf
Attachment 3: Learning_curves.png
Learning_curves.png
Attachment 4: Motion.png
Motion.png
  14710   Sun Jun 30 22:02:26 2019 MilindUpdateCamerasKeyed c1aux crate

I wanted to try out the unstick.py script on c1aux but kept running into timeout errors. I was also confronted by a blank GigE screen. Further, couldn't telnet into c1aux using telnet c1aux as described here. Therefore, I went in and keyed the c1aux crate (1Y1).

  14711   Sun Jun 30 22:21:07 2019 MilindUpdateIOOPMC and IMC locked again, some MEDM maintenance

Wrote the script. It currently lives at /users/milind/NonlinearControl/milind/unstick/unstick.py. Also pushed it to the repo here. It does the following:

  1. When run as python unstick.py c1aux (for instance) from the terminal, it parses the autoBurt.req file at /cvs/cds/caltech/target/c1aux/autoBurt.req and obtains the channels.
  2. Iterates through the channels and changes it to "some value"
    1. For channels corresponding to buttons on MEDM screen (Enable/Disable etc.) toggles between 0 and 1
    2. For channels corresponding to continuous values (such as say exposure time or the like) changes to abs(1+current_value)
  3. Resets to original value and then moves to the next channel

I tried print statements instead of actually writing to the channels as Gautam asked me to do that with supervision. Is this good enough?

Quote:

a useful piece of code that we should ask one of this summer's SURFs to write:

  1. read in a BURT .req file which is usually used to make the autosnap / restore.
  2. change ALL of the values to some value (slightly different from its current value)
  3. restore it to its current value

this will solve the sticky slider problem and do it in a systematic way. We can run it from the command line: e.g. 'unsticky.py c1psl c1ioo c1lsc'

Quote:

Aaron complained to me earlier that the PMC could not be locked. Turned out to be a classic sticky slider problem,

  14713   Mon Jul 1 11:02:05 2019 MilindUpdateIOOPMC and IMC locked again, some MEDM maintenance

Made changes as discussed in this issue. Further, I need to add signal handling capabilities to the code. I belive Jon has pointed me to some code. I will take a look at that ASAP.

Quote:

> For channels corresponding to continuous values (such as say exposure time or the like) changes to abs(1+current_value)

Why abs? Is the current_value is like -5.4321 (for example for the alignment slider), this returns +4.4321 and the suspension will suffer from huge motion (well it will be returned to the original value soon though). 

  14714   Mon Jul 1 20:11:34 2019 MilindUpdateCamerasSimulation enhancements

Today, I read a lot more about BRDF and modelling but could not make much headway regarding the implementation in the simulation. I've stopped for now and I'll take a crack at it tomorrow again.

Quote:

Yesterday, Rana asked me to look at Hiro Yamamoto's docs on the DCC to improve the simulation. I'm performing a first pass (=> Just skimming through to see if they're relevant, I will go through them more carefully soon!) and putting up stuff here for future reference. @Kruthi's help much appreciated!

  14715   Mon Jul 1 20:18:01 2019 MilindUpdateComputer Scripts / ProgramsPMC autolocker

I've begun working on this. Steps to complete:

  1. Convert the autolocker to python. Test that it works.
  2. Run the script with different settings of the servo gain adjust and DC output adjust parameters and obtain a plot of the average time of lock to determine what the best settings of the aforementioned parameters are.
Quote:

As Rana asked me to in the last meeting, I dug through the elogs to determine what had become of the previous autolockers. I stumbled upon this elog by Rana from before Gautam cleaned up the medm screen. Out of curiosity, I ran the autolocker script using the instructions in Rana's elog. I did this a total of 5 times and could lock the PMC 3 times fairly quickly. I attempted to decipher the details of the code but did not make much headway owing to my unfamiliarity with the language. From what I could make out from the medm screen while the autolocker was running, it appeared to be the same method as that in this elog. I will take a look at it again tomorrow. However, I intend to spend most of tomorrow working on preprocessing the data, developing the CNN script and then the simulation. 

Quote:
 
  1.  I shall also begin working on a script to autolock the PMC based on what Rana showed me on Monday. I will also take a look at the the contents of this elog and try to pick up from there. I hope to make significant progress by the next lab meeting.
  14717   Tue Jul 2 12:30:44 2019 MilindUpdateComputer Scripts / ProgramsPMC autolocker

Just finished a raw version of the autolocker!! Tested it once and was able to achieve lock! This is a python version of the code at /opt/rtcds/caltech/c1/scripts/PSL/PMC/AutoLock.sh.

The current code lives in my users directory. Gautam asked me to put the completed autolocker at /opt/rtcds/caltech/c1/scripts/PSL/PMC/ and that I needn't necessarily put it on git. However, I had previously added it to my Non-linear control repo. Not sure if I should take it off? The current script still lacks some checks like those that enable it to stop after a certain time of attempting to lock or those that handle interrupt signals. Will do that in some time.

P.S. As Koji says, Victory! :-P

P.P.S. Rana pointed out that this is not the objective and what we actually wanna do is run a search over the parameter space of the locking process. I will document my ideas about this process as soon as I do a little more reading. He also said that it would not do to have command line arguments as the main source from which parameters are procured and that .yml files ought to be used instead. I will make that change asap.

 

Quote:

I've begun working on this. Steps to complete:

  1. Convert the autolocker to python. Test that it works.
  2. Run the script with different settings of the servo gain adjust and DC output adjust parameters and obtain a plot of the average time of lock to determine what the best settings of the aforementioned parameters are.
Quote:

As Rana asked me to in the last meeting, I dug through the elogs to determine what had become of the previous autolockers. I stumbled upon this elog by Rana from before Gautam cleaned up the medm screen. Out of curiosity, I ran the autolocker script using the instructions in Rana's elog. I did this a total of 5 times and could lock the PMC 3 times fairly quickly. I attempted to decipher the details of the code but did not make much headway owing to my unfamiliarity with the language. From what I could make out from the medm screen while the autolocker was running, it appeared to be the same method as that in this elog. I will take a look at it again tomorrow. However, I intend to spend most of tomorrow working on preprocessing the data, developing the CNN script and then the simulation. 

Quote:
 
  1.  I shall also begin working on a script to autolock the PMC based on what Rana showed me on Monday. I will also take a look at the the contents of this elog and try to pick up from there. I hope to make significant progress by the next lab meeting.
  14723   Wed Jul 3 23:53:38 2019 MilindUpdateCamerasdata for nns

Tried collecting data today. Was unable to keep the camera_server code running for any length of time as it threw segfaults. Will take a shot again tomorrow.

Quote:

The GigE is focused now (judged by eye) and I have closed the lid. I'm attaching a picture of the MC2 beam spot, captured using GigE at an exposure time of 400µs.

What was the solution to resolving the flaky video streaming during the alignment process????

-> I think, the issue was with either the poor wireless network conection or the GigE-PoE ethernet cable.

Quote:

Turns out, focusing the GigE is actually a bit tricky. With pylon, everytime I change the exposure or the focus, I'm running into the error I had mentioned earlier in one of my elogs; so I tried using the python scripts to interact with the GigE. But whenever I try to change the focal plane distance by rotating the lens coupler, the ethernet cable connection becomes loose and the camera server needs to be relaunched every now and then. Also, everytime we want to change the distance between the lenses, the telescope needs to be dismantled and refocused again. I'll try to come up with a better telescope design for this.

Yesterday, I had focused the GigE using a low exposure time and small aperture of iris, to make sure that we are actually seeing a sharp image of the beam spot. I'm attaching a picture of the beam spot I had clicked while focusing it, unfortunately, I forgot to take a picture after I had focused it completely. I'm also attaching a picture of the final setup for future reference. 


Yesterday night, Rana asked me to lock the MC2. I figured that the PSL shutter was closed; I just opened it and was able to see the beam spot on the analog camera screen.

  14724   Thu Jul 4 10:47:37 2019 MilindUpdateGeneralEarthquake now

There was a magnitude 6.6 earthquake just a few minutes ago. I am attaching photographs of the monitor feeds for reference here. Is there a standard protocol to be followed in this situation? I'm looking through the wiki now.

Further, the IMC seems to be misaligned and is not locking! cryingcrying As Koji has let me know, I really hope this is not too serious and can be fixed easily.

Attachment 1: after_earthquake2.jpg
after_earthquake2.jpg
Attachment 2: after_earthquake.jpg
after_earthquake.jpg
  14726   Thu Jul 4 18:19:08 2019 MilindUpdateCamerasConvolutional neural networks for beam tracking

The quoted elog has figures which indicate that the network did not learn (train or generalize) on the used data. This is a scary thing as (in my experience) it indicates that something is fundamentally wrong with either the data or model and learning will not happen despite how hyperparameters are tuned. To check this, I ran the training experiment for nearly 25 hyperparameter settings (results here)with the old data and was able to successfully overfit the data. Why is this progress? Well, we know that we are on the right track and the task is to reduce overfitting. Whether, that will happen through more hyperparameter tuning, data collection or augmentation remains to be seen. See attachments for more details. 

Why is the fit so perfect at the start and bad later? Well, that's because the first 90% of the test data is  the training data I overfit to and the latter the validation data that the network has not generalized well to.

Quote:

And finally, a network is trained!

Result summary (TLDR :-P) : No memory was used. Model trained. Results were garbage. Will tune hyperparameters now. Code pushed to github.

 

More details of the experiment:

Aim:

  1. To train a network to check that training occurs and get a feel for what the learning might be like.
  2. To set up the necessary framework to perform mulitple experiments and record results in a manner facilitating comparison.
  3. To track beam spot motion.

What I did:

  1. Set up a network that learns a framewise mapping as described in here.
  2. Training data: 0.9 x 1791 frames. Validation data: 0.1 x 1791 frames. Test data (only prediction): all the 1791 frames
  3. Hyperparameters: Attachment #1
  4. Did no tuning of hyperparameters.
  5. Compiled and fit the models and saved the results.

 

What I saw

  1. Attachment #2: data fed to the network after pre-processing - median blur + crop
  2. Attachment #3: learning curves.
  3. Attachment #4: true and predicted motion. Nothing great.

What I think is going wrong-

  1. No hyperparameter tuning. This was only a first pass but is being reported as it will form the basis of all future experiments.
  2. Too little data.
  3. Maybe wrong architecture.

Well, what now?

  1. Tune hyperparmeters (try to get the network to overfit on the data and then test on that. We'll then know for sure that all we probably need is more data?)
  2. Currently the network has around 200k parameters. Maybe reduce that.
  3. Set up a network that takes as input (one example corresponding to one forward pass)  a bunch of frames and predicts a vector of position values that can be used as continuous data).
Quote:

I got to speak to Gabriele about the project today and he suggested that if I am using Rana's memory based approach, then I had better be careful to ensure that the network does not falsely learn to predict a sinusoid at all points in time and that if I use the frame wise approach I try to somehow incorporate the fact that certain magnitudes and frequencies of motion are simply not physically possible. Something that Rana and Gautam emphasized as well.

Quote:
 
  1. Network training for beam spot tracking: I will begin training the convolutional network with the data pre-processed as described above. I will also simultaneously prepare data acquired from the GigE and train networks on that. Note: I planned to experiment with framewize predictions and hence did some of the work described above. However, I will restrict the number of experiments on that and perform more of those that use 3D convolution. Rana also pointed out that it would be interesting to have the network output uncertainity in the predictions. I am not sure how this can be done, but I will look into it.
Attachment 1: Motion.pdf
Motion.pdf
Attachment 2: Error.pdf
Error.pdf
Attachment 3: Learning_curves.pdf
Learning_curves.pdf
  14731   Sun Jul 7 17:54:34 2019 MilindUpdateComputer Scripts / ProgramsPMC autolocker

I modified the autolocker code I wrote to read from a .yaml configuration file instead of commandline arguements (that option still exists if one wishes to override what the .yaml file contains). I have pushed the code to github. I started reading about MCMC and will put up details of the remaining part of the work ASAP.

Quote:
 

P.P.S.  He also said that it would not do to have command line arguments as the main source from which parameters are procured and that .yml files ought to be used instead. I will make that change asap.

  14734   Mon Jul 8 17:52:30 2019 MilindUpdateCamerasConvolutional neural networks for beam tracking

After the two earthquakes, I collected some data by dithering the optic and recording the QPD readings. Today, I set up scripts to process the data and then train networks on this data. I have pushed all the code to github. I attempted to train a bunch of networks on the new data to test if the code was alright but realised quickly that, training on my local machine is not feasilble at all as training for 10 epochs took roughly 6 minutes. Therefore, I have placed a request for access to the cluster and am waiting for a reply. I will now set up a bunch of experiments to tune hyperparameters for this data and see what the results are.

Trainng networks with memory

I set up a network to handle input volumes (stacks of frames) instead of individual frames. It still uses 2D convolution and not 3D convolution. I am currently training on the new data. However, I was curious to see if it would provide any improved performance over the results I put up in the previous elog. After a bit of hyperparameter tuning, I did get some decent results which I have attached below. However, this is for Pooja's old data which makes them, ah, not so relevant. Also, this testing isn't truly representative because the test data isn't entirely new to the network. I am going to train this network on the new data now with the following objectives (in the following steps):

  1. Train on data recorded at one frequency, generalize/ test on unseen data of the same frequency, large amplitude of motion
  2. Train on data recorded at one frequency, generallize/ test on unseen data of a different frequency, large amplitude of motion
  3. Train on data recorded at one frequency, generalize/ test on unseen data of  same/ different frequency, small amplitude of motion
  4. Train on data at different frequencies and generalize/ test on data with a mixture of frequencies at small amplitudes - Gautam pointed out that the network would truly be superb (good?) if we can just predict the QPD output from the video of the beam spot when nothing is being shaken.

I hope this looks alright? Rana also suggested I try LSTMs today. I'll maybe code it up tomorrow. What I have in mind- A conv layer encoder, flatten, followed by an LSTM layer (why not plain RNNs? well LSTMs handle vanishing gradients, so why the hassle).

Quote:

The quoted elog has figures which indicate that the network did not learn (train or generalize) on the used data. This is a scary thing as (in my experience) it indicates that something is fundamentally wrong with either the data or model and learning will not happen despite how hyperparameters are tuned. To check this, I ran the training experiment for nearly 25 hyperparameter settings (results here)with the old data and was able to successfully overfit the data. Why is this progress? Well, we know that we are on the right track and the task is to reduce overfitting. Whether, that will happen through more hyperparameter tuning, data collection or augmentation remains to be seen. See attachments for more details. 

Why is the fit so perfect at the start and bad later? Well, that's because the first 90% of the test data is  the training data I overfit to and the latter the validation data that the network has not generalized well to.

Attachment 1: Motion.pdf
Motion.pdf
  14737   Tue Jul 9 10:37:42 2019 MilindUpdateIOOkeyed psl crate, unstick.py, pmc autolocker code- working

Today, Gautam keyed the C1PSL crate and we got to test my unstick.py code. It seems to be working fine. Remarks:

  1. Gautam moved the unstick.py code to /opt/rtcds/caltech/c1/scripts/cds. Therefore, the steps to run this code are now:
    1. cd /opt/rtcds/caltech/c1/scripts/cds
    2. python unstick.py c1psl (for the c1psl machine)
  2. There is now a sleepTime global variable in the code which defines the amount of delay between successive channel toggles. We set this to 1ms and it took the code around 3s to run.
  3. Gautam was curious to see if this would work even if we set the sleepTime parameter to 0 but decided that that could be tested the next time something was keyed.
  4. I still need to add the signal handling thing to this code.

Following this, we tested my PMC autolocker code. The code ran for about a minute before achieveing lock. Remarks:

  1. Gautam moved my code (pmc_autolocker.py and autolocker_config.yaml) to /cvs/cds/rtcds/caltech/c1/scripts/PSL/PMC/ . Therefore, the steps to run this code are now:
    1. cd /cvs/cds/rtcds/caltech/c1/scripts/PSL/PMC/
    2. python pmc_autolocker.py (check code or use --help to see what the command line arguments do which is only for when you wanna override the details in the .yaml file)
  2. Gautam suggested that I add some delay between succesive steps of DC output adjust so that it locks quickly. I'll do that ASAP. For now, it works.
  14741   Tue Jul 9 22:13:26 2019 MilindUpdateCamerasConvolutional neural networks for beam tracking

I received access today. After some incredible hassle, I was able to set up my repository and code on the remote system. Following this, Gautam wrote to Gabriele to ask him about which GPUs to use and if there was a previously set up environment I could directly use. Gabriele suggested that I use pcdev2 / pcdev3 / pcdev11 as they have good gpus. He also said that I could use source ~gabriele.vajente/virtualenv/bin/activate to use a virtualenv with tensorflow, numpy etc. preinstalled. However, I could not get that working, Therefore I created my own virtual environment with the necessary tensorflow, keras, scipy, numpy etc. libraries and suitable versions. On ssh-ing into the cluster, it can be activated using source /home/millind.vaddiraju/beamtrack/bin/activate. How do I know everything works? Well, I trained a network on it! With the new data. Attached (see attachment #1) is the prediction data for completely new test data. Yeah, its not great, but I got to observe the time it takes for the network to train for 50 epochs-

  1. On pcdev5 CPU: one epoch took ~1500s which is roughly 25 minutes (see Attachment #2). Gautam suggested that I try to train my networks on Optimus. I think this evidence should be sufficient to decide against that idea.
  2. On my GTX 1060: one epoch took ~30s. Which is 25 minutes (for 50 epochs) to train a network.
  3. On pcdev11 GPU (Titan X I think): each epoch took ~16s which is a far more reasonable time.

Therefore, I will carry out all training only on this machine from now.

 


Note to self:

Steps to repeat what you did are:

  1. ssh in to the cluster using ssh albert.einstein@ssh.ligo.org as described here.
  2. activate virtualenv as descirbed above
  3. navigate to code  and run it.
Quote:

 I attempted to train a bunch of networks on the new data to test if the code was alright but realised quickly that, training on my local machine is not feasilble at all as training for 10 epochs took roughly 6 minutes. Therefore, I have placed a request for access to the cluster and am waiting for a reply. I will now set up a bunch of experiments to tune hyperparameters for this data and see what the results are.

Attachment 1: predicted_motion_first.pdf
predicted_motion_first.pdf
Attachment 2: pcdev5_time.png
pcdev5_time.png
  14746   Wed Jul 10 22:32:38 2019 MilindUpdateCamerasConvolutional neural networks for beam tracking

I trained a bunch (around 25 or so - to tune hyperparameters) of networks today. They were all CNNs. They all produced garbage. I also looked at lstm networks with CNN encoders (see this very useful link) and gave some thought to what kind of architecture we want to use and how to go about programming it (in Keras, will use tensorflow if I feel like I need more control). I will code it up tomorrow after some thought and discussion. I am not sure if abandoning CNNs is the right thing to do or if I should continue probing this with more architectures and tuning attempts. Any thoughts?

Right now, after speaking to Stuart (ldas_admin) I've decided on coding up the LSTM thing and then running that on one machine while probing the CNN thing on another.

 


Update on 10 July, 2019: I'm attaching all the results of training here in case anyone is interested in the future.

Quote:

I received access today. After some incredible hassle, I was able to set up my repository and code on the remote system. Following this, Gautam wrote to Gabriele to ask him about which GPUs to use and if there was a previously set up environment I could directly use. Gabriele suggested that I use pcdev2 / pcdev3 / pcdev11 as they have good gpus. He also said that I could use source ~gabriele.vajente/virtualenv/bin/activate to use a virtualenv with tensorflow, numpy etc. preinstalled. However, I could not get that working, Therefore I created my own virtual environment with the necessary tensorflow, keras, scipy, numpy etc. libraries and suitable versions. On ssh-ing into the cluster, it can be activated using source /home/millind.vaddiraju/beamtrack/bin/activate. How do I know everything works? Well, I trained a network on it! With the new data. Attached (see attachment #1) is the prediction data for completely new test data. Yeah, its not great, but I got to observe the time it takes for the network to train for 50 epochs-

  1. On pcdev5 CPU: one epoch took ~1500s which is roughly 25 minutes (see Attachment #2). Gautam suggested that I try to train my networks on Optimus. I think this evidence should be sufficient to decide against that idea.
  2. On my GTX 1060: one epoch took ~30s. Which is 25 minutes (for 50 epochs) to train a network.
  3. On pcdev11 GPU (Titan X I think): each epoch took ~16s which is a far more reasonable time.

Therefore, I will carry out all training only on this machine from now.

 


Note to self:

Steps to repeat what you did are:

  1. ssh in to the cluster using ssh albert.einstein@ssh.ligo.org as described here.
  2. activate virtualenv as descirbed above
  3. navigate to code  and run it.
Quote:

 I attempted to train a bunch of networks on the new data to test if the code was alright but realised quickly that, training on my local machine is not feasilble at all as training for 10 epochs took roughly 6 minutes. Therefore, I have placed a request for access to the cluster and am waiting for a reply. I will now set up a bunch of experiments to tune hyperparameters for this data and see what the results are.

  14760   Mon Jul 15 14:09:07 2019 MilindUpdateCamerasCNN LSTM for beam tracking

I've set up network with a CNN encoder (front end) feeding into a single LSTM cell followed by the output layer (see attachment #1). The network requires significantly more memory than the previous ones. It takes around 30s for one epoch of training. Attached are the predicted yaw motion and the fft of the same. The FFT looks rather curious. I still haven't done any tuning and these are only the preliminary results.

Quote:

 Rana also suggested I try LSTMs today. I'll maybe code it up tomorrow. What I have in mind- A conv layer encoder, flatten, followed by an LSTM layer (why not plain RNNs? well LSTMs handle vanishing gradients, so why the hassle).

Well, what about the previous conv nets?

What I did:

  1. Extensive tuning - of learning rate, batch size, dropout ratio, input size using a grid search
  2. Trained each network for 75 epochs and obtained weights, predicted motion and corresponding FFT, error etc.

What I observed:

  1. Loss curves look okay, validation loss isn't going up, so I don't think overfitting is the issue
  2. Training for over (even) 75 epochs seems to be pointless.

What I think is going wrong:

  1. Input size- relatively large input size: 350 x 350. Here, the input image size seems to be 128 x 128.
  2. Inadequate pre-processing.
    1. I have not applied any filters/blurs etc. to the frames.
    2. I have also not tried dimensionality reduction techniques such as PCA

What I will try now:

  1. Collect new data: with smaller amplitudes and different frequencies
  2. Tune the LSTM network for the data I have
  3. Try new CNN architectures with more aggressive max pooling and fewer parameters
  4. Ensembling the models (see this and this). Right now, I have multiple models trained either with same architecture and different hyperparameters or with different architectures. As a first pass, I intend to average the predictions of all the models and see if that improves performance.
Attachment 1: cnn-lstm.png
cnn-lstm.png
Attachment 2: fft_yaw.pdf
fft_yaw.pdf
Attachment 3: yaw_motion.pdf
yaw_motion.pdf
  14761   Mon Jul 15 14:53:40 2019 MilindUpdateIOOkeyed psl crate, unstick.py

Mode cleaner was not locked today. Koji came in and concluded that PSL had died. So we keyed it. Then we ran my unstick.py code. Mode cleaner is locked now.

Quote:

Today, Gautam keyed the C1PSL crate and we got to test my unstick.py code. It seems to be working fine. Remarks:

  1. Gautam moved the unstick.py code to /opt/rtcds/caltech/c1/scripts/cds. Therefore, the steps to run this code are now:
    1. cd /opt/rtcds/caltech/c1/scripts/cds
    2. python unstick.py c1psl (for the c1psl machine)
  2. There is now a sleepTime global variable in the code which defines the amount of delay between successive channel toggles. We set this to 1ms and it took the code around 3s to run.
  3. Gautam was curious to see if this would work even if we set the sleepTime parameter to 0 but decided that that could be tested the next time something was keyed.
  4. I still need to add the signal handling thing to this code.

Following this, we tested my PMC autolocker code. The code ran for about a minute before achieveing lock. Remarks:

  1. Gautam moved my code (pmc_autolocker.py and autolocker_config.yaml) to /cvs/cds/rtcds/caltech/c1/scripts/PSL/PMC/ . Therefore, the steps to run this code are now:
    1. cd /cvs/cds/rtcds/caltech/c1/scripts/PSL/PMC/
    2. python pmc_autolocker.py (check code or use --help to see what the command line arguments do which is only for when you wanna override the details in the .yaml file)
  2. Gautam suggested that I add some delay between succesive steps of DC output adjust so that it locks quickly. I'll do that ASAP. For now, it works.
  14779   Fri Jul 19 16:47:06 2019 MilindUpdateCamerasCNNs for beam tracking || Analysis of results

I did a whole lot of hyperparameter tuning for convolutional networks (without 3d convolution). Of the results I obtained, I am attaching the best results below.

Define "best"?

The lower the power of the error signal (difference between the true and predicted X and Y positions), essentially mse, on the test data, the better the performance of the model. Of the trained models I had, I chose the one with the lowest mse.

Attached results:

  1. Attachment 1: Training configuration
  2. Attachment 2: Predicted motion along the Y direction for the test data
  3. Attachment 3: Predicted motion along the Y direction for the training data
  4. Attachment 4: Learning curves
  5. Attachment 5: Error in test predictions
  6. Attachment 6: Video of image histogram plots
  7. Attachment 7: Plot of percentage of pixels with intensity over 240 with time

(Note: Attachment 6 and 7 present information regarding a fraction of the data. However, the behaviour remains the same for the rest of the data.)

Observations and analysis:

  1. Data:
    1. From attachemtns 2, 3, 5: Maximum deviation from true labels at the peaks of applied dither/motion. Possible reasons:
      1. Stupid Cropping? I checked (by watching the video of cropped frames, i.e visually) to ensure that the entire motion of the beam spot is captured. Therefore, this is not the case.
      2. Intensity variation: The intensity (brightness?) of the beam spot varies (decreases) significantly at the maximum displacement. This, I think, is creating a skewed dataset with very few frames with low intensity pixels. Therefore, I think it makes sense to even this out and get more data points (frames) with similar (lower) pixel intensities. I can think of two ways of doing this:
        1. Collect more data with lower amplitude of sinusoidal dither. I used an amplitude of 80 cts to dither the optic. Perhaps something like 40 is more feasible. This will ensure the dataset isn't too skewed.
        2. Increase exposure time. I used an exposure time of 500us to capture data. Perhaps a higher exposure time will ensure that the image of the beam spot doesn't fade out at the peak of motion.
    2. From attachment 5, Saturated images?: We would like to gun for a maximum deviation of 10% (0.1 in this case) from the true values in the predicted labels (Tbh, I'm not sure why this is a good baseline, I ought to give that some thought. I think the maximum deviation of the OpenCV thing I did at the start might also be a good baseline?). Clearly, we're not meeting that. One possible reason is that the video might be saturated- (too many pixels at 255, bleeding into surrounding pixles) leading to loss of information. I set the exposure time to 500us precisely to avoid this. However, I also created videos of the image histograms of the frames to make sure the frames weren't saturated (Is there some better standard way of doing it?). From attachements 6 and 7, I think it's evident that saturation is not an issue. Consequently, I think increasing the exposure time and collecting data is a good idea.
  2. The network:
    1. From attachment 4: Training post 25 epochs seems to produce overfitting, though it doesn't seem too terrible (from attachments 2 and 3). The network is still learning after 75 epochs, so I'll tinker with the learning rate, dropout and maybe put in annealing.
    2. I don't think there is a need to change the architecture yet. The model seems to generalize okay (valdiation error is close to training error), therefore I think it'll be a good idea to increase dropout for the fully connected layers and train for longer/ with a higher learning rate.

 


 

P.S. I will also try the 2D convolution followed by the 1D convolution thing now. 

P.P.S. Gabriele suggested that I try average pooling instead of max pooling as this is a regression task. I'll give that a shot.

 

Attachment 1: readme.txt
Experiment file: train_both.py
batch_size: 32
dropout_probability: 0.5
eta: 0.0001
filter_size: 1
filter_type: median
initializer: Xavier
memory_size: 10
num_epochs: 75
activation_function: relu
... 22 more lines ...
Attachment 2: yaw_motion_test.pdf
yaw_motion_test.pdf
Attachment 3: yaw_motion_train.pdf
yaw_motion_train.pdf
Attachment 4: Learning_curves_replotted.pdf
Learning_curves_replotted.pdf
Attachment 5: yaw_error_test.pdf
yaw_error_test.pdf
Attachment 6: intensity_histogram.mp4
Attachment 7: saturation_percentage.pdf
saturation_percentage.pdf
  14787   Sat Jul 20 14:43:45 2019 MilindUpdateCamerasCNNs for beam tracking || Analysis of results

<Adding details>

See Attachment #2.

Quote:

Make the MSE a subplot on the same axes as the time series for easier interpretation.

Training dataset:

  1. Peak to peak amplitue in physical units: ?
  2. Dither frequency: 0.2 Hz
  3. Video data: zoomed in video of the beam spot obtained from GigE camera 198.162.113.153 at 500us exposure time. Each frame has a resolution of 640 x 480 which I have cropped to 350 x 350. Attachment #1 is one such frame.
  4. Yes, therefore I am going to obtain video at lower amplitudes. I think that should help me avoid the problem of not-nominal-maximum value?
  5. Other details of the training dataset:
    1. Dataset created from four vides of duration ~ 30, 60, 60, 60 s at 25 FPS.
    2. 4032 training data points
      1. Input (one example/ data point): 10 successive frames stacked to form a 3D volume of shape 350 x 350 x 10
      2. Output (2 dimensional vector): QPD readings (C1:IOO-MC_TRANS_PIT_ERR, C1:IOO-MC_TRANS_YAW_ERR)
    3. Pre-processing: none
    4. Shuffling: Dataset was shuffled before every epoch
    5. No thresholding: Binary images are gonna be of little use if the expectation is that the network will learn to interpret intensity variations of pixels.

Do I need to provide any more details here?

Quote

Describe the training dataset - what is the pk-to-pk amplitude of the beam spot motion you are using for training in physical units? What was the frequency of the dither applied? Is this using a zoomed-in view of the spot or a zoomed out one with the OSEMs in it? If the excursion is large, and you are moving the spot by dithering MC2, the WFS servos may not have time to adjust the cavity alignment to the nominal maximum value.

?

Quote:

What is the minimum detectable motion given the CCD resolution?

see attachment #4.

Quote:
  1. Please upload a cartoon of the network architecture for easier visualization. What is the algorithm we are using? Is the approach the same as using the bright point scatterers to signal the beam spot motion that Gabriele demonstrated successfully

 

I wrote what I think is a handy script to observe if the frames are saturated. I thought this might be handy for if/when I collect data with higher exposure times. I assumed there was no saturation in the images because I'd set the exposure value to something low. I thought it'd be useful to just verify that. Attachment #3 has log scale on the x axis.

Quote:

What is the significance of Attachment #6? I think the x-axis of that plot should also be log-scaled.

 

Quote:
  1. Is the performance of the network still good if you feed it a time-shuffled test dataset? i.e. you have (pictures,Xcoord,Ycoord) tuples, which don't necessarily have to be given to the network in a time-ordered sequence in order to predict the beam spot position (unless the network is somehow using the past beam position to predict the new beam position).
  2. Is the time-sync problem Koji raised limiting this approach?

 

Attachment 1: frame0.pdf
frame0.pdf
Attachment 2: subplot_yaw_test.pdf
subplot_yaw_test.pdf
Attachment 3: intensity_histogram.mp4
Attachment 4: network2.pdf
network2.pdf
  14792   Sun Jul 21 19:27:25 2019 MilindUpdateLoss MeasurementMC2 loss map

I think ezca.Ezca().write() takes the string "CAM-ETMX_SNAP" as an argument and not C1:CAM-ETMX_SNAP. See this, line 47. Are you sure this is not the problem?

Quote:

The camera server keeps throwing the error: failed to grab frames. Milind suggested that it might a problem with the ethernet cable, so I even unplugged it and connected it again; it still had the same issue. One more thing I noticed was, it does take snapshots sometimes with the terminal command caput C1:CAM-ETMX_SNAP 1, but produces a segmentation fault when ezca.Ezca().write(C1:CAM-ETMX_SNAP, 1) is used via ipython. When the terminal command also fails to take snapshots, I noticed that the SNAP button on the GigE medm screen remains on and doesn't switch back to OFF like it is supposed to.

  14805   Wed Jul 24 12:24:43 2019 MilindUpdateIOOunstick.py and ifotest

Moved the unstick.py code to the ifotest repository here. It now handles signals like those generated by Ctrl-C and so forth. It can still be run as python unstick.py <machine1> <machine2> etc.

  14807   Wed Jul 24 20:05:47 2019 MilindUpdateCamerasCNNs for beam tracking || Tales of desperation

At the lab meeting today, Rana suggested that I use the Pylon app to collect more data if that's what I need. Following this, Jon helped me out by updating the pylon version and installing additional software to record video. Now I am collecting data at

  1. higher exposure rate - 600 us magically gives me a saturation percentage of around 1%, see attachment #1 (i.e around 1% of the pixels in the region containing the beam spot are over 240 in value). Ths is a consequence of my discussion with Gabriele where we concluded that I was losing information due the low exposure rate I was using.
  2. For much longer: roughly 10 minutes
    1. at an amplitude of 40 cts for 0.2 Hz
    2. at an amplitude of 20 cts for 0.2 Hz
    3. at an amplitude of 10 cts for 0.2 Hz
    4. at an amplitude of 40 cts for 0.4 Hz
    5. at an amplitude of 20 cts for 0.2 Hz
    6. Random motion

Consequently I have dithered the MC2 optic from around 9:00 PM.

Attachment 1: saturation_percentage.pdf
saturation_percentage.pdf
  14809   Thu Jul 25 00:26:47 2019 MilindUpdateCamerasConvolutional neural networks for beam tracking

Somehow I never got around to doing the pixel sum thing for the new real data from the GigE. Since I have to do it for the presentation, I'm putting up the results here anyway. I've normalized this and computed the SNR with the true readings.

SNR = (power in true readings)/ (power in error signal between true and predicted values)

Attachment #2 is SNR of best performing CNN for comparison.

Attachment 1: centroid.pdf
centroid.pdf
Attachment 2: subplot_yaw_test.pdf
subplot_yaw_test.pdf
  6008   Fri Nov 25 19:45:36 2011 Mirco, DenSummarySUSExcess Noise in Digital Filtering

Quote:

We are now trying to understand why the coherence between SUS-X_SUSPOS_IN1 and SUS-X_SUSPOS_OUT is lost below 1 Hz for X = MC1, MC2, MC3, BS, ITMX, ITMY, ETMX, ETMY, SRM. It is OKEY only for PRM but the different filteres are used there. For PRM - 30:0.0 and Bounce Roll, for all others - 30:0.0 and Cheby. The transfer functions between these two signals plotted in foton and fft tools are also not the same.

If we switch off all the filters between these 2 signals, than the coherence is one. If one of the filters is switched on, everything is also fine. But if there are several present, than they filter the signal in unexpected way.

Moreover it seems that the coherence is dependent on the input signal. The coherence is better with local dumping than without.

 We have done the following on the c1sus and c1lsc computers : provided excitation, let the signal pass through filters 30:0.0 and Cheby and plotted the coherence between in and out signals.

c1sus computer - coherence is corrupted

c1lsc computer - coherence is not corrupted

Attachment 1: sus_coh-crop.pdf
sus_coh-crop.pdf
Attachment 2: lsc_coh-crop.pdf
lsc_coh-crop.pdf
  5420   Thu Sep 15 17:12:29 2011 MirkoUpdateLSCRF modulation depth measurement

[Mirko, Kiwamu]

Put up a temp. setup on the laser table to measure the RF modulation depths using the optical spectrum analyzer. First with a pickoff beam with about 2mW => SNR of 8 of 1 peak per FSR.

Then with a beam with about 100mW. Much better SNR on the single peak but still no sidebands visible. Modematching not too good in either case. Shouldn't matter.

  5425   Thu Sep 15 20:30:52 2011 MirkoUpdateLSCPlan for long term optical spec. analyzer setup

Just to give some heads up on how the setup on the PSL table does / will look. We start out with one of the two reflections of a window. Power about 2mW.

DSC_3430b.jpg

  5426   Thu Sep 15 21:56:01 2011 MirkoUpdateCDSc1oaf check, possible shmem problem

After Jamie installed the c1oaf model ( entry 5424 ) I went and checked the intermodel communication.

Remember the config is:

c1lsc ->SHMEM-> c1oaf
c1oaf ->SHMEM-> c1lsc
c1pem ->SHMEM-> c1rfm ->PCIE-> c1oaf

I checked at least one of every communications type.

-All signals reach their destinations.
-c1lsc_to_c1oaf_via_shmem is more noisy adding noise to the signal. lsc runs at 16kHz and oaf at 2kHz but that should actually smooth things out.

c1lsc_to_c1oaf_via_shmem.png

 

Attachment 1: c1lsc_to_c1oaf_via_shmem.png
c1lsc_to_c1oaf_via_shmem.png
Attachment 2: c1oaf_to_c1lsc_via_shmem_fixed_sine_inj_at_100Hz.png
c1oaf_to_c1lsc_via_shmem_fixed_sine_inj_at_100Hz.png
Attachment 3: c1oaf_to_c1lsc_via_shmem_white_noise_inj.png
c1oaf_to_c1lsc_via_shmem_white_noise_inj.png
Attachment 4: c1pem_to_c1oaf_via_rfm.png
c1pem_to_c1oaf_via_rfm.png
  5462   Mon Sep 19 15:44:32 2011 MirkoUpdateLSCRF modulation depth measurement

Earlier measurements of the modulation index were less than optimal because we had too low transmission through the cavity. Contrary to what was believed you actually need to modematch onto the cavity.

Earlier transmitted power was about 8.5uW.

With a 250mm lens we archived 41uW.

Impinging power on the cavity is 1.7mW.

PD TF approx 0.1V / uW.

Carrier power: 4.1V => 41uW

41uW/1.7mW = 2.4 % transmission. Manufacturer clain for peak transmission: 20-30%.

11MHz SB: 28.8mV => m=0.17

55MHz SB:36mV => m=0.19


As you can see on the pic the SNR of the SBs is not too good.

P9190138.JPG

  5519   Thu Sep 22 15:53:37 2011 MirkoUpdateLSCRF modulation depth measurement again

Toyed around with the modematching some more today.

The outermost glass elements of the OSA are about 28cm apart.

With the OSA beeing a confocal cavity that should mean that the ROC of every mirror is 28cm on the cavity side. If the input surface is flat we need a 28cm focusing lens for good MM. If it's not we shouldn't need any MM.

Tried a f=250mm lens on different positions first. Got at best about 570mV (PD gain=10) in transmission of the OSA.

Then tried a f=1000mm lens. Best transmission 1.22V (7.2% transmission). SB were (PD gain =100) 11MHz: 87.2mV (m=0.17) , 55MHz: 59.2mV (m=0.14)

Lost the position while toying around. Left it then at 1.0V transmisison at 15:15 local time. Let's see how much it drifts. SBs for this were 11MHz: 52.8mV (m=0.17), 55MHz: 73.8mV (m=0.14)

[Ed by KA: If the carrier transmission was really 1.22V and 1.0V the modulation depths calculated are inconsistent with the measurement.]

Spacing between carrier 11MHz and 55MHz SBs seems consistent, and leads to a FSR measurement of 1.5GHz, also fine.

 

Update: After 90mins no change in carrier transmitted power. Next morning: Carrier transmission down by 10%.

DSC_3478.JPG

 DSC_3481.JPG

DSC_3480.JPG

  5530   Fri Sep 23 16:56:07 2011 MirkoUpdateLSCDesired MC modulation frequency measurement, tuning of modulation frequency

[ Mirko, Koji, Suresh ]

Looked into the modulation frequency that should pass the input MC. With a locked MC looked at the RF output of the PD in refl of the MC. Looked at the beat between 11MHz and 29.5MHz. Minimizing it by fine-tuning the 11MHz freq. ( which means maximizing the 11MHz transmission).

SB freq. [MHz]     Beat power [dBm]

11.065650          -75

11.065770          -80 (diving into spec. analyzer instrument noise)

11.066060          -80 (surfacing out of spec. analyzer instrument noise)

Set the freq. to the middle of the last two points: 11.065910MHz at 16:26.

ToDo: How big a problem is the AM?

  5567   Wed Sep 28 18:39:50 2011 MirkoUpdatePEMBLRM seismic channels in c1pem

[Mirko,Jenne]

Created 5-band BLRMS for seismometer data (Gur1, Gur2 and STS1 each in x,y,z respectively) and accelerometer 1 through 6.

Bands are:
0.1Hz-0.3Hz
0.3Hz-1Hz
1Hz-3Hz
3Hz-10Hz
10Hz-30Hz
each with a fitting 4th order butterworth bandpass.

Data is recorded at 256Hz as e.g. C1:PEM-ACC1_RMS_RMS_0p3_1_OUT_DQ. For the 75 channels we have that corresponds to the data rate of just 1.2 16kHz channels.

c1pem execution time increased fom 6-7us to 15-16us out of 480us available.

  5568   Wed Sep 28 21:23:23 2011 MirkoUpdateComputersISCY FE network card / cable not ok

[Mirko,Jenne]

We discovered that the left network cable is not rigidly connected to the back of the ISCY FE computer. You can easily pull it out a mm disconnecting it. It should click rigidly in place. Not clear if it's the cable or the network card.

  5569   Wed Sep 28 21:28:34 2011 MirkoUpdateComputersTorturing control computers. Fine again now

[Mirko, Jenne]

We tried to run an extended version of Matt's LMS adaptive filter c-code. We got the extension to compile separately in gcc first. Then after some tweaking of the code we could make-install c1oaf with the c-code.

This killed c1lsc (the FE computer running c1oaf). Not responding to ssh or even pings. We replaced the bad c-code with harmless code, then reset c1lsc via the hardware button. While looking for c1lsc we discovered the problem with c1iscy network card (see previous entry).

After c1lsc reboot, restart of the FB, and a BURT restore not ok yet

  5577   Thu Sep 29 20:37:12 2011 MirkoUpdateComputersNew c1oaf c-code: Breaking in new way

[Mirko, Jenne]

Programmed a new implementation of the LMS in C. Compiles fine in gcc. The full code still kills c1lsc computer. Tried to go through the code uncommenting more and more. Not perfect in reproducability. The attached version should compile and keep c1oaf running, but not actually produce an adaptive filter. At some point the code just keeps c1oaf from starting up. Leaves the c1lsc computer in working order. At some point I got error messages like ..................................................................
CA.Client.Exception...............................................
    Warning: "Identical process variable names on multiple servers"
    Context: "Channel: "C1:OAF-ADAPT_CORR_BS_Name08", Connecting to: 192.168.113.62:50970, Ignored: c1lsc:34533"
    Source File: ../cac.cpp line 1209
    Current Time: Thu Sep 29 2011 19:18:17.219208306
..................................................................
CA.Client.Exception...............................................
    Warning: "Identical process variable names on multiple servers"
    Context: "Channel: "C1:OAF-ADAPT_CORR_BS_Name09", Connecting to: 192.168.113.62:50970, Ignored: c1lsc:34533"
    Source File: ../cac.cpp line 1209
    Current Time: Thu Sep 29 2011 19:18:17.225999915
..................................................................
CA.Client.Exception...............................................
    Warning: "Identical process variable names on multiple servers"
    Context: "Channel: "C1:OAF-ADAPT_CORR_ETMX_SW1R", Connecting to: 192.168.113.62:50970, Ignored: c1lsc:34533"
    Source File: ../cac.cpp line 1209
    Current Time: Thu Sep 29 2011 19:18:17.243037042
..................................................................

This usually indicates that there are multiple carepeater running. Didn't find where that would be. After rebooting c1lsc and c1sus a couple of times everything seems fine.

Attachment 1: ADAPT_XFCODE11-09-29---20-12.c
// LMS algorithm adaptive filtering for the Caltech 40m -- Sept. 2011 by M. Prijatelj

#define nDOF 8
#define nWIT 21
#define nTAPS 1000

void ADAPT_XFCODE(double *in, int inSize, double *out, int outSize) {

//First the DOFs and their switches
//int nDOF=8;
... 136 more lines ...
  5611   Tue Oct 4 10:35:08 2011 MirkoUpdateCDSc1lsc and c1sus running again

[Alex, Mirko]

Alex fixed the computers this morning. It was in fact a dolphin problem:

Hi Jenne,  figured it out. Even though dxadmin said the Dolphin net was fine, it wasn't. Something happeneed to DIS networkmanager and I had to restart it. It is running on fb: 
controls@fb ~ $ ps -e | grep dis 12280 ?        00:00:00 dis_networkmgr
controls@fb ~ $ sudo /etc/init.d/dis_networkmgr restart
Once the restart was done both c1lsc and c1sus nodes were configured correctly, they printed the usual "node 12 is OK" "node 8 is OK" messages into the dmesg and was able to run /etc/start_fes.sh on lsc and sus to load all the FEs.  Alex

Some lights on c1lsc were still red: C1:DAQ-FBO_C1SYS_SYS and the smaller red light left of it. Restarted the fb. Didn't help. Restarted c1lsc, all green now.
Restored autoburt from Oct 3. 19:07 on c1lsc and c1sus.
  5642   Mon Oct 10 12:14:00 2011 MirkoUpdateComputer Scripts / ProgramsIMC simulations

[Mirko, Kiwamu]

I tried to answer two questions regarding the IMC:

1. What is the coupling of fluctuations in the SB freq. to SB transmitted power?
2. What (if any) is the influence of the IMC on the AM?

I ran into some weird things regarding the corresponding optickle simulations:
1. There seems to be some artifact at the beginning of every simulation sweep.
2. The position of features depends on the parameters of the sweep.

I mailed Matt asking if he sees some error in the simulations

opticke_xaxis.png

Attachment 2: DC_power.png
DC_power.png
Attachment 3: DC_power_B.png
DC_power_B.png
Attachment 4: IMC_simulation.zip
  5663   Thu Oct 13 21:44:48 2011 MirkoUpdateCDSSeismic BLRMS channels, new RMS calculation

[Rana, Koji, Mirko]

We looked into the CDS RMS block c-code as described in Rolfs RCG app guide. Seems the block uses a first order LP filter with a corner freq. / time of 20k execution cycles. There are also some weird thersholds at +-2000counts in there.

I was looking into implementing a hand-made RMS block, by squaring, filtering, rooting. The new RMS (left) seems nicer than the old one (bottom right). Signal was 141counts sinus at 4Hz.

Filters used: Before squaring: 4th order butterworth BP at given freq. & (new) 6th order inverse Chebyshew 20dB at 0.9*lower BP freq. and 1.1*upper BP freq. => about 1dB at BP freq.

                       After squaring: 4th order butterworth LP @ 1Hz.

C1PEM execution time increased from about 20us to about 45us.

Made a new medm screen with the respective filters in place of the empty C1PEM_OVERVIEW. Should go onto the sitemap.

New_RMS_vs_old_RMS.png

Original RMS LP is slower than 0.1Hz, see below for single LP at 0.1Hz in the new RMS. Original RMS is faster than single LP @ 0.01Hz

Original_RMS_LP_slower_than_0.1Hz.png

Some of the channels are recorded as 256Hz DAQ channels now. Need to figure out how to record these as 16Hz EPICS channls.

  5676   Mon Oct 17 10:43:14 2011 MirkoUpdateCDSCommited changes to c1rfm

I want to make changes to c1rfm. Found uncommited changes to it from Sept 27. Since we recompiled it since then it should be safe to commit them, so I did. See svn log for details.

  5677   Mon Oct 17 11:06:31 2011 MirkoUpdateCDSPiping data from c1lsc to c1oaf

Changed, recompiled, installed and restarted c1rfm and c1oaf to get the MC1-3 Pitch and Yaw data into the c1oaf model.
Also changed c1oaf to use MCL as a witness channel (as well as an actuator).
Added the changes to svn.

  5679   Mon Oct 17 14:26:22 2011 MirkoUpdateCDSSeismic BLRMS channels, new RMS calculation

Quote:

[Rana, Koji, Mirko]

We looked into the CDS RMS block c-code as described in Rolfs RCG app guide. Seems the block uses a first order LP filter with a corner freq. / time of 20k execution cycles. There are also some weird thersholds at +-2000counts in there.

I was looking into implementing a hand-made RMS block, by squaring, filtering, rooting. The new RMS (left) seems nicer than the old one (bottom right). Signal was 141counts sinus at 4Hz.

Filters used: Before squaring: 4th order butterworth BP at given freq. & (new) 6th order inverse Chebyshew 20dB at 0.9*lower BP freq. and 1.1*upper BP freq. => about 1dB at BP freq.

                       After squaring: 4th order butterworth LP @ 1Hz.

C1PEM execution time increased from about 20us to about 45us.

Made a new medm screen with the respective filters in place of the empty C1PEM_OVERVIEW. Should go onto the sitemap.

New_RMS_vs_old_RMS.png

Original RMS LP is slower than 0.1Hz, see below for single LP at 0.1Hz in the new RMS. Original RMS is faster than single LP @ 0.01Hz

Original_RMS_LP_slower_than_0.1Hz.png

Some of the channels are recorded as 256Hz DAQ channels now. Need to figure out how to record these as 16Hz EPICS channls.

 Channels are now going into EPICS channels (e.g. C1:PEM-ACC1_RMS_1_3 ). Adapted the PEM_SLOW.ini file. Channels don't yet show up in dataviewer. Probably due to other C1PEM maschine

  5695   Wed Oct 19 12:06:26 2011 MirkoUpdateCDSIncluded the MC servo channel in CDS

[Jamie, Mirko]

Included the 'Servo' output from the D040180 in c1ioo, which I hoped would be a better measure of the MC length fluctuations. It goes into ADC6, labeled CH7 on the physical board.
Servo-output => C1:IOO-MC_SERVO. (Already present is Out1-output => C1:IOO-MC_F).
At low freq. the servo signal is about 4.5dB bigger. Both are recorded at 256Hz now which is the reason for the downward slope at about 100Hz.

MC_F_versus_MC_SERVO.png

Coh_MC_F_MC_SERVO.png

ELOG V3.1.3-