40m QIL Cryo_Lab CTN SUS_Lab TCS_Lab OMC_Lab CRIME_Lab FEA ENG_Labs OptContFac Mariner WBEEShop
  40m Log  Not logged in ELOG logo
Message ID: 8148     Entry time: Sat Feb 23 16:16:11 2013
Author: Max Horton 
Type: Update 
Category: Summary Pages 
Subject: Multiprocessing 

Calendars:  The calendar issue discussed previously (http://nodus.ligo.caltech.edu:8080/40m/8098) where the numbers are squished together is very difficult for me to find.  I am not going to worry about it for the time being.

Multiprocessing:   Reviewed the implementation of Multiprocessing in python (using Multiprocessing package).  Wrote a simple test function and ran it on megatron, to verify that multiprocessing could successfully take advantage of megatron’s multiple cores – it could.  Now, I will work on implementing multiprocessing in the program.  I began testing at a section in the program where a for loop calls process_data() (which has a long runtime) multiple times.  The megatron terminals I had open began to run very slowly.  Why?  I believe that the process_data() function loads data into global variables to accomplish its task.  The global variables in the original implementation were cleared before the subsequent calls to process_data().  But in the multiprocessing version, the data is not cleared, meaning the memory fills quickly, which drastically reduces performance.  In the short term, I could try generating fewer processes at a time, wait for them to finish, then clearing the data, then generating more processes, etc.  This will probably generate a nominal performance boost.  In the long-term, restructuring of the way the program handles data may help (but not for sure).  In the coming week I will experiment with these techniques and try to decrease the run time of the program.

ELOG V3.1.3-