Calendars: The calendar issue discussed previously (http://nodus.ligo.caltech.edu:8080/40m/8098) where the numbers are squished together is very difficult for me to find. I am not going to worry about it for the time being.
Multiprocessing: Reviewed the implementation of Multiprocessing in python (using Multiprocessing package). Wrote a simple test function and ran it on megatron, to verify that multiprocessing could successfully take advantage of megatron’s multiple cores – it could. Now, I will work on implementing multiprocessing in the program. I began testing at a section in the program where a for loop calls process_data() (which has a long runtime) multiple times. The megatron terminals I had open began to run very slowly. Why? I believe that the process_data() function loads data into global variables to accomplish its task. The global variables in the original implementation were cleared before the subsequent calls to process_data(). But in the multiprocessing version, the data is not cleared, meaning the memory fills quickly, which drastically reduces performance. In the short term, I could try generating fewer processes at a time, wait for them to finish, then clearing the data, then generating more processes, etc. This will probably generate a nominal performance boost. In the long-term, restructuring of the way the program handles data may help (but not for sure). In the coming week I will experiment with these techniques and try to decrease the run time of the program. |