Multiprocessing: In its current form, the code uses multiprocessing to the maximal extent possible. It takes roughly 2600 seconds to run (times may vary depending on what else megatron is running, etc.). Multiprocessing is only used on the process_data() function calls, because this by far takes the longest. The other function calls after the process_data() calls take a combined ~120 seconds. See http://nodus.ligo.caltech.edu:8080/40m/8218 for details on the use of Multiprocessing to call process_data().
Crontab: I also updated the crontab in an attempt to fix the problem where data is only displayed until 5PM. Recall that previously (http://nodus.ligo.caltech.edu:8080/40m/8098) I found that the crontab wasn't even calling the summary_pages.py script after 5PM. I changed it then to be called at 11:59PM, which also didn't work because of the day change after midnight.
I decided it would be easiest to just call the function on the previous day's data at 12:01AM the next day. So, I changed the crontab.
59 5,11,17,23 * * * /users/public_html/40m-summary/bin/c1_summary_page.sh 2>&1
0 6,12,18 * * * /users/public_html/40m-summary/bin/c1_summary_page.sh 2>&1
1 0 * * * /users/public_html/40m-summary/bin/c1_summary_page.sh $(date "+%Y/%m/%d" --date="1 days ago") 2>&1
For some reason, as of 9:00PM today (March 4, 2013) I still don't see any data up, even though the change to the crontab was made on February 28. Even more bizarre is the fact that data is present for March 1-3. Perhaps some error was introduced into the code somehow, or I don't understand how crontab does its job. I will look into this now.
Once I fix the above problem, I will begin refactoring the code into different well-documented classes.