commit 53277867865a0192dc30d29c35e9558d85082421 Author: Damian Johnson atagar@torproject.org Date: Sat Oct 11 12:36:49 2014 -0700
Free memory prior to sending email
Frequently we run into OOM issues when sending a notification email at the end of our run...
Traceback (most recent call last): File "/srv/doctor.torproject.org/doctor/consensus_health_checker.py", line 705, in <module> util.send("Script Error", body_text = msg, destination = util.ERROR_ADDRESS) File "/srv/doctor.torproject.org/doctor/util.py", line 97, in send stderr = subprocess.PIPE, File "/usr/lib/python2.7/subprocess.py", line 679, in __init__ errread, errwrite) File "/usr/lib/python2.7/subprocess.py", line 1153, in _execute_child self.pid = os.fork() OSError: [Errno 12] Cannot allocate memory
This is because we call subprocess.Popen which forks our process, doubling the amount of memory we use. Sending an email obviously doesn't require much, but since we're using hundreds of megs for all the consensuses we download that's what is allocated.
http://stackoverflow.com/a/13329386/1067192
Attempting to address this by freeing descriptor content prior to forking our process. --- consensus_health_checker.py | 10 ++++++++++ 1 file changed, 10 insertions(+)
diff --git a/consensus_health_checker.py b/consensus_health_checker.py index 54c50da..8254dc5 100755 --- a/consensus_health_checker.py +++ b/consensus_health_checker.py @@ -7,6 +7,7 @@ Performs a variety of checks against the present votes and consensus. """
import datetime +import gc import time import traceback
@@ -219,6 +220,15 @@ def main(): for issue in issues: rate_limit_notice(issue)
+ # Reclaim memory of the consensus documents. This is ebecause sending an + # email forks our process, doubling memory usage. This can easily be a + # trigger of an OOM if we're still gobbling tons of memory for the + # descriptor content. + + del consensuses + del votes + gc.collect() + if TEST_RUN: print '\n'.join(map(str, issues)) else: