[tor-bugs] #12457 [DocTor]: consensus_health_checker repeatedly failing (was: Memory request for cappadocicum)

[Tor Bug Tracker & Wiki]

#12457: consensus_health_checker repeatedly failing
-----------------------------+----------------------
     Reporter:  atagar       |      Owner:
         Type:  enhancement  |     Status:  reopened
     Priority:  minor        |  Milestone:
    Component:  DocTor       |    Version:
   Resolution:               |   Keywords:
Actual Points:               |  Parent ID:
       Points:               |
-----------------------------+----------------------
Changes (by atagar):

 * status:  closed => reopened
 * resolution:  worksforme =>
 * component:  Tor Sysadmin Team => DocTor


Comment:

 Reopening as this has happened sixteen more times over the last couple
 days. One provided the aforementioned stacktrace but most simply say
 'killed'. I'm watching the host a bit right now. First thing that I
 noticed is that there were multiple python processes running in parallel.
 Cron should be spacing them out to avoid that, but it looks like the
 consensus checker is taking a lot longer to run of late...

 {{{
 grep "Checks finished"
 /srv/doctor.torproject.org/doctor/logs/consensus_health_checker | tail -n
 15
 07/03/2014 14:49:02 [DEBUG] Checks finished, runtime was 241.29 seconds
 07/03/2014 15:51:28 [DEBUG] Checks finished, runtime was 385.82 seconds
 07/03/2014 16:48:15 [DEBUG] Checks finished, runtime was 193.36 seconds
 07/03/2014 17:48:54 [DEBUG] Checks finished, runtime was 232.95 seconds
 07/03/2014 18:51:13 [DEBUG] Checks finished, runtime was 371.43 seconds
 07/03/2014 19:46:21 [DEBUG] Checks finished, runtime was 79.41 seconds
 07/04/2014 03:46:56 [DEBUG] Checks finished, runtime was 113.61 seconds
 07/04/2014 04:46:16 [DEBUG] Checks finished, runtime was 75.24 seconds
 07/04/2014 05:51:23 [DEBUG] Checks finished, runtime was 381.72 seconds
 07/04/2014 06:52:03 [DEBUG] Checks finished, runtime was 421.76 seconds
 07/04/2014 08:00:32 [DEBUG] Checks finished, runtime was 930.34 seconds
 07/04/2014 15:01:49 [DEBUG] Checks finished, runtime was 1006.77 seconds
 07/04/2014 17:00:36 [DEBUG] Checks finished, runtime was 934.79 seconds
 07/04/2014 17:58:50 [DEBUG] Checks finished, runtime was 827.66 seconds
 07/04/2014 19:02:07 [DEBUG] Checks finished, runtime was 1025.27 seconds
 }}}

 Changing the title and reassigning to my component as I'm really sure if
 this is actually memory related or not.

--
Ticket URL: <https://trac.torproject.org/projects/tor/ticket/12457#comment:5>
Tor Bug Tracker & Wiki <https://trac.torproject.org/>
The Tor Project: anonymity online