[tor-bugs] #28663 [Core Tor/sbws]: sbws stops accumulating, silently

[Tor Bug Tracker & Wiki]

#28663: sbws stops accumulating, silently
---------------------------+-----------------------------------
 Reporter:  stefani        |          Owner:  juga
     Type:  defect         |         Status:  assigned
 Priority:  Medium         |      Milestone:  sbws: 1.0.x-final
Component:  Core Tor/sbws  |        Version:  sbws: 1.0.2
 Severity:  Major          |     Resolution:
 Keywords:                 |  Actual Points:
Parent ID:  #28639         |         Points:
 Reviewer:                 |        Sponsor:
---------------------------+-----------------------------------

Comment (by juga):

 Replying to [comment:14 teor]:

 > > The point where sbws is stalled is in this loop:
 https://github.com/torproject/sbws/blob/ee64d76df54ceb3a3c9e1e2a797fd70d68bb0035/sbws/core/scanner.py#L362,
 which was added in #28061 in order to stop measuring the same relay by two
 threads when a new loop starts.
 >
 > This code can block forever ~~at a few point~~ for a few different
 reasons. I will open a new child ticket for every location that can block.
 >
 > > Turned out that there's 1 thread that is not measuring relays, it's
 the one storing the results, so that could be solved just changing "> 0"
 to "> 1", i'll try that.
 >
 > pending_results is a list of AsyncResults, not threads:
 >
 https://docs.python.org/2/library/multiprocessing.html#multiprocessing.pool.AsyncResult
 >
 > So this change doesn't do what you say it does.

 That's the logic thing to think, but i couldn't find enough documentation.
 Firstable, Pool manages tasks, jobs and results
 (https://github.com/python/cpython/blob/master/Lib/multiprocessing/pool.py#L186),
 and have at least 3 queues _inqueue, _outqueue, _taskqueue.
 I left sbws running extra logging the "jobs" and the queues, and the
 moment it was stalled, pending_results was 1, so was the difference
 between _inqueue and _outqueue, and the "job" i could see, was thread 1,
 resultdump waiting for more results to get.
 So while i'd agree with you, it's still not clear to me that that's the
 case.

--
Ticket URL: <https://trac.torproject.org/projects/tor/ticket/28663#comment:15>
Tor Bug Tracker & Wiki <https://trac.torproject.org/>
The Tor Project: anonymity online