[tor-bugs] #28866 [Core Tor/sbws]: ResultDump.queue.put() can hang if the queue is full

[Tor Bug Tracker & Wiki]

#28866: ResultDump.queue.put() can hang if the queue is full
---------------------------+-----------------------------------
 Reporter:  teor           |          Owner:  (none)
     Type:  defect         |         Status:  new
 Priority:  Medium         |      Milestone:  sbws: 1.0.x-final
Component:  Core Tor/sbws  |        Version:  sbws: 1.0.2
 Severity:  Normal         |     Resolution:
 Keywords:                 |  Actual Points:
Parent ID:  #28663         |         Points:
 Reviewer:                 |        Sponsor:
---------------------------+-----------------------------------

Comment (by juga):

 After looking more at this.

 Replying to [ticket:28866 teor]:
 > sbws calls `ResultDump.queue.put()` in blocking mode:
 >
 https://github.com/torproject/sbws/blob/ee64d76df54ceb3a3c9e1e2a797fd70d68bb0035/sbws/core/scanner.py#L303

 put is not blocking because it happens in result_dump's thread
 (result_dump launches its own thread before the the pool is instantiated)

 >
 > But multiprocessing callbacks need to return immediately:
 >
 https://docs.python.org/3/library/multiprocessing.html#multiprocessing.pool.Pool.apply_async
 >
 > So sbws should call put() without blocking, or with a (very small)
 timeout:
 > https://docs.python.org/3/library/queue.html#queue.Queue.put

 With put timeout it will always return Full unless the slot is inmediatly
 available (https://docs.python.org/3/library/queue.html#queue.Queue.put)
 It's also very unlikely that the queue will be full, since results happen
 with a difference of seconds and worst case there're only 6000.
 A way to solve this without a different thread, queue and lock in
 result_putter, would be to use a deque, but i don't think it's necessary,
 since i think the bug is in #28897.

--
Ticket URL: <https://trac.torproject.org/projects/tor/ticket/28866#comment:2>
Tor Bug Tracker & Wiki <https://trac.torproject.org/>
The Tor Project: anonymity online