[tor-bugs] #11978 [Ooni]: Recursion limit problem in TaskManager

[Tor Bug Tracker & Wiki]

#11978: Recursion limit problem in TaskManager
-------------------------+-------------------------
 Reporter:  cypherpunks  |          Owner:  hellais
     Type:  defect       |         Status:  new
 Priority:  normal       |      Milestone:
Component:  Ooni         |        Version:
 Keywords:               |  Actual Points:
Parent ID:               |         Points:
-------------------------+-------------------------
 This issue was automatically migrated from github issue
 https://github.com/TheTorProject/ooni-probe/issues/296.

 The task manager is currently designed to recursively call `_fillSlots`
 that will then call `_run` that then again will call `_fillSlots` on
 success or failure. This means that when there are a lot of tasks failing
 very quickly it is very likely that the default python recursion limit
 will be overcome (1000).

 To reproduce this bug you can try and run a test with a long invalid input
 for example http_requests:

 `ooniprobe blocking/http_requests -f data/complete.deck`

 Note that the fact that this test fails is correct, however it fails in a
 surprising manner:

 ```
 Unhandled error in Deferred:
 Unhandled Error
 Traceback (most recent call last):
   File "/ooni-probe/ooni/managers.py", line 153, in _failed
     super(LinkedTaskManager, self)._failed(result, task)
   File "/ooni-probe/ooni/managers.py", line 44, in _failed
     task.done.errback(failure)
   File "/.virtualenvs/ooni-probe/lib/python2.7/site-
 packages/twisted/internet/defer.py", line 423, in errback
     self._startRunCallbacks(fail)
   File "/.virtualenvs/ooni-probe/lib/python2.7/site-
 packages/twisted/internet/defer.py", line 490, in _startRunCallbacks
     self._runCallbacks()
 --- <exception caught here> ---
   File "/.virtualenvs/ooni-probe/lib/python2.7/site-
 packages/twisted/internet/defer.py", line 577, in _runCallbacks
     current.result = callback(current.result, *args, **kw)
   File "/ooni-probe/ooni/director.py", line 188, in measurementFailed
     log.msg("Failed doing measurement: %s" % measurement)
   File "/ooni-probe/ooni/utils/log.py", line 62, in msg
     print "%s" % msg
   File "/.virtualenvs/ooni-probe/lib/python2.7/site-
 packages/twisted/python/log.py", line 505, in write
     msg(message, printed=1, isError=self.isError)
   File "/.virtualenvs/ooni-probe/lib/python2.7/site-
 packages/twisted/python/threadable.py", line 53, in sync
     return function(self, *args, **kwargs)
   File "/.virtualenvs/ooni-probe/lib/python2.7/site-
 packages/twisted/python/log.py", line 185, in msg
     actualEventDict = (context.get(ILogContext) or {}).copy()
   File "/.virtualenvs/ooni-probe/lib/python2.7/site-
 packages/twisted/python/context.py", line 121, in getContext
     return self.currentContext().getContext(key, default)
 exceptions.RuntimeError: maximum recursion depth exceeded
 ```

 I think this bug is perhaps a good opportunity to discuss some possible
 refactoring of the task scheduler related code. It may be a good idea to
 draw some inspiration from: https://github.com/terrycojones/txrdq

--
Ticket URL: <https://trac.torproject.org/projects/tor/ticket/11978>
Tor Bug Tracker & Wiki <https://trac.torproject.org/>
The Tor Project: anonymity online