commit 30e2a75258e970bcf3a6643d739f62277e6596e6 Merge: 04f45ab 5b18cd5 Author: Damian Johnson atagar@torproject.org Date: Thu Jun 8 10:09:56 2017 -0700
Parallelize integration tests
For months I've been striving to improve our test performance but no matter what I did I ran into walls with python's GIL. At this year's PyCon however I discovered a module I wasn't aware of before: multiprocessing.
The multiprocessing module allows callers to easily spawn worker processes similar to threads, bypassing the python GIL. Using this to parallelize our long running tests to greatly improve our testing runtime on multi-core systems.
For me this drops the 'run_tests.py --all' runtime from 49s to 26s (47% faster). I'm on an old dual-core system so speedups should be more pronounced on anything modern. That said, the tests now saturate both cores for the duration of the tests so this hits the ceiling for what parallelization can buy me at present. When I upgrade there's more tests that are good candidates for parallelization.
docs/change_log.rst | 1 + run_tests.py | 44 +- stem/util/system.py | 14 +- stem/util/test_tools.py | 162 ++++++ test/integ/descriptor/extrainfo_descriptor.py | 28 +- test/integ/descriptor/microdescriptor.py | 18 +- test/integ/descriptor/networkstatus.py | 38 +- test/integ/descriptor/server_descriptor.py | 25 +- test/integ/installation.py | 141 +++-- test/integ/process.py | 715 ++++++++++++++------------ test/integ/util/system.py | 12 +- test/settings.cfg | 14 +- test/task.py | 1 + 13 files changed, 744 insertions(+), 469 deletions(-)