[tor-commits] [stem/master] Parallelize integration tests

[atagar at torproject.org]

commit 30e2a75258e970bcf3a6643d739f62277e6596e6
Merge: 04f45ab 5b18cd5
Author: Damian Johnson <atagar at torproject.org>
Date:   Thu Jun 8 10:09:56 2017 -0700

    Parallelize integration tests
    
    For months I've been striving to improve our test performance but no matter
    what I did I ran into walls with python's GIL. At this year's PyCon however I
    discovered a module I wasn't aware of before: multiprocessing.
    
    The multiprocessing module allows callers to easily spawn worker processes
    similar to threads, bypassing the python GIL. Using this to parallelize our
    long running tests to greatly improve our testing runtime on multi-core
    systems.
    
    For me this drops the 'run_tests.py --all' runtime from 49s to 26s (47%
    faster). I'm on an old dual-core system so speedups should be more pronounced
    on anything modern. That said, the tests now saturate both cores for the
    duration of the tests so this hits the ceiling for what parallelization can buy
    me at present. When I upgrade there's more tests that are good candidates for
    parallelization.

 docs/change_log.rst                           |   1 +
 run_tests.py                                  |  44 +-
 stem/util/system.py                           |  14 +-
 stem/util/test_tools.py                       | 162 ++++++
 test/integ/descriptor/extrainfo_descriptor.py |  28 +-
 test/integ/descriptor/microdescriptor.py      |  18 +-
 test/integ/descriptor/networkstatus.py        |  38 +-
 test/integ/descriptor/server_descriptor.py    |  25 +-
 test/integ/installation.py                    | 141 +++--
 test/integ/process.py                         | 715 ++++++++++++++------------
 test/integ/util/system.py                     |  12 +-
 test/settings.cfg                             |  14 +-
 test/task.py                                  |   1 +
 13 files changed, 744 insertions(+), 469 deletions(-)