[tor-bugs] #20458 [Core Tor/Tor]: Integration tests should be run locally before committing code changes

[Tor Bug Tracker & Wiki]

#20458: Integration tests should be run locally before committing code changes
--------------------------+------------------------------------
 Reporter:  chelseakomlo  |          Owner:
     Type:  enhancement   |         Status:  new
 Priority:  Medium        |      Milestone:  Tor: 0.3.0.x-final
Component:  Core Tor/Tor  |        Version:
 Severity:  Normal        |     Resolution:
 Keywords:  test, doc     |  Actual Points:
Parent ID:                |         Points:
 Reviewer:                |        Sponsor:
--------------------------+------------------------------------

Comment (by chelseakomlo):

 Here is latest progress:

 1. Changes to CodingStandards.md to have similar recommendations to
 WritingTests.md.

 We have `make tests-full` which runs the entire test suite. All I did was
 make this more consistent.

 see `github.com/chelseakomlo/tor_patches.git`, branch `testing`

 We talked about having one make task to cover all actions that are needed
 before pushing code. Is anyone in favor of this? For example, we could
 have a task `make full,` which would run `check-spaces` and `test-full.`

 2. Running integration tests in Jenkins (using docker)

 See `git at github.com:chelseakomlo/tor-integration-ci.git`, master branch

 This just needs to be set up in Jenkins. We can also test using different
 OS/versions as we do with unit tests.

 The blocker to this being more useful is that chutney tests are non-
 deterministic. For the short term, this could be a non-blocking Jenkins
 task.

 (I can open a different issue for this if we want to separate setting up
 integration tests in CI)

 3. Running integration tests locally

 I did a proof of concept for this using docker (it was a minimal change
 from #2 above). The nice thing about this is managing setup/teardown. The
 downside is having a dependency on docker.

 See here: `git at github.com:chelseakomlo/tor-integration-local.git`, master
 branch

 This is just a proof of concept, we can keep running tests as we do now.
 We would need to clean up processes that are left over after tests have
 been run (see #20409).

 What is left to do?
         1. Make chutney tests more deterministic.

 First, making test as determinisitc as possible. I know we should look for
 race conditions, but any other examples of ways to fix flakiness are
 welcome.

 It would also be great to have a deeper conversation about what an
 acceptable fail rate could be (best 2 out of 3, for example), as my
 understanding is that these tests can never be 100% deterministic.

         2. Allow integration tests to be run offline. Tickets have already
 been opened for these:

 https://trac.torproject.org/projects/tor/ticket/19573
 https://trac.torproject.org/projects/tor/ticket/16971

 Let me know any thoughts/ideas on these, particularly on making tests more
 deterministic. I think next steps can be 1) creating a Jenkins task for
 running integration tests and 2) making chutney tests more deterministic.

--
Ticket URL: <https://trac.torproject.org/projects/tor/ticket/20458#comment:7>
Tor Bug Tracker & Wiki <https://trac.torproject.org/>
The Tor Project: anonymity online