[tor-bugs] #9874 [BridgeDB]: Research/design a way to automate testing of BridgeDB's HTTPS and email distributors

[Tor Bug Tracker & Wiki]

#9874: Research/design a way to automate testing of BridgeDB's HTTPS and email
distributors
-------------------------+-------------------------------------------------
     Reporter:  isis     |      Owner:  isis
         Type:           |     Status:  needs_review
  enhancement            |  Milestone:
     Priority:  normal   |    Version:
    Component:           |   Keywords:  bridgedb-unittests, automation, ci,
  BridgeDB               |  bridgedb-gsoc-application
   Resolution:           |  Parent ID:
Actual Points:           |
       Points:           |
-------------------------+-------------------------------------------------

Comment (by isis):

 Replying to [comment:18 trygve]:
 > Thank you for feedback. I've attached a patch to fix the failing test.
 I'm new to git (used to cvs, svn and hg), so apologies if I've done this
 wrong.
 >

 Nope, it's cool. `git checkout -b YOURBRANCH develop` and then linking to
 a publicly available remote is generally the best way to go, mostly
 because straight patch files obviously lack commit metadata. I ''think''
 there is a way to use `git-hg` and `hg-git` to work on a git repo in
 mercurial, but then I'm not super familiar with Mercurial.

 > The ipv6 bridge has been changed to a vanilla ipv4 bridge. I've also
 added a new test to check for a scramblesuit bridge. Both of these
 surprised me because they had a different number of fields in each bridge
 line, but I've adjusted the tests accordingly.
 >

 Yep! There's actually a possibly infinite (sort of) amount of `k=v`s which
 can be on the end of bridge-extrainfo `transport` lines, which
 [https://gitweb.torproject.org/torspec.git/blob/HEAD:/pt-spec.txt#l113
 BridgeDB is supposed to put on the end of a bridge line].

 > A few comments:
 > * mechanize is in '.test.requirements.txt' but not 'requirements.txt'.
 I'm assuming this is intentional, but it means that the https tests fail
 when using 'requirements.txt'

 I actually need to remove `requirements.txt`, or remove at least the part
 in the `setup.py` script which reads `requirements.txt` and parses
 everything into `setuptools`/`distribute` dependencies. On the production
 server, we actually delete `requirements.txt` before installing, because
 otherwise `setuptools`/`distribute` drops down to the embedded
 `easy_install`, which we believe is quite unsafe.

 > * leakspin is in '.test.requirements.txt', but its dependency 'ipaddr'
 is not

 When `pip` installs dependencies, it looks into the
 `setuptools`/`distribute` dependencies listed in the `install_requires`
 and `dependency_links` parameters to the `setuptools.setup()` call, and
 also in the `MANIFEST.in` file, if one is present. So `pip` recursively
 downloads dependencies. Also `ipaddr` is specified with a rather
 strange/special syntax in BridgeDB's `requirements.txt`:

 {{{
 https://ipaddr-
 py.googlecode.com/files/ipaddr-2.1.10.tar.gz#sha1=c608450b077b19773d4f1b5f1ef88b26f6650ce0#egg=ipaddr-2.1.10-py2.7
 }}}

 because BridgeDB requires a specific version of `ipaddr` (due to upstream
 bugs, which I've pointed out to the maintainers). This version is

   1. missing from PyPI (although other versions are there)

   2. https://googlecode.com is what `pip` calls "an external index"
 (meaning that it is not https://pypi.org). I filed several CVEs against
 pip last year for unsafe behaviours while crawling indexes, and they very
 responsibly fixed these issues. (In fact, it was already fixed in alpha
 versions of `pip` at the time, just not the version in all the distros.)
 Because we ''need'' to use "an external index" for this dependency, we
 specify `https://` and the expected SHA-1 hash for the package, so that
 `pip` knows how to verify it.

 > * I had to change EMAIL_SMTP_PORT to 2525 in bridgedb.conf. This might
 not be obvious to others.
 > * I had to add "127.0.0.1" to EMAIL_DOMAINS in bridgedb.conf. This might
 not be obvious to others. It may be possible to change the scripts to send
 from one of the 3 allowed domains, as the SMTP server in the test script
 should intercept these emails without forwarding on to the real domains.

 This is possible, and I considered doing it, but I didn't really want to
 change your scripts while testing them out. It requires a change to
 `/etc/hosts`, however, which
 [https://gitweb.torproject.org/bridgedb.git/blob/HEAD:/.travis.yml#l26 we
 already do for Travis], but it's perhaps less nice to ask other people to
 do this in order to run the tests.

 We could change your `test_smtp.py` script to use `example.com`, set that
 in `/etc/hosts` for Travis, and then have your tests skip themselves if
 they aren't being run on Travis. Or...as you suggest:

 > * I had to perform a few more steps to get everything to work ('leakspin
 -n 100' and 'scripts/make-ssl-cert'). This was all documented in the
 excellent README, but would it be useful to create a little script that
 sets up the test environment automatically? You've added something that
 does this for Travis CI, but I've been working from the shell, entering
 commands manually.

 We could move the `before_install` section in `.travis.yml` to some
 script, but then we need to tell people to run a script which runs more
 scripts before testing. The nice thing about having the command separate
 in Travis is that you can see which one failed, versus a whole setup
 script failing. I'm not really sure which one is better. Unless you have
 more ideas?

 It occurs to me that your tests might fail on random machines, like if
 port 2525 was already in use, or they have a weird firewall configuration.
 Perhaps we should only expect them to run in special environments?

 We could do `export TESTING_BRIDGEDB=1` in the test environment setup
 script and Travis, and then check for that in your tests, rather than
 checking for `os.env.get("TRAVIS")` and only running on Travis. That way,
 your tests would only run if the test environment setup script had been
 run, or on CI machines, but would skip if someone tried to run all the
 tests on a random machine.

 Or we could just add the `sed` commands to the `README` and wish everyone
 the best of luck getting this crazy program to run. :/

--
Ticket URL: <https://trac.torproject.org/projects/tor/ticket/9874#comment:19>
Tor Bug Tracker & Wiki <https://trac.torproject.org/>
The Tor Project: anonymity online