commit f3804a4d6f488e2cd35001ec856ebc79ed769bbb Author: Nick Mathewson nickm@torproject.org Date: Thu Oct 8 10:55:31 2015 -0400
finish up doc/WritingTests.txt --- doc/WritingTests.txt | 144 +++++++++++++++++++++++++++++++++++++++++++++++--- 1 file changed, 136 insertions(+), 8 deletions(-)
diff --git a/doc/WritingTests.txt b/doc/WritingTests.txt index 62a17e3..977b836 100644 --- a/doc/WritingTests.txt +++ b/doc/WritingTests.txt @@ -66,6 +66,8 @@ and stuff like that.
=== Finding test coverage
+Test coverage is a measurement of which lines your tests actually visit. + When you configure Tor with the --enable-coverage option, it should build with support for coverage in the unit tests, and in a special "tor-cov" binary. @@ -245,29 +247,155 @@ And later, you can restore the original function with: For more information, see the definitions of this mocking logic in testsupport.h.
+=== Okay but what should my tests actually do? + +We talk above about "test coverage" -- making sure that your tests visit +every line of code, or every branch of code. But visiting the code isn't +enough: we want to verify that it's correct. + +So when writing tests, try to make tests that should pass with any correct +implementation of the code, and that should fail if the code doesn't do what +it's supposed to do. + +You can write "black-box" tests or "glass-box" tests. A black-box test is +one that you write without looking at the structure of the function. A +glass-box one is one you implement while looking at how the function is +implemented. + +In either case, make sure to consider common cases *and* edge cases; success +cases and failure csaes. + +For example, consider testing this function: + + /** Remove all elements E from sl such that E==element. Preserve + * the order of any elements before E, but elements after E can be + * rearranged. + */ + void smartlist_remove(smartlist_t *sl, const void *element); + +In order to test it well, you should write tests for at least all of the +following cases. (These would be black-box tests, since we're only looking +at the declared behavior for the function: + + * Remove an element that is in the smartlist. + * Remove an element that is not in the smartlist. + * Remove an element that appears in the smartlist more than once. + +And your tests should verify that it behaves correct. At minimum, you should +test: + + * That other elements before E are in the same order after you call the + functions. + * That the target element is really removed. + * That _only_ the target element is removed. + +When you consider edge cases, you might try: + + * Remove an element from an empty list. + * Remove an element from a singleton list containing that element. + * Remove an element for a list containing several instances of that + element, and nothing else. + +Now let's look at the implementation: + + void + smartlist_remove(smartlist_t *sl, const void *element) + { + int i; + if (element == NULL) + return; + for (i=0; i < sl->num_used; i++) + if (sl->list[i] == element) { + sl->list[i] = sl->list[--sl->num_used]; /* swap with the end */ + i--; /* so we process the new i'th element */ + sl->list[sl->num_used] = NULL; + } + } + +Based on the implementation, we now see three more edge cases to test: + + * Removing NULL from the list. + * Removing an element from the end of the list + * Removing an element from a position other than the end of the list. + + +=== What should my tests NOT do? + +Tests shouldn't require a network connection. + +Whenever possible, tests shouldn't take more than a second. Put the test +into test/slow if it genuinely needs to be run. + +Tests should not alter global state unless they run with TT_FORK: Tests +should not require other tests to be run before or after them. + +Tests should not leak memory or other resources. + +When possible, tests should not be over-fit to the implementation. That is, +the test should verify that the documented behavior is implemented, but +should not break if other permissible behavior is later implemented. +
=== Advanced techniques: Namespaces
-XXXX write this. danah boyd made us some really awesome stuff here. +Sometimes, when you're doing a lot of mocking at once, it's convenient to +isolate your identifiers within a single namespace. If this were C++, we'd +already have namespaces, but for C, we do the best we can with macros and +token-pasting. + +We have some macros defined for this purpose in src/test/test.h. To use +them, you define NS_MODULE to a prefix to be used for your identifiers, and +then use other macros in place of identifier names. See src/test/test.h for +more documentation.
Integration tests: Calling Tor from the outside -----------------------------------------------
-XXXX WRITEME +Some tests need to invoke Tor from the outside, and shouldn't run from the +same process as the Tor test program. Reasons for doing this might include: + + * Testing the actual behavior of Tor when run from the command line + * Testing that a crash-handler correctly logs a stack trace + * Verifying that a violating a sandbox or capability requirement will + actually crash the program. + * Needing to run as root in order to test capability inheritance or + user switching. + +To add one of these, you generally want a new C program in src/test. Add it +to TESTS and noinst_PROGRAMS if it can run on its own and return success or +failure. If it needs to be invoked multiple times, or it needs to be +wrapped, add a new shell script to TESTS, and the new program to +noinst_PROGRAMS. If you need access to any environment variable from the +makefile (eg ${PYTHON} for a python interpreter), then make sure that the +makefile exports them.
Writing integration tests with Stem -----------------------------------
-XXXX WRITEME +The 'stem' library includes extensive unit tests for the Tor controller +protocol. + +For more information on writing new tests for stem, have a look around +the tst/* directory in stem, and find a good example to emulate. You +might want to start with +https://gitweb.torproject.org/stem.git/tree/test/integ/control/controller.py +to improve Tor's test coverage. + +You can run stem tests from tor with "make test-stem", or see +https://stem.torproject.org/faq.html#how-do-i-run-the-tests .
System testing with Chutney ---------------------------
-XXXX WRITEME - -Who knows what evil lurks in the timings of networks? The Shadow knows! ------------------------------------------------------------------------ +The 'chutney' program configures and launches a set of Tor relays, +authorities, and clients on your local host. It has a 'test network' +functionality to send traffic through them and verify that the traffic +arrives correctly.
-XXXX WRITEME +You can write new test networks by adding them to 'networks'. To add +them to Tor's tests, add them to the test-network or test-network-all +targets in Makefile.am.
+(Adding new kinds of program to chutney will still require hacking the +code.)