[tor-dev] Stem Integ tests for Tor

[Nick Mathewson]

On Tue, Apr 30, 2013 at 10:29 AM, Ashish Patil <cool_asis_is at yahoo.in> wrote:
> On Mon, Apr 29, 2013 at 17:44 PM, Nick Mathewson <nickm at alum.mit.edu>
> wrote:
>> Right now, all chutney
>> does is configure and launch a small Tor network, but the small
>> network requires a little babysitting to make it bootstrap. (For
>> example, you sometimes need to send the clients a SIGHUP once the
>> authorities have made their first consensus.
>
> So, the first bits for improving chutney would be to automate the
> network generation as much as possible (leaving no loose ends)?

The network configuration works fine.  The bootstrapping is a bit
tricky.  Personally, though, I would focus on the actual testing
first: having a test for "did the bootstrapping finish" would be
invaluable if you're trying to actually make sure the bootstrapping
finishes.

>> What Chutney doesn't do (but should!) is actually test the network it
>> launches. Right now, to make sure the network has bootstrapped, you
>> have to read through its log files.  If you want to test the network
>> with any traffic, you need to use an application of your choice to
>> send traffic through one of the Tor clients, and verify that the
>> output as is expected. And if you want to make sure that Tor is
>> running without weird warnings, you need to manually inspect the
>> log files again.  It would be good if Chutney grew into, or were
>> replaced by, a tool that could actually do testing right that.
>
> Now, we want chutney to be robust enough to encompass all the manual
> that is mentioned above & automate it as much as possible or atleast
> print the tasks to be performed by the tester sequentially. We would
> need a set of static reliable web pages if we want to automate the
> traffic testing?

I think that instead, you'd want a little program that ran a server
locally.  Having an external webserver would mean that network
problems, or problems in the external webserver, would make Tor
untestable while they were getting fixed.

> Also, we would parse the log files for errors or warnings so that
> we dont have to manually check for them.
>
>   * Construct a circuit to an exit which allows port 80 and request a
> site through it.
>   * Do the same for an exit which rejects port 80.
>   * Construct a one-hop circuit through an exit.
>   * Try to use a circuit that runs through a bridge.
>   * Let tor pick the path of a circuit and assert that the first
> relay's a guard and the last is an exit.
>   * Attempt to request a page from a hidden service.
>   * Host a hidden service and request something from ourselves.
>   * Act as a relay and try to construct a circuit that includes ourselves.
>   * ... etc...
>
> As Damian listed down all the tests one needs to carry out for actual
> network testing (keeping in mind unreliability of the network at times),
> what all additional tests would we like chutney to have?

I think for a while at least, we'd want to be coverage-driven: looking
for an area of the code that isn't getting called during testing, and
adding a test for it, then iterating.

best wishes,
-- 
Nick