[ooni-dev] Exposing too many or too few options from ooni tests

[Will]

The model for test parameters has been a bit confusing to me. I think it's
because there are two different audiences that we're trying to cater to.

1. End users. For them, there should be good defaults, like the hard-coded
expected value here. In order to run the test, a user really shouldn't have
to think about anything, and something reasonable should happen. In the
same way, we should get to a point where we automatically fetch a
reasonable test deck and run reasonable tests for users who just want to
understand or report on what's going on without learning the intricacies of
ooni.

2. OONI developers / researchers debugging interference mechanisms.  Here,
someone is trying to dig deeper into what's going on through interactive
testing. In this case, it seems like it would be plausible that they might
want to have hooks exposed for changing how tests work, especially across a
bunch of remote probes, without needing to redeploy changed code. This
specific hook seems like it wouldn't commonly need to be changed, but I
could imagine someone wanting to re-use this test to see which domains
special-case lantern or if there are issues with lantern requesting back
into a country by modifying this parameter.

My gut feeling is that the easiest way to serve both audiences is to leave
a way to explicitly set the parameter, but focus on good, hard-coded
defaults that we expect to be used for collected reports.

--Will

On Tue, Mar 15, 2016 at 10:18 AM, Arturo Filastò <art at torproject.org> wrote:

> Hello Oonitarians,
>
> I was having a discussion with vasilis about some changes that I recently
> did to the lantern
> and psiphon test around exposing some extra configuration options and
> would like to know
> what your opinion is on the matter.
>
> Basically these tests will try to run the lantern and psiphon tool, then
> attempt to connect to
> a certain website with it and verify if the response it gets from the
> website is the expected one.
>
> In the past it was possible to configure the URL to be fetched and the
> response body that is
> expected with a sane default.
>
> I have changed this to no longer be the case and instead use a hardcoded
> value of a website
> that we expect to not change in the future and an expected result for it.
> In the specific case it’s http://www.google.com/humans.txt
>
> The reason for doing this is that I want to avoid the possibility of a
> user misconfiguring the
> URL and expected body to something that is not true (I say foo.com
> results “bar” while it
> actually returns “foo”) and leading to inconsistent results.
>
> The argument against this is that the website we use for testing may
> change in the future
> and if we don’t notice then we can still have inconsistent results.
>
> To this I believe that even if that were to become the case it’s more
> likely that us developers
> of the tool will notice and hence ship an update than expect the user to
> tweak their ooniprobe
> to provide valid measurements.
>
> I believe that exposing some settings that can lead to measurements that
> are not true is
> sub-optimal, but I would like to hear contrasting opinions.
>
> ~ Arturo
>
> _______________________________________________
> ooni-dev mailing list
> ooni-dev at lists.torproject.org
> https://lists.torproject.org/cgi-bin/mailman/listinfo/ooni-dev
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.torproject.org/pipermail/ooni-dev/attachments/20160315/27962566/attachment.html>