[ooni-dev] Exposing too many or too few options from ooni tests

[Arturo Filastò]

Hello Oonitarians,

I was having a discussion with vasilis about some changes that I recently did to the lantern
and psiphon test around exposing some extra configuration options and would like to know
what your opinion is on the matter.

Basically these tests will try to run the lantern and psiphon tool, then attempt to connect to
a certain website with it and verify if the response it gets from the website is the expected one.

In the past it was possible to configure the URL to be fetched and the response body that is
expected with a sane default.

I have changed this to no longer be the case and instead use a hardcoded value of a website
that we expect to not change in the future and an expected result for it.
In the specific case it’s http://www.google.com/humans.txt

The reason for doing this is that I want to avoid the possibility of a user misconfiguring the
URL and expected body to something that is not true (I say foo.com results “bar” while it
actually returns “foo”) and leading to inconsistent results.

The argument against this is that the website we use for testing may change in the future
and if we don’t notice then we can still have inconsistent results.

To this I believe that even if that were to become the case it’s more likely that us developers
of the tool will notice and hence ship an update than expect the user to tweak their ooniprobe
to provide valid measurements.

I believe that exposing some settings that can lead to measurements that are not true is
sub-optimal, but I would like to hear contrasting opinions.

~ Arturo