[ooni-dev] Exposing too many or too few options from ooni tests

[Simone Basso]

On 15/03/16 18:18, Arturo Filastò wrote:
> Hello Oonitarians,
> 
> I was having a discussion with vasilis about some changes that I recently did to the lantern
> and psiphon test around exposing some extra configuration options and would like to know
> what your opinion is on the matter.
> 
> Basically these tests will try to run the lantern and psiphon tool, then attempt to connect to
> a certain website with it and verify if the response it gets from the website is the expected one.
> 
> In the past it was possible to configure the URL to be fetched and the response body that is
> expected with a sane default.
> 
> I have changed this to no longer be the case and instead use a hardcoded value of a website
> that we expect to not change in the future and an expected result for it.
> In the specific case it’s http://www.google.com/humans.txt
> 
> The reason for doing this is that I want to avoid the possibility of a user misconfiguring the
> URL and expected body to something that is not true (I say foo.com results “bar” while it
> actually returns “foo”) and leading to inconsistent results.
> 
> The argument against this is that the website we use for testing may change in the future
> and if we don’t notice then we can still have inconsistent results.
> 
> To this I believe that even if that were to become the case it’s more likely that us developers
> of the tool will notice and hence ship an update than expect the user to tweak their ooniprobe
> to provide valid measurements.
> 
> I believe that exposing some settings that can lead to measurements that are not true is
> sub-optimal, but I would like to hear contrasting opinions.

Let's see if I can end up with a contrasting opinion...

So, in general I tend to think that users may know better than me and so
I am in favour of providing knobs to twist my programs behavior.

What I find optimal behavior is when a tool prevents me from doing
stupid things but at the same time allows me to --force it.

But in this case a strange thing happens. Usually command line flags
should be orthogonal but this is not the case, IMO.

In fact, my understanding of the problem statement is that you are
concerned that a user could supply inconsistent URL and content.

If there is a simple way to unify the flags, such that you only specify
the URL and then the program fetches the page using psiphon and also
another technique and compares that, I'd say it would be more Unix-ish
to expose this knob and let the user exercise her right to know best.

If that's not simple I am actually undecided.

I reckon that there may be use cases when an advanced ooniprobe user may
want to scan for a list of URLs that he/she is interested to using
Psiphon. So, if I would happen to be such user and the possiblity to
decide is removed, I'd probably be quite angry with the tool.

Maybe there are alternatives to that...

What about flagging the test as non-authoritative if the user has
supplied a non custom page, such that who analyzes the data is aware
that the test may be less reliable than others?

Or, what about disabling invoking the collector for such tests unless
the user forces the test to do so? (This seems more complex, tho.)

In the former case the person doing data analysis decide while in the
latter case it's the user who decides what to do. Maybe the most robust
thing is the former, unless there are concerns about resources usage.

Simone

-- 
Simone Basso
https://nexa.polito.it/people/sbasso

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 884 bytes
Desc: OpenPGP digital signature
URL: <http://lists.torproject.org/pipermail/ooni-dev/attachments/20160315/d689f453/attachment.sig>