Exposing too many or too few options from ooni tests
Hello Oonitarians, I was having a discussion with vasilis about some changes that I recently did to the lantern and psiphon test around exposing some extra configuration options and would like to know what your opinion is on the matter. Basically these tests will try to run the lantern and psiphon tool, then attempt to connect to a certain website with it and verify if the response it gets from the website is the expected one. In the past it was possible to configure the URL to be fetched and the response body that is expected with a sane default. I have changed this to no longer be the case and instead use a hardcoded value of a website that we expect to not change in the future and an expected result for it. In the specific case it’s http://www.google.com/humans.txt The reason for doing this is that I want to avoid the possibility of a user misconfiguring the URL and expected body to something that is not true (I say foo.com results “bar” while it actually returns “foo”) and leading to inconsistent results. The argument against this is that the website we use for testing may change in the future and if we don’t notice then we can still have inconsistent results. To this I believe that even if that were to become the case it’s more likely that us developers of the tool will notice and hence ship an update than expect the user to tweak their ooniprobe to provide valid measurements. I believe that exposing some settings that can lead to measurements that are not true is sub-optimal, but I would like to hear contrasting opinions. ~ Arturo
On 15/03/16 18:18, Arturo Filastò wrote:
Hello Oonitarians,
I was having a discussion with vasilis about some changes that I recently did to the lantern and psiphon test around exposing some extra configuration options and would like to know what your opinion is on the matter.
Basically these tests will try to run the lantern and psiphon tool, then attempt to connect to a certain website with it and verify if the response it gets from the website is the expected one.
In the past it was possible to configure the URL to be fetched and the response body that is expected with a sane default.
I have changed this to no longer be the case and instead use a hardcoded value of a website that we expect to not change in the future and an expected result for it. In the specific case it’s http://www.google.com/humans.txt
The reason for doing this is that I want to avoid the possibility of a user misconfiguring the URL and expected body to something that is not true (I say foo.com results “bar” while it actually returns “foo”) and leading to inconsistent results.
The argument against this is that the website we use for testing may change in the future and if we don’t notice then we can still have inconsistent results.
To this I believe that even if that were to become the case it’s more likely that us developers of the tool will notice and hence ship an update than expect the user to tweak their ooniprobe to provide valid measurements.
I believe that exposing some settings that can lead to measurements that are not true is sub-optimal, but I would like to hear contrasting opinions.
Let's see if I can end up with a contrasting opinion... So, in general I tend to think that users may know better than me and so I am in favour of providing knobs to twist my programs behavior. What I find optimal behavior is when a tool prevents me from doing stupid things but at the same time allows me to --force it. But in this case a strange thing happens. Usually command line flags should be orthogonal but this is not the case, IMO. In fact, my understanding of the problem statement is that you are concerned that a user could supply inconsistent URL and content. If there is a simple way to unify the flags, such that you only specify the URL and then the program fetches the page using psiphon and also another technique and compares that, I'd say it would be more Unix-ish to expose this knob and let the user exercise her right to know best. If that's not simple I am actually undecided. I reckon that there may be use cases when an advanced ooniprobe user may want to scan for a list of URLs that he/she is interested to using Psiphon. So, if I would happen to be such user and the possiblity to decide is removed, I'd probably be quite angry with the tool. Maybe there are alternatives to that... What about flagging the test as non-authoritative if the user has supplied a non custom page, such that who analyzes the data is aware that the test may be less reliable than others? Or, what about disabling invoking the collector for such tests unless the user forces the test to do so? (This seems more complex, tho.) In the former case the person doing data analysis decide while in the latter case it's the user who decides what to do. Maybe the most robust thing is the former, unless there are concerns about resources usage. Simone -- Simone Basso https://nexa.polito.it/people/sbasso
The model for test parameters has been a bit confusing to me. I think it's because there are two different audiences that we're trying to cater to. 1. End users. For them, there should be good defaults, like the hard-coded expected value here. In order to run the test, a user really shouldn't have to think about anything, and something reasonable should happen. In the same way, we should get to a point where we automatically fetch a reasonable test deck and run reasonable tests for users who just want to understand or report on what's going on without learning the intricacies of ooni. 2. OONI developers / researchers debugging interference mechanisms. Here, someone is trying to dig deeper into what's going on through interactive testing. In this case, it seems like it would be plausible that they might want to have hooks exposed for changing how tests work, especially across a bunch of remote probes, without needing to redeploy changed code. This specific hook seems like it wouldn't commonly need to be changed, but I could imagine someone wanting to re-use this test to see which domains special-case lantern or if there are issues with lantern requesting back into a country by modifying this parameter. My gut feeling is that the easiest way to serve both audiences is to leave a way to explicitly set the parameter, but focus on good, hard-coded defaults that we expect to be used for collected reports. --Will On Tue, Mar 15, 2016 at 10:18 AM, Arturo Filastò <art@torproject.org> wrote:
Hello Oonitarians,
I was having a discussion with vasilis about some changes that I recently did to the lantern and psiphon test around exposing some extra configuration options and would like to know what your opinion is on the matter.
Basically these tests will try to run the lantern and psiphon tool, then attempt to connect to a certain website with it and verify if the response it gets from the website is the expected one.
In the past it was possible to configure the URL to be fetched and the response body that is expected with a sane default.
I have changed this to no longer be the case and instead use a hardcoded value of a website that we expect to not change in the future and an expected result for it. In the specific case it’s http://www.google.com/humans.txt
The reason for doing this is that I want to avoid the possibility of a user misconfiguring the URL and expected body to something that is not true (I say foo.com results “bar” while it actually returns “foo”) and leading to inconsistent results.
The argument against this is that the website we use for testing may change in the future and if we don’t notice then we can still have inconsistent results.
To this I believe that even if that were to become the case it’s more likely that us developers of the tool will notice and hence ship an update than expect the user to tweak their ooniprobe to provide valid measurements.
I believe that exposing some settings that can lead to measurements that are not true is sub-optimal, but I would like to hear contrasting opinions.
~ Arturo
_______________________________________________ ooni-dev mailing list ooni-dev@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/ooni-dev
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA512 Hello list, On 15/03/16 14:18, Arturo Filastò wrote: [...]
Basically these tests will try to run the lantern and psiphon tool, then attempt to connect to a certain website with it and verify if the response it gets from the website is the expected one.
In the past it was possible to configure the URL to be fetched and the response body that is expected with a sane default.
I have changed this to no longer be the case and instead use a hardcoded value of a website that we expect to not change in the future and an expected result for it. In the specific case it’s http://www.google.com/humans.txt
I believe we can have both! By using default parameters -sane default we are allowing the tests to run without requiring extra arguments. This satisfies: - - Users that just want to run the test without any parameters set. Either because they are not aware of the "correct" parameters or they don't want to do any further tests rather than just check if lantern or psiphon bootstraps and works. - - Users that (probably) know what they do and would like to change the default (sane default) parameters. They would like to test if other URLs are reachable via lantern or psiphon or trigger different responses or do something else without doing changes into the code. Does this makes more sense? Just for reference the current sane default is to check the URL: http://www.google.com/humans.txt ~Vasilis -----BEGIN PGP SIGNATURE----- iQIcBAEBCgAGBQJW6HZBAAoJEF+/cLHRJgFiOqcQAK/snzWTv9RooeYxqgx5YnNN XMFj5zLs6en4ZcxQmmfNDG3dibiKR9ee2x82fYu8UC/UGDoWWMqy5TX5YiuqDLfD Ty/EE7ZeWNDubkXr+IhwcwImYAI6T7ANXNdGMDMon2TLCr5M6rcOlAQ4JDzm9X/Q P0jMbvWSAw0W+8o6oDS11arE1jyp68Xk/5m+MlpfdpjTfMYbuhPQcoXAfecYCJ93 I4ellb5efOKU0F3HHupag9RgNWMu9oJBn1GHz/tdNGiD/B3ECXkLQ531REs4xrhv /E1GLF8DG6p8NSWvj4zvv+Cf5xnjwTm5o+FUo0MNzo1XRQnTFw3QLT9rc1jIjwr0 +s7WP+yXlCo+uAqME4zCOkj8qNRkC15ByhKa+h0KcuUQpxmwSfT1spZqr15WFH7k 18x3YESLOOFCcvngx2Z6ZlGJmD+RcuD/0alTTXucBdz1vVxorh4tFViXoLfYwqj/ 0xX2tK33YrltB80MoqVt4fKApKRqIGSrMxkRfc+Z5fSFsAtyn5l0n1zdHriigU0Y kcfxr/bh/+KMlDZ7yRiVl91N+DR2j3v3bDQfAsTg/RZcd4aIvb1U3Kpjq6wOqB15 BUmfdKyR09QOnBO4ybqrEn3MQuPiwi3iD4GJ+XmNBqXOgyWowLd19psh/DVt9dcf ptV7ucaFznRH+nKdiaA2 =mf2H -----END PGP SIGNATURE-----
My opinion is that it would be best if we exposed such functionality, but disabled sending over the results of any such custom tests. I think we implicitly offer some guarantee that the metrics that we display publicly are accurate to a reasonable extent. This way the measurement records are kept accurate and users are free to run the tests as they wish. On Tue, 2016-03-15 at 18:18 +0100, Arturo Filastò wrote:
Hello Oonitarians,
I was having a discussion with vasilis about some changes that I recently did to the lantern and psiphon test around exposing some extra configuration options and would like to know what your opinion is on the matter.
Basically these tests will try to run the lantern and psiphon tool, then attempt to connect to a certain website with it and verify if the response it gets from the website is the expected one.
In the past it was possible to configure the URL to be fetched and the response body that is expected with a sane default.
I have changed this to no longer be the case and instead use a hardcoded value of a website that we expect to not change in the future and an expected result for it. In the specific case it’s http://www.google.com/humans.txt
The reason for doing this is that I want to avoid the possibility of a user misconfiguring the URL and expected body to something that is not true (I say foo.com results “bar” while it actually returns “foo”) and leading to inconsistent results.
The argument against this is that the website we use for testing may change in the future and if we don’t notice then we can still have inconsistent results.
To this I believe that even if that were to become the case it’s more likely that us developers of the tool will notice and hence ship an update than expect the user to tweak their ooniprobe to provide valid measurements.
I believe that exposing some settings that can lead to measurements that are not true is sub-optimal, but I would like to hear contrasting opinions.
~ Arturo
_______________________________________________ ooni-dev mailing list ooni-dev@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/ooni-dev -- poly @0xPoly https://darkdepths.net/pages/contact-keys.html
participants (5)
-
Arturo Filastò -
poly -
Simone Basso -
Vasilis -
Will