- ooni-dev - lists.torproject.org

Network Meter Design Document
by poly 17 Mar '16

17 Mar '16

Dear oonitarians, Following the discussion during the last meeting, we have created a design document [0] that outlines the future plans for network meter. It's still very much a living document, and is updated often. Feel free to edit or add suggestions. [0] https://pad.riseup.net/p/network-meter -- poly @0xPoly https://darkdepths.net/pages/contact-keys.html

1 0

reports.ooni.torproject.org
by Vasilis 16 Mar '16

16 Mar '16

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA512 Hello ooni people, The official OONI reports webserver is up and running! Canonical location: https://reports.ooni.torproject.org The sanitized public reports hosted there are currently updated hourly. Hope this helps. ~Vasilis -----BEGIN PGP SIGNATURE----- iQIcBAEBCgAGBQJW6XQdAAoJEF+/cLHRJgFiwpkQAJWFxbZUtqQH8y1ntKkGnv3Q wLf68T5cH0YbGJvE2koGqIzTj2sAP/0wuUT1NhWmA6lHwPLh85giiC6EuUfJD1Zg JpjkclAlyL2ahNg4hgTjrPa1fSIgOaWj41LRZgAfJD/0yQJg572aa2HzcHVC0GLH 1IREIJwD2HEef1myOPWmDMzBiH4eKIFbqIYq2Ua+XzAesOBmErKdxslHcm5PAy0u tWkJfNFtnrgLNgr5g/fYGA8RVGktMQubqHak0TiMT6XV2nwXc9igDvFbZ2t8LlX9 KWUMIXLCgHDPOVx1iyEFlsyvyuqhwZZoffQvfeIjB1MNg4Gm29QXRkSF9sCNDbVA uIOHau7GdoXzl6z2glWpjShy7nAcnPU8x3H/ahNkEcaTDjLmwyVOddiCWkW/vJ6a +JVO5esXmG9qChLFo3x8lXhLiRhYntXOJzDSajZsK9PgcCBqXO0TwYaQNFDFZesW wXsKE/gwV+IZ2gb5vIsypmCV1so1zhjgVwtWNXEWTOecrCkgx8Z3pJDfGE5UJISb S2BIqvdHxCA3FYDCcR2n5mwmZt/M4ox60ix6CX7PV1rsC8yAUvnwn8nRwlH5hYt5 dH1KMvoAmWM7U6E63qxhQiq8eIRt2lpN127SnAbbj82srGPyPVJLbidn14IOJjMm g8iRbLDZDgzEPUk7hqmK =+lTK -----END PGP SIGNATURE-----

1 1

Exposing too many or too few options from ooni tests
by Arturo Filastò 15 Mar '16

15 Mar '16

Hello Oonitarians, I was having a discussion with vasilis about some changes that I recently did to the lantern and psiphon test around exposing some extra configuration options and would like to know what your opinion is on the matter. Basically these tests will try to run the lantern and psiphon tool, then attempt to connect to a certain website with it and verify if the response it gets from the website is the expected one. In the past it was possible to configure the URL to be fetched and the response body that is expected with a sane default. I have changed this to no longer be the case and instead use a hardcoded value of a website that we expect to not change in the future and an expected result for it. In the specific case it’s http://www.google.com/humans.txt The reason for doing this is that I want to avoid the possibility of a user misconfiguring the URL and expected body to something that is not true (I say foo.com results “bar” while it actually returns “foo”) and leading to inconsistent results. The argument against this is that the website we use for testing may change in the future and if we don’t notice then we can still have inconsistent results. To this I believe that even if that were to become the case it’s more likely that us developers of the tool will notice and hence ship an update than expect the user to tweak their ooniprobe to provide valid measurements. I believe that exposing some settings that can lead to measurements that are not true is sub-optimal, but I would like to hear contrasting opinions. ~ Arturo

5 4

Reminder for weekly OONI IRC meeting: Monday 2016-03-14, 17:00 UTC in #ooni
by Vasilis 14 Mar '16

14 Mar '16

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA512 Hello, This is a reminder for today's weekly OONI IRC meeting. It will happen as usual on the #ooni channel on irc.oftc.net at 17:00 UTC (18:00 CEST, 12:00 EST, 09:00 PST). Event Times around the world: https://www.timeanddate.com/worldclock/fixedtime.html?iso=20160314T17 Everybody is welcome to join us and bring their questions and feedback. ~Vasilis -----BEGIN PGP SIGNATURE----- iQIcBAEBCgAGBQJW5sJnAAoJEF+/cLHRJgFia4cP/A39Hp7AdLgHjck4m71zpSe5 4sz/5w4gOjsyVWmUBYsYIZ3UvzbL2OtIdO57DGahG7ayktdhKGDmdlJWQQPkgxsc eaDV194ioxDKm76fYmyPLUnrVsgr/TiVFK1mIwnoIKmT6EGvpv38LCiiZj8X/R6s M7WCIzgffb+F0waCAB3j1zr8SRmVWvNUdWQpRv+NmdEWG14AGvcBNVDxGdfmC0ir Nfznd7+iNreSj0KA9zUjlU9ivMyll/ivt7IKuiI3AW1iCWOF6tXTE0SIveRddx29 3nckvINThlLIlvXA/ErxAfinm84ED91+rilFbkE73h76GsOcNl7LI7JbV/8hlYdz qlNw+GYLmAeJSG6HXo8WuP0yRa60Plvej+VybrFZSMCXacSIsDqQ09RPBxKkhtV6 /Vp6TvYtaUg0hmF6DEsZwHD7a7KYFQn6NDkTlRabJC/OTjToEXWe6SFRWk1wHnpY g8t+xtUmmaxalp3vLcu7sdUTS0utGKti/qnSBSKiS0Dq1/gPsefedqUYv77KprVp Po8vr0TMtwxbRE6IZobAx/m3KmgJrrl5G2tD1mtg9L+NsnYbjRCJSqiQZZWameGw 6mr0vJqRw8gmOYH/4xUafOi2I0RsFXN3NxTTaYQ7znMk8qW5nk9fiAdEhjIIQxfA w+QfKvE0ig2DoGA1NUEC =a8Jm -----END PGP SIGNATURE-----

1 0

Re: [ooni-dev] Request for feedback on resuming of publishing of the reports
by Vasilis 12 Mar '16

12 Mar '16

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA512 On 11/03/16 18:52, Arturo Filastò wrote: > >> On Mar 11, 2016, at 20:15, Vasilis <andz(a)torproject.org> wrote: >> >> The web server hosting the updated reports is ready and can be >> found here: http://141.20.103.26:8080 >> >> I'm waiting for the DNS A record of'measurements.ooni.io' to be >> added so that I can setup a letsencrypt certificate. > > > $ dig A +short measurements.ooni.io 141.20.103.26 Thanks! Waiting for the administrator to raise the ACL rules on 443. Until then the reports will be served on http://measurements.ooni.io:808 0/ ~Vasilis -----BEGIN PGP SIGNATURE----- iQIcBAEBCgAGBQJW5CQlAAoJEF+/cLHRJgFiiH4P/3tMZbNLuMEqqIMvULDqDQ/j Una9RvZjGlQLkj+1ASk1SeklUj4kIDJJvba9nmc9Ts8oKAt5koiwB0OPagIIH75Y dMX41fVvEolqxBficJHrJljuThzXUMbxMFMyW+5S51esrKYHAK2dE3sYcq0U5akH xnuRFv2ZWk71v02DXsrU/3VeE0RNBbYD94trbxoERe1rlXWrWCMi+gqh8k3GU/pA b8kttunk5gvvviG0kz/12yVrcJTM0TQe2BJbKMHJpNx2Oz18mVCLpGsdq+b2hrlb dnx+6XBW9OHE1JVUN3HqTIuR2uAA0zz2OYCpJ958mxSl+2GJW3HilxeXDDYILWoz xbSBtdOhQ/1Z4SkZUWPVV4+8+rbhdPpLLE5X+K/WgGTVQKRGFRfG+93oZHAR8whE XbHZ5NdL0MTAOfI4w6GoDQwys07yJWd8MBKjZ+K7YwdCHQieEE8w4+Suvmx2BC6D Qsde0RV0f7OKt8cRK7JvkF2GrAffm46Pcd68hNnDfrYTMqDVyDHgurXjcfFYrOE+ FEFVRchzE9Nh/grOqUXvXhgKaWjjpCMEs3HzwkmncjPswuBBOUzG5Vuc7WiR6scg 9nVt0H6DLOaGIkqpGM8zVDljlF/R3U7O0wr1ehL+E9q88/jTVshCF+1U+hJbiUZT umoNjGsj5JNwY0QPdmSn =C5TQ -----END PGP SIGNATURE-----

1 0

Re: [ooni-dev] Request for feedback on resuming of publishing of the reports
by Tyler Fisher 11 Mar '16

11 Mar '16

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 Hi Arturo, - --- Currently for our data processing needs we have begun to bucket reports by date (every date corresponds to when a certain report has been submitted to the collector). What I would like to know is of the two following options what would be most convenient to you for accessing the data. The options are: OPTION A: Have 1 JSON stream for every day of measurements (either gzipped or plai n) ex. - https://ooni.torproject.org/reports/json/2016-01-01.json - https://ooni.torproject.org/reports/json/2016-01-02.json - https://ooni.torproject.org/reports/json/2016-01-03.json etc. OPTION B: Have 1 JSON stream for every ooni-probe test run and publish them inside of a directory with the timestamp of when it was collected ex. - - https://ooni.torproject.org/reports/json/2016-01-01/20160101T204732Z NL-AS3265-http_requests-v1-probe.json.gz - - https://ooni.torproject.org/reports/json/2016-01-01/20160101T204732Z US-AS3265-dns_consistency-v1-probe.json.gz etc. Since we are internally using the daily batches for doing the processing and analysis of reports unless there is an explicit request to publish them on a test run basis we will probably end up going for option A, so don’t be shy to reply :) - --- I agree with David in that it will be easier to access specific ooni-probe test results using option (B) (i.e. the current solution). What benefits did you identify when considering to switch to option (A)? A few reasons to stick with option (B) include: - - Retaining the ability to run ooni-pipeline on a subset of reports associated with a given time period by filtering by date prefix, and substrings within key names; - - Retaining the ability to distribute small units of work easily among subprocesses; and - - Retaining the idempotent nature of ooni-pipeline, and the luigi framework - switching from lots of small files to a single large file for a given day will invariably increase the time required to recover from failures (i.e. if a small dnst-based test fails to normalise, you'll have to renormalise everything as opposed to a single test; - - Developers will not have to download hundreds of megabytes of data in order to access a traceroute test result that is only a few kilobytes in size; and - - It's generally easier to work with smaller files than it is to work with big files. Cheers, Tyler GPG fingerprint: 8931 45DF 609B EE2E BC32 5E71 631E 6FC3 4686 F0EB (tyler(a)tylerfisher.org) -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iQIcBAEBCAAGBQJWn1+qAAoJEGMeb8NGhvDr6hUP/R6XcXEejwT8DYuKLoVBpujs CqXtIj88A5JYhtt1npRF/a0peNihzFRbYuQpAUX/D1EdPa1UHDwCuqp1hO642xIJ WePgWHWIS7qzYK/i5LbMXC+oWmfAA0J25SawmjyNWclK+NCgIwQ1k7kleyFP7Ul5 PFjJKLCcuqJkQl1hnZlW7YhgLYZAf2QHOD1cJauLM5aDCNBDUSgfIP+/P/xfFLq3 XqLGFBfNrMXaWmOfDGLR7tV4mS3R4M5L7rL66AiomQULdld4cuLonAht4CWLDuhV MKyKrURixRqgTUoing59OcjgOcGEVQD5P5NaMuVruU1hFbHW0wr430/mEq8pdDW9 BQHZh/VZ/f2xz4rjWiE8Mfl3mgmGfbFiT6WMKQTRY3vr5mwbmefg0/IneJ1eHtIo A/XMX579DQt3V19tMa7rO4TjpdBKIWwJ8/6mwwaw9QrS/I2pmlg8AscLU0oQtMqc 3CcWELOdoV7uIPBVg3TfiL+RLDSxzIJIp0k6IM19tkwZAxGLmD+cZvDo+dxME3fd Y7+fYxovuQrt4vvhaPFU15EDzFWHMoMqlNUSOeC0FuIhpbYbM0Dqn1qL4EScJ+PA +p/rtMTZLiiIJLtYuZCRiZjbaMvqsfZAmPEZ5ZgSShJKsB4jhV4/5LsmHedYCHeW S/IdJ81xUzTfxMUZBqgI =kztT -----END PGP SIGNATURE-----

4 5

report_ids changed since conversion to JSON?
by David Fifield 11 Mar '16

11 Mar '16

Did report_ids get regenerated when reports were converted from YAML to JSON? I think they did but I want to make sure. For example, I have a copy of the old YAML report 20140428T232415Z-AS1241-http_requests_test-v1-probe.yaml.gz. It has a report_id ending in "zuj": report_filename: 20140428T232415Z-AS1241-http_requests_test-v1-probe.yaml report_id: 2014-04-28aqfgmdfzxjwmreodmroptzeugvanvtznhclirzuj The new corresponding JSON file seems to be https://ooni-public.s3.amazonaws.com/json/2014-02-28.json (requires an access key). Its report_id instead ends in "ois": "report_filename": "20140428T232415Z-AS1241-http_requests_test-v1-probe.yaml", "report_id": "2014-04-28nltgjbivffrtkqsvpoaudhgsgyafkbfldqncrois"

2 2

Reminder for weekly OONI dev gathering Monday 2016-02-15 17:00 UTC (18:00 CEST)
by Arturo Filastò 15 Feb '16

15 Feb '16

Hello Oonitarians, This is a reminder that today there will be the weekly OONI gathering. It will happen as usual on the #ooni channel on irc.oftc.net at 17:00 UTC (18:00 CEST, 12:00 EST, 09:00 PST). You can join via the web from: https://kiwiirc.com/client/irc.oftc.net/ooni (Note: sometimes Tor is blocked by OFTC, but it should mask your IP if you trust that stuff). Everybody is welcome to join us and bring their questions and feedback. See you later, ~ Arturo

1 0

Reminder for weekly OONI dev gathering Monday 2016-02-08 17:00 UTC (18:00 CEST)
by Arturo Filastò 08 Feb '16

08 Feb '16

Hello Oonitarians, This is a reminder that today there will be the weekly OONI gathering. It will happen as usual on the #ooni channel on irc.oftc.net at 17:00 UTC (18:00 CEST, 12:00 EST, 09:00 PST). You can join via the web from: https://kiwiirc.com/client/irc.oftc.net/ooni (Note: sometimes Tor is blocked by OFTC, but it should mask your IP if you trust that stuff). Everybody is welcome to join us and bring their questions and feedback. See you later, ~ Arturo

1 0

OONI team progress report for January 2016
by Arturo Filastò 03 Feb '16

03 Feb '16

# What we did in January 2016 * Merge the work done by Tyler on his version of the pipeline into our version of it. * Progress towards finalising the ooni-explorer. An MVP is now implemented and can be viewed at http://explorer.ooni.io/ (merge the pending PRs into master and scope of out the remaining tasks) * Setup redundant copy of the pipeline on the humboldt server * Start review of the ooni data formats to reflect the normalisations done in the pipeline: https://github.com/TheTorProject/ooni-spec/pull/49 * Start writing specification for the new test that should replace http_requests, dns_consistency and tcp_connect called “web_connectivity” https://github.com/TheTorProject/ooni-spec/pull/48 * Start implementing JSON support for the backend: https://github.com/TheTorProject/ooni-backend/pull/62 * Start implementing JSON support in ooni-probe: https://github.com/TheTorProject/ooni-probe/pull/456 * Start implementing the web_connectivity test helper in the backend: https://github.com/TheTorProject/ooni-backend/commit/c8bdf9b31ede6eb5551033… * Start implementing the web_connectivity test in ooni-probe: https://github.com/TheTorProject/ooni-probe/pull/457 * Progress on measurementkit to integrate traceroute test from portolan * Progress on measurementkit to support starting of tor * Ran the following weekly gatherings: http://meetbot.debian.net/ooni/2016/ooni.2016-01-04-16.59.html http://meetbot.debian.net/ooni/2016/ooni.2016-01-18-17.01.html http://meetbot.debian.net/ooni/2016/ooni.2016-01-25-17.00.html # What we plan on doing in February 2016 * Continue working on the web_connectivity test * Continue work on the web_connectivity backend * Restore the publishing of reports in the new data-format ~ Arturo

1 0