July 2014 - ooni-dev - lists.torproject.org

Bridge reachability results
by Arturo Filastò 11 Aug '14

11 Aug '14

As promised I published the bridge reachability measurements on the public ooni report hosting. You can find them here: https://ooni.torproject.org/reports/0.1/CN/ https://ooni.torproject.org/reports/0.1/RU/ https://ooni.torproject.org/reports/0.1/US/ Keep in mind that, as I was telling you, during some of the runs there were some issues with the measurements due to incompatibilities of ooniprobe with the old fedora version running on planetlab, so not all the measurements may be 100% accurate. They should still, at least, give you an idea of how the data format looks like and if it contains enough information for doing your parsing work. I would suggest we keep this discussion public and maintain the ooni-dev list in cc. ~ Art.

3 4

Detecting which Test Helpers are Running
by Taylor Hornby 02 Aug '14

02 Aug '14

Hello, We are working on scripts to create the Ooni bouncer configuration from multiple instances of Ooni running on M-Lab slices. So far, we have two scripts. One, which collects the portion of the bouncer.yaml out of an Ooni slice, and another script, which combines all of the parts into a single bouncer.yaml: https://github.com/LeastAuthority/ooni-support/blob/mlab-ns-simulator/bounc… https://github.com/LeastAuthority/ooni-support/blob/mlab-ns-simulator/bounc… One thing we would like the "getconfig.py" script to do is to dynamically determine which Test Helpers are enabled on the slice. I'm not sure how to do this, because the oonib.conf always contains entries for all the test helpers. So my question is this: How can we determine which Test Helpers are enabled by examining the oonib.conf or other files? Or, is this question misguided? Are *all* test helpers supposed to be in the bouncer.yaml file, even ones which are not running, and we don't want to have running? (For the first M-Lab deployment, there will be only one Test Helper, the http-return-headers one, so we could hard-code that one into the script, but we would like to make this script keep working when new tests are enabled). Thanks, -- Taylor Hornby

2 1

mlab-ns integration testing and easy-disable-feature.
by Nathan Wilcox 02 Aug '14

02 Aug '14

Dear Oonitarians, We're about to start testing some mlab-ns integration features on the M-Lab test slice. This will be potentially disruptive because we'll be reinitializing the slice, installing new bits and bobs, and we'll be using our own fork of ooni-support [1] rather than upstream ooni-support [2]. Please let us know if we should *not* do this soon. Today I planned to only poke mlab1.nuq0t.measurement-lab.org [3], but eventually we want to test at least two test slivvers. Meanwhile, what happens if we deploy some integration glue and it doesn't work, or has unanticipated problems in the future, or M-Lab policies change, etc? To anticipate those kinds of issues, we want to make sure that it's easy to toggle off the integration stuff. One of our key design goals has been to avoid modifying upstream ooni. We've just realized a new and related design goal: make sure it's easy to turn off the integration glue without requiring slice re-initialization, packages changes, repository changes, etc... So we just added a ticket for that [4]. Note that *while* we are doing this interactive testing, the two slivvers we're using will be pretty unstable, but *after* we reach a stable level of development, we should be able to flip the integration switch to off and the M-Lab test slice should operate as before (except with some extra stuff installed). If you have any concerns / feedback / comments, etc... feel free to email us, or respond to specific tickets. All of the tickets related to the mlab-ns integration in particular should live in the `ooni-support` issue tracker with the `LeastAuthority D` label [5]. Regards, Nathan links: [1] https://github.com/LeastAuthority/ooni-support [2] https://github.com/m-lab-tools/ooni-support [3] https://github.com/m-lab-tools/ooni-support/issues/50#issuecomment-50060748 [4] https://github.com/m-lab-tools/ooni-support/issues/54 [5] https://github.com/m-lab-tools/ooni-support/issues?direction=desc&labels=Le… -- Nathan Wilcox Least Authoritarian email: nathan(a)leastauthority.com twitter: @least_nathan PGP: 11169993 / AAAC 5675 E3F7 514C 67ED E9C9 3BFE 5263 1116 9993

2 1

http_requests test single responses
by jony-port＠bitmessage.ch 24 Jul '14

24 Jul '14

Hi, It seems that the single response of the http_requests requests test results in 'body_length_match' and 'headers_match' fields set to true. From my understanding a single response request should raise an error set by the 'control_failure' or 'experiment_failure' field. An example of the described issue: --- agent: agent body_length_match: true body_proportion: 1.0 control_failure: null experiment_failure: null factor: 0.8 headers_diff: !!set {} headers_match: true input: http://beemp3.com/ requests: - request: body: null headers: - - User-Agent - ['Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6'] method: GET tor: {is_tor: false} url: http://beemp3.com/ response: body: "<html>\r\n<head><title>301 Moved Permanently</title></head>\r\n<body bgcolor=\"\ white\">\r\n<center><h1>301 Moved Permanently</h1></center>\r\n<hr><center>nginx/1.2.7</center>\r\ \n</body>\r\n</html>\r\n" code: 301 headers: - - Content-Length - ['184'] - - Server - [nginx/1.2.7] - - Connection - [close] - - Location - ['http://beemp3s.org/'] - - Date - ['Tue, 17 Jun 2014 20:32:43 GMT'] - - Content-Type - [text/html] socksproxy: null ... ---

1 0

Fwd: Ooni / M-Lab Deployment Automation Script
by Nathan Wilcox 17 Jul '14

17 Jul '14

Hi ooni-dev. For your viewing pleasure, here is a forward about tickets related to deploying M-Lab on Ooni (without integration into mlab-ns). We'll send these announcements directly to ooni-dev henceforth. Enjoy. ---------- Forwarded message ---------- From: Taylor Hornby <taylor(a)leastauthority.com> Date: Wed, Jul 16, 2014 at 2:42 PM Subject: Ooni / M-Lab Deployment Automation Script To: Liz Pruszko Steininger <steiningerl(a)rfa.org>, Dan Meredith <meredithd(a)rfa.org>, lynna(a)rfa.org, Roger Dingledine <arma(a)mit.edu>, Arturo Filastò <art(a)torproject.org>, Meredith Whittaker <meredithrachel(a)google.com>, Will Hawkins <hawkinsw(a)opentechinstitute.org>, Jordan McCarthy <mccarthy(a)opentechinstitute.org>, critzo(a)opentechinstitute.org Cc: "consultancy(a)leastauthority.com" <consultancy(a)leastauthority.com>, taylor(a)leastauthority.com, Zooko Wilcox-OHearn <zooko(a)leastauthority.com>, Jessica Augustus <jessica(a)leastauthority.com>, Nathan Wilcox <nathan(a)leastauthority.com> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Dear OTF, Ooni, and M-Lab, We've finished our work for Milestone C. This milestone is about writing a script for automating the process of deploying Ooni to M-Lab slices. Since such a script had already been written before we arrived, we shifted our goals for this milestone as follows: 1. Usability and reliability testing of the existing deployment automation scripts. 2. Fix any issues that we identified during that process. Also part of Milestone C is the credential rotation deliverable, which is no longer relevant because the mechanism for distributing .ooni addresses has changed since the contract was negotiated. This is documented in the following ticket: https://github.com/m-lab-tools/ooni-support/issues/32 As part of the first (new) goal, we ran through a deployment several times using the scripts, which is documented in this ticket: https://github.com/m-lab-tools/ooni-support/issues/17 The issues we encountered are summarized in this umbrella ticket: https://github.com/m-lab-tools/ooni-support/issues/21 Each issue was split out into separate tickets: #23: Fix or document deployment gotcha of deleting $HOME https://github.com/m-lab-tools/ooni-support/issues/23 #24: Specify dependency on yum-cron for installation. https://github.com/m-lab-tools/ooni-support/issues/24 #25: Missing ``/etc/mlab/slice-functions`` https://github.com/m-lab-tools/ooni-support/issues/25 #26: Add root uid documentation and check in initialize.sh ... https://github.com/m-lab-tools/ooni-support/issues/26 #27: Fix initialize.sh to create ``/var/spool/mlab_ooni`` https://github.com/m-lab-tools/ooni-support/issues/27 #29: Ensure test_helpers can be reached from the public internet https://github.com/m-lab-tools/ooni-support/issues/29 #28: ``stop.sh`` failed to stop multiple processes. https://github.com/m-lab-tools/ooni-support/issues/28 #40: Make openssl an explicit dependency of the Ooni RPM https://github.com/m-lab-tools/ooni-support/issues/40 #12641: IStreamClientEndpointStringParser is Deprecated https://trac.torproject.org/projects/tor/ticket/12641#ticket #41: Install service_identity https://github.com/m-lab-tools/ooni-support/issues/41 #42: prepare.sh violates ooni-backend's README instructions https://github.com/m-lab-tools/ooni-support/issues/42 #44: Is dependency installation vulnerable to MITM attacks? https://github.com/m-lab-tools/ooni-support/issues/44 All of these tickets, with the exception of #40, #12641, #41, #42, and #44 are now closed. Ticket #40 is a minor issue, but would involve significant design decisions on M-Lab's part, so we left it open for M-Lab to close. Ticket #12641 is about the use of a deprecated function in Ooni, to be fixed by the Ooni team. Ticket #42 is about a missing dependency in Ooni for the Ooni team to fix. Ticket #44 is about a security vulnerability that requires Ooni collaboration to resolve (see below). We also found a new security vulnerability in Ooni: #12642: Can Network Attacker Downgrade Dependency Install Security? https://trac.torproject.org/projects/tor/ticket/12642#ticket Our fixes to the issues are contained in three pull requests: #36: Improvements to the README.md. https://github.com/m-lab-tools/ooni-support/pull/36 #37: Improvements to the initialize.sh script. https://github.com/m-lab-tools/ooni-support/pull/37 #43: Install dependencies according to ooni-backend README https://github.com/m-lab-tools/ooni-support/pull/43 Note that pull request #36 contains work from Milestone B as well. Please let us know if you have any suggestions, questions, or concerns. - -- Taylor Hornby Least Authoritarian Email: taylor(a)leastauthority.com PGP: CE3 F8ED D999 F066 C2E2 9124 F6D4 D32C E31C 99FE Twitter: @DefuseSec -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iQIcBAEBAgAGBQJTxvB6AAoJEPbU0yzjHJn+ccQQALHndy9a7kuz9MDifXrS+z2s uzzizfUK5EZB12G+mFaAfqF/t8pa/zcD2mZ2ycpna8AruhZPH5x9poxoZI/Agz59 gb8xlaJMwOJWFmeBHkn60Jz/zyaVZF0xTkQ8YhGKeqzXkfo1Vp+EI0ZFcanLKIvZ EaL+zHPZNyb5SQXOTiiy9OpyCXhboNOaXQru9GgxvBYJFosEeKA6aLVVyPx2ZSci irBg0KNt8jCkPQtH5YjkCrjKwjNI40niBpVU3B/jz5CvMb4f5B08ZjqL7t+Hhpul /c9dbYV7VILkq2/Q1/G5SNiosl8SUkjf3U8hDmb0pQpMeoZ/aE9V3AWDCrcABNvD dbJF9K3FD2YRrRjCBPNO0KWxXCU3X45oc58JAQbOuHbH6AVPazZB9WRgdu1pAisv Ikidl1yovoqxJkN3iEybfX3I2p1geMrDB4Q/z7FOdRP2dBNzTKR7zkTvJdXyulZf q1yI+Qav7MVQBGdCN87jX8xtt1eUXMEQXu7TVcxcNlvfgea5Uewv9s5l2/84fYa3 qu0Kp/+8BOioXIbG09PJREHzoHEeNSJvLqF7B6d5r3enBv5H0YvC194s8wjkZGTz sQBsAl4HI+7xEdeQ44vez+SV11i9NkEyHo1rwqh4T4glM8yXcdQ4buZaMwcXJ2V7 0UKWa6Sj2n563Dclb47K =RS7C -----END PGP SIGNATURE----- -- Nathan Wilcox Least Authoritarian email: nathan(a)leastauthority.com twitter: @least_nathan PGP: 11169993 / AAAC 5675 E3F7 514C 67ED E9C9 3BFE 5263 1116 9993

2 2

Fwd: Ooni Release / M-Lab Deployment Specifications
by Arturo Filastò 12 Jul '14

12 Jul '14

Forwarding this email to the public ooni-dev mailing list for posterity. -------- Original Message -------- Subject: Ooni Release / M-Lab Deployment Specifications Date: Thu, 10 Jul 2014 08:59:47 -0700 From: Nathan Wilcox <nathan(a)leastauthority.com> To: Liz Pruszko Steininger <steiningerl(a)rfa.org>, Dan Meredith <meredithd(a)rfa.org>, Adam Lynn <lynna(a)rfa.org>, Roger Dingledine <arma(a)mit.edu>, Arturo Filastò <art(a)torproject.org>, Meredith Whittaker <meredithrachel(a)google.com>, Will Hawkins <hawkinsw(a)opentechinstitute.org>, Jordan McCarthy <mccarthy(a)opentechinstitute.org> CC: consultancy(a)leastauthority.com <consultancy(a)leastauthority.com>, Taylor Hornby <taylor(a)leastauthority.com>, Zooko Wilcox-OHearn <zooko(a)leastauthority.com>, Jessica Augustus <jessica(a)leastauthority.com> (Note: There's an attached pdf which has the same content as the following message, in case the inline markup is too cumbersome for your taste.) Dearest OTFundies, Torish Ooniites, and M-Labysians, Summary ======= We've finished `Milestone B` of our `Ooni Release Engineering` contract with `Open Technology Fund`_, which is concerned with spelling out a clear release and deployment process for both M-Lab people and Tor people. This is the second out of six milestones. .. _`Open Technology Fund`: https://www.opentechfund.org/ Release & Deployment Documentation ================================== The heart of the work is in `our ooni-support README.md edits`_ to `the official ooni-support repository`_ (see `ooni-support Pull Request 36`_). Our hope is that someone new to the project who *does not interact closely* with the Ooni dev team can still deploy successfully on the M-Lab slice, which we believe makes the overall deployment more robust and flexible. .. _`our ooni-support README.md edits`: https://github.com/LeastAuthority/ooni-support/blob/improve-readme-add-ooni… .. _`the official ooni-support repository`: https://github.com/m-lab-tools/ooni-support/ .. _`ooni-support Pull Request 36`: https://github.com/m-lab-tools/ooni-support/pull/36 We were glad to see that there is already a well written `Ooni Release Procedure`_ for distributing well tested and packaged Ooni releases. This specification includes a link to test coverage, package signing, and target platforms. We've suggested two improvements in a `versioning specification ticket`_ and a `release planning specification ticket`_. Additionally, this document alludes to M-Lab acceptance criteria but there is no link, so we've documented that need in `a ticket for M-Lab acceptance criteria`_. .. _`Ooni Release Procedure`: https://github.com/TheTorProject/ooni-spec/blob/master/Release-Procedure.md .. _`versioning specification ticket`: https://trac.torproject.org/projects/tor/ticket/12586#ticket .. _`release planning specification ticket`: https://trac.torproject.org/projects/tor/ticket/12587#ticket .. _`a ticket for M-Lab acceptance criteria`: https://github.com/m-lab-tools/ooni-support/issues/33 Collaboration ============= All of our effort for this `Milestone B` is embodied in issue tickets or pull requests in these places: * the main Tor issue tracker `Ooni component tickets`_, and a narrower query for `only issues filed by Least Authority`_, and * and `the official ooni-support repository`_. .. _`Ooni component tickets`: https://trac.torproject.org/projects/tor/query?status=accepted&status=assig… .. _`only issues filed by Least Authority`: https://trac.torproject.org/projects/tor/query?status=accepted&status=assig… For example, here are `the tickets in ooni-support for Milestone B`_ which this email summarizes. .. _`the tickets in ooni-support for Milestone B`: https://github.com/m-lab-tools/ooni-support/issues?labels=LeastAuthority+B&… Ticketing Conventions --------------------- We've adopted several conventions in `the official ooni-support repository`_ issue tracker which the M-Lab and Ooni engineers should be aware of. We've added labels for our contractual milestones of the form ``LeastAuthority X`` where X is ``A - F``, such as `the tickets in ooni-support for Milestone B`_. It's important to distinguish the Least Authority contractual milestones, which these labels are associated with, from the `Github Milestone` feature. We use the latter for general engineering goals or topical areas, as seen in `the ooni-support github milestone list`_. .. _`the ooni-support github milestone list`: https://github.com/m-lab-tools/ooni-support/issues/milestones We've just started adding labels to indicate requests or blocking on M-Lab or Ooni input. Currently there is only `the M-Lab label`_ but we have not yet updated older tickets with this new convention. The ``LA - Close Me`` label is for our own (internal) review process. .. _`the M-Lab label`: https://github.com/m-lab-tools/ooni-support/issues?labels=M-Lab&milestone=&… Let us know if you have any suggestions, concerns, or questions.

1 0

OONI status report June 2014
by Arturo Filastò 08 Jul '14

08 Jul '14

# ooni status report June 2014 # Activities of June 2014 * Annotate on disk reports that have not been submitted to a collector. https://trac.torproject.org/projects/tor/ticket/11860 * Write tool for manually or automatically submitting reports to collectors. https://trac.torproject.org/projects/tor/ticket/11862 * Implement DNS test helper that can be used to discover the DNS resolver being used by a probe. https://gitweb.torproject.org/ooni-backend.git/commit/df43691bedae4e5024aa3… * Bugfixes related to test decks support: https://gitweb.torproject.org/ooni-probe.git/commit/a9a24464b4b61800fe57874… * Attend the tor developer meeting where we had a fruitful discussion on how to structure a network interference consortium. I will be doing a writeup on what came out of that session in the upcoming weeks. # Activities for July 2014 * Work with the BridgeDB team to do bridge reachability for rapid response to Tor blocking in specific countries. The master ticket for keeping track of this work can be found here: https://trac.torproject.org/projects/tor/ticket/12544 * Design a test deck to be run inside of Iran. * Make descriptive tickets of all that was discussed during the tor dev meeting. Given the fact that I am going to be quite busy during most of July this is all that I can commit to doing. I may be able to get some extra things done, but this is the minimum.

1 0