August 2014 - ooni-dev - lists.torproject.org

Ooni / M-Lab integration.
by Nathan Wilcox 05 Aug '14

05 Aug '14

Dear OTF, Ooni, and M-Lab, Summary ======= We've hashed out a design to integrate Ooni with mlab-ns on the M-Lab deployment, and we've implemented a fully functional deployment that approximates this by simulating mlab-ns (this is attached). This completes Milestone D of our contract with OTF. Design Goals ============ Our top goals for this integration are: It does not rely on any changes to upstream Ooni. (For example, probes still use a bouncer .onion, and the backend has stock bouncers, collectors, and test helpers running.) It can be disabled easily without redeploying the M-Lab backend. Our branch's ooni-support README.md has instructions to disable the integration, merely by editing a cron job to unset an ENABLED flag. There's no need to redeploy different versions of ooni-support. When enabled, it allows M-Lab operations to monitor collectors and test_helpers status with the same infrastructure as all other M-Lab tools. Future Architectural Changes ---------------------------- In the future, it may be nice to augment ooni / mlab-ns integration. For example, mlab-ns is designed to support different policies which may be useful to tools, such as geo-location of test_helpers. The Simulator ============= This deployment architecture uses a simulator. While it is fully functional and useful for testing it lacks security or robustness, so we want to emphasize *not to deploy this* to non-test environments. Rationale --------- There are three rationales for this approach: First, Least Authority didn't want to push through modifications to mlab-ns without first creating and testing a proof-of-concept. Second, we didn't want to block our effort on M-Lab engineering effort, so this allows a clean division of labor. Third, by creating and testing a working proof of concept we can help define the necessary changes to mlab-ns in a tightly scoped and concrete manner. Security -------- This system is insecure because it does not use the M-Lab nagios system to gather data, and instead lets anyone paste any data they want into the simulator. Nagios integration is future work captured in this ticket: * https://github.com/m-lab-tools/ooni-support/issues/10 Next Steps ========== Our contract with OTF proposes our next two milestones will focus on improving integration testing and unit test coverage. Our focus at that time was on test automation and documentation for diagnosing integration problems. Test automation has already been improved since that time, and we've accomplished most of the work for documentation: https://github.com/m-lab-tools/ooni-support/issues/60 Therefore, we propose to focus on some outstanding issues which will improve mlab-ns integration while continuing not to block on, or interfere with, M-Lab operations as follows: The primary change to mlab-ns will be to allow any tool to include arbitrary data per slivver to be gathered and distributed by mlab-ns. Ooni will use this to distribute data such as collector `.onion` addresses. The need for this change is discussed here: * https://github.com/m-lab-tools/ooni-support/issues/4 This proposed change is documented in this ticket: * https://github.com/m-lab-tools/ooni-support/issues/47 A secondary change is to implement `match=all` described in #47 above. It may not be necessary, so there is further investigation and testing necessary: https://github.com/m-lab-tools/ooni-support/issues/56 Along with these changes to `mlab-ns`, we need trivial updates to our integration scripts to work with mlab-ns rather than the simulator: * https://github.com/m-lab-tools/ooni-support/issues/10 * https://github.com/m-lab-tools/ooni-support/issues/11 Details & Links =============== Attached is a shortish overview of possible approaches to implement this integration. We've implemented a deployment with a mock mlab-ns (called mlab-ns-simulator) and the "arbitrary data" approach from the attached design document. The pull request is here: * https://github.com/m-lab-tools/ooni-support/pull/59 Specific details about this pull request: * A script for gathering necessary information from collectors and testhelpers, then updating the mlab-ns-simulator. * A script for updating a bouncer's state based on the mlan-ns-simulator. * A cron script to update the bouncer on an hourly schedule. * The mlab-ns-simulator itself, which approximates the production mlab-ns. * `.init/` script changes to automatically launch the simulator and bouncer on `mlab1.nuq0t.measurement-lab.org`. * Design documentation for mlab-ns integration (including this stepping stone architecture). * Each instructions to disable mlab-ns integration without any redeployment. We also created a subset pull request that has bug fixes but no mlab-ns integration features: * https://github.com/m-lab-tools/ooni-support/pull/58 Github Milestones ----------------- We split the mlab-ns-simulator deployment tasks out from the larger mlab-ns integration deployment. The mlab-ns-simulator milestone is at: * https://github.com/m-lab-tools/ooni-support/issues?q=is%3Aissue+milestone%3… The full mlab-ns integration milestone: * https://github.com/m-lab-tools/ooni-support/issues?q=is%3Aissue+milestone%3… As always, let us know if you have any feedback! -- Nathan Wilcox Least Authoritarian email: nathan(a)leastauthority.com twitter: @least_nathan PGP: 11169993 / AAAC 5675 E3F7 514C 67ED E9C9 3BFE 5263 1116 9993

3 8

OONI status report July 2014
by Arturo Filastò 05 Aug '14

05 Aug '14

# July This is what the OONI team was up to during the month of July. * Attended Tor dev meeting in Paris where we discussed about the bridge reachability project and the notion of an OONI consortium. * Define roadmap for the bridge reachability measurement project: https://trac.torproject.org/projects/tor/ticket/12544 * Fixed bug when a deck was missing some argument of a test https://gitweb.torproject.org/ooni-probe.git/commit/a9a24464b4b61800fe57874… * Improved error handling of oonibclient https://gitweb.torproject.org/ooni-probe.git/commit/d4cfecb34c146faf07c8726… * Published the bridge reachability results collected from CN, RU and US https://lists.torproject.org/pipermail/ooni-dev/2014-July/000118.html * Discussed with Choke point project some possible areas of collaboration: https://lists.torproject.org/pipermail/ooni-talk/2014-July/000003.html * Agreed to have weekly meetings on IRC #ooni every Friday at 5 PM CET. # August This month we plan to work on the following activities: * Specify the bridge reachability test: https://trac.torproject.org/projects/tor/ticket/12757 * Make ansible deployment script for bridge reachability probes on debian stable: https://trac.torproject.org/projects/tor/ticket/12769 * `daemonize` API breakage when upgrading from Twisted <= 13.1 to >= 13.2: https://trac.torproject.org/projects/tor/ticket/12644 * Probe Report should not leak the full PATH. https://trac.torproject.org/projects/tor/ticket/12706 * Have a list of all the the published reports so far https://trac.torproject.org/projects/tor/ticket/12770 * Add versioning information about the maxmind database being used https://trac.torproject.org/projects/tor/ticket/12772 * Design a test deck for Iran https://trac.torproject.org/projects/tor/ticket/12578 * "Failed to open reporter" might mean "the collector refused my report": https://trac.torproject.org/projects/tor/ticket/11855 ~ Art.

1 0

Detecting which Test Helpers are Running
by Taylor Hornby 02 Aug '14

02 Aug '14

Hello, We are working on scripts to create the Ooni bouncer configuration from multiple instances of Ooni running on M-Lab slices. So far, we have two scripts. One, which collects the portion of the bouncer.yaml out of an Ooni slice, and another script, which combines all of the parts into a single bouncer.yaml: https://github.com/LeastAuthority/ooni-support/blob/mlab-ns-simulator/bounc… https://github.com/LeastAuthority/ooni-support/blob/mlab-ns-simulator/bounc… One thing we would like the "getconfig.py" script to do is to dynamically determine which Test Helpers are enabled on the slice. I'm not sure how to do this, because the oonib.conf always contains entries for all the test helpers. So my question is this: How can we determine which Test Helpers are enabled by examining the oonib.conf or other files? Or, is this question misguided? Are *all* test helpers supposed to be in the bouncer.yaml file, even ones which are not running, and we don't want to have running? (For the first M-Lab deployment, there will be only one Test Helper, the http-return-headers one, so we could hard-code that one into the script, but we would like to make this script keep working when new tests are enabled). Thanks, -- Taylor Hornby

2 1

mlab-ns integration testing and easy-disable-feature.
by Nathan Wilcox 02 Aug '14

02 Aug '14

Dear Oonitarians, We're about to start testing some mlab-ns integration features on the M-Lab test slice. This will be potentially disruptive because we'll be reinitializing the slice, installing new bits and bobs, and we'll be using our own fork of ooni-support [1] rather than upstream ooni-support [2]. Please let us know if we should *not* do this soon. Today I planned to only poke mlab1.nuq0t.measurement-lab.org [3], but eventually we want to test at least two test slivvers. Meanwhile, what happens if we deploy some integration glue and it doesn't work, or has unanticipated problems in the future, or M-Lab policies change, etc? To anticipate those kinds of issues, we want to make sure that it's easy to toggle off the integration stuff. One of our key design goals has been to avoid modifying upstream ooni. We've just realized a new and related design goal: make sure it's easy to turn off the integration glue without requiring slice re-initialization, packages changes, repository changes, etc... So we just added a ticket for that [4]. Note that *while* we are doing this interactive testing, the two slivvers we're using will be pretty unstable, but *after* we reach a stable level of development, we should be able to flip the integration switch to off and the M-Lab test slice should operate as before (except with some extra stuff installed). If you have any concerns / feedback / comments, etc... feel free to email us, or respond to specific tickets. All of the tickets related to the mlab-ns integration in particular should live in the `ooni-support` issue tracker with the `LeastAuthority D` label [5]. Regards, Nathan links: [1] https://github.com/LeastAuthority/ooni-support [2] https://github.com/m-lab-tools/ooni-support [3] https://github.com/m-lab-tools/ooni-support/issues/50#issuecomment-50060748 [4] https://github.com/m-lab-tools/ooni-support/issues/54 [5] https://github.com/m-lab-tools/ooni-support/issues?direction=desc&labels=Le… -- Nathan Wilcox Least Authoritarian email: nathan(a)leastauthority.com twitter: @least_nathan PGP: 11169993 / AAAC 5675 E3F7 514C 67ED E9C9 3BFE 5263 1116 9993

2 1