October 2014 - ooni-dev - lists.torproject.org

Feedback on OONI data collection, aggregation, and visualization
by Karsten Loesing 15 Dec '14

15 Dec '14

Hi everyone, I just subscribed to this list, because Arturo asked me to comment on two postings here. As a very quick introduction, and because I don't know how distinct the Tor community and the OONI community are: I'm the developer behind the Tor network data collector CollecTor [-1] and the Tor Metrics website that aggregates and visualizes Tor network data [0]. Here's what Arturo asked on another thread: > With OONI what we are currently focusing on is Bridge Reachability > measurements. We have at this time 1 meter in China, 1 in Iran (a second > one is going to be setup soon), 1 in Russia and 1 in Ukraine. We have > some ideas of the sorts of information we would like to extract from > this data, but it would also be very good to have some more feedback > from you on what would be useful [1]. Long mail is long. Some random thoughts: - For Tor network data it has turned out to be quite useful to strictly separate data collection from data aggregation from data visualization. That is, don't worry too much about visualizing the right thing, but start with something, and if you don't like it, throw it away and do it differently. And if you're aggregating the wrong thing, then aggregate the previously collected data in a different way. Of course, if you figure out you collected the wrong thing, then you won't be able to go back in time and fix that. - I saw some discussion of "The pool from where the bridge has been extracted (private, tbb, BridgeDB https, BridgeDB email)". Note that isis and I are currently talking about removing sanitized bridge pool assignments from CollecTor. We're thinking about adding a new config line to tor that states the preferred bridge pool, which could be used here instead. Just as a heads-up, six months or so in advance. I can probably provide more details if this is relevant to you. > Another area that perhaps overlaps with the needs of the metrics is data > storage. Currently we have around 16 GB of uncompressed raw report data > that needs to be archived (currently it's being stored and published on > staticiforme, but I have a feeling that is not ideal especially when the > data will become much bigger) and indexed in some sort of database. > Once we put the data (or a subset of it) in a database producing > visualizations and exposing the data to end users will be much simpler. > The question is if this is a need also for > Metrics/BwAuth/ExitScanner/DocTor and if we can perhaps work out some > shared infrastructure to fit both of our goals. > Currently we have placed the data inside of MongoDB, but some concerns > with it have been raised [2]. Again, some random thoughts: - For Metrics, the choice of database is entirely an internal decision, and no user would ever see that. It's part of the aggregation part. If we ever decide to pick something else (than PostgreSQL in this case), we'd have to rewrite the aggregation scripts, which would then produce the same or similar output (which is an .csv file in our case). That being said, trying out MongoDB or another NoSQL variant might be worthwhile, but don't rely on it too much. - Would you want to add bridge reachability statistics to Tor Metrics? I'm currently working on opening it up and making it easier for people to contribute metrics. Maybe take a look at the website prototype that I posted to tor-dev@ a week ago [3] (and if you want, comment there). I could very well imagine adding a new section "Reachability" right next to "Diversity" with one or more graphs/tables provided by you. Please see the new "Contributing to Tor Metrics" section on the About page for the various options for contributing data or metrics. - Please ask weasel for a VM to host those 16 GB of report data; having it on staticiforme is probably a bad idea. Also, do you have any plans to synchronize reports between hosts? I'm planning such a thing for CollecTor where two or more instances fetch relay descriptors from directory authorities and automatically exchange missing descriptors. - I could imagine extending CollecTor to also collect and archive OONI reports, as a long-term thing. Right now CollecTor does that for Tor relay and bridge descriptors, TORDNSEL exit lists, BridgeDB pool assignment files, and Torperf performance measurement results. But note that it's written in Java and that I hardly have development time to keep it afloat; so somebody else would have to extend it towards supporting OONI reports. I'd be willing to review and merge things. We should also keep CollecTor pure Java, because I want to make it easier for others to run their own mirror and help us make data more redundant. Anyway, I can also imagine keeping the OONI report collector distinct from CollecTor and only exchange design ideas and experiences if that's easier. Lots of ideas. What do you think? All the best, Karsten [-1] https://collector.torproject.org/ [0] https://metrics.torproject.org/ [1] https://lists.torproject.org/pipermail/ooni-dev/2014-October/000176.html [2] https://lists.torproject.org/pipermail/ooni-dev/2014-October/000178.html [3] https://kloesing.github.io/metrics-2.0/

3 5

david's notes from yesterday
by David Stainton 25 Oct '14

25 Oct '14

* TODO create complete disaster recover procedure for Agregator server This can for example be a manual procedure... but in that case it must be fully documented. For instance the docs should specify how to build new Aggregator server given a freshly installed Debian stable server; this should involve copying some Docker scripts and running them to build Docker images that are configured properly for the different components. * Aggregator What are the different Aggregator Docker images? ** synchronization/subscriber of publishing collectors *** TODO create a Docker image to rsync data from the collectors and trigger removal afterwards This task can initially use rsync... but later will be replaced to a pub/sub system. ** YAML data sanitizer *** TODO create YAML data sanitizer Docker image This task makes the yaml reports data suitable to be published AND prepares for database import. ** data post processing + DB import *** TODO create post processing Docker image analyses the yaml reports creates database entries - only this Docker image contains mongodb write access credentials * non-aggregator Docker images ** mongodb This Docker image runs the MongoDB service. 1st iteration: listens on loopback interface 2nd iteration: move this Docker image to a different machine and listen to a publicly routed IP address. Must use crypto wire protocol + authentication ** web mirror *** TODO create docker image of web mirror * stateful directory structure ** task 0 rsync data to /data/raw # example: data/bouncer-nkvphnp3p6agi5qq/archive # /data/collector/archive ** task 1 sanitize data and move to /data/staging # example: none ** state 2 post-processing to prepare to DB import move data to /data/reports # example: none ** state 3 data that has been imported into the DB move data to /de # example: /data/reports * stateful directory backup strategy The backup strategy for the above 3-part stateful directory workflow is to make backup snapshots of all three state directories. * collector to aggregator state #1 For the moment we'll use rsync to copy the data from the collectors to the Aggregator. After rsyncing, the data will then be removed from the Collectors. Later iterations will involve a pub/sub system between the Collectors and Aggregators.

1 0

Background on bridge reachability project
by Arturo Filastò 24 Oct '14

24 Oct '14

Hello Oonitarians! To get an overall idea of the scope of the problem read the meeting proposal that explains to funders what we are trying to achieve with the hackfest [1] Most of the things in there you probably already know. During a meeting with some people from the visualization team we collected some notes about the infrastructure we need to setup and the sort of questions we are trying to answer with the data that we collect [2]. Now All of the infrastructure detailed there is in place and we have been collecting data from those 5 machines since about a month. To get a feeling of what the data we will be analyzing looks like see the published bridge reachability reports from China [3]. The ones from Russia, Ukraine, Iran are not yet published, but you will gain access to them during the meeting (though some of them first need to be redacted to strip out private bridge IPs). In this thread [4] what is in the notes above, is expanded and I recommend you read a bit of it. Some people have already started to do some mockups of visualization and you can see it here: Donuts : http://ooniviz.chokepointproject.net/donuts.htm Bar charts : http://ooniviz.chokepointproject.net/transports.htm Multi-line charts : http://ooniviz.chokepointproject.net/successes.htm Time-lines : http://ooniviz.chokepointproject.net/timelines.htm Let me know if you have any questions and if you want this evening we can get dinner or grab drinks later. We are going to be in Prenzlauer Berg at the address that you should have gotten in the other email. See you soon! ~ Arturo [1] https://pad.riseup.net/p/bridgereachability_meeting_proposal [2] https://pad.riseup.net/p/bridgereachability [3] https://ooni.torproject.org/reports/0.1/CN/ [4] https://lists.torproject.org/pipermail/ooni-dev/2014-October/000176.html

1 0

Some ideas on the visualization of OONI data
by kudrom 20 Oct '14

20 Oct '14

Hi all For the last couple of days i've been thinking about the visualization of the bridge reachability data and how it relates to the currently deployed ooni [7] system, here are the conclussions: == Variables == I think that the statistical variables for the bridge reachability reports are: - Success of the nettest (yes, no, errors) - The PT of the bridge (obfs3, obfs2, fte, vanilla) - The pool from where the bridge has been extracted (private, tbb, BridgeDB https, BridgeDB email) - The country "of the ooni-probe" With these variables I believe we can answer a lot of questions related to how much censorship is being taken, where and how. But there's something left: the timing. George sent an email [0] in which he proposes a timeline of events [5] of every bridge that would allow us to diagnose with much more precission how and why is a bridge being censored. To build that diagram we should define first the events that will be showed in the timeline. I think those events are the values of the pool variable and if the bridge is being blocked in a given country. With the events defined i think we can define another variable: - Time deltas between bridge events. So, for example, what this variable will answer is: how many {days, hours...} does it take China to block a bridge that is published in bridgeDB? Is China blocking new bridges at the same speed that Iran? How many days does it take China block a private bridge? There are some ambiguities related to the deltas, for example if the bridge is sometimes blocked and sometimes not in a country, which delta should we compute? Finally, in the etherpad [1] the tor's bootstrap is suggested as a variable, i don't understand why. Is it to detect some way of censorship? Can anyone explain a little more? == Data schema == In the last email Ruben, Laurier and Pascal "strongly recommended importing the reports into a database". I deeply believe the same. We should provide a service to query the values of the previous variables plus the timestamp of the nettest and the fingerprint of the bridge. With this database the inconsistencies between the data formats of the reports should be erased and the work with the data is much more easy. I think that we should also provide a way to export the queries to csv/json to allow other people to dig into the data. I also believe that we could use mongodb just because one reason: we can distribute it very easily. But let me explain why in the Future section. == Biased data == Can a malicious ooni-probe bias the data? For example, if it executes in bursts some tests the reports are going to be the same and the general picture could be biased. Any more ideas? == Geo Data == In the etherpad [1] it's suggested to increase the granularity of the geo data to detect geographical patterns, but it seems [2] that at least in China there's not such patterns so maybe we should discard the idea altogether. == Playing with data == So until now i've talked about data. Now i want to address how to present the data. I think we should provide a way to play with data to allow a more thoughtful and precise diagnosis of censorship. What i was thinking is to enhance the interactivity of the visualization by allowing the user a way to render the diagrams at the same time she thinks about the data. The idea is to allow the user to go from more general to more concret data patterns. So imagine that the user loads the visualization's page, first he sees a global heated map of censorship measured with the bridge reachability test, he is chinese so he clicks in his country and a histogram like [3] for China is stacked at the bottom of the global map, he then clicks on the obfs2 and a diagram like [4] is also stacked at the bottom but only showing the success variable for the obfs2 PT, then he clicks on the True value for the success variable and all the bridges that have been reached by all the nettests executions in that period of time in China are showed, finally he selects one bridge and it's timeline [5] plus it's link to atlas [6] is provided. This is only a particular scenario, the core idea is to provide the user with the enhanced capability to drive conclusions as much as she desires. The user started with the more general concept of the data, and he applied restrictions to the datapoints to dig more into the data. From general to specific he can start making hypothesis that he later discards or approves with more info displayed in the next diagram. There are some usability problems with the selection of diagram+variable and the diverse set of users that will use the system, but i'd be very glad to think about them if you like the idea. == Users == I think there are three set of users: 1- User of tor that is interested in the censorship performed in its country and how to avoid it. 2- Journalist that wants to write something about censorship but isn't that tech savvy. 3- Researcher that wants updated and detailed data about censorship. I believe we can provide a system that satisfies the three of them if we succeed in the previous bullet point. == Future == So, why do i think that we should index the data with mongodb? Because i think that this data repository should be provided as a new ooni-backend API related to the current collector. Right now the collectors can write down reports from any ooni-probe instance that chooses to do so and its API is completly separated from the bouncer API, which overall is a wise design decision because you can deploy ooni-backend to only work as a collector. So it's not unreasonable to think that we can have several collectors collecting different reports because the backend is designed to do so, therefore we need the data repository to be distributed. And mongodb is good at this. If we build the database for the bridge reachability nettests, i think that we should design it to index in the future all nettest reports and therefore generalize the desgin, implementation and deployment of all the work that we are going to do to the bridge reachability. That way an analyst can query the distributed database with a proper client that connects to the data repository ooni-backend API. So to sum up, I started talking about the bridge reachability visualization problem and finished with a much broader vision that intends to integrate the ongoing efforts of the bridge reachability to improve ooni as a whole. Hope the email is not too large. ciao [0] https://lists.torproject.org/pipermail/tor-dev/2014-October/007585.html [1] https://pad.riseup.net/p/bridgereachability [2] https://blog.torproject.org/blog/closer-look-great-firewall-china [3] http://ooniviz.chokepointproject.net/transports.htm [4] http://ooniviz.chokepointproject.net/successes.htm [5] https://people.torproject.org/~asn/bridget_vis/tbb_blocked_timeline.jpg [6] https://atlas.torproject.org [7] https://ooni.torproject.org/

3 5

ooniprobe v1.2.2 is out
by Arturo Filastò 17 Oct '14

17 Oct '14

Hi all, This mail is to inform you that a new version of ooniprobe has just been released. Here is the changelog: v1.2.2 (Fri, 17 Oct 2014) ------------------------- Who said friday 17th is only bad luck? * Add two new report entry keys test_start_time and test_runtime * Fix bug that lead to ooniresources not working properly It is now available for download from pypi: https://pypi.python.org/pypi/ooniprobe and soon will be available for download in debian. Have fun! ~ Arturo

1 0

OONI status report for September 2014
by Arturo Filastò 06 Oct '14

06 Oct '14

# Activities of September 2014 This month was a one dedicated mainly to system administration related tasks and on moving forwards the process of deploying OONI on M-Lab. All of the final details for the mlab deployment have been sorted out (also thanks to meeting with the mlab people in DC for the OTF summit) and should be set to have it deployed in the upcoming months. The bridge reachability project is also moving forward very well, with us having now probes in China, Iran, Russia, Ukraine, that run daily measurements to verify if a set of Tor bridges are working or not. Not all the collected data has yet been published since it also contains addresses of bridges that are not public. We also have released a new version of ooniprobe and oonibackend. You can read more about it's changelog here: https://lists.torproject.org/pipermail/ooni-dev/2014-October/000171.html Here are some of the tickets that we have solved this month: * Implemented support for making bouncer and client aware of policies: https://trac.torproject.org/projects/tor/ticket/12579 * Add support for writing oonibackend reports using the mlab format https://trac.torproject.org/projects/tor/ticket/13232 * Run a final end-to-end test https://trac.torproject.org/projects/tor/ticket/13289 * Get two boxes that are modern for running the bridge reachability tests: https://trac.torproject.org/projects/tor/ticket/12543 * Fix backward compatibility with twisted 13.1 in ooniprobe https://trac.torproject.org/projects/tor/ticket/13139 * Make ansible deployment script for bridge reachability probes on debian stable https://trac.torproject.org/projects/tor/ticket/12769 * Configure travis to run ooni-backend tests regularly with public reports https://trac.torproject.org/projects/tor/ticket/12244 * Attended the OTF summit in DC # Plans for October 2014 * Post process the data collected by the bridge DB collector https://trac.torproject.org/projects/tor/ticket/12546 * Sanitize .csv files https://trac.torproject.org/projects/tor/ticket/12548 * ooni's TrueHeaders uses a dict() for internal storage: https://trac.torproject.org/projects/tor/ticket/11991 * Make a test that measures both HTTP and DNS censorship: https://trac.torproject.org/projects/tor/ticket/11972 * ooni's TrueHeaders uses a dict() for internal storage: https://trac.torproject.org/projects/tor/ticket/11991 * Do not verify inputs if no policy is specified https://trac.torproject.org/projects/tor/ticket/12299 ~ Art.

1 0

Schedule change for ooni dev meeting at 19:00 CET on Thursday October 2 2014
by Arturo Filastò 02 Oct '14

02 Oct '14

Hello Oonitarians, This is a reminder to inform you that the weekly dev meeting will happen today at 19:00 CET Thursday October 2. Note: That the schedule has changed from Friday to Thursday. Adjust your calendaring software accordingly. See you! ~ Art.

1 0

Request to change the ooni meeting
by kudrom 02 Oct '14

02 Oct '14

Hi, As you all know the meeting of the ooni project is scheduled on fridays at 17:00 CET, and as some of you will know some of us have started the university classes in september, with the unfortunate personal event that i have a lab on friday from 16:00 to 18:00 CET, so i was wondering if the hour or day of the meeting can be changed. I propose on Thursdays at 17:00 CET ciao

2 2

ooniprobe 1.2.0 and oonibackend 1.1.4 are out!
by Arturo Filastò 01 Oct '14

01 Oct '14

Greetings! We are very pleased to announce the release of new versions of ooniprobe and oonibackend. One of the most interesting new features that are now part of ooniprobe is the ability to generate test decks for the country you are in a way that is much easier than before. As a matter of fact to start contributing useful measurements it's just a mater of 5 minutes of setup: https://pypi.python.org/pypi/ooniprobe#ooni-in-5-minutes Here are some more details of what this release includes: ooniprobe ========= * Introduce a new tool for generating ooniprobe test decks called oonideckgen. * Introduce a new tool for updating resources used for geoip lookup and deck generation. * Add support for policy aware bouncing in the client. https://trac.torproject.org/projects/tor/ticket/12579 * Various improvements to the bridge_reachability test (enable better tor logging and also log obfsproxy) * Fix backward compatibility with twisted 13.1 and add regression tests for this. https://trac.torproject.org/projects/tor/ticket/13139 oonibackend =========== * Add support for specifying the report archive path from config file * Write tor notice level logs to file * Fix bug that lead test helpers to not being started The new packages have been uploaded to pypy and can be found at the following links: https://pypi.python.org/pypi/ooniprobe https://pypi.python.org/pypi/oonibackend Soon ooniprobe will also be available inside of debian testing thanks to Lunar^. Have fun, ~ Art.

1 0