November 2014 - ooni-dev - lists.torproject.org

Feedback on OONI data collection, aggregation, and visualization
by Karsten Loesing 15 Dec '14

15 Dec '14

Hi everyone, I just subscribed to this list, because Arturo asked me to comment on two postings here. As a very quick introduction, and because I don't know how distinct the Tor community and the OONI community are: I'm the developer behind the Tor network data collector CollecTor [-1] and the Tor Metrics website that aggregates and visualizes Tor network data [0]. Here's what Arturo asked on another thread: > With OONI what we are currently focusing on is Bridge Reachability > measurements. We have at this time 1 meter in China, 1 in Iran (a second > one is going to be setup soon), 1 in Russia and 1 in Ukraine. We have > some ideas of the sorts of information we would like to extract from > this data, but it would also be very good to have some more feedback > from you on what would be useful [1]. Long mail is long. Some random thoughts: - For Tor network data it has turned out to be quite useful to strictly separate data collection from data aggregation from data visualization. That is, don't worry too much about visualizing the right thing, but start with something, and if you don't like it, throw it away and do it differently. And if you're aggregating the wrong thing, then aggregate the previously collected data in a different way. Of course, if you figure out you collected the wrong thing, then you won't be able to go back in time and fix that. - I saw some discussion of "The pool from where the bridge has been extracted (private, tbb, BridgeDB https, BridgeDB email)". Note that isis and I are currently talking about removing sanitized bridge pool assignments from CollecTor. We're thinking about adding a new config line to tor that states the preferred bridge pool, which could be used here instead. Just as a heads-up, six months or so in advance. I can probably provide more details if this is relevant to you. > Another area that perhaps overlaps with the needs of the metrics is data > storage. Currently we have around 16 GB of uncompressed raw report data > that needs to be archived (currently it's being stored and published on > staticiforme, but I have a feeling that is not ideal especially when the > data will become much bigger) and indexed in some sort of database. > Once we put the data (or a subset of it) in a database producing > visualizations and exposing the data to end users will be much simpler. > The question is if this is a need also for > Metrics/BwAuth/ExitScanner/DocTor and if we can perhaps work out some > shared infrastructure to fit both of our goals. > Currently we have placed the data inside of MongoDB, but some concerns > with it have been raised [2]. Again, some random thoughts: - For Metrics, the choice of database is entirely an internal decision, and no user would ever see that. It's part of the aggregation part. If we ever decide to pick something else (than PostgreSQL in this case), we'd have to rewrite the aggregation scripts, which would then produce the same or similar output (which is an .csv file in our case). That being said, trying out MongoDB or another NoSQL variant might be worthwhile, but don't rely on it too much. - Would you want to add bridge reachability statistics to Tor Metrics? I'm currently working on opening it up and making it easier for people to contribute metrics. Maybe take a look at the website prototype that I posted to tor-dev@ a week ago [3] (and if you want, comment there). I could very well imagine adding a new section "Reachability" right next to "Diversity" with one or more graphs/tables provided by you. Please see the new "Contributing to Tor Metrics" section on the About page for the various options for contributing data or metrics. - Please ask weasel for a VM to host those 16 GB of report data; having it on staticiforme is probably a bad idea. Also, do you have any plans to synchronize reports between hosts? I'm planning such a thing for CollecTor where two or more instances fetch relay descriptors from directory authorities and automatically exchange missing descriptors. - I could imagine extending CollecTor to also collect and archive OONI reports, as a long-term thing. Right now CollecTor does that for Tor relay and bridge descriptors, TORDNSEL exit lists, BridgeDB pool assignment files, and Torperf performance measurement results. But note that it's written in Java and that I hardly have development time to keep it afloat; so somebody else would have to extend it towards supporting OONI reports. I'd be willing to review and merge things. We should also keep CollecTor pure Java, because I want to make it easier for others to run their own mirror and help us make data more redundant. Anyway, I can also imagine keeping the OONI report collector distinct from CollecTor and only exchange design ideas and experiences if that's easier. Lots of ideas. What do you think? All the best, Karsten [-1] https://collector.torproject.org/ [0] https://metrics.torproject.org/ [1] https://lists.torproject.org/pipermail/ooni-dev/2014-October/000176.html [2] https://lists.torproject.org/pipermail/ooni-dev/2014-October/000178.html [3] https://kloesing.github.io/metrics-2.0/

3 5

Summary of data visualization OONI workshop
by Arturo Filastò 26 Nov '14

26 Nov '14

Greetings! Yesterday we did a small OONI workshop in Rome to focus on thinking about the problem of data visualization. I think what came out of it is very useful, though it is not yet some fancy graphics visualizations. The main value comes from having some people that have very little prior knowledge of OONI think about the problem of data visualization and learn from them what is their framework of thinking. One of my personal favorite takeaways is that when we think about the results of OONI what we should be talking about are "network irregularities". In the end what we are interested in spotting are deviations from what is to be considered a "normal" network behavior. This may or may not be a symptom of internet censorship. It could also be caused by some mess-up by the network operator or some bugs in our monitoring software. The point is that when we spot an irregularity we should investigate it further to discover if it is in fact an instance of censorship or something else. Starting from this assumption we started designing what should be the variables that a user interested in analyzing the data collected so far should be adjusting. We set our target audience to be somebody that is interested in integrating the data into some work they are doing. What came up often as a practical example of this is "the journalist working on an article on freedom of expression in country X". At the end of the day we came up with this chart that shows what are the variables that should be combined together to produce different views: https://people.torproject.org/~art/ooni/ooni-workshop-0x02-irregularities.p… What basically happens here is that the user fixes the timeframe they are interested in looking at. Once that is set they will be able to say "Show me all the _interesting_ measurements that are related to the country|test_name|input X". I think that now Viviana and Silvia have a clearer understanding of how OONI works and what it's measuring and they will be working on producing some mockups of UIs for exploring the data in this way. I will then implement these mockups based on the new REST API we have to the data. If you have questions on feedback on this it's greatly appreciated so please contribute to either this thread or post on this ticket: https://trac.torproject.org/projects/tor/ticket/13731 Have fun! ~ Arturo

1 0

Next OONI weekly meetings
by Arturo Filastò 24 Nov '14

24 Nov '14

Hello Oonitarians, Sorry for not sending this email sooner, but this week has been a bit hectic. A couple of people have said that they cannot attend the weekly meeting on thursday at 18:00, because it conflicts with another recurring appointment they have. For this reason I am going to suggest we reschedule the next OONI meetings to a date that works for everybody. Monday and Tuesdays seem to be the best option, so I am going to propose we skip next weeks meeting and instead opt for holding next weeks one on either a Monday or Tuesday at 18:00 UTC, 19:00 CET, 13:00 EST, 12:00 CST, 10:00:00 PST. How does this sound for everybody? I personally would prefer Monday over Tuesday, but if you would instead prefer Tuesday please send a reply to the ooni-dev mailing list. If nobody replies I am going to assume every Monday works for you all and the next meeting will be on Monday 24th of November. Have fun! ~ Arturo

2 2

OONI weekly dev meeting reminder
by Arturo Filastò 13 Nov '14

13 Nov '14

Hello Oonitarians, This is reminder that today (Thursday 6th of November 2014) there will be the weekly ooni dev meeting in #ooni on irc.oftc.net. The time will be: 18:00 UTC 19:00 CET 13:00 EST 12:00 CST 10:00:00 PST You are all invited to attend! ~ Arturo

1 1

OONI hackfest summary
by Arturo Filastò 04 Nov '14

04 Nov '14

>From October 24th to 26th the OONI team gathered in Berlin for a hackfest. Around 20 people ended up showing up and although most of them were seasoned Oonitarians some fairly new people joined us that I hope will become part of the growing OONI community. The scope of the hackfest was that of data analytics and visualization with special focus on the Tor bridge reachability study we are currently doing. # Bridge reachability study The goal of this study [1] is that of answering some questions concerning the blocking of Tor bridges [2] and pluggable transport [3] enabled bridges in the countries of China, Iran, Russia and Ukraine (test vantage points). To establish a baseline to eliminate the cases in which the bridge is marked as blocked, while it is in fact just offline, we measure also from a vantage point located in the Netherlands. For every test vantage point we perform two types of measurements: * A Bridge reachability measurement [4][5] that attempts to build a tor circuit using the bridge in question * A TCP connect measurement [6][7] that simply does a TCP connect to the bridge IP and port We run both of the measurements to further debug the reason why the blocking is happening, may this be due to a TCP RST or direct IP blocking or tor malfunction. So far this study has been running for a little less than 1 month. # OONI data pipeline In order to produce the aggregate data needed to build visualizations we have built a data pipeline [8] This consists of a series of operations that are done to the raw reports in order to strip out sensitive information and place the collected data into a database. The nice thing is that the data pipeline we have designed is not specific to this study, but can and will be in the future expanded to export data needed to visualize also the other types of measurements done by OONI. The data pipeline is comprised of 3 steps (or states, depending on how you want to look at it). When the data is submitted to a OONI collector it is synchronized with the aggregator. This is a central machine responsible for running all the data processing tasks, storing the collected data in a database and hosting a public interface to the sanitised reports. Since all the steps are independent from one another it is not necessary that they run on the machine, but it may also be more distributed. Once the data is on the aggregator machine it is said to be in the RAW state. The sanitise task is then run on the RAW data to remove sensitive information and strip out some superfluous information. A RAW copy of every report is also stored in a private compressed archive for future reference. Once the data is sanitised it is said to tbe in SANITISED state. At this point a import task is run on the data to place it inside of a database. The SANITISED reports are then place in a directory that is publicly exposed to the internet to allow people to download also a copy of the YAML reports. At this point is is possible to run any export task that performs queries on the database and produces as output some documents to be used in the data visualizations (think JSON, CSV, etc.). # The OONI hackfest The first day of the hackfest was spent going over the scope of the project we would be working on in the following days as well as working in groups that were interested in tacking the design of one aspect of the problem. Sticky notes were plentiful and helped us have a clear vision of what lied ahead of us. By the end of the first day we had clear what were the set of tasks that were needed to achieve our goals and which teams would be responsible for doing what. The second day was almost entirely dedicated to hacking and everybody had a task to complete that was either completed by the end of the day or sooner. Some people even completed their initially assigned task before the end of the day and came back asking for more! By the end of the second day we had a real data set to hand over to the visualization team, to start producing some pretty graphs based on real data. We decided that the first visualization we wanted to do should be kept as simple as possible and be something that we could also use to debug the data we had collected. It should tell us which bridges were working when and it should present the information in a way that would highlight the country involved and the pluggable transport type. A prototype of it can be seen here: http://reports.ooni.nu/analytics/bridge_reachability/timeline/ The code for this visualization can be found here: https://github.com/Shidash/OONI-Bridge-Reachability-Timeline # Next steps * Write scripts for generating the bridge_db.json document based on the data that is given to us from the bridge db team https://trac.torproject.org/projects/tor/ticket/13570 * Align the dates in the visual timeline https://trac.torproject.org/projects/tor/ticket/13639 * Better tokenising for bridges so that bridges that have the same fingerprint, but different transport are grouped properly https://trac.torproject.org/projects/tor/ticket/13638 * Finish setting up the docker containers for the steps of the data pipeline https://trac.torproject.org/projects/tor/ticket/13568 * Setup disaster recovery procedure and backup: https://trac.torproject.org/projects/tor/ticket/13584 * Setup monitoring of the probes. https://trac.torproject.org/projects/tor/ticket/12549 * Add support for obfs4 https://trac.torproject.org/projects/tor/ticket/13597 * Set upper bound in comparison with the control in the bridge reachability timeline https://trac.torproject.org/projects/tor/ticket/13640 * Make sure that the control measurement is for the specific bridge measurement https://trac.torproject.org/projects/tor/ticket/13655 Questions and comments should be directed to the ooni-dev mailing list or to the #ooni channel on irc.oftc.net. Have fun! ~ Arturo [1] https://lists.torproject.org/pipermail/ooni-dev/2014-October/000184.html [2] https://www.torproject.org/docs/bridges [3] https://www.torproject.org/docs/pluggable-transports.html.en [4] https://gitweb.torproject.org/ooni/spec.git/blob/HEAD:/test-specs/ts-011-br… [5] https://gitweb.torproject.org/ooni-probe.git/blob/HEAD:/ooni/nettests/block… [6] https://gitweb.torproject.org/ooni/spec.git/blob/HEAD:/test-specs/ts-008-tc… [7] https://gitweb.torproject.org/ooni-probe.git/blob/HEAD:/ooni/nettests/block… [8] https://github.com/TheTorProject/ooni-pipeline/blob/master/Readme.md#ooni-p…

2 1

OONI team status report for October 2014
by Arturo Filastò 03 Nov '14

03 Nov '14

# What we did in October 2014 ## ooniprobe * We released ooniprobe 1.2.0 and 1.2.2 that include two new tools oonideckgen and ooniresources making it possible to generate test decks for a specific country. For a full changelog see: https://gitweb.torproject.org/ooni-probe.git/blob/HEAD:/ChangeLog.rst#l13 https://gitweb.torproject.org/ooni-probe.git/blob/HEAD:/ChangeLog.rst#l4 * Add some extra keys to the ooniprobe report output: https://github.com/TheTorProject/ooni-spec/commit/73aa34f1123af603fab00d00b… https://github.com/TheTorProject/ooni-probe/commit/b59d9343354990a39a87fd6e… ## oonibackend * Fix bug in backend: https://github.com/TheTorProject/ooni-backend/commit/7e95433b66cb96b5c524a7… ## oonipipeline This is a new software component that deals with processing raw ooniprobe reports, placing them in a database and exposing them to the web. An overview of this software component can be found here: https://github.com/TheTorProject/ooni-pipeline#ooni-pipeline * Work on the script for putting the sanitised reports into the database https://trac.torproject.org/projects/tor/ticket/13564 * Work on script that generates the data formats for visualisation team https://trac.torproject.org/projects/tor/ticket/13565 * Sanitize .csv files https://trac.torproject.org/projects/tor/ticket/12548 * Post process the data collected by the bridge DB collector https://trac.torproject.org/projects/tor/ticket/12546 ## Data visualisation * Produce a prototype of the graphs for the bridge reachability study. ## Misc. * Organise and attend a hackfest in berlin to bootstrap the bridge reachability study. * Meet with censorship researchers in NYC during an event organised by mlab. # Next steps for November 2014 * Write scripts for generating the bridge_db.json document based on the data that is given to us from the bridge db team https://trac.torproject.org/projects/tor/ticket/13570 * Align the dates in the visual timeline https://trac.torproject.org/projects/tor/ticket/13639 * Better tokenising for bridges so that bridges that have the same fingerprint, but different transport are grouped properly https://trac.torproject.org/projects/tor/ticket/13638 * Finish setting up the docker containers for the steps of the data pipeline https://trac.torproject.org/projects/tor/ticket/13568 * Setup disaster recovery procedure and backup: https://trac.torproject.org/projects/tor/ticket/13584 * Setup monitoring of the probes. https://trac.torproject.org/projects/tor/ticket/12549 * Add support for obfs4 https://trac.torproject.org/projects/tor/ticket/13597 * Set upper bound in comparison with the control in the bridge reachability timeline https://trac.torproject.org/projects/tor/ticket/13640

1 0