- ooni-dev - lists.torproject.org

Some ideas on the visualization of OONI data
by kudrom 20 Oct '14

20 Oct '14

Hi all For the last couple of days i've been thinking about the visualization of the bridge reachability data and how it relates to the currently deployed ooni [7] system, here are the conclussions: == Variables == I think that the statistical variables for the bridge reachability reports are: - Success of the nettest (yes, no, errors) - The PT of the bridge (obfs3, obfs2, fte, vanilla) - The pool from where the bridge has been extracted (private, tbb, BridgeDB https, BridgeDB email) - The country "of the ooni-probe" With these variables I believe we can answer a lot of questions related to how much censorship is being taken, where and how. But there's something left: the timing. George sent an email [0] in which he proposes a timeline of events [5] of every bridge that would allow us to diagnose with much more precission how and why is a bridge being censored. To build that diagram we should define first the events that will be showed in the timeline. I think those events are the values of the pool variable and if the bridge is being blocked in a given country. With the events defined i think we can define another variable: - Time deltas between bridge events. So, for example, what this variable will answer is: how many {days, hours...} does it take China to block a bridge that is published in bridgeDB? Is China blocking new bridges at the same speed that Iran? How many days does it take China block a private bridge? There are some ambiguities related to the deltas, for example if the bridge is sometimes blocked and sometimes not in a country, which delta should we compute? Finally, in the etherpad [1] the tor's bootstrap is suggested as a variable, i don't understand why. Is it to detect some way of censorship? Can anyone explain a little more? == Data schema == In the last email Ruben, Laurier and Pascal "strongly recommended importing the reports into a database". I deeply believe the same. We should provide a service to query the values of the previous variables plus the timestamp of the nettest and the fingerprint of the bridge. With this database the inconsistencies between the data formats of the reports should be erased and the work with the data is much more easy. I think that we should also provide a way to export the queries to csv/json to allow other people to dig into the data. I also believe that we could use mongodb just because one reason: we can distribute it very easily. But let me explain why in the Future section. == Biased data == Can a malicious ooni-probe bias the data? For example, if it executes in bursts some tests the reports are going to be the same and the general picture could be biased. Any more ideas? == Geo Data == In the etherpad [1] it's suggested to increase the granularity of the geo data to detect geographical patterns, but it seems [2] that at least in China there's not such patterns so maybe we should discard the idea altogether. == Playing with data == So until now i've talked about data. Now i want to address how to present the data. I think we should provide a way to play with data to allow a more thoughtful and precise diagnosis of censorship. What i was thinking is to enhance the interactivity of the visualization by allowing the user a way to render the diagrams at the same time she thinks about the data. The idea is to allow the user to go from more general to more concret data patterns. So imagine that the user loads the visualization's page, first he sees a global heated map of censorship measured with the bridge reachability test, he is chinese so he clicks in his country and a histogram like [3] for China is stacked at the bottom of the global map, he then clicks on the obfs2 and a diagram like [4] is also stacked at the bottom but only showing the success variable for the obfs2 PT, then he clicks on the True value for the success variable and all the bridges that have been reached by all the nettests executions in that period of time in China are showed, finally he selects one bridge and it's timeline [5] plus it's link to atlas [6] is provided. This is only a particular scenario, the core idea is to provide the user with the enhanced capability to drive conclusions as much as she desires. The user started with the more general concept of the data, and he applied restrictions to the datapoints to dig more into the data. From general to specific he can start making hypothesis that he later discards or approves with more info displayed in the next diagram. There are some usability problems with the selection of diagram+variable and the diverse set of users that will use the system, but i'd be very glad to think about them if you like the idea. == Users == I think there are three set of users: 1- User of tor that is interested in the censorship performed in its country and how to avoid it. 2- Journalist that wants to write something about censorship but isn't that tech savvy. 3- Researcher that wants updated and detailed data about censorship. I believe we can provide a system that satisfies the three of them if we succeed in the previous bullet point. == Future == So, why do i think that we should index the data with mongodb? Because i think that this data repository should be provided as a new ooni-backend API related to the current collector. Right now the collectors can write down reports from any ooni-probe instance that chooses to do so and its API is completly separated from the bouncer API, which overall is a wise design decision because you can deploy ooni-backend to only work as a collector. So it's not unreasonable to think that we can have several collectors collecting different reports because the backend is designed to do so, therefore we need the data repository to be distributed. And mongodb is good at this. If we build the database for the bridge reachability nettests, i think that we should design it to index in the future all nettest reports and therefore generalize the desgin, implementation and deployment of all the work that we are going to do to the bridge reachability. That way an analyst can query the distributed database with a proper client that connects to the data repository ooni-backend API. So to sum up, I started talking about the bridge reachability visualization problem and finished with a much broader vision that intends to integrate the ongoing efforts of the bridge reachability to improve ooni as a whole. Hope the email is not too large. ciao [0] https://lists.torproject.org/pipermail/tor-dev/2014-October/007585.html [1] https://pad.riseup.net/p/bridgereachability [2] https://blog.torproject.org/blog/closer-look-great-firewall-china [3] http://ooniviz.chokepointproject.net/transports.htm [4] http://ooniviz.chokepointproject.net/successes.htm [5] https://people.torproject.org/~asn/bridget_vis/tbb_blocked_timeline.jpg [6] https://atlas.torproject.org [7] https://ooni.torproject.org/

3 5

ooniprobe v1.2.2 is out
by Arturo Filastò 17 Oct '14

17 Oct '14

Hi all, This mail is to inform you that a new version of ooniprobe has just been released. Here is the changelog: v1.2.2 (Fri, 17 Oct 2014) ------------------------- Who said friday 17th is only bad luck? * Add two new report entry keys test_start_time and test_runtime * Fix bug that lead to ooniresources not working properly It is now available for download from pypi: https://pypi.python.org/pypi/ooniprobe and soon will be available for download in debian. Have fun! ~ Arturo

1 0

OONI status report for September 2014
by Arturo Filastò 06 Oct '14

06 Oct '14

# Activities of September 2014 This month was a one dedicated mainly to system administration related tasks and on moving forwards the process of deploying OONI on M-Lab. All of the final details for the mlab deployment have been sorted out (also thanks to meeting with the mlab people in DC for the OTF summit) and should be set to have it deployed in the upcoming months. The bridge reachability project is also moving forward very well, with us having now probes in China, Iran, Russia, Ukraine, that run daily measurements to verify if a set of Tor bridges are working or not. Not all the collected data has yet been published since it also contains addresses of bridges that are not public. We also have released a new version of ooniprobe and oonibackend. You can read more about it's changelog here: https://lists.torproject.org/pipermail/ooni-dev/2014-October/000171.html Here are some of the tickets that we have solved this month: * Implemented support for making bouncer and client aware of policies: https://trac.torproject.org/projects/tor/ticket/12579 * Add support for writing oonibackend reports using the mlab format https://trac.torproject.org/projects/tor/ticket/13232 * Run a final end-to-end test https://trac.torproject.org/projects/tor/ticket/13289 * Get two boxes that are modern for running the bridge reachability tests: https://trac.torproject.org/projects/tor/ticket/12543 * Fix backward compatibility with twisted 13.1 in ooniprobe https://trac.torproject.org/projects/tor/ticket/13139 * Make ansible deployment script for bridge reachability probes on debian stable https://trac.torproject.org/projects/tor/ticket/12769 * Configure travis to run ooni-backend tests regularly with public reports https://trac.torproject.org/projects/tor/ticket/12244 * Attended the OTF summit in DC # Plans for October 2014 * Post process the data collected by the bridge DB collector https://trac.torproject.org/projects/tor/ticket/12546 * Sanitize .csv files https://trac.torproject.org/projects/tor/ticket/12548 * ooni's TrueHeaders uses a dict() for internal storage: https://trac.torproject.org/projects/tor/ticket/11991 * Make a test that measures both HTTP and DNS censorship: https://trac.torproject.org/projects/tor/ticket/11972 * ooni's TrueHeaders uses a dict() for internal storage: https://trac.torproject.org/projects/tor/ticket/11991 * Do not verify inputs if no policy is specified https://trac.torproject.org/projects/tor/ticket/12299 ~ Art.

1 0

Schedule change for ooni dev meeting at 19:00 CET on Thursday October 2 2014
by Arturo Filastò 01 Oct '14

01 Oct '14

Hello Oonitarians, This is a reminder to inform you that the weekly dev meeting will happen today at 19:00 CET Thursday October 2. Note: That the schedule has changed from Friday to Thursday. Adjust your calendaring software accordingly. See you! ~ Art.

1 0

Request to change the ooni meeting
by kudrom 01 Oct '14

01 Oct '14

Hi, As you all know the meeting of the ooni project is scheduled on fridays at 17:00 CET, and as some of you will know some of us have started the university classes in september, with the unfortunate personal event that i have a lab on friday from 16:00 to 18:00 CET, so i was wondering if the hour or day of the meeting can be changed. I propose on Thursdays at 17:00 CET ciao

2 2

ooniprobe 1.2.0 and oonibackend 1.1.4 are out!
by Arturo Filastò 01 Oct '14

01 Oct '14

Greetings! We are very pleased to announce the release of new versions of ooniprobe and oonibackend. One of the most interesting new features that are now part of ooniprobe is the ability to generate test decks for the country you are in a way that is much easier than before. As a matter of fact to start contributing useful measurements it's just a mater of 5 minutes of setup: https://pypi.python.org/pypi/ooniprobe#ooni-in-5-minutes Here are some more details of what this release includes: ooniprobe ========= * Introduce a new tool for generating ooniprobe test decks called oonideckgen. * Introduce a new tool for updating resources used for geoip lookup and deck generation. * Add support for policy aware bouncing in the client. https://trac.torproject.org/projects/tor/ticket/12579 * Various improvements to the bridge_reachability test (enable better tor logging and also log obfsproxy) * Fix backward compatibility with twisted 13.1 and add regression tests for this. https://trac.torproject.org/projects/tor/ticket/13139 oonibackend =========== * Add support for specifying the report archive path from config file * Write tor notice level logs to file * Fix bug that lead test helpers to not being started The new packages have been uploaded to pypy and can be found at the following links: https://pypi.python.org/pypi/ooniprobe https://pypi.python.org/pypi/oonibackend Soon ooniprobe will also be available inside of debian testing thanks to Lunar^. Have fun, ~ Art.

1 0

Complete list of Tor blocking events
by Sadia Afroz 30 Sep '14

30 Sep '14

Hi all, I am trying find a list of events when different countries blocked Tor to understand why and how those countries blocked it and how Tor resolved those issues. Do you know if anybody is maintaining such a list? I found censorshipwiki ( https://trac.torproject.org/projects/tor/wiki/doc/OONI/censorshipwiki) that has a similar list. But it seems that it's not maintained any more as it doesn't have the recent blocking incident by Iran. Also I'm not sure how comprehensive is this list as it's missing Tunisia. Do you have any suggestion on how to create a comprehensive list of Tor blocking by countries? My colleagues and I were thinking of two ways to go about this: 1. Get the total number of Tor users from every countries over time and look for anomalous fluctuations. When we created such a graph we noticed that huge number of bot controlled nodes that appear in almost every country's graph. It seems hard to identify real censorship events from graphs like this that can be affected by other events unrelated to censorship. 2. Search for tickets on https://trac.torproject.org/projects/tor <https://trac.torproject.org/projects/tor/ticket/12727> with keywords related to censorship, for example "censorship", "block", "dpi". This has the potential problem of missing many events where nobody complained or filed a bug report with different keywords. Let me know if there are other places I should look into to find censorship events. Thanks! -- Sadia

5 7

oonibackend Debian package
by jony-port＠bitmessage.ch 30 Sep '14

30 Sep '14

Hi, I was thinking that would be quite helpful creating a package for easy deployment of oonibackend similarly to ooniprobe debian packag. Is there any prior work in creating a Debian package for oonibackend?

2 1

Reminder OONI dev meeting happening tomorrow
by Arturo Filastò 18 Sep '14

18 Sep '14

Hi Oonitarians, This is a reminder to inform you that tomorrow (Friday 19th of September) there will the weekly dev meeting. It happens on IRC (irc.oftc.net) on the channel #ooni at 17:00 CET (15:00 UTC). Everyone is invited! See you tomorrow! ~ Art.

1 0

Rescheduling of OONI dev meeting of next week
by Arturo Filastò 05 Sep '14

05 Sep '14

Hello Oonitarians, Next week (on Friday 12th at 17:00 CET) we were supposed to have our weekly ooni dev meeting. Some of, though, will be at the OTF summit in DC and will therefore not be able to attend. After discussing this at todays weekly meeting we concluded to ask for feedback on whether we should skip next week and postpone it to next Friday 19th or if we should do it another day in the week of the 15th. If you have a strong opinion on doing either of these please let us know, if not it will by default be postponed to Friday 19th 17:00 CET. Have fun, ~ Art.

1 0