ooni-dev

ooni-dev@lists.torproject.org

306 discussions

Ubuntu PPA
by Iain R. Learmonth 11 Dec '14

11 Dec '14

Hi, Earlier in the week, an Ubuntu PPA was requested in IRC. I've since learnt how to put these together and now one exists [1]. Next up on this side of things I'll be looking at bug #10734 [2] for the Ubuntu package, and hopefully coming up with a patch that can be included easily in the Debian package too. Please do let me know how you get on if you use the PPA. Thanks, Iain. [1]: https://launchpad.net/~irl/+archive/ubuntu/ooni [2]: https://trac.torproject.org/projects/tor/ticket/10734 -- e: irl(a)fsfe.org w: iain.learmonth.me x: irl(a)jabber.fsfe.org t: +447875886930 c: 2M0STB g: IO87we p: 1F72 607C 5FF2 CCD5 3F01 600D 56FF 9EA4 E984 6C49

1 0

OONI Team Status Report for November
by Arturo Filastò 09 Dec '14

09 Dec '14

# What we did in November 2014? This month marks an important milestone. oonibackend is now deployed on the production mlab slices! This means that since November 26th probes running the http_header_field_manipulation test have been sending their reports to M-Lab production nodes. * Assist the M-Lab team in deploying oonibackend on the production slices * Set upper bound in comparison with the control in the bridge reachability timeline https://trac.torproject.org/projects/tor/ticket/13640 * Handle mlab-ns errors in update-bouncer.py script https://trac.torproject.org/projects/tor/ticket/13812 * Progress on Brainstorm ideas for possible visualisations: https://trac.torproject.org/projects/tor/ticket/13731 * Hold a small workshop on OONI data visualization in Rome: https://lists.torproject.org/pipermail/ooni-dev/2014-November/000194.html # What we plan to do in December 2014? * Align the dates in the visual timeline https://trac.torproject.org/projects/tor/ticket/13639 * Better tokenising for bridges so that bridges that have the same fingerprint, but different transport are grouped properly https://trac.torproject.org/projects/tor/ticket/13638 * Finish setting up the docker containers for the steps of the data pipeline https://trac.torproject.org/projects/tor/ticket/13568 * Setup disaster recovery procedure and backup: https://trac.torproject.org/projects/tor/ticket/13584 * Setup monitoring of the probes. https://trac.torproject.org/projects/tor/ticket/12549 * Add support for obfs4 https://trac.torproject.org/projects/tor/ticket/13597 * Implement some of the OONI tests inside of libight: https://github.com/bassosimone/libight/milestones/OONI%20tests%20implemente… ~ Arturo

1 0

Reminder for weekly OONI dev meeting Monday 18:00 UTC (19:00 CET)
by Arturo Filastò 01 Dec '14

01 Dec '14

Hi, This is a reminder that today there will be the weekly OONI meeting. As I am writing this I am on a plane heading towards London and will be working on a GlobaLeaks workshop in some office there. The schedule for the workshop is not very well defined so I am not 100% sure I will manage to be online at the above defined time. If I am not there please start without me and I will be sure to read the backlog and write feedback asynchronously. ~ Arturo

1 0

Summary of data visualization OONI workshop
by Arturo Filastò 26 Nov '14

26 Nov '14

Greetings! Yesterday we did a small OONI workshop in Rome to focus on thinking about the problem of data visualization. I think what came out of it is very useful, though it is not yet some fancy graphics visualizations. The main value comes from having some people that have very little prior knowledge of OONI think about the problem of data visualization and learn from them what is their framework of thinking. One of my personal favorite takeaways is that when we think about the results of OONI what we should be talking about are "network irregularities". In the end what we are interested in spotting are deviations from what is to be considered a "normal" network behavior. This may or may not be a symptom of internet censorship. It could also be caused by some mess-up by the network operator or some bugs in our monitoring software. The point is that when we spot an irregularity we should investigate it further to discover if it is in fact an instance of censorship or something else. Starting from this assumption we started designing what should be the variables that a user interested in analyzing the data collected so far should be adjusting. We set our target audience to be somebody that is interested in integrating the data into some work they are doing. What came up often as a practical example of this is "the journalist working on an article on freedom of expression in country X". At the end of the day we came up with this chart that shows what are the variables that should be combined together to produce different views: https://people.torproject.org/~art/ooni/ooni-workshop-0x02-irregularities.p… What basically happens here is that the user fixes the timeframe they are interested in looking at. Once that is set they will be able to say "Show me all the _interesting_ measurements that are related to the country|test_name|input X". I think that now Viviana and Silvia have a clearer understanding of how OONI works and what it's measuring and they will be working on producing some mockups of UIs for exploring the data in this way. I will then implement these mockups based on the new REST API we have to the data. If you have questions on feedback on this it's greatly appreciated so please contribute to either this thread or post on this ticket: https://trac.torproject.org/projects/tor/ticket/13731 Have fun! ~ Arturo

1 0

Next OONI weekly meetings
by Arturo Filastò 24 Nov '14

24 Nov '14

Hello Oonitarians, Sorry for not sending this email sooner, but this week has been a bit hectic. A couple of people have said that they cannot attend the weekly meeting on thursday at 18:00, because it conflicts with another recurring appointment they have. For this reason I am going to suggest we reschedule the next OONI meetings to a date that works for everybody. Monday and Tuesdays seem to be the best option, so I am going to propose we skip next weeks meeting and instead opt for holding next weeks one on either a Monday or Tuesday at 18:00 UTC, 19:00 CET, 13:00 EST, 12:00 CST, 10:00:00 PST. How does this sound for everybody? I personally would prefer Monday over Tuesday, but if you would instead prefer Tuesday please send a reply to the ooni-dev mailing list. If nobody replies I am going to assume every Monday works for you all and the next meeting will be on Monday 24th of November. Have fun! ~ Arturo

2 2

OONI weekly dev meeting reminder
by Arturo Filastò 13 Nov '14

13 Nov '14

Hello Oonitarians, This is reminder that today (Thursday 6th of November 2014) there will be the weekly ooni dev meeting in #ooni on irc.oftc.net. The time will be: 18:00 UTC 19:00 CET 13:00 EST 12:00 CST 10:00:00 PST You are all invited to attend! ~ Arturo

1 1

OONI hackfest summary
by Arturo Filastò 04 Nov '14

04 Nov '14

>From October 24th to 26th the OONI team gathered in Berlin for a hackfest. Around 20 people ended up showing up and although most of them were seasoned Oonitarians some fairly new people joined us that I hope will become part of the growing OONI community. The scope of the hackfest was that of data analytics and visualization with special focus on the Tor bridge reachability study we are currently doing. # Bridge reachability study The goal of this study [1] is that of answering some questions concerning the blocking of Tor bridges [2] and pluggable transport [3] enabled bridges in the countries of China, Iran, Russia and Ukraine (test vantage points). To establish a baseline to eliminate the cases in which the bridge is marked as blocked, while it is in fact just offline, we measure also from a vantage point located in the Netherlands. For every test vantage point we perform two types of measurements: * A Bridge reachability measurement [4][5] that attempts to build a tor circuit using the bridge in question * A TCP connect measurement [6][7] that simply does a TCP connect to the bridge IP and port We run both of the measurements to further debug the reason why the blocking is happening, may this be due to a TCP RST or direct IP blocking or tor malfunction. So far this study has been running for a little less than 1 month. # OONI data pipeline In order to produce the aggregate data needed to build visualizations we have built a data pipeline [8] This consists of a series of operations that are done to the raw reports in order to strip out sensitive information and place the collected data into a database. The nice thing is that the data pipeline we have designed is not specific to this study, but can and will be in the future expanded to export data needed to visualize also the other types of measurements done by OONI. The data pipeline is comprised of 3 steps (or states, depending on how you want to look at it). When the data is submitted to a OONI collector it is synchronized with the aggregator. This is a central machine responsible for running all the data processing tasks, storing the collected data in a database and hosting a public interface to the sanitised reports. Since all the steps are independent from one another it is not necessary that they run on the machine, but it may also be more distributed. Once the data is on the aggregator machine it is said to be in the RAW state. The sanitise task is then run on the RAW data to remove sensitive information and strip out some superfluous information. A RAW copy of every report is also stored in a private compressed archive for future reference. Once the data is sanitised it is said to tbe in SANITISED state. At this point a import task is run on the data to place it inside of a database. The SANITISED reports are then place in a directory that is publicly exposed to the internet to allow people to download also a copy of the YAML reports. At this point is is possible to run any export task that performs queries on the database and produces as output some documents to be used in the data visualizations (think JSON, CSV, etc.). # The OONI hackfest The first day of the hackfest was spent going over the scope of the project we would be working on in the following days as well as working in groups that were interested in tacking the design of one aspect of the problem. Sticky notes were plentiful and helped us have a clear vision of what lied ahead of us. By the end of the first day we had clear what were the set of tasks that were needed to achieve our goals and which teams would be responsible for doing what. The second day was almost entirely dedicated to hacking and everybody had a task to complete that was either completed by the end of the day or sooner. Some people even completed their initially assigned task before the end of the day and came back asking for more! By the end of the second day we had a real data set to hand over to the visualization team, to start producing some pretty graphs based on real data. We decided that the first visualization we wanted to do should be kept as simple as possible and be something that we could also use to debug the data we had collected. It should tell us which bridges were working when and it should present the information in a way that would highlight the country involved and the pluggable transport type. A prototype of it can be seen here: http://reports.ooni.nu/analytics/bridge_reachability/timeline/ The code for this visualization can be found here: https://github.com/Shidash/OONI-Bridge-Reachability-Timeline # Next steps * Write scripts for generating the bridge_db.json document based on the data that is given to us from the bridge db team https://trac.torproject.org/projects/tor/ticket/13570 * Align the dates in the visual timeline https://trac.torproject.org/projects/tor/ticket/13639 * Better tokenising for bridges so that bridges that have the same fingerprint, but different transport are grouped properly https://trac.torproject.org/projects/tor/ticket/13638 * Finish setting up the docker containers for the steps of the data pipeline https://trac.torproject.org/projects/tor/ticket/13568 * Setup disaster recovery procedure and backup: https://trac.torproject.org/projects/tor/ticket/13584 * Setup monitoring of the probes. https://trac.torproject.org/projects/tor/ticket/12549 * Add support for obfs4 https://trac.torproject.org/projects/tor/ticket/13597 * Set upper bound in comparison with the control in the bridge reachability timeline https://trac.torproject.org/projects/tor/ticket/13640 * Make sure that the control measurement is for the specific bridge measurement https://trac.torproject.org/projects/tor/ticket/13655 Questions and comments should be directed to the ooni-dev mailing list or to the #ooni channel on irc.oftc.net. Have fun! ~ Arturo [1] https://lists.torproject.org/pipermail/ooni-dev/2014-October/000184.html [2] https://www.torproject.org/docs/bridges [3] https://www.torproject.org/docs/pluggable-transports.html.en [4] https://gitweb.torproject.org/ooni/spec.git/blob/HEAD:/test-specs/ts-011-br… [5] https://gitweb.torproject.org/ooni-probe.git/blob/HEAD:/ooni/nettests/block… [6] https://gitweb.torproject.org/ooni/spec.git/blob/HEAD:/test-specs/ts-008-tc… [7] https://gitweb.torproject.org/ooni-probe.git/blob/HEAD:/ooni/nettests/block… [8] https://github.com/TheTorProject/ooni-pipeline/blob/master/Readme.md#ooni-p…

2 1

OONI team status report for October 2014
by Arturo Filastò 03 Nov '14

03 Nov '14

# What we did in October 2014 ## ooniprobe * We released ooniprobe 1.2.0 and 1.2.2 that include two new tools oonideckgen and ooniresources making it possible to generate test decks for a specific country. For a full changelog see: https://gitweb.torproject.org/ooni-probe.git/blob/HEAD:/ChangeLog.rst#l13 https://gitweb.torproject.org/ooni-probe.git/blob/HEAD:/ChangeLog.rst#l4 * Add some extra keys to the ooniprobe report output: https://github.com/TheTorProject/ooni-spec/commit/73aa34f1123af603fab00d00b… https://github.com/TheTorProject/ooni-probe/commit/b59d9343354990a39a87fd6e… ## oonibackend * Fix bug in backend: https://github.com/TheTorProject/ooni-backend/commit/7e95433b66cb96b5c524a7… ## oonipipeline This is a new software component that deals with processing raw ooniprobe reports, placing them in a database and exposing them to the web. An overview of this software component can be found here: https://github.com/TheTorProject/ooni-pipeline#ooni-pipeline * Work on the script for putting the sanitised reports into the database https://trac.torproject.org/projects/tor/ticket/13564 * Work on script that generates the data formats for visualisation team https://trac.torproject.org/projects/tor/ticket/13565 * Sanitize .csv files https://trac.torproject.org/projects/tor/ticket/12548 * Post process the data collected by the bridge DB collector https://trac.torproject.org/projects/tor/ticket/12546 ## Data visualisation * Produce a prototype of the graphs for the bridge reachability study. ## Misc. * Organise and attend a hackfest in berlin to bootstrap the bridge reachability study. * Meet with censorship researchers in NYC during an event organised by mlab. # Next steps for November 2014 * Write scripts for generating the bridge_db.json document based on the data that is given to us from the bridge db team https://trac.torproject.org/projects/tor/ticket/13570 * Align the dates in the visual timeline https://trac.torproject.org/projects/tor/ticket/13639 * Better tokenising for bridges so that bridges that have the same fingerprint, but different transport are grouped properly https://trac.torproject.org/projects/tor/ticket/13638 * Finish setting up the docker containers for the steps of the data pipeline https://trac.torproject.org/projects/tor/ticket/13568 * Setup disaster recovery procedure and backup: https://trac.torproject.org/projects/tor/ticket/13584 * Setup monitoring of the probes. https://trac.torproject.org/projects/tor/ticket/12549 * Add support for obfs4 https://trac.torproject.org/projects/tor/ticket/13597 * Set upper bound in comparison with the control in the bridge reachability timeline https://trac.torproject.org/projects/tor/ticket/13640

1 0

david's notes from yesterday
by David Stainton 25 Oct '14

25 Oct '14

* TODO create complete disaster recover procedure for Agregator server This can for example be a manual procedure... but in that case it must be fully documented. For instance the docs should specify how to build new Aggregator server given a freshly installed Debian stable server; this should involve copying some Docker scripts and running them to build Docker images that are configured properly for the different components. * Aggregator What are the different Aggregator Docker images? ** synchronization/subscriber of publishing collectors *** TODO create a Docker image to rsync data from the collectors and trigger removal afterwards This task can initially use rsync... but later will be replaced to a pub/sub system. ** YAML data sanitizer *** TODO create YAML data sanitizer Docker image This task makes the yaml reports data suitable to be published AND prepares for database import. ** data post processing + DB import *** TODO create post processing Docker image analyses the yaml reports creates database entries - only this Docker image contains mongodb write access credentials * non-aggregator Docker images ** mongodb This Docker image runs the MongoDB service. 1st iteration: listens on loopback interface 2nd iteration: move this Docker image to a different machine and listen to a publicly routed IP address. Must use crypto wire protocol + authentication ** web mirror *** TODO create docker image of web mirror * stateful directory structure ** task 0 rsync data to /data/raw # example: data/bouncer-nkvphnp3p6agi5qq/archive # /data/collector/archive ** task 1 sanitize data and move to /data/staging # example: none ** state 2 post-processing to prepare to DB import move data to /data/reports # example: none ** state 3 data that has been imported into the DB move data to /de # example: /data/reports * stateful directory backup strategy The backup strategy for the above 3-part stateful directory workflow is to make backup snapshots of all three state directories. * collector to aggregator state #1 For the moment we'll use rsync to copy the data from the collectors to the Aggregator. After rsyncing, the data will then be removed from the Collectors. Later iterations will involve a pub/sub system between the Collectors and Aggregators.

1 0

Background on bridge reachability project
by Arturo Filastò 24 Oct '14

24 Oct '14

Hello Oonitarians! To get an overall idea of the scope of the problem read the meeting proposal that explains to funders what we are trying to achieve with the hackfest [1] Most of the things in there you probably already know. During a meeting with some people from the visualization team we collected some notes about the infrastructure we need to setup and the sort of questions we are trying to answer with the data that we collect [2]. Now All of the infrastructure detailed there is in place and we have been collecting data from those 5 machines since about a month. To get a feeling of what the data we will be analyzing looks like see the published bridge reachability reports from China [3]. The ones from Russia, Ukraine, Iran are not yet published, but you will gain access to them during the meeting (though some of them first need to be redacted to strip out private bridge IPs). In this thread [4] what is in the notes above, is expanded and I recommend you read a bit of it. Some people have already started to do some mockups of visualization and you can see it here: Donuts : http://ooniviz.chokepointproject.net/donuts.htm Bar charts : http://ooniviz.chokepointproject.net/transports.htm Multi-line charts : http://ooniviz.chokepointproject.net/successes.htm Time-lines : http://ooniviz.chokepointproject.net/timelines.htm Let me know if you have any questions and if you want this evening we can get dinner or grab drinks later. We are going to be in Prenzlauer Berg at the address that you should have gotten in the other email. See you soon! ~ Arturo [1] https://pad.riseup.net/p/bridgereachability_meeting_proposal [2] https://pad.riseup.net/p/bridgereachability [3] https://ooni.torproject.org/reports/0.1/CN/ [4] https://lists.torproject.org/pipermail/ooni-dev/2014-October/000176.html

1 0

← Newer
1
...
19
20
21
22
23
24
25
...
31
Older →

Jump to page:

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

ooni-dev