January 2017 - tor-commits - lists.torproject.org

[ooni-probe/master] Add requirements needed to build docs
by art＠torproject.org 13 Jan '17

13 Jan '17

commit 289b9af7d9041871fca4a782ba292a39e083e153 Author: Arturo Filastò <arturo(a)filasto.net> Date: Mon Nov 21 18:26:26 2016 +0000 Add requirements needed to build docs --- requirements-docs.txt | 3 +++ 1 file changed, 3 insertions(+) diff --git a/requirements-docs.txt b/requirements-docs.txt new file mode 100644 index 0000000..0b9e096 --- /dev/null +++ b/requirements-docs.txt @@ -0,0 +1,3 @@ +sphinx +ghp-import +sphinx_rtd_theme

1 0

[ooni-probe/master] Update manpages
by art＠torproject.org 13 Jan '17

13 Jan '17

commit 03ec2d760cb38dedcee2c78d3d586ab3b5a33a9e Author: Arturo Filastò <arturo(a)filasto.net> Date: Sat Oct 29 22:57:37 2016 +0200 Update manpages --- data/oonideckgen.1 | 2 +- data/ooniprobe-agent.1 | 2 +- data/ooniprobe.1 | 2 +- data/oonireport.1 | 2 +- data/ooniresources.1 | 2 +- 5 files changed, 5 insertions(+), 5 deletions(-) diff --git a/data/oonideckgen.1 b/data/oonideckgen.1 index e3bcf62..3046041 100644 --- a/data/oonideckgen.1 +++ b/data/oonideckgen.1 @@ -1,6 +1,6 @@ .\" Man page generated from reStructuredText. . -.TH "OONIDECKGEN" "1" "October 14, 2016" "2.0.0" "OONI: Open Observatory of Network Interference" +.TH "OONIDECKGEN" "1" "October 29, 2016" "2.0.1" "OONI: Open Observatory of Network Interference" .SH NAME oonideckgen \- DEPRECATED ooni tool . diff --git a/data/ooniprobe-agent.1 b/data/ooniprobe-agent.1 index 61ace5a..ff25fd1 100644 --- a/data/ooniprobe-agent.1 +++ b/data/ooniprobe-agent.1 @@ -1,6 +1,6 @@ .\" Man page generated from reStructuredText. . -.TH "OONIPROBE-AGENT" "1" "October 14, 2016" "2.0.0" "OONI: Open Observatory of Network Interference" +.TH "OONIPROBE-AGENT" "1" "October 29, 2016" "2.0.1" "OONI: Open Observatory of Network Interference" .SH NAME ooniprobe-agent \- an internet censorship measurement tool, system daemon. . diff --git a/data/ooniprobe.1 b/data/ooniprobe.1 index 5f64a0a..0852165 100644 --- a/data/ooniprobe.1 +++ b/data/ooniprobe.1 @@ -1,6 +1,6 @@ .\" Man page generated from reStructuredText. . -.TH "OONIPROBE" "1" "October 14, 2016" "2.0.0" "OONI: Open Observatory of Network Interference" +.TH "OONIPROBE" "1" "October 29, 2016" "2.0.1" "OONI: Open Observatory of Network Interference" .SH NAME ooniprobe \- an internet censorship measurement tool . diff --git a/data/oonireport.1 b/data/oonireport.1 index 5c611ff..caa6cb7 100644 --- a/data/oonireport.1 +++ b/data/oonireport.1 @@ -1,6 +1,6 @@ .\" Man page generated from reStructuredText. . -.TH "OONIREPORT" "1" "October 14, 2016" "2.0.0" "OONI: Open Observatory of Network Interference" +.TH "OONIREPORT" "1" "October 29, 2016" "2.0.1" "OONI: Open Observatory of Network Interference" .SH NAME oonireport \- upload ooniprobe measurements . diff --git a/data/ooniresources.1 b/data/ooniresources.1 index 16b823d..3fb9eb8 100644 --- a/data/ooniresources.1 +++ b/data/ooniresources.1 @@ -1,6 +1,6 @@ .\" Man page generated from reStructuredText. . -.TH "OONIRESOURCES" "1" "October 14, 2016" "2.0.0" "OONI: Open Observatory of Network Interference" +.TH "OONIRESOURCES" "1" "October 29, 2016" "2.0.1" "OONI: Open Observatory of Network Interference" .SH NAME ooniresources \- DEPRECATED ooni tool .

1 0

[ooni-probe/master] Bump version number of master to 2.0.3.dev0
by art＠torproject.org 13 Jan '17

13 Jan '17

commit dd0a05a0fa0fd3b69ef8d7b29777ba7ffde2b74b Author: Arturo Filastò <arturo(a)filasto.net> Date: Wed Nov 16 15:02:19 2016 +0100 Bump version number of master to 2.0.3.dev0 --- ooni/__init__.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/ooni/__init__.py b/ooni/__init__.py index be61c65..9d479c9 100644 --- a/ooni/__init__.py +++ b/ooni/__init__.py @@ -1,7 +1,7 @@ # -*- encoding: utf-8 -*- __author__ = "Open Observatory of Network Interference" -__version__ = "2.0.2" +__version__ = "2.0.3.dev0" __all__ = [ 'agent',

1 0

[ooni-probe/master] Add ooniprobe v2.0.0 install instructions, cleanup, fix URLs (#649)
by art＠torproject.org 13 Jan '17

13 Jan '17

commit e6035577f33b8d34a10117bf30bffda45719896a Author: anadahz <anadahz(a)users.noreply.github.com> Date: Mon Nov 21 18:25:30 2016 +0000 Add ooniprobe v2.0.0 install instructions, cleanup, fix URLs (#649) * Add ooniprobe v2.0.0 install instructions, cleanup, fix URLs * Add installation instructions for ooniprobe v2.0.0 * Update git repository URLs * Cleanup obsolete old documentation * Remove extra dependencies, shape service install instructions * Add ooniprobe package install instructions fix port,vagrant option * Add Debian and Ubuntu package based install instructions * Fix web UI port and vagrant option * Remove end of life wily distribution install instructions --- README.rst | 229 ++++++++++++++++++------------------------------------------- 1 file changed, 67 insertions(+), 162 deletions(-) diff --git a/README.rst b/README.rst index d232188..0b9d3d2 100644 --- a/README.rst +++ b/README.rst @@ -47,50 +47,46 @@ probe is a liability for you, please be aware of this risk. OONI in 5 minutes ================= -On Debian testing or unstable:: - - sudo apt-get install ooniprobe +The latest ooniprobe version for Debian and Ubuntu releases can be found in the +deb.torproject.org package repository. On Debian stable (jessie):: - echo 'deb http://ftp.debian.org/debian jessie-backports main' | sudo tee -a /etc/apt/sources.list + gpg --keyserver keys.gnupg.net --recv A3C4F0F979CAA22CDBA8F512EE8CBC9E886DDD89 + gpg --export A3C4F0F979CAA22CDBA8F512EE8CBC9E886DDD89 | sudo apt-key add - + echo 'deb http://deb.torproject.org/torproject.org jessie main' | sudo tee /etc/apt/sources.list.d/ooniprobe.list sudo apt-get update - sudo apt-get install ooniprobe + sudo apt-get install ooniprobe deb.torproject.org-keyring -On Ubuntu 16.04 (xenial), 15.10 (wily) or 14.04 (trusty):: +On Debian testing:: - echo 'deb http://deb.torproject.org/torproject.org $RELEASE main' | sudo tee -a /etc/apt/sources.list + gpg --keyserver keys.gnupg.net --recv A3C4F0F979CAA22CDBA8F512EE8CBC9E886DDD89 + gpg --export A3C4F0F979CAA22CDBA8F512EE8CBC9E886DDD89 | sudo apt-key add - + echo 'deb http://deb.torproject.org/torproject.org testing main' | sudo tee /etc/apt/sources.list.d/ooniprobe.list sudo apt-get update - sudo apt-get install ooniprobe - -Note: You'll need to swap out ``$RELEASE`` for either ``xenial``, ``wily``, or -``trusty``. This will not happen automatically. You will also need to ensure -that you have the ``universe`` repository enabled. The ``universe`` repository -is enabled by default in a standard Ubuntu installation but may not be on some -minimal on not standard installations. - -On unix systems:: - - sudo pip install ooniprobe - -To install it from the current master run:: + sudo apt-get install ooniprobe deb.torproject.org-keyring - sudo pip install https://github.com/TheTorProject/ooni-probe/archive/master.zip +On Debian unstable:: -Then run:: - - mkdir my_decks - oonideckgen -o my_decks/ - -If you're using the Debian package, you will be asked when installing whether -you would like to run OONI daily. On other platforms, if you would like to -contribute measurements to OONI daily you can also add this to your crontab:: + gpg --keyserver keys.gnupg.net --recv A3C4F0F979CAA22CDBA8F512EE8CBC9E886DDD89 + gpg --export A3C4F0F979CAA22CDBA8F512EE8CBC9E886DDD89 | sudo apt-key add - + echo 'deb http://deb.torproject.org/torproject.org unstable main' | sudo tee /etc/apt/sources.list.d/ooniprobe.list + sudo apt-get update + sudo apt-get install ooniprobe deb.torproject.org-keyring - @daily ooniprobe $THE_OONI_COMMAND +On Ubuntu 16.10 (yakkety), 16.04 (xenial) or 14.04 (trusty):: -Run this command to automatically update your crontab:: + gpg --keyserver keys.gnupg.net --recv A3C4F0F979CAA22CDBA8F512EE8CBC9E886DDD89 + gpg --export A3C4F0F979CAA22CDBA8F512EE8CBC9E886DDD89 | sudo apt-key add - + echo 'deb http://deb.torproject.org/torproject.org $RELEASE main' | sudo tee /etc/apt/sources.list.d/ooniprobe.list + sudo apt-get update + sudo apt-get install ooniprobe deb.torproject.org-keyring - (crontab -l 2>/dev/null; echo "@daily ooniprobe $THE_OONI_COMMAND") | crontab - +Note: You'll need to swap out ``$RELEASE`` for either ``yakkety``, ``xenial`` or +``trusty``. This will not happen automatically. You will also need to ensure +that you have the ``universe`` repository enabled. The ``universe`` repository +is enabled by default in a standard Ubuntu installation but may not be on some +minimal on not standard installations. Installation ============ @@ -118,15 +114,19 @@ Make sure you have installed the following dependencies: * libffi-dev * tor (>=0.2.5.1 to run all the tor related tests) +Optional dependencies: + + * obfs4proxy + On debian based systems this can generally be done by running:: - sudo apt-get install -y build-essential libdumbnet-dev libpcap-dev libgeoip-dev libffi-dev python-dev python-pip tor + sudo apt-get install -y build-essential libdumbnet-dev libpcap-dev libgeoip-dev libffi-dev python-dev python-pip tor libssl-dev obfs4proxy Then you should be able to install ooniprobe by running:: sudo pip install ooniprobe -or as a user to install inside a virtualenv:: +or install ooniprobe as a user:: pip install ooniprobe @@ -148,20 +148,20 @@ On debian/ubuntu:: 3. Open a Terminal and run:: - git clone https://git.torproject.org/ooni-probe.git - git clone https://github.com/TheTorProject/ooni-backend.git + git clone https://github.com/TheTorProject/ooni-probe.git cd ooni-probe/ vagrant up 4. Login to the box with:: - vagrant ssh + vagrant ssh probe + +5. Start ooniprobe agent:: -ooniprobe will be installed in ``/ooni``. + ooniprobe-agent start -5. You can run tests with:: +6. Connect to the web UI on your host machine at http://localhost:8842/ - ooniprobe blocking/http_requests -f /ooni/var/example_inputs/alexa-top-1k.txt Using ooniprobe =============== @@ -179,134 +179,41 @@ Using ooniprobe Configuring ooniprobe --------------------- -You may edit the configuration for ooniprobe by editing the configuration file -found inside of ``~/.ooni/ooniprobe.conf``. - -By default ooniprobe will not include personal identifying information in the -test result, nor create a pcap file. This behavior can be personalized. - - -Updating resources ------------------- - -To generate decks you will have to update the input resources of ooniprobe. - -This can be done with:: - - ooniresources - -If you get a permission error, you may have to run the command as root or -change the ooniprobe data directory inside of `ooniprobe.conf`. - -Generating decks ----------------- - -You can generate decks for your country thanks to the oonideckgen command. - -If you wish, for example, to generate a deck to be run in the country of Italy, -you can do so (be sure to have updated the input resources first) by running:: - - oonideckgen --country-code IT --output ~/ - -You will now have in your home a folder called `deck-it`, containing the ooni -deck (ends with .deck) and the inputs. -Note: that you should not move the `deck-*` directory once it has been -generated as the paths to the inputs referenced by the test in the deck are -absolute. If you want your deck to live in another directory you must -regenerated it. - - -Running decks -------------- - -You will find all the installed decks inside of ``/usr/share/ooni/decks``. +After successfully installing ooniprobe you should be able to access the web UI +on your host machine at http://localhost:8842/ . -You may then run a deck by using the command line option ``-i``: +You should now be presented with the web UI setup wizard where you can read the +risks involved with running ooniprobe. Upon answering the quiz correctly you can +enable or disable ooniprobe tests, set how you can connect to the measurement's +collector and finally configure your privacy settings. -As root:: - - ooniprobe -i /usr/share/ooni/decks/mlab.deck - - -Or as a user:: - - ooniprobe -i /usr/share/ooni/decks/mlab_no_root.deck - - -Or: - -As root:: - - ooniprobe -i /usr/share/ooni/decks/complete.deck - - -Or as a user:: - - ooniprobe -i /usr/share/ooni/decks/complete_no_root.deck - - -The above tests will require around 20-30 minutes to complete depending on your network speed. - -If you would prefer to run some faster tests you should run: -As root:: - - ooniprobe -i /usr/share/ooni/decks/fast.deck - - -Or as a user:: - - ooniprobe -i /usr/share/ooni/decks/fast_no_root.deck - - -Running net tests ------------------ - -You may list all the installed stable net tests with:: - - - ooniprobe -s - - -You may then run a nettest by specifying its name for example:: - - - ooniprobe manipulation/http_header_field_manipulation - - -It is also possible to specify inputs to tests as URLs:: - - - ooniprobe blocking/http_requests -f httpo://ihiderha53f36lsd.onion/input/37e60e13536f6afe47a830bfb6b371b5cf65da66d7ad65137344679b24fdccd1 - - -You can find the result of the test in your current working directory. - -By default the report result will be collected by the default ooni collector -and the addresses of test helpers will be obtained from the default bouncer. +By default ooniprobe will not include personal identifying information in the +test results, nor create a pcap file. This behavior can be personalized. -You may also specify your own collector or bouncer with the options ``-c`` and -``-b``. +Run ooniprobe as a service (systemd) +------------------------------------ +Upon ooniprobe version 2.0.0 there is no need for cronjobs as ooniprobe-agent is +responsible for the tasks scheduling. -Bridges and obfsproxy bridges -============================= +You can ensure that ooniprobe-agent is always running by installing and enabling +the systemd unit `ooniprobe.service`:: -ooniprobe submits reports to oonib report collectors through Tor to a hidden -service endpoint. By default, ooniprobe uses the installed system Tor, but can -also be configured to launch Tor (see the advanced.start_tor option in -ooniprobe.conf), and ooniprobe supports bridges (and obfsproxy bridges, if -obfsproxy is installed). The tor.bridges option in ooniprobe.conf sets the path -to a file that should contain a set of "bridge" lines (of the same format as -used in torrc, and as returned by https://bridges.torproject.org) If obfsproxy -bridges are to be used, the path to the obfsproxy binary must be configured. -See option advanced.obfsproxy_binary, in ooniprobe.conf. + wget https://raw.githubusercontent.com/TheTorProject/ooni-probe/master/scripts/s… --directory-prefix=/etc/systemd/system + systemctl enable ooniprobe + systemctl start ooniprobe -(Optional) Install obfsproxy ----------------------------- +You should be able to see a similar output if ooniprobe (systemd) service is +active and loaded by running `systemctl status ooniprobe`:: -Install the latest version of obfsproxy for your platform. + ● ooniprobe.service - ooniprobe.service, network interference detection tool + Loaded: loaded (/etc/systemd/system/ooniprobe.service; enabled) + Active: active (running) since Thu 2016-10-20 09:17:42 UTC; 16s ago + Process: 311 ExecStart=/usr/local/bin/ooniprobe-agent start (code=exited, status=0/SUCCESS) + Main PID: 390 (ooniprobe-agent) + CGroup: /system.slice/ooniprobe.service + └─390 /usr/bin/python /usr/local/bin/ooniprobe-agent start -Download Obfsproxy: https://www.torproject.org/projects/obfsproxy.html.en Setting capabilities on your virtualenv python binary ===================================================== @@ -332,10 +239,8 @@ Contributing You can download the code for ooniprobe from the following git repository:: - git clone https://git.torproject.org/ooni-probe.git - + git clone https://github.com/TheTorProject/ooni-probe.git -It is also viewable on the web via: https://gitweb.torproject.org/ooni-probe.git. You should then submit patches for review as pull requests to this github repository:

1 0

[ooni-probe/master] Update to version 2.0.2
by art＠torproject.org 13 Jan '17

13 Jan '17

commit e5a89c46ac98f25c1339ffb3f9f140746ad7c878 Author: Arturo Filastò <arturo(a)filasto.net> Date: Tue Nov 15 21:05:40 2016 +0100 Update to version 2.0.2 * Add ChangeLog entry for 2.0.2 * Bump version * Update manpages --- ChangeLog.rst | 12 ++++++++++++ data/oonideckgen.1 | 2 +- data/ooniprobe-agent.1 | 2 +- data/ooniprobe.1 | 2 +- data/oonireport.1 | 2 +- data/ooniresources.1 | 2 +- ooni/__init__.py | 2 +- 7 files changed, 18 insertions(+), 6 deletions(-) diff --git a/ChangeLog.rst b/ChangeLog.rst index b2b8195..7798774 100644 --- a/ChangeLog.rst +++ b/ChangeLog.rst @@ -1,6 +1,18 @@ Changelog ========= +v2.0.2 (Wed, 16 Nov 2016) +-------------------------- + +Improvements: + +* Add support for specifying how the log rotation should happen + +Bug fixes: + +* Fix bug that lead to too many log lines being written: + https://github.com/TheTorProject/ooni-probe/issues/663 + v2.0.1 (Sat, 29 Oct 2016) -------------------------- diff --git a/data/oonideckgen.1 b/data/oonideckgen.1 index 3046041..466bf4d 100644 --- a/data/oonideckgen.1 +++ b/data/oonideckgen.1 @@ -1,6 +1,6 @@ .\" Man page generated from reStructuredText. . -.TH "OONIDECKGEN" "1" "October 29, 2016" "2.0.1" "OONI: Open Observatory of Network Interference" +.TH "OONIDECKGEN" "1" "November 16, 2016" "2.0.2" "OONI: Open Observatory of Network Interference" .SH NAME oonideckgen \- DEPRECATED ooni tool . diff --git a/data/ooniprobe-agent.1 b/data/ooniprobe-agent.1 index ff25fd1..889675e 100644 --- a/data/ooniprobe-agent.1 +++ b/data/ooniprobe-agent.1 @@ -1,6 +1,6 @@ .\" Man page generated from reStructuredText. . -.TH "OONIPROBE-AGENT" "1" "October 29, 2016" "2.0.1" "OONI: Open Observatory of Network Interference" +.TH "OONIPROBE-AGENT" "1" "November 16, 2016" "2.0.2" "OONI: Open Observatory of Network Interference" .SH NAME ooniprobe-agent \- an internet censorship measurement tool, system daemon. . diff --git a/data/ooniprobe.1 b/data/ooniprobe.1 index 0852165..743042e 100644 --- a/data/ooniprobe.1 +++ b/data/ooniprobe.1 @@ -1,6 +1,6 @@ .\" Man page generated from reStructuredText. . -.TH "OONIPROBE" "1" "October 29, 2016" "2.0.1" "OONI: Open Observatory of Network Interference" +.TH "OONIPROBE" "1" "November 16, 2016" "2.0.2" "OONI: Open Observatory of Network Interference" .SH NAME ooniprobe \- an internet censorship measurement tool . diff --git a/data/oonireport.1 b/data/oonireport.1 index caa6cb7..9b4edd8 100644 --- a/data/oonireport.1 +++ b/data/oonireport.1 @@ -1,6 +1,6 @@ .\" Man page generated from reStructuredText. . -.TH "OONIREPORT" "1" "October 29, 2016" "2.0.1" "OONI: Open Observatory of Network Interference" +.TH "OONIREPORT" "1" "November 16, 2016" "2.0.2" "OONI: Open Observatory of Network Interference" .SH NAME oonireport \- upload ooniprobe measurements . diff --git a/data/ooniresources.1 b/data/ooniresources.1 index 3fb9eb8..c0d5b7a 100644 --- a/data/ooniresources.1 +++ b/data/ooniresources.1 @@ -1,6 +1,6 @@ .\" Man page generated from reStructuredText. . -.TH "OONIRESOURCES" "1" "October 29, 2016" "2.0.1" "OONI: Open Observatory of Network Interference" +.TH "OONIRESOURCES" "1" "November 16, 2016" "2.0.2" "OONI: Open Observatory of Network Interference" .SH NAME ooniresources \- DEPRECATED ooni tool . diff --git a/ooni/__init__.py b/ooni/__init__.py index 385054e..be61c65 100644 --- a/ooni/__init__.py +++ b/ooni/__init__.py @@ -1,7 +1,7 @@ # -*- encoding: utf-8 -*- __author__ = "Open Observatory of Network Interference" -__version__ = "2.0.2.dev0" +__version__ = "2.0.2" __all__ = [ 'agent',

1 0

[ooni-probe/master] First pass at freshening up the architecture document (#580)
by art＠torproject.org 13 Jan '17

13 Jan '17

commit 128531b9bc1557d5d36580651bac9caaeee50a95 Author: Arturo Filastò <arturo(a)filasto.net> Date: Tue Nov 22 12:37:23 2016 +0000 First pass at freshening up the architecture document (#580) --- docs/source/architecture.rst | 351 ++++++++++++++++++++++++------------------- docs/source/conf.py | 2 +- 2 files changed, 201 insertions(+), 152 deletions(-) diff --git a/docs/source/architecture.rst b/docs/source/architecture.rst index 5e7d253..921fcbb 100644 --- a/docs/source/architecture.rst +++ b/docs/source/architecture.rst @@ -1,32 +1,119 @@ Architecture ============ -The goal of this document is provide an overview of how ooni works, what are -it's pieces and how they interact with one another. +Last Updated: 2016-08-01 -Keep in mind that this is the *big picture* and not all of the features and -compontent detailed here are implemented. -To get an idea of what is implemented and with what sort of quality see the -`Implementation status`_ section of this page. +The purpose of this goal is to illustrate the design goals of the various +components part of the OONI ecosystem, how they work and what is the +relationship between each other. -The two main components of ooni are `oonib`_ and `ooniprobe`_. +The following diagram gives you an idea of how the various OONI components +are related to each other. -.. image:: _static/images/ooniprobe-architecture.png - :width: 700px +.. graphviz:: -ooniprobe ---------- + digraph Architecture { -ooniprobe the client side component of ooni that is responsible for performing + subgraph cluster_0 { + style=filled; + color=lightgrey; + node [style=filled,color=white]; + "ooni-probe"; + "measurement-kit"; + label="clients"; + } + + "ooni-probe" -> "ooni-backend"; + "measurement-kit" -> "ooni-backend"; + "ooni-wui" -> "ooni-probe"; + "lepidopter" -> "ooni-probe"; + "ooni-backend" -> "ooni-pipeline"; + "ooni-pipeline" -> "ooni-explorer"; + } + + +The main software components are the following: + +* ooni-probe_: what users interested in contributing measurements will run. + It also includes a web based user interface for running measurements and + inspecting the results. + code repository: `<https://github.com/TheTorProject/ooni-probe>`_ + +* measurement-kit_: a portable C++ library that implements some ooniprobe + tests and is currently being used to port ooniprobe to mobile platforms + (Android and iOS). + In the future the measurement engine of ooniprobe will be replaced with + measurement-kit. + code repository: `<https://github.com/measurement-kit/measurement-kit>`_ + +* ooni-backend_: the software component that measurement clients communicate + with to learn the address of where they should submit results, submit results + (collector) and run certain tests against (see: `Test Helpers`_). + code repository: `<https://github.com/TheTorProject/ooni-backend>`_ + +* ooni-pipeline_: responsible for taking raw measurement data (from + collectors) normalising it, extracting insight from it and preparing it for + being presented inside of the `ooni-explorer`_ interface. + code repository: `<https://github.com/TheTorProject/ooni-pipeline>`_ + +* ooni-explorer_: a web front-end to the measurements collected by the OONI + platform. It features a world map view showcasing the countries where we have + identified network anomalies. + code repository: `<https://github.com/TheTorProject/ooni-explorer>`_ + +* ooni-wui_: web user interface assets and the implementation of the + ooni-probe web interface. Components in here are meant to be re-used across + the various software components (ooni-probe, ooni-explorer, net-probe, etc.), + though work on this from is not yet complete. + code repository: `<https://github.com/TheTorProject/ooni-wui>`_ + +* lepidopter_: a raspberry pi image for running ooniprobe. + code repository: `<https://github.com/TheTorProject/lepidopter>`_ + +* ooni-web_: the canonical ooni.torproject.org website. + code repository: `<https://github.com/TheTorProject/ooni-web>`_ + + + +.. _ooni-probe: +--------------- + +ooni-probe the client side component of OONI that is responsible for performing measurements on the to be tested network. -The main design goals for ooniprobe are: +Originally thought of as a tool to be used by users to investigate network +anomalies on their own and quickly implement new tests to check for new +censorship conditions, the focus is now shifting more towards something +meant to be used in an unattended manner. + +As such it's evolving into being a system daemon that is always running on +a users machine and automatically performs the network measurements the user +has instructed it to perform. + +Design goals +............. + +The current design goals are: + +**Unattended measurement collection** + +It should be possible for a user of the system to install it and forget about +it. This means that it shouldn't be necessary to constantly interact with the tool +itself. -Test specification decoupling -............................. +Previously some of the design considerations for ooni-probe used to be: -By this I mean that the definition of the test should be as loosely coupled to -the code that is used for running the test. +**Test specification decoupling** + +This design goal is still largely valid, though as ooni-probe grows as mainly +an enduser tool it's importance will be decreasing. + +Moreover the long-term plan for this is given the fact that tests are going to +be run based on measurement-kit_ is to have the testing framework logic be +implemented in the measurement-kit_ scripting language. + +The outline of this design goal nonetheless is that the definition of the test +should be as loosely coupled to the code that is used for running the test. This is achieved via what are called **Test Templates**. Test Templates a high level interface to the test developer specific to the protocol they are writing @@ -46,8 +133,7 @@ received, but a developer may with to include inside of their report the checksum of the of the content as is show in the example in `Writing Tests <writing_tests.html>`_. -Support for high concurrency -............................ +**Support for high concurrency** By this I mean that we want to be able to scan through big lists as fast as possible. @@ -68,68 +154,72 @@ For this purpose we have chosen to use the `Twisted networking framework If you have an argument for which you believe Twisted is not a good idea, I would love to know :). -Notes: -.. XXX +Running lot's of tests concurrently can reduce their accuracy. The ideal +strategy for dealing with this would involve adjusting the concurrency +based on failure rate. +Currently this is not implemented inside of ooniprobe and instead we use +a configurable concurrency value that is set to default as 3. -Running lot's of tests concurrently can reduce their accuracy. The strategy -for dealing with this involves doing proper error handling and adjusting the -concurrency window over time if the amount of error rates increases. +Implementation details +...................... -Currently the level of concurrency for tests is implemented inside of -:class:`ooni.inputunit`_, but we do not expose to the user a way of setting -this. Such feature will be something that will be controllable via the -ooniprobe API. +Below is a high level diagram of how the various modules of ooniprobe +are interrelated to each other. -Why Tor Hidden Services? -........................ +.. graphviz:: -We chose to use Tor Hidden Services as the means of exposing a backend -reporting system for the following reasons: + digraph ooniprobe_impl { -Easy addressing -_______________ + "agent" -> "director"; + "scheduler" -> "director"; -Using Tor Hidden Service allows us to have a globally unique identifier to be -passed to the ooni-probe clients. This identifier does not need to change even -if we decide to migrate the collector backend to a different machine (all we -have to do is copy the private key to the new box). + "director" -> "deck"; -It also allows people to run a collector backend if they do not have a public -IP address (if they are behing NAT for example). + "deck" -> "nettest"; + "deck" -> "backend_client"; + "deck" -> "nettests"; + } -Security -________ +ooni-probe is written in python using the `Twisted networking framework +<http://twistedmatrix.org>`_. -Tor Hidden Services give us for free and with little thought end to end -encryption and authentication. Once the address for the collector has been -transmitted to the probe you do not need to do any extra authenticatication, because -the address is self authenticating. +The two main concepts in ooniprobe are a decks and nettests. A nettest is a +particular network test that is designed to identify one class of anomalies. -Possible drawbacks -__________________ +A deck is a collection of one or more nettests and some associated inputs (such +as a list of URLs). -Supporting Tor Hidden Services as the only system for reporting means a -ooni-probe user is required to have Tor working to be able to submit reports to -a collector. In some cases this is not possible, because the user is in a -country where Tor is censored and they do not have any Tor bridges available. +The director is responsible for starting the measurement and reporting task +managers, starting tor, looking up the IP address of the probe and in general +controlling the lifecycle of the application. -Latency is also a big issue in Tor Hidden Services and this can make the -reporting process very long especially if the users network is not very good. +The schedulers are periodic tasks that need to be executed (think cron). Their +state is kept track of on disk (in particular the last time a successful +execution was performed). -For these reasons we plan to support in the future also non Tor HS based -reporting to oonib. -Currently this can easily be achieved by simply using tor2web.org. +The agent is responsible for starting director, the schedulers and exposing the +web user interface. -Standardization -............... +.. _measurement-kit: +-------------------- -.. TODO +Measurement-kit is a C++ library that implements network measurement primitives +and some of the ooniprobe tests. + +It has been developed with the goal of being able to target mobile platforms +(Android and iOS), but is growing with the intent of eventually replacing the +measurement engine of ooniprobe entirely with native code. + +There is work in progress to support calling it from python (see: +`<https://github.com/measurement-kit/measurement-kit/pull/697>`_) and there +are plans to implement a scripting interface around it to aid the development of +tests (see: `<https://github.com/measurement-kit/measurement-kit/issues/702>`_). -oonib ------ +.. _ooni-backend: +----------------- This is the backend component of OONI. It is responsible for exposing `test -helpers`_ and the `report collector`_. +helpers`_ , the `measurement collector`_ and the `bouncer service`_ Test Helpers ............ @@ -139,120 +229,79 @@ ooniprobes when running tests. If you would like to see a test helper implemented inside of oonib, thats great! -All you have to do is `open a ticket on trac -<https://trac.torproject.org/projects/tor/newticket?component=Ooni&keywords=…>`_. +All you have to do is `open a ticket on github +<https://github.com/TheTorProject/ooni-backend/issues/new?title=[new%20test-…>`_. To get an idea of the current implementation status of test helpers see the `oonib/testhelpers/ -<https://gitweb.torproject.org/ooni-probe.git/tree/HEAD:/oonib/testhelpers>`_ +<https://github.com/TheTorProject/ooni-backend/tree/master/oonib/testhelpers>`_ directory of the ooniprobe git repository. .. TODO write up the list of currently implemented test helpers and how to use them. -Report collector +Measurement collector ................ -.. autoclass:: oonib.report.file_collector.NewReportHandlerFile - :noindex: +This is the service that is used for submitting measurement results to. +The specification for the API of the measurement collector can be found here: +`<https://github.com/TheTorProject/ooni-spec/blob/master/oonib.md#20-collector>`_ -An ooniprobe run ----------------- - -Here we describe how an ooniprobe run should look like: - - 1. If configured to do so ooniprobe will start a connection to the Tor - network for the purpose of having a known good test channel and for - having a way of reporting to the backend collector - - 2. It will obtain it's IP Address from Tor via the getinfo addr Tor Ctrl port - request. - - 3. If a collect is specified it will connect to the reporting system and get - a report id that allows them to submit reports to the collector. - - 4. If inputs are specified it will slice them up into chunks of request to be - performed in parallel. - - 5. Once every chunk of inputs (called an InputUnit) will have completed the - report file and/or the collector will be updated. - - -OONIprobe Control Interface ---------------------------- -.. XXX update this section once interface is implemented. -The ooniprobe client provides a rich and simple JSON-based interface for -control over HTTP. While the implementation of this interface is currently -a work in progress, the specification may be found `here <control_interface.rst>`_. - - -Implementation status ---------------------- - -ooniprobe -......... - -**Reporting** - - * To flat YAML file: *alpha* - - * To remote httpo backend: *alpha* - -**Test templates** - - * HTTP test template: *alpha* - - * Scapy test template: *alpha* - - * DNS test template: *alpha* - - * TCP test template: *prototype* - -**Tests** - -To see the list of implemented tests see: -https://ooni.torproject.org/docs/#core-ooniprobe-tests - -**ooniprobe API** - - * Specification: *draft* - - * HTTP API: *not implemented* - -**ooniprobe HTML5/JS user interface** +Bouncer service +................ - Not implemented. +This is the service that is responsible for informing clients of where they +should be submitting their results to and what are the addresses of the +test-helpers they require to perform their measurements. -**ooniprobe build system** +The specification for the API of the bouncer can be found here: +`<https://github.com/TheTorProject/ooni-spec/blob/master/oonib.md#40-bouncer>`_ - Not implemented. +.. _ooni-pipeline: +------------------ -**ooniprobe command line interface** +When measurements are submitted to a measurement collector they are then +processed by the data pipeline. - Implemented in alpha quality, though needs to be ported to use the HTTP based - API. +The measurements are first normalised (to take into account the different data +formats that ooniprobe has supported over time), then sanitised (to redact from them +sensitive information such a private bridge IP address) and then put inside of a +database to be served via the ooni-explorer_. -oonib -..... +It is currently written in python using the `luigi workflow manager +<https://luigi.readthedocs.org>`_, but that may change in the near future. +For future plans see: `<https://github.com/TheTorProject/ooni-pipeline/issues/32>`_ -**Collector** +.. _ooni-explorer: +------------------ - * collection of YAML reports to flat file: *alpha* +This is the web interface that is used by end users to inspect measurements +collected by ooniprobe. - * collection of pcap reports: *not implemented* +It is written as a node.js web app (based on the strongloop framework), with +angular.js and d3.js. - * association of reports with test helpers: *not implemented* +.. _ooni-wui: +------------- -**Test helpers** +Web user interface assets and the implementation of the ooni-probe web +interface. Components in here are meant to be re-used across the various +software components (ooni-probe, ooni-explorer, net-probe, etc.), though work +on this from is not yet complete. - * HTTP Return JSON Helper: *alpha* +.. _lepidopter: +--------------- - * DNS Test helper: *prototype* +A raspberry pi image for running ooniprobe. - * Test Helper - collector mapping: *Not implemented* +Amongst other things it takes care of automatically updating ooniprobe to the +latest version and packaging all the dependencies required to run ooniprobe. - * TCP Test helper: *prototype* +.. _ooni-web: +------------- - * Daphn3 Test helper: *prototype* +The canonical ooni.torproject.org website. +It is implemented using `hugo <https://gohugo.io>`_ a golang based static +website generator. diff --git a/docs/source/conf.py b/docs/source/conf.py index e3c6544..2eba8e4 100644 --- a/docs/source/conf.py +++ b/docs/source/conf.py @@ -30,7 +30,7 @@ from ooni import __version__ as ooniprobe_version # Add any Sphinx extension module names here, as strings. They can be extensions # coming with Sphinx (named 'sphinx.ext.*') or your custom ones. extensions = ['sphinx.ext.todo', 'sphinx.ext.coverage', 'sphinx.ext.pngmath', -'sphinx.ext.viewcode', 'sphinx.ext.autodoc'] +'sphinx.ext.viewcode', 'sphinx.ext.autodoc', 'sphinx.ext.graphviz'] # Add any paths that contain templates here, relative to this directory. templates_path = ['_templates']

1 0

[ooni-probe/master] Implement sorting of measurements in measurements tab from newest to … (#665)
by art＠torproject.org 13 Jan '17

13 Jan '17

commit a9d86e09f4a02317e1ba2b6b7380620bd381d7bc Author: Arturo Filastò <arturo(a)filasto.net> Date: Tue Nov 29 12:10:26 2016 +0000 Implement sorting of measurements in measurements tab from newest to … (#665) * Implement sorting of measurements in measurements tab from newest to oldest * Fail aloud when argument is unknown --- ooni/measurements.py | 15 +++++++++++++-- ooni/ui/web/server.py | 2 +- 2 files changed, 14 insertions(+), 3 deletions(-) diff --git a/ooni/measurements.py b/ooni/measurements.py index 46a447e..d79fffb 100644 --- a/ooni/measurements.py +++ b/ooni/measurements.py @@ -1,4 +1,5 @@ import json +import operator from twisted.internet import defer from twisted.internet.threads import deferToThread @@ -129,7 +130,7 @@ def get_summary(measurement_id): return defer.succeed(json.load(f)) -def list_measurements(compute_size=False): +def list_measurements(compute_size=False, order=None): measurements = [] measurement_path = FilePath(config.measurements_directory) if not measurement_path.exists(): @@ -139,4 +140,14 @@ def list_measurements(compute_size=False): measurements.append(get_measurement(measurement_id, compute_size)) except: log.err("Failed to get metadata for measurement {0}".format(measurement_id)) - return measurements + + if order is None: + return measurements + + if order.lower() in ['asc', 'desc']: + reverse = {'asc': True, 'desc': False}[order.lower()] + measurements.sort(key=operator.itemgetter('test_start_time'), + reverse=reverse) + return measurements + else: + raise ValueError("order must be either 'asc' 'desc' or None") diff --git a/ooni/ui/web/server.py b/ooni/ui/web/server.py index fa35af0..bb4e45f 100644 --- a/ooni/ui/web/server.py +++ b/ooni/ui/web/server.py @@ -521,7 +521,7 @@ class WebUIAPI(object): @xsrf_protect(check=False) @requires_true(attrs=['_is_initialized']) def api_measurement_list(self, request): - measurements = list_measurements() + measurements = list_measurements(order='desc') return self.render_json({"measurements": measurements}, request) @app.route('/api/measurement/<string:measurement_id>', methods=["GET"])

1 0

[ooni-probe/master] Merge pull request #683 from TheTorProject/requirements-docs
by art＠torproject.org 13 Jan '17

13 Jan '17

commit c6a4e2f7b519b82476092f1a2a4a7e0b1606cb99 Merge: e603557 289b9af Author: anadahz <anadahz(a)users.noreply.github.com> Date: Tue Nov 22 01:01:54 2016 +0000 Merge pull request #683 from TheTorProject/requirements-docs Add requirements needed to build docs requirements-docs.txt | 3 +++ 1 file changed, 3 insertions(+)

1 0

[ooni-probe/master] Improvements to log handling
by art＠torproject.org 13 Jan '17

13 Jan '17

commit 1a2ba3577321ebfbfbb2e75e1d9719d0e3728af6 Author: Arturo Filastò <arturo(a)filasto.net> Date: Sat Nov 12 13:06:16 2016 +0100 Improvements to log handling * Add rotate setting to configure daily or length based rotation * Make MsecLogObserver inherit the log_level properties from LogLevelObserver * Add option to disable log rotation This can be used if you would rather handle log rotation via logrotate.d and similar system tools. * Support max_rotate_files option * Add some basic unittests for log related functions * Improve comments to rotation options * Disable duckduckgo geoip lookup service --- data/ooniprobe.conf.sample | 14 ++++++++++++-- ooni/geoip.py | 7 +++++-- ooni/settings.py | 10 ++++++++++ ooni/tests/test_utils.py | 47 ++++++++++++++++++++++++++++++++++++++++++++++ ooni/utils/log.py | 19 ++++++++++++++----- 5 files changed, 88 insertions(+), 9 deletions(-) diff --git a/data/ooniprobe.conf.sample b/data/ooniprobe.conf.sample index ee9a2c9..5880aee 100644 --- a/data/ooniprobe.conf.sample +++ b/data/ooniprobe.conf.sample @@ -4,8 +4,18 @@ basic: # Where OONIProbe should be writing its log file - logfile: /var/log/ooniprobe.log - loglevel: WARNING + # logfile: ooniprobe.log + # loglevel: WARNING + # Will rotate logs daily when "daily" or based on rotate_length + # when "length" is set. null to disable rotation. + # rotate: daily + # rotate_length: 1M + # Sets an upper bound on the number of rotated files, only works when + # "length" is the rotation mode + # max_rotated_files: null + # The maximum amount of data to store on disk. Once the quota is reached, + # we will start deleting older reports. + # measurement_quota: 1G privacy: # Should we include the IP address of the probe in the report? includeip: false diff --git a/ooni/geoip.py b/ooni/geoip.py index 9148e34..c765015 100644 --- a/ooni/geoip.py +++ b/ooni/geoip.py @@ -164,8 +164,11 @@ class ProbeIP(object): def __init__(self): self.geoIPServices = { - 'ubuntu': UbuntuGeoIP, - 'duckduckgo': DuckDuckGoGeoIP + 'ubuntu': UbuntuGeoIP + # We are disabling this because it sometimes creates parsing + # errors. + # See: https://github.com/TheTorProject/ooni-probe/issues/670 + # 'duckduckgo': DuckDuckGoGeoIP } self.geodata = { 'asn': 'AS0', diff --git a/ooni/settings.py b/ooni/settings.py index ed2d1e1..eb4f052 100644 --- a/ooni/settings.py +++ b/ooni/settings.py @@ -22,6 +22,13 @@ basic: # Where OONIProbe should be writing its log file # logfile: {logfile} # loglevel: WARNING + # Will rotate logs daily when "daily" or based on rotate_length + # when "length" is set. null to disable rotation. + # rotate: daily + # rotate_length: 1M + # Sets an upper bound on the number of rotated files, only works when + # "length" is the rotation mode + # max_rotated_files: null # The maximum amount of data to store on disk. Once the quota is reached, # we will start deleting older reports. # measurement_quota: 1G @@ -105,6 +112,9 @@ defaults = { "basic": { "loglevel": "WARNING", "logfile": "ooniprobe.log", + "rotate": "daily", + "rotate_length": "1M", + "max_rotated_files": None, "measurement_quota": "1G" }, "privacy": { diff --git a/ooni/tests/test_utils.py b/ooni/tests/test_utils.py index 4ef2926..f0f4ce0 100644 --- a/ooni/tests/test_utils.py +++ b/ooni/tests/test_utils.py @@ -1,5 +1,6 @@ import os import tempfile +from mock import patch from twisted.trial import unittest @@ -77,3 +78,49 @@ class TestUtils(unittest.TestCase): with open(os.path.join(tmp_dir, "subdir", "something.txt"), "w") as out_file: out_file.write("A"*1000) self.assertEqual(directory_usage(tmp_dir), 1000*2) + +class LoggingTests(unittest.TestCase): + def setUp(self): + self.dir = self.mktemp() + os.makedirs(self.dir) + self.name = "test.log" + self.path = os.path.join(self.dir, self.name) + + @patch('ooni.settings.config') + def test_length_rotate(self, mock_config): + mock_config.basic.loglevel = 'DEBUG' + mock_config.advanced.debug = False + mock_config.basic.rotate = 'length' + mock_config.basic.rotate_length = '1M' + + ooni_logger = log.OONILogger() + ooni_logger.start(logfile=self.path) + + ooni_logger.debug("1"*(1024**2)) + + # This will trigger the rotation + ooni_logger.debug("2") + + ooni_logger.stop() + self.assertTrue(os.path.exists("{0}.1".format(self.path))) + self.assertFalse(os.path.exists("{0}.2".format(self.path))) + + + @patch('ooni.settings.config') + def test_log_levels(self, mock_config): + mock_config.basic.loglevel = 'WARN' + mock_config.advanced.debug = False + mock_config.basic.rotate = None + + ooni_logger = log.OONILogger() + ooni_logger.start(logfile=self.path) + ooni_logger.msg("Should not print") + ooni_logger.warn("Will print") + ooni_logger.err("Will print") + ooni_logger.debug("Should not print") + ooni_logger.stop() + with open(self.path) as in_file: + self.assertEqual( + len(in_file.readlines()), + 2 + ) diff --git a/ooni/utils/log.py b/ooni/utils/log.py index b4136be..353b8b2 100644 --- a/ooni/utils/log.py +++ b/ooni/utils/log.py @@ -6,9 +6,10 @@ import logging from datetime import datetime from twisted.python import log as tw_log -from twisted.python.logfile import DailyLogFile +from twisted.python.logfile import DailyLogFile, LogFile from ooni.utils import mkdir_p +from ooni.utils.files import human_size_to_bytes from ooni import otime # Get rid of the annoying "No route found for @@ -101,7 +102,7 @@ class StdoutStderrObserver(LogLevelObserver): self.write(text + "\n") self.flush() -class MsecLogObserver(tw_log.FileLogObserver): +class MsecLogObserver(LogLevelObserver): def formatTime(self, when): """ Code from Twisted==16.4.1 modified to log microseconds. Although this @@ -172,10 +173,18 @@ class OONILogger(object): if config.advanced.debug: stdout_log_level = levels['DEBUG'] - daily_logfile = DailyLogFile(log_filename, log_folder) + if config.basic.rotate is 'daily': + logfile = DailyLogFile(log_filename, log_folder) + elif config.basic.rotate is 'length': + logfile = LogFile(log_filename, log_folder, + rotateLength=int(human_size_to_bytes( + config.basic.rotate_length + )), + maxRotatedFiles=config.basic.max_rotated_files) + else: + logfile = open(os.path.join(log_folder, log_filename), 'a') - self.fileObserver = MsecLogObserver(LogLevelObserver(daily_logfile, - log_level=file_log_level)) + self.fileObserver = MsecLogObserver(logfile, log_level=file_log_level) self.stdoutObserver = StdoutStderrObserver(sys.stdout, log_level=stdout_log_level)

1 0

[ooni-probe/master] Add Makefile for release workflow
by art＠torproject.org 13 Jan '17

13 Jan '17

commit 0e44933a48c2eac4fb5861be068e384bf5bb6965 Author: Arturo Filastò <arturo(a)filasto.net> Date: Wed Nov 16 14:48:12 2016 +0100 Add Makefile for release workflow --- Makefile | 15 +++++++++++++++ 1 file changed, 15 insertions(+) diff --git a/Makefile b/Makefile new file mode 100644 index 0000000..e22afb4 --- /dev/null +++ b/Makefile @@ -0,0 +1,15 @@ +VERSION = 2.0.2 + +man: + cd docs && make man + cp docs/build/man/* data/ + @echo "Now commit the updated man pages" + +sdist: + python setup.py sdist + +sign: + gpg --no-version --detach-sign --armor --local-user 0x702287F4 dist/ooniprobe-${VERSION}.tar.gz + +upload: + twine upload -r pypi dist/ooniprobe-${VERSION}.tar.gz dist/ooniprobe-${VERSION}.tar.gz.asc

1 0