- tor-commits - lists.torproject.org

[ooni-probe/master] No need to downgrade Twisted
by art＠torproject.org 29 Apr '16

29 Apr '16

commit 9963b3beda4ad29b050c60bbae2efdba6b3d79fc Author: anadahz <kojgelo(a)inbox.com> Date: Sun Dec 20 15:30:05 2015 +0100 No need to downgrade Twisted All Twister version are supported by ooniprobe! Fixed (https://trac.torproject.org/projects/tor/ticket/15944) --- scripts/install.sh | 4 ---- 1 file changed, 4 deletions(-) diff --git a/scripts/install.sh b/scripts/install.sh index beb9f60..01318ce 100755 --- a/scripts/install.sh +++ b/scripts/install.sh @@ -324,10 +324,6 @@ case "$lsb_dist" in set -x $sh_c 'apt-get install -y -q curl git-core python python-dev python-setuptools build-essential libdumbnet1 python-dumbnet python-libpcap tor tor-geoipdb libgeoip-dev libpcap0.8-dev libssl-dev libffi-dev libdumbnet-dev' $sh_c "PYTHONPATH=$PYTHONPATH pip install --install-option=\"--prefix=$PYTHON_PREFIX\" ooniprobe" - # Unistall Twisted and install a version >=12.2.0,<=14.0.0 - # https://trac.torproject.org/projects/tor/ticket/15944 - $sh_c "PYTHONPATH=$PYTHONPATH pip uninstall -y Twisted" - $sh_c "PYTHONPATH=$PYTHONPATH pip install --install-option=\"--prefix=$PYTHON_PREFIX\" 'Twisted>=12.2.0,<=14.0.0'" ) fi

1 0

[ooni-probe/master] README: proper path to example inputs
by art＠torproject.org 29 Apr '16

29 Apr '16

commit 474d20b5e672a3310d1ff43bd8878454175cbf7b Author: ecneladis <ecneladis(a)gmail.com> Date: Sat Jan 9 17:02:37 2016 +0100 README: proper path to example inputs --- README.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.rst b/README.rst index c824c44..a1b6386 100644 --- a/README.rst +++ b/README.rst @@ -174,7 +174,7 @@ ooniprobe will be installed in ``/ooni``. 5. You can run tests with:: - ooniprobe blocking/http_requests -f /ooni/example_inputs/alexa-top-1k.txt + ooniprobe blocking/http_requests -f /ooni/var/example_inputs/alexa-top-1k.txt Using ooniprobe ===============

1 0

[ooni-probe/master] Add description of what test helpers are in introduction.
by art＠torproject.org 29 Apr '16

29 Apr '16

commit 568b5cdd71755b3ba12861c72ec918c425b463c3 Author: Arturo Filastò <arturo(a)filasto.net> Date: Fri Jan 22 11:03:56 2016 +0100 Add description of what test helpers are in introduction. * Fix links --- docs/source/writing_test_helpers.rst | 24 ++++++++++++------------ 1 file changed, 12 insertions(+), 12 deletions(-) diff --git a/docs/source/writing_test_helpers.rst b/docs/source/writing_test_helpers.rst index 2206115..d391aa4 100644 --- a/docs/source/writing_test_helpers.rst +++ b/docs/source/writing_test_helpers.rst @@ -1,23 +1,23 @@ Writing Test Helpers ======== -OONI test helpers are... +OONI test helpers are used by OONI nettests to perform their meausrements. They can be used either to establish a ground truth or to exchange information with the probe to determine if some form of network manipulation is happening in the network path between the probe and the backend. Writing a Censorship Directionality Test Helper -------------------------- Our goal is to write an OONI test helper that helps an OONI-probe client determine the directionality of keyword censorship it has detected. To do this our helper will receive "encoded" data from an OONI-probe client, decode that data into a text string, and send the OONI-probe client a confirmation packet and a second packet containing the decoded string. -The OONI-backend code-base has many concise examples of test-helpers that you can build off to create your own. For this tutorial I used the `TCP echo test-helper <https://github.com/TheTorProject/ooni-backend/blob/479a1bb154037b834292ccc4…>` as my guide. +The OONI-backend code-base has many concise examples of test-helpers that you can build off to create your own. For this tutorial I used the `TCP echo test-helper <https://github.com/TheTorProject/ooni-backend/blob/479a1bb154037b834292ccc4…>`_ as my guide. -Following this tutorial requires basic knowledge of event-driven programming (specifically 'Twisted'). You will be more than ready to build and implement a test-helper after reading though one or two `tutorials online. <http://krondo.com/?page_id=1327>` +Following this tutorial requires basic knowledge of event-driven programming (specifically 'Twisted'). You will be more than ready to build and implement a test-helper after reading though one or two `tutorials online. <http://krondo.com/?page_id=1327>`_ Creating the test helper -------------- -OONI-backend keeps all the test-helpers in the `oonib/testhelpers directory <https://github.com/TheTorProject/ooni-backend/tree/master/oonib/testhelpers>` Each individual test helper is a twisted service. Most of the current test-helpers consists of a twisted Factory and a twisted Protocol defined in the test helpers directory and a `stock Twisted Server <https://twistedmatrix.com/documents/current/api/twisted.application.interne…>` that is defined in the backend code. We will follow this model in the tutorial. +OONI-backend keeps all the test-helpers in the `oonib/testhelpers directory <https://github.com/TheTorProject/ooni-backend/tree/master/oonib/testhelpers>`_ Each individual test helper is a twisted service. Most of the current test-helpers consists of a twisted Factory and a twisted Protocol defined in the test helpers directory and a `stock Twisted Server <https://twistedmatrix.com/documents/current/api/twisted.application.interne…>`_ that is defined in the backend code. We will follow this model in the tutorial. -Because of how simple this example test-helper is the job of our test-helper factory is merely to deploy a single instance of our protocol each time it's buildProtocol method is called. Because we have our factory inheret from the base `Factory object <https://twistedmatrix.com/trac/browser/tags/releases/twisted-15.5.0/twisted…>` we merely have to define its ``protocol`` property to point to our protocol.:: +Because of how simple this example test-helper is the job of our test-helper factory is merely to deploy a single instance of our protocol each time it's buildProtocol method is called. Because we have our factory inheret from the base `Factory object <https://twistedmatrix.com/trac/browser/tags/releases/twisted-15.5.0/twisted…>`_ we merely have to define its ``protocol`` property to point to our protocol.:: class TCPDirectionalityTestHelper(Factory): """ @@ -26,7 +26,7 @@ Because of how simple this example test-helper is the job of our test-helper fac protocol = TCPDirectionalityTestProtocol -The protocol for this helper needs to do two things. First, upon receiving encoded data it needs to send the OONI-probe client back confirmation that the data has been received. Second, it needs to send the decoded data back to the OONI-probe client. Because we are extending the `Protocol object <https://twistedmatrix.com/trac/browser/tags/releases/twisted-15.5.0/twisted…>` we can rewrite its ``dataReceived`` method which is called and passed data whenever it is received.:: +The protocol for this helper needs to do two things. First, upon receiving encoded data it needs to send the OONI-probe client back confirmation that the data has been received. Second, it needs to send the decoded data back to the OONI-probe client. Because we are extending the `Protocol object <https://twistedmatrix.com/trac/browser/tags/releases/twisted-15.5.0/twisted…>`_ we can rewrite its ``dataReceived`` method which is called and passed data whenever it is received.:: class TCPDirectionalityTestProtocol(Protocol): @@ -68,7 +68,7 @@ With this added we have completed the base of simple test-helper and now just ha Adding the helper to the config file ------ -OONI-backend uses a config file located at `/etc/oonibackend.conf <https://github.com/TheTorProject/ooni-backend/blob/master/oonib.conf.example>`. This file contains a `section where each test-helper can be configured. <https://github.com/TheTorProject/ooni-backend/blob/479a1bb154037b834292ccc4…>`. +OONI-backend uses a config file located at `/etc/oonibackend.conf <https://github.com/TheTorProject/ooni-backend/blob/master/oonib.conf.example>`_. This file contains a `section where each test-helper can be configured. <https://github.com/TheTorProject/ooni-backend/blob/479a1bb154037b834292ccc4…>`_. The test-helper will need to be given a unique identifyer so that it can be called from the config file. In this example we use ``tcp-directionality`` as our identifyer. @@ -81,17 +81,17 @@ For a helper to be used in the ooni-backend it needs to be given an identifyer s Adding the helper to the backend ------ -For a helper to be integrated into the ooni-backend it needs to be added to the initialization scripts contained within `oonibackend.py <https://github.com/TheTorProject/ooni-backend/blob/master/oonib/oonibackend…>`. +For a helper to be integrated into the ooni-backend it needs to be added to the initialization scripts contained within `oonibackend.py <https://github.com/TheTorProject/ooni-backend/blob/master/oonib/oonibackend…>`_. -The OONI test-helper system is a collection of `Twisted services <https://twistedmatrix.com/documents/current/core/howto/application.html>`. For our test-helper we will need to define a service that will run our test-helper factory.:: +The OONI test-helper system is a collection of `Twisted services <https://twistedmatrix.com/documents/current/core/howto/application.html>`_. For our test-helper we will need to define a service that will run our test-helper factory.:: # Create the service that will run our test-helpers factory. tcp_directionality_helper = internet.TCPServer(int(port), tcp_helpers.TCPDirectionalityTestHelper()) -**NOTE:** In this example I have placed the original service in the existing tcp_helpers file. If you created your own file for your test-helper you will have to make sure that you import that file at the top of `oonibackend.py <https://github.com/TheTorProject/ooni-backend/blob/master/oonib/oonibackend…>`. +**NOTE:** In this example I have placed the original service in the existing tcp_helpers file. If you created your own file for your test-helper you will have to make sure that you import that file at the top of `oonibackend.py <https://github.com/TheTorProject/ooni-backend/blob/master/oonib/oonibackend…>`_. -OONI uses a `Multi Service <https://twistedmatrix.com/documents/current/api/twisted.application.service…>` which allows them to combine all the OONI test-helpers and the report-collector into a singlular service for easier management. The next step for creating our test-helper is to add it to the OONI-backend `multiService <https://github.com/TheTorProject/ooni-backend/blob/479a1bb154037b834292ccc4…>`:: +OONI uses a `Multi Service <https://twistedmatrix.com/documents/current/api/twisted.application.service…>`_ which allows them to combine all the OONI test-helpers and the report-collector into a singlular service for easier management. The next step for creating our test-helper is to add it to the OONI-backend `multiService <https://github.com/TheTorProject/ooni-backend/blob/479a1bb154037b834292ccc4…>`_:: # Add the helper as a child of the backends multi-service test-helper multiService.addService(tcp_directionality_helper) @@ -103,7 +103,7 @@ Finally, we need to start our service.:: In order for our test-helper to be managed using the backend config file we will need to modify this code to check the config file for a test-helper that uses the identifyer we selected earlier. For the directionality helper we check to see if our test-helper had its port specified in the config file to determine if it should be run. I also added a default encoding in case -This snippet contains the final code that would be inserted into `oonibackend.py <https://github.com/TheTorProject/ooni-backend/blob/master/oonib/oonibackend…>`.:: +This snippet contains the final code that would be inserted into `oonibackend.py <https://github.com/TheTorProject/ooni-backend/blob/master/oonib/oonibackend…>`_.:: # Check to see if our test-helper was defined in the config if config.helpers['tcp-directionality'].port:

1 0

[ooni-probe/master] README: Vagrantfile requires ooni-backend in parent directory
by art＠torproject.org 29 Apr '16

29 Apr '16

commit 2c18041d9b1d13a788df27292fca83c0460bac04 Author: ecneladis <ecneladis(a)gmail.com> Date: Sat Jan 9 16:54:49 2016 +0100 README: Vagrantfile requires ooni-backend in parent directory --- README.rst | 1 + 1 file changed, 1 insertion(+) diff --git a/README.rst b/README.rst index bf52114..c824c44 100644 --- a/README.rst +++ b/README.rst @@ -162,6 +162,7 @@ On debian/ubuntu:: 3. Open a Terminal and run:: git clone https://git.torproject.org/ooni-probe.git + git clone https://github.com/TheTorProject/ooni-backend.git cd ooni-probe/ vagrant up

1 0

[ooni-probe/master] Fix typo in code snippet example
by art＠torproject.org 29 Apr '16

29 Apr '16

commit e0428e869f9dab9eb7dec9dab9425a0341eaedc6 Author: Arturo Filastò <arturo(a)filasto.net> Date: Fri Jan 22 11:08:22 2016 +0100 Fix typo in code snippet example --- docs/source/writing_test_helpers.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/source/writing_test_helpers.rst b/docs/source/writing_test_helpers.rst index d391aa4..a8601cc 100644 --- a/docs/source/writing_test_helpers.rst +++ b/docs/source/writing_test_helpers.rst @@ -72,9 +72,9 @@ OONI-backend uses a config file located at `/etc/oonibackend.conf <https://githu The test-helper will need to be given a unique identifyer so that it can be called from the config file. In this example we use ``tcp-directionality`` as our identifyer. -For a helper to be used in the ooni-backend it needs to be given an identifyer so that it can be called from the config file.:: +For a helper to be used in the ooni-backend it needs to be given an identifyer so that it can be called from the config file:: - tcp-echo: + tcp-directionality: encoding: rot13 port: 57009

1 0

[ooni-probe/master] Add link to test_helper_tutorial in index
by art＠torproject.org 29 Apr '16

29 Apr '16

commit b97304a0cf459437ad7849ec116f29c570393315 Author: Arturo Filastò <arturo(a)filasto.net> Date: Fri Jan 22 11:04:19 2016 +0100 Add link to test_helper_tutorial in index --- docs/source/index.rst | 1 + 1 file changed, 1 insertion(+) diff --git a/docs/source/index.rst b/docs/source/index.rst index 10c7c8f..1eabdee 100644 --- a/docs/source/index.rst +++ b/docs/source/index.rst @@ -508,6 +508,7 @@ More developer documentation tutorial writing_tests + writing_test_helpers reports nettests/modules api/*

1 0

[ooni-probe/master] Fix to make the psiphon test run
by art＠torproject.org 29 Apr '16

29 Apr '16

commit 5f7e947953c8af4ca56d0462e69e80ae991293b4 Author: Arturo Filastò <art(a)fuffa.org> Date: Wed Dec 9 17:34:51 2015 +0100 Fix to make the psiphon test run * Remove unneeded imports --- ooni/nettests/third_party/psiphon.py | 12 +++--------- 1 file changed, 3 insertions(+), 9 deletions(-) diff --git a/ooni/nettests/third_party/psiphon.py b/ooni/nettests/third_party/psiphon.py index 543e856..fb912a9 100644 --- a/ooni/nettests/third_party/psiphon.py +++ b/ooni/nettests/third_party/psiphon.py @@ -1,19 +1,12 @@ import tempfile -import stat import os import sys from twisted.internet import defer, reactor -from twisted.internet.endpoints import TCP4ClientEndpoint -from twisted.web.client import readBody from twisted.python import usage -from txsocksx.http import SOCKS5Agent - -from ooni.errors import handleAllFailures, TaskTimedOut from ooni.utils import log from ooni.templates import process, httpt -from ooni.templates.process import ProcessTest class UsageOptions(usage.Options): @@ -94,9 +87,9 @@ connect(False) def test_psiphon(self): log.debug('PsiphonTest.test_psiphon') + self.report['success'] = None if not os.path.exists(self.psiphonpath): log.err('psiphon path does not exists, is it installed?') - self.report['success'] = False self.report['psiphon_installed'] = False log.debug("Adding %s to report" % self.report) # XXX: the original code written by juga0 readed @@ -116,7 +109,8 @@ connect(False) # full with some block size and therefore the test would # terminate with error finished = self.run(self.command, - env=dict(PYTHONPATH=self.psiphonpath), + env=dict(PYTHONPATH=os.path.join(self.psiphonpath, + 'pyclient')), path=self.psiphonpath, usePTY=1)

1 0

[ooni-probe/master] changed report structure for ProcessTest
by art＠torproject.org 29 Apr '16

29 Apr '16

commit a309440621561c47d4cb7db5b04e73b342d96307 Author: juga0 <juga> Date: Wed Dec 23 16:51:35 2015 +0000 changed report structure for ProcessTest --- ooni/templates/process.py | 10 ++++------ 1 file changed, 4 insertions(+), 6 deletions(-) diff --git a/ooni/templates/process.py b/ooni/templates/process.py index 7ec4b2e..b7dafdd 100644 --- a/ooni/templates/process.py +++ b/ooni/templates/process.py @@ -103,12 +103,10 @@ class ProcessTest(NetTestCase): def processEnded(self, result, command): log.debug("Finished %s: %s" % (command, result)) - key = ' '.join(command) - self.report[key] = { - 'stdout': result['stdout'], - 'stderr': result['stderr'], - 'exit_reason': result['exit_reason'] - } + self.report['command_name'] = ' '.join(command) + self.report['command_stdout'] = result['stdout'] + self.report['command_stderr'] = result['stderr'] + self.report['command_exit_reason'] = result['exit_reason'] return result def run(self, command, finished=None, env={}, path=None, usePTY=0):

1 0

[ooni-probe/master] fix checking if commands exist, other minor changes
by art＠torproject.org 29 Apr '16

29 Apr '16

commit 0e975d3c47a1a6d9cdbe028b2c3ce08aca6431ce Author: juga0 <juga> Date: Wed Dec 23 19:50:07 2015 +0000 fix checking if commands exist, other minor changes * fix checking if commands exist * only install virtualenv if it is needed to create a new one * add/change debug ouput --- scripts/psiphon_install.sh | 50 ++++++++++++++++++++++++++++------------------ 1 file changed, 31 insertions(+), 19 deletions(-) diff --git a/scripts/psiphon_install.sh b/scripts/psiphon_install.sh index 117c823..12086c8 100755 --- a/scripts/psiphon_install.sh +++ b/scripts/psiphon_install.sh @@ -20,8 +20,10 @@ sh_c='sh -c' if [ "$user" != 'root' ]; then if command_exists sudo; then sh_c='sudo sh -c -E' + echo "[D] using sudo" elif command_exists su; then sh_c='su -c --preserve-environment' + echo "[D] using su" else echo >&2 'Error: this installer needs the ability to run commands as root.' echo >&2 'We are unable to find either "sudo" or "su" available to make this happen.' @@ -29,42 +31,49 @@ if [ "$user" != 'root' ]; then fi fi -echo "installing dependencies" +echo "[D] installing dependencies" $sh_c "apt-get -y install zlib1g-dev libssl-dev" -if [ ! "command_exists hg" ]; then +if ! command_exists hg; then + echo "[D] installing mercurial" $sh_c "apt-get -y install mercurial" fi +echo "[D] mercurial installed" cd $PSIPHON_PATH if [ ! -d "psiphon-circumvention-system" ]; then - echo "cloning psiphon repository" + echo "[D] cloning psiphon repository" hg clone $PSIPHON_REPO_URL fi -echo "psiphon repository cloned" +echo "[D] psiphon repository cloned" # optional, compile their ssh if [ ! -f "$PSIPHON_PYCLIENT_PATH/ssh" ]; then - echo "compiling psiphon ssh" + echo "[D] compiling psiphon ssh" cd psiphon-circumvention-system/Server/3rdParty/openssh-5.9p1/ ./configure make - mv ssh ../../../pyclient/ + mv ssh ../../../pyclient/pyclient/ make clean - echo "psiphon ssh compiled" + echo "[D] psiphon ssh compiled" fi # check if we are in a virtualenv, create it otherwise -echo "checking virtualenv" +echo "[D] checking virtualenv" if [ `python -c 'import sys; print hasattr(sys, "real_prefix")'` = "False" ]; then - # not in a virtualenv - # create a virtualenv - # FIXME: assuming debian version will have secure pip/virtualenv - if [ ! "command_exists virtualenv" ]; then - $sh_c "apt-get -y install python-virtualenv" - fi + echo "[D] not in a virtualenv" if [ ! -f $OONI_VIRTUALENV_PATH/bin/activate ]; then + echo "[D] virtualenv not found" + # create a virtualenv + # FIXME: assuming debian version will have secure pip/virtualenv + if ! command_exists virtualenv; then + echo "[D] installing virtualenv" + $sh_c "apt-get -y install python-virtualenv" + else + echo "[D] virtualenv command found" + fi + echo "[D] creating a virtualenv" # Set up the virtual environment mkdir -p $HOME/.virtualenvs virtualenv $OONI_VIRTUALENV_PATH @@ -72,12 +81,12 @@ if [ `python -c 'import sys; print hasattr(sys, "real_prefix")'` = "False" ]; th else . $OONI_VIRTUALENV_PATH/bin/activate fi + echo "[D] virtualenv activated" fi -echo "virtualenv activated" # create psi_client.dat -echo "creating servers data file" -echo "installing dependencies to create servers data file" +echo "[D] creating servers data file" +echo "[D] installing dependencies to create servers data file" pip install -v --timeout 60 wget cd /tmp cat <<EOF > psi_generate_dat.py @@ -107,9 +116,12 @@ EOF chmod +x psi_generate_dat.py ./psi_generate_dat.py -echo "servers data file created" +echo "[D] servers data file created" mv psi_client.dat $PSIPHON_PYCLIENT_PATH rm /tmp/psi_generate_dat.py -echo "[+] Installing all of the Python dependency requirements with pip in your virtualenv!"; +echo "[D] installing all of the Python dependency requirements with pip in the virtualenv"; pip install -v --timeout 60 jsonpickle pexpect + +echo "You can now run Psiphon: cd ~/psiphon-circumvention-system/pyclient/pyclient;python psi_client.py" +echo "NOTE that if OONI is not installed, you will not be able to run OONI Psiphon test"

1 0

[ooni-probe/master] leave success key to none in case psiphon fails
by art＠torproject.org 29 Apr '16

29 Apr '16

commit fbf9de395051b2f88d3f68a122f22ff25aa1cee0 Author: juga0 <juga> Date: Wed Dec 23 16:18:59 2015 +0000 leave success key to none in case psiphon fails --- ooni/nettests/third_party/psiphon.py | 1 - 1 file changed, 1 deletion(-) diff --git a/ooni/nettests/third_party/psiphon.py b/ooni/nettests/third_party/psiphon.py index 17c83c6..9314132 100644 --- a/ooni/nettests/third_party/psiphon.py +++ b/ooni/nettests/third_party/psiphon.py @@ -91,7 +91,6 @@ connect(False) self.report['psiphon_installed'] = None if not os.path.exists(self.psiphonpath): log.err('psiphon path does not exists, is it installed?') - self.report['success'] = False self.report['psiphon_installed'] = False log.debug("Adding %s to report" % self.report) # XXX: the original code written by juga0 readed

1 0