April 2016 - tor-commits - lists.torproject.org

[ooni-probe/master] Test that specific Failures are caught before parent Failures
by art＠torproject.org 29 Apr '16

29 Apr '16

commit 525a8438bd601592c4f878ca5d42d3dab8943be0 Author: srvetus <srvetus(a)users.noreply.github.com> Date: Thu Nov 19 14:16:42 2015 +0100 Test that specific Failures are caught before parent Failures --- ooni/tests/test_errors.py | 20 ++++++++++++++++++++ 1 file changed, 20 insertions(+) diff --git a/ooni/tests/test_errors.py b/ooni/tests/test_errors.py new file mode 100644 index 0000000..7baa1ea --- /dev/null +++ b/ooni/tests/test_errors.py @@ -0,0 +1,20 @@ +from twisted.trial import unittest + +import ooni.errors + +class TestErrors(unittest.TestCase): + + def test_catch_child_failures_before_parent_failures(self): + """ + Verify that more specific Failures are caught first by + handleAllFailures() and failureToString(). + + Fails if a subclass is listed after it's parent Failure. + """ + + # Check each Failure against all subsequent failures + for index, (failure, _) in enumerate(ooni.errors.known_failures): + for sub_failure, _ in ooni.errors.known_failures[index+1:]: + + # Fail if subsequent Failure inherits from the current Failure + self.assertNotIsInstance(sub_failure(None), failure)

1 0

[ooni-probe/master] Major fixes and enhancements to psipshon install script
by art＠torproject.org 29 Apr '16

29 Apr '16

commit 55f96585d62dcbbff41c01b9b8a97f5ee346c3d9 Author: anadahz <kojgelo(a)inbox.com> Date: Fri Nov 6 10:58:54 2015 +0100 Major fixes and enhancements to psipshon install script * Create required PSIPHON_PATH directory * Add command_exists function * Check if running as root user non sudo usage * Add required package dependencies * Fix check if hg command exits and if not install mercurial package * Check if virtualenv exists and if not install it * Fix minor format typos --- scripts/psiphon_install.sh | 42 ++++++++++++++++++++++++++++++++++++------ 1 file changed, 36 insertions(+), 6 deletions(-) diff --git a/scripts/psiphon_install.sh b/scripts/psiphon_install.sh index 1b6837d..208aa83 100755 --- a/scripts/psiphon_install.sh +++ b/scripts/psiphon_install.sh @@ -6,9 +6,35 @@ set -ex PSIPHON_PATH=$HOME/test PSIPHON_PYCLIENT_PATH=$PSIPHON_PATH/psiphon-circumvention-system/pyclient +mkdir -p $PSIPHON_PATH + +command_exists() { + command -v "$@" > /dev/null 2>&1 +} + +user="$(id -un 2>/dev/null || true)" + +sh_c='sh -c' + +if [ "$user" != 'root' ]; then + if command_exists sudo; then + sh_c='sudo sh -c -E' + elif command_exists su; then + sh_c='su -c --preserve-environment' + else + echo >&2 'Error: this installer needs the ability to run commands as root.' + echo >&2 'We are unable to find either "sudo" or "su" available to make this happen.' + exit 1 + fi +fi + echo "installing dependencies" -# FIXME: check if hg command exits and if not bail out? -sudo apt-get install -y mercurial +$sh_c "apt-get -y install zlib1g-dev libssl-dev" + +if [ -z "command_exists hg" ]; then + $sh_c "apt-get -y install mercurial" +fi + echo "cloning psiphon repository" cd $PSIPHON_PATH hg clone https://bitbucket.org/psiphon/psiphon-circumvention-system @@ -29,8 +55,9 @@ if [ python -c 'import sys; print hasattr(sys, "real_prefix")' = "False"];then # we are not in a virtualenv # create a virtualenv # FIXME: assuming debian version will have secure pip/virtualenv - sudo apt-get -y install python-virtualenv - + if [ -z "command_exists virtualenv" ]; then + $sh_c "apt-get -y install python-virtualenv" + fi if [ ! -f $HOME/.virtualenvs/ooniprobe/bin/activate ]; then # Set up the virtual environment mkdir -p $HOME/.virtualenvs @@ -72,9 +99,12 @@ dat["servers"] = json.load(open('server_list'))['data'].split() json.dump(dat, open('psi_client.dat', 'w')) EOF -chmod +x psi_generate_dat.py +chmod +x psi_generate_dat.py +./psi_generate_dat.py +echo "servers data file created" +chmod +x psi_generate_dat.py ./psi_generate_dat.py -echo "serers data file created" +echo "servers data file created" mv psi_client.dat $PSIPHON_PYCLIENT_PATH

1 0

[ooni-probe/master] Move general SOCKSError after more specific failures
by art＠torproject.org 29 Apr '16

29 Apr '16

commit 5f8bc2ca87dfcbc7ea666276c9a24b50e092c6c6 Author: srvetus <srvetus(a)users.noreply.github.com> Date: Thu Nov 19 13:52:47 2015 +0100 Move general SOCKSError after more specific failures --- ooni/errors.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/ooni/errors.py b/ooni/errors.py index 84c0afc..55b97e1 100644 --- a/ooni/errors.py +++ b/ooni/errors.py @@ -30,7 +30,6 @@ known_failures = [ (ResponseNeverReceived, 'response_never_received'), (DeferTimeoutError, 'deferred_timeout_error'), (GenericTimeoutError, 'generic_timeout_error'), - (SOCKSError, 'socks_error'), (MethodsNotAcceptedError, 'socks_methods_not_supported'), (AddressNotSupported, 'socks_address_not_supported'), (NetworkUnreachable, 'socks_network_unreachable'), @@ -43,6 +42,7 @@ known_failures = [ (ConnectionRefused, 'socks_connection_refused'), (TTLExpired, 'socks_ttl_expired'), (CommandNotSupported, 'socks_command_not_supported'), + (SOCKSError, 'socks_error'), (ProcessDone, 'process_done'), (ConnectionDone, 'connection_done'), (ConnectError, 'connect_error'),

1 0

[ooni-probe/master] Add psiphon install script
by art＠torproject.org 29 Apr '16

29 Apr '16

commit 75b88d7801fae016153e365600c3a6e649b613d0 Author: juga0 <juga> Date: Sun Sep 20 21:09:06 2015 +0000 Add psiphon install script --- scripts/psiphon_install.sh | 86 ++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 86 insertions(+) diff --git a/scripts/psiphon_install.sh b/scripts/psiphon_install.sh new file mode 100755 index 0000000..1b6837d --- /dev/null +++ b/scripts/psiphon_install.sh @@ -0,0 +1,86 @@ +#!/bin/sh + +#FIXME remove x +set -ex + +PSIPHON_PATH=$HOME/test +PSIPHON_PYCLIENT_PATH=$PSIPHON_PATH/psiphon-circumvention-system/pyclient + +echo "installing dependencies" +# FIXME: check if hg command exits and if not bail out? +sudo apt-get install -y mercurial +echo "cloning psiphon repository" +cd $PSIPHON_PATH +hg clone https://bitbucket.org/psiphon/psiphon-circumvention-system +echo "psiphon repository cloned" + +# optional, compile their ssh +echo "compiling psiphon ssh" +cd psiphon-circumvention-system/Server/3rdParty/openssh-5.9p1/ +./configure +make +mv ssh ../../../pyclient/ +make clean +echo "psiphon ssh compiled" + +# check if we are in a virtualenv, create it otherwise +echo "checking virtualenv" +if [ python -c 'import sys; print hasattr(sys, "real_prefix")' = "False"];then + # we are not in a virtualenv + # create a virtualenv + # FIXME: assuming debian version will have secure pip/virtualenv + sudo apt-get -y install python-virtualenv + + if [ ! -f $HOME/.virtualenvs/ooniprobe/bin/activate ]; then + # Set up the virtual environment + mkdir -p $HOME/.virtualenvs + virtualenv $HOME/.virtualenvs/ooniprobe + source $HOME/.virtualenvs/ooniprobe/bin/activate + else + source $HOME/.virtualenvs/ooniprobe/bin/activate + fi +fi +echo "virtualenv activated" + +# create psi_client.dat +echo "creating servers data file" +echo "installing dependencies to create servers data file" +pip install -v --timeout 60 wget +cd /tmp +cat <<EOF > psi_generate_dat.py +#!/usr/bin/env python + +import wget +import os +import json + +# Delete 'server_list' if exists +if os.path.exists("server_list"): + # os.remove("server_list") + # os.rename("server_list", "server_list") + pass +else: + # Download 'server_list' + url ="https://psiphon3.com/server_list" + wget.download(url) + +# convert server_list to psi_client.dat +dat = {} +dat["propagation_channel_id"] = "FFFFFFFFFFFFFFFF" +dat["sponsor_id"] = "FFFFFFFFFFFFFFFF" +dat["servers"] = json.load(open('server_list'))['data'].split() +json.dump(dat, open('psi_client.dat', 'w')) +EOF + +chmod +x psi_generate_dat.py +./psi_generate_dat.py +echo "serers data file created" +mv psi_client.dat $PSIPHON_PYCLIENT_PATH + + +echo "[+] Installing all of the Python dependency requirements with pip in your virtualenv!"; +pip install -v --timeout 60 jsonpickle pexpect + +# run psiphon +# cd $PSIPHON_PYCLIENT_PATH +# python psi_client.py

1 0

[ooni-probe/master] fixes and enhancements to psipshon install script
by art＠torproject.org 29 Apr '16

29 Apr '16

commit 955de5407b0ae3c4766ad9b48b99418ef2280a78 Author: juga0 <juga> Date: Tue Nov 10 17:35:09 2015 +0000 fixes and enhancements to psipshon install script * don't clone repository if the directory already exists * eliminate duplicated lines copying the server file * don't compile ssh client if the file already exists * fix syntax of line checking if a virtualenv is activated * fix psiphon directory path * remove x flag * remove commented lines for running psiphon * replace source command by dot * add variables for psiphon repo url and ooni virtualenv path * fix identation --- scripts/psiphon_install.sh | 89 +++++++++++++++++++++++----------------------- 1 file changed, 44 insertions(+), 45 deletions(-) diff --git a/scripts/psiphon_install.sh b/scripts/psiphon_install.sh index 208aa83..e5b2a78 100755 --- a/scripts/psiphon_install.sh +++ b/scripts/psiphon_install.sh @@ -3,13 +3,15 @@ #FIXME remove x set -ex -PSIPHON_PATH=$HOME/test +PSIPHON_PATH=$HOME PSIPHON_PYCLIENT_PATH=$PSIPHON_PATH/psiphon-circumvention-system/pyclient +PSIPHON_REPO_URL=https://bitbucket.org/psiphon/psiphon-circumvention-system +OONI_VIRTUALENV_PATH=$HOME/.virtualenvs/ooniprobe mkdir -p $PSIPHON_PATH command_exists() { - command -v "$@" > /dev/null 2>&1 + command -v "$@" > /dev/null 2>&1 } user="$(id -un 2>/dev/null || true)" @@ -17,55 +19,59 @@ user="$(id -un 2>/dev/null || true)" sh_c='sh -c' if [ "$user" != 'root' ]; then - if command_exists sudo; then - sh_c='sudo sh -c -E' - elif command_exists su; then - sh_c='su -c --preserve-environment' - else - echo >&2 'Error: this installer needs the ability to run commands as root.' - echo >&2 'We are unable to find either "sudo" or "su" available to make this happen.' - exit 1 - fi + if command_exists sudo; then + sh_c='sudo sh -c -E' + elif command_exists su; then + sh_c='su -c --preserve-environment' + else + echo >&2 'Error: this installer needs the ability to run commands as root.' + echo >&2 'We are unable to find either "sudo" or "su" available to make this happen.' + exit 1 + fi fi echo "installing dependencies" $sh_c "apt-get -y install zlib1g-dev libssl-dev" if [ -z "command_exists hg" ]; then - $sh_c "apt-get -y install mercurial" + $sh_c "apt-get -y install mercurial" fi echo "cloning psiphon repository" cd $PSIPHON_PATH -hg clone https://bitbucket.org/psiphon/psiphon-circumvention-system -echo "psiphon repository cloned" +if [ ! -d "psiphon-circumvention-system" ]; then + hg clone $PSIPHON_REPO_URL + echo "psiphon repository cloned" +fi # optional, compile their ssh -echo "compiling psiphon ssh" -cd psiphon-circumvention-system/Server/3rdParty/openssh-5.9p1/ -./configure -make -mv ssh ../../../pyclient/ -make clean -echo "psiphon ssh compiled" +if [ ! -f "$PSIPHON_PYCLIENT_PATH/ssh" ]; then + echo "compiling psiphon ssh" + cd psiphon-circumvention-system/Server/3rdParty/openssh-5.9p1/ + ./configure + make + mv ssh ../../../pyclient/ + make clean + echo "psiphon ssh compiled" +fi # check if we are in a virtualenv, create it otherwise echo "checking virtualenv" -if [ python -c 'import sys; print hasattr(sys, "real_prefix")' = "False"];then - # we are not in a virtualenv - # create a virtualenv - # FIXME: assuming debian version will have secure pip/virtualenv - if [ -z "command_exists virtualenv" ]; then - $sh_c "apt-get -y install python-virtualenv" - fi - if [ ! -f $HOME/.virtualenvs/ooniprobe/bin/activate ]; then - # Set up the virtual environment - mkdir -p $HOME/.virtualenvs - virtualenv $HOME/.virtualenvs/ooniprobe - source $HOME/.virtualenvs/ooniprobe/bin/activate - else - source $HOME/.virtualenvs/ooniprobe/bin/activate - fi +if [ `python -c 'import sys; print hasattr(sys, "real_prefix")'` = "False" ]; then + # not in a virtualenv + # create a virtualenv + # FIXME: assuming debian version will have secure pip/virtualenv + if [ -z "command_exists virtualenv" ]; then + $sh_c "apt-get -y install python-virtualenv" + fi + if [ ! -f $OONI_VIRTUALENV_PATH/bin/activate ]; then + # Set up the virtual environment + mkdir -p $HOME/.virtualenvs + virtualenv $OONI_VIRTUALENV_PATH + . $OONI_VIRTUALENV_PATH/bin/activate + else + . $OONI_VIRTUALENV_PATH/bin/activate + fi fi echo "virtualenv activated" @@ -83,7 +89,7 @@ import json # Delete 'server_list' if exists if os.path.exists("server_list"): - # os.remove("server_list") + # os.remove("server_list") # os.rename("server_list", "server_list") pass else: @@ -102,15 +108,8 @@ EOF chmod +x psi_generate_dat.py ./psi_generate_dat.py echo "servers data file created" -chmod +x psi_generate_dat.py -./psi_generate_dat.py -echo "servers data file created" mv psi_client.dat $PSIPHON_PYCLIENT_PATH - +rm /tmp/psi_generate_dat.py echo "[+] Installing all of the Python dependency requirements with pip in your virtualenv!"; pip install -v --timeout 60 jsonpickle pexpect - -# run psiphon -# cd $PSIPHON_PYCLIENT_PATH -# python psi_client.py

1 0

[ooni-probe/master] Refactor error handling to avoid inconsistencies
by art＠torproject.org 29 Apr '16

29 Apr '16

commit 29119b673f2880eedd47b0531d58bb8e7425dea8 Author: srvetus <srvetus(a)users.noreply.github.com> Date: Thu Sep 24 18:04:53 2015 +0200 Refactor error handling to avoid inconsistencies The error handling code in handleAllFailures() and failureToString() is inconsistent and does not correctly trap all of the Failures for which there are error messages defined. The revised code places the Exception and error message in a single place to avoid inconsistencies arising in future. --- ooni/errors.py | 153 ++++++++++++++------------------------------------------- 1 file changed, 38 insertions(+), 115 deletions(-) diff --git a/ooni/errors.py b/ooni/errors.py index cb18fe9..5751baf 100644 --- a/ooni/errors.py +++ b/ooni/errors.py @@ -20,39 +20,44 @@ from txsocksx.errors import TTLExpired, CommandNotSupported from socket import gaierror +known_failures = { + ConnectionRefusedError: 'connection_refused_error', + ConnectionLost: 'connection_lost_error', + ConnectError: 'connect_error', + CancelledError: 'task_timed_out', + gaierror: 'address_family_not_supported_error', + DNSLookupError: 'dns_lookup_error', + TCPTimedOutError: 'tcp_timed_out_error', + ResponseNeverReceived: 'response_never_received', + DeferTimeoutError: 'deferred_timeout_error', + GenericTimeoutError: 'generic_timeout_error', + SOCKSError: 'socks_error', + MethodsNotAcceptedError: 'socks_methods_not_supported', + AddressNotSupported: 'socks_address_not_supported', + NetworkUnreachable: 'socks_network_unreachable', + ConnectionError: 'socks_connect_error', + ConnectionLostEarly: 'socks_connection_lost_early', + ConnectionNotAllowed: 'socks_connection_not_allowed', + NoAcceptableMethods: 'socks_no_acceptable_methods', + ServerFailure: 'socks_server_failure', + HostUnreachable: 'socks_host_unreachable', + ConnectionRefused: 'socks_connection_refused', + TTLExpired: 'socks_ttl_expired', + CommandNotSupported: 'socks_command_not_supported', + ProcessDone: 'process_done', + ConnectionDone: 'connection_done', +} def handleAllFailures(failure): """ - Here we make sure to trap all the failures that are supported by the - failureToString function and we return the the string that represents the - failure. + Trap all the known Failures and we return a string that + represents the failure. Any unknown Failures will be reraised and + returned by failure.trap(). """ - failure.trap( - ConnectionRefusedError, - gaierror, - DNSLookupError, - TCPTimedOutError, - ResponseNeverReceived, - DeferTimeoutError, - GenericTimeoutError, - SOCKSError, - MethodsNotAcceptedError, - AddressNotSupported, - ConnectionError, - NetworkUnreachable, - ConnectionLostEarly, - ConnectionNotAllowed, - NoAcceptableMethods, - ServerFailure, - HostUnreachable, - ConnectionRefused, - TTLExpired, - CommandNotSupported, - ConnectError, - ConnectionLost, - CancelledError, - ConnectionDone) + # TODO: Should this function actually handle ALL failures and + # not just the failures configured in this list. + failure.trap(*known_failures.keys()) return failureToString(failure) @@ -69,95 +74,13 @@ def failureToString(failure): A string representing the HTTP response error message. """ - string = None - if isinstance(failure.value, ConnectionRefusedError): - # log.err("Connection refused.") - string = 'connection_refused_error' - elif isinstance(failure.value, ConnectionLost): - # log.err("Connection lost.") - string = 'connection_lost_error' - - elif isinstance(failure.value, ConnectError): - # log.err("Connect error.") - string = 'connect_error' - - elif isinstance(failure.value, gaierror): - # log.err("Address family for hostname not supported") - string = 'address_family_not_supported_error' - - elif isinstance(failure.value, DNSLookupError): - # log.err("DNS lookup failure") - string = 'dns_lookup_error' - - elif isinstance(failure.value, TCPTimedOutError): - # log.err("TCP Timed Out Error") - string = 'tcp_timed_out_error' - - elif isinstance(failure.value, ResponseNeverReceived): - # log.err("Response Never Received") - string = 'response_never_received' - - elif isinstance(failure.value, DeferTimeoutError): - # log.err("Deferred Timeout Error") - string = 'deferred_timeout_error' - - elif isinstance(failure.value, GenericTimeoutError): - # log.err("Time Out Error") - string = 'generic_timeout_error' - - elif isinstance(failure.value, ServerFailure): - # log.err("SOCKS error: ServerFailure") - string = 'socks_server_failure' - - elif isinstance(failure.value, ConnectionNotAllowed): - # log.err("SOCKS error: ConnectionNotAllowed") - string = 'socks_connection_not_allowed' - - elif isinstance(failure.value, NetworkUnreachable): - # log.err("SOCKS error: NetworkUnreachable") - string = 'socks_network_unreachable' - - elif isinstance(failure.value, HostUnreachable): - # log.err("SOCKS error: HostUnreachable") - string = 'socks_host_unreachable' - - elif isinstance(failure.value, ConnectionRefused): - # log.err("SOCKS error: ConnectionRefused") - string = 'socks_connection_refused' - - elif isinstance(failure.value, TTLExpired): - # log.err("SOCKS error: TTLExpired") - string = 'socks_ttl_expired' - - elif isinstance(failure.value, CommandNotSupported): - # log.err("SOCKS error: CommandNotSupported") - string = 'socks_command_not_supported' - - elif isinstance(failure.value, AddressNotSupported): - # log.err("SOCKS error: AddressNotSupported") - string = 'socks_address_not_supported' - - elif isinstance(failure.value, SOCKSError): - # log.err("Generic SOCKS error") - string = 'socks_error' - - elif isinstance(failure.value, CancelledError): - # log.err("Task timed out") - string = 'task_timed_out' - - elif isinstance(failure.value, ProcessDone): - string = 'process_done' - - elif isinstance(failure.value, ConnectionDone): - string = 'connection_done' - - else: - # log.err("Unknown failure type: %s" % type(failure.value)) - string = 'unknown_failure %s' % str(failure.value) - - return string + for failure_type, failure_string in known_failures.items(): + if isinstance(failure.value, failure_type): + return failure_string + # Did not find a matching failure message + return 'unknown_failure %s' % str(failure.value) class DirectorException(Exception): pass

1 0

[ooni-probe/master] Remove unneeded imports, fix call to variables out of scope.
by art＠torproject.org 29 Apr '16

29 Apr '16

commit 6a1a3c1f02802db6f1c006916b0469078fa5525b Author: Arturo Filastò <arturo(a)filasto.net> Date: Fri Feb 5 16:00:58 2016 +0100 Remove unneeded imports, fix call to variables out of scope. --- ooni/oonicli.py | 19 ++++++++----------- 1 file changed, 8 insertions(+), 11 deletions(-) diff --git a/ooni/oonicli.py b/ooni/oonicli.py index fc5eca0..b3e1013 100644 --- a/ooni/oonicli.py +++ b/ooni/oonicli.py @@ -8,7 +8,7 @@ import urlparse from twisted.python import usage from twisted.python.util import spewer -from twisted.internet import defer, reactor, protocol, task +from twisted.internet import defer, reactor, protocol from ooni import errors, __version__ @@ -123,8 +123,7 @@ def director_startup_handled_failures(failure): if isinstance(failure.value, errors.TorNotRunning): log.err("Tor does not appear to be running") - log.err("Reporting with the collector %s is not possible" % - global_options['collector']) + log.err("Reporting with a collector is not possible") log.msg( "Try with a different collector or disable collector reporting with -n") @@ -226,8 +225,8 @@ def setupCollector(global_options, net_test_loader): return collector -def createDeck(global_options, url=None, filename=None): - log.msg("Creating deck for: %s" % (url or filename,)) +def createDeck(global_options, url=None): + log.msg("Creating deck for: %s" % (url)) deck = Deck(no_collector=global_options['no-collector']) deck.bouncer = global_options['bouncer'] @@ -240,8 +239,6 @@ def createDeck(global_options, url=None, filename=None): test_file = nettest_to_path(global_options['test_file'], True) if url is not None: args = ('-u', url) - elif filename is not None: - args = ('-f', filename) else: args = tuple() if any(global_options['subargs']): @@ -271,9 +268,9 @@ def createDeck(global_options, url=None, filename=None): return deck -def runTestWithDirector(director, global_options, url=None, filename=None, +def runTestWithDirector(director, global_options, url=None, start_tor=True, check_incoherences=True): - deck = createDeck(global_options, url=url, filename=filename) + deck = createDeck(global_options, url=url) start_tor |= deck.requiresTor @@ -343,7 +340,7 @@ def runWithDirector(logging=True, start_tor=True, check_incoherences=True): sys.exit(0) if global_options.get('annotations') is not None: - annotations = setupAnnotations(global_options) + global_options['annotations'] = setupAnnotations(global_options) if global_options['no-collector']: log.msg("Not reporting using a collector") @@ -384,7 +381,7 @@ def runWithDaemonDirector(logging=True, start_tor=True, check_incoherences=True) director = Director() if global_options.get('annotations') is not None: - annotations = setupAnnotations(global_options) + global_options['annotations'] = setupAnnotations(global_options) if global_options['no-collector']: log.msg("Not reporting using a collector")

1 0

[ooni-probe/master] Squashed commit of the following:
by art＠torproject.org 29 Apr '16

29 Apr '16

commit 6d2728ca795a97f6929c37579432ebcf909097d8 Author: Daniel Ramsay <daniel(a)dretzq.org.uk> Date: Mon Aug 31 23:14:52 2015 +0100 Squashed commit of the following: Add support for ORG's blocked.org.uk scheduler Add: * ORG Queue integration * Support for additional values to send to ORG backend Refactor director startup * Moved collector setup to shared scope * Moved deck setup to shared scope * Moved annotations setup to shared scope * Moved global options to shared scope * fix some naming convention errors * move global counter to more sensible scope * fixes from pull request code review * remove another mobile browser * remove iPhone useragent to prevent mobile versions being returned to client * add HTTPS support for reporter * add authentication info from environment * import PROBE_ vars from environment into report * allow https collector in regex * AMQP connection details as cmdline parameter * more queue state, looping listen call * remove old bin script * exit after bounded random number of test decks * improve error handling * use proper custom exception for queue timeout reason * reduce duplicate serialization * add flag to prevent response body being written to report * add subargs to queue createdeck * fix global args passing * reuse cached geoip * reuse existing TOR connection * simplify for one-shot use * extend and report lifetime * probe exit on AMQP error * remove ooniresource from setup script --- Vagrantfile | 2 + bin/ooniprobequeue | 21 ++ ooni/director.py | 2 +- ooni/geoip.py | 57 +++-- ooni/nettests/blocking/http_requests.py | 3 + ooni/oonicli.py | 426 +++++++++++++++++++++++--------- ooni/reporter.py | 19 +- ooni/templates/httpt.py | 2 +- ooni/utils/net.py | 4 +- setup.py | 5 +- 10 files changed, 384 insertions(+), 157 deletions(-) diff --git a/Vagrantfile b/Vagrantfile index 16a6ba0..a06dc41 100644 --- a/Vagrantfile +++ b/Vagrantfile @@ -12,6 +12,8 @@ Vagrant.configure("2") do |config| # Place the ooni-backend source code in ../ooni-backend to sync it with the vagrant instance config.vm.synced_folder "../ooni-backend", "/backend" + config.vm.network :private_network, ip: "192.168.38.20" + end $script = <<SCRIPT diff --git a/bin/ooniprobequeue b/bin/ooniprobequeue new file mode 100755 index 0000000..f81f156 --- /dev/null +++ b/bin/ooniprobequeue @@ -0,0 +1,21 @@ +#!/usr/bin/env python +import sys +import copy_reg +from twisted.internet import reactor + +# This is a hack to overcome a bug in python +from ooni.utils.hacks import patched_reduce_ex +copy_reg._reduce_ex = patched_reduce_ex + +code=0 + +from ooni.oonicli import runWithDaemonDirector +d = runWithDaemonDirector() +(a)d.addBoth +def cb(result): + global code + if result is not None: + code=1 + reactor.stop() +reactor.run() +sys.exit(code) diff --git a/ooni/director.py b/ooni/director.py index 5ffee97..9bac49d 100644 --- a/ooni/director.py +++ b/ooni/director.py @@ -132,7 +132,7 @@ class Director(object): yield config.check_tor() if config.advanced.start_tor: yield self.startTor() - elif config.tor.control_port: + elif config.tor.control_port and config.tor_state is None: log.msg("Connecting to Tor Control Port...") yield self.getTorState() diff --git a/ooni/geoip.py b/ooni/geoip.py index 0ab2475..cb66675 100644 --- a/ooni/geoip.py +++ b/ooni/geoip.py @@ -190,34 +190,37 @@ class ProbeIP(object): @defer.inlineCallbacks def lookup(self): - try: - yield self.askTor() - log.msg("Found your IP via Tor %s" % self.address) - self.resolveGeodata() + if self.address: defer.returnValue(self.address) - except errors.TorStateNotFound: - log.debug("Tor is not running. Skipping IP lookup via Tor.") - except Exception: - log.msg("Unable to lookup the probe IP via Tor.") - - try: - yield self.askTraceroute() - log.msg("Found your IP via Traceroute %s" % self.address) - self.resolveGeodata() - defer.returnValue(self.address) - except errors.InsufficientPrivileges: - log.debug("Cannot determine the probe IP address with a traceroute, becase of insufficient privileges") - except: - log.msg("Unable to lookup the probe IP via traceroute") - - try: - yield self.askGeoIPService() - log.msg("Found your IP via a GeoIP service: %s" % self.address) - self.resolveGeodata() - defer.returnValue(self.address) - except Exception: - log.msg("Unable to lookup the probe IP via GeoIPService") - raise + else: + try: + yield self.askTor() + log.msg("Found your IP via Tor %s" % self.address) + self.resolveGeodata() + defer.returnValue(self.address) + except errors.TorStateNotFound: + log.debug("Tor is not running. Skipping IP lookup via Tor.") + except Exception: + log.msg("Unable to lookup the probe IP via Tor.") + + try: + yield self.askTraceroute() + log.msg("Found your IP via Traceroute %s" % self.address) + self.resolveGeodata() + defer.returnValue(self.address) + except errors.InsufficientPrivileges: + log.debug("Cannot determine the probe IP address with a traceroute, becase of insufficient priviledges") + except: + log.msg("Unable to lookup the probe IP via traceroute") + + try: + yield self.askGeoIPService() + log.msg("Found your IP via a GeoIP service: %s" % self.address) + self.resolveGeodata() + defer.returnValue(self.address) + except Exception: + log.msg("Unable to lookup the probe IP via GeoIPService") + raise @defer.inlineCallbacks def askGeoIPService(self): diff --git a/ooni/nettests/blocking/http_requests.py b/ooni/nettests/blocking/http_requests.py index 997fb4e..f21b041 100644 --- a/ooni/nettests/blocking/http_requests.py +++ b/ooni/nettests/blocking/http_requests.py @@ -17,6 +17,9 @@ class UsageOptions(usage.Options): ['url', 'u', None, 'Specify a single URL to test.'], ['factor', 'f', 0.8, 'What factor should be used for triggering censorship (0.8 == 80%)']] + optFlags = [ + ["withoutbody","B", "don't include HTTP response body inside of the report"], + ] class HTTPRequestsTest(httpt.HTTPTest): diff --git a/ooni/oonicli.py b/ooni/oonicli.py index 030014e..47aacfb 100644 --- a/ooni/oonicli.py +++ b/ooni/oonicli.py @@ -1,10 +1,14 @@ import sys + import os +import json import yaml +import random +import urlparse from twisted.python import usage from twisted.python.util import spewer -from twisted.internet import defer +from twisted.internet import defer, reactor, protocol, task from ooni import errors, __version__ @@ -17,6 +21,9 @@ from ooni.utils import log from ooni.utils.net import hasRawSocketPermission + +class LifetimeExceeded(Exception): pass + class Options(usage.Options): synopsis = """%s [options] [path to test].py """ % (os.path.basename(sys.argv[0]),) @@ -49,7 +56,8 @@ class Options(usage.Options): ["datadir", "d", None, "Specify a path to the ooniprobe data directory"], ["annotations", "a", None, - "Annotate the report with a key:value[, key:value] format."]] + "Annotate the report with a key:value[, key:value] format."], + ["queue", "Q", None, "AMQP Queue URL amqp://user:pass@host:port/vhost/queue"]] compData = usage.Completions( extraActions=[usage.CompleteFiles( @@ -102,11 +110,63 @@ def parseOptions(): return dict(cmd_line_options) -def runWithDirector(logging=True, start_tor=True, check_incoherences=True): - """ - Instance the director, parse command line options and start an ooniprobe - test! - """ +def director_startup_handled_failures(failure): + log.err("Could not start the director") + failure.trap(errors.TorNotRunning, + errors.InvalidOONIBCollectorAddress, + errors.UnableToLoadDeckInput, + errors.CouldNotFindTestHelper, + errors.CouldNotFindTestCollector, + errors.ProbeIPUnknown, + errors.InvalidInputFile, + errors.ConfigFileIncoherent) + + if isinstance(failure.value, errors.TorNotRunning): + log.err("Tor does not appear to be running") + log.err("Reporting with the collector %s is not possible" % + global_options['collector']) + log.msg( + "Try with a different collector or disable collector reporting with -n") + + elif isinstance(failure.value, errors.InvalidOONIBCollectorAddress): + log.err("Invalid format for oonib collector address.") + log.msg( + "Should be in the format http://<collector_address>:<port>") + log.msg("for example: ooniprobe -c httpo://nkvphnp3p6agi5qq.onion") + + elif isinstance(failure.value, errors.UnableToLoadDeckInput): + log.err("Unable to fetch the required inputs for the test deck.") + log.msg( + "Please file a ticket on our issue tracker: https://github.com/thetorproject/ooni-probe/issues") + + elif isinstance(failure.value, errors.CouldNotFindTestHelper): + log.err("Unable to obtain the required test helpers.") + log.msg( + "Try with a different bouncer or check that Tor is running properly.") + + elif isinstance(failure.value, errors.CouldNotFindTestCollector): + log.err("Could not find a valid collector.") + log.msg( + "Try with a different bouncer, specify a collector with -c or disable reporting to a collector with -n.") + + elif isinstance(failure.value, errors.ProbeIPUnknown): + log.err("Failed to lookup probe IP address.") + log.msg("Check your internet connection.") + + elif isinstance(failure.value, errors.InvalidInputFile): + log.err("Invalid input file \"%s\"" % failure.value) + + elif isinstance(failure.value, errors.ConfigFileIncoherent): + log.err("Incoherent config file") + + if config.advanced.debug: + log.exception(failure) + +def director_startup_other_failures(failure): + log.err("An unhandled exception occurred while starting the director!") + log.exception(failure) + +def setupGlobalOptions(logging, start_tor, check_incoherences): global_options = parseOptions() config.global_options = global_options config.set_paths() @@ -133,49 +193,44 @@ def runWithDirector(logging=True, start_tor=True, check_incoherences=True): log.err("Insufficient Privileges to capture packets." " See ooniprobe.conf privacy.includepcap") sys.exit(2) - - director = Director() - if global_options['list']: - print "# Installed nettests" - for net_test_id, net_test in director.getNetTests().items(): - print "* %s (%s/%s)" % (net_test['name'], - net_test['category'], - net_test['id']) - print " %s" % net_test['description'] - - sys.exit(0) - - elif global_options['printdeck']: - del global_options['printdeck'] - print "# Copy and paste the lines below into a test deck to run the specified test with the specified arguments" - print yaml.safe_dump([{'options': global_options}]).strip() - - sys.exit(0) - - if global_options.get('annotations') is not None: - annotations = {} - for annotation in global_options["annotations"].split(","): - pair = annotation.split(":") - if len(pair) == 2: - key = pair[0].strip() - value = pair[1].strip() - annotations[key] = value - else: - log.err("Invalid annotation: %s" % annotation) - sys.exit(1) - global_options["annotations"] = annotations - - if global_options['no-collector']: - log.msg("Not reporting using a collector") - global_options['collector'] = None - start_tor = False - else: - start_tor = True + return global_options + +def setupAnnotations(global_options): + annotations={} + for annotation in global_options["annotations"].split(","): + pair = annotation.split(":") + if len(pair) == 2: + key = pair[0].strip() + value = pair[1].strip() + annotations[key] = value + else: + log.err("Invalid annotation: %s" % annotation) + sys.exit(1) + global_options["annotations"] = annotations + return annotations + +def setupCollector(global_options, net_test_loader): + collector = None + if not global_options['no-collector']: + if global_options['collector']: + collector = global_options['collector'] + elif 'collector' in config.reports \ + and config.reports['collector']: + collector = config.reports['collector'] + elif net_test_loader.collector: + collector = net_test_loader.collector + + if collector and collector.startswith('httpo:') \ + and (not (config.tor_state or config.tor.socks_port)): + raise errors.TorNotRunning + return collector + + +def createDeck(global_options,url=None,filename=None): + log.msg("Creating deck for: %s" %(url or filename,) ) deck = Deck(no_collector=global_options['no-collector']) deck.bouncer = global_options['bouncer'] - if global_options['collector']: - start_tor |= True try: if global_options['testdeck']: @@ -183,7 +238,15 @@ def runWithDirector(logging=True, start_tor=True, check_incoherences=True): else: log.debug("No test deck detected") test_file = nettest_to_path(global_options['test_file'], True) - net_test_loader = NetTestLoader(global_options['subargs'], + if url is not None: + args = ('-u',url) + elif filename is not None: + args = ('-f',filename) + else: + args = tuple() + if any(global_options['subargs']): + args = global_options['subargs'] + args + net_test_loader = NetTestLoader(args, test_file=test_file) if global_options['collector']: net_test_loader.collector = global_options['collector'] @@ -205,6 +268,48 @@ def runWithDirector(logging=True, start_tor=True, check_incoherences=True): log.exception(e) log.err(e) sys.exit(5) + return deck + +def runWithDirector(logging=True, start_tor=True, check_incoherences=True): + """ + Instance the director, parse command line options and start an ooniprobe + test! + """ + + global_options = setupGlobalOptions(logging, start_tor, check_incoherences) + + director = Director() + if global_options['list']: + print "# Installed nettests" + for net_test_id, net_test in director.getNetTests().items(): + print "* %s (%s/%s)" % (net_test['name'], + net_test['category'], + net_test['id']) + print " %s" % net_test['description'] + + sys.exit(0) + + elif global_options['printdeck']: + del global_options['printdeck'] + print "# Copy and paste the lines below into a test deck to run the specified test with the specified arguments" + print yaml.safe_dump([{'options': global_options}]).strip() + + sys.exit(0) + + if global_options.get('annotations') is not None: + annotations = setupAnnotations(global_options) + + if global_options['no-collector']: + log.msg("Not reporting using a collector") + global_options['collector'] = None + start_tor = False + else: + start_tor = True + + if global_options['collector']: + start_tor |= True + + deck = createDeck(global_options) start_tor |= deck.requiresTor d = director.start(start_tor=start_tor, @@ -216,61 +321,6 @@ def runWithDirector(logging=True, start_tor=True, check_incoherences=True): except errors.UnableToLoadDeckInput as error: return defer.failure.Failure(error) - def director_startup_handled_failures(failure): - log.err("Could not start the director") - failure.trap(errors.TorNotRunning, - errors.InvalidOONIBCollectorAddress, - errors.UnableToLoadDeckInput, - errors.CouldNotFindTestHelper, - errors.CouldNotFindTestCollector, - errors.ProbeIPUnknown, - errors.InvalidInputFile, - errors.ConfigFileIncoherent) - - if isinstance(failure.value, errors.TorNotRunning): - log.err("Tor does not appear to be running") - log.err("Reporting with the collector %s is not possible" % - global_options['collector']) - log.msg( - "Try with a different collector or disable collector reporting with -n") - - elif isinstance(failure.value, errors.InvalidOONIBCollectorAddress): - log.err("Invalid format for oonib collector address.") - log.msg( - "Should be in the format http://<collector_address>:<port>") - log.msg("for example: ooniprobe -c httpo://nkvphnp3p6agi5qq.onion") - - elif isinstance(failure.value, errors.UnableToLoadDeckInput): - log.err("Unable to fetch the required inputs for the test deck.") - log.msg( - "Please file a ticket on our issue tracker: https://github.com/thetorproject/ooni-probe/issues") - - elif isinstance(failure.value, errors.CouldNotFindTestHelper): - log.err("Unable to obtain the required test helpers.") - log.msg( - "Try with a different bouncer or check that Tor is running properly.") - - elif isinstance(failure.value, errors.CouldNotFindTestCollector): - log.err("Could not find a valid collector.") - log.msg( - "Try with a different bouncer, specify a collector with -c or disable reporting to a collector with -n.") - - elif isinstance(failure.value, errors.ProbeIPUnknown): - log.err("Failed to lookup probe IP address.") - log.msg("Check your internet connection.") - - elif isinstance(failure.value, errors.InvalidInputFile): - log.err("Invalid input file \"%s\"" % failure.value) - - elif isinstance(failure.value, errors.ConfigFileIncoherent): - log.err("Incoherent config file") - - if config.advanced.debug: - log.exception(failure) - - def director_startup_other_failures(failure): - log.err("An unhandled exception occurred while starting the director!") - log.exception(failure) # Wait until director has started up (including bootstrapping Tor) # before adding tests @@ -285,19 +335,7 @@ def runWithDirector(logging=True, start_tor=True, check_incoherences=True): # deck is a singleton, the default collector set in # ooniprobe.conf will be used - collector = None - if not global_options['no-collector']: - if global_options['collector']: - collector = global_options['collector'] - elif 'collector' in config.reports \ - and config.reports['collector']: - collector = config.reports['collector'] - elif net_test_loader.collector: - collector = net_test_loader.collector - - if collector and collector.startswith('httpo:') \ - and (not (config.tor_state or config.tor.socks_port)): - raise errors.TorNotRunning + collector = setupCollector(global_options, net_test_loader) test_details = net_test_loader.testDetails test_details['annotations'] = global_options['annotations'] @@ -307,11 +345,163 @@ def runWithDirector(logging=True, start_tor=True, check_incoherences=True): collector) return director.allTestsDone - def start(): + + d.addCallback(setup_nettest) + d.addCallback(post_director_start) + d.addErrback(director_startup_handled_failures) + d.addErrback(director_startup_other_failures) + return d + + + +# this variant version of runWithDirector splits the process in two, +# allowing a single director instance to be reused with multiple decks. + +def runWithDaemonDirector(logging=True, start_tor=True, check_incoherences=True): + """ + Instance the director, parse command line options and start an ooniprobe + test! + """ + + try: + import pika + from pika import exceptions + from pika.adapters import twisted_connection + except ImportError: + print "Pika is required for queue connection." + print "Install with \"pip install pika\"." + sys.exit(7) + + + + global_options = setupGlobalOptions(logging, start_tor, check_incoherences) + + director = Director() + + if global_options.get('annotations') is not None: + annotations = setupAnnotations(global_options) + + if global_options['no-collector']: + log.msg("Not reporting using a collector") + global_options['collector'] = None + start_tor = False + else: + start_tor = True + + + def run_test(global_options, url=None, filename=None): + assert url is not None or filename is not None + + deck = createDeck(global_options, url=url, filename=filename) + + d = director.start(start_tor=True, + check_incoherences=check_incoherences) + + def setup_nettest(_): + try: + return deck.setup() + except errors.UnableToLoadDeckInput as error: + return defer.failure.Failure(error) + + + + # Wait until director has started up (including bootstrapping Tor) + # before adding tests + def post_director_start(_): + for net_test_loader in deck.netTestLoaders: + # Decks can specify different collectors + # for each net test, so that each NetTest + # may be paired with a test_helper and its collector + # However, a user can override this behavior by + # specifying a collector from the command-line (-c). + # If a collector is not specified in the deck, or the + # deck is a singleton, the default collector set in + # ooniprobe.conf will be used + + collector = setupCollector(global_options, net_test_loader) + + test_details = net_test_loader.testDetails + test_details['annotations'] = global_options['annotations'] + + director.startNetTest(net_test_loader, + global_options['reportfile'], + collector) + return director.allTestsDone + d.addCallback(setup_nettest) d.addCallback(post_director_start) d.addErrback(director_startup_handled_failures) d.addErrback(director_startup_other_failures) return d - return start() + finished = defer.Deferred() + + @defer.inlineCallbacks + def readmsg(_,channel, queue_object, consumer_tag, counter): + + # Wait for a message and decode it. + if counter >= lifetime: + queue_object.close(LifetimeExceeded()) + yield channel.basic_cancel(consumer_tag=consumer_tag) + finished.callback(None) + + else: + log.msg("Waiting for message") + + try: + ch, method, properties, body = yield queue_object.get() + log.msg("Got message") + data = json.loads(body) + counter += 1 + + log.msg("Received %d/%d: %s" %(counter, lifetime, data['url'],)) + # acknowledge the message + ch.basic_ack(delivery_tag=method.delivery_tag) + + d = run_test(global_options, url=data['url'].encode('utf8')) + # When the test has been completed, go back to waiting for a message. + d.addCallback(readmsg, channel, queue_object, consumer_tag, counter+1) + except exceptions.AMQPError,v: + finished.errback(v) + + + + @defer.inlineCallbacks + def runQueue(connection, name, qos): + # Set up the queue consumer. When a message is received, run readmsg + channel = yield connection.channel() + yield channel.basic_qos(prefetch_count=qos) + queue_object, consumer_tag = yield channel.basic_consume( + queue=name, + no_ack=False) + readmsg(None,channel,queue_object,consumer_tag, 0) + + + + # Create the AMQP connection. This could be refactored to allow test URLs + # to be submitted through an HTTP server interface or something. + urlp = urlparse.urlparse(config.global_options['queue']) + urlargs = dict(urlparse.parse_qsl(urlp.query)) + + # random lifetime requests counter + lifetime = random.randint(820,1032) + + # AMQP connection details are sent through the cmdline parameter '-Q' + + creds = pika.PlainCredentials(urlp.username or 'guest', + urlp.password or 'guest') + parameters = pika.ConnectionParameters(urlp.hostname, + urlp.port or 5672, + urlp.path.rsplit('/',1)[0] or '/', + creds, + heartbeat_interval=120, + ) + cc = protocol.ClientCreator(reactor, + twisted_connection.TwistedProtocolConnection, + parameters) + d = cc.connectTCP(urlp.hostname, urlp.port or 5672) + d.addCallback(lambda protocol: protocol.ready) + # start the wait/process sequence. + d.addCallback(runQueue, urlp.path.rsplit('/',1)[-1], int(urlargs.get('qos',1))) + + return finished diff --git a/ooni/reporter.py b/ooni/reporter.py index 36a0e88..5e702bb 100644 --- a/ooni/reporter.py +++ b/ooni/reporter.py @@ -240,7 +240,7 @@ class OONIBReporter(OReporter): Will raise :class:ooni.errors.InvalidOONIBCollectorAddress an exception if the oonib reporter is not valid. """ - regexp = '^(http|httpo):\/\/[a-zA-Z0-9\-\.]+(:\d+)?$' + regexp = '^(http|https|httpo):\/\/[a-zA-Z0-9\-\.]+(:\d+)?$' if not re.match(regexp, self.collectorAddress): raise errors.InvalidOONIBCollectorAddress @@ -265,7 +265,7 @@ class OONIBReporter(OReporter): request_json = json.dumps(request) log.debug("Sending %s" % request_json) - bodyProducer = StringProducer(json.dumps(request)) + bodyProducer = StringProducer(request_json) try: yield self.agent.request("PUT", url, @@ -288,6 +288,7 @@ class OONIBReporter(OReporter): # tor is started. from ooni.utils.hacks import SOCKS5Agent + from twisted.web.client import Agent from twisted.internet import reactor if self.collectorAddress.startswith('httpo://'): @@ -298,8 +299,13 @@ class OONIBReporter(OReporter): self.agent = SOCKS5Agent(reactor, proxyEndpoint=proxyEndpoint) elif self.collectorAddress.startswith('https://'): - # XXX add support for securely reporting to HTTPS collectors. - log.err("HTTPS based collectors are currently not supported.") + # not sure if there's something else it needs. Seems to work. + # Very difficult to get it to work with self-signed certs. + self.agent = Agent(reactor) + + elif self.collectorAddress.startswith('http://'): + log.msg("Warning using unencrypted collector") + self.agent = Agent(reactor) url = self.collectorAddress + '/report' @@ -319,12 +325,15 @@ class OONIBReporter(OReporter): # the backend. 'content': content } + # import values from the environment + request.update([(k.lower(),v) for (k,v) in os.environ.iteritems() + if k.startswith('PROBE_')]) log.msg("Reporting %s" % url) request_json = json.dumps(request) log.debug("Sending %s" % request_json) - bodyProducer = StringProducer(json.dumps(request)) + bodyProducer = StringProducer(request_json) log.msg("Creating report with OONIB Reporter. Please be patient.") log.msg("This may take up to 1-2 minutes...") diff --git a/ooni/templates/httpt.py b/ooni/templates/httpt.py index e77913d..a753955 100644 --- a/ooni/templates/httpt.py +++ b/ooni/templates/httpt.py @@ -136,7 +136,7 @@ class HTTPTest(NetTestCase): if response: request_response['response'] = { 'headers': list(response.headers.getAllRawHeaders()), - 'body': response_body, + 'body': response_body if self.localOptions.get('withoutbody',1) == 0 else '', 'code': response.code } if failure_string: diff --git a/ooni/utils/net.py b/ooni/utils/net.py index 014020c..96d62a1 100644 --- a/ooni/utils/net.py +++ b/ooni/utils/net.py @@ -27,8 +27,6 @@ except ImportError: # agents with largest anonymity set. userAgents = ("Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1.7) Gecko/20091221 Firefox/3.5.7", - "Mozilla/5.0 (iPhone; U; CPU iPhone OS 3 1 2 like Mac OS X; en-us)" - "AppleWebKit/528.18 (KHTML, like Gecko) Mobile/7D11", "Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6", "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6", "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6", @@ -169,4 +167,4 @@ def hasRawSocketPermission(): socket.socket(socket.AF_INET, socket.SOCK_RAW, socket.IPPROTO_RAW) return True except socket.error: - return False \ No newline at end of file + return False diff --git a/setup.py b/setup.py index 9b874f3..d99a90f 100644 --- a/setup.py +++ b/setup.py @@ -156,7 +156,7 @@ class install(_st_install): prefix = os.path.abspath(self.prefix) self.set_data_files(prefix) self.do_egg_install() - self.ooniresources() + #self.ooniresources() install_requires = [] dependency_links = [] @@ -204,7 +204,8 @@ setup( packages=packages, include_package_data=True, scripts=["bin/oonideckgen", "bin/ooniprobe", - "bin/oonireport", "bin/ooniresources"], + "bin/oonireport", "bin/ooniresources", + "bin/ooniprobequeue"], dependency_links=dependency_links, install_requires=install_requires, cmdclass={"install": install},

1 0

[ooni-probe/master] Improve Readme and fix typo in test_settings
by art＠torproject.org 29 Apr '16

29 Apr '16

commit b9dbbb6c918a11b3887dbcb8e21f4d47ef0585cd Author: Arturo <art(a)fuffa.org> Date: Fri Nov 20 13:49:49 2015 +0100 Improve Readme and fix typo in test_settings --- README.rst | 1 - ooni/tests/test_settings.py | 2 +- 2 files changed, 1 insertion(+), 2 deletions(-) diff --git a/README.rst b/README.rst index 5c043cd..71a253e 100644 --- a/README.rst +++ b/README.rst @@ -393,4 +393,3 @@ On Debian based systems a development environment can be setup as follows: (prer python setup.py install ooniprobe -s # if all went well, lists available tests - diff --git a/ooni/tests/test_settings.py b/ooni/tests/test_settings.py index 6b7c3b9..f94ca14 100644 --- a/ooni/tests/test_settings.py +++ b/ooni/tests/test_settings.py @@ -79,7 +79,7 @@ class TestSettings(ConfigTestCase): @defer.inlineCallbacks def test_check_tor_correct(self): """ - This test has been disabled because there is a starge concatenation of + This test has been disabled because there is a strange concatenation of conditions that make it not possible to run it on travis. The tests need to be run as root on travis so that the ones that use scapy will work properly. When running tor as root, though, it will by

1 0

[ooni-probe/master] Merge pull request #459 from TheTorProject/doc/fix-test-helpers-doc
by art＠torproject.org 29 Apr '16

29 Apr '16

commit a9d48f3efae4d29d616fdc5dfbcf318a760a6048 Merge: 1fac7cd b6c84cd Author: Arturo Filastò <arturo(a)filasto.net> Date: Fri Feb 5 13:57:00 2016 +0100 Merge pull request #459 from TheTorProject/doc/fix-test-helpers-doc Doc/fix test helpers doc docs/source/writing_test_helpers.rst | 26 +++++++++++++------------- 1 file changed, 13 insertions(+), 13 deletions(-)

1 0