November 2015 - tor-commits - lists.torproject.org

[ooni-probe/master] Make bin/ooniprobe-dev portable
by art＠torproject.org 20 Nov '15

20 Nov '15

commit cbeff0f327045c34ed4ea744a981a04224152e3b Author: Simone Basso <bassosimone(a)gmail.com> Date: Fri Oct 9 09:38:42 2015 +0200 Make bin/ooniprobe-dev portable - use a portable recipe to get the absolute path of the toplevel directory where ooni-probe sources are located [1] - make sure we exit with a meaningful error message if it is not possible to get the toplevel directory name - while there put quotes around $@ since according to [2] this is the best way to forward shell script arguments .. [1] http://serverfault.com/questions/40144 .. [2] http://stackoverflow.com/questions/255898/how-to-iterate-over-arguments-in-… --- bin/ooniprobe-dev | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/bin/ooniprobe-dev b/bin/ooniprobe-dev index b85d3a3..7eaaf40 100755 --- a/bin/ooniprobe-dev +++ b/bin/ooniprobe-dev @@ -2,5 +2,9 @@ # Developer script for running ooniprobe directly from the repository. # We don't automatically add "$PWD" to PYTHONPATH as that is a security risk # when run as /usr/bin/ooniprobe on an end user's computer. -cd "$(realpath "$(dirname "$0")")/.." -PYTHONPATH="$PWD" exec ./bin/ooniprobe $@ +ROOTDIR=$(cd $(dirname $(dirname $0)) && pwd -P) +if [ $? -ne 0 ]; then + echo "$0: cannot determine toplevel directory" 1>&2 + exit 1 +fi +PYTHONPATH="$ROOTDIR" exec ./bin/ooniprobe "$@"

1 0

[ooni-probe/master] Merge pull request #422 from TheTorProject/fix/oonireport-error-message
by art＠torproject.org 20 Nov '15

20 Nov '15

commit 7bf8c8bf2e51683d2599ef23a590ecc043794fd3 Merge: 8c6dad7 a3aad91 Author: Arturo Filastò <arturo(a)filasto.net> Date: Sun Nov 8 10:21:48 2015 +0100 Merge pull request #422 from TheTorProject/fix/oonireport-error-message Fixes spotted while debugging lepidopter issue bin/oonireport | 2 +- ooni/reporter.py | 2 ++ 2 files changed, 3 insertions(+), 1 deletion(-)

1 0

[ooni-probe/master] Add meek fronted requests documentation
by art＠torproject.org 20 Nov '15

20 Nov '15

commit 91cc6993c019fb40356dcb40822fb2f92129e776 Author: anadahz <kojgelo(a)inbox.com> Date: Fri Oct 16 01:33:35 2015 +0200 Add meek fronted requests documentation --- docs/source/tests/meek_fronted_requests.rst | 176 +++++++++++++++++++++++++++ 1 file changed, 176 insertions(+) diff --git a/docs/source/tests/meek_fronted_requests.rst b/docs/source/tests/meek_fronted_requests.rst new file mode 100644 index 0000000..b865e18 --- /dev/null +++ b/docs/source/tests/meek_fronted_requests.rst @@ -0,0 +1,176 @@ +Details +======= + +*Test Name*: Meek Fronted Request Test + +*Current version*: 0.0.1 + +*NetTest*: HTTP Requests (https://gitweb.torproject.org/ooni-probe.git/blob/HEAD:/ooni/nettests/block…) + +*Test Helper*: None + +*Test Type*: Content Blocking, Traffic Manipulation + +*Requires Root*: No + +Description +=========== + +Performs a HTTP GET request to a list of fronted domains with the Host +Header of the "inside" meek-server. The meek-server handles a GET request and +response with: "I’m just a happy little web server.\n". + +It tests if the fronted request/response to the meek server is successful. + + +How to run the test +=================== + +To test if the default meek servers are blocked: + +`ooniprobe blocking/meek_fronted_requests` + +To test a set of meek servers from a list containing hosts to test run: + +`ooniprobe blocking/meek_fronted_requests -f <input_file>` + +*input file* a list of domainName:hostHeader pairs to perform the test. + +Optional test options +===================== + +*-B* Expected body content from GET response (meek server default: 'I’m just a +happy little web server.\n') +*-D* Specify a single fronted domainName to test. +*-H* Specify "inside" Host Header to test. + +Sample report +============= + +`ooniprobe blocking/meek_fronted_requests` + +:: + +########################################### +# OONI Probe Report for meek_fronted_requests_test (0.0.1) +# Thu Oct 15 22:01:32 2015 +########################################### +--- +input_hashes: [] +options: [] +probe_asn: AS20676 +probe_cc: DE +probe_city: null +probe_ip: 127.0.0.1 +report_id: null +software_name: ooniprobe +software_version: 1.3.1 +start_time: 1444932091.0 +test_helpers: {} +test_name: meek_fronted_requests_test +test_version: 0.0.1 +... +--- +agent: agent +input: [a0.awsstatic.com, d2zfqthxsdq309.cloudfront.net] +requests: +- request: + body: null + headers: + - - Host + - [d2zfqthxsdq309.cloudfront.net] + method: GET + tor: {is_tor: false} + url: https://a0.awsstatic.com + response: + body: "I\u2019m just a happy little web server.\n" + code: 200 + headers: + - - Content-Length + - ['38'] + - - Via + - [1.1 15191055e43ba835d0fead01ae84015c.cloudfront.net (CloudFront)] + - - X-Cache + - [Hit from cloudfront] + - - Age + - ['530'] + - - Connection + - [close] + - - X-Amz-Cf-Id + - [PKUBrpXDpoi2rSZ-WV0YUzX1wPg6JylZ_37iQeRQJB-xDLtJddcxzw==] + - - Date + - ['Thu, 15 Oct 2015 19:52:47 GMT'] + - - Content-Type + - [text/plain; charset=utf-8] +socksproxy: null +success: true +test_runtime: 0.3198120594024658 +test_start_time: 1444932097.0 +... +--- +agent: agent +input: [ajax.aspnetcdn.com, az668014.vo.msecnd.net] +requests: +- request: + body: null + headers: + - - Host + - [az668014.vo.msecnd.net] + method: GET + tor: {is_tor: false} + url: https://ajax.aspnetcdn.com + response: + body: "I\u2019m just a happy little web server.\n" + code: 200 + headers: + - - Date + - ['Thu, 15 Oct 2015 20:01:37 GMT'] + - - Content-Length + - ['38'] + - - Content-Type + - [text/plain; charset=utf-8] + - - Connection + - [close] + - - Server + - [ECAcc (fcn/40C4)] +socksproxy: null +success: true +test_runtime: 0.4580512046813965 +test_start_time: 1444932097.0 +... +--- +agent: agent +input: [www.google.com, meek-reflect.appspot.com] +requests: +- request: + body: null + headers: + - - Host + - [meek-reflect.appspot.com] + method: GET + tor: {is_tor: false} + url: https://www.google.com + response: + body: "I\u2019m just a happy little web server.\n" + code: 200 + headers: + - - Alternate-Protocol + - ['443:quic,p=1'] + - - Content-Length + - ['38'] + - - Server + - [Google Frontend] + - - Connection + - [close] + - - Date + - ['Thu, 15 Oct 2015 20:01:38 GMT'] + - - Alt-Svc + - ['quic=":443"; p="1"; ma=604800'] + - - Content-Type + - [text/plain; charset=utf-8] +socksproxy: null +success: true +test_runtime: 0.5862560272216797 +test_start_time: 1444932097.0 +... + ...

1 0

[ooni-probe/master] Fix misleading error message
by art＠torproject.org 20 Nov '15

20 Nov '15

commit 7e31feb94ff1d1f6b751c1b9b698a08a100a9a1b Author: Simone Basso <bassosimone(a)gmail.com> Date: Wed Nov 4 16:02:32 2015 +0100 Fix misleading error message --- bin/oonireport | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/bin/oonireport b/bin/oonireport index 1335a83..6a79f4c 100755 --- a/bin/oonireport +++ b/bin/oonireport @@ -14,7 +14,7 @@ def failed(failure): r = failure.trap(exceptions.SystemExit, Exception) if r != exceptions.SystemExit: - log.err("Failed to run oonideckgen") + log.err("Failed to run oonireport") log.exception(failure) exitCode = 127 else:

1 0

[ooni-probe/master] Properly add BTC address
by art＠torproject.org 20 Nov '15

20 Nov '15

commit 866ed353f2f49a29aab73f41789d900325099066 Author: Arturo Filastò <arturo(a)filasto.net> Date: Wed Nov 11 19:15:12 2015 +0100 Properly add BTC address --- README.rst | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/README.rst b/README.rst index e4174d3..7c624dc 100644 --- a/README.rst +++ b/README.rst @@ -397,9 +397,10 @@ On Debian based systems a development environment can be setup as follows: (prer Donate ------- -Send bitcoins to: +Send bitcoins to .. image:: http://i.imgur.com/CIWHb5R.png :target: http://www.coindesk.com/information/how-can-i-buy-bitcoins/ - 1Ai9d4dhDBjxYVkKKf1pFXptEGfM1vxFBf + +1Ai9d4dhDBjxYVkKKf1pFXptEGfM1vxFBf

1 0

[ooni-probe/master] deal with existing, but empty, reporting.yml
by art＠torproject.org 20 Nov '15

20 Nov '15

commit a3aad913eccbad3bcd390b3cb73f52f11dddaadc Author: Simone Basso <bassosimone(a)gmail.com> Date: Thu Nov 5 14:47:13 2015 +0100 deal with existing, but empty, reporting.yml Spotted while helping @anadahz to debug lepidopter. --- ooni/reporter.py | 2 ++ 1 file changed, 2 insertions(+) diff --git a/ooni/reporter.py b/ooni/reporter.py index 36a0e88..b0df393 100644 --- a/ooni/reporter.py +++ b/ooni/reporter.py @@ -395,6 +395,8 @@ class OONIBReportLog(object): def get_report_log(self): with open(self.file_name) as f: report_log = yaml.safe_load(f) + if not report_log: + report_log = {} # consumers expect dictionary structure return report_log @property

1 0

[ooni-probe/master] psiphon: fix bug that psiphon hangs ooniprobe -ng
by art＠torproject.org 20 Nov '15

20 Nov '15

commit 1172083dff321c2544e4dfaa7c312528a0226512 Author: Simone Basso <bassosimone(a)gmail.com> Date: Sun Nov 15 16:58:37 2015 +0100 psiphon: fix bug that psiphon hangs ooniprobe -ng Marked my fix with XXX because I'm not sure whether this is the proper approach, or whether it would be better to work on a patch to teach ooniprobe's core to deal with that case. For now, and given that a psiphon MVP must be delivered as soon as possible, this fix will do. --- ooni/nettests/third_party/psiphon.py | 12 +++++++++--- 1 file changed, 9 insertions(+), 3 deletions(-) diff --git a/ooni/nettests/third_party/psiphon.py b/ooni/nettests/third_party/psiphon.py index 637a505..543e856 100644 --- a/ooni/nettests/third_party/psiphon.py +++ b/ooni/nettests/third_party/psiphon.py @@ -95,12 +95,18 @@ connect(False) log.debug('PsiphonTest.test_psiphon') if not os.path.exists(self.psiphonpath): - log.debug('psiphon path does not exists, is it installed?') + log.err('psiphon path does not exists, is it installed?') self.report['success'] = False self.report['psiphon_installed'] = False log.debug("Adding %s to report" % self.report) - #FIXME: this completes the test but ooni doesn't stop running, why? - return defer.succeed(None) + # XXX: the original code written by juga0 readed + # > return defer.succeed(None) + # but this caused `ooniprobe -ng` to hang forever, so I + # rewrote the code to return a deferred and simulate calling + # its callback method, to trigger an event. + # -sbs + reactor.callLater(0.0, self.bootstrapped.callback, None) + return self.bootstrapped self.report['psiphon_installed'] = True log.debug("Adding %s to report" % self.report)

1 0

[ooni-probe/master] Update Debian package info
by art＠torproject.org 20 Nov '15

20 Nov '15

commit 0945a51ae9db3ff06e43a7a0396f0a534f156765 Author: KwadroNaut <KwadroNaut(a)users.noreply.github.com> Date: Fri Oct 30 09:12:11 2015 +0100 Update Debian package info Package isn't in wheezy-backports anymore, but on it's way in the current Debian Stable: https://lists.debian.org/debian-backports/2015/10/msg00030.html --- README.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.rst b/README.rst index 68a56c0..5c043cd 100644 --- a/README.rst +++ b/README.rst @@ -53,7 +53,7 @@ On debian testing or unstable:: If you are running debian stable you can get it from backports via:: - sudo sh -c 'echo "deb http://http.debian.net/debian wheezy-backports main" >> /etc/apt/sources.list' + sudo sh -c 'echo "deb http://http.debian.net/debian jessie-backports main" >> /etc/apt/sources.list' sudo apt-get update && sudo apt-get install ooniprobe On unix systems::

1 0

[ooni-probe/master] initial psiphon test. Failure cases doesn't fail propertly
by art＠torproject.org 20 Nov '15

20 Nov '15

commit 2176d0dd113d0e9754d2efb4ce1f9a17716134b0 Author: juga0 <juga> Date: Sun Sep 20 21:09:06 2015 +0000 initial psiphon test. Failure cases doesn't fail propertly --- ooni/nettests/third_party/psiphon.py | 141 ++++++++++++++++++++++++++++++++++ 1 file changed, 141 insertions(+) diff --git a/ooni/nettests/third_party/psiphon.py b/ooni/nettests/third_party/psiphon.py new file mode 100644 index 0000000..f8cb82a --- /dev/null +++ b/ooni/nettests/third_party/psiphon.py @@ -0,0 +1,141 @@ +import tempfile +import stat +import os + +from twisted.internet import defer, reactor +from twisted.internet.endpoints import TCP4ClientEndpoint +from twisted.web.client import readBody +from twisted.python import usage + +from txsocksx.http import SOCKS5Agent + +from ooni.errors import handleAllFailures, TaskTimedOut +from ooni.utils import log +from ooni.templates import process, httpt +from ooni.templates.process import ProcessTest + + +class UsageOptions(usage.Options): + log.debug("UsageOptions") + optParameters = [ + ['url', 'u', None, 'Specify a single URL to test.'], + ['psiphonpath', 'p', None, 'Specify psiphon python client path.'], + ['socksproxy', 's', None, 'Specify psiphon socks proxy ip:port.'],] + + +class PsiphonTest(httpt.HTTPTest, process.ProcessTest): + + """ + This class tests Psiphon python client + + test_psiphon: + Starts a Psiphon, check if it bootstraps successfully + (print a line in stdout). + Then, perform an HTTP request using the proxy + """ + + name = "Psiphon Test" + description = "Bootstraps Psiphon and \ + does a HTTP GET for the specified URL" + author = "juga" + version = "0.0.1" + timeout = 20 + usageOptions = UsageOptions + + def _setUp(self): + # it is necessary to do this in _setUp instead of setUp + # because it needs to happen before HTTPTest's _setUp. + # incidentally, setting this option in setUp results in HTTPTest + # *saying* it is using this proxy while not actually using it. + log.debug('PiphonTest._setUp: setting socksproxy') + self.localOptions['socksproxy'] = '127.0.0.1:1080' + super(PsiphonTest, self)._setUp() + + def setUp(self): + log.debug('PsiphonTest.setUp') + + self.bootstrapped = defer.Deferred() + if self.localOptions['url']: + self.url = self.localOptions['url'] + else: + # FIXME: use http://google.com? + # self.url = 'https://wtfismyip.com/text' + self.url = 'https://check.torproject.orggg' + + if self.localOptions['psiphonpath']: + self.psiphonpath = self.localOptions['psiphonpath'] + else: + # FIXME: search for pyclient path instead of assuming is in the + # home? + # psiphon is not installable and to run it manually, it has to be + # run from the psiphon directory, so it wouldn't make sense to + # nstall it in the PATH + from os import path, getenv + self.psiphonpath = path.join( + getenv('HOME'), + 'psiphon-circumvention-system/pyclient') + log.debug('psiphon path: %s' % self.psiphonpath) + + x = """#!/usr/bin/env python +from psi_client import connect +connect(False) +""" + f = tempfile.NamedTemporaryFile(delete=False) + f.write(x) + f.close() + os.chmod(f.name, os.stat(f.name).st_mode | stat.S_IEXEC) + self.command = [f.name] + log.debug('command: %s' % ''.join(self.command)) + + def handleRead(self, stdout, stderr): + if 'Press Ctrl-C to terminate.' in self.processDirector.stdout: + if not self.bootstrapped.called: + log.debug("PsiphonTest.test_psiphon: calling bootstrapped.callback") + self.bootstrapped.callback(None) + + def test_psiphon(self): + log.debug('PsiphonTest.test_psiphon') + + if not os.path.exists(self.psiphonpath): + log.debug('psiphon path does not exists, is it installed?') + self.report['success'] = False + self.report['psiphon_installed'] = False + log.debug("Adding %s to report" % self.report) + #FIXME: this completes the test but ooni doesn't stop running, why? + return defer.succeed(None) + + self.report['psiphon_installed'] = True + log.debug("Adding %s to report" % self.report) + + finished = self.run(self.command, + env=dict(PYTHONPATH=self.psiphonpath), + path=self.psiphonpath, + usePTY=1) + + + def addFailureToReport(failure): + log.debug("PsiphonTest.test_psiphon.addFailureToReport") + log.debug(repr(failure )) + self.report['failure'] = handleAllFailures(failure) + self.report['success'] = False + log.debug("Adding %s to report" % self.report) + # FIXME: these keys are not being wroten in the report + # probably because report is being defined in NetTestCase as + # a class attribute that is initialized again in NetTescase._setUp + + def callDoRequest(_): + return self.doRequest(self.url) + self.bootstrapped.addCallback(callDoRequest) + + def cleanup(_): + log.debug('PsiphonTest:cleanup') + self.processDirector.transport.signalProcess('INT') + os.remove(self.command[0]) + return finished + + self.bootstrapped.addErrback(addFailureToReport) + self.bootstrapped.addBoth(cleanup) + return self.bootstrapped + + +

1 0

[ooni-probe/master] Merge pull request #420 from TheTorProject/doc/meek-fronted-requests-doc
by art＠torproject.org 20 Nov '15

20 Nov '15

commit 9e2d2a75626e329e6a321826c74e2d10b8320254 Merge: d14f7f6 91cc699 Author: Arturo Filastò <arturo(a)filasto.net> Date: Mon Nov 9 10:51:09 2015 +0100 Merge pull request #420 from TheTorProject/doc/meek-fronted-requests-doc Add meek fronted requests documentation docs/source/tests/meek_fronted_requests.rst | 176 +++++++++++++++++++++++++++ 1 file changed, 176 insertions(+)

1 0