April 2015 - tor-commits - lists.torproject.org

[ooni-probe/master] 13602: build a binary of bin/ooniprobe that we can setcap on to avoid running as root
by art＠torproject.org 15 Apr '15

15 Apr '15

commit 7b01492e082550a0917bb9141601fdfdd41fd79f Author: Ximin Luo <infinity0(a)pwned.gg> Date: Sun Mar 1 14:54:09 2015 +0100 13602: build a binary of bin/ooniprobe that we can setcap on to avoid running as root --- bin/Makefile | 81 +++++++++++++++++++++++++++++++++++++++++++++ bin/test/ooni/__init__.py | 1 + 2 files changed, 82 insertions(+) diff --git a/bin/Makefile b/bin/Makefile new file mode 100644 index 0000000..bc80843 --- /dev/null +++ b/bin/Makefile @@ -0,0 +1,81 @@ +# Wrappers for running ooniprobe as a non-root user. +# +# Build-Depends: cython, pythonX.Y-dev, libcap2-bin +# Depends: libpythonX.Y +# +# $ make && make check +# $ sudo make install # after installing the rest of ooni-probe +# $ make installcheck_unsafe # runs complete tests as non-root +# +# `make` builds a program that has file capabilities set on it. This is just +# ./ooniprobe compiled into a C program using Cython, so that one can set +# capabilities directly on the resulting binary. This way, we avoid the need +# for a separate child python interpreter with its own capabilities. Another +# advantage is that libpython.so (needed by the program) would be automatically +# upgraded by the system package manager. The version of python is hard-coded +# into the wrapper at build time; making this dynamic is possible, but much +# more complex and not yet implemented. +# +# Execution may additionally be limited to a particular unix group by using +# chgrp(1) and chmod(1) to 'o-x,g+x' after installation. +# + +# GNU Makefile conventions, see https://www.gnu.org/prep/standards/html_node/Makefile-Conventions.html +prefix = /usr/local +exec_prefix = $(prefix) +bindir = $(exec_prefix)/bin + +INSTALL = install +PYTHON = python +PYTHON_CONFIG = python-config +CYTHON = cython +SETCAP = setcap + +INSTALL_PROGRAM = $(INSTALL) +PY_CFLAGS = $(shell $(PYTHON_CONFIG) --cflags) +PY_LDFLAGS = $(shell $(PYTHON_CONFIG) --ldflags) + +BUILDDIR := ./build +SCRIPTDIR := . +TESTDIR := ./test +CAP_SCRIPT := ooniprobe +CAP_NEEDED := cap_net_admin,cap_net_raw + +# Unfortunately cython --embed ignores the arguments in the shebang line +# So we need to patch the generated code ourselves. +CYTHON_PRE_MAIN = extern int Py_IgnoreEnvironmentFlag; \ + Py_IgnoreEnvironmentFlag++; \ + extern int Py_NoUserSiteDirectory; \ + Py_NoUserSiteDirectory++; + +all: $(BUILDDIR)/$(CAP_SCRIPT) + +$(BUILDDIR)/$(CAP_SCRIPT): $(BUILDDIR)/$(CAP_SCRIPT).c Makefile + $(CC) $(PY_CFLAGS) $(PY_LDFLAGS) "$<" -o "$@" + +$(BUILDDIR)/$(CAP_SCRIPT).c: $(SCRIPTDIR)/$(CAP_SCRIPT) Makefile + mkdir -p "$(BUILDDIR)" + $(CYTHON) "$<" --embed=CYTHON_MAIN_SENTINEL -Werror -Wextra -o "$@" + sed -i \ + -e 's/$.*CYTHON_MAIN_SENTINEL.*{$/\1 $(CYTHON_PRE_MAIN)/g' \ + -e '/CYTHON_MAIN_SENTINEL[^{]*$$/,/{/s/{/{ $(CYTHON_PRE_MAIN)/g' \ + -e 's/CYTHON_MAIN_SENTINEL/main/g' "$@" + +check: $(BUILDDIR)/$(CAP_SCRIPT) + # test that setcapped binary ignores PYTHONPATH + BIN="$$(realpath "$<")" && cd "$(TESTDIR)" && PYTHONPATH=. $$BIN --version + +install: $(BUILDDIR)/$(CAP_SCRIPT) + mkdir -p "$(DESTDIR)$(bindir)" + $(INSTALL_PROGRAM) -t "$(DESTDIR)$(bindir)" "$(BUILDDIR)/$(CAP_SCRIPT)" + $(SETCAP) "$(CAP_NEEDED)"+eip "$(DESTDIR)$(bindir)/$(CAP_SCRIPT)" + +installcheck_unsafe: $(BUILDDIR)/$(CAP_SCRIPT) + # run a standard check. note that because of hardcoded paths (for security) + # this can only work after you've installed your development copy + "./$<" -i /usr/share/ooni/decks/complete.deck + +clean: + rm -rf "$(BUILDDIR)" + +.PHONY: clean all check install installcheck% diff --git a/bin/test/ooni/__init__.py b/bin/test/ooni/__init__.py new file mode 100644 index 0000000..f2cd353 --- /dev/null +++ b/bin/test/ooni/__init__.py @@ -0,0 +1 @@ +raise ValueError("test failed! wrapper did not ignore polluted PWD. either the wrapper is faulty, or ooni is still unpatched (Tor bug #13581)")

1 0

[spec/master] Merge pull request #36 from TheTorProject/feature/informed-consent
by art＠torproject.org 15 Apr '15

15 Apr '15

commit 87957c30924ef9be56d7614b87fc5ab03d59b18f Merge: fac37d8 897132e Author: Arturo Filastò <arturo(a)filasto.net> Date: Mon Mar 16 17:29:06 2015 +0100 Merge pull request #36 from TheTorProject/feature/informed-consent Feature/informed consent informed-consent/inform-users-long.md | 75 ++++++++++++++++++++++++++++++++ informed-consent/inform-users-short.md | 8 ++++ 2 files changed, 83 insertions(+)

1 0

[spec/master] Add long informed consent page
by art＠torproject.org 15 Apr '15

15 Apr '15

commit aa3f22c0701f583a6af3198a5980610322cca45b Author: Arturo Filastò <art(a)fuffa.org> Date: Mon Mar 16 13:31:19 2015 +0100 Add long informed consent page --- informed-consent/inform-users-long.md | 14 ++++++++++++++ 1 file changed, 14 insertions(+) diff --git a/informed-consent/inform-users-long.md b/informed-consent/inform-users-long.md index e76cfd2..e1e21ef 100644 --- a/informed-consent/inform-users-long.md +++ b/informed-consent/inform-users-long.md @@ -1,3 +1,17 @@ +# GOAL + +OONI is a project aimed at mapping network anomalies by performing measurements +from the vantage point of a users. +You may find details on what exactly it is that we measure by reading through +our [test specifications](https://github.com/TheTorProject/ooni-spec/tree/master/test…. +The data we collect is then made available to the public for further analysis. + +The goal of this study is to have a better understanding on how the internet +works around the world and what is stopping it from working as it should. + +Please take a minute to go through what can be the possible implications for +running ooniprobe and make sure you properly understand them before proceeding. + # LEGALITY Before running OONI inside of a country with controversial internet access

1 0

[spec/master] Fix issues identified by @bassosimone
by art＠torproject.org 15 Apr '15

15 Apr '15

commit 897132ef41c4e4201e4fcee4b370cab213725f9a Author: Arturo Filastò <art(a)fuffa.org> Date: Mon Mar 16 16:45:14 2015 +0100 Fix issues identified by @bassosimone --- informed-consent/inform-users-long.md | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/informed-consent/inform-users-long.md b/informed-consent/inform-users-long.md index e1e21ef..80e5134 100644 --- a/informed-consent/inform-users-long.md +++ b/informed-consent/inform-users-long.md @@ -18,8 +18,8 @@ Before running OONI inside of a country with controversial internet access related laws you should consult a lawyer. You may also reach out to us with specific inquiries at legal(a)ooni.nu. -Some things that OONI does may be against the terms of service of your ISP or -legally questionable by your country. +Some network tests that OONI does may be against the terms of service of your +ISP or legally questionable by your country. OONI's http test will download data from controversial websites, specifically targeting those which may be censored in your country. @@ -31,7 +31,7 @@ Even where these sites are not blocked, it may be illegal to access them. It may also be illegal to bypass censorship, as OONI attempts by using Tor. -In the most extreme case, any form of network monitoring could be +In the most extreme case, any form of active network measurement could be illegal or banned, or even considered a form of espionage. Some useful resources may be: @@ -47,7 +47,7 @@ about your internet connection to the whole world. Particular groups, such as your ISP and web services used by the ooni tests, will be able to discover even more detailed information about you. -THE PUBLIC will be able to see the information collected by OONIprobe. +THE PUBLIC will be able to see the information collected by ooniprobe. This will definitely include your approximate location, the network (ASN) you are connecting from, and when you ran ooniprobe. Other identifying information, such as your IP address, is not deliberately @@ -69,7 +69,7 @@ likely be able to link it to you personally. These organizations might include your government, your ISP, and your employer. ANYBODY WITH ACCESS TO YOUR COMPUTER, now or in the future, may be able -to detect that you have installed or run ooni +to detect that you have installed or run ooni. SERVICES CONNECTED TO BY OONI will be able to see your IP address, and -may be able to detect that you are using OONI +may be able to detect that you are using OONI.

1 0

[spec/master] Add test_helpers and report_id keys to report header
by art＠torproject.org 15 Apr '15

15 Apr '15

commit e21e84c736141955852dcd4bfcaa9b8a3e9feb04 Author: Arturo Filastò <art(a)fuffa.org> Date: Wed Mar 25 11:52:53 2015 +0100 Add test_helpers and report_id keys to report header --- data-formats/df-000-base.md | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-) diff --git a/data-formats/df-000-base.md b/data-formats/df-000-base.md index 8ca41c8..27120d8 100644 --- a/data-formats/df-000-base.md +++ b/data-formats/df-000-base.md @@ -24,7 +24,7 @@ format. This specification is of the basic data format common to all ooniprobe test outputs. -Data Format Version: df-000-base-000 +Data Format Version: df-000-base-001 ## Specification @@ -62,6 +62,12 @@ Data Format Version: df-000-base-000 data_format_version: The version string of the data format being used by the test (ex. httpt-000) + + report_id: + A 64 character mixed case string that is generated by the client used to identify the report. + + test_helpers: + A dictionary with as keys the names of the options and values the addresses of the test helpers used ... # Example output @@ -89,6 +95,8 @@ Data Format Version: df-000-base-000 start_time: 1359401639.0 test_name: HTTP Invalid Request Line test_version: 0.1.3 + test_helpers: {backend: "http://93.95.227.200"} + report_id: xxxxxxxXXXxXXXxxxxxxxxxxxxXXXxXXXxsxxxXXXxXXXxxxXXXxXXXxxxXXXxX ...

1 0

[spec/master] Make some changes to the wording of some of the texts
by art＠torproject.org 15 Apr '15

15 Apr '15

commit f5ee53e52ffb450acca5905770dc897dc1fa7759 Author: Arturo Filastò <art(a)fuffa.org> Date: Mon Mar 16 13:24:20 2015 +0100 Make some changes to the wording of some of the texts --- informed-consent/inform-users-long.md | 22 ++++++++++++++-------- informed-consent/inform-users-short.md | 18 ++++++++---------- 2 files changed, 22 insertions(+), 18 deletions(-) diff --git a/informed-consent/inform-users-long.md b/informed-consent/inform-users-long.md index fdfebbd..e76cfd2 100644 --- a/informed-consent/inform-users-long.md +++ b/informed-consent/inform-users-long.md @@ -1,7 +1,11 @@ # LEGALITY -OONI does several things which may be illegal in your country, and/or -banned by your ISP. +Before running OONI inside of a country with controversial internet access +related laws you should consult a lawyer. You may also reach out to us with +specific inquiries at legal(a)ooni.nu. + +Some things that OONI does may be against the terms of service of your ISP or +legally questionable by your country. OONI's http test will download data from controversial websites, specifically targeting those which may be censored in your country. @@ -16,8 +20,11 @@ using Tor. In the most extreme case, any form of network monitoring could be illegal or banned, or even considered a form of espionage. -[Include link to some resource on relevant laws globally. Someone like -the EFF must have one of these; does anybody have a link?] +Some useful resources may be: + +* [Tor Legal FAQ](https://www.eff.org/torchallenge/faq.html) + +* [EFF Know Your Rights](https://www.eff.org/issues/know-your-rights) # PRIVACY @@ -37,11 +44,10 @@ content based on your network location. You can see what information OONI releases to the public at https://ooni.torproject.org/reports/. You should expect this information -to remain online PERMANENTLY. [include details of retention policy, once -we have one] +to remain online PERMANENTLY. -THE OONI PROJECT will also be able to see your IP address [What other -info do we get?] +THE OONI PROJECT when running some tests against our backends will be able to +see your IP address. ORGANIZATIONS MONITORING YOUR INTERNET CONNECTION will be able to see all web traffic generated by OONI, including your IP address, and will diff --git a/informed-consent/inform-users-short.md b/informed-consent/inform-users-short.md index 0422d5a..1a6ac0c 100644 --- a/informed-consent/inform-users-short.md +++ b/informed-consent/inform-users-short.md @@ -1,10 +1,8 @@ -WARNING: Running OONI may be illegal in your country, or forbidden by -your ISP. By running OONI you will connect to web services which may be -banned, and use web censorship circumvention methods such as Tor. The -OONI project will publish data submitted by probes, possibly including -your IP address or other identifying information. In addition, your use -of OONI will be clear to anybody who has access to your computer, and to -anybody who can monitor your internet connection (such as your employer, -ISP or government). - -[link to long version] +WARNING: Running OONI may be against the terms of service of your ISP +or legally questionable in your country. +By running OONI you will connect to web services which may be banned, and use +web censorship circumvention methods such as Tor. The OONI project will publish +data submitted by probes, possibly including your IP address or other +identifying information. In addition, your use of OONI will be clear to anybody +who has access to your computer, and to anybody who can monitor your internet +connection (such as your employer, ISP or government).

1 0

[spec/master] Merge pull request #37 from TheTorProject/feature/report_id_test_helpers
by art＠torproject.org 15 Apr '15

15 Apr '15

commit c01997fa17dd21ed7e2d3865576f025d8012d90d Merge: 902a1ae e21e84c Author: Arturo Filastò <arturo(a)filasto.net> Date: Fri Mar 27 11:40:55 2015 +0100 Merge pull request #37 from TheTorProject/feature/report_id_test_helpers Add test_helpers and report_id keys to report header data-formats/df-000-base.md | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-)

1 0

[spec/master] Fix meek test spec (PR #38)
by art＠torproject.org 15 Apr '15

15 Apr '15

commit 45d28710b40effd941fd0af989b6de8cc532747d Author: anadahz <kojgelo(a)inbox.com> Date: Thu Apr 9 01:46:57 2015 +0200 Fix meek test spec (PR #38) --- test-specs/ts-013-meek_fronted_requests.md | 17 ++++++++++------- 1 file changed, 10 insertions(+), 7 deletions(-) diff --git a/test-specs/ts-013-meek_fronted_requests.md b/test-specs/ts-013-meek_fronted_requests.md index f211cc4..f78066e 100644 --- a/test-specs/ts-013-meek_fronted_requests.md +++ b/test-specs/ts-013-meek_fronted_requests.md @@ -36,9 +36,12 @@ Example: # Test description -Performs a HTTP GET request to a list of fronted domains with the Host -Header of the "inside" meek-server. The meek-server handles a GET request and -response with: "I’m just a happy little web server.\n". +Performs a HTTP GET request over TLS (HTTPS) to a list of fronted domains with +the Host Header of the "inside" meek-server. For diagnostic purposes the +meek-server handles a GET request and respond with: "I’m just a happy little +web server.\n". The GET request is sent over TLS to the root of the fronted +domain with the Host Header option of the desired meek-server host. + # Expected output @@ -55,12 +58,12 @@ df-001-httpt-000 * The received responses -* If the meek request is blocked +* If the meek server is blocked or unreachable ## Semantics -censored: - **boolean** indicates if an HTTP GET response to the meek server is +success: + **boolean** indicates if an HTTPS GET response to the meek server is successfull ## Possible conclusions @@ -71,7 +74,6 @@ If the fronted request/response to the meek server is successful. ``` agent: agent -censored: false input: ajax.aspnetcdn.com:az668014.vo.msecnd.net requests: - request: @@ -105,4 +107,5 @@ requests: - - Content-Type - [text/plain; charset=utf-8] socksproxy: null +success: true ```

1 0

[spec/master] Merge pull request #35 from irl/master
by art＠torproject.org 15 Apr '15

15 Apr '15

commit 902a1ae6c0857f51d172dfc299278541970d1f01 Merge: 87957c3 de49e3f Author: Arturo Filastò <arturo(a)filasto.net> Date: Wed Mar 25 11:54:18 2015 +0100 Merge pull request #35 from irl/master Added reverse traceroute test helper specification test-helpers/th-006-reverse-traceroute.md | 52 +++++++++++++++++++++++++++++ 1 file changed, 52 insertions(+)

1 0

[spec/master] Rename meek test for naming consistency
by art＠torproject.org 15 Apr '15

15 Apr '15

commit 84b8d1cafa42802ef98caad3013f870db7a62a25 Author: Arturo Filastò <art(a)fuffa.org> Date: Wed Apr 15 19:09:54 2015 +0200 Rename meek test for naming consistency --- test-specs/ts-013-meek-fronted-requests.md | 111 ++++++++++++++++++++++++++++ test-specs/ts-013-meek_fronted_requests.md | 111 ---------------------------- 2 files changed, 111 insertions(+), 111 deletions(-) diff --git a/test-specs/ts-013-meek-fronted-requests.md b/test-specs/ts-013-meek-fronted-requests.md new file mode 100644 index 0000000..f78066e --- /dev/null +++ b/test-specs/ts-013-meek-fronted-requests.md @@ -0,0 +1,111 @@ +# Specification version number + +2015-04-01-000 + +# Specification name + +Meek Fronted Request Test + +# Test preconditions + +* An internet connection + +# Expected impact + +Ability to detect uncensored fronted domains that can transmit requests to the +"inside" meek-server via an intermediary web service. + +# Expected inputs + +## Import document or import data format + +A list of domain names (fronted domains) and host headers (meek-server) of the +intermediary web service. + +## Semantics + +The input document may contain a domain name and a host header combination per +line separated by colon in the format: + + DomainName:HostHeader + +Example: + + www.google.com:meek-reflect.appspot.com + a0.awsstatic.com:d2zfqthxsdq309.cloudfront.net + +# Test description + +Performs a HTTP GET request over TLS (HTTPS) to a list of fronted domains with +the Host Header of the "inside" meek-server. For diagnostic purposes the +meek-server handles a GET request and respond with: "I’m just a happy little +web server.\n". The GET request is sent over TLS to the root of the fronted +domain with the Host Header option of the desired meek-server host. + + +# Expected output + +## Parent data format + +df-001-httpt-000 + +## Required output data + +* The domain name and host header used in the measurement + (DomainName:HostHeader) + +* The requests that have been made + +* The received responses + +* If the meek server is blocked or unreachable + +## Semantics + +success: + **boolean** indicates if an HTTPS GET response to the meek server is + successfull + +## Possible conclusions + +If the fronted request/response to the meek server is successful. + +## Example output sample + +``` +agent: agent +input: ajax.aspnetcdn.com:az668014.vo.msecnd.net +requests: +- request: + body: null + headers: + - - Host + - [az668014.vo.msecnd.net] + method: GET + tor: {is_tor: false} + url: https://ajax.aspnetcdn.com + response: + body: "I\u2019m just a happy little web server.\n" + code: 200 + headers: + - - Content-Length + - ['38'] + - - X-Cache + - [HIT] + - - X-Powered-By + - [ASP.NET] + - - Accept-Ranges + - [bytes] + - - Server + - [ECAcc (fcn/40C4)] + - - Last-Modified + - ['Wed, 01 Apr 2015 09:25:13 GMT'] + - - Connection + - [close] + - - Date + - ['Wed, 01 Apr 2015 10:01:37 GMT'] + - - Content-Type + - [text/plain; charset=utf-8] +socksproxy: null +success: true +``` diff --git a/test-specs/ts-013-meek_fronted_requests.md b/test-specs/ts-013-meek_fronted_requests.md deleted file mode 100644 index f78066e..0000000 --- a/test-specs/ts-013-meek_fronted_requests.md +++ /dev/null @@ -1,111 +0,0 @@ -# Specification version number - -2015-04-01-000 - -# Specification name - -Meek Fronted Request Test - -# Test preconditions - -* An internet connection - -# Expected impact - -Ability to detect uncensored fronted domains that can transmit requests to the -"inside" meek-server via an intermediary web service. - -# Expected inputs - -## Import document or import data format - -A list of domain names (fronted domains) and host headers (meek-server) of the -intermediary web service. - -## Semantics - -The input document may contain a domain name and a host header combination per -line separated by colon in the format: - - DomainName:HostHeader - -Example: - - www.google.com:meek-reflect.appspot.com - a0.awsstatic.com:d2zfqthxsdq309.cloudfront.net - -# Test description - -Performs a HTTP GET request over TLS (HTTPS) to a list of fronted domains with -the Host Header of the "inside" meek-server. For diagnostic purposes the -meek-server handles a GET request and respond with: "I’m just a happy little -web server.\n". The GET request is sent over TLS to the root of the fronted -domain with the Host Header option of the desired meek-server host. - - -# Expected output - -## Parent data format - -df-001-httpt-000 - -## Required output data - -* The domain name and host header used in the measurement - (DomainName:HostHeader) - -* The requests that have been made - -* The received responses - -* If the meek server is blocked or unreachable - -## Semantics - -success: - **boolean** indicates if an HTTPS GET response to the meek server is - successfull - -## Possible conclusions - -If the fronted request/response to the meek server is successful. - -## Example output sample - -``` -agent: agent -input: ajax.aspnetcdn.com:az668014.vo.msecnd.net -requests: -- request: - body: null - headers: - - - Host - - [az668014.vo.msecnd.net] - method: GET - tor: {is_tor: false} - url: https://ajax.aspnetcdn.com - response: - body: "I\u2019m just a happy little web server.\n" - code: 200 - headers: - - - Content-Length - - ['38'] - - - X-Cache - - [HIT] - - - X-Powered-By - - [ASP.NET] - - - Accept-Ranges - - [bytes] - - - Server - - [ECAcc (fcn/40C4)] - - - Last-Modified - - ['Wed, 01 Apr 2015 09:25:13 GMT'] - - - Connection - - [close] - - - Date - - ['Wed, 01 Apr 2015 10:01:37 GMT'] - - - Content-Type - - [text/plain; charset=utf-8] -socksproxy: null -success: true -```

1 0