August 2014 - tor-commits - lists.torproject.org

[spec/master] Merge pull request #28 from TheTorProject/feature/bridge_reachability_spec
by art＠torproject.org 19 Aug '14

19 Aug '14

commit 1c8b72ba95b25861b9ffb93927707732e78dafba Merge: 515855a f443707 Author: aagbsn <aagbsn(a)extc.org> Date: Wed Aug 13 16:57:55 2014 +0000 Merge pull request #28 from TheTorProject/feature/bridge_reachability_spec Add specification for bridge reachability test test-specs/ts-011-bridge-reachability.md | 224 ++++++++++++++++++++++++++++++ 1 file changed, 224 insertions(+)

1 0

[spec/master] Add specification for bridge reachability test
by art＠torproject.org 19 Aug '14

19 Aug '14

commit f443707f3193c63003a0c32b75c560ec1fd2235d Author: Arturo Filastò <art(a)fuffa.org> Date: Thu Aug 7 16:15:02 2014 +0200 Add specification for bridge reachability test --- test-specs/ts-011-bridge-reachability.md | 224 ++++++++++++++++++++++++++++++ 1 file changed, 224 insertions(+) diff --git a/test-specs/ts-011-bridge-reachability.md b/test-specs/ts-011-bridge-reachability.md new file mode 100644 index 0000000..4e94907 --- /dev/null +++ b/test-specs/ts-011-bridge-reachability.md @@ -0,0 +1,224 @@ +# Specification version number + +2014-08-06-001 + +# Specification name + +Bridge Reachability Test + +# Test preconditions + +* An internet connection + +# Expected impact + +Ability to detect which bridges are working from the given network vantage +point. + +# Expected inputs + +A list of bridge IP addressed, port numbers and pluggable transport type (when +applicable). + +## Semantics + +They should be in a flat text file one per line in the format: + + IP:ORPort + +Example: + + 203.0.113.27:1828 + 198.51.100.45:9045 + +For normal Tor bridges. + +For pluggable transport bridges the format is: + + TransportType IP:ORPort + +Example: + + obfs2 203.0.113.27:1828 + obfs3 198.51.100.45:9045 + + +# Test description + +For every bridge specified as input a new tor instance is started and it is +instructed to connect to Tor using that bridge. It will try to bootstrap to +100% and either succeed or timeout after a default time of 120 seconds. + +# Expected output + +## Parent data format + +None. + +## Required output data + +* The bridge address used in the measurement (bridge_address). + +* Whether or not it succeeded in connecting to the Tor networking using the + specified bridge (success). + +* The bootstrap percentage at which the test timed out (tor_progress). + +* The name of the transport used (transport_name). + +## Semantics + +error: + **string** indicating that an error occurred when running the test on the + given input. + + unsupported-tor-version: to indicate that the version of tor is too old to + connect to the specified bridge. + + missing-pyobfsproxy: to indicate that obfsproxy is not installed. + + missing-fteproxy: to indicate that fteproxy is not installed. + + null: no error was detected. + +success: + **boolean** indiciating if whether or not we were able to establish a + connection to the Tor network with the supplied bridge. + + True: indicates that we reached 100% bootstrap + + False: indicates that we were unable to reach 100% bootstrap before the + timeout was reached. + + null: indicates that tor exited before we were able to complete the test, + this can be due to some unhandled error. + + +timeout: + **integer** indicating the timeout, in seconds, after which we should + consider the bridge non-functioning. By default it is set to 120 seconds. + +transport_name: + **string** indicating the name of transport name used by the bridge. + + vanilla: a regular, non pluggable transport, bridge. + + obfs2: for [obfs2 obfsproxy](https://gitweb.torproject.org/pluggable-transports/obfsproxy.git…. + + obfs3: for [obfs3 obfsproxy](https://gitweb.torproject.org/pluggable-transports/obfsproxy.git…. + + fte: for [fte-proxy](https://fteproxy.org/) + + scramblesuit: for [ScrambleSuit](http://www.cs.kau.se/philwint/scramblesuit/) + +tor_version: + **string** indicating the version string of the tor client being used. + +tor_progress: + **integer** indicating the percentage of tor bootstrapping at which we + stopped. + +tor_progress_tag: + **string** a string giving a textual description of what the progress + percentage means. + +tor_progress_summary: + **string** a more verbose string giving a textual description of what the + progress percentage state is. + +tor_log: + **string** notice level log output of tor. + +bridge_address: + **string** the address of the bridge being used. + +## Possible conclusions + +Which bridges are functioning from the given network vantage point and which +ones are not. + +## Example output sample + +``` +########################################### +# OONI Probe Report for bridge_reachability (0.1) +# Thu Aug 7 16:10:50 2014 +########################################### +--- +input_hashes: [595c8ad9b63ff9f142eca6296021990df6367ab03b3c4eb1c69a8747f0cd41a1] +options: [-f, bridges.txt] +probe_asn: AS3269 +probe_cc: IT +probe_city: null +probe_ip: 127.0.0.1 +software_name: ooniprobe +software_version: 1.0.2 +start_time: 1407420650.560137 +test_name: bridge_reachability +test_version: '0.1' +... +--- +{bridge_address: '169.229.59.74:31493', error: null, input: 'obfs3 169.229.59.74:31493 + AF9F66B7B04F8FF6F32D455F05135250A16543C9', success: true, timeout: 120, tor_log: null, + tor_progress: 100, tor_progress_summary: Done, tor_progress_tag: done, tor_version: 0.2.4.20, + transport_name: obfs3} +... +--- +{bridge_address: '169.229.59.75:46328', error: null, input: 'obfs3 169.229.59.75:46328 + AF9F66B7B04F8FF6F32D455F05135250A16543C9', success: true, timeout: 120, tor_log: null, + tor_progress: 100, tor_progress_summary: Done, tor_progress_tag: done, tor_version: 0.2.4.20, + transport_name: obfs3} +... +--- +{bridge_address: '208.79.90.242:35658', error: null, input: 'obfs3 208.79.90.242:35658 + BA61757846841D64A83EA2514C766CB92F1FB41F', success: true, timeout: 120, tor_log: null, + tor_progress: 100, tor_progress_summary: Done, tor_progress_tag: done, tor_version: 0.2.4.20, + transport_name: obfs3} +... +--- +{bridge_address: '83.212.100.216:47870', error: null, input: 'obfs2 83.212.100.216:47870 + 1F01A7BB60F49FC96E0850A6BAD6D076DFEFAF80', success: false, timeout: 120, tor_log: '', + tor_progress: 0, tor_progress_summary: null, tor_progress_tag: null, tor_version: 0.2.4.20, + transport_name: obfs2} +... +--- +{bridge_address: '83.212.96.182:46602', error: null, input: 'obfs2 83.212.96.182:46602 + 6F058CBEF888EB20D1DEB9886909F1E812245D41', success: false, timeout: 120, tor_log: '', + tor_progress: 0, tor_progress_summary: null, tor_progress_tag: null, tor_version: 0.2.4.20, + transport_name: obfs2} +... +--- +{bridge_address: '70.182.182.109:54542', error: null, input: 'obfs2 70.182.182.109:54542 + 94C9E691688FAFDEC701A0788BD15BE8AD34ED35', success: false, timeout: 120, tor_log: '', + tor_progress: 0, tor_progress_summary: null, tor_progress_tag: null, tor_version: 0.2.4.20, + transport_name: obfs2} +... +--- +{bridge_address: '128.31.0.34:1051', error: null, input: 'obfs2 128.31.0.34:1051 CA7434F14A898C7D3427B8295A7F83446BC7F496', + success: false, timeout: 120, tor_log: '', tor_progress: 0, tor_progress_summary: null, + tor_progress_tag: null, tor_version: 0.2.4.20, transport_name: obfs2} +... +--- +{bridge_address: '83.212.101.2:45235', error: null, input: 'obfs2 83.212.101.2:45235 + 2ADFE7AA8D272C520D1FBFBF4E413F3A1B26313D', success: false, timeout: 120, tor_log: '', + tor_progress: 0, tor_progress_summary: null, tor_progress_tag: null, tor_version: 0.2.4.20, + transport_name: obfs2} +... +--- +{bridge_address: '83.212.101.2:42782', error: null, input: 'obfs3 83.212.101.2:42782 + 2ADFE7AA8D272C520D1FBFBF4E413F3A1B26313D', success: false, timeout: 120, tor_log: '', + tor_progress: 0, tor_progress_summary: null, tor_progress_tag: null, tor_version: 0.2.4.20, + transport_name: obfs3} +... +--- +{bridge_address: '83.212.101.2:443', error: null, input: 'obfs3 83.212.101.2:443 2ADFE7AA8D272C520D1FBFBF4E413F3A1B26313D', + success: false, timeout: 120, tor_log: '', tor_progress: 0, tor_progress_summary: null, + tor_progress_tag: null, tor_version: 0.2.4.20, transport_name: obfs3} +... +--- +{bridge_address: '209.141.36.236:45496', error: null, input: 'obfs3 209.141.36.236:45496 + 58D91C3A631F910F32E18A55441D5A0463BA66E2', success: false, timeout: 120, tor_log: '', + tor_progress: 0, tor_progress_summary: null, tor_progress_tag: null, tor_version: 0.2.4.20, + transport_name: obfs3} +... +```

1 0

[spec/master] Improve the document detailing our release procedure.
by art＠torproject.org 19 Aug '14

19 Aug '14

commit 604e5f8d64b4bb43c8fe7081ed428ef12927e775 Author: Arturo Filastò <art(a)fuffa.org> Date: Mon Aug 18 18:32:11 2014 +0200 Improve the document detailing our release procedure. --- Release-Procedure.md | 154 ++++++++++++++++++++++++++++++++++++++++++++++---- 1 file changed, 144 insertions(+), 10 deletions(-) diff --git a/Release-Procedure.md b/Release-Procedure.md index a928052..b33b6b9 100644 --- a/Release-Procedure.md +++ b/Release-Procedure.md @@ -24,24 +24,158 @@ they will be fit to run on their platform. Code coverage is measured via coveralls and the results for such tests can be found [here](https://coveralls.io/r/TheTorProject/ooni-probe). +Before a release is fit for release it must pass the automated and manual tests +listed below: + +## Unit tests + +``` +sudo trial ooni +``` + +Must produce no errors. + +## Build source distribution and install + +It should be possible to build a source distribution of ooniprobe. Install it +inside of a fresh virtuale environment and then run all the manual tests +needed. + +``` +rmvirtualenv ooni-testsdist; mkvirtualenv ooni-testsdist +pip install dist/ooniprobe-$OONIPROBE_VERSION.tar.gz +``` + +## Usage tests + +First cd into a temporary directory so that you don't mess up the code tree and +you import what you have just installed. + +``` +cd `mktemp -d 2>/dev/null || mktemp -d -t 'ooni'` +``` + +### Running decks + +``` +ooniprobe -i ~/.virtualenvs/ooni-testsdist/share/ooni/decks/mlab.deck +ooniprobe -i ~/.virtualenvs/ooni-testsdist/share/ooni/decks/fast_no_root.deck +ooniprobe -i ~/.virtualenvs/ooni-testsdist/share/ooni/decks/complete_no_root.deck +``` + +While this is running you may as well open another shell, get into that +virtualenv and make sure oonireport is working as it should. + +Check to see if by running: + +``` +oonireport status +``` + +You see a list of "Reports in progress". There should be at least 4 of them. + +The above commands should not throw any exceptions and you should now see +inside of your current working directory the following list of files: + +``` +report-dns_consistency-2014-08-18T134642Z.yamloo +report-http_header_field_manipulation-2014-08-18T134555Z.yamloo +report-http_header_field_manipulation-2014-08-18T134625Z.yamloo +report-http_header_field_manipulation-2014-08-18T134642Z.yamloo +report-http_host-2014-08-18T134642Z.yamloo +report-http_invalid_request_line-2014-08-18T134625Z.yamloo +report-http_invalid_request_line-2014-08-18T134642Z.yamloo +report-http_requests-2014-08-18T134642Z.yamloo +``` + +### Running single tests + +The following command should produce the list of tests installed on your +system: + +``` +ooniprobe -s +``` + +The following commands should all succeed and produce some measurements: + +``` +ooniprobe -n manipulation/http_header_field_manipulation +ooniprobe -n blocking/http_requests -f httpo://ihiderha53f36lsd.onion/input/37e60e13536f6afe47a830bfb6b371b5cf65da66d7ad65137344679b24fdccd1 +``` + +### Testing vagrant installation + +To setup and test ooniprobe on a fresh ubuntu installation with +[Vagrant](https://www.vagrantup.com): + +``` +vagrant destroy -f +vagrant up +vagrant ssh +``` + +You should now be logged into the vagrant machine with ooniprobe installed. + +To make sure everything went as expected try running: + +``` +sudo ooniprobe -i /usr/share/ooni/decks/fast.deck +``` + # Tagging and Signing of a release +To create a new release you should update the following files with the new +version information: + + * ooni/__init__.py: set __version__ = "$OONI_VERSION" + + * Changelog.md: to include a new changelog entry with the introduced + changes. + + * debian/changelog: with a new entry for the new release + +Once this is done you should commit the changes with a message like this: + +``` +git commit -a -m 'update ooniprobe to $OONI_VERSION' +``` + +You can now tag and sign a new release with: + +``` +git tag -s -a 'v$OONI_VERSION' +``` + +Then write a reasonable message inside of the tag message body. The format to +follow should be: + +``` +ooniprobe $OONI_VERSION + +$SOME_DESCRIPTION_OR_CHANGELOG_OR_WHATEVER +``` + All tags should be incremental and they will be signed with either of the following GPG keys: ``` -pub 4096R/150FE210 2011-10-23 [expires: 2014-05-04] -Key fingerprint = 46E5 EF37 DE26 4EA6 8DCF 53EA E3A2 1297 150F E210 -uid Arturo Filastò <art(a)torproject.org> -uid Arturo Filastò <arturo.filasto(a)logioshermes.org> -uid Arturo Filastò <arturo(a)filasto.net> -uid Arturo Filastò <art(a)fuffa.org> -uid Arturo Filastò <art(a)globaleaks.org> -uid Arturo Filastò <art(a)baculo.org> -uid Arturo Filastò <hellais(a)gmail.com> -sub 4096R/F423B27C 2011-10-23 [expires: 2014-05-04] +pub 4096R/150FE210 2011-10-23 [expires: 2014-11-24] + Key fingerprint = 46E5 EF37 DE26 4EA6 8DCF 53EA E3A2 1297 150F E210 + uid Arturo Filastò <art(a)torproject.org> + uid Arturo Filastò <arturo(a)filasto.net> + uid Arturo Filastò <arturo.filasto(a)logioshermes.org> + uid Arturo Filastò <art(a)fuffa.org> + uid Arturo Filastò <art(a)globaleaks.org> + uid Arturo Filastò <art(a)baculo.org> + uid Arturo Filastò <hellais(a)gmail.com> + sub 4096R/F423B27C 2011-10-23 [expires: 2014-11-24] ``` +## Versioning scheme + +All OONI tools follow [semantic versioning](http://semver.org/). + # Build systems & environments We currently only support packages for debian and ubuntu based systems.

1 0

[spec/master] Merge branch 'master' of github.com:TheTorProject/ooni-spec
by art＠torproject.org 19 Aug '14

19 Aug '14

commit b1deb3f6d97d44cef61d2174693fc2204a9c1b26 Merge: 604e5f8 1c8b72b Author: Arturo Filastò <art(a)fuffa.org> Date: Mon Aug 18 18:32:51 2014 +0200 Merge branch 'master' of github.com:TheTorProject/ooni-spec * 'master' of github.com:TheTorProject/ooni-spec: Add specification for bridge reachability test test-specs/ts-011-bridge-reachability.md | 224 ++++++++++++++++++++++++++++++ 1 file changed, 224 insertions(+)

1 0

[spec/master] Add obfsproxy_version key to the bridge reahcability spec.
by art＠torproject.org 19 Aug '14

19 Aug '14

commit 1378c1f7a67742a8ddedbad6e368a4387a18e751 Author: Arturo Filastò <art(a)fuffa.org> Date: Tue Aug 19 13:49:43 2014 +0200 Add obfsproxy_version key to the bridge reahcability spec. --- test-specs/ts-011-bridge-reachability.md | 3 +++ 1 file changed, 3 insertions(+) diff --git a/test-specs/ts-011-bridge-reachability.md b/test-specs/ts-011-bridge-reachability.md index 4e94907..194950a 100644 --- a/test-specs/ts-011-bridge-reachability.md +++ b/test-specs/ts-011-bridge-reachability.md @@ -132,6 +132,9 @@ tor_log: bridge_address: **string** the address of the bridge being used. +obfsproxy_version (since 0.1.1): + **string** indicating the version string of the obfsproxy client being used. + ## Possible conclusions Which bridges are functioning from the given network vantage point and which

1 0

[spec/master] Add command for making source dist.
by art＠torproject.org 19 Aug '14

19 Aug '14

commit 7a44e941287b2473398efadb1ab3399a3ef19551 Author: Arturo Filastò <art(a)fuffa.org> Date: Tue Aug 19 13:50:01 2014 +0200 Add command for making source dist. --- Release-Procedure.md | 1 + 1 file changed, 1 insertion(+) diff --git a/Release-Procedure.md b/Release-Procedure.md index b33b6b9..6910da9 100644 --- a/Release-Procedure.md +++ b/Release-Procedure.md @@ -42,6 +42,7 @@ inside of a fresh virtuale environment and then run all the manual tests needed. ``` +python setup.py sdist rmvirtualenv ooni-testsdist; mkvirtualenv ooni-testsdist pip install dist/ooniprobe-$OONIPROBE_VERSION.tar.gz ```

1 0

[spec/master] Add instructions on uploading a new ooniprobe version to pypi
by art＠torproject.org 19 Aug '14

19 Aug '14

commit abd539ebfb175df388327f9a388c2f3df8805d62 Author: Arturo Filastò <art(a)fuffa.org> Date: Tue Aug 19 15:36:21 2014 +0200 Add instructions on uploading a new ooniprobe version to pypi --- Release-Procedure.md | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/Release-Procedure.md b/Release-Procedure.md index 6910da9..89afb5f 100644 --- a/Release-Procedure.md +++ b/Release-Procedure.md @@ -177,6 +177,14 @@ pub 4096R/150FE210 2011-10-23 [expires: 2014-11-24] All OONI tools follow [semantic versioning](http://semver.org/). +## Updating packages + +To upload a new version to pypi run: + +``` +python setup.py sdist upload -s +``` + # Build systems & environments We currently only support packages for debian and ubuntu based systems.

1 0

[ooni-probe/master] update ooniprobe to 1.1.0
by art＠torproject.org 19 Aug '14

19 Aug '14

commit 3283a34cc487303cad2035f0257e88da04a38918 Author: Arturo Filastò <art(a)fuffa.org> Date: Tue Aug 19 15:32:56 2014 +0200 update ooniprobe to 1.1.0 --- debian/changelog | 6 ++++++ ooni/__init__.py | 4 ++-- 2 files changed, 8 insertions(+), 2 deletions(-) diff --git a/debian/changelog b/debian/changelog index af347a7..90ec33a 100644 --- a/debian/changelog +++ b/debian/changelog @@ -1,3 +1,9 @@ +ooniprobe (1.1.0-1) stable; urgency=low + + * ooniprobe 1.1.0 + + -- OONI Developers (signing key) <ooni-dev(a)lists.torproject.org> Tue, 19 Aug 2014 15:14:46 +0200 + ooniprobe (1.0.2-1) stable; urgency=low * ooniprobe 1.0.2 diff --git a/ooni/__init__.py b/ooni/__init__.py index c3c8050..821b197 100644 --- a/ooni/__init__.py +++ b/ooni/__init__.py @@ -1,8 +1,8 @@ # -*- encoding: utf-8 -*- __author__ = "Open Observatory of Network Interference" -__version__ = "1.1.0-rc1" +__version__ = "1.1.0" __all__ = ['config', 'inputunit', 'kit', - 'lib', 'nettest', 'oonicli', 'reporter', + 'lib', 'nettest', 'oonicli', 'report', 'reporter', 'templates', 'utils']

1 0

[ooni-probe/master] Update Vagrantfile to install ooniprobe from /ooni
by art＠torproject.org 19 Aug '14

19 Aug '14

commit dd9f96136b0e1f899e4c6254fe025b369a78b4af Author: Arturo Filastò <art(a)fuffa.org> Date: Tue Aug 19 15:23:28 2014 +0200 Update Vagrantfile to install ooniprobe from /ooni --- Vagrantfile | 11 +++-------- 1 file changed, 3 insertions(+), 8 deletions(-) diff --git a/Vagrantfile b/Vagrantfile index 74fe50f..7f20f5e 100644 --- a/Vagrantfile +++ b/Vagrantfile @@ -16,13 +16,8 @@ end $script = <<SCRIPT -echo "deb http://deb.ooni.nu/ooni wheezy main" >> /etc/apt/sources.list echo "deb http://deb.torproject.org/torproject.org precise main" >> /etc/apt/sources.list -# Install deb.ooni.nu key -gpg --keyserver pgp.mit.edu --recv-key 0x49B8CDF4 -gpg --export 89AB86D4788F3785FE9EDA31F9E2D9B049B8CDF4 | apt-key add - - # Install deb.torproject.org key gpg --keyserver keys.gnupg.net --recv 886DDD89 gpg --export A3C4F0F979CAA22CDBA8F512EE8CBC9E886DDD89 | apt-key add - @@ -30,11 +25,11 @@ gpg --export A3C4F0F979CAA22CDBA8F512EE8CBC9E886DDD89 | apt-key add - apt-get update apt-get install -y tor deb.torproject.org-keyring -apt-get install -y ooniprobe geoip-database-contrib # Setup for sniffer subsystem -apt-get install -y build-essential libdnet-dev libpcap-dev python-dev python-pip python-networkx -pip install PyDNET graphillion +apt-get install -y build-essential libdumbnet-dev python-dumbnet python-pypcap libpcap-dev python-dev python-pip libgeoip-dev +cd /ooni +python setup.py install echo "Login using 'vagrant ssh', and dont forget to run ooniprobe as root." echo "First run: 'sudo ooniprobe -i /usr/share/ooni/decks/fast.deck'"

1 0

[ooni-probe/master] Also install libffi-dev
by art＠torproject.org 19 Aug '14

19 Aug '14

commit f16457e07070b92103b339f9e367c96e36801d02 Author: Arturo Filastò <art(a)fuffa.org> Date: Tue Aug 19 15:25:46 2014 +0200 Also install libffi-dev --- Vagrantfile | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/Vagrantfile b/Vagrantfile index 7f20f5e..bd9d5db 100644 --- a/Vagrantfile +++ b/Vagrantfile @@ -27,7 +27,7 @@ apt-get update apt-get install -y tor deb.torproject.org-keyring # Setup for sniffer subsystem -apt-get install -y build-essential libdumbnet-dev python-dumbnet python-pypcap libpcap-dev python-dev python-pip libgeoip-dev +apt-get install -y build-essential libdumbnet-dev python-dumbnet python-pypcap libpcap-dev python-dev python-pip libgeoip-dev libffi-dev cd /ooni python setup.py install echo "Login using 'vagrant ssh', and dont forget to run ooniprobe as root."

1 0