September 2017 - tor-commits - lists.torproject.org

[ooni-probe/master] Make the geoip lookup more robust
by art＠torproject.org 22 Sep '17

22 Sep '17

commit 2a9e471250cebcbe6422a79e293d03d4d7b3d195 Author: Arturo Filastò <arturo(a)filasto.net> Date: Fri Apr 28 16:01:45 2017 +0000 Make the geoip lookup more robust related to #751 --- ooni/common/http_utils.py | 2 ++ ooni/geoip.py | 2 +- 2 files changed, 3 insertions(+), 1 deletion(-) diff --git a/ooni/common/http_utils.py b/ooni/common/http_utils.py index b44f5c0b..f54341c1 100644 --- a/ooni/common/http_utils.py +++ b/ooni/common/http_utils.py @@ -5,6 +5,8 @@ from base64 import b64encode def representBody(body): if not body: return body + if isinstance(body, unicode): + return body try: body = unicode(body, 'utf-8') except UnicodeDecodeError: diff --git a/ooni/geoip.py b/ooni/geoip.py index eda814e3..bb983eca 100644 --- a/ooni/geoip.py +++ b/ooni/geoip.py @@ -264,7 +264,7 @@ class ProbeIP(object): @d.addCallback def cb(result): self.strategy = 'tor_get_info_address' - self.address = result.values()[0] + self.address = str(result.values()[0]) return d else: raise errors.TorStateNotFound

1 0

[ooni-probe/master] Improve doc string for no-geoip option #686 / Documentation Update April 2017 #749 (#775)
by art＠torproject.org 22 Sep '17

22 Sep '17

commit b3a6b393c4b7cdb916aad544bc4ed20129d6ebf3 Author: Anatol <anatol(a)riseup.net> Date: Mon Jul 17 09:51:23 2017 +0200 Improve doc string for no-geoip option #686 / Documentation Update April 2017 #749 (#775) * Improved doc string for no-geoip option (#686) * Improved doc string for no-geoip option (#686) * Documentation Update April 2017 #749 Replace all contents of ./docs/source/index.rst except the footer with contents from ./README.rst * Documentation Update April 2017 #749 Added a line on how to start the web interface --- README.rst | 4 +- docs/source/index.rst | 452 +------------------------------------------------- ooni/ui/cli.py | 4 +- 3 files changed, 6 insertions(+), 454 deletions(-) diff --git a/README.rst b/README.rst index 2b3806e3..55e1f699 100644 --- a/README.rst +++ b/README.rst @@ -150,7 +150,9 @@ Configuring ooniprobe --------------------- After successfully installing ooniprobe you should be able to access the web UI -on your host machine at <http://localhost:8842/>. +on your host machine at <http://localhost:8842/> after running:: + ooniprobe -w +or starting the daemon. You should now be presented with the web UI setup wizard where you can read the risks involved with running ooniprobe. Upon answering the quiz correctly you can diff --git a/docs/source/index.rst b/docs/source/index.rst index d7574ac1..2e8e9c63 100644 --- a/docs/source/index.rst +++ b/docs/source/index.rst @@ -1,454 +1,4 @@ -.. OONI documentation master file. - -OONI: Open Observatory of Network Interference -============================================== - -The Open Observatory of Network Interference (OONI) is a global observation -network which collects high quality data, using open methodologies and Free and Open Source Software (FL/OSS), to share observations about the various types, methods, and amounts of network tampering in the world. - - - "The Net interprets censorship as damage and routes around it." - - John Gilmore; TIME magazine (6 December 1993) - -Are you interested in testing your network for signs of surveillance and censorship? Do you want to collect data so that you and others can better understand your network? If so, please read this document and we hope that ooniprobe will help you gather network data that will assist you with your endeavors! - -ooniprobe ---------- - -ooniprobe is a program that users can run to probe their network and to collect network measurement data. ooniprobe also allows users to choose what test should be run and what backend should be used for storing the test report and for assisting them in running the test. - -ooniprobe tests are divided into two categories: **Traffic Manipulation** and -**Content Blocking**. - -**Traffic Manipulation** tests aim to detect the presence of some sort of -tampering with the internet traffic between the probe and a remote test helper -backend. As such they usually require the selection of a oonib backend -component for running the test. - -**Content Blocking** tests aim to enumerate the kind of content that is -blocked from the probes network point of view. As such they usually require the specification of an input list for running the test. - -Read this before running ooniprobe! ------------------------------------ - -Running ooniprobe can potentially be a risky activity. This largely depends on the laws and jurisdiction of the country you are running ooniprobe from, as well as on the type of test you are running. Technically, it is possible for a person observing your internet connection to be aware of the fact that you are running ooniprobe. This means that if running network measurement tests is something that is considered to be illegal in your country then you could be spotted. - -Furthermore, ooniprobe takes no precautions to protect the install target machine from forensics analysis. If the fact that you have installed or used ooniprobe is a liability for you, please be aware of this risk. - -Using ooniprobe -=============== - -**Net test** is a set of measurements to assess what kind of internet censorship is occurring. - -**Decks** are collections of ooniprobe nettests with some associated inputs. - -**Collector** is a service used to report the results of measurements. - -**Test helper** is a service used by a probe for successfully performing its measurements. - -**Bouncer** is a service used to discover the addresses of test helpers and collectors. - -OONI in 5 minutes -================= - -On debian testing or unstable:: - - sudo apt-get install ooniprobe - -If you are running debian stable you can get it from backports via:: - - sudo sh -c 'echo "deb http://http.debian.net/debian wheezy-backports main" >> /etc/apt/sources.list' - sudo apt-get update && sudo apt-get install ooniprobe - -On unix systems:: - - sudo pip install ooniprobe - -To install it from the current master run:: - - sudo pip install https://github.com/TheTorProject/ooni-probe/archive/master.zip - -Then run:: - - mkdir my_decks - sudo ooniresources --update-inputs --update-geoip - oonideckgen -o my_decks/ - -**BUG** Note: -ooniprobe version 1.2.2 when installed from the debian repository will not -properly create the ooni home folder and if you run into an error in accessing -`~/.ooni/` run:: - - ooniprobe -n blocking/http_requests -u http://google.com/ - -This should generate the home and allow you to run oonideckgen. - -The output from the last command will tell you how to run ooniprobe to perform -the measurement. - -If you would like to contribute measurements to OONI daily you can also add this to your crontab:: - - @daily ooniprobe $THE_OONI_COMMAND - -Run this command to automatically update your crontab:: - - (crontab -l 2>/dev/null; echo "@daily ooniprobe $THE_OONI_COMMAND") | crontab - - -Installation -============ - -Debian based systems --------------------- - -If you are running Debian testing or Debian unstable you can install ooniprobe -simply with:: - - apt-get install ooniprobe - -If you are running Debian stable you can get it from backports via:: - - sudo sh -c 'echo "deb http://http.debian.net/debian wheezy-backports main" >> /etc/apt/sources.list' - sudo apt-get update && sudo apt-get install ooniprobe - -If you are running Ubuntu 14.04 LTS you can install it from the PPA -(https://launchpad.net/~irl/+archive/ubuntu/ooni/):: - - sudo add-apt-repository ppa:irl/ooni - sudo apt-get update && sudo apt-get install ooniprobe - -You will be warned that the packages are unauthenticated. This is due to the -PPA not being signed and is normal behaviour. If you would prefer to verify the -integrity of the package, use our private Debian repository below. - -Mac OS X --------- - -You can install ooniprobe on OSX if you have installed homebrew (http://mxcl.github.io/homebrew) with:: - - brew install ooniprobe - -Unix systems (with pip) ------------------------ - -Make sure you have installed the following depedencies: - - * build-essential - * python (>=2.7) - * python-dev - * pip - * libgeoip-dev - * libdumbnet-dev - * libpcap-dev - * libssl-dev - * libffi-dev - * tor (>=0.2.5.1 to run all the tor related tests) - -Then you should be able to install ooniprobe by running:: - - sudo pip install ooniprobe - -Configuring ooniprobe ---------------------- - -You may edit the configuration for ooniprobe by editing the configuration file -found inside of ``~/.ooni/ooniprobe.conf``. - -By default ooniprobe will not include personal identifying information in the -test result, nor create a pcap file. This behavior can be personalized. - - -Updating resources ------------------- - -To generate decks you will have to update the input resources of ooniprobe. - -This can be done with:: - - ooniresources --update-inputs - -If you get a permission error, you might have to run the command as root or -change the ooniprobe data directory inside of `ooniprobe.conf`. - -On some platforms, such as debian contrib, you will not get all the geoip -related files needed. In that case it is possible to manually download them -with ``ooniresources``:: - - ooniresources --update-geoip - -Generating decks ----------------- - -You can generate decks for your country with the oonideckgen command. - -If you wish, for example, to generate a deck to be run in the country of Italy, -you can do so (be sure to have updated the input resources first) by running:: - - oonideckgen --country-code IT --output ~/ - -You will now have in your home a folder called `deck-it`, containing the ooni -deck (ends with .deck) and the inputs. - -**Note:** You should not move the `deck-*` directory once it has been -generated as the paths to the inputs referenced by the test in the deck are -absolute. If you want your deck to live in another directory you must -regenerated it. - - -Running decks -------------- - -You will find all the installed decks inside of ``/usr/share/ooni/decks``. - -You may then run a deck by using the command line option ``-i``: - -As root:: - - ooniprobe -i /usr/share/ooni/decks/mlab.deck - - -Or as a user:: - - ooniprobe -i /usr/share/ooni/decks/mlab_no_root.deck - - -Or: - -As root:: - - ooniprobe -i /usr/share/ooni/decks/complete.deck - - -Or as a user:: - - ooniprobe -i /usr/share/ooni/decks/complete_no_root.deck - - -The above tests will require around 20-30 minutes to complete depending on your network speed. - -If you would prefer to run some faster tests you should run: -As root:: - - ooniprobe -i /usr/share/ooni/decks/fast.deck - - -Or as a user:: - - ooniprobe -i /usr/share/ooni/decks/fast_no_root.deck - - -Running net tests ------------------ - -You may list all the installed stable net tests with:: - - - ooniprobe -s - - -You may then run a nettest by specifying its name for example:: - - - ooniprobe manipulation/http_header_field_manipulation - - -It is also possible to specify inputs to tests as URLs:: - - - ooniprobe blocking/http_requests -f httpo://ihiderha53f36lsd.onion/input/37e60e13536f6afe47a830bfb6b371b5cf65da66d7ad65137344679b24fdccd1 - - -You can find the result of the test in your current working directory. - -By default the report result will be collected by the default ooni collector -and the addresses of test helpers will be obtained from the default bouncer. - -You may also specify your own collector or bouncer with the options ``-c`` and -``-b``. - - -Bridges and obfsproxy bridges -============================= - -ooniprobe submits reports to oonib report collectors through Tor to a hidden -service endpoint. By default, ooniprobe uses the installed system Tor, but can -also be configured to launch Tor (see the advanced.start_tor option in -ooniprobe.conf), and ooniprobe supports bridges (and obfsproxy bridges, if -obfsproxy is installed). The tor.bridges option in ooniprobe.conf sets the path -to a file that should contain a set of "bridge" lines (of the same format as -used in torrc, and as returned by https://bridges.torproject.org) If obfsproxy -bridges are to be used, the path to the obfsproxy binary must be configured. - -See option advanced.obfsproxy_binary, in ooniprobe.conf. - -(Optional) Install obfsproxy ----------------------------- - -Install the latest version of obfsproxy for your platform. - -Download Obfsproxy: https://www.torproject.org/projects/obfsproxy.html.en - -Setting capabilities on your virtualenv python binary -===================================================== - -If your distributation supports capabilities you can avoid needing to run OONI as root:: - - - setcap cap_net_admin,cap_net_raw+eip /path/to/your/virtualenv's/python - - -Reporting bugs -============== - -You can report bugs and issues you find with ooni-probe on The Tor Projec issue -tracker filing them under the "Ooni" component: https://trac.torproject.org/projects/tor/newticket?component=Ooni. - -You can either register an account or use the group account "cypherpunks" with -password "writecode". - -Contributing -============ - -You can download the code for ooniprobe from the following git repository:: - - - git clone https://git.torproject.org/ooni-probe.git - - -It is also viewable on the web via: https://gitweb.torproject.org/ooni-probe.git. - -You should then submit patches for review as pull requests to this github repository: - -https://github.com/TheTorProject/ooni-probe - -Read this article to learn how to create a pull request on github (https://help.github.com/articles/creating-a-pull-request) - -If you prefer not to use github (or don't have an account), you can also submit -patches as attachments to tickets. - -Be sure to format the patch (given that you are working on a feature branch -that is different from master) with:: - - - git format-patch master --stdout > my_first_ooniprobe.patch - - -Setting up development environment ----------------------------------- - -On debian based systems this can be done with:: - - sudo apt-get install libgeoip-dev python-virtualenv virtualenvwrapper - mkvirtualenv ooniprobe - python setup.py install - pip install -r requirements-dev.txt - - -Core ooniprobe Tests --------------------- - -The source for `Content blocking tests -<https://gitweb.torproject.org/ooni-probe.git/tree/HEAD:/ooni/nettests/block…>`_ -and `Traffic Manipulation tests -<https://gitweb.torproject.org/ooni-probe.git/tree/HEAD:/ooni/nettests/block…>`_ -can be found in the nettests/blocking and nettests/manipulation directories -respectively. - -Content Blocking Tests -^^^^^^^^^^^^^^^^^^^^^^ - - * `DNSConsistency <tests/dnsconsistency.html>`_ - - * `HTTP Requests <tests/http_requests.html>`_ - - * `TCP Connect <tests/tcpconnect.html>`_ - - -Traffic Manipulation Tests -^^^^^^^^^^^^^^^^^^^^^^^^^^^ - - * `HTTP Invalid Request Line: <tests/http_invalid_request_line.html>`_ - - * `DNS Spoof <tests/dnsspoof.html>`_ - - * `HTTP Header Field Manipulation <tests/http_header_field_manipulation.html>`_ - - * `Traceroute <tests/traceroute.html>`_ - - * `HTTP Host <tests/http_host.html>`_ - -Other tests -^^^^^^^^^^^ - -We also have some other tests that are currently not fully supported or still -being experimented with. - -You can find these in: - - * `ooni/nettests/experimental - <https://gitweb.torproject.org/ooni-probe.git/tree/HEAD:/ooni/nettests/exper…>`_ - -Tests that don't do a measurement but are useful for scanning can be found in: - - * `ooni/nettests/scanning - <https://gitweb.torproject.org/ooni-probe.git/tree/HEAD:/ooni/nettests/scann…>`_ - -Tests that involve running third party tools can be found in: - - * `ooni/nettests/third_party - <https://gitweb.torproject.org/ooni-probe.git/tree/HEAD:/ooni/nettests/third…>`_ - -ooni-backend ------------- - -This is the server side component of ooniprobe. It will store data -collected from ooniprobes and it will run a series of Test Helpers that assist -`Traffic Manipulation Tests`_ in performing their measurements. - -Test Helpers -^^^^^^^^^^^^ - -The currently implemented test helpers are the following: - - * `SSL Test Helpers - <https://gitweb.torproject.org/oonib.git/blob/HEAD:/oonib/testhelpers/ssl_he…>`_ - - * `HTTP Test Helpers - <https://gitweb.torproject.org/oonib.git/blob/HEAD:/oonib/testhelpers/http_h…>`_ - - * `TCP Test Helpers - <https://gitweb.torproject.org/oonib.git/blob/HEAD:/oonib/testhelpers/tcp_he…>`_ - - * `DNS Test Helpers - <https://gitweb.torproject.org/oonib.git/blob/HEAD:/oonib/testhelpers/dns_he…>`_ - -Threat Model ------------- - -Our adversary is capable of doing country wide network surveillance and -manipulation of network traffic. - -The goals of our adversary are to: - - * Restrict access to certain content, while not degrading overall quality of - the network - * Monitor the network in a way that they are able to identify misuse of it in - real time - -More specifc to the running of network filtering detection tests: - -1. Detect actors performing censorship detection tests -2. Fool people running such tests into believing that the network is - unrestricted - -**Note:** While 2) => 1) it is not true that 1) => 2) as the identification of -such actors does not necessarily have to happen in real time. - -While our intention is to minimize the risk of users running OONI probe being -identified, this comes with a trade-off in accuracy. It is therefore necessary in -certain tests to trade-off fingerprintability in favour of test accuracy. - -This is why we divide tests based on what risk the users running them can face, -allowing users to freely choose what threat model they wish to adere to. +.. include:: ../../README.rst More developer documentation diff --git a/ooni/ui/cli.py b/ooni/ui/cli.py index cf889a3c..4e92e960 100644 --- a/ooni/ui/cli.py +++ b/ooni/ui/cli.py @@ -26,8 +26,8 @@ class Options(usage.Options): optFlags = [["help", "h"], ["no-collector", "n", "Disable writing to collector"], ["no-njson", "N", "Disable writing to disk"], - ["no-geoip", "g", "Disable geoip lookup on start. " - "With this option your IP address can be disclosed in the report."], + ["no-geoip", "g", "Don't include geoip related information inside of the report." + "Note: Even with this option your IP address can be disclosed in the report."], ["list", "s", "List the currently installed ooniprobe " "nettests"], ["verbose", "v", "Show more verbose information"],

1 0

[ooni-probe/master] log the actual failure if Tor fails to start
by art＠torproject.org 22 Sep '17

22 Sep '17

commit c5dbca0bdc3e4185ca91e9507c1245b36f22c988 Author: meejah <meejah(a)meejah.ca> Date: Wed Apr 26 14:18:35 2017 -0600 log the actual failure if Tor fails to start --- ooni/utils/onion.py | 1 + 1 file changed, 1 insertion(+) diff --git a/ooni/utils/onion.py b/ooni/utils/onion.py index 691eda05..c4d5d4c3 100644 --- a/ooni/utils/onion.py +++ b/ooni/utils/onion.py @@ -337,6 +337,7 @@ class TorLauncherWithRetries(object): self.started.callback(state) def _setup_failed(self, failure): + log.msg("Starting Tor failed: {}".format(failure.value)) self.tor_output.seek(0) map(log.debug, self.tor_output.readlines()) self.tor_output.seek(0)

1 0

[ooni-probe/master] Randomise URLs when testing web_connectivity
by art＠torproject.org 22 Sep '17

22 Sep '17

commit 9cb4aa444ec2131f9fba77e9e67a13695d90db2e Author: Arturo Filastò <arturo(a)filasto.net> Date: Fri May 26 22:00:25 2017 +0000 Randomise URLs when testing web_connectivity This is aimed at reducing fingerprintabiliy of ooniprobe Fixes: https://github.com/TheTorProject/ooni-probe/issues/480 --- ooni/__init__.py | 2 +- ooni/nettests/blocking/web_connectivity.py | 9 +++++++++ 2 files changed, 10 insertions(+), 1 deletion(-) diff --git a/ooni/__init__.py b/ooni/__init__.py index abca74c1..804f1359 100644 --- a/ooni/__init__.py +++ b/ooni/__init__.py @@ -1,7 +1,7 @@ # -*- encoding: utf-8 -*- __author__ = "Open Observatory of Network Interference" -__version__ = "2.2.0" +__version__ = "2.2.1-dev" __all__ = [ 'agent', diff --git a/ooni/nettests/blocking/web_connectivity.py b/ooni/nettests/blocking/web_connectivity.py index 600a9e4e..3807a0de 100644 --- a/ooni/nettests/blocking/web_connectivity.py +++ b/ooni/nettests/blocking/web_connectivity.py @@ -1,6 +1,7 @@ # -*- encoding: utf-8 -*- import csv +import random from urlparse import urlparse from twisted.internet import defer @@ -36,6 +37,9 @@ class UsageOptions(usage.Options): ['retries', 'r', 1, 'Number of retries for the HTTP request'], ['timeout', 't', 240, 'Total timeout for this test'], ] + optFlags = [ + ['no-shuffle', '', 'Disable shuffling of URLs'], + ] class WebConnectivityTest(httpt.HTTPTest, dnst.DNSTest): @@ -123,6 +127,11 @@ class WebConnectivityTest(httpt.HTTPTest, dnst.DNSTest): else: fh.seek(0) generator = simple_file_generator(fh) + if self.localOptions['no-shuffle'] != True: + input_list = list(generator) + random.shuffle(input_list) + generator = input_list + for i in generator: if (not i.startswith("http://") and not i.startswith("https://")):

1 0

[ooni-probe/master] Merge pull request #772 from TheTorProject/fix/tor-binary
by art＠torproject.org 22 Sep '17

22 Sep '17

commit 625268f165f4539dade55c8ea7ab6d17f107d09a Merge: b3a6b393 626aedf0 Author: Arturo Filastò <arturo(a)filasto.net> Date: Tue Sep 19 11:11:49 2017 +0200 Merge pull request #772 from TheTorProject/fix/tor-binary Use `onion.find_tor_binary()` to obtain the path to the tor binary al… ooni/nettests/blocking/bridge_reachability.py | 11 +++++++---- ooni/nettests/blocking/vanilla_tor.py | 6 ++++-- ooni/tests/test_onion.py | 24 +++++++++++++++--------- ooni/utils/onion.py | 25 +++++++++++++++---------- setup.py | 1 - 5 files changed, 41 insertions(+), 26 deletions(-)

1 0

[ooni-probe/master] Use `onion.find_tor_binary()` to obtain the path to the tor binary always
by art＠torproject.org 22 Sep '17

22 Sep '17

commit edc76fab398474e28381df9cec3b0efef905ab34 Author: Arturo Filastò <arturo(a)filasto.net> Date: Sun Jun 25 17:59:50 2017 +0200 Use `onion.find_tor_binary()` to obtain the path to the tor binary always * Avoid setting `tor_details` globally as this could be called when the config file has not yet been parsed and therefore it may not respect `advanced-tor_binary` --- ooni/nettests/blocking/bridge_reachability.py | 11 +++++++---- ooni/nettests/blocking/vanilla_tor.py | 6 ++++-- ooni/tests/test_onion.py | 7 ++++--- ooni/utils/onion.py | 25 +++++++++++++++---------- setup.py | 1 - 5 files changed, 30 insertions(+), 20 deletions(-) diff --git a/ooni/nettests/blocking/bridge_reachability.py b/ooni/nettests/blocking/bridge_reachability.py index e5cc512e..870dcfa1 100644 --- a/ooni/nettests/blocking/bridge_reachability.py +++ b/ooni/nettests/blocking/bridge_reachability.py @@ -48,6 +48,8 @@ class BridgeReachability(nettest.NetTestCase): def setUp(self): self.tor_progress = 0 + self.tor_details = onion.get_tor_details() + self.obfsproxy_details = onion.get_obfsproxy_details() self.timeout = int(self.localOptions['timeout']) fd, self.tor_logfile = tempfile.mkstemp() @@ -60,12 +62,12 @@ class BridgeReachability(nettest.NetTestCase): self.report['success'] = None self.report['timeout'] = self.timeout self.report['transport_name'] = 'vanilla' - self.report['tor_version'] = str(onion.tor_details['version']) + self.report['tor_version'] = str(self.tor_details['version']) self.report['tor_progress'] = 0 self.report['tor_progress_tag'] = None self.report['tor_progress_summary'] = None self.report['tor_log'] = None - self.report['obfsproxy_version'] = str(onion.obfsproxy_details['version']) + self.report['obfsproxy_version'] = str(self.obfsproxy_details['version']) self.report['obfsproxy_log'] = None self.report['bridge_address'] = None @@ -128,7 +130,7 @@ class BridgeReachability(nettest.NetTestCase): config.DataDirectory = self.tor_datadir log.msg( "Connecting to %s with tor %s" % - (self.bridge, onion.tor_details['version'])) + (self.bridge, self.tor_details['version'])) transport_name = onion.transport_name(self.bridge) if transport_name == None: @@ -175,7 +177,8 @@ class BridgeReachability(nettest.NetTestCase): self.report['tor_progress_tag'] = tag self.report['tor_progress_summary'] = summary - d = txtorcon.launch_tor(config, reactor, timeout=self.timeout, + d = txtorcon.launch_tor(config, reactor, tor_binary=onion.find_tor_binary(), + timeout=self.timeout, progress_updates=updates) @d.addCallback diff --git a/ooni/nettests/blocking/vanilla_tor.py b/ooni/nettests/blocking/vanilla_tor.py index 717fa733..a147be89 100644 --- a/ooni/nettests/blocking/vanilla_tor.py +++ b/ooni/nettests/blocking/vanilla_tor.py @@ -39,6 +39,7 @@ class VanillaTor(nettest.NetTestCase): def setUp(self): self.tor_progress = 0 + self.tor_details = onion.get_tor_details() self.timeout = int(self.localOptions['timeout']) fd, self.tor_logfile = tempfile.mkstemp() @@ -49,7 +50,7 @@ class VanillaTor(nettest.NetTestCase): self.report['success'] = None self.report['timeout'] = self.timeout self.report['transport_name'] = 'vanilla' - self.report['tor_version'] = str(onion.tor_details['version']) + self.report['tor_version'] = str(self.tor_details['version']) self.report['tor_progress'] = 0 self.report['tor_progress_tag'] = None self.report['tor_progress_summary'] = None @@ -73,7 +74,8 @@ class VanillaTor(nettest.NetTestCase): self.report['tor_progress_tag'] = tag self.report['tor_progress_summary'] = summary - d = txtorcon.launch_tor(config, reactor, timeout=self.timeout, + d = txtorcon.launch_tor(config, reactor, tor_binary=onion.find_tor_binary(), + timeout=self.timeout, progress_updates=updates) @d.addCallback diff --git a/ooni/tests/test_onion.py b/ooni/tests/test_onion.py index 732614fb..0807ca02 100644 --- a/ooni/tests/test_onion.py +++ b/ooni/tests/test_onion.py @@ -26,9 +26,10 @@ class MockSuccessTorProtocol(object): class TestOnion(unittest.TestCase): def test_tor_details(self): - assert isinstance(onion.tor_details, dict) - assert onion.tor_details['version'] - assert onion.tor_details['binary'] + tor_details = onion.get_tor_details() + assert isinstance(tor_details, dict) + assert tor_details['version'] + assert tor_details['binary'] def test_transport_dicts(self): diff --git a/ooni/utils/onion.py b/ooni/utils/onion.py index 283579cd..2c2a2c81 100644 --- a/ooni/utils/onion.py +++ b/ooni/utils/onion.py @@ -98,15 +98,17 @@ def find_pt_executable(name): return bin_loc return None -tor_details = { - 'binary': find_tor_binary(), - 'version': tor_version() -} - -obfsproxy_details = { - 'binary': find_executable('obfsproxy'), - 'version': obfsproxy_version() -} +def get_tor_details(): + return { + 'binary': find_tor_binary(), + 'version': tor_version() + } + +def get_obfsproxy_details(): + return { + 'binary': find_executable('obfsproxy'), + 'version': obfsproxy_version() + } transport_bin_name = { 'fte': 'fteproxy', 'scramblesuit': 'obfsproxy', @@ -148,6 +150,9 @@ class OutdatedTor(Exception): pass def bridge_line(transport, log_file): + obfsproxy_details = get_obfsproxy_details() + tor_details = get_tor_details() + bin_name = transport_bin_name.get(transport) if not bin_name: raise UnrecognizedTransport @@ -387,7 +392,7 @@ class TorLauncherWithRetries(object): def _launch_tor(self): return launch_tor(self.tor_config, reactor, - tor_binary=config.advanced.tor_binary, + tor_binary=find_tor_binary(), progress_updates=self._progress_updates, stdout=self.tor_output, timeout=self.timeout, diff --git a/setup.py b/setup.py index 26ed2a19..a16891e7 100644 --- a/setup.py +++ b/setup.py @@ -279,7 +279,6 @@ def setup_package(): }, classifiers=[c for c in CLASSIFIERS.split('\n') if c] ) - setup(**metadata) if __name__ == "__main__":

1 0

[ooni-probe/master] Fix confusing message
by art＠torproject.org 22 Sep '17

22 Sep '17

commit efd6b477874b6b8edb2fb88ebc0603353c1c16eb Author: Arturo Filastò <arturo(a)filasto.net> Date: Sun May 28 14:49:22 2017 +0000 Fix confusing message --- ooni/nettest.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/ooni/nettest.py b/ooni/nettest.py index edba448c..2988db5b 100644 --- a/ooni/nettest.py +++ b/ooni/nettest.py @@ -575,8 +575,8 @@ class NetTest(object): log.msg("") log.msg("Status") log.msg("------") - log.msg("%d completed %d remaining" % (self._completedInputs, - self._totalInputs)) + log.msg("%d/%d completed" % (self._completedInputs, + self._totalInputs)) log.msg("%0.1f%% (ETA: %ds)" % (self.completionPercentage * 100, self.completionEta)) self.state.taskDone()

1 0

[ooni-probe/master] Fix Microsoft Azure meek endpoint, format (#746)
by art＠torproject.org 22 Sep '17

22 Sep '17

commit 3063f04fa3ce9b187778b382c464bfd9fa71cf3e Author: anadahz <anadahz(a)users.noreply.github.com> Date: Fri Jun 16 11:24:07 2017 -0300 Fix Microsoft Azure meek endpoint, format (#746) * Fix Microsoft Azure meek endpoint, format * Update default meek tor bridges Based on the tor Launcher preferences (default bridges): https://gitweb.torproject.org/builders/tor-browser-bundle.git/tree/Bundle-D… --- ooni/nettests/blocking/meek_fronted_requests.py | 21 ++++++++++----------- 1 file changed, 10 insertions(+), 11 deletions(-) diff --git a/ooni/nettests/blocking/meek_fronted_requests.py b/ooni/nettests/blocking/meek_fronted_requests.py index f516f84e..aee9c7ce 100644 --- a/ooni/nettests/blocking/meek_fronted_requests.py +++ b/ooni/nettests/blocking/meek_fronted_requests.py @@ -8,12 +8,12 @@ from ooni.utils import log class UsageOptions(usage.Options): optParameters = [ ['expectedBody', 'B', - 'I’m just a happy little web server.\n', - 'Expected body content from GET response.'], + 'I’m just a happy little web server.\n', + 'Expected body content from GET response.'], ['domainName', 'D', None, - 'Specify a single fronted domainName to test.'], + 'Specify a single fronted domainName to test.'], ['hostHeader', 'H', None, - 'Specify "inside" Host Header to test.'] + 'Specify "inside" Host Header to test.'] ] class meekTest(httpt.HTTPTest): @@ -23,21 +23,21 @@ class meekTest(httpt.HTTPTest): and response with: "I’m just a happy little web server.\n". The input file should be formatted as (one per line): "domainName:hostHeader" - ajax.aspnetcdn.com:az668014.vo.msecnd.net + ajax.aspnetcdn.com:meek.azureedge.net a0.awsstatic.com:d2zfqthxsdq309.cloudfront.net """ name = "Meek fronted requests test" description = "This test examines whether the domains used by Meek "\ "(a type of Tor bridge) work in your network." - version = "0.1.0" + version = "0.2.0" usageOptions = UsageOptions inputFile = ['file', 'f', None, - "File containing the domainName:hostHeader combinations to\ - be tested, one per line."] - inputs = [('ajax.aspnetcdn.com', 'az668014.vo.msecnd.net'), - ('a0.awsstatic.com', 'd2zfqthxsdq309.cloudfront.net')] + 'File containing the domainName:hostHeader combinations to \ + be tested, one per line.'] + inputs = [('ajax.aspnetcdn.com', 'meek.azureedge.net'), + ('a0.awsstatic.com', 'd2cly7j4zqgua7.cloudfront.net')] requiresRoot = False requiresTor = False @@ -76,4 +76,3 @@ class meekTest(httpt.HTTPTest): headers['Host'] = [self.header] return self.doRequest(self.domainName, method="GET", headers=headers, body_processor=process_body) -

1 0

[ooni-probe/master] Fix active meek tor bridges (#748)
by art＠torproject.org 22 Sep '17

22 Sep '17

commit 33f50fd9bc40fdf71b11085e272e257f44bae5ef Author: anadahz <anadahz(a)users.noreply.github.com> Date: Fri Jun 16 11:23:18 2017 -0300 Fix active meek tor bridges (#748) * Fix active meek tor bridges Based on the tor Launcher preferences (default bridges): https://gitweb.torproject.org/builders/tor-browser-bundle.git/tree/Bundle-D… --- data/ooniprobe.conf.sample | 5 ++--- ooni/settings.py | 5 ++--- 2 files changed, 4 insertions(+), 6 deletions(-) diff --git a/data/ooniprobe.conf.sample b/data/ooniprobe.conf.sample index 8a663824..a74e03f7 100644 --- a/data/ooniprobe.conf.sample +++ b/data/ooniprobe.conf.sample @@ -87,7 +87,6 @@ tor: #HTTPSProxyAuthenticator: user:password #UseBridges: 1 #Bridge: - #- "meek_lite 0.0.2.0:1 url=https://meek-reflect.appspot.com/ front=www.google.com" - #- "meek_lite 0.0.2.0:2 url=https://d2zfqthxsdq309.cloudfront.net/ front=a0.awsstatic.com" - #- "meek_lite 0.0.2.0:3 url=https://az786092.vo.msecnd.net/ front=ajax.aspnetcdn.com" + #- "meek_lite 0.0.2.0:1 url=https://d2cly7j4zqgua7.cloudfront.net/ front=a0.awsstatic.com" + #- "meek_lite 0.0.2.0:2 url=https://meek.azureedge.net/ front=ajax.aspnetcdn.com" #ClientTransportPlugin: "meek_lite exec /usr/bin/obfs4proxy" diff --git a/ooni/settings.py b/ooni/settings.py index 51009fb2..61e3e51f 100644 --- a/ooni/settings.py +++ b/ooni/settings.py @@ -105,9 +105,8 @@ tor: #HTTPSProxyAuthenticator: user:password #UseBridges: 1 #Bridge: - #- "meek_lite 0.0.2.0:1 url=https://meek-reflect.appspot.com/ front=www.google.com" - #- "meek_lite 0.0.2.0:2 url=https://d2zfqthxsdq309.cloudfront.net/ front=a0.awsstatic.com" - #- "meek_lite 0.0.2.0:3 url=https://az786092.vo.msecnd.net/ front=ajax.aspnetcdn.com" + #- "meek_lite 0.0.2.0:1 url=https://d2cly7j4zqgua7.cloudfront.net/ front=a0.awsstatic.com" + #- "meek_lite 0.0.2.0:2 url=https://meek.azureedge.net/ front=ajax.aspnetcdn.com" #ClientTransportPlugin: "meek_lite exec /usr/bin/obfs4proxy" """

1 0

[ooni-probe/master] Merge branch 'master' into fix/tor-binary
by art＠torproject.org 22 Sep '17

22 Sep '17

commit 626aedf03ec7c07d5f2e8dcd53b4bbbc5b683cfb Merge: eec13dff b3a6b393 Author: Arturo Filastò <arturo(a)filasto.net> Date: Mon Jul 17 09:51:35 2017 +0200 Merge branch 'master' into fix/tor-binary README.rst | 4 +- docs/source/index.rst | 452 +------------------------------------------------- ooni/ui/cli.py | 4 +- 3 files changed, 6 insertions(+), 454 deletions(-)

1 0