July 2016 - tor-commits - lists.torproject.org

[nyx/master] Migrate and test torrc panel
by atagar＠torproject.org 11 Jul '16

11 Jul '16

commit 8db31995c703b716b6108e08c408e78fed22e368 Merge: b2bee4b f388d3b Author: Damian Johnson <atagar(a)torproject.org> Date: Mon Jul 11 09:56:56 2016 -0700 Migrate and test torrc panel Smallest and simplest of our panels but no complains from me - glad this overhaul's nearly done! Just one more panel to go. Rather than breaking this panel up right now just testing the whole thing. nyx/panel/__init__.py | 2 +- nyx/panel/torrc.py | 51 ++++++++++++++++------------- test/panel/__init__.py | 1 + test/panel/connection.py | 50 +++++++++++++++++++--------- test/panel/torrc.py | 84 ++++++++++++++++++++++++++++++++++++++++++++++++ 5 files changed, 149 insertions(+), 39 deletions(-)

1 0

[obfs4/master] Use the correct epoch hour when crafting the obfs4 server ntor response.
by yawning＠torproject.org 11 Jul '16

11 Jul '16

commit 62057625eaba2a555967b9615039d43dda071360 Author: Yawning Angel <yawning(a)schwanenlied.me> Date: Mon Jul 11 00:51:28 2016 +0000 Use the correct epoch hour when crafting the obfs4 server ntor response. It's supposed to use the one derived from the client's handshake (assuming the clock skew is within acceptable limits), but it was using the one based off the current system time. --- ChangeLog | 2 ++ transports/obfs4/handshake_ntor.go | 3 +-- 2 files changed, 3 insertions(+), 2 deletions(-) diff --git a/ChangeLog b/ChangeLog index 92390b4..09655b3 100644 --- a/ChangeLog +++ b/ChangeLog @@ -2,6 +2,8 @@ Changes in version 0.0.7 - UNRELEASED: - Support configuring the obfs4 IAT parameter as the sole ServerTransportOption on bridges, and correctly checkpoint the argument to the state file. + - Correctly use the derived epoch hour when generating the server obfs4 + ntor handshake response to be more tollerant of clock skew. Changes in version 0.0.6 - 2016-01-25: - Delay transport factory initialization till after logging has been diff --git a/transports/obfs4/handshake_ntor.go b/transports/obfs4/handshake_ntor.go index 57de460..ee1bca8 100644 --- a/transports/obfs4/handshake_ntor.go +++ b/transports/obfs4/handshake_ntor.go @@ -356,8 +356,7 @@ func (hs *serverHandshake) generateHandshake() ([]byte, error) { // Calculate and write the MAC. hs.mac.Reset() hs.mac.Write(buf.Bytes()) - hs.epochHour = []byte(strconv.FormatInt(getEpochHour(), 10)) - hs.mac.Write(hs.epochHour) + hs.mac.Write(hs.epochHour) // Set in hs.parseClientHandshake() buf.Write(hs.mac.Sum(nil)[:macLength]) return buf.Bytes(), nil

1 0

[stem/master] Update torIRC link
by atagar＠torproject.org 10 Jul '16

10 Jul '16

commit 1033d98c9034dfc3fbc31c12c149998a30d7818e Author: Damian Johnson <atagar(a)torproject.org> Date: Sun Jul 10 13:44:20 2016 -0700 Update torIRC link The old torIRC hasn't been touched since 2013. Updating this link as requested by the writer of this alternative. --- docs/tutorials/double_double_toil_and_trouble.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/tutorials/double_double_toil_and_trouble.rst b/docs/tutorials/double_double_toil_and_trouble.rst index abab871..8dfbf4b 100644 --- a/docs/tutorials/double_double_toil_and_trouble.rst +++ b/docs/tutorials/double_double_toil_and_trouble.rst @@ -65,7 +65,7 @@ Applications `check_tor <http://anonscm.debian.org/gitweb/?p=users/lunar/check_tor.git;a=blob;f=chec…>`_ Nagios check to verify that a relay is participating in the Tor network. `munin-tor <https://github.com/mweinelt/munin-tor>`_ Plugin to provide Munin graphs. `tbbscraper <https://github.com/zackw/tbbscraper/blob/master/collector/lib/controller/co…>`_ Automated website scraper over Tor. -`torirc <https://github.com/alfred-gw/torirc>`_ IRC-like chat client using Tor hidden services. +`torIRC <https://gist.github.com/torifier/f1a7c1ac7b6b003cd9e1c187df2c5347>`_ IRC-like chat client using Tor hidden services. `exit-funding <https://github.com/torservers/exit-funding>`_ Script to estimate how much exit relays have been used. `torsearch <https://github.com/wfn/torsearch>`_ Search engine prototype for descriptor data. `or-applet <https://github.com/Yawning/or-applet>`_ GUI widget to provide circuit information and an interactive interpreter.

1 0

[ooni-probe/master] Fixes to support cloudfronting and https collectors
by art＠torproject.org 10 Jul '16

10 Jul '16

commit c9e7ac559bf4ca2754105495ab86b3f533287843 Author: Arturo Filastò <arturo(a)filasto.net> Date: Tue Jun 21 16:28:18 2016 +0200 Fixes to support cloudfronting and https collectors --- ooni/backend_client.py | 11 ++++++----- ooni/constants.py | 4 ++-- ooni/deck.py | 3 ++- ooni/oonicli.py | 1 + 4 files changed, 11 insertions(+), 8 deletions(-) diff --git a/ooni/backend_client.py b/ooni/backend_client.py index 71b774f..83869cc 100644 --- a/ooni/backend_client.py +++ b/ooni/backend_client.py @@ -36,19 +36,17 @@ class OONIBClient(object): self.base_headers = {} self.backend_type = settings.get('type', None) self.base_address = settings.get('address', address) + self.front = settings.get('front', '').encode('ascii') if self.backend_type is None: self.backend_type = guess_backend_type(self.base_address) self.backend_type = self.backend_type.encode('ascii') - if self.backend_type == 'cloudfront': - self.base_headers['Host'] = settings['front'].encode('ascii') - self._setupBaseAddress() self.settings = { 'type': self.backend_type, 'address': self.base_address, - 'front': settings.get('front', '').encode('ascii') + 'front': self.front } def _setupBaseAddress(self): @@ -64,8 +62,11 @@ class OONIBClient(object): parsed_address.netloc)) elif self.backend_type == 'http': self.base_address = ("http://%s" % parsed_address.netloc) - elif self.backend_type in ('https', 'cloudfront'): + elif self.backend_type == 'https': self.base_address = ("https://%s" % parsed_address.netloc) + elif self.backend_type == 'cloudfront': + self.base_headers['Host'] = [parsed_address.netloc] + self.base_address = ("https://%s" % self.front) self.base_address = self.base_address.encode('ascii') def isSupported(self): diff --git a/ooni/constants.py b/ooni/constants.py index ef1f3d7..881a932 100644 --- a/ooni/constants.py +++ b/ooni/constants.py @@ -1,8 +1,8 @@ CANONICAL_BOUNCER_ONION = 'httpo://nkvphnp3p6agi5qq.onion' CANONICAL_BOUNCER_HTTPS = 'https://bouncer.ooni.io' CANONICAL_BOUNCER_CLOUDFRONT = ( - 'XXXX.cloudfront.net', - 'https://a0.awsstatic.com/' + 'https://XXXX.cloudfront.net/', + 'a0.awsstatic.com' ) MEEK_BRIDGES = [ diff --git a/ooni/deck.py b/ooni/deck.py index 2aa958d..aee3da9 100644 --- a/ooni/deck.py +++ b/ooni/deck.py @@ -219,7 +219,8 @@ class Deck(InputFile): yield self.fetchAndVerifyNetTestInput(net_test_loader) if self.bouncer: - log.msg("Looking up collector and test helpers") + log.msg("Looking up collector and test helpers with {0}".format( + self.bouncer.base_address)) yield self.lookupCollectorAndTestHelpers() diff --git a/ooni/oonicli.py b/ooni/oonicli.py index 43654b6..c0ee5cd 100644 --- a/ooni/oonicli.py +++ b/ooni/oonicli.py @@ -345,6 +345,7 @@ def runWithDirector(global_options): test! """ from ooni.director import Director + start_tor = False director = Director() if global_options['list']: net_tests = [net_test for net_test in director.getNetTests().items()]

1 0

[ooni-probe/master] Implementing bridge failover in ooniprobe
by art＠torproject.org 10 Jul '16

10 Jul '16

commit 0ca814db9c9c87486012b20e4cdd951b6f6f85cc Author: Arturo Filastò <arturo(a)filasto.net> Date: Tue Jun 14 16:15:44 2016 +0300 Implementing bridge failover in ooniprobe We support failing over to obfs4 and meek when vanilla tor does not work. * This implements #538 --- ooni/utils/onion.py | 9 --------- 1 file changed, 9 deletions(-) diff --git a/ooni/utils/onion.py b/ooni/utils/onion.py index 51cd044..e18a6ee 100644 --- a/ooni/utils/onion.py +++ b/ooni/utils/onion.py @@ -214,16 +214,11 @@ def get_client_transport(transport): class TorLauncherWithRetries(object): -<<<<<<< 970cb5191f77b1da0d656e6441d77d42d53afb95 def __init__(self, tor_config, timeout=config.tor.timeout): -======= - def __init__(self, tor_config, timeout=200): ->>>>>>> Implementing bridge failover in ooniprobe self.retry_with = ["obfs4", "meek"] self.started = defer.Deferred() self.tor_output = StringIO.StringIO() self.tor_config = tor_config -<<<<<<< 970cb5191f77b1da0d656e6441d77d42d53afb95 if timeout is None: # XXX we will want to move setting the default inside of the # config object. @@ -243,10 +238,6 @@ class TorLauncherWithRetries(object): setattr(new_tor_config, key, getattr(self.tor_config, key)) self.tor_config = new_tor_config -======= - self.timeout = timeout - ->>>>>>> Implementing bridge failover in ooniprobe def _progress_updates(self, prog, tag, summary): log.msg("%d%%: %s" % (prog, summary))

1 0

[ooni-probe/master] Implementing bridge failover in ooniprobe
by art＠torproject.org 10 Jul '16

10 Jul '16

commit 46c623245bb7dbe791e5cf1cd2edaf2a46a5e7e0 Author: Arturo Filastò <arturo(a)filasto.net> Date: Tue Jun 14 16:15:44 2016 +0300 Implementing bridge failover in ooniprobe We support failing over to obfs4 and meek when vanilla tor does not work. * This implements #538 --- ooni/utils/onion.py | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/ooni/utils/onion.py b/ooni/utils/onion.py index e18a6ee..51cd044 100644 --- a/ooni/utils/onion.py +++ b/ooni/utils/onion.py @@ -214,11 +214,16 @@ def get_client_transport(transport): class TorLauncherWithRetries(object): +<<<<<<< 970cb5191f77b1da0d656e6441d77d42d53afb95 def __init__(self, tor_config, timeout=config.tor.timeout): +======= + def __init__(self, tor_config, timeout=200): +>>>>>>> Implementing bridge failover in ooniprobe self.retry_with = ["obfs4", "meek"] self.started = defer.Deferred() self.tor_output = StringIO.StringIO() self.tor_config = tor_config +<<<<<<< 970cb5191f77b1da0d656e6441d77d42d53afb95 if timeout is None: # XXX we will want to move setting the default inside of the # config object. @@ -238,6 +243,10 @@ class TorLauncherWithRetries(object): setattr(new_tor_config, key, getattr(self.tor_config, key)) self.tor_config = new_tor_config +======= + self.timeout = timeout + +>>>>>>> Implementing bridge failover in ooniprobe def _progress_updates(self, prog, tag, summary): log.msg("%d%%: %s" % (prog, summary))

1 0

[ooni-probe/master] Use ConfigNetTest case inside of test_nettest as well
by art＠torproject.org 10 Jul '16

10 Jul '16

commit a9e1e9d91299a6033db061344425553e20a8d55e Author: Arturo Filastò <arturo(a)filasto.net> Date: Mon Jun 20 14:31:01 2016 +0200 Use ConfigNetTest case inside of test_nettest as well --- ooni/tests/test_nettest.py | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/ooni/tests/test_nettest.py b/ooni/tests/test_nettest.py index 55134d3..4298fe9 100644 --- a/ooni/tests/test_nettest.py +++ b/ooni/tests/test_nettest.py @@ -219,7 +219,7 @@ dummyInputFile = 'dummyInputFile.txt' -class TestNetTest(unittest.TestCase): +class TestNetTest(ConfigTestCase): timeout = 1 def setUp(self): @@ -227,11 +227,13 @@ class TestNetTest(unittest.TestCase): with open(dummyInputFile, 'w') as f: for i in range(10): f.write("%s\n" % i) + super(TestNetTest, self).setUp() def tearDown(self): os.remove(dummyInputFile) if self.filename != "": os.remove(self.filename) + super(TestNetTest, self).tearDown() def assertCallable(self, thing): self.assertIn('__call__', dir(thing))

1 0

[ooni-probe/master] Implementing bridge failover in ooniprobe (#540)
by art＠torproject.org 10 Jul '16

10 Jul '16

commit 970cb5191f77b1da0d656e6441d77d42d53afb95 Author: Arturo Filastò <arturo(a)filasto.net> Date: Wed Jun 29 16:38:09 2016 +0200 Implementing bridge failover in ooniprobe (#540) * Implementing bridge failover in ooniprobe We support failing over to obfs4 and meek when vanilla tor does not work. * This implements #538 * Reset the DataDirectory when we the data_dir is not set Otherwise txtorcon will delete the datadirectory after it shutsdown and not re-create it. --- data/ooniprobe.conf.sample | 4 + ooni/__init__.py | 2 - ooni/constants.py | 76 ++++++++++++++++++ ooni/director.py | 58 ++----------- ooni/oonicli.py | 5 +- ooni/report/cli.py | 4 +- ooni/report/tool.py | 4 +- ooni/tests/test_director.py | 4 +- ooni/tests/test_onion.py | 45 ++++++++++- ooni/utils/onion.py | 192 +++++++++++++++++++++++++++++++++++++++++++- 10 files changed, 326 insertions(+), 68 deletions(-) diff --git a/data/ooniprobe.conf.sample b/data/ooniprobe.conf.sample index fec1026..be35364 100644 --- a/data/ooniprobe.conf.sample +++ b/data/ooniprobe.conf.sample @@ -58,6 +58,10 @@ tor: # This should be set to something to avoid having Tor download each time # the descriptors and consensus data. #data_dir: ~/.tor/ + # + # This is the timeout after which we consider to to not have + # bootstrapped properly. + timeout: 200 torrc: #HTTPProxy: host:port #HTTPProxyAuthenticator: user:password diff --git a/ooni/__init__.py b/ooni/__init__.py index 5280b1e..6f6e78f 100644 --- a/ooni/__init__.py +++ b/ooni/__init__.py @@ -5,8 +5,6 @@ __version__ = "1.5.2.dev1" # This is the version number of resources to be downloaded # when a release is made it should be aligned to __version__ __resources_version__ = "1.5.1" -canonical_bouncer = 'httpo://nkvphnp3p6agi5qq.onion' - __all__ = [ 'common', diff --git a/ooni/constants.py b/ooni/constants.py new file mode 100644 index 0000000..882edb4 --- /dev/null +++ b/ooni/constants.py @@ -0,0 +1,76 @@ +CANONICAL_BOUNCER_ONION = 'httpo://nkvphnp3p6agi5qq.onion' + +MEEK_BRIDGES = [ + ("meek 0.0.2.0:2 B9E7141C594AF25699E0079C1F0146F409495296 " + "url=https://d2zfqthxsdq309.cloudfront.net/ front=a0.awsstatic.com"), + ("meek 0.0.2.0:3 A2C13B7DFCAB1CBF3A884B6EB99A98067AB6EF44 " + "url=https://az786092.vo.msecnd.net/ front=ajax.aspnetcdn.com") +] + +# These are bridges taken from TBB +OBFS4_BRIDGES = [ + ("obfs4 154.35.22.10:41835 8FB9F4319E89E5C6223052AA525A192AFBC85D55 " + "cert=GGGS1TX4R81m3r0HBl79wKy1OtPPNR2CZUIrHjkRg65Vc2VR8fOyo64f9kmT1UAFG7j0HQ iat-mode=0"), + + ("obfs4 198.245.60.50:443 752CF7825B3B9EA6A98C83AC41F7099D67007EA5 " + "cert=xpmQtKUqQ/6v5X7ijgYE/f03+l2/EuQ1dexjyUhh16wQlu" + "/cpXUGalmhDIlhuiQPNEKmKw iat-mode=0"), + + ("obfs4 192.99.11.54:443 7B126FAB960E5AC6A629C729434FF84FB5074EC2 " + "cert=VW5f8+IBUWpPFxF+rsiVy2wXkyTQG7vEd" + "+rHeN2jV5LIDNu8wMNEOqZXPwHdwMVEBdqXEw iat-mode=0"), + + ("obfs4 109.105.109.165:10527 8DFCD8FB3285E855F5A55EDDA35696C743ABFC4E " + "cert=Bvg/itxeL4TWKLP6N1MaQzSOC6tcRIBv6q57DYAZc3b2AzuM" + "+/TfB7mqTFEfXILCjEwzVA iat-mode=0"), + + ("obfs4 83.212.101.3:41213 A09D536DD1752D542E1FBB3C9CE4449D51298239 " + "cert=lPRQ/MXdD1t5SRZ9MquYQNT9m5DV757jtdXdlePmRCudUU9CFUOX1Tm7" + "/meFSyPOsud7Cw iat-mode=0"), + + ("obfs4 104.131.108.182:56880 EF577C30B9F788B0E1801CF7E433B3B77792B77A " + "cert=0SFhfDQrKjUJP8Qq6wrwSICEPf3Vl" + "/nJRsYxWbg3QRoSqhl2EB78MPS2lQxbXY4EW1wwXA iat-mode=0"), + + ("obfs4 109.105.109.147:13764 BBB28DF0F201E706BE564EFE690FE9577DD8386D " + "cert=KfMQN/tNMFdda61hMgpiMI7pbwU1T+wxjTulYnfw" + "+4sgvG0zSH7N7fwT10BI8MUdAD7iJA iat-mode=0"), + + ("obfs4 154.35.22.11:49868 A832D176ECD5C7C6B58825AE22FC4C90FA249637 " + "cert=YPbQqXPiqTUBfjGFLpm9JYEFTBvnzEJDKJxXG5Sxzrr" + "/v2qrhGU4Jls9lHjLAhqpXaEfZw iat-mode=0"), + + ("obfs4 154.35.22.12:80 00DC6C4FA49A65BD1472993CF6730D54F11E0DBB " + "cert=N86E9hKXXXVz6G7w2z8wFfhIDztDAzZ" + "/3poxVePHEYjbKDWzjkRDccFMAnhK75fc65pYSg iat-mode=0"), + + ("obfs4 154.35.22.13:443 FE7840FE1E21FE0A0639ED176EDA00A3ECA1E34D " + "cert=fKnzxr+m+jWXXQGCaXe4f2gGoPXMzbL+bTBbXMYXuK0tMotd" + "+nXyS33y2mONZWU29l81CA iat-mode=0"), + + ("obfs4 154.35.22.10:80 8FB9F4319E89E5C6223052AA525A192AFBC85D55 " + "cert=GGGS1TX4R81m3r0HBl79wKy1OtPPNR2CZUIrHjkRg65Vc2VR8fOyo64f9kmT1UAFG7j0HQ iat-mode=0"), + + ("obfs4 154.35.22.10:443 8FB9F4319E89E5C6223052AA525A192AFBC85D55 " + "cert=GGGS1TX4R81m3r0HBl79wKy1OtPPNR2CZUIrHjkRg65Vc2VR8fOyo64f9kmT1UAFG7j0HQ iat-mode=0"), + + ("obfs4 154.35.22.11:443 A832D176ECD5C7C6B58825AE22FC4C90FA249637 " + "cert=YPbQqXPiqTUBfjGFLpm9JYEFTBvnzEJDKJxXG5Sxzrr" + "/v2qrhGU4Jls9lHjLAhqpXaEfZw iat-mode=0"), + + ("obfs4 154.35.22.11:80 A832D176ECD5C7C6B58825AE22FC4C90FA249637 " + "cert=YPbQqXPiqTUBfjGFLpm9JYEFTBvnzEJDKJxXG5Sxzrr" + "/v2qrhGU4Jls9lHjLAhqpXaEfZw iat-mode=0"), + + ("obfs4 154.35.22.9:60873 C73ADBAC8ADFDBF0FC0F3F4E8091C0107D093716 " + "cert=gEGKc5WN/bSjFa6UkG9hOcft1tuK" + "+cV8hbZ0H6cqXiMPLqSbCh2Q3PHe5OOr6oMVORhoJA iat-mode=0"), + + ("obfs4 154.35.22.9:80 C73ADBAC8ADFDBF0FC0F3F4E8091C0107D093716 " + "cert=gEGKc5WN/bSjFa6UkG9hOcft1tuK" + "+cV8hbZ0H6cqXiMPLqSbCh2Q3PHe5OOr6oMVORhoJA iat-mode=0"), + + ("obfs4 154.35.22.9:443 C73ADBAC8ADFDBF0FC0F3F4E8091C0107D093716 " + "cert=gEGKc5WN/bSjFa6UkG9hOcft1tuK" + "+cV8hbZ0H6cqXiMPLqSbCh2Q3PHe5OOr6oMVORhoJA iat-mode=0") +] diff --git a/ooni/director.py b/ooni/director.py index e6f864e..02b6743 100644 --- a/ooni/director.py +++ b/ooni/director.py @@ -7,13 +7,11 @@ from ooni.utils import log, generate_filename from ooni.utils.net import randomFreePort from ooni.nettest import NetTest, getNetTestInformation from ooni.settings import config -from ooni import errors from ooni.nettest import test_class_name_to_name -from txtorcon import TorConfig, TorState, launch_tor, build_tor_connection +from ooni.utils.onion import start_tor, connect_to_control_port -from twisted.internet import defer, reactor -from twisted.internet.endpoints import TCP4ClientEndpoint +from twisted.internet import defer class Director(object): @@ -133,8 +131,7 @@ class Director(object): if config.advanced.start_tor and config.tor_state is None: yield self.startTor() elif config.tor.control_port and config.tor_state is None: - log.msg("Connecting to Tor Control Port...") - yield self.getTorState() + yield connect_to_control_port() if config.global_options['no-geoip']: aux = [False] @@ -299,11 +296,6 @@ class Director(object): config.scapyFactory.registerProtocol(sniffer) log.msg("Starting packet capture to: %s" % filename_pcap) - @defer.inlineCallbacks - def getTorState(self): - connection = TCP4ClientEndpoint(reactor, '127.0.0.1', - config.tor.control_port) - config.tor_state = yield build_tor_connection(connection) def startTor(self): """ Starts Tor @@ -312,37 +304,7 @@ class Director(object): """ log.msg("Starting Tor...") - @defer.inlineCallbacks - def state_complete(state): - config.tor_state = state - log.msg("Successfully bootstrapped Tor") - log.debug("We now have the following circuits: ") - for circuit in state.circuits.values(): - log.debug(" * %s" % circuit) - - socks_port = yield state.protocol.get_conf("SocksPort") - control_port = yield state.protocol.get_conf("ControlPort") - - config.tor.socks_port = int(socks_port.values()[0]) - config.tor.control_port = int(control_port.values()[0]) - - def setup_failed(failure): - log.exception(failure) - raise errors.UnableToStartTor - - def setup_complete(proto): - """ - Called when we read from stdout that Tor has reached 100%. - """ - log.debug("Building a TorState") - config.tor.protocol = proto - state = TorState(proto.tor_protocol) - state.post_bootstrap.addCallback(state_complete) - state.post_bootstrap.addErrback(setup_failed) - return state.post_bootstrap - - def updates(prog, tag, summary): - log.msg("%d%%: %s" % (prog, summary)) + from txtorcon import TorConfig tor_config = TorConfig() if config.tor.control_port is None: @@ -388,14 +350,4 @@ class Director(object): tor_config.save() log.debug("Setting control port as %s" % tor_config.ControlPort) log.debug("Setting SOCKS port as %s" % tor_config.SocksPort) - - if config.advanced.tor_binary: - d = launch_tor(tor_config, reactor, - tor_binary=config.advanced.tor_binary, - progress_updates=updates) - else: - d = launch_tor(tor_config, reactor, - progress_updates=updates) - d.addCallback(setup_complete) - d.addErrback(setup_failed) - return d + return start_tor(tor_config) diff --git a/ooni/oonicli.py b/ooni/oonicli.py index 56b331d..74bf9ac 100644 --- a/ooni/oonicli.py +++ b/ooni/oonicli.py @@ -10,7 +10,8 @@ import urlparse from twisted.python import usage from twisted.internet import defer -from ooni import errors, __version__, canonical_bouncer +from ooni import errors, __version__ +from ooni.constants import CANONICAL_BOUNCER_ONION from ooni.settings import config from ooni.utils import log from backend_client import CollectorClient @@ -44,7 +45,7 @@ class Options(usage.Options): ["collector", "c", None, "Specify the address of the collector for " "test results. In most cases a user will " "prefer to specify a bouncer over this."], - ["bouncer", "b", canonical_bouncer, "Specify the bouncer used to " + ["bouncer", "b", CANONICAL_BOUNCER_ONION, "Specify the bouncer used to " "obtain the address of the " "collector and test helpers."], ["logfile", "l", None, "Write to this logs to this filename."], diff --git a/ooni/report/cli.py b/ooni/report/cli.py index 12ea83c..485ab52 100644 --- a/ooni/report/cli.py +++ b/ooni/report/cli.py @@ -3,7 +3,7 @@ from __future__ import print_function import os import sys -from ooni import canonical_bouncer +from ooni.constants import CANONICAL_BOUNCER_ONION from ooni.report import __version__ from ooni.report import tool from ooni.settings import config @@ -73,7 +73,7 @@ def run(args=sys.argv[1:]): config.read_config_file() if options['default-collector']: - options['bouncer'] = canonical_bouncer + options['bouncer'] = CANONICAL_BOUNCER_ONION if options['command'] == "upload" and options['report_file']: tor_check() diff --git a/ooni/report/tool.py b/ooni/report/tool.py index faa407f..f8af132 100644 --- a/ooni/report/tool.py +++ b/ooni/report/tool.py @@ -4,7 +4,7 @@ import sys from twisted.internet import defer -from ooni import canonical_bouncer +from ooni.constants import CANONICAL_BOUNCER_ONION from ooni.reporter import OONIBReporter, OONIBReportLog from ooni.utils import log @@ -62,7 +62,7 @@ def upload(report_file, collector=None, bouncer=None): log.msg("Could not find %s in reporting.yaml. Looking up " "collector with canonical bouncer." % report_file) collector_client = yield lookup_collector_client(report.header, - canonical_bouncer) + CANONICAL_BOUNCER_ONION) oonib_reporter = OONIBReporter(report.header, collector_client) log.msg("Creating report for %s with %s" % (report_file, diff --git a/ooni/tests/test_director.py b/ooni/tests/test_director.py index d2c4c82..0cc740a 100644 --- a/ooni/tests/test_director.py +++ b/ooni/tests/test_director.py @@ -68,8 +68,8 @@ class TestDirector(ConfigTestCase): assert 'http_header_field_manipulation' in nettests assert 'traceroute' in nettests - @patch('ooni.director.TorState', mock_TorState) - @patch('ooni.director.launch_tor', mock_launch_tor) + @patch('ooni.utils.onion.TorState', mock_TorState) + @patch('ooni.utils.onion.launch_tor', mock_launch_tor) def test_start_tor(self): @defer.inlineCallbacks def director_start_tor(): diff --git a/ooni/tests/test_onion.py b/ooni/tests/test_onion.py index 944f8f5..732614f 100644 --- a/ooni/tests/test_onion.py +++ b/ooni/tests/test_onion.py @@ -1,6 +1,9 @@ +from twisted.internet import defer from twisted.trial import unittest + from ooni.utils import onion from mock import Mock, patch +from txtorcon.interface import ITorControlProtocol sample_transport_lines = { 'fte': 'fte exec /fakebin --managed', @@ -10,14 +13,28 @@ sample_transport_lines = { 'obfs4': 'obfs4 exec /fakebin --enableLogging=true --logLevel=INFO' } +class MockTorState(object): + def __init__(self): + self.protocol = Mock() + self.protocol.get_state = lambda x: 8080 + self.protocol.post_bootstrap = defer.succeed(self) + +class MockSuccessTorProtocol(object): + def __init__(self): + self.tor_protocol = Mock(ITorControlProtocol) + self.tor_protocol.post_bootstrap = defer.succeed(MockTorState()) + class TestOnion(unittest.TestCase): def test_tor_details(self): assert isinstance(onion.tor_details, dict) assert onion.tor_details['version'] assert onion.tor_details['binary'] + def test_transport_dicts(self): - self.assertEqual( set(onion.transport_bin_name.keys()), - set(onion._transport_line_templates.keys()) ) + + self.assertEqual(set(onion.transport_bin_name.keys()), + set(onion._transport_line_templates.keys())) + def test_bridge_line(self): self.assertRaises(onion.UnrecognizedTransport, onion.bridge_line, 'rot13', '/log.txt') @@ -64,3 +81,27 @@ class TestOnion(unittest.TestCase): self.assertEqual(onion.is_onion_address( 'http://thirteenchars123.com'), False) + + def test_launcher_fail_once(self): + from ooni.utils.onion import TorLauncherWithRetries + from txtorcon import TorConfig + tor_config = TorConfig() + tor_launcher = TorLauncherWithRetries(tor_config) + + self.failures = 0 + def _launch_tor_fail_once(): + self.failures += 1 + if self.failures <= 1: + return defer.fail(Exception("Failed once")) + return defer.succeed(MockSuccessTorProtocol()) + + def _mock_setup_complete(protocol): + self.assertIsInstance(protocol, MockSuccessTorProtocol) + self.assertTrue( + tor_launcher.tor_config.ClientTransportPlugin.startswith("obfs4") + ) + tor_launcher.started.callback(None) + + tor_launcher._launch_tor = _launch_tor_fail_once + tor_launcher._setup_complete = _mock_setup_complete + return tor_launcher.launch() diff --git a/ooni/utils/onion.py b/ooni/utils/onion.py index fea75c2..e18a6ee 100644 --- a/ooni/utils/onion.py +++ b/ooni/utils/onion.py @@ -1,16 +1,28 @@ +import os import re import string +import StringIO import subprocess from distutils.spawn import find_executable from distutils.version import LooseVersion +from twisted.internet import reactor, defer +from twisted.internet.endpoints import TCP4ClientEndpoint + +from txtorcon import TorConfig, TorState, launch_tor, build_tor_connection from txtorcon.util import find_tor_binary as tx_find_tor_binary +from ooni import constants +from ooni import errors +from ooni.utils import log from ooni.settings import config ONION_ADDRESS_REGEXP = re.compile("^((httpo|http|https)://)?" "[a-z0-9]{16}\.onion") +TBB_PT_PATHS = ("/Applications/TorBrowser.app/Contents/MacOS/Tor" + "/PluggableTransports/",) + class TorVersion(LooseVersion): pass @@ -64,14 +76,22 @@ def transport_name(address): transport_name_chars = string.ascii_letters + string.digits if all(c in transport_name_chars for c in transport_name): return transport_name - else: - return None + return None def is_onion_address(address): return ONION_ADDRESS_REGEXP.match(address) != None +def find_pt_executable(name): + bin_loc = find_executable(name) + if bin_loc: + return bin_loc + for path in TBB_PT_PATHS: + bin_loc = os.path.join(path, name) + if os.path.isfile(bin_loc): + return bin_loc + return None tor_details = { 'binary': find_tor_binary(), @@ -107,7 +127,9 @@ _transport_line_templates = { _pyobfsproxy_line('obfs3', bin_loc, log_file), 'obfs4': lambda bin_loc, log_file: \ - "obfs4 exec %s --enableLogging=true --logLevel=INFO" % bin_loc } + "obfs4 exec %s --enableLogging=true --logLevel=INFO" % bin_loc, + +} class UnrecognizedTransport(Exception): pass @@ -139,3 +161,167 @@ def bridge_line(transport, log_file): raise OutdatedTor return _transport_line_templates[transport](bin_loc, log_file) + +pt_config = { + 'meek': [ + { + 'executable': 'obfs4proxy', + 'minimum_version': '0.0.6', + 'version_parse': lambda x: x.split('-')[1], + 'client_transport_line': 'meek exec {bin_loc}' + }, + { + 'executable': 'meek-client', + 'minimum_version': None, + 'client_transport_line': 'meek exec {bin_loc}' + } + ], + + 'obfs4': [ + { + 'executable': 'obfs4proxy', + 'minimum_version': None, + 'client_transport_line': 'obfs4 exec {bin_loc}' + } + ] + +} + +def get_client_transport(transport): + """ + + :param transport: + :return: client_transport_line + """ + + try: + pts = pt_config[transport] + except KeyError: + raise UnrecognizedTransport + + for pt in pts: + bin_loc = find_pt_executable(pt['executable']) + if bin_loc is None: + continue + if pt['minimum_version'] is not None: + pt_version = executable_version(bin_loc, pt['version_parse']) + if (pt_version is None or + pt_version < LooseVersion(pt['minimum_version'])): + continue + return pt['client_transport_line'].format(bin_loc=bin_loc) + + raise UninstalledTransport + + +class TorLauncherWithRetries(object): + def __init__(self, tor_config, timeout=config.tor.timeout): + self.retry_with = ["obfs4", "meek"] + self.started = defer.Deferred() + self.tor_output = StringIO.StringIO() + self.tor_config = tor_config + if timeout is None: + # XXX we will want to move setting the default inside of the + # config object. + timeout = 200 + self.timeout = timeout + + def _reset_tor_config(self): + """ + This is used to reset the Tor configuration to before launch_tor + modified it. This is in particular used to force the regeneration of the + DataDirectory. + """ + new_tor_config = TorConfig() + for key in self.tor_config: + if config.tor.data_dir is None and key == "DataDirectory": + continue + setattr(new_tor_config, key, getattr(self.tor_config, key)) + self.tor_config = new_tor_config + + def _progress_updates(self, prog, tag, summary): + log.msg("%d%%: %s" % (prog, summary)) + + @defer.inlineCallbacks + def _state_complete(self, state): + config.tor_state = state + log.msg("Successfully bootstrapped Tor") + log.debug("We now have the following circuits: ") + for circuit in state.circuits.values(): + log.debug(" * %s" % circuit) + + socks_port = yield state.protocol.get_conf("SocksPort") + control_port = yield state.protocol.get_conf("ControlPort") + + config.tor.socks_port = int(socks_port.values()[0]) + config.tor.control_port = int(control_port.values()[0]) + self.started.callback(state) + + def _setup_failed(self, failure): + self.tor_output.seek(0) + map(log.debug, self.tor_output.readlines()) + self.tor_output.seek(0) + + if len(self.retry_with) == 0: + self.started.errback(errors.UnableToStartTor()) + return + + while len(self.retry_with) > 0: + self._reset_tor_config() + self.tor_config.UseBridges = 1 + transport = self.retry_with.pop(0) + log.msg("Failed to start Tor. Retrying with {0}".format(transport)) + + try: + bridge_lines = getattr(constants, + '{0}_BRIDGES'.format(transport).upper()) + except AttributeError: + continue + + try: + self.tor_config.ClientTransportPlugin = get_client_transport(transport) + except UninstalledTransport: + log.err("Pluggable transport {0} is not installed".format( + transport)) + continue + except UnrecognizedTransport: + log.err("Unrecognized transport type") + continue + + self.tor_config.Bridge = bridge_lines + self.launch() + break + + def _setup_complete(self, proto): + """ + Called when we read from stdout that Tor has reached 100%. + """ + log.debug("Building a TorState") + config.tor.protocol = proto + state = TorState(proto.tor_protocol) + state.post_bootstrap.addCallbacks(self._state_complete, + self._setup_failed) + + def _launch_tor(self): + return launch_tor(self.tor_config, reactor, + tor_binary=config.advanced.tor_binary, + progress_updates=self._progress_updates, + stdout=self.tor_output, + timeout=self.timeout, + stderr=self.tor_output) + + def launch(self): + self._launched = self._launch_tor() + self._launched.addCallbacks(self._setup_complete, self._setup_failed) + return self.started + + +def start_tor(tor_config): + tor_launcher = TorLauncherWithRetries(tor_config) + return tor_launcher.launch() + + +(a)defer.inlineCallbacks +def connect_to_control_port(): + connection = TCP4ClientEndpoint(reactor, '127.0.0.1', + config.tor.control_port) + config.tor_state = yield build_tor_connection(connection)

1 0

[ooni-probe/master] Add support for specifying the test name without the category prefix. (#532)
by art＠torproject.org 10 Jul '16

10 Jul '16

commit 4f169919d41e60fd91709e8e7a3ca351c7945870 Author: Arturo Filastò <arturo(a)filasto.net> Date: Tue Jun 21 18:38:14 2016 +0200 Add support for specifying the test name without the category prefix. (#532) * Add support for specifying the test name without the category prefix. This implements: https://github.com/TheTorProject/ooni-probe/issues/483 * Remove redundant code --- ooni/deck.py | 25 +++++++++++++++++++++---- ooni/tests/test_deck.py | 11 ++++++++++- 2 files changed, 31 insertions(+), 5 deletions(-) diff --git a/ooni/deck.py b/ooni/deck.py index 657589d..82f98a1 100644 --- a/ooni/deck.py +++ b/ooni/deck.py @@ -73,6 +73,9 @@ def nettest_to_path(path, allow_arbitrary_paths=False): """ Takes as input either a path or a nettest name. + The nettest name may either be prefixed by the category of the nettest ( + blocking, experimental, manipulation or third_party) or not. + Args: allow_arbitrary_paths: @@ -85,11 +88,25 @@ def nettest_to_path(path, allow_arbitrary_paths=False): if allow_arbitrary_paths and os.path.exists(path): return path - fp = FilePath(config.nettest_directory).preauthChild(path + '.py') - if fp.exists(): - return fp.path - else: + test_name = path.rsplit("/", 1)[-1] + test_categories = [ + "blocking", + "experimental", + "manipulation", + "third_party" + ] + nettest_dir = FilePath(config.nettest_directory) + found_path = None + for category in test_categories: + p = nettest_dir.preauthChild(os.path.join(category, test_name) + '.py') + if p.exists(): + if found_path is not None: + raise Exception("Found two tests named %s" % test_name) + found_path = p.path + + if not found_path: raise e.NetTestNotFound(path) + return found_path class Deck(InputFile): diff --git a/ooni/tests/test_deck.py b/ooni/tests/test_deck.py index 1e311cf..e4f661f 100644 --- a/ooni/tests/test_deck.py +++ b/ooni/tests/test_deck.py @@ -4,7 +4,8 @@ from twisted.internet import defer from twisted.trial import unittest from hashlib import sha256 -from ooni.deck import InputFile, Deck +from ooni import errors +from ooni.deck import InputFile, Deck, nettest_to_path from ooni.tests.mocks import MockBouncerClient, MockCollectorClient net_test_string = """ @@ -202,3 +203,11 @@ class TestDeck(BaseTestCase): deck.netTestLoaders[1].localOptions['backend'], '2.2.2.2' ) + + def test_nettest_to_path(self): + path_a = nettest_to_path("blocking/http_requests") + path_b = nettest_to_path("http_requests") + self.assertEqual(path_a, path_b) + self.assertRaises(errors.NetTestNotFound, + nettest_to_path, + "invalid_test")

1 0

[ooni-probe/master] Add a configuration option that allows to prioritise non-onion backends
by art＠torproject.org 10 Jul '16

10 Jul '16

commit 9d52e351e79521175ce2e91e5b00dbe72d8f1814 Author: Arturo Filastò <arturo(a)filasto.net> Date: Thu Jun 16 20:03:57 2016 +0300 Add a configuration option that allows to prioritise non-onion backends This option allows a user to specify that they wish to use either a https, cloudfronted or onion backend server. Write unittests to use the priority address --- data/ooniprobe.conf.sample | 3 ++ ooni/backend_client.py | 26 +++++------ ooni/constants.py | 5 +++ ooni/deck.py | 109 ++++++++++++++++++++++++++------------------- ooni/errors.py | 4 ++ ooni/oonicli.py | 11 ++--- ooni/tests/bases.py | 10 +++-- ooni/tests/mocks.py | 9 ++++ ooni/tests/test_deck.py | 70 ++++++++++++++++++++++++++--- 9 files changed, 175 insertions(+), 72 deletions(-) diff --git a/data/ooniprobe.conf.sample b/data/ooniprobe.conf.sample index be35364..e310ddc 100644 --- a/data/ooniprobe.conf.sample +++ b/data/ooniprobe.conf.sample @@ -48,7 +48,10 @@ advanced: report_log_file: null inputs_dir: null decks_dir: null + # If we should support communicating to plaintext backends (via HTTP) insecure_backend: false + # The preferred backend type, can be one of onion, https or cloudfront + preferred_backend: onion tor: #socks_port: 8801 #control_port: 8802 diff --git a/ooni/backend_client.py b/ooni/backend_client.py index 0e85dd7..71b774f 100644 --- a/ooni/backend_client.py +++ b/ooni/backend_client.py @@ -19,6 +19,18 @@ from ooni.utils.net import BodyReceiver, StringProducer, Downloader from ooni.utils.socks import TrueHeadersSOCKS5Agent +def guess_backend_type(address): + if address is None: + raise e.InvalidAddress + if onion.is_onion_address(address): + return 'onion' + elif address.startswith('https://'): + return 'https' + elif address.startswith('http://'): + return 'http' + else: + raise e.InvalidAddress + class OONIBClient(object): def __init__(self, address=None, settings={}): self.base_headers = {} @@ -26,7 +38,7 @@ class OONIBClient(object): self.base_address = settings.get('address', address) if self.backend_type is None: - self._guessBackendType() + self.backend_type = guess_backend_type(self.base_address) self.backend_type = self.backend_type.encode('ascii') if self.backend_type == 'cloudfront': @@ -39,18 +51,6 @@ class OONIBClient(object): 'front': settings.get('front', '').encode('ascii') } - def _guessBackendType(self): - if self.base_address is None: - raise e.InvalidAddress - if onion.is_onion_address(self.base_address): - self.backend_type = 'onion' - elif self.base_address.startswith('https://'): - self.backend_type = 'https' - elif self.base_address.startswith('http://'): - self.backend_type = 'http' - else: - raise e.InvalidAddress - def _setupBaseAddress(self): parsed_address = urlparse(self.base_address) if self.backend_type == 'onion': diff --git a/ooni/constants.py b/ooni/constants.py index 882edb4..ef1f3d7 100644 --- a/ooni/constants.py +++ b/ooni/constants.py @@ -1,4 +1,9 @@ CANONICAL_BOUNCER_ONION = 'httpo://nkvphnp3p6agi5qq.onion' +CANONICAL_BOUNCER_HTTPS = 'https://bouncer.ooni.io' +CANONICAL_BOUNCER_CLOUDFRONT = ( + 'XXXX.cloudfront.net', + 'https://a0.awsstatic.com/' +) MEEK_BRIDGES = [ ("meek 0.0.2.0:2 B9E7141C594AF25699E0079C1F0146F409495296 " diff --git a/ooni/deck.py b/ooni/deck.py index 82f98a1..2aa958d 100644 --- a/ooni/deck.py +++ b/ooni/deck.py @@ -1,10 +1,11 @@ # -*- coding: utf-8 -*- from ooni.backend_client import CollectorClient, BouncerClient -from ooni.backend_client import WebConnectivityClient +from ooni.backend_client import WebConnectivityClient, guess_backend_type from ooni.nettest import NetTestLoader from ooni.settings import config from ooni.utils import log, onion +from ooni import constants from ooni import errors as e from twisted.python.filepath import FilePath @@ -120,7 +121,29 @@ class Deck(InputFile): no_collector=False): self.id = deck_hash self.no_collector = no_collector - self.bouncer = bouncer + + self.preferred_backend = config.advanced.get( + "preferred_backend", "onion" + ) + if self.preferred_backend not in ["onion", "https", "cloudfront"]: + raise e.InvalidPreferredBackend + + if bouncer is None: + bouncer_address = getattr( + constants, "CANONICAL_BOUNCER_{0}".format( + self.preferred_backend.upper() + ) + ) + if self.preferred_backend == "cloudfront": + self.bouncer = self._BouncerClient(settings={ + 'address': bouncer_address[0], + 'front': bouncer_address[1], + 'type': 'cloudfront' + }) + else: + self.bouncer = self._BouncerClient(bouncer_address) + else: + self.bouncer = self._BouncerClient(bouncer) self.requiresTor = False @@ -167,19 +190,23 @@ class Deck(InputFile): collector_address ) if test['options'].get('bouncer', None) is not None: - self.bouncer = test['options']['bouncer'] + self.bouncer = self._BouncerClient(test['options']['bouncer']) + if self.bouncer.backend_type is "onion": + self.requiresTor = True self.insert(net_test_loader) def insert(self, net_test_loader): """ Add a NetTestLoader to this test deck """ + if (net_test_loader.collector is not None + and net_test_loader.collector.backend_type is "onion"): + self.requiresTor = True try: net_test_loader.checkOptions() if net_test_loader.requiresTor: self.requiresTor = True except e.MissingTestHelper: - if not self.bouncer: - raise - self.requiresTor = True + if self.preferred_backend is "onion": + self.requiresTor = True self.netTestLoaders.append(net_test_loader) @@ -188,6 +215,7 @@ class Deck(InputFile): """ fetch and verify inputs for all NetTests in the deck """ log.msg("Fetching required net test inputs...") for net_test_loader in self.netTestLoaders: + # XXX figure out if we want to keep this or drop this. yield self.fetchAndVerifyNetTestInput(net_test_loader) if self.bouncer: @@ -196,44 +224,34 @@ class Deck(InputFile): def sortAddressesByPriority(self, priority_address, alternate_addresses): - onion_addresses= [] - cloudfront_addresses= [] - https_addresses = [] - plaintext_addresses = [] - - if onion.is_onion_address(priority_address): - priority_address = { - 'address': priority_address, - 'type': 'onion' - } - elif priority_address.startswith('https://'): - priority_address = { - 'address': priority_address, - 'type': 'https' - } - elif priority_address.startswith('http://'): - priority_address = { - 'address': priority_address, - 'type': 'http' - } - else: - raise e.InvalidOONIBCollectorAddress + prioritised_addresses = [] + + backend_type = guess_backend_type(priority_address) + priority_address = { + 'address': priority_address, + 'type': backend_type + } + address_priority = ['onion', 'https', 'cloudfront', 'http'] + address_priority.remove(self.preferred_backend) + address_priority.insert(0, self.preferred_backend) def filter_by_type(collectors, collector_type): - return filter(lambda x: x['type'] == collector_type, - collectors) - onion_addresses += filter_by_type(alternate_addresses, 'onion') - https_addresses += filter_by_type(alternate_addresses, 'https') - cloudfront_addresses += filter_by_type(alternate_addresses, - 'cloudfront') + return filter(lambda x: x['type'] == collector_type, collectors) + + if (priority_address['type'] != self.preferred_backend): + valid_alternatives = filter_by_type(alternate_addresses, + self.preferred_backend) + if len(valid_alternatives) > 0: + alternate_addresses += [priority_address] + priority_address = valid_alternatives[0] + alternate_addresses.remove(priority_address) - plaintext_addresses += filter_by_type(alternate_addresses, 'http') + prioritised_addresses += [priority_address] + for address_type in address_priority: + prioritised_addresses += filter_by_type(alternate_addresses, + address_type) - return ([priority_address] + - onion_addresses + - https_addresses + - cloudfront_addresses + - plaintext_addresses) + return prioritised_addresses @defer.inlineCallbacks def getReachableCollector(self, collector_address, collector_alternate): @@ -289,9 +307,12 @@ class Deck(InputFile): @defer.inlineCallbacks def getReachableTestHelpersAndCollectors(self, net_tests): for net_test in net_tests: + + primary_address = net_test['collector'] + alternate_addresses = net_test.get('collector-alternate', []) net_test['collector'] = yield self.getReachableCollector( - net_test['collector'], - net_test.get('collector-alternate', []) + primary_address, + alternate_addresses ) for test_helper_name, test_helper_address in net_test['test-helpers'].items(): @@ -307,8 +328,6 @@ class Deck(InputFile): @defer.inlineCallbacks def lookupCollectorAndTestHelpers(self): - oonibclient = self._BouncerClient(self.bouncer) - required_nettests = [] requires_test_helpers = False @@ -333,7 +352,7 @@ class Deck(InputFile): if not requires_test_helpers and not requires_collector: defer.returnValue(None) - response = yield oonibclient.lookupTestCollector(required_nettests) + response = yield self.bouncer.lookupTestCollector(required_nettests) try: provided_net_tests = yield self.getReachableTestHelpersAndCollectors(response['net-tests']) except e.NoReachableCollectors: diff --git a/ooni/errors.py b/ooni/errors.py index 8197f34..faa3627 100644 --- a/ooni/errors.py +++ b/ooni/errors.py @@ -316,3 +316,7 @@ class NoReachableCollectors(Exception): class NoReachableTestHelpers(Exception): pass + + +class InvalidPreferredBackend(Exception): + pass diff --git a/ooni/oonicli.py b/ooni/oonicli.py index 74bf9ac..43654b6 100644 --- a/ooni/oonicli.py +++ b/ooni/oonicli.py @@ -45,9 +45,9 @@ class Options(usage.Options): ["collector", "c", None, "Specify the address of the collector for " "test results. In most cases a user will " "prefer to specify a bouncer over this."], - ["bouncer", "b", CANONICAL_BOUNCER_ONION, "Specify the bouncer used to " - "obtain the address of the " - "collector and test helpers."], + ["bouncer", "b", None, "Specify the bouncer used to " + "obtain the address of the " + "collector and test helpers."], ["logfile", "l", None, "Write to this logs to this filename."], ["pcapfile", "O", None, "Write a PCAP of the ooniprobe session to " "this filename."], @@ -388,10 +388,11 @@ def runWithDirector(global_options): log.msg("Not reporting using a collector") global_options['collector'] = None start_tor = False - else: + elif config.advanced.get("preferred_backend", "onion") == "onion": start_tor = True - if global_options['collector']: + if (global_options['collector'] and + config.advanced.get("preferred_backend", "onion") == "onion"): start_tor |= True return runTestWithDirector(director=director, diff --git a/ooni/tests/bases.py b/ooni/tests/bases.py index 5d177fa..31cbf94 100644 --- a/ooni/tests/bases.py +++ b/ooni/tests/bases.py @@ -1,3 +1,5 @@ +import os +import shutil from twisted.trial import unittest from ooni.settings import config @@ -5,11 +7,13 @@ from ooni.settings import config class ConfigTestCase(unittest.TestCase): def setUp(self): - config.initialize_ooni_home("ooni_home") + self.ooni_home_dir = os.path.abspath("ooni_home") + self.config = config + self.config.initialize_ooni_home("ooni_home") + super(ConfigTestCase, self).setUp() def skipTest(self, reason): raise unittest.SkipTest(reason) def tearDown(self): - config.set_paths() - config.read_config_file() + shutil.rmtree("ooni_home") diff --git a/ooni/tests/mocks.py b/ooni/tests/mocks.py index db7b154..a377fba 100644 --- a/ooni/tests/mocks.py +++ b/ooni/tests/mocks.py @@ -222,6 +222,15 @@ class MockBouncerClient(object): 'version': net_test['version'], 'input-hashes': net_test['input-hashes'], 'collector': 'httpo://thirteenchars123.onion', + 'collector-alternate': [ + {'type': 'https', 'address': 'https://collector.ooni.io'}, + {'type': 'http', 'address': 'http://collector.ooni.io'}, + { + 'type': 'cloudfront', + 'address': 'https://address.cloudfront.net', + 'front': 'https://front.cloudfront.net' + }, + ], 'test-helpers': test_helpers }) return defer.succeed(ret) diff --git a/ooni/tests/test_deck.py b/ooni/tests/test_deck.py index e4f661f..f299e70 100644 --- a/ooni/tests/test_deck.py +++ b/ooni/tests/test_deck.py @@ -6,8 +6,11 @@ from twisted.trial import unittest from hashlib import sha256 from ooni import errors from ooni.deck import InputFile, Deck, nettest_to_path +from ooni.tests.bases import ConfigTestCase from ooni.tests.mocks import MockBouncerClient, MockCollectorClient +FAKE_BOUNCER_ADDRESS = "httpo://thirteenchars123.onion" + net_test_string = """ from twisted.python import usage from ooni.nettest import NetTestCase @@ -71,7 +74,7 @@ class BaseTestCase(unittest.TestCase): test_file: manipulation/http_invalid_request_line testdeck: null """ - + super(BaseTestCase, self).setUp() class TestInputFile(BaseTestCase): @@ -112,7 +115,7 @@ class TestInputFile(BaseTestCase): assert input_file.descriptorCached -class TestDeck(BaseTestCase): +class TestDeck(BaseTestCase, ConfigTestCase): def setUp(self): super(TestDeck, self).setUp() deck_hash = sha256(self.dummy_deck_content).hexdigest() @@ -127,9 +130,10 @@ class TestDeck(BaseTestCase): os.remove(self.deck_file) if self.filename != "": os.remove(self.filename) + super(TestDeck, self).tearDown() def test_open_deck(self): - deck = Deck(bouncer="httpo://foo.onion", + deck = Deck(bouncer=FAKE_BOUNCER_ADDRESS, decks_directory=".") deck.loadDeck(self.deck_file) assert len(deck.netTestLoaders) == 1 @@ -139,7 +143,7 @@ class TestDeck(BaseTestCase): "annotations": {"spam": "ham"}, "collector": "httpo://thirteenchars123.onion" } - deck = Deck(bouncer="httpo://foo.onion", + deck = Deck(bouncer=FAKE_BOUNCER_ADDRESS, decks_directory=".") deck.loadDeck(self.deck_file, global_options=global_options) @@ -153,7 +157,7 @@ class TestDeck(BaseTestCase): ) def test_save_deck_descriptor(self): - deck = Deck(bouncer="httpo://foo.onion", + deck = Deck(bouncer=FAKE_BOUNCER_ADDRESS, decks_directory=".") deck.loadDeck(self.deck_file) deck.load({'name': 'spam', @@ -169,8 +173,9 @@ class TestDeck(BaseTestCase): @defer.inlineCallbacks def test_lookup_test_helpers_and_collector(self): - deck = Deck(bouncer="httpo://foo.onion", + deck = Deck(bouncer=FAKE_BOUNCER_ADDRESS, decks_directory=".") + deck.bouncer = MockBouncerClient(FAKE_BOUNCER_ADDRESS) deck._BouncerClient = MockBouncerClient deck._CollectorClient = MockCollectorClient deck.loadDeck(self.deck_file) @@ -211,3 +216,56 @@ class TestDeck(BaseTestCase): self.assertRaises(errors.NetTestNotFound, nettest_to_path, "invalid_test") + + @defer.inlineCallbacks + def test_lookup_test_helpers_and_collector_cloudfront(self): + self.config.advanced.preferred_backend = "cloudfront" + deck = Deck(bouncer=FAKE_BOUNCER_ADDRESS, + decks_directory=".") + deck.bouncer = MockBouncerClient(FAKE_BOUNCER_ADDRESS) + deck._BouncerClient = MockBouncerClient + deck._CollectorClient = MockCollectorClient + deck.loadDeck(self.deck_file) + + self.assertEqual(len(deck.netTestLoaders[0].missingTestHelpers), 1) + + yield deck.lookupCollectorAndTestHelpers() + + self.assertEqual( + deck.netTestLoaders[0].collector.settings['address'], + 'https://address.cloudfront.net' + ) + self.assertEqual( + deck.netTestLoaders[0].collector.settings['front'], + 'https://front.cloudfront.net' + ) + + self.assertEqual( + deck.netTestLoaders[0].localOptions['backend'], + '127.0.0.1' + ) + + + @defer.inlineCallbacks + def test_lookup_test_helpers_and_collector_https(self): + self.config.advanced.preferred_backend = "https" + deck = Deck(bouncer=FAKE_BOUNCER_ADDRESS, + decks_directory=".") + deck.bouncer = MockBouncerClient(FAKE_BOUNCER_ADDRESS) + deck._BouncerClient = MockBouncerClient + deck._CollectorClient = MockCollectorClient + deck.loadDeck(self.deck_file) + + self.assertEqual(len(deck.netTestLoaders[0].missingTestHelpers), 1) + + yield deck.lookupCollectorAndTestHelpers() + + self.assertEqual( + deck.netTestLoaders[0].collector.settings['address'], + 'https://collector.ooni.io' + ) + + self.assertEqual( + deck.netTestLoaders[0].localOptions['backend'], + '127.0.0.1' + )

1 0