- tor-commits - lists.torproject.org

History had been rewritten: OONI-probe git repo rewind.
by Arturo Filastò 05 Apr '12

05 Apr '12

I wrongly merged a patch that contained a huge file (i.e. 22 MB) and to avoid the pain of having people download a 20 MB git repository I have had to rewrite the git history to exclude that file. Anybody that has cloned or pulled after commit 15c33719cf6dfea1b4482a9436d83820d0dfbf88 (yesterday) should clone a fresh copy of ooni-probe. Sorry for the inconvenience. - Art.

1 0

[ooni-probe/master] Updated TODO file
by art＠torproject.org 05 Apr '12

05 Apr '12

commit 69f731dcd501b6fb5044a63ff75e49e68d6cc937 Author: Isis Lovecruft <isis(a)patternsinthevoid.net> Date: Sat Mar 24 01:31:40 2012 -0700 Updated TODO file --- TODO | 5 +++++ 1 files changed, 5 insertions(+), 0 deletions(-) diff --git a/TODO b/TODO index 9c2cd0a..9ecae4b 100644 --- a/TODO +++ b/TODO @@ -21,6 +21,11 @@ effort: low, skill: low * All the DNS censorship detection tests (old/ooni/dnsooni.py) effort: medium, skill: medium + - The dnstamper test fulfills this requirement, although + there are several improvements which could be made, namely + figuring out if there is a pattern to the reverse resolved + names and parsing them in some manner to determine if that + server is authorized to serve the requested page. * Misc scripts to detect censorship that are not written in any version of OONI (old/ooni/plugins/old_stuff_to_convert/)

1 0

[ooni-probe/master] Removed nested if statements and pointed to crate.io for import exception
by art＠torproject.org 05 Apr '12

05 Apr '12

commit eac8bfad5142259a70ecfedd49750739bb41cbcc Author: Isis Lovecruft <isis(a)patternsinthevoid.net> Date: Fri Mar 23 20:16:44 2012 -0700 Removed nested if statements and pointed to crate.io for import exception --- plugoo/tests.py | 5 ++--- tests/httphost.py | 2 +- 2 files changed, 3 insertions(+), 4 deletions(-) diff --git a/plugoo/tests.py b/plugoo/tests.py index f7537de..f94ba04 100644 --- a/plugoo/tests.py +++ b/plugoo/tests.py @@ -78,9 +78,8 @@ class Test: if assets: self.logger.debug("Running through tests") - if extradata is not None: - if extradata['index']: - index = extradata['index'] + if extradata and extradata['index']: + index = extradata['index'] else: index = None diff --git a/tests/httphost.py b/tests/httphost.py index 0f33ca7..25adff2 100644 --- a/tests/httphost.py +++ b/tests/httphost.py @@ -23,7 +23,7 @@ monkey.patch_socket() try: from BeautifulSoup import BeautifulSoup except: - print "BeautifulSoup-3.2.1 is missing. Try doing: # easy_install beautifulsoup" + print "BeautifulSoup-3.2.1 is missing. Please see https://crate.io/packages/BeautifulSoup/" from plugoo.assets import Asset from plugoo.tests import Test

1 0

[ooni-probe/master] Fixed invalid call on NoneType object in plugoo/tests.py, added exception catcher for import statement on BeautifulSoup module in tests/httphost.py
by art＠torproject.org 05 Apr '12

05 Apr '12

commit 893fe90bffa55a90e332d748a186d44906227204 Author: Isis Lovecruft <isis(a)patternsinthevoid.net> Date: Fri Mar 23 15:23:40 2012 -0700 Fixed invalid call on NoneType object in plugoo/tests.py, added exception catcher for import statement on BeautifulSoup module in tests/httphost.py --- plugoo/tests.py | 5 +++-- tests/httphost.py | 5 ++++- 2 files changed, 7 insertions(+), 3 deletions(-) diff --git a/plugoo/tests.py b/plugoo/tests.py index 3d6b36b..f7537de 100644 --- a/plugoo/tests.py +++ b/plugoo/tests.py @@ -78,8 +78,9 @@ class Test: if assets: self.logger.debug("Running through tests") - if extradata['index']: - index = extradata['index'] + if extradata is not None: + if extradata['index']: + index = extradata['index'] else: index = None diff --git a/tests/httphost.py b/tests/httphost.py index 6446e1f..0f33ca7 100644 --- a/tests/httphost.py +++ b/tests/httphost.py @@ -20,7 +20,10 @@ import httplib # appearing on the wire. monkey.patch_socket() -from BeautifulSoup import BeautifulSoup +try: + from BeautifulSoup import BeautifulSoup +except: + print "BeautifulSoup-3.2.1 is missing. Try doing: # easy_install beautifulsoup" from plugoo.assets import Asset from plugoo.tests import Test

1 0

[ooni-probe/master] Apply Isis' patch for reverse DNS lookups
by art＠torproject.org 05 Apr '12

05 Apr '12

commit 915495eb20aa9b356192e3f9f76e82e2537fe696 Author: Arturo Filastò <hellais(a)gmail.com> Date: Thu Apr 5 13:14:54 2012 -0700 Apply Isis' patch for reverse DNS lookups --- HACKING | 2 +- ooni-probe.conf | 7 +++- plugoo/reports.py | 2 +- tests/dnstamper.py | 122 ++++++++++++++++++++++++++++++++++++++++++++-------- 4 files changed, 112 insertions(+), 21 deletions(-) diff --git a/HACKING b/HACKING index ed8d5dc..2530aeb 100644 --- a/HACKING +++ b/HACKING @@ -170,7 +170,7 @@ Method definitions inside of class are separated by a single blank line. Encoding ........ -Always use UTF-8 encoding. This can be specified by add the encoding cookie +Always use UTF-8 encoding. This can be specified by adding the encoding cookie to the beginning of your python files: # -*- coding: UTF-8 diff --git a/ooni-probe.conf b/ooni-probe.conf index 8702353..161a008 100644 --- a/ooni-probe.conf +++ b/ooni-probe.conf @@ -31,7 +31,12 @@ dns_experiment = top-1m.txt dns_experiment_dns = dns_servers.txt # This is the control known good DNS server -dns_control_server = 8.8.8.8 +dns_control_server = 91.191.136.152 + +# Specify whether the dnstamper test should attempt to remove +# GeoIP-based false positives by doing a reverse DNS resolve +# on positive results. +dns_reverse_lookup = true ### traceroute testing related config parameters diff --git a/plugoo/reports.py b/plugoo/reports.py index 90de498..009e86b 100644 --- a/plugoo/reports.py +++ b/plugoo/reports.py @@ -42,7 +42,7 @@ class Report: import paramiko except: self.scp = None - self.logger.warn("Could not import paramiko. SCP will not be disabled") + self.logger.warn("Could not import paramiko. SCP will be disabled") def __call__(self, data): """ diff --git a/tests/dnstamper.py b/tests/dnstamper.py index 68be12d..498ba04 100644 --- a/tests/dnstamper.py +++ b/tests/dnstamper.py @@ -1,23 +1,69 @@ +# -*- coding: utf-8 -*- +""" + dnstamper + ********* + + This test resolves DNS for a list of domain names, one per line, in the + file specified in the ooni-config under the setting "dns_experiment". If + the file is top-1m.txt, the test will be run using Amazon's list of top + one million domains. The experimental dns servers to query should + be specified one per line in assets/dns_servers.txt. + + The test reports censorship if the cardinality of the intersection of + the query result set from the control server and the query result set + from the experimental server is zero, which is to say, if the two sets + have no matching results whatsoever. + + NOTE: This test frequently results in false positives due to GeoIP-based + load balancing on major global sites such as google, facebook, and + youtube, etc. + + :copyright: (c) 2012 Arturo Filastò, Isis Lovecruft + :license: see LICENSE for more details +""" + +try: + from dns import resolver, reversename +except: + print "Error: dnspython is not installed! (http://www.dnspython.org/)" try: - from dns import resolver + import gevent except: - print "Error dnspython is not installed! (http://www.dnspython.org/)" -import gevent + print "Error: gevent is not installed! (http://www.gevent.org/)" + import os + import plugoo from plugoo.assets import Asset from plugoo.tests import Test - __plugoo__ = "DNST" __desc__ = "DNS censorship detection test" +class Top1MAsset(Asset): + """ + Class for parsing top-1m.txt as an asset. + """ + def __init__(self, file=None): + self = Asset.__init__(self, file) + + def parse_line(self, line): + self = Asset.parse_line(self, line) + return line.split(',')[1].replace('\n','') + class DNSTAsset(Asset): + """ + Creates DNS testing specific Assets. + """ def __init__(self, file=None): - self = asset.__init__(self, file) + self = Asset.__init__(self, file) class DNST(Test): def lookup(self, hostname, ns): + """ + Resolves a hostname through a DNS nameserver, ns, to the corresponding + IP address(es). + """ res = resolver.Resolver(configure=False) res.nameservers = [ns] answer = res.query(hostname) @@ -29,42 +75,82 @@ class DNST(Test): return ret + def reverse_lookup(self, ip, ns): + """ + Attempt to do a reverse DNS lookup to determine if the control and exp + sets from a positive result resolve to the same domain, in order to + remove false positives due to GeoIP load balancing. + """ + res = resolver.Resolver(configure=False) + res.nameservers = [ns] + n = reversename.from_address(ip) + revn = res.query(n, "PTR").__iter__().next().to_text()[:-1] + + return revn + def experiment(self, *a, **kw): + """ + Compares the lookup() sets of the control and experiment groups. + """ # this is just a dirty hack address = kw['data'][0] ns = kw['data'][1] config = self.config + ctrl_ns = config.tests.dns_control_server print "ADDRESS: %s" % address print "NAMESERVER: %s" % ns exp = self.lookup(address, ns) - control = self.lookup(address, config.tests.dns_control_server) + control = self.lookup(address, ctrl_ns) + + result = [] if len(set(exp) & set(control)) > 0: - print "%s : no tampering on %s" % (address, ns) - return (address, ns, False) + print "Address %s has not tampered with on DNS server %s\n" % (address, ns) + result = (address, ns, exp, control, False) + return result else: - print "%s : possible tampering on %s (%s, %s)" % (address, ns, exp, control) - return (address, ns, exp, control, True) + print "Address %s has possibly been tampered on %s:\nDNS resolution through %s yeilds:\n%s\nAlthough the control group DNS servers resolve to:\n%s" % (address, ns, ns, exp, control) + result = (address, ns, exp, control, True) + + if config.tests.dns_reverse_lookup: + + exprevn = [self.reverse_lookup(ip, ns) for ip in exp] + ctrlrevn = [self.reverse_lookup(ip, ctrl_ns) + for ip in control] + + if len(set(exprevn) & set(ctrlrevn)) > 0: + print "Further testing has eliminated this as a false positive." + else: + print "Reverse DNS on the results returned by %s returned:\n%s\nWhich does not match the expected domainname:\n%s\n" % (ns, exprevn, ctrlrevn) + return result + + else: + print "\n" + return result def run(ooni): - """Run the test + """ + Run the test. """ config = ooni.config urls = [] - dns_experiment = DNSTAsset(os.path.join(config.main.assetdir, \ - config.tests.dns_experiment)) - dns_experiment_dns = DNSTAsset(os.path.join(config.main.assetdir, \ + if (config.tests.dns_experiment == "top-1m.txt"): + dns_experiment = Top1MAsset(os.path.join(config.main.assetdir, + config.tests.dns_experiment)) + else: + dns_experiment = DNSTAsset(os.path.join(config.main.assetdir, + config.tests.dns_experiment)) + dns_experiment_dns = DNSTAsset(os.path.join(config.main.assetdir, config.tests.dns_experiment_dns)) assets = [dns_experiment, dns_experiment_dns] dnstest = DNST(ooni) - ooni.logger.info("starting test") - dnstest.run(assets) - ooni.logger.info("finished") - + ooni.logger.info("Beginning dnstamper test...") + dnstest.run(assets, {'index': 1}) + ooni.logger.info("Dnstamper test completed!")

1 0

[tor/release-0.2.2] Wrap long line; strlen("ides")<strlen("turtles").
by arma＠torproject.org 05 Apr '12

05 Apr '12

commit 439fc704f14af50364bee5d3cd91a2bca0d8611a Author: Nick Mathewson <nickm(a)torproject.org> Date: Wed Apr 4 21:05:19 2012 -0400 Wrap long line; strlen("ides")<strlen("turtles"). --- src/or/config.c | 3 ++- 1 files changed, 2 insertions(+), 1 deletions(-) diff --git a/src/or/config.c b/src/or/config.c index 681df57..090aba1 100644 --- a/src/or/config.c +++ b/src/or/config.c @@ -809,7 +809,8 @@ add_default_trusted_dir_authorities(authority_type_t type) "194.109.206.212:80 7EA6 EAD6 FD83 083C 538F 4403 8BBF A077 587D D755", "Tonga orport=443 bridge no-v2 82.94.251.203:80 " "4A0C CD2D DC79 9508 3D73 F5D6 6710 0C8A 5831 F16D", - "turtles orport=9090 no-v2 v3ident=27B6B5996C426270A5C95488AA5BCEB6BCC86956 " + "turtles orport=9090 no-v2 " + "v3ident=27B6B5996C426270A5C95488AA5BCEB6BCC86956 " "76.73.17.194:9030 F397 038A DC51 3361 35E7 B80B D99C A384 4360 292B", "gabelmoo orport=443 no-v2 " "v3ident=ED03BB616EB2F60BEC80151114BB25CEF515B226 "

1 0

[tor/release-0.2.2] put a _ before or_options_t elements that aren't configurable
by arma＠torproject.org 05 Apr '12

05 Apr '12

commit 5fed1ccd901d4751a3fed7dc01042771ca76f449 Author: Roger Dingledine <arma(a)torproject.org> Date: Sun Apr 1 15:59:38 2012 -0400 put a _ before or_options_t elements that aren't configurable it's fine with me if we change the current convention, but we should actually decide to change it if we want to. --- src/or/config.c | 6 +++--- src/or/directory.c | 5 +++-- src/or/or.h | 4 ++-- 3 files changed, 8 insertions(+), 7 deletions(-) diff --git a/src/or/config.c b/src/or/config.c index 1e7bd58..fbbd902 100644 --- a/src/or/config.c +++ b/src/or/config.c @@ -713,7 +713,7 @@ or_options_free(or_options_t *options) return; routerset_free(options->_ExcludeExitNodesUnion); - tor_free(options->BridgePassword_AuthDigest); + tor_free(options->_BridgePassword_AuthDigest); config_free(&options_format, options); } @@ -1310,8 +1310,8 @@ options_act(or_options_t *old_options) "BridgePassword."); return -1; } - options->BridgePassword_AuthDigest = tor_malloc(DIGEST256_LEN); - crypto_digest256(options->BridgePassword_AuthDigest, + options->_BridgePassword_AuthDigest = tor_malloc(DIGEST256_LEN); + crypto_digest256(options->_BridgePassword_AuthDigest, http_authenticator, strlen(http_authenticator), DIGEST_SHA256); tor_free(http_authenticator); diff --git a/src/or/directory.c b/src/or/directory.c index 9bc58e5..29cf10c 100644 --- a/src/or/directory.c +++ b/src/or/directory.c @@ -3069,7 +3069,7 @@ directory_handle_command_get(dir_connection_t *conn, const char *headers, } if (options->BridgeAuthoritativeDir && - options->BridgePassword_AuthDigest && + options->_BridgePassword_AuthDigest && connection_dir_is_encrypted(conn) && !strcmp(url,"/tor/networkstatus-bridges")) { char *status; @@ -3081,7 +3081,8 @@ directory_handle_command_get(dir_connection_t *conn, const char *headers, /* now make sure the password is there and right */ if (!header || - tor_memneq(digest, options->BridgePassword_AuthDigest, DIGEST256_LEN)) { + tor_memneq(digest, + options->_BridgePassword_AuthDigest, DIGEST256_LEN)) { write_http_status_line(conn, 404, "Not found"); tor_free(header); goto done; diff --git a/src/or/or.h b/src/or/or.h index 92592e5..a4e8e49 100644 --- a/src/or/or.h +++ b/src/or/or.h @@ -2492,8 +2492,8 @@ typedef struct { * for bridge statuses -- but only if the requests use this password. */ char *BridgePassword; /** If BridgePassword is set, this is a SHA256 digest of the basic http - * authenticator for it. */ - char *BridgePassword_AuthDigest; + * authenticator for it. Used so we can do a time-independent comparison. */ + char *_BridgePassword_AuthDigest; int UseBridges; /**< Boolean: should we start all circuits with a bridge? */ config_line_t *Bridges; /**< List of bootstrap bridge addresses. */

1 0

[tor/release-0.2.2] ides has become turtles, and gotten a new IP address
by arma＠torproject.org 05 Apr '12

05 Apr '12

commit b24487d1068dc09c08a8bde8f84fab9307ba9a27 Author: Sebastian Hahn <sebastian(a)torproject.org> Date: Thu Apr 5 01:51:04 2012 +0200 ides has become turtles, and gotten a new IP address As per ticket 5569 --- changes/ides-becomes-turtles | 4 ++++ src/or/config.c | 4 ++-- 2 files changed, 6 insertions(+), 2 deletions(-) diff --git a/changes/ides-becomes-turtles b/changes/ides-becomes-turtles new file mode 100644 index 0000000..09d964f --- /dev/null +++ b/changes/ides-becomes-turtles @@ -0,0 +1,4 @@ + o Directory authority changes: + - Change IP address for ides (v3 directory authority), and rename it to + turtles. + diff --git a/src/or/config.c b/src/or/config.c index fbbd902..681df57 100644 --- a/src/or/config.c +++ b/src/or/config.c @@ -809,8 +809,8 @@ add_default_trusted_dir_authorities(authority_type_t type) "194.109.206.212:80 7EA6 EAD6 FD83 083C 538F 4403 8BBF A077 587D D755", "Tonga orport=443 bridge no-v2 82.94.251.203:80 " "4A0C CD2D DC79 9508 3D73 F5D6 6710 0C8A 5831 F16D", - "ides orport=9090 no-v2 v3ident=27B6B5996C426270A5C95488AA5BCEB6BCC86956 " - "216.224.124.114:9030 F397 038A DC51 3361 35E7 B80B D99C A384 4360 292B", + "turtles orport=9090 no-v2 v3ident=27B6B5996C426270A5C95488AA5BCEB6BCC86956 " + "76.73.17.194:9030 F397 038A DC51 3361 35E7 B80B D99C A384 4360 292B", "gabelmoo orport=443 no-v2 " "v3ident=ED03BB616EB2F60BEC80151114BB25CEF515B226 " "212.112.245.170:80 F204 4413 DAC2 E02E 3D6B CF47 35A1 9BCA 1DE9 7281",

1 0

[tor/release-0.2.2] Merge branch 'maint-0.2.2' into release-0.2.2
by arma＠torproject.org 05 Apr '12

05 Apr '12

1 0

[tor/release-0.2.2] Merge remote-tracking branch 'karsten/geoip-march2012' into maint-0.2.2
by arma＠torproject.org 05 Apr '12

05 Apr '12

commit 9d5d3a7fd45265358069f0077857da1d7afaad61 Merge: e217569 c5d7ee7 Author: Nick Mathewson <nickm(a)torproject.org> Date: Thu Mar 8 10:50:03 2012 -0500 Merge remote-tracking branch 'karsten/geoip-march2012' into maint-0.2.2 changes/geoip-march2012 | 3 + src/config/geoip |26743 +++++++++++++++++++++++++++++++++++------------ 2 files changed, 19971 insertions(+), 6775 deletions(-)

1 0