[tor-commits] [ooni-probe/master] Merge branch 'master' into dnstamper-dnsconsistency
isis at torproject.org
isis at torproject.org
Sun Feb 24 19:10:38 UTC 2013
commit 70ecad5372fabadb5ca3d66c5c4a2b4835542921
Merge: 9dbe984 bcb2794
Author: Arturo Filastò <art at fuffa.org>
Date: Thu Feb 7 14:18:54 2013 +0100
Merge branch 'master' into dnstamper-dnsconsistency
* master:
Fix typo in dnstamper test example docs
Properly handle errors when DNS lookups
Fix typo in test writing documentation
Conflicts:
docs/source/tests/dnsconsistency.rst
docs/source/tests/dnsconsistency.rst | 2 +-
docs/source/writing_tests.rst | 4 ++--
nettests/blocking/dnsconsistency.py | 9 ++++++++-
3 files changed, 11 insertions(+), 4 deletions(-)
diff --cc docs/source/tests/dnsconsistency.rst
index 8a413bf,0000000..673661c
mode 100644,000000..100644
--- a/docs/source/tests/dnsconsistency.rst
+++ b/docs/source/tests/dnsconsistency.rst
@@@ -1,180 -1,0 +1,180 @@@
+Details
+=======
+
+*Test Name*: DNS Consistency (Ex DNS Tamper)
+
+*Current version*: 0.4
+
+*NetTest*: DNS Consistency Test (https://gitweb.torproject.org/ooni-probe.git/blob/HEAD:/nettests/blocking/dnsconsistency.py)
+
+*Test Helper*: DNS Test Helper (https://gitweb.torproject.org/ooni-probe.git/blob/HEAD:/oonib/testhelpers/dns_helpers.py)
+
+*Test Type*: Content Blocking
+
+*Requires Root*: No
+
+Description
+===========
+
+This test performs A queries to a set of test resolvers and a known good
+control resolver. If the two results do not match it will perform a reverse DNS
+lookup on the first A record address of both sets and check if they both
+resolve to the same name.
+
+NOTE: This test frequently results in false positives due to GeoIP-based
+load balancing on major global sites such as Google, Facebook, and
+Youtube, etc.
+
+How to run the test
+===================
+
+`./bin/ooniprobe nettests/core/dnsconsistency.py [-t <test resolvers>|-T <test resolver file>-f <input file> -b IP:PORT`
+
+*test resolvers* is a single test resolver (IP address)
+
+*test resolvers file* is a file containing the IP addresses of the resolvers to test for censorship, one per line.
+
+*input file* is a file containing hostnames or urls to check for tampering.
+
+*IP:PORT* is the address of the known good "control" resolver.
+
+Sample report
+=============
+
+From running:
- `./bin/ooniprobe nettests/core/dnsconsistency.py -t test_inputs/dns_tamper_test_resolvers.txt -f test_inputs/http_host_file.txt`
++`./bin/ooniprobe nettests/core/dnsconsistency.py -T test_inputs/dns_tamper_test_resolvers.txt -f test_inputs/http_host_file.txt`
+
+::
+
+ ###########################################
+ # OONI Probe Report for DNS tamper test
+ # Thu Nov 29 12:17:19 2012
+ ###########################################
+ ---
+ options:
+ collector: null
+ help: 0
+ logfile: null
+ pcapfile: null
+ reportfile: null
+ resume: 0
+ subargs: [-t, 8.8.8.8, -f, test_inputs/dns_tamper_file.txt]
+ test: nettests/blocking/dnsconsistency.py
+ probe_asn: null
+ probe_cc: null
+ probe_ip: 127.0.0.1
+ software_name: ooniprobe
+ software_version: 0.0.7.1-alpha
+ start_time: 1354184239.0
+ test_name: DNS tamper
+ test_version: '0.4'
+ ...
+ ---
+ input: torproject.org
+ report:
+ control_resolver: &id001 [8.8.8.8, 53]
+ queries:
+ - addrs: [86.59.30.40, 38.229.72.14, 38.229.72.16, 82.195.75.101]
+ answers:
+ - [<RR name=torproject.org type=A class=IN ttl=91s auth=False>, <A address=86.59.30.40
+ ttl=91>]
+ - [<RR name=torproject.org type=A class=IN ttl=91s auth=False>, <A address=38.229.72.14
+ ttl=91>]
+ - [<RR name=torproject.org type=A class=IN ttl=91s auth=False>, <A address=38.229.72.16
+ ttl=91>]
+ - [<RR name=torproject.org type=A class=IN ttl=91s auth=False>, <A address=82.195.75.101
+ ttl=91>]
+ query: '[Query(''torproject.org'', 1, 1)]'
+ query_type: A
+ resolver: *id001
+ - addrs: [86.59.30.40, 38.229.72.14, 38.229.72.16, 82.195.75.101]
+ answers:
+ - [<RR name=torproject.org type=A class=IN ttl=91s auth=False>, <A address=86.59.30.40
+ ttl=91>]
+ - [<RR name=torproject.org type=A class=IN ttl=91s auth=False>, <A address=38.229.72.14
+ ttl=91>]
+ - [<RR name=torproject.org type=A class=IN ttl=91s auth=False>, <A address=38.229.72.16
+ ttl=91>]
+ - [<RR name=torproject.org type=A class=IN ttl=91s auth=False>, <A address=82.195.75.101
+ ttl=91>]
+ query: '[Query(''torproject.org'', 1, 1)]'
+ query_type: A
+ resolver: [8.8.8.8, 53]
+ tampering: {8.8.8.8: false}
+ test_name: test_a_lookup
+ test_runtime: 0.0733950138092041
+ test_started: 1354187839.508863
+ ...
+ ---
+ input: google.com
+ report:
+ control_resolver: &id001 [8.8.8.8, 53]
+ queries:
+ - addrs: [173.194.69.100, 173.194.69.139, 173.194.69.113, 173.194.69.101, 173.194.69.138,
+ 173.194.69.102]
+ answers:
+ - [<RR name=google.com type=A class=IN ttl=54s auth=False>, <A address=173.194.69.100
+ ttl=54>]
+ - [<RR name=google.com type=A class=IN ttl=54s auth=False>, <A address=173.194.69.139
+ ttl=54>]
+ - [<RR name=google.com type=A class=IN ttl=54s auth=False>, <A address=173.194.69.113
+ ttl=54>]
+ - [<RR name=google.com type=A class=IN ttl=54s auth=False>, <A address=173.194.69.101
+ ttl=54>]
+ - [<RR name=google.com type=A class=IN ttl=54s auth=False>, <A address=173.194.69.138
+ ttl=54>]
+ - [<RR name=google.com type=A class=IN ttl=54s auth=False>, <A address=173.194.69.102
+ ttl=54>]
+ query: '[Query(''google.com'', 1, 1)]'
+ query_type: A
+ resolver: *id001
+ - addrs: [173.194.69.100, 173.194.69.139, 173.194.69.113, 173.194.69.101, 173.194.69.138,
+ 173.194.69.102]
+ answers:
+ - [<RR name=google.com type=A class=IN ttl=54s auth=False>, <A address=173.194.69.100
+ ttl=54>]
+ - [<RR name=google.com type=A class=IN ttl=54s auth=False>, <A address=173.194.69.139
+ ttl=54>]
+ - [<RR name=google.com type=A class=IN ttl=54s auth=False>, <A address=173.194.69.113
+ ttl=54>]
+ - [<RR name=google.com type=A class=IN ttl=54s auth=False>, <A address=173.194.69.101
+ ttl=54>]
+ - [<RR name=google.com type=A class=IN ttl=54s auth=False>, <A address=173.194.69.138
+ ttl=54>]
+ - [<RR name=google.com type=A class=IN ttl=54s auth=False>, <A address=173.194.69.102
+ ttl=54>]
+ query: '[Query(''google.com'', 1, 1)]'
+ query_type: A
+ resolver: [8.8.8.8, 53]
+ tampering: {8.8.8.8: false}
+ test_name: test_a_lookup
+ test_runtime: 0.08325004577636719
+ test_started: 1354187839.51091
+ ...
+ ---
+ input: measurementlab.net
+ report:
+ control_resolver: &id001 [8.8.8.8, 53]
+ queries:
+ - addrs: [72.249.86.184]
+ answers:
+ - [<RR name=measurementlab.net type=A class=IN ttl=600s auth=False>, <A address=72.249.86.184
+ ttl=600>]
+ query: '[Query(''measurementlab.net'', 1, 1)]'
+ query_type: A
+ resolver: *id001
+ - addrs: [72.249.86.184]
+ answers:
+ - [<RR name=measurementlab.net type=A class=IN ttl=600s auth=False>, <A address=72.249.86.184
+ ttl=600>]
+ query: '[Query(''measurementlab.net'', 1, 1)]'
+ query_type: A
+ resolver: [8.8.8.8, 53]
+ tampering: {8.8.8.8: false}
+ test_name: test_a_lookup
+ test_runtime: 0.2565779685974121
+ test_started: 1354187839.512434
+ ...
+
+Notes: Query is the string representation of :class:twisted.names.dns.Query
+
diff --cc nettests/blocking/dnsconsistency.py
index 6703478,0000000..7b6e7b9
mode 100644,000000..100644
--- a/nettests/blocking/dnsconsistency.py
+++ b/nettests/blocking/dnsconsistency.py
@@@ -1,166 -1,0 +1,173 @@@
+# -*- encoding: utf-8 -*-
+#
+# dnsconsistency
+# **************
+#
+# The test reports censorship if the cardinality of the intersection of
+# the query result set from the control server and the query result set
+# from the experimental server is zero, which is to say, if the two sets
+# have no matching results whatsoever.
+#
+# NOTE: This test frequently results in false positives due to GeoIP-based
+# load balancing on major global sites such as google, facebook, and
+# youtube, etc.
+#
+# :authors: Arturo Filastò, Isis Lovecruft
+# :licence: see LICENSE
+
+import pdb
+
+from twisted.python import usage
+from twisted.internet import defer
+
+from ooni.templates import dnst
+
+from ooni import nettest
+from ooni.utils import log
+
+class UsageOptions(usage.Options):
+ optParameters = [['backend', 'b', '8.8.8.8:53',
+ 'The OONI backend that runs the DNS resolver'],
+ ['testresolvers', 'T', None,
+ 'File containing list of DNS resolvers to test against'],
+ ['testresolver', 't', None,
+ 'Specify a single test resolver to use for testing']
+ ]
+
+class DNSConsistencyTest(dnst.DNSTest):
+
+ name = "DNS Consistency"
+ description = "DNS censorship detection test"
+ version = "0.5"
+ authors = "Arturo Filastò, Isis Lovecruft"
+ requirements = None
+
+ inputFile = ['file', 'f', None,
+ 'Input file of list of hostnames to attempt to resolve']
+
+ usageOptions = UsageOptions
+ requiredOptions = ['backend', 'file']
+
+ def setUp(self):
+ if (not self.localOptions['testresolvers'] and \
+ not self.localOptions['testresolver']):
+ raise usage.UsageError("You did not specify a testresolver")
+
+ elif self.localOptions['testresolvers']:
+ test_resolvers_file = self.localOptions['testresolvers']
+
+ elif self.localOptions['testresolver']:
+ self.test_resolvers = [self.localOptions['testresolver']]
+
+ try:
+ with open(test_resolvers_file) as f:
+ self.test_resolvers = [x.split('#')[0].strip() for x in f.readlines()]
+ self.report['test_resolvers'] = self.test_resolvers
+ f.close()
+
+ except IOError, e:
+ log.exception(e)
+ raise usage.UsageError("Invalid test resolvers file")
+
+ except NameError:
+ log.debug("No test resolver file configured")
+
+ dns_ip, dns_port = self.localOptions['backend'].split(':')
+ self.control_dns_server = (dns_ip, int(dns_port))
+
+ self.report['control_resolver'] = self.control_dns_server
+
+ @defer.inlineCallbacks
+ def test_a_lookup(self):
+ """
+ We perform an A lookup on the DNS test servers for the domains to be
+ tested and an A lookup on the known good DNS server.
+
+ We then compare the results from test_resolvers and that from
+ control_resolver and see if the match up.
+ If they match up then no censorship is happening (tampering: false).
+
+ If they do not we do a reverse lookup (PTR) on the test_resolvers and
+ the control resolver for every IP address we got back and check to see
+ if anyone of them matches the control ones.
+
+ If they do then we take not of the fact that censorship is probably not
+ happening (tampering: reverse-match).
+
+ If they do not match then censorship is probably going on (tampering:
+ true).
+ """
+ log.msg("Doing the test lookups on %s" % self.input)
+ list_of_ds = []
+ hostname = self.input
+
+ self.report['tampering'] = {}
+
+ control_answers = yield self.performALookup(hostname, self.control_dns_server)
+ if not control_answers:
+ log.err("Got no response from control DNS server %s," \
+ " perhaps the DNS resolver is down?" % self.control_dns_server[0])
+ self.report['tampering'][self.control_dns_server] = 'no_answer'
+ return
+
+ for test_resolver in self.test_resolvers:
+ log.msg("Testing resolver: %s" % test_resolver)
+ test_dns_server = (test_resolver, 53)
+
- experiment_answers = yield self.performALookup(hostname, test_dns_server)
++ try:
++ experiment_answers = yield self.performALookup(hostname, test_dns_server)
++ except Exception, e:
++ log.err("Problem performing the DNS lookup")
++ log.exception(e)
++ self.report['tampering'][test_resolver] = 'dns_lookup_error'
++ continue
++
+ if not experiment_answers:
+ log.err("Got no response, perhaps the DNS resolver is down?")
+ self.report['tampering'][test_resolver] = 'no_answer'
+ continue
+ else:
+ log.debug("Got the following A lookup answers %s from %s" % (experiment_answers, test_resolver))
+
+ def lookup_details():
+ """
+ A closure useful for printing test details.
+ """
+ log.msg("test resolver: %s" % test_resolver)
+ log.msg("experiment answers: %s" % experiment_answers)
+ log.msg("control answers: %s" % control_answers)
+
+ log.debug("Comparing %s with %s" % (experiment_answers, control_answers))
+ if set(experiment_answers) & set(control_answers):
+ lookup_details()
+ log.msg("tampering: false")
+ self.report['tampering'][test_resolver] = False
+ else:
+ log.msg("Trying to do reverse lookup")
+
+ experiment_reverse = yield self.performPTRLookup(experiment_answers[0], test_dns_server)
+ control_reverse = yield self.performPTRLookup(control_answers[0], self.control_dns_server)
+
+ if experiment_reverse == control_reverse:
+ log.msg("Further testing has eliminated false positives")
+ lookup_details()
+ log.msg("tampering: reverse_match")
+ self.report['tampering'][test_resolver] = 'reverse_match'
+ else:
+ log.msg("Reverse lookups do not match")
+ lookup_details()
+ log.msg("tampering: true")
+ self.report['tampering'][test_resolver] = True
+
+ def inputProcessor(self, filename=None):
+ """
+ This inputProcessor extracts domain names from urls
+ """
+ log.debug("Running dnsconsistency default processor")
+ if filename:
+ fp = open(filename)
+ for x in fp.readlines():
+ yield x.strip().split('//')[-1].split('/')[0]
+ fp.close()
+ else:
+ pass
More information about the tor-commits
mailing list