[tor-commits] [ooni-probe/master] Rename DNS Tamper Test to DNS Consistency Test

isis at torproject.org isis at torproject.org
Sun Feb 24 19:10:38 UTC 2013


commit 9dbe9841b16e8d678c5e32222a83cda4e4f5d40c
Author: Arturo Filastò <art at fuffa.org>
Date:   Tue Feb 5 00:50:26 2013 +0100

    Rename DNS Tamper Test to DNS Consistency Test
---
 decks/before_i_commit.testdeck             |    2 +-
 docs/source/index.rst                      |    2 +-
 docs/source/nettests/nettests.blocking.rst |    4 +-
 docs/source/tests/dnsconsistency.rst       |  180 ++++++++++++++++++++++++++++
 docs/source/tests/dnstamper.rst            |  180 ----------------------------
 docs/source/writing_tests.rst              |    2 +-
 inputs/README                              |    2 +-
 nettests/blocking/dnsconsistency.py        |  166 +++++++++++++++++++++++++
 nettests/blocking/dnstamper.py             |  166 -------------------------
 9 files changed, 352 insertions(+), 352 deletions(-)

diff --git a/decks/before_i_commit.testdeck b/decks/before_i_commit.testdeck
index 7877d74..525204d 100644
--- a/decks/before_i_commit.testdeck
+++ b/decks/before_i_commit.testdeck
@@ -13,7 +13,7 @@
     pcapfile: null
     reportfile: dns_tamper_test.yamloo
     subargs: [-T, example_inputs/dns_tamper_test_resolvers.txt, -f, example_inputs/dns_tamper_file.txt]
-    test: nettests/blocking/dnstamper.py
+    test: nettests/blocking/dnsconsistency.py
 - options:
     collector: null
     help: 0
diff --git a/docs/source/index.rst b/docs/source/index.rst
index ca74809..43ad1fe 100644
--- a/docs/source/index.rst
+++ b/docs/source/index.rst
@@ -48,7 +48,7 @@ respectively.
 Content Blocking Tests
 ......................
 
-  * `DNSTamper <tests/dnstamper.html>`_
+  * `DNSConsistency <tests/dnsconsistency.html>`_
 
   * `HTTP Requests <tests/http_requests.html>`_
 
diff --git a/docs/source/nettests/nettests.blocking.rst b/docs/source/nettests/nettests.blocking.rst
index 03a08b6..eaec0e8 100644
--- a/docs/source/nettests/nettests.blocking.rst
+++ b/docs/source/nettests/nettests.blocking.rst
@@ -1,10 +1,10 @@
 nettests blocking Package
 =========================
 
-:mod:`nettests.blocking.dnstamper` Module
+:mod:`nettests.blocking.dnsconsistency` Module
 -----------------------------------------
 
-.. automodule:: nettests.blocking.dnstamper
+.. automodule:: nettests.blocking.dnsconsistency
      :members:
      :undoc-members:
      :show-inheritance:
diff --git a/docs/source/tests/dnsconsistency.rst b/docs/source/tests/dnsconsistency.rst
new file mode 100644
index 0000000..8a413bf
--- /dev/null
+++ b/docs/source/tests/dnsconsistency.rst
@@ -0,0 +1,180 @@
+Details
+=======
+
+*Test Name*: DNS Consistency (Ex DNS Tamper)
+
+*Current version*: 0.4
+
+*NetTest*: DNS Consistency Test (https://gitweb.torproject.org/ooni-probe.git/blob/HEAD:/nettests/blocking/dnsconsistency.py)
+
+*Test Helper*: DNS Test Helper (https://gitweb.torproject.org/ooni-probe.git/blob/HEAD:/oonib/testhelpers/dns_helpers.py)
+
+*Test Type*: Content Blocking
+
+*Requires Root*: No
+
+Description
+===========
+
+This test performs A queries to a set of test resolvers and a known good
+control resolver. If the two results do not match it will perform a reverse DNS
+lookup on the first A record address of both sets and check if they both
+resolve to the same name.
+
+NOTE: This test frequently results in false positives due to GeoIP-based
+load balancing on major global sites such as Google, Facebook, and
+Youtube, etc.
+
+How to run the test
+===================
+
+`./bin/ooniprobe nettests/core/dnsconsistency.py [-t <test resolvers>|-T <test resolver file>-f <input file> -b IP:PORT`
+
+*test resolvers* is a single test resolver (IP address)
+
+*test resolvers file* is a file containing the IP addresses of the resolvers to test for censorship, one per line.
+
+*input file* is a file containing hostnames or urls to check for tampering.
+
+*IP:PORT* is the address of the known good "control" resolver.
+
+Sample report
+=============
+
+From running:
+`./bin/ooniprobe nettests/core/dnsconsistency.py -t test_inputs/dns_tamper_test_resolvers.txt -f test_inputs/http_host_file.txt`
+
+::
+
+    ###########################################
+    # OONI Probe Report for DNS tamper test
+    # Thu Nov 29 12:17:19 2012
+    ###########################################
+    ---
+    options:
+      collector: null
+      help: 0
+      logfile: null
+      pcapfile: null
+      reportfile: null
+      resume: 0
+      subargs: [-t, 8.8.8.8, -f, test_inputs/dns_tamper_file.txt]
+      test: nettests/blocking/dnsconsistency.py
+    probe_asn: null
+    probe_cc: null
+    probe_ip: 127.0.0.1
+    software_name: ooniprobe
+    software_version: 0.0.7.1-alpha
+    start_time: 1354184239.0
+    test_name: DNS tamper
+    test_version: '0.4'
+    ...
+    ---
+    input: torproject.org
+    report:
+      control_resolver: &id001 [8.8.8.8, 53]
+      queries:
+      - addrs: [86.59.30.40, 38.229.72.14, 38.229.72.16, 82.195.75.101]
+        answers:
+        - [<RR name=torproject.org type=A class=IN ttl=91s auth=False>, <A address=86.59.30.40
+            ttl=91>]
+        - [<RR name=torproject.org type=A class=IN ttl=91s auth=False>, <A address=38.229.72.14
+            ttl=91>]
+        - [<RR name=torproject.org type=A class=IN ttl=91s auth=False>, <A address=38.229.72.16
+            ttl=91>]
+        - [<RR name=torproject.org type=A class=IN ttl=91s auth=False>, <A address=82.195.75.101
+            ttl=91>]
+        query: '[Query(''torproject.org'', 1, 1)]'
+        query_type: A
+        resolver: *id001
+      - addrs: [86.59.30.40, 38.229.72.14, 38.229.72.16, 82.195.75.101]
+        answers:
+        - [<RR name=torproject.org type=A class=IN ttl=91s auth=False>, <A address=86.59.30.40
+            ttl=91>]
+        - [<RR name=torproject.org type=A class=IN ttl=91s auth=False>, <A address=38.229.72.14
+            ttl=91>]
+        - [<RR name=torproject.org type=A class=IN ttl=91s auth=False>, <A address=38.229.72.16
+            ttl=91>]
+        - [<RR name=torproject.org type=A class=IN ttl=91s auth=False>, <A address=82.195.75.101
+            ttl=91>]
+        query: '[Query(''torproject.org'', 1, 1)]'
+        query_type: A
+        resolver: [8.8.8.8, 53]
+      tampering: {8.8.8.8: false}
+    test_name: test_a_lookup
+    test_runtime: 0.0733950138092041
+    test_started: 1354187839.508863
+    ...
+    ---
+    input: google.com
+    report:
+      control_resolver: &id001 [8.8.8.8, 53]
+      queries:
+      - addrs: [173.194.69.100, 173.194.69.139, 173.194.69.113, 173.194.69.101, 173.194.69.138,
+          173.194.69.102]
+        answers:
+        - [<RR name=google.com type=A class=IN ttl=54s auth=False>, <A address=173.194.69.100
+            ttl=54>]
+        - [<RR name=google.com type=A class=IN ttl=54s auth=False>, <A address=173.194.69.139
+            ttl=54>]
+        - [<RR name=google.com type=A class=IN ttl=54s auth=False>, <A address=173.194.69.113
+            ttl=54>]
+        - [<RR name=google.com type=A class=IN ttl=54s auth=False>, <A address=173.194.69.101
+            ttl=54>]
+        - [<RR name=google.com type=A class=IN ttl=54s auth=False>, <A address=173.194.69.138
+            ttl=54>]
+        - [<RR name=google.com type=A class=IN ttl=54s auth=False>, <A address=173.194.69.102
+            ttl=54>]
+        query: '[Query(''google.com'', 1, 1)]'
+        query_type: A
+        resolver: *id001
+      - addrs: [173.194.69.100, 173.194.69.139, 173.194.69.113, 173.194.69.101, 173.194.69.138,
+          173.194.69.102]
+        answers:
+        - [<RR name=google.com type=A class=IN ttl=54s auth=False>, <A address=173.194.69.100
+            ttl=54>]
+        - [<RR name=google.com type=A class=IN ttl=54s auth=False>, <A address=173.194.69.139
+            ttl=54>]
+        - [<RR name=google.com type=A class=IN ttl=54s auth=False>, <A address=173.194.69.113
+            ttl=54>]
+        - [<RR name=google.com type=A class=IN ttl=54s auth=False>, <A address=173.194.69.101
+            ttl=54>]
+        - [<RR name=google.com type=A class=IN ttl=54s auth=False>, <A address=173.194.69.138
+            ttl=54>]
+        - [<RR name=google.com type=A class=IN ttl=54s auth=False>, <A address=173.194.69.102
+            ttl=54>]
+        query: '[Query(''google.com'', 1, 1)]'
+        query_type: A
+        resolver: [8.8.8.8, 53]
+      tampering: {8.8.8.8: false}
+    test_name: test_a_lookup
+    test_runtime: 0.08325004577636719
+    test_started: 1354187839.51091
+    ...
+    ---
+    input: measurementlab.net
+    report:
+      control_resolver: &id001 [8.8.8.8, 53]
+      queries:
+      - addrs: [72.249.86.184]
+        answers:
+        - [<RR name=measurementlab.net type=A class=IN ttl=600s auth=False>, <A address=72.249.86.184
+            ttl=600>]
+        query: '[Query(''measurementlab.net'', 1, 1)]'
+        query_type: A
+        resolver: *id001
+      - addrs: [72.249.86.184]
+        answers:
+        - [<RR name=measurementlab.net type=A class=IN ttl=600s auth=False>, <A address=72.249.86.184
+            ttl=600>]
+        query: '[Query(''measurementlab.net'', 1, 1)]'
+        query_type: A
+        resolver: [8.8.8.8, 53]
+      tampering: {8.8.8.8: false}
+    test_name: test_a_lookup
+    test_runtime: 0.2565779685974121
+    test_started: 1354187839.512434
+    ...
+
+Notes: Query is the string representation of :class:twisted.names.dns.Query
+
diff --git a/docs/source/tests/dnstamper.rst b/docs/source/tests/dnstamper.rst
deleted file mode 100644
index 8fdb572..0000000
--- a/docs/source/tests/dnstamper.rst
+++ /dev/null
@@ -1,180 +0,0 @@
-Details
-=======
-
-*Test Name*: DNS Tamper
-
-*Current version*: 0.4
-
-*NetTest*: DNS Tamper Test (https://gitweb.torproject.org/ooni-probe.git/blob/HEAD:/nettests/blocking/dnstamper.py)
-
-*Test Helper*: DNS Test Helper (https://gitweb.torproject.org/ooni-probe.git/blob/HEAD:/oonib/testhelpers/dns_helpers.py)
-
-*Test Type*: Content Blocking
-
-*Requires Root*: No
-
-Description
-===========
-
-This test performs A queries to a set of test resolvers and a known good
-control resolver. If the two results do not match it will perform a reverse DNS
-lookup on the first A record address of both sets and check if they both
-resolve to the same name.
-
-NOTE: This test frequently results in false positives due to GeoIP-based
-load balancing on major global sites such as Google, Facebook, and
-Youtube, etc.
-
-How to run the test
-===================
-
-`./bin/ooniprobe nettests/core/dnstamper.py [-t <test resolvers>|-T <test resolver file>-f <input file> -b IP:PORT`
-
-*test resolvers* is a single test resolver (IP address)
-
-*test resolvers file* is a file containing the IP addresses of the resolvers to test for censorship, one per line.
-
-*input file* is a file containing hostnames or urls to check for tampering.
-
-*IP:PORT* is the address of the known good "control" resolver.
-
-Sample report
-=============
-
-From running:
-`./bin/ooniprobe nettests/core/dnstamper.py -t test_inputs/dns_tamper_test_resolvers.txt -f test_inputs/http_host_file.txt`
-
-::
-
-    ###########################################
-    # OONI Probe Report for DNS tamper test
-    # Thu Nov 29 12:17:19 2012
-    ###########################################
-    ---
-    options:
-      collector: null
-      help: 0
-      logfile: null
-      pcapfile: null
-      reportfile: null
-      resume: 0
-      subargs: [-t, 8.8.8.8, -f, test_inputs/dns_tamper_file.txt]
-      test: nettests/blocking/dnstamper.py
-    probe_asn: null
-    probe_cc: null
-    probe_ip: 127.0.0.1
-    software_name: ooniprobe
-    software_version: 0.0.7.1-alpha
-    start_time: 1354184239.0
-    test_name: DNS tamper
-    test_version: '0.4'
-    ...
-    ---
-    input: torproject.org
-    report:
-      control_resolver: &id001 [8.8.8.8, 53]
-      queries:
-      - addrs: [86.59.30.40, 38.229.72.14, 38.229.72.16, 82.195.75.101]
-        answers:
-        - [<RR name=torproject.org type=A class=IN ttl=91s auth=False>, <A address=86.59.30.40
-            ttl=91>]
-        - [<RR name=torproject.org type=A class=IN ttl=91s auth=False>, <A address=38.229.72.14
-            ttl=91>]
-        - [<RR name=torproject.org type=A class=IN ttl=91s auth=False>, <A address=38.229.72.16
-            ttl=91>]
-        - [<RR name=torproject.org type=A class=IN ttl=91s auth=False>, <A address=82.195.75.101
-            ttl=91>]
-        query: '[Query(''torproject.org'', 1, 1)]'
-        query_type: A
-        resolver: *id001
-      - addrs: [86.59.30.40, 38.229.72.14, 38.229.72.16, 82.195.75.101]
-        answers:
-        - [<RR name=torproject.org type=A class=IN ttl=91s auth=False>, <A address=86.59.30.40
-            ttl=91>]
-        - [<RR name=torproject.org type=A class=IN ttl=91s auth=False>, <A address=38.229.72.14
-            ttl=91>]
-        - [<RR name=torproject.org type=A class=IN ttl=91s auth=False>, <A address=38.229.72.16
-            ttl=91>]
-        - [<RR name=torproject.org type=A class=IN ttl=91s auth=False>, <A address=82.195.75.101
-            ttl=91>]
-        query: '[Query(''torproject.org'', 1, 1)]'
-        query_type: A
-        resolver: [8.8.8.8, 53]
-      tampering: {8.8.8.8: false}
-    test_name: test_a_lookup
-    test_runtime: 0.0733950138092041
-    test_started: 1354187839.508863
-    ...
-    ---
-    input: google.com
-    report:
-      control_resolver: &id001 [8.8.8.8, 53]
-      queries:
-      - addrs: [173.194.69.100, 173.194.69.139, 173.194.69.113, 173.194.69.101, 173.194.69.138,
-          173.194.69.102]
-        answers:
-        - [<RR name=google.com type=A class=IN ttl=54s auth=False>, <A address=173.194.69.100
-            ttl=54>]
-        - [<RR name=google.com type=A class=IN ttl=54s auth=False>, <A address=173.194.69.139
-            ttl=54>]
-        - [<RR name=google.com type=A class=IN ttl=54s auth=False>, <A address=173.194.69.113
-            ttl=54>]
-        - [<RR name=google.com type=A class=IN ttl=54s auth=False>, <A address=173.194.69.101
-            ttl=54>]
-        - [<RR name=google.com type=A class=IN ttl=54s auth=False>, <A address=173.194.69.138
-            ttl=54>]
-        - [<RR name=google.com type=A class=IN ttl=54s auth=False>, <A address=173.194.69.102
-            ttl=54>]
-        query: '[Query(''google.com'', 1, 1)]'
-        query_type: A
-        resolver: *id001
-      - addrs: [173.194.69.100, 173.194.69.139, 173.194.69.113, 173.194.69.101, 173.194.69.138,
-          173.194.69.102]
-        answers:
-        - [<RR name=google.com type=A class=IN ttl=54s auth=False>, <A address=173.194.69.100
-            ttl=54>]
-        - [<RR name=google.com type=A class=IN ttl=54s auth=False>, <A address=173.194.69.139
-            ttl=54>]
-        - [<RR name=google.com type=A class=IN ttl=54s auth=False>, <A address=173.194.69.113
-            ttl=54>]
-        - [<RR name=google.com type=A class=IN ttl=54s auth=False>, <A address=173.194.69.101
-            ttl=54>]
-        - [<RR name=google.com type=A class=IN ttl=54s auth=False>, <A address=173.194.69.138
-            ttl=54>]
-        - [<RR name=google.com type=A class=IN ttl=54s auth=False>, <A address=173.194.69.102
-            ttl=54>]
-        query: '[Query(''google.com'', 1, 1)]'
-        query_type: A
-        resolver: [8.8.8.8, 53]
-      tampering: {8.8.8.8: false}
-    test_name: test_a_lookup
-    test_runtime: 0.08325004577636719
-    test_started: 1354187839.51091
-    ...
-    ---
-    input: measurementlab.net
-    report:
-      control_resolver: &id001 [8.8.8.8, 53]
-      queries:
-      - addrs: [72.249.86.184]
-        answers:
-        - [<RR name=measurementlab.net type=A class=IN ttl=600s auth=False>, <A address=72.249.86.184
-            ttl=600>]
-        query: '[Query(''measurementlab.net'', 1, 1)]'
-        query_type: A
-        resolver: *id001
-      - addrs: [72.249.86.184]
-        answers:
-        - [<RR name=measurementlab.net type=A class=IN ttl=600s auth=False>, <A address=72.249.86.184
-            ttl=600>]
-        query: '[Query(''measurementlab.net'', 1, 1)]'
-        query_type: A
-        resolver: [8.8.8.8, 53]
-      tampering: {8.8.8.8: false}
-    test_name: test_a_lookup
-    test_runtime: 0.2565779685974121
-    test_started: 1354187839.512434
-    ...
-
-Notes: Query is the string representation of :class:twisted.names.dns.Query
-
diff --git a/docs/source/writing_tests.rst b/docs/source/writing_tests.rst
index ccc529a..8f431d6 100644
--- a/docs/source/writing_tests.rst
+++ b/docs/source/writing_tests.rst
@@ -508,4 +508,4 @@ Report format
   test_started: 1354801371.980114
   ...
 
-For a more complex example, see: :mod:`nettests.blocking.dnstamper`
+For a more complex example, see: :mod:`nettests.blocking.dnsconsistency`
diff --git a/inputs/README b/inputs/README
index 9a4f9b7..2bbc11e 100644
--- a/inputs/README
+++ b/inputs/README
@@ -3,7 +3,7 @@ the correct functionality of the various OONIProbe tests.
 
 # DNS Tamper
 
-./bin/ooniprobe -o dns_tamper_test.yamloo nettests/blocking/dnstamper.py -t
+./bin/ooniprobe -o dns_tamper_test.yamloo nettests/blocking/dnsconsistency.py -t
 example_inputs/dns_tamper_test_resolvers.txt -f example_inputs/dns_tamper_file.txt
 
 less dns_tamper_test.yamloo
diff --git a/nettests/blocking/dnsconsistency.py b/nettests/blocking/dnsconsistency.py
new file mode 100644
index 0000000..6703478
--- /dev/null
+++ b/nettests/blocking/dnsconsistency.py
@@ -0,0 +1,166 @@
+# -*- encoding: utf-8 -*-
+#
+#  dnsconsistency
+#  **************
+#
+#  The test reports censorship if the cardinality of the intersection of
+#  the query result set from the control server and the query result set
+#  from the experimental server is zero, which is to say, if the two sets
+#  have no matching results whatsoever.
+#
+#  NOTE: This test frequently results in false positives due to GeoIP-based
+#  load balancing on major global sites such as google, facebook, and
+#  youtube, etc.
+#
+# :authors: Arturo Filastò, Isis Lovecruft
+# :licence: see LICENSE
+
+import pdb
+
+from twisted.python import usage
+from twisted.internet import defer
+
+from ooni.templates import dnst
+
+from ooni import nettest
+from ooni.utils import log
+
+class UsageOptions(usage.Options):
+    optParameters = [['backend', 'b', '8.8.8.8:53',
+                        'The OONI backend that runs the DNS resolver'],
+                     ['testresolvers', 'T', None,
+                        'File containing list of DNS resolvers to test against'],
+                     ['testresolver', 't', None,
+                         'Specify a single test resolver to use for testing']
+                    ]
+
+class DNSConsistencyTest(dnst.DNSTest):
+
+    name = "DNS Consistency"
+    description = "DNS censorship detection test"
+    version = "0.5"
+    authors = "Arturo Filastò, Isis Lovecruft"
+    requirements = None
+
+    inputFile = ['file', 'f', None,
+                 'Input file of list of hostnames to attempt to resolve']
+
+    usageOptions = UsageOptions
+    requiredOptions = ['backend', 'file']
+
+    def setUp(self):
+        if (not self.localOptions['testresolvers'] and \
+                not self.localOptions['testresolver']):
+            raise usage.UsageError("You did not specify a testresolver")
+
+        elif self.localOptions['testresolvers']:
+            test_resolvers_file = self.localOptions['testresolvers']
+
+        elif self.localOptions['testresolver']:
+            self.test_resolvers = [self.localOptions['testresolver']]
+
+        try:
+            with open(test_resolvers_file) as f:
+                self.test_resolvers = [x.split('#')[0].strip() for x in f.readlines()]
+                self.report['test_resolvers'] = self.test_resolvers
+            f.close()
+
+        except IOError, e:
+            log.exception(e)
+            raise usage.UsageError("Invalid test resolvers file")
+
+        except NameError:
+            log.debug("No test resolver file configured")
+
+        dns_ip, dns_port = self.localOptions['backend'].split(':')
+        self.control_dns_server = (dns_ip, int(dns_port))
+
+        self.report['control_resolver'] = self.control_dns_server
+
+    @defer.inlineCallbacks
+    def test_a_lookup(self):
+        """
+        We perform an A lookup on the DNS test servers for the domains to be
+        tested and an A lookup on the known good DNS server.
+
+        We then compare the results from test_resolvers and that from
+        control_resolver and see if the match up.
+        If they match up then no censorship is happening (tampering: false).
+
+        If they do not we do a reverse lookup (PTR) on the test_resolvers and
+        the control resolver for every IP address we got back and check to see
+        if anyone of them matches the control ones.
+
+        If they do then we take not of the fact that censorship is probably not
+        happening (tampering: reverse-match).
+
+        If they do not match then censorship is probably going on (tampering:
+        true).
+        """
+        log.msg("Doing the test lookups on %s" % self.input)
+        list_of_ds = []
+        hostname = self.input
+
+        self.report['tampering'] = {}
+
+        control_answers = yield self.performALookup(hostname, self.control_dns_server)
+        if not control_answers:
+                log.err("Got no response from control DNS server %s," \
+                        " perhaps the DNS resolver is down?" % self.control_dns_server[0])
+                self.report['tampering'][self.control_dns_server] = 'no_answer'
+                return
+
+        for test_resolver in self.test_resolvers:
+            log.msg("Testing resolver: %s" % test_resolver)
+            test_dns_server = (test_resolver, 53)
+
+            experiment_answers = yield self.performALookup(hostname, test_dns_server)
+            if not experiment_answers:
+                log.err("Got no response, perhaps the DNS resolver is down?")
+                self.report['tampering'][test_resolver] = 'no_answer'
+                continue
+            else:
+                log.debug("Got the following A lookup answers %s from %s" % (experiment_answers, test_resolver))
+
+            def lookup_details():
+                """
+                A closure useful for printing test details.
+                """
+                log.msg("test resolver: %s" % test_resolver)
+                log.msg("experiment answers: %s" % experiment_answers)
+                log.msg("control answers: %s" % control_answers)
+
+            log.debug("Comparing %s with %s" % (experiment_answers, control_answers))
+            if set(experiment_answers) & set(control_answers):
+                lookup_details()
+                log.msg("tampering: false")
+                self.report['tampering'][test_resolver] = False
+            else:
+                log.msg("Trying to do reverse lookup")
+
+                experiment_reverse = yield self.performPTRLookup(experiment_answers[0], test_dns_server)
+                control_reverse = yield self.performPTRLookup(control_answers[0], self.control_dns_server)
+
+                if experiment_reverse == control_reverse:
+                    log.msg("Further testing has eliminated false positives")
+                    lookup_details()
+                    log.msg("tampering: reverse_match")
+                    self.report['tampering'][test_resolver] = 'reverse_match'
+                else:
+                    log.msg("Reverse lookups do not match")
+                    lookup_details()
+                    log.msg("tampering: true")
+                    self.report['tampering'][test_resolver] = True
+
+    def inputProcessor(self, filename=None):
+        """
+        This inputProcessor extracts domain names from urls
+        """
+        log.debug("Running dnsconsistency default processor")
+        if filename:
+            fp = open(filename)
+            for x in fp.readlines():
+                yield x.strip().split('//')[-1].split('/')[0]
+            fp.close()
+        else:
+            pass
diff --git a/nettests/blocking/dnstamper.py b/nettests/blocking/dnstamper.py
deleted file mode 100644
index 1ef2c89..0000000
--- a/nettests/blocking/dnstamper.py
+++ /dev/null
@@ -1,166 +0,0 @@
-# -*- encoding: utf-8 -*-
-#
-#  dnstamper
-#  *********
-#
-#  The test reports censorship if the cardinality of the intersection of
-#  the query result set from the control server and the query result set
-#  from the experimental server is zero, which is to say, if the two sets
-#  have no matching results whatsoever.
-#
-#  NOTE: This test frequently results in false positives due to GeoIP-based
-#  load balancing on major global sites such as google, facebook, and
-#  youtube, etc.
-#
-# :authors: Arturo Filastò, Isis Lovecruft
-# :licence: see LICENSE
-
-import pdb
-
-from twisted.python import usage
-from twisted.internet import defer
-
-from ooni.templates import dnst
-
-from ooni import nettest
-from ooni.utils import log
-
-class UsageOptions(usage.Options):
-    optParameters = [['backend', 'b', '8.8.8.8:53',
-                        'The OONI backend that runs the DNS resolver'],
-                     ['testresolvers', 'T', None,
-                        'File containing list of DNS resolvers to test against'],
-                     ['testresolver', 't', None,
-                         'Specify a single test resolver to use for testing']
-                    ]
-
-class DNSTamperTest(dnst.DNSTest):
-
-    name = "DNS tamper"
-    description = "DNS censorship detection test"
-    version = "0.4"
-    authors = "Arturo Filastò, Isis Lovecruft"
-    requirements = None
-
-    inputFile = ['file', 'f', None,
-                 'Input file of list of hostnames to attempt to resolve']
-
-    usageOptions = UsageOptions
-    requiredOptions = ['backend', 'file']
-
-    def setUp(self):
-        if (not self.localOptions['testresolvers'] and \
-                not self.localOptions['testresolver']):
-            raise usage.UsageError("You did not specify a testresolver")
-
-        elif self.localOptions['testresolvers']:
-            test_resolvers_file = self.localOptions['testresolvers']
-
-        elif self.localOptions['testresolver']:
-            self.test_resolvers = [self.localOptions['testresolver']]
-
-        try:
-            with open(test_resolvers_file) as f:
-                self.test_resolvers = [x.split('#')[0].strip() for x in f.readlines()]
-                self.report['test_resolvers'] = self.test_resolvers
-            f.close()
-
-        except IOError, e:
-            log.exception(e)
-            raise usage.UsageError("Invalid test resolvers file")
-
-        except NameError:
-            log.debug("No test resolver file configured")
-
-        dns_ip, dns_port = self.localOptions['backend'].split(':')
-        self.control_dns_server = (dns_ip, int(dns_port))
-
-        self.report['control_resolver'] = self.control_dns_server
-
-    @defer.inlineCallbacks
-    def test_a_lookup(self):
-        """
-        We perform an A lookup on the DNS test servers for the domains to be
-        tested and an A lookup on the known good DNS server.
-
-        We then compare the results from test_resolvers and that from
-        control_resolver and see if the match up.
-        If they match up then no censorship is happening (tampering: false).
-
-        If they do not we do a reverse lookup (PTR) on the test_resolvers and
-        the control resolver for every IP address we got back and check to see
-        if anyone of them matches the control ones.
-
-        If they do then we take not of the fact that censorship is probably not
-        happening (tampering: reverse-match).
-
-        If they do not match then censorship is probably going on (tampering:
-        true).
-        """
-        log.msg("Doing the test lookups on %s" % self.input)
-        list_of_ds = []
-        hostname = self.input
-
-        self.report['tampering'] = {}
-
-        control_answers = yield self.performALookup(hostname, self.control_dns_server)
-        if not control_answers:
-                log.err("Got no response from control DNS server %s," \
-                        " perhaps the DNS resolver is down?" % self.control_dns_server[0])
-                self.report['tampering'][self.control_dns_server] = 'no_answer'
-                return
-
-        for test_resolver in self.test_resolvers:
-            log.msg("Testing resolver: %s" % test_resolver)
-            test_dns_server = (test_resolver, 53)
-
-            experiment_answers = yield self.performALookup(hostname, test_dns_server)
-            if not experiment_answers:
-                log.err("Got no response, perhaps the DNS resolver is down?")
-                self.report['tampering'][test_resolver] = 'no_answer'
-                continue
-            else:
-                log.debug("Got the following A lookup answers %s from %s" % (experiment_answers, test_resolver))
-
-            def lookup_details():
-                """
-                A closure useful for printing test details.
-                """
-                log.msg("test resolver: %s" % test_resolver)
-                log.msg("experiment answers: %s" % experiment_answers)
-                log.msg("control answers: %s" % control_answers)
-
-            log.debug("Comparing %s with %s" % (experiment_answers, control_answers))
-            if set(experiment_answers) & set(control_answers):
-                lookup_details()
-                log.msg("tampering: false")
-                self.report['tampering'][test_resolver] = False
-            else:
-                log.msg("Trying to do reverse lookup")
-
-                experiment_reverse = yield self.performPTRLookup(experiment_answers[0], test_dns_server)
-                control_reverse = yield self.performPTRLookup(control_answers[0], self.control_dns_server)
-
-                if experiment_reverse == control_reverse:
-                    log.msg("Further testing has eliminated false positives")
-                    lookup_details()
-                    log.msg("tampering: reverse_match")
-                    self.report['tampering'][test_resolver] = 'reverse_match'
-                else:
-                    log.msg("Reverse lookups do not match")
-                    lookup_details()
-                    log.msg("tampering: true")
-                    self.report['tampering'][test_resolver] = True
-
-    def inputProcessor(self, filename=None):
-        """
-        This inputProcessor extracts domain names from urls
-        """
-        log.debug("Running dnstamper default processor")
-        if filename:
-            fp = open(filename)
-            for x in fp.readlines():
-                yield x.strip().split('//')[-1].split('/')[0]
-            fp.close()
-        else:
-            pass





More information about the tor-commits mailing list