June 2014 - tor-commits - lists.torproject.org

[metrics-web/master] Move Q-and-A about user statistics to a text file.
by karsten＠torproject.org 26 Jun '14

26 Jun '14

commit f995b0f64febfe288af0f09efe0ef3b68c02100d Author: Karsten Loesing <karsten.loesing(a)gmx.net> Date: Thu Jun 26 16:43:03 2014 +0200 Move Q-and-A about user statistics to a text file. These questions and answers are likely read by less than 10% of visitors, and the remaining 90% wonder what that wall of text is. We should write a more general Q-and-A section covering the entire website. Whoever cares about how user statistics are calculated can read the text file. Tweak the Q's and A's a tiny bit while converting them to plain text. --- doc/users-q-and-a.txt | 94 +++++++++++++++++++++++++++++ website/web/WEB-INF/users.jsp | 131 +---------------------------------------- 2 files changed, 96 insertions(+), 129 deletions(-) diff --git a/doc/users-q-and-a.txt b/doc/users-q-and-a.txt new file mode 100644 index 0000000..15a1084 --- /dev/null +++ b/doc/users-q-and-a.txt @@ -0,0 +1,94 @@ +Questions and answers about user statistics +=========================================== + +Q: How is it even possible to count users in an anonymity network? +A: We actually don't count users, but we count requests to the directories +that clients make periodically to update their list of relays and estimate +user numbers indirectly from there. + +Q: Do all directories report these directory request numbers? +A: No, but we can see what fraction of directories reported them, and then +we can extrapolate the total number in the network. + +Q: How do you get from these directory requests to user numbers? +A: We put in the assumption that the average client makes 10 such requests +per day. A tor client that is connected 24/7 makes about 15 requests per +day, but not all clients are connected 24/7, so we picked the number 10 +for the average client. We simply divide directory requests by 10 and +consider the result as the number of users. Another way of looking at it, +is that we assume that each request represents a client that stays online +for one tenth of a day, so 2 hours and 24 minutes. + +Q: So, are these distinct users per day, average number of users connected +over the day, or what? +A: Average number of concurrent users, estimated from data collected over +a day. We can't say how many distinct users there are. + +Q: Are these tor clients or users? What if there's more than one user +behind a tor client? +A: Then we count those users as one. We really count clients, but it's +more intuitive for most people to think of users, that's why we say users +and not clients. + +Q: What if a user runs tor on a laptop and changes their IP address a few +times per day? Don't you overcount that user? +A: No, because that user updates their list of relays as often as a user +that doesn't change IP address over the day. + +Q: How do you know which countries users come from? +A: The directories resolve IP addresses to country codes and report these +numbers in aggregate form. This is one of the reasons why tor ships with +a GeoIP database. + +Q: Why are there so few bridge users that are not using the default OR +protocol or that are using IPv6? +A: Very few bridges report data on transports or IP versions yet, and by +default we consider requests to use the default OR protocol and IPv4. +Once more bridges report these data, the numbers will become more +accurate. + +Q: Why do the graphs end 2 days in the past and not today? +A: Relays and bridges report some of the data in 24-hour intervals which +may end at any time of the day. And after such an interval is over relays +and bridges might take another 18 hours to report the data. We cut off +the last two days from the graphs, because we want to avoid that the last +data point in a graph indicates a recent trend change which is in fact +just an artifact of the algorithm. + +Q: But I noticed that the last data point went up/down a bit since I last +looked a few hours ago. Why is that? +A: The reason is that we publish user numbers once we're confident enough +that they won't change significantly anymore. But it's always possible +that a directory reports data a few hours after we were confident enough, +but which then slightly changed the graph. + +Q: Why are no numbers available before September 2011? +A: We do have descriptor archives from before that time, but those +descriptors didn't contain all the data we use to estimate user numbers. + +Q: Why do you believe the current approach to estimate user numbers is +more accurate? +A: For direct users, we include all directories which we didn't do in the +old approach. We also use histories that only contain bytes written to +answer directory requests, which is more precise than using general byte +histories. + +Q: And what about the advantage of the current approach over the old one +when it comes to bridge users? +A: Oh, that's a whole different story. We wrote a 13 page long technical +report explaining the reasons for retiring the old approach. tl;dr: in +the old approach we measured the wrong thing, and now we measure the right +thing. + + https://research.torproject.org/techreports/counting-daily-bridge-users-201… + +Q: What are these red and blue dots indicating possible censorship +events? +A: We run an anomaly-based censorship-detection system that looks at +estimated user numbers over a series of days and predicts the user number +in the next days. If the actual number is higher or lower, this might +indicate a possible censorship event or release of censorship. For more +details, see our technical report. + + https://research.torproject.org/techreports/detector-2011-09-09.pdf + diff --git a/website/web/WEB-INF/users.jsp b/website/web/WEB-INF/users.jsp index 84cab43..0a31569 100644 --- a/website/web/WEB-INF/users.jsp +++ b/website/web/WEB-INF/users.jsp @@ -269,136 +269,9 @@ estimates. <hr> -<a name="questions-and-answers"></a> -Questions and answers - -Q: How is it even possible to count users in an anonymity network? -A: We actually don't count users, but we count requests to the directories -that clients make periodically to update their list of relays and estimate -user numbers indirectly from there. - - -Q: Do all directories report these directory request numbers? -A: No, but we can see what fraction of directories reported them, and then -we can extrapolate the total number in the network. - - -Q: How do you get from these directory requests to user numbers? -A: We put in the assumption that the average client makes 10 such requests -per day. A tor client that is connected 24/7 makes about 15 requests per -day, but not all clients are connected 24/7, so we picked the number 10 -for the average client. We simply divide directory requests by 10 and -consider the result as the number of users. Another way of looking at it, -is that we assume that each request represents a client that stays online -for 2 hours and 24 minutes. - - - -Q: So, are these distinct users per day, average number of users connected -over the day, or what? -A: Average number of concurrent users, estimated from data collected over -a day. We can't say how many distinct users there are. - - - -Q: Are these tor clients or users? What if there's more than one user -behind a tor client? -A: Then we count those users as one. We really count clients, but it's -more intuitive for most people to think of users, that's why we say users -and not clients. - - - -Q: What if a user runs tor on a laptop and changes their IP address a few -times per day? Don't you overcount that user? -A: No, because that user updates their list of relays as often as a user -that doesn't change IP address over the day. - - - -Q: How do you know which countries users come from? -A: The directories resolve IP addresses to country codes and report these -numbers in aggregate form. This is one of the reasons why tor ships with -a GeoIP database. - - - -Q: Why are there so few bridge users that are not using the default OR -protocol or that are using IPv6? -A: Very few bridges report data on transports or IP versions yet, and by -default we consider requests to use the default OR protocol and IPv4. -Once more bridges report these data, the numbers will become more -accurate. - - - -Q: Why do the graphs end 2 days in the past and not today? -A: Relays and bridges report some of the data in 24-hour intervals which -may end at any time of the day. And after such an interval is over relays -and bridges might take another 18 hours to report the data. We cut off -the last two days from the graphs, because we want to avoid that the last -data point in a graph indicates a recent trend change which is in fact -just an artifact of the algorithm. - - - -Q: But I noticed that the last data point went up/down a bit since I last -looked a few hours ago. Why is that? -A: You're an excellent observer! The reason is that we publish user -numbers once we're confident enough that they won't change significantly -anymore. But it's always possible that a directory reports data a few -hours after we were confident enough, but which then slightly changed the -graph. - - - -Q: Why are no numbers available before September 2011? -A: We do have descriptor archives from before that time, but those -descriptors didn't contain all the data we use to estimate user numbers. -We do have older user numbers from an earlier estimation approach -<a href="/data/old-user-number-estimates.tar.gz">here</a>, but we believe -the current approach is more accurate. - - - -Q: Why do you believe the current approach to estimate user numbers is -more accurate? -A: For direct users, we include all directories which we didn't do in the -old approach. We also use histories that only contain bytes written to -answer directory requests, which is more precise than using general byte -histories. - - - -Q: And what about the advantage of the current approach over the old one -when it comes to bridge users? -A: Oh, that's a whole different story. We wrote a 13 page long -<a href="https://research.torproject.org/techreports/counting-daily-bridge-users-201…">technical -report</a> explaining the reasons for retiring the old approach. -tl;dr: in the old approach we measured the wrong thing, and now we measure -the right thing. - - - -Q: Are the data and the source code for estimating these user numbers -available? -A: Sure, <a href="/data.html">data</a> and -<a href="https://gitweb.torproject.org/metrics-tasks.git/tree/HEAD:/task-8462">source -code</a> are publicly available. - - - -Q: What are these red and blue dots indicating possible censorship -events? -A: We run an anomaly-based censorship-detection system that looks at -estimated user numbers over a series of days and predicts the user number -in the next days. If the actual number is higher or lower, this might -indicate a possible censorship event or release of censorship. For more -details, see our -<a href="https://research.torproject.org/techreports/detector-2011-09-09.pdf">technical -report</a>. - +<a href="https://gitweb.torproject.org/metrics-web.git/blob/HEAD:/doc/users-q-and-a.…">Questions +and answers about users statistics</a> </div> </div>

1 0

[ooni-probe/master] Start implementing report log for keeping track of which reports have been created and which have failed.
by art＠torproject.org 26 Jun '14

26 Jun '14

commit 00ff99fe73158538b875c76e13e0ffc0249a86e9 Author: Arturo Filastò <art(a)fuffa.org> Date: Wed Jun 18 12:25:03 2014 +0200 Start implementing report log for keeping track of which reports have been created and which have failed. Related to: https://trac.torproject.org/projects/tor/ticket/11860 --- ooni/errors.py | 12 +++---- ooni/reporter.py | 57 ++++++++++++++++++++++++++++- ooni/settings.py | 2 ++ ooni/tests/test_reporter.py | 84 +++++++++++++++++++++++++++++++++++++------ 4 files changed, 135 insertions(+), 20 deletions(-) diff --git a/ooni/errors.py b/ooni/errors.py index 9f9ae75..fe6350b 100644 --- a/ooni/errors.py +++ b/ooni/errors.py @@ -190,10 +190,6 @@ class TorControlPortNotFound(Exception): pass -class ReportNotCreated(Exception): - pass - - class InsufficientPrivileges(Exception): pass @@ -202,10 +198,6 @@ class ProbeIPUnknown(Exception): pass -class GeoIPDataFilesNotFound(Exception): - pass - - class NoMoreReporters(Exception): pass @@ -282,6 +274,10 @@ class InvalidDestination(ReporterException): pass +class ReportLogExists(Exception): + pass + + def get_error(error_key): if error_key == 'test-helpers-key-missing': return CouldNotFindTestHelper diff --git a/ooni/reporter.py b/ooni/reporter.py index c5d07ae..7d7d709 100644 --- a/ooni/reporter.py +++ b/ooni/reporter.py @@ -401,7 +401,62 @@ class OONIBReporter(OReporter): def finish(self): url = self.collectorAddress + '/report/' + self.reportID + '/close' log.debug("Closing the report %s" % url) - response = yield self.agent.request("POST", str(url)) + yield self.agent.request("POST", str(url)) + + +class OONIBReportLog(object): + + def __init__(self, file_name=config.report_log_file): + self._lock = defer.DeferredLock() + self.file_name = file_name + + def create_report_log(self): + if os.path.exists(self.file_name): + raise errors.ReportLogExists + with open(self.file_name, 'w+') as f: + f.write(yaml.safe_dump({})) + + @contextmanager + def edit_report_log(self): + with open(self.file_name) as rfp: + report = yaml.safe_load(rfp) + with open(self.file_name, 'w+') as wfp: + yield report + wfp.write(yaml.safe_dump(report)) + + def _report_created(self, report_file, collector_address, report_id): + with self.edit_report_log() as report: + report[report_file] = { + 'created_at': datetime.now(), + 'status': 'created', + 'collector': collector_address, + 'report_id': report_id + } + + def report_created(self, report_file, collector_address, report_id): + return self._lock.run(self._report_created, report_file, + collector_address, report_id) + + def _report_creation_failed(self, report_file, collector_address): + with self.edit_report_log() as report: + report[report_file] = { + 'created_at': datetime.now(), + 'status': 'creation-failed', + 'collector': collector_address + } + + def report_creation_failed(self, report_file, collector_address): + return self._lock.run(self._report_creation_failed, report_file, + collector_address) + + def _report_closed(self, report_file): + with self.edit_report_log() as report: + if report[report_file]['status'] != "created": + raise errors.ReportNotCreated() + del report[report_file] + + def report_closed(self, report_file): + return self._lock.run(self._report_closed, report_file) class Report(object): diff --git a/ooni/settings.py b/ooni/settings.py index cd4e04b..0d1275e 100644 --- a/ooni/settings.py +++ b/ooni/settings.py @@ -8,6 +8,7 @@ from os.path import abspath, expanduser from ooni import otime, geoip from ooni.utils import Storage + class OConfig(object): _custom_home = None @@ -49,6 +50,7 @@ class OConfig(object): self.inputs_directory = os.path.join(self.ooni_home, 'inputs') self.decks_directory = os.path.join(self.ooni_home, 'decks') self.reports_directory = os.path.join(self.ooni_home, 'reports') + self.report_log_file = os.path.join(self.ooni_home, 'reporting.yml') if self.global_options.get('configfile'): config_file = self.global_options['configfile'] diff --git a/ooni/tests/test_reporter.py b/ooni/tests/test_reporter.py index 23c1190..5ca7bfa 100644 --- a/ooni/tests/test_reporter.py +++ b/ooni/tests/test_reporter.py @@ -1,3 +1,4 @@ +import os import yaml import json import time @@ -6,9 +7,9 @@ from mock import MagicMock from twisted.internet import defer from twisted.trial import unittest -from ooni.utils.net import StringProducer from ooni import errors as e -from ooni.reporter import YAMLReporter, OONIBReporter +from ooni.reporter import YAMLReporter, OONIBReporter, OONIBReportLog + class MockTest(object): _start_time = time.time() @@ -33,7 +34,9 @@ oonib_generic_error_message = { 'error': 'generic-error' } + class TestYAMLReporter(unittest.TestCase): + def setUp(self): pass @@ -51,37 +54,44 @@ class TestYAMLReporter(unittest.TestCase): entry = report_entries.next() # Check for first entry of report - assert all(x in entry \ - for x in ['report_content', 'input', \ - 'test_name', 'test_started', \ + assert all(x in entry + for x in ['report_content', 'input', + 'test_name', 'test_started', 'test_runtime']) + class TestOONIBReporter(unittest.TestCase): - + def setUp(self): self.mock_response = {} self.collector_address = 'http://example.com' - self.oonib_reporter = OONIBReporter(test_details, self.collector_address) + self.oonib_reporter = OONIBReporter( + test_details, + self.collector_address) self.oonib_reporter.agent = MagicMock() self.mock_agent_response = MagicMock() + def deliverBody(body_receiver): body_receiver.dataReceived(json.dumps(self.mock_response)) body_receiver.connectionLost(None) self.mock_agent_response.deliverBody = deliverBody - self.oonib_reporter.agent.request.return_value = defer.succeed(self.mock_agent_response) - + self.oonib_reporter.agent.request.return_value = defer.succeed( + self.mock_agent_response) + @defer.inlineCallbacks def test_create_report(self): self.mock_response = oonib_new_report_message yield self.oonib_reporter.createReport() - assert self.oonib_reporter.reportID == oonib_new_report_message['report_id'] + assert self.oonib_reporter.reportID == oonib_new_report_message[ + 'report_id'] @defer.inlineCallbacks def test_create_report_failure(self): self.mock_response = oonib_generic_error_message self.mock_agent_response.code = 406 - yield self.assertFailure(self.oonib_reporter.createReport(), e.OONIBReportCreationError) + yield self.assertFailure(self.oonib_reporter.createReport(), + e.OONIBReportCreationError) @defer.inlineCallbacks def test_write_report_entry(self): @@ -89,3 +99,55 @@ class TestOONIBReporter(unittest.TestCase): yield self.oonib_reporter.writeReportEntry(req) assert self.oonib_reporter.agent.request.called + +class TestOONIBReportLog(unittest.TestCase): + + def setUp(self): + self.report_log = OONIBReportLog('report_log') + self.report_log.create_report_log() + + def tearDown(self): + os.remove(self.report_log.file_name) + + @defer.inlineCallbacks + def test_report_created(self): + yield self.report_log.report_created("path_to_my_report.yaml", + 'httpo://foo.onion', + 'someid') + with open(self.report_log.file_name) as f: + report = yaml.safe_load(f) + assert "path_to_my_report.yaml" in report + + @defer.inlineCallbacks + def test_concurrent_edit(self): + d1 = self.report_log.report_created("path_to_my_report1.yaml", + 'httpo://foo.onion', + 'someid1') + d2 = self.report_log.report_created("path_to_my_report2.yaml", + 'httpo://foo.onion', + 'someid2') + yield defer.DeferredList([d1, d2]) + with open(self.report_log.file_name) as f: + report = yaml.safe_load(f) + assert "path_to_my_report1.yaml" in report + assert "path_to_my_report2.yaml" in report + + @defer.inlineCallbacks + def test_report_closed(self): + yield self.report_log.report_created("path_to_my_report.yaml", + 'httpo://foo.onion', + 'someid') + yield self.report_log.report_closed("path_to_my_report.yaml") + + with open(self.report_log.file_name) as f: + report = yaml.safe_load(f) + assert "path_to_my_report.yaml" not in report + + @defer.inlineCallbacks + def test_report_creation_failed(self): + yield self.report_log.report_creation_failed("path_to_my_report.yaml", + 'httpo://foo.onion') + with open(self.report_log.file_name) as f: + report = yaml.safe_load(f) + assert "path_to_my_report.yaml" in report + assert report["path_to_my_report.yaml"]["status"] == "creation-failed"

1 0

[ooni-probe/master] Cast two objects to string to avoid log_encode complaining when in debug mode.
by art＠torproject.org 26 Jun '14

26 Jun '14

commit 2d0289437255dd36460acc220a8226def261f24a Author: Arturo Filastò <art(a)fuffa.org> Date: Sat May 31 20:20:14 2014 +0200 Cast two objects to string to avoid log_encode complaining when in debug mode. --- ooni/managers.py | 2 +- ooni/templates/dnst.py | 8 +++----- 2 files changed, 4 insertions(+), 6 deletions(-) diff --git a/ooni/managers.py b/ooni/managers.py index f201a03..ca1cd5b 100644 --- a/ooni/managers.py +++ b/ooni/managers.py @@ -196,7 +196,7 @@ class ReportEntryManager(LinkedTaskManager): def succeeded(self, result, task): log.debug("Successfully performed report %s" % task) - log.debug(result) + log.debug(str(result)) def failed(self, failure, task): pass diff --git a/ooni/templates/dnst.py b/ooni/templates/dnst.py index 70c9c8e..7c9626c 100644 --- a/ooni/templates/dnst.py +++ b/ooni/templates/dnst.py @@ -3,13 +3,11 @@ # :authors: Arturo Filastò # :licence: see LICENSE -from twisted.internet import defer, udp, error, base +from twisted.internet import udp, error, base from twisted.internet.defer import TimeoutError from twisted.names import client, dns from twisted.names.client import Resolver -from twisted.names.error import DNSQueryRefusedError - from ooni.utils import log from ooni.nettest import NetTestCase from ooni.errors import failureToString @@ -32,7 +30,7 @@ def _bindSocket(self): # Make sure that if we listened on port 0, we update that to # reflect what the OS actually assigned us. self._realPortNumber = skt.getsockname()[1] - + # Here we remove the logging. # log.msg("%s starting on %s" % ( # self._getLogPrefix(self.protocol), self._realPortNumber)) @@ -144,7 +142,7 @@ class DNSTest(NetTestCase): query = [dns.Query(hostname, dnsType, dns.IN)] def gotResponse(message): log.debug(dns_type+" Lookup successful") - log.debug(message) + log.debug(str(message)) addrs = [] answers = [] if dns_server:

1 0

[ooni-probe/master] Handle also unhandled exceptions when starting the director.
by art＠torproject.org 26 Jun '14

26 Jun '14

commit 27f8f55d42d83174762cffab018638342a6f7ca1 Author: Arturo Filastò <art(a)fuffa.org> Date: Sat May 31 20:27:56 2014 +0200 Handle also unhandled exceptions when starting the director. --- ooni/oonicli.py | 23 +++++++++++++++-------- 1 file changed, 15 insertions(+), 8 deletions(-) diff --git a/ooni/oonicli.py b/ooni/oonicli.py index e29d20a..efcc7db 100644 --- a/ooni/oonicli.py +++ b/ooni/oonicli.py @@ -204,13 +204,15 @@ def runWithDirector(logging=True, start_tor=True): except errors.UnableToLoadDeckInput as error: return defer.failure.Failure(error) - def director_startup_failed(failure): - log.err("Failed to start the director") - r = failure.trap(errors.TorNotRunning, - errors.InvalidOONIBCollectorAddress, - errors.UnableToLoadDeckInput, errors.CouldNotFindTestHelper, - errors.CouldNotFindTestCollector, errors.ProbeIPUnknown, - errors.InvalidInputFile) + def director_startup_handled_failures(failure): + log.err("Could not start the director") + failure.trap(errors.TorNotRunning, + errors.InvalidOONIBCollectorAddress, + errors.UnableToLoadDeckInput, + errors.CouldNotFindTestHelper, + errors.CouldNotFindTestCollector, + errors.ProbeIPUnknown, + errors.InvalidInputFile) if isinstance(failure.value, errors.TorNotRunning): log.err("Tor does not appear to be running") @@ -245,6 +247,10 @@ def runWithDirector(logging=True, start_tor=True): if config.advanced.debug: log.exception(failure) + def director_startup_other_failures(failure): + log.err("An unhandled exception occurred while starting the director!") + log.exception(failure) + # Wait until director has started up (including bootstrapping Tor) # before adding tests def post_director_start(_): @@ -292,7 +298,8 @@ def runWithDirector(logging=True, start_tor=True): def start(): d.addCallback(setup_nettest) d.addCallback(post_director_start) - d.addErrback(director_startup_failed) + d.addErrback(director_startup_handled_failures) + d.addErrback(director_startup_other_failures) return d return start()

1 0

[ooni-probe/master] Pep8 related fixes.
by art＠torproject.org 26 Jun '14

26 Jun '14

commit 46eebbc9010481264d4ed02af96967ec7a55f8ad Author: Arturo Filastò <art(a)fuffa.org> Date: Wed Jun 18 11:52:05 2014 +0200 Pep8 related fixes. --- ooni/errors.py | 77 +++++++++++++++++++++++++++++++++----- ooni/reporter.py | 108 ++++++++++++++++++++++++++++++++---------------------- 2 files changed, 133 insertions(+), 52 deletions(-) diff --git a/ooni/errors.py b/ooni/errors.py index 3b62b2a..9f9ae75 100644 --- a/ooni/errors.py +++ b/ooni/errors.py @@ -15,23 +15,42 @@ from txsocksx.errors import HostUnreachable, ConnectionRefused from txsocksx.errors import TTLExpired, CommandNotSupported from socket import gaierror + + def handleAllFailures(failure): """ Here we make sure to trap all the failures that are supported by the failureToString function and we return the the string that represents the failure. """ - failure.trap(ConnectionRefusedError, gaierror, DNSLookupError, - TCPTimedOutError, ResponseNeverReceived, DeferTimeoutError, - GenericTimeoutError, - SOCKSError, MethodsNotAcceptedError, AddressNotSupported, - ConnectionError, NetworkUnreachable, ConnectionLostEarly, - ConnectionNotAllowed, NoAcceptableMethods, ServerFailure, - HostUnreachable, ConnectionRefused, TTLExpired, CommandNotSupported, - ConnectError, ConnectionLost, CancelledError) + failure.trap( + ConnectionRefusedError, + gaierror, + DNSLookupError, + TCPTimedOutError, + ResponseNeverReceived, + DeferTimeoutError, + GenericTimeoutError, + SOCKSError, + MethodsNotAcceptedError, + AddressNotSupported, + ConnectionError, + NetworkUnreachable, + ConnectionLostEarly, + ConnectionNotAllowed, + NoAcceptableMethods, + ServerFailure, + HostUnreachable, + ConnectionRefused, + TTLExpired, + CommandNotSupported, + ConnectError, + ConnectionLost, + CancelledError) return failureToString(failure) + def failureToString(failure): """ Given a failure instance return a string representing the kind of error @@ -119,7 +138,7 @@ def failureToString(failure): elif isinstance(failure.value, SOCKSError): log.err("Generic SOCKS error") string = 'socks_error' - + elif isinstance(failure.value, CancelledError): log.err("Task timed out") string = 'task_timed_out' @@ -130,99 +149,139 @@ def failureToString(failure): return string + class DirectorException(Exception): pass + class UnableToStartTor(DirectorException): pass + class InvalidOONIBCollectorAddress(Exception): pass + class InvalidOONIBBouncerAddress(Exception): pass + class AllReportersFailed(Exception): pass + class GeoIPDataFilesNotFound(Exception): pass + class ReportNotCreated(Exception): pass + class ReportAlreadyClosed(Exception): pass + class TorStateNotFound(Exception): pass + class TorControlPortNotFound(Exception): pass + class ReportNotCreated(Exception): pass + class InsufficientPrivileges(Exception): pass + class ProbeIPUnknown(Exception): pass + class GeoIPDataFilesNotFound(Exception): pass + class NoMoreReporters(Exception): pass + class TorNotRunning(Exception): pass + class OONIBError(Exception): pass + class OONIBReportError(OONIBError): pass + class OONIBReportUpdateError(OONIBReportError): pass + class OONIBReportCreationError(OONIBReportError): pass + class OONIBTestDetailsLookupError(OONIBReportError): pass + class UnableToLoadDeckInput(Exception): pass + class CouldNotFindTestHelper(Exception): pass + class CouldNotFindTestCollector(Exception): pass + class NetTestNotFound(Exception): pass + class MissingRequiredOption(Exception): pass + class FailureToLoadNetTest(Exception): pass + class NoPostProcessor(Exception): pass + class InvalidOption(Exception): pass + class TaskTimedOut(Exception): pass + class InvalidInputFile(Exception): pass + +class ReporterException(Exception): + pass + + +class InvalidDestination(ReporterException): + pass + + def get_error(error_key): if error_key == 'test-helpers-key-missing': return CouldNotFindTestHelper diff --git a/ooni/reporter.py b/ooni/reporter.py index 33b4fdf..c5d07ae 100644 --- a/ooni/reporter.py +++ b/ooni/reporter.py @@ -4,10 +4,13 @@ import json import os import re -from yaml.representer import * -from yaml.emitter import * -from yaml.serializer import * -from yaml.resolver import * +from datetime import datetime +from contextlib import contextmanager + +from yaml.representer import SafeRepresenter +from yaml.emitter import Emitter +from yaml.serializer import Serializer +from yaml.resolver import Resolver from twisted.python.util import untilConcludes from twisted.internet import defer from twisted.internet.error import ConnectionRefusedError @@ -20,6 +23,7 @@ try: from scapy.packet import Packet except ImportError: log.err("Scapy is not installed.") + class Packet(object): pass @@ -32,8 +36,7 @@ from ooni.utils.net import BodyReceiver, StringProducer from ooni.settings import config from ooni.tasks import ReportEntry, ReportTracker -class ReporterException(Exception): - pass + def createPacketReport(packet_list): """ @@ -45,16 +48,19 @@ def createPacketReport(packet_list): report = [] for packet in packet_list: report.append({'raw_packet': str(packet), - 'summary': str([packet])}) + 'summary': str([packet])}) return report + class OSafeRepresenter(SafeRepresenter): + """ This is a custom YAML representer that allows us to represent reports safely. It extends the SafeRepresenter to be able to also represent complex numbers and scapy packet. """ + def represent_data(self, data): """ This is very hackish. There is for sure a better way either by using @@ -82,29 +88,32 @@ class OSafeRepresenter(SafeRepresenter): OSafeRepresenter.add_representer(complex, OSafeRepresenter.represent_complex) + class OSafeDumper(Emitter, Serializer, OSafeRepresenter, Resolver): + """ This is a modification of the YAML Safe Dumper to use our own Safe Representer that supports complex numbers. """ + def __init__(self, stream, - default_style=None, default_flow_style=None, - canonical=None, indent=None, width=None, - allow_unicode=None, line_break=None, - encoding=None, explicit_start=None, explicit_end=None, - version=None, tags=None): + default_style=None, default_flow_style=None, + canonical=None, indent=None, width=None, + allow_unicode=None, line_break=None, + encoding=None, explicit_start=None, explicit_end=None, + version=None, tags=None): + Emitter.__init__(self, stream, canonical=canonical, - indent=indent, width=width, - allow_unicode=allow_unicode, line_break=line_break) + indent=indent, width=width, + allow_unicode=allow_unicode, line_break=line_break) Serializer.__init__(self, encoding=encoding, - explicit_start=explicit_start, explicit_end=explicit_end, - version=version, tags=tags) + explicit_start=explicit_start, + explicit_end=explicit_end, + version=version, tags=tags) OSafeRepresenter.__init__(self, default_style=default_style, - default_flow_style=default_flow_style) + default_flow_style=default_flow_style) Resolver.__init__(self) -class NoTestIDSpecified(Exception): - pass def safe_dump(data, stream=None, **kw): """ @@ -112,7 +121,9 @@ def safe_dump(data, stream=None, **kw): """ return yaml.dump_all([data], stream, Dumper=OSafeDumper, **kw) + class OReporter(object): + def __init__(self, test_details): self.testDetails = test_details @@ -149,10 +160,9 @@ class OReporter(object): return defer.maybeDeferred(self.writeReportEntry, test_report) -class InvalidDestination(ReporterException): - pass class YAMLReporter(OReporter): + """ These are useful functions for reporting to YAML format. @@ -160,11 +170,13 @@ class YAMLReporter(OReporter): the destination directory of the report """ - def __init__(self, test_details, report_destination='.', report_filename=None): + + def __init__(self, test_details, report_destination='.', + report_filename=None): self.reportDestination = report_destination if not os.path.isdir(report_destination): - raise InvalidDestination + raise errors.InvalidDestination if not report_filename: report_filename = "report-" + \ @@ -216,8 +228,11 @@ class YAMLReporter(OReporter): self._writeln("###########################################") - self._writeln("# OONI Probe Report for %s (%s)" % (self.testDetails['test_name'], - self.testDetails['test_version'])) + self._writeln("# OONI Probe Report for %s (%s)" % ( + self.testDetails['test_name'], + self.testDetails['test_version']) + ) + self._writeln("# %s" % otime.prettyDateNow()) self._writeln("###########################################") @@ -226,13 +241,16 @@ class YAMLReporter(OReporter): def finish(self): self._stream.close() + def collector_supported(collector_address): if collector_address.startswith('httpo') \ and (not (config.tor_state or config.tor.socks_port)): return False return True + class OONIBReporter(OReporter): + def __init__(self, test_details, collector_address): self.collectorAddress = collector_address self.validateCollectorAddress() @@ -265,7 +283,7 @@ class OONIBReporter(OReporter): url = self.collectorAddress + '/report' request = {'report_id': self.reportID, - 'content': content} + 'content': content} log.debug("Updating report with id %s (%s)" % (self.reportID, url)) request_json = json.dumps(request) @@ -274,8 +292,8 @@ class OONIBReporter(OReporter): bodyProducer = StringProducer(json.dumps(request)) try: - response = yield self.agent.request("PUT", url, - bodyProducer=bodyProducer) + yield self.agent.request("PUT", url, + bodyProducer=bodyProducer) except: # XXX we must trap this in the runner and make sure to report the # data later. @@ -298,10 +316,10 @@ class OONIBReporter(OReporter): if self.collectorAddress.startswith('httpo://'): self.collectorAddress = \ - self.collectorAddress.replace('httpo://', 'http://') - self.agent = SOCKS5Agent(reactor, - proxyEndpoint=TCP4ClientEndpoint(reactor, '127.0.0.1', - config.tor.socks_port)) + self.collectorAddress.replace('httpo://', 'http://') + proxyEndpoint = TCP4ClientEndpoint(reactor, '127.0.0.1', + config.tor.socks_port) + self.agent = SOCKS5Agent(reactor, proxyEndpoint=proxyEndpoint) elif self.collectorAddress.startswith('https://'): # XXX add support for securely reporting to HTTPS collectors. @@ -313,15 +331,16 @@ class OONIBReporter(OReporter): content += safe_dump(self.testDetails) content += '...\n' - request = {'software_name': self.testDetails['software_name'], + request = { + 'software_name': self.testDetails['software_name'], 'software_version': self.testDetails['software_version'], 'probe_asn': self.testDetails['probe_asn'], 'test_name': self.testDetails['test_name'], 'test_version': self.testDetails['test_version'], 'input_hashes': self.testDetails['input_hashes'], # XXX there is a bunch of redundancy in the arguments getting sent - # to the backend. This may need to get changed in the client and the - # backend. + # to the backend. This may need to get changed in the client and + # the backend. 'content': content } @@ -336,9 +355,11 @@ class OONIBReporter(OReporter): try: response = yield self.agent.request("POST", url, - bodyProducer=bodyProducer) + bodyProducer=bodyProducer) + except ConnectionRefusedError: - log.err("Connection to reporting backend failed (ConnectionRefusedError)") + log.err("Connection to reporting backend failed " + "(ConnectionRefusedError)") raise errors.OONIBReportCreationError except errors.HostUnreachable: @@ -366,8 +387,10 @@ class OONIBReporter(OReporter): if response.code == 406: # XXX make this more strict - log.err("The specified input or nettests cannot be submitted to this collector.") - log.msg("Try running a different test or try reporting to a different collector.") + log.err("The specified input or nettests cannot be submitted to " + "this collector.") + log.msg("Try running a different test or try reporting to a " + "different collector.") raise errors.OONIBReportCreationError self.reportID = parsed_response['report_id'] @@ -380,10 +403,9 @@ class OONIBReporter(OReporter): log.debug("Closing the report %s" % url) response = yield self.agent.request("POST", str(url)) -class ReportClosed(Exception): - pass class Report(object): + def __init__(self, reporters, reportEntryManager): """ This is an abstraction layer on top of all the configured reporters. @@ -499,7 +521,8 @@ class Report(object): log.err("Failed to write to %s reporter, giving up..." % reporter) self.reporters.remove(reporter) else: - log.err("Failed to write to (already) removed reporter %s" % reporter) + log.err("Failed to write to (already) removed reporter %s" % + reporter) # Don't forward the exception unless there are no more reporters if len(self.reporters) == 0: @@ -516,7 +539,6 @@ class Report(object): """ log.err("Failed to open %s reporter, giving up..." % reporter) log.err("Reporter %s failed, removing from report..." % reporter) - #log.exception(failure) if reporter in self.reporters: self.reporters.remove(reporter) # Don't forward the exception unless there are no more reporters

1 0

[ooni-probe/master] Implement system for keeping track of reports that have not been submitted.
by art＠torproject.org 26 Jun '14

26 Jun '14

commit 2e16103be4c892d8d02b7cefd4edc9ecd92833c3 Author: Arturo Filastò <art(a)fuffa.org> Date: Thu Jun 19 15:02:05 2014 +0200 Implement system for keeping track of reports that have not been submitted. --- ooni/director.py | 45 ++++--- ooni/nettest.py | 7 +- ooni/oonicli.py | 17 +-- ooni/reporter.py | 293 ++++++++++++++++++++++++------------------- ooni/tasks.py | 26 +--- ooni/tests/test_nettest.py | 32 +++-- ooni/tests/test_reporter.py | 12 +- 7 files changed, 220 insertions(+), 212 deletions(-) diff --git a/ooni/director.py b/ooni/director.py index 21ac9fc..1414184 100644 --- a/ooni/director.py +++ b/ooni/director.py @@ -13,10 +13,12 @@ from txtorcon import TorConfig, TorState, launch_tor, build_tor_connection from twisted.internet import defer, reactor from twisted.internet.endpoints import TCP4ClientEndpoint + class Director(object): + """ - Singleton object responsible for coordinating the Measurements Manager and the - Reporting Manager. + Singleton object responsible for coordinating the Measurements Manager + and the Reporting Manager. How this all looks like is as follows: @@ -55,6 +57,7 @@ class Director(object): +------+ """ + _scheduledTests = 0 # Only list NetTests belonging to these categories categories = ['blocking', 'manipulation'] @@ -91,9 +94,10 @@ class Director(object): def getNetTests(self): nettests = {} + def is_nettest(filename): return not filename == '__init__.py' \ - and filename.endswith('.py') + and filename.endswith('.py') for category in self.categories: dirname = os.path.join(config.nettest_directory, category) @@ -106,9 +110,11 @@ class Director(object): if nettest['id'] in nettests: log.err("Found a two tests with the same name %s, %s" % - (net_test_file, nettests[nettest['id']]['path'])) + (net_test_file, + nettests[nettest['id']]['path'])) else: - category = dirname.replace(config.nettest_directory, '') + category = dirname.replace(config.nettest_directory, + '') nettests[nettest['id']] = nettest return nettests @@ -214,7 +220,8 @@ class Director(object): self.allTestsDone.callback(None) @defer.inlineCallbacks - def startNetTest(self, net_test_loader, reporters): + def startNetTest(self, net_test_loader, report_filename, + collector_address=None): """ Create the Report for the NetTest and start the report NetTest. @@ -222,15 +229,19 @@ class Director(object): net_test_loader: an instance of :class:ooni.nettest.NetTestLoader """ + if self.allTestsDone.called: self.allTestsDone = defer.Deferred() if config.privacy.includepcap: if not config.reports.pcap: - config.reports.pcap = config.generate_pcap_filename(net_test_loader.testDetails) + config.reports.pcap = config.generate_pcap_filename( + net_test_loader.testDetails + ) self.startSniffing() - report = Report(reporters, self.reportEntryManager) + report = Report(net_test_loader.testDetails, report_filename, + self.reportEntryManager, collector_address) net_test = NetTest(net_test_loader, report) net_test.director = self @@ -255,7 +266,8 @@ class Director(object): config.scapyFactory = ScapyFactory(config.advanced.interface) if os.path.exists(config.reports.pcap): - log.msg("Report PCAP already exists with filename %s" % config.reports.pcap) + log.msg("Report PCAP already exists with filename %s" % + config.reports.pcap) log.msg("Renaming files with such name...") pushFilenameStack(config.reports.pcap) @@ -268,16 +280,16 @@ class Director(object): @defer.inlineCallbacks def getTorState(self): connection = TCP4ClientEndpoint(reactor, '127.0.0.1', - config.tor.control_port) + config.tor.control_port) config.tor_state = yield build_tor_connection(connection) - def startTor(self): """ Starts Tor Launches a Tor with :param: socks_port :param: control_port :param: tor_binary set in ooniprobe.conf """ log.msg("Starting Tor...") + @defer.inlineCallbacks def state_complete(state): config.tor_state = state @@ -328,9 +340,10 @@ class Director(object): if config.tor.bridges: tor_config.UseBridges = 1 if config.advanced.obfsproxy_binary: - tor_config.ClientTransportPlugin = \ - 'obfs2,obfs3 exec %s managed' % \ - config.advanced.obfsproxy_binary + tor_config.ClientTransportPlugin = ( + 'obfs2,obfs3 exec %s managed' % + config.advanced.obfsproxy_binary + ) bridges = [] with open(config.tor.bridges) as f: for bridge in f: @@ -347,12 +360,12 @@ class Director(object): tor_config.save() - if not hasattr(tor_config,'ControlPort'): + if not hasattr(tor_config, 'ControlPort'): control_port = int(randomFreePort()) tor_config.ControlPort = control_port config.tor.control_port = control_port - if not hasattr(tor_config,'SocksPort'): + if not hasattr(tor_config, 'SocksPort'): socks_port = int(randomFreePort()) tor_config.SocksPort = socks_port config.tor.socks_port = socks_port diff --git a/ooni/nettest.py b/ooni/nettest.py index 4598871..166a4f9 100644 --- a/ooni/nettest.py +++ b/ooni/nettest.py @@ -3,14 +3,12 @@ import re import time from hashlib import sha256 -from twisted.internet import defer, reactor +from twisted.internet import defer from twisted.trial.runner import filenameToModule from twisted.python import usage, reflect -from ooni import geoip from ooni.tasks import Measurement from ooni.utils import log, checkForRoot -from ooni import otime from ooni.settings import config from ooni import errors as e @@ -458,9 +456,6 @@ class NetTest(object): """ self.state.taskDone() - if len(self.report.reporters) == 0: - raise e.AllReportersFailed - return report_results def makeMeasurement(self, test_instance, test_method, test_input=None): diff --git a/ooni/oonicli.py b/ooni/oonicli.py index efcc7db..cc23ec5 100644 --- a/ooni/oonicli.py +++ b/ooni/oonicli.py @@ -13,7 +13,6 @@ from ooni import errors, __version__ from ooni.settings import config from ooni.director import Director from ooni.deck import Deck, nettest_to_path -from ooni.reporter import YAMLReporter, OONIBReporter from ooni.nettest import NetTestLoader from ooni.utils import log, checkForRoot @@ -280,19 +279,9 @@ def runWithDirector(logging=True, start_tor=True): test_details = net_test_loader.testDetails test_details['annotations'] = global_options['annotations'] - yaml_reporter = YAMLReporter(test_details, - report_filename=global_options['reportfile']) - reporters = [yaml_reporter] - - if collector: - log.msg("Reporting using collector: %s" % collector) - try: - oonib_reporter = OONIBReporter(test_details, collector) - reporters.append(oonib_reporter) - except errors.InvalidOONIBCollectorAddress, e: - raise e - - netTestDone = director.startNetTest(net_test_loader, reporters) + director.startNetTest(net_test_loader, + global_options['reportfile'], + collector) return director.allTestsDone def start(): diff --git a/ooni/reporter.py b/ooni/reporter.py index 7d7d709..9869837 100644 --- a/ooni/reporter.py +++ b/ooni/reporter.py @@ -35,7 +35,7 @@ from ooni.utils.net import BodyReceiver, StringProducer from ooni.settings import config -from ooni.tasks import ReportEntry, ReportTracker +from ooni.tasks import ReportEntry def createPacketReport(packet_list): @@ -189,7 +189,7 @@ class YAMLReporter(OReporter): log.msg("Report already exists with filename %s" % report_path) pushFilenameStack(report_path) - self.report_path = report_path + self.report_path = os.path.abspath(report_path) OReporter.__init__(self, test_details) def _writeln(self, line): @@ -396,6 +396,7 @@ class OONIBReporter(OReporter): self.reportID = parsed_response['report_id'] self.backendVersion = parsed_response['backend_version'] log.debug("Created report with id %s" % parsed_response['report_id']) + defer.returnValue(parsed_response['report_id']) @defer.inlineCallbacks def finish(self): @@ -406,26 +407,56 @@ class OONIBReporter(OReporter): class OONIBReportLog(object): + """ + Used to keep track of report creation on a collector backend. + """ + def __init__(self, file_name=config.report_log_file): - self._lock = defer.DeferredLock() self.file_name = file_name + self.create_report_log() + + def run(self, f, *arg, **kw): + lock = defer.DeferredFilesystemLock(self.file_name + '.lock') + d = lock.deferUntilLocked() + + def unlockAndReturn(r): + lock.unlock() + return r + + def execute(_): + d = defer.maybeDeferred(f, *arg, **kw) + d.addBoth(unlockAndReturn) + return d + + d.addCallback(execute) + return d def create_report_log(self): - if os.path.exists(self.file_name): - raise errors.ReportLogExists - with open(self.file_name, 'w+') as f: - f.write(yaml.safe_dump({})) + if not os.path.exists(self.file_name): + with open(self.file_name, 'w+') as f: + f.write(yaml.safe_dump({})) @contextmanager - def edit_report_log(self): + def edit_log(self): with open(self.file_name) as rfp: report = yaml.safe_load(rfp) with open(self.file_name, 'w+') as wfp: yield report wfp.write(yaml.safe_dump(report)) - def _report_created(self, report_file, collector_address, report_id): - with self.edit_report_log() as report: + def _not_created(self, report_file): + with self.edit_log() as report: + report[report_file] = { + 'created_at': datetime.now(), + 'status': 'not-created', + 'collector': None + } + + def not_created(self, report_file): + return self.run(self._not_created, report_file) + + def _created(self, report_file, collector_address, report_id): + with self.edit_log() as report: report[report_file] = { 'created_at': datetime.now(), 'status': 'created', @@ -433,35 +464,45 @@ class OONIBReportLog(object): 'report_id': report_id } - def report_created(self, report_file, collector_address, report_id): - return self._lock.run(self._report_created, report_file, - collector_address, report_id) + def created(self, report_file, collector_address, report_id): + return self.run(self._created, report_file, + collector_address, report_id) - def _report_creation_failed(self, report_file, collector_address): - with self.edit_report_log() as report: + def _creation_failed(self, report_file, collector_address): + with self.edit_log() as report: report[report_file] = { 'created_at': datetime.now(), 'status': 'creation-failed', 'collector': collector_address } - def report_creation_failed(self, report_file, collector_address): - return self._lock.run(self._report_creation_failed, report_file, - collector_address) + def creation_failed(self, report_file, collector_address): + return self.run(self._creation_failed, report_file, + collector_address) - def _report_closed(self, report_file): - with self.edit_report_log() as report: + def _incomplete(self, report_file): + with self.edit_log() as report: + if report[report_file]['status'] != "created": + raise errors.ReportNotCreated() + report[report_file]['status'] = 'incomplete' + + def incomplete(self, report_file): + return self.run(self._incomplete, report_file) + + def _closed(self, report_file): + with self.edit_log() as report: if report[report_file]['status'] != "created": raise errors.ReportNotCreated() del report[report_file] - def report_closed(self, report_file): - return self._lock.run(self._report_closed, report_file) + def closed(self, report_file): + return self.run(self._closed, report_file) class Report(object): - def __init__(self, reporters, reportEntryManager): + def __init__(self, test_details, report_filename, + reportEntryManager, collector_address=None): """ This is an abstraction layer on top of all the configured reporters. @@ -469,53 +510,79 @@ class Report(object): Args: - reporters: - a list of :class:ooni.reporter.OReporter instances + test_details: + A dictionary containing the test details. + + report_filename: + The file path for the report to be written. reportEntryManager: an instance of :class:ooni.tasks.ReportEntryManager + + collector: + The address of the oonib collector for this report. + """ - self.reporters = reporters + self.test_details = test_details + self.collector_address = collector_address + + self.report_log = OONIBReportLog() + + self.yaml_reporter = YAMLReporter(test_details, + report_filename=report_filename) + self.report_filename = self.yaml_reporter.report_path + + self.oonib_reporter = None + if collector_address: + self.oonib_reporter = OONIBReporter(test_details, + collector_address) self.done = defer.Deferred() self.reportEntryManager = reportEntryManager - self._reporters_openned = 0 - self._reporters_written = 0 - self._reporters_closed = 0 + def open_oonib_reporter(self): + def creation_failed(failure): + self.oonib_reporter = None + return self.report_log.creation_failed(self.report_filename) + + def created(report_id): + return self.report_log.created(self.report_filename, + self.collector_address, + report_id) + + d = self.oonib_reporter.createReport() + d.addErrback(creation_failed) + d.addCallback(created) + return d def open(self): """ This will create all the reports that need to be created and fires the created callback of the reporter whose report got created. """ - all_openned = defer.Deferred() + d = defer.Deferred() + deferreds = [] - def are_all_openned(): - if len(self.reporters) == self._reporters_openned: - all_openned.callback(self._reporters_openned) + def yaml_report_failed(failure): + d.errback(failure) - for reporter in self.reporters[:]: + def all_reports_openned(result): + if not d.called: + d.callback(None) - def report_created(result): - log.debug("Created report with %s" % reporter) - self._reporters_openned += 1 - are_all_openned() + if self.oonib_reporter: + deferreds.append(self.open_oonib_reporter()) + else: + deferreds.append(self.report_log.not_created(self.report_filename)) - def report_failed(failure): - try: - self.failedOpeningReport(failure, reporter) - except errors.NoMoreReporters, e: - all_openned.errback(defer.fail(e)) - else: - are_all_openned() - return + yaml_report_created = \ + defer.maybeDeferred(self.yaml_reporter.createReport) + yaml_report_created.addErrback(yaml_report_failed) - d = defer.maybeDeferred(reporter.createReport) - d.addCallback(report_created) - d.addErrback(report_failed) + dl = defer.DeferredList(deferreds) + dl.addCallback(all_reports_openned) - return all_openned + return d def write(self, measurement): """ @@ -532,75 +599,34 @@ class Report(object): been written or errbacks when no more reporters """ - all_written = defer.Deferred() - report_tracker = ReportTracker(self.reporters) - - for reporter in self.reporters[:]: - def report_completed(task): - report_tracker.completed() - if report_tracker.finished(): - all_written.callback(report_tracker) - - def report_failed(failure): - log.debug("Report Write Failure") - try: - report_tracker.failedReporters.append(reporter) - self.failedWritingReport(failure, reporter) - except errors.NoMoreReporters, e: - log.err("No More Reporters!") - all_written.errback(defer.fail(e)) - else: - report_tracker.completed() - if report_tracker.finished(): - all_written.callback(report_tracker) - return - - report_entry_task = ReportEntry(reporter, measurement) - self.reportEntryManager.schedule(report_entry_task) - - report_entry_task.done.addCallback(report_completed) - report_entry_task.done.addErrback(report_failed) - - return all_written - - def failedWritingReport(self, failure, reporter): - """ - This errback gets called every time we fail to write a report. - By fail we mean that the number of retries has exceeded. - Once a report has failed to be written with a reporter we give up and - remove the reporter from the list of reporters to write to. - """ + d = defer.Deferred() + deferreds = [] - # XXX: may have been removed already by another failure. - if reporter in self.reporters: - log.err("Failed to write to %s reporter, giving up..." % reporter) - self.reporters.remove(reporter) - else: - log.err("Failed to write to (already) removed reporter %s" % - reporter) + def yaml_report_failed(failure): + d.errback(failure) - # Don't forward the exception unless there are no more reporters - if len(self.reporters) == 0: - log.err("Removed last reporter %s" % reporter) - raise errors.NoMoreReporters - return + def oonib_report_failed(failure): + return self.report_log.incomplete(self.report_filename) - def failedOpeningReport(self, failure, reporter): - """ - This errback get's called every time we fail to create a report. - By fail we mean that the number of retries has exceeded. - Once a report has failed to be created with a reporter we give up and - remove the reporter from the list of reporters to write to. - """ - log.err("Failed to open %s reporter, giving up..." % reporter) - log.err("Reporter %s failed, removing from report..." % reporter) - if reporter in self.reporters: - self.reporters.remove(reporter) - # Don't forward the exception unless there are no more reporters - if len(self.reporters) == 0: - log.err("Removed last reporter %s" % reporter) - raise errors.NoMoreReporters - return + def all_reports_written(_): + if not d.called: + d.callback(None) + + write_yaml_report = ReportEntry(self.yaml_reporter, measurement) + self.reportEntryManager.schedule(write_yaml_report) + write_yaml_report.done.addErrback(yaml_report_failed) + deferreds.append(write_yaml_report.done) + + if self.oonib_reporter: + write_oonib_report = ReportEntry(self.oonib_reporter, measurement) + self.reportEntryManager.schedule(write_oonib_report) + write_oonib_report.done.addErrback(oonib_report_failed) + deferreds.append(write_oonib_report.done) + + dl = defer.DeferredList(deferreds) + dl.addCallback(all_reports_written) + + return d def close(self): """ @@ -611,20 +637,29 @@ class Report(object): all the reports have been closed. """ - all_closed = defer.Deferred() + d = defer.Deferred() + deferreds = [] + + def yaml_report_failed(failure): + d.errback(failure) + + def oonib_report_closed(result): + return self.report_log.closed(self.report_filename) + + def all_reports_closed(_): + if not d.called: + d.callback(None) - for reporter in self.reporters[:]: - def report_closed(result): - self._reporters_closed += 1 - if len(self.reporters) == self._reporters_closed: - all_closed.callback(self._reporters_closed) + close_yaml = defer.maybeDeferred(self.yaml_reporter.finish) + close_yaml.addErrback(yaml_report_failed) + deferreds.append(close_yaml) - def report_failed(failure): - log.err("Failed closing report") - log.exception(failure) + if self.oonib_reporter: + close_oonib = self.oonib_reporter.finish() + close_oonib.addCallback(oonib_report_closed) + deferreds.append(close_oonib) - d = defer.maybeDeferred(reporter.finish) - d.addCallback(report_closed) - d.addErrback(report_failed) + dl = defer.DeferredList(deferreds) + dl.addCallback(all_reports_closed) - return all_closed + return d diff --git a/ooni/tasks.py b/ooni/tasks.py index 578d8a9..a869254 100644 --- a/ooni/tasks.py +++ b/ooni/tasks.py @@ -126,34 +126,12 @@ class Measurement(TaskWithTimeout): def run(self): return self.netTestMethod() -class ReportTracker(object): - def __init__(self, reporters): - self.report_completed = 0 - self.reporters = reporters - self.failedReporters = [] - - def finished(self): - """ - Returns true if all the tasks are done. False if not. - """ - # If a reporter fails and is removed, the report - # is considered completed but failed, but the number - # of reporters is now decreased by the number of failed - # reporters. - if self.report_completed == (len(self.reporters) + len(self.failedReporters)): - return True - return False - - def completed(self): - """ - Called when a new report is completed. - """ - self.report_completed += 1 class ReportEntry(TaskWithTimeout): + def __init__(self, reporter, entry): self.reporter = reporter - self.entry = entry + self.entry = entry if config.advanced.reporting_timeout: self.timeout = config.advanced.reporting_timeout diff --git a/ooni/tests/test_nettest.py b/ooni/tests/test_nettest.py index 92278ce..6a550d6 100644 --- a/ooni/tests/test_nettest.py +++ b/ooni/tests/test_nettest.py @@ -1,22 +1,15 @@ import os -from StringIO import StringIO -from tempfile import TemporaryFile, mkstemp +from tempfile import mkstemp from twisted.trial import unittest from twisted.internet import defer, reactor from twisted.python.usage import UsageError from ooni.settings import config -from ooni.errors import MissingRequiredOption, InvalidOption, FailureToLoadNetTest +from ooni.errors import MissingRequiredOption from ooni.nettest import NetTest, NetTestLoader -from ooni.tasks import BaseTask from ooni.director import Director -from ooni.managers import TaskManager - -from ooni.tests.mocks import MockMeasurement, MockMeasurementFailOnce -from ooni.tests.mocks import MockNetTest, MockDirector, MockReporter -from ooni.tests.mocks import MockMeasurementManager net_test_string = """ from twisted.python import usage @@ -95,7 +88,6 @@ from ooni.errors import failureToString, handleAllFailures class UsageOptions(usage.Options): optParameters = [ ['url', 'u', None, 'Specify a single URL to test.'], - ['factor', 'f', 0.8, 'What factor should be used for triggering censorship (0.8 == 80%)'] ] class HTTPBasedTest(httpt.HTTPTest): @@ -107,15 +99,17 @@ class HTTPBasedTest(httpt.HTTPTest): dummyInputs = range(1) dummyArgs = ('--spam', 'notham') -dummyOptions = {'spam':'notham'} +dummyOptions = {'spam': 'notham'} dummyInvalidArgs = ('--cram', 'jam') -dummyInvalidOptions= {'cram':'jam'} +dummyInvalidOptions = {'cram': 'jam'} dummyArgsWithRequiredOptions = ('--foo', 'moo', '--bar', 'baz') -dummyRequiredOptions = {'foo':'moo', 'bar':'baz'} +dummyRequiredOptions = {'foo': 'moo', 'bar': 'baz'} dummyArgsWithFile = ('--spam', 'notham', '--file', 'dummyInputFile.txt') + class TestNetTest(unittest.TestCase): timeout = 1 + def setUp(self): with open('dummyInputFile.txt', 'w') as f: for i in range(10): @@ -211,7 +205,7 @@ class TestNetTest(unittest.TestCase): ntl.loadNetTestString(net_test_string_with_file) ntl.checkOptions() - nt = NetTest(ntl,None) + nt = NetTest(ntl, None) nt.initializeInputProcessor() # XXX: if you use the same test_class twice you will have consumed all @@ -249,7 +243,7 @@ class TestNetTest(unittest.TestCase): ntl.checkOptions() director = Director() - d = director.startNetTest(ntl, [MockReporter()]) + d = director.startNetTest(ntl, 'dummy_report.yaml') @d.addCallback def complete(result): @@ -259,24 +253,28 @@ class TestNetTest(unittest.TestCase): return d def test_require_root_succeed(self): - #XXX: will require root to run + # XXX: will require root to run ntl = NetTestLoader(dummyArgs) ntl.loadNetTestString(net_test_root_required) for test_class, method in ntl.testCases: self.assertTrue(test_class.requiresRoot) + class TestNettestTimeout(unittest.TestCase): + @defer.inlineCallbacks def setUp(self): from twisted.internet.protocol import Protocol, Factory from twisted.internet.endpoints import TCP4ServerEndpoint class DummyProtocol(Protocol): + def dataReceived(self, data): pass class DummyFactory(Factory): + def __init__(self): self.protocols = [] @@ -306,7 +304,7 @@ class TestNettestTimeout(unittest.TestCase): ntl.checkOptions() director = Director() - d = director.startNetTest(ntl, [MockReporter()]) + d = director.startNetTest(ntl, 'dummy_report.yaml') @d.addCallback def complete(result): diff --git a/ooni/tests/test_reporter.py b/ooni/tests/test_reporter.py index 5ca7bfa..69d9775 100644 --- a/ooni/tests/test_reporter.py +++ b/ooni/tests/test_reporter.py @@ -111,7 +111,7 @@ class TestOONIBReportLog(unittest.TestCase): @defer.inlineCallbacks def test_report_created(self): - yield self.report_log.report_created("path_to_my_report.yaml", + yield self.report_log.created("path_to_my_report.yaml", 'httpo://foo.onion', 'someid') with open(self.report_log.file_name) as f: @@ -120,10 +120,10 @@ class TestOONIBReportLog(unittest.TestCase): @defer.inlineCallbacks def test_concurrent_edit(self): - d1 = self.report_log.report_created("path_to_my_report1.yaml", + d1 = self.report_log.created("path_to_my_report1.yaml", 'httpo://foo.onion', 'someid1') - d2 = self.report_log.report_created("path_to_my_report2.yaml", + d2 = self.report_log.created("path_to_my_report2.yaml", 'httpo://foo.onion', 'someid2') yield defer.DeferredList([d1, d2]) @@ -134,10 +134,10 @@ class TestOONIBReportLog(unittest.TestCase): @defer.inlineCallbacks def test_report_closed(self): - yield self.report_log.report_created("path_to_my_report.yaml", + yield self.report_log.created("path_to_my_report.yaml", 'httpo://foo.onion', 'someid') - yield self.report_log.report_closed("path_to_my_report.yaml") + yield self.report_log.closed("path_to_my_report.yaml") with open(self.report_log.file_name) as f: report = yaml.safe_load(f) @@ -145,7 +145,7 @@ class TestOONIBReportLog(unittest.TestCase): @defer.inlineCallbacks def test_report_creation_failed(self): - yield self.report_log.report_creation_failed("path_to_my_report.yaml", + yield self.report_log.creation_failed("path_to_my_report.yaml", 'httpo://foo.onion') with open(self.report_log.file_name) as f: report = yaml.safe_load(f)

1 0

[ooni-probe/master] Improve readablity of output by removing some log messages.
by art＠torproject.org 26 Jun '14

26 Jun '14

commit c0a1e7fe37d6e7d59ac18f568435e6287d110551 Author: Arturo Filastò <art(a)fuffa.org> Date: Thu Jun 19 15:40:00 2014 +0200 Improve readablity of output by removing some log messages. --- ooni/director.py | 2 +- ooni/errors.py | 42 ++++++++++++++++++++---------------------- ooni/managers.py | 4 ++-- 3 files changed, 23 insertions(+), 25 deletions(-) diff --git a/ooni/director.py b/ooni/director.py index 1414184..901088f 100644 --- a/ooni/director.py +++ b/ooni/director.py @@ -194,7 +194,7 @@ class Director(object): return measurement def measurementFailed(self, failure, measurement): - log.msg("Failed doing measurement: %s" % measurement) + log.debug("Failed doing measurement: %s" % measurement) self.totalMeasurementRuntime += measurement.runtime self.failedMeasurements += 1 diff --git a/ooni/errors.py b/ooni/errors.py index fe6350b..d5673ed 100644 --- a/ooni/errors.py +++ b/ooni/errors.py @@ -64,87 +64,85 @@ def failureToString(failure): A string representing the HTTP response error message. """ - from ooni.utils import log - string = None if isinstance(failure.value, ConnectionRefusedError): - log.err("Connection refused.") + # log.err("Connection refused.") string = 'connection_refused_error' elif isinstance(failure.value, ConnectionLost): - log.err("Connection lost.") + # log.err("Connection lost.") string = 'connection_lost_error' elif isinstance(failure.value, ConnectError): - log.err("Connect error.") + # log.err("Connect error.") string = 'connect_error' elif isinstance(failure.value, gaierror): - log.err("Address family for hostname not supported") + # log.err("Address family for hostname not supported") string = 'address_family_not_supported_error' elif isinstance(failure.value, DNSLookupError): - log.err("DNS lookup failure") + # log.err("DNS lookup failure") string = 'dns_lookup_error' elif isinstance(failure.value, TCPTimedOutError): - log.err("TCP Timed Out Error") + # log.err("TCP Timed Out Error") string = 'tcp_timed_out_error' elif isinstance(failure.value, ResponseNeverReceived): - log.err("Response Never Received") + # log.err("Response Never Received") string = 'response_never_received' elif isinstance(failure.value, DeferTimeoutError): - log.err("Deferred Timeout Error") + # log.err("Deferred Timeout Error") string = 'deferred_timeout_error' elif isinstance(failure.value, GenericTimeoutError): - log.err("Time Out Error") + # log.err("Time Out Error") string = 'generic_timeout_error' elif isinstance(failure.value, ServerFailure): - log.err("SOCKS error: ServerFailure") + # log.err("SOCKS error: ServerFailure") string = 'socks_server_failure' elif isinstance(failure.value, ConnectionNotAllowed): - log.err("SOCKS error: ConnectionNotAllowed") + # log.err("SOCKS error: ConnectionNotAllowed") string = 'socks_connection_not_allowed' elif isinstance(failure.value, NetworkUnreachable): - log.err("SOCKS error: NetworkUnreachable") + # log.err("SOCKS error: NetworkUnreachable") string = 'socks_network_unreachable' elif isinstance(failure.value, HostUnreachable): - log.err("SOCKS error: HostUnreachable") + # log.err("SOCKS error: HostUnreachable") string = 'socks_host_unreachable' elif isinstance(failure.value, ConnectionRefused): - log.err("SOCKS error: ConnectionRefused") + # log.err("SOCKS error: ConnectionRefused") string = 'socks_connection_refused' elif isinstance(failure.value, TTLExpired): - log.err("SOCKS error: TTLExpired") + # log.err("SOCKS error: TTLExpired") string = 'socks_ttl_expired' elif isinstance(failure.value, CommandNotSupported): - log.err("SOCKS error: CommandNotSupported") + # log.err("SOCKS error: CommandNotSupported") string = 'socks_command_not_supported' elif isinstance(failure.value, AddressNotSupported): - log.err("SOCKS error: AddressNotSupported") + # log.err("SOCKS error: AddressNotSupported") string = 'socks_address_not_supported' elif isinstance(failure.value, SOCKSError): - log.err("Generic SOCKS error") + # log.err("Generic SOCKS error") string = 'socks_error' elif isinstance(failure.value, CancelledError): - log.err("Task timed out") + # log.err("Task timed out") string = 'task_timed_out' else: - log.err("Unknown failure type: %s" % type(failure.value)) + # log.err("Unknown failure type: %s" % type(failure.value)) string = 'unknown_failure %s' % str(failure.value) return string diff --git a/ooni/managers.py b/ooni/managers.py index ca1cd5b..d03d2c9 100644 --- a/ooni/managers.py +++ b/ooni/managers.py @@ -29,7 +29,7 @@ class TaskManager(object): The has failed to complete, we append it to the end of the task chain to be re-run once all the currently scheduled tasks have run. """ - log.err("Task %s has failed %s times" % (task, task.failures)) + log.debug("Task %s has failed %s times" % (task, task.failures)) if config.advanced.debug: log.exception(failure) @@ -42,7 +42,7 @@ class TaskManager(object): else: # This fires the errback when the task is done but has failed. - log.err('Permanent failure for %s' % task) + log.debug('Permanent failure for %s' % task) task.done.errback(failure) self._fillSlots()

1 0

[ooni-probe/master] Fix bug that lead to some reports not being submitted.
by art＠torproject.org 26 Jun '14

26 Jun '14

commit 74a146834ee6609fd4f66c0b5f5381654acf8feb Author: Arturo Filastò <art(a)fuffa.org> Date: Thu Jun 19 16:06:50 2014 +0200 Fix bug that lead to some reports not being submitted. Pep8 related fixes. --- ooni/deck.py | 14 ++++--- ooni/managers.py | 22 ++++++---- ooni/nettest.py | 123 +++++++++++++++++++++++++++++++++--------------------- ooni/oonicli.py | 81 +++++++++++++++++++---------------- ooni/reporter.py | 3 +- 5 files changed, 144 insertions(+), 99 deletions(-) diff --git a/ooni/deck.py b/ooni/deck.py index ad90d03..f38b098 100644 --- a/ooni/deck.py +++ b/ooni/deck.py @@ -7,10 +7,9 @@ from ooni.utils import log from ooni import errors as e from twisted.python.filepath import FilePath -from twisted.internet import reactor, defer +from twisted.internet import defer import os -import re import yaml import json from hashlib import sha256 @@ -187,12 +186,15 @@ class Deck(InputFile): response = yield self.oonibclient.lookupTestHelpers(required_test_helpers) for net_test_loader in self.netTestLoaders: - log.msg("Setting collector and test helpers for %s" % net_test_loader.testDetails['test_name']) + log.msg("Setting collector and test helpers for %s" % + net_test_loader.testDetails['test_name']) # Only set the collector if the no collector has been specified # from the command line or via the test deck. - if not required_test_helpers and net_test_loader in requires_collector: - log.msg("Using the default collector: %s" % response['default']['collector']) + if not net_test_loader.requiredTestHelpers and \ + net_test_loader in requires_collector: + log.msg("Using the default collector: %s" % + response['default']['collector']) net_test_loader.collector = response['default']['collector'].encode('utf-8') continue @@ -223,6 +225,6 @@ class Deck(InputFile): try: input_file.verify() except AssertionError: - raise e.UnableToLoadDeckInput, cached_path + raise e.UnableToLoadDeckInput i['test_class'].localOptions[i['key']] = input_file.cached_file diff --git a/ooni/managers.py b/ooni/managers.py index d03d2c9..65e4ce1 100644 --- a/ooni/managers.py +++ b/ooni/managers.py @@ -1,9 +1,9 @@ import itertools -from twisted.internet import defer from ooni.utils import log from ooni.settings import config + def makeIterable(item): """ Takes as argument or an iterable and if it's not an iterable object then it @@ -15,6 +15,7 @@ def makeIterable(item): iterable = iter([item]) return iterable + class TaskManager(object): retries = 2 concurrency = 10 @@ -60,7 +61,7 @@ class TaskManager(object): self._run(task) except StopIteration: break - except ValueError as exc: + except ValueError: # XXX this is a workaround the race condition that leads the # _tasks generator to throw the exception # ValueError: generator already called. @@ -103,8 +104,8 @@ class TaskManager(object): def schedule(self, task_or_task_iterator): """ - Takes as argument a single task or a task iterable and appends it to the task - generator queue. + Takes as argument a single task or a task iterable and appends it to + the task generator queue. """ log.debug("Starting this task %s" % repr(task_or_task_iterator)) @@ -136,7 +137,9 @@ class TaskManager(object): """ raise NotImplemented + class LinkedTaskManager(TaskManager): + def __init__(self): super(LinkedTaskManager, self).__init__() self.child = None @@ -160,10 +163,13 @@ class LinkedTaskManager(TaskManager): if self.parent: self.parent._fillSlots() + class MeasurementManager(LinkedTaskManager): + """ - This is the Measurement Tracker. In here we keep track of active measurements - and issue new measurements once the active ones have been completed. + This is the Measurement Tracker. In here we keep track of active + measurements and issue new measurements once the active ones have been + completed. MeasurementTracker does not keep track of the typology of measurements that it is running. It just considers a measurement something that has an input @@ -172,6 +178,7 @@ class MeasurementManager(LinkedTaskManager): NetTest on the contrary is aware of the typology of measurements that it is dispatching as they are logically grouped by test file. """ + def __init__(self): if config.advanced.measurement_retries: self.retries = config.advanced.measurement_retries @@ -186,7 +193,9 @@ class MeasurementManager(LinkedTaskManager): def failed(self, failure, measurement): pass + class ReportEntryManager(LinkedTaskManager): + def __init__(self): if config.advanced.reporting_retries: self.retries = config.advanced.reporting_retries @@ -200,4 +209,3 @@ class ReportEntryManager(LinkedTaskManager): def failed(self, failure, task): pass - diff --git a/ooni/nettest.py b/ooni/nettest.py index 166a4f9..1547617 100644 --- a/ooni/nettest.py +++ b/ooni/nettest.py @@ -16,9 +16,11 @@ from ooni import errors as e from inspect import getmembers from StringIO import StringIO + class NoTestCasesFound(Exception): pass + def get_test_methods(item, method_prefix="test_"): """ Look for test_ methods in subclasses of NetTestCase @@ -36,6 +38,7 @@ def get_test_methods(item, method_prefix="test_"): pass return test_cases + def getTestClassFromFile(net_test_file): """ Will return the first class that is an instance of NetTestCase. @@ -51,10 +54,12 @@ def getTestClassFromFile(net_test_file): except (TypeError, AssertionError): pass + def getOption(opt_parameter, required_options, type='text'): """ Arguments: - usage_options: a list as should be the optParameters of an UsageOptions class. + usage_options: a list as should be the optParameters of an UsageOptions + class. required_options: a list containing the strings of the options that are required. @@ -77,16 +82,19 @@ def getOption(opt_parameter, required_options, type='text'): required = False return {'description': description, - 'value': default, 'required': required, - 'type': type - } + 'value': default, 'required': required, + 'type': type + } + def getArguments(test_class): arguments = {} if test_class.inputFile: option_name = test_class.inputFile[0] - arguments[option_name] = getOption(test_class.inputFile, - test_class.requiredOptions, type='file') + arguments[option_name] = getOption( + test_class.inputFile, + test_class.requiredOptions, + type='file') try: list(test_class.usageOptions.optParameters) except AttributeError: @@ -94,16 +102,20 @@ def getArguments(test_class): for opt_parameter in test_class.usageOptions.optParameters: option_name = opt_parameter[0] - opt_type="text" + opt_type = "text" if opt_parameter[3].lower().startswith("file"): - opt_type="file" - arguments[option_name] = getOption(opt_parameter, - test_class.requiredOptions, type=opt_type) + opt_type = "file" + arguments[option_name] = getOption( + opt_parameter, + test_class.requiredOptions, + type=opt_type) return arguments + def test_class_name_to_name(test_class_name): - return test_class_name.lower().replace(' ','_') + return test_class_name.lower().replace(' ', '_') + def getNetTestInformation(net_test_file): """ @@ -122,21 +134,23 @@ def getNetTestInformation(net_test_file): test_id = os.path.basename(net_test_file).replace('.py', '') information = {'id': test_id, - 'name': test_class.name, - 'description': test_class.description, - 'version': test_class.version, - 'arguments': getArguments(test_class), - 'path': net_test_file, - } + 'name': test_class.name, + 'description': test_class.description, + 'version': test_class.version, + 'arguments': getArguments(test_class), + 'path': net_test_file, + } return information + class NetTestLoader(object): method_prefix = 'test' collector = None requiresTor = False def __init__(self, options, test_file=None, test_string=None): - self.onionInputRegex = re.compile("(httpo://[a-z0-9]{16}\.onion)/input/([a-z0-9]{64})$") + self.onionInputRegex = re.compile( + "(httpo://[a-z0-9]{16}\.onion)/input/([a-z0-9]{64})$") self.options = options self.testCases = [] @@ -204,20 +218,19 @@ class NetTestLoader(object): input_file_hashes.append(input_file['hash']) test_details = {'start_time': time.time(), - 'probe_asn': config.probe_ip.geodata['asn'], - 'probe_cc': config.probe_ip.geodata['countrycode'], - 'probe_ip': config.probe_ip.geodata['ip'], - 'probe_city': config.probe_ip.geodata['city'], - 'test_name': self.testName, - 'test_version': self.testVersion, - 'software_name': 'ooniprobe', - 'software_version': software_version, - 'options': self.options, - 'input_hashes': input_file_hashes - } + 'probe_asn': config.probe_ip.geodata['asn'], + 'probe_cc': config.probe_ip.geodata['countrycode'], + 'probe_ip': config.probe_ip.geodata['ip'], + 'probe_city': config.probe_ip.geodata['city'], + 'test_name': self.testName, + 'test_version': self.testVersion, + 'software_name': 'ooniprobe', + 'software_version': software_version, + 'options': self.options, + 'input_hashes': input_file_hashes + } return test_details - def _parseNetTestOptions(self, klass): """ Helper method to assemble the options into a single UsageOptions object @@ -360,7 +373,9 @@ class NetTestLoader(object): pass return test_cases + class NetTestState(object): + def __init__(self, allTasksDone): """ This keeps track of the state of a running NetTests case. @@ -407,6 +422,7 @@ class NetTestState(object): self.completedScheduling = True self.checkAllTasksDone() + class NetTest(object): director = None @@ -480,15 +496,17 @@ class NetTest(object): if self.director: measurement.done.addCallback(self.director.measurementSucceeded, - measurement) + measurement) measurement.done.addErrback(self.director.measurementFailed, - measurement) + measurement) return measurement @defer.inlineCallbacks def initializeInputProcessor(self): for test_class, _ in self.testCases: - test_class.inputs = yield defer.maybeDeferred(test_class().getInputProcessor) + test_class.inputs = yield defer.maybeDeferred( + test_class().getInputProcessor + ) if not test_class.inputs: test_class.inputs = [None] @@ -506,7 +524,10 @@ class NetTest(object): test_instance.summary = self.summary for method in test_methods: log.debug("Running %s %s" % (test_class, method)) - measurement = self.makeMeasurement(test_instance, method, input) + measurement = self.makeMeasurement( + test_instance, + method, + input) measurements.append(measurement.done) self.state.taskCreated() yield measurement @@ -519,6 +540,7 @@ class NetTest(object): # Call the postProcessor, which must return a single report # or a deferred post.addCallback(test_instance.postProcessor) + def noPostProcessor(failure, report): failure.trap(e.NoPostProcessor) return report @@ -526,30 +548,32 @@ class NetTest(object): post.addCallback(self.report.write) if self.report and self.director: - #ghetto hax to keep NetTestState counts are accurate + # ghetto hax to keep NetTestState counts are accurate [post.addBoth(self.doneReport) for _ in measurements] self.state.allTasksScheduled() + class NetTestCase(object): + """ This is the base of the OONI nettest universe. When you write a nettest you will subclass this object. * inputs: can be set to a static set of inputs. All the tests (the methods - starting with the "test" prefix) will be run once per input. At every run - the _input_ attribute of the TestCase instance will be set to the value of - the current iteration over inputs. Any python iterable object can be set - to inputs. + starting with the "test" prefix) will be run once per input. At every + run the _input_ attribute of the TestCase instance will be set to the + value of the current iteration over inputs. Any python iterable object + can be set to inputs. - * inputFile: attribute should be set to an array containing the command line - argument that should be used as the input file. Such array looks like - this: + * inputFile: attribute should be set to an array containing the command + line argument that should be used as the input file. Such array looks + like this: ``["commandlinearg", "c", "default value" "The description"]`` - The second value of such arrray is the shorthand for the command line arg. - The user will then be able to specify inputs to the test via: + The second value of such arrray is the shorthand for the command line + arg. The user will then be able to specify inputs to the test via: ``ooniprobe mytest.py --commandlinearg path/to/file.txt`` @@ -573,12 +597,14 @@ class NetTestCase(object): * requiresRoot: set to True if the test must be run as root. - * usageOptions: a subclass of twisted.python.usage.Options for processing of command line arguments + * usageOptions: a subclass of twisted.python.usage.Options for processing + of command line arguments * localOptions: contains the parsed command line arguments. Quirks: - Every class that is prefixed with test *must* return a twisted.internet.defer.Deferred. + Every class that is prefixed with test *must* return a + twisted.internet.defer.Deferred. """ name = "This test is nameless" author = "Jane Doe <foo(a)example.com>" @@ -603,6 +629,7 @@ class NetTestCase(object): requiresTor = False localOptions = {} + def _setUp(self): """ This is the internal setup method to be overwritten by templates. @@ -734,8 +761,8 @@ class NetTestCase(object): for required_option in self.requiredOptions: log.debug("Checking if %s is present" % required_option) if required_option not in self.localOptions or \ - self.localOptions[required_option] == None: - missing_options.append(required_option) + self.localOptions[required_option] is None: + missing_options.append(required_option) if missing_options: raise e.MissingRequiredOption(missing_options) diff --git a/ooni/oonicli.py b/ooni/oonicli.py index cc23ec5..f84ac2d 100644 --- a/ooni/oonicli.py +++ b/ooni/oonicli.py @@ -1,5 +1,3 @@ -#-*- coding: utf-8 -*- - import sys import os import yaml @@ -17,6 +15,7 @@ from ooni.nettest import NetTestLoader from ooni.utils import log, checkForRoot + class Options(usage.Options): synopsis = """%s [options] [path to test].py """ % (os.path.basename(sys.argv[0]),) @@ -34,27 +33,27 @@ class Options(usage.Options): ["verbose", "v"] ] - optParameters = [["reportfile", "o", None, "report file name"], - ["testdeck", "i", None, - "Specify as input a test deck: a yaml file containing the tests to run and their arguments"], - ["collector", "c", None, - "Address of the collector of test results. This option should not be used, but you should always use a bouncer."], - ["bouncer", "b", 'httpo://nkvphnp3p6agi5qq.onion', - "Address of the bouncer for test helpers. default: httpo://nkvphnp3p6agi5qq.onion"], - ["logfile", "l", None, "log file name"], - ["pcapfile", "O", None, "pcap file name"], - ["configfile", "f", None, - "Specify a path to the ooniprobe configuration file"], - ["datadir", "d", None, - "Specify a path to the ooniprobe data directory"], - ["annotations", "a", None, - "Annotate the report with a key:value[, key:value] format."] - ] + optParameters = [ + ["reportfile", "o", None, "report file name"], + ["testdeck", "i", None, + "Specify as input a test deck: a yaml file containing the tests to run and their arguments"], + ["collector", "c", None, + "Address of the collector of test results. This option should not be used, but you should always use a bouncer."], + ["bouncer", "b", 'httpo://nkvphnp3p6agi5qq.onion', + "Address of the bouncer for test helpers. default: httpo://nkvphnp3p6agi5qq.onion"], + ["logfile", "l", None, "log file name"], + ["pcapfile", "O", None, "pcap file name"], + ["configfile", "f", None, + "Specify a path to the ooniprobe configuration file"], + ["datadir", "d", None, + "Specify a path to the ooniprobe data directory"], + ["annotations", "a", None, + "Annotate the report with a key:value[, key:value] format."]] compData = usage.Completions( extraActions=[usage.CompleteFiles( - "*.py", descr="file | module | package | TestCase | testMethod", - repeat=True)],) + "*.py", descr="file | module | package | TestCase | testMethod", + repeat=True)],) tracer = None @@ -85,6 +84,7 @@ class Options(usage.Options): except: raise usage.UsageError("No test filename specified!") + def parseOptions(): print "WARNING: running ooniprobe involves some risk that varies greatly" print " from country to country. You should be aware of this when" @@ -94,12 +94,13 @@ def parseOptions(): cmd_line_options.getUsage() try: cmd_line_options.parseOptions() - except usage.UsageError, ue: + except usage.UsageError as ue: print cmd_line_options.getUsage() - raise SystemExit, "%s: %s" % (sys.argv[0], ue) + raise SystemExit("%s: %s" % (sys.argv[0], ue)) return dict(cmd_line_options) + def runWithDirector(logging=True, start_tor=True): """ Instance the director, parse command line options and start an ooniprobe @@ -122,9 +123,9 @@ def runWithDirector(logging=True, start_tor=True): try: checkForRoot() except errors.InsufficientPrivileges: - log.err("Insufficient Privileges to capture packets." - " See ooniprobe.conf privacy.includepcap") - sys.exit(2) + log.err("Insufficient Privileges to capture packets." + " See ooniprobe.conf privacy.includepcap") + sys.exit(2) director = Director() if global_options['list']: @@ -157,10 +158,10 @@ def runWithDirector(logging=True, start_tor=True): sys.exit(1) global_options["annotations"] = annotations - #XXX: This should mean no bouncer either! + # XXX: This should mean no bouncer either! if global_options['no-collector']: log.msg("Not reporting using a collector") - collector = global_options['collector'] = None + global_options['collector'] = None global_options['bouncer'] = None deck = Deck() @@ -178,16 +179,16 @@ def runWithDirector(logging=True, start_tor=True): log.debug("No test deck detected") test_file = nettest_to_path(global_options['test_file'], True) net_test_loader = NetTestLoader(global_options['subargs'], - test_file=test_file) + test_file=test_file) deck.insert(net_test_loader) - except errors.MissingRequiredOption, option_name: + except errors.MissingRequiredOption as option_name: log.err('Missing required option: "%s"' % option_name) print net_test_loader.usageOptions().getUsage() sys.exit(2) - except errors.NetTestNotFound, path: + except errors.NetTestNotFound as path: log.err('Requested NetTest file not found (%s)' % path) sys.exit(3) - except usage.UsageError, e: + except usage.UsageError as e: log.err(e) print net_test_loader.usageOptions().getUsage() sys.exit(4) @@ -217,24 +218,29 @@ def runWithDirector(logging=True, start_tor=True): log.err("Tor does not appear to be running") log.err("Reporting with the collector %s is not possible" % global_options['collector']) - log.msg("Try with a different collector or disable collector reporting with -n") + log.msg( + "Try with a different collector or disable collector reporting with -n") elif isinstance(failure.value, errors.InvalidOONIBCollectorAddress): log.err("Invalid format for oonib collector address.") - log.msg("Should be in the format http://<collector_address>:<port>") + log.msg( + "Should be in the format http://<collector_address>:<port>") log.msg("for example: ooniprobe -c httpo://nkvphnp3p6agi5qq.onion") elif isinstance(failure.value, errors.UnableToLoadDeckInput): log.err("Unable to fetch the required inputs for the test deck.") - log.msg("Please file a ticket on our issue tracker: https://github.com/thetorproject/ooni-probe/issues") + log.msg( + "Please file a ticket on our issue tracker: https://github.com/thetorproject/ooni-probe/issues") elif isinstance(failure.value, errors.CouldNotFindTestHelper): log.err("Unable to obtain the required test helpers.") - log.msg("Try with a different bouncer or check that Tor is running properly.") + log.msg( + "Try with a different bouncer or check that Tor is running properly.") elif isinstance(failure.value, errors.CouldNotFindTestCollector): log.err("Could not find a valid collector.") - log.msg("Try with a different bouncer, specify a collector with -c or disable reporting to a collector with -n.") + log.msg( + "Try with a different bouncer, specify a collector with -c or disable reporting to a collector with -n.") elif isinstance(failure.value, errors.ProbeIPUnknown): log.err("Failed to lookup probe IP address.") @@ -267,7 +273,8 @@ def runWithDirector(logging=True, start_tor=True): if not global_options['no-collector']: if global_options['collector']: collector = global_options['collector'] - elif 'collector' in config.reports and config.reports['collector']: + elif 'collector' in config.reports \ + and config.reports['collector']: collector = config.reports['collector'] elif net_test_loader.collector: collector = net_test_loader.collector diff --git a/ooni/reporter.py b/ooni/reporter.py index 60df7eb..8341b1a 100644 --- a/ooni/reporter.py +++ b/ooni/reporter.py @@ -546,7 +546,8 @@ class Report(object): def open_oonib_reporter(self): def creation_failed(failure): self.oonib_reporter = None - return self.report_log.creation_failed(self.report_filename) + return self.report_log.creation_failed(self.report_filename, + self.collector_address) def created(report_id): return self.report_log.created(self.report_filename,

1 0

[ooni-probe/master] Also record the PID of the process that created or failed to create the report.
by art＠torproject.org 26 Jun '14

26 Jun '14

commit 10d6f145baff27a65bd4e87a9b4358374b1f17cf Author: Arturo Filastò <art(a)fuffa.org> Date: Thu Jun 19 15:40:18 2014 +0200 Also record the PID of the process that created or failed to create the report. --- ooni/reporter.py | 3 +++ 1 file changed, 3 insertions(+) diff --git a/ooni/reporter.py b/ooni/reporter.py index 9869837..60df7eb 100644 --- a/ooni/reporter.py +++ b/ooni/reporter.py @@ -447,6 +447,7 @@ class OONIBReportLog(object): def _not_created(self, report_file): with self.edit_log() as report: report[report_file] = { + 'pid': os.getpid(), 'created_at': datetime.now(), 'status': 'not-created', 'collector': None @@ -458,6 +459,7 @@ class OONIBReportLog(object): def _created(self, report_file, collector_address, report_id): with self.edit_log() as report: report[report_file] = { + 'pid': os.getpid(), 'created_at': datetime.now(), 'status': 'created', 'collector': collector_address, @@ -471,6 +473,7 @@ class OONIBReportLog(object): def _creation_failed(self, report_file, collector_address): with self.edit_log() as report: report[report_file] = { + 'pid': os.getpid(), 'created_at': datetime.now(), 'status': 'creation-failed', 'collector': collector_address

1 0

[ooni-probe/master] Better handling of OONIB report closing.
by art＠torproject.org 26 Jun '14

26 Jun '14

commit 920eaebd5175f47f36e37c0d3ab666f054fa92a4 Author: Arturo Filastò <art(a)fuffa.org> Date: Thu Jun 19 17:47:31 2014 +0200 Better handling of OONIB report closing. --- ooni/nettests/blocking/dns_consistency.py | 7 +++---- ooni/reporter.py | 7 +++++-- 2 files changed, 8 insertions(+), 6 deletions(-) diff --git a/ooni/nettests/blocking/dns_consistency.py b/ooni/nettests/blocking/dns_consistency.py index 7959e99..23d7cfc 100644 --- a/ooni/nettests/blocking/dns_consistency.py +++ b/ooni/nettests/blocking/dns_consistency.py @@ -101,8 +101,8 @@ class DNSConsistencyTest(dnst.DNSTest): the control resolver for every IP address we got back and check to see if anyone of them matches the control ones. - If they do, then we take note of the fact that censorship is probably not - happening (tampering: reverse-match). + If they do, then we take note of the fact that censorship is probably + not happening (tampering: reverse-match). If they do not match then censorship is probably going on (tampering: true). @@ -131,9 +131,8 @@ class DNSConsistencyTest(dnst.DNSTest): try: experiment_answers = yield self.performALookup(hostname, test_dns_server) - except Exception as e: + except Exception: log.err("Problem performing the DNS lookup") - log.exception(e) self.report['tampering'][test_resolver] = 'dns_lookup_error' continue diff --git a/ooni/reporter.py b/ooni/reporter.py index 8341b1a..b149b79 100644 --- a/ooni/reporter.py +++ b/ooni/reporter.py @@ -398,11 +398,10 @@ class OONIBReporter(OReporter): log.debug("Created report with id %s" % parsed_response['report_id']) defer.returnValue(parsed_response['report_id']) - @defer.inlineCallbacks def finish(self): url = self.collectorAddress + '/report/' + self.reportID + '/close' log.debug("Closing the report %s" % url) - yield self.agent.request("POST", str(url)) + return self.agent.request("POST", str(url)) class OONIBReportLog(object): @@ -650,6 +649,9 @@ class Report(object): def oonib_report_closed(result): return self.report_log.closed(self.report_filename) + def oonib_report_failed(result): + log.err("Failed to close oonib report.") + def all_reports_closed(_): if not d.called: d.callback(None) @@ -661,6 +663,7 @@ class Report(object): if self.oonib_reporter: close_oonib = self.oonib_reporter.finish() close_oonib.addCallback(oonib_report_closed) + close_oonib.addErrback(oonib_report_failed) deferreds.append(close_oonib) dl = defer.DeferredList(deferreds)

1 0