March 2013 - tor-commits - lists.torproject.org

[ooni-probe/master] Merge pull request #63 from TheTorProject/changelog
by isis＠torproject.org 09 Mar '13

09 Mar '13

commit a62e36d449d246283e87255b05bbf91501cf5fb7 Merge: a677f33 1b67e05 Author: aagbsn <aagbsn(a)extc.org> Date: Tue Mar 5 06:38:05 2013 -0800 Merge pull request #63 from TheTorProject/changelog Add changelog for ooni ChangeLog.md | 253 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 files changed, 253 insertions(+), 0 deletions(-)

1 0

[ooni-probe/master] Merge pull request #65 from samsmith/patch-3
by isis＠torproject.org 09 Mar '13

09 Mar '13

commit 5c659652d042544cade3688f827988e934c85a95 Merge: a62e36d 5f585c2 Author: aagbsn <aagbsn(a)extc.org> Date: Fri Mar 8 15:24:23 2013 -0800 Merge pull request #65 from samsmith/patch-3 Update http_header_field_manipulation.py .../manipulation/http_header_field_manipulation.py | 4 ++-- 1 files changed, 2 insertions(+), 2 deletions(-)

1 0

[ooni-probe/master] Merge pull request #60 from hellais/feature/dns_injection
by isis＠torproject.org 09 Mar '13

09 Mar '13

commit 4bd8e051fbb3b41cccc8c4b683e88e4043362a28 Merge: 5c65965 989c83b Author: aagbsn <aagbsn(a)extc.org> Date: Fri Mar 8 18:55:42 2013 -0800 Merge pull request #60 from hellais/feature/dns_injection Add DNS injection test nettests/experimental/dns_injection.py | 63 ++++++++++++++++++++++++++++++++ 1 files changed, 63 insertions(+), 0 deletions(-)

1 0

[ooni-probe/master] Outline the Measurements and NetTest classes (#7852)
by isis＠torproject.org 09 Mar '13

09 Mar '13

commit f55e52c2b851f386ca3a040590f0f615d237b59e Author: Arturo Filastò <art(a)fuffa.org> Date: Thu Jan 10 16:16:40 2013 +0100 Outline the Measurements and NetTest classes (#7852) These wrap the NetTestCase and implement the failure and retry logic. NetTest is responsible for loading the test classes and methods. Measurement is responsible for bundling sets of test classes, methods and inputs. --- ooni/measurements.py | 116 ++++++++++++++++++++++++++++++++++++++++++++++++++ 1 files changed, 116 insertions(+), 0 deletions(-) diff --git a/ooni/measurements.py b/ooni/measurements.py new file mode 100644 index 0000000..81338c0 --- /dev/null +++ b/ooni/measurements.py @@ -0,0 +1,116 @@ +from .tasks import TaskWithTimeout + +class Measurement(TaskWithTimeout): + def __init__(self, test_class, test_method, test_input, net_test): + """ + test_class: + is the class, subclass of NetTestCase, of the test to be run + + test_method: + is a string representing the test method to be called to perform + this measurement + + test_input: + is the input to the test + """ + self.test_instance = test_class() + self.test_instance.input = test_input + self.test_instance.report = {} + self.test_instance._start_time = time.time() + self.test_instance._setUp() + self.test_instance.setUp() + self.test = getattr(self.test_instance, test_method) + + def succeeded(self): + self.net_test.measurementSuccess() + + def failed(self): + pass + + def timedOut(self): + self.net_test.measurementTimeOut() + + def run(self): + d = defer.maybeDeferred(self.test) + d.addCallback(self.success) + d.addErrback(self.failure) + return d + +class NetTest(object): + manager = None + + def __init__(self, net_test_file, inputs, options): + """ + net_test_file: + is a file object containing the test to be run. + + inputs: + is a generator containing the inputs to the net test. + + options: + is a dict containing the opitions to be passed to the net test. + """ + self.test_cases = self.loadNetTestFile(net_test_file) + self.inputs = inputs + self.options = options + + self.rateLimiter = StaticRateLimiter() + + def loadNetTestFile(self, net_test_file): + """ + Creates all the necessary test_cases (a list of tuples containing the + NetTestCase (test_class, test_method)) + + example: + [(test_classA, test_method1), + (test_classA, test_method2), + (test_classA, test_method3), + (test_classA, test_method4), + (test_classA, test_method5), + + (test_classB, test_method1), + (test_classB, test_method2)] + + Note: the inputs must be valid for test_classA and test_classB. + + net_test_file: + is a file like object that will be used to generate the test_cases. + """ + # XXX Not implemented + raise NotImplemented + + def measurementTimedOut(self, measurement): + """ + This gets called when a measurement has timed out. This may or may not + trigger a retry inside of MeasurementsTracker. + """ + self.manager.measurementTimedOut(measurement) + self.rateLimiter.timedOut(measurement) + + def measurementFailed(self, measurement, failure): + """ + This gets called when a measurement has failed in the sense that all + the retries have failed at successfully running the test. + This means that it's a definitive failure and we will no longer make + any attempts at re-running the measurement. + """ + self.manager.writeFailure(measurement, failure) + + def measurementSucceeded(self, measurement): + """ + This gets called when a measurement has failed. + """ + self.manager.writeReport(measurement) + + def generateMeasurements(self): + """ + This is a generator that yields measurements and sets their timeout + value and their netTest attribute. + """ + for test_input in self.inputs: + for test_class, test_method in self.test_cases: + measurement = Measurement(test_class, test_method, test_input) + measurement.netTest = self + measurement.timeout = self.rateLimiter.timeout + yield measurement +

1 0

[ooni-probe/master] Start outlining the Manager classes (#7852)
by isis＠torproject.org 09 Mar '13

09 Mar '13

commit 559c3fa2969346234c86a510ad9c78c6d0baa21b Author: Arturo Filastò <art(a)fuffa.org> Date: Thu Jan 10 16:08:45 2013 +0100 Start outlining the Manager classes (#7852) These will keep track of the running measurements and reporting tasks. --- ooni/managers.py | 130 ++++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 files changed, 130 insertions(+), 0 deletions(-) diff --git a/ooni/managers.py b/ooni/managers.py new file mode 100644 index 0000000..2bffe0f --- /dev/null +++ b/ooni/managers.py @@ -0,0 +1,130 @@ +import itertools + +from .ratelimiting import StaticRateLimiter +from .measurements import Measurement, NetTest + +class MeasurementsManager(object): + """ + This is the Measurement Tracker. In here we keep track of active measurements + and issue new measurements once the active ones have been completed. + + MeasurementTracker does not keep track of the typology of measurements that + it is running. It just considers a measurement something that has an input + and a method to be called. + + NetTest on the contrary is aware of the typology of measurements that it is + dispatching as they are logically grouped by test file. + """ + retries = 2 + + failures = [] + concurrency = 10 + + _measurements = iter() + _active_measurements = [] + + def __init__(self, manager, netTests=None): + self.netTests = netTests if netTests else [] + self.manager = manager + + @property + def failedMeasurements(self): + return len(self.failures) + + def start(self): + """ + Start running the measurements. + """ + self.populateMeasurements() + self.runMoreMeasurements() + + def populateMeasurements(self): + """ + Take all the setup netTests and create the measurements iterator from + them. + """ + for net_test in self.netTests: + self._measurements = itertools.chain(self._measurements, + net_test.generateMeasurements()) + + def availableSlots(self): + """ + Returns the number of available slots for running tests. + """ + return self.concurrency - len(self._active_measurements) + + def schedule(self, measurement): + self._active_measurements.append(measurement) + + d = measurement.run() + d.addCallback(self.done) + d.addCallback(self.failed) + return d + + def fillSlots(self): + """ + Called on test completion and schedules measurements to be run for the + available slots. + """ + for _ in range(self.availableSlots()): + try: + measurement = self._measurements.next() + self.schedule(measurement) + except StopIteration: + break + + def done(self, result, measurement): + """ + We have successfully completed a measurement. + """ + self._active_measurements.remove(measurement) + self.completedMeasurements += 1 + + self.fillSlots() + + def failed(self, failure, measurement): + """ + The measurement has failed to complete. + """ + self._active_measurements.remove(measurement) + self.failures.append((failure, measurement)) + + if measurement.failures < self.retries: + self._measurements = itertools.chain(self._measurements, + iter(measurement)) + + self.fillSlots() + +class OManager(object): + """ + Singleton object responsible for managing the Measurements Tracker and the + Reporting Tracker. + """ + + _scheduledTests = 0 + + def __init__(self, reporters=[]): + self.reporters = reporters + + self.netTests = [] + + self.measurements_tracker = MeasurementsTracker(manager=self, + netTests=self.netTests) + self.measurements_tracker.manager = self + + def writeReport(self, measurement): + """ + Write to all the configured reporters. + """ + for reporter in self.reporters: + reporter.write(measurement) + + def writeFailure(self, measurement, failure): + pass + + def addNetTest(self, net_test): + """ + This is called to add a NetTest to the list of running network tests. + """ + self.netTests.append(net_test) +

1 0

[ooni-probe/master] Outline the rate limiting abstract class (#7852)
by isis＠torproject.org 09 Mar '13

09 Mar '13

commit 6f5da8e07d7cbc357ba4a15f7a65d3cf57925a11 Author: Arturo Filastò <art(a)fuffa.org> Date: Thu Jan 10 16:20:29 2013 +0100 Outline the rate limiting abstract class (#7852) Classes that impose constraints on timeouts and issue rate go here. --- ooni/ratelimiting.py | 68 ++++++++++++++++++++++++++++++++++++++++++++++++++ 1 files changed, 68 insertions(+), 0 deletions(-) diff --git a/ooni/ratelimiting.py b/ooni/ratelimiting.py new file mode 100644 index 0000000..986cf82 --- /dev/null +++ b/ooni/ratelimiting.py @@ -0,0 +1,68 @@ +class RateLimiter(object): + """ + The abstract class that imposes limits over how measurements are scheduled, + how retries are handled and when we should be giving up on a certain + measurement. + """ + @property + def timeout(self): + """ + After what timeout a certain test should be considered to have failed + and attempt a retry if the maximum retry has not been reached. + """ + raise NotImplemented + + @property + def maxTimeout(self): + """ + This is the maximum value that timeout can reach. + """ + raise NotImplemented + + @property + def concurrency(self): + """ + How many concurrent requests should happen at the same time. + """ + raise NotImplemented + + def timedOut(self, measurement): + raise NotImplemented + + def completed(self, measurement): + raise NotImplemented + + def failed(self, measurement): + raise NotImplemented + +class StaticRateLimiter(RateLimiter): + """ + This is a static ratelimiter that returns constant values. + """ + @property + def timeout(self): + return 10 + + @property + def maxTimeout(self): + return 5 * 60 + + @property + def concurrency(self): + return 10 + + def timedOut(self, measurement): + pass + + def completed(self, measurement): + pass + + def failed(self, measurement, failure): + pass + +class TimeoutRateLimiter(RateLimiter): + pass + +class BandwidthRateLimiter(RateLimiter): + pass +

1 0

[ooni-probe/master] Move Measurement and ReportEntry into the tasks module
by isis＠torproject.org 09 Mar '13

09 Mar '13

commit 89c8bd37973b87f9eb405f0020f9caf82e07ce8d Author: Arturo Filastò <art(a)fuffa.org> Date: Fri Jan 11 20:57:45 2013 +0100 Move Measurement and ReportEntry into the tasks module --- ooni/tasks.py | 49 +++++++++++++++++++++++++++++++++++++++++++++++++ 1 files changed, 49 insertions(+), 0 deletions(-) diff --git a/ooni/tasks.py b/ooni/tasks.py index 361dccb..909f3e5 100644 --- a/ooni/tasks.py +++ b/ooni/tasks.py @@ -72,4 +72,53 @@ class TaskWithTimeout(BaseTask): """ pass +class Measurement(TaskWithTimeout): + def __init__(self, test_class, test_method, test_input, net_test): + """ + test_class: + is the class, subclass of NetTestCase, of the test to be run + + test_method: + is a string representing the test method to be called to perform + this measurement + + test_input: + is the input to the test + + net_test: + a reference to the net_test object such measurement belongs to. + """ + self.test_instance = test_class() + self.test_instance.input = test_input + self.test_instance.report = {} + self.test_instance._start_time = time.time() + self.test_instance._setUp() + self.test_instance.setUp() + self.test = getattr(self.test_instance, test_method) + + self.netTest = net_test + + def succeeded(self): + self.net_test.succeeded() + + def failed(self): + pass + + def timedOut(self): + self.net_test.timedOut() + + def run(self): + d = defer.maybeDeferred(self.test) + d.addCallback(self.success) + d.addErrback(self.failure) + return d + +class ReportEntry(TimedOutTask): + def __init__(self, reporter, measurement): + self.reporter = reporter + self.measurement = measurement + TimedOutTask.__init__(self) + + def run(self): + return self.reporter.writeReportEntry(self.measurement)

1 0

[ooni-probe/master] Refactoring of reporting managers
by isis＠torproject.org 09 Mar '13

09 Mar '13

commit 055993061eb1a69a9e9876091d9b07f894deda32 Author: Arturo Filastò <art(a)fuffa.org> Date: Fri Jan 11 20:56:45 2013 +0100 Refactoring of reporting managers Make the dependencies between modules much clearer Sketch out a graph that illustrates such depedencies --- ooni/managers.py | 281 ++++++++++++++++++++++++++++++++++++++++-------------- 1 files changed, 209 insertions(+), 72 deletions(-) diff --git a/ooni/managers.py b/ooni/managers.py index 3383840..ec47ca7 100644 --- a/ooni/managers.py +++ b/ooni/managers.py @@ -3,7 +3,108 @@ import itertools from .ratelimiting import StaticRateLimiter from .measurements import Measurement, NetTest -class MeasurementsManager(object): +class TaskManager(object): + retries = 2 + + failures = [] + concurrency = 10 + + _tasks = iter() + _active_tasks = [] + + def _failed(self, failure, task): + """ + The has failed to complete, we append it to the end of the task chain + to be re-run once all the currently scheduled tasks have run. + """ + self._active_tasks.remove(task) + self.failures.append((failure, task)) + + if task.failures < self.retries: + self._tasks = itertools.chain(self._tasks, + iter(task)) + + self.fillSlots() + + self.failed(failure, task) + + def _fillSlots(self): + """ + Called on test completion and schedules measurements to be run for the + available slots. + """ + for _ in range(self.availableSlots()): + try: + task = self._tasks.next() + self._run(task) + except StopIteration: + break + + def _suceeded(self, result, task): + """ + We have successfully completed a measurement. + """ + self._active_tasks.remove(task) + self.completedTasks += 1 + + self.fillSlots() + + self.suceeded(result, task) + + def _run(self, task): + """ + This gets called to add a task to the list of currently active and + running tasks. + """ + self._active_tasks.append(task) + + d = task.run() + d.addCallback(self.succeeded) + d.addCallback(self.failed) + + @property + def failedMeasurements(self): + return len(self.failures) + + def availableSlots(self): + """ + Returns the number of available slots for running tests. + """ + return self.concurrency - len(self._active_tasks) + + def schedule(self, task_or_task_iterator): + """ + Takes as argument a single task or a task iterable and appends it to the task + generator queue. + """ + self._tasks = itertools.chain(self._tasks, task_or_task_iterator) + + def start(self): + self.initializeTaskList() + self._fillSlots() + + def initializeTaskList(self): + """ + This should contain all the logic that gets run at first start to + pre-populate the list of tasks to be run and the tasks currently + running. + """ + raise NotImplemented + + def failed(self, failure, task): + """ + This method should be overriden by the subclass and should contains + logic for dealing with a failure that is subclass specific. + + The default failure handling logic is to reschedule the task up until + we reach the maximum number of retries. + """ + raise NotImplemented + + def succeeded(self, result, task): + raise NotImplemented + +class MeasurementsManager(TaskManager): """ This is the Measurement Tracker. In here we keep track of active measurements and issue new measurements once the active ones have been completed. @@ -20,113 +121,149 @@ class MeasurementsManager(object): failures = [] concurrency = 10 - _measurements = iter() - _active_measurements = [] + director = None - def __init__(self, manager, netTests=None): + def __init__(self, netTests=None): self.netTests = netTests if netTests else [] - self.manager = manager - @property - def failedMeasurements(self): - return len(self.failures) + def initializeTaskList(self): + for net_test in self.netTests: + self.schedule(net_test.generateMeasurements()) - def start(self): + def suceeded(self, result, measurement): + pass + + def failed(self, failure, measurement): + pass + +class Report(object): + def __init__(self, reporters, net_test): """ - Start running the measurements. + This will instantiate all the reporters and add them to the list of + available reporters. + + net_test: + is a reference to the net_test to which the report object belongs to. """ - self.populateMeasurements() - self.runMoreMeasurements() + self.netTest = net_test + self.reporters = [] + for r in reporters: + reporter = r() + self.reporters.append(reporter) + + self.createReports() - def populateMeasurements(self): + def createReports(self): """ - Take all the setup netTests and create the measurements iterator from - them. + This will create all the reports that need to be created. """ - for net_test in self.netTests: - self._measurements = itertools.chain(self._measurements, - net_test.generateMeasurements()) + for reporter in self.reporters: + reporter.createReport() - def availableSlots(self): + def write(self, measurement): """ - Returns the number of available slots for running tests. + This will write to all the reporters, by waiting on the created + callback to fire. """ - return self.concurrency - len(self._active_measurements) + for reporter in self.reporters: + @reporter.created.addCallback + def cb(result): + report_write_task = ReportWrite(reporter, measurement) - def schedule(self, measurement): - self._active_measurements.append(measurement) +class ReportEntryManager(object): - d = measurement.run() - d.addCallback(self.done) - d.addCallback(self.failed) - return d + director = None - def fillSlots(self): - """ - Called on test completion and schedules measurements to be run for the - available slots. - """ - for _ in range(self.availableSlots()): - try: - measurement = self._measurements.next() - self.schedule(measurement) - except StopIteration: - break + def __init__(self, manager, netTests=None): + self.netTests = netTests if netTests else [] + self.manager = manager - def done(self, result, measurement): - """ - We have successfully completed a measurement. - """ - self._active_measurements.remove(measurement) - self.completedMeasurements += 1 + def addNetTest(self, netTest): + self.netTests.append(netTest) - self.fillSlots() + def initializeTaskList(self): + pass + + def suceeded(self, result, measurement): + pass def failed(self, failure, measurement): - """ - The measurement has failed to complete. - """ - self._active_measurements.remove(measurement) - self.failures.append((failure, measurement)) + pass - if measurement.failures < self.retries: - self._measurements = itertools.chain(self._measurements, - iter(measurement)) +class Director(object): + """ + Singleton object responsible for coordinating the Measurements Manager and the + Reporting Manager. - self.fillSlots() + How this all looks like is as follows: -class OManager(object): - """ - Singleton object responsible for managing the Measurements Tracker and the - Reporting Tracker. - """ + +------------------------------------------------+ + | Director |<--+ + +------------------------------------------------+ | + ^ ^ | + | Measurement | | + +---------+ [---------] +--------------------+ | + | | | MeasurementManager | | + | NetTest | [---------] +--------------------+ | + | | | [----------------] | | + +---------+ [---------] | [----------------] | | + | | [----------------] | | + | +--------------------+ | + v | + +---------+ ReportEntry | + | | [---------] +--------------------+ | + | Report | | ReportEntryManager | | + | | [---------] +--------------------+ | + +---------+ | [----------------] | | + [---------] | [----------------] |-- + | [----------------] | + +--------------------+ + [------------] are Tasks + + +------+ + | | are TaskManagers + +------+ + | | + +------+ + + +------+ + | | are general purpose objects + +------+ + + """ _scheduledTests = 0 - def __init__(self, reporters=[]): + def __init__(self): self.reporters = reporters self.netTests = [] self.measurementsManager = MeasurementsManager(manager=self, netTests=self.netTests) - self.measurementsManager.manager = self + self.measurementsManager.director = self - self.reportingManager + self.reportingManager = ReportingEntryManager() + self.reportingManager.director = self - def writeReport(self, measurement): + def startTest(self, net_test_file, inputs, options): """ - Write to all the configured reporters. + Create the Report for the NetTest and start the report NetTest. """ - for reporter in self.reporters: - reporter.write(measurement) - - def writeFailure(self, measurement, failure): - pass + report = Report() + net_test = NetTest(net_test_file, inputs, options, report) + net_test.director = self - def addNetTest(self, net_test): + def measurementTimedOut(self, measurement): """ - This is called to add a NetTest to the list of running network tests. + This gets called every time a measurement times out independenty from + the fact that it gets re-scheduled or not. """ - self.netTests.append(net_test) + pass + + def measurementFailed(self, measurement, failure): + pass + + def writeFailure(self, measurement, failure): + pass

1 0

[ooni-probe/master] Renaming of measurements methods
by isis＠torproject.org 09 Mar '13

09 Mar '13

commit e211916588c55f162c1d14fde2c105decddc2aa1 Author: Arturo Filastò <art(a)fuffa.org> Date: Fri Jan 11 20:58:26 2013 +0100 Renaming of measurements methods --- ooni/measurements.py | 55 +++++++------------------------------------------ 1 files changed, 8 insertions(+), 47 deletions(-) diff --git a/ooni/measurements.py b/ooni/measurements.py index 81338c0..0ef8ef6 100644 --- a/ooni/measurements.py +++ b/ooni/measurements.py @@ -1,45 +1,9 @@ from .tasks import TaskWithTimeout -class Measurement(TaskWithTimeout): - def __init__(self, test_class, test_method, test_input, net_test): - """ - test_class: - is the class, subclass of NetTestCase, of the test to be run - - test_method: - is a string representing the test method to be called to perform - this measurement - - test_input: - is the input to the test - """ - self.test_instance = test_class() - self.test_instance.input = test_input - self.test_instance.report = {} - self.test_instance._start_time = time.time() - self.test_instance._setUp() - self.test_instance.setUp() - self.test = getattr(self.test_instance, test_method) - - def succeeded(self): - self.net_test.measurementSuccess() - - def failed(self): - pass - - def timedOut(self): - self.net_test.measurementTimeOut() - - def run(self): - d = defer.maybeDeferred(self.test) - d.addCallback(self.success) - d.addErrback(self.failure) - return d - class NetTest(object): - manager = None + director = None - def __init__(self, net_test_file, inputs, options): + def __init__(self, net_test_file, inputs, options, report): """ net_test_file: is a file object containing the test to be run. @@ -54,8 +18,6 @@ class NetTest(object): self.inputs = inputs self.options = options - self.rateLimiter = StaticRateLimiter() - def loadNetTestFile(self, net_test_file): """ Creates all the necessary test_cases (a list of tuples containing the @@ -79,28 +41,27 @@ class NetTest(object): # XXX Not implemented raise NotImplemented - def measurementTimedOut(self, measurement): + def timedOut(self, measurement): """ This gets called when a measurement has timed out. This may or may not trigger a retry inside of MeasurementsTracker. """ - self.manager.measurementTimedOut(measurement) - self.rateLimiter.timedOut(measurement) + self.director.measurementTimedOut(measurement) - def measurementFailed(self, measurement, failure): + def failed(self, measurement, failure): """ This gets called when a measurement has failed in the sense that all the retries have failed at successfully running the test. This means that it's a definitive failure and we will no longer make any attempts at re-running the measurement. """ - self.manager.writeFailure(measurement, failure) + self.director.measurementFailed(measurement, failure) - def measurementSucceeded(self, measurement): + def succeeded(self, measurement): """ This gets called when a measurement has failed. """ - self.manager.writeReport(measurement) + self.report.write(measurement) def generateMeasurements(self): """

1 0

[ooni-probe/master] Function that will transform an object into a listiterable
by isis＠torproject.org 09 Mar '13

09 Mar '13

commit c5513ba318ed75a78f010ad32702d55d4e0516e6 Author: Arturo Filastò <art(a)fuffa.org> Date: Sat Jan 12 14:53:13 2013 +0100 Function that will transform an object into a listiterable Will return the iterable if it's an iterable --- ooni/managers.py | 15 ++++++++++++--- 1 files changed, 12 insertions(+), 3 deletions(-) diff --git a/ooni/managers.py b/ooni/managers.py index ec47ca7..c4b4d2a 100644 --- a/ooni/managers.py +++ b/ooni/managers.py @@ -1,7 +1,16 @@ import itertools +from twisted.internet import defer -from .ratelimiting import StaticRateLimiter -from .measurements import Measurement, NetTest +def makeIterable(item): + """ + Takes as argument or an iterable and if it's not an iterable object then it + will return a listiterator. + """ + try: + iterable = iter(item) + except TypeError: + iterable = iter([item]) + return iterable class TaskManager(object): retries = 2

1 0