December 2012 - tor-commits - lists.torproject.org

[ooni-probe/master] Merge branch 'master' of github.com:hellais/ooni-probe
by art＠torproject.org 07 Dec '12

07 Dec '12

commit b829278e5f94b7b0faa261f466318e3c5efb39fa Merge: 9dcd60b 3b830b6 Author: Arturo Filastò <art(a)fuffa.org> Date: Thu Dec 6 22:39:44 2012 +0100 Merge branch 'master' of github.com:hellais/ooni-probe * 'master' of github.com:hellais/ooni-probe: Make nettests/* python modules Add nettest apidoc Set release version to 0.1-alpha Add references to classes in writing_tests Add documentation for the DNSSpoof test Add documentation for HTTPHeaderFieldManipulation Clean up writing_tests.rst Remove redundant heading Add automodule for tcpt to ooni.templates.rst Use section headers Add documentation and examples templates httpt and dnst Clean up index.rst Add docstrings to ooni/runner.py Remove automodule for ooni/__init__.py Add docstrings to network helper functions Add dnst to ooni.templates.rst Fix spelling in test documentation call launch_tor with the tor_binary from config Use HTTPS where available README.md | 14 +- docs/source/api/modules.rst | 7 - docs/source/api/ooni.rst | 8 - docs/source/api/ooni.templates.rst | 22 +- docs/source/api/ooni.utils.rst | 2 - docs/source/conf.py | 4 +- docs/source/index.rst | 5 +- docs/source/nettests/modules.rst | 8 + docs/source/nettests/nettests.blocking.rst | 26 ++ docs/source/nettests/nettests.examples.rst | 58 ++++ docs/source/nettests/nettests.experimental.rst | 58 ++++ docs/source/nettests/nettests.manipulation.rst | 58 ++++ docs/source/nettests/nettests.scanning.rst | 10 + docs/source/nettests/nettests.third_party.rst | 10 + docs/source/tests/dnsspoof.rst | 111 +++++++ docs/source/tests/dnstamper.rst | 6 +- .../tests/http_header_field_manipulation.rst | 346 ++++++++++++++++++++ docs/source/tests/http_host.rst | 10 +- docs/source/tests/http_invalid_request_line.rst | 4 +- docs/source/tests/http_requests.rst | 4 +- docs/source/tests/tcpconnect.rst | 2 +- docs/source/tests/traceroute.rst | 2 +- docs/source/writing_tests.rst | 271 +++++++++++++--- nettests/blocking/__init__.py | 1 + nettests/examples/example_http_checksum.py | 27 ++ ooni/runner.py | 8 + ooni/utils/txscapy.py | 5 + 27 files changed, 993 insertions(+), 94 deletions(-)

1 0

[ooni-probe/master] Make nettests/* python modules
by art＠torproject.org 07 Dec '12

07 Dec '12

commit f2504854b778899fdabdcf63ae555a86fc7882da Author: aagbsn <aagbsn(a)extc.org> Date: Thu Dec 6 19:29:41 2012 +0000 Make nettests/* python modules --- nettests/blocking/__init__.py | 1 + 1 files changed, 1 insertions(+), 0 deletions(-) diff --git a/nettests/__init__.py b/nettests/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/nettests/blocking/__init__.py b/nettests/blocking/__init__.py new file mode 100644 index 0000000..8b13789 --- /dev/null +++ b/nettests/blocking/__init__.py @@ -0,0 +1 @@ + diff --git a/nettests/examples/__init__.py b/nettests/examples/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/nettests/experimental/__init__.py b/nettests/experimental/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/nettests/manipulation/__init__.py b/nettests/manipulation/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/nettests/scanning/__init__.py b/nettests/scanning/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/nettests/third_party/__init__.py b/nettests/third_party/__init__.py new file mode 100644 index 0000000..e69de29

1 0

[ooni-probe/master] Merge pull request #18 from aagbsn/update_documentation
by art＠torproject.org 07 Dec '12

07 Dec '12

commit 3b830b654312ff6e47863fcde119f760514ccfb0 Merge: 0b40aa8 f250485 Author: Arturo Filastò <hellais(a)gmail.com> Date: Thu Dec 6 12:16:17 2012 -0800 Merge pull request #18 from aagbsn/update_documentation Update documentation docs/source/api/modules.rst | 7 - docs/source/api/ooni.rst | 8 - docs/source/api/ooni.templates.rst | 22 +- docs/source/api/ooni.utils.rst | 2 - docs/source/conf.py | 4 +- docs/source/index.rst | 5 +- docs/source/nettests/modules.rst | 8 + docs/source/nettests/nettests.blocking.rst | 26 ++ docs/source/nettests/nettests.examples.rst | 58 ++++ docs/source/nettests/nettests.experimental.rst | 58 ++++ docs/source/nettests/nettests.manipulation.rst | 58 ++++ docs/source/nettests/nettests.scanning.rst | 10 + docs/source/nettests/nettests.third_party.rst | 10 + docs/source/tests/dnsspoof.rst | 111 +++++++ docs/source/tests/dnstamper.rst | 6 +- .../tests/http_header_field_manipulation.rst | 346 ++++++++++++++++++++ docs/source/tests/http_host.rst | 10 +- docs/source/tests/http_invalid_request_line.rst | 4 +- docs/source/tests/http_requests.rst | 4 +- docs/source/tests/tcpconnect.rst | 2 +- docs/source/tests/traceroute.rst | 2 +- docs/source/writing_tests.rst | 271 +++++++++++++--- nettests/blocking/__init__.py | 1 + nettests/examples/example_http_checksum.py | 27 ++ ooni/runner.py | 7 + ooni/utils/txscapy.py | 5 + 26 files changed, 985 insertions(+), 87 deletions(-)

1 0

[ooni-probe/master] Add nettest apidoc
by art＠torproject.org 07 Dec '12

07 Dec '12

commit c0be39e758b52f797c88bd67782d6cae57838ab2 Author: aagbsn <aagbsn(a)extc.org> Date: Thu Dec 6 19:29:12 2012 +0000 Add nettest apidoc --- docs/source/api/modules.rst | 7 --- docs/source/index.rst | 2 + docs/source/nettests/modules.rst | 8 +++ docs/source/nettests/nettests.blocking.rst | 26 +++++++++++ docs/source/nettests/nettests.examples.rst | 58 ++++++++++++++++++++++++ docs/source/nettests/nettests.experimental.rst | 58 ++++++++++++++++++++++++ docs/source/nettests/nettests.manipulation.rst | 58 ++++++++++++++++++++++++ docs/source/nettests/nettests.scanning.rst | 10 ++++ docs/source/nettests/nettests.third_party.rst | 10 ++++ 9 files changed, 230 insertions(+), 7 deletions(-) diff --git a/docs/source/api/modules.rst b/docs/source/api/modules.rst deleted file mode 100644 index a9f85c8..0000000 --- a/docs/source/api/modules.rst +++ /dev/null @@ -1,7 +0,0 @@ -ooni -==== - -.. toctree:: - :maxdepth: 4 - - ooni diff --git a/docs/source/index.rst b/docs/source/index.rst index 1abb517..711440f 100644 --- a/docs/source/index.rst +++ b/docs/source/index.rst @@ -118,8 +118,10 @@ More developer documentation .. toctree:: :maxdepth: 2 + :glob: oonib writing_tests api/* + nettests/modules glossary diff --git a/docs/source/nettests/modules.rst b/docs/source/nettests/modules.rst new file mode 100644 index 0000000..408bbad --- /dev/null +++ b/docs/source/nettests/modules.rst @@ -0,0 +1,8 @@ +nettests +======== + +.. toctree:: + :glob: + :maxdepth: 2 + + * diff --git a/docs/source/nettests/nettests.blocking.rst b/docs/source/nettests/nettests.blocking.rst new file mode 100644 index 0000000..03a08b6 --- /dev/null +++ b/docs/source/nettests/nettests.blocking.rst @@ -0,0 +1,26 @@ +nettests blocking Package +========================= + +:mod:`nettests.blocking.dnstamper` Module +----------------------------------------- + +.. automodule:: nettests.blocking.dnstamper + :members: + :undoc-members: + :show-inheritance: + +:mod:`nettests.blocking.http_requests` Module +--------------------------------------------- + +.. automodule:: nettests.blocking.http_requests + :members: + :undoc-members: + :show-inheritance: + +:mod:`nettests.blocking.tcpconnect` Module +------------------------------------------ + +.. automodule:: nettests.blocking.tcpconnect + :members: + :undoc-members: + :show-inheritance: diff --git a/docs/source/nettests/nettests.examples.rst b/docs/source/nettests/nettests.examples.rst new file mode 100644 index 0000000..6027455 --- /dev/null +++ b/docs/source/nettests/nettests.examples.rst @@ -0,0 +1,58 @@ +nettests examples Package +========================= + +:mod:`nettests.examples.example_dnst` Module +-------------------------------------------- + +.. automodule:: nettests.examples.example_dnst + :members: + :undoc-members: + :show-inheritance: + +:mod:`nettests.examples.example_http_checksum` Module +----------------------------------------------------- + +.. automodule:: nettests.examples.example_http_checksum + :members: + :undoc-members: + :show-inheritance: + +:mod:`nettests.examples.example_httpt` Module +--------------------------------------------- + +.. automodule:: nettests.examples.example_httpt + :members: + :undoc-members: + :show-inheritance: + +:mod:`nettests.examples.example_myip` Module +-------------------------------------------- + +.. automodule:: nettests.examples.example_myip + :members: + :undoc-members: + :show-inheritance: + +:mod:`nettests.examples.example_scapyt` Module +---------------------------------------------- + +.. automodule:: nettests.examples.example_scapyt + :members: + :undoc-members: + :show-inheritance: + +:mod:`nettests.examples.example_scapyt_yield` Module +---------------------------------------------------- + +.. automodule:: nettests.examples.example_scapyt_yield + :members: + :undoc-members: + :show-inheritance: + +:mod:`nettests.examples.example_tcpt` Module +-------------------------------------------- + +.. automodule:: nettests.examples.example_tcpt + :members: + :undoc-members: + :show-inheritance: diff --git a/docs/source/nettests/nettests.experimental.rst b/docs/source/nettests/nettests.experimental.rst new file mode 100644 index 0000000..110cf14 --- /dev/null +++ b/docs/source/nettests/nettests.experimental.rst @@ -0,0 +1,58 @@ +nettests experimental Package +============================= + +:mod:`nettests.experimental.chinatrigger` Module +------------------------------------------------ + +.. automodule:: nettests.experimental.chinatrigger + :members: + :undoc-members: + :show-inheritance: + +:mod:`nettests.experimental.domclass_collector` Module +------------------------------------------------------ + +.. automodule:: nettests.experimental.domclass_collector + :members: + :undoc-members: + :show-inheritance: + +:mod:`nettests.experimental.http_keyword_filtering` Module +---------------------------------------------------------- + +.. automodule:: nettests.experimental.http_keyword_filtering + :members: + :undoc-members: + :show-inheritance: + +:mod:`nettests.experimental.http_uk_mobile_networks` Module +----------------------------------------------------------- + +.. automodule:: nettests.experimental.http_uk_mobile_networks + :members: + :undoc-members: + :show-inheritance: + +:mod:`nettests.experimental.keyword_filtering` Module +----------------------------------------------------- + +.. automodule:: nettests.experimental.keyword_filtering + :members: + :undoc-members: + :show-inheritance: + +:mod:`nettests.experimental.parasitictraceroute` Module +------------------------------------------------------- + +.. automodule:: nettests.experimental.parasitictraceroute + :members: + :undoc-members: + :show-inheritance: + +:mod:`nettests.experimental.squid` Module +----------------------------------------- + +.. automodule:: nettests.experimental.squid + :members: + :undoc-members: + :show-inheritance: diff --git a/docs/source/nettests/nettests.manipulation.rst b/docs/source/nettests/nettests.manipulation.rst new file mode 100644 index 0000000..c05f7a7 --- /dev/null +++ b/docs/source/nettests/nettests.manipulation.rst @@ -0,0 +1,58 @@ +nettests manipulation Package +============================= + +:mod:`nettests.manipulation.captiveportal` Module +-------------------------------------------------- + +.. automodule:: nettests.manipulation.captiveportal + :members: + :undoc-members: + :show-inheritance: + +:mod:`nettests.manipulation.daphne` Module +------------------------------------------ + +.. automodule:: nettests.manipulation.daphne + :members: + :undoc-members: + :show-inheritance: + +:mod:`nettests.manipulation.dnsspoof` Module +-------------------------------------------- + +.. automodule:: nettests.manipulation.dnsspoof + :members: + :undoc-members: + :show-inheritance: + +:mod:`nettests.manipulation.http_header_field_manipulation` Module +------------------------------------------------------------------ + +.. automodule:: nettests.manipulation.http_header_field_manipulation + :members: + :undoc-members: + :show-inheritance: + +:mod:`nettests.manipulation.http_host` Module +--------------------------------------------- + +.. automodule:: nettests.manipulation.http_host + :members: + :undoc-members: + :show-inheritance: + +:mod:`nettests.manipulation.http_invalid_request_line` Module +------------------------------------------------------------- + +.. automodule:: nettests.manipulation.http_invalid_request_line + :members: + :undoc-members: + :show-inheritance: + +:mod:`nettests.manipulation.traceroute` Module +---------------------------------------------- + +.. automodule:: nettests.manipulation.traceroute + :members: + :undoc-members: + :show-inheritance: diff --git a/docs/source/nettests/nettests.scanning.rst b/docs/source/nettests/nettests.scanning.rst new file mode 100644 index 0000000..8bad39b --- /dev/null +++ b/docs/source/nettests/nettests.scanning.rst @@ -0,0 +1,10 @@ +nettests scanning Package +========================= + +:mod:`nettests.scanning.http_url_list` Module +--------------------------------------------- + +.. automodule:: nettests.scanning.http_url_list + :members: + :undoc-members: + :show-inheritance: diff --git a/docs/source/nettests/nettests.third_party.rst b/docs/source/nettests/nettests.third_party.rst new file mode 100644 index 0000000..6d0e78a --- /dev/null +++ b/docs/source/nettests/nettests.third_party.rst @@ -0,0 +1,10 @@ +nettests third_party Package +============================ + +:mod:`nettests.third_party.netalyzr` Module +------------------------------------------- + +.. automodule:: nettests.third_party.netalyzr + :members: + :undoc-members: + :show-inheritance:

1 0

[ooni-probe/master] Set release version to 0.1-alpha
by art＠torproject.org 07 Dec '12

07 Dec '12

commit 4601552d7805786b4401c9bda85d06c091bf0e7c Author: aagbsn <aagbsn(a)extc.org> Date: Thu Dec 6 18:10:26 2012 +0000 Set release version to 0.1-alpha --- docs/source/conf.py | 4 ++-- 1 files changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/source/conf.py b/docs/source/conf.py index 570e4e6..1362e06 100644 --- a/docs/source/conf.py +++ b/docs/source/conf.py @@ -51,9 +51,9 @@ copyright = u'2012, The Tor Project' # built documents. # # The short X.Y version. -version = 'v0.0.7.1' +version = '0.1' # The full version, including alpha/beta/rc tags. -release = 'v0.0.7.1-alpha' +release = '0.1-alpha' # The language for content autogenerated by Sphinx. Refer to documentation # for a list of supported languages.

1 0

[ooni-probe/master] Use twisted fdesc when writing to files
by art＠torproject.org 07 Dec '12

07 Dec '12

commit 9dcd60b51dc692c6e3d042a0ea9b2c580aa803d8 Author: Arturo Filastò <art(a)fuffa.org> Date: Thu Dec 6 22:37:15 2012 +0100 Use twisted fdesc when writing to files * Add support for processing the ASN number of the probe Test reports shall follow the specification detailed inside of docs/reports.rst --- oonib/report/__init__.py | 3 -- oonib/report/file_collector.py | 61 +++++++++++++++++++++++++-------------- 2 files changed, 39 insertions(+), 25 deletions(-) diff --git a/oonib/report/__init__.py b/oonib/report/__init__.py index 96667b4..fcbf220 100644 --- a/oonib/report/__init__.py +++ b/oonib/report/__init__.py @@ -1,6 +1,3 @@ -def generateReportID(): - return otime.timestamp() + '_' + randomStr(20) - class MissingField(Exception): pass diff --git a/oonib/report/file_collector.py b/oonib/report/file_collector.py index 9896784..584a11d 100644 --- a/oonib/report/file_collector.py +++ b/oonib/report/file_collector.py @@ -4,8 +4,10 @@ import json import re import os -from oonib.report import generateReportID +from twisted.internet import fdesc + from oonib.report import MissingField, InvalidRequestField +from ooni.utils import randomStr from cyclone import web @@ -15,11 +17,14 @@ def parseNewReportRequest(request): """ version_string = re.compile("[0-9A-Za-z_\-\.]+$") name = re.compile("[a-zA-Z0-9_\- ]+$") + probe_asn = re.compile("AS[0-9]+$") - expected_request = {'software_name': name, + expected_request = { + 'software_name': name, 'software_version': version_string, 'test_name': name, - 'test_version': version_string + 'test_version': version_string, + 'probe_asn': probe_asn } parsed_request = json.loads(request) @@ -28,17 +33,20 @@ def parseNewReportRequest(request): value_to_check = parsed_request[k] except KeyError: raise MissingField(k) + print "Matching %s with %s | %s" % (regexp, value_to_check, k) if re.match(regexp, str(value_to_check)): continue else: raise InvalidRequestField(k) + return parsed_request class NewReportHandlerFile(web.RequestHandler): """ Responsible for creating and updating reports by writing to flat file. """ + def post(self): """ Creates a new report with the input @@ -49,7 +57,7 @@ class NewReportHandlerFile(web.RequestHandler): 'software_version': 'XXX', 'test_name': 'XXX', 'test_version': 'XXX', - 'progress': 'XXX', + 'probe_asn': 'XXX' 'content': 'XXX' } @@ -91,49 +99,58 @@ class NewReportHandlerFile(web.RequestHandler): software_version = report_data['software_version'] test_name = report_data['test_name'] test_version = report_data['test_version'] + probe_asn = report_data['test_version'] content = report_data['content'] - report_id = generateReportID() + if not probe_asn: + probe_asn = "AS0" - #report_filename = '_'.join((report_id, - # report_data['software_name'], - # report_data['software_version'], - # report_data['test_name'], - # report_data['test_version'])) + report_id = otime.timestamp() + '_' \ + + probe_asn + '_' \ + + randomStr(50) # The report filename contains the timestamp of the report plus a # random nonce report_filename = os.path.join(config.main.report_dir, report_id) - report_filename += '.yamloo' response = {'backend_version': backend_version, 'report_id': report_id - } + } + + self.writeToReport(report_filename, + report_data['content']) - fp = open(report_filename, 'w+') - fp.write(report_data['content']) - fp.close() self.write(response) + def writeToReport(report_filename, data): + with open(report_filename, 'w+') as fd: + fdesc.setNonBlocking(fd) + fdesc.writeToFD(data) + def put(self): """ Update an already existing report. - {'report_id': 'XXX', + { + 'report_id': 'XXX', 'content': 'XXX' } """ parsed_request = parseUpdateReportRequest(self.request.body) + report_id = parsed_request['report_id'] + print "Got this request %s" % parsed_request + report_filename = os.path.join(config.main.report_dir, + report_id) - report_filename = os.path.join(config.main.report_dir, report_id) - report_filename += '.yamloo' + self.updateReport(report_filename, parsed_request['content']) + + def updateReport(report_filename, data): try: - with open(report_filename, 'a+') as f: - # XXX-Twisted change this to use t.i.a.fdesc and perhaps make a - # nice little object for it. - f.write(parsed_request['content']) + with open(report_filename, 'a+') as fd: + fdesc.setNonBlocking(fd) + fdesc.writeToFD(data) except IOError as e: web.HTTPError(404, "Report not found")

1 0

[ooni-probe/master] Add support for sending the ASN number of the probe
by art＠torproject.org 07 Dec '12

07 Dec '12

commit 7edc89c9e8391975e42005ee8968b2f7abac70b1 Author: Arturo Filastò <art(a)fuffa.org> Date: Thu Dec 6 22:34:30 2012 +0100 Add support for sending the ASN number of the probe * The ASN number will get sent when creating a new report --- ooni/reporter.py | 49 ++++++++++++++++++++++++------------------------- 1 files changed, 24 insertions(+), 25 deletions(-) diff --git a/ooni/reporter.py b/ooni/reporter.py index 29e6049..e33d395 100644 --- a/ooni/reporter.py +++ b/ooni/reporter.py @@ -10,11 +10,12 @@ import traceback import itertools import logging -import sys -import os import time import yaml import json +import sys +import os +import re from yaml.representer import * from yaml.emitter import * @@ -116,7 +117,6 @@ def safe_dump(data, stream=None, **kw): """ return yaml.dump_all([data], stream, Dumper=OSafeDumper, **kw) -(a)defer.inlineCallbacks def getTestDetails(options): from ooni import __version__ as software_version @@ -126,28 +126,27 @@ def getTestDetails(options): config.privacy.includecountry or \ config.privacy.includecity: log.msg("Running geo IP lookup via check.torproject.org") - client_ip = yield geodata.myIP() - client_location = geodata.IPToLocation(client_ip) - else: - client_ip = "127.0.0.1" + client_geodata = geodata.IPToLocation(config.probe_ip) if config.privacy.includeip: - client_geodata['ip'] = client_ip + client_geodata['ip'] = config.probe_ip else: client_geodata['ip'] = "127.0.0.1" - client_geodata['asn'] = None - client_geodata['city'] = None - client_geodata['countrycode'] = None - - if config.privacy.includeasn: - client_geodata['asn'] = client_location['asn'] + # Here we unset all the client geodata if the option to not include then + # has been specified + if client_geodata and not config.privacy.includeasn: + client_geodata['asn'] = None + else: + # XXX this regexp should probably go inside of geodata + client_geodata['asn'] = \ + re.search('AS\d+', client_geodata['asn']).group(0) - if config.privacy.includecity: - client_geodata['city'] = client_location['city'] + if client_geodata and not config.privacy.includecity: + client_geodata['city'] = None - if config.privacy.includecountry: - client_geodata['countrycode'] = client_location['countrycode'] + if client_geodata and not config.privacy.includecountry: + client_geodata['countrycode'] = None test_details = {'start_time': otime.utcTimeNow(), 'probe_asn': client_geodata['asn'], @@ -158,7 +157,7 @@ def getTestDetails(options): 'software_name': 'ooniprobe', 'software_version': software_version } - defer.returnValue(test_details) + return test_details class OReporter(object): def __init__(self, cmd_line_options): @@ -230,14 +229,13 @@ class YAMLReporter(OReporter): self._write(safe_dump(entry)) self._write('...\n') - @defer.inlineCallbacks def createReport(self, options): self._writeln("###########################################") self._writeln("# OONI Probe Report for %s test" % options['name']) self._writeln("# %s" % otime.prettyDateNow()) self._writeln("###########################################") - test_details = yield getTestDetails(options) + test_details = getTestDetails(options) test_details['options'] = self.cmd_line_options self.writeReportEntry(test_details) @@ -305,13 +303,10 @@ class OONIBReporter(OReporter): """ Creates a report on the oonib collector. """ - test_name = options['name'] - test_version = options['version'] - url = self.backend_url + '/report/new' try: - test_details = yield getTestDetails(options) + test_details = getTestDetails(options) except Exception, e: log.exception(e) @@ -323,8 +318,12 @@ class OONIBReporter(OReporter): content += safe_dump(test_details) content += '...\n' + test_name = options['name'] + test_version = options['version'] + request = {'software_name': test_details['software_name'], 'software_version': test_details['software_version'], + 'probe_asn': test_details['probe_asn'], 'test_name': test_name, 'test_version': test_version, 'content': content

1 0

[ooni-probe/master] Add support for obtaining the probes IP address via getinfo address
by art＠torproject.org 07 Dec '12

07 Dec '12

commit a8a85f89fa458c74d89eda449dcefc378b6c708b Author: Arturo Filastò <art(a)fuffa.org> Date: Thu Dec 6 22:33:37 2012 +0100 Add support for obtaining the probes IP address via getinfo address * As detailed in https://trac.torproject.org/projects/tor/ticket/7447 --- ooni/config.py | 3 +++ ooni/runner.py | 5 +++++ 2 files changed, 8 insertions(+), 0 deletions(-) diff --git a/ooni/config.py b/ooni/config.py index b82936f..ce3fdcc 100644 --- a/ooni/config.py +++ b/ooni/config.py @@ -36,6 +36,9 @@ tor_control = None config_file = None sample_config_file = None +# This is used to store the probes IP address obtained via Tor +probe_ip = None + def get_root_path(): this_directory = os.path.dirname(__file__) root = os.path.join(this_directory, '..') diff --git a/ooni/runner.py b/ooni/runner.py index bc9b874..a948451 100644 --- a/ooni/runner.py +++ b/ooni/runner.py @@ -474,10 +474,15 @@ def startTor(): socks_port = yield state.protocol.get_conf("SocksPort") control_port = yield state.protocol.get_conf("ControlPort") + client_ip = yield state.protocol.get_info("address") config.tor.socks_port = int(socks_port.values()[0]) config.tor.control_port = int(control_port.values()[0]) + config.probe_ip = client_ip.values()[0] + + log.debug("Obtained our IP address from a Tor Relay %s" % config.privacy.client_ip) + def setup_failed(failure): log.exception(failure) raise UnableToStartTor

1 0

[ooni-probe/master] Make Beautiful soup a soft depedency
by art＠torproject.org 07 Dec '12

07 Dec '12

commit 40da6429be3348ae7a968658419402dba3b79dcc Author: Arturo Filastò <art(a)fuffa.org> Date: Thu Dec 6 22:36:10 2012 +0100 Make Beautiful soup a soft depedency * Remove un-needed dependencies from requirements.txt --- ooni/kit/domclass.py | 9 +++++++-- oonib/requirements.txt | 40 +++++++++++++++++----------------------- requirements.txt | 2 -- 3 files changed, 24 insertions(+), 27 deletions(-) diff --git a/ooni/kit/domclass.py b/ooni/kit/domclass.py index 1cf33a0..f287f98 100644 --- a/ooni/kit/domclass.py +++ b/ooni/kit/domclass.py @@ -28,12 +28,12 @@ # resepectively the eigenvalues for the probability matrix A and the # probability matrix B. # - import yaml import numpy -from bs4 import BeautifulSoup import time +from ooni import log + # All HTML4 tags # XXX add link to W3C page where these came from alltags = ['A', 'ABBR', 'ACRONYM', 'ADDRESS', 'APPLET', 'AREA', 'B', 'BASE', @@ -118,6 +118,11 @@ def readDOM(content=None, filename=None, debug=False): :filename: the filename to be read from for getting the content of the page. """ + try: + from bs4 import BeautifulSoup + except ImportError: + log.err("BeautifulSoup is not installed. This test canno run") + raise Exception if filename: f = open(filename) diff --git a/oonib/requirements.txt b/oonib/requirements.txt index e84e681..90fece5 100644 --- a/oonib/requirements.txt +++ b/oonib/requirements.txt @@ -1,24 +1,18 @@ -BeautifulSoup==3.2.1 -Jinja2==2.6 -PyYAML==3.10 -Pygments==1.5 -Sphinx==1.1.3 -Twisted==12.2.0 -argparse==1.2.1 -cyclone==1.0-rc13 -distribute==0.6.24 -dnspython==1.10.0 -docutils==0.9.1 -ipaddr==2.1.10 -numpy==1.6.2 -pyOpenSSL==0.13 -pygeoip==0.2.5 -repoze.sphinx.autointerface==0.7.1 -https://hg.secdev.org/scapy/archive/tip.zip#egg=scapy -storm==0.19 -transaction==1.3.0 +PyYAML>=3.10 +Pygments>=1.5 +Twisted>=12.2.0 +argparse>=1.2.1 +cyclone>=1.0-rc13 +distribute>=0.6.24 +docutils>=0.9.1 +ipaddr>=2.1.10 +pyOpenSSL>=0.13 +pygeoip>=0.2.5 +http://hg.secdev.org/scapy/archive/tip.zip#egg=scapy +storm>=0.19 +transaction>=1.3.0 txtorcon -wsgiref==0.1.2 -zope.component==4.0.0 -zope.event==4.0.0 -zope.interface==4.0.1 +wsgiref>=0.1.2 +zope.component>=4.0.0 +zope.event>=4.0.0 +zope.interface>=4.0.1 diff --git a/requirements.txt b/requirements.txt index ce2e721..1243806 100644 --- a/requirements.txt +++ b/requirements.txt @@ -1,7 +1,5 @@ -Jinja2>=2.6 PyYAML>=3.10 Pygments>=1.5 -Sphinx>=1.1.3 Twisted>=12.1.0 argparse>=1.2.1 distribute>=0.6.24

1 0

[ooni-probe/master] Add pointers to where to find the list of country codes
by art＠torproject.org 07 Dec '12

07 Dec '12

commit ef3a9e5448c1246f8f6b64e18e6feb1740bc7407 Author: Arturo Filastò <art(a)fuffa.org> Date: Thu Dec 6 19:08:53 2012 +0100 Add pointers to where to find the list of country codes --- docs/source/reports.rst | 9 ++++++--- 1 files changed, 6 insertions(+), 3 deletions(-) diff --git a/docs/source/reports.rst b/docs/source/reports.rst index 7a49dcb..00c2db9 100644 --- a/docs/source/reports.rst +++ b/docs/source/reports.rst @@ -18,12 +18,15 @@ following convention: ``dateInISO8601Format`` _ ``probeASNumber`` .yamloo -The date is expressed using `ISO 8601 <http://en.wikipedia.org/wiki/ISO_8601`_ +The date is expressed using `ISO 8601 <http://en.wikipedia.org/wiki/ISO_8601>`_ including seconds and with no ``:`` to delimit hours, minutes, days. -Like so +Like so: - ``YEAR`` - ``MONTH`` - ``DAY`` T ``HOURS`` ``MINUTES`` ``SECONDS`` Z +``YEAR`` - ``MONTH`` - ``DAY`` T ``HOURS`` ``MINUTES`` ``SECONDS`` Z + +Look `here for the up to date list of ISO 8601 country codes +<http://www.iso.org/iso/home/standards/country_codes/country_names_and_code_…>`_ The time is **always** expressed in UTC.

1 0