October 2012 - tor-commits - lists.torproject.org

r25839: {website} someone noticed andrew used to sign rpms too, list out the f (in website/trunk: about/en docs/en)
by Andrew Lewman 06 Oct '12

06 Oct '12

Author: phobos Date: 2012-10-06 01:48:05 +0000 (Sat, 06 Oct 2012) New Revision: 25839 Modified: website/trunk/about/en/financials.wml website/trunk/docs/en/signing-keys.wml Log: someone noticed andrew used to sign rpms too, list out the first two 2011 financial docs. Modified: website/trunk/about/en/financials.wml =================================================================== --- website/trunk/about/en/financials.wml 2012-10-05 21:54:42 UTC (rev 25838) +++ website/trunk/about/en/financials.wml 2012-10-06 01:48:05 UTC (rev 25839) @@ -13,18 +13,33 @@ <div id="maincol"> <h1>Tor: Financial Reports</h1> <dl> + <dt>Fiscal Year 2011</dt> + <dd><a href="findoc/2011-TorProject-Amended-Final-Report.pdf">2011 + Financial Statements and Audit Report</a></dd> + <dd><a href="findoc/2011-TorProject-DCF.pdf">2011 Dept of Commerce + Data Collection Form</a></dd> <dt>Fiscal Year 2010</dt> - <dd><a href="findoc/2010-TorProject-combined-Form990_PC_Audit_Results.pdf">2010 IRS Form 990, State of MA Form PC, and Independent Audit Results</a></dd> + <dd><a + href="findoc/2010-TorProject-combined-Form990_PC_Audit_Results.pdf">2010 + IRS Form 990, State of MA Form PC, and Independent Audit + Results</a></dd> <dt>Fiscal Year 2009</dt> - <dd><a href="findoc/2009-TorProject-Annual-Report.pdf">2009 Annual Report</a></dd> - <dd><a href="findoc/2009-TorProject-Form990andPC.pdf">2009 IRS Form 990 and State of MA Form PC</a></dd> - <dd><a href="findoc/2009-TorProject-FinancialStatements.pdf">2009 Financial Statements and Audit results</a></dd> - <dd><a href="findoc/2009-TorProject-DCF.pdf">2009 Dept of Commerce Data Collection Form</a></dd> + <dd><a href="findoc/2009-TorProject-Annual-Report.pdf">2009 + Annual Report</a></dd> + <dd><a href="findoc/2009-TorProject-Form990andPC.pdf">2009 IRS + Form 990 and State of MA Form PC</a></dd> + <dd><a href="findoc/2009-TorProject-FinancialStatements.pdf">2009 + Financial Statements and Audit results</a></dd> + <dd><a href="findoc/2009-TorProject-DCF.pdf">2009 Dept of Commerce + Data Collection Form</a></dd> <dt>Fiscal Year 2008</dt> - <dd><a href="findoc/2008-TorProject-Form990.pdf">2008 IRS Form 990</a></dd> - <dd><a href="findoc/2008-TorProject-FinancialStatements.pdf">2008 Financial Statements and Audit Results</a></dd> + <dd><a href="findoc/2008-TorProject-Form990.pdf">2008 IRS Form + 990</a></dd> + <dd><a href="findoc/2008-TorProject-FinancialStatements.pdf">2008 + Financial Statements and Audit Results</a></dd> <dt>Fiscal Year 2007</dt> - <dd><a href="findoc/2007-TorProject-Form990.pdf">2007 IRS Form 990</a></dd> + <dd><a href="findoc/2007-TorProject-Form990.pdf">2007 IRS Form + 990</a></dd> </dl> </div>  Modified: website/trunk/docs/en/signing-keys.wml =================================================================== --- website/trunk/docs/en/signing-keys.wml 2012-10-05 21:54:42 UTC (rev 25838) +++ website/trunk/docs/en/signing-keys.wml 2012-10-06 01:48:05 UTC (rev 25839) @@ -19,8 +19,8 @@ tarballs.</li> <li>Erinn Clark (0x63FEE659) signs the Tor Browser Bundles, Vidalia bundles, and many other packages. She signs RPMs with her other key - (0xF1F5C9B5). Andrew Lewman (0x31B0974B, 0x6B4D6475) used to sign packages for - Windows and OS X.</li> + (0xF1F5C9B5). Andrew Lewman (0x31B0974B, 0x6B4D6475) used to sign + packages for RPMs, Windows, and OS X.</li> <li>Tor Project Archive (0x886DDD89) signs the deb.torproject.org repositories and archives.</li> <li>Tomás Touceda (0x9A753A6B) signs current Vidalia tarballs. Matt @@ -79,9 +79,7 @@ pub 4096R/6B4D6475 2012-02-29 Key fingerprint = 0291 ECCB E42B 2206 8E68 5545 627D EE28 6B4D 6475 - uid Andrew Lewman <andrew(a)lewman.is> uid Andrew Lewman <andrew(a)torproject.org> - uid Andrew Lewman <andrew(a)lewman.com> uid Andrew Lewman <andrew(a)torproject.is> sub 4096R/BE713AC6 2012-02-29

1 0

r25838: {website} mike points out another characteristic that would be nice to (website/trunk/about/en)
by Roger Dingledine 05 Oct '12

05 Oct '12

Author: arma Date: 2012-10-05 21:54:42 +0000 (Fri, 05 Oct 2012) New Revision: 25838 Modified: website/trunk/about/en/jobs-projectcoordinator.wml Log: mike points out another characteristic that would be nice to have Modified: website/trunk/about/en/jobs-projectcoordinator.wml =================================================================== --- website/trunk/about/en/jobs-projectcoordinator.wml 2012-10-05 16:20:39 UTC (rev 25837) +++ website/trunk/about/en/jobs-projectcoordinator.wml 2012-10-05 21:54:42 UTC (rev 25838) @@ -61,6 +61,7 @@ working with distributed teams across different time-zones containing employees and volunteers of differing skill levels over multiple mediums, including email, instant messaging, and IRC.</li> +<li>Have experience maintaining long-term software projects. <li>Have basic familiarity with distributed version control systems.</li> </ul>

1 0

r25837: {} seed the audit and dcf reports. (website/trunk/about/findoc)
by Andrew Lewman 05 Oct '12

05 Oct '12

Author: phobos Date: 2012-10-05 16:20:39 +0000 (Fri, 05 Oct 2012) New Revision: 25837 Added: website/trunk/about/findoc/2011-TorProject-Amended-Final-Report.pdf website/trunk/about/findoc/2011-TorProject-Amended-Final-Report.pdf.asc website/trunk/about/findoc/2011-TorProject-DCF.pdf website/trunk/about/findoc/2011-TorProject-DCF.pdf.asc Log: seed the audit and dcf reports. Added: website/trunk/about/findoc/2011-TorProject-Amended-Final-Report.pdf =================================================================== (Binary files differ) Property changes on: website/trunk/about/findoc/2011-TorProject-Amended-Final-Report.pdf ___________________________________________________________________ Added: svn:mime-type + application/octet-stream Added: website/trunk/about/findoc/2011-TorProject-Amended-Final-Report.pdf.asc =================================================================== --- website/trunk/about/findoc/2011-TorProject-Amended-Final-Report.pdf.asc (rev 0) +++ website/trunk/about/findoc/2011-TorProject-Amended-Final-Report.pdf.asc 2012-10-05 16:20:39 UTC (rev 25837) @@ -0,0 +1,16 @@ +-----BEGIN PGP SIGNATURE----- + +iQIcBAABCAAGBQJQbwfdAAoJEGJ97ihrTWR1a4oQAKHa7GEYTU9hld8UznLLQ989 +ZuzZVNKlxzBXc264AM1As7xjyZb6jZ7JF3s3Leor7bgnC4c3re3wIgCteDpde04h +5UxaJwuvIbCo5hJBEQ0eas/5UDphNroqwypWJznoy3BPK0vuSy9eCenFMa3PNO1w +BiKneQ4H/TU50lU4ToLsB5t6r2j9es+92ZOnIYVn3UpKEceNFFGtOIbiGlRMu6+I +UY8tyDS1ewVOT89D3XygjIU0iyVeiWDicHMl1RUkxMWg7hM8KCC2VpLujjWLuWBs +N5ZfpdQaKJMKSSMUwXfQZU4TN3ffhfeBhfRNg8zJ3ryozfWtBjRYLtAGTzKEHCeJ +eN2ZUXFCU2PeTwgUA7Q8Zl41VoGDIr/OG3OYri2PCbFOmXpe6AMpuGp4eEHPmB7H +KitqFn4gHpdfYqGUQPsSU4RgyKPg7sArhRGTCetTouu/w2JjZ6W7hLVYzkMjmZaS +4IYt2+rTjinkBXii27M/Pv+FWBPlLn5bg5tawt4SrbNxDkFZZ+L7CF+MO4KfTTan +ChMg5VjrX+zjQmPwyp+GFfohq2OSnScAFeLBx/ScwZm5NJVFXwih5X96pYgAUnq7 +VBIWvN29TMls1msVJhAtWLQlpaJb/J/FCBUi+I7H6SiYCXrUWZGE2l3Iax6Kj8Gp +x+7uQDsngWNCdQQ59RGB +=h7ca +-----END PGP SIGNATURE----- Property changes on: website/trunk/about/findoc/2011-TorProject-Amended-Final-Report.pdf.asc ___________________________________________________________________ Added: svn:mime-type + text/plain Added: website/trunk/about/findoc/2011-TorProject-DCF.pdf =================================================================== (Binary files differ) Property changes on: website/trunk/about/findoc/2011-TorProject-DCF.pdf ___________________________________________________________________ Added: svn:mime-type + application/octet-stream Added: website/trunk/about/findoc/2011-TorProject-DCF.pdf.asc =================================================================== --- website/trunk/about/findoc/2011-TorProject-DCF.pdf.asc (rev 0) +++ website/trunk/about/findoc/2011-TorProject-DCF.pdf.asc 2012-10-05 16:20:39 UTC (rev 25837) @@ -0,0 +1,16 @@ +-----BEGIN PGP SIGNATURE----- + +iQIcBAABCAAGBQJQbwflAAoJEGJ97ihrTWR1QQkP/3CEEXtGhaf+7/xypcN1ZulI +R9m1EPmXpxQbTbA8YbrdCfLJ0ct9tcqG+zUoeNX6RZ1KIohiuBbvKrwIPpUvBmN+ +B7YBqNg1Wahk58JpOCCHloonLqhMtZTPIkyp5cBCbgtsIQrAV8Jwfzkmr+ur9ALn +TXU4w7nk7B8GkS65K+rVNwR+eKo73/zBFj4XFuWl7mAK7WbYVt1/Qz6ZjHZvPNtu +dcRSLhAIHq5yDJO+5w3PhMddq3njGQB97lhE/eMS2/HkbPpwUjpDnXf9Yc8suhcV +I4LZMT1htQz+BltgJ1Viu8EZ4kH2wjYGMHvlqsOs2afqp3syWzlU3Gqhn/A/9brS +UwcKqB56nP+g5Yf+Z6vGiXXz9nEH6v90NfxAsUMJMz8LLqv8qUXIcoNuRtkEB/f0 +uggRUY87Mn7BvihuZoAEwQ3aqKtCMPrCbjkO3my1c4HXCTE5GRv/qKUMhmB+lWbJ +Zl2OdXJb9zTMuZKsH/KswJXsS3rru6fPNmVPBJS0m0+a0E353DGiyTofUPdSaea2 +zzqcv91eVnzJT15U497tTlevr0TkCsuB1nvFDhpiigzxsjrHV3jcMoIXozozswla +bLxDbz+/e+6IwTEscSIR1BlD+Jr5UnG87kPNsX39vwSIWqPN1iMwrEem8qEBMOjk +R6dHsG3HVLJm4WDzRdV3 +=lJMW +-----END PGP SIGNATURE----- Property changes on: website/trunk/about/findoc/2011-TorProject-DCF.pdf.asc ___________________________________________________________________ Added: svn:mime-type + text/plain

1 0

[ooni-probe/master] Remove hardcoded sys.path from reporter
by art＠torproject.org 05 Oct '12

05 Oct '12

commit bf3a4c6648cfe987a759331637cc42e7bfeef567 Author: Arturo Filastò <arturo(a)filasto.net> Date: Thu Oct 4 13:38:54 2012 +0000 Remove hardcoded sys.path from reporter --- ooni/reporter.py | 1 - 1 files changed, 0 insertions(+), 1 deletions(-) diff --git a/ooni/reporter.py b/ooni/reporter.py index ee8427d..86c1fc1 100644 --- a/ooni/reporter.py +++ b/ooni/reporter.py @@ -3,7 +3,6 @@ import sys import yaml import itertools -sys.path.insert(0, '/home/x/Documents/pen_drive_bitcoin2012/ooni-probe/ENV/lib/python2.7/site-packages') from datetime import datetime from twisted.python.util import OrderedDict, untilConcludes from twisted.trial import unittest, reporter, runner

1 0

[ooni-probe/master] Fix date structure in reporting sytem.
by art＠torproject.org 05 Oct '12

05 Oct '12

commit 43cf06c1d2c5aa410aa8b220463d84f6ac7f0a26 Author: Arturo Filastò <arturo(a)filasto.net> Date: Tue Oct 2 23:45:07 2012 +0000 Fix date structure in reporting sytem. * Collect errors in httpt template * Rename testhelpers to avoid namespace conflicts * Miscelaneous bug fixing --- bin/ooniprobe | 2 +- nettests/example_httpt.py | 9 ++++- nettests/example_scapyt.py | 2 +- ooni/lib/txscapy.py | 2 +- ooni/nettest.py | 7 ++- ooni/reporter.py | 20 +++++---- ooni/runner.py | 6 ++- ooni/templates/httpt.py | 28 ++++++------- ooni/utils/date.py | 4 +- ooni/utils/log.py | 2 +- oonib/oonibackend.py | 5 ++- oonib/testhelpers/http.py | 74 ----------------------------------- oonib/testhelpers/httph.py | 92 ++++++++++++++++++++++++++++++++++++++++++++ 13 files changed, 143 insertions(+), 110 deletions(-) diff --git a/bin/ooniprobe b/bin/ooniprobe index c653756..4e400ce 100755 --- a/bin/ooniprobe +++ b/bin/ooniprobe @@ -1,4 +1,4 @@ -#!/usr/bin/python2 +#!/usr/bin/env python # -*- encoding: utf-8 -*- # # startup script based on twisted trial diff --git a/nettests/example_httpt.py b/nettests/example_httpt.py index 6501ca7..b113b23 100644 --- a/nettests/example_httpt.py +++ b/nettests/example_httpt.py @@ -10,7 +10,14 @@ class ExampleHTTP(httpt.HTTPTest): version = 0.1 inputs = ['http://google.com/', 'http://wikileaks.org/', - 'http://torproject.org/'] + 'http://torproject.org/'] + + def test_http(self): + if self.input: + url = self.input + return self.doRequest(url) + else: + raise Exception("No input specified") def processResponseBody(self, body): # XXX here shall go your logic diff --git a/nettests/example_scapyt.py b/nettests/example_scapyt.py index c7f342d..1cc8054 100644 --- a/nettests/example_scapyt.py +++ b/nettests/example_scapyt.py @@ -11,5 +11,5 @@ class ExampleScapy(scapyt.ScapyTest): version = 0.1 inputs = [IP(dst="8.8.8.8")/TCP(dport=31337), - IP(dst="ooni.nu")/TCP(dport=31337)] + IP(dst="ooni.nu")/TCP(dport=31337)] diff --git a/ooni/lib/txscapy.py b/ooni/lib/txscapy.py index ec605e5..d4f146e 100644 --- a/ooni/lib/txscapy.py +++ b/ooni/lib/txscapy.py @@ -84,7 +84,7 @@ class Scapy(object): """ min = 2 max = 6 - debug = True + debug = False write_only_answers = False pcapwriter = None recv = False diff --git a/ooni/nettest.py b/ooni/nettest.py index 403a022..d7188c6 100644 --- a/ooni/nettest.py +++ b/ooni/nettest.py @@ -81,8 +81,11 @@ class TestCase(unittest.TestCase): if self.inputFile: fp = open(self.inputFile) self.inputs = inputProcessor(fp) - - return {'inputs': self.inputs} + # XXX perhaps we may want to name and version to be inside of a + # different object that is not called options. + return {'inputs': self.inputs, + 'name': self.name, + 'version': self.version} def __repr__(self): return "<%s inputs=%s>" % (self.__class__, self.inputs) diff --git a/ooni/reporter.py b/ooni/reporter.py index 51a0166..3e108f9 100644 --- a/ooni/reporter.py +++ b/ooni/reporter.py @@ -7,6 +7,7 @@ sys.path.insert(0, '/home/x/Documents/pen_drive_bitcoin2012/ooni-probe/ENV/lib/p from datetime import datetime from twisted.python.util import OrderedDict, untilConcludes from twisted.trial import unittest, reporter, runner +from ooni.utils import date try: from scapy.all import packet @@ -70,19 +71,19 @@ class ReporterFactory(OReporter): self._testSuite = testSuite self._reporters = [] - def writeHeader(self): - pretty_date = "XXX Replace me with date.pretty_date()" + def writeHeader(self, options): self._writeln("###########################################") - self._writeln("# OONI Probe Report for Test %s" % "XXX replace with with the test suite name") - self._writeln("# %s" % pretty_date) + self._writeln("# OONI Probe Report for %s test" % options['name']) + self._writeln("# %s" % date.pretty_date()) self._writeln("###########################################") address = {'asn': 'XXX replace me with ASN', 'ip': 'XXX replace me with IP'} - test_details = {'start_time': datetime.now(), - 'asn': address['asn'], - 'test_name': 'XXX replace me with the test name', - 'addr': address['ip']} + test_details = {'startTime': repr(date.now()), + 'probeASN': address['asn'], + 'testName': options['name'], + 'testVersion': options['version'], + 'probeIP': address['ip']} self.writeYamlLine(test_details) self._writeln('') @@ -135,8 +136,9 @@ class OONIReporter(OReporter): test_input = repr(test.input) else: test_input = test.input + self._tests[idx]['input'] = test_input - self._tests[idx]['idx'] = idx + #self._tests[idx]['idx'] = idx self._tests[idx]['name'] = test.name #self._tests[idx]['test'] = test print "Now starting %s" % self._tests[idx] diff --git a/ooni/runner.py b/ooni/runner.py index 7883f07..0738bf8 100644 --- a/ooni/runner.py +++ b/ooni/runner.py @@ -129,6 +129,8 @@ def loadTestsAndOptions(classes): suiteFactory = InputTestSuite options = [] testCases = [] + names = [] + for klass in classes: try: k = klass() @@ -184,8 +186,8 @@ class ORunner(object): def run(self): #log.startLogging(sys.stdout) - log.start(True) - self.reporterFactory.writeHeader() + log.start() + self.reporterFactory.writeHeader(self.options) for inputUnit in InputUnitFactory(self.inputs): self.runWithInputUnit(inputUnit) diff --git a/ooni/templates/httpt.py b/ooni/templates/httpt.py index 51775f0..a537c6e 100644 --- a/ooni/templates/httpt.py +++ b/ooni/templates/httpt.py @@ -106,27 +106,25 @@ class HTTPTest(TestCase): """ pass - def doRequest(self, url): - d = self.build_request(url) + def doRequest(self, url, method="GET", headers=None, body=None): + try: + d = self.build_request(url, method, headers, body) + except Exception, e: + print e + self.report['error'] = e + + def errback(data): + print data + #self.report["error"] = data + def finished(data): - #self.mainDefer.callback() - return data + return + d.addErrback(errback) d.addCallback(self._cbResponse) d.addCallback(finished) return d - def test_http(self): - log.msg("Running experiment") - - if self.input: - url = self.input - else: - raise Exception("No input supplied") - - self.mainDefer = self.doRequest(url) - return self.mainDefer - def _cbResponse(self, response): self.response['headers'] = response.headers self.response['code'] = response.code diff --git a/ooni/utils/date.py b/ooni/utils/date.py index 59ec1f8..dcd8e36 100644 --- a/ooni/utils/date.py +++ b/ooni/utils/date.py @@ -3,10 +3,10 @@ from datetime import datetime class odate(datetime): def __str__(self): - return rfc3339(self) + return "%s" % rfc3339(self) def __repr__(self): - return "'%s'" % rfc3339(self) + return "%s" % rfc3339(self) def from_rfc(self, datestr): pass diff --git a/ooni/utils/log.py b/ooni/utils/log.py index 6ff6bf9..06bc3fa 100644 --- a/ooni/utils/log.py +++ b/ooni/utils/log.py @@ -73,7 +73,7 @@ class OONILogObserver(log.FileLogObserver): util.untilConcludes(self.write, timeStr + " " + msgStr) util.untilConcludes(self.flush) -def start(logstdout, logfile=None, verbosity=None): +def start(logfile=None, logstdout=True, verbosity=None): if log.defaultObserver: verbosity = _get_log_level(verbosity) diff --git a/oonib/oonibackend.py b/oonib/oonibackend.py index d6ba3f0..0232829 100644 --- a/oonib/oonibackend.py +++ b/oonib/oonibackend.py @@ -17,8 +17,8 @@ from twisted.web.microdom import escape from twisted.names import dns from oonib.lib import config -from oonib.testhelpers.http import HTTPBackend from oonib.lib.ssl import SSLContext +from oonib.testhelpers.httph import HTTPBackend, DebugHTTPServer from oonib.testhelpers.dns import ProxyDNSServer from oonib.testhelpers.daphn3 import Daphn3Server @@ -37,6 +37,9 @@ internet.SSLServer(int(config.main.ssl_port), SSLContext(config), ).setServiceParent(serviceCollection) +debugHTTPServer = DebugHTTPServer() +internet.TCPServer(8090, debugHTTPServer).setServiceParent(serviceCollection) + # Start the DNS Server related services TCPDNSServer = ProxyDNSServer() internet.TCPServer(int(config.main.dns_tcp_port), TCPDNSServer).setServiceParent(serviceCollection) diff --git a/oonib/testhelpers/http.py b/oonib/testhelpers/http.py deleted file mode 100644 index 79a487b..0000000 --- a/oonib/testhelpers/http.py +++ /dev/null @@ -1,74 +0,0 @@ -import json -import random -import string - -from twisted.application import internet, service -from twisted.internet import protocol, reactor, defer -from twisted.protocols import basic -from twisted.web import resource, server, static -from twisted.web.microdom import escape - -server.version = "Apache" - -class HTTPRandomPage(resource.Resource): - """ - This generates a random page of arbitrary length and containing the string - selected by the user. - The format is the following: - /random/<length>/<keyword> - """ - isLeaf = True - def _gen_random_string(self, length): - return ''.join(random.choice(string.letters) for x in range(length)) - - def genRandomPage(self, length=100, keyword=None): - data = self._gen_random_string(length/2) - if keyword: - data += keyword - data += self._gen_random_string(length - length/2) - data += '\n' - return data - - def render(self, request): - length = 100 - keyword = None - path_parts = request.path.split('/') - if len(path_parts) > 2: - length = int(path_parts[2]) - if length > 100000: - length = 100000 - - if len(path_parts) > 3: - keyword = escape(path_parts[3]) - - return self.genRandomPage(length, keyword) - -class HTTPReturnHeaders(resource.Resource): - """ - This returns the headers being sent by the client in JSON format. - """ - isLeaf = True - def render(self, request): - req_headers = request.getAllHeaders() - return json.dumps(req_headers) - -class HTTPSendHeaders(resource.Resource): - """ - This sends to the client the headers that they send inside of the POST - request encoded in json. - """ - isLeaf = True - def render_POST(self, request): - headers = json.loads(request.content.read()) - for header, value in headers.items(): - request.setHeader(str(header), str(value)) - return '' - -class HTTPBackend(resource.Resource): - def __init__(self): - resource.Resource.__init__(self) - self.putChild('random', HTTPRandomPage()) - self.putChild('returnheaders', HTTPReturnHeaders()) - self.putChild('sendheaders', HTTPSendHeaders()) - - diff --git a/oonib/testhelpers/httph.py b/oonib/testhelpers/httph.py new file mode 100644 index 0000000..217dbd5 --- /dev/null +++ b/oonib/testhelpers/httph.py @@ -0,0 +1,92 @@ +import json +import random +import string + +from twisted.application import internet, service +from twisted.internet import protocol, reactor, defer +from twisted.protocols import basic +from twisted.web import resource, server, static, http +from twisted.web.microdom import escape + +server.version = "Apache" + +class HTTPRandomPage(resource.Resource): + """ + This generates a random page of arbitrary length and containing the string + selected by the user. + The format is the following: + /random/<length>/<keyword> + """ + isLeaf = True + def _gen_random_string(self, length): + return ''.join(random.choice(string.letters) for x in range(length)) + + def genRandomPage(self, length=100, keyword=None): + data = self._gen_random_string(length/2) + if keyword: + data += keyword + data += self._gen_random_string(length - length/2) + data += '\n' + return data + + def render(self, request): + length = 100 + keyword = None + path_parts = request.path.split('/') + if len(path_parts) > 2: + length = int(path_parts[2]) + if length > 100000: + length = 100000 + + if len(path_parts) > 3: + keyword = escape(path_parts[3]) + + return self.genRandomPage(length, keyword) + +class HTTPReturnHeaders(resource.Resource): + """ + This returns the headers being sent by the client in JSON format. + """ + isLeaf = True + def render(self, request): + req_headers = request.getAllHeaders() + return json.dumps(req_headers) + +class HTTPSendHeaders(resource.Resource): + """ + This sends to the client the headers that they send inside of the POST + request encoded in json. + """ + isLeaf = True + def render_POST(self, request): + headers = json.loads(request.content.read()) + for header, value in headers.items(): + request.setHeader(str(header), str(value)) + return '' + +class HTTPBackend(resource.Resource): + def __init__(self): + resource.Resource.__init__(self) + self.putChild('random', HTTPRandomPage()) + self.putChild('returnheaders', HTTPReturnHeaders()) + self.putChild('sendheaders', HTTPSendHeaders()) + +class DebugProtocol(http.HTTPChannel): + def headerReceived(self, line): + print "[HEADER] %s" % line + http.HTTPChannel.headerReceived(self, line) + + def allContentReceived(self): + print self.requests[-1].getAllHeaders() + self.transport.loseConnection() + self.connectionLost("Normal closure") + +class DebugHTTPServer(http.HTTPFactory): + protocol = DebugProtocol + + def buildProtocol(self, addr): + print "Got connection from %s" % addr + p = protocol.ServerFactory.buildProtocol(self, addr) + p.timeOut = self.timeOut + return p +

1 0

[ooni-probe/master] Do some fixes to the reporter
by art＠torproject.org 05 Oct '12

05 Oct '12

commit f4ac83c43dd27ffb05686e311ded33b850e62acf Author: Arturo Filastò <arturo(a)filasto.net> Date: Wed Oct 3 12:25:02 2012 +0000 Do some fixes to the reporter * Remove useless print lines * Re-add hacks.py --- nettests/http_host.py | 47 ++++++++++++++++++++++++++++++++++++++ nettests/myip.py | 1 - ooni/reporter.py | 12 +++++++-- ooni/templates/httpt.py | 5 ---- ooni/utils/hacks.py | 58 +++++++++++++++++++++++++++++++++++++++++++++++ 5 files changed, 114 insertions(+), 9 deletions(-) diff --git a/nettests/http_host.py b/nettests/http_host.py new file mode 100644 index 0000000..4834ba3 --- /dev/null +++ b/nettests/http_host.py @@ -0,0 +1,47 @@ +# -*- encoding: utf-8 -*- +# +# HTTP Host Test +# ************** +# +# for more details see: +# https://trac.torproject.org/projects/tor/wiki/doc/OONI/Tests/HTTPHost +# +# :authors: Arturo Filastò +# :licence: see LICENSE + +from ooni.templates import httpt + +good_http_server = "http://127.0.0.1:8090/" + +class HTTPHost(httpt.HTTPTest): + name = "HTTP Host" + author = "Arturo Filastò" + version = 0.1 + + + inputs = ['google.com', 'wikileaks.org', + 'torproject.org'] + + def test_send_host_header(self): + headers = {} + headers["Host"] = [self.input] + return self.doRequest(good_http_server, headers=headers) + + def processResponseBody(self, body): + # XXX here shall go your logic + # for processing the body + if 'blocked' in body: + self.report['censored'] = True + else: + self.report['censored'] = False + + def processResponseHeaders(self, headers): + # XXX place in here all the logic for handling the processing of HTTP + # Headers. + if headers.hasHeader('location'): + self.report['redirect'] = True + + server = headers.getRawHeaders("Server") + if server: + self.report['http_server'] = str(server.pop()) + diff --git a/nettests/myip.py b/nettests/myip.py index e7e96f2..40a4849 100644 --- a/nettests/myip.py +++ b/nettests/myip.py @@ -7,7 +7,6 @@ from ooni.templates import httpt class MyIP(httpt.HTTPTest): inputs = ['https://check.torproject.org'] def processResponseBody(self, body): - print "FOOOO" import re regexp = "Your IP address appears to be: <b>(.+?)<\/b>" match = re.search(regexp, body) diff --git a/ooni/reporter.py b/ooni/reporter.py index 3e108f9..ee8427d 100644 --- a/ooni/reporter.py +++ b/ooni/reporter.py @@ -71,14 +71,20 @@ class ReporterFactory(OReporter): self._testSuite = testSuite self._reporters = [] - def writeHeader(self, options): + def writeHeader(self, options, geodata={}): self._writeln("###########################################") self._writeln("# OONI Probe Report for %s test" % options['name']) self._writeln("# %s" % date.pretty_date()) self._writeln("###########################################") - address = {'asn': 'XXX replace me with ASN', - 'ip': 'XXX replace me with IP'} + address = {'asn': 'unknown', + 'ip': 'unknown'} + if 'ip' in geodata: + address['ip'] = geodata['ip'] + + if 'asn' in geodata: + address['asn'] = geodata['asn'] + test_details = {'startTime': repr(date.now()), 'probeASN': address['asn'], 'testName': options['name'], diff --git a/ooni/templates/httpt.py b/ooni/templates/httpt.py index a537c6e..4b8fa9b 100644 --- a/ooni/templates/httpt.py +++ b/ooni/templates/httpt.py @@ -28,11 +28,6 @@ useragents = [("Mozilla/5.0 (Windows; U; Windows NT 5.1; en-GB; rv:1.8.1.6) Geck ("Mozilla/4.0 (compatible; MSIE 6.0; MSIE 5.5; Windows NT 5.1) Opera 7.02 [en]", "Opera 7.02, Windows XP"), ("Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.7.5) Gecko/20060127 Netscape/8.1", "Netscape 8.1, Windows XP")] - -class WebClientContextFactory(ClientContextFactory): - def getContext(self, hostname, port): - return ClientContextFactory.getContext(self) - class BodyReceiver(protocol.Protocol): def __init__(self, finished): self.finished = finished diff --git a/ooni/utils/hacks.py b/ooni/utils/hacks.py new file mode 100644 index 0000000..674f45e --- /dev/null +++ b/ooni/utils/hacks.py @@ -0,0 +1,58 @@ +# -*- encoding: utf-8 -*- +# +# :authors: Arturo Filastò +# :licence: see LICENSE + +import copy_reg + +def patched_reduce_ex(self, proto): + """ + This is a hack to overcome a bug in one of pythons core functions. It is + located inside of copy_reg and is called _reduce_ex. + + Some background on the issue can be found here: + http://stackoverflow.com/questions/569754/how-to-tell-for-which-object-attr… + http://stackoverflow.com/questions/2049849/why-cant-i-pickle-this-object + + There was also an open bug on the pyyaml trac repo, but it got closed because + they could not reproduce. + http://pyyaml.org/ticket/190 + + It turned out to be easier to patch the python core library than to monkey + patch yaml. + + XXX see if there is a better way. sigh... + """ + _HEAPTYPE = 1<<9 + assert proto < 2 + for base in self.__class__.__mro__: + if hasattr(base, '__flags__') and not base.__flags__ & _HEAPTYPE: + break + else: + base = object # not really reachable + if base is object: + state = None + elif base is int: + state = None + else: + if base is self.__class__: + raise TypeError, "can't pickle %s objects" % base.__name__ + state = base(self) + args = (self.__class__, base, state) + try: + getstate = self.__getstate__ + except AttributeError: + if getattr(self, "__slots__", None): + raise TypeError("a class that defines __slots__ without " + "defining __getstate__ cannot be pickled") + try: + dict = self.__dict__ + except AttributeError: + dict = None + else: + dict = getstate() + if dict: + return copy_reg._reconstructor, args, dict + else: + return copy_reg._reconstructor, args +

1 0

[ooni-probe/master] Add utility to obtain IP address of the client based on torproject.org
by art＠torproject.org 05 Oct '12

05 Oct '12

commit 67b024a0361a0585ebc3bf9ed615ef8bf7c793fd Author: Arturo Filastò <arturo(a)filasto.net> Date: Fri Oct 5 11:05:04 2012 +0000 Add utility to obtain IP address of the client based on torproject.org --- ooni/utils/geodata.py | 35 +++++++++++++++++++++++++++++++++++ 1 files changed, 35 insertions(+), 0 deletions(-) diff --git a/ooni/utils/geodata.py b/ooni/utils/geodata.py new file mode 100644 index 0000000..3e5202e --- /dev/null +++ b/ooni/utils/geodata.py @@ -0,0 +1,35 @@ +import re +from twisted.web.client import Agent +from twisted.internet import reactor, defer, protocol + +class BodyReceiver(protocol.Protocol): + def __init__(self, finished): + self.finished = finished + self.data = "" + + def dataReceived(self, bytes): + self.data += bytes + + def connectionLost(self, reason): + self.finished.callback(self.data) + +(a)defer.inlineCallbacks +def myIP(): + target_site = 'https://check.torproject.org/' + regexp = "Your IP address appears to be: <b>(.+?)<\/b>" + + myAgent = Agent(reactor) + result = yield myAgent.request('GET', target_site) + finished = defer.Deferred() + result.deliverBody(BodyReceiver(finished)) + body = yield finished + + match = re.search(regexp, body) + try: + myip = match.group(1) + except: + myip = "unknown" + + return myip + +

1 0

[ooni-probe/master] Merge branch 'master' of ssh://git-rw.torproject.org/ooni-probe
by art＠torproject.org 05 Oct '12

05 Oct '12

commit 9ed3bddfd0022058836d1fd2bd4ba448d300f3b1 Merge: 67b024a 7fdca5a Author: Arturo Filastò <arturo(a)filasto.net> Date: Fri Oct 5 14:08:52 2012 +0000 Merge branch 'master' of ssh://git-rw.torproject.org/ooni-probe pretty weird coflict :P Conflicts: ooni/utils/hacks.py INSTALL | 6 +- LICENSE | 2 +- ooni/plugins/bridget.py | 496 +++++++++++++++++++++++++++++++ ooni/plugins/new_bridget.py | 528 --------------------------------- ooni/plugoo/assets.py | 6 + ooni/plugoo/tests.py | 1 - ooni/utils/config.py | 68 +++++ ooni/utils/hacks.py | 3 +- ooni/utils/log.py | 10 +- ooni/utils/onion.py | 689 +++++++++++++++++++++++++++++++++++++++++++ ooni/utils/process.py | 54 ++++ ooni/utils/timer.py | 107 +++++++ 12 files changed, 1434 insertions(+), 536 deletions(-) diff --cc ooni/utils/hacks.py index 674f45e,5ef7c8b..cf5f080 --- a/ooni/utils/hacks.py +++ b/ooni/utils/hacks.py @@@ -11,7 -6,7 +11,8 @@@ def patched_reduce_ex(self, proto) located inside of copy_reg and is called _reduce_ex. Some background on the issue can be found here: - http://stackoverflow.com/questions/569754/how-to-tell-for-which-object-attr… ++ + http://stackoverflow.com/questions/569754/how-to-tell-for-which-object-attr… http://stackoverflow.com/questions/2049849/why-cant-i-pickle-this-object There was also an open bug on the pyyaml trac repo, but it got closed because

1 0

[ooni-probe/master] Add requirements.txt and add some directories to .gitignore
by art＠torproject.org 05 Oct '12

05 Oct '12

commit 019cd78fdd1249d214e6dc7d641eb7647bd2e9d7 Author: Arturo Filastò <arturo(a)filasto.net> Date: Fri Oct 5 11:03:35 2012 +0000 Add requirements.txt and add some directories to .gitignore --- .gitignore | 4 ++++ requirements.txt | 3 +++ 2 files changed, 7 insertions(+), 0 deletions(-) diff --git a/.gitignore b/.gitignore index dc569e9..5d6ba15 100644 --- a/.gitignore +++ b/.gitignore @@ -1,5 +1,6 @@ *.pyc assets/* +ENV/* build/* package/* proxy-lists/ip-cc.txt @@ -10,3 +11,6 @@ private/* /ooni/plugins/dropin.cache oonib/oonibackend.conf ooni/lib/txtorcon +fluff/ +pcaps/ +oonib/data/* diff --git a/requirements.txt b/requirements.txt new file mode 100644 index 0000000..faada4c --- /dev/null +++ b/requirements.txt @@ -0,0 +1,3 @@ +twisted +Scapy +pyyaml

1 0

[ooni-probe/master] Now we have the database properly hooked up to OONIB
by art＠torproject.org 05 Oct '12

05 Oct '12

commit b1ca4257c13dff18664c8b175ed6b57acc8874b3 Author: Arturo Filastò <arturo(a)filasto.net> Date: Fri Oct 5 10:58:57 2012 +0000 Now we have the database properly hooked up to OONIB * Support writing to DB of data relative to a report --- oonib/oonibackend.conf.sample | 1 + oonib/oonibackend.py | 36 ++++++++++++++----- oonib/report/api.py | 75 +++++++++++++++++++++++++++++----------- oonib/report/db/__init__.py | 20 +++++----- oonib/report/db/models.py | 44 +++++++++++++++++++++++- 5 files changed, 134 insertions(+), 42 deletions(-) diff --git a/oonib/oonibackend.conf.sample b/oonib/oonibackend.conf.sample index 021e0e2..dc0a662 100644 --- a/oonib/oonibackend.conf.sample +++ b/oonib/oonibackend.conf.sample @@ -1,4 +1,5 @@ [main] +reporting_port = 8888 http_port = 8080 dns_udp_port = 5354 dns_tcp_port = 8002 diff --git a/oonib/oonibackend.py b/oonib/oonibackend.py index 0232829..29f3f58 100644 --- a/oonib/oonibackend.py +++ b/oonib/oonibackend.py @@ -16,6 +16,7 @@ from twisted.web import resource, server, static from twisted.web.microdom import escape from twisted.names import dns +from oonib.report.api import reportingBackend from oonib.lib import config from oonib.lib.ssl import SSLContext from oonib.testhelpers.httph import HTTPBackend, DebugHTTPServer @@ -28,11 +29,13 @@ server.version = config.main.server_version application = service.Application('oonibackend') serviceCollection = service.IServiceCollection(application) -internet.TCPServer(int(config.main.http_port), +if config.main.http_port: + internet.TCPServer(int(config.main.http_port), server.Site(HTTPBackend()) ).setServiceParent(serviceCollection) -internet.SSLServer(int(config.main.ssl_port), +if config.main.ssl_port: + internet.SSLServer(int(config.main.ssl_port), server.Site(HTTPBackend()), SSLContext(config), ).setServiceParent(serviceCollection) @@ -41,11 +44,24 @@ debugHTTPServer = DebugHTTPServer() internet.TCPServer(8090, debugHTTPServer).setServiceParent(serviceCollection) # Start the DNS Server related services -TCPDNSServer = ProxyDNSServer() -internet.TCPServer(int(config.main.dns_tcp_port), TCPDNSServer).setServiceParent(serviceCollection) -UDPFactory = dns.DNSDatagramProtocol(TCPDNSServer) -internet.UDPServer(int(config.main.dns_udp_port), UDPFactory).setServiceParent(serviceCollection) - -# Start the ooni backend thing -daphn3 = Daphn3Server() -internet.TCPServer(int(config.main.daphn3_port), daphn3).setServiceParent(serviceCollection) +if config.main.dns_tcp_port: + TCPDNSServer = ProxyDNSServer() + internet.TCPServer(int(config.main.dns_tcp_port), + TCPDNSServer).setServiceParent(serviceCollection) + +if config.main.dns_udp_port: + UDPFactory = dns.DNSDatagramProtocol(TCPDNSServer) + internet.UDPServer(int(config.main.dns_udp_port), + UDPFactory).setServiceParent(serviceCollection) + +# Start the OONI daphn3 backend +if config.main.daphn3_port: + daphn3 = Daphn3Server() + internet.TCPServer(int(config.main.daphn3_port), + daphn3).setServiceParent(serviceCollection) + +if config.main.reporting_port: + internet.TCPServer(int(config.main.reporting_port), + reportingBackend).setServiceParent(serviceCollection) + + diff --git a/oonib/report/__init__.py b/oonib/report/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/oonib/report/api.py b/oonib/report/api.py index 04f70e0..ad0f62c 100644 --- a/oonib/report/api.py +++ b/oonib/report/api.py @@ -7,37 +7,61 @@ This is the async pcap reporting system. It requires the client to have created """ import random -import strings -from twisted.internet import reactor +import string +from twisted.internet import reactor, defer from cyclone import web + +from oonib.report.db import models backend_version = '0.0.1' def generateReportID(): size = 100 - id = ''.join(random.choice(strings.ascii_letters) for x in range(size)) - return id + report_id = ''.join(random.choice(string.ascii_letters) for x in range(size)) + return report_id +(a)defer.inlineCallbacks def newReport(software_name, software_version, test_name, test_version, progress, content): + + report_id = generateReportID() + + new_report = models.Report() + + new_report.report_id = unicode(report_id) + + new_report.software_name = unicode(software_name) + new_report.software_version = unicode(software_version) + new_report.test_name = unicode(test_name) + new_report.test_version = unicode(test_version) + new_report.progress = unicode(progress) + new_report.content = unicode(content) + print "Software Name: %s" % software_name print "Software Version: %s" % software_version print "Test Name: %s" % test_name print "Test Version: %s" % test_version print "Progress: %s" % progress print "Content: %s" % content - reportId = generateReportID() - return {'backend_version': backend_version, 'report_id': reportID} + + yield new_report.save() + + defer.returnValue({'backend_version': backend_version, 'report_id': + report_id}) def updateReport(report_id, content): print "Report ID: %s" % report_id print "Content: %s" % content - return {'backend_version': '0.1', 'report_id': 'FOOBAR'} -class DoReportHandler(web.RequestHandler): + return {'backend_version': backend_version, 'report_id': report_id} + +class NewReportHandler(web.RequestHandler): """ Responsible for creating and updating reports. """ - def post(self): + + @web.asynchronous + @defer.inlineCallbacks + def get(self): """ Creates a new report with the input @@ -51,6 +75,10 @@ class DoReportHandler(web.RequestHandler): 'content': 'XXX' } + Optional: + 'test_helper': 'XXX' + 'client_ip': 'XXX' + * Response {'backend_version': 'XXX', 'report_id': 'XXX'} @@ -61,15 +89,23 @@ class DoReportHandler(web.RequestHandler): 'test_name','test_version', 'progress', 'content'] report = {} - error = None for arg in arguments: if len(self.get_arguments(arg)) == 0: - print "No %s specified" % arg - error = arg - break + raise web.HTTPError(400, "%s not specified as argument of POST" + % arg) report[arg] = self.get_argument(arg) - if not error: - self.write(newReport(**report)) + + try: + test_helper = self.get_argument('test_helper') + + + except web.HTTPError: + pass + + new_report = yield newReport(**report) + + self.write(new_report) + self.finish() def put(self): """ @@ -88,11 +124,8 @@ class PCAPReportHandler(web.RequestHandler): def post(self): pass -handlers = [(r"/report/do", DoReportHandler), - (r"/report/pcap", PCAPReportHandler)] +spec = [(r"/report/new", NewReportHandler), + (r"/report/pcap", PCAPReportHandler)] -reporting = web.Application(handlers) -reactor.listenTCP(8888, reporting, interface="127.0.0.1") -print "starting bullshit" -reactor.run() +reportingBackend = web.Application(spec) diff --git a/oonib/report/db/__init__.py b/oonib/report/db/__init__.py index 1248c19..178abe9 100644 --- a/oonib/report/db/__init__.py +++ b/oonib/report/db/__init__.py @@ -6,28 +6,28 @@ from twisted.internet.defer import inlineCallbacks from storm.twisted.transact import Transactor, transact from storm.locals import * -dbtype = "sqlite" +from storm.uri import URI +from storm.databases.sqlite import SQLite -if dbtype == "sqlite": - from storm.uri import URI - from storm.databases.sqlite import SQLite - database = SQLite(URI('sqlite:///test.db')) +database = SQLite(URI('sqlite:///test.db')) threadpool = ThreadPool(0, 10) threadpool.start() transactor = Transactor(threadpool) +def getStore(): + store = Store(database) + return store + @inlineCallbacks def create_tables(): - def create(query): - store = Store(database) - store.execute(query) - store.commit() + from oonib.report.db import models for x in models.__all__: - query = getattr(models.__getattribute__(x), 'create_query') + query = getattr(models.__getattribute__(x), 'createQuery') try: yield transactor.run(create, query) except: log.msg("Failing in creating table for %s. Maybe it already exists?" % x) +create_tables() diff --git a/oonib/report/db/models.py b/oonib/report/db/models.py index b4e56d0..21e60eb 100644 --- a/oonib/report/db/models.py +++ b/oonib/report/db/models.py @@ -1,7 +1,26 @@ +__all__ = ['Report', 'TestHelperTMP'] from storm.twisted.transact import transact from storm.locals import * -class Report(object): +from oonib.report.db import getStore, transactor + +class OModel(object): + + transactor = transactor + + @transact + def create(query): + store = Store(database) + store.execute(query) + store.commit() + + @transact + def save(self): + store = getStore() + store.add(self) + store.commit() + +class Report(OModel): """ This represents an OONI Report as stored in the database. @@ -27,6 +46,14 @@ class Report(object): than 100 we should append to the YAML data structure that is currently stored in such field. """ + __storm_table__ = 'reports' + + createQuery = "CREATE TABLE " + __storm_table__ +\ + "(id INTEGER PRIMARY KEY, report_id VARCHAR, software_name VARCHAR,"\ + "software_version VARCHAR, test_name VARCHAR, test_version VARCHAR,"\ + "progress VARCHAR, content VARCHAR)" + + id = Int(primary=True) report_id = Unicode() @@ -39,3 +66,18 @@ class Report(object): content = Unicode() +class TestHelperTMP(OModel): + __storm_table__ = 'testhelpertmp' + + createQuery = "CREATE TABLE " + __storm_table__ +\ + "(id INTEGER PRIMARY KEY, report_id VARCHAR, test_helper VARCHAR,"\ + " client_ip VARCHAR, creation_time VARCHAR)" + + id = Int(primary=True) + + report_id = Unicode() + + test_helper = Unicode() + client_ip = Unicode() + + creation_time = Date()

1 0