May 2013 - tor-commits - lists.torproject.org

[tor/master] retroactively fix grammar
by arma＠torproject.org 18 May '13

18 May '13

commit 17090faf11f09025695596b02ffe2d5bf024a544 Author: Roger Dingledine <arma(a)torproject.org> Date: Sat May 18 17:50:34 2013 -0400 retroactively fix grammar --- ChangeLog | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/ChangeLog b/ChangeLog index 6a5ab8d..6528f7a 100644 --- a/ChangeLog +++ b/ChangeLog @@ -3696,7 +3696,7 @@ Changes in version 0.2.3.4-alpha - 2011-09-13 by an attacker who controls both an introduction point and a rendezvous point, and who uses the malleability of AES-CTR to alter the encrypted g^x portion of the INTRODUCE1 cell. We think - that these attacks is infeasible (requiring the attacker to send + that these attacks are infeasible (requiring the attacker to send on the order of zettabytes of altered cells in a short interval), but we'd rather block them off in case there are any classes of this attack that we missed. Reported by Willem Pinckaers.

1 0

[ooni-probe/master] Merge branch 'master' of ssh://git-rw.torproject.org/ooni-probe
by art＠torproject.org 18 May '13

18 May '13

commit 3f9035f84fc5d4e4ba59a018e49d1a1a98ba3ca8 Merge: ec46497 85a671b Author: Arturo Filastò <art(a)fuffa.org> Date: Sat May 18 19:13:30 2013 +0200 Merge branch 'master' of ssh://git-rw.torproject.org/ooni-probe * 'master' of ssh://git-rw.torproject.org/ooni-probe: Remove really old Makefile which seems to no longer point to valid directories. Remove oonib report archival script. Remove line to ignore oonib/oonibackend.conf from .gitignore. Remove log factory for oonib from ooni/utils/log.py. Remove old bin/oonib script. .gitignore | 1 - Makefile | 202 -------------------------------------------- bin/oonib | 36 -------- ooni/utils/log.py | 15 --- scripts/archive_reports.py | 174 -------------------------------------- 5 files changed, 0 insertions(+), 428 deletions(-)

1 0

[ooni-probe/master] Only process the input when we are dealing with URLs
by art＠torproject.org 18 May '13

18 May '13

commit ec464979088a4de69611010b68b9c59d9123bb05 Author: Arturo Filastò <art(a)fuffa.org> Date: Sat May 18 19:12:47 2013 +0200 Only process the input when we are dealing with URLs --- nettests/blocking/tcpconnect.py | 20 +++++++++++++------- 1 files changed, 13 insertions(+), 7 deletions(-) diff --git a/nettests/blocking/tcpconnect.py b/nettests/blocking/tcpconnect.py index 11c5558..5b432e0 100644 --- a/nettests/blocking/tcpconnect.py +++ b/nettests/blocking/tcpconnect.py @@ -48,16 +48,22 @@ class TCPConnectTest(nettest.NetTestCase): This inputProcessor extracts name:port pairs from urls XXX: Does not support unusual port numbers """ + def strip_url(address): + proto, path = x.strip().split('://') + proto = proto.lower() + host = path.split('/')[0] + if proto == 'http': + return "%s:80" % host + if proto == 'https': + return "%s:443" % host + if filename: fp = open(filename) for x in fp.readlines(): - proto, path = x.strip().split('://') - proto = proto.lower() - host = path.split('/')[0] - if proto == 'http': - yield "%s:80" % host - if proto == 'https': - yield "%s:443" % host + if x.startswith("http"): + yield strip_url(x) + else: + yield x.strip() fp.close() else: pass

1 0

[ooni-probe/master] Remove log factory for oonib from ooni/utils/log.py.
by isis＠torproject.org 18 May '13

18 May '13

commit b6412fb39a30e001d50d10ed555cd767e3dfa7b6 Author: Isis Lovecruft <isis(a)torproject.org> Date: Sat May 18 15:53:21 2013 +0000 Remove log factory for oonib from ooni/utils/log.py. --- ooni/utils/log.py | 15 --------------- 1 files changed, 0 insertions(+), 15 deletions(-) diff --git a/ooni/utils/log.py b/ooni/utils/log.py index 0740c10..8779fdb 100644 --- a/ooni/utils/log.py +++ b/ooni/utils/log.py @@ -64,18 +64,3 @@ def exception(error): else: exc_type, exc_value, exc_traceback = sys.exc_info() traceback.print_exception(exc_type, exc_value, exc_traceback) - -class LoggerFactory(object): - """ - This is a logger factory to be used by oonib - """ - def __init__(self, options): - pass - - def start(self, application): - # XXX parametrize this - start('oonib.log', "OONIB") - - def stop(self): - txlog.msg("Stopping OONIB") -

1 0

[ooni-probe/master] Remove line to ignore oonib/oonibackend.conf from .gitignore.
by isis＠torproject.org 18 May '13

18 May '13

commit 05a66ddbf8a6f55774cf2c218d0665ab0c62c424 Author: Isis Lovecruft <isis(a)torproject.org> Date: Sat May 18 15:54:52 2013 +0000 Remove line to ignore oonib/oonibackend.conf from .gitignore. --- .gitignore | 1 - 1 files changed, 0 insertions(+), 1 deletions(-) diff --git a/.gitignore b/.gitignore index 4523a7d..ec5a4a0 100644 --- a/.gitignore +++ b/.gitignore @@ -12,7 +12,6 @@ ooni/assets/bridge*.txt ooni/lib/txtorcon ooni/plugins/dropin.cache ooni/report*.yaml -oonib/oonibackend.conf fluff/ pcaps/ report*.yaml

1 0

[ooni-probe/master] Remove oonib report archival script.
by isis＠torproject.org 18 May '13

18 May '13

commit b56ab41246a832b7cc5cc4bddc5eaa8f4c83c2cf Author: Isis Lovecruft <isis(a)torproject.org> Date: Sat May 18 15:58:49 2013 +0000 Remove oonib report archival script. --- scripts/archive_reports.py | 174 -------------------------------------------- 1 files changed, 0 insertions(+), 174 deletions(-) diff --git a/scripts/archive_reports.py b/scripts/archive_reports.py deleted file mode 100755 index c49bd7a..0000000 --- a/scripts/archive_reports.py +++ /dev/null @@ -1,174 +0,0 @@ -#!/usr/bin/env python -import yaml -import sys -import glob -import fcntl -import os -import re -from ipaddr import IPAddress -from datetime import timedelta -from datetime import datetime -from ooni.otime import fromTimestamp, timestamp -from ooni.otime import InvalidTimestampFormat, utcDateNow -from ooni.utils import log - -############################################################################### -# You can set some config options here # -############################################################################### -report_age = 1 # hours -report_archive_dir = '/home/user/oonib/reports/archived' -report_source_dir = '/home/user/oonib/reports' -valid_test_versions = ['0.1', '0.1.1', '0.4', '0.1.3'] -default_probe_cc = '??' -target_permission = 0444 -path_permission = 0755 -retry_attempts = 100 -############################################################################### - -now = utcDateNow() -delta = timedelta(hours=report_age) - -def filter_reports_by_age(report): - try: - ts,__,__ = os.path.basename(report).split('_') - if now - fromTimestamp(ts) > delta: - return True - except (InvalidTimestampFormat, ValueError): - return False - -class InvalidReportField(Exception): - pass - -def validate_fields(fields): - log.debug("Report fields are: %s" % fields) - - # check report version - if fields['test_version'] not in valid_test_versions: - raise InvalidReportField('test_version') - - # check report CC - #XXX: confirm what value we use for default CC and whether - # or not we should support > 2 character CC - if fields['probe_cc'] is None: - fields['probe_cc'] = default_probe_cc - if not re.match('[A-Z\?]{2,4}', fields['probe_cc'].upper()): - raise InvalidReportField('probe_cc') - - # check report ASN - if fields['probe_asn'] is None: - fields['probe_asn'] = 'AS0' - if not re.match('^AS[0-9]{1,10}', fields['probe_asn'].upper()): - raise InvalidReportField('probe_asn') - - # check report timestamp - try: - datetime_ts = datetime.fromtimestamp(fields['start_time']) - datetime_str = timestamp(datetime_ts) - except InvalidTimestampFormat: - raise InvalidReportField('start_time') - - # check report IP - try: - IPAddress(fields['probe_ip']) - except ValueError: - raise InvalidReportField('probe_ip') - - # all looks good! - -def get_report_header_fields(report_header): - required_fields = ['probe_asn', 'probe_cc', 'probe_ip', 'start_time', - 'test_name', 'test_version'] - try: - return dict([(k,report_header[k]) for k in required_fields ]) - except KeyError: - return None - -def get_test_name(fields): - test_name = fields['test_name'].lower().replace(' ', '_') - return test_name - -def get_target_or_fail(fields, report): - # set the target filename - reportFormatVersion = fields['test_version'] - CC = fields['probe_cc'] - # XXX: wouldn't hurt to check timestamp for sanity again? - dateInISO8601Format,__,__ = os.path.basename(report).split('_') - probeASNumber = fields['probe_asn'] - testName = get_test_name(fields) - - # make sure path reportFormatVersion/CC exists - path = os.path.abspath(report_archive_dir) - for component in [reportFormatVersion, CC]: - path = os.path.join(path, component) - if not os.path.isdir(path): - try: - os.mkdir(path, path_permission) - log.debug("mkdir path: %s" % path) - except OSError: - return None - - # if the target file already exists, try to find another filename - filename = "%s-%s-%s.yamloo" % (testName, dateInISO8601Format, probeASNumber) - target = os.path.join(path, filename) - - # try to get a unique filename. os.open as used below requires - # that the file not already exist - naming_attempts = 1 - while os.path.exists(target) and naming_attempts < retry_attempts: - filename = "%s-%s-%s.%d.yamloo" % (testName, dateInISO8601Format, - probeASNumber, naming_attempts) - target = os.path.join(path, filename) - naming_attempts = naming_attempts + 1 - - if naming_attempts >= retry_attempts: - log.err("Failed getting unique filename %d times; skipping" % i) - return None - return target - -# grab list of reports -reports = glob.glob(report_source_dir+'/*') -reports_to_archive = filter(filter_reports_by_age, reports) - -# iterate over the reports to archive -for report in reports_to_archive: - log.debug("Parsing report: %s" % report) - try: - #XXX: verify that os.fdopen works as expected - f = os.fdopen(os.open(report, os.O_RDONLY|os.O_EXCL|os.O_NONBLOCK)) - except IOError: - log.err("Unable to get exclusive lock on %s; skipping" % report) - continue - - # parse the header and validate it - yamloo = yaml.safe_load_all(f) - report_header = yamloo.next() - fields = get_report_header_fields(report_header) - try: - validate_fields(fields) - except InvalidReportField, field_name: - log.err("Report %s contains invalid field called %s" % (report, field_name)) - continue - except: - log.err("An unhandled error occurred while processing %s" % report) - continue - - # get a target filename or fail - target = get_target_or_fail(fields, report) - if not target: - continue - - log.debug("target: %s" % target) - - try: - #XXX: My system does not have os.O_EXLOCK. Verify this works as is. - g = os.fdopen(os.open(target, os.O_CREAT|os.O_EXCL|os.O_NONBLOCK)) - - os.rename(report, target) - os.chmod(target, target_permission) - f.close() - g.close() - - except IOError: - # unable to lock the file... still held open? - log.err("Failed to lock target file. Possible race condition!") - continue

1 0

[ooni-probe/master] Merge remote-tracking branch 'tpo-common/master'
by isis＠torproject.org 18 May '13

18 May '13

commit e6e679d10c90be096243b9bf5cc572ce1fe1068d Merge: 80c99cb 59d6fad Author: Isis Lovecruft <isis(a)torproject.org> Date: Sat May 18 15:47:11 2013 +0000 Merge remote-tracking branch 'tpo-common/master' nettests/blocking/tcpconnect.py | 18 ++++++++++++++++++ nettests/manipulation/http_host.py | 10 ++++++++++ ooni/config.py | 23 ++++------------------- ooni/director.py | 23 ++++++++++++++--------- ooni/geoip.py | 2 +- ooni/nettest.py | 5 ++--- ooni/otime.py | 2 ++ 7 files changed, 51 insertions(+), 32 deletions(-)

1 0

[ooni-probe/master] Remove old bin/oonib script.
by isis＠torproject.org 18 May '13

18 May '13

commit 9a32c80096c3e6cf2acf8498dfc5d5b913b1b1a6 Author: Isis Lovecruft <isis(a)torproject.org> Date: Sat May 18 15:52:50 2013 +0000 Remove old bin/oonib script. --- bin/oonib | 36 ------------------------------------ 1 files changed, 0 insertions(+), 36 deletions(-) diff --git a/bin/oonib b/bin/oonib deleted file mode 100755 index 79885ba..0000000 --- a/bin/oonib +++ /dev/null @@ -1,36 +0,0 @@ -#!/usr/bin/env python - -import sys -import os -# Avoid the export PYTHONPATH insanity -sys.path[:] = map(os.path.abspath, sys.path) -this_directory = os.path.dirname(__file__) -root = os.path.abspath(os.path.join(this_directory, '..')) -sys.path.insert(0, root) - -backend_script = os.path.join(root, 'oonib', 'oonibackend.py') - -from twisted.python import log, usage -from twisted.internet import reactor -from twisted.application import app - -from oonib import runner -from oonib.oonibackend import application - -sys.argv[1:] = ['-ny', backend_script] - -# Uncomment this line to daemonize -#sys.argv[1:] = ['-y', backend_script] - -def runApp(config): - runner.OBaseRunner(config).run() - -config = runner.ServerOptions() -try: - config.parseOptions() -except usage.error, ue: - print config - print "%s: %s" % (sys.argv[0], ue) -else: - runApp(config) -

1 0

[ooni-probe/master] Remove really old Makefile which seems to no longer point to valid directories.
by isis＠torproject.org 18 May '13

18 May '13

commit 85a671b5eb4c8ecae6ec68811d11999fb76b414d Author: Isis Lovecruft <isis(a)torproject.org> Date: Sat May 18 16:01:59 2013 +0000 Remove really old Makefile which seems to no longer point to valid directories. --- Makefile | 202 -------------------------------------------------------------- 1 files changed, 0 insertions(+), 202 deletions(-) diff --git a/Makefile b/Makefile deleted file mode 100644 index 13d277e..0000000 --- a/Makefile +++ /dev/null @@ -1,202 +0,0 @@ -# -# .:| Open Observatory of Network Interference |:. -# -# /lib/Makefile -# ------------- -# For obtaining, building, and configuring dependencies for OONI installations. -# -# @authors: Isis Lovecruft, Arturo Filastó -# @version: 0.1.0-alpha -# @license: see included LICENSE file -# @copyright: 2012 Isis Lovecruft, Arturo Filastó -# -# XXX TODO eventually this should be converted into a distutils setup.py -# - -here_we_are = . -top_srcdir = ${here_we_are} - -srcdir_ooni = ${top_srcdir}/ooni -srcdir_ooni_kit = ${srcdir_ooni}/kit -srcdir_ooni_plugoo = ${srcdir_ooni}/plugoo -srcdir_ooni_protocols = ${srcdir_ooni}/protocols -srcdir_ooni_templates = ${srcdir_ooni}/templates - -srcdir_oonib = ${top_srcdir}/oonib -srcdir_oonib_lib = ${srcdir_oonib}/lib -srcdir_oonib_report = ${srcdir_oonib}/report -srcdir_oonib_report_db = ${srcdir_oonib_report}/db -srcdir_oonib_testhelpers = ${srcdir_oonib}/testhelpers - -nettests = ${top_srcdir}/nettests -nettests_core = ${nettests}/core -nettests_examples = ${nettests}/examples -nettests_experimental = ${nettests}/experimental -nettests_third_party = ${nettests}/third_party - -prefix = ${top_srcdir} -exec_prefix = ${srcdir_ooni} - -bindir = ${top_srcdir}/bin -docsdir = ${top_srcdir}/docs -libdir = ${srcdir_ooni}/lib -includedir = ${top_srcdir}/include -testsdir = ${top_srcdir}/tests - -#pkgtwisteddir = $(libdir)/Twisted-* -pkgscapydir = $(libdir)/scapy -pkgtxtorcondir = $(libdir)/txtorcon - -SUBDIRS = ${top_srcdir} \ - ${srcdir_ooni} \ - ${srcdir_ooni_kit} \ - ${srcdir_ooni_plugoo} \ - ${srcdir_ooni_protocols} \ - ${srcdir_ooni_templates} \ - ${srcdir_oonib} \ - ${srcdir_oonib_lib} \ - ${srcdir_oonib_report} \ - ${srcdir_oonib_report_db} \ - ${srcdir_oonib_testhelpers} \ - ${nettests} \ - ${nettests_core} \ - ${nettests_examples} \ - ${nettests_experimental} \ - ${nettests_third_party} \ - ${pkgscapydir} \ - ${pkgtxtorcondir} - -all: txtorcon - -dependencies: txtorcon non-pip - -non-pip: scapy-all - -clean: - rm -rf ${libdir}/txtorcon - -clean-all: - rm -rf ${libdir}/txtorcon - rm -rf ${libdir}/txtraceroute.py - rm -rf ${libdir}/txscapy.py - rm -rf ${libdir}/rfc3339.py - -clean-scapy-all: - rm -rf ${libdir}/scapy* - rm -rf ${libdir}/pypcap* - rm -rf ${libdir}/libdnet* - -clean-non-pip: clean-scapy-all - -clean-dependencies: - rm -rf ${libdir}/scapy* - rm -rf ${libdir}/pypcap* - rm -rf ${libdir}/libdnet* - rm -rf ${libdir}/txtorcon - -txtraceroute: - echo "Processing dependency txtraceroute..." - cd ${libdir} && \ - git clone https://github.com/hellais/txtraceroute.git txtraceroute.git && \ - mv txtraceroute.git/txtraceroute.py txtraceroute.py && \ - rm -rf txtraceroute.git && \ - cd ${top_srcdir} - -txtorcon: - echo "Processing dependency txtorcon..." - cd ${libdir} && \ - git clone https://github.com/meejah/txtorcon.git txtorcon.git && \ - mv txtorcon.git/txtorcon txtorcon && \ - rm -rf txtorcon.git && \ - cd ${top_srcdir} - -txscapy: - echo "Processing dependency txscapy" - cd ${libdir} && \ - git clone https://github.com/hellais/txscapy.git txscapy.git && \ - mv txscapy.git/txscapy.py txscapy.py && \ - rm -rf txscapy.git && \ - cd ${top_srcdir} - -rfc3339: - echo "Processing RFC3339 dependency" - cd ${libdir} && \ - hg clone https://bitbucket.org/henry/rfc3339 rfc3339 && \ - mv rfc3339/rfc3339.py rfc3339.py && \ - rm -rf rfc3339 \ - cd ${top_srcdir} - -## requires sudo for setup.py: -scapy: - echo "Processing dependency scapy..." - cd ${libdir} ; \ - wget --ca-certificate="secdev.org.pem" -O scapy.tar.gz https://www.secdev.org/projects/scapy/files/scapy-latest.tar.gz && \ - tar -xzf scapy.tar.gz && \ - cd scapy-* && sudo python setup.py install && \ - cd .. && rm scapy.tar.gz && cd ${top_srcdir} - -## requires sudo for setup.py -pypcap: - echo "Processing dependency pypcap..." - cd ${libdir} ; \ - wget -O pypcap.tar.gz https://pypcap.googlecode.com/files/pypcap-1.1.tar.gz && \ - tar -xzf pypcap.tar.gz && cd pypcap-* && \ - sudo python setup.py install && \ - cd .. && rm pypcap.tar.gz && \ - cd ${top_srcdir} -# make all && make build && make install && make test && \ - make cleandir distclean && cd ${libdir} && \ - rm -rf pypcap* - -libdnet: - echo "Processing dependency libdnet..." - cd ${libdir} ; \ - wget -O libdnet.tar.gz https://libdnet.googlecode.com/files/libdnet-1.12.tgz && \ - tar -xzf libdnet.tar.gz && cd libdnet-* && \ - configure && make && make install && make clean && \ - cd .. && rm -rf libdnet* && cd ${top_srcdir} - -scapy-all: scapy pypcap libdnet - -ETAGS = etags -ETAGSFLAGS = -ETAGSARGS = -## or change to ${SUBDIRS} -TAGME = ${topsrc_dir} - -tags-recursive: - list='$(TAGME)'; for subdir in $$list; do \ - test "$$subdir" = . || (cd $$subdir && make tags); \ - done - -tags: TAGS - -ID: $(HEADERS) $(SOURCES) $(LISP) $(TAGS_FILES) - list='$(SOURCES) $(HEADERS) $(LISP) $(TAGS_FILES)'; \ - unique=`for i in $$list; do \ - if test -f "$$i"; then echo $$i; fi; \ - done | \ - $(AWK) ' { files[$$0] = 1; } \ - END { for (i in files) print i; }'`; \ - mkid -fID $$unique - -TAGS: tags-recursive $(HEADERS) $(SOURCES) $(TAGS_DEPENDENCIES) \ - $(TAGS_FILES) $(LISP) - tags=; \ - here=`pwd`; \ - echo ${E_OR_V}; \ - list='$(TAGME)'; for subdir in $$list; do \ - if test "$$subdir" = .; then :; else \ - test -f $$subdir/TAGS && tags="$$tags -i $$here/$$subdir/TAGS"; \ - fi; \ - done; \ - list='$(SOURCES) $(HEADERS) $(LISP) $(TAGS_FILES)'; \ - unique=`for i in $$list; do \ - if test -f "$$i"; then echo $$i; fi; \ - done;` - test -z "$(ETAGS_ARGS)$$tags$$unique" \ - || $(ETAGS) $(ETAGSFLAGS) $(ETAGS_ARGS) $$tags $$unique - -awkward: - | $(AWK) ' { files[$$0] = 1; } \ - END { for (i in files) print i; }'`; \

1 0

[tor/master] Fix bug 8846: better log message on IP version confusion
by nickm＠torproject.org 17 May '13

17 May '13

commit bc56918e5abb48f67f31012838c689e9572eec69 Author: Nick Mathewson <nickm(a)torproject.org> Date: Fri May 17 14:50:45 2013 -0400 Fix bug 8846: better log message on IP version confusion --- changes/bug8846 | 4 ++++ src/or/connection_edge.c | 5 +++-- 2 files changed, 7 insertions(+), 2 deletions(-) diff --git a/changes/bug8846 b/changes/bug8846 new file mode 100644 index 0000000..377cc37 --- /dev/null +++ b/changes/bug8846 @@ -0,0 +1,4 @@ + o Minor bugfixes: + - Give a less useless error message when the user asks for an IPv4 + address on an IPv6-only port, or vice versa. Fixes bug 8846; bugfix + on 0.2.4.7-alpha. diff --git a/src/or/connection_edge.c b/src/or/connection_edge.c index 926fcab..4d317d0 100644 --- a/src/or/connection_edge.c +++ b/src/or/connection_edge.c @@ -1769,9 +1769,10 @@ connection_ap_get_begincell_flags(entry_connection_t *ap_conn) } if (flags == BEGIN_FLAG_IPV4_NOT_OK) { - log_warn(LD_BUG, "Hey; I'm about to ask a node for a connection that I " + log_warn(LD_EDGE, "I'm about to ask a node for a connection that I " "am telling it to fulfil with neither IPv4 nor IPv6. That's " - "probably not going to work."); + "not going to work. Did you perhaps ask for an IPv6 address " + "on an IPv4Only port, or vice versa?"); } return flags;

1 0