June 2015 - tor-commits - lists.torproject.org

[torflow/master] Add section to README.BwAuthorities with pip/peep instructions.
by mikeperry＠torproject.org 01 Jun '15

01 Jun '15

commit 412fd5c95d5552c26c489898603d32679c6b7807 Author: Isis Lovecruft <isis(a)torproject.org> Date: Wed May 20 06:58:13 2015 +0000 Add section to README.BwAuthorities with pip/peep instructions. --- NetworkScanners/BwAuthority/README.BwAuthorities | 38 ++++++++++++++++++++-- 1 file changed, 35 insertions(+), 3 deletions(-) diff --git a/NetworkScanners/BwAuthority/README.BwAuthorities b/NetworkScanners/BwAuthority/README.BwAuthorities index 576fead..dec3ed2 100644 --- a/NetworkScanners/BwAuthority/README.BwAuthorities +++ b/NetworkScanners/BwAuthority/README.BwAuthorities @@ -38,7 +38,41 @@ No special configure script options are needed, but again, both need to be running the master branch from tor git. -4. Download SQLAlchemy 0.7.x-latest and Elixir 0.7.x-latest +4. Install Dependencies + + +4.1. Dependencies from your distribution's package manager + +In Debian-based systems, the following packages are required: + + $ sudo apt-get install python2.6 libpython2.6-dev libsqlite3-dev + +If you want to use postgres support, you should also install python-psycopg2. + + +4.2. Python Dependencies + + +4.2.1. Using Pip or Peep: + +First, ensure that you've got a *recent* version of pip. If you already have +pip, do: + + $ pip install --upgrade pip + +Next, if you'd like to verify the correctness of the downloaded dependencies +with SHA-256 (rather than MD5, which is pip's default), do: + + $ pip install peep + +Finally, do: + + $ peep install -r .../NetworkScanners/BwAuthority/requirements.txt + +(Or, if you didn't install peep, do `$ pip install -r requirements.txt`.) + + +4.2.2. The Tedious Way The latest version of SQLAlchemy is 0.7.2 and the latest version of Elixir is 0.7.1 at the time of writing. While TorFlow is written to be compatible @@ -68,8 +102,6 @@ Elixir-0.7.1.tar.gz SQLAlchemy-0.7.2.tar.gz torflow-trunk Both these libraries also depend upon python-pysqlite2, which should be a package for your distribution (you want 2.3.x for SQLite 3.x). -If you want to use postgres support, you should also install python-psycopg2. - 5. Enable voting on bandwidths in your authority torrc

1 0

[torflow/master] Update paths
by mikeperry＠torproject.org 01 Jun '15

01 Jun '15

commit 9fd77676bfd87c56038e8eeb773b5347b9ce468b Author: Tom Ritter <tom(a)ritter.vg> Date: Thu May 21 12:59:09 2015 -0500 Update paths TorCtl is located in a different location if you actually look at the README and the shell script for the submodule. Tor is cloned without the '.git' by default, unless you name it that way, which is confusing. Dependencies have been updated (kinda.) --- NetworkScanners/BwAuthority/aggregate.py | 2 +- NetworkScanners/BwAuthority/bwauthority.py | 2 +- NetworkScanners/BwAuthority/run_scan.sh | 4 ++-- 3 files changed, 4 insertions(+), 4 deletions(-) diff --git a/NetworkScanners/BwAuthority/aggregate.py b/NetworkScanners/BwAuthority/aggregate.py index cbd6657..49ad3cc 100755 --- a/NetworkScanners/BwAuthority/aggregate.py +++ b/NetworkScanners/BwAuthority/aggregate.py @@ -58,7 +58,7 @@ MAX_SCAN_AGE = 60*60*24*1.5 # path to git repos (.git) PATH_TO_TORFLOW_REPO = '../../.git/' -PATH_TO_TORCTL_REPO = '../../TorCtl/.git/' +PATH_TO_TORCTL_REPO = '../../.git/modules/TorCtl/' def base10_round(bw_val): # This keeps the first 3 decimal digits of the bw value only diff --git a/NetworkScanners/BwAuthority/bwauthority.py b/NetworkScanners/BwAuthority/bwauthority.py index 4bab6c1..e190e7d 100755 --- a/NetworkScanners/BwAuthority/bwauthority.py +++ b/NetworkScanners/BwAuthority/bwauthority.py @@ -15,7 +15,7 @@ STOP_PCT_REACHED = 9 # path to git repos (.git) PATH_TO_TORFLOW_REPO = '../../.git/' -PATH_TO_TORCTL_REPO = '../../TorCtl/.git/' +PATH_TO_TORCTL_REPO = '../../.git/modules/TorCtl/' def main(argv): (branch, head) = get_git_version(PATH_TO_TORFLOW_REPO) diff --git a/NetworkScanners/BwAuthority/run_scan.sh b/NetworkScanners/BwAuthority/run_scan.sh index 7853647..6406f64 100755 --- a/NetworkScanners/BwAuthority/run_scan.sh +++ b/NetworkScanners/BwAuthority/run_scan.sh @@ -5,8 +5,8 @@ SCANNER_COUNT=5 # This tor must have the w status line fix as well as the stream bw fix # Ie git master or 0.2.2.x -TOR_EXE=../../../tor.git/src/or/tor -PYTHONPATH=../../../SQLAlchemy-0.7.3/lib:../../../Elixir-0.7.1/ +TOR_EXE=../../../tor/src/or/tor +PYTHONPATH=../../../SQLAlchemy-0.7.10/lib:../../../Elixir-0.7.1/ ! [ -e "./local.cfg" ] || . "./local.cfg"

1 0

[torflow/master] Use tjr's bwfile size values as the initial defaults.
by mikeperry＠torproject.org 01 Jun '15

01 Jun '15

commit 2c5faf72f7443cd68a68e490645c2449b611283c Author: Mike Perry <mikeperry-git(a)torproject.org> Date: Fri May 22 14:29:50 2015 -0700 Use tjr's bwfile size values as the initial defaults. --- NetworkScanners/BwAuthority/data/bwfiles | 15 +++++++++------ 1 file changed, 9 insertions(+), 6 deletions(-) diff --git a/NetworkScanners/BwAuthority/data/bwfiles b/NetworkScanners/BwAuthority/data/bwfiles index 7ad6b1a..267fdb8 100644 --- a/NetworkScanners/BwAuthority/data/bwfiles +++ b/NetworkScanners/BwAuthority/data/bwfiles @@ -1,9 +1,12 @@ -1 4M -7 2M -23 1M -53 512k -82 256k -95 128k +1 32M +7 16M +20 8M +37 4M +56 2M +72 1M +86 512k +93 256k +98 128k 99 64k 100 32k .

1 0

[torflow/master] Update TorCtl submodule.
by mikeperry＠torproject.org 01 Jun '15

01 Jun '15

commit 8d091d3660953f38c69954731b49d2e4a9523c78 Author: Mike Perry <mikeperry-git(a)torproject.org> Date: Sun May 31 22:35:32 2015 -0700 Update TorCtl submodule. --- TorCtl | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/TorCtl b/TorCtl index 4fdd203..c8fcb25 160000 --- a/TorCtl +++ b/TorCtl @@ -1 +1 @@ -Subproject commit 4fdd2031e6b231ed4bbaa79940f67e9b8f691382 +Subproject commit c8fcb25b079d52a20cafc7f7adf178e90ab76338

1 0

[torflow/master] We now support python2.6 or python2.7.
by mikeperry＠torproject.org 01 Jun '15

01 Jun '15

commit 9a9ed6921385870399dd9de460d649df71606156 Author: Mike Perry <mikeperry-git(a)torproject.org> Date: Tue May 26 15:42:29 2015 -0700 We now support python2.6 or python2.7. Adjust setup accordingly. Also bump other version deps. --- NetworkScanners/BwAuthority/README.BwAuthorities | 7 ++---- NetworkScanners/BwAuthority/cron.sh | 1 - NetworkScanners/BwAuthority/install-debs.sh | 27 ---------------------- NetworkScanners/BwAuthority/requirements.txt | 8 +++---- NetworkScanners/BwAuthority/run_scan.sh | 2 +- NetworkScanners/BwAuthority/setup.sh | 15 +++++++----- 6 files changed, 16 insertions(+), 44 deletions(-) diff --git a/NetworkScanners/BwAuthority/README.BwAuthorities b/NetworkScanners/BwAuthority/README.BwAuthorities index fcf1f54..7e055b4 100644 --- a/NetworkScanners/BwAuthority/README.BwAuthorities +++ b/NetworkScanners/BwAuthority/README.BwAuthorities @@ -41,12 +41,9 @@ prepare a virtualenv, or run through the setup manually. 2.1. Scripted virtualenv setup The easiest and most reliable setup method is to use the setup.sh script -to install a python 2.6 virtual environment. This script will download all +to install a python 2.6/2.7 virtual environment. This script will download all of the dependencies and install them for you, but it will require that you -have a copy of python2.6 installed and in your path. - -There is also a install-debs.sh script for Debian and Ubuntu systems that will -handle python2.6 and some additional package dependency installation for you. +have a copy of python2.6 or python2.7 installed and in your path. 2.2. Manual setup diff --git a/NetworkScanners/BwAuthority/cron.sh b/NetworkScanners/BwAuthority/cron.sh index d304031..26d4164 100755 --- a/NetworkScanners/BwAuthority/cron.sh +++ b/NetworkScanners/BwAuthority/cron.sh @@ -10,7 +10,6 @@ OUTPUT=$SCANNER_DIR/bwscan.V3BandwidthsFile cd $SCANNER_DIR # Needed for import to work properly. if [ -f bwauthenv/bin/activate ] then - echo "Using virtualenv..." . bwauthenv/bin/activate fi $SCANNER_DIR/aggregate.py $SCANNER_DIR/data $OUTPUT diff --git a/NetworkScanners/BwAuthority/install-debs.sh b/NetworkScanners/BwAuthority/install-debs.sh deleted file mode 100755 index 8357d0b..0000000 --- a/NetworkScanners/BwAuthority/install-debs.sh +++ /dev/null @@ -1,27 +0,0 @@ -#!/bin/bash - -if [ ! $(dpkg -s python2.6 python2.6-dev 2>/dev/null >/dev/null) ] -then - echo "We need python2.6 to be in the path. Press enter to try to install it." - echo "or control-c and find your own way to install it and re-run this script" - echo - echo -n "Hit enter to install python2.6: " - read - sudo apt-get install python2.6 python2.6-dev - if [ $? -ne 0 ] - then - echo - echo "Your distribution does not natively provide python2.6." - echo "Press enter to try to install from a ppa, or control-c to install on your own" - echo - echo -n "Hit enter to install from ppa:fkrull/deadsnakes: " - read - sudo apt-get install software-properties-common - sudo add-apt-repository ppa:fkrull/deadsnakes - sudo apt-get update - sudo apt-get install python2.6 python2.6-dev - fi -fi - -sudo apt-get install libsqlite3-dev python-virtualenv -sudo apt-get install autoconf2.13 automake make libevent-dev diff --git a/NetworkScanners/BwAuthority/requirements.txt b/NetworkScanners/BwAuthority/requirements.txt index 1cfee33..36999c4 100644 --- a/NetworkScanners/BwAuthority/requirements.txt +++ b/NetworkScanners/BwAuthority/requirements.txt @@ -19,11 +19,11 @@ # # (Or, if you didn't install peep, do `$ pip install -r requirements.txt`.) -# sha256: NUKQvRsYu2nhGUay2RP9mxXCpKp94uhWXSmUktt3NyE -pysqlite==2.6.0 +# sha256: _pw1IWv1bIWLNMS0yL5-NFZt3vKWcOWltD-cuOz7so0 +pysqlite==2.6.3 -# sha256: tYgFyp-PUSPmCfJTqRboeh4-rERqCq4Sjt5jBIXI9AY -SQLAlchemy==0.7.2 +# sha256: d6o51lydBD66a6Mps1n_hnQk_WxAO3wMsRK2XlB-HWY +SQLAlchemy==0.7.10 # sha256: p-9DfyW1ROT3T7Mjb8Q80l9db-tgN918ZpMQRtdUOek Elixir==0.7.1 diff --git a/NetworkScanners/BwAuthority/run_scan.sh b/NetworkScanners/BwAuthority/run_scan.sh index 6406f64..d2674ec 100755 --- a/NetworkScanners/BwAuthority/run_scan.sh +++ b/NetworkScanners/BwAuthority/run_scan.sh @@ -51,7 +51,7 @@ fi if [ -f bwauthenv/bin/activate ] then - echo "Using virtualenv..." + echo "Using virtualenv in bwauthenv..." . bwauthenv/bin/activate fi diff --git a/NetworkScanners/BwAuthority/setup.sh b/NetworkScanners/BwAuthority/setup.sh index 206f4a2..15ad931 100755 --- a/NetworkScanners/BwAuthority/setup.sh +++ b/NetworkScanners/BwAuthority/setup.sh @@ -3,18 +3,21 @@ SCANNER_DIR=$(dirname "$0") SCANNER_DIR=$(readlink -f "$SCANNER_DIR") -# 1. Install python2.6 if needed -if [ -z "$(which python2.6)" ] +PYTHON=$(which python2.6 || which python2.7) + +# 1. Install python if needed +if [ -z "$(which $PYTHON)" ] then - echo "We need python2.6 to be in the path." - echo "If you are on a Debian or Ubuntu system, you can try ./install-debs.sh" + echo "We need python2.6 or 2.7 to be in the path." + echo "If you are on a Debian or Ubuntu system, you can try: " + echo " sudo apt-get install python2.7 python2.7-dev libpython2.7-dev libsqlite3-dev python-virtualenv autoconf2.13 automake make libevent-dev" exit 1 fi if [ -z "$(which virtualenv)" ] then echo "We need virtualenv to be in the path. If you are on a debian system, try:" - echo " sudo apt-get install libsqlite3-dev python-virtualenv" + echo " sudo apt-get install python-dev libsqlite3-dev python-virtualenv autoconf2.13 automake make libevent-dev" exit 1 fi @@ -39,7 +42,7 @@ fi # 4. Initialize virtualenv if [ ! -f bwauthenv/bin/activate ] then - virtualenv -p python2.6 bwauthenv + virtualenv -p $PYTHON bwauthenv fi source bwauthenv/bin/activate

1 0

[torflow/master] Ignore SSL Cert Errors
by mikeperry＠torproject.org 01 Jun '15

01 Jun '15

commit a26bf21bb8fe417a708bccf62f3f2de3a5267f82 Author: Tom Ritter <tom(a)ritter.vg> Date: Thu May 21 13:00:41 2015 -0500 Ignore SSL Cert Errors A point release in python turned on cert validation (yay). This broke the script (boo). Ignore cert validation, as the only point of SSL here is to prevent caching, not confidentiality. --- NetworkScanners/BwAuthority/bwauthority_child.py | 17 +++++++++++++---- 1 file changed, 13 insertions(+), 4 deletions(-) diff --git a/NetworkScanners/BwAuthority/bwauthority_child.py b/NetworkScanners/BwAuthority/bwauthority_child.py index 4b4ac6f..13d4080 100755 --- a/NetworkScanners/BwAuthority/bwauthority_child.py +++ b/NetworkScanners/BwAuthority/bwauthority_child.py @@ -23,6 +23,7 @@ import ConfigParser import sqlalchemy import sets import re +import ssl import random sys.path.append("../../") @@ -134,23 +135,31 @@ def http_request(address): ''' perform an http GET-request and return 1 for success or 0 for failure ''' request = urllib2.Request(address) + try: + context = ssl._create_unverified_context() + except: + context = None request.add_header('User-Agent', user_agent) try: - reply = urllib2.urlopen(request) + if context: + reply = urllib2.urlopen(request, context=context) + else: + reply = urllib2.urlopen(request) decl_length = reply.info().get("Content-Length") read_len = len(reply.read()) plog("DEBUG", "Read: "+str(read_len)+" of declared "+str(decl_length)) return 1 - except (ValueError, urllib2.URLError): + except (ValueError, urllib2.URLError) as e: plog('ERROR', 'The http-request address ' + address + ' is malformed') + plog('ERROR', str(e)) return 0 - except (IndexError, TypeError): + except (IndexError, TypeError) as e: plog('ERROR', 'An error occured while negotiating socks5 with Tor') return 0 except KeyboardInterrupt: raise KeyboardInterrupt - except socks.Socks5Error, e: + except socks.Socks5Error as e: if e.value[0] == 6: plog("NOTICE", "Tor timed out our SOCKS stream request.") else:

1 0

[torflow/master] Add a 5th scanner for Unmeasured=1 nodes.
by mikeperry＠torproject.org 01 Jun '15

01 Jun '15

commit 4f272b1bf606df6e9788ad5bc60dbe3f9027eb23 Author: Mike Perry <mikeperry-git(a)torproject.org> Date: Thu May 21 18:18:55 2015 -0700 Add a 5th scanner for Unmeasured=1 nodes. --- NetworkScanners/BwAuthority/bwauthority_child.py | 36 ++++++++++++++++---- .../BwAuthority/data/scanner.1/bwauthority.cfg | 2 ++ .../BwAuthority/data/scanner.2/bwauthority.cfg | 2 ++ .../BwAuthority/data/scanner.3/bwauthority.cfg | 2 ++ .../BwAuthority/data/scanner.4/bwauthority.cfg | 2 ++ .../BwAuthority/data/scanner.5/bwauthority.cfg | 30 ++++++++++++++++ NetworkScanners/BwAuthority/run_scan.sh | 2 +- 7 files changed, 68 insertions(+), 8 deletions(-) diff --git a/NetworkScanners/BwAuthority/bwauthority_child.py b/NetworkScanners/BwAuthority/bwauthority_child.py index 2584991..4bac13a 100755 --- a/NetworkScanners/BwAuthority/bwauthority_child.py +++ b/NetworkScanners/BwAuthority/bwauthority_child.py @@ -95,9 +95,13 @@ def read_config(filename): pid_file = config.get('BwAuthority', 'pid_file') db_url = config.get('BwAuthority', 'db_url') + only_unmeasured = config.getint('BwAuthority', 'only_unmeasured') + min_unmeasured = config.getint('BwAuthority', 'min_unmeasured') + return (start_pct,stop_pct,nodes_per_slice,save_every, circs_per_node,out_dir,max_fetch_time,tor_dir, - sleep_start,sleep_stop,min_streams,pid_file,db_url) + sleep_start,sleep_stop,min_streams,pid_file,db_url,only_unmeasured, + min_unmeasured) def choose_url(percentile): # TODO: Maybe we don't want to read the file *every* time? @@ -205,7 +209,7 @@ class BwScanHandler(ScanSupport.SQLScanHandler): def speedrace(hdlr, start_pct, stop_pct, circs_per_node, save_every, out_dir, max_fetch_time, sleep_start_tp, sleep_stop_tp, slice_num, - min_streams, sql_file): + min_streams, sql_file, only_unmeasured): hdlr.set_pct_rstr(start_pct, stop_pct) attempt = 0 @@ -214,8 +218,6 @@ def speedrace(hdlr, start_pct, stop_pct, circs_per_node, save_every, out_dir, if hdlr.is_count_met(circs_per_node, successful): break hdlr.wait_for_consensus() - - # Check local time. Do not scan between 01:30 and 05:30 local time lt = time.localtime() sleep_start = time.mktime(lt[0:3]+sleep_start_tp+(0,0,0)+(lt[-1],)) @@ -239,7 +241,14 @@ def speedrace(hdlr, start_pct, stop_pct, circs_per_node, save_every, out_dir, # stream.. however, we count it as 'successful' below timer = threading.Timer(max_fetch_time, lambda: hdlr.close_streams(7)) timer.start() - url = choose_url(start_pct) + + # Always use median URL size for unmeasured nodes + # They may be too slow.. + if only_unmeasured: + url = choose_url(50) + else: + url = choose_url(start_pct) + plog("DEBUG", "Launching stream request for url "+url+" in "+str(start_pct)+'-'+str(stop_pct) + '%') ret = http_request(url) timer.cancel() @@ -297,7 +306,8 @@ def main(argv): TorUtil.read_config(argv[1]) (start_pct,stop_pct,nodes_per_slice,save_every,circs_per_node,out_dir, max_fetch_time,tor_dir,sleep_start,sleep_stop, - min_streams,pid_file_name,db_url) = read_config(argv[1]) + min_streams,pid_file_name,db_url,only_unmeasured, + min_unmeasured) = read_config(argv[1]) # make sure necessary out_dir directory exists path = os.getcwd()+'/'+out_dir @@ -332,9 +342,21 @@ def main(argv): socket.socket = socks.socksocket plog("INFO", "Set socks proxy to "+TorUtil.tor_host+":"+str(TorUtil.tor_port)) + hdlr.schedule_selmgr(lambda s: setattr(s, "only_unmeasured", only_unmeasured)) + hdlr.wait_for_consensus() + # We should go to sleep if there are less than 5 unmeasured nodes after + # consensus update + while min_unmeasured and hdlr.get_unmeasured() < min_unmeasured: + plog("NOTICE", "Less than "+str(min_unmeasured)+" unmeasured nodes ("+str(hdlr.get_unmeasured())+"). Sleeping for a bit") + time.sleep(3600) # Until next consensus arrives + plog("NOTICE", "Woke up from waiting for more unmeasured nodes. Getting consensus and checking again") + hdlr.wait_for_consensus() + pct_step = hdlr.rank_to_percent(nodes_per_slice) + plog("INFO", "Percent per slice is: "+str(pct_step)) + if pct_step > 100: pct_step = 100 # check to see if we are done if (slice_num * pct_step + start_pct > stop_pct): @@ -344,7 +366,7 @@ def main(argv): plog("DEBUG", "Starting slice number %s" % slice_num) successful = speedrace(hdlr, slice_num*pct_step + start_pct, (slice_num + 1)*pct_step + start_pct, circs_per_node, save_every, out_dir, max_fetch_time, sleep_start, sleep_stop, slice_num, - min_streams, sql_file) + min_streams, sql_file, only_unmeasured) # For debugging memory leak.. #TorUtil.dump_class_ref_counts(referrer_depth=1) diff --git a/NetworkScanners/BwAuthority/data/scanner.1/bwauthority.cfg b/NetworkScanners/BwAuthority/data/scanner.1/bwauthority.cfg index 4cced6b..7f81a0b 100644 --- a/NetworkScanners/BwAuthority/data/scanner.1/bwauthority.cfg +++ b/NetworkScanners/BwAuthority/data/scanner.1/bwauthority.cfg @@ -26,3 +26,5 @@ min_streams = 1 max_fetch_time = 600 sleep_start = 01:30 sleep_stop = 01:30 +only_unmeasured = 0 +min_unmeasured = 0 diff --git a/NetworkScanners/BwAuthority/data/scanner.2/bwauthority.cfg b/NetworkScanners/BwAuthority/data/scanner.2/bwauthority.cfg index 53dc469..28cfaa6 100644 --- a/NetworkScanners/BwAuthority/data/scanner.2/bwauthority.cfg +++ b/NetworkScanners/BwAuthority/data/scanner.2/bwauthority.cfg @@ -26,3 +26,5 @@ min_streams = 1 max_fetch_time = 300 sleep_start = 01:30 sleep_stop = 01:30 +only_unmeasured = 0 +min_unmeasured = 0 diff --git a/NetworkScanners/BwAuthority/data/scanner.3/bwauthority.cfg b/NetworkScanners/BwAuthority/data/scanner.3/bwauthority.cfg index 6df68e1..7b71aa8 100644 --- a/NetworkScanners/BwAuthority/data/scanner.3/bwauthority.cfg +++ b/NetworkScanners/BwAuthority/data/scanner.3/bwauthority.cfg @@ -26,3 +26,5 @@ min_streams = 1 max_fetch_time = 300 sleep_start = 01:30 sleep_stop = 01:30 +only_unmeasured = 0 +min_unmeasured = 0 diff --git a/NetworkScanners/BwAuthority/data/scanner.4/bwauthority.cfg b/NetworkScanners/BwAuthority/data/scanner.4/bwauthority.cfg index ff19b87..0dfea36 100644 --- a/NetworkScanners/BwAuthority/data/scanner.4/bwauthority.cfg +++ b/NetworkScanners/BwAuthority/data/scanner.4/bwauthority.cfg @@ -26,3 +26,5 @@ min_streams = 1 max_fetch_time = 300 sleep_start = 01:30 sleep_stop = 01:30 +only_unmeasured = 0 +min_unmeasured = 0 diff --git a/NetworkScanners/BwAuthority/data/scanner.5/bwauthority.cfg b/NetworkScanners/BwAuthority/data/scanner.5/bwauthority.cfg new file mode 100644 index 0000000..468a62e --- /dev/null +++ b/NetworkScanners/BwAuthority/data/scanner.5/bwauthority.cfg @@ -0,0 +1,30 @@ +[TorCtl] +loglevel=WARN +tor_host = 127.0.0.1 +tor_port = 9110 +control_host = 127.0.0.1 +control_port = 9111 +control_pass = +# XXX: Unused +meta_host = 127.0.0.1 +meta_port = 9112 + +[BwAuthority] +out_dir = ./data/scanner.5/scan-data +pid_file = ./data/scanner.5/bwauthority.pid +# if db_url is unset bwauthority will default to sqlite +db_url = +#db_url = mysql+mysqldb://bwscanner:password@127.0.0.1/BwScan4 +#db_url = postgresql://bwscanner:password@127.0.0.1/BwScan4 +tor_dir = ./data/tor +start_pct = 0 +stop_pct = 100 +save_every = 0 +nodes_per_slice = 50 +circs_per_node = 5 +min_streams = 1 +max_fetch_time = 360 +sleep_start = 01:30 +sleep_stop = 01:30 +only_unmeasured = 1 +min_unmeasured = 5 diff --git a/NetworkScanners/BwAuthority/data/scanner.5/scan-data/.gitignore b/NetworkScanners/BwAuthority/data/scanner.5/scan-data/.gitignore new file mode 100644 index 0000000..e69de29 diff --git a/NetworkScanners/BwAuthority/run_scan.sh b/NetworkScanners/BwAuthority/run_scan.sh index c21e871..7853647 100755 --- a/NetworkScanners/BwAuthority/run_scan.sh +++ b/NetworkScanners/BwAuthority/run_scan.sh @@ -1,7 +1,7 @@ #!/bin/sh # Number of scanners to run. -SCANNER_COUNT=4 +SCANNER_COUNT=5 # This tor must have the w status line fix as well as the stream bw fix # Ie git master or 0.2.2.x

1 0

[pytorctl/master] Really fix Bug #6131.
by mikeperry＠torproject.org 01 Jun '15

01 Jun '15

commit 88d09f8b269f68f8b4c7fa5d6de86698b6c1f9d6 Author: Mike Perry <mikeperry+git(a)torproject.org> Date: Thu May 21 22:14:38 2015 -0700 Really fix Bug #6131. Man, what happened here? I pointed out most of these issues in code review... https://trac.torproject.org/projects/tor/ticket/6131 --- TorCtl.py | 32 ++++++++++++++++++-------------- 1 file changed, 18 insertions(+), 14 deletions(-) diff --git a/TorCtl.py b/TorCtl.py index 562ea78..b74192d 100755 --- a/TorCtl.py +++ b/TorCtl.py @@ -374,12 +374,14 @@ class Router: self.idhex = idhex self.nickname = name if ns_bandwidth != None: - self.bw = ns_bandwidth + self.bw = max(ns_bandwidth,1) # Avoid div by 0 else: - self.bw = bw + self.bw = max(bw,1) # Avoid div by 0 if unmeasured: self.unmeasured = True - self.desc_bw = bw + else: + self.unmeasured = False + self.desc_bw = max(bw,1) # Avoid div by 0 self.exitpolicy = exitpolicy self.flags = flags # Technicaly from NS doc self.down = down @@ -510,9 +512,8 @@ class Router: def get_unmeasured_bw(self): # if unmeasured, the ratio of self.bw/self.desc_bw should be 1.0 - if self.unmeasured and self.bw > 0: return self.bw - elif self.desc_bw > 0: return self.desc_bw - else: return 1 + if self.unmeasured: return self.desc_bw + else: return self.bw class Connection: """A Connection represents a connection to the Tor process via the @@ -1232,9 +1233,11 @@ def ns_body_iter(data): w = re.search(r"^w Bandwidth=(\d+)(?:\s(Unmeasured)=1)?", nsline, re.M) unmeasured = None if w: - if w.groups(2): unmeasured = True + if w.group(2): + unmeasured = True yield NetworkStatus(*(m.groups()+(flags,)+(int(w.group(1))*1000,))+(unmeasured,)) else: + plog("WARN", "NS document has no bandwidth line: "+nsline) yield NetworkStatus(*(m.groups() + (flags,))) class EventSink: @@ -1637,9 +1640,10 @@ class ConsensusTracker(EventHandler): # if unmeasured is true... Then the ratio will work out to 1 that way." ratio_r = copy.copy(self.sorted_r) - ratio_r.sort(lambda x, y: cmp(float(y.bw)/y.get_unmeasured_bw(), - float(x.bw)/x.get_unmeasured_bw())) - for i in xrange(len(ratio_r)): ratio_r[i].ratio_rank = i + ratio_r.sort(lambda x, y: cmp(y.get_unmeasured_bw()/float(y.desc_bw), + x.get_unmeasured_bw()/float(x.desc_bw))) + for i in xrange(len(ratio_r)): + ratio_r[i].ratio_rank = i # XXX: Verification only. Can be removed. self._sanity_check(self.sorted_r) @@ -1726,8 +1730,8 @@ class ConsensusTracker(EventHandler): for i in xrange(len(self.sorted_r)): self.sorted_r[i].list_rank = i ratio_r = copy.copy(self.sorted_r) - ratio_r.sort(lambda x, y: cmp(float(y.bw)/y.get_unmeasured_bw(), - float(x.bw)/x.get_unmeasured_bw())) + ratio_r.sort(lambda x, y: cmp(y.get_unmeasured_bw()/y.desc_bw, + x.get_unmeasured_bw()/x.desc_bw)) for i in xrange(len(ratio_r)): ratio_r[i].ratio_rank = i plog("DEBUG", str(time.time()-d.arrived_at)+ " Read " + str(len(d.idlist)) +" ND => "+str(len(self.sorted_r))+" routers. Update: "+str(update)) @@ -1760,8 +1764,8 @@ class ConsensusTracker(EventHandler): for i in xrange(len(self.sorted_r)): self.sorted_r[i].list_rank = i ratio_r = copy.copy(self.sorted_r) - ratio_r.sort(lambda x, y: cmp(float(y.bw)/y.get_unmeasured_bw(), - float(x.bw)/x.get_unmeasured_bw())) + ratio_r.sort(lambda x, y: cmp(y.get_unmeasured_bw()/float(y.desc_bw), + x.get_unmeasured_bw()/float(x.desc_bw))) for i in xrange(len(ratio_r)): ratio_r[i].ratio_rank = i self._sanity_check(self.sorted_r)

1 0

[pytorctl/master] Create option to add in exits if we don't have enough in a slice.
by mikeperry＠torproject.org 01 Jun '15

01 Jun '15

commit e2561cb287796bd161648d3879ec562d17fbb623 Author: Mike Perry <mikeperry+git(a)torproject.org> Date: Wed May 27 16:05:21 2015 -0700 Create option to add in exits if we don't have enough in a slice. --- PathSupport.py | 44 +++++++++++++++++++++++++++++++++++++++++++- SQLSupport.py | 10 ++++++++-- ScanSupport.py | 7 +++++-- 3 files changed, 56 insertions(+), 5 deletions(-) diff --git a/PathSupport.py b/PathSupport.py index c9c5ee1..8d86d3f 100644 --- a/PathSupport.py +++ b/PathSupport.py @@ -317,6 +317,20 @@ class ConserveExitsRestriction(NodeRestriction): def __str__(self): return self.__class__.__name__+"()" +class ExitPortRestriction(NodeRestriction): + "Restriction to select exits that can exit to a port list" + def __init__(self, exit_ports=None): + self.exit_ports = exit_ports + + def r_is_ok(self, r): + for port in self.exit_ports: + if not r.will_exit_to("255.255.255.255", port): + return False + return True + + def __str__(self): + return self.__class__.__name__+"()" + class FlagsRestriction(NodeRestriction): "Restriction for mandatory and forbidden router flags" def __init__(self, mandatory, forbidden=[]): @@ -1030,7 +1044,7 @@ class SelectionManager(BaseSelectionManager): percent_fast, percent_skip, min_bw, use_all_exits, uniform, use_exit, use_guards,geoip_config=None, restrict_guards=False, extra_node_rstr=None, exit_ports=None, - order_by_ratio=False): + order_by_ratio=False, min_exits=0): BaseSelectionManager.__init__(self) self.__ordered_exit_gen = None self.pathlen = pathlen @@ -1049,6 +1063,8 @@ class SelectionManager(BaseSelectionManager): self.exit_ports = exit_ports self.extra_node_rstr=extra_node_rstr self.order_by_ratio = order_by_ratio + self.min_exits = min_exits + self.added_exits = [] def reconfigure(self, consensus=None): try: @@ -1126,6 +1142,21 @@ class SelectionManager(BaseSelectionManager): mid_rstr.add_restriction(self.extra_node_rstr) self.exit_rstr.add_restriction(self.extra_node_rstr) + # This is a hack just for the bw auths to avoid slices with no exits + if self.min_exits and self.exit_ports: + test_rstr = NodeRestrictionList( + [PctRstr(nonentry_skip, nonentry_fast, sorted_r), + ExitPortRestriction(self.exit_ports), + FlagsRestriction(["Running"], ["BadExit"])]) + exit_count = len(filter(lambda r: test_rstr.r_is_ok(r), sorted_r)) + if exit_count < self.min_exits: + self.added_exits = self.find_emergency_exits(sorted_r, + self.min_exits-exit_count) + plog("NOTICE", "Only "+str(exit_count)+" exits remain in slice "+str(nonentry_skip)+"-"+str(nonentry_fast)+" after restrictions. Adding in "+str(self.added_exits)) + idhex_list = map(IdHexRestriction, self.added_exits) + idhex_list.append(self.exit_rstr) + self.exit_rstr = NodeRestrictionList([OrNodeRestriction(idhex_list)]) + # GeoIP configuration if self.geoip_config: # Every node needs country_code @@ -1198,6 +1229,17 @@ class SelectionManager(BaseSelectionManager): exitgen, self.path_rstr) return + # Picks num of the top_n fastest exits that can handle our exit ports. + def find_emergency_exits(self, sorted_r, num, top_n=100): + new_exits = [] + test_rstr = NodeRestrictionList( + [ExitPortRestriction(self.exit_ports), + FlagsRestriction(["Running"], ["BadExit"])]) + for r in sorted_r: + if test_rstr.r_is_ok(r): new_exits.append(r.idhex) + if len(new_exits) >= top_n: break + return random.sample(new_exits, min(len(new_exits), num)) + def _set_exit(self, exit_name): # sets an exit, if bad, sets bad_exit exit_id = None diff --git a/SQLSupport.py b/SQLSupport.py index 28963fc..ee15d79 100644 --- a/SQLSupport.py +++ b/SQLSupport.py @@ -565,7 +565,7 @@ class RouterStats(Entity): tc_session.remove() compute = Callable(compute) - def write_stats(f, pct_low=0, pct_high=100, order_by=None, recompute=False, stat_clause=None, filter_clause=None, disp_clause=None): + def write_stats(f, pct_low=0, pct_high=100, order_by=None, recompute=False, stat_clause=None, filter_clause=None, disp_clause=None, ignore_by_idhex=[]): l_session = tc_session() if not order_by: @@ -626,6 +626,9 @@ class RouterStats(Entity): for s in RouterStats.query.filter(pct_clause).filter(stat_clause).\ filter(disp_clause).order_by(order_by).all(): + if s.router.idhex in ignore_by_idhex: + plog("NOTICE", "Skipping stats on added exit "+s.router.idhex+" in "+f.name) + continue f.write(s.router.idhex+" ("+s.router.nickname+")\n") f.write(" CF="+str(cvt(s.circ_from_rate,2))) f.write(" CT="+str(cvt(s.circ_to_rate,2))) @@ -649,7 +652,7 @@ class RouterStats(Entity): write_stats = Callable(write_stats) - def write_bws(f, pct_low=0, pct_high=100, order_by=None, recompute=False, stat_clause=None, filter_clause=None, disp_clause=None): + def write_bws(f, pct_low=0, pct_high=100, order_by=None, recompute=False, stat_clause=None, filter_clause=None, disp_clause=None, ignore_by_idhex=[]): l_session = tc_session() if not order_by: order_by=RouterStats.avg_first_ext @@ -678,6 +681,9 @@ class RouterStats(Entity): for s in RouterStats.query.filter(pct_clause).filter(stat_clause).\ filter(disp_clause).order_by(order_by).all(): + if s.router.idhex in ignore_by_idhex: + plog("NOTICE", "Skipping stats on added exit "+s.router.idhex+" in "+f.name) + continue f.write("node_id=$"+s.router.idhex+" nick="+s.router.nickname) f.write(" strm_bw="+str(cvt(s.sbw,0))) f.write(" filt_bw="+str(cvt(s.filt_sbw,0))) diff --git a/ScanSupport.py b/ScanSupport.py index e6752c5..797fe39 100644 --- a/ScanSupport.py +++ b/ScanSupport.py @@ -79,6 +79,7 @@ class ScanHandler(PathSupport.PathBuilder): cond.acquire() this.new_nym = True if this.selmgr.bad_restrictions: + # XXX: Maybe add in exits here? plog("NOTICE", "Clearing bad restrictions with reconfigure..") this.selmgr.reconfigure(this.current_consensus()) lines = this.c.sendAndRecv("SIGNAL CLEARDNSCACHE\r\n") @@ -198,7 +199,8 @@ class SQLScanHandler(ScanHandler): cond.acquire() SQLSupport.RouterStats.write_stats(file(rfilename, "w"), 0, 100, order_by=SQLSupport.RouterStats.sbw, - recompute=True, disp_clause=stats_filter) + recompute=True, disp_clause=stats_filter, + ignore_by_idhex=h.selmgr.added_exits) cond.notify() cond.release() cond.acquire() @@ -216,7 +218,8 @@ class SQLScanHandler(ScanHandler): f.write("slicenum="+str(slice_num)+"\n") SQLSupport.RouterStats.write_bws(f, 0, 100, order_by=SQLSupport.RouterStats.sbw, - recompute=False, disp_clause=stats_filter) + recompute=False, disp_clause=stats_filter, + ignore_by_idhex=this.selmgr.added_exits) f.close() cond.notify() cond.release()

1 0

[pytorctl/master] Hacks to support measurement of unmeasured=1 nodes.
by mikeperry＠torproject.org 01 Jun '15

01 Jun '15

commit c8fcb25b079d52a20cafc7f7adf178e90ab76338 Author: Mike Perry <mikeperry+git(a)torproject.org> Date: Sun May 31 19:03:15 2015 -0700 Hacks to support measurement of unmeasured=1 nodes. UnmeasuredPercentileRestriction is a little slow if there are a lot of nodes... --- PathSupport.py | 43 +++++++++++++++++++++++++++++++++++++++++-- ScanSupport.py | 27 ++++++++++++++++++++++++++- 2 files changed, 67 insertions(+), 3 deletions(-) diff --git a/PathSupport.py b/PathSupport.py index 8d86d3f..4e024b7 100644 --- a/PathSupport.py +++ b/PathSupport.py @@ -250,6 +250,35 @@ class RatioPercentileRestriction(NodeRestriction): def __str__(self): return self.__class__.__name__+"("+str(self.pct_skip)+","+str(self.pct_fast)+")" +class UnmeasuredPercentileRestriction(NodeRestriction): + """Restriction to only accept routers from a percentile range in the + unmeasured set""" + def __init__(self, pct_skip, pct_fast, r_list): + """Constructor. Sets up the restriction such that routers in the + 'pct_skip' to 'pct_fast' percentile of bandwidth rankings are + returned from the sorted list 'r_list'""" + self.pct_fast = pct_fast + self.pct_skip = pct_skip + self.sorted_r = filter(lambda r: r.unmeasured, r_list) + plog("DEBUG", "UnmeasuredPercentileRestriction built with "+str(len(self.sorted_r))+" routers") + + def r_is_ok(self, r): + "Returns true if r is in the unmeasured percentile boundaries" + if not r.unmeasured: return False + + # XXX: Can throw an exception somehow??? catch ValueError here.. + try: + idx = self.sorted_r.index(r) + except ValueError: + return False + + if idx < len(self.sorted_r)*self.pct_skip/100: return False + elif idx > len(self.sorted_r)*self.pct_fast/100: return False + return True + + def __str__(self): + return self.__class__.__name__+"("+str(self.pct_skip)+","+str(self.pct_fast)+")" + class UptimeRestriction(NodeRestriction): """Restriction to filter out routers with uptimes < min_uptime or > max_uptime""" @@ -263,6 +292,12 @@ class UptimeRestriction(NodeRestriction): if self.max_uptime and r.uptime > self.max_uptime: return False return True +class UnmeasuredRestriction(NodeRestriction): + """Restriction to select only unmeasured=1 routers""" + def r_is_ok(self, r): + "Returns true if r is unmeasured" + return r.unmeasured + class RankRestriction(NodeRestriction): """Restriction to cut out a list-rank slice of the network.""" def __init__(self, rank_skip, rank_stop): @@ -738,6 +773,7 @@ class ExactUniformGenerator(NodeGenerator): if lgen < self.position+1: for i in xrange(lgen, self.position+1): r._generated.append(0) + plog("DEBUG", "Rebuilt ExactUniformGenerator") class OrderedExitGenerator(NodeGenerator): @@ -1044,7 +1080,7 @@ class SelectionManager(BaseSelectionManager): percent_fast, percent_skip, min_bw, use_all_exits, uniform, use_exit, use_guards,geoip_config=None, restrict_guards=False, extra_node_rstr=None, exit_ports=None, - order_by_ratio=False, min_exits=0): + order_by_ratio=False, min_exits=0, only_unmeasured=False): BaseSelectionManager.__init__(self) self.__ordered_exit_gen = None self.pathlen = pathlen @@ -1065,6 +1101,7 @@ class SelectionManager(BaseSelectionManager): self.order_by_ratio = order_by_ratio self.min_exits = min_exits self.added_exits = [] + self.only_unmeasured = only_unmeasured def reconfigure(self, consensus=None): try: @@ -1102,7 +1139,9 @@ class SelectionManager(BaseSelectionManager): nonentry_skip = self.percent_skip nonentry_fast = self.percent_fast - if self.order_by_ratio: + if self.only_unmeasured: + PctRstr = UnmeasuredPercentileRestriction + elif self.order_by_ratio: PctRstr = RatioPercentileRestriction else: PctRstr = PercentileRestriction diff --git a/ScanSupport.py b/ScanSupport.py index 797fe39..d3bfbc1 100644 --- a/ScanSupport.py +++ b/ScanSupport.py @@ -128,7 +128,16 @@ class ScanHandler(PathSupport.PathBuilder): cond = threading.Condition() def notlambda(this): cond.acquire() - cond._pct = (100.0*rank)/len(this.sorted_r) # lol moar haxx + # This is ugly. If we are only scanning the unmeasured routers, + # we need a special count of just them.. + if this.selmgr.only_unmeasured: + # XXX: This can exceed 100% + count = 0 + for r in this.sorted_r: + if r.unmeasured: count += 1 + else: + count = len(this.sorted_r) + cond._pct = (100.0*rank)/count # XXX: Div 0 here if no unmeasured.. cond.notify() cond.release() cond.acquire() @@ -137,6 +146,22 @@ class ScanHandler(PathSupport.PathBuilder): cond.release() return cond._pct + def get_unmeasured(self): + cond = threading.Condition() + def notlambda(this): + cond.acquire() + count = 0 + for r in this.sorted_r: + if r.unmeasured: count += 1 + cond._count = count + cond.notify() + cond.release() + cond.acquire() + self.schedule_low_prio(notlambda) + cond.wait() + cond.release() + return cond._count + def percent_to_rank(self, pct): cond = threading.Condition() def notlambda(this):

1 0