May 2020 - tor-commits - lists.torproject.org

[tor/master] Add documentation about %include and seccomp sandbox limitations
by nickm＠torproject.org 12 May '20

12 May '20

commit cc169eb1204e1cf7d9fd45518edf86a1b397f6dc Author: Daniel Pinto <danielpinto52(a)gmail.com> Date: Wed May 6 01:42:24 2020 +0100 Add documentation about %include and seccomp sandbox limitations --- changes/doc34133 | 6 ++++++ doc/tor.1.txt | 6 ++++++ 2 files changed, 12 insertions(+) diff --git a/changes/doc34133 b/changes/doc34133 new file mode 100644 index 000000000..abe9db614 --- /dev/null +++ b/changes/doc34133 @@ -0,0 +1,6 @@ + o Documentation: + - Correctly document that we search for a system torrc file before + Document the limitations of using %include on config files with + seccomp sandbox enabled. No new files can be added to the + %included directories. Fixes documentation bug 34133; bugfix + on 0.3.1.1-alpha. Patch by Daniel Pinto. diff --git a/doc/tor.1.txt b/doc/tor.1.txt index fbe2915cf..bc09d634b 100644 --- a/doc/tor.1.txt +++ b/doc/tor.1.txt @@ -211,6 +211,8 @@ file will be parsed as if they were written where the %include option is. If the path is a folder, all files on that folder will be parsed following lexical order. Files starting with a dot are ignored. Files on subfolders are ignored. The %include option can be used recursively. +New configuration files or directories cannot be added to already running Tor +instance if **Sandbox** is enabled. By default, an option on the command line overrides an option found in the configuration file, and an option in a configuration file overrides one in @@ -855,6 +857,10 @@ forward slash (/) in the configuration file and on the command line. and **ORPort** are not allowed). Currently, if **Sandbox** is 1, **ControlPort** command "GETINFO address" will not work. + + + When using %include in the tor configuration files, reloading the tor + configuration is not supported after adding new configuration files or + directories. + + + (Default: 0) [[Schedulers]] **Schedulers** **KIST**|**KISTLite**|**Vanilla**::

1 0

[tor/master] Merge remote-tracking branch 'tor-github/pr/1885/head'
by nickm＠torproject.org 12 May '20

12 May '20

commit 86b5bcd80ad27083c9b8e5c5999927f235905ec6 Merge: b7a165228 cc169eb12 Author: Nick Mathewson <nickm(a)torproject.org> Date: Tue May 12 12:52:41 2020 -0400 Merge remote-tracking branch 'tor-github/pr/1885/head' changes/doc34133 | 6 ++++++ doc/tor.1.txt | 6 ++++++ 2 files changed, 12 insertions(+)

1 0

[translation/communitytpo-contentspot] https://gitweb.torproject.org/translation.git/commit/?h=communitytpo-contentspot
by translation＠torproject.org 12 May '20

12 May '20

commit adfdf7a14689710b3f368b779246f7c87ccb254f Author: Translation commit bot <translation(a)torproject.org> Date: Tue May 12 13:45:27 2020 +0000 https://gitweb.torproject.org/translation.git/commit/?h=communitytpo-conten… --- contents+es.po | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/contents+es.po b/contents+es.po index 5a9bdf4b5f..a583825a6c 100644 --- a/contents+es.po +++ b/contents+es.po @@ -2458,27 +2458,27 @@ msgstr "[ ] Llegar a un acuerdo sobre tomar fotos o no" #: https//community.torproject.org/training/checklist/ #: (content/training/checklist/contents+en.lrpage.body) msgid "[ ] Present the agenda" -msgstr "" +msgstr "[ ] Presenté la agenda" #: https//community.torproject.org/training/checklist/ #: (content/training/checklist/contents+en.lrpage.body) msgid "[ ] Introduce yourself and ask people to introduce themselves" -msgstr "" +msgstr "[ ] Preséntese y pida a la gente que se presente" #: https//community.torproject.org/training/checklist/ #: (content/training/checklist/contents+en.lrpage.body) msgid "[ ] \"All questions are welcome\"" -msgstr "" +msgstr "[ ] \"Todas las preguntas son bienvenidas\"" #: https//community.torproject.org/training/checklist/ #: (content/training/checklist/contents+en.lrpage.body) msgid "## After the Training" -msgstr "" +msgstr "## Después de la formación" #: https//community.torproject.org/training/checklist/ #: (content/training/checklist/contents+en.lrpage.body) msgid "[ ] Collect participants feedback" -msgstr "" +msgstr "[ ] Recoja las impresiones de los participantes" #: https//community.torproject.org/training/checklist/ #: (content/training/checklist/contents+en.lrpage.body)

1 0

[onionperf/master] Integrate reprocessing mode into analysis mode.
by karsten＠torproject.org 12 May '20

12 May '20

commit f0b985e4e1b2a20c1968014e25126f874de684c6 Author: Karsten Loesing <karsten.loesing(a)gmx.net> Date: Mon May 11 10:16:36 2020 +0200 Integrate reprocessing mode into analysis mode. Tweaked by acute to carry the -s and -t arguments to reprocessing. Implements #34142. --- onionperf/onionperf | 106 +++++++++-------------------------- onionperf/reprocessing.py | 13 +++-- onionperf/tests/test_reprocessing.py | 6 +- 3 files changed, 37 insertions(+), 88 deletions(-) diff --git a/onionperf/onionperf b/onionperf/onionperf index 536d6e2..cb1899c 100755 --- a/onionperf/onionperf +++ b/onionperf/onionperf @@ -63,18 +63,12 @@ The standard way to run this subcommand is to give the path to a TGen and/or a TorCtl file (e.g., those produced with the `measure` subcommand) using the `--tgen` and `--torctl` options, and the statistics file resulting from the analysis will be dumped to `onionperf.analysis.json.xz`. -(See https://collector.torproject.org/#type-torperf.) +Another way to run this subcommand is to give two paths to directories +containing TGen and TorCtl files to have files matched by filename and analysis +files dumped to `%Y-%m-%d.onionperf.analysis.json.xz`. +(See https://collector.torproject.org/#type-onionperf.) Stats files in the default Torperf format can also be exported. """ -DESC_REPROCESS = """ -Reprocesses results in bulk from the TGen traffic generator and Tor. - -This subcommand scans for TGen and Tor log files in the given paths, -matches them by filename and then runs the analysis command on each pair, -reproducing all analysis results. - -This is useful when reprocessing logs in bulk, for example -""" HELP_ANALYZE = """ Analyze Tor and TGen output @@ -86,9 +80,6 @@ and plots various interesting performance metrics to PDF files. HELP_VISUALIZE = """ Visualize OnionPerf analysis results """ -HELP_REPROCESS = """ -Reprocesses all OnionPerf log files in the given paths -""" logging.basicConfig(format='%(asctime)s %(created)f [onionperf] [%(levelname)s] %(message)s', level=logging.INFO, datefmt='%Y-%m-%d %H:%M:%S') logging.getLogger("stem").setLevel(logging.WARN) @@ -240,13 +231,13 @@ built-in Torperf (50KiB, 1MiB, 5MiB) traffic model""", analyze_parser.set_defaults(func=analyze, formatter_class=my_formatter_class) analyze_parser.add_argument('--tgen', - help="""a file PATH to a TGen logfile""", + help="""a file or directory PATH to a TGen logfile or logfile directory""", metavar="PATH", type=type_str_path_in, action="store", dest="tgen_logpath", default=None) analyze_parser.add_argument('--torctl', - help="""a file PATH to a TorCtl logfile (in the format output by the monitor subcommmand)""", + help="""a file or directory PATH to a TorCtl logfile or logfile directory (in the format output by the monitor subcommmand)""", metavar="PATH", type=type_str_path_in, action="store", dest="torctl_logpath", default=None) @@ -355,43 +346,6 @@ files generated by this script will be written""", action="store", dest="lineformats", default=util.LINEFORMATS) - # reprocess - reprocess_parser = sub_parser.add_parser('reprocess', description=DESC_REPROCESS, help=HELP_REPROCESS, - formatter_class=my_formatter_class) - reprocess_parser.set_defaults(func=reprocess, formatter_class=my_formatter_class) - - reprocess_parser.add_argument('--tgen-dir', - help="""a dictory PATH to a TGen logfile directory""", - metavar="PATH", type=type_str_path_in, - action="store", dest="tgen_dirpath", - default=None) - - reprocess_parser.add_argument('--torctl-dir', - help="""a directory PATH to a TorCtl logfile directory""", - metavar="PATH", type=type_str_path_in, - action="store", dest="torctl_dirpath", - default=None) - - reprocess_parser.add_argument('-p', '--prefix', - help="""A directory PATH prefix where the processed data -files generated by this script will be written""", - metavar="PATH", type=type_str_dir_path_out, - action="store", dest="prefix", - default=os.getcwd()) - - reprocess_parser.add_argument('-d', '--date-filter', - help="""a DATE string in the form YYYY-MM-DD, all log messages that did not occur on this date will be filtered out of the analysis""", - metavar="DATE", type=type_str_date_in, - action="store", dest="date_filter", - default=None) - - reprocess_parser.add_argument('-n', '--nickname', - help="""a nickname STRING that identifies the machine where the input logfiles were produced""", - metavar="STRING", type=str, - action="store", dest="nickname", - default=None) - - # get args and call the command handler for the chosen mode args = main_parser.parse_args() args.func(args) @@ -450,23 +404,32 @@ def measure(args): logging.info("Please fix path errors to continue") def analyze(args): - from onionperf.analysis import Analysis if args.tgen_logpath is None and args.torctl_logpath is None: logging.warning("No logfile paths were given, nothing will be analyzed") - return - - analysis = Analysis(nickname=args.nickname, ip_address=args.ip_address) - if args.tgen_logpath is not None: - analysis.add_tgen_file(args.tgen_logpath) - if args.torctl_logpath is not None: - analysis.add_torctl_file(args.torctl_logpath) + elif (args.tgen_logpath is None or os.path.isfile(args.tgen_logpath)) and (args.torctl_logpath is None or os.path.isfile(args.torctl_logpath)): + from onionperf.analysis import Analysis + analysis = Analysis(nickname=args.nickname, ip_address=args.ip_address) + if args.tgen_logpath is not None: + analysis.add_tgen_file(args.tgen_logpath) + if args.torctl_logpath is not None: + analysis.add_torctl_file(args.torctl_logpath) + analysis.analyze(args.do_simple, date_filter=args.date_filter) + analysis.save(output_prefix=args.prefix) + if args.save_torperf: + analysis.export_torperf_version_1_1(output_prefix=args.prefix, do_compress=False) + + elif args.tgen_logpath is not None and os.path.isdir(args.tgen_logpath) and args.torctl_logpath is not None and os.path.isdir(args.torctl_logpath): + from onionperf import reprocessing + tgen_logs = reprocessing.collect_logs(args.tgen_logpath, '*tgen.log*') + torctl_logs = reprocessing.collect_logs(args.torctl_logpath, '*torctl.log*') + log_pairs = reprocessing.match(tgen_logs, torctl_logs, args.date_filter) + logging.info("Found {0} matching log pairs to be reprocessed".format(len(log_pairs))) + reprocessing.multiprocess_logs(log_pairs, args.prefix, args.nickname, args.save_torperf, args.do_simple) - analysis.analyze(args.do_simple, date_filter=args.date_filter) - analysis.save(output_prefix=args.prefix) - if args.save_torperf: - analysis.export_torperf_version_1_1(output_prefix=args.prefix, do_compress=False) + else: + logging.error("Given paths were an unrecognized mix of file and directory paths, nothing will be analyzed") def visualize(args): from onionperf.visualization import TGenVisualization, TorVisualization @@ -489,21 +452,6 @@ def visualize(args): tgen_viz.plot_all(args.prefix) tor_viz.plot_all(args.prefix) -def reprocess(args): - from onionperf import reprocessing - if args.tgen_dirpath is None or args.torctl_dirpath is None: - logging.error("Required directory paths were not given - nothing will be reprocessed") - return - elif not os.path.isdir(args.tgen_dirpath) or not os.path.isdir(args.torctl_dirpath): - logging.error("One or more given paths do not exist or are not directories - nothing will be reprocessed") - return - else: - tgen_logs = reprocessing.collect_logs(args.tgen_dirpath, '*tgen.log') - torctl_logs = reprocessing.collect_logs(args.torctl_dirpath, '*torctl.log') - log_pairs = reprocessing.match(tgen_logs, torctl_logs, args.date_filter) - logging.info("Found {0} matching log pairs to be reprocessed".format(len(log_pairs))) - reprocessing.multiprocess_logs(log_pairs, args.prefix, args.nickname) - def type_nonnegative_integer(value): i = int(value) if i < 0: raise argparse.ArgumentTypeError("'%s' is an invalid non-negative int value" % value) diff --git a/onionperf/reprocessing.py b/onionperf/reprocessing.py index 7acf539..48f67bb 100644 --- a/onionperf/reprocessing.py +++ b/onionperf/reprocessing.py @@ -46,23 +46,24 @@ def match(tgen_logs, tor_logs, date_filter): return log_pairs -def analyze_func(prefix, nick, pair): +def analyze_func(prefix, nick, save_torperf, do_simple, pair): analysis = Analysis(nickname=nick) logging.info('Analysing pair for date {0}'.format(pair[2])) analysis.add_tgen_file(pair[0]) analysis.add_torctl_file(pair[1]) - analysis.analyze(do_simple=False, date_filter=pair[2]) + analysis.analyze(do_simple=do_simple, date_filter=pair[2]) analysis.save(output_prefix=prefix) - analysis.export_torperf_version_1_1( - output_prefix=prefix, do_compress=False) + if save_torperf: + analysis.export_torperf_version_1_1( + output_prefix=prefix, do_compress=False) return 1 -def multiprocess_logs(log_pairs, prefix, nick=None): +def multiprocess_logs(log_pairs, prefix, nick=None, save_torperf=False, do_simple=False): pool = Pool(cpu_count()) analyses = None try: - func = partial(analyze_func, prefix, nick) + func = partial(analyze_func, prefix, nick, save_torperf, do_simple) mr = pool.map_async(func, log_pairs) pool.close() while not mr.ready(): diff --git a/onionperf/tests/test_reprocessing.py b/onionperf/tests/test_reprocessing.py index efacc5f..a120587 100644 --- a/onionperf/tests/test_reprocessing.py +++ b/onionperf/tests/test_reprocessing.py @@ -61,7 +61,7 @@ def test_log_match_with_wrong_filter_date(): def test_analyze_func_json(): pair = (DATA_DIR + 'logs/onionperf_2019-01-10_23:59:59.tgen.log', DATA_DIR + 'logs/onionperf_2019-01-10_23:59:59.torctl.log', datetime.datetime(2019, 1, 10, 0, 0)) work_dir = tempfile.mkdtemp() - reprocessing.analyze_func(work_dir, None, pair) + reprocessing.analyze_func(work_dir, None, True, False, pair) json_file = os.path.join(work_dir, "2019-01-10.onionperf.analysis.json.xz") assert(os.path.exists(json_file)) for i in ['51200', '5242880', '1048576']: @@ -72,7 +72,7 @@ def test_analyze_func_json(): def test_multiprocess_logs(): pairs = [(DATA_DIR + 'logs/onionperf_2019-01-10_23:59:59.tgen.log', DATA_DIR + 'logs/onionperf_2019-01-10_23:59:59.torctl.log', datetime.datetime(2019, 1, 10, 0, 0))] work_dir = tempfile.mkdtemp() - reprocessing.multiprocess_logs(pairs, work_dir) + reprocessing.multiprocess_logs(pairs, work_dir, save_torperf=True) json_file = os.path.join(work_dir, "2019-01-10.onionperf.analysis.json.xz") assert(os.path.exists(json_file)) for i in ['51200', '5242880', '1048576']: @@ -85,7 +85,7 @@ def test_end_to_end(): torctl_logs = reprocessing.collect_logs(DATA_DIR, '*torctl.log') log_pairs = reprocessing.match(tgen_logs, torctl_logs, None) work_dir = tempfile.mkdtemp() - reprocessing.multiprocess_logs(log_pairs, work_dir) + reprocessing.multiprocess_logs(log_pairs, work_dir, save_torperf=True) json_file = os.path.join(work_dir, "2019-01-10.onionperf.analysis.json.xz") assert(os.path.exists(json_file)) for i in ['51200', '5242880', '1048576']:

1 0

[translation/communitytpo-contentspot] https://gitweb.torproject.org/translation.git/commit/?h=communitytpo-contentspot
by translation＠torproject.org 12 May '20

12 May '20

commit c60f813a1f0ad242ec4f5b4a66907c99a79d3347 Author: Translation commit bot <translation(a)torproject.org> Date: Tue May 12 10:15:29 2020 +0000 https://gitweb.torproject.org/translation.git/commit/?h=communitytpo-conten… --- contents+zh-CN.po | 15 ++++++++------- 1 file changed, 8 insertions(+), 7 deletions(-) diff --git a/contents+zh-CN.po b/contents+zh-CN.po index 9bde5e4489..37ccf1ad45 100644 --- a/contents+zh-CN.po +++ b/contents+zh-CN.po @@ -8,6 +8,7 @@ # 忻然李 <2544304894(a)qq.com>, 2020 # ff98sha, 2020 # Runzhe Liang <18051080098(a)163.com>, 2020 +# Scott Rhodes <starring169(a)gmail.com>, 2020 # msgid "" msgstr "" @@ -15,7 +16,7 @@ msgstr "" "Report-Msgid-Bugs-To: \n" "POT-Creation-Date: 2019-12-11 12:15+CET\n" "PO-Revision-Date: 2019-12-11 10:50+0000\n" -"Last-Translator: Runzhe Liang <18051080098(a)163.com>, 2020\n" +"Last-Translator: Scott Rhodes <starring169(a)gmail.com>, 2020\n" "Language-Team: Chinese (China) (https://www.transifex.com/otf/teams/1519/zh_CN/)\n" "MIME-Version: 1.0\n" "Content-Type: text/plain; charset=UTF-8\n" @@ -334,7 +335,7 @@ msgstr "" #: (content/relay-operations/technical-setup/contents+en.lrpage.body) msgid "" "* Do you want to run a Tor exit or non-exit (bridge/guard/middle) relay?" -msgstr "" +msgstr "* 你想运行一个Tor出口或非出口（网桥/守卫/中间）节点？" #: https//community.torproject.org/relay/setup/ #: (content/relay-operations/technical-setup/contents+en.lrpage.body) @@ -388,7 +389,7 @@ msgstr "" #: (content/relay-operations/technical-setup/contents+en.lrpage.body) msgid "" "* How much bandwidth/monthly traffic do you want to allow for Tor traffic?" -msgstr "" +msgstr "* 你想给Tor允许每月多少带宽/流量？" #: https//community.torproject.org/relay/setup/ #: (content/relay-operations/technical-setup/contents+en.lrpage.body) @@ -438,7 +439,7 @@ msgstr "# 带宽与连接" msgid "" "A non-exit relay should be able to handle at least 7000 concurrent " "connections." -msgstr "" +msgstr "一个非出口节点应能同时处理至少7000个连接。" #: https//community.torproject.org/relay/relays-requirements/ #: (content/relay-operations/relays-requirements/contents+en.lrpage.body) @@ -459,7 +460,7 @@ msgstr "" msgid "" "Fast exit relays (>=100 Mbit/s) usually have to handle a lot more concurrent" " connections (>100k)." -msgstr "" +msgstr "高速出口节点（≥100 Mb/s）通常要处理大许多的同时连接数（>100k）。" #: https//community.torproject.org/relay/relays-requirements/ #: (content/relay-operations/relays-requirements/contents+en.lrpage.body) @@ -486,12 +487,12 @@ msgstr "" msgid "" "If you do not know your bandwidth you can use http://beta.speedtest.net to " "measure it." -msgstr "" +msgstr "如果你不知道你的带宽你可以通过http://beta.speedtest.net测试。" #: https//community.torproject.org/relay/relays-requirements/ #: (content/relay-operations/relays-requirements/contents+en.lrpage.body) msgid "# Monthly Outbound Traffic" -msgstr "" +msgstr "# 每月出站流量" #: https//community.torproject.org/relay/relays-requirements/ #: (content/relay-operations/relays-requirements/contents+en.lrpage.body)

1 0

[onionperf/master] Enforce deterministic ordering in file walk.
by karsten＠torproject.org 12 May '20

12 May '20

commit 2ab97fc8082c03ff5f1c8d775f417bfee8936726 Author: Philipp Winter <phw(a)nymity.ch> Date: Fri May 8 10:20:15 2020 -0700 Enforce deterministic ordering in file walk. os.walk internally calls os.scandir, whose "entries are yielded in arbitrary order." This breaks (at least on my machine) two unit tests which rely on ordering: 1. test_reprocessing.test_log_collection_tgen 2. test_reprocessing.test_log_collection_torctl This patch fixes the problem by calling sorted on the arbitrarily ordered file names. --- onionperf/reprocessing.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/onionperf/reprocessing.py b/onionperf/reprocessing.py index c5351d4..7acf539 100644 --- a/onionperf/reprocessing.py +++ b/onionperf/reprocessing.py @@ -13,7 +13,7 @@ import sys def collect_logs(dirpath, pattern): logs = [] for root, dirnames, filenames in os.walk(dirpath): - for filename in fnmatch.filter(filenames, pattern): + for filename in fnmatch.filter(sorted(filenames), pattern): logs.append(os.path.join(root, filename)) return logs

1 0

[onionperf/master] Port OnionPerf to Python 3.
by karsten＠torproject.org 12 May '20

12 May '20

commit e5f15ea4328e1e0588e249314d002d7dbb647d07 Author: Karsten Loesing <karsten.loesing(a)gmx.net> Date: Tue May 5 16:21:43 2020 +0200 Port OnionPerf to Python 3. Fixes #29367. --- onionperf/analysis.py | 8 +++--- onionperf/docs/conf.py | 20 +++++++-------- onionperf/measurement.py | 20 ++++++++------- onionperf/model.py | 10 +++----- onionperf/onionperf | 4 +-- onionperf/tests/test_reprocessing.py | 4 +-- onionperf/tests/test_utils.py | 8 +++--- onionperf/util.py | 47 +++++++++--------------------------- onionperf/visualization.py | 8 +++--- run_tests.sh | 2 +- setup.py | 2 +- 11 files changed, 52 insertions(+), 81 deletions(-) diff --git a/onionperf/analysis.py b/onionperf/analysis.py index e90d005..82db3c8 100644 --- a/onionperf/analysis.py +++ b/onionperf/analysis.py @@ -17,7 +17,7 @@ from stem.response.events import CircuitEvent, CircMinorEvent, StreamEvent, Band from stem.response import ControlMessage, convert # onionperf imports -import util +from . import util ERRORS = { 'AUTH' : 'TGEN/AUTH', 'READ' : 'TGEN/READ', @@ -286,7 +286,7 @@ class Analysis(object): output.write("@type torperf 1.1\r\n") output_str = ' '.join("{0}={1}".format(k, d[k]) for k in sorted(d.keys()) if d[k] is not None).strip() output.write("{0}\r\n".format(output_str)) - except KeyError, e: + except KeyError as e: logging.warning("KeyError while exporting torperf file, missing key '{0}', skipping transfer '{1}'".format(str(e), xfer_db['transfer_id'])) continue @@ -431,8 +431,7 @@ class Transfer(object): d['elapsed_seconds']['payload_progress'] = {decile: self.payload_progress[decile] - e.unix_ts_start for decile in self.payload_progress if self.payload_progress[decile] is not None} return d -class Parser(object): - __metaclass__ = ABCMeta +class Parser(object, metaclass=ABCMeta): @abstractmethod def parse(self, source, do_simple): pass @@ -837,7 +836,6 @@ class TorCtlParser(Parser): except: continue source.close() - print len(self.streams), len(self.circuits) def get_data(self): return {'circuits': self.circuits, 'circuits_summary': self.circuits_summary, diff --git a/onionperf/docs/conf.py b/onionperf/docs/conf.py index f3d2b34..6344c90 100644 --- a/onionperf/docs/conf.py +++ b/onionperf/docs/conf.py @@ -19,14 +19,14 @@ sys.path.insert(0, os.path.abspath('..')) # -- Project information ----------------------------------------------------- -project = u'onionperf' -copyright = u'2019, Ana Custura' -author = u'Ana Custura' +project = 'onionperf' +copyright = '2019, Ana Custura' +author = 'Ana Custura' # The short X.Y version -version = u'' +version = '' # The full version, including alpha/beta/rc tags -release = u'' +release = '' # -- General configuration --------------------------------------------------- @@ -66,7 +66,7 @@ language = None # List of patterns, relative to source directory, that match files and # directories to ignore when looking for source files. # This pattern also affects html_static_path and html_extra_path . -exclude_patterns = [u'_build', 'Thumbs.db', '.DS_Store'] +exclude_patterns = ['_build', 'Thumbs.db', '.DS_Store'] # The name of the Pygments (syntax highlighting) style to use. pygments_style = 'sphinx' @@ -131,8 +131,8 @@ latex_elements = { # (source start file, target name, title, # author, documentclass [howto, manual, or own class]). latex_documents = [ - (master_doc, 'onionperf.tex', u'onionperf Documentation', - u'Ana Custura', 'manual'), + (master_doc, 'onionperf.tex', 'onionperf Documentation', + 'Ana Custura', 'manual'), ] @@ -141,7 +141,7 @@ latex_documents = [ # One entry per manual page. List of tuples # (source start file, name, description, authors, manual section). man_pages = [ - (master_doc, 'onionperf', u'onionperf Documentation', + (master_doc, 'onionperf', 'onionperf Documentation', [author], 1) ] @@ -152,7 +152,7 @@ man_pages = [ # (source start file, target name, title, author, # dir menu entry, description, category) texinfo_documents = [ - (master_doc, 'onionperf', u'onionperf Documentation', + (master_doc, 'onionperf', 'onionperf Documentation', author, 'onionperf', 'One line description of project.', 'Miscellaneous'), ] diff --git a/onionperf/measurement.py b/onionperf/measurement.py index 467ff61..91952a5 100644 --- a/onionperf/measurement.py +++ b/onionperf/measurement.py @@ -4,7 +4,7 @@ See LICENSE for licensing information ''' -import os, traceback, subprocess, threading, Queue, logging, time, datetime, re, shlex +import os, traceback, subprocess, threading, queue, logging, time, datetime, re, shlex from lxml import etree # stem imports @@ -14,7 +14,7 @@ from stem.version import Version, Requirement, get_system_tor_version from stem import __version__ as stem_version # onionperf imports -import analysis, monitor, model, util +from . import analysis, monitor, model, util def generate_docroot_index(docroot_path): root = etree.Element("files") @@ -22,7 +22,7 @@ def generate_docroot_index(docroot_path): for filename in filepaths: e = etree.SubElement(root, "file") e.set("name", filename) - with open("{0}/index.xml".format(docroot_path), 'wb') as f: print >> f, etree.tostring(root, pretty_print=True, xml_declaration=True) + with open("{0}/index.xml".format(docroot_path), 'wt') as f: print(etree.tostring(root, pretty_print=True, xml_declaration=True), file=f) def readline_thread_task(instream, q): # wait for lines from stdout until the EOF @@ -49,7 +49,8 @@ def watchdog_thread_task(cmd, cwd, writable, done_ev, send_stdin, ready_search_s # wait for a string to appear in stdout if requested if ready_search_str is not None: boot_re = re.compile(ready_search_str) - for line in iter(subp.stdout.readline, b''): + for bytes in iter(subp.stdout.readline, b''): + line = bytes.decode('utf-8') writable.write(line) if boot_re.search(line): break # got it! @@ -59,7 +60,7 @@ def watchdog_thread_task(cmd, cwd, writable, done_ev, send_stdin, ready_search_s ready_ev.set() # a helper will block on stdout and return lines back to us in a queue - stdout_q = Queue.Queue() + stdout_q = queue.Queue() t = threading.Thread(target=readline_thread_task, args=(subp.stdout, stdout_q)) t.start() @@ -67,9 +68,9 @@ def watchdog_thread_task(cmd, cwd, writable, done_ev, send_stdin, ready_search_s # sure that the subprocess is still alive and the master doesn't want us to quit while subp.poll() is None and done_ev.is_set() is False: try: - line = stdout_q.get(True, 1) - writable.write(line) - except Queue.Empty: + bytes = stdout_q.get(True, 1) + writable.write(bytes.decode('utf-8')) + except queue.Empty: # the queue is empty and the get() timed out, recheck loop conditions continue @@ -100,7 +101,8 @@ def watchdog_thread_task(cmd, cwd, writable, done_ev, send_stdin, ready_search_s # helper thread is done, make sure we drain the remaining lines from the stdout queue while not stdout_q.empty(): - writable.write(stdout_q.get_nowait()) + bytes = stdout_q.get_nowait() + writable.write(bytes.decode('utf-8')) # if we have too many failures, exit the watchdog to propogate the error up if len(failure_times) > 10: break diff --git a/onionperf/model.py b/onionperf/model.py index 3c057c5..a5e0787 100644 --- a/onionperf/model.py +++ b/onionperf/model.py @@ -5,16 +5,14 @@ ''' from abc import ABCMeta, abstractmethod -from cStringIO import StringIO +from io import StringIO from networkx import read_graphml, write_graphml, DiGraph -class TGenModel(object): +class TGenModel(object, metaclass=ABCMeta): ''' an action-dependency graph model for Shadow's traffic generator ''' - __metaclass__ = ABCMeta - def dump_to_string(self): s = StringIO() write_graphml(self.graph, s) @@ -42,9 +40,7 @@ class TGenLoadableModel(TGenModel): model_instance = cls(graph) return model_instance -class GeneratableTGenModel(TGenModel): - - __metaclass__ = ABCMeta +class GeneratableTGenModel(TGenModel, metaclass=ABCMeta): @abstractmethod def generate(self): diff --git a/onionperf/onionperf b/onionperf/onionperf index 45ead3c..536d6e2 100755 --- a/onionperf/onionperf +++ b/onionperf/onionperf @@ -1,4 +1,4 @@ -#!/usr/bin/env python +#!/usr/bin/env python3 ''' OnionPerf @@ -479,7 +479,7 @@ def visualize(args): tor_viz = TorVisualization() for (path, label) in args.datasets: - nextformat = lfcycle.next() + nextformat = next(lfcycle) anal = Analysis.load(filename=path) if anal is not None: diff --git a/onionperf/tests/test_reprocessing.py b/onionperf/tests/test_reprocessing.py index 76d5b4b..efacc5f 100644 --- a/onionperf/tests/test_reprocessing.py +++ b/onionperf/tests/test_reprocessing.py @@ -45,7 +45,7 @@ def test_log_match_no_log_date(): def test_log_match_with_filter_date(): tgen_logs = reprocessing.collect_logs(DATA_DIR, '*tgen.log') torctl_logs = reprocessing.collect_logs(DATA_DIR, '*torctl.log') - test_date = datetime.date(2019, 01, 10) + test_date = datetime.date(2019, 1, 10) log_pairs = reprocessing.match(tgen_logs, torctl_logs, test_date) well_known_list = [(DATA_DIR + 'logs/onionperf_2019-01-10_23:59:59.tgen.log', DATA_DIR + 'logs/onionperf_2019-01-10_23:59:59.torctl.log', datetime.datetime(2019, 1, 10, 0, 0))] assert_equals(log_pairs, well_known_list) @@ -53,7 +53,7 @@ def test_log_match_with_filter_date(): def test_log_match_with_wrong_filter_date(): tgen_logs = reprocessing.collect_logs(DATA_DIR, '*tgen.log') torctl_logs = reprocessing.collect_logs(DATA_DIR, '*torctl.log') - test_date = datetime.date(2017, 01, 01) + test_date = datetime.date(2017, 1, 1) log_pairs = reprocessing.match(tgen_logs, torctl_logs, test_date) well_known_list = [] assert_equals(log_pairs, well_known_list) diff --git a/onionperf/tests/test_utils.py b/onionperf/tests/test_utils.py index 71056ff..497c97e 100644 --- a/onionperf/tests/test_utils.py +++ b/onionperf/tests/test_utils.py @@ -95,7 +95,7 @@ def test_find_path_with_which(): """ temp_file = tempfile.NamedTemporaryFile() - os.chmod(temp_file.name, 0775) + os.chmod(temp_file.name, 0o775) work_path = util.find_path(None, temp_file.name, tempfile.tempdir) assert_equals(work_path, temp_file.name) temp_file.close() @@ -252,7 +252,7 @@ def test_file_writable(): test_writable.write("onionperf") test_writable.close() expected_checksum = "5001ed4ab25b52543946fa63da829d4eeab1bd254c89ffdad0877186e074b385" - with open(temp_file.name) as f: + with open(temp_file.name, 'rb') as f: file_bytes = f.read() file_checksum = hashlib.sha256(file_bytes).hexdigest() assert_equals(file_checksum, expected_checksum) @@ -270,8 +270,8 @@ def test_file_writable_compressed(): test_writable = util.FileWritable(temp_file.name, True) test_writable.write("onionperf") test_writable.close() - expected_checksum = "66a6256bc4b04529c7123fa9573d30de659ffaa0cce1cc9b189817c8bf30e813" - with open(temp_file.name) as f: + expected_checksum = "3556b3bee6bb56d0a42676cbbf5784ebe4151fe65b0797f42260f93212e2df11" + with open(temp_file.name, 'rb') as f: file_bytes = f.read() file_checksum = hashlib.sha256(file_bytes).hexdigest() assert_equals(file_checksum, expected_checksum) diff --git a/onionperf/util.py b/onionperf/util.py index d481150..7c8e80f 100644 --- a/onionperf/util.py +++ b/onionperf/util.py @@ -4,10 +4,9 @@ See LICENSE for licensing information ''' -import sys, os, socket, logging, random, re, shutil, datetime, urllib, gzip -from subprocess import Popen, PIPE, STDOUT +import sys, os, socket, logging, random, re, shutil, datetime, urllib.request, urllib.parse, urllib.error, gzip, lzma from threading import Lock -from cStringIO import StringIO +from io import StringIO from abc import ABCMeta, abstractmethod LINEFORMATS = "k-,r-,b-,g-,c-,m-,y-,k--,r--,b--,g--,c--,m--,y--,k:,r:,b:,g:,c:,m:,y:,k-.,r-.,b-.,g-.,c-.,m-.,y-." @@ -156,7 +155,7 @@ def get_ip_address(): """ ip_address = None try: - data = urllib.urlopen('https://check.torproject.org/').read() + data = urllib.request.urlopen('https://check.torproject.org/').read().decode('utf-8') ip_address = find_ip_address_url(data) if not ip_address: logging.error( @@ -195,18 +194,14 @@ class DataSource(object): self.filename = filename self.compress = compress self.source = None - self.xzproc = None def __iter__(self): if self.source is None: self.open() return self.source - def next(self): - return self.__next__() - - def __next__(self): # python 3 - return self.source.next() if self.source is not None else None + def __next__(self): + return next(self.source) if self.source is not None else None def open(self): if self.source is None: @@ -214,14 +209,12 @@ class DataSource(object): self.source = sys.stdin elif self.compress or self.filename.endswith(".xz"): self.compress = True - cmd = "xz --decompress --stdout {0}".format(self.filename) - xzproc = Popen(cmd.split(), stdout=PIPE) - self.source = xzproc.stdout + self.source = lzma.open(self.filename, mode='rt') elif self.filename.endswith(".gz"): self.compress = True - self.source = gzip.open(self.filename, 'rb') + self.source = gzip.open(self.filename, 'rt') else: - self.source = open(self.filename, 'r') + self.source = open(self.filename, 'rt') def get_file_handle(self): if self.source is None: @@ -230,12 +223,9 @@ class DataSource(object): def close(self): if self.source is not None: self.source.close() - if self.xzproc is not None: self.xzproc.wait() - -class Writable(object): - __metaclass__ = ABCMeta +class Writable(object, metaclass=ABCMeta): @abstractmethod def write(self, msg): pass @@ -251,8 +241,6 @@ class FileWritable(Writable): self.do_compress = do_compress self.do_truncate = do_truncate self.file = None - self.xzproc = None - self.ddproc = None self.lock = Lock() if self.filename == '-': @@ -275,14 +263,9 @@ class FileWritable(Writable): def __open_nolock(self): if self.do_compress: - self.xzproc = Popen("xz --threads=3 -".split(), stdin=PIPE, stdout=PIPE) - dd_cmd = "dd of={0}".format(self.filename) - # # note: its probably not a good idea to append to finalized compressed files - # if not self.do_truncate: dd_cmd += " oflag=append conv=notrunc" - self.ddproc = Popen(dd_cmd.split(), stdin=self.xzproc.stdout, stdout=open(os.devnull, 'w'), stderr=STDOUT) - self.file = self.xzproc.stdin + self.file = lzma.open(self.filename, mode='wt') else: - self.file = open(self.filename, 'w' if self.do_truncate else 'a', 0) + self.file = open(self.filename, 'wt' if self.do_truncate else 'at', 1) def close(self): self.lock.acquire() @@ -293,12 +276,6 @@ class FileWritable(Writable): if self.file is not None: self.file.close() self.file = None - if self.xzproc is not None: - self.xzproc.wait() - self.xzproc = None - if self.ddproc is not None: - self.ddproc.wait() - self.ddproc = None def rotate_file(self, filename_datetime=datetime.datetime.now()): self.lock.acquire() @@ -316,7 +293,7 @@ class FileWritable(Writable): self.__close_nolock() with open(self.filename, 'rb') as f_in, gzip.open(new_filename, 'wb') as f_out: shutil.copyfileobj(f_in, f_out) - with open(self.filename, 'a') as f_in: + with open(self.filename, 'ab') as f_in: f_in.truncate(0) self.__open_nolock() diff --git a/onionperf/visualization.py b/onionperf/visualization.py index c673617..a5dde54 100644 --- a/onionperf/visualization.py +++ b/onionperf/visualization.py @@ -46,9 +46,7 @@ pylab.rcParams.update({ }) ''' -class Visualization(object): - - __metaclass__ = ABCMeta +class Visualization(object, metaclass=ABCMeta): def __init__(self): self.datasets = [] @@ -349,7 +347,7 @@ class TGenVisualization(Visualization): if client not in dls[bytes]: dls[bytes][client] = 0 for sec in d["time_to_last_byte"][b]: dls[bytes][client] += len(d["time_to_last_byte"][b][sec]) for bytes in dls: - x, y = getcdf(dls[bytes].values(), shownpercentile=1.0) + x, y = getcdf(list(dls[bytes].values()), shownpercentile=1.0) pylab.figure(figs[bytes].number) pylab.plot(x, y, lineformat, label=label) @@ -555,7 +553,7 @@ def getcdf(data, shownpercentile=0.99, maxpoints=10000.0): frac = cf(data) k = len(data) / maxpoints x, y, lasty = [], [], 0.0 - for i in xrange(int(round(len(data) * shownpercentile))): + for i in range(int(round(len(data) * shownpercentile))): if i % k > 1.0: continue assert not numpy.isnan(data[i]) x.append(data[i]) diff --git a/run_tests.sh b/run_tests.sh index d45032c..7a30e61 100755 --- a/run_tests.sh +++ b/run_tests.sh @@ -1,3 +1,3 @@ #!/bin/sh -PYTHONPATH=. python -m nose --with-coverage --cover-package=onionperf +PYTHONPATH=. python3 -m nose --with-coverage --cover-package=onionperf diff --git a/setup.py b/setup.py index 07e46a6..41bba2f 100644 --- a/setup.py +++ b/setup.py @@ -1,4 +1,4 @@ -#!/usr/bin/env python +#!/usr/bin/env python3 from distutils.core import setup

1 0

[onionperf/master] Turn dict_keys and dict_values into lists.
by karsten＠torproject.org 12 May '20

12 May '20

commit 16556497d431f032e789294fa55191637e5a23aa Author: Philipp Winter <phw(a)nymity.ch> Date: Fri May 8 10:37:10 2020 -0700 Turn dict_keys and dict_values into lists. The tool 2to3 discovered these issues. --- onionperf/analysis.py | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/onionperf/analysis.py b/onionperf/analysis.py index 82db3c8..041d8c3 100644 --- a/onionperf/analysis.py +++ b/onionperf/analysis.py @@ -50,7 +50,7 @@ class Analysis(object): self.torctl_filepaths.append(filepath) def get_nodes(self): - return self.json_db['data'].keys() + return list(self.json_db['data'].keys()) def get_tor_bandwidth_summary(self, node, direction): try: @@ -160,13 +160,13 @@ class Analysis(object): continue xfers_by_filesize = {} - for xfer_db in self.json_db['data'][nickname]['tgen']['transfers'].values(): + for xfer_db in list(self.json_db['data'][nickname]['tgen']['transfers'].values()): xfers_by_filesize.setdefault(xfer_db['filesize_bytes'], []).append(xfer_db) streams_by_srcport, circuits = {}, [] if 'tor' in self.json_db['data'][nickname]: if 'streams' in self.json_db['data'][nickname]['tor']: - for streams_db in self.json_db['data'][nickname]['tor']['streams'].values(): + for streams_db in list(self.json_db['data'][nickname]['tor']['streams'].values()): if 'source' in streams_db: srcport = int(streams_db['source'].split(':')[1]) streams_by_srcport[srcport] = streams_db @@ -264,9 +264,9 @@ class Analysis(object): srcport = int(xfer_db['endpoint_local'].split(':')[2]) if srcport in streams_by_srcport: stream_db = streams_by_srcport[srcport] - if 'failure_reason_local' in stream_db.keys(): + if 'failure_reason_local' in list(stream_db.keys()): d['ERRORCODE'] += '_' + stream_db['failure_reason_local'] - if 'failure_reason_remote' in stream_db.keys(): + if 'failure_reason_remote' in list(stream_db.keys()): d['ERRORCODE'] += '_' + stream_db['failure_reason_remote'] circid = int(stream_db['circuit_id'] or 0) if circid in circuits: @@ -280,7 +280,7 @@ class Analysis(object): d['CIRC_ID'] = circid d['USED_AT'] = stream_db['unix_ts_end'] d['USED_BY'] = int(stream_db['stream_id']) - if 'ERRORCODE' in d.keys(): + if 'ERRORCODE' in list(d.keys()): d['ERRORCODE'] = ERRORS[d['ERRORCODE']] output.write("@type torperf 1.1\r\n")

1 0

[onionperf/master] Make installation instructions work with Python 3.
by karsten＠torproject.org 12 May '20

12 May '20

commit 0f43d7904c1690561667d7833f42e00b3f367647 Author: Philipp Winter <phw(a)nymity.ch> Date: Fri May 8 11:02:45 2020 -0700 Make installation instructions work with Python 3. For the most part, this is as simple as adding a "3" to all occurrences of "python" with the single exception of revising how one creates a virtual environment in Python 3. --- README.md | 22 +++++++++++----------- 1 file changed, 11 insertions(+), 11 deletions(-) diff --git a/README.md b/README.md index 51a9982..d7ff81b 100644 --- a/README.md +++ b/README.md @@ -12,6 +12,7 @@ produce a json stats database and files that can feed into Torperf, and can later be used to visualize changes in Tor client performance over time. For more information, see https://git.torproject.org/onionperf + For a dockerized setup, see https://github.com/hiromipaw/onionperf-docker ## Quick deployment instructions @@ -19,7 +20,7 @@ For a dockerized setup, see https://github.com/hiromipaw/onionperf-docker These are the quick deployment instructions for the current Debian stable distribution. ``` -sudo apt install git cmake make build-essential gcc libigraph0-dev libglib2.0-dev python-dev libxml2-dev python-lxml python-networkx python-scipy python-matplotlib python-numpy libevent-dev libssl-dev python-stem tor +sudo apt install git cmake make build-essential gcc libigraph0-dev libglib2.0-dev python3-dev libxml2-dev python3-lxml python3-networkx python3-scipy python3-matplotlib python3-numpy libevent-dev libssl-dev python3-stem tor git clone https://github.com/shadow/tgen.git cd tgen @@ -31,8 +32,8 @@ sudo ln -s ~/tgen/build/tgen /usr/bin/tgen git clone https://github.com/torproject/onionperf cd onionperf -sudo python setup.py build -sudo python setup.py install +sudo python3 setup.py build +sudo python3 setup.py install ``` ## Step-by-step installation instructions @@ -50,13 +51,13 @@ cd onionperf + **Tor** (>= v0.2.7.3-rc): libevent, openssl + **TGen** (Shadow >= v1.11.1): cmake, glib2, igraph - + **OnionPerf**: python + + **OnionPerf**: python3 The easiest way to satisfy all system dependencies is to use a package manager. TGen is not currently packaged and needs to be built from source. Note we only provide support for the current Debian Stable distribution. ``` -sudo apt install cmake make build-essential gcc libigraph0-dev libglib2.0-dev python-dev +sudo apt install cmake make build-essential gcc libigraph0-dev libglib2.0-dev python3-dev ``` ### Install Python modules @@ -68,7 +69,7 @@ sudo apt install cmake make build-essential gcc libigraph0-dev libglib2.0-dev py The easiest way to satisfy all system dependencies is to use a package manager. ``` -apt install tor libxml2-dev python-lxml python-networkx python-scipy python-matplotlib python-numpy python-stem +apt install tor libxml2-dev python3-lxml python3-networkx python3-scipy python3-matplotlib python3-numpy python3-stem ``` @@ -86,15 +87,14 @@ You must first satisfy the system/library requirements of each of the python mod Note: pip installation is not recommended as software installed by pip is not verified. ``` -sudo apt-get install python-dev libxml2 libxml2-dev libxslt1.1 libxslt1-dev libpng12-0 libpng12-dev libfreetype6 libfreetype6-dev +sudo apt-get install python3-dev libxml2 libxml2-dev libxslt1.1 libxslt1-dev libpng12-0 libpng12-dev libfreetype6 libfreetype6-dev ``` It is recommended to use virtual environments to keep all of the dependencies self-contained and to avoid conflicts with your other python projects. ``` -pip install virtualenv -virtualenv --no-site-packages venv +python3 -m venv venv source venv/bin/activate pip install -r requirements.txt # installs all required python modules for all OnionPerf subcommands deactivate @@ -169,8 +169,8 @@ pip install -I . Otherwise: ``` -python setup.py build -python setup.py install +python3 setup.py build +python3 setup.py install ``` ### Run OnionPerf

1 0

[onionperf/master] Open Tor control log files with \r\n as newline.
by karsten＠torproject.org 12 May '20

12 May '20

commit f863ee46e82a7acdf670752664eb4df93de0a247 Author: Karsten Loesing <karsten.loesing(a)gmx.net> Date: Sat May 9 19:51:42 2020 +0200 Open Tor control log files with \r\n as newline. Turns out that Tor control log files use \r\n for newlines. When we open these files without providing a newline parameter, line endings are translated by Python 3 into '\n' before being returned to the us. If we do specify a newline parameter, line endings are returned to us untranslated. We need \r\n as newlines for Stem to recognize and parse control events. On the other hand TGen log files use \n as newline, which is why we need to make newlines configurable. --- onionperf/analysis.py | 2 +- onionperf/util.py | 8 ++++---- 2 files changed, 5 insertions(+), 5 deletions(-) diff --git a/onionperf/analysis.py b/onionperf/analysis.py index 041d8c3..8e2b7df 100644 --- a/onionperf/analysis.py +++ b/onionperf/analysis.py @@ -825,7 +825,7 @@ class TorCtlParser(Parser): def parse(self, source, do_simple=True): self.do_simple = do_simple - source.open() + source.open(newline='\r\n') for line in source: # ignore line parsing errors try: diff --git a/onionperf/util.py b/onionperf/util.py index 7c8e80f..cb9f065 100644 --- a/onionperf/util.py +++ b/onionperf/util.py @@ -203,18 +203,18 @@ class DataSource(object): def __next__(self): return next(self.source) if self.source is not None else None - def open(self): + def open(self, newline=None): if self.source is None: if self.filename == '-': self.source = sys.stdin elif self.compress or self.filename.endswith(".xz"): self.compress = True - self.source = lzma.open(self.filename, mode='rt') + self.source = lzma.open(self.filename, mode='rt', newline=newline) elif self.filename.endswith(".gz"): self.compress = True - self.source = gzip.open(self.filename, 'rt') + self.source = gzip.open(self.filename, 'rt', newline=newline) else: - self.source = open(self.filename, 'rt') + self.source = open(self.filename, 'rt', newline=newline) def get_file_handle(self): if self.source is None:

1 0