tor-commits August 2018

tor-commits@lists.torproject.org

17 participants
2581 discussions

[sbws/master] Add forgotten destinations
by pastly＠torproject.org 09 Aug '18

09 Aug '18

commit 137cc27ba1dec5365c8926064594dc8bb34364d6 Author: juga0 <juga(a)riseup.net> Date: Wed Jul 25 01:19:22 2018 +0000 Add forgotten destinations which is actually the most important for the user --- docs/source/man_sbws.ini.rst | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-) diff --git a/docs/source/man_sbws.ini.rst b/docs/source/man_sbws.ini.rst index 522f898..b003bef 100644 --- a/docs/source/man_sbws.ini.rst +++ b/docs/source/man_sbws.ini.rst @@ -1,5 +1,5 @@ Simple Bandwidth Scanner - SBWS.INI(5) -===================================== +====================================== DESCRIPTION ----------- @@ -45,9 +45,17 @@ paths (Default ~/.sbws/log) destinations + STR = {on, off} + Name of destination. It is a name for the Web server from where to + download files in order to measure bandwidths. + usability_test_interval = INT How often to check if a destination is usable +destinations.STR + url = STR + The url to the destination + tor datadir = STR sbws' owned tor directory. (Default: ~/.sbws/tor)

1 0

[sbws/master] Merge branch 'state_file_02'
by pastly＠torproject.org 09 Aug '18

09 Aug '18

commit bddbf47a0fd9ab5aca7c6a257447f09716d95686 Merge: 137cc27 42b80dc Author: Matt Traudt <sirmatt(a)ksu.edu> Date: Wed Aug 1 18:06:32 2018 -0400 Merge branch 'state_file_02' CHANGELOG.md | 2 + README.md | 2 + docs/source/index.rst | 1 + docs/source/sbws.util.rst | 8 +++ docs/source/state.rst | 20 +++++++ sbws/config.default.ini | 2 +- sbws/core/scanner.py | 22 ++------ sbws/lib/v3bwfile.py | 34 ++++-------- sbws/util/config.py | 4 +- sbws/util/state.py | 99 +++++++++++++++++++++++++++++++++++ tests/unit/util/test_state.py | 119 ++++++++++++++++++++++++++++++++++++++++++ 11 files changed, 269 insertions(+), 44 deletions(-) diff --cc CHANGELOG.md index 910737c,9a5ceee..3cb31d4 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@@ -11,12 -11,8 +11,14 @@@ and this project adheres to [Semantic V - Log line on start up with sbws version, platform info, and library versions (trac#26751) +- Manual pages (#26926) + +### Fixed + +- Stop deleting the latest.v3bw symlink. Instead, do an atomic rename. + (#26740) + - State file for storing the last time `sbws scanner` was started, and able to + be used for storing many other types of state in the future. (GH#166) ### Changed diff --cc README.md index 49ca638,0fa6118..0b762a2 --- a/README.md +++ b/README.md @@@ -131,12 -131,13 +131,14 @@@ In this directory you will fin - `config.ini` The configuration file you should be editing if you want to modify sbws's behavior. -- `config.log.ini` The configuration file you should edit if you want to modify - how sbws logs. -- `datadir` Once your sbws scanner has started gathering results, it will dump +- `datadir/` Once your sbws scanner has started gathering results, it will dump them into this directory. Other sbws commands (such as generate and stats) read results from the files in this directory. +- `log/` If configured, this directory stores logs generated by all the sbws + commands in rotating log files. +- `v3bw/` This directory stores the v3bw files created with `sbws generate`. + - `state.dat` A file for storing state needed between sbws commands. See its + documentation for more information. ## Running tests diff --cc sbws/config.default.ini index 09053a3,121154d..036a987 --- a/sbws/config.default.ini +++ b/sbws/config.default.ini @@@ -1,10 -1,8 +1,10 @@@ [paths] datadir = ${sbws_home}/datadir v3bw_dname = ${sbws_home}/v3bw +# The latest bandwidth file is atomically symlinked to +# V3BandwidthsFile ${v3bw_dname}/latest.v3bw v3bw_fname = ${v3bw_dname}/{}.v3bw - started_filepath = ${sbws_home}/started_at + state_fname = ${sbws_home}/state.dat log_dname = ${sbws_home}/log [destinations]

1 0

[sbws/master] Add config times for cleaning v3bw files
by pastly＠torproject.org 09 Aug '18

09 Aug '18

commit ad6f1be916830cacb14355a31cbd153c6e387c28 Author: juga0 <juga(a)riseup.net> Date: Thu Jul 12 15:40:41 2018 +0000 Add config times for cleaning v3bw files --- sbws/config.default.ini | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/sbws/config.default.ini b/sbws/config.default.ini index 036a987..860e8db 100644 --- a/sbws/config.default.ini +++ b/sbws/config.default.ini @@ -14,6 +14,8 @@ usability_test_interval = 300 [general] # Days into the past that measurements are considered valid data_period = 5 +# Minutes into the past that the bandwidth list files need to be available (2h) +valid_mins_v3bw_files = 120 # Timeout in seconds to give to the python Requests library. It MUST be a # single float. Requests will use it both as the connect() timeout and the # timeout between bytes received from the server. See @@ -66,6 +68,11 @@ extra_lines = stale_days = 10 # After this many days, delete data files rotten_days = 90 +# After this many minutes, compress v3bw files +# Time during for the vote and time for the consensus (2h) +stale_mins_v3bw_files = 240 +# After this many minutes, delete v3bw files (1d) +rotten_mins_v3bw_files = 1440 [relayprioritizer] # Whether or not to measure authorities

1 0

[sbws/master] Add arg to clean v3bw files in tests
by pastly＠torproject.org 09 Aug '18

09 Aug '18

commit 30a191063d30ef7c072629e6bbe8985a4a8d804a Author: juga0 <juga(a)riseup.net> Date: Thu Jul 12 15:47:04 2018 +0000 Add arg to clean v3bw files in tests --- tests/unit/conftest.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tests/unit/conftest.py b/tests/unit/conftest.py index 4eab658..308eace 100644 --- a/tests/unit/conftest.py +++ b/tests/unit/conftest.py @@ -119,7 +119,7 @@ def sbwshome_only_datadir(sbwshome_empty): def args(sbwshome_empty, parser): """Args with sbws home in the tests tmp dir.""" args = _PseudoArguments(directory=sbwshome_empty, output=sbwshome_empty, - scale=False, log_level='debug') + scale=False, log_level='debug', v3bw=False) return args

1 0

[sbws/master] Refactor to clean v3bw files too
by pastly＠torproject.org 09 Aug '18

09 Aug '18

commit 9d6019f0921dcb4090e06d3489d0860d588c9f57 Author: juga0 <juga(a)riseup.net> Date: Thu Jul 12 15:46:14 2018 +0000 Refactor to clean v3bw files too --- sbws/core/cleanup.py | 87 +++++++++++++++++++++++++++++++++++++++++----------- 1 file changed, 69 insertions(+), 18 deletions(-) diff --git a/sbws/core/cleanup.py b/sbws/core/cleanup.py index 149cdd2..014b85d 100644 --- a/sbws/core/cleanup.py +++ b/sbws/core/cleanup.py @@ -10,6 +10,8 @@ import shutil import logging import time +from sbws.util.timestamp import unixts_to_dt_obj + log = logging.getLogger(__name__) @@ -27,17 +29,21 @@ def gen_parser(sub): p.add_argument('--dry-run', action='store_true', help='Don\'t actually compress or delete anything') + p.add_argument('--v3bw', action='store_true', help='Clean also v3bw files') + -def _get_older_files_than(dname, num_days_ago, extensions): +def _get_older_files_than(dname, time_delta, extensions, is_v3bw=False): assert os.path.isdir(dname) - assert isinstance(num_days_ago, int) + assert isinstance(time_delta, int) assert isinstance(extensions, list) for ext in extensions: assert isinstance(ext, str) assert ext[0] == '.' # Determine oldest allowed date today = datetime.utcfromtimestamp(time.time()) - oldest_day = today - timedelta(days=num_days_ago) + oldest_day = today - timedelta(days=time_delta) + if is_v3bw: + oldest = today - timedelta(minutes=time_delta) # Compile a regex that can extract a date from a file name that looks like # /path/to/foo/YYYY-MM-DD*.extension extensions = [re.escape(e) for e in extensions] @@ -50,38 +56,52 @@ def _get_older_files_than(dname, num_days_ago, extensions): for root, dirs, files in os.walk(dname): for f in files: fname = os.path.join(root, f) - match = regex.match(fname) - if not match: - log.debug('Ignoring %s because it doesn\'t look like ' - 'YYYY-MM-DD', fname) - continue - d = datetime(*[int(n) for n in match.group(1).split('-')]) - if d < oldest_day: - yield fname - - -def _remove_rotten_files(datadir, rotten_days, dry_run=True): + if is_v3bw: # or (v3bw_ext not in fname) + # not forcing files to have correct names just the extension + _, ext = os.path.splitext(fname) + if ext not in ['.v3bw']: + log.debug('Ignoring %s because it doesn\'t have extension ' + '%s', fname, ext) + continue + dt = unixts_to_dt_obj(os.path.getmtime(fname)) + if dt < oldest and os.path.splitext: + yield fname + else: + match = regex.match(fname) + if not match: + log.debug('Ignoring %s because it doesn\'t look like ' + 'YYYY-MM-DD', fname) + continue + d = datetime(*[int(n) for n in match.group(1).split('-')]) + if d < oldest_day: + yield fname + + +def _remove_rotten_files(datadir, rotten_days, dry_run=True, is_v3bw=False): assert os.path.isdir(datadir) assert isinstance(rotten_days, int) # Hold the lock for basically the entire time just in case someone else # moves files between when we get the list of files and when we try to # delete them. + exts = ['.txt', '.txt.gz'] if not is_v3bw else ['.v3bw'] with DirectoryLock(datadir): - for fname in _get_older_files_than(datadir, rotten_days, - ['.txt', '.txt.gz']): + for fname in _get_older_files_than(datadir, rotten_days, exts, + is_v3bw): log.info('Deleting %s', fname) if not dry_run: os.remove(fname) -def _compress_stale_files(datadir, stale_days, dry_run=True): +def _compress_stale_files(datadir, stale_days, dry_run=True, is_v3bw=False): assert os.path.isdir(datadir) assert isinstance(stale_days, int) # Hold the lock for basically the entire time just in case someone else # moves files between when we get the list of files and when we try to # compress them. + exts = ['.txt', '.txt.gz'] if not is_v3bw else ['.v3bw'] with DirectoryLock(datadir): - for fname in _get_older_files_than(datadir, stale_days, ['.txt']): + for fname in _get_older_files_than(datadir, stale_days, exts, + is_v3bw): log.info('Compressing %s', fname) if dry_run: continue @@ -92,6 +112,24 @@ def _compress_stale_files(datadir, stale_days, dry_run=True): os.remove(fname) +def _check_validity_periods(valid, stale, rotten): + if stale - 2 < valid: + fail_hard('For safetly, cleanup/stale_* (%d) must be at least 2 ' + 'days larger than general/data_period or general/valid_ * ' + '(%d)', stale, valid) + if rotten < stale: + fail_hard('cleanup/rotten_* (%d) must be the same or larger than ' + 'cleanup/stale_* (%d)', rotten, stale) + + if stale / 2 < valid: + log.warning( + 'cleanup/stale_ (%d) is less than twice ' + 'general/data_period or general/valid_*(%d). ' + 'For ease of parsing older results ' + 'if necessary, it is recommended to make stale at least ' + 'twice the data_period.', stale, valid) + + def main(args, conf): ''' Main entry point in to the cleanup command. @@ -126,3 +164,16 @@ def main(args, conf): _remove_rotten_files(datadir, rotten_days, dry_run=args.dry_run) _compress_stale_files(datadir, stale_days, dry_run=args.dry_run) + + if args.v3bw: + v3bw_dir = conf['paths']['v3bw_dname'] + if not os.path.isdir(datadir): + fail_hard('%s does not exist', v3bw_dir) + valid = conf.getint('general', 'valid_mins_v3bw_files') + stale = conf.getint('cleanup', 'stale_mins_v3bw_files') + rotten = conf.getint('cleanup', 'rotten_mins_v3bw_files') + _check_validity_periods(valid, stale, rotten) + _remove_rotten_files(v3bw_dir, rotten, dry_run=args.dry_run, + is_v3bw=True) + _compress_stale_files(v3bw_dir, stale, dry_run=args.dry_run, + is_v3bw=True)

1 0

[sbws/master] Add v3bw files cleanup config opts as valid
by pastly＠torproject.org 09 Aug '18

09 Aug '18

commit 1725c6cc5a9ae7cdbd9e5427ef0ea448fa670a24 Author: juga0 <juga(a)riseup.net> Date: Thu Jul 12 15:43:51 2018 +0000 Add v3bw files cleanup config opts as valid --- sbws/util/config.py | 3 +++ 1 file changed, 3 insertions(+) diff --git a/sbws/util/config.py b/sbws/util/config.py index 7ba530e..a1d5a15 100644 --- a/sbws/util/config.py +++ b/sbws/util/config.py @@ -174,6 +174,8 @@ def _validate_cleanup(conf): ints = { 'stale_days': {'minimum': 1, 'maximum': None}, 'rotten_days': {'minimum': 1, 'maximum': None}, + 'stale_mins_v3bw_files': {'minimum': 1, 'maximum': None}, + 'rotten_mins_v3bw_files': {'minimum': 1, 'maximum': None}, } all_valid_keys = list(ints.keys()) errors.extend(_validate_section_keys(conf, sec, all_valid_keys, err_tmpl)) @@ -187,6 +189,7 @@ def _validate_general(conf): err_tmpl = Template('$sec/$key ($val): $e') ints = { 'data_period': {'minimum': 1, 'maximum': None}, + 'valid_mins_v3bw_files': {'minimum': 1, 'maximum': None}, 'circuit_timeout': {'minimum': 1, 'maximum': None}, } floats = {

1 0

[sbws/master] Set sudo in travis
by pastly＠torproject.org 09 Aug '18

09 Aug '18

commit e8bc2d7185a32fe185152ee510f342c2f9c1eb17 Author: juga0 <juga(a)riseup.net> Date: Fri Jul 13 13:16:08 2018 +0000 Set sudo in travis --- .travis.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.travis.yml b/.travis.yml index 7b0e438..8d328d4 100644 --- a/.travis.yml +++ b/.travis.yml @@ -1,4 +1,4 @@ -sudo: false +sudo: true language: python python: - "3.4"

1 0

[sbws/master] Add entry to changelog about cleaning v3bw files
by pastly＠torproject.org 09 Aug '18

09 Aug '18

commit 311980cb1d048053b767b4589d6554937f308fd6 Author: juga0 <juga(a)riseup.net> Date: Thu Jul 12 21:38:36 2018 +0000 Add entry to changelog about cleaning v3bw files --- CHANGELOG.md | 1 + 1 file changed, 1 insertion(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index 3cb31d4..3c4f019 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -53,6 +53,7 @@ Summary of changes: Requires graphviz to be installed.(GHPR#226) - Add metadata to setup.py, useful for source/binary distributions. - Add possibility to log to system log. (#26683) +- Add option to cleanup v3bw files. (#26701) ### Fixed

1 0

[sbws/master] Revert "Refactor to clean v3bw files too"
by pastly＠torproject.org 09 Aug '18

09 Aug '18

commit 359f87a3dcec921f08af5a2c6b71a0913c8c0b2c Author: juga0 <juga(a)riseup.net> Date: Tue Jul 17 16:37:31 2018 +0000 Revert "Refactor to clean v3bw files too" This reverts commit 380461e56bd87bd2709abe591b6f4b11e1481e35. --- sbws/core/cleanup.py | 87 +++++++++++----------------------------------------- 1 file changed, 18 insertions(+), 69 deletions(-) diff --git a/sbws/core/cleanup.py b/sbws/core/cleanup.py index 014b85d..149cdd2 100644 --- a/sbws/core/cleanup.py +++ b/sbws/core/cleanup.py @@ -10,8 +10,6 @@ import shutil import logging import time -from sbws.util.timestamp import unixts_to_dt_obj - log = logging.getLogger(__name__) @@ -29,21 +27,17 @@ def gen_parser(sub): p.add_argument('--dry-run', action='store_true', help='Don\'t actually compress or delete anything') - p.add_argument('--v3bw', action='store_true', help='Clean also v3bw files') - -def _get_older_files_than(dname, time_delta, extensions, is_v3bw=False): +def _get_older_files_than(dname, num_days_ago, extensions): assert os.path.isdir(dname) - assert isinstance(time_delta, int) + assert isinstance(num_days_ago, int) assert isinstance(extensions, list) for ext in extensions: assert isinstance(ext, str) assert ext[0] == '.' # Determine oldest allowed date today = datetime.utcfromtimestamp(time.time()) - oldest_day = today - timedelta(days=time_delta) - if is_v3bw: - oldest = today - timedelta(minutes=time_delta) + oldest_day = today - timedelta(days=num_days_ago) # Compile a regex that can extract a date from a file name that looks like # /path/to/foo/YYYY-MM-DD*.extension extensions = [re.escape(e) for e in extensions] @@ -56,52 +50,38 @@ def _get_older_files_than(dname, time_delta, extensions, is_v3bw=False): for root, dirs, files in os.walk(dname): for f in files: fname = os.path.join(root, f) - if is_v3bw: # or (v3bw_ext not in fname) - # not forcing files to have correct names just the extension - _, ext = os.path.splitext(fname) - if ext not in ['.v3bw']: - log.debug('Ignoring %s because it doesn\'t have extension ' - '%s', fname, ext) - continue - dt = unixts_to_dt_obj(os.path.getmtime(fname)) - if dt < oldest and os.path.splitext: - yield fname - else: - match = regex.match(fname) - if not match: - log.debug('Ignoring %s because it doesn\'t look like ' - 'YYYY-MM-DD', fname) - continue - d = datetime(*[int(n) for n in match.group(1).split('-')]) - if d < oldest_day: - yield fname - - -def _remove_rotten_files(datadir, rotten_days, dry_run=True, is_v3bw=False): + match = regex.match(fname) + if not match: + log.debug('Ignoring %s because it doesn\'t look like ' + 'YYYY-MM-DD', fname) + continue + d = datetime(*[int(n) for n in match.group(1).split('-')]) + if d < oldest_day: + yield fname + + +def _remove_rotten_files(datadir, rotten_days, dry_run=True): assert os.path.isdir(datadir) assert isinstance(rotten_days, int) # Hold the lock for basically the entire time just in case someone else # moves files between when we get the list of files and when we try to # delete them. - exts = ['.txt', '.txt.gz'] if not is_v3bw else ['.v3bw'] with DirectoryLock(datadir): - for fname in _get_older_files_than(datadir, rotten_days, exts, - is_v3bw): + for fname in _get_older_files_than(datadir, rotten_days, + ['.txt', '.txt.gz']): log.info('Deleting %s', fname) if not dry_run: os.remove(fname) -def _compress_stale_files(datadir, stale_days, dry_run=True, is_v3bw=False): +def _compress_stale_files(datadir, stale_days, dry_run=True): assert os.path.isdir(datadir) assert isinstance(stale_days, int) # Hold the lock for basically the entire time just in case someone else # moves files between when we get the list of files and when we try to # compress them. - exts = ['.txt', '.txt.gz'] if not is_v3bw else ['.v3bw'] with DirectoryLock(datadir): - for fname in _get_older_files_than(datadir, stale_days, exts, - is_v3bw): + for fname in _get_older_files_than(datadir, stale_days, ['.txt']): log.info('Compressing %s', fname) if dry_run: continue @@ -112,24 +92,6 @@ def _compress_stale_files(datadir, stale_days, dry_run=True, is_v3bw=False): os.remove(fname) -def _check_validity_periods(valid, stale, rotten): - if stale - 2 < valid: - fail_hard('For safetly, cleanup/stale_* (%d) must be at least 2 ' - 'days larger than general/data_period or general/valid_ * ' - '(%d)', stale, valid) - if rotten < stale: - fail_hard('cleanup/rotten_* (%d) must be the same or larger than ' - 'cleanup/stale_* (%d)', rotten, stale) - - if stale / 2 < valid: - log.warning( - 'cleanup/stale_ (%d) is less than twice ' - 'general/data_period or general/valid_*(%d). ' - 'For ease of parsing older results ' - 'if necessary, it is recommended to make stale at least ' - 'twice the data_period.', stale, valid) - - def main(args, conf): ''' Main entry point in to the cleanup command. @@ -164,16 +126,3 @@ def main(args, conf): _remove_rotten_files(datadir, rotten_days, dry_run=args.dry_run) _compress_stale_files(datadir, stale_days, dry_run=args.dry_run) - - if args.v3bw: - v3bw_dir = conf['paths']['v3bw_dname'] - if not os.path.isdir(datadir): - fail_hard('%s does not exist', v3bw_dir) - valid = conf.getint('general', 'valid_mins_v3bw_files') - stale = conf.getint('cleanup', 'stale_mins_v3bw_files') - rotten = conf.getint('cleanup', 'rotten_mins_v3bw_files') - _check_validity_periods(valid, stale, rotten) - _remove_rotten_files(v3bw_dir, rotten, dry_run=args.dry_run, - is_v3bw=True) - _compress_stale_files(v3bw_dir, stale, dry_run=args.dry_run, - is_v3bw=True)

1 0

[sbws/master] Change description to include v3bw files
by pastly＠torproject.org 09 Aug '18

09 Aug '18

commit 2ecdcece94aff0654f1ee8d1acd9d6becf2bb42e Author: juga0 <juga(a)riseup.net> Date: Tue Jul 17 16:32:54 2018 +0000 Change description to include v3bw files and do not use concrete options since now there are several --- sbws/core/cleanup.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/sbws/core/cleanup.py b/sbws/core/cleanup.py index 149cdd2..bce90fb 100644 --- a/sbws/core/cleanup.py +++ b/sbws/core/cleanup.py @@ -20,8 +20,8 @@ def gen_parser(sub): :param argparse._SubParsersAction sub: what to add a sub-parser to ''' - d = 'Compress data files that are no longer fresh and delete data files '\ - 'that stopped being fresh a long time ago' + d = 'Compress and delete results and/or v3bw files old files.' \ + 'Configuration options are read to determine which are old files' p = sub.add_parser('cleanup', description=d, formatter_class=ArgumentDefaultsHelpFormatter) p.add_argument('--dry-run', action='store_true',

1 0

Jump to page:

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

tor-commits August 2018