August 2018 - tor-commits - lists.torproject.org

[sbws/master] Stop deleting the latest.v3bw symlink. Instead, do an atomic rename.
by pastly＠torproject.org 09 Aug '18

09 Aug '18

commit 2b111356fb813379ab0b4a0dc705706766788ad3 Author: teor <teor2345(a)gmail.com> Date: Wed Jul 18 11:12:34 2018 +1000 Stop deleting the latest.v3bw symlink. Instead, do an atomic rename. Also: * document the location of latest.v3bw * document the atomic creation of latest.v3bw * explain how to atomically create latest.v3bw when transferring it to another host Closes tor trac 26740. --- CHANGELOG.md | 6 +++++- sbws/config.default.ini | 2 ++ sbws/core/generate.py | 13 +++++++------ sbws/lib/v3bwfile.py | 19 +++++++++++++------ 4 files changed, 27 insertions(+), 13 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 604811e..48a7d37 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -12,9 +12,13 @@ and this project adheres to [Semantic Versioning](http://semver.org/spec/v2.0.0. - Log line on start up with sbws version, platform info, and library versions (trac#26751) +### Fixed + +- Stop deleting the latest.v3bw symlink. Instead, do an atomic rename. + (#26740) + ### Changed -- Document at which times should v3bw files be generated (#26740) - Remove test data v3bw file and generate it from the same test. (#26736) ### Fixed diff --git a/sbws/config.default.ini b/sbws/config.default.ini index b21ba0b..09053a3 100644 --- a/sbws/config.default.ini +++ b/sbws/config.default.ini @@ -1,6 +1,8 @@ [paths] datadir = ${sbws_home}/datadir v3bw_dname = ${sbws_home}/v3bw +# The latest bandwidth file is atomically symlinked to +# V3BandwidthsFile ${v3bw_dname}/latest.v3bw v3bw_fname = ${v3bw_dname}/{}.v3bw started_filepath = ${sbws_home}/started_at log_dname = ${sbws_home}/log diff --git a/sbws/core/generate.py b/sbws/core/generate.py index f952fbc..481529c 100644 --- a/sbws/core/generate.py +++ b/sbws/core/generate.py @@ -12,12 +12,13 @@ log = logging.getLogger(__name__) def gen_parser(sub): d = 'Generate a v3bw file based on recent results. A v3bw file is the '\ 'file Tor directory authorities want to read and base their '\ - 'bandwidth votes on.' \ - 'This file should be generated every hour at any minute, except ' \ - 'between 45 and 55 minutes past the hour because Tor read this file '\ - ' at minute 50 and except ' \ - 'between 15 and 25 minutes past the hour because Tor read this file '\ - ' at minute 20 during a consensus failure.' + 'bandwidth votes on. '\ + 'To avoid inconsistent reads, configure tor with '\ + '"V3BandwidthsFile /path/to/latest.v3bw". '\ + '(latest.v3bw is an atomically created symlink in the same '\ + 'directory as output.) '\ + 'If the file is transferred to another host, it should be written to '\ + 'a temporary path, then renamed to the V3BandwidthsFile path.' p = sub.add_parser('generate', description=d, formatter_class=ArgumentDefaultsHelpFormatter) p.add_argument('--output', default=None, type=str, diff --git a/sbws/lib/v3bwfile.py b/sbws/lib/v3bwfile.py index c1dc890..1899e3b 100644 --- a/sbws/lib/v3bwfile.py +++ b/sbws/lib/v3bwfile.py @@ -413,17 +413,24 @@ class V3BWFile(object): log.info("Writing to stdout is not supported.") return log.info('Writing v3bw file to %s', output) + # To avoid inconsistent reads, the bandwidth data is written to an + # archive path, then atomically symlinked to 'latest.v3bw' out_dir = os.path.dirname(output) out_link = os.path.join(out_dir, 'latest.v3bw') - if os.path.exists(out_link): - log.debug('Deleting existing symlink before creating a new one.') - os.remove(out_link) + out_link_tmp = out_link + '.tmp' with DirectoryLock(out_dir): with open(output, 'wt') as fd: fd.write(str(self.header)) for line in self.bw_lines: fd.write(str(line)) output_basename = os.path.basename(output) - log.debug('Creating symlink from {} to {}.' - .format(output_basename, out_link)) - os.symlink(output_basename, out_link) + # To atomically symlink a file, we need to create a temporary link, + # then rename it to the final link name. (POSIX guarantees that + # rename is atomic.) + log.debug('Creating symlink {} -> {}.' + .format(out_link_tmp, output_basename)) + os.symlink(output_basename, out_link_tmp) + log.debug('Renaming symlink {} -> {} to {} -> {}.' + .format(out_link_tmp, output_basename, + out_link, output_basename)) + os.rename(out_link_tmp, out_link)

1 0

[sbws/master] Add docstring in generator_started_from_file()
by pastly＠torproject.org 09 Aug '18

09 Aug '18

commit 5e596b71e57138fff48d524dbec0f43c1e6eb614 Author: Matt Traudt <sirmatt(a)ksu.edu> Date: Mon Jul 23 13:59:17 2018 -0400 Add docstring in generator_started_from_file() --- sbws/lib/v3bwfile.py | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/sbws/lib/v3bwfile.py b/sbws/lib/v3bwfile.py index 2784c2f..54ec98a 100644 --- a/sbws/lib/v3bwfile.py +++ b/sbws/lib/v3bwfile.py @@ -190,6 +190,10 @@ class V3BWHeader(object): @staticmethod def generator_started_from_file(conf): + ''' + ISO formatted timestamp for the time when the scanner process most + recently started. + ''' state = State(conf['paths']['state_fname']) if 'scanner_started' in state: return state['scanner_started']

1 0

[sbws/master] Make two tests actually test something
by pastly＠torproject.org 09 Aug '18

09 Aug '18

commit 6008858afa1ebdfd803ecb4c316dfbdaa2edc8b2 Author: Matt Traudt <sirmatt(a)ksu.edu> Date: Mon Jul 23 13:30:55 2018 -0400 Make two tests actually test something --- tests/unit/util/test_state.py | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/tests/unit/util/test_state.py b/tests/unit/util/test_state.py index 5e140fe..dd23d1a 100644 --- a/tests/unit/util/test_state.py +++ b/tests/unit/util/test_state.py @@ -68,7 +68,7 @@ def test_state_del(tmpdir): attempt_keys = (15983, None, True, -1.2, [], {}, set()) for key in attempt_keys: try: - del state[False] + del state[key] except TypeError: pass else: @@ -116,5 +116,4 @@ def test_state_iter(tmpdir): d = {'a': 1, 'b': 2, 'c': 3, 'd': 4} for key in d: state[key] = d[key] - for key in state: - pass + assert set([key for key in state]) == set(d)

1 0

[sbws/master] Add docstring for State class
by pastly＠torproject.org 09 Aug '18

09 Aug '18

commit 42b80dce5c418654bd275c2f11d8fb25d237def8 Author: Matt Traudt <sirmatt(a)ksu.edu> Date: Mon Jul 23 13:59:34 2018 -0400 Add docstring for State class --- sbws/util/state.py | 37 +++++++++++++++++++++++++++++++++++++ 1 file changed, 37 insertions(+) diff --git a/sbws/util/state.py b/sbws/util/state.py index 85e223c..a13ee13 100644 --- a/sbws/util/state.py +++ b/sbws/util/state.py @@ -4,6 +4,43 @@ import json class State: + ''' + State allows one to atomically access and update a simple state file on + disk across threads and across processes. + + To put it blunty, due to limited developer time and his inability to + quickly find a way to safely access and update more complex data types + (namely, collections like list, set, and dict), you may only store simple + types of data as enumerated in _ALLOWED_TYPES. Keys must be strings. + + Data is stored as JSON on disk in the provided file file. + + >>> state = State('foo.state') + >>> # state == {} + + >>> state['linux'] = True + >>> # 'foo.state' now exists on disk with the JSON for {'linux': True} + + >>> # We read 'foo.state' from disk in order to get the most up-to-date + >>> # state info. Pretend another process has updated 'linux' to be + >>> # False + >>> state['linux'] + >>> # returns False + + >>> # Pretend another process has added the user's age to the state file. + >>> # As before, we read the state file from disk for the most + >>> # up-to-date info. + >>> state['age'] + >>> # Returns 14 + + >>> # We now set their name. We read the state file first, set the option, + >>> # and then write it out. + >>> state['name'] = 'John' + + >>> # We can do many of the same things with a State object as with a dict + >>> for key in state: print(key) + >>> # Prints 'linux', 'age', and 'name' + ''' _ALLOWED_TYPES = (int, float, str, bool, type(None)) def __init__(self, fname):

1 0

[sbws/master] Add manual pages to conf.py, update author(s)
by pastly＠torproject.org 09 Aug '18

09 Aug '18

commit 41bbca65013d553ed2cc451a574b6ed11db4318c Author: juga0 <juga(a)riseup.net> Date: Wed Jul 25 00:46:50 2018 +0000 Add manual pages to conf.py, update author(s) --- docs/source/conf.py | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-) diff --git a/docs/source/conf.py b/docs/source/conf.py index d750bbd..5e260d4 100644 --- a/docs/source/conf.py +++ b/docs/source/conf.py @@ -25,7 +25,7 @@ def find_version(): project = 'Simple Bandwidth Scanner' copyright = 'Public Domain' -author = 'Matt Traudt' +author = 'Matt Traudt [sirmatt at ksu dot edu], juga [juga at riseup dot net]' # The short X.Y version version = find_version() @@ -139,7 +139,7 @@ latex_elements = { # author, documentclass [howto, manual, or own class]). latex_documents = [ (master_doc, 'simple-bw-scanner.tex', 'simple-bw-scanner Documentation', - 'pastly', 'manual'), + [author], 'manual'), ] @@ -148,8 +148,9 @@ latex_documents = [ # One entry per manual page. List of tuples # (source start file, name, description, authors, manual section). man_pages = [ - (master_doc, 'simple-bw-scanner', 'simple-bw-scanner Documentation', - [author], 1) + ('man_sbws', 'sbws', 'Simple Bandwidth Scanner', [author], 1), + ('man_sbws.ini', 'sbws.ini', + 'Simple Bandwidth Scanner configuration', [author], 5) ]

1 0

[sbws/master] Add to the title the name of the man
by pastly＠torproject.org 09 Aug '18

09 Aug '18

commit 28e36d74c25845af8728cd03d14ffbeb90a22acd Author: juga0 <juga(a)riseup.net> Date: Wed Jul 25 01:03:55 2018 +0000 Add to the title the name of the man --- docs/source/man_sbws.ini.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/source/man_sbws.ini.rst b/docs/source/man_sbws.ini.rst index ff8848c..522f898 100644 --- a/docs/source/man_sbws.ini.rst +++ b/docs/source/man_sbws.ini.rst @@ -1,5 +1,5 @@ -Simple Bandwidth Scanner configuration file manual -=================================================== +Simple Bandwidth Scanner - SBWS.INI(5) +===================================== DESCRIPTION -----------

1 0

[sbws/master] Add sbws man
by pastly＠torproject.org 09 Aug '18

09 Aug '18

commit 3272f5921825d7eeefd6391a82a7f7665768c5d5 Author: juga0 <juga(a)riseup.net> Date: Wed Jul 25 00:47:41 2018 +0000 Add sbws man --- docs/source/man_sbws.rst | 70 ++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 70 insertions(+) diff --git a/docs/source/man_sbws.rst b/docs/source/man_sbws.rst new file mode 100644 index 0000000..c48dbf4 --- /dev/null +++ b/docs/source/man_sbws.rst @@ -0,0 +1,70 @@ +Simple Bandwidth Scanner +======================== + +SYNOPSIS +-------- + +sbws [**Optional arguments**] [**Positional arguments**] + +sbws [**-h**] [**--version**] [**--log-level** {**debug,info,warning,error,critical**}] [**-d** DIRECTORY] {**cleanup,scanner,generate,init,stats**} + +DESCRIPTION +----------- + +Tor bandwidth scanner that generates bandwidth measurements files to be read by +the Directory Authorities. + +OPTIONS +------- + +Positional arguments +~~~~~~~~~~~~~~~~~~~~ + +{**cleanup,scanner,generate,init,stats**} + +Optional arguments +~~~~~~~~~~~~~~~~~~ + +-h, --help + Show help message and exit. + +--version + sbws version + +--log-level {debug,info,warning,error,critical} + Override the sbws log level (default: None) + +-d DIRECTORY, --directory DIRECTORY + Path to the sbws home directory (default: $HOME/.sbws) + +EXAMPLES +-------- + +sbws -c ~/.sbwsrc scanner + Run the scanner using the configuration file in ~/.sbwsrc + +sbws --log-level debug generate + Generate v3bw file in the default v3bw directory (~/.sbws/v3bw) + +sbws cleanup + Cleanup datadir and v3bw files older than XX in the default v3bw directory (~/.sbws/) + +FILES +----- + +$HOME/.sbws.ini + Default location for the sbws user configuration file. + +$HOME/.sbws + Default sbws home, where it stores measurement data files, + bandwidth list files and tor process data. + +SEE ALSO +--------- + +**sbws.ini** (5), https://sbws.readthedocs.org. + +BUGS +---- + +Please report bugs at https://trac.torproject.org/. \ No newline at end of file

1 0

[sbws/master] Add sbws configuration man
by pastly＠torproject.org 09 Aug '18

09 Aug '18

commit e935af835d105374a0de08013a221a129a7f29b9 Author: juga0 <juga(a)riseup.net> Date: Wed Jul 25 00:48:26 2018 +0000 Add sbws configuration man --- docs/source/man_sbws.ini.rst | 160 +++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 160 insertions(+) diff --git a/docs/source/man_sbws.ini.rst b/docs/source/man_sbws.ini.rst new file mode 100644 index 0000000..ff8848c --- /dev/null +++ b/docs/source/man_sbws.ini.rst @@ -0,0 +1,160 @@ +Simple Bandwidth Scanner configuration file manual +=================================================== + +DESCRIPTION +----------- + +Tor bandwidth scanner configuration file. + +SECTIONS +--------- + +general + data_period = INT + Days into the past that measurements are considered valid. (Default: 5) + http_timeout = INT + Timeout in seconds to give to the python Requests library. (Default: 10) + circuit_timeout = INT + Timeout in seconds to create circuits. (Default: 10) + reset_bw_ipv4_changes = {on, off} + Whether or not to reset the bandwidth measurements when the relay's IP + address changes. If it changes, we only consider results for the relay that + we obtained while the relay was located at its most recent IP address. + (Default: on) + reset_bw_ipv6_changes = off + NOT implemented for IPv6. + +paths + sbws_home = STR + sbws home directory. (Default: ~/.sbws) + datadir = STR + Directory where sbws stores temporal bandwidth results files. + (Default: ~/.sbws/datadir) + v3bw_dname = STR + Directory where sbws stores the bandwidth list files. + These are the files to be read by the Tor Directory Authorities. + (Default: ~/.sbws/v3bw) + v3bw_fname = STR + File names of the bandwidth list files. + The latest bandwidth file is symlinked by ``latests.v3bw`` + started_filepath = STR + File path to store the timestamp when the scanner was last started. + (Default: ~/.sbws/started_at) + log_dname = STR + Directory where to store log files when logging to files is enabled. + (Default ~/.sbws/log) + +destinations + usability_test_interval = INT + How often to check if a destination is usable + +tor + datadir = STR + sbws' owned tor directory. (Default: ~/.sbws/tor) + control_socket = STR + sbws's owned tor control socket file. + (Default: $XDG_RUNTIME_DIR/sbws/control) + pid = STR + sbws's owned tor pid file. (Default: $XDG_RUNTIME_DIR/sbws/tor.pid) + log = STR + sbws's owned tor directory log files. (Default: ~/.sbws/tor/log) + extra_lines = + sbws's tor extra configuration. (Default: None) + +scanner + nickname = STR + A human-readable string with chars in a-zA-Z0-9 to identify the scanner. + (Default: IDidntEditTheSBWSConfig) + download_toofast = INT + Limits on what download times are too fast/slow/etc. (Default: 1) + download_min = INT + Limits on what download times are too fast/slow/etc. (Default: 5) + download_target = INT + Limits on what download times are too fast/slow/etc. (Default: 6) + download_max = INT + Limits on what download times are too fast/slow/etc. (Default: 10) + num_rtts = INT + How many RTT measurements to make. (Default: 10) + num_downloads = INT + Number of downloads with acceptable times we must have for a relay before + moving on. (Default: 5) + initial_read_request = INT + The number of bytes to initially request from the server. (Default: 16384) + measurement_threads = INT + How many measurements to make in parallel. (Default: 3) + min_download_size = INT + Minimum number of bytes we should ever try to download in a measurement. + (Default: 1) + max_download_size = INT + Maximum number of bytes we should ever try to download in a measurement. + (Default: 1073741824) 1073741824 == 1 GiB + +relayprioritizer + measure_authorities = {on, off} + Whether or not to measure authorities. (Default: off) + fraction_relays = FLOAT + The target fraction of best priority relays we would like to return. + 0.05 is 5%. In a 7000 relay network, 5% is 350 relays. (Default: 0.05) + min_relays = INT + The minimum number of best priority relays we are willing to return. + (Default: 50) + +cleanup + stale_days = INT + After this many days, compress data files. (Default: 10) + rotten_days = INT + After this many days, delete data files. (Default: 90) + +logging + to_file = {yes, no} + Whether or not to log to a rotating file the directory paths.log_dname. + (Default: no) + to_stdout = {yes, no} + Whether or not to log to stdout. (Default: yes) + to_syslog = {yes, no} + Whether or not to log to syslog. (Default: no) + to_file_max_bytes = INT + If logging to file, how large (in bytes) should the file be allowed to get + before rotating to a new one. 10485760 is 10 MiB. If zero or number of + backups is zero, never rotate the log file. (Default: 10485760) + to_file_num_backups = INT + If logging to file, how many backups to keep. If zero or max bytes is zero, + never rotate the log file. (Default: 50) + level = {debug, info, warning, error, critical} + Level to log at. (Default: info) + to_file_level = {debug, info, warning, error, critical} + Level to log at when using files. (Default: info) + to_stdout_level = {debug, info, warning, error, critical} + Level to log at when using stdout. (Default: info) + to_syslog_level = {debug, info, warning, error, critical} + Level to log at when using syslog. (Default: info) + format = STR + Format string to use when logging. + (Default: [%(asctime)s] [%(name)s] [%(levelname)s] %(message)s) + to_file_format = STR + Format string to use when logging to files. (Default: $format) + to_stdout_format = STR + Format string to use when logging to stdout. (Default: $format) + to_syslog_format = STR + Format string to use when logging to syslog. + (Default: %(module)s[%(process)s]: <%(levelname)s> %(message)s) + +FILES +----- + +$HOME/.sbws.ini + Default location for the sbws user configuration file. + +$HOME/.sbws + Default sbws home, where it stores measurement data files, + bandwidth list files and tor process data. + +SEE ALSO +--------- + +**sbws** (1), https://sbws.readthedocs.org. + +BUGS +---- + +Please report bugs at https://trac.torproject.org/. \ No newline at end of file

1 0

[sbws/master] Include man pages in docs too
by pastly＠torproject.org 09 Aug '18

09 Aug '18

commit ef8f0c87c1f110a932f7ccb940563d5b3bb1c760 Author: juga0 <juga(a)riseup.net> Date: Wed Jul 25 01:04:14 2018 +0000 Include man pages in docs too --- docs/source/index.rst | 2 ++ 1 file changed, 2 insertions(+) diff --git a/docs/source/index.rst b/docs/source/index.rst index 43eb373..b237897 100644 --- a/docs/source/index.rst +++ b/docs/source/index.rst @@ -18,6 +18,8 @@ Contents INSTALL DEPLOY README + man_sbws + man_sbws.ini CHANGELOG CONTRIBUTING AUTHORS

1 0

[sbws/master] Add to the title the name of the man
by pastly＠torproject.org 09 Aug '18

09 Aug '18

commit e0f52b67dc95a585734ade41c6d31cc57323e9a5 Author: juga0 <juga(a)riseup.net> Date: Wed Jul 25 01:03:50 2018 +0000 Add to the title the name of the man --- docs/source/man_sbws.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/source/man_sbws.rst b/docs/source/man_sbws.rst index c48dbf4..94bb9f2 100644 --- a/docs/source/man_sbws.rst +++ b/docs/source/man_sbws.rst @@ -1,5 +1,5 @@ -Simple Bandwidth Scanner -======================== +Simple Bandwidth Scanner - SBWS(1) +=================================== SYNOPSIS --------

1 0