September 2014 - tor-commits - lists.torproject.org

[arm/master] Rewriting the header panel
by atagar＠torproject.org 09 Sep '14

09 Sep '14

commit f799eb451a5671446a2a0f728108d50d158b5fe6 Merge: 4c5c20b 934cab1 Author: Damian Johnson <atagar(a)torproject.org> Date: Mon Sep 8 19:01:21 2014 -0700 Rewriting the header panel Honestly this isn't a true feature branch (many miscellaneous revisions have slipped in over time), but still it's a landmark. Done rewriting our first curses panel! before: https://gitweb.torproject.org/arm.git/blob/e249dc8f5c4c282326324161cf8421f9… after: https://gitweb.torproject.org/arm.git/blob/HEAD:/arm/header_panel.py arm/__init__.py | 13 +- arm/arguments.py | 6 +- arm/config/strings.cfg | 44 +- arm/config_panel.py | 46 +- arm/connections/__init__.py | 11 +- arm/connections/circ_entry.py | 14 +- arm/connections/conn_entry.py | 48 +- arm/connections/conn_panel.py | 6 +- arm/connections/count_popup.py | 8 +- arm/connections/descriptor_popup.py | 25 +- arm/controller.py | 7 +- arm/graphing/__init__.py | 7 +- arm/graphing/bandwidth_stats.py | 44 +- arm/graphing/conn_stats.py | 2 +- arm/graphing/graph_panel.py | 18 +- arm/graphing/resource_stats.py | 6 +- arm/header_panel.py | 828 ++++++++++++------------------- arm/log_panel.py | 46 +- arm/menu/__init__.py | 6 +- arm/menu/menu.py | 6 +- arm/popups.py | 2 +- arm/starter.py | 97 ++-- arm/torrc_panel.py | 42 +- arm/util/__init__.py | 166 +------ arm/util/log.py | 44 ++ arm/util/panel.py | 19 +- arm/util/tor_config.py | 20 +- arm/util/tracker.py | 56 ++- arm/util/ui_tools.py | 144 +----- armrc.sample | 4 - run_tests.py | 32 +- test/__init__.py | 7 + test/util/__init__.py | 5 + test/util/authenticate.py | 73 --- test/util/init_controller.py | 67 --- test/util/tracker/__init__.py | 10 + test/util/tracker/connection_tracker.py | 32 +- test/util/tracker/daemon.py | 6 +- test/util/tracker/resource_tracker.py | 18 +- 39 files changed, 773 insertions(+), 1262 deletions(-)

1 0

[arm/master] Dropping get_my_bandwidth_observed() and get_my_bandwidth_measured()
by atagar＠torproject.org 09 Sep '14

09 Sep '14

commit e2f1998c608a9fead6d2ce5260f1f721f98135bf Author: Damian Johnson <atagar(a)torproject.org> Date: Mon Sep 8 14:57:06 2014 -0700 Dropping get_my_bandwidth_observed() and get_my_bandwidth_measured() With the prior revisions these are now just two line helper functions. No reason to keep them separate any longer. --- arm/graphing/bandwidth_stats.py | 36 +++++++----------------------------- arm/header_panel.py | 2 +- 2 files changed, 8 insertions(+), 30 deletions(-) diff --git a/arm/graphing/bandwidth_stats.py b/arm/graphing/bandwidth_stats.py index 20898af..980583a 100644 --- a/arm/graphing/bandwidth_stats.py +++ b/arm/graphing/bandwidth_stats.py @@ -407,8 +407,13 @@ class BandwidthStats(graph_panel.GraphStats): stats = [] bw_rate = get_my_bandwidth_rate(controller) bw_burst = get_my_bandwidth_burst(controller) - bw_observed = get_my_bandwidth_observed(controller) - bw_measured = get_my_bandwidth_measured(controller) + + my_server_descriptor = controller.get_server_descriptor(default = None) + bw_observed = getattr(my_server_descriptor, 'observed_bandwidth', None) + + my_router_status_entry = controller.get_network_status(default = None) + bw_measured = getattr(my_router_status_entry, 'bandwidth', None) + label_in_bytes = CONFIG['features.graph.bw.transferInBytes'] if bw_rate and bw_burst: @@ -546,30 +551,3 @@ def get_my_bandwidth_burst(controller): return int(effective_burst) else: return None - - -def get_my_bandwidth_observed(controller): - """ - Provides the relay's current observed bandwidth (the throughput determined - from historical measurements on the client side). This is used in the - heuristic used for path selection if the measured bandwidth is undefined. - This is fetched from the descriptors and hence will get stale if - descriptors aren't periodically updated. - """ - - my_descriptor = controller.get_server_descriptor(default = None) - return getattr(my_descriptor, 'observed_bandwidth', None) - - -def get_my_bandwidth_measured(controller): - """ - Provides the relay's current measured bandwidth (the throughput as noted by - the directory authorities and used by clients for relay selection). This is - undefined if not in the consensus or with older versions of Tor. Depending - on the circumstances this can be from a variety of things (observed, - measured, weighted measured, etc) as described by: - https://trac.torproject.org/projects/tor/ticket/1566 - """ - - my_status_entry = controller.get_network_status(default = None) - return getattr(my_status_entry, 'bandwidth', None) diff --git a/arm/header_panel.py b/arm/header_panel.py index e0ba247..2d13feb 100644 --- a/arm/header_panel.py +++ b/arm/header_panel.py @@ -106,7 +106,7 @@ class HeaderPanel(panel.Panel, threading.Thread): controller.connect() try: - controller.authenticate() + controller.authenticate() # TODO: should account for our chroot except stem.connection.MissingPassword: password = cli.popups.input_prompt('Controller Password: ')

1 0

[arm/master] Moving bandwidth population strings to config
by atagar＠torproject.org 09 Sep '14

09 Sep '14

commit e6e70bb4c3229ef9a1cbe038cf3607c0641ab83b Author: Damian Johnson <atagar(a)torproject.org> Date: Mon Sep 8 13:26:48 2014 -0700 Moving bandwidth population strings to config Moving the strings out of the code, and making a new 'msg.panle.*' grouping for the panel content. --- arm/config/strings.cfg | 8 ++++++-- arm/graphing/bandwidth_stats.py | 24 +++++++++--------------- arm/header_panel.py | 6 +++--- 3 files changed, 18 insertions(+), 20 deletions(-) diff --git a/arm/config/strings.cfg b/arm/config/strings.cfg index f44e168..465e208 100644 --- a/arm/config/strings.cfg +++ b/arm/config/strings.cfg @@ -5,6 +5,7 @@ # * config parsing or handling configuration options # * debug concerns the --debug argument # * misc anything that doesn't fit into a present namespace +# * panel used after startup by our curses panels # * setup notificaitons or issues arising while starting arm # * tracker related to tracking resource usage or connections # * usage usage information about starting and running arm @@ -19,8 +20,11 @@ msg.config.nothing_loaded No armrc loaded, using defaults. You can customize arm msg.debug.saving_to_path Saving a debug log to {path}, please check it for sensitive information before sharing it. msg.debug.unable_to_write_file Unable to write to our debug log file ({path}): {error} -msg.misc.fd_used_at_sixty_percent Tor's file descriptor usage is at {percentage}%. -msg.misc.fd_used_at_ninety_percent Tor's file descriptor usage is at {percentage}%. If you run out Tor will be unable to continue functioning. +msg.panel.graphing.prepopulation_all_successful Read the last day of bandwidth history from the state file +msg.panel.graphing.prepopulation_successful Read the last day of bandwidth history from the state file ({duration} is missing) +msg.panel.graphing.prepopulation_failure Unable to prepopulate bandwidth information ({error}) +msg.panel.header.fd_used_at_sixty_percent Tor's file descriptor usage is at {percentage}%. +msg.panel.header.fd_used_at_ninety_percent Tor's file descriptor usage is at {percentage}%. If you run out Tor will be unable to continue functioning. msg.setup.arm_is_running_as_root Arm is currently running with root permissions. This isn't a good idea, nor should it be necessary. Try starting arm with "sudo -u {tor_user} arm" instead. msg.setup.chroot_doesnt_exist The chroot path set in your config ({path}) doesn't exist. diff --git a/arm/graphing/bandwidth_stats.py b/arm/graphing/bandwidth_stats.py index 2a21a0a..20898af 100644 --- a/arm/graphing/bandwidth_stats.py +++ b/arm/graphing/bandwidth_stats.py @@ -14,6 +14,8 @@ from arm.util import tor_controller from stem.control import State from stem.util import conf, log, str_tools, system +from arm.util import msg + def conf_handler(key, value): if key == 'features.graph.bw.accounting.rate': @@ -39,9 +41,6 @@ COLLAPSE_WIDTH = 135 ACCOUNTING_ARGS = ('status', 'reset_time', 'read', 'written', 'read_limit', 'writtenLimit') -PREPOPULATE_SUCCESS_MSG = 'Read the last day of bandwidth history from the state file' -PREPOPULATE_FAILURE_MSG = 'Unable to prepopulate bandwidth information (%s)' - class BandwidthStats(graph_panel.GraphStats): """ @@ -168,8 +167,7 @@ class BandwidthStats(graph_panel.GraphStats): # results associated with this tor instance if not uptime or '-' not in uptime: - msg = PREPOPULATE_FAILURE_MSG % 'insufficient uptime' - log.notice(msg) + log.notice(msg('panel.graphing.prepopulation_failure', error = 'insufficient uptime')) return False # get the user's data directory (usually '~/.tor') @@ -177,8 +175,7 @@ class BandwidthStats(graph_panel.GraphStats): data_dir = controller.get_conf('DataDirectory', None) if not data_dir: - msg = PREPOPULATE_FAILURE_MSG % 'data directory not found' - log.notice(msg) + log.notice(msg('panel.graphing.prepopulation_failure', error = 'data directory not found')) return False # attempt to open the state file @@ -186,8 +183,7 @@ class BandwidthStats(graph_panel.GraphStats): try: state_file = open('%s%s/state' % (CONFIG['tor.chroot'], data_dir), 'r') except IOError: - msg = PREPOPULATE_FAILURE_MSG % 'unable to read the state file' - log.notice(msg) + log.notice(msg('panel.graphing.prepopulation_failure', error = 'unable to read the state file')) return False # get the BWHistory entries (ordered oldest to newest) and number of @@ -227,8 +223,7 @@ class BandwidthStats(graph_panel.GraphStats): missing_write_entries = int((time.time() - last_write_time) / 900) if not bw_read_entries or not bw_write_entries or not last_read_time or not last_write_time: - msg = PREPOPULATE_FAILURE_MSG % 'bandwidth stats missing from state file' - log.notice(msg) + log.notice(msg('panel.graphing.prepopulation_failure', error = 'bandwidth stats missing from state file')) return False # fills missing entries with the last value @@ -272,13 +267,12 @@ class BandwidthStats(graph_panel.GraphStats): del self.primary_counts[interval_index][self.max_column + 1:] del self.secondary_counts[interval_index][self.max_column + 1:] - msg = PREPOPULATE_SUCCESS_MSG missing_sec = time.time() - min(last_read_time, last_write_time) if missing_sec: - msg += ' (%s is missing)' % str_tools.time_label(missing_sec, 0, True) - - log.notice(msg) + log.notice(msg('panel.graphing.prepopulation_successful', duration = str_tools.time_label(missing_sec, 0, True))) + else: + log.notice(msg('panel.graphing.prepopulation_all_successful')) return True diff --git a/arm/header_panel.py b/arm/header_panel.py index 49ec6dc..e0ba247 100644 --- a/arm/header_panel.py +++ b/arm/header_panel.py @@ -17,7 +17,7 @@ import stem from stem.control import Listener from stem.util import conf, log, proc, str_tools, system -from util import msg, tor_controller, panel, tracker +from arm.util import msg, tor_controller, panel, tracker MIN_DUAL_COL_WIDTH = 141 # minimum width where we'll show two columns SHOW_FD_THRESHOLD = 60 # show file descriptor usage if usage is over this percentage @@ -449,11 +449,11 @@ class Sampling(object): fd_percent = 100 * self.fd_used / self.fd_limit if fd_percent >= 90: - log_msg = msg('misc.fd_used_at_ninety_percent', percentage = fd_percent) + log_msg = msg('panel.header.fd_used_at_ninety_percent', percentage = fd_percent) log.log_once('fd_used_at_ninety_percent', log.WARN, log_msg) log.DEDUPLICATION_MESSAGE_IDS.add('fd_used_at_sixty_percent') elif fd_percent >= 60: - log_msg = msg('misc.fd_used_at_sixty_percent', percentage = fd_percent) + log_msg = msg('panel.header.fd_used_at_sixty_percent', percentage = fd_percent) log.log_once('fd_used_at_sixty_percent', log.NOTICE, log_msg) def format(self, message, crop_width = None):

1 0

[arm/master] Replacing the get_my_bandwidth_rate() and get_my_bandwidth_burst() helpers
by atagar＠torproject.org 09 Sep '14

09 Sep '14

commit 0b1e813815009797242ae2bad49fcbd043052503 Author: Damian Johnson <atagar(a)torproject.org> Date: Mon Sep 8 16:42:37 2014 -0700 Replacing the get_my_bandwidth_rate() and get_my_bandwidth_burst() helpers Couple legacy methods from our old torTools module. Replacing them with a simpler helper that does the trick. --- arm/graphing/bandwidth_stats.py | 54 +++++++++++---------------------------- 1 file changed, 15 insertions(+), 39 deletions(-) diff --git a/arm/graphing/bandwidth_stats.py b/arm/graphing/bandwidth_stats.py index 980583a..ce8b5b1 100644 --- a/arm/graphing/bandwidth_stats.py +++ b/arm/graphing/bandwidth_stats.py @@ -405,8 +405,8 @@ class BandwidthStats(graph_panel.GraphStats): if not self._title_stats or not my_fingerprint or (event and my_fingerprint in event.idlist): stats = [] - bw_rate = get_my_bandwidth_rate(controller) - bw_burst = get_my_bandwidth_burst(controller) + bw_rate = _min_config(controller, 'BandwidthRate', 'RelayBandwidthRate', 'MaxAdvertisedBandwidth') + bw_burst = _min_config(controller, 'BandwidthBurst', 'RelayBandwidthBurst') my_server_descriptor = controller.get_server_descriptor(default = None) bw_observed = getattr(my_server_descriptor, 'observed_bandwidth', None) @@ -507,47 +507,23 @@ class BandwidthStats(graph_panel.GraphStats): self.accounting_last_updated = time.time() -def get_my_bandwidth_rate(controller): +def _min_config(controller, *attributes): """ - Provides the effective relaying bandwidth rate of this relay. Currently - this doesn't account for SETCONF events. + Provides the minimum of the given numeric bandwidth rate or burst config + options. """ - # effective relayed bandwidth is the minimum of BandwidthRate, - # MaxAdvertisedBandwidth, and RelayBandwidthRate (if set) + value = None - effective_rate = controller.get_conf('BandwidthRate', None) - relay_rate = controller.get_conf('RelayBandwidthRate', None) - - if effective_rate and relay_rate and relay_rate != '0': - effective_rate = min(int(effective_rate), int(relay_rate)) - - max_advertised = controller.get_conf('MaxAdvertisedBandwidth', None) - - if max_advertised: - effective_rate = min(int(effective_rate), int(max_advertised)) - - if effective_rate is not None: - return int(effective_rate) - else: - return None - - -def get_my_bandwidth_burst(controller): - """ - Provides the effective bandwidth burst rate of this relay. Currently this - doesn't account for SETCONF events. - """ - - # effective burst (same for BandwidthBurst and RelayBandwidthBurst) + for attr in attributes: + try: + attr_value = int(controller.get_conf(attr)) - effective_burst = controller.get_conf('BandwidthBurst', None) - relay_burst = controller.get_conf('RelayBandwidthBurst', None) + if attr_value == 0 and attr.startswith('Relay'): + continue # RelayBandwidthRate and RelayBandwidthBurst default to zero - if effective_burst and relay_burst and relay_burst != '0': - effective_burst = min(int(effective_burst), int(relay_burst)) + value = min(value, attr_value) if value else attr_value + except: + pass - if effective_burst is not None: - return int(effective_burst) - else: - return None + return value

1 0

[arm/master] First pass revising bandwidth stats
by atagar＠torproject.org 09 Sep '14

09 Sep '14

commit 24a11ea8d518066a32933e5fb65dd0317bee99f1 Author: Damian Johnson <atagar(a)torproject.org> Date: Mon Sep 8 13:01:52 2014 -0700 First pass revising bandwidth stats Just made three changes on this pass... * Changed double quotes to single quotes. * Fixed a couple spots where we call GETCONF with a None default then immedately convert to an int (raising a TypeError if we ever failed the call). * Changed a couple spots where we got our own descriptor to be a lot simpler thanks to getattr(). --- arm/graphing/bandwidth_stats.py | 241 ++++++++++++++++++--------------------- 1 file changed, 111 insertions(+), 130 deletions(-) diff --git a/arm/graphing/bandwidth_stats.py b/arm/graphing/bandwidth_stats.py index dedde65..2a21a0a 100644 --- a/arm/graphing/bandwidth_stats.py +++ b/arm/graphing/bandwidth_stats.py @@ -16,19 +16,19 @@ from stem.util import conf, log, str_tools, system def conf_handler(key, value): - if key == "features.graph.bw.accounting.rate": + if key == 'features.graph.bw.accounting.rate': return max(1, value) -CONFIG = conf.config_dict("arm", { - "features.graph.bw.transferInBytes": False, - "features.graph.bw.accounting.show": True, - "features.graph.bw.accounting.rate": 10, - "features.graph.bw.accounting.isTimeLong": False, - "tor.chroot": "", +CONFIG = conf.config_dict('arm', { + 'features.graph.bw.transferInBytes': False, + 'features.graph.bw.accounting.show': True, + 'features.graph.bw.accounting.rate': 10, + 'features.graph.bw.accounting.isTimeLong': False, + 'tor.chroot': '', }, conf_handler) -DL_COLOR, UL_COLOR = "green", "cyan" +DL_COLOR, UL_COLOR = 'green', 'cyan' # width at which panel abandons placing optional stats (avg and total) with # header in favor of replacing the x-axis label @@ -37,10 +37,10 @@ COLLAPSE_WIDTH = 135 # valid keys for the accounting_info mapping -ACCOUNTING_ARGS = ("status", "reset_time", "read", "written", "read_limit", "writtenLimit") +ACCOUNTING_ARGS = ('status', 'reset_time', 'read', 'written', 'read_limit', 'writtenLimit') -PREPOPULATE_SUCCESS_MSG = "Read the last day of bandwidth history from the state file" -PREPOPULATE_FAILURE_MSG = "Unable to prepopulate bandwidth information (%s)" +PREPOPULATE_SUCCESS_MSG = 'Read the last day of bandwidth history from the state file' +PREPOPULATE_FAILURE_MSG = 'Unable to prepopulate bandwidth information (%s)' class BandwidthStats(graph_panel.GraphStats): @@ -60,7 +60,7 @@ class BandwidthStats(graph_panel.GraphStats): # accounting data (set by _update_accounting_info method) self.accounting_last_updated = 0 - self.accounting_info = dict([(arg, "") for arg in ACCOUNTING_ARGS]) + self.accounting_info = dict([(arg, '') for arg in ACCOUNTING_ARGS]) # listens for tor reload (sighup) events which can reset the bandwidth # rate/burst and if tor's using accounting @@ -83,12 +83,12 @@ class BandwidthStats(graph_panel.GraphStats): self.initial_primary_total = 0 self.initial_secondary_total = 0 - read_total = controller.get_info("traffic/read", None) + read_total = controller.get_info('traffic/read', None) if read_total and read_total.isdigit(): self.initial_primary_total = int(read_total) / 1024 # Bytes -> KB - write_total = controller.get_info("traffic/written", None) + write_total = controller.get_info('traffic/written', None) if write_total and write_total.isdigit(): self.initial_secondary_total = int(write_total) / 1024 # Bytes -> KB @@ -113,7 +113,7 @@ class BandwidthStats(graph_panel.GraphStats): self._title_stats = [] # force reset of title self.new_desc_event(None) # updates title params - if event_type in (State.INIT, State.RESET) and CONFIG["features.graph.bw.accounting.show"]: + if event_type in (State.INIT, State.RESET) and CONFIG['features.graph.bw.accounting.show']: is_accounting_enabled = controller.get_info('accounting/enabled', None) == '1' if is_accounting_enabled != self.is_accounting: @@ -139,9 +139,9 @@ class BandwidthStats(graph_panel.GraphStats): # checks that this is a relay (if ORPort is unset, then skip) controller = tor_controller() - or_port = controller.get_conf("ORPort", None) + or_port = controller.get_conf('ORPort', None) - if or_port == "0": + if or_port == '0': return # gets the uptime (using the same parameters as the header panel to take @@ -153,8 +153,8 @@ class BandwidthStats(graph_panel.GraphStats): query_pid = controller.get_pid(None) if query_pid: - query_param = ["%cpu", "rss", "%mem", "etime"] - query_cmd = "ps -p %s -o %s" % (query_pid, ",".join(query_param)) + query_param = ['%cpu', 'rss', '%mem', 'etime'] + query_cmd = 'ps -p %s -o %s' % (query_pid, ','.join(query_param)) ps_call = system.call(query_cmd, None) if ps_call and len(ps_call) == 2: @@ -167,26 +167,26 @@ class BandwidthStats(graph_panel.GraphStats): # the state tracks a day's worth of data and this should only prepopulate # results associated with this tor instance - if not uptime or "-" not in uptime: - msg = PREPOPULATE_FAILURE_MSG % "insufficient uptime" + if not uptime or '-' not in uptime: + msg = PREPOPULATE_FAILURE_MSG % 'insufficient uptime' log.notice(msg) return False # get the user's data directory (usually '~/.tor') - data_dir = controller.get_conf("DataDirectory", None) + data_dir = controller.get_conf('DataDirectory', None) if not data_dir: - msg = PREPOPULATE_FAILURE_MSG % "data directory not found" + msg = PREPOPULATE_FAILURE_MSG % 'data directory not found' log.notice(msg) return False # attempt to open the state file try: - state_file = open("%s%s/state" % (CONFIG['tor.chroot'], data_dir), "r") + state_file = open('%s%s/state' % (CONFIG['tor.chroot'], data_dir), 'r') except IOError: - msg = PREPOPULATE_FAILURE_MSG % "unable to read the state file" + msg = PREPOPULATE_FAILURE_MSG % 'unable to read the state file' log.notice(msg) return False @@ -209,25 +209,25 @@ class BandwidthStats(graph_panel.GraphStats): # counter for the current sampling period. Hence, offsets are added to # account for both. - if line.startswith("BWHistoryReadValues"): - bw_read_entries = line[20:].split(",") + if line.startswith('BWHistoryReadValues'): + bw_read_entries = line[20:].split(',') bw_read_entries = [int(entry) / 1024.0 / 900 for entry in bw_read_entries] bw_read_entries.pop() - elif line.startswith("BWHistoryWriteValues"): - bw_write_entries = line[21:].split(",") + elif line.startswith('BWHistoryWriteValues'): + bw_write_entries = line[21:].split(',') bw_write_entries = [int(entry) / 1024.0 / 900 for entry in bw_write_entries] bw_write_entries.pop() - elif line.startswith("BWHistoryReadEnds"): - last_read_time = time.mktime(time.strptime(line[18:], "%Y-%m-%d %H:%M:%S")) - tz_offset + elif line.startswith('BWHistoryReadEnds'): + last_read_time = time.mktime(time.strptime(line[18:], '%Y-%m-%d %H:%M:%S')) - tz_offset last_read_time -= 900 missing_read_entries = int((time.time() - last_read_time) / 900) - elif line.startswith("BWHistoryWriteEnds"): - last_write_time = time.mktime(time.strptime(line[19:], "%Y-%m-%d %H:%M:%S")) - tz_offset + elif line.startswith('BWHistoryWriteEnds'): + last_write_time = time.mktime(time.strptime(line[19:], '%Y-%m-%d %H:%M:%S')) - tz_offset last_write_time -= 900 missing_write_entries = int((time.time() - last_write_time) / 900) if not bw_read_entries or not bw_write_entries or not last_read_time or not last_write_time: - msg = PREPOPULATE_FAILURE_MSG % "bandwidth stats missing from state file" + msg = PREPOPULATE_FAILURE_MSG % 'bandwidth stats missing from state file' log.notice(msg) return False @@ -276,7 +276,7 @@ class BandwidthStats(graph_panel.GraphStats): missing_sec = time.time() - min(last_read_time, last_write_time) if missing_sec: - msg += " (%s is missing)" % str_tools.time_label(missing_sec, 0, True) + msg += ' (%s is missing)' % str_tools.time_label(missing_sec, 0, True) log.notice(msg) @@ -284,7 +284,7 @@ class BandwidthStats(graph_panel.GraphStats): def bandwidth_event(self, event): if self.is_accounting and self.is_next_tick_redraw(): - if time.time() - self.accounting_last_updated >= CONFIG["features.graph.bw.accounting.rate"]: + if time.time() - self.accounting_last_updated >= CONFIG['features.graph.bw.accounting.rate']: self._update_accounting_info() # scales units from B to KB for graphing @@ -301,11 +301,11 @@ class BandwidthStats(graph_panel.GraphStats): if width <= COLLAPSE_WIDTH: # clears line - panel.addstr(labeling_line, 0, " " * width) + panel.addstr(labeling_line, 0, ' ' * width) graph_column = min((width - 10) / 2, self.max_column) - primary_footer = "%s, %s" % (self._get_avg_label(True), self._get_total_label(True)) - secondary_footer = "%s, %s" % (self._get_avg_label(False), self._get_total_label(False)) + primary_footer = '%s, %s' % (self._get_avg_label(True), self._get_total_label(True)) + secondary_footer = '%s, %s' % (self._get_avg_label(False), self._get_total_label(False)) panel.addstr(labeling_line, 1, primary_footer, self.get_color(True)) panel.addstr(labeling_line, graph_column + 6, secondary_footer, self.get_color(False)) @@ -314,50 +314,50 @@ class BandwidthStats(graph_panel.GraphStats): if self.is_accounting: if tor_controller().is_alive(): - status = self.accounting_info["status"] + status = self.accounting_info['status'] - hibernate_color = "green" + hibernate_color = 'green' - if status == "soft": - hibernate_color = "yellow" - elif status == "hard": - hibernate_color = "red" - elif status == "": + if status == 'soft': + hibernate_color = 'yellow' + elif status == 'hard': + hibernate_color = 'red' + elif status == '': # failed to be queried - status, hibernate_color = "unknown", "red" + status, hibernate_color = 'unknown', 'red' - panel.addstr(labeling_line + 2, 0, "Accounting (", curses.A_BOLD) + panel.addstr(labeling_line + 2, 0, 'Accounting (', curses.A_BOLD) panel.addstr(labeling_line + 2, 12, status, curses.A_BOLD, hibernate_color) - panel.addstr(labeling_line + 2, 12 + len(status), ")", curses.A_BOLD) + panel.addstr(labeling_line + 2, 12 + len(status), ')', curses.A_BOLD) - reset_time = self.accounting_info["reset_time"] + reset_time = self.accounting_info['reset_time'] if not reset_time: - reset_time = "unknown" + reset_time = 'unknown' - panel.addstr(labeling_line + 2, 35, "Time to reset: %s" % reset_time) + panel.addstr(labeling_line + 2, 35, 'Time to reset: %s' % reset_time) - used, total = self.accounting_info["read"], self.accounting_info["read_limit"] + used, total = self.accounting_info['read'], self.accounting_info['read_limit'] if used and total: - panel.addstr(labeling_line + 3, 2, "%s / %s" % (used, total), self.get_color(True)) + panel.addstr(labeling_line + 3, 2, '%s / %s' % (used, total), self.get_color(True)) - used, total = self.accounting_info["written"], self.accounting_info["writtenLimit"] + used, total = self.accounting_info['written'], self.accounting_info['writtenLimit'] if used and total: - panel.addstr(labeling_line + 3, 37, "%s / %s" % (used, total), self.get_color(False)) + panel.addstr(labeling_line + 3, 37, '%s / %s' % (used, total), self.get_color(False)) else: - panel.addstr(labeling_line + 2, 0, "Accounting:", curses.A_BOLD) - panel.addstr(labeling_line + 2, 12, "Connection Closed...") + panel.addstr(labeling_line + 2, 0, 'Accounting:', curses.A_BOLD) + panel.addstr(labeling_line + 2, 12, 'Connection Closed...') def get_title(self, width): stats = list(self._title_stats) while True: if not stats: - return "Bandwidth:" + return 'Bandwidth:' else: - label = "Bandwidth (%s):" % ", ".join(stats) + label = 'Bandwidth (%s):' % ', '.join(stats) if len(label) > width: del stats[-1] @@ -365,29 +365,29 @@ class BandwidthStats(graph_panel.GraphStats): return label def get_header_label(self, width, is_primary): - graph_type = "Download" if is_primary else "Upload" - stats = [""] + graph_type = 'Download' if is_primary else 'Upload' + stats = [''] # if wide then avg and total are part of the header, otherwise they're on # the x-axis if width * 2 > COLLAPSE_WIDTH: - stats = [""] * 3 - stats[1] = "- %s" % self._get_avg_label(is_primary) - stats[2] = ", %s" % self._get_total_label(is_primary) + stats = [''] * 3 + stats[1] = '- %s' % self._get_avg_label(is_primary) + stats[2] = ', %s' % self._get_total_label(is_primary) - stats[0] = "%-14s" % ("%s/sec" % str_tools.size_label((self.last_primary if is_primary else self.last_secondary) * 1024, 1, False, CONFIG["features.graph.bw.transferInBytes"])) + stats[0] = '%-14s' % ('%s/sec' % str_tools.size_label((self.last_primary if is_primary else self.last_secondary) * 1024, 1, False, CONFIG['features.graph.bw.transferInBytes'])) # drops label's components if there's not enough space - labeling = graph_type + " (" + "".join(stats).strip() + "):" + labeling = graph_type + ' (' + ''.join(stats).strip() + '):' while len(labeling) >= width: if len(stats) > 1: del stats[-1] - labeling = graph_type + " (" + "".join(stats).strip() + "):" + labeling = graph_type + ' (' + ''.join(stats).strip() + '):' else: - labeling = graph_type + ":" + labeling = graph_type + ':' break return labeling @@ -407,7 +407,7 @@ class BandwidthStats(graph_panel.GraphStats): if not controller.is_alive(): return # keep old values - my_fingerprint = controller.get_info("fingerprint", None) + my_fingerprint = controller.get_info('fingerprint', None) if not self._title_stats or not my_fingerprint or (event and my_fingerprint in event.idlist): stats = [] @@ -415,29 +415,29 @@ class BandwidthStats(graph_panel.GraphStats): bw_burst = get_my_bandwidth_burst(controller) bw_observed = get_my_bandwidth_observed(controller) bw_measured = get_my_bandwidth_measured(controller) - label_in_bytes = CONFIG["features.graph.bw.transferInBytes"] + label_in_bytes = CONFIG['features.graph.bw.transferInBytes'] if bw_rate and bw_burst: bw_rate_label = str_tools.size_label(bw_rate, 1, False, label_in_bytes) bw_burst_label = str_tools.size_label(bw_burst, 1, False, label_in_bytes) - # if both are using rounded values then strip off the ".0" decimal + # if both are using rounded values then strip off the '.0' decimal - if ".0" in bw_rate_label and ".0" in bw_burst_label: - bw_rate_label = bw_rate_label.replace(".0", "") - bw_burst_label = bw_burst_label.replace(".0", "") + if '.0' in bw_rate_label and '.0' in bw_burst_label: + bw_rate_label = bw_rate_label.replace('.0', '') + bw_burst_label = bw_burst_label.replace('.0', '') - stats.append("limit: %s/s" % bw_rate_label) - stats.append("burst: %s/s" % bw_burst_label) + stats.append('limit: %s/s' % bw_rate_label) + stats.append('burst: %s/s' % bw_burst_label) # Provide the observed bandwidth either if the measured bandwidth isn't # available or if the measured bandwidth is the observed (this happens # if there isn't yet enough bandwidth measurements). if bw_observed and (not bw_measured or bw_measured == bw_observed): - stats.append("observed: %s/s" % str_tools.size_label(bw_observed, 1, False, label_in_bytes)) + stats.append('observed: %s/s' % str_tools.size_label(bw_observed, 1, False, label_in_bytes)) elif bw_measured: - stats.append("measured: %s/s" % str_tools.size_label(bw_measured, 1, False, label_in_bytes)) + stats.append('measured: %s/s' % str_tools.size_label(bw_measured, 1, False, label_in_bytes)) self._title_stats = stats @@ -445,12 +445,12 @@ class BandwidthStats(graph_panel.GraphStats): total = self.primary_total if is_primary else self.secondary_total total += self.prepopulate_primary_total if is_primary else self.prepopulate_secondary_total - return "avg: %s/sec" % str_tools.size_label((total / max(1, self.tick + self.prepopulate_ticks)) * 1024, 1, False, CONFIG["features.graph.bw.transferInBytes"]) + return 'avg: %s/sec' % str_tools.size_label((total / max(1, self.tick + self.prepopulate_ticks)) * 1024, 1, False, CONFIG['features.graph.bw.transferInBytes']) def _get_total_label(self, is_primary): total = self.primary_total if is_primary else self.secondary_total total += self.initial_primary_total if is_primary else self.initial_secondary_total - return "total: %s" % str_tools.size_label(total * 1024, 1) + return 'total: %s' % str_tools.size_label(total * 1024, 1) def _update_accounting_info(self): """ @@ -461,12 +461,12 @@ class BandwidthStats(graph_panel.GraphStats): """ controller = tor_controller() - queried = dict([(arg, "") for arg in ACCOUNTING_ARGS]) - queried["status"] = controller.get_info("accounting/hibernating", None) + queried = dict([(arg, '') for arg in ACCOUNTING_ARGS]) + queried['status'] = controller.get_info('accounting/hibernating', None) # provides a nicely formatted reset time - end_interval = controller.get_info("accounting/interval-end", None) + end_interval = controller.get_info('accounting/interval-end', None) if end_interval: # converts from gmt to local with respect to DST @@ -476,10 +476,10 @@ class BandwidthStats(graph_panel.GraphStats): else: tz_offset = time.timezone - sec = time.mktime(time.strptime(end_interval, "%Y-%m-%d %H:%M:%S")) - time.time() - tz_offset + sec = time.mktime(time.strptime(end_interval, '%Y-%m-%d %H:%M:%S')) - time.time() - tz_offset - if CONFIG["features.graph.bw.accounting.isTimeLong"]: - queried["reset_time"] = ", ".join(str_tools.time_labels(sec, True)) + if CONFIG['features.graph.bw.accounting.isTimeLong']: + queried['reset_time'] = ', '.join(str_tools.time_labels(sec, True)) else: days = sec / 86400 sec %= 86400 @@ -487,22 +487,22 @@ class BandwidthStats(graph_panel.GraphStats): sec %= 3600 minutes = sec / 60 sec %= 60 - queried["reset_time"] = "%i:%02i:%02i:%02i" % (days, hours, minutes, sec) + queried['reset_time'] = '%i:%02i:%02i:%02i' % (days, hours, minutes, sec) # number of bytes used and in total for the accounting period - used = controller.get_info("accounting/bytes", None) - left = controller.get_info("accounting/bytes-left", None) + used = controller.get_info('accounting/bytes', None) + left = controller.get_info('accounting/bytes-left', None) if used and left: - used_comp, left_comp = used.split(" "), left.split(" ") + used_comp, left_comp = used.split(' '), left.split(' ') read, written = int(used_comp[0]), int(used_comp[1]) read_left, written_left = int(left_comp[0]), int(left_comp[1]) - queried["read"] = str_tools.size_label(read) - queried["written"] = str_tools.size_label(written) - queried["read_limit"] = str_tools.size_label(read + read_left) - queried["writtenLimit"] = str_tools.size_label(written + written_left) + queried['read'] = str_tools.size_label(read) + queried['written'] = str_tools.size_label(written) + queried['read_limit'] = str_tools.size_label(read + read_left) + queried['writtenLimit'] = str_tools.size_label(written + written_left) self.accounting_info = queried self.accounting_last_updated = time.time() @@ -517,20 +517,19 @@ def get_my_bandwidth_rate(controller): # effective relayed bandwidth is the minimum of BandwidthRate, # MaxAdvertisedBandwidth, and RelayBandwidthRate (if set) - effective_rate = int(controller.get_conf("BandwidthRate", None)) + effective_rate = controller.get_conf('BandwidthRate', None) + relay_rate = controller.get_conf('RelayBandwidthRate', None) - relay_rate = controller.get_conf("RelayBandwidthRate", None) + if effective_rate and relay_rate and relay_rate != '0': + effective_rate = min(int(effective_rate), int(relay_rate)) - if relay_rate and relay_rate != "0": - effective_rate = min(effective_rate, int(relay_rate)) - - max_advertised = controller.get_conf("MaxAdvertisedBandwidth", None) + max_advertised = controller.get_conf('MaxAdvertisedBandwidth', None) if max_advertised: - effective_rate = min(effective_rate, int(max_advertised)) + effective_rate = min(int(effective_rate), int(max_advertised)) if effective_rate is not None: - return effective_rate + return int(effective_rate) else: return None @@ -542,15 +541,15 @@ def get_my_bandwidth_burst(controller): """ # effective burst (same for BandwidthBurst and RelayBandwidthBurst) - effective_burst = int(controller.get_conf("BandwidthBurst", None)) - relay_burst = controller.get_conf("RelayBandwidthBurst", None) + effective_burst = controller.get_conf('BandwidthBurst', None) + relay_burst = controller.get_conf('RelayBandwidthBurst', None) - if relay_burst and relay_burst != "0": - effective_burst = min(effective_burst, int(relay_burst)) + if effective_burst and relay_burst and relay_burst != '0': + effective_burst = min(int(effective_burst), int(relay_burst)) if effective_burst is not None: - return effective_burst + return int(effective_burst) else: return None @@ -564,15 +563,8 @@ def get_my_bandwidth_observed(controller): descriptors aren't periodically updated. """ - my_fingerprint = controller.get_info("fingerprint", None) - - if my_fingerprint: - my_descriptor = controller.get_server_descriptor(my_fingerprint, None) - - if my_descriptor: - return my_descriptor.observed_bandwidth - - return None + my_descriptor = controller.get_server_descriptor(default = None) + return getattr(my_descriptor, 'observed_bandwidth', None) def get_my_bandwidth_measured(controller): @@ -585,16 +577,5 @@ def get_my_bandwidth_measured(controller): https://trac.torproject.org/projects/tor/ticket/1566 """ - # TODO: Tor is documented as providing v2 router status entries but - # actually looks to be v3. This needs to be sorted out between stem - # and tor. - - my_fingerprint = controller.get_info("fingerprint", None) - - if my_fingerprint: - my_status_entry = controller.get_network_status(my_fingerprint, None) - - if my_status_entry and hasattr(my_status_entry, 'bandwidth'): - return my_status_entry.bandwidth - - return None + my_status_entry = controller.get_network_status(default = None) + return getattr(my_status_entry, 'bandwidth', None)

1 0

[stem/master] Adding is_localhost() to the BaseController
by atagar＠torproject.org 09 Sep '14

09 Sep '14

commit beaae677118b6549ee959ba699f8a4bf664e2bb6 Author: Damian Johnson <atagar(a)torproject.org> Date: Mon Sep 8 19:31:01 2014 -0700 Adding is_localhost() to the BaseController Adding our socket's is_localhost() method to the BaseController. --- stem/control.py | 22 +++++++++++++++++----- 1 file changed, 17 insertions(+), 5 deletions(-) diff --git a/stem/control.py b/stem/control.py index c9a9a5f..3c26666 100644 --- a/stem/control.py +++ b/stem/control.py @@ -128,6 +128,7 @@ If you're fine with allowing your script to raise exceptions then this can be mo BaseController - Base controller class asynchronous message handling |- msg - communicates with the tor process |- is_alive - reports if our connection to tor is open or closed + |- is_localhost - returns if the connection is for the local system or not |- connection_time - time when we last connected or disconnected |- is_authenticated - checks if we're authenticated to tor |- connect - connects or reconnects to tor @@ -479,6 +480,17 @@ class BaseController(object): return self._socket.is_alive() + def is_localhost(self): + """ + Returns if the connection is for the local system or not. + + .. versionadded:: 1.3.0 + + :returns: **bool** that's **True** if the connection is for the local host and **False** otherwise + """ + + return self._socket.is_localhost() + def connection_time(self): """ Provides the unix timestamp for when our socket was either connected or @@ -1235,7 +1247,7 @@ class Controller(BaseController): if not user: user = self.get_info('process/user', None) - if not user and self.get_socket().is_localhost(): + if not user and self.is_localhost(): pid = self.get_pid(None) if pid: @@ -1245,7 +1257,7 @@ class Controller(BaseController): self._set_cache({'user': user}) return user elif default == UNDEFINED: - if self.get_socket().is_localhost(): + if self.is_localhost(): raise ValueError("Unable to resolve tor's user") else: raise ValueError("Tor isn't running locally") @@ -1276,7 +1288,7 @@ class Controller(BaseController): if getinfo_pid and getinfo_pid.isdigit(): pid = int(getinfo_pid) - if not pid and self.get_socket().is_localhost(): + if not pid and self.is_localhost(): pid_file_path = self.get_conf('PidFile', None) if pid_file_path is not None: @@ -1301,7 +1313,7 @@ class Controller(BaseController): self._set_cache({'pid': pid}) return pid elif default == UNDEFINED: - if self.get_socket().is_localhost(): + if self.is_localhost(): raise ValueError("Unable to resolve tor's pid") else: raise ValueError("Tor isn't running locally") @@ -2655,7 +2667,7 @@ class Controller(BaseController): owning_pid = self.get_conf('__OwningControllerProcess', None) - if owning_pid == str(os.getpid()) and self.get_socket().is_localhost(): + if owning_pid == str(os.getpid()) and self.is_localhost(): response = self.msg('TAKEOWNERSHIP') stem.response.convert('SINGLELINE', response)

1 0

[meek/master] Fix readIntoBuf when starting with a non-empty buffer.
by dcf＠torproject.org 09 Sep '14

09 Sep '14

commit 5b51cf7d301c6f8d5fd3a939dea602c7484111ed Author: David Fifield <david(a)bamsoftware.com> Date: Mon Sep 8 16:43:37 2014 -0700 Fix readIntoBuf when starting with a non-empty buffer. readIntoBuf concatenates results from multiple reads until a buffer is filled. There was a mistake whenever the current offset was not 0: the second argument is an absolute index, not a length relative to the first argument. The affected line of code was: this.buf.subarray(this.buf.length - this.bytesToRead, n).set(data) For example, if bytesToRead were 100, and readIntoBuf were called with 20 bytes available, the data would be stored in the buffer as: this.buf.subarray(100 - 100, 20).set(<array with length 20>) So far, so good. The first 20 elements of the array would be set. When the next read happened, say there are 80 bytes available, the call would be: this.buf.subarray(100 - 80, 80).set(<array with length 80>) The code was trying to store an 80-length array in a 60-length subarray, and failed with RangeError: invalid array length I'm pretty sure this bug was the cause of "tbb bundle with meek takes (literally) hours to connect" https://trac.torproject.org/projects/tor/ticket/11612 It would be more likely to happen during initial bootstrapping, because that's when the client sends some large bunches of data (they can be about 80 KB), which are more likely to be split into two sends. I was able to reproduce the described error by altering meek-client to split its sends into two, with a 500-millisecond delay in between. --- firefox/components/main.js | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/firefox/components/main.js b/firefox/components/main.js index ec98d15..1b71412 100644 --- a/firefox/components/main.js +++ b/firefox/components/main.js @@ -340,7 +340,7 @@ MeekHTTPHelper.RequestReader.prototype = { readIntoBuf: function(input) { var n = Math.min(input.available(), this.bytesToRead); var data = input.readByteArray(n); - this.buf.subarray(this.buf.length - this.bytesToRead, n).set(data) + this.buf.subarray(this.buf.length - this.bytesToRead).set(data); this.bytesToRead -= n; },

1 0

[meek/master] asyncWait doesn't take an argument.
by dcf＠torproject.org 09 Sep '14

09 Sep '14

commit 53371c4b2a37dbed6371246fd27f5a857e6a934a Author: David Fifield <david(a)bamsoftware.com> Date: Mon Sep 8 16:10:46 2014 -0700 asyncWait doesn't take an argument. --- firefox/components/main.js | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/firefox/components/main.js b/firefox/components/main.js index 0565af6..ec98d15 100644 --- a/firefox/components/main.js +++ b/firefox/components/main.js @@ -321,12 +321,12 @@ MeekHTTPHelper.RequestReader.prototype = { switch (this.state) { case this.STATE_READING_LENGTH: this.doStateReadingLength(input); - this.asyncWait(inputStream); + this.asyncWait(); break; case this.STATE_READING_OBJECT: this.doStateReadingObject(input); if (this.bytesToRead > 0) - this.asyncWait(inputStream); + this.asyncWait(); break; } } catch (e) {

1 0

[meek/master] Put "ExtORPort auto" in the meek-server man page examples.
by dcf＠torproject.org 08 Sep '14

08 Sep '14

commit 6b19a695ce5bb93c227496b178df0f696623459e Author: David Fifield <david(a)bamsoftware.com> Date: Sun Aug 10 23:12:22 2014 -0700 Put "ExtORPort auto" in the meek-server man page examples. --- doc/meek-server.1.txt | 2 ++ 1 file changed, 2 insertions(+) diff --git a/doc/meek-server.1.txt b/doc/meek-server.1.txt index cb3b151..855a20b 100644 --- a/doc/meek-server.1.txt +++ b/doc/meek-server.1.txt @@ -25,10 +25,12 @@ with plain HTTP. Configuration for meek-server usually appears in a torrc file. Here is a sample configuration using HTTPS: ---- +ExtORPort auto ServerTransportPlugin meek exec ./meek-server --port 8443 --cert cert.pem --key key.pem --log meek-server.log ---- Here is a sample configuration using plain HTTP: ---- +ExtORPort auto ServerTransportPlugin meek exec ./meek-server --port 8080 --disable-tls --log meek-server.log ----

1 0

[meek/master] Typo.
by dcf＠torproject.org 08 Sep '14

08 Sep '14

commit 7bf463f3948fc59c50e4bf09b2b6cd771bfef06a Author: David Fifield <david(a)bamsoftware.com> Date: Sun Aug 10 22:46:44 2014 -0700 Typo. --- meek-client/meek-client.go | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/meek-client/meek-client.go b/meek-client/meek-client.go index e98dd76..06077a9 100644 --- a/meek-client/meek-client.go +++ b/meek-client/meek-client.go @@ -351,7 +351,7 @@ func checkProxyURL(u *url.URL) error { // With the helper we can use HTTP and SOCKS (because it is the // browser that does the proxying, not us). // For the HTTP proxy with the Firefox helper: versions of - // Firefox before 32 , and Tor Browser before 3.6.2, leak the + // Firefox before 32, and Tor Browser before 3.6.2, leak the // covert Host header in HTTP proxy CONNECT requests. Using an // HTTP proxy cannot provide effective obfuscation without such // a patched Firefox.

1 0