tor-commits

Download

tor-commits@lists.torproject.org

December 2015

18 participants
1281 discussions

[stem/master] Only note style check content when there's something to show
by atagar＠torproject.org 06 Dec '15

06 Dec '15

commit c30362a4a5074d5fc6f97a79eff09a6fe7d94290 Author: Damian Johnson <atagar(a)torproject.org> Date: Thu Nov 12 09:35:30 2015 -0800 Only note style check content when there's something to show While making changes I got a bunch of... line 57 - W391 blank line at end of file () While it's not wrong I find the brackets annoying. Only providing the content if we're showing something, and using a pipe as a separator instead. --- run_tests.py | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/run_tests.py b/run_tests.py index 91c0adf..431a961 100755 --- a/run_tests.py +++ b/run_tests.py @@ -326,7 +326,8 @@ def _print_static_issues(static_check_issues): for line_number in sorted(line_to_issues.keys()): for msg, line in line_to_issues[line_number]: line_count = '%-4s' % line_number - println(' line %s - %-40s (%s)' % (line_count, msg, line.strip())) + content = ' | %s' % line.strip() if line.strip() else '' + println(' line %s - %-40s%s' % (line_count, msg, content)) println()

1 0

[stem/master] Module for Tor manual information
by atagar＠torproject.org 06 Dec '15

06 Dec '15

commit 53accd666eeceae94ab3ee7f54236c624fcfe174 Author: Damian Johnson <atagar(a)torproject.org> Date: Sat Nov 21 11:41:32 2015 -0800 Module for Tor manual information Nyx, Erebus, and Stem's interpreter all have use for human readable information about tor's configuration options. Nyx has long parsed the man page for this but like most of Nyx it was pretty cobbled together. No tests, poor performance, and not very general purpose. https://trac.torproject.org/projects/tor/ticket/8251 Moving the useful bits of Nyx over and expanding it to be an order of magnitude faster (by calling 'man -P cat' to avoid its pager) and have a great compliment of tests. Note that the tests presently expect the tor man page in our test directory but it doesn't exist. I'll be addressing this in an upcoming commit, but I don't want to add the man page to our git history (no need to artificially inflait our repository size). --- docs/api.rst | 1 + docs/api/manual.rst | 5 + docs/change_log.rst | 1 + docs/contents.rst | 1 + setup.py | 2 +- stem/manual.cfg | 264 +++++++++++++++++++++++++++++++++++++ stem/manual.py | 298 ++++++++++++++++++++++++++++++++++++++++++ test/settings.cfg | 2 + test/unit/manual.py | 183 ++++++++++++++++++++++++++ test/unit/tor.1_with_unknown | 51 ++++++++ 10 files changed, 807 insertions(+), 1 deletion(-) diff --git a/docs/api.rst b/docs/api.rst index 85cfdcb..208450b 100644 --- a/docs/api.rst +++ b/docs/api.rst @@ -15,6 +15,7 @@ Controller * **Types** * `stem.exit_policy <api/exit_policy.html>`_ - Relay policy for the destinations it will or won't allow traffic to. + * `stem.manual <api/manual.html>`_ - Information available about Tor from `its manual <https://www.torproject.org/docs/tor-manual.html.en>`_. * `stem.version <api/version.html>`_ - Tor versions that can be compared to determine Tor's capabilities. Descriptors diff --git a/docs/api/manual.rst b/docs/api/manual.rst new file mode 100644 index 0000000..7692673 --- /dev/null +++ b/docs/api/manual.rst @@ -0,0 +1,5 @@ +Manual +====== + +.. automodule:: stem.manual + diff --git a/docs/change_log.rst b/docs/change_log.rst index 70bb05e..080bc54 100644 --- a/docs/change_log.rst +++ b/docs/change_log.rst @@ -45,6 +45,7 @@ The following are only available within Stem's `git repository * **Controller** * Dramatic, `300x performance improvement <https://github.com/DonnchaC/stem/pull/1>`_ for reading from the control port with python 3 + * Added `stem.manual <api/manual.html>`_, which provides information available about Tor from `its manual <https://www.torproject.org/docs/tor-manual.html.en>`_ (:trac:`8251`) * :func:`~stem.connection.connect` and :func:`~stem.control.Controller.from_port` now connect to both port 9051 (relay's default) and 9151 (Tor Browser's default) (:trac:`16075`) * Added `support for NETWORK_LIVENESS events <api/response.html#stem.response.events.NetworkLivenessEvent>`_ (:spec:`44aac63`) * Added :func:`~stem.control.Controller.is_user_traffic_allowed` to the :class:`~stem.control.Controller` diff --git a/docs/contents.rst b/docs/contents.rst index ec41dda..930d4cf 100644 --- a/docs/contents.rst +++ b/docs/contents.rst @@ -34,6 +34,7 @@ Contents api/response api/exit_policy + api/manual api/version api/descriptor/descriptor diff --git a/setup.py b/setup.py index 9784441..002e982 100644 --- a/setup.py +++ b/setup.py @@ -17,6 +17,6 @@ distutils.core.setup( packages = ['stem', 'stem.descriptor', 'stem.interpreter', 'stem.response', 'stem.util'], keywords = 'tor onion controller', scripts = ['tor-prompt'], - package_data = {'stem.interpreter': ['settings.cfg'], 'stem.util': ['ports.cfg']}, + package_data = {'stem': ['manual.cfg'], 'stem.interpreter': ['settings.cfg'], 'stem.util': ['ports.cfg']}, ) diff --git a/stem/manual.cfg b/stem/manual.cfg new file mode 100644 index 0000000..f7a29c0 --- /dev/null +++ b/stem/manual.cfg @@ -0,0 +1,264 @@ +################################################################################ +# +# Information related to tor configuration options... +# +# * manual.important Most commonly used configuration options. +# * manual.summary Short summary describing the option. +# +################################################################################ + +manual.important BandwidthRate +manual.important BandwidthBurst +manual.important RelayBandwidthRate +manual.important RelayBandwidthBurst +manual.important ControlPort +manual.important HashedControlPassword +manual.important CookieAuthentication +manual.important DataDirectory +manual.important Log +manual.important RunAsDaemon +manual.important User + +manual.important Bridge +manual.important ExcludeNodes +manual.important MaxCircuitDirtiness +manual.important SocksPort +manual.important UseBridges + +manual.important BridgeRelay +manual.important ContactInfo +manual.important ExitPolicy +manual.important MyFamily +manual.important Nickname +manual.important ORPort +manual.important PortForwarding +manual.important AccountingMax +manual.important AccountingStart + +manual.important DirPortFrontPage +manual.important DirPort + +manual.important HiddenServiceDir +manual.important HiddenServicePort + +# General Config Options + +manual.summary.BandwidthRate Average bandwidth usage limit +manual.summary.BandwidthBurst Maximum bandwidth usage limit +manual.summary.MaxAdvertisedBandwidth Limit for the bandwidth we advertise as being available for relaying +manual.summary.RelayBandwidthRate Average bandwidth usage limit for relaying +manual.summary.RelayBandwidthBurst Maximum bandwidth usage limit for relaying +manual.summary.PerConnBWRate Average relayed bandwidth limit per connection +manual.summary.PerConnBWBurst Maximum relayed bandwidth limit per connection +manual.summary.ConnLimit Minimum number of file descriptors for Tor to start +manual.summary.ConstrainedSockets Shrinks sockets to ConstrainedSockSize +manual.summary.ConstrainedSockSize Limit for the received and transmit buffers of sockets +manual.summary.ControlPort Port providing access to tor controllers (nyx, vidalia, etc) +manual.summary.ControlListenAddress Address providing controller access +manual.summary.ControlSocket Socket providing controller access +manual.summary.HashedControlPassword Hash of the password for authenticating to the control port +manual.summary.CookieAuthentication If set, authenticates controllers via a cookie +manual.summary.CookieAuthFile Location of the authentication cookie +manual.summary.CookieAuthFileGroupReadable Group read permissions for the authentication cookie +manual.summary.ControlPortWriteToFile Path for a file tor writes containing its control port +manual.summary.ControlPortFileGroupReadable Group read permissions for the control port file +manual.summary.DataDirectory Location for storing runtime data (state, keys, etc) +manual.summary.DirServer Alternative directory authorities +manual.summary.AlternateDirAuthority Alternative directory authorities (consensus only) +manual.summary.AlternateHSAuthority Alternative directory authorities (hidden services only) +manual.summary.AlternateBridgeAuthority Alternative directory authorities (bridges only) +manual.summary.DisableAllSwap Locks all allocated memory so they can't be paged out +manual.summary.FetchDirInfoEarly Keeps consensus information up to date, even if unnecessary +manual.summary.FetchDirInfoExtraEarly Updates consensus information when it's first available +manual.summary.FetchHidServDescriptors Toggles if hidden service descriptors are fetched automatically or not +manual.summary.FetchServerDescriptors Toggles if the consensus is fetched automatically or not +manual.summary.FetchUselessDescriptors Toggles if relay descriptors are fetched when they aren't strictly necessary +manual.summary.Group GID for the process when started +manual.summary.HttpProxy HTTP proxy for connecting to tor +manual.summary.HttpProxyAuthenticator Authentication credentials for HttpProxy +manual.summary.HttpsProxy SSL proxy for connecting to tor +manual.summary.HttpsProxyAuthenticator Authentication credentials for HttpsProxy +manual.summary.Socks4Proxy SOCKS 4 proxy for connecting to tor +manual.summary.Socks5Proxy SOCKS 5 for connecting to tor +manual.summary.Socks5ProxyUsername Username for connecting to the Socks5Proxy +manual.summary.Socks5ProxyPassword Password for connecting to the Socks5Proxy +manual.summary.KeepalivePeriod Rate at which to send keepalive packets +manual.summary.Log Runlevels and location for tor logging +manual.summary.LogMessageDomains Includes a domain when logging messages +manual.summary.OutboundBindAddress Sets the IP used for connecting to tor +manual.summary.PidFile Path for a file tor writes containing its process id +manual.summary.ProtocolWarnings Toggles if protocol errors give warnings or not +manual.summary.RunAsDaemon Toggles if tor runs as a daemon process +manual.summary.LogTimeGranularity limits granularity of log message timestamps +manual.summary.SafeLogging Toggles if logs are scrubbed of sensitive information +manual.summary.User UID for the process when started +manual.summary.HardwareAccel Toggles if tor attempts to use hardware acceleration +manual.summary.AccelName OpenSSL engine name for crypto acceleration +manual.summary.AccelDir Crypto acceleration library path +manual.summary.AvoidDiskWrites Toggles if tor avoids frequently writing to disk +manual.summary.TunnelDirConns Toggles if directory requests can be made over the ORPort +manual.summary.PreferTunneledDirConns Avoids directory requests that can't be made over the ORPort if set +manual.summary.CircuitPriorityHalflife Overwrite method for prioritizing traffic among relayed connections +manual.summary.DisableIOCP Disables use of the Windows IOCP networking API +manual.summary.CountPrivateBandwidth Applies rate limiting to private IP addresses + +# Client Config Options + +manual.summary.AllowInvalidNodes Permits use of relays flagged as invalid by authorities +manual.summary.ExcludeSingleHopRelays Permits use of relays that allow single hop connections +manual.summary.Bridge Available bridges +manual.summary.LearnCircuitBuildTimeout Toggles adaptive timeouts for circuit creation +manual.summary.CircuitBuildTimeout Initial timeout for circuit creation +manual.summary.CircuitIdleTimeout Timeout for closing circuits that have never been used +manual.summary.CircuitStreamTimeout Timeout for shifting streams among circuits +manual.summary.ClientOnly Ensures that we aren't used as a relay or directory mirror +manual.summary.ExcludeNodes Relays or locales never to be used in circuits +manual.summary.ExcludeExitNodes Relays or locales never to be used for exits +manual.summary.ExitNodes Preferred final hop for circuits +manual.summary.EntryNodes Preferred first hops for circuits +manual.summary.StrictNodes Never uses notes outside of Entry/ExitNodes +manual.summary.FascistFirewall Only make outbound connections on FirewallPorts +manual.summary.FirewallPorts Ports used by FascistFirewall +manual.summary.HidServAuth Authentication credentials for connecting to a hidden service +manual.summary.ReachableAddresses Rules for bypassing the local firewall +manual.summary.ReachableDirAddresses Rules for bypassing the local firewall (directory fetches) +manual.summary.ReachableORAddresses Rules for bypassing the local firewall (OR connections) +manual.summary.LongLivedPorts Ports requiring highly reliable relays +manual.summary.MapAddress Alias mappings for address requests +manual.summary.NewCircuitPeriod Period for considering the creation of new circuits +manual.summary.MaxCircuitDirtiness Duration for reusing constructed circuits +manual.summary.NodeFamily Define relays as belonging to a family +manual.summary.EnforceDistinctSubnets Prevent use of multiple relays from the same subnet on a circuit +manual.summary.SocksPort Port for using tor as a Socks proxy +manual.summary.SocksListenAddress Address from which Socks connections can be made +manual.summary.SocksPolicy Access policy for the pocks port +manual.summary.SocksTimeout Time until idle or unestablished socks connections are closed +manual.summary.TrackHostExits Maintains use of the same exit whenever connecting to this destination +manual.summary.TrackHostExitsExpire Time until use of an exit for tracking expires +manual.summary.UpdateBridgesFromAuthority Toggles fetching bridge descriptors from the authorities +manual.summary.UseBridges Make use of configured bridges +manual.summary.UseEntryGuards Use guard relays for first hop +manual.summary.NumEntryGuards Pool size of guard relays we'll select from +manual.summary.SafeSocks Toggles rejecting unsafe variants of the socks protocol +manual.summary.TestSocks Provide notices for if socks connections are of the safe or unsafe variants +manual.summary.WarnUnsafeSocks Toggle warning of unsafe socks connection +manual.summary.VirtualAddrNetwork Address range used with MAPADDRESS +manual.summary.AllowNonRFC953Hostnames Toggles blocking invalid characters in hostname resolution +manual.summary.AllowDotExit Toggles allowing exit notation in addresses +manual.summary.FastFirstHopPK Toggle public key usage for the first hop +manual.summary.TransPort Port for transparent proxying if the OS supports it +manual.summary.TransListenAddress Address from which transparent proxy connections can be made +manual.summary.NATDPort Port for forwarding ipfw NATD connections +manual.summary.NATDListenAddress Address from which NATD forwarded connections can be made +manual.summary.AutomapHostsOnResolve Map addresses ending with special suffixes to virtual addresses +manual.summary.AutomapHostsSuffixes Address suffixes recognized by AutomapHostsOnResolve +manual.summary.DNSPort Port from which DNS responses are fetched instead of tor +manual.summary.DNSListenAddress Address for performing DNS resolution +manual.summary.ClientDNSRejectInternalAddresses Ignores DNS responses for internal addresses +manual.summary.ClientRejectInternalAddresses Disables use of Tor for internal connections +manual.summary.DownloadExtraInfo Toggles fetching of extra information about relays +manual.summary.FallbackNetworkstatusFile Path for a fallback cache of the consensus +manual.summary.WarnPlaintextPorts Toggles warnings for using risky ports +manual.summary.RejectPlaintextPorts Prevents connections on risky ports +manual.summary.AllowSingleHopCircuits Makes use of single hop exits if able + +# Server Config Options + +manual.summary.Address Overwrites address others will use to reach this relay +manual.summary.AllowSingleHopExits Toggles permitting use of this relay as a single hop proxy +manual.summary.AssumeReachable Skips reachability test at startup +manual.summary.BridgeRelay Act as a bridge +manual.summary.ContactInfo Contact information for this relay +manual.summary.ExitPolicy Traffic destinations that can exit from this relay +manual.summary.ExitPolicyRejectPrivate Prevent exiting connection on the local network +manual.summary.MaxOnionsPending Decryption queue size +manual.summary.MyFamily Other relays this operator administers +manual.summary.Nickname Identifier for this relay +manual.summary.NumCPUs Number of processes spawned for decryption +manual.summary.ORPort Port used to accept relay traffic +manual.summary.ORListenAddress Address for relay connections +manual.summary.PortForwarding Use UPnP or NAT-PMP if needed to relay +manual.summary.PortForwardingHelper Executable for configuring port forwarding +manual.summary.PublishServerDescriptor Types of descriptors published +manual.summary.ShutdownWaitLength Delay before quitting after receiving a SIGINT signal +manual.summary.HeartbeatPeriod Rate at which an INFO level heartbeat message is sent +manual.summary.AccountingMax Amount of traffic before hibernating +manual.summary.AccountingStart Duration of an accounting period +manual.summary.RefuseUnknownExits Prevents relays not in the consensus from using us as an exit +manual.summary.ServerDNSResolvConfFile Overriding resolver config for DNS queries we provide +manual.summary.ServerDNSAllowBrokenConfig Toggles if we persist despite configuration parsing errors or not +manual.summary.ServerDNSSearchDomains Toggles if our DNS queries search for addresses in the local domain +manual.summary.ServerDNSDetectHijacking Toggles testing for DNS hijacking +manual.summary.ServerDNSTestAddresses Addresses to test to see if valid DNS queries are being hijacked +manual.summary.ServerDNSAllowNonRFC953Hostnames Toggles if we reject DNS queries with invalid characters +manual.summary.BridgeRecordUsageByCountry Tracks geoip information on bridge usage +manual.summary.ServerDNSRandomizeCase Toggles DNS query case randomization +manual.summary.GeoIPFile Path to file containing geoip information +manual.summary.CellStatistics Toggles storing circuit queue duration to disk +manual.summary.DirReqStatistics Toggles storing network status counts and performance to disk +manual.summary.EntryStatistics Toggles storing client connection counts to disk +manual.summary.ExitPortStatistics Toggles storing traffic and port usage data to disk +manual.summary.ConnDirectionStatistics Toggles storing connection use to disk +manual.summary.ExtraInfoStatistics Publishes statistic data in the extra-info documents + +# Directory Server Options + +manual.summary.AuthoritativeDirectory Act as a directory authority +manual.summary.DirPortFrontPage Publish this html file on the DirPort +manual.summary.V1AuthoritativeDirectory Generates a version 1 consensus +manual.summary.V2AuthoritativeDirectory Generates a version 2 consensus +manual.summary.V3AuthoritativeDirectory Generates a version 3 consensus +manual.summary.VersioningAuthoritativeDirectory Provides opinions on recommended versions of tor +manual.summary.NamingAuthoritativeDirectory Provides opinions on fingerprint to nickname bindings +manual.summary.HSAuthoritativeDir Toggles accepting hidden service descriptors +manual.summary.HidServDirectoryV2 Toggles accepting version 2 hidden service descriptors +manual.summary.BridgeAuthoritativeDir Acts as a bridge authority +manual.summary.MinUptimeHidServDirectoryV2 Required uptime before accepting hidden service directory +manual.summary.DirPort Port for directory connections +manual.summary.DirListenAddress Address the directory service is bound to +manual.summary.DirPolicy Access policy for the DirPort +manual.summary.FetchV2Networkstatus Get the obsolete V2 consensus + +# Directory Authority Server Options + +manual.summary.RecommendedVersions Tor versions believed to be safe +manual.summary.RecommendedClientVersions Tor versions believed to be safe for clients +manual.summary.RecommendedServerVersions Tor versions believed to be safe for relays +manual.summary.ConsensusParams Params entry of the networkstatus vote +manual.summary.DirAllowPrivateAddresses Toggles allowing arbitrary input or non-public IPs in descriptors +manual.summary.AuthDirBadDir Relays to be flagged as bad directory caches +manual.summary.AuthDirBadExit Relays to be flagged as bad exits +manual.summary.AuthDirInvalid Relays from which the valid flag is withheld +manual.summary.AuthDirReject Relays to be dropped from the consensus +manual.summary.AuthDirListBadDirs Toggles if we provide an opinion on bad directory caches +manual.summary.AuthDirListBadExits Toggles if we provide an opinion on bad exits +manual.summary.AuthDirRejectUnlisted Rejects further relay descriptors +manual.summary.AuthDirMaxServersPerAddr Limit on the number of relays accepted per ip +manual.summary.AuthDirMaxServersPerAuthAddr Limit on the number of relays accepted per an authority's ip +manual.summary.BridgePassword Password for requesting bridge information +manual.summary.V3AuthVotingInterval Consensus voting interval +manual.summary.V3AuthVoteDelay Wait time to collect votes of other authorities +manual.summary.V3AuthDistDelay Wait time to collect the signatures of other authorities +manual.summary.V3AuthNIntervalsValid Number of voting intervals a consensus is valid for +manual.summary.V3BandwidthsFile Path to a file containing measured relay bandwidths +manual.summary.V3AuthUseLegacyKey Signs consensus with both the current and legacy keys +manual.summary.RephistTrackTime Discards old, unchanged reliability informaition + +# Hidden Service Options + +manual.summary.HiddenServiceDir Directory contents for the hidden service +manual.summary.HiddenServicePort Port the hidden service is provided on +manual.summary.PublishHidServDescriptors Toggles automated publishing of the hidden service to the rendezvous directory +manual.summary.HiddenServiceVersion Version for published hidden service descriptors +manual.summary.HiddenServiceAuthorizeClient Restricts access to the hidden service +manual.summary.RendPostPeriod Period at which the rendezvous service descriptors are refreshed + +# Testing Network Options + +manual.summary.TestingTorNetwork Overrides other options to be a testing network +manual.summary.TestingV3AuthInitialVotingInterval Overrides V3AuthVotingInterval for the first consensus +manual.summary.TestingV3AuthInitialVoteDelay Overrides TestingV3AuthInitialVoteDelay for the first consensus +manual.summary.TestingV3AuthInitialDistDelay Overrides TestingV3AuthInitialDistDelay for the first consensus +manual.summary.TestingAuthDirTimeToLearnReachability Delay until opinions are given about which relays are running or not +manual.summary.TestingEstimatedDescriptorPropagationTime Delay before clients attempt to fetch descriptors from directory caches + diff --git a/stem/manual.py b/stem/manual.py new file mode 100644 index 0000000..52f25e6 --- /dev/null +++ b/stem/manual.py @@ -0,0 +1,298 @@ +# Copyright 2015, Damian Johnson and The Tor Project +# See LICENSE for licensing information + +""" +Provides information available about Tor from `its manual +<https://www.torproject.org/docs/tor-manual.html.en>`_. + +**Module Overview:** + +:: + + is_important - Indicates if a configuration option is of particularly common importance. + + Manual - Information about Tor available from its manual. + +- from_man - Retrieves manual information from its man page. + +.. versionadded:: 1.5.0 +""" + +import collections +import os + +import stem.prereq +import stem.util.conf +import stem.util.enum +import stem.util.log +import stem.util.system + +try: + # added in python 2.7 + from collections import OrderedDict +except ImportError: + from stem.util.ordereddict import OrderedDict + +try: + # added in python 3.2 + from functools import lru_cache +except ImportError: + from stem.util.lru_cache import lru_cache + +Category = stem.util.enum.Enum('GENERAL', 'CLIENT', 'RELAY', 'DIRECTORY', 'AUTHORITY', 'HIDDEN_SERVICE', 'TESTING', 'UNKNOWN') +ConfigOption = collections.namedtuple('ConfigOption', ['category', 'name', 'usage', 'summary', 'description']) + +CATEGORY_SECTIONS = { + 'GENERAL OPTIONS': Category.GENERAL, + 'CLIENT OPTIONS': Category.CLIENT, + 'SERVER OPTIONS': Category.RELAY, + 'DIRECTORY SERVER OPTIONS': Category.DIRECTORY, + 'DIRECTORY AUTHORITY SERVER OPTIONS': Category.AUTHORITY, + 'HIDDEN SERVICE OPTIONS': Category.HIDDEN_SERVICE, + 'TESTING NETWORK OPTIONS': Category.TESTING, +} + + +@lru_cache() +def _config(): + """ + Provides a dictionary for our manual.cfg. This has a couple categories... + + * manual.important (list) - list of lowercase configuration options + considered to be important + + * manual.summary.* (str) - summary descriptions of config options, key uses + the lowercase configuration option + """ + + config = stem.util.conf.Config() + config_path = os.path.join(os.path.dirname(__file__), 'manual.cfg') + + try: + config.load(config_path) + config_dict = dict([(key.lower(), config.get_value(key)) for key in config.keys()]) + config_dict['manual.important'] = [name.lower() for name in config.get_value('manual.important', [], multiple = True)] + return config_dict + except Exception as exc: + stem.util.log.warn("BUG: stem failed to load its internal manual information from '%s': %s" % (config_path, exc)) + return {} + + +def is_important(option): + """ + Indicates if a configuration option of particularly common importance or not. + + :param str option: tor configuration option to check + + :returns: **bool** that's **True** if this is an important option and + **False** otherwise + """ + + return option.lower() in _config()['manual.important'] + + +class Manual(object): + """ + Parsed tor man page. Tor makes no guarantees about its man page format so + this may not always be compatible. If not you can use the cached manual + information stored with Stem. + + This does not include every bit of information from the tor manual. For + instance, I've excluded the 'THE CONFIGURATION FILE FORMAT' section. If + there's a part you'd find useful then `file an issue + <https://trac.torproject.org/projects/tor/wiki/doc/stem/bugs>`_ and we can + add it. + + :var str name: brief description of the tor command + :var str synopsis: brief tor command usage + :var str description: general description of what tor does + + :var dict commandline_options: mapping of commandline arguments to their descripton + :var dict signals: mapping of signals tor accepts to their description + :var dict files: mapping of file paths to their description + + :var dict config_option: **ConfigOption** tuples for tor configuration options + """ + + def __init__(self, name, synopsis, description, commandline_options, signals, files, config_options): + self.name = name + self.synopsis = synopsis + self.description = description + self.commandline_options = commandline_options + self.signals = signals + self.files = files + self.config_options = config_options + + @staticmethod + def from_man(man_path = 'tor'): + """ + Reads and parses a given man page. + + :param str man_path: path argument for 'man', for example you might want + '/path/to/tor/doc/tor.1' to read from tor's git repository + """ + + try: + man_output = stem.util.system.call('man -P cat %s' % man_path) + except OSError as exc: + raise IOError("Unable to run 'man -P cat %s': %s" % (man_path, exc)) + + categories, config_options = _get_categories(man_output), OrderedDict() + + for category_header, category_enum in CATEGORY_SECTIONS.items(): + _add_config_options(config_options, category_enum, categories.get(category_header, [])) + + for category in categories: + if category.endswith(' OPTIONS') and category not in CATEGORY_SECTIONS and category != 'COMMAND-LINE OPTIONS': + _add_config_options(config_options, Category.UNKNOWN, categories.get(category, [])) + + return Manual( + _join_lines(categories.get('NAME', [])), + _join_lines(categories.get('SYNOPSIS', [])), + _join_lines(categories.get('DESCRIPTION', [])), + _get_indented_descriptions(categories.get('COMMAND-LINE OPTIONS', [])), + _get_indented_descriptions(categories.get('SIGNALS', [])), + _get_indented_descriptions(categories.get('FILES', [])), + config_options, + ) + + +def _get_categories(content): + """ + The man page is headers followed by an indented section. First pass gets + the mapping of category titles to their lines. + """ + + # skip header and footer lines + + if content and 'TOR(1)' in content[0]: + content = content[1:] + + if content and 'TOR(1)' in content[-1]: + content = content[:-1] + + categories = {} + category, lines = None, [] + + for line in content: + # replace non-ascii characters + # + # \u2019 - smart single quote + # \u2014 - extra long dash + # \xb7 - centered dot + + char_for = chr if stem.prereq.is_python_3() else unichr + line = line.replace(char_for(0x2019), "'").replace(char_for(0x2014), '-').replace(char_for(0xb7), '*') + + if line and not line.startswith(' '): + if category: + if lines[-1] == '': + lines = lines[:-1] # sections end with an extra empty line + + categories[category] = lines + + category, lines = line.strip(), [] + else: + if line.startswith(' '): + line = line[7:] # contents of a section have a seven space indentation + + lines.append(line) + + if category: + categories[category] = lines + + return categories + + +def _get_indented_descriptions(lines): + """ + Parses the commandline argument and signal sections. These are options + followed by an indented description. For example... + + :: + + -f FILE + Specify a new configuration file to contain further Tor configuration + options OR pass - to make Tor read its configuration from standard + input. (Default: /usr/local/etc/tor/torrc, or $HOME/.torrc if that file + is not found) + + There can be additional paragraphs not related to any particular argument but + ignoring those. + """ + + options, last_arg = OrderedDict(), None + + for line in lines: + if line and not line.startswith(' '): + options[line], last_arg = [], line + elif last_arg and line.startswith(' '): + options[last_arg].append(line[4:]) + + return dict([(arg, ' '.join(desc_lines)) for arg, desc_lines in options.items() if desc_lines]) + + +def _add_config_options(config_options, category, lines): + """ + Parses a section of tor configuration options. These have usage information, + followed by an indented description. For instance... + + :: + + ConnLimit NUM + The minimum number of file descriptors that must be available to the + Tor process before it will start. Tor will ask the OS for as many file + descriptors as the OS will allow (you can find this by "ulimit -H -n"). + If this number is less than ConnLimit, then Tor will refuse to start. + + + You probably don't need to adjust this. It has no effect on Windows + since that platform lacks getrlimit(). (Default: 1000) + """ + + last_option, usage, description = None, None, [] + + if lines and lines[0].startswith('The following options'): + lines = lines[lines.index(''):] # drop the initial description + + for line in lines: + if line and not line.startswith(' '): + if last_option: + summary = _config().get('manual.summary.%s' % last_option.lower(), '') + config_options[last_option] = ConfigOption(category, last_option, usage, summary, _join_lines(description).strip()) + + if ' ' in line: + last_option, usage = line.split(' ', 1) + else: + last_option, usage = line, '' + + description = [] + else: + if line.startswith(' '): + line = line[4:] + + description.append(line) + + if last_option: + summary = _config().get('manual.summary.%s' % last_option.lower(), '') + config_options[last_option] = ConfigOption(category, last_option, usage, summary, _join_lines(description).strip()) + + +def _join_lines(lines): + """ + The man page provides line-wrapped content. Attempting to undo that. This is + close to a simple join, but we still want empty lines to provide newlines. + """ + + content = [] + + for line in lines: + if line: + if content and content[-1][-1] != '\n': + line = ' ' + line + + content.append(line) + else: + if content and content[-1][-1] != '\n': + content.append('\n\n') + + return ''.join(content) diff --git a/test/settings.cfg b/test/settings.cfg index fffbdcf..13fb342 100644 --- a/test/settings.cfg +++ b/test/settings.cfg @@ -135,6 +135,7 @@ pyflakes.ignore run_tests.py => 'unittest' imported but unused pyflakes.ignore stem/__init__.py => undefined name 'long' pyflakes.ignore stem/__init__.py => undefined name 'unicode' pyflakes.ignore stem/control.py => undefined name 'controller' +pyflakes.ignore stem/manual.py => undefined name 'unichr' pyflakes.ignore stem/prereq.py => 'RSA' imported but unused pyflakes.ignore stem/prereq.py => 'asn1' imported but unused pyflakes.ignore stem/prereq.py => 'unittest' imported but unused @@ -176,6 +177,7 @@ test.unit_tests |test.unit.exit_policy.rule.TestExitPolicyRule |test.unit.exit_policy.policy.TestExitPolicy |test.unit.version.TestVersion +|test.unit.manual.TestManual |test.unit.tutorial.TestTutorial |test.unit.tutorial_examples.TestTutorialExamples |test.unit.response.add_onion.TestAddOnionResponse diff --git a/test/unit/manual.py b/test/unit/manual.py new file mode 100644 index 0000000..c9e682a --- /dev/null +++ b/test/unit/manual.py @@ -0,0 +1,183 @@ +""" +Unit tessts for the stem.manual module. Test data comes from the following... + + * test/unit/tor.1 - Tor version 0.2.8.0-alpha-dev (git-3c6782395743a089) +""" + +import codecs +import os +import unittest + +import stem.manual +import stem.util.system +import test.runner + +from stem.manual import Category + +try: + # added in python 3.2 + from functools import lru_cache +except ImportError: + from stem.util.lru_cache import lru_cache + +TEST_MAN_PAGE = os.path.join(os.path.dirname(__file__), 'tor.1') + +EXPECTED_CATEGORIES = set([ + 'NAME', + 'SYNOPSIS', + 'DESCRIPTION', + 'COMMAND-LINE OPTIONS', + 'THE CONFIGURATION FILE FORMAT', + 'GENERAL OPTIONS', + 'CLIENT OPTIONS', + 'SERVER OPTIONS', + 'DIRECTORY SERVER OPTIONS', + 'DIRECTORY AUTHORITY SERVER OPTIONS', + 'HIDDEN SERVICE OPTIONS', + 'TESTING NETWORK OPTIONS', + 'SIGNALS', + 'FILES', + 'SEE ALSO', + 'BUGS', + 'AUTHORS', +]) + +EXPECTED_CLI_OPTIONS = set(['-h, -help', '-f FILE', '--allow-missing-torrc', '--defaults-torrc FILE', '--ignore-missing-torrc', '--hash-password PASSWORD', '--list-fingerprint', '--verify-config', '--service install [--options command-line options]', '--service remove|start|stop', '--nt-service', '--list-torrc-options', '--version', '--quiet|--hush']) +EXPECTED_SIGNALS = set(['SIGTERM', 'SIGINT', 'SIGHUP', 'SIGUSR1', 'SIGUSR2', 'SIGCHLD', 'SIGPIPE', 'SIGXFSZ']) + +EXPECTED_OPTION_COUNTS = { + Category.GENERAL: 74, + Category.CLIENT: 86, + Category.RELAY: 47, + Category.DIRECTORY: 5, + Category.AUTHORITY: 34, + Category.HIDDEN_SERVICE: 11, + Category.TESTING: 32, + Category.UNKNOWN: 0, +} + +EXPECTED_DESCRIPTION = """ +Tor is a connection-oriented anonymizing communication service. Users choose a source-routed path through a set of nodes, and negotiate a "virtual circuit" through the network, in which each node knows its predecessor and successor, but no others. Traffic flowing down the circuit is unwrapped by a symmetric key at each node, which reveals the downstream node. + +Basically, Tor provides a distributed network of servers or relays ("onion routers"). Users bounce their TCP streams - web traffic, ftp, ssh, etc. - around the network, and recipients, observers, and even the relays themselves have difficulty tracking the source of the stream. + +By default, tor will only act as a client only. To help the network by providing bandwidth as a relay, change the ORPort configuration option - see below. Please also consult the documentation on the Tor Project's website. +""".strip() + +EXPECTED_FILE_DESCRIPTION = 'Specify a new configuration file to contain further Tor configuration options OR pass - to make Tor read its configuration from standard input. (Default: /usr/local/etc/tor/torrc, or $HOME/.torrc if that file is not found)' + +EXPECTED_BANDWIDTH_RATE_DESCRIPTION = 'A token bucket limits the average incoming bandwidth usage on this node to the specified number of bytes per second, and the average outgoing bandwidth usage to that same value. If you want to run a relay in the public network, this needs to be at the very least 30 KBytes (that is, 30720 bytes). (Default: 1 GByte)\n\nWith this option, and in other options that take arguments in bytes, KBytes, and so on, other formats are also supported. Notably, "KBytes" can also be written as "kilobytes" or "kb"; "MBytes" can be written as "megabytes" or "MB"; "kbits" can be written as "kilobits"; and so forth. Tor also accepts "byte" and "bit" in the singular. The prefixes "tera" and "T" are also recognized. If no units are given, we default to bytes. To avoid confusion, we recommend writing "bytes" or "bits" explicitly, since it\'s easy to forget that "B" means bytes, not bits.' + + +@lru_cache() +def man_content(): + return stem.util.system.call('man -P cat %s' % TEST_MAN_PAGE) + + +class TestManual(unittest.TestCase): + def test_is_important(self): + self.assertTrue(stem.manual.is_important('ExitPolicy')) + self.assertTrue(stem.manual.is_important('exitpolicy')) + self.assertTrue(stem.manual.is_important('EXITPOLICY')) + + self.assertFalse(stem.manual.is_important('ConstrainedSockSize')) + + def test_get_categories(self): + if stem.util.system.is_windows(): + test.runner.skip(self, '(unavailable on windows)') + return + + categories = stem.manual._get_categories(man_content()) + self.assertEqual(EXPECTED_CATEGORIES, set(categories.keys())) + self.assertEqual(['tor - The second-generation onion router'], categories['NAME']) + self.assertEqual(['tor [OPTION value]...'], categories['SYNOPSIS']) + self.assertEqual(8, len(categories['DESCRIPTION'])) # check parsing of multi-line entries + + def test_escapes_non_ascii(self): + if stem.util.system.is_windows(): + test.runner.skip(self, '(unavailable on windows)') + return + + def check(content): + try: + codecs.ascii_encode(content, 'strict') + except UnicodeEncodeError as exc: + self.fail("Unable to read '%s' as ascii: %s" % (content, exc)) + + categories = stem.manual._get_categories(man_content()) + + for category, lines in categories.items(): + check(category) + + for line in lines: + check(line) + + def test_has_all_summaries(self): + if stem.util.system.is_windows(): + test.runner.skip(self, '(unavailable on windows)') + return + + test.runner.skip(self, 'coming soon!') # TODO: yup, got a few to fill in... + + manual = stem.manual.Manual.from_man(TEST_MAN_PAGE) + missing_summary = [] + + for config_option in manual.config_options.values(): + if not config_option.summary and config_option.category != Category.TESTING: + missing_summary.append(config_option.name) + + if missing_summary: + self.fail("The following config options are missing summaries: %s" % ', '.join(missing_summary)) + + def test_attributes(self): + if stem.util.system.is_windows(): + test.runner.skip(self, '(unavailable on windows)') + return + + manual = stem.manual.Manual.from_man(TEST_MAN_PAGE) + + self.assertEqual('tor - The second-generation onion router', manual.name) + self.assertEqual('tor [OPTION value]...', manual.synopsis) + self.assertEqual(EXPECTED_DESCRIPTION, manual.description) + + self.assertEqual(EXPECTED_CLI_OPTIONS, set(manual.commandline_options.keys())) + self.assertEqual('Display a short help message and exit.', manual.commandline_options['-h, -help']) + self.assertEqual(EXPECTED_FILE_DESCRIPTION, manual.commandline_options['-f FILE']) + + self.assertEqual(EXPECTED_SIGNALS, set(manual.signals.keys())) + self.assertEqual('Tor will catch this, clean up and sync to disk if necessary, and exit.', manual.signals['SIGTERM']) + + self.assertEqual(31, len(manual.files)) + self.assertEqual('The tor process stores keys and other data here.', manual.files['/usr/local/var/lib/tor/']) + + for category, expected_count in EXPECTED_OPTION_COUNTS.items(): + self.assertEqual(expected_count, len([entry for entry in manual.config_options.values() if entry.category == category])) + + option = manual.config_options['BandwidthRate'] + self.assertEqual(Category.GENERAL, option.category) + self.assertEqual('BandwidthRate', option.name) + self.assertEqual('N bytes|KBytes|MBytes|GBytes|KBits|MBits|GBits', option.usage) + self.assertEqual('Average bandwidth usage limit', option.summary) + self.assertEqual(EXPECTED_BANDWIDTH_RATE_DESCRIPTION, option.description) + + def test_with_unknown_options(self): + if stem.util.system.is_windows(): + test.runner.skip(self, '(unavailable on windows)') + return + + manual = stem.manual.Manual.from_man(TEST_MAN_PAGE + '_with_unknown') + + self.assertEqual('tor - The second-generation onion router', manual.name) + self.assertEqual('', manual.synopsis) + self.assertEqual('', manual.description) + self.assertEqual({}, manual.commandline_options) + self.assertEqual({}, manual.signals) + + self.assertEqual(2, len(manual.config_options)) + + option = [entry for entry in manual.config_options.values() if entry.category == Category.UNKNOWN][0] + self.assertEqual(Category.UNKNOWN, option.category) + self.assertEqual('SpiffyNewOption', option.name) + self.assertEqual('transport exec path-to-binary [options]', option.usage) + self.assertEqual('', option.summary) + self.assertEqual('Description of this new option.', option.description) diff --git a/test/unit/tor.1_with_unknown b/test/unit/tor.1_with_unknown new file mode 100644 index 0000000..b5e0b82 --- /dev/null +++ b/test/unit/tor.1_with_unknown @@ -0,0 +1,51 @@ +'\" t +.\" Title: tor +.\" Author: [see the "AUTHORS" section] +.\" Generator: DocBook XSL Stylesheets v1.76.1 <http://docbook.sf.net/> +.\" Date: 10/03/2015 +.\" Manual: Tor Manual +.\" Source: Tor +.\" Language: English +.\" +.TH "TOR" "1" "10/03/2015" "Tor" "Tor Manual" +.\" ----------------------------------------------------------------- +.\" * Define some portability stuff +.\" ----------------------------------------------------------------- +.\" ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +.\" http://bugs.debian.org/507673 +.\" http://lists.gnu.org/archive/html/groff/2009-02/msg00013.html +.\" ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +.ie \n(.g .ds Aq \(aq +.el .ds Aq ' +.\" ----------------------------------------------------------------- +.\" * set default formatting +.\" ----------------------------------------------------------------- +.\" disable hyphenation +.nh +.\" disable justification (adjust text to left margin only) +.ad l +.\" ----------------------------------------------------------------- +.\" * MAIN CONTENT STARTS HERE * +.\" ----------------------------------------------------------------- +.SH "NAME" +tor \- The second\-generation onion router +.SH "GENERAL OPTIONS" +.PP +\fBBandwidthRate\fR \fIN\fR \fBbytes\fR|\fBKBytes\fR|\fBMBytes\fR|\fBGBytes\fR|\fBKBits\fR|\fBMBits\fR|\fBGBits\fR +.RS 4 +A token bucket limits the average incoming bandwidth usage on this node to the specified number of bytes per second, and the average outgoing bandwidth usage to that same value\&. If you want to run a relay in the public network, this needs to be +\fIat the very least\fR +30 KBytes (that is, 30720 bytes)\&. (Default: 1 GByte) + + +With this option, and in other options that take arguments in bytes, KBytes, and so on, other formats are also supported\&. Notably, "KBytes" can also be written as "kilobytes" or "kb"; "MBytes" can be written as "megabytes" or "MB"; "kbits" can be written as "kilobits"; and so forth\&. Tor also accepts "byte" and "bit" in the singular\&. The prefixes "tera" and "T" are also recognized\&. If no units are given, we default to bytes\&. To avoid confusion, we recommend writing "bytes" or "bits" explicitly, since it\(cqs easy to forget that "B" means bytes, not bits\&. +.RE +.PP +.SH "NEW OPTIONS" +.PP +\fBSpiffyNewOption\fR \fItransport\fR exec \fIpath\-to\-binary\fR [options] +.RS 4 +Description of this new option. +.RE +.PP +

1 0

[stem/master] Update config summaries for the latest tor manual
by atagar＠torproject.org 06 Dec '15

06 Dec '15

commit 5f2ca351e0a70565e4d58545ddbbd7b79e357262 Author: Damian Johnson <atagar(a)torproject.org> Date: Fri Nov 20 09:42:18 2015 -0800 Update config summaries for the latest tor manual Arm's configuration summaries were written years ago. No surprise tor's man page has changed since then, adding new options and dropping others. Filling in the gaps and enabling the test that checks we're in sync. --- stem/manual.cfg | 150 ++++++++++++++++++++++++++++++++++++++++----------- stem/manual.py | 14 ++--- test/unit/manual.py | 19 ++++--- 3 files changed, 137 insertions(+), 46 deletions(-) diff --git a/stem/manual.cfg b/stem/manual.cfg index f7a29c0..512501e 100644 --- a/stem/manual.cfg +++ b/stem/manual.cfg @@ -22,7 +22,7 @@ manual.important User manual.important Bridge manual.important ExcludeNodes manual.important MaxCircuitDirtiness -manual.important SocksPort +manual.important SOCKSPort manual.important UseBridges manual.important BridgeRelay @@ -50,12 +50,21 @@ manual.summary.RelayBandwidthRate Average bandwidth usage limit for relaying manual.summary.RelayBandwidthBurst Maximum bandwidth usage limit for relaying manual.summary.PerConnBWRate Average relayed bandwidth limit per connection manual.summary.PerConnBWBurst Maximum relayed bandwidth limit per connection +manual.summary.ClientTransportPlugin Proxy when establishing bridge connections +manual.summary.ServerTransportPlugin Proxy when servicing bridge connections +manual.summary.ServerTransportListenAddr Endpoint for bridge's pluggable transport proxy +manual.summary.ServerTransportOptions Additional arguments for bridge's proxy +manual.summary.ExtORPort Endpoint for extended ORPort connections +manual.summary.ExtORPortCookieAuthFile Location of the ExtORPort's authentication cookie +manual.summary.ExtORPortCookieAuthFileGroupReadable Group read permissions for the ExtORPort's authentication cookie manual.summary.ConnLimit Minimum number of file descriptors for Tor to start +manual.summary.DisableNetwork Don't accept non-controller connections manual.summary.ConstrainedSockets Shrinks sockets to ConstrainedSockSize manual.summary.ConstrainedSockSize Limit for the received and transmit buffers of sockets manual.summary.ControlPort Port providing access to tor controllers (nyx, vidalia, etc) manual.summary.ControlListenAddress Address providing controller access manual.summary.ControlSocket Socket providing controller access +manual.summary.ControlSocketsGroupWritable Group read permissions for the control socket manual.summary.HashedControlPassword Hash of the password for authenticating to the control port manual.summary.CookieAuthentication If set, authenticates controllers via a cookie manual.summary.CookieAuthFile Location of the authentication cookie @@ -63,43 +72,49 @@ manual.summary.CookieAuthFileGroupReadable Group read permissions for the authen manual.summary.ControlPortWriteToFile Path for a file tor writes containing its control port manual.summary.ControlPortFileGroupReadable Group read permissions for the control port file manual.summary.DataDirectory Location for storing runtime data (state, keys, etc) -manual.summary.DirServer Alternative directory authorities +manual.summary.FallbackDir Fallback when unable to retrieve descriptor information +manual.summary.DirAuthority Alternative directory authorities +manual.summary.DirAuthorityFallbackRate Rate at which to use fallback directory manual.summary.AlternateDirAuthority Alternative directory authorities (consensus only) -manual.summary.AlternateHSAuthority Alternative directory authorities (hidden services only) manual.summary.AlternateBridgeAuthority Alternative directory authorities (bridges only) manual.summary.DisableAllSwap Locks all allocated memory so they can't be paged out +manual.summary.DisableDebuggerAttachment Limit information applications can retrieve about the process manual.summary.FetchDirInfoEarly Keeps consensus information up to date, even if unnecessary manual.summary.FetchDirInfoExtraEarly Updates consensus information when it's first available manual.summary.FetchHidServDescriptors Toggles if hidden service descriptors are fetched automatically or not manual.summary.FetchServerDescriptors Toggles if the consensus is fetched automatically or not manual.summary.FetchUselessDescriptors Toggles if relay descriptors are fetched when they aren't strictly necessary -manual.summary.Group GID for the process when started -manual.summary.HttpProxy HTTP proxy for connecting to tor -manual.summary.HttpProxyAuthenticator Authentication credentials for HttpProxy -manual.summary.HttpsProxy SSL proxy for connecting to tor -manual.summary.HttpsProxyAuthenticator Authentication credentials for HttpsProxy +manual.summary.HTTPProxy HTTP proxy for connecting to tor +manual.summary.HTTPProxyAuthenticator Authentication credentials for HTTPProxy +manual.summary.HTTPSProxy SSL proxy for connecting to tor +manual.summary.HTTPSProxyAuthenticator Authentication credentials for HTTPSProxy +manual.summary.Sandbox Run within a syscall sandbox manual.summary.Socks4Proxy SOCKS 4 proxy for connecting to tor manual.summary.Socks5Proxy SOCKS 5 for connecting to tor manual.summary.Socks5ProxyUsername Username for connecting to the Socks5Proxy manual.summary.Socks5ProxyPassword Password for connecting to the Socks5Proxy +manual.summary.SocksSocketsGroupWritable Group write permissions for the socks socket manual.summary.KeepalivePeriod Rate at which to send keepalive packets manual.summary.Log Runlevels and location for tor logging manual.summary.LogMessageDomains Includes a domain when logging messages manual.summary.OutboundBindAddress Sets the IP used for connecting to tor manual.summary.PidFile Path for a file tor writes containing its process id manual.summary.ProtocolWarnings Toggles if protocol errors give warnings or not +manual.summary.PredictedPortsRelevanceTime Duration to ensure circuits for previously used ports remain available manual.summary.RunAsDaemon Toggles if tor runs as a daemon process manual.summary.LogTimeGranularity limits granularity of log message timestamps +manual.summary.TruncateLogFile Overwrites log file rather than appending when restarted +manual.summary.SyslogIdentityTag Tag logs appended to the syslog as being from tor manual.summary.SafeLogging Toggles if logs are scrubbed of sensitive information manual.summary.User UID for the process when started manual.summary.HardwareAccel Toggles if tor attempts to use hardware acceleration manual.summary.AccelName OpenSSL engine name for crypto acceleration manual.summary.AccelDir Crypto acceleration library path manual.summary.AvoidDiskWrites Toggles if tor avoids frequently writing to disk -manual.summary.TunnelDirConns Toggles if directory requests can be made over the ORPort -manual.summary.PreferTunneledDirConns Avoids directory requests that can't be made over the ORPort if set manual.summary.CircuitPriorityHalflife Overwrite method for prioritizing traffic among relayed connections manual.summary.DisableIOCP Disables use of the Windows IOCP networking API +manual.summary.UserspaceIOCPBuffers Disable kernel-space IOCP TCP buffers +manual.summary.UseFilteringSSLBufferevents Use SSL for a chain of bufferevents manual.summary.CountPrivateBandwidth Applies rate limiting to private IP addresses # Client Config Options @@ -114,12 +129,15 @@ manual.summary.CircuitStreamTimeout Timeout for shifting streams among circuits manual.summary.ClientOnly Ensures that we aren't used as a relay or directory mirror manual.summary.ExcludeNodes Relays or locales never to be used in circuits manual.summary.ExcludeExitNodes Relays or locales never to be used for exits +manual.summary.GeoIPExcludeUnknown Don't use relays with an unknown locale in circuits manual.summary.ExitNodes Preferred final hop for circuits manual.summary.EntryNodes Preferred first hops for circuits manual.summary.StrictNodes Never uses notes outside of Entry/ExitNodes manual.summary.FascistFirewall Only make outbound connections on FirewallPorts manual.summary.FirewallPorts Ports used by FascistFirewall manual.summary.HidServAuth Authentication credentials for connecting to a hidden service +manual.summary.CloseHSClientCircuitsImmediatelyOnTimeout Close hidden service circuits that timeout +manual.summary.CloseHSServiceRendCircuitsImmediatelyOnTimeout Close hidden service rendezvous circuits that timeout manual.summary.ReachableAddresses Rules for bypassing the local firewall manual.summary.ReachableDirAddresses Rules for bypassing the local firewall (directory fetches) manual.summary.ReachableORAddresses Rules for bypassing the local firewall (OR connections) @@ -127,27 +145,36 @@ manual.summary.LongLivedPorts Ports requiring highly reliable relays manual.summary.MapAddress Alias mappings for address requests manual.summary.NewCircuitPeriod Period for considering the creation of new circuits manual.summary.MaxCircuitDirtiness Duration for reusing constructed circuits +manual.summary.MaxClientCircuitsPending Number of circuits that can be in construction at once manual.summary.NodeFamily Define relays as belonging to a family manual.summary.EnforceDistinctSubnets Prevent use of multiple relays from the same subnet on a circuit -manual.summary.SocksPort Port for using tor as a Socks proxy -manual.summary.SocksListenAddress Address from which Socks connections can be made +manual.summary.SOCKSPort Port for using tor as a Socks proxy +manual.summary.SOCKSListenAddress Address from which Socks connections can be made manual.summary.SocksPolicy Access policy for the pocks port manual.summary.SocksTimeout Time until idle or unestablished socks connections are closed +manual.summary.TokenBucketRefillInterval Frequency at which exhausted connections are checked for new traffic manual.summary.TrackHostExits Maintains use of the same exit whenever connecting to this destination manual.summary.TrackHostExitsExpire Time until use of an exit for tracking expires manual.summary.UpdateBridgesFromAuthority Toggles fetching bridge descriptors from the authorities manual.summary.UseBridges Make use of configured bridges manual.summary.UseEntryGuards Use guard relays for first hop +manual.summary.UseEntryGuardsAsDirGuards Retrieve descriptors via guards when able +manual.summary.GuardfractionFile File containing information with duration of our guards +manual.summary.UseGuardFraction Take guardfraction into account for path selection manual.summary.NumEntryGuards Pool size of guard relays we'll select from +manual.summary.NumDirectoryGuards Pool size of directory guards we'll select from +manual.summary.GuardLifetime Minimum time to keep entry guards manual.summary.SafeSocks Toggles rejecting unsafe variants of the socks protocol manual.summary.TestSocks Provide notices for if socks connections are of the safe or unsafe variants manual.summary.WarnUnsafeSocks Toggle warning of unsafe socks connection -manual.summary.VirtualAddrNetwork Address range used with MAPADDRESS +manual.summary.VirtualAddrNetworkIPv4 IPv4 address range to use when needing a virtual address +manual.summary.VirtualAddrNetworkIPv6 IPv6 address range to use when needing a virtual address manual.summary.AllowNonRFC953Hostnames Toggles blocking invalid characters in hostname resolution manual.summary.AllowDotExit Toggles allowing exit notation in addresses manual.summary.FastFirstHopPK Toggle public key usage for the first hop manual.summary.TransPort Port for transparent proxying if the OS supports it manual.summary.TransListenAddress Address from which transparent proxy connections can be made +manual.summary.TransProxyType Proxy type to be used manual.summary.NATDPort Port for forwarding ipfw NATD connections manual.summary.NATDListenAddress Address from which NATD forwarded connections can be made manual.summary.AutomapHostsOnResolve Map addresses ending with special suffixes to virtual addresses @@ -157,10 +184,27 @@ manual.summary.DNSListenAddress Address for performing DNS resolution manual.summary.ClientDNSRejectInternalAddresses Ignores DNS responses for internal addresses manual.summary.ClientRejectInternalAddresses Disables use of Tor for internal connections manual.summary.DownloadExtraInfo Toggles fetching of extra information about relays -manual.summary.FallbackNetworkstatusFile Path for a fallback cache of the consensus manual.summary.WarnPlaintextPorts Toggles warnings for using risky ports manual.summary.RejectPlaintextPorts Prevents connections on risky ports manual.summary.AllowSingleHopCircuits Makes use of single hop exits if able +manual.summary.OptimisticData Use exits without confirmation that prior connections succeeded +manual.summary.Tor2webMode Establish non-anonymous hidden service connections +manual.summary.Tor2webRendezvousPoints Rendezvous points to use for hidden services when in Tor2webMode +manual.summary.UseMicrodescriptors Retrieve microdescriptors rather than server descriptors +manual.summary.UseNTorHandshake Use ntor for establishing circuits with relays +manual.summary.PathBiasCircThreshold Number of circuits through a guard before applying bias checks +manual.summary.PathBiasNoticeRate Fraction of circuits that must succeed before logging a notice +manual.summary.PathBiasWarnRate Fraction of circuits that must succeed before logging a warning +manual.summary.PathBiasExtremeRate Fraction of circuits that must succeed before logging an error +manual.summary.PathBiasDropGuards Drop guards failing to establish circuits +manual.summary.PathBiasScaleThreshold Circuits through a guard before scaling past observations down +manual.summary.PathBiasUseThreshold Number of streams through a circuit before applying bias checks +manual.summary.PathBiasNoticeUseRate Fraction of streams that must succeed before logging a notice +manual.summary.PathBiasExtremeUseRate Fraction of streams that must succeed before logging an error +manual.summary.PathBiasScaleUseThreshold Streams through a circuit before scaling past observations down +manual.summary.ClientUseIPv6 Allow IPv6 connections to guards +manual.summary.ClientPreferIPv6ORPort Prefer a guard's IPv6 rather than IPv4 endpoint +manual.summary.PathsNeededToBuildCircuits Portion of relays to require information for before making circuits # Server Config Options @@ -169,9 +213,11 @@ manual.summary.AllowSingleHopExits Toggles permitting use of this relay as a sin manual.summary.AssumeReachable Skips reachability test at startup manual.summary.BridgeRelay Act as a bridge manual.summary.ContactInfo Contact information for this relay +manual.summary.ExitRelay Allow relaying of exit traffic manual.summary.ExitPolicy Traffic destinations that can exit from this relay manual.summary.ExitPolicyRejectPrivate Prevent exiting connection on the local network -manual.summary.MaxOnionsPending Decryption queue size +manual.summary.IPv6Exit Allow clients to use us for IPv6 traffic +manual.summary.MaxOnionQueueDelay Duration to reject new onionskins if we have more than we can process manual.summary.MyFamily Other relays this operator administers manual.summary.Nickname Identifier for this relay manual.summary.NumCPUs Number of processes spawned for decryption @@ -181,8 +227,10 @@ manual.summary.PortForwarding Use UPnP or NAT-PMP if needed to relay manual.summary.PortForwardingHelper Executable for configuring port forwarding manual.summary.PublishServerDescriptor Types of descriptors published manual.summary.ShutdownWaitLength Delay before quitting after receiving a SIGINT signal +manual.summary.SSLKeyLifetime Lifetime for our link certificate manual.summary.HeartbeatPeriod Rate at which an INFO level heartbeat message is sent manual.summary.AccountingMax Amount of traffic before hibernating +manual.summary.AccountingRule Method to determine when the accounting limit is reached manual.summary.AccountingStart Duration of an accounting period manual.summary.RefuseUnknownExits Prevents relays not in the consensus from using us as an exit manual.summary.ServerDNSResolvConfFile Overriding resolver config for DNS queries we provide @@ -193,48 +241,54 @@ manual.summary.ServerDNSTestAddresses Addresses to test to see if valid DNS quer manual.summary.ServerDNSAllowNonRFC953Hostnames Toggles if we reject DNS queries with invalid characters manual.summary.BridgeRecordUsageByCountry Tracks geoip information on bridge usage manual.summary.ServerDNSRandomizeCase Toggles DNS query case randomization -manual.summary.GeoIPFile Path to file containing geoip information +manual.summary.GeoIPFile Path to file containing IPv4 geoip information +manual.summary.GeoIPv6File Path to file containing IPv6 geoip information +manual.summary.TLSECGroup EC group for incoming SSL connections manual.summary.CellStatistics Toggles storing circuit queue duration to disk manual.summary.DirReqStatistics Toggles storing network status counts and performance to disk manual.summary.EntryStatistics Toggles storing client connection counts to disk manual.summary.ExitPortStatistics Toggles storing traffic and port usage data to disk manual.summary.ConnDirectionStatistics Toggles storing connection use to disk +manual.summary.HiddenServiceStatistics Toggles storing hidden service stats to disk manual.summary.ExtraInfoStatistics Publishes statistic data in the extra-info documents +manual.summary.ExtendAllowPrivateAddresses Allow circuits to be extended to the local network +manual.summary.MaxMemInQueues Threshold at which tor will terminate circuits to avoid running out of memory +manual.summary.SigningKeyLifetime Duration the Ed25519 signing key is valid for +manual.summary.OfflineMasterKey Don't generate the master secret key # Directory Server Options -manual.summary.AuthoritativeDirectory Act as a directory authority manual.summary.DirPortFrontPage Publish this html file on the DirPort -manual.summary.V1AuthoritativeDirectory Generates a version 1 consensus -manual.summary.V2AuthoritativeDirectory Generates a version 2 consensus -manual.summary.V3AuthoritativeDirectory Generates a version 3 consensus -manual.summary.VersioningAuthoritativeDirectory Provides opinions on recommended versions of tor -manual.summary.NamingAuthoritativeDirectory Provides opinions on fingerprint to nickname bindings -manual.summary.HSAuthoritativeDir Toggles accepting hidden service descriptors manual.summary.HidServDirectoryV2 Toggles accepting version 2 hidden service descriptors -manual.summary.BridgeAuthoritativeDir Acts as a bridge authority -manual.summary.MinUptimeHidServDirectoryV2 Required uptime before accepting hidden service directory manual.summary.DirPort Port for directory connections manual.summary.DirListenAddress Address the directory service is bound to manual.summary.DirPolicy Access policy for the DirPort -manual.summary.FetchV2Networkstatus Get the obsolete V2 consensus # Directory Authority Server Options -manual.summary.RecommendedVersions Tor versions believed to be safe +manual.summary.AuthoritativeDirectory Act as a directory authority +manual.summary.V3AuthoritativeDirectory Generates a version 3 consensus +manual.summary.VersioningAuthoritativeDirectory Provides opinions on recommended versions of tor +manual.summary.RecommendedVersions Suggested versions of tor +manual.summary.RecommendedPackageVersions Suggested versions of applications other than tor manual.summary.RecommendedClientVersions Tor versions believed to be safe for clients +manual.summary.BridgeAuthoritativeDir Acts as a bridge authority +manual.summary.MinUptimeHidServDirectoryV2 Required uptime before accepting hidden service directory manual.summary.RecommendedServerVersions Tor versions believed to be safe for relays manual.summary.ConsensusParams Params entry of the networkstatus vote manual.summary.DirAllowPrivateAddresses Toggles allowing arbitrary input or non-public IPs in descriptors -manual.summary.AuthDirBadDir Relays to be flagged as bad directory caches manual.summary.AuthDirBadExit Relays to be flagged as bad exits manual.summary.AuthDirInvalid Relays from which the valid flag is withheld manual.summary.AuthDirReject Relays to be dropped from the consensus -manual.summary.AuthDirListBadDirs Toggles if we provide an opinion on bad directory caches +manual.summary.AuthDirBadExitCCs Countries for which to flag all relays as bad exits +manual.summary.AuthDirInvalidCCs Countries for which the valid flag is withheld +manual.summary.AuthDirRejectCCs Countries for which relays aren't accepted into the consensus manual.summary.AuthDirListBadExits Toggles if we provide an opinion on bad exits -manual.summary.AuthDirRejectUnlisted Rejects further relay descriptors manual.summary.AuthDirMaxServersPerAddr Limit on the number of relays accepted per ip manual.summary.AuthDirMaxServersPerAuthAddr Limit on the number of relays accepted per an authority's ip +manual.summary.AuthDirFastGuarantee Advertised rate at which the Fast flag is granted +manual.summary.AuthDirGuardBWGuarantee Advertised rate necessary to be a guard +manual.summary.AuthDirPinKeys Don't accept descriptors with conflicting identity keypairs manual.summary.BridgePassword Password for requesting bridge information manual.summary.V3AuthVotingInterval Consensus voting interval manual.summary.V3AuthVoteDelay Wait time to collect votes of other authorities @@ -242,7 +296,10 @@ manual.summary.V3AuthDistDelay Wait time to collect the signatures of other auth manual.summary.V3AuthNIntervalsValid Number of voting intervals a consensus is valid for manual.summary.V3BandwidthsFile Path to a file containing measured relay bandwidths manual.summary.V3AuthUseLegacyKey Signs consensus with both the current and legacy keys -manual.summary.RephistTrackTime Discards old, unchanged reliability informaition +manual.summary.RephistTrackTime Discards old, unchanged reliability information +manual.summary.VoteOnHidServDirectoriesV2 Determines if the authority votes on hidden service directories +manual.summary.AuthDirHasIPv6Connectivity Descriptors can be retrieved over the authority's IPv6 ORPort +manual.summary.MinMeasuredBWsForAuthToIgnoreAdvertised Total measured value before advertised bandwidths are treated as unreliable # Hidden Service Options @@ -251,7 +308,12 @@ manual.summary.HiddenServicePort Port the hidden service is provided on manual.summary.PublishHidServDescriptors Toggles automated publishing of the hidden service to the rendezvous directory manual.summary.HiddenServiceVersion Version for published hidden service descriptors manual.summary.HiddenServiceAuthorizeClient Restricts access to the hidden service +manual.summary.HiddenServiceAllowUnknownPorts Allow rendezvous circuits on unrecognized ports +manual.summary.HiddenServiceMaxStreams Maximum streams per rendezvous circuit +manual.summary.HiddenServiceMaxStreamsCloseCircuit Closes rendezvous circuits that exceed the maximum number of streams manual.summary.RendPostPeriod Period at which the rendezvous service descriptors are refreshed +manual.summary.HiddenServiceDirGroupReadable Group read permissions for the hidden service directory +manual.summary.HiddenServiceNumIntroductionPoints Number of introduction points the hidden service will have # Testing Network Options @@ -259,6 +321,32 @@ manual.summary.TestingTorNetwork Overrides other options to be a testing network manual.summary.TestingV3AuthInitialVotingInterval Overrides V3AuthVotingInterval for the first consensus manual.summary.TestingV3AuthInitialVoteDelay Overrides TestingV3AuthInitialVoteDelay for the first consensus manual.summary.TestingV3AuthInitialDistDelay Overrides TestingV3AuthInitialDistDelay for the first consensus +manual.summary.TestingV3AuthVotingStartOffset Offset for the point at which the authority votes manual.summary.TestingAuthDirTimeToLearnReachability Delay until opinions are given about which relays are running or not manual.summary.TestingEstimatedDescriptorPropagationTime Delay before clients attempt to fetch descriptors from directory caches +manual.summary.TestingMinFastFlagThreshold Minimum value for the Fast flag +manual.summary.TestingServerDownloadSchedule Schedule for when we should download resources as a relay +manual.summary.TestingClientDownloadSchedule Schedule for when we should download resources as a client +manual.summary.TestingServerConsensusDownloadSchedule Schedule for when we should download the consensus as a relay +manual.summary.TestingClientConsensusDownloadSchedule Schedule for when we should download the consensus as a client +manual.summary.TestingBridgeDownloadSchedule Schedule for when we should download bridge descriptors +manual.summary.TestingClientMaxIntervalWithoutRequest Maximum time to wait to batch requests for missing descriptors +manual.summary.TestingDirConnectionMaxStall Duration to let directory connections stall before timing out +manual.summary.TestingConsensusMaxDownloadTries Retries for downloading the consensus +manual.summary.TestingDescriptorMaxDownloadTries Retries for downloading server descriptors +manual.summary.TestingMicrodescMaxDownloadTries Retries for downloading microdescriptors +manual.summary.TestingCertMaxDownloadTries Retries for downloading authority certificates +manual.summary.TestingDirAuthVoteExit Relays to give the Exit flag to +manual.summary.TestingDirAuthVoteExitIsStrict Only grant the Exit flag to relays listed by TestingDirAuthVoteExit +manual.summary.TestingDirAuthVoteGuard Relays to give the Guard flag to +manual.summary.TestingDirAuthVoteGuardIsStrict Only grant the Guard flag to relays listed by TestingDirAuthVoteGuard +manual.summary.TestingDirAuthVoteHSDir Relays to give the HSDir flag to +manual.summary.TestingDirAuthVoteHSDirIsStrict Only grant the HSDir flag to relays listed by TestingDirAuthVoteHSDir +manual.summary.TestingEnableConnBwEvent Allow controllers to request CONN_BW events +manual.summary.TestingEnableCellStatsEvent Allow controllers to request CELL_STATS events +manual.summary.TestingEnableTbEmptyEvent Allow controllers to request TB_EMPTY events +manual.summary.TestingMinExitFlagThreshold Lower bound for assigning the Exit flag +manual.summary.TestingLinkCertifetime Duration of our ed25519 certificate +manual.summary.TestingAuthKeyLifetime Duration for our ed25519 signing key +manual.summary.TestingLinkKeySlop Time before expiration that we replace our ed25519 key diff --git a/stem/manual.py b/stem/manual.py index 52f25e6..cf1be50 100644 --- a/stem/manual.py +++ b/stem/manual.py @@ -53,15 +53,15 @@ CATEGORY_SECTIONS = { @lru_cache() -def _config(): +def _config(lowercase = True): """ Provides a dictionary for our manual.cfg. This has a couple categories... - * manual.important (list) - list of lowercase configuration options - considered to be important + * manual.important (list) - configuration options considered to be important + * manual.summary.* (str) - summary descriptions of config options - * manual.summary.* (str) - summary descriptions of config options, key uses - the lowercase configuration option + :param bool lowercase: uses lowercase keys if **True** to allow for case + insensitive lookups """ config = stem.util.conf.Config() @@ -69,8 +69,8 @@ def _config(): try: config.load(config_path) - config_dict = dict([(key.lower(), config.get_value(key)) for key in config.keys()]) - config_dict['manual.important'] = [name.lower() for name in config.get_value('manual.important', [], multiple = True)] + config_dict = dict([(key.lower() if lowercase else key, config.get_value(key)) for key in config.keys()]) + config_dict['manual.important'] = [name.lower() if lowercase else name for name in config.get_value('manual.important', [], multiple = True)] return config_dict except Exception as exc: stem.util.log.warn("BUG: stem failed to load its internal manual information from '%s': %s" % (config_path, exc)) diff --git a/test/unit/manual.py b/test/unit/manual.py index c9e682a..60ae291 100644 --- a/test/unit/manual.py +++ b/test/unit/manual.py @@ -117,17 +117,20 @@ class TestManual(unittest.TestCase): test.runner.skip(self, '(unavailable on windows)') return - test.runner.skip(self, 'coming soon!') # TODO: yup, got a few to fill in... - manual = stem.manual.Manual.from_man(TEST_MAN_PAGE) - missing_summary = [] + present = set(manual.config_options.keys()) + expected = set([key[15:] for key in stem.manual._config(lowercase = False) if key.startswith('manual.summary.')]) + + # TODO: The 'Recognized' config name is due to our man page being slightly + # malformed. Sending a tor patch later to fix it. - for config_option in manual.config_options.values(): - if not config_option.summary and config_option.category != Category.TESTING: - missing_summary.append(config_option.name) + missing_options = present.difference(expected).difference(set(['Recognized'])) + extra_options = expected.difference(present) - if missing_summary: - self.fail("The following config options are missing summaries: %s" % ', '.join(missing_summary)) + if missing_options: + self.fail("The following config options are missing summaries: %s" % ', '.join(missing_options)) + elif extra_options: + self.fail("The following config options no longer exist in tor, so don't need summaries: %s" % ', '.join(extra_options)) def test_attributes(self): if stem.util.system.is_windows():

1 0

[stem/master] Function to download tor's latest man page
by atagar＠torproject.org 06 Dec '15

06 Dec '15

commit f581cf22a4760e0a8cf92f383f4cca32ccc0f28e Author: Damian Johnson <atagar(a)torproject.org> Date: Mon Nov 23 14:33:53 2015 -0800 Function to download tor's latest man page Adding a function to fetch tor's latest asciidoc manual via gitweb, then compile it into a man page. Cringing? Don't worry. It's well documented as being best-effort and handy not only for our users but our tests as well... * Nyx will want this so it can learn descriptions for new tor options without updating. * Our tests use this so we don't need to bundle a copy of the man page to exercise our parser. * Integ testing to give tor some assurance its asciidoc can still be compiled and looks right. --- stem/descriptor/reader.py | 1 + stem/descriptor/remote.py | 5 +- stem/manual.cfg | 352 ----------------------------------------- stem/manual.py | 116 +++++++++++++- stem/settings.cfg | 352 +++++++++++++++++++++++++++++++++++++++++ test/integ/manual.py | 213 +++++++++++++++++++++++++ test/integ/tor.1_with_unknown | 51 ++++++ test/settings.cfg | 1 + test/unit/manual.py | 263 ++++++++++++------------------ test/unit/tor.1_with_unknown | 51 ------ 10 files changed, 835 insertions(+), 570 deletions(-) diff --git a/stem/descriptor/reader.py b/stem/descriptor/reader.py index a96406d..2877975 100644 --- a/stem/descriptor/reader.py +++ b/stem/descriptor/reader.py @@ -218,6 +218,7 @@ def save_processed_files(path, processed_files): """ # makes the parent directory if it doesn't already exist + try: path_dir = os.path.dirname(path) diff --git a/stem/descriptor/remote.py b/stem/descriptor/remote.py index 095f125..b2b5b8e 100644 --- a/stem/descriptor/remote.py +++ b/stem/descriptor/remote.py @@ -80,9 +80,10 @@ import time import zlib try: - import urllib.request as urllib + # account for urllib's change between python 2.x and 3.x + import urllib.request as urllib except ImportError: - import urllib2 as urllib + import urllib2 as urllib import stem.descriptor diff --git a/stem/manual.cfg b/stem/manual.cfg deleted file mode 100644 index 512501e..0000000 --- a/stem/manual.cfg +++ /dev/null @@ -1,352 +0,0 @@ -################################################################################ -# -# Information related to tor configuration options... -# -# * manual.important Most commonly used configuration options. -# * manual.summary Short summary describing the option. -# -################################################################################ - -manual.important BandwidthRate -manual.important BandwidthBurst -manual.important RelayBandwidthRate -manual.important RelayBandwidthBurst -manual.important ControlPort -manual.important HashedControlPassword -manual.important CookieAuthentication -manual.important DataDirectory -manual.important Log -manual.important RunAsDaemon -manual.important User - -manual.important Bridge -manual.important ExcludeNodes -manual.important MaxCircuitDirtiness -manual.important SOCKSPort -manual.important UseBridges - -manual.important BridgeRelay -manual.important ContactInfo -manual.important ExitPolicy -manual.important MyFamily -manual.important Nickname -manual.important ORPort -manual.important PortForwarding -manual.important AccountingMax -manual.important AccountingStart - -manual.important DirPortFrontPage -manual.important DirPort - -manual.important HiddenServiceDir -manual.important HiddenServicePort - -# General Config Options - -manual.summary.BandwidthRate Average bandwidth usage limit -manual.summary.BandwidthBurst Maximum bandwidth usage limit -manual.summary.MaxAdvertisedBandwidth Limit for the bandwidth we advertise as being available for relaying -manual.summary.RelayBandwidthRate Average bandwidth usage limit for relaying -manual.summary.RelayBandwidthBurst Maximum bandwidth usage limit for relaying -manual.summary.PerConnBWRate Average relayed bandwidth limit per connection -manual.summary.PerConnBWBurst Maximum relayed bandwidth limit per connection -manual.summary.ClientTransportPlugin Proxy when establishing bridge connections -manual.summary.ServerTransportPlugin Proxy when servicing bridge connections -manual.summary.ServerTransportListenAddr Endpoint for bridge's pluggable transport proxy -manual.summary.ServerTransportOptions Additional arguments for bridge's proxy -manual.summary.ExtORPort Endpoint for extended ORPort connections -manual.summary.ExtORPortCookieAuthFile Location of the ExtORPort's authentication cookie -manual.summary.ExtORPortCookieAuthFileGroupReadable Group read permissions for the ExtORPort's authentication cookie -manual.summary.ConnLimit Minimum number of file descriptors for Tor to start -manual.summary.DisableNetwork Don't accept non-controller connections -manual.summary.ConstrainedSockets Shrinks sockets to ConstrainedSockSize -manual.summary.ConstrainedSockSize Limit for the received and transmit buffers of sockets -manual.summary.ControlPort Port providing access to tor controllers (nyx, vidalia, etc) -manual.summary.ControlListenAddress Address providing controller access -manual.summary.ControlSocket Socket providing controller access -manual.summary.ControlSocketsGroupWritable Group read permissions for the control socket -manual.summary.HashedControlPassword Hash of the password for authenticating to the control port -manual.summary.CookieAuthentication If set, authenticates controllers via a cookie -manual.summary.CookieAuthFile Location of the authentication cookie -manual.summary.CookieAuthFileGroupReadable Group read permissions for the authentication cookie -manual.summary.ControlPortWriteToFile Path for a file tor writes containing its control port -manual.summary.ControlPortFileGroupReadable Group read permissions for the control port file -manual.summary.DataDirectory Location for storing runtime data (state, keys, etc) -manual.summary.FallbackDir Fallback when unable to retrieve descriptor information -manual.summary.DirAuthority Alternative directory authorities -manual.summary.DirAuthorityFallbackRate Rate at which to use fallback directory -manual.summary.AlternateDirAuthority Alternative directory authorities (consensus only) -manual.summary.AlternateBridgeAuthority Alternative directory authorities (bridges only) -manual.summary.DisableAllSwap Locks all allocated memory so they can't be paged out -manual.summary.DisableDebuggerAttachment Limit information applications can retrieve about the process -manual.summary.FetchDirInfoEarly Keeps consensus information up to date, even if unnecessary -manual.summary.FetchDirInfoExtraEarly Updates consensus information when it's first available -manual.summary.FetchHidServDescriptors Toggles if hidden service descriptors are fetched automatically or not -manual.summary.FetchServerDescriptors Toggles if the consensus is fetched automatically or not -manual.summary.FetchUselessDescriptors Toggles if relay descriptors are fetched when they aren't strictly necessary -manual.summary.HTTPProxy HTTP proxy for connecting to tor -manual.summary.HTTPProxyAuthenticator Authentication credentials for HTTPProxy -manual.summary.HTTPSProxy SSL proxy for connecting to tor -manual.summary.HTTPSProxyAuthenticator Authentication credentials for HTTPSProxy -manual.summary.Sandbox Run within a syscall sandbox -manual.summary.Socks4Proxy SOCKS 4 proxy for connecting to tor -manual.summary.Socks5Proxy SOCKS 5 for connecting to tor -manual.summary.Socks5ProxyUsername Username for connecting to the Socks5Proxy -manual.summary.Socks5ProxyPassword Password for connecting to the Socks5Proxy -manual.summary.SocksSocketsGroupWritable Group write permissions for the socks socket -manual.summary.KeepalivePeriod Rate at which to send keepalive packets -manual.summary.Log Runlevels and location for tor logging -manual.summary.LogMessageDomains Includes a domain when logging messages -manual.summary.OutboundBindAddress Sets the IP used for connecting to tor -manual.summary.PidFile Path for a file tor writes containing its process id -manual.summary.ProtocolWarnings Toggles if protocol errors give warnings or not -manual.summary.PredictedPortsRelevanceTime Duration to ensure circuits for previously used ports remain available -manual.summary.RunAsDaemon Toggles if tor runs as a daemon process -manual.summary.LogTimeGranularity limits granularity of log message timestamps -manual.summary.TruncateLogFile Overwrites log file rather than appending when restarted -manual.summary.SyslogIdentityTag Tag logs appended to the syslog as being from tor -manual.summary.SafeLogging Toggles if logs are scrubbed of sensitive information -manual.summary.User UID for the process when started -manual.summary.HardwareAccel Toggles if tor attempts to use hardware acceleration -manual.summary.AccelName OpenSSL engine name for crypto acceleration -manual.summary.AccelDir Crypto acceleration library path -manual.summary.AvoidDiskWrites Toggles if tor avoids frequently writing to disk -manual.summary.CircuitPriorityHalflife Overwrite method for prioritizing traffic among relayed connections -manual.summary.DisableIOCP Disables use of the Windows IOCP networking API -manual.summary.UserspaceIOCPBuffers Disable kernel-space IOCP TCP buffers -manual.summary.UseFilteringSSLBufferevents Use SSL for a chain of bufferevents -manual.summary.CountPrivateBandwidth Applies rate limiting to private IP addresses - -# Client Config Options - -manual.summary.AllowInvalidNodes Permits use of relays flagged as invalid by authorities -manual.summary.ExcludeSingleHopRelays Permits use of relays that allow single hop connections -manual.summary.Bridge Available bridges -manual.summary.LearnCircuitBuildTimeout Toggles adaptive timeouts for circuit creation -manual.summary.CircuitBuildTimeout Initial timeout for circuit creation -manual.summary.CircuitIdleTimeout Timeout for closing circuits that have never been used -manual.summary.CircuitStreamTimeout Timeout for shifting streams among circuits -manual.summary.ClientOnly Ensures that we aren't used as a relay or directory mirror -manual.summary.ExcludeNodes Relays or locales never to be used in circuits -manual.summary.ExcludeExitNodes Relays or locales never to be used for exits -manual.summary.GeoIPExcludeUnknown Don't use relays with an unknown locale in circuits -manual.summary.ExitNodes Preferred final hop for circuits -manual.summary.EntryNodes Preferred first hops for circuits -manual.summary.StrictNodes Never uses notes outside of Entry/ExitNodes -manual.summary.FascistFirewall Only make outbound connections on FirewallPorts -manual.summary.FirewallPorts Ports used by FascistFirewall -manual.summary.HidServAuth Authentication credentials for connecting to a hidden service -manual.summary.CloseHSClientCircuitsImmediatelyOnTimeout Close hidden service circuits that timeout -manual.summary.CloseHSServiceRendCircuitsImmediatelyOnTimeout Close hidden service rendezvous circuits that timeout -manual.summary.ReachableAddresses Rules for bypassing the local firewall -manual.summary.ReachableDirAddresses Rules for bypassing the local firewall (directory fetches) -manual.summary.ReachableORAddresses Rules for bypassing the local firewall (OR connections) -manual.summary.LongLivedPorts Ports requiring highly reliable relays -manual.summary.MapAddress Alias mappings for address requests -manual.summary.NewCircuitPeriod Period for considering the creation of new circuits -manual.summary.MaxCircuitDirtiness Duration for reusing constructed circuits -manual.summary.MaxClientCircuitsPending Number of circuits that can be in construction at once -manual.summary.NodeFamily Define relays as belonging to a family -manual.summary.EnforceDistinctSubnets Prevent use of multiple relays from the same subnet on a circuit -manual.summary.SOCKSPort Port for using tor as a Socks proxy -manual.summary.SOCKSListenAddress Address from which Socks connections can be made -manual.summary.SocksPolicy Access policy for the pocks port -manual.summary.SocksTimeout Time until idle or unestablished socks connections are closed -manual.summary.TokenBucketRefillInterval Frequency at which exhausted connections are checked for new traffic -manual.summary.TrackHostExits Maintains use of the same exit whenever connecting to this destination -manual.summary.TrackHostExitsExpire Time until use of an exit for tracking expires -manual.summary.UpdateBridgesFromAuthority Toggles fetching bridge descriptors from the authorities -manual.summary.UseBridges Make use of configured bridges -manual.summary.UseEntryGuards Use guard relays for first hop -manual.summary.UseEntryGuardsAsDirGuards Retrieve descriptors via guards when able -manual.summary.GuardfractionFile File containing information with duration of our guards -manual.summary.UseGuardFraction Take guardfraction into account for path selection -manual.summary.NumEntryGuards Pool size of guard relays we'll select from -manual.summary.NumDirectoryGuards Pool size of directory guards we'll select from -manual.summary.GuardLifetime Minimum time to keep entry guards -manual.summary.SafeSocks Toggles rejecting unsafe variants of the socks protocol -manual.summary.TestSocks Provide notices for if socks connections are of the safe or unsafe variants -manual.summary.WarnUnsafeSocks Toggle warning of unsafe socks connection -manual.summary.VirtualAddrNetworkIPv4 IPv4 address range to use when needing a virtual address -manual.summary.VirtualAddrNetworkIPv6 IPv6 address range to use when needing a virtual address -manual.summary.AllowNonRFC953Hostnames Toggles blocking invalid characters in hostname resolution -manual.summary.AllowDotExit Toggles allowing exit notation in addresses -manual.summary.FastFirstHopPK Toggle public key usage for the first hop -manual.summary.TransPort Port for transparent proxying if the OS supports it -manual.summary.TransListenAddress Address from which transparent proxy connections can be made -manual.summary.TransProxyType Proxy type to be used -manual.summary.NATDPort Port for forwarding ipfw NATD connections -manual.summary.NATDListenAddress Address from which NATD forwarded connections can be made -manual.summary.AutomapHostsOnResolve Map addresses ending with special suffixes to virtual addresses -manual.summary.AutomapHostsSuffixes Address suffixes recognized by AutomapHostsOnResolve -manual.summary.DNSPort Port from which DNS responses are fetched instead of tor -manual.summary.DNSListenAddress Address for performing DNS resolution -manual.summary.ClientDNSRejectInternalAddresses Ignores DNS responses for internal addresses -manual.summary.ClientRejectInternalAddresses Disables use of Tor for internal connections -manual.summary.DownloadExtraInfo Toggles fetching of extra information about relays -manual.summary.WarnPlaintextPorts Toggles warnings for using risky ports -manual.summary.RejectPlaintextPorts Prevents connections on risky ports -manual.summary.AllowSingleHopCircuits Makes use of single hop exits if able -manual.summary.OptimisticData Use exits without confirmation that prior connections succeeded -manual.summary.Tor2webMode Establish non-anonymous hidden service connections -manual.summary.Tor2webRendezvousPoints Rendezvous points to use for hidden services when in Tor2webMode -manual.summary.UseMicrodescriptors Retrieve microdescriptors rather than server descriptors -manual.summary.UseNTorHandshake Use ntor for establishing circuits with relays -manual.summary.PathBiasCircThreshold Number of circuits through a guard before applying bias checks -manual.summary.PathBiasNoticeRate Fraction of circuits that must succeed before logging a notice -manual.summary.PathBiasWarnRate Fraction of circuits that must succeed before logging a warning -manual.summary.PathBiasExtremeRate Fraction of circuits that must succeed before logging an error -manual.summary.PathBiasDropGuards Drop guards failing to establish circuits -manual.summary.PathBiasScaleThreshold Circuits through a guard before scaling past observations down -manual.summary.PathBiasUseThreshold Number of streams through a circuit before applying bias checks -manual.summary.PathBiasNoticeUseRate Fraction of streams that must succeed before logging a notice -manual.summary.PathBiasExtremeUseRate Fraction of streams that must succeed before logging an error -manual.summary.PathBiasScaleUseThreshold Streams through a circuit before scaling past observations down -manual.summary.ClientUseIPv6 Allow IPv6 connections to guards -manual.summary.ClientPreferIPv6ORPort Prefer a guard's IPv6 rather than IPv4 endpoint -manual.summary.PathsNeededToBuildCircuits Portion of relays to require information for before making circuits - -# Server Config Options - -manual.summary.Address Overwrites address others will use to reach this relay -manual.summary.AllowSingleHopExits Toggles permitting use of this relay as a single hop proxy -manual.summary.AssumeReachable Skips reachability test at startup -manual.summary.BridgeRelay Act as a bridge -manual.summary.ContactInfo Contact information for this relay -manual.summary.ExitRelay Allow relaying of exit traffic -manual.summary.ExitPolicy Traffic destinations that can exit from this relay -manual.summary.ExitPolicyRejectPrivate Prevent exiting connection on the local network -manual.summary.IPv6Exit Allow clients to use us for IPv6 traffic -manual.summary.MaxOnionQueueDelay Duration to reject new onionskins if we have more than we can process -manual.summary.MyFamily Other relays this operator administers -manual.summary.Nickname Identifier for this relay -manual.summary.NumCPUs Number of processes spawned for decryption -manual.summary.ORPort Port used to accept relay traffic -manual.summary.ORListenAddress Address for relay connections -manual.summary.PortForwarding Use UPnP or NAT-PMP if needed to relay -manual.summary.PortForwardingHelper Executable for configuring port forwarding -manual.summary.PublishServerDescriptor Types of descriptors published -manual.summary.ShutdownWaitLength Delay before quitting after receiving a SIGINT signal -manual.summary.SSLKeyLifetime Lifetime for our link certificate -manual.summary.HeartbeatPeriod Rate at which an INFO level heartbeat message is sent -manual.summary.AccountingMax Amount of traffic before hibernating -manual.summary.AccountingRule Method to determine when the accounting limit is reached -manual.summary.AccountingStart Duration of an accounting period -manual.summary.RefuseUnknownExits Prevents relays not in the consensus from using us as an exit -manual.summary.ServerDNSResolvConfFile Overriding resolver config for DNS queries we provide -manual.summary.ServerDNSAllowBrokenConfig Toggles if we persist despite configuration parsing errors or not -manual.summary.ServerDNSSearchDomains Toggles if our DNS queries search for addresses in the local domain -manual.summary.ServerDNSDetectHijacking Toggles testing for DNS hijacking -manual.summary.ServerDNSTestAddresses Addresses to test to see if valid DNS queries are being hijacked -manual.summary.ServerDNSAllowNonRFC953Hostnames Toggles if we reject DNS queries with invalid characters -manual.summary.BridgeRecordUsageByCountry Tracks geoip information on bridge usage -manual.summary.ServerDNSRandomizeCase Toggles DNS query case randomization -manual.summary.GeoIPFile Path to file containing IPv4 geoip information -manual.summary.GeoIPv6File Path to file containing IPv6 geoip information -manual.summary.TLSECGroup EC group for incoming SSL connections -manual.summary.CellStatistics Toggles storing circuit queue duration to disk -manual.summary.DirReqStatistics Toggles storing network status counts and performance to disk -manual.summary.EntryStatistics Toggles storing client connection counts to disk -manual.summary.ExitPortStatistics Toggles storing traffic and port usage data to disk -manual.summary.ConnDirectionStatistics Toggles storing connection use to disk -manual.summary.HiddenServiceStatistics Toggles storing hidden service stats to disk -manual.summary.ExtraInfoStatistics Publishes statistic data in the extra-info documents -manual.summary.ExtendAllowPrivateAddresses Allow circuits to be extended to the local network -manual.summary.MaxMemInQueues Threshold at which tor will terminate circuits to avoid running out of memory -manual.summary.SigningKeyLifetime Duration the Ed25519 signing key is valid for -manual.summary.OfflineMasterKey Don't generate the master secret key - -# Directory Server Options - -manual.summary.DirPortFrontPage Publish this html file on the DirPort -manual.summary.HidServDirectoryV2 Toggles accepting version 2 hidden service descriptors -manual.summary.DirPort Port for directory connections -manual.summary.DirListenAddress Address the directory service is bound to -manual.summary.DirPolicy Access policy for the DirPort - -# Directory Authority Server Options - -manual.summary.AuthoritativeDirectory Act as a directory authority -manual.summary.V3AuthoritativeDirectory Generates a version 3 consensus -manual.summary.VersioningAuthoritativeDirectory Provides opinions on recommended versions of tor -manual.summary.RecommendedVersions Suggested versions of tor -manual.summary.RecommendedPackageVersions Suggested versions of applications other than tor -manual.summary.RecommendedClientVersions Tor versions believed to be safe for clients -manual.summary.BridgeAuthoritativeDir Acts as a bridge authority -manual.summary.MinUptimeHidServDirectoryV2 Required uptime before accepting hidden service directory -manual.summary.RecommendedServerVersions Tor versions believed to be safe for relays -manual.summary.ConsensusParams Params entry of the networkstatus vote -manual.summary.DirAllowPrivateAddresses Toggles allowing arbitrary input or non-public IPs in descriptors -manual.summary.AuthDirBadExit Relays to be flagged as bad exits -manual.summary.AuthDirInvalid Relays from which the valid flag is withheld -manual.summary.AuthDirReject Relays to be dropped from the consensus -manual.summary.AuthDirBadExitCCs Countries for which to flag all relays as bad exits -manual.summary.AuthDirInvalidCCs Countries for which the valid flag is withheld -manual.summary.AuthDirRejectCCs Countries for which relays aren't accepted into the consensus -manual.summary.AuthDirListBadExits Toggles if we provide an opinion on bad exits -manual.summary.AuthDirMaxServersPerAddr Limit on the number of relays accepted per ip -manual.summary.AuthDirMaxServersPerAuthAddr Limit on the number of relays accepted per an authority's ip -manual.summary.AuthDirFastGuarantee Advertised rate at which the Fast flag is granted -manual.summary.AuthDirGuardBWGuarantee Advertised rate necessary to be a guard -manual.summary.AuthDirPinKeys Don't accept descriptors with conflicting identity keypairs -manual.summary.BridgePassword Password for requesting bridge information -manual.summary.V3AuthVotingInterval Consensus voting interval -manual.summary.V3AuthVoteDelay Wait time to collect votes of other authorities -manual.summary.V3AuthDistDelay Wait time to collect the signatures of other authorities -manual.summary.V3AuthNIntervalsValid Number of voting intervals a consensus is valid for -manual.summary.V3BandwidthsFile Path to a file containing measured relay bandwidths -manual.summary.V3AuthUseLegacyKey Signs consensus with both the current and legacy keys -manual.summary.RephistTrackTime Discards old, unchanged reliability information -manual.summary.VoteOnHidServDirectoriesV2 Determines if the authority votes on hidden service directories -manual.summary.AuthDirHasIPv6Connectivity Descriptors can be retrieved over the authority's IPv6 ORPort -manual.summary.MinMeasuredBWsForAuthToIgnoreAdvertised Total measured value before advertised bandwidths are treated as unreliable - -# Hidden Service Options - -manual.summary.HiddenServiceDir Directory contents for the hidden service -manual.summary.HiddenServicePort Port the hidden service is provided on -manual.summary.PublishHidServDescriptors Toggles automated publishing of the hidden service to the rendezvous directory -manual.summary.HiddenServiceVersion Version for published hidden service descriptors -manual.summary.HiddenServiceAuthorizeClient Restricts access to the hidden service -manual.summary.HiddenServiceAllowUnknownPorts Allow rendezvous circuits on unrecognized ports -manual.summary.HiddenServiceMaxStreams Maximum streams per rendezvous circuit -manual.summary.HiddenServiceMaxStreamsCloseCircuit Closes rendezvous circuits that exceed the maximum number of streams -manual.summary.RendPostPeriod Period at which the rendezvous service descriptors are refreshed -manual.summary.HiddenServiceDirGroupReadable Group read permissions for the hidden service directory -manual.summary.HiddenServiceNumIntroductionPoints Number of introduction points the hidden service will have - -# Testing Network Options - -manual.summary.TestingTorNetwork Overrides other options to be a testing network -manual.summary.TestingV3AuthInitialVotingInterval Overrides V3AuthVotingInterval for the first consensus -manual.summary.TestingV3AuthInitialVoteDelay Overrides TestingV3AuthInitialVoteDelay for the first consensus -manual.summary.TestingV3AuthInitialDistDelay Overrides TestingV3AuthInitialDistDelay for the first consensus -manual.summary.TestingV3AuthVotingStartOffset Offset for the point at which the authority votes -manual.summary.TestingAuthDirTimeToLearnReachability Delay until opinions are given about which relays are running or not -manual.summary.TestingEstimatedDescriptorPropagationTime Delay before clients attempt to fetch descriptors from directory caches -manual.summary.TestingMinFastFlagThreshold Minimum value for the Fast flag -manual.summary.TestingServerDownloadSchedule Schedule for when we should download resources as a relay -manual.summary.TestingClientDownloadSchedule Schedule for when we should download resources as a client -manual.summary.TestingServerConsensusDownloadSchedule Schedule for when we should download the consensus as a relay -manual.summary.TestingClientConsensusDownloadSchedule Schedule for when we should download the consensus as a client -manual.summary.TestingBridgeDownloadSchedule Schedule for when we should download bridge descriptors -manual.summary.TestingClientMaxIntervalWithoutRequest Maximum time to wait to batch requests for missing descriptors -manual.summary.TestingDirConnectionMaxStall Duration to let directory connections stall before timing out -manual.summary.TestingConsensusMaxDownloadTries Retries for downloading the consensus -manual.summary.TestingDescriptorMaxDownloadTries Retries for downloading server descriptors -manual.summary.TestingMicrodescMaxDownloadTries Retries for downloading microdescriptors -manual.summary.TestingCertMaxDownloadTries Retries for downloading authority certificates -manual.summary.TestingDirAuthVoteExit Relays to give the Exit flag to -manual.summary.TestingDirAuthVoteExitIsStrict Only grant the Exit flag to relays listed by TestingDirAuthVoteExit -manual.summary.TestingDirAuthVoteGuard Relays to give the Guard flag to -manual.summary.TestingDirAuthVoteGuardIsStrict Only grant the Guard flag to relays listed by TestingDirAuthVoteGuard -manual.summary.TestingDirAuthVoteHSDir Relays to give the HSDir flag to -manual.summary.TestingDirAuthVoteHSDirIsStrict Only grant the HSDir flag to relays listed by TestingDirAuthVoteHSDir -manual.summary.TestingEnableConnBwEvent Allow controllers to request CONN_BW events -manual.summary.TestingEnableCellStatsEvent Allow controllers to request CELL_STATS events -manual.summary.TestingEnableTbEmptyEvent Allow controllers to request TB_EMPTY events -manual.summary.TestingMinExitFlagThreshold Lower bound for assigning the Exit flag -manual.summary.TestingLinkCertifetime Duration of our ed25519 certificate -manual.summary.TestingAuthKeyLifetime Duration for our ed25519 signing key -manual.summary.TestingLinkKeySlop Time before expiration that we replace our ed25519 key - diff --git a/stem/manual.py b/stem/manual.py index cf1be50..4f51d1f 100644 --- a/stem/manual.py +++ b/stem/manual.py @@ -10,15 +10,20 @@ Provides information available about Tor from `its manual :: is_important - Indicates if a configuration option is of particularly common importance. + download_man_page - Downloads tor's latest man page. Manual - Information about Tor available from its manual. - +- from_man - Retrieves manual information from its man page. + |- from_man - Retrieves manual information from its man page. + +- from_remote - Retrieves manual information remotely from tor's latest manual. .. versionadded:: 1.5.0 """ import collections import os +import shutil +import sys +import tempfile import stem.prereq import stem.util.conf @@ -38,9 +43,17 @@ try: except ImportError: from stem.util.lru_cache import lru_cache +try: + # account for urllib's change between python 2.x and 3.x + import urllib.request as urllib +except ImportError: + import urllib2 as urllib + Category = stem.util.enum.Enum('GENERAL', 'CLIENT', 'RELAY', 'DIRECTORY', 'AUTHORITY', 'HIDDEN_SERVICE', 'TESTING', 'UNKNOWN') ConfigOption = collections.namedtuple('ConfigOption', ['category', 'name', 'usage', 'summary', 'description']) +GITWEB_MANUAL_URL = 'https://gitweb.torproject.org/tor.git/plain/doc/tor.1.txt' + CATEGORY_SECTIONS = { 'GENERAL OPTIONS': Category.GENERAL, 'CLIENT OPTIONS': Category.CLIENT, @@ -55,7 +68,7 @@ CATEGORY_SECTIONS = { @lru_cache() def _config(lowercase = True): """ - Provides a dictionary for our manual.cfg. This has a couple categories... + Provides a dictionary for our settings.cfg. This has a couple categories... * manual.important (list) - configuration options considered to be important * manual.summary.* (str) - summary descriptions of config options @@ -65,7 +78,7 @@ def _config(lowercase = True): """ config = stem.util.conf.Config() - config_path = os.path.join(os.path.dirname(__file__), 'manual.cfg') + config_path = os.path.join(os.path.dirname(__file__), 'settings.cfg') try: config.load(config_path) @@ -90,6 +103,65 @@ def is_important(option): return option.lower() in _config()['manual.important'] +def download_man_page(path = None, file_handle = None, url = GITWEB_MANUAL_URL, timeout = 20): + """ + Downloads tor's latest man page from `gitweb.torproject.org + <https://gitweb.torproject.org/tor.git/plain/doc/tor.1.txt>`_. This method is + both slow and unreliable - please see the warnings on + :func:`~stem.manual.Manual.from_remote`. + + :param str path: path to save tor's man page to + :param file file_handle: file handler to save tor's man page to + :param str url: url to download tor's asciidoc manual from + :param int timeout: seconds to wait before timing out the request + + :raises: **IOError** if unable to retrieve the manual + """ + + if not path and not file_handle: + raise ValueError("Either the path or file_handle we're saving to must be provided") + elif not stem.util.system.is_available('a2x'): + raise IOError('We require a2x from asciidoc to provide a man page') + + dirpath = tempfile.mkdtemp() + asciidoc_path = os.path.join(dirpath, 'tor.1.txt') + manual_path = os.path.join(dirpath, 'tor.1') + + try: + try: + with open(asciidoc_path, 'wb') as asciidoc_file: + request = urllib.urlopen(url, timeout = timeout) + shutil.copyfileobj(request, asciidoc_file) + except: + exc = sys.exc_info()[1] + raise IOError("Unable to download tor's manual from %s to %s: %s" % (url, asciidoc_path, exc)) + + try: + stem.util.system.call('a2x -f manpage %s' % asciidoc_path) + + if not os.path.exists(manual_path): + raise OSError('no man page was generated') + except OSError as exc: + raise IOError("Unable to run 'a2x -f manpage %s': %s" % (asciidoc_path, exc)) + + if path: + try: + path_dir = os.path.dirname(path) + + if not os.path.exists(path_dir): + os.makedirs(path_dir) + + shutil.copyfile(manual_path, path) + except OSError as exc: + raise IOError(exc) + + if file_handle: + with open(manual_path) as manual_file: + shutil.copyfileobj(manual_file, file_handle) + finally: + shutil.rmtree(dirpath) + + class Manual(object): """ Parsed tor man page. Tor makes no guarantees about its man page format so @@ -129,6 +201,10 @@ class Manual(object): :param str man_path: path argument for 'man', for example you might want '/path/to/tor/doc/tor.1' to read from tor's git repository + + :returns: :class:`~stem.manual.Manual` for the system's man page + + :raises: **IOError** if unable to retrieve the manual """ try: @@ -155,6 +231,40 @@ class Manual(object): config_options, ) + @staticmethod + def from_remote(timeout = 20): + """ + Reads and parses the latest tor man page `from gitweb.torproject.org + <https://gitweb.torproject.org/tor.git/plain/doc/tor.1.txt>`_. Note that + while convenient, this reliance on GitWeb means you should alway call with + a fallback, such as... + + :: + + try: + manual = stem.manual.from_remote() + except IOError: + manual = stem.manual.from_cache() + + In addition to our GitWeb dependency this requires 'a2x' which is part of + `asciidoc <http://asciidoc.org/INSTALL.html>`_ and... isn't quick. + Personally this takes ~7.41s, breaking down for me as follows... + + * 1.67s to download tor.1.txt + * 5.57s to convert the asciidoc to a man page + * 0.17s for stem to read and parse the manual + + :param int timeout: seconds to wait before timing out the request + + :returns: latest :class:`~stem.manual.Manual` available for tor + + :raises: **IOError** if unable to retrieve the manual + """ + + with tempfile.NamedTemporaryFile() as tmp: + download_man_page(file_handle = tmp, timeout = timeout) + return Manual.from_man(tmp) + def _get_categories(content): """ diff --git a/stem/settings.cfg b/stem/settings.cfg new file mode 100644 index 0000000..512501e --- /dev/null +++ b/stem/settings.cfg @@ -0,0 +1,352 @@ +################################################################################ +# +# Information related to tor configuration options... +# +# * manual.important Most commonly used configuration options. +# * manual.summary Short summary describing the option. +# +################################################################################ + +manual.important BandwidthRate +manual.important BandwidthBurst +manual.important RelayBandwidthRate +manual.important RelayBandwidthBurst +manual.important ControlPort +manual.important HashedControlPassword +manual.important CookieAuthentication +manual.important DataDirectory +manual.important Log +manual.important RunAsDaemon +manual.important User + +manual.important Bridge +manual.important ExcludeNodes +manual.important MaxCircuitDirtiness +manual.important SOCKSPort +manual.important UseBridges + +manual.important BridgeRelay +manual.important ContactInfo +manual.important ExitPolicy +manual.important MyFamily +manual.important Nickname +manual.important ORPort +manual.important PortForwarding +manual.important AccountingMax +manual.important AccountingStart + +manual.important DirPortFrontPage +manual.important DirPort + +manual.important HiddenServiceDir +manual.important HiddenServicePort + +# General Config Options + +manual.summary.BandwidthRate Average bandwidth usage limit +manual.summary.BandwidthBurst Maximum bandwidth usage limit +manual.summary.MaxAdvertisedBandwidth Limit for the bandwidth we advertise as being available for relaying +manual.summary.RelayBandwidthRate Average bandwidth usage limit for relaying +manual.summary.RelayBandwidthBurst Maximum bandwidth usage limit for relaying +manual.summary.PerConnBWRate Average relayed bandwidth limit per connection +manual.summary.PerConnBWBurst Maximum relayed bandwidth limit per connection +manual.summary.ClientTransportPlugin Proxy when establishing bridge connections +manual.summary.ServerTransportPlugin Proxy when servicing bridge connections +manual.summary.ServerTransportListenAddr Endpoint for bridge's pluggable transport proxy +manual.summary.ServerTransportOptions Additional arguments for bridge's proxy +manual.summary.ExtORPort Endpoint for extended ORPort connections +manual.summary.ExtORPortCookieAuthFile Location of the ExtORPort's authentication cookie +manual.summary.ExtORPortCookieAuthFileGroupReadable Group read permissions for the ExtORPort's authentication cookie +manual.summary.ConnLimit Minimum number of file descriptors for Tor to start +manual.summary.DisableNetwork Don't accept non-controller connections +manual.summary.ConstrainedSockets Shrinks sockets to ConstrainedSockSize +manual.summary.ConstrainedSockSize Limit for the received and transmit buffers of sockets +manual.summary.ControlPort Port providing access to tor controllers (nyx, vidalia, etc) +manual.summary.ControlListenAddress Address providing controller access +manual.summary.ControlSocket Socket providing controller access +manual.summary.ControlSocketsGroupWritable Group read permissions for the control socket +manual.summary.HashedControlPassword Hash of the password for authenticating to the control port +manual.summary.CookieAuthentication If set, authenticates controllers via a cookie +manual.summary.CookieAuthFile Location of the authentication cookie +manual.summary.CookieAuthFileGroupReadable Group read permissions for the authentication cookie +manual.summary.ControlPortWriteToFile Path for a file tor writes containing its control port +manual.summary.ControlPortFileGroupReadable Group read permissions for the control port file +manual.summary.DataDirectory Location for storing runtime data (state, keys, etc) +manual.summary.FallbackDir Fallback when unable to retrieve descriptor information +manual.summary.DirAuthority Alternative directory authorities +manual.summary.DirAuthorityFallbackRate Rate at which to use fallback directory +manual.summary.AlternateDirAuthority Alternative directory authorities (consensus only) +manual.summary.AlternateBridgeAuthority Alternative directory authorities (bridges only) +manual.summary.DisableAllSwap Locks all allocated memory so they can't be paged out +manual.summary.DisableDebuggerAttachment Limit information applications can retrieve about the process +manual.summary.FetchDirInfoEarly Keeps consensus information up to date, even if unnecessary +manual.summary.FetchDirInfoExtraEarly Updates consensus information when it's first available +manual.summary.FetchHidServDescriptors Toggles if hidden service descriptors are fetched automatically or not +manual.summary.FetchServerDescriptors Toggles if the consensus is fetched automatically or not +manual.summary.FetchUselessDescriptors Toggles if relay descriptors are fetched when they aren't strictly necessary +manual.summary.HTTPProxy HTTP proxy for connecting to tor +manual.summary.HTTPProxyAuthenticator Authentication credentials for HTTPProxy +manual.summary.HTTPSProxy SSL proxy for connecting to tor +manual.summary.HTTPSProxyAuthenticator Authentication credentials for HTTPSProxy +manual.summary.Sandbox Run within a syscall sandbox +manual.summary.Socks4Proxy SOCKS 4 proxy for connecting to tor +manual.summary.Socks5Proxy SOCKS 5 for connecting to tor +manual.summary.Socks5ProxyUsername Username for connecting to the Socks5Proxy +manual.summary.Socks5ProxyPassword Password for connecting to the Socks5Proxy +manual.summary.SocksSocketsGroupWritable Group write permissions for the socks socket +manual.summary.KeepalivePeriod Rate at which to send keepalive packets +manual.summary.Log Runlevels and location for tor logging +manual.summary.LogMessageDomains Includes a domain when logging messages +manual.summary.OutboundBindAddress Sets the IP used for connecting to tor +manual.summary.PidFile Path for a file tor writes containing its process id +manual.summary.ProtocolWarnings Toggles if protocol errors give warnings or not +manual.summary.PredictedPortsRelevanceTime Duration to ensure circuits for previously used ports remain available +manual.summary.RunAsDaemon Toggles if tor runs as a daemon process +manual.summary.LogTimeGranularity limits granularity of log message timestamps +manual.summary.TruncateLogFile Overwrites log file rather than appending when restarted +manual.summary.SyslogIdentityTag Tag logs appended to the syslog as being from tor +manual.summary.SafeLogging Toggles if logs are scrubbed of sensitive information +manual.summary.User UID for the process when started +manual.summary.HardwareAccel Toggles if tor attempts to use hardware acceleration +manual.summary.AccelName OpenSSL engine name for crypto acceleration +manual.summary.AccelDir Crypto acceleration library path +manual.summary.AvoidDiskWrites Toggles if tor avoids frequently writing to disk +manual.summary.CircuitPriorityHalflife Overwrite method for prioritizing traffic among relayed connections +manual.summary.DisableIOCP Disables use of the Windows IOCP networking API +manual.summary.UserspaceIOCPBuffers Disable kernel-space IOCP TCP buffers +manual.summary.UseFilteringSSLBufferevents Use SSL for a chain of bufferevents +manual.summary.CountPrivateBandwidth Applies rate limiting to private IP addresses + +# Client Config Options + +manual.summary.AllowInvalidNodes Permits use of relays flagged as invalid by authorities +manual.summary.ExcludeSingleHopRelays Permits use of relays that allow single hop connections +manual.summary.Bridge Available bridges +manual.summary.LearnCircuitBuildTimeout Toggles adaptive timeouts for circuit creation +manual.summary.CircuitBuildTimeout Initial timeout for circuit creation +manual.summary.CircuitIdleTimeout Timeout for closing circuits that have never been used +manual.summary.CircuitStreamTimeout Timeout for shifting streams among circuits +manual.summary.ClientOnly Ensures that we aren't used as a relay or directory mirror +manual.summary.ExcludeNodes Relays or locales never to be used in circuits +manual.summary.ExcludeExitNodes Relays or locales never to be used for exits +manual.summary.GeoIPExcludeUnknown Don't use relays with an unknown locale in circuits +manual.summary.ExitNodes Preferred final hop for circuits +manual.summary.EntryNodes Preferred first hops for circuits +manual.summary.StrictNodes Never uses notes outside of Entry/ExitNodes +manual.summary.FascistFirewall Only make outbound connections on FirewallPorts +manual.summary.FirewallPorts Ports used by FascistFirewall +manual.summary.HidServAuth Authentication credentials for connecting to a hidden service +manual.summary.CloseHSClientCircuitsImmediatelyOnTimeout Close hidden service circuits that timeout +manual.summary.CloseHSServiceRendCircuitsImmediatelyOnTimeout Close hidden service rendezvous circuits that timeout +manual.summary.ReachableAddresses Rules for bypassing the local firewall +manual.summary.ReachableDirAddresses Rules for bypassing the local firewall (directory fetches) +manual.summary.ReachableORAddresses Rules for bypassing the local firewall (OR connections) +manual.summary.LongLivedPorts Ports requiring highly reliable relays +manual.summary.MapAddress Alias mappings for address requests +manual.summary.NewCircuitPeriod Period for considering the creation of new circuits +manual.summary.MaxCircuitDirtiness Duration for reusing constructed circuits +manual.summary.MaxClientCircuitsPending Number of circuits that can be in construction at once +manual.summary.NodeFamily Define relays as belonging to a family +manual.summary.EnforceDistinctSubnets Prevent use of multiple relays from the same subnet on a circuit +manual.summary.SOCKSPort Port for using tor as a Socks proxy +manual.summary.SOCKSListenAddress Address from which Socks connections can be made +manual.summary.SocksPolicy Access policy for the pocks port +manual.summary.SocksTimeout Time until idle or unestablished socks connections are closed +manual.summary.TokenBucketRefillInterval Frequency at which exhausted connections are checked for new traffic +manual.summary.TrackHostExits Maintains use of the same exit whenever connecting to this destination +manual.summary.TrackHostExitsExpire Time until use of an exit for tracking expires +manual.summary.UpdateBridgesFromAuthority Toggles fetching bridge descriptors from the authorities +manual.summary.UseBridges Make use of configured bridges +manual.summary.UseEntryGuards Use guard relays for first hop +manual.summary.UseEntryGuardsAsDirGuards Retrieve descriptors via guards when able +manual.summary.GuardfractionFile File containing information with duration of our guards +manual.summary.UseGuardFraction Take guardfraction into account for path selection +manual.summary.NumEntryGuards Pool size of guard relays we'll select from +manual.summary.NumDirectoryGuards Pool size of directory guards we'll select from +manual.summary.GuardLifetime Minimum time to keep entry guards +manual.summary.SafeSocks Toggles rejecting unsafe variants of the socks protocol +manual.summary.TestSocks Provide notices for if socks connections are of the safe or unsafe variants +manual.summary.WarnUnsafeSocks Toggle warning of unsafe socks connection +manual.summary.VirtualAddrNetworkIPv4 IPv4 address range to use when needing a virtual address +manual.summary.VirtualAddrNetworkIPv6 IPv6 address range to use when needing a virtual address +manual.summary.AllowNonRFC953Hostnames Toggles blocking invalid characters in hostname resolution +manual.summary.AllowDotExit Toggles allowing exit notation in addresses +manual.summary.FastFirstHopPK Toggle public key usage for the first hop +manual.summary.TransPort Port for transparent proxying if the OS supports it +manual.summary.TransListenAddress Address from which transparent proxy connections can be made +manual.summary.TransProxyType Proxy type to be used +manual.summary.NATDPort Port for forwarding ipfw NATD connections +manual.summary.NATDListenAddress Address from which NATD forwarded connections can be made +manual.summary.AutomapHostsOnResolve Map addresses ending with special suffixes to virtual addresses +manual.summary.AutomapHostsSuffixes Address suffixes recognized by AutomapHostsOnResolve +manual.summary.DNSPort Port from which DNS responses are fetched instead of tor +manual.summary.DNSListenAddress Address for performing DNS resolution +manual.summary.ClientDNSRejectInternalAddresses Ignores DNS responses for internal addresses +manual.summary.ClientRejectInternalAddresses Disables use of Tor for internal connections +manual.summary.DownloadExtraInfo Toggles fetching of extra information about relays +manual.summary.WarnPlaintextPorts Toggles warnings for using risky ports +manual.summary.RejectPlaintextPorts Prevents connections on risky ports +manual.summary.AllowSingleHopCircuits Makes use of single hop exits if able +manual.summary.OptimisticData Use exits without confirmation that prior connections succeeded +manual.summary.Tor2webMode Establish non-anonymous hidden service connections +manual.summary.Tor2webRendezvousPoints Rendezvous points to use for hidden services when in Tor2webMode +manual.summary.UseMicrodescriptors Retrieve microdescriptors rather than server descriptors +manual.summary.UseNTorHandshake Use ntor for establishing circuits with relays +manual.summary.PathBiasCircThreshold Number of circuits through a guard before applying bias checks +manual.summary.PathBiasNoticeRate Fraction of circuits that must succeed before logging a notice +manual.summary.PathBiasWarnRate Fraction of circuits that must succeed before logging a warning +manual.summary.PathBiasExtremeRate Fraction of circuits that must succeed before logging an error +manual.summary.PathBiasDropGuards Drop guards failing to establish circuits +manual.summary.PathBiasScaleThreshold Circuits through a guard before scaling past observations down +manual.summary.PathBiasUseThreshold Number of streams through a circuit before applying bias checks +manual.summary.PathBiasNoticeUseRate Fraction of streams that must succeed before logging a notice +manual.summary.PathBiasExtremeUseRate Fraction of streams that must succeed before logging an error +manual.summary.PathBiasScaleUseThreshold Streams through a circuit before scaling past observations down +manual.summary.ClientUseIPv6 Allow IPv6 connections to guards +manual.summary.ClientPreferIPv6ORPort Prefer a guard's IPv6 rather than IPv4 endpoint +manual.summary.PathsNeededToBuildCircuits Portion of relays to require information for before making circuits + +# Server Config Options + +manual.summary.Address Overwrites address others will use to reach this relay +manual.summary.AllowSingleHopExits Toggles permitting use of this relay as a single hop proxy +manual.summary.AssumeReachable Skips reachability test at startup +manual.summary.BridgeRelay Act as a bridge +manual.summary.ContactInfo Contact information for this relay +manual.summary.ExitRelay Allow relaying of exit traffic +manual.summary.ExitPolicy Traffic destinations that can exit from this relay +manual.summary.ExitPolicyRejectPrivate Prevent exiting connection on the local network +manual.summary.IPv6Exit Allow clients to use us for IPv6 traffic +manual.summary.MaxOnionQueueDelay Duration to reject new onionskins if we have more than we can process +manual.summary.MyFamily Other relays this operator administers +manual.summary.Nickname Identifier for this relay +manual.summary.NumCPUs Number of processes spawned for decryption +manual.summary.ORPort Port used to accept relay traffic +manual.summary.ORListenAddress Address for relay connections +manual.summary.PortForwarding Use UPnP or NAT-PMP if needed to relay +manual.summary.PortForwardingHelper Executable for configuring port forwarding +manual.summary.PublishServerDescriptor Types of descriptors published +manual.summary.ShutdownWaitLength Delay before quitting after receiving a SIGINT signal +manual.summary.SSLKeyLifetime Lifetime for our link certificate +manual.summary.HeartbeatPeriod Rate at which an INFO level heartbeat message is sent +manual.summary.AccountingMax Amount of traffic before hibernating +manual.summary.AccountingRule Method to determine when the accounting limit is reached +manual.summary.AccountingStart Duration of an accounting period +manual.summary.RefuseUnknownExits Prevents relays not in the consensus from using us as an exit +manual.summary.ServerDNSResolvConfFile Overriding resolver config for DNS queries we provide +manual.summary.ServerDNSAllowBrokenConfig Toggles if we persist despite configuration parsing errors or not +manual.summary.ServerDNSSearchDomains Toggles if our DNS queries search for addresses in the local domain +manual.summary.ServerDNSDetectHijacking Toggles testing for DNS hijacking +manual.summary.ServerDNSTestAddresses Addresses to test to see if valid DNS queries are being hijacked +manual.summary.ServerDNSAllowNonRFC953Hostnames Toggles if we reject DNS queries with invalid characters +manual.summary.BridgeRecordUsageByCountry Tracks geoip information on bridge usage +manual.summary.ServerDNSRandomizeCase Toggles DNS query case randomization +manual.summary.GeoIPFile Path to file containing IPv4 geoip information +manual.summary.GeoIPv6File Path to file containing IPv6 geoip information +manual.summary.TLSECGroup EC group for incoming SSL connections +manual.summary.CellStatistics Toggles storing circuit queue duration to disk +manual.summary.DirReqStatistics Toggles storing network status counts and performance to disk +manual.summary.EntryStatistics Toggles storing client connection counts to disk +manual.summary.ExitPortStatistics Toggles storing traffic and port usage data to disk +manual.summary.ConnDirectionStatistics Toggles storing connection use to disk +manual.summary.HiddenServiceStatistics Toggles storing hidden service stats to disk +manual.summary.ExtraInfoStatistics Publishes statistic data in the extra-info documents +manual.summary.ExtendAllowPrivateAddresses Allow circuits to be extended to the local network +manual.summary.MaxMemInQueues Threshold at which tor will terminate circuits to avoid running out of memory +manual.summary.SigningKeyLifetime Duration the Ed25519 signing key is valid for +manual.summary.OfflineMasterKey Don't generate the master secret key + +# Directory Server Options + +manual.summary.DirPortFrontPage Publish this html file on the DirPort +manual.summary.HidServDirectoryV2 Toggles accepting version 2 hidden service descriptors +manual.summary.DirPort Port for directory connections +manual.summary.DirListenAddress Address the directory service is bound to +manual.summary.DirPolicy Access policy for the DirPort + +# Directory Authority Server Options + +manual.summary.AuthoritativeDirectory Act as a directory authority +manual.summary.V3AuthoritativeDirectory Generates a version 3 consensus +manual.summary.VersioningAuthoritativeDirectory Provides opinions on recommended versions of tor +manual.summary.RecommendedVersions Suggested versions of tor +manual.summary.RecommendedPackageVersions Suggested versions of applications other than tor +manual.summary.RecommendedClientVersions Tor versions believed to be safe for clients +manual.summary.BridgeAuthoritativeDir Acts as a bridge authority +manual.summary.MinUptimeHidServDirectoryV2 Required uptime before accepting hidden service directory +manual.summary.RecommendedServerVersions Tor versions believed to be safe for relays +manual.summary.ConsensusParams Params entry of the networkstatus vote +manual.summary.DirAllowPrivateAddresses Toggles allowing arbitrary input or non-public IPs in descriptors +manual.summary.AuthDirBadExit Relays to be flagged as bad exits +manual.summary.AuthDirInvalid Relays from which the valid flag is withheld +manual.summary.AuthDirReject Relays to be dropped from the consensus +manual.summary.AuthDirBadExitCCs Countries for which to flag all relays as bad exits +manual.summary.AuthDirInvalidCCs Countries for which the valid flag is withheld +manual.summary.AuthDirRejectCCs Countries for which relays aren't accepted into the consensus +manual.summary.AuthDirListBadExits Toggles if we provide an opinion on bad exits +manual.summary.AuthDirMaxServersPerAddr Limit on the number of relays accepted per ip +manual.summary.AuthDirMaxServersPerAuthAddr Limit on the number of relays accepted per an authority's ip +manual.summary.AuthDirFastGuarantee Advertised rate at which the Fast flag is granted +manual.summary.AuthDirGuardBWGuarantee Advertised rate necessary to be a guard +manual.summary.AuthDirPinKeys Don't accept descriptors with conflicting identity keypairs +manual.summary.BridgePassword Password for requesting bridge information +manual.summary.V3AuthVotingInterval Consensus voting interval +manual.summary.V3AuthVoteDelay Wait time to collect votes of other authorities +manual.summary.V3AuthDistDelay Wait time to collect the signatures of other authorities +manual.summary.V3AuthNIntervalsValid Number of voting intervals a consensus is valid for +manual.summary.V3BandwidthsFile Path to a file containing measured relay bandwidths +manual.summary.V3AuthUseLegacyKey Signs consensus with both the current and legacy keys +manual.summary.RephistTrackTime Discards old, unchanged reliability information +manual.summary.VoteOnHidServDirectoriesV2 Determines if the authority votes on hidden service directories +manual.summary.AuthDirHasIPv6Connectivity Descriptors can be retrieved over the authority's IPv6 ORPort +manual.summary.MinMeasuredBWsForAuthToIgnoreAdvertised Total measured value before advertised bandwidths are treated as unreliable + +# Hidden Service Options + +manual.summary.HiddenServiceDir Directory contents for the hidden service +manual.summary.HiddenServicePort Port the hidden service is provided on +manual.summary.PublishHidServDescriptors Toggles automated publishing of the hidden service to the rendezvous directory +manual.summary.HiddenServiceVersion Version for published hidden service descriptors +manual.summary.HiddenServiceAuthorizeClient Restricts access to the hidden service +manual.summary.HiddenServiceAllowUnknownPorts Allow rendezvous circuits on unrecognized ports +manual.summary.HiddenServiceMaxStreams Maximum streams per rendezvous circuit +manual.summary.HiddenServiceMaxStreamsCloseCircuit Closes rendezvous circuits that exceed the maximum number of streams +manual.summary.RendPostPeriod Period at which the rendezvous service descriptors are refreshed +manual.summary.HiddenServiceDirGroupReadable Group read permissions for the hidden service directory +manual.summary.HiddenServiceNumIntroductionPoints Number of introduction points the hidden service will have + +# Testing Network Options + +manual.summary.TestingTorNetwork Overrides other options to be a testing network +manual.summary.TestingV3AuthInitialVotingInterval Overrides V3AuthVotingInterval for the first consensus +manual.summary.TestingV3AuthInitialVoteDelay Overrides TestingV3AuthInitialVoteDelay for the first consensus +manual.summary.TestingV3AuthInitialDistDelay Overrides TestingV3AuthInitialDistDelay for the first consensus +manual.summary.TestingV3AuthVotingStartOffset Offset for the point at which the authority votes +manual.summary.TestingAuthDirTimeToLearnReachability Delay until opinions are given about which relays are running or not +manual.summary.TestingEstimatedDescriptorPropagationTime Delay before clients attempt to fetch descriptors from directory caches +manual.summary.TestingMinFastFlagThreshold Minimum value for the Fast flag +manual.summary.TestingServerDownloadSchedule Schedule for when we should download resources as a relay +manual.summary.TestingClientDownloadSchedule Schedule for when we should download resources as a client +manual.summary.TestingServerConsensusDownloadSchedule Schedule for when we should download the consensus as a relay +manual.summary.TestingClientConsensusDownloadSchedule Schedule for when we should download the consensus as a client +manual.summary.TestingBridgeDownloadSchedule Schedule for when we should download bridge descriptors +manual.summary.TestingClientMaxIntervalWithoutRequest Maximum time to wait to batch requests for missing descriptors +manual.summary.TestingDirConnectionMaxStall Duration to let directory connections stall before timing out +manual.summary.TestingConsensusMaxDownloadTries Retries for downloading the consensus +manual.summary.TestingDescriptorMaxDownloadTries Retries for downloading server descriptors +manual.summary.TestingMicrodescMaxDownloadTries Retries for downloading microdescriptors +manual.summary.TestingCertMaxDownloadTries Retries for downloading authority certificates +manual.summary.TestingDirAuthVoteExit Relays to give the Exit flag to +manual.summary.TestingDirAuthVoteExitIsStrict Only grant the Exit flag to relays listed by TestingDirAuthVoteExit +manual.summary.TestingDirAuthVoteGuard Relays to give the Guard flag to +manual.summary.TestingDirAuthVoteGuardIsStrict Only grant the Guard flag to relays listed by TestingDirAuthVoteGuard +manual.summary.TestingDirAuthVoteHSDir Relays to give the HSDir flag to +manual.summary.TestingDirAuthVoteHSDirIsStrict Only grant the HSDir flag to relays listed by TestingDirAuthVoteHSDir +manual.summary.TestingEnableConnBwEvent Allow controllers to request CONN_BW events +manual.summary.TestingEnableCellStatsEvent Allow controllers to request CELL_STATS events +manual.summary.TestingEnableTbEmptyEvent Allow controllers to request TB_EMPTY events +manual.summary.TestingMinExitFlagThreshold Lower bound for assigning the Exit flag +manual.summary.TestingLinkCertifetime Duration of our ed25519 certificate +manual.summary.TestingAuthKeyLifetime Duration for our ed25519 signing key +manual.summary.TestingLinkKeySlop Time before expiration that we replace our ed25519 key + diff --git a/test/integ/manual.py b/test/integ/manual.py new file mode 100644 index 0000000..02de19e --- /dev/null +++ b/test/integ/manual.py @@ -0,0 +1,213 @@ +""" +Integ testing for the stem.manual module, fetching the latest man page from the +tor git repository and checking for new additions. +""" + +import codecs +import os +import tempfile +import unittest + +import stem.manual +import stem.util.system +import test.runner + +from stem.manual import Category + +EXPECTED_CATEGORIES = set([ + 'NAME', + 'SYNOPSIS', + 'DESCRIPTION', + 'COMMAND-LINE OPTIONS', + 'THE CONFIGURATION FILE FORMAT', + 'GENERAL OPTIONS', + 'CLIENT OPTIONS', + 'SERVER OPTIONS', + 'DIRECTORY SERVER OPTIONS', + 'DIRECTORY AUTHORITY SERVER OPTIONS', + 'HIDDEN SERVICE OPTIONS', + 'TESTING NETWORK OPTIONS', + 'SIGNALS', + 'FILES', + 'SEE ALSO', + 'BUGS', + 'AUTHORS', +]) + +EXPECTED_CLI_OPTIONS = set(['-h, -help', '-f FILE', '--allow-missing-torrc', '--defaults-torrc FILE', '--ignore-missing-torrc', '--hash-password PASSWORD', '--list-fingerprint', '--verify-config', '--service install [--options command-line options]', '--service remove|start|stop', '--nt-service', '--list-torrc-options', '--version', '--quiet|--hush']) +EXPECTED_SIGNALS = set(['SIGTERM', 'SIGINT', 'SIGHUP', 'SIGUSR1', 'SIGUSR2', 'SIGCHLD', 'SIGPIPE', 'SIGXFSZ']) + +EXPECTED_OPTION_COUNTS = { + Category.GENERAL: 74, + Category.CLIENT: 86, + Category.RELAY: 47, + Category.DIRECTORY: 5, + Category.AUTHORITY: 34, + Category.HIDDEN_SERVICE: 11, + Category.TESTING: 32, + Category.UNKNOWN: 0, +} + +EXPECTED_DESCRIPTION = """ +Tor is a connection-oriented anonymizing communication service. Users choose a source-routed path through a set of nodes, and negotiate a "virtual circuit" through the network, in which each node knows its predecessor and successor, but no others. Traffic flowing down the circuit is unwrapped by a symmetric key at each node, which reveals the downstream node. + +Basically, Tor provides a distributed network of servers or relays ("onion routers"). Users bounce their TCP streams - web traffic, ftp, ssh, etc. - around the network, and recipients, observers, and even the relays themselves have difficulty tracking the source of the stream. + +By default, tor will only act as a client only. To help the network by providing bandwidth as a relay, change the ORPort configuration option - see below. Please also consult the documentation on the Tor Project's website. +""".strip() + +EXPECTED_FILE_DESCRIPTION = 'Specify a new configuration file to contain further Tor configuration options OR pass - to make Tor read its configuration from standard input. (Default: @CONFDIR@/torrc, or $HOME/.torrc if that file is not found)' + +EXPECTED_BANDWIDTH_RATE_DESCRIPTION = 'A token bucket limits the average incoming bandwidth usage on this node to the specified number of bytes per second, and the average outgoing bandwidth usage to that same value. If you want to run a relay in the public network, this needs to be at the very least 75 KBytes for a relay (that is, 600 kbits) or 50 KBytes for a bridge (400 kbits) - but of course, more is better; we recommend at least 250 KBytes (2 mbits) if possible. (Default: 1 GByte)\n\nWith this option, and in other options that take arguments in bytes, KBytes, and so on, other formats are also supported. Notably, "KBytes" can also be written as "kilobytes" or "kb"; "MBytes" can be written as "megabytes" or "MB"; "kbits" can be written as "kilobits"; and so forth. Tor also accepts "byte" and "bit" in the singular. The prefixes "tera" and "T" are also recognized. If no units are given, we default to bytes. To avoid confusion, we recommend writing "bytes" or "bits" explicitly, sinc e it\'s easy to forget that "B" means bytes, not bits.' + + +class TestManual(unittest.TestCase): + @classmethod + def setUpClass(self): + self.man_path = None + self.man_content = None + self.skip_reason = None + self.download_error = None + + if stem.util.system.is_windows(): + self.skip_reason = '(unavailable on windows)' + elif test.runner.Target.ONLINE not in test.runner.get_runner().attribute_targets: + self.skip_reason = '(requires online target)' + elif not stem.util.system.is_available('a2x'): + self.skip_reason = '(requires asciidoc)' + else: + try: + with tempfile.NamedTemporaryFile(prefix = 'tor_man_page.', delete = False) as tmp: + stem.manual.download_man_page(file_handle = tmp) + self.man_path = tmp.name + + self.man_content = stem.util.system.call('man -P cat %s' % self.man_path) + except Exception as exc: + self.download_error = 'Unable to download the man page: %s' % exc + + @classmethod + def tearDownClass(self): + if self.man_path and os.path.exists(self.man_path): + os.remove(self.man_path) + + def requires_downloaded_manual(self): + if self.skip_reason: + test.runner.skip(self, self.skip_reason) + return True + elif self.download_error: + self.fail(self.download_error) + + return False + + def test_get_categories(self): + if self.requires_downloaded_manual(): + return + + categories = stem.manual._get_categories(self.man_content) + + present = set(categories.keys()) + missing_categories = present.difference(EXPECTED_CATEGORIES) + extra_categories = EXPECTED_CATEGORIES.difference(present) + + if missing_categories: + self.fail("Changed tor's man page? We expected the %s man page sections but they're no longer around, if expected then please update our test." % ', '.join(missing_categories)) + elif extra_categories: + self.fail("Changed tor's man page? We weren't expecting the %s man page sections, if expected then please update our test." % ', '.join(extra_categories)) + + self.assertEqual(['tor - The second-generation onion router'], categories['NAME']) + self.assertEqual(['tor [OPTION value]...'], categories['SYNOPSIS']) + self.assertEqual(8, len(categories['DESCRIPTION'])) # check parsing of multi-line entries + + def test_escapes_non_ascii(self): + if self.requires_downloaded_manual(): + return + + def check(content): + try: + codecs.ascii_encode(content, 'strict') + except UnicodeEncodeError as exc: + self.fail("Unable to read '%s' as ascii: %s" % (content, exc)) + + categories = stem.manual._get_categories(self.man_content) + + for category, lines in categories.items(): + check(category) + + for line in lines: + check(line) + + def test_has_all_summaries(self): + if self.requires_downloaded_manual(): + return + + manual = stem.manual.Manual.from_man(self.man_path) + present = set(manual.config_options.keys()) + expected = set([key[15:] for key in stem.manual._config(lowercase = False) if key.startswith('manual.summary.')]) + + # TODO: The 'Recognized' config name is due to our man page being slightly + # malformed. Sending a tor patch later to fix it. + + missing_options = present.difference(expected).difference(set(['Recognized'])) + extra_options = expected.difference(present) + + if missing_options: + self.fail("Changed tor's man page? Please update Stem's settings.cfg with summaries of the following config options: %s" % ', '.join(missing_options)) + elif extra_options: + self.fail("Changed tor's man page? Please remove the following summaries from Stem's settings.cfg: %s" % ', '.join(extra_options)) + + def test_attributes(self): + if self.requires_downloaded_manual(): + return + + def assert_equal(category, expected, actual): + if expected != actual: + self.fail("Changed tor's man page? The %s changed as follows...\n\nexpected: %s\n\nactual: %s" % (category, expected, actual)) + + manual = stem.manual.Manual.from_man(self.man_path) + + assert_equal('name', 'tor - The second-generation onion router', manual.name) + assert_equal('synopsis', 'tor [OPTION value]...', manual.synopsis) + assert_equal('description', EXPECTED_DESCRIPTION, manual.description) + + assert_equal('commandline options', EXPECTED_CLI_OPTIONS, set(manual.commandline_options.keys())) + assert_equal('help option', 'Display a short help message and exit.', manual.commandline_options['-h, -help']) + assert_equal('file option', EXPECTED_FILE_DESCRIPTION, manual.commandline_options['-f FILE']) + + assert_equal('signals', EXPECTED_SIGNALS, set(manual.signals.keys())) + assert_equal('sighup description', 'Tor will catch this, clean up and sync to disk if necessary, and exit.', manual.signals['SIGTERM']) + + assert_equal('number of files', 31, len(manual.files)) + assert_equal('lib path description', 'The tor process stores keys and other data here.', manual.files['@LOCALSTATEDIR@/lib/tor/']) + + for category, expected_count in EXPECTED_OPTION_COUNTS.items(): + assert_equal('number of %s category entries' % category, expected_count, len([entry for entry in manual.config_options.values() if entry.category == category])) + + option = manual.config_options['BandwidthRate'] + self.assertEqual(Category.GENERAL, option.category) + self.assertEqual('BandwidthRate', option.name) + self.assertEqual('N bytes|KBytes|MBytes|GBytes|KBits|MBits|GBits', option.usage) + self.assertEqual('Average bandwidth usage limit', option.summary) + self.assertEqual(EXPECTED_BANDWIDTH_RATE_DESCRIPTION, option.description) + + def test_with_unknown_options(self): + if stem.util.system.is_windows(): + test.runner.skip(self, '(unavailable on windows)') + return + + manual = stem.manual.Manual.from_man(os.path.join(os.path.dirname(__file__), 'tor.1_with_unknown')) + + self.assertEqual('tor - The second-generation onion router', manual.name) + self.assertEqual('', manual.synopsis) + self.assertEqual('', manual.description) + self.assertEqual({}, manual.commandline_options) + self.assertEqual({}, manual.signals) + self.assertEqual({}, manual.files) + + self.assertEqual(2, len(manual.config_options)) + + option = [entry for entry in manual.config_options.values() if entry.category == Category.UNKNOWN][0] + self.assertEqual(Category.UNKNOWN, option.category) + self.assertEqual('SpiffyNewOption', option.name) + self.assertEqual('transport exec path-to-binary [options]', option.usage) + self.assertEqual('', option.summary) + self.assertEqual('Description of this new option.', option.description) diff --git a/test/integ/tor.1_with_unknown b/test/integ/tor.1_with_unknown new file mode 100644 index 0000000..b5e0b82 --- /dev/null +++ b/test/integ/tor.1_with_unknown @@ -0,0 +1,51 @@ +'\" t +.\" Title: tor +.\" Author: [see the "AUTHORS" section] +.\" Generator: DocBook XSL Stylesheets v1.76.1 <http://docbook.sf.net/> +.\" Date: 10/03/2015 +.\" Manual: Tor Manual +.\" Source: Tor +.\" Language: English +.\" +.TH "TOR" "1" "10/03/2015" "Tor" "Tor Manual" +.\" ----------------------------------------------------------------- +.\" * Define some portability stuff +.\" ----------------------------------------------------------------- +.\" ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +.\" http://bugs.debian.org/507673 +.\" http://lists.gnu.org/archive/html/groff/2009-02/msg00013.html +.\" ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +.ie \n(.g .ds Aq \(aq +.el .ds Aq ' +.\" ----------------------------------------------------------------- +.\" * set default formatting +.\" ----------------------------------------------------------------- +.\" disable hyphenation +.nh +.\" disable justification (adjust text to left margin only) +.ad l +.\" ----------------------------------------------------------------- +.\" * MAIN CONTENT STARTS HERE * +.\" ----------------------------------------------------------------- +.SH "NAME" +tor \- The second\-generation onion router +.SH "GENERAL OPTIONS" +.PP +\fBBandwidthRate\fR \fIN\fR \fBbytes\fR|\fBKBytes\fR|\fBMBytes\fR|\fBGBytes\fR|\fBKBits\fR|\fBMBits\fR|\fBGBits\fR +.RS 4 +A token bucket limits the average incoming bandwidth usage on this node to the specified number of bytes per second, and the average outgoing bandwidth usage to that same value\&. If you want to run a relay in the public network, this needs to be +\fIat the very least\fR +30 KBytes (that is, 30720 bytes)\&. (Default: 1 GByte) + + +With this option, and in other options that take arguments in bytes, KBytes, and so on, other formats are also supported\&. Notably, "KBytes" can also be written as "kilobytes" or "kb"; "MBytes" can be written as "megabytes" or "MB"; "kbits" can be written as "kilobits"; and so forth\&. Tor also accepts "byte" and "bit" in the singular\&. The prefixes "tera" and "T" are also recognized\&. If no units are given, we default to bytes\&. To avoid confusion, we recommend writing "bytes" or "bits" explicitly, since it\(cqs easy to forget that "B" means bytes, not bits\&. +.RE +.PP +.SH "NEW OPTIONS" +.PP +\fBSpiffyNewOption\fR \fItransport\fR exec \fIpath\-to\-binary\fR [options] +.RS 4 +Description of this new option. +.RE +.PP + diff --git a/test/settings.cfg b/test/settings.cfg index 13fb342..6812a90 100644 --- a/test/settings.cfg +++ b/test/settings.cfg @@ -211,6 +211,7 @@ test.integ_tests |test.integ.descriptor.microdescriptor.TestMicrodescriptor |test.integ.descriptor.networkstatus.TestNetworkStatus |test.integ.version.TestVersion +|test.integ.manual.TestManual |test.integ.response.protocolinfo.TestProtocolInfo |test.integ.process.TestProcess |test.integ.socket.control_socket.TestControlSocket diff --git a/test/unit/manual.py b/test/unit/manual.py index 60ae291..7ca69da 100644 --- a/test/unit/manual.py +++ b/test/unit/manual.py @@ -1,77 +1,32 @@ """ -Unit tessts for the stem.manual module. Test data comes from the following... - - * test/unit/tor.1 - Tor version 0.2.8.0-alpha-dev (git-3c6782395743a089) +Unit testing for the stem.manual module. """ -import codecs -import os +import io import unittest +import stem.prereq import stem.manual -import stem.util.system -import test.runner -from stem.manual import Category +try: + # account for urllib's change between python 2.x and 3.x + import urllib.request as urllib +except ImportError: + import urllib2 as urllib + +try: + # added in python 3.3 + from unittest.mock import Mock, patch +except ImportError: + from mock import Mock, patch try: - # added in python 3.2 - from functools import lru_cache + # added in python 2.7 + from collections import OrderedDict except ImportError: - from stem.util.lru_cache import lru_cache - -TEST_MAN_PAGE = os.path.join(os.path.dirname(__file__), 'tor.1') - -EXPECTED_CATEGORIES = set([ - 'NAME', - 'SYNOPSIS', - 'DESCRIPTION', - 'COMMAND-LINE OPTIONS', - 'THE CONFIGURATION FILE FORMAT', - 'GENERAL OPTIONS', - 'CLIENT OPTIONS', - 'SERVER OPTIONS', - 'DIRECTORY SERVER OPTIONS', - 'DIRECTORY AUTHORITY SERVER OPTIONS', - 'HIDDEN SERVICE OPTIONS', - 'TESTING NETWORK OPTIONS', - 'SIGNALS', - 'FILES', - 'SEE ALSO', - 'BUGS', - 'AUTHORS', -]) - -EXPECTED_CLI_OPTIONS = set(['-h, -help', '-f FILE', '--allow-missing-torrc', '--defaults-torrc FILE', '--ignore-missing-torrc', '--hash-password PASSWORD', '--list-fingerprint', '--verify-config', '--service install [--options command-line options]', '--service remove|start|stop', '--nt-service', '--list-torrc-options', '--version', '--quiet|--hush']) -EXPECTED_SIGNALS = set(['SIGTERM', 'SIGINT', 'SIGHUP', 'SIGUSR1', 'SIGUSR2', 'SIGCHLD', 'SIGPIPE', 'SIGXFSZ']) - -EXPECTED_OPTION_COUNTS = { - Category.GENERAL: 74, - Category.CLIENT: 86, - Category.RELAY: 47, - Category.DIRECTORY: 5, - Category.AUTHORITY: 34, - Category.HIDDEN_SERVICE: 11, - Category.TESTING: 32, - Category.UNKNOWN: 0, -} - -EXPECTED_DESCRIPTION = """ -Tor is a connection-oriented anonymizing communication service. Users choose a source-routed path through a set of nodes, and negotiate a "virtual circuit" through the network, in which each node knows its predecessor and successor, but no others. Traffic flowing down the circuit is unwrapped by a symmetric key at each node, which reveals the downstream node. - -Basically, Tor provides a distributed network of servers or relays ("onion routers"). Users bounce their TCP streams - web traffic, ftp, ssh, etc. - around the network, and recipients, observers, and even the relays themselves have difficulty tracking the source of the stream. - -By default, tor will only act as a client only. To help the network by providing bandwidth as a relay, change the ORPort configuration option - see below. Please also consult the documentation on the Tor Project's website. -""".strip() - -EXPECTED_FILE_DESCRIPTION = 'Specify a new configuration file to contain further Tor configuration options OR pass - to make Tor read its configuration from standard input. (Default: /usr/local/etc/tor/torrc, or $HOME/.torrc if that file is not found)' - -EXPECTED_BANDWIDTH_RATE_DESCRIPTION = 'A token bucket limits the average incoming bandwidth usage on this node to the specified number of bytes per second, and the average outgoing bandwidth usage to that same value. If you want to run a relay in the public network, this needs to be at the very least 30 KBytes (that is, 30720 bytes). (Default: 1 GByte)\n\nWith this option, and in other options that take arguments in bytes, KBytes, and so on, other formats are also supported. Notably, "KBytes" can also be written as "kilobytes" or "kb"; "MBytes" can be written as "megabytes" or "MB"; "kbits" can be written as "kilobits"; and so forth. Tor also accepts "byte" and "bit" in the singular. The prefixes "tera" and "T" are also recognized. If no units are given, we default to bytes. To avoid confusion, we recommend writing "bytes" or "bits" explicitly, since it\'s easy to forget that "B" means bytes, not bits.' - - -@lru_cache() -def man_content(): - return stem.util.system.call('man -P cat %s' % TEST_MAN_PAGE) + from stem.util.ordereddict import OrderedDict + +URL_OPEN = 'urllib.request.urlopen' if stem.prereq.is_python_3() else 'urllib2.urlopen' class TestManual(unittest.TestCase): @@ -82,105 +37,89 @@ class TestManual(unittest.TestCase): self.assertFalse(stem.manual.is_important('ConstrainedSockSize')) - def test_get_categories(self): - if stem.util.system.is_windows(): - test.runner.skip(self, '(unavailable on windows)') - return - - categories = stem.manual._get_categories(man_content()) - self.assertEqual(EXPECTED_CATEGORIES, set(categories.keys())) - self.assertEqual(['tor - The second-generation onion router'], categories['NAME']) - self.assertEqual(['tor [OPTION value]...'], categories['SYNOPSIS']) - self.assertEqual(8, len(categories['DESCRIPTION'])) # check parsing of multi-line entries - - def test_escapes_non_ascii(self): - if stem.util.system.is_windows(): - test.runner.skip(self, '(unavailable on windows)') - return - - def check(content): - try: - codecs.ascii_encode(content, 'strict') - except UnicodeEncodeError as exc: - self.fail("Unable to read '%s' as ascii: %s" % (content, exc)) - - categories = stem.manual._get_categories(man_content()) - - for category, lines in categories.items(): - check(category) - - for line in lines: - check(line) - - def test_has_all_summaries(self): - if stem.util.system.is_windows(): - test.runner.skip(self, '(unavailable on windows)') - return - - manual = stem.manual.Manual.from_man(TEST_MAN_PAGE) - present = set(manual.config_options.keys()) - expected = set([key[15:] for key in stem.manual._config(lowercase = False) if key.startswith('manual.summary.')]) - - # TODO: The 'Recognized' config name is due to our man page being slightly - # malformed. Sending a tor patch later to fix it. - - missing_options = present.difference(expected).difference(set(['Recognized'])) - extra_options = expected.difference(present) - - if missing_options: - self.fail("The following config options are missing summaries: %s" % ', '.join(missing_options)) - elif extra_options: - self.fail("The following config options no longer exist in tor, so don't need summaries: %s" % ', '.join(extra_options)) - - def test_attributes(self): - if stem.util.system.is_windows(): - test.runner.skip(self, '(unavailable on windows)') - return - - manual = stem.manual.Manual.from_man(TEST_MAN_PAGE) - - self.assertEqual('tor - The second-generation onion router', manual.name) - self.assertEqual('tor [OPTION value]...', manual.synopsis) - self.assertEqual(EXPECTED_DESCRIPTION, manual.description) - - self.assertEqual(EXPECTED_CLI_OPTIONS, set(manual.commandline_options.keys())) - self.assertEqual('Display a short help message and exit.', manual.commandline_options['-h, -help']) - self.assertEqual(EXPECTED_FILE_DESCRIPTION, manual.commandline_options['-f FILE']) - - self.assertEqual(EXPECTED_SIGNALS, set(manual.signals.keys())) - self.assertEqual('Tor will catch this, clean up and sync to disk if necessary, and exit.', manual.signals['SIGTERM']) - - self.assertEqual(31, len(manual.files)) - self.assertEqual('The tor process stores keys and other data here.', manual.files['/usr/local/var/lib/tor/']) - - for category, expected_count in EXPECTED_OPTION_COUNTS.items(): - self.assertEqual(expected_count, len([entry for entry in manual.config_options.values() if entry.category == category])) - - option = manual.config_options['BandwidthRate'] - self.assertEqual(Category.GENERAL, option.category) - self.assertEqual('BandwidthRate', option.name) - self.assertEqual('N bytes|KBytes|MBytes|GBytes|KBits|MBits|GBits', option.usage) - self.assertEqual('Average bandwidth usage limit', option.summary) - self.assertEqual(EXPECTED_BANDWIDTH_RATE_DESCRIPTION, option.description) - - def test_with_unknown_options(self): - if stem.util.system.is_windows(): - test.runner.skip(self, '(unavailable on windows)') - return - - manual = stem.manual.Manual.from_man(TEST_MAN_PAGE + '_with_unknown') - - self.assertEqual('tor - The second-generation onion router', manual.name) + def test_download_man_page_without_arguments(self): + try: + stem.manual.download_man_page() + self.fail('we should fail without a path or file handler') + except ValueError as exc: + self.assertEqual("Either the path or file_handle we're saving to must be provided", str(exc)) + + @patch('stem.util.system.is_available', Mock(return_value = False)) + def test_download_man_page_requires_a2x(self): + try: + stem.manual.download_man_page('/tmp/no_such_file') + self.fail('we should require a2x to be available') + except IOError as exc: + self.assertEqual('We require a2x from asciidoc to provide a man page', str(exc)) + + @patch('tempfile.mkdtemp', Mock(return_value = '/no/such/path')) + @patch('shutil.rmtree', Mock()) + @patch('stem.manual.open', Mock(side_effect = IOError('unable to write to file')), create = True) + def test_download_man_page_when_unable_to_write(self): + try: + stem.manual.download_man_page('/tmp/no_such_file') + self.fail("we shouldn't be able to write to /no/such/path") + except IOError as exc: + self.assertEqual("Unable to download tor's manual from https://gitweb.torproject.org/tor.git/plain/doc/tor.1.txt to /no/such/path/tor.1.txt: unable to write to file", str(exc)) + + @patch('tempfile.mkdtemp', Mock(return_value = '/no/such/path')) + @patch('shutil.rmtree', Mock()) + @patch('stem.manual.open', Mock(return_value = io.BytesIO()), create = True) + @patch(URL_OPEN, Mock(side_effect = urllib.URLError('<urlopen error [Errno -2] Name or service not known>'))) + def test_download_man_page_when_download_fails(self): + try: + stem.manual.download_man_page('/tmp/no_such_file', url = 'https://www.atagar.com/foo/bar') + self.fail("downloading from test_invalid_url.org shouldn't work") + except IOError as exc: + self.assertEqual("Unable to download tor's manual from https://www.atagar.com/foo/bar to /no/such/path/tor.1.txt: <urlopen error <urlopen error [Errno -2] Name or service not known>>", str(exc)) + + @patch('tempfile.mkdtemp', Mock(return_value = '/no/such/path')) + @patch('shutil.rmtree', Mock()) + @patch('stem.manual.open', Mock(return_value = io.BytesIO()), create = True) + @patch('stem.util.system.call', Mock(side_effect = OSError('call failed'))) + @patch(URL_OPEN, Mock(return_value = io.BytesIO(b'test content'))) + def test_download_man_page_when_a2x_fails(self): + try: + stem.manual.download_man_page('/tmp/no_such_file', url = 'https://www.atagar.com/foo/bar') + self.fail("downloading from test_invalid_url.org shouldn't work") + except IOError as exc: + self.assertEqual("Unable to run 'a2x -f manpage /no/such/path/tor.1.txt': call failed", str(exc)) + + @patch('tempfile.mkdtemp', Mock(return_value = '/no/such/path')) + @patch('shutil.rmtree', Mock()) + @patch('stem.manual.open', create = True) + @patch('stem.util.system.call') + @patch('os.path.exists', Mock(return_value = True)) + @patch(URL_OPEN, Mock(return_value = io.BytesIO(b'test content'))) + def test_download_man_page_when_successful(self, call_mock, open_mock): + open_mock.side_effect = lambda path, *args: { + '/no/such/path/tor.1.txt': io.BytesIO(), + '/no/such/path/tor.1': io.BytesIO(b'a2x output'), + }[path] + + call_mock.return_value = Mock() + + output = io.BytesIO() + stem.manual.download_man_page(file_handle = output) + self.assertEqual(b'a2x output', output.getvalue()) + call_mock.assert_called_once_with('a2x -f manpage /no/such/path/tor.1.txt') + + @patch('stem.util.system.call', Mock(side_effect = OSError('man -P cat tor returned exit status 16'))) + def test_from_man_when_manual_is_unavailable(self): + try: + stem.manual.Manual.from_man() + self.fail("fetching the manual should fail when it's unavailable") + except IOError as exc: + self.assertEqual("Unable to run 'man -P cat tor': man -P cat tor returned exit status 16", str(exc)) + + @patch('stem.util.system.call', Mock(return_value = [])) + def test_when_man_is_empty(self): + manual = stem.manual.Manual.from_man() + + self.assertEqual('', manual.name) self.assertEqual('', manual.synopsis) self.assertEqual('', manual.description) self.assertEqual({}, manual.commandline_options) self.assertEqual({}, manual.signals) - - self.assertEqual(2, len(manual.config_options)) - - option = [entry for entry in manual.config_options.values() if entry.category == Category.UNKNOWN][0] - self.assertEqual(Category.UNKNOWN, option.category) - self.assertEqual('SpiffyNewOption', option.name) - self.assertEqual('transport exec path-to-binary [options]', option.usage) - self.assertEqual('', option.summary) - self.assertEqual('Description of this new option.', option.description) + self.assertEqual({}, manual.files) + self.assertEqual(OrderedDict(), manual.config_options) diff --git a/test/unit/tor.1_with_unknown b/test/unit/tor.1_with_unknown deleted file mode 100644 index b5e0b82..0000000 --- a/test/unit/tor.1_with_unknown +++ /dev/null @@ -1,51 +0,0 @@ -'\" t -.\" Title: tor -.\" Author: [see the "AUTHORS" section] -.\" Generator: DocBook XSL Stylesheets v1.76.1 <http://docbook.sf.net/> -.\" Date: 10/03/2015 -.\" Manual: Tor Manual -.\" Source: Tor -.\" Language: English -.\" -.TH "TOR" "1" "10/03/2015" "Tor" "Tor Manual" -.\" ----------------------------------------------------------------- -.\" * Define some portability stuff -.\" ----------------------------------------------------------------- -.\" ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -.\" http://bugs.debian.org/507673 -.\" http://lists.gnu.org/archive/html/groff/2009-02/msg00013.html -.\" ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -.ie \n(.g .ds Aq \(aq -.el .ds Aq ' -.\" ----------------------------------------------------------------- -.\" * set default formatting -.\" ----------------------------------------------------------------- -.\" disable hyphenation -.nh -.\" disable justification (adjust text to left margin only) -.ad l -.\" ----------------------------------------------------------------- -.\" * MAIN CONTENT STARTS HERE * -.\" ----------------------------------------------------------------- -.SH "NAME" -tor \- The second\-generation onion router -.SH "GENERAL OPTIONS" -.PP -\fBBandwidthRate\fR \fIN\fR \fBbytes\fR|\fBKBytes\fR|\fBMBytes\fR|\fBGBytes\fR|\fBKBits\fR|\fBMBits\fR|\fBGBits\fR -.RS 4 -A token bucket limits the average incoming bandwidth usage on this node to the specified number of bytes per second, and the average outgoing bandwidth usage to that same value\&. If you want to run a relay in the public network, this needs to be -\fIat the very least\fR -30 KBytes (that is, 30720 bytes)\&. (Default: 1 GByte) - - -With this option, and in other options that take arguments in bytes, KBytes, and so on, other formats are also supported\&. Notably, "KBytes" can also be written as "kilobytes" or "kb"; "MBytes" can be written as "megabytes" or "MB"; "kbits" can be written as "kilobits"; and so forth\&. Tor also accepts "byte" and "bit" in the singular\&. The prefixes "tera" and "T" are also recognized\&. If no units are given, we default to bytes\&. To avoid confusion, we recommend writing "bytes" or "bits" explicitly, since it\(cqs easy to forget that "B" means bytes, not bits\&. -.RE -.PP -.SH "NEW OPTIONS" -.PP -\fBSpiffyNewOption\fR \fItransport\fR exec \fIpath\-to\-binary\fR [options] -.RS 4 -Description of this new option. -.RE -.PP -

1 0

[stem/master] Test that tor's config options match the man page
by atagar＠torproject.org 06 Dec '15

06 Dec '15

commit 5bd9f98cb5b2d48ffba5c8298650b8f7b08059ab Author: Damian Johnson <atagar(a)torproject.org> Date: Mon Nov 23 18:41:15 2015 -0800 Test that tor's config options match the man page Adding an integ test to check that the configuration options tor provides matches the man page. This way when new additions are made to tor we'll have a nice reminder to document them in the manual as well. In doing this ran into a number of minor issues with the man page. Shot Nick some patches... https://trac.torproject.org/projects/tor/ticket/17666 https://trac.torproject.org/projects/tor/ticket/17665 --- stem/settings.cfg | 2 +- test/integ/manual.py | 196 +++++++++++++++++++++++++++++++++++++------------- 2 files changed, 148 insertions(+), 50 deletions(-) diff --git a/stem/settings.cfg b/stem/settings.cfg index 512501e..d770dfc 100644 --- a/stem/settings.cfg +++ b/stem/settings.cfg @@ -346,7 +346,7 @@ manual.summary.TestingEnableConnBwEvent Allow controllers to request CONN_BW eve manual.summary.TestingEnableCellStatsEvent Allow controllers to request CELL_STATS events manual.summary.TestingEnableTbEmptyEvent Allow controllers to request TB_EMPTY events manual.summary.TestingMinExitFlagThreshold Lower bound for assigning the Exit flag -manual.summary.TestingLinkCertifetime Duration of our ed25519 certificate +manual.summary.TestingLinkCertLifetime Duration of our ed25519 certificate manual.summary.TestingAuthKeyLifetime Duration for our ed25519 signing key manual.summary.TestingLinkKeySlop Time before expiration that we replace our ed25519 key diff --git a/test/integ/manual.py b/test/integ/manual.py index 02de19e..72410d2 100644 --- a/test/integ/manual.py +++ b/test/integ/manual.py @@ -99,26 +99,42 @@ class TestManual(unittest.TestCase): return False - def test_get_categories(self): - if self.requires_downloaded_manual(): + def test_with_unknown_options(self): + """ + Check that we can read a local mock man page that contains unrecognized + options. Unlike most other tests this doesn't require network access. + """ + + if stem.util.system.is_windows(): + test.runner.skip(self, '(unavailable on windows)') # needs to run 'man' return - categories = stem.manual._get_categories(self.man_content) + manual = stem.manual.Manual.from_man(os.path.join(os.path.dirname(__file__), 'tor.1_with_unknown')) - present = set(categories.keys()) - missing_categories = present.difference(EXPECTED_CATEGORIES) - extra_categories = EXPECTED_CATEGORIES.difference(present) + self.assertEqual('tor - The second-generation onion router', manual.name) + self.assertEqual('', manual.synopsis) + self.assertEqual('', manual.description) + self.assertEqual({}, manual.commandline_options) + self.assertEqual({}, manual.signals) + self.assertEqual({}, manual.files) - if missing_categories: - self.fail("Changed tor's man page? We expected the %s man page sections but they're no longer around, if expected then please update our test." % ', '.join(missing_categories)) - elif extra_categories: - self.fail("Changed tor's man page? We weren't expecting the %s man page sections, if expected then please update our test." % ', '.join(extra_categories)) + self.assertEqual(2, len(manual.config_options)) - self.assertEqual(['tor - The second-generation onion router'], categories['NAME']) - self.assertEqual(['tor [OPTION value]...'], categories['SYNOPSIS']) - self.assertEqual(8, len(categories['DESCRIPTION'])) # check parsing of multi-line entries + option = [entry for entry in manual.config_options.values() if entry.category == Category.UNKNOWN][0] + self.assertEqual(Category.UNKNOWN, option.category) + self.assertEqual('SpiffyNewOption', option.name) + self.assertEqual('transport exec path-to-binary [options]', option.usage) + self.assertEqual('', option.summary) + self.assertEqual('Description of this new option.', option.description) def test_escapes_non_ascii(self): + """ + Check that our manual parser escapes all non-ascii characters. If this + fails then that means someone probably added a new type of non-ascii + character. Easy to fix: please simply add an escape for it in + stem/manual.py's _get_categories(). + """ + if self.requires_downloaded_manual(): return @@ -136,26 +152,12 @@ class TestManual(unittest.TestCase): for line in lines: check(line) - def test_has_all_summaries(self): - if self.requires_downloaded_manual(): - return - - manual = stem.manual.Manual.from_man(self.man_path) - present = set(manual.config_options.keys()) - expected = set([key[15:] for key in stem.manual._config(lowercase = False) if key.startswith('manual.summary.')]) - - # TODO: The 'Recognized' config name is due to our man page being slightly - # malformed. Sending a tor patch later to fix it. - - missing_options = present.difference(expected).difference(set(['Recognized'])) - extra_options = expected.difference(present) - - if missing_options: - self.fail("Changed tor's man page? Please update Stem's settings.cfg with summaries of the following config options: %s" % ', '.join(missing_options)) - elif extra_options: - self.fail("Changed tor's man page? Please remove the following summaries from Stem's settings.cfg: %s" % ', '.join(extra_options)) - def test_attributes(self): + """ + General assertions against a few manual fields. If you update tor's manual + then go ahead and simply update these assertions. + """ + if self.requires_downloaded_manual(): return @@ -189,25 +191,121 @@ class TestManual(unittest.TestCase): self.assertEqual('Average bandwidth usage limit', option.summary) self.assertEqual(EXPECTED_BANDWIDTH_RATE_DESCRIPTION, option.description) - def test_with_unknown_options(self): - if stem.util.system.is_windows(): - test.runner.skip(self, '(unavailable on windows)') + def test_has_all_categories(self): + """ + Check that the categories in tor's manual matches what we expect. If these + change then we likely want to add/remove attributes from Stem's Manual + class to match. + """ + + if self.requires_downloaded_manual(): return - manual = stem.manual.Manual.from_man(os.path.join(os.path.dirname(__file__), 'tor.1_with_unknown')) + categories = stem.manual._get_categories(self.man_content) - self.assertEqual('tor - The second-generation onion router', manual.name) - self.assertEqual('', manual.synopsis) - self.assertEqual('', manual.description) - self.assertEqual({}, manual.commandline_options) - self.assertEqual({}, manual.signals) - self.assertEqual({}, manual.files) + present = set(categories.keys()) + missing_categories = present.difference(EXPECTED_CATEGORIES) + extra_categories = EXPECTED_CATEGORIES.difference(present) - self.assertEqual(2, len(manual.config_options)) + if missing_categories: + self.fail("Changed tor's man page? We expected the %s man page sections but they're no longer around. Might need to update our Manual class." % ', '.join(missing_categories)) + elif extra_categories: + self.fail("Changed tor's man page? We weren't expecting the %s man page sections. Might need to update our Manual class." % ', '.join(extra_categories)) - option = [entry for entry in manual.config_options.values() if entry.category == Category.UNKNOWN][0] - self.assertEqual(Category.UNKNOWN, option.category) - self.assertEqual('SpiffyNewOption', option.name) - self.assertEqual('transport exec path-to-binary [options]', option.usage) - self.assertEqual('', option.summary) - self.assertEqual('Description of this new option.', option.description) + self.assertEqual(['tor - The second-generation onion router'], categories['NAME']) + self.assertEqual(['tor [OPTION value]...'], categories['SYNOPSIS']) + self.assertEqual(8, len(categories['DESCRIPTION'])) # check parsing of multi-line entries + + def test_has_all_summaries(self): + """ + Check that we have brief, human readable summaries for all of tor's + configuration options. If you add a new config entry then please take a sec + to write a little summary. They're located in 'stem/settings.cfg'. + """ + + if self.requires_downloaded_manual(): + return + + manual = stem.manual.Manual.from_man(self.man_path) + present = set(manual.config_options.keys()) + expected = set([key[15:] for key in stem.manual._config(lowercase = False) if key.startswith('manual.summary.')]) + + # TODO: Typo in man page (s/TestingLinkCertifetime/TestingLinkCertLifetime) + + present.remove('TestingLinkCertifetime') + present.add('TestingLinkCertLifetime') + + # TODO: The 'Recognized' config name is due to our man page being slightly + # malformed. Sending a tor patch later to fix it. + + missing_options = present.difference(expected).difference(set(['Recognized'])) + extra_options = expected.difference(present) + + if missing_options: + self.fail("Changed tor's man page? Please update Stem's settings.cfg with summaries of the following config options: %s" % ', '.join(missing_options)) + elif extra_options: + self.fail("Changed tor's man page? Please remove the following summaries from Stem's settings.cfg: %s" % ', '.join(extra_options)) + + def test_has_all_tor_config_options(self): + """ + Check that all the configuration options tor supports are in the man page. + """ + + if self.requires_downloaded_manual(): + return + + with test.runner.get_runner().get_tor_controller() as controller: + config_options_in_tor = set([line.split()[0] for line in controller.get_info('config/names').splitlines()]) + + # options starting with an underscore are hidden by convention + + for name in list(config_options_in_tor): + if name.startswith('_'): + config_options_in_tor.remove(name) + + # hidden service options are a special snowflake + + if 'HiddenServiceOptions' in config_options_in_tor: + config_options_in_tor.remove('HiddenServiceOptions') + + # TODO: Addressing some errors in the man page I'll be sending fixes for. + + config_options_in_tor.remove('SocksPort') + config_options_in_tor.add('SOCKSPort') + + config_options_in_tor.remove('SocksListenAddress') + config_options_in_tor.add('SOCKSListenAddress') + + config_options_in_tor.remove('TestingLinkCertLifetime') + config_options_in_tor.add('TestingLinkCertifetime') + + config_options_in_tor.remove('TestingSigningKeySlop') + config_options_in_tor.remove('TestingAuthKeySlop') + + config_options_in_tor.remove('RecommendedPackages') + config_options_in_tor.add('RecommendedPackageVersions') + + config_options_in_tor.add('Recognized') + config_options_in_tor.add('VoteOnHidServDirectoriesV2') + config_options_in_tor.add('HidServDirectoryV2') + + # TODO: Looks like options we should remove from tor... + # + # https://trac.torproject.org/projects/tor/ticket/17665 + + config_options_in_tor.remove('SchedulerMaxFlushCells__') + config_options_in_tor.remove('SchedulerLowWaterMark__') + config_options_in_tor.remove('SchedulerHighWaterMark__') + + manual = stem.manual.Manual.from_man(self.man_path) + config_options_in_manual = set(manual.config_options.keys()) + + missing_from_manual = config_options_in_tor.difference(config_options_in_manual) + + if missing_from_manual: + self.fail("The %s config options supported by tor isn't in its man page. Maybe we need to add them?" % ', '.join(missing_from_manual)) + + extra_in_manual = config_options_in_manual.difference(config_options_in_tor) + + if extra_in_manual: + self.fail("The %s config options in our man page aren't presently supported by tor. Maybe we need to remove them?" % ', '.join(extra_in_manual))

1 0

[stem/master] Example for using our manual module
by atagar＠torproject.org 06 Dec '15

06 Dec '15

commit 4b90bd04a1b48780957e4d8435cb2fd236604aee Author: Damian Johnson <atagar(a)torproject.org> Date: Tue Nov 24 09:30:41 2015 -0800 Example for using our manual module Expanding our manual module's header docs. Describing how our various methods of getting manual information differ and adding an example. --- docs/_static/example/manual_config_options.py | 30 ++++++++++++++++++++++++ docs/_static/manual_output.png | Bin 0 -> 54570 bytes stem/manual.py | 31 ++++++++++++++++++++++--- 3 files changed, 58 insertions(+), 3 deletions(-) diff --git a/docs/_static/example/manual_config_options.py b/docs/_static/example/manual_config_options.py new file mode 100644 index 0000000..964ff52 --- /dev/null +++ b/docs/_static/example/manual_config_options.py @@ -0,0 +1,30 @@ +from stem.manual import Manual +from stem.util import term + +try: + print("Downloading tor's manual information, please wait...") + manual = Manual.from_remote() + print(" done\n") +except IOError as exc: + print(" unsuccessful (%s), using information provided with stem\n" % exc) + manual = Manual.from_cache() # fall back to our bundled manual information + +print('Which tor configuration would you like to learn about? (press ctrl+c to quit)\n') + +try: + while True: + requested_option = raw_input('> ').strip() + + if requested_option: + if requested_option in manual.config_options: + option = manual.config_options[requested_option] + print(term.format('%s %s' % (option.name, option.usage), term.Color.GREEN, term.Attr.BOLD)) + print(term.format(option.summary, term.Color.GREEN)) # brief description provided by stem + + print(term.format('\nFull Description:\n', term.Color.GREEN, term.Attr.BOLD)) + print(term.format(option.description + '\n', term.Color.GREEN)) + else: + print(term.format("Sorry, we don't have any information about %s. Are you sure it's an option?" % requested_option, term.Color.RED)) +except KeyboardInterrupt: + pass # user pressed ctrl+c + diff --git a/docs/_static/manual_output.png b/docs/_static/manual_output.png new file mode 100644 index 0000000..4fccd94 Binary files /dev/null and b/docs/_static/manual_output.png differ diff --git a/stem/manual.py b/stem/manual.py index 4f51d1f..1e2a8f1 100644 --- a/stem/manual.py +++ b/stem/manual.py @@ -2,8 +2,33 @@ # See LICENSE for licensing information """ -Provides information available about Tor from `its manual -<https://www.torproject.org/docs/tor-manual.html.en>`_. +Information available about Tor from `its manual +<https://www.torproject.org/docs/tor-manual.html.en>`_. This provides three +methods of getting this information... + +* :func:`~stem.manual.Manual.from_cache` provides manual content bundled with + Stem. This is the fastest and most reliable method but only as up-to-date as + Stem's release. + +* :func:`~stem.manual.Manual.from_man` reads Tor's local man page for + information about it. + +* :func:`~stem.manual.Manual.from_remote` fetches the latest manual information + remotely. This is the slowest and least reliable method but provides the most + recent information about Tor. + +Manual information includes arguments, signals, and probably most usefully the +torrc configuration options. For example, say we want a little script that told +us what our torrc options do... + +.. literalinclude:: /_static/example/manual_config_options.py + :language: python + +| + +.. image:: /_static/manual_output.png + +| **Module Overview:** @@ -263,7 +288,7 @@ class Manual(object): with tempfile.NamedTemporaryFile() as tmp: download_man_page(file_handle = tmp, timeout = timeout) - return Manual.from_man(tmp) + return Manual.from_man(tmp.name) def _get_categories(content):

1 0

[stem/master] Drop tor man page workarounds
by atagar＠torproject.org 06 Dec '15

06 Dec '15

commit f65af3782f39e966197ad2e911f39a36d2e2d2d2 Author: Damian Johnson <atagar(a)torproject.org> Date: Wed Nov 25 07:59:24 2015 -0800 Drop tor man page workarounds Nick's merged our man page fixes so we can now drop our workarounds - yay! \o/ https://trac.torproject.org/projects/tor/ticket/17666 I'm dropping our assertions on exact counts of configuration options since they'll break with every man page update and aren't really helpful. Instead just checking that the categories themselves still exist and that we recognize all the categories. --- stem/settings.cfg | 14 +++++++------- test/integ/manual.py | 52 +++++++++----------------------------------------- 2 files changed, 16 insertions(+), 50 deletions(-) diff --git a/stem/settings.cfg b/stem/settings.cfg index d770dfc..78bf014 100644 --- a/stem/settings.cfg +++ b/stem/settings.cfg @@ -22,7 +22,7 @@ manual.important User manual.important Bridge manual.important ExcludeNodes manual.important MaxCircuitDirtiness -manual.important SOCKSPort +manual.important SocksPort manual.important UseBridges manual.important BridgeRelay @@ -148,8 +148,8 @@ manual.summary.MaxCircuitDirtiness Duration for reusing constructed circuits manual.summary.MaxClientCircuitsPending Number of circuits that can be in construction at once manual.summary.NodeFamily Define relays as belonging to a family manual.summary.EnforceDistinctSubnets Prevent use of multiple relays from the same subnet on a circuit -manual.summary.SOCKSPort Port for using tor as a Socks proxy -manual.summary.SOCKSListenAddress Address from which Socks connections can be made +manual.summary.SocksPort Port for using tor as a Socks proxy +manual.summary.SocksListenAddress Address from which Socks connections can be made manual.summary.SocksPolicy Access policy for the pocks port manual.summary.SocksTimeout Time until idle or unestablished socks connections are closed manual.summary.TokenBucketRefillInterval Frequency at which exhausted connections are checked for new traffic @@ -259,7 +259,6 @@ manual.summary.OfflineMasterKey Don't generate the master secret key # Directory Server Options manual.summary.DirPortFrontPage Publish this html file on the DirPort -manual.summary.HidServDirectoryV2 Toggles accepting version 2 hidden service descriptors manual.summary.DirPort Port for directory connections manual.summary.DirListenAddress Address the directory service is bound to manual.summary.DirPolicy Access policy for the DirPort @@ -270,7 +269,7 @@ manual.summary.AuthoritativeDirectory Act as a directory authority manual.summary.V3AuthoritativeDirectory Generates a version 3 consensus manual.summary.VersioningAuthoritativeDirectory Provides opinions on recommended versions of tor manual.summary.RecommendedVersions Suggested versions of tor -manual.summary.RecommendedPackageVersions Suggested versions of applications other than tor +manual.summary.RecommendedPackages Suggested versions of applications other than tor manual.summary.RecommendedClientVersions Tor versions believed to be safe for clients manual.summary.BridgeAuthoritativeDir Acts as a bridge authority manual.summary.MinUptimeHidServDirectoryV2 Required uptime before accepting hidden service directory @@ -297,7 +296,6 @@ manual.summary.V3AuthNIntervalsValid Number of voting intervals a consensus is v manual.summary.V3BandwidthsFile Path to a file containing measured relay bandwidths manual.summary.V3AuthUseLegacyKey Signs consensus with both the current and legacy keys manual.summary.RephistTrackTime Discards old, unchanged reliability information -manual.summary.VoteOnHidServDirectoriesV2 Determines if the authority votes on hidden service directories manual.summary.AuthDirHasIPv6Connectivity Descriptors can be retrieved over the authority's IPv6 ORPort manual.summary.MinMeasuredBWsForAuthToIgnoreAdvertised Total measured value before advertised bandwidths are treated as unreliable @@ -348,5 +346,7 @@ manual.summary.TestingEnableTbEmptyEvent Allow controllers to request TB_EMPTY e manual.summary.TestingMinExitFlagThreshold Lower bound for assigning the Exit flag manual.summary.TestingLinkCertLifetime Duration of our ed25519 certificate manual.summary.TestingAuthKeyLifetime Duration for our ed25519 signing key -manual.summary.TestingLinkKeySlop Time before expiration that we replace our ed25519 key +manual.summary.TestingLinkKeySlop Time before expiration that we replace our ed25519 link key +manual.summary.TestingAuthKeySlop Time before expiration that we replace our ed25519 authentication key +manual.summary.TestingSigningKeySlop Time before expiration that we replace our ed25519 signing key diff --git a/test/integ/manual.py b/test/integ/manual.py index 72410d2..6a01526 100644 --- a/test/integ/manual.py +++ b/test/integ/manual.py @@ -37,17 +37,6 @@ EXPECTED_CATEGORIES = set([ EXPECTED_CLI_OPTIONS = set(['-h, -help', '-f FILE', '--allow-missing-torrc', '--defaults-torrc FILE', '--ignore-missing-torrc', '--hash-password PASSWORD', '--list-fingerprint', '--verify-config', '--service install [--options command-line options]', '--service remove|start|stop', '--nt-service', '--list-torrc-options', '--version', '--quiet|--hush']) EXPECTED_SIGNALS = set(['SIGTERM', 'SIGINT', 'SIGHUP', 'SIGUSR1', 'SIGUSR2', 'SIGCHLD', 'SIGPIPE', 'SIGXFSZ']) -EXPECTED_OPTION_COUNTS = { - Category.GENERAL: 74, - Category.CLIENT: 86, - Category.RELAY: 47, - Category.DIRECTORY: 5, - Category.AUTHORITY: 34, - Category.HIDDEN_SERVICE: 11, - Category.TESTING: 32, - Category.UNKNOWN: 0, -} - EXPECTED_DESCRIPTION = """ Tor is a connection-oriented anonymizing communication service. Users choose a source-routed path through a set of nodes, and negotiate a "virtual circuit" through the network, in which each node knows its predecessor and successor, but no others. Traffic flowing down the circuit is unwrapped by a symmetric key at each node, which reveals the downstream node. @@ -181,8 +170,14 @@ class TestManual(unittest.TestCase): assert_equal('number of files', 31, len(manual.files)) assert_equal('lib path description', 'The tor process stores keys and other data here.', manual.files['@LOCALSTATEDIR@/lib/tor/']) - for category, expected_count in EXPECTED_OPTION_COUNTS.items(): - assert_equal('number of %s category entries' % category, expected_count, len([entry for entry in manual.config_options.values() if entry.category == category])) + for category in Category: + if len([entry for entry in manual.config_options.values() if entry.category == category]) == 0 and category != Category.UNKNOWN: + self.fail('We had an empty %s section, did we intentionally drop it?' % category) + + unknown_options = [entry for entry in manual.config_options.values() if entry.category == Category.UNKNOWN] + + if unknown_options: + self.fail("We don't recognize the category for the %s options. Maybe a new man page section? If so then please update the Category enum in stem/manual.py." % ', '.join(unknown_options)) option = manual.config_options['BandwidthRate'] self.assertEqual(Category.GENERAL, option.category) @@ -230,15 +225,7 @@ class TestManual(unittest.TestCase): present = set(manual.config_options.keys()) expected = set([key[15:] for key in stem.manual._config(lowercase = False) if key.startswith('manual.summary.')]) - # TODO: Typo in man page (s/TestingLinkCertifetime/TestingLinkCertLifetime) - - present.remove('TestingLinkCertifetime') - present.add('TestingLinkCertLifetime') - - # TODO: The 'Recognized' config name is due to our man page being slightly - # malformed. Sending a tor patch later to fix it. - - missing_options = present.difference(expected).difference(set(['Recognized'])) + missing_options = present.difference(expected) extra_options = expected.difference(present) if missing_options: @@ -268,27 +255,6 @@ class TestManual(unittest.TestCase): if 'HiddenServiceOptions' in config_options_in_tor: config_options_in_tor.remove('HiddenServiceOptions') - # TODO: Addressing some errors in the man page I'll be sending fixes for. - - config_options_in_tor.remove('SocksPort') - config_options_in_tor.add('SOCKSPort') - - config_options_in_tor.remove('SocksListenAddress') - config_options_in_tor.add('SOCKSListenAddress') - - config_options_in_tor.remove('TestingLinkCertLifetime') - config_options_in_tor.add('TestingLinkCertifetime') - - config_options_in_tor.remove('TestingSigningKeySlop') - config_options_in_tor.remove('TestingAuthKeySlop') - - config_options_in_tor.remove('RecommendedPackages') - config_options_in_tor.add('RecommendedPackageVersions') - - config_options_in_tor.add('Recognized') - config_options_in_tor.add('VoteOnHidServDirectoriesV2') - config_options_in_tor.add('HidServDirectoryV2') - # TODO: Looks like options we should remove from tor... # # https://trac.torproject.org/projects/tor/ticket/17665

1 0

[stem/master] Line wrapping by man is platform specific
by atagar＠torproject.org 06 Dec '15

06 Dec '15

commit 9ef7a8844c00d13bd79245bdf09df3a84cfdf855 Author: Damian Johnson <atagar(a)torproject.org> Date: Fri Nov 27 18:33:28 2015 -0800 Line wrapping by man is platform specific Seems where the man command wraps lines varies from platform to platform... ====================================================================== FAIL: test_has_all_categories ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/atagar/Desktop/stem/test/integ/manual.py", line 212, in test_has_all_categories self.assertEqual(8, len(categories['DESCRIPTION'])) # check parsing of multi-line entries AssertionError: 8 != 14 Considering this is on my netbook my guess is that it's related to terminal dimensions. Loosening this assertion (all we really care about is that we're picking up that there's multiple lines. --- test/integ/manual.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/test/integ/manual.py b/test/integ/manual.py index 6a01526..3fbf89a 100644 --- a/test/integ/manual.py +++ b/test/integ/manual.py @@ -209,7 +209,7 @@ class TestManual(unittest.TestCase): self.assertEqual(['tor - The second-generation onion router'], categories['NAME']) self.assertEqual(['tor [OPTION value]...'], categories['SYNOPSIS']) - self.assertEqual(8, len(categories['DESCRIPTION'])) # check parsing of multi-line entries + self.assertTrue(len(categories['DESCRIPTION']) > 5) # check parsing of multi-line entries def test_has_all_summaries(self): """

1 0

[stem/master] Document namedtuples we provide
by atagar＠torproject.org 06 Dec '15

06 Dec '15

commit 281fc886e8976169f2c48d6e364a36fd5e7f9ab7 Author: Damian Johnson <atagar(a)torproject.org> Date: Wed Nov 25 08:56:54 2015 -0800 Document namedtuples we provide Sphinx generates useless docs for namedtuples that simply says 'Alias for field number 0'. Using a tip from the following for geneating useful docs... https://stackoverflow.com/questions/13785150/how-can-i-provide-sphinx-docum… --- stem/control.py | 109 +++++++++++++------------- stem/descriptor/hidden_service_descriptor.py | 32 ++++---- stem/descriptor/networkstatus.py | 28 +++---- stem/manual.py | 17 +++- stem/util/connection.py | 30 ++++--- stem/util/test_tools.py | 18 +++-- 6 files changed, 118 insertions(+), 116 deletions(-) diff --git a/stem/control.py b/stem/control.py index 72f0552..3b1edfa 100644 --- a/stem/control.py +++ b/stem/control.py @@ -368,30 +368,49 @@ SERVER_DESCRIPTORS_UNSUPPORTED = "Tor is currently not configured to retrieve \ server descriptors. As of Tor version 0.2.3.25 it downloads microdescriptors \ instead unless you set 'UseMicrodescriptors 0' in your torrc." -AccountingStats = collections.namedtuple('AccountingStats', [ - 'retrieved', - 'status', - 'interval_end', - 'time_until_reset', - 'read_bytes', - 'read_bytes_left', - 'read_limit', - 'written_bytes', - 'write_bytes_left', - 'write_limit', -]) - -UserTrafficAllowed = collections.namedtuple('UserTrafficAllowed', [ - 'inbound', - 'outbound', -]) - -CreateHiddenServiceOutput = collections.namedtuple('CreateHiddenServiceOutput', [ - 'path', - 'hostname', - 'hostname_for_client', - 'config', -]) + +class AccountingStats(collections.namedtuple('AccountingStats', ['retrieved', 'status', 'interval_end', 'time_until_reset', 'read_bytes', 'read_bytes_left', 'read_limit', 'written_bytes', 'write_bytes_left', 'write_limit'])): + """ + Accounting information, determining the limits where our relay suspends + itself. + + :var float retrieved: unix timestamp for when this was fetched + :var str status: hibernation status of 'awake', 'soft', or 'hard' + :var datetime interval_end: time when our limits reset + :var int time_until_reset: seconds until our limits reset + :var int read_bytes: number of bytes we've read relaying + :var int read_bytes_left: number of bytes we can read until we suspend + :var int read_limit: reading threshold where we suspend + :var int written_bytes: number of bytes we've written relaying + :var int write_bytes_left: number of bytes we can write until we suspend + :var int write_limit: writing threshold where we suspend + """ + + +class UserTrafficAllowed(collections.namedtuple('UserTrafficAllowed', ['inbound', 'outbound'])): + """ + Indicates if we're likely to be servicing direct user traffic or not. + + :var bool inbound: if **True** we're likely providing guard or bridge connnections + :var bool outbound: if **True** we're likely providng exit connections + """ + + +class CreateHiddenServiceOutput(collections.namedtuple('CreateHiddenServiceOutput', ['path', 'hostname', 'hostname_for_client', 'config'])): + """ + Attributes of a hidden service we've created. + + Both the **hostnames** and **hostname_for_client** attributes can only be + provided if we're able to read the hidden service directory. If the method + was called with **client_names** then we may provide the + **hostname_for_client**, and otherwise can provide the **hostnames**. + + :var str path: hidden service directory + :var str hostname: content of the hostname file if available + :var dict hostname_for_client:mapping of client names to their onion address + if available + :var dict config: tor's new hidden service configuration + """ def with_default(yields = False): @@ -1321,25 +1340,13 @@ class Controller(BaseController): get_accounting_stats(default = UNDEFINED) Provides stats related to our relaying limitations if AccountingMax was set - in our torrc. This provides a **namedtuple** with the following - attributes... - - * retrieved (float) - unix timestamp for when this was fetched - * status (str) - hibernation status of 'awake', 'soft', or 'hard' - * interval_end (datetime) - * time_until_reset (int) - seconds until our limits reset - * read_bytes (int) - * read_bytes_left (int) - * read_limit (int) - * written_bytes (int) - * write_bytes_left (int) - * write_limit (int) + in our torrc. .. versionadded:: 1.3.0 :param object default: response if the query fails - :returns: **namedtuple** with our accounting stats + :returns: :class:`~stem.control.AccountingStats` with our accounting stats :raises: :class:`stem.ControllerError` if unable to determine the listeners and no default was provided @@ -1521,8 +1528,9 @@ class Controller(BaseController): .. versionadded:: 1.5.0 - :returns: **namedtuple** with an **inbound** and **outbound** boolean - attribute to indicate if we're likely to have user traffic there + :returns: :class:`~stem.cotroller.UserTrafficAllowed` with **inbound** and + **outbound** boolean attributes to indicate if we're likely servicing + direct user traffic """ inbound_allowed, outbound_allowed = False, False @@ -2439,26 +2447,14 @@ class Controller(BaseController): def create_hidden_service(self, path, port, target_address = None, target_port = None, auth_type = None, client_names = None): """ Create a new hidden service. If the directory is already present, a - new port is added. This provides a **namedtuple** of the following... - - * path (str) - hidden service directory - - * hostname (str) - Content of the hostname file, if no **client_names** - are provided this is the onion address of the service. This is only - retrieved if we can read the hidden service directory. - - * hostname_for_client (dict) - mapping of client names to their onion - address, this is only set if the **client_names** was provided and we - can read the hidden service directory - - * config (dict) - tor's new hidden service configuration + new port is added. Our *.onion address is fetched by reading the hidden service directory. However, this directory is only readable by the tor user, so if unavailable the **hostname** will be **None**. - **As of Tor 0.2.7.1 there's two ways for creating hidden services. This is - no longer the recommended method.** Rather, try using + **As of Tor 0.2.7.1 there's two ways for creating hidden services, and this + method is no longer recommended.** Rather, try using :func:`~stem.control.Controller.create_ephemeral_hidden_service` instead. .. versionadded:: 1.3.0 @@ -2474,7 +2470,8 @@ class Controller(BaseController): :param str auth_type: authentication type: basic, stealth or None to disable auth :param list client_names: client names (1-16 characters "A-Za-z0-9+-_") - :returns: **CreateHiddenServiceOutput** if we create or update a hidden service, **None** otherwise + :returns: :class:`~stem.cotroller.CreateHiddenServiceOutput` if we create + or update a hidden service, **None** otherwise :raises: :class:`stem.ControllerError` if the call fails """ diff --git a/stem/descriptor/hidden_service_descriptor.py b/stem/descriptor/hidden_service_descriptor.py index 935b432..1d9ecae 100644 --- a/stem/descriptor/hidden_service_descriptor.py +++ b/stem/descriptor/hidden_service_descriptor.py @@ -80,7 +80,17 @@ SINGLE_INTRODUCTION_POINT_FIELDS = [ BASIC_AUTH = 1 STEALTH_AUTH = 2 -IntroductionPoint = collections.namedtuple('IntroductionPoints', INTRODUCTION_POINTS_ATTR.keys()) + +class IntroductionPoints(collections.namedtuple('IntroductionPoints', INTRODUCTION_POINTS_ATTR.keys())): + """ + :var str identifier: hash of this introduction point's identity key + :var str address: address of this introduction point + :var int port: port where this introduction point is listening + :var str onion_key: public key for communicating with this introduction point + :var str service_key: public key for communicating with this hidden service + :var list intro_authentication: tuples of the form (auth_type, auth_data) for + establishing a connection + """ class DecryptionFailure(Exception): @@ -245,21 +255,9 @@ class HiddenServiceDescriptor(Descriptor): @lru_cache() def introduction_points(self, authentication_cookie = None): """ - Provided this service's introduction points. This provides a list of - IntroductionPoint instances, which have the following attributes... - - * **identifier** (str): hash of this introduction point's identity key - * **address** (str): address of this introduction point - * **port** (int): port where this introduction point is listening - * **onion_key** (str): public key for communicating with this introduction point - * **service_key** (str): public key for communicating with this hidden service - * **intro_authentication** (list): tuples of the form (auth_type, auth_data) - for establishing a connection - - :param str authentication_cookie: cookie to decrypt the introduction-points - if it's encrypted + Provided this service's introduction points. - :returns: **list** of IntroductionPoints instances + :returns: **list** of :class:`~stem.descriptor.hidden_service_descriptor.IntroductionPoints` :raises: * **ValueError** if the our introduction-points is malformed @@ -358,7 +356,7 @@ class HiddenServiceDescriptor(Descriptor): @staticmethod def _parse_introduction_points(content): """ - Provides the parsed list of IntroductionPoint for the unencrypted content. + Provides the parsed list of IntroductionPoints for the unencrypted content. """ introduction_points = [] @@ -405,6 +403,6 @@ class HiddenServiceDescriptor(Descriptor): auth_type, auth_data = auth_value.split(' ')[:2] auth_entries.append((auth_type, auth_data)) - introduction_points.append(IntroductionPoint(**attr)) + introduction_points.append(IntroductionPoints(**attr)) return introduction_points diff --git a/stem/descriptor/networkstatus.py b/stem/descriptor/networkstatus.py index 2db94a6..ce6dd73 100644 --- a/stem/descriptor/networkstatus.py +++ b/stem/descriptor/networkstatus.py @@ -52,16 +52,6 @@ For more information see :func:`~stem.descriptor.__init__.DocumentHandler`... KeyCertificate - Certificate used to authenticate an authority DocumentSignature - Signature of a document by a directory authority DirectoryAuthority - Directory authority as defined in a v3 network status document - - -.. data:: PackageVersion - - Latest recommended version of a package that's available. - - :var str name: name of the package - :var str version: latest recommended version - :var str url: package's url - :var dict digests: mapping of digest types to their value """ import collections @@ -91,13 +81,6 @@ from stem.descriptor.router_status_entry import ( RouterStatusEntryMicroV3, ) -PackageVersion = collections.namedtuple('PackageVersion', [ - 'name', - 'version', - 'url', - 'digests', -]) - # Version 2 network status document fields, tuples of the form... # (keyword, is_mandatory) @@ -215,6 +198,17 @@ PARAM_RANGE = { } +class PackageVersion(collections.namedtuple('PackageVersion', ['name', 'version', 'url', 'digests'])): + """ + Latest recommended version of a package that's available. + + :var str name: name of the package + :var str version: latest recommended version + :var str url: package's url + :var dict digests: mapping of digest types to their value + """ + + def _parse_file(document_file, document_type = None, validate = False, is_microdescriptor = False, document_handler = DocumentHandler.ENTRIES, **kwargs): """ Parses a network status and iterates over the RouterStatusEntry in it. The diff --git a/stem/manual.py b/stem/manual.py index 1e2a8f1..be25353 100644 --- a/stem/manual.py +++ b/stem/manual.py @@ -75,8 +75,6 @@ except ImportError: import urllib2 as urllib Category = stem.util.enum.Enum('GENERAL', 'CLIENT', 'RELAY', 'DIRECTORY', 'AUTHORITY', 'HIDDEN_SERVICE', 'TESTING', 'UNKNOWN') -ConfigOption = collections.namedtuple('ConfigOption', ['category', 'name', 'usage', 'summary', 'description']) - GITWEB_MANUAL_URL = 'https://gitweb.torproject.org/tor.git/plain/doc/tor.1.txt' CATEGORY_SECTIONS = { @@ -90,6 +88,19 @@ CATEGORY_SECTIONS = { } +class ConfigOption(collections.namedtuple('ConfigOption', ['category', 'name', 'usage', 'summary', 'description'])): + """ + Tor configuration attribute found in its torrc. + + :var stem.manual.Category category: category the config option was listed + under, this is Category.UNKNOWN if we didn't recognize the category + :var str name: name of the configuration option + :var str usage: arguments accepted by the option + :var str summary: brief description of what the option does + :var str description: longer manual description with details + """ + + @lru_cache() def _config(lowercase = True): """ @@ -207,7 +218,7 @@ class Manual(object): :var dict signals: mapping of signals tor accepts to their description :var dict files: mapping of file paths to their description - :var dict config_option: **ConfigOption** tuples for tor configuration options + :var dict config_option: :class:`~stem.manual.ConfigOption` tuples for tor configuration options """ def __init__(self, name, synopsis, description, commandline_options, signals, files, config_options): diff --git a/stem/util/connection.py b/stem/util/connection.py index 88d70d5..951b020 100644 --- a/stem/util/connection.py +++ b/stem/util/connection.py @@ -74,14 +74,6 @@ Resolver = enum.Enum( ('BSD_PROCSTAT', 'procstat (bsd)') ) -Connection = collections.namedtuple('Connection', [ - 'local_address', - 'local_port', - 'remote_address', - 'remote_port', - 'protocol', -]) - FULL_IPv4_MASK = '255.255.255.255' FULL_IPv6_MASK = 'FFFF:FFFF:FFFF:FFFF:FFFF:FFFF:FFFF:FFFF' @@ -140,16 +132,22 @@ RESOLVER_FILTER = { } +class Connection(collections.namedtuple('Connection', ['local_address', 'local_port', 'remote_address', 'remote_port', 'protocol'])): + """ + Network connection information. + + :var str local_address: ip address the connection originates from + :var int local_port: port the connection originates from + :var str remote_address: destionation ip address + :var int remote_port: destination port + :var str protocol: protocol of the connection ('tcp', 'udp', etc) + """ + + def get_connections(resolver, process_pid = None, process_name = None): """ Retrieves a list of the current connections for a given process. This - provides a list of Connection instances, which have five attributes... - - * **local_address** (str) - * **local_port** (int) - * **remote_address** (str) - * **remote_port** (int) - * **protocol** (str, generally either 'tcp' or 'udp') + provides a list of :class:`~stem.util.connection.Connection`. .. versionadded:: 1.1.0 @@ -157,7 +155,7 @@ def get_connections(resolver, process_pid = None, process_name = None): :param int process_pid: pid of the process to retrieve :param str process_name: name of the process to retrieve - :returns: **list** of Connection instances + :returns: **list** of :class:`~stem.util.connection.Connection` instances :raises: * **ValueError** if using **Resolver.PROC** or **Resolver.BSD_PROCSTAT** diff --git a/stem/util/test_tools.py b/stem/util/test_tools.py index dac9fc7..7c97444 100644 --- a/stem/util/test_tools.py +++ b/stem/util/test_tools.py @@ -31,11 +31,15 @@ CONFIG = stem.util.conf.config_dict('test', { 'exclude_paths': [], }) -Issue = collections.namedtuple('Issue', [ - 'line_number', - 'message', - 'line', -]) + +class Issue(collections.namedtuple('Issue', ['line_number', 'message', 'line'])): + """ + Issue encountered by pyflakes or pep8. + + :var int line_number: line number the issue occured on + :var str message: description of the issue + :var str line: content of the line the issue is about + """ def clean_orphaned_pyc(paths): @@ -168,7 +172,7 @@ def stylistic_issues(paths, check_newlines = False, check_exception_keyword = Fa :param bool prefer_single_quotes: standardize on using single rather than double quotes for strings, when reasonable - :returns: **dict** of the form ``path => [(line_number, message)...]`` + :returns: dict of paths list of :class:`stem.util.test_tools.Issue` instances """ issues = {} @@ -256,7 +260,7 @@ def pyflakes_issues(paths): :param list paths: paths to search for problems - :returns: dict of the form ``path => [(line_number, message)...]`` + :returns: dict of paths list of :class:`stem.util.test_tools.Issue` instances """ issues = {}

1 0

[stem/master] Use man arguments to help with parsing
by atagar＠torproject.org 06 Dec '15

06 Dec '15

commit 72786c6edaeb4a6de955bf7c0a459746181222a1 Author: Damian Johnson <atagar(a)torproject.org> Date: Fri Nov 27 20:55:39 2015 -0800 Use man arguments to help with parsing Using a couple arguments to simplify parsing... * MANWIDTH so we don't need to contend with line wrapping. * --encoding=ascii to normalize our output. Seems on my netbook this is the default, causing some long dashes to be replaced with '--'. --- stem/manual.py | 21 ++++----------------- stem/util/system.py | 8 ++++++-- test/integ/manual.py | 12 ++++++------ test/unit/manual.py | 4 ++-- 4 files changed, 18 insertions(+), 27 deletions(-) diff --git a/stem/manual.py b/stem/manual.py index be25353..138380e 100644 --- a/stem/manual.py +++ b/stem/manual.py @@ -244,9 +244,9 @@ class Manual(object): """ try: - man_output = stem.util.system.call('man -P cat %s' % man_path) + man_output = stem.util.system.call('man --encoding=ascii -P cat %s' % man_path, env = {'MANWIDTH': '10000000'}) except OSError as exc: - raise IOError("Unable to run 'man -P cat %s': %s" % (man_path, exc)) + raise IOError("Unable to run 'man --encoding=ascii -P cat %s': %s" % (man_path, exc)) categories, config_options = _get_categories(man_output), OrderedDict() @@ -425,20 +425,7 @@ def _add_config_options(config_options, category, lines): def _join_lines(lines): """ - The man page provides line-wrapped content. Attempting to undo that. This is - close to a simple join, but we still want empty lines to provide newlines. + Simple join, except we want empty lines to still provide a newline. """ - content = [] - - for line in lines: - if line: - if content and content[-1][-1] != '\n': - line = ' ' + line - - content.append(line) - else: - if content and content[-1][-1] != '\n': - content.append('\n\n') - - return ''.join(content) + return ''.join([line if line else '\n\n' for line in lines]) diff --git a/stem/util/system.py b/stem/util/system.py index 4f74f81..7976e48 100644 --- a/stem/util/system.py +++ b/stem/util/system.py @@ -951,7 +951,7 @@ def files_with_suffix(base_path, suffix): yield os.path.join(root, filename) -def call(command, default = UNDEFINED, ignore_exit_status = False): +def call(command, default = UNDEFINED, ignore_exit_status = False, env = None): """ call(command, default = UNDEFINED, ignore_exit_status = False) @@ -959,10 +959,14 @@ def call(command, default = UNDEFINED, ignore_exit_status = False): results. This is not actually ran in a shell so pipes and other shell syntax are not permitted. + .. versionchanged:: 1.5.0 + Added env argument. + :param str,list command: command to be issued :param object default: response if the query fails :param bool ignore_exit_status: reports failure if our command's exit status was non-zero + :param dict env: environment variables :returns: **list** with the lines of output from the command @@ -978,7 +982,7 @@ def call(command, default = UNDEFINED, ignore_exit_status = False): is_shell_command = command_list[0] in SHELL_COMMANDS start_time = time.time() - process = subprocess.Popen(command_list, stdout = subprocess.PIPE, stderr = subprocess.PIPE, shell = is_shell_command) + process = subprocess.Popen(command_list, stdout = subprocess.PIPE, stderr = subprocess.PIPE, shell = is_shell_command, env = env) stdout, stderr = process.communicate() stdout, stderr = stdout.strip(), stderr.strip() diff --git a/test/integ/manual.py b/test/integ/manual.py index 3fbf89a..d9cf171 100644 --- a/test/integ/manual.py +++ b/test/integ/manual.py @@ -40,14 +40,14 @@ EXPECTED_SIGNALS = set(['SIGTERM', 'SIGINT', 'SIGHUP', 'SIGUSR1', 'SIGUSR2', 'SI EXPECTED_DESCRIPTION = """ Tor is a connection-oriented anonymizing communication service. Users choose a source-routed path through a set of nodes, and negotiate a "virtual circuit" through the network, in which each node knows its predecessor and successor, but no others. Traffic flowing down the circuit is unwrapped by a symmetric key at each node, which reveals the downstream node. -Basically, Tor provides a distributed network of servers or relays ("onion routers"). Users bounce their TCP streams - web traffic, ftp, ssh, etc. - around the network, and recipients, observers, and even the relays themselves have difficulty tracking the source of the stream. +Basically, Tor provides a distributed network of servers or relays ("onion routers"). Users bounce their TCP streams -- web traffic, ftp, ssh, etc. -- around the network, and recipients, observers, and even the relays themselves have difficulty tracking the source of the stream. -By default, tor will only act as a client only. To help the network by providing bandwidth as a relay, change the ORPort configuration option - see below. Please also consult the documentation on the Tor Project's website. +By default, tor will only act as a client only. To help the network by providing bandwidth as a relay, change the ORPort configuration option -- see below. Please also consult the documentation on the Tor Project's website. """.strip() EXPECTED_FILE_DESCRIPTION = 'Specify a new configuration file to contain further Tor configuration options OR pass - to make Tor read its configuration from standard input. (Default: @CONFDIR@/torrc, or $HOME/.torrc if that file is not found)' -EXPECTED_BANDWIDTH_RATE_DESCRIPTION = 'A token bucket limits the average incoming bandwidth usage on this node to the specified number of bytes per second, and the average outgoing bandwidth usage to that same value. If you want to run a relay in the public network, this needs to be at the very least 75 KBytes for a relay (that is, 600 kbits) or 50 KBytes for a bridge (400 kbits) - but of course, more is better; we recommend at least 250 KBytes (2 mbits) if possible. (Default: 1 GByte)\n\nWith this option, and in other options that take arguments in bytes, KBytes, and so on, other formats are also supported. Notably, "KBytes" can also be written as "kilobytes" or "kb"; "MBytes" can be written as "megabytes" or "MB"; "kbits" can be written as "kilobits"; and so forth. Tor also accepts "byte" and "bit" in the singular. The prefixes "tera" and "T" are also recognized. If no units are given, we default to bytes. To avoid confusion, we recommend writing "bytes" or "bits" explicitly, sinc e it\'s easy to forget that "B" means bytes, not bits.' +EXPECTED_BANDWIDTH_RATE_DESCRIPTION = 'A token bucket limits the average incoming bandwidth usage on this node to the specified number of bytes per second, and the average outgoing bandwidth usage to that same value. If you want to run a relay in the public network, this needs to be at the very least 75 KBytes for a relay (that is, 600 kbits) or 50 KBytes for a bridge (400 kbits) -- but of course, more is better; we recommend at least 250 KBytes (2 mbits) if possible. (Default: 1 GByte)\n\nWith this option, and in other options that take arguments in bytes, KBytes, and so on, other formats are also supported. Notably, "KBytes" can also be written as "kilobytes" or "kb"; "MBytes" can be written as "megabytes" or "MB"; "kbits" can be written as "kilobits"; and so forth. Tor also accepts "byte" and "bit" in the singular. The prefixes "tera" and "T" are also recognized. If no units are given, we default to bytes. To avoid confusion, we recommend writing "bytes" or "bits" explicitly, sin ce it\'s easy to forget that "B" means bytes, not bits.' class TestManual(unittest.TestCase): @@ -259,9 +259,9 @@ class TestManual(unittest.TestCase): # # https://trac.torproject.org/projects/tor/ticket/17665 - config_options_in_tor.remove('SchedulerMaxFlushCells__') - config_options_in_tor.remove('SchedulerLowWaterMark__') - config_options_in_tor.remove('SchedulerHighWaterMark__') + for option in ('SchedulerMaxFlushCells__', 'SchedulerLowWaterMark__', 'SchedulerHighWaterMark__'): + if option in config_options_in_tor: + config_options_in_tor.remove(option) manual = stem.manual.Manual.from_man(self.man_path) config_options_in_manual = set(manual.config_options.keys()) diff --git a/test/unit/manual.py b/test/unit/manual.py index 7ca69da..1f5ea74 100644 --- a/test/unit/manual.py +++ b/test/unit/manual.py @@ -104,13 +104,13 @@ class TestManual(unittest.TestCase): self.assertEqual(b'a2x output', output.getvalue()) call_mock.assert_called_once_with('a2x -f manpage /no/such/path/tor.1.txt') - @patch('stem.util.system.call', Mock(side_effect = OSError('man -P cat tor returned exit status 16'))) + @patch('stem.util.system.call', Mock(side_effect = OSError('man --encoding=ascii -P cat tor returned exit status 16'))) def test_from_man_when_manual_is_unavailable(self): try: stem.manual.Manual.from_man() self.fail("fetching the manual should fail when it's unavailable") except IOError as exc: - self.assertEqual("Unable to run 'man -P cat tor': man -P cat tor returned exit status 16", str(exc)) + self.assertEqual("Unable to run 'man --encoding=ascii -P cat tor': man --encoding=ascii -P cat tor returned exit status 16", str(exc)) @patch('stem.util.system.call', Mock(return_value = [])) def test_when_man_is_empty(self):

1 0

Jump to page: