March 2018 - tor-commits - lists.torproject.org

[webwml/master] remove redundant execdir position
by arma＠torproject.org 29 Mar '18

29 Mar '18

commit b0fa2fa81f99f90d500e21ed593320bb3d416095 Author: Roger Dingledine <arma(a)torproject.org> Date: Thu Mar 29 16:04:42 2018 -0400 remove redundant execdir position don't list executive director as a previous job title when we're asking for one right now --- about/en/jobs.wml | 1 - 1 file changed, 1 deletion(-) diff --git a/about/en/jobs.wml b/about/en/jobs.wml index 2bdf7a76..0f96e6fc 100644 --- a/about/en/jobs.wml +++ b/about/en/jobs.wml @@ -34,7 +34,6 @@ <li>Lead Automation Engineer</li> <li>Firefox/Thunderbird Extension Developer</li> <li>Director of Communications</li> - <li>Executive Director</li> <li>Executive Administrator</li> <li>Browser Developers for Mobile (Android)</li> <li>Chief Financial and Grants Officer</li>

1 0

[webwml/master] all roles are supportive, but only some roles are support
by arma＠torproject.org 29 Mar '18

29 Mar '18

commit 7db96cf6ed74310dec2d2ab81193955ed259f9ca Author: Roger Dingledine <arma(a)torproject.org> Date: Thu Mar 29 16:04:17 2018 -0400 all roles are supportive, but only some roles are support --- about/en/jobs-useradvocate.wml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/about/en/jobs-useradvocate.wml b/about/en/jobs-useradvocate.wml index c6b10a0f..c9bfb768 100644 --- a/about/en/jobs-useradvocate.wml +++ b/about/en/jobs-useradvocate.wml @@ -47,7 +47,7 @@ coordinate with our teams to ensure the problems are addressed. The ideal candidate will be someone skilled at working with developers to understand the root causes of user issues while still maintaining a broad, user-centric view of the problems and areas where Tor can -improve. Previous experience working in a supportive role is a huge +improve. Previous experience working in a support role is a huge plus. </p>

1 0

[webwml/master] link it in to the main jobs page
by arma＠torproject.org 29 Mar '18

29 Mar '18

commit cd659f0788b9034fc935a796712727a9a723e60c Author: Roger Dingledine <arma(a)torproject.org> Date: Thu Mar 29 15:57:37 2018 -0400 link it in to the main jobs page and call it out as a mozilla fellowship there, to be very clear --- about/en/jobs.wml | 1 + 1 file changed, 1 insertion(+) diff --git a/about/en/jobs.wml b/about/en/jobs.wml index f27fd37b..2bdf7a76 100644 --- a/about/en/jobs.wml +++ b/about/en/jobs.wml @@ -19,6 +19,7 @@ <p>Specifically, we're looking for the following people currently:</p> <ul> <li><a href="<page about/jobs-execdirector>">Executive Director</a></li> + <li><a href="<page about/jobs-useradvocate>">User Advocate (Mozilla fellowship)</a></li> <li><a href="<page about/jobs-userresearchcoordinator>">User Research Coordinator</a></li> <li><a href="<page about/jobs-communityliaison>">Community Liaison</a></li> </ul>

1 0

[webwml/master] add job description for our new user advocate spot
by arma＠torproject.org 29 Mar '18

29 Mar '18

commit 11f13dd369c7d89796b047afe22d2fc601a935de Author: Roger Dingledine <arma(a)torproject.org> Date: Thu Mar 29 15:19:27 2018 -0400 add job description for our new user advocate spot --- about/en/jobs-useradvocate.wml | 117 +++++++++++++++++++++++++++++++++++++++++ 1 file changed, 117 insertions(+) diff --git a/about/en/jobs-useradvocate.wml b/about/en/jobs-useradvocate.wml new file mode 100644 index 00000000..c6b10a0f --- /dev/null +++ b/about/en/jobs-useradvocate.wml @@ -0,0 +1,117 @@ +## translation metadata +# Revision: $Revision$ +# Translation-Priority: 3-low + +#include "head.wmi" TITLE="Tor Project: Jobs (User Advocate)" CHARSET="UTF-8" +<div id="content" class="clearfix"> + <div id="breadcrumbs"> + <a href="<page index>">Home » </a> + <a href="<page about/overview>">About » </a> + <a href="<page about/jobs>">Jobs</a> + </div> + <div id="maincol"> + +<h1>User Advocate (Mozilla Fellowship)</h1> + +<h2>March 29, 2018</h2> + +<p> +As an organization dedicated to safeguarding the +open internet, the Tor Project is participating in the <a +href="https://advocacy.mozilla.org/en-US/open-web-fellows/overview">Mozilla +Open Web fellow</a> program as a host. Through this fellowship, +we're seeking to host a User Advocate to bridge the gap between Tor +users and Tor developers. If you're passionate about internet freedom +and improving software to make it more user-friendly, this might be the +perfect job for you. +</p> + +<p> +Tor is widely considered the strongest internet privacy software, +and we want to make it the most user-friendly. Millions of people use +Tor every day to protect their privacy, safety, and freedom of expression +on the internet. Because Tor's users are distributed across the globe +and have varying levels of technical skill, we are looking for someone +who can gather data from this diverse group of people to help us make +usability improvements. +</p> + +<p> +Your job will be to identify common user problems: where people +get stuck setting up or using Tor, and what we can do to address those +issues. After identifying and documenting the common problems, you will +coordinate with our teams to ensure the problems are addressed. +</p> + +<p> +The ideal candidate will be someone skilled at working with developers +to understand the root causes of user issues while still maintaining +a broad, user-centric view of the problems and areas where Tor can +improve. Previous experience working in a supportive role is a huge +plus. +</p> + +<h2>The ideal candidate will have the following:</h2> + +<ul> +<li>Experience with public interest technology and commitment to +addressing issues like privacy, security, and inclusion online</li> +<li>Familiarity with the ins and outs of Tor Browser, the Tor network, +and the various applications that integrate Tor's privacy and security +protections</li> +<li>Exceptional patience and willingness to talk through issues and +problems with people across many cultures - the more detail you can get +from affected users, the better!</li> +<li>Be detail-oriented in filing comprehensive bug reports (Trac is +the primary tool Tor developers use to discuss and deploy new features +and bug-fixes)</li> +<li>Capability of keeping the big picture in mind without becoming overly +distracted by any single area</li> +<li>Excitement for being a user advocate and for getting common user +issues prioritized in the development roadmaps for each team</li> +<li>Willingness and ability to travel internationally during the term +of this fellowship</li> +<li>Fluency in languages other than English is also a major plus.</li> +</ul> + +<p> +The fellow will receive a $60,000 stipend, paid in monthly installments, +as well as supplements for childcare and health insurance. This is a +full-time fellowship, either working from our Seattle office or remotely. +</p> + +<h2>How to apply:</h2> + +<p> +To apply, visit <a +href="https://foundation.mozilla.org/fellowships/apply/">Mozilla's +application page</a>, create an account, and select “Open Web.” The +deadline for applications is April 20, 2018 at 5pm EDT. +</p> + +<h2>About Tor:</h2> + +<p> +The Tor Project, Inc., is a 501(c)(3) organization headquartered +in Seattle that provides the technical infrastructure for privacy +protection over the Internet, helping millions of activists, journalists +and others around the world communicate securely. With paid staff and +contractors of around 35 engineers and operational support people, +plus many volunteers all over the world who contribute to our work. Our +mission is to advance human rights and freedoms by creating and deploying +free and open anonymity and privacy technologies, supporting their +unrestricted availability and use, and furthering their scientific and +popular understanding. +</p> + + </div> +  + <div id = "sidecol"> +#include "side.wmi" +#include "info.wmi" + </div> +  +</div> + +#include <foot.wmi> +

1 0

[webwml/staging] add job description for our new user advocate spot
by arma＠torproject.org 29 Mar '18

29 Mar '18

commit 11f13dd369c7d89796b047afe22d2fc601a935de Author: Roger Dingledine <arma(a)torproject.org> Date: Thu Mar 29 15:19:27 2018 -0400 add job description for our new user advocate spot --- about/en/jobs-useradvocate.wml | 117 +++++++++++++++++++++++++++++++++++++++++ 1 file changed, 117 insertions(+) diff --git a/about/en/jobs-useradvocate.wml b/about/en/jobs-useradvocate.wml new file mode 100644 index 00000000..c6b10a0f --- /dev/null +++ b/about/en/jobs-useradvocate.wml @@ -0,0 +1,117 @@ +## translation metadata +# Revision: $Revision$ +# Translation-Priority: 3-low + +#include "head.wmi" TITLE="Tor Project: Jobs (User Advocate)" CHARSET="UTF-8" +<div id="content" class="clearfix"> + <div id="breadcrumbs"> + <a href="<page index>">Home » </a> + <a href="<page about/overview>">About » </a> + <a href="<page about/jobs>">Jobs</a> + </div> + <div id="maincol"> + +<h1>User Advocate (Mozilla Fellowship)</h1> + +<h2>March 29, 2018</h2> + +<p> +As an organization dedicated to safeguarding the +open internet, the Tor Project is participating in the <a +href="https://advocacy.mozilla.org/en-US/open-web-fellows/overview">Mozilla +Open Web fellow</a> program as a host. Through this fellowship, +we're seeking to host a User Advocate to bridge the gap between Tor +users and Tor developers. If you're passionate about internet freedom +and improving software to make it more user-friendly, this might be the +perfect job for you. +</p> + +<p> +Tor is widely considered the strongest internet privacy software, +and we want to make it the most user-friendly. Millions of people use +Tor every day to protect their privacy, safety, and freedom of expression +on the internet. Because Tor's users are distributed across the globe +and have varying levels of technical skill, we are looking for someone +who can gather data from this diverse group of people to help us make +usability improvements. +</p> + +<p> +Your job will be to identify common user problems: where people +get stuck setting up or using Tor, and what we can do to address those +issues. After identifying and documenting the common problems, you will +coordinate with our teams to ensure the problems are addressed. +</p> + +<p> +The ideal candidate will be someone skilled at working with developers +to understand the root causes of user issues while still maintaining +a broad, user-centric view of the problems and areas where Tor can +improve. Previous experience working in a supportive role is a huge +plus. +</p> + +<h2>The ideal candidate will have the following:</h2> + +<ul> +<li>Experience with public interest technology and commitment to +addressing issues like privacy, security, and inclusion online</li> +<li>Familiarity with the ins and outs of Tor Browser, the Tor network, +and the various applications that integrate Tor's privacy and security +protections</li> +<li>Exceptional patience and willingness to talk through issues and +problems with people across many cultures - the more detail you can get +from affected users, the better!</li> +<li>Be detail-oriented in filing comprehensive bug reports (Trac is +the primary tool Tor developers use to discuss and deploy new features +and bug-fixes)</li> +<li>Capability of keeping the big picture in mind without becoming overly +distracted by any single area</li> +<li>Excitement for being a user advocate and for getting common user +issues prioritized in the development roadmaps for each team</li> +<li>Willingness and ability to travel internationally during the term +of this fellowship</li> +<li>Fluency in languages other than English is also a major plus.</li> +</ul> + +<p> +The fellow will receive a $60,000 stipend, paid in monthly installments, +as well as supplements for childcare and health insurance. This is a +full-time fellowship, either working from our Seattle office or remotely. +</p> + +<h2>How to apply:</h2> + +<p> +To apply, visit <a +href="https://foundation.mozilla.org/fellowships/apply/">Mozilla's +application page</a>, create an account, and select “Open Web.” The +deadline for applications is April 20, 2018 at 5pm EDT. +</p> + +<h2>About Tor:</h2> + +<p> +The Tor Project, Inc., is a 501(c)(3) organization headquartered +in Seattle that provides the technical infrastructure for privacy +protection over the Internet, helping millions of activists, journalists +and others around the world communicate securely. With paid staff and +contractors of around 35 engineers and operational support people, +plus many volunteers all over the world who contribute to our work. Our +mission is to advance human rights and freedoms by creating and deploying +free and open anonymity and privacy technologies, supporting their +unrestricted availability and use, and furthering their scientific and +popular understanding. +</p> + + </div> +  + <div id = "sidecol"> +#include "side.wmi" +#include "info.wmi" + </div> +  +</div> + +#include <foot.wmi> +

1 0

[stem/master] Expanded descriptor compression support
by atagar＠torproject.org 29 Mar '18

29 Mar '18

commit 72700087b94f2889b5b364738a1178c324862ba5 Author: Damian Johnson <atagar(a)torproject.org> Date: Wed Mar 28 11:57:10 2018 -0700 Expanded descriptor compression support We supported plaintext and gzip when downloading descriptors, but recently tor added lzma and zstd support as well... https://gitweb.torproject.org/torspec.git/commit/?id=1cb56af In practice I'm having difficulty finding an example of this working in the live tor network... https://trac.torproject.org/projects/tor/ticket/25667 That said, gonna go ahead and push this since even setting lzma/zstd aside it's a nice step forward since it allows callers to specify compression headers. --- docs/change_log.rst | 1 + stem/descriptor/remote.py | 159 ++++++++++++++++++++++++++++++---------- stem/version.py | 2 + test/integ/descriptor/remote.py | 20 +++++ test/unit/descriptor/remote.py | 50 ++++++++++--- 5 files changed, 182 insertions(+), 50 deletions(-) diff --git a/docs/change_log.rst b/docs/change_log.rst index bb42c982..a67aae2d 100644 --- a/docs/change_log.rst +++ b/docs/change_log.rst @@ -55,6 +55,7 @@ The following are only available within Stem's `git repository * **Descriptors** * `Fallback directory v2 support <https://lists.torproject.org/pipermail/tor-dev/2017-December/012721.html>`_, which adds *nickname* and *extrainfo* + * Added zstd and lzma compression support (:spec:`1cb56af`) * Reduced maximum descriptors fetched by the remote module to match tor's new limit (:trac:`24743`) * Consensus **shared_randomness_*_reveal_count** attributes undocumented, and unavailable if retrieved before their corresponding shared_randomness_*_value attribute (:trac:`25046`) * Allow 'proto' line to have blank values (:spec:`a8455f4`) diff --git a/stem/descriptor/remote.py b/stem/descriptor/remote.py index 9181dbcf..ab61d3a7 100644 --- a/stem/descriptor/remote.py +++ b/stem/descriptor/remote.py @@ -80,6 +80,21 @@ content. For example... Maximum number of microdescriptors that can requested at a time by their hashes. + +.. data:: Compression (enum) + + Compression when downloading descriptors. + + .. versionadded:: 1.7.0 + + =============== =========== + Compression Description + =============== =========== + **PLAINTEXT** Uncompressed data. + **GZIP** `GZip compression <https://www.gnu.org/software/gzip/>`_. + **ZSTD** `Zstandard compression <https://www.zstd.net>`_ + **LZMA** `LZMA compression <https://en.wikipedia.org/wiki/LZMA>`_. + =============== =========== """ import io @@ -91,6 +106,13 @@ import threading import time import zlib +import stem.descriptor +import stem.prereq +import stem.util.enum + +from stem import Flag +from stem.util import _hash_attr, connection, log, str_tools, tor_tools + try: # added in python 2.7 from collections import OrderedDict @@ -103,11 +125,24 @@ try: except ImportError: import urllib2 as urllib -import stem.descriptor -import stem.prereq +try: + # added in python 3.3 + import lzma + LZMA_SUPPORTED = True +except ImportError: + LZMA_SUPPORTED = False -from stem import Flag -from stem.util import _hash_attr, connection, log, str_tools, tor_tools +ZSTD_SUPPORTED = False + +Compression = stem.util.enum.Enum( + ('PLAINTEXT', 'identity'), + ('GZIP', 'gzip'), # can also be 'deflate' + ('ZSTD', 'x-zstd'), + ('LZMA', 'x-tor-lzma'), +) + +ZSTD_UNAVAILABLE_MSG = 'ZSTD is not yet supported' +LZMA_UNAVAILABLE_MSG = 'LZMA compression was requested but requires the lzma module, which was added in python 3.3' # Tor has a limited number of descriptors we can fetch explicitly by their # fingerprint or hashes due to a limit on the url length by squid proxies. @@ -224,7 +259,7 @@ class Query(object): from stem.descriptor.remote import Query query = Query( - '/tor/server/all.z', + '/tor/server/all', block = True, timeout = 30, ) @@ -243,7 +278,7 @@ class Query(object): print('Current relays:') - for desc in Query('/tor/server/all.z', 'server-descriptor 1.0'): + for desc in Query('/tor/server/all', 'server-descriptor 1.0'): print(desc.fingerprint) In either case exceptions are available via our 'error' attribute. @@ -256,28 +291,37 @@ class Query(object): =============================================== =========== Resource Description =============================================== =========== - /tor/server/all.z all present server descriptors - /tor/server/fp/<fp1>+<fp2>+<fp3>.z server descriptors with the given fingerprints - /tor/extra/all.z all present extrainfo descriptors - /tor/extra/fp/<fp1>+<fp2>+<fp3>.z extrainfo descriptors with the given fingerprints - /tor/micro/d/<hash1>-<hash2>.z microdescriptors with the given hashes - /tor/status-vote/current/consensus.z present consensus - /tor/status-vote/current/consensus-microdesc.z present microdescriptor consensus - /tor/keys/all.z key certificates for the authorities - /tor/keys/fp/<v3ident1>+<v3ident2>.z key certificates for specific authorities + /tor/server/all all present server descriptors + /tor/server/fp/<fp1>+<fp2>+<fp3> server descriptors with the given fingerprints + /tor/extra/all all present extrainfo descriptors + /tor/extra/fp/<fp1>+<fp2>+<fp3> extrainfo descriptors with the given fingerprints + /tor/micro/d/<hash1>-<hash2> microdescriptors with the given hashes + /tor/status-vote/current/consensus present consensus + /tor/status-vote/current/consensus-microdesc present microdescriptor consensus + /tor/keys/all key certificates for the authorities + /tor/keys/fp/<v3ident1>+<v3ident2> key certificates for specific authorities =============================================== =========== - The '.z' suffix can be excluded to get a plaintext rather than compressed - response. Compression is handled transparently, so this shouldn't matter to - the caller. + **LZMA** compression requires the `lzma module + <https://docs.python.org/3/library/lzma.html>`_ which was added in Python + 3.3. + + For legacy reasons if our resource has a '.z' suffix then our **compression** + argument is overwritten with Compression.GZIP. + + .. versionchanged:: 1.7.0 + Added the compression argument. - :var str resource: resource being fetched, such as '/tor/server/all.z' + :var str resource: resource being fetched, such as '/tor/server/all' :var str descriptor_type: type of descriptors being fetched (for options see :func:`~stem.descriptor.__init__.parse_file`), this is guessed from the resource if **None** :var list endpoints: (address, dirport) tuples of the authority or mirror we're querying, this uses authorities if undefined + :var list compression: list of :data:`stem.descriptor.remote.Compression` + we're willing to accept, when none are mutually supported downloads fall + back to Compression.PLAINTEXT :var int retries: number of times to attempt the request if downloading it fails :var bool fall_back_to_authority: when retrying request issues the last @@ -305,17 +349,37 @@ class Query(object): the same as running **query.run(True)** (default is **False**) """ - def __init__(self, resource, descriptor_type = None, endpoints = None, retries = 2, fall_back_to_authority = False, timeout = None, start = True, block = False, validate = False, document_handler = stem.descriptor.DocumentHandler.ENTRIES, **kwargs): + def __init__(self, resource, descriptor_type = None, endpoints = None, compression = None, retries = 2, fall_back_to_authority = False, timeout = None, start = True, block = False, validate = False, document_handler = stem.descriptor.DocumentHandler.ENTRIES, **kwargs): if not resource.startswith('/'): raise ValueError("Resources should start with a '/': %s" % resource) - self.resource = resource + if resource.endswith('.z'): + compression = [Compression.GZIP] + resource = resource[:-2] + elif compression is None: + compression = [Compression.PLAINTEXT] + else: + if isinstance(compression, str): + compression = [compression] # caller provided only a single option + + if Compression.LZMA in compression and not LZMA_SUPPORTED: + log.log_once('stem.descriptor.remote.lzma_unavailable', log.INFO, LZMA_UNAVAILABLE_MSG) + compression.remove(Compression.LZMA) + + if Compression.ZSTD in compression and not ZSTD_SUPPORTED: + log.log_once('stem.descriptor.remote.zstd_unavailable', log.INFO, ZSTD_UNAVAILABLE_MSG) + compression.remove(Compression.ZSTD) + + if not compression: + compression = [Compression.PLAINTEXT] if descriptor_type: self.descriptor_type = descriptor_type else: self.descriptor_type = _guess_descriptor_type(resource) + self.resource = resource + self.compression = compression self.endpoints = endpoints if endpoints else [] self.retries = retries self.fall_back_to_authority = fall_back_to_authority @@ -352,7 +416,7 @@ class Query(object): self._downloader_thread = threading.Thread( name = 'Descriptor Query', target = self._download_descriptors, - args = (self.retries,) + args = (self.compression, self.retries,) ) self._downloader_thread.setDaemon(True) @@ -435,26 +499,41 @@ class Query(object): if use_authority or not self.endpoints: directories = get_authorities().values() - picked = random.choice(directories) + picked = random.choice(list(directories)) address, dirport = picked.address, picked.dir_port else: address, dirport = random.choice(self.endpoints) return 'http://%s:%i/%s' % (address, dirport, self.resource.lstrip('/')) - def _download_descriptors(self, retries): + def _download_descriptors(self, compression, retries): try: use_authority = retries == 0 and self.fall_back_to_authority self.download_url = self._pick_url(use_authority) - self.start_time = time.time() - response = urllib.urlopen(self.download_url, timeout = self.timeout).read() - if self.download_url.endswith('.z'): - response = zlib.decompress(response) + response = urllib.urlopen( + urllib.Request( + self.download_url, + headers = {'Accept-Encoding': ', '.join(compression)}, + ), + timeout = self.timeout, + ) + + data = response.read() + encoding = response.info().getheader('Content-Encoding') - self.content = response.strip() + if encoding in (Compression.GZIP, 'deflate'): + # The '32' is for automatic header detection... + # https://stackoverflow.com/questions/3122145/zlib-error-error-3-while-decomp… + data = zlib.decompress(data, zlib.MAX_WBITS | 32) + elif encoding == Compression.ZSTD: + pass # TODO: implement + elif encoding == Compression.LZMA and LZMA_SUPPORTED: + data = lzma.decompress(data) + + self.content = data.strip() self.runtime = time.time() - self.start_time log.trace("Descriptors retrieved from '%s' in %0.2fs" % (self.download_url, self.runtime)) except: @@ -462,7 +541,7 @@ class Query(object): if retries > 0: log.debug("Unable to download descriptors from '%s' (%i retries remaining): %s" % (self.download_url, retries, exc)) - return self._download_descriptors(retries - 1) + return self._download_descriptors(compression, retries - 1) else: log.debug("Unable to download descriptors from '%s': %s" % (self.download_url, exc)) self.error = exc @@ -539,7 +618,7 @@ class DescriptorDownloader(object): fingerprints (this is due to a limit on the url length by squid proxies). """ - resource = '/tor/server/all.z' + resource = '/tor/server/all' if isinstance(fingerprints, str): fingerprints = [fingerprints] @@ -548,7 +627,7 @@ class DescriptorDownloader(object): if len(fingerprints) > MAX_FINGERPRINTS: raise ValueError('Unable to request more than %i descriptors at a time by their fingerprints' % MAX_FINGERPRINTS) - resource = '/tor/server/fp/%s.z' % '+'.join(fingerprints) + resource = '/tor/server/fp/%s' % '+'.join(fingerprints) return self.query(resource, **query_args) @@ -569,7 +648,7 @@ class DescriptorDownloader(object): fingerprints (this is due to a limit on the url length by squid proxies). """ - resource = '/tor/extra/all.z' + resource = '/tor/extra/all' if isinstance(fingerprints, str): fingerprints = [fingerprints] @@ -578,7 +657,7 @@ class DescriptorDownloader(object): if len(fingerprints) > MAX_FINGERPRINTS: raise ValueError('Unable to request more than %i descriptors at a time by their fingerprints' % MAX_FINGERPRINTS) - resource = '/tor/extra/fp/%s.z' % '+'.join(fingerprints) + resource = '/tor/extra/fp/%s' % '+'.join(fingerprints) return self.query(resource, **query_args) @@ -613,7 +692,7 @@ class DescriptorDownloader(object): if len(hashes) > MAX_MICRODESCRIPTOR_HASHES: raise ValueError('Unable to request more than %i microdescriptors at a time by their hashes' % MAX_MICRODESCRIPTOR_HASHES) - return self.query('/tor/micro/d/%s.z' % '-'.join(hashes), **query_args) + return self.query('/tor/micro/d/%s' % '-'.join(hashes), **query_args) def get_consensus(self, authority_v3ident = None, microdescriptor = False, **query_args): """ @@ -643,7 +722,7 @@ class DescriptorDownloader(object): if authority_v3ident: resource += '/%s' % authority_v3ident - consensus_query = self.query(resource + '.z', **query_args) + consensus_query = self.query(resource, **query_args) # if we're performing validation then check that it's signed by the # authority key certificates @@ -672,7 +751,7 @@ class DescriptorDownloader(object): if 'endpoint' not in query_args: query_args['endpoints'] = [(authority.address, authority.dir_port)] - return self.query(resource + '.z', **query_args) + return self.query(resource, **query_args) def get_key_certificates(self, authority_v3idents = None, **query_args): """ @@ -694,7 +773,7 @@ class DescriptorDownloader(object): squid proxies). """ - resource = '/tor/keys/all.z' + resource = '/tor/keys/all' if isinstance(authority_v3idents, str): authority_v3idents = [authority_v3idents] @@ -703,7 +782,7 @@ class DescriptorDownloader(object): if len(authority_v3idents) > MAX_FINGERPRINTS: raise ValueError('Unable to request more than %i key certificates at a time by their identity fingerprints' % MAX_FINGERPRINTS) - resource = '/tor/keys/fp/%s.z' % '+'.join(authority_v3idents) + resource = '/tor/keys/fp/%s' % '+'.join(authority_v3idents) return self.query(resource, **query_args) @@ -711,7 +790,7 @@ class DescriptorDownloader(object): """ Issues a request for the given resource. - :param str resource: resource being fetched, such as '/tor/server/all.z' + :param str resource: resource being fetched, such as '/tor/server/all' :param query_args: additional arguments for the :class:`~stem.descriptor.remote.Query` constructor diff --git a/stem/version.py b/stem/version.py index 9de2f1a5..9036effb 100644 --- a/stem/version.py +++ b/stem/version.py @@ -35,6 +35,7 @@ easily parsed and compared, for instance... Requirement Description ===================================== =========== **AUTH_SAFECOOKIE** SAFECOOKIE authentication method + **DESCRIPTOR_COMPRESSION** `Expanded compression support for ZSTD and LZMA <https://gitweb.torproject.org/torspec.git/commit/?id=1cb56afdc1e55e303e3e6b…>`_ **DROPGUARDS** DROPGUARDS requests **EVENT_AUTHDIR_NEWDESCS** AUTHDIR_NEWDESC events **EVENT_BUILDTIMEOUT_SET** BUILDTIMEOUT_SET events @@ -353,6 +354,7 @@ safecookie_req.greater_than(Version('0.2.3.13')) Requirement = stem.util.enum.Enum( ('AUTH_SAFECOOKIE', safecookie_req), + ('DESCRIPTOR_COMPRESSION', Version('0.3.1.1-alpha')), ('DROPGUARDS', Version('0.2.5.1-alpha')), ('EVENT_AUTHDIR_NEWDESCS', Version('0.1.1.10-alpha')), ('EVENT_BUILDTIMEOUT_SET', Version('0.2.2.7-alpha')), diff --git a/test/integ/descriptor/remote.py b/test/integ/descriptor/remote.py index 3123ef06..4e10eac2 100644 --- a/test/integ/descriptor/remote.py +++ b/test/integ/descriptor/remote.py @@ -16,6 +16,26 @@ import test.require class TestDescriptorDownloader(unittest.TestCase): @test.require.only_run_once @test.require.online + def test_compression(self): + """ + Issue a request for a plaintext descriptor. + """ + + moria1 = stem.descriptor.remote.get_authorities()['moria1'] + + descriptors = list(stem.descriptor.remote.Query( + '/tor/server/fp/%s' % moria1.fingerprint, + 'server-descriptor 1.0', + endpoints = [(moria1.address, moria1.dir_port)], + timeout = 30, + validate = True, + ).run()) + + self.assertEqual(1, len(descriptors)) + self.assertEqual('moria1', descriptors[0].nickname) + + @test.require.only_run_once + @test.require.online def test_shorthand_aliases(self): """ Quick sanity test that we can call our shorthand aliases for getting diff --git a/test/unit/descriptor/remote.py b/test/unit/descriptor/remote.py index ac150d5c..3661a54b 100644 --- a/test/unit/descriptor/remote.py +++ b/test/unit/descriptor/remote.py @@ -11,6 +11,8 @@ import stem.descriptor.remote import stem.prereq import stem.util.conf +from stem.descriptor.remote import Compression + try: # added in python 2.7 from collections import OrderedDict @@ -29,6 +31,8 @@ except ImportError: URL_OPEN = 'urllib.request.urlopen' if stem.prereq.is_python_3() else 'urllib2.urlopen' +TEST_RESOURCE = '/tor/server/fp/9695DFC35FFEB861329B9F1AB04C46397020CE31' + # Output from requesting moria1's descriptor from itself... # % curl http://128.31.0.39:9131/tor/server/fp/9695DFC35FFEB861329B9F1AB04C46397020C… @@ -108,6 +112,33 @@ FALLBACK_ENTRY = b"""\ class TestDescriptorDownloader(unittest.TestCase): + def test_gzip_url_override(self): + query = stem.descriptor.remote.Query(TEST_RESOURCE, start = False) + self.assertEqual([Compression.PLAINTEXT], query.compression) + self.assertEqual(TEST_RESOURCE, query.resource) + + query = stem.descriptor.remote.Query(TEST_RESOURCE + '.z', compression = Compression.PLAINTEXT, start = False) + self.assertEqual([Compression.GZIP], query.compression) + self.assertEqual(TEST_RESOURCE, query.resource) + + def test_zstd_support_check(self): + with patch('stem.descriptor.remote.ZSTD_SUPPORTED', True): + query = stem.descriptor.remote.Query(TEST_RESOURCE, compression = Compression.ZSTD, start = False) + self.assertEqual([Compression.ZSTD], query.compression) + + with patch('stem.descriptor.remote.ZSTD_SUPPORTED', False): + query = stem.descriptor.remote.Query(TEST_RESOURCE, compression = Compression.ZSTD, start = False) + self.assertEqual([Compression.PLAINTEXT], query.compression) + + def test_lzma_support_check(self): + with patch('stem.descriptor.remote.LZMA_SUPPORTED', True): + query = stem.descriptor.remote.Query(TEST_RESOURCE, compression = Compression.LZMA, start = False) + self.assertEqual([Compression.LZMA], query.compression) + + with patch('stem.descriptor.remote.LZMA_SUPPORTED', False): + query = stem.descriptor.remote.Query(TEST_RESOURCE, compression = Compression.LZMA, start = False) + self.assertEqual([Compression.PLAINTEXT], query.compression) + @patch(URL_OPEN) def test_query_download(self, urlopen_mock): """ @@ -117,13 +148,14 @@ class TestDescriptorDownloader(unittest.TestCase): urlopen_mock.return_value = io.BytesIO(TEST_DESCRIPTOR) query = stem.descriptor.remote.Query( - '/tor/server/fp/9695DFC35FFEB861329B9F1AB04C46397020CE31', + TEST_RESOURCE, 'server-descriptor 1.0', endpoints = [('128.31.0.39', 9131)], + compression = Compression.PLAINTEXT, validate = True, ) - expeced_url = 'http://128.31.0.39:9131/tor/server/fp/9695DFC35FFEB861329B9F1AB04C46397020C…' + expeced_url = 'http://128.31.0.39:9131' + TEST_RESOURCE self.assertEqual(expeced_url, query._pick_url()) descriptors = list(query) @@ -135,7 +167,7 @@ class TestDescriptorDownloader(unittest.TestCase): self.assertEqual('9695DFC35FFEB861329B9F1AB04C46397020CE31', desc.fingerprint) self.assertEqual(TEST_DESCRIPTOR.strip(), desc.get_bytes()) - urlopen_mock.assert_called_once_with(expeced_url, timeout = None) + self.assertEqual(1, urlopen_mock.call_count) @patch(URL_OPEN) def test_query_with_malformed_content(self, urlopen_mock): @@ -147,9 +179,10 @@ class TestDescriptorDownloader(unittest.TestCase): urlopen_mock.return_value = io.BytesIO(descriptor_content) query = stem.descriptor.remote.Query( - '/tor/server/fp/9695DFC35FFEB861329B9F1AB04C46397020CE31', + TEST_RESOURCE, 'server-descriptor 1.0', endpoints = [('128.31.0.39', 9131)], + compression = Compression.PLAINTEXT, validate = True, ) @@ -171,7 +204,7 @@ class TestDescriptorDownloader(unittest.TestCase): urlopen_mock.side_effect = socket.timeout('connection timed out') query = stem.descriptor.remote.Query( - '/tor/server/fp/9695DFC35FFEB861329B9F1AB04C46397020CE31', + TEST_RESOURCE, 'server-descriptor 1.0', endpoints = [('128.31.0.39', 9131)], fall_back_to_authority = False, @@ -180,10 +213,6 @@ class TestDescriptorDownloader(unittest.TestCase): ) self.assertRaises(socket.timeout, query.run) - urlopen_mock.assert_called_with( - 'http://128.31.0.39:9131/tor/server/fp/9695DFC35FFEB861329B9F1AB04C46397020C…', - timeout = 5, - ) self.assertEqual(3, urlopen_mock.call_count) @patch(URL_OPEN) @@ -191,9 +220,10 @@ class TestDescriptorDownloader(unittest.TestCase): urlopen_mock.return_value = io.BytesIO(TEST_DESCRIPTOR) query = stem.descriptor.remote.Query( - '/tor/server/fp/9695DFC35FFEB861329B9F1AB04C46397020CE31', + TEST_RESOURCE, 'server-descriptor 1.0', endpoints = [('128.31.0.39', 9131)], + compression = Compression.PLAINTEXT, validate = True, )

1 0

[tor/master] Merge branch 'maint-0.3.3'
by nickm＠torproject.org 29 Mar '18

29 Mar '18

commit e8c1d4c8b05247e3e3cfec20ca11bafa5369bf41 Merge: 2c1afc2de 6aaafb067 Author: Nick Mathewson <nickm(a)torproject.org> Date: Thu Mar 29 15:00:46 2018 -0400 Merge branch 'maint-0.3.3' doc/tor.1.txt | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-)

1 0

[tor/release-0.3.3] Document torrc options that only work on v2 onion services
by nickm＠torproject.org 29 Mar '18

29 Mar '18

commit 6aaafb0672ec7d2f454720ea0bed41fb91d9744e Author: teor <teor2345(a)gmail.com> Date: Wed Mar 28 17:10:31 2018 +1100 Document torrc options that only work on v2 onion services RendPostPeriod only works on v2 onion services. HiddenServiceAuthorizeClient is not yet implemented for v3 onion services. Closes ticket #25661, bugfix on 0.3.2.1-alpha. --- doc/tor.1.txt | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/doc/tor.1.txt b/doc/tor.1.txt index 8ecc4175d..80769412f 100644 --- a/doc/tor.1.txt +++ b/doc/tor.1.txt @@ -2678,7 +2678,8 @@ The following options are used to configure a hidden service. spaces). If this option is set, the hidden service is not accessible for clients without authorization any more. Generated authorization data can be found in the hostname file. Clients need to put this authorization data in - their configuration file using **HidServAuth**. + their configuration file using **HidServAuth**. This option is only for v2 + services. [[HiddenServiceAllowUnknownPorts]] **HiddenServiceAllowUnknownPorts** **0**|**1**:: If set to 1, then connections to unrecognized ports do not cause the @@ -2700,7 +2701,8 @@ The following options are used to configure a hidden service. Every time the specified period elapses, Tor uploads any rendezvous service descriptors to the directory servers. This information is also uploaded whenever it changes. Minimum value allowed is 10 minutes and - maximum is 3.5 days. (Default: 1 hour) + maximum is 3.5 days. This option is only for v2 services. + (Default: 1 hour) [[HiddenServiceDirGroupReadable]] **HiddenServiceDirGroupReadable** **0**|**1**:: If this option is set to 1, allow the filesystem group to read the

1 0

[tor/release-0.3.3] Merge branch 'maint-0.3.3' into release-0.3.3
by nickm＠torproject.org 29 Mar '18

29 Mar '18

commit e47080a4d7e8f532e2278ddf75c7a144a6596b84 Merge: 720f044a2 6aaafb067 Author: Nick Mathewson <nickm(a)torproject.org> Date: Thu Mar 29 15:00:46 2018 -0400 Merge branch 'maint-0.3.3' into release-0.3.3 doc/tor.1.txt | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-)

1 0

[tor/maint-0.3.3] Document torrc options that only work on v2 onion services
by nickm＠torproject.org 29 Mar '18

29 Mar '18

commit 6aaafb0672ec7d2f454720ea0bed41fb91d9744e Author: teor <teor2345(a)gmail.com> Date: Wed Mar 28 17:10:31 2018 +1100 Document torrc options that only work on v2 onion services RendPostPeriod only works on v2 onion services. HiddenServiceAuthorizeClient is not yet implemented for v3 onion services. Closes ticket #25661, bugfix on 0.3.2.1-alpha. --- doc/tor.1.txt | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/doc/tor.1.txt b/doc/tor.1.txt index 8ecc4175d..80769412f 100644 --- a/doc/tor.1.txt +++ b/doc/tor.1.txt @@ -2678,7 +2678,8 @@ The following options are used to configure a hidden service. spaces). If this option is set, the hidden service is not accessible for clients without authorization any more. Generated authorization data can be found in the hostname file. Clients need to put this authorization data in - their configuration file using **HidServAuth**. + their configuration file using **HidServAuth**. This option is only for v2 + services. [[HiddenServiceAllowUnknownPorts]] **HiddenServiceAllowUnknownPorts** **0**|**1**:: If set to 1, then connections to unrecognized ports do not cause the @@ -2700,7 +2701,8 @@ The following options are used to configure a hidden service. Every time the specified period elapses, Tor uploads any rendezvous service descriptors to the directory servers. This information is also uploaded whenever it changes. Minimum value allowed is 10 minutes and - maximum is 3.5 days. (Default: 1 hour) + maximum is 3.5 days. This option is only for v2 services. + (Default: 1 hour) [[HiddenServiceDirGroupReadable]] **HiddenServiceDirGroupReadable** **0**|**1**:: If this option is set to 1, allow the filesystem group to read the

1 0