April 2018 - tor-commits - lists.torproject.org

[translation/tails-persistence-setup_completed] Update translations for tails-persistence-setup_completed
by translation＠torproject.org 01 Apr '18

01 Apr '18

commit 2fad2a15fb560bab4c78f50faceb1f7e64f211d0 Author: Translation commit bot <translation(a)torproject.org> Date: Sun Apr 1 15:45:54 2018 +0000 Update translations for tails-persistence-setup_completed --- pt_BR/pt_BR.po | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/pt_BR/pt_BR.po b/pt_BR/pt_BR.po index e21318972..cd66d1362 100644 --- a/pt_BR/pt_BR.po +++ b/pt_BR/pt_BR.po @@ -3,7 +3,7 @@ # This file is distributed under the same license as the PACKAGE package. # # Translators: -# Communia <ameaneantie(a)riseup.net>, 2013-2017 +# Communia <ameaneantie(a)riseup.net>, 2013-2018 # Bruno Geronimo <bruuh(a)outlook.com>, 2012 # carlo giusepe tadei valente sasaki <carlo.gt.valente(a)gmail.com>, 2014 # Communia <ameaneantie(a)riseup.net>, 2013 @@ -21,8 +21,8 @@ msgstr "" "Project-Id-Version: The Tor Project\n" "Report-Msgid-Bugs-To: Tails developers <tails(a)boum.org>\n" "POT-Creation-Date: 2017-05-15 13:51+0200\n" -"PO-Revision-Date: 2018-02-20 18:51+0000\n" -"Last-Translator: Diego Santos <diegospsi(a)opmbx.org>\n" +"PO-Revision-Date: 2018-04-01 15:37+0000\n" +"Last-Translator: Communia <ameaneantie(a)riseup.net>\n" "Language-Team: Portuguese (Brazil) (http://www.transifex.com/otf/torproject/language/pt_BR/)\n" "MIME-Version: 1.0\n" "Content-Type: text/plain; charset=UTF-8\n" @@ -64,7 +64,7 @@ msgstr "Perfis do Pidgin e chaveiro OTR (Off-The-Record)" #: ../lib/Tails/Persistence/Configuration/Presets.pm:88 msgid "Thunderbird" -msgstr "Perfis no Thunderbird et e-mails armazenados localmente" +msgstr "Thunderbird" #: ../lib/Tails/Persistence/Configuration/Presets.pm:90 msgid "Thunderbird profiles and locally stored email"

1 0

[translation/tails-persistence-setup] Update translations for tails-persistence-setup
by translation＠torproject.org 01 Apr '18

01 Apr '18

commit b78d50c002c5b13a5f6bb4a0c409a9849ced17b8 Author: Translation commit bot <translation(a)torproject.org> Date: Sun Apr 1 15:45:48 2018 +0000 Update translations for tails-persistence-setup --- pt_BR/pt_BR.po | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/pt_BR/pt_BR.po b/pt_BR/pt_BR.po index e21318972..cd66d1362 100644 --- a/pt_BR/pt_BR.po +++ b/pt_BR/pt_BR.po @@ -3,7 +3,7 @@ # This file is distributed under the same license as the PACKAGE package. # # Translators: -# Communia <ameaneantie(a)riseup.net>, 2013-2017 +# Communia <ameaneantie(a)riseup.net>, 2013-2018 # Bruno Geronimo <bruuh(a)outlook.com>, 2012 # carlo giusepe tadei valente sasaki <carlo.gt.valente(a)gmail.com>, 2014 # Communia <ameaneantie(a)riseup.net>, 2013 @@ -21,8 +21,8 @@ msgstr "" "Project-Id-Version: The Tor Project\n" "Report-Msgid-Bugs-To: Tails developers <tails(a)boum.org>\n" "POT-Creation-Date: 2017-05-15 13:51+0200\n" -"PO-Revision-Date: 2018-02-20 18:51+0000\n" -"Last-Translator: Diego Santos <diegospsi(a)opmbx.org>\n" +"PO-Revision-Date: 2018-04-01 15:37+0000\n" +"Last-Translator: Communia <ameaneantie(a)riseup.net>\n" "Language-Team: Portuguese (Brazil) (http://www.transifex.com/otf/torproject/language/pt_BR/)\n" "MIME-Version: 1.0\n" "Content-Type: text/plain; charset=UTF-8\n" @@ -64,7 +64,7 @@ msgstr "Perfis do Pidgin e chaveiro OTR (Off-The-Record)" #: ../lib/Tails/Persistence/Configuration/Presets.pm:88 msgid "Thunderbird" -msgstr "Perfis no Thunderbird et e-mails armazenados localmente" +msgstr "Thunderbird" #: ../lib/Tails/Persistence/Configuration/Presets.pm:90 msgid "Thunderbird profiles and locally stored email"

1 0

[torbirdy/master] Update Enigmail keyserver settings
by sukhbir＠torproject.org 01 Apr '18

01 Apr '18

commit 819842ba97d1c2358ced5c648b527162995b35e2 Author: Sukhbir Singh <sukhbir(a)torproject.org> Date: Sun Apr 1 10:25:20 2018 -0400 Update Enigmail keyserver settings Assume that a user has dirmngr running and configured with `use-tor' in dirmngr.conf and set the relevant keyserver settings (similar to what Tails has been doing); otherwise the keyserver lookup will fail-safe. This should work for the daemon and Tor Browser since as per gnupg/dirmngr/dns-stuff.c, it tries both 9050 and 9150. This is probably better than the older setting because a) it actually works now, b) it will still fail-safe in case Tor is not running. --- chrome/content/preferences.js | 8 ++------ components/torbirdy.js | 4 ++-- 2 files changed, 4 insertions(+), 8 deletions(-) diff --git a/chrome/content/preferences.js b/chrome/content/preferences.js index 8acd6ba..2980857 100644 --- a/chrome/content/preferences.js +++ b/chrome/content/preferences.js @@ -42,14 +42,10 @@ if (!org.torbirdy.prefs) org.torbirdy.prefs = new function() { opts += "--throw-keyids "; } if (! pub.prefs.getBoolPref("extensions.torbirdy.gpg_already_torified")) { - var proxy = "socks5h://127.0.0.1:9150"; if (anonService === "jondo") { - proxy = "http://127.0.0.1:4001"; + let proxy = "http://127.0.0.1:4001"; + opts += "--keyserver-options=no-try-dns-srv,http-proxy=" + proxy + " "; } - if (anonService === "custom") { - proxy = "socks5h://" + pub.prefs.getCharPref("network.proxy.socks") + ":" + pub.prefs.getIntPref("network.proxy.socks_port"); - } - opts += "--keyserver-options=no-try-dns-srv,http-proxy=" + proxy + " "; } return opts + diff --git a/components/torbirdy.js b/components/torbirdy.js index e8d2cd2..6c2be57 100644 --- a/components/torbirdy.js +++ b/components/torbirdy.js @@ -274,8 +274,8 @@ var TorBirdyPrefs = { "--no-comments " + // We want to force UTF-8 everywhere "--display-charset utf-8 " + - // We want to ensure that Enigmail is proxy aware even when it runs gpg in a shell - "--keyserver-options http-proxy=socks5h://127.0.0.1:9150 ", + // Set additional keyserver options + "--keyserver-options no-auto-key-retrieve ", // The default key server should be a hidden service; use the Tor OnionBalance hidden service pool (https://sks-keyservers.net/overview-of-pools.php#pool_tor) "extensions.enigmail.keyserver": "hkp://jirk5u4osbsr34t5.onion",

1 0

[stem/master] Drop compression integ test
by atagar＠torproject.org 01 Apr '18

01 Apr '18

commit f7949b644eaadf9a0e731547deafb40cf106e8a8 Author: Damian Johnson <atagar(a)torproject.org> Date: Sat Mar 31 21:29:06 2018 -0700 Drop compression integ test None of the dirauths evidently support zstd, and individual descriptors can't be provided via lzma (evidently only the whole consensus will honor that compression due to its heavier cpu costs). This doesn't leave us with great options for integ testing. Oh well, at least we have unit tests. --- test/integ/descriptor/remote.py | 20 -------------------- 1 file changed, 20 deletions(-) diff --git a/test/integ/descriptor/remote.py b/test/integ/descriptor/remote.py index 4e10eac2..3123ef06 100644 --- a/test/integ/descriptor/remote.py +++ b/test/integ/descriptor/remote.py @@ -16,26 +16,6 @@ import test.require class TestDescriptorDownloader(unittest.TestCase): @test.require.only_run_once @test.require.online - def test_compression(self): - """ - Issue a request for a plaintext descriptor. - """ - - moria1 = stem.descriptor.remote.get_authorities()['moria1'] - - descriptors = list(stem.descriptor.remote.Query( - '/tor/server/fp/%s' % moria1.fingerprint, - 'server-descriptor 1.0', - endpoints = [(moria1.address, moria1.dir_port)], - timeout = 30, - validate = True, - ).run()) - - self.assertEqual(1, len(descriptors)) - self.assertEqual('moria1', descriptors[0].nickname) - - @test.require.only_run_once - @test.require.online def test_shorthand_aliases(self): """ Quick sanity test that we can call our shorthand aliases for getting

1 0

[stem/master] Working zstd support
by atagar＠torproject.org 01 Apr '18

01 Apr '18

commit 5da64e67948c07a3c509825e6fecef32cd7bbd8d Author: Damian Johnson <atagar(a)torproject.org> Date: Sat Mar 31 21:13:18 2018 -0700 Working zstd support Oops, turns out we weren't using the officially suggested zstd python module. Moving to the following did the trick... https://pypi.python.org/pypi/zstandard --- stem/descriptor/remote.py | 52 +++++++++++++++++++++++++++++++---------------- 1 file changed, 35 insertions(+), 17 deletions(-) diff --git a/stem/descriptor/remote.py b/stem/descriptor/remote.py index 9b71be8d..e1190fc7 100644 --- a/stem/descriptor/remote.py +++ b/stem/descriptor/remote.py @@ -92,8 +92,8 @@ content. For example... =============== =========== **PLAINTEXT** Uncompressed data. **GZIP** `GZip compression <https://www.gnu.org/software/gzip/>`_. - **ZSTD** `Zstandard compression <https://www.zstd.net>`_. - **LZMA** `LZMA compression <https://en.wikipedia.org/wiki/LZMA>`_. + **ZSTD** `Zstandard compression <https://www.zstd.net>`_, this requires the `zstandard module <https://pypi.python.org/pypi/zstandard>`_. + **LZMA** `LZMA compression <https://en.wikipedia.org/wiki/LZMA>`_, this requires the 'lzma module <https://docs.python.org/3/library/lzma.html>`_. =============== =========== """ @@ -133,9 +133,18 @@ except ImportError: LZMA_SUPPORTED = False try: - # https://pypi.python.org/pypi/zstd + # We use the suggested python zstd library... + # + # https://pypi.python.org/pypi/zstandard + # + # Unfortunately this installs as a zstd module which can be confused with... + # + # https://pypi.python.org/pypi/zstd + # + # As such checking for the specific decompression class we'll need. + import zstd - ZSTD_SUPPORTED = True + ZSTD_SUPPORTED = hasattr(zstd, 'ZstdDecompressor') except ImportError: ZSTD_SUPPORTED = False @@ -146,8 +155,8 @@ Compression = stem.util.enum.Enum( ('LZMA', 'x-tor-lzma'), ) -ZSTD_UNAVAILABLE_MSG = 'ZSTD is not yet supported' -LZMA_UNAVAILABLE_MSG = 'LZMA compression was requested but requires the lzma module, which was added in python 3.3' +ZSTD_UNAVAILABLE_MSG = 'ZSTD compression requires the zstandard module (https://pypi.python.org/pypi/zstandard)' +LZMA_UNAVAILABLE_MSG = 'LZMA compression requires the lzma module (https://docs.python.org/3/library/lzma.html)' # Tor has a limited number of descriptors we can fetch explicitly by their # fingerprint or hashes due to a limit on the url length by squid proxies. @@ -307,9 +316,9 @@ class Query(object): /tor/keys/fp/<v3ident1>+<v3ident2> key certificates for specific authorities =============================================== =========== - **LZMA** compression requires the `lzma module - <https://docs.python.org/3/library/lzma.html>`_ which was added in Python - 3.3. + **ZSTD** compression requires `zstandard + <https://pypi.python.org/pypi/zstandard>`_, and **LZMA** requires the `lzma + module <https://docs.python.org/3/library/lzma.html>`_. For legacy reasons if our resource has a '.z' suffix then our **compression** argument is overwritten with Compression.GZIP. @@ -367,14 +376,14 @@ class Query(object): if isinstance(compression, str): compression = [compression] # caller provided only a single option - if Compression.LZMA in compression and not LZMA_SUPPORTED: - log.log_once('stem.descriptor.remote.lzma_unavailable', log.INFO, LZMA_UNAVAILABLE_MSG) - compression.remove(Compression.LZMA) - if Compression.ZSTD in compression and not ZSTD_SUPPORTED: log.log_once('stem.descriptor.remote.zstd_unavailable', log.INFO, ZSTD_UNAVAILABLE_MSG) compression.remove(Compression.ZSTD) + if Compression.LZMA in compression and not LZMA_SUPPORTED: + log.log_once('stem.descriptor.remote.lzma_unavailable', log.INFO, LZMA_UNAVAILABLE_MSG) + compression.remove(Compression.LZMA) + if not compression: compression = [Compression.PLAINTEXT] @@ -528,13 +537,22 @@ class Query(object): data = response.read() encoding = response.info().getheader('Content-Encoding') - if encoding in (Compression.GZIP, 'deflate'): - # The '32' is for automatic header detection... - # https://stackoverflow.com/questions/3122145/zlib-error-error-3-while-decomp… + # Tor doesn't include compression headers. As such when using gzip we + # need to include '32' for automatic header detection... + # + # https://stackoverflow.com/questions/3122145/zlib-error-error-3-while-decomp… + # + # ... and with zstd we need to use the streaming API. + if encoding in (Compression.GZIP, 'deflate'): data = zlib.decompress(data, zlib.MAX_WBITS | 32) elif encoding == Compression.ZSTD and ZSTD_SUPPORTED: - data = zstd.decompress(data) + output_buffer = io.BytesIO() + + with zstd.ZstdDecompressor().write_to(output_buffer) as decompressor: + decompressor.write(data) + + data = output_buffer.getvalue() elif encoding == Compression.LZMA and LZMA_SUPPORTED: data = lzma.decompress(data)

1 0

[stem/master] Unit test descriptor compression
by atagar＠torproject.org 01 Apr '18

01 Apr '18

commit 522422427ec2b8427f9eb7baefb6261941772e85 Author: Damian Johnson <atagar(a)torproject.org> Date: Sat Mar 31 12:00:54 2018 -0700 Unit test descriptor compression Mocking download responses so we can add unit test coverage of our compression support. --- test/unit/descriptor/__init__.py | 9 +++ test/unit/descriptor/data/compressed_gzip | Bin 0 -> 1543 bytes test/unit/descriptor/data/compressed_identity | 52 +++++++++++++++++ test/unit/descriptor/data/compressed_lzma | 0 test/unit/descriptor/data/compressed_zstd | Bin 0 -> 1550 bytes test/unit/descriptor/remote.py | 77 +++++++++++++++++++++++++- 6 files changed, 136 insertions(+), 2 deletions(-) diff --git a/test/unit/descriptor/__init__.py b/test/unit/descriptor/__init__.py index bf0e8619..bce8d3d0 100644 --- a/test/unit/descriptor/__init__.py +++ b/test/unit/descriptor/__init__.py @@ -25,6 +25,15 @@ def get_resource(filename): return os.path.join(DESCRIPTOR_TEST_DATA, filename) +def read_resource(filename): + """ + Provides test data. + """ + + with open(get_resource(filename), 'rb') as resource_file: + return resource_file.read() + + def base_expect_invalid_attr(cls, default_attr, default_value, test, desc_attrs, attr = None, expected_value = None): return base_expect_invalid_attr_for_text(cls, default_attr, default_value, test, cls.content(desc_attrs), attr, expected_value) diff --git a/test/unit/descriptor/data/compressed_gzip b/test/unit/descriptor/data/compressed_gzip new file mode 100644 index 00000000..2b2dc642 Binary files /dev/null and b/test/unit/descriptor/data/compressed_gzip differ diff --git a/test/unit/descriptor/data/compressed_identity b/test/unit/descriptor/data/compressed_identity new file mode 100644 index 00000000..cd5b56b4 --- /dev/null +++ b/test/unit/descriptor/data/compressed_identity @@ -0,0 +1,52 @@ +router moria1 128.31.0.34 9101 0 9131 +identity-ed25519 +-----BEGIN ED25519 CERT----- +AQQABnxNAQS9ja600v/ZodOUiu7NepTkbPIOrFPgEVQE+03rGBtPAQAgBADKnR/C +2nhpr9UzJkkbPy83sqbfNh63VgFnCpkSTULAcq52z8xM7raRDCiTJTu/FK/BJGgE +dJcFQ8MgZJOuYgFKcMVyQ6j2FGbhDI0zQTK1+TAPNRG4ixiF7h7wqDT9Ugw= +-----END ED25519 CERT----- +master-key-ed25519 yp0fwtp4aa/VMyZJGz8vN7Km3zYet1YBZwqZEk1CwHI +platform Tor 0.3.4.0-alpha-dev on Linux +proto Cons=1-2 Desc=1-2 DirCache=1-2 HSDir=1-2 HSIntro=3-4 HSRend=1-2 Link=1-5 LinkAuth=1,3 Microdesc=1-2 Relay=1-2 +published 2018-03-31 04:17:41 +fingerprint 9695 DFC3 5FFE B861 329B 9F1A B04C 4639 7020 CE31 +uptime 295196 +bandwidth 512000 62914560 3403447 +extra-info-digest 393861CB4D9A0480E5A58A3005A88DD7F09271E3 CCbwxMembtsEKAVkU2bqCiBhKaokRlJv6077uy5kI4Q +caches-extra-info +onion-key +-----BEGIN RSA PUBLIC KEY----- +MIGJAoGBAKsNybH/i/nMLDwSr6WYd77EV6IgiuhfD7UBbsifsDywTe+YrZU3Z4O+ +DY9jemMPRA7wYioJrmXyMVTsbktjWUtWFge1oUj2xsxO3cufCavLmmkmqpkHL3Wn +xRJraupgfLK72/UN8wXB0duSAU/DH9hTg7WRhkFZdRJK7rostJvZAgMBAAE= +-----END RSA PUBLIC KEY----- +signing-key +-----BEGIN RSA PUBLIC KEY----- +MIGJAoGBALtJ9uD7cD7iHjqNA3AgsX9prES5QN+yFQyr2uOkxzhvunnaf6SNhzWW +bkfylnMrRm/qCz/czcjZO6N6EKHcXmypehvP566B7gAQ9vDsb+l7VZVWgXvzNc2s +tl3P7qpC08rgyJh1GqmtQTCesIDqkEyWxwToympCt09ZQRq+fIttAgMBAAE= +-----END RSA PUBLIC KEY----- +onion-key-crosscert +-----BEGIN CROSSCERT----- +A9x30r1LTl6rRUe4/irB6/cdNGmd95+bOuArjrmuwSI8Crerx3uNbbN8/iRlRSec +fBqElxb8SvEGONvtyjGVxp+t5QQS381Bah7DRMz7MfRvOgD8UoZ4DEcpMBnJnA+O +5OnAoBVOProdfUvFfEnBWJY+ELB3ShG4A3TL4c8QrwA= +-----END CROSSCERT----- +ntor-onion-key-crosscert 0 +-----BEGIN ED25519 CERT----- +AQoABnadAcqdH8LaeGmv1TMmSRs/Lzeypt82HrdWAWcKmRJNQsByAEnIlLlEtELa +EZA/y3Q5wXxA20upqmSKoCt1s8a6cXHqnsbha61MVbRLVCgWlWmKNjBSu+GiXeXC +IJ9Tr+aYawM= +-----END ED25519 CERT----- +hidden-service-dir +contact 1024D/28988BF5 arma mit edu +ntor-onion-key a3r4k2huGP4YdHTQ8ldJhZkm5O+1tLaKE//o5DSibU0= +reject *:* +tunnelled-dir-server +router-sig-ed25519 FkY6h3SMPWPt1fQlU8jsFllGWIy+lw7SGTjTV2FZVNlrHYHcS1F2L7bMGu38ljZ7J+VIBssRZnaiQVEXbI9uDQ +router-signature +-----BEGIN SIGNATURE----- +UzHGCcQK1XGntL3MVvcgcj0kpCsEMf5UIEj4eWOtCjWyriOX7nqwayc5vWOubZLN +pAZ27MjVmMbVGgtS0xFZ/UQrGEeznITaZwjLNd1IOliadKPdOp4w4OFcjbkCytk6 +MdAEyxwnZzxUwCex0Kj+3FWXQTr09VnXMg5/+vdbCUY= +-----END SIGNATURE----- diff --git a/test/unit/descriptor/data/compressed_lzma b/test/unit/descriptor/data/compressed_lzma new file mode 100644 index 00000000..e69de29b diff --git a/test/unit/descriptor/data/compressed_zstd b/test/unit/descriptor/data/compressed_zstd new file mode 100644 index 00000000..b3c3269b Binary files /dev/null and b/test/unit/descriptor/data/compressed_zstd differ diff --git a/test/unit/descriptor/remote.py b/test/unit/descriptor/remote.py index 41585fc2..05cf1d57 100644 --- a/test/unit/descriptor/remote.py +++ b/test/unit/descriptor/remote.py @@ -11,6 +11,7 @@ import stem.prereq import stem.util.conf from stem.descriptor.remote import Compression +from test.unit.descriptor import read_resource try: # added in python 2.7 @@ -110,14 +111,18 @@ FALLBACK_ENTRY = b"""\ """ -def _urlopen_mock(data): +def _urlopen_mock(data, encoding = 'identity'): urlopen_mock = Mock() urlopen_mock().read.return_value = data - urlopen_mock().info().getheader.return_value = 'identity' + urlopen_mock().info().getheader.return_value = encoding return urlopen_mock class TestDescriptorDownloader(unittest.TestCase): + def tearDown(self): + # prevent our mocks from impacting other tests + stem.descriptor.remote.SINGLETON_DOWNLOADER = None + def test_gzip_url_override(self): query = stem.descriptor.remote.Query(TEST_RESOURCE, start = False) self.assertEqual([Compression.PLAINTEXT], query.compression) @@ -145,6 +150,74 @@ class TestDescriptorDownloader(unittest.TestCase): query = stem.descriptor.remote.Query(TEST_RESOURCE, compression = Compression.LZMA, start = False) self.assertEqual([Compression.PLAINTEXT], query.compression) + @patch(URL_OPEN, _urlopen_mock(read_resource('compressed_identity'), encoding = 'identity')) + def test_compression_plaintext(self): + """ + Download a plaintext descriptor. + """ + + descriptors = list(stem.descriptor.remote.get_server_descriptors( + '9695DFC35FFEB861329B9F1AB04C46397020CE31', + compression = Compression.PLAINTEXT, + validate = True, + )) + + self.assertEqual(1, len(descriptors)) + self.assertEqual('moria1', descriptors[0].nickname) + + @patch(URL_OPEN, _urlopen_mock(read_resource('compressed_gzip'), encoding = 'gzip')) + def test_compression_gzip(self): + """ + Download a gip compressed descriptor. + """ + + descriptors = list(stem.descriptor.remote.get_server_descriptors( + '9695DFC35FFEB861329B9F1AB04C46397020CE31', + compression = Compression.GZIP, + validate = True, + )) + + self.assertEqual(1, len(descriptors)) + self.assertEqual('moria1', descriptors[0].nickname) + + @patch(URL_OPEN, _urlopen_mock(read_resource('compressed_zstd'), encoding = 'x-zstd')) + def test_compression_zstd(self): + """ + Download a zstd compressed descriptor. + """ + + if not stem.descriptor.remote.ZSTD_SUPPORTED: + self.skipTest('(requires zstd module)') + return + + descriptors = list(stem.descriptor.remote.get_server_descriptors( + '9695DFC35FFEB861329B9F1AB04C46397020CE31', + compression = Compression.ZSTD, + validate = True, + )) + + self.assertEqual(1, len(descriptors)) + self.assertEqual('moria1', descriptors[0].nickname) + + @patch(URL_OPEN, _urlopen_mock(read_resource('compressed_lzma'), encoding = 'x-tor-lzma')) + def test_compression_lzma(self): + """ + Download a lzma compressed descriptor. + """ + + if not stem.descriptor.remote.LZMA_SUPPORTED: + self.skipTest('(requires lzma module)') + return + + descriptors = list(stem.descriptor.remote.get_server_descriptors( + '9695DFC35FFEB861329B9F1AB04C46397020CE31', + compression = Compression.LZMA, + validate = True, + )) + + self.assertEqual(1, len(descriptors)) + self.assertEqual('moria1', descriptors[0].nickname) + @patch(URL_OPEN, _urlopen_mock(TEST_DESCRIPTOR)) def test_query_download(self): """

1 0

[stem/master] Initial zstd attempt
by atagar＠torproject.org 01 Apr '18

01 Apr '18

commit 7c09e7ec3d079a9295eea2ebe326c83569bde6b8 Author: Damian Johnson <atagar(a)torproject.org> Date: Sat Mar 31 19:58:31 2018 -0700 Initial zstd attempt Doesn't yet properly decompress. Still figuring out this and lzma... :/ --- stem/descriptor/remote.py | 13 +++++++++---- 1 file changed, 9 insertions(+), 4 deletions(-) diff --git a/stem/descriptor/remote.py b/stem/descriptor/remote.py index ab61d3a7..9b71be8d 100644 --- a/stem/descriptor/remote.py +++ b/stem/descriptor/remote.py @@ -92,7 +92,7 @@ content. For example... =============== =========== **PLAINTEXT** Uncompressed data. **GZIP** `GZip compression <https://www.gnu.org/software/gzip/>`_. - **ZSTD** `Zstandard compression <https://www.zstd.net>`_ + **ZSTD** `Zstandard compression <https://www.zstd.net>`_. **LZMA** `LZMA compression <https://en.wikipedia.org/wiki/LZMA>`_. =============== =========== """ @@ -132,7 +132,12 @@ try: except ImportError: LZMA_SUPPORTED = False -ZSTD_SUPPORTED = False +try: + # https://pypi.python.org/pypi/zstd + import zstd + ZSTD_SUPPORTED = True +except ImportError: + ZSTD_SUPPORTED = False Compression = stem.util.enum.Enum( ('PLAINTEXT', 'identity'), @@ -528,8 +533,8 @@ class Query(object): # https://stackoverflow.com/questions/3122145/zlib-error-error-3-while-decomp… data = zlib.decompress(data, zlib.MAX_WBITS | 32) - elif encoding == Compression.ZSTD: - pass # TODO: implement + elif encoding == Compression.ZSTD and ZSTD_SUPPORTED: + data = zstd.decompress(data) elif encoding == Compression.LZMA and LZMA_SUPPORTED: data = lzma.decompress(data)

1 0

[stem/master] Fix lzma unit test
by atagar＠torproject.org 01 Apr '18

01 Apr '18

commit 9aa1f7eee610db1d0f55d529d5092691b09bb46e Author: Damian Johnson <atagar(a)torproject.org> Date: Sat Mar 31 20:12:52 2018 -0700 Fix lzma unit test Lzma test data was empty since it doesn't work for individual descriptors... https://trac.torproject.org/projects/tor/ticket/25667#comment:10 I downloaded the lzma consensus and we can decompress it, but we can't include it as test data (too large). As such using the lzma library to compress a single descriptor for this test. --- test/unit/descriptor/data/compressed_lzma | Bin 0 -> 1652 bytes 1 file changed, 0 insertions(+), 0 deletions(-) diff --git a/test/unit/descriptor/data/compressed_lzma b/test/unit/descriptor/data/compressed_lzma index e69de29b..e3297504 100644 Binary files a/test/unit/descriptor/data/compressed_lzma and b/test/unit/descriptor/data/compressed_lzma differ

1 0

[stem/master] Expanded descriptor compression support
by atagar＠torproject.org 01 Apr '18

01 Apr '18

commit 079b48126fdc97d83df84d96e7423f99870f2584 Merge: 4a8f8d04 f7949b64 Author: Damian Johnson <atagar(a)torproject.org> Date: Sat Mar 31 21:39:00 2018 -0700 Expanded descriptor compression support We supported plaintext and gzip when downloading descriptors, but recently tor added lzma and zstd support as well... https://gitweb.torproject.org/torspec.git/commit/?id=1cb56af Lzma support was added in Python 3.3... https://docs.python.org/3/library/lzma.html ... and zstd is covered by the zstandard module... https://pypi.python.org/pypi/zstandard docs/change_log.rst | 1 + stem/descriptor/remote.py | 182 ++++++++++++++++++++------ stem/version.py | 2 + test/unit/descriptor/__init__.py | 9 ++ test/unit/descriptor/data/compressed_gzip | Bin 0 -> 1543 bytes test/unit/descriptor/data/compressed_identity | 52 ++++++++ test/unit/descriptor/data/compressed_lzma | Bin 0 -> 1652 bytes test/unit/descriptor/data/compressed_zstd | Bin 0 -> 1550 bytes test/unit/descriptor/remote.py | 174 ++++++++++++++++++------ 9 files changed, 341 insertions(+), 79 deletions(-)

1 0