Core Tor Team July 2017 Report
In July we reviewed, triaged and fixed misc bugs related to releases 0.3.0 and 0.3.1:
* Never send a consensus which the downloader already has. [1]
This was an issue we identified that make the most trouble in test networks, where the consensus update rate is so fast that relays frequently try to download a consensus. We investigated and identified what caused the issue and fixed it.
* LZMA coder causes crash when the sandbox is enabled. [2]
While doing measurements for this project we noticed that Tor instances running as relays or authorities would sometimes crash when the sandbox is enabled. This is due to the MALLOC_MP_LIM value in sandbox.c, which is currently set to 16 MB, being too low. We limit our LZMA coder to only use 16 MB, but the coder allocates some additional data other than its internal buffer, which leads to the crash.
* Investigated: Assertion failure in consensus_cache_entry_handle_get on Windows.[3]
We created a mitigation and diagnostic branch. It doesn't fix this, but it makes it nonfatal and tries to get more useful info if it happens again. And receive a report that is still present in 0.3.1.4-alpha. Will continue investigation in August.
* Bug: src/common/compress.c:576: tor_compress_process: [4]
We fixed a bug found 0.3.1 at a stable Gentoo Linux server.
* Bridge unavailable during differential consensus update [5]
Bug reported for Raspberry Pi running Raspbian. we revised our threadpool implementation to prevent this background work from affecting other background work. There are 3 changelog entries for it in 0.3.1.5-alpha.
** Major bugfixes (relay, performance): Perform circuit handshake operations at a higher priority than we use for consensus diff creation and compression. This should prevent circuits from starving when a relay or bridge receives a new consensus, especially on lower-powered machines. Fixes bug 22883; bugfix on 0.3.1.1-alpha.
** Minor features (directory cache, consensus diff): Add a new MaxConsensusAgeForDiffs option to allow directory cache operators with low-resource environments to adjust the number of consensuses they'll store and generate diffs from. Most cache operators should leave it unchanged. Helps to work around bug 22883.
** Minor features (relay, performance): Always start relays with at least two worker threads, to prevent priority inversion on slow tasks. Part of the fix for bug 22883. Allow background work to be queued with different priorities, so that a big pile of slow low-priority jobs will not starve out higher priority jobs.This lays the groundwork for a fix for bug 22883.
* zstd tests fail with libzstd 1.3.0 [6]
Write zstd epilogues correctly when the epilogue requires reallocation of the output buffer, even with zstd 1.3.0. (Previously, we worked on 1.2.0 and failed with 1.3.0).
* consdiff test fails on OS X [7]
test_consdiff_base64cmp would fail on OS X because while OS X follows the standard of (less than zero/zero/greater than zero), it doesn't follow the convention of (-1/0/+1). Make the test comply with the standard.
We started to land updates to dir-spec.txt for prop #278 in GL/ahf/torspec/tree/prop278-in-dir-spec [8] And concluded our review of lz4 compressor [9]. For that we added a simple LZ4 test to our own little benchmarking tool [10].
Our results [11] (results) showed that LZ4 is a lot faster than anything else we have (even beats gzip at compression level 1), but the compression ratio is also worse than anything else we have for the cached-consensusdocument (~2.2 MB of structured text).
Since we are currently batch processing the compression of our "larger" files in the background we decided we wouldn't win much by including LZ4 here.
For August we feel that we are pretty close from finishing this deliverable and we plan to publish a blog post about compression + consensus diff work done with this project. Hope to share that soon.
[1] https://trac.torproject.org/projects/tor/ticket/22702 [2] https://trac.torproject.org/projects/tor/ticket/22751 [3] https://trac.torproject.org/projects/tor/ticket/22752 [4] https://trac.torproject.org/projects/tor/ticket/22719 [5] https://trac.torproject.org/projects/tor/ticket/22883 [6] https://trac.torproject.org/projects/tor/ticket/22927 [7] https://trac.torproject.org/projects/tor/ticket/22870 [8] https://trac.torproject.org/projects/tor/ticket/22275 [9] https://trac.torproject.org/projects/tor/ticket/22819 [10] https://gitlab.com/ahf/tor-sponsor4-compression/tree/master/bench [11] https://docs.google.com/spreadsheets/d/1WzFXQGfH8yI4WCCRAEQBt1Z_sC-3hMkmE6HN...