December 2015 - tor-dev - lists.torproject.org

Have the Tor daemon use more cores
by Jeremy 03 Dec '15

03 Dec '15

On 12/02/2015 07:00 AM, tor-dev-request(a)lists.torproject.org wrote: >> >I've been working on the volunteer project described here >> >https://www.torproject.org/getinvolved/volunteer.html.en#useMoreCores >> >but can't spend much more time on it. >> > >> >Right now, I have refactored circuit_receive_relay_cell() in relay.c >> >(which calls relay_crypt() and eventually the AES crypt routines) to >> >use the workqueue.c infrastructure similar to cpuworker.c. >> > >> >When the refactored code runs in single threaded mode, all seems >> >good. Once I activate the thread pool and start sending it work with >> >threadpool_queue_work(), it Bootstraps 100% okay and runs for several >> >minutes before crashing on cells it doesn't handle properly. >> > >> >I'm happy to share my code with anyone interested. > Sure I think I'm the person that used the workqueue stuff last, so I'd > be up for doing the rest of the work (nb: I am traveling/in meetings for > the rest of the week, so it'll be next week at the earliest that I can > really spend quality time). > > Regards, > > -- Yawning Angel Yawning, or anyone else: If you assess the code has potential, I can assist & work on it more. If it is a big miss, then I might start with a smaller contribution. Thanks, Jeremy

1 0

txtorcon 0.14.2
by meejah 03 Dec '15

03 Dec '15

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 With Twisted 15.5.0, a deprecated endpoint interface class is still being used. This release backports the fix for that to the 0.14.x branch. You can download the release from PyPI or GitHub (or of course "pip install txtorcon"): https://pypi.python.org/pypi/txtorcon/0.14.2 https://github.com/meejah/txtorcon/releases/tag/v0.14.2 Releases are also available from the hidden service: http://timaq4ygg2iegci7.onion/txtorcon-0.14.2.tar.gz http://timaq4ygg2iegci7.onion/txtorcon-0.14.2.tar.gz.asc http://timaq4ygg2iegci7.onion/txtorcon-0.14.2-py2-none-any.whl http://timaq4ygg2iegci7.onion/txtorcon-0.14.2-py2-none-any.whl.asc You can verify the sha256sum of both by running the following 4 lines in a shell wherever you have the files downloaded: cat <<EOF | sha256sum --check fbc95c41e924b0e10156c46227eac2b4acf42a3b8d01f0ea3022897a10dc059a txtorcon-0.14.2-py2-none-any.whl f99819b1a71b8dea9e80317ec83c990b4ff608c98bc78a9fc1dc9991d349d13f txtorcon-0.14.2.tar.gz EOF thanks, meejah -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.12 (GNU/Linux) iQEcBAEBAgAGBQJWX3WRAAoJEMJgKAMSgGmnOasH/i3NuaGRXu/wgdSHwDldPCpt FCVFXN9RT8nTV8EN72V1iwqW0ir38YG8rM9TVNbImZlPFbMLaV0zJsb+rNId5xgb uNBnlzrzdJOtlgofg4aKHBHjvDioV4xrZiypzOOc5u2Hyd8xuOPkI3fEVx+QfaKA Cjbys6lCM2HZpeWkHlPKM3xnN9BoJ7IZodbv54fEehR5MgljkgwCoQJOdDy8tf9L FQmpDXb7ptNXqbHNUltg8zkyIt345u6XcvZWX8wCxVu4PHCucdUW0VWzq33dNCuA jbGvsesrJsr6+2a6wMq0FXJByaU1ymkwdp0OV55ASdNg5KBcfh4Sf4gMNiYGQw8= =bBHR -----END PGP SIGNATURE-----

1 0

Have the Tor daemon use more cores
by Jeremy 03 Dec '15

03 Dec '15

>> I've been working on the volunteer project described here >> >https://www.torproject.org/getinvolved/volunteer.html.en#useMoreCores but >> >can't spend much more time on it. >> > >> >Right now, I have refactored circuit_receive_relay_cell() in relay.c (which >> >calls relay_crypt() and eventually the AES crypt routines) to use the >> >workqueue.c infrastructure similar to cpuworker.c. >> > >> >When the refactored code runs in single threaded mode, all seems good. Once >> >I activate the thread pool and start sending it work with >> >threadpool_queue_work(), it Bootstraps 100% okay and runs for several >> >minutes before crashing on cells it doesn't handle properly. >> > >> >I'm happy to share my code with anyone interested. > Hi Jeremy! > > This is great! Thanks for this. You should definitely post your code (link to > branch or .diff) to ticket #7572 [1] so someone at some point can take a look > at it and help you out. > > Thanks! > David > > [1]https://trac.torproject.org/projects/tor/ticket/7572 > Thank you. I have posted my code and tried to explain my thought process on ticket #7572.

1 0

Scaling Tor Metrics
by Karsten Loesing 01 Dec '15

01 Dec '15

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hello devs, the Tor Metrics website [0] claims to be "the primary place to learn interesting facts about the Tor network" and invites its visitors who "come across something that is missing" to contact the website authors about it. That's a bold statement I put there! :) Yet, there's considerable product backlog with possible enhancements [1] that doesn't seem to ever become shorter. Even worse, it can be expected that the backlog will refill quickly once the community notices that feature requests are suddenly considered. The main reason for this unfortunate situation is that Tor Metrics contains many moving parts, including some heavy database lifting that takes place below the surface, that all want to be maintained. Adding more parts just makes the whole thing even more likely to break. At the same time, knowing about the situation that Tor Metrics has become almost closed to contributions is painful. This posting shall discuss possible solutions. The goal is to let Tor Metrics grow in a healthy fashion that encourages contributions from the community. These solutions are not mutually exclusive, and the best solution may use parts of more than one solution sketched out here. 1 Make Tor Metrics better and bigger, internally The obvious solution is that the maintainers of Tor Metrics could just work harder to overcome the problems stated above. Let's think this through. 1.1 Add more development resources If only the current Tor Metrics maintainers had more time to devote to cleaning up existing parts and to add new parts, that would solve our problem. They could refactor parts that are hard to maintain, and they could work off the serious backlog that has piled up. Of course, this means dropping or handing over responsibilities for other products, and it may mean finding (and paying) new developers to help maintain Tor Metrics. It's unclear whether anything like this would fit into Tor's budget, and whether these changed priorities would make users of tools that had to be dropped or handed over unhappy. 1.2 Rewrite internal parts of Tor Metrics to encourage external contributions Most of Tor Metrics would have run 10 or 15 years ago with only minor modifications. It's not necessarily a bad thing to use established technologies. But maybe, if we rewrite it using modern data-processing, web, and visualization frameworks, it becomes more attractive to other developers to contribute code and help maintain existing (well, then rewritten) code. The result would be a larger Tor Metrics website that is easier to maintain and hopefully maintained by more people. It's unclear how realistic this plan is, though, and it requires attention by Tor Metrics maintainers to bring it enough into shape for external contributors to get involved. 2 Add more ways to contribute to Tor Metrics externally It may be possible to further grow Tor Metrics without adding more code to it, hence not making it any harder to maintain. However, if code to generate visualizations is run elsewhere, there's a certain risk that results are not perceived as trustworthy as if that code were run as part of Metrics. This is primarily a problem of setting user expectations right. We could add different ways for contributing to Tor Metrics, depending on the level of commitment that contributors are willing to make. Possible new ways (in addition to filing a Trac ticket, which is already possible, though not very effective) are: 2.1 Accept contribution of static data or static graphs Somebody might contribute data (in a tarball, download link, etc.) or a static graph (static as in "doesn't break, ever", not "static HTML with a tiny amount of JavaScript that will surely never break"). The Tor Metrics team reviews that and puts it on the Tor Metrics website, together with a short description, author information, license, etc. There are plenty of visualizations on Trac and on the mailing lists, so we'll have to define criteria what we add and what not, and we'll need a good process for making that happen. 2.2 Link to external websites Somebody might write a website that visualizes Tor network data. The Tor Metrics team reviews the idea behind it, but not necessarily look at its code, and adds an external link to Tor Metrics. It becomes obvious that the authors remain responsible for their visualization, so there's no risk involved for Tor Metrics, but users may not trust it as much, because it doesn't have the Tor Metrics label. Note that we're already doing this approach by linking to the visualizations showing "Tor users as percentage of larger Internet population" [2] and "Data flow in the Tor network" [3]. Also note that we could as well have hosted the former directly on Tor Metrics with appropriate attribution, because it's a static image. This is not the case with the latter. 2.3 Run an externally developed website as if it were part of Tor Metrics Let's imagine that somebody produces a visualization of Tor network data and would like to make it part of Tor Metrics but without limiting themselves to the technology used by Tor Metrics. We could let them write their visualization as website and integrate it into Tor Metrics after reviewing its code. Technically, part of this integration would be to "redress" the website by applying the Tor Metrics design (which has lots of room for improvement, but let's just say the result will look as seamlessly integrated into Tor Metrics as the "Network bubble graphs" [4]). Another part would probably be to rewrite web requests, so that users still think they're talking to https://metrics.torproject.org/, but really they're talking to another webserver behind that. Regarding hosting and maintenance, in theory, the website could be hosted by the original creators, but that effectively means that the Tor Metrics team gives up part of the control about what's on the Tor Metrics website. The creators of the external website could change parts or add new parts that wouldn't be reviewed by Tor Metrics developers, but they would be perceived as part of Metrics, which seems bad. The Tor Metrics team could run the externally developed website on a separate host or on the same host as Tor Metrics. We could imagine variants where the original creator stays around to fix any issues as they come up, or we could imagine that they donate their visualization that the Tor Metrics people will then maintain. We could even imagine that the Tor Metrics maintainers some day decide to integrate the originally external website into Tor Metrics proper, but that would not be required for this model to work. All these ideas require writing down guidelines, criteria, and processes. In particular, they require more thoughts and input from other people who are not currently involved in Tor Metrics maintenance and who can be expected more objective. And once these ideas are implemented, we'll need more Tor Metrics maintainer than just one. What are your thoughts? All the best, Karsten [0] https://metrics.torproject.org/ [1] https://trac.torproject.org/projects/tor/query?status=!closed&component=Met… [2] https://metrics.torproject.org/oxford-anonymous-internet.html [3] https://metrics.torproject.org/uncharted-data-flow.html [4] https://metrics.torproject.org/bubbles.html -----BEGIN PGP SIGNATURE----- Comment: GPGTools - http://gpgtools.org iQEcBAEBAgAGBQJWVdmJAAoJEJD5dJfVqbCrmlEH/j6IjRNEkXzRVJBIBcFMKIwR eAA958Zg+DCzKPHI6Y7KZ/jGCHMP21r+YGIFevbJV4LDos9D2G0RP681/5PK7/dW if4Pz0xhl/LqbhLoOSqU2wGJG+GdWgjuTxO1TBMPUhYK71lwJsZ0cUzbNee7iAqJ zjdl3W3o4XJma0ZjZFH/gVVPbemvQHftrO8d0v7L2gFHfXmJg0kEwSZ4lJW0hfOx wIizEadFSx9/7CrINvbcHbyck0N+DRSfRYyMNSjpyFbnYI7HjYk7/+jvLWALEYZT 0MZbuL/zl/PFpkTDNY0jzE5fPaKjtS2pEoZn85Wn1+0kCjeFLE/Hvulvn6ZFxsA= =n5p1 -----END PGP SIGNATURE-----

6 5

Have the Tor daemon use more cores
by Jeremy 01 Dec '15

01 Dec '15

I've been working on the volunteer project described here https://www.torproject.org/getinvolved/volunteer.html.en#useMoreCores but can't spend much more time on it. Right now, I have refactored circuit_receive_relay_cell() in relay.c (which calls relay_crypt() and eventually the AES crypt routines) to use the workqueue.c infrastructure similar to cpuworker.c. When the refactored code runs in single threaded mode, all seems good. Once I activate the thread pool and start sending it work with threadpool_queue_work(), it Bootstraps 100% okay and runs for several minutes before crashing on cells it doesn't handle properly. I'm happy to share my code with anyone interested. Thanks.

3 2

BridgeDB 0.3.4 is released
by isis 01 Dec '15

01 Dec '15

Hey, I've just tagged and released BridgeDB 0.3.4. For those interested, here are the changes (c.f. https://gitweb.torproject.org/bridgedb.git/tree/CHANGELOG) Changes in version 0.3.4 - 0.3.5 - 2015-11-30 * FIXES #14685 https://bugs.torproject.org/14685 This disables distribution of obfs2 bridges. This pluggable transport has known distiguishers which allow adversaries to identify client connections to obfs2 bridges, which in turn allows these connections to be blocked/censored. With numerous obfs3 and obfs4 bridges both readily available, users should not be presented with an easily-configurable choice that is known to be unsafe for the majority of users. And includes the following general changes: * ADDS error pages to BridgeDB's web interface, to provide friendlier explanations for downtime, missing pages, and internal server errors. For example: https://bridges.torproject.org/404 -- ♥Ⓐ isis agora lovecruft _________________________________________________________ OpenPGP: 4096R/0A6A58A14B5946ABDE18E207A3ADB67A2CDB8B35 Current Keys: https://blog.patternsinthevoid.net/isis.txt

1 0

[Applications Team] Meeting next Tuesday 19:00 UTC
by Georg Koppen 01 Dec '15

01 Dec '15

Hi, just a reminder that the applications team is having its monthly meeting next week on Tuesday, December 1 at 19:00 UTC in #tor-project on irc.oftc.net. There is no proposed topic for this meeting yet. Please come up with suggestions if you feel there is something we should discuss. See you there, Georg

2 1

BridgeDB 0.3.3 is released
by isis 01 Dec '15

01 Dec '15

Hello! I released BridgeDB 0.3.3 a few weeks ago, and deployed it on the production server. However, I completely forgot to email the list to notify you all of the changes. Oops, sorry! For those who are curious, BridgeDB-0.3.3 brought about the following changes (and, as always, the current changelog is available at https://gitweb.torproject.org/bridgedb.git/tree/CHANGELOG) Changes in version 0.3.3 - 2015-10-25 * FIXES #12029 https://bugs.torproject.org/12029 BridgeDB now has an API for creating Bridge Distributors. See the bridgedb.distribute module, or its developer documentation at https://pythonhosted.org/bridgedb/bridgedb.distribute.html. * FIXES PART OF #12506 https://bugs.torproject.org/12506 BridgeDB's two Distributors (HTTPS and Email) are now entirely modularised and self-contained within separate subdirectories in the source code. This is the first step to redesigning these Distributors into their own separate processes, which will allow the Distributors to remain functional while BridgeDB is reparsing bridge descriptors. * FIXES #15968 https://bugs.torproject.org/15968 BridgeDB now sends a Content-Security-Policy header which explicitly allows Javascript, images, CSS, and fonts, from https://bridges.torproject.org. All other types of content are forbidden, including: - embedding https://bridges.torproject.org within <iframe>, <embed>, or <object>, and attempting to source additional resources into its embedded context - inline Javascript, including Javascript within SVG files - inline CSS - externally hosted fonts - inline SVG, e.g. via the HTML5 <svg> tag - any and all connections made via Javascript XMLHttpRequests, WebSockets, sendBeacon(), and Web Workers - plugins - applets BridgeDB's Content-Security-Policy does not yet make use of certain newer, lesser supported, Content-Security-Policy v2.0 directives, such as "reflected-xss" and "frame-ancestors", but may someday. * FIXES #16273 https://bugs.torproject.org/16273 Several links to Tor Project gitweb URLs within the developer documentation were outdated in that they still used the old gitweb URL format. These are now updated. Thanks to David Fifield for the bug report and patches. * FIXES #16330 https://bugs.torproject.org/16330 BridgeDB can now handle bridge-server-descriptors with extra-info-digest fields which have two values, as well as both bridge-server-descriptors and bridge-extrainfo descriptors which contain Ed25519 key material and signatures. See Tor proposals #220 and #228 for more information on the changes to these descriptors. Note that BridgeDB can now parse this information, but does not yet make use of any Ed25519 cryptographic material within bridge descriptors. https://gitweb.torproject.org/torspec.git/tree/proposals/220-ecc-id-keys.txt https://gitweb.torproject.org/torspec.git/tree/proposals/228-cross-certific… Thanks to Atagar for patching Stem. * FIXES #16616 https://bugs.torproject.org/16616 The HSDir flag can now be included within bridge-networkstatus documents. BridgeDB now has unittests which guarantee that its parsers safely ignore this flag, as well as any flags unknown to BridgeDB which may appear in the future. Thanks to Roger Dingledine for alerting me about the change. * FIXES #16649 https://bugs.torproject.org/16649 Mobile users, and other users with small screen pixel ratios, will find that the UI of BridgeDB's HTTPS Distributor has greatly increased in usability and readability. And includes the following general changes: * FIXES an error when requesting the non-HTML version of the bridges page (e.g. https://bridges.torproject.org/bridges?format=plain) * REMOVES the `bridgedb test` commandline option. BridgeDB's tests can be run via `python setup.py test` or `make test` (or `make coverage` for generating HTML test coverage statistics). * CHANGES the HTTPS Distributor to HTML-encode Bridge Lines. Previously, a malicious Pluggable Transport Bridge could include in its PT arguments something like "evil=<script>[…]</script>" and if such a Bridge were to be distributed to a user, that user's web browser would execute the script (if Javacript was enabled). Other characters, including non-ASCII, control characters, double quotes, and backslashes, are also sanitised from Bridge Lines. Thanks to Robert Ransom for the patches. * CHANGES BridgeDB's module/package version numbers to be compliant with PEP440. * CHANGES the layout of BridgeDB's source code directories. Rather than storing BridgeDB's source in "lib/bridgedb/", it is now kept in "bridgedb/". Similarly, the directory containing BridgeDB's tests has been moved from "lib/bridgedb/test/" to "test/", which means that the tests are no longer installed when running `python setup.py install` or `make install`. * ADDS several improvements to the developer documentation at https://pythonhosted.org/bridgedb. * UPDATE English (en_US) translations. * UPDATE English (en) translations. * ADD Serbian (sr) translations. Thanks to obj.petit.a, Ivan Radeljic, and Milenko Doder. * UPDATE Arabic (ar) translations. Thanks to A. Hassan, debo debo, KACIMI LAMINE, and Nudroid A. * UPDATE Catalan (ca) translations. Thanks to laia_. * UPDATE Czech (cs) translations. Thanks to Tomas Palik and Vlastimil Burián. * UPDATE Danish (da) translations. Thanks to Mogelbjerg. * UPDATE German (de) translations. Thanks to jschfr, Junge Limba, and Toralf Förster. * UPDATE English (en_GB) translations. Thanks to Andi Chandler. * UPDATE Farsi (fa) translations. Thanks to some awesome anonymous person for helping out. * UPDATE Finish (fi) translations. Thanks to Riku Viitanen. * UPDATE French (fr) translations. Thanks to elouann, Trans-fr, and Towinet. * UPDATE French (fr_CA) translations. Thanks to Trans-fr. * UPDATE Croatian (hr_HR) translations. Thanks to some awesome anonymous person for helping out. * UPDATE Hungarian (hu) translations. Thanks to some awesome anonymous person for helping out. * UPDATE Indonesian (id) translations. Thanks to Anthony Santana, Astryd Viandila Dahlan, cholif yulian, constantius damar wicaksono, Dwi Cahyono, L1Nus, km242saya, and Zamani Karmana. * UPDATE Italian (it) translations. Thanks to Random_R. * UPDATE Japanese (ja) translations. Thanks to ABE Tsunehiko. * UPDATE Latvian (lv) translations. Thanks to Ojārs Balcers. * UPDATE Norwegian Bokmål (nb) translations. Thanks to Erik Matson and Kristian Andre Henriksen. * UPDATE Dutch (nl) translations. Thanks to Mart3000. * UPDATE Polish (pl) translations. Thanks to Karol Obartuch. * UPDATE Portuguese (pt) translations. Thanks to Bruno D. Rodrigues and MMSRS. * UPDATE Brazillian Portuguese (pt_BR) translations. Thanks to Communia. * UPDATE Romanian (ro) translations. Thanks to Ana, axel_89, and Di N. * UPDATE Russian (ru) translations. Thanks to Ivan. * UPDATE Slovak (sk_SK) translations. Thanks to StefanH. * UPDATE Albanian (sq) translations. Thanks to some awesome unknown anonymous person who didn't add their name to the list of translators. * UPDATE Swedish (sv) translations. Thanks to Peter Michanek. * UPDATE Turkish (tr) translations. Thanks to Bullgeschichte and Fomas. * UPDATE Ukranian (uk) translations. Thanks to Yasha. * UPDATE Chinese Mandarin (zh_CN) translations. Thanks to khi. * UPDATE Taiwanese Mandarin (zh_TW) translations. Thanks to x4r. -- ♥Ⓐ isis agora lovecruft _________________________________________________________ OpenPGP: 4096R/0A6A58A14B5946ABDE18E207A3ADB67A2CDB8B35 Current Keys: https://blog.patternsinthevoid.net/isis.txt

1 0