- tor-dev - lists.torproject.org

tor 0.2.7.5 make error
by grarpamp 08 Dec '15

08 Dec '15

"Makefile", line <nums>: Need an operator make: fatal errors encountered -- cannot continue 6772:export TESTING_TOR_BINARY=$(top_builddir)/src/or/tor 6794:export PYTHON= 6795:export SHELL=/bin/sh 6796:export abs_top_srcdir=/tmp/tor-0.2.7.5 6797:export builddir=. This is on a FreeBSD 8.x from early 2014. The autotools is probably too old there. Haven't got around to supplying a better fix than commenting them out of .in and not using autoreconf -fiv. 8.x is EOL anyways.

2 2

Tor Messenger 0.1.0b4 is released
by Sukhbir Singh 08 Dec '15

08 Dec '15

We are pleased to announce another public beta release of Tor Messenger. This release addresses a number of stability and usability issues, and includes the default bridge configurations for pluggable transports. The initial public release [0] was a success in that it garnered a lot of useful feedback. We tried to respond to all your concerns in the comments of the blog post [1] but also collected and aggregated a FAQ of the most common questions [2]. * Before Upgrading Before upgrading to the new release, you will need to backup your OTR keys or simply generate new ones. Please see the following steps to back them up [3]. In our eagerness to build on work done by Tor Browser, we made the decision to store your profile directory inside the application bundle. This complicates matters when you want to use the same accounts and keys across updates, especially while we don't have an automatic updater [4]. Also, as was vociferously pointed out by some of our early adopters, this probably isn't a very intuitive user experience. Copying the extracted application to someone else's computer would unknowingly transfer your accounts and OTR keys. It's unclear if this is commonly done and we'd love feedback on this point to understand the urgency of the issue. In future releases, we plan on revisiting this decision. The number one item on our roadmap is porting Tor Browser's updater patches [5] so that keeping Tor Messenger up-to-date is seamless and automatic. We also plan to add a UI to make importing OTR keys and accounts from Pidgin, and other clients, as easy as possible [6]. * Downloads Please note that Tor Messenger is _still in beta_. The purpose of this release is to help test the application and provide feedback. *At-risk users should not depend on it for their privacy and safety*. Linux (32-bit) https://dist.torproject.org/tormessenger/0.1.0b4/tor-messenger-linux32-0.1.… Linux (64-bit) https://dist.torproject.org/tormessenger/0.1.0b4/tor-messenger-linux64-0.1.… Windows https://dist.torproject.org/tormessenger/0.1.0b4/tormessenger-install-0.1.0… OS X (Mac) https://dist.torproject.org/tormessenger/0.1.0b4/TorMessenger-0.1.0b4-osx64… sha256sums.txt https://dist.torproject.org/tormessenger/0.1.0b4/sha256sums.txt sha256sums.txt.asc https://dist.torproject.org/tormessenger/0.1.0b4/sha256sums.txt.asc The sha256sums.txt file containing hashes of the bundles is signed with the key 0x6887935AB297B391 (fingerprint: 3A0B 3D84 3708 9613 6B84 5E82 6887 935A B297 B391). * Changelog Here is the complete changelog since v0.1.0b2: Tor Messenger 0.1.0b4 -- November 22 2015 All Platforms Bug 17492: Include default bridges configuration Use tor and the pluggable transports from tor-browser 5.0.4 Bug 17552: Instantbird should handle XMPP message stanzas with subjects ctypes-otr Bug 17539: Pass username when interpolating resent string Bug 15179: Add an OTR Preferences item to the Tools menu Use the FIREFOX_42_0_RELEASE tag on mozilla-release Use the THUNDERBIRD_42_0b2_RELEASE tag on comm-release Bug 16489: Prevent automatic logins at startup Update Tor Messenger logo in Tor Launcher Mac Bug 16476: Themes preference is positioned incorrectly Bug 17456: Application hang when navigating the preferences menu Tor Messenger 0.1.0b3 -- October 30 2015 Windows Bug 17453: Fix Tor Messenger crash when starting up in Windows [0] - https://blog.torproject.org/blog/tor-messenger-beta-chat-over-tor-easily/ [1] - https://blog.torproject.org/blog/tor-messenger-beta-chat-over-tor-easily/#c… [2] - https://trac.torproject.org/projects/tor/wiki/doc/TorMessenger/FAQ [3] - https://trac.torproject.org/projects/tor/wiki/doc/TorMessenger/FAQ#Whereare… [4] - https://bugs.torproject.org/13861 [5] - https://bugs.torproject.org/14388 [6] - https://bugs.torproject.org/16526 -- Sukhbir

1 0

Shared random value calculation edge cases (proposal 250)
by George Kadianakis 07 Dec '15

07 Dec '15

Hello there believers of prop250, you can find the latest version of the proposal in the upstream torpec repo: https://gitweb.torproject.org/torspec.git/tree/proposals/250-commit-reveal-… Implementation is also constantly moving forward as you can see in the ticket: https://trac.torproject.org/projects/tor/ticket/16943 Now that we have ironed out the whole voting procedure, it's time to finish this up by figuring out the last details of the shared random value calculation. The logic in the current proposal seems reasonable (see section 3.3), but I have some doubts that I wanted to communicate with you. - I'm afraid of edge cases where different authorities will calculate different shared random values. This is bad because if it happens at the wrong moment, it might break the consensus. For example, imagine that there are 5 authorities that are doing the prop250 protocol. Since there are 5 auths, they can create a valid consensus on their own with SR info in it. Now imagine that one of them, Alice, has a different view of the previous_SRV than the others (maybe because she _just_ booted up before voting and she doesn't have a previous consensus). In this case, Alice will calculate a different SRV than the other 4, and hence the consensus will break because 5 signatures could not be collected. Is this something likely to happen? If yes, a way to protect against it is to add a flag on the votes denoting support for the SR protocol, and have dirauths toggle it off if they are voting for 00:00UTC and they don't know the previous_SRV or if they don't have a consensus. This can also be used as a torrc-enabled killswitch, if the SR protocol bugs out completely and we need to disable it for the sake of the network. What do you think? - Another bothersome thing is the disaster SRV calculation. Specifically, the proposal says: =============================== prop 250: ==================================== If the consensus at 00:00UTC fails to be created, then there will be no fresh shared random value for the day. In this case, and assuming there is a previous shared random value, directory authorities should use the following construction as the shared random value of the day: SRV = HMAC(previous_SRV, "shared-random-disaster") where "previous_SRV" is the previous shared random value. =============================================================================== this logic is not implemented in the current code, and it's not straightforward to implement. Again because the previous_SRV is blended in the formula, but also because it's not easy to know that "the consensus at 00:00UTC fails to be created". For example, if you are a dirauth that just started up at 00:30UTC, and you asked for the previous consensus and you were given the 23:00UTC consensus, then you won't know that the 00:00 consensus was not created and that you need to do the disaster procedure. This will again break the consensus. Not sure if this is a likely scenario as well, and if we should protect against it. What do you think? It depends on the logic that authorities have for fetching consensuses. Do they ensure that they always have the latest consensus? Do we need to add such logic as part of prop250? :/ A way to protect against it is to use the "SR support" vote flag I talked about before, and toggle it off if you are a dirauth and you don't have the latest consensus. Terrible? Would that even allow dirauths to bootstrap SR? - Another interesting part of prop250 is: =============================== prop 250: ==================================== If the shared random value contains reveal contributions by less than 3 directory authorities, it MUST NOT be created. Instead, the old shared random value should be used as specified in section [SRDISASTER]. =============================================================================== do you think this is useful? The fact that we use consensus methods, ensures us that at least 5 dirauths understand the SR protocol, otherwise we don't do it. Should we care about the number of reveal values? And why should the constant be 3, and not 5 or 2? Those were my doubts. Sorry for the extra confusion, but I'm currently reading the proposal and trying to minimize the amount of edge cases that can happen in the implementation (especially if they result in breaking the consensus for everyone). Maybe I'm trying too hard, and there are already multiple such edge cases in the consensus protocol that just never happen and are not worth fixing. In any case, the shared random value calculation seems to be the last piece of the puzzle here, so let's figure it out and finish up! Thanks!

6 16

Graphs - Estimated Traffic Capacity
by David Goulet 07 Dec '15

07 Dec '15

Hello everyone! I would like to share this graph I've been working on along side with Aaron Johnson (thanks to him for the algorithm I'll be describing). This shows both the maximum additional and total traffic the network can sustain for both HS and Exit. The "additional" traffic here means that we consider the current load on the network thus the additional value is what can be added more on top of the current traffic. http://ygzf7uqcusp4ayjs.onion/tor-health/tor-health/bandwidth_capacity.html (Please ignore the initial downward spike, technical issue) Here is the algorithm we are using for this. It's in no ways an accurate value but rather an estimation as you will see with the following algorithm to compute the traffic capacity you see. 1. Using the latest consensus, build a list of relays that are Running, Valid and Fast (weight selection only picks relays with Fast flag). Then compute their positionnal consensus weights. 2. For each of them, set their total capacity to min(avg BW, observed BW). 3. For each of them, compute the unused capacity that is use the extrainfo descriptor read/write bytes history for the amount of traffic used and substract that to the total capacity. max(mean(write_history_values) / write_interval, mean(read_history_values) / read_interval) 4. For HS traffic, that is a 6-hops circuit, we go over all relays and find the relay that has the lowest unused capacity divided by it's probability of being picked in the Guard/Middle position: picked_prob = (guard_weight * 2) + (middle_weight * 4) unused = relay.unused_capacity / picked_prob We use time 2 here since there are two guards in the 6 hops circuit and 4 middle nodes. We get the minimum value between each relays. 5. The minimum "unused" value times the picked_prob is the amount of bytes that we'll give to all relays thus reducing their unused capacity: relay.unused_capacity -= (min_unused * picked_prob) At each iteration of the algorithm, this will make at least one relay go to 0 unused capacity so when this happens, remove that relay from the set of relays to be picked. Once we can't pick a relay anymore for the 6 hops circuit (that is no more guards or middle), we stop. 6. Add min_unused value to the total number of bytes. Goto step 4. Ok... this can maybe be not that trivial to understand at first, feel free to ask questions but the basic idea is that at each iteration of the algorithm you spread a specific amount of bytes to each relay depending on the probability of being picked until you have no more relays capacity for such a circuit. For the "Max Total" traffic, the relay.unused_capacity is equal to the relay capacity. For Exit traffic, we do the same logic but with a 3 hops circuit where each position is multiplied by 1 since there is one guard, middle and exit (in normal circumstances). I know that we have sometimes 1, 4, 5, 6 or 7 hops circuit but the general case is considered here and we have no stats on how frequent those unusual circuits are. Anyway, if you think this algorithm could be improved, please respond. If you think this algorithm is wrong, please respond. If you can reproduce the result on your own with this algo, omg please respond! :) The above could be totally wrong but hopefully we did a fairly good job. Please remember, this is an estimate. :) Thanks! David

5 7

metrics-lib 1.0.0 is released
by Karsten Loesing 05 Dec '15

05 Dec '15

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hello devs, I just released metrics-lib 1.0.0: https://dist.torproject.org/descriptor/1.0.0/ metrics-lib, which also goes by the name DescripTor, is a Java API that fetches Tor descriptors from a variety of sources like cached descriptors and directory authorities/mirrors. The DescripTor API is useful to support statistical analysis of the Tor network data and for building services and applications. # Changes in version 1.0.0 - 2015-12-05 * Major changes - This is the initial release after four years of development. Happy 4th birthday! Special thanks to iwakeh for helping with metrics-lib development over the past year including pushing for finally putting out this release! Happy descriptor parsing! All the best, Karsten -----BEGIN PGP SIGNATURE----- Comment: GPGTools - http://gpgtools.org iQEcBAEBAgAGBQJWYqQ2AAoJEJD5dJfVqbCrHjUH/2Yavypi6BK6F1R3bAyMfCUd t140r4FlH1FBOuWi9qt5p5fbkGroef7ttMHZz3cXqjUJWyfOC3TynB7g152hIflQ E6JeilxeCY62RiOpJ2nRF0KBAKmL1QyVinvE3USORXyT+RvPAhtVpNPU6A8AQlxn lUWsjeD7HNMn9s1VXy3O3m0fs3CGxflbwfbbS8kTP6OINu7nP5f3ldcAm2ykDB0R mG1BgkvGoamhuQmimWVv8S5c/N+pOPLN9La75kSC4BppgBAiD5MCP6zJHhkvMmAG h9ciWsZdAxuKjEdLVCIYRXkNIGAMLikr7HpYB5ewBu8tcGkBHQaCXweNA5sI0Po= =jY3h -----END PGP SIGNATURE-----

1 0

Have the Tor daemon use more cores
by Jeremy 03 Dec '15

03 Dec '15

On 12/02/2015 07:00 AM, tor-dev-request(a)lists.torproject.org wrote: >> >I've been working on the volunteer project described here >> >https://www.torproject.org/getinvolved/volunteer.html.en#useMoreCores >> >but can't spend much more time on it. >> > >> >Right now, I have refactored circuit_receive_relay_cell() in relay.c >> >(which calls relay_crypt() and eventually the AES crypt routines) to >> >use the workqueue.c infrastructure similar to cpuworker.c. >> > >> >When the refactored code runs in single threaded mode, all seems >> >good. Once I activate the thread pool and start sending it work with >> >threadpool_queue_work(), it Bootstraps 100% okay and runs for several >> >minutes before crashing on cells it doesn't handle properly. >> > >> >I'm happy to share my code with anyone interested. > Sure I think I'm the person that used the workqueue stuff last, so I'd > be up for doing the rest of the work (nb: I am traveling/in meetings for > the rest of the week, so it'll be next week at the earliest that I can > really spend quality time). > > Regards, > > -- Yawning Angel Yawning, or anyone else: If you assess the code has potential, I can assist & work on it more. If it is a big miss, then I might start with a smaller contribution. Thanks, Jeremy

1 0

txtorcon 0.14.2
by meejah 03 Dec '15

03 Dec '15

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 With Twisted 15.5.0, a deprecated endpoint interface class is still being used. This release backports the fix for that to the 0.14.x branch. You can download the release from PyPI or GitHub (or of course "pip install txtorcon"): https://pypi.python.org/pypi/txtorcon/0.14.2 https://github.com/meejah/txtorcon/releases/tag/v0.14.2 Releases are also available from the hidden service: http://timaq4ygg2iegci7.onion/txtorcon-0.14.2.tar.gz http://timaq4ygg2iegci7.onion/txtorcon-0.14.2.tar.gz.asc http://timaq4ygg2iegci7.onion/txtorcon-0.14.2-py2-none-any.whl http://timaq4ygg2iegci7.onion/txtorcon-0.14.2-py2-none-any.whl.asc You can verify the sha256sum of both by running the following 4 lines in a shell wherever you have the files downloaded: cat <<EOF | sha256sum --check fbc95c41e924b0e10156c46227eac2b4acf42a3b8d01f0ea3022897a10dc059a txtorcon-0.14.2-py2-none-any.whl f99819b1a71b8dea9e80317ec83c990b4ff608c98bc78a9fc1dc9991d349d13f txtorcon-0.14.2.tar.gz EOF thanks, meejah -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.12 (GNU/Linux) iQEcBAEBAgAGBQJWX3WRAAoJEMJgKAMSgGmnOasH/i3NuaGRXu/wgdSHwDldPCpt FCVFXN9RT8nTV8EN72V1iwqW0ir38YG8rM9TVNbImZlPFbMLaV0zJsb+rNId5xgb uNBnlzrzdJOtlgofg4aKHBHjvDioV4xrZiypzOOc5u2Hyd8xuOPkI3fEVx+QfaKA Cjbys6lCM2HZpeWkHlPKM3xnN9BoJ7IZodbv54fEehR5MgljkgwCoQJOdDy8tf9L FQmpDXb7ptNXqbHNUltg8zkyIt345u6XcvZWX8wCxVu4PHCucdUW0VWzq33dNCuA jbGvsesrJsr6+2a6wMq0FXJByaU1ymkwdp0OV55ASdNg5KBcfh4Sf4gMNiYGQw8= =bBHR -----END PGP SIGNATURE-----

1 0

Have the Tor daemon use more cores
by Jeremy 03 Dec '15

03 Dec '15

>> I've been working on the volunteer project described here >> >https://www.torproject.org/getinvolved/volunteer.html.en#useMoreCores but >> >can't spend much more time on it. >> > >> >Right now, I have refactored circuit_receive_relay_cell() in relay.c (which >> >calls relay_crypt() and eventually the AES crypt routines) to use the >> >workqueue.c infrastructure similar to cpuworker.c. >> > >> >When the refactored code runs in single threaded mode, all seems good. Once >> >I activate the thread pool and start sending it work with >> >threadpool_queue_work(), it Bootstraps 100% okay and runs for several >> >minutes before crashing on cells it doesn't handle properly. >> > >> >I'm happy to share my code with anyone interested. > Hi Jeremy! > > This is great! Thanks for this. You should definitely post your code (link to > branch or .diff) to ticket #7572 [1] so someone at some point can take a look > at it and help you out. > > Thanks! > David > > [1]https://trac.torproject.org/projects/tor/ticket/7572 > Thank you. I have posted my code and tried to explain my thought process on ticket #7572.

1 0

Scaling Tor Metrics
by Karsten Loesing 01 Dec '15

01 Dec '15

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hello devs, the Tor Metrics website [0] claims to be "the primary place to learn interesting facts about the Tor network" and invites its visitors who "come across something that is missing" to contact the website authors about it. That's a bold statement I put there! :) Yet, there's considerable product backlog with possible enhancements [1] that doesn't seem to ever become shorter. Even worse, it can be expected that the backlog will refill quickly once the community notices that feature requests are suddenly considered. The main reason for this unfortunate situation is that Tor Metrics contains many moving parts, including some heavy database lifting that takes place below the surface, that all want to be maintained. Adding more parts just makes the whole thing even more likely to break. At the same time, knowing about the situation that Tor Metrics has become almost closed to contributions is painful. This posting shall discuss possible solutions. The goal is to let Tor Metrics grow in a healthy fashion that encourages contributions from the community. These solutions are not mutually exclusive, and the best solution may use parts of more than one solution sketched out here. 1 Make Tor Metrics better and bigger, internally The obvious solution is that the maintainers of Tor Metrics could just work harder to overcome the problems stated above. Let's think this through. 1.1 Add more development resources If only the current Tor Metrics maintainers had more time to devote to cleaning up existing parts and to add new parts, that would solve our problem. They could refactor parts that are hard to maintain, and they could work off the serious backlog that has piled up. Of course, this means dropping or handing over responsibilities for other products, and it may mean finding (and paying) new developers to help maintain Tor Metrics. It's unclear whether anything like this would fit into Tor's budget, and whether these changed priorities would make users of tools that had to be dropped or handed over unhappy. 1.2 Rewrite internal parts of Tor Metrics to encourage external contributions Most of Tor Metrics would have run 10 or 15 years ago with only minor modifications. It's not necessarily a bad thing to use established technologies. But maybe, if we rewrite it using modern data-processing, web, and visualization frameworks, it becomes more attractive to other developers to contribute code and help maintain existing (well, then rewritten) code. The result would be a larger Tor Metrics website that is easier to maintain and hopefully maintained by more people. It's unclear how realistic this plan is, though, and it requires attention by Tor Metrics maintainers to bring it enough into shape for external contributors to get involved. 2 Add more ways to contribute to Tor Metrics externally It may be possible to further grow Tor Metrics without adding more code to it, hence not making it any harder to maintain. However, if code to generate visualizations is run elsewhere, there's a certain risk that results are not perceived as trustworthy as if that code were run as part of Metrics. This is primarily a problem of setting user expectations right. We could add different ways for contributing to Tor Metrics, depending on the level of commitment that contributors are willing to make. Possible new ways (in addition to filing a Trac ticket, which is already possible, though not very effective) are: 2.1 Accept contribution of static data or static graphs Somebody might contribute data (in a tarball, download link, etc.) or a static graph (static as in "doesn't break, ever", not "static HTML with a tiny amount of JavaScript that will surely never break"). The Tor Metrics team reviews that and puts it on the Tor Metrics website, together with a short description, author information, license, etc. There are plenty of visualizations on Trac and on the mailing lists, so we'll have to define criteria what we add and what not, and we'll need a good process for making that happen. 2.2 Link to external websites Somebody might write a website that visualizes Tor network data. The Tor Metrics team reviews the idea behind it, but not necessarily look at its code, and adds an external link to Tor Metrics. It becomes obvious that the authors remain responsible for their visualization, so there's no risk involved for Tor Metrics, but users may not trust it as much, because it doesn't have the Tor Metrics label. Note that we're already doing this approach by linking to the visualizations showing "Tor users as percentage of larger Internet population" [2] and "Data flow in the Tor network" [3]. Also note that we could as well have hosted the former directly on Tor Metrics with appropriate attribution, because it's a static image. This is not the case with the latter. 2.3 Run an externally developed website as if it were part of Tor Metrics Let's imagine that somebody produces a visualization of Tor network data and would like to make it part of Tor Metrics but without limiting themselves to the technology used by Tor Metrics. We could let them write their visualization as website and integrate it into Tor Metrics after reviewing its code. Technically, part of this integration would be to "redress" the website by applying the Tor Metrics design (which has lots of room for improvement, but let's just say the result will look as seamlessly integrated into Tor Metrics as the "Network bubble graphs" [4]). Another part would probably be to rewrite web requests, so that users still think they're talking to https://metrics.torproject.org/, but really they're talking to another webserver behind that. Regarding hosting and maintenance, in theory, the website could be hosted by the original creators, but that effectively means that the Tor Metrics team gives up part of the control about what's on the Tor Metrics website. The creators of the external website could change parts or add new parts that wouldn't be reviewed by Tor Metrics developers, but they would be perceived as part of Metrics, which seems bad. The Tor Metrics team could run the externally developed website on a separate host or on the same host as Tor Metrics. We could imagine variants where the original creator stays around to fix any issues as they come up, or we could imagine that they donate their visualization that the Tor Metrics people will then maintain. We could even imagine that the Tor Metrics maintainers some day decide to integrate the originally external website into Tor Metrics proper, but that would not be required for this model to work. All these ideas require writing down guidelines, criteria, and processes. In particular, they require more thoughts and input from other people who are not currently involved in Tor Metrics maintenance and who can be expected more objective. And once these ideas are implemented, we'll need more Tor Metrics maintainer than just one. What are your thoughts? All the best, Karsten [0] https://metrics.torproject.org/ [1] https://trac.torproject.org/projects/tor/query?status=!closed&component=Met… [2] https://metrics.torproject.org/oxford-anonymous-internet.html [3] https://metrics.torproject.org/uncharted-data-flow.html [4] https://metrics.torproject.org/bubbles.html -----BEGIN PGP SIGNATURE----- Comment: GPGTools - http://gpgtools.org iQEcBAEBAgAGBQJWVdmJAAoJEJD5dJfVqbCrmlEH/j6IjRNEkXzRVJBIBcFMKIwR eAA958Zg+DCzKPHI6Y7KZ/jGCHMP21r+YGIFevbJV4LDos9D2G0RP681/5PK7/dW if4Pz0xhl/LqbhLoOSqU2wGJG+GdWgjuTxO1TBMPUhYK71lwJsZ0cUzbNee7iAqJ zjdl3W3o4XJma0ZjZFH/gVVPbemvQHftrO8d0v7L2gFHfXmJg0kEwSZ4lJW0hfOx wIizEadFSx9/7CrINvbcHbyck0N+DRSfRYyMNSjpyFbnYI7HjYk7/+jvLWALEYZT 0MZbuL/zl/PFpkTDNY0jzE5fPaKjtS2pEoZn85Wn1+0kCjeFLE/Hvulvn6ZFxsA= =n5p1 -----END PGP SIGNATURE-----

6 5

Have the Tor daemon use more cores
by Jeremy 01 Dec '15

01 Dec '15

I've been working on the volunteer project described here https://www.torproject.org/getinvolved/volunteer.html.en#useMoreCores but can't spend much more time on it. Right now, I have refactored circuit_receive_relay_cell() in relay.c (which calls relay_crypt() and eventually the AES crypt routines) to use the workqueue.c infrastructure similar to cpuworker.c. When the refactored code runs in single threaded mode, all seems good. Once I activate the thread pool and start sending it work with threadpool_queue_work(), it Bootstraps 100% okay and runs for several minutes before crashing on cells it doesn't handle properly. I'm happy to share my code with anyone interested. Thanks.

3 2