December 2015 - tor-dev - lists.torproject.org

Weekly Tor developers meetings; IRC time change!
by Nick Mathewson 10 Dec '15

10 Dec '15

Hi, all! (This is about the IRC meeting on OFTC on #tor-dev to talk about developing the program called "tor".) We have have fewer people at the new Wednesday time than we did at the old one. That's not great. The Monday/Tuesday time is convenient for a bunch of folks, though, so that's nice. So for next week at least, we will move Wednesday earlier to be closer to the old time, and keep Monday/Tuesday as it is. That makes the meeting times: TUESDAY 0100 UTC (== 8 pm eastern on Monday. No change.) WEDNESDAY 1330 UTC (==0830 am eastern on Wednesday. Changed.) There's no need to show up to both, but you can if that's convenient. If both are equally convenient for you and you only want to do one, please favor the Wednesday one. One of the topics at the next meetings will be Good Meeting Times and Getting More People to Come. If you want to talk about meeting times, but neither of these times will work, just send me a note or find me online so I know. Let's keep trying till we get this right! cheers, -- Nick

1 0

Hidden service patch workshop on #tor-dev IRC channel (17/12/2015)
by George Kadianakis 10 Dec '15

10 Dec '15

Dear tor-dev, Everyone who is working on any kind of improvement to hidden services or related applications and would like to give or receive assistance (code reviews, advice, etc.) is welcome on: Thursday December 17, 16:00 UTC @ #tor-dev OFTC channel This might become a regular biweekly thing, if it proves to be successful. Cheers!

1 0

tor control port filtration with a series of tubes
by David Stainton 09 Dec '15

09 Dec '15

or-ctl-sieve - granular bidirectional control port filtration https://github.com/david415/or-ctl-sieve or-ctl-sieve does what it says on the tin and comes with a handy config file that we use with arm (tor-arm): https://github.com/david415/or-ctl-sieve/blob/master/tor-arm-filter.json I wrote this in Tubes... the new async message passing API for twisted... and this caused me to draw the this ASCII diagram of the flow graph: https://github.com/david415/or-ctl-sieve/blob/master/or_ctl_sieve/sieve.py#… cheers, david

1 0

Better relay uptime visualisation
by Philipp Winter 08 Dec '15

08 Dec '15

I spent some time improving the existing relay uptime visualisation [0]. Inspired by a research paper [1], the new algorithm uses single-linkage clustering with Pearson's correlation coefficient as distance function. The idea is that relays are grouped next to each other if their uptime (basically a binary sequence) is highly correlated. Check out the following gallery. It contains monthly relay uptime images, dating back to 2007: <https://nymity.ch/sybilhunting/uptime-visualisation/> If you aren't familiar with this type of visualisation: Every image shows the uptime of all Tor relays that were online in a given month. Every row is a consensus and every column is a relay. White pixels mean that a relay was offline and black pixels means that a relay was online. Red pixels are used to highlight suspiciously similar clusters. A nice example is the Heartbleed incident: <https://nymity.ch/sybilhunting/uptime-visualisation/slide_2014-04.html> The huge red block on the left shows all the relays that were removed by the directory authorities because they didn't rotate their key pairs in time. The downside of single-linkage clustering is that it takes longer to compute. On my laptop, I can create an image covering one month in under three minutes, so it's tolerable. Another practical problem is that it's cumbersome to learn the relay fingerprint of a given column. I'm looking into JavaScript/HTML tricks that can show text when you hover over a region in the image. Perhaps somebody knows more? [0] <https://bugs.torproject.org/12813> [1] <http://nms.csail.mit.edu/papers/clustering-imw2002.pdf>, Section 2 Cheers, Philipp

6 13

tor 0.2.7.5 make error
by grarpamp 08 Dec '15

08 Dec '15

"Makefile", line <nums>: Need an operator make: fatal errors encountered -- cannot continue 6772:export TESTING_TOR_BINARY=$(top_builddir)/src/or/tor 6794:export PYTHON= 6795:export SHELL=/bin/sh 6796:export abs_top_srcdir=/tmp/tor-0.2.7.5 6797:export builddir=. This is on a FreeBSD 8.x from early 2014. The autotools is probably too old there. Haven't got around to supplying a better fix than commenting them out of .in and not using autoreconf -fiv. 8.x is EOL anyways.

2 2

Tor Messenger 0.1.0b4 is released
by Sukhbir Singh 08 Dec '15

08 Dec '15

We are pleased to announce another public beta release of Tor Messenger. This release addresses a number of stability and usability issues, and includes the default bridge configurations for pluggable transports. The initial public release [0] was a success in that it garnered a lot of useful feedback. We tried to respond to all your concerns in the comments of the blog post [1] but also collected and aggregated a FAQ of the most common questions [2]. * Before Upgrading Before upgrading to the new release, you will need to backup your OTR keys or simply generate new ones. Please see the following steps to back them up [3]. In our eagerness to build on work done by Tor Browser, we made the decision to store your profile directory inside the application bundle. This complicates matters when you want to use the same accounts and keys across updates, especially while we don't have an automatic updater [4]. Also, as was vociferously pointed out by some of our early adopters, this probably isn't a very intuitive user experience. Copying the extracted application to someone else's computer would unknowingly transfer your accounts and OTR keys. It's unclear if this is commonly done and we'd love feedback on this point to understand the urgency of the issue. In future releases, we plan on revisiting this decision. The number one item on our roadmap is porting Tor Browser's updater patches [5] so that keeping Tor Messenger up-to-date is seamless and automatic. We also plan to add a UI to make importing OTR keys and accounts from Pidgin, and other clients, as easy as possible [6]. * Downloads Please note that Tor Messenger is _still in beta_. The purpose of this release is to help test the application and provide feedback. *At-risk users should not depend on it for their privacy and safety*. Linux (32-bit) https://dist.torproject.org/tormessenger/0.1.0b4/tor-messenger-linux32-0.1.… Linux (64-bit) https://dist.torproject.org/tormessenger/0.1.0b4/tor-messenger-linux64-0.1.… Windows https://dist.torproject.org/tormessenger/0.1.0b4/tormessenger-install-0.1.0… OS X (Mac) https://dist.torproject.org/tormessenger/0.1.0b4/TorMessenger-0.1.0b4-osx64… sha256sums.txt https://dist.torproject.org/tormessenger/0.1.0b4/sha256sums.txt sha256sums.txt.asc https://dist.torproject.org/tormessenger/0.1.0b4/sha256sums.txt.asc The sha256sums.txt file containing hashes of the bundles is signed with the key 0x6887935AB297B391 (fingerprint: 3A0B 3D84 3708 9613 6B84 5E82 6887 935A B297 B391). * Changelog Here is the complete changelog since v0.1.0b2: Tor Messenger 0.1.0b4 -- November 22 2015 All Platforms Bug 17492: Include default bridges configuration Use tor and the pluggable transports from tor-browser 5.0.4 Bug 17552: Instantbird should handle XMPP message stanzas with subjects ctypes-otr Bug 17539: Pass username when interpolating resent string Bug 15179: Add an OTR Preferences item to the Tools menu Use the FIREFOX_42_0_RELEASE tag on mozilla-release Use the THUNDERBIRD_42_0b2_RELEASE tag on comm-release Bug 16489: Prevent automatic logins at startup Update Tor Messenger logo in Tor Launcher Mac Bug 16476: Themes preference is positioned incorrectly Bug 17456: Application hang when navigating the preferences menu Tor Messenger 0.1.0b3 -- October 30 2015 Windows Bug 17453: Fix Tor Messenger crash when starting up in Windows [0] - https://blog.torproject.org/blog/tor-messenger-beta-chat-over-tor-easily/ [1] - https://blog.torproject.org/blog/tor-messenger-beta-chat-over-tor-easily/#c… [2] - https://trac.torproject.org/projects/tor/wiki/doc/TorMessenger/FAQ [3] - https://trac.torproject.org/projects/tor/wiki/doc/TorMessenger/FAQ#Whereare… [4] - https://bugs.torproject.org/13861 [5] - https://bugs.torproject.org/14388 [6] - https://bugs.torproject.org/16526 -- Sukhbir

1 0

Shared random value calculation edge cases (proposal 250)
by George Kadianakis 07 Dec '15

07 Dec '15

Hello there believers of prop250, you can find the latest version of the proposal in the upstream torpec repo: https://gitweb.torproject.org/torspec.git/tree/proposals/250-commit-reveal-… Implementation is also constantly moving forward as you can see in the ticket: https://trac.torproject.org/projects/tor/ticket/16943 Now that we have ironed out the whole voting procedure, it's time to finish this up by figuring out the last details of the shared random value calculation. The logic in the current proposal seems reasonable (see section 3.3), but I have some doubts that I wanted to communicate with you. - I'm afraid of edge cases where different authorities will calculate different shared random values. This is bad because if it happens at the wrong moment, it might break the consensus. For example, imagine that there are 5 authorities that are doing the prop250 protocol. Since there are 5 auths, they can create a valid consensus on their own with SR info in it. Now imagine that one of them, Alice, has a different view of the previous_SRV than the others (maybe because she _just_ booted up before voting and she doesn't have a previous consensus). In this case, Alice will calculate a different SRV than the other 4, and hence the consensus will break because 5 signatures could not be collected. Is this something likely to happen? If yes, a way to protect against it is to add a flag on the votes denoting support for the SR protocol, and have dirauths toggle it off if they are voting for 00:00UTC and they don't know the previous_SRV or if they don't have a consensus. This can also be used as a torrc-enabled killswitch, if the SR protocol bugs out completely and we need to disable it for the sake of the network. What do you think? - Another bothersome thing is the disaster SRV calculation. Specifically, the proposal says: =============================== prop 250: ==================================== If the consensus at 00:00UTC fails to be created, then there will be no fresh shared random value for the day. In this case, and assuming there is a previous shared random value, directory authorities should use the following construction as the shared random value of the day: SRV = HMAC(previous_SRV, "shared-random-disaster") where "previous_SRV" is the previous shared random value. =============================================================================== this logic is not implemented in the current code, and it's not straightforward to implement. Again because the previous_SRV is blended in the formula, but also because it's not easy to know that "the consensus at 00:00UTC fails to be created". For example, if you are a dirauth that just started up at 00:30UTC, and you asked for the previous consensus and you were given the 23:00UTC consensus, then you won't know that the 00:00 consensus was not created and that you need to do the disaster procedure. This will again break the consensus. Not sure if this is a likely scenario as well, and if we should protect against it. What do you think? It depends on the logic that authorities have for fetching consensuses. Do they ensure that they always have the latest consensus? Do we need to add such logic as part of prop250? :/ A way to protect against it is to use the "SR support" vote flag I talked about before, and toggle it off if you are a dirauth and you don't have the latest consensus. Terrible? Would that even allow dirauths to bootstrap SR? - Another interesting part of prop250 is: =============================== prop 250: ==================================== If the shared random value contains reveal contributions by less than 3 directory authorities, it MUST NOT be created. Instead, the old shared random value should be used as specified in section [SRDISASTER]. =============================================================================== do you think this is useful? The fact that we use consensus methods, ensures us that at least 5 dirauths understand the SR protocol, otherwise we don't do it. Should we care about the number of reveal values? And why should the constant be 3, and not 5 or 2? Those were my doubts. Sorry for the extra confusion, but I'm currently reading the proposal and trying to minimize the amount of edge cases that can happen in the implementation (especially if they result in breaking the consensus for everyone). Maybe I'm trying too hard, and there are already multiple such edge cases in the consensus protocol that just never happen and are not worth fixing. In any case, the shared random value calculation seems to be the last piece of the puzzle here, so let's figure it out and finish up! Thanks!

6 16

Graphs - Estimated Traffic Capacity
by David Goulet 07 Dec '15

07 Dec '15

Hello everyone! I would like to share this graph I've been working on along side with Aaron Johnson (thanks to him for the algorithm I'll be describing). This shows both the maximum additional and total traffic the network can sustain for both HS and Exit. The "additional" traffic here means that we consider the current load on the network thus the additional value is what can be added more on top of the current traffic. http://ygzf7uqcusp4ayjs.onion/tor-health/tor-health/bandwidth_capacity.html (Please ignore the initial downward spike, technical issue) Here is the algorithm we are using for this. It's in no ways an accurate value but rather an estimation as you will see with the following algorithm to compute the traffic capacity you see. 1. Using the latest consensus, build a list of relays that are Running, Valid and Fast (weight selection only picks relays with Fast flag). Then compute their positionnal consensus weights. 2. For each of them, set their total capacity to min(avg BW, observed BW). 3. For each of them, compute the unused capacity that is use the extrainfo descriptor read/write bytes history for the amount of traffic used and substract that to the total capacity. max(mean(write_history_values) / write_interval, mean(read_history_values) / read_interval) 4. For HS traffic, that is a 6-hops circuit, we go over all relays and find the relay that has the lowest unused capacity divided by it's probability of being picked in the Guard/Middle position: picked_prob = (guard_weight * 2) + (middle_weight * 4) unused = relay.unused_capacity / picked_prob We use time 2 here since there are two guards in the 6 hops circuit and 4 middle nodes. We get the minimum value between each relays. 5. The minimum "unused" value times the picked_prob is the amount of bytes that we'll give to all relays thus reducing their unused capacity: relay.unused_capacity -= (min_unused * picked_prob) At each iteration of the algorithm, this will make at least one relay go to 0 unused capacity so when this happens, remove that relay from the set of relays to be picked. Once we can't pick a relay anymore for the 6 hops circuit (that is no more guards or middle), we stop. 6. Add min_unused value to the total number of bytes. Goto step 4. Ok... this can maybe be not that trivial to understand at first, feel free to ask questions but the basic idea is that at each iteration of the algorithm you spread a specific amount of bytes to each relay depending on the probability of being picked until you have no more relays capacity for such a circuit. For the "Max Total" traffic, the relay.unused_capacity is equal to the relay capacity. For Exit traffic, we do the same logic but with a 3 hops circuit where each position is multiplied by 1 since there is one guard, middle and exit (in normal circumstances). I know that we have sometimes 1, 4, 5, 6 or 7 hops circuit but the general case is considered here and we have no stats on how frequent those unusual circuits are. Anyway, if you think this algorithm could be improved, please respond. If you think this algorithm is wrong, please respond. If you can reproduce the result on your own with this algo, omg please respond! :) The above could be totally wrong but hopefully we did a fairly good job. Please remember, this is an estimate. :) Thanks! David

5 7

metrics-lib 1.0.0 is released
by Karsten Loesing 05 Dec '15

05 Dec '15

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hello devs, I just released metrics-lib 1.0.0: https://dist.torproject.org/descriptor/1.0.0/ metrics-lib, which also goes by the name DescripTor, is a Java API that fetches Tor descriptors from a variety of sources like cached descriptors and directory authorities/mirrors. The DescripTor API is useful to support statistical analysis of the Tor network data and for building services and applications. # Changes in version 1.0.0 - 2015-12-05 * Major changes - This is the initial release after four years of development. Happy 4th birthday! Special thanks to iwakeh for helping with metrics-lib development over the past year including pushing for finally putting out this release! Happy descriptor parsing! All the best, Karsten -----BEGIN PGP SIGNATURE----- Comment: GPGTools - http://gpgtools.org iQEcBAEBAgAGBQJWYqQ2AAoJEJD5dJfVqbCrHjUH/2Yavypi6BK6F1R3bAyMfCUd t140r4FlH1FBOuWi9qt5p5fbkGroef7ttMHZz3cXqjUJWyfOC3TynB7g152hIflQ E6JeilxeCY62RiOpJ2nRF0KBAKmL1QyVinvE3USORXyT+RvPAhtVpNPU6A8AQlxn lUWsjeD7HNMn9s1VXy3O3m0fs3CGxflbwfbbS8kTP6OINu7nP5f3ldcAm2ykDB0R mG1BgkvGoamhuQmimWVv8S5c/N+pOPLN9La75kSC4BppgBAiD5MCP6zJHhkvMmAG h9ciWsZdAxuKjEdLVCIYRXkNIGAMLikr7HpYB5ewBu8tcGkBHQaCXweNA5sI0Po= =jY3h -----END PGP SIGNATURE-----

1 0

Have the Tor daemon use more cores
by Jeremy 03 Dec '15

03 Dec '15

On 12/02/2015 07:00 AM, tor-dev-request(a)lists.torproject.org wrote: >> >I've been working on the volunteer project described here >> >https://www.torproject.org/getinvolved/volunteer.html.en#useMoreCores >> >but can't spend much more time on it. >> > >> >Right now, I have refactored circuit_receive_relay_cell() in relay.c >> >(which calls relay_crypt() and eventually the AES crypt routines) to >> >use the workqueue.c infrastructure similar to cpuworker.c. >> > >> >When the refactored code runs in single threaded mode, all seems >> >good. Once I activate the thread pool and start sending it work with >> >threadpool_queue_work(), it Bootstraps 100% okay and runs for several >> >minutes before crashing on cells it doesn't handle properly. >> > >> >I'm happy to share my code with anyone interested. > Sure I think I'm the person that used the workqueue stuff last, so I'd > be up for doing the rest of the work (nb: I am traveling/in meetings for > the rest of the week, so it'll be next week at the earliest that I can > really spend quality time). > > Regards, > > -- Yawning Angel Yawning, or anyone else: If you assess the code has potential, I can assist & work on it more. If it is a big miss, then I might start with a smaller contribution. Thanks, Jeremy

1 0