- tor-dev - lists.torproject.org

Request to expunge project 'torsocks' on Google Code
by Virgil Griffith 26 Oct '14

26 Oct '14

Sitting with David Goulet, it's problematic that when I search for "torsocks" I get the old Google Code page instead of the up-to-date page on http://gitweb.torproject.org This is problematic because people report bugs to the Google Code repository instead of the torproject repository. Jake/Robert: You are the project owners on Google Code. Could you please remove the project? Much love, -Virgil

6 8

tor-fw-helper redux
by Yawning Angel 26 Oct '14

26 Oct '14

Hello, "You are in a maze of twisty little firmwares, all terrible". I'm at the point where the new and improved firewall helper could use some additional testing by various users, though there's some issues with the design that still need to be resolved. But not being one to keep issues inherited from the original from stopping progress... Code: https://github.com/yawning/go-fw-helper Bug: https://trac.torproject.org/projects/tor/ticket/13338 Yes, it's another Go app. It should work both as a helper for tor (PortForwardingHelper in the torrc), and for flashproxy. I am currently only concerned about the latter use case since it will immediately make flashproxy more useful in various environments, and I'm not sure people that can't setup port forwarding should be running relays in the first place. How to test it: $ go build github.com/yawning/go-fw-helper One of: * Play with it by hand. "-h" dumps usage. * Edit the torrc-defaults file shipped with Tor Browser to have flashproxy use the helper. On the Linux bundle the file in question is located at tor-browser_en-US/Browser/TorBrowser/Data/Tor The flashproxy ClientTransportPlugin line should look something like: ClientTransportPlugin flashproxy exec ./TorBrowser/Tor/PluggableTransports/flashproxy-client --port-forwarding-helper=/path/to/go-fw-helper --register :0 :9000 You *can* edit where it says "9000" to use a different port, but having it auto assigned will lead to it being harder to remove when done. * Try running a tor relay using PortForwardingHelper. I personally don't recommend this, and haven't tested it at all, but there's no reason why it won't work unless the tor side of the code rotted (unlikely?). Caveats: * Not sure which version of the Go compiler/runtime this requires. Development was done on 1.3.3. It probably requires 1.2.x but it may work on older versions (Not a bug/WONTFIX). * UPnP discovery requires being able to listen on a UDP traffic and accept incoming packets. Your local firewall may prevent this. (Not a bug/WONTFIX). * flashproxy does not know how to deal with mappings expiring, so things will stop working after 2 hours if NAT-PMP is used. * Neither flashproxy nor tor know how to use go-fw-helper to delete mappings, because tor-fw-helper did not have such a thing. WARNING: If UPnP is used as the protocol, mappings are indefinite. You will need to use the router's admin interface or "go-fw-helper -d" to remove it. Yes, there is a very good reason why this is like this, despite the protocol on paper having the length as a registration time parameter. Note: NAT-PMP mappings obtained by go-fw-helper last for 2 hours. * go-fw-helper's "-T" option doesn't do everything tor-fw-helper's does. (Meh?) * The UPnP mapping description is hard coded to "Tor relay" to match tor-fw-helper. Useful extensions over tor-fw-helper: Dump all current mappings with: $ go-fw-helper -l Remove mappings with: $ go-fw-helper -d ([<external port>]:<internal port>] Both of those options only work with UPnP because NAT-PMP does not have a method of querying mapping information, and because at least one NAT-PMP implementation deployed on a lot of routers does not handle removal correctly (Bug reported to maintainer, and fixed in master but people do not update router firmware enough. There is a way to force go-fw-helper to let you delete port forwarding entries, but it's intentionally undocumented, because I don't want to support it.) Force a certain protocol (Case sensitive): $ go-fw-helper --protocol=NAT-PMP ... $ go-fw-helper --protocol=UPnP ... Tested on: * 64 bit Linux * 64 bit FreeBSD 9.3 * 32 bit Windows 7 Need testing on: * Darwin (esp with NAT-PMP) * With more routers than the 2 I have immediate access to. If something breaks "-v" gets you debug output. Questions, Comments, Feedback appreciated, -- Yawning Angel

2 2

repo: TLS vs. GPG signed files (#12871)
by Nusenu 24 Oct '14

24 Oct '14

Hi Ondrej, [I felt it is better to discuss this via email if you feel otherwise feel free to move the discussion back to trac.] even though it was also me requesting the use of HTTPS for the repos [1] - and I'm glad it has been (partially) accepted and implemented I do not follow your comment that HTTPS is "better" than repo_gpgcheck [2]. It is my opinion that even in the case of HTTPS GPG signatures provide a security improvement since (I hope) the private GPG key used to sign the repo is less exposed than the wildcard certificate for *.tpo. (I filed #13553 [4] to address rogue CAs / certificate pinning for yum.) Could you elaborate on your issue regarding repo_gpgcheck not showing fingerprints? (It does show the gpg key fingerprint on a fc20 system after adding repo_gpgcheck=1 and running 'yum update' [3]). thanks for providing and maintaining the RPM repo, Nusenu [1] https://trac.torproject.org/projects/tor/ticket/12897 [2] https://trac.torproject.org/projects/tor/ticket/12871#comment:8 [3] Importing GPG key 0x5AC001F1: Userid : "torproject.org RPM signing key" Fingerprint: 3b9e eeb9 7b1e 827b cf0a 0d96 8af5 653c 5ac0 01f1 From : https://deb.torproject.org/torproject.org/rpm/RPM-GPG-KEY-torproject.org.asc Is this ok [y/N]: [4] https://trac.torproject.org/projects/tor/ticket/13553

2 1

Pluggable transports meeting tomorrow (16:00UTC Wednesday 22nd of October 2014)
by George Kadianakis 21 Oct '14

21 Oct '14

Hello! just wanted to remind you that the regular biweekly pluggable transports meeting is going to occur tomorrow at 16:00 UTC. Place is the #tor-dev IRC channel in the OFTC network. Thanks for your attention!

1 0

Re: [tor-dev] Potential projects for SponsorR (Hidden Services)
by str4d 21 Oct '14

21 Oct '14

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA512 George Kadianakis wrote: > == Opt-in HS indexing service == > > This seems like a fun project that can be used in various ways in > the future. Of course, the feature must remain opt-in so that only > services that want to be public will surface. > > For this project, we could make some sort of 'HS authority' which > collects HS information (the HS descriptor?) from volunteering > HSes. It's unclear who will run an HS authority; maybe we can work > with ahmia so that they integrate it in their infrastructure? > > If we are more experimental, we can even build a basic petname > system using the HS authority [2]. Maybe just a "simple" NAME <-> > PUBKEY database where HSes can register themselves in a FIFO > fashion. This might cause tons of domain camping and attempts for > dirty sybil attacks, but it might develop into something useful. > Worst case we can shut it down and call the experiment done? > AFAIK, I2P has been doing something similar at > https://geti2p.net/en/docs/naming > We have been running our petname system for at least five years (probably longer), mostly without incident. The above linked page covers details of the current system; see [0] for a (slightly dated) more general discussion of the reasons behind the I2P naming system, common arguments and possible alternatives. We have also started looking at GNS as an option[1,2]. I like the concept, and the ability for users to easily control their own domain name zones, which replaces the problem of a user losing their Destination keys (.onion equivalent) down to only needing to securely maintain their zone key. But the UI/UX needs vast improvement if everyday users are to understand it, and we need to research the trade-offs carefully. I am happy to discuss this further if anyone is interested, because a combined / general approach would benefit multiple networks. str4d [0] https://geti2p.net/en/docs/discussions/naming [1] http://zzz.i2p/topics/1545 (in I2P) [2] http://trac.i2p2.de/wiki/GNS -----BEGIN PGP SIGNATURE----- iQIcBAEBCgAGBQJURgOtAAoJEIA97kkaNHPnfUQP/RQIBUCjCQcRvDyXL5l+D0Cv yoE7JT2vr7mqQx+/4nT9w/EhuQvakS5lK8gsod3yHDErtbNa8Ot/NylDbPYvdSo0 mXYOWRtaozv/8VIrRZuFc1NLne/08RYAIrnC3A9NCbpCcvwVLMxWXdl3DzEuZ+R5 rVijvHXW2nhVKjEYFez4dtErA+qTC/nKtdlG5hG1gq4mTTbWyYPEVmUonZe7t8l6 +iVc5jG9bKS6ng1Mcvmj5XwDN2rRRk3igo+64gyk7GsHSi3IXxq2S8H+hVD16hk+ gPmS0+oX5Vy6Ww04xM+x55X3NBHbLzkb+dEMYC/gYVLZrmxeG76iAQ/vBDCJncGC 8JI+KhoCx/hQGzvIby1T2VEf8t/fyREKwDq/jmAQdraxqoA5/z5xHwgpetNRRJOM 6LuOtc3jkxe0fOrsJcvHXLQybOKE8bLrLf0jLlmOpKR60EqmFAZaXC8qbzfItvaA KBHKO9t4e1F0t96PH0qhTJjlqqJVunfHyaulC6i9/HBRIZAnjrDtxvN9oQgtyYzI TRw/uxqR0o4c3iDet472YOjLR0JDwz23OHGT66VG/xiNYN2siNpSeBoVXmpc7YuC PGqjgZJkl3EZ/AIU+2mS+lAvmouT9UmW58fIF0BD+bmsr1//zfCIxXH8/EeOZBCw sXUQa30VVMdDYCmQc8dX =HUK1 -----END PGP SIGNATURE-----

1 0

Reminder: weekly Tor development meeting (Wed), and weekly patch workshop (Tue)
by Nick Mathewson 21 Oct '14

21 Oct '14

Hi, all! You probably know that we do our regular meetings for tor (the program) development on Wednesdays at 1330 UTC on the #tor-dev IRC channel on irc.oftc.net. But did you know everybody who's interested in doing anything with tor is welcome to attend? And did you know that anybody who's interested is welcome to lurk? Also, we've started doing a weekly patch workshop where we all look at each other's patches and try to help get them done, try to help code get better, and so forth. That's on Tuesdays at 1700 UTC on the same IRC channel! (Thanks for rl1987 for suggesting this.) cheers, -- Nick

1 1

tor2web 3.1.20 released!
by Giovanni Pellerano 19 Oct '14

19 Oct '14

Hi all, i've released a 3.1.20 version that includes some networking bugfixing and optimizations. it also includes the following two new features: 1) with numes we discussed https://github.com/globaleaks/Tor2web-3.0/issues/157 in past i added a feature in order to update the blocklist of a tor2web node mergin a remote bloklist. numes asked for the possibility to replace local blocklist with a remote one in order to implement different policies. the result is that we now have two modes for this REPLACE mode and MERGE mode 2) https://github.com/globaleaks/Tor2web-3.0/issues/158 in globaleaks in past we has the need to ad a TRANSLATION mode of tor2web to make a single node as a proxy for only a single hidden service: e.g., demo.globaleaks.org -> demo-HS now i've added an /etc/hosts like that enable a mapping like the following: demo.globaleaks.org hiddenservice1.onion adopter1.demo.globaleaks.org hiddenservice2.onion adopter2.demo.globaleaks.org hiddenservice3.onion this would result particular interesting in order to implement mnemonic names translation for specific hidden services e.g.: puppetname.tor2web.org antani123.onion as usual you will find documentation for this at https://github.com/globaleaks/Tor2web-3.0/wiki/Configuration-Guide and in /etc/tor2web.conf.example ciao! evilaliv3

1 0

Re: [tor-dev] Building TOR using Visual Studio
by teor 18 Oct '14

18 Oct '14

On 18 Oct 2014, at 13:29 , tor-dev-request(a)lists.torproject.org wrote: > Date: Fri, 17 Oct 2014 22:29:02 -0400 > From: Nick Mathewson <nickm(a)freehaven.net> > To: tor-dev(a)lists.torproject.org > Subject: Re: [tor-dev] Building TOR using Visual Studio > * Some people want to use paid versions of Visual Studio, and have > paid for a version earlier than VS2013, and don't want to pay for a > newer one. I sympathize with this: I've been on paid upgrade > treadmills myself, and it's always tempting to save money by skipping > steps in the upgrade treadmill. I've even paid for commercial > compilers in my misspent youth. Are there no-cost, non-license-restricted compilers available for Windows that support C99? This could be a way out for those who don't wish to pay for the VS 2013 upgrade. But it's a bit more of a barrier than using an existing compiler on the system. > * Some compilers for weird old hardware have never been upgraded to > even rudimentary C99 support, and trying to build code with those > weird old compilers is a good way to expose some bugs. I sympathize > with this too: there was one guy who would always compile new versions > of Tor on his old Irix boxes, and he always turned up a new warning or > two when he did. Static analysers, better compiler warnings, and runtime checks are starting to fill the role previous occupied by obscure systems. And mobile/embedded platforms help with this too :-) I think we may be able to compensate for lack of C89 support for old compilers, by using a combination of coverity, clang --analyze, gcc/clang -ftrapv , and clang -fsanitize=undefined-trap -fsanitize-undefined-trap-on-error. Oh, and unit tests :-) teor pgp 0xABFED1AC hkp://pgp.mit.edu/ https://gist.github.com/teor2345/d033b8ce0a99adbc89c5 http://0bin.net/paste/Mu92kPyphK0bqmbA#Zvt3gzMrSCAwDN6GKsUk7Q8G-eG+Y+BLpe7w…

6 5

Re: [tor-dev] Building TOR using Visual Studio
by teor 18 Oct '14

18 Oct '14

On 17 Oct 2014, at 23:00 , tor-dev-request(a)lists.torproject.org wrote: > Date: Fri, 17 Oct 2014 15:09:38 +0400 > From: Vladimir <vilgeforce(a)gmail.com> > To: tor-dev(a)lists.torproject.org > Subject: Re: [tor-dev] Building TOR using Visual Studio > > VS2013 if free only in Express version, and Express version is limited: > doesn't have a profiler, for example. Moreover, there are limitations for > using Express version, it's written in license. > > I'v attached a diff where I changed a code for VS2008 (tested on ubuntu-64 > gcc 4.8.2 too). Also I changed readme: it contained instruction to run > ./configure, but there's no such file. >

6 6

ooniprobe v1.2.2 is out
by Arturo Filastò 17 Oct '14

17 Oct '14

Hi all, This mail is to inform you that a new version of ooniprobe has just been released. Here is the changelog: v1.2.2 (Fri, 17 Oct 2014) ------------------------- Who said friday 17th is only bad luck? * Add two new report entry keys test_start_time and test_runtime * Fix bug that lead to ooniresources not working properly It is now available for download from pypi: https://pypi.python.org/pypi/ooniprobe and soon will be available for download in debian. Have fun! ~ Arturo

1 0

Building TOR using Visual Studio
by Владимир Мартьянов 17 Oct '14

17 Oct '14

Hi all! I'm new at this list :-) I decided to understand how TOR works and I want to build it in VS to debug it and explore it's internals. I have Visual Studio on my first PC and I got errors during build on address.c. I investigated the reason: commit 0ca83872468af59b94e14fe7fdfcb38cb5a3f496 I have Visual Studio express 2013 on my second PC and I didn't have any problems building the TOR. So I have two questions: did you decide not to support old Visual Studio versions or it'll be better to build TOR in VS2008 too? If old versions aren't supported, should it be some #error directives in sources to explain this decision? It was really hard to understand where's the problem, so I think #error will be very helpfull. If you give me some instructions about the problem, I'll try to commit the changes. Thank you.

4 5

Some ideas on the visualization of OONI data
by kudrom 16 Oct '14

16 Oct '14

Hi all For the last couple of days i've been thinking about the visualization of the bridge reachability data and how it relates to the currently deployed ooni [7] system, here are the conclussions: == Variables == I think that the statistical variables for the bridge reachability reports are: - Success of the nettest (yes, no, errors) - The PT of the bridge (obfs3, obfs2, fte, vanilla) - The pool from where the bridge has been extracted (private, tbb, BridgeDB https, BridgeDB email) - The country "of the ooni-probe" With these variables I believe we can answer a lot of questions related to how much censorship is being taken, where and how. But there's something left: the timing. George sent an email [0] in which he proposes a timeline of events [5] of every bridge that would allow us to diagnose with much more precission how and why is a bridge being censored. To build that diagram we should define first the events that will be showed in the timeline. I think those events are the values of the pool variable and if the bridge is being blocked in a given country. With the events defined i think we can define another variable: - Time deltas between bridge events. So, for example, what this variable will answer is: how many {days, hours...} does it take China to block a bridge that is published in bridgeDB? Is China blocking new bridges at the same speed that Iran? How many days does it take China block a private bridge? There are some ambiguities related to the deltas, for example if the bridge is sometimes blocked and sometimes not in a country, which delta should we compute? Finally, in the etherpad [1] the tor's bootstrap is suggested as a variable, i don't understand why. Is it to detect some way of censorship? Can anyone explain a little more? == Data schema == In the last email Ruben, Laurier and Pascal "strongly recommended importing the reports into a database". I deeply believe the same. We should provide a service to query the values of the previous variables plus the timestamp of the nettest and the fingerprint of the bridge. With this database the inconsistencies between the data formats of the reports should be erased and the work with the data is much more easy. I think that we should also provide a way to export the queries to csv/json to allow other people to dig into the data. I also believe that we could use mongodb just because one reason: we can distribute it very easily. But let me explain why in the Future section. == Biased data == Can a malicious ooni-probe bias the data? For example, if it executes in bursts some tests the reports are going to be the same and the general picture could be biased. Any more ideas? == Geo Data == In the etherpad [1] it's suggested to increase the granularity of the geo data to detect geographical patterns, but it seems [2] that at least in China there's not such patterns so maybe we should discard the idea altogether. == Playing with data == So until now i've talked about data. Now i want to address how to present the data. I think we should provide a way to play with data to allow a more thoughtful and precise diagnosis of censorship. What i was thinking is to enhance the interactivity of the visualization by allowing the user a way to render the diagrams at the same time she thinks about the data. The idea is to allow the user to go from more general to more concret data patterns. So imagine that the user loads the visualization's page, first he sees a global heated map of censorship measured with the bridge reachability test, he is chinese so he clicks in his country and a histogram like [3] for China is stacked at the bottom of the global map, he then clicks on the obfs2 and a diagram like [4] is also stacked at the bottom but only showing the success variable for the obfs2 PT, then he clicks on the True value for the success variable and all the bridges that have been reached by all the nettests executions in that period of time in China are showed, finally he selects one bridge and it's timeline [5] plus it's link to atlas [6] is provided. This is only a particular scenario, the core idea is to provide the user with the enhanced capability to drive conclusions as much as she desires. The user started with the more general concept of the data, and he applied restrictions to the datapoints to dig more into the data. From general to specific he can start making hypothesis that he later discards or approves with more info displayed in the next diagram. There are some usability problems with the selection of diagram+variable and the diverse set of users that will use the system, but i'd be very glad to think about them if you like the idea. == Users == I think there are three set of users: 1- User of tor that is interested in the censorship performed in its country and how to avoid it. 2- Journalist that wants to write something about censorship but isn't that tech savvy. 3- Researcher that wants updated and detailed data about censorship. I believe we can provide a system that satisfies the three of them if we succeed in the previous bullet point. == Future == So, why do i think that we should index the data with mongodb? Because i think that this data repository should be provided as a new ooni-backend API related to the current collector. Right now the collectors can write down reports from any ooni-probe instance that chooses to do so and its API is completly separated from the bouncer API, which overall is a wise design decision because you can deploy ooni-backend to only work as a collector. So it's not unreasonable to think that we can have several collectors collecting different reports because the backend is designed to do so, therefore we need the data repository to be distributed. And mongodb is good at this. If we build the database for the bridge reachability nettests, i think that we should design it to index in the future all nettest reports and therefore generalize the desgin, implementation and deployment of all the work that we are going to do to the bridge reachability. That way an analyst can query the distributed database with a proper client that connects to the data repository ooni-backend API. So to sum up, I started talking about the bridge reachability visualization problem and finished with a much broader vision that intends to integrate the ongoing efforts of the bridge reachability to improve ooni as a whole. Hope the email is not too large. ciao [0] https://lists.torproject.org/pipermail/tor-dev/2014-October/007585.html [1] https://pad.riseup.net/p/bridgereachability [2] https://blog.torproject.org/blog/closer-look-great-firewall-china [3] http://ooniviz.chokepointproject.net/transports.htm [4] http://ooniviz.chokepointproject.net/successes.htm [5] https://people.torproject.org/~asn/bridget_vis/tbb_blocked_timeline.jpg [6] https://atlas.torproject.org [7] https://ooni.torproject.org/

1 0

Prop 237 (Make all relays directory caches) Patch
by Matthew Finkel 15 Oct '14

15 Oct '14

Hi Nick, All, I've made some minor corrections to proposal 237. Mostly these are cosmetic changes, but I did remove some...overstatements, as well. Attached is the patch, but it's only available as a branch in my personal repo[0] as prop237-clarifications. Let me know if you have any comments/suggestions (the overall proposal is unchanged). If not, I'll start implementing this within the next few days. Thanks! Matt [0] https://git.torproject.org/user/sysrqb/torspec.git

1 0

Patches to improve mobile hidden service performance
by Michael Rogers 13 Oct '14

13 Oct '14

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 Hi all, I've been experimenting with small changes to Tor to improve the performance of mobile hidden services. The following patches for Tor and jtorctl make two performance improvements: 1. Each time the network's enabled, don't try to build introduction circuits until we've successfully built a circuit. This avoids a problem where we'd try to build introduction circuits immediately, all the circuits would fail, and we'd wait for 5 minutes before trying again. 2. Added a command to the control protocol to purge any cached state relating to a specified hidden service. This command can be used before trying to connect to the service to force Tor to download a fresh descriptor. It's a more fine-grained alternative to SIGNAL NEWNYM, which purges all cached descriptors and also discards circuits. https://code.briarproject.org/akwizgran/briar/blob/9e5e2e2df24d84135f14adaa… https://code.briarproject.org/akwizgran/briar/blob/9e5e2e2df24d84135f14adaa… The Tor patch is based on the tor-0.2.24 tag, and the jtorctl patch is based on the Guardian Project's repo, which is ahead of the upstream repo (n8fr8 needs commit privileges for upstream, I think). https://github.com/guardianproject/jtorctl I've only done small-scale testing of these patches so far. If they seems like they might be useful I'll create a trac ticket to merge them. Cheers, Michael -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.12 (GNU/Linux) iQEcBAEBCAAGBQJUM6W7AAoJEBEET9GfxSfM5FkIAL+gPk2ZmGvSnt/gVQGZwqH5 mpqGoPIfZPycfhm/dgOjE4MSANcGo7kTqnh85ir9TQvjtNAEkfkTL7GB9G+DOd2X ghje+HCWNRU8WlFD27rrxSUzT+8IvfVaJaj+sCYtA44ib3qpz4XXKXee8xTW2epV dnCLtW7TBUjQEHqbL0aGtsKEHvgVQcLWPxusEioefBPXo6a+8cRKLY+EJvqxaMze mdOQBh+wa7SIyMQi4JqgzjtKAoAqpVzXYsIrmxT1hlCOOaNDjbFUN35QbrZMT6B+ zNCuWIIC6JfBWccfxDsAew2MC8WNf05nvgM67cTFU2nCI6MwYYf4x2xtEOcA5qs= =LweI -----END PGP SIGNATURE-----

4 5

Optimising Tor node selection probabilities
by Steven Murdoch 13 Oct '14

13 Oct '14

I’ve just published a new paper on selecting the node selection probabilities (consensus weights) in Tor. It takes a queuing-theory approach and shows that what Tor used to do (distributing traffic to nodes in proportion to their contribution to network capacity) is not the best approach. Counter-intuitively the paper shows that some of the slowest nodes should not be used at all, because if they are used they will slow down the average performance for all users. The proportion of nodes which shouldn’t be used depends on the relationship between network usage and network capacity, so will vary over time. It’s not clear that there is a closed-form solution to the problem of calculating node selection probabilities (I couldn’t find one), but this paper shows that the optimisation surface is convex and so gradient-based optimisation methods will find the global optimum (rather than some local optimum which depends on the starting position of the optimisation process). Although the process outlined in the paper requires knowing the relationship between network capacity and usage, it isn’t highly sensitive to minor inaccuracies in measuring this value. For example if it is assumed the network is loaded at 50% then the solution will outperform Tor’s old approach provided the true network load is between 0% and 60%. After this work was done, Tor moved to actively measuring the network performance and manipulating the consensus weights in response to changes. This seems to have ended up with roughly the same outcome. The advantage of Tor’s new approach is that it doesn’t require knowing network usage and node capacity; however the disadvantage is that it can only react slowly to changes in network characteristics. For more details, see the paper: http://www.cl.cam.ac.uk/~sjm217/papers/#pub-el14optimising Note that this is published in IET Electronics Letters, which is a bit different to the usual Computer Science publication venues. It jumps straight into the maths and leaves it to the reader to understand the background and implications. The advantage is that it’s 2 pages long; the disadvantage is that to understand it you need to know a reasonable amount about Tor and queuing theory to make much sense of it. Best wishes, Steven

2 2

Re: [tor-dev] Optimising Tor node selection probabilities
by teor 12 Oct '14

12 Oct '14

On 11 Oct 2014, at 23:00 , tor-dev-request(a)lists.torproject.org wrote: > Date: Fri, 10 Oct 2014 14:33:52 +0100 > From: Steven Murdoch <Steven.Murdoch(a)cl.cam.ac.uk> > To: tor-dev(a)lists.torproject.org > Subject: [tor-dev] Optimising Tor node selection probabilities > Message-ID: <FDECA8F4-5F99-4738-8391-CD60D156D774(a)cl.cam.ac.uk> > Content-Type: text/plain; charset=windows-1252 > > I?ve just published a new paper on selecting the node selection probabilities (consensus weights) in Tor. It takes a queuing-theory approach and shows that what Tor used to do (distributing traffic to nodes in proportion to their contribution to network capacity) is not the best approach. > > Counter-intuitively the paper shows that some of the slowest nodes should not be used at all, because if they are used they will slow down the average performance for all users. The proportion of nodes which shouldn?t be used depends on the relationship between network usage and network capacity, so will vary over time. > > It?s not clear that there is a closed-form solution to the problem of calculating node selection probabilities (I couldn?t find one), but this paper shows that the optimisation surface is convex and so gradient-based optimisation methods will find the global optimum (rather than some local optimum which depends on the starting position of the optimisation process). > > Although the process outlined in the paper requires knowing the relationship between network capacity and usage, it isn?t highly sensitive to minor inaccuracies in measuring this value. For example if it is assumed the network is loaded at 50% then the solution will outperform Tor?s old approach provided the true network load is between 0% and 60%. > > After this work was done, Tor moved to actively measuring the network performance and manipulating the consensus weights in response to changes. This seems to have ended up with roughly the same outcome. The advantage of Tor?s new approach is that it doesn?t require knowing network usage and node capacity; however the disadvantage is that it can only react slowly to changes in network characteristics. > > For more details, see the paper: > http://www.cl.cam.ac.uk/~sjm217/papers/#pub-el14optimising > > Note that this is published in IET Electronics Letters, which is a bit different to the usual Computer Science publication venues. It jumps straight into the maths and leaves it to the reader to understand the background and implications. The advantage is that it?s 2 pages long; the disadvantage is that to understand it you need to know a reasonable amount about Tor and queuing theory to make much sense of it. > > Best wishes, > Steven This is fantastic, Steven - and although we've changed Tor's consensus weights algorithm, we still waste bandwidth telling clients about relays that wold slow the network down. Your result further supports recent proposals to remove the slowest relays from the consensus entirely. teor pgp 0xABFED1AC hkp://pgp.mit.edu/ https://gist.github.com/teor2345/d033b8ce0a99adbc89c5 http://0bin.net/paste/Mu92kPyphK0bqmbA#Zvt3gzMrSCAwDN6GKsUk7Q8G-eG+Y+BLpe7w…

2 1

Tor Metrics website redesign
by Karsten Loesing 11 Oct '14

11 Oct '14

Hi everyone, I started redesigning the Tor Metrics website and would like to hear your opinion. The most recent version of the redesign is available here: https://kloesing.github.io/metrics-2.0/ This is a non-functional design prototype in an early development state. Each page contains a note at the bottom that explains what design decisions were made. For reference, here's the current Tor Metrics website: https://metrics.torproject.org/ Feedback much appreciated. This is the perfect time to consider your ideas. Thanks! All the best, Karsten

1 0

hidden service???
by EBOLA 11 Oct '14

11 Oct '14

is hidden server hack possible ? plase respon t y

2 1

Guardiness: Yet another external dirauth script
by George Kadianakis 09 Oct '14

09 Oct '14

==Guardiness: Yet another external dirauth script== ====Introduction==== One well-known problem with Tor relays, is that Guards will suffer a big loss of traffic as soon as they get the Guard flag. This happens because clients pick guards every 2-3 months, so young guards will not get picked by old clients and mainly attract new clients. This is documented in 'phase three' of Roger's blog post: https://blog.torproject.org/blog/lifecycle-of-a-new-relay The problem gets even worse if we extend the guard lifetime to 8-9 months. The plan to solve this problem is to make client load balancing a bit smarter by priotizing guards that suffer this traffic loss as middle relays. The reason I'm sending this email is because this feature is by far the trickiest part of prop236 (guard node security) and I wanted to inform all dirauths of our plan and ask for feedback on the deployment procedure. ====How guardiness works==== Authorities calculate for each relay how many consensuses it has been a guard for the past 2-3 months, and then they note that fraction down in the consensus. Then clients parse the consensus and if they see a guard that has been a guard for 55% of the past consensuses, they will consider that relay as 55% guard and 45% non-guard (that's 100% - 55%). You can find more information at: https://gitweb.torproject.org/torspec.git/blob/HEAD:/proposals/236-single-g… The idea was that the guardiness script will be an external script that is run by Tor in a similar fashion to the bandwidth auth scripts. We chose that because we could write the script in a high-level language and because it could be modular and we could change the algorithm in the future if we wanted. Unfortunately, it seems that external scripts in dirauths are a PITA to maintain as can be seen by the lack of bwauth operators. ====How the guardiness script works==== The guardiness script, is supposed to parse 2-3 months worth of consensuses (but should also be to do the same for 9 months worth of consensuses) and calculate the guard fraction of each guard, save it to a file, and have the dirauth read it to update its routerstatuses. One problem I encountered from early on, is that stem takes about 30mins to parse 3 months of consesuses (~2000 consensuses). Since this script should ideally be run every hour before each authority votes, such long parsing time is unacceptable. I mentioned this problem at https://trac.torproject.org/projects/tor/ticket/9321#comment:19 and stated a few possible solutions. I received some feedback from Nick, and the solution I decided to take in the end is to have another script that is called first and summarizes consensuses to summary files. Summary files are then saved to disk, and parsed by the guardiness script to produce an output file that is read by dirauths. Summary files are designed to be quick to parse (even with Python) and contain all the necessary information for guardiness. For example, parsing 2000 summary files in my laptop takes about 10 seconds. FWIW, the guardiness scripts are ready for review and can be found here: https://gitweb.torproject.org/user/asn/hax.git/shortlog/refs/heads/guardine… ====How the guardiness script will be deployed==== The idea is that dirauths will add another script to their crontab that is called every hour (before or after the bwauth scripts). The script first calls the summarizer script, which goes to the consensus/ directory and summarizes all consensuses it finds and puts them in the summary/ directory. The summarizer script then deletes all the consensuses that got summarized. Then the script calls the the guardiness script, which goes to the summary/ directory, parses all summary files it finds, and outputs a guardiness output file that gets parsed by the dirauth prior to voting. That should be all. Easy, eh? :) Now I will start a FAQ section where I state my doubts and fears. ====FAQ==== - Q: Where do dirauths find all those old consensuses? There are various ways for dirauths to populate their consensus/ directory. They could fetch consensuses from metrics, or they could add a cron job that copies cached-consensus to a directory every hour. However, I think the cleanest solution is to use Daniel Martí's upcoming consensus diff changes. Daniel will add a torrc option that allows Tor to save consensuses to a directory. My idea was to get dirauths to use Daniel's code to populate their consensus/ directory for two or three months. And then, after two or three months enable the guardiness scripts. To make sure that this is indeed the best approach, I need to learn from Nick when he plans to merge Daniel's code to Tor. - Q: How does guardiness look like in the consensus? Here is how a guard with guardiness (GuardFraction) 10% looks like in the consensus: r test006r HyS1DRHzEojbQVPZ1B3zAHc/HY0 9St4yWfV4huz5V86mt24HL3Yi2I 2014-09-06 13:44:28 127.0.0.1 5006 7006 s Exit Fast Guard HSDir Running Stable V2Dir Valid v Tor 0.2.6.0-alpha-dev w Bandwidth=111 Unmeasured=1 GuardFraction=10 - Q: What are you afraid of? I'm mainly afraid of misconfiguration problems. This guardiness system is a bit complex and I'm not expecting dirauths to learn how to use it and debug it, so it should work easily and well... Here are some specific issues: -- File management For example, I'm afraid of the file management mess that summary files cause. We need to make sure that we don't leave old consensuses/summary files rot in the filesystem. Or that we don't summarize the same consensuses over and over again. To do that, I added some optional cleanup switches to both scripts: Specifically, the summarizer script can delete consensus files that already got summarized and can also delete consensus files older than 3 months (or N months). Similarly, the guardiness.py script can delete summary files older than 3 months (or N months). The idea is that every time the cron job triggers, both scripts will auto-delete the oldest summary/consensus file, keeping in disk only the useful files. -- Incomplete consensus data set I'm afraid that a directory authority might not have a properly populated consensus directory and hence advertise wrong guard fractions. For example, maybe it only has 10 consensuses in its consensus directory instead of 1900 consensuses. Since the authorities only state the guardiness percentage in the consensus, it's not possible to learn how many consensuses were in their dataset. Maybe we need to add a "guardiness-consensus-parsed" in their votes, to easier debug such issues? Also, 3 months worth of consensuses is 2160 consensuses. Because dirauths sometimes misbehave, it's certain that not all 2160 consensuses will have been issued and that's normal. But how do we understand if dirauths have a sufficiently good consensus data set? Is 2000 out of 2160 consensuses an OK data set? What about 1000 out of 2160 consensuses? Furthermore, we need to make sure that dirauths don't consider old consensuses in their GuardFraction calculations. To achieve this, both scripts have a mandatory switch that allows operators to specify the maximum consensus age that is acceptable. So for example, if you call the summarizer script with 3 months of consensus age, it will not parse consensuses older than 3 months. Furthermore, there is a CLI switch that allows the scripts to delete expired consensuses. - Q: Why do you slow stem instead of parsing consensuses with Python on your own? This is another part where I might have taken the wrong design decision, but I decided to not get into the consensus parsing business and just rely on stem. This is also because I was hoping to use stem to verify consensus signatures. However, now that we might use Daniel's patch to populate our consensus database, maybe we don't need to treat consensuses as untrusted anymore. If you think that I should try to parse the consensuses on my own, please tell me and I will give it a try. Maybe it will be fast. Definitely not as fast as summary files, but maybe we can parse 3 months worth of consesuses in 15 to 40 seconds. - Q: Why do you mess with multiple summary files and you don't just have *one* summary file? Because of the rolling nature of guardiness (we always want to consider the past 3 months), every hour we need to _discard_ the oldest observations (the consensus from 3 months ago) and start considering the newest consensus. Because we need to discard that oldest consensus, it's hard to keep information about each consensus in a single summary file. And that's why I chose to have a summary file for each consensus. Maybe it's the wrong decision though... - Q: What's up with the name "guardiness"? It's quite terrible I know but it was the name that we used from quite early on about this project. I think before finalizing this task I'm going to rename everything to 'GuardFraction' since it's more self-explanatory. I'm also considering names like "GuardTrafficLoadBalancer" etc.

3 8

GetTor dev meeting (Friday 10th, 5.30pm UTC)
by ilv 08 Oct '14

08 Oct '14

Hi people! We are having our GetTor dev meeting on Friday 10th at 5.30pm UTC. It will take place at the #tor-dev IRC channel in the OFTC network. This is the last meeting before we deploy the new version of GetTor. Everyone is welcome to participate! Have a nice day. -- 4096R/540BFC0E

1 0

Re: [tor-dev] On the visualization of OONI bridge reachability data
by Arturo Filastò 08 Oct '14

08 Oct '14

On 10/7/14, 6:18 AM, M. C. McGrath wrote: >> The kind of data that they end up generating looks something like >> this: >> http://arturo.filasto.net/vizPlayground/bridge_rearchability.csv > > Nice- though the link seems to be dead for me (I get a 404). But it is > great that you are working on this as having this even in a rough form > makes it possible to get started on visualizations. Yeah sorry I mistyped. The correct link is: http://arturo.filasto.net/vizPlayground/bridge_reachability.csv ~ Art.

2 2

Pluggable transports meeting today (16:00UTC Wednesday 8th of October 2014)
by George Kadianakis 08 Oct '14

08 Oct '14

Hello! just wanted to remind you that the regular biweekly pluggable transports meeting is going to occur today at 16:00 UTC. Place is the #tor-dev IRC channel in the OFTC network. Thanks for your attention!

1 0

[PATCH] torspec/rend-spec-ng: Link to proof of the keyblinding scheme.
by George Kadianakis 07 Oct '14

07 Oct '14

I attach a patch that adds a link to Nick Hopper's proof of the keyblinding scheme in rend-spec-ng.txt. Thanks!

2 1

Re: [tor-dev] On the visualization of OONI bridge reachability data
by Matthew Finkel 06 Oct '14

06 Oct '14

On Sat, Oct 04, 2014 at 06:27:22PM -0700, M. C. McGrath wrote: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA256 > > Hi, > These were a few possibilities for visualization that we came up with > at the OTF summit (I can send the full notes from that discussion if > everyone is okay with it): > - - Timelines (by protocol, pool, country) > - - Pie charts for above > - - Timeline/graph of time it takes to block bridge from when added to > TBB (github parser) Similar to the next one, I wonder if showing a map cooresponding to this data would also help. At t0, zero countries block the built-in bridges, at t1 = only China blocks, at t2 = China + Iran block, at t3 = China + Iran + Syria block, t4 = t3 + Turkey, etc. I'm thinking this would be nice in addition to the timeline which George sketched (where some of the time points are clickable and update the map). I don't actually know how difficult this is to make. > - - Geographic breakdown by region (if enough data points) Could be > similar to this map of % of internet users who use Tor by country > https://transparencytoolkit.org/tormap.html That's cool. Are you able to add a legend to it? Having something like this or similar to RSF's Press Freedom Index [0], based on the number of bridge users, would be nice. This is doable today, using the available metrics data. We'll probably never be able to know the number of users per protocol per country, but at least we can visualize where in the world bridges (in general) are used most and if this changes over time. [0] http://rsf.org/index2014/en-index2014.php But, it would also be really cool if we can create a map like this based on the reachability of bridges per country per protocol and maybe, in addition, color-code/denote how the ISPs/country are interfering with the connection (e.g. throttling, DNS cache poisoning, IP addr/port blocking). > - - At what point in the tor bootstrapping does it fail (may be > difficult to determine, especially anonymized)? Yes, but there's already a risk to running ooni-probe (at least right now, hopefully this will change in time). We will eventually need probes running in most countries if we want a good understanding of what network interference is taking place and who is affected. > - - In all visualizations, compare with control (filter, line break, > plot alongside, etc) > > And the variables we thought would be relevant to visualizations: > Protocol > Pool > Country (and region)- Iran, China, Netherlands (control) > Time it takes to be blocked > Point in bootstrap where it fails > Classify the bridges by commercial/residential connection > Time we started scanning the bridge from where > Maybe latency measurements per protocol? Initially, I'm thinking "the time is takes to download a consensus from the bridge" but there are many variables that may affect this. Anyone have a better idea? I think this mostly covers it. The only addition can think of right now is comparing different control countries against each other (and different ISPs within the control countries). Maybe we'll find something interesting. > It should be relatively simple to make rough versions of a lot of > visualizations to see what works once we have a parser/converter that > will generate JSONs (or similar) from OONI output that include the > variables listed above. > Is someone already working on this? I'm not really volunteering, merely curious if this is in progress. :) > Are there any other variables that would be particularly helpful to > track or visualize? And are there any visualizations (listed or > otherwise) that anyone would find particularly helpful? > > > On 10/04/2014 06:10 PM, George Kadianakis wrote: > > == What is bridge reachability data? == > > > > By bridge reachability data I'm referring to information about > > which Tor bridges are censored in different parts of the world. > > > > The OONI project has been developing a test that allows probes in > > censored countries to test which bridges are blocked and which are > > not. The test simply takes as input a list of bridges and tests > > whether they work. It's also able to test obfuscated bridges with > > various pluggable transports (PTs). > > > > == Why do we care about this bridgability data? == > > > > A few different parties care about the results of the bridge > > reachability test [0]. Some examples: > > > > Tor developers and censorship researchers can study the bridge > > reachability data to learn which PTs are currently useful around > > the world, by seeing which pluggable transports get blocked and > > where. We can also learn which bridge distribution mechanisms are > > busted and which are not. > > > > Bridge operators, the press, funders and curious people, can learn > > which countries conduct censorship and how advanced technology > > they use. They can also learn how long it takes jurisdictions to > > block public bridges. And in general, they can get a better > > understanding of how well Tor is doing in censorship circumvention > > around the world. > > > > Finally, censored users and world travelers can use the data to > > learn which PTs are safe to use in a given jurisdiction. > > > > == Visualizing bridge reachability data == > > > > So let's look at the data. > > > > Currently, OONI bridge reachability reports look like this: > > https://ooni.torproject.org/reports/0.1/CN/bridge_reachability-2014-07-02T0… > > > > > and you can retrieve them from this directory listing: > > https://ooni.torproject.org/reports/0.1/ > > > > That's nice, but I doubt that many people will be able to access > > (let alone understand) those reports. Hence, we need some kind of > > visualization (and better dir listing) to conveniently display the > > data to human beings. > > > > However, a simple x-to-y graph will not suffice: our ploblem is > > multidimensional. There are many use cases for the data and > > bridges have various characteristics (obfuscation method, > > distribution method, etc.) hence there are more than one useful > > ways to visualize this dataset. > > > > To give you an idea, I will show you two mockups of visualizations > > that I would find useful. Please don't pay attention to the data > > itself, I just made some things up while on a train. > > > > Here is one that shows which PTs are blocked in which countries: > > https://people.torproject.org/~asn/bridget_vis/countries_pts.jpg > > The list would only include countries that are blocking at least a > > bridge. Green is "works", red is "blocked". Also, you can imagine > > the same visualization, but instead of PT names for columns it has > > distribution methods ("BridgeDB HTTP distributor", "BridgeDB mail > > distributor", "Private bridge", etc.). > > > > And here is another one that shows how fast jurisdictions block > > the default TBB bridges: > > https://people.torproject.org/~asn/bridget_vis/tbb_blocked_timeline.jpg > > > > These visualizations could be helpful, but they are not the only > > ones. > > > > What other use cases do you imagine using this dataset for? > > > > What graphs or visualizations would you like to see? > > > > [0]: Here are some use cases: > > > > Tor developers / Researcers: *** Which pluggable transports are > > blocked and where? *** Do they do DPI? Or did they just block the > > TBB hardcoded bridges? *** Which jurisdictions are most aggressive > > and what blocking technology do they use? *** Do they block based > > on IP or on (IP && PORT)? > > > > Users: *** Which pluggable transport should I use in my > > jurisdiction? > > > > Bridge operators / Press / Funders / Curious people: *** Which > > jurisdictions conduct Tor censorship? (block pluggable > > transports/distribution methods) *** How quickly do jurisdictions > > block bridges? *** How many users/traffic (and which locations) did > > the blocked bridges serve? **** Can be found out through extrainfo > > descriptors. *** How well are Tor bridges doing in censorship > > circumvention? > >

2 1

Re: [tor-dev] [tor-relays] French Flag
by Karsten Loesing 03 Oct '14

03 Oct '14

[Please don't cross-post. Removed tor-relays@.] On 02/10/14 17:00, Sebastian Urbach wrote: > Hi, > > I saw that a lot of services around tor are using a bright blue in the > french flag. It should be a dark blue like the one used in the Dutch > flag for example ... > > The following services are concerned: > > Atlas > Globe Please open tickets in both components, ideally with references to each other. Maybe they're using the same library. Though I think we shouldn't start modifying flag images ourselves. The better approach would be to report this problem to whoever made these flags, ask them to get the colors right, and upgrade. > Compass I think Compass doesn't use country flags. All the best, Karsten > Oniontip (already fixed, thanks Donncha) > > If there are more services around that i don't know about then i would > like to ask if someone could check them as well. > > It hurts a bit to see the Tricolore like that ... > > Thanks

2 1