May 2015 - tor-dev - lists.torproject.org

Summary of meek's costs, April 2015
by David Fifield 07 May '15

07 May '15

Here's the summary of meek's CDN fees for April 2015. App Engine + Amazon + Azure = total by month February 2014 $0.09 + -- + -- = $0.09 March 2014 $0.00 + -- + -- = $0.00 April 2014 $0.73 + -- + -- = $0.73 May 2014 $0.69 + -- + -- = $0.69 June 2014 $0.65 + -- + -- = $0.65 July 2014 $0.56 + $0.00 + -- = $0.56 August 2014 $1.56 + $3.10 + -- = $4.66 September 2014 $4.02 + $4.59 + $0.00 = $8.61 October 2014 $40.85 + $130.29 + $0.00 = $171.14 November 2014 $224.67 + $362.60 + $0.00 = $587.27 December 2014 $326.81 + $417.31 + $0.00 = $744.12 January 2015 $464.37 + $669.02 + $0.00 = $1133.39 February 2015 $650.53 + $604.83 + $0.00 = $1255.36 March 2015 $690.29 + $815.68 + $0.00 = $1505.97 April 2015 $886.43 + $785.37 + $0.00 = $1671.80 -- total by CDN $3292.25 + $3792.79 + $0.00 = $7085.04 grand total https://metrics.torproject.org/userstats-bridge-transport.html?graph=userst… The number of simultaneous users increased hugely in April, from 2000 to around 5000. There are big increases around April 9 and April 15, which I conjecture were caused by a performance improvement in meek-azure and its coverage in TWN. meek-azure used to be hugely bottlenecked, but now it should handle as many users as the others. https://lists.torproject.org/pipermail/tor-dev/2015-April/008637.html https://blog.torproject.org/blog/tor-weekly-news-%E2%80%94-april-15th-2015 We moved over 10 TB in April. Because of meek's extra layer of indirection, it doesn't correctly compute per-country user statistics (the server sees the IP address of the CDN, not of the actual user). Now that a nontrivial fraction of bridge users use meek, a nontrivial fraction of bridge users are being miscounted. Keep that in mind when you look at the per-country bridge user graphs--if users are connecting through meek, they don't count towards that country's total. Here is a ticket that aims to fix the issue by passing the client IP through the CDN. https://trac.torproject.org/projects/tor/ticket/13171 If you want to help reduce costs, you can 1. Use meek-azure; it's still covered through a grant for the next four months. 2. Set up your own App Engine or CDN account. Then you can pay for your own usage (it might even be free depending on how much you use). Here are instructions on how to set up your own: https://gitweb.torproject.org/pluggable-transports/meek.git/tree/appengine/… https://trac.torproject.org/projects/tor/wiki/doc/meek#AmazonCloudFront https://trac.torproject.org/projects/tor/wiki/doc/meek#MicrosoftAzure Then you will have to enter a bridge line manually. Follow the instructions at https://trac.torproject.org/projects/tor/wiki/doc/meek#Howtochangethefrontd… but instead of changing the "front=" part, change the "url=" part. For example, bridge meek 0.0.2.0:1 url=https://<myappname>.appspot.com/ front=www.google.com == App Engine a.k.a. meek-google == Both bandwidth and instance hours were up about 19%. Here is how the Google costs broke down: 6304 GB $756.59 2597 instance hours $129.84 Compared to the previous month: 5316 GB $637.93 2173 instance hours $108.64 https://globe.torproject.org/#/bridge/88F745840F47CE0C6A4FE61D827950B06F9E4… == Amazon a.k.a. meek-amazon == Both usage and cost were down a bit from the previous month. Asia Pacific (Singapore) 114M requests $136.79 884 GB $117.92 Asia Pacific (Sydney) 443K requests $0.55 3 GB $0.34 Asia Pacific (Tokyo) 51M requests $61.03 305 GB $39.32 EU (Ireland) 173M requests $208.17 1598 GB $127.21 South America (Sao Paulo) 5M requests $9.91 23 GB $5.31 US East (Northern Virginia) 46M requests $45.56 418 GB $33.26 -- total 389M requests $462.01 3231 GB $323.36 https://globe.torproject.org/#/bridge/3FD131B74D9A96190B1EE5D31E91757FADA1A… == Azure a.k.a. meek-azure == meek-azure increased a lot because of improved performance this month. https://onionoo.torproject.org/bandwidth?fingerprint=AA033EEB61601B2B7312D8… Estimated bandwidth use this month: 2015-04 1982 GB Compared to last month: 2015-03 737 GB https://globe.torproject.org/#/bridge/AA033EEB61601B2B7312D89B62AAA23DC3ED8… David Fifield Earlier reports in this series: https://lists.torproject.org/pipermail/tor-dev/2014-August/007429.html https://lists.torproject.org/pipermail/tor-dev/2014-October/007576.html https://lists.torproject.org/pipermail/tor-dev/2014-November/007716.html https://lists.torproject.org/pipermail/tor-dev/2014-December/007916.html https://lists.torproject.org/pipermail/tor-dev/2015-January/008082.html https://lists.torproject.org/pipermail/tor-dev/2015-February/008235.html https://lists.torproject.org/pipermail/tor-dev/2015-March/008427.html https://lists.torproject.org/pipermail/tor-dev/2015-April/008596.html

6 14

exitmap feature requests?
by nusenu 07 May '15

07 May '15

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA512 Hi Philipp, do you consider feature requests via [1] or would you recommend forking and implementing it oneself? thanks, nusenu [1] https://github.com/NullHypothesis/exitmap/issues -----BEGIN PGP SIGNATURE----- iQIcBAEBCgAGBQJVSlpSAAoJEFv7XvVCELh0lCQQALCIDN+o7o5rubDUP7+O2J01 pI1ezUWqRk1BkKXYOPhB17gJke/5XzYCQkA/VW8FLmjf52WnucFg25c7TJ6hbRFd T73jfQvIAoNm938c/ZRHND8xOCOxaO4oa+sd5+oPNDYCfEg3gHfFxHtjgdvUzF+P Y2CqbO271drrFzkWk0ILSoRJihmZjSZHJl21gvfmeDKRsyycig/yKip/Lyq9n8QF O9rneAsVXtT72uCKnjd7rAqVwetTgqiNKQMrOG4hZ61BoGgrwYueWpTVN9wSNNOO 755TnSjZw+/rjYJxIQngbhdFSeFOX2MvE2gaUsut4b1tUa23QSQ5bLhPVEw4P6u2 Hm9lo06RCPNoYe16Hr+ff2rLMK/UU2LfoYkBZk2f0FM7LcYzEOXs6mmeUREBoJw8 V6m+O1orczjwppF9aPBLXihUr8FZQyuhIYMPpwYnMdrtysiFAKvOusOxr7svmM/w nQvPrEy3SuiyyoQa8VGZt0DejeUfyXwhkndbTC/goxKf44pWGDVUB15fyUIKh17l 9vNVdMbCmmRVHa8Mto3RmjlNRG9c3gqsmaMteAJohXGAbe8qoSMqmRF095HYjlhb MD5UPiuNIoAqOBjC2FMTYnLT2bmrRmcQD0i0dUEtK6wg5w11M3pSbk+jR/7AD5uZ gPjSfXUM718ly0avzegm =+LN2 -----END PGP SIGNATURE-----

2 1

Tor + Apache Traffic Server w/ SOCKS - works now!
by CJ Ess 05 May '15

05 May '15

So I've been looking for a long time for something modern to sit between my browser and Tor -- something modern, capable, and efficient (i.e. doesn't fork every connection). Years ago Yahoo got some proxy software from an acquisition, a few years later they made it open source as Apache Traffic Server ( http://trafficserver.apache.org/) and today its the backbone of Yahoo's infrastructure. They have a number of full time engineers that work on it full time, they use it in production, and they are implementing cutting edge features like IPv6, SPDY, and HTTP/2 support. SOCKS is was one of the legacy features of Apache Traffic Server. However, it hasn't been maintained. If you build from git right now you'll find SOCKS support completely broken at least four ways (a couple bad asserts, wrong byte order, and an uninitialized field). They took the documentation on the SOCKS feature out a while ago but never got around to removing the code. Since it was there I spent some time over the weekend and fixed it. There are still some issues around SOCKS still but it works well enough that you can surf though tor with it. If there is interest in it here I'd be happy to put together a how-to for Linux and MacOS to get it built and configured. I'd also like to encourage people to make some noise - Yahoo does have SOCKS servers internally but they don't test using Traffic Server with them because they don't think anyone uses the feature (and they are right, there is no way the code works for anyone in the present state). But if there was interest then maybe they'd keep the code fresh going forward. I'm including a copy of the patch with this e-mail just to get it out. You can pull their git repository (https://github.com/apache/trafficserver) and apply it to the master master branch.

3 3

Re: [tor-dev] working back to socks_request_t
by teor 05 May '15

05 May '15

> Date: Mon, 4 May 2015 16:07:20 -0400 > From: CJ Ess <zxcvbn4038(a)gmail.com> > > Thanks for going into so much detail, you've given me a lot to think about. > The real solution is probably the one that nobody wants to take on - having > an application HTTP port that could take direct input from HTTP aware stuff > and utilize a richer set of information then SOCKS allows for. I've spent a > couple evenings looking to see if I could take the code stuff from the > dirport and use it for that purpose. I need to spend another couple > evenings and and go back and look at the SOCKS4 stuff, I've just recently > realized that the state machine for that is closer to an http > request/response. CJ, Pluggable transports are designed to arbitrarily change how tor makes outgoing connections. The existing transports are used for obfuscation, but they could be used for dynamic proxy authentication as well. (Some do take authentication arguments.) You'd just have to work out which user to authenticate each tor connection with - which is a difficult question of policy. This arises because tor only makes a small number of long-term connections to a few guard nodes, and multiplexes multiple streams over these connections. teor teor2345 at gmail dot com pgp 0xABFED1AC https://gist.github.com/teor2345/d033b8ce0a99adbc89c5 teor at blah dot im OTR D5BE4EC2 255D7585 F3874930 DB130265 7C9EBBC7

1 0

Re: [tor-dev] working back to socks_request_t
by teor 04 May '15

04 May '15

> Date: Sun, 3 May 2015 12:34:51 -0400 > From: CJ Ess <zxcvbn4038(a)gmail.com> > > On Sun, May 3, 2015 at 11:06 AM, teor <teor2345(a)gmail.com> wrote: > >>> Date: Sun, 3 May 2015 02:50:46 -0400 >>> From: CJ Ess <zxcvbn4038(a)gmail.com> >>> >>> So I'm doing a bit of an experiment, the idea being that if you have a >>> group of tor users sharing common infrastructure then its a slightly >>> different situation then one lone user, and you wantto emphasize that >>> resources should not be shared, caching should be minimal and >>> non-persistent, you need to keep usage from standing out, etc. … >>> So plan B is everyone involved runs their socks speaking browser on their >>> desktop/laptop, everyone runs a tor client on the same device as their >>> browser, we use the HTTPProxy/HTTPSProxy feature of the clients to navigate >>> the firewall, everyone uses their own credentials instead of having one ID >>> draw attention for high utilization, and the presence of >>> the Proxy-Authorization header takes care of any caching/session sharing >>> issues along the way. >>> >>> To make that work, the one question I have for tor-dev is if its possible >>> Here: >>> >>> https://github.com/torproject/tor/blob/24f170a11f59e26dec3a24d076b749c8acc7… >>> >>> To work back to the socks_req, so that I can pass through the username and >>> password to the upstream proxy instead of the one global username/password? >> >> Hi CJ, >> >> It sounds like you're looking for one of the HTTP(S)ProxyAuthenticator >> options - you can configure a different username and password in the torrc >> file on each client's desktop/laptop. >> >> … >> >> If these options aren't what you're looking for, can you explain what you >> want done with the SOCKS request in a bit more detail? > > So underlying idea in this case is to pass thru the proxy credentials from > the browser, so they don't have to be had coded in plain text in the tor > config - you exit the browser and the credential goes away (or maybe its > encrypted in the browser password manager), if you change your password you > don't have to go update the tor config and bounce it, if its a shared > device you don't have to reconfigure and possibly leave your credentials > around, etc. > > That sounds good conceptually, but to implement I somehow need to work back > from the connection_t passed into the function I mentioned to something > that has the socks info (circuit? associated edge connection?) I tried to > trace it though and nothing jumped out at me but maybe there is some type > casting happening and I'm missing it. The other option would be to pass the > info down through extra arguments or copied into extra field members. > Either way I'm speculating there might be a really simple way to do this > and worth the time writing up the question. If I get it working I'd be > happy to send in a patch to the this list. Hi CJ, There isn't a one-to-one correspondence between web browser SOCKS connections and Tor connections: 1. Each SOCKS connection results in a stream, and tor multiplexes multiple streams over multiple circuits through the same connection to the first hop in the path. 2. tor also initiates streams, such as directory fetches, without any corresponding SOCKS connection from the browser. 3. tor pre-emptively launches circuits, then uses them for client connections once a SOCKS request is received. 4. tor sometimes cannibalizes existing circuits by adding a hop to them. tor makes a connection to 1-3 guard nodes, and those connections carry the majority of the web browser SOCKS traffic. They are authenticated at tor launch, before any browser makes a request, and I'm pretty sure that's it for authentication from then on. If you could get authentication passthrough working, you'd still run into the following compatibility issues: 5. Tor Browser uses the input SOCKS username (I think) to manage stream-based isolation for different tabs, based on the URL. This would prevent you using Tor Browser with your modified version of tor, leaving you with a browser with fewer anonymity protections. 6. Some pluggable transports use the output proxy username to pass arguments. This would prevent you using pluggable transports with your modified version of tor. And the following anonymity / usage issues: 7. If each SOCKS connection has a corresponding circuit, and you identify the user who initiated the circuit, you make it easier to de-anonymise the user behind the circuit. 8. Given that tor can multiplex multiple users over each circuit, who is responsible for bandwidth over-use? If you're still keen on continuing, the closest that the socks authentication details get to where you need them is in conn->socks_request in circuit_get_open_circ_or_launch(). The call stack from then on is: circuit_launch_by_extend_info() circuit_establish_circuit() circuit_handle_first_hop() channel_connect_for_circuit() channel_connect() channel_tls_connect() connection_or_connect() Which initialises the or_connection_t struct "conn", then makes the connection to the proxy (if set). You could add the username and password to the conn data structure here. When the connection has finished connecting, the callback is: connection_or_finished_connecting() Which calls: connection_proxy_connect() Which could then use the username and password from the conn data structure, if they exist - otherwise, it could use the default set in the config. I still think you'll get the default authentication most of the time, because of tor's pre-emptive guard connections. When you reply, can you avoid top-posting? Some mail clients top-post by default. But it's easier to follow the conversation if replies come after the initial email. Feel free to reply in the middle of the email as well. teor teor2345 at gmail dot com pgp 0xABFED1AC https://gist.github.com/teor2345/d033b8ce0a99adbc89c5 teor at blah dot im OTR D5BE4EC2 255D7585 F3874930 DB130265 7C9EBBC7

2 1

Re: [tor-dev] design for a Tor router without anonymity compromises
by teor 04 May '15

04 May '15

> Date: Sun, 3 May 2015 16:23:08 -0700 > From: coderman <coderman(a)gmail.com> > >> I suggest also adding mandatory audit logging to the scope of the >> router software. In my opinion any and all state changes, whether >> automatic (Tor circuit change) or manual (administrator changing >> configuration) must be logged. > > this is an important detail; thank you for bringing it up. i will add > the expected run logging and troubleshooting logging output collected > on device and available to the owner via privacy director > administrative access. Some users will want as little logging as possible, as it can lead to privacy leaks if the device is compromised - may I suggest you turn it off by default? teor teor2345 at gmail dot com pgp 0xABFED1AC https://gist.github.com/teor2345/d033b8ce0a99adbc89c5 teor at blah dot im OTR D5BE4EC2 255D7585 F3874930 DB130265 7C9EBBC7

2 1

Damian's Status Report - April 2015
by Damian Johnson 04 May '15

04 May '15

Hi all! My April was split between SoP prep and Nyx. Changes for the later unfortunately aren't yet ready so this is gonna be a shorter status report than usual. -------------------------------------------------------------------------------- Tor Summer of Privacy -------------------------------------------------------------------------------- It's a thing! Of our twenty-two applicants sadly we only had funding for four, but all have great projects and we look forward to working with them this summer! https://blog.torproject.org/blog/tor-summer-privacy-projects Personally I'm particularly excited since this will be the first time I've mentored since 2012. Cristobal's working on a web dashboard for relay operators - think nyx, but ajax... https://leivaburto.github.io/sop-proposal/ -------------------------------------------------------------------------------- Few other noteworthy things were... * Built on David's hidden service descriptor work by adding a get_hidden_service_descriptor() method to Stem. Next up is support for Yawning's ephemeral hidden services... * Sambuddha revised Stem's Twitter example to support Twitter's v2 API. (#9003) * Stem test for installation via our setup.py. (#15258) Cheers! -Damian

1 0

Re: [tor-dev] working back to socks_request_t
by teor 03 May '15

03 May '15

> Date: Sun, 3 May 2015 02:50:46 -0400 > From: CJ Ess <zxcvbn4038(a)gmail.com> > > So I'm doing a bit of an experiment, the idea being that if you have a > group of tor users sharing common infrastructure then its a slightly > different situation then one lone user, and you wantto emphasize that > resources should not be shared, caching should be minimal and > non-persistent, you need to keep usage from standing out, etc. The problem > with my original idea is that everything that does HTTP <> SOCKS is one or > two decades old, and draws a lot of attention because it forks for every > connection or is some strange process that nobody has ever seen before. > > So plan B is everyone involved runs their socks speaking browser on their > desktop/laptop, everyone runs a tor client on the same device as their > browser, we use the HTTPProxy/HTTPSProxy feature of the clients to navigate > the firewall, everyone uses their own credentials instead of having one ID > draw attention for high utilization, and the presence of > the Proxy-Authorization header takes care of any caching/session sharing > issues along the way. > > To make that work, the one question I have for tor-dev is if its possible > Here: > > https://github.com/torproject/tor/blob/24f170a11f59e26dec3a24d076b749c8acc7… > > To work back to the socks_req, so that I can pass through the username and > password to the upstream proxy instead of the one global username/password? Hi CJ, It sounds like you're looking for one of the HTTP(S)ProxyAuthenticator options - you can configure a different username and password in the torrc file on each client's desktop/laptop. If you are going to run a SOCKS-speaking browser, why not run the Tor Browser? It does a lot more to protect your anonymity than most. From the tor manual page: HTTPProxyAuthenticator username:password If defined, Tor will use this username:password for Basic HTTP proxy authentication, as in RFC 2617. This is currently the only form of HTTP proxy authentication that Tor supports; feel free to submit a patch if you want it to support others. HTTPSProxyAuthenticator username:password If defined, Tor will use this username:password for Basic HTTPS proxy authentication, as in RFC 2617. This is currently the only form of HTTPS proxy authentication that Tor supports; feel free to submit a patch if you want it to support others. If these options aren't what you're looking for, can you explain what you want done with the SOCKS request in a bit more detail? teor teor2345 at gmail dot com pgp 0xABFED1AC https://gist.github.com/teor2345/d033b8ce0a99adbc89c5 teor at blah dot im OTR D5BE4EC2 255D7585 F3874930 DB130265 7C9EBBC7

2 1

working back to socks_request_t
by CJ Ess 03 May '15

03 May '15

So I'm doing a bit of an experiment, the idea being that if you have a group of tor users sharing common infrastructure then its a slightly different situation then one lone user, and you wantto emphasize that resources should not be shared, caching should be minimal and non-persistent, you need to keep usage from standing out, etc. The problem with my original idea is that everything that does HTTP <> SOCKS is one or two decades old, and draws a lot of attention because it forks for every connection or is some strange process that nobody has ever seen before. So plan B is everyone involved runs their socks speaking browser on their desktop/laptop, everyone runs a tor client on the same device as their browser, we use the HTTPProxy/HTTPSProxy feature of the clients to navigate the firewall, everyone uses their own credentials instead of having one ID draw attention for high utilization, and the presence of the Proxy-Authorization header takes care of any caching/session sharing issues along the way. To make that work, the one question I have for tor-dev is if its possible Here: https://github.com/torproject/tor/blob/24f170a11f59e26dec3a24d076b749c8acc7… To work back to the socks_req, so that I can pass through the username and password to the upstream proxy instead of the one global username/password?

1 0

onionoo: historic details.json data
by Nusenu 03 May '15

03 May '15

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA512 Hi Karsten, can one download historic onionoo documents (details.json) archived somewhere or would one have to setup onionoo + feed old data into it to achieve that? thanks, Nusenu -----BEGIN PGP SIGNATURE----- iQIcBAEBCgAGBQJVIaqIAAoJEFv7XvVCELh0OQgP/3FiBM3uMifPW9A0u8sMuh1E 7iFDmU52MNmkeToBUnuxuUK+7ig96tz/p5Ds6OyaAtC0Ld/IzRlIU5ELjqDwRqMD LPaafWm21QHO8y4fgW5FfGDfFTeLL/b3Lm1E9VmJ8wpOsgpzNnBMUoeDrpDyE3Ow zHXM/IfCSROqmO9lOeU6YGQTwInucc0HSveAWQ9i/x6iks9kMguzGnuWTxgfwA61 RQT4KUKZI+b5w//8V3/IpR5jlQ2N4OJrURpVhaKKNYMy+mqOfl5/a1ZPyQhhDaxj eLfg0qQmsk9IJY/6dZa0XogsmAzRCcEO/Ha0b0ZoiWs+VFo97cczoWdVWHoKpsyS X/qeehkOS13l22mw8va+yuj4zoRBrgK2chzcG2iPgVBeTu1H7B+x57PQrQXaCdaf rq8wwlS7mgaMQ6mgHyOQ1niZDTnvJaY0e3gVsgucR7YOTdALxnnHhQkyNboCWq/E 180SetWZfIHbL3cV38DRxYKzijwSu1gqmkwTJWffylb5SMja1TdW9cCq/RzidqyF v7vLXHQwmrRQ0XOy9UULe5XtU0Fe4YCm/zMNzBHHGIrePd664mYKqqXP2dlempwb BbbVgMLuFC2sy2eqxtwyQrrDUoWrEovUxg/BeoKO8RYXzNhPtha8eQkAhexhNux2 2mosesv/fooaQnt17qAo =q6LM -----END PGP SIGNATURE-----

6 16