- tor-dev - lists.torproject.org

[Measurement Team] Next IRC meeting happens on Thursday, Sep 24, 14:00 UTC in #tor-dev
by Karsten Loesing 23 Sep '15

23 Sep '15

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hello list, This is an announcement/reminder that there will be an IRC meeting of the Measurement Team on Thursday, September 24, 2015, 14:00 UTC in #tor-dev https://www.timeanddate.com/worldclock/fixedtime.html?iso=20150924T14 This will be the last IRC meeting before the Berlin dev meeting. Maybe we'll hold an in-person team meeting next week at the same time. We'll resume the IRC meeting format the week after. Talk to you tomorrow! All the best, Karsten -----BEGIN PGP SIGNATURE----- Version: GnuPG v1 Comment: GPGTools - http://gpgtools.org iQEcBAEBAgAGBQJWArn+AAoJEJD5dJfVqbCrm/YIAMTJvTAPpmbg0B85By/GfzLL mtTRB108vCLl+mkAOOkefo3o0u7RXcGlWeuSIUsqfoS4AcNpKVTF8dMrOyxCm80c nIOcwC8QX9gHGNXJXKIXaDCSeZ5MRcSLa2gLpllc/M844LbrfILIVQiF9AKafEiM DhJ7Hw41E9dNma1Qfm8LQXYwd9Sp1DLjRIFuGs3hOurZH9b2H+9Z9S+8M4D0I/hM FDyDo11MbRHEWcombVAX6IdkCUBjcZnjOUIu3tvca3pvf67or3gsRKI1GCLIRwan /Ff5TxDPkltTQwmU2E4xLpuI09baKcRpdiInkrsg1E6JfH1tT2VESJxNYb+1STM= =QMdU -----END PGP SIGNATURE-----

1 0

Looking for resources
by TOR USER 21 Sep '15

21 Sep '15

Hello, I was going through the Tor proposals which have now been closed. I have been looking for discussion regarding those proposals (here - https://gitweb.torproject.org/torspec.git) Could you link me to some discussion threads because I couldn't find much on them? Thank you

2 1

Onionoo as_number / as_name data
by Gareth Llewellyn 19 Sep '15

19 Sep '15

Hello, How often does Onionoo update its ASN information? Relays 518FF8708698E1DA09C823C36D35DF89A2CAD956, F92B3CB9BBE0CB22409843FB1AE4DBCD5EFAC835 & 707C1B61AC72227B34487B56D04BAA3BA1179CE8 are relatively new and whilst their IPs recently received updated GeoIP information (from Unassigned to UK) the ASN (AS28715) and ISP name aren't showing up. Gareth

2 1

Summary of meek's costs, August 2015
by David Fifield 18 Sep '15

18 Sep '15

Here's the summary of meek's CDN fees for August 2015. App Engine + Amazon + Azure = total by month February 2014 $0.09 + -- + -- = $0.09 March 2014 $0.00 + -- + -- = $0.00 April 2014 $0.73 + -- + -- = $0.73 May 2014 $0.69 + -- + -- = $0.69 June 2014 $0.65 + -- + -- = $0.65 July 2014 $0.56 + $0.00 + -- = $0.56 August 2014 $1.56 + $3.10 + -- = $4.66 September 2014 $4.02 + $4.59 + $0.00 = $8.61 October 2014 $40.85 + $130.29 + $0.00 = $171.14 November 2014 $224.67 + $362.60 + $0.00 = $587.27 December 2014 $326.81 + $417.31 + $0.00 = $744.12 January 2015 $464.37 + $669.02 + $0.00 = $1133.39 February 2015 $650.53 + $604.83 + $0.00 = $1255.36 March 2015 $690.29 + $815.68 + $0.00 = $1505.97 April 2015 $886.43 + $785.37 + $0.00 = $1671.80 May 2015 $871.64 + $896.39 + $0.00 = $1768.03 June 2015 $601.83 + $820.00 + $0.00 = $1421.83 July 2015 $732.01 + $837.08 + $0.00 = $1569.09 August 2015 $656.76 + $819.59 + $154.89 = $1631.24 -- total by CDN $6154.49 + $7165.85 + $154.89 = $13475.23 grand total https://metrics.torproject.org/userstats-bridge-transport.html?graph=userst… This month there was a big rebound as a temporary outage (not related to censorship) with meek-azure was resolved. The user count is now almost as high as it has ever been, at about 5000 simultaneous users. This month is the first time we have actual usage numbers from Azure. They told us what the cost would have been, but did not actually charge us. This month (September) is the last month on the free grant we have for Azure. After that, we will probably have to throttle Azure just like the other backends. If you want to help make domain fronting more efficient and future-proof, a good place to direct your attention is to this ticket, which aims to start using domain fronting as a cheap, low-volume interface to BridgeDB, instead of as a bulk data transfer channel: https://bugs.torproject.org/16650 If you want to set up your own bridge and CDN instance without rate limiting, I can help you do it. Here are some docs to look at: https://trac.torproject.org/projects/tor/wiki/doc/meek#Howtorunameek-server… https://trac.torproject.org/projects/tor/wiki/doc/meek#GoogleAppEngine https://trac.torproject.org/projects/tor/wiki/doc/meek#AmazonCloudFront https://trac.torproject.org/projects/tor/wiki/doc/meek#MicrosoftAzure == App Engine a.k.a. meek-google == Here is how the Google costs broke down: 2920 GB $350.38 6128 instance hours $306.38 Compared to the previous month: 2934 GB $352.11 7598 instance hours $379.90 https://globe.torproject.org/#/bridge/88F745840F47CE0C6A4FE61D827950B06F9E4… == Amazon a.k.a. meek-amazon == Asia Pacific (Singapore) 150M requests $180.30 1191 GB $157.13 Asia Pacific (Sydney) 1M requests $2.26 7 GB $0.98 Asia Pacific (Tokyo) 28M requests $34.17 177 GB $22.80 EU (Ireland) 174M requests $209.36 905 GB $70.40 South America (Sao Paulo) 7M requests $16.87 49 GB $11.68 US East (Northern Virginia) 75M requests $75.07 490 GB $38.60 -- total 437M requests $518.03 2821 GB $301.59 https://globe.torproject.org/#/bridge/3FD131B74D9A96190B1EE5D31E91757FADA1A… == Azure a.k.a. meek-azure == This is the first month where we have actual (not estimated) usage numbers, as a result of the new kind of account we're using. Unfortunately (also as a result of the account transition), meek-azure was broken until August 14, so these number really only represent half a month. Zone 1 1062 GB $92.40 Zone 2 453 GB $62.49 -- total 1515 GB $154.89 https://globe.torproject.org/#/bridge/AA033EEB61601B2B7312D89B62AAA23DC3ED8… Earlier reports in this series: https://lists.torproject.org/pipermail/tor-dev/2014-August/007429.html https://lists.torproject.org/pipermail/tor-dev/2014-October/007576.html https://lists.torproject.org/pipermail/tor-dev/2014-November/007716.html https://lists.torproject.org/pipermail/tor-dev/2014-December/007916.html https://lists.torproject.org/pipermail/tor-dev/2015-January/008082.html https://lists.torproject.org/pipermail/tor-dev/2015-February/008235.html https://lists.torproject.org/pipermail/tor-dev/2015-March/008427.html https://lists.torproject.org/pipermail/tor-dev/2015-April/008596.html https://lists.torproject.org/pipermail/tor-dev/2015-May/008767.html https://lists.torproject.org/pipermail/tor-dev/2015-June/008932.html https://lists.torproject.org/pipermail/tor-dev/2015-July/009030.html https://lists.torproject.org/pipermail/tor-dev/2015-August/009213.html

1 0

Re: [tor-dev] Proposal: Single onion services
by tordev123＠Safe-mail.net 18 Sep '15

18 Sep '15

Doesn't your proposal imply that you are turning all relays into exit-nodes lite? The last relay in the path will know what service you are connecting to (at least if that service is hosted with a unique relay), right? Have you considered all the implications?

7 7

Re: [tor-dev] Towards a new version of the PT spec...
by Adam Pritchard 18 Sep '15

18 Sep '15

At Psiphon we often discuss (and get asked about) using Tor's pluggable transports directly. The cost/benefit balance hasn't yet been in favour of doing this, but if there's discussion of a big PT revamp... maybe Psiphon should indicate how the cost side of the balance could come down for us. We're obviously not a priority for what Tor does with PTs, but there's surely no harm in us mentioning our wishlist. So... What would be best for us is if PTs were written in Go and implemented the net.Conn[1] interface. We have had good results with the composability of net.Conn implementations: an obfuscated SSH net.Conn on top of a meek net.Conn[2] on top of a upstream proxy net.Conn[3] on top of a TCPConn net.Conn[4]. Layering in a PT net.Conn would be sane and clean and reasonably easy. Conversely: Python is difficult-to-impossible due to runtime distribution. Separate executables are difficult-to-impossible due to Android PIE requirements and distribution size bloat. Anyway, if this is of any interest we can discuss it further. (Note: Probably Lantern people are reading this too. And probably they would benefit from the same things we would, since their architecture is similar to ours.) [1]: https://golang.org/pkg/net/#Conn [2]: https://github.com/Psiphon-Labs/psiphon-tunnel-core/blob/master/psiphon/mee… [3]: https://github.com/Psiphon-Labs/psiphon-tunnel-core/tree/master/psiphon/ups… [4]: https://golang.org/pkg/net/#TCPConn Adam Pritchard Psiphon Inc. > Date: Mon, 7 Sep 2015 22:45:52 +0000 > From: Yawning Angel <yawning(a)schwanenlied.me> > To: tor-dev(a)lists.torproject.org > Subject: [tor-dev] Towards a new version of the PT spec... > Message-ID: <20150907224552.6959bcbc(a)schwanenlied.me> > Content-Type: text/plain; charset="us-ascii" > > So, we currently have a Pluggable Transport (PT) spec, and it kind-of > sort-of works (The documentation is a mess that I'm working on > cleaning up, but it's an orthogonal issue for how well it works). > > There are a number of problems with the current PT spec that require > breaking backward compatibility to fix, so eventually I would like to > do so. > > I'm soliciting input on what people would also like to see in a > (currently hypothetical) PT spec 2.0 beyond what I already have in mind: > > MUST haves: > * Support dual stack Bridges correctly (Multiple server endpoints per > transport) > * Increase the argument space beyond 510 bytes (Prop. #227). > * Mandatory ExtORPort support (currently optional, but metrics are > good). > * Centralized logging by the calling process (Probably via stderr). > * AF_UNIX support where sensible for better sandboxing. > > MIGHT haves: > * Rename the env vars to not start with "TOR_PT". Some people claim > that this is a good idea (I think it is stupid and cosmetic). > * Ability to force at least clients to stop network activity without > tearing the PT down. > * Deprecate SOCKS4a, and make SOCKS5 mandatory for clients. > > UNLIKELY: > * Specify an interface for where fork()/exec() isn't possible (iOS). > I don't think this is makes sense because it is probably too > platform/caller specific. > * Allow operating both as a client and a server simultaneously. I > don't see a problem with running 2 copies of something for this > use case. > > I probably missed some things. If people have strong opinions about > this, do reply, otherwise I *will* design something that I like, which > will not include what other people want. > > Regards, > > -- > Yawning Angel -- Adam Pritchard Psiphon Inc.

3 2

How long does it take for 25/50/75% of relays to update to a new Tor version?
by Karsten Loesing 18 Sep '15

18 Sep '15

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hello Sebastian and list, Sebastian again suggested a fun task for the latest measurement team 1-1-1 task exchange round [0] that I picked up. He asked the following question: When a new Tor version is released, how long does it take for 25, 50, 75% of all relays to update? How often do relays update in general? Is there an effect when for example Debian updates its packages? It would help me to get an idea what I need to answer these questions :-) "update" means you're running, then you go down, then within a few minutes you're back up again with a changed version Here's what I came up with: I suggest we use two data sets as input for this task. The first data set consists of archived consensuses, available at: https://collector.torproject.org/archive/relay-descriptors/consensuses/ We'll need "valid-after" and "server-versions" from the header and "v" and "w" lines from all contained entries. Example: valid-after 2015-09-18 08:00:00 server-versions 0.2.4.23,0.2.4.24,0.2.4.25,0.2.4.26,0.2.4.27,0.2.5.8-rc,0.2.5.9-rc,0.2.5.10,0.2.5.11,0.2.5.12,0.2.6.5-rc,0.2.6.6,0.2.6.7,0.2.6.8,0.2.6.9,0.2.6.10,0.2.7.1-alpha,0.2.7.2-alpha v Tor 0.2.7.2-alpha-dev w Bandwidth=450 v Tor 0.2.6.10 w Bandwidth=23 [...] We can use those lines to *count* the number of running relays with a given version string, and we can sum up the *fraction* of consensus weights of relays running a given version string. Both can be useful, though I suspect the latter to be more meaningful. We should aggregate these consensus parts into daily statistics to make them easier to handle. We can do that by keeping counters per (date, version) tuple and per (date) for a) number of relays and b) consensus weight, and then we divide the (date, version) numbers by the (date) numbers to obtain averages. We'd also determine whether a version is recommended, whether it's the latest recommended version in its series, and whether it's the latest recommended stable version. The result would be a .csv file like the following (example data!): date,version,recommended,series,stable,count,fraction 2015-09-18,0.2.6.10,TRUE,TRUE,TRUE,0.387,0.451 2015-09-18,0.2.6.9,TRUE,FALSE,FALSE,0.063,0.0102 2015-09-18,0.2.6.8,TRUE,FALSE,FALSE,0.021,0.001 2015-09-18,0.2.6.7,TRUE,FALSE,FALSE,0.016,0.009 Read this as: "On 2015-09-18, version 0.2.6.10, which was both recommended and the latest in its series, was run by 38.7% of relays by count or 45.1% by consensus weight." (again, example data!) You could then draw graphs similar to https://metrics.torproject.org/versions.html, but with a percent scale on y. Though I'm not sure how readable those graphs would be with a few dozen lines in them. Still, they might be useful to explore the data set. The second data set that would be useful here is events related to versions. The following events come to mind: - tagged as version in Git (use a command similar to [1]), - tagged as alpha, beta, rc, or stable (look at version string), - first/last recommended by directory authorities for relays (parse from consensuses), - last recommended as newest version in a series or newest stable version (also parse from consensuses), or - first released in Debian stable (unclear how to obtain those dates). A possible data format would be: date,version,event 2015-07-12,0.2.6.10,git_tag 2015-06-10,0.2.6.9,git_tag 2015-05-19,0.2.6.8,git_tag 2015-04-06,0.2.6.7,git_tag For exploratory purposes, you could add these as colored vertical lines to the graphs above. Of course, adding more elements to a probably already overloaded graph doesn't exactly make it easier to understand. But if you limit the x axis on just a month or two, it might be useful. So, regarding your original question: "How long does it take for 25/50/75% of relays to update to a new Tor version?" I'm not sure whether that's really the best question to ask/answer here. Some versions will never be deployed on 75%, 50%, or even 25% of relays, because a new version was released shortly after. But that doesn't mean that the previous version was bad. This question also weights all versions the same, though in reality some releases are more important than others. If you get one data point for each version and then aggregate them by taking the average, what exactly does that tell you? I think that question could be rephrased to: "How long does it take for 25/50/75% of relays to update to a new Tor version *series*?" And I think that's something you could answer with the first data set above. But there are other interesting questions you could answer with the two data sets. For example, - What fraction of relays (by number or consensus weight) is running (un-)recommended versions? - What fraction of relays (by number of consensus weight) is (not) running the latest version in a series or latest stable version? Those could again be answered by a graph that uses the first data set, dates on x and fractions on y, and possibly events from the second data set as vertical lines. Explore, explore. There, the hour is over ("It took an hour to write, I thought it would take an hour to read.", Fry, Futurama). Hope this is useful in any way. And maybe others have good/better ideas for you as well? If you come up with some interesting answers, please post them here. All the best, Karsten [0] 1-1-1 task exchange: you get 1 minute to describe a task that would take somebody else roughly 1 hour and that they will do for you within 1 week (review a document, write some analysis code, fix a small bug, etc.; better come prepared to get the most out of this; give 1, take 1) [1] `git log --tags --simplify-by-decoration --pretty="format:%ci %d" | grep "tag: tor" | less` -----BEGIN PGP SIGNATURE----- Version: GnuPG v1 Comment: GPGTools - http://gpgtools.org iQEcBAEBAgAGBQJV++DHAAoJEJD5dJfVqbCr+sYH/1nViWWJgie+BD6Mo+9pC+g9 qEfMQ+NBvYWr3/SFcn/zWjydgZf+YPVXcVsOA1B1YcLX2s8OY1cWM5t5cDCstvfh /vOYmsDw1nddBh8VQxggYPkTLfEWRRSB0IeMiFWWUdVVKkcepOxgCfXMWJlBEAgG 8EtarcTkxSl1YMrurH3iuL8PrHGRiGf+mEKOZOMFtpmkSii/bWpO7BOAEyOXC7Ib EX2OouhCvae5X3Lyp2CnlQZLSrRGtAFV8sQUw5gcQb5/nouk38M7/ayrkBW/H3Ng qM9k0Y9EHAs1I1rn1ZY5lfGMKZ0WR+1NNCQ3mdsBSF5Qah01i1By7uk2FiH0e4U= =H6/Z -----END PGP SIGNATURE-----

1 0

List Administrivia [...encrypted onion services...]
by grarpamp 17 Sep '15

17 Sep '15

fabio, sky, et al... Please stop top quoting massive amounts of useless text that everyone's already seen before. It's seriously annoying to peoples workflow trying to figure out what youre replying to and if youve inlined other stuff below. And it needlessly bloats peoples mailboxes and thus search results. Trim the original to relavent bits and write your reply below that in context. Thanks, users everywhere.

1 0

Proposal: End-to-end encrypted onion services for non-Tor clients
by Donncha O'Cearbhaill 17 Sep '15

17 Sep '15

Hello all, I have been thinking about ideas to make Tor hidden services more available and secure for non-Tor users. Inline I've included a draft proposal which describes an end-to-end encrypted Tor2Web-like system. I'm really interested in hearing any suggestions, comments or criticism about this proposal. In particular I'd like to know if the trust requirements for the entry proxies and resolvers seem reasonable? Does this proposal make sense and is it something worth implementing? Thank you to David Stainton and Leif Ryge for your feedback on this proposal. Kind Regards, Donncha --- Filename: xxx-encrypted-onion-services-for-non-tor-clients.txt Title: End-to-end encrypted onion services for non-Tor clients Author: Donncha Ó Cearbhaill Created: 22-Aug-2015 Status: Draft 1. Overview: This proposal describes a system to allow non-Tor users to securely access location-hidden services via a domain name chosen by the hidden service operator. 2. Motivation: Tor hidden services have typically been used to provide anonymity to both services and clients. There are however use cases where a publisher or content provider requires anonymity but their users do not. The publishers typically would like their content to be available securely and to the widest possible audience. Tor2Web currently allows non-Tor users to access hidden services but it has a number of limitations. Tor2Web deployments are limited due to requirement to entrust Tor2Web service providers with users location information and the content of their requests. The onion subdomain based addressing in Tor2Web poses a usability issue and can be unwieldy for users without providing any self-authenticating properties. Tor2Web can be configured with a custom domain for an individual hidden service but this requires manual configuration and additional infrastructure. Ideally non-Tor users should be able to access location-hidden services under a standard domain name with end-to-end encryption from the client's browser to the hidden service web server. The system should be decentralized and it should not require the hidden service operator to run any clearnet systems. 3. Proposal: At a high level this proposal specifies a system of TCP entry proxies which transparently proxy the TLS connections of non-Tor clients to a hidden service. A client who requests a domain name such as https://myblog.com should get connected to a hidden service endpoint which has a valid SSL certificate for myblog.com. 3.1. Hidden Service configuration A hidden service operator obtains a public domain name and a corresponding CA signed SSL certificate. The operator configures the hidden service web server with the SSL certificate for the external domain. The domain name is pointed at an authoritative nameserver provider. The operator should also set a TXT DNS record containing the onion address of their hidden service. 3.2. Entry Proxy Modern web browsers send a Server Name Indication (SNI) field in the initial ClientHello of the TLS handshake. The entry proxy reads this field to determine the clients destination domain. The proxy performs a DNS lookup at that domain for a TXT record which specifies the corresponding onion address for the provided domain. The TLS entry proxy connects to the onion service via a Tor2Web-type circuit and then begins transparently proxying TLS traffic between the clients browser and the destination onion service. 3.3. Resolving Nameserver The key component of this system is a DNS name server which can direct clients to an online entry proxy via a round-robin type system. In a basic solution the onion service operator could manually specify one or more entry proxies in the A records for their domain on their existing authoritative DNS provider. In practice entry proxy churn would result in changing set of entry proxies. The nameserver will need regularly update its set of online entry proxies an remove proxies which are malfunctioning, malicious or otherwise unusable. The resolver may run a scanner to check its known proxies or load a list from an external service. Additionally the resolver should detect if an entry proxy blacklists a domain for which it is responsible and avoid routing clients to that entry proxy. It is expected that independent service providers will run their own domain->onion resolving nameserver in diverse jurisdictions as free or paid services. 4. Implementation: The entry proxy component could be implemented as part of the current Tor relay code base. Integration directly within Tor would allow use of the existing network consensus and bandwidth measurement systems to be used to discover available entry proxies. It would also allow for malicious entry proxies to be blacklisted. Alternatively the entry proxy could be implemented in the existing Tor2Web software or as a standalone software package. Implementing outside of Tor would be faster and it would avoid the risk of losing Tor relay capacity as a result of legal threats to the entry proxies. The resolving nameserver is the most complicated component of this system. The component will eventually require a DNS server, a management interface, and a set of network monitoring tools. 5. Security and resiliency implications: 5.1. Availability Attacks: Adversaries can attack the availability of a publicly-proxied hidden service at a number of levels: * Censorship or shutdown of entry proxy: Attacks on individual entry proxies are mitigated by performing DNS based round-robin between many online entry proxies. The Resolver system should be able to quickly remove entry proxies which misbehave or which go offline. * Censorship or seizure of the hidden service public domain: DNS based blocks are widely deployed for censorship and may be difficult to avoid. Domain registrars can also be forced to suspend domain names. Service operators should considering running their service under a TLD which is less vulnerable to these type of coercive threats. * Takedown of a nameserver provider: Multiple resolving nameservers can be configure for each forwarded domain. Using nameservers maintained by different providers can provide resilience to attacks against a single nameserver provider. 5.2. Security Attacks: Entry proxies and exit relays have a similar ability to monitor and interfere with client traffic. This is greater risk of targeted interference from entry proxies as they can also determine the client's network location. * Man-in-the-middle HTTP connections: Entry proxies have the ability to man-in-the-middle HTTP connections. Service operators should send HSTS header to force clients to automatically use TLS for all future connections. * TLS man-in-the-middle with CA-signed certificate Some commercial CA cert providers allow for domain ownership to be validated by providing a file over HTTP at the domain. A malicious entry proxy could successfully obtain a CA-signed certificate from one of these certificate authorities. Service operators can minimize their exposure to this type of attack by using HPKP headers to limit the set of valid certificate authorities for their domain.

5 5

multiple relay identities on a single IP:port, bug or feature?
by nusenu 16 Sep '15

16 Sep '15

Hi, in the last two days someone started generating a steady amount of new relay fingerprints on a single IP:port (2 per hour, actually a lot more than that but only to make it into the consensus) [1]. I'm surprised that actually both of them end up in the consensus. example: https://collector.torproject.org/recent/relay-descriptors/consensuses/2015-… search for "67.173.119.40 51256" - you will find it twice. Does that make sense or is this a bug? btw: Why do dir auths include non-running relays in their votes (flag line is "s" only), instead of not voting for them at all? thanks! [1] http://article.gmane.org/gmane.network.onion-routing.ornetradar/113

2 1