- tor-dev - lists.torproject.org

Onionoo as_number / as_name data
by Gareth Llewellyn 19 Sep '15

19 Sep '15

Hello, How often does Onionoo update its ASN information? Relays 518FF8708698E1DA09C823C36D35DF89A2CAD956, F92B3CB9BBE0CB22409843FB1AE4DBCD5EFAC835 & 707C1B61AC72227B34487B56D04BAA3BA1179CE8 are relatively new and whilst their IPs recently received updated GeoIP information (from Unassigned to UK) the ASN (AS28715) and ISP name aren't showing up. Gareth

2 1

Summary of meek's costs, August 2015
by David Fifield 19 Sep '15

19 Sep '15

Here's the summary of meek's CDN fees for August 2015. App Engine + Amazon + Azure = total by month February 2014 $0.09 + -- + -- = $0.09 March 2014 $0.00 + -- + -- = $0.00 April 2014 $0.73 + -- + -- = $0.73 May 2014 $0.69 + -- + -- = $0.69 June 2014 $0.65 + -- + -- = $0.65 July 2014 $0.56 + $0.00 + -- = $0.56 August 2014 $1.56 + $3.10 + -- = $4.66 September 2014 $4.02 + $4.59 + $0.00 = $8.61 October 2014 $40.85 + $130.29 + $0.00 = $171.14 November 2014 $224.67 + $362.60 + $0.00 = $587.27 December 2014 $326.81 + $417.31 + $0.00 = $744.12 January 2015 $464.37 + $669.02 + $0.00 = $1133.39 February 2015 $650.53 + $604.83 + $0.00 = $1255.36 March 2015 $690.29 + $815.68 + $0.00 = $1505.97 April 2015 $886.43 + $785.37 + $0.00 = $1671.80 May 2015 $871.64 + $896.39 + $0.00 = $1768.03 June 2015 $601.83 + $820.00 + $0.00 = $1421.83 July 2015 $732.01 + $837.08 + $0.00 = $1569.09 August 2015 $656.76 + $819.59 + $154.89 = $1631.24 -- total by CDN $6154.49 + $7165.85 + $154.89 = $13475.23 grand total https://metrics.torproject.org/userstats-bridge-transport.html?graph=userst… This month there was a big rebound as a temporary outage (not related to censorship) with meek-azure was resolved. The user count is now almost as high as it has ever been, at about 5000 simultaneous users. This month is the first time we have actual usage numbers from Azure. They told us what the cost would have been, but did not actually charge us. This month (September) is the last month on the free grant we have for Azure. After that, we will probably have to throttle Azure just like the other backends. If you want to help make domain fronting more efficient and future-proof, a good place to direct your attention is to this ticket, which aims to start using domain fronting as a cheap, low-volume interface to BridgeDB, instead of as a bulk data transfer channel: https://bugs.torproject.org/16650 If you want to set up your own bridge and CDN instance without rate limiting, I can help you do it. Here are some docs to look at: https://trac.torproject.org/projects/tor/wiki/doc/meek#Howtorunameek-server… https://trac.torproject.org/projects/tor/wiki/doc/meek#GoogleAppEngine https://trac.torproject.org/projects/tor/wiki/doc/meek#AmazonCloudFront https://trac.torproject.org/projects/tor/wiki/doc/meek#MicrosoftAzure == App Engine a.k.a. meek-google == Here is how the Google costs broke down: 2920 GB $350.38 6128 instance hours $306.38 Compared to the previous month: 2934 GB $352.11 7598 instance hours $379.90 https://globe.torproject.org/#/bridge/88F745840F47CE0C6A4FE61D827950B06F9E4… == Amazon a.k.a. meek-amazon == Asia Pacific (Singapore) 150M requests $180.30 1191 GB $157.13 Asia Pacific (Sydney) 1M requests $2.26 7 GB $0.98 Asia Pacific (Tokyo) 28M requests $34.17 177 GB $22.80 EU (Ireland) 174M requests $209.36 905 GB $70.40 South America (Sao Paulo) 7M requests $16.87 49 GB $11.68 US East (Northern Virginia) 75M requests $75.07 490 GB $38.60 -- total 437M requests $518.03 2821 GB $301.59 https://globe.torproject.org/#/bridge/3FD131B74D9A96190B1EE5D31E91757FADA1A… == Azure a.k.a. meek-azure == This is the first month where we have actual (not estimated) usage numbers, as a result of the new kind of account we're using. Unfortunately (also as a result of the account transition), meek-azure was broken until August 14, so these number really only represent half a month. Zone 1 1062 GB $92.40 Zone 2 453 GB $62.49 -- total 1515 GB $154.89 https://globe.torproject.org/#/bridge/AA033EEB61601B2B7312D89B62AAA23DC3ED8… Earlier reports in this series: https://lists.torproject.org/pipermail/tor-dev/2014-August/007429.html https://lists.torproject.org/pipermail/tor-dev/2014-October/007576.html https://lists.torproject.org/pipermail/tor-dev/2014-November/007716.html https://lists.torproject.org/pipermail/tor-dev/2014-December/007916.html https://lists.torproject.org/pipermail/tor-dev/2015-January/008082.html https://lists.torproject.org/pipermail/tor-dev/2015-February/008235.html https://lists.torproject.org/pipermail/tor-dev/2015-March/008427.html https://lists.torproject.org/pipermail/tor-dev/2015-April/008596.html https://lists.torproject.org/pipermail/tor-dev/2015-May/008767.html https://lists.torproject.org/pipermail/tor-dev/2015-June/008932.html https://lists.torproject.org/pipermail/tor-dev/2015-July/009030.html https://lists.torproject.org/pipermail/tor-dev/2015-August/009213.html

1 0

Re: [tor-dev] Proposal: Single onion services
by tordev123＠Safe-mail.net 19 Sep '15

19 Sep '15

Doesn't your proposal imply that you are turning all relays into exit-nodes lite? The last relay in the path will know what service you are connecting to (at least if that service is hosted with a unique relay), right? Have you considered all the implications?

7 7

Re: [tor-dev] Towards a new version of the PT spec...
by Adam Pritchard 18 Sep '15

18 Sep '15

At Psiphon we often discuss (and get asked about) using Tor's pluggable transports directly. The cost/benefit balance hasn't yet been in favour of doing this, but if there's discussion of a big PT revamp... maybe Psiphon should indicate how the cost side of the balance could come down for us. We're obviously not a priority for what Tor does with PTs, but there's surely no harm in us mentioning our wishlist. So... What would be best for us is if PTs were written in Go and implemented the net.Conn[1] interface. We have had good results with the composability of net.Conn implementations: an obfuscated SSH net.Conn on top of a meek net.Conn[2] on top of a upstream proxy net.Conn[3] on top of a TCPConn net.Conn[4]. Layering in a PT net.Conn would be sane and clean and reasonably easy. Conversely: Python is difficult-to-impossible due to runtime distribution. Separate executables are difficult-to-impossible due to Android PIE requirements and distribution size bloat. Anyway, if this is of any interest we can discuss it further. (Note: Probably Lantern people are reading this too. And probably they would benefit from the same things we would, since their architecture is similar to ours.) [1]: https://golang.org/pkg/net/#Conn [2]: https://github.com/Psiphon-Labs/psiphon-tunnel-core/blob/master/psiphon/mee… [3]: https://github.com/Psiphon-Labs/psiphon-tunnel-core/tree/master/psiphon/ups… [4]: https://golang.org/pkg/net/#TCPConn Adam Pritchard Psiphon Inc. > Date: Mon, 7 Sep 2015 22:45:52 +0000 > From: Yawning Angel <yawning(a)schwanenlied.me> > To: tor-dev(a)lists.torproject.org > Subject: [tor-dev] Towards a new version of the PT spec... > Message-ID: <20150907224552.6959bcbc(a)schwanenlied.me> > Content-Type: text/plain; charset="us-ascii" > > So, we currently have a Pluggable Transport (PT) spec, and it kind-of > sort-of works (The documentation is a mess that I'm working on > cleaning up, but it's an orthogonal issue for how well it works). > > There are a number of problems with the current PT spec that require > breaking backward compatibility to fix, so eventually I would like to > do so. > > I'm soliciting input on what people would also like to see in a > (currently hypothetical) PT spec 2.0 beyond what I already have in mind: > > MUST haves: > * Support dual stack Bridges correctly (Multiple server endpoints per > transport) > * Increase the argument space beyond 510 bytes (Prop. #227). > * Mandatory ExtORPort support (currently optional, but metrics are > good). > * Centralized logging by the calling process (Probably via stderr). > * AF_UNIX support where sensible for better sandboxing. > > MIGHT haves: > * Rename the env vars to not start with "TOR_PT". Some people claim > that this is a good idea (I think it is stupid and cosmetic). > * Ability to force at least clients to stop network activity without > tearing the PT down. > * Deprecate SOCKS4a, and make SOCKS5 mandatory for clients. > > UNLIKELY: > * Specify an interface for where fork()/exec() isn't possible (iOS). > I don't think this is makes sense because it is probably too > platform/caller specific. > * Allow operating both as a client and a server simultaneously. I > don't see a problem with running 2 copies of something for this > use case. > > I probably missed some things. If people have strong opinions about > this, do reply, otherwise I *will* design something that I like, which > will not include what other people want. > > Regards, > > -- > Yawning Angel -- Adam Pritchard Psiphon Inc.

3 2

How long does it take for 25/50/75% of relays to update to a new Tor version?
by Karsten Loesing 18 Sep '15

18 Sep '15

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hello Sebastian and list, Sebastian again suggested a fun task for the latest measurement team 1-1-1 task exchange round [0] that I picked up. He asked the following question: When a new Tor version is released, how long does it take for 25, 50, 75% of all relays to update? How often do relays update in general? Is there an effect when for example Debian updates its packages? It would help me to get an idea what I need to answer these questions :-) "update" means you're running, then you go down, then within a few minutes you're back up again with a changed version Here's what I came up with: I suggest we use two data sets as input for this task. The first data set consists of archived consensuses, available at: https://collector.torproject.org/archive/relay-descriptors/consensuses/ We'll need "valid-after" and "server-versions" from the header and "v" and "w" lines from all contained entries. Example: valid-after 2015-09-18 08:00:00 server-versions 0.2.4.23,0.2.4.24,0.2.4.25,0.2.4.26,0.2.4.27,0.2.5.8-rc,0.2.5.9-rc,0.2.5.10,0.2.5.11,0.2.5.12,0.2.6.5-rc,0.2.6.6,0.2.6.7,0.2.6.8,0.2.6.9,0.2.6.10,0.2.7.1-alpha,0.2.7.2-alpha v Tor 0.2.7.2-alpha-dev w Bandwidth=450 v Tor 0.2.6.10 w Bandwidth=23 [...] We can use those lines to *count* the number of running relays with a given version string, and we can sum up the *fraction* of consensus weights of relays running a given version string. Both can be useful, though I suspect the latter to be more meaningful. We should aggregate these consensus parts into daily statistics to make them easier to handle. We can do that by keeping counters per (date, version) tuple and per (date) for a) number of relays and b) consensus weight, and then we divide the (date, version) numbers by the (date) numbers to obtain averages. We'd also determine whether a version is recommended, whether it's the latest recommended version in its series, and whether it's the latest recommended stable version. The result would be a .csv file like the following (example data!): date,version,recommended,series,stable,count,fraction 2015-09-18,0.2.6.10,TRUE,TRUE,TRUE,0.387,0.451 2015-09-18,0.2.6.9,TRUE,FALSE,FALSE,0.063,0.0102 2015-09-18,0.2.6.8,TRUE,FALSE,FALSE,0.021,0.001 2015-09-18,0.2.6.7,TRUE,FALSE,FALSE,0.016,0.009 Read this as: "On 2015-09-18, version 0.2.6.10, which was both recommended and the latest in its series, was run by 38.7% of relays by count or 45.1% by consensus weight." (again, example data!) You could then draw graphs similar to https://metrics.torproject.org/versions.html, but with a percent scale on y. Though I'm not sure how readable those graphs would be with a few dozen lines in them. Still, they might be useful to explore the data set. The second data set that would be useful here is events related to versions. The following events come to mind: - tagged as version in Git (use a command similar to [1]), - tagged as alpha, beta, rc, or stable (look at version string), - first/last recommended by directory authorities for relays (parse from consensuses), - last recommended as newest version in a series or newest stable version (also parse from consensuses), or - first released in Debian stable (unclear how to obtain those dates). A possible data format would be: date,version,event 2015-07-12,0.2.6.10,git_tag 2015-06-10,0.2.6.9,git_tag 2015-05-19,0.2.6.8,git_tag 2015-04-06,0.2.6.7,git_tag For exploratory purposes, you could add these as colored vertical lines to the graphs above. Of course, adding more elements to a probably already overloaded graph doesn't exactly make it easier to understand. But if you limit the x axis on just a month or two, it might be useful. So, regarding your original question: "How long does it take for 25/50/75% of relays to update to a new Tor version?" I'm not sure whether that's really the best question to ask/answer here. Some versions will never be deployed on 75%, 50%, or even 25% of relays, because a new version was released shortly after. But that doesn't mean that the previous version was bad. This question also weights all versions the same, though in reality some releases are more important than others. If you get one data point for each version and then aggregate them by taking the average, what exactly does that tell you? I think that question could be rephrased to: "How long does it take for 25/50/75% of relays to update to a new Tor version *series*?" And I think that's something you could answer with the first data set above. But there are other interesting questions you could answer with the two data sets. For example, - What fraction of relays (by number or consensus weight) is running (un-)recommended versions? - What fraction of relays (by number of consensus weight) is (not) running the latest version in a series or latest stable version? Those could again be answered by a graph that uses the first data set, dates on x and fractions on y, and possibly events from the second data set as vertical lines. Explore, explore. There, the hour is over ("It took an hour to write, I thought it would take an hour to read.", Fry, Futurama). Hope this is useful in any way. And maybe others have good/better ideas for you as well? If you come up with some interesting answers, please post them here. All the best, Karsten [0] 1-1-1 task exchange: you get 1 minute to describe a task that would take somebody else roughly 1 hour and that they will do for you within 1 week (review a document, write some analysis code, fix a small bug, etc.; better come prepared to get the most out of this; give 1, take 1) [1] `git log --tags --simplify-by-decoration --pretty="format:%ci %d" | grep "tag: tor" | less` -----BEGIN PGP SIGNATURE----- Version: GnuPG v1 Comment: GPGTools - http://gpgtools.org iQEcBAEBAgAGBQJV++DHAAoJEJD5dJfVqbCr+sYH/1nViWWJgie+BD6Mo+9pC+g9 qEfMQ+NBvYWr3/SFcn/zWjydgZf+YPVXcVsOA1B1YcLX2s8OY1cWM5t5cDCstvfh /vOYmsDw1nddBh8VQxggYPkTLfEWRRSB0IeMiFWWUdVVKkcepOxgCfXMWJlBEAgG 8EtarcTkxSl1YMrurH3iuL8PrHGRiGf+mEKOZOMFtpmkSii/bWpO7BOAEyOXC7Ib EX2OouhCvae5X3Lyp2CnlQZLSrRGtAFV8sQUw5gcQb5/nouk38M7/ayrkBW/H3Ng qM9k0Y9EHAs1I1rn1ZY5lfGMKZ0WR+1NNCQ3mdsBSF5Qah01i1By7uk2FiH0e4U= =H6/Z -----END PGP SIGNATURE-----

1 0

List Administrivia [...encrypted onion services...]
by grarpamp 17 Sep '15

17 Sep '15

fabio, sky, et al... Please stop top quoting massive amounts of useless text that everyone's already seen before. It's seriously annoying to peoples workflow trying to figure out what youre replying to and if youve inlined other stuff below. And it needlessly bloats peoples mailboxes and thus search results. Trim the original to relavent bits and write your reply below that in context. Thanks, users everywhere.

1 0

Proposal: End-to-end encrypted onion services for non-Tor clients
by Donncha O'Cearbhaill 17 Sep '15

17 Sep '15

Hello all, I have been thinking about ideas to make Tor hidden services more available and secure for non-Tor users. Inline I've included a draft proposal which describes an end-to-end encrypted Tor2Web-like system. I'm really interested in hearing any suggestions, comments or criticism about this proposal. In particular I'd like to know if the trust requirements for the entry proxies and resolvers seem reasonable? Does this proposal make sense and is it something worth implementing? Thank you to David Stainton and Leif Ryge for your feedback on this proposal. Kind Regards, Donncha --- Filename: xxx-encrypted-onion-services-for-non-tor-clients.txt Title: End-to-end encrypted onion services for non-Tor clients Author: Donncha Ó Cearbhaill Created: 22-Aug-2015 Status: Draft 1. Overview: This proposal describes a system to allow non-Tor users to securely access location-hidden services via a domain name chosen by the hidden service operator. 2. Motivation: Tor hidden services have typically been used to provide anonymity to both services and clients. There are however use cases where a publisher or content provider requires anonymity but their users do not. The publishers typically would like their content to be available securely and to the widest possible audience. Tor2Web currently allows non-Tor users to access hidden services but it has a number of limitations. Tor2Web deployments are limited due to requirement to entrust Tor2Web service providers with users location information and the content of their requests. The onion subdomain based addressing in Tor2Web poses a usability issue and can be unwieldy for users without providing any self-authenticating properties. Tor2Web can be configured with a custom domain for an individual hidden service but this requires manual configuration and additional infrastructure. Ideally non-Tor users should be able to access location-hidden services under a standard domain name with end-to-end encryption from the client's browser to the hidden service web server. The system should be decentralized and it should not require the hidden service operator to run any clearnet systems. 3. Proposal: At a high level this proposal specifies a system of TCP entry proxies which transparently proxy the TLS connections of non-Tor clients to a hidden service. A client who requests a domain name such as https://myblog.com should get connected to a hidden service endpoint which has a valid SSL certificate for myblog.com. 3.1. Hidden Service configuration A hidden service operator obtains a public domain name and a corresponding CA signed SSL certificate. The operator configures the hidden service web server with the SSL certificate for the external domain. The domain name is pointed at an authoritative nameserver provider. The operator should also set a TXT DNS record containing the onion address of their hidden service. 3.2. Entry Proxy Modern web browsers send a Server Name Indication (SNI) field in the initial ClientHello of the TLS handshake. The entry proxy reads this field to determine the clients destination domain. The proxy performs a DNS lookup at that domain for a TXT record which specifies the corresponding onion address for the provided domain. The TLS entry proxy connects to the onion service via a Tor2Web-type circuit and then begins transparently proxying TLS traffic between the clients browser and the destination onion service. 3.3. Resolving Nameserver The key component of this system is a DNS name server which can direct clients to an online entry proxy via a round-robin type system. In a basic solution the onion service operator could manually specify one or more entry proxies in the A records for their domain on their existing authoritative DNS provider. In practice entry proxy churn would result in changing set of entry proxies. The nameserver will need regularly update its set of online entry proxies an remove proxies which are malfunctioning, malicious or otherwise unusable. The resolver may run a scanner to check its known proxies or load a list from an external service. Additionally the resolver should detect if an entry proxy blacklists a domain for which it is responsible and avoid routing clients to that entry proxy. It is expected that independent service providers will run their own domain->onion resolving nameserver in diverse jurisdictions as free or paid services. 4. Implementation: The entry proxy component could be implemented as part of the current Tor relay code base. Integration directly within Tor would allow use of the existing network consensus and bandwidth measurement systems to be used to discover available entry proxies. It would also allow for malicious entry proxies to be blacklisted. Alternatively the entry proxy could be implemented in the existing Tor2Web software or as a standalone software package. Implementing outside of Tor would be faster and it would avoid the risk of losing Tor relay capacity as a result of legal threats to the entry proxies. The resolving nameserver is the most complicated component of this system. The component will eventually require a DNS server, a management interface, and a set of network monitoring tools. 5. Security and resiliency implications: 5.1. Availability Attacks: Adversaries can attack the availability of a publicly-proxied hidden service at a number of levels: * Censorship or shutdown of entry proxy: Attacks on individual entry proxies are mitigated by performing DNS based round-robin between many online entry proxies. The Resolver system should be able to quickly remove entry proxies which misbehave or which go offline. * Censorship or seizure of the hidden service public domain: DNS based blocks are widely deployed for censorship and may be difficult to avoid. Domain registrars can also be forced to suspend domain names. Service operators should considering running their service under a TLD which is less vulnerable to these type of coercive threats. * Takedown of a nameserver provider: Multiple resolving nameservers can be configure for each forwarded domain. Using nameservers maintained by different providers can provide resilience to attacks against a single nameserver provider. 5.2. Security Attacks: Entry proxies and exit relays have a similar ability to monitor and interfere with client traffic. This is greater risk of targeted interference from entry proxies as they can also determine the client's network location. * Man-in-the-middle HTTP connections: Entry proxies have the ability to man-in-the-middle HTTP connections. Service operators should send HSTS header to force clients to automatically use TLS for all future connections. * TLS man-in-the-middle with CA-signed certificate Some commercial CA cert providers allow for domain ownership to be validated by providing a file over HTTP at the domain. A malicious entry proxy could successfully obtain a CA-signed certificate from one of these certificate authorities. Service operators can minimize their exposure to this type of attack by using HPKP headers to limit the set of valid certificate authorities for their domain.

5 5

multiple relay identities on a single IP:port, bug or feature?
by nusenu 16 Sep '15

16 Sep '15

Hi, in the last two days someone started generating a steady amount of new relay fingerprints on a single IP:port (2 per hour, actually a lot more than that but only to make it into the consensus) [1]. I'm surprised that actually both of them end up in the consensus. example: https://collector.torproject.org/recent/relay-descriptors/consensuses/2015-… search for "67.173.119.40 51256" - you will find it twice. Does that make sense or is this a bug? btw: Why do dir auths include non-running relays in their votes (flag line is "s" only), instead of not voting for them at all? thanks! [1] http://article.gmane.org/gmane.network.onion-routing.ornetradar/113

2 1

Draft Proposal: Random Number Generation During Tor Voting
by George Kadianakis 16 Sep '15

16 Sep '15

Hello there, we are glad to release a first draft of our proposal on distributed random generation using the Tor voting process. It specifies how Tor dirauths can generate a fresh random value every day using a commit-and-reveal protocol. The protocol piggybacks on top of the regular Tor voting procedure which happens every hour. Our proposal spends lots of effort resolving the various edge cases and engineering issues that come up when you are trying to fit such a protocol into the Tor voting system. It also introduces a new consensus flavor document which is used to host shared randomness information, so that the regular consensus does not get affected if this feature is flawed. We are especially interested in feedback from people who are familiar with the Tor voting protocol and can tell us whether we have messed something up, or whether there are attacks that we did not consider. We hope the document (and the illustration) is helpful for understanding the protocol. Let us know if you have any questions, or suggestions for improving the proposal. We believe this proposal is fundamental for the security of hidden services and for developing proposal 224, hence we invite everyone to provide feedback and improvements. Thanks! ----------------------------------------------------------------------------------- Filename: xxx-commit-reveal-consensus.txt Title: Random Number Generation During Tor Voting Authors: David Goulet, George Kadianakis Created: 2015-08-03 Status: Draft 1. Introduction 1.1. Motivation For the next generation hidden services project, we need the Tor network to produce a fresh random value every day in such a way that it cannot be predicted in advance or influenced by an attacker. Currently we need this random value to make the HSDir hash ring unpredictable (#8244), which should resolve a wide class of hidden service DoS attacks and should make it harder for people to gauge the popularity and activity of target hidden services. Furthermore, this random value can be used by other Tor-related protocols (or even non-Tor-related) like OnioNS to introduce unpredictability to the protocol. 1.2. Previous work Proposal 225 specifies a commit-and-reveal protocol that can be run as an external script and have the results be fed to the directory authorities. However, directory authority operators feel unsafe running a third-party script that opens TCP ports and accepts connections from the Internet. Hence, this proposal aims to embed the commit-and-reveal idea in the Tor voting process which should makes it smoother to deploy and maintain. Another idea proposed specifically for Tor is Nick Hopper's "A threshold signature-based proposal for a shared RNG" which was never turned into an actual Tor proposal. 2. Overview This proposal alters the Tor consensus protocol such that a random number is generated by the directory authorities during the regular voting process. The distributed random generator scheme is based on a commit-and-reveal technique. The proposal also specifies how the final shared random value is embedded in consensus documents so that clients who need it can get it. 2.1. Ten thousand feet view Our commit-and-reveal protocol aims to produce a fresh shared random value everyday at 12:00UTC. The final fresh random value is embedded in the microdescriptor consensus document at that time. Our protocol uses a *new* consensus flavor document called "shared randomness document" (SR doc). We use a new consensus document as a way to keep ground truth state and also as a way to apply the majority (see section [MAJORITY]) rule on commit and reveal values. It also allows rebooting authorities to rejoin the protocol in some cases. Our protocol has two phases and uses the hourly voting procedure of Tor. Each phase lasts 12 hours, which means that 12 voting rounds happen in between. In short, the protocol works as follows: Commit phase: Starting at 12:00UTC and for a period of 12 hours, authorities every hour send their commitments in their votes. They also include any received commitments from other authorities, if available. From those votes, a shared random document consensus is computed containing the commitments decided by the majority. Reveal phase: At 00:00UTC, the reveal phase starts and lasts till the end of the protocol at 12:00UTC. In this stage, authorities must reveal the value they committed to in the previous phase. The commitment and revealed values from other authorities, when available, are also added to the vote. Then a shared random document consensus is computed containing the commitments and the revealed values agreed on. Shared Randomness Calculation: At 12:00UTC, the shared random value is computed from the agreed revealed values located in the shared random document and finally added to the microdescriptor consensus. This concludes the commit-and-reveal procedure at 12:00UTC everyday. 2.2. Commit & Reveal Our commit-and-reveal protocol aims to produce a fresh shared random value everyday at 12:00UTC. In the beginning of that time period, each authority generates a new random value and keeps it for the whole day. The authority cryptographically hashes the random value and calls the output its "commitment" value. The original random value is called the "reveal value". Given a reveal value you can verify that it corresponds to a given commitment value. However given a commitment value you cannot derive the underlying reveal value. 2.3. Microdescriptor Consensus [MDCONS] Every hour, the microdescriptor consensus documents need to include the shared random value of the day, as well as the shared random value of the previous day. That's because either of these values might be needed at a given time for a Tor client to access a hidden service according to section [TIME-OVERLAP] of proposal 224. These means that these two values also need to be included in votes and in SR documents as well. Microdescriptor consensuses include: (a) The shared random value of the current time period. This is derived from the reveal values sent by the authorities during the voting session. (b) The shared random value of the previous time period. This is the same shared random value that was included in the votes. 2.4. Shared Random Document [SRDOC] The Shared Random document is a consensus flavor that contains the current state of our commit & reveal protocol. Since it uses the consensus mechanism of Tor, we use it as a way to enforce majority voting on the commitments and reveals without messing with the actual network status consensus. See section [REBOOT] for detail on how this document is handled when an authority reboots. During the commitment phase, the SR doc is populated with the commitments of all authorities. Then during the reveal phase, it's also used to store the reveal values as well. As discussed previously, the shared random values from the current and previous time period must be present in the document at all times if they are available. A shared random document requires 50% + 1 authority signatures to be considered valid. As this proposal is being written, there are 9 authorities thus we would need 5. 2.5. Protocol Illustration We have prepared an illustration to help you understand the protocol. You can find it here: https://people.torproject.org/~asn/hs_notes/shared_rand.jpg For every hour, it shows the authority votes and the resulting SR doc and microdescriptor consensus. The chain 'A_1 -> c_1 -> r_1' denotes that the authority committed to the value c_1 which corresponds to the reveal value r_1. The illustration depicts the first 25 hours of running the protocol. It starts with the very first commit round, then moves on to the second commit round, and then skips directly to the last commit round. Then the reveal phase starts, where we again show the first, second and last rounds. After the reveal phase is done, we generate the shared randomness (SR_1) and we start the new commit phase. The illustration finishes with the second round of this new commit phase. We advice you to revisit this after you have read the whole document. 3. Protocol In this section we give a detailed specification of the protocol. We describe the protocol participants' logic and the messages they send. The encoding of the messages is specified in the next section ([SPEC]). Now we go through the phases of the protocol: 3.1 Commitment Phase [COMMITMENTPHASE] The commit phase lasts from 12:00UTC to 00:00UTC. The goal is that at the end of this phase, the shared random document MUST contain a single commitment value from each authority (or none, if that authority did not participate in this phase). 3.1.1. Voting During Commitment Phase During the commit phase, each authority includes in its votes: - A commitment value for this consensus period. - Any commitments received from other authorities. - The two previous shared random values produced by the protocol (if any). After all votes have been received or pulled in, the authorities collectively generate the shared random document containing the commitments. 3.1.2. Shared Random Document During Commitment Phase [SRDOCCOMMIT] During the commitment phase, the shared random document contains: - The commitments received by the majority of authorities - The two previous shared random values produced by the protocol (if any). A commitment should only be transcribed to the shared random document if and only if the majority of the voting authorities agreed that a particular commitment was sent by a particular authority. Appendix section [COMMITEXAMPLE] contains an example of this procedure. The commit phase lasts for 12 hours, so authorities have multiple chances to commit their values. An authority can commit a second value during a subsequent round of the commit phase, but only the last value should be transcribed to the shared random document and only if it has been seen by the majority. Also, an authority should not be able to register a commitment value for a different authority. Hence, an authority X should only vote and place in the SR doc commitments by authority Y, iff authority Y included that commitment in its vote. 3.1.3. First & Last Round Of Commitment Phase [FIRSTLASTROUND] It's worth mentioning that during the very first round of the commitment phase at 12:00UTC, each authority votes its own commitment and is unaware of the commitments of the other authorities. For this reason, it's unlikely that a majority opinion of commitments will be created at 12:00UTC. Instead authorities are expected to form a majority opinion and transcribe commitments to the SR doc during the voting period of 13:00UTC or at least until the reveal phase. Similarly, an authority will not be able to commit to a new value during the last round of the commitment phase. That's because there won't be enough time for the other authorities to form a majority opinion about this value before the reveal phase. Hence, Tor authorities SHOULD NOT commit new values during the last round of the commitment phase at 23:00UTC. 3.2 Reveal Phase The reveal phase lasts from 00:00UTC to 12:00UTC. Now that the commitments have been agreed on, it's time for authorities to reveal their random values. 3.2.1. Voting During Reveal Phase During the reveal phase, each authority includes in its votes: - Its reveal value that was previously committed in the commit phase. - All the commitments and reveals received from other authorities. - The two previous shared random values produced by the protocol (if any). The set of commitments have been established during the commitment phase and must remain the same. If an authority tries to change its commitment during the reveal phase or introduce a new commitment, the entire vote MUST be ignored for the purposes of this system. To do so, authorities during the first reveal round MUST check that received votes contain the same commitments as the last SR document of the commitment phase. In subsequent reveal rounds, authorities check the previous hour SR document for commitment validation. After all votes have been received, authorities generate the shared random document along with the consensus. 3.2.2. Shared Random Document During Reveal Phase [SRDOCREVEAL] During the reveal phase, the shared random document contains: - The commitments agreed on during the commitment phase. - The corresponding reveal values from the majority of authorities. - The two previous shared random values produced by this system (if any). Similar to the commitment phase, authorities transcribe reveal values to the shared random document if and only if the majority of the voting authorities have voted on that particular reveal value. An example of this can be seen in section [REVEALEXAMPLE]. Section [FIRSTLASTROUND] also applies for the reveal phase. This means that Tor authorities SHOULD NOT reveal new values during the last round of the reveal phase at 11:00UTC. 3.3. Shared Random Value Calculation At 12:00UTC Finally, at 12:00UTC every day, authorities compute a fresh shared random value and this value must be added to the microdescriptor consensus so clients can use it. Authorities calculate the shared random value using the reveal values in the latest shared random document as specified in subsection [SRCALC]. If the shared random value contains reveal contributions by less than 3 directory authorities, it MUST NOT be created. Instead, the old shared random value should be used as specified in section [SRDISASTER]. Authorities at 12:00UTC start including this new shared random value in their votes, replacing the one from two protocol runs ago. Authorities also start including this new shared random value in the SR document and in the microdescriptor consensus as well. Apart from that, authorities proceed voting normally as they would in the first round of the commitment phase (section [COMMITMENTPHASE]). 3.3.1. Shared Randomness Calculation [SRCALC] An authority that wants to derive the shared random value V, should use the appropriate reveal values for that time period and calculate V as follows: V = H(ID_a | R_a | ID_b | R_b | ...) where the ID_k value is the identity fingerprint of directory authority k and R_k is its corresponding reveal value of that authority for the current period. H is sha256 for protocol version 1. XXX Should the hashing here include more elements? Like the previous random value for chaining? Or the current date? See how the NIST beacon does it in case we can steal some additional RNG security properties: http://hackaday.com/2014/12/19/nist-randomness-beacon/ 3.4. Bootstrapping Procedure As described in [MDCONS], two shared random values are required for the HSDir overlay periods to work properly as specified in proposal 224. Hence clients MUST NOT use the randomness of this system till it has bootstrapped completely; that is, until two shared random values are included in a consensus. This should happen after three 12:00UTC consensuses have been produced, which takes 48 hours. 3.5. Rebooting Directory Authorities [REBOOT] The shared randomness protocol must be able to support directory authorities who leave or join in the middle of the protocol execution. An authority that commits in the Commitment Phase and then leaves SHOULD store its reveal value on disk so that it continues participating in the protocol if it returns before or during the Reveal Phase. The reveal value MUST be stored timestamped to avoid sending it on wrong protocol runs. For this reason, other authorities should carry the commitment values of absent authorities in the shared randomness document until the end of the protocol. The shared randomness document can be used to verify that the commitment values are carried properly. An authority that misses the Commitment Phase cannot commit anymore, so it's unable to participate in the protocol for that run. Same thing for an authority that misses the Reveal phase. Authorities who do not participate in the protocol SHOULD still carry commits and reveals of others in their vote. 3.6. How we define majority [MAJORITY] The shared randomness protocol must be able to support directory authorities who participate in the consensus protocol but not in the shared randomness protocol. It must also be able to tolerate authorities who drop or join in the middle of the protocol. The security of this proposal strongly relies on forming majority opinion so it's important for the number of participants to always be well defined: In the voting session right before creating the SR doc, we define the number of active participants to be the number of directory authorities that included commit/reveal values in their votes. As specified in sections [SRDOCCOMMIT] and [SRDOCREVEAL], a commit/reveal value should be transcribed to the SR doc if and only if the majority voted for it. So for example, if there are 6 active participants, a commit value will only be transcribed if 4 or more participants agreed on it. Furthermore, as specified in section [SRDOC], the shared random document is considered valid only if it is signed by 50% + 1 authorities. XXX The number of active participants is dynamic as authorities leave and join the protocol. Since the number of active participants is dynamic , an attacker could trick some authorities believing there are N participants and some others believing there are N-1 participants, by sending different votes to different auths. Should we worry? [asn] A way to avoid a dynamic number of participants could be to set the number of participants to be the number of auths who committed during the very first commitment phase round. 3.7. Shared Randomness Disaster Recovery [SRDISASTER] If the consensus at 12:00UTC fails to be created, then there will be no new shared random value for the day. Directory authorities should keep including the previous shared random values in the consensus till the next 12:00UTC commit-and-reveal session. The time period needs to be updated to reflect the current time period even if the random value stays the same. Clients should keep on using this shared random values. 4. Specification [SPEC] 4.1 Voting This section describes how commitments, reveals and SR values are encoded in votes. We describe how to encode both the authority's own commits/reveals and also the commits/reveals received from the other authorities. Commits and reveals share the same line, but reveals are optional. 4.1.1 Encoding the authority's own commit/reveal value An authority that wants to commit (or reveal) a value during a vote, should generate a random 256-bit value REVEAL, and include its commitment COMMIT in its 12:00UTC vote as follows: "shared-rand-commitment" SP algname SP COMMIT [SP REVEAL] NL During the Reveal Phase, an authority can also optionally reveal the value REVEAL. The "algname" is the hash algorithm that should be used to compute COMMIT and REVEAL if any. It should be "sha256" for this version. The commitment value COMMIT is constructed as follows: C = base64-encode( SHA256(REVEAL) ) 4.1.2 Encoding commit/reveal values received by other authorities [COMMITOTHER] An authority puts in its vote the commitments and reveals it has seen from the other authorities. To do so, it includes the following in its votes: "shared-rand-received-commitment" SP identity SP algname SP COMMIT [SP REVEAL] NL when "identity" is the hex-encoded commitment's authority fingerprint and COMMIT is the received commitment value. Authorities can also optionally include the reveal value REVEAL. There MUST be only one line per authority else the vote is considered invalid. Finally, the "algname" is the hash algorithm that should be used to compute COMMIT and REVEAL if any. 4.1.3. Shared Random value Authorities include a shared random value in their votes using the following encoding for the previous and current value respectively: "shared-rand-previous-value" SP value NL "shared-rand-current-value" SP value NL where "value" is the actual shared random value. It's computed as specified in the section [SRCALC]. To maintain consistent ordering, the shared random values of the previous period should be listed before the values of the current period. 4.2. Shared Random Document As a way to keep ground truth state in this protocol, we introduce a new consensus flavor document. We call it the "Shared Random Document". This document is only used by directory authorities. This new consensus flavor should be signed with the sha256 signature format as documented in proposal 162. 4.2.1 Format [SRFORMAT] This document has a very strict format because authorities need to generate the exact same document. It contains a preamble, a commitment and reveal section, a list of shared random values and finally a footer. The preamble (or header) contains the following items. They MUST occur in the order given here: "shared-random-version" SP version SP flavor NL [At start, exactly once.] A document format version. For this specification, version is "1". The flavor is always "shared-random". "created" SP YYYY-MM-DD SP HH:MM:SS NL [Exactly once] The creation time of this document. "valid-until" SP YYYY-MM-DD SP HH:MM:SS NL [Exactly once] After this time, this document is invalid and shouldn't be used nor trusted. The validity time period is 3 hours. "protocol-phase" SP phase NL [Exactly once] The current protocol phase when this document is generated. The accepted values are: "commitment" and "reveal". The following details the commitment and reveal section. "shared-rand-commitment" SP algname SP identity SP commitment-value [SP revealed-value] NL [Exactly once per authority] This is the commitment or/and reveal value agreed upon by the majority from one authority. The algname is always "sha256" in version 1. The "identity" is the authority hex-encoded digest of the authority identity key of the signing authority from which the values are from. Finally, "{commitment|revealed}-value" is the value as specified in section [SPEC]. Next is the shared random value section. "shared-rand-previous-value" SP value NL [At most once] This is the previous shared random value agreed on at the previous period. The "value" is defined in section [SRCALC]. "shared-rand-current-value" SP value NL [At most once] This is the latest shared random value. The "value" is defined in section [SRCALC]. Finally, the footer of the document: "shared-random-footer" NL [Exactly once] It contains one subsection, the authority signatures. "authority-signature" SP algname SP identity SP signing-key-digest NL Signature NL [Exactly once per authority] The "identity" is the hex-encoded digest of the authority identity key and the "signing-key-digest" is the hex-encoded digest of the current authority signing key of the signing authority. The "algname" item is the algorithm used to compute the hash of the document before signing it. As proposal 162 proposed, "sha256" should be used. The authority-signature entry MUST be ignored if "algname" is unrecognized. See dir-spec.txt for a specification for the Signature item. 4.3. Shared Random Value in Consensus [SRCONSENSUS] Authorities insert the two shared random values in the consensus following the same encoding format as in [SRFORMAT]. 5. Security Analysis 5.1. Security of commit-and-reveal and future directions The security of commit-and-reveal protocols is well understood, and has certain flaws. Basically, the protocol is insecure to the extent that an adversary who controls b of the authorities gets to choose among 2^b outcomes for the result of the protocol. However, an attacker who is not a dirauth should not be able to influence the outcome at all. We believe that this system offers sufficient security especially compared to the current situation. More secure solutions require much more advanced crypto and more complex protocols so this seems like an acceptable solution for now. 5.2. Is there a need for a final agreement phase? Commit-and-reveal protocols usually also end with an agreement phase, during which participants agree on which reveal values should be used to make the shared random value. An agreement phase is needed, because if the protocol ended with the reveal phase, an evil authority could wait until the last reveal round, and reveal its value to half of the authorities. That would partition the authorities into two sets: the ones who think that the shared random value should contain this new reveal, and the rest who don't know about it. This would result in a tie and two different SR docs. However, we believe that an agreement phase is not necessary in our protocol since reveal values are transcribed in the SR document if only if the majority agrees. Hence, a tie is not enough to confuse the authorities since it's not majority and the offending value would just be discarded. That said, an attack that could still work here would be if an authority can make half of the authorities believe that the value should be discarded, and make the other half of the authorities believe that the value should be included. That could be achieved if the attacker could force honest authorities to send different votes to different authorities. We believe this should not be the case currently, but we should look more into this. XXX Needs feedback by a person who knows the voting protocol well!!! 5.3. Predicting the shared random value during reveal phase The reveal phase lasts 12 hours, and most authorities will send their reveal value on the first round of the reveal phase. This means that an attacker can predict the final shared random value about 12 hours before it's generated. This does not pose a problem for the HSDir hash ring, since we impose an uptime restriction on HSDir nodes, so 12 hours predictability is not an issue. Any other protocols using the shared random value from this system should be aware of this property. 6. Discussion 6.1. Why the added complexity from proposal 225? The complexity difference between this proposal and prop225 is in part because prop225 doesn't specify how the shared random value gets to the clients. This proposal spends lots of effort specifying how the two shared random values can always be readily accessible to clients. 6.2. Why do you do a commit-and-reveal protocol in 24 rounds? The reader might be wondering why we span the protocol over the course of a whole day (24 hours), when only 3 rounds would be sufficient to generate a shared random value. We decided to do it this way, because we piggyback on the Tor voting protocol which also happens every hour. We could instead only do the shared randomness protocol from 21:00 to 00:00 every day. Or to do it multiple times a day. However, we decided that since the shared random value needs to be in every consensus anyway, carrying the commitments/reveals as well will not be a big problem. Also, this way we give more chances for a failing dirauth to recover and rejoin the protocol. 6.3. Why can't we recover if we fail to do a consensus at 12:00UTC? Section [SRDISASTER] specifies that if the 12:00UTC consensus or SR doc fails to be created, we fall back to the random value of the previous day meaning authorities will carry the last valid SR values from the previous microdescriptor consensus to the new one. Theoretically, we could recover by calculating the shared randomness of the day at 13:00UTC instead. However, adding such fallback logic would complicate the protocol even further, so we have not yet considered it. 7. Appendix 7.1. Example commitment majority [COMMITEXAMPLE] Here is an example of voting during the commitment phase. The table below represents the votes of 6 individual authorities A_i (one vote per column). Since it's the commitment phase, votes include the authorities commitments and all commitments received. For example, below all authorities believe that A_1 has registered the value 7 as its commitment. +------------+------------+-------------+-------------+-------------+-----------+ | A_1 vote | A_2 vote | A_3 vote | A_4 vote | A_5 vote | A_6 vote | +------------+------------+-------------+-------------+-------------+-----------+ | A_1 -> 7 | A_1 -> 7 | A_1 -> 7 | A_1 -> 7 | A_1 -> 7 | A_1 -> 7 | | A_2 -> 66 | A_2 -> 66 | A_2 -> 42 | A_2 -> 42 | A_2 -> 42 | A_2 -> 42 | | A_3 -> 16 | A_3 -> 16 | A_3 -> 16 | A_3 -> 16 | A_3 -> 16 | A_3 -> 16 | | A_4 -> 22 | A_4 -> 22 | A_4 -> 22 | BLANK | A_4 -> 22 | BLANK | | A_5 -> 9 | A_5 -> 9 | A_5 -> 9 | A_5 -> 9 | A_5 -> 9 | A_5 -> 9 | | A_6 -> 33 | A_6 -> 33 | A_6 -> 33 | A_6 -> 33 | A_6 -> 33 | BLANK | +------------+------------+-------------+-------------+-------------+-----------+ In this case, following the majority rule, the final SR doc will contain: +-------------+ | SR Document | +-------------+ | A_1 -> 7 | | A_2 -> 42 | | A_3 -> 16 | | A_4 -> 22 | | A_5 -> 9 | | A_6 -> 33 | +-------------+ 7.2. Example reveal phase [REVEALEXAMPLE] Here is an example of voting during the reveal phase. The table below represents 6 votes by 6 different authorities A_i (one vote per column). Since it's the reveal phase, votes include all reveals received (commitments have been hidden for simplicity). For example, below all authorities believe that A_1 has revealed the value 444. Let's say that a malicious dirauth is trying to partition the group into two sets, by sending different votes to different auths. The attacker has splitted the group into two sets, the auths who think that A_6 has revealed the value 123, and the rest who have not seen a reveal from A_6. +------------+------------+-------------+-------------+-------------+------------+ | A_1 vote | A_2 vote | A_3 vote | A_4 vote | A_5 vote | A_6 vote | +------------+------------+-------------+-------------+-------------+------------+ | A_1 -> 444 | A_1 -> 444 | A_1 -> 444 | A_1 -> 444 | A_1 -> 444 | A_1 -> 444 | | A_2 -> 110 | A_2 -> 110 | A_2 -> 110 | A_2 -> 110 | A_2 -> 110 | A_2 -> 110 | | A_3 -> 420 | A_3 -> 420 | A_3 -> 420 | A_3 -> 420 | A_3 -> 420 | A_3 -> 420 | | BLANK | BLANK | A_4 -> 980 | BLANK | A_4 -> 980 | BLANK | | A_5 -> 666 | A_5 -> 555 | A_5 -> 555 | A_5 -> 555 | A_5 -> 555 | A_5 -> 555 | | A_6 -> 123 | A_6 -> 123 | A_6 -> 123 | BLANK | BLANK | BLANK | +------------+------------+-------------+-------------+-------------+------------+ Following the rules of the reveal phase, the reveal of A_4 should be ignored since it was not voted by > 3 authorities. The reveal from A_6 should also be ignored since it was only seen by half of the auths (3/6) which is not majority (it would require at least 4/6 votes). Hence, the final shared random document should contain: +-------------+ | SR Document | +-------------+ | A_1 -> 444 | | A_2 -> 110 | | A_3 -> 420 | | BLANK | | A_5 -> 555 | | BLANK | +-------------+

10 26

Towards a new version of the PT spec...
by Yawning Angel 15 Sep '15

15 Sep '15

So, we currently have a Pluggable Transport (PT) spec, and it kind-of sort-of works (The documentation is a mess that I'm working on cleaning up, but it's an orthogonal issue for how well it works). There are a number of problems with the current PT spec that require breaking backward compatibility to fix, so eventually I would like to do so. I'm soliciting input on what people would also like to see in a (currently hypothetical) PT spec 2.0 beyond what I already have in mind: MUST haves: * Support dual stack Bridges correctly (Multiple server endpoints per transport) * Increase the argument space beyond 510 bytes (Prop. #227). * Mandatory ExtORPort support (currently optional, but metrics are good). * Centralized logging by the calling process (Probably via stderr). * AF_UNIX support where sensible for better sandboxing. MIGHT haves: * Rename the env vars to not start with "TOR_PT". Some people claim that this is a good idea (I think it is stupid and cosmetic). * Ability to force at least clients to stop network activity without tearing the PT down. * Deprecate SOCKS4a, and make SOCKS5 mandatory for clients. UNLIKELY: * Specify an interface for where fork()/exec() isn't possible (iOS). I don't think this is makes sense because it is probably too platform/caller specific. * Allow operating both as a client and a server simultaneously. I don't see a problem with running 2 copies of something for this use case. I probably missed some things. If people have strong opinions about this, do reply, otherwise I *will* design something that I like, which will not include what other people want. Regards, -- Yawning Angel

5 6