- tor-dev - lists.torproject.org

Fwd: Re: Support for mix integration research
by Sebastian G. <bastik.tor> 25 Feb '16

25 Feb '16

I'm forwarding this to the list, because it might contain relevant information. Mostly within George's reply. Initial reason to not send my reply to the list was to not bother too many people with seemingly irrelevant questions on my part. I agree with the reply given in the forwarded message. It has to be considered what 'it' does to the network. Best Regards, Sebastian G. (bastik) -------------- Forwarded Message -------------- Subject: Re: [tor-dev] Support for mix integration research Date: Thu, 25 Feb 2016 19:58:56 +0100 From: George Kadianakis <desnacked(a)riseup.net> To: Sebastian G. <bastik.tor> <bastik.tor(a)googlemail.com> "Sebastian G. <bastik.tor>" <bastik.tor(a)googlemail.com> writes: > 25.02.2016, 13:08 George Kadianakis: >> b) Does this increase the hard disk or memory requirement of people running Tor >> relays? That is in high-latency mode, will Tor relays need to store client's >> traffic for longer? How does that impact the memory and hard disk >> requirement of people running Tor relays? > > Hello, > > this has been intentionally sent off list. > > I just don't understand the question. How would you delay traffic > without storing it for longer? Whenever in memory or disk, the data that > are being delayed have to be stored somewhere, as far as I understand. > > Could one delay traffic on a wire? E.g. like with a truck that drives on > a road between two storage facilities where you couldn't say it is > already stored in the facilities. > > I fail to see how it would be possible to have a system that delays > something without holding back whatever is delayed. > I agree. I'm wondering how this increases the memory and hard disk requirement of running a relay. And how much would the current relays contribute to such a high-latency system? And how much memory / CPU will be required to contribute meaningfully to this. And what about DDoS opportunities? All these things will need to be thought of so that we don't overwhelm the current Tor network by introducing this high-latency mode. Because if that's the case, we should probably have a separate network for the high-latency mode. Pity you replied off list, this conversation is of sufficient interest. Cheers. > If you currently take in x and give out y at the same rate, what needs > to be stored is what has to be buffered. If now the rate of y decreases > while the rate for x remains the same, more stuff needs be buffered. > Unless you reduce the rate for x, more storage is required. > > Sorry, I just don't get it. Please show me where my error is. Can be > short. :) >

1 0

Fwd: Downloadable content: Fonts!
by Nathan Freitas 25 Feb '16

25 Feb '16

Mozilla is adding some new runtime installation features to reduce the size of the mobile Firefox APK. Is this happening at all on desktop? It makes me nervous as the "default" config could very much more greatly, not to mention having a new centralized attack channel. ----- Original message ----- From: Sebastian Kaspari <s.kaspari(a)gmail.com> To: "mobile-firefox-dev" <mobile-firefox-dev(a)mozilla.org> Subject: Downloadable content: Fonts! Date: Fri, 19 Feb 2016 11:56:42 +0000 Good news, everyone! Our first step to downloadable content has been enabled in Nightly: This means we now stopped to ship fonts[1] in the APK and instead download them at runtime (Bug 1194338 [2]). With that we reduced the size of the APK by roughly 6.4% (~ 2.7MB) [3]. Without having the fonts downloaded (yet) our users can still browse websites but they may look less nice. And in fact, as things go, a bug caused just that to happen in Nightly (We don't download any fonts): bug 1249354 [4]. So if websites are currently looking a bit weird on Nightly then that's because of that. The bug should be resolved soon and after that let me know if you see any new weird issues related to (wrong) fonts. :) Our plans for the future: * Right now we ship the list of fonts and the location to download with the application. We want to synchronize this catalog of content from a Kinto instance: https://bugzilla.mozilla.org/show_bug.cgi?id=1201059 * We want to download hyphenation dictionaries at runtime too: https://bugzilla.mozilla.org/show_bug.cgi?id=1095719 * Eventually we might even want to download (some) localization files at runtime: https://bugzilla.mozilla.org/show_bug.cgi?id=945123 Best, Sebastian [1] https://www.youtube.com/watch?v=6J2rrFiN1Jw [2] https://bugzilla.mozilla.org/show_bug.cgi?id=1194338 [3] https://twitter.com/Anti_Hype/status/699905577196134400 [4] https://bugzilla.mozilla.org/show_bug.cgi?id=1249354 _______________________________________________ mobile-firefox-dev mailing list mobile-firefox-dev(a)mozilla.org https://mail.mozilla.org/listinfo/mobile-firefox-dev

5 7

Tor Consensus Transparency, take two
by Linus Nordberg 25 Feb '16

25 Feb '16

Hi, Some time ago, like 1,5 years, this list received a proposal for how to build Tor Consensus Transparency. I got some good feedback and then i got busy with other things. Here's an updated version with feedback incorporated, change in wordings, more of an actual specification, some changes in data structures and over all an easier read, I hope. I'll follow up shortly with a separate email describing the status of an experimental implementation. --8<---------------cut here---------------start------------->8--- Filename: xxx-tor-consensus-transparency.txt Title: Tor Consensus Transparency Author: Linus Nordberg Created: 2014-06-28 Status: Draft 0. Introduction This document describes how to provide and use public, append-only, verifiable logs containing Tor consensus and vote status documents, much like what Certificate Transparency [CT] does for TLS certificates, making it possible for log monitors to detect false consensuses and votes. Tor clients and relays can refuse using a consensus not present in a set of logs of their choosing, as well as provide possible evidence of misissuance by submitting such a consensus to any number of logs. 1. Overview Tor status documents, consensuses as well as votes, are stored in one or more public, append-only, externally verifiable log using a history tree like the one described in [CrosbyWallach]. Consensus-users, i.e. Tor clients and relays, expect to receive one or more "proof of inclusions" with new consensus documents. A proof of inclusion is a hash sum representing the tree head of a log, signed by the logs private key, and an audit path listing the nodes in the tree needed to recreate the tree head. Consensus-users are configured to use one or more logs by listing a log address and a public key for each log. This is enough for verifying that a given consensus document is present in a given log. Submission of status documents to a log can be done by anyone with an internet connection (and the Tor network, in case of logs only on a .onion address). The submitter gets a signed tree head and a proof of inclusion in return. Directory authorities are expected to submit to one or more logs and include the proofs when serving consensus documents. Directory caches and consensus-users receiving a consensus not including a proof of inclusion may submit the document and use the proof they receive in return. Auditing log behaviour and monitoring the contents of logs is performed in cooperation between the Tor network and external services. Relays act as log auditors with help from Tor clients gossiping about what they see. Directory authorities are good candidates for monitoring log content since they know what votes they have sent and received as well as what consensus documents they have issued. Anybody can run both an auditor and a monitor though, which is an important property of the proposed system. 2. Motivation Popping a handful of boxes (currently five) or factoring the same number of RSA keys should not be ruled out as a possible attack against a subset of Tor users. An attacker controlling a majority of the directory authorities signing keys can, using man-in-the-middle or man-on-the-side attacks, serve consensus documents listing relays under their control. If mounted on a small subset of Tor users on the internet, the chance of detection is probably low. Implementation of this proposal increases the cost for such an attack by raising the chances of it being detected. Note that while the proposed solution gives each individual some degree of protection against using a false consensus this is not the primary goal but more of a nice side effect. The primary goal is to detect correctly signed consensus documents which differ from the consensus of the directory authoritites. This raises the risk of exposure of an attacker capable of producing a consensus and feed it to users. The complexity of the proposed solution is motivated by the fact that the log key is not just another key on top of the directory authority keys since the log doesn't have to be trusted. Another value is the decentralisation given -- anybody can run their own log and use it. Anybody can audit all existing logs and verify their correct behaviour. This empowers people outside the group of Tor directory authority operators and the people who trust them for one reason or the other. 3. Design Communication with logs is done over HTTP using TLS or Tor onion services for transport, similar to what is defined in [rfc6962-bis-12]. Parameters for POSTs and all responses are encoded as name/value pairs in JSON objects [RFC4627]. Summary of proposed changes to Tor: - Configuration is added for listing known logs and for describing policy for using them. - Directory authorities start submitting newly created consensuses to at least one public log. - Tor clients and relays receiving a consensus not accompanied by a proof of inclusion start submitting that consensus to at least one public log. - Consensus-users start rejecting consensuses accompanied by an invalid proof of inclusion. - A new cell type LOG_STH is defined, for clients and relays to exchange information about seen tree heads and their validity. - Consensus-users send seen tree heads to relays acting as log auditors. - Relays acting as log auditors validate tree heads (section 3.2.2) received from consensus-users and send results back. - Consensus-users start rejecting consensuses for which valid proofs of inclusion can not be obtained. Definitions: - Log id: The SHA-256 hash of the log's public key, to be treated as an opaque byte string identifying the log. 3.1. Consensus submission Logs accept consensus submissions from anyone as long as the consensus is signed by a majority of the Tor directory authorities of the Tor network that it's logging. Consensus documents are POST:ed to a well-known URL as defined in section 5.2. The output is what we call a proof of inclusion. 3.2. Verification 3.2.1. Log entry membership verification Calculate a tree head from the hash of the received consensus and the audit path in the accompanying proof. Verify that the calculated tree head is identical to the tree head in the proof. This can easily be done by consensus-users for each received consensus. We now know that the consensus is part of a tree which the log claims to be The Tree. Whether this tree is the same tree that everybody else see is unknown at this point. 3.2.2. Log consistency verification Ask the log for a consistency proof between the tree head to verify and a previously known good tree head from the pool. Section 5.3 specifies how to fetch a consistency proof. [[TBD require auditors to fetch and store the tree head for the empty tree as part of bootstrapping, in order to avoid the case where there's no older tree to verify against?]] [[TODO description of verification of consistency goes here]] Relays acting as auditors cache results to minimise calculations and communication with log servers. [[TBD have clients verify consistency as well? NOTE: we still want relays to see tree heads in order to catch a lying log (the split-view attack)]] We now know that the verified tree is a superset of a known good tree. 3.3. Log auditing A log auditor verifies two things: - A logs append-only property, i.e. that no entries once accepted by a log are ever altered or removed. - That a log presents the same view to all of its users [[TODO describe the Tor networks role in auditing more than what's found in section 3.2.2]] A log auditor typically doesn't care about the contents of the log entries, other than calculating their hash sums for auditing purposes. Tor relays should act as log auditors. 3.4. Log monitoring A log monitor downloads and investigates each entry in a log searching for anomalies according to its monitoring policy. This document doesn't define monitoring policies but does outline a few strategies for monitoring in section [[TBD]]. Note that there can be more than one valid consensus documents for a given point in time. One reason for this is that the number of signatures can differ due to consensus voting timing details. [[TODO Are there more reasons?]] [[TODO expand on monitoring strategies -- even if this is not part of the proposed extensions to the Tor network it's good for understanding. a) dirauths can verify consensus documents byte for byte; b) anyone can look for diffs larger than D per time T, where "diffs" certainly can be smarter than a plain text diff]] 3.5. Consensus-user behaviour [[TODO move most of this to section 5]] Keep an on-disk cache of consensus documents. Mark them as being in one of three states: LOG_STATE_UNKNOWN -- don't know whether it's present in enough logs or not LOG_STATE_LOGGED -- have seen good proof(s) of inclusion LOG_STATE_LOGGED_GOOD -- confident about the tree head representing a good tree Newly arrived consensus documents start in UNKNOWN or LOGGED depending on whether they are accompanied by enough proofs or not. There are two possible state transitions: - UNKNOWN --> LOGGED: When enough correctly verifying proofs of inclusion (section 3.2.1) have been seen. The number of good proofs required is a policy setting in the configuration of the consensus-user. - LOGGED --> LOGGED_GOOD: When the tree head in enough of the inclusion proofs have been verified (section 3.2.2) or enough LOG_STH cells vouching for the same tree heads have been seen. The number of verifications required is a policy setting in the configuration of the consensus-user. Consensuses in state UNKNOWN are not used but are instead submitted to one or more logs. If the submission succeeds, this will take the consensus to state LOGGED. Consensuses in state LOGGED are used despite not being fully verified with regard to logging. LOG_STH cells containing tree heads from received proofs are being sent to relays for verification. Clients send to all relays that they have a circuit to, i.e. their guard relay(s). Relays send to three random relays that they have a circuit to. 3.6. Relay behaviour when acting as an auditor In order to verify the append-only property of a log, relays acting as log auditors verify the consistency of tree heads received in LOG_STH cells. An auditor keeps a copy of 2+N known good tree heads in a pool stored on persistent media [[TBD where N is either a fixed number in the range 32-128 or is a function of the log size]]. Two of them are the oldest and newest tree heads seen, respectively. The rest, N, are randomly chosen from the tree heads seen. [[TODO describe or refer to an algorithm for "randomly chosen", hopefully not subjective to flushing attacks (or other attacks)]]. 3.7. Notable differences from Certificate Transparency - The data logged is "strictly time-stamped", i.e. ordered. - Much shorter lifetime of logged data -- a day rather than a year. Is the effects of this difference of importance only for "one-shot attacks"? - Directory authorities have consensus about what they're signing -- there are no "web sites knowing better". - Submitters are not in the same hurry as CA:s and can wait minutes rather than seconds for a proof of inclusion. 4. Security implications TODO 5. Specification 5.0. Data structures Data structures are defined as described in [RFC5246] section 4, i.e. TLS 1.2 presentation language. While it is tempting to try to avoid yet another format, the cost of redefining the data structures in [rfc6962-bis-12] outweighs this consideration. The burden of redefining, reimplementing and testing is extra true for those structures which need precise definitions because they are to be signed. 5.1. Signed Tree Head (STH) An STH is a TransItem structure of type "signed_tree_head" as defined in [rfc6962-bis-12] section 5.8. 5.2. Submitting a consensus document to a log POST https://<log server>/tct/v1/add-consensus Input: consensus: A consensus status document as defined in [dir-spec] section 3.4.1 [[TBD gziped and base64 encoded to save 50%?]] Output: sth: A signed tree head as defined in section 5.1 refering to a tree in which the submitted document is included. inclusion: An inclusion proof as specified for the "inclusion" output in [rfc6962-bis-12] section 6.5. 5.3. Getting a consistency proof from a log GET https://<log server>/tct/v1/get-sth-consistency Input and output as specified in [rfc6962-bis-12] section 6.4. 5.x. LOG_STH cells A LOG_STH cell is a variable-length cell with the following fields: TBDname [TBD octets] TBDname [TBD octets] TBDname [TBD octets] 6. Compatibility TBD 7. Implementation TBD 8. Performance and scalability notes TBD A. Open issues / TODOs - TODO: Add SCTs from CT, at least as a practical "cookie" (i.e. no need to send them around or include them anywhere). Logs should be given more time for distributing than we're willing to wait on an HTTP response for. - TODO: explain why no hash function and signing algorithm agility, [[rfc6962-bis-12] section 10 - TODO: add a blurb about the values of publishing logs as onion services - TODO: discuss compromise of log keys B. Acknowledgements This proposal leans heavily on [rfc6962-bis-12]. Some definitions are copied verbatim from that document. Valuable feedback has been received from Ben Laurie, Karsten Loesing and Ximin Luo. C. References [CrosbyWallach] http://static.usenix.org/event/sec09/tech/full_papers/crosby.pdf [dir-spec] https://gitweb.torproject.org/torspec.git/blob/HEAD:/dir-spec.txt [RFC4627] https://tools.ietf.org/html/rfc4627 [rfc6962-bis-12] https://datatracker.ietf.org/doc/draft-ietf-trans-rfc6962-bis/12 [CT] https://https://www.certificate-transparency.org/ --8<---------------cut here---------------end--------------->8---

2 1

Support for mix integration research
by Katharina Kohls 25 Feb '16

25 Feb '16

Hi everyone, we are a team of 4 PHD students in the field of IT security, working at the Ruhr-University Bochum at the chair for systems security and the information security group. Currently we work on a research project with the goal to leverage the security of Tor against timing attacks by integrating mixes in Tor nodes. The general idea is to differentiate high-latency and low-latency traffic among the network for applying additional delays to the former type of packets. Based on this the success of traffic analysis attacks should be decreased without restricting the low latency assurance of Tor. We plan to integrate the mix into Tor version 0.2.5.10 and analyze its performance along with the Shadow simulator. As there are a lot of details to consider, both regarding the technical aspects of the integration as well as practical assumptions, e.g., "how do we get DiffServ-like nodes?", we would be pleased to receive some feedback on the idea and support for the implementation of the mix. Further details on the mix and stuff will sure be provided if needed! Cheers, Katharina -- M.Sc. Katharina Kohls Ruhr-University Bochum Research Group Information Security Universitätsstrasse 150 ID 2/123 44780 Bochum / Germany Phone: +49 234 / 32 - 26991 Web: www.infsec.rub.de

4 4

Nyx Project Ideas
by Damian Johnson 25 Feb '16

25 Feb '16

Hi wonderful relay operators. It's GSoC season again, where students can be funded to make open source projects like Tor even better! Nyx (previously known as arm [1]) has been my main focus this last year and is inching ever closer to release. For those unfamiliar with it, Nyx is an ncurses monitor for Tor relays providing a bandwidth graph, event log, connections, config editor, and more. Rather than add new features my work has focused on making Nyx simpler and faster, but GSoC provides us an opportunity to do even more. So I'm curious - what do you want from an ncurses monitor? The answer may be 'keep it simple'. Feature creep does us no favors. But if there's a good fit I'd love to mentor a project that makes your lives even better! I'm not overly fond of the ideas I've had so far... * Windows support. This poses a few challenges. [2] * When running multiple tor instances on a single system connect to them all, aggregating the information. So anything come to mind? Cheers! -Damian [1] https://www.atagar.com/arm/ [2] https://trac.torproject.org/projects/tor/wiki/doc/arm#Windows

1 0

Tor consensus documents now in a public append-only verifiable log
by Linus Nordberg 24 Feb '16

24 Feb '16

Hi, One of the directory authoritites (maatuska) is submitting new consensus documents to a public append-only verifiable log, similar to what Certificate Transparency uses. If you think this is exciting and want to help out with this experiment, consider setting up a monitor and start looking for anomalies, or at least an auditor verifying proper log behaviour. Or help making the submission code in Tor more useful (f.ex. submitting to the log's .onion address, submitting votes too, making the POST chunked). Below are two examples of how to investigate what's in the log so far. Fetching one document, using curl --------------------------------- Here's an example of how to fetch the 80th entry in the log: $ curl -sA '' -x socks4a://localhost:9050/ 'http://mvkhztpvqcxpdbn3.onion/open/gaol/v1/get-entries?start=79&end=79' The result is a large JSON document (~740kB): --8<---------------cut here---------------start------------->8--- { "entries" : [ { "leaf_input" : "AAAAAAFTA+pimwAACHcGH4sIAAAAAAACA+y96XLi2rYu+H8/hX7WDTKN+sa [...] rUcfehxefa/xjpqq45E6dZEPzPosd1cg8mSGi03G/r+LikK1rl7GQ/I7yOzvfvGm3jeWrD/g6++f2yN9oeI7dH+31/w3O7wS+HI8YAAAA", "extra_data" : "AAAA" } ] } --8<---------------cut here---------------end--------------->8--- "leaf_input" contains a gziped, base64 encoded Tor consensus document. Note that this format _will_ change in the future! Fetching all documents, using fetchallblobs.py from catlfish ------------------------------------------------------------ Given a local copy of branch gaol6 of catlfish in my private repo [1] and a copy of the public key for the log [2], here's how to download all the entries from the log and store them in ZIP files in directory 'gaol-store': $ torsocks catlfish/tools/fetchallblobs.py --store gaol-store --publickey logs gaol.ct.nordu.net.pem http://mvkhztpvqcxpdbn3.onion/ The base64 encoded entry can be extracted using this awk script: /-----END BLOB-----/{blob=0} //{if (blob) print} /-----BEGIN BLOB-----/{blob=1} Feeding that to base64 -d and gzip -d should produce a Tor consensus document. For example: $ cat 00000080 | awk '/-----END BLOB-----/{blob=0}//{if (blob) print}/-----BEGIN BLOB-----/{blob=1}'| base64 -d | gzip -d [1] $ git clone -b gaol6 https://git.nordu.net/user/linus/catlfish.git [2] https://www.ct.nordu.net/gaol.ct.nordu.net.pem https://www.ct.nordu.net/gaol.ct.nordu.net.pem.asc

1 0

Post-quantum symetric crypto
by Jeff Burdges 22 Feb '16

22 Feb '16

Symmetric crypto might start worrying more about being post-quantum soon : http://arxiv.org/abs/1602.05973 Best, Jeff

1 0

Help with TOR on UDP/QUIC
by Xiaofan Li 20 Feb '16

20 Feb '16

Hi everyone: I'm new to Tor dev community and my name is Xiaofan Li, currently a senior at Carnegie Mellon University studying ECE and CS. My friend Kevin Ku and I are taking a graduate class <http://www.cs.cmu.edu/%7Esrini/15-744/S16/> on computer networks and we decided to examine the possibilities of substituting TCP with the Google QUIC protocol for Tor in order to improve performance. We are emailing you because: 1. We want to get some points of contact with the Tor community in case of future integration and/or testing. 2. We want to know if *anyone else* has done (or is doing) Tor with QUIC. If so, what their status is; and if not, why not? 3. We want to get *your opinions* on this idea. Attached is our (very) preliminary plan and goals for the project. Any feedback is welcomed. 4. Any *implementation recommendations*. My plan is to find a clean layer of abstraction where I can substitute TCP with QUIC. Any ideas? On a first look, I'm thinking about either *or/channel.c* or *or/transports.c * 5. Any *testing suggestions*? How do Tor engineers test new stuff? Please take a look at our outline attached below. Thank you! Looking forward to hearing from you soon! Li.

3 3

Summary of meek's costs moved to tor-project list
by David Fifield 20 Feb '16

20 Feb '16

I decided to move the meek cost emails to the tor-project list, because they are more project-y than dev-y. Here is the email for January 2016: https://lists.torproject.org/pipermail/tor-project/2016-February/000101.html There's a table of all previous summaries here: https://trac.torproject.org/projects/tor/wiki/doc/meek#Costs

1 0

Analysis of ASan usage
by Jens Kubieziel 18 Feb '16

18 Feb '16

Hi, FYI: oss-security lately had a posting with the title »Address Sanitizer local root« (<URL:http://www.openwall.com/lists/oss-security/2016/02/17/9>) The author showed that building a suid binary with ASan enables local root exploits. He also shows some other problems with this approach. In his posting he mentions the Tor Browser and recommends to not use the word »hardened«, because it is misleading. -- Jens Kubieziel http://www.kubieziel.de Vielleicht verdirbt Geld tatsächlich den Charakter. Auf keinen Fall aber macht ein Mangel an Geld ihn besser. Jonathan Swift

1 0

Last network team/dev meetings of February: Wednesday and Friday and Monday!
by Nick Mathewson 17 Feb '16

17 Feb '16

Hi, all! Wednesday Feb 17 morning at 0830 EST (==1330 UTC) is the regular Tor/Network team development meeting. Friday Feb 19 at 0900 EST (==1400 UTC) is the discussion meeting for proposals 257 and 258. Monday Feb 22 at 2000 EST (==Tuesday at 0100 UTC) is the secondary meeting for folks who find the Wednesday meetings totally inconvenient. I'll try not to be asleep this time! We'll be skipping online meetings Feb 24 through Mar 2, since many people will be traveling. Or meeting face-to-face! We've been scheduling things on https://trac.torproject.org/projects/tor/wiki/org/teams/NetworkTeam/Meeting… . So far, it's been working well! peace to all, -- Nick

1 0

Two major improvements to the Tor Metrics back-end
by Karsten Loesing 15 Feb '16

15 Feb '16

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hello list, as some of you may have noticed, Tor Metrics had some trouble updating its aggregate data lately, and the issues there are by no means completely gone now. But I managed to tweak two of the aggregations that previously took many hours. Here's a graph: https://people.torproject.org/~karsten/volatile/metrics-modules-2016-01-07.… The green dots at the end are from the latest update. Changes affecting the 70-run-hidserv-stats.sh module: https://gitweb.torproject.org/metrics-web.git/commit/?id=c0f4c76 https://gitweb.torproject.org/metrics-web.git/commit/?id=6054ddd Changes affecting the 80-run-clients-stats.sh module: https://gitweb.torproject.org/metrics-web.git/commit/?id=44acfe5 Boom! All the best, Karsten -----BEGIN PGP SIGNATURE----- Comment: GPGTools - http://gpgtools.org iQEcBAEBAgAGBQJWjprFAAoJEJD5dJfVqbCrPT0IALX6dNrL3PK2gbbwAGmm4OBg 4kuz7qFMshTNFp8iT6HB5H9G9/wldgAPanZn+zLlFM0sOBoY1d16RoL2+ay+VKXQ xTeY85tgqMAnbaS2lQcBRZy5j8RXPui94g/ZcKnw9Bg3TpwLFO2+cMXqh1nhMLtb V+Lj0EE+y7qjdtOQO/IbOQzeUr/S9xtze44dRkGlZ0NmeKQsHXaR3m8w8ReIP32k vZekymZkUgbfzYeRoB+jI+3M3l0CF5agwIelhhHlnWwkJ70jPPVrAGX3rhb8X9SX iSNMUJTG62AC3WLqdRkhyY2YJ5OHSMLtpvQwOaOzqSs1yHWylHDVSLTZx+XCQmM= =tvfK -----END PGP SIGNATURE-----

3 4

Re: [tor-dev] Existing Tor Guard Selection Algorithm
by George Kadianakis 15 Feb '16

15 Feb '16

Reinaldo Junior <rjunior(a)thoughtworks.com> writes: > On Thu, Feb 11, 2016 at 7:55 AM, George Kadianakis <desnacked(a)riseup.net> > wrote: > >> Chelsea Komlo <ckomlo(a)thoughtworks.com> writes: >> >> > Hi George, >> > >> > Thanks for your help with this! >> > >> > We wrote up our high-level understanding of the current Tor guard >> selection >> > algorithm here: >> > >> > https://gist.github.com/chelseakomlo/2acbe15314b5a809c6f4 >> > >> > This has more than our python simulation, but less than the actual Tor >> > implementation. For example, it is missing conditions like prioritization >> > by uptime, capacity, etc. >> > >> > >> https://github.com/twstrike/tor_guardsim/blob/develop/lib/original_client.py >> > >> > If you wouldn't mind taking a look at this and letting us know anything >> > that is missing/should change, that would be really helpful. >> > >> >> Your pseudocode looks pretty accurate based on what I remember from the >> guard >> algorithm and from quickly skimming the code. >> >> One inaccuracy might be that if IIRC in RECEIVE_NEW_CONSENSUS we don't >> remove >> guards that are not listed in the latest consensus. Instead we mark them >> as 'bad' >> which rules them out of retries etc. till they reappear on the consensus. >> See entry_guard_set_status() and its caller for how this works. >> > > <snip> > > See: > https://github.com/twstrike/tor_guardsim/commit/1fb7400c0c8e167508753ede15b… Hmm, looks pretty good! One note: I feel that you are hiding information behind the simple phrase "Build LIVE_GUARDS". Interesting things happen in populate_live_entry_guards() that are not documented by your algo. For example, in entry_is_live() we call entry_is_time_to_retry(), which periodically retries guards based on time: struct guard_retry_period_s periods[] = { { 6*60*60, 60*60 }, /* For first 6 hrs., retry hourly; */ { 3*24*60*60, 4*60*60 }, /* Then retry every 4 hrs. until the 3-day mark; */ { 7*24*60*60, 18*60*60 }, /* After 3 days, retry every 18 hours until 1 week mark. */ { TIME_MAX, 36*60*60 } /* After 1 week, retry every 36 hours. */ }; Also, another function that I'm not sure if you have seen (and whether it fits with the algorithm) is entries_retry_all(). BTW, I like the way you have splitted the algo into various events (RECEIVE_NEW_CONSENSUS, BUILD_NEW_CIRCUIT, CHOOSE_A_GUARD, CONNECT_ENTRY_GUARD, etc.). We should maybe try to shape Ola's new algorithm to fit this event structure exactly since it resembles the actual Tor networking API. Cheers!

1 0

Possibly rescheduling monday meeting for prop#265
by Nick Mathewson 15 Feb '16

15 Feb '16

Hi! Looking at the schedule at https://trac.torproject.org/projects/tor/wiki/org/teams/NetworkTeam/Meeting… , it does not appear that there is a time when Mike (the proposal author) could make it. I'll check my email in the morning, but it's possible we'll have to discuss prop#265 on another day.

1 0

Applications team prep for Valencia
by Isabela 13 Feb '16

13 Feb '16

Hello there! As we approach our 2016 Winter Tor's Dev Meeting in Valencia [1], I am trying to help the different teams prep for discussions we will have there. I hope the Applications team take some time to review your retrospective notes[2] to see what the team have done from that list and what it haven't. Also, to think about other points that could help improve the team organization process. We hope to have another retrospective session in Valencia and that will help prep for it. Another exercise I would like each dev team that plans to be in Valencia to do is to review the roadmaps they created in Berlin. We will be planning another 6 months roadmap (from Winter Dev meeting till Summer Dev meeting), it will be useful for the meeting to look at what you planned, what got done, what you want to carry on for this next 6months we will be planning and what you think its better to drop or leave for latter. I added Tor Browser and Tor Messenger roadmaps to the Applications wiki page [3]. I plan on adding other Applications dev teams stuff there too, if you need help finding the roadmap you did you let me know. Or if you need help with anything else. I am available on IRC as well if you have questions. Cheers, Isabela [1] https://trac.torproject.org/projects/tor/wiki/org/meetings/2016WinterDevMee… [2] https://trac.torproject.org/projects/tor/wiki/org/meetings/2015SummerDevMee… [3] https://trac.torproject.org/projects/tor/wiki/org/teams/ApplicationsTeam -- PM at TorProject.org gpg fingerprint = 8F2A F9B6 D4A1 4D03 FDF1 B298 3224 4994 1506 4C7B @isa

1 0

Proposal: Rendezvous Single Onion Services
by Tim Wilson-Brown - teor 12 Feb '16

12 Feb '16

Hi All, Please find below and attached a proposal: Rendezvous Single Onion Services. This is an updated and expanded version of "Direct Onion Services: Fast-but-not-hidden services”. It also borrows heavily from "Single Onion Services" (Proposal #252). The proposal is available in the branch “feature-17178-rsos” at https://github.com/teor2345/torspec.git <https://github.com/teor2345/torspec.git> as torspec/proposals/ideas/xxx-rend-single-onion.txt Work on this proposal is being tracked in trac ticket #17178 at https://trac.torproject.org/projects/tor/ticket/17178 <https://trac.torproject.org/projects/tor/ticket/17178> There is a basic implementation in the branch “feature-17178-rsos” at https://github.com/teor2345/tor.git <https://github.com/teor2345/tor.git> This can be tested with the chutney branch "feature-17178-rsos” at https://github.com/teor2345/chutney.git <https://github.com/teor2345/chutney.git> using the command: src/test/test-network.sh --flavor rsos-min Regards, Tim ----- Filename: xxx-rend-single-onion.txt Title: Rendezvous Single Onion Services Author: Tim Wilson-Brown, John Brooks, Aaron Johnson, Rob Jansen, George Kadianakis, Paul Syverson, Roger Dingledine Created: 2015-10-17 Status: Draft 1. Overview Rendezvous single onion services are an alternative design for single onion services, which trade service-side location privacy for improved performance, reliability, and scalability. Rendezvous single onion services have a .onion address identical to any other onion service. The descriptor contains the same information as the existing double onion (hidden) service descriptors. The introduction point and rendezvous protocols occur as in double onion services, with one modification: one-hop connections are made from the onion server to the introduction and rendezvous points. This proposal is a revision of the unnumbered proposal Direct Onion Services: Fast-but-not-hidden services by Roger Dingledine, and George Kadianakis at https://lists.torproject.org/pipermail/tor-dev/2015-April/008625.html It incorporates much of the discussion around hidden services since April 2015, including content from Single Onion Services (Proposal #252) by John Brooks, Paul Syverson, and Roger Dingledine. 2. Motivation Rendezvous single onion services are best used by sites which: * Don’t require location anonymity * Would appreciate lower latency or self-authenticated addresses * Would like to work with existing tor clients and relays * Can’t accept connections to an open ORPort Rendezvous single onion services have a few benefits over double onion services: * Connection latency is lower, as one-hop circuits are built to the introduction and rendezvous points, rather than three-hop circuits * Stream latency is reduced on a four-hop circuit * Less Tor network capacity is consumed by the service, as there are fewer hops (4 rather than 6) between the client and server via the rendezvous point Rendezvous single onion services have a few benefits over single onion services: * A rendezvous single onion service can load-balance over multiple rendezvous backends (see proposal #255) * A rendezvous single onion service doesn't need an accessible ORPort (it works behind a NAT, and in server enclaves that only allow outward connections) * A rendezvous single onion service is compatible with existing tor clients, hidden service directories, introduction points, and rendezvous points Rendezvous single onion services have a few drawbacks over single onion services: * Connection latency is higher, as one-hop circuits are built to the introduction and rendezvous points. Single onion services perform one extend to the single onion service’s ORPort only It should also be noted that, while single onion services receive many incoming connections from different relays, rendezvous single onion services make many outgoing connections to different relays. This should be taken into account when planning the connection capacity of the infrastructure supporting the onion service. Rendezvous single onion services are not location hidden on the service side, but clients retain all of the benefits and privacy of onion services. (The rationale for the 'single' and 'double' nomenclature is described in section 7.4 of proposal #252.) We believe that it is important for the Tor community to be aware of the alternative single onion service designs, so that we can reach consensus on the features and tradeoffs of each design. However, we recognise that each additional flavour of onion service splits the anonymity set of onion service users. Therefore, it may be best for user anonymity that not all designs are adopted, or that mitigations are implemented along with each additional flavour. (See sections 8 & 9 for a further discussion.) 3. Onion descriptors The rendezvous single onion descriptor format is identical to the double onion descriptor format. 4. Reaching a rendezvous single onion service as a client Clients reach rendezvous single onion services in an identical fashion to double onion services. The rendezvous design means that clients do not know whether they are talking to a double or rendezvous single onion service, unless that service tells them. (This may be a security issue.) However, the use of a four-hop path between client and rendezvous single onion service may be statistically distinguishable. (See section 8 for further discussion of security issues.) (Please note that this proposal follows the hop counting conventions in the tor source code. A circuit with a single connections between the client and the endpoint is one-hop, a circuit with 4 connections (and 3 nodes) between the client and endpoint is four-hop.) 5. Publishing a rendezvous single onion service To act as a rendezvous single onion service, a tor instance (or cooperating group of tor instances) must: * Publish onion descriptors in the same manner as any onion service, using three-hop circuits. This avoids service blocking by IP address, proposal #224 (next-generation hidden services) avoids blocking by onion address. * Perform the rendezvous protocol in the same manner as a double onion service, but make the intro and rendezvous connections one-hop. (This may allow intro and rendezvous points to block the service.) 5.1. Configuration options 5.1.1 RendezvousSingleOnionServiceNonAnonymousServer The tor instance operating a rendezvous single onion service must make one-hop circuits to the introduction and rendezvous points: RendezvousSingleOnionServiceNonAnonymousServer 0|1 If set, make one-hop circuits between the Rendezvous Single Onion Service server, and the introduction and rendezvous points. This option makes every onion service instance hosted by this tor instance a Rendezvous Single Onion Service. (Default: 0) Because of the grave consequences of misconfiguration here, we have added ‘NonAnonymous’ to the name of the torrc option. Furthermore, Tor MUST issue a startup warning message to operators of the onion service if this feature is enabled. [Should the name start with ‘NonAnonymous’ instead?] As RendezvousSingleOnionServiceNonAnonymousServer modifies the behaviour of every onion service on a tor instance, it is impossible to run hidden (double onion) services and rendezvous single onion services on the same tor instance. This is considered a feature, as it prevents hidden services from being discovered via rendezvous single onion services on the same tor instance. 5.1.2 Recommended Additional Options: Correctness Based on the experiences of Tor2Web with one-hop paths, operators should consider using the following options with every rendezvous single onion service, and every single onion service: UseEntryGuards 0 One-hop paths do not use entry guards. This also deactivates the entry guard pathbias code, which is not compatible with one-hop paths. Entry guards are a security measure against Sybil attacks. Unfortunately, they also act as the bottleneck of busy onion services and overload those Tor relays. LearnCircuitBuildTimeout 0 Learning circuit build timeouts is incompatible with one-hop paths. It also creates additional, unnecessary connections. Perhaps these options should be set automatically on (rendezvous) single onion services. Tor2Web sets these options automatically: UseEntryGuards 0 LearnCircuitBuildTimeout 0 5.1.3 Recommended Additional Options: Performance LongLivedPorts The default LongLivedPorts setting creates additional, unnecessary connections. This specifies no long-lived ports (the empty list). PredictedPortsRelevanceTime 0 seconds The default PredictedPortsRelevanceTime setting creates additional, unnecessary connections. RendPostPeriod 0 seconds This option typically hides the startup time of a hidden service by randomly posting over a 2 hour period. Since single onion services value speed over anonymity, they can post descriptors straight away. (Actually, 30 seconds after they bootstrap, for descriptor stability.) However, we do not recommend setting the following option to 1, unless bug #17359 is resolved so tor onion services can bootstrap without predicted circuits. __DisablePredictedCircuits 0 This option disables all predicted circuits. It is equivalent to: LearnCircuitBuildTimeout 0 LongLivedPorts PredictedPortsRelevanceTime 0 seconds And turning off hidden service server preemptive circuits, which is currently unimplemented (#17360) 5.1.3 Recommended Additional Options: Security We recommend that no other services are run on a rendezvous single onion service tor instance. Since tor runs as a client (and not a relay) by default, rendezvous single onion service operators should set: SocksPort 0 Disallow connections from client applications to the tor network via this tor instance. ClientOnly 1 Even if the defaults file configures this instance to be a relay, never relay any traffic or serve any descriptors. 5.2. Publishing descriptors A single onion service must publish descriptors in the same manner as any onion service, as defined by rend-spec. 5.3. Authorization Client authorization for a rendezvous single onion service is possible via the same methods used for double onion services. 6. Related Proposals, Tools, and Features 6.1. Load balancing High capacity services can distribute load and implement failover by: * running multiple instances that publish to the same onion service directories, * publishing descriptors containing multiple introduction points (OnionBalance), * publishing different introduction points to different onion service directories (OnionBalance upcoming(?) feature), * handing off rendezvous to a different tor instance via control port messages (proposal #255), or by a combination of these methods. 6.2. Ephemeral single onion services (ADD_ONION) The ADD_ONION control port command could be extended to support ephemerally configured rendezvous single onion services. Given that RendezvousSingleOnionServiceNonAnonymousServer modifies the behaviour of all onion services on a tor instance, if it is set, any ephemerally configured onion service should become a rendezvous single onion service. 6.3. Proposal 224 ("Next-Generation Hidden Services") This proposal is compatible with proposal 224, with onion services acting just like a next-generation hidden service, but making one-hop paths to the introduction and rendezvous points. 6.4. Proposal 246 ("Merging Hidden Service Directories and Intro Points") This proposal is compatible with proposal 246. The onion service will publish its descriptor to the introduction points in the same manner as any other onion service. Clients will use the merged hidden service directory and introduction point just as they do for other onion services. 6.5. Proposal 252 ("Single Onion Services") This proposal is compatible with proposal 252. The onion service will publish its descriptor to the introduction points in the same manner as any other onion service. Clients can then choose to extend to the single onion service, or continue with the rendezvous protocol. Running a rendezvous single onion service and single onion service allows older clients to connect via rendezvous, and newer clients to connenct via extend. This is useful for the transition period where not all clients support single onion services. 6.5. Proposal 255 ("Hidden Service Load Balancing") This proposal is compatible with proposal 255. The onion service will perform the rendezvous protocol in the same manner as any other onion service. Controllers can then choose to handoff the rendezvous point connection to another tor instance, which should also be configured as a rendezvous single onion service. 7. Considerations 7.1 Modifying RendezvousSingleOnionServiceNonAnonymousServer at runtime Implementations should not reuse introduction points or introduction point circuits if the value of RendezvousSingleOnionServiceNonAnonymousServer is different than it was when the introduction point was selected. This is because these circuits will have an undesirable length. There is specific code in tor that preserves introduction points on a HUP, if RendezvousSingleOnionServiceNonAnonymousServer has changed, all circuits should be closed, and all introduction points must be discarded. 7.2 Delaying connection expiry Tor clients typically expire connections much faster than tor relays [citation needed]. (Rendezvous) single onion service operators may find that keeping connections open saves on connection latency. However, it may also place an additional load on the service. (This could be implemented by increasing the configured connection expiry time.) 7.3. (No) Benefit to also running a Tor relay In tor Trac ticket #8742, running a relay and hidden onion service on the same tor instance was disabled for security reasons. While there may be benefits to running a relay on the same instance as a rendezvous single onion service (existing connections mean lower latency, it helps the tor network overall), a security analysis of this configuration has not yet been performed. In addition, a potential drawback is overloading a busy single onion service. 6.4 Predicted circuits We should look whether we can optimize further the predicted circuits that Tor makes as a onion service for this mode. 8. Security Implications 8.1 Splitting the Anonymity Set Each additional flavour of onion service, and each additional externally visible onion service feature, provides oportunities for fingerprinting. Also, each additional type of onion service shrinks the anonymity set for users of double onion (hidden) services who require server location anonymity. These users benefit from the cover provided by current users of onion services, who use them for client anonymity, self-authentication, NAT-punching, or other benefits. For this reason, features that shrink the double onion service anonymity set should be carefully considered. The benefits and drawbacks of additional features also often depend on a particular threat model. It may be that a significant number of users and sites adopt (rendezvous) single onion services due to their benefits. This could increase the traffic on the tor network, therefore increasing anonymity overall. However, the unique behaviour of each type of onion service may still be distinguishable from both the client and server ends of the connection. 8.2 Hidden Service Designs can potentially be more secure As a side-effect, by optimizing for performance in this feature, it allows us to lean more heavily towards security decisions for regular onion services. 8.3 One-hop onion service paths may encourage more attacks There's a possible second-order effect here since both encrypted services and hidden services will have foo.onion addresses and it's not clear based on the address whether the service will be hidden -- if *some* .onion addresses are easy to track down, are we encouraging adversaries to attack all rendezvous points just in case? 9. Further Work Further proposals or research could attempt to mitigate the anonymity-set splitting described in section 8. Here are some initial ideas. 9.1 Making Client Exit connections look like Client Onion Service Connections A mitigation to this fingerprinting is to make each (or some) exit connections look like onion service connections. This provides cover for particular types of onion service connections. Unfortunately, it is not possible to make onion service connections look like exit connections, as there are no suitable dummy servers to exit to on the Internet. 9.1.1 Making Client Exit connections perform Descriptor Downloads (Some) exit connections could perform a dummy descriptor download. (However, descriptors for recently accessed onion services are cached, so dummy downloads should only be performed occasionally.) Exit connections already involve a four-hop "circuit" to the server (including the connection between the exit and the server on the Internet). The server on the Internet is not included in the consensus. Therefore, this mitigation would effectively cover single onion services which are not relays. 9.1.2 Making Client Exit connections perform the Rendezvous Protocol (Some) exit connections could perform a dummy rendezvous protocol. Exit connections already involve a four-hop "circuit" to the server (including the connection between the exit and the server on the Internet). Therefore, this mitigation would effectively cover rendezvous single onion services, as long as a dummy descriptor download was also performed occasionally. 9.1.3 Making Single Onion Service rendezvous points perform name resolution Currently, Exits perform DNS name resolution, and changing this behaviour would cause unacceptable connection latency. Therefore, we could make onion service connections look like exit connections by making the rendezvous point do name resolution (that is, descriptor fetching), and, if needed, the introduction part of the protocol. This could potentially *reduce* the latency of single onion service connections, depending on the length of the paths used by the rendezvous point. However, this change makes rendezvous points almost as powerful as Exits, a careful security analysis will need to be performed before this is implemented. There is also a design issue with rendezvous name resolution: a client wants to leave resolution (descriptor download) to the RP, but it doesn't know whether it can use the exit-like protocol with an RP until it has downloaded the descriptor. This might mean that single onion services of both flavours need a different address style or address namespace. We could use .single.onion or something. (This would require an update to the HSDir code.) 9.2 Performing automated and common queries over onion services Tor could create cover traffic for a flavour of onion service by performing automated or common queries via an onion service of that type. In addition, onion service-based checks have security benefits over DNS-based checks. See Genuine Onion, Syverson and Boyce, 2015, at http://www.nrl.navy.mil/itd/chacs/syverson-genuine-onion-simple-fast-flexib… Here are some examples of automated queries that could be performed over an onion service: 9.2.1 torcheck over onion service torcheck ("Congratulations! This browser is configured to use Tor.") could be retrieved from an onion service. Incidentally, this would resolve the exitmap issues in #17297, but it would also fail to check that exit connections work, which is important for many Tor Browser users. 9.2.2 Tor Browser version checks over onion service Running tor browser version checks over an onion service seems to be an excellent use-case for onion services. It would also have the Tor Project "eating its own dogfood", that is, using onion services for its essential services. 9.2.3 Tor Browser downloads over onion service Running tor browser downloads over an onion service might require some work on the onion service codebase to support high loads, load-balancing, and failover. It is a good use case for a (rendezvous) single onion service, as the traffic over the tor network is only slightly higher than for Tor Browser downloads over tor. (4 hops for [R]SOS, 3 hops for Exit.) 9.2.4 SSL Observatory submissions over onion service HTTPS certificates could be submitted to HTTPS Everywhere's SSL Observatory over an onion service. This option is disabled in Tor Browser by default. Perhaps some users would be more comfortable enabling submission over an onion service, due to the additional security benefits. -----

2 2

Proposal: Load-balancing hidden services by splitting introduction from rendezvous
by Tom van der Woerdt 12 Feb '16

12 Feb '16

Hey all, I'd like your thoughts and comments on this proposal. Tom PS: If you want to deliver them in person, I'm in Berlin. Filename: xxx-intro-rendezvous-controlsocket.txt Title: Load-balancing hidden services by splitting introduction from rendezvous Author: Tom van der Woerdt Created: 2015-09-30 Status: draft 1. Overview and motivation To address scaling concerns with the onion web, we want to be able to spread the load of hidden services across multiple machines. OnionBalance is a great stab at this, and it can currently give us 60x the capacity by publishing 6 separate descriptors, each with 10 introduction points, but more is better. This proposal aims to address hidden service scaling up to a point where we can handle millions of concurrent connections. The basic idea involves splitting the 'introduce' from the 'rendezvous', in the tor implementation, and adding new events and commands to the control specification to allow intercepting introductions and transmitting them to different nodes, which will then take care of the actual rendezvous. External controller code could relay the data to another node or a pool of nodes, all which are run by the hidden service operator, effectively distributing the load of hidden services over multiple processes. By cleverly utilizing the current descriptor methods, we could publish up to sixty unique introduction points, which could translate to many thousands of parallel tor workers. This should allow hidden services to go multi-threaded, with a few small changes. 2. Specification We propose two additions to the control specification, of which one is an event and the other is a new command. We also introduce a new configuration option. 2.1. DisableAutomaticRendezvous configuration option The syntax is: "DisableAutomaticRendezvous" SP [1|0] CRLF This configuration option is defined to be a boolean toggle which, if set, stops the tor implementation from automatically doing a rendezvous when an INTRODUCE2 cell is received. Instead, an event will be sent to the controllers. If no controllers are present, the introduction cell should be dropped, as acting on it instead of dropping it could open a window for a DoS. For security reasons, the configuration should be made available only in the configuration files, and not as an option settable by the controller. 2.2. The "INTRODUCE" event The syntax is: "650" SP "INTRODUCE" SP RendezvousData CRLF RendezvousData = implementation-specific, but must not contain whitespace, must only contain human-readable characters, and should be no longer than 512 bytes The INTRODUCE event should contain sufficient data to allow continuing the rendezvous from another Tor instance. The exact format is left unspecified and left up to the implementation. From this follows that only matching versions can be used safely to coordinate the rendezvous of hidden service connections. 2.3. "PERFORM-RENDEZVOUS" command The syntax is: "PERFORM-RENDEZVOUS" SP RendezvousData CRLF This command allows a controller to perform a rendezvous using data received through an INTRODUCE event. The format of RendezvousData is not specified other than that it must not contain whitespace, and should be no longer than 512 bytes. 3. Compatibility and security The implementation of these methods should, ideally, not change anything in the network, and all control changes are opt-in, so this proposal is fully backwards compatible. Controllers handling this data must be careful to not leak rendezvous data to untrusted parties, as it could be used to intercept and manipulate hidden services traffic. 4. Example Let's take an example where a client (Alice) tries to contact Bob's hidden service. To do this, Bob follows the normal hidden service specification, except he sets up ten servers to do this. One of these publishes the descriptor, the others have this desabled. When the INTRODUCE2 cell arrives at the node which published the descriptor, it does not immediately try to perform the rendezvous, but instead outputs this to the controller. Through an out-of-band process this message is relayed to a controller of another node of Bob's, and this transmits the "PERFORM-RENDEZVOUS" command to that node. This node finally performs the rendezvous, and will continue to serve data to Alice, whose client will now not have to talk to the introduction point anymore. 5. Other considerations We have left the actual format of the rendezvous data in the control protocol unspecified, so that controllers do not need to worry about the various types of hidden service connections, most notably proposal 224. The decision to not implement the actual cell relaying in the tor implementation itself was taken to allow more advanced configurations, and to leave the actual load-balancing algorithm to the implementor of the controller. The developer of the tor implementation should not have to choose between a round-robin algorithm and something that could pull CPU load averages from a centralized monitoring system.

5 9

Existing Tor Guard Selection Algorithm
by Chelsea Komlo 11 Feb '16

11 Feb '16

Hi George, Thanks for your help with this! We wrote up our high-level understanding of the current Tor guard selection algorithm here: https://gist.github.com/chelseakomlo/2acbe15314b5a809c6f4 This has more than our python simulation, but less than the actual Tor implementation. For example, it is missing conditions like prioritization by uptime, capacity, etc. https://github.com/twstrike/tor_guardsim/blob/develop/lib/original_client.py If you wouldn't mind taking a look at this and letting us know anything that is missing/should change, that would be really helpful. Thanks! Chelsea

2 1

Re: [tor-dev] Entry guards, primary guards, dir guards
by George Kadianakis 10 Feb '16

10 Feb '16

Ola Bini <obini(a)thoughtworks.com> writes: > Hi, > >> So maybe the simple answer here is that if prop247 is enabled (this could be a >> NumGuards=N argument to our algorithm), instead of always returning the first >> reachable guard, we instead build a list of the first N reachable guards, and >> randomly choose one of them. Could this work? (see #12466 for a weird behavior >> that will make us skip bugs if NumEntryGuards != 1) > Ah, I understand. We could definitely do that. I think the easiest way > is to just have a parameter to the existing algorithm with > ExcludeNodes. We can then repeat it N times sending in the previous > nodes chosen as ExcludeNodes. This will generate the N reachable first > guards. Then, we can just choose randomly from them. > > Does that sound reasonable? > Yes, that sounds like a reasonable way to achieve this. Maybe N_PRIMARY_GUARDS should also scale up in that case, so that N_PRIMARY_GUARDS >= N.

2 1

Re: [tor-dev] Detailed algorithm
by George Kadianakis 09 Feb '16

09 Feb '16

Ola Bini <obini(a)thoughtworks.com> writes: > OK, with your feedback and thinking a bit more about it, here is a > revision of the algorithm from yesterday. I think we are starting to > get close so we will rip out the original simulation code and > implement something that matches this now. Hopefully, the changes will > be smaller. > > - Start of algorithm (arguments: USED_GUARDS, EXCLUDE_NODES) > - If selecting directory guards, GUARDS is all guards from the consensus with the V2Flag, > otherwise GUARDS is all guards from the consensus > - Set UTOPIC_GUARDS to be all guards to use under utopic conditions from GUARDS > - Set DYSTOPIC_GUARDS to be all guards to use under dystopic conditions from GUARDS > - Set REMAINING_UTOPIC_GUARDS to be UTOPIC_GUARDS without EXCLUDE_NODES > - Set REMAINING_DYSTOPIC_GUARDS to be DYSTOPIC_GUARDS without EXCLUDE_NODES > - Create a list of PRIMARY_GUARDS that contain N_PRIMARY_GUARDS that are not bad by: > - Taking the next entry from USED_GUARDS > - If USED_GUARDS is empty: > - randomly select an entry from UTOPIC_GUARDS, weighted by bandwidth > - Set TRIED_GUARDS to be an empty set > - Set TRIED_DYSTOPIC_GUARDS to be an empty set > - Set state = STATE_PRIMARY_GUARDS > > - Each iteration of algorithm > - If a new consensus has arrived: > - Update all guard profiles with new bad/non-bad information > - If any PRIMARY_GUARDS have become bad: > - re-add to the list of PRIMARY_GUARDS using the same procedure > - If any USED_GUARDS have become non-bad: > - add it back to PRIMARY_GUARDS at the place it would have been if > it was non-bad when running the start of the algorithm. If this > results in PRIMARY_GUARDS being larger than N_PRIMARY_GUARDS, > remove from the end of the list until the list is N_PRIMARY_GUARDS long > - Ensure that UTOPIC_GUARDS and DYSTOPIC_GUARDS are updated with the changes > from the consensus > > - If it was at least 3 minutes since we tried the primary guards and we are not in STATE_PRIMARY_GUARDS: > - save previous state > - set state = STATE_PRIMARY_GUARDS > > - STATE_PRIMARY_GUARDS: > - return each entry in PRIMARY_GUARDS in turn > - mark each entry as "unreachable" if algorithm doesn't terminate > - restore previous state (or STATE_TRY_UTOPIC if no previous) > > - STATE_TRY_UTOPIC: > - for each entry in TRIED_GUARDS that was marked as unreachable more than 20 minutes ago > - add it back to REMAINING_UTOPIC_GUARDS > - return each remaining entry from USED_GUARDS in turn > - for each entry, if algorithm doesn't terminate > - mark entry as "unreachable" > - add entry to TRIED_GUARDS > - if the number of entries in TRIED_GUARDS that were tried within GUARDS_TRY_THRESHOLD_TIME > is larger than GUARDS_TRY_THRESHOLD, return failure from the algorithm > - if the number of entries in TRIED_GUARDS is larger than a GUARDS_FAILOVER_THRESHOLD > proportion of UTOPIC_GUARDS, set state = STATE_TRY_DYSTOPIC > - return each entry from REMAINING_UTOPIC_GUARDS randomly selected, weighted by bandwidth > - remove the returned entry from REMAINING_UTOPIC_GUARDS > - for each entry, if algorithm doesn't terminate > - mark entry as "unreachable" > - add entry to TRIED_GUARDS > - if the number of entries in TRIED_GUARDS that were tried within GUARDS_TRY_THRESHOLD_TIME > is larger than GUARDS_TRY_THRESHOLD, return failure from the algorithm > - if the number of entries in TRIED_GUARDS is larger than a GUARDS_FAILOVER_THRESHOLD > proportion of UTOPIC_GUARDS, set state = STATE_TRY_DYSTOPIC > > - STATE_TRY_DYSTOPIC: > - for each entry in TRIED_DYSTOPIC_GUARDS that was marked as unreachable more than 20 minutes ago > - add it back to REMAINING_DYSTOPIC_GUARDS > - return each remaining DYSTOPIC entry from USED_GUARDS in turn > - for each entry, if algorithm doesn't terminate > - mark entry as "unreachable" > - add entry to TRIED_DYSTOPIC_GUARDS > - if the number of entries in TRIED_GUARDS+TRIED_DYSTOPIC_GUARDS that were tried within GUARDS_TRY_THRESHOLD_TIME > is larger than GUARDS_TRY_THRESHOLD, return failure from the algorithm > - if the number of entries in TRIED_DYSTOPIC_GUARDS is larger than a GUARDS_FAILOVER_THRESHOLD > proportion of DYSTOPIC_GUARDS: > - mark all guards in PRIMARY_GUARDS, TRIED_GUARDS and TRIED_DYSTOPIC_GUARDS as not "unreachable" > - return failure from the algorithm > - return each entry from REMAINING_DYSTOPIC_GUARDS randomly selected, weighted by bandwidth > - remove the returned entry from REMAINING_DYSTOPIC_GUARDS > - for each entry, if algorithm doesn't terminate > - mark entry as "unreachable" > - add entry to TRIED_DYSTOPIC_GUARDS > - if the number of entries in TRIED_GUARDS+TRIED_DYSTOPIC_GUARDS that were tried within GUARDS_TRY_THRESHOLD_TIME > is larger than GUARDS_TRY_THRESHOLD, return failure from the algorithm > - if the number of entries in TRIED_DYSTOPIC_GUARDS is larger than a GUARDS_FAILOVER_THRESHOLD > proportion of DYSTOPIC_GUARDS: > - mark all guards in PRIMARY_GUARDS, TRIED_GUARDS and TRIED_DYSTOPIC_GUARDS as not "unreachable" > - return failure from the algorithm > > - End of algorithm > - If circuit is set up correctly, let algorithm know > - Algorithm marks the guard chosen as used and makes sure it is in USED_GUARDS > - Otherwise do another run of the algorithm > Ah great! Good improvements. I think we are going the right way. Here is another quick review. I include a second copy of the algorithm and comment inline: > > - Start of algorithm (arguments: USED_GUARDS, EXCLUDE_NODES) > - If selecting directory guards, GUARDS is all guards from the consensus with the V2Flag, > otherwise GUARDS is all guards from the consensus > - Set UTOPIC_GUARDS to be all guards to use under utopic conditions from GUARDS > - Set DYSTOPIC_GUARDS to be all guards to use under dystopic conditions from GUARDS > - Set REMAINING_UTOPIC_GUARDS to be UTOPIC_GUARDS without EXCLUDE_NODES > - Set REMAINING_DYSTOPIC_GUARDS to be DYSTOPIC_GUARDS without EXCLUDE_NODES Maybe we also need to exclude USED_GUARDS from these two lists? > - Create a list of PRIMARY_GUARDS that contain N_PRIMARY_GUARDS that are not bad by: > - Taking the next entry from USED_GUARDS > - If USED_GUARDS is empty: > - randomly select an entry from UTOPIC_GUARDS, weighted by bandwidth > - Set TRIED_GUARDS to be an empty set > - Set TRIED_DYSTOPIC_GUARDS to be an empty set > - Set state = STATE_PRIMARY_GUARDS > > - Each iteration of algorithm > - If a new consensus has arrived: > - Update all guard profiles with new bad/non-bad information > - If any PRIMARY_GUARDS have become bad: > - re-add to the list of PRIMARY_GUARDS using the same procedure > - If any USED_GUARDS have become non-bad: > - add it back to PRIMARY_GUARDS at the place it would have been if > it was non-bad when running the start of the algorithm. If this > results in PRIMARY_GUARDS being larger than N_PRIMARY_GUARDS, > remove from the end of the list until the list is N_PRIMARY_GUARDS long > - Ensure that UTOPIC_GUARDS and DYSTOPIC_GUARDS are updated with the changes > from the consensus > Not sure if this is part of this algorithm, or it's actually another helper algorithm that is called when a consensus arrives. I feel it might be cleaner if we do it as a separate algo, but we can proceed like this as well since it's not too confusing. > - If it was at least 3 minutes since we tried the primary guards and we are not in STATE_PRIMARY_GUARDS: > - save previous state > - set state = STATE_PRIMARY_GUARDS > > - STATE_PRIMARY_GUARDS: > - return each entry in PRIMARY_GUARDS in turn > - mark each entry as "unreachable" if algorithm doesn't terminate So IIUC the "algorithm" referenced here is _not_ the algorithm that we are describing right now (let's call it ALGO_CHOOSE_GUARD). Instead the "algorithm" here is the _caller of ALGO_CHOOSE_GUARD_, which is the algorithm responsible for creating circuits, testing whether they work and reporting the results back to CHOOSE_GUARD_ALGO (let's call this other algorithm ALGO_BUILD_CIRCUIT). Maybe we can clarify this? Also, do PRIMARY_GUARDS go into TRIED_GUARDS and count against our thresholds? > - restore previous state (or STATE_TRY_UTOPIC if no previous) > > - STATE_TRY_UTOPIC: > - for each entry in TRIED_GUARDS that was marked as unreachable more than 20 minutes ago > - add it back to REMAINING_UTOPIC_GUARDS > - return each remaining entry from USED_GUARDS in turn > - for each entry, if algorithm doesn't terminate In general, I think we should make the context of this algorithm (ALGO_CHOOSE_GUARD) a bit clearer, so that we know when "Start of algorithm" is supposed to run, and when "End of algorithm" is supposed to run. Because for example one could think here that in an asynchronous setting, when we try a entry from USED_GUARDS and find out whether the circuit succeeded or not, we need to run the algorithm from the beginning including the "Start of algorithm" step. Whereas if I understand correctly you assume that we will just drop in and continue exactly from this point. I feel that this confusion is caused by the ALGO_CHOOSE_GUARD algorithm being pseudo-asynchronous. To address this I suggested additional states for when we are waiting for the results of a circuit construction, but I agree that this might complicate things too much. Is there a way we can make this cleaner? > - mark entry as "unreachable" > - add entry to TRIED_GUARDS > - if the number of entries in TRIED_GUARDS that were tried within GUARDS_TRY_THRESHOLD_TIME > is larger than GUARDS_TRY_THRESHOLD, return failure from the algorithm > - if the number of entries in TRIED_GUARDS is larger than a GUARDS_FAILOVER_THRESHOLD > proportion of UTOPIC_GUARDS, set state = STATE_TRY_DYSTOPIC > - return each entry from REMAINING_UTOPIC_GUARDS randomly selected, weighted by bandwidth > - remove the returned entry from REMAINING_UTOPIC_GUARDS > - for each entry, if algorithm doesn't terminate > - mark entry as "unreachable" > - add entry to TRIED_GUARDS > - if the number of entries in TRIED_GUARDS that were tried within GUARDS_TRY_THRESHOLD_TIME > is larger than GUARDS_TRY_THRESHOLD, return failure from the algorithm > - if the number of entries in TRIED_GUARDS is larger than a GUARDS_FAILOVER_THRESHOLD > proportion of UTOPIC_GUARDS, set state = STATE_TRY_DYSTOPIC > > <snip> Finally, what kind of statistics and measurements does the simulator conduct? For example, I can think of reachability related stats like: * Time (or number of guards) till we manage to build first circuit * Time till we manage to recover from flaky network and I can also think of security related stats like: * Number of guards we tried before succeeding first circuit * Number of guards we exposed ourselves to after time t etc.

2 1

Re: [tor-dev] Entry guards, primary guards, dir guards
by George Kadianakis 09 Feb '16

09 Feb '16

Ola Bini <obini(a)thoughtworks.com> writes: > Hey, > > Maybe I misunderstood the hard part - I thought the problem was to > choose the NUM longlived vanguards - since there are only ever NUM > possible guards at each level, not to choose which one to use among > the NUM guards. For the first, it felt like using 259 would work, and > the for the second use case, just randomly choose one of the NUM > guards. > Hmm, let's try to formalize this slightly. Maybe it's helpful. I'm doing this a bit rushed, so maybe I'm wrong. So far we've been designing an algorithm that: a) populates and manages our guardlist b) picks the right guard to use from our guardlist We could in theory decouple these two steps, but so far we've been mixing them together. Step (b) has been easy so far because Tor uses a single guard node for general circuits, so we always pick the very first reachable guard from our guardlist every time. However, when we consider prop247 or directory guards we don't always want to pick the very first available guard. For example, when considering the layer-3 guards of prop247, we want to pick amongst N guards everytime we build a circuit so that we load balance our traffic amongst the guards. When considering directory guards, we want to pick randomly amongst the first 3 directory guards every time, so that if the first guard lied to us, we will also try the second one who might not lie. So maybe the simple answer here is that if prop247 is enabled (this could be a NumGuards=N argument to our algorithm), instead of always returning the first reachable guard, we instead build a list of the first N reachable guards, and randomly choose one of them. Could this work? (see #12466 for a weird behavior that will make us skip bugs if NumEntryGuards != 1) > The other alternative is to simply initialize USED_GUARDS to be > layer-2 guardlist and run the algorithm, then init USED_GUARDS to be > layer-3 guardlist and run the algorithm. Makes sense or am I missing > something? > Hmm, in this case we would just use the first reachable guard from USED_GUARDS. We would only reach other guards if the first guards are unreachable. That will not work for load balancing prop247 for example. (got to go. will be back later!)

1 0

Onion (Hidden) Service Proposal Discussion
by Tim Wilson-Brown - teor 08 Feb '16

08 Feb '16

Hi, We just had a meeting to discuss the following tor proposals[0] in the #tor-dev IRC channel[1]. Proposal 252: Single Onion Services Proposal 260: Rendezvous Single Onion Services Proposal 255: Controller features to allow for load-balancing hidden services Proposal 246: Merging Hidden Service Directories and Introduction Points A quick summary of each proposal: Some onion (hidden) service websites don't need to hide their location. They can have faster connection setup and bandwidth, and put less load on the tor network, by having 3 relays between the client and onion service. Proposal 252 has the onion service open an ORPort, and then clients extend from their third relay to the ORPort. Proposal 260 has the onion service connect directly to the introduction and rendezvous points. The other proposals improve onion service speed in different ways: Proposal 255 improves hidden or onion service load balancing by handing off the rendezvous to another tor instance. Proposal 246 improves hidden or onion service setup time by using the HSDirs as introduction points, and teaching clients to re-use the HSDir connection for the introduction. And a quick summary of our thoughts: Proposal 252 and Proposal 260 achieve similar outcomes. 260 is simpler to code, preserves NAT-punching (which some website providers need), and has already been coded[2]. It's also compatible with 255, which 252 is not. But 252 has a faster connection set-up time, because it skips the rendezvous protocol entirely. We'd like to see more research into the performance differences between 252 and 260. We want to focus on Proposal 224 (next-generation hidden services), and we were concerned that too much other work on onion service proposals would slow that down. So we'd like to finish 260 in the short term, and then reconsider 252 based on resourcing and research outcomes. We thought that 255 was a good idea, but noted that it increases connection set-up time. We noted that 246 already had concerns raised about it on the mailing list. That said, we could use 246 to improve the performance of 260. Tim [0]: https://gitweb.torproject.org/torspec.git/tree/proposals <https://gitweb.torproject.org/torspec.git/tree/proposals> [1]: http://meetbot.debian.net/tor-dev/2016/tor-dev.2016-02-08-22.00.log.html <http://meetbot.debian.net/tor-dev/2016/tor-dev.2016-02-08-22.00.log.html> [2]: https://trac.torproject.org/projects/tor/ticket/17178 <https://trac.torproject.org/projects/tor/ticket/17178> Tim Wilson-Brown (teor) teor2345 at gmail dot com PGP 968F094B teor at blah dot im OTR CAD08081 9755866D 89E2A06F E3558B7F B5A9D14F

1 0

Re: [tor-dev] Entry guards, primary guards, dir guards
by George Kadianakis 08 Feb '16

08 Feb '16

Ola Bini <obini(a)thoughtworks.com> writes: > Hey, > > Thanks- this is very helpful. > > When it comes to vanguards, I've already read through the > proposal. I'm not exactly sure I understand how much different 259 > would need to be to support the 247 needs. It seems we should be able > to just run the algorithm NUM_SECOND_GUARDS * NUM_THIRD_GUARDS times > to choose the sets of vanguards for each layer, right? > Hmm, how would that work exactly? Let's say I'm a prop247 hidden service. I just received an introduction and want to setup my rendezvous circuit. To setup my circuit, I would need to do three guard picks, one for every layer. Each layer has a different guard list. First, I use my layer-1 guardlist to pick my layer-1 guard. That's easy, I use a single guard for layer-1, so I always pick the first reachable non-bad guard from the layer-1 guardlist. Then I need to use my layer-2 guardlist to pick my layer-2 guard. Proposal 247 says that each HS has two layer-2 guards , so I would need to pick a guard out of the two top guards in my layer-2 guardlist. How does this happen exactly? A similar thing needs to happen for layer-3.

1 0

Roadmap - Hidden service next generation (prop224)
by David Goulet 08 Feb '16

08 Feb '16

Hi everyone! For a while now, we've had an idea on how to proceed for the engineering part of proposal 224 (next generation hidden service). So, we've put up this pad with *5* large milestones that will in theory happen one after the other. Step 1 is shared randomness for which the code and proposal has been finalized and under review for upstream merge. Yay progress! :) You'll notice that they are pretty huge milestone. As we go forward, we'll break them down in much more details (++tickets). You can already see step 2) has a pad for that already. Our hope is to update that pad regurlarly as implementation moves forward and update tor-dev@ as well with relevant information for the community. You to can help with this huge effort, please come find us on #tor-dev! Code review, comment on designs, testing, etc..., we'll need a lot of these. https://storm.torproject.org/shared/TlPhwNbdgPwLw5h23G3gEDtkWdqxVe9Gqn3KiqV… Don't hesitate to comment on the roadmap using this thread or you can also do it in the pad (which is a "Can Comment" link). Thanks! David

2 2

Re: [tor-dev] Entry guards, primary guards, dir guards
by George Kadianakis 08 Feb '16

08 Feb '16

Ola Bini <obini(a)thoughtworks.com> writes: > Hi again, > > Two questions - first, hopefully I simple one. What are directory > guards, exactly? > > Second, we found a weird behavior in the current code > base. Specifically, when Tor first downloads the microdescriptors, it > chooses 3 guards for this with the V2Dir flag. That's a little bit > less than 1600 nodes right now. However, these three guards gets > persistes as primary guards, which means that almost all Tor users > will have primary guards among those 1600, not among the 1900 that > have the Guard flag. Is this intented behavior? > Very relevant questions. Directory guards were intoduced by proposal 207. They are basically guard nodes for directory connections, so that Tor clients expose themselves to the minimum number of directory relays when downloading the various Tor documents (microdescriptors, etc.). Currently Tor uses the same guardlist for directory guards and entry guards. That's a feature because ideally you would have a single guard that would be both your entry guard and your directory guard. Unfortunately, that's not always possible because some entry guards are not directories (they don't have the V2Dir flag) so they can't be picked as directory guards. And that's basically the weird behavior that you noticed. Because Tor pick its directory guards first and they are picked from a slightly restricted set compared the total set of guards nodes. Eventually when #12538 gets to stable Tor, all relays will be directories, and hence all guards will be potential directory guards. --- So everything is explained now except from why Tor picks 3 directory guards and not one. That's because we have decided to keep the number of directory guards to 3, instead of using a single guard like we do for normal circuits. Otherwise, that single directory guard could launch some hard-to-detect route biasing attacks. For more details see: https://lists.torproject.org/pipermail/tor-dev/2014-May/006820.html https://lists.torproject.org/pipermail/tor-dev/2014-June/006944.html So basically when Tor picks a directory guard for a directory connection, it finds the first three reachable entry guards in its guardlist, and picks one of them. It might be smart to be able to support "multiple guards" mode in the algorithms that we are currently designing. That's because we might keep the "3 directory guards" design for some more time, but even more importantly because if we ever go with a layered guards design we will need to have multiple guards in some layers. For example, consider proposal 247 which specifies a layer guard design to defend against hidden service guard discovery attacks. The proposal basically makes hidden services pick guard nodes for their second and third hops. So instead of looking like this: -> middle_1 -> middle_A -> middle_2 -> middle_B -> middle_3 -> middle_C -> middle_4 -> middle_D HS -> guard -> middle_5 -> middle_E -> Rendezvous Point -> middle_6 -> middle_F -> middle_7 -> middle_G -> middle_8 -> middle_H -> ... -> ... -> middle_n -> middle_n it will look like this: -> guard_3A_A -> guard_2_A -> guard_3A_B -> guard_3A_C -> Rendezvous Point HS -> guard_1 -> guard_3B_D -> guard_2_B -> guard_3B_E -> guard_3B_F -> Rendezvous Point As you can see in this latter design, we pick amongst "2 guard nodes" for the second hop, and amongst "6 guard nodes" for the third hop. Feel free to read the introduction of the proposal for more info. So yeah... it would be nice to support this multiple guard node design in our algorithms as well. We discussed this a bit during our proposal meeting some weeks ago but we kind of let it drop. Do you think it's possible? :)

1 0