April 2018 - tor-dev - lists.torproject.org

Bandwidth scanner - Summer of Privacy
by juga 23 Apr '18

23 Apr '18

Hi, I will be working on the bandwidth scanner project during the Tor Summer of Privacy. This would include implementing features to the scanner code, writing tests and documentation, deployment in a test network, comparing results, modify little-tor code. The goal is to have bandwidth scanner code that is easier to maintain, understand, run and gives better measurements. There will be more details online and with reports. I've being already contributing on this project, for instance, writing a draft specification for the bandwidth measurements format [0] or reporting/fixing bugs on bwscanner code [1]. I'm MS in computer engineering, senior developer and have worked as sysadmin and researcher. Best regards, juga. GnuPG key fp: 2DA81D01455C3A0032198850F305447AF806D46B [0] https://lists.torproject.org/pipermail/tor-dev/2018-April/013071.html [1] https://github.com/TheTorProject/bwscanner

1 0

Notes from 19 April 2018 Simple Bandwidth Scanner Meeting
by Matt Traudt 19 Apr '18

19 Apr '18

Next meeting is scheduled 26 April 2018 at 2200 UTC in #tor-meeting, **BUT** we are probably going to look into finding a time that works better across three halves of the globe. Pad is here, and is copy/pasted below. https://pad.riseup.net/p/BenwGdgpz0uS Meeting IRC log http://meetbot.debian.net/tor-meeting/2018/tor-meeting.2018-04-19-21.59.log… Simple Bandwidth Scanner meeting 19 April 2018 ============================================== Previous weeks notes: https://lists.torproject.org/pipermail/tor-dev/2018-April/013059.html Updates -------------------------------------------------------- pastly: - Started running an authority in the test net. Don't think it's working yet and/or people have started trusting it - Switched to standard python logging module - Added periodic reachability tests for helper relays - (Helped) add FAQ and glosasry to docs - Learned: no part of sbws is CPU bound. Weird results were due to a helper that is not capable of much bandwidth - Time is still passing. In less than 2w the open source paperwork should be back next week: - Put docs on an onion service - Ask dirauths what they want to do about sbws servers In the coming weeks, I think I need to be transitioning away from "big" feature type stuff that isn't absolutely necessary, and instead focus my time on docs, tests, and making sbws easy to use. I think literal next steps need to include a practice deployment and transition in the testnet. It's one thing to run an sbws scanner and server. I've done that. I want to see other people try it so we can learn what isn't obvious to anyone but me. I want to exercise getting scan results to the dirauth so it can put them in the consensus. i've tried scanner and server and reported problems :) * list tasks for next months: https://pad.riseup.net/p/bfikl6PM0YV9 * plan proposed by teor with testnet deployment: * ... * it'll be very useful to have sbws doc public! juga: last week: * 1st draft spec * setup dirauth, sbws server, sbws scanner * dirauth not yet in consensus * start doc about what we have deployed so far (started to feel a bit lost about what we have/need) * added draft faq and glossary to doc next week: * finish spec * check the status of config files vs cli args, with debug (sorry pastly, didn't follow close the changes) and see whether #91 is really needed * check the status of tests/chutney * implement actually some unit test? * think how to make easier deployment at some point and add something on it? (cron/Makefile/system services?) questions: * pastly: once sbws repo is public, can we work on new releases publicly? [pastly: says yes] teor: last week: * occasional suggestions this week: * make my test dirauth a trust pastly's and juga's dirauths * eventually try to set up sbws on the testnet

1 0

Proposal: The move to two guard nodes
by Mike Perry 18 Apr '18

18 Apr '18

In-line below for ease of comment. Also available at: https://gitweb.torproject.org/user/mikeperry/torspec.git/tree/proposals/xxx… =========================== Filename: xxx-two-guard-nodes.txt Title: The move to two guard nodes Author: Mike Perry Created: 2018-03-22 Supersedes: Proposal 236 0. Background Back in 2014, Tor moved from three guard nodes to one guard node[1,2,3]. We made this change primarily to limit points of observability of entry into the Tor network for clients and onion services, as well as to reduce the ability of an adversary to track clients as they move from one internet connection to another by their choice of guards. 1. Proposed changes 1.1. Switch to two guards per client When this proposal becomes effective, clients will switch to using two guard nodes. The guard node selection algorithms of Proposal 271 will remain unchanged. Instead of having one primary guard "in use", Tor clients will always use two. This will be accomplished by setting the guard-n-primary-guards-to-use consensus parameter to 2, as well as guard-n-primary-guards to 2. (Section 3.1 covers the reason for both parameters). This is equivalent to using the torrc option NumEntryGuards=2, which can be used for testing behavior prior to the consensus update. 1.2. Enforce Tor's path restrictions across this guard layer In order to ensure that Tor can always build circuits using two guards without resorting to a third, they must be chosen such that Tor's path restrictions could still build a path with at least one of them, regardless of the other nodes in the path. In other words, we must ensure that both guards are not chosen from the same /16 or the same node family. In this way, Tor will always be able to build a path using these guards, preventing the use of a third guard. 2. Discussion 2.1. Why two guards? The main argument for switching to two guards is that because of Tor's path restrictions, we're already using two guards, but we're using them in a suboptimal and potentially dangerous way. Tor's path restrictions enforce the condition that the same node cannot appear twice in the same circuit, nor can nodes from the same /16 subnet or node family be used in the same circuit. Tor's paths are also built such that the exit node is chosen first and held fixed during guard node choice, as are the IP, HSDIR, and RPs for onion services. This means that whenever one of these nodes happens to be the guard[4], or be in the same /16 or node family as the guard, Tor will build that circuit using a second "primary" guard, as per proposal 271[7]. Worse still, the choice of RP, IP, and exit can all be controlled by an adversary (to varying degrees), enabling them to force the use of a second guard at will. Because this happens somewhat infrequently in normal operation, a fresh TLS connection will typically be created to the second "primary" guard, and that TLS connection will be used only for the circuit for that particular request. This property makes all sorts of traffic analysis attacks easier, because this TLS connection will not benefit from any multiplexing. This is more serious than traffic injection via an already in-use guard because the lack of multiplexing means that the data retention level required to gain information from this activity is very low, and may exist for other reasons. To gain information from this behavior, an adversary needs only connection 5-tuples + timestamps, as opposed to detailed timeseries data that is polluted by other concurrent activity and padding. In the most severe form of this attack, the adversary can take a suspect list of Tor client IP addresses (or the list of all Guard node IP addresses) and observe when secondary Tor connections are made to them at the time when they cycle through all guards as RPs for connections to an onion service. This adversary does not require collusion on the part of observers beyond the ability to provide 5-tuple connection logs (which ISPs may retain for reasons such as netflow accounting, IDS, or DoS protection systems). A fully passive adversary can also make use of this behavior. Clients unlucky enough to pick guard nodes in heavily used /16s or in large node families will tend to make use of a second guard more frequently even without effort from the adversary. In these cases, the lack of multiplexing also means that observers along the path to this secondary guard gain more information per observation. 2.2. Why not MOAR guards? We do not want to increase the number of observation points for client activity into the Tor network[1]. We merely want better multiplexing for the cases where this already happens. 2.3. Can you put some numbers on that? The Changing of the Guards[13] paper studies this from a few different angles, but one of the crucially missing graphs is how long a client can expect to run with N guards before it chooses a malicious guard. However, we do have tables in section 3.2.1 of proposal 247 that cover this[14]. There are three tables there: one for a 1% adversary, one for a 5% adversary, and one for a 10% adversary. You can see the probability of adversary success for one and two guards in terms of the number of rotations needed before the adversary's node is chosen. Not surprisingly, the two guard adversary gets to compromise clients roughly twice as quickly, but the timescales are still rather large even for the 10% adversary: they only have 50% chance of success after 4 rotations, which will take about 14 months with Tor's 3.5 month guard rotation. 2.4. What about guard fingerprinting? More guards also means more fingerprinting[8]. However, even one guard may be enough to fingerprint a user who moves around in the same area, if that guard is low bandwidth or there are not many Tor users in that area. Furthermore, our use of separate directory guards (and three of them) means that we're not really changing the situation much with the addition of another regular guard. Right now, directory guard use alone is enough to track all Tor users across the entire world. While the directory guard problem could be fixed[12] (and should be fixed), it is still the case that another mechanism should be used for the general problem of guard-vs-location management[9]. 3. Alternatives There are two other solutions that also avoid the use of secondary guard in the path restriction case. 3.1. Eliminate path restrictions entirely If Tor decided to stop enforcing /16, node family, and also allowed the guard node to be chosen twice in the path, then under normal conditions, it should retain the use of its primary guard. This approach is not as extreme as it seems on face. In fact, it is hard to come up with arguments against removing these restrictions. Tor's /16 restriction is of questionable utility against monitoring, and it can be argued that since only good actors use node family, it gives influence over path selection to bad actors in ways that are worse than the benefit it provides to paths through good actors[10,11]. However, while removing path restrictions will solve the immediate problem, it will not address other instances where Tor temporarily opts use a second guard due to congestion, OOM, or failure of its primary guard, and we're still running into bugs where this can be adversarially controlled or just happen randomly[5]. While using two guards means twice the surface area for these types of bugs, it also means that instances where they happen simultaneously on both guards (thus forcing a third guard) are much less likely than with just one guard. (In the passive adversary model, consider that one guard fails at any point with probability P1. If we assume that such passive failures are independent events, both guards would fail concurrently with probability P1*P2. Even if the events are correlated, the maximum chance of concurrent failure is still MIN(P1,P2)). Note that for this analysis to hold, we have to ensure that nodes that are at RESOURCELIMIT or otherwise temporarily unresponsive do not cause us to consider other primary guards beyond than the two we have chosen. This is accomplished by setting guard-n-primary-guards to 2 (in addition to setting guard-n-primary-guards-to-use to 2). With this parameter set, the proposal 271 algorithm will avoid considering more than our two guards, unless *both* are down at once. 3.2. No Guard-flagged nodes as exit, RP, IP, or HSDIRs Similar to 3.1, we could instead forbid the use of Guard-flagged nodes for the exit, IP, RP, and HSDIR positions. This solution has two problems: First, like 3.1, it also does not handle the case where resource exhaustion could force the use of a second guard. Second, it requires clients to upgrade to the new behavior and stop using Guard flagged nodes before it can be deployed. 4. The future is confluxed An additional benefit of using a second guard is that it enables us to eventually use conflux[6]. Conflux works by giving circuits a 256bit cookie that is sent to the exit/RP, and circuits that are then built to the same exit/RP with the same cookie can then be fused together. Throughput estimates are used to balance traffic between these circuits, depending on their performance. We have unfortunately signaled to the research community that conflux is not worth pursuing, because of our insistence on a single guard. While not relevant to this proposal (indeed, conflux requires its own proposal and also concurrent research), it is worth noting that whichever way we go here, the door remains open to conflux because of its utility against similar issues. If our conflux implementation includes packet acking, then circuits can still survive the loss of one guard node due to DoS, OOM, or other failures because the second half of the path will remain open and usable (see the probability of concurrent failure arguments in Section 3.1). If exits remember this cookie for a short period of time after the last circuit is closed, the technique can be used to protect against DoS/OOM/guard downtime conditions that take down both guard nodes or destroy many circuits to confirm both guard node choices. In these cases, circuits could be rebuilt along an alternate path and resumed without end-to-end circuit connectivity loss. This same technique will also make things like ephemeral bridges (ie Snowflake/Flashproxy) more usable, because bridge uptime will no longer be so crucial to usability. It will also improve mobile usability by allowing us to resume connections after mobile Tor apps are briefly suspended, or if the user switches between cell and wifi networks. Furthermore, it is likely that conflux will also be useful against traffic analysis and congestion attacks. Since the load balancing is dynamic and hard to predict by an external observer and also increases overall traffic multiplexing, traffic correlation and website traffic fingerprinting attacks will become harder, because the adversary can no longer be sure what percentage of the traffic they have seen (depending on their position and other potential concurrent activity). Similarly, it should also help dampen congestion attacks, since traffic will automatically shift away from a congested guard. References: 1. https://blog.torproject.org/improving-tors-anonymity-changing-guard-paramet… 2. https://trac.torproject.org/projects/tor/ticket/12206 3. https://gitweb.torproject.org/torspec.git/tree/proposals/236-single-guard-n… 4. https://trac.torproject.org/projects/tor/ticket/14917 5. https://trac.torproject.org/projects/tor/ticket/25347#comment:14 6. https://www.cypherpunks.ca/~iang/pubs/conflux-pets.pdf 7. https://gitweb.torproject.org/torspec.git/tree/proposals/271-another-guard-… 8. https://trac.torproject.org/projects/tor/ticket/9273#comment:3 9. https://tails.boum.org/blueprint/persistent_Tor_state/ 10. https://trac.torproject.org/projects/tor/ticket/6676#comment:3 11. https://bugs.torproject.org/15060 12. https://trac.torproject.org/projects/tor/ticket/10969 13. https://www.freehaven.net/anonbib/cache/wpes12-cogs.pdf 14. https://gitweb.torproject.org/torspec.git/tree/proposals/247-hs-guard-disco… -- Mike Perry

4 7

[release] [protocol] Onionoo 6.0-1.13.0 and metrics-lib 2.3.0
by iwakeh 18 Apr '18

18 Apr '18

Hi there! Two new releases are available at: https://dist.torproject.org/onionoo/6.0-1.13.0/ https://dist.torproject.org/metrics-lib/2.3.0/ This Onionoo release provides the announced major version change to 6.0 Protocol change (also summarized in [0], [1]): Change the "exit_addresses" field to not exclude current OR addresses anymore. This release also includes some bug fixes, please take a look at the change log [2]. The changes are already deployed on all onionoo.torproject.org instances. The new metrics-lib release adds support for reading web server logs contained in tar-archives and a backwards compatible API change detailed in the changelog [3] and described in the javadoc documentation [4]. Please direct comments and questions to the metrics-team mailing list [5]. Cheers, iwakeh [0] https://metrics.torproject.org/onionoo.html#versions_6_0 [1] https://metrics.torproject.org/onionoo.html#details_relay_exit_addresses [2] https://gitweb.torproject.org/onionoo.git/plain/CHANGELOG.md [3] https://gitweb.torproject.org/metrics-lib.git/plain/CHANGELOG.md [4] https://metrics.torproject.org/metrics-lib/org/torproject/descriptor/Server… [5] https://lists.torproject.org/cgi-bin/mailman/listinfo/metrics-team

1 0

Tor Bandwidth Measurements Document Format
by juga 17 Apr '18

17 Apr '18

Hi, as commented with teor and pastly, i send in-line a draft specification for the document format that the bandwidth scanner implementations should produce. I've left my own questions/notes in square brackets. Thanks, juga. ======================================= Tor Bandwidth Measurements Document Format [juga: which name should we give to this document?] 1. Scope and preliminaries This document describes the format of Tor's bandwidth measurements document, version X.X.X [juga: which version should be this?] and later. Since Tor version X.X.X [juga: which tor version?] the directory authorities use the bandwidth measurements document called "V3BandwidthsFile" and produced by Torflow [1] (format described in README.spec.txt [2]). The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in RFC 2119. 1.2. Acknowledgements The original bandwidth measurement scanner (Torflow) and format was created by mike. Teor suggested to write this specification while contributing on pastly's new bandwidth scanner implementation. This specification was revised after feedback from: XXX 1.3 Outline The bandwidth measurements mentioned in sections 3.4.1 and 3.4.2 of dir-spec.txt [3] are obtained by bandwidth authorities, which are either directory authorities or other servers running bandwidth measurement scanners and sending the results to the former. [juga: it seems that bandwidth authorities have not been formally before] 2. Format details Bandwidth measurements MUST contain the following sections: - Header (exactly once) - Relays measurements (zero or more times) Each section (or entry) ends with a separator. 2.1. Nonterminals The following nonterminals are defined in the Onionoo details document specification [4]: fingerprint nickname In the bandwidth measurement documents nickname is optional. The following nonterminals are defined in the in dir-spec.txt: NL (newline) SP (space) "bw" = INT, the aggregated measured bandwidth of this relay, in kilobytes per second. We introduce the following nonterminals: [juga: this should probably be defined more formally and should probably link to other documents, which ones?] "version" = The name and the version of the bandwidth scannner software, such as "sbws 0.1.0". The name of the software, if absent, is assumed to be "torflow". [juga: which should be the version if absent?] "timestamp" = INT, the Unix Epoch time when the file was created. 2.2. Header format It MUST consists of: "timestamp" timestamp NL "version" version NL 2.3. Relay measurements format Relays measurements MUST consist of the following items. "node_id" fingerprint SP "bw" bandwidth SP When there are no more items, the "bw" item ends with NL instead of SP. 2.4. Optional extra items Different implementations of the bandwidth measurements scanners MAY include other items per relay. For instance, sbws includes: "rtt" = INT, Round Trip Time (to obtain 1B) Every relay measurement in sbws consists of: "node_id" fingerprint SP "bw" bandwidth SP "nick=" nickname SP "rtt=" rtt SP "time=" timestamp NL Every relay measurement in Torflow consists of: "node_id" fingerprint SP "bw" bandwidth SP "nick=" nickname SP "measured_at=" slice timestamp NL The "measured_at" does not correspond to the "time" in sbws. [juga: actually, if bwauths use "measured_at", then the code on them or sbws should be changed]. Torflow includes other items that are out of the scope of this document. References: 1. https://gitweb.torproject.org/torflow.git 2. https://gitweb.torproject.org/torflow.git/tree/NetworkScanners/BwAuthority/… 3. https://gitweb.torproject.org/torspec.git/tree/dir-spec.txt 4. https://metrics.torproject.org/onionoo.html#details A. Sample data A.1. Torflow 1523911758 node_id=$68A483E05A2ABDCA6DA5A3EF8DB5177638A27F80 bw=392760 nick=Test measured_at=1523911725 updated_at=1523911725 pid_error=4.11374090719 pid_error_sum=4.11374090719 pid_bw=57136645 pid_delta=2.12168374577 circ_fail=0.2 scanner=/filepath A.2. sbws 1523911758 version=0.1.0 node_id=$68A483E05A2ABDCA6DA5A3EF8DB5177638A27F80 bw=392760 nick=Test rtt=380 time=1523911725

2 1

Hogging of second_elapsed_callback()
by Linus Nordberg 15 Apr '18

15 Apr '18

Hi, How long time can we spend signing status documents before tor gets sad? I ask because I'm planning on putting dirauth signing keys on a sloooooow HSM and would like to understand if I'd have to make format_networkstatus_vote() and networkstatus_compute_consensus() (and their callers) be "restartable", much like dirvote_act() is "restartable". Current measurements indicate a worst case figure of 6.5 seconds for a hashing and signing operation. This may even increase some over time. (It may also shrink considerably with some optimisations but this is where we are today.)

2 2

Notes from 12 April 2018 Simple Bandwidth Scanner Meeting
by Matt Traudt 13 Apr '18

13 Apr '18

See below for the pad notes. Next meeting is scheduled 19 April 2018 at 2200 UTC in #tor-meeting. (This one was held in #tor-dev, but we should use meetbot). ------------- Simple Bandwidth Scanner meeting 12 April 2018 #### Updates/Status messages #### pastly: What's on my plate? <- doesn't have to be all in your plate :P - Test coverage getting closer to 100% - Immediate future: switch to standard python logging module, which is quite good - Improving documentation - Checking results against torflow - Monitor CPU of sbws client/server - +1 on considering asyncio - See how chutney generates random strings - Run testnet authority - Reach out to current auths about running sbws/torflow and adding me as an auth juga: - open/close PRs/issues about things to improve in doc, refactor code, etc..., but not changing functionality - re. doc: - thought to update sbws spec (or create other) to doc differences with Torflow, not sure it's useful - i'd document further some of the classes/functions (as measure_relay) - code doc vs spec (see below) - find box to run other sbws, bwauth also in testnet? ## Topic: what is still missing for milestone 1? (aka 1st release, v1.0.0) - could we create all tickets needed to achive it? - maybe previous list is enough? Missing: - A consensus parameter stating the scaling factor - sbws config option to set fallback if no consensus param - `sbws generate` code to use the consensus param - https://stem.torproject.org/api/descriptor/networkstatus.html#stem.descript… - Correlation coefficient on comparision graphs ## Topic: comparing to torflow tah - Can we make the test sbws deployment a little bigger? - What else needs to be compared? teor: actually running it in a voting network, to check the feedback loop (if any) the scaling - Conclusions after comparing? - what we could think to change/improve after comparing? Graphs pastly can explain: - sbws vs moria, sorted by sbws: https://share.riseup.net/#-W_zqcv-08AX4SnOgTatUw - sorted by moria: https://share.riseup.net/#URXp6NccZHEhOPFJQcfO4w teor: the correlation seems good here If we're going to use these charts to compare, please compare two existing bwauths See: https://share.riseup.net/#lPGcIrgHp3ftnvTHUKqOKg (but ignore the sbws-scaled line, it's wrong wrong wrong) ## Topic: convincing people to run sbws juga: maybe something to do when 1st sbws release? pastly: yes, probalby. unless we need to convince testnet people <- ah, right i was thinking on the Tor net ## Topic: status of open sourcing sbws - No real update. Time is still passing. ## Topic: specifications torflow/BwAuthority: https://gitweb.torproject.org/torflow.git/tree/NetworkScanners/BwAuthority/…, https://ohmygodel.com/publications/peerflow-popets2017.pdf has a section that also makes a nice summary sbws: https://github.com/pastly/simple-bw-scanner/blob/master/docs/source/specifi… (ask Pastly for access) bwscanner: no spec, but reading https://github.com/TheTorProject/bwscanner/blob/develop/bwscanner/circuit.p… it looks like a Torflow clone <- almost :) We need a spec for the v3bw file that tor reads (in torspec/dir-spec.txt) We need a spec for bwauth migration, including acceptance criteria for new bwauth implementations Scanners should have their own detailed design documents

3 5

Consensus-health single-relay data
by Tom Ritter 07 Apr '18

07 Apr '18

teor suggested the other day that it'd be really useful to be able to see the vote data for a single relay; since the _entire_ detailed page is huge and unwieldy. I've been pondering how I could support this without complicating the server, which results in a few constraints: a) I really don't want to create one file per-relay b) I really don't want to make any server-side logic, I'd like to keep it as static HTML and client-side javascript. (With the client-side javascript optional for most of the information.) I think I can make something work using AJAX and Range requests though. Before I fly forward with that though, I wanted to understand a bit more about the use cases where one wants to see per-relay vote data. The biggest question I have for you teor (and anyone else who wants to chime in) is how do you know what relay you want the information for? Do you know the fingerprint? If so, I could try adding a text box at the bottom of the index page saying "Supply a fingerprint, and I will populate vote data for this relay inline". -tom

3 5

[release] [protocol] Onionoo 5.2-1.12.0
by iwakeh 06 Apr '18

06 Apr '18

Hi there! Onionoo's protocol was extended and has a minor version jump to 5.2. Reminder:there will be a major *upcoming version* jump 6.0 after April 14, 2018. Take note of the field "next_major_version_scheduled" in documents. Download available at: https://dist.torproject.org/onionoo/5.2-1.12.0/ Protocol changes (also summarized in [0]): There is now a 'version_status' field, which is the status of the Tor software version of this relay based on the versions recommended by the directory authorities [1]. Operational changes: In order to always provide 'fresh' data Onionoo now fetches descriptors from both CollecTor instances and won't be affected as much anymore when one CollecTor instance is not reachable. For more please take a look at the change log [3]. The changes are already deployed on all onionoo.torproject.org instances. Please direct comments and questions to the metrics-team mailing list [3]. Cheers, iwakeh [0] https://metrics.torproject.org/onionoo.html#versions_5_2 [1] https://metrics.torproject.org/onionoo.html#details_relay_version_status https://metrics.torproject.org/onionoo.html#details_bridge_version_status [2] https://gitweb.torproject.org/onionoo.git/plain/CHANGELOG.md [3] https://lists.torproject.org/cgi-bin/mailman/listinfo/metrics-team

1 0

The case for Tor-over-QUIC
by Mike Perry 02 Apr '18

02 Apr '18

In Rome, I held a session about network protocol upgrades. My intent was to cover the switch to two guards, conflux, datagram transports, and QUIC. We ended up touching only briefly on everything but QUIC, but we went into enough depth on QUIC itself that it was a worthwhile and very productive session. Our notes are here: https://trac.torproject.org/projects/tor/wiki/org/meetings/2018Rome/Notes/F… I wanted to give a bit of background and some historical perspective about datagram transports for Tor, as well as explain those notes in long form, to get everybody on the same page about this idea. With the forthcoming IETF standard ratification of QUIC along with several solid reference implementations (canonical list: https://github.com/quicwg/base-drafts/wiki/Implementations) I believe we are close to the point where we can finally put together a plan (and a funding proposal) to undertake this work. Consider this mail a pre-proposal to temperature check and solicit early feedback about a Tor-over-QUIC deployment, before we invest the effort to deep dive into the framing, protocol, and implementation details that will be necessary for a full proposal. # Historical Background So, first: Tor has been considering UDP transports for a long time. There have been a series of research papers on the topic, dating back to Proposal 100 -- the very first Tor proposal: https://gitweb.torproject.org/torspec.git/tree/proposals/100-tor-spec-udp.t… That proposal only examines adding an unreliable DTLS link between relays, to enable transmission of unreliable (UDP) traffic end-to-end. Despite being marked as Dead, it is a good conceptual starting point, because it examines the relay-to-relay link crypto properties we need from a datagram layer and has some good comments at the bottom. However, adding the ability to simply carry UDP is not sufficient. What we have always really wanted is to transition the network to some form of end-to-end drop-based reliable congestion and flow control. There are numerous benefits from such a transition. Drop-signaled end-to-end congestion control means that instead of having to maintain arbitrary queues at relays (to make room for Tor's fixed SENDME window sizes for every single Tor circuit -- which as we have seen does not scale and is vulnerabe to DoS and OOM attacks), we can instead use a global fixed-length queue at each relay, and have each relay simply drop packets when this queue is full. This behavior mirrors how the internet works, and will not only put an upper bound on the end-to-end queuing latency of the Tor network, but it will also allow us to better ensure fairness, eliminate OOM conditions, and mitigate DoS and congestion attacks. QUIC's stream management and byte-acked teardown mechanisms will also eliminate a class of sidechannel issues in Tor where an adversary gets to send ignored data on invalid and/or recently closed streams: https://trac.torproject.org/projects/tor/ticket/25573 Furthermore, the switch to a secure datagram link protocol will eliminate RST spamming attacks, where an adversary spoofs RSTs to/from relay IP addresses to tear down our TLS links to gain information from the resulting circuit failures and retries across the network. Prior to the advent of QUIC, we studied many options for achieving end-to-end congestion control, but have found all of them to be lacking in one way or another. In 2011, Steven Murdoch did a literature review of these approaches: https://research.torproject.org/techreports/datagram-comparison-2011-11-07.… Section 4 of that review is the most useful thing to skim -- it explains why we found each of the end-to-end transport modes available at the time to be either insufficient for Tor's use, or encumbered by licensing or engineering incompatibilities. # QUIC Overview QUIC, however, is different. It basically serves as a complete drop-in replacement for Tor's n:1 streams-on-circuit connection model. QUIC has a connection layer, which corresponds to Tor's circuit concept, and a framing layer, which supports multiplexing for multiple reliable streams inside a single connection (corresponding to Tor's streams). Each QUIC connection provides an outer layer of drop-based congestion control. Unlike latency-based congestion control like bittorrent's utp, which was designed to compensate for poor queuing management at edge routers (as well as yield to TCP flows), QUIC's drop-based congestion control is designed to find the common bottleneck anywhere on the connection's path, and use packet drops from that queue to signal congestion and ensure fairness among connections at that bottleneck point. QUIC also provides optimizations to handle higher latency and drop-heavy links (SACK, SACK ranges, early retransmit, reordering tolerance, RTT estimates): https://quicwg.github.io/base-drafts/draft-ietf-quic-recovery.html Additionally, QUIC provides flow control at the connection level, as well as at the individual stream level. This is analogous to Tor's flow control model, but unlike Tor, QUIC implementations can adjust these window sizes automatically based on RTT measurements and congestion feedback: https://datatracker.ietf.org/doc/html/draft-ietf-quic-transport#section-3.4 https://datatracker.ietf.org/doc/html/draft-ietf-quic-transport#section-11 https://docs.google.com/document/d/1F2YfdDXKpy20WVKJueEf4abn_LVZHhMUMS5gX6P… It is perhaps no small coincidence that QUIC's architecture fits so perfectly into Tor's circuit model. While at an IETF meeting years ago, an attendee quietly whispered in my ear, "You know, we're designing IETF QUIC with Tor-like networks in mind." I always love me a good prospiracy (https://www.urbandictionary.com/define.php?term=prospiracy ;). # Using QUIC in Tor While QUIC does indeed solve a great many problems for Tor out of the box, the path to Tor-over-QUIC requires that we make a few design and engineering decisions specific to Tor. There are two research implementations that study using Tor-over-QUIC: Quux and QuTor. Quux was implemented by wrapping Chromium's C++ QUIC implementation in C bindings: https://www.benthamsgaze.org/2016/09/29/quux-a-quic-un-multiplexing-of-the-… https://www.benthamsgaze.org/wp-content/uploads/2016/09/393617_Alastair_Cla… QuTor is unfortunately paywalled at researchgate, so it is hard for me to tell what they did: https://www.researchgate.net/profile/Raik_Aissaoui/publication/292392094_QU… Unfortunately, the Quux implementation appears to use QUIC at a suboptimal position -- they replace Tor's TLS connections with QUIC, and use QUIC streams in place of Tor's circuit ID -- but only between relays. This means that it does not benefit from QUIC's end-to-end congestion control for the entire path of the circuit. Such a design will not solve the queuing and associated OOM problems at Tor relays, since relays would be unable to drop cells to signal backpressure to endpoints. Drops will instead block every circuit on a connection between relays, and even then, due to end-to-end reliability, relays will still need to queue without bound, subject to Tor's current (and inadequate) flow control. The optimal design is to use QUIC end-to-end. (Based on the researchgate abstract, it looks like this might be what QuTor did, but it is hard to tell.) Instead of replacing TLS with QUIC, we should use an unreliable link encryption protocol between relays (such as DTLS), and replace Tor's circuits with QUIC connections. In this design, Tor applications connect to the local SOCKS port using TCP, and the tor client converts it to a QUIC stream on a QUIC connection-circuit that runs all the way to the exit or onion service, which then will convert the stream back to TCP. The exit/service keeps buffering to a minimum by using QUIC pushback on the Tor side, and TCP pushback on the server side. While IETF QUIC provides its own crypto layer based on TLS 1.3, we won't really need this encryption. Instead, we will layer QUIC inside of Tor's layered onion encryption, with an ntor handshake to each hop in the path. # QUIC Deployment Considerations Past the above high-level decision of how to use QUIC, a handful of deployment considerations remain. ## Implementation As mentioned previously, there are a growing number of QUIC implementations out there, in several languages, with varying degrees of IETF compatibility: https://github.com/quicwg/base-drafts/wiki/Implementations For our purposes, the most interesting ones are ngtcp2 (native C implementation, BoringSSL, MIT licensed), picoquic (C, MIT license), and mozquic (C++ with C bindings, NSS, MPL licensed). The QUIC in Chromium is not IETF compliant, and is C++ with no C bindings. Sadly Rust implementations of QUIC are not complete. Google does also maintain a Go QUIC, which is complete. This suggests that experimental designs using one of the golang Tor implementations might be worthwhile. I'm told that the IETF hopes to finalize the QUIC standard within the year. ## Framing QUIC actually differentiates between connection-level packet sizes and frame sizes. If we want to preserve Tor's wire behavior, we will want to enforce a one-frame-per-packet policy from our QUIC library, and use QUIC's padding frames to make packet sizes uniform. ## Upgrade Path The upgrade path to QUIC will require some thorough thinking -- if all relays on a path don't yet support datagram links, should we embed QUIC inside an opaque RELAY cell type, or should wait for a flag day to switch over once sufficient relays have upgraded? Embedding QUIC in opaque RELAY cells seems like the natural choice, but will require additional engineering considerations. Additionally, a small and declining percentage of networks currently block all UDP other than DNS. At last measurement, the rate of QUIC failure was below 5%. However, like Google, we will likely want to retain the ability to fall back to TLS over TCP for these situations. ## Signaling Because we need to transition to unreliable relay links to get the most out of QUIC, we will need to figure out how to handle circuit setup -- there is currently no mechanism to ensure reliability for onionskins in Tor, since this is provided by TLS today. The simplest solution is to retain TLS between relays to transmit circuit extend cells out-of-band. Padding could be used to obscure timing information of this signaling. The alternative would be to implement some onionskin-level acking between relays, so that the datagram transport could be used for all traffic. However, since our deployment path will likely involve migration from TCP anyway, and since a small (and shrinking) percentage of networks simply block all non-DNS traffic, we will likely want to retain TCP connections for some time, making out-of-band signaling a natural first choice. ## Native UDP Support Unfortunately, during the standardization process, the IETF QUIC decided to drop support for unreliable datagram streams inside the QUIC connection, to simplify the layering of their implementation. However, because we will still need to layer QUIC inside of an onion layer, we can use that layer to carry either QUIC or UDP (at the expense of an additional field to differentiate this). # QUIC Research Questions A couple areas of open research into how QUIC will behave in a Tor deployment remain. It is my belief that while these areas are worth further study, they do not represent blockers to QUIC, as a naive deployment of QUIC will substantially outperform our current (lack of) congestion and poor flow control. ## End-to-end Congestion Control Study Unfortunately, it appears that the Quux QUIC paper studied QUIC at the wrong position - between relays, and the QuTor implementation is unclear. This means that it may still be an open question as to if QUIC's congestion control algorithms will behave optimally in a Tor-like network under heavy load. However, Google has conducted extensive internet-scale research into the congestion control algorithm, which very likely covers enough networks with Tor-like latency and drop characteristics. Because of this, I do not expect us to have to do a lot of tuning of their congestion control here, but it is worth investigating. ## Queue Management Fairness among flows in drop-based congestion control is heavily influenced by how the queue is managed, and particularly by the decision of which packet to drop when the queue is full. Modern favored algorithms involve probabilistic early dropping from the middle of the queue. The most popular approaches are the adaptive variants of RED and Blue: https://en.wikipedia.org/wiki/Random_early_detection https://en.wikipedia.org/wiki/Blue_(queue_management_algorithm) The simplest possible queue management design would be to set Tor's queues to 0, and allow the OS kernel or upstream router to apply its drop strategy. While this will be sufficient to ensure backpressure under congestion, it is not optimal for fairness. To enforce fairness, Tor would want to take information about QUIC connection IDs into account when dropping packets, similar to what is done with circuit ids by EWMA currently. This information will be unavailable to the kernel and upstream routers, due to the DTLS link encryption that will conceal the multiplexing of QUIC connections on connections to relays. Moreover, ensuring that sufficient queuing happens inside the Tor daemon (as opposed to the kernel) will require an aggressive socket-reading strategy. Tor's single-threaded nature will make this difficult -- to ensure that queuing happens inside the Tor daemon, packets must be continuously read from the kernel with minimal processing delay or delay from other events. Even in this case, optimality will still depend upon the bottleneck being Tor, and not in the kernel or in an upstream internet router. Thus, optimal queue management in Tor for QUIC remains an open research and engineering problem. (However, this is not really a deployment blocker. The deployment of QUIC on the internet has similar shortcomings -- the vast majority of internet routers also do not take QUIC connection ID into account when making UDP drop decisions, and yet the protocol still outperforms TCP in link utilization and drop recovery). ## Anonymity Since a Tor with bounded-length queues will have more predictable latency characteristics, a faster Tor based on QUIC may be more vulnerable to inferences about location that can be made by connection latency. Particularly, middle nodes may be able to tell how far the client is from the guard by measuring RTT. One solution to this may be adding delay in cases where RTT could be inferred. Another solution is to have two middle nodes for four hop paths, so that it is no longer clear which middle node you are in a QUIC circuit. There may be other attacks that come out of using end-to-end congestion control, but I am personally having a hard time coming up with cases where such side channels and resource starvation conditions could be any worse than those that already exist, or are unique to the OOM and DoS conditions currently possible due to Tor's lack of congestion control. -- Mike Perry

3 4