New Proposal 306: A Tor Implementation of IPv6 Happy Eyeballs
Hi tor-dev@ mailing list, I have a new proposal: A Tor Implementation of IPv6 Happy Eyeballs This is to implement Tor IPv6 Happy Eyeballs and acts as an alternative to Prop299 as requested here: https://trac.torproject.org/projects/tor/ticket/29801 The GitHub pull request is here: https://github.com/torproject/torspec/pull/87 Thank You, Neel Chauhan === https://www.neelc.org/
Hi Neel, Thanks for this proposal.
Here's the proposal content, with my comments:
Supercedes: 299
When we implement this feature in tor, it would be a good idea to call the two addresses "preferred" and "alternate" address. With this design, the low-level connection code doesn't have to know about reachable addresses, or IPv4/IPv6 preferences. It just has to try them in order.
Tor's code already does this check: we won't need to change it.
That's not quite true: most clients use IPv4 by default, but they can be configured to prefer IPv6, or only allow certain addresses. And bridge clients automatically use IPv6 if they are configured with an IPv6 bridge.
In the worst case scenario, users see Tor Browser hang for 15 seconds before it makes a successful connection. That's not acceptable. Depending on their location, most tor clients authenticate to the first hop within 0.5-1.5 seconds. So I suggest we use a 1.5 second delay: https://metrics.torproject.org/onionperf-buildtimes.html In RFC 8305, the default delay is 250 milliseconds, and the maximum delay is 2 seconds. So 1.5 seconds is reasonable for TLS and tor link authentication. https://tools.ietf.org/html/rfc8305#section-8 (This delay will mainly affect initial bootstrap, because all of Tor's other connections are pre-emptive, or re-used.) A small number of clients may do wasted authentication. That's ok. Tor already does multiple bootstrap and guard connections. We have talked about this design in the team over the last few months. Our key insights are that: * TCP connections are cheap, but TLS is expensive * most failed TCP connections fail immediately in the kernel, some fail quickly with a response from the router, and others are blackholed and time out * it's unlikely that a client will fail to authenticate to a relay over one IP version, but succeed over the other IP version, because the directory authorities authenticate to each relay when they check reachability * some censorship systems only break authentication over IPv4, but they are rare So here are some alternative designs: 1. Tor connects to the preferred address and tries to authenticate. On failure, or after a 1.5 second delay, it connects to the alternate address and tries to authenticate. On the first successful authentication, it closes the other connection. This design places the least connection load on the network, but might add a bit of extra TLS load. 2. Tor connects via TCP to the preferred address. On failure, or after a 250 ms delay, it connects via TCP to the alternate address. On the first TCP success, tor attempts to authenticate immediately. On authentication failure, or after a 1.5 s delay, tor attempts to authenticate over the second TCP connection. On the first successful authentication, it closes the other connection. This design is the most reliable for clients, but it also puts a bit more connection load on dual-stack guards and authorities. 3. Tor connects via TCP to the preferred address. On failure, or after a 250ms delay, it connects via TCP to the alternate address. On the first TCP success, tor attempts to authenticate, and closes the other connection. This design looks similar to a web browser's implementation of Happy Eyeballs, because it closely follows the RFC. That might help hide tor from censors. It adds some extra connection load, but no extra TLS load. I suggest that we put all 3 alternative designs in the proposal, but start by implementing and testing alternative 1. When we implement this code, let's put the happy eyeballs part in a separate module, as much as possible. That helps us review the code, and make sure it has good test coverage. It also stops existing files and functions getting too big.
T
Hi Neel, > On 27 Jun 2019, at 11:10, neel@neelc.org wrote: > > Thank you so much for your feedback. I have pushed your changes (and revert my changes to 000-index.txt) as fixup commits to the GitHub PR. > > Could you please review the new changes? I'm going to wait a week or two for other people to provide feedback on this thread. T >> On 2019-06-25 23:33, teor wrote: >> Hi Neel, >> Thanks for this proposal. >>> On 26 Jun 2019, at 11:15, neel@neelc.org wrote: >>> I have a new proposal: A Tor Implementation of IPv6 Happy Eyeballs >>> This is to implement Tor IPv6 Happy Eyeballs and acts as an >>> alternative to Prop299 as requested here: >>> https://trac.torproject.org/projects/tor/ticket/29801 >>> The GitHub pull request is here: >>> https://github.com/torproject/torspec/pull/87 >> Here's the proposal content, with my comments: >>> Filename: 306-ipv6-happy-eyeballs.txt >>> Title: A Tor Implementation of IPv6 Happy Eyeballs >>> Author: Neel Chauhan >>> Created: 25-Jun-2019 >> Supercedes: 299 >>> Status: Open >>> Ticket: https://trac.torproject.org/projects/tor/ticket/29801 >>> 1. Introduction >>> As IPv4 address space becomes scarce, ISPs and organizations will >>> deploy >>> IPv6 in their networks. Right now, Tor clients connect to guards >>> using >>> IPv4 connectivity by default. >>> When networks first transition to IPv6, both IPv4 and IPv6 will >>> be enabled >>> on most networks in a so-called "dual-stack" configuration. This >>> is to not >>> break existing IPv4-only applications while enabling IPv6 >>> connectivity. >>> However, IPv6 connectivity may be unreliable and clients should >>> be able >>> to connect to the guard using the most reliable technology, >>> whether IPv4 >>> or IPv6. >>> In ticket #27490, we introduced the option ClientAutoIPv6ORPort >>> which >>> lets a client randomly choose between IPv4 or IPv6. However, this >>> random decision does not take into account unreliable >>> connectivity >>> or falling back to the competing IP version should one be >>> unreliable >>> or unavailable. >>> One way to select between IPv4 and IPv6 on a dual-stack network >>> is a >>> so-called "Happy Eyeballs" algorithm as per RFC 8305. In one, a >>> client >>> attempts an IP family, whether IPv4 or IPv6. Should it work, the >>> client >>> sticks with the working IP family. Otherwise, the client attempts >>> the >>> opposing version. This means if a dual-stack client has both IPv4 >>> and >>> IPv6, and IPv6 is unreliable, the client uses IPv4, and vice >>> versa. >>> In Proposal 299, we have attempted a IP fallback mechanism using >>> failure >>> counters and preferring IPv4 and IPv6 based on the state of the >>> counters. >>> However, Prop299 was not standard Happy Eyeballs and an >>> alternative, >>> standards-compliant proposal was requested in [P299-TRAC] to >>> avoid issues >>> from complexity caused by randomness. >>> This proposal describes a Tor implementation of Happy Eyeballs >>> and is >>> intended as a successor to Proposal 299. >>> 2. Address Selection >>> To be able to handle Happy Eyeballs in Tor, we will need to >>> modify the >>> data structures used for connections to guards, namely the extend >>> info >>> structure. >>> The extend info structure should contain both an IPv4 and an IPv6 >>> address. >>> This will allow us to try IPv4 and the IPv6 addresses should both >>> be >>> available on a relay and the client is dual-stack. >>> When parsing relay descriptors and filling in the extend info >>> data >>> structure, we need to fill in both the IPv4 and IPv6 address if >>> they both >>> are available. If only one family is available for a relay (IPv4 >>> or IPv6), >>> we should fill in the address for available family and leave the >>> opposing >>> family null. >> When we implement this feature in tor, it would be a good idea to call >> the >> two addresses "preferred" and "alternate" address. With this design, >> the low-level connection code doesn't have to know about reachable >> addresses, or IPv4/IPv6 preferences. It just has to try them in order. >>> 3. Connecting To A Relay >>> When a client connects to a guard using an extend info data >>> structure, we >>> should first check if there is an existing authenticated >>> connection. If >>> there is, we should use it. >> Tor's code already does this check: we won't need to change it. >>> If there is no existing authenticated connection for an extend >>> info, we >>> should attempt to connect using the first available, allowed, and >>> preferred >>> address. At the time of writing, this is IPv4. >> That's not quite true: most clients use IPv4 by default, but they can >> be >> configured to prefer IPv6, or only allow certain addresses. And bridge >> clients >> automatically use IPv6 if they are configured with an IPv6 bridge. >>> We should also schedule a timer for connecting using the other >>> address >>> should one be available and allowed, and the first attempted >>> version >>> fails. This should be higher than most client's successful TLS >>> authentication time. I propose that the timer is 15 seconds. The >>> reason >>> for this is to accommodate high-latency connections such as >>> dial-up and >>> satellite. >> In the worst case scenario, users see Tor Browser hang for 15 seconds >> before it makes a successful connection. That's not acceptable. >> Depending on their location, most tor clients authenticate to the >> first >> hop within 0.5-1.5 seconds. So I suggest we use a 1.5 second delay: >> https://metrics.torproject.org/onionperf-buildtimes.html >> In RFC 8305, the default delay is 250 milliseconds, and the maximum >> delay is 2 seconds. So 1.5 seconds is reasonable for TLS and tor link >> authentication. >> https://tools.ietf.org/html/rfc8305#section-8 >> (This delay will mainly affect initial bootstrap, because all of Tor's >> other connections are pre-emptive, or re-used.) >> A small number of clients may do wasted authentication. >> That's ok. Tor already does multiple bootstrap and guard connections. >> We have talked about this design in the team over the last few months. >> Our key insights are that: >> * TCP connections are cheap, but TLS is expensive >> * most failed TCP connections fail immediately in the kernel, some >> fail quickly with a response from the router, and others are >> blackholed >> and time out >> * it's unlikely that a client will fail to authenticate to a relay >> over one >> IP version, but succeed over the other IP version, because the >> directory >> authorities authenticate to each relay when they check reachability >> * some censorship systems only break authentication over IPv4, >> but they are rare >> So here are some alternative designs: >> 1. Tor connects to the preferred address and tries to authenticate. >> On failure, or after a 1.5 second delay, it connects to the >> alternate address >> and tries to authenticate. >> On the first successful authentication, it closes the other >> connection. >> This design places the least connection load on the network, but might >> add >> a bit of extra TLS load. >> 2. Tor connects via TCP to the preferred address. >> On failure, or after a 250 ms delay, it connects via TCP to the >> alternate >> address. >> On the first TCP success, tor attempts to authenticate immediately. >> On authentication failure, or after a 1.5 s delay, tor attempts to >> authenticate over the second TCP connection. >> On the first successful authentication, it closes the other >> connection. >> This design is the most reliable for clients, but it also puts a bit >> more >> connection load on dual-stack guards and authorities. >> 3. Tor connects via TCP to the preferred address. >> On failure, or after a 250ms delay, it connects via TCP to the >> alternate >> address. >> On the first TCP success, tor attempts to authenticate, and closes >> the >> other connection. >> This design looks similar to a web browser's implementation of Happy >> Eyeballs, because it closely follows the RFC. That might help hide tor >> from censors. It adds some extra connection load, but no extra TLS >> load. >> I suggest that we put all 3 alternative designs in the proposal, but >> start >> by implementing and testing alternative 1. >> When we implement this code, let's put the happy eyeballs part in a >> separate module, as much as possible. That helps us review the code, >> and make sure it has good test coverage. It also stops existing files >> and >> functions getting too big. >>> 4. Handling Connection Successes And Failures >>> Should a connection to a guard succeed and is authenticated via >>> TLS, we >>> can then use the connection. In this case, we should cancel all >>> other >>> connection timers and in-progress connections. Cancelling the >>> timers is >>> so we don't attempt new unnecessary connections when our existing >>> connection is successful, preventing denial-of-service risks. >>> However, if we fail all available and allowed connections, we >>> should tell >>> the rest of Tor that the connection has failed. This is so we can >>> attempt >>> another guard relay. >>> 5. Acknowledgments >>> Thank you so much to teor for the discussion of the happy >>> eyeballs proposal. >>> I wouldn't have been able to do this has it not been for your >>> help. >>> 6. Appendix >>> [P299-TRAC]: >>> https://trac.torproject.org/projects/tor/ticket/29801
Hi, My comments are inline.
The big problem that happy eyeballs was meant to solve was that often you might have something announcing an IPv6 prefix but that routing was not properly configured, so while the operating system thought it had IPv6 Internet it was actually just broken. In some cases, the IPv6 Internet would be partitioned as there weren't enough backup routes to fail over to in times of outages. For most purposes, as I understand it, this means either IPv6 connectivity to a host is there or it's not. There's not really a middle ground where it sometimes works but is flaky (i.e. where you can maintain a connection but it has high packet loss).
This sounds like a good candidate for a consensus parameter, such that we can switch the preference for all clients at once, not just the ones that have updated to the version we switch the preference in. There may also be other ordering parameters for the address candidates. We might want to avoid using IPv6 addresses that are using 6to4 or Teredo as we *know* those are tunnels and thus have encapsulation overhead, higher latency, and funnel all the traffic through centralised (even if distributed) points in the network.
The Happy Eyeballs specification doesn't just talk about having one v4 and one v6 address. In some cases, relays may be multihomed and so may have multiple v4 or v6 addresses. We should be able to race all the candidates.
To match the IETF protocol more closely, we should have a list of candidate addresses and order them according to our preferences.
The delay seems arbitrary. OnionPerf collects data on latency in the Tor network, and could be used to inform better timing choices for the best end user performance (the happiest eyeballs). If we choose to take this route, we should open new connections with a timeout of ~250ms, and only change the condition for deciding which is the connection we will use.
Creating TCP connections is not a huge issue, and we should be racing the connections with the ~250ms timeout anyway. All the designs will have this issue.
This is probably also the "simplest" to implement, as it means that the happy eyeballs algorithm is contained to the socket handling code. I don't believe that requiring authentication to complete is going to do anything more than generate load on relays. Either the packet loss is high enough that the three way handshake fails, or there is low packet loss. I don't think the case where requiring an additional few packets make it through helps you choose a better connection is going to be that common. Of course it is always possible to add a "PreferredAddressFamily" option to torrc for those that know they are on a bad IPv6 network.
I'd want to see some justification with some experimental (or even anecdotal) data as to why first successful authentication is the way to go. 3.1.3 is going to be the simpler option and, in my opinion, the best place to start. 3.1.3 can likely be implemented using exactly the algorithm in section 5 of RFC 8305, excluding portions relating to DNS because we already have all the candidates from the server descriptor.
Some issues that come to mind: - I wonder how many relay IPv6 addresses are actually using tunnels. At the levels of throughput they use, that overhead adds up. What is the additional bandwidth cost and what is the impact of reduced MSS? - What are the tunables? RFC8305 has some that would be applicable, and probably all of them could be consensus parameters if we wanted to tune them: * First Address Family Count * Connection Attempt Delay * Minimum Connection Attempt Delay * Maximum Connection Attempt Delay - How do we know what is going on? We do not collect metrics from clients about their usage, but we do collect metrics from relays. Are there any counters we should be adding to extra info descriptors to help us see whether or not this is working? Could clients help relays by reporting that a connection is being closed because they have another connection? (I don't know the answer, but RFC8305 does explicitly point out that it is a mitigation technique designed to hide problems from the user, which means that those problems might come back to haunt us later if we're not on top of them.) Thanks, Iain.
Hi Iain, Thanks for your review!
You're right, I think our worst-case scenario in the current tor implementation is 100% packet loss, which happens when a firewall is configured to drop packets. We should be much clearer about these two scenarios in the proposal (IPv4/IPv6 failure, and IPv4/IPv6 timeout). Another common scenario is very slow (DirPort) speeds, as a defence against old clients on tor26. But the DirPort is out of scope for this proposal.
Tor already has these IPv4 and IPv6 torrc options: * ClientUseIPv4 - use IPv4, on by default * ClientUseIPv6 - use IPv6, off by default, overridden by explicit bridge, PT, and proxy configs * ClientPreferIPv6ORPort - prefer IPv6, off by default At the moment, these options work well: * ClientUseIPv4 1 Only use IPv4 (other options are ignored) * ClientPreferIPv6ORPort 1 Try to use IPv6 as much as possible (overrides ClientUseIPv4 1 and ClientUseIPv6 0) * ClientUseIPv4 0 Only use IPv6 (other options are ignored) After this proposal is fully deployed, all valid combinations of options should work well. In particular: * the default should be: ClientUseIPv4 1 ClientUseIPv6 1 ClientPreferIPv6ORPort 0 (for load-balancing reasons) * tor clients should work with these defaults on IPv4-only, dual-stack, and IPv6-only networks (and they should continue to work on all these networks if ClientPreferIPv6ORPort is 1) * we should have consensus parameters for: ClientUseIPv6 (emergency use) ClientPreferIPv6ORPort (if most of the guards have IPv6, and it's fast) We should probably ClientUseIPv6 0 in the first alpha release, and then change the consensus parameter and torrc defaults after we've done enough testing. We should be clearer about these torrc options, consensus parameters, testing, and deployment in the proposal.
I'm not sure how this feature would work: most of the time, when tor is ordering addresses, it has already chosen a relay. It has exactly one IPv4 address, and an optional IPv6 address. This kind of ordering of multiple IPv6 addresses requires a pool of addresses from multiple relays. It's out of scope for this proposal, but it could be implemented as part of our pool refactor: https://trac.torproject.org/projects/tor/ticket/30817#comment:3
Tor relays only advertise 1 IPv4 address: https://gitweb.torproject.org/torspec.git/tree/dir-spec.txt#n392 and 0 or 1 IPv6 address: https://gitweb.torproject.org/torspec.git/tree/dir-spec.txt#n764 in their descriptor. The consensus only contains 1 IPv4 address: https://gitweb.torproject.org/torspec.git/tree/dir-spec.txt#n2297 and 0 or 1 IPv6 address: https://gitweb.torproject.org/torspec.git/tree/dir-spec.txt#n2316 per relay. Adding extra addresses is out of scope for this proposal. We could do it in a separate proposal, but it might not be the best use of limited space in the consensus. (If a relay machine is down, all its addresses are down. It's rare for a client to not be able to reach one IP address on a relay, but be able to reach another address on the same relay in the *same* IP family.)
With the current descriptor and consensus implementation, there will only ever be 1 or 2 addresses in the list for each relay. (There is one extend info data structure per relay connection request. Modifying other parts of the tor implementation is out of scope for this proposal.)
Here are the design tradeoffs for this section, which we should add to the proposal: * launching multiple TCP connections places up to 2x the socket load on dual-stack relays and authorities, because both connections may succeed, * launching multiple TLS connections places up to 2x the CPU load on dual-stack relays and authorities, because both connections may succeed, * increasing the delays between connections mitigates these issues, but reduces perceived performance, particularly at bootstrap time (pre-emptive circuits hide these delays after bootstrap).
The 1.5 second delay is based on Onionperf data, and we should reference the Onionperf figures in the proposal. See my previous review of an earlier draft of this proposal:
Tor already does multiple bootstrap and guard connections over IPv4, so I'm not sure exactly what design you're proposing. Can you give me an example?
Neel, that's not what I wrote in my last email:
A small number of clients will take longer than 1.5 seconds to authenticate. So we should only close a connection when the other connection to the relay successfully authenticates.
That's not true: Tor's last connection level denial of service event was November 2017 - February 2018. And there are occasional connection spikes on authorities and fallbacks. These connection DoSes need to be mentioned in the proposal.
I'm not sure exactly what issue you're referring to?
Middleboxes that only break IPv4 TLS are rare, but they do exist:
But we still want tor to work by default on those networks, so we should try IPv4 and IPv6 all the way up to TLS.
Of course it is always possible to add a "PreferredAddressFamily" option to torrc for those that know they are on a bad IPv6 network.
Tor already has this torrc option: * ClientPreferIPv6ORPort - prefer IPv6, off by default
It increases the risk of network-wide DoS, and fails to work around some censored networks. But it might be good for a simple initial test implementation.
All supported Tor client versions use microdescriptors, not server descriptors. Since consensus method 28 in tor 0.3.3.6, microdesc consensuses contain IPv6 addresses. (This is important during bootstrap.) See proposal 283 for context: https://gitweb.torproject.org/torspec.git/tree/proposals/283-ipv6-in-micro-c... We also intend to use this proposal to connect to the hard-coded fallbacks and authorities, some of which have IPv6 addresses. Ideally, we shouldn't need to change any of the code from proposal 283. But we might need to change the relay selection logic, because otherwise tor could chose a run of IPv4-only relays, and fail to bootstrap on an IPv6-only network. So we need to add another section to the proposal, I guess.
Here's one way we can mitigate this overhead: * tor clients prefer IPv4 by default, * tor uses a 1.5 second delay between IPv4 and IPv6 connections That way, most clients that can use IPv4, will end up using IPv4, and avoid this overhead. The clients that don't will fall into two categories: * IPv6-only, so the overhead is a small price to pay for connectivity, or * high-latency, so the overhead might not be noticeable anyway.
This value must be fixed at 1. Tor's code only connects to 1 relay at a time, and that relay only has 1 address from each family. Increasing the number of addresses per relay or per "happy eyeballs" attempt is out of scope for this proposal.
* Connection Attempt Delay
From Onionperf data, I think this should default to 1.5 seconds. But I'm happy to modify it based on testing, or future Onionperf measurements. Let's make it a torrc option and consensus parameter?
* Minimum Connection Attempt Delay
Dynamically adjusting the delay per client is out of scope for this proposal. It also carries privacy risks, unless we add some jitter. Let's fix the minimum at 10 milliseconds as recommended in RFC 8305, and adjust it network-wide using the "Connection Attempt Delay" consensus parameter.
* Maximum Connection Attempt Delay
As above, but if we choose to include TLS in the delay, we should set the maximum much higher than the RFC 8305 recommendation of 2 seconds. Let's make it 30 seconds, to match tor's existing timeout. (Users might want to set the delay this high on very slow networks.)
We should definitely be collecting the number of IPv4 and IPv6 connections to ORPorts. We should probably also distinguish authenticated (relay, authority reachability) and unauthenticated (client, bridge) connections. We should also be including these stats in the heartbeat logs. We were going to wait for PrivCount for these stats, but we didn't manage to implement it in the sponsored time we had available. So I don't think it makes sense to block further stats on PrivCount at this time.
Clients don't report circuit or stream close reasons to relays, to preserve privacy and avoid information leaks. Clients can't always report connection close reasons over the Tor protocol, because it sits below the TLS layer, but connections can be closed at the TCP stage. (Or any subsequent stage, including TLS, link, or link authentication.) T
Hi, I'm really sorry about the delay in responding to your review. I was busy with an internship (unrelated to Tor, but still related to security) and was out a lot in my "free time". I have implemented your requested changes and the GitHub PR is here: https://github.com/torproject/torspec/pull/87 Hopefully I have not missed anything. Most of these changes you (Iain and Teor) suggested sound good. I'm not a huge fan of preferring IPv4 in the case of tunneled IPv6 connections (reason: we stay with IPv4 longer than we should), but understand why you have it (reason: better network performance) and have added this change anyways. Thanks, Neel Chauhan On 2019-07-02 07:15, teor wrote:
Hi, On July 11, 2019 12:37:03 AM UTC, neel@neelc.org wrote:
Thanks for these revisions. I have some overall comments on the proposal: 1. The proposal is very large now. Let's add an intro section that splits the changes into: * initial feasability testing (for initial developer review) * minimum viable product (for testing in Tor Browser Alpha) * parameter tuning (for performance, load, and user experience) * relay statistics (for ongoing monitoring) * optional features, and how we will know if we need them (if users experience particular bugs) 2. Bridges can't be configured with an IPv4 and an IPv6 address in tor, because tor only accepts one IP address per bridge. Let's be clear that bridges are out of scope. (Tor already attempts to connect to all? its configured bridges.) 3. Each revision of this proposal has added text. Is there any text that is redundant or not essential? Can we make it shorter? After these revisions, I will do a final review. I hope we can get another tor developer to also do a final review. T -- teor ----------------------------------------------------------------------
Hi, I have split up the sections. The GitHub PR is here: https://github.com/torproject/torspec/pull/87 This was done as a fixup commit. If you want a new PR, please let me know. -Neel === https://www.neelc.org/ On 2019-07-13 12:47, teor wrote:
Just a reminder that this proposal (Prop306) needs to be reviewed: https://github.com/torproject/torspec/pull/87 -Neel === https://www.neelc.org/ On 2019-07-21 10:30, neel@neelc.org wrote:
Hi Neel,
On 30 Jul 2019, at 03:11, Neel Chauhan <neel@neelc.org> wrote:
Just a reminder that this proposal (Prop306) needs to be reviewed: https://github.com/torproject/torspec/pull/87
I can't find these changes that I requested to the proposal:
These parts of the proposal exist, but they are not in separate sections:
These parts of the proposal do not exist:
* relay statistics (for ongoing monitoring)
This modification is not in the current proposal:
2. Bridges can't be configured with an IPv4 and an IPv6 address in tor, because tor only accepts one IP address per bridge. Let's be clear that bridges are out of scope. (Tor already attempts to connect to all? its configured bridges.)
I added some suggest text to make bridges out of scope for this proposal. I don't see any deleted text:
3. Each revision of this proposal has added text. Is there any text that is redundant or not essential? Can we make it shorter?
David suggested we delete one section, and I suggested we keep some of it, but delete most of it. I also suggested we move some alternative options to an appendix. Is there anything else you think we can delete? T
Hi, I see this staying longer with IPv4 longer than we should also problematic, we are at the point that there are providers out there who do have more clients than IPv4 space which results in having them making carrier grade NAT. Some of them have the problem that their NAT gear gets maxed out at peak hours which results in heavy packet loss and very slow speeds, while their IPv6 connection is perfectly fine. That will not get better, i guess it will get worse in the future. So i would also prefer to use IPv6 if it works better. On 11.07.2019 02:37, neel@neelc.org wrote:
Hi,
On 2 Aug 2019, at 09:20, NOC <tor@afo-tm.org> wrote:
I see this staying longer with IPv4 longer than we should also problematic, we are at the point that there are providers out there who do have more clients than IPv4 space which results in having them making carrier grade NAT. Some of them have the problem that their NAT gear gets maxed out at peak hours which results in heavy packet loss and very slow speeds, while their IPv6 connection is perfectly fine. That will not get better, i guess it will get worse in the future. So i would also prefer to use IPv6 if it works better.
Currently, Tor clients don't use IPv6 unless they are specifically configured to use it. Some apps (OnionBrowser) use the OS network APIs to automatically configure Tor, but most don't. This proposal makes sure that Tor clients try IPv4, then try IPv6 after a short delay. If either works, the client will connect to the Tor network. At this stage, only 20% of guards support IPv6. But we are going to make sure at least one of the three client primary guards has IPv6. Ensuring at least one IPv6 client guard will increase traffic to IPv6 guards by up to 1.7x, which could cause load balancing issues. So we need to counter this load imbalance by trying IPv6 after IPv4. Once 33% of non-exit guards support IPv6, we can reduce the delay, or try IPv6 first at random. Once 67% of non-exit guards support IPv6, we can try IPv6 first. We are working on a funding proposal that will increase the number of IPv6 relays by automatically detecting, testing, and using IPv6 addresses. T
That would imply that guard relays run at 100% capacity which i can't confirm for any of my guards. Also seeing decline in usage would maybe give a incentive for some relay operators to configure IPv6 in their torrc, because there are thousands of relays which do have IPv6 connectivity, but the operator simply didn't configure it because it is not as easy as putting "ORPort 9001" in your torrc and it listens on IPv4 and IPv6. Sadly ORPort 9001 gives you a working relay which listens only on IPv4, so i would say the lack of IPv6 enabled relays is a problem which is created by how the config works and that a default torrc file loses no word about IPv6. Beside that if i look at https://metrics.torproject.org/bandwidth-flags.html the Tor network could lose their entire IPv4 only guards and still work, but after that guards would indeed run at 100% capacity. On 02.08.2019 02:05, teor wrote:
Hi,
On 2 Aug 2019, at 13:02, NOC <tor@afo-tm.org> wrote:
That would imply that guard relays run at 100% capacity which i can't confirm for any of my guards.
I have run guards at 100% before, but it's not ideal. Generally, Tor load-balances traffic to make sure that clients get consistently good bandwidth and latency. Adding significant extra load to some relays makes it hard to achieve this goal. In particular, 100% relay usage can significantly increase latency.
Also seeing decline in usage would maybe give a incentive for some relay operators to configure IPv6 in their torrc,
But IPv4-only relays *will* see a decrease in traffic, and IPv6 relays will see an increase. Sending 33% of client traffic to the 20% of guards that have IPv6 will cause an increase in usage for IPv6 relays (and a decrease for IPv4 relays). Even if we check the relay's IPv6 address after its IPv4 address, the traffic is still going to that relay. Tor clients on IPv6-only networks will also add extra to IPv6 relays. These clients don't work now, so we don't know how many there are. Once we've tested IPv6 in at least one stable release, we will have a better idea of the load balancing impacts of IPv6. And we can think about changing the address order. Another reason that we want to check IPv4 first is that is preserves tor's current behaviour. Adding an extra attempt at IPv6 once IPv4 has failed, is a smaller change that is unlikely to have any negative impacts. (Our last attempt at adding automatic client IPv6 didn't work, and we had to disable it in Tor Browser alpha.)
because there are thousands of relays which do have IPv6 connectivity, but the operator simply didn't configure it because it is not as easy as putting "ORPort 9001" in your torrc and it listens on IPv4 and IPv6.
Sadly ORPort 9001 gives you a working relay which listens only on IPv4, so i would say the lack of IPv6 enabled relays is a problem which is created by how the config works
We are working on a funding proposal that will increase the number of IPv6 relays by automatically detecting, testing, and using IPv6 addresses.
and that a default torrc file loses no word about IPv6.
You're right, I opened a ticket to add IPv6 ORPorts to the example torrcs: https://trac.torproject.org/projects/tor/ticket/31320
Beside that if i look at https://metrics.torproject.org/bandwidth-flags.html the Tor network could lose their entire IPv4 only guards and still work, but after that guards would indeed run at 100% capacity.
See my answer above: 100% capacity is not good for the network or clients. T
On Tue, Jun 25, 2019 at 9:24 PM <neel@neelc.org> wrote:
Hi, Neel! Thanks for working on this; I believe it's come a long way in the last month! Here are a few questions based on the current PR. * We need to revise the "relay selection changes" to match the vocabulary of guard-spec.txt. It's easy to say "select at least one relay with an ipv6 address", but it's not trivial to do so in practice. (Also, do we do this always, or do we do this only when we think we can connect to ipv6 addresses?) * We also need to think about what this algorithm means in terms of guard-spec.txt's data structures. Does it mean that each connection to a guard is replaced with two? Does it mean that some of the reachability variables are replaced by two? * The proposal considers TCP success vs authentication success as indicating that a connection has succeeded. There is a good alternative that reduces CPU load, however. The TLS handshake has multiple phases, and the expensive CPU stuff all happens after we receive a ServerHello message. If we treat an incoming ServerHello as meaning that the connection will be successful, we can avoid most wasted handshakes. [This would definitely not handle the problem where one of a server's addresses is correct but the other address is a different server entirely, but I hope we can catch that earlier in data flow, possibly at the authorities.] * The 1.5 second delay, and associated other hardcore times, should be a network parameter, transmitted in the consensus. 1.5 seconds can be the default, but we will want to keep the ability to tune it later on. * For pluggable transports, do we want to manage this process ourselves, or delegate the decisions to the PT? Each option has its own benefits and risks. cheers, -- Nick
Hi Nick,
On the pull request, I suggested that we make sure that at least one of the three primary guards has IPv6. (This change might also place a similar IPv6 requirement on the larger sets of guards chosen by clients. Is there a nice Venn diagram of all the guard sets?) Here's one way we could implement an IPv6 guard requirement: When choosing the last primary guard (or rotating any primary guard), if there are no IPv6 primary guards, pass a new flag CRN_NEED_IPV6 to router_choose_random_node(). CRN_NEED_IPV6 can be implemented like CRN_PREF_ADDR, but with a hard-coded preference for IPv6. /* On clients, only provide nodes that satisfy ClientPreferIPv6OR */ CRN_PREF_ADDR = 1<<7, (I cover the non-guard cases below.)
(Also, do we do this always, or do we do this only when we think we can connect to ipv6 addresses?)
Happy eyeballs does not and should not require the client to guess IPv6 reachability. Tor can't reliably get that information, because the results of OS network APIs may be unreliable, unavailable, or incorrect. (And past connectivity is not a reliable guide to future connectivity, particularly on mobile.) If we want to try to guess, that's an optimisation, which belongs in the "optional optimisations" section of the proposal.
I would prefer a new low-level network module that takes an IPv4 and IPv6 address for each connection request, and reports success if either address succeeds. And failure if both fail. (Note that IPv4-only, dual-stack, and IPv6-only are all valid address combinations. Relays, authorities, and fallbacks are IPv4 or dual stack, bridge lines are currently IPv4-only or IPv6-only, and v3 single onion service rendezvous direct connections can be all three.) This design would have a minimal impact on existing guard data structures and guard code. I'd like to put any other guard changes in the "optional optimisations" section of the proposal. Unless we are sure that they are essential.
Sounds sensible. Let's use the ServerHellos as the minimal viable product for merging and release in an alpha. So this feature belongs in the "minimal viable product" section of the proposal. Initial feasibility testing can just use TCP connections though.
Authority IPv4 or IPv6 reachability checks should catch this issue and mark the relayMas not Running. (And therefore it won't be in the client's consensus 2-4 hours after the bad address is in the descriptor or on the machine.) IPv4 reachability checks on relays should also catch most IPv4 misconfigurations. We also have a funding proposal to do IPv6 reachability checks on relays, which will catch IPv6 misconfigurations before relays upload their descriptors.
David and I suggested this change on the pull request.
I suggested on the proposal pull request that we make changing bridge lines (including pluggable transports) out of scope for this proposal. Bridge and PT lines don't currently support multiple IP addresses, and some transports (meek, snowflake) use a placeholder address, and the transport doesn't tell tor the real address. So they can use the new module API for consistency, but they will only ever pass it an IPv4-only or IPv6-only bridge connect request. Perhaps we need an "out of scope" section in the proposal. T
On Tue, Jun 25, 2019 at 9:24 PM <neel@neelc.org> wrote: FWIW, this proposal is now in the repository. Let's do the pending changes as patches on what's there now. -- Nick
participants (7)
-
Hajar El -
Iain Learmonth -
Neel Chauhan -
neel＠neelc.org
-
Nick Mathewson -
NOC -
teor