Emerald Onion's new relays
Hello, Beginning this Wednesday, 2019-Apr-03, Emerald Onion will be starting 40 new uncapped and unfiltered exit relays using HardenedBSD in Seattle. We will turn on 10 new relays per day and be monitoring performance. In tandem, we will be publishing our updated edge routing and Tor relay configurations on our Github. We'll also publish an updated transparency report (spoiler: zero new requests). This work is part of our efforts to saturate our new unmetered 10Gbps transit link in the Westin data center. There we peer with the Seattle Internet Exchange with a separate 10Gbps connection. We'll be using our existing IP scopes: 2620:18C::/36 23.129.64.0/24 We are in the process of creating an RPKI ROA for our prefixes, and we are still rebuilding our DNS resolver (thanks nusenu for the feedback). These things will be completed before we turn on the new relays. If you donated to Emerald Onion (a 501(c)(3)) nonprofit), thank you! We'll be naming the relays accordingly. We will still accept new donations for relay naming rights-- email me directly for more information. Lastly, and If you're unaware, I spoke about our work at DEFCON 26: https://www.youtube.com/watch?v=cs6a1i4Owic Cheers, -- Christopher Sheats Executive Director for Emerald Onion Email: yawnbox@emeraldonion.org Office (& Signal): +1-206-739-3390 Website: https://emeraldonion.org/
On Tue, Apr 02, 2019 at 04:36:37AM +0000, Christopher Sheats wrote:
Beginning this Wednesday, 2019-Apr-03, Emerald Onion will be starting 40 new uncapped and unfiltered exit relays using HardenedBSD in Seattle.
Great to hear! And for those who are thinking "omg what if that makes them too large a fraction of the network"... that means it's the perfect time for some of the other relay running nonprofits to step up and add some capacity too. :) --Roger
Does this include me? :)
On 2. Apr 2019, at 06:50, Roger Dingledine <arma@torproject.org> wrote:
to step up and add some capacity too. :)
--Roger
_______________________________________________ tor-relays mailing list tor-relays@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays
‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐ On Tuesday, April 2, 2019 4:50 AM, Roger Dingledine <arma@torproject.org> wrote:
On Tue, Apr 02, 2019 at 04:36:37AM +0000, Christopher Sheats wrote:
it's the perfect time for some of the other relay running nonprofits to step up and add some capacity too. :)
Would if I could (BrassHorn) :) UK fibre (and London co-location) is obscenely expensive :/
Hi Christopher / Emerald, This is all awesome news! Best of luck with your roll out, and please do not hesitate to reach out if there is anything we can do to assist you. I look forward to watching these relays come online!
On Apr 1, 2019, at 11:36 PM, Christopher Sheats <yawnbox@emeraldonion.org> wrote:
Hello,
Beginning this Wednesday, 2019-Apr-03, Emerald Onion will be starting 40 new uncapped and unfiltered exit relays using HardenedBSD in Seattle. We will turn on 10 new relays per day and be monitoring performance. In tandem, we will be publishing our updated edge routing and Tor relay configurations on our Github. We'll also publish an updated transparency report (spoiler: zero new requests). This work is part of our efforts to saturate our new unmetered 10Gbps transit link in the Westin data center. There we peer with the Seattle Internet Exchange with a separate 10Gbps connection. We'll be using our existing IP scopes:
2620:18C::/36 23.129.64.0/24
We are in the process of creating an RPKI ROA for our prefixes, and we are still rebuilding our DNS resolver (thanks nusenu for the feedback). These things will be completed before we turn on the new relays.
If you donated to Emerald Onion (a 501(c)(3)) nonprofit), thank you! We'll be naming the relays accordingly. We will still accept new donations for relay naming rights-- email me directly for more information. Lastly, and If you're unaware, I spoke about our work at DEFCON 26: https://www.youtube.com/watch?v=cs6a1i4Owic
Cheers,
-- Christopher Sheats Executive Director for Emerald Onion Email: yawnbox@emeraldonion.org Office (& Signal): +1-206-739-3390 Website: https://emeraldonion.org/
_______________________________________________ tor-relays mailing list tor-relays@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays
We are in the process of creating an RPKI ROA for our prefixes
Thanks for taking the extra steps to create a RPKI ROA to reduce the impact of BGP routing attacks on your prefixes. Extra points for doing RPKI-based Route Origin Validation on your BGP routers. I hope to convince everyone with such a high concentration of tor network capacity to make use of tor's OfflineMasterKey mode to safeguard your relay identity keys even in the event of a system compromise. Which basically implies automation because no one wants to handle (renew) more than 3 keys manually. I'm also encouraging you to use separate IP addresses for exit traffic [1] because that helps eliminate the impact on relay-to-relay communication when ISPs are ordered to BGP blackhole some exit IP addresses (as we have seen recently in the news).
40 new uncapped and unfiltered exit relays
I would suggest to not run uncapped tor instances but to set a per-instance limit of around 80-90% what a single core is able to handle, to avoid poor performance for the user. With growing bandwidth the CPU will spend considerable amount of resources just handling packets (kernel).
This work is part of our efforts to saturate our new unmetered 10Gbps transit link
As teor usually says, saturated links is not what we should be aiming for if we like performance. Thanks for adding such a significant amount of exit capacity. [1] https://2019.www.torproject.org/docs/tor-manual.html.en#OutboundBindAddressE... -- https://twitter.com/nusenu_ https://mastodon.social/@nusenu
I'm also encouraging you to use separate IP addresses for exit traffic [1] because that helps eliminate the impact on relay-to-relay communication when ISPs are ordered to BGP blackhole some exit IP addresses (as we have seen recently in the news).
I've been assigning a second set of IP addresses to my servers in case I want to run another instance of Tor. Would it be more prudent to use that second set of IP addresses as an OutboundBindAddressExit instead and use different ports as a better practice? Thanks, Conrad On Tue, Apr 2, 2019 at 12:35 PM nusenu <nusenu-lists@riseup.net> wrote:
We are in the process of creating an RPKI ROA for our prefixes
Thanks for taking the extra steps to create a RPKI ROA to reduce the impact of BGP routing attacks on your prefixes. Extra points for doing RPKI-based Route Origin Validation on your BGP routers.
I hope to convince everyone with such a high concentration of tor network capacity to make use of tor's OfflineMasterKey mode to safeguard your relay identity keys even in the event of a system compromise. Which basically implies automation because no one wants to handle (renew) more than 3 keys manually.
I'm also encouraging you to use separate IP addresses for exit traffic [1] because that helps eliminate the impact on relay-to-relay communication when ISPs are ordered to BGP blackhole some exit IP addresses (as we have seen recently in the news).
40 new uncapped and unfiltered exit relays
I would suggest to not run uncapped tor instances but to set a per-instance limit of around 80-90% what a single core is able to handle, to avoid poor performance for the user. With growing bandwidth the CPU will spend considerable amount of resources just handling packets (kernel).
This work is part of our efforts to saturate our new unmetered 10Gbps transit link
As teor usually says, saturated links is not what we should be aiming for if we like performance.
Thanks for adding such a significant amount of exit capacity.
[1] https://2019.www.torproject.org/docs/tor-manual.html.en#OutboundBindAddressE...
-- https://twitter.com/nusenu_ https://mastodon.social/@nusenu
_______________________________________________ tor-relays mailing list tor-relays@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays
-- Conrad Rockenhaus https://www.rockenhaus.com Cell: (254) 292-3350 Fax: (254) 875-0459
On 4/4/19, Conrad Rockenhaus <conrad@rockenhaus.com> wrote:
when ISPs are ordered to BGP blackhole some exit IP addresses
I've been assigning a second set of IP addresses to my servers in case I want to run another instance of Tor. Would it be more prudent to use that second set of IP addresses as an OutboundBindAddressExit instead and use different ports as a better practice?
ISP traffic filtering sinks, from the tor browser perspective, affecting traffic exiting a relay out through its exitpolicy to clearnet, can be... - dst based "sink traffic to there", appears as "cnn.com down", a minor issue, depending on scope of the sink. - src based "sink traffic from there", appears as "Internet down", a major issue, depending on scope of the sink. Unlike websites, and unless they're tied playing [geo]politics, ISP's really don't like to keep these sinks in place for a long time. Relay management such as OS updates, ssh, wget could get blocked if those addresses are in consensus. Then there is relay-to-relay traffic types that don't "exit", but can still get found and blocked. And the OR IP must be obviously not be blocked, else depending on scope, the relay won't receive traffic to move out any horizon. Tor should still allow config of 2 tor instances on one IP. If IP's are "free", and if operator survey says the exit functions are getting knocked off the tor network more often than entire OR's, try putting out the OutboundBindAddressExit on IP for sacrifice, instead of burning entire OR's which could otherwise be used more quietly as middle relays etc. An operators own cost, management, and ISP relationships may show running more relays is better IP, or net traffic pushed, wise than enduring a few sinks now and then. Probably every situation is different. Or try both and see. Common options from the manpage... Address address ORPort [address:]PORT|auto [flags] OutboundBindAddress IP OutboundBindAddressOR IP OutboundBindAddressExit IP First one implemented was OutboundBindAddress, then came OutboundBindAddressOR and OutboundBindAddressExit. All for different matrix of reasons.
Hello, Just a report-back: https://twitter.com/EmeraldOnion/status/1160694122069422080 "It took us 128 days to go from 1Gbps to 10Gbps advertised bandwidth with the progressive deployment of up to 56 relays (April 2 to August 8, 2019). In terms of actual bandwidth, we have a sustained throughout of roughly 5-6Gbps. We can sustain up to 15Gbps and burst to 20Gbps." Currently we're the fifth largest exit family at 1.2220% consensus weight, 1247.67 MiB/s advertised bandwidth, and 5.5546% exit probability. It costs us $1,600 per month, in donations, to manage this service. As you may know, we are all volunteers. There was some additional feedback that we got from Nusenu regarding capping bandwidth of individual Tor processes, and using a second IP for exit traffic. We are still trying to determine the per-process max bandwidth on our two hardware platforms, but we are planning on enabling these caps for quality of the service. When we originally deployed many new relays using Ansible-Relayor, we used a second IP per process. However, due to routing troubleshooting and relays not showing up on Relay Search, we removed the second IPs. We are still using only one per relay. We also found it challenging to manage this many relay IPs with Ansible-Relayor. Ansible-Relayor uses sequential IPs based on the listing from ifconfig. This presents a challenge because it is difficult to setup forward and reverse DNS in a predictable way. The second issue we have with running a secondary IP per Tor process is system load. Having more IPs opens more sockets, and we are already putting a lot of load on these multi-core servers. The third issue is that when people block our IPs, they block the scope. Should we use a secondary IP per relay if the IP is in the same scope? Should we then use two /24 scopes, and wouldn't both scopes end up getting blocked? If we were to use a second /24 for relays, how will Ansible-Relayor know to use a second IP scope for exiting? IPv4 dependency is a real burden. We would like to see Tor Project help Tor network operators more directly, both financially and securing IPv4 scopes for nonprofit organizations to take ownership of. The latter is needed until we can stop using IPv4 completely and operate only with IPv6. It is discouraging to see so many small and large network operators not using IPv6. Why is this such a problem? Tor Project, please increase your #IPv6 awareness/outreach similar to how ARIN and the other RIRs try very hard to do. Cheers, -- Christopher Sheats Executive Director for Emerald Onion Email: yawnbox@emeraldonion.org Office (& Signal): +1-206-739-3390 Website: https://emeraldonion.org/ -----Original Message----- From: tor-relays <tor-relays-bounces@lists.torproject.org> On Behalf Of grarpamp Sent: Thursday, April 4, 2019 12:45 AM To: tor-relays@lists.torproject.org Subject: Re: [tor-relays] Emerald Onion's new relays On 4/4/19, Conrad Rockenhaus <conrad@rockenhaus.com> wrote:
when ISPs are ordered to BGP blackhole some exit IP addresses
I've been assigning a second set of IP addresses to my servers in case I want to run another instance of Tor. Would it be more prudent to use that second set of IP addresses as an OutboundBindAddressExit instead and use different ports as a better practice?
ISP traffic filtering sinks, from the tor browser perspective, affecting traffic exiting a relay out through its exitpolicy to clearnet, can be... - dst based "sink traffic to there", appears as "cnn.com down", a minor issue, depending on scope of the sink. - src based "sink traffic from there", appears as "Internet down", a major issue, depending on scope of the sink. Unlike websites, and unless they're tied playing [geo]politics, ISP's really don't like to keep these sinks in place for a long time. Relay management such as OS updates, ssh, wget could get blocked if those addresses are in consensus. Then there is relay-to-relay traffic types that don't "exit", but can still get found and blocked. And the OR IP must be obviously not be blocked, else depending on scope, the relay won't receive traffic to move out any horizon. Tor should still allow config of 2 tor instances on one IP. If IP's are "free", and if operator survey says the exit functions are getting knocked off the tor network more often than entire OR's, try putting out the OutboundBindAddressExit on IP for sacrifice, instead of burning entire OR's which could otherwise be used more quietly as middle relays etc. An operators own cost, management, and ISP relationships may show running more relays is better IP, or net traffic pushed, wise than enduring a few sinks now and then. Probably every situation is different. Or try both and see. Common options from the manpage... Address address ORPort [address:]PORT|auto [flags] OutboundBindAddress IP OutboundBindAddressOR IP OutboundBindAddressExit IP First one implemented was OutboundBindAddress, then came OutboundBindAddressOR and OutboundBindAddressExit. All for different matrix of reasons. _______________________________________________ tor-relays mailing list tor-relays@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays
Hi,
On 12 Aug 2019, at 10:46, Christopher Sheats <yawnbox@emeraldonion.org> wrote:
The second issue we have with running a secondary IP per Tor process is system load. Having more IPs opens more sockets, and we are already putting a lot of load on these multi-core servers.
In the next few months, we want to add support for the Rust ed25519-dalek library in Tor. (It's currently our best candidate for useful but optional Rust code.) Having faster crypto should reduce the need for more tor instances.
...
IPv4 dependency is a real burden. We would like to see Tor Project help Tor network operators more directly, both financially and securing IPv4 scopes for nonprofit organizations to take ownership of. The latter is needed until we can stop using IPv4 completely and operate only with IPv6.
In general, The Tor Project Inc. tries to avoid operating Tor network infrastructure. So IPv4 netblock allocations are a good task for individual volunteers, and Torservers and similar organisations.
It is discouraging to see so many small and large network operators not using IPv6. Why is this such a problem?
Tor relays don't automatically detect IPv6 addresses, and they don't test the reachability of IPv6 ORPorts. We are working on a grant application to add this support in Tor. (It's more complex than it seems, because we need to split the reachability checks per-ORPort, and add IPv6 extend support to Tor relays.)
Tor Project, please increase your #IPv6 awareness/outreach similar to how ARIN and the other RIRs try very hard to do.
I'll add an awareness objective to our grant application. T
On 12.08.2019 02:46, Christopher Sheats wrote:
It is discouraging to see so many small and large network operators not using IPv6. Why is this such a problem? Tor Project, please increase your #IPv6 awareness/outreach similar to how ARIN and the other RIRs try very hard to do.
+1 Twitter :-( Wish you're on Mastodon, too. -- Ciao Marco!
On Mon, 12 Aug 2019 00:46:50 +0000 Christopher Sheats <yawnbox@emeraldonion.org> wrote:
Tor Project, please increase your #IPv6 awareness/outreach similar to how ARIN and the other RIRs try very hard to do.
Before outreach Tor would need some actual IPv6 support, as in using it for the actual traffic of relay-to-relay communication. I tried running a few relays with very fast IPv6 and slow IPv4 (due to a common NAT frontend which was the bottleneck), but it was a complete nonstarter. Tor supports IPv6 very poorly and nobody cares much. -- With respect, Roman
Hi,
On 13 Aug 2019, at 05:08, Roman Mamedov <rm@romanrm.net> wrote:
On Mon, 12 Aug 2019 00:46:50 +0000 Christopher Sheats <yawnbox@emeraldonion.org> wrote:
Tor Project, please increase your #IPv6 awareness/outreach similar to how ARIN and the other RIRs try very hard to do.
Before outreach Tor would need some actual IPv6 support, as in using it for the actual traffic of relay-to-relay communication. I tried running a few relays with very fast IPv6 and slow IPv4 (due to a common NAT frontend which was the bottleneck), but it was a complete nonstarter.
Tor relays currently don't connect over IPv6. When 10% of the network supported IPv6, there wasn't much point, because putting a very small number of paths over IPv6 has privacy risks. So we focused on client, guard, and exit IPv6 support. But currently, about 30% of the consensus weight supports IPv6. So we are working on a grant for IPv6 support (see below). We won't be able to prefer IPv6 until 50-67% of relays support IPv6, for load-balancing and privacy reasons. But we plan on using the "Happy Eyeballs" (RFC 8305) algorithm on dual-stack relays. So sufficiently slow IPv4 will cause relays to connect over IPv6. (And we can tune the load-balancing using the IPv4 to IPv6 delay.)
Tor supports IPv6 very poorly and nobody cares much.
Lots of us care about IPv6. Our problem is finding *funders* who care enough to pay for the time we need to implement this complex feature. But we're working on a grant application right now: On 12 Aug 2019, at 11:54, teor <teor@riseup.net> wrote:
It is discouraging to see so many small and large network operators not using IPv6. Why is this such a problem?
Tor relays don't automatically detect IPv6 addresses, and they don't test the reachability of IPv6 ORPorts. We are working on a grant application to add this support in Tor. (It's more complex than it seems, because we need to split the reachability checks per-ORPort, and add IPv6 extend support to Tor relays.)
Tor Project, please increase your #IPv6 awareness/outreach similar to how ARIN and the other RIRs try very hard to do.
I'll add an awareness objective to our grant application.
T
Hi On 12.08.2019 23:39, teor wrote:
Hi,
On 13 Aug 2019, at 05:08, Roman Mamedov <rm@romanrm.net <mailto:rm@romanrm.net>> wrote:
On Mon, 12 Aug 2019 00:46:50 +0000 Christopher Sheats <yawnbox@emeraldonion.org <mailto:yawnbox@emeraldonion.org>> wrote:
Tor Project, please increase your #IPv6 awareness/outreach similar to how ARIN and the other RIRs try very hard to do.
Before outreach Tor would need some actual IPv6 support, as in using it for the actual traffic of relay-to-relay communication. I tried running a few relays with very fast IPv6 and slow IPv4 (due to a common NAT frontend which was the bottleneck), but it was a complete nonstarter.
Tor relays currently don't connect over IPv6. When 10% of the network supported IPv6, there wasn't much point, because putting a very small number of paths over IPv6 has privacy risks. So we focused on client, guard, and exit IPv6 support.
But currently, about 30% of the consensus weight supports IPv6. So we are working on a grant for IPv6 support (see below).
We won't be able to prefer IPv6 until 50-67% of relays support IPv6, for load-balancing and privacy reasons. But we plan on using the "Happy Eyeballs" (RFC 8305) algorithm on dual-stack relays. So sufficiently slow IPv4 will cause relays to connect over IPv6. (And we can tune the load-balancing using the IPv4 to IPv6 delay.)
I still would say that these stats are deeply flawed. Looking at the Autonomous Systems where the relays are located from the top100, 99 of them do support IPv6 (85,7625& consensus weight), the only one which doesn't support is AS4224 but since they manage their AS themselves they would only need to ask their LIR and would get IPv6. So my conclusion is not that there is any need to wait for adaption, only for software support. Release one stable from which point you need IPv6 and the operator will see that there is something to be done. You won't affect older versions since they still can speak with you but you won't get in the consensus from that point because you don't fulfill all requirements for it.
Hi,
On 14 Aug 2019, at 03:42, NOC <tor@afo-tm.org> wrote:
On 12.08.2019 23:39, teor wrote:
On 13 Aug 2019, at 05:08, Roman Mamedov <rm@romanrm.net> wrote:
On Mon, 12 Aug 2019 00:46:50 +0000 Christopher Sheats <yawnbox@emeraldonion.org> wrote:
Tor Project, please increase your #IPv6 awareness/outreach similar to how ARIN and the other RIRs try very hard to do.
Before outreach Tor would need some actual IPv6 support, as in using it for the actual traffic of relay-to-relay communication. I tried running a few relays with very fast IPv6 and slow IPv4 (due to a common NAT frontend which was the bottleneck), but it was a complete nonstarter.
Tor relays currently don't connect over IPv6. When 10% of the network supported IPv6, there wasn't much point, because putting a very small number of paths over IPv6 has privacy risks. So we focused on client, guard, and exit IPv6 support.
But currently, about 30% of the consensus weight supports IPv6. So we are working on a grant for IPv6 support (see below).
We won't be able to prefer IPv6 until 50-67% of relays support IPv6, for load-balancing and privacy reasons. But we plan on using the "Happy Eyeballs" (RFC 8305) algorithm on dual-stack relays. So sufficiently slow IPv4 will cause relays to connect over IPv6. (And we can tune the load-balancing using the IPv4 to IPv6 delay.) I still would say that these stats are deeply flawed. Looking at the Autonomous Systems where the relays are located from the top100, 99 of them do support IPv6 (85,7625& consensus weight), the only one which doesn't support is AS4224 but since they manage their AS themselves they would only need to ask their LIR and would get IPv6.
The top 100 relays are only 13-18% of the total advertised bandwidth: https://metrics.torproject.org/advbwdist-relay.html?start=2019-05-16&end=201... https://metrics.torproject.org/bandwidth.html
So my conclusion is not that there is any need to wait for adaption, only for software support.
It's hard to draw accurate conclusions from less than 20% of the network. What about the other 80% of the network?
Release one stable from which point you need IPv6 and the operator will see that there is something to be done. You won't affect older versions since they still can speak with you but you won't get in the consensus from that point because you don't fulfill all requirements for it.
I don't think kicking relay operators off the network when they upgrade to a new version is helpful. We *want* people to upgrade. Also, abruptly reducing the capacity of the network is a bad experience for Tor users. They will have slow traffic or connection failures. We call these kinds of abrupt changes "flag days", and we try hard to avoid them. Here's a plan that will take longer, but help more relay operators get on IPv6: Automatically detect and use IPv6 on relays: 1. Release a tor version that automatically detects IPv6 on relays, and uses it if it works. But turn it off by default, and have an option and a consensus parameter to turn it on. 2. Ask relay operators to test the new feature in the alphas and release candidates. 3. Once the feature has had enough testing, turn it on for all relays using the consensus parameter. 4. Encourage relay operators to configure IPv6 on their relay's network stack and upstream connection Reduce the consensus weight for IPv4-only relays: 1. Wait until we have enough dual-stack relays, that we can afford to send them more traffic. (We need metrics for IPv6 traffic, which we don't have right now.) 2. Add a consensus parameter and consensus method that reduces the consensus weight for IPv4-only relays by a percentage. 3. Gradually reduce the weight of IPv4 relays. 4. Encourage relay operators to configure IPv6 on their relay's network stack and upstream connection Wait until we have seen some good research on non-clique anonymity networks, or remove IPv4 and allow IPv6 at the same time. Allow IPv6-only guards: 1. Add automatic IPv6 support to clients, so clients use IPv4 and IPv6 when available 2. Optional: add automatic IPv6 support to bridges, so bridges use IPv4 and IPv6 when available 3. Wait until we have enough dual-stack middle relays, and dual-stack or IPv6 clients and bridges 4. Allow IPv6-only guards 5. Optional: allow IPv6-only bridges Allow IPv6-only exits: 1. Improve Tor client and Tor exit support for IPv6 exit connections, allowing clients to discover and remember whether a site is IPv4-only, dual-stack, or IPv6-only 2. Wait until we have enough dual-stack middle relays, and enough internet sites that support IPv6 3. Allow IPv6-only exits Allow IPv6-only middles: 1. Wait until we have enough dual-stack guards and exits 2. Allow IPv6-only middles Remove IPv4-only relays: 1. Wait until the proportion of IPv4-only guards, middles, or exits is small enough 2. Remove IPv4-only relays from that role (we can turn guards and exits into middles) I've added this draft plan to the IPv6 roadmap page: https://trac.torproject.org/projects/tor/wiki/org/roadmaps/Tor/IPv6Features#... T
On 15.08.2019 00:50, teor wrote:
Hi,
On 14 Aug 2019, at 03:42, NOC <tor@afo-tm.org <mailto:tor@afo-tm.org>> wrote:
On 12.08.2019 23:39, teor wrote:
On 13 Aug 2019, at 05:08, Roman Mamedov <rm@romanrm.net <mailto:rm@romanrm.net>> wrote:
On Mon, 12 Aug 2019 00:46:50 +0000 Christopher Sheats <yawnbox@emeraldonion.org <mailto:yawnbox@emeraldonion.org>> wrote:
Tor Project, please increase your #IPv6 awareness/outreach similar to how ARIN and the other RIRs try very hard to do.
Before outreach Tor would need some actual IPv6 support, as in using it for the actual traffic of relay-to-relay communication. I tried running a few relays with very fast IPv6 and slow IPv4 (due to a common NAT frontend which was the bottleneck), but it was a complete nonstarter.
Tor relays currently don't connect over IPv6. When 10% of the network supported IPv6, there wasn't much point, because putting a very small number of paths over IPv6 has privacy risks. So we focused on client, guard, and exit IPv6 support.
But currently, about 30% of the consensus weight supports IPv6. So we are working on a grant for IPv6 support (see below).
We won't be able to prefer IPv6 until 50-67% of relays support IPv6, for load-balancing and privacy reasons. But we plan on using the "Happy Eyeballs" (RFC 8305) algorithm on dual-stack relays. So sufficiently slow IPv4 will cause relays to connect over IPv6. (And we can tune the load-balancing using the IPv4 to IPv6 delay.)
I still would say that these stats are deeply flawed. Looking at the Autonomous Systems where the relays are located from the top100, 99 of them do support IPv6 (85,7625% consensus weight), the only one which doesn't support is AS4224 but since they manage their AS themselves they would only need to ask their LIR and would get IPv6.
The top 100 relays are only 13-18% of the total advertised bandwidth: https://metrics.torproject.org/advbwdist-relay.html?start=2019-05-16&end=201... https://metrics.torproject.org/bandwidth.html
I never wrote about the top100 relays, relays don't matter, they come and go. It is important who does host them, that is why i looked at the AS, because the providers won't stop offer IPv6 if they have deployed it once. And that is why i think the complete roadmap is not useful at all and will delay everything just unnecessary.
Hi,
On 16 Aug 2019, at 01:02, NOC <tor@afo-tm.org> wrote:
On 15.08.2019 00:50, teor wrote:
On 14 Aug 2019, at 03:42, NOC <tor@afo-tm.org> wrote:
On 12.08.2019 23:39, teor wrote:
On 13 Aug 2019, at 05:08, Roman Mamedov <rm@romanrm.net> wrote:
On Mon, 12 Aug 2019 00:46:50 +0000 Christopher Sheats <yawnbox@emeraldonion.org> wrote:
Tor Project, please increase your #IPv6 awareness/outreach similar to how ARIN and the other RIRs try very hard to do.
Before outreach Tor would need some actual IPv6 support, as in using it for the actual traffic of relay-to-relay communication. I tried running a few relays with very fast IPv6 and slow IPv4 (due to a common NAT frontend which was the bottleneck), but it was a complete nonstarter.
Tor relays currently don't connect over IPv6. When 10% of the network supported IPv6, there wasn't much point, because putting a very small number of paths over IPv6 has privacy risks. So we focused on client, guard, and exit IPv6 support.
But currently, about 30% of the consensus weight supports IPv6. So we are working on a grant for IPv6 support (see below).
We won't be able to prefer IPv6 until 50-67% of relays support IPv6, for load-balancing and privacy reasons. But we plan on using the "Happy Eyeballs" (RFC 8305) algorithm on dual-stack relays. So sufficiently slow IPv4 will cause relays to connect over IPv6. (And we can tune the load-balancing using the IPv4 to IPv6 delay.)
I still would say that these stats are deeply flawed. Looking at the Autonomous Systems where the relays are located from the top100, 99 of them do support IPv6 (85,7625% consensus weight), the only one which doesn't support is AS4224 but since they manage their AS themselves they would only need to ask their LIR and would get IPv6.
The top 100 relays are only 13-18% of the total advertised bandwidth: https://metrics.torproject.org/advbwdist-relay.html?start=2019-05-16&end=201... https://metrics.torproject.org/bandwidth.html
I never wrote about the top100 relays, relays don't matter, they come and go. It is important who does host them, that is why i looked at the AS, because the providers won't stop offer IPv6 if they have deployed it once. And that is why i think the complete roadmap is not useful at all and will delay everything just unnecessary.
We try to add new features, while keeping the network stable and fast, and protecting user anonymity. Sometimes we are too cautious, other times we break things. We'll be able to judge the right speed once we've released the first few new IPv6 features. Having funding will also help us go faster. And it would really help if we had some researchers tell us how to do anonymity in a mixed IPv4-only/dual-stack/IPv6-only network. T -- teor ----------------------------------------------------------------------
On 19.08.2019 05:03, teor wrote:
We'll be able to judge the right speed once we've released the first few new IPv6 features. Having funding will also help us go faster.
Maybe this year before Christmas a donation call for IPv6 features could be made ;-) Similar to last year, when the Mozilla Foundation doubled donations. -- ╰_╯ Ciao Marco!
On 20 Aug 2019, at 06:20, lists@for-privacy.net wrote:
On 19.08.2019 05:03, teor wrote:
We'll be able to judge the right speed once we've released the first few new IPv6 features. Having funding will also help us go faster.
Maybe this year before Christmas a donation call for IPv6 features could be made ;-) Similar to last year, when the Mozilla Foundation doubled donations.
I'm not sure what our end of year campaign will be this year. But we are working on funding for IPv6: On 13 Aug 2019, at 07:39, teor <teor@riseup.net> wrote:
Tor supports IPv6 very poorly and nobody cares much.
Lots of us care about IPv6. Our problem is finding *funders* who care enough to pay for the time we need to implement this complex feature. But we're working on a grant application right now:
On 12 Aug 2019, at 11:54, teor <teor@riseup.net> wrote:
It is discouraging to see so many small and large network operators not using IPv6. Why is this such a problem?
Tor relays don't automatically detect IPv6 addresses, and they don't test the reachability of IPv6 ORPorts. We are working on a grant application to add this support in Tor. (It's more complex than it seems, because we need to split the reachability checks per-ORPort, and add IPv6 extend support to Tor relays.)
Tor Project, please increase your #IPv6 awareness/outreach similar to how ARIN and the other RIRs try very hard to do.
I'll add an awareness objective to our grant application.
T
When we originally deployed many new relays using Ansible-Relayor, we used a second IP per process. However, due to routing troubleshooting and relays not showing up on Relay Search, we removed the second IPs.
was this an onionoo issue or did they not show up in the tor consensus either? Making use of OutboundBindAddressExit should not actually affect the reachability/visiblity on Relay Search.
We are still using only one per relay. We also found it challenging to manage this many relay IPs with Ansible-Relayor. Ansible-Relayor uses sequential IPs based on the listing from ifconfig.
yes, that is the default behavior. You can also set it manually (which ) using the vars: tor_v4ips tor_available_public_ipv4s These role variables are not actually documented in the README because I aimed for automation (no need to set them manually out of the box), but yes, there might be rare cases where you want to set them manually.
This presents a challenge because it is difficult to setup forward and reverse DNS in a predictable way.
Ideally you would automate that if your DNS provider has an API since the behavior should be deterministic. One other IP usage strategy that I had in mind to make increasing and decreasing the instance count less painful for DNS changes is the following: current behavior when 4 IPs are available and tor_dedicatedExitIP is used: .1 OR IP#1 .2 OR IP#2 .3 exit IP #1 .4 exit IP #2 potentially new strategy: .1 OR IP #1 .2 exit IP #1 .3 OR IP #2 .4 exit IP #2 but that would actually be counterproductive is you want to use two distinct /24 prefixes for in and outbound IPs (but still possible via manually setting above vars).
The second issue we have with running a secondary IP per Tor process is system load. Having more IPs opens more sockets,
I'm not sure I understood you there. Running multiple instances increases the amount of sockets but using OutboundBindAddressExit should not actually increase the amount of TCP connections/sockets used (when compared to not using that tor feature) since exit connections are always new TCP connections, no matter whether it uses the same or a distinct IP address for the exit connection. Maybe you can elaborate? If you worry about load/sockets you could decrease the amount of tor instances since you can probably do your current exit probability with a lot less instances.
The third issue is that when people block our IPs, they block the scope.
Some will, but most automated blocking mechanisms probably just block the source IP they observed or the IP address as seen in the tor consensus. Maybe it is even enough to have just distinct (smaller) inetnum objects instead of distinct /24 prefixes.
Should we use a secondary IP per relay if the IP is in the same scope?
I do believe it is still valuable to use OutboundBindAddressExit even if they reside in the same /24 prefix. That said, that feature is aimed at operators that have the IPs available, you probably should not use it if you are short on IP addresses.
If we were to use a second /24 for relays, how will Ansible-Relayor know to use a second IP scope for exiting?
by using the vars mentioned above. to illustrate that: In combination with tor_dedicatedExitIP ansible-relayor would make use of 198.51.100.x for exiting while using 192.0.2.x for OR ports this way: tor_v4ips: - 192.0.2.1 - 192.0.2.2 tor_available_public_ipv4s: - 192.0.2.1 - 192.0.2.2 - 198.51.100.1 - 198.51.100.2
Tor Project, please increase your #IPv6 awareness/outreach similar to how ARIN and the other RIRs try very hard to do.
+1 -- https://twitter.com/nusenu_ https://mastodon.social/@nusenu
Conrad Rockenhaus:
I'm also encouraging you to use separate IP addresses for exit traffic [1] because that helps eliminate the impact on relay-to-relay communication when ISPs are ordered to BGP blackhole some exit IP addresses (as we have seen recently in the news).
I've been assigning a second set of IP addresses to my servers in case I want to run another instance of Tor. Would it be more prudent to use that second set of IP addresses as an OutboundBindAddressExit instead and use different ports as a better practice?
yes, I think so -- https://twitter.com/nusenu_ https://mastodon.social/@nusenu
participants (12)
-
Christopher Sheats -
Colin Childs -
Conrad Rockenhaus -
Gareth Llewellyn -
grarpamp -
lists＠for-privacy.net -
niftybunny -
NOC -
nusenu -
Roger Dingledine -
Roman Mamedov -
teor