The current limit is 2 connections per IP address. This affects single onion services, because they don't use guards.
Can you please make sure that you only have one or two Single Onion Services on each outbound IP address?
I think the NYT is okay (separate IPs?) but if I understand this right, this is going to hamper EOTK development, since I have ~ 12 worker onions spread over 6 quad-core machines, and then publish up to 10 additional "service" addresses via OnionBalance ... all behind my single DSL NAT firewall that protects them from inbound traffic.
Hmmm...
- a
Because the circuit-creation limit is applied at the guard, wouldn’t this affect hidden sevices instead of single onion services?
Aaron
On Feb 1, 2018, at 7:08 PM, Alec Muffett alec.muffett@gmail.com wrote:
The current limit is 2 connections per IP address. This affects single onion services, because they don't use guards.
Can you please make sure that you only have one or two Single Onion Services on each outbound IP address?
I think the NYT is okay (separate IPs?) but if I understand this right, this is going to hamper EOTK development, since I have ~ 12 worker onions spread over 6 quad-core machines, and then publish up to 10 additional "service" addresses via OnionBalance ... all behind my single DSL NAT firewall that protects them from inbound traffic.
Hmmm...
- a
tor-onions mailing list tor-onions@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-onions
I am not going to pretend that I fully understand the DDoS mitigations yet, but experience at two jobs has taught me that at least three entire countries essentially present themselves from behind small numbers of heavily NATed addresses, so I hope that the mitigations are NAT-friendly.
ISTR that UAE and Singapore are two such, I forget the third?
-a
On 2 Feb 2018 00:10, "A. Johnson" aaron.m.johnson@nrl.navy.mil wrote:
Because the circuit-creation limit is applied at the guard, wouldn’t this affect hidden sevices instead of single onion services?
On 02 Feb (00:23:14), Alec Muffett wrote:
I am not going to pretend that I fully understand the DDoS mitigations yet, but experience at two jobs has taught me that at least three entire countries essentially present themselves from behind small numbers of heavily NATed addresses, so I hope that the mitigations are NAT-friendly.
ISTR that UAE and Singapore are two such, I forget the third?
I've been running the circuit creation mitigation for weeks now in different forms which had much more aggressive threshold in the beginning.
At most, my Guard identified 550 ish client address for which I've investigated a bit. They were all from big hosting corp that is dedicatedpanel.com, vultr.com LeaseWeb and Hetzner (the OVH clients were gone at that time).
The majority (82%) was Hetzner.
Thus so far I would say that it is not impacting that much single countries NATed in some ways or another.
This doesn't mean it won't be *especially* when 70% of the network will be rolling out those defenses. We really need to keep a sharp eye on this and adjust accordingly.
Cheers! David
Hi,
I didn't give much detail here:
On 2 Feb 2018, at 11:08, Alec Muffett alec.muffett@gmail.com wrote:
The current limit is 2 connections per IP address. This affects single onion services, because they don't use guards.
Can you please make sure that you only have one or two Single Onion Services on each outbound IP address?
So I'm going to quote from my email to tor-relays earlier today:
Here are the mitigations again:
o Major features:
- Give relays some defenses against the recent network overload. We start with three defenses (default parameters in parentheses). First: if a single client address makes too many connections (>100), hang up on further connections. Second: if a single client address makes circuits too quickly (more than 3 per second, with an allowed burst of 90) while also having too many connections open (3), refuse new create cells for the next while (1-2 hours).
We could patch clients so they never exceed this number of circuits by default. But that would penalise large clients that have a dedicated IP address.
For Alec's home development use case, limiting client bursts may be helpful.
Third: (DoS mitigation) if a client asks to establish a rendezvous point to you directly, ignore the request. These defenses can be manually controlled by new torrc options, but relays will also take guidance from consensus parameters, so there's no need to configure anything manually. Implements ticket 24902.
This shuts down Tor2web, which is a major source of the current load.
On 2 Feb 2018, at 03:04, David Goulet dgoulet@torproject.org wrote:
On 01 Feb (04:01:10), grarpamp wrote:
Applications that use a lot of resources will have to rate-limit themselves. Otherwise, relays will rate-limit them.
It's possible if relays figure that stuff by #2 might not be an attack per se, but could be user activities... that relays might push back on that one by...
- Seeking significantly higher default values committed
- Seeking default action committed as off
- Setting similar on their own relays if commits don't
work. And by not being default off, it should be prominently documented if #2 affects users activities [1].
That I agree. We've set up default values for now and they will probably be adapted over time so for now this is experimental to see how much we make people unhappy (well except for the people doing the DoS ;).
But I do agree that we should document some "real life" use cases that could trigger defenses at the relay in some very public way (blog post or wiki) before this goes wide in the network. Large amount of tor clienst behind NAT is one I have in mind, IPv6 as well...
We did some analysis when we were choosing these figures.
It takes a few hundred clients behind an IP address, to have a 50% probability of 3 clients choosing the same large guard. That's unusual. And if clients see their guard timeout, they will move to another guard.
Here are some other scenarios:
Peer-to-peer clients like Ricochet, when the user has >90 contacts, but only when there are hundreds of other clients on the same IP address.
Any Tor client that doesn't use guards. For example:
Bridges with multiple users, but only when there are >=3 bridges per outbound IP address. (This is unlikely, because bridges need their own IPv4 address. If you have multiple bridges using the default route, and multiple IP addresses, set OutboundBindAddress on each bridge.) We will need to consider this issue when we allow IPv6 bridges without a public IPv4 address. Perhaps an appropriate solution is to make bridge clients use vanguards.
Multiple (>=3) Tor2web or single onion services using separate tor instances, behind a single IP address, making large numbers of circuits. This is a likely source of our current issues.
Alec wrote:
I think the NYT is okay (separate IPs?)
I hear Facebook is ok as well.
but if I understand this right, this is going to hamper EOTK development, since I have ~ 12 worker onions spread over 6 quad-core machines, and then publish up to 10 additional "service" addresses via OnionBalance ... all behind my single DSL NAT firewall that protects them from inbound traffic.
For IP addresses with 3 or more connections to a single guard, the guard imposes a limit of 1 circuit every 3 seconds, with a 90 circuit burst allowance.
Are these single onion services or do they use guards?
If they are single onion services, then the probability that 3 of your 13? instances will choose the same relays for their directory, HSDir, intro or rend points is small (< 10%). If it happens, and if they build more than 90 circuits to the same relay, the defence will trigger. Then both instances will try another relay. You should probably limit your circuit construction rate if this happens a lot.
If they use guards, then the probability that 3 of your 13? instances will choose the same guard is negligible (1 in 1 million). The defence will not trigger.
Aaron wrote:
Because the circuit-creation limit is applied at the guard, wouldn’t this affect hidden sevices instead of single onion services?
It will only trigger if hundreds of guard-using clients are behind a single IP address. See above.
Alec wrote:
I am not going to pretend that I fully understand the DDoS mitigations yet, but experience at two jobs has taught me that at least three entire countries essentially present themselves from behind small numbers of heavily NATed addresses, so I hope that the mitigations are NAT-friendly.
ISTR that UAE and Singapore are two such, I forget the third?
They are as NAT-friendly as possible under the circumstances. We may have issues if thousands of users are behind a single IP.
In this case, we could recommend bridges, or increase the DDoS mitigation limits.
T
On Fri, Feb 02, 2018 at 02:23:24PM +1100, teor wrote:
For IP addresses with 3 or more connections to a single guard, the guard imposes a limit of 1 circuit every 3 seconds, with a 90 circuit burst allowance.
3 circuits every 1 seconds, actually. Think of it like a token bucket with a size of 90 circuits and a refill rate of 3 per second.
If it happens, and if they build more than 90 circuits to the same relay, the defence will trigger. Then both instances will try another relay.
I think Tor clients who have all their create cells responded to with destroy cells won't abandon that relay. That is, getting a destroy cell in response to a create cell is not an indication that the relay is broken, so it won't convince us to stop trying that one.
That "feature" is actually part of the calculus here, since we want to think very carefully about how our choices shape the behavior of the millions of enthusiastic high-bandwidth Tor clients that are overwhelming the network.
Because the circuit-creation limit is applied at the guard, wouldn???t this affect hidden sevices instead of single onion services?
It will only trigger if hundreds of guard-using clients are behind a single IP address.
I expect a popular onion service that doesn't use guards and that runs many Tor instances on the same IP address will trigger the defense often: because it doesn't use guards, each new circuit it builds in response to a rendezvous request will pick an entry point at random, and if some of the conversations with clients last for a while, then outgoing connections will accumulate, eventually reaching the threshold for each relay to decide that that address is being unfair.
--Roger
Hi All,
I think I see the shape of the DDoS mitigations now, and to test my understanding I'm going to try to recap/quote some of the thread as I understand it; plus, I'll voice some of the questions which linger at the back of my head.
Stuff where I am inferring (possibly wrongly) behaviour from context, is in <angle> brackets.
Observation: I had a hard time following whether a bare reference to "client" in any given explanation, referred to a client-ip-address, or to one of the tor-daemons ("clients behind an ip-address"), or to something (eg: curl) actually using tor. So I may have misapprehended / got some of this wrong...
# The three defences are...
1) Connection-Limiting: if a <given client ip-address> makes too many connections to <a solitary given relay> then <that solitary relay> will start to <destroy further circuits that the ip-address creates, irrespective of which tor-daemon behind that address, created them>
Question: as above, I am really uncertain whether the circuits are destroyed because they are attributed to a given ip-address (potential for collateral damage?) or to a specific overzealous tor-daemon/client *behind* a given ip-address.
Question: I am not sure whether this mitigation is per-relay, or whether the status that <given client ip-address> is being "unfair" gets somehow propagated to the rest of the Tor network?
2) Rate-Limiting: if a <given client ip-address> makes connections too quickly to <a solitary given relay>, when ALSO there are lots of existing connections <to that relay>, then <that solitary relay> will start to ignore that <given client ip-address>
Question: <The same two questions as-previous, apply.>
3) DoS Mitigation: if a <non-relay / random ip-address somewhere on the internet> asks <a relay> to <become> a rendezvous point, it gets ignored, which thereby kills Tor2web.
Notes: initially this worried me because I misread it as "if a non-relay CONNECTS to an RP" - which would ban SingleOnions because no guards - but I am pretty sure of my revised interpretation and I am fairly certain of why this would be a good idea; but if I can make this misinterpretation, then others might do, too.
# Re: my situation...
Teor: yep everything is SingleOnion in EOTK by default, not least because testing onionification of video streams with a bank of RaspberryPi on a single DSL connection is laggy enough without a couple of extra hops.
Questions:
The reason I am worried (above) about "notification of bad behaviour is somehow propagated?" is that I am worried that if I trip a trigger, am I going to effectively be banned from the entire Tor network for "a couple of hours", or am I just going to lose <a solitary relay>?
Reading the text further down, Teor writes: "...the defence will trigger. Then both instances will try another relay..." - so I am pretty sure that this is not a "banhammer" that would kick me off Tor entirely; instead it's more of a "choke".
# Re: EOTK...
Any given EOTK "worker" currently gets 2x tor daemons provisioned to it; this is partly to make better use of the compute bandwidth available on a multiprocessor box, and partly to mitigate exant problems of Tor daemons going "stale" - having 2x daemons makes it less likely to lose the entire worker.
Also: I am looking at #24973 "Tor should be more gentle when launching dozens of circuits at once" - which resonates with my worker pool: I am running 6x2 = 12 worker-daemons (and that figure may double, soon) not to mention all the other tor daemons that I am running (eg: TBB, OnionBalance)
Also: per #24992, I have seen quite a lot of "exceeded launch limit with 10 intro points in the last 180 seconds"-type messages, recently; I am not sure why this has been happening because the tor daemons in question have all been workers running a single HS in each. Perhaps it was having problems establishing ANY introduction points at all, because of the DDoS attack, but the failures still counted?
To summarise: all of this "bad" behaviour on my part, explains why I am/was worried about getting blocked from Tor entirely. :-)
# Re: NYT...
I am still working with NYT to confirm their IP setup on AWS; until now, one of the selling points of enterprise onion networking for me has been the capability to keep everything in one/more "NAT enclaves" which - combined with SingleOnion - mitigates inbound-connection attacks like flooding, whilst permitting "horizontal scaling" which is important with a rewriter-proxy like EOTK.
I'm a little bummed that that may no longer be viable / that in order to do Tor at high bandwidth then your worker fleet will need to have separate and visible IP addresses, because DDoS attribution; however, this is an evolving environment so I suppose that we'll adapt.
In the case of AWS this likely means buying "Elastic IPs" for each worker.
# Speculation...
Aaron wrote: "Because the circuit-creation limit is applied at the guard, wouldn't this affect hidden sevices instead of single onion services?"
Teor wrote: "It will only trigger if hundreds of guard-using clients are behind a single IP address."
This exchange makes me think a couple of things:
a) if my EOTK development farm was using NormalOnions not SingleOnions, then I might actually be in a worse situation because I would be more heavily regulated... in fact:
b) I am wondering if <bad people> might leverage the DDoS mitigations to knock non-SingleOnion onionsites off-air? The attack would be to use a wide diversity of client IPs (eg: botnet) to "just enough" connections to the onion site that the connections back, through the guard, to the rendezvous sites, would trip the DDoS code?
Whether I am right would depend in-part on my proper understanding of defences 1 & 2 at the top of this post, plus also how quickly the onionsite would "give up" on the guard which is now ignoring it.
It would be a bit like a DNS amplification attack; but I'm presuming that I'm somehow wrong; Roger's final paragraph speaks directly to Onions-that-use-guards being apparently safer:
"I expect a popular onion service that doesn't use guards and that runs many Tor instances on the same IP address will trigger the defense often: because it doesn't use guards, each new circuit it builds in response to a rendezvous request will pick an entry point at random, and if some of the conversations with clients last for a while, then outgoing connections will accumulate, eventually reaching the threshold for each relay to decide that that address is being unfair."
...however it is speaking to the protections of relays as a whole, rather than the connectivity of a given onion site through a given guard.
-a
Experimental DoS mitigation is in tor master https://lists.torproject.org/pipermail/tor-relays/2018-January/014357.html
Full thread above, some potential affected usage below...
https://lists.torproject.org/pipermail/tor-relays/2018-February/014393.html https://lists.torproject.org/pipermail/tor-relays/2018-February/014396.html
tor-onions@lists.torproject.org
-
A. Johnson -
Alec Muffett -
David Goulet -
grarpamp -
Roger Dingledine -
teor