Circuit Creation Madness: Anyone else still experiencing (extremely) excessive clients / (possibly) modified relays creating millions upon millions of circuits?
@tor-relays:
Sorry for being quite noisy recently but I really need to know how many people are suffering from the same madness I am encountering right now.
Quick excerpt from the log:
... Mar 22 09:48:10 <hostname_redacted> tor[pid_redacted]: Mar 22 09:48:10.000 [warn] Your computer is too slow to handle this many circuit creation requests! Please consider using the MaxAdvertisedBandwidth config option or choosing a more restricted exit policy. [12420 similar message(s) suppressed in last 120 seconds] Mar 22 09:49:10 <hostname_redacted> tor[pid_redacted]: Mar 22 09:49:10.000 [warn] Your computer is too slow to handle this many circuit creation requests! Please consider using the MaxAdvertisedBandwidth config option or choosing a more restricted exit policy. [31764 similar message(s) suppressed in last 60 seconds] Mar 22 09:50:10 <hostname_redacted> tor[pid_redacted]: Mar 22 09:50:10.000 [warn] Your computer is too slow to handle this many circuit creation requests! Please consider using the MaxAdvertisedBandwidth config option or choosing a more restricted exit policy. [104748 similar message(s) suppressed in last 60 seconds] Mar 22 09:51:10 <hostname_redacted> tor[pid_redacted]: Mar 22 09:51:10.000 [warn] Your computer is too slow to handle this many circuit creation requests! Please consider using the MaxAdvertisedBandwidth config option or choosing a more restricted exit policy. [364165 similar message(s) suppressed in last 60 seconds] Mar 22 09:52:10 <hostname_redacted> tor[pid_redacted]: Mar 22 09:52:10.000 [warn] Your computer is too slow to handle this many circuit creation requests! Please consider using the MaxAdvertisedBandwidth config option or choosing a more restricted exit policy. [509474 similar message(s) suppressed in last 60 seconds] Mar 22 09:53:10 <hostname_redacted> tor[pid_redacted]: Mar 22 09:53:10.000 [warn] Your computer is too slow to handle this many circuit creation requests! Please consider using the MaxAdvertisedBandwidth config option or choosing a more restricted exit policy. [241332 similar message(s) suppressed in last 60 seconds] ...
This then goes on for a while, stopping at a few million suppressed messages / circuit creation attempts.
Sorry, but 1 million circuit creation requests in just 5 minutes, there is no way that this is legitimate behavior we are seeing - this is also what was previously used to get my relay oom-killed but that I have fixed so the legitimate clients hopefully don't suffer too much anymore.
If any other relay operators are encountering the same log entries or behavior, please don't hesitate to reply.
Added tor-dev@lists.torproject.org as a CC as they might want to know about this.
@tor-dev:
I suspect some kind of denial-of-service attack against onion services or a more targeted attack against singular relays for guard discovery / traffic confirmation attacks.
Might be smart to add some code which, if this scenario is triggered, lists offenders by hashes of their signing keys (if relay), or IP addresses (if client).
There doesn't seem to be a defense against this, and the new connect() rate-limit added through ticket 40253 also won't handle this as the connection is already ACK'd and established, and a malicious relay with custom source code could do whatever it was programmed to do anyway.
- William
On Mon, Mar 22, 2021 at 07:54:43PM +0000, William Kane wrote:
Sorry for being quite noisy recently but I really need to know how many people are suffering from the same madness I am encountering right now.
Quick excerpt from the log:
Mar 22 09:48:10 <hostname_redacted> tor[pid_redacted]: Mar 22 09:48:10.000 [warn] Your computer is too slow to handle this many circuit creation requests! Please consider using the MaxAdvertisedBandwidth config option or choosing a more restricted exit policy. [12420 similar message(s) suppressed in last 120 seconds]
Yes, it could help to hear if many people are experiencing these log messages or just a few.
There are several known situations where the log messages could happen to a small subset of the relays at any given time. For example, if somebody is trying to flood a particular onion service, then the six or eight HSDirs for that onion address for that day could see a lot of overload (which would last for as much as that day), and also the introduction points listed in the descriptors could see a lot of overload (which would last a lot less than a day probably).
Might be smart to add some code which, if this scenario is triggered, lists offenders by hashes of their signing keys (if relay), or IP addresses (if client).
For the variants of the overloads that we've seen so far, they are done via Tor, i.e. your relay doesn't actually know who is starting the circuits, so those logs would be at best useless. (We built an anonymity system, and now it keeps people anonymous. We can't be *too* unhappy here. :)
I think the long term answer for these attacks are the options outlined by George in this blog post: https://blog.torproject.org/stop-the-onion-denial
I'm especially interested in the proof-of-work variant, which doesn't need an interface where the human does stuff, doesn't need to be hooked together with a global ecash system that everybody wants a piece of, etc: https://gitweb.torproject.org/torspec.git/tree/proposals/327-pow-over-intro....
But as they say, more work remains.
--Roger
Roger:
For the variants of the overloads that we've seen so far, they are done via Tor, i.e. your relay doesn't actually know who is starting the circuits, so those logs would be at best useless. (We built an anonymity system, and now it keeps people anonymous. We can't be *too* unhappy here. :)
I'm fully aware of that, but wouldn't it take a "malicious" guard in order to forward all these circuit creation requests anyway?
Last time I checked, the authorities were configured to vote on / publish reasonable thresholds of consecutive connections and circuit creation requests for relays to adapt (however most of the mitigation only takes place on guards, which makes sense), so even if a client is doing all this and we obviously can't get their IP address, the guard would have to be configured in a way that allows this scenario to happen in the first place, in my opinion making them bad relays - right now my relay only takes place as a middle in a circuit, so figuring out the guard is possible (not considering the onion service scenario right now).
- William
On 23/03/2021, Roger Dingledine arma@torproject.org wrote:
On Mon, Mar 22, 2021 at 07:54:43PM +0000, William Kane wrote:
Sorry for being quite noisy recently but I really need to know how many people are suffering from the same madness I am encountering right now.
Quick excerpt from the log:
Mar 22 09:48:10 <hostname_redacted> tor[pid_redacted]: Mar 22 09:48:10.000 [warn] Your computer is too slow to handle this many circuit creation requests! Please consider using the MaxAdvertisedBandwidth config option or choosing a more restricted exit policy. [12420 similar message(s) suppressed in last 120 seconds]
Yes, it could help to hear if many people are experiencing these log messages or just a few.
There are several known situations where the log messages could happen to a small subset of the relays at any given time. For example, if somebody is trying to flood a particular onion service, then the six or eight HSDirs for that onion address for that day could see a lot of overload (which would last for as much as that day), and also the introduction points listed in the descriptors could see a lot of overload (which would last a lot less than a day probably).
Might be smart to add some code which, if this scenario is triggered, lists offenders by hashes of their signing keys (if relay), or IP addresses (if client).
For the variants of the overloads that we've seen so far, they are done via Tor, i.e. your relay doesn't actually know who is starting the circuits, so those logs would be at best useless. (We built an anonymity system, and now it keeps people anonymous. We can't be *too* unhappy here. :)
I think the long term answer for these attacks are the options outlined by George in this blog post: https://blog.torproject.org/stop-the-onion-denial
I'm especially interested in the proof-of-work variant, which doesn't need an interface where the human does stuff, doesn't need to be hooked together with a global ecash system that everybody wants a piece of, etc: https://gitweb.torproject.org/torspec.git/tree/proposals/327-pow-over-intro....
But as they say, more work remains.
--Roger
tor-relays mailing list tor-relays@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays
tor-relays@lists.torproject.org
-
Roger Dingledine -
William Kane