I run the relay 8F6A78B1EA917F2BF221E87D14361C050A70CCC3
Like most relays mine has been targeted by the DoS attack. Hundreds of VPS IPs creating millions of IP connections. This I mitigated with rules in my firewall. Looking at the firewall counters it looks like that attack has now stopped.
However the relay is still overloaded from lots of circuit creations. Normally my little relay reports around 100K circuits open in the log file but since the overloading started it's closer to 1M circuits open. All these circuit creations put a strain on the CPU, sometime pegging it at 100% (4 core i5-4670K CPU @ 3.40GHz). Worse, it seems to eat memory. Normally the tor process uses about 3GB (out of 8GB) but I have seen it quickly shoot up to using all memory and all swap. I assume this is because of the circuit creation DoS (it's new behavior) and what causes the oom killer to kill the tor process at least once a day or so.
So, how do I mitigate the circuit creation DoS? My immediate thought is to identify if there are a few IPs responsible for the majority and add those to my firewall naughty list.
Is there a way to query the tor process about number of open circuits mapped to IP addresses?
On 2022-10-19 17:10, Chris wrote:
You may want to check these links:
https://gitlab.torproject.org/tpo/community/support/-/issues/40093
Thank you for the reply and the links. From what I can understand those links concern "connections". I believe my firewall rules handles that fine (they're based on Toralf's example).
My concern is about circuits. As I understand it one connection can create many circuits. If the attacker keeps the connections down to avoid being blacklisted they can create lots of circuits. And one circuit created affects 3 relays.
So what I'm looking for is a way to get the IP of big circuit creators. I understand that many circuits will come from other relays but on my guard relay I assume the attacker also connect directly. If I can blacklist non-relays that create too many circuits I can help my relay and those downstream.
Excerpts from Logforme's message of October 19, 2022 8:47 am:
I run the relay 8F6A78B1EA917F2BF221E87D14361C050A70CCC3
Like most relays mine has been targeted by the DoS attack. Hundreds of VPS IPs creating millions of IP connections. This I mitigated with rules in my firewall. Looking at the firewall counters it looks like that attack has now stopped.
However the relay is still overloaded from lots of circuit creations. Normally my little relay reports around 100K circuits open in the log file but since the overloading started it's closer to 1M circuits open. All these circuit creations put a strain on the CPU, sometime pegging it at 100% (4 core i5-4670K CPU @ 3.40GHz). Worse, it seems to eat memory. Normally the tor process uses about 3GB (out of 8GB) but I have seen it quickly shoot up to using all memory and all swap. I assume this is because of the circuit creation DoS (it's new behavior) and what causes the oom killer to kill the tor process at least once a day or so.
So, how do I mitigate the circuit creation DoS? My immediate thought is to identify if there are a few IPs responsible for the majority and add those to my firewall naughty list.
Is there a way to query the tor process about number of open circuits mapped to IP addresses? _______________________________________________ tor-relays mailing list tor-relays@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays
this is a good idea, which is why someone's already had it (me, possibly also other people). there are a few problems with this plan:
1. tor doesn't expose this information in log (without SafeLogging 0, highly discouraged and possibly illegal except for testing) or control port, because it could potentially be used to deanonymize clients, made worse because...
2. it might not possible to know in general whether the clients with many connections are "abusive" clients or "abused" onion services. additionally, you definitely need to filter out relays, which will obviously have lots of open circuits. there are a couple of decent ways to do that, but it's non-trivial.
for these reasons, I haven't aggressively pursued this plan. I have some more ideas based on intra-family correlation, but they also have similar problems as well as more implementation problems. in the long term, our best hope is the PoW scheme (note: not cryptocurrency) being somewhat quietly worked on.
cheers, Alex.
On 2022-10-20 03:35, Alex Xu (Hello71) via tor-relays wrote:
for these reasons, I haven't aggressively pursued this plan. I have some more ideas based on intra-family correlation, but they also have similar problems as well as more implementation problems. in the long term, our best hope is the PoW scheme (note: not cryptocurrency) being somewhat quietly worked on.
Good points and PoW will be great when available but I'm looking for something I as an operator can do to mitigate the circuit creation DoS until then. Oh well
tor-relays@lists.torproject.org
-
Alex Xu (Hello71) -
Chris -
Logforme