Overload (dropped ntor) due to DDoS??
Hi All I have a non exit relay running on a root server (4 AMD Epyc cores, 8 GB RAM, 2.5 GBit/s Ethernet) I have limited tor to numcpus 2, relaybandwidthburst 15 MB, hardwareaccel 1, maxadvertisedbandwidth 10 MB, maxmeminqueues 3GB Usually it takes less than 1 CPU core, and like 1 GB of RAM. But recently my relay is foten shown as obverloaded. I have these LOG entries: Tor[814]: General overload -> Ntor dropped (290376) fraction 5.3451% is above threshold of 0.5000% Is this due to DDoS attacks or a misconfigration on my side? Is there something that I can do to aleviate this issue? CU, Ricsi
Richard Menedetter wrote:
Hi All
I have a non exit relay running on a root server (4 AMD Epyc cores, 8 GB RAM, 2.5 GBit/s Ethernet) I have limited tor to numcpus 2, relaybandwidthburst 15 MB, hardwareaccel 1, maxadvertisedbandwidth 10 MB, maxmeminqueues 3GB
Thanks for running a relay! didn't you also use RelayBandwidthRate along with RelayBandwidthBurst ?
Usually it takes less than 1 CPU core, and like 1 GB of RAM. But recently my relay is foten shown as obverloaded. I have these LOG entries: Tor[814]: General overload -> Ntor dropped (290376) fraction 5.3451% is above threshold of 0.5000%
You are not the only one, it's an ongoing DoS attack on the network, targeting onion services.
Is this due to DDoS attacks or a misconfigration on my side?
Besides the question above about RelayBandwidthRate I don't see anything wrong.
Is there something that I can do to aleviate this issue?
Nope, there is nothing you can do, unfortunately. Tor has some defenses against DoS and will blacklist / mark the abusing addresses, etc. as much as it can. But as you know DoS is a never ending battle, usually won by having "larger pipe", and it's something hard to tickle in an environment where anonymity is the grounding law. What you can do is maintain your relay up and running in good shape with the latest version of Tor until this "attack" gets through. As I said, I guess most of relays are getting this at present times. The DoS "attack" is not targeted at your relay, what you are seeing is just a side effect of someone creating large amounts of circuits (heavy usage of Tor) which is reflected network-wide anyways.
CU, Ricsi
Hi Thanx for the explanation. I have 0.4.7.8 and try to run the latest version. So it seems the overload is entirely due to the DDoS and not my config. I have removed the maxadvertised bandwidth limit, it will now again send the measured value instead of being limited to 10 MB. I have these limits: RelayBandwidthRate 15 MB RelayBandwidthBurst 30 MB BandwidthRate 50 MB NumCPUs 2 MaxMemInQueues 3072 MB CU, Ricsi
Gesendet: Freitag, 05. August 2022 um 01:11 Uhr Von: "s7r" <s7r@sky-ip.org> An: tor-relays@lists.torproject.org Betreff: Re: [tor-relays] Overload (dropped ntor) due to DDoS??
Richard Menedetter wrote:
Hi All
I have a non exit relay running on a root server (4 AMD Epyc cores, 8 GB RAM, 2.5 GBit/s Ethernet) I have limited tor to numcpus 2, relaybandwidthburst 15 MB, hardwareaccel 1, maxadvertisedbandwidth 10 MB, maxmeminqueues 3GB
Thanks for running a relay!
didn't you also use RelayBandwidthRate along with RelayBandwidthBurst ?
Usually it takes less than 1 CPU core, and like 1 GB of RAM. But recently my relay is foten shown as obverloaded. I have these LOG entries: Tor[814]: General overload -> Ntor dropped (290376) fraction 5.3451% is above threshold of 0.5000%
You are not the only one, it's an ongoing DoS attack on the network, targeting onion services.
Is this due to DDoS attacks or a misconfigration on my side?
Besides the question above about RelayBandwidthRate I don't see anything wrong.
Is there something that I can do to aleviate this issue?
Nope, there is nothing you can do, unfortunately. Tor has some defenses against DoS and will blacklist / mark the abusing addresses, etc. as much as it can. But as you know DoS is a never ending battle, usually won by having "larger pipe", and it's something hard to tickle in an environment where anonymity is the grounding law.
What you can do is maintain your relay up and running in good shape with the latest version of Tor until this "attack" gets through. As I said, I guess most of relays are getting this at present times. The DoS "attack" is not targeted at your relay, what you are seeing is just a side effect of someone creating large amounts of circuits (heavy usage of Tor) which is reflected network-wide anyways.
CU, Ricsi
_______________________________________________ tor-relays mailing list tor-relays@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays
On Friday, August 5, 2022 1:11:27 AM CEST s7r wrote:
Richard Menedetter wrote:
I have a non exit relay running on a root server (4 AMD Epyc cores, 8 GB RAM, 2.5 GBit/s Ethernet) I have limited tor to numcpus 2, Why? Do you have other services on the server? Otherwise, omit num CPU. Let the tor daemon use all CPU's for crypto stuff.
relaybandwidthburst 15 MB, hardwareaccel 1, maxadvertisedbandwidth 10 MB, maxmeminqueues 3GB
Thanks for running a relay!
didn't you also use RelayBandwidthRate along with RelayBandwidthBurst ?
Usually it takes less than 1 CPU core, and like 1 GB of RAM. But recently my relay is foten shown as obverloaded. I have these LOG entries: Tor[814]: General overload -> Ntor dropped (290376) fraction 5.3451% is above threshold of 0.5000%
You are not the only one, it's an ongoing DoS attack on the network, targeting onion services.
Is this due to DDoS attacks or a misconfigration on my side?
Besides the question above about RelayBandwidthRate I don't see anything wrong.
Is there something that I can do to aleviate this issue?
Nope, there is nothing you can do, unfortunately. Tor has some defenses against DoS and will blacklist / mark the abusing addresses, etc. as much as it can. But as you know DoS is a never ending battle, usually won by having "larger pipe", and it's something hard to tickle in an environment where anonymity is the grounding law.
What you can do is maintain your relay up and running in good shape with the latest version of Tor until this "attack" gets through. As I said, I guess most of relays are getting this at present times. The DoS "attack" is not targeted at your relay, what you are seeing is just a side effect of someone creating large amounts of circuits (heavy usage of Tor) which is reflected network-wide anyways. Sometimes 100.000-1.000.000 connections from one IP! I block the worst with 2 nftables egress rules.
toralf has developed some smarter ddos scripts: https://github.com/toralf/torutils -- ╰_╯ Ciao Marco! Debian GNU/Linux It's free software and it gives you freedom!
Hi
I have a non exit relay running on a root server (4 AMD Epyc cores, 8 GB RAM, 2.5 GBit/s Ethernet) I have limited tor to numcpus 2, Why? Do you have other services on the server? Otherwise, omit num CPU. Let the tor daemon use all CPU's for crypto stuff.
Yes, there are other services running on that host, and that are the ressources I am willing to contribute to tor. 2 EPYC cores can deal easily with the crypto, especially as they are using HW crypto acceleration. CU, Ricsi
lists@for-privacy.net wrote:
On Friday, August 5, 2022 1:11:27 AM CEST s7r wrote:
Richard Menedetter wrote:
I have a non exit relay running on a root server (4 AMD Epyc cores, 8 GB RAM, 2.5 GBit/s Ethernet) I have limited tor to numcpus 2, Why? Do you have other services on the server? Otherwise, omit num CPU. Let the tor daemon use all CPU's for crypto stuff.
The OP has a EPYC chip with four cores and presumably eight CPU threads, so he should specify at least "NumCPUs 8" as a bare minimum. "NumCPUs 16", "NumCPUs 24", or "NumCPUs 32" would be better. These onion skin decryption threads use hardly any actual CPU time throughout the course of a busy day, but having many extras available for work can mean drastically lower times for a decryption task to remain in the work queue and therefore a much greater chance of getting through a high connection count period without dropping any due to the high load. I've never seen an overload message like the one the OP showed, and my system is ridiculously slower than the OP's EPYC. When I last started tor 41 days ago, it was started with "NumCPUs 12", up from 10 in the previous iteration. I've since changed torrc to "NumCPUs 16" because of the connection and circuit counts during the recent troubles, so the next time I start it it will be with four onion skin decryptors per core. This is on an ancient QX9650 (Core 2 Extreme quad-core, so only one CPU thread per core) bumped up to 3.3 GHz. I hope that will be sufficient to get it through such attacks by making my Internet connection speed, rather than my tor config, be the weak spot. During the recent events I've seen tor's main thread take up 15% to 16% of CPU time on one core, while each of the worker threads consumed far less than 1% apiece. FWIW, mprime runs nice'd to 20 on all four cores, as well. Also, I normally renice tor to -6 about 15 seconds after starting it, to keep it mostly prioritized higher than anything I might be doing interactively and also other stuff running all the time (e.g., top(1), privoxy(1)).
relaybandwidthburst 15 MB, hardwareaccel 1, maxadvertisedbandwidth 10 MB, maxmeminqueues 3GB
Thanks for running a relay!
didn't you also use RelayBandwidthRate along with RelayBandwidthBurst ?
Usually it takes less than 1 CPU core, and like 1 GB of RAM.
During the couple of hours of the attack the other day my tor instance never seemed to use more than ~2700 MB.
But recently my relay is foten shown as obverloaded. I have these LOG entries: Tor[814]: General overload -> Ntor dropped (290376) fraction 5.3451% is above threshold of 0.5000%
You are not the only one, it's an ongoing DoS attack on the network, targeting onion services.
Is this due to DDoS attacks or a misconfigration on my side?
Besides the question above about RelayBandwidthRate I don't see anything wrong.
Is there something that I can do to aleviate this issue?
Again, I suggest increasing NumCPUs by several times to try to avoid the overload message.
Nope, there is nothing you can do, unfortunately. Tor has some defenses against DoS and will blacklist / mark the abusing addresses, etc. as much as it can. But as you know DoS is a never ending battle, usually won by having "larger pipe", and it's something hard to tickle in an environment where anonymity is the grounding law.
What you can do is maintain your relay up and running in good shape with the latest version of Tor until this "attack" gets through. As I said, I guess most of relays are getting this at present times. The DoS "attack" is not targeted at your relay, what you are seeing is just a side effect of someone creating large amounts of circuits (heavy usage of Tor) which is reflected network-wide anyways.
Sometimes 100.000-1.000.000 connections from one IP! I block the worst with 2 nftables egress rules.
toralf has developed some smarter ddos scripts: https://github.com/toralf/torutils
Scott Bennett, Comm. ASMELG, CFIAG ********************************************************************** * Internet: bennett at sdf.org *xor* bennett at freeshell.org * *--------------------------------------------------------------------* * "A well regulated and disciplined militia, is at all times a good * * objection to the introduction of that bane of all free governments * * -- a standing army." * * -- Gov. John Hancock, New York Journal, 28 January 1790 * **********************************************************************
participants (4)
-
lists＠for-privacy.net -
Richard Menedetter -
s7r -
Scott Bennett