connlimit: better to use "DROP" or "REJECT --reject-with tcp-reset"?
For relay operators using iptables connlimit to mitigate DoS attacks (or increased load from new clients), is it better for the Tor network to use "DROP" rules, or should we use something like "REJECT --reject-with tcp-reset"?
On Fri, Jan 5, 2018 at 1:44 PM, tor <tor@anondroid.com> wrote:
For relay operators using iptables connlimit to mitigate DoS attacks (or increased load from new clients), is it better for the Tor network to use "DROP" rules, or should we use something like "REJECT --reject-with tcp-reset"?
REJECT is friendlier to clients that are not misbehaving but happen to be caught in the crossfire, and to the Internet as a whole. I personally think DROP should only ever be used as a desperation measure when the DoS load is so high that you can't even afford to send RSTs. zw
On 6 Jan 2018, at 06:05, Zack Weinberg <zackw@cmu.edu> wrote:
On Fri, Jan 5, 2018 at 1:44 PM, tor <tor@anondroid.com> wrote: For relay operators using iptables connlimit to mitigate DoS attacks (or increased load from new clients), is it better for the Tor network to use "DROP" rules, or should we use something like "REJECT --reject-with tcp-reset"?
REJECT is friendlier to clients that are not misbehaving but happen to be caught in the crossfire, and to the Internet as a whole.
I personally think DROP should only ever be used as a desperation measure when the DoS load is so high that you can't even afford to send RSTs.
I've tried various ways of limiting Tor's RAM and CPU. MaxAdvertisedBandwidth was effective, as was limiting Tor's file descriptors and DisableOOSCheck 1. MaxMemInQueues had a minor impact. So I decided to use a firewall to limit connections. If I send RST, a Tor client will immediately try another guard. If I DROP, the Tor client will timeout before trying another guard. For misbehaving clients, I want to DROP. For regular clients, I want to RST. (Or let them connect.) So I want to set a high connection limit, and use DROP. I see up to 700 connections per IP, and I see normal residential IP addresses start at around 80. I don't think that clients with more than 100 connections per IP will get much bandwidth anyway. I used this command: netstat -n | grep ESTABLISHED | awk '{print $5}' | cut -d: -f1 | sort | uniq -c | sort -rn | head All the connections over 100 were at Hetzner, OVH, and Leaseweb. (This command gives /16s for IPv6, but these clients aren't using IPv6.) So I set up this firewall rule: /sbin/iptables -A INPUT -p tcp --syn ! --dport 22 -m connlimit --connlimit-above 100 -j DROP You should replace 22 with the list of ports you use for SSH and other important connections, just in case. And I installed iptables-persistent to save the rules. (It might be Debian-specific). After I set up the firewall, the connections slowly dropped from 700 down to around 100. It only affects new connections, so it might take a while. T -- Tim / teor PGP C855 6CED 5D90 A0C5 29F6 4D43 450C BA7F 968F 094B ricochet:ekmygaiu4rzgsk6n ------------------------------------------------------------------------
Hi, Still having load trouble on your relay? Try dropping rapid connection attempts.
On 9 Jan 2018, at 16:32, teor <teor2345@gmail.com> wrote:
I've tried various ways of limiting Tor's RAM and CPU. MaxAdvertisedBandwidth was effective, as was limiting Tor's file descriptors and DisableOOSCheck 1. MaxMemInQueues had a minor impact.
So I decided to use a firewall to limit connections.
...
So I set up this firewall rule:
/sbin/iptables -A INPUT -p tcp --syn ! --dport 22 -m connlimit --connlimit-above 100 -j DROP
You should replace 22 with the list of ports you use for SSH and other important connections, just in case.
And I installed iptables-persistent to save the rules. (It might be Debian-specific).
After I set up the firewall, the connections slowly dropped from 700 down to around 100. It only affects new connections, so it might take a while.
This worked well, but Tor was still using a lot of CPU with its OOS checks. And it was using 4GB of RAM, which is good, but not sustainable on my machine. Today, I added these firewall rules to drop rapid connection attempts from the same IP address, even if there are under 100 connections: iptables -I INPUT -p tcp --syn ! --dport 22 -m state --state NEW -m recent --set iptables -I INPUT -p tcp --syn ! --dport 22 -m state --state NEW -m recent --update --seconds 60 --hitcount 100 -j DROP They drop connection attempts after there have been 100 attempts in a minute. So if there were 100 clients, that would be 1 connection per client per minute. This reduced Tor's CPU usage and OOS warnings within a few minutes. I'm hoping RAM will go down over time. I made the rules permanent using: iptables-save > /etc/iptables/rules.v4 This might be Debian-specific. T -- Tim / teor PGP C855 6CED 5D90 A0C5 29F6 4D43 450C BA7F 968F 094B ricochet:ekmygaiu4rzgsk6n ------------------------------------------------------------------------
On 01/10/2018 06:39 AM, teor wrote:
iptables -I INPUT -p tcp --syn ! --dport 22 -m state --state NEW -m recent --set iptables -I INPUT -p tcp --syn ! --dport 22 -m state --state NEW -m recent --update --seconds 60 --hitcount 100 -j DROP
What's about the following approach; IPT="/sbin/iptables" $IPT -P INPUT DROP $IPT -P OUTPUT ACCEPT $IPT -P FORWARD DROP # trust already established connections # $IPT -A INPUT --match conntrack --ctstate ESTABLISHED -j ACCEPT $IPT -A INPUT --match conntrack --ctstate RELATED -j ACCEPT $IPT -A INPUT --match conntrack --ctstate INVALID -j DROP # Tor # for p in 443 80 do $IPT -A INPUT -p tcp --syn --destination-port $p --match connlimit --connlimit-above 2 --connlimit-mask 32 -j DROP $IPT -A INPUT -p tcp --destination-port $p -j ACCEPT done Those rules should not prevent clients behind a NAT from accessing the relay as long as the clients do not come in in parallel. Objections ? -- Toralf PGP C4EACDDE 0076E94E
On 11 Jan 2018, at 08:10, Toralf Förster <toralf.foerster@gmx.de> wrote:
On 01/10/2018 06:39 AM, teor wrote:
iptables -I INPUT -p tcp --syn ! --dport 22 -m state --state NEW -m recent --set iptables -I INPUT -p tcp --syn ! --dport 22 -m state --state NEW -m recent --update --seconds 60 --hitcount 100 -j DROP
What's about the following approach; IPT="/sbin/iptables"
$IPT -P INPUT DROP $IPT -P OUTPUT ACCEPT $IPT -P FORWARD DROP
# trust already established connections # $IPT -A INPUT --match conntrack --ctstate ESTABLISHED -j ACCEPT $IPT -A INPUT --match conntrack --ctstate RELATED -j ACCEPT $IPT -A INPUT --match conntrack --ctstate INVALID -j DROP
# Tor # for p in 443 80 do $IPT -A INPUT -p tcp --syn --destination-port $p --match connlimit --connlimit-above 2 --connlimit-mask 32 -j DROP $IPT -A INPUT -p tcp --destination-port $p -j ACCEPT done
Those rules should not prevent clients behind a NAT from accessing the relay as long as the clients do not come in in parallel.
As far as I can tell, this single rule has the same effect: # Tor # for p in 443 80 do $IPT -A INPUT -p tcp --syn --destination-port $p --match connlimit --connlimit-above 2 --connlimit-mask 32 -j DROP done
Objections ?
Bridges and hidden services often come in parallel. As do large client deployments in areas without much IPv4 allocation. We allow 2 relays per IPv4 address, and each relay makes 1-2 connections to each other relay. (Or more, if the connections start failing. This is a bug we want to fix.) So if you're going to do this, please set a much higher limit than 2. I would suggest at least 4, but 10 or more is better. You might be able to set it higher if you put a limit on repeated connection attempts. T -- Tim Wilson-Brown (teor) teor2345 at gmail dot com PGP C855 6CED 5D90 A0C5 29F6 4D43 450C BA7F 968F 094B ricochet:ekmygaiu4rzgsk6n ------------------------------------------------------------------------
On 01/11/2018 02:10 AM, teor wrote:
We allow 2 relays per IPv4 address, and each relay makes 1-2 connections to each other relay. (Or more, if the connections start failing. This is a bug we want to fix.)
So if you're going to do this, please set a much higher limit than 2. I would suggest at least 4, but 10 or more is better.
You might be able to set it higher if you put a limit on repeated connection attempts. Right - good hint !
-- Toralf PGP C4EACDDE 0076E94E
On 12 Jan 2018, at 08:46, Toralf Förster <toralf.foerster@gmx.de> wrote:
On 01/11/2018 02:10 AM, teor wrote: As far as I can tell, this single rule has the same effect: Even if " -P INPUT DROP" is et ?
I think that applying the single rule to the default config, has the same effect as applying all the rules you specified, including " -P INPUT DROP". But I'm not sure, I don't have much iptables experience. T
On 01/11/2018 02:10 AM, teor wrote:
So if you're going to do this, please set a much higher limit than 2. I would suggest at least 4, but 10 or more is better.
You might be able to set it higher if you put a limit on repeated connection attempts.
The simple approach (allowing 8 syn requests from an address at ORport and at DirPort respectively) worked flawlessley for a while - just few dozen/hundreds DROPs per hour. Since yesterday however I get > 100K DROPs per hour. Could a side effect of that traffic be that I lost the Exit flag ? -- Toralf PGP C4EACDDE 0076E94E
On 21 Jan 2018, at 22:34, Toralf Förster <toralf.foerster@gmx.de> wrote:
On 01/11/2018 02:10 AM, teor wrote:
So if you're going to do this, please set a much higher limit than 2. I would suggest at least 4, but 10 or more is better.
You might be able to set it higher if you put a limit on repeated connection attempts.
The simple approach (allowing 8 syn requests from an address at ORport and at DirPort respectively) worked flawlessley for a while - just few dozen/hundreds DROPs per hour. Since yesterday however I get > 100K DROPs per hour.
Your relays are now handling extra load, because they lost the exit flag and became guards.
Could a side effect of that traffic be that I lost the Exit flag ?
No, the exit flag is determined by your exit policy, and the Tor version running on the majority of directory authorities. Recently, a majority of authorities upgraded to 0.3.2 or later. They require ports 80 and 443 for the Exit flag: https://trac.torproject.org/projects/tor/ticket/23637 Your exit policy does not include port 80, so your relays are not useful for clients to build general-purpose exit circuits. Please allow port 80 to regain the Exit flag. (The majority of Tor traffic is web traffic. Some of that traffic is unencrypted. This is bad, but enforcing port 443 on Tor clients would sacrifice usability and anonymity for security.) T -- Tim / teor PGP C855 6CED 5D90 A0C5 29F6 4D43 450C BA7F 968F 094B ricochet:ekmygaiu4rzgsk6n ------------------------------------------------------------------------
On 10 Jan 2018, at 16:39, teor <teor2345@gmail.com> wrote:
Hi,
Still having load trouble on your relay? Try dropping rapid connection attempts.
On 9 Jan 2018, at 16:32, teor <teor2345@gmail.com> wrote:
I've tried various ways of limiting Tor's RAM and CPU. MaxAdvertisedBandwidth was effective, as was limiting Tor's file descriptors and DisableOOSCheck 1. MaxMemInQueues had a minor impact.
So I decided to use a firewall to limit connections.
...
So I set up this firewall rule:
/sbin/iptables -A INPUT -p tcp --syn ! --dport 22 -m connlimit --connlimit-above 100 -j DROP
You should replace 22 with the list of ports you use for SSH and other important connections, just in case.
...
This worked well, but Tor was still using a lot of CPU with its OOS checks. And it was using 4GB of RAM, which is good, but not sustainable on my machine.
Today, I added these firewall rules to drop rapid connection attempts from the same IP address, even if there are under 100 connections:
iptables -I INPUT -p tcp --syn ! --dport 22 -m state --state NEW -m recent --set iptables -I INPUT -p tcp --syn ! --dport 22 -m state --state NEW -m recent --update --seconds 60 --hitcount 100 -j DROP
They drop connection attempts after there have been 100 attempts in a minute. So if there were 100 clients, that would be 1 connection per client per minute.
This reduced Tor's CPU usage and OOS warnings within a few minutes. I'm hoping RAM will go down over time.
I made the rules permanent using:
iptables-save > /etc/iptables/rules.v4
This might be Debian-specific.
I tried a few configs over the past week. Now I have: * MaxMemInQueues 2 GB * 15000 file descriptors per tor instance * DisableOOSCheck 0 * A limit of 20 established connections per IP * A limit of 6 connection attempts per IP per minute I left this over the weekend, and my relays are stable, and using: * 3 GB - 6 GB RAM * 5000 - 11000 file descriptors * 50 - 120% CPU They are also not logging too many OOS warnings or other warnings, apart from the normal "assign to cpuworker failed" and "attempt to establish rendezvous". Thanks to everyone for your suggestions in this and other threads. We are also working on a few different ways to limit the load in Tor. T -- Tim / teor PGP C855 6CED 5D90 A0C5 29F6 4D43 450C BA7F 968F 094B ricochet:ekmygaiu4rzgsk6n ------------------------------------------------------------------------
participants (4)
-
teor -
tor -
Toralf Förster -
Zack Weinberg