Hello Everyone,
Before I make changes to [my scripts](https://github.com/Enkidu-6/tor-ddos), I need to understand a few things and any help is much appreciated.
- First, Does an Exit relay with zero Guard probability and zero middle relay probability need to initiate circuits with a Guard or middle relay and assuming it fails, would that affect my relay, relaying traffic to that Exit node as a part of my own circuit? To clarify the question, I have 2 Established connections to two Or Ports of an exit relay, the relay has no connections to me, Does that break anything?
- I have a few Exit relays as permanent residents in my block list, not because I want them to be there but because, no matter how many times I remove them, day or night, they'll be back in seconds for making too many concurrent attempts. I'm assuming this is due to the fact that Exits are being used to attack other relays but I may be wrong which is why I need someone to clarify this for me and hence another reason for question one.
- Each relay has Established connections to many other relays and if they're guard they will also have many connections to regular users and their Tor browsers until they have enough traffic to reach their MaxAdvertisedBandwidth. Obviously we don't Establish connections to all 6300 relays out there. So if we do not allow each IP more than two connections and they need 4, They'll have two from us and they'll move on to another relay and get the other two and get the job done and we will reach our Max Bandwidth anyway by accepting traffic from other relays. Diversity of relays as opposed to concentration of some relays. Am I correct in my assumption that this will have little to no effect on the health of the Tor network as a whole?
Thanks for reading, I'm not going to make this longer than it is already.
Hi,
thanks for raising these questions and trying to understand before deploying/changes to filters.
A good understanding of how tor relays and connections work is important when trying to defend against overload attacks, without breaking functionality with packet filters that cause false positive blocks, especially when such a long standing limit like the relays per IPv4 address limit is changed.
- I have a few Exit relays as permanent residents in my block list, not
because I want them to be there but because, no matter how many times I remove them, day or night, they'll be back in seconds for making too many concurrent attempts.
As a relay operator you should allow all other relays to connect to your relay's ORPort no matter what flags or onionoo guard/middle/exit probability they might have. Place known relay IPs on an exception list so your filters don't block them and update that list at least every hour.
If you have a problem on your relay's ORPort with a source IP that is also used by an exit relay please try to contact the operator by looking at their contactinfo, if they don't have a contactinfo, join the 'require a usable contactinfo' lobby ;) for this very reason and maybe ask on this list if they can drop you an email.
You can also not be sure whether it is an actual authenticated relay to relay connection or a client to relay connection just by looking at the source IP. Some upcoming MetricsPort enhancement might help you there in the future not per source IP but as a general overview for your relay's ORPort connections. An exit can share it's source IP with tor clients for example behind NAT, but I don't think that is common and it is also discouraged. Exits should have dedicated IPs that is not shared for unrelated things.
Blocking relay to relay communication should not be done as it breaks a core assumption of the tor network (every relay can talk to all other relays ORPorts). Upcoming tooling that detects broken relay to relay links might also detect and flag your relay if your filters break relay to relay communication.
As an option of last resort - after verifying it is an authenticated relay to relay connection that is causing you trouble and not some tor client using the same source IP you might contact the bad-relays list. That is still better than blocking another relay from reaching your relay's ORPort.
I've seen other problematic filter practices for relay to relay connections and I'll write up some recommendations in a separate email so it doesn't get lost in this lengthy email.
I'm assuming this is due to the fact that Exits are being used to attack other relays
In a vanilla current tor version it is not possible to use a tor client to connect via an exit to another relay's ORPort so this is very unlikely. The background here is that tor does not allow such connections to prevent an attacker from reentering the tor network via an exit relay.
You can test that by opening this URL in tor browser, you will get a "Unable to connect" very fast:
https://185.220.102.242/ because it is this ORPort: https://metrics.torproject.org/rs.html#details/0A2366980A2842D770EF8E136A7DA...
the answer is very fast and not a slow timeout because a tor client can predict that is inaccessible before even trying to create a stream because exits will not allow such connections to relay ORPorts.
I have 2 Established connections to two Or Ports of an exit relay
I was not able to follow you there. I was unable to find any exit relay that has more than one ORPort on IPv4 (I identify relays by fingerprint not by IP address). Maybe you can list the specific exit relay fingerprint and timestamp so I can cross check for bugs in my tooling/onionoo?
- Each relay has Established connections to many other relays and if
they're guard they will also have many connections to regular users and their Tor browsers until they have enough traffic to reach their MaxAdvertisedBandwidth. Obviously we don't Establish connections to all 6300 relays out there.
It is best to actually expect that.
So if we do not allow each IP more than two connections and they need 4, They'll have two from us and they'll move on to another relay and get the other two and get the job done and we will reach our Max Bandwidth anyway by accepting traffic from other relays. Diversity of relays as opposed to concentration of some relays. Am I correct in my assumption that this will have little to no effect on the health of the Tor network as a whole?
I think there is a misunderstanding there. One important point to take away: Relays do not chose the next hop in a circuit, tor clients do. So if my tor clients picks a circuit with these relays: client -> A -> B -> C -> example.com and B can not reach C, B can not say "oh I can not reach C I'll pick D instead for you", a relay has no say in that. I hope that helps to reinforce the importance of ubiquitous reachability between relays. Relays have to obey a tor client's orders and a tor client expects that all relays can talk to each other. A few years ago David Stainton published some actual scan results on tor-dev that showed that this expectation might not be true in reality but close enough.
I fear that overly aggressive relay to relay filter actually help the adversaries more than the network and would advise against filtering practices between relays. The first step should always be on a social level: Try to reach the operator if you feel they attack your relay and NOT iptables DROP without notice.
kind regards, nusenu
@nusenu
Thank you very much for taking the time to help me understand things better. I can use all the help I can get.
> You can also not be sure whether it is an actual authenticated relay to relay > connection or a client to relay connection just by looking at the source IP.
> In a vanilla current tor version it is not possible to use a tor client > to connect via an exit to another relay's ORPort so this is very unlikely.
> An exit can share it's source IP with tor clients for example behind NAT
Sorry I'm confused here. Those statements seem to contradict each other or there's something here I misunderstood. Even if that happens, why would a client connect directly to an Exit and get the Exit to connect to another relay or Guard using the Exit's IP address? Wouldn't the same protection you mentioned kick in and stop a client?
> I was not able to follow you there. I was unable to find any exit relay that > has more than one ORPort on IPv4 (I identify relays by fingerprint not by IP address).
Sorry, I should have been more careful in using my words. In the world of firewalls and iptables we identify connections by IPAddress:port pairs hence my inaccurate wording. A relay has only one ORPort but an IP address can have 4.
> I think there is a misunderstanding there. One important point to take away: > Relays do not chose the next hop in a circuit, tor clients do. > So if my tor clients picks a circuit with these relays: > client -> A -> B -> C -> example.com > and B can not reach C, B can not say "oh I can not reach C I'll pick D instead for you",
I understand that and I think it might be my fault again for not being clear. It doesn't matter who makes that choice. What matters is that the choice is being made. My understanding is that the moment you open your Tor Browser, it starts establishing connections and building multiple circuits and keeps them in a pool.
In the old days of vidalia you could actually see that in real time. Many established circuits were just sitting there waiting. Each time you try to browse a page, Tor doesn't say, oh let's find a relay. It picks a circuit from a pool of already Established ones. If Tor tries to build a circuit with a relay and it's unable to, it will try another one that works and keeps it in the pool for when it's needed. Again, I may be wrong. Feel free to tell me if I am.
If that wasn't the case, any time I rebooted, someone would scream for being disconnected from the Internet. Evey single relay is important but not that important.
> Blocking relay to relay communication should not be done > as it breaks a core assumption of the tor network (every relay can talk > to all other relays ORPorts)
Okay, now let's be clear about DDoS mitigations. There's no such thing as DDoS mitigation without rate limiting. Both my rules and @toralf rules and frankly any other rules that anyone could come up with by themselves, includes rate limiting.This means that at any given time, someone, somewhere is not allowed to connect due to their behaviour.
The idea that all relays must be able to connect to other relays any time and in any shape or form they choose can not exist in real world of DDoS mitigation. Right at this moment I have about 1600 relays that have established connections to me and I probably have as many connections to other relays and my relay is happily operating at about its Max Advertised Bandwidth. Do I really need to keep 6000 relays in my allow list and let them do whatever they want? Giving 6000 servers a free reign on your system is unheard of in the security world.
DDoS mitigation like anything else in life is about compromise. Ideals are sacrificed in favour of "good to haves" and "good to haves" are sacrificed in favour of survival. Otherwise you'd be a part of the currently [over 2000 relays that are overloaded](https://github.com/Enkidu-6/tor-relay-lists/blob/main/overloaded.txt) and passing it along.
When I'm attacked, it's not just about me. I'm relaying that attack to the next relay and they relay that to the next one. So the idea that I should accept the attack when it's coming from another relay is simply unacceptable.
Again I'd like to be very clear, this is not about blocking relays, it's about rate limiting. They do get to connect but at a reasonable rate. For example, there's no justifiable reason for a relay to try to connect to another relay at a rate of 10 times a second for 15 minutes straight.
So if for example we say a relay can have two Established connections to me, we're not blocking them. If they do need the connection, they use it. If they close a connection and at some point they need one, they can open another one. But they can't have 10 connections at the same time.
Again, My goal here with my questions is to find a way to keep the balance of that compromise in our favour. But thinking that we can go on without making those compromises would be naive.
Again thanks for reading and thank you for your time and response.
On 2/6/2023 6:17 PM, nusenu wrote:
Hi,
thanks for raising these questions and trying to understand before deploying/changes to filters.
A good understanding of how tor relays and connections work is important when trying to defend against overload attacks, without breaking functionality with packet filters that cause false positive blocks, especially when such a long standing limit like the relays per IPv4 address limit is changed.
- I have a few Exit relays as permanent residents in my block list, not
because I want them to be there but because, no matter how many times I remove them, day or night, they'll be back in seconds for making too many concurrent attempts.
As a relay operator you should allow all other relays to connect to your relay's ORPort no matter what flags or onionoo guard/middle/exit probability they might have. Place known relay IPs on an exception list so your filters don't block them and update that list at least every hour.
If you have a problem on your relay's ORPort with a source IP that is also used by an exit relay please try to contact the operator by looking at their contactinfo, if they don't have a contactinfo, join the 'require a usable contactinfo' lobby ;) for this very reason and maybe ask on this list if they can drop you an email.
You can also not be sure whether it is an actual authenticated relay to relay connection or a client to relay connection just by looking at the source IP. Some upcoming MetricsPort enhancement might help you there in the future not per source IP but as a general overview for your relay's ORPort connections. An exit can share it's source IP with tor clients for example behind NAT, but I don't think that is common and it is also discouraged. Exits should have dedicated IPs that is not shared for unrelated things.
Blocking relay to relay communication should not be done as it breaks a core assumption of the tor network (every relay can talk to all other relays ORPorts). Upcoming tooling that detects broken relay to relay links might also detect and flag your relay if your filters break relay to relay communication.
As an option of last resort - after verifying it is an authenticated relay to relay connection that is causing you trouble and not some tor client using the same source IP you might contact the bad-relays list. That is still better than blocking another relay from reaching your relay's ORPort.
I've seen other problematic filter practices for relay to relay connections and I'll write up some recommendations in a separate email so it doesn't get lost in this lengthy email.
I'm assuming this is due to the fact that Exits are being used to attack other relays
In a vanilla current tor version it is not possible to use a tor client to connect via an exit to another relay's ORPort so this is very unlikely. The background here is that tor does not allow such connections to prevent an attacker from reentering the tor network via an exit relay.
You can test that by opening this URL in tor browser, you will get a "Unable to connect" very fast:
https://185.220.102.242/ because it is this ORPort: https://metrics.torproject.org/rs.html#details/0A2366980A2842D770EF8E136A7DA...
the answer is very fast and not a slow timeout because a tor client can predict that is inaccessible before even trying to create a stream because exits will not allow such connections to relay ORPorts.
I have 2 Established connections to two Or Ports of an exit relay
I was not able to follow you there. I was unable to find any exit relay that has more than one ORPort on IPv4 (I identify relays by fingerprint not by IP address). Maybe you can list the specific exit relay fingerprint and timestamp so I can cross check for bugs in my tooling/onionoo?
- Each relay has Established connections to many other relays and if
they're guard they will also have many connections to regular users and their Tor browsers until they have enough traffic to reach their MaxAdvertisedBandwidth. Obviously we don't Establish connections to all 6300 relays out there.
It is best to actually expect that.
So if we do not allow each IP more than two connections and they need 4, They'll have two from us and they'll move on to another relay and get the other two and get the job done and we will reach our Max Bandwidth anyway by accepting traffic from other relays. Diversity of relays as opposed to concentration of some relays. Am I correct in my assumption that this will have little to no effect on the health of the Tor network as a whole?
I think there is a misunderstanding there. One important point to take away: Relays do not chose the next hop in a circuit, tor clients do. So if my tor clients picks a circuit with these relays: client -> A -> B -> C -> example.com and B can not reach C, B can not say "oh I can not reach C I'll pick D instead for you", a relay has no say in that. I hope that helps to reinforce the importance of ubiquitous reachability between relays. Relays have to obey a tor client's orders and a tor client expects that all relays can talk to each other. A few years ago David Stainton published some actual scan results on tor-dev that showed that this expectation might not be true in reality but close enough.
I fear that overly aggressive relay to relay filter actually help the adversaries more than the network and would advise against filtering practices between relays. The first step should always be on a social level: Try to reach the operator if you feel they attack your relay and NOT iptables DROP without notice.
kind regards, nusenu
Even if that happens, why would a client connect directly to an Exit and get the Exit to connect to another relay or Guard using the Exit's IP address?
You mentioned the exit flag, but you didn't specify whether that relay also had the guard flag.
Generally speaking it is correct, that if you fire up a fresh tor client with default settings for the first time it will most likely pick a relay with the guard flag and no exit flag as their guard relay, but...:
Tor clients keep local state over a considerable amount of time. Most notably they pick their guard relays rarely.
It is possible that a relay that has the guard+exit flags today, only had the guard flag yesterday and the operator decided to enable exiting by changing the exit policy just this morning. So if that relay was picked as guard yesterday, their guard probability of today might not matter so much. I recall a gitlab.tpo issue that discussed the details of whether tor clients should change guards when their picked guard lost/gained flags. Maybe someone else could paste a link to it.
related content for the interested reader: https://lists.torproject.org/pipermail/tor-relays/2020-October/019002.html https://gitlab.torproject.org/tpo/core/tor/-/issues/40230
Wouldn't the same protection you mentioned kick in and stop a client?
The protection I mentioned (reenter the tor network via exits blocking) does not interfere with a tor client directly connecting to a guard.
In the old days of vidalia you could actually see that in real time.
for anyone missing that experience, onioncircuits can help you there :) but it has less features than vidalia used to have. https://gitlab.tails.boum.org/tails/onioncircuits
Each time you try to browse a page, Tor doesn't say, oh let's find a relay. It picks a circuit from a pool of already Established ones. If Tor tries to build a circuit with a relay and it's unable to, it will try another one that works and keeps it in the pool for when it's needed. Again, I may be wrong. Feel free to tell me if I am.
Yes tor clients preemptively create circuits before creating actual streams to reduce latency.
There's no such thing as DDoS mitigation without rate limiting.
DDoS rate limit filters do not require an all or nothing approach, different source IPs can be handled differently see toralf's use of onionoo to feed ipsets as an example. I would recommend to use tor's controlport as a source of information instead though because onionoo is not meant to be used for such real time needs.
I don't think relays should silently drop other relays packets without first trying: - to confirm that accepting that IP would render the relay (mostly) unusable (by first running in a mode that accepts relay IPs) - to understand the actual root cause - to contact the relay operator at the other - to report relays they consider malicious
Not investigating such cases is a missed opportunity to find potential bugs or a new detection mechanism for malicious relays. It is also a missed opportunity to help protect the tor network at a higher level, because it is unlikely that everyone is using (the same) filters.
Filters that result in blocks for a large fraction of the tor network are more likely a sign of an unsuitable filters than an indicator that most of the tor network is engaging in attack against other relays, especially when they include well known and long term trusted operators.
This a good topic to be added to the Expectations for Relay Operators document. https://gitlab.torproject.org/tpo/community/team/-/wikis/Expectations-for-Re...
At the very least relays blocking/dropping some packets of other relays should be very transparent about it.
kind regards, nusenu
On Wed, Feb 08, 2023 at 12:07:22AM +0100, nusenu wrote:
I recall a gitlab.tpo issue that discussed the details of whether tor clients should change guards when their picked guard lost/gained flags. Maybe someone else could paste a link to it.
This might be the one you want: https://gitlab.torproject.org/tpo/core/torspec/-/issues/141
It has quite the complicated story, and I think our current behavior ("move to the next guard in the list if our current guard doesn't have the Guard flag") is good for performance but bad for security, but it's a complicated enough analysis that reasonable people can disagree.
--Roger
DDoS rate limit filters do not require an all or nothing approach, different source IPs can be handled differently see toralf's use of onionoo to feed ipsets as an example. I would recommend to use tor's controlport as a source of information instead though because onionoo is not meant to be used for such real time needs.
We do the same. You can take a look at the rules. There's no all or nothing approach. Regardless, this doesn't change the fact that in either approach there's always a time when someone, somewhere will not be able to connect due to their behaviour.
The ipsets are pulled from [my repository](https://github.com/Enkidu-6/tor-relay-lists) to avoid putting undue pressure on oniooo server and my lists are updated every half an hour. This means one query as opposed to very many for each user.
We do have a ban list and cron jobs to remove relays from the list periodically -even though I'm not a fan of it- to avoid an all out ban. However the number of allowed connections remain the same regardless whether you're in the ban list or not.
ipsets are set to allow authorities and snowflake servers to have as many connections as they need. Dual-or relays are in another ipset and they can have more connections per IP. That will still stay limited. How many would be necessary, is the main reason I started this discussion.
I don't think relays should silently drop other relays packets without first trying:
- to confirm that accepting that IP would render the relay (mostly) unusable (by first running in a mode that accepts relay IPs)
- to understand the actual root cause
- to contact the relay operator at the other
- to report relays they consider malicious
Firewalls and iptables don't work that way. They make those decisions in fractions of milliseconds and do what they're told. Now, if you're suggesting that I should do all of that before writing a rule to protect my server, I do what I can with the time I have and to a reasonable level.
No new versions of rules are released unless I run them for 2 weeks on my own servers and check for all possible side effects. I spend time to investigate the relays that are misbehaving to the best of my knowledge and ability and spend as much time as I'm willing to spend on it. Again it's all about that compromise I was talking about. I'm not looking to achieve an ideal solution for an ideal world. That's not achievable and would be a waste of time trying. My purpose for this discussion is to find the best way to keep the balance of that compromise in our favour and not do damage to the overall health of the Tor network.
Now to clarify a few things, I'll explain my process. The rules are tested on two relays with 20 MiB Max Advertised bandwidth each and each one has full use of 12 CPUs and about 5-7 GB of RAM and NumCPUs of 30. By the way this is far better than the systems majority of relays are run on. Majority of operators run their relays on VPS systems with 1-4 CPUs and 1-4 GB of RAM.
The idea is that if a ruleset causes my relay to go to the overloaded status, it will certainly do more damage to weaker relays. I can tell you with absolute certainty that with the current state of the DDoS we're experiencing if we allow more than 2 connections per IP (1 would be better) the average relay will have overloaded status at least 2-3 times a week if not more. Again we're talking about average relays not beasts.
Now to get back to my original question, I'll try to make it a bit more clear. It may help you answer my question better. Here are the current statistics:
Since the start of the new rule of 4 relays per IP, 14 operators have decided to take advantage of it. Together combined, they are operating 214 relays, out which about more or less 100 something relays are actually new relays added to the network. Some added two, some added only one. Among them I've seen fake email addresses, those who haven't even added the new relays to their family, etc.. By the way those numbers haven't increased in the past 2 or3 days and remained pretty much the same.
All of these relays with 3-4 Tor instances are running on a total of 54 IP addresses, 39 of which running purely Exit relays on all instances of Tor on the IP with 0% guard probability and 0% middle probability. Their status might change after a couple of weeks to a month when they join the network with full capacity and I'll be monitoring that.
So I ask my question again, with the current state, right at this moment, If I allow those limited IP addresses only 2 established connections at a time instead of 4, will that break anything or cause any harm to the health of the Tor network as a whole? My own answer is absolutely not, but I eagerly await everyone's opinion.
Cheers.
On 2/7/2023 6:07 PM, nusenu wrote:
Even if that happens, why would a client connect directly to an Exit and get the Exit to connect to another relay or Guard using the Exit's IP address?
You mentioned the exit flag, but you didn't specify whether that relay also had the guard flag.
Generally speaking it is correct, that if you fire up a fresh tor client with default settings for the first time it will most likely pick a relay with the guard flag and no exit flag as their guard relay, but...:
Tor clients keep local state over a considerable amount of time. Most notably they pick their guard relays rarely.
It is possible that a relay that has the guard+exit flags today, only had the guard flag yesterday and the operator decided to enable exiting by changing the exit policy just this morning. So if that relay was picked as guard yesterday, their guard probability of today might not matter so much. I recall a gitlab.tpo issue that discussed the details of whether tor clients should change guards when their picked guard lost/gained flags. Maybe someone else could paste a link to it.
related content for the interested reader: https://lists.torproject.org/pipermail/tor-relays/2020-October/019002.html
https://gitlab.torproject.org/tpo/core/tor/-/issues/40230
Wouldn't the same protection you mentioned kick in and stop a client?
The protection I mentioned (reenter the tor network via exits blocking) does not interfere with a tor client directly connecting to a guard.
In the old days of vidalia you could actually see that in real time.
for anyone missing that experience, onioncircuits can help you there :) but it has less features than vidalia used to have. https://gitlab.tails.boum.org/tails/onioncircuits
Each time you try to browse a page, Tor doesn't say, oh let's find a relay. It picks a circuit from a pool of already Established ones. If Tor tries to build a circuit with a relay and it's unable to, it will try another one that works and keeps it in the pool for when it's needed. Again, I may be wrong. Feel free to tell me if I am.
Yes tor clients preemptively create circuits before creating actual streams to reduce latency.
There's no such thing as DDoS mitigation without rate limiting.
DDoS rate limit filters do not require an all or nothing approach, different source IPs can be handled differently see toralf's use of onionoo to feed ipsets as an example. I would recommend to use tor's controlport as a source of information instead though because onionoo is not meant to be used for such real time needs.
I don't think relays should silently drop other relays packets without first trying:
- to confirm that accepting that IP would render the relay (mostly)
unusable (by first running in a mode that accepts relay IPs)
- to understand the actual root cause
- to contact the relay operator at the other
- to report relays they consider malicious
Not investigating such cases is a missed opportunity to find potential bugs or a new detection mechanism for malicious relays. It is also a missed opportunity to help protect the tor network at a higher level, because it is unlikely that everyone is using (the same) filters.
Filters that result in blocks for a large fraction of the tor network are more likely a sign of an unsuitable filters than an indicator that most of the tor network is engaging in attack against other relays, especially when they include well known and long term trusted operators.
This a good topic to be added to the Expectations for Relay Operators document. https://gitlab.torproject.org/tpo/community/team/-/wikis/Expectations-for-Re...
At the very least relays blocking/dropping some packets of other relays should be very transparent about it.
kind regards, nusenu
On Mittwoch, 8. Februar 2023 00:07:22 CET nusenu wrote:
I don't think relays should silently drop other relays packets without first trying:
- to confirm that accepting that IP would render the relay (mostly) unusable
(by first running in a mode that accepts relay IPs) - to understand the actual root cause
- to contact the relay operator at the other
- to report relays they consider malicious
Not investigating such cases is a missed opportunity to find potential bugs or a new detection mechanism for malicious relays. It is also a missed opportunity to help protect the tor network at a higher level, because it is unlikely that everyone is using (the same) filters.
Filters that result in blocks for a large fraction of the tor network are more likely a sign of an unsuitable filters than an indicator that most of the tor network is engaging in attack against other relays, especially when they include well known and long term trusted operators.
This a good topic to be added to the Expectations for Relay Operators document. https://gitlab.torproject.org/tpo/community/team/-/wikis/Expectations-for-R elay-Operators
At the very least relays blocking/dropping some packets of other relays should be very transparent about it.
On exits, i don't block|rate limit via nftables I block via exit policy. Alex from Artikel10 created a nice python script: https://github.com/artikel10/surgeprotector I'm running the script with 10000 TCP connections as the LIMIT.
Once issue ¹#40676 is resolved, the relays will no longer need to be restarted. A reload would close already established outbound connections.
¹https://gitlab.torproject.org/tpo/core/tor/-/issues/40676
@Enkidu
As an user of your filtering script, I want to first say thank you for maintaining the script!
The idea that all relays must be able to connect to other relays any time
and in any shape or form they choose can not exist in real world of DDoS mitigation.
I totally agree, however I want to also add a note here regarding privacy: If an adversary can nudge a user to choose some particular circuit / relay and avoid other circuits, they can break the privacy guarantee of Tor network. Aggressive automated rate limiting might provide adversaries with this opportunity, as an adversary might find ways to "trigger" the filtering and cause more circuits between "good" relays to be blocked, and clients will automatically fall back to circults that go through two or more malicious relays operated by the adversary (leading to deanonymization).
This attack surface is a pure theoretical speculation and probably not happening right now. Still, at a high level we need to be careful when designing automated filtering mechanisms, and I suggest erring on the more conservative side. E.g., 5 incoming connections per relay instance, so if an IP has 4 or 8 relay instances it can have 4*5=20 or 8*5=40 connections. (Of course, the quota should depend on flags, so an attacker can't just spin up 8 new relays per IP.)
Also, it might be good to have a more conservative (higher) rate limit by default and let users turn it down as necessary. -- Danny
On Tue, Feb 7, 2023 at 8:34 AM Chris Enkidu-6 tor@wcbsecurity.com wrote:
@nusenu
Thank you very much for taking the time to help me understand things better. I can use all the help I can get.
> You can also not be sure whether it is an actual authenticatedrelay to relay > connection or a client to relay connection just by looking at the source IP.
> In a vanilla current tor version it is not possible to use a torclient > to connect via an exit to another relay's ORPort so this is very unlikely.
> An exit can share it's source IP with tor clients for examplebehind NAT
Sorry I'm confused here. Those statements seem to contradict each other or there's something here I misunderstood. Even if that happens, why would a client connect directly to an Exit and get the Exit to connect to another relay or Guard using the Exit's IP address? Wouldn't the same protection you mentioned kick in and stop a client?
> I was not able to follow you there. I was unable to find any exitrelay that > has more than one ORPort on IPv4 (I identify relays by fingerprint not by IP address).
Sorry, I should have been more careful in using my words. In the world of firewalls and iptables we identify connections by IPAddress:port pairs hence my inaccurate wording. A relay has only one ORPort but an IP address can have 4.
> I think there is a misunderstanding there. One important point totake away: > Relays do not chose the next hop in a circuit, tor clients do. > So if my tor clients picks a circuit with these relays: > client -> A -> B -> C -> example.com > and B can not reach C, B can not say "oh I can not reach C I'll pick D instead for you",
I understand that and I think it might be my fault again for not being clear. It doesn't matter who makes that choice. What matters is that the choice is being made. My understanding is that the moment you open your Tor Browser, it starts establishing connections and building multiple circuits and keeps them in a pool.
In the old days of vidalia you could actually see that in real time. Many established circuits were just sitting there waiting. Each time you try to browse a page, Tor doesn't say, oh let's find a relay. It picks a circuit from a pool of already Established ones. If Tor tries to build a circuit with a relay and it's unable to, it will try another one that works and keeps it in the pool for when it's needed. Again, I may be wrong. Feel free to tell me if I am.
If that wasn't the case, any time I rebooted, someone would scream for being disconnected from the Internet. Evey single relay is important but not that important.
> Blocking relay to relay communication should not be done > as it breaks a core assumption of the tor network (every relay cantalk > to all other relays ORPorts)
Okay, now let's be clear about DDoS mitigations. There's no such thing as DDoS mitigation without rate limiting. Both my rules and @toralf rules and frankly any other rules that anyone could come up with by themselves, includes rate limiting.This means that at any given time, someone, somewhere is not allowed to connect due to their behaviour.
The idea that all relays must be able to connect to other relays any time and in any shape or form they choose can not exist in real world of DDoS mitigation. Right at this moment I have about 1600 relays that have established connections to me and I probably have as many connections to other relays and my relay is happily operating at about its Max Advertised Bandwidth. Do I really need to keep 6000 relays in my allow list and let them do whatever they want? Giving 6000 servers a free reign on your system is unheard of in the security world.
DDoS mitigation like anything else in life is about compromise. Ideals are sacrificed in favour of "good to haves" and "good to haves" are sacrificed in favour of survival. Otherwise you'd be a part of the currently [over 2000 relays that are overloaded]( https://github.com/Enkidu-6/tor-relay-lists/blob/main/overloaded.txt) and passing it along.
When I'm attacked, it's not just about me. I'm relaying that attack to the next relay and they relay that to the next one. So the idea that I should accept the attack when it's coming from another relay is simply unacceptable.
Again I'd like to be very clear, this is not about blocking relays, it's about rate limiting. They do get to connect but at a reasonable rate. For example, there's no justifiable reason for a relay to try to connect to another relay at a rate of 10 times a second for 15 minutes straight.
So if for example we say a relay can have two Established connections to me, we're not blocking them. If they do need the connection, they use it. If they close a connection and at some point they need one, they can open another one. But they can't have 10 connections at the same time.
Again, My goal here with my questions is to find a way to keep the balance of that compromise in our favour. But thinking that we can go on without making those compromises would be naive.
Again thanks for reading and thank you for your time and response.
On 2/6/2023 6:17 PM, nusenu wrote:
Hi,
thanks for raising these questions and trying to understand before deploying/changes to filters.
A good understanding of how tor relays and connections work is important when trying to defend against overload attacks, without breaking functionality with packet filters that cause false positive blocks, especially when such a long standing limit like the relays per IPv4 address limit is changed.
- I have a few Exit relays as permanent residents in my block list, not
because I want them to be there but because, no matter how many times I remove them, day or night, they'll be back in seconds for making too many concurrent attempts.
As a relay operator you should allow all other relays to connect to your relay's ORPort no matter what flags or onionoo guard/middle/exit probability they might have. Place known relay IPs on an exception list so your filters don't block them and update that list at least every hour.
If you have a problem on your relay's ORPort with a source IP that is also used by an exit relay please try to contact the operator by looking at their contactinfo, if they don't have a contactinfo, join the 'require a usable contactinfo' lobby ;) for this very reason and maybe ask on this list if they can drop you an email.
You can also not be sure whether it is an actual authenticated relay to relay connection or a client to relay connection just by looking at the source IP. Some upcoming MetricsPort enhancement might help you there in the future not per source IP but as a general overview for your relay's ORPort connections. An exit can share it's source IP with tor clients for example behind NAT, but I don't think that is common and it is also discouraged. Exits should have dedicated IPs that is not shared for unrelated things.
Blocking relay to relay communication should not be done as it breaks a core assumption of the tor network (every relay can talk to all other relays ORPorts). Upcoming tooling that detects broken relay to relay links might also detect and flag your relay if your filters break relay to relay communication.
As an option of last resort - after verifying it is an authenticated relay to relay connection that is causing you trouble and not some tor client using the same source IP you might contact the bad-relays list. That is still better than blocking another relay from reaching your relay's ORPort.
I've seen other problematic filter practices for relay to relay connections and I'll write up some recommendations in a separate email so it doesn't get lost in this lengthy email.
I'm assuming this is due to the fact that Exits are being used to attack other relays
In a vanilla current tor version it is not possible to use a tor client to connect via an exit to another relay's ORPort so this is very unlikely. The background here is that tor does not allow such connections to prevent an attacker from reentering the tor network via an exit relay.
You can test that by opening this URL in tor browser, you will get a "Unable to connect" very fast:
https://185.220.102.242/ because it is this ORPort:
https://metrics.torproject.org/rs.html#details/0A2366980A2842D770EF8E136A7DA...
the answer is very fast and not a slow timeout because a tor client can predict that is inaccessible before even trying to create a stream because exits will not allow such connections to relay ORPorts.
I have 2 Established connections to two Or Ports of an exit relay
I was not able to follow you there. I was unable to find any exit relay that has more than one ORPort on IPv4 (I identify relays by fingerprint not by IP address). Maybe you can list the specific exit relay fingerprint and timestamp so I can cross check for bugs in my tooling/onionoo?
- Each relay has Established connections to many other relays and if
they're guard they will also have many connections to regular users and their Tor browsers until they have enough traffic to reach their MaxAdvertisedBandwidth. Obviously we don't Establish connections to all 6300 relays out there.
It is best to actually expect that.
So if we do not allow each IP more than two connections and they need 4, They'll have two from us and they'll move on to another relay and get the other two and get the job done and we will reach our Max Bandwidth anyway by accepting traffic from other relays. Diversity of relays as opposed to concentration of some relays. Am I correct in my assumption that this will have little to no effect on the health of the Tor network as a whole?
I think there is a misunderstanding there. One important point to take away: Relays do not chose the next hop in a circuit, tor clients do. So if my tor clients picks a circuit with these relays: client -> A -> B -> C -> example.com and B can not reach C, B can not say "oh I can not reach C I'll pick D instead for you", a relay has no say in that. I hope that helps to reinforce the importance of ubiquitous reachability between relays. Relays have to obey a tor client's orders and a tor client expects that all relays can talk to each other. A few years ago David Stainton published some actual scan results on tor-dev that showed that this expectation might not be true in reality but close enough.
I fear that overly aggressive relay to relay filter actually help the adversaries more than the network and would advise against filtering practices between relays. The first step should always be on a social level: Try to reach the operator if you feel they attack your relay and NOT iptables DROP without notice.
kind regards, nusenu
tor-relays mailing list tor-relays@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays
Hi, Danny
Those theoretical concerns may or may not be valid as I don't have enough expertise about how Tor operates under the hood to comment on it, but I can tell you that currently there are a few different DDoS attacks with different purposes but they don't seem to have the surgical accuracy you're suggesting. What I can tell you though is what I mentioned in my other post. Allowing the numbers of established connections per IP as you suggest would not end well for average operator.
My script also generates a file by the name rules.sh for users to examine and see what actually happened to their system when they ran my script. It's nothing but iptables rules in plain text with no fancy scripts or codes. Any operator can modify that file as they wish and let's say change 2 to 5 and run it again. The new rules will apply and replace the original. I did that on purpose because I believe users shouldn't run a script when they may not have the expertise to be able to tell what the script does simply because they trust me and then hope for the best.
However if you choose to change those numbers be prepared to be the test subject instead of me. :)
Cheers
On 2/8/2023 12:09 AM, Xiaoqi Chen (Danny) wrote:
@Enkidu
As an user of your filtering script, I want to first say thank you for maintaining the script!
The idea that all relays must be able to connect to other relays any
time and in any shape or form they choose can not exist in real world of DDoS mitigation.
I totally agree, however I want to also add a note here regarding privacy: If an adversary can nudge a user to choose some particular circuit / relay and avoid other circuits, they can break the privacy guarantee of Tor network. Aggressive automated rate limiting might provide adversaries with this opportunity, as an adversary might find ways to "trigger" the filtering and cause more circuits between "good" relays to be blocked, and clients will automatically fall back to circults that go through two or more malicious relays operated by the adversary (leading to deanonymization).
This attack surface is a pure theoretical speculation and probably not happening right now. Still, at a high level we need to be careful when designing automated filtering mechanisms, and I suggest erring on the more conservative side. E.g., 5 incoming connections per relay instance, so if an IP has 4 or 8 relay instances it can have 4*5=20 or 8*5=40 connections. (Of course, the quota should depend on flags, so an attacker can't just spin up 8 new relays per IP.)
Also, it might be good to have a more conservative (higher) rate limit by default and let users turn it down as necessary. -- Danny
On Tue, Feb 7, 2023 at 8:34 AM Chris Enkidu-6 <tor@wcbsecurity.com mailto:tor@wcbsecurity.com> wrote:
@nusenu Thank you very much for taking the time to help me understand things better. I can use all the help I can get. > You can also not be sure whether it is an actual authenticated relay to relay > connection or a client to relay connection just by looking at the source IP. > In a vanilla current tor version it is not possible to use a tor client > to connect via an exit to another relay's ORPort so this is very unlikely. > An exit can share it's source IP with tor clients for example behind NAT Sorry I'm confused here. Those statements seem to contradict each other or there's something here I misunderstood. Even if that happens, why would a client connect directly to an Exit and get the Exit to connect to another relay or Guard using the Exit's IP address? Wouldn't the same protection you mentioned kick in and stop a client? > I was not able to follow you there. I was unable to find any exit relay that > has more than one ORPort on IPv4 (I identify relays by fingerprint not by IP address). Sorry, I should have been more careful in using my words. In the world of firewalls and iptables we identify connections by IPAddress:port pairs hence my inaccurate wording. A relay has only one ORPort but an IP address can have 4. > I think there is a misunderstanding there. One important point to take away: > Relays do not chose the next hop in a circuit, tor clients do. > So if my tor clients picks a circuit with these relays: > client -> A -> B -> C -> example.com <http://example.com> > and B can not reach C, B can not say "oh I can not reach C I'll pick D instead for you", I understand that and I think it might be my fault again for not being clear. It doesn't matter who makes that choice. What matters is that the choice is being made. My understanding is that the moment you open your Tor Browser, it starts establishing connections and building multiple circuits and keeps them in a pool. In the old days of vidalia you could actually see that in real time. Many established circuits were just sitting there waiting. Each time you try to browse a page, Tor doesn't say, oh let's find a relay. It picks a circuit from a pool of already Established ones. If Tor tries to build a circuit with a relay and it's unable to, it will try another one that works and keeps it in the pool for when it's needed. Again, I may be wrong. Feel free to tell me if I am. If that wasn't the case, any time I rebooted, someone would scream for being disconnected from the Internet. Evey single relay is important but not that important. > Blocking relay to relay communication should not be done > as it breaks a core assumption of the tor network (every relay can talk > to all other relays ORPorts) Okay, now let's be clear about DDoS mitigations. There's no such thing as DDoS mitigation without rate limiting. Both my rules and @toralf rules and frankly any other rules that anyone could come up with by themselves, includes rate limiting.This means that at any given time, someone, somewhere is not allowed to connect due to their behaviour. The idea that all relays must be able to connect to other relays any time and in any shape or form they choose can not exist in real world of DDoS mitigation. Right at this moment I have about 1600 relays that have established connections to me and I probably have as many connections to other relays and my relay is happily operating at about its Max Advertised Bandwidth. Do I really need to keep 6000 relays in my allow list and let them do whatever they want? Giving 6000 servers a free reign on your system is unheard of in the security world. DDoS mitigation like anything else in life is about compromise. Ideals are sacrificed in favour of "good to haves" and "good to haves" are sacrificed in favour of survival. Otherwise you'd be a part of the currently [over 2000 relays that are overloaded](https://github.com/Enkidu-6/tor-relay-lists/blob/main/overloaded.txt) and passing it along. When I'm attacked, it's not just about me. I'm relaying that attack to the next relay and they relay that to the next one. So the idea that I should accept the attack when it's coming from another relay is simply unacceptable. Again I'd like to be very clear, this is not about blocking relays, it's about rate limiting. They do get to connect but at a reasonable rate. For example, there's no justifiable reason for a relay to try to connect to another relay at a rate of 10 times a second for 15 minutes straight. So if for example we say a relay can have two Established connections to me, we're not blocking them. If they do need the connection, they use it. If they close a connection and at some point they need one, they can open another one. But they can't have 10 connections at the same time. Again, My goal here with my questions is to find a way to keep the balance of that compromise in our favour. But thinking that we can go on without making those compromises would be naive. Again thanks for reading and thank you for your time and response. On 2/6/2023 6:17 PM, nusenu wrote: > Hi, > > thanks for raising these questions and trying to understand before > deploying/changes to filters. > > A good understanding of how tor relays and connections work is > important when trying > to defend against overload attacks, without breaking functionality > with packet filters that cause false positive blocks, especially when > such a > long standing limit like the relays per IPv4 address limit is changed. > >> - I have a few Exit relays as permanent residents in my block list, not >> because I want them to be there but because, no matter how many times I >> remove them, day or night, they'll be back in seconds for making too >> many concurrent attempts. > > As a relay operator you should allow all other relays > to connect to your relay's ORPort no matter what flags or onionoo > guard/middle/exit probability they might have. > Place known relay IPs on an exception list so your filters don't block > them > and update that list at least every hour. > > If you have a problem on your relay's ORPort with a source IP that > is also used by an exit relay please try to contact > the operator by looking at their contactinfo, if they don't > have a contactinfo, join the 'require a usable contactinfo' lobby ;) > for this very reason and maybe ask on this list if they can drop you > an email. > > You can also not be sure whether it is an actual authenticated relay > to relay > connection or a client to relay connection just by looking at the > source IP. > Some upcoming MetricsPort enhancement might help you there in the future > not per source IP but as a general overview for your relay's ORPort > connections. > An exit can share it's source IP with tor clients for example behind NAT, > but I don't think that is common and it is also discouraged. Exits > should have dedicated > IPs that is not shared for unrelated things. > > Blocking relay to relay communication should not be done > as it breaks a core assumption of the tor network (every relay can talk > to all other relays ORPorts). Upcoming tooling that detects > broken relay to relay links might also detect and flag your relay > if your filters break relay to relay communication. > > As an option of last resort - after verifying it is an authenticated > relay to relay connection that is causing you trouble and not some tor > client using the same source IP > you might contact the bad-relays list. That is still better than blocking > another relay from reaching your relay's ORPort. > > I've seen other problematic filter practices for relay to relay > connections and I'll > write up some recommendations in a separate email so it doesn't get > lost in this lengthy email. > >> I'm assuming this is due to the fact that >> Exits are being used to attack other relays > > In a vanilla current tor version it is not possible to use a tor client > to connect via an exit to another relay's ORPort so this is very > unlikely. The background here > is that tor does not allow such connections to prevent an attacker > from reentering > the tor network via an exit relay. > > You can test that by opening this URL in tor browser, you will get a > "Unable to connect" very fast: > > https://185.220.102.242/ > because it is this ORPort: > https://metrics.torproject.org/rs.html#details/0A2366980A2842D770EF8E136A7DA14876360447 > > > the answer is very fast and not a slow timeout because a tor client > can predict > that is inaccessible before even trying to create a stream > because exits will not allow such connections to relay ORPorts. > >> I have 2 Established connections to two Or Ports of an exit relay > > I was not able to follow you there. I was unable to find any exit > relay that > has more than one ORPort on IPv4 (I identify relays by fingerprint not > by IP address). > Maybe you can list the specific exit relay fingerprint and timestamp > so I can cross > check for bugs in my tooling/onionoo? > >> - Each relay has Established connections to many other relays and if >> they're guard they will also have many connections to regular users and >> their Tor browsers until they have enough traffic to reach their >> MaxAdvertisedBandwidth. Obviously we don't Establish connections to all >> 6300 relays out there. > > It is best to actually expect that. > >> So if we do not allow each IP more than two >> connections and they need 4, They'll have two from us and they'll move >> on to another relay and get the other two and get the job done and we >> will reach our Max Bandwidth anyway by accepting traffic from other >> relays. Diversity of relays as opposed to concentration of some relays. >> Am I correct in my assumption that this will have little to no effect on >> the health of the Tor network as a whole? > > I think there is a misunderstanding there. One important point to take > away: > Relays do not chose the next hop in a circuit, tor clients do. > So if my tor clients picks a circuit with these relays: > client -> A -> B -> C -> example.com <http://example.com> > and B can not reach C, B can not say "oh I can not reach C I'll pick D > instead for you", > a relay has no say in that. I hope that helps to reinforce the > importance of > ubiquitous reachability between relays. Relays have to obey a tor > client's > orders and a tor client expects that all relays can talk to each other. > A few years ago David Stainton published some actual scan results on > tor-dev > that showed that this expectation might not be true in reality but > close enough. > > I fear that overly aggressive relay to relay filter actually help the > adversaries more > than the network and would advise against filtering practices between > relays. > The first step should always be on a social level: Try to reach the > operator if you feel > they attack your relay and NOT iptables DROP without notice. > > kind regards, > nusenu > _______________________________________________ tor-relays mailing list tor-relays@lists.torproject.org <mailto:tor-relays@lists.torproject.org> https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays
tor-relays@lists.torproject.org
-
Chris Enkidu-6 -
lists＠for-privacy.net -
nusenu -
Roger Dingledine -
Xiaoqi Chen (Danny)