Reapply exit policy on reload
Hi to all dear exit operators, If you are interested in applying the exit policy on reload and not by restarting tor please note: https://gitlab.torproject.org/tpo/core/tor/-/merge_requests/735#note_3051797 Quote David: "Can you give us a sense of how many exit operators use this? If there is a large enough need for this, we can evaluate this for next release but it needs to be for more than 1 operator for such feature." Related Issue: https://gitlab.torproject.org/tpo/core/tor/-/issues/40676 -- ╰_╯ Ciao Marco! Debian GNU/Linux It's free software and it gives you freedom!
to be clear about what this feature does: it was already possible to add more rules, and these rules would apply to new connections made from your exit, but it would *not* kill existing connections which violate the new policy. `ReevaluateExitPolicy` allows reevaluating the new exit policy on existing connections, killing any connection that would no longer be allowed. This was previously possible only by restarting the relay, killing every good connection in the process. This feature has been available in 0.4.9 for some time now, but that version is not considered stable yet. If as an exit relay operator you are running 0.4.8 and regularly restart your relay to force-apply a new exit policy on old connections, or have wished you could do that, but didn't because of the large side effects, please make it known either by replying in this thread, or by thumbs-up-ing boldsuck's comment on https://gitlab.torproject.org/tpo/core/tor/-/merge_requests/735 . regards, trinity-1686a On Wed, 24 Jul 2024 at 16:11, boldsuck <lists@for-privacy.net> wrote:
Hi to all dear exit operators,
If you are interested in applying the exit policy on reload and not by restarting tor please note:
https://gitlab.torproject.org/tpo/core/tor/-/merge_requests/735#note_3051797 Quote David: "Can you give us a sense of how many exit operators use this? If there is a large enough need for this, we can evaluate this for next release but it needs to be for more than 1 operator for such feature."
Related Issue: https://gitlab.torproject.org/tpo/core/tor/-/issues/40676
-- ╰_╯ Ciao Marco!
Debian GNU/Linux
It's free software and it gives you freedom!_______________________________________________ tor-relays mailing list tor-relays@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays
I would definitely want to be able to change my exit policy by just sending a simple "kill -SIGHUP $pid". So yeah, consider myself interested in this functionality. But, don't we already have that implemented? I remember changing my exit policy then doing "systemctl reload tor" and after a few hours, Metrics showed that SSH was now also rejected. Weird. All the best, George Hartley On Wednesday, July 24th, 2024 at 4:10 PM, boldsuck <lists@for-privacy.net> wrote:
Hi to all dear exit operators,
If you are interested in applying the exit policy on reload and not by restarting tor please note:
https://gitlab.torproject.org/tpo/core/tor/-/merge_requests/735#note_3051797 Quote David: "Can you give us a sense of how many exit operators use this? If there is a large enough need for this, we can evaluate this for next release but it needs to be for more than 1 operator for such feature."
Related Issue: https://gitlab.torproject.org/tpo/core/tor/-/issues/40676
-- ╰_╯ Ciao Marco!
Debian GNU/Linux
It's free software and it gives you freedom!_______________________________________________ tor-relays mailing list tor-relays@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays
On Dienstag, 30. Juli 2024 18:34:44 CEST George Hartley via tor-relays wrote:
I would definitely want to be able to change my exit policy by just sending a simple "kill -SIGHUP $pid".
So yeah, consider myself interested in this functionality.
But, don't we already have that implemented?
I remember changing my exit policy then doing "systemctl reload tor" and after a few hours, Metrics showed that SSH was now also rejected.
It's not about changing the exit policy via reload. Yes, that's always been possible. It's about killing _existing_ connections that are currently DOSing us. Example: 500K connections from IP 1.2.3.4 You create the reject policy, ExitPolicy reject 1.2.3.4/32:* do a reload and the _existing_ connections are terminated. In order for this to work you have to use the new config option: ReevaluateExitPolicy 1 # (Default 0) And of course a version of Tor in which trinity's commit was merged ;-) -- ╰_╯ Ciao Marco! Debian GNU/Linux It's free software and it gives you freedom!
This is already impossible, as both circuit and concurrent connection DoS both gets detected and the IP in question flagged and blacklisted. Please see the manual on this: https://2019.www.torproject.org/docs/tor-manual.html.en#DoSCircuitCreationEn... All the best, George On Sunday, August 4th, 2024 at 12:30 AM, lists@for-privacy.net <lists@for-privacy.net> wrote:
On Dienstag, 30. Juli 2024 18:34:44 CEST George Hartley via tor-relays wrote:
I would definitely want to be able to change my exit policy by just sending a simple "kill -SIGHUP $pid".
So yeah, consider myself interested in this functionality.
But, don't we already have that implemented?
I remember changing my exit policy then doing "systemctl reload tor" and after a few hours, Metrics showed that SSH was now also rejected.
It's not about changing the exit policy via reload. Yes, that's always been possible.
It's about killing existing connections that are currently DOSing us.
Example: 500K connections from IP 1.2.3.4 You create the reject policy, ExitPolicy reject 1.2.3.4/32:* do a reload and the existing connections are terminated.
In order for this to work you have to use the new config option: ReevaluateExitPolicy 1 # (Default 0)
And of course a version of Tor in which trinity's commit was merged ;-)
-- ╰_╯ Ciao Marco!
Debian GNU/Linux
It's free software and it gives you freedom!_______________________________________________ tor-relays mailing list tor-relays@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays
On Mittwoch, 7. August 2024 14:30:27 CEST George Hartley via tor-relays wrote:
This is already impossible, as both circuit and concurrent connection DoS both gets detected and the IP in question flagged and blacklisted.
No. DoS has been a topic of conversation at nearly all relay meetings for over 2 years. Enkidu and Toralf have developed Tor-ddos IPtables rules for the community. Article10 specifically for Tor exits and trinity has developed the patch. https://gitlab.torproject.org/tpo/core/tor/-/issues/40676 Roger, Mike, Nick and Perry certainly wouldn't have let Trinity develop the feature if the current DoS mitigations in Tor had helped.
Please see the manual on this:
https://2019.www.torproject.org/docs/tor-manual.html.en#DoSCircuitCreationEn abled
This is a client to relay detection only. "auto" means use the consensus parameter. (Default: auto) It _is_ defined in the consensus: https://consensus-health.torproject.org/#consensusparams
Example: 500K connections from IP 1.2.3.4 These are numbers from reality and not fantasy. AFAIK, Article10 and relayon already had 1,000,000 connections per IP!
-- ╰_╯ Ciao Marco! Debian GNU/Linux It's free software and it gives you freedom!
Then these must be targeted attacks, as I have never encountered something like this during 10 years of relay operation under different providers and aliases. Sorry, but the Tor logs that I am seeing suggest that most DoS gets mitigated. As far as I know, the concurrent connection (not circuit!) DoS defense is relatively new, so give the developers some time. Also, any default IPTables rule-set should automatically either reject or just drop connections above a certain threshold. All the best, George On Friday, August 9th, 2024 at 8:59 PM, boldsuck <lists@for-privacy.net> wrote:
On Mittwoch, 7. August 2024 14:30:27 CEST George Hartley via tor-relays wrote:
This is already impossible, as both circuit and concurrent connection DoS both gets detected and the IP in question flagged and blacklisted.
No. DoS has been a topic of conversation at nearly all relay meetings for over 2 years. Enkidu and Toralf have developed Tor-ddos IPtables rules for the community. Article10 specifically for Tor exits and trinity has developed the patch.
https://gitlab.torproject.org/tpo/core/tor/-/issues/40676 Roger, Mike, Nick and Perry certainly wouldn't have let Trinity develop the feature if the current DoS mitigations in Tor had helped.
Please see the manual on this:
https://2019.www.torproject.org/docs/tor-manual.html.en#DoSCircuitCreationEn abled
This is a client to relay detection only. "auto" means use the consensus parameter. (Default: auto) It is defined in the consensus: https://consensus-health.torproject.org/#consensusparams
Example: 500K connections from IP 1.2.3.4
These are numbers from reality and not fantasy. AFAIK, Article10 and relayon already had 1,000,000 connections per IP!
-- ╰_╯ Ciao Marco!
Debian GNU/Linux
It's free software and it gives you freedom!_______________________________________________ tor-relays mailing list tor-relays@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays
On 8/10/24 00:58, George Hartley via tor-relays wrote:
As far as I know, the concurrent connection (not circuit!) DoS defense is relatively new, so give the developers some time.
Well, I do have metrics here in place since a longer time. That and how DDoS prevention works for rejected circuits, is seen in the metrics just after 1st of August (see attached screen shot), not before. These are metrics from my non-exit relays running git HEAD. BTW an interesting example from a tiny VPS Tor (guard+middle) which shows an interesting transient process for that feature. -- Toralf
The DoSCircuitCreation/DoSConnection configs are unrelated to what ReevaluateExitPolicy allows. DoSCircuitCreation/DoSConnection are enacted by guards, to protect themselves, and to some extent the rest of the network, from "noisy IPs" trying to connect to Tor. ReevaluateExitPolicy is not a DoS option, it doesn't take any action automatically. It is only useful on exit nodes, and is roughly the equivalent to running the right tcpkill incantation to kill all already established connection to ip/ports not allowed a new ExitPolicy (but that were allowed when these connections were initiated). On Sat, 10 Aug 2024 at 01:23, George Hartley via tor-relays <tor-relays@lists.torproject.org> wrote:
Then these must be targeted attacks, as I have never encountered something like this during 10 years of relay operation under different providers and aliases.
Sorry, but the Tor logs that I am seeing suggest that most DoS gets mitigated.
As far as I know, the concurrent connection (not circuit!) DoS defense is relatively new, so give the developers some time.
Also, any default IPTables rule-set should automatically either reject or just drop connections above a certain threshold.
All the best, George
On Friday, August 9th, 2024 at 8:59 PM, boldsuck <lists@for-privacy.net> wrote:
On Mittwoch, 7. August 2024 14:30:27 CEST George Hartley via tor-relays wrote:
This is already impossible, as both circuit and concurrent connection DoS both gets detected and the IP in question flagged and blacklisted.
No. DoS has been a topic of conversation at nearly all relay meetings for over 2 years. Enkidu and Toralf have developed Tor-ddos IPtables rules for the community. Article10 specifically for Tor exits and trinity has developed the patch.
https://gitlab.torproject.org/tpo/core/tor/-/issues/40676 Roger, Mike, Nick and Perry certainly wouldn't have let Trinity develop the feature if the current DoS mitigations in Tor had helped.
Please see the manual on this:
https://2019.www.torproject.org/docs/tor-manual.html.en#DoSCircuitCreationEn abled
This is a client to relay detection only. "auto" means use the consensus parameter. (Default: auto) It is defined in the consensus: https://consensus-health.torproject.org/#consensusparams
Example: 500K connections from IP 1.2.3.4
These are numbers from reality and not fantasy. AFAIK, Article10 and relayon already had 1,000,000 connections per IP!
-- ╰_╯ Ciao Marco!
Debian GNU/Linux
It's free software and it gives you freedom!_______________________________________________ tor-relays mailing list tor-relays@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays____________... tor-relays mailing list tor-relays@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays
I am very well aware of that and how it works, I have seen your commit that got merged, and am a C/C++ programmer as well. Nevertheless, this is a feature I wanted anyway, so I could just reload the config and block IP's or even ranges if SSH range / portscans are done using my exit. Right now I reject 22 exits fully, but this might change soon thanks to your patch. Thank you for your contribution :) George On Saturday, August 10th, 2024 at 12:48 PM, trinity pointard <trinity.pointard@gmail.com> wrote:
The DoSCircuitCreation/DoSConnection configs are unrelated to what ReevaluateExitPolicy allows. DoSCircuitCreation/DoSConnection are enacted by guards, to protect themselves, and to some extent the rest of the network, from "noisy IPs" trying to connect to Tor. ReevaluateExitPolicy is not a DoS option, it doesn't take any action automatically. It is only useful on exit nodes, and is roughly the equivalent to running the right tcpkill incantation to kill all already established connection to ip/ports not allowed a new ExitPolicy (but that were allowed when these connections were initiated).
On Sat, 10 Aug 2024 at 01:23, George Hartley via tor-relays tor-relays@lists.torproject.org wrote:
Then these must be targeted attacks, as I have never encountered something like this during 10 years of relay operation under different providers and aliases.
Sorry, but the Tor logs that I am seeing suggest that most DoS gets mitigated.
As far as I know, the concurrent connection (not circuit!) DoS defense is relatively new, so give the developers some time.
Also, any default IPTables rule-set should automatically either reject or just drop connections above a certain threshold.
All the best, George
On Friday, August 9th, 2024 at 8:59 PM, boldsuck lists@for-privacy.net wrote:
On Mittwoch, 7. August 2024 14:30:27 CEST George Hartley via tor-relays wrote:
This is already impossible, as both circuit and concurrent connection DoS both gets detected and the IP in question flagged and blacklisted.
No. DoS has been a topic of conversation at nearly all relay meetings for over 2 years. Enkidu and Toralf have developed Tor-ddos IPtables rules for the community. Article10 specifically for Tor exits and trinity has developed the patch.
https://gitlab.torproject.org/tpo/core/tor/-/issues/40676 Roger, Mike, Nick and Perry certainly wouldn't have let Trinity develop the feature if the current DoS mitigations in Tor had helped.
Please see the manual on this:
https://2019.www.torproject.org/docs/tor-manual.html.en#DoSCircuitCreationEn abled
This is a client to relay detection only. "auto" means use the consensus parameter. (Default: auto) It is defined in the consensus: https://consensus-health.torproject.org/#consensusparams
Example: 500K connections from IP 1.2.3.4
These are numbers from reality and not fantasy. AFAIK, Article10 and relayon already had 1,000,000 connections per IP!
-- ╰_╯ Ciao Marco!
Debian GNU/Linux
It's free software and it gives you freedom!_______________________________________________ tor-relays mailing list tor-relays@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays____________... tor-relays mailing list tor-relays@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays
_______________________________________________ tor-relays mailing list tor-relays@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays
On Samstag, 10. August 2024 14:38:27 CEST George Hartley via tor-relays wrote:
I am very well aware of that and how it works, I have seen your commit that got merged, and am a C/C++ programmer as well.
Nevertheless, this is a feature I wanted anyway, so I could just reload the config and block IP's or even ranges if SSH range / portscans are done using my exit.
You've always been able to do that, my script does it several times a day. Every 10 minutes if necessary. But that has nothing to do with 'ReevaluateExitPolicy 1'. This has now been explained several times in this thread -- ╰_╯ Ciao Marco! Debian GNU/Linux It's free software and it gives you freedom!
On Samstag, 10. August 2024 00:58:29 CEST George Hartley via tor-relays wrote:
Then these must be targeted attacks, as I have never encountered something like this during 10 years of relay operation under different providers and aliases.
Of course, these are targeted attacks and have been extreme since the Ukraine war. If a handful of servers of large relay orgs with hundreds of relays are brought down, it can affect 20-30% of the total exit traffic. The attacks pushed the Junipers to their limits and the entire IX was at risk. In the next few days I can show a live example of targeted hidden service attacks. 2 hs are currently public in the client software and are being attacked. 2 more will be added tomorrow and I am sure that DDoS will start shortly after. You can see this very clearly in PoW metrics on Grafa. https://db4n0nym3.grafana.net/public-dashboards/ 71ad3412bfde44058993dccb07a5e593
Sorry, but the Tor logs that I am seeing suggest that most DoS gets mitigated.
You can't see everything in the Tor logs. Toralf has developed some tools to better monitor relays. Specifically ddos-inbound.sh helped me to develop rules. https://github.com/toralf/torutils
As far as I know, the concurrent connection (not circuit!) DoS defense is relatively new, so give the developers some time.
Also, any default IPTables rule-set should automatically either reject or just drop connections above a certain threshold.
That's why we have developed dynamic IP/NFtables rules for the guards. The whole story began here: https://gitlab.torproject.org/tpo/community/support/-/issues/40093 For Tor exits, the policy reject is of course more effective. And from 10-20G you can no longer use conntrack. Linux does not scale. You can't do much with table inet filter. I drop the most stubborn IPs with ethtool using NIC hardware filtration. The rest with nftables dynamic sets in the ingress hook before prerouting. -- ╰_╯ Ciao Marco! Debian GNU/Linux It's free software and it gives you freedom!
P.S: If this is a client to guard detection only, then why does my exit node also block a significant amount of DoS (I had around the same statistics when my guard probability fraction was still zero, so clearly something is working):
Aug 09 21:08:36 matrix tor[XXX]: Aug 09 21:08:36.000 [notice] Heartbeat: DoS mitigation since startup: 6 circuits killed with too many cells, 865308797 circuits rejected, 691 marked addresses, 0 marked addresses for max queue, 0 same address concurrent connections rejected, 0 connections rejected, 0 single hop clients refused, 0 INTRODUCE2 rejected.
Thank you, George On Friday, August 9th, 2024 at 8:59 PM, boldsuck lists@for-privacy.net wrote:
On Mittwoch, 7. August 2024 14:30:27 CEST George Hartley via tor-relays wrote:
This is already impossible, as both circuit and concurrent connection DoS both gets detected and the IP in question flagged and blacklisted.
No. DoS has been a topic of conversation at nearly all relay meetings for over 2 years. Enkidu and Toralf have developed Tor-ddos IPtables rules for the community. Article10 specifically for Tor exits and trinity has developed the patch.
https://gitlab.torproject.org/tpo/core/tor/-/issues/40676 Roger, Mike, Nick and Perry certainly wouldn't have let Trinity develop the feature if the current DoS mitigations in Tor had helped.
Please see the manual on this:
https://2019.www.torproject.org/docs/tor-manual.html.en#DoSCircuitCreationEn abled
This is a client to relay detection only. "auto" means use the consensus parameter. (Default: auto) It is defined in the consensus: https://consensus-health.torproject.org/#consensusparams
Example: 500K connections from IP 1.2.3.4
These are numbers from reality and not fantasy. AFAIK, Article10 and relayon already had 1,000,000 connections per IP!
-- ╰_╯ Ciao Marco!
Debian GNU/Linux
It's free software and it gives you freedom!_______________________________________________ tor-relays mailing list tor-relays@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays
On Samstag, 10. August 2024 05:25:51 CEST George Hartley via tor-relays wrote:
If this is a client to guard detection only, then why does my exit node also block a significant amount of DoS (I had around the same statistics when my guard probability fraction was still zero, so clearly something is working):
- It says so in the man page from 2019 that you linked above. - And well, an exit is also a guard, an HsDir, Intro, Rdv and whatever else. -- ╰_╯ Ciao Marco! Debian GNU/Linux It's free software and it gives you freedom!
participants (5)
-
boldsuck -
George Hartley -
lists＠for-privacy.net
-
Toralf Förster -
trinity pointard