Hello friendly relay operators,
Another day, another weird thing with the Tor network. This time we have some jerk bombing the directory authorities with directory fetches, and doing it via exits: https://lists.torproject.org/pipermail/network-health/2021-January/000661.ht...
The network is mostly holding together, but I wouldn't say it is pretty.
One of the long-term fixes will be ticket #2667: https://gitlab.torproject.org/tpo/core/tor/-/issues/2667 where exit relays refuse to let users connect back into the Tor network.
David and I made a branch this evening that implements #2667, and it could use some testing. If you're comfortable building your exit relay from a git branch, please do, and let us know how it goes. It is the "ticket2667" branch on either https://git.torproject.org/user/arma/tor or https://gitlab.torproject.org/arma/tor/
And if your relay is currently using 100% cpu and/or way more bandwidth than usual, you might be especially excited to try out this patch. :)
When the defense triggers, you will see an info-level log line like "%s tried to connect back to a known relay address. Closing." (where %s is the destination, so don't get upset at them. :)
You can let us know how it's going either by mail just to me, or by a reply on the list, whichever you prefer. Once we know that you're running the branch, we can also probe your relay remotely to verify that it is correctly refusing those connections.
Thanks! --Roger
Have a tor exit running in the US ; fingerprint is 3DE567C1350C0E858C6147AECB06EA9B3EAF3261 and OR address is 71.174.105.126:9001
Just built and launched the ticket-2667 branch; came up as:
[notice] Tor 0.4.6.0-alpha-dev running on Linux with Libevent 2.0.21-stable, OpenSSL 1.0.1t, Zlib 1.2.8, Liblzma N/A, Libzstd N/A and Glibc 2.19 as libc.
I'll monitor the log notices but feel free to probe to test. Thanks!
Roger Dingledine mailto:arma@torproject.org January 28, 2021 at 1:40 AM Hello friendly relay operators,
Another day, another weird thing with the Tor network. This time we have some jerk bombing the directory authorities with directory fetches, and doing it via exits: https://lists.torproject.org/pipermail/network-health/2021-January/000661.ht...
The network is mostly holding together, but I wouldn't say it is pretty.
One of the long-term fixes will be ticket #2667: https://gitlab.torproject.org/tpo/core/tor/-/issues/2667 where exit relays refuse to let users connect back into the Tor network.
David and I made a branch this evening that implements #2667, and it could use some testing. If you're comfortable building your exit relay from a git branch, please do, and let us know how it goes. It is the "ticket2667" branch on either https://git.torproject.org/user/arma/tor or https://gitlab.torproject.org/arma/tor/
And if your relay is currently using 100% cpu and/or way more bandwidth than usual, you might be especially excited to try out this patch. :)
When the defense triggers, you will see an info-level log line like "%s tried to connect back to a known relay address. Closing." (where %s is the destination, so don't get upset at them. :)
You can let us know how it's going either by mail just to me, or by a reply on the list, whichever you prefer. Once we know that you're running the branch, we can also probe your relay remotely to verify that it is correctly refusing those connections.
Thanks! --Roger
tor-relays mailing list tor-relays@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays
Unless you already ruled out that hypothesis by looking at the attack distribution by source IP:
If dir auths (some or all) are willing to share (privately or publicly) the distribution of attack load (frequency, bandwidth, ...) by exit source IP in total or relative values I can correlate this data to strengthen a hypothesis that malicious/suspicious exits are involved to a greater extend than well-known long term exits. That could mean that they are not (exclusively) attacking via but _from_ servers that also happen to run tor exits.
From another angle this is an interesting precedence
because the tor project uses it's access to protect dir auths from exit relays. Why is that interesting? Because no one else that gets attacked via exit relays has that "luxury" to "filter" it at the "source" (exits).
On Fri, Jan 29, 2021 at 12:34:28AM +0100, nusenu wrote:
If dir auths (some or all) are willing to share (privately or publicly) the distribution of attack load (frequency, bandwidth, ...) by exit source IP in total or relative values I can correlate this data to strengthen a hypothesis that malicious/suspicious exits are involved to a greater extend than well-known long term exits.
I'll send you out-of-band a little snapshot of requests from relay IP addresses -- 160k requests over a 24 minute period from yesterday early evening.
At one point later in the evening I was getting several tens of millions of requests per hour. That's when I started to realize that exit relay operators were probably seeing this increased load too.
That could mean that they are not (exclusively) attacking via but _from_ servers that also happen to run tor exits.
Well, there are definitely other addresses -- the overload from last week was non-relay addresses, and that's still going.
It's possible that there are exits that are generating more than their "fair" share of requests. I didn't see that pattern obviously happening, and confirming it would be complicated by the fact that some relays probably have less or more congestion, which would cause the attacks to be more efficient or less efficient through them.
We had a long debugging session in #tor-dev on irc last night, and there will be more of those as we proceed. We've found a bunch of short-term fragile distinguishers, which we could use to block the "bad" traffic right now, but which wouldn't hold up if the bad traffic adapts a bit.
More broadly, we're trying to walk the fine line between doing our analysis and patches in public (yay transparency), vs being aware that whoever is doing this is probably reading these threads too. The destination we want is that we have defenses that are robust to the attacker knowing about them, but things will be a bit bumpy as we get to that destination.
I'm also trying to make sure everybody continues to think about the privacy side -- the directory authorities or fallbackdirs don't know what paths clients build, or what destinations they reach with them, but they can know at what timestamps some IP addresses seemed to be using Tor. And like most things, that information is better private by default.
From another angle this is an interesting precedence because the tor project uses it's access to protect dir auths from exit relays. Why is that interesting? Because no one else that gets attacked via exit relays has that "luxury" to "filter" it at the "source" (exits).
Actually, the #2667 patch protects all relays from exit relays. That is, exit relays will decline to exit to known ORPorts or DirPorts of any relay. There are two benefits here: (a) people can't do circuit-level amplification attacks (happy to elaborate on these once the defense is more in place), and (b) people can't create directory requests which blend with the directory requests that the relay itself does.
These two issues are Tor-specific, and the second one is an especially good argument I think, because the relay is reserving for itself the ability to make its dir connections in a way that the destination can know that the relay is the one making the connection. (Another option would be to add more authentication to the connection, but most ways of doing that are heavier-weight, not lighter-weight.)
--Roger
Roger Dingledine:
On Fri, Jan 29, 2021 at 12:34:28AM +0100, nusenu wrote:
If dir auths (some or all) are willing to share (privately or publicly) the distribution of attack load (frequency, bandwidth, ...) by exit source IP in total or relative values I can correlate this data to strengthen a hypothesis that malicious/suspicious exits are involved to a greater extend than well-known long term exits.
I'll send you out-of-band a little snapshot of requests from relay IP addresses -- 160k requests over a 24 minute period from yesterday early evening.
I've looked at the data and found no clear indicators to support a hypothesis that malicious/suspicious exit operators are involved to a greater extend then others, but I'm not sure if 24 minutes is enough to draw any conclusions. 24 hours would probably more suitable.
Expected request frequency for exit IPs would also be interesting when looking and evaluating such data.
At one point later in the evening I was getting several tens of millions of requests per hour. That's when I started to realize that exit relay operators were probably seeing this increased load too.
Did any exit operator actually see increased load on their exits?
kind regards, nusenu
Roger Dingledine wrote:
Hello friendly relay operators,
Another day, another weird thing with the Tor network. This time we have some jerk bombing the directory authorities with directory fetches, and doing it via exits: https://lists.torproject.org/pipermail/network-health/2021-January/000661.ht...
The network is mostly holding together, but I wouldn't say it is pretty.
One of the long-term fixes will be ticket #2667: https://gitlab.torproject.org/tpo/core/tor/-/issues/2667 where exit relays refuse to let users connect back into the Tor network.
David and I made a branch this evening that implements #2667, and it could use some testing. If you're comfortable building your exit relay from a git branch, please do, and let us know how it goes. It is the "ticket2667" branch on either https://git.torproject.org/user/arma/tor or https://gitlab.torproject.org/arma/tor/
And if your relay is currently using 100% cpu and/or way more bandwidth than usual, you might be especially excited to try out this patch. :)
When the defense triggers, you will see an info-level log line like "%s tried to connect back to a known relay address. Closing." (where %s is the destination, so don't get upset at them. :)
You can let us know how it's going either by mail just to me, or by a reply on the list, whichever you prefer. Once we know that you're running the branch, we can also probe your relay remotely to verify that it is correctly refusing those connections.
Thanks! --Roger
Hello,
Indeed the defense is triggered more often than I expected. Very nice.
I tried to read on that ticket that's a decade old but didn't understand what was the final resolution.
This defense is only implemented at the Exit relay, and it blocks all the relay:ORPort / DirPort combinations that exit knows about according to its version of the consensus?
What happens if the abuser has a different valid consensus (newer/older) that has some relays which are not known by the Exit? Or if the abuser uses for re-entry a relay configured with `PublishServerDescriptor 0`? How does it treat bridges?
And most important, does this means that an Exit will no longer try to connect to other middle relay? At this moment can't an exit relay (of course with a small probability) be used as a middle or entry? Won't this become a mess if we have 80% of relays Exits and thus used more as 1st or 2nd hops, or in a perfect future where 100% or relays are all exits?
On Thu, Feb 04, 2021 at 01:20:58AM +0200, s7r wrote:
Indeed the defense is triggered more often than I expected. Very nice.
Btw, a better version of the #2667 patch is now included in all of the current Tor releases: 0.3.5.13, 0.4.3.8, 0.4.4.7, 0.4.5.5-rc.
So if you are still trying out my experiment patch, thank you, but now please stop doing that and move to one of the actual releases. :)
This defense is only implemented at the Exit relay, and it blocks all the relay:ORPort / DirPort combinations that exit knows about according to its version of the consensus?
The design that we decided on was "block relay:ORPort, dirauth:ORPort, and dirauth:DirPort". That is, it doesn't try to block relay:DirPort connections.
We chose that balance because if we'd blocked all relay dirports too, we would have broken relay dirport reachability tests, because the way those tests work is by building an exit circuit and then "exiting" back to the relay's dirport to see if it works.
Ultimately I expect we're going to phase out the concept of a dirport on non-dir-auth relays, since they mostly go unused so it's just yet another thing to maintain that isn't worth it. But now we can do that phase-out completely separate from this topic.
What happens if the abuser has a different valid consensus (newer/older) that has some relays which are not known by the Exit?
If there are edge cases where different sides have different knowledge, then those edge cases will 'get through'. But hopefully they'll be rare, or even if they're not rare, hopefully they will be low-impact.
Or if the abuser uses for re-entry a relay configured with `PublishServerDescriptor 0`? How does it treat bridges?
They are all allowed.
In fact, one of the Tor use cases we're going to break here is people who torify all their traffic (at the network or host level) and then run Tor Browser also. The suggested workaround for them is that if they really need to do this, they should configure their Tor Browser to use a bridge. There will be a bunch of confused users who don't know they're doing this, and don't know why it broke, though.
I've been working on https://gitlab.torproject.org/tpo/core/tor/-/issues/40271 to do a little bit toward that explanation side.
And most important, does this means that an Exit will no longer try to connect to other middle relay? At this moment can't an exit relay (of course with a small probability) be used as a middle or entry? Won't this become a mess if we have 80% of relays Exits and thus used more as 1st or 2nd hops, or in a perfect future where 100% or relays are all exits?
No, exiting is different from extending. Exit relays can still extend to other relays -- this is a part of the Tor protocol that lets a relay make a TLS connection with another relay. The difference is in who the "ends" of that TLS connection are: in the extend case, the ends are the two relays. In the 'exit and reenter' case, one end is the client and the other is the destination relay.
--Roger
tor-relays@lists.torproject.org
-
Chris Dagdigian -
nusenu -
Roger Dingledine -
s7r