Guard relay here appears to have come under steadily increasing abuse over the last several months. Belive the two previous threads relate to the same issue:
Failing because we have 4063 connections already // Number of file descriptors
DoS attacks are real
Several times a day a large burst of circuit extends are attempted resulting in log flooding with
[Warning] assign_to_cpuworker failed. Ignoring.
where the above indicates a circuit-launch failed due to a full circuit request queue. Presently the guard runs on an old system lacking AES-NI, and the operation is expansive rather than trivial. Originally thought the events were very brief, but after reducing MaxClientCircuitsPending from a larger value to the default it appears they last between five and ten minutes.
The abuser also contrives to create huge circuit queues, which resulted in an OOM kill of the daemon a couple of days back. Lowered MaxMemInQueues to 1G, set vm.overcommit_memory=2 with vm.overcommit_ratio=X (X such that /proc/meminfo:CommitLimit is comfortably less than physical memory) and now instead of a dameon take-out see
[Warning] We're low on memory. Killing circuits with over-long queues. (This behavior is controlled by MaxMemInQueues.)
Removed 1060505952 bytes by killing 1 circuits; 19k circuits remain alive. Also killed 0 non- linked directory connections.
As you can see the one circuit was consuming all of MaxMemInQueues.
And today this showed up in the middle of a "assign_to_cpuworker failed" blast:
[Warning] Failing because we have Y connections already. . .
Digging into the source, the message indicates ENOMEM/ENOBUFS was returned from an attempt to create a socket. Socket max on the system is much higher than Y so kernel memory exhaustion is the cause. Implication is a burst of client connections associated with the events, but haven't verified that.
An old server was dusted off after a hardware fail and the machine is a bit underpowered, but certainly up to the load that corresponds with the connection speed and assigned consensus weight. AFAICT normal Tor clients experience acceptable performance. The less-than-blazing current hardware illuminates the abuse/attack incidents and inspired the writing of this post.
My relay ran out of connections once and also crashed once so I followed the suggestions in the "DoS attacks are real (probably)" thread and implemented connection limits in my firewall. Everything have run smoothly since.
My only concern is how low I can set the number of connections per IP address. Someone wrote a legit client will only open max 2 tcp connections. I'd like this verified before I lower my limits further.
On 18 Dec 2017, at 02:45, Logforme m7527@abc.se wrote:
My relay ran out of connections once and also crashed once so I followed the suggestions in the "DoS attacks are real (probably)" thread and implemented connection limits in my firewall. Everything have run smoothly since.
My only concern is how low I can set the number of connections per IP address. Someone wrote a legit client will only open max 2 tcp connections. I'd like this verified before I lower my limits further.
A standard tor client will only open one connection to each guard.
But please don't assume there is only one client per IPv4 address. Many networks and even entire countries have a very small IPv4 address allocation. If you restrict it to one connection per IP address, you will be restricting some of the people who need tor the most. And you will push the load onto a smaller set of guards.
Using 256 per IP is probably reasonable.
If we manage to fix some bugs in the socket limits in Tor, we can activate them only when the relay is under heavy load, which is even better.
T
On 19 Dec 2017, at 08:38, Toralf Förster toralf.foerster@gmx.de wrote:
On 12/17/2017 10:24 PM, teor wrote:
Using 256 per IP is probably reasonable.
Is this a rather arbitrary limit or does this limit fit the use of NATed addresses entirely ?
That's an arbitrary safe upper bound.
The number of active connections that can be NATed per IP address is limited by the number of ports: 65535. (Technically, it's 65535 per remote IP address and port, but most NATs don't have that much RAM or bandwidth.)
Also, genuine users behind a NAT would likely have multiple Tor and non-Tor connections open. And spare ports are needed for NAT to manage port churn and the TCP delay wait state on connection close.
To be more precise: * if all 65535 connections on an IP were open to the Tor network, and * the biggest Tor Guard has 0.91% Guard probability[0], then * it would expect to see 597 connections.
Feel free to do the sums for your own guard's probability.
(We are aware of the issue, and we are working on a more permanent fix.)
[0]: https://atlas.torproject.org/#details/9844B981A80B3E4B50897098E2D65167E6AEF1...
T
-- Tim Wilson-Brown (teor)
teor2345 at gmail dot com PGP C855 6CED 5D90 A0C5 29F6 4D43 450C BA7F 968F 094B ricochet:ekmygaiu4rzgsk6n xmpp: teor at torproject dot org ------------------------------------------------------------------------
I don't quite understand the last calculation.
"if all 65535 connections on an IP were open" => I'm guessing you mean ports "the biggest Tor Guard has 0.91% Guard probability" => percentage of all entries into the network handled by this guard
=> 0.91% of all user connections but how many user connections are there at a time?
and then don't understand how probability and ports availability can be combined?
Please elaborate. Thanks
On Mon, 18 Dec 2017 at 23:11 teor teor2345@gmail.com wrote:
On 19 Dec 2017, at 08:38, Toralf Förster toralf.foerster@gmx.de wrote:
On 12/17/2017 10:24 PM, teor wrote:
Using 256 per IP is probably reasonable.
Is this a rather arbitrary limit or does this limit fit the use of NATed
addresses entirely ?
That's an arbitrary safe upper bound.
The number of active connections that can be NATed per IP address is limited by the number of ports: 65535. (Technically, it's 65535 per remote IP address and port, but most NATs don't have that much RAM or bandwidth.)
Also, genuine users behind a NAT would likely have multiple Tor and non-Tor connections open. And spare ports are needed for NAT to manage port churn and the TCP delay wait state on connection close.
To be more precise:
- if all 65535 connections on an IP were open to the Tor network, and
- the biggest Tor Guard has 0.91% Guard probability[0], then
- it would expect to see 597 connections.
Feel free to do the sums for your own guard's probability.
(We are aware of the issue, and we are working on a more permanent fix.)
T
-- Tim Wilson-Brown (teor)
teor2345 at gmail dot com PGP C855 6CED 5D90 A0C5 29F6 4D43 450C BA7F 968F 094B ricochet:ekmygaiu4rzgsk6n xmpp: teor at torproject dot org
tor-relays mailing list tor-relays@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays
On 19 Dec 2017, at 10:10, r1610091651 r1610091651@telenet.be wrote:
I don't quite understand the last calculation.
It's a slightly better approximation that my wild guess.
"if all 65535 connections on an IP were open" => I'm guessing you mean ports
No, I mean: "let's use 65535 as the approximate limit on connections from behind a NAT, even though it could be larger or smaller depending on the bandwidth and RAM available to the NAT"
The technical limit is the number of unique (source IP, source port, destination IP, destination port) tuples, or 65535 per source IP to every unique Tor ORPort. But that's hardly ever reached.
"the biggest Tor Guard has 0.91% Guard probability" => percentage of all entries into the network handled by this guard
=> 0.91% of all user connections but how many user connections are there at a time?
Let's assume there are at most 65535 connections at once time from every source IP into the Tor network.
and then don't understand how probability and ports availability can be combined?
If there are 65535 connections open from a source IP, and they all go to Tor Guards, and the clients weight connections according to Guard probability, then the largest guard will have 0.91% of 65535 connections, or approximately 597.
Most guards would see 10-200 connections per IP.
T
On Mon, 18 Dec 2017 at 23:11 teor teor2345@gmail.com wrote:
On 19 Dec 2017, at 08:38, Toralf Förster toralf.foerster@gmx.de wrote:
On 12/17/2017 10:24 PM, teor wrote:
Using 256 per IP is probably reasonable.
Is this a rather arbitrary limit or does this limit fit the use of NATed addresses entirely ?
That's an arbitrary safe upper bound.
The number of active connections that can be NATed per IP address is limited by the number of ports: 65535. (Technically, it's 65535 per remote IP address and port, but most NATs don't have that much RAM or bandwidth.)
Also, genuine users behind a NAT would likely have multiple Tor and non-Tor connections open. And spare ports are needed for NAT to manage port churn and the TCP delay wait state on connection close.
To be more precise:
- if all 65535 connections on an IP were open to the Tor network, and
- the biggest Tor Guard has 0.91% Guard probability[0], then
- it would expect to see 597 connections.
Feel free to do the sums for your own guard's probability.
(We are aware of the issue, and we are working on a more permanent fix.)
T
-- Tim Wilson-Brown (teor)
teor2345 at gmail dot com PGP C855 6CED 5D90 A0C5 29F6 4D43 450C BA7F 968F 094B ricochet:ekmygaiu4rzgsk6n xmpp: teor at torproject dot org
tor-relays mailing list tor-relays@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays
tor-relays mailing list tor-relays@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays
El 19/12/17 a las 11:13, teor escribió: …
If there are 65535 connections open from a source IP, and they all go to Tor Guards, and the clients weight connections according to Guard probability, then the largest guard will have 0.91% of 65535 connections, or approximately 597.
Most guards would see 10-200 connections per IP.
…
My relay B33BFA9AA0005730C1C0E8F7E6F53CF3C5716BD6 is not currently tagged as Guard, and I am seeing more than twenty IPv4s with more than 10 connections, and one with 147. Should that be considered normal for a non-guard relay?
Cheers,
-- Santiago
On Wed, December 20, 2017 12:10 pm, Santiago wrote: ...
My relay B33BFA9AA0005730C1C0E8F7E6F53CF3C5716BD6 is not currently tagged as Guard, and I am seeing more than twenty IPv4s with more than 10 connections, and one with 147. Should that be considered normal for a non-guard relay?
Cheers,
-- Santiago
147 is a bit high for a non-exit, non-guard, for a single IP. check https://atlas.torproject.org/ and see if this IP is part of Tor network.
cheers.
-- x9p | PGP : 0x03B50AF5EA4C8D80 / 5135 92C1 AD36 5293 2BDF DDCC 0DFA 74AE 1524 E7EE
On 20 Dec 2017, at 16:39, x9p wrote:
On Wed, December 20, 2017 12:10 pm, Santiago wrote: ...
My relay B33BFA9AA0005730C1C0E8F7E6F53CF3C5716BD6 is not currently tagged as Guard, and I am seeing more than twenty IPv4s with more than 10 connections, and one with 147. Should that be considered normal for a non-guard relay?
147 is a bit high for a non-exit, non-guard, for a single IP. check https://atlas.torproject.org/ and see if this IP is part of Tor network.
My relay is regularly struggling a bit nowadays, with some source IP's crossing over the 1000 connections, but quite a few at 50-100. The one with 1000 connections, and for some random IP's none of their IP's being listed as an Tor node on atlas. Seems to be a lot of IP's out of 54.36.51.0/24 that tend to open a lot of sessions. Whereby the ones checked are not on Atlas.
At some point the entire conntrack table was full and OOM kills for the tor process. This only left me to put in some connection limits. Despite being advices against. I currently have: 200 connections per /24, if that's used then at least allow 24 connections per /32.
I'm currently running with 6600 connections just fine; when it crosses the 15k it becomes troublesome.
Now blocking some connections might be far-far from ideal, but better ~6000 connections served with bandwidth then to remove my relay from the tor network in my view.
That said it would be good if the Tor program itself would have some protections, to the extend possible, with the current protocol. For instance dropping clients (source IP's) that frequently connect but are not behaving. I understand this might have it's implications when under censorship/censorship countermeasures.
On 12/20/2017 04:39 PM, x9p wrote:
My relay B33BFA9AA0005730C1C0E8F7E6F53CF3C5716BD6 is not currently tagged as Guard, and I am seeing more than twenty IPv4s with more than 10 connections, and one with 147. Should that be considered normal for a non-guard relay?
Cheers,
-- Santiago
147 is a bit high for a non-exit, non-guard, for a single IP. check https://atlas.torproject.org/ and see if this IP is part of Tor network.
? IMO relays don't open more than 1 connection to another relay.
Same shit here. It looks like this:
https://i.imgur.com/rokqahz.png https://i.imgur.com/rokqahz.png
Markus
“Cheery was aware that Commander Vimes didn't like the phrase 'The innocent have nothing to fear', believing the innocent had everything to fear, mostly from the guilty but in the longer term even more from those who say things like 'The innocent have nothing to fear'.”
― Terry Pratchett, Snuff
On 20. Dec 2017, at 17:18, Toralf Förster toralf.foerster@gmx.de wrote:
On 12/20/2017 04:39 PM, x9p wrote:
My relay B33BFA9AA0005730C1C0E8F7E6F53CF3C5716BD6 is not currently tagged as Guard, and I am seeing more than twenty IPv4s with more than 10 connections, and one with 147. Should that be considered normal for a non-guard relay?
Cheers,
-- Santiago
147 is a bit high for a non-exit, non-guard, for a single IP. check https://atlas.torproject.org/ and see if this IP is part of Tor network.
? IMO relays don't open more than 1 connection to another relay.
-- Toralf PGP C4EACDDE 0076E94E
tor-relays mailing list tor-relays@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays
On 12/18/2017 11:10 PM, teor wrote:
The number of active connections that can be NATed per IP address is limited by the number of ports: 65535. (Technically, it's 65535 per remote IP address and port, but most NATs don't have that much RAM or bandwidth.)
Also, genuine users behind a NAT would likely have multiple Tor and non-Tor connections open. And spare ports are needed for NAT to manage port churn and the TCP delay wait state on connection close.
To be more precise:
- if all 65535 connections on an IP were open to the Tor network, and
- the biggest Tor Guard has 0.91% Guard probability[0], then
- it would expect to see 597 connections.
good example, Thx teor !
Hi everybody
- if all 65535 connections on an IP were open to the Tor network, and
- the biggest Tor Guard has 0.91% Guard probability[0], then
- it would expect to see 597 connections.
Sorry if this is a silly question, but do we know if these are Tor clients connecting our guards? We see many connects but not much circuits.
Could someone get state by: openssl s_client -connect tor-guard-ip:tor-guard-orport -tls1 and establish awfull many tls connects without any circuit ?
In this case there are like 64k outbound ports available and the necessary memory/cpu for openssl is much lower than for a regular Tor client.
On 21 Dec 2017, at 06:48, Felix zwiebel@quantentunnel.de wrote:
Hi everybody
- if all 65535 connections on an IP were open to the Tor network, and
- the biggest Tor Guard has 0.91% Guard probability[0], then
- it would expect to see 597 connections.
Sorry if this is a silly question, but do we know if these are Tor clients connecting our guards? We see many connects but not much circuits.
Some of us have analysed the details of this attack on our relays. The clients perform SSL, the Tor link protocol, and parts of the circuit protocol. Are they real Tor clients? Possibly not.
We're working on a fix, please see this email for details: https://lists.torproject.org/pipermail/tor-relays/2017-December/013881.html
T
My relay ran out of connections once and also crashed once so I followed the suggestions in the "DoS attacks are real (probably)" thread and implemented connection limits in my firewall. Everything has run smoothly since.
I missed this thread, thank you for highlighting it!
My only concern is how low I can set the number of connections per IP address. Someone wrote a legit client will only open max 2 tcp connections. I'd like this verified before I lower my limits further.
Two connection limit is fine for single-IP clients, but will penalize multiple clients operating behind NAT IPs. I've decided that's too bad for them for the moment. . .
Limiting connections-per-IP fixes it. I set
-m connlimit --connlimit-above 2 --connlimit-mask 32 -j DROP
and obtained good mitigation. The attacker relies on opening tons of connections and this simple rules squashes it. Rule accumulated 15 million hits over a short span.
I have kept an eye on the number of peer-relay connections and client connections for a long time and the client connection count has been artificially high of late. With the above rule it went right back to the usual level.
The limit can be higher than two and it should work as well--the rate of new connections appears to be critical to the attack. Possibly the low performance CPU here lacking AES hardware mitigates it since each connection appears to require an onionskin calculation, and whatever old-connection cleanup logic exists in Tor easily keeps pace. This has been going on for months and only became a problem now with the attacker enhancing it to somehow queue huge amounts of data on circuits--but per my initial post that's simple to mitigate.
tor-relays@lists.torproject.org
-
Felix -
Logforme -
niftybunny -
r1610091651 -
Santiago -
starlight.2017q4＠binnacle.cx -
Stijn Jonker -
teor -
tor -
Toralf Förster -
x9p