Upgraded exit to 0.3.3.3 and now seeing a curious CPU saturation attack. Whatever the cause, result is the main event-worker thread going from a normal load level of about 30%/core to 100%/core and staying there for about 30 seconds; then CPU consumption declines back to 30%. Gradual change on ascent and decent. Another characteristic is egress traffic slightly higher than ingress traffic, perhaps 3-4%, where normally egress and ingress flows match precisly. Checked browsing via the node and performance seems fine--no obvious degradation. Elevated NTor circuit creation rates as-of the last heartbeat, from roughly 300k to 700k per-report, but not extreme (at least in a relative sense since late December). Anyone else observed this? Have any idea how the attack works? Captured a debug-level log of a cycle from normal load to full-on-attack but won't have time to analyzed it for a couple of weeks.
On 03/04/2018 07:41 PM, Dhalgren Tor wrote:
the main event-worker thread going from a normal load level of about 30%/core to 100%/core and staying there for about 30 seconds; I do wonder if this is just the normal behaviour when - IIRC correctly - consensus documents are compressed before sending.
-- Toralf PGP C4EACDDE 0076E94E
On Sun, Mar 4, 2018 at 7:06 PM, Toralf Förster <toralf.foerster@gmx.de> wrote:
On 03/04/2018 07:41 PM, Dhalgren Tor wrote:
the main event-worker thread going from a normal load level of about 30%/core to 100%/core and staying there for about 30 seconds; I do wonder if this is just the normal behaviour when - IIRC correctly - consensus documents are compressed before sending.
No chance whatsoever. Relay runs for months-on-end never exceeding 40% CPU. Have seen the same or a similar attack, twice before I believe under 0.2.9.14. Just realized the ISP added some bugs to their data graphs: in this case _ingress_ traffic is 3-4% higher than egress (they reversed the labels along with breaking long-term historical). Earlier observed a similar attack where _egress_ traffic was 10-15% higher than ingress traffic. What's interesting here is the crypto-worker threads are near zero (normal) in contrast to circuit-extend attacks where the crypto threads peg at 100%. Did see one brief, intense crypto- worker CPU spike today but it's not characteristic of this event in general.
Found other ones: December 24 where egress was much higher then ingress (but crypto-workers were pegged, not main thread). December 28 & 29, attack like today. Feburary 1 & 2, like today with ingress higher than egress. In today's and the latter-two above the main event thread was pegged either continuously or intermittently with ingress higher than egress. Just looked again and see all threads, crypto-worker and main-event pegging episodically.
Dear Dhalgren Tor, I have been running a middle node with the ingress connexions being 5-10% fewer than egress connexions since it started about 11 months ago. changing Tor versions twice has not made any difference in this, the current version being 0.3.2.10 that came out yesterday. But I'm only running only about 2,000 on each side (19xx ingress, 21xx egress). I assumed that was normal... Sent with ProtonMail Secure Email. ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐ On March 4, 2018 3:08 PM, Dhalgren Tor <dhalgren.tor@gmail.com> wrote:
Found other ones: December 24 where egress was much higher then
ingress (but crypto-workers were pegged, not main thread). December
28 & 29, attack like today. Feburary 1 & 2, like today with ingress
higher than egress.
In today's and the latter-two above the main event thread was pegged
either continuously or intermittently with ingress higher than egress.
Just looked again and see all threads, crypto-worker and main-event
pegging episodically.
tor-relays mailing list
tor-relays@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays
participants (3)
-
Dhalgren Tor -
Toralf Förster -
torix＠protonmail.com