[tor-relays] max TCP interruption before Tor circuit teardown?

[Gordon Morehouse]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

Tor CPU usage has dropped way back to 15-40% in the last few minutes,
even as I was increasing my total inbound connection limit to 150
connections.

This is a hell of a log line on a Raspberry Pi, which also just popped
out:

Oct 31 10:13:49.000 [notice] Circuit handshake stats since last time:
61533/218956 TAP, 30/30 NTor.

Wow.

Best,
- -Gordon M.



Gordon Morehouse:
> I've just seen the most amazing headshot of my Tor relay by a
> sudden massive SYN flood yet. I was online and started noticing
> problems with DNS on my local router. I checked my so-called
> monitoring setup, a window with a permanent ping to my router, and
> noticed a lot of timeouts. Obviously, that means trouble.
> 
> Checked my Raspberry Pi tor relay setup, and there was an
> incredible SYN flood just starting. I have attached an image where
> the vertical scale reaches up to 5 megabits per second and where is
> column is two seconds. This is absolutely not established tor
> connection behavior. I don't know what *all* of it is, since once
> the Tor daemon dies, the SYN traffic seems to be steady at about
> 50KB/sec (of *just* SYNs inbound, and 100+KB/sec of outbound ICMP
> port unreachable packets).  But that huge tsunami marks when the
> flood / circuit creation storm really got going.
> 
> My relay crashed faster than I've ever seen it crash before, even
> with my newer protections in place. In under 5 minutes the out of
> memory killer reaped Tor.  In previous situations, I've observed
> during floods that Tor's share of physical memory doesn't seem to
> increase. I could be wrong about that, but I think the thing eating
> all the RAM is TCP open/half-open sockets and/or associated tables
> in the Linux kernel - once RAM pressure becomes too intense, Tor is
> just the biggest thing around, so the oom-killer picks it and bam.
> 
> The truly amazing and disturbing thing is that it's an hour and a
> half later now, and my router is still under extreme load from the
> incoming SYN packets. It hasn't yet crashed.
> 
> In the meantime I added an iptables rule right under the
> "ESTABLISHED" rule suggested by David Serrano:
> 
> 
> Chain INPUT (policy ACCEPT) target     prot opt source
> destination
> 
> ACCEPT     all  --  anywhere             anywhere
> state ESTABLISHED
> 
> DROP       tcp  --  anywhere             anywhere tcpflags:
> FIN,SYN,RST,ACK/SYN #conn src/0 > 75
> 
> SYN_THROTTLE  tcp  --  anywhere             anywhere multiport
> dports 31923,31924 state NEW
> 
> 
> (those weird params are from a connlimit suggestion I found for 
> limiting the total number of TCP connections which may be handled
> over a chain.)  I started off at 50, and am now up to 100.  This
> is obviously a stopgap solution for an ongoing event, but it
> suggests some further ways that slower single-board computers can
> be made to weather such storms, possibly without (see earlier
> discussion on this thread) using fail2ban at all, which is very
> inefficient.
> 
> What's quite alarming is that when I raise the limit a bit, to get
> the restarted Tor relay better connected, the SYN flood logs go
> crazy for a minute or two before instantaneously stopping when, I
> presume, the connection limit has been reached.  Since the dropped
> packets above the global inbound connection limit are not logged,
> the sudden start/stop of the SYN flood logging (in the SYN_THROTTLE
> chain, they're logged) tells me I am still under intense SYN
> flood.
> 
> After adding connection limits on the Tor box, my router recovered
> and is seeing ping times, ballpark 2x normal (0.8-1.2ms is normal,
> now it's more like 1.0-2.0,s), but things are working handily.  I
> have also been able to connect to other services through the Tor
> relay again, with considerable difficulty.
> 
> I notice that Tor is consuming all available CPU, even though it
> is, for the moment, relaying a pretty consistent 50-80KB/sec.  I
> suspect that this is mostly circuit creation requests coming in
> over established Tor connections, which Tor is ... processing, I
> don't know how, but unless there's been some turnover and they get
> lucky, once another peer attempts a TCP/TLS handshake, its packets
> are likely to be dropped.  This is probably not ideal.
> 
> As long as the Raspberry Pi manages to stay up and keep logging
> I'll have all the data to go through later. (I already captured a
> lot.) I also have the logs from the other incident that I caught
> and watched in real time. I'm planning to do an analysis on the
> number of IPs involved, whether they are Tor relays are not, and
> other interesting things that can be gleaned from the logs. I
> promise some graphs and charts, punch and pie not so much.
> Unfortunately my life is quite busy right now so the report may
> take a while although it's kind of a high priority thing for me at
> the moment.  This is pretty crazy.  I can't verify it, but my
> suspicion is this is happening when I get my Stable flag (I have no
> idea if I'd gotten it back this morning or not) or shortly
> thereafter.
> 
> Node is VastCatbox, flood started around 8am Pacific.
> 
> Best, -Gordon M.
> 
> 
> 
> 
> _______________________________________________ tor-relays mailing
> list tor-relays at lists.torproject.org 
> https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays
> 

- -- 
Sent from my thing that sends email.
-----BEGIN PGP SIGNATURE-----

iQEcBAEBCgAGBQJScpDnAAoJED/jpRoe7/ujHW0H/0ylbsd2e1r5FighEgfzCxMd
aCoSsJkv6GH1H+P/TWgGHDS7H2f6DxuKgoWehTp/S9hO5IhYJrxQ2+451tEsVKH+
fCM3GvJZgZHCtjtBpOZA64Avna6KW4AC7siN/LWiYYsf8tSfrFvmMbV0j7zUNJak
y2Vrz3+GcxauZ2o7lWkRAih4dQQQjtuFAyzv66hwtA7jsfx9T2hko1D4LIQfCQWE
gDPogVjxTvJCyG731VrY2ev4ileOwPFbIzWlcTqCBg69gDwBxwMlaUgsrgNcShjl
LLolqsGFRn0NIk36LBy8X4kmPpVbkZpIKWZitkUaPKASsgAnbomb3ztEENQSGB8=
=d6rN
-----END PGP SIGNATURE-----