[tor-relays] DoS attacks on multiple relays

[Scott Bennett]

null <null at omuravpn.com> wrote:

> We're experiencing what looks like a DoS attack on multiple relays in our
> family:
>
> https://atlas.torproject.org/#search/family:CBEAE10CBBB86C51059246B2EF92EB2CB4E111BC
>
> The relays are currently running Tor 0.3.1.9 on Linux kernel 4.4.0
> (although when the problem started the relays were running Tor 0.3.1.8).
>
> The attack knocked 3 of 6 relays offline overnight. By the time we looked
> at logs, the Tor service had stopped and this was the last line in the log:
>
> "Tor[xyz]: Failing because we have 16351 connections already. Please read
> doc/TUNING for guidance."
>
> The attack is still ongoing. When it's happening, the number of connections
> rises very rapidly, until the attack succeeds in stopping the service.
>
> $ ss -s
> Total: 15855 (kernel 0)
> TCP:   24520 (estab 23969, closed 305, orphaned 31, synrecv 0, timewait
> 261/0), ports 0
>
> Transport Total     IP        IPv6
> *   0         -         -
> RAW   0         0         0
> UDP   8         4         4
> TCP   24215     24213     2
> INET   24223     24217     6
> FRAG   0         0         0
>
> ... and only a few seconds later:
>
> $ ss -s
> Total: 12120 (kernel 0)
> TCP:   27389 (estab 20026, closed 1906, orphaned 45, synrecv 0, timewait
> 1587/0), ports 0
>
> Transport Total     IP        IPv6
> *   0         -         -
> RAW   0         0         0
> UDP   8         4         4
> TCP   25483     25481     2
> INET   25491     25485     6
> FRAG   0         0         0
>
> That's obviously much larger than the normal number of connections, more
> than we've ever seen, and seems like more connections than would be needed
> for a relay.
>
     What you are seeing is most likely the same phenomenon brought up on
this list repeatedly over at least the last decade or so.  That phenomenon
is providing HSDir service, or perhaps a rendez-vous point, for a popular
hidden service.  As soon as your node is associated with that hidden service
and that association begins to be distributed by the HSDir population to
clients looking for that hidden service, the number of connections to your
node will increase fairly rapidly to a level corresponding to that hidden
service's level of popularity.  If you don't like it, you can set

HidServDirectoryV2	0

which will stop clients from trying to get hidden service descriptors from
your node, which will eliminate most of the bursts of connections you're
seeing, but will not prevent your node from being a rendez-vous point because
every tor relay is expected to provide that function as part of the relay
protocols.

> We have file descriptors (/proc/sys/fs/file-max) set to 64000, but it looks
> like Tor sets MAX_FILEDESCRIPTORS to 16384 per /etc/init.d/tor:
>
>   elif [ "$system_max" -gt "40000" ] ; then
>     MAX_FILEDESCRIPTORS=16384
>
> Surely that is high enough for normal service?

     If by normal you mean "low traffic", then yes, it's probably enough.
However, that's really not very high in a general sense.  Consider also that
some installed packages place high demands upon the supply of file descriptors.
(E.g., I gather you do not have a graphics port/package called piglit installed
on your system, which recommends at least 50000 be available for its runs, so
I have

kern.maxfiles="50000"

in /boot/loader.conf on my FreeBSD system.  I don't think I can recall tor
ever handling many more than 5000 (i.e., 10% of that figure) at one time on my
low-traffic node.)  The faster, larger-capacity tor nodes often have
considerably higher settings to keep tor from exhausting the fd limits on
those hosts.
>
> We haven't started looking into where the traffic is coming from or other
> characteristics. We are wondering if: 1) this is a known attack, 2) if
> other operators are experiencing it, 3) if there are any ideas for
> mitigating it, and 4) if any additional information would be helpful.
>
     Other than refusing to be a hidden service directory server, there is
probably nothing to be done about it.  Adjust your settings accordingly, along
with your expectations. :-)


                                  Scott Bennett, Comm. ASMELG, CFIAG
**********************************************************************
* Internet:   bennett at sdf.org   *xor*   bennett at freeshell.org  *
*--------------------------------------------------------------------*
* "A well regulated and disciplined militia, is at all times a good  *
* objection to the introduction of that bane of all free governments *
* -- a standing army."                                               *
*    -- Gov. John Hancock, New York Journal, 28 January 1790         *
**********************************************************************