[tor-relays] DoS attacks on multiple relays

[Scott Bennett]

null <null at omuravpn.com> wrote:

> @ x9p:
>
> > # netstat -tupan | grep ESTABLISHED | grep /tor | awk '{print $5}' | awk
> > -F: '{print $1}' | awk -F. '{print $1"."$2"."$3}' | sort | uniq -c | sort
> > | egrep -v '      1 |      2 |      3 '
> >
> > with this information in hand, double the max of it (mine was 10
> > connections from 188.214.30.0/24):
> >
> >      10 188.214.30
> >
> > iptables -A INPUT -i eth0 -p tcp -m connlimit --connlimit-above 20
> > --connlimit-mask 24 -j REJECT --reject-with tcp-reset
>
> Thank you! This was extremely helpful.
>
> In our case we found a handful of IPs that had *thousands* of
> concurrent connections on several of our relays. The offending IPs
> were not in the consensus. After restarting the Tor service, these
> suspect connections come back rapidly, again across several of our
> relays. Since our relays are all in the same declared family, it is
> very difficult to see how this traffic is legitimate. If it's valid

     But it could be legitimate.  As has been discussed here previously,
they may be connections from a relay that actually is in the consensus,
but either a) uses an OutboundBindAddress or b) is on a LAN that has
multiple connections to the Internet.  Those addresses (of either kind)
are not published by tor, and the tor project has deferred to the alleged
desires of some unknown portion of the operators of such relays to have
those addresses remain secret at the expense of those of us who wish both
to protect our systems, especially those systems that support the tor
relay network, by the judicious use of packet filters and to give a free
pass through those packet filters to currently running relays to access
our relays' ORPorts and DirPorts.  (I currently have such a "free pass"
system for relays' IP addresses that appear in the most recently fetched
consensus or the most recently fetched list of exit IP addresses.  It
also includes a handful of addresses mentioned on this list as being
relay addresses that might not appear in the consensus.)
     No one on this list, BTW, has yet given a rationale for prioritizing
the allegedly desired secrecy of those relay operators over our efforts
to help to secure our tor relays and our tor relays' hosts.  However, a
couple of people have resorted to name-calling and absurd accusations of
"censorship".  (Now, tell me again, *why* is it that I've run a tor relay
most of the time for over a decade?  To practice censorship?  Such stunning
illogic is rather appalling.)
     A script similar to the one used to reveal and make available the
otherwise unidentified source IP addresses of exits could be run by the
project to gather the hidden addresses of currently running relays, and
a list of such addresses could be made available on a compromise basis,
e.g., by having a relay at the project that would serve those lists only
over tunneled directory connections *from relays*, were it not for obstinacy.
Such a list could then be included into our packet filters' "free pass"
lists without putting the list up on the project's web site like the exit
list is.  A torrc option to fetch the list whenever updates were available
could default to not fetching the list, so relays whose operators who do
not use packet filter defenses would not automatically fetch the hidden
address list of non-exit relays.

> Tor clients, they are behaving very strangely, and in either case we
> need to limit their impact. As such we've implemented connlimits by
> /24 as suggested (with a much higher limit to err on the side of not
> rejecting valid traffic). We can already see that this has improved
> our situation.
>
     And it will likely get you roundly denounced by tor project members
and certain other individuals on this list. :-(  You will also see your
Fast and HSDir flags come and go at random, depending upon how many
authorities creating testing circuits to reach and test your node(s)
go through a node that used a hidden outbound address as the source
address that fails your filter to connect to your node.
     Nevertheless, without knowing whether the sources from which you
claim many connections are made to your relay(s) are themselves relays,
you cannot know whether they are legitimate.  For example, suppose a
high-capacity relay is serving as the hop prior to your relay in many
circuits, quite possibly from large numbers of users in many different
countries.  All you see is that a high-capacity relay has a lot of
circuits created by clients that have chosen that relay, followed by
yours, in their circuits.  If you run an exit, you don't know whether
that relay was the entry relay or the middle relay for any particular
circuit.  There may well be some of each.
>
> @ Scott Bennett:
>
> >      What you are seeing is most likely the same phenomenon brought up on
> > this list repeatedly over at least the last decade or so.  That phenomenon
> > is providing HSDir service, or perhaps a rendez-vous point, for a popular
> > hidden service.
>
> > If you don't like it, you can set
> >
> > HidServDirectoryV2      0
>
> Thanks for your reply. The data suggests this was not the case (this
> time). Some of these relays have been up almost a year with the same
> configuration, often with the HSDir flag. The recent issues just
> started occurring, and happened across several relays during the same
> timeframe.
>
> Nonetheless, we're not opposed to disabling HidServDirectoryV2 to rule
> it out. However it appears that option is deprecated (on 0.3.1.9).
> Enabling it causes this in the log:
>
>   [WARN] Skipping obsolete configuration option 'HidServDirectoryV2'
>
> It's also no longer listed in the Tor manual
> (https://www.torproject.org/docs/tor-manual.html.en). It looks like we
> might be able to achieve the same effect with something like this

     Sigh.  My apologies.  You are indeed correct on this matter.  It had
slipped my mind that tor no longer is distributed with a man page.  Instead,
it has a document as HTML, not intended to be read directly by humans, but
rather by a web browser, which humans must then use to dig around in their
local file systems to find wherever the HTML page got installed in order to
read it with the aid of the browser. :-(

> instead:
>
>   MinUptimeHidServDirectoryV2 52 weeks
>
> Anyone have any info on why HidServDirectoryV2 is consider obsolete?

     Not entirely sure, but I *think* I remember reading somewhere that
HS v2 is no longer preferred and has been superseded by HS v3.  I hope
someone will confirm/correct this recollection.

> Is using MinUptimeHidServDirectoryV2 instead going to achieve the same
> effect?
>
     As teor has answered you already, I'll skip this.
>
> >> We have file descriptors (/proc/sys/fs/file-max) set to 64000, but it looks
> >> like Tor sets MAX_FILEDESCRIPTORS to 16384 per /etc/init.d/tor:
> >>
> >>   elif [ "$system_max" -gt "40000" ] ; then
> >>     MAX_FILEDESCRIPTORS=16384
> >>
> >> Surely that is high enough for normal service?
> >
> >      If by normal you mean "low traffic", then yes, it's probably enough.
> > However, that's really not very high in a general sense.
>
>
> Why does the default /etc/init.d/tor (from the Debian/Ubuntu pkg) cap
> MAX_FILEDESCRIPTORS at 16384 then? We have it set to 64000 otherwise.
> Obviously we could comment out these lines, but it seems odd the
> default config tries to cap it at 16384 if that's too low for a high
> traffic relay.
>
     teor has addressed this one with a question.  I am not a LINUX user,
so I have no idea.  Perhaps an operator of one or more high-capacity relays
on any OS can provide some idea of what maxima might be reasonable for
relays of different capacity scales (e.g., 500 KB/s, 1 MB/s, 5 MB/s, 10 MB/s--
Oleg?  Are you still out there?  Moritz?).


                                  Scott Bennett, Comm. ASMELG, CFIAG
**********************************************************************
* Internet:   bennett at sdf.org   *xor*   bennett at freeshell.org  *
*--------------------------------------------------------------------*
* "A well regulated and disciplined militia, is at all times a good  *
* objection to the introduction of that bane of all free governments *
* -- a standing army."                                               *
*    -- Gov. John Hancock, New York Journal, 28 January 1790         *
**********************************************************************