null null@omuravpn.com wrote:
We're experiencing what looks like a DoS attack on multiple relays in our family:
https://atlas.torproject.org/#search/family:CBEAE10CBBB86C51059246B2EF92EB2C...
The relays are currently running Tor 0.3.1.9 on Linux kernel 4.4.0 (although when the problem started the relays were running Tor 0.3.1.8).
The attack knocked 3 of 6 relays offline overnight. By the time we looked at logs, the Tor service had stopped and this was the last line in the log:
"Tor[xyz]: Failing because we have 16351 connections already. Please read doc/TUNING for guidance."
The attack is still ongoing. When it's happening, the number of connections rises very rapidly, until the attack succeeds in stopping the service.
$ ss -s Total: 15855 (kernel 0) TCP: 24520 (estab 23969, closed 305, orphaned 31, synrecv 0, timewait 261/0), ports 0
Transport Total IP IPv6
- 0 - -
RAW 0 0 0 UDP 8 4 4 TCP 24215 24213 2 INET 24223 24217 6 FRAG 0 0 0
... and only a few seconds later:
$ ss -s Total: 12120 (kernel 0) TCP: 27389 (estab 20026, closed 1906, orphaned 45, synrecv 0, timewait 1587/0), ports 0
Transport Total IP IPv6
- 0 - -
RAW 0 0 0 UDP 8 4 4 TCP 25483 25481 2 INET 25491 25485 6 FRAG 0 0 0
That's obviously much larger than the normal number of connections, more than we've ever seen, and seems like more connections than would be needed for a relay.
What you are seeing is most likely the same phenomenon brought up on this list repeatedly over at least the last decade or so. That phenomenon is providing HSDir service, or perhaps a rendez-vous point, for a popular hidden service. As soon as your node is associated with that hidden service and that association begins to be distributed by the HSDir population to clients looking for that hidden service, the number of connections to your node will increase fairly rapidly to a level corresponding to that hidden service's level of popularity. If you don't like it, you can set
HidServDirectoryV2 0
which will stop clients from trying to get hidden service descriptors from your node, which will eliminate most of the bursts of connections you're seeing, but will not prevent your node from being a rendez-vous point because every tor relay is expected to provide that function as part of the relay protocols.
We have file descriptors (/proc/sys/fs/file-max) set to 64000, but it looks like Tor sets MAX_FILEDESCRIPTORS to 16384 per /etc/init.d/tor:
elif [ "$system_max" -gt "40000" ] ; then MAX_FILEDESCRIPTORS=16384
Surely that is high enough for normal service?
If by normal you mean "low traffic", then yes, it's probably enough. However, that's really not very high in a general sense. Consider also that some installed packages place high demands upon the supply of file descriptors. (E.g., I gather you do not have a graphics port/package called piglit installed on your system, which recommends at least 50000 be available for its runs, so I have
kern.maxfiles="50000"
in /boot/loader.conf on my FreeBSD system. I don't think I can recall tor ever handling many more than 5000 (i.e., 10% of that figure) at one time on my low-traffic node.) The faster, larger-capacity tor nodes often have considerably higher settings to keep tor from exhausting the fd limits on those hosts.
We haven't started looking into where the traffic is coming from or other characteristics. We are wondering if: 1) this is a known attack, 2) if other operators are experiencing it, 3) if there are any ideas for mitigating it, and 4) if any additional information would be helpful.
Other than refusing to be a hidden service directory server, there is probably nothing to be done about it. Adjust your settings accordingly, along with your expectations. :-)
Scott Bennett, Comm. ASMELG, CFIAG ********************************************************************** * Internet: bennett at sdf.org *xor* bennett at freeshell.org * *--------------------------------------------------------------------* * "A well regulated and disciplined militia, is at all times a good * * objection to the introduction of that bane of all free governments * * -- a standing army." * * -- Gov. John Hancock, New York Journal, 28 January 1790 * **********************************************************************