[tor-relays] How to Run High Capacity Tor Relays

[Green Dream]

Pardon the thread necromancy, but I'm wondering if this document ever made
its way off this mailing list and onto a blog? Or perhaps there is some
other modern doc covering this topic?

I've recently setup a relay on a Gb/s fiber connection, and am struggling
to understand how to optimize performance. It's not clear 5 years later
which if any of the tweaks listed below are still relevant. I'm running a
modern debian-based system.

Thanks in advance.


> After talking to Moritz and Olaf privately and asking them about their
> nodes, and after running some experiments with some high capacity
> relays, I've begun to realize that running a fast Tor relay is a
> pretty black art, with a lot of ad-hoc practice. Only a few people
> know how to do it, and if you just use Linux and Tor out of the box,
> your relay will likely underperform on 100Mbit links and above.
> In the interest of trying to help grow and distribute the network, my
> ultimate plan is to try to collect all of this lore, use Science to
> divine out what actually matters, and then write a more succinct blog
> post about it.
> However, that is a lot of work. It's also not totally necessary to do
> all this work, when you can get a pretty good setup with a rough
> superset of all of the ad-hoc voodoo. This post is thus about that
> voodoo.
> Hopefully others will spring forth from the darkness to dump their own
> voodoo in this thread, as I suspect there is one hell of a lot of it
> out there, some (much?) of which I don't yet know. Likewise, if any
> blasphemous heretic wishes to apply Science to this voodoo, they
> should yell out, "Stand Back, I'm Doing Science!" (at home please, not
> on this list) and run some experiments to try to eliminate options
> that are useless to Tor performance. Or cite academic research papers.
> (But that's not Science, that's computerscience - which is a religion
> like voodoo, but with cathedrals).
> Anyway, on with the draft:
>
> == Machine Specs ==
> First, you want to run your OS in x64 mode because openssl should do
> crypto faster in 64bit.
> Tor is currently not fully multithreaded, and tends not to benefit
> beyond 2 cores per process. Even then, the benefit is still marginal
> beyond just 1 core. 64bit Tor nodes require about one 2Ghz Xeon/Core2
> core per 100Mbit of capacity.
> Thus, to fill an 800Mbit link, you need at least a dual socket, quad
> core cpu config.  You may be able to squeeze a full gigabit out of one
> of these machines. As far as I know, no one has ever done this with
> Tor, on any one machine.
> The i7's also just came out in this form factor, and can do
> hyperthreading (previous models may list 'ht' in cpuinfo, but actually
> don't support it). This should give you a decent bonus if you set
> NumCPUs to 2, since ht tends to work better with pure integer math
> (like crypto). We have not benchmarked this config yet though, but I
> suspect it should fill a gigabit link fairly easily, possibly
> approaching 2Gbit.
> At full capacity, exit node Tor processes running at this rate consume
> about 500M of ram. You want to ensure your ram speed is sufficient,
> but most newish hardware is good. Using on this chart:
>
> https://secure.wikimedia.org/wikipedia/en/wiki/List_of_device_bandwidths#Memory_Interconnect.2FRAM_buses
> you can do the math and see that with a dozen memcpys in each
> direction, you come out needing DDR2 to be able to push 1Gbit full
> duplex.
> As far as ethernet cards, the Intel e1000e *should* be theoretically
> good, but they seem to fail at properly irq balancing across multiple
> CPUs on recent kernels, which can cause you to bottleneck at 100% CPU
> on one core. At least that has been Moritz's experience. In our
> experiments, the RTL-8169 works fine (once tweaked, see below).
>
> == System Tweakscript Wibbles and Config Splatters ==
> First, you want to ensure that you run no more than 2 Tor instances
> per IP. Any more than this and clients will ignore them.
> Next, paste the following smattering into the shell (or just read it
> and make your own script):
> # Set the hard limit of open file descriptors really high.
> # Tor will also potentially run out of ports.
> ulimit -SHn 65000
> # Set the txqueuelen high, to prevent premature drops
> ifconfig eth0 txqueuelen 20000
> # Tell our ethernet card (interrupt found from /proc/interrupts)
> # to balance its IRQs across one whole CPU socket (4 cpus, mask 0f).
> # You only want one socket for optimal ISR and buffer caching.
> #
> # Note that e1000e does NOT seem to obey this, but RTL-8169 will.
> echo 0f > /proc/irq/17/smp_affinity
> # Make sure you have auxiliary nameservers. I've seen many ISP
> # nameservers fall over under load from fast tor nodes, both on our
> # nodes and from scans. Or run caching named and closely monitor it.
> echo "nameserver 8.8.8.8" >> /etc/resolv.conf
> echo "nameserver 4.2.2.2" >> /etc/resolv.conf
> # Load an amalgam of gigabit-tuning sysctls from:
> # http://datatag.web.cern.ch/datatag/howto/tcp.html
> # http://fasterdata.es.net/TCP-tuning/linux.html
> # http://www.acc.umu.se/~maswan/linux-netperf.txt
> # http://www.psc.edu/networking/projects/tcptune/#Linux
> # and elsewhere...
> # We have no idea which of these are needed yet for our actual use
> # case, but they do help (especially the nf-contrack ones):
> sysctl -p << EOF
> net.ipv4.tcp_rmem = 4096 87380 16777216
> net.ipv4.tcp_wmem = 4096 65536 16777216
> net.core.netdev_max_backlog = 2500
> net.ipv4.tcp_no_metrics_save = 1
> net.ipv4.tcp_moderate_rcvbuf = 1
> net.core.rmem_max = 1048575
> net.core.wmem_max = 1048575
> net.ipv4.ip_local_port_range = 1025 61000
> net.ipv4.tcp_synack_retries = 3
> net.ipv4.tcp_tw_recycle = 1
> net.ipv4.tcp_max_syn_backlog = 10240
> net.ipv4.tcp_fin_timeout = 30
> net.ipv4.tcp_keepalive_time = 1200
> net.netfilter.nf_conntrack_tcp_timeout_established=7200
> net.netfilter.nf_conntrack_checksum=0
> net.netfilter.nf_conntrack_max=131072
> net.netfilter.nf_conntrack_tcp_timeout_syn_sent=15
> net.ipv4.tcp_keepalive_time=60
> net.ipv4.tcp_keepalive_time = 60
> net.ipv4.tcp_keepalive_intvl = 10
> net.ipv4.tcp_keepalive_probes = 3
> net.ipv4.ip_local_port_range = 1025 65530
> net.core.netdev_max_backlog=300000
> net.core.somaxconn=20480
> net.ipv4.tcp_max_tw_buckets=2000000
> net.ipv4.tcp_timestamps=0
> vm.min_free_kbytes = 65536
> net.ipv4.ip_forward=1
> net.ipv4.tcp_syncookies = 1
> net.ipv4.tcp_synack_retries = 2
> net.ipv4.conf.default.forwarding=1
> net.ipv4.conf.default.proxy_arp = 1
> net.ipv4.conf.all.rp_filter = 1
> net.ipv4.conf.default.send_redirects = 1
> net.ipv4.conf.all.send_redirects = 0
> EOF
> # XXX: ethtool wibbles
> # You may also have to tweak some parameters with ethtool, possibly
> # also enabling some checksum offloading or irq coalescing options to
> # spare CPU, but for us this hasn't been needed yet.
>
> == Setting up the Torrc ==
> Basically you can just read through the stock example torrc, but there
> are some as-yet undocumented magic options, and options that need new
> defaults.
> # NumCPUs doesn't provide any benefit beyond 2, and setting it higher
> # may cause cache misses.
> NumCPUs 2
> # These options have archaic maximums of 2-5Mbyte
> BandwidthRate 100 MB
> BandwidthBurst 200 MB
>
> == Waiting for the Bootstrap and Measurement Process ==
> Perhaps the most frustrating part of this setup is how long it takes
> for you to acquire traffic. If you are starting new at an ISP, I would
> consider only 200-400Mbit for your first month. Hitting that by the
> end of the month may be a challenge, mostly because their may be dips
> and total setbacks along the way.
> The slow rampup is primarily due to limitations in Tor's ability to
> rapidly publish descriptor updates, and to measure relays.
> It ends up taking about 2-3 days to hit an observed bandwidth of
> 2Mbyte/sec per relay, but it can take well over a week or more (Moritz, do
> you have a better number?) to reach 8-9Mbyte/relay. This is for an
> Exit node. A middle node will likely gather traffic slower. Also once
> you crash, you lose it. This bug is about that issue:
> https://trac.torproject.org/projects/tor/ticket/1863
> There is also a potential dip when you get the Guard flag, as our load
> balancing formulas try to avoid you, but no clients have chosen you
> yet. Changes to the authority voting on Guards in Tor 0.2.2.15 should
> make this less drastic. It is also possible that your observed
> bandwidth will be greater than without it. However, it will still take
> up to 2 months for clients to choose you as their new Guard.
>
> == Running temporary auxiliary nodes ==
> One way to shortcut this process and avoid paying for bandwidth you
> don't use is to spin up a bunch of temporary nodes to utilize the CPU
> and quickly gather that easy first 2MB/sec of observed bandwidth.
> But you need the spare IPs to do this.
>
> == Monitoring ==
> Personally, I prefer console-based options like nload, top, and
> Damian's arm (http://www.atagar.com/arm/) because I don't like the
> idea of running extra services to publish my monitoring data to the
> world.
> Other people have web-based monitoring using things like munin and
> mrtg. It would be nice to get a script/howto for that too.
>
> == Current 1-Box Capacity Record ==
> Our setup has topped at 450Mbit, but averages between 300-400Mbit. We
> are currently having uptime issues due to heat (melting, poorly
> ventilated harddrives). It is likely that once we resolve this, we
> will continually increase to our CPU ceiling.
> I believe Moritz and Olaf also push this much capacity, possibly a bit
> more, but with less nodes (4 as opposed to our 8). I hear Jake is also
> ramping up some Guard nodes (or maybe I didn't? Did I just betray you
> again Jake?)
>
> == Did I leave anything out? ==
> Well, did I?
>
> --
> Mike Perry
> Mad Computer Scientist
> fscked.org evil labs
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.torproject.org/pipermail/tor-relays/attachments/20150721/b6d239a4/attachment-0001.html>