High speed Tor relay advice
Hi all, I've ran multiple Tor relays before but I have moved to a new server and would like some advice. My new server has 10Gb/s connection (I've observed it at 900MB/s to the drives) with plenty of CPU and RAM to complement. I typically use default configurations on my relays but I feel that to get the most out of this one I'll need to do some configuration to tweak it. Does anyone have advice on getting the most out of this server, in terms of speed? Thanks
Hey, That's neat! Thanks for contributing. How many CPU's / CPU cores does this new server have and does it use AES-NI? How much RAM? Does it have multiple public IP addresses? Currently it's complicated for a single Tor process to saturate a 10Gb/s line, because it's not yet able to use all CPU cores. What I would do if I had multiple public IP addresses: make 4-5 virtual machines, with 1 CPU core each and reasonable RAM (say 8GB per virtual machine) and run 4-5 different relays that would all combined come close to saturate the 10Gb/s link. On 8/14/2016 5:09 PM, i3 wrote:
Hi all,
I've ran multiple Tor relays before but I have moved to a new server and would like some advice.
My new server has 10Gb/s connection (I've observed it at 900MB/s to the drives) with plenty of CPU and RAM to complement. I typically use default configurations on my relays but I feel that to get the most out of this one I'll need to do some configuration to tweak it.
Does anyone have advice on getting the most out of this server, in terms of speed?
Thanks
On Aug 14, 2016 9:28 AM, "s7r" <s7r@sky-ip.org> wrote:
Currently it's complicated for a single Tor process to saturate a 10Gb/s line, because it's not yet able to use all CPU cores.
Out of curiosity, what is the maximum speed a single Tor instance can achieve? Are there any plans for multi-core support?
Hey! So the server is technically a VPS, it is a slice of a larger server that is shared with 5 other people. Though I still have full root access. So the whole 10gb/s is not just for me, but from my tests I can at least get a few gigabit in real world speeds sustained. CPU: 6x Xeon E5-2620v3 vCores RAM: 10GB I only get one IP address to myself by default. I could probably get more though if I feel it is worth it. On 14/08/16 15:27, s7r wrote:
Hey,
That's neat! Thanks for contributing.
How many CPU's / CPU cores does this new server have and does it use AES-NI? How much RAM? Does it have multiple public IP addresses?
Currently it's complicated for a single Tor process to saturate a 10Gb/s line, because it's not yet able to use all CPU cores.
What I would do if I had multiple public IP addresses: make 4-5 virtual machines, with 1 CPU core each and reasonable RAM (say 8GB per virtual machine) and run 4-5 different relays that would all combined come close to saturate the 10Gb/s link.
On 8/14/2016 5:09 PM, i3 wrote:
Hi all,
I've ran multiple Tor relays before but I have moved to a new server and would like some advice.
My new server has 10Gb/s connection (I've observed it at 900MB/s to the drives) with plenty of CPU and RAM to complement. I typically use default configurations on my relays but I feel that to get the most out of this one I'll need to do some configuration to tweak it.
Does anyone have advice on getting the most out of this server, in terms of speed?
Thanks
_______________________________________________ tor-relays mailing list tor-relays@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays
Hello, In this case I suggest you run two separate Tor instances on the same IP address (it is the maximum allowed). When I say 2 Tor instances I mean 2 separate data directories with separate identities. If it's a VPS it's already a VM. It does not make sense to create a vm inside a vm. You can use linux containers or FreeBSD jails if you are on FreeBSD which work fine in virtual machines, one for each Tor instance. P.S. If you can, make them exit relays if they're not already. It's not that much of a hassle, if the provider is mentally sane and just forwards you the abuse complaints with request to reply within 24 hours. Most of them are automated emails which do not require replies, and the rare ones which require reply are from people who just don't know what Tor is, just explain and they will understand. On 8/14/2016 5:39 PM, i3 wrote:
Hey!
So the server is technically a VPS, it is a slice of a larger server that is shared with 5 other people. Though I still have full root access. So the whole 10gb/s is not just for me, but from my tests I can at least get a few gigabit in real world speeds sustained.
CPU: 6x Xeon E5-2620v3 vCores RAM: 10GB
I only get one IP address to myself by default. I could probably get more though if I feel it is worth it.
On 14/08/16 15:27, s7r wrote:
Hey,
That's neat! Thanks for contributing.
How many CPU's / CPU cores does this new server have and does it use AES-NI? How much RAM? Does it have multiple public IP addresses?
Currently it's complicated for a single Tor process to saturate a 10Gb/s line, because it's not yet able to use all CPU cores.
What I would do if I had multiple public IP addresses: make 4-5 virtual machines, with 1 CPU core each and reasonable RAM (say 8GB per virtual machine) and run 4-5 different relays that would all combined come close to saturate the 10Gb/s link.
On 8/14/2016 5:09 PM, i3 wrote:
Hi all,
I've ran multiple Tor relays before but I have moved to a new server and would like some advice.
My new server has 10Gb/s connection (I've observed it at 900MB/s to the drives) with plenty of CPU and RAM to complement. I typically use default configurations on my relays but I feel that to get the most out of this one I'll need to do some configuration to tweak it.
Does anyone have advice on getting the most out of this server, in terms of speed?
Thanks
Hi i3, Thanks for running relays! Agreed you will want to run multiple instances to make the most of your host. The Xeon E5-2620v3 does have AES-NI, which is good. Other items to consider: - On most linux/unix systems the ulimits will be set too low by default. On debian-like linux, higher limits for Tor might look like the following in /etc/security/limits.conf: debian-tor soft nofile 64000 debian-tor hard nofile 64000 - There are a lot of possible syctl tweaks. You could check out https://raw.githubusercontent.com/torservers/server-config-templates/master/.... I use these values on my non-exit relays: net.core.wmem_max = 12582912 net.core.rmem_max = 12582912 net.ipv4.tcp_rmem = 10240 87380 12582912 net.ipv4.tcp_wmem = 10240 87380 12582912 net.ipv4.conf.default.send_redirects = 1 net.ipv4.conf.all.rp_filter = 0 net.ipv4.conf.all.send_redirects = 0 kernel.sysrq = 0 net.ipv4.icmp_echo_ignore_broadcasts = 1 net.ipv4.conf.all.accept_redirects = 0 net.ipv4.icmp_ignore_bogus_error_responses = 1 net.ipv4.tcp_no_metrics_save = 1 net.ipv4.tcp_moderate_rcvbuf = 1 net.ipv4.tcp_tw_recycle = 1 net.ipv4.tcp_fin_timeout = 4 net.ipv4.tcp_keepalive_time = 60 net.ipv4.tcp_keepalive_intvl = 10 net.ipv4.tcp_keepalive_probes = 3 net.ipv4.ip_local_port_range = 1025 65530 net.core.somaxconn = 20480 net.ipv4.tcp_timestamps = 0 fs.file-max = 64000 - You should install ntp make sure your clock is synced. - You should install a local DNS resolver (unbound, named, dnsmasq, something along those lines). This guide has a lot of helpful info: https://www.torservers.net/wiki/setup/server
On 15 Aug 2016, at 06:54, Green Dream <greendream848@gmail.com> wrote:
- You should install ntp make sure your clock is synced.
Alternately, run ntpdate via cron every few hours, to avoid running an unnecessary network service. (Recent security issues in ntp remain unpatched in some distributions.) Tim Tim Wilson-Brown (teor) teor2345 at gmail dot com PGP C855 6CED 5D90 A0C5 29F6 4D43 450C BA7F 968F 094B ricochet:ekmygaiu4rzgsk6n xmpp: teor at torproject dot org
On 08/14/16 20:48, teor wrote:
On 15 Aug 2016, at 06:54, Green Dream <greendream848@gmail.com> wrote:
- You should install ntp make sure your clock is synced.
Alternately, run ntpdate via cron every few hours, to avoid running an unnecessary network service. (Recent security issues in ntp remain unpatched in some distributions.)
There's actually a technical problem with running ntpdate periodically. NTP works by slowly and carefully adjusting the time, accounting for any local gaps without breaking any precise-time requiring daemons or functions, like databases. ntpdate might be used on startup (or with ntpd_sync_on_start), but it's deprecated last I read. Tools like rdate are not replacements for an ntp daemon on a production system. You might even notice the tor daemon isn't fond of abrupt time adjustments, and will bark in the log about it. Don't like the standard ntpd? Try openntpd (.org) which is probably in your OS's port/pkg system, if not in its base already. Totally adequate, secure and small ntpd replacement. g -- 5F77 765E 40D6 5340 A0F5 3401 4997 FF11 A86F 44E2
On Sun, 14 Aug 2016 20:57:13 -0400 George <george@queair.net> wrote:
Alternately, run ntpdate via cron every few hours, to avoid running an unnecessary network service. (Recent security issues in ntp remain unpatched in some distributions.)
There's actually a technical problem with running ntpdate periodically. NTP works by slowly and carefully adjusting the time, accounting for any local gaps without breaking any precise-time requiring daemons or functions, like databases. ntpdate might be used on startup (or with ntpd_sync_on_start), but it's deprecated last I read. Tools like rdate are not replacements for an ntp daemon on a production system.
You might even notice the tor daemon isn't fond of abrupt time adjustments, and will bark in the log about it.
I do the same (ntpdate in crontab) and have never seen any issues with that. If you run ntpdate often enough (in my case I find every 6 hours suffices) your computer clock will never have the chance to drift far enough away, so that the ntpdate adjustment would disrupt anything. -- With respect, Roman
At 17:27 8/14/2016 +0300, s7r wrote:
What I would do if I had multiple public IP addresses: make 4-5 virtual machines
Have to disagree here. VMs are extremely bad for latency. Just go single physical machine and create different user IDs to isolate the daemons if you are concerned about someone breaking into an instance and taking over the rest. Another way to isolate without sacrificing performance is to put each instance in a Linux container. All of it more trouble than it's worth IMO.
Suggest running multiple instances of the Tor daemon and placing them in a single family. One daemon per physical core up to perhaps eight. Only two daemons can be configured for any two IP addresses so you need at least that many IPs, but one-IP-per-daemon is ideal. When deciding how many daemons to run, take into account any FUP (fair use policy) and/or true available bandwidth which could be substantially less than 10G. You will see better performance if you isolate interrupts to one or two cores and bind the Tor daemons to other cores, each with it's own physical core. If you end up with extra cores, look into mapping the crypto threads to the spares though this probably won't make much difference. Possibly disable SMT/hyperthreading if you can since virtual cores give worse latency than what can be achieved with a one-core/one-thread config. At 15:09 8/14/2016 +0100, you wrote:
Hi all,
I've ran multiple Tor relays before but I have moved to a new server and would like some advice.
My new server has 10Gb/s connection (I've observed it at 900MB/s to the drives) with plenty of CPU and RAM to complement. I typically use default configurations on my relays but I feel that to get the most out of this one I'll need to do some configuration to tweak it.
Does anyone have advice on getting the most out of this server, in terms of speed?
On 8/14/16, i3 <i3jm2y7yftyix4kobdra@posteo.ee> wrote:
My new server has 10Gb/s connection (I've observed it at 900MB/s to the drives
Depending on whether you meant MiB/s or MB/s, you may find your network calculations off by 350Mbps, which is a sizable tor relay's worth itself. Standard use is decimal and bits for network "Mbps", and binary and bytes for disk "MiB". "MB/s" is neither. Your drives quote yields 7.2Gbps minimum.
CPU: 6x Xeon E5-2620v3 vCores
Being subject to vps provider, and depending on how the cpu's and clocks are parceled out, core count can help there. This cpu is 2.4GHz. "5 other people" sounds like you may have a whole core. Anyway, you'll enjoy taking it for spin :) For those doing dedi or colo, clock speed and cpu generation per price can be large factor, fastest is 4.0GHz, lots of 4x cores near there. Some synthetic comparison charts... http://cpubenchmark.net/singleThread.html http://cpubenchmark.net/high_end_cpus.html VPS might have some careful reason to even look at... http://www.cpubenchmark.net/multi_cpu.html Intel released some skylake v5 etc since last I looked... http://ark.intel.com/Search/Advanced?s=t&RetailSkuAvailable=true&FilterCurre...
On Mon, 15 Aug 2016 02:35:49 -0400 grarpamp <grarpamp@gmail.com> wrote:
On 8/14/16, i3 <i3jm2y7yftyix4kobdra@posteo.ee> wrote:
My new server has 10Gb/s connection (I've observed it at 900MB/s to the drives
Depending on whether you meant MiB/s or MB/s, you may find your network calculations off by 350Mbps,
To me these seem to be just two loosely related facts, the latter merely supporting the notion that the network connection is capable of way more than 1Gbps. I don't see any "network calculations" being presented.
Standard use is decimal and bits for network "Mbps", and binary and bytes for disk "MiB". "MB/s" is neither.
https://en.wikipedia.org/wiki/Megabyte: "The megabyte is a multiple of the unit byte for digital information. Its recommended unit symbol is MB." MB/s is a long-accepted shorthand for Megabyte per second, and yes, Mb/s is megabit per second. But please, take your "MiB" lunacy elsewhere. -- With respect, Roman
On 2016-08-15 at 08:52, Roman Mamedov wrote:
https://en.wikipedia.org/wiki/Megabyte: "The megabyte is a multiple of the unit byte for digital information. Its recommended unit symbol is MB." MB/s is a long-accepted shorthand for Megabyte per second, and yes, Mb/s is megabit per second. But please, take your "MiB" lunacy elsewhere.
Well, "MiB" is a completely other unit than MB, so that's no lunacy, but in fact a different unit called "Mebibyte". Mebibytes are exactly 1024 x 1024 bytes or 2^10 multiplied by 2^10 bytes and those binary prefixes give a more "accurate" representation of the size than the SI prefixes. https://en.wikipedia.org/wiki/Mebibyte Best, Michael
On 8/15/16, Roman Mamedov <rm@romanrm.net> wrote:
To me these seem to be just two loosely related facts, the latter merely I don't see any "network calculations" being presented.
Was an fyi for the OP, who may or may not be doing calculations, regardless of presentation to us.
https://en.wikipedia.org/wiki/Megabyte: "The megabyte is a multiple of the unit byte for digital information. Its recommended unit symbol is MB." MB/s is a long-accepted shorthand for Megabyte per second, and yes, Mb/s is megabit per second. But please, take your "MiB" lunacy elsewhere.
The current published standards of standards bodies, you would find that "M" for binary is deprecated in favor of Mi, thus making MiB no such lunacy. Then there is proper context, where manufacturers are getting sued, and some unixes and apps are correcting their stats output, be it in underlying quantities and/or in naming with bits(b) for network in decimal (for interfacing with network providers and hardware), and bytes(B) for disk and ram in binary (for interfacing with read() call of 2^3=8 bits (byte) [1]). MB is thus bit of deprecated legacy mashup. What is lunacy is perpetuating legacy ambiguous terminology and context against standards. Things like mars climate orbiter happen when you do that. [1] Though I'd suggest having switches to optional display disk/ram rates per sec in decimal bits for easier network throughput checks at those location in system.) No worries, the next pint of beer is on me, while I bitch about the "8GB" stick I tested having far less than 8E9 bytes ;) https://en.wikipedia.org/wiki/IEC_60027 https://en.wikipedia.org/wiki/ISO/IEC_80000 https://en.wikipedia.org/wiki/Binary_prefix http://www.bipm.org/en/publications/si-brochure/ http://physics.nist.gov/cuu/Units/binary.html http://members.optus.net/alexey/prefBin.xhtml
participants (10)
-
George -
grarpamp -
Green Dream -
i3 -
Michael Armbruster -
Roman Mamedov -
s7r -
starlight.2016q3＠binnacle.cx -
teor -
Tristan