Hardware sizing for physical exit node
Hi everyone, we are planning to get some hardware to run a physical Tor exit node, starting with a 1Gbps dedicated, unmetered uplink (10Gbps downlink). We will also route a /24 on it, so we will have large availability of addresses to run multiple instances. We have been running a few exit nodes so far, but never on our own hardware. Which is the bandwith limit per core/Tore instance? Or what can we expect to be the bottleneck? Due to some other requirements we need for some experiments (SFP ports, coreboot support, etc) we can mainly choose between these 2 CPUs: Intel i5-1235U Intel i7-1255U The cost between the two models is significant enough in our case to pick the i7 only if it's really useful. In both cases with 32GB of DDR5 RAM (we can max to 64 if needed, but is it?). Should this allow us to saturate the uplink? To summarize, with this bandwith, this hardware and a /24 how many Tor exit nodes should be ideal to run considering that each of them could have their own address? Thanks!
On Mittwoch, 10. Juli 2024 00:32:04 CEST Osservatorio Nessuno via tor-relays wrote:
we are planning to get some hardware to run a physical Tor exit node, starting with a 1Gbps dedicated, unmetered uplink (10Gbps downlink). We will also route a /24 on it, so we will have large availability of addresses to run multiple instances. We have been running a few exit nodes so far, but never on our own hardware.
Your bottleneck is the 1G uplink. For comparison, I have 2x Xeon E5-2680v2 10C/20T and 256Gb RAM 2x 10G nic (LACP bond) and I can not achieve 10G throughput with it. As a rule of thumb, I would always count one instance per thread or core. I have 40T and 40 tor exit instances. F3Netze has specified the hardware in Contact info: https://metrics.torproject.org/rs.html#search/185.220.100.
Which is the bandwith limit per core/Tore instance? Or what can we expect to be the bottleneck?
That depends on the CPU clock speed. Fast Ryzen or Epyc's can do 50-70 MiB/s per core/instance.
Due to some other requirements we need for some experiments (SFP ports, coreboot support, etc) we can mainly choose between these 2 CPUs: Intel i5-1235U Intel i7-1255U
The cost between the two models is significant enough in our case to pick the i7 only if it's really useful.
In both cases with 32GB of DDR5 RAM (we can max to 64 if needed, but is it?).
Should this allow us to saturate the uplink?
Guards need more resources than exits since the introduction of congestion- control and because of DDoS I would use 64GB RAM for a guard. With your IP space and 1G uplink, I would take the i5 with 32Gb, save the money and maybe add a second server later. Or if you build the hardware yourself, look for a used Epyc or Ryzen server. 16 or 32 core with high _base_ clock. Used server hardware from the data center is like new.
To summarize, with this bandwith, this hardware and a /24 how many Tor exit nodes should be ideal to run considering that each of them could have their own address?
https://metrics.torproject.org/rs.html#search/185.220.101. We are 5 relay orgs sharing a /24. Currently 5x 2x10G(or 25G) With now 8 relays per IP, over 2000 instances can run in a /24 subnet. It would be nice if you share the subnet with 1-2 other relay operators. -- ╰_╯ Ciao Marco! Debian GNU/Linux It's free software and it gives you freedom!
Hi, Imo mobile chips with mostly low-performance 'efficient cores' are a suboptimal fit for running Tor at scale. A couple of reasons: - Performance between the different core types (efficient vs. performance) is very different. You would need to lock Tor processes to specific cores and accept that you can only run 2 fast relays (and 8 slow ones), or have 10 of them capped at the efficient core's performance profile (because the Tor processes would get swapped around all the time). - These CPUs aren't build for sustained loads so their real world sustained clock speed is significantly lower than their turbo speed, hampering their performance (especially compared to benchmarks). That being said, especially the 2 performance cores are quite decent and should be able to push some nice amount of Tor traffic (with some help from the efficient cores). My guess is that the i7 would not saturate a 1 Gb/s link (probably 600-650 Mb/s when utilizing all cores effectively) if your relays have the guard flag. Middle only relays would push more traffic. So with some quirks/workarounds, I think you could certainly make these CPUs work (but I still wouldn't recommend them). Do you need SFP or SFP+? Both aren't a feature of the CPU (except for some embedded SoCs) and are generally easily/cheaply available. For example the Intel X520-DA2 and Mellanox ConnectX-3 are extremely affordable second hand or refurbished and both of the CPUs have enough PCI-e lanes to run them at full speed. Regarding i5 vs i7: the i7 effectively only has a higher sustained clock speed and marginally better GPU (which is irrelevant for Tor) but otherwise is pretty much identical. If the difference would be small (i.e. ~20 euro/dollar) I would go for the i7 but otherwise I wouldn't bother. Your mileage may vary of course :-). About RAM: Tor is (for the most part) single threaded so quite a few operators run one Tor process per physical core or per thread. In your case this would mean 10 or 12 Tor relays. Especially guard relays can go out of memory very easily nowadays because of all sorts of attacks, so I wouldn't recommend less than 4 GB of available RAM per relay (so 40-48 GB for Tor + X GB for OS/networking/routing/firewall/services/etc.). 90% of the time you don't need it, but when you do need it it makes sure Tor doesn't crash, your kernel doesn't panic and your server stays responsive (e.g. ssh) etc. You could also run less relays (or limit their bandwidth) and keep it at 32 GB of RAM, but then you would have a harder time to saturate the uplink. Cheers, tornth Jul 10, 2024, 11:30 by tor-relays@lists.torproject.org:
Hi everyone, we are planning to get some hardware to run a physical Tor exit node, starting with a 1Gbps dedicated, unmetered uplink (10Gbps downlink). We will also route a /24 on it, so we will have large availability of addresses to run multiple instances. We have been running a few exit nodes so far, but never on our own hardware.
Which is the bandwith limit per core/Tore instance? Or what can we expect to be the bottleneck?
Due to some other requirements we need for some experiments (SFP ports, coreboot support, etc) we can mainly choose between these 2 CPUs: Intel i5-1235U Intel i7-1255U
The cost between the two models is significant enough in our case to pick the i7 only if it's really useful.
In both cases with 32GB of DDR5 RAM (we can max to 64 if needed, but is it?).
Should this allow us to saturate the uplink?
To summarize, with this bandwith, this hardware and a /24 how many Tor exit nodes should be ideal to run considering that each of them could have their own address?
Thanks! _______________________________________________ tor-relays mailing list tor-relays@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays
Hi, my personnal experience with many many instances on never redundant hardware (I know diversity in hardware, locations...): This is Exit specific: 1) never more than 4GB ram per core and never less than 2 cores per IP, let me explain: a) most people will tell you that instances run per core, so why 2 cores? + because of DDOS and similar attacks, then the instance is jumping to the other core and is much a higher cost to kill than on their own (1 core). This has the effect of putting the attack investment way above average and is going to save your instance from falling back months earlier than its current reputation level. b) An IP will receive complaints and be banned (even when not from solely your ISP), not a VPS contract. + you want to avoid compromising other instances and this is why many IPs-max4GB is important: this spreads your hardware potential accross diversified IP affected by individual probabilities of being banned somewhere. 2) CPU work from tore is basic, RAM is the amplitude: you want a maximum of modern to medium-old CPUs threads, save on your cpu choice (but study their known vulnerabilities) for an even number of heads, prefer cpus with most threads. TO CONCLUDE: 64GB ram for 10Gbps is normally overkill : 10Gbps (octets) 24/7 is 3300 TeraB-Y-T-E-S / month, this one hardware cap. I have never witnessed a conventional (relay) 1Gpbs-2BG-1thread tor instance outperforming 10MB-Y-T-E-S, this is kind of a Tor-cap. This is inviting to never overkill with a 10Gbps+ connection less than 100 x 10MB-Y-T-E-S (and this even when stipulating max bandwidth to some infinite number in your torrc). My answer: with 10Gbps unlimited bandwidth, have 100 threads (50 to 100 cores) at the cheapest CPU and 2GB to 4 GB per thread (more than 64GB in total). My answer for virtualized instances : when you do make the unsecured choice of polling your ram on KVM (or alternative) then the above works with 64GB ram. This is a bad idea. This is real world experience and I understand that theory is suggesting very different perfomances namely : what?!!! 10MB for 2GB ram?!!!. yep. Carlos. On 7/10/24 12:32 AM, Osservatorio Nessuno via tor-relays wrote:
Hi everyone, we are planning to get some hardware to run a physical Tor exit node, starting with a 1Gbps dedicated, unmetered uplink (10Gbps downlink). We will also route a /24 on it, so we will have large availability of addresses to run multiple instances. We have been running a few exit nodes so far, but never on our own hardware.
Which is the bandwith limit per core/Tore instance? Or what can we expect to be the bottleneck?
Due to some other requirements we need for some experiments (SFP ports, coreboot support, etc) we can mainly choose between these 2 CPUs: Intel i5-1235U Intel i7-1255U
The cost between the two models is significant enough in our case to pick the i7 only if it's really useful.
In both cases with 32GB of DDR5 RAM (we can max to 64 if needed, but is it?).
Should this allow us to saturate the uplink?
To summarize, with this bandwith, this hardware and a /24 how many Tor exit nodes should be ideal to run considering that each of them could have their own address?
Thanks! _______________________________________________ tor-relays mailing list tor-relays@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays
-- PGP updated every second week : please actualize our communication every time.
participants (5)
-
eff_03675549＠posteo.se -
lists＠for-privacy.net -
mail＠nothingtohide.nl -
Osservatorio Nessuno -
Toralf Förster