I am wondering if anyone with experience in this area could advise me some on recommended specifications for a 1Gbps exit I think my latest once needs a package upgrade to handle it, currently seems stuck at around 100Mbps worth of traffic. Currently it has access to 3 v cores @1.33Ghz and 1GB of RAM and I think the latter might be becoming the limit instead now I had thought it was the CPU when it had just the one core but that alone hasn't helped.
I am not entirely sure if tor is infact using the other cores though watching with top it is sitting at 100 or a fraction over like 104% or so. MaxCPU's is set in torrc to 4 as the OS sees 4 virtual cores but is capped to 300% by the hypervisor perhaps it should be set to 3 not sure if there is any benefit in setting to 4 I have limited experience with virtual servers but unfortunately a fully dedicated server for tor is financially out of my budget at the moment.
Also, just to confirm while I'm aware it's very possible that the limitation could be bandwidth given it shares with the other VPS'es on the host I don't believe that was the case, I pulled a backup from there to another server of mine on a different AS and continent and it transferred at 250Mbps while tor was running at 100Mbps the entire time, so unless the contention is over the download which seems less likely for a server I think we should be able to get a good 300Mbps of exit capacity out of this possibly more.
So can anyone can advise on recommended CPU, RAM have to consider budget but will try and meet them, also if there are any settings I should check in torrc?
Hi Matt,
Tor indeed does not use more than one core for most of its operations, effectively limiting throughput to ~100MBps per Tor process on a non-AES-NI machine.
A CPU with AES-NI support can do up to 300-400MBps per Tor process.
You will have to run multiple Tor processes.
See https://www.torservers.net/wiki/setup/server#multiple_tor_processes for a handy initscript.
On 26.02.2013 14:46, Matt Joyce wrote:
I am wondering if anyone with experience in this area could advise me some on recommended specifications for a 1Gbps exit I think my latest once needs a package upgrade to handle it, currently seems stuck at around 100Mbps worth of traffic. Currently it has access to 3 v cores @1.33Ghz and 1GB of RAM and I think the latter might be becoming the limit instead now I had thought it was the CPU when it had just the one core but that alone hasn't helped.
I am not entirely sure if tor is infact using the other cores though watching with top it is sitting at 100 or a fraction over like 104% or so. MaxCPU's is set in torrc to 4 as the OS sees 4 virtual cores but is capped to 300% by the hypervisor perhaps it should be set to 3 not sure if there is any benefit in setting to 4 I have limited experience with virtual servers but unfortunately a fully dedicated server for tor is financially out of my budget at the moment.
Also, just to confirm while I'm aware it's very possible that the limitation could be bandwidth given it shares with the other VPS'es on the host I don't believe that was the case, I pulled a backup from there to another server of mine on a different AS and continent and it transferred at 250Mbps while tor was running at 100Mbps the entire time, so unless the contention is over the download which seems less likely for a server I think we should be able to get a good 300Mbps of exit capacity out of this possibly more.
So can anyone can advise on recommended CPU, RAM have to consider budget but will try and meet them, also if there are any settings I should check in torrc?
tor-relays mailing list tor-relays@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays
Thanks, just set it up with 3 a few hours ago, will take a while before starts advertising the full bandwidth again I guess.
I'm a little confused though for some reason only two of the instances show up in atlas, the other one just keeps complaining it isn't in the cached consensus and isn't seeing any usage either consensus health over at metrics mentions that instance but nothing more it's reporting blank for all of them what am I missing?
On 26/02/13 13:57, Moritz Bartl wrote:
Hi Matt,
Tor indeed does not use more than one core for most of its operations, effectively limiting throughput to ~100MBps per Tor process on a non-AES-NI machine.
A CPU with AES-NI support can do up to 300-400MBps per Tor process.
You will have to run multiple Tor processes.
See https://www.torservers.net/wiki/setup/server#multiple_tor_processes for a handy initscript.
On 26.02.2013 14:46, Matt Joyce wrote:
I am wondering if anyone with experience in this area could advise me some on recommended specifications for a 1Gbps exit I think my latest once needs a package upgrade to handle it, currently seems stuck at around 100Mbps worth of traffic. Currently it has access to 3 v cores @1.33Ghz and 1GB of RAM and I think the latter might be becoming the limit instead now I had thought it was the CPU when it had just the one core but that alone hasn't helped.
I am not entirely sure if tor is infact using the other cores though watching with top it is sitting at 100 or a fraction over like 104% or so. MaxCPU's is set in torrc to 4 as the OS sees 4 virtual cores but is capped to 300% by the hypervisor perhaps it should be set to 3 not sure if there is any benefit in setting to 4 I have limited experience with virtual servers but unfortunately a fully dedicated server for tor is financially out of my budget at the moment.
Also, just to confirm while I'm aware it's very possible that the limitation could be bandwidth given it shares with the other VPS'es on the host I don't believe that was the case, I pulled a backup from there to another server of mine on a different AS and continent and it transferred at 250Mbps while tor was running at 100Mbps the entire time, so unless the contention is over the download which seems less likely for a server I think we should be able to get a good 300Mbps of exit capacity out of this possibly more.
So can anyone can advise on recommended CPU, RAM have to consider budget but will try and meet them, also if there are any settings I should check in torrc?
tor-relays mailing list tor-relays@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays
On Tue, Feb 26, 2013 at 11:54:59PM +0000, Matt Joyce wrote:
I'm a little confused though for some reason only two of the instances show up in atlas, the other one just keeps complaining it isn't in the cached consensus and isn't seeing any usage either consensus health over at metrics mentions that instance but nothing more it's reporting blank for all of them what am I missing?
The tor consensus will only list at most 2 relays per IP address. So if you want to run 4 relays, put them on two IP addresses.
https://gitweb.torproject.org/torspec.git/blob/HEAD:/proposals/109-no-sharin...
--Roger
On 26/02/13 23:57, Roger Dingledine wrote:
On Tue, Feb 26, 2013 at 11:54:59PM +0000, Matt Joyce wrote:
I'm a little confused though for some reason only two of the instances show up in atlas, the other one just keeps complaining it isn't in the cached consensus and isn't seeing any usage either consensus health over at metrics mentions that instance but nothing more it's reporting blank for all of them what am I missing?
The tor consensus will only list at most 2 relays per IP address. So if you want to run 4 relays, put them on two IP addresses.
https://gitweb.torproject.org/torspec.git/blob/HEAD:/proposals/109-no-sharin...
--Roger
tor-relays mailing list tor-relays@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays
Ah, looks like I will need to look at requesting another IP address, again finding myself glad I asked the question rather than doing my usual try to figure it out myself because I was at a complete loss.
Just thought I would add a follow up on this in case you were interested to know what happened, have had the system upgrades already so now has 4 cores and 4GB of RAM, when I initially tried running 2 processes there was little gain the server only had 1GB of guaranteed RAM at the time and I'm pretty sure that was the limitation there.
After the upgrade I've already seen the two between them pumping 150Mbit in each direction with CPU usages of ~90% and ~60%, the new IP is expected to be online today so should have two further processes up shortly but I am pretty sure that configuration can probably easily achieve 400Mbit of capacity for us.
My other concern regarding what the actual network capacity after sharing might be seems to be looking pretty good after trying a few (admittedly unscientific) tests. Couple of test downloads from thinkbroadband's server in UK to the relay in Amsterdam with 8,16 and 64 threads getting a good 70-80MB/s despite the two instances already running.
Of course being a server any contention is more likely going to be on the other side, but while I can find gigabit capable servers to try pulling from finding one to try pulling from me is entirely another story. I did make a test file if anyone has the connection and 1GB of bw to try please let me know what you get http://torexit2.mttjocy.co.uk/1GBtest.bin
On 27/02/13 00:01, Matt Joyce wrote:
On 26/02/13 23:57, Roger Dingledine wrote:
On Tue, Feb 26, 2013 at 11:54:59PM +0000, Matt Joyce wrote:
I'm a little confused though for some reason only two of the instances show up in atlas, the other one just keeps complaining it isn't in the cached consensus and isn't seeing any usage either consensus health over at metrics mentions that instance but nothing more it's reporting blank for all of them what am I missing?
The tor consensus will only list at most 2 relays per IP address. So if you want to run 4 relays, put them on two IP addresses.
https://gitweb.torproject.org/torspec.git/blob/HEAD:/proposals/109-no-sharin...
--Roger
tor-relays mailing list tor-relays@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays
Ah, looks like I will need to look at requesting another IP address, again finding myself glad I asked the question rather than doing my usual try to figure it out myself because I was at a complete loss.
tor-relays mailing list tor-relays@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays
On Mon, 04 Mar 2013 18:37:01 +0000 Matt Joyce toradmin@mttjocy.co.uk allegedly wrote:
Of course being a server any contention is more likely going to be on the other side, but while I can find gigabit capable servers to try pulling from finding one to try pulling from me is entirely another story. I did make a test file if anyone has the connection and 1GB of bw to try please let me know what you get http://torexit2.mttjocy.co.uk/1GBtest.bin
Here you go: http://rlogin.net/tor/torexit2.txt
Deeply unscientific, but real world.
Mick
---------------------------------------------------------------------
blog: baldric.net gpg fingerprint: FC23 3338 F664 5E66 876B 72C0 0A1F E60B 5BAD D312
---------------------------------------------------------------------
On Mon, 04 Mar 2013 18:37:01 +0000 Matt Joyce toradmin@mttjocy.co.uk allegedly wrote:
Of course being a server any contention is more likely going to be on the other side, but while I can find gigabit capable servers to try pulling from finding one to try pulling from me is entirely another story. I did make a test file if anyone has the connection and 1GB of bw to try please let me know what you get http://torexit2.mttjocy.co.uk/1GBtest.bin
Matt
A thought. You could try for yourself using the same service I used at https://www.digitalocean.com/features if you wanted to run some more tests. Digital Ocean sell their "droplets" by the hour. So you could easily fire up a test VM for less than the cost of a coffee and doughnuts...
Mick
---------------------------------------------------------------------
blog: baldric.net gpg fingerprint: FC23 3338 F664 5E66 876B 72C0 0A1F E60B 5BAD D312
---------------------------------------------------------------------
On 05/03/13 03:29, Steve Snyder wrote:
On 02/26/2013 08:46 AM, Matt Joyce wrote:
I am wondering if anyone with experience in this area could advise me some on recommended specifications for a 1Gbps exit
[snip]
What DNS configuration will/are you using to handle the avalanche of resolution requests? _______________________________________________ tor-relays mailing list tor-relays@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays
I considered that a possible concern also I have set up a local bind 9 daemon on the server as a DNS cache and also because I believe it is probably better at handling issues if an upstream server is performing poorly especially when give multiple options for forwarding. Currently it forwards on to the hosts DNS or alternatively to google public DNS who have servers very close response times on both (8.8.8.8 and 8.8.4.4) are around 4.5msec so I'm guessing that Google likely has a facility close by probably in the Amsterdam area for the peering opportunities at AMS-IX.
The setup seems to be working well enough for now, usage is at 110Mbit/s each direction and I'm not seeing any unusual number eventdns messages, there are some but presumably tor can't tell if it is receiving a servfail reply because of a local issue or because the authoritative servers for the domain are failing which will produce the same. Not sure if it makes much of a difference as network doesn't appear to be a bottleneck but traffic prioritisation is set with TC such that UDP dpt 53 is handled at a higher priority than the outgoing TCP traffic, would rather get the small time sensitive packets on the wire than have them waiting for a batch of full sized TCP packets which I figure can better handle a que anyway, after all a TCP receiver application has to expect and hopefully sanely handle retransmission delay of 2*RTT or more.
Having the local DNS server does have a small cost the named process uses ~1-2% of a core worth of CPU time and around 2% RAM but it's fairly minor, average incoming query load calculated based on the totals for ~29h is approx 11.234qps. 7,980 SERVFAIL results from 1,169,495 queries or around 0.7% low enough I suspect most are probably failure of the authoritative server or transient reachability issues.
Query RTT Stats (Based on 1,651,610 queries sent) 541,253 queries with RTT < 10ms 616,239 queries with RTT 10-100ms 247,249 queries with RTT 100-500ms 12,565 queries with RTT 500-800ms 17,740 queries with RTT 800-1600ms 2,307 queries with RTT > 1600ms
The DNS server does have DNSSEC validation also, disabling that would almost certainly reduce load as it results in additional queries for the needed records DNSKEY, DS etc but at the moment it's working well so doesn't hurt to leave it on and hopefully help make the network that little bit more resistant to possible cache poisoning attacks as well, at least for clients that don't have their own validating resolvers which I believe still includes a number of browsers unless plugins have been installed to add it.
tor-relays@lists.torproject.org
-
Matt Joyce -
mick -
Moritz Bartl -
Roger Dingledine -
Steve Snyder