[tor-relays] Getting max bandwidth out of a relay

[Jeroen Massar]

On 2013-09-11 16:48 , Moritz Bartl wrote:
> Hi Jeroen,
> 
> On 09/11/2013 02:21 PM, Jeroen Massar wrote:
>> What else is there to tune except for maybe running multiple Tor nodes
>> on the same box? Which would require it to use multiple IPs right as one
>> can only run 2 nodes on the same IP I understand.
> 
> You will start to see error messages in your logs.

You mean the endless repeats of:

Sep 10 20:37:42.000 [warn] failed to get unique circID.
Sep 10 20:37:43.000 [warn] No unused circ IDs. Failing.
Sep 10 20:37:43.000 [warn] failed to get unique circID.
Sep 10 20:37:43.000 [warn] No unused circ IDs. Failing.
Sep 10 20:37:43.000 [warn] failed to get unique circID.

? :)

Those they hit once in a while, but further it is quite quiet.

> It is very unlikely
> that you will be able to satisy the interface with just one Tor process.

Are we aware why this limit gets reached? Is is because Tor does not use
all cores or simply because the instructions needed are maxed out?

> Best I've seen is 400 MBit/s per Tor process on modern machines with AES-NI.

50mbit down + up is nowhere there yet.

Are boxes that are doing these speeds running at a CPU or a network cap?
Or maybe better asked, they do run at 100% usage of their cores or do
they just use two/three cores to the max?

If it is a CPU cap, is it because it is using only one proc or only a
few cores, or is it because the AES-NI instruction set is fully loaded,
in which case we could have partial requests going through that and
others through standard CPU/FPU math.... ? :)

For instance the manning2 node and some others are pushing quite a bit
more than 50mbit, what config/setup is that? Or is it really limited to
the 68 day thing and that traffic slowly picks up as long as cpu limits
and network limits allow?

I would at least expect huge bursts in the mean time from clients that
are fast, maybe I have to force a Tor client to use my nodes as
multi-hops and do a speed test that way.

Thus: me -> box1 -> box2 -> box3 -> hiddenservice

all under my control so that I know the network properties and am not
flooding anything.

> How long did you leave your relay up and running? Let me quote from
> Roger's recent great blog post:

That post is great indeed (saw it earlier today too), the relays are up
(that is active, I restart them once in a while for version upgrades)
for more than two weeks now, thus they have not made it to the 70 days+
range. They still need to gain 'named' flag too.

But I am wondering if that will add a lot more traffic to the box or
not, seeing that two cores are nearly maxed out by Tor.

Btw, Tor has 5G virtual and 2.6G resident memory in use at the moment...

Another nice artifact since the last update to 2.5.x branch is that nTor
connections is going up:

Sep 11 04:44:33.000 [notice] Circuit handshake stats since last time:
361258/361258 TAP, 418/418 NTor.
Sep 11 05:44:33.000 [notice] Circuit handshake stats since last time:
434515/434515 TAP, 407/407 NTor.
Sep 11 06:44:33.000 [notice] Circuit handshake stats since last time:
564997/564997 TAP, 467/467 NTor.
Sep 11 07:44:33.000 [notice] Circuit handshake stats since last time:
583556/583556 TAP, 607/607 NTor.
Sep 11 08:44:33.000 [notice] Circuit handshake stats since last time:
925657/929808 TAP, 784/784 NTor.
Sep 11 09:44:33.000 [notice] Circuit handshake stats since last time:
1394250/1600424 TAP, 923/923 NTor.
Sep 11 10:45:22.000 [notice] Circuit handshake stats since last time:
1468992/1487163 TAP, 1117/1117 NTor.
Sep 11 10:45:27.000 [notice] Heartbeat: Tor's uptime is 1 day 0:00
hours, with 54130 circuits open. I've sent 287.59 GB and received 281.98 GB.
Sep 11 10:45:27.000 [notice] Average packaged cell fullness: 99.055%
Sep 11 10:45:27.000 [notice] TLS write overhead: 10%
Sep 11 11:44:30.000 [notice] Circuit handshake stats since last time:
1276400/1746616 TAP, 1181/1182 NTor.
Sep 11 12:44:30.000 [notice] Circuit handshake stats since last time:
1150424/1808004 TAP, 1275/1275 NTor.
Sep 11 13:44:30.000 [notice] Circuit handshake stats since last time:
1095543/1959589 TAP, 1429/1429 NTor.
Sep 11 14:44:30.000 [notice] Circuit handshake stats since last time:
1111451/1971801 TAP, 1500/1501 NTor.

Thus traffic it does push and more NTor is coming in!

>> Would there maybe be a way to run multiple Tor processes with the same
>> key/identity but with a TCP load-balancer in front of it which
>> distributes the incoming connections to the processes? The only thing
>> then is that only one of them should report their details to the
>> authorities and the others should not publish; would that be possible or
>> would it mess up for instance performance stats?
> 
> I have not tried such a thing, and I don't think anyone else has.

Seems that if I have some cycles left somewhere next week I'll try to
see if I can make something like that work. Especially with a non-AES-NI
openssl it might show if we can get extra juice out of a box that way.

I wonder though a bit how the bandwidth measurements work and what data
is submitted to the DirAuths, I'll have to read up in the source on that
to see if these kind of submits could be merged between processes.

> It's very easy to run and manage multiple Tor processes, and so far
every ISP
> was able to provide more than one IP.

With gigabit-at-home becoming more common and relays not causing any
'exit' and thus possible abusive/"copyright infringing" annoyance, doing
it 'at home' might become doable[1], though I am fairly sure one of
those ISPs will slap one with a "fair usage" if one uses it that way ;)
But in most of those cases 1 IP is all one will get...

Also, from the view of the MyFamily argument, it is easier and possibly
better and clearer for the Tor network to have a single Node than having
that Node effectively spread over multiple IPs but actually being the
same node.

Greets,
 Jeroen

[1] though definitely not recommended, as losing your shiny link that
way is not what you likely want at home...