Is tor being slowed up to ~20ms per hop by TCP delayed ACK?

[John Brooks]

This pretty much mirrors my experiences. I have a tor node running on a dedi
used for other purposes as well. It's capped to 300 KB/s currently, and
marked as fast, guard, and stable by all directories, so it handles that
much traffic 24/7. Tor generally has pretty high memory usage (over 200mb
before on 0.1.x, about 100mb on 0.2.1.x), but very low CPU usage (<3% on
average). This is not on a powerful CPU by any means.

It seems pretty obvious to me that the bottleneck in tor lies mostly with
insufficient bandwidth, not overhead from encryption (minimal) or tcp
(seriously?). Improvements in those two areas could help, but probably only
in regards to reducing the load of bandwidth on nodes. What the network
really seems to need is more high-speed nodes, so more connections can be
handled without bottlenecking at some slower or too busy node. There is
probably room for improvement in the node selection algorithm, too.

I think energy would be better spent trying to recruit more node providers,
finding a way to give back to node providers, and ensuring their legal
status. Technical improvements only go so far when, by design,  tor will add
3 very busy bottleneck points to the route.

As of writing, 260 KB/s steadily coming from tor.

- John Brooks

On Thu, Jul 24, 2008 at 1:01 AM, Scott Bennett <bennett at cs.niu.edu> wrote:

>     On Wed, 23 Jul 2008 21:24:43 -0400 phobos at rootme.org wrote:
> >On Sun, Jul 20, 2008 at 10:52:44AM -0700, adam_richter2004 at yahoo.comwrote 2.5K bytes in 53 lines about:
> >: tor-0.2.1.2-alpha or libevent-1.3e.  Have I missed some other library
> >: that tor is using to avoid unnecessary Nagle delays?
> >:
> >:      I am thinking about modifying tor so that, _when it has
> >: nothing else to do_, it will set and clear TCP_NODELAY on any sockets
> >: that the outgoing data is flushed.  Outgoing TCP data would continue
> >: to be coalesced when tor has more input pending from any connection.
> >: The pushes would only be done if tor is about to go to sleep in
> >: select() and has transmitted unpushed data pending.
> >:
> >:      Am I overlooking something?  Has anyone else looked into this?
> >
> >Others have raised the possibility that tcp overhead and delays are the
> >leading cause of tor slowness, not encryption overhead.  I don't know
> >that anyone has done a rigorous analysis of tcp congestion/nagle/delays
> >over Tor.
> >
>      My server is running under FreeBSD 6.3-STABLE on a Dell Inspiron XPS,
> which has a 3.4 GHz P4 with HTT enabled and 2GB of memory.  My ISP
> connection is an ADSL currently limited to ~4.56 Mb/s reception and ~698
> Kb/s transmission.  tor input traffic varies from a few KB/s to, say, 180
> KB/s, but peak average over 10 - 15 seconds is probably ~110 KB/s.  The
> busier times usually have sustained rates running from 45 - 80 KB/s.  It
> rarely gets up to such levels for more than a few seconds until after it
> has been running long enough to get the Guard flag in consensus from the
> directory authorities, but then may spend a lot of the time in that
> activity range.  The server offers exit service for many ports, but not
> port 80.  It also offers directory mirror service, so some of the limited
> transmit bandwidth can be tied up by directory replies.  tor is configured
> with NumCPUs 2 and has 6 threads most of the time.  When my system is
> essentially inactive except for tor with its client side idle, top(1)
> usually displays 99% - 100% idle time.  It occasionally drops lower by 1%
> to 6% for as long as a few seconds, but no longer.  I don't know what it
> tor is doing at those times that is different from the rest of the time.
> Once in a great while, a couple of tor threads will chew up considerably
> more, usually while incorporating cached-descriptors.new into
> cached-descriptors.  Note that by using the system idle time as an
> indicator, all overhead required by networking activities is already
> included with the tor process's CPU time.  There is undoubtedly a certain
> amount of time consumed by other, mostly idle processes in the system,
> too, but that is so trivial that it can be safely ignored for the
> conditions described above.
>     So my impression of tor is that it is basically a highly network-bound
> application, contrary to many other reports of high CPU consumption by tor
> that I've read over time on this list.  It looks to me as though my system
> would easily handle ten times my current available bandwidth while still
> leaving 90% or more CPU time available for other activities.  Of course,
> it would be fun to try it out on a connection with huge bandwidth available
> for a few weeks to see whether those performance characteristics hold up,
> but until fiber optics installed to residences is available to me and at a
> rate affordable by me, which is not currently detectable on the horizon:-(,
> I have no way of finding out.  But it does make me curious to know whether
> my little machine could handle a load like that of blutmagie, for example,
> if only it had a 100 Mb/s or faster connection. :-)
>
>
>                                  Scott Bennett, Comm. ASMELG, CFIAG
> **********************************************************************
> * Internet:       bennett at cs.niu.edu                              *
> *--------------------------------------------------------------------*
> * "A well regulated and disciplined militia, is at all times a good  *
> * objection to the introduction of that bane of all free governments *
> * -- a standing army."                                               *
> *    -- Gov. John Hancock, New York Journal, 28 January 1790         *
> **********************************************************************
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.torproject.org/pipermail/tor-talk/attachments/20080724/0c0fb999/attachment.htm>