Improvement of memory allocation possible?

[Nick Mathewson]

On Thu, May 11, 2006 at 09:21:05PM +0200, Joerg Maschtaler wrote:
> Hi,
> 
> if it possible to add an option which allows to shrink memory buffers
> immediatly if they are not full? [1]

I don't think you would want that; the CPU usage would be *insanely*
high.  Every time you transmitted any information at all, you'd need
to shrink the buffer, and then immediately re-grow the buffer the
buffer when you had more data to transmit.

Right now, Tor shrinks buffers ever 60 seconds, down to the next
largest power of two above the largest amount of the buffer at any
time in the last 60 seconds.  A 60-second lag here probably does no
harm memory-wise, but the power-of-two thing will, on average, make
25% of your buffer space unused.

The only thing that would actually help trade cpu for RAM here won't
be a more frequent shrinking; instead, we'd have to switch off the
power-of-two buffers implementation.  But if we're going to do *that*,
we may as well move to an mbuf/skbuff-style implementation, and get
improved RAM usage and improved CPU usage at the same time.  (That
approach will make our SSL frame-size uniformity code a little
trickier, but I think we can handle that.)

> I run an Tor server on a virtual server in which the amount of RAM is
> the bottleneck of this system.
> Since ~ 3 weeks i measure the "resident" memory allocation and the
> corresponding traffic of the Tor server and the thing i realize is that
> the allocation only shrinks if i shutdown (and restart) the Tor
> server. [2]

Hm.  I should look at a breakdown of buffer size; I'll try to do that
later tonight, once I've had my server running for a bit.  It's
probably important to know whether our real problem is wedged
connections whose buffers get impossibly large, or buffers whose
capacities are larger than they have to be.

yrs,
-- 
Nick Mathewson
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 654 bytes
Desc: not available
URL: <http://lists.torproject.org/pipermail/tor-talk/attachments/20060511/86e66b57/attachment.pgp>