Improvement of memory allocation possible?

[Watson Ladd]

The default system one should if large blocks are allocated and deallocated
at once.

On 5/11/06, Ben Wilhelm <zorba at pavlovian.net> wrote:
>
>
> Does your allocator actually return memory to the OS? Many don't, and in
> my (admittedly brief) look through the source, I don't remember seeing a
> custom allocator.
>
> If it doesn't return memory to the OS, it'll just sit at its maximum
> allocated size for all eternity, despite not using much of this memory.
> (Although your buffer-shrinking would help reduce that
> maximum-allocated-size.)
>
> -Ben
>
> Nick Mathewson wrote:
> > On Thu, May 11, 2006 at 09:21:05PM +0200, Joerg Maschtaler wrote:
> >> Hi,
> >>
> >> if it possible to add an option which allows to shrink memory buffers
> >> immediatly if they are not full? [1]
> >
> > I don't think you would want that; the CPU usage would be *insanely*
> > high.  Every time you transmitted any information at all, you'd need
> > to shrink the buffer, and then immediately re-grow the buffer the
> > buffer when you had more data to transmit.
> >
> > Right now, Tor shrinks buffers ever 60 seconds, down to the next
> > largest power of two above the largest amount of the buffer at any
> > time in the last 60 seconds.  A 60-second lag here probably does no
> > harm memory-wise, but the power-of-two thing will, on average, make
> > 25% of your buffer space unused.
> >
> > The only thing that would actually help trade cpu for RAM here won't
> > be a more frequent shrinking; instead, we'd have to switch off the
> > power-of-two buffers implementation.  But if we're going to do *that*,
> > we may as well move to an mbuf/skbuff-style implementation, and get
> > improved RAM usage and improved CPU usage at the same time.  (That
> > approach will make our SSL frame-size uniformity code a little
> > trickier, but I think we can handle that.)
> >
> >> I run an Tor server on a virtual server in which the amount of RAM is
> >> the bottleneck of this system.
> >> Since ~ 3 weeks i measure the "resident" memory allocation and the
> >> corresponding traffic of the Tor server and the thing i realize is that
> >> the allocation only shrinks if i shutdown (and restart) the Tor
> >> server. [2]
> >
> > Hm.  I should look at a breakdown of buffer size; I'll try to do that
> > later tonight, once I've had my server running for a bit.  It's
> > probably important to know whether our real problem is wedged
> > connections whose buffers get impossibly large, or buffers whose
> > capacities are larger than they have to be.
> >
> > yrs,
>
>


-- 
"Those who would give up Essential Liberty to purchase a little Temporary
Safety deserve neither  Liberty nor Safety."
-- Benjamin Franklin
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.torproject.org/pipermail/tor-talk/attachments/20060511/3806b2e9/attachment.htm>