[tor-relays] relay memory leak?

[David Goulet]

On 22 Jul (23:28:51), Fran via tor-relays wrote:
> Hey,
> 
> new non-exit relay, Debian 11, tor 0.4.7.8-1~d11.bullseye+1, ~ 1 week old
> (-> no guard)
> 
> KVM VM with atm 4 cores, host passthrough AMD EPYC (-> AES HW accel.).
> 
> As can be seen at the attached screenshots memory consumption is irritating
> as well as the quite high CPU load.
> 
> All was fine when it had ~100 Mbit/s but then onion skins exploded (110 per
> second -> up to 4k per second) as well as CPU and memory.
> 
> 
> Tor complains:
> 
> > Your computer is too slow to handle this many circuit creation requests!
> Please consider using the MaxAdvertisedBandwidth config option or choosing a
> more restricted exit policy.
> 
> 
> And from time to time memory killer takes action
> 
> 
> torrc is pretty basic:
> 
> Nickname 123
> ContactInfo 123
> RunAsDaemon 1
> Log notice syslog
> RelayBandwidthRate 2X MBytes
> RelayBandwidthBurst 2X MBytes
> SocksPort 0
> ControlSocket 0
> CookieAuthentication 0
> AvoidDiskWrites 1
> Address  xxxx
> OutboundBindAddress yyyy
> ORPort xxxx:yyy
> Address  [zzzz]
> OutboundBindAddress [zzz]
> ORPort [zzz]:xxx
> MetricsPort hhhh:sss
> MetricsPortPolicy accept fffffff
> DirPort yy
> Sandbox 1
> NoExec 1
> CellStatistics 1
> ExtraInfoStatistics 1
> ConnDirectionStatistics 1
> EntryStatistics 1
> ExitPortStatistics 1
> HiddenServiceStatistics 1
> 
> 
> Ideas/suggestions (apart from limiting BW) to fix this?

We are currently seeing huge memory pressure on relays. I was unsuccessful at
finding any kind of memory leaks at the moment and so there is a distinct
possibility that relays have been accumulating somehow legit memory. We are
still heavily investigating all this and coming up with ways to reduce the
footprint.

In the meantime, we know that the "CellStatistics" option is very very memory
hungry and so you could disable that one and see if this stabilizes thing for
you.

The other option that can help with memory pressure usually is the
"MaxMemInQueues" (man 1 tor). Essentially, it tells "tor" when to start
running its "out of memory handler" (OOM). It is usually set around 75% of your
total memory but you could reduce it and see if this helps.

I would although, in the current network conditions, really NOT put it below
2GB. And if the OOM gets triggered too many times and you can spare memory,
bump it up to 4GB at the very least.

The current network conditions are abnormal and, often couple with other
things, creates these resource pressure on relays that we rarely experience
and so our team needs to investigate a needle in a haystack when it happens.

Thanks for the report! And thanks to all to help us through these difficult
times for our relays and users.

Cheers!
David

-- 
gBLT7DYMF2Nm0xKoiSACNi4AD0YA2FrdU7AvJ0fWCVc=
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 488 bytes
Desc: not available
URL: <http://lists.torproject.org/pipermail/tor-relays/attachments/20220725/e13a54dc/attachment.sig>