David Fifield david@bamsoftware.com wrote Sat, 24 Sep 2022 20:14:18 -0600:
I deployed more optimizations aimed at decreasing memory usage per client. https://bugs.torproject.org/tpo/anti-censorship/pluggable-transports/snowfla...
FWIW, more RAM is on its way. I hope to install another 48GB today or tomorrow European time, doubling the amount of RAM in the system. Expect downtime while this is being done. I will open a ticket as soon as I know more about the timing.
I attached a graph of interface bandwidth for the past few days. Outgoing bandwidth reached well over 300 MB/s on September 24. At this moment, traffic is approaching the daily minimum, which is still around 200 MB/s. We'll see what tomorrow brings. If we run into more memory pressure, we have another easy mitigation, which is to decrease the size of client send queues. https://bugs.torproject.org/tpo/anti-censorship/pluggable-transports/snowfla... CPU pressure, I don't see any quick fixes. In an emergency, we could hack the tor binary to use a static ExtORPort authentication cookie, and remove the extor-static-cookie shim from the pipeline.
That might be useful also for bringing down the number of context switches in the system. Not sure that's a problem though, will investigate.
Possibly related, the number of packets per second seems to be bound to 400 kpps so something is going full at a nice and round figure here:
More granularity shows symmetric "dips" for in- and outbound with a frequency of about 10-15 seconds:
CPU interrupts, softirq and softnet show matching dips:
It seems likely that we're hitting a limit of some sort and next thing is to figure out if it's a soft limit that we can influence through system configuration or if it's a hardware resource limit.
