[anti-censorship-team] Need to increase number of tor instances on snowflake-01 bridge, increased usage since yesterday

[Linus Nordberg]

David Fifield <david at bamsoftware.com> wrote
Sat, 24 Sep 2022 20:14:18 -0600:

> I deployed more optimizations aimed at decreasing memory usage per
> client.
> https://bugs.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/40177

FWIW, more RAM is on its way. I hope to install another 48GB today or
tomorrow European time, doubling the amount of RAM in the system. Expect
downtime while this is being done. I will open a ticket as soon as I
know more about the timing.


> I attached a graph of interface bandwidth for the past few days.
> Outgoing bandwidth reached well over 300 MB/s on September 24. At this
> moment, traffic is approaching the daily minimum, which is still around
> 200 MB/s. We'll see what tomorrow brings. If we run into more memory
> pressure, we have another easy mitigation, which is to decrease the size
> of client send queues.
> https://bugs.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/40086#note_2837328For CPU pressure, I don't see any quick fixes. In an emergency, we could
> hack the tor binary to use a static ExtORPort authentication cookie, and
> remove the extor-static-cookie shim from the pipeline.

That might be useful also for bringing down the number of context
switches in the system. Not sure that's a problem though, will
investigate.

Possibly related, the number of packets per second seems to be bound to
400 kpps so something is going full at a nice and round figure here:
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 2022-09-25-snowflake-pps-obe.jpg
Type: image/jpeg
Size: 203440 bytes
Desc: 2022-09-25 pps
URL: <http://lists.torproject.org/pipermail/anti-censorship-team/attachments/20220926/a7647e0e/attachment-0001.jpg>
-------------- next part --------------

More granularity shows symmetric "dips" for in- and outbound with a
frequency of about 10-15 seconds:
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 2022-09-26-pps.png
Type: image/png
Size: 55060 bytes
Desc: not available
URL: <http://lists.torproject.org/pipermail/anti-censorship-team/attachments/20220926/a7647e0e/attachment-0004.png>
-------------- next part --------------


CPU interrupts, softirq and softnet show matching dips:
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 2022-09-26-interrupts.png
Type: image/png
Size: 62816 bytes
Desc: not available
URL: <http://lists.torproject.org/pipermail/anti-censorship-team/attachments/20220926/a7647e0e/attachment-0005.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 2022-09-26-softirq.png
Type: image/png
Size: 91103 bytes
Desc: not available
URL: <http://lists.torproject.org/pipermail/anti-censorship-team/attachments/20220926/a7647e0e/attachment-0006.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 2022-09-26-softnet.png
Type: image/png
Size: 108516 bytes
Desc: not available
URL: <http://lists.torproject.org/pipermail/anti-censorship-team/attachments/20220926/a7647e0e/attachment-0007.png>
-------------- next part --------------

It seems likely that we're hitting a limit of some sort and next thing
is to figure out if it's a soft limit that we can influence through
system configuration or if it's a hardware resource limit.