[anti-censorship-team] Need to increase number of tor instances on snowflake-01 bridge, increased usage since yesterday

[David Fifield]

On Thu, Sep 22, 2022 at 11:25:54PM -0600, David Fifield wrote:
> On Thu, Sep 22, 2022 at 09:24:47AM -0600, David Fifield wrote:
> > There is increased usage of the snowflake-01 bridge since yesterday,
> > likely related to protests/shutdowns in Iran. The 4 load-balanced tor
> > instances, which recently were at about 60% CPU at the steady state, are
> > currently near 100%.
> > 
> > I am planning to increase the number of instances today.
> 
> I increased the number of instances without incident:
> https://bugs.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/40173

I increased the number of instances again, from 8 to 12.
https://bugs.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/40176

> > I am also seeing a lot of "no address in clientID-to-IP map (capacity
> > 10240)" (dozens per second), so I will increase that parameter at the
> > same time.
> 
> This is among some performance changes that I hope to deploy tomorrow.
> I've actually deployed them on the snowflake-02 bridge for testing
> already.
> https://bugs.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/40086#note_2836854

I deployed more optimizations aimed at decreasing memory usage per
client.
https://bugs.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/40177

I attached a graph of interface bandwidth for the past few days.
Outgoing bandwidth reached well over 300 MB/s on September 24. At this
moment, traffic is approaching the daily minimum, which is still around
200 MB/s. We'll see what tomorrow brings. If we run into more memory
pressure, we have another easy mitigation, which is to decrease the size
of client send queues.
https://bugs.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/40086#note_2837328
For CPU pressure, I don't see any quick fixes. In an emergency, we could
hack the tor binary to use a static ExtORPort authentication cookie, and
remove the extor-static-cookie shim from the pipeline.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: snowflake-01-bw-20220924.png
Type: image/png
Size: 18685 bytes
Desc: not available
URL: <http://lists.torproject.org/pipermail/anti-censorship-team/attachments/20220924/3a613cb0/attachment-0001.png>