[anti-censorship-team] Need to increase number of tor instances on snowflake-01 bridge, increased usage since yesterday

[Linus Nordberg]

David Fifield <david at bamsoftware.com> wrote
Tue, 27 Sep 2022 08:54:53 -0600:

> On Mon, Sep 26, 2022 at 10:39:42AM +0200, Linus Nordberg via anti-censorship-team wrote:
>> It seems likely that we're hitting a limit of some sort and next thing
>> is to figure out if it's a soft limit that we can influence through
>> system configuration or if it's a hardware resource limit.
>
> tor has a default bandwidth limit, but we should be nowhere close to it,
> especially disitributed across 12 instances:
>
>     BandwidthRate N bytes|KBytes|MBytes|GBytes|TBytes|KBits|MBits|GBits|TBits
> 	A token bucket limits the average incoming bandwidth usage on this node
> 	to the specified number of bytes per second, and the average outgoing
> 	bandwidth usage to that same value. If you want to run a relay in the
> 	public network, this needs to be at the very least 75 KBytes for a
> 	relay (that is, 600 kbits) or 50 KBytes for a bridge (400 kbits) — but
> 	of course, more is better; we recommend at least 250 KBytes (2 mbits)
> 	if possible. (Default: 1 GByte)
>
> I do not see any rate limit enabled in /etc/haproxy/haproxy.cfg.
>
> I checked the number of sockets connected to the haproxy frontend port,
> thinking that we may be running out of localhost 4-tuples. It's still in
> bounds (but we may have to figure something out for that eventually).
>
>     # ss -n | grep -c '127.0.0.1:10000\s*$'
>     27314
>     # sysctl net.ipv4.ip_local_port_range
>     net.ipv4.ip_local_port_range = 15000    64000

Would more IP addresses and DNS round robin work?


> According to https://stackoverflow.com/a/3923785, some other parameters
> that may be important are
>
>     # sysctl net.ipv4.tcp_fin_timeout
>     net.ipv4.tcp_fin_timeout = 60
>     # cat /proc/sys/net/netfilter/nf_conntrack_max
>     262144

Yes, we'd better keep an eye on the conntrack count and either raise the
max or get rid of the connection tracking somehow. I've seen warnings
from netdata about the count raising above 85%.

# cat /proc/sys/net/netfilter/nf_conntrack_{count,max} 
181053
262144

One thing I would like to do soon is to hook up the other NIC and put
sshd and wireguard on that while keeping snowflake traffic on the
current 10G. That way we could start playing with ethtool to instruct
the NIC to do some fancy stuff suggested by anarcat (see below).

I've created
https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40186
to track this.


>     # sysctl net.core.netdev_max_backlog
>     net.core.netdev_max_backlog = 1000
>     Ethernet txqueuelen (1000)
>
> net.core.netdev_max_backlog is the "maximum number of packets, queued on
> the INPUT side, when the interface receives packets faster than kernel
> can process them."
> https://www.kernel.org/doc/html/latest/admin-guide/sysctl/net.html#netdev-max-backlogBut if we were having trouble with backlog buffer sizes, I would expect
> to see lots of dropped packets, and I don't:
>
>     # ethtool -S eno1 | grep dropped
>          rx_dropped: 0
>          tx_dropped: 0

Yes, the lack of drops makes me think we should look elsewhere.


> It may be something inside snowflake-server, for example some central
> scheduling algorithm that cannot run any faster. (Though if that were
> the case, I'd expect to see one CPU core at 100%, which I do not.) I
> suggest doing another round of profiling now that we have taken care of
> the more obvious hotspots in
> https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/merge_requests/100

After an interesting chat with anarcat I think that we are CPU bound and
in particular by handling so many interrupts from the NIC and dealing
with such a high number of context switches. I have two suggestions on
how to move forward with this.

First, let's patch tor to get rid of the extor processes, as suggested
by David earlier when discussing RAM pressure. This should bring down
context switches.

Second, once we've got #40186 sorted, do what's suggested in [1] to
bring the interrupt frequency down. This should take some load off the
CPU's.

[1] https://www.kernel.org/doc/html/v4.20/networking/i40e.html#interrupt-rate-limiting