[tor-relays] inet_csk_bind_conflict

Mon Dec 5 20:48:55 UTC 2022

> May I ask what your set up is?
> Are you running your relays on separate VMs on the main system or are
> you using a different set up like having all IP addresses on the same OS
> and using OutboundBindAddress , routing, etc... to separate them? If I
> know more, I might be able to make a script specific to your set up.

Thank you. Yes, of course.

Ubuntu server 22.04 runs on bare metal. Ansible-relayor manages 20 exit relays on each system. Netplan has each IP individually listed (sub-divided as a /25 per server from within a dedicated /24, similarly for v6 addresses). I believe an available IP is randomly picked by ansible-relayor and used statically in each torrc file.

Here is an example torrc:

# ansible-relayor generated torrc configuration file
# Note: manual changes will be OVERWRITTEN on the next ansible-playbook run

OfflineMasterKey 1
RunAsDaemon 0
Log notice syslog
OutboundBindAddress 23.129.64.130
SocksPort 0
User _tor-23.129.64.130_443
DataDirectory /var/lib/tor-instances/23.129.64.130_443
ORPort 23.129.64.130:443
ORPort [2620:18c:0:192::130]:443
OutboundBindAddress [2620:18c:0:192::130]

DirPort 23.129.64.130:80
Address 23.129.64.130

SyslogIdentityTag 23.129.64.130_443

ControlSocket /var/run/tor-instances/23.129.64.130_443/control GroupWritable RelaxDirModeCheck

Nickname ageis
ContactInfo url:emeraldonion.org proof:uri-rsa ciissversion:2 tech at emeraldonion.org

Sandbox 1
NoExec 1

# we are an exit relay!
ExitRelay 1
IPv6Exit 1
DirPort [2620:18c:0:192::130]:80 NoAdvertise
DirPortFrontPage /etc/tor/instances/tor-exit-notice.html

ExitPolicy reject 23.129.64.128/25:*,reject6 [2613:18c:0:192::]/64:*,accept *:*,accept6 *:*

MyFamily <snip>
# end of torrc

--
Christopher Sheats (yawnbox)
Executive Director
Emerald Onion
Signal: +1 206.739.3390
Website: https://emeraldonion.org/
Mastodon: https://digitalcourage.social/@EmeraldOnion/

> On Dec 4, 2022, at 10:08 PM, Chris <tor at wcbsecurity.com> wrote:
> 
> Sorry to hear it wasn't much help. Even though the additions I suggested
> didn't help they certainly couldn't cause any harm and can't be
> responsible for the drops in traffic.
> 
> As for the torutils scripts, I'm sure toralf would be able to better
> investigate that but I have a feeling you have a certain set up that
> might not have worked with the script. May I ask what your set up is?
> Are you running your relays on separate VMs on the main system or are
> you using a different set up like having all IP addresses on the same OS
> and using OutboundBindAddress , routing, etc... to separate them? If I
> know more, I might be able to make a script specific to your set up.
> 
> On 12/3/2022 2:07 PM, Christopher Sheats wrote:
>> Hello,
>> 
>> Thank you for this information. After 24-hours of testing, these
>> configurations brought Tor to a halt.
>> 
>> At first I started with the sysctl modifications. After a few hours
>> with just that, there was no improvement in ~75%
>> inet_csk_bind_conflict utilization. I then installed Torutils for both
>> IPv4 and IPv6. After only a couple of hours, Tor dropped to below 15
>> Mbps across both servers (40 relays). 16 hours later, Tor dropped
>> below 2 Mbps.
>> 
>> I've removed all of these new settings and restarted.
>> 
>> --
>> Christopher Sheats (yawnbox)
>> Executive Director
>> Emerald Onion
>> Signal: +1 206.739.3390
>> Website: https://emeraldonion.org/
>> Mastodon: https://digitalcourage.social/@EmeraldOnion/
>> 
>> 
>> 
>> 
>>> On Dec 2, 2022, at 7:30 AM, Chris <tor at wcbsecurity.com> wrote:
>>> 
>>> Hi,
>>> 
>>> As I'm sure you've already gathered, your system is maxing out trying to
>>> deal with all the connection requests. When inet_csk_get_port is called
>>> and the port is found to be occupied then inet_csk_bind_conflict is
>>> called to resolve the conflict. So in normal circumstances you shouldn't
>>> see it in perf top much less at 79%. There are two ways to deal with it,
>>> and each method should be complimented by the other. One way is to try
>>> to increase the number of ports and reduce the wait time which you have
>>> somehow tried. I would add the following:
>>> 
>>> net.ipv4.tcp_fin_timeout = 20
>>> 
>>> net.ipv4.tcp_max_tw_buckets = 1200
>>> 
>>> net.ipv4.tcp_keepalive_time = 1200
>>> 
>>> net.ipv4.tcp_syncookies = 1
>>> 
>>> net.ipv4.tcp_max_syn_backlog = 8192
>>> 
>>> The complimentary method to the above is to lower the number of
>>> connection requests by removing the frivolous connection requests out of
>>> the equation using a few iptables rules.
>>> 
>>> I'm assuming the increased load you're experiencing is due to the
>>> current DDos attacks and I'm not sure if you're using anything to
>>> mitigate that but you should consider it.
>>> 
>>> You may find something useful at the following  links
>>> 
>>> [1](https://github.com/Enkidu-6/tor-ddos)
>>> 
>>> [2](https://github.com/toralf/torutils)
>>> 
>>> [background](https://gitlab.torproject.org/tpo/community/support/-/issues/40093)
>>> 
>>> Cheers.
>>> 
>>> On 12/1/2022 3:35 PM, Christopher Sheats wrote:
>>>> Hello tor-relays,
>>>> 
>>>> We are using Ubuntu server currently for our exit relays.
>>>> Occasionally, exit throughput will drop from ~4Gbps down to ~200Mbps
>>>> and the only observable data point that we have is a significant
>>>> increase in inet_csk_bind_conflict, as seen via 'perf top', where it
>>>> will hit 85% [kernel] utilization.
>>>> 
>>>> A while back we thought we solved with with two /etc/sysctl.conf
>>>> settings:
>>>> net.ipv4.ip_local_port_range = 1024 65535
>>>> net.ipv4.tcp_tw_reuse = 1
>>>> 
>>>> However we are still experiencing this problem.
>>>> 
>>>> Both of our (currently, two) relay servers suffer from the same
>>>> problem, at the same time. They are AMD Epyc 7402P bare-metal servers
>>>> each with 96GB RAM, each has 20 exit relays on them. This issue
>>>> persists after upgrading to 0.4.7.11.
>>>> 
>>>> Screenshots of perf top are shared
>>>> here: https://digitalcourage.social/@EmeraldOnion/109440197076214023
>>>> 
>>>> Does anyone have experience troubleshooting and/or fixing this problem?
>>>> 
>>>> Cheers,
>>>> 
>>>> --
>>>> Christopher Sheats (yawnbox)
>>>> Executive Director
>>>> Emerald Onion
>>>> Signal: +1 206.739.3390
>>>> Website: https://emeraldonion.org/
>>>> Mastodon: https://digitalcourage.social/@EmeraldOnion/
>>>> 
>>>> 
>>>> 
>>>> 
>>>> 
>>>> _______________________________________________
>>>> tor-relays mailing list
>>>> tor-relays at lists.torproject.org
>>>> https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays
>> 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.torproject.org/pipermail/tor-relays/attachments/20221205/bed0a068/attachment-0001.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: Message signed with OpenPGP
URL: <http://lists.torproject.org/pipermail/tor-relays/attachments/20221205/bed0a068/attachment-0001.sig>