<html><head><meta http-equiv="content-type" content="text/html; charset=us-ascii"></head><body style="overflow-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;">Hello,<div><br></div><div>Thank you for this information. After 24-hours of testing, these configurations brought Tor to a halt.</div><div><br></div><div>At first I started with the sysctl modifications. After a few hours with just that, there was no improvement in ~75% inet_csk_bind_conflict utilization. I then installed Torutils for both IPv4 and IPv6. After only a couple of hours, Tor dropped to below 15 Mbps across both servers (40 relays). 16 hours later, Tor dropped below 2 Mbps.</div><div><br></div><div>I've removed all of these new settings and restarted.</div><div><br><div>
<meta charset="UTF-8"><div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; overflow-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;"><div>--</div><div>Christopher Sheats (yawnbox)</div><div>Executive Director</div><div>Emerald Onion</div><div>Signal: +1 206.739.3390</div><div>Website: https://emeraldonion.org/</div><div>Mastodon: https://digitalcourage.social/@EmeraldOnion/</div><div><br></div></div><br class="Apple-interchange-newline"><br class="Apple-interchange-newline">
</div>
<div><br><blockquote type="cite"><div>On Dec 2, 2022, at 7:30 AM, Chris <tor@wcbsecurity.com> wrote:</div><br class="Apple-interchange-newline"><div><div>Hi,<br><br>As I'm sure you've already gathered, your system is maxing out trying to<br>deal with all the connection requests. When inet_csk_get_port is called<br>and the port is found to be occupied then inet_csk_bind_conflict is<br>called to resolve the conflict. So in normal circumstances you shouldn't<br>see it in perf top much less at 79%. There are two ways to deal with it,<br>and each method should be complimented by the other. One way is to try<br>to increase the number of ports and reduce the wait time which you have<br>somehow tried. I would add the following:<br><br>net.ipv4.tcp_fin_timeout = 20<br><br>net.ipv4.tcp_max_tw_buckets = 1200<br><br>net.ipv4.tcp_keepalive_time = 1200<br><br>net.ipv4.tcp_syncookies = 1<br><br>net.ipv4.tcp_max_syn_backlog = 8192<br><br>The complimentary method to the above is to lower the number of<br>connection requests by removing the frivolous connection requests out of<br>the equation using a few iptables rules.<br><br>I'm assuming the increased load you're experiencing is due to the<br>current DDos attacks and I'm not sure if you're using anything to<br>mitigate that but you should consider it.<br><br>You may find something useful at the following links<br><br>[1](https://github.com/Enkidu-6/tor-ddos)<br><br>[2](https://github.com/toralf/torutils)<br><br>[background](https://gitlab.torproject.org/tpo/community/support/-/issues/40093)<br><br>Cheers.<br><br>On 12/1/2022 3:35 PM, Christopher Sheats wrote:<br><blockquote type="cite">Hello tor-relays,<br><br>We are using Ubuntu server currently for our exit relays.<br>Occasionally, exit throughput will drop from ~4Gbps down to ~200Mbps<br>and the only observable data point that we have is a significant<br>increase in inet_csk_bind_conflict, as seen via 'perf top', where it<br>will hit 85% [kernel] utilization.<br><br>A while back we thought we solved with with two /etc/sysctl.conf settings:<br>net.ipv4.ip_local_port_range = 1024 65535<br>net.ipv4.tcp_tw_reuse = 1<br><br>However we are still experiencing this problem.<br><br>Both of our (currently, two) relay servers suffer from the same<br>problem, at the same time. They are AMD Epyc 7402P bare-metal servers<br>each with 96GB RAM, each has 20 exit relays on them. This issue<br>persists after upgrading to 0.4.7.11.<br><br>Screenshots of perf top are shared<br>here: https://digitalcourage.social/@EmeraldOnion/109440197076214023<br><br>Does anyone have experience troubleshooting and/or fixing this problem?<br><br>Cheers,<br><br>--<br>Christopher Sheats (yawnbox)<br>Executive Director<br>Emerald Onion<br>Signal: +1 206.739.3390<br>Website: https://emeraldonion.org/<br>Mastodon: https://digitalcourage.social/@EmeraldOnion/<br><br><br><br><br><br>_______________________________________________<br>tor-relays mailing list<br>tor-relays@lists.torproject.org<br>https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays<br></blockquote></div></div></blockquote></div><br></div></body></html>