<html><head><meta http-equiv="content-type" content="text/html; charset=us-ascii"></head><body style="overflow-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;"><div>server1:~$ ss -s</div><div>Total: 454644</div><div>TCP: 465840 (estab 368011, closed 36634, orphaned 7619, timewait 11466)</div><div><br></div><div>Transport Total IP IPv6</div><div>RAW<span class="Apple-tab-span" style="white-space:pre"> </span> 0 0 0 </div><div>UDP<span class="Apple-tab-span" style="white-space:pre"> </span> 48 48 0 </div><div>TCP<span class="Apple-tab-span" style="white-space:pre"> </span> 429206 413815 15391 </div><div>INET<span class="Apple-tab-span" style="white-space:pre"> </span> 429254 413863 15391 </div><div>FRAG<span class="Apple-tab-span" style="white-space:pre"> </span> 0 0 0 </div><div><br></div><div>81% inet_csk_bind_conflict</div><div><br></div><div>server2:~$ ss -s</div><div>Total: 460089</div><div>TCP: 477026 (estab 367786, closed 42817, orphaned 7456, timewait 17239)</div><div><br></div><div>Transport Total IP IPv6</div><div>RAW<span class="Apple-tab-span" style="white-space:pre"> </span> 0 0 0 </div><div>UDP<span class="Apple-tab-span" style="white-space:pre"> </span> 71 71 0 </div><div>TCP<span class="Apple-tab-span" style="white-space:pre"> </span> 434209 418235 15974 </div><div>INET<span class="Apple-tab-span" style="white-space:pre"> </span> 434280 418306 15974 </div><div>FRAG<span class="Apple-tab-span" style="white-space:pre"> </span> 1 1 0 </div><div><br></div><div>80% inet_csk_bind_conflict</div><div><br></div><div>(total combined throughput at the time of measurement was ~650 Mbps symmetrical per transit provider metrics, this low throughput volume is common when <span style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0);">inet_csk_bind_conflict is this high</span>)</div><div><br></div><div><div>Re OutboundBindAddress - yes, for both v4 and v6</div><div><br></div><div>Re kernel version - 5.15.0-56-generic (jammy). Foundation for Applied Privacy recommended that we try the nightly repo which apparently includes the IP_BIND_ADDRESS_NO_PORT change. However that merge request mentions a workaround of modifying net.ipv4.ip_local_port_range, which we've already performed.</div></div><div><br></div><div>
<meta charset="UTF-8"><div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; overflow-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;"><div>--</div><div>Christopher Sheats (yawnbox)</div><div>Executive Director</div><div>Emerald Onion</div><div>Signal: +1 206.739.3390</div><div>Website: https://emeraldonion.org/</div><div>Mastodon: https://digitalcourage.social/@EmeraldOnion/</div><div><br></div></div><br class="Apple-interchange-newline"><br class="Apple-interchange-newline">
</div>
<div><br><blockquote type="cite"><div>On Dec 3, 2022, at 3:02 AM, Anders Trier Olesen <anders.trier.olesen@gmail.com> wrote:</div><br class="Apple-interchange-newline"><div><meta http-equiv="Content-Type" content="text/html; charset=utf-8"><div dir="ltr"><div dir="auto">Hi Christopher</div><div dir="auto"><br></div><div>How many open connections do you have? (`ss -s`)<br></div><div dir="auto">Do you happen to use OutboundBindAddress in your torrc?</div><div dir="auto"><br></div><div>What I think we need is for the Tor developers to include this PR in a release: <a href="https://gitlab.torproject.org/tpo/core/tor/-/merge_requests/579">https://gitlab.torproject.org/tpo/core/tor/-/merge_requests/579</a></div><div>Once that has happened, I think the problem should go away, as long as you run a recent enough Linux kernel that supports IP_BIND_ADDRESS_NO_PORT (since Linux 4.2).</div><div><br></div><div>- Anders<br></div><div dir="auto"><br></div></div><div dir="auto"><br></div><div dir="auto"><br></div><div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">fre. 2. dec. 2022 kl. 09.24 skrev Christopher Sheats <<a href="mailto:yawnbox@emeraldonion.org" target="_blank">yawnbox@emeraldonion.org</a>>:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div>Hello tor-relays,<div><br></div><div>We are using Ubuntu server currently for our exit relays. Occasionally, exit throughput will drop from ~4Gbps down to ~200Mbps and the only observable data point that we have is a significant increase in inet_csk_bind_conflict, as seen via 'perf top', where it will hit 85% [kernel] utilization.</div><div><br></div><div>A while back we thought we solved with with two /etc/sysctl.conf settings:</div><div><div>net.ipv4.ip_local_port_range = 1024 65535</div><div>net.ipv4.tcp_tw_reuse = 1</div></div><div><br></div><div>However we are still experiencing this problem.</div><div><br></div><div>Both of our (currently, two) relay servers suffer from the same problem, at the same time. They are AMD Epyc 7402P bare-metal servers each with 96GB RAM, each has 20 exit relays on them. This issue persists after upgrading to 0.4.7.11.</div><div><br></div><div>Screenshots of perf top are shared here: <a href="https://digitalcourage.social/@EmeraldOnion/109440197076214023" target="_blank">https://digitalcourage.social/@EmeraldOnion/109440197076214023</a></div><div><br></div><div>Does anyone have experience troubleshooting and/or fixing this problem?</div><div><br></div><div>Cheers,</div><div><br><div>
<div dir="auto" style="letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; text-decoration: none;"><div>--</div><div>Christopher Sheats (yawnbox)</div><div>Executive Director</div><div>Emerald Onion</div><div>Signal: +1 206.739.3390</div><div>Website: <a href="https://emeraldonion.org/" target="_blank">https://emeraldonion.org/</a></div><div>Mastodon: <a href="https://digitalcourage.social/@EmeraldOnion/" target="_blank">https://digitalcourage.social/@EmeraldOnion/</a></div><div><br></div></div><br><br>
</div>
<br></div></div>_______________________________________________<br>
tor-relays mailing list<br>
<a href="mailto:tor-relays@lists.torproject.org" target="_blank">tor-relays@lists.torproject.org</a><br>
<a href="https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays" rel="noreferrer" target="_blank">https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays</a><br>
</blockquote></div></div>
_______________________________________________<br>tor-relays mailing list<br>tor-relays@lists.torproject.org<br>https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays<br></div></blockquote></div><br></body></html>