<html><head></head><body><div class="ydpec96470dyahoo-style-wrap" style="font-family:Helvetica Neue, Helvetica, Arial, sans-serif;font-size:16px;"><div><span style="color: rgb(38, 40, 42); font-size: 13px;">On Saturday, December 10, 2022, 7:23:28 AM PST, David Fifield <david@bamsoftware.com> wrote:</span><br></div><div id="ydpec96470dyahoo_quoted_1639263456" class="ydpec96470dyahoo_quoted"><div style="font-family:'Helvetica Neue', Helvetica, Arial, sans-serif;font-size:13px;color:#26282a;"><div><div id="ydpec96470dyiv5618260027"><div><div style="font-family:Helvetica Neue, Helvetica, Arial, sans-serif;font-size:16px;" class="ydpec96470dyiv5618260027ydpf69caf64yahoo-style-wrap"><div id="ydpec96470dyiv5618260027ydpf69caf64yahoo_quoted_1235353697" class="ydpec96470dyiv5618260027ydpf69caf64yahoo_quoted"><div style="font-family:'Helvetica Neue', Helvetica, Arial, sans-serif;font-size:13px;color:#26282a;"><div id="ydpec96470dyiv5618260027yqtfd90513" class="ydpec96470dyiv5618260027yqt0729580083">
<div><br clear="none"></div>
<div>On Sat, Dec 10, 2022 at 09:59:14AM +0100, Anders Trier Olesen wrote:<br clear="none"></div></div><div><div id="ydpec96470dyiv5618260027yqtfd61573" class="ydpec96470dyiv5618260027yqt0729580083">>> IP_BIND_ADDRESS_NO_PORT did not fix your somewhat similar problem in your<br clear="none">>> Haproxy setup, because all the connections are to the same dst tuple <ip, port><br clear="none">>> (i.e 127.0.0.1:ExtORPort).<br clear="none">>> The connect() system call is looking for a unique 5-tuple <protocol, srcip,<br clear="none">>> srcport, dstip, dstport>. In the Haproxy setup, the only free variable is<br clear="none">>> srcport <tcp, 127.0.0.1, srcport, 127.0.0.1, ExtORPort>, so toggling<br clear="none">>> IP_BIND_ADDRESS_NO_PORT makes no difference.<br clear="none"><br clear="none">> No—that is what I thought too, at first, but experimentally it is not<br clear="none">> the case. Removing the IP_BIND_ADDRESS_NO_PORT option from Haproxy and<br clear="none">> *doing nothing else* is sufficient to resolve the problem. Haproxy ends<br clear="none">> up binding to the same address it would have bound to with<br clear="none">> IP_BIND_ADDRESS_NO_PORT, and there are the same number of 5-tuples to<br clear="none">> the same endpoints, but the EADDRNOTAVAIL errors stop. It is<br clear="none">> counterintuitive and unexpected, which why I took the trouble to write<br clear="none">> it up.<br clear="none"><br clear="none">> As I wrote at #40201, there are divergent code paths for connect in the<br clear="none">> kernel when the port is already bound versus when it is not bound. It's<br clear="none">> not as simple as filling in blanks in a 5-tuple in otherwise identical<br clear="none">> code paths.<br clear="none"><br clear="none">> Anyway, it is not true that all connections go to the same (IP, port).<br clear="none">> (There would be no need to use a load balancer if that were the case.)<br clear="none">> At the time, we were running 12 tor processes with 12 different<br clear="none">> ExtORPorts (each ExtORPort on a different IP address, even: 127.0.3.1,<br clear="none">> 127.0.3.2, etc.). We started to have EADDRNOTAVAIL problems at around<br clear="none">> 3000 connections per ExtORPort, which is far too few to have exhausted<br clear="none">> the 5-tuple space. Please check the discussion at #40201 again, because<br clear="none">> I documented this detail there.<br clear="none"><br clear="none">> I urge you to run an experient yourself, if these observations are not<br clear="none">> what you expect. I was surprised, as well.</div><div id="ydpec96470dyiv5618260027ydpf69caf64yqtfd66090" class="ydpec96470dyiv5618260027ydpf69caf64yqt7324022753"><br clear="none"></div><div id="ydpec96470dyiv5618260027ydpf69caf64yqtfd66090" class="ydpec96470dyiv5618260027ydpf69caf64yqt7324022753"><span style="color:rgb(29, 34, 40);font-size:16px;"><br clear="none"></span></div><div id="ydpec96470dyiv5618260027ydpf69caf64yqtfd66090" class="ydpec96470dyiv5618260027ydpf69caf64yqt7324022753"><span style="color:rgb(29, 34, 40);font-size:16px;">I wonder if IP_BIND_ADDRESS_NO_PORT is better implemented in Nginx?</span></div><div id="ydpec96470dyiv5618260027ydpf69caf64yqtfd66090" class="ydpec96470dyiv5618260027ydpf69caf64yqt7324022753"><span style="color:rgb(29, 34, 40);font-size:16px;"><br></span></div><div id="ydpec96470dyiv5618260027ydpf69caf64yqtfd66090" class="ydpec96470dyiv5618260027ydpf69caf64yqt7324022753"><a href="https://www.nginx.com/blog/overcoming-ephemeral-port-exhaustion-nginx-plus/" rel="nofollow" target="_blank">https://www.nginx.com/blog/overcoming-ephemeral-port-exhaustion-nginx-plus/</a><span style="color:rgb(29, 34, 40);font-size:16px;"><br></span></div><div id="ydpec96470dyiv5618260027ydpf69caf64yqtfd66090" class="ydpec96470dyiv5618260027ydpf69caf64yqt7324022753"><span style="color:rgb(29, 34, 40);font-size:16px;"><br clear="none"></span></div><div id="ydpec96470dyiv5618260027ydpf69caf64yqtfd66090" class="ydpec96470dyiv5618260027ydpf69caf64yqt7324022753"><span style="color:rgb(29, 34, 40);font-size:16px;">Respectfully,</span></div><div id="ydpec96470dyiv5618260027ydpf69caf64yqtfd66090" class="ydpec96470dyiv5618260027ydpf69caf64yqt7324022753"><span style="color:rgb(29, 34, 40);font-size:16px;"><br clear="none"></span></div><div id="ydpec96470dyiv5618260027ydpf69caf64yqtfd66090" class="ydpec96470dyiv5618260027ydpf69caf64yqt7324022753"><span style="color:rgb(29, 34, 40);font-size:16px;"><br clear="none"></span></div><div id="ydpec96470dyiv5618260027ydpf69caf64yqtfd66090" class="ydpec96470dyiv5618260027ydpf69caf64yqt7324022753"><span style="color:rgb(29, 34, 40);font-size:16px;">Gary</span></div><div id="ydpec96470dyiv5618260027ydpf69caf64yqtfd66090" class="ydpec96470dyiv5618260027ydpf69caf64yqt7324022753"><span style="color:rgb(29, 34, 40);font-size:16px;">—</span><br clear="none" style="color:rgb(29, 34, 40);font-size:16px;"><span style="color:rgb(29, 34, 40);font-size:16px;">This Message Originated by the Sun.</span><br clear="none" style="color:rgb(29, 34, 40);font-size:16px;"><span style="color:rgb(29, 34, 40);font-size:16px;">iBigBlue 63W Solar Array (~12 Hour Charge)</span><br clear="none" style="color:rgb(29, 34, 40);font-size:16px;"><span style="color:rgb(29, 34, 40);font-size:16px;">+ 2 x Charmast 26800mAh Power Banks</span><br clear="none" style="color:rgb(29, 34, 40);font-size:16px;"><span style="color:rgb(29, 34, 40);font-size:16px;">= iPhone XS Max 512GB (~2 Weeks Charged)</span><div id="ydpec96470dyiv5618260027yqtfd65405" class="ydpec96470dyiv5618260027yqt0729580083"><br clear="none"></div></div></div><div id="ydpec96470dyiv5618260027yqtfd29582" class="ydpec96470dyiv5618260027yqt0729580083">
</div></div><div id="ydpec96470dyiv5618260027yqtfd20339" class="ydpec96470dyiv5618260027yqt0729580083">
</div></div></div></div></div></div>
</div>
</div></div></body></html>