<html><head><meta http-equiv="content-type" content="text/html; charset=utf-8"></head><body style="overflow-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;"><div>I am happy to report that we have upgraded all our relays to Tor 0.4.8.0-alpha-dev and for the pst 8 days since the upgrade the bind conflict has ceased. No firewall rules are being used. No sysctl settings helped.</div><br><div>
<meta charset="UTF-8"><div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; overflow-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;"><div>--</div><div>Christopher Sheats (yawnbox)</div><div>Executive Director</div><div>Emerald Onion</div><div>Signal: +1 206.739.3390</div><div>Website: https://emeraldonion.org/</div><div>Mastodon: https://digitalcourage.social/@EmeraldOnion/</div><div><br></div></div><br class="Apple-interchange-newline"><br class="Apple-interchange-newline">
</div>
<div><br><blockquote type="cite"><div>On Dec 12, 2022, at 1:18 PM, Anders Trier Olesen <anders.trier.olesen@gmail.com> wrote:</div><br class="Apple-interchange-newline"><div><meta http-equiv="Content-Type" content="text/html; charset=utf-8"><div dir="ltr">>
It is surprising, isn't it? It certainly feels like calling connect<br>
> without first binding to an address should have the same effect as<br>
> manually binding to an address and then calling connect, especially if<br>
> the address you bind to is the same as the kernel would have chosen<br> > automatically. It seems like it might be a bug, but I'm not qualified to<br><div>> judge that.</div><div>Yes, I'm starting to think so too. And strange that Cloudflare doesn't mention stumbling upon this problem in their blogpost on running out of ephemeral ports. [1]<br></div><div>If I find the time, I'll make an attempt at understanding exactly what is going on in the kernel.<br></div><div><br></div><div>> If I am interpreting your results correctly, it means that either of the<br>
> two extremes is safe</div><div>Yes. That is what I think too.<br></div><div><br></div><div>>
Anyway, thank your for the insight. I apologize if I was inconsiderate<br>
> in my prior reply.</div><div>Likewise!<br></div><div><br></div><div>Best regards<br></div><div>Anders Trier Olesen<br></div><div><br></div><div>[1] <a href="https://blog.cloudflare.com/how-to-stop-running-out-of-ephemeral-ports-and-start-to-love-long-lived-connections/">https://blog.cloudflare.com/how-to-stop-running-out-of-ephemeral-ports-and-start-to-love-long-lived-connections/</a></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, Dec 12, 2022 at 4:16 PM David Fifield <<a href="mailto:david@bamsoftware.com">david@bamsoftware.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">On Mon, Dec 12, 2022 at 12:39:50AM +0100, Anders Trier Olesen wrote:<br>
> I wrote some tests[1] which showed behaviour I did not expect.<br>
> IP_BIND_ADDRESS_NO_PORT seems to work as it should, but calling bind without it<br>
> enabled turns out to be even worse than I thought.<br>
> This is what I think is happening: A successful bind() on a socket without<br>
> IP_BIND_ADDRESS_NO_PORT enabled, with or without an explicit port configured,<br>
> makes the assigned (or supplied) port unavailable for new connect()s (on<br>
> different sockets), no matter the destination. I.e if you exhaust the entire<br>
> net.ipv4.ip_local_port_range with bind() (no matter what IP you bind to!),<br>
> connect() will stop working - no matter what IP you attempt to connect to. You<br>
> can work around this by manually doing a bind() (with or without an explicit<br>
> port, but without IP_BIND_ADDRESS_NO_PORT) on the socket before connect().<br>
> <br>
> What blows my mind is that after running test2, you cannot connect to anything<br>
> without manually doing a bind() beforehand (as shown by test1 and test3 above)!<br>
> This also means that after running test2, software like ssh stops working:<br>
> <br>
> When using IP_BIND_ADDRESS_NO_PORT, we don't have this problem (1 5 6 can be<br>
> run in any order):<br>
<br>
Thank you for preparing that experiment. It's really valuable, and it<br>
looks a lot like what I was seeing on the Snowflake bridge: calls to<br>
connect would fail with EADDRNOTAVAIL unless first bound concretely to a<br>
port number. IP_BIND_ADDRESS_NO_PORT causes bind not to set a concrete<br>
port number, so in that respect it's the same as calling connect without<br>
calling bind first.<br>
<br>
It is surprising, isn't it? It certainly feels like calling connect<br>
without first binding to an address should have the same effect as<br>
manually binding to an address and then calling connect, especially if<br>
the address you bind to is the same as the kernel would have chosen<br>
automatically. It seems like it might be a bug, but I'm not qualified to<br>
judge that.<br>
<br>
If I am interpreting your results correctly, it means that either of the<br>
two extremes is safe: either everything that needs to bind to a source<br>
address should call bind with IP_BIND_ADDRESS_NO_PORT, or else<br>
everything (whether it needs a specific source address or not) should<br>
call bind *without* IP_BIND_ADDRESS_NO_PORT. (The latter situation is<br>
what we've arrived at on the Snowflake bridge.) The middle ground, where<br>
some connections use IP_BIND_ADDRESS_NO_PORT and some do not, is what<br>
causes trouble, because connections that do not use<br>
IP_BIND_ADDRESS_NO_PORT somehow "poison" the ephemeral port pool for<br>
connections that do use IP_BIND_ADDRESS_NO_PORT (and for connections<br>
that do not bind at all). It would explain why causing HAProxy not to<br>
use IP_BIND_ADDRESS_NO_PORT resolved errors in my case.<br>
<br>
> > Removing the IP_BIND_ADDRESS_NO_PORT option from Haproxy and<br>
> > *doing nothing else* is sufficient to resolve the problem.<br>
><br>
> Maybe there are other processes on the same host which calls bind() without<br>
> IP_BIND_ADDRESS_NO_PORT, and blocks the ports? E.g OutboundBindAddress or<br>
> similar in torrc?<br>
<br>
OutboundBindAddress is a likely culprit. We did end up setting<br>
OutboundBindAddress on the bridge during the period of intense<br>
performance debugging at the end of September.<br>
<br>
One thing doesn't quite add up, though. The earliest EADDRNOTAVAIL log<br>
messages started at 2022-09-28 10:57:26:<br>
<a href="https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40198" rel="noreferrer" target="_blank">https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40198</a><br>
Whereas according to the change history of /etc on the bridge,<br>
OutboundBindAddress was first set some time between 2022-09-29 21:38:37<br>
and 2022-09-29 22:37:06, over 30 hours later. I would be tempted to say<br>
this is a case of what you initially suspected, simple tuple exhaustion<br>
between two static IP addresses, if not for the fact that pre-binding an<br>
address resolved the problem in that case as well ("I get EADDRNOTAVAIL<br>
sometimes even with netcat, making a connection to the haproxy port—but<br>
not if I specify a source address in netcat"). But I only ran that<br>
netcat test after OutboundBindAddress had been set, so there may have<br>
been many factors being conflated.<br>
<br>
Anyway, thank your for the insight. I apologize if I was inconsiderate<br>
in my prior reply.<br>
_______________________________________________<br>
tor-relays mailing list<br>
<a href="mailto:tor-relays@lists.torproject.org" target="_blank">tor-relays@lists.torproject.org</a><br>
<a href="https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays" rel="noreferrer" target="_blank">https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays</a><br>
</blockquote></div>
_______________________________________________<br>tor-relays mailing list<br>tor-relays@lists.torproject.org<br>https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays<br></div></blockquote></div><br></body></html>