I wrote some tests[1] which showed behaviour I did not expect. IP_BIND_ADDRESS_NO_PORT seems to work as it should, but calling bind without it enabled turns out to be even worse than I thought.
This is what I think is happening: A successful bind() on a socket without IP_BIND_ADDRESS_NO_PORT enabled, with or without an explicit port configured, makes the assigned (or supplied) port unavailable for new connect()s (on different sockets), no matter the destination. I.e if you exhaust the entire net.ipv4.ip_local_port_range with bind() (no matter what IP you bind to!), connect() will stop working - no matter what IP you attempt to connect to. You can work around this by manually doing a bind() (with or without an explicit port, but without IP_BIND_ADDRESS_NO_PORT) on the socket before connect().
$ uname -a
Linux laptop 5.15.0-56-generic #62-Ubuntu SMP Tue Nov 22 19:54:14 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
# sysctl -w net.ipv4.ip_local_port_range="40000 40100"
$ cd server && cargo run &
$ ../connect.py
Raised RLIMIT_NOFILE softlimit from 1024 to 200000
Select test (1-6): 2
#### Test 2 ####
Error on bind: [Errno 98] Address already in use
Made 101 connections. Expected to be around 101.
Select test (1-6): 1
#### Test 1 ####
Error on connect: [Errno 99] Cannot assign requested address
Made 0 connections. Expected to be around 101.
Select test (1-6): 3
#### Test 3 ####
Error on bind: [Errno 98] Address already in use
Made 200 connections. Expected to be around 202.
What blows my mind is that after running test2, you cannot connect to anything without manually doing a bind() beforehand (as shown by test1 and test3 above)! This also means that after running test2, software like ssh stops working:
$ ssh -v
mirrors.dotsrc.org[...]
debug1: connect to address 130.225.254.116 port 22: Cannot assign requested address
When using IP_BIND_ADDRESS_NO_PORT, we don't have this problem (1 5 6 can be run in any order):
$ ./connect.py
Raised RLIMIT_NOFILE softlimit from 1024 to 200000
Select test (1-6): 5
#### Test 5 ####
Error on connect: [Errno 99] Cannot assign requested address
Made 90 connections. Expected to be around 101.
Select test (1-6): 6
#### Test 6 ####
Error on connect: [Errno 99] Cannot assign requested address
Made 180 connections. Expected to be around 202.
Select test (1-6): 1
#### Test 1 ####
Error on connect: [Errno 99] Cannot assign requested address
Made 90 connections. Expected to be around 101.
> Removing the IP_BIND_ADDRESS_NO_PORT option from Haproxy and
> *doing nothing else* is sufficient to resolve the problem.
Maybe there are other processes on the same host which calls bind() without IP_BIND_ADDRESS_NO_PORT, and blocks the ports? E.g OutboundBindAddress or similar in torrc?