I updated the ticket to say something like this :-)
Thanks Tim, thanks Sebastian. Pushed a fix to improve this...
https://gitweb.torproject.org/stem.git/commit/?id=66a9b77
Also, are you aware that inbound connections can come in on ORPorts that are configured like this, and therefore aren't bound to a local address at all?
ORPort 91.121.230.208:9001 ORPort [2001:41d0:d:22a6::1:0]:9001
Gotcha. Think at this point I'm willing to say 'good enough'. If this becomes an actual problem in practice we'll cross that bridge.