Forwarding Andrew's message here, since it was accidentally not sent to the list:
Andrew said:
On Fri, Dec 20, 2013 at 03:08:01AM -0800, desnacked@riseup.net wrote
1.7K bytes in 0
lines about: : For this reason we started wondering whether DNS-round-robin-like : scalability is actually worth such trouble. AFAIK most big websites : use DNS round-robin, but is it necessary? What about application-layer : solutions like HAProxy? Do application-layer load balancing solutions : exist for other (stateful) protocols (IRC, XMPP, etc.)?
In my experience in running large websites and services, we didn't use DNS round-robin. If large sites do it themselves, versus outsourcing it to a content delivery network, they look into anycast, geoip-based proxy servers, or load balancing proxy servers (3DNS/BigIP, NetScalar, etc) DNS round-robin is for smaller websites which want to simply spread the load across redundant servers--this is what tor does now.
If scaling hidden services is going to be a large challenge and consume a lot of time, it sounds like making HS work more reliably and with stronger crypto is a better return on effort. The simple answer for scaling has been to copy around the private/public keys and host the same HS descriptors on multiple machines. I'm not sure we have seen a popular enough hidden service to warrant the need for massive scaling now.
Maybe changing HAProxy to support .onion links is a fine option too.