[Question to sysadmins and HS operators:] How should Hidden Services scale?
Hello people, you might have noticed that there is a redesign of hidden services going on. For a while we've been told that "hidden services don't scale" and "there is a max number of clients that a hidden service can handle" so we decided to also consider hidden service scalability as part of the upcoming redesign. Unfortunately, we are not experienced in maintaining busy hidden services so we need some help here. For example, a load balancing technique that hidden services are missing is DNS round-robin; that is, something that load balances on the IP layer (you send some clients to one IP and the rest to another IP). To do the equivalent in hidden services you have to do the load distribution in the Introduction Point layer. Some people have been thinking about this: https://lists.torproject.org/pipermail/tor-dev/2013-October/005556.html https://lists.torproject.org/pipermail/tor-dev/2013-October/005674.html Unfortunately, the above designs are not trivial to implement. They require the hidden service peers to communicate with each other so that they can generate and recycle keys, and it also puts Introduction Points in a position where they can find out the status or number of hidden service peers. For this reason we started wondering whether DNS-round-robin-like scalability is actually worth such trouble. AFAIK most big websites use DNS round-robin, but is it necessary? What about application-layer solutions like HAProxy? Do application-layer load balancing solutions exist for other (stateful) protocols (IRC, XMPP, etc.)? What should we do to make Hidden Services scale to a large number of clients? Also, is "scalability" the correct phrase here? Should we replace it with a combination of "load balancing", "high availability", or something else? [I sent this mail twice because the first did not get posted in tor-dev.]
On Fri, 20 Dec 2013, George Kadianakis wrote:
For this reason we started wondering whether DNS-round-robin-like scalability is actually worth such trouble. AFAIK most big websites use DNS round-robin, but is it necessary? What about application-layer solutions like HAProxy? Do application-layer load balancing solutions exist for other (stateful) protocols (IRC, XMPP, etc.)?
I think HAProxy and DNS round-robin solve different kinds of scalability issues: - the application running behind the hidden service is taking too much ressources for a single node. Something like HAProxy can help split the application on multiple nodes, but all traffic still go through a single node. - the network bandwidth that a single node can have or the number of connections it can handle is not enough. A solution like HAProxy doesn't help, because everything still has to go through a single node. I think the DNS round-robin is used to solve this kind of problem.
Forwarding Andrew's message here, since it was accidentally not sent to the list: Andrew said:
On Fri, Dec 20, 2013 at 03:08:01AM -0800, desnacked@riseup.net wrote 1.7K bytes in 0 lines about: : For this reason we started wondering whether DNS-round-robin-like : scalability is actually worth such trouble. AFAIK most big websites : use DNS round-robin, but is it necessary? What about application-layer : solutions like HAProxy? Do application-layer load balancing solutions : exist for other (stateful) protocols (IRC, XMPP, etc.)?
In my experience in running large websites and services, we didn't use DNS round-robin. If large sites do it themselves, versus outsourcing it to a content delivery network, they look into anycast, geoip-based proxy servers, or load balancing proxy servers (3DNS/BigIP, NetScalar, etc) DNS round-robin is for smaller websites which want to simply spread the load across redundant servers--this is what tor does now.
If scaling hidden services is going to be a large challenge and consume a lot of time, it sounds like making HS work more reliably and with stronger crypto is a better return on effort. The simple answer for scaling has been to copy around the private/public keys and host the same HS descriptors on multiple machines. I'm not sure we have seen a popular enough hidden service to warrant the need for massive scaling now.
Maybe changing HAProxy to support .onion links is a fine option too.
Also forwarding George's message. The original thread had a wrong address for tor-dev, and all their messages are not posted in tor-dev... George Kargiotakis said:
On Fri, 20 Dec 2013 11:58:27 -0500 andrew@torproject.org wrote:
On Fri, Dec 20, 2013 at 03:08:01AM -0800, desnacked@riseup.net wrote 1.7K bytes in 0 lines about: : For this reason we started wondering whether DNS-round-robin-like : scalability is actually worth such trouble. AFAIK most big websites : use DNS round-robin, but is it necessary? What about application-layer : solutions like HAProxy? Do application-layer load balancing solutions : exist for other (stateful) protocols (IRC, XMPP, etc.)?
In my experience in running large websites and services, we didn't use DNS round-robin. If large sites do it themselves, versus outsourcing it to a content delivery network, they look into anycast, geoip-based proxy servers, or load balancing proxy servers (3DNS/BigIP, NetScalar, etc) DNS round-robin is for smaller websites which want to simply spread the load across redundant servers--this is what tor does now.
If scaling hidden services is going to be a large challenge and consume a lot of time, it sounds like making HS work more reliably and with stronger crypto is a better return on effort. The simple answer for scaling has been to copy around the private/public keys and host the same HS descriptors on multiple machines. I'm not sure we have seen a popular enough hidden service to warrant the need for massive scaling now.
Maybe changing HAProxy to support .onion links is a fine option too.
Hello all,
For a while we've been told that "hidden services don't scale" and "there is a max number of clients that a hidden service can handle" so we decided to also consider hidden service scalability as part of the upcoming redesign. Unfortunately, we are not experienced in maintaining busy hidden services so we need some help here.
to solve a problem you need to strictly define it first. Where exactly is the bottleneck here ? I've never run a .onion that "couldn't scale" because of many clients visiting, so I don't have a first hand experience with such issues. If it's because it's slow to open many connections to hidden services then imho simply adding an .onion-aware HAProxy/varnish won't solve these problems in the long run. There will be a time where one HAProxy/varnish won't be enough and it will always be a SPOF.
Most big websites do geoip (to distribute the load between DCs in different regions), then they do something like HAProxy/LVS to spread the load across multiple workers in the same DC, and of course they put static files on CDNs.
Each of the above serves quite a different purpose. Reducing latency through geoip, load-balancing and graceful fail-over with LVS/HAProxy and CDNs are doing both at the same time but for different types of requests.
Since geoip does not make sense in the Tor world, maybe making multiple hosts advertise the same .onion address at the same time in the database would make some sense. If that were true, people could also implement .onion CDN services. I'm not so sure what can be done for an LVS-like setup in the Tor world though.
I hope this helps a tiny bit.
Regards, -- George Kargiotakis https://void.gr GPG KeyID: 0x897C03177011E02C
You'll probably run out of cpu crunching circuits before bandwidth or anything else. Multipublishing the same onion doesn't really work. Some services do code different onions in urls embedded in their pages to serve as their own cdn/scale but still take the first onion handoff hit. DNS over Tor does work fine with onioncat ipv6, but tor-devs seems determined to break this style of transport overlay with the new HS proposals, without providing a replacement. There is some partial utility in tcp resolvers on the host but it's not as useful as an overlay.
As some people said, another aspect of this project is increasing the 'availability' of hidden services. That is, increasing the number of nodes of a Hidden Service so that even if one is down, the rest can still handle clients. This is not something that application-layer solutions like HAProxy can solve. This even has security implications for Hidden Services since DoS attacks can lead to privacy attacks. For example, if someone suspects that a Hidden Service is hosted on a specific IP, they can shut down the IP and do a confirmation attack by checking if the HS is reachable anymore. If it's not reachable, chances are that the suspicion was correct. If we could somehow allow multiple nodes to handle introductions and rendezvous requests, we might be able to make HSes more secure against DoS confirmation attacks like the above. Unfortunately, this doesn't seem easy to do properly [0]. And of course, an attacker can still do confirmation attacks after the rendezvous is done, by sending specific amounts of data to the HS and checking the network of the suspected IP for that amount of incoming data... [0]: For example, the proposed "scalable HS" solutions allow multiple nodes to handle introductions for an HS, but in the proposed designs the same node that handles the introduction is also the one that handles the rendezvous. If an attacker suspects a specific node to be part of an HS, she can do an introduction to it, then DoS it, and see if the node ever appears in the rendezvous point. If it doesn't, then the confirmation might be successful. There are probably even more subtle attacks that an attacker can do if she wants to do a confirmation attack against an HS node during the introduction or rendezvous steps.
participants (3)
-
George Kadianakis -
grarpamp -
Nicolas Vigier