Proposal: Load-balancing hidden services by splitting introduction from rendezvous
Hey all, I'd like your thoughts and comments on this proposal. Tom PS: If you want to deliver them in person, I'm in Berlin. Filename: xxx-intro-rendezvous-controlsocket.txt Title: Load-balancing hidden services by splitting introduction from rendezvous Author: Tom van der Woerdt Created: 2015-09-30 Status: draft 1. Overview and motivation To address scaling concerns with the onion web, we want to be able to spread the load of hidden services across multiple machines. OnionBalance is a great stab at this, and it can currently give us 60x the capacity by publishing 6 separate descriptors, each with 10 introduction points, but more is better. This proposal aims to address hidden service scaling up to a point where we can handle millions of concurrent connections. The basic idea involves splitting the 'introduce' from the 'rendezvous', in the tor implementation, and adding new events and commands to the control specification to allow intercepting introductions and transmitting them to different nodes, which will then take care of the actual rendezvous. External controller code could relay the data to another node or a pool of nodes, all which are run by the hidden service operator, effectively distributing the load of hidden services over multiple processes. By cleverly utilizing the current descriptor methods, we could publish up to sixty unique introduction points, which could translate to many thousands of parallel tor workers. This should allow hidden services to go multi-threaded, with a few small changes. 2. Specification We propose two additions to the control specification, of which one is an event and the other is a new command. We also introduce a new configuration option. 2.1. DisableAutomaticRendezvous configuration option The syntax is: "DisableAutomaticRendezvous" SP [1|0] CRLF This configuration option is defined to be a boolean toggle which, if set, stops the tor implementation from automatically doing a rendezvous when an INTRODUCE2 cell is received. Instead, an event will be sent to the controllers. If no controllers are present, the introduction cell should be dropped, as acting on it instead of dropping it could open a window for a DoS. For security reasons, the configuration should be made available only in the configuration files, and not as an option settable by the controller. 2.2. The "INTRODUCE" event The syntax is: "650" SP "INTRODUCE" SP RendezvousData CRLF RendezvousData = implementation-specific, but must not contain whitespace, must only contain human-readable characters, and should be no longer than 512 bytes The INTRODUCE event should contain sufficient data to allow continuing the rendezvous from another Tor instance. The exact format is left unspecified and left up to the implementation. From this follows that only matching versions can be used safely to coordinate the rendezvous of hidden service connections. 2.3. "PERFORM-RENDEZVOUS" command The syntax is: "PERFORM-RENDEZVOUS" SP RendezvousData CRLF This command allows a controller to perform a rendezvous using data received through an INTRODUCE event. The format of RendezvousData is not specified other than that it must not contain whitespace, and should be no longer than 512 bytes. 3. Compatibility and security The implementation of these methods should, ideally, not change anything in the network, and all control changes are opt-in, so this proposal is fully backwards compatible. Controllers handling this data must be careful to not leak rendezvous data to untrusted parties, as it could be used to intercept and manipulate hidden services traffic. 4. Example Let's take an example where a client (Alice) tries to contact Bob's hidden service. To do this, Bob follows the normal hidden service specification, except he sets up ten servers to do this. One of these publishes the descriptor, the others have this desabled. When the INTRODUCE2 cell arrives at the node which published the descriptor, it does not immediately try to perform the rendezvous, but instead outputs this to the controller. Through an out-of-band process this message is relayed to a controller of another node of Bob's, and this transmits the "PERFORM-RENDEZVOUS" command to that node. This node finally performs the rendezvous, and will continue to serve data to Alice, whose client will now not have to talk to the introduction point anymore. 5. Other considerations We have left the actual format of the rendezvous data in the control protocol unspecified, so that controllers do not need to worry about the various types of hidden service connections, most notably proposal 224. The decision to not implement the actual cell relaying in the tor implementation itself was taken to allow more advanced configurations, and to leave the actual load-balancing algorithm to the implementor of the controller. The developer of the tor implementation should not have to choose between a round-robin algorithm and something that could pull CPU load averages from a centralized monitoring system.
On 30 Sep 2015, at 17:27, Tom van der Woerdt <info@tvdw.eu> wrote:
...
Filename: xxx-intro-rendezvous-controlsocket.txt Title: Load-balancing hidden services by splitting introduction from rendezvous Author: Tom van der Woerdt Created: 2015-09-30 Status: draft
1. Overview and motivation
To address scaling concerns with the onion web, we want to be able to spread the load of hidden services across multiple machines. OnionBalance is a great stab at this, and it can currently give us 60x the capacity by publishing 6 separate descriptors, each with 10 introduction points, but more is better. This proposal aims to address hidden service scaling up to a point where we can handle millions of concurrent connections.
The basic idea involves splitting the 'introduce' from the 'rendezvous', in the tor implementation, and adding new events and commands to the control specification to allow intercepting introductions and transmitting them to different nodes, which will then take care of the actual rendezvous. … 2.1. DisableAutomaticRendezvous configuration option
The syntax is: "DisableAutomaticRendezvous" SP [1|0] CRLF
This configuration option is defined to be a boolean toggle which, if set, stops the tor implementation from automatically doing a rendezvous when an INTRODUCE2 cell is received. Instead, an event will be sent to the controllers. If no controllers are present, the introduction cell should be dropped, as acting on it instead of dropping it could open a window for a DoS.
For security reasons, the configuration should be made available only in the configuration files, and not as an option settable by the controller.
I’m not sure it’s necessary to prevent the controller setting this option. We trust the controller, and might need it to be able to set this option for compatibility with ephemeral hidden services. What is the threat model where a controller could set this option, but not do things that are much worse?
2.2. The "INTRODUCE" event
The syntax is: "650" SP "INTRODUCE" SP RendezvousData CRLF
RendezvousData = implementation-specific, but must not contain whitespace, must only contain human-readable characters, and should be no longer than 512 bytes
I don’t think 512 bytes is enough for the current implementation, I recommend at least 2048 bytes. (See below.)
The INTRODUCE event should contain sufficient data to allow continuing the rendezvous from another Tor instance. The exact format is left unspecified and left up to the implementation. From this follows that only matching versions can be used safely to coordinate the rendezvous of hidden service connections.
I would appreciate a list of the data needed by the current version of the hidden service protocol to rendezvous, even if we don’t want to specify the exact format, or specify data items for future implementations. This helps ensure that the limits in the proposal are sane, and that the proposal doesn’t have any unexpected implementation issues. From reading rend_service_receive_introduction think the data is at least: * service_id - the hidden service address (16 base32 bytes) * intro_key - the introduction-point specific key (128 binary bytes, 171 base64 bytes) * request - the encrypted portion of the INTRODUCE2 cell (up to 476 binary bytes(?), 635 base64 bytes) Therefore, I think the minimum for the current hidden service implementation is around 830 bytes, at least if we want to offload the maximum processing to the rendezvous instances by sending the entire encrypted INTRODUCE2 cell. Therefore, I’d suggest that a limit of 2048 bytes is much more reasonable for future-proofing this proposal. It also looks like you might need to split rend_service_t into: * introduction point-specific data * rendezvous-specific data * shared data Does any data need to be shared, and, if so, how do you intend to keep the shared data synchronised? (Putting it in the RendezvousData each time might blow out the size considerably.) I’d also appreciate an example of which parts of rend_service_receive_introduction could be performed by each of the cooperating tor instances. I assume that sending the data “as early as possible” would offload the most processing to the rendezvous side. I think that the split could happen right before the decryption of the cell, at the lines: stage_descr = "decryption"; /* Now try to decrypt it */ This would avoid having to share the intro point encrypted replay cache (intro_point->accepted_intro_rsa_parts), but there’s still the hidden service Diffie-Hellman handshake cache (service->accepted_intro_dh_parts). If we don’t share that: * two backend instances could accidentally compete for the same rendezvous point if the client times out * a client could more easily DoS the hidden service by using the same Diffie-Hellman handshake We’d have to decide if this security issue outweighs the benefit of doing the decryption on multiple rendezvous-side instances. In general, I’m concerned that we need to think through the implementation of this proposal more carefully, because it will help us decide whether it’s compatible with: * Current Hidden Services * Next-Generation Hidden Services And perhaps make changes to any of these proposals to make them work together. I’d also note that it’s definitely not compatible with Single Onion Services as specified in Proposal #252, as there is no rendezvous in that protocol. Tim Tim Wilson-Brown (teor) teor2345 at gmail dot com PGP 968F094B teor at blah dot im OTR CAD08081 9755866D 89E2A06F E3558B7F B5A9D14F
Hi Tim, Thanks for your great comments, very much appreciated! Comments inline. Op 30/09/15 om 19:40 schreef Tim Wilson-Brown - teor:
On 30 Sep 2015, at 17:27, Tom van der Woerdt <info@tvdw.eu <mailto:info@tvdw.eu>> wrote:
...
Filename: xxx-intro-rendezvous-controlsocket.txt Title: Load-balancing hidden services by splitting introduction from rendezvous Author: Tom van der Woerdt Created: 2015-09-30 Status: draft
1. Overview and motivation
To address scaling concerns with the onion web, we want to be able to spread the load of hidden services across multiple machines. OnionBalance is a great stab at this, and it can currently give us 60x the capacity by publishing 6 separate descriptors, each with 10 introduction points, but more is better. This proposal aims to address hidden service scaling up to a point where we can handle millions of concurrent connections.
The basic idea involves splitting the 'introduce' from the 'rendezvous', in the tor implementation, and adding new events and commands to the control specification to allow intercepting introductions and transmitting them to different nodes, which will then take care of the actual rendezvous. … 2.1. DisableAutomaticRendezvous configuration option
The syntax is: "DisableAutomaticRendezvous" SP [1|0] CRLF
This configuration option is defined to be a boolean toggle which, if set, stops the tor implementation from automatically doing a rendezvous when an INTRODUCE2 cell is received. Instead, an event will be sent to the controllers. If no controllers are present, the introduction cell should be dropped, as acting on it instead of dropping it could open a window for a DoS.
For security reasons, the configuration should be made available only in the configuration files, and not as an option settable by the controller.
I’m not sure it’s necessary to prevent the controller setting this option. We trust the controller, and might need it to be able to set this option for compatibility with ephemeral hidden services.
What is the threat model where a controller could set this option, but not do things that are much worse?
You're right, this addresses an irrelevant threat model.
2.2. The "INTRODUCE" event
The syntax is: "650" SP "INTRODUCE" SP RendezvousData CRLF
RendezvousData = implementation-specific, but must not contain whitespace, must only contain human-readable characters, and should be no longer than 512 bytes
I don’t think 512 bytes is enough for the current implementation, I recommend at least 2048 bytes. (See below.)
Agreed
The INTRODUCE event should contain sufficient data to allow continuing the rendezvous from another Tor instance. The exact format is left unspecified and left up to the implementation. From this follows that only matching versions can be used safely to coordinate the rendezvous of hidden service connections.
I would appreciate a list of the data needed by the current version of the hidden service protocol to rendezvous, even if we don’t want to specify the exact format, or specify data items for future implementations. This helps ensure that the limits in the proposal are sane, and that the proposal doesn’t have any unexpected implementation issues.
From reading rend_service_receive_introduction think the data is at least: * service_id - the hidden service address (16 base32 bytes) * intro_key - the introduction-point specific key (128 binary bytes, 171 base64 bytes) * request - the encrypted portion of the INTRODUCE2 cell (up to 476 binary bytes(?), 635 base64 bytes) Therefore, I think the minimum for the current hidden service implementation is around 830 bytes, at least if we want to offload the maximum processing to the rendezvous instances by sending the entire encrypted INTRODUCE2 cell. Therefore, I’d suggest that a limit of 2048 bytes is much more reasonable for future-proofing this proposal.
It also looks like you might need to split rend_service_t into: * introduction point-specific data * rendezvous-specific data * shared data Does any data need to be shared, and, if so, how do you intend to keep the shared data synchronised? (Putting it in the RendezvousData each time might blow out the size considerably.)
I’d also appreciate an example of which parts of rend_service_receive_introduction could be performed by each of the cooperating tor instances. I assume that sending the data “as early as possible” would offload the most processing to the rendezvous side. I think that the split could happen right before the decryption of the cell, at the lines: stage_descr = "decryption"; /* Now try to decrypt it */
This would avoid having to share the intro point encrypted replay cache (intro_point->accepted_intro_rsa_parts), but there’s still the hidden service Diffie-Hellman handshake cache (service->accepted_intro_dh_parts). If we don’t share that: * two backend instances could accidentally compete for the same rendezvous point if the client times out * a client could more easily DoS the hidden service by using the same Diffie-Hellman handshake We’d have to decide if this security issue outweighs the benefit of doing the decryption on multiple rendezvous-side instances.
Just spent a tiny bit of time trying to separate the functions as much as I can : https://github.com/TvdW/tor/commit/115389e1659d400eb8fcb6c2d5db3c00fb4b80e2 The prototype of the new function is : int rend_service_perform_rendezvous(rend_intro_cell_t *parsed_req, rend_service_t *service, rend_intro_point_t *intro_point, crypto_pk_t *intro_key, char *rend_pk_digest) intro_point is only needed to count how many handshakes we've seen, should probably be moved to a different location but I couldn't find a way to do that without changing the behavior. parsed_req is just the parsed cell, could easily be serialized and unserialized later. service would have to be synchronized across machines, but since it's basically a configuration struct, we can leave that up to the operator to make sure configurations are similar or equal. intro_key should probably just be transferred to the controller. Same for rend_pk_digest Post-224 a few things may need to change, as there are more keys to deal with, but imho we can just give those to the controller as well as I don't see that becoming a performance issue any time soon. As for the DH replay cache: we still perform the normal replay checks, what's the worst thing that can happen if we see the same DH data twice?
In general, I’m concerned that we need to think through the implementation of this proposal more carefully, because it will help us decide whether it’s compatible with: * Current Hidden Services * Next-Generation Hidden Services And perhaps make changes to any of these proposals to make them work together.
Thoughts welcome! I don't think I'm the right person to address those.
I’d also note that it’s definitely not compatible with Single Onion Services as specified in Proposal #252, as there is no rendezvous in that protocol.
Indeed. Tom Draft 2: Filename: TBD.txt Title: Load-balancing hidden services by splitting introduction from rendezvous Author: Tom van der Woerdt Created: 2015-09-30 Status: draft 1. Overview and motivation To address scaling concerns with the onion web, we want to be able to spread the load of hidden services across multiple machines. OnionBalance is a great stab at this, and it can currently give us 60x the capacity by publishing 6 separate descriptors, each with 10 introduction points, but more is better. This proposal aims to address hidden service scaling up to a point where we can handle millions of concurrent connections. The basic idea involves splitting the 'introduce' from the 'rendezvous', in the tor implementation, and adding new events and commands to the control specification to allow intercepting introductions and transmitting them to different nodes, which will then take care of the actual rendezvous. External controller code could relay the data to another node or a pool of nodes, all which are run by the hidden service operator, effectively distributing the load of hidden services over multiple processes. By cleverly utilizing the current descriptor methods, we could publish up to sixty unique introduction points, which could translate to many thousands of parallel tor workers. This should allow hidden services to go multi-threaded, with a few small changes. 2. Specification We propose two additions to the control specification, of which one is an event and the other is a new command. We also introduce a new configuration option. 2.1. HiddenServiceAutomaticRendezvous configuration option The syntax is: "HiddenServiceAutomaticRendezvous" SP [1|0] CRLF This configuration option is defined to be a boolean toggle which, if zero, stops the tor implementation from automatically doing a rendezvous when an INTRODUCE2 cell is received. Instead, an event will be sent to the controllers. If no controllers are present, the introduction cell should be dropped, as acting on it instead of dropping it could open a window for a DoS. This configuration option can be specified on a per-hidden service level, and can be set through the controller for ephemeral hidden services as well. 2.2. The "INTRODUCE" event The syntax is: "650" SP "INTRODUCE" SP RendezvousData CRLF RendezvousData = implementation-specific, but must not contain whitespace, must only contain human-readable characters, and should be no longer than 2048 bytes The INTRODUCE event should contain sufficient data to allow continuing the rendezvous from another Tor instance. The exact format is left unspecified and left up to the implementation. From this follows that only matching versions can be used safely to coordinate the rendezvous of hidden service connections. 2.3. "PERFORM-RENDEZVOUS" command The syntax is: "PERFORM-RENDEZVOUS" SP RendezvousData CRLF This command allows a controller to perform a rendezvous using data received through an INTRODUCE event. The format of RendezvousData is not specified other than that it must not contain whitespace, and should be no longer than 2048 bytes. 3. Compatibility and security The implementation of these methods should, ideally, not change anything in the network, and all control changes are opt-in, so this proposal is fully backwards compatible. Controllers handling this data must be careful to not leak rendezvous data to untrusted parties, as it could be used to intercept and manipulate hidden services traffic. 4. Example Let's take an example where a client (Alice) tries to contact Bob's hidden service. To do this, Bob follows the normal hidden service specification, except he sets up ten servers to do this. One of these publishes the descriptor, the others have this disabled. When the INTRODUCE2 cell arrives at the node which published the descriptor, it does not immediately try to perform the rendezvous, but instead outputs this to the controller. Through an out-of-band process this message is relayed to a controller of another node of Bob's, and this transmits the "PERFORM-RENDEZVOUS" command to that node. This node finally performs the rendezvous, and will continue to serve data to Alice, whose client will now not have to talk to the introduction point anymore. 5. Other considerations We have left the actual format of the rendezvous data in the control protocol unspecified, so that controllers do not need to worry about the various types of hidden service connections, most notably proposal 224. The decision to not implement the actual cell relaying in the tor implementation itself was taken to allow more advanced configurations, and to leave the actual load-balancing algorithm to the implementor of the controller. The developer of the tor implementation should not have to choose between a round-robin algorithm and something that could pull CPU load averages from a centralized monitoring system.
On 2 Oct 2015, at 14:43, Tom van der Woerdt <info@tvdw.eu> wrote:
Hi Tim,
Thanks for your great comments, very much appreciated!
Comments inline.
Op 30/09/15 om 19:40 schreef Tim Wilson-Brown - teor:
On 30 Sep 2015, at 17:27, Tom van der Woerdt <info@tvdw.eu <mailto:info@tvdw.eu>> wrote:
...
Filename: xxx-intro-rendezvous-controlsocket.txt Title: Load-balancing hidden services by splitting introduction from rendezvous Author: Tom van der Woerdt Created: 2015-09-30 Status: draft
1. Overview and motivation
To address scaling concerns with the onion web, we want to be able to spread the load of hidden services across multiple machines. OnionBalance is a great stab at this, and it can currently give us 60x the capacity by publishing 6 separate descriptors, each with 10 introduction points, but more is better. This proposal aims to address hidden service scaling up to a point where we can handle millions of concurrent connections.
The basic idea involves splitting the 'introduce' from the 'rendezvous', in the tor implementation, and adding new events and commands to the control specification to allow intercepting introductions and transmitting them to different nodes, which will then take care of the actual rendezvous. …
In general, I’m concerned that we need to think through the implementation of this proposal more carefully, because it will help us decide whether it’s compatible with: * Current Hidden Services * Next-Generation Hidden Services And perhaps make changes to any of these proposals to make them work together.
Thoughts welcome! I don't think I'm the right person to address those.
I’d also note that it’s definitely not compatible with Single Onion Services as specified in Proposal #252, as there is no rendezvous in that protocol.
Indeed.
Splitting the introduction and rendezvous is another use case for NAT-punching single-rendezvous-hop onion services. However, splitting hidden services into multiple different implementations provides less cover for users who really need three-hop hidden services. We’ll need to decide what the tradeoff is here. Tim Tim Wilson-Brown (teor) teor2345 at gmail dot com PGP 968F094B teor at blah dot im OTR CAD08081 9755866D 89E2A06F E3558B7F B5A9D14F
Op 02/10/15 om 14:56 schreef Tim Wilson-Brown - teor:
On 2 Oct 2015, at 14:43, Tom van der Woerdt <info@tvdw.eu <mailto:info@tvdw.eu>> wrote:
Hi Tim,
Thanks for your great comments, very much appreciated!
Comments inline.
Op 30/09/15 om 19:40 schreef Tim Wilson-Brown - teor:
On 30 Sep 2015, at 17:27, Tom van der Woerdt <info@tvdw.eu <mailto:info@tvdw.eu> <mailto:info@tvdw.eu>> wrote:
...
Filename: xxx-intro-rendezvous-controlsocket.txt Title: Load-balancing hidden services by splitting introduction from rendezvous Author: Tom van der Woerdt Created: 2015-09-30 Status: draft
1. Overview and motivation
To address scaling concerns with the onion web, we want to be able to spread the load of hidden services across multiple machines. OnionBalance is a great stab at this, and it can currently give us 60x the capacity by publishing 6 separate descriptors, each with 10 introduction points, but more is better. This proposal aims to address hidden service scaling up to a point where we can handle millions of concurrent connections.
The basic idea involves splitting the 'introduce' from the 'rendezvous', in the tor implementation, and adding new events and commands to the control specification to allow intercepting introductions and transmitting them to different nodes, which will then take care of the actual rendezvous. …
In general, I’m concerned that we need to think through the implementation of this proposal more carefully, because it will help us decide whether it’s compatible with: * Current Hidden Services * Next-Generation Hidden Services And perhaps make changes to any of these proposals to make them work together.
Thoughts welcome! I don't think I'm the right person to address those.
I’d also note that it’s definitely not compatible with Single Onion Services as specified in Proposal #252, as there is no rendezvous in that protocol.
Indeed.
Splitting the introduction and rendezvous is another use case for NAT-punching single-rendezvous-hop onion services.
However, splitting hidden services into multiple different implementations provides less cover for users who really need three-hop hidden services. We’ll need to decide what the tradeoff is here.
Tim
Tim Wilson-Brown (teor)
teor2345 at gmail dot com PGP 968F094B
teor at blah dot im OTR CAD08081 9755866D 89E2A06F E3558B7F B5A9D14F
Had some time, implemented the rest of the proposal as well to check feasibility and updated the proposal with some new insights. Looking forward to all comments on this! Patch 1, splitting the intro and rendezvous code (basically a no-op): https://github.com/TvdW/tor/commit/0443e38d776a114458e2f56e435f324c38e7a17a Patch 2, actually implementing the proposal: https://github.com/TvdW/tor/commit/b8d41f66efdb856b7813c4394f8a81c82e1f2e07 Simple controller implementation that proves my proposal works: https://gist.github.com/TvdW/3f720b9c6ffcd71967c1 Tom ================================================================== Filename: TBD.txt Title: Load-balancing hidden services by splitting introduction from rendezvous Author: Tom van der Woerdt Created: 2015-09-30 Status: draft 1. Overview and motivation To address scaling concerns with the onion web, we want to be able to spread the load of hidden services across multiple machines. OnionBalance is a great stab at this, and it can currently give us 60x the capacity by publishing 6 separate descriptors, each with 10 introduction points, but more is better. This proposal aims to address hidden service scaling up to a point where we can handle millions of concurrent connections. The basic idea involves splitting the 'introduce' from the 'rendezvous', in the tor implementation, and adding new events and commands to the control specification to allow intercepting introductions and transmitting them to different nodes, which will then take care of the actual rendezvous. External controller code could relay the data to another node or a pool of nodes, all which are run by the hidden service operator, effectively distributing the load of hidden services over multiple processes. By cleverly utilizing the current descriptor methods through OnionBalance, we could publish up to sixty unique introduction points, which could translate to many thousands of parallel tor workers after implementing this proposal. This should allow hidden services to go multi-threaded with a few small changes, and continue scaling for a long time. 2. Specification We propose two additions to the control specification, of which one is an event and the other is a new command. We also introduce two new configuration options. 2.1. HiddenServiceAutomaticRendezvous configuration option The syntax is: "HiddenServiceAutomaticRendezvous" SP [1|0] CRLF This configuration option is defined to be a boolean toggle which, if zero, stops the tor implementation from automatically doing a rendezvous when an INTRODUCE2 cell is received. Instead, an event will be sent to the controllers. If no controllers are present, the introduction cell should be dropped, as acting on it instead of dropping it could open a window for a DoS. This configuration option can be specified on a per-hidden service level, and can be set through the controller for ephemeral hidden services as well. 2.2. HiddenServiceTag configuration option The syntax is: "HiddenServiceTag" SP [a-zA-Z0-9] CRLF To identify groups of hidden services more easily across nodes, a name/tag can be given to a hidden service. Defaults to the storage path of the hidden service (HiddenServiceDir). 2.3. The "INTRODUCE" event The syntax is: "650" SP "INTRODUCE" SP HSTag SP RendezvousData CRLF HSTag = the tag of the hidden service RendezvousData = implementation-specific, but must not contain whitespace, must only contain human-readable characters, and should be no longer than 2048 bytes The INTRODUCE event should contain sufficient data to allow continuing the rendezvous from another Tor instance. The exact format is left unspecified and left up to the implementation. From this follows that only matching versions can be used safely to coordinate the rendezvous of hidden service connections. 2.4. "PERFORM-RENDEZVOUS" command The syntax is: "PERFORM-RENDEZVOUS" SP HSTag SP RendezvousData CRLF This command allows a controller to perform a rendezvous using data received through an INTRODUCE event. The format of RendezvousData is not specified other than that it must not contain whitespace, and should be no longer than 2048 bytes. 3. Compatibility and security The implementation of these methods should, ideally, not change anything in the network, and all control changes are opt-in, so this proposal is fully backwards compatible. Controllers handling this data must be careful to not leak rendezvous data to untrusted parties, as it could be used to intercept and manipulate hidden services traffic. 4. Example Let's take an example where a client (Alice) tries to contact Bob's hidden service. To do this, Bob follows the normal hidden service specification, except he sets up ten servers to do this. One of these publishes the descriptor, the others have this disabled. When the INTRODUCE2 cell arrives at the node which published the descriptor, it does not immediately try to perform the rendezvous, but instead outputs this to the controller. Through an out-of-band process this message is relayed to a controller of another node of Bob's, and this transmits the "PERFORM-RENDEZVOUS" command to that node. This node finally performs the rendezvous, and will continue to serve data to Alice, whose client will now not have to talk to the introduction point anymore. 5. Other considerations We have left the actual format of the rendezvous data in the control protocol unspecified, so that controllers do not need to worry about the various types of hidden service connections, most notably proposal 224. The decision to not implement the actual cell relaying in the tor implementation itself was taken to allow more advanced configurations, and to leave the actual load-balancing algorithm to the implementor of the controller. The developer of the tor implementation should not have to choose between a round-robin algorithm and something that could pull CPU load averages from a centralized monitoring system.
On 3 Oct 2015, at 13:34, Tom van der Woerdt <info@tvdw.eu> wrote: ... 3. Compatibility and security
The implementation of these methods should, ideally, not change anything in the network, and all control changes are opt-in, so this proposal is fully backwards compatible.
Controllers handling this data must be careful to not leak rendezvous data to untrusted parties, as it could be used to intercept and manipulate hidden services traffic.
After thinking through this, I wonder if the rendezvous data should contain the decrypted cell, rather than the introduction point key and the encrypted cell. That way, if an INTRODUCE event is exposed, only the one rendezvous referred to by the event is vulnerable. (Exposure of the introduction point key means that all introductions from that point are vulnerable until it is rotated, however, there are other layers of encryption protecting the INTRODUCE2 cells [but we shouldn’t rely on these, because we want defence-in-depth].) This is also slightly more efficient, as we are transmitting less data in the INTRODUCE event. The drawback of this change is that decryption places slightly more load on the tor instance that receives the INTRODUCE2 cell. Tim Tim Wilson-Brown (teor) teor2345 at gmail dot com PGP 968F094B teor at blah dot im OTR CAD08081 9755866D 89E2A06F E3558B7F B5A9D14F
Op 04/10/15 om 06:46 schreef Tim Wilson-Brown - teor:
On 3 Oct 2015, at 13:34, Tom van der Woerdt <info@tvdw.eu <mailto:info@tvdw.eu>> wrote: ... 3. Compatibility and security
The implementation of these methods should, ideally, not change anything in the network, and all control changes are opt-in, so this proposal is fully backwards compatible.
Controllers handling this data must be careful to not leak rendezvous data to untrusted parties, as it could be used to intercept and manipulate hidden services traffic.
After thinking through this, I wonder if the rendezvous data should contain the decrypted cell, rather than the introduction point key and the encrypted cell. That way, if an INTRODUCE event is exposed, only the one rendezvous referred to by the event is vulnerable. (Exposure of the introduction point key means that all introductions from that point are vulnerable until it is rotated, however, there are other layers of encryption protecting the INTRODUCE2 cells [but we shouldn’t rely on these, because we want defence-in-depth].)
This is also slightly more efficient, as we are transmitting less data in the INTRODUCE event.
The drawback of this change is that decryption places slightly more load on the tor instance that receives the INTRODUCE2 cell.
I don't have a particular opinion on which to pick. The proposal leaves this decision up to the implementation for exactly that reason. There are several things to consider that I can think of: * If the crypto is done on the rendezvous side, HSes could potentially scale better and be more resistant to DoSes; * With up to 60 introduction points (= processes) made possible by OnionBalance, the perf difference may not matter any time soon, and 224 should make HSes perform better anyway; * If the crypto is done on the introduction side, DH replays could be detected better, which may be good against DoSes; * The controller is considered trusted, and connections between the controllers needed for this proposal can be encrypted; * The overhead of constantly adding the same key into the events can largely be negated by using a fast compression algorithm such as Snappy -- although I don't think that these introduce events will ever become a bottleneck. Tom PS: So far this thread has been between Tim and myself... Does anyone else have an opinion? ;-)
Tom van der Woerdt <info@tvdw.eu> writes:
Op 04/10/15 om 06:46 schreef Tim Wilson-Brown - teor:
On 3 Oct 2015, at 13:34, Tom van der Woerdt <info@tvdw.eu <mailto:info@tvdw.eu>> wrote: ... 3. Compatibility and security
The implementation of these methods should, ideally, not change anything in the network, and all control changes are opt-in, so this proposal is fully backwards compatible.
Controllers handling this data must be careful to not leak rendezvous data to untrusted parties, as it could be used to intercept and manipulate hidden services traffic.
After thinking through this, I wonder if the rendezvous data should contain the decrypted cell, rather than the introduction point key and the encrypted cell. That way, if an INTRODUCE event is exposed, only the one rendezvous referred to by the event is vulnerable. (Exposure of the introduction point key means that all introductions from that point are vulnerable until it is rotated, however, there are other layers of encryption protecting the INTRODUCE2 cells [but we shouldn’t rely on these, because we want defence-in-depth].)
This is also slightly more efficient, as we are transmitting less data in the INTRODUCE event.
The drawback of this change is that decryption places slightly more load on the tor instance that receives the INTRODUCE2 cell.
I don't have a particular opinion on which to pick. The proposal leaves this decision up to the implementation for exactly that reason.
There are several things to consider that I can think of: * If the crypto is done on the rendezvous side, HSes could potentially scale better and be more resistant to DoSes; * With up to 60 introduction points (= processes) made possible by OnionBalance, the perf difference may not matter any time soon, and 224 should make HSes perform better anyway; * If the crypto is done on the introduction side, DH replays could be detected better, which may be good against DoSes; * The controller is considered trusted, and connections between the controllers needed for this proposal can be encrypted; * The overhead of constantly adding the same key into the events can largely be negated by using a fast compression algorithm such as Snappy -- although I don't think that these introduce events will ever become a bottleneck.
Tom
PS: So far this thread has been between Tim and myself... Does anyone else have an opinion? ;-)
Hello, I really like this proposal! I don't have a strong opinion on whether the decryption should happen on the master or on the slaves (need better terminology here). Some thoughts: ==== Security ===== With regards to security, both proposals seem almost the same thing. That is if decryption happens on the slaves as originally proposed, and the attacker is MITMing the master <-> slaves control channel and is able to snatch the introduction key, then yes she can decrypt all future encrypted introductions from that IP. On the alternative design where decryption happens on the master, the adversary who MITMs the control channel can just read the introductions trivially since they are already decrypted. It seems to me that in both cases the adversary wins equally if she is able to MITM the control channel, which seems to be what needs to be protected here if we even care about these attacks. Finally, what's the deal with the replay caches here? Can the replay check be done on the master if we do the "decrypt at slaves" design? IIRC the replaycache stores the decrypted data, so probably not. Is this a big problem? The proposal does not seem to be mentioning it. We probably don't want to do replay protection on the slaves, since that might leak the number of slaves we have. ==== Performance ==== Now, with regards to performance, it indeed seems slightly faster to decrypt on the slaves. However as you said, with onionbalance enabled, the operator can have multiple master nodes, which should make this not that big of a deal. ==== Simplicity ==== In general, the "decryption at slaves" design requires you to pass encrypted blobs around. I think this is suboptimal for backwards/future compatibility especially without a version field. If we end up going with this design, I'd prefer if the blob is specified a bit more. Maybe like Nick suggested here: https://trac.torproject.org/projects/tor/ticket/17254#comment:8 ==== Usefuleness ==== As I've mentioned before, if we end up going with the "decryption at master" design, we can also use this proposal to implement the "rendezvous approver" functionality of #16059, which would be quite useful under DoS scenarios. This could look like this: IP_1 \ /-- rend_slave_1 \ /--- rend_slave_2 IP_2 ---> HS master -> rendezvous approver -> /---- rend_slave_3 / \---- rend_slave_4 IP_3 / \--- rend_slave_5 \-- rend_slave_6 The "rendezvous approver" can also be implemented with the "decryption at slave" design, but the approver will need to be attached to every rendezvous slave. ==== In any case I like this proposal regardless of which design decision we choose in the end. If I were to chose right now, I'd probably pick the "decrypt at master" design decision, since it seems to be slightly better for security (replay cache), and also a bit simpler (no reason to pass any encrypted blobs anymore), as well as allowing the "rendezvous approver" functionality. Thanks for the proposal!
On Wed, Sep 30, 2015 at 11:27 AM, Tom van der Woerdt <info@tvdw.eu> wrote:
Hey all,
I'd like your thoughts and comments on this proposal.
Tom
PS: If you want to deliver them in person, I'm in Berlin.
Filename: xxx-intro-rendezvous-controlsocket.txt Title: Load-balancing hidden services by splitting introduction from rendezvous
IMO great idea. I ignored it until the Berlin meeting because the title didn't reflect what it actually does in a way I understood. Instead I would suggest a title more like: "Controller features to so hidden-service introduce2 handling to happen on a separate host from rendezvous2 sending"
Author: Tom van der Woerdt Created: 2015-09-30 Status: draft
1. Overview and motivation
To address scaling concerns with the onion web, we want to be able to spread the load of hidden services across multiple machines. OnionBalance is a great stab at this, and it can currently give us 60x the capacity by publishing 6 separate descriptors, each with 10 introduction points, but more is better. This proposal aims to address hidden service scaling up to a point where we can handle millions of concurrent connections.
The basic idea involves splitting the 'introduce' from the 'rendezvous', in the tor implementation, and adding new events and commands to the control specification to allow intercepting introductions and transmitting them to different nodes, which will then take care of the actual rendezvous. External controller code could relay the data to another node or a pool of nodes, all which are run by the hidden service operator, effectively distributing the load of hidden services over multiple processes.
By cleverly utilizing the current descriptor methods, we could publish up to sixty unique introduction points, which could translate to many thousands of parallel tor workers. This should allow hidden services to go multi-threaded, with a few small changes.
2. Specification
We propose two additions to the control specification, of which one is an event and the other is a new command. We also introduce a new configuration option.
2.1. DisableAutomaticRendezvous configuration option
The syntax is: "DisableAutomaticRendezvous" SP [1|0] CRLF
This configuration option is defined to be a boolean toggle which, if set, stops the tor implementation from automatically doing a rendezvous when an INTRODUCE2 cell is received. Instead, an event will be sent to the controllers. If no controllers are present, the introduction cell should be dropped, as acting on it instead of dropping it could open a window for a DoS.
For security reasons, the configuration should be made available only in the configuration files, and not as an option settable by the controller.
2.2. The "INTRODUCE" event
The syntax is: "650" SP "INTRODUCE" SP RendezvousData CRLF
RendezvousData = implementation-specific, but must not contain whitespace, must only contain human-readable characters, and should be no longer than 512 bytes
The INTRODUCE event should contain sufficient data to allow continuing the rendezvous from another Tor instance. The exact format is left unspecified and left up to the implementation. From this follows that only matching versions can be used safely to coordinate the rendezvous of hidden service connections.
Recommendation: Allow it to be longer than 512 bytes (futureproofing), rename it to something like "INTRODUCE_REQUEST_RECEIVED". Recommendation: Specify what it would look like as implemented for today's Tor.
2.3. "PERFORM-RENDEZVOUS" command
The syntax is: "PERFORM-RENDEZVOUS" SP RendezvousData CRLF
This command allows a controller to perform a rendezvous using data received through an INTRODUCE event. The format of RendezvousData is not specified other than that it must not contain whitespace, and should be no longer than 512 bytes.
Recommendation: Allow it to be longer than 512 bytes (futureproofing), rename it to something like "ANSWER_RENDEZVOUS". Recommendation: Specify what it would look like as implemented for today's Tor.
3. Compatibility and security
The implementation of these methods should, ideally, not change anything in the network, and all control changes are opt-in, so this proposal is fully backwards compatible.
Controllers handling this data must be careful to not leak rendezvous data to untrusted parties, as it could be used to intercept and manipulate hidden services traffic.
4. Example
Let's take an example where a client (Alice) tries to contact Bob's hidden service. To do this, Bob follows the normal hidden service specification, except he sets up ten servers to do this. One of these publishes the descriptor, the others have this desabled. When the INTRODUCE2 cell arrives at the node which published the descriptor, it does not immediately try to perform the rendezvous, but instead outputs this to the controller. Through an out-of-band process this message is relayed to a controller of another node of Bob's, and this transmits the "PERFORM-RENDEZVOUS" command to that node. This node finally performs the rendezvous, and will continue to serve data to Alice, whose client will now not have to talk to the introduction point anymore.
5. Other considerations
We have left the actual format of the rendezvous data in the control protocol unspecified, so that controllers do not need to worry about the various types of hidden service connections, most notably proposal 224.
IMO we need to specify what this looks like for current hidden services, and for hidden services under proposal 224, or else this proposal is not complete.
The decision to not implement the actual cell relaying in the tor implementation itself was taken to allow more advanced configurations, and to leave the actual load-balancing algorithm to the implementor of the controller. The developer of the tor implementation should not have to choose between a round-robin algorithm and something that could pull CPU load averages from a centralized monitoring system.
_______________________________________________ tor-dev mailing list tor-dev@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev
On Wed, Sep 30, 2015 at 05:27:28PM +0200, Tom van der Woerdt wrote:
I'd like your thoughts and comments on this proposal.
Tom
Hi Tom! I am slowly turning back into a Tor developer. I like this proposal. Here are some thoughts: - The INTRODUCE event should happen independent of the value of HiddenServiceAutomaticRendezvous. From reading the proposal I thought you meant this, but then looking at your suggested patches it looks like you only deliver the event if automaticrendezvous is disabled. The controller should have the option to just observe if it wants. (In the implementation, you might want to check if there are any controllers that care about the event, before constructing the rendezvousblob.) - We do indeed need a clearer name for INTRODUCE -- but my reason is different from the others presented. I want it clearer because in the future the *client* side could have an event when it sends its introduction cell, and that one will start out being called INTRODUCE too. So maybe HS_SERVER_INTRODUCE or something. We need to look into the future when we have events for all the various rendezvous steps, and pick a naming scheme now that won't be too terrible then. - I also have a weak preference towards having the first Tor do the decryption -- I think George's point about the replaycache is a good one. - I agree with Nick that we need a specification for what version "1" of the rendezvousblob format will look like. A specification that says it won't specify the core piece of the design is not cool. :)
If no controllers are present, the introduction cell should be dropped
So if the controller dies for some reason, the onion service will now quietly ignore all introduction requests. That's a bit sad. But I think it's still the best design -- and controllers that setconf AutomaticRendezvous to 0 can use the TAKEOWNERSHIP command to make sure Tor exits rather than remains silently useless.
Let's take an example where a client (Alice) tries to contact Bob's hidden service. To do this, Bob follows the normal hidden service specification, except he sets up ten servers to do this. One of these publishes the descriptor, the others have this desabled.
Do we also want a way to tell an onion service to not bother establishing or maintaining intro points? Maybe via letting HiddenServiceNumIntroductionPoints be 0? (I recommend not letting this idea block any of the rest of the implementation or deployment. :) --Roger
participants (5)
-
George Kadianakis -
Nick Mathewson -
Roger Dingledine -
Tim Wilson-Brown - teor -
Tom van der Woerdt