Garbage collection of abandoned onion servers
Hello, We've built a solution to scale our onion server horizontally using Kubernetes horizontal autoscaler. It's basically an onionbalance & a set of wrappers to regenerate its config file depending on the set of running TorHiddenService instances. Each time we scale the set of servers up, we start new Tor onion servers on random addresses (that's what onionbalance expects), and their descriptors are then merged into the single one by onionbalance. When we delete old instances, their onion addresses are abandoned. We want to be good citizens and not to clutter Tor's DHT with abandoned records. My question basically is whether the Tor ecosystem automatically deletes abandoned descriptors or not. If yes, we can keep our system simpler. If not, we'll have to make the system stateful and try to reuse our existing onions as much as possible. Thanks.
On Sun, Dec 25, 2022 at 08:14:30PM +0100, p13dz p13dz via tor-onions wrote:
Hello, We've built a solution to scale our onion server horizontally using Kubernetes horizontal autoscaler. It's basically an onionbalance & a set of wrappers to regenerate its config file depending on the set of running TorHiddenService instances.
Very interesting! I wonder if you plan to release this solution as a free/open source software at some point.
Each time we scale the set of servers up, we start new Tor onion servers on random addresses (that's what onionbalance expects), and their descriptors are then merged into the single one by onionbalance. When we delete old instances, their onion addresses are abandoned. We want to be good citizens and not to clutter Tor's DHT with abandoned records. My question basically is whether the Tor ecosystem automatically deletes abandoned descriptors or not. If yes, we can keep our system simpler. If not, we'll have to make the system stateful and try to reuse our existing onions as much as possible.
TL;DR I think you can keep your system simple :) Please others correct if I'm wrong, but as far as I understand you don't have to worry about having disposable Onion Service addresses. This does not overload Tor's "DHT-like" HSDir structure (which in fact is a hash ring in constant renewal). From the [rend-spec-v3]: 2.2.2. When to publish a hidden service descriptor [WHEN-HSDESC] Hidden services periodically publish their descriptor to the responsible HSDirs. The set of responsible HSDirs is determined as specified in [WHERE-HSDESC]. Specifically, every time a hidden service publishes its descriptor, it also sets up a timer for a random time between 60 minutes and 120 minutes in the future. When the timer triggers, the hidden service needs to publish its descriptor again to the responsible HSDirs for that time period. [...] 2.2.5. Expiring hidden service descriptors [EXPIRE-DESC] Hidden services set their descriptor's "descriptor-lifetime" field to 180 minutes (3 hours). Hidden services ensure that their descriptor will remain valid in the HSDir caches, by republishing their descriptors periodically as specified in [WHEN-HSDESC]. So a descriptor vanishes from the HSDirs after a while if it's not regularly published. (See also section [HASHRING] and other sections in that document for more info). [rend-spec-v3]: https://gitlab.torproject.org/tpo/core/torspec/-/blob/main/rend-spec-v3.txt -- Silvio Rhatto pronouns he/him
On Thu, Feb 9, 2023 at 11:12 PM rhatto <rhatto@torproject.org> wrote:
On Sun, Dec 25, 2022 at 08:14:30PM +0100, p13dz p13dz via tor-onions wrote:
Hello, We've built a solution to scale our onion server horizontally using Kubernetes horizontal autoscaler. It's basically an onionbalance & a set of wrappers to regenerate its config file depending on the set of running TorHiddenService instances.
Very interesting!
I wonder if you plan to release this solution as a free/open source software at some point.
That's definitely possible. We'd need to separate it from the rest of our product, clean everything up, write docs, etc. As a side note, after running our solution in production under a few (D?)DoS attacks, we found some downsides to it (and to the Onion system limits in general): - Very low limit on the descriptor size in the Onion protocol. Every server instance needs to have its own entry in the descriptor, and at 7-10 instances the descriptor gets saturated, which makes further scaling up impossible. - The current implementation of onionbalance doesn't support automatic adjustment of the number of intro points per server instance. As an example, when I have 1 server instance running, I might want to have 6 intro points to make sure they don't become an easy target for DoS. When we scale up the service, say 3x, 3*6=18 intro points no longer fit in the descriptor, and onionbalance breaks down and stops publishing updates at all. It could be smart enough to automatically lower the "replication factor" until the message fits in the descriptor. If the number of instances is too high, it should publish at least some of them instead of nothing. At the level of the Onion protocol, it would be great to increase the number of intro points per descriptor to a higher number (say 10x from what we have today). If the Tor team could assist with that, we'd publish our autoscaler and together make Tor a cool platform for building services resilient to serious attacks. Let me know what you think.
Each time we scale the set of servers up, we start new Tor onion servers on random addresses (that's what onionbalance expects), and their descriptors are then merged into the single one by onionbalance. When we delete old instances, their onion addresses are abandoned. We want to be good citizens and not to clutter Tor's DHT with abandoned records. My question basically is whether the Tor ecosystem automatically deletes abandoned descriptors or not. If yes, we can keep our system simpler. If not, we'll have to make the system stateful and try to reuse our existing onions as much as possible.
TL;DR I think you can keep your system simple :)
Please others correct if I'm wrong, but as far as I understand you don't have to worry about having disposable Onion Service addresses.
This does not overload Tor's "DHT-like" HSDir structure (which in fact is a hash ring in constant renewal).
From the [rend-spec-v3]:
2.2.2. When to publish a hidden service descriptor [WHEN-HSDESC]
Hidden services periodically publish their descriptor to the responsible HSDirs. The set of responsible HSDirs is determined as specified in [WHERE-HSDESC].
Specifically, every time a hidden service publishes its descriptor, it also sets up a timer for a random time between 60 minutes and 120 minutes in the future. When the timer triggers, the hidden service needs to publish its descriptor again to the responsible HSDirs for that time period.
[...]
2.2.5. Expiring hidden service descriptors [EXPIRE-DESC]
Hidden services set their descriptor's "descriptor-lifetime" field to 180 minutes (3 hours). Hidden services ensure that their descriptor will remain valid in the HSDir caches, by republishing their descriptors periodically as specified in [WHEN-HSDESC].
So a descriptor vanishes from the HSDirs after a while if it's not regularly published. (See also section [HASHRING] and other sections in that document for more info).
[rend-spec-v3]: https://gitlab.torproject.org/tpo/core/torspec/-/blob/main/rend-spec-v3.txt
That's perfect, thanks!
-- Silvio Rhatto pronouns he/him
On Sat, 18 Feb 2023 at 06:53, p13dz p13dz via tor-onions < tor-onions@lists.torproject.org> wrote:
At the level of the Onion protocol, it would be great to increase the number of intro points per descriptor to a higher number (say 10x from what we have today).
Speaking as someone who experimented with that, doing so actually did not have the benefits which we thought it would. Introduction points are not actually used very much - essentially once per fresh connection attempt, and then cached locally - and having (e.g.) 10 introduction points to register didn't assist performance, in fact it made recovery times (after an outage) somewhat worse because there were more IPs to register. This latter is also part of why I stopped recommending using OnionBalance to big sites like Reddit, Twitter & the BBC; simpler instead to run 2 or 3 instances which farm out access to a proper web-tier, since those 2 or 3 instances do not actually have to do much work; this (plus the practice of descriptor caching) also implements an ersatz "round-robin"-style loadbalancing amongst the HSDir, if you tune the tor instances to republish their descriptors at different times. - alec -- https://alecmuffett.com/about
Well done! I would be interested on knowing more details about the project and eventually help with kubernetes setup, if that's somehow needed. Cheers, mane ------- Original Message ------- On Thursday, February 2nd, 2023 at 1:29 AM, p13dz p13dz via tor-onions <tor-onions@lists.torproject.org> wrote:
Hello, We've built a solution to scale our onion server horizontally using Kubernetes horizontal autoscaler. It's basically an onionbalance & a set of wrappers to regenerate its config file depending on the set of running TorHiddenService instances. Each time we scale the set of servers up, we start new Tor onion servers on random addresses (that's what onionbalance expects), and their descriptors are then merged into the single one by onionbalance. When we delete old instances, their onion addresses are abandoned. We want to be good citizens and not to clutter Tor's DHT with abandoned records. My question basically is whether the Tor ecosystem automatically deletes abandoned descriptors or not. If yes, we can keep our system simpler. If not, we'll have to make the system stateful and try to reuse our existing onions as much as possible. Thanks.
What details? On Sat, Feb 18, 2023, 1:53 AM mane via tor-onions < tor-onions@lists.torproject.org> wrote:
Well done!
I would be interested on knowing more details about the project and eventually help with kubernetes setup, if that's somehow needed.
Cheers, mane
------- Original Message ------- On Thursday, February 2nd, 2023 at 1:29 AM, p13dz p13dz via tor-onions < tor-onions@lists.torproject.org> wrote:
Hello, We've built a solution to scale our onion server horizontally using Kubernetes horizontal autoscaler. It's basically an onionbalance & a set of wrappers to regenerate its config file depending on the set of running TorHiddenService instances. Each time we scale the set of servers up, we start new Tor onion servers on random addresses (that's what onionbalance expects), and their descriptors are then merged into the single one by onionbalance. When we delete old instances, their onion addresses are abandoned. We want to be good citizens and not to clutter Tor's DHT with abandoned records. My question basically is whether the Tor ecosystem automatically deletes abandoned descriptors or not. If yes, we can keep our system simpler. If not, we'll have to make the system stateful and try to reuse our existing onions as much as possible. Thanks.
_______________________________________________ tor-onions mailing list tor-onions@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-onions
participants (5)
-
Alec Muffett -
Daniel Robinson -
mane -
p13dz p13dz -
rhatto