Intent to Minimise Effort: Fallback Directory Mirrors
Hi, Fallback directory mirrors help Tor clients find the tor network, and take load off Tor directory authorities. [0] Since Tor 0.2.8, I've generated a fallback directory mirror list every 6 months or so. I'd like to minimise the amount of effort I put into this in future. (But I think it's valuable, so I won't abandon it.) I'm happy to spend half a day every 6 months to update the input lists and run automated checks on a bunch of relays [1]. But contacting relay operators to agree to go on the list (opt-in), or check their relay details, is a time-consuming process. Here's what you can do to help: * We ask relay operators to opt-in. We could ask them to opt-out (contact us to be taken off the list) instead. The opt-in list hasn't been updated for about a year, so we will have to switch to opt-outs soon to get enough good fallbacks. - If we do opt-ins, who is going to contact relay operators? - If we do opt-outs, how do we make sure operators know they can opt-out? - (And how do we make sure we get stable relays?) * I wrote down the opt-in process in [1]. Maybe it's too complicated. - Can you design a better process? * If you're good at mail merges or scripting mail, that's one way we could contact a bunch of relay operators without it being time-consuming and error-prone. - Can you do mail merges? (I can't!) * We can improve the fallback script to make it easier to use. (See [1] and [2].) - Can you write python? If anyone wants to help out with any of these things, let me know. I'm going to do a fallback directory mirror refresh some time in the next few weeks. About 12% of fallbacks have failed, and it looks like this is placing extra load on the directory authorities. [3] There's already a ticket for it [4], and I'm happy to do a minimal refresh. If we don't get 150 fallbacks from a minimal refresh, I'll switch to opt-out and see how that goes. [0]: https://trac.torproject.org/projects/tor/wiki/doc/FallbackDirectoryMirrors [1]: https://trac.torproject.org/projects/tor/wiki/doc/UpdatingFallbackDirectoryM... [2]: https://trac.torproject.org/projects/tor/query?status=accepted&status=assign... [3]: https://lists.torproject.org/pipermail/tor-relays/2017-August/012886.html [4]: https://trac.torproject.org/projects/tor/ticket/22271 T -- Tim Wilson-Brown (teor) teor2345 at gmail dot com PGP C855 6CED 5D90 A0C5 29F6 4D43 450C BA7F 968F 094B ricochet:ekmygaiu4rzgsk6n ------------------------------------------------------------------------
To reduce the amount of work required to maintain the list I'd propose to remove the emailing step (or at least reduce it to a one way email without the need to parse replies) from the process and use ContactInfo as a way to collect the intention of relay operators to opt-in/opt-out of the fallback directory mirrors list. This comes at the price of increasing the descriptor size, but (temporary) 5 chars might be ok? You can probably reduce it to 3 chars if 5 is to much. This could be something like: Add " fd:0" at the end of your contactInfo for opt-out and " fd:1" for opt-in. or f0 and f1 Once you defined a format you could simply email that info to tor-relays and known fallbackdirs and watch how well it is received by looking at contactInfo data. This way you could probably update the list with every release without requiring a lot of manual work. -- https://mastodon.social/@nusenu https://twitter.com/nusenu_
On 26 Aug 2017, at 08:30, nusenu <nusenu-lists@riseup.net> wrote:
use ContactInfo as a way to collect the intention of relay operators to opt-in/opt-out of the fallback directory mirrors list.
This is a good idea, *if* we still think an opt-in is necessary. It seems that fallbacks are widely accepted by relay operators. I've never had any fallback operator complain about the extra load. (Or anything else, for that matter.) And consensus diffs will make for even *less* load in future. So I can't see any reasons to keep running opt-ins. But it does cut out the conversation about keeping relay details the same. I wonder how much this helps fallback operators, and if we should keep it. Regardless of my level of opt-in effort, we still lose around 10% of relays on the list every 6 months. (Which is about the rate we expected, so we designed Tor to cope well with 20% loss.) If someone else wants to run the opt-in machinery: emails, conversations, and relay lists; I'm happy to support it. And I'm happy to support any process redesign. But if no-one wants to do the opt-in or implement the redesign, I'd like to try running opt-out for a while, and see if we get a similar failure rate. So let's try for a process that's as simple as possible, but still achieves the redundancy we want.
This comes at the price of increasing the descriptor size, but (temporary) 5 chars might be ok? You can probably reduce it to 3 chars if 5 is to much.
I wouldn't worry about descriptor size: compression and microdescriptors will take care of the extra bytes. T -- Tim Wilson-Brown (teor) teor2345 at gmail dot com PGP C855 6CED 5D90 A0C5 29F6 4D43 450C BA7F 968F 094B ricochet:ekmygaiu4rzgsk6n xmpp: teor at torproject dot org ------------------------------------------------------------------------
use ContactInfo as a way to collect the intention of relay operators to opt-in/opt-out of the fallback directory mirrors list. This is a good idea, *if* we still think an opt-in is necessary.
Isn't that way to collect operator intentions also useful in opt-out mode? (instead of continuing to use emails) Or how do you plan to collect the opt-out info? -- https://mastodon.social/@nusenu https://twitter.com/nusenu_
On 26/08/17 15:44, nusenu wrote:
use ContactInfo as a way to collect the intention of relay operators to opt-in/opt-out of the fallback directory mirrors list. This is a good idea, *if* we still think an opt-in is necessary.
Isn't that way to collect operator intentions also useful in opt-out mode? (instead of continuing to use emails) Or how do you plan to collect the opt-out info?
Why are emails preferred over other methods? Another options would be wrapping a script that ops can run to create a cyberpunk trac ticket and opt-out. This is probably easier than merging emails. But maybe we do not want to handle this through trac. I'd be happy to help w/ that eventually, and the fallback script. -hiro
_______________________________________________ tor-project mailing list tor-project@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-project
On 28 Aug 2017, at 18:15, Linus Nordberg <linus@torproject.org> wrote:
teor <teor2345@gmail.com> wrote Sat, 26 Aug 2017 22:39:52 +1000:
So I can't see any reasons to keep running opt-ins.
Are we listing non-exits?
Yes. Non-exits and under-used exits are the best fallback directory mirrors, because they are under less load.
I suppose those might see a radical change in their "IP reputation" when they end up on the blocklists after the next ransomware attack. Exits are already in the lists.
Yes, there is a higher risk that a fallback will end up on one of these lists. But all public Tor relays are at risk of ending up on a list like this.
On 28 Aug 2017, at 19:34, Silvia [Hiro] <hiro@torproject.org> wrote:
On 26/08/17 15:44, nusenu wrote:
use ContactInfo as a way to collect the intention of relay operators to opt-in/opt-out of the fallback directory mirrors list. This is a good idea, *if* we still think an opt-in is necessary.
Isn't that way to collect operator intentions also useful in opt-out mode? (instead of continuing to use emails) Or how do you plan to collect the opt-out info?
Why are emails preferred over other methods? Another options would be wrapping a script that ops can run to create a cyberpunk trac ticket and opt-out. This is probably easier than merging emails. But maybe we do not want to handle this through trac.
I'd be happy to help w/ that eventually, and the fallback script.
This is a good idea. At the moment, I take emails sent to me or tor-relays, and turn them into trac comments or trac tickets. If we put it directly on trac, anyone could handle the ticket. T -- Tim Wilson-Brown (teor) teor2345 at gmail dot com PGP C855 6CED 5D90 A0C5 29F6 4D43 450C BA7F 968F 094B ricochet:ekmygaiu4rzgsk6n ------------------------------------------------------------------------
Hi all, I'd like to move this thread to the tor-dev list. I made a list of Core Tor tickets at the end of this email. Let's talk about them on tor-dev. Here's some background: Rebuilding the fallback list takes a lot of effort! So I'm picking up this tor-project thread while the latest fallback rebuild is fresh in my mind. The rebuild took a day or two of my time for regular changes. And another day or two for once-off changes. We made a few changes to streamline the process: * atagar and I wrote scripts that make it easier to generate fallback lines, and get operator contact details [0] * pastly, atagar and irl helped out with code reviews and list format changes, which should make the list easier to read and parse * pastly and nickm helped out with fallback list updates and backport Having this help meant that I had time to make the process better. Thanks! But there's still more we can do:
On 29 Aug 2017, at 11:12, teor <teor2345@gmail.com> wrote:
On 28 Aug 2017, at 18:15, Linus Nordberg <linus@torproject.org> wrote:
teor <teor2345@gmail.com> wrote Sat, 26 Aug 2017 22:39:52 +1000:
So I can't see any reasons to keep running opt-ins.
Are we listing non-exits?
Yes. Non-exits and under-used exits are the best fallback directory mirrors, because they are under less load.
I suppose those might see a radical change in their "IP reputation" when they end up on the blocklists after the next ransomware attack. Exits are already in the lists.
Yes, there is a higher risk that a fallback will end up on one of these lists. But all public Tor relays are at risk of ending up on a list like this.
Because of the higher risk, we will continue to ask relay operators to opt-in [1]. So we need to find people to help with this process every 6-12 months. And make it more efficient.
On 28 Aug 2017, at 19:34, Silvia [Hiro] <hiro@torproject.org> wrote:
On 26/08/17 15:44, nusenu wrote:
use ContactInfo as a way to collect the intention of relay operators to opt-in/opt-out of the fallback directory mirrors list. This is a good idea, *if* we still think an opt-in is necessary.
Isn't that way to collect operator intentions also useful in opt-out mode? (instead of continuing to use emails) Or how do you plan to collect the opt-out info?
Why are emails preferred over other methods? Another options would be wrapping a script that ops can run to create a cyberpunk trac ticket and opt-out. This is probably easier than merging emails. But maybe we do not want to handle this through trac.
I'd be happy to help w/ that eventually, and the fallback script.
This is a good idea.
At the moment, I take emails sent to me or tor-relays, and turn them into trac comments or trac tickets. If we put it directly on trac, anyone could handle the ticket.
The most time-consuming part of the rebuild is the opt-in process. Here's what we do at the moment: [2] 0. Send an email to tor-relays to ask operators to opt-in 1. Run a script to find fallbacks whose details have changed, and email tor-relays to ask operators if their details are permanent Here are the most time-consuming steps: 2a. Receive personal emails and emails to the list with fallback details 2b. Generate an updated fallback line for each relay [0] 2c. Update the whitelist And these steps are quick: 3. Run the update script to generate a new list 4. Write a changes file 5. Announce the new list 6. Backport the list to all supported Tor versions Here are some things I'd like to change: Just have relay fingerprints in the fallback whitelist [3] The fallback script checks relay details, but it can just rely on Onionoo to say when the address last changed. This removes step 1 and step 2b. Add a torrc option and descriptor line to opt-in as a FallbackDir [4] This is inspired by nusenu's "ContactInfo" idea, and Hiro's "bot" idea. Having a descriptor line and torrc option gets us a more reliable implementation, and it's easier for relay operators to use. And there's no need for an email bot or trac bot. This removes step 2a and step 2c, for tor versions with this feature. For tor versions without this feature, we can use the simpler fingerprint list. Make the hard-coded authorities into a separate include file with a standard format [5] We'll also change the fallback file format (again, sorry!) to make them consistent. This makes it easier for stem, metrics-lib, and other Tor implementations to use the same authorities and fallbacks. This makes step 5 easier (for libraries that use fallbacks). Let's discuss these changes on the tor-dev list! T [0]: https://gitweb.torproject.org/tor.git/tree/scripts/maint/generateFallbackDir... https://gitweb.torproject.org/tor.git/tree/scripts/maint/lookupFallbackDirCo... [1]: https://trac.torproject.org/projects/tor/ticket/24789 [2]: https://trac.torproject.org/projects/tor/wiki/doc/UpdatingFallbackDirectoryM... [3]: https://trac.torproject.org/projects/tor/ticket/24838 [4]: https://trac.torproject.org/projects/tor/ticket/24839 [5]: https://trac.torproject.org/projects/tor/ticket/24818 -- Tim / teor PGP C855 6CED 5D90 A0C5 29F6 4D43 450C BA7F 968F 094B ricochet:ekmygaiu4rzgsk6n ------------------------------------------------------------------------
teor <teor2345@gmail.com> wrote Sat, 26 Aug 2017 22:39:52 +1000:
So I can't see any reasons to keep running opt-ins.
Are we listing non-exits? I suppose those might see a radical change in their "IP reputation" when they end up on the blocklists after the next ransomware attack. Exits are already in the lists.
participants (4)
-
Linus Nordberg -
nusenu -
Silvia [Hiro] -
teor