Re: [tor-dev] [tor-project] Intent to Minimise Effort: Fallback Directory Mirrors
Hi all, I'd like to move this thread to the tor-dev list. I made a list of Core Tor tickets at the end of this email. Let's talk about them on tor-dev. Here's some background: Rebuilding the fallback list takes a lot of effort! So I'm picking up this tor-project thread while the latest fallback rebuild is fresh in my mind. The rebuild took a day or two of my time for regular changes. And another day or two for once-off changes. We made a few changes to streamline the process: * atagar and I wrote scripts that make it easier to generate fallback lines, and get operator contact details [0] * pastly, atagar and irl helped out with code reviews and list format changes, which should make the list easier to read and parse * pastly and nickm helped out with fallback list updates and backport Having this help meant that I had time to make the process better. Thanks! But there's still more we can do:
On 29 Aug 2017, at 11:12, teor <teor2345@gmail.com> wrote:
On 28 Aug 2017, at 18:15, Linus Nordberg <linus@torproject.org> wrote:
teor <teor2345@gmail.com> wrote Sat, 26 Aug 2017 22:39:52 +1000:
So I can't see any reasons to keep running opt-ins.
Are we listing non-exits?
Yes. Non-exits and under-used exits are the best fallback directory mirrors, because they are under less load.
I suppose those might see a radical change in their "IP reputation" when they end up on the blocklists after the next ransomware attack. Exits are already in the lists.
Yes, there is a higher risk that a fallback will end up on one of these lists. But all public Tor relays are at risk of ending up on a list like this.
Because of the higher risk, we will continue to ask relay operators to opt-in [1]. So we need to find people to help with this process every 6-12 months. And make it more efficient.
On 28 Aug 2017, at 19:34, Silvia [Hiro] <hiro@torproject.org> wrote:
On 26/08/17 15:44, nusenu wrote:
use ContactInfo as a way to collect the intention of relay operators to opt-in/opt-out of the fallback directory mirrors list. This is a good idea, *if* we still think an opt-in is necessary.
Isn't that way to collect operator intentions also useful in opt-out mode? (instead of continuing to use emails) Or how do you plan to collect the opt-out info?
Why are emails preferred over other methods? Another options would be wrapping a script that ops can run to create a cyberpunk trac ticket and opt-out. This is probably easier than merging emails. But maybe we do not want to handle this through trac.
I'd be happy to help w/ that eventually, and the fallback script.
This is a good idea.
At the moment, I take emails sent to me or tor-relays, and turn them into trac comments or trac tickets. If we put it directly on trac, anyone could handle the ticket.
The most time-consuming part of the rebuild is the opt-in process. Here's what we do at the moment: [2] 0. Send an email to tor-relays to ask operators to opt-in 1. Run a script to find fallbacks whose details have changed, and email tor-relays to ask operators if their details are permanent Here are the most time-consuming steps: 2a. Receive personal emails and emails to the list with fallback details 2b. Generate an updated fallback line for each relay [0] 2c. Update the whitelist And these steps are quick: 3. Run the update script to generate a new list 4. Write a changes file 5. Announce the new list 6. Backport the list to all supported Tor versions Here are some things I'd like to change: Just have relay fingerprints in the fallback whitelist [3] The fallback script checks relay details, but it can just rely on Onionoo to say when the address last changed. This removes step 1 and step 2b. Add a torrc option and descriptor line to opt-in as a FallbackDir [4] This is inspired by nusenu's "ContactInfo" idea, and Hiro's "bot" idea. Having a descriptor line and torrc option gets us a more reliable implementation, and it's easier for relay operators to use. And there's no need for an email bot or trac bot. This removes step 2a and step 2c, for tor versions with this feature. For tor versions without this feature, we can use the simpler fingerprint list. Make the hard-coded authorities into a separate include file with a standard format [5] We'll also change the fallback file format (again, sorry!) to make them consistent. This makes it easier for stem, metrics-lib, and other Tor implementations to use the same authorities and fallbacks. This makes step 5 easier (for libraries that use fallbacks). Let's discuss these changes on the tor-dev list! T [0]: https://gitweb.torproject.org/tor.git/tree/scripts/maint/generateFallbackDir... https://gitweb.torproject.org/tor.git/tree/scripts/maint/lookupFallbackDirCo... [1]: https://trac.torproject.org/projects/tor/ticket/24789 [2]: https://trac.torproject.org/projects/tor/wiki/doc/UpdatingFallbackDirectoryM... [3]: https://trac.torproject.org/projects/tor/ticket/24838 [4]: https://trac.torproject.org/projects/tor/ticket/24839 [5]: https://trac.torproject.org/projects/tor/ticket/24818 -- Tim / teor PGP C855 6CED 5D90 A0C5 29F6 4D43 450C BA7F 968F 094B ricochet:ekmygaiu4rzgsk6n ------------------------------------------------------------------------
On 8 January 2018 at 20:56, teor <teor2345@gmail.com> wrote:
Add a torrc option and descriptor line to opt-in as a FallbackDir [4]
Setting a config entry is easy and requires no thought. It's easy to set without understanding the requirements or implications. Getting a personal email and request for one's relay to keep a stable IP lends a lot of gravity to the importance of it. Are you worried about that? Work-wise, the worst thing that would happen is that the list just needs to be regenerated more frequently - I think? But user-wise people on those older versions will have significantly increased startup times if only 20% of their FallbackDirs are working... -tom
On 9 Jan 2018, at 14:23, Tom Ritter <tom@ritter.vg> wrote:
On 8 January 2018 at 20:56, teor <teor2345@gmail.com> wrote: Add a torrc option and descriptor line to opt-in as a FallbackDir [4]
Setting a config entry is easy and requires no thought. It's easy to set without understanding the requirements or implications. Getting a personal email and request for one's relay to keep a stable IP lends a lot of gravity to the importance of it.
Are you worried about that?
My experience is that personal emails don't actually make much difference. Operators try hard to keep things stable, but sometimes their provider makes changes, or their circumstances change. Here's what I've tried in the past: * I sent personal emails the first few times, * I sent list emails, * I sent no emails at all, because we wanted to get seized relays off the list quickly. Every time, we rebuilt the list after 6-12 months. So this time, we only selected fallbacks that had the same address for at least 3 months. I'm hoping that makes a difference. Even if it doesn't change fallback stability, it's still good to tell people. I would be happy if someone else sent personal emails to: * potential operators of high-bandwidth relays * every fallback operator when there is a new list * any fallback operator when their relay details change, or their fallback goes down, asking them to fix it if they can I think a bot or a human could do a very good job of these things. (But I'm not that human.) Hiro, did you want to write an email bot? Or anyone else? (The bot would need to work out how to deal with obfuscated email addresses. We might need a human to code this part, and a list of any particularly difficult ones.)
Work-wise, the worst thing that would happen is that the list just needs to be regenerated more frequently - I think?
I want to reduce the time it takes to rebuild the list. It takes days right now. If I'm going to keep on doing it, I'd like to cut that down to hours. The features I suggested let me do that. And they make it easier for others to rebuild the list. (I don't want to be the only one who can rebuild the list.) And they don't stop people sending personal emails. They just mean that the process doesn't rely on them.
But user-wise people on those older versions will have significantly increased startup times if only 20% of their FallbackDirs are working...
Tor is designed to bootstrap if 0% of the fallbacks and 50% of the authorities are working. Recent tor versions wait 6 seconds before trying an authority. (Older versions try an authority and a fallback straight away.) They will try 5 fallbacks and 3 authorities in the first 30 seconds. And we backport the new fallback list to all supported versions. T
participants (2)
-
teor -
Tom Ritter