[Re-sending, because my first attempt apparently got lost somewhere. Apologies if gets found and this is now a duplicate.]
Hi devs, hi dir-spec.txt authors,
when I reviewed the open proposals related to the directory protocol, I wondered how to integrate the proposed changes into dir-spec.txt. For example, status opinions (proposal 147) are going to add a new variant of the status data format and a new operation that makes authorities produce and exchange opinions at some point in the voting process.
I also thought back when I wrote the patch to add microdescriptors to the spec and remembered that I had a hard time finding the right places to add things.
So, here's my question: can we reorder a few things in dir-spec.txt to make it easier to read and easier to extend? (To be clear, I'm happy to submit patches for this, I'm just asking if patches are welcome.)
In particular, I'd want to start by making the following changes (if you need a table of contents, here's how I generated mine: `grep -Ev "^ |^$|XXXX" dir-spec.txt | less`):
Sections 3 ("Formats produced by directory authorities") and 4 ("Directory server operation") apparently started out by separating passive parts (data formats) from active parts (operations). But then 3.4 ("Assigning flags in a vote") and 3.5 ("Computing a consensus from a set of votes") broke this abstraction.
One way to fix this would be to move the two mentioned subsections 3.4 and 3.5 to Section 4. Though, personally I'm not a big fan of the passive-active separation, because the description of passive parts implies a lot about how they're going to be used. And then we end up describing things twice, once implicitly and once explicitly.
The better fix, IMHO, would be to turn Section 3 into "Directory authority operation and formats" and Section 4 into "Directory cache operation". Section 3 would specify operations in causal order (not sure if this the right term) and specify data formats together with the operation that generates them: - "creating key certificates", - "accepting router descriptors and extra-info documents", - "computing microdescriptors", - "exchanging votes", - "downloading missing certificates", - "downloading router descriptors and extra-info documents", - "computing the consensus", - "computing consensus flavors", - "exchanging detached signatures", and - "publishing the signed consensus".
Section 4 would then only contain the subsections from the current Section 4 that are relevant for caches. In cases where cache operations are equivalent or similar to authority operations, there would be a simple reference to the earlier section.
Also on the top-level of sections, I'd want to merge Sections 5 ("Client operation: downloading information") and 6 ("Using directory information") into a single section "Client operation".
Would you accept patches making these changes?
All the best, Karsten
For example, status opinions (proposal 147)...
Hi Karsten. Just looked at 147 to figure out the context for overhauling the dir-spec and I'm curious about if we really need a new document type for this. I'm not clear from the proposal - which problem are we trying to solve?
1. Relays publish their descriptor to a single authority, but that authority has no way of notifying the rest beside their v2 document.
2. Generally all authorities *do* inform each other of the descriptors, but might be off by one or two. We want a method for them to be fully in sync besides a v2 document prior to a vote.
In both cases a specialized DirPort method to relay the descriptor data seems appropriate rather than inventing a new document type.
Cheers! -Damian
On 1/5/14 7:13 PM, Damian Johnson wrote:
For example, status opinions (proposal 147)...
Hi Karsten. Just looked at 147 to figure out the context for overhauling the dir-spec
Hi Damian,
proposal 147 was just one example. Another example is proposal 164 which suggests to add vote infos as another status document type. Changing the thread subject to reflect that we're talking about proposal 147 here.
and I'm curious about if we really need a new document type for this. I'm not clear from the proposal - which problem are we trying to solve?
- Relays publish their descriptor to a single authority, but that
authority has no way of notifying the rest beside their v2 document.
- Generally all authorities *do* inform each other of the
descriptors, but might be off by one or two. We want a method for them to be fully in sync besides a v2 document prior to a vote.
I'd say 1. Even with prevoting opinions exchanged minutes before generating votes there's no guarantee that no new relays appear and send their descriptors only to a subset of authorities. But prevoting opinions mitigate the problem by reducing the window from 1 hour to a few minutes.
In both cases a specialized DirPort method to relay the descriptor data seems appropriate rather than inventing a new document type.
I'm not sure what you mean by "specialized DirPort method". Can you explain that in more detail?
Also, if you're curious about proposal 147, see my review on this list from a few weeks ago:
https://lists.torproject.org/pipermail/tor-dev/2013-December/005986.html
All the best, Karsten
I'm not sure what you mean by "specialized DirPort method". Can you explain that in more detail?
No doubt I'm missing something basic about either the problem we're trying to solve or how votes work. :P
What I meant by a "specialized DirPort method" was the methods in section 4.2 ('/tor/status-vote/current/consensus' and friends). My rough guess from the proposal is that the trouble is...
1. A new relay publishes their descriptor to a single authority. (?) 2. The authority communicates the new descriptor to the rest via their v2 document. 3. Authorities now have an enumeration of all relays, and generate their vote.
... and we don't want #2 to use v2 documents. If that's the case then the simple solution seems to be something like...
2. When an authority receives a descriptor it forwards it to all the other authorities.
My understanding is that votes don't need to precisely match. If a descriptor comes in during the window where votes take place then that's fine - the majority of authorities won't have it and it simply won't be included until the following consensus.
Cheers! -Damian
On 1/6/14 9:22 PM, Damian Johnson wrote:
I'm not sure what you mean by "specialized DirPort method". Can you explain that in more detail?
No doubt I'm missing something basic about either the problem we're trying to solve or how votes work. :P
What I meant by a "specialized DirPort method" was the methods in section 4.2 ('/tor/status-vote/current/consensus' and friends). My rough guess from the proposal is that the trouble is...
- A new relay publishes their descriptor to a single authority. (?)
- The authority communicates the new descriptor to the rest via their
v2 document. 3. Authorities now have an enumeration of all relays, and generate their vote.
... and we don't want #2 to use v2 documents. If that's the case then the simple solution seems to be something like...
- When an authority receives a descriptor it forwards it to all the
other authorities.
That approach scales a lot less than exchanging a single status opinion document once per voting period.
My understanding is that votes don't need to precisely match. If a descriptor comes in during the window where votes take place then that's fine - the majority of authorities won't have it and it simply won't be included until the following consensus.
That's correct. The question is whether we want to fix the situation that it might take until the next consensus to include a relay that could have been included in the current consensus.
See also Nick's comment on the other proposal 147 review thread:
Also, ISTR that Roger told me that this whole proposal didn't actually seem to be necessary in practice. I wish I could remember the rationale, though.
Personally, I don't feel strongly about either adding prevoting opinions or living with the 1-hour delay in edge cases. But the current situation where we keep publishing v2 statuses is not ideal.
All the best, Karsten
On Sun, Jan 5, 2014 at 2:56 AM, Karsten Loesing karsten@torproject.org wrote:
[Re-sending, because my first attempt apparently got lost somewhere. Apologies if gets found and this is now a duplicate.]
Hi devs, hi dir-spec.txt authors,
[...]
Would you accept patches making these changes?
I'm okay with re-ordering for readability and maintainability, and the order you describe seems okay. I think that we'll need more changes for readability, but there's no reason to let the perfect be the enemy of the good here. One request:
The patches that move text should do literally nothing other than move text. They should not exit, reformat, delete, renumber, or anything. That should happen in other patches.
Rationale: It is much more difficult to review patches that move and edit than patches which only move things.
Rationale: It is much more difficult to review patches that move and edit than patches which only move things.
I hate to +1 threads but I was thinking the same when this was first proposed. I read all control-spec and dir-spec commits to determine if they impact Stem. Making reformatting changes have zero functional impact would be very, very appreciated.
On 1/6/14 6:04 PM, Nick Mathewson wrote:
On Sun, Jan 5, 2014 at 2:56 AM, Karsten Loesing karsten@torproject.org wrote:
[Re-sending, because my first attempt apparently got lost somewhere. Apologies if gets found and this is now a duplicate.]
Hi devs, hi dir-spec.txt authors,
[...]
Would you accept patches making these changes?
I'm okay with re-ordering for readability and maintainability, and the order you describe seems okay. I think that we'll need more changes for readability, but there's no reason to let the perfect be the enemy of the good here.
Great! I'll make the described changes, and then I'd like to hear what other changes you have in mind. But first things first.
One request:
The patches that move text should do literally nothing other than move text. They should not exit, reformat, delete, renumber, or anything. That should happen in other patches.
Rationale: It is much more difficult to review patches that move and edit than patches which only move things.
Okay, I was planning to avoid changes other than moving text except for the renumbering part. But sure, I can do the renumbering in a separate commit.
All the best, Karsten
On 1/6/14 6:04 PM, Nick Mathewson wrote:
On Sun, Jan 5, 2014 at 2:56 AM, Karsten Loesing karsten@torproject.org wrote:
[Re-sending, because my first attempt apparently got lost somewhere. Apologies if gets found and this is now a duplicate.]
Hi devs, hi dir-spec.txt authors,
[...]
Would you accept patches making these changes?
I'm okay with re-ordering for readability and maintainability, and the order you describe seems okay. I think that we'll need more changes for readability, but there's no reason to let the perfect be the enemy of the good here.
So, I finished a first revision in my branch reorder-dirspec:
https://gitweb.torproject.org/user/karsten/torspec.git/shortlog/refs/heads/r...
I'm counting 35 commits, so I really tried to keep changes in each commit small.
Here are the guidelines that I tried to follow:
- There's supposed to be one section per role in the directory protocol (client, authority, cache, client) with one subsection per operation performed by this role. If necessary, subsubsections help structure more complex operations.
- Changes are ordered from the top of the document to the bottom. There's typically one commit (or commit set) for every changed section or subsection.
- For every change that moves more than a few lines of text, there are typically three commits: the first commit only moves text, the second commit only fixes references, and the third commit edits content.
Want to have a look?
(Let me know if you prefer this review to happen in a ticket rather than here.)
All the best, Karsten
On Tue, Jan 14, 2014 at 1:56 PM, Karsten Loesing karsten@torproject.org wrote: [...]
(Let me know if you prefer this review to happen in a ticket rather than here.)
Thanks, Karsten! I think it should ideally be a ticket?
On 1/16/14 8:59 PM, Nick Mathewson wrote:
On Tue, Jan 14, 2014 at 1:56 PM, Karsten Loesing karsten@torproject.org wrote: [...]
(Let me know if you prefer this review to happen in a ticket rather than here.)
Thanks, Karsten! I think it should ideally be a ticket?
Sure. I created #10645 for this. Thanks!
All the best, Karsten
On Sun, Jan 5, 2014 at 2:56 AM, Karsten Loesing karsten@torproject.org wrote:
[Re-sending, because my first attempt apparently got lost somewhere. Apologies if gets found and this is now a duplicate.]
Hi devs, hi dir-spec.txt authors,
when I reviewed the open proposals related to the directory protocol, I wondered how to integrate the proposed changes into dir-spec.txt.
Oops; I missed this sentence, and should send a quick reminder: we don't actually integrate proposals into the specs until after the code is merged. (See proposal 001-process.txt) We learned this lesson from the bad old days when our specifications were a mix of reality and aspiration.
I figure that you probably knew this, but just in case, I should probably stop you from wasting time on merging unimplemented proposals.
(To clarify: Nothing I've said in this email means we *shouldn't* update dir-spec.txt for readability and maintainability.)
peace,
On 1/6/14 7:58 PM, Nick Mathewson wrote:
On Sun, Jan 5, 2014 at 2:56 AM, Karsten Loesing karsten@torproject.org wrote:
[Re-sending, because my first attempt apparently got lost somewhere. Apologies if gets found and this is now a duplicate.]
Hi devs, hi dir-spec.txt authors,
when I reviewed the open proposals related to the directory protocol, I wondered how to integrate the proposed changes into dir-spec.txt.
Oops; I missed this sentence, and should send a quick reminder: we don't actually integrate proposals into the specs until after the code is merged. (See proposal 001-process.txt) We learned this lesson from the bad old days when our specifications were a mix of reality and aspiration.
I figure that you probably knew this, but just in case, I should probably stop you from wasting time on merging unimplemented proposals.
Sure. That's what I meant: *after* proposals are implemented. But thanks for pointing it out!
All the best, Karsten
(To clarify: Nothing I've said in this email means we *shouldn't* update dir-spec.txt for readability and maintainability.)
peace,
-
Damian Johnson -
Karsten Loesing -
Nick Mathewson