Tor proposal status (December 2013)
This list of open Tor proposals is based on one I sent out last month[0], based on the one I did in June of last year[1], based on the one I did in May[2] of the year before. If you're looking for something to review, think about, or comment on: Review 212 (using older consensuses) or 215 (obsoleting consensus methods) if you understand the directory system even a little bit; they are quite simple. Review 219 if you're a DNS geek, or you'd like Tor to work better with more DNS types. Review 220 (ed25519 identity keys) if you like designing signature things, if you have good ideas about future-proofing key type migration, or if you care about making Tor servers' identity keys stronger. Review 223 (ACE handshake) if you're a cryptographer, or a cryptography implementer, and you'd like an even faster replacement for the ntor handshake. Review 224 if you want to look through a big, complex protocol with a lot of pieces. Also review it if you care about hidden services and making them better. Review something else if you want to take a possibly good idea that needs more momentum and promote it, fix it up, or finally kill it off. Finally: if you've sent something to tor-dev or to me that should have a proposal number, but doesn't have one yet, please ping me again to remind me! [0] https://lists.torproject.org/pipermail/tor-dev/2013-November/005798.html [1] https://lists.torproject.org/pipermail/tor-dev/2012-June/003641.html [2] https://lists.torproject.org/pipermail/tor-dev/2011-May/002637.html **NOTE**: The dates after each paragraph indicate when I last revised the paragraph. 127 Relaying dirport requests to Tor download site / website The idea here was to make it easier to fetch and learn about Tor by making it easy for relays to automatically act as proxies to the Tor website. It needs more discussion, and there are some significant details to work out. It's not at all clear whether this is actually a good idea or not. Probably, there are better choices when it comes to distributing software and updates. (11/2013) 131 Help users to verify they are using Tor [NEEDS-REVISION] This one is not a crazy idea, but I marked it as needs-revision since it doesn't seem to work so well with our current designs. It seems mostly superseded by proposal 211. (11/2013) 132 A Tor Web Service For Verifying Correct Browser Configuration This proposal was meant to give users a way to see if their browser and privoxy (yes, it was a while ago) are correctly configured by running a local webserver on 127.0.0.1. I'm not sure the status here. Generally, I'm skeptical of designs that run webservers on localhost, since they become a target for cross-site attacks. (11/2013) 133 Incorporate Unreachable ORs into the Tor Network This proposal had an idea for letting ORs that can only make outgoing connections still relay data usefully in the network. It's something we should keep in mind, and it's a pretty neat idea, but it radically changes the network topology. Anybody who wants to analyze new network topologies should definitely have a look. (5/2011) 140 Provide diffs between consensuses This proposal describes a way to transmit less directory traffic by sending only differences between consensuses, rather than the consensuses themselves. It is mainly languishing for lack of an appropriately licensed, well-written, very small, pure-C implementation of the "diff" and "patch" algorithms. (The good diffs seem to be GPL (which we can't use without changing Tor's license), or spaghetti code, or not easily usable as a library, or not written in C, or very large, or some combination of those.) (5/2011) 141 Download server descriptors on demand The idea of this proposal was for clients to only download the consensus, and later ask nodes for their own server descriptors while building the circuit through them. It would make each circuit more time-consuming to build, but make bootstrapping much cheaper. Microdescriptors obsolete a lot of this proposal, and present some difficulties in using in a way compatible with it. (6/2012) 143 Improvements of Distributed Storage for Tor Hidden Service Descriptors Here's a proposal from Karsten about making the hidden service DHT more reliable and secure to use. It could use more discussion and analysis. We should look into it as part of efforts to improve designs for the next generation of hidden services. One problem with the analysis here, though, is that it assumes a fixed set of servers that doesn't change. One reason that we upload to N servers at each chosen point in the ring is that the hidden service host and the hidden service client may have different views of which servers exist. We need to re-do the analysis with some fraction of recent consensuses. (11/2013) 144 Increase the diversity of circuits by detecting nodes belonging the same provider This is a version of the good idea, "Let's do routing in a way that tries to keep from routing traffic through the same provider too much!" There are complex issues here that the proposal doesn't completely address, but I think it might be a fine idea for somebody to see how much more we know now than we did in 2008, particularly in light of the relevant paper(s) by Matt Edmann and Paul Syverson. (5/2011) 145 Separate "suitable as a guard" from "suitable as a new guard" [NEEDS-RESEARCH] Currently, the Guard flag means both "You can use this node as a guard if you need to pick a new guard" and "If this node is currently your guard, you can keep using it as a guard." This proposal tries to separate these two concepts, so that clients can stop picking a router once it is full of existing clients using it as a guard, but the clients currently on it won't all drop it. It's not clear whether this has anonymity issues, and it's not clear whether the imagined performance gains are actually worthwhile. (5/2011) 147 Eliminate the need for v2 directories in generating v3 directories This proposal explains a way that we can phase out the vestigial use of v2 directory documents in keeping authorities well-informed enough to generating the v3 consensus. It's still correct; somebody should implement it before the v2 directory code rots any further. (5/2011) 156 Tracking blocked ports on the client side This proposal provides a way for clients to learn which ports they are (and aren't) able to connect to, and connect to the ones that work. It comes with a patch, too. It also lets routers track ports that _they_ can't connect to. I'm a little unconvinced that this will help a great deal: most clients that have some ports blocked will need bridges, not just restriction to a smaller set of ports. This could be good behind restrictive firewalls, though. The router-side part is a little iffy: routers that can't connect to each other violate one of our network topology assumptions, and even if we do want to track failed router->router connections, the routers need to be sure that they aren't fooled into trying to connect repeatedly to a series of nonexistent addresses in an attempt to make them believe that (say) they can't reach port 443. This one is a paradigmatic "open" proposal: it needs more discussion. The patch probably also needs to be ported to 0.2.3.x; it touches some code that has changed. (5/2011) 159 Exit Scanning This is an overview of SoaT, with some ideas for how to integrate it into Tor. (5/2011) 164 Reporting the status of server votes This proposal explains a way for authorities to provide a slightly more verbose document that relay operators can use to diagnose reasons that their router was or was not listed in the consensus. These documents would be like slightly more verbose versions of the authorities' votes, and would explain *why* the authority voted as it did. It wouldn't be too hard to implement, and would be a fine project for somebody who wants to get to know the directory code. (5/2011) 165 Easy migration for voting authority sets This is a design for how to change the set of authorities without having a flag day where the authority operators all reconfigure their authorities at once. It needs more discussion. One difficulty here is that we aren't talking much about changing the set of authorities, but that may be a chicken-and-egg issue, since changing the set is so onerous. If anybody is interested, it would be great to move the discussion ahead here. (5/2011) 168 Reduce default circuit window This proposal reduces the default window for circuit sendme cells. I think it's implemented (or mostly implemented) in 0.2.1.20? If so, we should make sure that tor-spec.txt is updated and close it. (11/2013) 172 GETINFO controller option for circuit information 173 GETINFO Option Expansion These would help controllers (particularly arm) provide more useful information about a running Tor process. They're accepted and some parts of 173 are even implemented: somebody just needs to implement the rest. (5/2011) 175 Automatically promoting Tor clients to nodes Here's Steven's proposal for adding a mode between "client mode" and "relay mode" for "self-test to see if you would be a good relay, and if so become one." It didn't get enough attention when it was posted to the list; more people should review it. (5/2011) 177 Abstaining from votes on individual flags Here's my proposal for letting authorities have opinions about some (flag,router) combinations without voting on whether _every_ router should have that flag. It's simple, and I think it's basically right. With more discussion and review, somebody could/should build it, I think. (11/2013) 182 Credit Bucket This proposal suggests an alternative approach to our current token-bucket based rate-limiting, that promises better performance, less buffering insanity, and a possible end to double-gating issues. (6/2012) 185 Directory caches without DirPort The old HTTP directory port feature is no longer used by clients and relays under most circumstances. The proposal explains how we can get rid of the requirement that non-bridge directories have an open directory port. (6/2012) 188 Bridge Guards and other anti-enumeration defenses This proposal suggests some ways to make it harder for a relay on the Tor network to enumerate a list of Tor bridges. Worth investigating and possibly implementing. (6/2012) 189 AUTHORIZE and AUTHORIZED cells 190 Bridge Client Authorization Based on a Shared Secret [NEEDS-REVISION] 191 Bridge Detection Resistance against MITM-capable Adversaries Proposal 187 reserved the AUTHORIZE cell type; these proposals suggests how it could work to try to make it harder to probe for Tor bridges. They need more alternatives and attention, and possibly some revision and analysis. Number 190 needs revision, since its protocol isn't actually so great. (11/2013) 192 Automatically retrieve and store information about bridges This proposal gives an enhancement to the bridge information protocol, where clients remember more things about bridges, and are able to update what they know about them over time. Could help a lot with bridge churn. (6/2012) 194 Mnemonic .onion URLs Here's one of several competing "let's make .onion URLs human-usable" proposals. This one makes sentences using a fixed map. This kind of approach is likely to obsoleted if we go ahead with current plans for hidden services that would make .onion addresses much longer, though. (11/2013) 195 TLS certificate normalization for Tor 0.2.4.x Here's the followup to proposal 179, containing all the parts of proposal 179 that didn't get built, and a couple of other tricks besides to try to make Tor's default protocol less detectable. I'm pretty psyched about the part where we let relays drop in any any self-signed or CA-issued certificate that they like. Some of this is done in ticket #7145; we should decide, however, how much we want to push towards normalizing the main Tor protocol. (11/2013) 196 Extended ORPort and TransportControlPort Here are some remaining pieces of the pluggable transport protocol that give Tor finer control over the behavior of its transports. Much of this is implemented in 0.2.5 now; we should figure out what's left, and whether we want to build that. (11/2013) 197 Message-based Inter-Controller IPC Channel This proposal is for an architectural enhancement in Tor deployments, where Tor coordinates communications between the various programs (Vidalia, TorBrowser, etc) that are using it. (6/2012) 199 Integration of BridgeFinder and BridgeFinderHelper Here's a proposal for how Tor can integrate with a client program that finds bridges for it. I've seen some work being done on things called "BridgeFinder"; I don't know what the status of the current proposal is, though. (11/2013) 201 Make bridges report statistics on daily v3 network status requests Here's a proposal for bridges to better estimate the number of bridge users. (6/2012) 202 Two improved relay encryption protocols for Tor cells Here's a sketch of the two broad classes of alternatives for improving how relay encryption works. Right now, progress on this proposal is stalled waiting for the ideal wide-block construction to come along the line. (11/2013) 203 Avoiding censorship by impersonating an HTTPS server This one is a design for making a bridge that acts like an HTTPS server (by *being* an HTTPS server) until the user proves they know it's a bridge. (11/2013) 209 Tuning the Parameters for the Path Bias Defense In this proposal, Mike discusses alternative parameters for getting better result out of the path-bias-attack detection code. (11/2013) 210 Faster Headless Consensus Bootstrapping This proposal suggests that we get our initial consensus by launching multiple connections in parallel, and fetching the consensus from whichever one completes. In my opinion, that would be a fine idea when we're fetching our initial consensus from non-Authority DirSources, but we shouldnt' do anything to increase the load on authorities. (11/2013) 211 Internal Mapaddress for Tor Configuration Testing Here, the idea is to serve an XML document over HTTP that would let the know when it's using Tor. The XML document would be returned when you make a request over Tor for a magic address in 127.0.0.0/8. I think we need to do _something_to solve this problem, but I'm not thrilled with the idea of having any more magical addresses like this; we got rid of .noconnect for a reason, after all. (11/2013) 212 Increase Acceptable Consensus Age This proposal suggests that we increase the maximum age of a consensus that clients are willing to use when they can't find a new one, in order to make the network robust for longer against a failure to reach consensus. In my opinion, we should do that. If I recall correctly, there was some tor-dev discussion on this one that should get incorporated into a final, implementable version. (11/2013) 215 Let the minimum consensus method change with time This proposal describes how we can raise the minimum allowable consensus method that all authorities must support, since the ancient "consensus method 1" would not actually be viable to keep the Tor network running. We should do this; see ticket #10163. (11/2013) 219 Support for full DNS and DNSSEC resolution in Tor Here's a design to allow Tor to support a full range of DNS request types. It probably isn't adequate on its to make DNSSEC work realistically, since naive DNSSEC requires many round trips that wouldn't be practical over Tor. It has a ton of inline discussion that needs to get resolved before this is buildable. One thing to consider here is whether we can get the server-side done with reasonable confidence, and figure out the client side once more servers have upgraded. (12/2013) 220 Migrate server identity keys to Ed25519 This one is an intial design to migrate server identity keys to Ed25519 for improved security margin. It needs more analysis of long-term migration to other signing key types -- what do we do if we want to add support for EdDSA over Curve2617 or something? Can we make it easier than this? And it also needs analysis to make sure it enables us to eventually drop RSA1024 keys entirely. I've started building this, though, so we'd better figure out out fairly soon. Other proposals, like 224, depend on this one. (12/2013) 223 Ace: Improved circuit-creation key exchange Here's an interesting one. It proposes a replacement for the ntor handshake, using the multi-exponentiation optimization, to run a bit faster at an equivalent security level. Assuming that more cryptographers like the security proof, and that the ntor handshake winds up being critical-path in profiles as more clients upgrade to 0.2.4 or 0.2.5, this is something we should consider. (12/2013) 224 Next-Generation Hidden Services in Tor This proposal outlines a more or less completely revised version of the Tor hidden services protocol, improved to accomodate better cryptography, better scalability, and defenses for several attacks we'd never considered when we did the original design. Some parts of this one are clearly right; some (like scalability) are entirely unwritten. This proposal needs a lot of attention and improvements to get it done right. I hope to implement this over the course of 2014. (12/2013) 225 Strawman proposal: commit-and-reveal shared rng Proposal 224's solutions for bug #8244 require that authorities regularly agree upon a shared random value which none of them could have influenced or predicted in advance. This proposal outlines a simple one that isn't perfect (it's vulnerable to DOS and to limited influence by one or more collaborating hostile authorities), but it's quite simple, and it's meant to start discussion. I hope that we never build this, but instead replace it with something better. Some alternatives have already been discussed on tor-dev; more work is needed, though. (12/2013) Since last time: 223, 224, and 225 are new. 157 was finally finished and closed; see ticket #10162.
On 12/17/13 10:31 PM, Nick Mathewson wrote:
212 Increase Acceptable Consensus Age
This proposal suggests that we increase the maximum age of a consensus that clients are willing to use when they can't find a new one, in order to make the network robust for longer against a failure to reach consensus. In my opinion, we should do that. If I recall correctly, there was some tor-dev discussion on this one that should get incorporated into a final, implementable version. (11/2013)
Hi Nick, I agree with the idea that clients should accept an old consensus up to 3 days instead of 1. It's stressful enough to nag directory authority operators to look after their machines if they fail to produce a consensus for a few hours. I did that a couple of times, and it stressed me out every single time. I don't want to imagine how bad such a situation would be during the holidays or CCC. You mention a tor-dev discussion above that should get incorporated. Do you have a link? A quick search in my inbox didn't help. Here's some feedback from reading the proposal: - Section 6.1 of dir-spec.txt says that "Circuits SHOULD NOT be built until the client has [...] a live consensus network status", but that means 3 hours after valid-after, AFAIK. Should we rather specify here that clients MAY use a consensus for up to 3 days after its valid-after time if they don't find a more recent one? Or is this something to leave to the implementation and leave open in dir-spec.txt? - If the new 3 days constant should become part of dir-spec.txt, what about the suggested time after which old router descriptors may safely be removed from caches? (Would you accept patches to dir-spec.txt that specify related time constants that are currently only written to the code?) - Do we really plan to raise the 3 days to something higher when the "proposals related to ticket #7126 [...] are complete and implemented"? If so, would it make sense to make the 3 days constant a new consensus parameter, rather than hard-code it? - I didn't review the Implementation Notes part in detail, yet. But it feels wrong that OLD_ROUTER_DESC_MAX_AGE is now only 3 days 18 hours when it was 5 days before. All the best, Karsten
On Fri, Dec 20, 2013 at 12:54 PM, Karsten Loesing <karsten@torproject.org> wrote:
On 12/17/13 10:31 PM, Nick Mathewson wrote:
212 Increase Acceptable Consensus Age
This proposal suggests that we increase the maximum age of a consensus that clients are willing to use when they can't find a new one, in order to make the network robust for longer against a failure to reach consensus. In my opinion, we should do that. If I recall correctly, there was some tor-dev discussion on this one that should get incorporated into a final, implementable version. (11/2013)
Hi Nick,
I agree with the idea that clients should accept an old consensus up to 3 days instead of 1. It's stressful enough to nag directory authority operators to look after their machines if they fail to produce a consensus for a few hours. I did that a couple of times, and it stressed me out every single time. I don't want to imagine how bad such a situation would be during the holidays or CCC.
You mention a tor-dev discussion above that should get incorporated. Do you have a link? A quick search in my inbox didn't help.
I'm afraid I can't find it either. "Some time in 2012" would be my guess. I think I was thinking of the discussion on #7986 .
Here's some feedback from reading the proposal:
- Section 6.1 of dir-spec.txt says that "Circuits SHOULD NOT be built until the client has [...] a live consensus network status", but that means 3 hours after valid-after, AFAIK. Should we rather specify here that clients MAY use a consensus for up to 3 days after its valid-after time if they don't find a more recent one? Or is this something to leave to the implementation and leave open in dir-spec.txt?
I think it should go into dir-spec.txt once this proposal is done. Alternatively, we could increase the valid-until interval and have the valid-until time be 3 days after valid-after. That seems like a cleaner solution to me. I wonder why we didn't spec it like that. Perhaps a more careful reading of the proposal or of #7986 will tell me why...
- If the new 3 days constant should become part of dir-spec.txt, what about the suggested time after which old router descriptors may safely be removed from caches? (Would you accept patches to dir-spec.txt that specify related time constants that are currently only written to the code?)
Sure.
- Do we really plan to raise the 3 days to something higher when the "proposals related to ticket #7126 [...] are complete and implemented"? If so, would it make sense to make the 3 days constant a new consensus parameter, rather than hard-code it?
Possibly. peace, -- Nick
On 12/17/13 10:31 PM, Nick Mathewson wrote:
215 Let the minimum consensus method change with time
This proposal describes how we can raise the minimum allowable consensus method that all authorities must support, since the ancient "consensus method 1" would not actually be viable to keep the Tor network running. We should do this; see ticket #10163. (11/2013)
Hi Nick, I'm probably missing something important here, but I don't know what. Right now, if a directory authority learns from the votes that more than 2/3 of authorities support a consensus method higher that it can support itself, it falls back to consensus method 1. That authority then produces a consensus that won't have enough signatures for any client to use it, so it's useless. The proposal suggests that this authority produces a consensus using a higher method than 1, but still lower than what the other authorities are going to produce. But this consensus will still not contain enough signatures. What's the point? The last paragraph in the proposal makes most sense to me:
We might want to have the behavior when we see that everybody else will be using a method we don't support be "Don't make a consensus at all." That's harder to program, though.
Can you say why this solution is harder to program? It seems like the cleaner design. But even if it's too difficult to program (or would likely add new bugs), why not keep the fall-back-to-method-1 workaround? Does it cause any harm? There are probably edge cases I didn't consider. I wonder which ones that are. All the best, Karsten
On Sun, Dec 22, 2013 at 4:07 AM, Karsten Loesing <karsten@torproject.org> wrote: [...]
Hi Nick,
I'm probably missing something important here, but I don't know what.
Right now, if a directory authority learns from the votes that more than 2/3 of authorities support a consensus method higher that it can support itself, it falls back to consensus method 1. That authority then produces a consensus that won't have enough signatures for any client to use it, so it's useless.
The proposal suggests that this authority produces a consensus using a higher method than 1, but still lower than what the other authorities are going to produce. But this consensus will still not contain enough signatures.
What's the point?
The last paragraph in the proposal makes most sense to me:
We might want to have the behavior when we see that everybody else will be using a method we don't support be "Don't make a consensus at all." That's harder to program, though.
Can you say why this solution is harder to program? It seems like the cleaner design.
The issue is that right now, I think the consensus code in Tor doesn't have a "don't publish a consensus" case.
But even if it's too difficult to program (or would likely add new bugs), why not keep the fall-back-to-method-1 workaround? Does it cause any harm?
So IIRC there are two important ideas in proposal 215. * One is to define a new minimum consensus method which everybody must support. * The other is to eventually drop support for earlier consensus methods. Note that the change to the fallback behavior when we can't support the chosen consensus method isn't one of the two important changes. It's a side-effect of dropping support for ancient consensus methods. So my reasoning for not retaining the "fall back to method 1" logic is that eventually I would like Tor not to *have* any method-1 code left in it, since we should never use it. -- Nick
On 12/17/13 10:31 PM, Nick Mathewson wrote:
165 Easy migration for voting authority sets
This is a design for how to change the set of authorities without having a flag day where the authority operators all reconfigure their authorities at once. It needs more discussion. One difficulty here is that we aren't talking much about changing the set of authorities, but that may be a chicken-and-egg issue, since changing the set is so onerous.
If anybody is interested, it would be great to move the discussion ahead here. (5/2011)
Hi Nick, (just in case you're wondering, I'm going through all proposals in your list that have to do with the directory protocol and try to review them) Proposal 165 looks like a fine idea, and the algorithm looks plausible to me. I'd say let's do it! So, what discussion would you want to see here? Are you hoping for some kind of "proof" that the suggested algorithm cannot break under certain assumptions? I don't know how to write one. But this could be a fine question for a grad student or researcher. For example, a few days ago we have been asked for research questions for small doctoral projects, and this could be a fine topic. Or did you expect to hear from current and prospective authority operators? As a former authority operator I can say that I'd really have appreciated a two-phase process where everyone first configures a second voting set and then removes the first voting set. Or were you hoping for somebody to implement the proposal? Doesn't seem terribly difficult, so maybe we'd find somebody if we created a Trac ticket for it. Here's some feedback, though nothing really important: - Branch prop165tweaks in my public torspec repository has a few tweaks. - You say in "Migration issues" that we should keep track somewhere which Tor client versions recognized which authorities. Would it be sufficient to write a little shell script that searches the git history of config.c for changes to trusted authorities and prints out which tags first contained those commits. Let me know if I can help with anything here. All the best, Karsten
On Sun, Dec 22, 2013 at 10:27 AM, Karsten Loesing <karsten@torproject.org> wrote:
On 12/17/13 10:31 PM, Nick Mathewson wrote:
165 Easy migration for voting authority sets
This is a design for how to change the set of authorities without having a flag day where the authority operators all reconfigure their authorities at once. It needs more discussion. One difficulty here is that we aren't talking much about changing the set of authorities, but that may be a chicken-and-egg issue, since changing the set is so onerous.
If anybody is interested, it would be great to move the discussion ahead here. (5/2011)
Hi Nick,
(just in case you're wondering, I'm going through all proposals in your list that have to do with the directory protocol and try to review them)
Proposal 165 looks like a fine idea, and the algorithm looks plausible to me. I'd say let's do it!
So, what discussion would you want to see here? Are you hoping for some kind of "proof" that the suggested algorithm cannot break under certain assumptions? I don't know how to write one. But this could be a fine question for a grad student or researcher. For example, a few days ago we have been asked for research questions for small doctoral projects, and this could be a fine topic.
Maybe; I'm not sure I'd want to have anybody do this for their doctoral thesis and expect that the results would be important to Tor. (IOW, Tor could use a solution here, but we're not suffering from the lack of it. Further, anything that's much more complex than the scheme in 165 is probably not going to get implemented, and I wouldn't want to disappoint a PhD candidate through giving them inaccurate expectations.) I'm not expecting a proof that the scheme described in prop165 works in any robust way given byzantine inputs; rather, I just want to be sure that it works properly if a majority of authorities are configured correctly. And I believe it does. Also, there's an issue we should solve. It says, Ties are broken in favor of some arbitrary function of the identity keys of the authorities in the set. We should specify which arbitrary function we mean, and describe how it works when we do a migration where we _replace_ an authority.
Or did you expect to hear from current and prospective authority operators? As a former authority operator I can say that I'd really have appreciated a two-phase process where everyone first configures a second voting set and then removes the first voting set.
Or were you hoping for somebody to implement the proposal? Doesn't seem terribly difficult, so maybe we'd find somebody if we created a Trac ticket for it.
Agreed. The trickiest parts will IMO be: 1) Handling the fact that we can know about directory authorities which are not the directory authorities we treat as canonical. Right now, we have only one set of authorities we believe in. That would need to broaden into three sets. First "prospective co-voters" would be the authorities to which we send our votes, from which we download votes, from which the authorities we download certificates (?). "Actual co-voters" would be the set whose votes we use as input to our consensus, and whom we collect signatures signatures from. Finally, we'd maintain the set of authorities that we believe in as a regular Tor node, which we use for checking consensuses. 2) Testing!
Here's some feedback, though nothing really important:
- Branch prop165tweaks in my public torspec repository has a few tweaks.
Merged.
- You say in "Migration issues" that we should keep track somewhere which Tor client versions recognized which authorities. Would it be sufficient to write a little shell script that searches the git history of config.c for changes to trusted authorities and prints out which tags first contained those commits.
Maybe, though that script would need to parse C to stay viable long-term. Better IMO to introduce a little change log comment around that part of the code. Possibly it could be built using such a shell script, though. peace, -- Nick
On 12/17/13 10:31 PM, Nick Mathewson wrote:
185 Directory caches without DirPort
The old HTTP directory port feature is no longer used by clients and relays under most circumstances. The proposal explains how we can get rid of the requirement that non-bridge directories have an open directory port. (6/2012)
Hey Nick, this proposal looks like a fine idea. Two remarks: - I'm not sure why a) relays with a working DirPort shouldn't include "dir-cache 1" in their router descriptor and b) authorities shouldn't assign the "DirCache" flag to relays with a working DirPort that don't have a "dir-cache 1" line in their router descriptor. I understand that neither of the two actions are necessary to make the proposal work. But this could be a chance to get rid of the DirPort concept entirely and only rely on "dir-cache 1" and "DirCache" flags in the future. - We should write into ChangeLog and torrc.sample that relay operators shouldn't rush setting their DirPort to 0 and rely on DirCache to turn their relay into a directory mirror. Older clients that don't recognize what a DirCache is will put more load on mirrors with non-zero DirPort, and it's unclear how long it will take them to upgrade. All the best, Karsten
On Mon, Dec 23, 2013 at 1:43 PM, Karsten Loesing <karsten@torproject.org> wrote:
On 12/17/13 10:31 PM, Nick Mathewson wrote:
185 Directory caches without DirPort
The old HTTP directory port feature is no longer used by clients and relays under most circumstances. The proposal explains how we can get rid of the requirement that non-bridge directories have an open directory port. (6/2012)
Hey Nick,
this proposal looks like a fine idea. Two remarks:
- I'm not sure why a) relays with a working DirPort shouldn't include "dir-cache 1" in their router descriptor and b) authorities shouldn't assign the "DirCache" flag to relays with a working DirPort that don't have a "dir-cache 1" line in their router descriptor. I understand that neither of the two actions are necessary to make the proposal work. But this could be a chance to get rid of the DirPort concept entirely and only rely on "dir-cache 1" and "DirCache" flags in the future.
- We should write into ChangeLog and torrc.sample that relay operators shouldn't rush setting their DirPort to 0 and rely on DirCache to turn their relay into a directory mirror. Older clients that don't recognize what a DirCache is will put more load on mirrors with non-zero DirPort, and it's unclear how long it will take them to upgrade.
I agree on both counts. I've patched the proposal in 8ef1e83ed9695603d4 -- Nick
On 12/17/13 10:31 PM, Nick Mathewson wrote:
147 Eliminate the need for v2 directories in generating v3 directories
This proposal explains a way that we can phase out the vestigial use of v2 directory documents in keeping authorities well-informed enough to generating the v3 consensus. It's still correct; somebody should implement it before the v2 directory code rots any further. (5/2011)
This proposal looks plausible to me. Some minor remarks: - The proposal suggests that authorities send an opinion document to the other authorities "at the regular vote upload URL". URLs are cheap, why not use a different URL to keep things separated, e.g., /tor/post/opinion ? - Should dir-spec.txt suggest a timing for pushing-and-pulling opinion documents? Authorities could send their opinions at :45:00 and fetch missing opinions at :47:30. This could be defined by a new OpinionSeconds part contained in "voting-delay" lines. This would be a SHOULD requirement, not a MUST requirement. - The proposal doesn't say what lines must be contained in opinion documents. It seems that an authority that parses an opinion document is only interested in a) relay fingerprint, b) descriptor publication time, and c) descriptor digest; unless there's more information that helps authorities decide whether "they might accept" a descriptor. If not, opinion documents only need to contain a small subset of headers and all the "r" lines that would be contained in a later vote. - The proposal doesn't explicitly say this, so just to be sure: when an authority finds that it's missing a router descriptor that it then downloads, it also downloads the corresponding extra-info descriptor afterwards, right? - Another thing that is left implicit in the proposal: the opinion document will always contain the valid-after time of the *next* consensus. Well, the URL /tor/status-vote/next/opinion implies that, but maybe we should explicitly mention this in dir-spec.txt. All the best, Karsten P. S.: Two more to go: 164 and 177...
On Tue, Dec 24, 2013 at 3:55 AM, Karsten Loesing <karsten@torproject.org> wrote:
On 12/17/13 10:31 PM, Nick Mathewson wrote:
147 Eliminate the need for v2 directories in generating v3 directories
This proposal explains a way that we can phase out the vestigial use of v2 directory documents in keeping authorities well-informed enough to generating the v3 consensus. It's still correct; somebody should implement it before the v2 directory code rots any further. (5/2011)
This proposal looks plausible to me. Some minor remarks:
- The proposal suggests that authorities send an opinion document to the other authorities "at the regular vote upload URL". URLs are cheap, why not use a different URL to keep things separated, e.g., /tor/post/opinion ?
sure.
- Should dir-spec.txt suggest a timing for pushing-and-pulling opinion documents? Authorities could send their opinions at :45:00 and fetch missing opinions at :47:30. This could be defined by a new OpinionSeconds part contained in "voting-delay" lines. This would be a SHOULD requirement, not a MUST requirement.
This is plausible.
- The proposal doesn't say what lines must be contained in opinion documents. It seems that an authority that parses an opinion document is only interested in a) relay fingerprint, b) descriptor publication time, and c) descriptor digest; unless there's more information that helps authorities decide whether "they might accept" a descriptor. If not, opinion documents only need to contain a small subset of headers and all the "r" lines that would be contained in a later vote.
This also seems okay. It would however mean that we can't use the same parsing logic as we use for regular votes.
- The proposal doesn't explicitly say this, so just to be sure: when an authority finds that it's missing a router descriptor that it then downloads, it also downloads the corresponding extra-info descriptor afterwards, right?
I suppose it should.
- Another thing that is left implicit in the proposal: the opinion document will always contain the valid-after time of the *next* consensus. Well, the URL /tor/status-vote/next/opinion implies that, but maybe we should explicitly mention this in dir-spec.txt.
Hm. maybe valid-after and valid-until should just get ignored on opinions. Or omitted. Also, ISTR that Roger told me that this whole proposal didn't actually seem to be necessary in practice. I wish I could remember the rationale, though. yrs, -- Nick
On 1/6/14 7:55 PM, Nick Mathewson wrote:
On Tue, Dec 24, 2013 at 3:55 AM, Karsten Loesing <karsten@torproject.org> wrote:
On 12/17/13 10:31 PM, Nick Mathewson wrote:
147 Eliminate the need for v2 directories in generating v3 directories
This proposal explains a way that we can phase out the vestigial use of v2 directory documents in keeping authorities well-informed enough to generating the v3 consensus. It's still correct; somebody should implement it before the v2 directory code rots any further. (5/2011)
This proposal looks plausible to me. Some minor remarks:
- The proposal suggests that authorities send an opinion document to the other authorities "at the regular vote upload URL". URLs are cheap, why not use a different URL to keep things separated, e.g., /tor/post/opinion ?
sure.
Okay, starting a patch with proposal 147 tweaks and changing this URL as suggested.
- Should dir-spec.txt suggest a timing for pushing-and-pulling opinion documents? Authorities could send their opinions at :45:00 and fetch missing opinions at :47:30. This could be defined by a new OpinionSeconds part contained in "voting-delay" lines. This would be a SHOULD requirement, not a MUST requirement.
This is plausible.
Added to the proposal.
- The proposal doesn't say what lines must be contained in opinion documents. It seems that an authority that parses an opinion document is only interested in a) relay fingerprint, b) descriptor publication time, and c) descriptor digest; unless there's more information that helps authorities decide whether "they might accept" a descriptor. If not, opinion documents only need to contain a small subset of headers and all the "r" lines that would be contained in a later vote.
This also seems okay. It would however mean that we can't use the same parsing logic as we use for regular votes.
True. Added as two comments to the proposal.
- The proposal doesn't explicitly say this, so just to be sure: when an authority finds that it's missing a router descriptor that it then downloads, it also downloads the corresponding extra-info descriptor afterwards, right?
I suppose it should.
Added.
- Another thing that is left implicit in the proposal: the opinion document will always contain the valid-after time of the *next* consensus. Well, the URL /tor/status-vote/next/opinion implies that, but maybe we should explicitly mention this in dir-spec.txt.
Hm. maybe valid-after and valid-until should just get ignored on opinions. Or omitted.
Added as comments.
Also, ISTR that Roger told me that this whole proposal didn't actually seem to be necessary in practice. I wish I could remember the rationale, though.
I talked to Roger on IRC, and here's why this proposal may indeed be overkill: As of January 2013, there is only a single version 3 directory authority left that serves version 2 statuses: dizum. moria1 and tor26 have been rejecting version 2 requests for a long time, and it's mostly an oversight that dizum still serves them. The other six authorities have never generated version 2 statuses for others to be used as pre-voting opinions. So, it's basically not true that version 2 statuses are required for the version 3 protocol to work properly. Here's a possible way to move this forward. - Please review and merge my prop147tweaks branch that contains tweaks from our discussion above, regardless of whether this proposal will be implemented or not. - I'm going to run a quick analysis of archived vote documents to see how much authorities would have benefited from the others' votes before generating their own votes. - I'm going to ask Alex to disable version 2 statuses on dizum using DisableV2DirectoryInfo_ 1 to see what that does to the network. We should probably finish the 2048 bits RSA keys upgrade first before changing yet another variable. - If there's no convincing argument to implement opinion documents, we close this proposal as rejected. What do you think? All the best, Karsten
On 1/15/14 1:08 PM, Karsten Loesing wrote:
On 1/6/14 7:55 PM, Nick Mathewson wrote:
On Tue, Dec 24, 2013 at 3:55 AM, Karsten Loesing <karsten@torproject.org> wrote:
On 12/17/13 10:31 PM, Nick Mathewson wrote:
147 Eliminate the need for v2 directories in generating v3 directories
This proposal explains a way that we can phase out the vestigial use of v2 directory documents in keeping authorities well-informed enough to generating the v3 consensus. It's still correct; somebody should implement it before the v2 directory code rots any further. (5/2011)
This proposal looks plausible to me. Some minor remarks:
- The proposal suggests that authorities send an opinion document to the other authorities "at the regular vote upload URL". URLs are cheap, why not use a different URL to keep things separated, e.g., /tor/post/opinion ?
sure.
Okay, starting a patch with proposal 147 tweaks and changing this URL as suggested.
- Should dir-spec.txt suggest a timing for pushing-and-pulling opinion documents? Authorities could send their opinions at :45:00 and fetch missing opinions at :47:30. This could be defined by a new OpinionSeconds part contained in "voting-delay" lines. This would be a SHOULD requirement, not a MUST requirement.
This is plausible.
Added to the proposal.
- The proposal doesn't say what lines must be contained in opinion documents. It seems that an authority that parses an opinion document is only interested in a) relay fingerprint, b) descriptor publication time, and c) descriptor digest; unless there's more information that helps authorities decide whether "they might accept" a descriptor. If not, opinion documents only need to contain a small subset of headers and all the "r" lines that would be contained in a later vote.
This also seems okay. It would however mean that we can't use the same parsing logic as we use for regular votes.
True. Added as two comments to the proposal.
- The proposal doesn't explicitly say this, so just to be sure: when an authority finds that it's missing a router descriptor that it then downloads, it also downloads the corresponding extra-info descriptor afterwards, right?
I suppose it should.
Added.
- Another thing that is left implicit in the proposal: the opinion document will always contain the valid-after time of the *next* consensus. Well, the URL /tor/status-vote/next/opinion implies that, but maybe we should explicitly mention this in dir-spec.txt.
Hm. maybe valid-after and valid-until should just get ignored on opinions. Or omitted.
Added as comments.
Also, ISTR that Roger told me that this whole proposal didn't actually seem to be necessary in practice. I wish I could remember the rationale, though.
I talked to Roger on IRC, and here's why this proposal may indeed be overkill:
As of January 2013, there is only a single version 3 directory authority left that serves version 2 statuses: dizum. moria1 and tor26 have been rejecting version 2 requests for a long time, and it's mostly an oversight that dizum still serves them. The other six authorities have never generated version 2 statuses for others to be used as pre-voting opinions. So, it's basically not true that version 2 statuses are required for the version 3 protocol to work properly.
Here's a possible way to move this forward.
- Please review and merge my prop147tweaks branch that contains tweaks from our discussion above, regardless of whether this proposal will be implemented or not.
- I'm going to run a quick analysis of archived vote documents to see how much authorities would have benefited from the others' votes before generating their own votes.
From January 1 to 7, 2014, only 0.4 relays on average were not included in a consensus because they were listed in less than 5 votes. These 0.4 relays could probably have been included with pre-voting opinions.
(Here's how I found out: extract the votes-2014-01.tar.bz2 tarball, run `grep -R "^r " 0[1-7] | cut -c 4-22,112- | cut -d" " -f1,3 | sort | uniq -c | sort | grep " [1-4] " | wc -l`, result is 63, divide by 7*24 published consensuses, obtain 0.375 as end result.)
- I'm going to ask Alex to disable version 2 statuses on dizum using DisableV2DirectoryInfo_ 1 to see what that does to the network. We should probably finish the 2048 bits RSA keys upgrade first before changing yet another variable.
- If there's no convincing argument to implement opinion documents, we close this proposal as rejected.
What do you think?
All the best, Karsten
On Wed, Jan 15, 2014 at 01:08:03PM +0100, Karsten Loesing wrote:
I talked to Roger on IRC, and here's why this proposal may indeed be overkill:
As of January 2013, there is only a single version 3 directory authority left that serves version 2 statuses: dizum. moria1 and tor26 have been rejecting version 2 requests for a long time, and it's mostly an oversight that dizum still serves them. The other six authorities have never generated version 2 statuses for others to be used as pre-voting opinions. So, it's basically not true that version 2 statuses are required for the version 3 protocol to work properly.
See git commits 2e692bd8 and eaf5487d, which went into 0.2.2.12-alpha: o Major bugfixes: - Many relays have been falling out of the consensus lately because not enough authorities know about their descriptor for them to get a majority of votes. When we deprecated the v2 directory protocol, we got rid of the only way that v3 authorities can hear from each other about other descriptors. Now authorities examine every v3 vote for new descriptors, and fetch them from that authority. Bugfix on 0.2.1.23. That was the stopgap that made proposal 147 not so critical. I think based on Karsten's recent results that maybe it's enough. --Roger
On Wed, Jan 15, 2014 at 9:15 PM, Roger Dingledine <arma@mit.edu> wrote:
On Wed, Jan 15, 2014 at 01:08:03PM +0100, Karsten Loesing wrote:
I talked to Roger on IRC, and here's why this proposal may indeed be overkill:
As of January 2013, there is only a single version 3 directory authority left that serves version 2 statuses: dizum. moria1 and tor26 have been rejecting version 2 requests for a long time, and it's mostly an oversight that dizum still serves them. The other six authorities have never generated version 2 statuses for others to be used as pre-voting opinions. So, it's basically not true that version 2 statuses are required for the version 3 protocol to work properly.
See git commits 2e692bd8 and eaf5487d, which went into 0.2.2.12-alpha: o Major bugfixes: - Many relays have been falling out of the consensus lately because not enough authorities know about their descriptor for them to get a majority of votes. When we deprecated the v2 directory protocol, we got rid of the only way that v3 authorities can hear from each other about other descriptors. Now authorities examine every v3 vote for new descriptors, and fetch them from that authority. Bugfix on 0.2.1.23.
That was the stopgap that made proposal 147 not so critical. I think based on Karsten's recent results that maybe it's enough.
Sounds good to me. Is this in dir-spec.txt? I'm not finding it at first glance. If it isn't, Karsten, would you be able to add it? Probably we should do that _after_ merging your dirspec branch. -- Nick
On 1/16/14 10:06 PM, Nick Mathewson wrote:
On Wed, Jan 15, 2014 at 9:15 PM, Roger Dingledine <arma@mit.edu> wrote:
On Wed, Jan 15, 2014 at 01:08:03PM +0100, Karsten Loesing wrote:
I talked to Roger on IRC, and here's why this proposal may indeed be overkill:
As of January 2013, there is only a single version 3 directory authority left that serves version 2 statuses: dizum. moria1 and tor26 have been rejecting version 2 requests for a long time, and it's mostly an oversight that dizum still serves them. The other six authorities have never generated version 2 statuses for others to be used as pre-voting opinions. So, it's basically not true that version 2 statuses are required for the version 3 protocol to work properly.
See git commits 2e692bd8 and eaf5487d, which went into 0.2.2.12-alpha: o Major bugfixes: - Many relays have been falling out of the consensus lately because not enough authorities know about their descriptor for them to get a majority of votes. When we deprecated the v2 directory protocol, we got rid of the only way that v3 authorities can hear from each other about other descriptors. Now authorities examine every v3 vote for new descriptors, and fetch them from that authority. Bugfix on 0.2.1.23.
That was the stopgap that made proposal 147 not so critical. I think based on Karsten's recent results that maybe it's enough.
Sounds good to me.
Is this in dir-spec.txt? I'm not finding it at first glance. If it isn't, Karsten, would you be able to add it? Probably we should do that _after_ merging your dirspec branch.
Current Section 4.4 ("Downloading and storing router descriptors (authorities and caches)") mentions this, though rather implicitly:
Periodically (currently, every 10 seconds), directory servers check whether there are any specific descriptors that they do not have and that they are not currently trying to download. [...]; authorities identify them by hash in vote (if publication date is more recent than the descriptor we currently have). If so, the directory server launches requests to the authorities for these descriptors, such that each authority is only asked for descriptors listed in its most recent vote (if the requester is an authority) [...]. If we're an authority, and more than one authority lists the descriptor, we choose which to ask at random.
We could this make more explicit by adding a sentence or two to the end of the current Section 4.2 ("Voting (authorities only)"), which would be the new Section 3.4 ("Exchanging votes") in my reorder-dirspec branch. This sentence or two should probably explain why it's important that authorities learn descriptors contained in other authorities' votes. And sure, happy to add this clarification after merging the branch. All the best, Karsten
On Wed, Jan 15, 2014 at 7:08 AM, Karsten Loesing <karsten@torproject.org> wrote: [...]
I talked to Roger on IRC, and here's why this proposal may indeed be overkill:
As of January 2013, there is only a single version 3 directory authority left that serves version 2 statuses: dizum. moria1 and tor26 have been rejecting version 2 requests for a long time, and it's mostly an oversight that dizum still serves them. The other six authorities have never generated version 2 statuses for others to be used as pre-voting opinions. So, it's basically not true that version 2 statuses are required for the version 3 protocol to work properly.
Here's a possible way to move this forward.
- Please review and merge my prop147tweaks branch that contains tweaks from our discussion above, regardless of whether this proposal will be implemented or not.
Done.
- I'm going to run a quick analysis of archived vote documents to see how much authorities would have benefited from the others' votes before generating their own votes.
- I'm going to ask Alex to disable version 2 statuses on dizum using DisableV2DirectoryInfo_ 1 to see what that does to the network. We should probably finish the 2048 bits RSA keys upgrade first before changing yet another variable.
Soynds good.
- If there's no convincing argument to implement opinion documents, we close this proposal as rejected.
Great.
What do you think?
Sounds good.
On 1/16/14 10:03 PM, Nick Mathewson wrote:
On Wed, Jan 15, 2014 at 7:08 AM, Karsten Loesing <karsten@torproject.org> wrote: [...]
I talked to Roger on IRC, and here's why this proposal may indeed be overkill:
As of January 2013, there is only a single version 3 directory authority left that serves version 2 statuses: dizum. moria1 and tor26 have been rejecting version 2 requests for a long time, and it's mostly an oversight that dizum still serves them. The other six authorities have never generated version 2 statuses for others to be used as pre-voting opinions. So, it's basically not true that version 2 statuses are required for the version 3 protocol to work properly.
Here's a possible way to move this forward.
- Please review and merge my prop147tweaks branch that contains tweaks from our discussion above, regardless of whether this proposal will be implemented or not.
Done.
- I'm going to run a quick analysis of archived vote documents to see how much authorities would have benefited from the others' votes before generating their own votes.
- I'm going to ask Alex to disable version 2 statuses on dizum using DisableV2DirectoryInfo_ 1 to see what that does to the network. We should probably finish the 2048 bits RSA keys upgrade first before changing yet another variable.
Soynds good.
Okay, will post here how that went. Might take a week or maybe longer, depending on how far the 2048 bits RSA keys upgrade is. All the best, Karsten
- If there's no convincing argument to implement opinion documents, we close this proposal as rejected.
Great.
What do you think?
Sounds good. _______________________________________________ tor-dev mailing list tor-dev@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev .
On 17/01/14 16:39, Karsten Loesing wrote:
On 1/16/14 10:03 PM, Nick Mathewson wrote:
On Wed, Jan 15, 2014 at 7:08 AM, Karsten Loesing <karsten@torproject.org> wrote:
- I'm going to ask Alex to disable version 2 statuses on dizum using DisableV2DirectoryInfo_ 1 to see what that does to the network. We should probably finish the 2048 bits RSA keys upgrade first before changing yet another variable.
Soynds good.
Okay, will post here how that went. Might take a week or maybe longer, depending on how far the 2048 bits RSA keys upgrade is.
So, dizum is now running with this new setting for a week, and it looks like the Tor network survived quite well. Yay!
- If there's no convincing argument to implement opinion documents, we close this proposal as rejected.
Please find my branch prop147reject which also mentions more explicitly that authorities scan the other votes for descriptors they don't know. https://gitweb.torproject.org/user/karsten/torspec.git/shortlog/refs/heads/p... All the best, Karsten
On 12/17/13 10:31 PM, Nick Mathewson wrote:
177 Abstaining from votes on individual flags
Here's my proposal for letting authorities have opinions about some (flag,router) combinations without voting on whether _every_ router should have that flag. It's simple, and I think it's basically right. With more discussion and review, somebody could/should build it, I think. (11/2013)
This proposal looks useful, too. There's just one thing that surprised me in the proposal:
A flag is listed in the consensus if it is in the known-flags section of at least one voter, and in the known-flags or extra-flags section of at least three voters (or half the authorities, whichever set is smaller).
The previous requirement for a flag to be listed in the consensus was:
Known-flags is the union of all flags known by any voter.
If I'm not mistaken, the new requirement that at least three voters need to at least sometimes have an opinion on a flag is new, and it seems unrelated to being able to abstain from votes on individual flags. Even if nobody uses extra-flags, a flag that is only contained in two known-flags lines suddenly won't make it into the consensus when the new consensus method is used. I'm not saying this new requirement is bad, but I didn't expect it to be introduced in this proposal. Maybe there should be a separate (tiny) proposal that requires at least three voters to know a flag. Or maybe the overview of this proposal and a later ChangeLog entry and dir-spec.txt patch should state this new requirement more explicitly. Or maybe I just didn't read dir-spec.txt well enough to find that it already contains this requirement... All the best, Karsten
On Tue, Dec 24, 2013 at 4:58 AM, Karsten Loesing <karsten@torproject.org> wrote:
On 12/17/13 10:31 PM, Nick Mathewson wrote:
177 Abstaining from votes on individual flags
Here's my proposal for letting authorities have opinions about some (flag,router) combinations without voting on whether _every_ router should have that flag. It's simple, and I think it's basically right. With more discussion and review, somebody could/should build it, I think. (11/2013)
This proposal looks useful, too.
There's just one thing that surprised me in the proposal:
A flag is listed in the consensus if it is in the known-flags section of at least one voter, and in the known-flags or extra-flags section of at least three voters (or half the authorities, whichever set is smaller).
The previous requirement for a flag to be listed in the consensus was:
Known-flags is the union of all flags known by any voter.
If I'm not mistaken, the new requirement that at least three voters need to at least sometimes have an opinion on a flag is new, and it seems unrelated to being able to abstain from votes on individual flags. Even if nobody uses extra-flags, a flag that is only contained in two known-flags lines suddenly won't make it into the consensus when the new consensus method is used. I'm not saying this new requirement is bad, but I didn't expect it to be introduced in this proposal. Maybe there should be a separate (tiny) proposal that requires at least three voters to know a flag. Or maybe the overview of this proposal and a later ChangeLog entry and dir-spec.txt patch should state this new requirement more explicitly.
So, I think my reasoning here was that we should not allow any single authority to be a dictator for a flag, and we shouldn't let a single authority add a huge number of flags on their own. We could split it into a new proposal, I guess. Want to write that? peace, -- Nick
On 12/17/13 10:31 PM, Nick Mathewson wrote:
164 Reporting the status of server votes
This proposal explains a way for authorities to provide a slightly more verbose document that relay operators can use to diagnose reasons that their router was or was not listed in the consensus. These documents would be like slightly more verbose versions of the authorities' votes, and would explain *why* the authority voted as it did. It wouldn't be too hard to implement, and would be a fine project for somebody who wants to get to know the directory code. (5/2011)
Hi Nick, I very much like this proposal! I want to help move it forward and integrate the additional information into Onionoo, so that people can diagnose the network better. Knowing why an authority rejected a descriptor, when it last performed a successful and an unsuccessful reachability test, why it didn't include a relay in its vote, why it assigned which relay flags, etc. can be very helpful information. Here's some feedback: - The URL /tor/status-vote-info/current/authority[.z] doesn't really fit into the schema that prefixes everything related to the voting process with /tor/status-vote/(next|current)/. A more consistent choice would be /tor/status-vote/current/vote-info[.z]. - The WFU and MTBF thresholds are already contained in votes in "flag-thresholds" lines since February 2013 (admittedly, four years after the proposal was written). We should either use the same line format in vote-info documents, or leave out flag thresholds here. - The proposal says in two places that explanations should be given in English. The better approach, IMO, would be to enumerate all possible explanations in dir-spec.txt and assign error codes to them. Reasons include: a) requires fewer bytes in an authority's memory; b) requires fewer bytes in vote-info documents; c) easier for applications to process vote-info documents; d) forces us to enumerate reasons for rejecting a router descriptor or not including a router in a vote and explicitly specify them in dir-spec.txt. (Happy to help enumerating reasons.) - I'd want to make the format of vote-info documents more compact, though I don't have a good suggestion yet. (But I also didn't want to delay sending this email, so here's my half-baked thought.) Ideally, every status entry has one "r" line to identify the relay and then one line per noteworthy event. Noteworthy events are: a) the authority receives a router descriptor and decides whether to accept or reject it; b) the authority performs a reachability test that is either successful or not; c) the authority produces a vote document and decides whether to include a relay and what flags to assign. (Are there more events that are worth including?) I'm aware that you mentioned the same information in the proposal, I'm just wondering about better ways to represent it. As I said, this thought is only half-baked and will hopefully become clearer when going through the code. - The proposal says under "Risks" that it doesn't make provisions for caching these documents. But authorities have to cache these documents! An authority can only generate a vote-info document at the same time as it generates a vote document. Any later attempts to say why it voted the way it did would require the authority to keep state that it doesn't need for anything else. The authority should simply write its vote-info document to disk and serve it whenever somebody asks for it. (To be extra precise, it should only serve a vote-info document when the consensus becomes valid.) There, that concludes my review of directory-protocol related proposals. Looking forward to what you think. Please let me know how I can best move these proposals forward, e.g., by writing patches to the proposals or dir-spec.txt, by writing code, etc. All the best, Karsten
On Thu, Dec 26, 2013 at 6:05 AM, Karsten Loesing <karsten@torproject.org> wrote:
On 12/17/13 10:31 PM, Nick Mathewson wrote:
164 Reporting the status of server votes
This proposal explains a way for authorities to provide a slightly more verbose document that relay operators can use to diagnose reasons that their router was or was not listed in the consensus. These documents would be like slightly more verbose versions of the authorities' votes, and would explain *why* the authority voted as it did. It wouldn't be too hard to implement, and would be a fine project for somebody who wants to get to know the directory code. (5/2011)
Hi Nick,
I very much like this proposal! I want to help move it forward and integrate the additional information into Onionoo, so that people can diagnose the network better. Knowing why an authority rejected a descriptor, when it last performed a successful and an unsuccessful reachability test, why it didn't include a relay in its vote, why it assigned which relay flags, etc. can be very helpful information.
Here's some feedback:
- The URL /tor/status-vote-info/current/authority[.z] doesn't really fit into the schema that prefixes everything related to the voting process with /tor/status-vote/(next|current)/. A more consistent choice would be /tor/status-vote/current/vote-info[.z].
Sounds okay to me.
- The WFU and MTBF thresholds are already contained in votes in "flag-thresholds" lines since February 2013 (admittedly, four years after the proposal was written). We should either use the same line format in vote-info documents, or leave out flag thresholds here.
If they're already in votes, then we should just include them verbatim in vote-info. vote-info should not diverge from vote needlessly.
- The proposal says in two places that explanations should be given in English. The better approach, IMO, would be to enumerate all possible explanations in dir-spec.txt and assign error codes to them. Reasons include: a) requires fewer bytes in an authority's memory; b) requires fewer bytes in vote-info documents; c) easier for applications to process vote-info documents; d) forces us to enumerate reasons for rejecting a router descriptor or not including a router in a vote and explicitly specify them in dir-spec.txt. (Happy to help enumerating reasons.)
There will always be more explanations we didn't think of; what if we do something that uses enumerated error codes *and* explanatory messages? The total size of vote-info documents won't actually be affected if we store them compressed. Perhaps we should make compression mandatory.
- I'd want to make the format of vote-info documents more compact, though I don't have a good suggestion yet. (But I also didn't want to delay sending this email, so here's my half-baked thought.) Ideally, every status entry has one "r" line to identify the relay and then one line per noteworthy event. Noteworthy events are: a) the authority receives a router descriptor and decides whether to accept or reject it; b) the authority performs a reachability test that is either successful or not; c) the authority produces a vote document and decides whether to include a relay and what flags to assign. (Are there more events that are worth including?) I'm aware that you mentioned the same information in the proposal, I'm just wondering about better ways to represent it. As I said, this thought is only half-baked and will hopefully become clearer when going through the code.
- The proposal says under "Risks" that it doesn't make provisions for caching these documents. But authorities have to cache these documents! An authority can only generate a vote-info document at the same time as it generates a vote document. Any later attempts to say why it voted the way it did would require the authority to keep state that it doesn't need for anything else. The authority should simply write its vote-info document to disk and serve it whenever somebody asks for it. (To be extra precise, it should only serve a vote-info document when the consensus becomes valid.)
I meant that these new documents aren't cached at directory caches. (And they possibly shouldn't be.) But this opens a risk of creating more traffic at authorities, which wouldn't be good.
There, that concludes my review of directory-protocol related proposals. Looking forward to what you think. Please let me know how I can best move these proposals forward, e.g., by writing patches to the proposals or dir-spec.txt, by writing code, etc.
For proposal 164, I'd love it if you can patch the proposal to make the uncontroversial changes above. (that is, the changes that we can agree on how to do.) Writing patches to merge stuff into dir-spec isn't useful without code; we don't change dir-spec until after the code changes, per 001-process.txt. As for writing code, that sounds fine! Let me know which ones you're most interested in working on some time and we can figure out how to prioritize? Most of these are currently in "Lorax" status[*], meaning that I agree that they'd be a good idea, but nobody is currently writing code for them or making a plan to do so. (I think I got a partial implementation of 185 at some point.) In some cases (like 212) the biggest issue is testing. [*] "Unless someone like you cares a whole awful lot Nothing is going to get better. It's not!" --Dr. Seuss, _The Lorax_ -- Nick
participants (4)
-
Karsten Loesing -
Nick Mathewson -
Nick Mathewson -
Roger Dingledine