commit 739702d077dc724d58e95758f10ba2876b546026 Author: Nick Mathewson nickm@torproject.org Date: Tue May 9 10:00:43 2017 -0400

Merge prop140 into dir-spec.txt --- dir-spec.txt | 110 +++++++++++++++++++++++++++++++++++++- proposals/000-index.txt | 4 +- proposals/140-consensus-diffs.txt | 3 +- tor-spec.txt | 2 + 4 files changed, 115 insertions(+), 4 deletions(-)

diff --git a/dir-spec.txt b/dir-spec.txt index 4223c57..3565c7a 100644 --- a/dir-spec.txt +++ b/dir-spec.txt @@ -1899,7 +1899,17 @@ authority votes required for a fresh shared random value to be written in the consensus (this rule only applies on the first commit round of the shared randomness protocol). - Min: 1. Max: INT32_MAX. Default: 2/3 of the total number of dirauth. + Min: 1. Max: INT32_MAX. Default: 2/3 of the total number of + dirauth. + + "max-consensuses-age-to-cache-for-diff" -- Determines how + much consensus history (in hours) relays should try to cache + in order to serve diffs. (min 0, max 8192, default 72) + + "try-diff-for-consensus-newer-than" -- This parameter + determines how old a consensus can be (in hours) before a + client should no longer try to find a diff for it. (min 0, + max 8192, default 72)

"shared-rand-previous-value" SP NumReveals SP Value NL

@@ -3109,6 +3119,7 @@ The following methods have incorrect implementations; authorities SHOULD /tor/status-vote/(current|next)/consensus-F.z. and /tor/status-vote/(current|next)/consensus-F/<FP1>+....z.

+ 4. Directory cache operation

All directory caches implement this section, except as noted. @@ -3193,6 +3204,60 @@ The following methods have incorrect implementations; authorities SHOULD documents are missing. Caches download from authorities. We follow the same splitting and back-off rules as in section 4.2.

+4.5. Consensus diffs + + Instead of downloading an entire consensus, clients may download + a "diff" document containing an ed-style diff from a previous + consensus document. Caches (and authorities) make these diffs as + they learn about new consensuses. To do so, they must store a + record of older consensuses. + + (Support for consensus diffs was added in 0.3.1.1-alpha, and is + advertised with the DirCache protocol version "2" or later.) + +4.5.1. Consensus diff format + + Consensus diffs are formatted as follows: + + The first line is "network-status-diff-version 1" NL + + The second line is + "hash" SP FromDigest SP ToDigest NL + + where FromDigest is the hex-encoded SHA3-256 digest of the _signed + part_ of the consensus that the diff should be applied to, and + ToDigest is the hex-encoded SHA3-256 digest of the _entire_ + consensus resulting from applying the diff. (See 3.4.1 for + information on that part of a consensus is signed.) + + The third and subsequent lines encode the diff from FromDigest to + ToDigest in a limited subset of the ed diff format, as specified + in appendix E. + +4.5.2. Serving and requesting diffs. + + When downloading the current consensus, a client may include an + HTTP header of the form + + X-Or-Diff-From-Consensus: HASH1, HASH2, ... + + where the HASH values are hex-encoded SHA3-256 digests of the + _signed part_ of one or more consensuses that the client knows + about. + + If a cache knows a consensus diff from one of those consensuses + to the most recent consensus of the requested flavor, it may + send that diff instead of the specified consensus. + + Caches also serve diffs from the URIs: + + /tor/status-vote/current/consensus/diff/<HASH>/<FPRLIST>.z + /tor/status-vote/current/consensus-<FLAVOR>/diff/<HASH>/<FPRLIST>.z + + where FLAVOR is the consensus flavor, defaulting to "ns", and + FPRLIST is +-separated list of recognized authority identity + fingerprints as in appendix B. + 5. Client operation

Every Tor that is not a directory server (that is, those that do @@ -3641,3 +3706,46 @@ D. Inferring missing proto lines.

For Microdesc and Cons, Tor versions before 0.2.7.stable should be taken to support version 1; 0.2.7.stable and later should have 1-2. + +E. Limited ed diff format + + We support the following format for consensus diffs. It's a + subset of the ed diff format, but clients MUST NOT accept other + ed commands. + + We support the following ed commands, each on a line by itself: + - "<n1>d" Delete line n1 + - "<n1>,<n2>d" Delete lines n1 through n2, inclusive + - "<n1>,$d" Delete line n1 through the end of the file, inclusive. + - "<n1>c" Replace line n1 with the following block + - "<n1>,<n2>c" Replace lines n1 through n2, inclusive, with the + following block. + - "<n1>a" Append the following block after line n1. + - "a" Append the following block after the current line. + + Note that line numbers always apply to the file after all previous + commands have already been applied. Note also that line numbers + are 1-indexed. + + The commands MUST apply to the file from back to front, such that + lines are only ever referred to by their position in the original + file. + + If there are any directory signatures on the original document, the + first command MUST be a "<n1>,$d" form to remove all of the directory + signatures. Using this format ensures that the client will + successfully apply the diff even if they have an unusual encoding for + the signatures. + + The "current line" is either the first line of the file, if this is + the first command, the last line of a block we added in an append or + change command, or the line immediate following a set of lines we just + deleted (or the last line of the file if there are no lines after + that). + + The replace and append command take blocks. These blocks are simply + appended to the diff after the line with the command. A line with + just a period (".") ends the block (and is not part of the lines + to add). Note that it is impossible to insert a line with just + a single dot. + diff --git a/proposals/000-index.txt b/proposals/000-index.txt index 2bfebbb..eca98ce 100644 --- a/proposals/000-index.txt +++ b/proposals/000-index.txt @@ -60,7 +60,7 @@ Proposals by number: 137 Keep controllers informed as Tor bootstraps [CLOSED] 138 Remove routers that are not Running from consensus documents [CLOSED] 139 Download consensus documents only when it will be trusted [CLOSED] -140 Provide diffs between consensuses [ACCEPTED] +140 Provide diffs between consensuses [CLOSED] 141 Download server descriptors on demand [OBSOLETE] 142 Combine Introduction and Rendezvous Points [DEAD] 143 Improvements of Distributed Storage for Tor Hidden Service Descriptors [SUPERSEDED] @@ -255,7 +255,6 @@ Proposals by status: 276 Report bandwidth with lower granularity in consensus documents [for 0.3.1.x-alpha] 277 Detect multiple relay instances running with same ID [for 0.3.??] ACCEPTED: - 140 Provide diffs between consensuses 172 GETINFO controller option for circuit information 173 GETINFO Option Expansion 188 Bridge Guards and other anti-enumeration defenses @@ -309,6 +308,7 @@ Proposals by status: 137 Keep controllers informed as Tor bootstraps [in 0.2.1.x] 138 Remove routers that are not Running from consensus documents [in 0.2.1.2-alpha] 139 Download consensus documents only when it will be trusted [in 0.2.1.x] + 140 Provide diffs between consensuses [in 0.3.1.1-alpha] 148 Stream end reasons from the client side should be uniform [in 0.2.1.9-alpha] 150 Exclude Exit Nodes from a circuit [in 0.2.1.3-alpha] 152 Optionally allow exit from single-hop circuits [in 0.2.1.6-alpha] diff --git a/proposals/140-consensus-diffs.txt b/proposals/140-consensus-diffs.txt index 5a8be3f..7793f84 100644 --- a/proposals/140-consensus-diffs.txt +++ b/proposals/140-consensus-diffs.txt @@ -2,7 +2,8 @@ Filename: 140-consensus-diffs.txt Title: Provide diffs between consensuses Author: Peter Palfrader Created: 13-Jun-2008 -Status: Accepted +Implemented-In: 0.3.1.1-alpha +Status: Closed

0. History

diff --git a/tor-spec.txt b/tor-spec.txt index 927173b..ccbe7fb 100644 --- a/tor-spec.txt +++ b/tor-spec.txt @@ -1750,6 +1750,8 @@ see tor-design.pdf.

"1" -- supports all features in Tor 0.2.4.19.

+ "2" -- adds support for consensus diffs. + 9.8. "Desc"

Describes features present or absent in descriptors.