Proposal 162: Publish the consensus in multiple flavors
nickm at torproject.org
Fri May 15 17:05:41 UTC 2009
Title: Publish the consensus in multiple flavors
Author: Nick Mathewson
This proposal describes a way to publish each consensus in
multiple simultaneous formats, or "flavors". This will reduce the
amount of time needed to deploy new consensus-like documents, and
reduce the size of consensus documents in the long term.
In the future, we will almost surely want different fields and
data in the network-status document. Examples include:
- Publishing hashes of microdescriptors instead of hashes of
full descriptors (Proposal 158).
- Including different digests of descriptors, instead of the
Note that in both cases, from the client's point of view, this
information _replaces_ older information. If we're using a
SHA256 hash, we don't need to see the SHA1. If clients only want
microdescriptors, they don't (necessarily) need to see hashes of
Our past approach to cases like this has been to shovel all of
the data into the consensus document. But this is rather poor
for bandwidth. Adding a single SHA256 hash to a consensus for
each router increases the compressed consensus size by 47%. In
comparison, replacing a single SHA1 hash with a SHA256 hash for
each listed router increases the consensus size by only 18%.
Design in brief:
Let the voting process will remain as it is, until a consensus is
generated. With future versions of the voting algorithm, instead
of just a single consensus being generated, multiple consensus
"flavors" are produced.
Consensuses (all of them) include a list of which flavors are
being generated. Caches fetch and serve all flavors of consensus
that are listed, regardless of whether they can parse or validate
them, and serve them to clients. Thus, once this design is in
place, we won't need to deploy more cache changes in order to get
new flavors of consensus to be cached.
Clients download only the consensus flavor they want.
A note on hashes:
Everything in this document is specified to use SHA256, and to be
upgradeable to use better hashes in the future.
1. URLs and changes to the current consensus format.
Every consensus flavor has a name consisting of a sequence of one
or more alphanumeric characters and dashes. For compatibility
current descriptor flavor is called "ns".
The supported consensus flavors are defined as part of the
authorities' consensus method.
For each supported flavors, every authority calculates another
consensus document of as-yet-unspecified format, and exchange
detached signatures for these documents as in the current consensus
In addition to the consensus currently served at
/tor/status-vote/(current|next)/consensus.z , authorities serve
another consensus of each flavor "F" from the location
When caches serve these documents, they do so from the same
2. Document format: generic consensus.
The format of a flavored consensus is as-yet-unspecified, except
that the first line is:
"network-status-version" SP version SP flavor NL
where version is 3 or higher, and the flavor is a string
consisting of alphanumeric characters and dashes, matching the
corresponding flavor listed in the unflavored consensus.
3. Document format: detached signatures.
In addition to the current detached signature format, we allow
the first line to take the form,
"consensus-digest" SP flavor SP 1*(Algname "=" Digest) NL
The consensus-signatures URL should contain the signatures
for _all_ flavors of consensus.
4. The consensus index:
Authorities additionally generate and serve a consensus-index
document. Its format is:
Header ValidAfter ValidUntil Documents Signatures
Header = "consensus-index" SP version NL
ValidAfter = as in a consensus
ValidUntil = as in a consensus
Documents = Document*
Document = "document" SP flavor SP SignedLength
1*(SP AlgorithmName "=" Digest) NL
Signatures = Signature *
Signature = "directory-signature" SP algname SP identity
SP signing-key-digest NL signature
There must be one Document line for each generated consensus flavor
Each Document line describes the length of the signed portion of
a consensus (the signatures themselves are not included), along
with one or more digests of that signed portion. Digests are
given in hex. The algorithm "sha256" MUST be included; others
The algname part of a signature describes what algorithm was
used to hash the identity and signing keys, and to compute the
signature. The algorithm "sha256" MUST be recognized;
signatures with unrecognized algorithms MUST be ignored.
The consensus index is made available at
Caches should fetch this document so they can check the
correctness of the different consensus documents they fetch.
They do not need to check anything about an unrecognized
consensus document beyond its digest.
4.1. The "sha256" signature format.
The 'SHA256' signature format for directory objects is defined as
the RSA signature of the OAEP+-padded SHA256 digest of the SHA256
digest of the the item to be signed. When checking signatures,
the signature MUST be treated as valid if the signed material
begins with SHA256(SHA256(document)); this allows us to add other
- We should not create a new flavor of consensus when adding a
field wouldn't be too onerous.
- We should not proliferate flavors lightly: clients will be
distinguishable based on which flavor they download.
- Stage one: authorities begin generating and serving
- Stage two: Caches begin downloading consenusus-index files,
validating them, and using them to decide what flavors of
consensus documents to cache. They download all listed
documents, and compare them to the digests given in the
- Stage three: Once we want to make a significant change to the
consensus format, we deploy another flavor of consensus at the
authorities. This will immediately start getting cached by the
caches, and clients can start fetching the new flavor without
waiting a version or two for enough caches to begin supporting
More information about the tor-dev