[or-cvs] r16857: {} Rename spec to be useful ; remove redundant copy of matt's s (updater/trunk/specs)

nickm at seul.org nickm at seul.org
Thu Sep 11 19:58:23 UTC 2008


Author: nickm
Date: 2008-09-11 15:58:23 -0400 (Thu, 11 Sep 2008)
New Revision: 16857

Added:
   updater/trunk/specs/glider-spec.txt
Removed:
   updater/trunk/specs/U1-automatic-updates.txt
   updater/trunk/specs/U2-formats.txt
Log:
Rename spec to be useful ; remove redundant copy of matt's spec draft.

Deleted: updater/trunk/specs/U1-automatic-updates.txt
===================================================================
--- updater/trunk/specs/U1-automatic-updates.txt	2008-09-11 19:57:46 UTC (rev 16856)
+++ updater/trunk/specs/U1-automatic-updates.txt	2008-09-11 19:58:23 UTC (rev 16857)
@@ -1,377 +0,0 @@
-
-Filename: xxx-automatic-updates.txt
-Title: Automatic Software Update Protocol
-Version: $Revision$
-Last-Modified: $Date$
-Author: Matt Edman
-Created: 30-July-2008
-Status: Open
-
-
-Scope
-
-  This proposal specifies the method by which an automatic update client can
-  determine the most recent recommended Tor installation package for the
-  user's platform, download the package, and then verify that the package was
-  downloaded successfully. While this proposal focuses on only the Tor
-  software, the protocol defined is sufficiently extensible such that other
-  components of the Tor bundles, like Vidalia, Polipo, and Torbutton, can be
-  managed and updated by the automatic update client as well.
-
-  The initial target platform for the automatic update framework is Windows,
-  given that's the platform used by a majority of our users and that it lacks
-  a sane package management system that many Linux distributions already have.
-  Our second target platform will be Mac OS X, and so the protocol will be
-  designed with this near-future direction in mind.
-
-  Other client-side aspects of the automatic update process, such as user
-  interaction, the interface presented, and actual package installation
-  procedure, are outside the scope of this proposal.
-
-
-Motivation
-
-  Tor releases new versions frequently, often with important security,
-  anonymity, and stability fixes. Thus, it is important for users to be able
-  to promptly recognize when new versions are available and to easily
-  download, authenticate, and install updated Tor and Tor-related software
-  packages.
-  
-  Tor's control protocol [2] provides a method by which controllers can
-  identify when the user's Tor software is obsolete or otherwise no longer
-  recommended. Currently, however, no mechanism exists for clients to
-  automatically download and install updated Tor and Tor-related software for
-  the user.
-
-
-Design Overview
-
-  The core of the automatic update framework is a well-defined file called a
-  "recommended-packages" file. The recommended-packages file is accessible via
-  HTTP[S] at one or more well-defined URLs. An example recommended-packages
-  URL may be:
-
-    https://updates.torproject.org/recommended-packages
-
-  The recommended-packages document is formatted according to Section 1.2
-  below and specifies the most recent recommended installation package
-  versions for Tor or Tor-related software, as well as URLs at which the
-  packages and their signatures can be downloaded.
-
-  An automatic update client process runs on the Tor user's computer and
-  periodically retrieves the recommended-packages file according to the method
-  described in Section 2.0. As described further in Section 1.2, the
-  recommended-packages file is signed and can be verified by the automatic
-  update client with one or more public keys included in the client software.
-  Since it is signed, the recommended-packages file can be mirrored by
-  multiple hosts (e.g., Tor directory authorities), whose URLs are included in
-  the automatic update client's configuration.
-
-  After retrieving and verifying the recommended-packages file, the automatic
-  update client compares the versions of the recommended software packages
-  listed in the file with those currently installed on the end-user's
-  computer. If one or more of the installed packages is determined to be out
-  of date, an updated package and its signature will be downloaded from one of
-  the package URLs listed in the recommended-packages file as described in
-  Section 2.2.
-
-  The automatic update system uses a multilevel signing key scheme for package
-  signatures. There are a small number of entities we call "packaging
-  authorities" that each have their own signing key. A packaging authority is
-  responsible for signing and publishing the recommended-packages file.
-  Additionally, each individual packager responsible for producing an
-  installation package for one or more platforms has their own signing key.
-  Every packager's signing key must be signed by at least one of the packaging
-  authority keys.
-
-
-Specification
-
-  1. recommended-packages Specification
-
-  In this section we formally specify the format of the published
-  recommended-packages file.
-
-  1.1. Document Meta-format
-  
-  The recommended-packages document follows the lightweight extensible
-  information format defined in Tor's directory protocol specification [1]. In
-  the interest of self-containment, we have reproduced the relevant portions
-  of that format's specification in this Section. (Credits to Nick Mathewson
-  for much of the original format definition language.)
-  
-  The highest level object is a Document, which consists of one or more
-  Items.  Every Item begins with a KeywordLine, followed by zero or more
-  Objects. A KeywordLine begins with a Keyword, optionally followed by
-  whitespace and more non-newline characters, and ends with a newline.  A
-  Keyword is a sequence of one or more characters in the set [A-Za-z0-9-].
-  An Object is a block of encoded data in pseudo-Open-PGP-style
-  armor. (cf. RFC 2440)
-
-  More formally:
-
-    Document     ::= (Item | NL)+
-    Item         ::= KeywordLine Object*
-    KeywordLine  ::= Keyword NL | Keyword WS ArgumentChar+ NL
-    Keyword      ::= KeywordChar+
-    KeywordChar  ::= 'A' ... 'Z' | 'a' ... 'z' | '0' ... '9' | '-'
-    ArgumentChar ::= any printing ASCII character except NL.
-    WS           ::= (SP | TAB)+
-    Object       ::= BeginLine Base-64-encoded-data EndLine
-    BeginLine    ::= "-----BEGIN " Keyword "-----" NL
-    EndLine      ::= "-----END " Keyword "-----" NL
-
-    The BeginLine and EndLine of an Object must use the same keyword.
-  
-  In our Document description below, we also tag Items with a multiplicity in
-  brackets. Possible tags are:
-
-    "At start, exactly once": These items MUST occur in every instance of the
-    document type, and MUST appear exactly once, and MUST be the first item in
-    their documents.
-        
-    "Exactly once": These items MUST occur exactly one time in every
-    instance of the document type.
-
-    "Once or more": These items MUST occur at least once in any instance
-    of the document type, and MAY occur more than once.
-
-    "At end, exactly once": These items MUST occur in every instance of
-    the document type, and MUST appear exactly once, and MUST be the
-    last item in their documents.
-
-  1.2. recommended-packages Document Format
-
-  When interpreting a recommended-packages Document, software MUST ignore
-  any KeywordLine that starts with a keyword it doesn't recognize; future
-  implementations MUST NOT require current automatic update clients to
-  understand any KeywordLine not currently described.
-
-  In lines that take multiple arguments, extra arguments SHOULD be
-  accepted and ignored.
-
-  The currently defined Items contained in a recommended-packages document
-  are:
-
-    "recommended-packages-format" SP number NL
-
-      [Exactly once]
-
-      This Item specifies the version of the recommended-packages format that
-      is contained in the subsequent document. The version defined in this
-      proposal is version "1". Subsequent iterations of this protocol MUST
-      increment this value if they introduce incompatible changes to the
-      document format and MAY increment this value if they only introduce
-      additional Keywords.
-
-    "published" SP YYYY-MM-DD SP HH:MM:SS NL
-
-      [Exactly once]
-
-      The time, in GMT, when this recommended-packages document was generated.
-      Automatic update clients SHOULD ignore Documents over 60 days old.
-
-    "tor-stable-win32-version" SP TorVersion NL
-
-      [Exactly once]
-
-      This keyword specifies the latest recommended release of Tor's "stable"
-      branch for the Windows platform that has an installation package
-      available. Note that this version does not necessarily correspond to the
-      most recently tagged stable Tor version, since that version may not yet
-      have an installer package available, or may have known issues on
-      Windows.
-
-      The TorVersion field is formatted according to Section 2 of Tor's 
-      version specification [3].
-
-    "tor-stable-win32-package" SP Url NL
-
-      [Once or more]
-
-      This Item specifies the location from which the most recent
-      recommended Windows installation package for Tor's stable branch can be
-      downloaded.
-    
-      When this Item appears multiple times within the Document, automatic
-      update clients SHOULD select randomly from the available package
-      mirrors.
-
-    "tor-dev-win32-version" SP TorVersion NL
-
-      [Exactly once]
-    
-      This Item specifies the latest recommended release of Tor's
-      "development" branch for the Windows platform that has an installation
-      package available. The same caveats from the description of
-      "tor-stable-win32-version" also apply to this keyword.
-
-      The TorVersion field is formatted according to Section 2 of Tor's
-      version specification [3].
-
-    "tor-dev-win32-package" SP Url NL
-  
-      [Once or more]
-
-      This Item specifies the location from which the most recent recommended
-      Windows installation package and its signature for Tor's development
-      branch can be downloaded.
-
-      When this Keyword appears multiple times within the Document, automatic
-      update clients SHOULD select randomly from the available package
-      mirrors.
-
-    "signature" NL SIGNATURE NL
-
-      [At end, exactly once]
-
-      The "SIGNATURE" Object contains a PGP signature (using a packaging
-      authority signing key) of the entire document, taken from the beginning
-      of the "recommended-packages-format" keyword, through the newline after
-      the "signature" Keyword.
-
-
-  2. Automatic Update Client Behavior
-
-  The client-side component of the automatic update framework is an
-  application that runs on the end-user's machine. It is responsible for
-  fetching and verifying a recommended-packages document, as well as
-  downloading, verifying, and subsequently installing any necessary updated
-  software packages.
-
-  2.1. Download and verify a recommended-packages document
-
-  The first step in the automatic update process is for the client to download
-  a copy of the recommended-packages file. The automatic update client
-  contains a (hardcoded and/or user-configurable) list of URLs from which it
-  will attempt to retrieve a recommended-packages file.
-
-  Connections to each of the recommended-packages URLs SHOULD be attempted in
-  the following order:
-
-    1) HTTPS over Tor
-    2) HTTP over Tor
-    3) Direct HTTPS
-    4) Direct HTTP
-
-  If the client fails to retrieve a recommended-packages document via any of
-  the above connection methods from any of the configured URLs, the client
-  SHOULD retry its download attempts following an exponential back-off
-  algorithm. After the first failed attempt, the client SHOULD delay one hour
-  before attempting again, up to a maximum of 24 hours delay between retry
-  attempts.
-
-  After successfully downloading a recommended-packages file, the automatic
-  update client will verify the signature using one of the public keys
-  distributed with the client software. If more than one recommended-packages
-  file is downloaded and verified, the file with the most recent "published"
-  date that is verified will be retained and the rest discarded.
-
-  2.2. Download and verify the updated packages
-
-  The automatic update client next compares the latest recommended package
-  version from the recommended-packages document with the currently installed
-  Tor version. If the user currently has installed a Tor version from Tor's
-  "development" branch, then the version specified in "tor-dev-*-version" Item
-  is used for comparison. Similarly, if the user currently has installed a Tor
-  version from Tor's "stable" branch, then the version specified in the
-  "tor-stable-*version" Item is used for comparison. Version comparisons are
-  done according to Tor's version specification [3].
-
-  If the automatic update client determines an installation package newer than
-  the user's currently installed version is available, it will attempt to
-  download a package appropriate for the user's platform and Tor branch from a
-  URL specified by a "tor-[branch]-[platform]-package" Item. If more than one
-  mirror for the selected package is available, a mirror will be chosen at
-  random from all those available.
-
-  The automatic update client must also download a ".asc" signature file for
-  the retrieved package. The URL for the package signature is the same as that
-  for the package itself, except with the extension ".asc" appended to the
-  package URL.
-
-  Connections to download the updated package and its signature SHOULD be
-  attempted in the same order described in Section 2.1.
-
-  After completing the steps described in Sections 2.1 and 2.2, the automatic
-  update client will have downloaded and verified a copy of the latest Tor
-  installation package. It can then take whatever subsequent platform-specific
-  steps are necessary to install the downloaded software updates.
-
-  2.3. Periodic checking for updates
-
-  The automatic update client SHOULD maintain a local state file in which it
-  records (at a minimum) the timestamp at which it last retrieved a
-  recommended-packages file and the timestamp at which the client last
-  successfully downloaded and installed a software update.
-
-  Automatic update clients SHOULD check for an updated recommended-packages
-  document at most once per day but at least once every 30 days.
-
-
-  3. Future Extensions
-
-  There are several possible areas for future extensions of this framework.
-  The extensions below are merely suggestions and should be the subject of
-  their own proposal before being implemented.
-  
-  3.1. Additional Software Updates
-  
-  There are several software packages often included in Tor bundles besides
-  Tor, such as Vidalia, Privoxy or Polipo, and Torbutton. The versions and
-  download locations of updated installation packages for these bundle
-  components can be easily added to the recommended-packages document
-  specification above.
-
-  3.2. Including ChangeLog Information
-
-  It may be useful for automatic update clients to be able to display for
-  users a summary of the changes made in the latest Tor or Tor-related
-  software release, before the user chooses to install the update. In the
-  future, we can add keywords to the specification in Section 1.2 that specify
-  the location of a ChangeLog file for the latest recommended package
-  versions. It may also be desirable to allow localized ChangeLog information,
-  so that the automatic update client can fetch release notes in the
-  end-user's preferred language.
-
-  3.3. Weighted Package Mirror Selection
-
-  We defined in Section 1.2 a method by which automatic update clients can
-  select from multiple available package mirrors. We may want to add a Weight
-  argument to the "*-package" Items that allows the recommended-packages file
-  to suggest to clients the probability with which a package mirror should be
-  chosen. This will allow clients to more appropriately distribute package
-  downloads across available mirrors proportional to their approximate
-  bandwidth.
-
-
-Implementation
-
-  Implementation of this proposal will consist of two separate components.
-
-  The first component is a small "au-publish" tool that takes as input a
-  configuration file specifying the information described in Section 1.2 and a
-  private key. The tool is run by a "packaging authority" (someone responsible
-  for publishing updated installation packages), who will be prompted to enter
-  the passphrase for the private key used to sign the recommended-packages
-  document. The output of the tool is a document formatted according to
-  Section 1.2, with a signature appended at the end. The resulting document
-  can then be published to any of the update mirrors.
-
-  The second component is an "au-client" tool that is run on the end-user's
-  machine. It periodically checks for updated installation packages according
-  to Section 2 and fetches the packages if necessary. The public keys used
-  to sign the recommended-packages file and any of the published packages are
-  included in the "au-client" tool.
-
-
-References
-  
-  [1] Tor directory protocol (version 3),
-  https://tor-svn.freehaven.net/svn/tor/trunk/doc/spec/dir-spec.txt
-  
-  [2] Tor control protocol (version 2),
-  https://tor-svn.freehaven.net/svn/tor/trunk/doc/spec/control-spec.txt
-
-  [3] Tor version specification,
-  https://tor-svn.freehaven.net/svn/tor/trunk/doc/spec/version-spec.txt

Deleted: updater/trunk/specs/U2-formats.txt
===================================================================
--- updater/trunk/specs/U2-formats.txt	2008-09-11 19:57:46 UTC (rev 16856)
+++ updater/trunk/specs/U2-formats.txt	2008-09-11 19:58:23 UTC (rev 16857)
@@ -1,627 +0,0 @@
-
-0. Preliminaries
-
-0.0. Scope
-
-   This document describes a system for distributing Tor bundle updates.
-
-0.1. Proposed code name
-
-   Since "auto-update" is so generic, I've been thinking about going with
-   "glider", based on the sugar glider you get when you search for "handy
-   pocket creature".  I haven't yet done a search to find out whether
-   somebody else is using the name, so we shouldn't get too attached to it
-   before we see if it's taken.
-
-0.2. Goals
-
-   Once Tor was a single executable that you could just run.  Then it
-   required Privoxy.  Now, thanks to the Tor Browser Bundle and related
-   projects, a full installation can contain Tor, Privoxy, Torbutton,
-   Firefox, and more.
-
-   We need to keep this software updated.  When we make security fixes,
-   quick uptake helps narrow the window in which attackers can exploit
-   them.
-
-   We need updates to be easy.  Each additional step a user must take to
-   get updated means that more users will stay with older insecure
-   versions.
-
-   We need updates to be secure.  We're supposed to be good at crypto;
-   let's act like it.  There is no good reason in this day and age to
-   subject users to rollback attacks or unsigned packages or whatever.
-
-   We need administration to be simple.  Tor doesn't have a release
-   engineering team, so we can't add too many hard steps to putting out
-   a new release.
-
-   The system should be easy to implement; we may need to do multiple
-   implementations on the client side at least.
-
-0.2.1. Goals for package formats and PKIs
-
-   It should be possible to mirror a repository using only rsync and
-   cron.
-
-   Separate keys should be used for different people and different
-   roles.
-
-   Only a minimal set of keys should have to be kept online to keep
-   the system running.
-
-   The system should handle any single computer or system or person
-   being unavailable.
-
-   The formats and protocols should be pretty future-proof.
-
-0.3. Non-goals
-
-   This is not a general-purpose package manager like yum or apt: it
-   assumes that users will want to have one or more of a set of
-   "bundles", not an arbitrary selection of packages dependant on one
-   another.  (Rationale: these systems do what they do pretty well.)
-
-   This is also not a general-purpose package format.  It assumes the
-   existence of an external package format that can handle install,
-   update, remove, and version query.  (Rationale:
-
-1. System overview
-
-   The basic unit of updatability is a "bundle".  A bundle is a set of
-   software components, or "packages", plus some rules about installing
-   them.  Example bundles could be "Tor Browser, stable series" or
-   "Basic Tor, development series".
-
-   When Glider has responsibility for keeping a bundle up to date, we
-   say that a user has "subscribed" to that bundle.
-
-   Conceptually, there are four parts to keeping a bundle up to date:
-
-      Polling:
-        - Periodically, Glider asks a mirror whether there is a newer
-          version of some bundle that a user has subscribed to.  If so,
-          Glider determines what's in the bundle.
-
-      Fetching:
-        - If the bundle contains packages that Glider hasn't installed
-          or hasn't cached, it needs to download them from a mirror.
-          This can happen over any protocol; v1 should support at least
-          http and https-over-Tor.  V1 should also support resuming
-          partial downloads, since many users have unreliable
-          connections.
-
-          Later versions could support Bittorrent, or whatever.
-
-      Validation:
-        - Throughout the process, Glider must ensure that all the
-          bundles are signed correctly, all the packages are signed
-          correctly, and everything is up-to-date.
-
-          We want to specify this so that users can't be tricked about
-          the contents of a bundle, can't install a malicious package,
-          and can't be fooled into believing that an old bundle is
-          actually the latest.
-
-      Installation:
-        - Now Glider has a set of packages to install.  The format of
-          these packages will be platform-dependent: they could be pkg
-          files on OSX, MSI files on Win32, RPMs or DEBs on Linux, and
-          so on.  Glider should query the user for permission to start,
-          then install the packages.
-
-1.1. The repository
-
-   Each Glider instance knows about one or more "repositories".  A
-   repository is a filesystem somewhere that contains the packages in a
-   set of bundles, and some associated metadata.  A repository must
-   exist at one or more canonical hosts, and may have a number of full
-   or partial mirrors.
-
-   In v1, each Glider instance will know about only one repository.
-
-1.2. The PKI
-
-   The trust root for the whole system is, necessarily, whatever users
-   download when they first download a copy of Glider.  We need to make
-   sure that the first download happens from a site we trust, using
-   HTTPS.
-
-   Glider ships with root keys, which in turn are used to verify the
-   keys for all the other roles.  There are a few root keys, operated by
-   trusted admins for the system.  If root keys ever need to be changed,
-   we can just ship an update of Glider: it's supposed to be
-   self-updating anyway.
-
-   The root keys are only used to sign a 'key list' of all the other
-   keys and their roles.  A key list is valid if it has been signed by a
-   threshold of root keys.
-
-   Each package is signed with the key of its authorized builder.  For
-   example, one volunteer may be authorized to build the mac versions of
-   several packages, and another may be authorized to build the windows
-   version of just one.
-
-   Each bundle is signed with the key of its maintainer.  It's assumed
-   that the bundle maintainer might be the package maintainer for some
-   but not all of the packages.
-
-   The list of mirrors is also signed.  If the mirror list is
-   automatically updated, this key must be kept online; otherwise, it
-   can be offline.
-
-   To prevent an adversary from replaying an out-of-date signed
-   document, an automated process periodically signs a timestamped
-   statement containing the hashes of the mirror list, the latest
-   bundles, and the key list, using yet another special-purpose key.
-   This key must be kept online.
-
-1.3. Threat Model And Analysis
-
-   We assume an adversary who can operate compromised mirrors, and who
-   can possibly compromise the main repository.  At worst, such an
-   adversary can DOS users in a way that they can detect.
-
-   We're assuming for the moment an OSX/Win32-like execution model,
-   where all packages will run equal privilege, but occasionally
-   installation will require higher privilege.  This means that once a
-   hostile package is installed, it can basically do whatever it
-   wants.  As rootkit writers demonstrate, compromise is really
-   tenuous: any attacker who can induce a user to install a hostile
-   piece of code has, in effect, permanently compromised that user
-   until they reinstall.
-
-   Thus, if an adversary compromises enough keys to sign a compromised
-   package, or tricks a packager into signing a compromised package,
-   and manages to get that package into a signed bundle, the best we
-   can do is to limit the number of users who are affected.  We do
-   this by compartmentalizing signing keys so that only the package
-   and bundle in question are at risk.
-
-   (If we had replicated build processes and a bit-by-bit reliable
-   build process, we could have multiple packagers test that a binary
-   was built properly, and multiply sign it.  This would be effective
-   against an adversary compromising a single packaging key, but not
-   against one compromising a source repository.)
-
-2. The repository layout
-
-   The filesystem layout in the repository is used for two purposes:
-     - To give mirrors an easy way to mirror only some of the repository.
-     - To specify which parts of the repository a given key has the
-       authority to sign.
-
-   The following files exist in all repositories and mirrors:
-
-    /meta/keys.txt
-
-         Signed by the root keys; indicates keys and roles.
-         [???? I'm using the txt extension here.  Is that smart?]
-
-    /meta/mirrors.txt
-
-         Signed by the mirror key; indicates which parts of the
-         repository are mirrored at what mirrors.
-
-    /meta/timestamp.txt
-
-         Signed by the timestamp key; indicates hashes and timestamps
-         for the latest versions of keys.txt and mirrors.txt.  Also
-         indicates the latest version of each bundle for each os/arch.
-
-         This is the only file that needs to be downloaded for polling.
-
-    /bundleinfo/bundlename/os-arch/bundlename-os-arch-bundleversion.txt
-
-         Signed by the appropriate bundle key.  Describes what
-         packages make up a bundle, and what order to install,
-         uninstall, and upgrade them in.
-
-    /pkginfo/packagename/os-arch/version/packagename-os-arch-packageversion.txt
-
-         Signed by the appropriate package key.  Tells the name of the
-         file that makes up a package, its hash, and what procedure
-         is used to install it.
-
-    /packages/packagename/os-arch/version/(some filename)
-
-         The actual package file.  Its naming convention will depend
-         on the underlying packaging system.
-
-3. Document formats
-
-3.1. Metaformat
-
-   All documents use Rivest's SEXP meta-format as documented at
-     http://people.csail.mit.edu/rivest/sexp.html
-   with the restriction that no "display hint" fields are to be used,
-   and the base64 transit encoding isn't used either.
-
-   (We use SEXP because it's really easy to parse, really portable,
-   and unlike most other tagged data formats, has a
-   trivially-specified canonical format suitable for hashing.)
-
-   In descriptions of syntax below, we use regex-style qualifiers, so
-   that in
-        (sofa slipcover? occupant* leg+)
-   the sofa will have an optional slipcover, zero or more occupants,
-   and one or more legs.  This pattern matches (sofa leg) and (sofa
-   slipcover occupant occupant leg leg leg leg) but not (sofa leg
-   slipcover).
-
-   We also use a braces notation to indicate elements that can occur
-   in any order.  For example,
-        (bread {flour+ eggs? yeast})
-   matches a list starting with "bread", and then containing one or
-   more  of flours, zero or one occurrences of eggs, and one
-   occurrence of yeast, in any order.  This pattern matches (bread eggs
-   yeast flour) but not (bread yeast) or (bread flour eggs yeast
-   macadamias).
-
-3.2. File formats: general principles
-
-   We use tagged lists (lists whose first element is a string) to
-   indicate typed objects.  Tags are generally lower-case, with
-   hyphens used for separation.  Think Lispy.
-
-   We use attrlists [lists of (key value) lists] to indicate a
-   multimap from keys to values.  Clients MUST accept unrecognized
-   keys in these attrlists.  The syntax for an attrlist with two
-   recognized and required keys is typically given as ({(key1 val1)
-   (key2 val2) (ATTR VAL)*}), indicating that the keys can occur in
-   any order, intermixed with other attributes.
-
-   Timestamp files will be downloaded very frequently; all other files
-   will be much smaller in size than package files.  Thus,
-   size-optimization for timestamp files makes sense and most other
-   other space optimizations don't.
-
-   Versions are represented as lists of the form (v I1 I2 I3 I4 ...)
-   where each item is a number or alphanumeric version component.  For
-   example, the version "0.2.1.5-alpha" is represented as (v 0 2 1 5
-   alpha).
-
-   All signed files are of the format:
-
-       (signed
-          X
-          (signature ({(keyid K) (method M) (ATTR VAL)*}) SIG)+
-       )
-
-   where: X is a list whose first element describes the signed object.
-          K is the identifier of a key signing the document
-          M is the method to be used to make the signature
-          (ATTR VAL) is an arbitrary list whose first element is a
-             string.
-          SIG is a signature of the canonical encoding of X using the
-          identified key.
-
-   We define two signing methods at present:
-       sha256-oaep : A signature of the SHA256 hash of the canonical
-         encoding of X, using OAEP+ padding. [XXXX say more about mgf]
-
-   All times are given as strings of the format "YYYY-MM-DD HH:MM:SS",
-   in UTC.
-
-   All keys are of the format:
-      (pubkey ({(type TYPE) (ATTR VAL)*}) KEYVAL)
-   where TYPE is a string describing the type of the key and how it's
-   used to sign documents.  The type determines the interpretation of
-   KEYVAL.
-
-   The ID of a key is the type field concatenated with the SHA-256
-   hash of the canonical encoding of the KEYVAL field.
-
-   We define one keytype at present: 'rsa'.  The KEYVAL in this case
-   is a 2-element list of (e n), with both values given in big-endian
-   binary format.  [This makes keys 45-60% more compact than using
-   decimal integers.]
-
-   All RSA keys must be at least 2048 bits long.
-
-
-   Every role in the system is associated with a key.  Replacing
-   anything but a root key is supposed to be relatively easy.
-
-   Root-keys sign other keys, and certify them as belonging to roles.
-   Clients are configured to know the root keys.
-
-   Bundle keys certify the contents of a bundle.
-
-   Package keys certify packages for a given program or set of
-   programs.
-
-   Mirror keys certify a list of mirrors.  We expect this to be an
-   automated process.
-
-   Timestamp keys certify that given versions of other metadata
-   documents are up-to-date.  They are the only keys that absolutely
-   need to be kept online.  (If they are not, timestamps won't be
-   generated.)
-
-3.3. File formats: key list
-
-   The key list file is signed by multiple root keys.  It indicates
-   which keys are authorized to sign which parts of the repository.
-
-   (keylist
-     (ts TIME)
-     (keys
-       ((key ({(roles (ROLE PATH)+) (ATTR VAL)*}) KEY)*)
-     ...
-   )
-
-   The "ts" line describes when the keys file was updated.  Clients
-   MUST NOT replace a file with an older one, and SHOULD NOT accept a
-   file too far in the future.
-
-   A ROLE is one of "timestamp" "mirrors" "bundle" or "package".
-
-   PATH is a path relative to the top of the directory hierarchy.  It
-   may contain "*" elements to indicate "any file", and may end with a
-   "/**" element to indicate all files under a given point.
-
-3.4. File formats: mirror list
-
-   The mirror list is signed by a mirror key.  It indicates which
-   mirrors are active and believed to be mirroring which parts of the
-   repository.
-
-   (mirrorlist
-     (ts TIME)
-     (mirrors
-       ( (mirror ({(name N) (urlbase U) (contents PATH+) (weight W)
-                   (official)?  (ATTR VAL)})) * )
-     ...
-  )
-
-  Every mirror is a copy of some or all of the directory hierarchy
-  containing at least the /meta, /bundles/, and /pkginfo directories.
-
-  N is a descriptive name for the mirror; U is the URL of the mirror's
-  base (i.e., the parent of the "meta" directory); and the PATH
-  elements are the components describing how much of the packages
-  directory is mirrored.  Their format is as in the keylist file.
-
-  W is an integer used to weight mirrors when picking at random;
-  mirrors with more bandwidth should have higher weigths.   The
-  "official" element should only be present if the mirror is (one of
-  the) official repositories operated by the Tor Project.
-
-3.5. File formats: timestamp files
-
-  The timestamp file is signed by a timestamp key.  It indicates the
-  latest versions of other files, and contains a regularly updated
-  timestamp to prevent rollback attacks.
-
-  (ts
-    ({(at TIME)
-      (m TIME MIRRORLISTHASH)
-      (k TIME KEYLISTHASH)
-      (b NAME VERSION TIME PATH HASH)*})
-  )
-
-  TIME is when the timestamp was signed.  MIRRORLISTHASH is the digest
-  of the mirror-list file; KEYLISTHASH is the digest of the key list
-  file; and the 'b' entries are a list of the latest version of each
-  bundles and their locations and hashes.
-
-3.6. File formats: bundle files
-
-  (bundle
-    (at TIME)
-    (os OS)
-    [(arch ARCH)]
-    (version V)
-    (packages
-      (NAME VERSION PATH HASH ({(order INST UPDATE REMOVE)
-                                (optional)?
-                                (gloss LANG TEXT)*
-                                (longloss LANG TEXT)*
-                                 (ATTR VAL)*})? )* )
-  )
-
-  Most elements are self-explanatory; the INST, UPDATE, and REMOVE
-  elements of the order element are numbers defining the order in
-  which the packages are installed, updated, and removed respectively.
-  The "optional" element is present if the package is optional.
-  "Gloss" is a short utf-8 human-readable string explaining what the
-  package provides for the bundle; "longloss" is a longer such
-  utf-8 string.
-
-  (Note that the gloss strings are meant, not to describe the package,
-  but to describe what the package provides for the bundle.  For
-  example, "The Anonymous Email Bundle needs the Python Runtime to run
-  Mixminion.")
-
-  Multiple gloss strings are allowed; each should have a different
-  language. The UI should display the must appropriate language to the
-  user.
-
-3.7. File formats: package files
-
-  (package
-    ({(name NAME)
-     (version VERSION)
-     (format FMT ((ATTR VAL)*)? )
-     (path PATH)
-     (ts TIME)
-     (digest HASH)
-     (shortdesc LANG TEXT)*
-     (longdesc LANG TEXT)*
-     (ATTR VAL)* })
-  )
-
-  Most elements are self-explanatory.  The "FMT" element describes the
-  file format of the package, which should give enough information
-  about how to install it.
-
-  No two package files in the same repository should have the same
-  name and version.  If a package needs to be changed, the version
-  MUST be incremented.
-
-  Descriptions are tagged with languages in the same way as glosses.
-
-4. Detailed Workflows
-
-4.1. The client application
-
-  Periodically, the client updater fetches a timestamp file from a
-  mirror.  If the timestamp in the file is up-to-date, the client
-  first checks to see whether the keys file listed is one that the
-  client has.  If not, the client fetches it, makes sure the hash of
-  the keys file matches the hash in the timestamp file, makes sure its
-  date is more recent than any keys file they have but not too far in
-  the future, and that it is signed by enough root keys that the
-  client recognizes.
-
-       [If the timestamp file is not up-to-date, the client tries a
-       few mirrors until it finds one with a good timestamp.]
-
-       [If the keys file from a mirror does not match the timestamp
-       file, the client tries a new mirror for both.]
-
-       [If the keys file is not signed by enough root keys, the client
-       warns the user and tries another mirror for both the timestamp
-       file and the keys file.]
-
-  Once the client has an up-to-date keys file, the client checks the
-  signature on the timestamp file.  Assuming it checks out, the client
-  refreshes the mirror list as needed, and refreshes any bundle files
-  to which the user is subscribed if the client does not have
-  the latest version of those files.  The client checks signatures on
-  these files, and fetches package metadata for any packages listed in
-  the bundle file that the client does not have, checks signatures on
-  these, and fetches binaries for packages that might need to be
-  installed or updated.  As the packages arrive, clients check their
-  hashes.
-
-  Once the client has gotten enough packages, it informs the user that
-  new packages have arrived, and asks them if they want to update.
-
-  Clients SHOULD cache at least the latest versions they have received
-  of all files.
-
-4.1.1. Download preferences
-
-  Users should be able to specify that packages must be only
-  downloaded over Tor, or must only be downloaded over encrypted
-  protocols, or both.  Users should also be able to force 
-
-4.2. Mirrors
-
-  Periodically, mirrors do an rsync or equivalent to fetch the latest
-  version of whatever parts of the repository have changed since the
-  version they currently hold.  Mirrors SHOULD replace older versions
-  of the repository idempotently, so that clients are less likely to
-  see inconsistent state.  Mirrors SHOULD validate the information
-  they receive, and not serve partial or inconsistent files.
-
-4.3. Workflow: Packagers
-
-  When a new binary package is done, the person making the package
-  runs a tool to generate and sign a package file, and sends both the
-  package and the package file to a repository admin.  Typically, the
-  base package file will be generated by inserting a version into a
-  template.
-
-  Packages MAY have as part of their build process a script to
-  generate the appropriately versioned package file.  This script
-  should at a minimum demand a build version, or use a timestamp in
-  place of a build version, to prevent two packages with the same
-  version from being created.
-
-4.4. Workflow: bundlers
-
-  When the packages in a bundle are done, the bundler runs a tool on
-  the package files to generate and sign a bundle file.  Typically,
-  this tool uses a template bundle file.
-
-4.5. Workflow: repository administrators
-
-  Repository administrators use a tool to validate signed files into the
-  repository.  The repository should not be altered manually.
-
-  This tool acts as follows:
-     - Package files may be added, but never replaced.
-     - Bundle files may be added, but never replaced.
-     - No file may be added unless it is syntactically valid and
-       signed by a key in the keys file authorized to sign files of
-       this type in this file's location location.
-
-     - A package file may not be added unless all of its binary
-       packages match their hashes.
-
-     - A bundle file may not be added unless all of its package files
-       are present and match their hashes.
-
-     - When adding a new keylist, bundle, or mirrors list, the
-       timestamp file must be regenerated immediately.
-
-5. Parameter setting and corner cases.
-
-5.1. Timing:
-
-  The timestamp file SHOULD be regenerated every 15 minutes.  Mirrors
-  SHOULD attempt to update every hour.  Clients SHOULD accept a
-  timestamp file up to 6 hours old.
-
-5.2. Format versioning and forward-compatibility:
-
-  All of the above formats include the ability to add more
-  attribute-value fields for backwards-compatible format changes.  If
-  we need to make a backwards incompatible format change, we create a
-  new filename for the new format.
-
-5.3. Key management and migration:
-
-  Root keys should be kept offline.  All keys except timestamp and
-  mirror keys should be stored encrypted.
-
-  All the formats above allow for multiple keys to sign a single
-  document.  To replace a compromised root key, it suffices to sign
-  keylist documents with both the compromised key and its replacement
-  until all clients have updated to a new version of the autoupdater.
-
-  To replace another key, it suffices to authorize the new key in the
-  keylist.  Note that a new package or bundle key must re-sign and
-  issue new versions of all packages or bundles it has generated.
-
-
-
-F. Future directions and open questions
-
-F.1. Package decomposition
-
-  It would be neat to decouple existing packages.  Right now, we'd
-  never want a windows user to have to fetch an openssl dll and Tor
-  separately.  But if they're using an auto-update tool, it'd be
-  pretty keen to have them not need to fetch a new openssl every time
-  Tor has a bugfix.
-
-F.2. Caching at Tor servers.
-
-  See Tor Proposal number 127.
-
-F.3. Support for more download methods
-
-  Ozymandns, chunked downloads, and bittorrent would all be neat
-  ideas.
-
-
-R. Ideas I'm rejecting for the moment
-
-R.1. Considering recommended versions from Tor consensus directory documents
-
-  This requires a working Tor to update Tor; that's not necessarily a
-  great idea.
-
-R.2. Integration with existing GPG signatures
-
-  The OpenPGP signature and key format is so complicated that you'd
-  have to be mad to touch it.
-
-
-
-
-

Copied: updater/trunk/specs/glider-spec.txt (from rev 16854, updater/trunk/specs/U2-formats.txt)
===================================================================
--- updater/trunk/specs/glider-spec.txt	                        (rev 0)
+++ updater/trunk/specs/glider-spec.txt	2008-09-11 19:58:23 UTC (rev 16857)
@@ -0,0 +1,627 @@
+
+0. Preliminaries
+
+0.0. Scope
+
+   This document describes a system for distributing Tor bundle updates.
+
+0.1. Proposed code name
+
+   Since "auto-update" is so generic, I've been thinking about going with
+   "glider", based on the sugar glider you get when you search for "handy
+   pocket creature".  I haven't yet done a search to find out whether
+   somebody else is using the name, so we shouldn't get too attached to it
+   before we see if it's taken.
+
+0.2. Goals
+
+   Once Tor was a single executable that you could just run.  Then it
+   required Privoxy.  Now, thanks to the Tor Browser Bundle and related
+   projects, a full installation can contain Tor, Privoxy, Torbutton,
+   Firefox, and more.
+
+   We need to keep this software updated.  When we make security fixes,
+   quick uptake helps narrow the window in which attackers can exploit
+   them.
+
+   We need updates to be easy.  Each additional step a user must take to
+   get updated means that more users will stay with older insecure
+   versions.
+
+   We need updates to be secure.  We're supposed to be good at crypto;
+   let's act like it.  There is no good reason in this day and age to
+   subject users to rollback attacks or unsigned packages or whatever.
+
+   We need administration to be simple.  Tor doesn't have a release
+   engineering team, so we can't add too many hard steps to putting out
+   a new release.
+
+   The system should be easy to implement; we may need to do multiple
+   implementations on the client side at least.
+
+0.2.1. Goals for package formats and PKIs
+
+   It should be possible to mirror a repository using only rsync and
+   cron.
+
+   Separate keys should be used for different people and different
+   roles.
+
+   Only a minimal set of keys should have to be kept online to keep
+   the system running.
+
+   The system should handle any single computer or system or person
+   being unavailable.
+
+   The formats and protocols should be pretty future-proof.
+
+0.3. Non-goals
+
+   This is not a general-purpose package manager like yum or apt: it
+   assumes that users will want to have one or more of a set of
+   "bundles", not an arbitrary selection of packages dependant on one
+   another.  (Rationale: these systems do what they do pretty well.)
+
+   This is also not a general-purpose package format.  It assumes the
+   existence of an external package format that can handle install,
+   update, remove, and version query.  (Rationale:
+
+1. System overview
+
+   The basic unit of updatability is a "bundle".  A bundle is a set of
+   software components, or "packages", plus some rules about installing
+   them.  Example bundles could be "Tor Browser, stable series" or
+   "Basic Tor, development series".
+
+   When Glider has responsibility for keeping a bundle up to date, we
+   say that a user has "subscribed" to that bundle.
+
+   Conceptually, there are four parts to keeping a bundle up to date:
+
+      Polling:
+        - Periodically, Glider asks a mirror whether there is a newer
+          version of some bundle that a user has subscribed to.  If so,
+          Glider determines what's in the bundle.
+
+      Fetching:
+        - If the bundle contains packages that Glider hasn't installed
+          or hasn't cached, it needs to download them from a mirror.
+          This can happen over any protocol; v1 should support at least
+          http and https-over-Tor.  V1 should also support resuming
+          partial downloads, since many users have unreliable
+          connections.
+
+          Later versions could support Bittorrent, or whatever.
+
+      Validation:
+        - Throughout the process, Glider must ensure that all the
+          bundles are signed correctly, all the packages are signed
+          correctly, and everything is up-to-date.
+
+          We want to specify this so that users can't be tricked about
+          the contents of a bundle, can't install a malicious package,
+          and can't be fooled into believing that an old bundle is
+          actually the latest.
+
+      Installation:
+        - Now Glider has a set of packages to install.  The format of
+          these packages will be platform-dependent: they could be pkg
+          files on OSX, MSI files on Win32, RPMs or DEBs on Linux, and
+          so on.  Glider should query the user for permission to start,
+          then install the packages.
+
+1.1. The repository
+
+   Each Glider instance knows about one or more "repositories".  A
+   repository is a filesystem somewhere that contains the packages in a
+   set of bundles, and some associated metadata.  A repository must
+   exist at one or more canonical hosts, and may have a number of full
+   or partial mirrors.
+
+   In v1, each Glider instance will know about only one repository.
+
+1.2. The PKI
+
+   The trust root for the whole system is, necessarily, whatever users
+   download when they first download a copy of Glider.  We need to make
+   sure that the first download happens from a site we trust, using
+   HTTPS.
+
+   Glider ships with root keys, which in turn are used to verify the
+   keys for all the other roles.  There are a few root keys, operated by
+   trusted admins for the system.  If root keys ever need to be changed,
+   we can just ship an update of Glider: it's supposed to be
+   self-updating anyway.
+
+   The root keys are only used to sign a 'key list' of all the other
+   keys and their roles.  A key list is valid if it has been signed by a
+   threshold of root keys.
+
+   Each package is signed with the key of its authorized builder.  For
+   example, one volunteer may be authorized to build the mac versions of
+   several packages, and another may be authorized to build the windows
+   version of just one.
+
+   Each bundle is signed with the key of its maintainer.  It's assumed
+   that the bundle maintainer might be the package maintainer for some
+   but not all of the packages.
+
+   The list of mirrors is also signed.  If the mirror list is
+   automatically updated, this key must be kept online; otherwise, it
+   can be offline.
+
+   To prevent an adversary from replaying an out-of-date signed
+   document, an automated process periodically signs a timestamped
+   statement containing the hashes of the mirror list, the latest
+   bundles, and the key list, using yet another special-purpose key.
+   This key must be kept online.
+
+1.3. Threat Model And Analysis
+
+   We assume an adversary who can operate compromised mirrors, and who
+   can possibly compromise the main repository.  At worst, such an
+   adversary can DOS users in a way that they can detect.
+
+   We're assuming for the moment an OSX/Win32-like execution model,
+   where all packages will run equal privilege, but occasionally
+   installation will require higher privilege.  This means that once a
+   hostile package is installed, it can basically do whatever it
+   wants.  As rootkit writers demonstrate, compromise is really
+   tenuous: any attacker who can induce a user to install a hostile
+   piece of code has, in effect, permanently compromised that user
+   until they reinstall.
+
+   Thus, if an adversary compromises enough keys to sign a compromised
+   package, or tricks a packager into signing a compromised package,
+   and manages to get that package into a signed bundle, the best we
+   can do is to limit the number of users who are affected.  We do
+   this by compartmentalizing signing keys so that only the package
+   and bundle in question are at risk.
+
+   (If we had replicated build processes and a bit-by-bit reliable
+   build process, we could have multiple packagers test that a binary
+   was built properly, and multiply sign it.  This would be effective
+   against an adversary compromising a single packaging key, but not
+   against one compromising a source repository.)
+
+2. The repository layout
+
+   The filesystem layout in the repository is used for two purposes:
+     - To give mirrors an easy way to mirror only some of the repository.
+     - To specify which parts of the repository a given key has the
+       authority to sign.
+
+   The following files exist in all repositories and mirrors:
+
+    /meta/keys.txt
+
+         Signed by the root keys; indicates keys and roles.
+         [???? I'm using the txt extension here.  Is that smart?]
+
+    /meta/mirrors.txt
+
+         Signed by the mirror key; indicates which parts of the
+         repository are mirrored at what mirrors.
+
+    /meta/timestamp.txt
+
+         Signed by the timestamp key; indicates hashes and timestamps
+         for the latest versions of keys.txt and mirrors.txt.  Also
+         indicates the latest version of each bundle for each os/arch.
+
+         This is the only file that needs to be downloaded for polling.
+
+    /bundleinfo/bundlename/os-arch/bundlename-os-arch-bundleversion.txt
+
+         Signed by the appropriate bundle key.  Describes what
+         packages make up a bundle, and what order to install,
+         uninstall, and upgrade them in.
+
+    /pkginfo/packagename/os-arch/version/packagename-os-arch-packageversion.txt
+
+         Signed by the appropriate package key.  Tells the name of the
+         file that makes up a package, its hash, and what procedure
+         is used to install it.
+
+    /packages/packagename/os-arch/version/(some filename)
+
+         The actual package file.  Its naming convention will depend
+         on the underlying packaging system.
+
+3. Document formats
+
+3.1. Metaformat
+
+   All documents use Rivest's SEXP meta-format as documented at
+     http://people.csail.mit.edu/rivest/sexp.html
+   with the restriction that no "display hint" fields are to be used,
+   and the base64 transit encoding isn't used either.
+
+   (We use SEXP because it's really easy to parse, really portable,
+   and unlike most other tagged data formats, has a
+   trivially-specified canonical format suitable for hashing.)
+
+   In descriptions of syntax below, we use regex-style qualifiers, so
+   that in
+        (sofa slipcover? occupant* leg+)
+   the sofa will have an optional slipcover, zero or more occupants,
+   and one or more legs.  This pattern matches (sofa leg) and (sofa
+   slipcover occupant occupant leg leg leg leg) but not (sofa leg
+   slipcover).
+
+   We also use a braces notation to indicate elements that can occur
+   in any order.  For example,
+        (bread {flour+ eggs? yeast})
+   matches a list starting with "bread", and then containing one or
+   more  of flours, zero or one occurrences of eggs, and one
+   occurrence of yeast, in any order.  This pattern matches (bread eggs
+   yeast flour) but not (bread yeast) or (bread flour eggs yeast
+   macadamias).
+
+3.2. File formats: general principles
+
+   We use tagged lists (lists whose first element is a string) to
+   indicate typed objects.  Tags are generally lower-case, with
+   hyphens used for separation.  Think Lispy.
+
+   We use attrlists [lists of (key value) lists] to indicate a
+   multimap from keys to values.  Clients MUST accept unrecognized
+   keys in these attrlists.  The syntax for an attrlist with two
+   recognized and required keys is typically given as ({(key1 val1)
+   (key2 val2) (ATTR VAL)*}), indicating that the keys can occur in
+   any order, intermixed with other attributes.
+
+   Timestamp files will be downloaded very frequently; all other files
+   will be much smaller in size than package files.  Thus,
+   size-optimization for timestamp files makes sense and most other
+   other space optimizations don't.
+
+   Versions are represented as lists of the form (v I1 I2 I3 I4 ...)
+   where each item is a number or alphanumeric version component.  For
+   example, the version "0.2.1.5-alpha" is represented as (v 0 2 1 5
+   alpha).
+
+   All signed files are of the format:
+
+       (signed
+          X
+          (signature ({(keyid K) (method M) (ATTR VAL)*}) SIG)+
+       )
+
+   where: X is a list whose first element describes the signed object.
+          K is the identifier of a key signing the document
+          M is the method to be used to make the signature
+          (ATTR VAL) is an arbitrary list whose first element is a
+             string.
+          SIG is a signature of the canonical encoding of X using the
+          identified key.
+
+   We define two signing methods at present:
+       sha256-oaep : A signature of the SHA256 hash of the canonical
+         encoding of X, using OAEP+ padding. [XXXX say more about mgf]
+
+   All times are given as strings of the format "YYYY-MM-DD HH:MM:SS",
+   in UTC.
+
+   All keys are of the format:
+      (pubkey ({(type TYPE) (ATTR VAL)*}) KEYVAL)
+   where TYPE is a string describing the type of the key and how it's
+   used to sign documents.  The type determines the interpretation of
+   KEYVAL.
+
+   The ID of a key is the type field concatenated with the SHA-256
+   hash of the canonical encoding of the KEYVAL field.
+
+   We define one keytype at present: 'rsa'.  The KEYVAL in this case
+   is a 2-element list of (e n), with both values given in big-endian
+   binary format.  [This makes keys 45-60% more compact than using
+   decimal integers.]
+
+   All RSA keys must be at least 2048 bits long.
+
+
+   Every role in the system is associated with a key.  Replacing
+   anything but a root key is supposed to be relatively easy.
+
+   Root-keys sign other keys, and certify them as belonging to roles.
+   Clients are configured to know the root keys.
+
+   Bundle keys certify the contents of a bundle.
+
+   Package keys certify packages for a given program or set of
+   programs.
+
+   Mirror keys certify a list of mirrors.  We expect this to be an
+   automated process.
+
+   Timestamp keys certify that given versions of other metadata
+   documents are up-to-date.  They are the only keys that absolutely
+   need to be kept online.  (If they are not, timestamps won't be
+   generated.)
+
+3.3. File formats: key list
+
+   The key list file is signed by multiple root keys.  It indicates
+   which keys are authorized to sign which parts of the repository.
+
+   (keylist
+     (ts TIME)
+     (keys
+       ((key ({(roles (ROLE PATH)+) (ATTR VAL)*}) KEY)*)
+     ...
+   )
+
+   The "ts" line describes when the keys file was updated.  Clients
+   MUST NOT replace a file with an older one, and SHOULD NOT accept a
+   file too far in the future.
+
+   A ROLE is one of "timestamp" "mirrors" "bundle" or "package".
+
+   PATH is a path relative to the top of the directory hierarchy.  It
+   may contain "*" elements to indicate "any file", and may end with a
+   "/**" element to indicate all files under a given point.
+
+3.4. File formats: mirror list
+
+   The mirror list is signed by a mirror key.  It indicates which
+   mirrors are active and believed to be mirroring which parts of the
+   repository.
+
+   (mirrorlist
+     (ts TIME)
+     (mirrors
+       ( (mirror ({(name N) (urlbase U) (contents PATH+) (weight W)
+                   (official)?  (ATTR VAL)})) * )
+     ...
+  )
+
+  Every mirror is a copy of some or all of the directory hierarchy
+  containing at least the /meta, /bundles/, and /pkginfo directories.
+
+  N is a descriptive name for the mirror; U is the URL of the mirror's
+  base (i.e., the parent of the "meta" directory); and the PATH
+  elements are the components describing how much of the packages
+  directory is mirrored.  Their format is as in the keylist file.
+
+  W is an integer used to weight mirrors when picking at random;
+  mirrors with more bandwidth should have higher weigths.   The
+  "official" element should only be present if the mirror is (one of
+  the) official repositories operated by the Tor Project.
+
+3.5. File formats: timestamp files
+
+  The timestamp file is signed by a timestamp key.  It indicates the
+  latest versions of other files, and contains a regularly updated
+  timestamp to prevent rollback attacks.
+
+  (ts
+    ({(at TIME)
+      (m TIME MIRRORLISTHASH)
+      (k TIME KEYLISTHASH)
+      (b NAME VERSION TIME PATH HASH)*})
+  )
+
+  TIME is when the timestamp was signed.  MIRRORLISTHASH is the digest
+  of the mirror-list file; KEYLISTHASH is the digest of the key list
+  file; and the 'b' entries are a list of the latest version of each
+  bundles and their locations and hashes.
+
+3.6. File formats: bundle files
+
+  (bundle
+    (at TIME)
+    (os OS)
+    [(arch ARCH)]
+    (version V)
+    (packages
+      (NAME VERSION PATH HASH ({(order INST UPDATE REMOVE)
+                                (optional)?
+                                (gloss LANG TEXT)*
+                                (longloss LANG TEXT)*
+                                 (ATTR VAL)*})? )* )
+  )
+
+  Most elements are self-explanatory; the INST, UPDATE, and REMOVE
+  elements of the order element are numbers defining the order in
+  which the packages are installed, updated, and removed respectively.
+  The "optional" element is present if the package is optional.
+  "Gloss" is a short utf-8 human-readable string explaining what the
+  package provides for the bundle; "longloss" is a longer such
+  utf-8 string.
+
+  (Note that the gloss strings are meant, not to describe the package,
+  but to describe what the package provides for the bundle.  For
+  example, "The Anonymous Email Bundle needs the Python Runtime to run
+  Mixminion.")
+
+  Multiple gloss strings are allowed; each should have a different
+  language. The UI should display the must appropriate language to the
+  user.
+
+3.7. File formats: package files
+
+  (package
+    ({(name NAME)
+     (version VERSION)
+     (format FMT ((ATTR VAL)*)? )
+     (path PATH)
+     (ts TIME)
+     (digest HASH)
+     (shortdesc LANG TEXT)*
+     (longdesc LANG TEXT)*
+     (ATTR VAL)* })
+  )
+
+  Most elements are self-explanatory.  The "FMT" element describes the
+  file format of the package, which should give enough information
+  about how to install it.
+
+  No two package files in the same repository should have the same
+  name and version.  If a package needs to be changed, the version
+  MUST be incremented.
+
+  Descriptions are tagged with languages in the same way as glosses.
+
+4. Detailed Workflows
+
+4.1. The client application
+
+  Periodically, the client updater fetches a timestamp file from a
+  mirror.  If the timestamp in the file is up-to-date, the client
+  first checks to see whether the keys file listed is one that the
+  client has.  If not, the client fetches it, makes sure the hash of
+  the keys file matches the hash in the timestamp file, makes sure its
+  date is more recent than any keys file they have but not too far in
+  the future, and that it is signed by enough root keys that the
+  client recognizes.
+
+       [If the timestamp file is not up-to-date, the client tries a
+       few mirrors until it finds one with a good timestamp.]
+
+       [If the keys file from a mirror does not match the timestamp
+       file, the client tries a new mirror for both.]
+
+       [If the keys file is not signed by enough root keys, the client
+       warns the user and tries another mirror for both the timestamp
+       file and the keys file.]
+
+  Once the client has an up-to-date keys file, the client checks the
+  signature on the timestamp file.  Assuming it checks out, the client
+  refreshes the mirror list as needed, and refreshes any bundle files
+  to which the user is subscribed if the client does not have
+  the latest version of those files.  The client checks signatures on
+  these files, and fetches package metadata for any packages listed in
+  the bundle file that the client does not have, checks signatures on
+  these, and fetches binaries for packages that might need to be
+  installed or updated.  As the packages arrive, clients check their
+  hashes.
+
+  Once the client has gotten enough packages, it informs the user that
+  new packages have arrived, and asks them if they want to update.
+
+  Clients SHOULD cache at least the latest versions they have received
+  of all files.
+
+4.1.1. Download preferences
+
+  Users should be able to specify that packages must be only
+  downloaded over Tor, or must only be downloaded over encrypted
+  protocols, or both.  Users should also be able to force 
+
+4.2. Mirrors
+
+  Periodically, mirrors do an rsync or equivalent to fetch the latest
+  version of whatever parts of the repository have changed since the
+  version they currently hold.  Mirrors SHOULD replace older versions
+  of the repository idempotently, so that clients are less likely to
+  see inconsistent state.  Mirrors SHOULD validate the information
+  they receive, and not serve partial or inconsistent files.
+
+4.3. Workflow: Packagers
+
+  When a new binary package is done, the person making the package
+  runs a tool to generate and sign a package file, and sends both the
+  package and the package file to a repository admin.  Typically, the
+  base package file will be generated by inserting a version into a
+  template.
+
+  Packages MAY have as part of their build process a script to
+  generate the appropriately versioned package file.  This script
+  should at a minimum demand a build version, or use a timestamp in
+  place of a build version, to prevent two packages with the same
+  version from being created.
+
+4.4. Workflow: bundlers
+
+  When the packages in a bundle are done, the bundler runs a tool on
+  the package files to generate and sign a bundle file.  Typically,
+  this tool uses a template bundle file.
+
+4.5. Workflow: repository administrators
+
+  Repository administrators use a tool to validate signed files into the
+  repository.  The repository should not be altered manually.
+
+  This tool acts as follows:
+     - Package files may be added, but never replaced.
+     - Bundle files may be added, but never replaced.
+     - No file may be added unless it is syntactically valid and
+       signed by a key in the keys file authorized to sign files of
+       this type in this file's location location.
+
+     - A package file may not be added unless all of its binary
+       packages match their hashes.
+
+     - A bundle file may not be added unless all of its package files
+       are present and match their hashes.
+
+     - When adding a new keylist, bundle, or mirrors list, the
+       timestamp file must be regenerated immediately.
+
+5. Parameter setting and corner cases.
+
+5.1. Timing:
+
+  The timestamp file SHOULD be regenerated every 15 minutes.  Mirrors
+  SHOULD attempt to update every hour.  Clients SHOULD accept a
+  timestamp file up to 6 hours old.
+
+5.2. Format versioning and forward-compatibility:
+
+  All of the above formats include the ability to add more
+  attribute-value fields for backwards-compatible format changes.  If
+  we need to make a backwards incompatible format change, we create a
+  new filename for the new format.
+
+5.3. Key management and migration:
+
+  Root keys should be kept offline.  All keys except timestamp and
+  mirror keys should be stored encrypted.
+
+  All the formats above allow for multiple keys to sign a single
+  document.  To replace a compromised root key, it suffices to sign
+  keylist documents with both the compromised key and its replacement
+  until all clients have updated to a new version of the autoupdater.
+
+  To replace another key, it suffices to authorize the new key in the
+  keylist.  Note that a new package or bundle key must re-sign and
+  issue new versions of all packages or bundles it has generated.
+
+
+
+F. Future directions and open questions
+
+F.1. Package decomposition
+
+  It would be neat to decouple existing packages.  Right now, we'd
+  never want a windows user to have to fetch an openssl dll and Tor
+  separately.  But if they're using an auto-update tool, it'd be
+  pretty keen to have them not need to fetch a new openssl every time
+  Tor has a bugfix.
+
+F.2. Caching at Tor servers.
+
+  See Tor Proposal number 127.
+
+F.3. Support for more download methods
+
+  Ozymandns, chunked downloads, and bittorrent would all be neat
+  ideas.
+
+
+R. Ideas I'm rejecting for the moment
+
+R.1. Considering recommended versions from Tor consensus directory documents
+
+  This requires a working Tor to update Tor; that's not necessarily a
+  great idea.
+
+R.2. Integration with existing GPG signatures
+
+  The OpenPGP signature and key format is so complicated that you'd
+  have to be mad to touch it.
+
+
+
+
+



More information about the tor-commits mailing list