- tor-dev - lists.torproject.org

ooniprobe 1.1.0 is out!
by Arturo Filastò 19 Aug '14

19 Aug '14

Hello beautiful humans! We are very pleased to announce that a new version of ooniprobe is now out. In this new release of ooniprobe we have added a new command line tool for listing the reports that have not been published to a collector and that allows the probe operator to choose which ones they would like to upload. We have also made some privacy improvements to the reports (we will sanitize all things that may look like file paths) and added metadata associated with the maxmind database being used by the probe operator. Here is a more detailed list of what has been done: * Annotate on disk which reports we have submitted and which ones we have not: https://trac.torproject.org/projects/tor/ticket/11860 * Add tool called oonireport for publishing unpublished ooniprobe reports to a collector: https://trac.torproject.org/projects/tor/ticket/11862 * Probe Report does not leak filepaths anymore: https://trac.torproject.org/projects/tor/ticket/12706 * Reports now include version information about the maxmind database being used: https://trac.torproject.org/projects/tor/ticket/12771 * We now do integrity checks on the ooniprobe.conf file so that we don't start the tool if the config file is missing some settings or is not consistent: https://trac.torproject.org/projects/tor/ticket/11983 (thanks to Alejandro López (kudrom)) * Improvements have been made to the sniffer subsystem (thanks to Alejandro López (kudrom)) * Fix the multi protocol traceroute test. https://trac.torproject.org/projects/tor/ticket/12883 Minor bug fixes: * Fix dns_spoof test (by kudrom) https://trac.torproject.org/projects/tor/ticket/12486 * ooni might not look at requiresTor: https://trac.torproject.org/projects/tor/ticket/11858 * ooni spits out gobs of tracebacks if Tor is not running and the OONI config says it will be: https://trac.torproject.org/projects/tor/ticket/11859 * The README for ooni-probe should mention the bugtracker and repository https://trac.torproject.org/projects/tor/ticket/11980 For the moment you can download the new version from pypi here: https://pypi.python.org/pypi/ooniprobe. Soon a there will also be a new debian package for it (thank to Lunar^). As always feedback is greatly appreciated so please submit any bugs you may find to our bug tracker: https://trac.torproject.org/projects/tor/newticket?component=Ooni Have fun! ~ Art.

1 0

Re: [tor-dev] Proposal: All Relays are Directory Servers
by Tim 19 Aug '14

19 Aug '14

On 19 Aug 2014, at 22:00 , tor-dev-request(a)lists.torproject.org wrote: > Date: Tue, 19 Aug 2014 01:32:25 +0000 > From: Matthew Finkel <matthew.finkel(a)gmail.com> > To: tor-dev(a)lists.torproject.org > Subject: Re: [tor-dev] Proposal: All Relays are Directory Servers > Message-ID: <20140819013224.GB3689@localhost> > Content-Type: text/plain; charset=utf-8 > > On Mon, Aug 18, 2014 at 02:17:28PM -0400, Nick Mathewson wrote: >> On Wed, Aug 13, 2014 at 11:53 AM, Matthew Finkel >> <matthew.finkel(a)gmail.com> wrote: > >>> Filename: xxx-directory-servers-for-all.txt >>> Title: All relays are directory servers >>> Author: Matthew Finkel >>> Created: 29-Jul-2014 >>> Status: >>> Target: 0.2.6.x >>> >>> Overview: >>> >>> This proposal aims at removing part of the distinction between the >>> relay and the directory server. Currently operators have the options >>> of being one of: a relay, a directory server, or both. With the >>> acceptance of this proposal the options will be simplified to being >>> either only a directory server or a combined relay and directory >>> server. All relays will serve directory requests. >> >> FWIW, we don't support being only a directory server right now, do we? >> > > Nope, you're right. Bad assumption. > > https://gitweb.torproject.org/tor.git/blob/390728d85644a6c062d6807ffc096343… Looking at the details of parse_ports(), the warning/comment isn't entirely accurate: it seems that any configured "port" is enough to disable the warning (even a DirPort, ControlPort/ControlSocket(!), or ExtORPort). Which is why I'm not seeing it on the directory-only tor instance I have configured. Then again, the core message is accurate - a tor instance that doesn't act as a tor client or relay isn't going to do much. Tim

1 0

Re: [tor-dev] Proposal: All Relays are Directory Servers
by Tim 19 Aug '14

19 Aug '14

> On 19 Aug 2014, at 11:30, Nick Mathewson <nickm(a)freehaven.net> wrote: > > Date: Mon, 18 Aug 2014 14:17:28 -0400 > From: Nick Mathewson <nickm(a)freehaven.net> > To: tor-dev(a)lists.torproject.org > Subject: Re: [tor-dev] Proposal: All Relays are Directory Servers > > >> On Wed, Aug 13, 2014 at 11:53 AM, Matthew Finkel >> <matthew.finkel(a)gmail.com> wrote: >> Hi All, >> >> Below is the proposal for #12538 [0], with some changes after George's >> review and some other revisions. >> >> Feedback welcome! >> >> Thanks, >> Matt >> >> >> [0] https://trac.torproject.org/projects/tor/ticket/12538 > > Thanks! This is now proposal 237. Any revisions should be sent in as > patches against the one in the torspec repository. > >> Filename: xxx-directory-servers-for-all.txt >> Title: All relays are directory servers >> Author: Matthew Finkel >> Created: 29-Jul-2014 >> Status: >> Target: 0.2.6.x >> >> Overview: >> >> This proposal aims at removing part of the distinction between the >> relay and the directory server. Currently operators have the options >> of being one of: a relay, a directory server, or both. With the >> acceptance of this proposal the options will be simplified to being >> either only a directory server or a combined relay and directory >> server. All relays will serve directory requests. > > FWIW, we don't support being only a directory server right now, do we? I've recently configured a tor instance [for HTTP fuzzing] that does nothing but cache the entire directory - all that is required is setting the DirPort non-zero, and the ORPort to 0. No warnings are emitted, and the instance stays stable, up to date with the consensus and descriptors, and answers [local] directory requests [as configured]. Because it's a local/test instance, I have specifically disabled external access. I have also disabled the default relay/directory server behaviour of submitting its own descriptor to the authorities. So I don't actually know if it would submit a descriptor if I enabled that option. However, the man page says that either one of a directory or a router port is sufficient for a server to submit its descriptor to the consensus. I also don't know if clients would connect to a directory server that didn't have an ORPort. Do clients select directory guards based on flags that it's impossible for a directory-only Tor instance to obtain? What about HSDirs? Nick, which of these things did you mean by "support"? Does the consensus include directory-only Tor servers? Do tor clients connect to directory-only Tor servers? Will a directory-only Tor server ever get the flags needed to become a "directory guard"? Will a directory-only Tor server ever get the flags needed to become a HSDir? (I think we agree that a directory-only server is a far less useful configuration of an instance than a router would be.) Tim

1 0

Proposal: All Relays are Directory Servers
by Matthew Finkel 19 Aug '14

19 Aug '14

Hi All, Below is the proposal for #12538 [0], with some changes after George's review and some other revisions. Feedback welcome! Thanks, Matt [0] https://trac.torproject.org/projects/tor/ticket/12538 ----------------------------------------------------------------------- Filename: xxx-directory-servers-for-all.txt Title: All relays are directory servers Author: Matthew Finkel Created: 29-Jul-2014 Status: Target: 0.2.6.x Overview: This proposal aims at removing part of the distinction between the relay and the directory server. Currently operators have the options of being one of: a relay, a directory server, or both. With the acceptance of this proposal the options will be simplified to being either only a directory server or a combined relay and directory server. All relays will serve directory requests. Motivation: Fetching directory documents and descriptors is sometimes a non-trivial operation for clients. If they do not have a consensus then they must contact a directory authority (until directory sources are added or clients are able to use a fallback consensus). If they have a consensus and have at least one entry guard then the client can query that guard for documents. If the document isn't available then after a period of time the client will attempt to retry downloading it. If the entry guard isn't a directory server, as well, a directory server and/or directory guard must be chosen (based on the server having an open DirPort) and queried for the document. At a minimum, this has a potential performance impact, at worst it's another attack vector that allows for profiling clients and partitioning users. With the orthogonally proposed move to clients using a single guard, the potential performance bottleneck and ability to profile users could be exacerbated. If the client selects an entry guard and it is not a directory server then the client may select a distinct directory guard which will leak client behavior to a second node. In the case where the client does not use guards, it is important to have the largest possible amount of diversity in the set of directory servers. In a network where every relay is a directory server, the profiling and partitioning attack vector is reduced to the guard (for clients who use them), which is already in a privileged position for this. In addition, with the increased set size relay descriptors and documents are more readily available and it diversifies the providers. Design: The changes needed to achieve this should be simple. Currently all relays download and cache the majority of relay documents in any case, so the slight increased memory usage from downloading all of them should have minimal consequences. There will be necessary logical changes in the client, router, and directory code. Currently directory servers are defined as such if they advertise having an open directory port. We can no longer assume this is true. To this end, we will introduce a new server descriptor line. "tunnelled-dir-server" NL The presence of this line indicates that the relay accepts tunnelled directory requests. For a relay that implements this proposal, this line MUST be added to its descriptor if it does not advertise a directory port, and MAY be added if it also advertises an open directory port. In addition to this, relays will now download and cache all descriptors and documents listed in the consensus, regardless of whether they are deemed useful or usable, exactly like the current directory servers. All relays will also accept directory requests when they are tunnelled over a connection established with a BEGIN_DIR cell, the same way these connections are already accepted by bridges and directory servers with an open DirPort. Directory Authorities will now assign the V2Dir flag to a server if it supports a version of the directory protocol which is useful to clients and it has at least an open directory port or it has an open and reachable OR port and advertises "tunnelled-dir-server" in its server descriptor. Clients choose a directory by using the current criteria with the additional criterion that a server only needs the V2Dir status flag instead of requiring an open DirPort. When the client chooses which directory server it will query, it checks if the server has an open directory port and uses begindir if it does not have one. Directory servers should not be able to determine which version of Tor the client is using (or a lower-bound on the version), if possible. Continuing to prefer direct directory connections over begin may help mitigate a potential partitioning attack. Security Considerations and Implications: Currently all directory servers are explicitly configured. This is necessary because they must have a configured and reachable external port. However, this is a restriction and results in a reduced number of directory servers on the network. As a result, this could allow an adversary to control a significant fraction of the servers. By increasing the number of directory servers on the network the likelihood of selecting one that is malicious is reduced. Also, with this proposal, it will be more likely that a client's entry guard is also a directory server (as alluded to in Proposal 207). However, the reduced anonymity set caused when the guard does not have, or is unwilling to distribute, a specific document still exists. With the increased diversity in the available servers, the impact of this should be reduced. Another question that may need further consideration is whether we trust bad directories to be good guards and exits. Specification: The version 3 directory protocol specification does not currently document the use of directory guards. This spec should be updated to mention the preferred use of directory guards during directory requests. In addition, the new criteria for assigning the V2Dir flag should be documented. Impact on local resources: Should relays attempt to download documents from another mirror before asking an authority? All relays will now prefer contacting the authorities first, but this will not scale well and will partition users from relays. If all relays become directory servers, they will choose to download all documents, regardless of whether they are useful, in case another client does want them. This will have very little impact on the "typical" relay, however on memory constrained relays (BeagleBone, Raspberry Pi, and similar), every megabyte allocated to directory documents is not available for new circuits. Should we add a config option that allows operators to disable being a directory server? Is it more worthwhile for them to serve these documents or to relay cells? Future Considerations: Should the DirPort be deprecated at some point in the future?

2 2

Final Fog Combiner GSoC Report and Fog Status/Description
by quinn jarrell 19 Aug '14

19 Aug '14

Hi all, This report is going to be a bit longer and more general than usual as it'll cover a lot more about the current state of Fog and a description of it. This is my last report for GSoC, but I will probably continue submitting patches to Fog. Here's a general description of what I worked on this summer. Fog provides an easy way to chain pluggable transports together. It allows users to obfuscate their Tor data in multiple layers by using the output of one pluggable transport as the input for another. This will hopefully make pluggable transports much more difficult to block and provide more anonymity for users. Fog has two components, a client written in Python and a server written in Go. It appears to Tor as a normal pluggable transport and acts like one. But instead of obfuscating the data itself like other pluggable transports, it proxies the data through other pluggable transports. It then receives the obfuscated data and passes it onto the next pluggable transport in the chain and so on. After the last pluggable transport obfuscates the data and sends it back to Fog, Fog sends it onto the bridge address where the Fog server is waiting. The Fog server works similarly, but reverses the order of the pluggable transports, allowing each pluggable transport to peel off a layer of obfuscation. When it completely deobfuscates the data after traveling through the chain in reverse, Fog sends the data onto Tor as if it is a normal pluggable transport. The nice part about Fog is that it requires very little if any modification of existing pluggable transports to create chains. As long as a pluggable transport talks obeys the PT-Spec[0] and sends its results to a prespecified server, it can be used. So for instance all the chains: b64|obfs3|scramblesuit, scramblesuit|fte|bananaphone|obfs3 will work without any modification of those pluggable transports. Now the exceptions to this are pluggable transports that send obfuscated data to different servers on every startup. This mainly includes the flash proxy and Meek pluggable transports. These can only be used currently at the end of a chain. The reason being that since they send to different servers every time, it is not possible to direct them to a SOCKS shim as they are configured to only talk a specific protocol instead of just TCP. They rely on receiving some sort of confirmation or direction instead of acting as dumb obfuscators. More specifically, this summer I've been working on generalizing a prototype that David Fifield(dcf1) and Ximin Luo(infinity0) built chaining together the pluggable transports obfs3 and flashproxy. The main features I've added to Fog include Generalizing Fog to use any pluggable transports instead of just obfs3 and flashproxy Generalizing Fog to use any number of pluggable transports to obfuscate data I modified how Tor connects to Fog, it now uses a SOCKS server to recieve the request from Tor instead of receiving TCP from the first pluggable transport as it did before. Added configuration systems to both Fog client and Fog Server to parse a format similar to the torrc config files. Added the ability to support multiple chains with one instance of Fog. The Fog client will only launch one copy of each pluggable transport even if that pluggable transport is used in multiple chains. Over the next few weeks outside of GSoC I plan on adding A system of passing ClientTransportOptions and ServerTransportOptions to the chain of pluggable transports. (I worked on this the last two weeks but was not able to submit it for code review before GSoC ended) Tests for everything! (Some are written but they are currently in code review, others will follow) Documentation. This email is just a general description of Fog, I'll be adding a bunch more documentation about the structure of fog and a guide to use it. Attempting to package Fog as a binary similar to Obfsproxy, though I've heard bad things about Py2Exe. In the future, Fog may need to Modify the PT-Spec to force all pluggable transport servers to transmit metadata about what IP/port the connection came from. This will let the Fog server use one copy of pluggable transport for multiple chains and fix an issue with high traffic causing data to be sent back in the wrong order. And thats about it for me. I'm headed back to UIUC next week but as when I have free time, I probably will continue contributed patches to Fog and maybe other parts of the Tor Project. I've had a great summer working for the Tor Project. Its been an awesome time developing Fog and I've learned a massive amount. I can't thank Infinity0 enough for being my main mentor and teaching me a ton about what's needed to do open source. He's really guided me to be a better coder and I'm thankful that he was able to put up with my at times messy/lazy code. I thought I knew something about how to write clean code at the beginning of summer. Now I know that I knew nothing lol. Seriously, Infinity0, thanks so much! I would also like to thank dcf for being my backup mentor and helping with questions about how the Fog server worked/potential enhancements to it. You can see the latest version of Fog at [1] Quinn [0] https://gitweb.torproject.org/torspec.git/blob/HEAD:/pt-spec.txt [1] https://github.com/infinity0/fog

1 0

[GSoC] HTTPS Everywhere's Secure Ruleset Updater - Final Report
by Red 19 Aug '14

19 Aug '14

Hello everyone! Today is the official "pencils down" date for Google Summer of Code, so I am going to outline the results of my work this Summer as a way of concluding my project. The first thing I had done at the start of my project was learn about Mozilla's XPCOM components and how they are used to interface with Gecko and other Firefox internals from extension code. I also learned about special files included with any extension such as install.rdf which are used to describe an extension's version information and signature-verifying public key, among other things. This lead me to understanding how extensions are updated and discover the kinds of limitations I would find in trying to update HTTPS Everywhere's rulesets library without changing the rest of the extension. The next thing I did was draft the specification for the format of the JSON manifest used to describe ruleset updates and explain how the process of updating the rulesets library would work. This work has since been built on and improved, and the existing specification document can be found in my fork of the extension, and is linked below[0]. This document describes everything one needs to know about the ruleset updater. It details: * How an object signing certificate can be generated using certutil from the NSS toolset. * How to create an update.json manifest file using the ruleset_update_manifest.py utility script I created.[1] * How to sign the update.json file using pk1sign and an object signing certificate. * How update information is fetched and how download failures are handled. * How the contents of update.json and the rulesets library database file are verified to be authentic. * What the significance of each field in update.json is. * Approximately how the ruleset updating mechanism functions in pseudocode. After the details of how updating would work were outlined in updateJSONSpec.md, I began implementing the ruleset updater module code for the extension. I also started working on a way to write tests for HTTPS Everywhere that I would initially use to test my code. I discovered that Firefox extensions could import XPCOM components exported by other extensions added to a user's browser, and so began work on a separate extension[2] that could be installed and used solely for testing HTTPS Everywhere. This work was soon improved upon by my mentor, Yan, who had the idea of creating a skeleton of an extension inside of HTTPS Everywhere's own codebase[3]. This extension used the more modern Addon SDK, which includes a unit testing suite, enabling HTTPS Everywhere developers to start writing proper unit tests that could be included in the HTTPS-Everywhere repository and run from the command line. While the unit testing environment I now had available was very effective, I had run into some surprising difficulty with Mozilla's nsIDataSignatureVerified XPCOM component, used to verify the signatures we were generating. By this point, Yan and I had agreed to try to compute the signature over the SHA256 hash of update.json's contents, so that this data could be more easily transcribed from a regular work computer to the airgapped machine EFF uses to house an offline signing key. It was at this point that we found that the key and signatures OpenSSL were generating were, somehow, incompatible with nsIDataSignatureVerifier, and Yan managed to get the verification unit test to pass by using certutil and pk1sign. The same problem of verifying signatures was encountered again when I got around to testing the actual ruleset updater code again, and had to generate new data, despite using certutil and pk1sign. Quite unfortunately, this problem drained almost four whole weeks of my time, as I had been looking into solutions involving porting a Firefox extension signing tool called Uhura (written in Perl), which meant also trying to understand what the tool was doing with ASN.1 specifications, among other things. Finally, I found that the nsIDataSignatureVerifier tool successfully verified the signature I had created for testing when it was not the hash of update.json's contents that were signed and supplied to the verifier, but the contents of the file themselves. The component is poorly documented and, after reaching out to developers in Mozilla's ecosystem, I found that it isn't very well understood in general. As such, even now I do not really understand why the verifier did not behave as expected in this scenario. Regardless, while it is somewhat less convenient here to sign update.json, the signature verification process works. Finally, I was able to move on to solve the problems of downloading and verifying the authenticity of the rulesets database file, and of reinitializing the extension's rulesets with those of the new database file. These problems were technically nuanced, but have been solved. The ruleset updater has been integrated into my fork of the extension's repository on github. My tests of its performance have passed, and I am pleased to say that it appears to be working. I have since submitted a pull request[4] to have my work reviewed and merged, and have even made changes requested by Yan, who has reviewed my work already. I'd like to conclude by giving my thanks to everyone in both of these mailing lists. I have received some invaluable support and advice that has had a profoundly positive impact on my experience. I learned a lot from everyone that lent a hand, and I feel the project has benefited greatly from each of the contributions I received to my efforts. My mentor, Yan, has also been a pleasure to work with. Our weekly meetings added very significant structure to my workflow, helped to keep me organized, and were also a great opportunity for me to pique her brain for ideas and possible solutions to some of the tougher problems I've encountered. I sincerely regard this as having been a positive experience, and I hope that everyone who has helped me until now is satisfied with my efforts and the actual results of my work. Thanks, everyone! [0] - https://github.com/redwire/https-everywhere/blob/master/doc/updateJSONSpec.… [1] - https://github.com/redwire/https-everywhere/blob/master/utils/ruleset_updat… [2] - https://github.com/redwire/https-everywhere-testing [3] - https://github.com/EFForg/https-everywhere/tree/master/https-everywhere-tes… [4] - https://github.com/EFForg/https-everywhere/pull/437

1 0

[PATCH 1/4] Remove some unused variables
by Daniel Martí 18 Aug '14

18 Aug '14

Signed-off-by: Daniel Martí <mvdan(a)mvdan.cc> --- lib/chutney/TorNet.py | 5 ++--- lib/chutney/Traffic.py | 1 - 2 files changed, 2 insertions(+), 4 deletions(-) diff --git a/lib/chutney/TorNet.py b/lib/chutney/TorNet.py index 3685ec0..9d8efd1 100644 --- a/lib/chutney/TorNet.py +++ b/lib/chutney/TorNet.py @@ -297,7 +297,6 @@ class LocalNodeBuilder(NodeBuilder): """ datadir = self._env['dir'] tor = self._env['tor'] - idfile = os.path.join(datadir,'keys',"identity_key") cmdline = [ tor, "--quiet", @@ -445,9 +444,9 @@ class LocalNodeController(NodeController): def hup(self): """Send a SIGHUP to this node, if it's running.""" pid = self.getPid() - running = self.isRunning() + running = self.isRunning(pid) nick = self._env['nick'] - if self.isRunning(): + if running: print "Sending sighup to %s"%nick os.kill(pid, signal.SIGHUP) return True diff --git a/lib/chutney/Traffic.py b/lib/chutney/Traffic.py index 7202c3d..57250f7 100644 --- a/lib/chutney/Traffic.py +++ b/lib/chutney/Traffic.py @@ -312,7 +312,6 @@ import sys def main(): """Test the TrafficTester by sending and receiving some data.""" DATA = "a foo is a bar" * 1000 - DATA_CORRUPT = "a foo is a baz" * 1000 proxy = ('localhost', 9008) bind_to = ('localhost', int(sys.argv[1])) -- Daniel Martí - mvdan(a)mvdan.cc - http://mvdan.cc/ PGP: A9DA 13CD F7A1 4ACD D3DE E530 F4CA FFDB 4348 041C

2 1

GSoC final progress update
by Белоус Михаил 18 Aug '14

18 Aug '14

During GSoC I have done following: Threadpool using pththread Benchmark for this pool Multithread version of command_proccess_cell and fix most of the data racing places Current version of tor daemon, passes chutney test, and chutney + valgrind for data racing test. I glad I work for tor, thank you all.

1 0

Proposal 220 (revised): Migrate server identity keys to Ed25519
by Nick Mathewson 18 Aug '14

18 Aug '14

I've revised proposal 220 based on commentary from Roger. The biggest changes is tweaking all of the things called "certificates" to make them actually follow the same format to greatest the extent possible. To see diffs, you can use git, or browse the gitweb site at https://gitweb.torproject.org/torspec.git/history/HEAD:/proposals/220-ecc-i… Filename: 220-ecc-id-keys.txt Title: Migrate server identity keys to Ed25519 Authors: Nick Mathewson Created: 12 August 2013 Target: 0.2.x.x Status: Draft [Note: This is a draft proposal; I've probably made some important mistakes, and there are parts that need more thinking. I'm publishing it now so that we can do the thinking together.] 0. Introduction In current Tor designs, router identity keys are limited to 1024-bit RSA keys. Clearly, that should change, because RSA doesn't represent a good performance-security tradeoff nowadays, and because 1024-bit RSA is just plain too short. We've already got an improved circuit extension handshake protocol that uses curve25519 in place of RSA1024, and we're using (where supported) P256 ECDHE in our TLS handshakes, but there are more uses of RSA1024 to replace, including: * Router identity keys * TLS link keys * Hidden service keys This proposal describes how we'll migrate away from using 1024-bit RSA in the first two, since they're tightly coupled. Hidden service crypto changes will be complex, and will merit their own proposal. In this proposal, we'll also (incidentally) be extirpating a number of SHA1 usages. 1. Overview When this proposal is implemented, every router will have an Ed25519 identity key in addition to its current RSA1024 public key. Ed25519 (specifically, Ed25519-SHA-512 as described and specified at http://ed25519.cr.yp.to/) is a desirable choice here: it's secure, fast, has small keys and small signatures, is bulletproof in several important ways, and supports fast batch verification. (It isn't quite as fast as RSA1024 when it comes to public key operations, since RSA gets to take advantage of small exponents when generating public keys.) (For reference: In Ed25519 public keys are 32 bytes long, private keys are 64 bytes long, and signatures are 64 bytes long.) To mirror the way that authority identity keys work, we'll fully support keeping Ed25519 identity keys offline; they'll be used to sign long-ish term signing keys, which in turn will do all of the heavy lifting. A signing key will get used to sign the things that RSA1024 identity keys currently sign. 1.1. 'Personalized' signatures Each of the keys introduced here is used to sign more than one kind of document. While these documents should be unambiguous, I'm going to forward-proof the signatures by specifying each signature to be generated, not on the document itself, but on the document prefixed with some distinguishing string. 2. Certificates and Router descriptors. 2.1. Certificates When generating a signing key, we also generate a certificate for it. Unlike the certificates for authorities' signing keys, these certificates need to be sent around frequently, in significant numbers. So we'll choose a compact representation. VERSION [1 Byte] CERT_TYPE [1 Byte] EXPIRATION_DATE [3 Bytes] CERT_KEY_TYPE [1 byte] CERTIFIED_KEY [32 Bytes] EXTENSIONS [variable length, up to length of certificate minus 64 bytes.] SIGNATURE [64 Bytes] The "VERSION" field holds the value [01]. The "CERT_TYPE" field holds a value depending on the type of certificate. (See appendix A.1.) The CERTIFIED_KEY field is an Ed25519 public key if CERT_KEY_TYPE is [01], or a SHA256 hash of some other key type depending on the value of CERT_KEY_TYPE. The EXPIRATION_DATE is a date, given in HOURS since the epoch, after which this certificate isn't valid. (A three-byte field here will work fine until 5797 A.D.) The EXTENSIONS field contains zero or more extensions, each of the format: ExtLength [1 or 2 bytes] ExtType [1 or 2 bytes] ExtData [Length bytes] The ExtLength and ExtType fields can represent values between 0 and 2^15-1, representing values under 128 as "0xxxxxxx" and values over 128 as "1xxxxxxx yyyyyyyy". The meaning of the ExtData field in an extension is type-dependent. It is an error for an extension to be truncated; such a certificate is invalid. Before processing any certificate, parties MUST know which identity key it is supposed to be signed by, and then check the signature. The signature is formed by signing the first N-64 bytes of the certificate prefixed with the string "Tor node signing key certificate v1". 2.2. Basic extensions 2.2.1. Signed-with-ed25519-key extension [type 04] In several places, it's desirable to bundle the key signing a certificate along with the certificate. We do so with this extension. ExtLength = 32 ExtData = An ed25519 key [32 bytes] When this extension is present, it MUST match the key used to sign the certificate. This 2.3. Revoking keys. We also specify a revocation document for revoking a signing key or an identity key. Its format is: FIXED_PREFIX [8 Bytes] VERSION [1 Byte] KEYTYPE [1 Byte] IDENTITY_KEY [32 Bytes] REVOKED_KEY [32 Bytes] PUBLISHED [8 Bytes] REV_EXTENSIONS [variable length, up to length of revocation document minus 64 bytes] SIGNATURE [64 Bytes] FIXED_PREFIX is "REVOKEID" or "REVOKESK". VERSION is [01]. KEYTYPE is [01] for revoking a signing key or [02] for revoking an identity key. REVOKED_KEY is the key being revoked; IDENTITY_KEY is the node's Ed25519 identity key. PUBLISHED is the time that the document was generated, in seconds since the epoch. REV_EXTENSIONS is left for a future version of this document. The SIGNATURE is generated with the same key as in IDENTITY_KEY, and covers the entire revocation, prefixed with "Tor key revocation v1". Using these revocation documents is left for a later specification. 2.4. Managing keys By default, we can keep the easy-to-setup key management properties that Tor has now, so that node operators aren't required to have offline public keys: * When a Tor node starts up with no Ed25519 identity keys, it generates a new identity keypair. * When a Tor node has an Ed25519 identity keypair, and it has no signing key, or its signing key is going to expire within the next 48 hours, it generates a new signing key to last 30 days. But we also support offline identity keys: * When a Tor node starts with an Ed25519 public identity key but no private identity key, it checks whether it has a currently valid certified signing keypair. If it does, it starts. Otherwise, it refuses to start. * If a Tor node's signing key is going to expire soon, it starts warning the user. If it is expired, then the node shuts down. 2.5. Router descriptors We specify the following element that may appear at most once in each router descriptor: "identity-ed25519" SP certificate NL The identity-key and certificate are base64-encoded with terminating =s removed. When this element is present, it MUST appear as the first or second element in the router descriptor. [XXX The rationale here is to allow extracting the identity key and signing key and checking the signature before fully parsing the rest of the document. -NM] The certificate has CERT_TYPE of [04]. It must include a signed-with-ed25519-key extension (see section 2.2.1), so that we can extract the identity key. When an identity-ed25519 element is present, there must also be a "router-signature-ed25519" element. It MUST be the next-to-last element in the descriptor, appearing immediately before the RSA signature. It MUST contain an ed25519 signature of the entire document, from the first character up to but not including the "router-signature-ed25519" element, prefixed with the string "Tor router descriptor signature v1". Its format is: "router-signature-ed25519" SP signature NL Where 'signature' is encoded in base64 with terminating =s removed. The signing key in the certificate MUST be the one used to sign the document. Note that these keys cross-certify as follows: the ed25519 identity key signs the ed25519 signing key in the certificate. The ed25519 signing key signs itself and the ed25519 identity key and the RSA identity key as part of signing the descriptor. And the RSA identity key also signs all three keys as part of signing the descriptor. 2.5.1. Checking descriptor signatures. Current versions of Tor will handle these new formats by ignoring the new fields, and not checking any ed25519 information. New versions of Tor will have a flag that tells them whether to check ed25519 information. When it is set, they must check: * All RSA information and signatures that Tor implementations currently check. * If the identity-ed25519 line is present, it must be well-formed, and the certificate must be well-formed and correctly signed, and there must be a valid router-signature-ed25519 signature. * If we require an ed25519 key for this node (see 3.1 below), the ed25519 key must be present. Authorities and directory caches will have this flag always-on. For clients, it will be controlled by a torrc option and consensus option, to be set to "always-on" in the future once enough clients support it. 2.5.2. Extra-info documents Extra-info documents now include "identity-ed25519" and "router-signature-ed25519" fields in the same positions in which they appear in router descriptors. Additionally, we add the base64-encoded, =-stripped SHA256 digest of a node's extra-info document field to the extra-info-digest line in the router descriptor. (All versions of Tor that recognize this line allow an extra field there.) 2.5.3. A note on signature verification Here and elsewhere, we're receiving a certificate and a document signed with the key certified by that certificate in the same step. This is a fine time to use the batch signature checking capability of Ed25519, so that we can check both signatures at once without (much) additional overhead over checking a single signature. 3. Consensus documents and authority operation 3.1. Handling router identity at the authority When receiving router descriptors, authorities must track mappings between RSA and Ed25519 keys. Rule 1: Once an authority has seen an Ed25519 identity key and an RSA identity key together on the same (valid) descriptor, it should no longer accept any descriptor signed by that RSA key with a different Ed25519 key, or that Ed25519 key with a different RSA key. Rule 2: Once an authority has seen an Ed25519 identity key and an RSA identity key on the same descriptor, it should no longer accept any descriptor signed by that RSA key unless it also has that Ed25519 key. These rules together should enforce the property that, even if an attacker manages to steal or factor a node's RSA identity key, the attacker can't impersonate that node to the authorities, even when that node is identified by its RSA key. Enforcement of Rule 1 should be advisory-only for a little while (a release or two) while node operators get experience having Ed25519 keys, in case there are any bugs that cause or force identity key replacement. Enforcement of Rule 2 should be advisory-only for little while, so that node operators can try 0.2.5 but downgrade to 0.2.4 without being de-listed from the consensus. [XXX I could specify a way to do a signed "I'm downgrading for a while!" statement, and kludge some code back into 0.2.4.x to better support that?] 3.2. Formats Vote and microdescriptor documents now contain an optional "id" field for each routerstatus section. Its format is: "id" SP "ed25519" SP ed25519-identity NL where ed25519-identity is base64-encoded, with trailing = characters omitted. In vote documents, it may be replaced by the format: "id" SP "ed25519" SP "none" NL which indicates that the node does not have an ed25519 identity. (In a microdescriptor, a lack of "id" line means that the node has no ed25519 identity.) [XXXX Should the id entries in consensuses go into microdescriptors instead? I think perhaps so. -NM] A vote or consensus document is ill-formed if it includes the same ed25519 identity key twice. 3.3. Generating votes An authority should pick which descriptor to choose for a node as before, and include the ed25519 identity key for the descriptor if it's present. As a transition, before Rule 1 and Rule 2 in 3.1 are fully enforced, authorities need a way to deal with the possibility that there might be two nodes with the same ed25519 key but different RSA keys. In that case, it votes for the one with the most recent publication date. (The existing rules already prevent an authority from voting for two servers with the same RSA identity key.) 3.4. Generating a consensus from votes This proposal requires a new consensus vote method. When we deploy it, we'll pick the next available vote method in sequence to use for this. When the new consensus method is in use, we must choose nodes first by ECC key, then by RSA key. First, for every {ECC identity key, RSA identity key} pair listed by over half of the voting authorities, list it, unless some other RSA identity key digest is listed more popularly for the ECC key. Break ties in favor of low RSA digests. Treat all routerstatus entries that mention this <ECC,RSA> pair as being for the same router, and all routerstatus entries that mention the same RSA key with an unspecified ECC key as being for the same router. Then, for every node that has previously not been listed, perform the current routerstatus algorithm: listing a node if it has been listed by at least N/2 voting authorities, and treating all routerstatuses containing the same identity as the same router. In other words: Let Entries = [] for each ECC ID listed by any voter: Find the RSA key associated with that ECC ID by the most voters, breaking ties in favor of low RSA keys. If that ECC ID and RSA key ID are listed by > N/2 voting authorities: Add the consensus of the routerstatus entries for those voters, along with the routerstatus entry for every voter that included that RSA key with no ECC key, to Entries. Include the ECC ID in the consensus. For each RSA key listed by any voter: If that RSA key is already in Entries, skip it. If the RSA key is listed by > N/2 voting authorities: Add the consensus of the routerstatus entries for those voters to Entries. Do not include an ECC key in the consensus. [XXX Think about this even more.] 4. The link protocol 4.1. Overview of the status quo This section won't make much sense unless you grok the v3 link protocol as described in tor-spec.txt, first proposed in proposal 195. So let's review. In the v3 link protocol, the client completes a TLS handshake with the server, in which the server uses an arbitrary certificate signed with an RSA key. The client then sends a VERSIONS cell. The server replies with a VERSIONS cell to negotiate version 3 or higher. The server also sends a CERTS cell and an AUTH_CHALLENGE cell and a NETINFO cell. The CERTS cell from the server contains a set of one or more certificates that authenticate the RSA key used in the TLS handshake. (Right now there's one self-signed RSA identity key certificate, and one certificate signing the RSA link key with the identity key. These certificates are X509.) Having received a CERTS cell, the client has enough information to authenticate the server. At this point, the client may send a NETINFO cell to finish the handshake. But if the client wants to authenticate as well, it can send a CERTS cell and an AUTENTICATE cell. The client's CERTS cell also contains certs of the same general kinds as the server's key file: a self-signed identity certificate, and an authentication certificate signed with the identity key. The AUTHENTICATE cell contains a signature of various fields, including the contents of the AUTH_CHALLENGE which the server sent cell, using the client's authentication key. These cells allow the client to authenticate to the server. 4.2. Link protocol changes for ECC ID keys We add four new CertType values for use in CERTS cells: 4: Ed25519 signing key 5: Link key certificate certified by Ed25519 signing key 6: Ed25519 TLS authentication key certified by Ed25519 signing key 7: RSA cross-certificate for Ed25519 identity key These correspond to types used in the CERT_TYPE field of the certificates. The content of certificate type [04] (Ed25519 signing key) is as in section 2.5 above, containing an identity key and the signing key, both signed by the identity key. Certificate type [05] (Link certificate signed with Ed25519 signing key) contains a SHA256 digest of the X.509 link certificate used on the TLS connection in its key field; it is signed with the signing key. Certificate type [06] (Ed25519 TLS authentication signed with Ed25519 signing key) has the signing key used to sign the AUTHENTICATE cell described later in this section. Certificate type [07] (Cross-certification of Ed25519 identity with RSA key) contains the following data: ED25519_KEY [32 bytes] EXPIRATION_DATE [4 bytes] SIGNATURE [128 bytes] Here, the Ed25519 identity key is signed with router's RSA identity key, to indicate that authenticating with a key certified by the Ed25519 key counts as certifying with RSA identity key. (The signature is computed on the SHA256 hash of the non-signature parts of the certificate, prefixed with the string "Tor TLS RSA/Ed25519 cross-certificate".) (There's no reason to have a corresponding Ed25519-signed-RSA-key certificate here, since we do not treat authenticating with an RSA key as proving ownership of the Ed25519 identity.) Relays with Ed25519 keys should always send these certificate types in addition to their other certificate types. Non-bridge relays with Ed25519 keys should generate TLS link keys of appropriate strength, so that the certificate chain from the Ed25519 key to the link key is strong enough. We add a new authentication type for AUTHENTICATE cells: "Ed25519-TLSSecret", with AuthType value 2. Its format is the same as "RSA-SHA256-TLSSecret", except that the CID and SID fields support more key types; some strings are different, and the signature is performed with Ed25519 using the authentication key from a type-6 cert. Clients can send this AUTHENTICATE type if the server lists it in its AUTH_CHALLENGE cell. Modified values and new fields below are marked with asterisks. TYPE: The characters "AUTH0002"* [8 octets] CID: A SHA256 hash of the initiator's RSA1024 identity key [32 octets] SID: A SHA256 hash of the responder's RSA1024 identity key [32 octets] *CID_ED: The initiator's Ed25519 identity key [32 octets] *SID_ED: The responder's Ed25519 identity key, or all-zero. [32 octets] SLOG: A SHA256 hash of all bytes sent from the responder to the initiator as part of the negotiation up to and including the AUTH_CHALLENGE cell; that is, the VERSIONS cell, the CERTS cell, the AUTH_CHALLENGE cell, and any padding cells. [32 octets] CLOG: A SHA256 hash of all bytes sent from the initiator to the responder as part of the negotiation so far; that is, the VERSIONS cell and the CERTS cell and any padding cells. [32 octets] SCERT: A SHA256 hash of the responder's TLS link certificate. [32 octets] TLSSECRETS: A SHA256 HMAC, using the TLS master secret as the secret key, of the following: - client_random, as sent in the TLS Client Hello - server_random, as sent in the TLS Server Hello - the NUL terminated ASCII string: "Tor V3 handshake TLS cross-certification with Ed25519"* [32 octets] RAND: A 24 byte value, randomly chosen by the initiator. [24 octets] *SIG: A signature of all previous fields using the initiator's Ed25519 authentication flags. [variable length] If you've got a consensus that lists an ECC key for a node, but the node doesn't give you an ECC key, then refuse this connection. 5. The extend protocol We add a new NSPEC node specifier for use in EXTEND2 cells, with LSTYPE value [03]. Its length must be 32 bytes; its content is the Ed25519 identity key of the target node. Clients should use this type only when: * They know an Ed25519 identity key for the destination node. * The source node supports EXTEND2 cells * A torrc option is set, _or_ a consensus value is set. We'll leave the consensus value off for a while until more clients support this, and then turn it on. When picking a channel for a circuit, if this NSPEC value is provided, then the RSA identity *and* the Ed25519 identity must match. If we have a channel with a given Ed25519 ID and RSA identity, and we have a request for that Ed25519 ID and a different RSA identity, we do not attempt to make another connection: we just fail and DESTROY the circuit. If we receive an EXTEND or EXTEND2 request for a node listed in the consensus, but that EXTEND/EXTEND2 request does not include an Ed25519 identity key, the node SHOULD treat the connection as failed if the Ed25519 identity key it receives does not match the one in the consensus. 6. Naming nodes in the interface Anywhere in the interface that takes an $identity should be able to take an ECC identity too. ECC identities are case-sensitive base64 encodings of Ed25519 identity keys. You can use $ to indicate them as well; we distinguish RSA identity digests by length. When we need to indicate an Ed25519 identity key in a hostname format (as in a .exit address), we use the lowercased version of the name, and perform a case-insensitive match. (This loses us a little less than one bit per byte of name, leaving plenty of bits to make sure we choose the right node.) Nodes must not list Ed25519 identities in their family lines; clients and authorities must not honor them there. (Doing so would make different clients change paths differently in a possibly manipulatable way.) Clients shouldn't accept .exit addresses with Ed25519 names on SOCKS or DNS ports by default, even when AllowDotExit is set. We can add another option for them later if there's a good reason to have this. We need an identity-to-node map for ECC identity and for RSA identity. The controller interface will need to accept and report Ed25519 identity keys as well as (or instead of) RSA identity keys. That's a separate proposal, though. 7. Hidden service changes out of scope Hidden services need to be able to identify nodes by ECC keys, just as they will need to include ntor keys as well as TAP keys. Not just yet though. This needs to be part of a bigger hidden service revamping strategy. 8. Proposed migration steps Once a few versions have shipped with Ed25519 key support, turn on "Rule 1" on the authorities. (Don't allow an Ed25519<->RSA pairing to change.) Once the release with these changes is in beta or rc, turn on the consensus option for everyone who receives descriptors with Ed25519 identity keys to check them. Once the release with these changes is in beta or rc, turn on the consensus option for clients to generate EXTEND2 requests with Ed25519 identity keys. Once the release with these changes has been stable for a month or two, turn on "Rule 2" on authorities. (Don't allow nodes that have advertised an Ed25519 key to stop.) 9. Future proposals * Ed25519 identity support on the controller interface * Supporting nodes without RSA keys * Remove support for nodes without Ed25519 keys * Ed25519 support for hidden services * Bridge identity support. * Ed25519-aware family support A.1. List of certificate types The values marked with asterisks are not types corresponding to the certificate format of section 2.1. Instead, they are reserved for RSA-signed certificates to avoid conflicts between the certificate type enumeration of the CERTS cell and the certificate type enumeration of in our Ed25519 certificates. **[00],[01],[02],[03] - Reserved to avoid conflict with types used in CERTS cells. [04] - signing a signing key with an identity key (Section 2.5) [05] - TLS link certificate signed with ed25519 signing key (Section 4.2) [06] - Ed25519 authentication key signed with ed25519 signing key (Section 4.2) **[07] - reserved for RSA identity cross-certification (Section 4.2) A.2. List of extension types [01] - signed-with-ed25519-key (section 2.2.1) A.3. List of signature prefixes We describe various documents as being signed with a prefix. Here are those prefixes: "Tor router descriptor signature v1" (section 2.5) "Tor node signing key certificate v1" (section 2.1) A.4. List of certified key types [01] ed25519 key [02] SHA256 hash of an RSA key [03] SHA256 hash of an X.509 certificate A.5. Reserved numbers We need a new consensus algorithm number to encompass checking ed25519 keys and putting them in microdescriptors. We need new CertType values for use in CERTS cells. We reserved in section 4.2. 4: Ed25519 signing key 5: Link key certificate certified by Ed25519 signing key 6: TLS authentication key certified by Ed25519 signing key 7: RSA cross-certificate for Ed25519 identity key

4 3

PKCS#1 ASN.1 Public Key Encoding
by Gareth Owen 17 Aug '14

17 Aug '14

Hi all I wonder if someone might be able to help me with the above. I understand, that to generate the digest, the PK must be encoded in PKCS#1 format. And further to this, the public keys in the router descriptors are NOT in PKCS#1 format, but plain ASN.1. I'm trying to generate the fingerprint given just the pubilc key in Java and after almost a whole day I'm about to give up. Does anyone have a sample PKCS#1 encoded public key that is used immediately before SHA-1 to generate the fingerprint? e.g. a hex string is what I'm after. It seems there are subtle ways that an PKCS#1 can vary while encoding the same information which affects the hash, Java seems to be doing it one way, OpenSSL another, an example on stack overflow adds an extra field, etc. Many thanks Gareth

2 2