[RFC] Directory structure of prop224 onion services
Hey list, with service-side prop224 implementation moving forward, we need to pin down the directory structure of prop224 onion services. This will be very similar to the current directory structure, but with some mods to facilitate assymetric client authorization keys and offline keys. As people have pointed out, the HS directory structure matters less after the introduction of ephemeral ADD_ONION onion services, but still it's an important part of onion service sysadmin UX. So the HiddenServiceDir directory will contain the following items: - "./hostname" [FILE] This is a file containing the onion address of the onion service. As you can see it's the same filename as in v2. Should we suffix it with v3 to make it clear that it's v3 onion? Would we ever have v2 and v3 onions living in the same directory? - "./private_key_ed25519" [FILE] This is the file containing the private master ed25519 key of the onion service. If offline keys are _enabled_, then this file doesn't exist and instead a directory is made containing blinded keys for every day [TODO: The directory format here will be specified in the future]. - "./client_authorized_pubkeys" [FILE] If client authorization is _enabled_, this is a newline-separated file of "<client name> <pubkey>" entries for authorized clients. You can think of it as the ~/.ssh/authorized_keys of onion services. - "./client_authorized_privkeys/" [DIRECTORY] "./client_authorized_privkeys/alice" [FILE] "./client_authorized_privkeys/bob" [FILE] "./client_authorized_privkeys/charlie" [FILE] If client authorization is _enabled_ _AND_ if the hidden service is responsible for generating and distributing private keys for its clients, then this directory contains files with client's private keys. The idea is that these files can be shredded and deleted after the private key has been passed to the client. For more context here, please read the client authorization thread in [tor-dev] and see 'Appendix F' of prop224 for more details on how this works. So this is it. The above should handle most uses of onion services + client authorization. The directory format of offline keys will be specified as we move forward with implementation. Hope things here are not too controversial. Looking forward to your feedback. In a few days, I will add a small Appendix section to prop224 with the above, and also fix the parts of 'Appendix F' that got outdated since then. Cheers!
On 26 Jan (15:05:26), George Kadianakis wrote:
Hey list,
Hi! First, big thanks for this write up!
I don't believe we should suffix here because for almost 10 years, users/apps have been exposed to "hostname" and it does make sense that it's the goto file for that. Current implementation doesn't allow two services in the same HiddenServiceDir and for prop224, the ongoing implementation doesn't allow it either. Sharing a directory brings all sorts of uneeded complexity. So if the directory is v3, everything in it will be v3.
If that file doesn't exists, the public key is needed else the service can't derive the .onion and create the hostname file. The offline case is an extra use case but I suspect we would use "public_key_ed25519" along with the blinded keys specific file name. (Unless we make our "tor-genkey" tool generate the hostname file as well. #bikesheding)
Small clarification. The "<client name>" field in the the pubkey file is the same for the privkey file name. So if "alice" is in the pubkey file, it will be "alice" in this privkey directory.
Also, expected behavior that we should go for when implementing this within the "tor" code base. We could think of many ways to make this more complex that it could be but going *simple* is what I'm aiming for: - The torrc option HiddenServiceAuthorizeClient as to match the list of client in the pubkey file in so if the pubkey file has extra entries, we error at startup. With this in mind, here are the behaviors: i) if a privkey file exists but no entry in the pubkey file, add the entry to pubkey file as long as the client name is found in HiddenServiceAuthorizeClient. ii) a pubkey entries does NOT need a corresponding privkey to be used. As long as the client name is found in HiddenServiceAuthorizeClient. iii) if a client name is specified in the HiddenServiceAuthorizeClient option but NOT in the pubkey file, generate pubkey/privkey unless the privkey file exists which is (i) behavior where pubkey is derived from privkey. So the great thing about this is that you can create a keypair on a different machine (or client side), put the privkey file in the client_authorized_privkeys directory and add the client name to torrc, HUP tor and done. We could see ultimately an auto update of the service configuration with the client name but I'm not a big fan of changing the torrc file automagically... Thanks! David
-- LPDGvjmCLxb+bnFoopeN77JIq714VtyU6DdFrKs4R08=
David Goulet <dgoulet@ev0ke.net> writes:
Hey David, thanks for the useful comments. Please check my torspec branch `prop224-directory-format`. FWIW, I agree with all the expected behavior details you noted at the end of your email. I encoded some of those behaviors in the spec, but I didn't provide a complete formal algorithm of how the whole process works because I don't think it's spec material and also because I feel that during implementation we will get new insights on how this should work. Let me know how you feel about the spec patch :) Cheers!
On 30 Jan (16:16:07), George Kadianakis wrote:
Good stuff! And yes, I don't think it's spec material at all but good to have in an Appendfix for reference. Once this file structure will be released in a tor version, we *must* update the man page FILES section. Thanks! David
Cheers!
-- IOA3vUozzhhCnEz3vEUUkbz+AJwvgSYAkuxkheMCHHA=
Hi, David Goulet wrote:
Directory structure looks good to me. I agree the file containing the address should just be called 'hostname' with no version suffix or anything, it will be redundant and create confusions as soon as v2 is entirely deprecated. Also, 'hostname' file is only responsible to provide the right address that can be used to connect to the onion service, it should not care about the version or include this information it the containing file name. What we will do here to avoid accidents is that we will use the public_key_ed25519 file as the starting point. If this file is present, we will assume the onion service is generated and existent. If not, compute it and save to disk either from private_key_ed25519 (if available) or the blinded keys (if available). I like the protocol suggested by David very much for managing auth keys for clients. Nice, logic and should leave no room for accidents. Here's another protocol for managing the onion service identity (keys). This is not urgent but I am writing it now so I won't forget something and we will have it here when needed: 1. Check if private_key_ed25519 exists. If it doesn't exist, move to step 2. 1.1 If it exists, compute public_key_ed25519 and hostname, or validate the latter two if they exist. If there is a mismatch, exit with error, user accidentally mixed something. 2. Check if public_key_ed25519 exists. If it doesn't exist, move to step 3. 2.1 If it exists, compute hostname or validate the latter one if it exists, If there is a mismatch, exit with error, used accidentally mixed something. 3. Check if a blinded key exists. 3.1 If it doesn't exist, and private_key_ed25519 is available, compute it and go to step 3.2, otherwise go to step 4. 3.2 If it exists and is valid, compute public_key_ed25519 and hostname, or validate the latter two if they exist. If there is a mismatch, exit with error, user accidentally mixed something. 3.3 If it is invalid and private_key_ed25519 is available, compute a new one and go back to step 3.2, otherwise exit with error asking for manual further action with genkey tool. 4. If offline onion service keys is not set, create a new private_key_ed25519 and go to step 1 again. 4.1 If offline onion service keys is set, exit with error asking for manual further action with genkey tool. As always, epic work. Thanks !
This works for applications that simply display the hostname without any further processing. But code that expects a short hostname file may become confused when exposed to a longer hostname. It may fail silently, refuse to work, or have some other issue. Have you tested any applications that use the hostname file with longer names?
How does an application tell the difference between a v2 and v3 directory? What's the supported method, that we will continue to support in future, regardless of key or algorithm changes? T -- Tim Wilson-Brown (teor) teor2345 at gmail dot com PGP C855 6CED 5D90 A0C5 29F6 4D43 450C BA7F 968F 094B ricochet:ekmygaiu4rzgsk6n xmpp: teor at torproject dot org ------------------------------------------------------------------------
On Tue, Jan 31, 2017 at 09:02:35AM +1100, teor wrote:
I guess by looking at the address and checking its validity and/or looking at the vesion field in v3 (see recent discussion on onion address encoding). So if there will be crypto changes there will be version field changes as well. -- Ivan Makrin
On 31 Jan (09:02:35), teor wrote:
To be honest, I would really hope applications just don't blindly open the file and regex on a 16 char onion... They will be really sad with client auth. But, we can't assume they are all perfect so yes that could be a problem. And I hope that if it's the case, they error gracefully if regex fails. It's kind of unreasonable to ask us to make sure we don't break any applications out there doing such. The only assertion they should make on the "hostname" file is that it contains the address of the address. Your next question is a good one actually so jumping into it!
Right now, apart from the "key tag" in the public key file that will be something like "v3", I don't see any :S ... And I think you are raising a good point. How can we make it easy for any application *only* looking on disk what version the hidden service is? The onion address has the version encoded into it *but* that would require the application to do some base32 decode and truncation magic. Not user friendly enough? Maybe but apart from that, we would need to make it obvious either with a filename or litterally an extra "version file". However, there is kind of an issue rising from this. Imagine that v4 changes the onion address format because new crypto. We'll end up with a problem where the how to extract the version from the address is actually version specific... A solution to that is that "whatever size/encoding the address is, version will ALWAYS be the last 1 byte." Thoughts?
What's the supported method, that we will continue to support in future, regardless of key or algorithm changes?
Not 100% sure what you mean by "supported method" but I'll take a guess. Those are the one that comes to mind: - Single onion service - Client auth (only basic for now, we don't have stealth specified) Also, every single Hidden Service torrc option will be supported for v3 *EXCEPT* the Tor2Web one. Hope that answers. Thanks! David
-- laomWpXMwfgJQSD00XsC6WR0haEh1gK8WeNTaAPQf20=
I think this would be a good idea: * version 2 hidden services have no version file * version 3 hidden services have a version file with "3" in it and so on...
I think it is a good idea to make the version the last byte of the address. But I also think that a version file is a good idea to make it easy for applications to discover the on-disk version. Otherwise, the algorithm would have to be something like: * look for a hostname file * read the first line * find the address in that line * if the address is N characters long, version 2 * do we promise we will never have addresses this long in future versions? * base32 decode that line * do we promise addresses will always be base32? * read the last byte * do we promise addresses will always have the version in the last byte?
Actually, what I meant was: "What can an application do to find the hidden service version, that we always promise will work in any future version?" That's what we're discussing above. T -- Tim Wilson-Brown (teor) teor2345 at gmail dot com PGP C855 6CED 5D90 A0C5 29F6 4D43 450C BA7F 968F 094B ricochet:ekmygaiu4rzgsk6n xmpp: teor at torproject dot org ------------------------------------------------------------------------
On Wed, Feb 01, 2017 at 09:36:54AM +1100, teor wrote:
Sure, if stays here it's effectively a version suffix (label). Another version will imply address length, encoding, checksum, etc.
This entire idea to do something for some apps to detect onion service version "just by looking at disk" looks like a feature creep to me. Why should we bother at all while we have onion address that is self-descriptive?* It's not a big deal really to parse onion address from a 'hostname' file. Especially for an app that wants to know the protocol version (what for?). Personally I belive that there are no(t so many) apps that would mess with onion services without using an onion-related library such as stem which will make it as simple as one function call over onion address. [*] Yeah, for that there should always be Base32 and version byte at the end. -- Ivan Markin
David Goulet <dgoulet@ev0ke.net> writes:
I'm +1 on keeping the filename as "hostname". txtorcon doesn't do processing on it, per se. Might be worth noting here that it's not always "a" hostname: if you have an authenticated hidden-service, it is multiple lines. In "basic" auth case, each line starts with the same .onion address -- in "stealth" case, each line has its own .onion address. cheers, -- meejah
On 26 Jan (15:05:26), George Kadianakis wrote:
Ok small modification here. Tor has an API to load/generate ed25519 keys that uses standard suffix for the keys. Basically, we can use any filename we want but the secret and public key will be suffixed with "_public_key" and "_secret_key". Thus, I'll make a slight change to the proposal for the key files: "hs_ed25519_public_key" "hs_ed25519_secret_key" The public key is added because for the offline key case. Note that currently a relay secret key file is: "ed25519_master_id_secret_key" so we added the "hs_" part to make it obvious for instance when an operator is backing up keys that those are for HS. Furthermore, those are the only keys that will touch the disk (except for blinded keys in case of offline keys but not implemented yet) thus they are obviously the "master id" key of the HS. Thanks! David
-- 7ubUkzha3XBwjjDlr6nT01SbL9JGW7YactfQDuqq6yw=
participants (6)
-
David Goulet -
George Kadianakis -
Ivan Markin -
meejah -
s7r -
teor