-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256
Hi,
Tor 0.2.7.x will support Ed25519 router identities along with the traditional 1024-bit RSA ones which will be used simultaneously for some time, until we will completely deprecate RSA router identities.
I would like to document what Tor needs to do (or don't do) by default in some cases to prevent mistakes which confused operators could do. Please review and add/edit/remove if something is wrong, doesn't make sense or is incomplete. When finished, I will split it in more parts and open tickets on trac based on priorities. Some cases are already covered (Tor does what it should by default). We also need to compose the messages Tor will print in the log files in a way that they will be of a reasonable length and expressive/explicit/polite/whatever.
Take in consideration the design: For an Ed25519 identity, Tor will generate these files in $datadirectory/keys:
- - ed25519_master_id_secret_key - the master identity key. Optionally this can be kept offline or encrypted. This is the file one needs in case he wants to restore a relay. Never expires. Sensitive (if reached it will compromise the identity of the relay forever).
- - ed25519_master_id_public_key - the public key. This is the actual router identity which Tor publishes in the descriptors. It is computed from the secret key and it is not sensitive. Never expires.
- - ed25519_signing_secret_key - a temporary signing key. This will be used for circuits (onionskin) encryption. It is valid only temporary for as long as ed25519_signing_cert says so.
- - ed25519_signing_cert - a cert signed by the master ID key (ed25519_master_id_secret_key) attesting that the ed25519_signing_secret_key 'x' is valid for 'n' days/months. The validity period can be configured with the torrc option SigningKeyLifetime 'n' days/weeks/months (default is 30 days).
Also take in consideration that the directory authorities will pin relays with paired RSA-Ed25519 identities and reject them if one changes. For example, if relay A has ID RSA_A & Ed25519_A, and later changes to RSA_B & Ed25519_A or RSA_A & Ed25519_B, it will be rejected.
The text is also available on Etherpad: https://etherpad.mozilla.org/efIBq6nDwt - ------------------------------------------------------ Ed25519 router identity - default behavior - ------------------------------------------------------ We don't touch the RSA ID key behavior. We let it as it was, use it if secret ID file is there, generate new one if not. The behavior below should be implemented by taking in consideration only the ed25519_* files in $datadirectory/keys (should not care or notice the RSA related files). We don't say anything about offline Ed25519 ID keys or encrypted Ed25519 ID keys. Whoever wants to use these features will have to read the FAQ first and understand some things.
The default is: generate ed25519_master_id_secret_key (unencrypted) in $datadirectory/keys and use that forever, by automatically renewing ed25519_signing_secret_key and ed25519_signing_cert with the default validity period of 30 days, unless otherwise specified in torrc. This is already implemented and it is how Tor works now.
At every start, restart, reload, server reboot, signal sent by user or another app, etc. Tor should: 1) check if file ed25519_master_id_secret_key exists in $datadirectory/keys and it's a valid curve25519 private key file; 2) check if file ed25519_master_id_secret_key_encrypted exists in $datadirectory/keys; 3) check if file ed25519_master_id_public_key exists in $datadirectory/keys and it's a valid curve25519 public key file; 4) check if file ed25519_signing_cert exists in $datadirectory/keys; 5) check if ed25519_signing_cert in $datadirectory/keys is expired; 6) check if ed25519_signing_cert correlates with ed25519_master_id_public_key; 7) check if file ed25519_signing_secret_key exists in $datadirectory/keys; 8) check if ed25519_signing_secret_key correlates with ed25519_signing_cert (non-expired cert refers to that signing key);
1. If ed25519_master_id_secret_key && ed25519_master_id_public_key && ed25519_signing_secret_key && ed25519_signing_cert are all missing from $datadirectory/keys:
- -> We assume it's a new relay or a relay which just upgraded to a Tor that supports Ed25519 router identities. Generate a new Ed25519 identity unencrypted (ed25519_master_id_secret_key, ed25519_master_id_public_key, ed25519_signing_secret_key and ed25519_signing_cert with a validity for 30 days unless specified otherwise in torrc) and publish a descriptor using that, proceed as normal after.
2. If ed25519_master_id_secret_key is missing or exists encrypted && ed25519_master_id_public_key exists in $datadirectory/keys and ed25519_signing_secret_key and ed25519_signing_cert are expired:
- -> We assume this can happen suddenly, while running unattended. Disable relay functionality and stop publishing descriptors (close ORPort and (if open) DirPort). Leave client functionality since that could be used by something else on that machine and it shouldn't be affected. Print warnings in the log file that action is required (generate new ed25519_signing_secret_key and ed25519_signing_cert or move the ed25519_master_id_secret_key unencrypted in $datadirectory/keys so Tor can do this by itself). The operator needs to signal Tor with a reload signal (HUP) or restart/stop & start after completing the action. Tor will check for the valid ed25519_signing_secret_key and ed25519_signing_cert by itself every time server restarts (along with Tor process) or at every automated reload signal (HUP).
3. If ed25519_master_id_secret_key exists unencrypted in $datadirectory/keys and there is no ed25519_master_id_public_key, ed25519_signing_secret_key or ed25519_signing_cert (or ed25519_signing_secret_key and ed25519_signing_cert are expired):
- -> We assume the relay has just been restored and the Ed25519 identity was backed up. Automatically generate ed25519_master_id_public_key, ed25519_signing_secret_key and ed25519_signing_cert with a validity period of 30 days (unless otherwise specified in torrc). - -> Since we have the ed25519_master_id_secret_key unencrypted, and at startup we always check for availability/validity of the other ed25519* files by default anyway, we should generate automatically in this case: ed25519_master_id_public_key (if missing), ed25519_signing_secret_key (if missing or expired) and ed25519_signing_cert (if missing or expired).
4. If ed25519_master_id_secret_key exists encrypted in $datadirectory/keys && there is no ed25519_master_id_public_key, ed25519_signing_secret_key or ed25519_signing_cert (even if just one of these 3 is missing):
- -> We assume the relay identity existed previously and ed25519_master_id_secret_key_encrypted was restored form a backup. Do not start Tor at all and print explicit warnings that action is required - Tor cannot decrypt the ed25519_master_id_secret_key_encrypted file. Generate ed25519_master_id_public_key, ed25519_signing_secret_key and ed25519_signing_cert or decrypt the ed25519_master_id_secret_key_encrypted in $datadirectory/keys so Tor can do it by itself.
5. If the chain validation fails (ed25519_signing_cert doesn't correlate to ed25519_master_id_public_key AND/OR ed25519_signing_secret_key doesn't correlate to ed25519_signing_cert):
- -> We assume the operator probably runs many relays and holds many ed25519_master_id_secret_key files maybe accidentally mixes them and generates temporary signing key and key cert using the master key from another relay. Do no start Tor service at all and print warnings: the ed25519_signing_secret_key and ed25519_signing_cert are generated with the ed25519_master_id_secret_key of another relay. These files should be removed and replaced with the correct ones (search for the ed25519_master_id_secret_key which belongs to this relay). Otherwise it's useless (since we already reject RSA-Ed25519 which don't match), if the ed25519_master_id_secret_key is lost permanently, nothing else for the operator to do but to manually generate fresh identity both RSA and Ed25519 so the relay can be part of the consensus (start from scratch).
6. If ed25519_master_id_secret_key is missing or _encrypted AND ed25519_master_id_public_key is also missing from $datadirectory/keys AND a) we have an ed25519_signing_secret_key and ed25519_signing_cert which are not expired:
- ->Automatically compute ed25519_master_public_key from the ed25519_signing_cert and save it in $datadirectory/keys.
b) if ed25519_signing_secret_key and ed25519_signing_cert are expired:
- ->Do no start Tor service at all and print warnings that user action is required (generate new ed25519_signing_secret_key and ed25519_signing_cert or move the ed25519_master_id_secret_key unencrypted in $datadirectory/keys so Tor can do this by itself). The operator needs to signal Tor with a reload signal (HUP) or restart/stop & start after completing the action. Tor will check for the valid ed25519_signing_secret_key and ed25519_signing_cert by itself every time server restarts (along with Tor process) or at every automated reload signal (HUP).
- ------------------------------------------------------ Commands related to Ed25519 keys - ------------------------------------------------------ 1. tor --keygen should be replaced with tor --ecckeygen (keygen is not explicit enough since we are talking about 2 router identities which are used simultaneously [RSA and Ed25519]). - - Ask if user wants to encrypt the key with a passphrase and if yes ask 2 passphrase confirmations. Only allow ASCII, check and return invalid passphrase otherwise (local language keyboards mess things up). I suggest the Encrypt key? question [y/N] should be a No by default. - - Unless user specifies --out /path/to/dir to save the output files, save them in working directory (where the command is run). Do not create the folder and save them in $HOME/.tor. - - Only generate an ed25519_master_id_secret_key with this command (no ed25519_master_id_public_key, ed25519_signing_secret_key or ed25519_signing_cert). Print a message confirming this: ed25519_master_id_secret_key was successfully generated and saved encrypted/unencrypted (depending how it was generated) in $location ($location = either working directory where the command was run either the location specified separately with --out argument included in the - --ecckeygen command). Append _encrypted at the end of the file name in case a passphrase was provided.
2. tor --encryptkey <ed25519_master_id_secret_key file> or /path/to/<key file> - - check if it's a valid ed25519_master_id_secret_key unencrypted, ask for 2 passphrase confirmations and encrypt it. Only allow ASCII, check and return invalid passphrase otherwise (local language keyboards mess things up). - - ask if the user wants to delete the old plaintext ed25519_master_id_secret_key and only remain with the encrypted copy. [Y/n], Yes by default. Delete it in a secure way if yes, obviously. - - save the encrypted key in the working directory where the command is run or with an argument --out /path/to/dir. Print a message confirming this: ed25519_master_id_secret_key was successfully encrypted and saved in $location ($location = either working directory where the command was run either the location specified separately with --out argument included in the --encryptkey command).
3. tor --decryptkey <ed25519_master_id_secret_key file> or /path/to/<key file> - - ask for confirmation: Are you sure you wan to decrypt your ed25519_master_id_secret_key? This will save the master ID key in plaintext, not protected by a passphrase. [Y/n], Yes by default. - - ask for the passphrase and decrypt. - - ask if the user wants to delete the old encrypted ed25519_master_id_secret_key and only remain with the unencrypted copy of it. [y/N], No by default. - - save the plaintext key in the working directory where the command is run or with an argument --out /path/to/dir. Print a message confirming this: ed25519_master_id_secret_key was successfully decrypted permanently and saved in $location ($location = either working directory where the command was run either the location specified separately with --out argument included in the --decryptkey command).
4. tor --newpasskey <ed25519_master_id_secret_key file> or /path/to/<key file> - - ask for the current passphrase. - - ask for the new passphrase 2 times. Only allow ASCII, check and return invalid passphrase otherwise (local language keyboards mess things up). - - update the ed25519_master_id_secret_key file with then new passphrase in place, don't save a separate new copy with this command. - - print a message confirming that the passphrase has been successfully and permanently changed.
5. tor --gensignkey <ed25519_master_id_secret_key file> or /path/to/<key file> - - ask for the passphrase if key file is encrypted. - - generate new ed25519_signing_secret_key and ed25519_signing_cert valid for 30 days unless specified otherwise in torrc. If there is a SigningKeyLifetime in torrc, read it and ask the user: The signing key lifetime in your torrc is n days/weeks/months, use this validity period? [Y/n], yes default. If user enters N, ask the user to manually enter the lifetime. - - save the output files in the working directory where the command is run or with an argument --out /path/to/dir. Print a message which explains that these files need to be moved in Tor's $datadirectory/keys. - - Print a message confirming this: ed25519_master_id_public_key, ed25519_signing_secret_key and ed25519_signing_cert with a validity period of $validity have been successfully created and saved in $location. Please move them to your Tor $datadirectory/keys folder and make sure Tor can access/read them. ($validity = SigningKeyLifetime in torrc or lifetime manually specified by the user or 30 days, the default if there is no torrc setting for this and no input from the user; $location = either working directory where the command was run either the location specified separately with --out argument included in the --gensignkey command).
6. tor --getpubkey <ed25519_master_id_secret_key file> or /path/to/<key file> - - ask for the passphrase if key file is encrypted - - compute the ed2551_master_id_public_key - - save the output file ed25519_master_id_public_key in working directory where the command is run or with an argument --out /path/to/dir. - - Print a message confirming that the ed25519_master_id_public_key has been saved in $location ($location = either working directory where the command was run either the location specified separately with - --out argument included in the --getpubkey command).
- ------------------------------------------------------ Permissions - a thing many users don't know/care about: - - If Tor cannot complete a command from above because it does not have permission to read the ed25519_master_id_secret_key print an error message and instruct the user that he should run the tor command as the rightful user or change permissions over the ed25519_master_id_secret_key file. - - If Tor cannot save the output files resulted from a command because it does not have write permissions in the working directory where the command is run or the destination directory manually specified with - --out argument included in command print an error message and instruct the user that he should either run the tor command as the rightful user or select a destination directory where current user has write permissions. - - Always save the output files when using above commands with the correct permissions (I guess these are chmod 0600?). - - Can Tor work if the files in $datadirectory/keys folder are owned by other user than the one running Tor? If not, we should try to detect what user runs Tor on that system (not everywhere Tor runs under debian-tor, in FreeBSD it is _tor for example) and change the owner of the files after they are generated and saved.
On Mon, Aug 3, 2015 at 6:55 PM, s7r s7r@sky-ip.org wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256
Hi,
Tor 0.2.7.x will support Ed25519 router identities along with the traditional 1024-bit RSA ones which will be used simultaneously for some time, until we will completely deprecate RSA router identities.
Hi, s7r!
This is an impressive writeup; thanks!
One thing that makes it hard for me to follow this document is that I'm not sure which parts are describing how things work _now_, and which parts describe how things _should_ work. Would it be possible to split up descriptions of current/revised behavior, and mark each?
many thanks,
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256
On 8/4/2015 5:42 PM, Nick Mathewson wrote:
Hi, s7r!
This is an impressive writeup; thanks!
One thing that makes it hard for me to follow this document is that I'm not sure which parts are describing how things work _now_, and which parts describe how things _should_ work. Would it be possible to split up descriptions of current/revised behavior, and mark each?
many thanks,
Hi Nick,
My pleasure. Done and written. It looks long but I hope it reads fast and makes sense. I know it's a PITA to fix things by reading about them on email, so do let me know what are the priorities and how should I open tickets on trac to be cleaner and easier for you (a single big ticket? more tickets for each case to the master ticket where we discuss the Ed25519 FAQ?).
I have assigned status flags to cases as OK (already doing what it should), NOT OK (mostly doing what it should, needs just little adjustments), DEFECT (really broken behavior).
One thing which I've noticed is that Tor thinks very highly about ed25519_master_id_public_key, it treats it like the identity key itself (and I don't know why, since it can compute this from ed25519_signing_cert?) - Tor will go as far as generating a new identity if we take ed25519_master_id_public_key away. See each case documented below.
Note: I was only able to use restart (not reload signal [HUP]) because it doesn't work for me after playing with ed25519 key files ( reported in https://trac.torproject.org/projects/tor/ticket/16703 )
1. No ed25519* files at all in $datadirectory/keys:
What Tor should do: - - We assume it's a new relay or a relay which just upgraded to a Tor that supports Ed25519 router identities. Generate a new Ed25519 identity unencrypted (ed25519_master_id_secret_key, ed25519_master_id_public_key, ed25519_signing_secret_key and ed25519_signing_cert with a validity for 30 days unless specified otherwise in torrc) and publish a descriptor using that, proceed as normal after.
What Tor currently does: - - Generates a new ed25519 identity unencrypted (ed25519_master_id_secret_key, ed25519_master_id_public_key, ed25519_signing_secret_key and ed25519_signing_cert with a validity of 30 days, unless specified otherwise in torrc).
Status: OK - --------------------------------------------------------------------
2. If ed25519_master_id_secret_key is missing or encrypted and ed25519_master_id_public_key exists in $datadirectory/keys (without ed25519_signing_secret_key and ed25519_signing_cert):
What Tor should do: - - Don't start at all, something is wrong. Print warnings that user action is required (either generate valid ed25519_signing_secret_key and ed25519_signing_cert either move the ed25519_master_id_secret_key unencrypted in $datadirectory/keys so Tor can do it by itself).
What Tor currently does: Doesn't start: Aug 04 10:53:36.000 [warn] Missing identity key Aug 04 10:53:36.000 [err] do_main_loop(): Bug: Error initializing keys; exiting (on Tor 0.2.7.2-alpha-dev )
Status: NOT OK. Behavior is correct. We should just change the log message with a better, more explicit one and remove 'Bug' (it isn't a bug). - --------------------------------------------------------------------
3. If ed25519_master_id_secret_key unencrypted and ed25519_master_id_public_key both exist in $datadirectory/keys and ed25519_signing_secret_key, ed25519_signing_cert are missing:
What Tor should do: - - Assume it's a reinstalled relay with backups of the identity. Generate valid ed25519_signing_secret_key and ed25519_signing_cert (with a validity period of 30 days unless specified otherwise in torrc) and start, proceed as normal after.
What Tor currently does: - - Generates valid ed25519_signing_secret_key and ed25519_signing_cert (with a validity period of 30 days unless specified otherwise in torrc), starts and proceeds as normal.
Status: OK - --------------------------------------------------------------------
4. If only ed25519_master_id_secret_key unencrypted exists in $datadirectory/keys (ed25519_master_id_public_key, ed25519_signing_secret_key and ed25519_signing_cert are missing):
What Tor should do: - - We assume the relay has just been restored and the Ed25519 identity was backed up. Automatically generate ed25519_master_id_public_key, ed25519_signing_secret_key and ed25519_signing_cert with a validity period of 30 days (unless otherwise specified in torrc).
What Tor currently does: - - It starts and generates valid ed25519_signing_secret_key and ed25519_signing_cert first time but DOES NOT generate ed25519_master_id_public_key. Starts fine first time.
After a service tor restart, it dies: Aug 04 11:45:46.000 [warn] The signing cert we have was not signed with the master key we loaded! Aug 04 11:45:46.000 [err] do_main_loop(): Bug: Error initializing keys; exiting (on Tor 0.2.7.2-alpha-dev )
Here is what Tor is actually doing: - - It sees the valid unencrypted ed25519_master_id_secret_key file and generates ed25519_signing_secret_key and ed25519_signing_cert because these files are missing. OK. but it doesn't check or care about ed25519_master_id_public_key which is wrong. Starts. - - After first restart, because it sees the ed25519_signing_secret_key and ed25519_signing_cert proceeds further to validation and notices that ed25519_master_id_public_key is missing. Instead of computing it from the ed25519_signing_cert, it generates an entire new identity (ed25519_master_id_secret_key file [yes, it will overwrite the old existent file and change the identity forever] and ed25519_master_id_public_key file). - - Now it insanely checks the old ed25519_signing_secret_key and ed25519_signing_cert with the new generated ed25519_master_id_secret_key and ed25519_master_id_public_key in an endless loop - obviously they cannot validate, so it dies like this: [warn] The signing cert we have was not signed with the master key we loaded!
No matter how many restarts after this, it will still say: [warn] The signing cert we have was not signed with the master key we loaded!
Only way to start Tor now is to delete manually ed25519_signing_secret_key and ed25519_signing_cert from $datadirectory/keys (Tor will generate valid new ones for the correct new unrequested identity) or delete the new unrequested ed25519_master_id_secret_key and ed25519_master_id_public_key and bring back the old ed25519_master_id_public_key which validates with the ed25519_signing_secret_key and ed25519_signing_cert.
Status: DEFECT. Order of operations is not logic. Tor should check for existence of files, generate what is missing and can be generated, validate, die or proceed. - --------------------------------------------------------------------
5. If only ed25519_master_id_secret_key_encrypted exists in $datadirectory/keys (ed25519_master_id_public_key, ed25519_signing_secret_key and ed25519_signing_cert are missing):
What Tor should do: - - We assume the relay identity existed previously and ed25519_master_id_secret_key_encrypted was restored form a backup. Do not start Tor at all and print explicit warnings that action is required - Tor cannot decrypt the ed25519_master_id_secret_key_encrypted file. Generate ed25519_master_id_public_key, ed25519_signing_secret_key and ed25519_signing_cert or decrypt the ed25519_master_id_secret_key_encrypted in $datadirectory/keys so Tor can do it by itself.
What Tor currently does: - - Ignores the ed25519_master_id_secret_key_encrypted and generates entire new identity + signing key and signing cert and starts as normal.
Status: DEFECT. If there is an ed25519_master_id_secret_key_encrypted in $datadirectory/keys we never generate automatically another new identity, instead we push the user to decrypt it or manually generate valid ed25519_signing_secret_key and ed25519_signing_cert. - --------------------------------------------------------------------
6. If ed25519_master_id_secret_key_encrypted and ed25519_master_id_public_key both exist in $datadirectory/keys (ed25519_signing_secret_key and ed25519_signing_cert are missing):
What Tor should do: - - We assume the relay identity existed previously and ed25519_master_id_secret_key_encrypted was restored form a backup. Do not start Tor at all and print explicit warnings that action is required - Tor cannot decrypt the ed25519_master_id_secret_key_encrypted file. Generate ed25519_master_id_public_key, ed25519_signing_secret_key and ed25519_signing_cert or decrypt the ed25519_master_id_secret_key_encrypted in $datadirectory/keys so Tor can do it by itself.
What Tor currently does: - - It does not start at all because it doesn't have a valid ed25519_signing_secret_key and ed25519_signing_cert: Aug 04 11:55:20.000 [warn] Missing identity key Aug 04 11:55:20.000 [err] do_main_loop(): Bug: Error initializing keys; exiting (on Tor 0.2.7.2-alpha-dev )
Status: NOT OK. Behavior is correct. We should just change the log message with a better, more explicit one and remove 'Bug' (it isn't a bug). - --------------------------------------------------------------------
7. If ed25519_master_id_secret_key unencrypted and valid ed25519_signing_secret_key and ed25519_signing_cert exist in $datadirecotry/keys (ed25519_master_id_public_key is missing):
What Tor should do: - - Compute ed25519_master_id_public_key from ed25519_signing_cert (in order to keep a logic), validate and start. Proceed as normal after (regenerate automatically after expiration if master key unencrypted still available).
What Tor currently does: - - Generates a new identity (ed25519_master_id_secret_key [yes, it overwrites the old identity key file] and ed25519_master_id_public_key) and tries to validate the existent ed25519_signing_secret_key and ed25519_signing_cert with the new unrequested identity - obviously it fails and dies (see Case 4 above):
Aug 04 10:57:43.000 [warn] The signing cert we have was not signed with the master key we loaded! Aug 04 10:57:43.000 [err] do_main_loop(): Bug: Error initializing keys; exiting (on Tor 0.2.7.2-alpha-dev )
It becomes an endless loop, and won't ever start, always tries to validate the old cert and key with the new identity. Only way to start is to delete signing key and cert or delete the new unrequested identity and bring back the old ed25519_master_id_public_key.
Status: DEFECT. Order of operations is not logic. Tor should check for existence of files, generate what is missing and can be generated, validate, die or proceed. - --------------------------------------------------------------------
8. If only ed25519_master_id_public_key and valid ed25519_signing_secret_key and valid ed25519_signing_cert exist in $datadirectory/keys:
What Tor should do: - - Start normally, all good.
What Tor currently does: - - Starts normally, all good.
Status: OK - --------------------------------------------------------------------
9. If only valid ed25519_signing_secret_key and valid ed25519_signing_cert exist in $datadirectory/keys (ed25519_master_id_public_key missing):
What Tor should do: - - Compute ed25519_master_id_public_key from ed25519_signing_cert (in order to keep a logic), validate and start. Proceed as normal after (regenerate automatically after expiration if master key unencrypted is still available).
What Tor currently does: - - Generates a new identity (ed25519_master_id_secret_key [yes, it overwrites the old identity key file] and ed25519_master_id_public_key) and tries to validate the existent ed25519_signing_secret_key and ed25519_signing_cert with the new unrequested identity - obviously it fails and dies (see Cases 4 and 7 from above).
Status: DEFECT. Order of operations is not logic. Tor should check for existence of files, generate what is missing and can be generated, validate, die or proceed. - --------------------------------------------------------------------
10. If we have in $datadirectory/keys an ed25519_master_id_secret_key and ed25519_master_id_public_key which doesn't belong to the secret key:
What Tor should do: - - Don't start at all - obviously something is wrong. Print messages in log file that user action is required, we need either the old identity ed25519_master_id_secret_key unencrypted if it was backed up, either a whole new identity (so delete all edd25519* files from $datadirectory/keys and start fresh) either a valid ed25519_signing_secret_key and valid ed25519_signing_cert).
What Tor currently does: - - Doesn't start at all: Aug 04 12:10:25.000 [warn] /var/lib/tor/keys/ed25519_master_id_public_key does not match /var/lib/tor/keys/ed25519_master_id_secret_key! Aug 04 12:10:25.000 [warn] Missing identity key Aug 04 12:10:25.000 [err] do_main_loop(): Bug: Error initializing keys; exiting (on Tor 0.2.7.2-alpha-dev )
And it also doesn't generate a new unrequested identity.
Status: NOT OK. Behavior is correct, we just need to adjust the log messages. Not a Bug. - --------------------------------------------------------------------
11. Probably not a bug: key files have to be owned by the same user running Tor, or they cannot be read (chmod is 0600 for all of them):
Aug 04 11:54:09.000 [warn] Could not open "/var/lib/tor/keys/ed25519_signing_secret_key": Permission denied Aug 04 11:54:09.000 [warn] Could not open "/var/lib/tor/keys/ed25519_master_id_public_key": Permission denied Aug 04 11:54:09.000 [warn] Unable to read /var/lib/tor/keys/ed25519_master_id_public_key: Permission denied
####### THE END #######
On Tue, Aug 4, 2015 at 8:24 PM, s7r s7r@sky-ip.org wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256
On 8/4/2015 5:42 PM, Nick Mathewson wrote:
Hi, s7r!
This is an impressive writeup; thanks!
One thing that makes it hard for me to follow this document is that I'm not sure which parts are describing how things work _now_, and which parts describe how things _should_ work. Would it be possible to split up descriptions of current/revised behavior, and mark each?
many thanks,
Hi Nick,
My pleasure. Done and written. It looks long but I hope it reads fast and makes sense. I know it's a PITA to fix things by reading about them on email, so do let me know what are the priorities and how should I open tickets on trac to be cleaner and easier for you (a single big ticket? more tickets for each case to the master ticket where we discuss the Ed25519 FAQ?).
Thanks; this is incredibly helpful!
I've started a branch to do a test case to demonstrate all these bugs ; it's called "ed25519_keygen" in my public repository. It also adds a couple more features to '--keygen'. It does cases 2...4 so far; I want to make it cover 5..10.
Once I've got it demonstrating all the cases, I'll try to clean it up and start hammering down the bugs.
best wishes,
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256
That is great.
One thing which could help identifying the bugs faster: Tor gives too much importance to ed25519_master_id_public_key (if it doesn't see this file, it will generate a new unrequested identity regardless if it has a valid signing_cert and signing_secret_key and/or even if it has the ed25519_master_id_secret_key unencrypted).
These commands would be useful as well: --getpubkey; --encryptkey; - --decryptkey; --newpass; --gensignkey.
On 8/6/2015 4:14 AM, Nick Mathewson wrote:
On Tue, Aug 4, 2015 at 8:24 PM, s7r s7r@sky-ip.org wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256
On 8/4/2015 5:42 PM, Nick Mathewson wrote:
Hi, s7r!
This is an impressive writeup; thanks!
One thing that makes it hard for me to follow this document is that I'm not sure which parts are describing how things work _now_, and which parts describe how things _should_ work. Would it be possible to split up descriptions of current/revised behavior, and mark each?
many thanks,
Hi Nick,
My pleasure. Done and written. It looks long but I hope it reads fast and makes sense. I know it's a PITA to fix things by reading about them on email, so do let me know what are the priorities and how should I open tickets on trac to be cleaner and easier for you (a single big ticket? more tickets for each case to the master ticket where we discuss the Ed25519 FAQ?).
Thanks; this is incredibly helpful!
I've started a branch to do a test case to demonstrate all these bugs ; it's called "ed25519_keygen" in my public repository. It also adds a couple more features to '--keygen'. It does cases 2...4 so far; I want to make it cover 5..10.
Once I've got it demonstrating all the cases, I'll try to clean it up and start hammering down the bugs.
best wishes,
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256
Thanks; this is incredibly helpful!
I've started a branch to do a test case to demonstrate all these bugs ; it's called "ed25519_keygen" in my public repository. It also adds a couple more features to '--keygen'. It does cases 2...4 so far; I want to make it cover 5..10.
Once I've got it demonstrating all the cases, I'll try to clean it up and start hammering down the bugs.
Finished tests on branch ed25519_keygen, commit: b71dafcb3358fac7
Things are indeed much much better. I am sending you the new (hopefully last) cases which need adjustments. Note that the case numbers do not match the ones in my previous email, since I am completely removing the cases where now Tor does what it should.
We have cases with two possible status codes: LOG MESSAGE - where we need to change the log message and not die with [err] do_main_loop(): Bug: - I am also trying to provide suggestions for log messages, but we certainly need to improve them.
DEFECT - where we also need to adjust the behavior (only one defect fortunately).
- ----------------------------------------------------------
1. If ed25519_master_id_secret_key is missing or encrypted and we only have ed25519_master_id_public_key available in $datadirectory/keys (no ed25519_signing_cert and 25519_signing_secret_key):
What Tor should do: - - Don't start at all and print log messages requiring user action.
What Tor currently does: - - Doesn't start at all: Aug 06 15:15:01.000 [warn] We needed to load a secret key from /root/torsrc/data/keys/ed25519_master_id_secret_key, but couldn't find it. Aug 06 15:15:01.000 [warn] Missing identity key Aug 06 15:15:01.000 [err] do_main_loop(): Bug: Error initializing keys; exiting (on Tor 0.2.7.2-alpha-dev b71dafcb3358fac7)
The behavior is correct, it detects the file ed25519_master_id_public_key, it doesn't see ed25519_signing_cert and ed25519_signing_secret_key so it tries to look for ed25519_master_id_secret_key (unencrypted) for generating them.
STATUS: LOG MESSAGE -> Change the log message, it is not a bug: Failed to start: Missing a valid signing key and certificate. Please generate manually a valid signing key and certificate and move them to $datadirectory/keys or move the plaintext (unencrypted) master key in $datadirectory/keys so Tor can do this automatically. - ----------------------------------------------------------
2. If only ed25519_master_id_secret_key_encrypted exists in $datadirectory/keys (ed25519_master_id_public_key, ed25519_signing_secret_key and ed25519_signing_cert are missing):
What Tor should do: - - We assume the relay identity existed previously and ed25519_master_id_secret_key_encrypted was restored form a backup. Do not start Tor at all and print explicit warnings that action is required - Tor cannot decrypt the ed25519_master_id_secret_key_encrypted file. Generate ed25519_master_id_public_key, ed25519_signing_secret_key and ed25519_signing_cert or decrypt the ed25519_master_id_secret_key_encrypted in $datadirectory/keys so Tor can do it by itself.
What Tor currently does: - - Doesn't start: Aug 06 15:03:25.000 [warn] Found an encrypted secret key, but not public key file /root/torsrc/data/keys/ed25519_master_id_public_key! Aug 06 15:03:25.000 [warn] Missing identity key Aug 06 15:03:25.000 [err] do_main_loop(): Bug: Error initializing keys; exiting (on Tor 0.2.7.2-alpha-dev b71dafcb3358fac7)
ed25519_master_id_public_key missing is not the issue here, the issue is actually not finding valid ed25519_signing_cert and ed25519_signing_secret_key.
STATUS: LOG MESSAGE -> Change the log message, it is not a bug: Failed to start: The master key in $datadirectory/keys is encrypted and we don't have a valid signing key and certificate. Please generate manually a valid signing key and certificate and move them to $datadirectory/keys or move the plaintext (unencrypted) master key in $datadirectory/keys so Tor can do this automatically. - ----------------------------------------------------------
3. If ed25519_master_id_secret_key_encrypted and ed25519_master_id_public_key both exist in $datadirectory/keys (ed25519_signing_secret_key and ed25519_signing_cert are missing):
What Tor should do: - - We assume the relay identity existed previously and ed25519_master_id_secret_key_encrypted was restored form a backup. Do not start Tor at all and print explicit warnings that action is required - Tor cannot decrypt the ed25519_master_id_secret_key_encrypted file. Generate ed25519_master_id_public_key, ed25519_signing_secret_key and ed25519_signing_cert or decrypt the ed25519_master_id_secret_key_encrypted in $datadirectory/keys so Tor can do it by itself.
STATUS: LOG MESSAGE -> Change the log message, it is not a bug: Failed to start: The master key in $datadirectory/keys is encrypted and we don't have a valid signing key and certificate. Please generate manually a valid signing key and certificate and move them to $datadirectory/keys or move the plaintext (unencrypted) master key in $datadirectory/keys so Tor can do this automatically. - ----------------------------------------------------------
4. If only valid ed25519_signing_secret_key and valid ed25519_signing_cert exist in $datadirectory/keys (ed25519_master_id_secret_key and ed25519_master_id_public_key missing):
What Tor should do: - - Compute ed25519_master_id_public_key from ed25519_signing_secret_key, validate and start. Proceed as normal after (regenerate automatically after expiration if master key unencrypted is still available).
What Tor currently does: - - It starts (this is good), but does not compute and save to disk ed25519_master_id_public_key from ed25519_signing_cert:
Aug 06 15:33:21.000 [warn] No key found in /root/torsrc/data/keys/ed25519_master_id_secret_key or /root/torsrc/data/keys/ed25519_master_id_public_key. Aug 06 15:33:21.000 [warn] Master public key was absent; inferring from public key in signing certificate
STATUS: DEFECT -> Tor should also save to disk in $datadirectory/keys ed25519_master_id_public_key (computed from ed25519_signing_cert) not just start, in order to avoid future warnings for the same reason.
We have to compute and save to disk *ed25519_master_id_public_key of ed25519_master_id_secret_key*, not the public key of ed25519_signing_secret_key. - ----------------------------------------------------------
5. If we have in $datadirectory/keys an ed25519_master_id_secret_key and ed25519_master_id_public_key which doesn't belong to the secret key:
What Tor should do: - - Don't start at all - something is obviously wrong and user has to clean it up.
What Tor currently does: - - Doesn't start: Aug 06 15:45:36.000 [warn] /root/torsrc/data/keys/ed25519_master_id_public_key does not match /root/torsrc/data/keys/ed25519_master_id_secret_key! Aug 06 15:45:36.000 [warn] Missing identity key Aug 06 15:45:36.000 [err] do_main_loop(): Bug: Error initializing keys; exiting (on Tor 0.2.7.2-alpha-dev b71dafcb3358fac7)
STATUS: LOG MESSAGE -> Change the log message, it is not a bug: Failed to start: The ed25519_master_id_public_key and ed25519_master_id_secret_key do not match. If you are trying to restore this relay from a backup, make sure you didn't accidentally mix the key files. If you are sure the ed25519_master_id_secret_key is the right identity file for this relay, delete ed25519_master_id_public_key. - ----------------------------------------------------------
6. ed25519_signing_cert and ed25519_signing_secret_key are valid but do not validate against ed25519_master_id_public_key:
What Tor should do: - - Don't start at all - something is wrong.
What Tor currently does: - - Doesn't start: Aug 06 15:54:20.000 [warn] The signing cert we have was not signed with the master key we loaded! Aug 06 15:54:20.000 [err] do_main_loop(): Bug: Error initializing keys; exiting (on Tor 0.2.7.2-alpha-dev b71dafcb3358fac7)
STATUS: LOG MESSAGE -> Change the log message, it is not a bug: Change the log message, not a bug: Failed to start: The certificate was not signed with the master key we loaded! Maybe we have in $datadirectory/keys the signing key and certificate generated with another relay's master key, please double check! - ----------------------------------------------------------
7. Only expired ed25519_signing_cert and ed25519_signing_secret_key:
What Tor should do: - - Don't start.
What Tor currently does: - - Doesn't start: Aug 06 15:48:56.000 [warn] No key found in /root/torsrc/data/keys/ed25519_master_id_secret_key or /root/torsrc/data/keys/ed25519_master_id_public_key. Aug 06 15:48:56.000 [warn] Missing identity key Aug 06 15:48:56.000 [err] do_main_loop(): Bug: Error initializing keys; exiting (on Tor 0.2.7.2-alpha-dev b71dafcb3358fac7)
The behavior is correct, Tor first checks the certificate's expiration date, sees that it is expired and proceeds looking for ed25519_master_id_secret_key unencrypted to generate fresh new cert and key (very good).
STATUS: LOG MESSAGE -> Change the log message, it is not a bug: Failed to start: the signing key and certificate are expired. Generate a fresh valid signing key and certificate and move them to $datadirectory/keys or move the ed25519_master_id_secret_key unencrypted to $datadirectory/keys so Tor can do this automatically.
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256
I am also sending the steps I imagine Tor should take when started as a relay. Apologies if I am missing something obvious.
They are expressed as simple as possible, Tor's interpretation is way more complex than this, but I think/hope this might help with ordering and architecture of the code.
The ed25519_keygen branch behaves _very_ _good_ (report in my previous email), so I am sending this only for a fast verification. It is easier to spot if the code jumps over a step if we have logic in ordering:
[0] If there are no ed25519* files at all in $datadirectory/keys, generate a fresh new identity, signing key and cert, everything needed (valid for 30 days unless otherwise specified in torrc) and use those.
1. Check if file ed25519_master_id_secret_key exists: - - Do nothing if it doesn't exist. Proceed.
2. Check if file ed25519_master_id_secret_key_encrypted exists: - - Do nothing if it doesn't exist. Proceed.
3. Check if file ed25519_master_id_public_key exists: - - If it doesn't exist, try to generate it from ed25519_master_id_secret_key; - - If ed25519_master_id_secret_key is missing or _encrypted, do nothing. Proceed.
4. Validate ed25519_master_id_public_key against ed25519_master_id_secret_key: - - Do nothing if ed25519_master_id_secret_key is missing. Proceed; - - Do nothing if ed25519_master_id_secret_key_encrypted exists. Proceed; - - Skip this step / do nothing if ed25519_master_id_public_key doesn't exist and couldn't be generated at step 3. Proceed.
5. Check if file ed25519_signing_cert exists: - - If it doesn't exist, try to generate a valid one from ed25519_master_id_secret_key with the SigningKeyLifetime in torrc (30 days unless specified otherwise). Generate an ed25519_signing_secret_key along with it to use together. Proceed; - - Die if it is missing and cannot be generated because ed25519_master_id_secret_key is missing or _encrypted.
6. Check if ed25519_signing_cert is expired: - - If it is expired, try to generate a valid one from ed25519_master_id_secret_key with the SigningKeyLifetime in torrc (30 days unless specified otherwise). Generate an ed25519_signing_secret_key along with it to use together. Proceed; - - Die if it is expired and a new one cannot be generated because ed25519_master_id_secret_key is missing or _encrypted.
7. Validate ed25519_signing_cert against ed25519_master_id_public_key: - - Do nothing if ed25519_master_id_public_key is missing. Proceed; - - Die if it doesn't match.
8. Check if file ed25519_signing_secret_key exists: - - Die if it doesn't exist.
9. Check if ed25519_signing_secret_key matches with ed25519_signing_cert: - - Die if it doesn't match.
10. Finally, if we didn't complete step 7 because ed25519_master_id_public_key was missing: - - Compute and save to disk file ed25519_master_id_secret_key from ed25519_signing_cert;
Use that to create descriptors, start the Tor magic, publish descriptors, relay traffic, save the world.
On 8/7/2015 12:18 AM, s7r wrote:
Thanks; this is incredibly helpful!
I've started a branch to do a test case to demonstrate all these bugs ; it's called "ed25519_keygen" in my public repository. It also adds a couple more features to '--keygen'. It does cases 2...4 so far; I want to make it cover 5..10.
Once I've got it demonstrating all the cases, I'll try to clean it up and start hammering down the bugs.
Finished tests on branch ed25519_keygen, commit: b71dafcb3358fac7
Things are indeed much much better. I am sending you the new (hopefully last) cases which need adjustments. Note that the case numbers do not match the ones in my previous email, since I am completely removing the cases where now Tor does what it should.
On Thu, Aug 6, 2015 at 6:26 PM, s7r s7r@sky-ip.org wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256
I am also sending the steps I imagine Tor should take when started as a relay. Apologies if I am missing something obvious.
They are expressed as simple as possible, Tor's interpretation is way more complex than this, but I think/hope this might help with ordering and architecture of the code.
The ed25519_keygen branch behaves _very_ _good_ (report in my previous email), so I am sending this only for a fast verification. It is easier to spot if the code jumps over a step if we have logic in ordering:
[0] If there are no ed25519* files at all in $datadirectory/keys, generate a fresh new identity, signing key and cert, everything needed (valid for 30 days unless otherwise specified in torrc) and use those.
Almost. Here's what I think is going on:
1) Load the secret signing key signing certificate. If they are absent, or expired, or if --keygen was called, we'll need to generate a new one. If it's going to expire soon, we _want_ to generate a new one.
2) If we need or want to generate a new signing key, load the master ID secret key. Otherwise, don't try. If we try to load it and it's absent or encrypted, log a message. If we need to generate a new signing key then exit on error; otherwise just warn.
2b) If we fail to load the master ID secret key, and there were no other keys in the keys directory, then generate a master ID secret key and save it.
3) Load the master ID public key. If we loaded a secret key, and it doesn't match, log and quit. If it doesn't match the master ID public key in a certificate we loaded, log and quit. If we have the public key from one of those other sources and the master ID public key file is missing, recreate it.
4) At this point, if we need to generate a new signing key and cert, and we don't have a secret master ID key, exit.
5) If we have a have a secret master ID key, and we need or want to generate a new signing key and cert, do so, and save them.
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256
Hi,
Things look good in ed25519_keygen - git-018082ef88b688e2.
I can confirm the last defect was fixed (now it saves to disk ed25519_master_id_public_key if it only has ed25519_signing_cert - valid and ed25519_signing_secret_key).
Log messages are fine, no longer saying 'bug' as far as I was able to try it.
I see that the operations will have the same result as what I've suggested, so it should be ok.
Can you please ensure that they are called on all signals? Currently, in the last commit, these operations are not called when we send a reload (HUP) signal (ed25519_master_id_public_key is not saved to disk if missing, etc.).
On 8/10/2015 6:04 PM, Nick Mathewson wrote:
Almost. Here's what I think is going on:
- Load the secret signing key signing certificate. If they are
absent, or expired, or if --keygen was called, we'll need to generate a new one. If it's going to expire soon, we _want_ to generate a new one.
- If we need or want to generate a new signing key, load the
master ID secret key. Otherwise, don't try. If we try to load it and it's absent or encrypted, log a message. If we need to generate a new signing key then exit on error; otherwise just warn.
2b) If we fail to load the master ID secret key, and there were no other keys in the keys directory, then generate a master ID secret key and save it.
- Load the master ID public key. If we loaded a secret key, and
it doesn't match, log and quit. If it doesn't match the master ID public key in a certificate we loaded, log and quit. If we have the public key from one of those other sources and the master ID public key file is missing, recreate it.
- At this point, if we need to generate a new signing key and
cert, and we don't have a secret master ID key, exit.
- If we have a have a secret master ID key, and we need or want
to generate a new signing key and cert, do so, and save them.
-
Nick Mathewson -
s7r