Hi,
The Warning: DO NOT USE MY BRANCH YET, IT HAS HAD MINIMAL TESTING AND REVIEW. IT IS NOT DONE. IT WILL BROADCAST YOUR SECRETS TO THE NSA'S ORBITAL SPACE STATION.
Trac Ticket: https://trac.torproject.org/projects/tor/ticket/6411 Branch: https://github.com/Yawning/tor/compare/feature6411
With that out of the way, I've been poking at implementing a way to add hidden services at runtime via the control port. I'm calling this "ephemeral hidden services" because the HS state is not written out to disk, and they disappear when the tor damon is shutdown.
Yes, this means that if you run "kittensomgmewmew.onion" and are scared of the NSA's persistent attempts to extract your hidden service key, via ultrasonic laser beamed from their satellites, you could run your tor instance entirely in a ram disk, and load the HS key manually each time from a USB token you wear around your neck.
What I've come up with looks like this:
ADD_EPH_HS - Adds a new ephemeral hidden service.
The syntax is:
"ADD_EPH_HS" keytype:keyblob SP 1*(port SP addrPort) CRLF keytype = "RSA1024", "NEW" keyblob = opaque ascii serialized key material, or an algorithm specifier, in the case of "keytype" == "NEW". port = the "VIRTPORT" addrPort = the "TARGET"
A hidden service is created using the key and list of port/targets, that will persist till configuration reload or the termination of the tor process.
The currently supported keytypes and keyblobs are:
"RSA1024" - The keyblob contains the Base64 encoding of the PKCS#1 layout DER representation of a 1024 bit RSA private key.
"NEW" - The tor instance should generate a key with the keytype specified in the keyblob parameter. A type of "BEST" will generate a key using the best supported algorithm.
Returns:
"250-ServiceID=serviceID" - The hidden service's onion address without the ".onion" suffix.
"250-PrivateKey=keytype:keyblob" - The newly generated hidden service's private key, if a keytype of "NEW" was specified.
"250 OK" - Everything went well.
Proper key management, is left as an exercise for the application invoking the controller command.
DEL_EPH_HS - Removes an existing ephemeral hidden service.
The syntax is:
"DEL_EPH_HS" serviceID CRLF serviceID = the hidden service identifier, without the trailing ".onion" suffix
The hidden service specified by the given service ID is removed, and the intro points are closed. Existing connections are left as is, and it is up to the application to close them if it wishes to do so.
Examples:
* Adding a new ephemeral hidden service, with a pre-generated RSA key: > ADD_EPH_HS RSA1024:[Base64 blob] 80 127.0.0.1:8080 < 250-ServiceID=blahblahblahblah < 250 OK
Hidden service is reachable at blahblahblahblah.onion:80
* Adding a new ephemeral hidden service, tor generates a new RSA key: > ADD_EPH_HS NEW:RSA1024 22 127.0.0.1:22 < 250-ServiceID=hogehogehogehoge < 250-PrivateKey=RSA1024:[Base64 blob] < 250 OK
Hidden service is reachable at hogehogehogehoge.onion:22. At a later date, the value passed back as "PrivateKey" can be used to re-create the hidden service.
* Add a new ephemeral hidden service, tor picks the algorithm: > ADD_EPH_HS NEW:BEST 1245 192.168.1.5:23 < 250-ServiceID=hemohemohemohemo < 250-PrivateKey=Ed25519:[Base64 blob] < 250 OK
Hidden service is reachable at hemohemohemohemo:23. (nb: Ed25519 key support is not actually implemented yet, and is used as an example of a better algorithm.)
* Remove the hidden service configured in the first example: > DEL_EPH_HS blahblahblahblah < 250 OK
Note: ADD_EPH_HS can take more than one VIRTPORT/TARGET pair at once, the examples only show one pair for brevity.
This does not support `HiddenServiceAuthorizeClient` yet in any way (neither basic nor stealth). It may in the future, but coming up with a sane interface for it is way more pain than I am willing to self-inflict at the moment.
Design question that I want feedback on, primarily from control port library maintainers, and authors of applications that use the control port:
* meejah suggested that ephemeral hidden services should have their lifetime tied to the originating control port connection. I think this is a good idea, but this would be the only control port command that does this sort of thing.
My motivations for strongly considering implementing this is that, I don't think that adding a command to enumerate hidden services is a good idea (SketchORChat has no business in seeing what hidden services JankORFileSharing is using), and that this will automate cleanup in the event of an application crashing, or not removing it's hidden services before terminating.
* Does this interface expose enough functionality, in a way that is intuitive and easy to use? (The initial version I was planning on proposing made key generation entirely the application's problem, but nickm convinced me otherwise).
Questions, comments, feedback appreciated,
On 2/14/15 1:45 AM, Yawning Angel wrote:
Hi,
The Warning: DO NOT USE MY BRANCH YET, IT HAS HAD MINIMAL TESTING AND REVIEW. IT IS NOT DONE. IT WILL BROADCAST YOUR SECRETS TO THE NSA'S ORBITAL SPACE STATION.
Trac Ticket: https://trac.torproject.org/projects/tor/ticket/6411 Branch: https://github.com/Yawning/tor/compare/feature6411
I'm fine with the proposal.
That's an important part of a set of tickets designed to enable the use of a Tor integration without ever touching the filesystem within third party application and application controllers (such as TxTorCon/ORbot) .
This complement the already closed #13865 (https://trac.torproject.org/projects/tor/ticket/6411) and the yet-to-be-discussed #14899 (Enable Tor to work without using filesystem for cached files https://trac.torproject.org/projects/tor/ticket/14899).
When #6411 will be integrated and #14899 will be implemented: - Tor AppArmor profile can be imporved by completely disabling filesystem read/write (when integrated with a third part app) - Third party App can fully use Tor by keeping all it's configuration directive, keys (for TorHS) and caches (for descriptors/consensus) in the application database
Once all of that will be possible, we'll be able to make a 100% clean Tor integration into GlobaLeaks (that's undergoing an architecture refactor to have a master/slave process).
Yawning Angel yawning@schwanenlied.me writes:
Cool! I added a quick try at using this in txtorcon, with an example; see the following branch:
https://github.com/meejah/txtorcon/tree/yawning-feature-6411
(Most of that example code would actually end up in the endpoint.listen() code and be hidden from most users of the library, too. And would be the default instead of "launch a new tor").
This does not support `HiddenServiceAuthorizeClient` yet in any way (neither basic nor stealth). It may in the future, but coming up with a sane interface for it is way more pain than I am willing to self-inflict at the moment.
What about a similar API: add + remove a client-key. And similarly, you could push a key yourself, or ask for a new one. The only difference would be including the onion-address. Like:
ADD_EPH_HS_ADD_CLIENT_AUTH kittenmeowae41f nickname0 stealth NEW:BEST ADD_EPH_HS_ADD_CLIENT_AUTH kittenmeowae41f nickname1 basic NEW:BEST
The return value would be the cookie (for basic) or both the cookie and the key-blob for stealth. Or "NEW:BEST" could be "RSA1024:....." like the other API?
For deleting you'd pass the onion ID and the nickname I guess. Like:
ADD_EPH_HS_DEL_CLIENT_AUTH kittenmeowae41f nickname1
- meejah suggested that ephemeral hidden services should have their lifetime tied to the originating control port connection. I think this is a good idea, but this would be the only control port command that does this sort of thing.
__OwningControllerProcess is similar, I guess, but instead ties to the whole lifetime of Tor (which sounds way easier to implement ;)
Questions, comments, feedback appreciated,
I think it's looking quite nice!
Hi Yawning, nice addition! As requested moving from irc to here.
One gotcha to think about is that ADD_EPH_HS is using a variable number of positional arguments. This will limit your ability to expand this command in the future with new arguments. Also, I'd suggest renaming addrPort to addrTarget (to avoid making this sound restricted to a simple port).
Other than that spec looks great. Looking forward to it! -Damian
On Fri, Feb 13, 2015 at 4:45 PM, Yawning Angel yawning@schwanenlied.me wrote:
Hi,
The Warning: DO NOT USE MY BRANCH YET, IT HAS HAD MINIMAL TESTING AND REVIEW. IT IS NOT DONE. IT WILL BROADCAST YOUR SECRETS TO THE NSA'S ORBITAL SPACE STATION.
Trac Ticket: https://trac.torproject.org/projects/tor/ticket/6411 Branch: https://github.com/Yawning/tor/compare/feature6411
With that out of the way, I've been poking at implementing a way to add hidden services at runtime via the control port. I'm calling this "ephemeral hidden services" because the HS state is not written out to disk, and they disappear when the tor damon is shutdown.
Yes, this means that if you run "kittensomgmewmew.onion" and are scared of the NSA's persistent attempts to extract your hidden service key, via ultrasonic laser beamed from their satellites, you could run your tor instance entirely in a ram disk, and load the HS key manually each time from a USB token you wear around your neck.
What I've come up with looks like this:
ADD_EPH_HS - Adds a new ephemeral hidden service.
The syntax is: "ADD_EPH_HS" keytype:keyblob SP 1*(port SP addrPort) CRLF keytype = "RSA1024", "NEW" keyblob = opaque ascii serialized key material, or an algorithm specifier, in the case of "keytype" == "NEW". port = the "VIRTPORT" addrPort = the "TARGET" A hidden service is created using the key and list of port/targets, that will persist till configuration reload or the termination of the tor process. The currently supported keytypes and keyblobs are: "RSA1024" - The keyblob contains the Base64 encoding of the PKCS#1 layout DER representation of a 1024 bit RSA private key. "NEW" - The tor instance should generate a key with the keytype specified in the keyblob parameter. A type of "BEST" will generate a key using the best supported algorithm. Returns: "250-ServiceID=serviceID" - The hidden service's onion address without the ".onion" suffix. "250-PrivateKey=keytype:keyblob" - The newly generated hidden service's private key, if a keytype of "NEW" was specified. "250 OK" - Everything went well. Proper key management, is left as an exercise for the application invoking the controller command.
DEL_EPH_HS - Removes an existing ephemeral hidden service.
The syntax is: "DEL_EPH_HS" serviceID CRLF serviceID = the hidden service identifier, without the trailing ".onion" suffix The hidden service specified by the given service ID is removed, and the intro points are closed. Existing connections are left as is, and it is up to the application to close them if it wishes to do so.
Examples:
Adding a new ephemeral hidden service, with a pre-generated RSA key:
ADD_EPH_HS RSA1024:[Base64 blob] 80 127.0.0.1:8080
< 250-ServiceID=blahblahblahblah < 250 OK
Hidden service is reachable at blahblahblahblah.onion:80
Adding a new ephemeral hidden service, tor generates a new RSA key:
ADD_EPH_HS NEW:RSA1024 22 127.0.0.1:22
< 250-ServiceID=hogehogehogehoge < 250-PrivateKey=RSA1024:[Base64 blob] < 250 OK
Hidden service is reachable at hogehogehogehoge.onion:22. At a later date, the value passed back as "PrivateKey" can be used to re-create the hidden service.
Add a new ephemeral hidden service, tor picks the algorithm:
ADD_EPH_HS NEW:BEST 1245 192.168.1.5:23
< 250-ServiceID=hemohemohemohemo < 250-PrivateKey=Ed25519:[Base64 blob] < 250 OK
Hidden service is reachable at hemohemohemohemo:23. (nb: Ed25519 key support is not actually implemented yet, and is used as an example of a better algorithm.)
Remove the hidden service configured in the first example:
DEL_EPH_HS blahblahblahblah
< 250 OK
Note: ADD_EPH_HS can take more than one VIRTPORT/TARGET pair at once, the examples only show one pair for brevity.
This does not support `HiddenServiceAuthorizeClient` yet in any way (neither basic nor stealth). It may in the future, but coming up with a sane interface for it is way more pain than I am willing to self-inflict at the moment.
Design question that I want feedback on, primarily from control port library maintainers, and authors of applications that use the control port:
meejah suggested that ephemeral hidden services should have their lifetime tied to the originating control port connection. I think this is a good idea, but this would be the only control port command that does this sort of thing.
My motivations for strongly considering implementing this is that, I don't think that adding a command to enumerate hidden services is a good idea (SketchORChat has no business in seeing what hidden services JankORFileSharing is using), and that this will automate cleanup in the event of an application crashing, or not removing it's hidden services before terminating.
Does this interface expose enough functionality, in a way that is intuitive and easy to use? (The initial version I was planning on proposing made key generation entirely the application's problem, but nickm convinced me otherwise).
Questions, comments, feedback appreciated,
-- Yawning Angel
tor-dev mailing list tor-dev@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev
On Sat, 14 Feb 2015 13:46:04 -0800 Damian Johnson atagar@torproject.org wrote:
One gotcha to think about is that ADD_EPH_HS is using a variable number of positional arguments. This will limit your ability to expand this command in the future with new arguments. Also, I'd suggest renaming addrPort to addrTarget (to avoid making this sound restricted to a simple port).
Hmm ok. Judging by the feedback, I'm thinking the following:
* Auth is still a low-ish priority, I want to get the basic ephemeral stuff done first, and I need to read up more on how it works, and how the code is structured, before I can promise things in this area.
* People seem to be ok with the tying ephemeral HSes to the originating control port (and if the only major argument against is "it's a bit weird, relative to other things, well, eph. HSes are weird, and this solves a bunch of problems). So this will be implemented as planned.
* ADD_EPH_HS syntax changed to something like:
"ADD_EPH_HS" keytype:keyblob 1*(SP "Port=" virtPort "," addrTarget) CRLF
So, basically, space separated instances of "Port=virtPort,addrTarget" entries, of which there must be at least one. This clearly indicates the argument type and should be future proof (and also has the side benefit of being easier for me to validate.
Thoughts?
On 14 Feb (00:45:24), Yawning Angel wrote:
Hey Yawning, great stuff btw! I have a questions below regarding meejah's comment and https://trac.torproject.org/projects/tor/ticket/6411#comment:32
"Ephemeral hidden services are tied to the control port connection that created them. This means, that when the control connection goes away, so does the hidden service intro point. Closing client connections is left as an exercise for the application."
[...]
Design question that I want feedback on, primarily from control port library maintainers, and authors of applications that use the control port:
meejah suggested that ephemeral hidden services should have their lifetime tied to the originating control port connection. I think this is a good idea, but this would be the only control port command that does this sort of thing.
My motivations for strongly considering implementing this is that, I don't think that adding a command to enumerate hidden services is a good idea (SketchORChat has no business in seeing what hidden services JankORFileSharing is using), and that this will automate cleanup in the event of an application crashing, or not removing it's hidden services before terminating.
Can you explain the rational for this? This email states in the "unofficial-tor-spec-patch":
A hidden service is created using the key and list of port/targets, that will persist till configuration reload or the termination of the tor process.
Now, an HS bound to a control connection might be a good idea, I'm not 100% sure but I can see issues with this. Let's say "ControlListenAddress" is used, this means a TCP socket and it can timeout if no activity, so if that happens, I loose my HS?
This also put quite a requirement on the user side to add an HS on its tor-ramdisk for instance but has to use a client that keeps the control connection opens for its lifetime?... How does that work with stem, it would have to keep the control connection open and the script using it can't quit else the socket gets closed by the OS?
Thanks! David
On Mon, 16 Feb 2015 10:17:51 -0500 David Goulet dgoulet@ev0ke.net wrote: [snip]
A hidden service is created using the key and list of port/targets, that will persist till configuration reload or the termination of the tor process.
Now, an HS bound to a control connection might be a good idea, I'm not 100% sure but I can see issues with this. Let's say "ControlListenAddress" is used, this means a TCP socket and it can timeout if no activity, so if that happens, I loose my HS?
That's correct, though unless tor or the controller library has code to stomp on long dormant connections (which a casual look says there isn't), this shouldn't happen, because TCP/IP in itself has no idle timeout (See RFC 1122 4.2.3.6 regarding keep alives, which would also not cause HS loss, since the OS will respond to the probe).
There may be certain broken middleboxes (loadbalancers etc) that stomp on long idle TCP connections, but anyone that is running a control port connection through such a thing, and sending RSA keying material in the clear, probably has bigger things to worry about.
This also put quite a requirement on the user side to add an HS on its tor-ramdisk for instance but has to use a client that keeps the control connection opens for its lifetime?... How does that work with stem, it would have to keep the control connection open and the script using it can't quit else the socket gets closed by the OS?
Yup, I don't see "you need to leave stem running" as being all that bad, since this is mostly targeted at managed applications (chat, filesharing, global leaks, etc).
If someone has a suggestion for an alternative interface that can handle applications crashing (possibly before they persist the list of HSes they need to clean up), applications that are just poorly written (and not cleaning up all the ephemeral HSes), and (optionally, though lacking this would be a reduction in features) limiting cross application HS enumeration, I'd be more inclined to change things.
Regards,
On Mon, Feb 16, 2015 at 03:47:07PM +0000, Yawning Angel wrote:
On Mon, 16 Feb 2015 10:17:51 -0500 David Goulet dgoulet@ev0ke.net wrote: [snip]
A hidden service is created using the key and list of port/targets, that will persist till configuration reload or the termination of the tor process.
Now, an HS bound to a control connection might be a good idea, I'm not 100% sure but I can see issues with this. Let's say "ControlListenAddress" is used, this means a TCP socket and it can timeout if no activity, so if that happens, I loose my HS?
That's correct, though unless tor or the controller library has code to stomp on long dormant connections (which a casual look says there isn't), this shouldn't happen, because TCP/IP in itself has no idle timeout (See RFC 1122 4.2.3.6 regarding keep alives, which would also not cause HS loss, since the OS will respond to the probe).
There may be certain broken middleboxes (loadbalancers etc) that stomp on long idle TCP connections, but anyone that is running a control port connection through such a thing, and sending RSA keying material in the clear, probably has bigger things to worry about.
This also put quite a requirement on the user side to add an HS on its tor-ramdisk for instance but has to use a client that keeps the control connection opens for its lifetime?... How does that work with stem, it would have to keep the control connection open and the script using it can't quit else the socket gets closed by the OS?
Yup, I don't see "you need to leave stem running" as being all that bad, since this is mostly targeted at managed applications (chat, filesharing, global leaks, etc).
If someone has a suggestion for an alternative interface that can handle applications crashing (possibly before they persist the list of HSes they need to clean up), applications that are just poorly written (and not cleaning up all the ephemeral HSes), and (optionally, though lacking this would be a reduction in features) limiting cross application HS enumeration, I'd be more inclined to change things.
First, thanks for doing this! I think it's a great feature which will make it much easier to create new hidden service applications.
I like the idea of tying HS lifetime to the control port connection for the reasons you state, namely that cleanup is automatic when applications crash.
However, it seems like in the case of applications which are not HS-specific this will necessitate keeping another process running just to keep the HS alive. I'd rather see two modes: one as you describe, and another in which the ephemeral HS stays running until a new control port connection requests that it be stopped. To avoid allowing enumeration of running services, the "stop" command could require that the requestor already knows some details of the HS - either a cookie generated at creation time, or perhaps just the private key that was provided when it was started.
This of course wouldn't result in crashed applications' HSes being cleaned up automatically, but having a few stale HSes sitting around isn't the end of the world. One approach for cleaning them up could be that tor could remove them automatically after it sees connection refused a few times.
~leif
On Mon, 16 Feb 2015 16:11:55 +0000 Leif Ryge leif@synthesize.us wrote: [snippity]
However, it seems like in the case of applications which are not HS-specific this will necessitate keeping another process running just to keep the HS alive. I'd rather see two modes: one as you describe, and another in which the ephemeral HS stays running until a new control port connection requests that it be stopped. To avoid allowing enumeration of running services, the "stop" command could require that the requestor already knows some details of the HS - either a cookie generated at creation time, or perhaps just the private key that was provided when it was started.
dgoulet suggested "Detach=true" as an optional argument, which is what the add side interface would look like if I did this.
This of course wouldn't result in crashed applications' HSes being cleaned up automatically, but having a few stale HSes sitting around isn't the end of the world. One approach for cleaning them up could be that tor could remove them automatically after it sees connection refused a few times.
I'm not quite sure how I feel about this yet. The code for doing all of this isn't that difficult, but I'd want to hear from a few more people about what the right thing to do here would be.
Most importantly since the `ADD_EPH_HS` interface uses key/value pairs for the port/target now, this would be easy to add on at a later date even if it doesn't get included in the first iteration.
Something to discuss at the dev-meeting if consensus hasn't been reached by then.
Regards,
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256
(CCing the hidden-services list.)
On 16/02/15 16:11, Leif Ryge wrote:
If someone has a suggestion for an alternative interface that can handle applications crashing (possibly before they persist the list of HSes they need to clean up), applications that are just poorly written (and not cleaning up all the ephemeral HSes), and (optionally, though lacking this would be a reduction in features) limiting cross application HS enumeration, I'd be more inclined to change things.
First, thanks for doing this! I think it's a great feature which will make it much easier to create new hidden service applications.
Seconded!
I like the idea of tying HS lifetime to the control port connection for the reasons you state, namely that cleanup is automatic when applications crash.
As an app developer this strikes me as the right approach. But having said that, I wouldn't actually need this feature because Briar already uses __OwningControllerProcess to shut down Tor if the control connection is closed. I imagine the same would apply to any app that manages its own Tor process - so this feature would only be useful for apps that share a Tor process with other apps and communicate directly with it via the control port, rather than indirectly via a controller such as Orbot.
Are there any such apps, and is it a good idea to support such apps (has the rest of the control protocol been examined for cross-controller data leaks, what happens if apps tread on each other's configuration changes, etc)?
However, it seems like in the case of applications which are not HS-specific this will necessitate keeping another process running just to keep the HS alive.
Dormant processes are essentially free, so does this matter?
Cheers, Michael
On Mon, 16 Feb 2015 19:35:58 +0000 Michael Rogers michael@briarproject.org wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256
(CCing the hidden-services list.)
(Wonder if my reply will bounce.)
On 16/02/15 16:11, Leif Ryge wrote:
If someone has a suggestion for an alternative interface that can handle applications crashing (possibly before they persist the list of HSes they need to clean up), applications that are just poorly written (and not cleaning up all the ephemeral HSes), and (optionally, though lacking this would be a reduction in features) limiting cross application HS enumeration, I'd be more inclined to change things.
First, thanks for doing this! I think it's a great feature which will make it much easier to create new hidden service applications.
Seconded!
I like the idea of tying HS lifetime to the control port connection for the reasons you state, namely that cleanup is automatic when applications crash.
As an app developer this strikes me as the right approach. But having said that, I wouldn't actually need this feature because Briar already uses __OwningControllerProcess to shut down Tor if the control connection is closed. I imagine the same would apply to any app that manages its own Tor process - so this feature would only be useful for apps that share a Tor process with other apps and communicate directly with it via the control port, rather than indirectly via a controller such as Orbot.
Yep. I suspect that well behaved applications won't need this for the most part, but by defining things this way, it avoids unpleasant surprises for apps that aren't written well, or those that do try to share a tor instance.
Are there any such apps, and is it a good idea to support such apps (has the rest of the control protocol been examined for cross-controller data leaks, what happens if apps tread on each other's configuration changes, etc)?
I haven't looked at other concerns in depth, and AFAIK it's a huge problem area. My concerns in this area are more "does my branch make the current situation any worse", rather than "solve all the control port problems, and make sure this is totally a safe/fine/recommended thing to do" (It's none of those things).
If most if not all apps will set _OwningControllerProcess, the current behavior doesn't hurt either since the tor instance will die anyway, and on the off chance that someone writes a not-so-great app (heaven forbid), the right thing happens.
However, it seems like in the case of applications which are not HS-specific this will necessitate keeping another process running just to keep the HS alive.
Dormant processes are essentially free, so does this matter?
I have this mindset too.
As an app developer this strikes me as the right approach. But having said that, I wouldn't actually need this feature because Briar already uses __OwningControllerProcess to shut down Tor if the control connection is closed. I imagine the same would apply to any app that manages its own Tor process - so this feature would only be useful for apps that share a Tor process with other apps and communicate directly with it via the control port, rather than indirectly via a controller such as Orbot.
This feature is going to change how tor controllers operate... and whether the tor controller is a separate process or an API that the application uses seems to have been overlooked in this last statement of yours. Here's an example of what I mean; In the case of Twisted Python applications... olde apps written a decade ago are made to work with Tor hidden services without a single line of code changing... because the endpoint is dynamically loaded and the controlling of tor is abstracted by this Twisted endpoint.
pip install txtorcon && twistd -n web --port "onion:80" --path ~/public_html
In this case... can twistd be said to manage it's own tor process? It certainly does not contain any Tor specific code whatsoever. It is managed in the same process but the details are abstracted away by the txtorcon controller API.
Furthermore it is not limited to apps that share a tor process with other applications... Perhaps the computer has multiple instances of tor running; the Tor controller could be setup to connect to any of these control ports. This is an observation about a deployment issue more so than an API issue.
However, it seems like in the case of applications which are not HS-specific this will necessitate keeping another process running just to keep the HS alive.
responding to Leif's comment above:
This seems like a false statement in the context of using txtorcon with this feature. The above txtorcon twistd example could be made to work with the tor control port and it would not imply "keeping another process running". Furthermore, the above example currently relies on txtorcon launching it's own Tor process which I think is convenient but not so good for "security isolation"... and implies ambient authority, that is, why should the python process have that much control over the tor process? The python app only needs the privilege to create and remove it's Tor hidden service (principal of least privilege ftw).
Dormant processes are essentially free, so does this matter?
Yes, it seems like it doesn't matter much.. The local connection from the tor process would fail to connect to the local service. I'm not sure if there are other implications that are more important than this.
From my perspective, the entire point of this feature is to allow
applications to use "the system Tor" (or, at least "some already-running tor") to put their hidden services on.
(Or, looking at it another way, if you don't want to share a tor instance with other applications, you can do that easily today -- you start up a tor, and can use __OwningControllerProcess as you see fit).
The *only* reason that txtorcon by default launches a new tor process for the "onion:" endpoints is because nothing else will work reliably.
I'd very much like the default to be that it adds a new onion using Yawning's API. This will still fall back to "launch a new tor process", but if you know what you're doing you can have a system (or personal) Tor instance running (with cookie authentication) that can have hidden services added and removed from it.
Obviously, this will be a lot faster than launching a new Tor (and less load or DirAuths). As a bonus, Yawning's API is nice and simple (versus SETCONF, which makes it super easy to nuke all the *other* hidden services you configured).
What I would expect of people using this feature is that the process that's "adding the hidden service" is the very same one that's "providing the service"...
(If you're doing something with a separate nginx or whatever process running your hidden-service, than you should know enough to be able to add a couple lines to your torrc and make a normal, "permanent" hidden-service).
So, in that context, if your application code isn't perfect and might sometimes crash, definitely the right thing is to nuke the hidden service. If the app didn't even save its private keys before that happened, well, too bad.
I guess to put another way: I can't see a use-case to keep the hidden- service around if the application that added it went away.
On Tue, 17 Feb 2015 02:22:54 +0400 meejah meejah@meejah.ca wrote:
From my perspective, the entire point of this feature is to allow applications to use "the system Tor" (or, at least "some already-running tor") to put their hidden services on.
The design the way it is with more isolation than usual also allows for people that use the tails/whonix/or my control port tinfoil hat filter to potentially whitelist the commands as well, since there isn't any information disclosure (unless I screwed up the implementation).
Modifying the filters is left up to the implementors (and I probably won't change or-ctl-filter to allow them since I don't use HSes[0]), but I view it as a step forward for including/using applications that use HSes into such environments.
(Or, looking at it another way, if you don't want to share a tor instance with other applications, you can do that easily today -- you start up a tor, and can use __OwningControllerProcess as you see fit).
The *only* reason that txtorcon by default launches a new tor process for the "onion:" endpoints is because nothing else will work reliably.
I'd very much like the default to be that it adds a new onion using Yawning's API. This will still fall back to "launch a new tor process", but if you know what you're doing you can have a system (or personal) Tor instance running (with cookie authentication) that can have hidden services added and removed from it.
Obviously, this will be a lot faster than launching a new Tor (and less load or DirAuths). As a bonus, Yawning's API is nice and simple (versus SETCONF, which makes it super easy to nuke all the *other* hidden services you configured).
What I would expect of people using this feature is that the process that's "adding the hidden service" is the very same one that's "providing the service"...
(If you're doing something with a separate nginx or whatever process running your hidden-service, than you should know enough to be able to add a couple lines to your torrc and make a normal, "permanent" hidden-service).
Or write a trivial script that prompts for a password, decrypts a HS key, and loads it into a tor instance over the control port. I assume if/when my branch gets merged into mainline tor that the controler APIs will support it in a easy to use manner.
So, in that context, if your application code isn't perfect and might sometimes crash, definitely the right thing is to nuke the hidden service. If the app didn't even save its private keys before that happened, well, too bad.
I guess to put another way: I can't see a use-case to keep the hidden- service around if the application that added it went away.
It seems a lose consensus is reached here that this is ok, so I'm going to leave the design as is and write the control-spec.txt patch.
Let me chime in on saying that this looks to me like a great development. I even imagine that in a couple of years most end-to-end encrypted services on the Internet may be using this interface, so for the sake of accessibility for future devs, I would suggest something totally superficial:
On Sat, Feb 14, 2015 at 12:45:24AM +0000, Yawning Angel wrote:
ADD_EPH_HS - Adds a new ephemeral hidden service.
The "ephemeral" vs current behavior seems to be an aspect that might go away soon anyway, so I would leave the cryptic "EPH" out. Also, the idea that hidden services are hidden is a bit outdated - many are intentionally not hidden at all and only interested in the aspect of providing authenticated end- to-end encryption and/or anonymization for the calling entity, so "HS" isn't exactly an acronym which speaks a clear language. Thus I would think it will be easier for future generations of devs and sysadmins to grok this, if the functions were simply named
ADD_SERVICE, REMOVE_SERVICE.
Concerning the "ephemerality" of it, I can imagine services being configured en passant by a cat >> socket from a shell script or so, while the actual application was merely told to receive connections on localhost:something and has no awareness of Tor at all, like most Tor HS are, I presume. So in the spirit of Leif's suggestion I would make the dependency on a ControlPort link staying up the non-default.
KUTGW!
carlo von lynX lynX@time.to.get.psyced.org writes:
I like your suggestion, and while we're bikeshedding ;) what if we use the recently-proposed naming system and call it "ADD_ONION"?
Concerning the "ephemerality" of it, I can imagine services being configured en passant by a cat >> socket from a shell script or so, [..]
You still need to authenticate. I do like the simplicity, but it will be a little more complex than that. I guess it's a bit of extra work to keep such a connection around. But really, it's just storing a PID and killing it when you're done.
It's still, I think, worth distinguishing somehow between an onion service added via SETCONF (which will get written to disc, and written to the torrc potentially) and the API Yawning has added that will vanish if the tor instance is re-started (and has no "hidden service dir" at all).
Another option could always be added in the future, like "lifetime={controller,tor}" or something if the "goes away with process" makes it harder than necessary to use. My instincts still say that "controller connection lifetime" is a good API, but that's not a super compelling argument ;)
Concerning the "ephemerality" of it, I can imagine services being configured en passant by a cat >> socket from a shell script or so, [..]
On Tue, Feb 24, 2015 at 09:05:38PM +0400, meejah wrote:
You still need to authenticate. I do like the simplicity, but it will be a little more complex than that. I guess it's a bit of extra work to keep such a connection around. But really, it's just storing a PID and killing it when you're done.
posix has user/group access rights for file sockets and linux has acls on top of that. using a localhost socket is less secure, i agree.
It's still, I think, worth distinguishing somehow between an onion service added via SETCONF (which will get written to disc, and written to the torrc potentially) and the API Yawning has added that will vanish if the tor instance is re-started (and has no "hidden service dir" at all).
that is too inflexible.. i may want non-ephemeral services activated by a cat to a socket. i don't see the point in an artificial requirement to have config things frozen into a text file.
Another option could always be added in the future, like "lifetime={controller,tor}" or something if the "goes away with process" makes it harder than necessary to use. My instincts still say that "controller connection lifetime" is a good API, but that's not a super compelling argument ;)
the advantages of that aren't obvious to me. why would i need to make every networking app hold the hand of its router to let it know it's still needed? do we do this for regular networking, too? tor is on its way to becoming an AF_TOR - a networking esssential. making a hidden service could one day be as simple as doing listen() on an AF_TOR socket...
On Wed, 25 Feb 2015 05:06:37 +0000, carlo von lynX wrote: ....
the advantages of that aren't obvious to me. why would i need to make every networking app hold the hand of its router to let it know it's still needed?
You answer your question yourself:
tor is on its way to becoming an AF_TOR - a networking esssential. making a hidden service could one day be as simple as doing listen() on an AF_TOR socket...
The AF_TOR listener would go away with closing the listener socket as well (and thus is bound to the lifetime of the process); so binding a hidden service to the control connection is the obvious analogy.
Whether tor as such should exit with loss of control connections is (more) debatable - if my app starts the tor process, I'd expect it to terminate when it detects that I'm no longer there. If it's a system/shared daemon we'd need to talk about how it comes into existence in the first place, and how it is shared.
Also, would you entrust your hidden service keys to a system-wide tor process? :-)
Andreas
On Wed, Feb 25, 2015 at 07:41:01AM +0100, Andreas Krey wrote:
The AF_TOR listener would go away with closing the listener socket as well (and thus is bound to the lifetime of the process); so binding a hidden service to the control connection is the obvious analogy.
Yes, but as it stands AF_TOR is not the #1 API deployed in network applications. The majority of hidden services are $whatever configured to listen on port localhost:something and zero awareness of any tor router doing the rest of the work. Having to change hundreds of existing apps so that they can work with tor without having to edit torrc is a worse tradeoff than having to edit torrc.
What is useful here is if I can use existing $app with existing tor router and just have a shell script drop the glue instructions into the tor unix socket. This allows to configure hidden services without editing neither torrc nor the code of the $app. A bit like Vidalia does it, but without messing with the torrc and without being Vidalia.
Also, would you entrust your hidden service keys to a system-wide tor process? :-)
It's like asking if I trust my bank to keep my money. If you don't trust your Tor process, try not using a computer. If you don't trust your cheap server hosting provider, don't put hidden services on cheap rented servers or virtual machines. But if you don't trust your OS, there is no place where you can hide your hidden service keys.
Hello,
Not sure about the renaming thing. I'm open to doing so, but a lot of the user facing stuff (the torrc configuration variables etc) use the HS terminology, so I'm not sure if being inconsistent is good.
On Wed, 25 Feb 2015 13:51:59 +0100 carlo von lynX lynX@time.to.get.psyced.org wrote:
On Wed, Feb 25, 2015 at 07:41:01AM +0100, Andreas Krey wrote:
The AF_TOR listener would go away with closing the listener socket as well (and thus is bound to the lifetime of the process); so binding a hidden service to the control connection is the obvious analogy.
Yes, but as it stands AF_TOR is not the #1 API deployed in network applications. The majority of hidden services are $whatever configured to listen on port localhost:something and zero awareness of any tor router doing the rest of the work. Having to change hundreds of existing apps so that they can work with tor without having to edit torrc is a worse tradeoff than having to edit torrc.
So, write a python script that pulls in txtorcon or stem, does the appropriate controller goo, and subprocess.call()s $whatever.
Anyway now that master is 0.2.7.x, I'm looking to revisit this in the form of:
* Renaming the commands if necessary. * Minor code cleanups (that occurred to me after I let it sit for a while). * Adding an argument to `ADD_EPH_HS` to suppress the private key being returned over the control port, when the caller requests that tor generate the keypair (for cases where the HS is truely oneshot and the user does not wish to preserve it). * Documentation.
Regards,
On Wed, 25 Feb 2015 16:14:47 +0000 Yawning Angel yawning@schwanenlied.me wrote:
Anyway now that master is 0.2.7.x, I'm looking to revisit this in the form of:
[snip]
I talked to a few people at Valencia about this, including discussing the outstanding issues and made the following changes since the last discussion:
* ADD_EPH_HS/DEL_EPH_HS are now ADD_ONION/DEL_ONION. * An optional "DiscardPK" argument can now be provided to suppress including the newly generated private key in the response (for one-shot, never-want-to-recreate-ever HSes, there's no reason to pass the private key around). * Minor cleanups.
The code: https://github.com/Yawning/tor/compare/feature6411 The spec: https://github.com/Yawning/torspec/compare/feature6411
The spec change can probably use proofreading, corrections, and clarification, but those aren't my strong point(s). Assuming that no unpleasant surprises pop up, I will rebase/squash down my feature branch and seek code review/merge into master (0.2.7.x).
Questions, comments, feedback appreciated[0],
On Wed, 11 Mar 2015 02:35:10 +0000 Yawning Angel yawning@schwanenlied.me wrote:
The code: https://github.com/Yawning/tor/compare/feature6411 The spec: https://github.com/Yawning/torspec/compare/feature6411
Minor updates to both over the course of yesterday, thanks to all that gave useful feedback and corrections. The notable changes are:
* "DiscardPK" is now "Flags=DiscardPK", to allow for.....
[0]: Onions added this way will remain tied to the control port connection that created them for now. That particular dead pony has been loaded onto the cart and shipped off to the glue factory, and is no longer available for beatings.
* I lied. People that want to shoot themselves in the foot can now specify the "Detach" flag which explicitly unlinks the newly created HS from any control port instance. "Detach"ed Onion Services will persist till tor termination, or explicitly removed by "DEL_ONION".
* "DEL_ONION" now will allow the removal of Onion Services belonging to the current connection in addition to any Onion Service created with the "Detach" flag.
* Added "GET_ONIONS" that returns a list of Onion Services belonging to the current connection in addition to all Onion Services created with the "Detach" flag.
I strongly recommend ignoring the fact that the "Detach" flag and "GET_ONIONS" command exists, beyond reviewing my code and making sure they're implemented correctly.
That said, I know multiple developers will probably write applications that implements "cleanup" as "DEL_ONION ALL THE THINGS", stomping over unrelated "Detach"-ed services[0].
Unless something comes up, I plan to rebase/squash my feature branch for review sometime by early next week, so if people have strong opinions on this feature, they should speak up now.
Regards,
Hello all,
Just a quick heads up for where this is going: * 'GET_ONIONS' was changed to be 'GETINFO' subcommands for consistency. * Code is in 'needs_review', for the 2nd round.
As a proof of concept I wrote: https://github.com/Yawning/onionwrap
It's a dumb quick and dirty hack that works like this:
$ TOR_CONTROL_PORT=9251 ./onionwrap -port="80,8080" godoc -http=%TADDR INFO: Created onion: 5rn73le64hri2zmp.onion:80 -> 127.0.0.1:8080
[onionwrap blocks till killed]
The %TADDR notation is a convenience thing for automation, there's also %TPORT, and %VPORT that get rewritten as appropriate.
The onionwrap code is still incomplete, so I wouldn't recommend using it quite yet, especially since it:
* Only supports creating one shot Onion Services. * Should support talking to the real server over AF_UNIX. * Needs documentation. * Requires a really experimental branch based on tor's master. * Maybe should use stem/txtorcon instead of bulb[0].
But as a proof of concept and a demonstration of the feature, I think it gets the point across. Thanks to special for inspiring me to write this.
Regards,
On Wed, 25 Feb 2015 13:51:59 +0000, carlo von lynX wrote: ...
What is useful here is if I can use existing $app with existing tor router and just have a shell script drop the glue instructions into the tor unix socket.
One way to do that would be to tie the hidden service to the existence of the PID of your app - just exec the app in the script after setting up the HS. (I seem to remember that being an option in some form already.)
Alternatively tor could check whether the listener the HS is accessing is still open, and discard the HS when that is no longer the case. (Possibly new idea.)
(And hopefully your application isn't giving extended authority to clients connecting from 127.0.0.1.)
Andreas
Thanks "Angel", appreciate your effort.
On Thu, Feb 26, 2015 at 09:29:05AM +0100, Andreas Krey wrote:
On Wed, 25 Feb 2015 13:51:59 +0000, carlo von lynX wrote: ...
What is useful here is if I can use existing $app with existing tor router and just have a shell script drop the glue instructions into the tor unix socket.
One way to do that would be to tie the hidden service to the existence of the PID of your app - just exec the app in the script after setting up the HS. (I seem to remember that being an option in some form already.)
Not exactly the intended behaviour when somebody has to restart the web server and doesn't expect Tor to stop servicing it... or when the web server is written in $occasionalcoredumpstyle.
Alternatively tor could check whether the listener the HS is accessing is still open, and discard the HS when that is no longer the case. (Possibly new idea.)
Yes, and then hope not for a racing condition.
(And hopefully your application isn't giving extended authority to clients connecting from 127.0.0.1.)
Depends on the specific constellation.. if noone is web browsing on the same system.. if processes are not separated by uid anyway, because that actually takes some work, und finally nobody else has a login, warning about unsecured control ports or suchlike is crying wolf and educating users to ignore such warnings.
The current default way to run the Tor router is with the same uid as the user herself, right? Putting an authentication method on the control port is pretty pointless - if an attacker manages to break into her browser he doesn't have to look very far for her Tor state. So all the warnings about localhost being not safe enough yet even though for the majority of users it is the factual configuration appears somewhat counter-productive to me. We should first introduce a habit of having Tor neither launched by TBB nor by vidalia nor as root but using its own isolated uid.
Then again, whichever way you give the user a way to control the Tor router opens up an attack vector for somebody who managed to break into a faulty client application. So to me the entire lets-not-trust-localhost logic doesn't work out in my head. It either produces bureaucratic complications or false positives in the warning log.
Maybe I overlooked something.
On Sat, Feb 28, 2015 at 02:40:29PM +0100, carlo von lynX wrote:
Thanks "Angel", appreciate your effort.
On Thu, Feb 26, 2015 at 09:29:05AM +0100, Andreas Krey wrote:
On Wed, 25 Feb 2015 13:51:59 +0000, carlo von lynX wrote: ...
What is useful here is if I can use existing $app with existing tor router and just have a shell script drop the glue instructions into the tor unix socket.
One way to do that would be to tie the hidden service to the existence of the PID of your app - just exec the app in the script after setting up the HS. (I seem to remember that being an option in some form already.)
Not exactly the intended behaviour when somebody has to restart the web server and doesn't expect Tor to stop servicing it... or when the web server is written in $occasionalcoredumpstyle.
I think this is an important point I hadn't considered - at the very least, it will be necessary to make sure that Tor handles well the case where the same HS is destroyed and then immediately recreated.
Alternatively tor could check whether the listener the HS is accessing is still open, and discard the HS when that is no longer the case. (Possibly new idea.)
Yes, and then hope not for a racing condition.
(And hopefully your application isn't giving extended authority to clients connecting from 127.0.0.1.)
Depends on the specific constellation.. if noone is web browsing on the same system.. if processes are not separated by uid anyway, because that actually takes some work, und finally nobody else has a login, warning about unsecured control ports or suchlike is crying wolf and educating users to ignore such warnings.
The current default way to run the Tor router is with the same uid as the user herself, right? Putting an authentication method on the control port is pretty pointless - if an attacker manages to break into her browser he doesn't have to look very far for her Tor state. So all the warnings about localhost being not safe enough yet even though for the majority of users it is the factual configuration appears somewhat counter-productive to me. We should first introduce a habit of having Tor neither launched by TBB nor by vidalia nor as root but using its own isolated uid.
FWIW this is already how Debian (and presumably other distros') tor packages work: tor runs as a dedicated user. Already it is possible to grant other users access to the control port (from which they can already create and remove hidden services). The reason why HS applications that create their own HSes generally run their own instance of tor as their own uid is that the hidden service data (key and hostname) written by tor is currently only readable by the tor user. There is another patch to address this issue (in progress or possibly already merged, sorry I'm not looking up the ticket right now) to allow this data to be written with permissions for another group to read it, but this ephemeral HS plan of delivering the information over the control port is obviously much better/more flexible.
From Valencia,
~leif
Then again, whichever way you give the user a way to control the Tor router opens up an attack vector for somebody who managed to break into a faulty client application. So to me the entire lets-not-trust-localhost logic doesn't work out in my head. It either produces bureaucratic complications or false positives in the warning log.
Maybe I overlooked something.
-- E-mail is public! Talk to me in private using Tor. torify telnet loupsycedyglgamf.onion DON'T SEND ME irc://loupsycedyglgamf.onion:67/lynX PRIVATE EMAIL http://loupsycedyglgamf.onion/LynX/ OR FACEBOOGLE
FWIW this is already how Debian (and presumably other distros') tor packages work: tor runs as a dedicated user. Already it is possible to grant other users access to the control port (from which they can already create and remove hidden services). The reason why HS applications that create their own HSes generally run their own instance of tor as their own uid is that the hidden service data (key and hostname) written by tor is currently only readable by the tor user. There is another patch to address this issue (in progress or possibly already merged, sorry I'm not looking up the ticket right now) to
Already merged in 2.6.x.
allow this data to be written with permissions for another group to read it, but this ephemeral HS plan of delivering the information over the control port is obviously much better/more flexible.
From Valencia, ~leif
From Valencia... the other end of the table from Leif in the same tiny room.
David
On Fri, Feb 13, 2015, at 04:45 PM, Yawning Angel wrote:
Yes, this means that if you run "kittensomgmewmew.onion" and are scared of the NSA's persistent attempts to extract your hidden service key, via ultrasonic laser beamed from their satellites,
...or the backdoor in the firmware on your hard disk... :)
--dave