On Sat, 14 Feb 2015 13:46:04 -0800 Damian Johnson <atagar@torproject.org> wrote:

One gotcha to think about is that ADD_EPH_HS is using a variable number of positional arguments. This will limit your ability to expand this command in the future with new arguments. Also, I'd suggest renaming addrPort to addrTarget (to avoid making this sound restricted to a simple port).

Hmm ok. Judging by the feedback, I'm thinking the following: * Auth is still a low-ish priority, I want to get the basic ephemeral stuff done first, and I need to read up more on how it works, and how the code is structured, before I can promise things in this area. * People seem to be ok with the tying ephemeral HSes to the originating control port (and if the only major argument against is "it's a bit weird, relative to other things, well, eph. HSes are weird, and this solves a bunch of problems). So this will be implemented as planned. * ADD_EPH_HS syntax changed to something like: "ADD_EPH_HS" keytype:keyblob 1*(SP "Port=" virtPort "," addrTarget) CRLF So, basically, space separated instances of "Port=virtPort,addrTarget" entries, of which there must be at least one. This clearly indicates the argument type and should be future proof (and also has the side benefit of being easier for me to validate. Thoughts? -- Yawning Angel

On Mon, 16 Feb 2015 10:17:51 -0500 David Goulet <dgoulet@ev0ke.net> wrote: [snip]

A hidden service is created using the key and list of port/targets, that will persist till configuration reload or the termination of the tor process.

Now, an HS bound to a control connection might be a good idea, I'm not 100% sure but I can see issues with this. Let's say "ControlListenAddress" is used, this means a TCP socket and it can timeout if no activity, so if that happens, I loose my HS?

That's correct, though unless tor or the controller library has code to stomp on long dormant connections (which a casual look says there isn't), this shouldn't happen, because TCP/IP in itself has no idle timeout (See RFC 1122 4.2.3.6 regarding keep alives, which would also not cause HS loss, since the OS will respond to the probe). There may be certain broken middleboxes (loadbalancers etc) that stomp on long idle TCP connections, but anyone that is running a control port connection through such a thing, and sending RSA keying material in the clear, probably has bigger things to worry about.

This also put quite a requirement on the user side to add an HS on its tor-ramdisk for instance but has to use a client that keeps the control connection opens for its lifetime?... How does that work with stem, it would have to keep the control connection open and the script using it can't quit else the socket gets closed by the OS?

Yup, I don't see "you need to leave stem running" as being all that bad, since this is mostly targeted at managed applications (chat, filesharing, global leaks, etc). If someone has a suggestion for an alternative interface that can handle applications crashing (possibly before they persist the list of HSes they need to clean up), applications that are just poorly written (and not cleaning up all the ephemeral HSes), and (optionally, though lacking this would be a reduction in features) limiting cross application HS enumeration, I'd be more inclined to change things. Regards, -- Yawning Angel

On Mon, 16 Feb 2015 16:11:55 +0000 Leif Ryge <leif@synthesize.us> wrote: [snippity]

However, it seems like in the case of applications which are not HS-specific this will necessitate keeping another process running just to keep the HS alive. I'd rather see two modes: one as you describe, and another in which the ephemeral HS stays running until a new control port connection requests that it be stopped. To avoid allowing enumeration of running services, the "stop" command could require that the requestor already knows some details of the HS - either a cookie generated at creation time, or perhaps just the private key that was provided when it was started.

dgoulet suggested "Detach=true" as an optional argument, which is what the add side interface would look like if I did this.

This of course wouldn't result in crashed applications' HSes being cleaned up automatically, but having a few stale HSes sitting around isn't the end of the world. One approach for cleaning them up could be that tor could remove them automatically after it sees connection refused a few times.

I'm not quite sure how I feel about this yet. The code for doing all of this isn't that difficult, but I'd want to hear from a few more people about what the right thing to do here would be. Most importantly since the `ADD_EPH_HS` interface uses key/value pairs for the port/target now, this would be easy to add on at a later date even if it doesn't get included in the first iteration. Something to discuss at the dev-meeting if consensus hasn't been reached by then. Regards, -- Yawning Angel

On Mon, 16 Feb 2015 19:35:58 +0000 Michael Rogers <michael@briarproject.org> wrote:

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256

(CCing the hidden-services list.)

(Wonder if my reply will bounce.)

On 16/02/15 16:11, Leif Ryge wrote:

...

...

If someone has a suggestion for an alternative interface that can handle applications crashing (possibly before they persist the list of HSes they need to clean up), applications that are just poorly written (and not cleaning up all the ephemeral HSes), and (optionally, though lacking this would be a reduction in features) limiting cross application HS enumeration, I'd be more inclined to change things.

First, thanks for doing this! I think it's a great feature which will make it much easier to create new hidden service applications.

Seconded!

...

I like the idea of tying HS lifetime to the control port connection for the reasons you state, namely that cleanup is automatic when applications crash.

As an app developer this strikes me as the right approach. But having said that, I wouldn't actually need this feature because Briar already uses __OwningControllerProcess to shut down Tor if the control connection is closed. I imagine the same would apply to any app that manages its own Tor process - so this feature would only be useful for apps that share a Tor process with other apps and communicate directly with it via the control port, rather than indirectly via a controller such as Orbot.

Yep. I suspect that well behaved applications won't need this for the most part, but by defining things this way, it avoids unpleasant surprises for apps that aren't written well, or those that do try to share a tor instance.

Are there any such apps, and is it a good idea to support such apps (has the rest of the control protocol been examined for cross-controller data leaks, what happens if apps tread on each other's configuration changes, etc)?

I haven't looked at other concerns in depth, and AFAIK it's a huge problem area. My concerns in this area are more "does my branch make the current situation any worse", rather than "solve all the control port problems, and make sure this is totally a safe/fine/recommended thing to do" (It's none of those things). If most if not all apps will set _OwningControllerProcess, the current behavior doesn't hurt either since the tor instance will die anyway, and on the off chance that someone writes a not-so-great app (heaven forbid), the right thing happens.

...

However, it seems like in the case of applications which are not HS-specific this will necessitate keeping another process running just to keep the HS alive.

Dormant processes are essentially free, so does this matter?

I have this mindset too. -- Yawning Angel

From my perspective, the entire point of this feature is to allow applications to use "the system Tor" (or, at least "some already-running tor") to put their hidden services on.

(Or, looking at it another way, if you don't want to share a tor instance with other applications, you can do that easily today -- you start up a tor, and can use __OwningControllerProcess as you see fit). The *only* reason that txtorcon by default launches a new tor process for the "onion:" endpoints is because nothing else will work reliably. I'd very much like the default to be that it adds a new onion using Yawning's API. This will still fall back to "launch a new tor process", but if you know what you're doing you can have a system (or personal) Tor instance running (with cookie authentication) that can have hidden services added and removed from it. Obviously, this will be a lot faster than launching a new Tor (and less load or DirAuths). As a bonus, Yawning's API is nice and simple (versus SETCONF, which makes it super easy to nuke all the *other* hidden services you configured). What I would expect of people using this feature is that the process that's "adding the hidden service" is the very same one that's "providing the service"... (If you're doing something with a separate nginx or whatever process running your hidden-service, than you should know enough to be able to add a couple lines to your torrc and make a normal, "permanent" hidden-service). So, in that context, if your application code isn't perfect and might sometimes crash, definitely the right thing is to nuke the hidden service. If the app didn't even save its private keys before that happened, well, too bad. I guess to put another way: I can't see a use-case to keep the hidden- service around if the application that added it went away. -- meejah

On Tue, 17 Feb 2015 02:22:54 +0400 meejah <meejah@meejah.ca> wrote:

From my perspective, the entire point of this feature is to allow applications to use "the system Tor" (or, at least "some already-running tor") to put their hidden services on.

The design the way it is with more isolation than usual also allows for people that use the tails/whonix/or my control port tinfoil hat filter to potentially whitelist the commands as well, since there isn't any information disclosure (unless I screwed up the implementation). Modifying the filters is left up to the implementors (and I probably won't change or-ctl-filter to allow them since I don't use HSes[0]), but I view it as a step forward for including/using applications that use HSes into such environments.

(Or, looking at it another way, if you don't want to share a tor instance with other applications, you can do that easily today -- you start up a tor, and can use __OwningControllerProcess as you see fit).

The *only* reason that txtorcon by default launches a new tor process for the "onion:" endpoints is because nothing else will work reliably.

I'd very much like the default to be that it adds a new onion using Yawning's API. This will still fall back to "launch a new tor process", but if you know what you're doing you can have a system (or personal) Tor instance running (with cookie authentication) that can have hidden services added and removed from it.

Obviously, this will be a lot faster than launching a new Tor (and less load or DirAuths). As a bonus, Yawning's API is nice and simple (versus SETCONF, which makes it super easy to nuke all the *other* hidden services you configured).

What I would expect of people using this feature is that the process that's "adding the hidden service" is the very same one that's "providing the service"...

(If you're doing something with a separate nginx or whatever process running your hidden-service, than you should know enough to be able to add a couple lines to your torrc and make a normal, "permanent" hidden-service).

Or write a trivial script that prompts for a password, decrypts a HS key, and loads it into a tor instance over the control port. I assume if/when my branch gets merged into mainline tor that the controler APIs will support it in a easy to use manner.

So, in that context, if your application code isn't perfect and might sometimes crash, definitely the right thing is to nuke the hidden service. If the app didn't even save its private keys before that happened, well, too bad.

I guess to put another way: I can't see a use-case to keep the hidden- service around if the application that added it went away.

It seems a lose consensus is reached here that this is ok, so I'm going to leave the design as is and write the control-spec.txt patch. -- Yawning Angel [0]: The first HS I ever set up was when I finished my first pass implementation, and got the code to a working state.

carlo von lynX <lynX@time.to.get.psyced.org> writes: I like your suggestion, and while we're bikeshedding ;) what if we use the recently-proposed naming system and call it "ADD_ONION"?

Concerning the "ephemerality" of it, I can imagine services being configured en passant by a cat >> socket from a shell script or so, [..]

You still need to authenticate. I do like the simplicity, but it will be a little more complex than that. I guess it's a bit of extra work to keep such a connection around. But really, it's just storing a PID and killing it when you're done. It's still, I think, worth distinguishing somehow between an onion service added via SETCONF (which will get written to disc, and written to the torrc potentially) and the API Yawning has added that will vanish if the tor instance is re-started (and has no "hidden service dir" at all). Another option could always be added in the future, like "lifetime={controller,tor}" or something if the "goes away with process" makes it harder than necessary to use. My instincts still say that "controller connection lifetime" is a good API, but that's not a super compelling argument ;) -- meejah

On Wed, 25 Feb 2015 05:06:37 +0000, carlo von lynX wrote: ....

the advantages of that aren't obvious to me. why would i need to make every networking app hold the hand of its router to let it know it's still needed?

You answer your question yourself:

tor is on its way to becoming an AF_TOR - a networking esssential. making a hidden service could one day be as simple as doing listen() on an AF_TOR socket...

The AF_TOR listener would go away with closing the listener socket as well (and thus is bound to the lifetime of the process); so binding a hidden service to the control connection is the obvious analogy. Whether tor as such should exit with loss of control connections is (more) debatable - if my app starts the tor process, I'd expect it to terminate when it detects that I'm no longer there. If it's a system/shared daemon we'd need to talk about how it comes into existence in the first place, and how it is shared. Also, would you entrust your hidden service keys to a system-wide tor process? :-) Andreas -- "Totally trivial. Famous last words." From: Linus Torvalds <torvalds@*.org> Date: Fri, 22 Jan 2010 07:29:21 -0800

Hello, Not sure about the renaming thing. I'm open to doing so, but a lot of the user facing stuff (the torrc configuration variables etc) use the HS terminology, so I'm not sure if being inconsistent is good. On Wed, 25 Feb 2015 13:51:59 +0100 carlo von lynX <lynX@time.to.get.psyced.org> wrote:

On Wed, Feb 25, 2015 at 07:41:01AM +0100, Andreas Krey wrote:

...

The AF_TOR listener would go away with closing the listener socket as well (and thus is bound to the lifetime of the process); so binding a hidden service to the control connection is the obvious analogy.

Yes, but as it stands AF_TOR is not the #1 API deployed in network applications. The majority of hidden services are $whatever configured to listen on port localhost:something and zero awareness of any tor router doing the rest of the work. Having to change hundreds of existing apps so that they can work with tor without having to edit torrc is a worse tradeoff than having to edit torrc.

So, write a python script that pulls in txtorcon or stem, does the appropriate controller goo, and subprocess.call()s $whatever. Anyway now that master is 0.2.7.x, I'm looking to revisit this in the form of: * Renaming the commands if necessary. * Minor code cleanups (that occurred to me after I let it sit for a while). * Adding an argument to `ADD_EPH_HS` to suppress the private key being returned over the control port, when the caller requests that tor generate the keypair (for cases where the HS is truely oneshot and the user does not wish to preserve it). * Documentation. Regards, -- Yawning Angel

On Wed, 11 Mar 2015 02:35:10 +0000 Yawning Angel <yawning@schwanenlied.me> wrote:

The code: https://github.com/Yawning/tor/compare/feature6411 The spec: https://github.com/Yawning/torspec/compare/feature6411

Minor updates to both over the course of yesterday, thanks to all that gave useful feedback and corrections. The notable changes are: * "DiscardPK" is now "Flags=DiscardPK", to allow for.....

[0]: Onions added this way will remain tied to the control port connection that created them for now. That particular dead pony has been loaded onto the cart and shipped off to the glue factory, and is no longer available for beatings.

* I lied. People that want to shoot themselves in the foot can now specify the "Detach" flag which explicitly unlinks the newly created HS from any control port instance. "Detach"ed Onion Services will persist till tor termination, or explicitly removed by "DEL_ONION". * "DEL_ONION" now will allow the removal of Onion Services belonging to the current connection in addition to any Onion Service created with the "Detach" flag. * Added "GET_ONIONS" that returns a list of Onion Services belonging to the current connection in addition to all Onion Services created with the "Detach" flag. I strongly recommend ignoring the fact that the "Detach" flag and "GET_ONIONS" command exists, beyond reviewing my code and making sure they're implemented correctly. That said, I know multiple developers will probably write applications that implements "cleanup" as "DEL_ONION ALL THE THINGS", stomping over unrelated "Detach"-ed services[0]. Unless something comes up, I plan to rebase/squash my feature branch for review sometime by early next week, so if people have strong opinions on this feature, they should speak up now. Regards, -- Yawning Angel [0]: I have my "not a bug"/"wontfix" ready for such situations.

On Wed, 25 Feb 2015 13:51:59 +0000, carlo von lynX wrote: ...

What is useful here is if I can use existing $app with existing tor router and just have a shell script drop the glue instructions into the tor unix socket.

One way to do that would be to tie the hidden service to the existence of the PID of your app - just exec the app in the script after setting up the HS. (I seem to remember that being an option in some form already.) Alternatively tor could check whether the listener the HS is accessing is still open, and discard the HS when that is no longer the case. (Possibly new idea.) (And hopefully your application isn't giving extended authority to clients connecting from 127.0.0.1.) Andreas -- "Totally trivial. Famous last words." From: Linus Torvalds <torvalds@*.org> Date: Fri, 22 Jan 2010 07:29:21 -0800

On Sat, Feb 28, 2015 at 02:40:29PM +0100, carlo von lynX wrote:

Thanks "Angel", appreciate your effort.

On Thu, Feb 26, 2015 at 09:29:05AM +0100, Andreas Krey wrote:

...

On Wed, 25 Feb 2015 13:51:59 +0000, carlo von lynX wrote: ...

...

What is useful here is if I can use existing $app with existing tor router and just have a shell script drop the glue instructions into the tor unix socket.

One way to do that would be to tie the hidden service to the existence of the PID of your app - just exec the app in the script after setting up the HS. (I seem to remember that being an option in some form already.)

Not exactly the intended behaviour when somebody has to restart the web server and doesn't expect Tor to stop servicing it... or when the web server is written in $occasionalcoredumpstyle.

I think this is an important point I hadn't considered - at the very least, it will be necessary to make sure that Tor handles well the case where the same HS is destroyed and then immediately recreated.

...

Alternatively tor could check whether the listener the HS is accessing is still open, and discard the HS when that is no longer the case. (Possibly new idea.)

Yes, and then hope not for a racing condition.

...

(And hopefully your application isn't giving extended authority to clients connecting from 127.0.0.1.)

Depends on the specific constellation.. if noone is web browsing on the same system.. if processes are not separated by uid anyway, because that actually takes some work, und finally nobody else has a login, warning about unsecured control ports or suchlike is crying wolf and educating users to ignore such warnings.

The current default way to run the Tor router is with the same uid as the user herself, right? Putting an authentication method on the control port is pretty pointless - if an attacker manages to break into her browser he doesn't have to look very far for her Tor state. So all the warnings about localhost being not safe enough yet even though for the majority of users it is the factual configuration appears somewhat counter-productive to me. We should first introduce a habit of having Tor neither launched by TBB nor by vidalia nor as root but using its own isolated uid.

FWIW this is already how Debian (and presumably other distros') tor packages work: tor runs as a dedicated user. Already it is possible to grant other users access to the control port (from which they can already create and remove hidden services). The reason why HS applications that create their own HSes generally run their own instance of tor as their own uid is that the hidden service data (key and hostname) written by tor is currently only readable by the tor user. There is another patch to address this issue (in progress or possibly already merged, sorry I'm not looking up the ticket right now) to allow this data to be written with permissions for another group to read it, but this ephemeral HS plan of delivering the information over the control port is obviously much better/more flexible.

From Valencia, ~leif

Then again, whichever way you give the user a way to control the Tor router opens up an attack vector for somebody who managed to break into a faulty client application. So to me the entire lets-not-trust-localhost logic doesn't work out in my head. It either produces bureaucratic complications or false positives in the warning log.

Maybe I overlooked something.

-- E-mail is public! Talk to me in private using Tor. torify telnet loupsycedyglgamf.onion DON'T SEND ME irc://loupsycedyglgamf.onion:67/lynX PRIVATE EMAIL http://loupsycedyglgamf.onion/LynX/ OR FACEBOOGLE