February 2016 - tor-dev - lists.torproject.org

Roadmap - Hidden service next generation (prop224)
by David Goulet 08 Feb '16

08 Feb '16

Hi everyone! For a while now, we've had an idea on how to proceed for the engineering part of proposal 224 (next generation hidden service). So, we've put up this pad with *5* large milestones that will in theory happen one after the other. Step 1 is shared randomness for which the code and proposal has been finalized and under review for upstream merge. Yay progress! :) You'll notice that they are pretty huge milestone. As we go forward, we'll break them down in much more details (++tickets). You can already see step 2) has a pad for that already. Our hope is to update that pad regurlarly as implementation moves forward and update tor-dev@ as well with relevant information for the community. You to can help with this huge effort, please come find us on #tor-dev! Code review, comment on designs, testing, etc..., we'll need a lot of these. https://storm.torproject.org/shared/TlPhwNbdgPwLw5h23G3gEDtkWdqxVe9Gqn3KiqV… Don't hesitate to comment on the roadmap using this thread or you can also do it in the pad (which is a "Can Comment" link). Thanks! David

2 2

Re: [tor-dev] Entry guards, primary guards, dir guards
by George Kadianakis 08 Feb '16

08 Feb '16

Ola Bini <obini(a)thoughtworks.com> writes: > Hi again, > > Two questions - first, hopefully I simple one. What are directory > guards, exactly? > > Second, we found a weird behavior in the current code > base. Specifically, when Tor first downloads the microdescriptors, it > chooses 3 guards for this with the V2Dir flag. That's a little bit > less than 1600 nodes right now. However, these three guards gets > persistes as primary guards, which means that almost all Tor users > will have primary guards among those 1600, not among the 1900 that > have the Guard flag. Is this intented behavior? > Very relevant questions. Directory guards were intoduced by proposal 207. They are basically guard nodes for directory connections, so that Tor clients expose themselves to the minimum number of directory relays when downloading the various Tor documents (microdescriptors, etc.). Currently Tor uses the same guardlist for directory guards and entry guards. That's a feature because ideally you would have a single guard that would be both your entry guard and your directory guard. Unfortunately, that's not always possible because some entry guards are not directories (they don't have the V2Dir flag) so they can't be picked as directory guards. And that's basically the weird behavior that you noticed. Because Tor pick its directory guards first and they are picked from a slightly restricted set compared the total set of guards nodes. Eventually when #12538 gets to stable Tor, all relays will be directories, and hence all guards will be potential directory guards. --- So everything is explained now except from why Tor picks 3 directory guards and not one. That's because we have decided to keep the number of directory guards to 3, instead of using a single guard like we do for normal circuits. Otherwise, that single directory guard could launch some hard-to-detect route biasing attacks. For more details see: https://lists.torproject.org/pipermail/tor-dev/2014-May/006820.html https://lists.torproject.org/pipermail/tor-dev/2014-June/006944.html So basically when Tor picks a directory guard for a directory connection, it finds the first three reachable entry guards in its guardlist, and picks one of them. It might be smart to be able to support "multiple guards" mode in the algorithms that we are currently designing. That's because we might keep the "3 directory guards" design for some more time, but even more importantly because if we ever go with a layered guards design we will need to have multiple guards in some layers. For example, consider proposal 247 which specifies a layer guard design to defend against hidden service guard discovery attacks. The proposal basically makes hidden services pick guard nodes for their second and third hops. So instead of looking like this: -> middle_1 -> middle_A -> middle_2 -> middle_B -> middle_3 -> middle_C -> middle_4 -> middle_D HS -> guard -> middle_5 -> middle_E -> Rendezvous Point -> middle_6 -> middle_F -> middle_7 -> middle_G -> middle_8 -> middle_H -> ... -> ... -> middle_n -> middle_n it will look like this: -> guard_3A_A -> guard_2_A -> guard_3A_B -> guard_3A_C -> Rendezvous Point HS -> guard_1 -> guard_3B_D -> guard_2_B -> guard_3B_E -> guard_3B_F -> Rendezvous Point As you can see in this latter design, we pick amongst "2 guard nodes" for the second hop, and amongst "6 guard nodes" for the third hop. Feel free to read the introduction of the proposal for more info. So yeah... it would be nice to support this multiple guard node design in our algorithms as well. We discussed this a bit during our proposal meeting some weeks ago but we kind of let it drop. Do you think it's possible? :)

1 0

Propsal 263 Quantum-safe Hybrid handshake for Tor, updated feature request v1.2
by Zhenfei Zhang 08 Feb '16

08 Feb '16

Hello list, We had a very constructive discussion on this proposal last Thursday. I have updated the feature request based on the feedback from the discussion. We also like to thank Roger for grammar fixes on the proposal. The diff file is available at: https://github.com/zhenfeizhang/ntru-tor/commit/dc48c2065d1772c5497cd6ad900… Major tech updates are: 1. Use SHA3 and SHAKE instead of HMAC_SHA3 2. In previous proposal, when server sends back the message, it sends a ciphertext C = ENCRYPT( P | B, QSPK), and its one-time ECC public key Y = e^y; where P is the secret that server chose, B is servers long term public key. In this new version, we also include servers short term public key in the plaintext, i.e., C = ENCRYPT( P | B | Y, QSPK) and Y is no longer send to the client separately. The client will need to decrypt C first and retrieve Y from the message. Y is not revealed to any one other than the client. This is proposed by Nick. It was also suggested to us by Aniket Kate. We thank Nick and Aniket for the suggestion. This modification will save 32 bytes of traffic as Y is no longer transmitted. Also since Y is now encrypted it will be a little harder for the attacker to attack the 25519 keys. 3. Choosing algorithms and parameters for the protocol. We have identified NTRUEncrypt and R-LWE as two candidates for this protocol. We have agreed to use NTRUEncrypt for now, in the meantime, keep the option open for "newesthope" version of R-LWE type key exchange when it becomes available. Also, for NTRUEncrypt we will be using parameters sets with SHA3, to align with Tor specs. I have modified the feature request to reflect this change. Also in the discussion we were talking about the possibility of using non-product form polynomial version of NTRUEncrypt, as this version will become patent free by Aug 2017, while the patent for product form will last for another 4 years. The main concern is that Debian will not allow patented software in the package. However, through our discussion, it turns out that we may be able to include this proposal in the next one of two release of Tor. From this point of view, both patent (basic NTRU patent, till 2017 and product form patent, till 2021) are going to be an issue if Debian does not agree with SI's patent statement. So does it make sense to keep the product form polynomials as they enables roughly 3 times faster operations on both client side and server side? Thanks for your time, and please let us know if you have any further comments/suggestions. Best regards, Zhenfei

2 1

Re: [tor-dev] Detailed algorithm
by George Kadianakis 08 Feb '16

08 Feb '16

> Hey, > > So - thinking through your comments and thoughts, I realized that the > better way to proceed was to describe the algorithm, and then put it > into a proposal format. The following is updated with most of your > comments and should be a bit clearer. I decided to remove #241 again, > since it's unclear how it actually fits together with #259. The idea > is that the algo is divided into three pieces. If failure is reported > from the algorithm, the caller can wait for a while or just restart it > from scratch immediately: > I feel that this algorithm is a step in the right direction! It's interface logic is quite similar to how Tor picks its guards, and it's more specific than the previous proposals. It's a good first step so I'll CC tor-dev so that Nick, Isis and other people can take a look if they want. After we pin this down, we also need to collect all its security properties in one place. For example, we should state clearly what security properties the thresholds GUARDS_TRY_THRESHOLD_TIME and GUARDS_FAILOVER_THRESHOLD provide. I inline various questions below. I can take a deeper look tomorrow as well. > - Start of algorithm > - Ensure USED_GUARDS is populated (from state file etc) > - Create a list of PRIMARY_GUARDS that contain N_PRIMARY_GUARDS that are not bad by: > - Taking the next entry from USED_GUARDS > - If USED_GUARDS is empty: > - randomly select an entry from UTOPIC_GUARDS, weighted by bandwidth Hmm. PRIMARY_GUARDS stays like this for ever? It probably needs to be updated when a primary guard get removed from the consensus (become bad), and put the next entry of USED_GUARDS in its place. Maybe we need another event (and algorithm) for "receive new consensus"? Or is this considered out of scope? > - Set TRIED_GUARDS to be an empty set Hm. What is the point of TRIED_GUARDS? It's like a list of guards that we found unreachable at some point in time. It's never cleaned, so it will eventually accumulate many nodes. IIUC, we only use that list to enforce our thresholds by looking at timestamps. Can we do without this list, just by checking the timestamps of unreachable USED_GUARDS? Or the logic will be much more complicated? > - Set TRIED_DYSTOPIC_GUARDS to be an empty set > - Set state = STATE_PRIMARY_GUARDS > > - Each iteration of algorithm > - If it was at least 3 minutes since we tried the primary guards and we are not in STATE_PRIMARY_GUARDS: > - save previous state > - set state = STATE_PRIMARY_GUARDS > > - STATE_PRIMARY_GUARDS: > - return each entry in PRIMARY_GUARDS in turn > - mark each entry as "unreachable" if algorithm doesn't terminate What is the "algorithm" here? Is it the guard selection algorithm, or the circuit construction algorithm? I guess the latter? If it's indeed the circuit construction algorithm, then the steps here are not asynchronous anymore, which diverges from how Tor networking is designed. Maybe this could be fixed by introducing some new states for when we are waiting for a circuit to connect and we are now learning whether it was succesful or not. Basically what entry_guard_register_connect_status() does in Tor right now. For example we could have STATE_WAITING_FOR_PRIMARY_GUARD_CIRCUIT for when we are waiting for a primary guard circuit to connect. If it fails, then we need to mark the entry as unreachable. Maybe we also need STATE_WAITING_FOR_UTOPIC_GUARD_CIRCUIT etc. > - restore previous state (or STATE_TRY_UTOPIC if no previous) > > > - STATE_TRY_UTOPIC: > - for each entry in TRIED_GUARDS that was marked as unreachable more than 20 minutes ago > - add to the front of remaining UTOPIC_GUARDS Isn't it already in UTOPIC_GUARDS? I don't see us ever removing stuff from UTOPIC_GUARDS. BTW, what is this step supposed to do? Setup previously TRIED_GUARDS to be retried? > - return each remaining entry from USED_GUARDS in turn > - for each entry, if algorithm doesn't terminate > - mark entry as "unreachable" > - add entry to TRIED_GUARDS Add it there even if it it's already there (TRIED_GUARDS is never cleaned)? Or maybe update a timestamp if it's already there? How to specify this nicely? > - if the number of entries in TRIED_GUARDS that were tried within GUARDS_TRY_THRESHOLD_TIME > is larger than GUARDS_TRY_THRESHOLD, return failure from the algorithm > - if the number of entries in TRIED_GUARDS is larger than a GUARDS_FAILOVER_THRESHOLD > proportion of UTOPIC_GUARDS, set state = STATE_TRY_DYSTOPIC > - return each remaining entry from UTOPIC_GUARDS randomly selected, weighted by bandwidth > - for each entry, if algorithm doesn't terminate > - mark entry as "unreachable" > - add entry to TRIED_GUARDS > - if the number of entries in TRIED_GUARDS that were tried within GUARDS_TRY_THRESHOLD_TIME > is larger than GUARDS_TRY_THRESHOLD, return failure from the algorithm > - if the number of entries in TRIED_GUARDS is larger than a GUARDS_FAILOVER_THRESHOLD > proportion of UTOPIC_GUARDS, set state = STATE_TRY_DYSTOPIC > TRIED_GUARDS is never cleaned, so this will be almost always true after a while. > - STATE_TRY_DYSTOPIC: > - for each entry in TRIED_DYSTOPIC_GUARDS that was marked as unreachable more than 20 minutes ago > - add to the front of remaining UTOPIC_GUARDS Does it matter if we add it to the front or the back of UTOPIC_GUARDS? >From what I understand, we always sample from UTOPIC_GUARDS based on relay bandwidth so the list order should not matter, right? [This comment and also the comments below also apply for the STATE_TRY_UTOPIC case above] > - return each remaining DYSTOPIC entry from USED_GUARDS in turn Do we even try previously-found unreachable guards here? > - for each entry, if algorithm doesn't terminate > - mark entry as "unreachable" > - add entry to TRIED_DYSTOPIC_GUARDS Order matters here right? We probably want FIFO. > - if the number of entries in TRIED_GUARDS+TRIED_DYSTOPIC_GUARDS that were tried within GUARDS_TRY_THRESHOLD_TIME > is larger than GUARDS_TRY_THRESHOLD, return failure from the algorithm > - if the number of entries in TRIED_DYSTOPIC_GUARDS is larger than a GUARDS_FAILOVER_THRESHOLD > proportion of DYSTOPIC_GUARDS: > - mark all guards in PRIMARY_GUARDS, TRIED_GUARDS and TRIED_DYSTOPIC_GUARDS as not "unreachable" > - return failure from the algorithm > - return each remaining entry from DYSTOPIC_GUARDS randomly selected, weighted by bandwidth > - for each entry, if algorithm doesn't terminate > - mark entry as "unreachable" > - add entry to TRIED_DYSTOPIC_GUARDS > - if the number of entries in TRIED_GUARDS+TRIED_DYSTOPIC_GUARDS that were tried within GUARDS_TRY_THRESHOLD_TIME > is larger than GUARDS_TRY_THRESHOLD, return failure from the algorithm > - if the number of entries in TRIED_DYSTOPIC_GUARDS is larger than a GUARDS_FAILOVER_THRESHOLD > proportion of DYSTOPIC_GUARDS: > - mark all guards in PRIMARY_GUARDS, TRIED_GUARDS and TRIED_DYSTOPIC_GUARDS as not "unreachable" Hm, not sure what this is supposed to do. Is it trying to setup the algorithm to retry all guards from the beginning? Shouldn't we edit USED_+GUARDS in that case? > - return failure from the algorithm > Instead of failing in this case, shouldn't we just retry from the top without adding any new guards if we hit these thresholds? How do we do this? Or is this logic considered out of scope? :x > - End of algorithm > - If circuit is set up correctly, let algorithm know > - Algorithm marks the guard chosen as used and makes sure it is in USED_GUARDS > - Otherwise do another run of the algorithm > When does this "End of algorithm" happen?

1 0

Configuration of tor relay using setup files (use of API via Tor Expert Bundle)
by Nathan Bliss 07 Feb '16

07 Feb '16

Hi, Is there a way to configure a bridge in tor (e.g. meek) via the config files from the command line without having to use the GUI in the Tor browser? I've been searching for documentation on this, so if I've missed it I would be grateful for a pointer to where this is in the docs... I found something about the torrc but this was an option for configuring a tor server node as a bridge whereas I want to configure the bridge as a client (in client mode). I need to do this in order to have automation scripts working with search engines such as google. Google always seems to be able to detect http connection via a script or via the tor interface running in standard mode. Only a client connection via google-meek bridge seems to work and I'm looking for a way to do this via the tor stem API or via the config files. If this feature isn't currently exposed as an API hook, I'd be happy to contribute this feature to the source code if someone can give me a few pointers as I'm new to the project and haven't yet seen any of the source [&#X1f60a] Many thanks, Nathan

2 1

Re: [tor-dev] Hashring understanding
by George Kadianakis 07 Feb '16

07 Feb '16

Ola Bini <obini(a)thoughtworks.com> writes: > Hi, > >> I think it's fine to use randomness to generate it for now; that's what tor >> also currently does [0]. > Cool - we have a local clone of the specs and updating it with this understanding. > Great! For better or worse I think making design decisions (or at least simulating them) as well as improving and finalizing the spec is a big part of this task :) >> The real lifetime of guards is currently a random value between 2 to 3 months >> (see chosen_on_date) This is regardless of whether the guard is your primary >> guard or not. We might increase the lifetime in the future, but probably the >> precise value should not matter too much since it's a constant >> change. > Does this mean the choice of guard is saved in the state file or > config file? Indeed. It is saved in the state file. Check the end of: https://lists.torproject.org/pipermail/tor-dev/2016-February/010355.html also see here for another explanation of how parts of it work: https://lists.torproject.org/pipermail/tor-dev/2014-June/007042.html > Should the PRIMARY_GUARDS chosen by #259 here be saved as well? Hmm. Not sure how this will work exactly in practice. The way I imagine it, is that the first N_PRIMARY_GUARDS active guards on your guard list are considered to be your primary guards. It's more of a label, than a static set. That's because the set of primary guards changes over time, as nodes go inactive. Like prop259 says, "by 'active' we mean that we only consider guards that are present in the latest consensus as primary". > I suspect that might not work very well because of the difference between > dystopic and utopic networks, and when people move around between > networks. Or maybe we should have two sets, UTOPIC and DYSTOPIC > PRIMARY_GUARDS that are more long lived? Hmm... Not sure how this would work if utopic and dystopic guard lists are separated. That is if there should be both dystopic primary guards _and_ entropic primary guards. I'd personally prefer the approach where we don't have such strictly separated guard lists. I tried to explain parts of it in the top of this email: https://lists.torproject.org/pipermail/tor-dev/2016-February/010355.html but it probably requires more design work. The way I'm thinking of it, we have a main guardlist GUARDS, where we put every guard we have sampled and connected or planning to connect to. We would sample both normal and 80/443 guards for this guardlist. This guardlist should work for most clients (except from 80/443 firewalled clients). To handle these 80/443 firewalled clients, if we try more than GUARDLIST_FAILOVER_THRESHOLD nodes from GUARDS and we can't connect, then we consider it to be a firewall issue, and go into "dystopic firewall mode". During this mode we only sample dystopic guards (potentially from a separate DYSTOPIC_GUARDS guardlist). If we try more than GUARDLIST_FAILOVER_THRESHOLD of guards from the dystopic guardlist, then we complain about network issues. (There are various optimizations and heuristics that can be applied to the above system, but let's not get to them now.) [The reason I don't like having separate disjoint utopic and dystopic guardlists and first trying the former and then the latter, is because it means that normal people will use the utopic guards, and the dystopic guards will only be used by people that have 80/443 firewalled. We don't really know how big these user sets are right now so it's hard to make load-balancing assumptions, and from a security PoV we should probably not isolate those user sets if we can help it.] Bleh sorry for the mess... It's been a while since I've thought of this stuff.

2 3

Re: [tor-dev] Guards selection - current algorithm simulation
by George Kadianakis 05 Feb '16

05 Feb '16

Iván Pazmiño <ipazmino(a)thoughtworks.com> writes: > Hello, thanks for your reply. > > On 02/04/2016 04:28 PM, George Kadianakis wrote: >> We don't actually and it's a pity because it would have been very useful for >> this particular case indeed. > We will write one then. > Exciting! >> >> However, the entry guard code in Tor is not particularly clean or well >> abstracted. So finding the underlying algorithm will not be a trivial task. >> That's also another reason we are trying to move to a more formalized >> algorithm, so that it's more easily studied. > We've started looking at the code and so far we believe the relevant > scenarios would be those when you invoke choose_random_entry_impl with a > NULL state. Would that be correct? > I _think_ that also the non-NULL state scenario is relevant. choose_random_entry_impl() is called with an initialized state in onion_extend_cpath(). That's the codepath when "we are building a circuit and we need an entry guard for it". choose_random_entry_impl() is called with a NULL state in add_an_entry_guard(). That's the codepath for "shit we are out of entry guards. We need a new one." I think both of these codepaths are relevant. Maybe to make sure, add some logging calls in choose_random_entry_impl() and start up your Tor, to see if it's called with a NULL state or not. > Cheers, > Iván

1 0

Re: [tor-dev] Hashring understanding
by George Kadianakis 04 Feb '16

04 Feb '16

Ola Bini <obini(a)thoughtworks.com> writes: > Hi, > > Sorry for the string of emails! > > Hopefully a simple question: > The current proposal contains logic for keeping track of network > up/down and setting timeouts for exponential backoff to test the > network again. But if I understand correctly, this proposal is > basically about replacing the algorithm used for > choose_good_entry_server() - correct? So it seems like keeping track > of network status doesn't really belong inside this algorithm at > all. Wouldn't it make sense to return a specific failure to the caller > and let the caller be in charge of when to retry? > Hmm. It's quite hard (if not impossible) for a user-land application like Tor to actually keep track of whether the network is up or down in a multiplatform, secure and scalable manner (e.g. without using the dirauths as an oracle). So instead all we do is connect to Tor nodes and check whether we could reach them or not. I think currently if a relay fails to answer a CREATE cell, we treat the relay as unreachable and we mark it as such in our guardlist (see entry_guard_register_connect_status()). This logic is considered suboptimal and it sometimes ends up marking good relays as unreachable. The retry logic in prop259 tries to compensate for this, by eventually retrying nodes that might have been marked as offline by a shaky network. I agree it would be nice to decouple the above behavior from the guard picking behavior. However, the two behaviors are certainly linked with each other, since the only nodes that a Tor client connects to are its guard nodes. I'm not sure how separating these two behaviors would look like, but if you guys think it would simplify things I'd definitely be interested in hearing about it :) WRT current code, if you see choose_random_entry_impl() (which is called by choose_good_entry_server()) you will notice that the status of relays is very central to this function since populate_live_entry_guards() will remove any guards that are inactive or previously found unreachable from the guardlist. choose_random_entry_impl() does not contain logic about the network itself being up and down, because currently Tor does not have such logic. Instead, Tor will endlessly keep on cycling through guards till the network comes up. Hopefully I answered the right question here. BTW, if you guys want we can have another meeting next week Tuesday same time iff that will be helpful to you. cheers!

1 0

Re: [tor-dev] Guards selection - current algorithm simulation
by George Kadianakis 04 Feb '16

04 Feb '16

Iván Pazmiño <ipazmino(a)thoughtworks.com> writes: > Hey guys, > > Just wondering if by any chance you have a simulator for the current > algorithm to select guards. It would be very useful to compare against > new algorithm's numbers. > > Thanks, Hello, We don't actually and it's a pity because it would have been very useful for this particular case indeed. However, the entry guard code in Tor is not particularly clean or well abstracted. So finding the underlying algorithm will not be a trivial task. That's also another reason we are trying to move to a more formalized algorithm, so that it's more easily studied.

1 0

Re: [tor-dev] Hashring understanding
by George Kadianakis 04 Feb '16

04 Feb '16

Ola Bini <obini(a)thoughtworks.com> writes: > Hi, > > Thanks for the confirmation of our understanding! Very helpful. > > If I understand things correctly, we are supposed to take > GUARDLIST_FAILOVER_THRESHOLD guards from the list of all guards and > generate the GUARDLIST from that. Is that process supposed to be > deterministic for a specific client on a specific network, or is it > fine to use randomness to generate it? > Hello, I think it's fine to use randomness to generate it for now; that's what tor also currently does [0]. I think coming up with the right way to do deterministic sampling based on specific networks can be the subject of a separate proposal :). I imagine that there will be engineering and security issues that will complicate any easy approaches. So you can ignore this part of the problem for now, and just use fresh local randomness everytime you sample a guard :] > And how long lived are the PRIMARY_GUARDS supposed to be? The real lifetime of guards is currently a random value between 2 to 3 months (see chosen_on_date) This is regardless of whether the guard is your primary guard or not. We might increase the lifetime in the future, but probably the precise value should not matter too much since it's a constant change. [0]: see choose_good_entry_server() -> router_choose_random_node() -> smartlist_choose_node_by_bandwidth_weights()

1 0