- tor-dev - lists.torproject.org

Configuration of tor relay using setup files (use of API via Tor Expert Bundle)
by Nathan Bliss 07 Feb '16

07 Feb '16

Hi, Is there a way to configure a bridge in tor (e.g. meek) via the config files from the command line without having to use the GUI in the Tor browser? I've been searching for documentation on this, so if I've missed it I would be grateful for a pointer to where this is in the docs... I found something about the torrc but this was an option for configuring a tor server node as a bridge whereas I want to configure the bridge as a client (in client mode). I need to do this in order to have automation scripts working with search engines such as google. Google always seems to be able to detect http connection via a script or via the tor interface running in standard mode. Only a client connection via google-meek bridge seems to work and I'm looking for a way to do this via the tor stem API or via the config files. If this feature isn't currently exposed as an API hook, I'd be happy to contribute this feature to the source code if someone can give me a few pointers as I'm new to the project and haven't yet seen any of the source [&#X1f60a] Many thanks, Nathan

2 1

Re: [tor-dev] Hashring understanding
by George Kadianakis 07 Feb '16

07 Feb '16

Ola Bini <obini(a)thoughtworks.com> writes: > Hi, > >> I think it's fine to use randomness to generate it for now; that's what tor >> also currently does [0]. > Cool - we have a local clone of the specs and updating it with this understanding. > Great! For better or worse I think making design decisions (or at least simulating them) as well as improving and finalizing the spec is a big part of this task :) >> The real lifetime of guards is currently a random value between 2 to 3 months >> (see chosen_on_date) This is regardless of whether the guard is your primary >> guard or not. We might increase the lifetime in the future, but probably the >> precise value should not matter too much since it's a constant >> change. > Does this mean the choice of guard is saved in the state file or > config file? Indeed. It is saved in the state file. Check the end of: https://lists.torproject.org/pipermail/tor-dev/2016-February/010355.html also see here for another explanation of how parts of it work: https://lists.torproject.org/pipermail/tor-dev/2014-June/007042.html > Should the PRIMARY_GUARDS chosen by #259 here be saved as well? Hmm. Not sure how this will work exactly in practice. The way I imagine it, is that the first N_PRIMARY_GUARDS active guards on your guard list are considered to be your primary guards. It's more of a label, than a static set. That's because the set of primary guards changes over time, as nodes go inactive. Like prop259 says, "by 'active' we mean that we only consider guards that are present in the latest consensus as primary". > I suspect that might not work very well because of the difference between > dystopic and utopic networks, and when people move around between > networks. Or maybe we should have two sets, UTOPIC and DYSTOPIC > PRIMARY_GUARDS that are more long lived? Hmm... Not sure how this would work if utopic and dystopic guard lists are separated. That is if there should be both dystopic primary guards _and_ entropic primary guards. I'd personally prefer the approach where we don't have such strictly separated guard lists. I tried to explain parts of it in the top of this email: https://lists.torproject.org/pipermail/tor-dev/2016-February/010355.html but it probably requires more design work. The way I'm thinking of it, we have a main guardlist GUARDS, where we put every guard we have sampled and connected or planning to connect to. We would sample both normal and 80/443 guards for this guardlist. This guardlist should work for most clients (except from 80/443 firewalled clients). To handle these 80/443 firewalled clients, if we try more than GUARDLIST_FAILOVER_THRESHOLD nodes from GUARDS and we can't connect, then we consider it to be a firewall issue, and go into "dystopic firewall mode". During this mode we only sample dystopic guards (potentially from a separate DYSTOPIC_GUARDS guardlist). If we try more than GUARDLIST_FAILOVER_THRESHOLD of guards from the dystopic guardlist, then we complain about network issues. (There are various optimizations and heuristics that can be applied to the above system, but let's not get to them now.) [The reason I don't like having separate disjoint utopic and dystopic guardlists and first trying the former and then the latter, is because it means that normal people will use the utopic guards, and the dystopic guards will only be used by people that have 80/443 firewalled. We don't really know how big these user sets are right now so it's hard to make load-balancing assumptions, and from a security PoV we should probably not isolate those user sets if we can help it.] Bleh sorry for the mess... It's been a while since I've thought of this stuff.

2 3

Re: [tor-dev] Guards selection - current algorithm simulation
by George Kadianakis 05 Feb '16

05 Feb '16

Iván Pazmiño <ipazmino(a)thoughtworks.com> writes: > Hello, thanks for your reply. > > On 02/04/2016 04:28 PM, George Kadianakis wrote: >> We don't actually and it's a pity because it would have been very useful for >> this particular case indeed. > We will write one then. > Exciting! >> >> However, the entry guard code in Tor is not particularly clean or well >> abstracted. So finding the underlying algorithm will not be a trivial task. >> That's also another reason we are trying to move to a more formalized >> algorithm, so that it's more easily studied. > We've started looking at the code and so far we believe the relevant > scenarios would be those when you invoke choose_random_entry_impl with a > NULL state. Would that be correct? > I _think_ that also the non-NULL state scenario is relevant. choose_random_entry_impl() is called with an initialized state in onion_extend_cpath(). That's the codepath when "we are building a circuit and we need an entry guard for it". choose_random_entry_impl() is called with a NULL state in add_an_entry_guard(). That's the codepath for "shit we are out of entry guards. We need a new one." I think both of these codepaths are relevant. Maybe to make sure, add some logging calls in choose_random_entry_impl() and start up your Tor, to see if it's called with a NULL state or not. > Cheers, > Iván

1 0

Re: [tor-dev] Hashring understanding
by George Kadianakis 04 Feb '16

04 Feb '16

Ola Bini <obini(a)thoughtworks.com> writes: > Hi, > > Sorry for the string of emails! > > Hopefully a simple question: > The current proposal contains logic for keeping track of network > up/down and setting timeouts for exponential backoff to test the > network again. But if I understand correctly, this proposal is > basically about replacing the algorithm used for > choose_good_entry_server() - correct? So it seems like keeping track > of network status doesn't really belong inside this algorithm at > all. Wouldn't it make sense to return a specific failure to the caller > and let the caller be in charge of when to retry? > Hmm. It's quite hard (if not impossible) for a user-land application like Tor to actually keep track of whether the network is up or down in a multiplatform, secure and scalable manner (e.g. without using the dirauths as an oracle). So instead all we do is connect to Tor nodes and check whether we could reach them or not. I think currently if a relay fails to answer a CREATE cell, we treat the relay as unreachable and we mark it as such in our guardlist (see entry_guard_register_connect_status()). This logic is considered suboptimal and it sometimes ends up marking good relays as unreachable. The retry logic in prop259 tries to compensate for this, by eventually retrying nodes that might have been marked as offline by a shaky network. I agree it would be nice to decouple the above behavior from the guard picking behavior. However, the two behaviors are certainly linked with each other, since the only nodes that a Tor client connects to are its guard nodes. I'm not sure how separating these two behaviors would look like, but if you guys think it would simplify things I'd definitely be interested in hearing about it :) WRT current code, if you see choose_random_entry_impl() (which is called by choose_good_entry_server()) you will notice that the status of relays is very central to this function since populate_live_entry_guards() will remove any guards that are inactive or previously found unreachable from the guardlist. choose_random_entry_impl() does not contain logic about the network itself being up and down, because currently Tor does not have such logic. Instead, Tor will endlessly keep on cycling through guards till the network comes up. Hopefully I answered the right question here. BTW, if you guys want we can have another meeting next week Tuesday same time iff that will be helpful to you. cheers!

1 0

Re: [tor-dev] Guards selection - current algorithm simulation
by George Kadianakis 04 Feb '16

04 Feb '16

Iván Pazmiño <ipazmino(a)thoughtworks.com> writes: > Hey guys, > > Just wondering if by any chance you have a simulator for the current > algorithm to select guards. It would be very useful to compare against > new algorithm's numbers. > > Thanks, Hello, We don't actually and it's a pity because it would have been very useful for this particular case indeed. However, the entry guard code in Tor is not particularly clean or well abstracted. So finding the underlying algorithm will not be a trivial task. That's also another reason we are trying to move to a more formalized algorithm, so that it's more easily studied.

1 0

Re: [tor-dev] Hashring understanding
by George Kadianakis 04 Feb '16

04 Feb '16

Ola Bini <obini(a)thoughtworks.com> writes: > Hi, > > Thanks for the confirmation of our understanding! Very helpful. > > If I understand things correctly, we are supposed to take > GUARDLIST_FAILOVER_THRESHOLD guards from the list of all guards and > generate the GUARDLIST from that. Is that process supposed to be > deterministic for a specific client on a specific network, or is it > fine to use randomness to generate it? > Hello, I think it's fine to use randomness to generate it for now; that's what tor also currently does [0]. I think coming up with the right way to do deterministic sampling based on specific networks can be the subject of a separate proposal :). I imagine that there will be engineering and security issues that will complicate any easy approaches. So you can ignore this part of the problem for now, and just use fresh local randomness everytime you sample a guard :] > And how long lived are the PRIMARY_GUARDS supposed to be? The real lifetime of guards is currently a random value between 2 to 3 months (see chosen_on_date) This is regardless of whether the guard is your primary guard or not. We might increase the lifetime in the future, but probably the precise value should not matter too much since it's a constant change. [0]: see choose_good_entry_server() -> router_choose_random_node() -> smartlist_choose_node_by_bandwidth_weights()

1 0

Re: [tor-dev] Hashring understanding
by George Kadianakis 04 Feb '16

04 Feb '16

Ola Bini <obini(a)thoughtworks.com> writes: > Hi, > > Sorry for this - had some questions about the hashring component of > #259 that I haven't been able to figure out myself. I'm sure it's just > me being unused to the Tor code base and how you write your proposals, > but it would be super helpful if I can get a quick answer to a few > questions. > > First, my simplified understanding of the data structures is this: > GUARDS are all the guards in the consensus > > UTOPIC_GUARDS is all utopic guards from GUARDS > DYSTOPIC_GUARDS is all dystopic guards from GUARDS > > UTOPIC_GUARDLIST is a subset of UTOPIC_GUARDS > DYSTOPIC_GUARDLIST is a subset of DYSTOPIC_GUARDS > > We will only ever choose guards to use from UTOPIC/DYSTOPIC_GUARDLIST > > The idea of the hashring is to be a structure to make it possible for us to > go from UTOPIC/DYSTOPIC_GUARDS to UTOPIC/DYSTOPIC_GUARDLIST. > The reason we want to avoid shifting is because we want the hashring to be a > long lived data structure that can be cheaply updated over p> several consensuses. > > Here are my questions: > - Is the above understanding correct? Yes, that's also my understanding. I think what prop259 tries to do with that hashring construction is to provide a deterministic way to sample guards, with minimum shifting as time passes. The aim here is to make it harder to track and fingerprint clients based on their guard lists. The idea is that when you are in WiFi "home" you always have one set of guards, and when you move to the WiFi "cafe" you have a different set of guards. See #10969 for some background and check http://meetbot.debian.net/tor-dev/2016/tor-dev.2016-01-19-16.03.log.html starting from 16:21:58 for some recent discussion. Personally, I think that the hashring is kind of a red herring in the sense that it makes prop259 more complex than it needs to be, and it also tries to tackle a problem that we don't really know how to solve. I mean even if we have the hashring construction, how do we use it? Do we just blend in the MAC of the network gateway when we sample guards? What happens if an evil gateway changes its MAC constantly, till we sample the guards it wants? I think solving the guard fingerprinting issue should be the subject of a separate proposal. Isis suggested this during the meeting but I was too dump at that point. Until we have a proper solution here I'd suggest we stick to the KISS thing, which probably is to sample guards from a simple list weighted by their bandwidth like Tor is currently doing (see node_sl_choose_by_bandwidth()). Of course, if you guys are interested in this and want to simulate the hashring idea (or any othe similar idea) that's fine as well. That's the good part of a simulation; that we don't need to commit to any particular construction. > - At what point do we add/update/change guards in UTOPIC/DYSTOPIC_GUARDLIST? I think prop259 suggests that we populate the guardlists the first time Tor attempts to connect to a network. This is different to how Tor currently does it, where it starts with a minimal guardlist (one guard), and adds new guards to it on demand (when the first guard fails). Both approaches have their merits. I'm not sure which one is better. We add more guards to guardlists if all the previous guards don't work anymore. This can be seen in step 4 of §2. We also need to mark (update) guards as unreachable when they get removed from the consensus, or when we fail to connect to them. Indeed the proposal should be updated to better specify when these actions happen and how they work. > - Since each key in the hashring structure depends acutely on bw_weight_total > and the BW of the guard, how will it be possible to update it > efficiently? My assumption is that the bw_weight_total will change on every > consensus, and at that point all the old keys are invalid?

1 0

(no subject)
by Christian Sturm 04 Feb '16

04 Feb '16

Hello, a while ago I set up a BSD buildbot for tor[1]. It's actually not just for BSDs, but also for other systems, such as Solaris. I wanted to make it easier to notice when something breaks on non-popular systems, not actively used by the developers. There also is reporting through an IRC bot and I wanted to ask if it would be interesting to have it report there. Currently it just lurks in there and lacks the permission to report build failures. So it would be great if the user 'bsdbuildbot' could get those permissions. I was wondering whether this is something that developers are interested in and if so what kind of reporting they'd like to have. Currently (if it would be allowed to) it would report individual build failures and success, but of course if that's too noisy could also change to just notify when this changes. Thanks to weasel there also is a FreeBSD builder for Jenkins. However that's only FreeBSD for now. The buildbot already has some other operating systems as build slaves and it's rather simple to set it up. Chris [1] http://buildbot.pixelminers.net/

1 1

Next (and subsequent) proposal discussion meetings
by Nick Mathewson 04 Feb '16

04 Feb '16

Hi, friends! This is a followup to the previous thread on the subject of proposal reviews. If you didn't read it, you can find it here: https://lists.torproject.org/pipermail/tor-dev/2016-January/010219.html [If you don't know how proposals work, see Lunar's explanation here: https://lists.torproject.org/pipermail/tor-dev/2016-January/010246.html ] First, the next meeting is scheduled on #tor-dev this Thursday, at 9:00 am EST (=1400 UTC). We'll be discussing these proposals: * Proposal 263: Postquantum circuit extension * Proposal 249: Large create cells Second, I'm trying to use the wiki to plan and schedule these meetings going forward. I've asked people who need or want to attend them to comment on a set of proposed times, and tried to list the people who really need to be at each one. The wiki page is at: https://trac.torproject.org/projects/tor/wiki/org/teams/NetworkTeam/Meeting… I'll try to start outlining meeting times roughly a month in advance there, and finalizing them at least a week in advance. (No promises; it will take a while to ramp up.) best wishes, -- Nick

5 7

Re: [tor-dev] Getting people started on guard selection algorithms [prop259]
by George Kadianakis 03 Feb '16

03 Feb '16

> On Tue, Feb 02, 2016 at 02:29:22PM -0500, Ola Bini wrote: >> Hi, >> >> We have now started looking at the proposal and the existing code. Our >> current plan is to first of all code more of the simulations referred >> to in stuff-to-test.txt. We also noticed that the hashring >> implementation for choosing the [DYSTOPIC/UTOPIC]_GUARDLIST hasn't >> been implemented in the simulation code - in fact, in the code it >> seems the [DYSTOPIC/UTOPIC]_GUARDS is used as the guardlist. We are >> planning on implementing the two different varieties of the guardlist >> selection algorithm as well. >> Sounds good. Another thing to note is that during our last meeting, we decided to not have the utopic/dystopic guardlists be disjoint (mainly for load balancing reasons): https://lists.torproject.org/pipermail/tor-dev/2016-January/010265.html Check the meeting logs if you care for more info. Unfortunately, I think prop259 has not been updated to specify this new behavior. This might be too researchy for what you are trying to do, but finding a nice behavior here would be very helpful. Here are is an example idea of doing the 80/443 fascist firewall detection heuristic without two disjoint guard pools: Alice initializes a single guard list with all the guard nodes. Then she does steps 1 to 4 from §2 of prop259. Then in step 5, if Alice has tried more than GUARDLIST_FAILOVER_THRESHOLD guards from her guard list, she goes into "dystopic firewall" mode. During this mode, Alice only picks 80/443 nodes as guards (maybe from a separate dystopic guardlist). If those don't work either (she tries GUARDLIST_FAILOVER_THRESHOLD of them), then Alice "should make a note to herself that the network has potentially gone down" as suggested by step 5.b. So in the above idea, there are two guardlists. Guardlist GUARDS has all the guards in the network, and then there is a DYSTOPIC_GUARDS guard list (which is a subset of GUARDS) which is only used during "dystopic firewall" mode. I think this has better load balancing and anonymity properties. But there might be even better behaviors. Feel free to come up with your own and test them! >> After that, we are going to start looking at where it fits in the main >> Tor code base. >> Sounds good. You might enjoy choose_random_entry_impl() as a starting point. Feel free to ask us any questions you have about the Tor code base. Either here or on IRC. >> The only thing I'm a bit unclear about from the specification is the >> idea of primary guards, and what the procedure is when no primary >> guards are possible - 259.§2.3 talks about "all available and fitting >> entry guards" - is this from the list of primary guards or the >> guardlist? >> IIUC it's from the whole guardlist, but this should only happen if the primary guards are unreachable. The idea with primary guards is that in an ideal world, a Tor client would always only connect to the top guard of its guardlist. To expose itself minimally to the network. Unfortunately, the network is fiddly so this is not possible because the top guard will eventually go down. The concept of primary guards tries to compensate for that, by going to extra lengths to ensure that at least you always connect to one of your N=3 top guards in your guard list. It does this by periodically checking the reachability of those top N=3 guards, and marking them online if they are (see step 2 of §2). So, even if you or your guards have reachability issues and you drift on your 12th guard or something, you will eventually come back to one of your primary guards when they are found online again. Also, for the above heuristic to work, nodes that are not listed in the latest consensus should not be considered primary guards. >> 259.§2.4 says "adds a new entry guard" - is that adding it to the list >> of primary guards or something else? >> Good question. I'm not sure what the proposal means there. Maybe isis can clarify this further? Here is an attempt to help. Hope I don't confuse you further. The currently implemented Tor guard algorithm keeps a list USED_GUARDS of the guards it has already connected to (it's also saved on disk and is the guard list you see in your Tor state file). Everytime Tor tries a new guard node, it adds it to USED_GUARDS. The top N guards of USED_GUARDS are the primary guards. If you have exhausted all of the guards in USED_GUARDS and you still can't connect, then you need to add a new node to USED_GUARDS and attempt to connect to it. I imagine that the "list of all available and fitting entry guards" referenced in step 3 is something like USED_GUARDS. (It's actually not called USED_GUARDS in the code. I just named it like this for this email.)

1 0