April 2016 - tor-dev - lists.torproject.org

txtorcon: fixing APIs, painting bikesheds
by meejah 11 Apr '16

11 Apr '16

There are several rough edges and "regrettable" API choices in txtorcon that I'd like to fix as a "1.x" release. I also had several productive discssions with Brian Warner (thanks a lot!) suggesting some things to make using txtorcon easier. To that end, I am collecting some of these thoughts as well as massively re-organizing (re-doing?) the documentation in a branch called "release-1.x". ReadTheDocs is building this now, too. Some highlights of what I'm doing or find annoying currently: - Move a bunch of things to "private" attrs/functions that were never supposed to be public (i.e. prefix a bunch more things with "_") - Eliminate need to deal with the "bootstrap" Deferreds (via factory functions and/or other help methods) (mostly this is true already) - Python3 (works in release-1.x branch now, minus txsocksx) - Re-org docs, add "programming guide" (instead of walkthrough, tutorial and HOWTO) - consolidate examples (there's too many, and some are confusing) - introduce a "Tor" object that abstracts "a running tor instance" (whether launched by txtorcon or just connected-to) as a Builder or Factory for other useful objects. This would be "the" high-level API that most users would interact with. Something like: @inlineCallbacks def main(reactor): tor = yield txtorcon.launch(reactor, ...) # or .connect() torstate = yield tor.create_state() print("all circuits:", torstate.circuits.values()) endpoint = tor.create_client_endpoint('torproject.org', 443) onion = tor.create_onion_endpoint(port=80, private_key=None) # etc. This would get rid of "launch_tor", "build_local_tor_connection" and others, giving just "launch()" or "connect()". get_global_tor() would also return a "Tor" instance. Also, launch() would *not* take an arbitrary TorConfig (as launch_tor does now) but instead would expose a very few critical options as kwargs (and if you need to mess around with obscure options, you'd do so after launch). - .get_info() should never have returned a dict (instead: list of results, in same order as keys) - consolidate all the various types of "onion/hidden-service" things to support ephemeral and "on disk" services, as well as authenticated. There are currently conceptually 6 variants from tor: ephemeral: plain, basic-auth (not in tor), stealth-auth (not in tor) on-disk: plain, basic-auth, stealth-auth This is further complicated by the fact that both the authenticated services are really, logically, a "list of clients" each of which has a name, a secret (and for stealth, a separate .onion address). My current design *ideas* are in the release-1.x branch, but boil down to interfaces: IOnionClients (for basic + stealth auth) which is a collection of IOnionClient instances (has secret + name, and is a subclass of IOnionService) and IOnionService for all the rest (has hostname, ports, private-key). This stuff is the most in-flux still. "create a new onion service" would be available at the high level via one or several method-calls on Tor() instances which would return new concrete IStreamClientEndpoint or IStreamServerEndpoint implementers - For "I want a Stream over Circuit X", add a ".stream_to(endpoint)" method on Circuit instances that gives you a Twisted IStreamClientEndpoint implementation causing the stream resulting from .connect() to be attached to that specific circuit. - Introduce a configurable "CircuitBuilder" to more-easily control construction of "types of" circuits (instead of just "build a circuit through these exact relays" as .build_circuit() provides). For more, see: https://github.com/meejah/txtorcon/issues/125 - re-consider all the magic (attribute-style access) of TorConfig - Add a "composible"-style API for listening to Circuit and Stream events (instead of having to implement ICircuitListener etc). I would be happy to hear of (other) existing APIs that are awkward or confusing as well as opinions on the general colour of my bikeshed. Please beware that this branch is very much "in progress" ;) As far as current users of 0.x releases: I will not break these, and will backport any critical fixes to them. My preference would be to simply work with existing users (e.g. via pull-requests to your repos) to let them seamlessly upgrade to 1.x as soon as 1.0.0 is pushed out the door. Where possible I can provide wrappers for backwards-compatiblity for any removed/re-named methods My goal here is *not* to just gratuitously break things "because aesthetics" but to make the "API surface" of txtorcon smaller, easier to understand and better documented. How to help: - look at http://txtorcon.readthedocs.org/en/release-1.x/ (note that some of the docs are lies as the code doesn't always exist yet, or worse yet exists but isn't what I want) - express opinions (here, #tor-dev, via any of the methods at https://meejah.ca/contact) - *maaaybe* look at the new code (but I'm trying to write the docs first, so those are the most-useful to look at). I want the docs to reflect the upcoming reality, and will change the code to match. Thanks, meejah

1 1

Notes from the prop224 proposal reading group
by George Kadianakis 09 Apr '16

09 Apr '16

Hello, so we had a meeting about the future of "Next Generation Hidden Services" aka prop224. It was a good meeting. We spent most of the time discussing the topics brought up here: https://lists.torproject.org/pipermail/tor-dev/2016-March/010534.html Please read the above mail to get up to speed with the topics of discussion. So these are basically two topics: a) Should we try to maintain backwards compatibility with old intro/rend points? We ended up deciding that it's actually worthwhile keeping backwards compatibility if we can help it: Our rationale was that coding the logic for the extra cells that are required to maintain backwards compatibility is probably less painful than coordinating a perfect bug-free incremental deployment of prop224 in such a way that the whole network has upgraded before we enable client support. So our current plan is to bake backwards compat in the code, and also introduce a consensus parameter of "Should we connect to prop224 onion services using legacy IPs/RPs?" which we will switch off when a good percentage of the network has upgraded to support prop224. At that point, we will be able to rip out the backwards compat code as well. As an added point to that, we should implement the backwards compat code in a way that it's not spagghetied with the normal code, so that it's easy to analyze and remove when needed. Furthermore, we should look back in prop224 and see if we can simplify the backwards compat logic even more. FWIW, prop224 does not have backwards compatibility for HSDirs, so we will still have to do careful deployment there, and make sure that a good percentage of HSDirs have upgraded to prop224 before we start using them (maybe a new consensus parameter is needed here). b) In prop224, why do intro points need to know the "intro point encryption key" and also what's the point of UPDATE-KEYS-SUBCMD? Nick told us that the main point of UPDATE-KEYS-SUBCMD is that so hidden services can rotate their intro point encryption key periodically, so that they can reset their replay caches. That's a fair point. The big question here is, is this worth the complexity that MAINT_INTRO and UPDATE-KEYS-SUBCMD add to the protocol logic? Unfortunately we don't know the answer to this yet, because we actually don't have stats on how our replay caches are doing currently. Here are some things we need to look into: i) Speak with facebook or some other big onion hoster, and try to get some information about their replay cache. Is it ready to bust? How much space is it occupying? ii) The above will help us gauge replay caches in terms of normal activity, but we also need to think of how replay caches should act in cases of a DoS. iii) Also think about INTRODUCE1 replay attacks in general, and understand how dangerous they can be, so that we know how robust our replay caches should be. Is it _that_ dangerous if an attacker manages to sneak in a replay attack every once in a while? For example, imagine an attacker who fills up our replay cache on purpose, so that she can sneak in one replay attack when we reset it. If an attacker can send legit INTRODUCE2 cells to fill up our replay cache, what does she need the replays for? What other types of attacker are there? After we learn more about the above, we will understand *how much we care* about onion services being able to rotate intro pint encryption keys on the fly using UPDATE-KEYS-SUBCMD. Here are some possible scenarios here: - We end up deciding that INTRODUCE1 replay attacks are *dangerous* and we need to seriously defend ourselves against them. In this case, having some sort of UPDATE-KEYS-SUBCMD mechanism sort of makes sense, since onion services should be able to rotate keys at will and let their clients know, without them having to fetch a new HS descriptor. - We end up deciding that INTRODUCE1 are a concern but not something we should complicate our design considerably for. In this case, we might want to consider improving the quality of our replay caches, by using more compact designs (= more space for entries), or by using bloom filters or a similar tech. A proposal will need to be made if so. Yawning posted the following relevant paper: https://home.kookmin.ac.kr/~mkyoon/publications/2010TKDE.pdf - We end up deciding it's no big deal if the attacker sneaks in a replay attack every once in a while. In this case, we can just go with the most minimal approach and ditch the UPDATE-KEYS-SUBCMD mechanism for now, and have intro points be oblivious about encryption keys (which further simplifies our cell formats, etc.). If we take this approach, and in the future future we change our minds about the importance of replay attacks, we can still add some sort of UPDATE-KEYS-SUBCMD mechanism using the extensions mechanism of ESTABLISH_INTRO cells. Furthermore on this topic, we discussed some alternate approaches for doing replay detection that would require less heavy replay caches: i) Maybe we can have an increasing counter in the HS descriptor, that clients need to also include in their INTRODUCE1 cell. If an onion service receives an INTRODUCE1 cell with a counter value different than the one in the descriptor, it ignores it. Onion services update the counter and publish a new descriptor when their replay cache is close to getting filled. I believe various subtle engineering details are hidden in this scheme, along with edge cases and race conditions like "What happens if the client fetches a descriptor with counter value N, right before the onion service updates the descriptor with a new counter value?". ii) When an intro point replay cache is almost full, the onion service will make new encryption keys, tear down the old intro circuit and make a new one, and also pass a signed "HS moved to another circuit" message to the intro point. The intro point will forward that message to any clients that ask for the onion service using its old keys, so that clients know the right place to ask. Lots of engineering details hidden in this proposal as well. In the end we decided that Roger might have some useful feedback for us here :] More research is required! Cheers!

4 9

Using Let's Encrypt for meek bridges
by David Fifield 09 Apr '16

09 Apr '16

I'm looking for ideas of good ways to handle TLS certificates and their renewal for meek bridges. I want to use Let's Encrypt for this process, and I hope that someone who knows Let's Encrypt well can contribute some ideas. All three of the meek bridges use HTTPS to receive connections from the CDN, so they need TLS certificates. For example, when you use meek-azure, your traffic is forwarded to the bridge at https://meek.bamsoftware.com/. How it works now is I do the usual domain validation procedure with a CA, receive an email to show that I control the domain, install the cert and key, and then run the server like this: meek-server --cert /etc/meek/cert.pem --key /etc/meek/key.pem When I used Let's Encrypt in the past, using the --webroot option, it wanted to write a file to the URL path "/.well-known/acme-challenge/{token}". That won't work for meek-server as it exists today, because meek-server never serves files from the filesystem. But it could. Perhaps we could add an option like --acme-webroot that would allow serving files from a single whitelisted directory. I notice that there are other ways of proving domain ownership (HTTP, TLS SNI, DNS). Maybe we could use one of those? https://ietf-wg-acme.github.io/acme/#identifier-validation-challenges I also note that there are third-party plugins: https://github.com/letsencrypt/letsencrypt/wiki/Plugins Maybe there could be a plugin for meek-server (ideally not requiring new complicated code in meek-server itself). Currently you have to restart meek-server in order to make it notice a changed certificate and key file. It would be better if that were not necessary--maybe we could periodically stat the files, and re-load them if they have changed? This is going to be an issue for Snowflake as well, because we will want to use WebSocket over TLS for the server component.

2 2

Introduction: CollecTor Improvement Project
by iwakeh 08 Apr '16

08 Apr '16

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 Hi there! With the first nice spring days in some parts of the Northern Hemisphere the CollecTor Improvement Project started. You probably know that CollecTor [0] is the friendly data-collecting service in the Tor network. It fetches data from various nodes and services in the public Tor network and makes it available to the world. The current project aims at making CollecTor more robust against data loss and improve its source base, development structure, as well as its installation procedure and configuration. More details can be found in the project's wiki page [1]. Further general discussion of the CollecTor Improvement Project will take place in the metrics-team mailing list [2]. So, if you're interested your can subscribe there, too. Cheers, iwakeh [0]: https://collector.torproject.org/ [1]: https://trac.torproject.org/projects/tor/wiki/doc/CollecTor/Improvements [2]: https://lists.torproject.org/cgi-bin/mailman/listinfo/metrics-team -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iF4EAREIAAYFAlcHjlgACgkQsANAxTOj/aYIVQD+KwlYx5FFXy+uiCtF8wCIAfeA SJQ3Fnk4YQC598/jo9QA/RUaZ/Z99Yjey8DsniBZTOJ/lTNlCOcKpXyXLIaOezOB =CSaG -----END PGP SIGNATURE-----

2 1

Research on Breaking reCAPTCHA
by bancfc＠openmailbox.org 08 Apr '16

08 Apr '16

While I don't understand why researchers are doing free labor for Google its still an interesting read on the innards of reCAPTCHAs and its weaknesses. Some tests were done with Tor: http://www.cs.columbia.edu/~polakis/papers/sivakorn_eurosp16.pdf

1 0

Re: [tor-dev] Update on 259
by Ola Bini 08 Apr '16

08 Apr '16

Hey, > > - OrPort vs DirPort > > ORPort is used for regular circuits, while DirPort is used when getting directory information. We need to interpret reachable stuff > > differently depending on the purpose. > > > > I'm not actually sure what the comment means here. This was more for our own benefit. The OrPort vs DirPort distinction has been a bit complicated so far. The comment basically means, when we are looking up directory information, we should use the DirPort to decide reachability and so on instead, correct? > Ensuring a min percentage of dirguards in our sampled set could work. Then, > when we need a directory guard, we could filter the sampled set and only > examine guards that can do directory requests. Yeah, we talked about this yesterday and our current thinking is to have a sampled set that contains every kind of thing, and then we dynamically filter it based on config and so on during START. > Hm, are you talking about the guardlists here? What's the question? > > BTW, if we have the ability to do "ensure a min percentage of X in our sampled > set", couldn't we just ensure a min percentage of dystopic guards in our > sampled set? And forget about the two separate guardlists? > > In this case we can have the percentage value be the actual portion of the > network that is dystopic guards. So if 20% of the total guard bandwidth is > dystopic, we could ensure that at least 20% of our sampled set is > dystopic". Well, the problem is really that the idea of dystopic doesn't necessarily make sense, since it's so dependent on the current network position of the client. Our current thinking is to do away with that concept as well. =) > > - DYSTOPIC - is there value in trying 80 and 443? > > Probably not. > > > > What does "trying" mean in this case? Falling back to guards with 80 and 443. > Restart pending guard selection algorithms on a SIGHUP? Plausible, but I don't > know how hard it would be to implement this. Well, the alternative is to just finish the running guard selections with the old settings, but use the new settings for new algorithm instances. > That's not very nice because the USED_GUARDS set that was created when > ClientsUseIPv6 or FascistFirewall were on will have reduced diversity. Then > even if we switch off those options, we are still stuck with reduced diversity. > > I'm not sure what's the right way to do this here! > > We could imagine having multiple USED_GUARDS sets, where we make a new set for > each possible filter. This might be worth considering, but I imagine there will > be technical difficulties. e.g. when a guard goes down, you need to update its > state in all the USED_GUARDS sets that it's in. Also, a person who toggles the > FascistFirewall option frequently, will end up using two different sets of > guards all the time which is suboptimal. Well, one thing you could do is hash the settings (and maybe also reachable ports) and use that as a key to differentiate the different USED_GUARDS. That would solve the problem, but might lead to a single client using lots of different guards in different locations. Might that be OK? > > - Can we make the lists smaller? > > Probably. Maybe a sampled set of 30 guards? Or 1.5%? > > > > Plausible. However, if we take the filtering approach but use a small sampled > guards list, it could happen that the list is not able to satisfy some of our > filtering restrictions. > > e.g. maybe in our 30 guards there are no IPv6 guards at all, and the user just > turned on ClientUseIPv6. What to do now? > > This is important to understand, because currently there is no mechanism to add > stuff to the sampled guards list if a restriction cannot be satisfied. So what > will Tor do, if a user enables ClientsUseIPv6 _and_ FascistFirewall but there > are no IPv6+80/443 guards in our sampled guards list? Yeah, we talked about that yesterday. Our suggestion is to do something like this: - if the filtered/reduced sample-set contains less than X (5?) guards, expand SAMPLED guards using the regular process. - If SAMPLE guards reach SAMPLED_MAX (50?) size, we fail closed with an error saying something like "your current network settings make it impossible for us to safely choose an entry guard. If you really need to connect under these circumstances, consider explicitly setting the EntryGuards configuration option" > I think it asks "What happens when guards in our sampled set drop out of the > consensus and get marked as bad?" (see bad_since in entrynodes.c) . > > This is also a great question. Especially when combined with the planned > "ensure a min percentage of X in our sampled set" logic. > > Like, what happens if suddenly most of our sampled IPv6 guards drop out of the > consensus when we have ClientsUseIPv6 on? Should we replace them? And if yes, > don't we need to replace them with other IPv6 guards to maintain the minimum > percentage? Well, I think if we replace, we should just replace randomly just like we always expand the sampled set. If most ipv6 guards drop out, we will have fewer ipv6 guards in our sampled set, but that also reflects the Tor network. I suspect a plausible thing to do is to wait a few consensus rounds with expanding the sampled set to replace "bad" guards - they might come back, and under most circumstances we shouldn't need to use the sampled set anyway. > The current proposal says the following about SAMPLED_UTOPIC_GUARDS: > > It will be filled in by the algorithm if it's empty, or if it contains > less than SAMPLE_SET_THRESHOLD guards after winnowing out older guards. > > which I think is a good suggestion. However, what should we do if we end up > going with the "ensure minimum percentage" logic? Yeah, my suggestion is not have minimum percentage of X type guards - instead just fill it randomly, and use the expanding process if we can't find enough guards for a specific purpose. > > - EntryNodes > > If this is set, never use the algorithm for regular circuits - we should still use it for directory server connections though. > > If this is set we should not use our algorithm, but we should instead pick one > of the guards in the EntryNodes list. This is for people who want to hardcode > their guard. It's used a lot by people currently. Yeah. Is the guard picked randomly from this list, or using something more complicated? > > - UseEntryGuardsAsDirGuards > > I don't understand exactly what this settings does. > > I'm not sure either. I'd just let it keep the exact same semantics it currently has. Yeah, except we don't exactly understand what does semantics are, and if we need to change something in our code to match it. =) Thanks for all the feedback! Hopefully we're getting close to the final iteration of this spec. =) Cheers -- Ola Bini (https://olabini.se) "Yields falsehood when quined" yields falsehood when quined.

5 10

Tor debugging
by SMTP Test 07 Apr '16

07 Apr '16

Hi all, I am working on a project that needs to do some modifications on the source code of Tor, to add functionalities to Tor relay. To debug I used chutney to setup Tor network locally, and checked the logs of the relay. But it is still painful and slow: I need to restart the script, and wait the network to be up every time I make changes. My question is: is there any fast way to test/debug the functionalities of Tor relay? Thanks!

2 1

extension for chrome and mozilla
by Mukul Gupta 07 Apr '16

07 Apr '16

hello tor-dev I want to make extension which can control network in my browser. so that i can connect to my some other computer which has VPN . And after that the data is exchanged between VPN server and the user. And the intermediate nodes won't be able to identify whats going on. This is my project proposal. Can you help me in this how to initiate this project.

1 0

Re: [tor-dev] Update on 259
by tdruiva＠riseup.net 06 Apr '16

06 Apr '16

Adding tor-dev. On 04/06/2016 08:53 AM, Tania Silva wrote: > Hey, > >>>>>> Yeah, we talked about that yesterday. Our suggestion is to do >>>>>> something like this: >>>>>> - if the filtered/reduced sample-set contains less than X (5?) guards, >>>>>> expand SAMPLED guards using the regular process. >>>>>> - If SAMPLE guards reach SAMPLED_MAX (50?) size, we fail closed with >>>>>> an error saying something like "your current network settings make it >>>>>> impossible for us to safely choose an entry guard. If you really need >>>>>> to connect under these circumstances, consider explicitly setting the >>>>>> EntryGuards configuration option" >>>> >>>> Oh, wow, I don't think failing closed is a good idea. >>>> It means users that move around a lot (and clients which have a longer state history) could fail at some arbitrary time. >>>> Why not simply continue to add guards that satisfy the restrictions? >> Well, users that move around a lot will only have an expanded sampled >> set if they move between several different networks that have severe >> restrictions - but mutually exclusive such restrictions. And we would >> only ever hit this fail closed if we can't find anything in the >> sampled set that matches the current needed restrictions. If we keep >> adding guards, the idea of the sampled set as a measure to minimize >> exposure to too many guards fly out the window. >> >> The problem really comes down to this - if you have a network that is >> actively firewalling every guard that is not under their control, if >> we keep expanding we will sooner or later be forced to use a guard >> under adversary control. By failing closed, we can avoid that >> eventuality. > > We thought in to have a minimum size for this set after been filtered [1]. > > ``` > MINIMUM_FILTERED_SAMPLE_SIZE > The minimum size of the sampled set after filtering out nodes based on > client configuration (FILTERED_SAMPLED). Proposed value is ???. > ``` > > In order to expose less guards when we are expanding sample set, we > thought in to have a maximum size before to fail. > What do you think if this is a configurable value by user? (we can have > as default if not provided) > > ``` > MAXIMUM_SAMPLE_SIZE_THRESHOLD > In order to guarantee a minimum size of guards after filtering, we > expand SAMPLED_GUARDS until a limit. This fraction of GUARDS will be > used as an upper bound when expanding SAMPLED_GUARDS. Proposed value is > 0.03. > ``` > > 1 - > https://github.com/twstrike/torspec/blob/review/proposals/259-guard-selecti… >

1 0

Adding one more (optional) step to Trac's ticket work-flow
by Isabela 06 Apr '16

06 Apr '16

Hello tor-dev! tl;dr; we want to add one more step to Trac ticket work-flow called 'merge_ready' - if you don't want to use it you don't need to During Tor's Dev Meeting in Valencia, the Network Team discussed a code review process[1] for the team and we concluded that for us a two stage review process would make things much better for the team. In order to organize that on our ticket work-flow on Trac, we decided to add the step 'merge_ready'[2] that would come after either 'needs_review' or 'needs_revision' steps. For our team, what that will mean is that we have reviewed the code and believe is ready for Nick to do a second review and merge. The first review will remove load from Nick and we want to eventually have more folks helping Nick on the second review as well. But this change alone will be already a big step. What does that means for someone working on a ticket that is not part of Core Tor development flow? Well, if you don't want to use it in your team or your project flow, you can just ignore it and keep using Trac as normal. We are sending this note to make sure folks knows about it before it happens and of course, to hear any concerns or feedback. Cheers, Isabela [1] https://trac.torproject.org/projects/tor/wiki/org/meetings/2016WinterDevMee… [2] https://storm.torproject.org/shared/pt9FtdJNG_gB_afqayg9dXTJwdDFncgU0HVXJEq… -- PM at TorProject.org gpg fingerprint = 8F2A F9B6 D4A1 4D03 FDF1 B298 3224 4994 1506 4C7B @isa

3 2