- tor-dev - lists.torproject.org

Is it possible to leak huge load of data over onions?
by Ivan Markin 04 Apr '16

04 Apr '16

Recently someone leaked enormous amount of docs (2.6 TiB) to the journalists [1]. It's still hard to do such thing even over plain old Internet. Highly possible that these docs were transfered on a physical hard drive despite doing so is really *risky*. Anyways, in the framework of anonymous whistleblowing, i.e. SecureDrop and Tor specifically it's seems to be an interesting case. I'm wondering about the following aspects: o Even if we use exit mode/non-anonymous onions (RSOS) is such leaking reliable? The primary issue here is time of transmission. It's much longer than any time period we have in Tor. o What is going to happen with the connection after the HS republishes its descriptor? Long after? [This one is probably fine if we are not using IPs, but...] o Most importantly, is transferring data on >1 TiB scale (or just transferring data for days) safe at all? At least the source should not change their location/RP/circuits. Or need to pack all this stuff into chunks and send them separately. It's not obvious how it can be done properly. So at what point the source should stop the transmission (size/time/etc)/change location or the guard/ pick new RP? -- [1] http://panamapapers.sueddeutsche.de/articles/56febff0a1bb8d3c3495adf4/ -- Happy hacking, Ivan Markin

2 2

[::]/8 is marked as private network, why?
by Sebastian Hahn 04 Apr '16

04 Apr '16

Hey, I've been wondering about the private_nets const in src/or/policies. It was added in a96c0affcb4cda1a2e0d83d123993d10efc6e396 but Nick doesn't remember why, and I'm hoping someone has an idea (maybe teor, who I've CCed here, who documented this in a later commit?). If nobody knows why we do this I think we should remove it as likely incorrect. Cheers and thanks Sebastian

4 3

Yawning's CFC, web caching, and PETs
by Jeff Burdges 04 Apr '16

04 Apr '16

Should we try to organize some public chat about web caching at PETs or HotPETs this summer? By that I mean, a discussion with anonymity researchers on security and anonymity concerns around making tools like Yawning's CFC a long-term solution to the CloudFlare problem? Aside from our not knowing if CloudFlare will become more accommodating, a trustworthy web cache would enable more serious efforts towards alpha-mixing, either in Tor itself, or with mixnets on the side of Tor. And archival tools make the web better in numerous ways, like by making it harder to removing anything. There are interesting problems in this space like : Big scary adversary issues. Archiving TLS sessions along with HTML transformations so that subsequent clients can verify the original site's certificate. How best to one distribute the cache. Jeff

2 1

A few ideas about improved design/modularity in Tor
by Nick Mathewson 04 Apr '16

04 Apr '16

(Hi! Here's a document I've been poking at , on and off, for a while. There's a lot more to say here, but I think it's ready to share with tor-dev for comment. Thanks!) ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ +++ 0. PRELIMINARIES. ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ ============== WHY? If you've ever looked at even slightly serious software engineering methodology, you've seen some discussion of modularity: the idea that programs should be divided into a number of smaller modules; that complexity should be hidden within modules and not in the interfaces between them; that which modules can call which other modules should be limited; and other little rules like that. We'd like modules isolated from one another for these reasons: * Make them simpler to test * Minimize what is allowed to touch what * Make tor easier to understand * Make tor easier to hack on * Let us write parts of tor in other languages. * Make sandboxing more powerful This is actually several problems all in one! - OO/Isolation/Modularity design that doesn't actually split into multiple processes, but makes the callgraph much much simpler. - Multiprocess sandbox framework with RPC implementing the above abstraction - Finding the right stuff to carve out into separate sandboxes - Finding the right stuff to carve into separate modules. ============ Steps/phases: Over the past 6-8 months we've gone a lot of work to simplify Tor's callgraph in preparation for this work. We've got our callgraph simplified to a pretty extreme degree, for example. Here are some of our next steps, some of which can be done in parallel. 1. Figure out which modules should be able to call each others. Isolate them based on their ability to do so. Refactor until they actually work that way. - In some cases we will need to split - hide big piles of modules behind single entry points. - separate src/common and src/or into separate parts. - Add tools to enforce this separation - Add tools to separate Tor functions safely and not too error-prone. (See sections 1 and 3 below.) 2. Add backend abstractions as needed to minimize module coupling. These should be abstractions that are friendly to in- and multi-process implementations. We will need at least: - publish/subscribe{,/acknowledge}. (See https://en.wikipedia.org/wiki/Publish%E2%80%93subscribe_pattern. The 'acknowledge' abstraction allows the publisher to wait for every subscriber to acknowledge receipt. More on this in section 4 below.) - handles (We need frequently need handles to objects where the handles need to be persistent while the objects themselves can go away. We use several patterns for this now. We should just have a handle pattern for this instead. This would need separate implementations for in-process and cross-process access. See section 2 below for more information.) - Need a way to mark functions as "internal to module". We're going to have a larger number of multi-file modules. Right now, our only restriction for making functions less visible is "static", and the fact that we don't allow src/common to call src/or. Both of these should be cleaned up. 3. Write abstractions for splitting code into multiple processes that talk over IPC. (See section 5 below) ========================= Metrics We can measure our success in a number of ways: * How large are the strongly connected components among modules calling another? Right now the largest SCC is 52 C files. * How many layer/module violations can we find? This needs to be an automated process. * How many exported functions from each module can we remove? * How much can we remove from the main linux sandbox code? How many functions can we isolate to purpose-built modules? ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ +++ 0. THE MODULES ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ ========================= Modules in src/or LOWEST LEVEL These should only get called from higher-level components. They shouldn't call anything of a higher level. Types and data structures. - part of configuration. - parts of statefile - routerset - policy (some) - torcert Various backends and utilities. - confparse - geoip - fp_pair? - reasons - replaycache - buffers Nodelist data structures, excluding routerinfo. - nodelist - microdesc - networkstatus - routerparse.c (parts) SLIGHTLY HIGHER These should only get called from higher-level components, and only call lower-level components. When they currently call a higher-level piece, they should use a callback or a publish/subscribe API or something. Nothing in here should really be tor-specific. mainloop: - connections (common, genericparts) - main (nonspecific parts) - config (maybe) - workqueue - scheduler - periodic - cpuworker (parts) - hibernate ? (Parts?) HIGHER THAN THAT These modules are where we start making Tor do Tor. They should only call lower-level modules onionrouting: - channels - onion* - cpuworker (parts) - connection_or - connection_edge - circuitlist - command - relay - parts of router.c - transports HIGHER STILL These modules can call downwards, though they should really be made as independent as possible from each other and from one another. We should enumerate which of these can call which others; mostly, they should leave one another alone. hidden services: - rend* controller: - control.c auxiliary: - rephist authority: - dirvote - dirserver (parts) - dirauth - dircollate - keypin relay: - ext_orport - parts of router.c - routerkeys.c cache: - parts of directory - dirserver (parts) - routerlist (parts) - routerparse.c (parts) exit: - dns client: - dnsserv - most circuitbuild, circuituse stuff - circpathbias - circuitstats directory: - directory - dirserver - HIGHEST LEVEL Here's the parts of the application that know all the other parts, and call down. Nothing else calls up to these modules. application (highest level) - Main (parts) - parts of config - parts of statefile - ntmain - status ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ +++ 2. INFRASTRUCTURE: HANDLES. ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ ======================== Handles in C * We need a light-weight handle mechanism. (A mechanism where we can have a reference to X that can still live even when X dies, where lookups of X through the reference are safe even when X is freed, and where lookups of X through the reference are FAST.) * It's okay if the things being handled need to be marked as 'handleable'. * When the handle is in a separate process, we probably need a table-based implementation: * Unique handle (64/96 bit int) for each thingie; hashtable to look up by handle. * When the reference is local, can get as fast as pointer lookup. * Threadsafety can be important but is not always required. For an example implementation, see #18362. ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ +++ 3. INFRASTRUCTURE: ENFORCING ISOLATION. ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ === Making isolation happen. It'll do us no good to declare that modules are isolated if they actually aren't. Once we have defined the allowed call patterns, we need to make sure that calls that fall outside of those patterns aren't permitted. We can build something pretty simple out of 'nm', I believe. See ticket #18617. This is a work in progress. === Moving functions around. I'm working on a tool to move functions from one file to another in a deterministic way based on annotations in the original file. This lets us version our movement, and have the movement happen deterministically so that the patches are easier to verify. If we integrate it with our callgraph-examination tools above, we can make sure that our movement plans actually make the code more conformant with our modularity plans. See ticket #18618. This is a work in progress. ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ +++ 4. INFRASTRUCTURE: PUBLISH/SUBSCRIBE ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Many of our modularity violations happen because when some property changes, or some event occurs, we need to inform a wide variety of other modules. The publish/subscribe{,/acknowledge} pattern, and the similar observer pattern, were made for this. ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ +++ 5. ADVANCED INFRASTRUCTURE: MULTIPROCESS SANDBOXING ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ ======================== Multiprocess abstraction / design We'll consider multiprocess for these cases: * Easier/stronger/more-portable sandboxing. * Writing subcomponents in languages whose runtime/link rules don't play so nicely with C. * Enforcing modularity when modularity delivers security. For sandboxing, here are some high-risk privileges we should make sure that other code doesn't have. * Filesystem calls -- anything that can open a file. * Exec -- anything that can launch a process. * Access to identity or signing keys. * Invoking getaddrinfo() [because it uses a fairly huge segment of the underlying operating system] And here is some higher-risk code that we could most safely isolate from the main modules. * Anything that parses an object. * Consensus diff processing. Our basic abstractions are those described above (pub/sub, handles, hooks), with the addition of: * Blocking RPC * Capabilities???? * message queues. Controversial: * We should NOT require that most of this be blindingly fast. Correct is perfectly adequate. Fast is only needed in the critical path. Here's the general architecture I have in mind: * At the highest level, have a launcher/supervisor process whose only job is to start the other processes, monitor them, and notice if they go down. * Additionally, have a communications-hub process whose role is to pass messages between other processes, performing access control on the messages to make sure that each message is only sent by a process that is allowed to send it, to a process that it is allowed to send to. Processes may only communicate with the hub. * Messages should use a simple RPC format. I vote for protobufs or capnproto. They sure have a speed/simplicity tradeoff though. * pipes or socketpairs should get used on Unix. Windows can use windows Pipes, which aren't quite FDs. Windows will need to have its pipes in a different thread from the socket-based libevent loop (if there is one.) * We need a way to pass fds or handles back and forth. Both Windows and Unix can do this. (via DuplicateHandle() on Windows and sendmsg() on PF_UNIX sockets on unix.) The protocol needs to handle this explicitly. Here's the development plan I would suggest: * In parallel, work on improving Tor's modularity as the earlier sections of this document suggest. Unless we get more modular, we won't be able to make anything actually separate besides the privilege-restriction sandboxing parts above (like isolating keys and FS access). * Research what exactly windows and osx allow. * Write the top-level supervisor process and hub process. * Ensure that the APIs we expose can be written in C in a fairly mechanism-agnostic way, so that we can migrate to this architecture with minimal additional effort. * Move privilege-restriction sandboxing parts into subprocesses. And here is an alternative plan: * Investigate the chromium sandbox code; see if it can be extracted, or how much of it can be extracted. ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ +++ A. APPENDIX. ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ === progress so far I've gotten these done so far: * The starts of a build-time callgraph enforcement mechanism using nm and readelf. * A handle-based-reference module. (See ticket #18362, but cpunks doesn't like it.) * A draft refactoring tool that lets you annotate code with what should move where, so that you can deterministicly try the code movement and see what happens, and so you can see what effects it will have on the module-level callgraph.

9 8

Re: [tor-dev] Is it possible to leak huge load of data over onions?
by Ryan Carboni 04 Apr '16

04 Apr '16

I've never seen anything download faster than ten megabits per second on Tor. Presumably the inverse is true if you have upload.

1 0

Re: [tor-dev] Advice regarding Cloudflare
by Ryan Carboni 03 Apr '16

03 Apr '16

> > (as opposed to the people that seem to think that Exits > should actively combat abuse by having the capability for censorship). > > Well, a large number of exit nodes already have the capability for a man-in-the-middle attack. This capability could very well be a default option. b) In your magic world, how would accessing any site that uses > multiple hosts for content to work? > > Yes, yes. It is you who is being imposed upon, not Cloudflare, not the businesses that serve content. In my magic world, people produce things for free! This might seem patronizing, but you seem genuinely ignorant. Cloudflare runs a business. They also get paid for it. That business is to protect email addresses from scrapes, and so forth. If they tell their customers that malicious actors can do those things, but only through Tor, because Tor does good work, their customers will take their business elsewhere. In a libertarian world, people can bar entry to their property from people who seem suspicious. You do not believe that Cloudflare should be allowed to bar entry out of some egocentric concern. By any reasonable definition of ethics, one must find a middle ground, and essentially, Cloudflare has all the negotiating power, unless you plan on personally battering down the doors of Cloudflare. A good step would be to ask Cloudflare for statistics on Tor misuse. Perhaps a maximum of 63 domain names (forgot Cloudflare only has a dozen IPs) per Tor circuit could be done.

3 2

Advice regarding Cloudflare
by Ryan Carboni 03 Apr '16

03 Apr '16

I could see why cloudflare is annoyed with you, you are annoying activists from their perspective, although you folks aren't chaining yourselves to coal power plants . But I also use Tor from time to time, so I'll offer some advice. On the Tor side, a way to minimize abuse is for exit nodes not to allow multiple IP addresses from a single circuit. This would make web crawling with Tor expensive, although it will disable the ability for a single tab to use a single exit node. Another method is to request a sort of proof of work on the Cloudflare side. Given current protocols, the simplest would be to require ECDHE with secp521r1 with each Tor connection and disallow unencrypted connections. It will be more bearable than a captcha.

2 1

Re: [tor-dev] Quantum-safe Hybrid handshake for Tor
by Ryan Carboni 03 Apr '16

03 Apr '16

I just want to note you only need an algorithm that protects against 2^80 quantum operations for short-term keys. Regardless, I doubt anyone is going to be spending a billion dollars to crack data sent over a single Tor connection.

1 0

(no subject)
by Nick Mathewson 29 Mar '16

29 Mar '16

Hi, folks! The Tor git master branch is now 0.2.9.0-alpha-dev. Development on 0.2.8.x continues in the branch "maint-0.2.8". I expect that 0.2.8.2-alpha will be the last 0.2.8.x-alpha, and then we can move on to rc and stable. We now return you to your semiregularly semischeduled Tor development. yrs, -- Nick

1 0

Re: [tor-dev] Should we disable the collection of some stats published in extra-infos?
by Karsten Loesing 29 Mar '16

29 Mar '16

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 15/01/16 23:00, Rob Jansen wrote: > Hello, Hi Rob, I'm moving this discussion from metrics-team@ to tor-dev@, because I think it's relevant for little-t-tor devs who are not subscribed to metrics-team@. Hope you don't mind. > I was recently reviewing the statistics that Tor allows relays to > collect and report to the dir servers [1], which then get published > in extra-info documents [2]. Most of this can be enabled by simply > setting a torrc option. There are quite a few statistics that I > feel should not be collected. I'm wondering if the original purpose > for collecting many of these statistics still exists, and if we > still feel that the privacy compromises that were made when the > collection was implemented are still valid in most cases. > > Here are the stats I am most worried about, and why: > > [unique ips per country code] *-ips (there are many of these, e.g. > "entry-ips") Usually this involves storing individual user IP > addresses in memory (in order to track uniqueness) over some period > of time (usually 24 hours), sometimes for longer than the user > would have otherwise been known to Tor (if a user's session is 1 > hour, Tor could remember the IP for at most 23 additional hours). > This is reported, e.g., per entry; there are many cases in the data > where it is very likely that only one user is connecting to a guard > from a given country (because it is rounded up to 8). Users in > small countries have the greatest risk (intersection attacks become > really easy). I agree that might just lose these statistics. We used them in the past as first approximation to counting users, but obviously that only works as long as clients only connect to a single relay. The only place where we're still using them is in a workaround for estimating bridge users. See #15469 for more details and #8786 for something we'd have to implement before taking these statistics out. > [exit statistics by port number] exit-kibibytes-written > exit-kibibytes-read exit-streams-opened Tor is classifying its > traffic into ports, which could uniquely identify the application > being used by the client. They also track bandwidth usage per port > (and per exit); again, this is bad for those using a random or > unique looking ports (that a given exit does not see very often) > because it could be used to create a fingerprint. Intersection > attacks become easier with this information. Agreed, I can see us dropping these statistics, too. We're currently not using them. But also see my suggestion below. > The less problematic stats: > > [circuit-based cell statistics] cell-processed-cells > cell-queued-cells cell-time-in-queue cell-circuits-per-decile This > provides queue timings and number of cells being processed at a > relay. The number of cells can be used to compute bandwidth of > circuits. It may be possible to launch some attacks that create > several circuits with the intent of moving which decile buckets > some legitimate circuits get placed into, but this is less > worrisome of an attack than the others. I'm less worried about this one. But, suggestion below. > Should Tor still be collecting these things? Should Tor disable the > collection of these statistics until we have a more > privacy-preserving way to collect and aggregate them? > > The good news is that privacy-preserving techniques exist that can > reduce information leakage. I'm developing a tool based on the > secret-sharing variant of PrivEx [3] to collect some of these types > of statistics while providing privacy guarantees. We are currently > using it to collect only those stats that are useful for producing > Tor traffic models. A great advantage of this tool is that the > various counters that we store during the collection phase get > noise added and are randomized during initialization; only the > aggregates are ever known and revealed by the aggregation server, > limiting the information that is lost if a relay is compromised. > This is a large improvement over the current collection method, > which only adds noise before publication and reveals statistics on > a per-relay basis. Suggestion: How about we evaluate these statistics published by relays in the past years to see if there are other benefits or risks we didn't think of, and then we decide whether to leave them in, modify them, or take them out? The reason is that I'd want to avoid removing this code only to realize shortly after that we overlooked a good reason for keeping it. These statistics are being collected for years now, and it might take another year or so for relays to upgrade to stop collecting them. So what's another month. Thanks for (re-)starting this discussion! > All the best, Rob All the best, Karsten > [1] https://gitweb.torproject.org/torspec.git/tree/dir-spec.txt [2] > https://collector.torproject.org/recent/relay-descriptors/extra-infos/ > > [3] www.cypherpunks.ca/~iang/pubs/privex-ccs14.pdf > > > > _______________________________________________ metrics-team > mailing list metrics-team(a)lists.torproject.org > https://lists.torproject.org/cgi-bin/mailman/listinfo/metrics-team > -----BEGIN PGP SIGNATURE----- Comment: GPGTools - http://gpgtools.org iQEcBAEBAgAGBQJWnffFAAoJEJD5dJfVqbCrSREH/1G3nZDwC7a8FGFEySgQX3MZ n1dlMcylDU6kypgGMEYs2KmMRwhNYmCVFn6fRJgrFN/KgJ0oZJG1pRcglnIFwNwH 5VUWDZp3+frhY5jBbJ3JXqA4Exhi/0mUYCTRwfcFQ/JOyzjlcFbLRW/rkqFHn62H wLidaVeagiJ+TI/T8zgwtqQjTDSZrZPpmlxvlO57D1bGhW1ZPJVbOeUzuudhNpRS WVu//MH51juCzcML32MeiMV6wWYYOm1irKZ8lZHVCPbsL98qoh3ewtO6fb62THIa sWcQZlwMQdcC0+509PmMeYwkRn+40MUwkk84/glyP57dfpgtOwzZCohL3yQRQ1s= =F6F+ -----END PGP SIGNATURE-----

2 4