- tor-dev - lists.torproject.org

Exitmap's control flow
by Philipp Winter 17 May '16

17 May '16

Hi Mridul, I'm copying tor-dev@, so other folks can chime in. On Thu, May 12, 2016 at 10:16:45AM +0530, Mridul Malpotra wrote: > a) Can you give me a short description about the program flow on how the > EventHandler class enables modules to be executed in exitmap? From my > initial pondering over the code, the circuits seem to be created in > succession. Then for each circuit creation, the listener catches the event > and starts by calling module_closure for invoking probe() function specific > to each module. However, I am having trouble understanding the role of the > command utility and IPC queue, for which I can see a separate queue having > the exit_fingerprint and socket pairs but fail to comprehend how it is > being used. Like you said, circuits are created sequentially in exitmap.py:419. We have configurable inter-circuit delay (exitmap.py:442) to reduce the load on the Tor network. Before exitmap asks Stem to create circuits, it registers event listeners for all circuit and stream events (exitmap.py:371). So whenever Tor tells Stem that a circuit or stream has changed, Stem notifies exitmap. We only care about a subset of all circuit and stream events, though. All the event code is in eventhandler.py. Once Tor managed to create a new circuit, it tells Stem, which tells exitmap. We catch this event in the new_circuit() function (eventhandler.py:254). Now here's where the command utility come in. Exitmap modules can do one of two things. They can either use pure Python to do their scan (e.g., by using httplib to fetch a web site), or use an external tool and parse its output (e.g., by running the openssl command line tool). For the first case, we offer the function run_python_over_tor() (command.py:37). It basically monkey-patches Python's socket.socket and makes it go over Tor. External programs are a little bit more tricky, and handled by the Command object (command.py:58). We will have multiple instances of our command line tool running at the same time, and they all connect to Tor's SOCKS port. Exitmap then maps a stream (e.g., whatever openssl does) to a circuit. But how does exitmap know which one among the, say, 10 streams belongs to a given circuit? If we don't attach them correctly, we can still detect MitM attacks, but we will get the alert for the wrong exit relay. To correctly attach streams to circuits, exitmap modules remember the source port of the command line tool and hand it back to exitmap (note that a module runs in a different process than exitmap) over the queue. We can get the source port by parsing the output of torsocks. However, the current version of torsocks does not implement this yet. > b) For calculating all possible exits and creating circuits, > ServerDescriptors is set to 0 probably to avoid conflict with changed > consensus data in the middle of module execution. Also, for each module run > in the same execution, the original consensus downloaded during > bootstrapping Tor is being used. In my use case however, which involves > long duration scanning, we will need to update the cached-consensus after > some iterations. One way I think this can be made possible is by having an > asynchronous task that updates the consensus after say, an hour or so. > Conflicts could be avoided mid-module execution by either stalling > execution if near the DA consensus time period or use the old > cached-consensus. I would like to ask you whether my conclusion is correct > and if so, what other ways can I explore? Yes, that sounds reasonable to me. In fact, we might as well periodically download server descriptors instead of consensuses, so we can also scan relays that don't have the Exit flag, but have an exit policy. The consensus does not necessarily tell us what a relay's exit policy looks like. For more information, see: <https://github.com/NullHypothesis/exitmap/issues/13> > c) In exitmap, we are taking in the analysis_dir parameter in the command > line and storing it in a global variable in util.py. However, nowhere is > the dump_to_file function [1] being called to report on bad exits. What was > the working that was thought of behind this function? Was exitmap to dump > the log entries in files inside the analysis_dir for every false negative? > Can't seem to find code where the function is called. dump_to_file() is only used in modules, and not by exitmap itself. You certainly have a point here; the way exitmap logs stuff is inconsistent because it's entirely module-driven. Seems worthy of some improvements. I hope that clears things up a bit. Let me know if you want me to elaborate on something. Cheers, Philipp

1 0

Request for feedback/victims: cfc-0.0.2
by Yawning Angel 16 May '16

16 May '16

Hello, Thanks for the feedback so far. [ PEOPLE THAT HAVE BIG SCARY ADVERSARIES IN THEIR THREAT MODEL STILL SHOULD NOT USE THIS. ] New version with changes some that add functionality, some code of quality stuff, hence a version bump to 0.0.2, especially since it'll probably be a bit before I can focus on tackling the TODO items. Source: https://git.schwanenlied.me/yawning/cfc XPI: https://people.torproject.org/~yawning/volatile/cfc-20160327/ Major changes: * Properly deregister the HTTP event listeners on addon unload. * Toned down the snark when I rewrite the CloudFlare captcha page, since I wasn't very nice. * Additional quality of life/privacy improvements courtesy of Will Scott, both optional and enabled by default. * (QoL) Skip useless landing pages (github.com/twitter.com will be auto-redirected to the "search" pages). * (Privacy) Kill twitter's outbound link tracking (t.co URLs) by rewriting the DOM to go to the actual URL when possible. Since DOM changes made from content scripts are isolated from page scripts, this shouldn't substantially alter behavior. * (Code quality) Use a pref listener to handle preference changes. TODO: * Try to figure out a way to mitigate the ability for archive.is to track you. The IFRAME based approach might work here, needs more investigation. * Handle custom CloudFlare captcha pages (In general my philosophy is to minimize false positives, over avoiding false negatives). Looking at the regexes in dcf's post, disabling the title check may be all that's needed. * Handle CloudFlare 503 pages. * Get samples of other common blanket CDN based Tor blocking/major sites that block Tor, and implement bypass methods similar to how CloudFlare is handled. * Look into adding a "contact site owner" button as suggested by Jeff Burdges et al (Difficult?). * Support a user specified "always use archive.is for these sites" list. * UI improvements. * More Quality of Life/Privacy improvements (Come for the Street Signs, stay for the user scripts). * I will eventually get annoyed enough at being linked to mobile wikipedia that I will rewrite URLs to strip out the ".m.". * Test this on Fennec. * Maybe throw this up on addons.mozilla.org. Regards, -- Yawning Angel

6 10

[GSoC] Tails Server - status report
by segfault 16 May '16

16 May '16

Hi everyone, this is the first status report on the Tails Server GSoC project. I officially began working on it on April 25th, although I already did some work in the weeks before. This is what I have done so far: * Updating the blueprint of the Tails Server [1] * Implementing two iterations of prototypes of the GUI. The most recent one is available on gitlab [2]. * Discussing the design of the GUI on the Tails-UX mailing list [3][4] * I began implementing the CLI, the current code (not ready for review yet) is also available on gitlab [2] Next I plan to continue the design of the GUI, after others commented on the new prototype. There are still some features missing which I will implement in time. In parallel I will continue implementing the CLI and discussing the design decisions. I did quite a lot during the last two weeks and I will have some other work to do in the next two weeks, so I expect to get less work done until the next report. Cheers! [1] https://tails.boum.org/blueprint/tails_server/ [2] https://gitlab.com/segfault_/tails_server [3] https://mailman.boum.org/pipermail/tails-ux/2016-April/thread.html#919 [4] https://mailman.boum.org/pipermail/tails-ux/2016-May/thread.html#953

2 2

Firejail Integration
by bancfc＠openmailbox.org 13 May '16

13 May '16

Hi Yawning. We are thinking about how to integrate your Firejail profile [0] in Whonix. Do you plan to upstream the changes in the start-tor-browser script soon? If not then what is the best way to ensure the custom script survives TBB upgrades? [0] https://git.schwanenlied.me/yawning/tor-firejail

2 1

exitmap modules that make *lots* of connections
by Zack Weinberg 13 May '16

13 May '16

I'm working on an exitmap module that wants to feed order of 5000 short-lived streams through each exit relay. I think this is running foul of some sort of upper limit (in STEM, or in Tor itself, not sure) on the number of streams a circuit can be used for, or how long, or something. What I see in the logs (note: I've modified eventhandler.py for more detailed debug logs) looks like 2016-04-22 16:07:53,306 [DEBUG]: Circuit status change: CIRC 6 LAUNCHED PURPOSE=GENERAL TIME_CREATED=2016-04-22T20:07:53.305851 2016-04-22 16:07:53,325 [DEBUG]: Circuit status change: CIRC 6 EXTENDED [fp] PURPOSE=GENERAL TIME_CREATED=2016-04-22T20:07:53.305851 2016-04-22 16:07:54,114 [DEBUG]: Circuit status change: CIRC 6 EXTENDED [fp],[fp] PURPOSE=GENERAL TIME_CREATED=2016-04-22T20:07:53.305851 2016-04-22 16:07:54,115 [DEBUG]: Circuit status change: CIRC 6 BUILT [fp],[fp] PURPOSE=GENERAL TIME_CREATED=2016-04-22T20:07:53.305851 2016-04-22 16:07:54,136 [DEBUG]: Port 47488 preparing to attach to circuit 6 2016-04-22 16:07:54,136 [DEBUG]: Port 47488 circuit 6 waiting for stream. 2016-04-22 16:07:54,155 [DEBUG]: Attempting to attach stream 65 to circuit 6. 2016-04-22 16:07:54,387 [DEBUG]: Port 47492 preparing to attach to circuit 6 2016-04-22 16:07:54,387 [DEBUG]: Port 47492 circuit 6 waiting for stream. 2016-04-22 16:07:54,388 [DEBUG]: Attempting to attach stream 67 to circuit 6. 2016-04-22 16:07:54,809 [DEBUG]: Port 47496 preparing to attach to circuit 6 2016-04-22 16:07:54,810 [DEBUG]: Port 47496 circuit 6 waiting for stream. 2016-04-22 16:07:54,810 [DEBUG]: Attempting to attach stream 69 to circuit 6. 2016-04-22 16:07:55,060 [DEBUG]: Port 47502 preparing to attach to circuit 6 2016-04-22 16:07:55,060 [DEBUG]: Port 47502 circuit 6 waiting for stream. 2016-04-22 16:07:55,061 [DEBUG]: Attempting to attach stream 72 to circuit 6. 2016-04-22 16:07:55,468 [DEBUG]: Port 47506 preparing to attach to circuit 6 2016-04-22 16:07:55,468 [DEBUG]: Port 47506 circuit 6 waiting for stream. 2016-04-22 16:07:55,469 [DEBUG]: Attempting to attach stream 74 to circuit 6. 2016-04-22 16:07:55,720 [DEBUG]: Port 47508 preparing to attach to circuit 6 2016-04-22 16:07:55,720 [DEBUG]: Port 47508 circuit 6 waiting for stream. 2016-04-22 16:07:55,990 [DEBUG]: Port 47512 preparing to attach to circuit 6 2016-04-22 16:07:55,990 [DEBUG]: Port 47512 circuit 6 waiting for stream. 2016-04-22 16:07:55,990 [DEBUG]: Attempting to attach stream 77 to circuit 6. 2016-04-22 16:07:56,241 [DEBUG]: Port 47518 preparing to attach to circuit 6 2016-04-22 16:07:56,241 [DEBUG]: Port 47518 circuit 6 waiting for stream. 2016-04-22 16:07:56,242 [DEBUG]: Attempting to attach stream 80 to circuit 6. 2016-04-22 16:07:56,492 [DEBUG]: Port 47528 preparing to attach to circuit 6 2016-04-22 16:07:56,492 [DEBUG]: Port 47528 circuit 6 waiting for stream. 2016-04-22 16:07:56,495 [DEBUG]: Attempting to attach stream 85 to circuit 6. 2016-04-22 16:07:56,836 [DEBUG]: Port 47536 preparing to attach to circuit 6 2016-04-22 16:07:56,836 [DEBUG]: Port 47536 circuit 6 waiting for stream. 2016-04-22 16:07:56,836 [DEBUG]: Attempting to attach stream 89 to circuit 6. 2016-04-22 16:07:57,100 [DEBUG]: Port 47540 preparing to attach to circuit 6 2016-04-22 16:07:57,100 [DEBUG]: Attempting to attach stream 91 to circuit 6. 2016-04-22 16:07:57,351 [DEBUG]: Port 47544 preparing to attach to circuit 6 2016-04-22 16:07:57,351 [DEBUG]: Port 47544 circuit 6 waiting for stream. 2016-04-22 16:07:57,352 [DEBUG]: Attempting to attach stream 93 to circuit 6. 2016-04-22 16:07:57,769 [DEBUG]: Port 47550 preparing to attach to circuit 6 2016-04-22 16:07:57,769 [DEBUG]: Port 47550 circuit 6 waiting for stream. 2016-04-22 16:07:57,769 [DEBUG]: Attempting to attach stream 96 to circuit 6. 2016-04-22 16:07:58,118 [DEBUG]: Port 47554 preparing to attach to circuit 6 2016-04-22 16:07:58,118 [DEBUG]: Port 47554 circuit 6 waiting for stream. 2016-04-22 16:07:58,118 [DEBUG]: Attempting to attach stream 98 to circuit 6. 2016-04-22 16:08:04,697 [DEBUG]: Circuit status change: CIRC 6 CLOSED [fp],[fp] PURPOSE=PATH_BIAS_TESTING TIME_CREATED=2016-04-22T20:07:53.305851 REASON=FINISHED 2016-04-22 16:08:05,878 [DEBUG]: Port 47690 preparing to attach to circuit 6 2016-04-22 16:08:05,878 [DEBUG]: Port 47690 circuit 6 waiting for stream. 2016-04-22 16:08:05,879 [DEBUG]: Attempting to attach stream 166 to circuit 6. 2016-04-22 16:08:05,879 [WARNING]: Failed to attach stream because: Unknown circuit "6" You can see that circuit 6 is no longer available, but the module is still trying to use it. It looks like it lasted for almost exactly ten seconds, which smells like a time limit, but I can't find any relevant configuration parameters in the documentation. Rather than trying to make a circuit survive as long as necessary, which might not even be possible, it'd probably be better if exitmap could notice that it's lost a circuit that's still in use and create a new one. However, I'm not sure how to do that in the current architecture. The existing code has "one circuit per module invocation = one circuit per exit node" as a pretty deeply embedded design constraint. Anyone have any ideas? Full logfile available on request, but it's huge. zw

2 2

Fewer Failures in Shared Random
by Tim Wilson-Brown - teor 12 May '16

12 May '16

Hi All, Currently, the shared random system treats the first vote of the next commit phase specially - it's the only time it tries to agree on a new shared random value. This makes one consensus in each day particularly attractive to adversaries (or particularly vulnerable to failures), because if we fail to agree on that consensus, the next shared random value is never agreed. This makes hidden service behaviour far more predictable for the next 24 hours. But we can try to agree 12 times on a new shared random value like this: * for the first consensus in the new commit period: * vote the calculated shared random value based on the commits and reveals we've seen; * for subsequent consensuses in the new commit period: * if we have an agreed shared random value from a trusted, previous consensus in the period, vote that value; * if not, (if the new shared random value is missing from all previous consensuses in that period, or there is no trusted consensus), continue to vote our calculated value. This way, we try up to 12 times to agree on a shared random value. (But we never change an agreed value after we've agreed on it.) This is very similar to how the previous shared random value is determined. I've added #19045 to keep track of this: https://trac.torproject.org/projects/tor/ticket/19045 Tim Tim Wilson-Brown (teor) teor2345 at gmail dot com PGP 968F094B ricochet:ekmygaiu4rzgsk6n

1 0

RFC-ish: basket2 (aka obfs5)
by Yawning Angel 12 May '16

12 May '16

Hello, So I've been somewhat productive as of late and have been working on the successor to obfs4. I have a "oh my god, you wrote how much code, with no documentation" minimum-viable-product-ish release that appears to work, though ABSOLUTELY NO ONE SHOULD USE IT YET, because I will break bridge line/protocol wire compatibility intentionally at least once before release. The code: https://git.schwanenlied.me/yawning/basket2 (Really, don't use it, at all. I will ignore cries of help, mercilessly mock people that attempt to chase a moving target, and laugh as I break things). The documentation: Hahahahahahahahahahahahahahaha (It should be self explanatory). A brief overview of the improvements: * Client can now negotiate the link padding/delay algorithm at handshake time with the server (no more `iat-mode=blah` argument), and defaults to something which injects delay. * Link cryptography changed to a XChaCha20/Poly1305 construct, with AVX2 used for the ChaCha20 when available, and the framing construct it self is a lot better designed. * Handshake is X25519 + NewHope or X448 + NewHope based (available algorithms specified as part of the bridge line), with X25519 (Elligator2) for authentication. * SHA-3/SHAKE are used as the handshake routines. * When adding IAT on a burst basis, the code attempts to detect bulk data transfer resulting in less delay for large downloads/uploads. * Server side buffering reduced by 50% (not sure if that'll stick, I need to think about some of this stuff). TODO: * Implement a "expensive" padding option for bridges that can afford to run such a thing. (WTF-Pad, CS-BuFlo, or Tamaraw, not sure yet. WTF-Pad is a bit harder to implement than I'd like...) * Improve the lightweight padding algorithms instead of using what is essentially the same as obfs * Finish support for user authentication. * Write a formal specification. * Debug, debug, debug. * Write an AVX2 Poly1305 so it goes faster on boxes that matter (my laptop). * (Maybe) Write NEON and 32 bit x86 assembly optimized versions of the custom crypto, so it runs faster on various low end shit boxes. NEON is more important to me than 32 bit Intel (and a low end Atom does ChaCha20 at 40 MiB/s or so anyway...). Anyway, this is sort of a heads up that something like this is in the works, and is approaching alpha state, though DID I MENTION NOT TO USE IT YET? Questions/Comments/Feedback welcome as always. Regards, -- Yawning Angel

1 0

Directory authority of type bridge?
by Nicholas R. Parker (RIT Student) 11 May '16

11 May '16

Hey again all, got another one for you. When we've started adding bridges to the network, they send a warning "Publishing server descriptor to directory authorities of type 'Bridge', but no authorities of that type listed!" Not sure how to have a directory authority as a bridge type given that if you set BridgeRelay 1 on an authority it drops the DirPort and just becomes a bridge relay. How does one list an authority as a bridge type? Nicholas R. Parker Rochester Institute of Technology 5thYear, BS/MS Computing Security

2 2

Re: [tor-dev] [Proposal] Obfuscating the Tor Browser Bundle initial download
by Blake Hadley 09 May '16

09 May '16

> The environment you're were in was mounting a MITM attack to break TLS, > or has compromised your box, because the only component of the URL that > is visible otherwise is the host in the SNI field. > > In such an environment, gettor in general isn't unblockable because > there is no privacy/security for the request/response messages. > > Regards, > > -- > Yawning Angel Sorry, I'm not completely familiar with how TLS works. The environment requires an HTTPS proxy to reach the World Web Web. Do HTTP proxies inherently create a situation similar to MITM?

4 3

[Proposal] Obfuscating the Tor Browser Bundle initial download
by Blake Hadley 09 May '16

09 May '16

Hey everyone, [How it's currently done] Distributed by gettor(a)torproject.com, the URL makes it pretty clear what you're downloading. Dropbox: https://www.dropbox.com/s/mz9ug2rzvj85791/torbrowser-install-5.5.5_en-US.ex… Google Drive: https://docs.google.com/uc?id=0B76pDbk5No54VHowTEprZnBfWlU&export=download GitHub: https://github.com/TheTorProject/gettorbrowser/releases/download/v5.5.5/tor… [Security problem] The download URL on Google Drive is somewhat obfuscated, but once the download is started, the filename that the browser requests is 'torbrowser[...]' An environment I was working in has started to block the files based on name, and it would be very easy for an adversary monitoring network traffic to detect users downloading it. [Solution proposed] When the user emails gettor, they could also request obfuscation. An application would randomize the filename and upload it to a mainstream host (Google, Dropbox, GitHub, AWS). Maybe even protect the file from scanning by making an AES encrypted ZIP file, and giving the user the password in the reply email. I'd be happy to make a proof-of-concept. What do you all think of this? Does anyone have any better ideas? Anything that uses less processing resource?

2 1