- tor-dev - lists.torproject.org

[GSoC '16] Status Report #2 - Exitmap Improvements Project
by Mridul Malpotra 27 Jun '16

27 Jun '16

Hey everyone. This is my second status report for the Exitmap improvements project. These 2 weeks, I was building up on the browser automation aspect to allow for browser interactions on Exitmap scans. These will better help disguise scanner activity against malicious exit nodes that might not act like an adversary when they detect scanner-like behavior in the client. Work and details: ----------------------- 1. I am using the Selenium WebDriver Python bindings to help in simulating browser actions. Have implemented some simple actions in Selenium and have also taken inspiration from the Tor Browser Selenium library suggested by the community. 2. Have included actions like opening a random number of links on the webpage in the same page, new tab or another window. Tried to get clickable elements of a webpage but facing some issues with dynamic pages where buttons do not enabled instantly. 3. Added browser_simulation.py that allows for a module to import the functions use the simulate() function with the intended URL to performs these operations. Some problems in these functions with some websites. Could be due to dynamic content. Needs to be fixed. Problems faced: ---------------------- 1. Major issue that I am facing is with the multiple invocations of the module for every exit node. The asynchronous calls are throttled due to the more serial Selenium calls to the TBB driver. I have not found a working solution for this as of now, which makes the tests running with browser automation to be much slower. 2. For some operations like opening tabs, I could not find defined library calls, leading to more hackier ways to complete these operations. This often leads to unexpected behavior and needs to be fixed. Future plans: ----------------- As continuous scanning is more important toward the project, I will be focusing on that for the moment and revisiting browser automation after I complete the former. My next report is due on 1st July. I also have submitted the report a week late due to unexpected complete disconnect from internet in the area I was at that moment for 6 days. This caused my work to stop for some time and I apologize for the irregularity. Immediately having shifted to a metropolitan city, I will ensure my reports will now be on time and work will progress without hindrance. Regards, -- Mridul ============================= PGP keyID: 0xb716e33ab6d0a653

1 0

TorBirdy 0.2.0: Sixth Beta Release
by Sukhbir Singh 27 Jun '16

27 Jun '16

We are pleased to announce the sixth beta release of TorBirdy and the first in the 0.2 series: TorBirdy 0.2.0. All users are encouraged to upgrade as this release fixes numerous security and privacy issues. Notable changes include fixing local timestamp disclosure in the date and the message-ID headers, as detailed in tickets #6314 [0] and #6315 [1]. The patch for sanitizing the date header is shipped [2] with TorBirdy. The patch for the message-ID header was submitted upstream [3] to Mozilla and merged in Thunderbird 45, and it is therefore recommended that you upgrade to Thunderbird 45 if possible. There are currently no known leaks in TorBirdy but please note that we are still in beta, so the usual caveats apply. If you are using TorBirdy for the first time, visit the wiki [4] to get started. Other changes in this release include: 0.2.0, 27 Jun 2016 * Bug #6314 [5]: Prevent local timestamp disclosure via Date header * Bug #6315 [6]: Prevent local timestamp disclosure via Message-ID header * Bug #13721 [7]: Fix usage of wrong locale * Bug #17426 [8]: Allow configuration of default email protocol * Bug #15459 [9]: Add support for deterministic XPI generation * Bug #11387 [10], #13006 [11]: Fix non-standard EHLO argument * Bug #17118 [12]: Allow manual account configuration for Gmail with OAuth2 * Bug #19031 [13]: Add and audit support for RSS reader * Bug #7847 [14]: Audit and update support for NNTP * Bug #10683 [15]: Update Thunderbird UI to reflect TorBirdy's state * Bug #19330 [16]: Set secure defaults for outgoing mail servers * Removed compatibility for older versions of Thunderbird and added support for Thunderbird 37+ * Added support for automatic configuration of Riseup email accounts * Updated various privacy and security settings (see commit 2bdeffbb [17] for a list of the changes) * Update translations for current languages Many thanks to Arthur Edelstein and the Tails Developers for this release! We offer two ways of installing TorBirdy -- either by visiting our website [18] (GPG signature [19]; signed by 0xB01C8B006DA77FAA) or by visiting the Mozilla Add-ons page [20] for TorBirdy. Please note that there may be a delay -- which can range from a few hours to days -- before the extension is reviewed by Mozilla and updated on the Add-ons page. (Packages for Debian GNU/Linux will be created and uploaded shortly.) [0] - https://bugs.torproject.org/6314 [1] - https://bugs.torproject.org/6315 [2] - https://gitweb.torproject.org/torbirdy.git/commit/?id=bfa822de6071e894dad3d… [3] - https://bugzilla.mozilla.org/show_bug.cgi?id=902580 [4] - https://trac.torproject.org/projects/tor/wiki/torbirdy [5] - https://bugs.torproject.org/6314 [6] - https://bugs.torproject.org/6315 [7] - https://bugs.torproject.org/13721 [8] - https://bugs.torproject.org/17426 [9] - https://bugs.torproject.org/15459 [10] - https://bugs.torproject.org/11387 [11] - https://bugs.torproject.org/13006 [12] - https://bugs.torproject.org/17118 [13] - https://bugs.torproject.org/19031 [14] - https://bugs.torproject.org/7847 [15] - https://bugs.torproject.org/10683 [16] - https://bugs.torproject.org/19330 [17] - https://gitweb.torproject.org/torbirdy.git/commit/?id=2bdeffbb [18] - https://dist.torproject.org/torbirdy/torbirdy-current.xpi [19] - https://dist.torproject.org/torbirdy/torbirdy-current.xpi.asc [20] - https://addons.mozilla.org/en-us/thunderbird/addon/torbirdy/ [Also on https://blog.torproject.org/blog/torbirdy-020-sixth-beta-release] -- Sukhbir

1 0

Help complete the list of official & community projects
by Nima Fatemi 27 Jun '16

27 Jun '16

Hi there, I'm working with some folks to compile a list of most active official and community projects we have. There are some projects listed on our volunteer page[1] but I'm not confident if the list is complete or even up to date. Please have a look at that page and let me know if your project is listed and it's not up to date, and definitely let me know if it's not listed at all. So if you've recently wrote a Tor library in a language only 5 people on this planet know about, this is a good chance for you to let us know of your project. :) Here's the information I need from you: Name of the project, along with a paragraph explaining the significance of the project and why it matters. Link to the project, and name or the handle of the maintainer. Examples: ====================================================================== ## Stem Stem is a Python controller library for Tor. With it you can use Tor's control protocol to script against the Tor process. https://stem.torproject.org/ maintainer: atagar ## OONI A free software, global observation network for detecting censorship, surveillance and traffic manipulation on the internet. https://ooni.torproject.org maintainer: ooni-team ====================================================================== [1] https://www.torproject.org/getinvolved/volunteer.html.en#Projects Alternatively we can create a new wiki page just for this projects, until #16576 is resolved. Except I'm only looking for active projects at this point. Happy Friday! -- Nima 0X58C4B928A3E218F6 | @mrphs "I disapprove of what you say, but I will defend to the death your right to say it" --Evelyn Beatrice Hall

8 7

RB Trees
by U+039b 25 Jun '16

25 Jun '16

Hello! I had a discussion with David Goulet about RB trees [0]. More particularly about the usage of RB trees for some Tor operations like [1] or HSDir cache. David said that Nick wants to use a FreeBSD implementation - is this one[2]? If it is, I think it could be hard for new contributors like me to maintain/write/debug code using it. Since inline functions are as fast as macros [3], is it conceivable to use functional implementation instead of macro implementation? If it is, I can propose a generic C implementation of RB tree. It seems that Tor uses some kind of hashmap [4]. What about using hashmaps with RB tree as bucket and thus ensure bounded time complexity? In some cases, it could be interesting to store RB tree nodes in hashmap to ensure lookup in O(1) while keeping entries sorted. Regards, L. [0] https://en.wikipedia.org/wiki/Red%E2%80%93black_tree [1] https://trac.torproject.org/projects/tor/ticket/13739 [2] https://github.com/freebsd/freebsd/blob/master/sys/sys/tree.h [3] https://gcc.gnu.org/onlinedocs/gcc/Inline.html [4] https://gitweb.torproject.org/tor.git/tree/src/common/container.h

1 0

Freenet + Onioncat: Is the traffic welcome?
by konstant＠mail2tor.com 25 Jun '16

25 Jun '16

I posted steps on how to connect Freenet nodes over Onioncat and Garlicat for Tor/I2P. I am looking to scale it into an Opennet inside Tor with a lot of peers: https://emu.freenetproject.org/pipermail/devl/2016-June/039056.html https://emu.freenetproject.org/pipermail/devl/2016-June/039059.html Is the extra traffic desirable in Tor? Reading asn's comment, I was under the impression that you are interested in adding higher latency traffic such as Freenet or mixnets for better anonymity: https://blog.torproject.org/blog/crowdfunding-future-hidden-services Using both projects in tandem can finally realize the vision of FreeHaven. You are the best at firewall circumvention, performance and accessing the web, Freenet supplies users with censorship resistant publishing and p2p services. There is a HotPETs 16 paper co-authored by George Danezis on renewing interest in anonymous storage networks: http://www0.cs.ucl.ac.uk/staff/M.Isaakidis/p/isaakidis-p2pstorageservices-h…

4 8

We need a new RPM maintainer
by Nick Mathewson 24 Jun '16

24 Jun '16

Hi, all! Ondrej Mikle, who has built our RPMs for a while, is no longer able to do so. (He isn't using RPM distros any more, doesn't have as much free time, and etc.) First: Thanks, Ondrej, for all your hard work! Here's the instructions he sent on how to do it (quoted with permission): {{{ Basically every part of the process is documented here: https://gitweb.torproject.org/user/hiviah/rpm-build-scripts.git/tree/README It's maybe not the best README, because the actual merge-and-build part is in the middle (but I always use it, otherwise I'd just forget all the necessary steps). Nevertheless, the introduction maybe necessary for someone not familiar with the process. The process in the short is: 0. have the environment ready (has to be done just once) 1. clone my own RPM fork which has RPM changes (https://gitweb.torproject.org/user/hiviah/rpm-tor.git) 2. Merge and build according to "MERGING REPO USED FOR RPM BUILDING FROM ORIGIN" 3. Download, review, sign the built RPMs (I used feddei.tpo VM for this, but you can have your own VM as I had last time since feddei.tpo was down) 4. upload to palmeri.tpo 5. distribute to everyone else via "static-update-component deb.torproject.org" on palmeri.tpo I could walk somebody through this, as I said, this is not rocket surgery, mostly attention to detail :) If there ever was a critical need for a special build due to some unforseen bug, I could still do it, but this is for exceptional case only. }}} The easiest way for RPMs to happen would be if somebody who already has a Tor LDAP account wants to step forward and volunteer for a while. It would be harder if this were somebody that we didn't know who volunteered: until our RPM process creates reproducible builds, we need to have some amount of trust in the person who creates them. There may be other issues here too. I am no packaging expert. :) best wishes, -- Nick Mathewson

6 7

[GSOC16] Fingerprint Central - Status report n°2
by Pierre Laperdrix 24 Jun '16

24 Jun '16

Hi everyone, Here is my second status report for my GSOC project. A little reminder that the repo is located on GitHub: https://github.com/plaperdr/fp-central 1 - I have progressed faster than I expected in the last two weeks. Here is everything that I have done: - Storage of fingerprints in a MongoDB database - Adding a small API to get statistics on stored variables - Adding support of hashed variables for faster stats computation - Adding collection of new attributes and support of HTTP headers - Adding support of translation with the start of a French version 2 - I also started development of a page to tell if a user has an "acceptable" fingerprint or not (I haven't pushed the code to GitHub yet). So far, I'm verifying that the screen resolution is in the correct bounds (i.e. not fullscreen) and that there are no plugins in the browser. If anyone has any idea that I could implement to help users have a less recognizable fingerprint, I'll be happy to add it. I have also added steps to follow to help people better configure their browser. 3 - I have tried to add a webpage where I can detect the level of the security slider. This way, I could give recommendations to users to maybe try a higher security level or it would be a way to know the distribution of Tor users on that feature. However, it has proven to be much harder than anticipated. * For "Medium-low", I verify that MathML is disabled. * For "High", I verify that there are either no JavaScript or no SVG elements. * I have troubles to detect the "Medium-High" level. I tried detecting the support of OpenType SVG fonts but it seems that I haven't found the right set of instructions to detect a difference. I'm using a font that I modified where I'm able to display a difference depending on the level of the security slider but I can't detect that difference through JavaScript. When SVG support is present, the displayed character is bigger than the HTML element but I can't detect that it is out of bounds. If anyone has any idea to detect the "Medium-high" level of the security slider, I'll be very happy about it. My goal in the next two weeks is to finish both the "acceptable fingerprint" page and the "slider" page. I also want to start working on a complete statistics page (outside of the main fingerprinting page). Hopefully, in two weeks, I'll have a version that is more complete and from there, I'll start digging into more complicated features like dealing with returning users. Have a great week-end, Pierre

3 6

Revisiting prop224 time periods and HS descriptor upload/downloads
by George Kadianakis 24 Jun '16

24 Jun '16

Hello, during March we discussed the cell formats of prop224: https://lists.torproject.org/pipermail/tor-dev/2016-March/010534.html The prop224 topic for this month has to do with the way descriptors get uploaded and downloaded, how this is scheduled using time periods and how the shared randomness subsystem interacts with all that. Here are some discussion topics. Lots of text on the first two, less text on the rest: - My main goal was to understand the prop224 sections [TIME-PERIODS] and [TIME-OVERLAP]. Those sections specify a system where hidden services decide in a probabilistic manner _when_ to publish their descriptor so that not all hidden services publish their descriptors at the same moment and cause a thundering herd that stampedes the network. For this to work, time is split into time periods of k hours each. A few hours before each time period, there is an overlap period where hidden services start publishing their _next_ descriptors to HSDirs, so that when the upcoming time period starts, all the HSDirs have already received the descriptors and are ready to serve them. Consider the overlap period at the end of time period #N. During that overlap period, hidden services publish their descriptors for future time period #N+1. In this case, hidden services also need to know the shared random value that will be active during time period #N+1, since it needs to be used to find the responsible HSDirs. This means, that the shared random value for time period #N+1 needs to be published _before_ the overlap period starts. This is not the case in current proposal 224, since time is split into time periods of 25 hours, which means that each day the start time shifts by one hour forward. Since the start/end times of the time periods keep on shifting, there will be cases where the right shared random value will not be accessible when the overlap period starts. So what to do? To fix this, I suggest we change the time period length to a day (24 hours). I also suggest we start time periods every day at 12:00 and finish after 24 hours same time, so that it works well with the current shared randomness schedule (where the new shared random value gets published at 00:00 every day). [It might actually be wiser to actually reverse those schedules: create the SRV at 12:00 and start the time period at 00:00] In any case, this is how this might look like: +------------------------------------------------------------------+ | | | 00:00 12:00 00:00 12:00 00:00 12:00 | | SRV#1 TP#1 SRV#2 TP#2 SRV#3 TP#3 | | | | $ |-----------$-----======|-----------$-----======| | | overlap12 overlap23 | | | +------------------------------------------------------------------+ Legend: [TP#1 = Time Period #1] [SRV#1 = Shared Random Value #1] So, this basically gives a space of 12 hours between the SRV generation and the start of the next time period. We can then easily fit an overlap period of 6 hours before the next time periods starts. In the above diagram, the "equal sign" segments are the overlap periods. 'overlap12' is the overlap period from TP#1 to TP#2. Do you think that's reasonable? And do you see any problems with changing the time period length from 25 hours to 24 hours? - So now that we have ironed out the time period stuff slightly, let's discuss the behavior that hidden services, clients and HSDirs should inherit. This email is quite long already so I'm going to go with examples, instead of formal specification. However, this stuff needs to go formally in the proposal IMO, so any help in formalizing it would be great. + Hidden Service behavior: Example 1: Our hidden service boots up at 14:00 of TP#1. In this case, we are nowhere close to the overlap period, so the hidden service should just publish its TP#1 descriptor to the HSDir hash ring using SRV#1 (which at that point should be in consensuses as "shared-rand-current-value"). The hidden service might also want to calculate its overlap OFFSET (as specified in [TIME-OVERLAP]) and schedule a time callback for publishing its TP#2 descriptors. Example 2: Our hidden service boots up at 03:00 of TP#1. That's outside of the overlap period again, but this time the hidden service needs to use the SRV from "shared-rand-previous-value" because the SRV was rotated at midnight. Example 3: Our hidden service boots up at 09:00 of TP#1. That's inside the overlap period, so the hidden service should calculate its overlap OFFSET and compare it with the current time. If it has not passed, then we are in the exact same case as Example 2. If the overlap OFFSET _has_ passed, then the hidden service needs to act as Example 2, and _also_ publish its TP#2 descriptors to a second set of HSDirs using SRV#2. I think these are all the cases for the hidden service, but I would like to formalize this in a way that can be written in the spec. Particularly, I'm not sure how to formalize which SRV to pick at a given time point. + Client behavior My current intuition with regards to client behavior is that they should always fetch descriptors from the HSDirs of the _current_ time period. They should not concern themselves with the overlap stuff _at all_. The overlap system is there so that by the time the new time period starts, all the HSDirs have received the descriptors and are ready to help the clients. Clients should never notice the overlap stuff happening. For this reason I think we can remove this paragraph from the spec: When a client is looking for a service, it must calculate its key both for the current and for the subsequent period, to decide whether the next period's key is valid yet. What do you think? + HSDir behavior Currently the spec says the following: Hidden service directories should accept descriptors at least [TODO: how much?] minutes before they would become valid, and retain them for at least [TODO: how much?] minutes after the end of the period. After discussion with David, we thought of chopping off the first part of that paragraph and not imposing any such weak restrictions for accepting descriptors (see #18332). We still have not decided about the second part of that paragraph, that is how long descriptors should be retained after the end of the period. We currently think clock skew is the only thing that can bring clients to the wrong HSDir after the end of the period. Maybe an hour is OK? David suggested 12 hours. The current Tor is doing 48 hours... Any ideas? And this half-assedly sums up the behaviors of clients/HSes and HSDirs with regards to descriptor uploads and downloads. What is missing, and do you agree that parts of this should be in the proposal? - We should revert the torspec commit: "prop224: avoid replicas with the same blinded key" https://gitweb.torproject.org/torspec.git/commit/?id=8df8c0584392240aa8fecb… It adds a whole lot of complexity to prop224 with no clear security benefit against realistic adversaries. Furthermore, the time period and descriptor download/upload logic of Tor gets very complicated with it. I discussed this with teor and special and found it reasonable. - The randomized revision-counter logic should also be simplified or even removed: https://gitweb.torproject.org/torspec.git/commit/?id=01119bf1291a40aa309dfb… I haven't looked much into this yet. If someone has thoughts please let me know. - We should use fresh salt every time we rebuild the descriptor, but not for every replica: https://gitweb.torproject.org/torspec.git/commit/?id=01e865d592ffcbb67a0e66… - teor says we should revert the double hashing here, and just use tor's random API: https://gitweb.torproject.org/torspec.git/commit/?id=93f47f4f4e7614d4b3debf… peace

5 25

[RELEASE] Torsocks 2.2.0-rc1
by David Goulet 24 Jun '16

24 Jun '16

Hi everyone! I'm releasing a long overdue version of torsocks 2.2.0 release candidate 1. Unfortunately, some other platforms than Linux can break so please let me know either by email or using https://trac.torproject.org what broke. Source: https://people.torproject.org/~dgoulet/torsocks/torsocks-2.2.0-rc1.tar.bz2 Sig: https://people.torproject.org/~dgoulet/torsocks/torsocks-2.2.0-rc1.tar.bz2.… Here is the ChangeLog: https://gitweb.torproject.org/torsocks.git/commit/?id=bcf6ae8c122afff1cfc6a… Worth noting the new command line option to specify the SocksPort address and port of the tor daemon: -a, --address HOST Specify Tor address -P, --port PORT Specify Tor port Also, if you are still experiencing issues with some applications, let me know which one specifically and try to be as detailed as possible with your report. Finally, if anyone with OS X knowledge could check this bug, it would be of great help because this is kind of a show stopper for OS X users that I would like to fix before the stable release. https://trac.torproject.org/projects/tor/ticket/17980 Now is also a good time to speak up if you want something in this release. Cheers! David

3 2

Re: [tor-dev] Freenet + Onioncat: Is the traffic welcome?
by konstant＠mail2tor.com 23 Jun '16

23 Jun '16

[konstant] arma: Do you give this a thumbs up? https://lists.torproject.org/pipermail/tor-dev/2016-June/011108.html [arma] not necessarily? how much traffic is it? [arma] "just add more traffic" mostly hurts tor, alas [arma] since it doesn't really improve security, and it slows it down [arma] if we had a way to add more traffic in a way that helped, that would be grand [konstant] arma: I can't give an exact estimate without working metrics. Freenet has 10KiB/s minimum bandwidth requirement. Optimistically the max. number of nodes in Tor will be around a hundred [konstant] Does this land on the heavy side? [arma] konstant: so 1MB/s of traffic. that's not so bad really. if a few of those freenode nodes set themselves up as tor relays, it would balance out. [arma] the main rule is that if you're going to add traffic to tor, run some relays to match [arma] for hidden services, that's 1MB/s of traffic onto 6 places, so 6MB/s [konstant] I know the seednodes will have 24/7 uptime so they are good relay candidates [konstant] Imma post your responses on the dev ML for posterity

1 0