- tor-dev - lists.torproject.org

Re: [tor-dev] Understanding the HS subsystem
by George Kadianakis 04 Feb '15

04 Feb '15

[Declassifying this discussion and posting on [tor-dev]] David Goulet <dgoulet(a)ev0ke.net> writes: > Hello HS elves! > > I wrote a document to organize my thought and also list what we have in > the bug tracker right now about HS behaviours that we want to > understand/measure/assess/track. > > It's a bit long but you can pass the first section describing the > tickets and go right into the How and The Work to be done. > > Nick, you will see there is a SponsorS component but I didn't go into > hard details there. We all know we need a testing network but for now > I'm more focuses on making sure we can collect the right data (for HS). > > Very important part I would like feedback on is the "HS health service" > for which I would like that we all agree of it's usefulness and way to > do it properly. > > Cheers! > David > > This document describes the methodology and technical details of an hidden > service measurement framework/tool/<insert what this is>. > > NOTE: This is NOT intended to be run in the real Tor network. ONLY testing. > > Why and What > ----- > > The goal is to answer some questions we have regarding HS behaviours. Most > of them have a ticket assigned to them but needs an experiment or/and added > feature(s) so we can measure what we need. > > - Is rend_cache_clean_v2_descs_as_dir cutoff crazy high? > https://trac.torproject.org/projects/tor/ticket/13207 > > In order to address this, it seems we need a way to measure all the > interactions with the cache of an HSDir and a client. We need to assess > the rend cache cleanup timing values which will also helps with the upload > and refetch timings. > > - What's the average number of hsdir fetches before we get the hsdesc? > https://trac.torproject.org/projects/tor/ticket/13208 > > Using the control port for that is trivial but this needs a testing > network to be setup and has actual load on it. > > It could also be setup as a feature of an "HS health measurement tool" > with a client fetching over and over the same .onion address randomly over > time. > > - Write a hidden service hsdir health measurer > https://trac.torproject.org/projects/tor/ticket/13209 > > This is a useful one, being able to correlate relay churn and HS desc. > fetch. This one needs more brainstorming on how we could setup some sort > of client or service that report/logs the results on crunching the > consensus for HSDir for a specific .onion address that we know and > control. > > - Refactor rend_client_refetch_v2_renddesc() > https://trac.torproject.org/projects/tor/ticket/13223 > > Insure correctness of this very important function that do fetches for the > client. It's in there that the HSDir (with replicas) are looped on so the > descriptor can be fetched. > > - Maybe we want three preemptive internal circs for hidden services? > https://trac.torproject.org/projects/tor/ticket/13239 > > That's pretty trivial to measure and quantify with the tracing > instrumentation added in Tor. No need for a new feature but an experiment > has to be designed to measure 2 internal circuits versus 3. > > - rend_consider_services_upload() sets initial next_upload_time which is > clobbered when first intro point established? > https://trac.torproject.org/projects/tor/ticket/13483 > > Do the RendPostPeriod option is working correctly. What's the exact > relation in time of service->desc_is_dirty and upload time of a new > descriptor. > > - Do we have edge cases with rend_consider_descriptor_republication()? Can > we refactor it to be cleaner? > https://trac.torproject.org/projects/tor/ticket/13484 > > This is a core function that is called every second so we should make sure > it behaving as expected and not trying to do uneeded upload. > Hello, nice list of tickets. Here are some more ideas if you are looking for more brainstorming action. There is #3733 which is about a behavior that affects performance and could benefit from a testing network. And there is #8950 which is about the number of IPs per hidden service. It's very unclear whether this functionality works as intended or whether it's a good privacy idea. And there is also #13222 but it's probably easier to hack the solution here, than to measure its severity. > > How > ----- > > Here are some steps I think are needed to be able to measure and answer the > Why section. > 2> 1) Dump the uploaded/fetched HS in a human readable way. > * Allows us to track descriptor over time while testing and analyse them > afterwards by correlating events with a readable desc. This kind of > feature will also be useful for people crawling HS on SponsorR. > * Should be a control event like for instance (ONLY client side): > > setconf HSDESC_DUMP /tmp/my/dir > > 2) On how many HSDir (including replicas) have been probed for one > single .onion request. (Which should be repeated a lot for significant > results.) > * Why have we probed 1 or 5? > * What made us retry? Failure code? > * Did the descriptor was actually alive on the HSDir? If not, when did > it move? (Correlate timings between HSdir and client in a testing network) > > 3) HS desc cache tracker. We want to know, very precisely, how things are > moving in the cache especially on the HSDir cache side. > * When and why an HS desc is removed? > * Why it hasn't been stored in the cache? > * Count and when a descriptor is requested. > > 4) Track the HS descriptor upload. Log at what time it was done. Use this > to correlate with RendPostPeriod or when desc_is_dirty is set. Also should > be correlate with the actual state of the HSDir. Did it already have it? > Is the HSDir gone? > > > What to be done > ---------------- > > * Collect data > > "Collect it all" --> https://i.imgur.com/tVXAcGGl.jpg > > It's clear that we have to collect more data from the HS subsystem. Most of > it can be collected through the control port but some are missing. > Measuring precise timing of HS actions (for instance let say descriptor > store) is not possible with the control port right now and also might not be > that relevant since the job of this feature is to report high level events > and push command to the tor daemon. > > Tracing should be used here with a set of events added to the HS subsystem > to collect the information we need so it can be analyzed after the > experiment is run. This is only for performance measurement, the rest should > as much as possible use the control port. > > * Testing network (much SponsorS) > > Once we are able to extract all the data we need, time to design experiment > that allows us to run scenarios and collect/analyze what we want. A scenario > could be this example with a set of questions we want to answer going with > it: > > * 50 clients randomly accessing an HS in a busy tor network. > - What is the failure rate of desc. fetch, RP establishment, ...? > - What are the timings of each component of the HS subsystem? > - What are the outliers of the whole process of establishing a connection > to the HS? > - How much relay churn affected HS reachability. > > And dump a human readable report/graphs whatever is useful for us to > investiguate or assess the HS functionnalities. > > * HS health service > > ref: https://trac.torproject.org/projects/tor/ticket/13209 > > What about a web page that prints the result of: > > 1) Fetch last 3 concensuses (thus 3 hours) > 2) Find the union of all HSDir responsible for a.onion (we control that > HS service and should be up at all time else the results are meaningless.) > 3) Fetch the descriptor on each of them > 4) Graph/log how many of them had it thus giving us a probability of > reaching the HS within a time period. > > So 3) is the tricky one. There are multiple ways of achieving that possibly: > > i) New SOCKS command to tor that a client could use. > - Command would have an onion address with it and the reply should be 0 > or 1 (successful attempt or not) with the HSDir fingerprint with it. > > ii) Control event. > > setconf HSDESC_FETCH_ALL <this_is_a.onion> > [...] > Prints out the results as they come in with the HSDir information. > > iii) A weird way of doing this with an option "tor --fetch-on-all-hs-dir > this_address.onion", print out the results and quit. > > I much prefer i) and ii) here. Not sure which one is best though. Hm, I think I like (ii) here. It doesn't seem to be much more work than (i) and a few researchers have been asking for such functionality for years.

1 0

ANN: txtorcon 0.12.0
by meejah 03 Feb '15

03 Feb '15

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 I'm pleased to announce txtorcon 0.12.0. Full list of improvements: * doc, code and import cleanups from Kali Kaneko * HiddenServiceDirGroupReadable support * Issue #80: honour "ControlPort 0" in incoming TorConfig instance. The caller owns both pieces: you have to figure out when it's bootstraped, and are responsible for killing it off. * Issue #88: clarify documentation and fix appending to some config lists * If GeoIP data isn't loaded in Tor, it sends protocol errors; if txtorcon also hasn't got GeoIP data, the queries for country-code fail; this error is now ignored. * 100% unit-test coverage! * PyPy support (as in: all tests pass) * TCP4HiddenServiceEndpoint now waits for descriptor upload before the listen() call does its callback (this means when using "onion:" endpoint strings, or any of the endpoints APIs your hidden service is 100% ready for action when you receive the callback) * "TorControlProtocol now has an ".all_routers" member, which is a set() of all Routers * TimeIntervalCommaList from Tor config supported * documentation fix from "sammyshj" You can download the release from PyPI or GitHub (or of course "pip install txtorcon"): https://pypi.python.org/pypi/txtorcon/0.12.0 https://github.com/meejah/txtorcon/releases/tag/v0.12.0 Releases are also available from the hidden service: http://timaq4ygg2iegci7.onion/txtorcon-0.12.0.tar.gz http://timaq4ygg2iegci7.onion/txtorcon-0.12.0.tar.gz.asc You can verify the sha256sum of both by running the following 4 lines in a shell wherever you have the files downloaded: cat <<EOF | sha256sum --check 206b1bd8a840119c12d9b85d638ab9defec5b376436fa36be9139ab1ebc8cd78 txtorcon-0.12.0.tar.gz 4e4f6aa2ec677f6c27bff41d17888d31a979f6b831a20501101b39ca93ede9da txtorcon-0.12.0-py2-none-any.whl EOF thanks, meejah -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.12 (GNU/Linux) iQEcBAEBAgAGBQJU0TWHAAoJEMJgKAMSgGmnDEoH/AriLbAyImmcfmRy5C2YcVth eJyXRhHTXc6WBh0tBgryt88o5n55XAwiqNoAwslLvS4RS6w9NEzA9zusimuVYEZs CV10NeC5PiXHJ6qDlcJ+FPsHhWk4zt49wGtaqyfGK/8aZm9enQwiMH6j9Iwx6il0 rLFwm7RoukPTW8dn3oR67QFYwdpHD7cCQW8e6uajpQuBCeNr2nljRpVLFM/6hueh shIAAmPGfBQ2vR16QOQDyJMCLk7oKj0xtzok8O4fWBof1+h3JvMQqphEYlgtE5Lh 9W0XgG/ugrrhjIIoKc29+4Io5vaqKjHcGfqEyJyEimppgGB/6YTX2fJNw2R3Op4= =vaqH -----END PGP SIGNATURE-----

1 0

[Proposal 241] Resisting guard-turnover attacks [DRAFT]
by Nick Mathewson 03 Feb '15

03 Feb '15

Filename: 241-suspicious-guard-turnover.txt Title: Resisting guard-turnover attacks Author: Aaron Johnson, Nick Mathewson Created: 2015-01-27 Status: Draft 1. Introduction Tor uses entry guards to prevent an attacker who controls some fraction of the network from observing a fraction of every user's traffic. If users chose their entries and exits uniformly at random from the list of servers every time they build a circuit, then an adversary who had (k/N) of the network would deanonymize F=(k/N)^2 of all circuits... and after a given user had built C circuits, the attacker would see them at least once with probability 1-(1-F)^C. With large C, the attacker would get a sample of every user's traffic with probability 1. To prevent this from happening, Tor clients choose a small number of guard nodes (currently 1: see proposal 236). These guard nodes are the only nodes that the client will connect to directly. If they are not compromised, the user's paths are not compromised. But attacks remain. Consider an attacker who can run a firewall between a target user and the Tor network, and make many of the guards they don't control appear to be unreachable. Or consider an attacker who can identify a user's guards, and mount denial-of-service attacks on them until the user picks a guard that the attacker controls. In the presence of these attacks, we can't continue to connect to the Tor network unconditionally. Doing so would eventually result in the user chosing a hostile node as their guard, and losing anonymity. 2. Proposed behavior Keep a record of all the guards we've tried to connect to, connected to, or extended circuits through in the last PERIOD days. (We have connected to a guard if we authenticate its identity. We have extended a circuit through a guard if we built a multi-hop circuit with it.) If the number of guards we have *tried* to connect to in the last PERIOD days is greater than CANDIDATE_THRESHOLD, do not attempt to connect to any other guards; only attempt the ones we have previously *tried* to connect to. If the number of guards we *have* connected to in the last PERIOD days is greater than CONNECTED_THRESHOLD, do not attempt to connect to any other guards; only attempt ones we have already *successfully* connected to. If we fail to connect to NET_THRESHOLD guards in a row, conclude that the network is likely down. Stop/notify the user; retry later; add no new guards for consideration. [[ optional If we notice that USE_THRESHOLD guards that we *used for circuits* in the last FAST_REACT_PERIOD days are not working, but some other guards are, assume that an attack is in progress, and stop/notify the user. ]] 2.1. Suggested parameter thresholds. PERIOD -- 60 days FAST_REACT_PERIOD -- 10 days CONNECTED_THRESHOLD -- 8 CANDIDATE_THRESHOLD -- 20 NET_THRESHOLD -- 10 (< CANDIDATE_THRESHOLD) [[ optional USE_THRESHOLD -- 3 (< CONNECTED_THRESHOLD) ]] (Each of the above should have a corresponding consensus parameter.) 2.2. What do we mean by "Stop/warn"? By default, we should probably give warnings in most of the above cases for the first version that deploys them. We can have an on/off/auto setting for whether we will build circuits at all if we're in a "stopped" mode. Default should be auto, meaning off for now. The warning needs to be carefully chosen, and suggest a workaround better than "get a better network" or "clear your state file". 2.3. What's with making USE_THRESHOLD optional? Aaron thinks that getting rid of it might help in the fascistfirewall case. I'm a little unclear whether that makes any of the attacks easier. 3. State storage requirements Right now, we save for each guard that we have made contact with: ID Added is dircache? down-since last-attempted bad-since chosen-on-date, chosen-by-version path bias info (circ_attempts, successes, close_success) To implement the above proposal, we'll need to add, for each guard *or guard candidate*: when did we first decide to try connecting to it? when did we last do one of: decide to try connecting to it? connect to it? build a multihop circuit through it? which one was it? Probably round these to the nearest day or so. 4. Future work We need to make this play nicely with mobility. When a user has three guards on port 9001 and they move to a firewall that only allows 80/443, we'd prefer that they not simply grind to a halt. If nodes are configured to stop when too many of their guards have gone away, this will confuse them. If people need to turn FascistFirewall on and off, great. But if they just clear their state file as a workaround, that's not so good. If we could tie guard choice to location, that would help a great deal, but we'd need to answer the question, "Where am I on the network", which is not so easy to do passively if you're behind a NAT. Appendix A. Scenario analysis A.1. Example attacks * Filter Alice's connection so they can only talk to your guards. * Whenever Alice is using a guard you don't control, DOS it. A.2. Example non-attacks * Alice's guard goes down. * Alice is on a laptop that is sometimes behind a firewall that blocks a guard, and sometimes is not. * Alice is on a laptop that's behind a firewall that blocks a lot of the tor network, (like, everything not on 80/443). * Alice has a network connection that sometimes turns off and turns on again. * Alice reboots her computer periodically, and tor starts a little while before the network is live. Appendix B. Acknowledgements Thanks to Rob Jansen and David Goulet for comments on earlier versions of this draft.

3 2

[RESEARCH] Student of University of Verona, Thesis about TOR
by Diego Sempreboni 02 Feb '15

02 Feb '15

Hi, I am a student of Engineering of the University of Verona. For my master's thesis, I decided to analyze TOR network, analyzing various attack scenarios. By consulting several documents have not been able to obtain a pattern of messages exchanged during the handshake protocol nor other details about the packets exchanged later. It would be possible to have some information about this? I write hoping someone can help me. Thank you.

3 2

Estimating censorship lag by obfs4 blocking
by David Fifield 02 Feb '15

02 Feb '15

According to a couple of blog comments on February 1, 2015, the default obfs4 bridges are blocked by GFW. https://blog.torproject.org/blog/tor-browser-45a3-released#comment-86416 https://blog.torproject.org/blog/tor-browser-45a3-released#comment-86907 The obfs4 bridges were introduced on November 17, 2014. https://blog.torproject.org/blog/tor-browser-45-alpha-1-released I have a message from Yawning Angel verifying that the obfs4 bridges were reachable on December 2, 2014. With this information, we can put rough bounds on how long it took the GFW to react to a new circumvention system: between 2 and 10 weeks. Does anyone know information that could tighten up the bounds, a date between December 2 and February 1 when obfs4 bridges were known to be either accessible or inaccessible? David Fifield

1 0

Damian's Status Report - January 2015
by Damian Johnson 01 Feb '15

01 Feb '15

Greetings wonderful world. January has been delightfully focused on a couple substantial feature branches that are now merged! -------------------------------------------------------------------------------- Descriptor Lazy Loading -------------------------------------------------------------------------------- When reading descriptors without validation (which is now the default) Stem is 25-70% faster! https://lists.torproject.org/pipermail/tor-dev/2015-January/008211.html This is because we now parse descriptors fields on-demand. Just need a server descriptor's exit policy? That's all we'll bother reading. This is why the heavier the descriptor type the larger the benefit. -------------------------------------------------------------------------------- Unified Python 2/3 Codebase -------------------------------------------------------------------------------- Thanks to Foxboron Stem now has a unified python 2/3 codebase! https://trac.torproject.org/projects/tor/ticket/14075 This means that Stem now runs directly with any version of python from 2.6 to 3.x. Prior to this Stem did a 2to3 export when you installed it to provide cross-version compatibility. For users, the benefit of Foxboron's overhaul is that python3 installation and test runs are now much, much quicker. -------------------------------------------------------------------------------- Beside these a few other noteworthy tasks were... * Nick had a handful of suggestions for testing improvements, neatest of which was to include the command to re-run just individual tests when we have a failure. (#14113) * Our port_usage() function always returned None. (#14046) * Proc connection resolution could fail on busy systems. (#14048) * Fixed how the 'strict' argument works with exit policies. (#14314) Cheers! -Damian

1 0

Feature freeze schedule for Tor 0.2.6
by Nick Mathewson 31 Jan '15

31 Jan '15

Hi! Here's my plan for the Tor 0.2.6 feature freeze. I had hoped to to the freeze today, but I'm going to be travelling to a kid's birthday party for most of the day. So instead, I'm aiming for *Monday* for merging all features that are done, and giving possible extensions to high-value features that aren't yet implemented, but which aren't too far away. Until Feb 17, I'll take bugfixes and features that got extensions. After that, the only bugs I want to merge fixes for are: * bugs that have trivial obvious fixes * regressions * significant security bugs. (Note that the absence of a feature is not a bug). I hope this will let us get 0.2.6 stabilized in March. I'm planning to open an 0.2.7 development branch in early March one way or another. I'm going to write another email or blog post or something about all the cool stuff that's going into Tor 0.2.6 (and some of the stuff that I hope will go into 0.2.7) -- the development team has done a really excellent job, and I'm really happy with how well we've all been working together. Special thanks to all the excellent volunteers as well. best wishes, -- Nick

1 0

Re: [tor-dev] [Cryptography] traffic analysis
by grarpamp 27 Jan '15

27 Jan '15

On Tue, Jan 27, 2015 at 3:23 PM, Ben Laurie <benl(a)google.com> wrote: > Yeah, but ... who can realistically afford that bandwidth? One can afford what they wish to give to and thus expect to get from the network in return, no more, no less, as always. (Be it literally from the physical NIC network, or the logical networks created by their applications riding over and within the physical.) > To every possible recipient? Clearly you have to make a tradeoff. Full meshing of chaff addressed to all participants seems unnecessary. > grarpamp wrote: > Is there so much (possibly far less than correct) thought out there > that fill bandwidth is evil, untolerable, unmanageable, and blocking > of usability such that these networks are moot to even try coding > for general deployment? On Tue, Jan 27, 2015 at 1:35 PM, Jerry Leichter <leichter(a)lrw.com> wrote: > Google Fiber offers 1Gb/second - but how many customers running all > out will overload any possible backbone behind the single link from the > house to the concentrator? > > If everyone starts sending constant cover traffic, links will be quickly > overloaded all over the place. At which point the providers will start > charging [...] nobody will be happy That's the simple man's kneejerk response when initially contemplating chaffed networks. > There's room to do much better. For one thing, you don't need to saturate your link with cover traffic - you need to send enough cover traffic so that a listener can't tell the difference between cover and real traffic. This depends on if you're network is a low latency one, or if it uses a store and forward model. Even that is tricky due to needs to hide the size of data each endpoint exchanges from passive adversaries. It begins to approach constant higher rate the more you data wish to pass securely. > If your cover traffic rate equals your average rate over some period of time, > you're not adding more traffic - you're simply replacing some of your cover > messages with real messages. But ... what happens when you have a > peak demand way above your average? Then you must wait until you can pass your wheat. Or plan your needs according to the give and get model. > As Stealthmonger has commented concerning anonymity, if you want > security against traffic analysis, you have to accept delays: Set your > cover traffic rate somewhat higher than your average rate, and you'll > *eventually* catch up with peaks (though as with any queueing system, > the delays can grow without bound - requiring unbounded memory > *somewhere*). Delays (trading latency) are not the only way to achieve security. You may also elect to trade available throughput bandwidth in a wheat/chaff system. > I'm not aware of any open research on these kinds of questions - though > it may well be out there. What's the optimum cover traffic rate under > various assumptions about the real traffic rate and distribution? When > is it safe to use the traffic other users present as cover for your own? > Clearly if there's only one other user sending traffic, you can't use him > for cover as *he* can tell which of the packets are yours. But is there a > way to mix traffic from multiple users in a way that requires large numbers > of them to conspire to reveal anything? The mixmaster stuff looks at this > specifically from the point of view of a store-and-forward node - is there > some suitable useful analogue on a single link? Can we somehow get > the same guarantees without storage inside the network? There were some research references posted in the threads below. > we're now going to have to change our attitudes toward traffic analysis. Yes, a lot of oppurtunity exists to create new working networks in this area that utilize fill traffic and/or delay. You may want to review the recent guardian-dev and tor-dev side threads below on this exact subject of link padding, latency and analysis. Relavent papers and talk have been posted. https://lists.mayfirst.org/pipermail/guardian-dev/2014-November/004040.html https://lists.mayfirst.org/pipermail/guardian-dev/2014-November/004069.html https://lists.torproject.org/pipermail/tor-dev/2014-November/007741.html https://lists.torproject.org/pipermail/tor-dev/2014-December/007934.html https://lists.torproject.org/pipermail/tor-dev/2015-January/008039.html http://www.metzdowd.com/pipermail/cryptography/2015-January/024479.html https://lists.torproject.org/pipermail/tor-dev/2015-January/008099.html [Copied a few places simply to include these ongoing links as reference for anyone interested.]

1 0

Regarding contributing to torproject
by Vipul Sharma 27 Jan '15

27 Jan '15

Hello developers, I have some experience in Python/Django and looking to contribute to torproject. I've read about many projects mentioned at: https://www.torproject.org/getinvolved/volunteer.html.en#Projects but still I need your help and advice to pick a project. Please suggest me which project should I look for, considering that I am new to torproject :) Apart from this, there are few questions I would like to ask: - Which is the best place to ask questions: mailing list or irc(#tor-dev) (I don't want to spam with my questions in this mailing list) ? - How should I communicate or notify before sending a PR ? - Is it necessary to issue a ticket before sending a PR ? (Supposing that I've found an issue) I know these are pretty novice questions but I need your help :) Regards, Vipul Sharma

1 0

Reminder for weekly OONI dev meeting Monday 2015-01-26 18:00 UTC (19:00 CET)
by Arturo Filastò 26 Jan '15

26 Jan '15

Hello Oonitarians, This is a reminder that today there will be the weekly OONI meeting. It will happen as usual on the #ooni channel on irc.oftc.net at 18:00 UTC (19:00 CET, 13:00 EST, 10:00 PST). Everybody is welcome to join us and bring their questions and feedback. See you later, ~ Arturo

1 0