- tor-dev - lists.torproject.org

Re: [tor-dev] On the visualization of OONI bridge reachability data
by Matthew Finkel 06 Oct '14

06 Oct '14

On Sat, Oct 04, 2014 at 06:27:22PM -0700, M. C. McGrath wrote: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA256 > > Hi, > These were a few possibilities for visualization that we came up with > at the OTF summit (I can send the full notes from that discussion if > everyone is okay with it): > - - Timelines (by protocol, pool, country) > - - Pie charts for above > - - Timeline/graph of time it takes to block bridge from when added to > TBB (github parser) Similar to the next one, I wonder if showing a map cooresponding to this data would also help. At t0, zero countries block the built-in bridges, at t1 = only China blocks, at t2 = China + Iran block, at t3 = China + Iran + Syria block, t4 = t3 + Turkey, etc. I'm thinking this would be nice in addition to the timeline which George sketched (where some of the time points are clickable and update the map). I don't actually know how difficult this is to make. > - - Geographic breakdown by region (if enough data points) Could be > similar to this map of % of internet users who use Tor by country > https://transparencytoolkit.org/tormap.html That's cool. Are you able to add a legend to it? Having something like this or similar to RSF's Press Freedom Index [0], based on the number of bridge users, would be nice. This is doable today, using the available metrics data. We'll probably never be able to know the number of users per protocol per country, but at least we can visualize where in the world bridges (in general) are used most and if this changes over time. [0] http://rsf.org/index2014/en-index2014.php But, it would also be really cool if we can create a map like this based on the reachability of bridges per country per protocol and maybe, in addition, color-code/denote how the ISPs/country are interfering with the connection (e.g. throttling, DNS cache poisoning, IP addr/port blocking). > - - At what point in the tor bootstrapping does it fail (may be > difficult to determine, especially anonymized)? Yes, but there's already a risk to running ooni-probe (at least right now, hopefully this will change in time). We will eventually need probes running in most countries if we want a good understanding of what network interference is taking place and who is affected. > - - In all visualizations, compare with control (filter, line break, > plot alongside, etc) > > And the variables we thought would be relevant to visualizations: > Protocol > Pool > Country (and region)- Iran, China, Netherlands (control) > Time it takes to be blocked > Point in bootstrap where it fails > Classify the bridges by commercial/residential connection > Time we started scanning the bridge from where > Maybe latency measurements per protocol? Initially, I'm thinking "the time is takes to download a consensus from the bridge" but there are many variables that may affect this. Anyone have a better idea? I think this mostly covers it. The only addition can think of right now is comparing different control countries against each other (and different ISPs within the control countries). Maybe we'll find something interesting. > It should be relatively simple to make rough versions of a lot of > visualizations to see what works once we have a parser/converter that > will generate JSONs (or similar) from OONI output that include the > variables listed above. > Is someone already working on this? I'm not really volunteering, merely curious if this is in progress. :) > Are there any other variables that would be particularly helpful to > track or visualize? And are there any visualizations (listed or > otherwise) that anyone would find particularly helpful? > > > On 10/04/2014 06:10 PM, George Kadianakis wrote: > > == What is bridge reachability data? == > > > > By bridge reachability data I'm referring to information about > > which Tor bridges are censored in different parts of the world. > > > > The OONI project has been developing a test that allows probes in > > censored countries to test which bridges are blocked and which are > > not. The test simply takes as input a list of bridges and tests > > whether they work. It's also able to test obfuscated bridges with > > various pluggable transports (PTs). > > > > == Why do we care about this bridgability data? == > > > > A few different parties care about the results of the bridge > > reachability test [0]. Some examples: > > > > Tor developers and censorship researchers can study the bridge > > reachability data to learn which PTs are currently useful around > > the world, by seeing which pluggable transports get blocked and > > where. We can also learn which bridge distribution mechanisms are > > busted and which are not. > > > > Bridge operators, the press, funders and curious people, can learn > > which countries conduct censorship and how advanced technology > > they use. They can also learn how long it takes jurisdictions to > > block public bridges. And in general, they can get a better > > understanding of how well Tor is doing in censorship circumvention > > around the world. > > > > Finally, censored users and world travelers can use the data to > > learn which PTs are safe to use in a given jurisdiction. > > > > == Visualizing bridge reachability data == > > > > So let's look at the data. > > > > Currently, OONI bridge reachability reports look like this: > > https://ooni.torproject.org/reports/0.1/CN/bridge_reachability-2014-07-02T0… > > > > > and you can retrieve them from this directory listing: > > https://ooni.torproject.org/reports/0.1/ > > > > That's nice, but I doubt that many people will be able to access > > (let alone understand) those reports. Hence, we need some kind of > > visualization (and better dir listing) to conveniently display the > > data to human beings. > > > > However, a simple x-to-y graph will not suffice: our ploblem is > > multidimensional. There are many use cases for the data and > > bridges have various characteristics (obfuscation method, > > distribution method, etc.) hence there are more than one useful > > ways to visualize this dataset. > > > > To give you an idea, I will show you two mockups of visualizations > > that I would find useful. Please don't pay attention to the data > > itself, I just made some things up while on a train. > > > > Here is one that shows which PTs are blocked in which countries: > > https://people.torproject.org/~asn/bridget_vis/countries_pts.jpg > > The list would only include countries that are blocking at least a > > bridge. Green is "works", red is "blocked". Also, you can imagine > > the same visualization, but instead of PT names for columns it has > > distribution methods ("BridgeDB HTTP distributor", "BridgeDB mail > > distributor", "Private bridge", etc.). > > > > And here is another one that shows how fast jurisdictions block > > the default TBB bridges: > > https://people.torproject.org/~asn/bridget_vis/tbb_blocked_timeline.jpg > > > > These visualizations could be helpful, but they are not the only > > ones. > > > > What other use cases do you imagine using this dataset for? > > > > What graphs or visualizations would you like to see? > > > > [0]: Here are some use cases: > > > > Tor developers / Researcers: *** Which pluggable transports are > > blocked and where? *** Do they do DPI? Or did they just block the > > TBB hardcoded bridges? *** Which jurisdictions are most aggressive > > and what blocking technology do they use? *** Do they block based > > on IP or on (IP && PORT)? > > > > Users: *** Which pluggable transport should I use in my > > jurisdiction? > > > > Bridge operators / Press / Funders / Curious people: *** Which > > jurisdictions conduct Tor censorship? (block pluggable > > transports/distribution methods) *** How quickly do jurisdictions > > block bridges? *** How many users/traffic (and which locations) did > > the blocked bridges serve? **** Can be found out through extrainfo > > descriptors. *** How well are Tor bridges doing in censorship > > circumvention? > >

2 1

Re: [tor-dev] [tor-relays] French Flag
by Karsten Loesing 03 Oct '14

03 Oct '14

[Please don't cross-post. Removed tor-relays@.] On 02/10/14 17:00, Sebastian Urbach wrote: > Hi, > > I saw that a lot of services around tor are using a bright blue in the > french flag. It should be a dark blue like the one used in the Dutch > flag for example ... > > The following services are concerned: > > Atlas > Globe Please open tickets in both components, ideally with references to each other. Maybe they're using the same library. Though I think we shouldn't start modifying flag images ourselves. The better approach would be to report this problem to whoever made these flags, ask them to get the colors right, and upgrade. > Compass I think Compass doesn't use country flags. All the best, Karsten > Oniontip (already fixed, thanks Donncha) > > If there are more services around that i don't know about then i would > like to ask if someone could check them as well. > > It hurts a bit to see the Tricolore like that ... > > Thanks

2 1

Log output cause for concern?
by Steve Snyder 03 Oct '14

03 Oct '14

Over 3 days I've gotten 9 instances of the sequence shown below. This on Tor v0.2.5.8-rc / x86_64. I am at 80% CPU utilization with BandwidthRate set to 90Mbps on this 100Mbps interface. Should I be concerned about this log output, or is it just verbose because v0.2.5.x is still in development? --------------------------------------- Sep 30 15:30:08.000 [warn] No unused circIDs found on channel without wide circID support, with 0 inbound and 0 outbound circuits. Found 0 circuit IDs in use by circuits, and 64 with pending destroy cells. (0 of those were marked bogusly.) The ones with pending destroy cells have been marked unusable for an average of 16204 seconds and a maximum of 32369 seconds. This channel is 61794 seconds old. Failing a circuit. [3827 similar message(s) suppressed in last 3600 seconds] Sep 30 15:30:08.000 [warn] Circuitmux on this channel has 0 circuits, of which 0 are active. It says it has 32669 destroy cells queued. Sep 30 15:30:08.000 [warn] Channel 2454544 (at 0x7f3ca1b12720) with transport TLS channel (connection 5592182) is in state open (2) Sep 30 15:30:08.000 [warn] * Channel 2454544 was created at 1412029214 (61794 seconds ago) and last active at 1412090927 (81 seconds ago) Sep 30 15:30:08.000 [warn] * Channel 2454544 says it is connected to an OR with digest 7CF62B2A0EA73C3621B79118A40F0A9673FA3146 and no known nickname Sep 30 15:30:08.000 [warn] * Channel 2454544 says its remote address is [scrubbed], and gives a canonical description of "[scrubbed]" and an actual description of "[scrubbed]" Sep 30 15:30:08.000 [warn] * Channel 2454544 has these marks: !bad_for_new_circs canonical is_canonical_is_reliable !client !local outgoing Sep 30 15:30:08.000 [warn] * Channel 2454544 has 0 queued incoming cells and 0 queued outgoing cells Sep 30 15:30:08.000 [warn] * Channel 2454544 has 0 active circuits out of 0 in total Sep 30 15:30:08.000 [warn] * Channel 2454544 was last used by a client at 0 (1412091008 seconds ago) Sep 30 15:30:08.000 [warn] * Channel 2454544 was last drained at 1412090927 (81 seconds ago) Sep 30 15:30:08.000 [warn] * Channel 2454544 last received a cell at 1412090006 (1002 seconds ago) Sep 30 15:30:08.000 [warn] * Channel 2454544 last transmitted a cell at 1412090927 (81 seconds ago) Sep 30 15:30:08.000 [warn] * Channel 2454544 has received 381 cells and transmitted 11081 Sep 30 15:30:08.000 [warn] * Channel 2454544 has averaged 162.188976 seconds between received cells Sep 30 15:30:08.000 [warn] * Channel 2454544 has averaged 5.576573 seconds between transmitted cells Sep 30 19:43:59.000 [notice] Heartbeat: Tor's uptime is 5 days 18:00 hours, with 10312 circuits open. I've sent 3545.65 GB and received 3417.71 GB. Sep 30 19:43:59.000 [notice] Average packaged cell fullness: 99.112% Sep 30 19:43:59.000 [notice] TLS write overhead: 3% Sep 30 19:43:59.000 [notice] Circuit handshake stats since last time: 352835/352843 TAP, 113291/113292 NTor.

1 0

Scaling tor for a global population
by Andrew Lewman 03 Oct '14

03 Oct '14

I had a conversation with a vendor yesterday. They are interested in including Tor as their "private browsing mode" and basically shipping a re-branded tor browser which lets people toggle the connectivity to the Tor network on and off. They very much like Tor Browser and would like to ship it to their customer base. Their product is 10-20% of the global market, this is of roughly 2.8 billion global Internet users. As Tor Browser is open source, they are already working on it. However ,their concern is scaling up to handling some percent of global users with "tor mode" enabled. They're willing to entertain offering their resources to help us solve the scalability challenges of handling hundreds of millions of users and relays on Tor. As this question keeps popping up by the business world looking at privacy as the next "must have" feature in their products, I'm trying to compile a list of tasks to solve to help us scale. The old 2008 three-year roadmap looks at performance, https://www.torproject.org/press/2008-12-19-roadmap-press-release.html.en I've been through the specs, https://gitweb.torproject.org/torspec.git/tree/HEAD:/proposals to see if there are proposals for scaling the network or directory authorities. I didn't see anything directly related. The last research paper I see directly addressing scalability is Torsk (http://www.freehaven.net/anonbib/bibtex.html#ccs09-torsk) or PIR-Tor (http://www.freehaven.net/anonbib/bibtex.html#usenix11-pirtor) Is there a better list available for someone new to Tor to read up on the scalability challenges? -- Andrew pgp 0x6B4D6475 https://www.torproject.org/ +1-781-948-1982

17 35

RFC: Trunnel -- a binary parser generator for Tor and more
by Nick Mathewson 02 Oct '14

02 Oct '14

Hi! Back when the Heartbleed vulnerability went public, I thought to myself: "If we keep generating writing C code to parse binary formats by hand, the same thing will happen to us sooner or later." So on and off, I've been working on a little tool to automatically generate binary encoding and parsing code based on C-like structure descriptions. It's a bit like nail[0], but more lightweight: it doesn't support backtracking, sub-byte alignment, or any of several other features that Tor doesn't actually want. You can see a pre-alpha version of the tool (called "Trunnel") in one of my git repositories [1]. The "doc" subdirectory has a simple manual [2]. To make sure that it would suit our needs, I wrote up trunnel specification files for the current Tor protocols, for SOCKS, and for the new protocols in proposals 220, 224, and 229. It seems to work okay. In case you want to see what the tool does without running it yourself, you can find the SOCKS+proposal229 specification here [3], and compare it with the generated header [4] and C implementation [5]. Trunnel is written in Python; my tests have 99% line coverage on the Python code. The C code which is generated by those tests itself has 99% line coverage from other unit tests, so I'm thinking that it's reasonably tested. (Branch coverage is lower, but still pretty high.) So, please check this out and have a look at it! My open questions are: * Is this a good idea? * Is it a good idea to use this in Tor? * Are there any tricky bugs left in the generated code? * What am I forgetting to think of? FWIW, my current plan is not to do a blanket replacement of our existing parse code with Trunnel (yet) but rather to use Trunnel to generate parsing code for new cell formats that are getting added in 0.2.6 and later. [0] https://github.com/jbangert/nail [1] https://gitweb.torproject.org/user/nickm/trunnel.git . [2] Also online at http://www.wangafu.net/~nickm/trunnel-manual.html [3] https://gitweb.torproject.org/user/nickm/trunnel.git/blob/HEAD:/examples/so… [4] http://www.wangafu.net/~nickm/volatile/socks5.h [5] http://www.wangafu.net/~nickm/volatile/socks5.c cheers, -- Nick Mathewson

1 1

Schedule change for ooni dev meeting at 19:00 CET on Thursday October 2 2014
by Arturo Filastò 01 Oct '14

01 Oct '14

Hello Oonitarians, This is a reminder to inform you that the weekly dev meeting will happen today at 19:00 CET Thursday October 2. Note: That the schedule has changed from Friday to Thursday. Adjust your calendaring software accordingly. See you! ~ Art.

1 0

ooniprobe 1.2.0 and oonibackend 1.1.4 are out!
by Arturo Filastò 01 Oct '14

01 Oct '14

Greetings! We are very pleased to announce the release of new versions of ooniprobe and oonibackend. One of the most interesting new features that are now part of ooniprobe is the ability to generate test decks for the country you are in a way that is much easier than before. As a matter of fact to start contributing useful measurements it's just a mater of 5 minutes of setup: https://pypi.python.org/pypi/ooniprobe#ooni-in-5-minutes Here are some more details of what this release includes: ooniprobe ========= * Introduce a new tool for generating ooniprobe test decks called oonideckgen. * Introduce a new tool for updating resources used for geoip lookup and deck generation. * Add support for policy aware bouncing in the client. https://trac.torproject.org/projects/tor/ticket/12579 * Various improvements to the bridge_reachability test (enable better tor logging and also log obfsproxy) * Fix backward compatibility with twisted 13.1 and add regression tests for this. https://trac.torproject.org/projects/tor/ticket/13139 oonibackend =========== * Add support for specifying the report archive path from config file * Write tor notice level logs to file * Fix bug that lead test helpers to not being started The new packages have been uploaded to pypy and can be found at the following links: https://pypi.python.org/pypi/ooniprobe https://pypi.python.org/pypi/oonibackend Soon ooniprobe will also be available inside of debian testing thanks to Lunar^. Have fun, ~ Art.

1 0

Summary of meek's costs, August 2014
by David Fifield 01 Oct '14

01 Oct '14

The "meek" pluggable transport has started getting some users since my blog post two weeks ago (https://blog.torproject.org/blog/how-use-%E2%80%9Cmeek%E2%80%9D-pluggable-t…) See the attached file userstats-bridge-transport-meek-2014-08-01-2014-08-29.png. Today I logged in to the Google and Amazon accounts to see what the usage looks like. Here is a summary. In short, App Engine cost $1.02 this month, and Amazon cost $2.54. meek-appspot-2014-08-29.png shows the Google App Engine bytes/second for the month of August. App Engine costs $0.12 per GB, with 1 GB for free each day. It's finally at the point where it's costing money every day: All of January 2014: $ -- All of February 2014: $0.10 All of March 2014: $ -- All of April 2014: $0.83 All of May 2014: $0.75 All of June 2014: $0.73 All of July 2014: $0.59 August 2014 to date: $1.02 Total to date $4.02 Here are daily totals in the past two weeks: 2014-08-14 $ -- 2014-08-15 $ -- 2014-08-16 $ -- 2014-08-17 $ -- 2014-08-18 $0.09 2014-08-19 $0.00 2014-08-20 $0.04 2014-08-21 $0.07 2014-08-22 $0.09 2014-08-23 $0.07 2014-08-24 $0.03 2014-08-25 $0.09 2014-08-26 $0.19 2014-08-27 $0.07 2014-08-28 $0.28 meek-aws-2014-08-29.png shows the AWS megabytes per day (note that the y-axis is different from the other graph so they are not directly comparable). The charges break down quite differently: there's no cost for bandwidth so far, and all the charges are for the sheer number of HTTPS requests. The Amazon bill is broken down by region: Asia Pacific (Singapore) Region $0.0090 per 10,000 HTTP Requests 3 Requests $0.01 Asia Pacific (Sydney) Region $ -- EU (Ireland) Region $ -- South America (Sao Paulo) Region $ -- Asia Pacific (Tokyo) Region $0.0090 per 10,000 HTTP Requests 38 Requests $0.01 US East (Northern Virginia) Region $0.0100 per 10,000 Proxy HTTPS Requests (US) 2,513,557 Requests $2.51 $0.0075 per 10,000 HTTP Requests 3 Requests $0.01 Total to date $2.54 David Fifield

1 1

Re: [tor-dev] Scaling tor for a global population
by teor 01 Oct '14

01 Oct '14

[thread reconstructed] >> On 09/30/2014 06:28 AM, AFO-Admin wrote: >>> if we would get multithread support, this would boost the bandwith >>> that is avaible, there i'm sure. >>> All my relays run in CPU limit because i don't think wasting even more >>> ipv4 addresses is great, today you get more and more cpu cores that is >>> not linier with the IPC increase. >>> >>> E.g. you have a Server with 2x E5-2683 v3 v3 and a 10 Gbit/s pipe you >>> would need atleast 14 IP's to use most of the CPU. And every IP's gets >>> blacklistet, with that much Tor nodes in the same /24 maybe the entire >>> /24. So most ISP's wouldn't be happy with that and with the IPv4 >>> shortage this days im also not very happy with that. >>> We really shouldn't waste more IPv4 IP's then needed and the only >>> solution is to change the max amount of Tor Processes from 2 to a >>> higher number or move to IPv6 or get multithreading working. [Moritz quoted] >>> E.g. you have a Server with 2x E5-2683 v3 v3 and a 10 Gbit/s pipe you >>> would need atleast 14 IP's to use most of the CPU. > > Moritz Bartl transcribed 0.5K bytes: >> Raising the limit from 2 relays per IP to x per IP has been discussed in the >> past and would be an easy change. > Date: Tue, 30 Sep 2014 20:35:21 +0000 From: isis <isis(a)torproject.org> > We *still* have that limit? I thought we killed it a long time ago. > > Can we kill it now? It's not going to do anything to prevent Sybils, it'll > only prevent good relay operators on larger machines from giving more > bandwidth. Allowing multiple (>2) tor instances would alleviate AFO's issue in the short term, although in this particular case, they might need 14(!) instances (or AES-NI). Given the shortage of IPv4s, and the availability of multi-processor, high-bandwidth servers, we could trial raising the Tor instance limit per IP. (As this is an authority parameter, the change could happen much sooner than predominantly-IPv6 tor or multithreaded tor.) 4 would allow 1 tor process per logical processor in many server machines (e.g. 4x1 and 2x2). At ~320Mbps per tor process (the maximum bandwidth in the current network), this could saturate a 1 Gbps link with 1 IP. 8 would allow 1 tor process per logical processor in almost all servers (e.g. 4x2 and 8x1), and could saturate a 2.5 Gbps link with 1 IP. In AFO's 10 Gbps case, they'd need 32 processes, or 4 IPs. (Which doesn't seem as unreasonable as 14 IPs.) The only drawback I can see is that IPs with slow connections/few CPUs could then launch 4 or 8 instances, and slow down the network. This could exacerbate the "wasted consensus entry" issue, where the consensus bytes used for a router outweigh its contribution. (But this seems unlikely.) In the short term, can we trial raising the Tor instance limit per IP to 4 or 8? (In the longer term, I'm happy to help with (network) performance, multithreading, or IPv6 - probably in that order.) T teor pgp 0xABFED1AC hkp://pgp.mit.edu/ https://gist.github.com/teor2345/d033b8ce0a99adbc89c5 http://0bin.net/paste/Mu92kPyphK0bqmbA#Zvt3gzMrSCAwDN6GKsUk7Q8G-eG+Y+BLpe7w…

1 0

Damian's Status Report - September 2014
by Damian Johnson 30 Sep '14

30 Sep '14

Code! Beautiful code! How I've missed thee! As mentioned before I'm hip deep in the middle of an arm rewrite. Not only to overhaul the arm codebase, but to ensure Stem will be up to snuff for a Tor GUI or web dashboard (projects I'd *love* to jump into someday). I'm delighted to announce a nice bit of progress on this front. September went toward rewriting its header panel (the top portion of arm's interface)... * Old zomg my eyes, they bleed! https://gitweb.torproject.org/arm.git/blob/e249dc8f5c4c282326324161cf8421f9… * New hotness https://gitweb.torproject.org/arm.git/blob/HEAD:/arm/header_panel.py As mentioned though this is really more about Stem than arm. Stem improvements made along the way this month include... * Far better handling for the 'private' keyword (the default prefix) and the default suffix in our ExitPolicy class. (#10107) * New BaseController.connection_time() method that provides when our control connection was established or detached. [1] * New Controller.get_accounting_stats() method which reports accounting information for our relay. [2] * Controller now fetches our own descriptor if no fingerprint is provided to get_server_descriptor(), get_network_status(), or get_microdescriptor(). * New crop() and join() methods for the str_tools module. [3][4] * Dropped a redundant get_* prefix from most of our util function names. Old names still work as aliases. * Our launch_tor_with_config() function raised an OSError if called too many times, caught thanks to RSenet. (#13141) DocTor, the monitor for directory authority health, also got some love this month... * Corrected suppression behavior when we have staggered alerts. This should greatly cut down on the amount of noise on the list. * We now check if too many relays lack a bandwidth authority measurement... or we would if this wouldn't perpetually be in alarm. (#12989, #13046) * Revised DocTor's schedule as requested by Sebastian. (#13210) * Expanded suppression for messages about turtles and kicked off a conversation to at long last sort this out. (#13296) One final note is that I uploaded my Tor Ecosystem presentation to YouTube so volunteer page visitors can see a streaming presentation rather than downloading a 53 MB video... https://www.youtube.com/watch?v=fb6iqZcQsSg Cheers! -Damian [1] https://stem.torproject.org/api/control.html#stem.control.BaseController.co… [2] https://stem.torproject.org/api/control.html#stem.control.Controller.get_ac… [3] https://stem.torproject.org/api/util/str_tools.html#stem.util.str_tools.crop [4] https://stem.torproject.org/api/util/str_tools.html#stem.util.str_tools.join

1 0