- tor-dev - lists.torproject.org

Scaling Tor Metrics, Round 2
by Karsten Loesing 28 Dec '15

28 Dec '15

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hi everyone, I posted some thoughts on Scaling Tor Metrics [0] almost two weeks ago and received very useful feedback from George, David, Thomas, and Letty. Thanks for that! In fact, this is such a big topic and this was so much feedback that I decided not to respond to every idea individually but instead start over and suggest a new plan that incorporates all feedback I saw. Otherwise, I'd be worried that we'd lose ourselves in the details and miss the big picture. Maybe this also makes this topic somewhat easier to follow and respond to, which I hope many people will do. The problem to solve is still that the Tor Metrics website has a huge product backlog and that we want it to remain "the primary place to learn interesting facts about the Tor network", either by making it better and bigger internally, or by adding more ways to let others contribute to it externally. - From the feedback in round 1, I observe three major areas where we need to improve Tor Metrics: 1 Metrics frontend 2 Metrics backend 3 External contributions There are low-hanging fruit in each area, but there are many fruit overall and some are hanging higher than we might think. I'll go through them area by area and assign numbers to the tasks. 1 Metrics frontend The frontend is the part of the Tor Metrics website that takes pre-aggregated data in the .csv format as input and produces somewhat interactive visualizations. The current frontend uses Java servlets/JSP as web framework and R/ggplot2 for graphs. This code is hard to extend, even for me, and the result isn't pretty, but therefore the website doesn't require any JavaScript. So, one task would be: #1 decide whether we can still ignore JavaScript and what it has to offer. I agree that D3.js is cool, I even used it myself in the past, though I know very little about it. This decision would mean that we develop new visualizations in D3.js and phase out the existing R/ggplot2 visualizations one by one. This is a tough decision, but one with a lot of potential. I understand how we're excited about this as developers, but I'd want to ask Metrics users about this first. Another task would be: #2 website redesign. In fact, what you see on the website right now is a redesign half-way through. Believe me, the website was even less readable a year or two ago, and this statement alone tells you how slow things are moving. But the remaining step is just to replace the start page with a gallery, well, and to apply a Bootstrap design to everything, because why not. One challenge here is that the current graphs all look quite similar making them hard to distinguish in a gallery, but that's probably still more useful than putting text there. It sounds like Letty and Thomas might be willing to help out with this, which would be great. Yet another task would be: #3 replace the website framework with something more recent. This can be something simple, as long as it supports some basic filtering and maybe searching on the start page. I'd say let's pick something Python-based here. However, maybe we should first replace the existing graphs that are deeply tied into the current website framework. If we switch to D3.js and have replaced all existing graphs, this switch to a new website framework will hurt a lot less. Another high-hanging fruit would be: #4 build something like Thomas' Visionion, where users can create generic visualizations on the fly. This is an exciting idea, really, but I think we have to accept that it's out of reach for now. 2 Metrics backend The backend of Tor Metrics consists of a bunch of programs that run once per day to fetch new data from CollecTor and produce a set of .csv files for the frontend. There are no strict requirements to languages and databases, as long as tools are available in Debian stable. Some programs use PostgreSQL, but most of them just use files. Ironically, it's the database-based tools that have major performance problems, whereas the file-based ones work just fine. Most programs are written in Java, very few in Python. One rather low-hanging fruit would be: #5 document backend programs and say what's required to add one more to the bunch. The biggest challenge in writing such a program is that it needs to stay reasonably fast even over the next couple of years and even if the network doubles or triples in size. I started documenting things a while ago, but got distracted by other things. I wouldn't mind help. Another rather low-hanging fruit would be: #6 use Big Data to produce pre-aggregated data for the frontend. As said before, it doesn't matter whether a backend program uses files or PostgreSQL or another database engine. What matters is that it reads data from CollecTor and produces a data that the frontend can use. This could be a CSV or JSON file. We should probably pick the next visualization project as test case for applying Big Data tools, or we could rewrite one that needs to be rewritten for performance reasons anyway. Here's a not-so-low-hanging fruit for the backend: #7 have the backend provide an API to the frontend. This is potentially more difficult. The part that I'm very much afraid of is performance. It's just too easy to build a tool that performs reasonable during testing but that can't handle the load of 10 or 100 people looking at a frontend visualization at once. In particular, it's easy to build an API that works just fine and then add another feature that looks harmless, which later turns out to hurt performance a lot. I'd say postpone. 3 External contributions Most of the discussion in round 1 circled around how external contributions are great, but that they need to be handled with care. I understand the risks here, and I think I'll postpone the part where we're including externally developed websites after "redressing" them. Let's instead try to either keep those contributions as external links or properly integrate them into Tor Metrics. The lowest-hanging fruit here is: #8 keep adding links to external websites as we already do, assuming that we clearly mark these contributions as external. Another low-hanging fruit is: #9 add static data or static graphs. I heard a single +1, but I guess once we have concrete suggestions what data or graphs to add, we'll have more discussion. That's fine. One important and still somewhat low-hanging fruit is: #10 give external developers more support when developing visualizations that could later be added to Metrics. This requires better documentation, but it also requires making it easier to install Tor Metrics locally and test new additions before submitting them. The latter is a good goal, but we're not there yet. The documentation part doesn't seem crazy though. David, if you don't mind being the guinea pig yet once more, I'd want to try this out with your latest visualizations. This is pending on the JavaScript decision though. Now to the higher-hanging fruit: #11 Build or adapt a tool like Munin to prototype new visualizations quickly. I didn't fully understand how Munin makes it easier to prototype a visualization than just writing a Python/Stem script for the data pre-processing and using any graphing engine for the visualization. But maybe there's something to learn here. Still, this seems like a quite high goal at the moment. Another one: #12 provide a public API like Onionoo but for Metrics data. This seems somewhat out of reach for the moment. It's a cool idea, but doing it right is not at all trivial. I'd want to put this to the lower end of the list. Another high-hanging fruit: #13 adapt the Gist idea where external contributors write some code that magically turn into new visualizations. I think that most new visualizations require writing backend code to aggregate different parts of the available data or to aggregate the same data differently. It's a neat idea that external contributors could simply write some code that then magically turns into new visualizations. But I think it's a long way until we're there. 4 Summary Here's the list of low-hanging fruit: #1 decide whether we can still ignore JavaScript #2 website redesign #5 document backend programs #6 use Big Data to produce pre-aggregated data #8 keep adding links to external websites #9 add static data or static graphs #10 give external developers more support And here are the tasks that I think we should postpone a bit longer: #3 replace the website framework #4 build something like Thomas' Visionion #7 have the backend provide an API to the frontend #11 Build or adapt a tool like Munin #12 provide a public API like Onionoo but for Metrics data #13 adapt the Gist idea How does this sound? What did I miss (sorry!), on a high level without going into all the details just yet? Who wants to help? All the best, Karsten [0] https://lists.torproject.org/pipermail/tor-dev/2015-November/009983.html -----BEGIN PGP SIGNATURE----- Comment: GPGTools - http://gpgtools.org iQEcBAEBAgAGBQJWZFnNAAoJEJD5dJfVqbCrowMIAIcx5kd7BfHN9HsgEj+JZ6e6 YsIUfrrCcTeMlZyHHG9amWodac+spE11fZCAkZ2B2gYf8I6KT23pex4w23WCVlxo 0qq5pZSADedAYBsocEhZLh2bZyzIhEPueGJW2/HeyW2Smd9T+YJ8hA1kkoebX/Xv xb/2051bc48sQW767gst3ClTv4va8+24pKytJxvfRYDrs3xv3VxPlyCvLsrgz9ji pnxd2C+Syo4sT3vHFb9JNhvd1ZaEmqjXBzywQ2SNnOqZNQE77zfvB766FFSJEg6a MMgwt0RcjElTnZRZR2utA2WLYzJDiof4ULfw51xoYYcT6kmGQw9IOqOuzgBvSr4= =Yo+I -----END PGP SIGNATURE-----

4 12

metrics-lib 1.1.0 is released
by Karsten Loesing 28 Dec '15

28 Dec '15

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hello devs, I just released metrics-lib 1.1.0: https://dist.torproject.org/descriptor/1.1.0/ - From the change log: # Changes in version 1.1.0 - 2015-12-28 * Medium changes - Parse flag thresholds in bridge network statuses, and parse the "ignoring-advertised-bws" flag threshold in relay network status votes. - Support parsing of .xz-compressed tarballs using Apache Commons Compress and XZ for Java. Applications only need to add XZ for Java as dependency if they want to parse .xz-compressed tarballs. - Introduce a new ExitList.Entry type for exit list entries instead of the ExitListEntry type which is now deprecated. The main difference between the two is that ExitList.Entry can hold more than one exit address and scan time which were previously parsed as multiple ExitListEntry instances. - Introduce four new types to distinguish between relay and bridge descriptors: RelayServerDescriptor, RelayExtraInfoDescriptor, BridgeServerDescriptor, and BridgeExtraInfoDescriptor. The existing types, ServerDescriptor and ExtraInfoDescriptor, are still usable and will not be deprecated, because applications may not care whether a relay or a bridge published a descriptor. - Support Ed25519 certificates, Ed25519 master keys, SHA-256 digests, and Ed25519 signatures thereof in server descriptors and extra-info descriptors, and support Ed25519 master keys in votes. - Include RSA-1024 signatures of SHA-1 digests of extra-info descriptors, which were parsed and discarded before. - Support hidden-service statistics in extra-info descriptors. - Support onion-key and ntor-onion-key cross certificates in server descriptors. * Minor changes - Start using Java 7 features like the diamond operator and switch on String, and use StringBuilder correctly in many places. Thanks to iwakeh for driving many of the changes contained in this release and to thms for spotting missing features that are now included. Happy descriptor parsing! All the best, Karsten -----BEGIN PGP SIGNATURE----- Comment: GPGTools - http://gpgtools.org iQEcBAEBAgAGBQJWgPSdAAoJEJD5dJfVqbCrYdEIAI+zZzM8iDRplpmlJ62ehkGi r07yo14VfFHxa2od+abh8A6pOM+hdwJjeYOQ1dz/HGf6xIl7GmdDt1Tt+KMM0bAc F7r+1Yi4eo8G7ZqNIDIsJdey59orMLjpDafUFuXYv4RM22ol5zj5b3Hj94OIQM9A +gZXReoCxJQS+WxOfsMVP2yPjdQsoptZCJQjIJMiu5fL0lQF3tkCLuJ6r5teLnLj 1VGx+YsJWKE8czM42yfpY4/SmnTkUtjmYO2Ivq/WOMuCh7RcbBSGkr09PB7fxrHT ZmoML92TqI0+zwEY7yI52pjn+Q+EX5BY8qvKeCen14b/BwKCIKXCRqaye15RTX4= =VkEL -----END PGP SIGNATURE-----

1 0

Traffic correlation attacks on Hidden Services
by Virgil Griffith 24 Dec '15

24 Dec '15

I've been looking into simple graph-theoretic metrics for Roster to quantifying Tor's susceptibility to traffic correlation attacks, mostly using BGPStream, https://bgpstream.caida.org/ . All of the academic literature I've read talks about the risk to Tor users of an AS being in the path between client <-> guard + exit <-> destination. Questions: (1) To ensure I'm not measuring the wrong thing, can someone be more specific on the correlation attack scenario for Tor hidden services? (2) Just guessing, but would be it be the same but replace "exit <-> destination" with: "HS server <-> HS guard" ? (3) If yes to (2), the natural solution is simply to install a Tor relay on the HS server itself so that there's no ASpath between the two? Comments greatly appreciated. I'm not an internet routing expert and I want to ensure Roster is incentivizing the right things to harden the network. -V

2 1

IDE for Tor?
by Lain Iwakura 15 Dec '15

15 Dec '15

Hello guys; Sorry such silly questions novice. But what IDE you use, and why? In Linux, of course. ... For C and python. I see Eclipse (Mars), but it seems more to own java. Codeblocks, perhaps? Excuse me still could not find me in the documentation. thank you : )

4 3

Reminder: Tomorrow's Wednesday Tor dev meeting is 1330 UTC!
by Nick Mathewson 15 Dec '15

15 Dec '15

---------- Forwarded message ---------- From: Nick Mathewson <nickm(a)freehaven.net> Date: Thu, Dec 10, 2015 at 2:49 PM Subject: Weekly Tor developers meetings; IRC time change! To: tor-dev(a)lists.torproject.org Hi, all! (This is about the IRC meeting on OFTC on #tor-dev to talk about developing the program called "tor".) We have have fewer people at the new Wednesday time than we did at the old one. That's not great. The Monday/Tuesday time is convenient for a bunch of folks, though, so that's nice. So for next week at least, we will move Wednesday earlier to be closer to the old time, and keep Monday/Tuesday as it is. That makes the meeting times: TUESDAY 0100 UTC (== 8 pm eastern on Monday. No change.) WEDNESDAY 1330 UTC (==0830 am eastern on Wednesday. Changed.) There's no need to show up to both, but you can if that's convenient. If both are equally convenient for you and you only want to do one, please favor the Wednesday one. One of the topics at the next meetings will be Good Meeting Times and Getting More People to Come. If you want to talk about meeting times, but neither of these times will work, just send me a note or find me online so I know. Let's keep trying till we get this right! cheers, -- Nick

1 0

apparmor in lxc containers [#17754]
by Peter Palfrader 15 Dec '15

15 Dec '15

Hi Intri, https://bugs.torproject.org/17754 reports that tor no longer works in LXC containers. I have set up an ubuntu wily VM, and a wily LXC container in it, and I can confirm that with the AppArmorProfile= line in the service file, tor will not launch. Do you have any ideas how to properly fix this? Or what the best workaround would be to document? Thanks, weasel -- | .''`. ** Debian ** Peter Palfrader | : :' : The universal https://www.palfrader.org/ | `. `' Operating System | `- https://www.debian.org/

3 2

Summary of meek's costs, November 2015
by David Fifield 14 Dec '15

14 Dec '15

Here's the summary of meek's CDN fees for November 2015. App Engine + Amazon + Azure = total by month February 2014 $0.09 + -- + -- = $0.09 March 2014 $0.00 + -- + -- = $0.00 April 2014 $0.73 + -- + -- = $0.73 May 2014 $0.69 + -- + -- = $0.69 June 2014 $0.65 + -- + -- = $0.65 July 2014 $0.56 + $0.00 + -- = $0.56 August 2014 $1.56 + $3.10 + -- = $4.66 September 2014 $4.02 + $4.59 + $0.00 = $8.61 October 2014 $40.85 + $130.29 + $0.00 = $171.14 November 2014 $224.67 + $362.60 + $0.00 = $587.27 December 2014 $326.81 + $417.31 + $0.00 = $744.12 January 2015 $464.37 + $669.02 + $0.00 = $1133.39 February 2015 $650.53 + $604.83 + $0.00 = $1255.36 March 2015 $690.29 + $815.68 + $0.00 = $1505.97 April 2015 $886.43 + $785.37 + $0.00 = $1671.80 May 2015 $871.64 + $896.39 + $0.00 = $1768.03 June 2015 $601.83 + $820.00 + $0.00 = $1421.83 July 2015 $732.01 + $837.08 + $0.00 = $1569.09 August 2015 $656.76 + $819.59 + $154.89 = $1631.24 September 2015 $617.08 + $710.75 + $490.58 = $1818.41 October 2015 $672.01 + $110.72 + $300.64 = $1083.37 November 2015 $602.35 + $474.13 + $174.18 = $1250.66 -- total by CDN $8045.93 + $8461.45 + $1120.29 = $17627.67 grand total https://metrics.torproject.org/userstats-bridge-transport.html?graph=userst… The number of users is holding steady at around 4,000–5,000. meek-google had an outage as a result of a server migration between 23 November and 4 December. If you want to set up your own bridge and CDN instance without rate limiting, I can help you do it. Here are some docs to look at: https://trac.torproject.org/projects/tor/wiki/doc/meek#Howtorunameek-server… https://trac.torproject.org/projects/tor/wiki/doc/meek#GoogleAppEngine https://trac.torproject.org/projects/tor/wiki/doc/meek#AmazonCloudFront https://trac.torproject.org/projects/tor/wiki/doc/meek#MicrosoftAzure == App Engine a.k.a. meek-google == Here is how the Google costs broke down: 2162 GB $259.52 6856 instance hours $342.83 Compared to the previous month: 2842 GB $341.06 6619 instance hours $330.95 https://globe.torproject.org/#/bridge/88F745840F47CE0C6A4FE61D827950B06F9E4… == Amazon a.k.a. meek-amazon == Asia Pacific (Singapore) 39M requests $47.84 361 GB $52.28 Asia Pacific (Sydney) 1M requests $2.24 4 GB $0.64 Asia Pacific (Tokyo) 16M requests $20.27 106 GB $13.78 EU (Ireland) 122M requests $146.93 736 GB $58.25 South America (Sao Paulo) 3M requests $8.43 11 GB $2.71 US East (Northern Virginia) 78M requests $78.74 535 GB $42.01 -- total 263M requests $304.45 1755 GB $169.67 https://globe.torproject.org/#/bridge/F4AD82B2032EDEF6C02C5A529C42CFAFE5165… == Azure a.k.a. meek-azure == Zone 1 1467 GB $126.31 Zone 2 350 GB $47.87 -- total 1817 GB $174.18 https://globe.torproject.org/#/bridge/AA033EEB61601B2B7312D89B62AAA23DC3ED8… Earlier reports in this series: https://lists.torproject.org/pipermail/tor-dev/2014-August/007429.html https://lists.torproject.org/pipermail/tor-dev/2014-October/007576.html https://lists.torproject.org/pipermail/tor-dev/2014-November/007716.html https://lists.torproject.org/pipermail/tor-dev/2014-December/007916.html https://lists.torproject.org/pipermail/tor-dev/2015-January/008082.html https://lists.torproject.org/pipermail/tor-dev/2015-February/008235.html https://lists.torproject.org/pipermail/tor-dev/2015-March/008427.html https://lists.torproject.org/pipermail/tor-dev/2015-April/008596.html https://lists.torproject.org/pipermail/tor-dev/2015-May/008767.html https://lists.torproject.org/pipermail/tor-dev/2015-June/008932.html https://lists.torproject.org/pipermail/tor-dev/2015-July/009030.html https://lists.torproject.org/pipermail/tor-dev/2015-August/009213.html https://lists.torproject.org/pipermail/tor-dev/2015-September/009533.html https://lists.torproject.org/pipermail/tor-dev/2015-October/009672.html https://lists.torproject.org/pipermail/tor-dev/2015-November/009941.html

1 0

Help me help you : )
by Lain Iwakura 11 Dec '15

11 Dec '15

Hello guys; My name is Lain, I'm from Brazil. only it's not my real name. :) And I love it. I identify a lot with the Tor project, and would like a simple guideline. I work with programming (Pascal) in the last five years, I want to contribute to developing the project. but do not know where to start, take that language, which would be more effective my acting. ... something that a veteran is tired of doing I could do, and learn from that expedite the work. Anyway, give me some tips. Hey! I am learning English for that. help me to be useful. :)

3 3

Bitcoin-paid hidden meek relays?
by Jacek Wielemborek 11 Dec '15

11 Dec '15

List, I'm starting this thread in relation to the following discussion: https://security.stackexchange.com/questions/107490/how-will-france-block-t… I definitely don't have a perfect knowledge of Tor, so correct me if I'm wrong at any point. I heard that Tor entry nodes right now are distributed to users as IP addresses, which are scarce resources that could be blocked. In order to make this more difficult, pluggable transports were introduced - those try to use other protocols to define communication with Tor network, lowering the cost of introducing new way to connect with it. One that particularly caught my attention is Meek, which uses CDNs to evade censorship. CDNs share same IPs for many critical Internet serices, so blocking them is not an option and having them behind TLS makes it even more complicated. The problem I noticed though is that the costs of Meek go up and if I read the reports from David Fifield (the maintainer of Meek), the bandwidth has to be limited to avoid abuse. This slows the transport down and I thought of another approach. Instead of using a limited amount of Meek nodes, we could encourage users to run those on their own and distribute them in a way similar to regular Tor entry nodes. The catch would be that those nodes would require a small Bitcoin pre-payment that covers the cost of bandwidth used. The node would require the user to pass some proof of payment during the connection and not respond at all if there was no payment made. Here's an example way this could work: 1. User sends a message to bridges(a)bridges.torproject.org with a request for a meek bridge and a GPG public key attached, 2. The bridge service replies with a Bitcoin address created just for the user, 3. As soon as the user sends any Bitcoins, it receives another e-mail with the hostname of the Meek relay and a token that one should use Together with Amazon AWS, this could be an automated solution to the bandwidth and payment problem that would result in short-lived relays that are difficult to detect. What do you think about it? Hopefully I didn't mess it up in the fundamentals and this could actually help any little. Cheers, d33tah

5 5

Weekly Tor developers meetings; IRC time change!
by Nick Mathewson 10 Dec '15

10 Dec '15

Hi, all! (This is about the IRC meeting on OFTC on #tor-dev to talk about developing the program called "tor".) We have have fewer people at the new Wednesday time than we did at the old one. That's not great. The Monday/Tuesday time is convenient for a bunch of folks, though, so that's nice. So for next week at least, we will move Wednesday earlier to be closer to the old time, and keep Monday/Tuesday as it is. That makes the meeting times: TUESDAY 0100 UTC (== 8 pm eastern on Monday. No change.) WEDNESDAY 1330 UTC (==0830 am eastern on Wednesday. Changed.) There's no need to show up to both, but you can if that's convenient. If both are equally convenient for you and you only want to do one, please favor the Wednesday one. One of the topics at the next meetings will be Good Meeting Times and Getting More People to Come. If you want to talk about meeting times, but neither of these times will work, just send me a note or find me online so I know. Let's keep trying till we get this right! cheers, -- Nick

1 0