- tor-dev - lists.torproject.org

[measurement team] analytics server proposal
by thomas lörtsch 09 Oct '15

09 Oct '15

analytics server proposal ========================= tl;dr ----- measurement team is setting up a server for you to play with raw metrics data. please check below if it meets your requirements and if you have any suggestions for the setup. Motivation + Setup ------------------ An idea came up during the Berlin dev meeting: it would be good to have all measurement data on one server, loaded into a database, with a suite of analytics software readily set up, available to everybody who needs to do aggregation, analytics, research etc on the data. Right now everybody who wants to analyze Tor metrics data first has to downlod raw data from the metrics website, load it into a local database, massage it, setup a query environment etc. That can be cumbersome and use siginificant computing resources, making your notebook unusable for hours or eating up hundreds of gigabytes of disk space if you don’t optimize properly. That makes on the fly quick and dirty prototyping and analysis quite unfeasable and prohibitively tedious. We plan to change this by setting up an analytics server free for everybody to use: * containing as much raw metrics data as possible * in a HBase database on Hadoop/HDFS * with analytic tools like Spark and Drill on top * supporting diverse languages - SQL, R, Python, Java, Scala, Clojure - * with gateways to desktop-ish tools like Tableau and MongoDB Please see the detailed description below, tell us if you have any suggestions concerning the setup and request a login (to be available soon)! Data ---- The data will be available as raw as possible, like it is served by CollecTor, preloaded into a database and maybe also in some pre-aggregated sets that concentrate on “popular” aspects of the data. The complete dataset is about 50 GB compressed, or 5 TB uncompressed. For a start we may need to concentrate on the last 2 or 3 years. Analytics ———— The amount of data practically rules out the usual RDBMS suspects (PostgreSQL etc). We also want to have easy access to the analytical tools that are available in the big data ecosystem. Hadoop [0] is a popular candidate for Big Data processing, but has a reputation of being slightly outdated, not so well engineered [1] and is not available in Debian stable (which likely won’t change [2]). It seems like choosing Hadoop means risking to run into nasty little problems down the road. Most solutions outlined below are or can be based on Hadoop File System (HDFS) but that is available independently of the Hadoop MapReduce system. Disco [3] is another MapReduce engine. Seems solid, heavyly uses Python, but relies on it’s own file system [4] which would make it harder or impossible to use with other tools like Spark, that require HDFS (or compatible). Would need more research/googling to be really sure but right now it doesn't seems worth the effort. Spark [5] is faster than Hadoop MapReduce when working in memory and can work equally well from disk. Storage is based on Hadoop and "works with any Hadoop compatible data source including HDFS, HBase, Cassandra, etc.” [6]. It can work with Amazons Elastic MapReduce EMR instances (which in turn allows running Spark on EMR). EMR pricing is not prohibitive [7]. Ooni uses this setup [8]. Spark supports R, Scala, Java, Python and Clojure. It provides an interactive shell for Python and Scala and plugins for SQL-like queries, MLib (machine learning lib) and some graph processing. Its performance but also the SQL plugin and the wide range of supported programming languages make Spark quite attractive. Hive [9] is another analytics engine on top of Hadoop that has similar features to Spark [10] (its feature set was lagging behind Spark for some time but has catched up recently). Since Ooni uses Spark and the two are not very different with respect to feature set we favor Spark (and hope for synergies). MongoDB [11], which is available in Debian stable and supports Map Reduce [12], may not be performant enough and is in some ways rather shoddy. Redis [13] has a good reputation and while optimized for in-memory operations it can work from disk too. It is available as Debian stable [14]. Map reduce operations are available as add ons [15] but not in Debian stable. Like MongoDB there is not much interesting analytics software available besides MapReduce. Drill [16] is a nice analytic tool: a 'schema-free SQL query engine for Big Data’ which accepts a variety of data sources including HDFS, MongoDB, Amazon S3, with impressive performance and a wide range of possible applications. Both Spark and Drill support Tableau, a nice Desktop data visualization tool that allows to work with data in a rather intuitive way. In summary it seems like Spark and Drill are the aggregation and analytic softwares that we want to have. Together they support a nice array of languages (SQL, R, Java, Python, Scala, Clojure) and desktop-ish tools (MongoDB, Tableau) which should make them readily useable for a lot of people. Storage ------- Amazons EMR is an option but for now we’ll install our own box. We could go with CSV files in the Hadoop Distributed File System HDFS to keep complexity down but given the amount of data a proper DBMS seems justified. Two databases are supported by both Spark and Drill: Apache Cassandra [17][18] and Apache HBase [19][20][21]. Both are column-oriented key-value-store. HBase follows the same design principles as Cassandra but comes out of the Hadoop community and is better integrated wiht other offerings. Also while Cassandra is more optimized for easy scaling and write performance HBase has the emphasis on fast read [22]. It also seems like getting Drill to work with Cassandra is not as easy as advertized [23]. So HBase seems like the best fit. Hopefully Drill and Spark will work from the same data in HBase and not want to import it into their own store (thereby duplicating it) but we’re not sure about that right now. If they do import the data into their own stores anyway then HBase might be overkill and CSV files on HDFS sufficient. Or Parquet [24][25]? We plan to figure that out during the next days. Any help and remarks appreciated! Redis and MongoDB, the 2 systems that are available as Debian stable packages, are not well enough supported by analytic softwares to be useful. Conclusion ---------- Measurement team will set up a hosted server with 32 GB RAM and 5 TB storage (partly as SSD), install Apache HBase as database and Apache Spark and Apache Drill as analytics softwares and import a lot of raw measurement data into HBase. We then give login to the machine on request and hope that you have fun with map reducing, data sciencing, visualizing... This is a test: we plan to keep the server running for at least the next 3 months or so. If it doesn’t prove useful we will probably shut it down afterwards, but we hope otherwise. --- [0] http://hadoop.apache.org/ [1] https://news.ycombinator.com/item?id=4298284 - see the 4. comment thread for a nice rundown on the cons of Hadoop. [2] https://wiki.debian.org/Hadoop [3] http://discoproject.org/ [4] http://disco.readthedocs.org/en/develop/howto/ddfs.html?highlight=hdfs [5] http://spark.apache.org/ [6] http://www.infoq.com/articles/apache-spark-introduction [7] https://aws.amazon.com/elasticmapreduce/pricing/ [8] https://github.com/TheTorProject/ooni-pipeline [9] http://hive.apache.org/ [10] http://www.zdnet.com/article/sql-and-hadoop-its-complicated/ [11] https://www.mongodb.org/ [12] https://docs.mongodb.org/manual/core/map-reduce/ [13] https://redis.io [14] https://packages.debian.org/search?suite=jessie&arch=any&searchon=names&key… [15] http://heynemann.github.io/r3/ - maybe also others although Google didn’t really help [16] https://drill.apache.org/, https://drill.apache.org/faq/ [17] http://cassandra.apache.org/ [18] http://www.planetcassandra.org/getting-started-with-apache-spark-and-cassan… [19] http://hbase.apache.org/ [20] http://www.vidyasource.com/blog/Programming/Scala/Java/Data/Hadoop/Analytic… [21] http://www.slideshare.net/cloudera/5-h-base-schemahbasecon2012 [22] http://www.infoworld.com/article/2848722/nosql/mongodb-cassandra-hbase-thre… [23] http://stackoverflow.com/questions/31017755 [24] https://parquet.apache.org/documentation/latest/ [25] http://arnon.me/2015/08/spark-parquet-s3/

3 6

Concerns regarding a multilingual version of TBB,
by Yet Another Tor User 09 Oct '15

09 Oct '15

I saw an email on the Wiki mailing list concerning a multilingual version of Tor. If that was made possible your language tag be used to track you. This could break your anonymity. Your thoughts?

3 2

Summary of meek's costs, September 2015
by David Fifield 09 Oct '15

09 Oct '15

Here's the summary of meek's CDN fees for September 2015. App Engine + Amazon + Azure = total by month February 2014 $0.09 + -- + -- = $0.09 March 2014 $0.00 + -- + -- = $0.00 April 2014 $0.73 + -- + -- = $0.73 May 2014 $0.69 + -- + -- = $0.69 June 2014 $0.65 + -- + -- = $0.65 July 2014 $0.56 + $0.00 + -- = $0.56 August 2014 $1.56 + $3.10 + -- = $4.66 September 2014 $4.02 + $4.59 + $0.00 = $8.61 October 2014 $40.85 + $130.29 + $0.00 = $171.14 November 2014 $224.67 + $362.60 + $0.00 = $587.27 December 2014 $326.81 + $417.31 + $0.00 = $744.12 January 2015 $464.37 + $669.02 + $0.00 = $1133.39 February 2015 $650.53 + $604.83 + $0.00 = $1255.36 March 2015 $690.29 + $815.68 + $0.00 = $1505.97 April 2015 $886.43 + $785.37 + $0.00 = $1671.80 May 2015 $871.64 + $896.39 + $0.00 = $1768.03 June 2015 $601.83 + $820.00 + $0.00 = $1421.83 July 2015 $732.01 + $837.08 + $0.00 = $1569.09 August 2015 $656.76 + $819.59 + $154.89 = $1631.24 September 2015 $617.08 + $710.75 + $490.58 = $1818.41 -- total by CDN $6771.57 + $7876.60 + $645.47 = $15293.64 grand total https://metrics.torproject.org/userstats-bridge-transport.html?graph=userst… Last month was pretty stable at about 5,000 simultaneous users all month. This month is the first time we have actual usage numbers from Azure. They told us what the cost would have been, but did not actually charge us. This month (September) is the last month on the free grant we have for Azure. After that, we will probably have to throttle Azure just like the other backends. If you want to help make domain fronting more efficient and future-proof, a good place to direct your attention is to this ticket, which aims to start using domain fronting as a cheap, low-volume interface to BridgeDB, instead of as a bulk data transfer channel: https://bugs.torproject.org/16650 If you want to set up your own bridge and CDN instance without rate limiting, I can help you do it. Here are some docs to look at: https://trac.torproject.org/projects/tor/wiki/doc/meek#Howtorunameek-server… https://trac.torproject.org/projects/tor/wiki/doc/meek#GoogleAppEngine https://trac.torproject.org/projects/tor/wiki/doc/meek#AmazonCloudFront https://trac.torproject.org/projects/tor/wiki/doc/meek#MicrosoftAzure == App Engine a.k.a. meek-google == Here is how the Google costs broke down: 2871 GB $344.53 5451 instance hours $272.55 Compared to the previous month: 2920 GB $350.38 6128 instance hours $306.38 https://globe.torproject.org/#/bridge/88F745840F47CE0C6A4FE61D827950B06F9E4… == Amazon a.k.a. meek-amazon == Asia Pacific (Singapore) 102M requests $122.83 909 GB $120.81 Asia Pacific (Sydney) 2M requests $2.53 6 GB $0.88 Asia Pacific (Tokyo) 9M requests $11.60 57 GB $7.54 EU (Ireland) 188M requests $226.56 1131 GB $89.42 South America (Sao Paulo) 6M requests $14.37 40 GB $9.45 US East (Northern Virginia) 70M requests $70.04 441 GB $34.73 -- total 379M requests $447.93 2586 GB $262.83 https://globe.torproject.org/#/bridge/3FD131B74D9A96190B1EE5D31E91757FADA1A… == Azure a.k.a. meek-azure == This is the first month where we have actual (not estimated) usage numbers, as a result of the new kind of account we're using. Unfortunately (also as a result of the account transition), meek-azure was broken until August 14, so these number really only represent half a month. Zone 1 3726 GB $324.20 Zone 2 1206 GB $166.39 -- total 4932 GB $490.58 https://globe.torproject.org/#/bridge/AA033EEB61601B2B7312D89B62AAA23DC3ED8… Earlier reports in this series: https://lists.torproject.org/pipermail/tor-dev/2014-August/007429.html https://lists.torproject.org/pipermail/tor-dev/2014-October/007576.html https://lists.torproject.org/pipermail/tor-dev/2014-November/007716.html https://lists.torproject.org/pipermail/tor-dev/2014-December/007916.html https://lists.torproject.org/pipermail/tor-dev/2015-January/008082.html https://lists.torproject.org/pipermail/tor-dev/2015-February/008235.html https://lists.torproject.org/pipermail/tor-dev/2015-March/008427.html https://lists.torproject.org/pipermail/tor-dev/2015-April/008596.html https://lists.torproject.org/pipermail/tor-dev/2015-May/008767.html https://lists.torproject.org/pipermail/tor-dev/2015-June/008932.html https://lists.torproject.org/pipermail/tor-dev/2015-July/009030.html https://lists.torproject.org/pipermail/tor-dev/2015-August/009213.html https://lists.torproject.org/pipermail/tor-dev/2015-September/009533.html

1 0

UniformDH question
by Jeff Burdges 08 Oct '15

08 Oct '15

What is the advantage of using X or p-X in UniformDH in obfsproxy? https://gitweb.torproject.org/pluggable-transports/obfsproxy.git/tree/d oc/obfs3/obfs3-protocol-spec.txt#n65 Isn't just X itself dense pretty quickly anyways? Jeff

2 1

Re: [tor-dev] Trac priorities and severities
by Hugo Maxwell Connery 08 Oct '15

08 Oct '15

Hi, Yes, renaming to a more meaningful thing is great. Also, if only some group can grade entries, then it is wise to have an 'uncategorised' category which appears in a grading list towards, but not at the bottom. Field experience in these systems tells me that people get annoyed at being auto-classified at the bottom. I advise putting 'uncategorised' at 5 of 7 or something similar. e.g Critical, High Import, Import, Medium, Uncategorised, Small, Trivial. This also helps to raise the visibility of Uncategorised items; i.e Categorise them :) Regards, Hugo

1 0

Re: [tor-dev] Crux: Privacy-preserving statistics for Tor
by str4d 07 Oct '15

07 Oct '15

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA512 Vasilios Mavroudis wrote: > Hello, > > I would like to introduce our project "Crux", which enables the > computation of privacy preserving statistics on sensitive data. The > project was developed at University College London (UCL) by me, in > close collaboration with George Danezis. This is fantastic work. Haven't read all of your thesis yet, but it looks similar to the stats collection system that we ran for a while inside I2P, but with the privacy protections that I wanted it to have. Well done! > > "Crux" was designed with Tor hidden services in mind (in accordance > with this > <https://research.torproject.org/techreports/hidden-service-stats-2015 - -04-28.pdf> > > Tor > Tech Report), but it can be applied on various kinds of data (given > an appropriate parser). Excellent. I want to use this in I2P :) > > It supports three statistics (i.e. mean, median, variance) and its > threat model is compatible Tor's (see technical document below). I will look into this, but I doubt there will be any insurmountable incompatibilities with I2P's threat model. > Currently, we don't have a parser specific for the data collected > by the Tor relays, but we plan to keep adding functionality. > > The source code can be found here: > https://github.com/mavroudisv/Crux The theoretical background, > threat model, security argument and technical details can be found > here: link <http://mavroudisv.eu/files/thesis.pdf> > > > Ideas, comments, feedback much appreciated! Firstly: needs more docs. I shouldn't have to read your entire thesis in order to figure out how to run it :P The README is a good start, but the command parameters need clarification/definitions, and it is not at all obvious how the relays work. An XLS file is also mentioned without explanation. Secondly, you could probably simplify configuration by managing hidden service connections yourself. I believe the Stem library can do this? I2P has a python library that can be used to listen on and connect to I2P Destinations, with a socket-like interface. It's not clear to me from your thesis whether you plan to merge the relay component into the Tor relay code (and have the Tor relays talk directly to the query processors), or have your relays running separately and interfacing with the Tor relays to fetch stats. This will potentially alter how it would be used with other networks. In I2P's case I'd imagine it would be easiest (in terms of usability by those setting up relays) to write an I2P plugin in Java that performs the relay role, but I wouldn't want to be duplicating a lot of logic that subsequently needs to be kept in-sync with upstream :) str4d > > Vasilis > > > > _______________________________________________ tor-dev mailing > list tor-dev(a)lists.torproject.org > https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev > -----BEGIN PGP SIGNATURE----- iQIcBAEBCgAGBQJWFP9HAAoJEBO17ljAn7PgAq0P/jrjLnPxg2+rfOX1H46xClpG wfeeNuDltlrvTtNwtDO+lqSzpz+LY/PkUbHow2B7wprRgGvav2kCtGh3xga4BVEX SoYiNcWhRKFmrw6jNjAGPh060hcC7WzBbAjxQ28kotl7NiXO4ZbeMdQ5vqDVE5+K 2f0lSFnBYmj4SYxaVWIxGir3FRn+VdACWjQPL1JV4LDbAPXmNlg17nBwvj5g9lhS n9InsXUqoyhJ5zbVqwHoNl8N2IMui+i2ppF3ZiuHQCHw88+6qJxVW5u11zP2TccJ ycFNE5jcxxNtK9+1jwdfpen5yAnAlaaeKHkHZ+oNN1nb156QbLhcIwqt8IeRFcZv SzaCQk6NIAZFbOVPNzTJ6akXjajd7AbkdVnGBeziD8z8yD1Qa+OH0vBoT/jTP5Ec OdU9f0a3Vc3zftoSaRsCHaQX7+1OEKjDkyk6Gs6lt2PUWVvnNu2+ztXBttI2v8Ee UjEbAC7OR8NpyBAn0p+ja/353zCZU0TXzjccEFb7YtjCCy0hACQGuxwjEECcpbQH MKRZa0JmbSVbDsZoRuMNEwPzg4/sXTVaS8mGmOurotXR1iPgs80DxiAym0kLOe0d xllTDtf561gekxed/+Zm5hjH5FAR7uMI4cpywRUMqyCzJGSKZKr/2uxjnv832j4H ecujGrMEByFdgfG3co5P =etz3 -----END PGP SIGNATURE-----

2 1

#9623 [Tor Browser]: Referers being sent from hidden service websites
by Tim Wilson-Brown - teor 07 Oct '15

07 Oct '15

Hi All, Currently there’s an information leak in Tor Browser: it sends referrer headers containing .onion site addresses when the user clicks on a link on the .onion site. There’s a fix in the works, but we were wondering: Does anyone’s hidden service depend on the referrer header? The currently favoured fix is to stop sending referrers cross-origin (between different .onion sites, and between .onion sites and sites on the internet). But this may break sites that are set up with multiple .onion addresses and use referrers to check that requests are coming from the parent site. (People sometimes set up different .onion sites to serve different types of content, such as images.) In general, I would discourage people from using referrers in this way, because they aren’t secure and can be faked. But does anyone have a compelling use case for cross-origin referrers, or is using them at the moment? We could include a preference if removing them would break too many sites. Tim Tim Wilson-Brown (teor) teor2345 at gmail dot com PGP 968F094B teor at blah dot im OTR CAD08081 9755866D 89E2A06F E3558B7F B5A9D14F

4 3

[Measurement Team] Next IRC meeting happens on Thursday, Oct 8, 14:00 UTC in #tor-dev
by Karsten Loesing 07 Oct '15

07 Oct '15

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hello list, This is an announcement/reminder that there will be an IRC meeting of the Measurement Team on Thursday, October 8, 2015, 14:00 UTC in #tor-dev https://www.timeanddate.com/worldclock/fixedtime.html?iso=20151008T14 This will be the first IRC meeting since the Berlin dev meeting, and we're going to talk about the outcomes of that, among other things. For example, we talked about meeting every two weeks on IRC, instead of every week, and moving more communication to email. Be sure to be around tomorrow if you want to discuss things like that. Talk to you tomorrow! All the best, Karsten -----BEGIN PGP SIGNATURE----- Version: GnuPG v1 Comment: GPGTools - http://gpgtools.org iQEcBAEBAgAGBQJWFKklAAoJEJD5dJfVqbCrgV4H/3DoPjbDOk+mMmbCnc5ui0rM sihlWNx7xgKSAKJ5fK6BfCBnSeBHdWC3AzzpZGre21JpZp7FgQqkVOq65z70tANG WaGDHuHmNvtmu45nIDgSDZrgt3teHG2T5EALTcBy7bo3JdBKIVICkheBhO99tvCE Do4/MzHUF4kQ6EMD9bo2n7ASnIi7Tr/RPwA/mNDvHYGlGztJOkH9um+HK99+PLDf hx0c7G3jL2/jiItH+NnBFAoXAahC7F6x3Pga5EhhQ5gy1C4mT//ZWECMAEo5eMDg m9t2y+30ecpJ0x0HIXvx6v5XH0rjOHWu6ASaMim63goQJ1NYy8GAL5dvQZjCR+w= =JnLB -----END PGP SIGNATURE-----

1 0

Networks Blocking Tor's SSL Connections
by Tim Wilson-Brown - teor 07 Oct '15

07 Oct '15

Hi All, This morning I observed a “free wifi” network blocking tor’s SSL connections. While other SSL connections from my machine went through, I observed multiple network traces of tor completing a TCP 3-way handshake, and then getting no reply to the first SSL packet it sent. I think they may have been blocking unknown or untested certificates, but I can’t be sure. Still, I was able to use meek(-google) to access tor. Has anyone else seen this kind of blocking behaviour? (Is this the right list?) Tim Tim Wilson-Brown (teor) teor2345 at gmail dot com PGP 968F094B teor at blah dot im OTR CAD08081 9755866D 89E2A06F E3558B7F B5A9D14F

2 1

Crux: Privacy-preserving statistics for Tor
by Vasilios Mavroudis 07 Oct '15

07 Oct '15

Hello, I would like to introduce our project "Crux", which enables the computation of privacy preserving statistics on sensitive data. The project was developed at University College London (UCL) by me, in close collaboration with George Danezis. "Crux" was designed with Tor hidden services in mind (in accordance with this <https://research.torproject.org/techreports/hidden-service-stats-2015-04-28…> Tor Tech Report), but it can be applied on various kinds of data (given an appropriate parser). It supports three statistics (i.e. mean, median, variance) and its threat model is compatible Tor's (see technical document below). Currently, we don't have a parser specific for the data collected by the Tor relays, but we plan to keep adding functionality. The source code can be found here: https://github.com/mavroudisv/Crux The theoretical background, threat model, security argument and technical details can be found here: link <http://mavroudisv.eu/files/thesis.pdf> Ideas, comments, feedback much appreciated! Vasilis -- Vasilios Mavroudis

1 0