- tor-dev - lists.torproject.org

Tor Metrics project - past, present, future
by Karsten Loesing 19 Dec '13

19 Dec '13

Dear devs, you probably know the Tor Metrics project with its most visible part, the metrics website [0]. The metrics website and the less visible parts around it have grown organically since 2009. I'm maintaining the code behind most of these parts, which is why I'm periodically trying to bring order into the chaos. Sometimes this means removing parts which are mostly unused or require too many resources, and removing working parts typically leads to sad faces. But sometimes I'm also adding new parts which has the potential of turning sad faces into happy faces again. When I talked to Tom on Thursday, he suggested that I should take a step back and explain here what metrics is for and what it's not for, and list the things that are great, the things that can go, and the things we might like in the future. Great idea, let me do that! Before I start, may I call your attention to exhibit A: https://metrics.torproject.org/tools.html This is a diagram showing all the different metrics parts and how they are connected. Really, please take a quick look at it, because I'm going to refer to the diagram below. So, *what* is metrics for? Perhaps it's easier to answer this question by saying *who* metrics is for: 1. For the researchers! The metrics project archives Tor network data, that is safe to be archived, and provides it to anyone on a public website [1, 2]. This is a valuable resource for researchers who are trying to back their research with actual data about the deployed Tor network. (Green parts in the diagram.) 2. For the community! The metrics website contains all kinds of statistics showing how the Tor network grew over time [3, 4], including details about relays, bridges, and clients as far as user privacy is not at risk. Focus is on the Tor network as a whole, from 2007 until today. (Purple parts in the diagram.) 3. For the operators! Onionoo [5] processes Tor network data and provides it in a convenient format for web applications [6, 7, 8] designed for relay or bridge operators, people helping to debug the Tor network, and more generally Tor users. Focus is on single relays or bridges right now or in the very recent past. (Orange part in the diagram.) And what is metrics *not* for? If you take another look at exhibit A, I'll explain three parts which I think are either not used anymore or require too many resources. If people think we really need to keep any of these parts, I'd like to know: 1. We added statistics on uni-/bidirectional connection usage a few years back. I think these statistics were useful for researchers at the time, but I haven't heard from anybody using them recently. We should remove them from little-t-tor and from metrics [9]. (Part of performance.html in the diagram.) 2. We introduced the notion of a fast exit for sponsor J, and we added graphs showing how many of these fast exits or almost fast exits are running in the Tor network. The sponsor contract ended, and the notion is too specific to be useful for anything else. We should remove the graphs. (fast-exits.html in the diagram.) 3. The relay-search service can find any relay that was running since 2007. This is certainly useful for debugging the Tor network, but it requires us to keep a full database of relays which doesn't scale anymore [10]. (relay-search.html in the diagram.) Last, but certainly not least, I have a couple of things in mind that we might like in the *future*. It would be good to know which of these things sound interesting to others, so I can direct my time on the most popular features first: 1. Start archiving microdescriptors and microdescriptor consensuses (#2785). While both descriptor types can be derived from server descriptors and unflavored consensuses, archiving the originals would be more convenient for researchers and developers. (Green parts in the diagram.) 2. Resume to add country codes and maybe AS numbers to sanitized bridge descriptors. We stopped resolving bridge IP addresses to country codes, because it's difficult to do that in a reproducible way with monthly changing GeoIP databases. But having country and maybe AS information would for sure be useful. (Green parts in the diagram.) 3. Keep contact lines in sanitized bridge descriptors (as discussed in #9854). Having contact lines of bridge operators would allow more people to contact them in case of problems, and it would allow bridge operators to find their bridge more easily in services like Atlas [6] or Globe [8]. (Green parts in the diagram.) 4. Make graph on the number of running obfsbridges (#9187). Bridges report which pluggable transports they support in their extra-info descriptors, and it's not trivial to extract this information and count how many bridges per transport are running at a given time. But it's also not terribly difficult. (Purple parts in the diagram.) 5. Re-enable relays-by-country graph on the metrics website (#8127). This is difficult for similar reasons as 2. (Purple parts in the diagram.) 6. Make bridges-by-country graph for the metrics website. Similar to 5, depends on 2. (Purple parts in the diagram.) 7. Include votes in Onionoo (#9778). Votes contain some interesting information about relays, like missing or extra relay flags, or measured bandwidth. The main problem would be potential performance issues. (Orange part in the diagram.) 8. Provide per-bridge usage statistics in Onionoo (#10331). We could tell bridge operators some estimates on the number of clients connecting to their bridge by country, transport, or IP version. Maybe this encourages bridge operators to run more bridges, in particular if they can show their social circle how their bridge helps actual people in the world. (Orange part in the diagram.) 9. Provide relay comparison metrics in Onionoo. We could define some simple metrics on the usefulness of a relay, like provided bandwidth or uptime, in comparison to other relays. A possible statement from these metrics could be: "your relay provides more bandwidth than 95% of relays in the network." Similar to 8. If Atlas [6] or Globe [8] or a yet-to-be-written Facebook application or a also-yet-to-be-written Twitter integration into Tor Weather (#10372) tell the world how successful someone's running Tor relays, maybe that encourages others to run relays, too. We could even invent a points system for running relays, with additional points for running exits, if that makes the Tor network better. Probably needs input from a community coordinator person. (Orange part in the diagram.) Whee, that was a comprehensive discourse on the past, present, and future of the metrics project. Thanks for reading! Feedback much appreciated! All the best, Karsten [0] https://metrics.torproject.org/ [1] https://metrics.torproject.org/data.html [2] https://metrics.torproject.org/formats.html [3] https://metrics.torproject.org/network.html [4] https://metrics.torproject.org/users.html [5] https://onionoo.torproject.org/ [6] https://atlas.torproject.org/ [7] https://compass.torproject.org/ [8] https://globe.torproject.org/ [9] https://lists.torproject.org/pipermail/tor-dev/2013-December/005919.html [10] https://lists.torproject.org/pipermail/tor-talk/2013-December/031310.html

2 2

InjectSOCKS: 2nd try
by tor＠herr-der-mails.de 17 Dec '13

17 Dec '13

Hello, I've first sent this e-mail to help(a)rt.torproject.org and the answer was to send a copy of it to the "tor-dev mailing list". So that's what I do: I just wanted to let you know that I've created a small new tool for Windows called InjectSOCKS that can force other Windows software to do TCP connections via SOCKS. This way software not supporting SOCKS can be used together with Tor. You don't need any additional HTTP proxy or other proxies. As an example it works for passive FTP, too. Additionally it handles the DNS requests of that other software in a way that while creating the SOCKS connection, Tor gets the textual address - so the exit node can resolve it (which is the way favored by the Tor developers). This way Tor hidden services work as well. And it works per Windows process, so it doesn't influence the whole operating system. In case you're interested in my software, I've put it on sourceforge to make it open source: http://sourceforge.net/projects/injectsocks The tool is far from being perfect yet, but I think some of the ideas are interesting. By the way, I've also created DNS2SOCKS, which is already listed on Tor's Wiki: http://sourceforge.net/projects/dns2socks It seems like several people like it, so I hope that some people will also like InjectSOCKS. Regards, ghostmaker

3 6

Gitian Build Failure (pt branch only?)
by Kevin P Dyer 17 Dec '13

17 Dec '13

Hi Mike/dcf1, Building the pt branch [1] of tor-browser-bundle on Ubuntu 13.04, 64-bit resulted in the following error: <<< ./bin/gbuild:21:in `system!': failed to run on-target setarch i386 bash -x < var/build-script > var/build.log 2>&1 (RuntimeError) from ./bin/gbuild:121:in `build_one_configuration' from ./bin/gbuild:223 from ./bin/gbuild:218:in `each' from ./bin/gbuild:218 from ./bin/gbuild:216:in `each' from ./bin/gbuild:216 make: *** [build] Error 1 >>> dcf1 suggested that I downgrade to the 3.2.0 kernel. This wasn't (obviously) supported in 13.04, so I decided to downgrade my system to 12.04. On a clean 12.04, 64-bit, install I got the same issue. Then, the following resolved the problem for me: <<< $ sudo apt-get remove linux-image-3.8.0-29-generic linux-image-generic-lts-quantal linux-image-generic-lts-raring $ sudo apt-get --reinstall install linux-image-generic >>> I did not encounter this problem when building [2] last week. When the failure occurs, var/build.log contains the "invalid instruction suffix" errors as reported in [3]. -Kevin [1] https://gitweb.torproject.org/user/dcf/tor-browser-bundle.git [2] https://gitweb.torproject.org/builders/tor-browser-bundle.git [3] https://github.com/cloudmatrix/myppy/issues/7

4 7

fteproxy v0.2.2
by Kevin P Dyer 16 Dec '13

16 Dec '13

I've response to the good feedback [1] from asn, nickm, and yawning, I've tagged v0.2.2 [2] of fteproxy. This release includes the removal of gmpy as a dependency, additional documentation to explain the significance of language theoretical algorithms, and bounds checking of the input/output of our (un)ranking algorithms. I look forward to feedback on v0.2.2! -Kevin [1] https://trac.torproject.org/projects/tor/ticket/10362#comment:2 [2] https://github.com/kpdyer/fteproxy/tree/384a4b0ba5a5

1 0

Slight obfsproxy API change (#10342)
by George Kadianakis 13 Dec '13

13 Dec '13

Hello Philipp and David, I'm sending you an email because you are the people who are currently developing obfsproxy modules. We are thinking of slightly changing the API of obfsproxy, so that it becomes more friendly to transport authors (or at least we hope so). Specifically, we are thinking of having the circuit be an element of the transport class, instead of being passed as an argument in the receivedDownstream and receivedUpstream methods. I think David and Leif were also thinking of this change. Furthermore, we are also thinking of renaming the 'handshake' method to 'circuitConnected'. (More details can be found in #10342.) Do you agree with the changes? If yes, I will merge Ximin's patch for #10342, and you will have to update your transport codebase appropriately. Please let me know. Thanks!

4 4

Tor project automation work
by Nicolas Vigier 12 Dec '13

12 Dec '13

Hello, If you don't know me yet, I am the new person working on automation : https://www.torproject.org/about/jobs-lead-automation.html.en To get started working on tor automation, I am currently looking at all components that are part of tor project, to find what is the current status regarding build, packaging, testing, and what should be done. I'm going to write a summary of this, with some proposals and a general plan for the work to be done during the coming months, and it will be available at the following URLs : https://people.torproject.org/~boklm/automation/tor-automation-review.html https://people.torproject.org/~boklm/automation/tor-automation-review.pdf (you can find an early version there) In order to help me doing that, I'm very interested to receive from developers of any tor components : - a description or ticket number of bugs that you wish could have been detected earlier with automated tests - any wish or specific needs that you may have - anything else that you think might be useful for me to know If you want to discuss this on IRC, I'm 'boklm' on #tor-dev. Thanks! Nicolas

3 7

Proposal 225: Strawman proposal: commit-and-reveal shared rng
by Nick Mathewson 11 Dec '13

11 Dec '13

This used to be an appendix in my drafts for proposal 224, but I split it into its own document since it's orthogonal to the rest of 224. See section 1 for appropriate provisos and disclaimers. I'm writing this as a proposal mainly so other researchers can write better proposals for this kind of thing. Filename: 225-strawman-shared-rand.txt Title: Strawman proposal: commit-and-reveal shared rng Author: Nick Mathewson Created: 2013-11-29 Status: Draft 1. Introduction This is a strawman proposal: I don't think we should actually build it. It's just a simple writeup of the more trivial commit-then-reveal protocol for generating a shared random value. It's insecure to the extent that an adversary who controls b of the authorities gets to choose among 2^b outcomes for the result of the protocol. See proposal 224, section HASHRING for some motivation of why we want one of these in Tor. Let's do better! [TODO: Are we really stuck with Tor's nasty metaformat here?] 2. The protocol Here's a protocol for producing a shared random value. It should run less frequently than the directory consensus algorithm. It runs in these phases. 1. COMMITMENT 2. REVEAL 3. COMPUTE SHARED RANDOM It should be implemented by software other than Tor, which should be okay for authorities. Note: This is not a great protocol. It has a number of failure modes. Better protocols seem hard to implement, though, and it ought to be possible to drop in a replacement here, if we do it right. At the start of phase 1, each participating authority publishes a statement of the form: shared-random 1 shared-random-type commit signing-key-certification (certification here; see proposal 220) commitment-key-certification (certification here; see proposal 220) published YYYY-MM-DD HH:MM:SS period-start YYYY-MM-DD HH:MM:SS attempt INT commitment sha512 C signature (made with commitment key; see proposal 220) The signing key is the one used for consensus votes, signed by the directory authority identity key. The commitment key is used for this protocol only. The signature is made with the commitment key. The period-start value is the start of the period for which the shared random value should be in use. The attempt value starts at 1, and increments by 1 for each time that the protocol fails. The other fields should be self-explanatory. The commitment value C is a base64-encoded SHA-512 hash of a 256-bit random value R. During the rest of phase 1, every authority collects the commitments from other authorities, and publishes them to other authorities, as they do today with directory votes. At the start of phase 2, each participating authority publishes: shared-random 1 shared-random-type reveal signing-key-certification (certification here; see proposal 220) commitment-key-certification (certification here; see proposal 220) received-commitment ID sig received-commitment ID sig published YYYY-MM-DD HH:MM:SS period-start YYYY-MM-DD HH:MM:SS attempt INT commitment sha512 C reveal R signature (made with commitment key; see proposal 220) The R value is the one used to generate C. The received-commitment lines are the signatures on the documents from other authorities in phase 1. All other fields are as in the commitments. During the rest of phase 2, every authority collects the reveals from other authorities, as above with commitments. At the start of phase 3, each participating authority either has a reveal from every authority that it received a commitment from, or it does not. Each participating authority then says shared-random 1 shared-random-type finish signing-key-certification (certification here; see proposal 220) commitment-key-certification (certification here; see proposal 220) received-commitment ID sig R received-commitment ID sig R ... published YYYY-MM-DD HH:MM:SS period-start YYYY-MM-DD HH:MM:SS attempt INT consensus C signature (made with commitment key; see proposal 220) Where C = SHA256(ID | R | ID | R | ID | R | ...) where the ID values appear in ascending order and the R values appear after their corresponding ID values. See [SHAREDRANDOM-REFS] for more discussion here. (TODO: should this be its own spec? If so, does it have to use our regular metaformat or can it use something less sucky?)

3 4

Debug-assert in 'test util/logging/sigsafe_err'
by Gisle Vanem 11 Dec '13

11 Dec '13

Running "test --verbose util/logging/sigsafe_err" gives me an debug-assert in write(STDERR_FILENO,...). Assert that failed is: "(_osfile(h) & FOPEN)" Here's the call-stack in WinDbg: MSVCR100D!_write+0x126 test!tor_log_err_sigsafe_write(char * s = 0x00707290 ".====================== ====================================== T=")+0x66 [g:\mingw32\src\inet\tor-project\src\common\log.c @ 463] test!tor_log_err_sigsafe(char * m = 0x006bae50 "Minimal..")+0xdf [g:\mingw32\src\inet\tor-project\src\common\log.c @ 491] test!test_sigsafe_err(void * arg = 0x00000000)+0xb0 [g:\mingw32\src\inet\tor-project\src\test\test_logging.c @ 98] ... It makes sense that "(_osfile(h) & FOPEN)" is FALSE, since in test_logging.c I see: close(STDERR_FILENO); <<< !!! what good is this? log_err(LD_BUG, "Say, this isn't too cool."); tor_log_err_sigsafe("Minimal.\n", NULL); If you close stderr, shouldn't handle 2 point to something else? I cannot see it does. All this is on Win-XP SP3, MSVC v16 (debug). --gv

1 0

fteproxy v0.2.1
by Kevin P Dyer 11 Dec '13

11 Dec '13

Hi dcf1/asn, Yesterday I tagged release v0.2.1 of fteproxy. In this release I focused on breaking away from heavyweight dependencies: OpenFST and boost. Cross-platform PTTBB binaries are available on the fteproxy website: https://fteproxy.org/download. asn - Please have a look at the code and confirm that you're happy. I had to make a few changes to the Python modules written in C++, as I can no longer rely upon boost::python. I've also added additional information to the function-level descriptions, which should help explain the significance of the language theoretical algorithms. It is also worth noting that the pseudocode is available in Appendix A of the full version of our paper [0]. dcf1 - I finally have the gitian build process producing vanilla binaries. However, I don't see anything specific to pluggable transports in [1] or [2]. Is there a fork of this code that has logic for building the pluggable transports? Or is that still an open task? -Kevin [0] http://eprint.iacr.org/2012/494 [1] https://gitweb.torproject.org/builders/tor-browser-bundle.git [2] https://gitweb.torproject.org/builders/gitian-builder.git

2 1

txtorcon 0.8.2
by meejah 10 Dec '13

10 Dec '13

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 First, sorry this announcement is a little late, this was in my drafts :( There was a bug in txtorcon 0.8.1 where the TCP4HiddenServiceEndpoint wasn't ensuring that only 127.0.0.1 was being listened on. GitHub's master and the tag v0.8.2 fix this issue, and (thanks to Lunar) the Debian packages are also updated. PyPI packages are also updated (via Twine this time, which is really nice). If you don't use TCP4HiddenServiceEndpoint, there is no need to upgrade. If you *do* use it, you should upgrade. Thanks, meejah -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.10 (GNU/Linux) iQEcBAEBAgAGBQJSp03QAAoJEJ0UOBRRgEVQWPcIAMV0aupijakh3GUBqF2s0zEp TxPr658w0KDZuWH9cON9h/Apkss0BQnW2LuoPKP4spAXh8qOlUd6sBzj+fat2RwE 09qkJPKuyt6kGZyxD5jRL3pRNEBU8qs0C6AIcdysfpfWRGOx53ZOjJgBoanzzmRA JmdFQBtAVBCW0IzImFpf8rCIQpx/8SXtA898RqQODANdtxucwqRVqC/Y5uX+7a29 xdtgFpr7NcqW2KC0WMInubQ6lo6KfPgCfnrSU2FBcafwdXlh97cmdovzUX0/raSM bHJvgruPyF1hJBDuTHXzKbnb41Xb0Vk6MMALw3ad3t505jFDhZT1g8kBw0mo1Qk= =PM/Z -----END PGP SIGNATURE-----

1 0