- tor-dev - lists.torproject.org

Re: [tor-dev] Pluggable transport weekly meeting
by vmonmoonshine＠gmail.com 06 Sep '13

06 Sep '13

> From: Yawning Angel <yawning(a)schwanenlied.me> > > Assuming not today (2013/09/16) isn't "soon", the times specified works > for me and I will make an effort to attend. > No, I didn't mean today. I just wanted to send it as soon as found out it hasn't been out. Perhapse it was better to wait few hours. But I would like to get some feedback first. Once we agree on a time (which seems Friday works for most people), I'll send a reminder before the meeting to dev list (hopefully to the correct one). Thanks for the effort ;) Vmon

1 0

[GSoC 2013] Status report - Searchable metrics archive
by Kostas Jakeliunas 06 Sep '13

06 Sep '13

Hello, This status update is less extensive/dramatic compared to the last one, but I'm still happy to report to be slowly moving ahead towards a stable searchable archive system. In short, I've been working on what I said I should work on in the last report, more or less: I should now move on with implementing / extending the Onionoo API, in > particular, working on date range queries, and refining/rewriting the "list > status entries" API point (see below). Need to carefully plan some > things, and always keep an updated API document. (Also need to update and > publish a separate, more detailed specification document.) The searchable metrics backend (encompassing a part of the database and the Onionoo-like API) [2] is still happily chugging along online; quite a few folk ran some queries on it, including myself. I'm in the process of expanding my benchmark.py tool to generate realistic-looking parallel/asynchronous traffic for different kinds of relays and API points; from my limited parallelized benchmarking so far, everything looks good - the bottlenecks are still localized to the individual queries. I should try and generate human-readable more-or-less rigorous benchmarking reports and publish them. I've briefly run the backend on an EC2 instance again, to compare benchmark outputs and average query times. Natural database caching seems to be helping quite a bit for the current online database (at ts.mkj.lt), by which I mean, indexes and some query results get cached via natural usage of the backend/database. I've been tinkering on a simple system to pre-warm indexes upon application start (without the need for any PostgreSQL extensions), for more uniformly distributed query times. Overall though, we seem to be doing OK in regards to actual query times. Made updates to the Onionoo API, but haven't pushed them yet (hoped to do that until this report, but finally learning not to delay); expect them soon (and subsequent updates to the Onionoo API doc [3] as well, once that happens.) I'm now trying to incorporate the most recent Karsten's feedback in regards to API points, parameters and some preliminary simplistic caching. Basically, once the date range parameters are working nicely for all three types of documents currently provided by the API, and once the status entry API point is returning valid-after summaries/ranges in a more intuitive document format, the whole thing will hopefully be able to satisfy actual usefulness criteria to a significant extent. Together with caching, it will hopefully be able to be considered as an almost-proper (smallish, as of now) subset of the Onionoo API/backend. Did some experiments with PostgreSQL's pg_trgm module for full-text search (so that search strings matching only the middle of some descriptor field could work); I realize that's not priority now, but I was curious to see if it would work. Nothing conclusive thus far, unfortunately. Specification document explaining more detailed design and applicable use cases for the Onionoo-like API coming along. I should continue finalizing the Onionoo-like API into a working, non-hacky state; keep the codebase clean and maybe do some cleanup refactoring; continue observing database performance; write more documentation; and if all is well, expand the list of fields contained in the three status documents. Besides all this, - update the database to the latest consensuses and descriptors; - turn on the cronjob for rsync and import of newest archives; - caching; - hopefully later integration with the bandwidth and weights documents in Onionoo proper; - import 2008 year data, etc. These couple of weeks coincide with an (not quite planned for) apartment-moving period, but I *really* hope I have the vast majority of the things mentioned working in a decent state by the end of the next week, to leave ourselves some wiggle space to ensure the resulting system is as stable as possible before the official date for the end of the coding period. I'll probably be available throughout the weekend/s, just in case. Cheers to you all Kostas. [1]: https://lists.torproject.org/pipermail/tor-dev/2013-August/005310.html [2]: https://lists.torproject.org/pipermail/tor-dev/2013-August/005311.html [3]: https://github.com/wfn/torsearch/blob/master/docs/onionoo_api.md

1 0

IRC meeting to discuss sponsor F progress on Tue September 3, 18:00 to 19:00 UTC in #tor-dev
by Karsten Loesing 05 Sep '13

05 Sep '13

Hi all, I'd like to schedule an IRC meeting to discuss what progress we made on sponsor F deliverables in August. Time and place are: Tue September 3, 18:00 to 19:00 UTC in #tor-dev That's in (exactly) 4 1/2 days from now (sorry for the short notice). The time in other timezones is: 11:00 in San Francisco 14:00 in Boston 20:00 in Berlin 21:00 in Athens 23:30 in New Delhi People who should attend are: Roger, Nick, George, David, Ximin, Nathan, Matt, Colin, Andrew, Tom, and anyone else who wants to attend. If your name is on that list and you're working on sponsor F stuff, but the date or time doesn't work for you, please email me your progress and I'll report it for you at the meeting. Thanks, Karsten

3 4

Traffic Obfuscation
by josef.winger＠email.de 05 Sep '13

05 Sep '13

Can a developer please explain to me why something like the following obfuscation of 'torified traffic' is exploitable? Suppose a scenario where a collective of authorities is able to observe large parts of the www. Then observing traffic correlation can unreveal a connection through the network. But why can't we just alter the pattern inside the network, such that there is no correlation between 'incomming' and 'outgoing' data anymore? Suppose I'm connected to a server and there is a lot of traffic between from the server to me. Through the TOR network, of course. Data is encrypted and the exploit measures the raw data stream pattern. But why not change the data stream inside the network? Suppose the server A is outside of the TOR network, i.e. it is not a hidden service. The data stream into our network then is out of our controll. Encrypted or not, we can't change it. Now it flows to node B (exit node) and inside TOR. Then node B streams the data to node C, than node C streams the data to node D and node D exits the stream to me. (simplification) Ok now node B got the data from the 'outside world'. Then B and C first make a handshake to define a shared key for a private encryption protocoll only valid for some time. Now node B does not stream the data to node C, but obfuscates it. That means if there are n packages it transforms them into m packages in some unpredictable way and each new packages gets a small amount of additional random-data. (The point is that the new stream will not look at all like the old one) Only node B nows the way to de-obfuscate this. But B and C did a handshake and using this encryption B shares with C how to de-obfuscate the data. Now C recovers the real data and then does another secret handshake with D. (separated from the shared secret with B of course) Then C obfuscates the data again and only D will know how to recover the original data. This repeats until I recive an obfuscatet stream and my client can recover the original data. ====== The point is, that patterns of the in-stream of the server A aren't correlated to what streams from D to me anymore. Hence an observer isn't able to see correlations anymore. Number, size and pattern of the stream packages are different. Above this one could try to add randomly zero-information streams between the network and its clients. That way an observer can't even be sure if a client recieves information at al... ====== Now can someone please downtalk this approach? best /jo

3 2

regarding control spec for hidden service descriptor
by Qingping Hou 04 Sep '13

04 Sep '13

Hi all, I am currently working on ticket 8510: https://trac.torproject.org/projects/tor/ticket/8510 However, I notice that in the control spec, there is no event related to hidden service descriptors. Should we add a new event in the spec? For example, we can have something like the following: HS_DESC SP REQUESTED SP OnionAddress SP DescptorID SP AuthType SP HsDir SP Any thoughts? Cheers, qp

4 10

Re: [tor-dev] Pluggable Transport Browser Bundle FTE Integration
by Arlo Breault 03 Sep '13

03 Sep '13

> On Sat, Aug 31, 2013 at 09:43:21PM -0400, Kevin P Dyer wrote: > > === Building/Testing Tor on Linux/OSX/Windows > > I'm trying to understand exactly how the current build/release process works > > for tor. In regards to the PT TBB it seems like there are a few resources > > [1,2,3]. However, is there a canonical documentation on how the release process > > works? I'm especially interested in what you guys are doing to produce builds > > on Windows. Are you using virtualization or do you have a few physical build > > machines? > > [1] https://gitweb.torproject.org/pluggable-transports/bundle.git > > [2] https://lists.torproject.org/pipermail/tor-dev/2013-June/005056.html > > [3] https://gitweb.torproject.org/builders/tor-browser-bundle.git > > > > https://gitweb.torproject.org/pluggable-transports/bundle.git/blob/HEAD:/bu… You may have an easier time with MinGW than Cygwin. One way to get a compatible shell environment is to install Git Bash from, http://git-scm.com/downloads

1 0

Draft of proposal "Migrate HS identity keys to Ed25519"
by George Kadianakis 03 Sep '13

03 Sep '13

Greetz, I'm posting the draft of a proposal that specifies how to upgrade the identity keys of HSes (currently RSA-1024) to use Ed25519. This proposal is supposed to go along with a proposal that specifies how to hide HS descriptors from HSDirs. I'm going to post that second proposal in a few minutes. This proposal is incredibly drafty in the sense that I might have forgotten to specify things that need to be specified. On the other hand, "release early; release often" they say, so here it goes. Inlining: Filename: xxx-hs-ecc-id-keys.txt Title: Migrate HS identity keys to Ed25519 Author: George Kadianakis Created: 10 August 2013 Target: 0.2.5.x Status: Draft [More draft than Guiness.] 0. Overview: This proposal suggests the adoption of ECDSA keys as the long-term identity keys of Hidden Services. It also proposes the adoption of ECDSA keys for the per-introduction-point service key that was introduced in the v2 hidden service protocol. This proposal specifies the key generation procedure, paths where the keys should be stored, and flagday/migration procedures. XXX 1. Motivation: Hidden Services currently use an RSA-1024 keypair as their long-term identity keys. RSA-1024 is considered weak and should be replaced. Dan Bernstein's ed25519 ECDSA scheme provides 2^128 bits of security and better signature performance than RSA. Furthermore, the keysize is much smaller than the keys of RSA. 2. Related proposals This document is part of a proposal triptych. Upcoming sister proposals are: - "Stop HS address enumeration by HSDirs", where we propose an alternative HS descriptor format that will not allow HSDirs to learn the addresses of the Hidden Services it serves. - "Deployment strategy for recent HS proposals", where we specify how these proposals should be deployed without making people angry. 3. Changes to the current HS protocol 3.0. Overview of changes to the current HS protocol There are two places where RSA-1024 is used in the current HS: Longterm RSA-1024 "identity keys" are used by the HSes to authenticate themselves to clients. Multiple short-term RSA-1024 "service keys" are used by the HSes for each introduction point so that they don't reveal their identity key. This proposal specifies: a) The generation of the new ed25519 long-term identity keys and service keys (#KEYGEN) b) The new HS descriptor format (v3) that contains identity keys and service keys (#NEWDESC) c) A way for clients to upload and fetch v3 descriptors from HSDirs (#HSDIRV3) 3.1. Generation of ed25519 keypairs (#KEYGEN) 3.1.0. Generation of long-term ed25519 identity key Compliant Hidden Services need to generate a fresh ed25519 keypair and store the private key in: $HiddenServiceDirectory/hs_ed25519_privkey For deployment and migration reasons, the new ed25519 keys must be kept alongside with the old RSA-1024 keys. 3.1.1. Generation of short-term ed25519 service keys Hidden Services generate an ed25519 service key for each introduction point -- instead of the RSA-1024 key they currently generate. 3.2. New v3 Hidden Service descriptors 3.2.0. New v3 Hidden Service Descriptor Format (#NEWDESC) v2 Hidden Service descriptors contain both the long-term identity public key of the Hidden Service, and service keys for each introduction point. It also contains a signature of the descriptor. We need to change all these fields to carry our brand new ed25519 keys. For v3 Hidden Service descriptors, we keep the v2 format and perform the following changes: [*] The "permanent-key" field is replaced by "permanent-key-ed25519": "permanent-key-ed25519" NL an ed25519 public key [Exactly once] The public key of the hidden service which is required to verify the "descriptor-id" and the "signature-ed25519". In base64 format with terminating =s removed. [*] The "service-key" field is replaced by "service-key-ed25519': "service-key-ed25519" NL an ed25519 public key [Exactly once] The public key that can be used to encrypt messages to the hidden service. In base64 format with terminating =s removed. [*] The "signature" field is replaced by "signature-ed25519': "signature-ed25519" NL signature-string [At end, exactly once] A signature of all fields above with the private ed25519 key of the hidden service. In base64 format with terminating =s removed. 3.2.1. Uploading v3 HS descriptors to HSDirs (#HSDIRV3) A new type of Hidden Service Directory Server must be established which understands v3 Hidden Service descriptors. The Hidden Service follows the same publishing procedure as for v2 descriptors but instead of sending an HTTP 'POST' to "/tor/rendezvous2/publish", the HS sends the 'POST' request to "/tor/rendezvous3/publish". (This part of the proposal conflicts with the "Stop HS address enumeration by HSDirs" proposal.) 3.3. Fetching v3 HS descriptors from HSDirs (#HSDIRV3) When a client needs to fetch a v3 Hidden Service Descriptor from an HSDir, it follows the exact same procedure as for v2 descriptors but instead of sending an HTTP 'GET' to "/tor/rendezvous2/<z>", it sends an HTTP 'GET' to "/tor/rendezvous3/<z>". (This part of the proposal conflicts with the "Stop HS address enumeration by HSDirs" proposal) 3.4. Service keys and INTRODUCE cells In INTRODUCE cells, when v2 descriptors are used, PK_ID is defined as the hash of the service key. This means that we don't need to change the format of INTRODUCE cells since we can just use the hash of the ed25519 service key. # XXX will this cause problems?

3 5

Re: [tor-dev] Help with rebasing arma's n23-5 tor branch to current master
by Karsten Loesing 03 Sep '13

03 Sep '13

(Moving this discussion back to tor-dev@ with Charlie's permission.) On 8/6/13 4:10 AM, Charlie Belmer wrote: > Hey Karsten, > > I've been reading up on Shadow & Scallion to do some performance testing on > N23. Before I go too far down that road, I wanted to check in with you. > > Are there existing blueprints or best practices I should be following? Thanks for looking into this! So, I wouldn't call the following notes blueprints or best practices, but they are a description of what I would do to run Shadow simulations of your branch. First of all, be sure to read the Shadow wiki which has contains lots of interesting stuff: https://github.com/shadow/shadow/wiki The next step would be to get a vanilla Tor master (that your branch is based on) running in a minimal or tiny Shadow network. It's probably easiest to use a local Ubuntu VM for this. Or you can start an m1.large EC2 instance for the tiny network, or an even smaller instance for the minimal network. Once that simulation succeeds, you'll want to run another simulation using your branch, extract performance metrics using Shadow's analyze script, produce a PDF with graphs also using the analyze script, and then wonder if things are faster than before. The minimal or tiny networks are mostly there to learn that things are working as expected. You'll probably want larger networks for real performance results, but starting small to get everything working is probably easier. Sounds doable, I guess. However, here's where things start to get tricky: in order to simulate Tor master (or a branch based on it) in Shadow, you'll have to perform quite a few extra steps like compiling your own CMake and Clang/LLVM. Not really hard, but you need to get all pieces together before Shadow does what you want. I'm pasting detailed instructions that I used to simulate Steven's uTP branch in Shadow below as an example. Not everything applies to your branch, but it should give you the general idea. If something remains unclear, please feel free to ask! > Depending how it goes, I'd also look at re-trying adaptive 23 or tuning the > existing implementation. Neat! We should probably include Mashael in that discussion once you have a working and simulated n23-5 branch. Maybe she has ideas what parts need tweaking. I'm cc'ing her in this mail so she knows what you're working on. In any case, having a working rebased n23-5 branch is already a big step forward. Thanks! Karsten === START OF INSTRUCTIONS === Launch new EC2 instance running Ubuntu Server 13.04 64 bit Set ulimit -n to 25000, see Shadow wiki for details https://github.com/shadow/shadow/wiki/Preparing-your-machine#system-configs… $ sudo apt-get update $ sudo apt-get upgrade Clone and test-compile modified libutp $ sudo apt-get install build-essential git automake libssl-dev libevent-dev $ mkdir src $ cd src/ $ git clone https://github.com/sjmurdoch/libutp $ cd libutp/ $ make $ cd ../ Clone and test-compile Tor branch utp $ git clone https://git.torproject.org/tor.git $ cd tor/ $ git remote add sjm217 https://git.torproject.org/sjm217/tor.git $ git fetch sjm217 $ git checkout -b utp sjm217/utp $ ./autogen.sh $ LDFLAGS="-L/home/ubuntu/src/libutp" CFLAGS="-I/home/ubuntu/src/libutp" LIBS="-lutp -lrt" ./configure --disable-asciidoc $ make $ make distclean $ cd ../ Modify Tor to use uTP for all links (or "0 &&" for not using uTP for any link) diff --git a/src/or/channeltls.c b/src/or/channeltls.c index 0551b73..b7b36e1 100644 --- a/src/or/channeltls.c +++ b/src/or/channeltls.c @@ -418,7 +418,7 @@ channel_tls_connect(const tor_addr_t *addr, uint16_t port, /* Create a uTP connection */ tor_addr_to_sockaddr(addr, port, (struct sockaddr*)&sin, sizeof(sin)); tor_addr_to_str(addr_str, addr, sizeof(addr_str), 0); - if (!strncmp(addr_str, "128.232.10.129", sizeof(addr_str))) { + if (1 || !strncmp(addr_str, "128.232.10.129", sizeof(addr_str))) { log_info(LD_CHANNEL, "Trying uTP connection to %s", addr_str); tlschan->utp = UTP_Create(tor_UTPSendToProc, tlschan, (const struct sockaddr*)&sin, Install CMake 2.8.10 $ wget http://www.cmake.org/files/v2.8/cmake-2.8.10.2.tar.gz $ tar xf cmake-2.8.10.2.tar.gz $ cd cmake-2.8.10.2/ $ ./configure $ make $ export PATH=~/src/cmake-2.8.10.2/bin:$PATH $ cd ../ $ which cmake # result: /home/ubuntu/src/cmake-2.8.10.2/bin/cmake Build Clang and LLVM from source $ wget http://llvm.org/releases/3.2/llvm-3.2.src.tar.gz $ wget http://llvm.org/releases/3.2/clang-3.2.src.tar.gz $ tar xaf llvm-3.2.src.tar.gz $ tar xaf clang-3.2.src.tar.gz $ cp -R clang-3.2.src llvm-3.2.src/tools/clang $ cd llvm-3.2.src/ $ mkdir build $ cd build/ $ cmake -DCMAKE_INSTALL_PREFIX=/home/ubuntu/.local ../. $ make $ make install $ export PATH=~/.local/bin/:$PATH $ cd ../../ $ which clang # result: /home/ubuntu/.local/bin//clang Install Shadow $ sudo apt-get install gcc xz-utils make automake autoconf cmake tidy libtidy-dev libglib2.0 libglib2.0-dev dstat pdftk python2.7 python-matplotlib python-numpy python-scipy $ git clone https://github.com/shadow/shadow.git $ cd shadow $ ./setup dependencies $ ./setup build -g -i /home/ubuntu/src/libutp -i /usr/include/x86_64-linux-gnu/c++/4.7 -l /home/ubuntu/src/libutp --tor-prefix /home/ubuntu/src/tor --tor-lib utp $ ./setup install $ export PATH=~/.shadow/bin/:$PATH Simulate tiny network in Shadow $ cd resource/examples/scallion/ $ tar xf tiny-4GB-m1.large.tar.xz $ cd tiny-4GB-m1.large/ $ scallion Graph simulation results Add one line to beginning of analyze.py: import matplotlib; matplotlib.use('Agg') For additional traffic simulations: $ grep -v "\[traffic-" scallion.log > scallion-notraffic.log $ cd ~/src/ $ mkdir results $ cd results/ $ python ~/src/shadow/contrib/analyze.py parse --output uTP-1 ~/src/shadow/resource/examples/scallion/tiny-4GB-m1.large/data-utp-1/scallion.log $ python ~/src/shadow/contrib/analyze.py parse --output uTP-0 ~/src/shadow/resource/examples/scallion/tiny-4GB-m1.large/data-utp-0/scallion.log $ python ~/src/shadow/contrib/analyze.py parse --output vanilla ~/src/shadow/resource/examples/scallion/tiny-4GB-m1.large/data-vanilla/scallion.log $ python ~/src/shadow/contrib/analyze.py plot --title "sjm217's utp branch (#9166)" --prefix "sjm217-utp" --data vanilla/ "vanilla 0.2.4.4-alpha" --data uTP-1/ "uTP for all links" --data uTP-0/ "uTP for none of the links" === END OF INSTRUCTIONS ===

1 1

[RELEASE] Torsocks 2.0.0-rc2
by David Goulet 02 Sep '13

02 Sep '13

Greetings everyone! After a week or so, the release candidate 2 is now out after receiving various contributions for BSD and OS X support. A quick note. Please use the Github bug tracker for any issues and *not* trac.torproject.org. Until this code base is accepted as a potential replacement for the old version, please use the current repository issue tracker. Here is the ChangeLog: 2013-09-02 torsocks 2.0.0-rc2 * Fix: remove FAQ file from Makefile * Fix: remove out of date and inaccurate FAQ * Tests: add connection object unit test * Fix: Improve README file * Tests: add onion pool subsystem unit test * Use extern for tsocks_libc_* in torsocks.h * Define LIBC_SYSCALL_ for OS X * Make sure __darwin__ is defined * Fix: explicitly ignore fileno return value * Use AC_CHECK_FUNC rather than AC_LINK_IFELSE. * Find out if we really need libdl. * Define LIBC_SYSCALL_ for FreeBSD. * Use SYS_ from <sys/syscall.h>. * Include <sys/socket.h> for AF_INET*. * Use getsockname(2) for finding out socket address family Please continue to test, review and contribute it! :) Tarball: https://github.com/dgoulet/torsocks/archive/v2.0.0-rc2.tar.gz Big thanks to all contributors for that release! Cheers! David

1 0

Searchable metrics archive - Onionoo-like API available online for probing
by Kostas Jakeliunas 02 Sep '13

02 Sep '13

Hello, this accompanies my status report [1], and includes info how to query the searchable metrics archive for anyone curious. I also refer to the original (now semi-outdated) project proposal/document. [0] Only sending to tor-dev@for now. The Onionoo-like backend is listening on http://ts.mkj.lt:5555/ (backup URI where it's actually running (domain unrelated): ravinesmp.com:5555) This document details how it can be queried: https://github.com/wfn/torsearch/blob/master/docs/onionoo_api.md (It is, by design, an almost-subset (it does change some things as of now, though) of the Onionoo API [2].) Try not to DoS the thing, it is still fragile! :) (or, maybe do, but gently?) I will try to follow-up with a more detailed email, but this will have to do for now. Anyone curious should refer to the API doc. I'll soon add date range parameters, so it will become somewhat more useful. Examples: http://ts.mkj.lt:5555/details?search=mor http://ts.mkj.lt:5555/details?search=79. http://ts.mkj.lt:5555/statuses?lookup=9695DFC35FFEB861329B9F1AB04C46397020C… Please report any inconsistencies / errors / time-outs / anything that takes a few seconds or more to execute. I'm logging the queries (together with IP addresses for now - for shame!), so will be able to later correlate activity with database load, which will hopefully provide some realistic semi-benchmark-like data. [0]: http://kostas.mkj.lt/gsoc2013/gsoc2013.html [1]: https://lists.torproject.org/pipermail/tor-dev/2013-August/005310.html [2]: https://onionoo.torproject.org/ -- Kostas. 0x0e5dce45 @ pgp.mit.edu

2 3