November 2012 - tor-dev - lists.torproject.org

questions about parallel Tor circuits en masse & adoption drivers
by Simon 27 Nov '12

27 Nov '12

Hello, I'm interesting in discussing the development aspects of using Tor as a massively distributed anonymous file server where blocks (e.g. 4KB) of a particular file (e.g. even several GB in size) might be redundantly distributed on very many hidden service nodes (e.g. a unique 4KB block of a file redundantly duplicated on ~ x hidden nodes from y million total hidden service nodes) and access to the file would be high through-put because very many (e.g. cable bandwidth / 4KB, e.g. 50 Mbps is 6.25 MB/s or 1600 * 4KB blocks per second) concurrent but high latency Tor circuits. Thus each Tor circuit would still be the usual high latency, but very many Tor circuits in parallel would deliver fantastic through-put. What if Tor could handle several rolling window bundles of e.g. 1,600 parallel Tor circuits which request one small block of info from 1,600 unique hidden service nodes? Does Tor parallelized, high bandwidth file sharing already exist? Does Tor already handle a massively parallelized number of Tor circuits, on the scale of thousands or tens of thousands of Tor circuits? Is many concurrent Tor circuits desirable? Who is working on it? Would using Tor as a massively distributed anonymous file server dramatically increase the number of Tor nodes (due to file sharing being more desirable than anonymity) and therefore make the entire Tor system more resilient to attack due to high growth and orders of magnitude more nodes? Is such an expansion -- on the back of a new functional focal point; i.e. not anonymity -- politically desirable for the Tor project? Thanks, Simon Experienced network programming enthusiast from Vancouver, Canada

1 0

tor exit node modules in Tor source codes
by esolve esolve 27 Nov '12

27 Nov '12

I'm doing research on Tor and I want to modify the tor exit node functions. I downloaded the source codes from this link https://gitweb.torproject.org/tor.git/tree/b13c6becc892d9716d8645f16b5d03bd… but I can't find the modules/files which are related to tor exit functions(like receive cells from previous relay router, send HTTP/TCP connection request to webserver/remote_host, buffer management) can anyone give some hints? thanks!

1 0

Re: [tor-dev] RIAA v Verizon: 404
by Christian Kujau 27 Nov '12

27 Nov '12

On Mon, 26 Nov 2012 at 17:40, leez wrote: > The lost document was actually this one: > https://www.eff.org/document/dc-circuit-opinion-granting-verizons-motion-qu… > > I've added a redirect from the broken link, so either will get you > there now. Thanks for fixing this! Christian. -- BOFH excuse #171: NOTICE: alloc: /dev/null: filesystem full

1 0

[PATCH 1/2] New option LocalOutboundBindAddress
by Andrew Cady 27 Nov '12

27 Nov '12

This works just like OutboundBindAddress, but instead of excluding connections to loopback addresses, it affects only connections to loopback addresses. The whitespace in this commit is a bit nutty, in order to put whitespace changes into a separate commit. --- doc/tor.1.txt | 11 +++++++++++ src/or/config.c | 25 ++++++++++++++++--------- src/or/connection.c | 17 ++++++++++------- src/or/or.h | 6 ++++++ 4 files changed, 43 insertions(+), 16 deletions(-) diff --git a/doc/tor.1.txt b/doc/tor.1.txt index 1e1ff1e..f25234d 100644 --- a/doc/tor.1.txt +++ b/doc/tor.1.txt @@ -477,6 +477,17 @@ GENERAL OPTIONS This setting will be ignored for connections to the loopback addresses (127.0.0.0/8 and ::1). +**LocalOutboundBindAddress** __IP__:: + Like OutboundBindAddress, but _only_ for connections to loopback + addresses (e.g., inbound connections to local hidden services). + This is useful to distinguish Tor traffic from local traffic in + the log files of your hidden services. + + + + The IP should be in the 127.0.0.0/8 range for maximum compatibility. + I recommend using 127.84.111.114, since 84.111.114 is the ASCII + encoding of "Tor". I also recommend adding \'127.84.111.114 tor' to + your /etc/hosts file. + **PidFile** __FILE__:: On startup, write our PID to FILE. On clean shutdown, remove FILE. diff --git a/src/or/config.c b/src/or/config.c index 206ccc8..03e07f9 100644 --- a/src/or/config.c +++ b/src/or/config.c @@ -285,6 +285,7 @@ static config_var_t option_vars_[] = { OBSOLETE("IgnoreVersion"), V(KeepalivePeriod, INTERVAL, "5 minutes"), VAR("Log", LINELIST, Logs, NULL), + V(LocalOutboundBindAddress, LINELIST, NULL), V(LogMessageDomains, BOOL, "0"), OBSOLETE("LinkPadding"), OBSOLETE("LogLevel"), @@ -5556,15 +5557,20 @@ getinfo_helper_config(control_connection_t *conn, static int parse_outbound_addresses(or_options_t *options, int validate_only, char **msg) { - const config_line_t *lines = options->OutboundBindAddress; - int found_v4 = 0, found_v6 = 0; + int i; + for (i=0; i<2; ++i) { + + tor_addr_t *out4 = i ? &options->OutboundBindAddressIPv4_ : &options->LocalOutboundBindAddressIPv4_; + tor_addr_t *out6 = i ? &options->OutboundBindAddressIPv6_ : &options->LocalOutboundBindAddressIPv6_; if (!validate_only) { - memset(&options->OutboundBindAddressIPv4_, 0, - sizeof(options->OutboundBindAddressIPv4_)); - memset(&options->OutboundBindAddressIPv6_, 0, - sizeof(options->OutboundBindAddressIPv6_)); + memset(out4, 0, sizeof(*out4)); + memset(out6, 0, sizeof(*out6)); } + + const config_line_t *lines = i ? options->OutboundBindAddress : options->LocalOutboundBindAddress; + int found_v4 = 0, found_v6 = 0; + while (lines) { tor_addr_t addr, *dst_addr = NULL; int af = tor_addr_parse(&addr, lines->value); @@ -5577,7 +5583,7 @@ parse_outbound_addresses(or_options_t *options, int validate_only, char **msg) return -1; } found_v4 = 1; - dst_addr = &options->OutboundBindAddressIPv4_; + dst_addr = out4; break; case AF_INET6: if (found_v6) { @@ -5587,7 +5593,7 @@ parse_outbound_addresses(or_options_t *options, int validate_only, char **msg) return -1; } found_v6 = 1; - dst_addr = &options->OutboundBindAddressIPv6_; + dst_addr = out6; break; default: if (msg) @@ -5599,7 +5605,8 @@ parse_outbound_addresses(or_options_t *options, int validate_only, char **msg) tor_addr_copy(dst_addr, &addr); lines = lines->next; } - return 0; + } + return 0; } /** Load one of the geoip files, <a>family</a> determining which diff --git a/src/or/connection.c b/src/or/connection.c index dbcfc41..397a2dc 100644 --- a/src/or/connection.c +++ b/src/or/connection.c @@ -1451,14 +1451,17 @@ connection_connect(connection_t *conn, const char *address, make_socket_reuseable(s); - if (!tor_addr_is_loopback(addr)) { + { + const int is_local = tor_addr_is_loopback(addr); const tor_addr_t *ext_addr = NULL; - if (protocol_family == AF_INET && - !tor_addr_is_null(&options->OutboundBindAddressIPv4_)) - ext_addr = &options->OutboundBindAddressIPv4_; - else if (protocol_family == AF_INET6 && - !tor_addr_is_null(&options->OutboundBindAddressIPv6_)) - ext_addr = &options->OutboundBindAddressIPv6_; + const tor_addr_t *maybe_ext_addr4 = is_local ? &options->LocalOutboundBindAddressIPv4_ : &options->OutboundBindAddressIPv4_; + const tor_addr_t *maybe_ext_addr6 = is_local ? &options->LocalOutboundBindAddressIPv6_ : &options->OutboundBindAddressIPv6_; + + if (protocol_family == AF_INET && !tor_addr_is_null(maybe_ext_addr4)) + ext_addr = maybe_ext_addr4; + else if (protocol_family == AF_INET6 && !tor_addr_is_null(maybe_ext_addr6)) + ext_addr = maybe_ext_addr6; + if (ext_addr) { struct sockaddr_storage ext_addr_sa; socklen_t ext_addr_len = 0; diff --git a/src/or/or.h b/src/or/or.h index a8645f8..a832e4f 100644 --- a/src/or/or.h +++ b/src/or/or.h @@ -3181,6 +3181,12 @@ typedef struct { tor_addr_t OutboundBindAddressIPv4_; /** IPv6 address derived from OutboundBindAddress. */ tor_addr_t OutboundBindAddressIPv6_; + /** Local address to bind outbound sockets -- for loopback connections (e.g., hidden services) */ + config_line_t *LocalOutboundBindAddress; + /** IPv4 address derived from LocalOutboundBindAddress. */ + tor_addr_t LocalOutboundBindAddressIPv4_; + /** IPv6 address derived from LocalOutboundBindAddress. */ + tor_addr_t LocalOutboundBindAddressIPv6_; /** Directory server only: which versions of * Tor should we tell users to run? */ config_line_t *RecommendedVersions; -- 1.7.10.4

1 1

RIAA v Verizon: 404
by Christian Kujau 26 Nov '12

26 Nov '12

Hi, the Tor DMCA response[0] has a link inside for the RIAA v. Verizon case: https://www.eff.org/legal/cases/RIAA_v_Verizon/opinion-20031219.pdf But this document is no longer available at this address, thus producing a 404. @EFF Webmaster, could you please fix this or provide the new address of this document? Is this perhaps the following: https://www.eff.org/sites/default/files/filenode/RIAA_v_Verizon/20030121-ri… I've cc'ed tor-dev, because the link is included in contrib/tor-exit-notice.html for the Tor exit nodes to present. Thanks, Christian. [0] https://www.torproject.org/eff/tor-dmca-response -- BOFH excuse #60: system has been recalled

2 1

Re: [tor-dev] Request for comments on notes on parallel cell crypto
by Nick Mathewson 19 Nov '12

19 Nov '12

On Sun, Nov 18, 2012 at 7:43 PM, Andrea Shepard <andrea(a)torproject.org> wrote: > I've made some notes on parallelizing cell crypto on relays which I've > attached to this mail and added to the wiki page on this [1], which I > would like comment on. Particularly, I want to resolve the following > points: > > * Should we operate on cell_t or packed_cell_t? After reading your comments, I don't feel strongly that packed_cell_t is correct. Either would probably be okay. I think I might have said "packed_cell_t" on the grounds that we already have a specialized allocator for them, whereas we don't currently heap-allocate as many cell_t as we would need for this. I might also have said "packed_cell_t" on the theory that copying data from cell_t to packed_cell_t would waste cycles, but I'm not sure that's significant here. Or maybe I was thinking of the amount of space wasted in packed_cell_t vs cell_t, and worried about the number of queued cells we're likely to have at once. For queueing the cell_ts, and where to put a next pointer: I *think* (but can't be sure) that most of our cell_t objects are currently short-lived, and most of our long-lived cell objects are currently packed_cell_t. So adding fields to cell_t wouldn't hurt much, since they don't currently represent very much of our total allocation, I think. (We should double-check that!) > * Is it acceptable for the main thread to call functions to > manipulate job-dispatching data structures that need to > acquire a lock if we can be sure everything that ever > holds the lock only does so very briefly to manipulate > those structures and will never block while holding it, > or do we need to design something that's lock-free for > the main thread? I don't think the main thread needs to be lock-free so long as it won't often block for too long. I was initially planning to have locks protect the work-queues initially, and only substitute in a lock-free work-queue implementation if it turned out to matter. (How long the lock is held by the main thread is less important for performance IIUC than whether the main thread will spend much time blocked waiting for the lock, or how much lock contention there will be.) > * The functions that currently call relay_crypt()/ > relay_crypt_one_payload() will have to be split into > two parts, one that queues a cell to be crypted and > another that handles the finished, crypted cell. > How will the worker threads cause the second part to > be called in the main thread? (NickM, there's probably > some libevent voodoo I don't know about yet for this?) So, I'd suggest the we start with a work queue each way: one queue (or set of queues) to enqueue stuff for worker threads, and one to enqueue the crypted stuff for the main cell to process it. That said, we'll need a way to alert the main thread "Hey, you have more data on your queue!" The easiest way to do that is, using Libevent 2.0 or later, to just call event_active() on a "struct event" whose callback processes the data. (This needs to require Libevent 2.0 or later, since Libevent 1.4 and earlier didn't let you do this threadsafely.) The next easiest alternative would be to construct a socketpair, have the main thread listen on the read side of it (using a read event), and have the worker thread write a byte to the socketpair whenever the queue went from "observed" . There are faster ways to do all of the above, and more ways we could optimize libevent to help with our particular case, but for now I'll suggest we ignore them: getting even one extra core to do crypto with will more than pay for all the overhead in a v1 implementation. We can find the painful points and optimize them later -- and waiting to do so will ensure we don't waste time optimizing outside of critical paths. In fact, we should probably be doing this for the rest of the design: looking to see if there are cases where even if we think we might need to get complex eventually, we can still do them simply at first. (For the case of sending work to the worker threads, a mutex/condition-var pair will do. Windows Vista added real condition variables, and there's plenty of compat code around to fake them with earlier Windowses.) I suggest that for the first implementation of this, we can just make the multithreaded crypto stuff require Libevent 2.0 or later. We'll want a compile option to turn it off for old OpenBSDs anyway. Turning multithreaded cypto off should be easy, right? Instead of "Thread1: enqueue. Thread2: unqueue, encrypt, enqueue. Thread1: unqueue, process", you just replace the first "enqueue" operation with "encrypt and process" -- perhaps making a stub relaycrypt_dispatcher_t that doesn't actually dispatch anything, but just does the crypto operation immediately. This might even be a good intermediate step while refactoring the crypto. > Once these are resolved I'll be making a branch for this and > writing the headers with proposed data structures and interfaces > for review. Cool; thank you! Another observation about your design sketch: If we implement this nicely, I think it will be possible to stress-test it and load-test the relaycrypt_dispatcher_t part of it without actually requiring the rest of Tor to be running. IOW, we could have a performance-test for this component that didn't require any actual Tor traffic to be happening, or circuits to exist. IMO that would be a pretty good idea, since it would help us optimize as needed, and it would encourage loose coupling. cheers, -- Nick

3 3

Proposal 215: Let the minimum consensus method change with time
by Nick Mathewson 19 Nov '12

19 Nov '12

Filename: 215-update-min-consensus-ver.txt Title: Let the minimum consensus method change with time Author: Nick Mathewson Created: 15 Nov 2012 Status: Open 0. Overview This proposal suggests that we drop the requirement that authorities support the very old consensus method "1", and instead move to a wider window of recognized consensus methods as Tor evolves. 1. Background and Motivation When we designed the directory voting system, we added the notion of "consensus method" so that we could smoothly upgrade the voting process over time. We also said that all authorities must support the consensus method '1', and must fall back to it if they don't support the method that the supermajority of authorities will choose. Consensus method 1 is no longer viable for the Tor network. It doesn't result in a microdescriptor consensus, and omits other fields that clients need in order to work well. Consensus methods under 12 have security issues, since they let a single authority set a consensus parameter. In the future, new consensus methods will be needed so that microdescriptor-using clients can use IPv6 exits and ECC onion-keys. Rolling back from those would degrade functionality. We need a way to change the minimum consensus method over time. 2. Design I propose that we change the minimum consensus method about once per release cycle, or once per ever other release cycle. As a rule of thumb, let the minimum consensus method in Tor series X be the highest method supported by the oldest version that "anybody reasonable" would use for running an authority. Typically, that's the stable version of the previous release series. For flexibility, it might make sense to choose a slightly older method, if falling back to that method wouldn't cause security problems. For example, while Tor 0.2.4.x is under development, authorities should really not be running anything before Tor 0.2.3.x. Tor 0.2.3.x has supported consensus method 13 since 0.2.3.21-rc, so it's okay for 0.2.4.x to require 13 as the minimum method. We even might go back to method 12, since the worst outcome of not using 13 would be some warnings in client logs. Consensus method 12 was a security improvement, so we don't want to roll back before that. 2.1. Behavior when the method used is one we don't know The spec currently says that if an authority sees that a method will be used that it doesn't support, it should act as if the consensus method will be "1". This attempt will be doomed, since the other authorities will be computing the consensus with a more recent method, and any attempt to use method "1" won't get enough signatures. Instead, let's say that authorities fall back to the most recent method that they *do* support. This isn't any likelier to reach consensus, but it is less likely to result in anybody signing something they don't like. 3. Likely outcomes If a bunch of authorities were to downgrade to a much older version, all at once, then newer authorities would not be able to sign the consensus they made. That's probably for the best: if a bunch of authorities were to suddenly start running 0.2.0.x, consensing along with them would be a poor idea. 4. Alternatives We might choose a less narrow window of allowable method, when we can do so securely. Maybe two release series, rather than one, would be a good interval to do when the consensus format isn't changing rapidly. We might want to have the behavior when we see that everybody else will be using a method we don't support be "Don't make a consensus at all." That's harder to program, though.

1 0

Free and Open Source Software internships for women
by Sathyanarayanan Gunasekaran 16 Nov '12

16 Nov '12

Free and Open Source Software (FOSS) is software that gives the user the freedom to use, copy, study, change, and improve it. FOSS contributors believe that this is the best way to develop software because it benefits society, creates a fun collaborative community around a project, and allows anyone to make innovative changes that reach many people. In an effort to get more women involved in FOSS, a number of organizations are offering Outreach Program for Women internships from January 2 to April 2, 2013. The application deadline is December 3, 2012. The applicants need to get in touch with individual projects they are interested in working on and make a small contribution ahead of the application deadline. Participants will work remotely from home, while getting guidance from an assigned mentor and collaborating within their project’s team and the rest of the FOSS community. The projects include software development, system administration, web development, user experience design, graphic design, documentation, and marketing. The stipend for the program is $5,000 (USD). Any woman available for a full time internship who has not previously participated in the Outreach Program for Women or Google Summer of Code is welcome to apply. College women from the Southern Hemisphere who will have a school summer break during most of this time are particularly encouraged to apply. Please visit https://www.torproject.org/getinvolved/volunteer.html.en#opw to learn more and apply.

1 0

Proposal 214: Allow 4-byte circuit IDs in a new link protocol
by Nick Mathewson 10 Nov '12

10 Nov '12

Filename: 214-longer-circids.txt Title: Allow 4-byte circuit IDs in a new link protocol Author: Nick Mathewson Created: 6 Nov 2012 Status: Open 0. Overview Relays are running out of circuit IDs. It's time to make the field bigger. 1. Background and Motivation Long ago, we thought that 65535 circuit IDs would be enough for anybody. It wasn't. But our cell format in link protocols is still: Cell [512 bytes] CircuitID [2 bytes] Command [1 byte] Payload [509 bytes] Variable-length cell [Length+5 bytes] CircID [4 bytes] Command [1 byte] Length [2 bytes] Payload [Length bytes] This means that a relay can run out of circuit IDs pretty easily. 2. Design I propose a new link cell format for relays that support it. It should be: Cell [514 bytes] CircuitID [4 bytes] Command [1 byte] Payload [509 bytes] Variable cell (Length+7 bytes) CircID [4 bytes] Command [1 byte] Length [2 bytes] Payload [Length bytes] We need to keep the payload size in fixed-length cells to its current value, since otherwise the relay protocol won't work. This new cell format should be used only when the link protocol is 4. (To negotiation link protocol 4, both sides need to use the "v3" handshake, and include "4" in their version cells. If version 4 or later is negotiated, this is the cell format to use.) 2.1. Better allocation of circuitID space In the current Tor design, circuit ID allocation is determined by whose RSA public key has the lower modulus. How ridiculous! Instead, I propose that when the version 4 link protocol is in use, the connection initiator use the low half of the circuit ID space, and the responder use the low half of the circuit ID space. 3. Discussion * Why 4 bytes? Because 3 would result in an odd cell size, and 8 seems like overkill. * Will this be distinguishable from the v3 protocol? Yes. Anybody who knows they're seeing the Tor protocol can probably tell by the TLS record sizes which version of the protocol is in use. Probably not a huge deal though; which approximate range of versions of Tor a client or server is running is not something we've done much to hide in the past. * Why a new link protocol and not a new cell type? Because pretty much every cell has a meaningful circuit ID. * Okay, why a new link protocol and not a new _set of_ cell types? Because it's a bad idea to mix short and long circIDs on the same channel. (That would leak which cells go with what kind of circuits ID, potentially.) * How hard is this to implement? I wasn't sure, so I coded it up. I've got a probably-buggy implementation in branch "wide_cird_ids" in my public repository. Be afraid! More testing is needed!

5 6

authentication questions on Tor controller
by jiang song 08 Nov '12

08 Nov '12

I just start to use tor controller, I notice the mannual says: pass Tor the "--CookieAuthentication 1" option when you start it. Tor will create a file in its data directory called "control_auth_cookie". All your controller needs to do is to pass the contents of this file to authenticate() when it connects to Tor. there are two kinds of Tor, one is installed and can be run by command "tor" the other is Tor Browser Bundle, 1) with command "tor", when it establishes, if I run example.py the result is "Connection refused. Is the ControlPort enabled?" I tried command: "tor --CookieAuthentication 1" to start Tor, but I can't find the so called data directory, and the running information says： Could not open "/etc/tor/torrc": Permission denied 2) with TBB, after I started TBB, if I run example.py, it seems it can connect to ControlPort, but it prompts for a password there is a Data directory in TBB, but I can't find the so called " control_auth_cookie" file, so when I run TBB, how can I pass "--CookieAuthentication 1" to tor? example.py : import time import TorCtl class BandwidthListener(TorCtl.PostEventListener): def __init__(self): TorCtl.PostEventListener.__init__(self) def bandwidth_event(self, event): print "tor read %i bytes and wrote %i bytes" % (event.read, event.written) # constructs a listener that prints BW events myListener = BandwidthListener() # initiates a TorCtl connection, returning None if it was unsuccessful conn = TorCtl.connect() if conn: # tells tor to send us BW events conn.set_events(["BW"]) # attaches the listener so it'll receive BW events conn.add_event_listener(myListener) # run until we get a keyboard interrupt try: while True: time.sleep(10) except KeyboardInterrupt: pass

2 1