November 2011 - tor-dev - lists.torproject.org

Sanitizing and publishing our web server logs
by Karsten Loesing 01 Dec '11

01 Dec '11

Hi everyone, we have been discussing sanitizing and publishing our web server logs for quite a while now. The idea is to remove all potentially sensitive parts from the logs, publish them in monthly tarballs on the metrics website, and analyze them for top visited pages, top downloaded packages, etc. See the tickets #1641 and #2489 for details. Here's a suggested sanitizing procedure for our web logs, which are in Apache's combined log format: - Ignore everything except GET requests. - Ignore all requests that resulted in a 404 status code. - Rewrite log lines so that they only contain the following fields: - IP address 0.0.0.0 for HTTP request or 0.0.0.1 for HTTPS requests (as logged by our Apache configuration), - the request date (with the time part set to 00:00:00), - the requested URL (cut off at the first encountered "?"), - the HTTP version, - the server's HTTP status code, and - the size of the returned object. - Write all lines from a given virtual host and day to a single output file. - Sort the output file alphanumerically to conceal the original order of requests. Here's a sample sanitized log file for www.torproject.org from May 1, 2009 (462K): http://freehaven.net/~karsten/volatile/www.torproject.org-access.log-200905… Is there still anything sensitive in that log file that we should remove? For example: - Do the logs reveal how many pages were cached already on the requestor's site (e.g. as repeat accesses)? Note that log files are sorted before being published. - Are there other concerns about making these sanitized log files publicly available? Are the decisions to remove parts from the logs reasonable? In particular: - Do we have to take out all requests with 404 status codes? Some of these requests for non-existing URLs contain typos which may not be safe to make public. Should we instead put in some placeholder for the URL part and keep the 404 lines to know how many 404's we have per day? - Is there any good reason to keep the portion of a URL after a "?"? - Is it possible to leave some part of Referers in the logs that helps us figure out where our traffic originates and what search terms people use to find us? - Can we resolve client IP addresses to country codes and include those in the logs instead of our 0.0.0.0/0.0.0.1 code for HTTP/HTTPS? How would we handle countries with only a few users per day, e.g., should there be a threshold below which we consider requests to come from "a country with less than XY users?" Thanks, Karsten

6 10

Failed test in UNICODE
by Gisle Vanem 30 Nov '11

30 Nov '11

I just built Tor and the test suite with UNICODE enabled (MS Visual-C v16 on Win-XP SP3). With some patching, the test-programs compiled and ran mostly fine. Except test_util_spawn_background_ok() failed for a reason I did't understand initially: > test --warn util/spawn_background_ok: Nov 23 14:35:55.531 [warn] Failed to create child process test-child.exe: Systemet finner ikke angitt fil. FAIL test_util.c:1405: assert(process_handle.status == expected_status): -1 vs 1 [spawn_background_ok FAILED] util/spawn_background_partial_read: Nov 23 14:35:55.546 [warn] Failed to create child process test-child.exe: Systemet finner ikke angitt fil. FAIL test_util.c:1521: assert(process_handle.status == expected_status): -1 vs 1 [spawn_background_partial_read FAILED] ("Systemet finner ikke angitt fil" -> "System doesn't find specified file"). I think the reason is that CreateProcessW() is used in common/util.c. The function is passed a 'const char *const filename'. Either we should always use CreateProcessA() or convert 'filename' using mbstowcs(). What do you think? --gv

2 1

Periodic alert to outdated Tor operators?
by Fabio Pietrosanti (naif) 30 Nov '11

30 Nov '11

Hi all, should we send a periodic email reminder to all operators running outdated Tor routers? >From a brief look at cached-descriptors: grep 'Tor 0\.2\.0' /var/lib/tor/cached-descriptors | wc -l 111 grep 'Tor 0\.2\.1' /var/lib/tor/cached-descriptors | wc -l 5631 grep 'Tor 0\.2\.2' /var/lib/tor/cached-descriptors | wc -l 11572 grep 'Tor 0\.2\.3' /var/lib/tor/cached-descriptors | wc -l 1182 It would be nice to have an automatic system that spam periodically all Tor operators that run outdated Tor routers. -naif

3 2

Re: [tor-dev] Proposal 178: Require majority of authorities to vote for consensus parameters
by Nick Mathewson 27 Nov '11

27 Nov '11

On Tue, Feb 22, 2011 at 1:34 AM, Sebastian Hahn <hahn.seb(a)web.de> wrote: > Design: > > When the consensus is generated, the directory authorities ensure that > a param is only included in the list of params if at least half of the > total number of authorities votes for that param. The value chosen is > the low-median of all the votes. We don't mandate that the authorities > have to vote on exactly the same value for it to be included because > some consensus parameters could be the result of active measurements > that individual authorities make. This is possibly bikeshed, but I would suggest that instead of requiring half of existing authorities to vote on a particular parameter, we require 3 or more to vote on it. (As a degenerate case, fall back to "at least half" if there are fewer than 6 authorities in the clique.) I think we don't want the number to be _less_ than 3, since 3 is the smallest number of parties who can come up with a low-median that isn't just the diktat of a single member (1 party), or as low as either member wants it to be (2 parties). yrs, -- Nick

2 7

Bug triage status
by Nick Mathewson 24 Nov '11

24 Nov '11

Hi, Roger! As discussed, I did some Tor bug triage tonight. Here's what I did: * I made a new "Tor: very long term" (formerly "Tor: distant future") milestone for stuff too long-term to be in "Tor: unspecified." Now we will no longer have "unspecified" mean "maybe never" to some of us and "I honestly don't know" to others. * I made a new "tor: 0.2.4.x" milestone * I went through every ticket in "post 0.2.3.x" (there were like 3 or 4) and reassigned them someplace better. * I went through every ticket without a component. * I went through every ticket in Tor: unspecified to see if it belonged there. Surprisingly, very few of them belonged in "Tor: very long term". * I went through every ticket without a milestone that was assigned to one of the core Tor components (that is, Tor Bridge, Tor Hidden Service, Tor Directory Authority, Tor Client, Tor Relay). Now they all have milestones; some few of them also have comments, or are closed. Still to do, possibly by me, possibly not: * Triage all 233 open 0.2.3.x tickets. The ones that are feature-like, big, and not planned to get done by nov 30 should get punted to "unspecified" or 0.2.4.x or "very long term." The ones that are feature-like and small should get ordinary triage. The ones that are bugs should get priorities assigned, or marked as deferrable, or such. * Triage all 23 open 0.2.2.x-final tickets. Some of these are bugs that we believe *should* have their fixes applied on 0.2.2; some are merely bugs that *exist* in 0.2.2, but which might not be important enough to alter a stable release. * Triage all core-Tor bugs in a milestone other than 0.2.2.x-final, 0.2.3.x-final, 0.2.4.x-final, unspecified, or very long term. Next steps for Roger here: * Go over the "Unspecified" stuff I've triaged to see if you agree. It should be cleaner than it was before, happy Thanksgiving, -- Nick

1 0

Re: [tor-dev] Tor glossary
by Shondoit 23 Nov '11

23 Nov '11

During translating last week it occurred to me that we do not have a consistent translation for some common phrases. I've set up a first draft of all common words that are used throughout the various Tor projects. Why I'm sending this to tor-dev is the following: - You as developers know which phrases are important for a clear communication of Tor's (and others) workings. I would very much like if you can add to the lists the words that are essential to be consistently translated. (as of writing this, I know I forgot to add 'Introduction Point') - Sometimes different phrases are used throughout the applications. Sometimes it's called an 'exit node' other times it's an 'exit relay'. It would benefit consistency when devs use the glossary for common terms. - Translators sometimes don't know the definition of words, or the context changes. A directory in the Tor context is different from a directory in an OS context. I'm not sure what the best way is to go about doing this, but we need definitions. The last version of the list can be found in my github: https://github.com/Shondoit/torglossary If you have additions to the list, definitions, suggestions, you can either email or clone/fork the repo and add it yourself. Thanks in advance

1 0

Re: [tor-dev] Regarding Metrics project
by Karsten Loesing 17 Nov '11

17 Nov '11

Hi Qiang, I'm cc'ing tor-dev, because I think your original request to work on Java codebases went there. On 11/17/11 2:13 AM, Qiang Wang wrote: > This is Qiang, and I am very interested in Tor project, and want to > contribute something. I was wondering if I can provide any help about > metrics project? > > Could you please let me know your recommendations? Sure thing. Three ideas come to mind: - Improve the consensus-health script: We have a Java program that downloads the network statuses from all eight directory authorities and compares them to each other to detect problems. One of the outputs is a web page [0], another output is a status email [1], and a third is a message sent to an IRC bot [2]. The code [3] needs some love. - Answer the question what fraction of exit relays exit from a different IP than is in their descriptor. This is a typical question that can be answered using the metrics data [4] we have. We'd want to publish the analysis code in the metrics-tasks repository [5], so that others can reproduce the results or refine the analysis. There's already a lot of Java code in that repository, because that's what I use when analyzing metrics data. We have plenty of other analysis questions similar to this one, so if you don't like this one, take a look at the Analysis component in our bug tracker [6]. - Implement an efficient relay-search database. We have a web site for searching relays by IP, nickname, or fingerprint [7], but it's really slow. The main problem is that the database originally was designed for aggregating statistics about relays, not for searching relays. I have two ideas here: The first is to design a separate PostgreSQL database [8] and go crazy with indexes, the second is to try out CouchDB for this [9]. This task isn't really that Java-specific, except for the fact that I'm using Java to import data and send queries. If anything sounds interesting to you, please let me know. I have more information about these tasks and can help you get started. Also look at the research section of the metrics website [10] to learn more. Best, Karsten [0] https://metrics.torproject.org/consensus-health.html [1] https://lists.torproject.org/pipermail/tor-consensus-health/2011-November/0… [2] See the IRC bot nsa in #tor-bots on OFTC, e.g., "< nsa> or: [consensus-health] The following directory authorities set conflicting or invalid consensus parameters: ides bwauthbestratio=1 bwauthcircs=0 bwauthdescbw=1 bwauthkp=10000 bwauthpid=1 bwauthtd=0 bwauthti=0 bwauthtidecay=5000 cbtnummodes=3 refuseunknownexits=1" [3] https://gitweb.torproject.org/metrics-web.git/tree/HEAD:/src/org/torproject… [4] https://metrics.torproject.org/data.html [5] https://gitweb.torproject.org/metrics-tasks.git [6] https://trac.torproject.org/projects/tor/query?status=!closed&component=Ana… [7] https://metrics.torproject.org/relay-search.html [8] https://trac.torproject.org/projects/tor/ticket/2922 [9] https://trac.torproject.org/projects/tor/ticket/4440 [10] https://metrics.torproject.org/research.html

1 0

About JTor project
by Qiang Wang 17 Nov '11

17 Nov '11

Hello there, I am very interested in Tor project, and I want to contribute something to it. In particular, I'd like to work on JTor. Could anyone tell me whom I can contact with for further information regarding getting involved in? -- Thanks & best regards Qiang Wang

2 2

What Should Tor Bridges and Clients Do When They Get Hosed?
by George Kadianakis 12 Nov '11

12 Nov '11

This is not a proposal. So, since some people in this list, who are not in IRC, seemed interested in censorship resistance, I pose this new question: What should Tor bridges and clients do when they hosed in previous schemes? All those previous proposals dealed with ways to detect fingerprinting adversaries, but not with ways to persuade them that what they are seeing is not Tor. There should be a proposal specifying exactly that, and this is a discussion thread (even monologue can be considered a discussion). Let's start with a quick threat model: --- We consider a nation-state adversary who: a) has a list of hosts suspected to be Tor bridges and wants to block them without suffering huge false positives. OR b) sails the Internet to find Tor bridges. The adversary is controlling the state's Internet infrastructure and can inject, modify or block incoming or outgoing traffic. He can build software that speaks and understands network protocols but he cannot deploy humans to do the necessary testing. To complete his goals, the adversary can directly probe hosts to see what protocol they speak, OR he can MITM all connections towards those hosts to see what protocol their clients speak. --- tl;dr we are dealing with MITMs and direct probes here. Naturally, it would be cool if we could just say "Oh right, all bridges and clients speak HTTPS in case of detection; all is cool", but remember that HTTPS is quite hard to spoof right, both in the server and in the client case [1], especially when it has to be done in C and in the tor.git codebase. Ideally, we would want to be as application-layer-neutral [2] as possible, and with the minimum implementation cost possible. So let's handle client and bridge cases separately and have a go. Clients: Proposal 191 allows clients to detect MITM attempts during the SSL handshake, this is good because we don't have to pretend speaking a specific application layer protocol. I think a good solution would be to send an SSL Alert of "Bad Certificate" type and close the connection when clients detect an SSL MITM attempt. That's what Firefox does when a user selects "Get Me Out Of Here!" on the self-signed certificate dialog, and that's what all applications should do (?) when an MITM attack is detected either by certificate or public-key pinning. In the future we might also want to send innocuous HTTP queries like GETting index.html, but we have to be aware of [1]. Bridges: Bridges have it harder. Proposal 187, allows bridges to detect probing attempts after the SSL handshake, when the probing client sends any sort of packet or a badly formed AUTHORIZE cell. This forces bridges to respond with an application-layer-specific message (or close the connection, but that would be rude). This is where I'm not sure what should be done. I think all implementations MUST provide a way for the bridge operator to configure a network service of his own (FTPS, HTTPS, SMTPS, etc.) and have the bridge send all the adversary traffic to that service. That should do it for technical bridge operators, but there probably should also be something on by default, for bridge operators who don't know how to setup net services on their own. The easy choice is an "HTTPS" server with the default Apache "It Works!", or a closed basic access authentication, but really implementing a spoofed HTTPS server in tor will be a PITA, because censors can easily test us by provoking one of [0] (there is a reason that HTTP servers usually require lots of LoCs to work). Maybe we should ship a configured Apache server with the long-term future "Anti-censorship Tor Bundle"? Also, what happens to Tor on Linux when it can't listen on port 443? Or when port 443 is already taken? HTTPS servers on 9001 sure look sketchy. Any ideas are welcome. Any services widely used, frequently seen with SSL support, that handle traffic that kinda looks like Tor's and are easily implementable, are also welcome. [0]: https://en.wikipedia.org/wiki/List_of_HTTP_status_codes [1]: https://en.wikipedia.org/wiki/List_of_HTTP_header_fields https://trac.torproject.org/projects/tor/ticket/2907 [2]: Let's place SSL on the transport layer for now.

2 1

[Patch] test_util.c
by Gisle Vanem 12 Nov '11

12 Nov '11

Those '{}' constructs are not well liked by MSVC (cl v.16.xx). An easy fix: diff --git a/src/test/test_util.c b/src/test/test_util.c index 2146299..64bf52e 100644 --- a/src/test/test_util.c +++ b/src/test/test_util.c @@ -1597,7 +1597,7 @@ test_util_join_win_cmdline(void *ptr) {"a\\\\\\b", "de fg", "H", NULL}, // Backslashes {"a\\\"b", "\\c", "D\\", NULL}, // Backslashes before quote {"a\\\\b c", "d", "E", NULL}, // Backslashes not before quote - {} // Terminator + { NULL } // Terminator }; const char *cmdlines[] = { @@ -1649,7 +1649,7 @@ test_util_split_lines(void *ptr) {"\n\rfoo\n\rbar\r\n", 12, {"foo", "bar", NULL}}, {"fo o\r\nb\tar", 10, {"fo o", "b.ar", NULL}}, {"\x0f""f\0o\0\n\x01""b\0r\0\r", 12, {".f.o.", ".b.r.", NULL}}, - {NULL, 0, {}} + {NULL, 0, { NULL }} }; int i, j; --gv

2 1