- tor-commits - lists.torproject.org

r25679: {website} link to latest 2 news articles. (website/trunk/press/en)
by Andrew Lewman 18 Jun '12

18 Jun '12

Author: phobos Date: 2012-06-18 18:12:14 +0000 (Mon, 18 Jun 2012) New Revision: 25679 Modified: website/trunk/press/en/inthemedia.wml Log: link to latest 2 news articles. Modified: website/trunk/press/en/inthemedia.wml =================================================================== --- website/trunk/press/en/inthemedia.wml 2012-06-18 14:38:45 UTC (rev 25678) +++ website/trunk/press/en/inthemedia.wml 2012-06-18 18:12:14 UTC (rev 25679) @@ -33,6 +33,16 @@ </tr> </thead> <tr style="background-color: #e5e5e5;"> +<td>2012 Jun 18</td> +<td>Poynter</td> +<td><a href="http://www.poynter.org/latest-news/top-stories/177642/news-challenge-funds-…">News Challenge funds 6 projects focused on networks</td> +</tr> +<tr> +<td>2012 Jun 18</td> +<td>Knight Foundation</td> +<td><a href="http://www.knightfoundation.org/grants/20121802/">Knight Foundation funds project to protect journalists and their sources worldwide by providing them with a toolkit to preserve their anonymity online.</td> +</tr> +<tr style="background-color: #e5e5e5;"> <td>2012 Apr 30</td> <td>Forbes</td> <td><a href="http://www.forbes.com/sites/andygreenberg/2012/04/30/the-tor-projects-new-t…">The Tor Project's New Tool Aims To Map Out Internet Censorship</a></td>

1 0

[onionoo/master] Add a DESIGN document.
by karsten＠torproject.org 18 Jun '12

18 Jun '12

commit 77fc6e110c1c57b28ad36a0ccd00c61a96c034ff Author: Karsten Loesing <karsten.loesing(a)gmx.net> Date: Mon Jun 18 18:57:12 2012 +0200 Add a DESIGN document. --- DESIGN | 157 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 files changed, 157 insertions(+), 0 deletions(-) diff --git a/DESIGN b/DESIGN new file mode 100644 index 0000000..6afbdb3 --- /dev/null +++ b/DESIGN @@ -0,0 +1,157 @@ +Onionoo design document +======================= + +This short document describes Onionoo's design in a mostly informal and +language-independent way. The goal is to be able to discuss design +decisions with non-Java programmers and to provide a blueprint for porting +Onionoo to other programming languages. This document cannot describe all +the details, but it can provide a rough overview. + +There are two main building blocks of Onionoo that are described here: + + 1) an hourly cronjob processing newly published Tor descriptors and + + 2) a web service component answering client requests. + +The interface between the two building blocks is a directory in the local +file system that can be read and written by component 1 and can be read by +component 2. In theory, the two components can be implemented in two +entirely different programming languages. In a possible port from Java to +another programming language, the two components can easily be ported +subsequently. + +The purpose of the hourly batch processor is to read updated Tor +descriptors from the metrics service and transform them to be read by the +web service component. Answering a client request in component 2 of +Onionoo needs to be highly efficient which is why any data aggregation +needs to happen beforehand. Parsing descriptors on-the-fly is not an +option. + +The hourly batch processor is run in a cron job at :15 every hour that +usually takes up to five minutes and that contains the following substeps: + + 1.1) Rsync new Tor descriptors from metrics. + + 1.2) Read previously stored status data about relays and bridges that + have been running in the last seven days to memory. These data + include for each relay or bridge: nickname, fingerprint, primary + OR address and port, additional OR addresses and ports, exit + addresses, network status publication time, directory port, relay + flags, consensus weight, country code, host name as obtained by + reverse domain name lookup, and timestamp of last reverse domain + name lookup. + + 1.3) Import any new relay network status consensuses that have been + published since the last run. + + 1.4) Set the running bit for all relays that are contained in the last + known relay network status consensus. + + 1.5) Look up all relay IP addresses in a local GeoIP database and in a + local AS number database. Extract country codes and names, city + names, geo coordinates, AS name and number, etc. + + 1.6) Import any new bridge network statuses that have been published + since the last run. + + 1.7) Start reverse domain name lookups for all relay IP addresses. Run + in background, only refresh lookups for previously looked up IP + address every 12 hours, run up to five lookups in parallel, and + set timeouts for single requests and for the general lookup + process. In theory, this step could happen a few steps before, + but not before step 1.3. + + 1.8) Import any new relay server descriptors that have been published + since the last run. + + 1.9) Import any new exit lists that have been published since the last + run. + + 1.10) Import any new bridge server descriptors that have been published + since the last run. + + 1.11) Import any new bridge pool assignments that have been published + since the last run. + + 1.12) Make sure that reverse domain name lookups are finished or the + timeout for running lookups has expired. This step cannot happen + at any time later than step 1.13 and shouldn't happen long before. + + 1.13) Rewrite all details files that have changed. Details files + combine information from all previously imported descriptory + types, database lookups, and performed reverse domain name + lookups. The web service component needs to be able to retrieve a + details file for a given relay or bridge without grabbing + information from different data sources. It's best to write the + details file part for a give relay or bridge to a single file in + the target JSON format, saved under the relay's or bridge's + fingerprint. If a database is used, the raw string should be + saved for faster processing. + + 1.14) Import relays' and bridges' bandwidth histories from extra-info + descriptors that have been published since the last run. There + must be internally stored bandwidth histories for each relay and + bridge, regardless of whether they have been running in the last + seven days. The original bandwidth histories, which are available + on 15-minute detail, can be aggregated to longer time periods the + farther the interval lies in the past. The interal bandwidth + histories are different from the bandwidth files described in 1.15 + which are written to be given out to clients. + + 1.15) Rewrite bandwidth files that have changed. Bandwidth files + aggregate bandwidth history information on varying levels of + detail, depending on how far observations lie in the past. It's + inevitable to write JSON-formatted bandwidth files for all relays + and bridges in the hourly cronjob. Any attempts to process years + of bandwidth data while answering a web request can only fail. + The previously aggregated bandwidth files are stored under the + relay's or bridge's fingerprint for quick lookup. + + 1.16) Update the summary file listing all relays and bridges that have + been running in the last seven days which was previously read in + step 1.2. This is the last step in the hourly process. The web + service component checks the modification time of this file to + decide whether it needs to reload its view on the network. If + this step was not the last step, the web service component might + list relays or bridges for which there are no details or bandwidth + files available yet. (With the approach taken here, it's + conveivable that a bandwidth file of a relay or bridge that hasn't + been running for a week has been deleted before step 1.16. This + case has been found acceptable, because it's highly unlikely. If + a database would have been used, steps 1.2 to 1.16 would have + happened in a single database transaction.) + +The web service component has the purpose of answering client requests. +It uses previously prepared data from the hourly cronjob to respond to +requests very quickly. + +During initialization, or whenever the hourly cronjob has finished, the +web service component does the following substeps: + + 2.1) Read the summary file that was produced by the hourly cronjob in + step 1.16. + + 2.2) Keep the list of relays and bridges in memory, including all + information that is used for filtering or sorting results. + + 2.3) Prepare summary lines for all relays and bridges. The summary + resource is a JSON file with a single line per relay or bridge. + This line contains only very few fields as compared to details + files that a client might use for further filtering results. + +When responding to a request, the web service component does the following +steps: + + 2.4) Parse request and its parameters. + + 2.5) Possibly filter relays and bridges. + + 2.6) Possibly re-order and limit results. + + 2.7) Write response or error code. + +Again, (and this can hardly be overstated!) steps 2.4 to 2.7 need to +happen *extremely* fast. Any steps that go beyond file system reads or +simple database lookups need to happen either in the hourly cronjob (1.1 +to 1.16) or in the web service component initialization (2.1 to 2.3). +

1 0

[torspec/master] Merge proposal 198 into tor-spec.txt
by nickm＠torproject.org 18 Jun '12

18 Jun '12

commit 1c30043671507c521cfa6150c554d7028e1daf96 Author: Nick Mathewson <nickm(a)torproject.org> Date: Fri Jun 15 17:07:18 2012 -0400 Merge proposal 198 into tor-spec.txt The client side is implemented in 0.2.3.17-beta; technically, we are in compliance with the server side. --- tor-spec.txt | 67 ++++++++++++++++++++++++++++++++++++++++++++++++++------- 1 files changed, 58 insertions(+), 9 deletions(-) diff --git a/tor-spec.txt b/tor-spec.txt index 28a51c9..86aa49f 100644 --- a/tor-spec.txt +++ b/tor-spec.txt @@ -189,10 +189,8 @@ see tor-design.pdf. party sending a two-certificate chain as in "certificates up-front". The initiator's ClientHello MUST include at least one ciphersuite not in the list above -- that's how the initiator indicates that it can - handle this handshake. The responder SHOULD NOT select any - ciphersuite besides those in the list above. - [The above "should not" is because some of the ciphers that - clients list may be fake.] + handle this handshake. For other considerations on the initiator's + ClientHello, see section 2.1 below. In "in-protocol" (a.k.a. "the v3 handshake"), the initiator sends no certificates, and the @@ -249,11 +247,6 @@ see tor-design.pdf. initiator SHOULD choose a list of ciphersuites and TLS extensions to mimic one used by a popular web browser. - Responders MUST NOT select any TLS ciphersuite that lacks ephemeral keys, - or whose symmetric keys are less then KEY_LEN bits, or whose digests are - less than HASH_LEN bits. Responders SHOULD NOT select any SSLv3 - ciphersuite other than those listed above. - Even though the connection protocol is identical, we will think of the initiator as either an onion router (OR) if it is willing to relay traffic for other Tor users, or an onion proxy (OP) if it only handles @@ -299,6 +292,62 @@ see tor-design.pdf. their IP address changes. Clients MAY send certificates using any of the above handshake variants. +2.1. Picking TLS ciphersuites + + Clients SHOULD send a ciphersuite list chosen to emulate some popular + web browser or other program common on the internet. Clients may send + the "Fixed Cipheruite List" below. If they do not, they MUST NOT + advertise any ciphersuite that they cannot actually support, unless that + cipher is one not supported by OpenSSL 1.0.1. + + The fixed ciphersuite list is: + TLS1_ECDHE_ECDSA_WITH_AES_256_CBC_SHA + TLS1_ECDHE_RSA_WITH_AES_256_CBC_SHA + TLS1_DHE_RSA_WITH_AES_256_SHA + TLS1_DHE_DSS_WITH_AES_256_SHA + TLS1_ECDH_RSA_WITH_AES_256_CBC_SHA + TLS1_ECDH_ECDSA_WITH_AES_256_CBC_SHA + TLS1_RSA_WITH_AES_256_SHA + TLS1_ECDHE_ECDSA_WITH_RC4_128_SHA + TLS1_ECDHE_ECDSA_WITH_AES_128_CBC_SHA + TLS1_ECDHE_RSA_WITH_RC4_128_SHA + TLS1_ECDHE_RSA_WITH_AES_128_CBC_SHA + TLS1_DHE_RSA_WITH_AES_128_SHA + TLS1_DHE_DSS_WITH_AES_128_SHA + TLS1_ECDH_RSA_WITH_RC4_128_SHA + TLS1_ECDH_RSA_WITH_AES_128_CBC_SHA + TLS1_ECDH_ECDSA_WITH_RC4_128_SHA + TLS1_ECDH_ECDSA_WITH_AES_128_CBC_SHA + SSL3_RSA_RC4_128_MD5 + SSL3_RSA_RC4_128_SHA + TLS1_RSA_WITH_AES_128_SHA + TLS1_ECDHE_ECDSA_WITH_DES_192_CBC3_SHA + TLS1_ECDHE_RSA_WITH_DES_192_CBC3_SHA + SSL3_EDH_RSA_DES_192_CBC3_SHA + SSL3_EDH_DSS_DES_192_CBC3_SHA + TLS1_ECDH_RSA_WITH_DES_192_CBC3_SHA + TLS1_ECDH_ECDSA_WITH_DES_192_CBC3_SHA + SSL3_RSA_FIPS_WITH_3DES_EDE_CBC_SHA + SSL3_RSA_DES_192_CBC3_SHA + [*] The "extended renegotiation is supported" ciphersuite, 0x00ff, is + not counted when checking the list of ciphersuites. + + If the client sends the Fixed Ciphersuite List, the responder MUST NOT + select any ciphersuite besides TLS_DHE_RSA_WITH_AES_256_CBC_SHA, + TLS_DHE_RSA_WITH_AES_128_CBC_SHA, SSL_DHE_RSA_WITH_3DES_EDE_CBC_SHA, and + SSL_DHE_DSS_WITH_3DES_EDE_CBC_SHA: such ciphers might not actually be + supported by the client. + + If the client sends a v2+ ClientHello with a list of ciphers other then + the Fixed Ciphersuite List, the responder can trust that the client + supports every cipher advertised in that list, so long as that ciphersuite + is also supported by OpenSSL 1.0.1. + + Responders MUST NOT select any TLS ciphersuite that lacks ephemeral keys, + or whose symmetric keys are less then KEY_LEN bits, or whose digests are + less than HASH_LEN bits. Responders SHOULD NOT select any SSLv3 + ciphersuite other than the DHE+3DES suites listed above. + 3. Cell Packet format The basic unit of communication for onion routers and onion

1 0

[tor/master] Clarify some messages about publishing hidden service descriptors
by nickm＠torproject.org 18 Jun '12

18 Jun '12

commit 87409771c45abdd8bcf82c84a6ab7fccc5c17ba7 Author: Nick Mathewson <nickm(a)torproject.org> Date: Fri Jun 15 15:25:46 2012 -0400 Clarify some messages about publishing hidden service descriptors Fix for bug 3311. --- changes/bug3311 | 5 +++++ src/or/rendservice.c | 6 +++--- 2 files changed, 8 insertions(+), 3 deletions(-) diff --git a/changes/bug3311 b/changes/bug3311 new file mode 100644 index 0000000..014d18d --- /dev/null +++ b/changes/bug3311 @@ -0,0 +1,5 @@ + o Minor bugfixes: + - Replace "Sending publish request" log messages with "Launching + upload", so that they no longer confusingly imply that we're + sending somtheing to a directory we might not even be connected + to yet. Fixes bug 3311; bugfix on 0.2.0.10-alpha. diff --git a/src/or/rendservice.c b/src/or/rendservice.c index 5ffc586..e88d2b3 100644 --- a/src/or/rendservice.c +++ b/src/or/rendservice.c @@ -1918,7 +1918,7 @@ directory_post_to_hs_dir(rend_service_descriptor_t *renddesc, continue; node = node_get_by_id(hs_dir->identity_digest); if (!node || !node_has_descriptor(node)) { - log_info(LD_REND, "Not sending publish request for v2 descriptor to " + log_info(LD_REND, "Not launching upload for for v2 descriptor to " "hidden service directory %s; we don't have its " "router descriptor. Queuing for later upload.", safe_str_client(routerstatus_describe(hs_dir))); @@ -1934,7 +1934,7 @@ directory_post_to_hs_dir(rend_service_descriptor_t *renddesc, base32_encode(desc_id_base32, sizeof(desc_id_base32), desc->desc_id, DIGEST_LEN); hs_dir_ip = tor_dup_ip(hs_dir->addr); - log_info(LD_REND, "Sending publish request for v2 descriptor for " + log_info(LD_REND, "Launching upload for v2 descriptor for " "service '%s' with descriptor ID '%s' with validity " "of %d seconds to hidden service directory '%s' on " "%s:%d.", @@ -2032,7 +2032,7 @@ upload_service_descriptor(rend_service_t *service) } /* Post the current descriptors to the hidden service directories. */ rend_get_service_id(service->desc->pk, serviceid); - log_info(LD_REND, "Sending publish request for hidden service %s", + log_info(LD_REND, "Launching upload for hidden service %s", serviceid); directory_post_to_hs_dir(service->desc, descs, serviceid, seconds_valid);

1 0

[tor/master] Merge remote-tracking branch 'public/bug3311'
by nickm＠torproject.org 18 Jun '12

18 Jun '12

commit e6782b355ae77455aaff3af9fe3831c07312a5c8 Merge: 72acdfe 8740977 Author: Nick Mathewson <nickm(a)torproject.org> Date: Mon Jun 18 12:07:39 2012 -0400 Merge remote-tracking branch 'public/bug3311' changes/bug3311 | 5 +++++ src/or/rendservice.c | 6 +++--- 2 files changed, 8 insertions(+), 3 deletions(-)

1 0

[tor/master] Merge branch 'bug4748_squashed'
by nickm＠torproject.org 18 Jun '12

18 Jun '12

commit 72acdfe13465939150cb61943b23495ec85e9441 Merge: 4cbd6e4 f5e86bc Author: Nick Mathewson <nickm(a)torproject.org> Date: Mon Jun 18 12:01:56 2012 -0400 Merge branch 'bug4748_squashed' changes/bug4748 | 4 +++ doc/tor.1.txt | 65 +++++++++++++++++++++++++++++++++++++++++++++--------- 2 files changed, 58 insertions(+), 11 deletions(-)

1 0

[tor/master] Document 0.2.3.x torrc/default-torrc/command line semantics changes
by nickm＠torproject.org 18 Jun '12

18 Jun '12

commit f5e86bcd6c06d43ff3af5acd8135bd8b577bc3fd Author: Nick Mathewson <nickm(a)torproject.org> Date: Fri Jun 15 16:41:30 2012 -0400 Document 0.2.3.x torrc/default-torrc/command line semantics changes Bug 4748 squash! Document 0.2.3.x torrc/default-torrc/command line semantics changes Incorporates fixes suggested by rransom. --- changes/bug4748 | 4 +++ doc/tor.1.txt | 65 +++++++++++++++++++++++++++++++++++++++++++++--------- 2 files changed, 58 insertions(+), 11 deletions(-) diff --git a/changes/bug4748 b/changes/bug4748 new file mode 100644 index 0000000..9c0f9cf --- /dev/null +++ b/changes/bug4748 @@ -0,0 +1,4 @@ + o Documentation: + - Document the --defaults-torrc option, and the new (in 0.2.3) + semantics for overriding, extending, and clearing lists of + options. Closes bug 4748. diff --git a/doc/tor.1.txt b/doc/tor.1.txt index feb7215..a4a0ed7 100644 --- a/doc/tor.1.txt +++ b/doc/tor.1.txt @@ -28,13 +28,21 @@ Users bounce their TCP streams -- web traffic, ftp, ssh, etc -- around the routers, and recipients, observers, and even the routers themselves have difficulty tracking the source of the stream. -OPTIONS -------- +COMMAND-LINE OPTIONS +-------------------- **-h**, **-help**:: Display a short help message and exit. **-f** __FILE__:: - FILE contains further "option value" pairs. (Default: @CONFDIR@/torrc) + Specify a new configuration file to contain further Tor configuration + options. (Default: $HOME/.torrc, or @CONFDIR@/torrc if that file is not + found) + +**--defaults-torrc** __FILE__:: + Specify a file in which to find default values for Tor options. The + contents of this file are overridden by those in the regular + configuration file, and by those on the command line. (Default: + @CONFDIR@/torrc-defaults.) **--hash-password**:: Generates a hashed password for control port access. @@ -70,14 +78,49 @@ OPTIONS which tells Tor to only send warnings and errors to the console, or with the **--quiet** option, which tells Tor not to log to the console at all. -Other options can be specified either on the command-line (--option - value), or in the configuration file (option value or option "value"). - Options are case-insensitive. C-style escaped characters are allowed inside - quoted values. Options on the command line take precedence over - options found in the configuration file, except indicated otherwise. To - split one configuration entry into multiple lines, use a single \ before - the end of the line. Comments can be used in such multiline entries, but - they must start at the beginning of a line. +Other options can be specified on the command-line in the format "--option +value", in the format "option value", or in a configuration file. For +instance, you can tell Tor to start listening for SOCKS connections on port +9999 by passing --SOCKSPort 9999 or SOCKPort 9999 to it on the command line, +or by putting "SOCKSPort 9999" in the configuration file. You will need to +quote options with spaces in them: if you want Tor to log all debugging +messages to debug.log, you will probably need to say --Log 'debug file +debug.log'. + +Options on the command line override those in configuration files. See the +next section for more information. + +THE CONFIGURATION FILE FORMAT +----------------------------- + +All configuration options in a configuration are written on a single line by +default. They take the form of an option name and a value, or an option name +and a quoted value (option value or option "value"). Anything after a # +character is treated as a comment. Options are +case-insensitive. C-style escaped characters are allowed inside quoted +values. To split one configuration entry into multiple lines, use a single +backslash character (\) before the end of the line. Comments can be used in +such multiline entries, but they must start at the beginning of a line. + +By default, an option on the command line overrides an option found in the +configuration file, and an option in a configuration file overrides one in +the defaults file. + +This rule is simple for options that take a single value, but it can become +complicated for options that are allowed to occur more than once: if you +specify four SOCKSPorts in your configuration file, and one more SOCKSPort on +the command line, the option on the command line will replace __all__ of the +SOCKSPorts in the configuration file. If this isn't what you want, prefix +the option name with a plus sign, and it will be appended to the previous set +of options instead. + +Alternatively, you might want to remove every instance of an option in the +configuration file, and not replace it at all: you might want to say on the +command line that you want no SOCKSPorts at all. To do that, prefix the +option name with a forward slash. + +GENERAL OPTIONS +--------------- **BandwidthRate** __N__ **bytes**|**KB**|**MB**|**GB**:: A token bucket limits the average incoming bandwidth usage on this node to

1 0

[tor/master] More sophisticated attempt at detecting working linker options
by nickm＠torproject.org 18 Jun '12

18 Jun '12

commit adbdeafad9a85226ef9d232e85360bea54f78a73 Author: Nick Mathewson <nickm(a)torproject.org> Date: Fri Jun 15 21:04:07 2012 -0400 More sophisticated attempt at detecting working linker options On some platforms, the linker is perfectly happy to produce binaries that won't run if you give it the wrong set of flags. So when not cross-compiling, try to link-and-run a little test program, rather than just linking it. Possible fix for 6173. --- acinclude.m4 | 9 +++++++-- changes/bug6173 | 5 +++++ 2 files changed, 12 insertions(+), 2 deletions(-) diff --git a/acinclude.m4 b/acinclude.m4 index 4328059..ae14411 100644 --- a/acinclude.m4 +++ b/acinclude.m4 @@ -71,9 +71,14 @@ AC_DEFUN([TOR_CHECK_LDFLAGS], [ CFLAGS="$CFLAGS -pedantic -Werror" LDFLAGS="$LDFLAGS $2 $1" LIBS="$LIBS $3" - AC_TRY_LINK([], [return 0;], + AC_RUN_IFELSE([ +#include <stdio.h> +int main(int argc, char **argv) { fputs("", stdout); return 0; }], [AS_VAR_SET(VAR,yes)], - [AS_VAR_SET(VAR,no)]) + [AS_VAR_SET(VAR,no)], + [AC_TRY_LINK([], [return 0;], + [AS_VAR_SET(VAR,yes)], + [AS_VAR_SET(VAR,no)])]) CFLAGS="$tor_saved_CFLAGS" LDFLAGS="$tor_saved_LDFLAGS" LIBS="$tor_saved_LIBS" diff --git a/changes/bug6173 b/changes/bug6173 new file mode 100644 index 0000000..3b467a9 --- /dev/null +++ b/changes/bug6173 @@ -0,0 +1,5 @@ + o Major bugfixes: + - Make our linker option detection code more robust against linkers + where a bad combination of options completes successfully but + makes an unrunnable binary. Fixes bug 6173; bugfix on 0.2.3.17-beta. +

1 0

[tor/master] Merge branch 'bug6173_rebased'
by nickm＠torproject.org 18 Jun '12

18 Jun '12

commit 4cbd6e46ef6d64e293dd03b0c453a2750684563a Merge: 4432fa4 8c01581 Author: Nick Mathewson <nickm(a)torproject.org> Date: Mon Jun 18 11:52:03 2012 -0400 Merge branch 'bug6173_rebased' acinclude.m4 | 9 ++++++--- changes/bug6173 | 5 +++++ 2 files changed, 11 insertions(+), 3 deletions(-)

1 0

[tor/master] Merge remote-tracking branch 'andrea/bug6028'
by nickm＠torproject.org 18 Jun '12

18 Jun '12

commit 4432fa40dda2a79da7c2d88d2fe83462ed919d7b Merge: 26855fe 1f7f10e Author: Nick Mathewson <nickm(a)torproject.org> Date: Mon Jun 18 11:51:55 2012 -0400 Merge remote-tracking branch 'andrea/bug6028' changes/bug6028 | 4 ++++ src/or/connection.c | 6 +++++- 2 files changed, 9 insertions(+), 1 deletions(-)

1 0