- tor-commits - lists.torproject.org

[tor/maint-0.2.1] Avoid crash in any_pending_bridge_descriptor_fetches
by nickm＠torproject.org 08 Mar '11

08 Mar '11

commit 9a6df215395750286383eed77f3c49d2b3ef34df Author: Nick Mathewson <nickm(a)torproject.org> Date: Tue Mar 8 15:17:41 2011 -0500 Avoid crash in any_pending_bridge_descriptor_fetches This is based on shitlei's fix for bug2629, with extra parens removed. Fixes bug 2629, bugfix on 0.2.1.2-alpha. --- changes/bug2629 | 5 +++++ src/or/circuitbuild.c | 3 ++- 2 files changed, 7 insertions(+), 1 deletions(-) diff --git a/changes/bug2629 b/changes/bug2629 new file mode 100644 index 0000000..87817cf --- /dev/null +++ b/changes/bug2629 @@ -0,0 +1,5 @@ + o Minor bugfixes + - Fix a crash bug that could occur occasionally when a client was + configured with a large number of bridges. Fixes bug 2629; bugfix + on 0.2.1.2-alpha. Bugfix by trac user "shitlei". + diff --git a/src/or/circuitbuild.c b/src/or/circuitbuild.c index 065eb05..76713e6 100644 --- a/src/or/circuitbuild.c +++ b/src/or/circuitbuild.c @@ -3200,7 +3200,8 @@ any_pending_bridge_descriptor_fetches(void) conn->purpose == DIR_PURPOSE_FETCH_SERVERDESC && TO_DIR_CONN(conn)->router_purpose == ROUTER_PURPOSE_BRIDGE && !conn->marked_for_close && - conn->linked && !conn->linked_conn->marked_for_close) { + conn->linked && + conn->linked_conn && !conn->linked_conn->marked_for_close) { log_debug(LD_DIR, "found one: %s", conn->address); return 1; }

1 0

[tor/maint-0.2.2] make nickm's proposed convention from 2003 be gospel
by arma＠torproject.org 08 Mar '11

08 Mar '11

commit 95edd51116c17327f314bc132a0a5f86499e410a Author: Roger Dingledine <arma(a)torproject.org> Date: Tue Mar 8 14:59:30 2011 -0500 make nickm's proposed convention from 2003 be gospel --- doc/HACKING | 10 +++++----- 1 files changed, 5 insertions(+), 5 deletions(-) diff --git a/doc/HACKING b/doc/HACKING index 486fe6d..bdb86c0 100644 --- a/doc/HACKING +++ b/doc/HACKING @@ -312,11 +312,11 @@ operation. If a library function is currently called such that failure always means ERR, then the library function should log WARN and let the caller log ERR. -[XXX Proposed convention: every message of severity INFO or higher should -either (A) be intelligible to end-users who don't know the Tor source; or (B) -somehow inform the end-users that they aren't expected to understand the -message (perhaps with a string like "internal error"). Option (A) is to be -preferred to option (B). -NM] +Every message of severity INFO or higher should either (A) be intelligible +to end-users who don't know the Tor source; or (B) somehow inform the +end-users that they aren't expected to understand the message (perhaps +with a string like "internal error"). Option (A) is to be preferred to +option (B). Doxygen ~~~~~~~~

1 0

[metrics-web/master] Warn when bandwidth scanners are failing.
by karsten＠torproject.org 08 Mar '11

08 Mar '11

commit c3567ba120dd3c09c45de2f74a385cde621cfdab Author: Karsten Loesing <karsten.loesing(a)gmx.net> Date: Tue Mar 8 15:16:36 2011 +0100 Warn when bandwidth scanners are failing. --- .../ernie/cron/ConsensusHealthChecker.java | 25 ++++++++++++++++++++ 1 files changed, 25 insertions(+), 0 deletions(-) diff --git a/src/org/torproject/ernie/cron/ConsensusHealthChecker.java b/src/org/torproject/ernie/cron/ConsensusHealthChecker.java index ac07754..65a8cce 100644 --- a/src/org/torproject/ernie/cron/ConsensusHealthChecker.java +++ b/src/org/torproject/ernie/cron/ConsensusHealthChecker.java @@ -105,6 +105,7 @@ public class ConsensusHealthChecker { StringBuilder authorityVersionsResults = new StringBuilder(); SortedSet<String> allKnownFlags = new TreeSet<String>(); SortedSet<String> allKnownVotes = new TreeSet<String>(); + SortedSet<String> runningBandwidthScanners = new TreeSet<String>(); SortedMap<String, String> consensusAssignedFlags = new TreeMap<String, String>(); SortedMap<String, SortedSet<String>> votesAssignedFlags = @@ -383,6 +384,7 @@ public class ConsensusHealthChecker { + " <td>" + voteContainsBandwidthWeights + " Measured values in w lines</td>\n" + " </tr>\n"); + runningBandwidthScanners.add(dirSource); } } @@ -404,6 +406,29 @@ public class ConsensusHealthChecker { + "directory authorities: " + sb.toString().substring(2)); } + /* Check if less than 4 bandwidth scanners are running. TODO make this + * configurable */ + SortedSet<String> knownBandwidthScanners = new TreeSet<String>( + Arrays.asList("ides,urras,moria1,gabelmoo".split(","))); + for (String dir : runningBandwidthScanners) { + knownBandwidthScanners.remove(dir); + } + if (!knownBandwidthScanners.isEmpty()) { + StringBuilder sb = new StringBuilder(); + for (String dir : knownBandwidthScanners) { + sb.append(", " + dir); + } + String message = "The following directory authorities are not " + + "reporting bandwidth scanner results: " + + sb.toString().substring(2); + this.logger.warning(message); + if (runningBandwidthScanners.size() >= 3) { + nagiosWarnings.add(message); + } else { + nagiosCriticals.add(message); + } + } + try { /* Keep the past two consensus health statuses. */

1 0

[torperf/master] Add scripts for filtering and visualizing Torperf data.
by karsten＠torproject.org 08 Mar '11

08 Mar '11

commit a257846fda8e01261817f9c286de43fa49c9f1c2 Author: tomb <tomb(a)torproject.org> Date: Wed Mar 2 10:39:21 2011 -0500 Add scripts for filtering and visualizing Torperf data. Implements #2563. --- README | 14 ++++- metrics/HOWTO | 109 ++++++++++++++++++++++++++++++++++++ metrics/filter.R | 149 ++++++++++++++++++++++++++++++++++++++++++++++++++ metrics/timematrix.R | 36 ++++++++++++ 4 files changed, 305 insertions(+), 3 deletions(-) diff --git a/README b/README index c4769d4..48b9617 100644 --- a/README +++ b/README @@ -8,9 +8,9 @@ Contents via SOCKS 4a/5 and outputs timing information util.c: Utility functions for trivsocks-client.c util.h: Utility function declarations for trivsocks-client.c - + Makefile: Builds and tests trivsocks-client - + [run_test.py: Script to automate running of trivsocks-client -- defect] [plot_results.R: Plot the results from run_test.py -- defect] @@ -27,5 +27,13 @@ Contents performance data and path data LICENSE: The Tor license (3-clause BSD) - README: This file + README: This file + +Subdirectory /metrics +------------ -------- + +A set of utilities for filtering and graphing Tor performance data. + filter.R: filters torperf data and prepares it for graphing + timematrix.R: graphs tordata for interpretation and visualization + HOWTO: documentation and examples diff --git a/metrics/HOWTO b/metrics/HOWTO new file mode 100644 index 0000000..ce6f7eb --- /dev/null +++ b/metrics/HOWTO @@ -0,0 +1,109 @@ +HOWTO -- How to generate nifty graphs of tor performance + +Welcome traveler! You have reached the howto for some tor performance +and metrics stuff. You will find here some techniques and scripts +developed during several tasks including: +#1919; in which we examine torperfs with fixed entry guards +#2543; in which we create graphs of #1919 data +#2563; in which we generalize techniques from #2543 for the future + +The remainder of this HOWTO will walk you through what you need to do +to use the generalized techniques to generate graphs from performance +data. We will use #2543 as an example, because it is from this +example that the generalized technique was derived. This is intended +to be a living document. If something is unclear, or if you wish to +request a feature, please open a ticket: +https://trac.torproject.org/projects/tor/newticket + +As far as I know, this document was written by Karsten, Mike Perry, +and Tom Benjamin. If you are also an author of this document, please +add yourself to this list. + +Step 1: Download Torperf request files +-------------------------------------- + +The 15 Torperf request files are available here: + + https://metrics.torproject.org/data.html#performance + +The wget commands to download all of them are: + + wget https://metrics.torproject.org/data/torperf-50kb.data + wget https://metrics.torproject.org/data/torperf-1mb.data + wget https://metrics.torproject.org/data/torperf-5mb.data + wget https://metrics.torproject.org/data/torperffastratio-50kb.data + wget https://metrics.torproject.org/data/torperffastratio-1mb.data + wget https://metrics.torproject.org/data/torperffastratio-5mb.data + wget https://metrics.torproject.org/data/torperffast-50kb.data + wget https://metrics.torproject.org/data/torperffast-1mb.data + wget https://metrics.torproject.org/data/torperffast-5mb.data + wget https://metrics.torproject.org/data/torperfslow-50kb.data + wget https://metrics.torproject.org/data/torperfslow-1mb.data + wget https://metrics.torproject.org/data/torperfslow-5mb.data + wget https://metrics.torproject.org/data/torperfslowratio-50kb.data + wget https://metrics.torproject.org/data/torperfslowratio-1mb.data + wget https://metrics.torproject.org/data/torperfslowratio-5mb.data + +Note that the torperf-*.data files are quite big already (25M+). + + +Step 2: Install R and ggplot2 +----------------------------- + +Install R 2.8 or higher. + +Run R as user and install ggplot2, quit R, start R again and try to load +ggplot2: + + $ R + > install.packages("ggplot2") + > q() # No need to save the workspace image, ever. + $ R + > library(ggplot2) + > q() + + +Step 3: Filter the data +----------------------- + +Before actually graphing the Torperf data, we should filter it to avoid +reading 29M of data for each graph. filter.R is a script that +accomplishes this task, writing it's output to filtered.csv +It is used as follows: + +1) Decide which files you are interested in. If you only want graphs +based on the fast guard nodes, you only need to crunch those files. + +2) Decide what date range you are interested in. The default is to +include all data since 2001-02-01 until 2099-12-31, by which time I +expect this script may be obsolete. + +usage: R --slave -f filter.R --args [-start=DATE] [-end=DATE] FILENAME(S) + +filename must be of the form guardname-basesizeSUFFIX.data +where SUFFIX is one of kb, mb, gb, tb + eg: R --slave -f filter.R --args -start=2011-02-01 -end=2099-12-31 *.data + eg: R --slave -f filter.R --args torperf-50kb.data + +So, to filter all data from #1919 you would execute: + $ R --slave -f filter.R --args *.data + +The script may take some time to run if the data files are large. + + +Step 4: Visualize the data +-------------------------- + +Let's start with plotting a matrix of completion times graphs for every +file size and guard selection. + + $ R --slave -f timematrix.R + +This execution may take around 15 seconds. + + +Step 5: Find a more useful visualization of the data +---------------------------------------------------- + +... TODO ... + diff --git a/metrics/filter.R b/metrics/filter.R new file mode 100644 index 0000000..f069856 --- /dev/null +++ b/metrics/filter.R @@ -0,0 +1,149 @@ +## A new and "improved" genericised version of the old filter script +## This version was created for task 2563 +## See HOWTO to put this in context +## +## usage: R -f filter.R --args [-start=DATE] [-end=DATE] FILENAME(S) +## filename must be of the form guardname-basesizeSUFFIX.data +## where SUFFIX is one of kb, mb, gb, tb +## +## eg: R -f filter.R --args -start=2011-02-01 -end=2099-12-31 *.data +## eg: R -f filter.R --args torperf-50kb.data +## +## This R script reads in Torperf files as specified on the command line +## and writes a filtered version to filtered.csv for later processing. + +FilterMain <- function(ARGV) { + kDebug <- FALSE # set TRUE for debugging output + kVersion <- 0.3 + if (kDebug) { cat("filter.R version ", kVersion, "\n\n") } + files <- NULL # files is a list of torperfFiles as definied below + setClass("torperfFile", + representation( + filename = "character", + guardLabel = "character", + filesizeLabel = "character", + filesize = "numeric" + ) + ) + + ## default values + ## cutoff dates for observations + start <- as.POSIXct("2011-02-01", origin = "1970-01-01") + end <- as.POSIXct("2099-12-31", origin = "1970-01-01") + + ## process command line arguments + args <- unlist(strsplit(ARGV, " ")) + + ## there are better ways to process command line args, but this works for me :-) + for (arg in args) { + if (kDebug) { cat('arg: ', arg, "\n") } + ## if start date specified + if (length(splitArgL <- unlist(strsplit(arg, "-start="))) == 2) { + if (kDebug) { cat('Starting from ', splitArgL[2], '\n') } + start <- as.POSIXct(splitArgL[2], origin = "1970-01-01") + next + } + ## if end date specified + if (length(splitArgL <- unlist(strsplit(arg, "-end="))) == 2) { + if (kDebug) { cat('Ending at ', splitArgL[2], '\n') } + end <- as.POSIXct(splitArgL[2], origin = "1970-01-01") + next + } + ## if the argument is -start= or -end= we will not reach this line + ## now, if it isn't a parameter add it to the file list + ## parse filename for metadata... + ## examples: + ## "torperf-50kb.data" should result in + ## filename = "torperf-50kb.data" + ## guardLabel = "torperf" + ## filesizeLabel = "50kb" + ## filesize = 50 * 1024 + my.file <- new("torperfFile", filename = arg) + + ## get base filename (strip out leading parts of filename such as dirname) + baseFilename <- basename(my.file@filename) + parseFileStr <- unlist(strsplit(baseFilename, "-")) ## split the two parts of the filename string + if (length(parseFileStr) != 2) { + cat("error: filenames must be of the form guard-filesize.data, you said \"", baseFilename, "\"\n") + quit("no", 1) + } + my.file@guardLabel <- parseFileStr[1] + cdr <- parseFileStr[2] + parseFilesize <- unlist(strsplit(cdr, "\\.")) + if (length(parseFilesize) != 2) { + cat("error: tail of filename must be filesize.data, you said \"", cdr, "\"\n") + quit("no", 1) + } + my.file@filesizeLabel <- tolower(parseFilesize[1]) ## smash case to make our life easier + + fileBaseSize <- as.integer(unlist(strsplit(my.file@filesizeLabel, "[a-z]"))[1]) + fileSizeMultiplierStr <- unlist(strsplit(my.file@filesizeLabel, '[0-9]')) + fileSizeMultiplierStr <- fileSizeMultiplierStr[length(fileSizeMultiplierStr)] + fileSizeMultiplier <- 1 ## assume no suffix + if (fileSizeMultiplierStr == "kb") { fileSizeMultiplier <- 1024 } + if (fileSizeMultiplierStr == "mb") { fileSizeMultiplier <- 1024 * 1024 } + if (fileSizeMultiplierStr == "gb") { fileSizeMultiplier <- 1024 * 1024 * 1024} + ## yeah right, like we are really pushing TB of data + if (fileSizeMultiplierStr == "tb") { fileSizeMultiplier <- 1024 * 1024 * 1024 * 1024 } + my.file@filesize <- fileBaseSize * fileSizeMultiplier + + if (kDebug) { + cat("i will read file: ", my.file@filename, ' ', + my.file@guardLabel, ' ', + my.file@filesizeLabel, ' ', + my.file@filesize, "\n") + } + + files <- c(files, my.file) + } + + ## sanity check arguments + if (start >= end) { + cat("error: start date must be before end date\n"); + quit("no", 1) + } + if (length(files) == 0) { + cat("error: input files must be specified as arguments\n") + quit("no", 1) ## terminate with non-zero errlev + } + + if (kDebug) { + cat("filtering from ", as.character.POSIXt(start), " to ", + as.character.POSIXt(end), "\n") + } + + ## Turn a given Torperf file into a data frame with the information we care + ## about. + read <- function(filename, guards, filesize, bytes) { + x <- read.table(filename) + x <- x[as.POSIXct(x$V1, origin = "1970-01-01") >= start & + as.POSIXct(x$V1, origin = "1970-01-01") <= end, ] + if (length(x$V1) == 0) + NULL + else + data.frame( + started = as.POSIXct(x$V1, origin = "1970-01-01"), + timeout = x$V17 == 0, + failure = x$V17 > 0 & x$V20 < bytes, + completemillis = ifelse(x$V17 > 0 & x$V20 >= bytes, + round((x$V17 * 1000 + x$V18 / 1000) - + (x$V1 * 1000 + x$V19 / 1000), 0), NA), + guards = guards, + filesize = filesize) + } + + ## Read in files and bind them to a single data frame. + filtered <- NULL + for (file in files) { + if (kDebug) { cat('Processing ', file@filename, "...\n") } + filtered <- rbind(filtered, + read(file@filename, file@guardLabel, file@filesizeLabel, file@filesize) + ) + } + + # Write data frame to a csv file for later processing. + write.csv(filtered, "filtered.csv", quote = FALSE, row.names = FALSE) + +} + +FilterMain(commandArgs(TRUE)) diff --git a/metrics/timematrix.R b/metrics/timematrix.R new file mode 100644 index 0000000..ec01a25 --- /dev/null +++ b/metrics/timematrix.R @@ -0,0 +1,36 @@ +# Load ggplot library without printing out stupid warnings. +options(warn = -1) +suppressPackageStartupMessages(library("ggplot2")) + +# Read in filtered data. +data <- read.csv("filtered.csv", stringsAsFactors = FALSE) + +# Remove NA's +data <- na.omit(data) + +# Remove "outliers" +data <- data[(data$filesize == "50kb" & data$completemillis < 60000) | + (data$filesize == "1mb" & data$completemillis < 120000) | + (data$filesize == "5mb" & data$completemillis < 300000), ] + +# Plot a matrix of scatter plots; the first step is to define which data +# we want to plot (here: data) and what to put on x and y axis. +ggplot(data, aes(x = as.POSIXct(started), y = completemillis / 1000)) + + +# Draw a point for every observation, but with an alpha value of 1/10 to +# reduce overplotting +geom_point(alpha = 1/10) + + +# Draw a matrix of these graphs with different filesizes and different +# guards. +facet_grid(filesize ~ guards, scales = "free_y") + + +# Rename y axis. +scale_y_continuous(name = "Completion time in seconds") + + +# Rename x axis. +scale_x_datetime(name = "Starting time") + +# Save the result to a large PNG file. +ggsave("timematrix.png", width = 10, height = 10, dpi = 150) +

1 0

[torflow/master] Submodule..
by mikeperry＠torproject.org 08 Mar '11

08 Mar '11

commit 51dcda9bcd3e4ab1370d53b1bf23dbcf7ba3a89e Author: Mike Perry <mikeperry-git(a)fscked.org> Date: Mon Mar 7 18:14:02 2011 -0800 Submodule.. --- TorCtl | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/TorCtl b/TorCtl index 6eeffeb..b99ecc1 160000 --- a/TorCtl +++ b/TorCtl @@ -1 +1 @@ -Subproject commit 6eeffeb4f20aefb2f859a9f5754c34342abc307e +Subproject commit b99ecc1de55cdefbcd803e267d2635534ff9e511

1 0

[pytorctl/master] Fix some issues with SmartSocket.
by mikeperry＠torproject.org 08 Mar '11

08 Mar '11

commit b99ecc1de55cdefbcd803e267d2635534ff9e511 Author: Mike Perry <mikeperry-git(a)fscked.org> Date: Mon Mar 7 18:10:41 2011 -0800 Fix some issues with SmartSocket. Also remove some traceback debugging statements. --- PathSupport.py | 19 ++++++++++++------- 1 files changed, 12 insertions(+), 7 deletions(-) diff --git a/PathSupport.py b/PathSupport.py index e3ddfd1..4e9ad6d 100644 --- a/PathSupport.py +++ b/PathSupport.py @@ -1318,6 +1318,12 @@ class SmartSocket(_SocketWrapper): port_table = set() _table_lock = threading.Lock() + def __init__(self, family=2, type=1, proto=0, _sock=None): + ret = super(SmartSocket, self).__init__(family, type, proto, _sock) + self.__local_addr = None + plog("DEBUG", "New socket constructor") + return ret + def connect(self, args): ret = super(SmartSocket, self).connect(args) myaddr = self.getsockname() @@ -1341,14 +1347,13 @@ class SmartSocket(_SocketWrapper): return ret def __del__(self): - SmartSocket._table_lock.acquire() - try: + if self.__local_addr: + SmartSocket._table_lock.acquire() SmartSocket.port_table.remove(self.__local_addr) - except AttributeError,e: - traceback.print_exc() - plog("WARN", "Hrm. Socket instance without local_addr attribute?") - SmartSocket._table_lock.release() - plog("DEBUG", "Removed "+self.__local_addr+" from our local port list") + plog("DEBUG", "Removed "+self.__local_addr+" from our local port list") + SmartSocket._table_lock.release() + else: + plog("DEBUG", "Got a socket deletion with no address") def table_size(): SmartSocket._table_lock.acquire()

1 0

r24312: {website} Add a missing horizontal rule (website/trunk/docs/en)
by Robert Ransom 08 Mar '11

08 Mar '11

Author: rransom Date: 2011-03-08 00:25:31 +0000 (Tue, 08 Mar 2011) New Revision: 24312 Modified: website/trunk/docs/en/faq.wml Log: Add a missing horizontal rule Modified: website/trunk/docs/en/faq.wml =================================================================== --- website/trunk/docs/en/faq.wml 2011-03-07 23:26:15 UTC (rev 24311) +++ website/trunk/docs/en/faq.wml 2011-03-08 00:25:31 UTC (rev 24312) @@ -560,6 +560,8 @@ director for information on making grants or major donations. </p> + <hr> + <a id="Metrics"></a> <h3><a class="anchor" href="#Metrics">How many people use Tor? How many relays or exit nodes are there?</a></h3>

1 0

r24311: {website} Undo a bit more of the damage done by css/reset.css, this ti (website/trunk/css)
by Robert Ransom 07 Mar '11

07 Mar '11

Author: rransom Date: 2011-03-07 23:26:15 +0000 (Mon, 07 Mar 2011) New Revision: 24311 Modified: website/trunk/css/typography.css Log: Undo a bit more of the damage done by css/reset.css, this time to sup/sub This won't fix the font size, but I can't test a change to two properties at once with Firebug. Modified: website/trunk/css/typography.css =================================================================== --- website/trunk/css/typography.css 2011-03-07 23:10:28 UTC (rev 24310) +++ website/trunk/css/typography.css 2011-03-07 23:26:15 UTC (rev 24311) @@ -95,6 +95,13 @@ font-family: Inconsolata, "Lucida Console", "Droid Sans Mono", Cousine, monospace; } +sup { + vertical-align: top; +} +sub { + vertical-align: bottom; +} + /* NAVIGATION ------------*/ #nav ul { font-size: 1.167em; }

1 0

r24310: {website} Remove wiki markup accidentally pasted in FAQ page (website/trunk/docs/en)
by Robert Ransom 07 Mar '11

07 Mar '11

Author: rransom Date: 2011-03-07 23:10:28 +0000 (Mon, 07 Mar 2011) New Revision: 24310 Modified: website/trunk/docs/en/faq.wml Log: Remove wiki markup accidentally pasted in FAQ page Modified: website/trunk/docs/en/faq.wml =================================================================== --- website/trunk/docs/en/faq.wml 2011-03-07 16:18:22 UTC (rev 24309) +++ website/trunk/docs/en/faq.wml 2011-03-07 23:10:28 UTC (rev 24310) @@ -1415,8 +1415,6 @@ <hr> -[https://www.torproject.org/docs/faq#EntryGuards Answer moved to our new FAQ page] - <a id="EntryGuards"></a> <h3><a class="anchor" href="#EntryGuards">What are Entry Guards?</a></h3>

1 0

[tor/master] Merge remote branch 'sebastian/bug2660' into maint-0.2.2
by nickm＠torproject.org 07 Mar '11

07 Mar '11

commit 3bd83b8fb64c070210c589f65ed8cc023dd47077 Merge: 35fcec3 f83debb Author: Nick Mathewson <nickm(a)torproject.org> Date: Mon Mar 7 17:09:23 2011 -0500 Merge remote branch 'sebastian/bug2660' into maint-0.2.2 changes/bug2660 | 7 +++++++ src/common/address.c | 18 +++++++++--------- 2 files changed, 16 insertions(+), 9 deletions(-)

1 0