- tor-commits - lists.torproject.org

[tor/master] Sockets are unsigned on windows
by nickm＠torproject.org 09 Aug '11

09 Aug '11

commit 0a5338e03cdf14ef80584c6ff8adeb49200b8a76 Author: Sebastian Hahn <sebastian(a)torproject.org> Date: Tue Aug 9 10:59:03 2011 +0200 Sockets are unsigned on windows this gets rid of a warning about signed/unsigned comparison --- src/common/compat.h | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/src/common/compat.h b/src/common/compat.h index 98642e2..8e271ba 100644 --- a/src/common/compat.h +++ b/src/common/compat.h @@ -401,7 +401,7 @@ typedef int socklen_t; #ifdef MS_WINDOWS #define tor_socket_t intptr_t -#define SOCKET_OK(s) ((s) != INVALID_SOCKET) +#define SOCKET_OK(s) ((unsigned)(s) != INVALID_SOCKET) #else #define tor_socket_t int #define SOCKET_OK(s) ((s) >= 0)

1 0

[tor/master] Get rid of an unused parameter warning on win
by nickm＠torproject.org 09 Aug '11

09 Aug '11

commit bed79c47f4ec0ee72b19e2b81c54131d516d07ef Author: Sebastian Hahn <sebastian(a)torproject.org> Date: Tue Aug 9 11:00:25 2011 +0200 Get rid of an unused parameter warning on win --- src/common/util.c | 2 ++ 1 files changed, 2 insertions(+), 0 deletions(-) diff --git a/src/common/util.c b/src/common/util.c index 601f2be..542093f 100644 --- a/src/common/util.c +++ b/src/common/util.c @@ -1736,6 +1736,8 @@ check_private_dir(const char *dirname, cpd_check_t check, struct passwd *pw = NULL; uid_t running_uid; gid_t running_gid; +#else + (void)effective_user; #endif tor_assert(dirname);

1 0

[tor/master] Sticking a size_t into long generates a warn on win
by nickm＠torproject.org 09 Aug '11

09 Aug '11

commit 7c340a97b1252e1165dc646d4e75ce4d837c65e6 Author: Sebastian Hahn <sebastian(a)torproject.org> Date: Tue Aug 9 11:34:21 2011 +0200 Sticking a size_t into long generates a warn on win --- src/or/connection_edge.c | 10 ++++++---- 1 files changed, 6 insertions(+), 4 deletions(-) diff --git a/src/or/connection_edge.c b/src/or/connection_edge.c index 7dd6ee0..72c4224 100644 --- a/src/or/connection_edge.c +++ b/src/or/connection_edge.c @@ -161,7 +161,8 @@ connection_edge_process_inbuf(edge_connection_t *conn, int package_partial) "data from edge while in '%s' state. Sending it anyway. " "package_partial=%d, buflen=%ld", conn_state_to_string(conn->_base.type, conn->_base.state), - package_partial, connection_get_inbuf_len(TO_CONN(conn))); + package_partial, + (long)connection_get_inbuf_len(TO_CONN(conn))); if (connection_edge_package_raw_inbuf(conn, package_partial, NULL)<0) { /* (We already sent an end cell if possible) */ connection_mark_for_close(TO_CONN(conn)); @@ -2442,9 +2443,10 @@ connection_ap_handshake_send_begin(edge_connection_t *ap_conn) ap_conn->sending_optimistic_data) && connection_ap_supports_optimistic_data(ap_conn)) { log_info(LD_APP, "Sending up to %ld + %ld bytes of queued-up data", - connection_get_inbuf_len(TO_CONN(ap_conn)), - ap_conn->sending_optimistic_data ? - generic_buffer_len(ap_conn->sending_optimistic_data) : 0); + (long)connection_get_inbuf_len(TO_CONN(ap_conn)), + ap_conn->sending_optimistic_data ? + (long)generic_buffer_len(ap_conn->sending_optimistic_data) : + 0); if (connection_edge_package_raw_inbuf(ap_conn, 1, NULL) < 0) { connection_mark_for_close(TO_CONN(ap_conn)); }

1 0

[metrics-tasks/master] Add a more useful README for detector.py (#2718).
by karsten＠torproject.org 09 Aug '11

09 Aug '11

commit 56889ddfad3b3a1fdc1b259d4f95f3895e764972 Author: Karsten Loesing <karsten.loesing(a)gmx.net> Date: Tue Aug 9 15:52:59 2011 +0200 Add a more useful README for detector.py (#2718). --- task-2718/README | 120 ++++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 files changed, 120 insertions(+), 0 deletions(-) diff --git a/task-2718/README b/task-2718/README new file mode 100644 index 0000000..8f1d1e2 --- /dev/null +++ b/task-2718/README @@ -0,0 +1,120 @@ +Tor Censorship Detector +======================= + +The Tor Censorship Detector is a script that reads a file containing the +number of daily Tor users and finds anomalies that might be indicative of +censorship. This README explains how to use the script and then makes an +attempt to describe how the math behind the script works. + +We start with downloading the estimated Tor user numbers from the metrics +website: + + $ wget https://metrics.torproject.org/csv/direct-users.csv + +This file contains estimated daily Tor users with columns being country +codes and rows being dates. A detailed description of how these estimates +are obtained is contained in this tech report: + + https://metrics.torproject.org/papers/countingusers-2010-11-30.pdf + +An excerpt of direct-users.csv is: + + date,??,a1,a2,ad,ae,...,zw,all + 2011-08-06,6936,448,61,24,2460,...,13,337627 + 2011-08-07,5904,398,53,15,2335,...,13,335626 + +The "date" column contains the ISO-formatted date. The number in the "??" +column stands for all IP addresses that could not be resolved to a country +by the GeoIP database. The columns "a1" to "zw" contain the number of +users by country, including MaxMind-specific country codes described at +http://www.maxmind.com/app/iso3166 . The "all" column contains the sum of +all Tor users on a given day. + +In order to run the Python script to detect anomalies, make an img/ +directory and install the required Python packages: + + $ mkdir img/ + $ sudo apt-get install python-numpy python-scipy python-matplotlib + +Run the detector.py script (this may take a while): + + $ python detector.py + +The output consists of a file direct-users-ranges.csv containing the +expected range of users per day and country, a file img/summary.txt with +an overview of possible censorship events, and graphs in img/ for all +countries with possible censorship events. + +An excerpt of direct-users-ranges.csv is: + + date,country,minusers,maxusers + 2011-08-06,ae,1559.43780955,3880.20765967 + 2011-08-07,ae,1460.8116866,3452.1615707 + +This output means that the expected number of users on August 6, 2011 for +the United Arab Emirates was 1559 to 3880 users. The observed number of +users in direct-users.csv was 2460, so that the script wouldn't suspect a +censorship event here. + +The img/summary.txt file begins with the following lines: + + ======================= + Report for 2011-02-03 to 2011-08-07 + ======================= + sc -- down: 17 (up: 25 affected: 148) + ly -- down: 13 (up: 17 affected: 29) + py -- down: 10 (up: 8 affected: 137) + +This output means that, for example, in Libya there were 13 possible +censorship events (downturns) and 17 possible releases of censorship +(upturns) between February 3 and August 7, 2011. + +The graph img/013-ly-censor.png visualizes these downturns and upturns in +a time plot. + +The core of the censorship detector script is contained in the functions +make_tendencies_minmax() and write_all() in detector.py. + +In make_tendencies_minmax(), the detector is given the user number series +of the top-50 countries by users based on the last day in direct-users.csv +for the entire interval in direct-users.csv. For example, the last 10 +values in the series for the United States are: + + 66171,72866,76900,76292,77753,75749,81680,68084,77526,75499 + +For each of these countries, the detector computes the quotients between +the number of users on a given day and 1 week before. These quotients for +the series above are: + + 1.048,1.075,1.015,0.987,1.148,0.959,1.176,1.029,1.064,0.982 + +For every day in direct-users.csv, the detector considers all non-zero +quotients of the top-50 countries. It then discards outliers which are +more than 4 times the interquartile range away from the median. For +August 7, 2011, all 50 quotients were in the interval from 0.697 to 1.263 +with no outliers. + +In the next step, the detector fits a normal distribution to these +quotients and uses the inverse cumulative function to look up the 0.01-th +and 99.99-th percentiles. These values are the hypothetic quotients that +are greater than 0.01% and 99.99% of all quotients, respectively. In the +data mentioned above, the fitted normal distribution has a mean of 0.992 +and a standard deviation of 0.091, and the looked up percentiles are 0.654 +and 1.33. + +In write_all(), the detector calculates an estimated minimum and maximum +for a given country and date based on the user number 1 week ago. The +detector first looks up the 0.01-th and the 99.99-th percentile of a +Poisson distribution with the mean being the user number from 1 week ago. +It then weights these percentiles with the network-wide quotients +calculated above. + +For example, when looking at the user numbers from the United States, +there were 75499 users on August 7, 2011 and 76900 users 1 week before on +July 31, 2011. The 0.01-th and the 99.99th percentiles of the Poisson +distribution with a mean of 76900 are 75871 and 77933. The estimated +range of users on August 7, 2011 goes from 0.654 * 75871 = 49620 to +1.33 * 77933 = 103651. The actually observed 75499 users are within this +interval, so there's no suspected censorship event, nor release of +censorship in the United States on August 7, 2011. +

1 0

[metrics-tasks/master] Remove outdated censorship detector code (#2718).
by karsten＠torproject.org 09 Aug '11

09 Aug '11

commit 4426cf9cfa1d8965057c3fe34184c5ce459fa390 Author: Karsten Loesing <karsten.loesing(a)gmx.net> Date: Tue Aug 9 15:52:38 2011 +0200 Remove outdated censorship detector code (#2718). --- task-2718/README | 27 -------------------- task-2718/convert-dirreqs-sql.R | 8 ------ task-2718/detect-censorship.R | 52 --------------------------------------- 3 files changed, 0 insertions(+), 87 deletions(-) diff --git a/task-2718/README b/task-2718/README deleted file mode 100644 index 8c60847..0000000 --- a/task-2718/README +++ /dev/null @@ -1,27 +0,0 @@ -Here's how you run the censorship detector prototype: - - $ wget https://metrics.torproject.org/csv/direct-users.csv - $ R --slave -f detect-censorship.R - -------------------------------------------------------------------------- - -Extracting raw directory requests from the metrics-web database: - -- Export dirreq_stats table from the metrics-web database via psql: - - # \f ',' - # \a - # \t - # \o dirreqs-sql.csv - # SELECT * FROM dirreq_stats ORDER BY statsend; - # \o - # \t - # \a - -- Transform the huge (!) CSV file (104M) from long to wide format. Note - that this takes a while: - - $ R --slave -f convert-dirreqs-sql.R - -- The result is in dirreqs.csv (8.8M). - diff --git a/task-2718/convert-dirreqs-sql.R b/task-2718/convert-dirreqs-sql.R deleted file mode 100644 index e330307..0000000 --- a/task-2718/convert-dirreqs-sql.R +++ /dev/null @@ -1,8 +0,0 @@ -library(ggplot2) -data <- read.csv("dirreqs-sql.csv", header = FALSE) -data <- data.frame(fingerprint = data$V1, statsend = data$V2, - seconds = data$V3, country = data$V4, requests = data$V5) -data <- cast(data, fingerprint + statsend + seconds ~ country, - value = "requests") -write.csv(data, file = "dirreqs.csv", quote = FALSE, row.names = FALSE) - diff --git a/task-2718/detect-censorship.R b/task-2718/detect-censorship.R deleted file mode 100644 index f27e863..0000000 --- a/task-2718/detect-censorship.R +++ /dev/null @@ -1,52 +0,0 @@ -# Tor Censorship Detector -# Usage: R --slave < detect-censorship.R - -# Read CSV file containing daily user number estimates. -direct <- read.csv("direct-users.csv") - -# Start plotting everything to a single PDF (with multiple pages). -pdf("detect-censorship.pdf") - -# Convert the column containing daily Iranian users to a time series -# object, starting on the 263th day of 2009 with a frequency of 365 days. -# We're probably off by a day or two, but this should be fine for now. -all <- ts(direct$ir, start = c(2009, 263), frequency = 365) - -# Uncomment to print the time series values. -#print(all) - -# Let's try our approach for the last 365 days to see if we detect any -# blocking in that time period. In the final version of this script, we'd -# only have a single run with i = 1. -for (i in 365:1) { - idx <- length(direct$date) - i - - # Convert the daily Iranian users until i days in the past to a time - # series object. - x <- ts(direct$ir[1:idx], start = c(2009, 263), frequency = 365) - - # Apply an ARIMA(1, 0, 1) model to the time series. - x.fit = arima(x, order = c(1, 0, 1)) - - # Predict 10 dates ahead. - x.fore=predict(x.fit, n.ahead=10) - - # Calculate a lower bound. Here we use the predicted value minus three - # standard errors. - L = x.fore$pred - 3*x.fore$se - - # If the observed daily user number is lower than our predicted lower - # bound, plot the data and lower bound. - if (direct$ir[idx + 1] < L[1]) { - - # Plot the full time series. - ts.plot(all) - - # Add a line for the ten predicted values. - lines(L, col = "red", lwd = 2) - } -} - -# Close the PDF device. -dev.off() -

1 0

[torbutton/master] Document torbutton release process.
by mikeperry＠torproject.org 09 Aug '11

09 Aug '11

commit 4fd677d0499e786e7529d6e972b8eee636be320d Author: Mike Perry <mikeperry-git(a)fscked.org> Date: Sat Jul 2 12:02:27 2011 -0700 Document torbutton release process. Documented in as close to script form as possible. In the bright bright future, the whole release process will be automated. --- README.RELEASE | 74 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 files changed, 74 insertions(+), 0 deletions(-) diff --git a/README.RELEASE b/README.RELEASE new file mode 100644 index 0000000..cec3ee8 --- /dev/null +++ b/README.RELEASE @@ -0,0 +1,74 @@ +# Magical Torbutton Release Process Incantations +# +# "May this part of my job one day be replaced by a small shell script" +# + +#. Decide the version number. + + export VERSION="1.4.0" + +#. Test upgrades of candidate XPI in latest Firefox and TBB product matrix + + # TODO: Currently manual + +#. Generate changelog from git shortlog + + # TODO: Currently manual + +#. tx noise: https://svn.torproject.org/svn/translation/trunk/documentation/howto.txt + + cd torbutton.git/trans_tools/po + torsocks svn up + torsocks tx pull -a + cd .. + ./validate_all.sh | less + torsocks ./new_tb_strings.sh + ./mkmoz.sh + ./mvmoz.sh + cd .. + git diff . + git commit . + +#. Tag the release: + + git tag -s ${VERSION} + torsocks git push origin-push --tags + +#. Create the .xpi + + cd /tmp/ + torsocks git clone git://git.torproject.org/git/torbutton.git torbutton-release + cd torbutton-release + git tag -v ${VERSION} # verify signature against git protocol haaAAXXXX + git checkout -b tag-${VERSION} ${VERSION} + sh makexpi.sh + +#. SHA1 + Sign the .xpi + + gpg -abs ./pkg/torbutton-${VERSION}.xpi + sha1sum ./pkg/torbutton-${VERSION}.xpi + +#. Upload .xpi to website + + torsocks scp ./pkg/torbutton-${VERSION}.xpi* vescum:/srv/www-master.torproject.org/htdocs/dist/torbutton/ + torsocks scp ./pkg/torbutton-${VERSION}.xpi vescum:/srv/www-master.torproject.org/htdocs/dist/torbutton/torbutton-curre… + torsocks scp ./pkg/torbutton-${VERSION}.xpi.asc vescum:/srv/www-master.torproject.org/htdocs/dist/torbutton/torbutton-curre… + +#. Update website + + cd website-svn + vim ./include/versions.wmi # Add sha1 and version + vim ./torbutton/update.rdf # Add <li> and <description> w/ sha1 and version + torsocks svn up . + torsocks svn commit . + +#. Test firefox autoupdate + + # Wait for mirrors to sync first.. Pester phobos/weasel/helix/Sebastian for + # website push + dist chown + mirror update. + # TODO: Currently manual + +#. Announce on tor-talk + + # FIXME: May need eventual minor improvements + echo "Troll food." | mail tor-talk(a)lists.torproject.org -s "Feed the trolls"

1 0

[torbutton/master] Bug #3665: Use loadGroup to get window if callbacks are absent
by mikeperry＠torproject.org 09 Aug '11

09 Aug '11

commit cc51fe06551be998f70d1653433f8332b2895f98 Author: Mike Perry <mikeperry-git(a)fscked.org> Date: Mon Aug 8 17:00:01 2011 -0700 Bug #3665: Use loadGroup to get window if callbacks are absent Thanks to Georg Koppen for catching this oversight. --- src/chrome/content/stanford-safecache.js | 28 +++++++++++++++++++--------- 1 files changed, 19 insertions(+), 9 deletions(-) diff --git a/src/chrome/content/stanford-safecache.js b/src/chrome/content/stanford-safecache.js index 3daed58..31cdf45 100644 --- a/src/chrome/content/stanford-safecache.js +++ b/src/chrome/content/stanford-safecache.js @@ -97,19 +97,29 @@ SSC_RequestListener.prototype = onModifyRequest: function(channel) { var parent = null; - if (channel.notificationCallbacks) { - try { - var wind = channel.notificationCallbacks.QueryInterface( - Components.interfaces.nsIInterfaceRequestor).getInterface( - Components.interfaces.nsIDOMWindow); - parent = wind.window.top.location; - } catch(e) { - } - SSC_dump("Parent "+parent+" for "+ channel.URI.spec); + if (channel.notificationCallbacks || + channel.loadGroup && channel.loadGroup.notificationCallbacks) { + var callbacks = null; + if (channel.notificationCallbacks) { + callbacks = channel.notificationCallbacks; + } else { + callbacks = channel.loadGroup.notificationCallbacks; + } + try { + var wind = callbacks.QueryInterface( + Components.interfaces.nsIInterfaceRequestor).getInterface( + Components.interfaces.nsIDOMWindow); + parent = wind.window.top.location; + } catch(e) { + } + SSC_dump("Parent "+parent+" for "+ channel.URI.spec); } if (channel.documentURI && channel.documentURI == channel.URI) { parent = null; // first party interaction + } else if(!parent) { + // Questionable first party interaction.. + SSC_dump("No parent parent for "+ channel.URI.spec); } // Same-origin policy

1 0

[torbutton/master] Bug #3665: Use referer host if we can't find the owner window
by mikeperry＠torproject.org 09 Aug '11

09 Aug '11

commit bf2a44b97fd3fad9d766a0cac6ed6148dd9c27b5 Author: Mike Perry <mikeperry-git(a)fscked.org> Date: Mon Aug 8 17:45:59 2011 -0700 Bug #3665: Use referer host if we can't find the owner window Add a fallback to use the referer host if we can't find the owner window through either loadGroup or notificationCallbacks.. This might still leave https:// urls sourced from http:// frames un-isolated, as well as the reverse... --- src/chrome/content/stanford-safecache.js | 33 +++++++++++++++++------------ 1 files changed, 19 insertions(+), 14 deletions(-) diff --git a/src/chrome/content/stanford-safecache.js b/src/chrome/content/stanford-safecache.js index 31cdf45..bf2fb16 100644 --- a/src/chrome/content/stanford-safecache.js +++ b/src/chrome/content/stanford-safecache.js @@ -42,7 +42,7 @@ var SSC_debug = true; */ function SSC_dump(msg) { if (SSC_debug) - torbutton_log(2, "SSC: " + msg); + torbutton_log(3, "SSC: " + msg); } //////////////////////////////////////////////////////////////////////////// @@ -77,7 +77,7 @@ SSC_RequestListener.prototype = subject.QueryInterface(Components.interfaces.nsICachingChannel); this.onExamineResponse(subject); } - } catch(e) {try {SSC_dump(e);} catch(ex) {}} + } catch(e) {try {torbutton_log(4, "SSC: "+e);} catch(ex) {}} }, bypassCache: function(channel) { @@ -96,7 +96,7 @@ SSC_RequestListener.prototype = }, onModifyRequest: function(channel) { - var parent = null; + var parent_host = null; if (channel.notificationCallbacks || channel.loadGroup && channel.loadGroup.notificationCallbacks) { var callbacks = null; @@ -109,26 +109,30 @@ SSC_RequestListener.prototype = var wind = callbacks.QueryInterface( Components.interfaces.nsIInterfaceRequestor).getInterface( Components.interfaces.nsIDOMWindow); - parent = wind.window.top.location; + parent_host = wind.window.top.location.hostname; } catch(e) { } - SSC_dump("Parent "+parent+" for "+ channel.URI.spec); + SSC_dump("Parent "+parent_host+" for "+ channel.URI.spec); } if (channel.documentURI && channel.documentURI == channel.URI) { - parent = null; // first party interaction - } else if(!parent) { + parent_host = null; // first party interaction + } else if(!parent_host) { // Questionable first party interaction.. - SSC_dump("No parent parent for "+ channel.URI.spec); + if (!channel.referrer) { + torbutton_safelog(3, "SSC: No parent for ", channel.URI.spec); + } else { + parent_host = channel.referrer.host; + } } // Same-origin policy var referrer; - if (parent && parent.hostname != channel.URI.host) { + if (parent_host && parent_host != channel.URI.host) { SSC_dump("Segmenting " + channel.URI.host + - " content loaded by " + parent.host); - this.setCacheKey(channel, parent.hostname); - referrer = parent.hostname; + " content loaded by " + parent_host); + this.setCacheKey(channel, parent_host); + referrer = parent_host; } else { referrer = channel.URI.host; if(!this.readCacheKey(channel.cacheKey)) { @@ -139,9 +143,9 @@ SSC_RequestListener.prototype = } if (this.controller.getBlockThirdPartyCache()) { - if(parent && parent.hostname != channel.URI.host) { + if(parent_host && parent_host != channel.URI.host) { //SSC_dump("Third party cache blocked for " + channel.URI.spec + - //" content loaded by " + parent.spec); + //" content loaded by " + parent_host); this.bypassCache(channel); } } @@ -227,6 +231,7 @@ SSC_RequestListener.prototype = if(setCookie) { var parent = null; + // XXX: need to use loadGroup here if null.. if (channel.notificationCallbacks) { var wind = channel.notificationCallbacks.QueryInterface( Components.interfaces.nsIInterfaceRequestor).getInterface(

1 0

[tor/master] Merge remote-tracking branch 'origin/maint-0.2.1' into maint-0.2.2
by nickm＠torproject.org 08 Aug '11

08 Aug '11

commit f570470830503c93b4d0491eb4812dfa59e585a1 Merge: 718252b c75ee94 Author: Nick Mathewson <nickm(a)torproject.org> Date: Mon Aug 8 17:21:21 2011 -0400 Merge remote-tracking branch 'origin/maint-0.2.1' into maint-0.2.2 changes/geoip-august2011 | 3 + src/config/geoip | 6191 ++++++++++++++++++++++++---------------------- 2 files changed, 3235 insertions(+), 2959 deletions(-)

1 0

[tor/master] Merge remote-tracking branch 'origin/maint-0.2.2'
by nickm＠torproject.org 08 Aug '11

08 Aug '11

commit 10d9bcb2a2fe4e49afd88251fe92181c2ffbc231 Merge: a72b181 f570470 Author: Nick Mathewson <nickm(a)torproject.org> Date: Mon Aug 8 17:21:39 2011 -0400 Merge remote-tracking branch 'origin/maint-0.2.2' changes/geoip-august2011 | 3 + src/config/geoip | 6191 ++++++++++++++++++++++++---------------------- 2 files changed, 3235 insertions(+), 2959 deletions(-)

1 0