- tor-commits - lists.torproject.org

[tor/master] Merge branch 'maint-0.2.2'
by arma＠torproject.org 12 Mar '11

12 Mar '11

commit 0b07b5dc989b50cb9fad82efe946f160713420cf Merge: 4740311 879f171 Author: Roger Dingledine <arma(a)torproject.org> Date: Fri Mar 11 17:40:16 2011 -0500 Merge branch 'maint-0.2.2' ReleaseNotes | 51 +++++++++++++++++++++++++++++++++++++++++++++++++++ 1 files changed, 51 insertions(+), 0 deletions(-)

1 0

[tor/maint-0.2.2] put the 0.2.1.30 changes into releasenotes
by arma＠torproject.org 12 Mar '11

12 Mar '11

commit 151ab614421eed48ba2f39518695b8ee08206904 Author: Roger Dingledine <arma(a)torproject.org> Date: Fri Mar 11 17:38:55 2011 -0500 put the 0.2.1.30 changes into releasenotes --- ReleaseNotes | 51 +++++++++++++++++++++++++++++++++++++++++++++++++++ 1 files changed, 51 insertions(+), 0 deletions(-) diff --git a/ReleaseNotes b/ReleaseNotes index 9562133..7ba473e 100644 --- a/ReleaseNotes +++ b/ReleaseNotes @@ -3,6 +3,57 @@ This document summarizes new features and bugfixes in each stable release of Tor. If you want to see more detailed descriptions of the changes in each development snapshot, see the ChangeLog file. +Changes in version 0.2.1.30 - 2011-02-23 + Tor 0.2.1.30 fixes a variety of less critical bugs. The main other + change is a slight tweak to Tor's TLS handshake that makes relays + and bridges that run this new version reachable from Iran again. + We don't expect this tweak will win the arms race long-term, but it + buys us time until we roll out a better solution. + + o Major bugfixes: + - Stop sending a CLOCK_SKEW controller status event whenever + we fetch directory information from a relay that has a wrong clock. + Instead, only inform the controller when it's a trusted authority + that claims our clock is wrong. Bugfix on 0.1.2.6-alpha; fixes + the rest of bug 1074. + - Fix a bounds-checking error that could allow an attacker to + remotely crash a directory authority. Bugfix on 0.2.1.5-alpha. + Found by "piebeer". + - If relays set RelayBandwidthBurst but not RelayBandwidthRate, + Tor would ignore their RelayBandwidthBurst setting, + potentially using more bandwidth than expected. Bugfix on + 0.2.0.1-alpha. Reported by Paul Wouters. Fixes bug 2470. + - Ignore and warn if the user mistakenly sets "PublishServerDescriptor + hidserv" in her torrc. The 'hidserv' argument never controlled + publication of hidden service descriptors. Bugfix on 0.2.0.1-alpha. + + o Minor features: + - Adjust our TLS Diffie-Hellman parameters to match those used by + Apache's mod_ssl. + - Update to the February 1 2011 Maxmind GeoLite Country database. + + o Minor bugfixes: + - Check for and reject overly long directory certificates and + directory tokens before they have a chance to hit any assertions. + Bugfix on 0.2.1.28. Found by "doorss". + - Bring the logic that gathers routerinfos and assesses the + acceptability of circuits into line. This prevents a Tor OP from + getting locked in a cycle of choosing its local OR as an exit for a + path (due to a .exit request) and then rejecting the circuit because + its OR is not listed yet. It also prevents Tor clients from using an + OR running in the same instance as an exit (due to a .exit request) + if the OR does not meet the same requirements expected of an OR + running elsewhere. Fixes bug 1859; bugfix on 0.1.0.1-rc. + + o Packaging changes: + - Stop shipping the Tor specs files and development proposal documents + in the tarball. They are now in a separate git repository at + git://git.torproject.org/torspec.git + - Do not include Git version tags as though they are SVN tags when + generating a tarball from inside a repository that has switched + between branches. Bugfix on 0.2.1.15-rc; fixes bug 2402. + + Changes in version 0.2.1.29 - 2011-01-15 Tor 0.2.1.29 continues our recent code security audit work. The main fix resolves a remote heap overflow vulnerability that can allow remote

1 0

[tor/maint-0.2.2] Merge branch 'maint-0.2.1' into maint-0.2.2
by arma＠torproject.org 12 Mar '11

12 Mar '11

commit 879f1718ad59682387ffc54b9d8589badb5f1e81 Merge: 48c4d53 151ab61 Author: Roger Dingledine <arma(a)torproject.org> Date: Fri Mar 11 17:39:54 2011 -0500 Merge branch 'maint-0.2.1' into maint-0.2.2 ReleaseNotes | 51 +++++++++++++++++++++++++++++++++++++++++++++++++++ 1 files changed, 51 insertions(+), 0 deletions(-)

1 0

[tor/maint-0.2.1] put the 0.2.1.30 changes into releasenotes
by arma＠torproject.org 12 Mar '11

12 Mar '11

commit 151ab614421eed48ba2f39518695b8ee08206904 Author: Roger Dingledine <arma(a)torproject.org> Date: Fri Mar 11 17:38:55 2011 -0500 put the 0.2.1.30 changes into releasenotes --- ReleaseNotes | 51 +++++++++++++++++++++++++++++++++++++++++++++++++++ 1 files changed, 51 insertions(+), 0 deletions(-) diff --git a/ReleaseNotes b/ReleaseNotes index 9562133..7ba473e 100644 --- a/ReleaseNotes +++ b/ReleaseNotes @@ -3,6 +3,57 @@ This document summarizes new features and bugfixes in each stable release of Tor. If you want to see more detailed descriptions of the changes in each development snapshot, see the ChangeLog file. +Changes in version 0.2.1.30 - 2011-02-23 + Tor 0.2.1.30 fixes a variety of less critical bugs. The main other + change is a slight tweak to Tor's TLS handshake that makes relays + and bridges that run this new version reachable from Iran again. + We don't expect this tweak will win the arms race long-term, but it + buys us time until we roll out a better solution. + + o Major bugfixes: + - Stop sending a CLOCK_SKEW controller status event whenever + we fetch directory information from a relay that has a wrong clock. + Instead, only inform the controller when it's a trusted authority + that claims our clock is wrong. Bugfix on 0.1.2.6-alpha; fixes + the rest of bug 1074. + - Fix a bounds-checking error that could allow an attacker to + remotely crash a directory authority. Bugfix on 0.2.1.5-alpha. + Found by "piebeer". + - If relays set RelayBandwidthBurst but not RelayBandwidthRate, + Tor would ignore their RelayBandwidthBurst setting, + potentially using more bandwidth than expected. Bugfix on + 0.2.0.1-alpha. Reported by Paul Wouters. Fixes bug 2470. + - Ignore and warn if the user mistakenly sets "PublishServerDescriptor + hidserv" in her torrc. The 'hidserv' argument never controlled + publication of hidden service descriptors. Bugfix on 0.2.0.1-alpha. + + o Minor features: + - Adjust our TLS Diffie-Hellman parameters to match those used by + Apache's mod_ssl. + - Update to the February 1 2011 Maxmind GeoLite Country database. + + o Minor bugfixes: + - Check for and reject overly long directory certificates and + directory tokens before they have a chance to hit any assertions. + Bugfix on 0.2.1.28. Found by "doorss". + - Bring the logic that gathers routerinfos and assesses the + acceptability of circuits into line. This prevents a Tor OP from + getting locked in a cycle of choosing its local OR as an exit for a + path (due to a .exit request) and then rejecting the circuit because + its OR is not listed yet. It also prevents Tor clients from using an + OR running in the same instance as an exit (due to a .exit request) + if the OR does not meet the same requirements expected of an OR + running elsewhere. Fixes bug 1859; bugfix on 0.1.0.1-rc. + + o Packaging changes: + - Stop shipping the Tor specs files and development proposal documents + in the tarball. They are now in a separate git repository at + git://git.torproject.org/torspec.git + - Do not include Git version tags as though they are SVN tags when + generating a tarball from inside a repository that has switched + between branches. Bugfix on 0.2.1.15-rc; fixes bug 2402. + + Changes in version 0.2.1.29 - 2011-01-15 Tor 0.2.1.29 continues our recent code security audit work. The main fix resolves a remote heap overflow vulnerability that can allow remote

1 0

r24339: {website} enable spanish and russian, since they build currently (website/trunk)
by Roger Dingledine 12 Mar '11

12 Mar '11

Author: arma Date: 2011-03-11 22:26:38 +0000 (Fri, 11 Mar 2011) New Revision: 24339 Modified: website/trunk/Makefile.common Log: enable spanish and russian, since they build currently Modified: website/trunk/Makefile.common =================================================================== --- website/trunk/Makefile.common 2011-03-11 12:38:16 UTC (rev 24338) +++ website/trunk/Makefile.common 2011-03-11 22:26:38 UTC (rev 24339) @@ -18,7 +18,7 @@ -D STABLETAG=$(STABLETAG) #LANGS=ar bms de en es et fa it fi fr ja ko nl no pl pt ru se tr zh-cn -LANGS=en fr +LANGS=en es fr ru WMLFILES=$(wildcard $(patsubst %, %/*.wml, $(LANGS))) WMIFILES=$(wildcard $(patsubst %, %/*.wmi, $(LANGS)) $(WMLBASE)/include/*.wmi )

1 0

[metrics-tasks/master] Add code for 2680.
by karsten＠torproject.org 12 Mar '11

12 Mar '11

commit ee7df6a759f6788579f61403fb9771996ea5c988 Author: Karsten Loesing <karsten.loesing(a)gmx.net> Date: Fri Mar 11 14:51:12 2011 +0100 Add code for 2680. --- task-2680/.gitignore | 6 + task-2680/ProcessRelayConsensuses.java | 85 ++++++++ task-2680/ProcessSanitizedBridges.java | 327 ++++++++++++++++++++++++++++++++ task-2680/README | 145 ++++++++++++++ task-2680/verify.R | 27 +++ 5 files changed, 590 insertions(+), 0 deletions(-) diff --git a/task-2680/.gitignore b/task-2680/.gitignore new file mode 100644 index 0000000..134e86d --- /dev/null +++ b/task-2680/.gitignore @@ -0,0 +1,6 @@ +*.class +*.csv +bridge-descriptors/ +commons-codec-1.4.jar +consensuses/ + diff --git a/task-2680/ProcessRelayConsensuses.java b/task-2680/ProcessRelayConsensuses.java new file mode 100644 index 0000000..44d9ce3 --- /dev/null +++ b/task-2680/ProcessRelayConsensuses.java @@ -0,0 +1,85 @@ +import java.io.*; +import java.util.*; +import org.apache.commons.codec.binary.*; +import org.apache.commons.codec.digest.*; + +public class ProcessRelayConsensuses { + public static void main(String[] args) throws IOException { + + /* Validate command-line arguments. */ + if (args.length != 1 || !new File(args[0]).exists()) { + System.out.println("Usage: java ProcessRelayConsensuses <dir>"); + System.exit(1); + } + + /* Find all files that we should parse. Somewhat fragile, but should + * work. */ + System.out.println("Creating list of files we should parse."); + SortedMap<String, File> consensuses = new TreeMap<String, File>(); + Stack<File> files = new Stack<File>(); + files.add(new File(args[0])); + while (!files.isEmpty()) { + File file = files.pop(); + String filename = file.getName(); + if (file.isDirectory()) { + files.addAll(Arrays.asList(file.listFiles())); + } else if (filename.endsWith("-consensus")) { + consensuses.put(filename, file); + } + } + System.out.println("We found " + consensuses.size() + + " consensuses."); + + /* Parse consensuses. */ + if (!consensuses.isEmpty()) { + System.out.println("Parsing consensuses."); + BufferedWriter bw = new BufferedWriter(new FileWriter( + "relays.csv")); + bw.write("consensus,fingerprint\n"); + int parsedConsensuses = 0, totalConsensuses = consensuses.size(), + writtenOutputLines = 1; + long started = System.currentTimeMillis(); + for (File file : consensuses.values()) { + BufferedReader br = new BufferedReader(new FileReader(file)); + String line, validAfter = null; + while ((line = br.readLine()) != null) { + if (line.startsWith("valid-after ")) { + validAfter = line.substring("valid-after ".length()); + } else if (line.startsWith("r ")) { + if (validAfter == null) { + System.out.println("Found an r line before the valid-after " + + "line in " + file.getName() + ". Please check. " + + "Exiting."); + System.exit(1); + } + String fingerprint = DigestUtils.shaHex(Base64.decodeBase64( + line.split(" ")[2] + "=")); + bw.write(validAfter + "," + fingerprint + "\n"); + writtenOutputLines++; + } + } + br.close(); + parsedConsensuses++; + if (parsedConsensuses % (totalConsensuses / 10) == 0) { + double fractionDone = (double) (parsedConsensuses) / + (double) totalConsensuses; + double fractionLeft = 1.0D - fractionDone; + long now = System.currentTimeMillis(); + double millisLeft = ((double) (now - started)) * fractionLeft / + fractionDone; + long secondsLeft = (long) millisLeft / 1000L; + System.out.println(" " + (parsedConsensuses / (totalConsensuses + / 10)) + "0% done, " + secondsLeft + " seconds left."); + } + } + bw.close(); + System.out.println("Parsed " + parsedConsensuses + " consensuses " + + "and wrote " + writtenOutputLines + " lines to relays.csv."); + } + + /* This is it. */ + System.out.println("Terminating."); + } +} + + diff --git a/task-2680/ProcessSanitizedBridges.java b/task-2680/ProcessSanitizedBridges.java new file mode 100644 index 0000000..1f0e00e --- /dev/null +++ b/task-2680/ProcessSanitizedBridges.java @@ -0,0 +1,327 @@ +import java.io.*; +import java.text.*; +import java.util.*; +import org.apache.commons.codec.binary.*; + +public class ProcessSanitizedBridges { + public static void main(String[] args) throws IOException, + ParseException { + + /* Validate command-line arguments. */ + if (args.length != 1 || !new File(args[0]).exists()) { + System.out.println("Usage: java ProcessSanitizedBridges <dir>"); + System.exit(1); + } + + /* Find all files that we should parse. Somewhat fragile, but should + * work. */ + System.out.println("Creating list of files we should parse."); + SortedMap<String, File> statuses = new TreeMap<String, File>(); + SortedMap<String, File> serverDescriptors = + new TreeMap<String, File>(); + SortedMap<String, File> extraInfoDescriptors = + new TreeMap<String, File>(); + Stack<File> files = new Stack<File>(); + files.add(new File(args[0])); + while (!files.isEmpty()) { + File file = files.pop(); + String path = file.getAbsolutePath(); + String filename = file.getName(); + if (file.isDirectory()) { + files.addAll(Arrays.asList(file.listFiles())); + } else if (path.contains("statuses")) { + statuses.put(filename, file); + } else if (path.contains("server-descriptors")) { + serverDescriptors.put(filename, file); + } else if (path.contains("extra-infos")) { + extraInfoDescriptors.put(filename, file); + } + } + System.out.println("We found\n " + statuses.size() + " statuses,\n " + + serverDescriptors.size() + " server descriptors, and\n " + + extraInfoDescriptors.size() + " extra-info descriptors."); + + /* Parse statuses. */ + if (!statuses.isEmpty()) { + System.out.println("Parsing statuses."); + List<String> knownFlags = new ArrayList<String>(Arrays.asList( + ("Authority,BadExit,BadDirectory,Exit,Fast,Guard,Named,Stable," + + "Running,Valid,V2Dir").split(","))); + BufferedWriter bw = new BufferedWriter(new FileWriter( + "statuses.csv")); + bw.write("status,fingerprint,descriptor,published,address,orport," + + "dirport"); + for (String knownFlag : knownFlags) { + bw.write("," + knownFlag.toLowerCase()); + } + bw.write("\n"); + int parsedStatuses = 0, totalStatuses = statuses.size(), + writtenOutputLines = 1; + long started = System.currentTimeMillis(); + for (File file : statuses.values()) { + String filename = file.getName(); + if (filename.length() != ("20110101-000703-" + + "4A0CCD2DDC7995083D73F5D667100C8A5831F16D").length()) { + System.out.println("Status filename has wrong length: '" + + filename + "' Please check. Exiting."); + System.exit(1); + } + String statusDateTime = filename.substring(0, 4) + "-" + + filename.substring(4, 6) + "-" + filename.substring(6, 8) + + " " + filename.substring(9, 11) + ":" + + filename.substring(11, 13) + ":" + + filename.substring(13, 15); + BufferedReader br = new BufferedReader(new FileReader(file)); + String line; + while ((line = br.readLine()) != null) { + if (line.startsWith("r ")) { + String[] parts = line.split(" "); + if (parts.length != 9) { + System.out.println("r line doesn't have the correct number " + + "of entries: '" + line + "'. Please check. Exiting."); + System.exit(1); + } + String fingerprint = Hex.encodeHexString(Base64.decodeBase64( + parts[2] + "=")); + String descriptor = Hex.encodeHexString(Base64.decodeBase64( + parts[2] + "=")); + String published = parts[4] + " " + parts[5]; + String address = parts[6]; + String orPort = parts[7]; + String dirPort = parts[8]; + bw.write(statusDateTime + "," + fingerprint + "," + descriptor + + "," + published + "," + address + "," + orPort + "," + + dirPort); + } else if (line.equals("s") || line.startsWith("s ")) { + String flags = line.substring(1); + for (String flag : knownFlags) { + if (flags.contains(" " + flag)) { + bw.write(",TRUE"); + } else { + bw.write(",FALSE"); + } + } + bw.write("\n"); + writtenOutputLines++; + } + } + br.close(); + parsedStatuses++; + if (parsedStatuses % (totalStatuses / 10) == 0) { + double fractionDone = (double) (parsedStatuses) / + (double) totalStatuses; + double fractionLeft = 1.0D - fractionDone; + long now = System.currentTimeMillis(); + double millisLeft = ((double) (now - started)) * fractionLeft / + fractionDone; + long secondsLeft = (long) millisLeft / 1000L; + System.out.println(" " + (parsedStatuses / (totalStatuses + / 10)) + "0% done, " + secondsLeft + " seconds left."); + } + } + bw.close(); + System.out.println("Parsed " + parsedStatuses + " statuses and " + + "wrote " + writtenOutputLines + " lines to statuses.csv."); + } + + /* Parse server descriptors and extra-info descriptors. */ + if (!serverDescriptors.isEmpty()) { + System.out.println("Parsing server descriptors and extra-info " + + "descriptors."); + List<String> knownCountries = new ArrayList<String>(Arrays.asList( + ("?? A1 A2 AD AE AF AG AI AL AM AN AO AP AQ AR AS AT AU AW AX " + + "AZ BA BB BD BE BF BG BH BI BJ BM BN BO BR BS BT BV BW BY BZ " + + "CA CD CF CG CH CI CK CL CM CN CO CR CS CU CV CY CZ DE DJ DK " + + "DM DO DZ EC EE EG ER ES ET EU FI FJ FK FM FO FR GA GB GD GE " + + "GF GG GH GI GL GM GN GP GQ GR GT GU GW GY HK HN HR HT HU ID " + + "IE IL IM IN IO IQ IR IS IT JE JM JO JP KE KG KH KI KM KN KP " + + "KR KW KY KZ LA LB LC LI LK LR LS LT LU LV LY MA MC MD ME MF " + + "MG MH MK ML MM MN MO MP MQ MR MS MT MU MV MW MX MY MZ NA NC " + + "NE NF NG NI NL NO NP NR NU NZ OM PA PE PF PG PH PK PL PM PR " + + "PS PT PW PY QA RE RO RS RU RW SA SB SC SD SE SG SH SI SJ SK " + + "SL SM SN SO SR ST SV SY SZ TC TD TG TH TJ TK TL TM TN TO TR " + + "TT TV TW TZ UA UG UM US UY UZ VA VC VE VG VI VN VU WF WS YE " + + "YT ZA ZM ZW").toLowerCase().split(" "))); + BufferedWriter bw = new BufferedWriter(new FileWriter( + "descriptors.csv")); + bw.write("descriptor,fingerprint,published,address,orport,dirport," + + "version,platform,uptime,bridgestatsend,bridgestatsseconds"); + for (String country : knownCountries) { + bw.write("," + country); + } + bw.write(",bridgestatscountries,bridgestatstotal\n"); + int parsedServerDescriptors = 0, parsedExtraInfoDescriptors = 0, + parsedGeoipStats = 0, skippedGeoipStats = 0, + parsedBridgeStats = 0, + totalServerDescriptors = serverDescriptors.size(), + writtenOutputLines = 1; + SimpleDateFormat timeFormat = new SimpleDateFormat( + "yyyy-MM-dd HH:mm:ss"); + timeFormat.setTimeZone(TimeZone.getTimeZone("UTC")); + long started = System.currentTimeMillis(); + for (File file : serverDescriptors.values()) { + String filename = file.getName(); + BufferedReader br = new BufferedReader(new FileReader(file)); + String line, fingerprint = null, published = null, address = null, + orPort = null, dirPort = null, version = null, + platform = null, uptime = null, extraInfoDigest = null, + bridgeStatsEnd = null, bridgeStatsSeconds = null; + SortedMap<String, String> bridgeStatsIps = + new TreeMap<String, String>(); + long bridgeStatsTotal = 0L; + while ((line = br.readLine()) != null) { + if (line.startsWith("opt ")) { + line = line.substring(4); + } + if (line.startsWith("router ")) { + String[] parts = line.split(" "); + address = parts[2]; + orPort = parts[3]; + dirPort = parts[4]; + } else if (line.startsWith("platform ")) { + version = line.split(" ")[2]; + platform = line.substring(line.indexOf("on ") + + "on ".length()); + if (platform.contains("Windows")) { + platform = "Windows"; + } else if (platform.contains("Linux")) { + platform = "Linux"; + } else if (platform.contains("Darwin")) { + platform = "Mac OS X"; + } else if (platform.contains("BSD")) { + platform = "*BSD"; + } else { + platform = "Other"; + } + } else if (line.startsWith("published ")) { + String[] parts = line.split(" "); + published = parts[1] + " " + parts[2]; + } else if (line.startsWith("fingerprint ")) { + fingerprint = line.substring("fingerprint".length()). + replaceAll(" ", "").toLowerCase(); + } else if (line.startsWith("uptime ")) { + uptime = line.split(" ")[1]; + } else if (line.startsWith("extra-info-digest ")) { + extraInfoDigest = line.substring("extra-info-digest ". + length()).toLowerCase(); + if (extraInfoDescriptors.containsKey(extraInfoDigest)) { + parsedExtraInfoDescriptors++; + BufferedReader br2 = new BufferedReader(new FileReader( + extraInfoDescriptors.get(extraInfoDigest))); + String geoipStartTime = null, bridgeStatsEndLine = null; + while ((line = br2.readLine()) != null) { + if (line.startsWith("geoip-start-time ")) { + geoipStartTime = line.substring("geoip-start-time ". + length()); + } else if (line.startsWith("geoip-client-origins ") && + line.split(" ").length > 1 && published != null && + geoipStartTime != null) { + if (version.startsWith("0.2.2.")) { + skippedGeoipStats++; + } else { + parsedGeoipStats++; + bridgeStatsEnd = published; + bridgeStatsSeconds = "" + + + (timeFormat.parse(published).getTime() + - timeFormat.parse(geoipStartTime).getTime()) + / 1000L; + for (String pair : line.split(" ")[1].split(",")) { + String country = pair.substring(0, 2); + String ips = pair.substring(3); + bridgeStatsIps.put(country, ips); + bridgeStatsTotal += Long.parseLong(ips); + } + } + } else if (line.startsWith("bridge-stats-end ")) { + bridgeStatsEndLine = line; + } else if (line.startsWith("bridge-ips ") && + line.length() > "bridge-ips ".length() && + bridgeStatsEndLine != null) { + parsedBridgeStats++; + String[] parts = bridgeStatsEndLine.split(" "); + bridgeStatsEnd = parts[1] + " " + parts[2]; + bridgeStatsSeconds = parts[3].substring(1); + for (String pair : line.split(" ")[1].split(",")) { + String country = pair.substring(0, 2); + String ips = pair.substring(3); + bridgeStatsIps.put(country, ips); + bridgeStatsTotal += Long.parseLong(ips); + } + } + } + br2.close(); + } + } + } + br.close(); + if (fingerprint == null || published == null || address == null || + orPort == null || dirPort == null || version == null || + platform == null || uptime == null) { + System.out.println("Server descriptor " + filename + " is " + + "missing critical information. Please check. Exiting."); + System.exit(1); + } + bw.write(filename + "," + fingerprint + "," + published + "," + + address + "," + orPort + "," + dirPort + "," + version + "," + + platform + "," + uptime); + if (bridgeStatsEnd != null) { + bw.write("," + bridgeStatsEnd + "," + bridgeStatsSeconds); + int bridgeStatsCountries = bridgeStatsIps.size(); + for (String country : knownCountries) { + if (bridgeStatsIps.containsKey(country)) { + bw.write("," + bridgeStatsIps.remove(country)); + } else { + bw.write(",0"); + } + } + if (!bridgeStatsIps.isEmpty()) { + StringBuilder message = new StringBuilder(); + for (String country : bridgeStatsIps.keySet()) { + message.append(", " + country); + } + System.out.println("Unknown " + (bridgeStatsIps.size() == 1 ? + "country" : "countries") + " " + message.toString(). + substring(2) + " in extra-info descriptor " + + extraInfoDigest + ". Please check. Exiting."); + System.exit(1); + } + bw.write("," + bridgeStatsCountries + "," + bridgeStatsTotal + + "\n"); + } else { + bw.write(",NA,NA"); + for (String country : knownCountries) { + bw.write(",NA"); + } + bw.write(",NA,NA\n"); + } + writtenOutputLines++; + parsedServerDescriptors++; + if (parsedServerDescriptors % (totalServerDescriptors / 100) + == 0) { + double fractionDone = (double) (parsedServerDescriptors) / + (double) totalServerDescriptors; + double fractionLeft = 1.0D - fractionDone; + long now = System.currentTimeMillis(); + double millisLeft = ((double) (now - started)) * fractionLeft / + fractionDone; + long secondsLeft = (long) millisLeft / 1000L; + System.out.println(" " + (parsedServerDescriptors / + (totalServerDescriptors / 100)) + "% done, " + secondsLeft + + " seconds left."); + } + } + bw.close(); + System.out.println("Parsed " + parsedServerDescriptors + " server " + + "descriptors and " + parsedExtraInfoDescriptors + + " extra-info descriptors.\nParsed " + parsedGeoipStats + + " geoip-stats and " + parsedBridgeStats + " bridge-stats.\n" + + "Skipped " + skippedGeoipStats + " broken geoip-stats of " + + "0.2.2.x bridges.\nWrote " + writtenOutputLines + " to " + + "descriptors.csv."); + } + + /* This is it. */ + System.out.println("Terminating."); + } +} + diff --git a/task-2680/README b/task-2680/README new file mode 100644 index 0000000..a00856f --- /dev/null +++ b/task-2680/README @@ -0,0 +1,145 @@ +This ticket contains Java and R code to + + a) process bridge and relay data to convert them to a format that is more + useful for researchers and + b) verify that the output data files are valid. + +This README has a separate section for each Java or R code snippet. + +The Java applications produce three output formats containing bridge +descriptors, bridge status lines, and hashed relay identities. The data +formats are described below. + +-------------------------------------------------------------------------- + +ProcessSanitizedBridges.java + + - Download sanitized bridge descriptors from the metrics website, e.g., + https://metrics.torproject.org/data/bridge-descriptors-2011-01.tar.bz2, + and extract them in a local directory, e.g., bridge-descriptors/. + + - Download Apache Commons Codec 1.4 or higher and put in in this + directory. + + - Compile the Java class, e.g., + $ javac -cp commons-codec-1.4.jar ProcessSanitizedBridges.java + + - Run the Java class, e.g., + $ java -cp .:commons-codec-1.4.jar ProcessSanitizedBridges + bridge-descriptors/ + + - Once the Java application is done, you'll find the two files + statuses.csv and descriptors.csv in this directory. + +-------------------------------------------------------------------------- + +ProcessRelayConsensuses.java + + - Download v3 relay consensuses from the metrics website, e.g., + https://metrics.torproject.org/data/consensuses-2011-01.tar.bz2, and + extract them in a local directory, e.g., consensuses/. + + - Download Apache Commons Codec 1.4 or higher and put in in this + directory, unless you haven't already done this above for + ProcessSanitizedBridges.java. + + - Compile the Java class, e.g., + $ javac -cp commons-codec-1.4.jar ProcessRelayConsensuses.java + + - Run the Java class, e.g., + $ java -cp .:commons-codec-1.4.jar ProcessRelayConsensuses consensuses/ + + - Once the Java application is done, you'll find a file relays.csv in + this directory. + +-------------------------------------------------------------------------- + +verify.R + + - Run the R verification script like this: + $ R --slave -f verify.R + +-------------------------------------------------------------------------- + +descriptors.csv + +The descriptors.csv file contains one line for each bridge descriptor that +a bridge has published. This descriptor consists of fields coming from +the bridge's server descriptor and the bridge's extra-info descriptor that +was published at the same time. + +The columns in descriptors.csv are: + + - descriptor: Hex-formatted descriptor identifier + - fingerprint: Hex-formatted SHA-1 hash of identity fingerprint + - published: ISO-formatted descriptor publication time + - address: Sanitized IPv4 address in dotted notation + - orport: OR port + - dirport: Dir port + - version: Tor version + - platform: Operating system family (Windows, Linux, etc.) + - uptime: Uptime in seconds + - bridgestatsend: ISO-formatted time when stats interval ended + - bridgestatsseconds: Stats interval length in seconds + - ??: Unique client IP addresses that could not be resolved + - a1: Unique client IP addresses from anonymous proxies + - a2: Unique client IP addresses from satellite providers + - ad: Unique client IP addresses from Andorra + - ae: Unique client IP addresses from the United Arab Emirates + - [...] See ISO 3166-1 alpha-2 country codes + - zw: Unique client IP addresses from Zimbabwe + - bridgestatscountries: Number of countries with non-zero unique IPs + - bridgestatstotal: Total number of unique IPs + +There are two sources for the bridgestats* and country-code columns, +depending on Tor's version. Bridges running Tor version 0.2.1.x or +earlier use dynamic stats intervals from a few hours to a few days. +Bridges running early 0.2.2.x versions published faulty stats and are +therefore removed from descriptors.csv. Bridges running 0.2.2.x or higher +(except the faulty 0.2.2.x versions) collect stats in 24-hour intervals. + +-------------------------------------------------------------------------- + +statuses.csv + +The statuses.csv file contains one line for every bridge that is +referenced in a bridge network status. Note that if a bridge is running +for, say, 12 hours, it will be contained in 24 half-hourly published +statuses in that time and will be listed 24 times in statuses.csv. + +The columns in statuses.csv are: + + - status: ISO-formatted status publication time + - fingerprint: Hex-formatted SHA-1 hash of identity fingerprint + - descriptor: Hex-formatted descriptor identifier + - published: ISO-formatted descriptor publication time + - address: Sanitized IPv4 address in dotted notation + - orport: OR port + - dirport: Dir port + - authority: TRUE if bridge has the Authority flag, FALSE otherwise + - badexit: TRUE if bridge has the BadExit flag, FALSE otherwise + - baddirectory: TRUE if bridge has the BadDirectory flag, FALSE otherwise + - exit: TRUE if bridge has the Exit flag, FALSE otherwise + - fast: TRUE if bridge has the Fast flag, FALSE otherwise + - guard: TRUE if bridge has the Guard flag, FALSE otherwise + - named: TRUE if bridge has the Named flag, FALSE otherwise + - stable: TRUE if bridge has the Stable flag, FALSE otherwise + - running: TRUE if bridge has the Running flag, FALSE otherwise + - valid: TRUE if bridge has the Valid flag, FALSE otherwise + - v2dir: TRUE if bridge has the V2Dir flag, FALSE otherwise + +-------------------------------------------------------------------------- + +relays.csv + +The relays.csv file contains SHA-1 hashes of identity fingerprints of +normal relays. If a bridge uses the same identity key that it also used +as a relay, it might observe more users than it would observe as a pure +bridge. Therefore, bridges that have been running as relays before should +be excluded from bridge statistics. + +The columns in relays.csv are: + + - consensus: ISO-formatted consensus publication time + - fingerprint: Hex-formatted SHA-1 hash of identity fingerprint + diff --git a/task-2680/verify.R b/task-2680/verify.R new file mode 100644 index 0000000..63ef233 --- /dev/null +++ b/task-2680/verify.R @@ -0,0 +1,27 @@ +# Usage: R --slave -f verify.R + +if (file.exists("descriptors.csv")) { + cat("Verifying descriptors.csv. This may take a while.\n") + d <- read.csv("descriptors.csv", stringsAsFactors = FALSE) + cat(" ", length(na.omit(d$bridgestatsend)), "of", length(d$descriptor), + "descriptors contain bridge stats.\n") +} else { + cat("descriptors.csv does not exist\n") +} + +if (file.exists("statuses.csv")) { + cat("Verifying statuses.csv. This may take a while.\n") + s <- read.csv("statuses.csv", stringsAsFactors = FALSE) + cat(" ", length(s[s$running == TRUE, "running"]), "of", + length(s$running), "bridges contained in the statuses have the", + "Running flag.\n") +} else { + cat("statuses.csv does not exist\n") +} + +if (file.exists("relays.csv")) { + cat("Verifying relays.csv. This may take a while.\n") + r <- read.csv("relays.csv", stringsAsFactors = FALSE) + summary(as.POSIXct(r$consensus)) +} +

1 0

r24338: {website} add the NyTeknik story. (website/trunk/press/en)
by Andrew Lewman 11 Mar '11

11 Mar '11

Author: phobos Date: 2011-03-11 12:38:16 +0000 (Fri, 11 Mar 2011) New Revision: 24338 Modified: website/trunk/press/en/inthemedia.wml Log: add the NyTeknik story. Modified: website/trunk/press/en/inthemedia.wml =================================================================== --- website/trunk/press/en/inthemedia.wml 2011-03-11 01:36:18 UTC (rev 24337) +++ website/trunk/press/en/inthemedia.wml 2011-03-11 12:38:16 UTC (rev 24338) @@ -29,6 +29,13 @@ <th>Topic</th> </tr> </thead> +<tr> +<td>2011 March 10</td> +<td>NyTeknik</td> +<td><a +href="http://www.nyteknik.se/nyheter/it_telekom/internet/article3123594.ece">Svenska +biståndspengar till Facebookrevolutionen</a></td> +</tr> <tr style="background-color: #e5e5e5;"> <td>2011 March 09</td> <td>Washington Post</td>

1 0

r24337: {website} support the old anchor name too (website/trunk/download/en)
by Roger Dingledine 11 Mar '11

11 Mar '11

Author: arma Date: 2011-03-11 01:36:18 +0000 (Fri, 11 Mar 2011) New Revision: 24337 Modified: website/trunk/download/en/download.wml Log: support the old anchor name too Modified: website/trunk/download/en/download.wml =================================================================== --- website/trunk/download/en/download.wml 2011-03-11 01:15:16 UTC (rev 24336) +++ website/trunk/download/en/download.wml 2011-03-11 01:36:18 UTC (rev 24337) @@ -183,6 +183,7 @@  <div class="warning"> <a name="warning"></a> +<a name="Warning"></a> <h2><a class="anchor" href="#warning">Want Tor to really work?</a></h2> <p>...then please don't just install it and go on. You need to change some of your habits, and reconfigure your software! Tor by itself is <em>NOT</em> all you need to maintain your anonymity. There are several major pitfalls to watch out for: </p>

1 0

r24336: {website} somewhere along the line we stopped setting svn props (in website/trunk: about/en docs/en donate/en download/en en getinvolved/en manpages/en press/en projects/en torbutton/en)
by Roger Dingledine 11 Mar '11

11 Mar '11

Author: arma Date: 2011-03-11 01:15:16 +0000 (Fri, 11 Mar 2011) New Revision: 24336 Modified: website/trunk/about/en/sidenav.wmi website/trunk/docs/en/proxychain.wml website/trunk/docs/en/sidenav.wmi website/trunk/donate/en/info.wmi website/trunk/donate/en/sidenav.wmi website/trunk/download/en/download-easy.wml website/trunk/en/navigation.wmi website/trunk/getinvolved/en/sidenav.wmi website/trunk/manpages/en/tor-manual-dev.wml website/trunk/press/en/info.wmi website/trunk/press/en/sidenav.wmi website/trunk/projects/en/arm.wml website/trunk/projects/en/gettor.wml website/trunk/projects/en/sidenav.wmi website/trunk/torbutton/en/sidenav.wmi Log: somewhere along the line we stopped setting svn props Modified: website/trunk/about/en/sidenav.wmi =================================================================== --- website/trunk/about/en/sidenav.wmi 2011-03-11 01:12:39 UTC (rev 24335) +++ website/trunk/about/en/sidenav.wmi 2011-03-11 01:15:16 UTC (rev 24336) @@ -1,5 +1,5 @@ ## translation metadata -# Revision: $Revision: 0 $ +# Revision: $Revision$ # Translation-Priority: 2-medium # this structure defines the side nav bar for the /about pages Property changes on: website/trunk/about/en/sidenav.wmi ___________________________________________________________________ Added: svn:keywords + Author Date Id Revision Added: svn:eol-style + native Modified: website/trunk/docs/en/proxychain.wml =================================================================== --- website/trunk/docs/en/proxychain.wml 2011-03-11 01:12:39 UTC (rev 24335) +++ website/trunk/docs/en/proxychain.wml 2011-03-11 01:15:16 UTC (rev 24336) @@ -1,5 +1,5 @@ ## translation metadata -# Revision: $Revision: 23689 $ +# Revision: $Revision$ # Translation-Priority: 1-high #include "head.wmi" TITLE="Tor Project: Configuring Tor to use a Proxy" CHARSET="UTF-8" Property changes on: website/trunk/docs/en/proxychain.wml ___________________________________________________________________ Added: svn:keywords + Author Date Id Revision Added: svn:eol-style + native Modified: website/trunk/docs/en/sidenav.wmi =================================================================== --- website/trunk/docs/en/sidenav.wmi 2011-03-11 01:12:39 UTC (rev 24335) +++ website/trunk/docs/en/sidenav.wmi 2011-03-11 01:15:16 UTC (rev 24336) @@ -1,5 +1,5 @@ ## translation metadata -# Revision: $Revision: 0 $ +# Revision: $Revision$ # Translation-Priority: 2-medium # this structure defines the side nav bar for the /about pages Property changes on: website/trunk/docs/en/sidenav.wmi ___________________________________________________________________ Added: svn:keywords + Author Date Id Revision Added: svn:eol-style + native Modified: website/trunk/donate/en/info.wmi =================================================================== --- website/trunk/donate/en/info.wmi 2011-03-11 01:12:39 UTC (rev 24335) +++ website/trunk/donate/en/info.wmi 2011-03-11 01:15:16 UTC (rev 24336) @@ -1,5 +1,5 @@ ## translation metadata -# Revision: $Revision: 0 $ +# Revision: $Revision$ # Translation-Priority: 2-medium <div class="img-shadow"> Property changes on: website/trunk/donate/en/info.wmi ___________________________________________________________________ Added: svn:keywords + Author Date Id Revision Added: svn:eol-style + native Modified: website/trunk/donate/en/sidenav.wmi =================================================================== --- website/trunk/donate/en/sidenav.wmi 2011-03-11 01:12:39 UTC (rev 24335) +++ website/trunk/donate/en/sidenav.wmi 2011-03-11 01:15:16 UTC (rev 24336) @@ -1,7 +1,7 @@ #!/usr/bin/wml ## translation metadata -# Revision: $Revision: 0 $ +# Revision: $Revision$ # Translation-Priority: 2-medium # this structure defines the side nav bar for the /about pages Property changes on: website/trunk/donate/en/sidenav.wmi ___________________________________________________________________ Added: svn:keywords + Author Date Id Revision Added: svn:eol-style + native Modified: website/trunk/download/en/download-easy.wml =================================================================== --- website/trunk/download/en/download-easy.wml 2011-03-11 01:12:39 UTC (rev 24335) +++ website/trunk/download/en/download-easy.wml 2011-03-11 01:15:16 UTC (rev 24336) @@ -1,5 +1,5 @@ ## translation metadata -# Revision: $Revision: 21396 $ +# Revision: $Revision$ # Translation-Priority: 1-high #include "head.wmi" TITLE="Tor: Easy Download Selector" CHARSET="UTF-8" ANNOUNCE_RSS="yes" Property changes on: website/trunk/download/en/download-easy.wml ___________________________________________________________________ Added: svn:keywords + Author Date Id Revision Added: svn:eol-style + native Modified: website/trunk/en/navigation.wmi =================================================================== --- website/trunk/en/navigation.wmi 2011-03-11 01:12:39 UTC (rev 24335) +++ website/trunk/en/navigation.wmi 2011-03-11 01:15:16 UTC (rev 24336) @@ -1,7 +1,7 @@ #!/usr/bin/wml ## translation metadata -# Revision: $Revision: 22069 $ +# Revision: $Revision$ # Translation-Priority: 1-high <: Property changes on: website/trunk/en/navigation.wmi ___________________________________________________________________ Added: svn:keywords + Author Date Id Revision Modified: website/trunk/getinvolved/en/sidenav.wmi =================================================================== --- website/trunk/getinvolved/en/sidenav.wmi 2011-03-11 01:12:39 UTC (rev 24335) +++ website/trunk/getinvolved/en/sidenav.wmi 2011-03-11 01:15:16 UTC (rev 24336) @@ -1,7 +1,7 @@ #!/usr/bin/wml ## translation metadata -# Revision: $Revision: 0 $ +# Revision: $Revision$ # Translation-Priority: 2-medium # this structure defines the side nav bar for the /about pages Property changes on: website/trunk/getinvolved/en/sidenav.wmi ___________________________________________________________________ Added: svn:keywords + Author Date Id Revision Added: svn:eol-style + native Modified: website/trunk/manpages/en/tor-manual-dev.wml =================================================================== --- website/trunk/manpages/en/tor-manual-dev.wml 2011-03-11 01:12:39 UTC (rev 24335) +++ website/trunk/manpages/en/tor-manual-dev.wml 2011-03-11 01:15:16 UTC (rev 24336) @@ -1,5 +1,5 @@ ## translation metadata -# Revision: $Revision: 23689 $ +# Revision: $Revision$ #include "head.wmi" TITLE="Tor Project: manual" Property changes on: website/trunk/manpages/en/tor-manual-dev.wml ___________________________________________________________________ Added: svn:keywords + Author Date Id Revision Added: svn:eol-style + native Modified: website/trunk/press/en/info.wmi =================================================================== --- website/trunk/press/en/info.wmi 2011-03-11 01:12:39 UTC (rev 24335) +++ website/trunk/press/en/info.wmi 2011-03-11 01:15:16 UTC (rev 24336) @@ -1,5 +1,5 @@ ## translation metadata -# Revision: $Revision: 0 $ +# Revision: $Revision$ # Translation-Priority: 2-medium #!/usr/bin/env wml <div class="img-shadow"> Property changes on: website/trunk/press/en/info.wmi ___________________________________________________________________ Added: svn:keywords + Author Date Id Revision Added: svn:eol-style + native Modified: website/trunk/press/en/sidenav.wmi =================================================================== --- website/trunk/press/en/sidenav.wmi 2011-03-11 01:12:39 UTC (rev 24335) +++ website/trunk/press/en/sidenav.wmi 2011-03-11 01:15:16 UTC (rev 24336) @@ -1,7 +1,7 @@ #!/usr/bin/wml ## translation metadata -# Revision: $Revision: 0 $ +# Revision: $Revision$ # Translation-Priority: 2-medium # this structure defines the side nav bar for the /about pages Property changes on: website/trunk/press/en/sidenav.wmi ___________________________________________________________________ Added: svn:keywords + Author Date Id Revision Added: svn:eol-style + native Modified: website/trunk/projects/en/arm.wml =================================================================== --- website/trunk/projects/en/arm.wml 2011-03-11 01:12:39 UTC (rev 24335) +++ website/trunk/projects/en/arm.wml 2011-03-11 01:15:16 UTC (rev 24336) @@ -1,5 +1,5 @@ ## translation metadata -# Revision: $Revision: 23704 $ +# Revision: $Revision$ # Translation-Priority: 4-optional #include "head.wmi" TITLE="Tor Project: Arm" CHARSET="UTF-8" ANNOUNCE_RSS="yes" Property changes on: website/trunk/projects/en/arm.wml ___________________________________________________________________ Added: svn:keywords + Author Date Id Revision Added: svn:eol-style + native Modified: website/trunk/projects/en/gettor.wml =================================================================== --- website/trunk/projects/en/gettor.wml 2011-03-11 01:12:39 UTC (rev 24335) +++ website/trunk/projects/en/gettor.wml 2011-03-11 01:15:16 UTC (rev 24336) @@ -1,5 +1,5 @@ ## translation metadata -# Revision: $Revision: 23689 $ +# Revision: $Revision$ # Translation-Priority: 3-low #include "head.wmi" TITLE="Tor Project: GetTor email autoresponder" CHARSET="UTF-8" ANNOUNCE_RSS="yes" Property changes on: website/trunk/projects/en/gettor.wml ___________________________________________________________________ Added: svn:keywords + Author Date Id Revision Added: svn:eol-style + native Modified: website/trunk/projects/en/sidenav.wmi =================================================================== --- website/trunk/projects/en/sidenav.wmi 2011-03-11 01:12:39 UTC (rev 24335) +++ website/trunk/projects/en/sidenav.wmi 2011-03-11 01:15:16 UTC (rev 24336) @@ -1,7 +1,7 @@ #!/usr/bin/wml ## translation metadata -# Revision: $Revision: 0 $ +# Revision: $Revision$ # Translation-Priority: 2-medium # this structure defines the side nav bar for the /about pages Property changes on: website/trunk/projects/en/sidenav.wmi ___________________________________________________________________ Added: svn:keywords + Author Date Id Revision Added: svn:eol-style + native Modified: website/trunk/torbutton/en/sidenav.wmi =================================================================== --- website/trunk/torbutton/en/sidenav.wmi 2011-03-11 01:12:39 UTC (rev 24335) +++ website/trunk/torbutton/en/sidenav.wmi 2011-03-11 01:15:16 UTC (rev 24336) @@ -1,7 +1,7 @@ #!/usr/bin/wml ## translation metadata -# Revision: $Revision: 0 $ +# Revision: $Revision$ # Translation-Priority: 2-medium # this structure defines the side nav bar for the /about pages Property changes on: website/trunk/torbutton/en/sidenav.wmi ___________________________________________________________________ Added: svn:keywords + Author Date Id Revision Added: svn:eol-style + native

1 0

r24335: {website} minor fix (website/trunk/about/en)
by Roger Dingledine 11 Mar '11

11 Mar '11

Author: arma Date: 2011-03-11 01:12:39 +0000 (Fri, 11 Mar 2011) New Revision: 24335 Modified: website/trunk/about/en/gsoc.wml Log: minor fix Modified: website/trunk/about/en/gsoc.wml =================================================================== --- website/trunk/about/en/gsoc.wml 2011-03-10 23:20:43 UTC (rev 24334) +++ website/trunk/about/en/gsoc.wml 2011-03-11 01:12:39 UTC (rev 24335) @@ -186,7 +186,7 @@ If you're interested, you can either contact the <a href="<page about/contact>">tor-assistants list</a> with a brief summary of your proposal and we'll give you feedback, or just jump right in and post your ideas - and goals to the <a href="<page docs/documentation>#MailingLists">or-talk + and goals to the <a href="<page docs/documentation>#MailingLists">tor-talk mailing list</a>. Make sure to be responsive during the application selection period; if we like your application but you never answer our mails asking for more information, that's not a good sign.

1 0