September 2012 - tor-commits - lists.torproject.org

r25797: {website} add some recent press. (website/trunk/press/en)
by Andrew Lewman 19 Sep '12

19 Sep '12

Author: phobos Date: 2012-09-19 15:11:19 +0000 (Wed, 19 Sep 2012) New Revision: 25797 Modified: website/trunk/press/en/inthemedia.wml Log: add some recent press. Modified: website/trunk/press/en/inthemedia.wml =================================================================== --- website/trunk/press/en/inthemedia.wml 2012-09-18 14:20:57 UTC (rev 25796) +++ website/trunk/press/en/inthemedia.wml 2012-09-19 15:11:19 UTC (rev 25797) @@ -33,6 +33,14 @@ </tr> </thead> <tr style="background-color: #e5e5e5;"> +<td>2012 Sep 19</td> +<td><a href="http://www.infobae.com/notas/667840-Proyecto-Tor-comunicaciones-en-la-web-1…">Proyecto Tor: comunicaciones en la web 100% anónimas y seguras</a></td> +</td> +<tr> +<td>2012 Aug 24</td> +<td><a href="https://secure.security.nl/artikel/42807/1/Tor_gaat_naar_Facebook_ondanks_p…">Tor gaat naar Facebook ondanks privacyrisico</a></td> +</tr> +<tr style="background-color: #e5e5e5;"> <td>2012 Jul 19</td> <td>Philadelphia Inquirer</td> <td><a href="http://www.philly.com/philly/business/20120719_Tech_Life__Fostering_anonymi…">Tech Life: Fostering anonymity online</a></td>

1 0

[tor/master] Merge branch 'bug6843'
by nickm＠torproject.org 19 Sep '12

19 Sep '12

commit 0e0a16774261cf2f9f98ffe42ccd8aefe3a36a95 Merge: f612a9e a78eeb4 Author: Nick Mathewson <nickm(a)torproject.org> Date: Wed Sep 19 09:01:15 2012 -0400 Merge branch 'bug6843' changes/bug6843 | 5 +++++ doc/include.am | 30 ++++++++++++++++++++++++++---- 2 files changed, 31 insertions(+), 4 deletions(-)

1 0

[tor/master] Fix dependency checking on manpage builds
by nickm＠torproject.org 19 Sep '12

19 Sep '12

commit a78eeb4cbc99e7c92cad57c30e0d1957881951fb Author: Nick Mathewson <nickm(a)torproject.org> Date: Fri Sep 14 10:36:42 2012 -0400 Fix dependency checking on manpage builds Fixes bug 6843; bugfix on 0.2.4.1-alpha. --- changes/bug6843 | 5 +++++ doc/include.am | 30 ++++++++++++++++++++++++++---- 2 files changed, 31 insertions(+), 4 deletions(-) diff --git a/changes/bug6843 b/changes/bug6843 new file mode 100644 index 0000000..1acfdbd --- /dev/null +++ b/changes/bug6843 @@ -0,0 +1,5 @@ + o Minor bugfixes: + - Our new buildsystem was overzealous about rebuilding manpages: it + would rebuild them all whenever any one of them changed. Now our + dependency checking should be correct. Fixes bug 6843; bugfix on + 0.2.4.1-alpha. diff --git a/doc/include.am b/doc/include.am index bb01038..9eb919b 100644 --- a/doc/include.am +++ b/doc/include.am @@ -46,24 +46,46 @@ asciidoc_product = $(nodist_man1_MANS) $(doc_DATA) # Generate the html documentation from asciidoc, but don't do # machine-specific replacements yet -$(html_in) : $(txt_in) +$(html_in) : $(AM_V_GEN)$(top_srcdir)/doc/asciidoc-helper.sh html @ASCIIDOC@ $(top_srcdir)/$@ # Generate the manpage from asciidoc, but don't do # machine-specific replacements yet -$(man_in) : $(txt_in) +$(man_in) : $(AM_V_GEN)$(top_srcdir)/doc/asciidoc-helper.sh man @A2X@ $(top_srcdir)/$@ +doc/tor.1.in: doc/tor.1.txt +doc/tor-gencert.1.in: doc/tor-gencert.1.txt +doc/tor-resolve.1.in: doc/tor-resolve.1.txt +doc/torify.1.in: doc/torify.1.txt +doc/tor-fw-helper.1.in: doc/tor-fw-helper.1.txt + +doc/tor.html.in: doc/tor.1.txt +doc/tor-gencert.html.in: doc/tor-gencert.1.txt +doc/tor-resolve.html.in: doc/tor-resolve.1.txt +doc/torify.html.in: doc/torify.1.txt +doc/tor-fw-helper.html.in: doc/tor-fw-helper.1.txt + # use ../config.status to swap all machine-specific magic strings # in the asciidoc with their replacements. -$(asciidoc_product) : $(txt_in) $(man_in) +$(asciidoc_product) : $(AM_V_GEN)$(MKDIR_P) $(@D) $(AM_V_at)if test -e $(top_srcdir)/$@.in && ! test -e $@.in ; then \ cp $(top_srcdir)/$@.in $@; \ fi $(AM_V_at)./config.status -q --file=$@; -$(doc_DATA) : $(html_in) +doc/tor.html: doc/tor.html.in +doc/tor-gencert.html: doc/tor-gencert.html.in +doc/tor-resolve.html: doc/tor-resolve.html.in +doc/torify.html: doc/torify.html.in +doc/tor-fw-helper.html: doc/tor-fw-helper.html.in + +doc/tor.1: doc/tor.1.in +doc/tor-gencert.1: doc/tor-gencert.1.in +doc/tor-resolve.1: doc/tor-resolve.1.in +doc/torify.1: doc/torify.1.in +doc/tor-fw-helper.1: doc/tor-fw-helper.1.in CLEANFILES+= $(asciidoc_product) config.log DISTCLEANFILES+= $(html_in) $(man_in)

1 0

[tor/master] Clarify that hidden services are TCP only
by nickm＠torproject.org 19 Sep '12

19 Sep '12

commit b9c86948e4f4f0af15e88f74a7297f424ed37f6b Author: Nick Mathewson <nickm(a)torproject.org> Date: Wed Sep 19 08:18:19 2012 -0400 Clarify that hidden services are TCP only Also remove some trailing whitespace. Patch from maker; fixes bug 6024. --- changes/bug6024 | 2 ++ doc/tor.1.txt | 19 ++++++++++--------- 2 files changed, 12 insertions(+), 9 deletions(-) diff --git a/changes/bug6024 b/changes/bug6024 new file mode 100644 index 0000000..743e6ef --- /dev/null +++ b/changes/bug6024 @@ -0,0 +1,2 @@ + o Documentation fixes: + - Clarify that hidden services are TCP only. Fixes bug 6024. diff --git a/doc/tor.1.txt b/doc/tor.1.txt index 5aac054..773fccf 100644 --- a/doc/tor.1.txt +++ b/doc/tor.1.txt @@ -237,7 +237,7 @@ GENERAL OPTIONS recommend that you leave this alone unless you know what you're doing, since giving attackers access to your control listener is really dangerous. This directive can be specified multiple - times to bind to multiple addresses/ports. (Default: 127.0.0.1) + times to bind to multiple addresses/ports. (Default: 127.0.0.1) **ControlSocket** __Path__:: Like ControlPort, but listens on a Unix domain socket, rather than a TCP @@ -762,7 +762,7 @@ The following options are useful only for clients (that is, if purposes, e.g., for Tor controllers. This option may be used multiple times for different hidden services. If a hidden service uses authorization and this option is not set, the hidden service is not accessible. Hidden - services can be configured to require authorization using the + services can be configured to require authorization using the **HiddenServiceAuthorizeClient** option. **CloseHSClientCircuitsImmediatelyOnTimeout** **0**|**1**:: @@ -1017,7 +1017,7 @@ The following options are useful only for clients (that is, if Open this port to listen for transparent proxy connections. Set this to 0 if you don't want to allow transparent proxy connections. Set the port to "auto" to have Tor pick a port for you. This directive can be - specified multiple times to bind to multiple addresses/ports. See + specified multiple times to bind to multiple addresses/ports. See SOCKSPort for an explanation of isolation flags. + + TransPort requires OS support for transparent proxies, such as BSDs' pf or @@ -1055,7 +1055,7 @@ The following options are useful only for clients (that is, if **AutomapHostsOnResolve** **0**|**1**:: When this option is enabled, and we get a request to resolve an address that ends with one of the suffixes in **AutomapHostsSuffixes**, we map an - unused virtual address to that address, and return the new virtual address. + unused virtual address to that address, and return the new virtual address. This is handy for making ".onion" addresses work with applications that resolve an address and then connect to it. (Default: 0) @@ -1747,10 +1747,11 @@ The following options are used to configure a hidden service. Configure a virtual port VIRTPORT for a hidden service. You may use this option multiple times; each time applies to the service using the most recent hiddenservicedir. By default, this option maps the virtual port to - the same port on 127.0.0.1. You may override the target port, address, or - both by specifying a target of addr, port, or addr:port. You may also have - multiple lines with the same VIRTPORT: when a user connects to that - VIRTPORT, one of the TARGETs from those lines will be chosen at random. + the same port on 127.0.0.1 over TCP. You may override the target port, + address, or both by specifying a target of addr, port, or addr:port. + You may also have multiple lines with the same VIRTPORT: when a user + connects to that VIRTPORT, one of the TARGETs from those lines will be + chosen at random. **PublishHidServDescriptors** **0**|**1**:: If set to 0, Tor will run any hidden services you configure, but it won't @@ -1775,7 +1776,7 @@ The following options are used to configure a hidden service. their configuration file using **HidServAuth**. **RendPostPeriod** __N__ **seconds**|**minutes**|**hours**|**days**|**weeks**:: - Every time the specified period elapses, Tor uploads any rendezvous + Every time the specified period elapses, Tor uploads any rendezvous service descriptors to the directory servers. This information is also uploaded whenever it changes. (Default: 1 hour)

1 0

[tor/master] Merge remote-tracking branch 'origin/maint-0.2.3'
by nickm＠torproject.org 19 Sep '12

19 Sep '12

commit f612a9eb1bfc32849f649d2071b9d4d9c9b51053 Merge: d1c4cf2 b9c8694 Author: Nick Mathewson <nickm(a)torproject.org> Date: Wed Sep 19 08:26:07 2012 -0400 Merge remote-tracking branch 'origin/maint-0.2.3' changes/bug6024 | 2 ++ doc/tor.1.txt | 19 ++++++++++--------- 2 files changed, 12 insertions(+), 9 deletions(-)

1 0

[tor/maint-0.2.3] Clarify that hidden services are TCP only
by nickm＠torproject.org 19 Sep '12

19 Sep '12

commit b9c86948e4f4f0af15e88f74a7297f424ed37f6b Author: Nick Mathewson <nickm(a)torproject.org> Date: Wed Sep 19 08:18:19 2012 -0400 Clarify that hidden services are TCP only Also remove some trailing whitespace. Patch from maker; fixes bug 6024. --- changes/bug6024 | 2 ++ doc/tor.1.txt | 19 ++++++++++--------- 2 files changed, 12 insertions(+), 9 deletions(-) diff --git a/changes/bug6024 b/changes/bug6024 new file mode 100644 index 0000000..743e6ef --- /dev/null +++ b/changes/bug6024 @@ -0,0 +1,2 @@ + o Documentation fixes: + - Clarify that hidden services are TCP only. Fixes bug 6024. diff --git a/doc/tor.1.txt b/doc/tor.1.txt index 5aac054..773fccf 100644 --- a/doc/tor.1.txt +++ b/doc/tor.1.txt @@ -237,7 +237,7 @@ GENERAL OPTIONS recommend that you leave this alone unless you know what you're doing, since giving attackers access to your control listener is really dangerous. This directive can be specified multiple - times to bind to multiple addresses/ports. (Default: 127.0.0.1) + times to bind to multiple addresses/ports. (Default: 127.0.0.1) **ControlSocket** __Path__:: Like ControlPort, but listens on a Unix domain socket, rather than a TCP @@ -762,7 +762,7 @@ The following options are useful only for clients (that is, if purposes, e.g., for Tor controllers. This option may be used multiple times for different hidden services. If a hidden service uses authorization and this option is not set, the hidden service is not accessible. Hidden - services can be configured to require authorization using the + services can be configured to require authorization using the **HiddenServiceAuthorizeClient** option. **CloseHSClientCircuitsImmediatelyOnTimeout** **0**|**1**:: @@ -1017,7 +1017,7 @@ The following options are useful only for clients (that is, if Open this port to listen for transparent proxy connections. Set this to 0 if you don't want to allow transparent proxy connections. Set the port to "auto" to have Tor pick a port for you. This directive can be - specified multiple times to bind to multiple addresses/ports. See + specified multiple times to bind to multiple addresses/ports. See SOCKSPort for an explanation of isolation flags. + + TransPort requires OS support for transparent proxies, such as BSDs' pf or @@ -1055,7 +1055,7 @@ The following options are useful only for clients (that is, if **AutomapHostsOnResolve** **0**|**1**:: When this option is enabled, and we get a request to resolve an address that ends with one of the suffixes in **AutomapHostsSuffixes**, we map an - unused virtual address to that address, and return the new virtual address. + unused virtual address to that address, and return the new virtual address. This is handy for making ".onion" addresses work with applications that resolve an address and then connect to it. (Default: 0) @@ -1747,10 +1747,11 @@ The following options are used to configure a hidden service. Configure a virtual port VIRTPORT for a hidden service. You may use this option multiple times; each time applies to the service using the most recent hiddenservicedir. By default, this option maps the virtual port to - the same port on 127.0.0.1. You may override the target port, address, or - both by specifying a target of addr, port, or addr:port. You may also have - multiple lines with the same VIRTPORT: when a user connects to that - VIRTPORT, one of the TARGETs from those lines will be chosen at random. + the same port on 127.0.0.1 over TCP. You may override the target port, + address, or both by specifying a target of addr, port, or addr:port. + You may also have multiple lines with the same VIRTPORT: when a user + connects to that VIRTPORT, one of the TARGETs from those lines will be + chosen at random. **PublishHidServDescriptors** **0**|**1**:: If set to 0, Tor will run any hidden services you configure, but it won't @@ -1775,7 +1776,7 @@ The following options are used to configure a hidden service. their configuration file using **HidServAuth**. **RendPostPeriod** __N__ **seconds**|**minutes**|**hours**|**days**|**weeks**:: - Every time the specified period elapses, Tor uploads any rendezvous + Every time the specified period elapses, Tor uploads any rendezvous service descriptors to the directory servers. This information is also uploaded whenever it changes. (Default: 1 hour)

1 0

[metrics-tasks/master] Fix two bugs in current #1854 code.
by karsten＠torproject.org 19 Sep '12

19 Sep '12

commit 221002f39f852d60fdd71b620545ae755748d7c5 Author: Karsten Loesing <karsten.loesing(a)gmx.net> Date: Tue Sep 18 19:13:10 2012 -0400 Fix two bugs in current #1854 code. Relays shouldn't be excluded based on their position-dependent weighted consensus weight, but on their consensus weight directly taken from the consensus. No multiplying by Wxx weights necessary for that. The current for loop to remove routers with min cw removed only every second element. Use a while loop instead. --- task-1854/pyentropy.py | 24 ++++-------------------- 1 files changed, 4 insertions(+), 20 deletions(-) diff --git a/task-1854/pyentropy.py b/task-1854/pyentropy.py index 4ee003c..3899810 100644 --- a/task-1854/pyentropy.py +++ b/task-1854/pyentropy.py @@ -36,7 +36,6 @@ class Router: self.as_no = None self.is_exit = None self.is_guard = None - self.cw = None def add_router_info(self, values): hex_digest = b2a_hex(a2b_base64(values[2]+"=")) @@ -116,20 +115,8 @@ def run(file_name): if len(routers) <= 0: return - # calculate consensus weight for each relay - for router in routers: - if not router.bandwidth: - # should we consider relays with 'None' bandwidth? - continue - if router.is_guard and router.is_exit: - router.cw = Wgd*Wed*router.bandwidth - elif router.is_guard: - router.cw = Wgg*router.bandwidth - elif router.is_exit: - router.cw = Wee*router.bandwidth - # sort list of routers based on consensus weight - routers.sort(key=lambda router: router.cw) + routers.sort(key=lambda router: router.bandwidth) while(len(routers)>1): total_bw, total_exit_bw, total_guard_bw = 0, 0, 0 @@ -137,7 +124,7 @@ def run(file_name): bw_countries, bw_as = {}, {} max_entropy, max_entropy_as, max_entropy_guard, max_entropy_country, max_entropy_exit = 0.0, 0.0, 0.0, 0.0, 0.0 # first relay has smallest cw - min_cw = routers[0].cw + min_cw = routers[0].bandwidth for router in routers: if not router.bandwidth: @@ -223,11 +210,8 @@ def run(file_name): str(max_entropy_as)])) # remove routers with min cw - for id, router in enumerate(routers): - if router.cw == min_cw: - del routers[id] - else: - break + while len(routers) > 0 and routers[0].bandwidth == min_cw: + del routers[0] return '\n'.join(result_string)

1 0

[metrics-tasks/master] Add .gitignore (#1854).
by karsten＠torproject.org 19 Sep '12

19 Sep '12

commit 9e28f9cf2b3d6e57b5763b0d04af2d01f7f3a4f4 Author: Karsten Loesing <karsten.loesing(a)gmx.net> Date: Tue Sep 18 19:12:47 2012 -0400 Add .gitignore (#1854). --- task-1854/.gitignore | 7 +++++++ 1 files changed, 7 insertions(+), 0 deletions(-) diff --git a/task-1854/.gitignore b/task-1854/.gitignore new file mode 100644 index 0000000..ada3ff3 --- /dev/null +++ b/task-1854/.gitignore @@ -0,0 +1,7 @@ +GeoIP.dat +GeoIPASNum.dat +entropy.csv +in/ +pygeoip/ +stem/ +

1 0

[metrics-tasks/master] Add graphing code for #1854.
by karsten＠torproject.org 19 Sep '12

19 Sep '12

commit 2ed7526d2b22b66dfebd035baaeb47f1cb94657a Author: Karsten Loesing <karsten.loesing(a)gmx.net> Date: Tue Sep 18 19:45:37 2012 -0400 Add graphing code for #1854. --- task-1854/.gitignore | 2 + task-1854/plot-entropy.R | 53 ++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 55 insertions(+), 0 deletions(-) diff --git a/task-1854/.gitignore b/task-1854/.gitignore index ada3ff3..f6115f4 100644 --- a/task-1854/.gitignore +++ b/task-1854/.gitignore @@ -4,4 +4,6 @@ entropy.csv in/ pygeoip/ stem/ +Rplots.pdf +*.png diff --git a/task-1854/plot-entropy.R b/task-1854/plot-entropy.R new file mode 100644 index 0000000..e28193f --- /dev/null +++ b/task-1854/plot-entropy.R @@ -0,0 +1,53 @@ +library(ggplot2) +library(reshape) +library(scales) + +d <- read.csv("entropy.csv", header = FALSE, + col.names = c("validafter", "min_cw", "relays", "all", "max_all", + "exit", "max_exit", "guard", "max_guard", "country", "max_country", + "as", "max_as"), stringsAsFactor = FALSE) +max_validafter <- max(d$validafter) +d <- d[d$validafter == max_validafter, ] + +e <- d[d$min_cw <= 10000, ] +e <- data.frame(min_cw = e$min_cw, all = e$all / e$max_all, + exit = e$exit / e$max_exit, guard = e$guard / e$max_guard, + country = e$country / e$max_country, as = e$as / e$max_as) +e <- melt(e, "min_cw") +e <- data.frame(min_cw = e$min_cw, variable = + ifelse(e$variable == "all", "All relays", + ifelse(e$variable == "exit", "All exits", + ifelse(e$variable == "guard", "All guards", + ifelse(e$variable == "country", "All countries", + "All ASes")))), value = e$value) +ggplot(e, aes(x = min_cw, y = value)) + +geom_line() + +facet_wrap(~ variable) + +scale_x_continuous(name = "\nMinimum consensus weight") + +scale_y_continuous(name = "", limits = c(0, 1), labels = percent) + +opts(title = paste("Degree of anonymity based on consensus from ", + max_validafter, "\n", sep = "")) +ggsave("degree-of-anonymity-min-cw.png", width = 8, height = 6, dpi = 100) + +f <- d[d$min_cw <= 10000, ] +f <- rbind( + data.frame(min_cw = f$min_cw, entropy = f$all, max = f$max_all, + type = "All relays"), + data.frame(min_cw = f$min_cw, entropy = f$exit, max = f$max_exit, + type = "All exits"), + data.frame(min_cw = f$min_cw, entropy = f$guard, max = f$max_guard, + type = "All guards"), + data.frame(min_cw = f$min_cw, entropy = f$country, max = f$max_country, + type = "All countries"), + data.frame(min_cw = f$min_cw, entropy = f$as, max = f$max_as, + type = "All ASes")) +f <- melt(f, c("min_cw", "type")) +ggplot(f, aes(x = min_cw, y = value, colour = variable)) + +geom_line() + +facet_wrap(~ type) + +scale_x_continuous(name = "\nMinimum consensus weight") + +scale_y_continuous(name = "", limits = c(0, max(f$value))) + +opts(title = paste("Entropy and maximum entropy based on consensus from ", + max_validafter, "\n", sep = ""), legend.position = "none") +ggsave("entropy-min-cw.png", width = 8, height = 6, dpi = 100) +

1 0

[metrics-tasks/master] Add entropy code for #1854
by karsten＠torproject.org 19 Sep '12

19 Sep '12

commit ad848ad07f14ed75b5e464b20a069928076f8e36 Author: Sathyanarayanan Gunasekaran <gsathya.ceg(a)gmail.com> Date: Sat Sep 15 17:49:56 2012 +0530 Add entropy code for #1854 --- task-1854/pyentropy.py | 262 ++++++++++++++++++++++++++++++++++++++++++++++++ 1 files changed, 262 insertions(+), 0 deletions(-) diff --git a/task-1854/pyentropy.py b/task-1854/pyentropy.py new file mode 100644 index 0000000..4ee003c --- /dev/null +++ b/task-1854/pyentropy.py @@ -0,0 +1,262 @@ +""" +Usage - python pyentropy.py -h +Output - A CSV file of the format (without newlines): + <valid-after>, + <min consensus weight>, + <number of relays>, + <entropy for all nodes>, + <max entropy for all nodes>, + <entropy for exit nodes>, + <max entropy for exit nodes>, + <entropy for guard nodes>, + <max entropy for guard nodes>, + <entropy for countries>, + <max entropy for countries>, + <entropy for AS>, + <max entropy for AS> +rsync -arz --delete metrics.torproject.org::metrics-recent/relay-descriptors/consensuses in +""" + +import sys +import math +import os +import pygeoip +import StringIO +import stem.descriptor + +from optparse import OptionParser +from binascii import b2a_hex, a2b_base64, a2b_hex +from stem.descriptor.server_descriptor import RelayDescriptor, BridgeDescriptor + +class Router: + def __init__(self): + self.bandwidth = None + self.advertised_bw = None + self.country = None + self.as_no = None + self.is_exit = None + self.is_guard = None + self.cw = None + + def add_router_info(self, values): + hex_digest = b2a_hex(a2b_base64(values[2]+"=")) + self.advertised_bw = self.get_advertised_bw(hex_digest) + ip = values[5] + self.country = gi_db.country_code_by_addr(ip) + self.as_no = self.get_as_details(ip) + + def add_weights(self, values): + self.bandwidth = int(values[0].split('=')[1]) + + def add_flags(self, values): + if "Exit" in values and not "BadExit" in values: + self.is_exit = True + if "Guard" in values: + self.is_guard = True + + def get_as_details(self, ip): + try: + value = as_db.org_by_addr(str(ip)).split() + return value[0] + except: + return "" + + def get_advertised_bw(self, hex_digest): + try: + with open(options.server_desc+hex_digest) as f: + data = f.read() + + desc_iter = stem.descriptor.server_descriptor.parse_file(StringIO.StringIO(data)) + desc_entries = list(desc_iter) + desc = desc_entries[0] + return min(desc.average_bandwidth, desc.burst_bandwidth, desc.observed_bandwidth) + except: + return 0 + +def parse_bw_weights(values): + data = {} + try: + for value in values: + key, value = value.split("=") + data[key] = float(value) / 10000 + return data + except: + return None + +def run(file_name): + routers = [] + router = None + result_string = [] + Wed, Wee, Wgd, Wgg = 1, 1, 1, 1 + # parse consensus + with open(file_name, 'r') as f: + for line in f.readlines(): + key = line.split()[0] + values = line.split()[1:] + if key =='r': + router = Router() + routers.append(router) + router.add_router_info(values) + elif key == 's': + router.add_flags(values) + elif key == 'w': + router.add_weights(values) + elif key == 'valid-after': + valid_after = ' '.join(values) + elif key == 'bandwidth-weights': + data = parse_bw_weights(values) + try: + Wed = data['Wed'] + Wee = data['Wee'] + Wgd = data['Wgd'] + Wgg = data['Wgg'] + except: + pass + + if len(routers) <= 0: + return + + # calculate consensus weight for each relay + for router in routers: + if not router.bandwidth: + # should we consider relays with 'None' bandwidth? + continue + if router.is_guard and router.is_exit: + router.cw = Wgd*Wed*router.bandwidth + elif router.is_guard: + router.cw = Wgg*router.bandwidth + elif router.is_exit: + router.cw = Wee*router.bandwidth + + # sort list of routers based on consensus weight + routers.sort(key=lambda router: router.cw) + + while(len(routers)>1): + total_bw, total_exit_bw, total_guard_bw = 0, 0, 0 + guards_no, exits_no = 0, 0 + bw_countries, bw_as = {}, {} + max_entropy, max_entropy_as, max_entropy_guard, max_entropy_country, max_entropy_exit = 0.0, 0.0, 0.0, 0.0, 0.0 + # first relay has smallest cw + min_cw = routers[0].cw + + for router in routers: + if not router.bandwidth: + continue + total_bw += router.bandwidth + if router.is_guard and router.is_exit: + total_guard_bw += Wgd*router.bandwidth + total_exit_bw += Wed*router.bandwidth + guards_no += 1 + exits_no += 1 + elif router.is_guard: + total_guard_bw += Wgg*router.bandwidth + guards_no += 1 + elif router.is_exit: + total_exit_bw += Wee*router.bandwidth + exits_no += 1 + if bw_countries.has_key(router.country): + bw_countries[router.country] += router.bandwidth + else: + bw_countries[router.country] = router.bandwidth + if bw_as.has_key(router.as_no): + bw_as[router.as_no] += router.bandwidth + else: + bw_as[router.as_no] = router.bandwidth + + if total_bw == 0: + return + + entropy, entropy_exit, entropy_guard, entropy_country, entropy_as = 0.0, 0.0, 0.0, 0.0, 0.0 + for router in routers: + p = float(router.bandwidth) / float(total_bw) + if p != 0: + entropy += -(p * math.log(p, 2)) + if router.is_guard and router.is_exit: + p = float(Wgd*router.bandwidth) / float(total_guard_bw) + if p != 0: + entropy_guard += -(p * math.log(p, 2)) + p = float(Wed*router.bandwidth) / float(total_exit_bw) + if p != 0: + entropy_exit += -(p * math.log(p, 2)) + elif router.is_guard: + p = float(Wgg*router.bandwidth) / float(total_guard_bw) + if p != 0: + entropy_guard += -(p * math.log(p, 2)) + elif router.is_exit: + p = float(Wee*router.bandwidth) / float(total_exit_bw) + if p != 0: + entropy_exit += -(p * math.log(p, 2)) + + for country in bw_countries.iterkeys(): + p = float(bw_countries[country]) / float(total_bw) + if p != 0: + entropy_country += -(p * math.log(p, 2)) + + for as_no in bw_as.iterkeys(): + p = float(bw_as[as_no]) / float(total_bw) + if p !=0: + entropy_as += -(p * math.log(p, 2)) + + # Entropy of uniform distribution of 'n' possible values: log(n) + max_entropy = math.log(len(routers), 2) + if guards_no: + max_entropy_guard = math.log(guards_no, 2) + if exits_no: + max_entropy_exit = math.log(exits_no, 2) + if bw_countries: + max_entropy_country = math.log(len(bw_countries), 2) + if bw_as: + max_entropy_as = math.log(len(bw_as), 2) + + result_string.append(','.join([valid_after, + str(min_cw), + str(len(routers)), + str(entropy), + str(max_entropy), + str(entropy_exit), + str(max_entropy_exit), + str(entropy_guard), + str(max_entropy_guard), + str(entropy_country), + str(max_entropy_country), + str(entropy_as), + str(max_entropy_as)])) + + # remove routers with min cw + for id, router in enumerate(routers): + if router.cw == min_cw: + del routers[id] + else: + break + + return '\n'.join(result_string) + +def parse_args(): + usage = "Usage - python pyentropy.py [options]" + parser = OptionParser(usage) + + parser.add_option("-g", "--geoip", dest="gi_db", default="GeoIP.dat", + help="Input GeoIP database") + parser.add_option("-a", "--as", dest="as_db", default="GeoIPASNum.dat", + help="Input AS GeoIP database") + parser.add_option("-s", "--server_desc", dest="server_desc", + default="data/relay-descriptors/server-descriptors/", help="Server descriptors directory") + parser.add_option("-o", "--output", dest="output", default="entropy.csv", + help="Output filename") + parser.add_option("-c", "--consensus", dest="consensus", default="in/consensus", + help="Input consensus dir") + + (options, args) = parser.parse_args() + + return options + +if __name__ == "__main__": + options = parse_args() + gi_db = pygeoip.GeoIP(options.gi_db) + as_db = pygeoip.GeoIP(options.as_db) + + with open(options.output, 'w') as f: + for file_name in os.listdir(options.consensus): + string = run(os.path.join(options.consensus, file_name)) + if string: + f.write("%s\n" % (string))

1 0