January 2014 - tor-commits - lists.torproject.org

[bridgedb/master] Clarify which bridges are used and which ones are skipped
by isis＠torproject.org 12 Jan '14

12 Jan '14

commit 907ab23075b358c2106cbded24ab7cbdfda17ca0 Author: Matthew Finkel <Matthew.Finkel(a)gmail.com> Date: Thu Aug 15 05:29:25 2013 +0000 Clarify which bridges are used and which ones are skipped If we receive a descriptor for a bridge which was not listed in the network status, then we should not use it. We do not use it, but this is not obvious without some investigation. Make this obvious. --- lib/bridgedb/Main.py | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/lib/bridgedb/Main.py b/lib/bridgedb/Main.py index 8e8179b..87b95fc 100644 --- a/lib/bridgedb/Main.py +++ b/lib/bridgedb/Main.py @@ -134,6 +134,10 @@ def load(state, splitter, clear=False): # single ip:port So for now, we will only consider the bridges # primary IP:port bridge.or_addresses = addresses.get(bridge.getID()) + # We attempt to insert all bridges. If the bridge is not + # running, then it is skipped during the insertion process. Also, + # if we have a descriptor for the bridge but it was not in the + # ns, then we skip it there, too. splitter.insert(bridge) if bridge.getID() in timestamps.keys():

1 0

[bridgedb/master] We now support extra-info documents and pluggable transports
by isis＠torproject.org 12 Jan '14

12 Jan '14

commit aa41406f594a4d7173d108f1b1ae89b38d5554cd Author: Matthew Finkel <Matthew.Finkel(a)gmail.com> Date: Thu Aug 15 03:39:57 2013 +0000 We now support extra-info documents and pluggable transports BridgeDB supports parsing extra-info documents which allows it to retrieve information regarding the pluggable transports a bridge accepts. --- bridge-db-spec.txt | 40 +++++++++++++++++++++++++++++++++------- 1 file changed, 33 insertions(+), 7 deletions(-) diff --git a/bridge-db-spec.txt b/bridge-db-spec.txt index 83a6d7e..e3735c9 100644 --- a/bridge-db-spec.txt +++ b/bridge-db-spec.txt @@ -17,10 +17,11 @@ 1. Importing bridge network statuses and bridge descriptors - BridgeDB learns about bridges from parsing bridge network statuses and - bridge descriptors as specified in Tor's directory protocol. - BridgeDB parses one bridge network status file first and at least one - bridge descriptor file afterwards. + BridgeDB learns about bridges by parsing bridge network statuses, + bridge descriptors, and extra info documents as specified in Tor's + directory protocol. BridgeDB parses one bridge network status file + first and at least one bridge descriptor file and potentially one extra + info file afterwards. BridgeDB scans its files on sighup. @@ -85,6 +86,30 @@ the bridge network status parsed before, it removes that bridge from the set of bridges to be given out to bridge users. +1.3. Parsing extra-info documents + + BridgeDB learns if a bridge supports a pluggable transport by parsing + extra-info documents. + Extra-info documents contain the name of the bridge (but only if it is + named), the bridge's fingerprint, the type of pluggable transport(s) it + supports, and the IP address and port number on which each transport + listens, respectively. + + Extra-info documents may contain zero or more entries per bridge. We expect + an extra-info entry to contain the following lines in the stated order: + + "extra-info" SP name SP fingerprint NL + "transport" SP transport SP IP ":" PORT ARGS NL + + BridgeDB parses the fingerprint, transport type, IP address, port and any + arguments that are specified on these lines. BridgeDB skips the name. If + the fingerprint is invalid, BridgeDB skips the entry. + BridgeDB memorizes the transport type, IP address, port number, and any + arguments that are be provided and then it assigns them to the + corresponding bridge based on the fingerprint. Arguments are space + separated and are of the form k=v. + Bridges that do not have an associated extra-info entry are not invalid. + 2. Assigning bridges to distributors A "distributor" is a mechanism by which bridges are given (or not @@ -270,9 +295,10 @@ The distributor is one out of "email", "https", or "unallocated". - Both "email" and "https" distributors support adding keys for "port" - and "flag" and the port number and flag name as values to indicate that - a bridge matches certain port or flag criteria of requests. + Both "email" and "https" distributors support adding keys for "port", + "flag" and "transport". Respectively, the port number, flag name, and + transport types are the values. These are used to indicate that + a bridge matches certain port, flag, transport criteria of requests. The "https" distributor also allows the key "ring" with a number as value to indicate to which IP address area the bridge is returned.

1 0

[bridgedb/master] We use the "a" line of a bridge from the NS for its OR Address
by isis＠torproject.org 12 Jan '14

12 Jan '14

commit e77ae419b1fddf22b27bff6474aa586418096669 Author: Matthew Finkel <Matthew.Finkel(a)gmail.com> Date: Thu Aug 15 03:45:36 2013 +0000 We use the "a" line of a bridge from the NS for its OR Address The "a" line in the network status is checked and signed by the bridge authority, so we prefer this over the "or-address" line in the bridge descriptor. --- bridge-db-spec.txt | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/bridge-db-spec.txt b/bridge-db-spec.txt index e3735c9..0e07dcc 100644 --- a/bridge-db-spec.txt +++ b/bridge-db-spec.txt @@ -40,10 +40,13 @@ "r" SP nickname SP identity SP digest SP publication SP IP SP ORPort SP DirPort NL + "a" SP address ":" port NL (no more than 8 instances) "s" SP Flags NL - BridgeDB parses the identity from the "r" line and the assigned flags - from the "s" line. + BridgeDB parses the identity and the publication timestamp from the "r" + line, the OR address(es) and ORPort(s) from the "a" line(s), and the + assigned flags from the "s" line, specifically checking the assignment + of the "Running" and "Stable" flags. BridgeDB memorizes all bridges that have the Running flag as the set of running bridges that can be given out to bridge users. BridgeDB memorizes assigned flags if it wants to ensure that sets of

1 0

[bridgedb/master] Clarify how we parse bridge descriptors
by isis＠torproject.org 12 Jan '14

12 Jan '14

commit 3a636411f34754c5e5c68bb79ae4765598106116 Author: Matthew Finkel <Matthew.Finkel(a)gmail.com> Date: Thu Aug 15 03:56:11 2013 +0000 Clarify how we parse bridge descriptors Bring the description inline with how the the implementation handles the descriptors and bridges. --- bridge-db-spec.txt | 11 ++++++----- 1 file changed, 6 insertions(+), 5 deletions(-) diff --git a/bridge-db-spec.txt b/bridge-db-spec.txt index 0e07dcc..aa8af20 100644 --- a/bridge-db-spec.txt +++ b/bridge-db-spec.txt @@ -70,11 +70,12 @@ "@purpose" SP purpose NL "router" SP nickname SP IP SP ORPort SP SOCKSPort SP DirPort NL ["opt" SP] "fingerprint" SP fingerprint NL + "router-signature" NL Signature NL - BridgeDB parses the purpose, IP, ORPort, and fingerprint from these - lines. + BridgeDB parses the purpose, IP, ORPort, nickname, and fingerprint + from these lines. BridgeDB skips bridge descriptors if the fingerprint is not contained - in the bridge network status parsed before or if the bridge does not + in the bridge network status parsed earlier or if the bridge does not have the Running flag. BridgeDB discards bridge descriptors which have a different purpose than "bridge". BridgeDB can be configured to only accept descriptors @@ -86,8 +87,8 @@ BridgeDB memorizes the IP address and OR port of the most recently parsed bridge descriptor. If BridgeDB does not find a bridge descriptor for a bridge contained in - the bridge network status parsed before, it removes that bridge from - the set of bridges to be given out to bridge users. + the bridge network status parsed before, it does not add that bridge + to the set of bridges to be given out to bridge users. 1.3. Parsing extra-info documents

1 0

[bridgedb/master] Reword and improve description of the distributor
by isis＠torproject.org 12 Jan '14

12 Jan '14

commit 76e57aeb25f28e3d013a4b0306f9b22789cf04f1 Author: Matthew Finkel <Matthew.Finkel(a)gmail.com> Date: Thu Aug 15 04:06:33 2013 +0000 Reword and improve description of the distributor --- bridge-db-spec.txt | 29 +++++++++++++++++------------ 1 file changed, 17 insertions(+), 12 deletions(-) diff --git a/bridge-db-spec.txt b/bridge-db-spec.txt index aa8af20..567e362 100644 --- a/bridge-db-spec.txt +++ b/bridge-db-spec.txt @@ -139,16 +139,19 @@ Upon receiving a client request, a BridgeDB distributor provides a subset of the bridges assigned to it. BridgeDB only gives out bridges that are contained in the most recently - parsed bridge network status and that have the Running flag set. + parsed bridge network status and that have the Running flag set (see + Section 1). BridgeDB may be configured to give out a different number of bridges - (typically 3) depending on the distributor. - BridgeDB may define an arbitrary number of rules saying that a certain - number of bridges should have a given OR port or a given bridge relay - flag. + (typically 4) depending on the distributor. + BridgeDB may define an arbitrary number of rules. These rules may + specify the criteria by which a bridge is selected. Specifically, + the available rules restrict the IP address version, OR port number, + transport type, bridge relay flag, or country in which the bridge + should not be blocked. 4. Selecting bridges to be given out based on IP addresses - BridgeDB may be configured to support one or more distributors that + BridgeDB may be configured to support one or more distributors which gives out bridges based on the requestor's IP address. Currently, this is how the HTTPS distributor works. The goal is to avoid handing out all the bridges to users in a similar @@ -158,11 +161,13 @@ BridgeDB fixes the set of bridges to be returned for a defined time period. - BridgeDB considers all IP addresses coming from the same /24 as the - same IP address and returns the same set of bridges. + BridgeDB considers all IP addresses coming from the same /24 network + as the same IP address and returns the same set of bridges. From here on, + this non-unique address will be referred to as the IP address's 'area'. BridgeDB divides the IP address space equally into a small number of - areas (typically 4) and return different results to requests coming - from these areas. +# Note, changed term from "areas" to "disjoint clusters" -MF + disjoint clusters (typically 4) and returns different results for requests + coming from addresses that are placed into different clusters. # I found that BridgeDB is not strict in returning only bridges for a # given area. If a ring is empty, it considers the next one. Is this # expected behavior? -KL @@ -200,8 +205,8 @@ When giving out bridges based on a position in a ring, BridgeDB first looks at flag requirements and port requirements. For example, BridgeDB may be configured to "Give out at least L bridges with port - 443, and at least M bridges with Stable, and at least N bridges - total." To do this, BridgeDB adds to the results: + 443, and at least M bridges with Stable, and at most N bridges + total." To do this, BridgeDB combines to the results: - The first L bridges in the ring after the position that have the port 443, and - The first M bridges in the ring after the position that have the

1 0

[bridgedb/master] Describe how we decide the number of bridges we will return
by isis＠torproject.org 12 Jan '14

12 Jan '14

commit ff82ea4dcd31375f46569c0581c2269ee25038e6 Author: Matthew Finkel <Matthew.Finkel(a)gmail.com> Date: Thu Sep 26 22:00:14 2013 +0000 Describe how we decide the number of bridges we will return --- bridge-db-spec.txt | 20 ++++++++++++++++++++ 1 file changed, 20 insertions(+) diff --git a/bridge-db-spec.txt b/bridge-db-spec.txt index 96dd83c..c7adf38 100644 --- a/bridge-db-spec.txt +++ b/bridge-db-spec.txt @@ -222,6 +222,23 @@ - The first N-L-M bridges in the ring after the position that it has not already decided to give out. + After BridgeDB selects appropriate bridges to return to the requestor, it + then prioritises the ordering of them in a list so that as many criteria + are fulfilled as possible within the first few bridges. This list is then + truncated to N bridges, if possible. N is currently defined as a + piecewise function of the number of bridges in the ring such that: + + / + | 1, if len(ring) < 20 + | + N = | 2, if 20 <= len(ring) <= 100 + | + | 3, if 100 <= len(ring) + \ + + The bridges in this sublist, containing no more than N bridges, are the + bridges returned to the requestor. + 5. Selecting bridges to be given out based on email addresses BridgeDB can be configured to support one or more distributors that are @@ -273,6 +290,9 @@ - It hands out bridges starting at that position, based on the port/flag requirements, as specified at the end of section 4. + See section 4 for the details of how bridges are selected from the ring + and returned to the requestor. + 6. Selecting unallocated bridges to be stored in file buckets # Kaner should have a look at this section. -NM

1 0

[bridgedb/master] Improve description of IP-based distributor
by isis＠torproject.org 12 Jan '14

12 Jan '14

commit 07faf6cdba0e98a7501b32d0eba6aa0acf54b2da Author: Matthew Finkel <Matthew.Finkel(a)gmail.com> Date: Thu Aug 15 04:16:32 2013 +0000 Improve description of IP-based distributor We now use a CAPTCHA to weed out (most) scrapers. Also, improve description of how bridges are selected based on filters and rings. --- bridge-db-spec.txt | 16 ++++++++++++---- 1 file changed, 12 insertions(+), 4 deletions(-) diff --git a/bridge-db-spec.txt b/bridge-db-spec.txt index 567e362..96dd83c 100644 --- a/bridge-db-spec.txt +++ b/bridge-db-spec.txt @@ -178,8 +178,11 @@ set of bridges to requests coming from these IP addresses. The bridges returned to proxy IP addresses do not come from the same set as those for the general IP address space. + BridgeDB can be configured to include bridge fingerprints in replies along with bridge IP addresses and OR ports. + BridgeDB can be configured to display a CAPTCHA which the user must solve + prior to returning the requested bridges. The current algorithm is as follows. An IP-based distributor splits the bridges uniformly into a set of "rings" based on an HMAC of their @@ -191,10 +194,15 @@ maps the IP into an "area" (that is, a /24), and then uses an HMAC to map the area to one of the area rings. - Once the IP-based distributor knows what ring it is handing out bridges - from, it maps the current "epoch" (N-hour period) and the IP's area - (/24) to a point in the ring based on HMAC, and hands out bridges at - that point. + When the IP-based distributor determines from which area ring it is handing + out bridges, it identifies which rules it will use to choose appropriate + bridges. Using this information, it searches its cache of rings for one + that already adheres to the criteria specified in this request. If one + exists, then BridgeDB maps the current "epoch" (N-hour period) and the + IP's area (/24) to a point on the ring based on HMAC, and hands out + bridges at that point. If a ring does not already exist which satisfies this + request, then a new ring is created and filled with bridges that fulfill + the requirements. This ring is then used to select bridges as described. "Mapping X to Y based on an HMAC" above means one of the following: - We keep all of the elements of Y in some order, with a mapping

1 0

[bridgedb/master] We can sign all outgoing emails now
by isis＠torproject.org 12 Jan '14

12 Jan '14

commit 4bc923c732ff441b11e51d5976c7075e7d3e8ce6 Author: Matthew Finkel <Matthew.Finkel(a)gmail.com> Date: Thu Aug 15 04:24:21 2013 +0000 We can sign all outgoing emails now --- bridge-db-spec.txt | 1 + 1 file changed, 1 insertion(+) diff --git a/bridge-db-spec.txt b/bridge-db-spec.txt index c7adf38..e3f0b10 100644 --- a/bridge-db-spec.txt +++ b/bridge-db-spec.txt @@ -275,6 +275,7 @@ # elaborate? -KL BridgeDB can be configured to include bridge fingerprints in replies along with bridge IP addresses and OR ports. + BridgeDB can be configured to sign all replies using a PGP signing key. BridgeDB periodically discards old email-address-to-bridge mappings. BridgeDB rejects too frequent email requests coming from the same normalized address.

1 0

[bridgedb/master] Add description about how we display bridge addresses
by isis＠torproject.org 12 Jan '14

12 Jan '14

commit ef0328bddd63426058d5e39e75952914e2854d42 Author: Matthew Finkel <Matthew.Finkel(a)gmail.com> Date: Tue Aug 13 23:06:15 2013 +0000 Add description about how we display bridge addresses --- bridge-db-spec.txt | 30 +++++++++++++++++++++++++++--- 1 file changed, 27 insertions(+), 3 deletions(-) diff --git a/bridge-db-spec.txt b/bridge-db-spec.txt index 59750f5..e9facf8 100644 --- a/bridge-db-spec.txt +++ b/bridge-db-spec.txt @@ -102,8 +102,8 @@ Extra-info documents may contain zero or more entries per bridge. We expect an extra-info entry to contain the following lines in the stated order: - "extra-info" SP name SP fingerprint NL - "transport" SP transport SP IP ":" PORT ARGS NL + "extra-info" SP name SP fingerprint NL + "transport" SP transport SP IP ":" PORT ARGS NL BridgeDB parses the fingerprint, transport type, IP address, port and any arguments that are specified on these lines. BridgeDB skips the name. If @@ -336,7 +336,31 @@ # This should be handled the same as if the file bucket is reduced in size. # If X returns, then it should be added to the appropriate distributor. -MF -7. Writing bridge assignments for statistics +7. Displaying Bridge Information + + After bridges are selected using one of the methods described in + Sections 4 - 6, they are output in one of two formats. Bridges are + formatted as: + + <address:port> NL + + Pluggable transports are formatted as: + + <transportname> SP <address:port> [SP arglist] NL + + where arglist is an optional space-separated list of key-value pairs in + the form of k=v. + + Previously, each line was prepended with the "bridge" keyword, such as + + "bridge" SP <address:port> NL + + "bridge" SP <transportname> SP <address:port> [SP arglist] NL + +# We don't do this anymore because Vidalia and TorLauncher don't expect it. +# See the commit message for b70347a9c5fd769c6d5d0c0eb5171ace2999a736. + +8. Writing bridge assignments for statistics BridgeDB can be configured to write bridge assignments to disk for statistical analysis.

1 0

[bridgedb/master] Attempt to answer some old questions, ask a new one.
by isis＠torproject.org 12 Jan '14

12 Jan '14

commit c5f615ecbf12c721db42d4909678b0dea7789d2d Author: Matthew Finkel <Matthew.Finkel(a)gmail.com> Date: Thu Aug 15 04:30:00 2013 +0000 Attempt to answer some old questions, ask a new one. --- bridge-db-spec.txt | 20 ++++++++++++++++++++ 1 file changed, 20 insertions(+) diff --git a/bridge-db-spec.txt b/bridge-db-spec.txt index e3f0b10..59750f5 100644 --- a/bridge-db-spec.txt +++ b/bridge-db-spec.txt @@ -171,6 +171,10 @@ # I found that BridgeDB is not strict in returning only bridges for a # given area. If a ring is empty, it considers the next one. Is this # expected behavior? -KL +# +# This does not appear to be the case, anymore. If a ring is empty, then +# BridgeDB simply returns an empty set of bridges. -MF +# # I also found that BridgeDB does not make the assignment to areas # persistent in the database. So, if we change the number of rings, it # will assign bridges to other rings. I assume this is okay? -KL @@ -253,6 +257,7 @@ ones that RFC2822 allows. BridgeDB may be configured to reject email addresses containing other characters it might not process correctly. +# I don't think we do this, is it worthwhile? -MF BridgeDB rejects email addresses coming from other domains than a configured set of permitted domains. BridgeDB normalizes email addresses by removing "." characters and by @@ -273,6 +278,18 @@ # problem exists for the IP distributor). -NM # I'm afraid I don't fully understand what you mean here. Can you # elaborate? -KL +# +# Assuming an average churn rate, if we use short time periods, then a +# requestor will receive new bridges based on rate-limiting and will (likely) +# eventually work their way around the ring; eventually exhausting all bridges +# available to them from this distributor. If we use a longer time period, +# then each time the period expires there will be more bridges in the ring +# thus reducing the likelihood of all bridges being blocked and increasing +# the time and effort required to enumerate all bridges. (This is my +# understanding, not from Nick) -MF +# Also, we presently need the cache to prevent replays and because if a user +# sent multiple requests with different criteria in each then we would leak +# additional bridges otherwise. -MF BridgeDB can be configured to include bridge fingerprints in replies along with bridge IP addresses and OR ports. BridgeDB can be configured to sign all replies using a PGP signing key. @@ -315,6 +332,9 @@ # cannot put it back in file bucket A, because it's full. Are we going to # add it to a different file bucket? Doesn't that mean that most bridges # will be contained in most file buckets over time? -KL +# +# This should be handled the same as if the file bucket is reduced in size. +# If X returns, then it should be added to the appropriate distributor. -MF 7. Writing bridge assignments for statistics

1 0