February 2014 - tor-commits - lists.torproject.org

[flashproxy/master] Remove unused TEST_ALL.
by dcf＠torproject.org 02 Feb '14

02 Feb '14

commit 4b24df6aed873397c10a6110b345853cdfe28941 Author: David Fifield <david(a)bamsoftware.com> Date: Sat Feb 1 17:09:00 2014 -0800 Remove unused TEST_ALL. --- Makefile.client | 1 - 1 file changed, 1 deletion(-) diff --git a/Makefile.client b/Makefile.client index efdef7c..3d25848 100644 --- a/Makefile.client +++ b/Makefile.client @@ -46,7 +46,6 @@ DST_DOC = $(SRC_DOC) DST_ALL = $(DST_SCRIPT) $(DST_DOC) $(DST_MAN1) TEST_PY = flashproxy-client-test.py -TEST_ALL = $(TEST_PY) REBUILD_MAN = 1

1 0

[flashproxy/master] Use resolved listener addresses for port forwarding.
by dcf＠torproject.org 02 Feb '14

02 Feb '14

commit 945418905fd844913fed6328a2e3e664785ca398 Author: David Fifield <david(a)bamsoftware.com> Date: Sat Feb 1 16:32:51 2014 -0800 Use resolved listener addresses for port forwarding. Previously we used remote_addr, the actual address given on the command line. The difference is that after opening the listener, we know what port ":0" became, so this is now possible: ./flashproxy-client --external --port-forwarding :9001 :0 The other thing we can do is skip IPv6 addresses, because it appears that UPnP and NAT-PMP are for TCP/IPv4 ports only. The above command will produce output like 2014-02-01 17:07:01 Listening remote on 0.0.0.0:40138. 2014-02-01 17:07:01 Listening remote on [::]:55179. 2014-02-01 17:07:01 Listening local on 127.0.0.1:9001. 2014-02-01 17:07:01 Listening local on [::1]:9001. 2014-02-01 17:07:01 Not forwarding to [::]:55179 because it is not an IPv4 address. 2014-02-01 17:07:01 Running port forwarding command: tor-fw-helper -p 40138:40138 --- flashproxy-client | 36 ++++++++++++++++++++++++++---------- 1 file changed, 26 insertions(+), 10 deletions(-) diff --git a/flashproxy-client b/flashproxy-client index ee0b068..6a7667e 100755 --- a/flashproxy-client +++ b/flashproxy-client @@ -19,7 +19,7 @@ import threading import time import traceback -from flashproxy.util import parse_addr_spec, format_addr +from flashproxy.util import parse_addr_spec, addr_family, format_addr from hashlib import sha1 @@ -678,6 +678,27 @@ def forward_ports(pairs): return False return True +def forward_listeners(listeners): + """Attempt to forward the ports belonging to the given listening sockets. + Non-IPv4 addresses are ignored. If options.port_forwarding_external is not + None, only the first IPv4 address in the list will be forwarded.""" + forward_list = [] + for listener in remote_listen: + host, port = socket.getnameinfo(listener.getsockname(), socket.NI_NUMERICHOST | socket.NI_NUMERICSERV) + port = int(port) + af = addr_family(host) + if af != socket.AF_INET: + # I guess tor-fw-helper can only handle IPv4. + log(u"Not forwarding to %s because it is not an IPv4 address." % format_addr((host, port))) + continue + if options.port_forwarding_external is not None: + forward_list.append((options.port_forwarding_external, port)) + # A fixed external address means we can forward only one port. + break + else: + forward_list.append((port, port)) + forward_ports(forward_list) + register_condvar = threading.Condition() # register_flag true means registration_thread_func should register at its next # opportunity. @@ -1165,15 +1186,6 @@ def main(): for method in register_methods: options.register_commands.append(build_register_command(method)) - # Attempt to forward ports if requested. - if options.port_forwarding: - internal = remote_addr[1] - if options.port_forwarding_external is not None: - external = options.port_forwarding_external - else: - external = internal - forward_ports(((external, internal),)) - # Remote sockets, accepting remote WebSocket connections from proxies. remote_listen = [] for addr in options.remote_addrs: @@ -1206,6 +1218,10 @@ def main(): if options.managed: pt_cmethods_done() + # Attempt to forward ports if requested. + if options.port_forwarding: + forward_listeners(remote_listen) + # New remote sockets waiting to finish their WebSocket negotiation. websocket_pending = [] # Remote connection sockets.

1 0

[flashproxy/master] ChangeLog for forwarding port 0.
by dcf＠torproject.org 02 Feb '14

02 Feb '14

commit 8923e2b4f70bf5dc25b8f85260ce039e5f03fda3 Author: David Fifield <david(a)bamsoftware.com> Date: Sat Feb 1 17:33:25 2014 -0800 ChangeLog for forwarding port 0. --- ChangeLog | 3 +++ 1 file changed, 3 insertions(+) diff --git a/ChangeLog b/ChangeLog index 2fcc7b5..da1190e 100644 --- a/ChangeLog +++ b/ChangeLog @@ -1,3 +1,6 @@ + o Allowed the --port-forwarding option to work when the remote port + number is given as 0. + Changes in version 1.5 o Add manpages for the facilitator and nodejs proxy, automatically generated by help2man.

1 0

[bridgedb/develop] Add draft status and related proposals for XXX-bridgedb-social-distribution.txt.
by isis＠torproject.org 01 Feb '14

01 Feb '14

commit d2a29558b5495950be87e6ccfa58012c8ab6b5d4 Author: Isis Lovecruft <isis(a)torproject.org> Date: Wed Oct 23 11:49:28 2013 +0000 Add draft status and related proposals for XXX-bridgedb-social-distribution.txt. --- doc/proposals/XXX-bridgedb-social-distribution.txt | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/doc/proposals/XXX-bridgedb-social-distribution.txt b/doc/proposals/XXX-bridgedb-social-distribution.txt index 538f6ae..5f44fcf 100644 --- a/doc/proposals/XXX-bridgedb-social-distribution.txt +++ b/doc/proposals/XXX-bridgedb-social-distribution.txt @@ -4,7 +4,9 @@ Filename: XXX-social-bridge-distribution.txt Title: Social Bridge Distribution Author: Isis Agora Lovecruft Created: 18 July 2013 -Status: Open +Related Proposals: 199-bridgefinder-integration.txt + XXX-bridgedb-database-improvements.txt +Status: Draft * I. Overview

1 0

[bridgedb/develop] Update social bridge distributor proposal to include terminology.
by isis＠torproject.org 01 Feb '14

01 Feb '14

commit 578abc670ad7e9576f9977d78584a2c8779184c0 Author: Isis Lovecruft <isis(a)torproject.org> Date: Tue Oct 22 02:42:18 2013 +0000 Update social bridge distributor proposal to include terminology. * ADD beginning of threat model as well. * ADD defⁿ table for constants/variable names in doc/proposal/XXX-bridgedb-social-distributor.txt. --- doc/proposals/XXX-bridgedb-social-distribution.txt | 101 ++++++++++++++++---- 1 file changed, 85 insertions(+), 16 deletions(-) diff --git a/doc/proposals/XXX-bridgedb-social-distribution.txt b/doc/proposals/XXX-bridgedb-social-distribution.txt index 199302c..538f6ae 100644 --- a/doc/proposals/XXX-bridgedb-social-distribution.txt +++ b/doc/proposals/XXX-bridgedb-social-distribution.txt @@ -16,7 +16,7 @@ Status: Open such malicious users and/or censoring entities from joining the pool of Tor clients who are able to receive distributed bridges. -* II. Motivation and Problem Scope +* II. Motivation & Problem Scope As it currently stands, Tor bridges which are stored within BridgeDB may be freely requested by any entity at nearly any time. While the openness, that @@ -69,9 +69,72 @@ Status: Open roughly 1000 bridges in the Email Distributor's pool, distributing 3 bridges per email response, -* III. Design +* III. Terminology & Notations +** III.A. Terminology Definitions + + User := A client connecting to BridgeDB in order to obtain bridges. + +** III.B. Notations + +|--------------------+---------------------------------------------------------------------------------------------| +| Symbol | Definition | +|--------------------+---------------------------------------------------------------------------------------------| +| U | A user connecting to BridgeDB in order to obtain bridges, identified by a User Credential | +| D | The bridge distributor, i.e. BridgeDB | +| Gᵐᵃˣ | Upper limit (maximum) number of bridge users for a bridge Bᵢ | +| Gˢᵗᵃʳᵗ | Number of starting users | +| Gᵃᵛᵍ | Average number of users per bridge | +| M | Fraction of users which are malicious | +| B | A bridge | +| {B₁, …, Bᵢ, …, Bₖ} | The set of bridges assigned and given to user U | +| k | The number of bridges which have been given to user U | +| Tᵐⁱⁿ | The minimum time which a bridge must remain reachable | +| Tᶜᵘʳ | The current time, given in Unix Era (UE) seconds notation (an integer, seconds since epoch) | +| Tᵐᵃˣ | The upper bound on the time for which a user U can earn coins from Bᵢ | +| τᵢ | The time when bridge Bᵢ was first given to user U | +| tᵢ | The time from when U was first given Bᵢ to either Tᶜᵘʳ or ßᵢ, whichever is greater | +| ßᵢ | The time when bridge Bᵢ was first considered blocked; if not blocked, ßᵢ = 0 | +| ϕ | Total coins owned by user U | +| φᵢ | The coins which user U has earned thus far from bridge Bᵢ | +| ϱᵢ | Rate of earning coins from bridge Bᵢ | +| λᵢ | The probability that bridge Bᵢ has been blocked | +| ω | The last time that U requested and Invite Ticket from D | +|--------------------+---------------------------------------------------------------------------------------------| + +** III.C. Unicode characters + +* III. Threat Model + + In the original rBridge scheme, there are two separate proposals: the first + does not make any attempt to hide information such as the user's (U) + identity, the list of bridges given to U, the from BridgeDBBridgeDB is + + In our modifications to the rBridge social bridge distribution scheme, + BridgeDB is considered a trusted party, that is to say, BridgeDB is + assumed to be honest in all protocols, and no protections are taken to + protect clients from malicious behaviour from BridgeDB. + +** III.A. Modifications -** III.A. Overview + + + Modification: allow BridgeDB to be a malicious actor (protecting against it + at this point is too costly, instead we want to eliminate BridgeDB's + ability to obtain a social graph for Tor bridge users.) + + +*** 1. BridgeDB is permitted to know the following information: + + + + + XXX finishme + + + +* IV. Design + +** IV.A. Overview As mentioned, most of this proposal is based upon §IV of the rBridge paper, which is the non-privacy preserving portion of the paper. [0] The @@ -88,15 +151,7 @@ Status: Open XXX finishme -** III.B. Threat Model - - Modification: allow BridgeDB to be a malicious actor (protecting against it - at this point is too costly, instead we want to eliminate BridgeDB's - ability to obtain a social graph for Tor bridge users.) - - XXX finishme - -** III.C. Data Formats +** IV.C. Data Formats *** 1. User Credential @@ -136,9 +191,9 @@ Status: Open *** XXX other formats -* IV. Databases +* V. Databases -** IV.A. Scalability Requirements +** V.A. Scalability Requirements Databases SHOULD be implemented in a manner which is ammenable to using a distributed storage system; this is necessary because certain types of data @@ -214,6 +269,13 @@ Status: Open XXX evaluation on data by calling the sha1 for a serverside Lua script [#] [#]: http://redis.io/commands/evalsha + XXX mediawiki notes and references on switching to redis + [#]: https://www.mediawiki.org/wiki/Redis + + XXX using redis as a message queue for job scheduling + [#]: http://www.restmq.com/ + + **** 2.a. Data Structures which should be stored in a RDBMS - User Credentials @@ -222,9 +284,9 @@ Status: Open - Spent Credits -* IV. Open Questions +* VI. Open Questions -** IV.A. In which component of the Tor ecosystem should the client application code go? +** VI.A. In which component of the Tor ecosystem should the client application code go? *** 1. Should this be done as a Pluggable Transport? @@ -290,3 +352,10 @@ Status: Open [6]: https://github.com/andymccurdy/redis-py/ [7]: http://www.dr-josiah.com/2012/03/why-we-didnt-use-bloom-filter.html [8]: http://redis.io/topics/data-types §"Strings" + +[#]: Naor, Moni, and Benny Pinkas. "Efficient oblivious transfer protocols." + Proceedings of the twelfth annual ACM-SIAM symposium on Discrete algorithms. + Society for Industrial and Applied Mathematics, 2001. + http://www.wisdom.weizmann.ac.il/%7Enaor/PAPERS/eotp.ps + https://gitweb.torproject.org/user/isis/bridgedb.git/tree/refs/heads/featur… +

1 0

[bridgedb/develop] Merge branch 'fix/9127-https-interface-ipv6' into develop
by isis＠torproject.org 01 Feb '14

01 Feb '14

commit 95dd010dd9ef9faf69e6034f952a7a95b6743c3b Merge: a90e586 b57f054 Author: Isis Lovecruft <isis(a)torproject.org> Date: Tue Jan 28 19:30:20 2014 +0000 Merge branch 'fix/9127-https-interface-ipv6' into develop lib/bridgedb/templates/base.html | 25 +++++++++++++++++++++++++ 1 file changed, 25 insertions(+)

1 0

[bridgedb/develop] First cobbled together social distributor proposal
by isis＠torproject.org 01 Feb '14

01 Feb '14

commit 4e26736660fb285f8ffb8a6bb7b1c2ad58d72eed Author: Isis Lovecruft <isis(a)torproject.org> Date: Sat Oct 19 12:56:38 2013 +0000 First cobbled together social distributor proposal --- doc/proposals/XXX-bridgedb-social-distribution.txt | 292 ++++++++++++++++++++ 1 file changed, 292 insertions(+) diff --git a/doc/proposals/XXX-bridgedb-social-distribution.txt b/doc/proposals/XXX-bridgedb-social-distribution.txt new file mode 100644 index 0000000..199302c --- /dev/null +++ b/doc/proposals/XXX-bridgedb-social-distribution.txt @@ -0,0 +1,292 @@ +# -*- coding: utf-8 ; mode: org -*- + +Filename: XXX-social-bridge-distribution.txt +Title: Social Bridge Distribution +Author: Isis Agora Lovecruft +Created: 18 July 2013 +Status: Open + +* I. Overview + + This proposal specifies a system for social distribution of the + centrally-stored bridges within BridgeDB. It is primarily based upon Part + IV of the rBridge paper, [0] utilising a coin-based incentivisation scheme + to ensure that malicious users and/or censoring entities are deterred from + blocking bridges, as well as socially-distributed invite tickets to prevent + such malicious users and/or censoring entities from joining the pool of + Tor clients who are able to receive distributed bridges. + +* II. Motivation and Problem Scope + + As it currently stands, Tor bridges which are stored within BridgeDB may be + freely requested by any entity at nearly any time. While the openness, that + is to say public accessibility, of any anonymity system certainly + provisions its users with the protections of a more greatly diversified + anonymity set, the damages to usability, and the efficacy of such an + anonymity system for censorship circumvention, are devastatingly impacted + due to the equal capabilities of both a censoring/malicious entity and an + honest user to request new Tor bridges. + + Thus far, very little has been done to protect the volunteered bridges from + eventually being blocked in various regions. This severely restricts the + overall usability of Tor for clients within these regions, who, arguably, + can be even more in need of the identity protections and free speech + enablement which Tor can provide, given their political contexts. + +** II.A. Current Tor bridge distribution mechanisms and known pitfalls: + +*** 1. HTTP(S) Distributor + + At https://bridges.torproject.org, users may request new bridges, provided + that they are able to pass a CAPTCHA test. Requests through the HTTP(S) + Distributor are not allowed to be made from any current Tor exit relay, + and a hash of the user's actual IP address is used to place them within a + hash ring so that only a subset of the bridges allotted to the HTTP(S) + Distributor's pool may become known to a(n) adversary/user at that IP + address. + +**** 1.a. Known attacks/pitfalls: + + 1) An adversary with a diverse and large IP address space can easily + retrieve some significant portion of the bridges in the HTTPS + Distributor pool. + + 2) The relatively low cost of employing humans to solve CAPTCHAs is not + sufficient to deter adversaries with requisite economic resources from + doing so to obtain bridges. [XXX cost of employment] + +*** 2. Email Distributor + + Clients may send email to bridges(a)bridges.torproject.org with the line + "get bridges" in the body of the email to obtain new bridges. Such emails + must be sent from a Gmail or Yahoo! account, which is required under the + assumption that such accounts are non-trivial to obtain. + +**** 2.a. Known attacks/pitfalls: + + 1) Mechanisms for purchasing pre-registered Gmail accounts en masse + exists, charging between USD$0.25 and USD$0.70 per account. With + roughly 1000 bridges in the Email Distributor's pool, distributing 3 + bridges per email response, + +* III. Design + +** III.A. Overview + + As mentioned, most of this proposal is based upon §IV of the rBridge + paper, which is the non-privacy preserving portion of the paper. [0] The + reasons for deferring implementation of §V include: + + - Adding a simpler out-of-band distribution of bridges. Requiring users to + copy+paste Bridge lines into their torrc is ridiculous. + + - XXX + + Modifications: + + - Remove OT, keep blind signatures and Pedersen's Commitments. + + XXX finishme + +** III.B. Threat Model + + Modification: allow BridgeDB to be a malicious actor (protecting against it + at this point is too costly, instead we want to eliminate BridgeDB's + ability to obtain a social graph for Tor bridge users.) + + XXX finishme + +** III.C. Data Formats + +*** 1. User Credential + + A Credential is a signed document obtained from BridgeDB. It contains all + of the state required to verify honest client behavior, and is formatted + as a JSON object with the following format: + + { "Bridges" : [ + { "BridgeLine" : BridgeLine, + "LearnedTS" : TimeStamp, + "CreditsEarned" : INT + }, + ... + ], + "CrenditialTS" : TimeStamp, + "TotalUnspentCredits" : INT + } NL + + BridgeLine := <Bridge line from BridgeDB> + TimeStamp := INT + NumCredits := INT + + The Timestamp in this case is the time which a user first learned the + existence of that bridge. + + Example: + + {'Bridges': [ + {'BridgeLine': '1.2.3.4:6666 obfs3 adc83b19e793491b1c6ea0fd8b46cd9f32e592fc', + 'CreditsEarned': 5, + 'Timestamp': 1382078292.864117}, + {'BridgeLine': '6.6.6.6:1234 d929c82d2ee727ccbea9c50c669a71075249899f', + 'CreditsEarned': 5, + 'LearnedTS': 1382078292.864117}], + 'CredentialTS': 982398423, + 'TotalUnspentCredits': 10} + +*** XXX other formats + +* IV. Databases + +** IV.A. Scalability Requirements + + Databases SHOULD be implemented in a manner which is ammenable to using a + distributed storage system; this is necessary because certain types of data + MUST be stored permanently, such as the list of hashes of spent tokens, or + the list of hashes of used invite tickets. + + Additionally, doing so promotes modularisation the components of BridgeDB, + such that the BridgeDistributor XXX can be separated from the backend + storage system, BridgeDB. + +*** 1. Distributed Database System + + A distributed database system SHOULD be used for BridgeDB, in order to + scale resources as the number of Tor bridge users grows. This database + system, hereafter referred to as DDBS. + + The DDBS MUST be capable of working within Twisted's asynchronous + framework. If possible, a Object-Relational Mapper (ORM) SHOULD be used to + abstract the database backend's structure and query syntax from the + Twisted Python classes which interact with it, so that the type of + database may be swapped out for another with less code refactoring. + + The DDBM SHALL be used for persistent storage of complex data structures + such as the bridges, which MAY include additional information from both + the XXX @type-bridge-relay descriptors and the @type-bridge-extra-info + descriptors. + + [#]: https://github.com/couchbase/couchbase-python-client#twisted-api + +**** 1.a. Data Structures which should be stored in a DDBS: + + - RedactedDB - The Database of Blocked Bridges + + The RedactedDB will hold entries of bridges which have been discovered + to be unreachable from BridgeDB network vantage point, or have been + reported unreachable by clients. + + - + +*** 2. Relational Database Mapping Server + + For simpler data structures which must be persistently stored, such as the + list of hashes of previously seen Invite Tickets, or the list of + previously spent Tokens, a Relational Database Mapping Server (RDBMS) + SHALL be used for optimisation of queries. + + Redis and Memcached are two examples of RDBMS which are well tested and + are known to work well with Twisted. The major difference between the two + is that Memcached is volatile, while Redis supports command for + transferring objects into persistent on-disk storage. There are several + (see Twisted's MemCacheProtocol class [1] [2] or txyam [3] for Memcached, + and txredis [4] or txredisapi [5] for Redis). For non-Twisted Python Redis + APIs, there is redis-py, which provides a connection pool that could + likely be interfaced with from Twisted Python without too much + difficultly. [6] + + In order to further decrease the need for lookups in the backend + databases, Bloom Filters can used to eliminate extraneous + queries. However, this optimization would only be beneficial for string + lookups, i.e. querying for a user's credential, and SHOULD NOT be used for + queries within any of the hash lists, i.e. the list of hashes of + previously seen invite tickets. [7] It might be possible to use Redis' + GETBIT and SETBIT commands to store a Bloom Filter within a Redis cache + system; [8] doing so would offload the severe memory requirements of + loading the Bloom Filter into memory in Python when inserting new entries, + reducing the time complexity to order O(1) from some (polynomial) time + complexity that is proportional to the integral of the number of bridge + users over the rate of change of bridge users over time. + + XXX expire credentials [#] redis key datatype + [#]: http://redis.io/commands/pexpireat + + XXX evaluation on data by calling the sha1 for a serverside Lua script [#] + [#]: http://redis.io/commands/evalsha + +**** 2.a. Data Structures which should be stored in a RDBMS + + - User Credentials + + - Invite Tickets + + - Spent Credits + +* IV. Open Questions + +** IV.A. In which component of the Tor ecosystem should the client application code go? + +*** 1. Should this be done as a Pluggable Transport? + + Considerations: + +**** a. It doesn't need to modify the user's application-level traffic + + The clientside will eventually need to be able to build a circuit to the + BridgeDB backend, but it is not necessary that the clientside handle + any of the user's application level traffic. However, the clientside + system of rBridge must start when TBB (or tor) is started. + +**** b. It needs to be able to start tor. + + This is necessary because the lines: + {{{ + UseBridges 1 + Bridge [...] + }}} + must be present before tor is started; tor will not reload these + settings via SIGHUP. + +**** c. TorLaucher is not the correct place for this functionality. + + I am *not* adding this to TorLauncher. The clientside of rBridge will + eventually need to handle a lot of complicated new cryptographic + primitives, including commitments and zero-knowledge proofs. This is + dangerous enough, period, because there aren't really any libraries + for Pairing-Based Cryptography yet (though Tanya Lange has mentioned + to me that a student of theirs should have a good one finished some + time this year -- but I'm still going to count that as existing like + a unicorn). If I am to write this, I am doing it in + C/Python/Python-extensions. Not JS. + +***** c.i It could possibly launch TorLauncher + + In other words, this thing edits the torrc according to it's state, + and then either launches tor (if the user wants to use an installed + tor binary) or launches TorLauncher if we're running TBB. + +**** d. Little-t tor is not the correct place for this either. + + It might be possible, instead of (b) or (c), to add this to little-t + tor. However, I feel like the bridge distribution problem is a + separate to tor, which should be (more or less) strictly an + implementation of the onion-routing design. Additionally, I do not + wish to pile more code or maintenance upon eith Nick or Andrea, nor + do I wish to make little-t tor more monolithic. + + I talked with Nick briefly about this at the Summer 2013 Tor Dev + meeting in München, and he agreed that little-t tor isn't where this + code should go. + + +* References + +[0]: http://www-users.cs.umn.edu/~hopper/rbridge_ndss13.pdf +[1]: https://twistedmatrix.com/documents/current/api/twisted.protocols.memcache.… +[2]: http://stackoverflow.com/a/5162203 +[3]: http://findingscience.com/twisted/python/memcache/2012/06/09/txyam:-yet-ano… +[4]: https://pypi.python.org/pypi/txredis +[5]: https://github.com/fiorix/txredisapi +[6]: https://github.com/andymccurdy/redis-py/ +[7]: http://www.dr-josiah.com/2012/03/why-we-didnt-use-bloom-filter.html +[8]: http://redis.io/topics/data-types §"Strings"

1 0

[bridgedb/develop] Separate backend database improvements into their own proposal.
by isis＠torproject.org 01 Feb '14

01 Feb '14

commit 92ee780e102f3574f57f18191641a3ca1e9c0af1 Author: Isis Lovecruft <isis(a)torproject.org> Date: Wed Oct 23 18:32:02 2013 +0000 Separate backend database improvements into their own proposal. --- .../XXX-bridgedb-database-improvements.txt | 260 ++++++++++++++++++++ 1 file changed, 260 insertions(+) diff --git a/doc/proposals/XXX-bridgedb-database-improvements.txt b/doc/proposals/XXX-bridgedb-database-improvements.txt new file mode 100644 index 0000000..2d25bd2 --- /dev/null +++ b/doc/proposals/XXX-bridgedb-database-improvements.txt @@ -0,0 +1,260 @@ +# -*- coding: utf-8 ; mode: org -*- + +Filename: XXX-bridgedb-database-improvements.txt +Title: "Scalability and Stability Improvements to BridgeDB: Switching to a + Distributed Database System and RDBMS" +Author: Isis Agora Lovecruft +Created: 12 Oct 2013 +Related Proposals: XXX-social-bridge-distribution.txt +Status: Open + +* I. Overview + + BridgeDB is Tor's Bridge Distribution system, which currently has two major + Bridge Distribution mechanisms: the HTTPS Distributor and an Email + Distributor. [0] + + BridgeDB is written largely in Twisted Python, and uses Python2's builtin + sqlite3 as its database backend. Unfortunately, this backend system is + already showing strain through increased times for queries, and sqlite's + memory usage is not up-to-par with modern, more efficient, NoSQL databases. + + In order to better facilitate the implementation of newer, more complex + Bridge Distribution mechanisms, several improvements should be made to the + underlying database system of BridgeDB. Additionally, I propose that a + clear distinction in terms, as well as a modularisation of the codebase, be + drawn between the mechanisms for Bridge Distribution versus the backend + Bridge Database (BridgeDB) storage system. + + This proposal covers the design and implementation of a scalable NoSQL ― + Document-Based and Key-Value Relational ― database backend for storing data + on Tor Bridge relays, in an efficient manner that is ammenable to + interfacing with the Twisted Python asynchronous networking code of current + and future Bridge Distribution mechanisms. + +* II. Terminology + + BridgeDistributor := A program which decides when and how to hand out + information on a Tor Bridge relay, and to whom. + + BridgeDB := The backend system of databases and object-relational mapping + servers, which interfaces with the BridgeDistributor in order + to hand out bridges to clients, and to obtain and process new, + incoming ``@type bridge-server-descriptors``, + ``@type bridge-networkstatus`` documents, and + ``@type bridge-extrainfo`` descriptors. [3] + + BridgeFinder := A client-side program for an Onion Proxy (OP) which handles + interfacing with a BridgeDistributor in order to obtain new + Bridge relays for a client. A BridgeFinder also interfaces + with a local Tor Controller (such as TorButton or ARM) to + handle automatic, transparent Bridge configuration (no more + copy+pasting into a torrc) without being given any + additional privileges over the Tor process, [1] and relies + on the Tor Controller to interface with the user for + control input and displaying up-to-date information + regarding available Bridges, Pluggable Transport methods, + and potentially Invite Tickets and Credits (a cryptographic + currency without fiat value which is generated + automatically by clients whose Bridges remain largely + uncensored, and is used to purchase new Bridges), should a + Social Bridge Distributor be implemented. [2] + +* III. Databases +** III.A. Scalability Requirements + + Databases SHOULD be implemented in a manner which is ammenable to using a + distributed storage system; this is necessary because many potential + datatypes required by future BridgeDistributors MUST be stored permanently. + For example, in the designs for the Social Bridge Distributor, the list of + hash digests of spent Credits, and the list of hash digests of redeemed + Invite Tickets MUST be stored forever to prevent either from being replayed + ― or double-spent ― by a malicious user who wishes to block bridges faster. + Designing the BridgeDB backend system such that additional nodes may be + added in the future will allow the system to freely scale in relation to + the storage requirements of future BridgeDistributors. + + Additionally, requiring that the implementation allow for distributed + database backends promotes modularisation the components of BridgeDB, such + that BridgeDistributors can be separated from the backend storage system, + BridgeDB, as all queries will be issued through a simplified, common API, + regardless of the number of nodes system, or the design of future + BridgeDistributors. + +*** 1. Distributed Database System + + A distributed database system SHOULD be used for BridgeDB, in order to + scale resources as the number of Tor bridge users grows. This database + system, hereafter referred to as DDBS. + + The DDBS MUST be capable of working within Twisted's asynchronous + framework. If possible, a Object-Relational Mapper (ORM) SHOULD be used to + abstract the database backend's structure and query syntax from the Twisted + Python classes which interact with it, so that the type of database may be + swapped out for another with less code refactoring. + + The DDBM SHALL be used for persistent storage of complex data structures + such as the bridges, which MAY include additional information from both the + `@type bridge-server-descriptor`s and the `@type bridge-extra-info` + descriptors. [3] + +**** 1.a. Choice of DDBS + + CouchDB is chosen for its simple HTTP API, ease of use, speed, and official + support for Twisted Python applications. [4] Additionally, its + document-based data model is very similar to the current archetecture of + tor's Directory Server/Mirror system, in that an HTTP API is used to + retrieve data stored within virtual directories. Internally, it uses JSON + to store data and JavaScript as its query language, both of which are + likely friendlier to various other components of the Tor Metrics + infrastructure which sanitise and analyse portions of the Bridge + descriptors. At the very least, friendlier than hardcoding raw SQL queries + as Python strings. + +**** 1.b. Data Structures which should be stored in a DDBS: + + - RedactedDB - The Database of Blocked Bridges + + The RedactedDB will hold entries of bridges which have been discovered to + be unreachable from BridgeDB network vantage point, or have been reported + unreachable by clients. + + - BridgeDB - The Database of Bridges + + BridgeDB holds information on available Bridges, obtained via bridge + descriptors and networkstatus documents from the BridgeAuthority. Because + a Bridge may have multiple `ORPort`s and multiple + `ServerTransportListenAddress`es, attaching additional data to each of + these addresses which MAY include the following information on a blocking + event: + - Geolocational country code of the reported blocking event + - Timestamp for when the blocking event was first reported + - The method used for discovery of the block + - an the believed mechanism which is causing the block + would quickly become unwieldy, the RedactedDB and BridgeDB SHOULD be kept + separate. + + - User Credentials + + For the Social BridgeDistributor, these are rather complex, + increasingly-growing, concatenations (or tuples) of several datatypes, + including Non-Interactive Proofs-of-Knowledge (NIPK) of Commitments to + k-TAA Blind Signatures, and NIPK of Commitments to a User's current + number of Credits and timestamps of requests for Invite Tickets. + +*** 2. Key-Value Relational Database Mapping Server + + For simpler data structures which must be persistently stored, such as the + list of hashes of previously seen Invite Tickets, or the list of + previously spent Tokens, a Relational Database Mapping Server (RDBMS) + SHALL be used for optimisation of queries. + + Redis and Memcached are two examples of RDBMS which are well tested and + are known to work well with Twisted. The major difference between the two + is that Memcaches are stored only within volatile memory, while Redis + additionally supports commands for transferring objects into persistent, + on-disk storage. + + There are several support modules for interfacing with both Memcached and + Redis from Twisted Python, see Twisted's MemCacheProtocol class [5] [6] or + txyam [7] for Memcached, and txredis [8] or txredisapi [9] for + Redis. Additionally, numerous big name projects both use Redis as part of + their backend systems, and also provide helpful documentation on their own + experience of the process of switching over to the new systems. [17] For + non-Twisted Python Redis APIs, there is redis-py, which provides a + connection pool that could likely be interfaced with from Twisted Python + without too much difficultly. [10] [11] + +**** 2.a. Data Structures which should be stored in a RDBMS + + Simple, mostly-flat datatypes, and data which must be frequently indexed + should be stored in a RDBMS, such as large lists of hashes, or arbitrary + strings with assigned point-values (i.e. the "Uniform Mapping" for the + current HTTPS BridgeDistributor). + + For the Social BridgeDistributor, hash digests of the following datatypes + SHOULD be stored in the RDBMS, in order to prevent double-spending and + replay attacks: + + - Invite Tickets + + These are anonymous, unlinkable, unforgeable, and verifiable tokens + which are occasionally handed out to well-behaved Users by the Social + BridgeDistributor to permit new Users to be invited into the system. + When they are redeemed, the Social BridgeDistributor MUST store a hash + digest of their contents to prevent replayed Invite Tickets. + + - Spent Credits + + These are Credits which have already been redeemed for new Bridges. + The Social BridgeDistributor MUST also store a hash digest of Spent + Credits to prevent double-spending. + +*** 3. Bloom Filters and Other Database Optimisations + + In order to further decrease the need for lookups in the backend + databases, Bloom Filters can used to eliminate extraneous + queries. However, this optimization would only be beneficial for string + lookups, i.e. querying for a User's Credential, and SHOULD NOT be used for + queries within any of the hash lists, i.e. the list of hashes of + previously seen Invite Tickets. [14] + +**** 3.a. Bloom Filters within Redis + + It might be possible to use Redis' GETBIT and SETBIT commands to store a + Bloom Filter within a Redis cache system; [15] doing so would offload the + severe memory requirements of loading the Bloom Filter into memory in + Python when inserting new entries, reducing the time complexity from some + polynomial time complexity that is proportional to the integral of the + number of bridge users over the rate of change of bridge users over time, + to a time complexity of order O(1). + +**** 3.b. Expiration of Stale Data + + Some types of data SHOULD be safe to expire, such as User Credentials + which have not been updated within a certain timeframe. This idea should + be further explored to assess the safety and potential drawbacks to + removing old data. + + If there is data which SHOULD expire, the PEXPIREAT command provided by + Redis for the key datatype would allow the RDBMS itself to handle cleanup + of stale data automatically. [16] + +**** 4. Other potential uses of the improved Bridge database system + + Redis provides mechanisms for evaluations to be made on data by calling + the sha1 for a serverside Lua script. [15] While not required in the + slightest, it is a rather neat feature, as it would allow Tor's Metrics + infrastructure to offload some of the computational overhead of gathering + data on Bridge usage to BridgeDB (as well as diminish the security + implications of storing Bridge descriptors). + + Also, if Twisted's IProducer and IConsumer interfaces do not provide + needed interface functionality, or it is desired that other components of + the Tor software ecosystem be capable of scheduling jobs for BridgeDB, + there are well-tested mechanisms for using Redis as a message + queue/scheduling system. [16] + +* References + +[0]: https://bridges.torproject.org + mailto:bridges@bridges.torproject.org +[1]: See proposals 199-bridgefinder-integration.txt at + https://gitweb.torproject.org/torspec.git/blob/HEAD:/proposals/199-bridgefi… +[2]: See XXX-social-bridge-distribution.txt at + https://gitweb.torproject.org/user/isis/bridgedb.git/blob/refs/heads/featur… +[3]: https://metrics.torproject.org/formats.html#descriptortypes +[4]: https://github.com/couchbase/couchbase-python-client#twisted-api +[5]: https://twistedmatrix.com/documents/current/api/twisted.protocols.memcache.… +[6]: http://stackoverflow.com/a/5162203 +[7]: http://findingscience.com/twisted/python/memcache/2012/06/09/txyam:-yet-ano… +[8]: https://pypi.python.org/pypi/txredis +[9]: https://github.com/fiorix/txredisapi +[10]: https://github.com/andymccurdy/redis-py/ +[11]: http://degizmo.com/2010/03/22/getting-started-redis-and-python/ +[12]: http://www.dr-josiah.com/2012/03/why-we-didnt-use-bloom-filter.html +[13]: http://redis.io/topics/data-types §"Strings" +[14]: http://redis.io/commands/pexpireat +[15]: http://redis.io/commands/evalsha +[16]: http://www.restmq.com/ +[17]: https://www.mediawiki.org/wiki/Redis

1 0

[bridgedb/develop] Add a CHANGELOG.
by isis＠torproject.org 01 Feb '14

01 Feb '14

commit aa495a95b1072565b4f093117b0b8e55eaeba39b Author: Isis Lovecruft <isis(a)torproject.org> Date: Tue Jan 28 22:23:24 2014 +0000 Add a CHANGELOG. --- CHANGELOG | 69 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 69 insertions(+) diff --git a/CHANGELOG b/CHANGELOG new file mode 100644 index 0000000..27415f9 --- /dev/null +++ b/CHANGELOG @@ -0,0 +1,69 @@ +Changes in version 0.1.0 - 2014-XXX +BridgeDB 0.1.1 includes fixes for the following bugs: + * FIXES #9127 + * FIXES #10737 + +Including the following general changes: + * ADDS A general simple error page to display rather than + webserver tracebacks. See #6127. + +Changes in version 0.1.0 - 2014-01-14 +BridgeDB 0.1.0 includes fixes for the following bugs: + * FIXES #1606 Write a BridgeDB spec + * FIXES #3573 bridges.torproject.org doesn't have a robots.txt + * FIXES #6127 bridges.tpo runs in development mode + * FIXES #9013 BridgeDB should pass pluggable transport + shared-secrets to clients + * FIXES #9157 Persian and Arabic should be right aligned on bridges.tpo + * FIXES #9462 BridgeDB netstatus descriptor parsers need refactoring + * FIXES #9959 BridgeDB seems to be missing English translations + * FIXES #9865 Add automated code coverage report generation + * FIXES #9872 Create a test runner for BridgeDB unittests + * FIXES #9873 Convert BridgeDB's old unittests to use twisted.trial + * FIXES #9937 Create fake non-sanitised bridge descriptors for + BridgeDB testing purposes + * FIXES #10333 Indexing list-like objects by 0L in + Bridges.getConfigLine + * FIXES #10446 BridgeDB is/was using a GeoIP module which is + incompatible with virtualenvs + * FIXES #10559 BridgeDB writes `keyid=` before fingerprints + +Including the following general changes: + * ADD Numerous unittests and automated continuous integration testing. + * ADD Patches by sysrqb to correctly render right-to-left languages. + * FIXES fallback languages for translations. + * ADD Sphinx makefiles, substantial amounts of documentation. + * ADD Documentations builds (currently at + https://para.noid.cat/bridgedb) + +Changes in version 0.0.1 - 2013-08-20 +BridgeDB 0.0.1 includes fixes for the following bugs: + * FIXES #5332 Update BridgeDB documentation with deployment + instructions + * FIXES #9156 BridgeDB: Users try to add obfsbridges to their + normal TBB + These commits were added to fix a compatibility issue with + Vidalia, though they may be reverted to switch back to the old + behaviour of returning bridge lines in the form: + Bridge [transport_method] address:port [keyid=fingerprint] [K=v] […] + in order to work with torrc files and the new TBB-3.x packages + which use TorLauncher instead of Vidalia. + * PARTIAL FIX #9264 Problem with transport lines in BridgeDB's + bridge pool assignment files. + * FIXES #9425 Create and document a better BridgeDB (re)deployment + strategy + +Including the following general changes: + * UPDATE translations files with finished files from Transifex for + the strings for the newly-refactored web interface created by + gsathya. + * ADD an automatic version-numbering system as part of developing + a better deployment strategy. + * CHANGE setup.py script to automatically install Python-based + dependencies from requirements.txt. + * REMOVE MANIFEST.in and put equivalent 'include' directives into + setup.py (the less places we have for manually keeping track of + files, the better). + * REMOVE the "bridge " prefix from the lines returned on the web + interface. +

1 0

[bridgedb/develop] Fix alignment of topic headers in XXX-bdb-soc-dist.
by isis＠torproject.org 01 Feb '14

01 Feb '14

commit 58eb425869b34cc76ae2f76e751e160aa6ad229e Author: Isis Lovecruft <isis(a)torproject.org> Date: Wed Oct 23 11:55:28 2013 +0000 Fix alignment of topic headers in XXX-bdb-soc-dist. --- doc/proposals/XXX-bridgedb-social-distribution.txt | 18 ++++++++---------- 1 file changed, 8 insertions(+), 10 deletions(-) diff --git a/doc/proposals/XXX-bridgedb-social-distribution.txt b/doc/proposals/XXX-bridgedb-social-distribution.txt index 5f44fcf..7274132 100644 --- a/doc/proposals/XXX-bridgedb-social-distribution.txt +++ b/doc/proposals/XXX-bridgedb-social-distribution.txt @@ -8,7 +8,7 @@ Related Proposals: 199-bridgefinder-integration.txt XXX-bridgedb-database-improvements.txt Status: Draft -* I. Overview +* I. Overview This proposal specifies a system for social distribution of the centrally-stored bridges within BridgeDB. It is primarily based upon Part @@ -18,7 +18,7 @@ Status: Draft such malicious users and/or censoring entities from joining the pool of Tor clients who are able to receive distributed bridges. -* II. Motivation & Problem Scope +* II. Motivation & Problem Scope As it currently stands, Tor bridges which are stored within BridgeDB may be freely requested by any entity at nearly any time. While the openness, that @@ -35,7 +35,7 @@ Status: Draft can be even more in need of the identity protections and free speech enablement which Tor can provide, given their political contexts. -** II.A. Current Tor bridge distribution mechanisms and known pitfalls: +** II.A. Current Tor bridge distribution mechanisms and known pitfalls: *** 1. HTTP(S) Distributor @@ -71,7 +71,7 @@ Status: Draft roughly 1000 bridges in the Email Distributor's pool, distributing 3 bridges per email response, -* III. Terminology & Notations +* III. Terminology & Notations ** III.A. Terminology Definitions User := A client connecting to BridgeDB in order to obtain bridges. @@ -103,9 +103,7 @@ Status: Draft | ω | The last time that U requested and Invite Ticket from D | |--------------------+---------------------------------------------------------------------------------------------| -** III.C. Unicode characters - -* III. Threat Model +* IV. Threat Model In the original rBridge scheme, there are two separate proposals: the first does not make any attempt to hide information such as the user's (U) @@ -294,7 +292,7 @@ Status: Draft Considerations: -**** a. It doesn't need to modify the user's application-level traffic +**** 1a. It doesn't need to modify the user's application-level traffic The clientside will eventually need to be able to build a circuit to the BridgeDB backend, but it is not necessary that the clientside handle @@ -311,7 +309,7 @@ Status: Draft must be present before tor is started; tor will not reload these settings via SIGHUP. -**** c. TorLaucher is not the correct place for this functionality. +**** 1c. TorLaucher is not the correct place for this functionality. I am *not* adding this to TorLauncher. The clientside of rBridge will eventually need to handle a lot of complicated new cryptographic @@ -329,7 +327,7 @@ Status: Draft and then either launches tor (if the user wants to use an installed tor binary) or launches TorLauncher if we're running TBB. -**** d. Little-t tor is not the correct place for this either. +**** 1d. Little-t tor is not the correct place for this either. It might be possible, instead of (b) or (c), to add this to little-t tor. However, I feel like the bridge distribution problem is a

1 0