[tor-bugs] #12031 [BridgeDB]: Create a Key-Value database system for simple/flat datatypes in BridgeDB

[Tor Bug Tracker & Wiki]

#12031: Create a Key-Value database system for simple/flat datatypes in BridgeDB
-------------------------+-------------------------------------------------
     Reporter:  isis     |      Owner:  isis
         Type:           |     Status:  needs_revision
  enhancement            |  Milestone:
     Priority:  major    |    Version:
    Component:           |   Keywords:  bridgedb-1.0.x, bridgedb-db,
  BridgeDB               |  proposal-226
   Resolution:           |  Parent ID:
Actual Points:           |
       Points:           |
-------------------------+-------------------------------------------------

Comment (by isis):

 My work on this so far is going faster than expected, but it's still in no
 way ready to merge. You can see my latest branch, `fix/12031-redis_r1`
 [https://gitweb.torproject.org/user/isis/bridgedb.git/shortlog/refs/heads/fix/12031-redis_r1
 here].

 So far I have finished:

  * A
 [https://gitweb.torproject.org/user/isis/bridgedb.git/commitdiff/b91b4c799ca7c78f7520ee4a002fda77b508c9d3
 mechanism for queuing batches of transactions] with a Redis server, with a
 [https://github.com/deldotdr/txRedis/blob/master/txredis/client.py#L18
 txredis.client.RedisClient] assigned to each batch. This turns out to be
 the fastest, safest way to do this, even though it unfortunately makes the
 callback chains more complicated.

  *
 [https://gitweb.torproject.org/user/isis/bridgedb.git/commitdiff/4e7b01a1b89b4b1df4b69ce1fd46fea9fe95829c
 Fixed a bug] due to a design flaw in the internal queuing mechanism of the
 `txredis` library we're using. I have
 [https://github.com/deldotdr/txRedis/issues/53 reported the design flaw
 upstream] and offered to fix it.

  * I've
 [https://gitweb.torproject.org/user/isis/bridgedb.git/blob/refs/heads/fix/12031-redis_r1:/lib/bridgedb/database/redis.py#l296
 tested my implementation] so far by using the `bridgedb.parse.descriptors`
 module (from my `fix/9380-stem_r2` branch, which is not merged yet) to
 parse 250 `@type bridge-networkstatus` documents with Stem, serialise
 them, and store them in Redis. That entire process takes a mean time 12ms,
 and scales linearly with additional descriptors. (As opposed to the
 linearithmic scaling which BridgeDB is currently dealing with.)

 There is a tiny bit of entanglement with my
 [https://gitweb.torproject.org/user/isis/bridgedb.git/shortlog/refs/heads/fix/12029
 -dist-api_r1 branch] for #12029, `fix/12029-dist-api_r1`, due to there
 being some commits at the beginning of the `fix/12031-redis_r1` branch
 which the `fix/12029-dist-api_r1` needs, these will need to be separated
 properly, depending on whichever one is merged first. The commits which
 are necessary for both branches are:

 {{{
 5f1243d75af619b8c39fe98be48ba1bc43f63944 Make `answerParameters` be a
 @property of `Distributor`s.
 2e0f705e0edbdea0c4db3ad91e5e7bbed9b670e5 Move
 `b.Bridges.BridgeRingParameters` → `b.bridgerequest.AnswerParameters`.
 5495af3b0877b58af1005caaa14655f32b20d084 Rename `b.hashring.BridgeRing` to
 `b.hashring.Hashring` and cleanup docstrings.
 f9632e97c1ca0bed825def9e5dee503e6d87849d Make `b.hashring.BridgeRing` an
 implementer of `b.hashring.IHashring`.
 fb28ea8b9dfa9cc3e0d04bd0a405c5058d125f4f Move `b.Bridges.BridgeRing` →
 `b.hashring.BridgeRing`.
 8b4b08c0f38ea3dae723eff2b27e7b338bf9edb7 Add IHashring interface
 specification.
 60814f6b7f6d7f952cd6dfd62eb1bfdd7db6d769 Move old `b.Dist.Distributor`
 doctest to `b.Dist.IPBasedDistributor`.
 b93b8410b075f09576984d5819a92e5dfa5eba3f Remove old
 bridgedb.Dist.Distributor class.
 36069a806bd3f0aa5bef27fb7a14e59745f43ecc Add basic implementation of the
 IDistribute interface.
 d956f1011af69ea6e0513d0907e4d7d440812eda Add initial IDistribute interface
 specification for a distributor.
 99fd8189785917dd09d9122270692a89f896b061 Fix old Tests.py;
 networkstatus.parseALine returns a string.
 5c8bf76f2379d0039637877ef9ae4bdf11a72877 Support requiring distribution of
 bridges with the "Running" flag.
 ca25134def3ac064cdd1f5d21ce583013d7318fd Rip out
 `BridgeHolder.assignmentsArePersistent()`, it's never used.
 541f927475e8574a96a03c94fc99f74b37b4d6b4 Make `isValidIP()` backwards
 compatible with deprecated `is_valid_ip()`.
 6a59652077571d700266c5ffa01ead8c55d58325 Move old b.Bridges.is_valid_ip()
 doctest to b.p.addr.isValidIP().
 20ddda862453494bfbc98c34ff70f4e00ace3bae Add note that
 bridgedb.Bridges.ID_LEN isn't used.
 fbde0bf1702347bf524452401672ac263c2ace04 Remove
 bridgedb.Bridges.HEX_DIGEST_LEN; it's completely unused.
 4d7a9f34d66d71f71c6814f219a3e3c207431d8a Deprecate
 bridgedb.Bridges.is_valid_fingerprint().
 afa33faaf90982c68e4f501c0d44646d3437eb04 Add bridgedb.interfaces module
 which simply collects all interfaces.
 d78e56e424f644da7d9f8da99ff7012b2cc922c2 Rename IBridgeRequest →
 IRequestBridges. It's cuter.
 0ab4fbc2a531b9f7895655a18e702e3d48647f51 Add Bridges.is_valid_ip doctests
 to b.p.addr.isValidIP docstring.
 }}}

 ...and some of those may even be general enough to warrant inclusion into
 the `develop` branch ''before'' anything gets merged.

 I estimate that this ticket is somewhere between 30-50% complete.

--
Ticket URL: <https://trac.torproject.org/projects/tor/ticket/12031#comment:4>
Tor Bug Tracker & Wiki <https://trac.torproject.org/>
The Tor Project: anonymity online