[tor-bugs] #2688 [BridgeDB]: BridgeDB re-assigns unallocated bridges from/to file buckets without need

[Tor Bug Tracker & Wiki]

#2688: BridgeDB re-assigns unallocated bridges from/to file buckets without need
----------------------+-----------------------------------------------------
 Reporter:  karsten   |          Owner:  kaner
     Type:  defect    |         Status:  new  
 Priority:  major     |      Milestone:       
Component:  BridgeDB  |        Version:       
 Keywords:            |         Parent:       
   Points:            |   Actualpoints:       
----------------------+-----------------------------------------------------
 It looks like BridgeDB re-assigns unallocated bridges from/to file buckets
 without need.  That is, a bridge that keeps running from one network
 status to the next might be removed from a file bucket and replaced with
 another bridge.  This leads to quick enumeration of all bridges in the
 unallocated pool when using file buckets.

 A second bug seems to be that BridgeDB appends bridges to file buckets
 instead of overwriting these files.  The result is that there are
 duplicate entries in files that external distributors use.

 Here's how one can reproduce the problem using sanitized bridge
 descriptors.  Yes, this description is lengthy and ugly, but it works for
 testing BridgeDB in general, even if one doesn't have the original bridge
 descriptors handy.

 Download and extract the sanitized bridge descriptors from January 2009:

   https://metrics.torproject.org/data/bridge-descriptors-2009-01.tar.bz2

 Create a single bridge-descriptors file containing all bridge descriptors
 from that month.

   $ cd bridge-descriptors-2009-01/
   $ echo "@purpose bridge" > purpose
   $ echo "router-signature" > routersignature
   $ find server-descriptors/ -type f | xargs -I{} cat purpose {}
 routersignature > bridge-descriptors

 Copy the new bridge-descriptors file to BridgeDB's working directory
 (here: `~/run/`).

   $ cd ~/run/
   $ cp bridge-descriptors-2009-01/bridge-descriptors .

 Also copy the sanitized network status file from 2009-01-10 00:07:04 to
 BridgeDB's working directory and rename it to networkstatus-bridges.

   $ cp bridge-
 descriptors-2009-01/statuses/10/20090110-000704-4A0CCD2DDC7995083D73F5D667100C8A5831F16D
 networkstatus-bridges

 Configure BridgeDB to write 4 bridges to file bucket `twitter`, otherwise
 keep the default configuration.  Start BridgeDB (note that it may take 30
 seconds to digest the 8.5M bridge-descriptors file) and dump bridges to
 file buckets.

 The result is a new file `twitter-2011-03-09.brdgs` with this content (may
 vary for you):

 {{{
 10.134.79.249:443
 10.236.199.173:443
 10.116.76.140:9001
 10.51.76.151:18443
 }}}

 It also writes this `unallocated-2011-03-09.brdgs` file (again, content
 may vary):

 {{{
 10.126.198.237:443
 10.251.69.61:9003
 10.31.186.235:49001
 10.81.88.5:9001
 }}}

 Replace the networkstatus-bridges with the one from roughly 30 minutes
 later:

   $ cp bridge-
 descriptors-2009-01/statuses/10/20090110-030709-4A0CCD2DDC7995083D73F5D667100C8A5831F16D
 networkstatus-bridges

 Give BridgeDB a HUP, wait at least 30 seconds, and tell it to dump bridges
 to file buckets again.

 Here's my new `twitter-2011-03-09.brdgs` file:

 {{{
 10.134.79.249:443
 10.236.199.173:443
 10.116.76.140:9001
 10.51.76.151:18443
 10.51.76.151:18443
 10.237.143.0:443
 10.241.115.62:443
 10.239.76.198:443
 }}}

 And my `unallocated-2011-03-09.brdgs` file:

 {{{
 10.126.198.237:443
 10.251.69.61:9003
 10.31.186.235:49001
 10.81.88.5:9001
 10.126.198.237:443
 10.251.69.61:9003
 10.31.186.235:49001
 10.81.88.5:9001
 10.134.79.249:443
 10.236.199.173:443
 10.116.76.140:9001
 }}}

 There are two bugs:

  - Why was 10.236.199.173:443 (2nd line in `twitter-2011-03-09.brdgs`)
 removed from this file bucket and put back in the unallocated ring again
 (last but one line in `unallocated-2011-03-09.brdgs`)?  I confirmed that
 this is the same bridge using the new pool assignments patch that is not
 merged yet.  This is the first bug described above.

  - Why are IP:port lines appended to these files?  This is the second bug
 described above.

-- 
Ticket URL: <https://trac.torproject.org/projects/tor/ticket/2688>
Tor Bug Tracker & Wiki <https://trac.torproject.org/>
The Tor Project: anonymity online