[tor-bugs] #11216 [BridgeDB]: BridgeDB is parsing PTs from `cached-extrainfo*` files cumulatively

[Tor Bug Tracker & Wiki]

#11216: BridgeDB is parsing PTs from `cached-extrainfo*` files cumulatively
-------------------------+-------------------------------------------------
     Reporter:  isis     |      Owner:  isis
         Type:  defect   |     Status:  closed
     Priority:           |  Milestone:
  critical               |    Version:
    Component:           |   Keywords:  bridgedb-0.3.0, metrics, bridgedb-
  BridgeDB               |  parsers, stem, isis2014Q3Q4, isisExB
   Resolution:  fixed    |  Parent ID:
Actual Points:           |
       Points:           |
-------------------------+-------------------------------------------------
Changes (by isis):

 * keywords:  bridgedb-0.2.x, metrics, bridgedb-parsers, isis2014Q3Q4,
     isisExB => bridgedb-0.3.0, metrics, bridgedb-parsers, stem,
     isis2014Q3Q4, isisExB
 * status:  needs_revision => closed
 * resolution:   => fixed


Comment:

 This was fixed by commits
 [https://gitweb.torproject.org/user/isis/bridgedb.git/commit/?h=fix/9380-stem_r10&id=7869e4c7cd1e43f9354480f6cafba0794fd86433
 7869e4c7cd1e43f9354480f6cafba0794fd86433],
 [https://gitweb.torproject.org/user/isis/bridgedb.git/commit/?h=fix/9380-stem_r10&id=65f18cd7c31a97274a8277688f0512b69ac1e3e3
 65f18cd7c31a97274a8277688f0512b69ac1e3e3], and
 [https://gitweb.torproject.org/user/isis/bridgedb.git/commit/?h=fix/9380-stem_r10&id=fe70415269693948bdfc5c8ea3abfab2b1d86c49
 fe70415269693948bdfc5c8ea3abfab2b1d86c49] in my `fix/9380-stem_r10`
 [https://gitweb.torproject.org/user/isis/bridgedb.git/log/?h=fix/9380-stem_r10
 branch].

 Those commits introduce the `bridgedb.parse.descriptors.deduplicate()`
 [https://gitweb.torproject.org/user/isis/bridgedb.git/tree/lib/bridgedb/parse/descriptors.py?h=fix/9380-stem_r10#n149
 function], which is called in the
 `bridgedb.parse.descriptors.parseExtraInfoFiles()`
 [https://gitweb.torproject.org/user/isis/bridgedb.git/tree/lib/bridgedb/parse/descriptors.py?h=fix/9380-stem_r10#n219
 function]. The former deduplicates all descriptors for every bridge,
 selecting only the newest descriptor for a particular bridge.
 Additionally, if any Bridge has multiple `@type bridge-extrainfo`
 descriptors with exactly the same timestamps, then a
 `bridgedb.parse.descriptors.DescriptorWarning`
 [https://gitweb.torproject.org/user/isis/bridgedb.git/tree/lib/bridgedb/test/test_parse_descriptors.py?h=fix/9380-stem_r10#n371
 will be issued], since perfectly identical descriptors shouldn't be
 something an unmodified tor is capable of doing (and thus would imply that
 there is either a drastic regression in tor, or that someone has created a
 possibly-malicious OR implementation). Unittests and integration
 [https://gitweb.torproject.org/user/isis/bridgedb.git/tree/lib/bridgedb/test/test_parse_descriptors.py?h=fix/9380-stem_r10#n381
 tests] which verify that these behaviours are functioning as expected have
 also been added.

 (Incidentally, these same changes also fix #2895.)

 Since #9380 has been merged into `develop` for the next release, this
 issue should be fixed in `bridgedb-0.3.0`.

--
Ticket URL: <https://trac.torproject.org/projects/tor/ticket/11216#comment:9>
Tor Bug Tracker & Wiki <https://trac.torproject.org/>
The Tor Project: anonymity online