[tor-bugs] #13600 [Onionoo]: Improve bulk imports of descriptor archives

[Tor Bug Tracker & Wiki]

#13600: Improve bulk imports of descriptor archives
-----------------------------+-----------------
     Reporter:  karsten      |      Owner:
         Type:  enhancement  |     Status:  new
     Priority:  normal       |  Milestone:
    Component:  Onionoo      |    Version:
   Resolution:               |   Keywords:
Actual Points:               |  Parent ID:
       Points:               |
-----------------------------+-----------------

Comment (by karsten):

 Replying to [comment:12 leeroy]:
 > Sorry for the delay. I've now checked re-parsing identical data with the
 following results.
 >
 >  * Reprocessing data rewrites the timestamp of
 last_changed_or_address_or_port from a valid value to -1.

 Interesting.  I only observed the other way around; which is not good
 either.  There is a bug here that we need to fix.  I'm listing this below,
 so that we don't forget about it.

 >  * The host_name key gets removed. I did see this before. I believe this
 depends on the rdns resolving component failing. I use a system tor for
 dns resolution so this makes sense (and sounds mostly harmless).

 Makes sense.  This is not a bug, I think.

 > That's it, besides the artifacts mentioned previously. This all combines
 to produce smaller data stores after reprocessing. If rewriting
 last_changed_or_address_or_port sounds harmless too then I think this
 ticket can be closed now that the branch was merged. The rest can be
 handled in their own tickets.

 Not harmless, I'm afraid.  See the list below.

 > Unless, would you prefer to deal with the command-line parsing before
 closing?

 That can happen in its own ticket.

 So, I finally reproduced some of these issues and discovered quite a few
 more.  I'm listing all issues here, so that we can either fix them in this
 ticket or open new tickets for some or all of them.  I assume some of
 these are closely related, which is why I didn't open new tickets just
 yet.

 __1. Bandwidth statuses contain overlapping intervals__

 {{{
 status/bandwidth/0/0/0011BD2485AD45D984EC4159C88FC066E5E3300E
  w 2015-07-04 20:56:24 2015-07-04 21:56:24 6862711808
  w 2015-07-04 21:56:24 2015-07-04 22:56:24 8487956480
 +w 2015-07-04 22:11:24 2015-07-04 22:56:24 5816468480
  w 2015-07-04 22:56:24 2015-07-04 23:56:24 7400038400

  w 2015-07-05 15:56:24 2015-07-05 16:56:24 8741448704
 -w 2015-07-05 16:11:24 2015-07-05 16:56:24 6629600256
  w 2015-07-05 16:56:24 2015-07-05 17:56:24 8642562048

  w 2015-07-05 20:56:24 2015-07-05 21:56:24 10966376448
 -w 2015-07-05 21:56:24 2015-07-05 22:56:24 9769795584
 +w 2015-07-05 21:56:24 2015-07-05 22:11:24 2572811264
  w 2015-07-05 22:56:24 2015-07-05 23:56:24 7191084032

  r 2015-07-04 21:56:24 2015-07-04 22:56:24 8327266304
 +r 2015-07-04 22:11:24 2015-07-04 22:56:24 5703036928
  r 2015-07-04 22:56:24 2015-07-04 23:56:24 7253635072
 }}}

 __2. Relay flags contain empty string__

 {{{
 status/details/B/A/BA2067E5ACA2417EA3DF3D883CCD411DCE79A4E0
 -"relay_flags":[]
 +"relay_flags":[""]
 }}}

 __3. Last changed OR address or port differs__

 {{{
 status/details/F/D/FD2836E402083EAF1E40635EC6EBD4CF83126988
 -"last_changed_or_address_or_port":-1
 +"last_changed_or_address_or_port":1436405873000

 status/details/F/C/FC75ECCBB64F3786B079D78F52F33E9A00529C2B
 -"last_changed_or_address_or_port":1436349600000
 +"last_changed_or_address_or_port":1435708800000
 }}}

 See also `status/summary` with node statuses.

 __4. Middle probability differs__

 {{{
 status/details/F/F/FF89E8901B433F2546F3E705B44918CA1F33F541
 -"middle_probability":7.0548704E-6
 +"middle_probability":7.2019566E-6
 }}}

 __5. Consensus weight differs__

 {{{
 F4E92F76C532F407968C2FA5396AE4E30064D418
 -"consensus_weight":0
 +"consensus_weight":-1
 }}}

 __6. Contact differs__

 {{{
 status/summary
 -noah r mer <memeticpox at gmail dot com>
 +noah r?mer <memeticpox at gmail dot com>

 -0x2b3fc09b375594c0 sebastian m ki <sebastian at tico.fi> -
 1j7fbivh6kf8ujsgp23fej4knms3x5px1v
 +0x2b3fc09b375594c0 sebastian m?ki <sebastian at tico.fi> -
 1j7fbivh6kf8ujsgp23fej4knms3x5px1v
 }}}

--
Ticket URL: <https://trac.torproject.org/projects/tor/ticket/13600#comment:13>
Tor Bug Tracker & Wiki <https://trac.torproject.org/>
The Tor Project: anonymity online