[tor-bugs] #13600 [Onionoo]: Improve bulk imports of descriptor archives

[Tor Bug Tracker & Wiki]

#13600: Improve bulk imports of descriptor archives
-----------------------------+-----------------
     Reporter:  karsten      |      Owner:
         Type:  enhancement  |     Status:  new
     Priority:  normal       |  Milestone:
    Component:  Onionoo      |    Version:
   Resolution:               |   Keywords:
Actual Points:               |  Parent ID:
       Points:               |
-----------------------------+-----------------

Comment (by leeroy):

 Keep in mind that unpatched master will introduce update artifacts as
 mentioned. It's not going to be a guaranteed fix to just remove the
 archive after parsing once. What if you import the latest month's archive,
 then import recent? (You get the same problems). That being said....

 You can expect the numbers quoted for time, space to still be accurate.
 The implementation imports by the archive anyway so you might as well make
 your life easy and import by the month. Master doesn't make any mitigation
 against losses on failure. Unless you want to redo the results (in
 particular the ones which try to create memory error), then go ahead. I
 see no benefit though. The implementation is heap based. We know this
 already.

 Speaking of loss mitigation. Writing a history file and writing it early
 (as opposed to the end of the entire updater run, which may be days). If
 this sounds useful see #16426 (which requires #16540). This *shouldn't*
 been seen as a fix for the update artifacts mentioned above.

 Besides that you might also find #16612 useful (which requires #16424).

 I hope that clears things up. Let me know if you need further details.

--
Ticket URL: <https://trac.torproject.org/projects/tor/ticket/13600#comment:18>
Tor Bug Tracker & Wiki <https://trac.torproject.org/>
The Tor Project: anonymity online