[tor-bugs] #25815 [Metrics/Onionoo]: Speed up hourly updater performance

[Tor Bug Tracker & Wiki]

#25815: Speed up hourly updater performance
-----------------------------+--------------------------
 Reporter:  karsten          |          Owner:  karsten
     Type:  enhancement      |         Status:  accepted
 Priority:  Medium           |      Milestone:
Component:  Metrics/Onionoo  |        Version:
 Severity:  Normal           |     Resolution:
 Keywords:                   |  Actual Points:
Parent ID:                   |         Points:
 Reviewer:                   |        Sponsor:
-----------------------------+--------------------------

Comment (by karsten):

 So, I ran some performance measurements here. In particular, I used a
 recent backup from one of the public Onionoo hosts and ran an update. By
 doing so I identified two possible areas of improvements:

  1. While writing details documents, we spend almost 40% of CPU time on
 (de-)serializing JSON documents. We might want to try out another JSON
 library than Gson. A quick search says that Jackson might be a lot faster.
 Maybe we can save 10% or 20% of the overall time here.

  2. Also while writing details documents, we read almost all details
 status files and (re-)write a corresponding details document file. That's
 almost 1M files and takes roughly 55% of the overall CPU time, including
 the 40% for JSON handling. If we can find a way to exclude files that
 don't need to be updated, we might be able to save another 10% or 20% of
 overall time. (The overall savings are related to what we do to JSON
 handling.)

 I'd like to keep this ticket for a while longer and dive deeper into these
 two ideas. But I thought I'd share these first results to hear early if
 they're a bad idea for some reason.

--
Ticket URL: <https://trac.torproject.org/projects/tor/ticket/25815#comment:3>
Tor Bug Tracker & Wiki <https://trac.torproject.org/>
The Tor Project: anonymity online