[tor-bugs] #25815 [Metrics/Onionoo]: Speed up hourly updater performance

[Tor Bug Tracker & Wiki]

#25815: Speed up hourly updater performance
---------------------------------+--------------------------
     Reporter:  karsten          |      Owner:  metrics-team
         Type:  enhancement      |     Status:  new
     Priority:  Medium           |  Milestone:
    Component:  Metrics/Onionoo  |    Version:
     Severity:  Normal           |   Keywords:
Actual Points:                   |  Parent ID:
       Points:                   |   Reviewer:
      Sponsor:                   |
---------------------------------+--------------------------
 When looking at Onionoo's logs with irl the other day we noticed that the
 hourly updater sometimes takes as long as 45 minutes.

 We obviously need to be careful not to let this time get closer to 60
 minutes or even exceed that time.

 I just looked a little bit closer at the logs and made a graph. It shows
 three phases of the hourly updater:
  - before: from starting the updater to writing details documents;
  - write: writing details documents;
  - after: from after writing details documents to the end.

 The graph compares total runtimes on the x axis with the three phases on
 the y axis. I'm reading two things in this graph.
  - Writing details documents really takes a long time. It's never faster
 than 15 minutes and can take longer than 30 minutes in some cases.
  - Whenever an hourly update takes longer than, say, 40 minutes, it's not
 just the details documenting that takes longer than usual, but also the
 phase before takes longer. I guess the system is just generally slower at
 those times.

 I'd say let's look at details document writing for optimizing runtime.
 Maybe we can decide early that we don't have to update a file. But that's
 just guessing, we should add more logging to that code, as right now the
 logs are silent for those 30 to 45 minutes.

--
Ticket URL: <https://trac.torproject.org/projects/tor/ticket/25815>
Tor Bug Tracker & Wiki <https://trac.torproject.org/>
The Tor Project: anonymity online