[metrics-bugs] #32660 [Metrics/Onionoo]: onionoo-backend is killing the ganeti cluster

[Tor Bug Tracker & Wiki]

#32660: onionoo-backend is killing the ganeti cluster
-----------------------------+------------------------------
 Reporter:  anarcat          |          Owner:  metrics-team
     Type:  defect           |         Status:  merge_ready
 Priority:  Medium           |      Milestone:
Component:  Metrics/Onionoo  |        Version:
 Severity:  Normal           |     Resolution:
 Keywords:                   |  Actual Points:
Parent ID:                   |         Points:
 Reviewer:                   |        Sponsor:
-----------------------------+------------------------------

Comment (by karsten):

 Oh, I'm not concerned about increase in CPU load. I used the same code to
 compute SHA-256 file digests for #31204 where doing this for 643M of data
 took merely 5 seconds.

 Other mechanisms like timestamps don't work here, because we're actually
 producing new content for these files and only then learn that previously
 written files already have the same content.

 We could do something smarter by detecting whether something has changed
 to the underlying data that we're formatting and writing, but that
 requires more development time that we currently have available.

 I'll deploy this change now, and then we can see how much that bought us.
 If this doesn't fully solve the issue, which might well be the case, we
 can think more about that discrepancy between the two numbers.

--
Ticket URL: <https://trac.torproject.org/projects/tor/ticket/32660#comment:13>
Tor Bug Tracker & Wiki <https://trac.torproject.org/>
The Tor Project: anonymity online