[tor-bugs] #27181 [Metrics/Onionoo]: Avoid unnecessary disk writes

[Tor Bug Tracker & Wiki]

#27181: Avoid unnecessary disk writes
---------------------------------+--------------------------
     Reporter:  irl              |      Owner:  metrics-team
         Type:  enhancement      |     Status:  new
     Priority:  Medium           |  Milestone:
    Component:  Metrics/Onionoo  |    Version:
     Severity:  Normal           |   Keywords:
Actual Points:                   |  Parent ID:
       Points:                   |   Reviewer:
      Sponsor:                   |
---------------------------------+--------------------------
 Right now if you run the updater, every status document will be touched by
 Onionoo regardless of whether or not the content actually changed.
 Additionally every file in the out directory will be touched, although I'm
 less concerned about this as I've seen that the out directory is
 reproducibly built from the status directory where no changes have
 occurred. My primary motivation for this ticket is enabling consistency in
 backups and the creation of test instances where we'd like to copy out the
 status directory between updater runs, where there is a limited time
 window.

 I propose that we add a field to the Document class that marks a document
 as having changed since it was loaded from disk. Initially this flag would
 be clear but if any field in the document is changed then this would
 trigger a write. There may be conditions where a value is set to the value
 it already has and we should detect these cases and not set the flag to
 perform a write.

 Alternatively, we can store the serialization of the document along with
 the parsed fields. When serializing the document we can compare the
 document with its original serialization to determine if it needs to be
 written to disk.

 We will need to have good test coverage for this as if we miss setting the
 flag to write data we may end up with gaps in data on the Onionoo hosts.

--
Ticket URL: <https://trac.torproject.org/projects/tor/ticket/27181>
Tor Bug Tracker & Wiki <https://trac.torproject.org/>
The Tor Project: anonymity online