[tor-bugs] #22990 [Metrics/Library]: Add a heartbeat log message indicating progress and estimated time left

[Tor Bug Tracker & Wiki]

#22990: Add a heartbeat log message indicating progress and estimated time left
-----------------------------+--------------------------------
 Reporter:  karsten          |          Owner:  iwakeh
     Type:  enhancement      |         Status:  needs_revision
 Priority:  Medium           |      Milestone:
Component:  Metrics/Library  |        Version:
 Severity:  Normal           |     Resolution:
 Keywords:  metrics-2017     |  Actual Points:
Parent ID:                   |         Points:
 Reviewer:  iwakeh           |        Sponsor:
-----------------------------+--------------------------------

Comment (by karsten):

 Replying to [comment:9 iwakeh]:
 > I'd like to use a Logger different from the standard Logger.  Maybe,
 'statistics' as in Onionoo, i.e.,
 `LoggerFactory.getLogger("statistics")...`?  The progress information is
 rather of statistical nature,  the log statements will show up in the
 general ('-all.log') anyway; and it would be possible to have the
 information separated by simple logback configuration.

 Okay.

 > The percent calculation should only be done once: `100 * progress /
 total`.
 > Why not only keep track of the percentage done, i.e.,
 previousPercentDone and percentDone or similar?

 Sure, why not.

 > More importantly, `readTarballs` needs to catch the IOE to avoid halting
 the process just b/c of one tarball being faulty.

 Ugh, very good point. That's an oversight.

 > Information about skipped tarballs should also be logged to the
 'statistics' logger

 Not sure about that one. That's something different than what we're trying
 to achieve here: have an easy way to learn about progress when importing
 large amounts of data.

 > and the tarball skipped should be logged as warning as before.

 No, it's perfectly fine to leave a tarball in place and use the history
 file to avoid reading it once again. If we want to log it, it should be on
 debug.

 > Maybe, think about other possible statistics, which could be easily
 logged?

 Nothing came to mind yesterday. The new message was already very useful
 when importing one year of descriptors which took five hours. After three
 or four hours one wonders whether it's still doing anything useful or if
 it's going to take days or weeks, and then it's really useful to have this
 message. I'd say let's save future statistics for when we find out that we
 need them.

 I can revise the patch today or tomorrow, unless you beat me to it.

--
Ticket URL: <https://trac.torproject.org/projects/tor/ticket/22990#comment:10>
Tor Bug Tracker & Wiki <https://trac.torproject.org/>
The Tor Project: anonymity online