[tor-bugs] #6395 [Metrics Data Processor]: Can we remove GetTor statistics from metrics-*?

[Tor Bug Tracker & Wiki]

#6395: Can we remove GetTor statistics from metrics-*?
------------------------------------+---------------------------------------
 Reporter:  karsten                 |          Owner:     
     Type:  task                    |         Status:  new
 Priority:  normal                  |      Milestone:     
Component:  Metrics Data Processor  |        Version:     
 Keywords:                          |         Parent:     
   Points:                          |   Actualpoints:     
------------------------------------+---------------------------------------

Comment(by karsten):

 Replying to [comment:14 ioerror]:
 > I'm not going to stop you from writing something or changing how these
 things are done, obviously. :)

 Okay.

 > Regarding the no one noticing - I admit, I don't know how to check if
 such things are broken. :(

 Well, the graphed line suddenly stopped in the middle of the graph,
 indicating that there were no newer values available.  You can't overlook
 that.  I guess nobody looked at the graph at all, that's why nobody
 noticed.  Well, and if nobody looks at the graph, it can't be as
 important.  For other graphs, I hear from people within a few days if
 they're broken.

 > I really do believe that these statistics have been very useful for
 understanding the deployment of client software at key points in time.
 >
 > I do think that it might make sense to just specify a generic graphing
 interface where given a dataset for "foo" we count the daily instances of
 "bar" occurring. That is a rather minimal thing, which should never
 change, which is basically just a string and an integer updated on a daily
 basis. I'd understand if you don't want to do this but I guess removing
 the code that already runs seems weird, if only because well, we all
 noticed (late) and some of us care.

 Again, there's much more code involved in metrics-* than graphing a line.
 metrics-db grabs and archives GetTor's stats file, metrics-lib provides a
 simple parser, metrics-web has the format specification, imports the stats
 file into a database, has the graphing code written in R, and makes a web
 form available to customize the graph.

 > Clearly, we're going to follow your lead on this either way. :)

 Glad to hear.  How do you like the
 [https://trac.torproject.org/projects/tor/attachment/ticket/6395/plot.py
 attached Python script]?  It requires matplot which is `apt-get install
 python-matplotlib` on Debian.  Run the script with `-h` to see what
 options you have.  The result should be quite similar to what you know
 from metrics, but with a lot less code.  I can tweak it if you tell me in
 what direction.

 kaner, can you add the script to GetTor's repo?  And can you merge
 [https://metrics.torproject.org/data/gettor_stats.txt past GetTor stats]
 into the current stats and stop truncating old values?  The file won't be
 downloaded as often anymore, so that should be fine from a bandwidth
 perspective.  Thanks!

-- 
Ticket URL: <https://trac.torproject.org/projects/tor/ticket/6395#comment:15>
Tor Bug Tracker & Wiki <https://trac.torproject.org/>
The Tor Project: anonymity online