[tor-bugs] #4634 [Company]: Need a VM to set up web log analysis tools

[Tor Bug Tracker & Wiki]

#4634: Need a VM to set up web log analysis tools
---------------------+------------------------------------------------------
 Reporter:  runa     |          Owner:  weasel
     Type:  task     |         Status:  new   
 Priority:  normal   |      Milestone:        
Component:  Company  |        Version:        
 Keywords:           |         Parent:        
   Points:           |   Actualpoints:        
---------------------+------------------------------------------------------

Comment(by karsten):

 Replying to [comment:1 phobos]:
 > Why two VMs? Why not just sanitize the logs on the same VM, purge the
 unsanitized logs, and let webalizer/awstats run against them? My concern
 is we're creating lots of admin overhead for little gain. The webserver
 logs are already fairly sanitized. It's not the end of the world if they
 are leaked. And, we don't need to keep all of this data hanging around.
 The output of webalizer or awstats should be sufficient to see the
 necessary statistics. If the machine dies and we lose all of our logs, oh
 well. We don't use them now anyway. We're assigning too much value to
 these logs and the data contained within them.

 The idea was to sanitize web logs on the same VM that we use for
 sanitizing bridge descriptors and have a separate VM for running AWStats
 and/or Webalizer.

 As for throwing away original logs, I don't mind as much.  My idea was to
 test the sanitized log format and visualization a bit before throwing away
 original data.  But I can give up on that idea.  I wouldn't want to do
 this for bridge descriptors, but I don't care as much for web logs.  Just
 don't be sad if we lack history to compare download numbers or anything
 over time.

 > Why webalizer AND awstats? We should choose one and be done with it.

 We're not sure yet if both of them work as expected.  For example, Runa
 had difficulties with AWStats on AWS.  It might be that we'll run into
 more problems when deploying this.  Both tools work, so we can try
 deploying both, and if one fails, we take the other.  Really, setting up
 these two tools on the same VM working with the same data set isn't the
 major effort here.

 > And while I'm whining, stats.tpo is too generic, how about webstats or
 weblogs for the domain?

 (Leaving this question for Runa.)

-- 
Ticket URL: <https://trac.torproject.org/projects/tor/ticket/4634#comment:2>
Tor Bug Tracker & Wiki <https://trac.torproject.org/>
The Tor Project: anonymity online