[tor-bugs] #22428 [Metrics/CollecTor]: add webstats module to collector

[Tor Bug Tracker & Wiki]

#22428: add webstats module to collector
-------------------------------+-----------------------------------
 Reporter:  iwakeh             |          Owner:  iwakeh
     Type:  enhancement        |         Status:  needs_information
 Priority:  Medium             |      Milestone:
Component:  Metrics/CollecTor  |        Version:
 Severity:  Normal             |     Resolution:
 Keywords:                     |  Actual Points:
Parent ID:                     |         Points:
 Reviewer:                     |        Sponsor:
-------------------------------+-----------------------------------

Comment (by iwakeh):

 Replying to [comment:9 karsten]:
 > First bit of feedback (more may follow): wouldn't it be problematic to
 leave out the hostname of the collecting server in the output path? There
 are several hosts serving the main Tor website, and each of them will
 create a log file for that domain name for a given day.

 Good point!  The hostname is used as directory name on
 webstats.torproject.org, but doesn't seem to be part of current
 shell/python processing, which takes place in a directory containing the
 files to be cleaned (afaict).

 >
 > Note that I don't have access to the current webstats host, but I would
 imagine that not all files are stored in a directory without
 subdirectories. The existing code might reveal something about that, but I
 haven't looked.

 As mentioned above, but it is very easy to simply look for files in
 subfolder, too.

 So all in all, the main question left is how the access logs and the log
 meta data are provided.  So far, meta data seems to be in the log-filename
 and the path (i.e. the hostname).
 Could the logs be 'delivered' as compressed files having meta data in
 their filename, e.g.,
 {{{<measuring-host>-<served-hostname>-access.log-<year-month-day>.xz}}}
 This is just an  example any other combination will be fine, important is
 what and how meta-data is communicated.

--
Ticket URL: <https://trac.torproject.org/projects/tor/ticket/22428#comment:10>
Tor Bug Tracker & Wiki <https://trac.torproject.org/>
The Tor Project: anonymity online