[tor-bugs] #32747 [Metrics/CollecTor]: Avoid reprocessing webstats files

[Tor Bug Tracker & Wiki]

#32747: Avoid reprocessing webstats files
-----------------------------------+----------------------
     Reporter:  karsten            |      Owner:  karsten
         Type:  defect             |     Status:  assigned
     Priority:  Medium             |  Milestone:
    Component:  Metrics/CollecTor  |    Version:
     Severity:  Normal             |   Keywords:
Actual Points:                     |  Parent ID:
       Points:                     |   Reviewer:
      Sponsor:                     |
-----------------------------------+----------------------
 Web servers typically provide us with the last 14 days of request logs. We
 shouldn't process the whole 14 days over and over. Instead we should only
 process new logs files and any other log files containing log lines from
 newly written dates.

 In some cases web servers stop serving a given virtual host or stop acting
 as web server at all. However, in these cases we're left with 14 days of
 logs per virtual host. Ideally, these logs would get cleaned up, but until
 that's the case, we should at least not reprocess these files over and
 over.

--
Ticket URL: <https://trac.torproject.org/projects/tor/ticket/32747>
Tor Bug Tracker & Wiki <https://trac.torproject.org/>
The Tor Project: anonymity online