[metrics-bugs] #25317 [Metrics/CollecTor]: Enable webstats to process large (> 2G) logfiles

[Tor Bug Tracker & Wiki]

#25317: Enable webstats to process large (> 2G) logfiles
-------------------------------+------------------------------
 Reporter:  iwakeh             |          Owner:  iwakeh
     Type:  defect             |         Status:  needs_review
 Priority:  High               |      Milestone:
Component:  Metrics/CollecTor  |        Version:
 Severity:  Normal             |     Resolution:
 Keywords:                     |  Actual Points:
Parent ID:                     |         Points:
 Reviewer:                     |        Sponsor:
-------------------------------+------------------------------
Changes (by iwakeh):

 * status:  needs_revision => needs_review


Comment:

 Please review another
 [https://gitweb.torproject.org/user/iwakeh/collector.git/commit/?h=task-25317
 commit] on the current patch branch.

 Timing with the test batch is a little faster and memory usage looks way
 healthier now (stays between 2G and 9G mostly around 5G with very few
 peaks at 11G).  A more concave build-up, that drops down again to even
 less than 2G often.  The higher memory usage is during the compressing and
 writing phase which makes sense.

 Processing yearly batches for larger imports is much faster by about a
 factor of three on the aroides logs.

--
Ticket URL: <https://trac.torproject.org/projects/tor/ticket/25317#comment:5>
Tor Bug Tracker & Wiki <https://trac.torproject.org/>
The Tor Project: anonymity online