[metrics-bugs] #25317 [Metrics/CollecTor]: Enable webstats to process large (> 2G) logfiles

Tor Bug Tracker & Wiki blackhole at torproject.org
Mon Feb 26 10:43:17 UTC 2018

#25317: Enable webstats to process large (> 2G) logfiles
 Reporter:  iwakeh             |          Owner:  iwakeh
     Type:  defect             |         Status:  needs_review
 Priority:  High               |      Milestone:
Component:  Metrics/CollecTor  |        Version:
 Severity:  Normal             |     Resolution:
 Keywords:                     |  Actual Points:
Parent ID:                     |         Points:
 Reviewer:                     |        Sponsor:
Changes (by iwakeh):

 * status:  needs_revision => needs_review


 Please review another
 commit] on the current patch branch.

 Timing with the test batch is a little faster and memory usage looks way
 healthier now (stays between 2G and 9G mostly around 5G with very few
 peaks at 11G).  A more concave build-up, that drops down again to even
 less than 2G often.  The higher memory usage is during the compressing and
 writing phase which makes sense.

 Processing yearly batches for larger imports is much faster by about a
 factor of three on the aroides logs.

Ticket URL: <https://trac.torproject.org/projects/tor/ticket/25317#comment:5>
Tor Bug Tracker & Wiki <https://trac.torproject.org/>
The Tor Project: anonymity online

More information about the metrics-bugs mailing list