[tor-bugs] #25161 [Metrics/CollecTor]: Fix another memory problem with the webstats bulk import

[Tor Bug Tracker & Wiki]

#25161: Fix another memory problem with the webstats bulk import
-------------------------------+------------------------------
 Reporter:  karsten            |          Owner:  metrics-team
     Type:  defect             |         Status:  new
 Priority:  Medium             |      Milestone:
Component:  Metrics/CollecTor  |        Version:
 Severity:  Normal             |     Resolution:
 Keywords:                     |  Actual Points:
Parent ID:                     |         Points:
 Reviewer:                     |        Sponsor:
-------------------------------+------------------------------

Comment (by iwakeh):

 Quote from #25100 comment 16:

      The second quarter 2017 of logs from weschniakowii amounts to 32M
 (compressed) and can be processed in 36min using 8G. The entire year won't
 work with just 8G.

      85min and 16G are needed for the entire available archives of
 meronense and weschniakowii together (compressed 59M). The used heap
 median usage is 8.5G and the max 15.8G.

      So, depending on the hardware a conservative import strategy might be
 to import slices of quarters with subsequent CollecTor runs.

--
Ticket URL: <https://trac.torproject.org/projects/tor/ticket/25161#comment:2>
Tor Bug Tracker & Wiki <https://trac.torproject.org/>
The Tor Project: anonymity online