[tor-bugs] #25100 [Metrics/CollecTor]: Make CollecTor's webstats module use less RAM and wall time
Tor Bug Tracker & Wiki
blackhole at torproject.org
Mon Feb 5 16:12:42 UTC 2018
#25100: Make CollecTor's webstats module use less RAM and wall time
-------------------------------+------------------------------
Reporter: karsten | Owner: iwakeh
Type: enhancement | Status: needs_review
Priority: High | Milestone:
Component: Metrics/CollecTor | Version:
Severity: Normal | Resolution:
Keywords: | Actual Points:
Parent ID: | Points:
Reviewer: | Sponsor:
-------------------------------+------------------------------
Comment (by iwakeh):
I didn't find any more bugs processing the following benchmarks for
determining how to approach the import using the two latest branches of
this and the child ticket.
The second quarter 2017 of logs from weschniakowii amounts to 32M
(compressed) and can be processed in 36min using 8G. The entire year
won't work with just 8G.
85min and 16G are needed for the entire available archives of meronense
and weschniakowii together (compressed 59M). The used heap median usage
is 8.5G and the max 15.8G.
So, depending on the hardware a conservative import strategy might be to
import slices of quarters with subsequent CollecTor runs.
--
Ticket URL: <https://trac.torproject.org/projects/tor/ticket/25100#comment:18>
Tor Bug Tracker & Wiki <https://trac.torproject.org/>
The Tor Project: anonymity online
More information about the tor-bugs
mailing list