[metrics-bugs] #25259 [Metrics/Statistics]: Tune advbwdist module of metrics-web
Tor Bug Tracker & Wiki
blackhole at torproject.org
Wed Feb 14 18:16:29 UTC 2018
#25259: Tune advbwdist module of metrics-web
--------------------------------+------------------------------
Reporter: iwakeh | Owner: metrics-team
Type: enhancement | Status: new
Priority: Medium | Milestone:
Component: Metrics/Statistics | Version:
Severity: Normal | Resolution:
Keywords: | Actual Points:
Parent ID: | Points:
Reviewer: | Sponsor:
--------------------------------+------------------------------
Comment (by iwakeh):
Please review [https://gitweb.torproject.org/user/iwakeh/metrics-
web.git/commit/?h=task-25259 these tuning steps].
From the commit comment:
Processing advbwdist-validafter.csv (350M) took 150 seconds and used up to
7G.
Performing pre-processing separately, helping R by defining read types,
and avoiding multiple casting operations led to halving the processing
time (to 77 seconds) and reducing the necessary memory to about 25%
(approx. 1.8G). The resulting advbwdist.csv are identical.
In future it will be necessary to split the aggregating process into
years. Or, to store data for years that won't have changes anymore and
combine existing with freshly computed data.
--
Ticket URL: <https://trac.torproject.org/projects/tor/ticket/25259#comment:1>
Tor Bug Tracker & Wiki <https://trac.torproject.org/>
The Tor Project: anonymity online
More information about the metrics-bugs
mailing list