[metrics-bugs] #25259 [Metrics/Statistics]: Tune advbwdist module of metrics-web

Tor Bug Tracker & Wiki blackhole at torproject.org
Wed Feb 14 18:16:29 UTC 2018

#25259: Tune advbwdist module of metrics-web
 Reporter:  iwakeh              |          Owner:  metrics-team
     Type:  enhancement         |         Status:  new
 Priority:  Medium              |      Milestone:
Component:  Metrics/Statistics  |        Version:
 Severity:  Normal              |     Resolution:
 Keywords:                      |  Actual Points:
Parent ID:                      |         Points:
 Reviewer:                      |        Sponsor:

Comment (by iwakeh):

 Please review [https://gitweb.torproject.org/user/iwakeh/metrics-
 web.git/commit/?h=task-25259 these tuning steps].
 From the commit comment:
 Processing advbwdist-validafter.csv (350M) took 150 seconds and used up to
 Performing pre-processing separately, helping R by defining read types,
 and avoiding multiple casting operations led to halving the processing
 time (to 77 seconds) and reducing the necessary memory to about 25%
 (approx. 1.8G). The resulting advbwdist.csv are identical.

 In future it will be necessary to split the aggregating process into
 years.  Or, to store data for years that won't have changes anymore and
 combine existing with freshly computed data.

Ticket URL: <https://trac.torproject.org/projects/tor/ticket/25259#comment:1>
Tor Bug Tracker & Wiki <https://trac.torproject.org/>
The Tor Project: anonymity online

More information about the metrics-bugs mailing list