[tor-bugs] #25100 [Metrics/CollecTor]: Make CollecTor's webstats module use less RAM and CPU time

[Tor Bug Tracker & Wiki]

#25100: Make CollecTor's webstats module use less RAM and CPU time
-------------------------------+--------------------------
 Reporter:  karsten            |          Owner:  iwakeh
     Type:  enhancement        |         Status:  accepted
 Priority:  High               |      Milestone:
Component:  Metrics/CollecTor  |        Version:
 Severity:  Normal             |     Resolution:
 Keywords:                     |  Actual Points:
Parent ID:                     |         Points:
 Reviewer:                     |        Sponsor:
-------------------------------+--------------------------

Comment (by iwakeh):

 Please review
 [https://gitweb.torproject.org/user/iwakeh/collector.git/log/?h=next two
 commits] one adapting to the changes in #25103
 ([https://gitweb.torproject.org/user/iwakeh/metrics-
 lib.git/commit/?h=next&id=f6cfd4b271601ea97a014d9ae68736376280ed9c this
 commit]) and one using more parallelization.

 The two more metrics-lib tweaks, the adaption for collector as well as
 adding more parallelization improved time and memory usage a little.
 The initial median memory usage was 5.867G (max 6.671G).  Now, it is
 possible to process the 516 meronense logs with memory of 6G (median
 4.541G, max 5.647G) in less than three minutes (2:36min).

 Testing with 8867 logs ...

--
Ticket URL: <https://trac.torproject.org/projects/tor/ticket/25100#comment:5>
Tor Bug Tracker & Wiki <https://trac.torproject.org/>
The Tor Project: anonymity online