[tor-bugs] #25100 [Metrics/CollecTor]: Make CollecTor's webstats module use less RAM and wall time

[Tor Bug Tracker & Wiki]

#25100: Make CollecTor's webstats module use less RAM and wall time
-------------------------------+------------------------------
 Reporter:  karsten            |          Owner:  iwakeh
     Type:  enhancement        |         Status:  needs_review
 Priority:  High               |      Milestone:
Component:  Metrics/CollecTor  |        Version:
 Severity:  Normal             |     Resolution:
 Keywords:                     |  Actual Points:
Parent ID:                     |         Points:
 Reviewer:                     |        Sponsor:
-------------------------------+------------------------------
Changes (by iwakeh):

 * status:  needs_revision => needs_review


Comment:

 Please review
 [https://gitweb.torproject.org/user/iwakeh/collector.git/log/?h=task-25100
 two small commits] applying parallelization tweaks and making use of the
 metrics-lib changes.

 With [https://gitweb.torproject.org/user/iwakeh/metrics-
 lib.git/log/?h=task-25103 metrics-lib task-25103 branch] meronense's 516
 logs can be processed using 2G (heap usage throughout was mostly below
 1.5G and with three peaks slightly above 1.5G) in 2.5 min.

 Test runs processing of larger imports using 8G are still on the way.  So
 far, the memory handling looks fine; mostly well below 4G, only the
 processing of aus1.tp.o logs, which are quite large needed the 8G to work.
 Considering that these are the logs for a year (327 days) this should be
 ok and CollecTor is surely able to handle the regular daily imports and
 occasionally imports of a few months now with modest RAM and time demand.

 I test further and post results, but I think the changes here and those in
 metrics-lib can be reviewed now.

--
Ticket URL: <https://trac.torproject.org/projects/tor/ticket/25100#comment:14>
Tor Bug Tracker & Wiki <https://trac.torproject.org/>
The Tor Project: anonymity online