[tor-bugs] #25100 [Metrics/CollecTor]: Make CollecTor's webstats module use less RAM and CPU time

[Tor Bug Tracker & Wiki]

#25100: Make CollecTor's webstats module use less RAM and CPU time
-------------------------------+--------------------------------
 Reporter:  karsten            |          Owner:  iwakeh
     Type:  enhancement        |         Status:  needs_revision
 Priority:  High               |      Milestone:
Component:  Metrics/CollecTor  |        Version:
 Severity:  Normal             |     Resolution:
 Keywords:                     |  Actual Points:
Parent ID:                     |         Points:
 Reviewer:                     |        Sponsor:
-------------------------------+--------------------------------

Comment (by iwakeh):

 True, so far we didn't trade memory for time, but got some improvements
 that could be picked easily even winning some time here.
 Keeping counts of different sanitized lines in memory could also help and
 might be only a small change; I'm looking into this next.

 But first, we should make sure that the performance tuning focuses on the
 usual scenario before starting the bigger changes.

 1. The usual import amount will logs of a few days, not the yearly logs,
 right?
 2. Major bulk imports like the initial one should work, but also appear
 very rarely.  Correct?

 Do you have some reasonable figures as example for each?

--
Ticket URL: <https://trac.torproject.org/projects/tor/ticket/25100#comment:7>
Tor Bug Tracker & Wiki <https://trac.torproject.org/>
The Tor Project: anonymity online