[metrics-bugs] #25161 [Metrics/CollecTor]: Fix another memory problem with the webstats bulk import

Tor Bug Tracker & Wiki blackhole at torproject.org
Tue Feb 6 15:50:58 UTC 2018

#25161: Fix another memory problem with the webstats bulk import
 Reporter:  karsten            |          Owner:  metrics-team
     Type:  defect             |         Status:  new
 Priority:  Medium             |      Milestone:
Component:  Metrics/CollecTor  |        Version:
 Severity:  Normal             |     Resolution:
 Keywords:                     |  Actual Points:
Parent ID:                     |         Points:
 Reviewer:                     |        Sponsor:

Comment (by karsten):

 Replying to [comment:1 iwakeh]:
 > I didn't take a close look at the proposed code changes/additions, yet.
 In #25100 I suggested to partition the import by date, b/c that gives the
 reduction of heap usage and also is fine for the bulk import where we know
 one-file-one-date is true.

 Yes, my code implements such a partition by date. Though you have a point
 there with one-file-one-date. I could simplify the code a lot. Let me do
 that. (I think it won't affect the memory issue, though.)

 > > Or is this just the operation where we happen to run out of memory
 from accumulating stuff over time, and where fixing this issue would just
 mean that we're failing somewhere else, shortly after?
 > I think this is the case.  We're also - opposed to all other CollecTor
 modules - compressing before writing.

 Okay. So maybe there's room for improvement here.

 > What heap setting is used, 8G?  How many cores are available?

 `-Xmx16g` with 16 GiB RAM available. 4 cores.

Ticket URL: <https://trac.torproject.org/projects/tor/ticket/25161#comment:3>
Tor Bug Tracker & Wiki <https://trac.torproject.org/>
The Tor Project: anonymity online

More information about the metrics-bugs mailing list