[tor-bugs] #20335 [Metrics/CollecTor]: ReferenceChecker causes OOM

[Tor Bug Tracker & Wiki]

#20335: ReferenceChecker causes OOM
-------------------------------+---------------------
 Reporter:  iwakeh             |          Owner:
     Type:  defect             |         Status:  new
 Priority:  High               |      Milestone:
Component:  Metrics/CollecTor  |        Version:
 Severity:  Normal             |     Resolution:
 Keywords:                     |  Actual Points:
Parent ID:                     |         Points:
 Reviewer:                     |        Sponsor:
-------------------------------+---------------------

Comment (by iwakeh):

 Replying to [comment:10 karsten]:
 > I ''believe'' that's a smart thing to do.  I'm not sure if that would
 break something.

 Unfortunately, you're right and this was not the reason for the oom :-/

 >
 > But I think it doesn't solve the issue here.  Or, I ran out of memory
 with a similar patch applied.  And I think I know why:
 >
 > {{{
 > -rw-r--r--  1 karsten  staff   2.1G Oct 18 11:52
 /Users/karsten/src/collector/recent/relay-
 descriptors/votes/2016-10-18-09-22-46-votes
 > }}}
 >
 > With the recent change to append all votes to a single file, we're
 creating a 2.1G large file when doing the first synchronization run.
 However, metrics-lib wants to read that entire file to memory which
 probably doubles in size when it's parsed.  Ugh!
 >
 > Note that the CollecTor's reference checker is just one application that
 breaks when input files are too large.  Other applications using metrics-
 lib would similarly be affected.
 >
 > A longer-term fix might be to change metrics-lib to read through large
 files and pause when it has parsed enough descriptors that the application
 still has to process.

 Did you verify that metrics-lib is the culprit? Like, reading the 2G file
 for parsing?

 >
 > A short-term fix would be to stick with the current naming scheme for
 files in the `recent/` directory, as that would have avoided this issue.
 >

 The commit for the review in #18910 is easily adaptable, if necessary.

 > Fun times!

 Wouldn't it be boring without the unforeseen ;-)

--
Ticket URL: <https://trac.torproject.org/projects/tor/ticket/20335#comment:11>
Tor Bug Tracker & Wiki <https://trac.torproject.org/>
The Tor Project: anonymity online