[tor-bugs] #20335 [Metrics/CollecTor]: ReferenceChecker causes OOM

[Tor Bug Tracker & Wiki]

#20335: ReferenceChecker causes OOM
-------------------------------+---------------------
 Reporter:  iwakeh             |          Owner:
     Type:  defect             |         Status:  new
 Priority:  High               |      Milestone:
Component:  Metrics/CollecTor  |        Version:
 Severity:  Normal             |     Resolution:
 Keywords:                     |  Actual Points:
Parent ID:                     |         Points:
 Reviewer:                     |        Sponsor:
-------------------------------+---------------------
Changes (by karsten):

 * priority:  Medium => High


Comment:

 I ''believe'' that's a smart thing to do.  I'm not sure if that would
 break something.

 But I think it doesn't solve the issue here.  Or, I ran out of memory with
 a similar patch applied.  And I think I know why:

 {{{
 -rw-r--r--  1 karsten  staff   2.1G Oct 18 11:52
 /Users/karsten/src/collector/recent/relay-
 descriptors/votes/2016-10-18-09-22-46-votes
 }}}

 With the recent change to append all votes to a single file, we're
 creating a 2.1G large file when doing the first synchronization run.
 However, metrics-lib wants to read that entire file to memory which
 probably doubles in size when it's parsed.  Ugh!

 Note that the CollecTor's reference checker is just one application that
 breaks when input files are too large.  Other applications using metrics-
 lib would similarly be affected.

 A longer-term fix might be to change metrics-lib to read through large
 files and pause when it has parsed enough descriptors that the application
 still has to process.

 A short-term fix would be to stick with the current naming scheme for
 files in the `recent/` directory, as that would have avoided this issue.

 Fun times!

--
Ticket URL: <https://trac.torproject.org/projects/tor/ticket/20335#comment:10>
Tor Bug Tracker & Wiki <https://trac.torproject.org/>
The Tor Project: anonymity online