[tor-bugs] #22514 [Metrics/metrics-lib]: Replace DescriptorReader's setMaxDescriptorFilesInQueue() with setMaxMemory()

[Tor Bug Tracker & Wiki]

#22514: Replace DescriptorReader's setMaxDescriptorFilesInQueue() with
setMaxMemory()
-------------------------------------+--------------------------
     Reporter:  karsten              |      Owner:  metrics-team
         Type:  enhancement          |     Status:  new
     Priority:  Medium               |  Milestone:
    Component:  Metrics/metrics-lib  |    Version:
     Severity:  Normal               |   Keywords:
Actual Points:                       |  Parent ID:
       Points:                       |   Reviewer:
      Sponsor:                       |
-------------------------------------+--------------------------
 We currently let applications define a limit for parsed descriptor files
 in `DescriptorReader#setMaxDescriptorFilesInQueue()`.  If the background
 can parse descriptor files faster than the application can process them,
 it pauses when the configured number of descriptor files is contained in
 the queue and only resumes as soon as the application has dequeued a
 descriptor file.

 As we say in the docs, "the default is 100, but if descriptor files
 contain hundreds or even thousands of descriptors, that default may be too
 high".  But honestly, it's impossible to define a useful number there that
 works for all descriptor types, from microdescriptors to votes or even
 files containing tens or hundreds of concatenated votes.

 The real purpose of having a limit is to avoid running out of memory when
 parsing descriptors.  So, we should just replace
 `setMaxDescriptorFilesInQueue()` with `setMaxMemory()`.  This can't be a
 hard limit, because we cannot put partial descriptors in the queue (or not
 even partial descriptor files right now).  But this limit can define a
 magnitute of available memory, which could be anything between, say, 50M
 and 1G.

 This change is useful on its own, but it also prepares #22141 (after which
 we won't add `DescriptorFile` instances to the queue anymore, but
 `Descriptor` instances) and #21751 (when we'll likely want to define a
 `setMaxThreads()` method).

 I'll attach a branch as soon as I have a ticket number.  The code there is
 for discussion, not meant to be merged just yet.  It provides another
 method to define an upper limit for descriptors (not descriptor files)
 which I briefly considered but deemed less useful than a limit for
 descriptor bytes.

--
Ticket URL: <https://trac.torproject.org/projects/tor/ticket/22514>
Tor Bug Tracker & Wiki <https://trac.torproject.org/>
The Tor Project: anonymity online