[tor-bugs] #20395 [Metrics/Library]: Add capability to handle large descriptor files

[Tor Bug Tracker & Wiki]

#20395: Add capability to handle large descriptor files
-----------------------------+--------------------------------
 Reporter:  iwakeh           |          Owner:  karsten
     Type:  defect           |         Status:  needs_revision
 Priority:  Medium           |      Milestone:
Component:  Metrics/Library  |        Version:
 Severity:  Normal           |     Resolution:
 Keywords:                   |  Actual Points:
Parent ID:                   |         Points:
 Reviewer:  iwakeh           |        Sponsor:
-----------------------------+--------------------------------
Changes (by iwakeh):

 * status:  needs_review => needs_revision


Comment:

 Thinking for a while and looking at the proposed branch I'd say it's
 finally time for an API change.

 The proposed changes pretty much result in a blocking queue that knows
 when no more items will be added.

 Therefore, it might be very useful for clients using metrics-lib to have
 `BlockingQueue<Descriptor> descriptorQueue` in `DescriptorReader`, where
 they can choose if they want to call a blocking method or a timeout
 method.  The new `BlockingQueue` could be based on `ArrayBlockingQueue`
 with the additional methods `boolean outOfDescriptors()` and `boolean
 setOutOfDescriptors()`.

 The queue would also facilitate more than one consumer to access the
 descriptors made available instead of iterating or waiting for a list of
 descriptors.  Adding elements in parallel would also be made easier.

 We could provide the new method `BlockingQueue<Descriptor>
 descriptorQueue` while deprecating the old `Iterable<Descriptor>
 readDescriptors` before removing the old method.  Thus, paving the way to
 faster more efficient processing on both the client's and metrics-lib's
 side.

--
Ticket URL: <https://trac.torproject.org/projects/tor/ticket/20395#comment:17>
Tor Bug Tracker & Wiki <https://trac.torproject.org/>
The Tor Project: anonymity online