[tor-bugs] #4439 [Metrics Utilities]: Develop a Java/Python API that wraps relay descriptor sources and provides unified access to them

[Tor Bug Tracker & Wiki]

#4439: Develop a Java/Python API that wraps relay descriptor sources and provides
unified access to them
-------------------------------+--------------------------------------------
 Reporter:  karsten            |          Owner:  karsten
     Type:  task               |         Status:  new    
 Priority:  normal             |      Milestone:         
Component:  Metrics Utilities  |        Version:         
 Keywords:                     |         Parent:         
   Points:                     |   Actualpoints:         
-------------------------------+--------------------------------------------

Comment(by atagar):

 > that makes various Tor descriptor types available for statistical
 analysis and for building services and applications.

 The main value of this library is being able to pull consensus information
 via several methods without a tor instance, yes? This description seems to
 focus more on the use cases you're planning rather than what the library
 does. Maybe alternatively phrase this as "that directly fetches consensus
 information from a variety of sources like cached descriptors and
 directory authorities/mirrors."

 > include relay and bridge descriptors

 How does a bridge descriptor differ from relays? I don't think that I've
 ever dealt with them.

 > The descriptor source then makes the descriptors available in a
 descriptor store.

 Ahhh interesting, I hadn't thought of eagerly loaded consensus snapshots.
 As the last sentence mentions this works well for batch jobs, being
 simpler for callers and more faithful to how consensuses are published.
 However, it also comes with the drawbacks of a lengthy initialization and
 high memory usage. Can we follow an iteration or callback pattern instead
 so we can process the descriptors as they come in (and free the memory)?

 I had been planning on an api where we lazy load and cache descriptors
 requested by our caller. That would have been better for some use cases,
 but certainly not when we want to process the majority of the consensus.
 It might also not be realistic with how we can fetch consensuses
 information when detached from the control socket (I assume when dealing
 with authorities and cached consensuses fetching is more of an all-or-
 nothing operation).

 Cheers! -Damian

-- 
Ticket URL: <https://trac.torproject.org/projects/tor/ticket/4439#comment:9>
Tor Bug Tracker & Wiki <https://trac.torproject.org/>
The Tor Project: anonymity online