[tor-bugs] #7987 [Metrics Website]: Descriptor types missing network-status-microdesc-consensus-3

[Tor Bug Tracker & Wiki]

#7987: Descriptor types missing network-status-microdesc-consensus-3
-----------------------------+----------------------------------------------
 Reporter:  atagar           |          Owner:     
     Type:  defect           |         Status:  new
 Priority:  normal           |      Milestone:     
Component:  Metrics Website  |        Version:     
 Keywords:                   |         Parent:     
   Points:                   |   Actualpoints:     
-----------------------------+----------------------------------------------

Comment(by karsten):

 Replying to [comment:4 atagar]:
 > Hmmm. Would you mind expanding the @type annotations to include things
 not on the metrics site?

 Not at all.  Which annotations should we add?

 > You earlier made a list of tasks we needed to make stem the primary
 descriptor parsing library (ideally so we don't need to continue
 maintaining metrics-lib as well). Thoughts on the next step?
 >
 > https://trac.torproject.org/projects/tor/wiki/doc/stem#Projects

 Quoting the not yet done steps here:

 > Parse microdescriptors - Smaller replacement for server descriptors.

 Microdescriptors are not relevant for metrics.

 > Parse bridge pool assignments - Published by BridgeDB and sanitized by
 metrics.

 This may become relevant once we make progress on the metrics-web
 replacement.  This might be in the next few weeks.

 > Parse exit list entry - Published by DNSEL or TorBEL to indicate what ip
 address exit relay X had at timestamp Y.

 Same as bridge pool assignments.

 > Parse  Torperf output - Performance data measured by making periodic
 requests over the Tor network. We'll want to implement #3036 first.

 This one should be postponed until after the Torperf rewrite.  Hopefully,
 we'll have a new Torperf data format in a month from now.

 > Port  Onionoo - See #6452 - One of the chief users of metrics-lib,
 Onionoo is the data provider for  Atlas. There's a  design document which
 might be a good starting point.

 Sathya and I recently agreed to postpone this and resume Onionoo
 development in Java.  Porting Onionoo isn't as simple as it seems.  We'll
 need somebody with two months of free time for this.  We should revisit
 this in 6 months from now.

 > Remote descriptor fetching - Ability to fetch descriptors via an
 authority or directory mirror's DirPort. This involves some tricky, but
 important performance optimizations like making requests to directories in
 parallel, requesting up to 96 descriptors in a single HTTP GET, using .z
 compression, etc.
 > Port DocTor - Fetches current consensuses and votes and outputs
 consensus problems.

 These two would be nice.  This is not related to the other stuff we're
 currently working on.  If I have to do this myself, that's going to take
 months.  But this could be done by an interested volunteer.  Happy to
 guide that volunteer and review code.

 > Relay class - Convenience class for commonly requested relay attributes,
 which lazily loads a relay's server descriptor, microdescriptor, and
 network status entity as needed. This is more a todo item for the
 Controller class since it requires a control socket.

 Okay, unrelated to metrics, it seems.

 > Port the statistics-aggregating portion of metrics-web - Provides data
 for  graphs on the metrics website. Once we have that we only need a new
 metrics website before we can kill metrics-web.

 I'm currently working on this together with a volunteer.  Soon there will
 be code.

 > Port  ExoneraTor - A website that tells you whether some IP address was
 a Tor relay. We could probably re-use the existing database schema with
 minor tweaks.
 > Searchable Tor descriptor and metrics data  archive - Once we have that,
 we can turn off  relay search and maybe even  ExoneraTor.

 I think I'd prefer a new searchable data archive that makes ExoneraTor
 obsolete over a simple rewrite of ExoneraTor.

 > Replace metrics-lib - Part of the goal of this project is to deprecate
 metrics-lib in favor of stem so we have one fewer services to maintain.
 When the above is done we should be very, very close - last step is to
 double check that we haven't missed anything. Realistically, we're talking
 about years here, not months.

 Ack.

-- 
Ticket URL: <https://trac.torproject.org/projects/tor/ticket/7987#comment:6>
Tor Bug Tracker & Wiki <https://trac.torproject.org/>
The Tor Project: anonymity online