[metrics-bugs] #30219 [Metrics/CollecTor]: Add Tom's bandwidth file archive to CollecTor

[Tor Bug Tracker & Wiki]

#30219: Add Tom's bandwidth file archive to CollecTor
-------------------------------------------------+-------------------------
 Reporter:  irl                                  |          Owner:
                                                 |  metrics-team
     Type:  enhancement                          |         Status:  new
 Priority:  Medium                               |      Milestone:
Component:  Metrics/CollecTor                    |        Version:
 Severity:  Normal                               |     Resolution:
 Keywords:  tor-bwauth,tor-dirauth,metrics-      |  Actual Points:
  roadmap-2019-q2                                |
Parent ID:  #21378                               |         Points:
 Reviewer:                                       |        Sponsor:
-------------------------------------------------+-------------------------

Comment (by karsten):

 Okay, I ''finally'' got around to trying a local import with CollecTor. At
 least importing a small sample of these bandwidth files worked okay.

 However, I'm wondering if we need to make an enhancement before import
 these bandwidth files into CollecTor. Consider these subdirectories in
 Tom's tarball:

 {{{
  24G    bastet
 8.2G    faravahar
  22G    gabelmoo
  29G    maatuska
 2.2G    maatuska-21697
 1.9G    maatuska-fastly
 616M    maatuska-nodns
  58M    maatuska-nofasthop
 3.5G    maatuska-vanilla
  26G    moria1
 }}}

 As of now, when we import these files into CollecTor we're losing meta
 data like ''source'' and human-readable ''annotations'' like whether DNS
 was broken or which bandwidth file server was used. We could provide those
 annotations via some timeline (e.g., by saying when maatuska switched to
 Fastly), but there's no good way to retain source information in these
 files.

 Note that we're facing the same issue with current bandwidth files. We
 just said that they'll be referenced from votes which provides source
 information indirectly.

 Also note that we briefly discussed including source information in the
 file name. But even if we do that, we should consider adding something to
 the file contents, most likely as an annotation in order to keep the
 digest unchanged. We should not put relevant information ''only'' in the
 file name.

 Hmmm.

--
Ticket URL: <https://trac.torproject.org/projects/tor/ticket/30219#comment:3>
Tor Bug Tracker & Wiki <https://trac.torproject.org/>
The Tor Project: anonymity online