[tor-bugs] #17321 [CollecTor]: Index to better support downloaders

[Tor Bug Tracker & Wiki]

#17321: Index to better support downloaders
-------------------------+---------------------
 Reporter:  atagar       |          Owner:
     Type:  enhancement  |         Status:  new
 Priority:  major        |      Milestone:
Component:  CollecTor    |        Version:
 Keywords:               |  Actual Points:
Parent ID:               |         Points:
  Sponsor:               |
-------------------------+---------------------
 Hi Karsten, as discussed at the dev meeting it would be sweeeeeet for Stem
 to have a CollecTorDownloader class that supports getting and processing
 descriptors from CollecTor.

 Presently you read and process the Apache index html, but it would it
 could be a lot nicer to have a machine readable index.json file instead.
 This is the ticket for it.

 To support future expansion it would be nice to be able to include
 additional metadata, so I propose a 'contents' hash, such as...

 {{{
 https://collector.torproject.org/recent/index.json

 {
   'resource': 'https://collector.torproject.org/recent/index.json',
   'contents': [
     {
       'name': 'bridge-descriptors',
       'type': 'directory',
       'last_modified': '31-May-2014 09:49',  # or a unix timestam?
       'url': 'https://collector.torproject.org/recent/bridge-descriptors/'
     },
     {
        'name': 'exit-lists',
        'type': 'directory',
        'last_modified': '11-Oct-2015 15:02',
        'url': 'https://collector.torproject.org/recent/exit-lists/'
     },
     {
         'name': 'example-file.txt',
         'type': 'file',
         'last_modified': '31-May-2014 09:50',
         'size': 42901,
         'url': 'https://collector.torproject.org/recent/example-file.txt'
     }
   ]
 }
 }}}

--
Ticket URL: <https://trac.torproject.org/projects/tor/ticket/17321>
Tor Bug Tracker & Wiki <https://trac.torproject.org/>
The Tor Project: anonymity online