[tor-bugs] #6709 [pyonionoo]: Extend Pyonionoo to return details/bandwidth/weights documents

[Tor Bug Tracker & Wiki]

#6709: Extend Pyonionoo to return details/bandwidth/weights documents
-------------------------+--------------------------------------------------
 Reporter:  karsten      |          Owner:  gsathya       
     Type:  enhancement  |         Status:  needs_revision
 Priority:  normal       |      Milestone:                
Component:  pyonionoo    |        Version:                
 Keywords:               |         Parent:                
   Points:               |   Actualpoints:                
-------------------------+--------------------------------------------------
Changes (by karsten):

  * status:  assigned => needs_revision


Comment:

 Having details/bandwidth/weights info in the database would be better.
 Files have the problem that they could change while we're reading them.
 This is actually a weakness of Onionoo that we should overcome in
 pyonionoo.  But it's not as high priority, so maybe add a TODO and leave
 it as it is for now.

 So, I looked at the diff between origin/master and your branch, but I
 couldn't execute your code because of my barely existing Internet
 connectivity.  The following comments are solely from reading your code:
  - It seems that the code for returning details and bandwidth files is 99%
 the same.  In the longer term we should merge their code and add a
 parameter for the type of requested document.  But for the shorter term,
 can you copy that code a third time to support weights files, so that we
 can get closer to deploying pyonionoo?
  - As of September 10, bandwidth and weights files are stored in
 subdirectories, e.g., out/bandwidth/1/2/1234567.. (fingerprint 1234567..)
 or out/weights/f/e/fedcba9.. (fingerprint fedcba9..) to avoid performance
 problems from storing too many files in a single directory.  From the line
 `"path = os.path.join('out/bandwidth', row[1])"` it seems the patch
 doesn't take that into account.  I can't produce a new out/ tarball
 easily, so please git pull onionoo.git and run it to generate a new set of
 out/ data for testing.
  - The `"import pyonionoo.database as get_router"` lines look like they
 are not used.
  - There's a comment asking if order of elements in the response matters.
 The answer is: no.
  - The method for getting details files is `"GET details"` with an s.  It
 seems that web.py thinks it's detail without an s.

 Rest looks good.  I can probably test it more thoroughly next week.
 Thanks!

-- 
Ticket URL: <https://trac.torproject.org/projects/tor/ticket/6709#comment:6>
Tor Bug Tracker & Wiki <https://trac.torproject.org/>
The Tor Project: anonymity online