Hi Megan, hi Erik,
cc'ing Sathya and tor-dev, hoping you don't mind.
On 7/27/12 5:21 PM, Megan Chang wrote:
We are currently almost done with the first draft of pyonionoo.
Awesome! :)
We were wondering what some of the entries might look like so we would be able to see what information was being parsed to fill in some of the fields such as the bandwidth handler's "write_history" and "read_history". We noticed that the router descriptor format includes "write_history" and "read_history" according to torspec but again, we aren't too sure on what these entries look like.
The bandwidth documents are tricky, because one cannot just look up these values from Tor descriptors. They're based on histories in extra-info descriptors, but they aggregate these values on different levels of detail, and they're specifically designed for graphing. I don't know about your schedule, but if you're short on time, I'd say let's discuss summary and details documents first and ignore bandwidth documents for the moment. (Of course, if you have plenty of time left, happy to discuss bandwidth documents more!)
Is your code online somewhere? I'd like to take a look and give you some feedback about Tor descriptor specific things (not so much about Python/Twisted (?) things).
Also, we don't know how to approach the fields in the details handler that require GeoIP information. Any tips on this?
Onionoo uses Maxmind's Java API, their GeoIP database, and their AS database to resolve IP addresses to country/AS information. There's a Python API, too, which I think Sathya used for his network entropy code:
https://gitweb.torproject.org/metrics-tasks.git/tree/HEAD:/task-6232/pygeoip
If you're unsure how to use it, best ask Sathya for more details.
Best, Karsten
Hi,
We were wondering what some of the entries might look like so we would be able to see what information was being parsed to fill in some of the fields such as the bandwidth handler's "write_history" and "read_history". We noticed that the router descriptor format includes "write_history" and "read_history" according to torspec but again, we aren't too sure on what these entries look like.
If you want some sample summary, bandwidth, or details documents to see see how they look like, I can send you the ones I got from Karsten during the Tor dev meeting.
Also, we don't know how to approach the fields in the details handler that require GeoIP information. Any tips on this?
Onionoo uses Maxmind's Java API, their GeoIP database, and their AS database to resolve IP addresses to country/AS information. There's a Python API, too, which I think Sathya used for his network entropy code:
https://gitweb.torproject.org/metrics-tasks.git/tree/HEAD:/task-6232/pygeoip
Here is the Python API for parsing the GeoIP database: http://appliedsec.github.com/pygeoip/
Although, I'm not entirely sure why the web service part of pyonionoo would have to parse GeoIP data or calculate bandwidth histories?
Also, I have a ticket to track the progress of the pyonionoo backend in case anyone's interested: https://trac.torproject.org/projects/tor/ticket/6452
Thanks! -Sathya
On 7/28/12 11:44 AM, Sathyanarayanan Gunasekaran wrote:
If you want some sample summary, bandwidth, or details documents to see see how they look like, I can send you the ones I got from Karsten during the Tor dev meeting.
Oh, I wasn't aware that Sathya is working on the backend and Megan and Erik are working on the frontend. In that case my previous reply doesn't make as much sense anymore.
Here's a current tarball of Onionoo's out/ directory that the frontend should use (16.8M):
https://people.torproject.org/~karsten/volatile/onionoo-out-2012-07-28.tar.b...
Although, I'm not entirely sure why the web service part of pyonionoo would have to parse GeoIP data or calculate bandwidth histories?
Indeed. Should become clearer when looking at the tarball above. The details and bandwidth files (as well as the new weights files) are ready to be served to clients. No need to look up anything while doing so. This is the part that's supposed to be crazy fast.
Best, Karsten
-
Karsten Loesing -
Sathyanarayanan Gunasekaran