Hi Karsten,
We have a preliminary version of the Pyonionoo front-end at
git://github.com/meganchang/pyonionoo.git
Make sure to look at the database branch. We'd be happy to hear feedback.
Unfortunately, this isn't as far as we wanted to be. A run-down of the main open issues:
* We ran into issues with getting sqlite in-memory databases to play nicely with multiple threads (stop-gap solution: use an on-disk database).
* Right now it just has the summary handler, though adding the other handlers probably isn't a problem (probably a bit of minor refactoring).
* The search GET parameter isn't implemented yet, in part because we realized that implementing it as specified in the protocol isn't so straightforward, at least in SQLite (which does not, in an obvious way, support searching for initial substrings).
- Norman
On 8/9/12 5:28 PM, Norman Danner wrote:
Hi Karsten,
We have a preliminary version of the Pyonionoo front-end at
git://github.com/meganchang/pyonionoo.gitMake sure to look at the database branch. We'd be happy to hear feedback.
Unfortunately, this isn't as far as we wanted to be. A run-down of the main open issues:
- We ran into issues with getting sqlite in-memory databases to play
nicely with multiple threads (stop-gap solution: use an on-disk database).
It occurs to me that twisted.enterprise.adbapi probably fixes provides a solution to these issues. If it were documented, I'd know better... We'll play with it tomorrow and see what happens.
- Norman
Hi Norman,
On 8/10/12 3:15 AM, Norman Danner wrote:
On 8/9/12 5:28 PM, Norman Danner wrote:
Hi Karsten,
We have a preliminary version of the Pyonionoo front-end at
git://github.com/meganchang/pyonionoo.gitMake sure to look at the database branch. We'd be happy to hear feedback.
First feedback is that I'm having trouble installing cyclone. :/ What steps does it take from, say, a vanilla Debian Squeeze to an environment where `twistd -n cyclone -r pyonionoo.web.Application` works? Ideally, these steps only include `apt-get install`, but if something else is needed, I'll try to convince weasel to run that on the Tor machine.
Some more feedback after a _very_ quick look through the files:
- Why do we need something like scripts/cookie_secret.py?
- Do the two handlers bandwidth.py and detail.py really need to list all the fields in the respective documents? That seems hard to maintain when we make changes to the back-end. Ideally, the front-end would simply consider these files as valid JSON strings without parsing them at all.
- pyonionoo/summary looks like example data that shouldn't be in the Git repo. Similarly, summary.db is a binary file and probably shouldn't be in the Git repo. Maybe the latter should be in .gitignore.
I'll have more feedback after the weekend, and even more once I get this thing to run. :) Very much looking forward to that!
Thanks! Karsten
Unfortunately, this isn't as far as we wanted to be. A run-down of the main open issues:
- We ran into issues with getting sqlite in-memory databases to play
nicely with multiple threads (stop-gap solution: use an on-disk database).
It occurs to me that twisted.enterprise.adbapi probably fixes provides a solution to these issues. If it were documented, I'd know better... We'll play with it tomorrow and see what happens.
- Norman
Hi Karsten,
On 8/10/12 8:29 AM, Karsten Loesing wrote:
Hi Norman,
On 8/10/12 3:15 AM, Norman Danner wrote:
On 8/9/12 5:28 PM, Norman Danner wrote:
Hi Karsten,
We have a preliminary version of the Pyonionoo front-end at
git://github.com/meganchang/pyonionoo.gitMake sure to look at the database branch. We'd be happy to hear feedback.
First feedback is that I'm having trouble installing cyclone. :/ What steps does it take from, say, a vanilla Debian Squeeze to an environment where `twistd -n cyclone -r pyonionoo.web.Application` works? Ideally, these steps only include `apt-get install`, but if something else is needed, I'll try to convince weasel to run that on the Tor machine.
Oh dear; hadn't realized this was an issue. We installed Twisted and Cyclone from source.
There is a debian package for twisted (python-twisted), but not for Cyclone.
The issues below are pretty straightforward. This one seems more of a problem.
Some more feedback after a _very_ quick look through the files:
- Why do we need something like scripts/cookie_secret.py?
We probably don't. We just jumped off of Sathya's first pass, and didn't play with the Cyclone details at all.
- Do the two handlers bandwidth.py and detail.py really need to list all
the fields in the respective documents? That seems hard to maintain when we make changes to the back-end. Ideally, the front-end would simply consider these files as valid JSON strings without parsing them at all.
No, those are old versions and will be completely replaced. We'll follow your approach for these handlers: use the query parameters to determine the fingerprints of routers that match the parameters, then just respond with the corresponding JSON files from the filesystem.
Hopefully we'll have an example of one of these that should enable one to plug in Sathya's new back-end without modifying the handler classes, just the database module.
- pyonionoo/summary looks like example data that shouldn't be in the Git
repo. Similarly, summary.db is a binary file and probably shouldn't be in the Git repo. Maybe the latter should be in .gitignore.
Correct, that shouldn't be there.
Hi Norman,
On 8/10/12 4:10 PM, Norman Danner wrote:
Oh dear; hadn't realized this was an issue. We installed Twisted and Cyclone from source.
There is a debian package for twisted (python-twisted), but not for Cyclone.
I managed to install Cyclone with Sathya's apt-get/pip instructions. (I also had to apt-get install python-dev, or pip would fail; not that its error message was helpful, but Google was.)
So, I don't have a clear preference for or against Cyclone. On the one hand, getting rid of that dependency would be nice, in particular given that pip fails to handle basic problems such as a missing python-dev. But on the other hand, I'd like to get the current pyonionoo running without having to rewrite major parts of it first. How about we stick with Cyclone for the moment and postpone the decision of a possible rewrite towards a Twisted-only version?
Some more feedback after a _very_ quick look through the files:
- Why do we need something like scripts/cookie_secret.py?
We probably don't. We just jumped off of Sathya's first pass, and didn't play with the Cyclone details at all.
Sathya, what do you think about that file/feature? Can we get rid of it?
- Do the two handlers bandwidth.py and detail.py really need to list all
the fields in the respective documents? That seems hard to maintain when we make changes to the back-end. Ideally, the front-end would simply consider these files as valid JSON strings without parsing them at all.
No, those are old versions and will be completely replaced. We'll follow your approach for these handlers: use the query parameters to determine the fingerprints of routers that match the parameters, then just respond with the corresponding JSON files from the filesystem.
Hopefully we'll have an example of one of these that should enable one to plug in Sathya's new back-end without modifying the handler classes, just the database module.
Okay. Focusing on summary documents first then.
Thanks, Karsten
Hi,
Some more feedback after a _very_ quick look through the files:
- Why do we need something like scripts/cookie_secret.py?
We probably don't. We just jumped off of Sathya's first pass, and didn't play with the Cyclone details at all.
Sathya, what do you think about that file/feature? Can we get rid of it?
We don't need it. Feel free to expunge it. (If not, I'll do it)
--Sathya
First feedback is that I'm having trouble installing cyclone. :/ What steps does it take from, say, a vanilla Debian Squeeze to an environment where `twistd -n cyclone -r pyonionoo.web.Application` works? Ideally, these steps only include `apt-get install`, but if something else is needed, I'll try to convince weasel to run that on the Tor machine.
The 'apt-get' way to install python packages is 'pip'.
$ apt-get install python-twisted-web python-twisted-core python-pip $ pip install cyclone
Thanks, -Sathya
-
Karsten Loesing -
Norman Danner -
Sathyanarayanan Gunasekaran