Hi, in the last couple of days I thought about a topic for a gsoc application.
After getting ideas from Karsten and Sathya, I started making a rough roadmap of things that I want to do for globe, in the next couple of months (and maybe use it as a idea for a gsoc topic).
- add new onionoo features to globe - add compass functionality + open compass tickets to globe - improve/add help documentation
The central point would be to add compass functionality to globe. This has the advantage that the user can use the same application to search for relay/bridge details and do queries with grouped results.
This could be accomplished by changing the UI and modify advanced search filters, add options for grouping and grouped search results. Another suggestion was to add a group detail page that displays aggregated data for multiple details.
Currently compass pulls all onionoo relay details and queries them locally. Karsten suggested that globe could use the same method and load the onionoo data every hour or so and do the currently supported search and compass functionality on it. The idea is that globe could store the dump in a db (e.g. mongodb) and use its features for searching, grouping and aggregating data.
This would be a major rewrite away from calling the Onionoo-API, to querying the locally available dump.
I would like to know, if you think that integrating Compass into globe would be a good idea? Do you think that the compass integration would be a reasonable project for the gsoc? Do you have other ideas on how to improve globe?
Thanks, Christian
Hey Christian,
On Tue, Mar 4, 2014 at 2:28 PM, Christian me@rndm.de wrote:
Hi, in the last couple of days I thought about a topic for a gsoc application.
After getting ideas from Karsten and Sathya, I started making a rough roadmap of things that I want to do for globe, in the next couple of months (and maybe use it as a idea for a gsoc topic).
- add new onionoo features to globe
Why? Do we want to deprecate onionoo?
- add compass functionality + open compass tickets to globe
- improve/add help documentation
These would be awesome!
The central point would be to add compass functionality to globe. This has the advantage that the user can use the same application to search for relay/bridge details and do queries with grouped results.
Sounds good.
This could be accomplished by changing the UI and modify advanced search filters, add options for grouping and grouped search results. Another suggestion was to add a group detail page that displays aggregated data for multiple details.
Currently compass pulls all onionoo relay details and queries them locally. Karsten suggested that globe could use the same method and load the onionoo data every hour or so and do the currently supported search and compass functionality on it. The idea is that globe could store the dump in a db (e.g. mongodb) and use its features for searching, grouping and aggregating data.
Is this going to be part of globe-node? I don't think there is a need to rewrite the Compass backend(the logic) -- it exposes a REST API which can just be consumed by Globe. Majority of the open Compass tickets are bugs in the frontend.
This would be a major rewrite away from calling the Onionoo-API, to querying the locally available dump.
I don't think this is a good idea. There's no need to rewrite the existing Globe code.
I would like to know, if you think that integrating Compass into globe would be a good idea?
Yes.
Do you think that the compass integration would be a reasonable project for the gsoc?
Yes. Bonus points if you can leverage the data provided by Compass for some nice graphs like heat maps (based on probability) or something like that.
Do you have other ideas on how to improve globe?
1) Add the Compass frontend to Globe 2) Refactor the Compass UI -- I don't think the current compass UI is intuitive enough and could be extended to display more information in a nicer way. 3) Add visualizations and graphs based on Compass data
Thanks, --Sathya
Hi Christian, Sathya,
On 05/03/14 01:39, Sathyanarayanan Gunasekaran wrote:
Hey Christian,
On Tue, Mar 4, 2014 at 2:28 PM, Christian me@rndm.de wrote:
Hi, in the last couple of days I thought about a topic for a gsoc application.
After getting ideas from Karsten and Sathya, I started making a rough roadmap of things that I want to do for globe, in the next couple of months (and maybe use it as a idea for a gsoc topic).
- add new onionoo features to globe
Why? Do we want to deprecate onionoo?
Christian and I discussed this in private mail before moving this thread to tor-dev@. This bullet point should have said "Add support for new Onionoo features to Globe", or something like that. Examples are vote details (#9778), bridge user graphs (#10331), monthly stats per relay/bridge (#11041), list of transports (#11052), etc. These are all fine things to add to Globe, and maybe it's fine to mention them in the second half of a GSoC proposal. But they all depend on the Onionoo guy to write code first, so a Globe GSoC project shouldn't depend on them.
- add compass functionality + open compass tickets to globe
- improve/add help documentation
These would be awesome!
Agreed!
The central point would be to add compass functionality to globe. This has the advantage that the user can use the same application to search for relay/bridge details and do queries with grouped results.
Sounds good.
Agreed!
This could be accomplished by changing the UI and modify advanced search filters, add options for grouping and grouped search results. Another suggestion was to add a group detail page that displays aggregated data for multiple details.
Right. For example, a group detail page could show bandwidth and weights graphs for all relays in the group.
Currently compass pulls all onionoo relay details and queries them locally. Karsten suggested that globe could use the same method and load the onionoo data every hour or so and do the currently supported search and compass functionality on it. The idea is that globe could store the dump in a db (e.g. mongodb) and use its features for searching, grouping and aggregating data.
Is this going to be part of globe-node? I don't think there is a need to rewrite the Compass backend(the logic) -- it exposes a REST API which can just be consumed by Globe. Majority of the open Compass tickets are bugs in the frontend.
This sounds like a fine idea to integrate Compass into Globe very quickly and ask for user feedback early in the process. Agile!
But in the long term I suggest we move the grouping logic of Compass to Globe and decomission Compass. Reasons: - that's one tool less to maintain, and Christian seems to be more of a JavaScript person, not Python (IIRC); - Globe is going to be client-server soon in order to remove the need for JavaScript in the browser, and then it's only a small step to do the same things that Compass does right now; - Onionoo's search functionality is kinda hard to extend, so it would be better if Globe did its own search based on locally cached documents.
However, as I said above, fine to start by querying Compass!
This would be a major rewrite away from calling the Onionoo-API, to querying the locally available dump.
I don't think this is a good idea. There's no need to rewrite the existing Globe code.
(See above.)
I would like to know, if you think that integrating Compass into globe would be a good idea?
Yes.
Do you think that the compass integration would be a reasonable project for the gsoc?
Yes. Bonus points if you can leverage the data provided by Compass for some nice graphs like heat maps (based on probability) or something like that.
I think that's something for the metrics website, not for Globe. Onionoo and Globe are meant to provide status information on relays or bridges, whereas the metrics website provides aggregate data covering the entire Tor network. There are exceptions like the bubble graphs (https://metrics.torproject.org/bubbles.html) which use Onionoo's data, but in theory those should be implemented in the metrics website only. So, I'd say let's not mix scopes here, and let's not try to do too much in a single summer.
Do you have other ideas on how to improve globe?
- Add the Compass frontend to Globe
- Refactor the Compass UI -- I don't think the current compass UI is
intuitive enough and could be extended to display more information in a nicer way. 3) Add visualizations and graphs based on Compass data
Looks like these ideas are already discussed above, unless I'm overlooking something?
All the best, Karsten
Hi Karsten,
On Wed, Mar 5, 2014 at 2:50 AM, Karsten Loesing karsten@torproject.org wrote:
Hi Christian, Sathya,
On 05/03/14 01:39, Sathyanarayanan Gunasekaran wrote:
Why? Do we want to deprecate onionoo?
Christian and I discussed this in private mail before moving this thread to tor-dev@. This bullet point should have said "Add support for new Onionoo features to Globe", or something like that. Examples are vote details (#9778), bridge user graphs (#10331), monthly stats per relay/bridge (#11041), list of transports (#11052), etc. These are all fine things to add to Globe, and maybe it's fine to mention them in the second half of a GSoC proposal. But they all depend on the Onionoo guy to write code first, so a Globe GSoC project shouldn't depend on them.
Ah, makes sense.
Is this going to be part of globe-node? I don't think there is a need to rewrite the Compass backend(the logic) -- it exposes a REST API which can just be consumed by Globe. Majority of the open Compass tickets are bugs in the frontend.
This sounds like a fine idea to integrate Compass into Globe very quickly and ask for user feedback early in the process. Agile!
Yep :)
But in the long term I suggest we move the grouping logic of Compass to Globe and decomission Compass. Reasons:
- that's one tool less to maintain, and Christian seems to be more of a
JavaScript person, not Python (IIRC);
I'm just worried that we've spent so much time looking into the tricky bits of logic/code that I don't want to throw that away. But if it makes sense to rewrite it then that's fine too :)
- Globe is going to be client-server soon in order to remove the need
for JavaScript in the browser, and then it's only a small step to do the same things that Compass does right now;
I see. I'm fine with that. I just don't want the client side node to download all the Onionoo data and parse it.
- Onionoo's search functionality is kinda hard to extend, so it would
be better if Globe did its own search based on locally cached documents.
Agreed.
Yes. Bonus points if you can leverage the data provided by Compass for some nice graphs like heat maps (based on probability) or something like that.
I think that's something for the metrics website, not for Globe. Onionoo and Globe are meant to provide status information on relays or bridges, whereas the metrics website provides aggregate data covering the entire Tor network. There are exceptions like the bubble graphs (https://metrics.torproject.org/bubbles.html) which use Onionoo's data, but in theory those should be implemented in the metrics website only. So, I'd say let's not mix scopes here, and let's not try to do too much in a single summer.
Makes sense. I just didn't think just adding the Compass to Globe would be enough for a GSoC project. From our conversation on IRC, it looks like there are other things that need to be incorporated into Globe which could be done as part of this GSoC project.
Do you have other ideas on how to improve globe?
- Add the Compass frontend to Globe
- Refactor the Compass UI -- I don't think the current compass UI is
intuitive enough and could be extended to display more information in a nicer way. 3) Add visualizations and graphs based on Compass data
Looks like these ideas are already discussed above, unless I'm overlooking something?
Yeah, you're right.
Thanks, --Sathya
-
Christian -
Karsten Loesing -
Sathyanarayanan Gunasekaran -
Sathyanarayanan Gunasekaran