Looks pretty awesome!<span></span><div><br></div><div>You might enjoy this data feed for giving a parent organization to each relay:</div><div><br></div><div><a href="http://www.caida.org/data/as-organizations/">http://www.caida.org/data/as-organizations/</a><br><div><br>On Tuesday, 31 May 2016, Anathema <<a href="mailto:anathema@anche.no">anathema@anche.no</a>> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hi everyone,<br>
I've participated to the last metrics team meeting on IRC, and I've<br>
"presented" a new tool. Since I'll not be able to attend the next<br>
meeting, I'm going to present the full project here.<br>
<br>
A little bit of background:<br>
I was trying to find out some information related to Tor node stats like<br>
"how many nodes from country X have been activated in the last Y months"<br>
or "how many hosts with hostname X on platform Y there are" and so on.<br>
I found Atlas really helpful in some aspects but not so good in others,<br>
mainly I was not able to answer the above questions.<br>
Plus, there are some cons:<br>
- it's slow<br>
- it returns limited results<br>
- its query language doesn't allow complex and combined queries<br>
<br>
So I started writing a tool that fulfill my requirements. When it was<br>
almost finished, I thought: "well this is cool, maybe the Tor community<br>
can be interested in it". And here we are.<br>
<br>
What's all this about? OnionStats<br>
First, a note: the software described below can be integrated in Atlas.<br>
I created one from scratch because was easier for me, but if we don't<br>
want to use two services we can think about integrating mine into Atlas<br>
or Atlas into mine.<br>
<br>
So the name is OnionStats because, you know, Tor, Onionoo, onions. (it<br>
was TorStats but then Karsten suggested a better name :)<br>
<br>
The software stack is as follow:<br>
- Semantic-ui + jQuery as a frontend<br>
- Tornado as a backend<br>
- Monogdb as a DBMS<br>
- Elasticsearch as a search engine<br>
<br>
Here is the link to a live instance: <a href="http://138.201.90.124:8080" target="_blank">http://138.201.90.124:8080</a> (it's<br>
a cheap VPS so it may be slow due to a lack of resources - please be<br>
gentle and don't hammer it).<br>
<br>
How things work:<br>
Basically, there is a python script that runs in the background (cron)<br>
every 12h that fetches the nodes information using Onioon protocol and<br>
save the information into the mongodb schema.<br>
mongodb-collector automatically pushes the data into Elasticsearch.<br>
When you search through the web UI, the backend makes an Elasticsearch<br>
query and returns the data back to the web UI which displays the data.<br>
<br>
Easy, clean, fast.<br>
<br>
Pro:<br>
- it's fast. Really.<br>
- huge results cap: I've hardcoded a limit of 2000 results per query for<br>
testing but it can be easily increased in production with better hardware.<br>
- easy to audit: Atlas is made of AngularJS, which is great but for<br>
someone who doens't know anything about it, it's a big learning curve. I<br>
think that's a little bit overkill. My code is just plain jQuery and<br>
DataTables. That's all I needed.<br>
- complex queries: it can be possible to leverage almost all the<br>
Elasticsearch syntax features. More information in the "Syntax" section<br>
<br>
Cons:<br>
- updates data every 12h. In the IRC meeting someone told me that I can<br>
decrease the sleep time, so it may be possible to reduce to every 6h or<br>
maybe 1h?<br>
<br>
There are few HTML glitches so I apologize, I'm not a frontend coder and<br>
I'll try to fix them ASAP.<br>
<br>
I didn't push the code to my <github|bitbucket> repository but if you<br>
want to take a look at the code I'm more then welcome to publish it,<br>
just let me know which of the two do you prefer or if you prefer another<br>
way of sharing (like a link to a tarball on the server).<br>
<br>
Hope you like it and I'd be more than welcome to help integrate it into<br>
Atlas or integrate some Atlas' feature into OnionStats (and maybe, find<br>
a better name :)<br>
<br>
Let me know what you think.<br>
<br>
Thank you,<br>
Regards<br>
<br>
<br>
--<br>
Anathema<br>
<br>
+--------------------------------------------------------------------+<br>
|GPG/PGP KeyID: CFF94F0A available on <a href="http://pgpkeys.mit.edu:11371/" target="_blank">http://pgpkeys.mit.edu:11371/</a> |<br>
|Fingerprint: 80CE EC23 2D16 143F 6B25 6776 1960 F6B4 CFF9 4F0A |<br>
| |<br>
|<a href="https://keybase.io/davbarbato" target="_blank">https://keybase.io/davbarbato</a> |<br>
+--------------------------------------------------------------------+<br>
<br>
<br>
_______________________________________________<br>
metrics-team mailing list<br>
<a href="javascript:;" onclick="_e(event, 'cvml', 'metrics-team@lists.torproject.org')">metrics-team@lists.torproject.org</a><br>
<a href="https://lists.torproject.org/cgi-bin/mailman/listinfo/metrics-team" target="_blank">https://lists.torproject.org/cgi-bin/mailman/listinfo/metrics-team</a><br>
</blockquote></div></div>