Hi all!
On Mon, Jun 9, 2014 at 10:22 AM, Karsten Loesing karsten@torproject.org wrote:
On 09/06/14 01:26, Damian Johnson wrote:
Oh, and another quick thought - you once mentioned that a descriptor search service would make ExoneraTor obsolete, and in looking it over I agree. The search functionality ExoneraTor provides is trivial. The only reason it requires such a huge database is because it's storing a copy of every descriptor ever made.
I suspect the actual right solution isn't to rewrite ExoneraTor at all, but rather develop a new service that can be queried for this descriptor data. That would make for a *much* more worthwhile project.
ExoneraTor? Nice to have. Descriptor archive service? Damn useful. :)
I agree, that was the idea behind Kostas' GSoC project last year. And I still think it's a good idea. It's just not trivial to get right.
Indeed, not trivial at all!
I'll use this space to mention the running metrics archive backend modulo ExoneraTor stuff / what could be sorta-relevant here.
fwiw, the onionoo-like backend is still running at an obscure address:port: http://ts.mkj.lt:5555/
TL;DR "what can I do with that" is: look at:
https://github.com/wfn/torsearch/blob/master/docs/use_cases_examples.md
In particular, regarding ExoneraTor-like queries (incl. arbitrary subnet / part-of-ip lookups):
https://github.com/wfn/torsearch/blob/master/docs/use_cases_examples.md#exon...
Not sure if it's worth discussing all the weaknesses of this archive backend in this thread, but the short relevant version is that the ExoneraTor-like functionality does mostly work, but I would need to look into it so see how reliable the results are ("is this relay ip address field really the one we should be using?", etc.)
But what's nice is that it is possible to do arbitrary queries on all consensuses since ~2008, with no date specified (if you don't want to.) (Which is to say, "it's possible", not necessarily "this is the right way to do the solution for the problems in this thread")
So e.g. this is the ip address where moria runs, and we want to see what relays have ever run on it:
http://ts.mkj.lt:5555/details?search=128.31.0.34
Take the fingerprint of the one that is currently running (moria1), and look up its last 500 statuses (in a kind of condensed/summary form): http://ts.mkj.lt:5555/statuses?lookup=9695DFC35FFEB861329B9F1AB04C46397020CE...
"from", "to" date ranges can be specified as e.g. 2009, 2009-02, 2009-02-10, 2009-02-10 02:00:00. limit/offset/parameters/etc. specified here: https://github.com/wfn/torsearch/blob/master/docs/onionoo_api.md
(Descriptors/digests aren't currently included (I think they used to), but they can be, etc.)
The point is probably mostly about "this is some evidence that it can be done." ("But there are nuances, things are imperfect, time is needed, etc.")
The question really is regarding the actual scope of this rewrite, I suppose.
I'd probably agree with Karsten that just doing a port of the ExoneraTor functionality as it currently is on exonerator.torproject.org would be the safe bet. See how that goes, venture into more exotic lands later on maybe, etc. (That doesn't mean that I wouldn't be excited to put the current backend to good use, and/or use the knowledge I gained to help you folks in some way!)
Regarding your comment about storing a copy of every descriptor ever made, I believe that users trust ExoneraTor's results more if they see the actual descriptors that lead to results. Of course, I'm saying that without knowing what ExoneraTor users actually want. But let's not drop descriptor copies from the database easily.
And, heh, when you say that the search functionality ExoneraTor provides is trivial, a little part of me is dying. It's the part that spent a few weeks on getting the search functionality fast enough for production. That was not at all trivial. The oraddress24, oraddress48, and exitaddress24 fields as well as the indexes are the result of me running lots and lots of sample queries and wondering about Postgres' EXPLAIN ANALYZE results. Just saying that it's not going to be trivial to generalize the search functionality towards other fields than IP addresses and dates.
Hear hear, I can only imagine! These things and exonerator stuff is not easy to be done in a way that would provide **consistently** good/great performance.
I spent some days of the last summer also looking at EXPLAIN ANALYZE results (it was a great feeling to start to understand what they mean and how I can make them better), but eventually things start making sense. (And when they do, I also get that same feeling that NoSQL stuff doesn't magically solve things.)
If others want to follow, here's the SQL code I'm talking about:
https://gitweb.torproject.org/exonerator.git/blob/HEAD:/db/exonerator.sql
So, I'm happy to talk about writing a searchable descriptor archive. It could _start_ with ExoneraTor's functionality (minus the target address and port thing discussed in that other email), and then we could consider adding more searches.
fwiw, imho this sounds like a sane plan to me. (Of course it could also be possible to work on the onionoo-like archive backend (or fork it, or smash it into parts and steal some of them, etc., but I can see why this might yield unclear deliverables, etc.) (So a short document of "what is wanted" would help, yeah.)
Pretty sure that Kostas is reading this (in fact, I just cc'ed him), so let me make one remark about optimizing Postgres defaults: I wrote quite a few database queries in the past, and some of them perform horribly (relay search) whereas others perform really well (ExoneraTor). I believe that the majority of performance gains can be achieved by designing good tables, indexes, and queries. Only as a last resort we should consider optimizing the Postgres defaults.
Ha, at this point I probably have a sort of "premature optimizer" label in your mind, Karsten. :) (And I kinda deserved it by at one point focusing on very-low-level postgres caching mechanisms last summer, etc etc.)
I've actually come to really appreciate good schema and query design[1] and the wonders that they do. That being said, I'd actually be curious to know how large the indexes of relay-search and current exonerator are.[2] I (still) bet increasing postgres' shared_buffers and effective_cache_size (totally normal practice!) might help! (Oh, is this one of those vim-vs-emacs things? If it is, sorry.)
But the point is that (to invoke a cliche) there is no free lunch, and (2) postgresql can really do wonders and scale well when used right.
You realize that a searchable descriptor archives focuses much more on database optimization than the ExoneraTor rewrite from Java to Python (which would leave the database untouched)?
"leaving database untouched" probably implies (very) significantly less work, so it would be a nice/clear starting point. (caveat, i may be lacking context, etc.)
[1]: also, fun things like "sometimes indexes won't be used because a sequential read will be faster, because if parts of indexes to be used are in various parts across the disk (not all of them are in memory), random seek + read a bit into memory + repeat is slower than 'just read a lot of continuous data into memory'", etc etc.)
[2]: if you're feeling adventuruous, you can run this on each of postgres databases, to see how large the indexes (among all other things) are, and which parts of them are loaded into memory https://github.com/wfn/torsearch/blob/master/misc/buffercache.sql
--
Kostas.
0x0e5dce45 @ pgp.mit.edu