<div dir="ltr">On Wed, Oct 23, 2013 at 2:32 PM, Karsten Loesing <span dir="ltr"><<a href="mailto:karsten@torproject.org" target="_blank">karsten@torproject.org</a>></span> wrote:<br><div class="gmail_extra"><div class="gmail_quote">
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">On 10/11/13 4:05 PM, Kostas Jakeliunas wrote:<br>
<br>
Oops! Sorry for the delay in responding! Responding now.<br>
<div><div class="h5"><br>
> On Fri, Oct 11, 2013 at 12:00 PM, Karsten Loesing <<a href="mailto:karsten@torproject.org">karsten@torproject.org</a>>wrote:<br>
><br>
>> Hi Kostas,<br>
>><br>
>> should we move this thread to tor-dev@?<br>
>><br>
><br>
> Hi Karsten!<br>
><br>
> sure.<br>
><br>
>>From our earlier conversation about your GSoC project:<br>
>>> In particular, we should discuss how to integrate your project into<br>
>>> Onionoo. I could imagine that we:<br>
>>><br>
>>> - create a database on the Onionoo machine;<br>
>>> - run your database importer cronjob right after the current Onionoo<br>
>>> cronjob;<br>
>>> - make your code produce statuses documents and store them on disk,<br>
>>> similar to details/weights/bandwidth documents;<br>
>>> - let the ResourceServlet use your database to return the<br>
>>> fingerprints to return documents for; and<br>
>>> - extend the ResourceServlet to support the new statuses documents.<br>
>>><br>
>>> Maybe I'm overlooking something and you have a better plan? In any<br>
>>> case, we should take the path that implies writing as little code as<br>
>>> possible to integrate your code in Onionoo.<br>
>><br>
>> Let me know what you think!<br>
>><br>
><br>
> Sounds good. Responding to particular points:<br>
><br>
>> - create a database on the Onionoo machine;<br>
>> - run your database importer cronjob right after the current Onionoo<br>
>> cronjob;<br>
><br>
> These should be no problem and make perfect sense. It's always best to use<br>
> raw SQL table creation routines to make sure the database looks exactly<br>
> like the one on the dev machine I guess (cf. using SQLAlchemy abstractions<br>
> to do that (I did that before)).<br>
><br>
> Current SQL script to do that is at [1]. I'll look over it. For example,<br>
> I'd (still) like to generate some plots showing the chances of two<br>
> fingerprints having the same substring (this is for the intermediate<br>
> fingerprint table.) (One axis would be substring length, another would be<br>
> the possibility in (portions of) %.) As of now, we still use<br>
> substr(fingerprint, 0, 12), and it is reflected in the schema.<br>
><br>
> Overall though, no particular snags here.<br>
<br>
</div></div>I don't follow. But before we get into details here, I must admit that<br>
I was too optimistic about running your code on the current Onionoo<br>
machine. I ran a few benchmark tests on it last week to compare it to<br>
new hardware, and those tests almost made it fall over. We should not<br>
even think about adding new load to the current machine.<br>
<br>
New plan: can you run an Onionoo instance with your changes on a<br>
different machine? (If you need anything from me, like a tarball of the<br>
status/ and out/ directories, I'm happy to provide them to you.) I<br>
think we should run this instance for a while to see how reliable it is.<br>
And once we're confident enough, we'll likely have new hardware for the<br>
new Onionoo, so that we can move it there.<br></blockquote><div><br></div><div>This sounds like a very good idea. Ok, I can try and do this. Sorry for delaying my response as well, I'll try and follow up with what I need (if anything).</div>
<div><br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div class="im">
>> - make your code produce statuses documents and store them on disk,<br>
>> similar to details/weights/bandwidth documents;<br>
><br>
> Right, so if we are planning to support all V3 network statuses for all<br>
> fingerprints, how are we to store all the status documents? The idea is to<br>
> preprocess and serve static JSON documents, correct (as in the current<br>
> Onionoo)? (cf. the idea of simply caching documents: if we serve a<br>
> particular status document, it gets cached, and depending on the query<br>
> parameters (date range restriction, e.g.) it may be set not to expire at<br>
> all.)<br>
><br>
> Or should we try and actually store all the statuses (the condensed status<br>
> document version [2], of course)?<br>
<br>
</div>Let's do it as the current Onionoo does it. This code does not exist,<br>
right?<br></blockquote><div><br></div><div>I've done some small testing on a local system, it seems the Onionoo way is plausible, since the generation of all the old(er) status etc. documents needs to happen only once (obviously, but now I understand this means the number of resulting status documents and their size is not such a big deal after all.) I don't have good code for it as of yet.</div>
<div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div class="im">
>> - let the ResourceServlet use your database to return the<br>
>> fingerprints to return documents for; and<br>
>> - extend the ResourceServlet to support the new statuses documents.<br>
><br>
> Sounds good. I assume you are very busy with other things as well, so<br>
> ideally maybe you had in mind that I could try and do the Java part? :)<br>
> Though, since you are much more familiar with (your own) code, you could<br>
> probably do it faster than me. Not sure.<br>
> Any particular technical issues/nuances here (re: ResourceServlet)?<br>
<br>
</div>Can you give it a try? Happy to help with specific questions about<br>
ResourceServlet, and I'll try hard to reply faster this time. Again,<br>
sorry for the delay!<br></blockquote><div><br></div><div>Okay! I've been tinkering a bit, actually. Will see if I can produce something decent and reliable.</div><div> </div><div>Best wishes</div><div>Kostas.</div><div>
<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div class="HOEnZb"><div class="h5">
><br>
> [1]: <a href="https://github.com/wfn/torsearch/blob/master/db/db_create.sql" target="_blank">https://github.com/wfn/torsearch/blob/master/db/db_create.sql</a><br>
> [2]:<br>
> <a href="https://github.com/wfn/torsearch/blob/master/docs/onionoo_api.md#network-status-entry-documents" target="_blank">https://github.com/wfn/torsearch/blob/master/docs/onionoo_api.md#network-status-entry-documents</a><br>
> (e.g.<br>
> <a href="http://ts.mkj.lt:5555/statuses?lookup=9695DFC35FFEB861329B9F1AB04C46397020CE31&condensed=true" target="_blank">http://ts.mkj.lt:5555/statuses?lookup=9695DFC35FFEB861329B9F1AB04C46397020CE31&condensed=true</a><br>
> )<br>
><br></div></div></blockquote></div></div></div>