<div dir="ltr"><div class="gmail_extra"><div class="gmail_quote">On Tue, Apr 8, 2014 at 12:59 PM, Karsten Loesing <span dir="ltr"><<a href="mailto:karsten@torproject.org" target="_blank">karsten@torproject.org</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div class="">On 05/04/14 17:46, Lukas Erlacher wrote:<br>
> Hello Nikita, Karsten,<br>
><br>
> On 04/05/2014 05:03 PM, Nikita Borisov wrote:<br>
>> On Sat, Apr 5, 2014 at 3:58 PM, Karsten Loesing<br>
>> <<a href="mailto:karsten@torproject.org">karsten@torproject.org</a>> wrote:<br>
>>> Installing packages using Python-specific package managers is<br>
>>> going to make our sysadmins sad, so we should have a very good<br>
>>> reason for wanting such a package. In general, we don't need<br>
>>> the latest and greatest package. Unless we do.<br>
>> What about virtualenv? Part of the premise behind it is that you<br>
>> can configure appropriate packages as a developer / operator<br>
>> without having to bother sysadmins and making them worried about<br>
>> system-wide effects.<br>
>><br>
>> - Nikita<br>
><br>
> I was going to mention virtualenv as well, but I have to admit that<br>
> I find it weird and scary, especially since I haven't found good<br>
> documentation for it. If there is somebody who is familiar with<br>
> virtualenv that would probably be the best solution.<br>
<br>
</div>I'm afraid I don't know enough about Python or virtualenv. So far, it<br>
was almost zero effort for our sysadmins to install a package from the<br>
repositories and keep that up-to-date. I'd like to stick with the<br>
apt-get approach and save the virtualenv approach for situations when<br>
we really need a package that is not contained in the repositories.<br>
<br>
Thanks for the suggestion, though!<br>
<div class=""><br>
> On 04/05/2014 04:58 PM, Karsten Loesing wrote:<br>
>> My hope with challenger is that it's written quickly, working<br>
>> quietly for a year, and then disappearing without anybody<br>
>> noticing. I'd rather not want to maintain yet another thing.<br>
>> So, maybe Weather is a better candidate for using onion-py than<br>
>> challenger.<br>
><br>
> Yes, I understand.<br>
><br>
>> Yeah, I think we'll want to define a maximum lifetime of cache<br>
>> entries, or the poor cache will explode pretty soon.<br>
><br>
> What usage patterns do we have to expect? Do we want to hit onionoo<br>
> to check if the cache is still valid for every request, or should<br>
> we do "hard caching" for several minutes? The best UX solution<br>
> would be to have a background task that keeps the cache current so<br>
> user requests can be delivered without hitting onionoo at all.<br>
<br>
</div>That's a fine question. I can see various caching approaches here.<br>
But I just realize that this is premature optimization. Let's first<br>
build the thing and download whatever we need and whenever we need it.<br>
And once we know what caching needs we have, let's build the cache.<br>
<div class=""><br>
> In other words, unless we do something intelligent with the cache,<br>
> the cache is not actually going to be very useful.<br>
<br>
</div>Valid point. :)<br>
<div class=""><br>
>> Great, your help would be much appreciated! Want to send me a<br>
>> pull request whenever you have something to merge?<br>
><br>
> Will do.<br>
<br>
</div>Great. Thanks!<br></blockquote><div><br></div><div>Hi Karsten and others,</div><div><br></div><div>I got to run the challenger script by chance[1], and spotted a small mistake that was preventing Lukas' onion.py downloader code from working. Ended up forking and creating a separate branch:</div>
<div><br></div><div><a href="https://github.com/wfn/challenger/commits/wfn_fix_luk3s_download">https://github.com/wfn/challenger/commits/wfn_fix_luk3s_download</a><br></div><div><br></div><div>Relevant commits:</div><div>
<br></div><div> - 38d88bcb1136f97881f81152d3d883c4e9480188[2] (enables downloader)</div><div> - 39c800643c040474402fc62d2a2db75c25889dfc[3] (this is the one with the small thingie-fix)</div><div><br></div><div>(It was a very small thing with the way the 'requests' module handles/provides json documents.)</div>
<div><br></div><div>I was doing this to be able to give Roger the 'combined-*.json' files for currently vulnerable (re: openssl) relays (he wanted to see which part of the combined weight fraction they comprise, etc.)</div>
<div><br></div><div>Fingerprints for those relays are here, fwiw: <a href="http://ravinesmp.com/volatile/challenger-stuff/vuln_fingerprints.txt">http://ravinesmp.com/volatile/challenger-stuff/vuln_fingerprints.txt</a> (the original link that Roger gave me was <a href="http://fpaste.org/92688/">http://fpaste.org/92688/</a> )</div>
<div>(count: 1024.)</div><div><br></div><div>If you download these fingerprints, you can just run `python challenge.py -f vuln_fingerprints.txt`</div><div><br></div><div>(for anyone using virtualenv, you might need to `pip install requests`, and then things should work. For anyone who's just cloned the thing, everything should probably work after simply installing the 'requests' python module, if it's not there. I see that 'python-requests' is available in the repos.)</div>
<div><br></div><div>I guess the code hasn't been tested for those amounts of fingerprints before. Good news: it works (where 'works' means 'i opened the resulting files and they contained all those fingerprints, and/or they contained lots of numbers.') Kinda-bad news: Onionoo doesn't seem to share the enthusiasm, and hiccups, and spits 502 Proxy Error some time after the lookups for the first document (combined bandwidth) are made.</div>
<div><br></div><div>My cheap quick hack was to insert time.sleep() here and there:</div><div><br></div><div> - 7425ef6fc00dedf3b2b7f2649e832fb4c93909ae[4]</div><div><br></div><div>(cheap hack is cheap, but it worked. Note: takes time to download everything. Didn't time it yet - sorry.)</div>
<div><br></div><div>For anyone interested, these are the resulting 'combined-*.json' files from all those fingerprints:</div><div><br></div><div> - <a href="http://ravinesmp.com/volatile/challenger-stuff/vuln1024-combined-bandwidth.json">http://ravinesmp.com/volatile/challenger-stuff/vuln1024-combined-bandwidth.json</a></div>
<div> - <a href="http://ravinesmp.com/volatile/challenger-stuff/vuln1024-combined-weights.json">http://ravinesmp.com/volatile/challenger-stuff/vuln1024-combined-weights.json</a></div><div> - <a href="http://ravinesmp.com/volatile/challenger-stuff/vuln1024-combined-clients.json">http://ravinesmp.com/volatile/challenger-stuff/vuln1024-combined-clients.json</a> [uh oh, this one's empty. Why is it empty? Didn't look into it.]</div>
<div> - <a href="http://ravinesmp.com/volatile/challenger-stuff/vuln1024-combined-uptime.json">http://ravinesmp.com/volatile/challenger-stuff/vuln1024-combined-uptime.json</a></div><div><br></div><div>I haven't much looked into them, at least not yet.</div>
<div><br></div><div>Roger wants to get some information about those vulnerable relays, and he thinks this challenger stuff can help with that. Those combined-* documents seem useful. I made a separate ML thread for this:</div>
<div><br></div><div><a href="https://lists.torproject.org/pipermail/tor-relays/2014-April/004262.html">https://lists.torproject.org/pipermail/tor-relays/2014-April/004262.html</a> [wow, i should switch to plaintext email probably..]<br>
</div><div><br></div><div>[1]: where 'by chance' means 'fell under arma's irc-Jedi spells somehow' / didn't plan to / i might be wrong about things or the things i did to the script, so beware</div>
<div>[2]: <a href="https://github.com/wfn/challenger/commit/38d88bcb1136f97881f81152d3d883c4e9480188">https://github.com/wfn/challenger/commit/38d88bcb1136f97881f81152d3d883c4e9480188</a></div><div>[3]: <a href="https://github.com/wfn/challenger/commit/39c800643c040474402fc62d2a2db75c25889dfc">https://github.com/wfn/challenger/commit/39c800643c040474402fc62d2a2db75c25889dfc</a></div>
<div class="gmail_quote">[4]: <a href="https://github.com/wfn/challenger/commit/7425ef6fc00dedf3b2b7f2649e832fb4c93909ae">https://github.com/wfn/challenger/commit/7425ef6fc00dedf3b2b7f2649e832fb4c93909ae</a></div><br class="">
take it easy</div><div class="gmail_quote"><br><div><div dir="ltr"><div>--</div><div><br></div><div>kostas / wfn</div><div><br></div>0x0e5dce45 @ <a href="http://pgp.mit.edu/" target="_blank">pgp.mit.edu</a></div></div></div>
</div></div>