[tor-relays] Metrics for assessing EFF's Tor relay challenge?

Wed Apr 9 00:51:22 UTC 2014

On Tue, Apr 8, 2014 at 12:59 PM, Karsten Loesing <karsten at torproject.org>wrote:

> On 05/04/14 17:46, Lukas Erlacher wrote:
> > Hello Nikita, Karsten,
> >
> > On 04/05/2014 05:03 PM, Nikita Borisov wrote:
> >> On Sat, Apr 5, 2014 at 3:58 PM, Karsten Loesing
> >> <karsten at torproject.org> wrote:
> >>> Installing packages using Python-specific package managers is
> >>> going to make our sysadmins sad, so we should have a very good
> >>> reason for wanting such a package.  In general, we don't need
> >>> the latest and greatest package.  Unless we do.
> >> What about virtualenv? Part of the premise behind it is that you
> >> can configure appropriate packages as a developer / operator
> >> without having to bother sysadmins and making them worried about
> >> system-wide effects.
> >>
> >> - Nikita
> >
> > I was going to mention virtualenv as well, but I have to admit that
> > I find it weird and scary, especially since I haven't found good
> > documentation for it. If there is somebody who is familiar with
> > virtualenv that would probably be the best solution.
>
> I'm afraid I don't know enough about Python or virtualenv.  So far, it
> was almost zero effort for our sysadmins to install a package from the
> repositories and keep that up-to-date.  I'd like to stick with the
> apt-get approach and save the virtualenv approach for situations when
> we really need a package that is not contained in the repositories.
>
> Thanks for the suggestion, though!
>
> > On 04/05/2014 04:58 PM, Karsten Loesing wrote:
> >> My hope with challenger is that it's written quickly, working
> >> quietly for a year, and then disappearing without anybody
> >> noticing.  I'd rather not want to maintain yet another thing.
> >> So, maybe Weather is a better candidate for using onion-py than
> >> challenger.
> >
> > Yes, I understand.
> >
> >> Yeah, I think we'll want to define a maximum lifetime of cache
> >> entries, or the poor cache will explode pretty soon.
> >
> > What usage patterns do we have to expect? Do we want to hit onionoo
> > to check if the cache is still valid for every request, or should
> > we do "hard caching" for several minutes? The best UX solution
> > would be to have a background task that keeps the cache current so
> > user requests can be delivered without hitting onionoo at all.
>
> That's a fine question.  I can see various caching approaches here.
> But I just realize that this is premature optimization.  Let's first
> build the thing and download whatever we need and whenever we need it.
>  And once we know what caching needs we have, let's build the cache.
>
> > In other words, unless we do something intelligent with the cache,
> > the cache is not actually going to be very useful.
>
> Valid point. :)
>
> >> Great, your help would be much appreciated!  Want to send me a
> >> pull request whenever you have something to merge?
> >
> > Will do.
>
> Great.  Thanks!
>

Hi Karsten and others,

I got to run the challenger script by chance[1], and spotted a small
mistake that was preventing Lukas' onion.py downloader code from working.
Ended up forking and creating a separate branch:

https://github.com/wfn/challenger/commits/wfn_fix_luk3s_download

Relevant commits:

  - 38d88bcb1136f97881f81152d3d883c4e9480188[2] (enables downloader)
  - 39c800643c040474402fc62d2a2db75c25889dfc[3] (this is the one with the
small thingie-fix)

(It was a very small thing with the way the 'requests' module
handles/provides json documents.)

I was doing this to be able to give Roger the 'combined-*.json' files for
currently vulnerable (re: openssl) relays (he wanted to see which part of
the combined weight fraction they comprise, etc.)

Fingerprints for those relays are here, fwiw:
http://ravinesmp.com/volatile/challenger-stuff/vuln_fingerprints.txt (the
original link that Roger gave me was http://fpaste.org/92688/ )
(count: 1024.)

If you download these fingerprints, you can just run `python challenge.py
-f vuln_fingerprints.txt`

(for anyone using virtualenv, you might need to `pip install requests`, and
then things should work. For anyone who's just cloned the thing, everything
should probably work after simply installing the 'requests' python module,
if it's not there. I see that 'python-requests' is available in the repos.)

I guess the code hasn't been tested for those amounts of fingerprints
before. Good news: it works (where 'works' means 'i opened the resulting
files and they contained all those fingerprints, and/or they contained lots
of numbers.') Kinda-bad news: Onionoo doesn't seem to share the enthusiasm,
and hiccups, and spits 502 Proxy Error some time after the lookups for the
first document (combined bandwidth) are made.

My cheap quick hack was to insert time.sleep() here and there:

  - 7425ef6fc00dedf3b2b7f2649e832fb4c93909ae[4]

(cheap hack is cheap, but it worked. Note: takes time to download
everything. Didn't time it yet - sorry.)

For anyone interested, these are the resulting 'combined-*.json' files from
all those fingerprints:

  -
http://ravinesmp.com/volatile/challenger-stuff/vuln1024-combined-bandwidth.json
  -
http://ravinesmp.com/volatile/challenger-stuff/vuln1024-combined-weights.json
  -
http://ravinesmp.com/volatile/challenger-stuff/vuln1024-combined-clients.json[uh
oh, this one's empty. Why is it empty? Didn't look into it.]
  -
http://ravinesmp.com/volatile/challenger-stuff/vuln1024-combined-uptime.json

I haven't much looked into them, at least not yet.

Roger wants to get some information about those vulnerable relays, and he
thinks this challenger stuff can help with that. Those combined-* documents
seem useful. I made a separate ML thread for this:

https://lists.torproject.org/pipermail/tor-relays/2014-April/004262.html[wow,
i should switch to plaintext email probably..]

[1]: where 'by chance' means 'fell under arma's irc-Jedi spells somehow' /
didn't plan to / i might be wrong about things or the things i did to the
script, so beware
[2]:
https://github.com/wfn/challenger/commit/38d88bcb1136f97881f81152d3d883c4e9480188
[3]:
https://github.com/wfn/challenger/commit/39c800643c040474402fc62d2a2db75c25889dfc
[4]:
https://github.com/wfn/challenger/commit/7425ef6fc00dedf3b2b7f2649e832fb4c93909ae

take it easy

--

kostas / wfn

0x0e5dce45 @ pgp.mit.edu
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.torproject.org/pipermail/tor-relays/attachments/20140409/af86cd2b/attachment-0001.html>