[metrics-team] TorDNSEL rewrite using Onionoo

[Nicolas Braud-Santoni]

Hi !

Sorry for the late answer, I was sorting out some heavy personal badness,
and also travel to FOSDEM (organising a Tor relays meetup there).

On Fri, Jan 19, 2018 at 05:17:51PM -0500, Arlo Breault wrote:
> > On Jan 8, 2018, at 11:06 PM, Philipp Winter <phw at torproject.org> wrote:
> > Yeah, as David mentioned, [making exitmap report exit addresses is] easy to
> > do.  We already have a module for that:
> > <https://github.com/NullHypothesis/exitmap/blob/master/src/modules/checktest.py>
> > 
> > The variable check_addr at line 68 contains the exit address, as seen by
> > check.torproject.org.

Thanks !

@Iain: Could that be something you work on (feeding that data into onionoo)?
       I feel there is arcane metrics team knowledge required.  :3



> It would be great if someone maintained a TorDNSEL-like
> service, since our current one if suffering from a
> dearth of attention, most likely because it's written
> in Haskell.

I'm aware, I actually tried to review and refresh the patchset that has been
rotting on trac for years, that tried to modernize it.  I never really got
anywhere due to the near-impossibility to get someone to look at my own Haskell
changes.  :(


> It lacks IPv6 support [0] and has at least one open bug
> resulting in false negatives [1].
> [...]
> It should be noted that there's a bit more to running
> a service that answers "Tor or not" than serving the
> lastest result of an exit scan.  A consensus is valid
> for a certain period of time (16hrs?) and a relay could
> come back online at any point therein.  Also, oddly,
> some relays rotate their exit ip frequently and others
> that seem to roundrobin a choice of IPs.  Anyways,
> the point is that check tended to err on the side of
> false positives for a better user experience.  There's
> a bug about the discrepancy [5].

Thanks for the pointers; I might try to write tests documenting
those behaviours, so we can more easily check that “new” TorDNSEL
doesn't accidentally differ from the old one.


> At the time that I was actively working on check [2],
> I started to rewrite it [3] rather than use exitmap
> because exitmap lacked support for continuous scanning [4]
> and was not very fast (less important because scanning
> each exit at every consensus period probably doesn't
> scale anyways).

Continuous scanning sounds like it would be a useful feature
anyhow;  I've already done a bit on work with exitmap (writing
two scan modules), so I will poke Philipp and ask whether whether
the branch mentionned in the ticket ever got anywhere, and have a
stab at it.


> Hopefully the above ramblings are useful.  Let me
> know if I can be of help,

They were super helpful, thanks  :)

From all that, I guess I should try adding continuous scanning to
exitmap, and have a first stab at an onionoo-backed TorDNSEL (even
while onionoo's data source is still “old” TorDNSEL).


Best,

  nicoo

> [0] https://trac.torproject.org/projects/tor/ticket/16947
> [1] https://trac.torproject.org/projects/tor/ticket/20114#comment:3
> [2] https://gitweb.torproject.org/check.git/
> [3] https://github.com/arlolra/exitaddr
> [4] https://github.com/NullHypothesis/exitmap/issues/7
> [5] https://trac.torproject.org/projects/tor/ticket/18968
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: not available
URL: <http://lists.torproject.org/pipermail/metrics-team/attachments/20180125/8735c218/attachment.sig>