[metrics-team] OnionStats - roadmap?

[Karsten Loesing]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hi Anathema,

moving this thread back to the team mailing list with your permission.

On 03/08/16 01:48, Anathema wrote:
> Sorry for flooding you with emails, but I've found few time to work
> on this project and I'd like to update you. So, I've release a
> 'beta' version right know.
> 
> I'm sending this to you first so you can validate the code and the
> service so we can discuss in the weekly meeting this Thursday.
> 
> Here you can find the code for review:
> https://github.com/davinerd/onionoo-ng Below the address which you
> can use to poke with it: http://138.201.90.124:8080/details

We already discussed this a bit last Thursday, but I said I'd take a
closer look and respond via email, which I'm doing know.  Note that I
didn't look at code or prototype, just at this email.

> My implementation differs slightly (in positive) from the standard
> one for the following reasons [0]:

I already mentioned the following on Thursday, but let me explain it
more here:

We need to be careful with protocol changes, in particular if changes
would have to be updated.  We distinguish backward-compatible changes
which current clients wouldn't care about from backward-incompatible
changes that would require clients to adapt first.  Let's try to avoid
the latter if we can.  If there's a convincing reason for doing them
anyway, let's talk about that.  But if not, let's just keep things as
they are.

> - results are returned for both bridges and relays when using
> 'limit' and 'offset'

The reason for the current behavior is that clients can easily
implement paging of results by setting limit to the number of results
they want to display and offset to the number of results on earlier
pages that they want to skip.  That won't work anymore with the
suggested change.  I'd consider this a bad backward-incompatible
change, unless there's a reason for making this change that I'm
overlooking.

> - 'order' parameter's value can be any field, so it's not limited
> to the 'consensum_weight'

Oh yes, this one is a good change, and it should be
backward-compatible.  We might have to specify how sorting works for
some fields.  For example, are or_addresses sorted alphanumerically?
Do we sort by first address in or_addresses or by the
(alphanumerically) smallest?  How do we handle missing values?

So, now that I'm listing these possible issues, would it be easier to
start with the fields that are easy and add more complicated fields later?

> - 'fields' parameter's value can be any field

I think we already resolved this at the meeting.  This is already the
case.

> The (negative) differences are: - 'lookup' is not implemented: I
> was not able to find a difference between 'lookup' and
> 'fingerprint': can you provide some real examples?

The difference between those two parameters is specified on the
protocol page:

https://onionoo.torproject.org/protocol.html

Here's where we added the fingerprint parameter:

https://gitweb.torproject.org/onionoo.git/commit/?id=8f63e74709cd05cd812e33f95ffe51b05d6d537c

If it turns out to be difficult to implement that parameter, let's
talk more.  Maybe we don't need it anymore.  Removing that would be a
backward-incompatible change, but let's see.

> - 'search' does not implement: "any 4 hex characters of a
> space-separated fingerprint" and "beginning of a base64-encoded
> fingerprint without trailing equal signs": I was not able to find
> any relevant case for those

Also mentioned at the meeting:

Sometimes people paste fingerprints from other sources into Atlas or
other Onionoo clients, and we should return matching relays to them.
We're only returning relays and bridges matching all search terms, so
we'll have to store all 4 hex character blocks of a fingerprint and
make them searchable.  Let me know if you have more questions about this.

> I'd also advice you to test all the features, in particularly the
> 'first_seen_days' and 'last_seen_days', since I'm not sure if it's
> completely consistent (bear in mind that the data used it's a
> snapshot taken on the 11 December 2015).
> 
> Some questions remain, but they are outlined in the other emails.
> 
> Please let me know what needs to be done, what to fix, and your
> thoughts about it.
> 
> Thanks! Have a nice day
> 
> 
> [0] it would take too much time to comply with the standard 3.1
> protocol version, since ElasticSearch offers great flexibility. So
> implementing the current protocol strictly would have meant to
> cripple the software's features and benefits.

Hope this helps to get your code closer to the current Onionoo
protocol.  Ideally, you'd be able to deploy an Atlas version that
points to your Onionoo server and offer that to users.  Let me know if
you need help with that.

Thanks!

All the best,
Karsten

-----BEGIN PGP SIGNATURE-----
Comment: GPGTools - http://gpgtools.org

iQEcBAEBAgAGBQJXqEZqAAoJEC3ESO/4X7XBGZYIAMsk3EYWT1JOkiorJW6ryC7i
/kB/BuVhpvbnnxgiPrb4gy3gFdxhdGE71NEDiz/F7EWTQy7c5OUASCejr45Sitij
PxT/RXQ7cffu49tR+lj3EM+6rq3PVmlliff3Db2XJ0VhGKhWXC5XOM9nAK30tIp+
wESUaz7ANsiVip6iEF08C5txw7AQ1YVXD6Y5B7X7sR/3NGk9olemH9fRr1OWlAqF
T4JxugPo9UoESO97iquaBmK+pOwdnzNyuEDcKBpMTAaFvYLmvwkZl7E/7yKXLBuz
F/usVpsJ5X4tBHDtoqHtsUTO9byqTh0iA4FrM4B4ksTcaFw9sZlgMYBxzyimrN4=
=Ssds
-----END PGP SIGNATURE-----