[metrics-team] Bringing back Tor Weather - volunteering developer

[Eran Sandler]

On Fri, 2018-07-20 at 18:38 +0000, nusenu wrote:
> > 
> once an hour should be fine 
> (even though onionoo's update process
> is not done in a single transaction and you might get different data
> for the same relays_published timestamp).

I see. Any reason why it onionoo doesn't switch the data in a single
transaction? I would assume it relates to the way it reads the data,
however in 2 hours I assume it can read the consensus data for the
whole network.

Maybe a good way would be to have "active" and "in-work" data for
onionoo. Once the "in-work" data is considered complete it will
override the active one. That way, when querying the "active" data it
would be considered a true snapshot in a certain time of the network
and someone really wants to get a more up-to-date data they can query
the "in-work" data.

> 
> I'm not sure I understood you here. Tor network data is updated once
> an hour
> by dir auths, anything faster than that will not make much sense
> unless 
> the tor network consensus interval is also changed.
> onionoo is about relays and bridges only (not Tor clients)

I see. I think I mixed a few terms together. But I now (better)
understand how it work (will read more of the code later).

> > I think that actually Onionoo is the right place to parse the
> > contact
> > info and serve it as part of its meta data. 
> 
> I like the idea of having this in onionoo but I guess this will not
> happen anytime soon and
> Tor Weather shouldn't depend on it.

I agree it shouldn't depend on it but there is no real need in parsing
it for the weather system if we don't actually use the information for
anything (email validation is unrelated to the contact info in the
node).

> 
> if this service will NOT use its own email infrastructure I think
> we should have a discussion on which provider is used and what
> requirements it should meet. I have a few things in mind here.
> I'll add that to the ticket eventually.

I truly think that in today's email climate its almost impossible to
run your own SMTP server and have emails be deliverable in a reasonable
rate. If TorProject.org already have an SMTP email that is known and
valid it would be great at some point to use it, but for now we'll have
to use a 3rd party for the email alerts to be useful.

> 
> Eran: Since this is not an official tor service, which issue tracker
> do you want to use? Github repo? I wouldn't mind something other than
> trac since trac has
> no email integration.

I'll be doing the dev on github so I guess github issues should be more
than fine.

> 
> > All of these system assumes there is a domain for that. I'll get
> > one
> > (or use some domain I already have).
> 
> please ensure it supports DNSSEC

Sure. Might need a bit help in configuring DNSSEC (haven't done that
before but I'm sure can figure most of it out).

> 
> this is a great point! so we no longer (or at least less) have to
> worry about 
> who runs the service or the SMTP servers

Perhaps we should start with this approach. That way if someone really
wants Email alerts they can use IFTTT or whatever system they want to
listen to the feed.

It also make things rather easier in terms of running this on limited
hardware. Consider this design:

* Process to read data from onionoo and generate the "events" based on
that data, storing it in a database.

* Process to generate static feed files (RSS/Atom/JSON/Whatever). 

* Serve these files from a predefined directory structure (probably
have the node fingerprint as the filename or part of the path).

Since there are only ~8123  relays and ~439 bridges (so total of ~8562
nodes to generate feeds every 1 hour or so). it shouldn't take more
than a simple VPN to do all that.

> 
> > * An event for when the node first appeared
> 
> that sounds a bit like onionoo's first_seen timestamp

Yeah, that's probably what I'll use for that.

> 
> 
> as the operator of a onionoo dependend service you might like to
> subscribe
> to the metrics-alerts list which tells you when onionoo has problems.

Thanks, just did that.


Eran
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 659 bytes
Desc: This is a digitally signed message part
URL: <http://lists.torproject.org/pipermail/metrics-team/attachments/20180720/75e5d690/attachment.sig>