[metrics-team] [network-team] Back from Montreal, anything I should do first?

[Karsten Loesing]

On 2017-10-17 17:29, Iain R. Learmonth wrote:
> Hi,

Hi!

> On 16/10/17 19:53, Karsten Loesing wrote:
>> I'd love to work more closely with you and other Python folks. And I
>> don't rule out that it might become a reality at some point in the
>> future. But I think it's safe to assume that this future won't be in the
>> next 12 months which is our current planning horizon.
> 
> This is also something that I've been considering.
> 
> I am primarily a Python developer when considering LOC/time. I did an
> undergraduate degree that was mostly Java and so it's not a barrier for
> me but I've had to read docs to catch up to Java 8.

Okay.

>>> That aside, at the dev meeting I floated the idea of pyoninoo because
>>> after years of work Nyx is *finally* nearing release so I'll be coming
>>> up for air soon to look for another project. Present ideas I have
>>> are...
> 
> pyonionoo was a Python version of the server that is no longer used? I'm
> not sure I get what the idea is.

Pyonionoo was a planned rewrite of Onionoo in Python that was never
completed. At this point there's probably not much use in looking at the
old, unfinished pyonionoo code. It's more the idea of rewriting Onionoo.

However, I already mentioned to atagar that we might want to merge other
services together with Onionoo or a rewritten Onionoo. Like, why not
answer ExoneraTor queries using the same data? And if the database
backend is powerful enough or can be easily distributed, why not use it
to aggregate statistics for the visualizations on Tor Metrics?

All in all, Pyonionoo is currently not a priority for the metrics team,
which is also the reason why we're not doing it ourselves. But if atagar
wants to use it as an experiment, I'd be curious to look at least at the
database parts to see how he got it efficient enough to import new data
every hour and the same time handle the number of responses that the
current Onionoo handles.

>>> 1. Look into Python/Rust bindings like rust-cpython. We actually have
>>> *three* descriptor parsers: metrics-lib (java), stem (python), and tor
>>> itself (C). That last is about to be reimplemented in Rust. Maybe I
>>> should write it? Research needed, but if the bindings are good enough
>>> maybe Stem and Tor can share a parser.
> 
> Assuming Rust can build libraries that have a C ABI, which I think was
> one of the points of it, you can call Rust functions with JNI.
> 
>> This is more for the network team, I guess.
>>
>>> 2. Pyonionoo. This might dovetail nicely with my day job depending on
>>> how things go at work as a way of learning Boto.
>>
>> As I said in Montreal, it would be best to focus on the database parts
>> of such a re-implementation. I'd be curious.
> 
> Still not sure what this is.
> 
>>> 3. Take over Atlas maintenance. For a long time Atlas has looked like
>>> a fun project to me and could be a neat way of improving my
>>> javascript-fu. This is why I cringed when you mentioned merging Atlas
>>> with the metrics site. Metrics is primarily a service for researchers
>>> whereas Atlas is a site for relay operators. I do see the overlap you
>>> described but personally I think these sites should remain separate.
> 
> The Metrics team provides a number of tools that are useful for relay
> operators, and by integrating these tools together, they become more
> discoverable.
> 
> https://metrics.torproject.org/operation.html
> 
> If you'd like to work on some Atlas tickets, I'm happy to do a triage
> pass and tag some of the tickets with "metrics-help" to indicate which
> it would be useful to work on first.
> 
>>> 5. Web dashboard for relay operators. My original plan was to follow
>>> up Nyx with a web version where relay operators could browse localhost
>>> to see bandwidth graphs, event logs, etc. But honestly with so many
>>> other interesting projects I'll probably defer this.
>>
>> You should check with relay operators how much they'd care about this
>> vs. other efforts. Maybe also check with irl who generally has a good
>> sense of relay operators' needs from receiving lots of questions and
>> other feedback from Atlas users.
> 
> Prometheus + Grafana + https://github.com/atalax/prometheus-tor_exporter
> 
> If anything has gone particularly wrong, you can always just start up
> arm to investigate. The above solution will also allow monitoring of
> multiple relays. You could look over the solution and check that it's
> doing everything right, and doing everything it should.
> 
> Thanks,
> Iain.

I guess the rest of your response is mostly for atagar. I don't have
much to add to that.

All the best,
Karsten

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 528 bytes
Desc: OpenPGP digital signature
URL: <http://lists.torproject.org/pipermail/metrics-team/attachments/20171017/88e078fd/attachment-0001.sig>