[metrics-team] OnionStats

[Karsten Loesing]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hi Anathema,

On 04/06/16 00:34, Anathema wrote:
> To reply to myself (again):

Heh, let me enter this conversation again, so that you don't have to
reply to yourself the next time. :)

First, here's a bug report (from a few days ago when I made my first
attempt in responding, not sure if this bug still exists): When I
enter "nickname:default platform:linux" in the search field and select
"Last 6 months" for "First Seen", I get 1 result which was first seen
2016-03-06 08:00:00.  But when I switch to "Last 3 months" (2016-03-02
to 2016-06-02) I get 0 results.  Likewise, when I pick
"nickname:default" as search, last 6 months gets me 290 entries and
last 3 months 0.

Second, maybe you noticed that Onionoo has become faster lately.  The
reason was a problem with Apache's disk cache, causing all ~1M/h
requests to go through the Java process.  That's what you can see in
the last few weeks in the following graph.  We're now back to a few 10k/h.

https://people.torproject.org/~karsten/volatile/onionoo-total-requests-2016-06-05.png

I wonder how your stack compares to these new statistics:

Request statistics (2016-06-06 11:20:55, 3600 s):
Total processed requests: 15599
Most frequently requested resource: details (15278), summary (118),
bandwidth (99)
Most frequently requested parameter combinations: [lookup, fields]
(14474), [lookup] (722), [search] (245)
Matching relays per request: .500<2, .900<2, .990<4, .999<16384
Matching bridges per request: .500<1, .900<1, .990<1, .999<8192
Written characters per response: .500<256, .900<512, .990<16384,
.999<16777216
Milliseconds to handle request: .500<16, .900<16, .990<64, .999<256
Milliseconds to build response: .500<4, .900<16, .990<512, .999<16384

Third, when you say you want to throw the entire CollecTor archive
into Elasticsearch and have it indexed and searchable in a useful
manner, I think you're underestimating how much data is available
there.  Well, assuming similar computing resources, that is.  But if
you believe I'm wrong, here's a possible use case:
https://exonerator.torproject.org/.  Would you want to build a
prototype that throws all consensuses since 2007 into Elasticsearch
and lets users search by relay IP address or /24 and valid-after date
+/- 1 day?

Last, sorry for not responding to all details in this thread yet.  I
appreciate your effort in contributing to Tor metrics, I'm just not
good at replying to long threads, even if they're really interesting.
 The shorter the message and the easier to reply to, the earlier
you'll get a response, promised! :)

Thanks!

All the best,
Karsten

-----BEGIN PGP SIGNATURE-----
Comment: GPGTools - http://gpgtools.org

iQEcBAEBAgAGBQJXVWjQAAoJEC3ESO/4X7XBkiQH/36yEhMruL2qhGvPDGGVjjfy
vdZm9jdZKlFV76LDpXy/pNlYxFrJUjy6WQuff+o7YNrbgIbVq8qFO8I/McS6JgCm
t9vwBDkQLck8FJtDU+lmEFZu66h1NZ8ZiY53EsZpRiRvwRyW1j3P+yomiiz4XF5W
Z+6LRFGOQ1EouCCnLE2JOQ70vYTjIMMgT6faFRHXKJHE3Gf8eZyXhmaL31bDmWHY
gf2/IYU8yStblVhFdpwirG0cUv4U23w4g+lYE1uDsK+HdSe3PcgftJt1zhmxWdOD
SBzsARccIY24wU0gENd3hpAfmrCYy3g2SdmqC0IOnXqzTlToWEZ0h8zp6TSU+/s=
=a34D
-----END PGP SIGNATURE-----