[metrics-team] Bringing back Tor Weather - volunteering developer

[Karsten Loesing]

Hi!

Thanks for working on bringing back Tor Weather! Much appreciated!

On 2018-07-20 21:30, nusenu wrote:
>>> once an hour should be fine 
>>> (even though onionoo's update process
>>> is not done in a single transaction and you might get different data
>>> for the same relays_published timestamp).
>>
>> I see. Any reason why it onionoo doesn't switch the data in a single
>> transaction? I would assume it relates to the way it reads the data,
>> however in 2 hours I assume it can read the consensus data for the
>> whole network.
> 
> everytime you fetch 
> https://onionoo.torproject.org/details?type=relay&running=true
> you should get all relays that were running at consensus time
> matching the relays_published timestamp and we should do so at least once
> an hour, karsten might tell us what is a good time past the hour

Okay, I'll try to give some more insights into these 2 hours that are
mentioned several times on this thread, by giving two examples:

Imagine that a relay first drops out of the consensus with a valid-after
time of 02:00 on any given day. Onionoo waits until some time between
02:15 and 02:20 to start its hourly update run. The reason for waiting
15 to 20 minutes is that it needs the most recent consensus, but also
the newly sanitized bridge descriptors as input. Onionoo then runs until
about 02:45 to 02:50, and Weather can learn about changes to the network
shortly after. That's a delay of a bit under 1 hour.

Now, where do the 2 hours come from? Imagine that a relay goes offline
at 00:51 on that day. The directory authorities had just included it as
running in their votes that they exchange at 00:50 and altogether will
include it in their consensus with valid-after time 01:00. Now it's
going to take until 01:50 before they vote again and not list that relay
anymore in the 02:00 consensus. Altogether, that's a delay of roughly 2
hours between the relay going down and Weather notifying the operator.

All in all, the fact that directory authorities vote once per hour and
take 10 minutes for that vote process contributes between 10 minutes and
70 minutes to the delay, and Onionoo adds another 50 minutes.

There's not much we can do in Onionoo to further reduce this delay. We
could maybe save another 10 or 20 minutes by parallelizing things more.
But that's easier said than done, and at the moment it's not a priority
for the metrics team.

What you could do in Weather is query Onionoo every few minutes for an
hourly update. Just be sure to include a "If-Modified-Since" header:

https://metrics.torproject.org/onionoo.html#caching

Oh, and also be sure to use the fields parameter to reduce the response
size to whichever fields you actually need:

https://metrics.torproject.org/onionoo.html#parameters_fields

All the best,
Karsten

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 528 bytes
Desc: OpenPGP digital signature
URL: <http://lists.torproject.org/pipermail/metrics-team/attachments/20180723/24ca1286/attachment.sig>