[tor-talk] Shutting down the relay-search service by the end of the year

[Karsten Loesing]

On 12/8/13 7:37 PM, Roger Dingledine wrote:
> On Sun, Dec 01, 2013 at 10:44:48AM +0100, Karsten Loesing wrote:
>> I'm thinking about shutting down a service that is currently running as
>> part of the metrics website:
>>
>>   https://metrics.torproject.org/relay-search.html
> [snip]
>> The main feature that these two alternatives are missing is that they
>> don't tell you about relays that have been offline for *more* than a week.
> 
> As usual, I think I'm the one using more of the metrics services than
> others.
> 
> For the most recent case where I found it useful, see
> https://lists.torproject.org/pipermail/tor-relays/2013-December/003455.html

Hi Roger,

so, the information you learned from relay-search was that the relay
"was up for that one consensus period and then down after that."

You could have learned this specific piece of information from Onionoo,
too, by looking at the "first_seen" and "last_seen" fields, e.g.,

https://onionoo.torproject.org/details?search=110.146.133.98

I'm pasting relevant parts of the output here, because these relays
aren't running anymore and will disappear from Onionoo's output after
being offline for a week (which already happened for the relay showing
up briefly on December 1):

{"relays_published":"2013-12-09 08:00:00",
"relays":[
{"nickname":"alphadet",
"fingerprint":"5B99C1D84FC05F5D069C392E77687F5C9C40D1B5",
"or_addresses":["110.146.133.98:9001"],
"last_seen":"2013-12-03 04:00:00",
"first_seen":"2013-12-03 04:00:00",
"last_changed_address_or_port":"2013-12-03 04:00:00",
"running":false,
"flags":["Running","Valid"],
...
},
{"nickname":"alphadet",
"fingerprint":"EE16D7A4FBCF6494FEE75C856D76782295CB9DC4",
"or_addresses":["110.146.133.98:9001"],
"last_seen":"2013-12-02 16:00:00",
"first_seen":"2013-12-02 16:00:00",
"last_changed_address_or_port":"2013-12-02 16:00:00",
"running":false,
"flags":["Running","Valid"],
...
}
],
"bridges_published":"2013-12-09 07:37:04",
"bridges":[
]}


However, I admit that the above would have worked in this specific use
case only.  Or maybe there are similar use cases where Atlas or Globe or
Onionoo have the information you're looking for.  But in general,
Onionoo provides less information than the original consensus entries
and relay descriptors that relay-search returns.

Now, how can we make sure that you (and others) have the tools you need
to debug the network?  I can see three possible ways, starting from the
one I like most:

1) We teach you how to use stem to debug the network using downloaded
descriptors from metrics.  I think Damian would be happy to help you
with this.  And we shut down relay-search by the end of the year.

2) Whenever Onionoo or Atlas/Globe are missing information that you need
to debug the network, you tell us about it, and we try to extend the
Onionoo protocol and the Atlas/Globe user interfaces.  We'll have to be
careful what to add to Onionoo in order not to run into new performance
problems, though.  And we shut down relay-search by the end of the year.

3) We keep relay-search by making it a service of its own, separate from
the metrics website.  That won't solve the problem that the database is
growing and growing, but it will allow me to continue refactoring the
aggregation code for the metrics website and eventually add new
statistics that you care about.  The main downside is that we'll have
yet one more service to maintain.

What do you think?

>> The major argument against keeping the relay-search service is that it's
>> one of the few remaining things that require us to keep a full database
>> of relays on our already overloaded metrics machine.
> 
> How sad. I guess if it has to go, it should go. But I do use it. :)

Let's try to find a solution that doesn't lead to sadness.

>> My current thinking is that the relay-search service is only used,
>> because it's linked from various websites including Tor Weather and the
>> Relay Configuration Instructions on Debian/Ubuntu.  Those links could be
>> changed to Atlas or Globe easily.
> 
> Changing the links sounds like a fine plan to me.

Okay, will do.

All the best,
Karsten