[tor-relays] significant decrease in onionoo AS data coverage after switch to ipfire geoip db?

[Iain Learmonth]

Hi,

What we do get is consistency between the data displayed in Tor Metrics
and the data provided by Onionoo as now both are using the same upstream
database.

The database might not have the best coverage however, and that's
expected as IP Fire do not have the same resources as MaxMind have
available.

The lazy answer to this would be to point you at the database and tell
you to improve it:

https://git.ipfire.org/?p=location/location-database.git

A better answer would be to think about what we actually want from the
AS data.

It's nice to have the IP Fire database because the lookups can be
performed offline and there is an archived history in their git
repository of all the changes. For GeoIP, which is pretty hard as a
problem, this works great.

When we look at IP to AS resolution though, the clear way to do this is
with live BGP data because that's easily available to us. There's a
ticket for doing this that you filed and that I updated this morning:

https://gitlab.torproject.org/tpo/metrics/onionoo/-/issues/26585

The easier way to implement this would probably be to have Onionoo
perform live lookups through canid during the hourly update, but instead
of just showing the digest of the geoip file in use we'd now be looking
at each request querying a temporally dependent data source, so we
should add some new field to capture that information.

If we wanted to keep the IP Fire AS number as well as have the RIPEstat
one, then we do need to think a bit about what the documents look like
to support that, with metadata for each result.

Thanks,
Iain.

On 18/06/2021 23:34, nusenu wrote:
> Hi,
> 
> the following table shows how many relays in onionoo data have no
> as_name data.
> 
> Since a long time the underlying DB has not been updated and there is a
> plan to improve things
> https://gitlab.torproject.org/tpo/metrics/relay-search/-/issues/40004
> 
> Now I'm wondering if this improvement made things worse?
> 
> +------------+----------+
> | 2021-06-10 |      419 |
> | 2021-06-11 |      422 |
> | 2021-06-12 |      422 |
> | 2021-06-13 |      424 |
> | 2021-06-14 |      421 |
> | 2021-06-15 |      422 |
> | 2021-06-16 |      420 |
> | 2021-06-17 |      414 |
> | 2021-06-18 |     1748 |
> 
> 
> kind regards,
> nusenu
> 
> 
> 
>