significant decrease in onionoo AS data coverage after switch to ipfire geoip db?
Hi, the following table shows how many relays in onionoo data have no as_name data. Since a long time the underlying DB has not been updated and there is a plan to improve things https://gitlab.torproject.org/tpo/metrics/relay-search/-/issues/40004 Now I'm wondering if this improvement made things worse? +------------+----------+ | 2021-06-10 | 419 | | 2021-06-11 | 422 | | 2021-06-12 | 422 | | 2021-06-13 | 424 | | 2021-06-14 | 421 | | 2021-06-15 | 422 | | 2021-06-16 | 420 | | 2021-06-17 | 414 | | 2021-06-18 | 1748 | kind regards, nusenu -- https://nusenu.github.io
Hi, What we do get is consistency between the data displayed in Tor Metrics and the data provided by Onionoo as now both are using the same upstream database. The database might not have the best coverage however, and that's expected as IP Fire do not have the same resources as MaxMind have available. The lazy answer to this would be to point you at the database and tell you to improve it: https://git.ipfire.org/?p=location/location-database.git A better answer would be to think about what we actually want from the AS data. It's nice to have the IP Fire database because the lookups can be performed offline and there is an archived history in their git repository of all the changes. For GeoIP, which is pretty hard as a problem, this works great. When we look at IP to AS resolution though, the clear way to do this is with live BGP data because that's easily available to us. There's a ticket for doing this that you filed and that I updated this morning: https://gitlab.torproject.org/tpo/metrics/onionoo/-/issues/26585 The easier way to implement this would probably be to have Onionoo perform live lookups through canid during the hourly update, but instead of just showing the digest of the geoip file in use we'd now be looking at each request querying a temporally dependent data source, so we should add some new field to capture that information. If we wanted to keep the IP Fire AS number as well as have the RIPEstat one, then we do need to think a bit about what the documents look like to support that, with metadata for each result. Thanks, Iain. On 18/06/2021 23:34, nusenu wrote:
Hi,
the following table shows how many relays in onionoo data have no as_name data.
Since a long time the underlying DB has not been updated and there is a plan to improve things https://gitlab.torproject.org/tpo/metrics/relay-search/-/issues/40004
Now I'm wondering if this improvement made things worse?
+------------+----------+ | 2021-06-10 | 419 | | 2021-06-11 | 422 | | 2021-06-12 | 422 | | 2021-06-13 | 424 | | 2021-06-14 | 421 | | 2021-06-15 | 422 | | 2021-06-16 | 420 | | 2021-06-17 | 414 | | 2021-06-18 | 1748 |
kind regards, nusenu
Hi, I've been in contact with upstream (ipfire). Please update to an ipfire database version generated after 9 Jul 2021 23:10:48 UTC this should solve the major part of the issue, at least for ARIN space which was missing. I would appreciate if you update the db file regularly (ideally automatically) and allow users to easily learn when you update and what version you use. Thanks for improving onionoo's ASN data. kind regards, nusenu -- https://nusenu.github.io
participants (2)
-
Iain Learmonth -
nusenu