Hi All,
This is my monthly status report for work complete during April 2021.
There was a ramp-up period as I got back into the flow of things, including reconfiguring GitLab access (with 2FA, SSH and GPG keys) and also my LDAP account. There are still some access issues to be resolved relating to uploading releases to dist.torproject.org but otherwise I am all set up now.
In order to catch up on what's been happening since I last worked on Tor Metrics I read through the network-health and metrics-team mailing lists since the start of the year.
Throughout this project until the end of July I am working on documenting Tor Metrics in preparation for the full time Metrics person to take over. All the documentation is being collected at:
https://gitlab.torproject.org/tpo/metrics/team/-/wikis/home
This is as opposed to documentation being scattered across various project wikis and repositories, as between the move from Trac wiki to GitLab and git.tpo migration to GitLab it's important to ensure that knowledge already collected isn't lost. This may not be the permanent home for the documentation but as the GitLab wiki is really just a collection of MarkDown files it's rather portable should moving it be desired.
The two main articles in this documentation I have worked on so far are the Java survival guide and the CollecTor development, deployment and operation documentation. Onionoo, Exonerator, and metrics-web are all still to follow. In collaboration with acute, OnionPerf's wiki has been merged into this central wiki.
In putting together a "pager playbook" for each service, I'm finding that monitoring for some features is lacking. I have spoken with anarcat about ways to improve this situation including writing simple exporters to get some monitoring data into the services Prometheus instance rather than sending emails via cron job.
In addition to documentation I have three main tasks:
1. Update the GeoIP database used by Onionoo
There are no Java bindings availabile for the IP Fire database that is now used in core Tor. As such we need a means of querying the file. A compact format for the file is used by core Tor, generated by a Rust tool. nickm has extended this tool to allow for AS numbers to be included in the generated file alongside country codes.
I have written a parser and lookup function for metrics-lib which is included in the 2.16.0 release. This will allow for Onionoo to consume the generated GeoIP files.
In order to ease access to the generated files, and not require Metrics developers to have a Rust toolchain on their machines, I have set up a scheduled CI job to generate the files: https://tpo.pages.torproject.net/metrics/geoip-data/ (it would be nice to add a date/time for the last update into this page, if anyone wants to submit a MR on the repo for that it could be an easy first contribution).
2. Updating the Metrics timeline in metrics-web
I have not yet started on this task.
3. bridgestrap/rdsys/Collector/Onionoo
This issue will require collaboration with the anti-censorship team. The first step here will be to get the state of bridgestrap into the Metrics pipeline, which is waiting on https://gitlab.torproject.org/tpo/anti-censorship/bridgestrap/-/issues/15.
Finally, I have been around in Matrix/IRC to answer Metrics questions and on occasion to provide a little end-user support in #tor. I've been attending the weekly network health meetings and will continue to do so. If you've got any questions about anything I'm working on, just send me an email or chat on Matrix/IRC.
Thanks, Iain.