Hello David,
again, thanks for your work on adding more metrics to tor's MetricsPort! Many relay operators will love this and documentation will be useful [1].
I reported https://gitlab.torproject.org/tpo/core/tor/-/issues/40699 which got closed yesterday, but there was likely a misunderstanding and the changes did not solve the underlying issue.
The core issue is: The metric called tor_relay_connections(_total) [2][3] contains a mix of different types (gauge, counter).
When mixing types in a single metric, the TYPE definition will always be wrong for one or the other value. For example grafana will show this if you use a counter metric without rate(): "Selected metric is a counter. Consider calculating rate of counter by adding rate()."
It is a best practice to avoid mixing different types in a single metric.
From the prometheus best practices [4]:
"As a rule of thumb, either the sum() or the avg() over all dimensions of a given metric should be meaningful (though not necessarily useful). If it is not meaningful, split the data up into multiple metrics. For example, having the capacity of various queues in one metric is good, while mixing the capacity of a queue with the current number of elements in the queue is not."
An idea to address the underlying issue: One connection metric for counter and one for gauge:
- tor_relay_connections_total for counters, like the current label state="created" - tor_relay_connections for gauge metrics, like the current label state="opened". "rejected" also appears to be a gauge metric.
Another nice feature of these metrics would be to have a label for what type of system is connecting (src="relay", src="non-relay") - more on that in yesterday's email. A tool by toralf [4] also shows these and uses the source IP but tor itself does not need to look at the source IP to determine the type, something discussed in last week's relay operator meetup.
best regards, nat
[1] https://gitlab.torproject.org/tpo/web/support/-/issues/312 [2] https://gitlab.torproject.org/tpo/core/tor/-/commit/06a26f18727d3831339c138c... [3] https://gitlab.torproject.org/tpo/core/tor/-/commit/6d40e980fb149549bbef5d9e... [4] https://prometheus.io/docs/practices/naming/