[tor-relays] MetricsPort: tor_relay_connections_total type confusion

[nat at danwin1210.de]

Hello David,

again, thanks for your work on adding more metrics to tor's MetricsPort!
Many relay operators will love this and documentation will be useful [1].

I reported
https://gitlab.torproject.org/tpo/core/tor/-/issues/40699
which got closed yesterday, but there was likely a misunderstanding and
the changes did not solve the underlying issue.

The core issue is: The metric called
tor_relay_connections(_total) [2][3]
contains a mix of different types (gauge, counter).

When mixing types in a single metric, the TYPE definition will always be
wrong for one or the other value.
For example grafana will show this if you use a counter metric without
rate():
"Selected metric is a counter. Consider calculating rate of counter by
adding rate()."

It is a best practice to avoid mixing different types in a single metric.
>From the prometheus best practices [4]:
"As a rule of thumb, either the sum() or the avg() over all dimensions of
a given metric should be meaningful (though not necessarily useful). If it
is not meaningful, split the data up into multiple metrics. For example,
having the capacity of various queues in one metric is good, while mixing
the capacity of a queue with the current number of elements in the queue
is not."

An idea to address the underlying issue:
One connection metric for counter and one for gauge:

- tor_relay_connections_total for counters, like the current label
state="created"
- tor_relay_connections for gauge metrics, like the current label
state="opened". "rejected" also appears to be a gauge metric.

Another nice feature of these metrics would be to have a label for what
type of system is connecting (src="relay", src="non-relay") - more on that
in yesterday's email.
A tool by toralf [4] also shows these and uses the source IP but tor
itself does not need to look at the source IP to determine the type,
something discussed in last week's relay operator meetup.

best regards,
nat

[1] https://gitlab.torproject.org/tpo/web/support/-/issues/312
[2]
https://gitlab.torproject.org/tpo/core/tor/-/commit/06a26f18727d3831339c138ccec07ea2f7935014
[3]
https://gitlab.torproject.org/tpo/core/tor/-/commit/6d40e980fb149549bbef5d9e80dbdf886d87d207
[4] https://prometheus.io/docs/practices/naming/