Hej, taking another look and comparing it to onion_services exporter there is a slight difference in the metrics output. prometheus-onion-service-exporter: onion_service_up{address="foobar:587",name="mail_v3_587",type="tcp"} 1 vs output from tor: tor_hs_app_write_bytes_total{onion=foobar,port=80} 298891 The working onion-service-exporter double quotes the labels but tor doesn't. Debug messages from prometheus are also not very helpful: [...] msg="Append failed" err="expected label value, got \"INVALID\"" But after searching around for a long time I found that promtool can help with debugging scraping erros. Promtool complains about the missing double quotes $ GET http://<IP>:9035/metrics | promtool check metrics error while linting: text format parsing error in line 3: expected '"' at start of label value, found 'f' Saving the exporter output to a file and quoting the onion and port label values: $ cat quoted-output.txt | promtool check metrics tor_hs_intro_established_count non-histogram and non-summary metrics should not have "_count" suffix tor_hs_rdv_established_count non-histogram and non-summary metrics should not have "_count" suffix After removing the criticized "_count" in the file promtool is happy. So, how to fix this to not make my weekend project fail? Relabel label values! First try (for testing): relabel_configs: - action: labeldrop regex: "port" - action: labeldrop regex: "onion" not working, still 'expected label value, got "INVALID"'. What about: relabel_configs: - target_label: onion replacement: foo - target_label: port replacement: bar now I got not only the labels 'instance="<IP>:9035"job="tor_metrics"' but also 'onion="foo" port="bar"' but still the 'expected label value, got "INVALID"' error. Out of ideas for the moment. Best fran On 1/23/22 01:20, Fran wrote:

Hej,

I just found out about the integrated prometheus exporter in tor - yay, thanks for that!

I can scrape it with curl:

$ curl http://<ip>:9035/metrics # HELP tor_hs_rdv_num_total Total number of rendezvous circuit created # TYPE tor_hs_rdv_num_total counter tor_hs_rdv_num_total{onion=<onionv3adr>} 4 # HELP tor_hs_app_write_bytes_total Total number of bytes written to the application # TYPE tor_hs_app_write_bytes_total counter tor_hs_app_write_bytes_total{onion=<onionv3adr>,port=80} 15211 # HELP tor_hs_intro_num_total Total number of introduction received # TYPE tor_hs_intro_num_total counter tor_hs_intro_num_total{onion=<onionv3adr>} 4 # HELP tor_hs_rdv_established_count Total number of established rendezvous circuit # TYPE tor_hs_rdv_established_count gauge tor_hs_rdv_established_count{onion=<onionv3adr>} 1 # HELP tor_hs_app_read_bytes_total Total number of bytes read from the application # TYPE tor_hs_app_read_bytes_total counter tor_hs_app_read_bytes_total{onion=<onionv3adr>,port=80} 319667 # HELP tor_hs_intro_established_count Total number of established introduction circuit # TYPE tor_hs_intro_established_count gauge tor_hs_intro_established_count{onion=<onionv3adr>} 18

Which looks like proper data for me. Unfortunately prometheus thinks otherwise. In the prometheus webGUI under "/targets" it complains:

...

expected label value, got "INVALID"

The Prometheus config part is:

  - job_name: tor_metrics     metrics_path: /metrics     static_configs:     - targets:       - <server1>:9035       - <server2>:9035

Servers are running Debian 11, Prometheus 2.32.1 and tor 0.4.6.9-1~d11.bullseye+1

I searched the relays mailinglist and tried to find something in the internet, unfortunately not very successfully.

Any ideas?

Thanks a lot!

fran

Hej David,

Thanks for reporting and investigating!! :)

Thank you for fixing and implementing in the first place! Really looking forward to see some nice graphs showing how many people use our services via onion. BTW: Implementation of nusenu feature request[1] for exposing more relay health metrics for prometheus would be awesome! :) Best, fran [1] https://gitlab.torproject.org/tpo/core/tor/-/issues/40194