[tor-relays] tor prometheus metrics -> expected label value, got "INVALID"

[Fran]

Hej,

taking another look and comparing it to onion_services exporter there is 
a slight difference in the metrics output.

prometheus-onion-service-exporter:

onion_service_up{address="foobar:587",name="mail_v3_587",type="tcp"} 1

vs output from tor:

tor_hs_app_write_bytes_total{onion=foobar,port=80} 298891

The working onion-service-exporter double quotes the labels but tor doesn't.

Debug messages from prometheus are also not very helpful:

[...] msg="Append failed" err="expected label value, got \"INVALID\""

But after searching around for a long time I found that promtool can 
help with debugging scraping erros.

Promtool complains about the missing double quotes

$ GET http://<IP>:9035/metrics | promtool check metrics
error while linting: text format parsing error in line 3: expected '"' 
at start of label value, found 'f'

Saving the exporter output to a file and quoting the onion and port 
label values:

$ cat quoted-output.txt | promtool check metrics
tor_hs_intro_established_count non-histogram and non-summary metrics 
should not have "_count" suffix
tor_hs_rdv_established_count non-histogram and non-summary metrics 
should not have "_count" suffix

After removing the criticized "_count" in the file promtool is happy.

So, how to fix this to not make my weekend project fail? Relabel label 
values!

First try (for testing):

     relabel_configs:
     - action: labeldrop
       regex: "port"
     - action: labeldrop
       regex: "onion"

not working, still 'expected label value, got "INVALID"'. What about:

     relabel_configs:
     - target_label: onion
       replacement: foo
     - target_label: port
       replacement: bar

now I got not only the labels 'instance="<IP>:9035"job="tor_metrics"' 
but also 'onion="foo" port="bar"' but still the 'expected label value, 
got "INVALID"' error.

Out of ideas for the moment.

Best fran


On 1/23/22 01:20, Fran wrote:
> Hej,
> 
> I just found out about the integrated prometheus exporter in tor - yay, 
> thanks for that!
> 
> I can scrape it with curl:
> 
> $ curl http://<ip>:9035/metrics
> # HELP tor_hs_rdv_num_total Total number of rendezvous circuit created
> # TYPE tor_hs_rdv_num_total counter
> tor_hs_rdv_num_total{onion=<onionv3adr>} 4
> # HELP tor_hs_app_write_bytes_total Total number of bytes written to the 
> application
> # TYPE tor_hs_app_write_bytes_total counter
> tor_hs_app_write_bytes_total{onion=<onionv3adr>,port=80} 15211
> # HELP tor_hs_intro_num_total Total number of introduction received
> # TYPE tor_hs_intro_num_total counter
> tor_hs_intro_num_total{onion=<onionv3adr>} 4
> # HELP tor_hs_rdv_established_count Total number of established 
> rendezvous circuit
> # TYPE tor_hs_rdv_established_count gauge
> tor_hs_rdv_established_count{onion=<onionv3adr>} 1
> # HELP tor_hs_app_read_bytes_total Total number of bytes read from the 
> application
> # TYPE tor_hs_app_read_bytes_total counter
> tor_hs_app_read_bytes_total{onion=<onionv3adr>,port=80} 319667
> # HELP tor_hs_intro_established_count Total number of established 
> introduction circuit
> # TYPE tor_hs_intro_established_count gauge
> tor_hs_intro_established_count{onion=<onionv3adr>} 18
> 
> Which looks like proper data for me. Unfortunately prometheus thinks 
> otherwise. In the prometheus webGUI under "/targets" it complains:
> 
>  > expected label value, got "INVALID"
> 
> The Prometheus config part is:
> 
>    - job_name: tor_metrics
>      metrics_path: /metrics
>      static_configs:
>      - targets:
>        - <server1>:9035
>        - <server2>:9035
> 
> Servers are running Debian 11, Prometheus 2.32.1 and tor 
> 0.4.6.9-1~d11.bullseye+1
> 
> 
> I searched the relays mailinglist and tried to find something in the 
> internet, unfortunately not very successfully.
> 
> Any ideas?
> 
> Thanks a lot!
> 
> fran