[tor-bugs] #33027 [Internal Services/Tor Sysadmin Team]: Add blackbox exporter to prometheus ext

[Tor Bug Tracker & Wiki]

#33027: Add blackbox exporter to prometheus ext
-------------------------------------------------+---------------------
 Reporter:  hiro                                 |          Owner:  tpa
     Type:  defect                               |         Status:  new
 Priority:  Medium                               |      Milestone:
Component:  Internal Services/Tor Sysadmin Team  |        Version:
 Severity:  Normal                               |     Resolution:
 Keywords:                                       |  Actual Points:
Parent ID:  #30152                               |         Points:
 Reviewer:                                       |        Sponsor:
-------------------------------------------------+---------------------

Comment (by anarcat):

 the grafana dashboard is, as expected, kind of undecipherable. :) so much
 JSON! so we can review this once it's pulled in the GUI... no harm done
 their either way, so that's okay.

 you might have missed it, but `modules/profile/files/grafana/dashboards`
 is its own repository. it's one of those dark corners of the Puppet
 repository, which is currently a huge monorepo, but that does have some
 subrepos somehow. i'm looking into how to deal with that problem in
 #30770, in general. in particular, my remote for this repo right now is
 `git at gitlab.com:anarcat/grafana-dashboards.git` because i didn't want to
 go through the trouble of creating a project in gitolite and all that
 stuff. i could either give you access to that, or we could move it
 somewhere else, or i could just push the changes myself.

 in any case, it would be great if you could document the dashboard's in
 the README file next to it so we know where it comes from and so on.

 as for the puppet work, the blackbox exporter class looks sane. i don't
 see it included anywhere in the patch though, so i'm not sure how it
 should work.

 lower down, the `scrape_configs` array looks a bit strange to me.
 normally, the targets there look like `example.com:9091`, ie. with a port
 number at least. unless https://bridges.torproject.org/metrics actually
 gives out prometheus-compatible metrics (aka "OpenMetrics"). that doesn't
 seem to be the case right now (it's a 404).

 >  What is missing is the possibility for the service admins to add more
 targets without having to send patches to our puppet.

 That's going to be the tough part. Because the blackbox exporter is
 managed through puppet, in this configuration, we forcibly control the
 config file and local changes would be lost. So it's really one of three
 things:

  1. they send us the target and so we need to intervene when that needs to
 be changed
  2. we patch the module to allow us to *not* manage the blackbox config
 (yet another patch to the module, we can take it)
  3. we just *not* manage the blackbox exporter through puppet at all, and
 add it manually to the `scrape_configs` target.

 I would do option 2, eventually, and start with option 1. makes sense?

--
Ticket URL: <https://trac.torproject.org/projects/tor/ticket/33027#comment:2>
Tor Bug Tracker & Wiki <https://trac.torproject.org/>
The Tor Project: anonymity online