[tor-bugs] #33027 [Internal Services/Tor Sysadmin Team]: Add blackbox exporter to prometheus ext

[Tor Bug Tracker & Wiki]

#33027: Add blackbox exporter to prometheus ext
-------------------------------------------------+---------------------
 Reporter:  hiro                                 |          Owner:  tpa
     Type:  defect                               |         Status:  new
 Priority:  Medium                               |      Milestone:
Component:  Internal Services/Tor Sysadmin Team  |        Version:
 Severity:  Normal                               |     Resolution:
 Keywords:                                       |  Actual Points:
Parent ID:  #30152                               |         Points:
 Reviewer:                                       |        Sponsor:
-------------------------------------------------+---------------------

Comment (by hiro):

 Replying to [comment:2 anarcat]:
 > the grafana dashboard is, as expected, kind of undecipherable. :) so
 much JSON! so we can review this once it's pulled in the GUI... no harm
 done their either way, so that's okay.
 >



 Sure. I pulled it from the dashboard database. But I have should have
 added the link to the dashboard itself. I am going to do this.



 > you might have missed it, but `modules/profile/files/grafana/dashboards`
 is its own repository. it's one of those dark corners of the Puppet
 repository, which is currently a huge monorepo, but that does have some
 subrepos somehow. i'm looking into how to deal with that problem in
 #30770, in general. in particular, my remote for this repo right now is
 `git at gitlab.com:anarcat/grafana-dashboards.git` because i didn't want to
 go through the trouble of creating a project in gitolite and all that
 stuff. i could either give you access to that, or we could move it
 somewhere else, or i could just push the changes myself.



 I can create a git repository or we can just leave it like it is. Whatever
 works better for you in this case.



 >
 > in any case, it would be great if you could document the dashboard's in
 the README file next to it so we know where it comes from and so on.
 >
 > as for the puppet work, the blackbox exporter class looks sane. i don't
 see it included anywhere in the patch though, so i'm not sure how it
 should work.



 I haven't included it yet. This will come in the next iteration.



 >
 > lower down, the `scrape_configs` array looks a bit strange to me.
 normally, the targets there look like `example.com:9091`, ie. with a port
 number at least. unless https://bridges.torproject.org/metrics actually
 gives out prometheus-compatible metrics (aka "OpenMetrics"). that doesn't
 seem to be the case right now (it's a 404).
 >
 > >  What is missing is the possibility for the service admins to add more
 targets without having to send patches to our puppet.
 >
 > That's going to be the tough part. Because the blackbox exporter is
 managed through puppet, in this configuration, we forcibly control the
 config file and local changes would be lost. So it's really one of three
 things:
 >
 >  1. they send us the target and so we need to intervene when that needs
 to be changed
 >  2. we patch the module to allow us to *not* manage the blackbox config
 (yet another patch to the module, we can take it)
 >  3. we just *not* manage the blackbox exporter through puppet at all,
 and add it manually to the `scrape_configs` target.
 >
 > I would do option 2, eventually, and start with option 1. makes sense?

 Yep makes sense.

--
Ticket URL: <https://trac.torproject.org/projects/tor/ticket/33027#comment:3>
Tor Bug Tracker & Wiki <https://trac.torproject.org/>
The Tor Project: anonymity online