[tor-bugs] #30028 [Internal Services/Tor Sysadmin Team]: additional prometheus/grafana exporters/dashboards

[Tor Bug Tracker & Wiki]

#30028: additional prometheus/grafana exporters/dashboards
-------------------------------------------------+-------------------------
 Reporter:  anarcat                              |          Owner:  anarcat
     Type:  project                              |         Status:  closed
 Priority:  Medium                               |      Milestone:
Component:  Internal Services/Tor Sysadmin Team  |        Version:
 Severity:  Normal                               |     Resolution:  fixed
 Keywords:                                       |  Actual Points:
Parent ID:  #29681                               |         Points:
 Reviewer:                                       |        Sponsor:
-------------------------------------------------+-------------------------
Description changed by anarcat:

Old description:

> our munin replacement is not entirely complete, as there are key parts of
> the infrastructure that are not monitored. here's a short inventory of
> what I found in #29681:
>
> '''email servers monitoring (eugeni, etc? postfix)'''
>
> * [https://github.com/kumina/postfix_exporter in debian],
> [https://github.com/kumina/postfix_exporter/issues/21 possible dashboard]
> * another approach: [https://github.com/cherti/mailexporter email
> delivery tests]
>
> '''mailman monitoring'''
>
> no known exporter or dashboard
>
> '''databases'''
>
> * [https://github.com/wrouesnel/postgres_exporter/ postgres exporter in
> debian], [https://github.com/wrouesnel/postgres_exporter/issues/218 no
> offocial dashboard], but
> [https://grafana.com/dashboards?dataSource=prometheus&search=postgres
> many possible dashboards]
> * [https://github.com/prometheus/mysqld_exporter mysqld exporter in
> debian] - [https://grafana.com/dashboards/625 possible dashboard]
> [https://github.com/percona/grafana-dashboards another from  percona],
> [https://github.com/prometheus/mysqld_exporter/issues/286 not officially
> documented]
> * [https://github.com/free/sql_exporter generic sql exporter], in debian
> - [https://github.com/credativ/elephant-shed/tree/master/sql-exporter
> credativ config] and [https://github.com/credativ/elephant-
> shed/tree/master/grafana dashboard]
>
> '''DNS / bind'''
>
> - [https://github.com/digitalocean/bind_exporter/ in debian],
> [https://grafana.com/dashboards/1666 official dashboard]
>
> '''GitLab'''
>
> there is
> [https://docs.gitlab.com/ee/administration/monitoring/prometheus/ builtin
> support for prometheus] that has to be
> [https://docs.gitlab.com/ee/administration/monitoring/prometheus/gitlab_metrics.html
> configured]
>
> those are the other missing things I found during the audit performed
> while removing Munin:
>
>  * '''spamassassin''': ham/spam/total counts, looks for `spamd:
> ((processing|checking) message|identified spam|clean message)` in
> mail.log, could be replaced with [https://github.com/google/mtail ​mtail]
>  * '''postgres-wal-traffic_''': should be covered by the
> postgres_exporter mentioned above, otherwise hook `psql -p "$port" --no-
> align --command 'SELECT * FROM pg_current_xlog_insert_location()'
> --tuples-only --quiet | tr -d /,` into the node_exporter
>  * '''ksm stats''': extra memory statistics, might not be very important
>  * '''haproxy''': https://github.com/prometheus/haproxy_exporter
>  * '''per VM disk usage''': see  #29816
>  * '''vsftpd''': custom mtail plugin, no known exporter or dashboard
>
> See the full review in #29682 for details on those.
>
> There were also demands from other teams for monitoring, see #29863 and
> #30006 for now.

New description:

 our munin replacement is not entirely complete, as there are key parts of
 the infrastructure that are not monitored. here's a short inventory of
 what I found in #29681:

 '''email servers monitoring (eugeni, etc? postfix)'''

 * [https://github.com/kumina/postfix_exporter in debian],
 [https://github.com/kumina/postfix_exporter/issues/21 possible dashboard]
 * another approach: [https://github.com/cherti/mailexporter email delivery
 tests]

 '''mailman monitoring'''

 no known exporter or dashboard

 '''databases'''

 * [https://github.com/wrouesnel/postgres_exporter/ postgres exporter], in
 debian, [https://github.com/wrouesnel/postgres_exporter/issues/218 no
 offocial dashboard], but
 [https://grafana.com/dashboards?dataSource=prometheus&search=postgres many
 possible dashboards]
 * [https://github.com/prometheus/mysqld_exporter mysqld exporter], in
 debian - [https://grafana.com/dashboards/625 possible dashboard]
 [https://github.com/percona/grafana-dashboards another from  percona],
 [https://github.com/prometheus/mysqld_exporter/issues/286 not officially
 documented]
 * [https://github.com/free/sql_exporter generic sql exporter], in debian -
 [https://github.com/credativ/elephant-shed/tree/master/sql-exporter
 credativ config] and [https://github.com/credativ/elephant-
 shed/tree/master/grafana dashboard]

 '''DNS / bind'''

 - [https://github.com/digitalocean/bind_exporter/ in debian],
 [https://grafana.com/dashboards/1666 official dashboard]

 '''GitLab'''

 there is [https://docs.gitlab.com/ee/administration/monitoring/prometheus/
 builtin support for prometheus] that has to be
 [https://docs.gitlab.com/ee/administration/monitoring/prometheus/gitlab_metrics.html
 configured]

 those are the other missing things I found during the audit performed
 while removing Munin:

  * '''spamassassin''': ham/spam/total counts, looks for `spamd:
 ((processing|checking) message|identified spam|clean message)` in
 mail.log, could be replaced with [https://github.com/google/mtail ​mtail]
  * '''postgres-wal-traffic_''': should be covered by the postgres_exporter
 mentioned above, otherwise hook `psql -p "$port" --no-align --command
 'SELECT * FROM pg_current_xlog_insert_location()' --tuples-only --quiet |
 tr -d /,` into the node_exporter
  * '''ksm stats''': extra memory statistics, might not be very important
  * '''haproxy''': https://github.com/prometheus/haproxy_exporter
  * '''per VM disk usage''': see  #29816
  * '''vsftpd''': custom mtail plugin, no known exporter or dashboard

 See the full review in #29682 for details on those.

 There were also demands from other teams for monitoring, see #29863 and
 #30006 for now.

--

--
Ticket URL: <https://trac.torproject.org/projects/tor/ticket/30028#comment:10>
Tor Bug Tracker & Wiki <https://trac.torproject.org/>
The Tor Project: anonymity online