[tor-bugs] #31159 [Internal Services/Tor Sysadmin Team]: Monitor anti-censorship www services with prometheus

[Tor Bug Tracker & Wiki]

#31159: Monitor anti-censorship www services with prometheus
-------------------------------------------------+---------------------
 Reporter:  phw                                  |          Owner:  tpa
     Type:  task                                 |         Status:  new
 Priority:  Medium                               |      Milestone:
Component:  Internal Services/Tor Sysadmin Team  |        Version:
 Severity:  Normal                               |     Resolution:
 Keywords:                                       |  Actual Points:
Parent ID:  #30152                               |         Points:  1
 Reviewer:                                       |        Sponsor:
-------------------------------------------------+---------------------

Comment (by anarcat):

 Replying to [comment:2 phw]:
 > Replying to [comment:1 hiro]:
 > > There is also another aspect to consider, in the case of a service
 like gettor, monitoring the https endpoint will only give us some info
 about the static html we are serving with apache. Gettor itself (the
 service sending emails) is a twisted service instead.
 > [[br]]
 > Gotcha. We have a similar problem with BridgeDB because it is exposed
 over an Apache reverse proxy and you cannot directly talk to BridgeDB.
 However, if BridgeDB is down, bridges.torproject.org responds with an
 internal server error if I remember correctly, so we can still monitor
 BridgeDB despite the reverse proxy, right?

 Should, yes.

 > To monitor BridgeDB, we need to set up an exporter, right?

 In Prometheus, yes. This could be a simple configuration in a "blackbox
 exporter":

 https://github.com/prometheus/blackbox_exporter/

 > > Maybe we can consider an approach in which services expose an http
 endpoint that we can use to know that the service is alive. Otherwise I
 think we could do some other monitoring via nagios checks.
 >
 > I think we already have that for BridgeDB and snowflake's website but
 not for GetTor.

 From what I can tell, we check bridges.torproject.org:

 {{{
   -
     name: bridges.tpo web service
     nrpe: "/usr/lib/nagios/plugins/check_http -H bridges.torproject.org -S
 --string=bridge"
     hosts: polyanthum
     depends: network service - https
 }}}

 We also check onionoo:

 {{{
  # non-tpa services
  ####
   -
     name: network service - onionoo backend
     nrpe: "/usr/lib/nagios/plugins/tor-check-onionoo 127.0.0.1:8080"
     hostgroups: onionoo-backend
     depends: "process - haproxy - master"
     contacts: +metrics
   -
     name: network service - onionoo varnish
     nrpe: "/usr/lib/nagios/plugins/tor-check-onionoo 127.0.0.1:6081"
     hostgroups: onionoo-backend
     depends: "process - haproxy - master"
     contacts: +metrics
   -
     name: network service - onionoo haproxy
     nrpe: "/usr/lib/nagios/plugins/tor-check-onionoo -s
 onionoo.torproject.org"
     hostgroups: onionoo-backend
     depends: "process - haproxy - master"
     contacts: +metrics
 }}}

 ... but those are all TPA machines, so they can be monitored by Nagios.

--
Ticket URL: <https://trac.torproject.org/projects/tor/ticket/31159#comment:3>
Tor Bug Tracker & Wiki <https://trac.torproject.org/>
The Tor Project: anonymity online