[tor-bugs] #31916 [Internal Services/Tor Sysadmin Team]: reliability issues with hetzner-nbg1-01

[Tor Bug Tracker & Wiki]

#31916: reliability issues with hetzner-nbg1-01
-------------------------------------------------+-------------------------
 Reporter:  anarcat                              |          Owner:  anarcat
     Type:  defect                               |         Status:
                                                 |  assigned
 Priority:  Medium                               |      Milestone:
Component:  Internal Services/Tor Sysadmin Team  |        Version:
 Severity:  Blocker                              |     Resolution:
 Keywords:                                       |  Actual Points:
Parent ID:                                       |         Points:
 Reviewer:                                       |        Sponsor:
-------------------------------------------------+-------------------------
Description changed by anarcat:

Old description:

> The new Prometheus monitoring server (hetzner-nbg1-01.torproject.org) is
> seeing intermittent networking issues. It's proving very difficult to get
> reliable metrics out of it, in any case. From its perspective, random
> hosts blink in and out of existence unreliably, with almost *all* hosts
> (63 of the ~80 monitored) are affected over a period of a week. This
> leads me to believe the problem is not with *all* hosts, but with the
> monitoring server itself. The attached screenshot (tpo-overview.png)
> shows the randomness of the problem, as seen from hetzner-
> nbg1-01.torproject.org during the last 7 days.
>
> We have another monitoring server hosted in the Hetzner cloud (hetzner-
> hel1-01.torproject.org) which doesn't seem to suffer from the same
> problems. From its perspective, most hosts are healthy over the same
> period, with an average availability of 99.876% over all hosts, which
> includes at least one outlier at 88%. The other (nagios) monitoring
> server sees the new monitoring server with only a 99.728% availbility,
> with a total 30 minutes downtime over the last 7 days. Note that those
> statistics have a large margin of error as the Nagios checks are much
> less frequent than the Prometheus ones, with a granularity ranging in
> tens of minutes instead of seconds.
>
> The alert history graph (second attachment,  histogram.cgi-nbg1-01.png)
> shows more clearly the problem, especially when compared to a similar
> host in the vincinity (hetzner-nbg01-02, third attachement, histogram
> .cgi-nbg1-02.png).
>
> I would therefore conclude there is a severe and intermittent routing
> issue with this server.

New description:

 The new Prometheus monitoring server (hetzner-nbg1-01.torproject.org) is
 seeing intermittent networking issues. It's proving very difficult to get
 reliable metrics out of it, in any case. From its perspective, random
 hosts blink in and out of existence unreliably, with almost *all* hosts
 (63 of the ~80 monitored) are affected over a period of a week. This leads
 me to believe the problem is not with *all* hosts, but with the monitoring
 server itself. The attached screenshot (tpo-overview.png) shows the
 randomness of the problem, as seen from hetzner-nbg1-01.torproject.org
 during the last 7 days.

 [[Image(tpo-overview.png​, 700)]]

 We have another monitoring server hosted in the Hetzner cloud (hetzner-
 hel1-01.torproject.org) which doesn't seem to suffer from the same
 problems. From its perspective, most hosts are healthy over the same
 period, with an average availability of 99.876% over all hosts, which
 includes at least one outlier at 88%. The other (nagios) monitoring server
 sees the new monitoring server with only a 99.728% availbility, with a
 total 30 minutes downtime over the last 7 days. Note that those statistics
 have a large margin of error as the Nagios checks are much less frequent
 than the Prometheus ones, with a granularity ranging in tens of minutes
 instead of seconds.

 The alert history graph (second attachment,  histogram.cgi-nbg1-01.png)
 shows more clearly the problem, especially when compared to a similar host
 in the vincinity (hetzner-nbg01-02, third attachement, histogram.cgi-
 nbg1-02.png).

 [[Image(histogram.cgi-nbg1-01.png​, 700)]]
 [[Image(histogram.cgi-nbg1-02.png​, 700)]]

 I would therefore conclude there is a severe and intermittent routing
 issue with this server.

--

--
Ticket URL: <https://trac.torproject.org/projects/tor/ticket/31916#comment:1>
Tor Bug Tracker & Wiki <https://trac.torproject.org/>
The Tor Project: anonymity online