February 2022 - metrics-alerts - lists.torproject.org

** RECOVERY Service Alert: onionoo-frontend-02/network service - onionoo varnish is OK **
by nagios＠hetzner-hel1-01.torproject.org 28 Feb '22

28 Feb '22

***** Icinga ***** Notification Type: RECOVERY Service: network service - onionoo varnish Host: onionoo-frontend-02 Address: 49.12.57.137 State: OK Date/Time: Mon Feb 28 23:36:10 UTC 2022 Additional Info: OK: Valid response with recent timestamps (5765 sec): relays_published=2022-02-28 22:00:00, bridges_published=2022-02-28 22:45:56

1 0

** RECOVERY Service Alert: onionoo-frontend-01/network service - onionoo full service check is OK **
by nagios＠hetzner-hel1-01.torproject.org 28 Feb '22

28 Feb '22

***** Icinga ***** Notification Type: RECOVERY Service: network service - onionoo full service check Host: onionoo-frontend-01 Address: 116.202.120.171 State: OK Date/Time: Mon Feb 28 23:36:10 UTC 2022 Additional Info: OK: Valid response with recent timestamps (6609 sec): relays_published=2022-02-28 22:00:00, bridges_published=2022-02-28 21:45:56

1 0

** PROBLEM Service Alert: onionoo-frontend-01/network service - onionoo full service check is WARNING **
by nagios＠hetzner-hel1-01.torproject.org 28 Feb '22

28 Feb '22

***** Icinga ***** Notification Type: PROBLEM Service: network service - onionoo full service check Host: onionoo-frontend-01 Address: 116.202.120.171 State: WARNING Date/Time: Mon Feb 28 23:09:00 UTC 2022 Additional Info: WARNING: Either timestamp is not recent (12177 sec): relays_published=2022-02-28 20:00:00, bridges_published=2022-02-28 19:45:56

1 0

** PROBLEM Service Alert: onionoo-frontend-02/network service - onionoo varnish is WARNING **
by nagios＠hetzner-hel1-01.torproject.org 28 Feb '22

28 Feb '22

***** Icinga ***** Notification Type: PROBLEM Service: network service - onionoo varnish Host: onionoo-frontend-02 Address: 49.12.57.137 State: WARNING Date/Time: Mon Feb 28 23:05:50 UTC 2022 Additional Info: WARNING: Either timestamp is not recent (11985 sec): relays_published=2022-02-28 20:00:00, bridges_published=2022-02-28 19:45:56

1 0

[FIRING:1] Onionperf disk space
by alertmanager＠hetzner-nbg1-02.torproject.org 28 Feb '22

28 Feb '22

Total firing alerts: 1 Total resolved alerts: 0 ## Firing Alerts ----- Time: 2022-02-28 22:55:57.544031668 +0000 UTC Summary: This onionperf instace is using more than 80% disk space available. Description: Onionperf instance disk space used is more than 80% ----- ##Resolved Alerts

1 0

Cron <onionoo@onionoo-backend-01> /srv/onionoo.torproject.org/onionoo/monitoring/network/onionoo
by root＠onionoo-backend-01.torproject.org 28 Feb '22

28 Feb '22

Traceback (most recent call last): File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 159, in _new_conn (self._dns_host, self.port), self.timeout, **extra_kw) File "/usr/lib/python3/dist-packages/urllib3/util/connection.py", line 57, in create_connection for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM): File "/usr/lib/python3.7/socket.py", line 748, in getaddrinfo for res in _socket.getaddrinfo(host, port, family, type, proto, flags): socket.gaierror: [Errno -3] Temporary failure in name resolution During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 600, in urlopen chunked=chunked) File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 343, in _make_request self._validate_conn(conn) File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 841, in _validate_conn conn.connect() File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 301, in connect conn = self._new_conn() File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 168, in _new_conn self, "Failed to establish a new connection: %s" % e) urllib3.exceptions.NewConnectionError: <urllib3.connection.VerifiedHTTPSConnection object at 0x7fed0184ef98>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/lib/python3/dist-packages/requests/adapters.py", line 449, in send timeout=timeout File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 638, in urlopen _stacktrace=sys.exc_info()[2]) File "/usr/lib/python3/dist-packages/urllib3/util/retry.py", line 398, in increment raise MaxRetryError(_pool, url, error or ResponseError(cause)) urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='onionoo.torproject.org', port=443): Max retries exceeded with url: /details (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7fed0184ef98>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/srv/onionoo.torproject.org/onionoo/monitoring/network/onionoo", line 11, in <module> details = json.loads(requests.get("https://onionoo.torproject.org/details").text) File "/usr/lib/python3/dist-packages/requests/api.py", line 75, in get return request('get', url, params=params, **kwargs) File "/usr/lib/python3/dist-packages/requests/api.py", line 60, in request return session.request(method=method, url=url, **kwargs) File "/usr/lib/python3/dist-packages/requests/sessions.py", line 533, in request resp = self.send(prep, **send_kwargs) File "/usr/lib/python3/dist-packages/requests/sessions.py", line 646, in send r = adapter.send(request, **kwargs) File "/usr/lib/python3/dist-packages/requests/adapters.py", line 516, in send raise ConnectionError(e, request=request) requests.exceptions.ConnectionError: HTTPSConnectionPool(host='onionoo.torproject.org', port=443): Max retries exceeded with url: /details (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7fed0184ef98>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution'))

1 0

Cron <onionoo@onionoo-backend-02> /srv/onionoo.torproject.org/onionoo/monitoring/network/onionoo
by root＠onionoo-backend-02.torproject.org 28 Feb '22

28 Feb '22

Traceback (most recent call last): File "/srv/onionoo.torproject.org/onionoo/monitoring/network/onionoo", line 11, in <module> details = json.loads(requests.get("https://onionoo.torproject.org/details").text) File "/usr/lib/python3.7/json/__init__.py", line 348, in loads return _default_decoder.decode(s) File "/usr/lib/python3.7/json/decoder.py", line 337, in decode obj, end = self.raw_decode(s, idx=_w(s, 0).end()) File "/usr/lib/python3.7/json/decoder.py", line 355, in raw_decode raise JSONDecodeError("Expecting value", s, err.value) from None json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

1 0

Cron <onionoo@onionoo-backend-01> /srv/onionoo.torproject.org/onionoo/monitoring/network/onionoo
by root＠onionoo-backend-01.torproject.org 28 Feb '22

28 Feb '22

Traceback (most recent call last): File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 159, in _new_conn (self._dns_host, self.port), self.timeout, **extra_kw) File "/usr/lib/python3/dist-packages/urllib3/util/connection.py", line 80, in create_connection raise err File "/usr/lib/python3/dist-packages/urllib3/util/connection.py", line 70, in create_connection sock.connect(sa) TimeoutError: [Errno 110] Connection timed out During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 600, in urlopen chunked=chunked) File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 343, in _make_request self._validate_conn(conn) File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 841, in _validate_conn conn.connect() File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 301, in connect conn = self._new_conn() File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 168, in _new_conn self, "Failed to establish a new connection: %s" % e) urllib3.exceptions.NewConnectionError: <urllib3.connection.VerifiedHTTPSConnection object at 0x7f331a42d048>: Failed to establish a new connection: [Errno 110] Connection timed out During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/lib/python3/dist-packages/requests/adapters.py", line 449, in send timeout=timeout File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 638, in urlopen _stacktrace=sys.exc_info()[2]) File "/usr/lib/python3/dist-packages/urllib3/util/retry.py", line 398, in increment raise MaxRetryError(_pool, url, error or ResponseError(cause)) urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='onionoo.torproject.org', port=443): Max retries exceeded with url: /details (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7f331a42d048>: Failed to establish a new connection: [Errno 110] Connection timed out')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/srv/onionoo.torproject.org/onionoo/monitoring/network/onionoo", line 11, in <module> details = json.loads(requests.get("https://onionoo.torproject.org/details").text) File "/usr/lib/python3/dist-packages/requests/api.py", line 75, in get return request('get', url, params=params, **kwargs) File "/usr/lib/python3/dist-packages/requests/api.py", line 60, in request return session.request(method=method, url=url, **kwargs) File "/usr/lib/python3/dist-packages/requests/sessions.py", line 533, in request resp = self.send(prep, **send_kwargs) File "/usr/lib/python3/dist-packages/requests/sessions.py", line 646, in send r = adapter.send(request, **kwargs) File "/usr/lib/python3/dist-packages/requests/adapters.py", line 516, in send raise ConnectionError(e, request=request) requests.exceptions.ConnectionError: HTTPSConnectionPool(host='onionoo.torproject.org', port=443): Max retries exceeded with url: /details (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7f331a42d048>: Failed to establish a new connection: [Errno 110] Connection timed out'))

1 0

Cron <onionoo@onionoo-backend-02> /srv/onionoo.torproject.org/onionoo/monitoring/network/onionoo
by root＠onionoo-backend-02.torproject.org 28 Feb '22

28 Feb '22

Traceback (most recent call last): File "/srv/onionoo.torproject.org/onionoo/monitoring/network/onionoo", line 12, in <module> bandwidth = json.loads(requests.get("https://onionoo.torproject.org/bandwidth").text) File "/usr/lib/python3.7/json/__init__.py", line 348, in loads return _default_decoder.decode(s) File "/usr/lib/python3.7/json/decoder.py", line 337, in decode obj, end = self.raw_decode(s, idx=_w(s, 0).end()) File "/usr/lib/python3.7/json/decoder.py", line 355, in raw_decode raise JSONDecodeError("Expecting value", s, err.value) from None json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

1 0

** RECOVERY Service Alert: onionoo-frontend-02/network service - onionoo varnish is OK **
by nagios＠hetzner-hel1-01.torproject.org 28 Feb '22

28 Feb '22

***** Icinga ***** Notification Type: RECOVERY Service: network service - onionoo varnish Host: onionoo-frontend-02 Address: 49.12.57.137 State: OK Date/Time: Mon Feb 28 18:40:50 UTC 2022 Additional Info: OK: Valid response with recent timestamps (6041 sec): relays_published=2022-02-28 17:00:00, bridges_published=2022-02-28 17:15:55

1 0