September 2025 - metrics-alerts - lists.torproject.org

Cron <onionoo@onionoo-backend-03> /srv/onionoo.torproject.org/onionoo/monitoring/network/onionoo
by root＠onionoo-backend-03.torproject.org 25 Sep '25

25 Sep '25

Traceback (most recent call last): File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 198, in _new_conn sock = connection.create_connection( (self._dns_host, self.port), ...<2 lines>... socket_options=self.socket_options, ) File "/usr/lib/python3/dist-packages/urllib3/util/connection.py", line 60, in create_connection for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM): ~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.13/socket.py", line 977, in getaddrinfo for res in _socket.getaddrinfo(host, port, family, type, proto, flags): ~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ socket.gaierror: [Errno -3] Temporary failure in name resolution The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 787, in urlopen response = self._make_request( conn, ...<10 lines>... **response_kw, ) File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 488, in _make_request raise new_e File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 464, in _make_request self._validate_conn(conn) ~~~~~~~~~~~~~~~~~~~^^^^^^ File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 1093, in _validate_conn conn.connect() ~~~~~~~~~~~~^^ File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 704, in connect self.sock = sock = self._new_conn() ~~~~~~~~~~~~~~^^ File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 205, in _new_conn raise NameResolutionError(self.host, self, e) from e urllib3.exceptions.NameResolutionError: <urllib3.connection.HTTPSConnection object at 0x7fa5c7566f90>: Failed to resolve 'onionoo.torproject.org' ([Errno -3] Temporary failure in name resolution) The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/lib/python3/dist-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( method=request.method, ...<9 lines>... chunked=chunked, ) File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 841, in urlopen retries = retries.increment( method, url, error=new_e, _pool=self, _stacktrace=sys.exc_info()[2] ) File "/usr/lib/python3/dist-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='onionoo.torproject.org', port=443): Max retries exceeded with url: /details (Caused by NameResolutionError("<urllib3.connection.HTTPSConnection object at 0x7fa5c7566f90>: Failed to resolve 'onionoo.torproject.org' ([Errno -3] Temporary failure in name resolution)")) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/srv/onionoo.torproject.org/onionoo/monitoring/network/onionoo", line 31, in <module> details = json.loads(requests.get("https://onionoo.torproject.org/details").text) ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3/dist-packages/requests/api.py", line 73, in get return request("get", url, params=params, **kwargs) File "/usr/lib/python3/dist-packages/requests/api.py", line 59, in request return session.request(method=method, url=url, **kwargs) ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3/dist-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) File "/usr/lib/python3/dist-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) File "/usr/lib/python3/dist-packages/requests/adapters.py", line 700, in send raise ConnectionError(e, request=request) requests.exceptions.ConnectionError: HTTPSConnectionPool(host='onionoo.torproject.org', port=443): Max retries exceeded with url: /details (Caused by NameResolutionError("<urllib3.connection.HTTPSConnection object at 0x7fa5c7566f90>: Failed to resolve 'onionoo.torproject.org' ([Errno -3] Temporary failure in name resolution)"))

1 0

Cron <onionoo@onionoo-backend-03> /srv/onionoo.torproject.org/onionoo/monitoring/services/onionoo
by root＠onionoo-backend-03.torproject.org 25 Sep '25

25 Sep '25

Error fetching minimal summary document: <urlopen error [Errno -3] Temporary failure in name resolution>

1 0

Cron <onionoo@onionoo-backend-03> /srv/onionoo.torproject.org/onionoo/monitoring/network/onionoo
by root＠onionoo-backend-03.torproject.org 25 Sep '25

25 Sep '25

Traceback (most recent call last): File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 198, in _new_conn sock = connection.create_connection( (self._dns_host, self.port), ...<2 lines>... socket_options=self.socket_options, ) File "/usr/lib/python3/dist-packages/urllib3/util/connection.py", line 60, in create_connection for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM): ~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.13/socket.py", line 977, in getaddrinfo for res in _socket.getaddrinfo(host, port, family, type, proto, flags): ~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ socket.gaierror: [Errno -3] Temporary failure in name resolution The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 787, in urlopen response = self._make_request( conn, ...<10 lines>... **response_kw, ) File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 488, in _make_request raise new_e File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 464, in _make_request self._validate_conn(conn) ~~~~~~~~~~~~~~~~~~~^^^^^^ File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 1093, in _validate_conn conn.connect() ~~~~~~~~~~~~^^ File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 704, in connect self.sock = sock = self._new_conn() ~~~~~~~~~~~~~~^^ File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 205, in _new_conn raise NameResolutionError(self.host, self, e) from e urllib3.exceptions.NameResolutionError: <urllib3.connection.HTTPSConnection object at 0x7f61a1b66f90>: Failed to resolve 'onionoo.torproject.org' ([Errno -3] Temporary failure in name resolution) The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/lib/python3/dist-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( method=request.method, ...<9 lines>... chunked=chunked, ) File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 841, in urlopen retries = retries.increment( method, url, error=new_e, _pool=self, _stacktrace=sys.exc_info()[2] ) File "/usr/lib/python3/dist-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='onionoo.torproject.org', port=443): Max retries exceeded with url: /details (Caused by NameResolutionError("<urllib3.connection.HTTPSConnection object at 0x7f61a1b66f90>: Failed to resolve 'onionoo.torproject.org' ([Errno -3] Temporary failure in name resolution)")) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/srv/onionoo.torproject.org/onionoo/monitoring/network/onionoo", line 31, in <module> details = json.loads(requests.get("https://onionoo.torproject.org/details").text) ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3/dist-packages/requests/api.py", line 73, in get return request("get", url, params=params, **kwargs) File "/usr/lib/python3/dist-packages/requests/api.py", line 59, in request return session.request(method=method, url=url, **kwargs) ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3/dist-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) File "/usr/lib/python3/dist-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) File "/usr/lib/python3/dist-packages/requests/adapters.py", line 700, in send raise ConnectionError(e, request=request) requests.exceptions.ConnectionError: HTTPSConnectionPool(host='onionoo.torproject.org', port=443): Max retries exceeded with url: /details (Caused by NameResolutionError("<urllib3.connection.HTTPSConnection object at 0x7f61a1b66f90>: Failed to resolve 'onionoo.torproject.org' ([Errno -3] Temporary failure in name resolution)"))

1 0

Cron <onionoo@onionoo-backend-03> /srv/onionoo.torproject.org/onionoo/monitoring/services/onionoo
by root＠onionoo-backend-03.torproject.org 25 Sep '25

25 Sep '25

Error fetching minimal summary document: <urlopen error [Errno -3] Temporary failure in name resolution>

1 0

[FIRING:1] JobDown -- onionoo_service
by alertmanager＠prometheus2.torproject.org 25 Sep '25

25 Sep '25

Total firing alerts: 1 ## Firing Alerts ----- Time: 2025-09-25 15:40:06.784 +0000 UTC Summary: Exporter job "onionoo_service" on onionoo.torproject.org:443 is down Description: Exporter job "onionoo_service" on onionoo.torproject.org:443 has been unreachable for more than 15 minutes. playbook: https://gitlab.torproject.org/tpo/tpa/team/-/wikis/service/prometheus#expor… -----

1 0

Cron <onionoo@onionoo-backend-03> /srv/onionoo.torproject.org/onionoo/monitoring/services/onionoo
by root＠onionoo-backend-03.torproject.org 25 Sep '25

25 Sep '25

Error fetching minimal summary document: <urlopen error [Errno -3] Temporary failure in name resolution>

1 0

Cron <onionoo@onionoo-backend-03> /srv/onionoo.torproject.org/onionoo/monitoring/network/onionoo
by root＠onionoo-backend-03.torproject.org 25 Sep '25

25 Sep '25

Traceback (most recent call last): File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 198, in _new_conn sock = connection.create_connection( (self._dns_host, self.port), ...<2 lines>... socket_options=self.socket_options, ) File "/usr/lib/python3/dist-packages/urllib3/util/connection.py", line 60, in create_connection for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM): ~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.13/socket.py", line 977, in getaddrinfo for res in _socket.getaddrinfo(host, port, family, type, proto, flags): ~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ socket.gaierror: [Errno -3] Temporary failure in name resolution The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 787, in urlopen response = self._make_request( conn, ...<10 lines>... **response_kw, ) File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 488, in _make_request raise new_e File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 464, in _make_request self._validate_conn(conn) ~~~~~~~~~~~~~~~~~~~^^^^^^ File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 1093, in _validate_conn conn.connect() ~~~~~~~~~~~~^^ File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 704, in connect self.sock = sock = self._new_conn() ~~~~~~~~~~~~~~^^ File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 205, in _new_conn raise NameResolutionError(self.host, self, e) from e urllib3.exceptions.NameResolutionError: <urllib3.connection.HTTPSConnection object at 0x7ff810e82f90>: Failed to resolve 'onionoo.torproject.org' ([Errno -3] Temporary failure in name resolution) The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/lib/python3/dist-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( method=request.method, ...<9 lines>... chunked=chunked, ) File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 841, in urlopen retries = retries.increment( method, url, error=new_e, _pool=self, _stacktrace=sys.exc_info()[2] ) File "/usr/lib/python3/dist-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='onionoo.torproject.org', port=443): Max retries exceeded with url: /details (Caused by NameResolutionError("<urllib3.connection.HTTPSConnection object at 0x7ff810e82f90>: Failed to resolve 'onionoo.torproject.org' ([Errno -3] Temporary failure in name resolution)")) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/srv/onionoo.torproject.org/onionoo/monitoring/network/onionoo", line 31, in <module> details = json.loads(requests.get("https://onionoo.torproject.org/details").text) ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3/dist-packages/requests/api.py", line 73, in get return request("get", url, params=params, **kwargs) File "/usr/lib/python3/dist-packages/requests/api.py", line 59, in request return session.request(method=method, url=url, **kwargs) ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3/dist-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) File "/usr/lib/python3/dist-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) File "/usr/lib/python3/dist-packages/requests/adapters.py", line 700, in send raise ConnectionError(e, request=request) requests.exceptions.ConnectionError: HTTPSConnectionPool(host='onionoo.torproject.org', port=443): Max retries exceeded with url: /details (Caused by NameResolutionError("<urllib3.connection.HTTPSConnection object at 0x7ff810e82f90>: Failed to resolve 'onionoo.torproject.org' ([Errno -3] Temporary failure in name resolution)"))

1 0

[FIRING:1] JobDown -- metrics_web_service
by alertmanager＠prometheus2.torproject.org 25 Sep '25

25 Sep '25

Total firing alerts: 1 ## Firing Alerts ----- Time: 2025-09-25 15:55:51.784 +0000 UTC Summary: Exporter job "metrics_web_service" on metrics.torproject.org:443 is down Description: Exporter job "metrics_web_service" on metrics.torproject.org:443 has been unreachable for more than 15 minutes. playbook: https://gitlab.torproject.org/tpo/tpa/team/-/wikis/service/prometheus#expor… -----

1 0

[FIRING:1] JobDown -- metrics-standard
by alertmanager＠prometheus2.torproject.org 25 Sep '25

25 Sep '25

Total firing alerts: 1 ## Firing Alerts ----- Time: 2025-09-25 15:40:36.784 +0000 UTC Summary: Exporter job "metrics-standard" on probetelemetry-01.torproject.org:443 is down Description: Exporter job "metrics-standard" on probetelemetry-01.torproject.org:443 has been unreachable for more than 15 minutes. playbook: https://gitlab.torproject.org/tpo/tpa/team/-/wikis/service/prometheus#expor… -----

1 0

[FIRING:1] JobDown -- onionoo_network
by alertmanager＠prometheus2.torproject.org 25 Sep '25

25 Sep '25

Total firing alerts: 1 ## Firing Alerts ----- Time: 2025-09-25 15:14:36.784 +0000 UTC Summary: Exporter job "onionoo_network" on onionoo.torproject.org:443 is down Description: Exporter job "onionoo_network" on onionoo.torproject.org:443 has been unreachable for more than 15 minutes. playbook: https://gitlab.torproject.org/tpo/tpa/team/-/wikis/service/prometheus#expor… -----

1 0