September 2025 - metrics-alerts - lists.torproject.org

[FIRING:1] JobDown -- metrics_web_network
by alertmanager＠prometheus2.torproject.org 25 Sep '25

25 Sep '25

Total firing alerts: 1 ## Firing Alerts ----- Time: 2025-09-25 15:55:36.784 +0000 UTC Summary: Exporter job "metrics_web_network" on metrics.torproject.org:443 is down Description: Exporter job "metrics_web_network" on metrics.torproject.org:443 has been unreachable for more than 15 minutes. playbook: https://gitlab.torproject.org/tpo/tpa/team/-/wikis/service/prometheus#expor… -----

1 0

[FIRING:1] JobDown -- tordnsel_data
by alertmanager＠prometheus2.torproject.org 25 Sep '25

25 Sep '25

Total firing alerts: 1 ## Firing Alerts ----- Time: 2025-09-25 15:55:36.784 +0000 UTC Summary: Exporter job "tordnsel_data" on check.torproject.org:443 is down Description: Exporter job "tordnsel_data" on check.torproject.org:443 has been unreachable for more than 15 minutes. playbook: https://gitlab.torproject.org/tpo/tpa/team/-/wikis/service/prometheus#expor… -----

1 0

[FIRING:1] JobDown -- metrics_web_data
by alertmanager＠prometheus2.torproject.org 25 Sep '25

25 Sep '25

Total firing alerts: 1 ## Firing Alerts ----- Time: 2025-09-25 15:55:06.784 +0000 UTC Summary: Exporter job "metrics_web_data" on metrics.torproject.org:443 is down Description: Exporter job "metrics_web_data" on metrics.torproject.org:443 has been unreachable for more than 15 minutes. playbook: https://gitlab.torproject.org/tpo/tpa/team/-/wikis/service/prometheus#expor… -----

1 0

[FIRING:6] JobDown -- onionperf_metrics_port
by alertmanager＠prometheus2.torproject.org 25 Sep '25

25 Sep '25

Total firing alerts: 6 ## Firing Alerts ----- Time: 2025-09-25 15:25:21.784 +0000 UTC Summary: Exporter job "onionperf_metrics_port" on op-de8a-conflux.onionperf.torproject.net:9143 is down Description: Exporter job "onionperf_metrics_port" on op-de8a-conflux.onionperf.torproject.net:9143 has been unreachable for more than 15 minutes. playbook: https://gitlab.torproject.org/tpo/tpa/team/-/wikis/service/prometheus#expor… ----- ----- Time: 2025-09-25 15:23:51.784 +0000 UTC Summary: Exporter job "onionperf_metrics_port" on op-de8a.onionperf.torproject.net:9143 is down Description: Exporter job "onionperf_metrics_port" on op-de8a.onionperf.torproject.net:9143 has been unreachable for more than 15 minutes. playbook: https://gitlab.torproject.org/tpo/tpa/team/-/wikis/service/prometheus#expor… ----- ----- Time: 2025-09-25 15:23:36.784 +0000 UTC Summary: Exporter job "onionperf_metrics_port" on op-hk8a-conflux.onionperf.torproject.net:9143 is down Description: Exporter job "onionperf_metrics_port" on op-hk8a-conflux.onionperf.torproject.net:9143 has been unreachable for more than 15 minutes. playbook: https://gitlab.torproject.org/tpo/tpa/team/-/wikis/service/prometheus#expor… ----- ----- Time: 2025-09-25 15:23:51.784 +0000 UTC Summary: Exporter job "onionperf_metrics_port" on op-hk8a.onionperf.torproject.net:9143 is down Description: Exporter job "onionperf_metrics_port" on op-hk8a.onionperf.torproject.net:9143 has been unreachable for more than 15 minutes. playbook: https://gitlab.torproject.org/tpo/tpa/team/-/wikis/service/prometheus#expor… ----- ----- Time: 2025-09-25 15:23:51.784 +0000 UTC Summary: Exporter job "onionperf_metrics_port" on op-us8a-conflux.onionperf.torproject.net:9143 is down Description: Exporter job "onionperf_metrics_port" on op-us8a-conflux.onionperf.torproject.net:9143 has been unreachable for more than 15 minutes. playbook: https://gitlab.torproject.org/tpo/tpa/team/-/wikis/service/prometheus#expor… ----- ----- Time: 2025-09-25 15:24:06.784 +0000 UTC Summary: Exporter job "onionperf_metrics_port" on op-us8a.onionperf.torproject.net:9143 is down Description: Exporter job "onionperf_metrics_port" on op-us8a.onionperf.torproject.net:9143 has been unreachable for more than 15 minutes. playbook: https://gitlab.torproject.org/tpo/tpa/team/-/wikis/service/prometheus#expor… -----

1 0

[FIRING:2] JobDown -- collector
by alertmanager＠prometheus2.torproject.org 25 Sep '25

25 Sep '25

Total firing alerts: 2 ## Firing Alerts ----- Time: 2025-09-25 15:08:36.784 +0000 UTC Summary: Exporter job "collector" on collector.torproject.org:443 is down Description: Exporter job "collector" on collector.torproject.org:443 has been unreachable for more than 15 minutes. playbook: https://gitlab.torproject.org/tpo/tpa/team/-/wikis/service/prometheus#expor… ----- ----- Time: 2025-09-25 15:29:36.784 +0000 UTC Summary: Exporter job "collector" on collector2.torproject.org:443 is down Description: Exporter job "collector" on collector2.torproject.org:443 has been unreachable for more than 15 minutes. playbook: https://gitlab.torproject.org/tpo/tpa/team/-/wikis/service/prometheus#expor… -----

1 0

Cron <collector@collector-02> /srv/collector.torproject.org/collector/monitoring/services/collector
by root＠collector-02.torproject.org 25 Sep '25

25 Sep '25

Error fetching or parsing index from collector.torproject.org: <urlopen error [Errno -3] Temporary failure in name resolution> Could not fetch or parse index.json from collector.torproject.org

1 0

Cron <onionoo@onionoo-backend-03> /srv/onionoo.torproject.org/onionoo/monitoring/network/onionoo
by root＠onionoo-backend-03.torproject.org 25 Sep '25

25 Sep '25

Traceback (most recent call last): File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 198, in _new_conn sock = connection.create_connection( (self._dns_host, self.port), ...<2 lines>... socket_options=self.socket_options, ) File "/usr/lib/python3/dist-packages/urllib3/util/connection.py", line 60, in create_connection for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM): ~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.13/socket.py", line 977, in getaddrinfo for res in _socket.getaddrinfo(host, port, family, type, proto, flags): ~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ socket.gaierror: [Errno -3] Temporary failure in name resolution The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 787, in urlopen response = self._make_request( conn, ...<10 lines>... **response_kw, ) File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 488, in _make_request raise new_e File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 464, in _make_request self._validate_conn(conn) ~~~~~~~~~~~~~~~~~~~^^^^^^ File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 1093, in _validate_conn conn.connect() ~~~~~~~~~~~~^^ File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 704, in connect self.sock = sock = self._new_conn() ~~~~~~~~~~~~~~^^ File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 205, in _new_conn raise NameResolutionError(self.host, self, e) from e urllib3.exceptions.NameResolutionError: <urllib3.connection.HTTPSConnection object at 0x7fd876292f90>: Failed to resolve 'onionoo.torproject.org' ([Errno -3] Temporary failure in name resolution) The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/lib/python3/dist-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( method=request.method, ...<9 lines>... chunked=chunked, ) File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 841, in urlopen retries = retries.increment( method, url, error=new_e, _pool=self, _stacktrace=sys.exc_info()[2] ) File "/usr/lib/python3/dist-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='onionoo.torproject.org', port=443): Max retries exceeded with url: /details (Caused by NameResolutionError("<urllib3.connection.HTTPSConnection object at 0x7fd876292f90>: Failed to resolve 'onionoo.torproject.org' ([Errno -3] Temporary failure in name resolution)")) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/srv/onionoo.torproject.org/onionoo/monitoring/network/onionoo", line 31, in <module> details = json.loads(requests.get("https://onionoo.torproject.org/details").text) ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3/dist-packages/requests/api.py", line 73, in get return request("get", url, params=params, **kwargs) File "/usr/lib/python3/dist-packages/requests/api.py", line 59, in request return session.request(method=method, url=url, **kwargs) ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3/dist-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) File "/usr/lib/python3/dist-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) File "/usr/lib/python3/dist-packages/requests/adapters.py", line 700, in send raise ConnectionError(e, request=request) requests.exceptions.ConnectionError: HTTPSConnectionPool(host='onionoo.torproject.org', port=443): Max retries exceeded with url: /details (Caused by NameResolutionError("<urllib3.connection.HTTPSConnection object at 0x7fd876292f90>: Failed to resolve 'onionoo.torproject.org' ([Errno -3] Temporary failure in name resolution)"))

1 0

Cron <onionoo@onionoo-backend-03> /srv/onionoo.torproject.org/onionoo/monitoring/services/onionoo
by root＠onionoo-backend-03.torproject.org 25 Sep '25

25 Sep '25

Error fetching minimal summary document: <urlopen error [Errno -3] Temporary failure in name resolution>

1 0

Cron <onionoo@onionoo-backend-03> /srv/onionoo.torproject.org/onionoo/monitoring/services/onionoo
by root＠onionoo-backend-03.torproject.org 25 Sep '25

25 Sep '25

Error fetching minimal summary document: <urlopen error [Errno -3] Temporary failure in name resolution>

1 0

Cron <onionoo@onionoo-backend-03> /srv/onionoo.torproject.org/onionoo/monitoring/network/onionoo
by root＠onionoo-backend-03.torproject.org 25 Sep '25

25 Sep '25

Traceback (most recent call last): File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 198, in _new_conn sock = connection.create_connection( (self._dns_host, self.port), ...<2 lines>... socket_options=self.socket_options, ) File "/usr/lib/python3/dist-packages/urllib3/util/connection.py", line 60, in create_connection for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM): ~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.13/socket.py", line 977, in getaddrinfo for res in _socket.getaddrinfo(host, port, family, type, proto, flags): ~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ socket.gaierror: [Errno -3] Temporary failure in name resolution The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 787, in urlopen response = self._make_request( conn, ...<10 lines>... **response_kw, ) File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 488, in _make_request raise new_e File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 464, in _make_request self._validate_conn(conn) ~~~~~~~~~~~~~~~~~~~^^^^^^ File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 1093, in _validate_conn conn.connect() ~~~~~~~~~~~~^^ File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 704, in connect self.sock = sock = self._new_conn() ~~~~~~~~~~~~~~^^ File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 205, in _new_conn raise NameResolutionError(self.host, self, e) from e urllib3.exceptions.NameResolutionError: <urllib3.connection.HTTPSConnection object at 0x7f5237732f90>: Failed to resolve 'onionoo.torproject.org' ([Errno -3] Temporary failure in name resolution) The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/lib/python3/dist-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( method=request.method, ...<9 lines>... chunked=chunked, ) File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 841, in urlopen retries = retries.increment( method, url, error=new_e, _pool=self, _stacktrace=sys.exc_info()[2] ) File "/usr/lib/python3/dist-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='onionoo.torproject.org', port=443): Max retries exceeded with url: /details (Caused by NameResolutionError("<urllib3.connection.HTTPSConnection object at 0x7f5237732f90>: Failed to resolve 'onionoo.torproject.org' ([Errno -3] Temporary failure in name resolution)")) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/srv/onionoo.torproject.org/onionoo/monitoring/network/onionoo", line 31, in <module> details = json.loads(requests.get("https://onionoo.torproject.org/details").text) ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3/dist-packages/requests/api.py", line 73, in get return request("get", url, params=params, **kwargs) File "/usr/lib/python3/dist-packages/requests/api.py", line 59, in request return session.request(method=method, url=url, **kwargs) ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3/dist-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) File "/usr/lib/python3/dist-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) File "/usr/lib/python3/dist-packages/requests/adapters.py", line 700, in send raise ConnectionError(e, request=request) requests.exceptions.ConnectionError: HTTPSConnectionPool(host='onionoo.torproject.org', port=443): Max retries exceeded with url: /details (Caused by NameResolutionError("<urllib3.connection.HTTPSConnection object at 0x7f5237732f90>: Failed to resolve 'onionoo.torproject.org' ([Errno -3] Temporary failure in name resolution)"))

1 0