- metrics-alerts - lists.torproject.org

** PROBLEM Service Alert: onionoo-backend-01/network service - onionoo varnish is WARNING **
by nagios＠hetzner-hel1-01.torproject.org 21 Jul '22

21 Jul '22

***** Icinga ***** Notification Type: PROBLEM Service: network service - onionoo varnish Host: onionoo-backend-01 Address: 116.202.120.163 State: WARNING Date/Time: Wed Jul 20 23:11:48 UTC 2022 Additional Info: WARNING: Either timestamp is not recent (11504 sec): relays_published=2022-07-20 20:00:00, bridges_published=2022-07-20 21:37:36

1 0

Cron <onionoo@onionoo-backend-02> /srv/onionoo.torproject.org/onionoo/monitoring/network/onionoo
by root＠onionoo-backend-02.torproject.org 21 Jul '22

21 Jul '22

Traceback (most recent call last): File "/srv/onionoo.torproject.org/onionoo/monitoring/network/onionoo", line 28, in <module> details = json.loads(requests.get("https://onionoo.torproject.org/details").text) File "/usr/lib/python3.9/json/__init__.py", line 346, in loads return _default_decoder.decode(s) File "/usr/lib/python3.9/json/decoder.py", line 337, in decode obj, end = self.raw_decode(s, idx=_w(s, 0).end()) File "/usr/lib/python3.9/json/decoder.py", line 355, in raw_decode raise JSONDecodeError("Expecting value", s, err.value) from None json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

1 0

Cron <onionoo@onionoo-backend-01> /srv/onionoo.torproject.org/onionoo/monitoring/network/onionoo
by root＠onionoo-backend-01.torproject.org 20 Jul '22

20 Jul '22

Traceback (most recent call last): File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 699, in urlopen httplib_response = self._make_request( File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 382, in _make_request self._validate_conn(conn) File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 1012, in _validate_conn conn.connect() File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 411, in connect self.sock = ssl_wrap_socket( File "/usr/lib/python3/dist-packages/urllib3/util/ssl_.py", line 449, in ssl_wrap_socket ssl_sock = _ssl_wrap_socket_impl( File "/usr/lib/python3/dist-packages/urllib3/util/ssl_.py", line 493, in _ssl_wrap_socket_impl return ssl_context.wrap_socket(sock, server_hostname=server_hostname) File "/usr/lib/python3.9/ssl.py", line 500, in wrap_socket return self.sslsocket_class._create( File "/usr/lib/python3.9/ssl.py", line 1040, in _create self.do_handshake() File "/usr/lib/python3.9/ssl.py", line 1309, in do_handshake self._sslobj.do_handshake() ConnectionResetError: [Errno 104] Connection reset by peer During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/lib/python3/dist-packages/requests/adapters.py", line 439, in send resp = conn.urlopen( File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 755, in urlopen retries = retries.increment( File "/usr/lib/python3/dist-packages/urllib3/util/retry.py", line 532, in increment raise six.reraise(type(error), error, _stacktrace) File "/usr/lib/python3/dist-packages/six.py", line 718, in reraise raise value.with_traceback(tb) File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 699, in urlopen httplib_response = self._make_request( File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 382, in _make_request self._validate_conn(conn) File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 1012, in _validate_conn conn.connect() File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 411, in connect self.sock = ssl_wrap_socket( File "/usr/lib/python3/dist-packages/urllib3/util/ssl_.py", line 449, in ssl_wrap_socket ssl_sock = _ssl_wrap_socket_impl( File "/usr/lib/python3/dist-packages/urllib3/util/ssl_.py", line 493, in _ssl_wrap_socket_impl return ssl_context.wrap_socket(sock, server_hostname=server_hostname) File "/usr/lib/python3.9/ssl.py", line 500, in wrap_socket return self.sslsocket_class._create( File "/usr/lib/python3.9/ssl.py", line 1040, in _create self.do_handshake() File "/usr/lib/python3.9/ssl.py", line 1309, in do_handshake self._sslobj.do_handshake() urllib3.exceptions.ProtocolError: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/srv/onionoo.torproject.org/onionoo/monitoring/network/onionoo", line 28, in <module> details = json.loads(requests.get("https://onionoo.torproject.org/details").text) File "/usr/lib/python3/dist-packages/requests/api.py", line 76, in get return request('get', url, params=params, **kwargs) File "/usr/lib/python3/dist-packages/requests/api.py", line 61, in request return session.request(method=method, url=url, **kwargs) File "/usr/lib/python3/dist-packages/requests/sessions.py", line 542, in request resp = self.send(prep, **send_kwargs) File "/usr/lib/python3/dist-packages/requests/sessions.py", line 655, in send r = adapter.send(request, **kwargs) File "/usr/lib/python3/dist-packages/requests/adapters.py", line 498, in send raise ConnectionError(err, request=request) requests.exceptions.ConnectionError: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))

1 0

Cron <onionoo@onionoo-backend-01> /srv/onionoo.torproject.org/onionoo/monitoring/network/onionoo
by root＠onionoo-backend-01.torproject.org 20 Jul '22

20 Jul '22

Traceback (most recent call last): File "/srv/onionoo.torproject.org/onionoo/monitoring/network/onionoo", line 29, in <module> bandwidth = json.loads(requests.get("https://onionoo.torproject.org/bandwidth").text) File "/usr/lib/python3.9/json/__init__.py", line 346, in loads return _default_decoder.decode(s) File "/usr/lib/python3.9/json/decoder.py", line 337, in decode obj, end = self.raw_decode(s, idx=_w(s, 0).end()) File "/usr/lib/python3.9/json/decoder.py", line 355, in raw_decode raise JSONDecodeError("Expecting value", s, err.value) from None json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

1 0

Cron <onionoo@onionoo-backend-02> /srv/onionoo.torproject.org/onionoo/monitoring/network/onionoo
by root＠onionoo-backend-02.torproject.org 20 Jul '22

20 Jul '22

Traceback (most recent call last): File "/srv/onionoo.torproject.org/onionoo/monitoring/network/onionoo", line 28, in <module> details = json.loads(requests.get("https://onionoo.torproject.org/details").text) File "/usr/lib/python3.9/json/__init__.py", line 346, in loads return _default_decoder.decode(s) File "/usr/lib/python3.9/json/decoder.py", line 337, in decode obj, end = self.raw_decode(s, idx=_w(s, 0).end()) File "/usr/lib/python3.9/json/decoder.py", line 355, in raw_decode raise JSONDecodeError("Expecting value", s, err.value) from None json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

1 0

Cron <onionoo@onionoo-backend-01> /srv/onionoo.torproject.org/onionoo/monitoring/network/onionoo
by root＠onionoo-backend-01.torproject.org 20 Jul '22

20 Jul '22

Traceback (most recent call last): File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 169, in _new_conn conn = connection.create_connection( File "/usr/lib/python3/dist-packages/urllib3/util/connection.py", line 96, in create_connection raise err File "/usr/lib/python3/dist-packages/urllib3/util/connection.py", line 86, in create_connection sock.connect(sa) TimeoutError: [Errno 110] Connection timed out During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 699, in urlopen httplib_response = self._make_request( File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 382, in _make_request self._validate_conn(conn) File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 1012, in _validate_conn conn.connect() File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 353, in connect conn = self._new_conn() File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 181, in _new_conn raise NewConnectionError( urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPSConnection object at 0x7fe7f739ad00>: Failed to establish a new connection: [Errno 110] Connection timed out During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/lib/python3/dist-packages/requests/adapters.py", line 439, in send resp = conn.urlopen( File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 755, in urlopen retries = retries.increment( File "/usr/lib/python3/dist-packages/urllib3/util/retry.py", line 574, in increment raise MaxRetryError(_pool, url, error or ResponseError(cause)) urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='onionoo.torproject.org', port=443): Max retries exceeded with url: /details (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7fe7f739ad00>: Failed to establish a new connection: [Errno 110] Connection timed out')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/srv/onionoo.torproject.org/onionoo/monitoring/network/onionoo", line 28, in <module> details = json.loads(requests.get("https://onionoo.torproject.org/details").text) File "/usr/lib/python3/dist-packages/requests/api.py", line 76, in get return request('get', url, params=params, **kwargs) File "/usr/lib/python3/dist-packages/requests/api.py", line 61, in request return session.request(method=method, url=url, **kwargs) File "/usr/lib/python3/dist-packages/requests/sessions.py", line 542, in request resp = self.send(prep, **send_kwargs) File "/usr/lib/python3/dist-packages/requests/sessions.py", line 655, in send r = adapter.send(request, **kwargs) File "/usr/lib/python3/dist-packages/requests/adapters.py", line 516, in send raise ConnectionError(e, request=request) requests.exceptions.ConnectionError: HTTPSConnectionPool(host='onionoo.torproject.org', port=443): Max retries exceeded with url: /details (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7fe7f739ad00>: Failed to establish a new connection: [Errno 110] Connection timed out'))

1 0

Cron <onionoo@onionoo-backend-01> /srv/onionoo.torproject.org/onionoo/monitoring/network/onionoo
by root＠onionoo-backend-01.torproject.org 20 Jul '22

20 Jul '22

Traceback (most recent call last): File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 169, in _new_conn conn = connection.create_connection( File "/usr/lib/python3/dist-packages/urllib3/util/connection.py", line 96, in create_connection raise err File "/usr/lib/python3/dist-packages/urllib3/util/connection.py", line 86, in create_connection sock.connect(sa) TimeoutError: [Errno 110] Connection timed out During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 699, in urlopen httplib_response = self._make_request( File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 382, in _make_request self._validate_conn(conn) File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 1012, in _validate_conn conn.connect() File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 353, in connect conn = self._new_conn() File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 181, in _new_conn raise NewConnectionError( urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPSConnection object at 0x7fbe8dab0d00>: Failed to establish a new connection: [Errno 110] Connection timed out During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/lib/python3/dist-packages/requests/adapters.py", line 439, in send resp = conn.urlopen( File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 755, in urlopen retries = retries.increment( File "/usr/lib/python3/dist-packages/urllib3/util/retry.py", line 574, in increment raise MaxRetryError(_pool, url, error or ResponseError(cause)) urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='onionoo.torproject.org', port=443): Max retries exceeded with url: /details (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7fbe8dab0d00>: Failed to establish a new connection: [Errno 110] Connection timed out')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/srv/onionoo.torproject.org/onionoo/monitoring/network/onionoo", line 28, in <module> details = json.loads(requests.get("https://onionoo.torproject.org/details").text) File "/usr/lib/python3/dist-packages/requests/api.py", line 76, in get return request('get', url, params=params, **kwargs) File "/usr/lib/python3/dist-packages/requests/api.py", line 61, in request return session.request(method=method, url=url, **kwargs) File "/usr/lib/python3/dist-packages/requests/sessions.py", line 542, in request resp = self.send(prep, **send_kwargs) File "/usr/lib/python3/dist-packages/requests/sessions.py", line 655, in send r = adapter.send(request, **kwargs) File "/usr/lib/python3/dist-packages/requests/adapters.py", line 516, in send raise ConnectionError(e, request=request) requests.exceptions.ConnectionError: HTTPSConnectionPool(host='onionoo.torproject.org', port=443): Max retries exceeded with url: /details (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7fbe8dab0d00>: Failed to establish a new connection: [Errno 110] Connection timed out'))

1 0

Cron <exonerator@materculae> cd /srv/exonerator.torproject.org/exonerator/ && ~exonerator/bin/run-exonerator
by root＠materculae.torproject.org 20 Jul '22

20 Jul '22

2022-07-20 18:22:02,224 ERROR o.t.m.e.ExoneraTorDatabaseImporter:167 Lock file /srv/exonerator.torproject.org/exonerator/exonerator-lock is less than 6 hours old. Either make sure that there are no other ExoneraTor database importers running and manually delete that file, or wait until the file is 6 hours old when it will be overwritten automatically. Exiting.

1 0

Cron <exonerator@materculae> cd /srv/exonerator.torproject.org/exonerator/ && ~exonerator/bin/run-exonerator
by root＠materculae.torproject.org 20 Jul '22

20 Jul '22

2022-07-20 11:22:02,293 WARN o.t.m.e.ExoneraTorDatabaseImporter:174 Lock file /srv/exonerator.torproject.org/exonerator/exonerator-lock is at least 6 hours old. Overwriting and continuing with the database import.

1 0

Cron <exonerator@materculae> cd /srv/exonerator.torproject.org/exonerator/ && ~exonerator/bin/run-exonerator
by root＠materculae.torproject.org 20 Jul '22

20 Jul '22

2022-07-20 16:22:02,117 ERROR o.t.m.e.ExoneraTorDatabaseImporter:167 Lock file /srv/exonerator.torproject.org/exonerator/exonerator-lock is less than 6 hours old. Either make sure that there are no other ExoneraTor database importers running and manually delete that file, or wait until the file is 6 hours old when it will be overwritten automatically. Exiting.

1 0