January 2023 - metrics-alerts - lists.torproject.org

Cron <onionoo@onionoo-backend-02> /srv/onionoo.torproject.org/onionoo/monitoring/network/onionoo
by root＠onionoo-backend-02.torproject.org 25 Jan '23

25 Jan '23

Traceback (most recent call last): File "/srv/onionoo.torproject.org/onionoo/monitoring/network/onionoo", line 28, in <module> details = json.loads(requests.get("https://onionoo.torproject.org/details").text) File "/usr/lib/python3.9/json/__init__.py", line 346, in loads return _default_decoder.decode(s) File "/usr/lib/python3.9/json/decoder.py", line 337, in decode obj, end = self.raw_decode(s, idx=_w(s, 0).end()) File "/usr/lib/python3.9/json/decoder.py", line 355, in raw_decode raise JSONDecodeError("Expecting value", s, err.value) from None json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

1 0

** RECOVERY Service Alert: onionoo-backend-02/network service - onionoo backend is OK **
by nagios＠hetzner-hel1-01.torproject.org 25 Jan '23

25 Jan '23

***** Icinga ***** Notification Type: RECOVERY Service: network service - onionoo backend Host: onionoo-backend-02 Address: 116.202.120.178 State: OK Date/Time: Wed Jan 25 14:34:05 UTC 2023 Additional Info: OK: Valid response with recent timestamps (9915 sec): relays_published=2023-01-25 12:00:00, bridges_published=2023-01-25 11:48:48

1 0

Cron <metrics@meronense> /srv/metrics.torproject.org/metrics/monitoring/network/website
by root＠meronense.torproject.org 25 Jan '23

25 Jan '23

sys:1: DtypeWarning: Columns (3,4) have mixed types.Specify dtype option on import or set low_memory=False.

1 0

Cron <onionoo@onionoo-backend-02> /srv/onionoo.torproject.org/onionoo/monitoring/network/onionoo
by root＠onionoo-backend-02.torproject.org 25 Jan '23

25 Jan '23

Traceback (most recent call last): File "/srv/onionoo.torproject.org/onionoo/monitoring/network/onionoo", line 28, in <module> details = json.loads(requests.get("https://onionoo.torproject.org/details").text) File "/usr/lib/python3.9/json/__init__.py", line 346, in loads return _default_decoder.decode(s) File "/usr/lib/python3.9/json/decoder.py", line 337, in decode obj, end = self.raw_decode(s, idx=_w(s, 0).end()) File "/usr/lib/python3.9/json/decoder.py", line 355, in raw_decode raise JSONDecodeError("Expecting value", s, err.value) from None json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

1 0

Cron <onionoo@onionoo-backend-01> /srv/onionoo.torproject.org/onionoo/monitoring/network/onionoo
by root＠onionoo-backend-01.torproject.org 25 Jan '23

25 Jan '23

Traceback (most recent call last): File "/srv/onionoo.torproject.org/onionoo/monitoring/network/onionoo", line 29, in <module> bandwidth = json.loads(requests.get("https://onionoo.torproject.org/bandwidth").text) File "/usr/lib/python3.9/json/__init__.py", line 346, in loads return _default_decoder.decode(s) File "/usr/lib/python3.9/json/decoder.py", line 337, in decode obj, end = self.raw_decode(s, idx=_w(s, 0).end()) File "/usr/lib/python3.9/json/decoder.py", line 355, in raw_decode raise JSONDecodeError("Expecting value", s, err.value) from None json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

1 0

** PROBLEM Service Alert: onionoo-backend-02/network service - onionoo backend is WARNING **
by nagios＠hetzner-hel1-01.torproject.org 25 Jan '23

25 Jan '23

***** Icinga ***** Notification Type: PROBLEM Service: network service - onionoo backend Host: onionoo-backend-02 Address: 116.202.120.178 State: WARNING Date/Time: Wed Jan 25 14:09:14 UTC 2023 Additional Info: WARNING: Either timestamp is not recent (12028 sec): relays_published=2023-01-25 11:00:00, bridges_published=2023-01-25 10:48:45

1 0

Cron <onionoo@onionoo-backend-02> /srv/onionoo.torproject.org/onionoo/monitoring/network/onionoo
by root＠onionoo-backend-02.torproject.org 25 Jan '23

25 Jan '23

Traceback (most recent call last): File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 699, in urlopen httplib_response = self._make_request( File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 445, in _make_request six.raise_from(e, None) File "<string>", line 3, in raise_from File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 440, in _make_request httplib_response = conn.getresponse() File "/usr/lib/python3.9/http/client.py", line 1347, in getresponse response.begin() File "/usr/lib/python3.9/http/client.py", line 307, in begin version, status, reason = self._read_status() File "/usr/lib/python3.9/http/client.py", line 276, in _read_status raise RemoteDisconnected("Remote end closed connection without" http.client.RemoteDisconnected: Remote end closed connection without response During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/lib/python3/dist-packages/requests/adapters.py", line 439, in send resp = conn.urlopen( File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 755, in urlopen retries = retries.increment( File "/usr/lib/python3/dist-packages/urllib3/util/retry.py", line 532, in increment raise six.reraise(type(error), error, _stacktrace) File "/usr/lib/python3/dist-packages/six.py", line 718, in reraise raise value.with_traceback(tb) File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 699, in urlopen httplib_response = self._make_request( File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 445, in _make_request six.raise_from(e, None) File "<string>", line 3, in raise_from File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 440, in _make_request httplib_response = conn.getresponse() File "/usr/lib/python3.9/http/client.py", line 1347, in getresponse response.begin() File "/usr/lib/python3.9/http/client.py", line 307, in begin version, status, reason = self._read_status() File "/usr/lib/python3.9/http/client.py", line 276, in _read_status raise RemoteDisconnected("Remote end closed connection without" urllib3.exceptions.ProtocolError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/srv/onionoo.torproject.org/onionoo/monitoring/network/onionoo", line 28, in <module> details = json.loads(requests.get("https://onionoo.torproject.org/details").text) File "/usr/lib/python3/dist-packages/requests/api.py", line 76, in get return request('get', url, params=params, **kwargs) File "/usr/lib/python3/dist-packages/requests/api.py", line 61, in request return session.request(method=method, url=url, **kwargs) File "/usr/lib/python3/dist-packages/requests/sessions.py", line 542, in request resp = self.send(prep, **send_kwargs) File "/usr/lib/python3/dist-packages/requests/sessions.py", line 655, in send r = adapter.send(request, **kwargs) File "/usr/lib/python3/dist-packages/requests/adapters.py", line 498, in send raise ConnectionError(err, request=request) requests.exceptions.ConnectionError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))

1 0

Cron <onionoo@onionoo-backend-01> /srv/onionoo.torproject.org/onionoo/monitoring/network/onionoo
by root＠onionoo-backend-01.torproject.org 25 Jan '23

25 Jan '23

Traceback (most recent call last): File "/srv/onionoo.torproject.org/onionoo/monitoring/network/onionoo", line 28, in <module> details = json.loads(requests.get("https://onionoo.torproject.org/details").text) File "/usr/lib/python3.9/json/__init__.py", line 346, in loads return _default_decoder.decode(s) File "/usr/lib/python3.9/json/decoder.py", line 337, in decode obj, end = self.raw_decode(s, idx=_w(s, 0).end()) File "/usr/lib/python3.9/json/decoder.py", line 353, in raw_decode obj, end = self.scan_once(s, idx) json.decoder.JSONDecodeError: Unterminated string starting at: line 682 column 98 (char 1525320)

1 0

** RECOVERY Service Alert: onionoo-frontend-02/network service - onionoo varnish is OK **
by nagios＠hetzner-hel1-01.torproject.org 25 Jan '23

25 Jan '23

***** Icinga ***** Notification Type: RECOVERY Service: network service - onionoo varnish Host: onionoo-frontend-02 Address: 49.12.57.137 State: OK Date/Time: Wed Jan 25 10:24:14 UTC 2023 Additional Info: OK: Valid response with recent timestamps (5728 sec): relays_published=2023-01-25 09:00:00, bridges_published=2023-01-25 08:48:45

1 0

** RECOVERY Service Alert: onionoo-backend-01/network service - onionoo varnish is OK **
by nagios＠hetzner-hel1-01.torproject.org 25 Jan '23

25 Jan '23

***** Icinga ***** Notification Type: RECOVERY Service: network service - onionoo varnish Host: onionoo-backend-01 Address: 116.202.120.163 State: OK Date/Time: Wed Jan 25 10:22:14 UTC 2023 Additional Info: OK: Valid response with recent timestamps (5608 sec): relays_published=2023-01-25 09:00:00, bridges_published=2023-01-25 08:48:45

1 0