January 2023 - metrics-alerts - lists.torproject.org

** PROBLEM Service Alert: onionoo-frontend-01/network service - onionoo full service check is WARNING **
by nagios＠hetzner-hel1-01.torproject.org 25 Jan '23

25 Jan '23

***** Icinga ***** Notification Type: PROBLEM Service: network service - onionoo full service check Host: onionoo-frontend-01 Address: 116.202.120.171 State: WARNING Date/Time: Wed Jan 25 00:09:14 UTC 2023 Additional Info: WARNING: Either timestamp is not recent (12028 sec): relays_published=2023-01-24 22:00:00, bridges_published=2023-01-24 20:48:45

1 0

** PROBLEM Service Alert: onionoo-frontend-02/network service - onionoo varnish is WARNING **
by nagios＠hetzner-hel1-01.torproject.org 24 Jan '23

24 Jan '23

***** Icinga ***** Notification Type: PROBLEM Service: network service - onionoo varnish Host: onionoo-frontend-02 Address: 49.12.57.137 State: WARNING Date/Time: Tue Jan 24 23:59:14 UTC 2023 Additional Info: WARNING: Either timestamp is not recent (15028 sec): relays_published=2023-01-24 20:00:00, bridges_published=2023-01-24 19:48:45

1 0

Cron <onionoo@onionoo-backend-01> /srv/onionoo.torproject.org/onionoo/monitoring/network/onionoo
by root＠onionoo-backend-01.torproject.org 24 Jan '23

24 Jan '23

Traceback (most recent call last): File "/usr/lib/python3/dist-packages/urllib3/response.py", line 697, in _update_chunk_length self.chunk_left = int(line, 16) ValueError: invalid literal for int() with base 16: b'' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/lib/python3/dist-packages/urllib3/response.py", line 438, in _error_catcher yield File "/usr/lib/python3/dist-packages/urllib3/response.py", line 764, in read_chunked self._update_chunk_length() File "/usr/lib/python3/dist-packages/urllib3/response.py", line 701, in _update_chunk_length raise InvalidChunkLength(self, line) urllib3.exceptions.InvalidChunkLength: InvalidChunkLength(got length b'', 0 bytes read) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/lib/python3/dist-packages/requests/models.py", line 753, in generate for chunk in self.raw.stream(chunk_size, decode_content=True): File "/usr/lib/python3/dist-packages/urllib3/response.py", line 572, in stream for line in self.read_chunked(amt, decode_content=decode_content): File "/usr/lib/python3/dist-packages/urllib3/response.py", line 793, in read_chunked self._original_response.close() File "/usr/lib/python3.9/contextlib.py", line 135, in __exit__ self.gen.throw(type, value, traceback) File "/usr/lib/python3/dist-packages/urllib3/response.py", line 455, in _error_catcher raise ProtocolError("Connection broken: %r" % e, e) urllib3.exceptions.ProtocolError: ("Connection broken: InvalidChunkLength(got length b'', 0 bytes read)", InvalidChunkLength(got length b'', 0 bytes read)) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/srv/onionoo.torproject.org/onionoo/monitoring/network/onionoo", line 29, in <module> bandwidth = json.loads(requests.get("https://onionoo.torproject.org/bandwidth").text) File "/usr/lib/python3/dist-packages/requests/api.py", line 76, in get return request('get', url, params=params, **kwargs) File "/usr/lib/python3/dist-packages/requests/api.py", line 61, in request return session.request(method=method, url=url, **kwargs) File "/usr/lib/python3/dist-packages/requests/sessions.py", line 542, in request resp = self.send(prep, **send_kwargs) File "/usr/lib/python3/dist-packages/requests/sessions.py", line 697, in send r.content File "/usr/lib/python3/dist-packages/requests/models.py", line 831, in content self._content = b''.join(self.iter_content(CONTENT_CHUNK_SIZE)) or b'' File "/usr/lib/python3/dist-packages/requests/models.py", line 756, in generate raise ChunkedEncodingError(e) requests.exceptions.ChunkedEncodingError: ("Connection broken: InvalidChunkLength(got length b'', 0 bytes read)", InvalidChunkLength(got length b'', 0 bytes read))

1 0

Cron <onionoo@onionoo-backend-02> /srv/onionoo.torproject.org/onionoo/monitoring/network/onionoo
by root＠onionoo-backend-02.torproject.org 24 Jan '23

24 Jan '23

Traceback (most recent call last): File "/usr/lib/python3/dist-packages/urllib3/response.py", line 697, in _update_chunk_length self.chunk_left = int(line, 16) ValueError: invalid literal for int() with base 16: b'' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/lib/python3/dist-packages/urllib3/response.py", line 438, in _error_catcher yield File "/usr/lib/python3/dist-packages/urllib3/response.py", line 764, in read_chunked self._update_chunk_length() File "/usr/lib/python3/dist-packages/urllib3/response.py", line 701, in _update_chunk_length raise InvalidChunkLength(self, line) urllib3.exceptions.InvalidChunkLength: InvalidChunkLength(got length b'', 0 bytes read) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/lib/python3/dist-packages/requests/models.py", line 753, in generate for chunk in self.raw.stream(chunk_size, decode_content=True): File "/usr/lib/python3/dist-packages/urllib3/response.py", line 572, in stream for line in self.read_chunked(amt, decode_content=decode_content): File "/usr/lib/python3/dist-packages/urllib3/response.py", line 793, in read_chunked self._original_response.close() File "/usr/lib/python3.9/contextlib.py", line 135, in __exit__ self.gen.throw(type, value, traceback) File "/usr/lib/python3/dist-packages/urllib3/response.py", line 455, in _error_catcher raise ProtocolError("Connection broken: %r" % e, e) urllib3.exceptions.ProtocolError: ("Connection broken: InvalidChunkLength(got length b'', 0 bytes read)", InvalidChunkLength(got length b'', 0 bytes read)) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/srv/onionoo.torproject.org/onionoo/monitoring/network/onionoo", line 29, in <module> bandwidth = json.loads(requests.get("https://onionoo.torproject.org/bandwidth").text) File "/usr/lib/python3/dist-packages/requests/api.py", line 76, in get return request('get', url, params=params, **kwargs) File "/usr/lib/python3/dist-packages/requests/api.py", line 61, in request return session.request(method=method, url=url, **kwargs) File "/usr/lib/python3/dist-packages/requests/sessions.py", line 542, in request resp = self.send(prep, **send_kwargs) File "/usr/lib/python3/dist-packages/requests/sessions.py", line 697, in send r.content File "/usr/lib/python3/dist-packages/requests/models.py", line 831, in content self._content = b''.join(self.iter_content(CONTENT_CHUNK_SIZE)) or b'' File "/usr/lib/python3/dist-packages/requests/models.py", line 756, in generate raise ChunkedEncodingError(e) requests.exceptions.ChunkedEncodingError: ("Connection broken: InvalidChunkLength(got length b'', 0 bytes read)", InvalidChunkLength(got length b'', 0 bytes read))

1 0

** PROBLEM Service Alert: global/collector is WARNING **
by nagios＠hetzner-hel1-01.torproject.org 24 Jan '23

24 Jan '23

***** Icinga ***** Notification Type: PROBLEM Service: collector Host: global Address: global State: WARNING Date/Time: Tue Jan 24 23:22:14 UTC 2023 Additional Info: WARNING: Got a valid response from https://collector.torproject.org/index/index.json with at least one timestamp being outdated: /recent/exit-lists/ = 2023-01-24 20:02 (3 = 3 hours ago)

1 0

** PROBLEM Service Alert: onionoo-backend-02/network service - onionoo varnish is CRITICAL **
by nagios＠hetzner-hel1-01.torproject.org 24 Jan '23

24 Jan '23

***** Icinga ***** Notification Type: PROBLEM Service: network service - onionoo varnish Host: onionoo-backend-02 Address: 116.202.120.178 State: CRITICAL Date/Time: Tue Jan 24 23:15:04 UTC 2023 Additional Info: CHECK_NRPE STATE CRITICAL: Socket timeout after 50 seconds.

1 0

** PROBLEM Service Alert: onionoo-frontend-01/network service - onionoo varnish is WARNING **
by nagios＠hetzner-hel1-01.torproject.org 24 Jan '23

24 Jan '23

***** Icinga ***** Notification Type: PROBLEM Service: network service - onionoo varnish Host: onionoo-frontend-01 Address: 116.202.120.171 State: WARNING Date/Time: Tue Jan 24 23:09:14 UTC 2023 Additional Info: WARNING: Either timestamp is not recent (15629 sec): relays_published=2023-01-24 19:00:00, bridges_published=2023-01-24 18:48:44

1 0

** PROBLEM Service Alert: onionoo-backend-02/network service - onionoo varnish is WARNING **
by nagios＠hetzner-hel1-01.torproject.org 24 Jan '23

24 Jan '23

***** Icinga ***** Notification Type: PROBLEM Service: network service - onionoo varnish Host: onionoo-backend-02 Address: 116.202.120.178 State: WARNING Date/Time: Tue Jan 24 22:59:24 UTC 2023 Additional Info: WARNING: Either timestamp is not recent (15030 sec): relays_published=2023-01-24 19:00:00, bridges_published=2023-01-24 18:48:44

1 0

** PROBLEM Service Alert: onionoo-backend-02/network service - onionoo backend is WARNING **
by nagios＠hetzner-hel1-01.torproject.org 24 Jan '23

24 Jan '23

***** Icinga ***** Notification Type: PROBLEM Service: network service - onionoo backend Host: onionoo-backend-02 Address: 116.202.120.178 State: WARNING Date/Time: Tue Jan 24 22:44:14 UTC 2023 Additional Info: WARNING: Either timestamp is not recent (14129 sec): relays_published=2023-01-24 19:00:00, bridges_published=2023-01-24 18:48:44

1 0

** PROBLEM Service Alert: onionoo-backend-02/network service - onionoo varnish is CRITICAL **
by nagios＠hetzner-hel1-01.torproject.org 24 Jan '23

24 Jan '23

***** Icinga ***** Notification Type: PROBLEM Service: network service - onionoo varnish Host: onionoo-backend-02 Address: 116.202.120.178 State: CRITICAL Date/Time: Tue Jan 24 22:00:14 UTC 2023 Additional Info: CHECK_NRPE STATE CRITICAL: Socket timeout after 50 seconds.

1 0