October 2023 - metrics-alerts - lists.torproject.org

Cron <onionoo@onionoo-backend-02> /srv/onionoo.torproject.org/onionoo/monitoring/network/onionoo
by root＠onionoo-backend-02.torproject.org 01 Oct '23

01 Oct '23

Traceback (most recent call last): File "/srv/onionoo.torproject.org/onionoo/monitoring/network/onionoo", line 29, in <module> bandwidth = json.loads(requests.get("https://onionoo.torproject.org/bandwidth").text) File "/usr/lib/python3.9/json/__init__.py", line 346, in loads return _default_decoder.decode(s) File "/usr/lib/python3.9/json/decoder.py", line 337, in decode obj, end = self.raw_decode(s, idx=_w(s, 0).end()) File "/usr/lib/python3.9/json/decoder.py", line 355, in raw_decode raise JSONDecodeError("Expecting value", s, err.value) from None json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

1 0

** PROBLEM Service Alert: onionoo-backend-02/network service - onionoo backend is WARNING **
by nagios＠hetzner-hel1-01.torproject.org 01 Oct '23

01 Oct '23

***** Icinga ***** Notification Type: PROBLEM Service: network service - onionoo backend Host: onionoo-backend-02 Address: 116.202.120.178 State: WARNING Date/Time: Sun Oct 1 08:17:32 UTC 2023 Additional Info: WARNING: Either timestamp is not recent (11842 sec): relays_published=2023-10-01 05:00:00, bridges_published=2023-10-01 07:03:13

1 0

** PROBLEM Service Alert: onionoo-backend-02/network service - onionoo varnish is WARNING **
by nagios＠hetzner-hel1-01.torproject.org 01 Oct '23

01 Oct '23

***** Icinga ***** Notification Type: PROBLEM Service: network service - onionoo varnish Host: onionoo-backend-02 Address: 116.202.120.178 State: WARNING Date/Time: Sun Oct 1 08:16:12 UTC 2023 Additional Info: WARNING: Either timestamp is not recent (11769 sec): relays_published=2023-10-01 05:00:00, bridges_published=2023-10-01 07:03:13

1 0

** RECOVERY Service Alert: global/collector is OK **
by nagios＠hetzner-hel1-01.torproject.org 01 Oct '23

01 Oct '23

***** Icinga ***** Notification Type: RECOVERY Service: collector Host: global Address: global State: OK Date/Time: Sun Oct 1 08:11:52 UTC 2023 Additional Info: OK: Got a valid response from https://collector.torproject.org/index/index.json with recent timestamps: /archive/bridgedb-metrics/ = 2023-09-30 06:24 (25 72 hours ago), /archive/bridge-descriptors/extra-infos/ = 2023-09-30 06:42 (25 72 hours ago), /archive/bridge-descriptors/server-descriptors/ = 2023-09-30 06:23 (25 72 hours ago), /archive/bridge-descriptors/statuses/ = 2023-09-30 05:52 (26 72 hours ago), /archive/bridge-pool-assignments/ = 2023-09-30 06:24 (25 72 hours ago), /archive/exit-lists/ = 2023-09-30 03:28 (28 72 hours ago), /archive/onionperf/ = 2023-09-30 03:28 (28 72 hours ago), /archive/relay-descriptors/ = 2023-09-30 03:28 (28 720 hours ago), /archive/relay-descriptors/bandwidths/ = 2023-09-30 05:01 (27 72 hours ago), /archive/relay-descriptors/consensuses/ = 2023-09-30 03:34 (28 72 hours ago), /archive/relay-descriptors/extra-infos/ = 2023-09-30 06:23 (25 72 hours ago), /archive/relay-descriptors/microdescs/ = 2023-09-30 03:34 (28 72 hours ago), /archive/r elay-descriptors/server-descriptors/ = 2023-09-30 05:48 (26 72 hours ago), /archive/relay-descriptors/votes/ = 2023-09-30 05:46 (26 72 hours ago), /archive/snowflakes/ = 2023-09-30 06:23 (25 72 hours ago), /archive/webstats/ = 2023-09-30 06:42 (25 72 hours ago), /recent/bridgedb-metrics/ = 2023-09-30 21:40 (10 60 hours ago), /recent/bridge-descriptors/extra-infos/ = 2023-10-01 07:09 (1 3 hours ago), /recent/bridge-descriptors/server-descriptors/ = 2023-10-01 07:09 (1 3 hours ago), /recent/bridge-descriptors/statuses/ = 2023-10-01 07:09 (1 3 hours ago), /recent/bridge-pool-assignments/ = 2023-10-01 00:09 (8 30 hours ago), /recent/exit-lists/ = 2023-10-01 08:02 (0 3 hours ago), /recent/onionperf/ = 2023-10-01 02:10 (6 30 hours ago), /recent/relay-descriptors/bandwidths/ = 2023-10-01 08:06 (0 3 hours ago), /recent/relay-descriptors/consensuses/ = 2023-10-01 08:06 (0 3 hours ago), /recent/relay-descriptors/extra-infos/ = 2023-10-01 08:05 (0 3 hours ago), /recent/relay-desc riptors/microdescs/consensus-microdesc/ = 2023-10-01 08:05 (0 3 hours ago), /recent/relay-descriptors/microdescs/micro/ = 2023-10-01 08:06 (0 3 hours ago), /recent/relay-descriptors/server-descriptors/ = 2023-10-01 08:06 (0 3 hours ago), /recent/relay-descriptors/votes/ = 2023-10-01 08:05 (0 3 hours ago), /recent/snowflakes/ = 2023-09-30 17:40 (14 30 hours ago), /recent/webstats/ = 2023-10-01 07:42 (0 30 hours ago)

1 0

[FIRING:1] Onionperf disk space
by alertmanager＠hetzner-nbg1-02.torproject.org 01 Oct '23

01 Oct '23

Total firing alerts: 1 Total resolved alerts: 0 ## Firing Alerts ----- Time: 2023-10-01 08:11:12.544 +0000 UTC Summary: op-hk7a.onionperf.torproject.net:9100 is using 95.03170445974771% of disk space available. Description: op-hk7a.onionperf.torproject.net:9100 disk space used is more than 95% ----- ##Resolved Alerts

1 0

Cron <onionoo@onionoo-backend-02> /srv/onionoo.torproject.org/onionoo/monitoring/network/onionoo
by root＠onionoo-backend-02.torproject.org 01 Oct '23

01 Oct '23

Traceback (most recent call last): File "/srv/onionoo.torproject.org/onionoo/monitoring/network/onionoo", line 28, in <module> details = json.loads(requests.get("https://onionoo.torproject.org/details").text) File "/usr/lib/python3.9/json/__init__.py", line 346, in loads return _default_decoder.decode(s) File "/usr/lib/python3.9/json/decoder.py", line 337, in decode obj, end = self.raw_decode(s, idx=_w(s, 0).end()) File "/usr/lib/python3.9/json/decoder.py", line 355, in raw_decode raise JSONDecodeError("Expecting value", s, err.value) from None json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

1 0

Cron <onionoo@onionoo-backend-02> /srv/onionoo.torproject.org/onionoo/monitoring/network/onionoo
by root＠onionoo-backend-02.torproject.org 01 Oct '23

01 Oct '23

Traceback (most recent call last): File "/srv/onionoo.torproject.org/onionoo/monitoring/network/onionoo", line 29, in <module> bandwidth = json.loads(requests.get("https://onionoo.torproject.org/bandwidth").text) File "/usr/lib/python3.9/json/__init__.py", line 346, in loads return _default_decoder.decode(s) File "/usr/lib/python3.9/json/decoder.py", line 337, in decode obj, end = self.raw_decode(s, idx=_w(s, 0).end()) File "/usr/lib/python3.9/json/decoder.py", line 355, in raw_decode raise JSONDecodeError("Expecting value", s, err.value) from None json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

1 0

Cron <onionoo@onionoo-backend-01> /srv/onionoo.torproject.org/onionoo/monitoring/network/onionoo
by root＠onionoo-backend-01.torproject.org 01 Oct '23

01 Oct '23

Traceback (most recent call last): File "/srv/onionoo.torproject.org/onionoo/monitoring/network/onionoo", line 29, in <module> bandwidth = json.loads(requests.get("https://onionoo.torproject.org/bandwidth").text) File "/usr/lib/python3.9/json/__init__.py", line 346, in loads return _default_decoder.decode(s) File "/usr/lib/python3.9/json/decoder.py", line 337, in decode obj, end = self.raw_decode(s, idx=_w(s, 0).end()) File "/usr/lib/python3.9/json/decoder.py", line 355, in raw_decode raise JSONDecodeError("Expecting value", s, err.value) from None json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

1 0

Cron <onionoo@onionoo-backend-01> /srv/onionoo.torproject.org/onionoo/monitoring/services/onionoo
by root＠onionoo-backend-01.torproject.org 01 Oct '23

01 Oct '23

Traceback (most recent call last): File "/srv/onionoo.torproject.org/onionoo/monitoring/services/onionoo", line 38, in <module> write_to_textfile("/srv/onionoo.torproject.org/onionoo/service/metrics", registry=registry) File "/usr/lib/python3/dist-packages/prometheus_client/exposition.py", line 202, in write_to_textfile with open(tmppath, 'wb') as f: OSError: [Errno 28] No space left on device: '/srv/onionoo.torproject.org/onionoo/service/metrics.3013047.140236624144192'

1 0

Cron <onionoo@onionoo-backend-01> /srv/onionoo.torproject.org/onionoo/monitoring/network/onionoo
by root＠onionoo-backend-01.torproject.org 01 Oct '23

01 Oct '23

Traceback (most recent call last): File "/srv/onionoo.torproject.org/onionoo/monitoring/network/onionoo", line 678, in <module> write_to_textfile('/srv/onionoo.torproject.org/onionoo/network/metrics', registry) File "/usr/lib/python3/dist-packages/prometheus_client/exposition.py", line 202, in write_to_textfile with open(tmppath, 'wb') as f: OSError: [Errno 28] No space left on device: '/srv/onionoo.torproject.org/onionoo/network/metrics.3012828.139855634839360'

1 0