October 2023 - metrics-alerts - lists.torproject.org

Cron <onionoo@onionoo-backend-01> /srv/onionoo.torproject.org/onionoo/monitoring/network/onionoo
by root＠onionoo-backend-01.torproject.org 02 Oct '23

02 Oct '23

Traceback (most recent call last): File "/srv/onionoo.torproject.org/onionoo/monitoring/network/onionoo", line 678, in <module> write_to_textfile('/srv/onionoo.torproject.org/onionoo/network/metrics', registry) File "/usr/lib/python3/dist-packages/prometheus_client/exposition.py", line 202, in write_to_textfile with open(tmppath, 'wb') as f: OSError: [Errno 28] No space left on device: '/srv/onionoo.torproject.org/onionoo/network/metrics.3087563.140302584768320'

1 0

Cron <onionoo@onionoo-backend-01> /srv/onionoo.torproject.org/onionoo/monitoring/services/onionoo
by root＠onionoo-backend-01.torproject.org 02 Oct '23

02 Oct '23

Traceback (most recent call last): File "/srv/onionoo.torproject.org/onionoo/monitoring/services/onionoo", line 38, in <module> write_to_textfile("/srv/onionoo.torproject.org/onionoo/service/metrics", registry=registry) File "/usr/lib/python3/dist-packages/prometheus_client/exposition.py", line 202, in write_to_textfile with open(tmppath, 'wb') as f: OSError: [Errno 28] No space left on device: '/srv/onionoo.torproject.org/onionoo/service/metrics.3087567.140048183326528'

1 0

Cron <onionoo@onionoo-backend-01> /srv/onionoo.torproject.org/onionoo/monitoring/network/onionoo
by root＠onionoo-backend-01.torproject.org 02 Oct '23

02 Oct '23

Traceback (most recent call last): File "/srv/onionoo.torproject.org/onionoo/monitoring/network/onionoo", line 678, in <module> write_to_textfile('/srv/onionoo.torproject.org/onionoo/network/metrics', registry) File "/usr/lib/python3/dist-packages/prometheus_client/exposition.py", line 202, in write_to_textfile with open(tmppath, 'wb') as f: OSError: [Errno 28] No space left on device: '/srv/onionoo.torproject.org/onionoo/network/metrics.3087377.140668699637568'

1 0

Cron <onionoo@onionoo-backend-01> /srv/onionoo.torproject.org/onionoo/monitoring/services/onionoo
by root＠onionoo-backend-01.torproject.org 02 Oct '23

02 Oct '23

Traceback (most recent call last): File "/srv/onionoo.torproject.org/onionoo/monitoring/services/onionoo", line 38, in <module> write_to_textfile("/srv/onionoo.torproject.org/onionoo/service/metrics", registry=registry) File "/usr/lib/python3/dist-packages/prometheus_client/exposition.py", line 202, in write_to_textfile with open(tmppath, 'wb') as f: OSError: [Errno 28] No space left on device: '/srv/onionoo.torproject.org/onionoo/service/metrics.3087374.140282171696960'

1 0

Cron <onionoo@onionoo-backend-01> /srv/onionoo.torproject.org/onionoo/monitoring/network/onionoo
by root＠onionoo-backend-01.torproject.org 02 Oct '23

02 Oct '23

Traceback (most recent call last): File "/srv/onionoo.torproject.org/onionoo/monitoring/network/onionoo", line 678, in <module> write_to_textfile('/srv/onionoo.torproject.org/onionoo/network/metrics', registry) File "/usr/lib/python3/dist-packages/prometheus_client/exposition.py", line 202, in write_to_textfile with open(tmppath, 'wb') as f: OSError: [Errno 28] No space left on device: '/srv/onionoo.torproject.org/onionoo/network/metrics.3087146.140137501886272'

1 0

Cron <onionoo@onionoo-backend-01> /srv/onionoo.torproject.org/onionoo/monitoring/services/onionoo
by root＠onionoo-backend-01.torproject.org 02 Oct '23

02 Oct '23

Traceback (most recent call last): File "/srv/onionoo.torproject.org/onionoo/monitoring/services/onionoo", line 38, in <module> write_to_textfile("/srv/onionoo.torproject.org/onionoo/service/metrics", registry=registry) File "/usr/lib/python3/dist-packages/prometheus_client/exposition.py", line 202, in write_to_textfile with open(tmppath, 'wb') as f: OSError: [Errno 28] No space left on device: '/srv/onionoo.torproject.org/onionoo/service/metrics.3087154.140118743590720'

1 0

Cron <onionoo@onionoo-backend-01> /srv/onionoo.torproject.org/onionoo/monitoring/network/onionoo
by root＠onionoo-backend-01.torproject.org 02 Oct '23

02 Oct '23

Traceback (most recent call last): File "/srv/onionoo.torproject.org/onionoo/monitoring/network/onionoo", line 678, in <module> write_to_textfile('/srv/onionoo.torproject.org/onionoo/network/metrics', registry) File "/usr/lib/python3/dist-packages/prometheus_client/exposition.py", line 202, in write_to_textfile with open(tmppath, 'wb') as f: OSError: [Errno 28] No space left on device: '/srv/onionoo.torproject.org/onionoo/network/metrics.3086926.140080186488640'

1 0

Cron <onionoo@onionoo-backend-01> /srv/onionoo.torproject.org/onionoo/monitoring/services/onionoo
by root＠onionoo-backend-01.torproject.org 02 Oct '23

02 Oct '23

Traceback (most recent call last): File "/srv/onionoo.torproject.org/onionoo/monitoring/services/onionoo", line 38, in <module> write_to_textfile("/srv/onionoo.torproject.org/onionoo/service/metrics", registry=registry) File "/usr/lib/python3/dist-packages/prometheus_client/exposition.py", line 202, in write_to_textfile with open(tmppath, 'wb') as f: OSError: [Errno 28] No space left on device: '/srv/onionoo.torproject.org/onionoo/service/metrics.3086925.140492197467968'

1 0

[FIRING:1] Onionperf disk space
by alertmanager＠hetzner-nbg1-02.torproject.org 02 Oct '23

02 Oct '23

Total firing alerts: 1 Total resolved alerts: 0 ## Firing Alerts ----- Time: 2023-10-02 07:06:12.544 +0000 UTC Summary: op-hk7a.onionperf.torproject.net:9100 is using 95.02853595763926% of disk space available. Description: op-hk7a.onionperf.torproject.net:9100 disk space used is more than 95% ----- ##Resolved Alerts

1 0

** RECOVERY Service Alert: onionoo-frontend-02/network service - onionoo varnish is OK **
by nagios＠hetzner-hel1-01.torproject.org 02 Oct '23

02 Oct '23

***** Icinga ***** Notification Type: RECOVERY Service: network service - onionoo varnish Host: onionoo-frontend-02 Address: 49.12.57.137 State: OK Date/Time: Mon Oct 2 07:04:32 UTC 2023 Additional Info: OK: Valid response with recent timestamps (7463 sec): relays_published=2023-10-02 05:00:00, bridges_published=2023-10-02 05:03:14

1 0