July 2024 - metrics-alerts - lists.torproject.org

** RECOVERY Service Alert: global/collector2 is OK **
by nagios＠hetzner-hel1-01.torproject.org 03 Jul '24

03 Jul '24

***** Icinga ***** Notification Type: RECOVERY Service: collector2 Host: global Address: global State: OK Date/Time: Wed Jul 3 14:44:17 UTC 2024 Additional Info: OK: Got a valid response from https://collector2.torproject.org/index/index.json with recent timestamps: /recent/relay-descriptors/bandwidths/ = 2024-07-03 14:31 (0 3 hours ago), /recent/relay-descriptors/consensuses/ = 2024-07-03 14:20 (0 3 hours ago), /recent/relay-descriptors/extra-infos/ = 2024-07-03 14:31 (0 3 hours ago), /recent/relay-descriptors/microdescs/consensus-microdesc/ = 2024-07-03 14:20 (0 3 hours ago), /recent/relay-descriptors/microdescs/micro/ = 2024-07-03 14:24 (0 3 hours ago), /recent/relay-descriptors/server-descriptors/ = 2024-07-03 14:31 (0 3 hours ago), /recent/relay-descriptors/votes/ = 2024-07-03 14:20 (0 3 hours ago)

1 0

Configuration error - Default route: [FIRING:1] JobDown
by alertmanager＠hetzner-nbg1-02.torproject.org 03 Jul '24

03 Jul '24

Total firing alerts: 1 Total resolved alerts: 0 ## Firing Alerts ----- Time: 2024-07-03 14:30:47.366 +0000 UTC Summary: Job tordnsel_data@TordnselData is down Description: Job tordnsel_data on TordnselData has been down for more than 5 minutes. ----- ##Resolved Alerts

1 0

** PROBLEM Service Alert: global/collector is CRITICAL **
by nagios＠hetzner-hel1-01.torproject.org 03 Jul '24

03 Jul '24

***** Icinga ***** Notification Type: PROBLEM Service: collector Host: global Address: global State: CRITICAL Date/Time: Wed Jul 3 14:28:37 UTC 2024 Additional Info: CRITICAL: Got a valid response from https://collector.torproject.org/index/index.json with at least one timestamp being critically outdated: /recent/relay-descriptors/bandwidths/ = 2024-07-03 04:49 (9 = 3 + 3 hours ago), /recent/relay-descriptors/consensuses/ = 2024-07-03 03:56 (10 = 3 + 3 hours ago), /recent/relay-descriptors/extra-infos/ = 2024-07-03 04:43 (9 = 3 + 3 hours ago), /recent/relay-descriptors/microdescs/consensus-microdesc/ = 2024-07-03 04:47 (9 = 3 + 3 hours ago), /recent/relay-descriptors/microdescs/micro/ = 2024-07-03 04:34 (9 = 3 + 3 hours ago), /recent/relay-descriptors/server-descriptors/ = 2024-07-03 04:43 (9 = 3 + 3 hours ago), /recent/relay-descriptors/votes/ = 2024-07-03 04:31 (9 = 3 + 3 hours ago)

1 0

Configuration error - Default route: [FIRING:1] JobDown
by alertmanager＠hetzner-nbg1-02.torproject.org 03 Jul '24

03 Jul '24

Total firing alerts: 1 Total resolved alerts: 0 ## Firing Alerts ----- Time: 2024-07-03 14:16:17.366 +0000 UTC Summary: Job collector@CollecTor is down Description: Job collector on CollecTor has been down for more than 5 minutes. ----- ##Resolved Alerts

1 0

Cron <onionoo@onionoo-backend-01> cd /srv/onionoo.torproject.org/onionoo/bin/onionoo/ && ./runner
by root＠onionoo-backend-01.torproject.org 03 Jul '24

03 Jul '24

Run onionoo update service. Next run will start in 60 minutes.

1 0

** PROBLEM Service Alert: global/collector is UNKNOWN **
by nagios＠hetzner-hel1-01.torproject.org 03 Jul '24

03 Jul '24

***** Icinga ***** Notification Type: PROBLEM Service: collector Host: global Address: global State: UNKNOWN Date/Time: Wed Jul 3 14:14:37 UTC 2024 Additional Info: (Service Check Timed Out)

1 0

** PROBLEM Service Alert: global/collector2 is CRITICAL **
by nagios＠hetzner-hel1-01.torproject.org 03 Jul '24

03 Jul '24

***** Icinga ***** Notification Type: PROBLEM Service: collector2 Host: global Address: global State: CRITICAL Date/Time: Wed Jul 3 14:14:17 UTC 2024 Additional Info: CRITICAL: Got a valid response from https://collector2.torproject.org/index/index.json with at least one timestamp being critically outdated: /recent/relay-descriptors/bandwidths/ = 2024-06-21 04:27 (297 = 3 + 3 hours ago), /recent/relay-descriptors/consensuses/ = 2024-06-21 04:20 (297 = 3 + 3 hours ago), /recent/relay-descriptors/extra-infos/ = 2024-06-21 04:27 (297 = 3 + 3 hours ago), /recent/relay-descriptors/microdescs/consensus-microdesc/ = 2024-06-21 04:20 (297 = 3 + 3 hours ago), /recent/relay-descriptors/microdescs/micro/ = 2024-06-21 04:20 (297 = 3 + 3 hours ago), /recent/relay-descriptors/server-descriptors/ = 2024-06-21 04:27 (297 = 3 + 3 hours ago), /recent/relay-descriptors/votes/ = 2024-06-21 04:20 (297 = 3 + 3 hours ago)

1 0

** PROBLEM Service Alert: global/collector2 is UNKNOWN **
by nagios＠hetzner-hel1-01.torproject.org 03 Jul '24

03 Jul '24

***** Icinga ***** Notification Type: PROBLEM Service: collector2 Host: global Address: global State: UNKNOWN Date/Time: Wed Jul 3 13:59:17 UTC 2024 Additional Info: UNKNOWN: Error fetching https://collector2.torproject.org/index/index.json: [Errno 111] Connection refused

1 0

Configuration error - Default route: [FIRING:1] JobDown
by alertmanager＠hetzner-nbg1-02.torproject.org 03 Jul '24

03 Jul '24

Total firing alerts: 1 Total resolved alerts: 0 ## Firing Alerts ----- Time: 2024-07-03 13:55:32.366 +0000 UTC Summary: Job collector@CollecTor is down Description: Job collector on CollecTor has been down for more than 5 minutes. ----- ##Resolved Alerts

1 0

Cron <exonerator@materculae> cd /srv/exonerator.torproject.org/exonerator/ && ~exonerator/bin/run-exonerator
by root＠materculae.torproject.org 03 Jul '24

03 Jul '24

2024-07-03 12:22:02,305 ERROR o.t.m.e.ExoneraTorDatabaseImporter:167 Lock file /srv/exonerator.torproject.org/exonerator/exonerator-lock is less than 6 hours old. Either make sure that there are no other ExoneraTor database importers running and manually delete that file, or wait until the file is 6 hours old when it will be overwritten automatically. Exiting.

1 0