November 2024 - metrics-alerts - lists.torproject.org

Cron <exonerator@materculae> cd /srv/exonerator.torproject.org/exonerator/ && ~exonerator/bin/run-exonerator
by root＠materculae.torproject.org 03 Nov '24

03 Nov '24

2024-11-03 21:22:02,369 ERROR o.t.m.e.ExoneraTorDatabaseImporter:167 Lock file /srv/exonerator.torproject.org/exonerator/exonerator-lock is less than 6 hours old. Either make sure that there are no other ExoneraTor database importers running and manually delete that file, or wait until the file is 6 hours old when it will be overwritten automatically. Exiting.

1 0

Cron <exonerator@materculae> cd /srv/exonerator.torproject.org/exonerator/ && ~exonerator/bin/run-exonerator
by root＠materculae.torproject.org 03 Nov '24

03 Nov '24

2024-11-03 20:22:01,828 ERROR o.t.m.e.ExoneraTorDatabaseImporter:167 Lock file /srv/exonerator.torproject.org/exonerator/exonerator-lock is less than 6 hours old. Either make sure that there are no other ExoneraTor database importers running and manually delete that file, or wait until the file is 6 hours old when it will be overwritten automatically. Exiting.

1 0

Cron <exonerator@materculae> cd /srv/exonerator.torproject.org/exonerator/ && ~exonerator/bin/run-exonerator
by root＠materculae.torproject.org 03 Nov '24

03 Nov '24

2024-11-03 19:22:02,303 ERROR o.t.m.e.ExoneraTorDatabaseImporter:167 Lock file /srv/exonerator.torproject.org/exonerator/exonerator-lock is less than 6 hours old. Either make sure that there are no other ExoneraTor database importers running and manually delete that file, or wait until the file is 6 hours old when it will be overwritten automatically. Exiting.

1 0

[FIRING:12] Collector Index Stale -- collector
by alertmanager＠hetzner-nbg1-02.torproject.org 03 Nov '24

03 Nov '24

Total firing alerts: 12 Total resolved alerts: 0 ## Firing Alerts ----- Time: 2024-10-18 08:06:12.544 +0000 UTC Summary: Collector index stale for more than 10 minutes Description: The collector index freshness metric has been in a critical state for more than 10 minutes for /recent/relay-descriptors/bandwidths/ on ----- ----- Time: 2024-10-18 08:06:12.544 +0000 UTC Summary: Collector index stale for more than 10 minutes Description: The collector index freshness metric has been in a critical state for more than 10 minutes for /recent/relay-descriptors/consensuses/ on ----- ----- Time: 2024-10-18 08:06:12.544 +0000 UTC Summary: Collector index stale for more than 10 minutes Description: The collector index freshness metric has been in a critical state for more than 10 minutes for /recent/relay-descriptors/extra-infos/ on ----- ----- Time: 2024-10-18 08:06:12.544 +0000 UTC Summary: Collector index stale for more than 10 minutes Description: The collector index freshness metric has been in a critical state for more than 10 minutes for /recent/relay-descriptors/microdescs/consensus-microdesc/ on ----- ----- Time: 2024-10-18 08:06:12.544 +0000 UTC Summary: Collector index stale for more than 10 minutes Description: The collector index freshness metric has been in a critical state for more than 10 minutes for /recent/relay-descriptors/microdescs/micro/ on ----- ----- Time: 2024-10-18 08:06:12.544 +0000 UTC Summary: Collector index stale for more than 10 minutes Description: The collector index freshness metric has been in a critical state for more than 10 minutes for /recent/relay-descriptors/server-descriptors/ on ----- ----- Time: 2024-10-18 08:06:12.544 +0000 UTC Summary: Collector index stale for more than 10 minutes Description: The collector index freshness metric has been in a critical state for more than 10 minutes for /recent/relay-descriptors/votes/ on ----- ----- Time: 2024-11-03 13:00:27.544 +0000 UTC Summary: Collector index stale for more than 10 minutes Description: The collector index freshness metric has been in a critical state for more than 10 minutes for /archive/relay-descriptors/consensuses/ on ----- ----- Time: 2024-11-01 14:05:27.544 +0000 UTC Summary: Collector index stale for more than 10 minutes Description: The collector index freshness metric has been in a critical state for more than 10 minutes for /recent/bridge-descriptors/extra-infos/ on ----- ----- Time: 2024-11-03 13:00:27.544 +0000 UTC Summary: Collector index stale for more than 10 minutes Description: The collector index freshness metric has been in a critical state for more than 10 minutes for /recent/bridge-descriptors/server-descriptors/ on ----- ----- Time: 2024-11-03 13:00:27.544 +0000 UTC Summary: Collector index stale for more than 10 minutes Description: The collector index freshness metric has been in a critical state for more than 10 minutes for /recent/bridge-descriptors/statuses/ on ----- ----- Time: 2024-11-03 18:55:27.544 +0000 UTC Summary: Collector index stale for more than 10 minutes Description: The collector index freshness metric has been in a critical state for more than 10 minutes for /recent/webstats/ on ----- ##Resolved Alerts

1 0

[FIRING:17] Collector Index Stale -- collector
by alertmanager＠hetzner-nbg1-02.torproject.org 03 Nov '24

03 Nov '24

Total firing alerts: 17 Total resolved alerts: 0 ## Firing Alerts ----- Time: 2024-10-18 08:06:12.544 +0000 UTC Summary: Collector index stale for more than 10 minutes Description: The collector index freshness metric has been in a critical state for more than 10 minutes for /recent/relay-descriptors/bandwidths/ on ----- ----- Time: 2024-10-18 08:06:12.544 +0000 UTC Summary: Collector index stale for more than 10 minutes Description: The collector index freshness metric has been in a critical state for more than 10 minutes for /recent/relay-descriptors/consensuses/ on ----- ----- Time: 2024-10-18 08:06:12.544 +0000 UTC Summary: Collector index stale for more than 10 minutes Description: The collector index freshness metric has been in a critical state for more than 10 minutes for /recent/relay-descriptors/extra-infos/ on ----- ----- Time: 2024-10-18 08:06:12.544 +0000 UTC Summary: Collector index stale for more than 10 minutes Description: The collector index freshness metric has been in a critical state for more than 10 minutes for /recent/relay-descriptors/microdescs/consensus-microdesc/ on ----- ----- Time: 2024-10-18 08:06:12.544 +0000 UTC Summary: Collector index stale for more than 10 minutes Description: The collector index freshness metric has been in a critical state for more than 10 minutes for /recent/relay-descriptors/microdescs/micro/ on ----- ----- Time: 2024-10-18 08:06:12.544 +0000 UTC Summary: Collector index stale for more than 10 minutes Description: The collector index freshness metric has been in a critical state for more than 10 minutes for /recent/relay-descriptors/server-descriptors/ on ----- ----- Time: 2024-10-18 08:06:12.544 +0000 UTC Summary: Collector index stale for more than 10 minutes Description: The collector index freshness metric has been in a critical state for more than 10 minutes for /recent/relay-descriptors/votes/ on ----- ----- Time: 2024-11-03 13:00:27.544 +0000 UTC Summary: Collector index stale for more than 10 minutes Description: The collector index freshness metric has been in a critical state for more than 10 minutes for /archive/relay-descriptors/consensuses/ on ----- ----- Time: 2024-11-01 14:05:27.544 +0000 UTC Summary: Collector index stale for more than 10 minutes Description: The collector index freshness metric has been in a critical state for more than 10 minutes for /recent/bridge-descriptors/extra-infos/ on ----- ----- Time: 2024-11-03 13:00:27.544 +0000 UTC Summary: Collector index stale for more than 10 minutes Description: The collector index freshness metric has been in a critical state for more than 10 minutes for /recent/bridge-descriptors/server-descriptors/ on ----- ----- Time: 2024-11-03 13:00:27.544 +0000 UTC Summary: Collector index stale for more than 10 minutes Description: The collector index freshness metric has been in a critical state for more than 10 minutes for /recent/bridge-descriptors/statuses/ on ----- ----- Time: 2024-11-03 17:50:27.544 +0000 UTC Summary: Collector index stale for more than 10 minutes Description: The collector index freshness metric has been in a critical state for more than 10 minutes for /recent/relay-descriptors/bandwidths/ on ----- ----- Time: 2024-11-03 18:35:27.544 +0000 UTC Summary: Collector index stale for more than 10 minutes Description: The collector index freshness metric has been in a critical state for more than 10 minutes for /recent/relay-descriptors/extra-infos/ on ----- ----- Time: 2024-11-03 18:35:27.544 +0000 UTC Summary: Collector index stale for more than 10 minutes Description: The collector index freshness metric has been in a critical state for more than 10 minutes for /recent/relay-descriptors/microdescs/consensus-microdesc/ on ----- ----- Time: 2024-11-03 18:35:27.544 +0000 UTC Summary: Collector index stale for more than 10 minutes Description: The collector index freshness metric has been in a critical state for more than 10 minutes for /recent/relay-descriptors/microdescs/micro/ on ----- ----- Time: 2024-11-03 18:40:27.544 +0000 UTC Summary: Collector index stale for more than 10 minutes Description: The collector index freshness metric has been in a critical state for more than 10 minutes for /recent/relay-descriptors/server-descriptors/ on ----- ----- Time: 2024-11-03 18:40:27.544 +0000 UTC Summary: Collector index stale for more than 10 minutes Description: The collector index freshness metric has been in a critical state for more than 10 minutes for /recent/relay-descriptors/votes/ on ----- ##Resolved Alerts

1 0

[FIRING:15] Collector Index Stale -- collector
by alertmanager＠hetzner-nbg1-02.torproject.org 03 Nov '24

03 Nov '24

Total firing alerts: 15 Total resolved alerts: 0 ## Firing Alerts ----- Time: 2024-10-18 08:06:12.544 +0000 UTC Summary: Collector index stale for more than 10 minutes Description: The collector index freshness metric has been in a critical state for more than 10 minutes for /recent/relay-descriptors/bandwidths/ on ----- ----- Time: 2024-10-18 08:06:12.544 +0000 UTC Summary: Collector index stale for more than 10 minutes Description: The collector index freshness metric has been in a critical state for more than 10 minutes for /recent/relay-descriptors/consensuses/ on ----- ----- Time: 2024-10-18 08:06:12.544 +0000 UTC Summary: Collector index stale for more than 10 minutes Description: The collector index freshness metric has been in a critical state for more than 10 minutes for /recent/relay-descriptors/extra-infos/ on ----- ----- Time: 2024-10-18 08:06:12.544 +0000 UTC Summary: Collector index stale for more than 10 minutes Description: The collector index freshness metric has been in a critical state for more than 10 minutes for /recent/relay-descriptors/microdescs/consensus-microdesc/ on ----- ----- Time: 2024-10-18 08:06:12.544 +0000 UTC Summary: Collector index stale for more than 10 minutes Description: The collector index freshness metric has been in a critical state for more than 10 minutes for /recent/relay-descriptors/microdescs/micro/ on ----- ----- Time: 2024-10-18 08:06:12.544 +0000 UTC Summary: Collector index stale for more than 10 minutes Description: The collector index freshness metric has been in a critical state for more than 10 minutes for /recent/relay-descriptors/server-descriptors/ on ----- ----- Time: 2024-10-18 08:06:12.544 +0000 UTC Summary: Collector index stale for more than 10 minutes Description: The collector index freshness metric has been in a critical state for more than 10 minutes for /recent/relay-descriptors/votes/ on ----- ----- Time: 2024-11-03 13:00:27.544 +0000 UTC Summary: Collector index stale for more than 10 minutes Description: The collector index freshness metric has been in a critical state for more than 10 minutes for /archive/relay-descriptors/consensuses/ on ----- ----- Time: 2024-11-01 14:05:27.544 +0000 UTC Summary: Collector index stale for more than 10 minutes Description: The collector index freshness metric has been in a critical state for more than 10 minutes for /recent/bridge-descriptors/extra-infos/ on ----- ----- Time: 2024-11-03 13:00:27.544 +0000 UTC Summary: Collector index stale for more than 10 minutes Description: The collector index freshness metric has been in a critical state for more than 10 minutes for /recent/bridge-descriptors/server-descriptors/ on ----- ----- Time: 2024-11-03 13:00:27.544 +0000 UTC Summary: Collector index stale for more than 10 minutes Description: The collector index freshness metric has been in a critical state for more than 10 minutes for /recent/bridge-descriptors/statuses/ on ----- ----- Time: 2024-11-03 17:50:27.544 +0000 UTC Summary: Collector index stale for more than 10 minutes Description: The collector index freshness metric has been in a critical state for more than 10 minutes for /recent/relay-descriptors/bandwidths/ on ----- ----- Time: 2024-11-03 18:35:27.544 +0000 UTC Summary: Collector index stale for more than 10 minutes Description: The collector index freshness metric has been in a critical state for more than 10 minutes for /recent/relay-descriptors/extra-infos/ on ----- ----- Time: 2024-11-03 18:35:27.544 +0000 UTC Summary: Collector index stale for more than 10 minutes Description: The collector index freshness metric has been in a critical state for more than 10 minutes for /recent/relay-descriptors/microdescs/consensus-microdesc/ on ----- ----- Time: 2024-11-03 18:35:27.544 +0000 UTC Summary: Collector index stale for more than 10 minutes Description: The collector index freshness metric has been in a critical state for more than 10 minutes for /recent/relay-descriptors/microdescs/micro/ on ----- ##Resolved Alerts

1 0

Cron <exonerator@materculae> cd /srv/exonerator.torproject.org/exonerator/ && ~exonerator/bin/run-exonerator
by root＠materculae.torproject.org 03 Nov '24

03 Nov '24

2024-11-03 18:22:02,374 ERROR o.t.m.e.ExoneraTorDatabaseImporter:167 Lock file /srv/exonerator.torproject.org/exonerator/exonerator-lock is less than 6 hours old. Either make sure that there are no other ExoneraTor database importers running and manually delete that file, or wait until the file is 6 hours old when it will be overwritten automatically. Exiting.

1 0

[FIRING:20] Collector Index Stale -- collector
by alertmanager＠hetzner-nbg1-02.torproject.org 03 Nov '24

03 Nov '24

Total firing alerts: 20 Total resolved alerts: 0 ## Firing Alerts ----- Time: 2024-10-18 08:06:12.544 +0000 UTC Summary: Collector index stale for more than 10 minutes Description: The collector index freshness metric has been in a critical state for more than 10 minutes for /recent/relay-descriptors/bandwidths/ on ----- ----- Time: 2024-10-18 08:06:12.544 +0000 UTC Summary: Collector index stale for more than 10 minutes Description: The collector index freshness metric has been in a critical state for more than 10 minutes for /recent/relay-descriptors/consensuses/ on ----- ----- Time: 2024-10-18 08:06:12.544 +0000 UTC Summary: Collector index stale for more than 10 minutes Description: The collector index freshness metric has been in a critical state for more than 10 minutes for /recent/relay-descriptors/extra-infos/ on ----- ----- Time: 2024-10-18 08:06:12.544 +0000 UTC Summary: Collector index stale for more than 10 minutes Description: The collector index freshness metric has been in a critical state for more than 10 minutes for /recent/relay-descriptors/microdescs/consensus-microdesc/ on ----- ----- Time: 2024-10-18 08:06:12.544 +0000 UTC Summary: Collector index stale for more than 10 minutes Description: The collector index freshness metric has been in a critical state for more than 10 minutes for /recent/relay-descriptors/microdescs/micro/ on ----- ----- Time: 2024-10-18 08:06:12.544 +0000 UTC Summary: Collector index stale for more than 10 minutes Description: The collector index freshness metric has been in a critical state for more than 10 minutes for /recent/relay-descriptors/server-descriptors/ on ----- ----- Time: 2024-10-18 08:06:12.544 +0000 UTC Summary: Collector index stale for more than 10 minutes Description: The collector index freshness metric has been in a critical state for more than 10 minutes for /recent/relay-descriptors/votes/ on ----- ----- Time: 2024-11-03 10:45:27.544 +0000 UTC Summary: Collector index stale for more than 10 minutes Description: The collector index freshness metric has been in a critical state for more than 10 minutes for /archive/bridge-descriptors/extra-infos/ on ----- ----- Time: 2024-11-03 10:10:27.544 +0000 UTC Summary: Collector index stale for more than 10 minutes Description: The collector index freshness metric has been in a critical state for more than 10 minutes for /archive/bridge-descriptors/server-descriptors/ on ----- ----- Time: 2024-11-03 09:35:27.544 +0000 UTC Summary: Collector index stale for more than 10 minutes Description: The collector index freshness metric has been in a critical state for more than 10 minutes for /archive/bridge-descriptors/statuses/ on ----- ----- Time: 2024-11-03 10:15:27.544 +0000 UTC Summary: Collector index stale for more than 10 minutes Description: The collector index freshness metric has been in a critical state for more than 10 minutes for /archive/bridge-pool-assignments/ on ----- ----- Time: 2024-11-03 10:15:27.544 +0000 UTC Summary: Collector index stale for more than 10 minutes Description: The collector index freshness metric has been in a critical state for more than 10 minutes for /archive/bridgedb-metrics/ on ----- ----- Time: 2024-11-03 06:40:27.544 +0000 UTC Summary: Collector index stale for more than 10 minutes Description: The collector index freshness metric has been in a critical state for more than 10 minutes for /archive/exit-lists/ on ----- ----- Time: 2024-11-03 06:40:27.544 +0000 UTC Summary: Collector index stale for more than 10 minutes Description: The collector index freshness metric has been in a critical state for more than 10 minutes for /archive/onionperf/ on ----- ----- Time: 2024-11-03 13:00:27.544 +0000 UTC Summary: Collector index stale for more than 10 minutes Description: The collector index freshness metric has been in a critical state for more than 10 minutes for /archive/relay-descriptors/consensuses/ on ----- ----- Time: 2024-11-01 14:05:27.544 +0000 UTC Summary: Collector index stale for more than 10 minutes Description: The collector index freshness metric has been in a critical state for more than 10 minutes for /recent/bridge-descriptors/extra-infos/ on ----- ----- Time: 2024-11-03 13:00:27.544 +0000 UTC Summary: Collector index stale for more than 10 minutes Description: The collector index freshness metric has been in a critical state for more than 10 minutes for /recent/bridge-descriptors/server-descriptors/ on ----- ----- Time: 2024-11-03 13:00:27.544 +0000 UTC Summary: Collector index stale for more than 10 minutes Description: The collector index freshness metric has been in a critical state for more than 10 minutes for /recent/bridge-descriptors/statuses/ on ----- ----- Time: 2024-11-03 16:15:27.544 +0000 UTC Summary: Collector index stale for more than 10 minutes Description: The collector index freshness metric has been in a critical state for more than 10 minutes for /recent/onionperf/ on ----- ----- Time: 2024-11-03 17:50:27.544 +0000 UTC Summary: Collector index stale for more than 10 minutes Description: The collector index freshness metric has been in a critical state for more than 10 minutes for /recent/relay-descriptors/bandwidths/ on ----- ##Resolved Alerts

1 0

[FIRING:19] Collector Index Stale -- collector
by alertmanager＠hetzner-nbg1-02.torproject.org 03 Nov '24

03 Nov '24

Total firing alerts: 19 Total resolved alerts: 0 ## Firing Alerts ----- Time: 2024-10-18 08:06:12.544 +0000 UTC Summary: Collector index stale for more than 10 minutes Description: The collector index freshness metric has been in a critical state for more than 10 minutes for /recent/relay-descriptors/bandwidths/ on ----- ----- Time: 2024-10-18 08:06:12.544 +0000 UTC Summary: Collector index stale for more than 10 minutes Description: The collector index freshness metric has been in a critical state for more than 10 minutes for /recent/relay-descriptors/consensuses/ on ----- ----- Time: 2024-10-18 08:06:12.544 +0000 UTC Summary: Collector index stale for more than 10 minutes Description: The collector index freshness metric has been in a critical state for more than 10 minutes for /recent/relay-descriptors/extra-infos/ on ----- ----- Time: 2024-10-18 08:06:12.544 +0000 UTC Summary: Collector index stale for more than 10 minutes Description: The collector index freshness metric has been in a critical state for more than 10 minutes for /recent/relay-descriptors/microdescs/consensus-microdesc/ on ----- ----- Time: 2024-10-18 08:06:12.544 +0000 UTC Summary: Collector index stale for more than 10 minutes Description: The collector index freshness metric has been in a critical state for more than 10 minutes for /recent/relay-descriptors/microdescs/micro/ on ----- ----- Time: 2024-10-18 08:06:12.544 +0000 UTC Summary: Collector index stale for more than 10 minutes Description: The collector index freshness metric has been in a critical state for more than 10 minutes for /recent/relay-descriptors/server-descriptors/ on ----- ----- Time: 2024-10-18 08:06:12.544 +0000 UTC Summary: Collector index stale for more than 10 minutes Description: The collector index freshness metric has been in a critical state for more than 10 minutes for /recent/relay-descriptors/votes/ on ----- ----- Time: 2024-11-03 10:45:27.544 +0000 UTC Summary: Collector index stale for more than 10 minutes Description: The collector index freshness metric has been in a critical state for more than 10 minutes for /archive/bridge-descriptors/extra-infos/ on ----- ----- Time: 2024-11-03 10:10:27.544 +0000 UTC Summary: Collector index stale for more than 10 minutes Description: The collector index freshness metric has been in a critical state for more than 10 minutes for /archive/bridge-descriptors/server-descriptors/ on ----- ----- Time: 2024-11-03 09:35:27.544 +0000 UTC Summary: Collector index stale for more than 10 minutes Description: The collector index freshness metric has been in a critical state for more than 10 minutes for /archive/bridge-descriptors/statuses/ on ----- ----- Time: 2024-11-03 10:15:27.544 +0000 UTC Summary: Collector index stale for more than 10 minutes Description: The collector index freshness metric has been in a critical state for more than 10 minutes for /archive/bridge-pool-assignments/ on ----- ----- Time: 2024-11-03 10:15:27.544 +0000 UTC Summary: Collector index stale for more than 10 minutes Description: The collector index freshness metric has been in a critical state for more than 10 minutes for /archive/bridgedb-metrics/ on ----- ----- Time: 2024-11-03 06:40:27.544 +0000 UTC Summary: Collector index stale for more than 10 minutes Description: The collector index freshness metric has been in a critical state for more than 10 minutes for /archive/exit-lists/ on ----- ----- Time: 2024-11-03 06:40:27.544 +0000 UTC Summary: Collector index stale for more than 10 minutes Description: The collector index freshness metric has been in a critical state for more than 10 minutes for /archive/onionperf/ on ----- ----- Time: 2024-11-03 13:00:27.544 +0000 UTC Summary: Collector index stale for more than 10 minutes Description: The collector index freshness metric has been in a critical state for more than 10 minutes for /archive/relay-descriptors/consensuses/ on ----- ----- Time: 2024-11-01 14:05:27.544 +0000 UTC Summary: Collector index stale for more than 10 minutes Description: The collector index freshness metric has been in a critical state for more than 10 minutes for /recent/bridge-descriptors/extra-infos/ on ----- ----- Time: 2024-11-03 13:00:27.544 +0000 UTC Summary: Collector index stale for more than 10 minutes Description: The collector index freshness metric has been in a critical state for more than 10 minutes for /recent/bridge-descriptors/server-descriptors/ on ----- ----- Time: 2024-11-03 13:00:27.544 +0000 UTC Summary: Collector index stale for more than 10 minutes Description: The collector index freshness metric has been in a critical state for more than 10 minutes for /recent/bridge-descriptors/statuses/ on ----- ----- Time: 2024-11-03 16:15:27.544 +0000 UTC Summary: Collector index stale for more than 10 minutes Description: The collector index freshness metric has been in a critical state for more than 10 minutes for /recent/onionperf/ on ----- ##Resolved Alerts

1 0

Cron <exonerator@materculae> cd /srv/exonerator.torproject.org/exonerator/ && ~exonerator/bin/run-exonerator
by root＠materculae.torproject.org 03 Nov '24

03 Nov '24

2024-11-03 15:22:02,901 ERROR o.t.m.e.ExoneraTorDatabaseImporter:167 Lock file /srv/exonerator.torproject.org/exonerator/exonerator-lock is less than 6 hours old. Either make sure that there are no other ExoneraTor database importers running and manually delete that file, or wait until the file is 6 hours old when it will be overwritten automatically. Exiting.

1 0