November 2024 - metrics-alerts - lists.torproject.org

Cron <exonerator@materculae> cd /srv/exonerator.torproject.org/exonerator/ && ~exonerator/bin/run-exonerator
by root＠materculae.torproject.org 04 Nov '24

04 Nov '24

2024-11-04 08:22:01,988 WARN o.t.m.e.ExoneraTorDatabaseImporter:174 Lock file /srv/exonerator.torproject.org/exonerator/exonerator-lock is at least 6 hours old. Overwriting and continuing with the database import. 2024-11-04 13:58:18,726 ERROR o.t.m.e.ExoneraTorDatabaseImporter:413 Caught an SQL exception while importing status entry with base64-encoded fingerprint 9vWbZLJJTymJnoByvLDms+BwqRc and valid-after time 2024-11-01T08:00. Check the exception for details. Exiting. org.postgresql.util.PSQLException: ERROR: duplicate key value violates unique constraint "statusentry_oraddress_validafter_fingerprint_id_oraddress_key" Detail: Key (validafter, fingerprint_id, oraddress)=(2024-11-01 08:00:00, 462606, 192.42.116.24) already exists. Where: SQL statement "INSERT INTO statusentry_oraddress (validafter, fingerprint_id, oraddress, nickname_id, exit) VALUES (insert_validafter, insert_fingerprint_id, insert_oraddress::INET, insert_nickname_id, insert_exit)" PL/pgSQL function insert_statusentry_oraddress(timestamp without time zone,character,text,character,character varying,boolean) line 14 at SQL statement at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2440) at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2183) at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:308) at org.postgresql.jdbc.PgStatement.executeInternal(PgStatement.java:441) at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:365) at org.postgresql.jdbc.PgPreparedStatement.executeWithFlags(PgPreparedStatement.java:143) at org.postgresql.jdbc.PgCallableStatement.executeWithFlags(PgCallableStatement.java:77) at org.postgresql.jdbc.PgPreparedStatement.execute(PgPreparedStatement.java:132) at org.torproject.metrics.exonerator.ExoneraTorDatabaseImporter.importStatusentry(ExoneraTorDatabaseImporter.java:410) at org.torproject.metrics.exonerator.ExoneraTorDatabaseImporter.parseConsensus(ExoneraTorDatabaseImporter.java:338) at org.torproject.metrics.exonerator.ExoneraTorDatabaseImporter.parseDescriptors(ExoneraTorDatabaseImporter.java:276) at org.torproject.metrics.exonerator.ExoneraTorDatabaseImporter.main(ExoneraTorDatabaseImporter.java:68)

1 0

[FIRING:8] Collector Index Stale -- collector
by alertmanager＠hetzner-nbg1-02.torproject.org 04 Nov '24

04 Nov '24

Total firing alerts: 8 Total resolved alerts: 0 ## Firing Alerts ----- Time: 2024-10-18 08:06:12.544 +0000 UTC Summary: Collector index stale for more than 10 minutes Description: The collector index freshness metric has been in a critical state for more than 10 minutes for /recent/relay-descriptors/bandwidths/ on ----- ----- Time: 2024-10-18 08:06:12.544 +0000 UTC Summary: Collector index stale for more than 10 minutes Description: The collector index freshness metric has been in a critical state for more than 10 minutes for /recent/relay-descriptors/consensuses/ on ----- ----- Time: 2024-10-18 08:06:12.544 +0000 UTC Summary: Collector index stale for more than 10 minutes Description: The collector index freshness metric has been in a critical state for more than 10 minutes for /recent/relay-descriptors/extra-infos/ on ----- ----- Time: 2024-10-18 08:06:12.544 +0000 UTC Summary: Collector index stale for more than 10 minutes Description: The collector index freshness metric has been in a critical state for more than 10 minutes for /recent/relay-descriptors/microdescs/consensus-microdesc/ on ----- ----- Time: 2024-10-18 08:06:12.544 +0000 UTC Summary: Collector index stale for more than 10 minutes Description: The collector index freshness metric has been in a critical state for more than 10 minutes for /recent/relay-descriptors/microdescs/micro/ on ----- ----- Time: 2024-10-18 08:06:12.544 +0000 UTC Summary: Collector index stale for more than 10 minutes Description: The collector index freshness metric has been in a critical state for more than 10 minutes for /recent/relay-descriptors/server-descriptors/ on ----- ----- Time: 2024-10-18 08:06:12.544 +0000 UTC Summary: Collector index stale for more than 10 minutes Description: The collector index freshness metric has been in a critical state for more than 10 minutes for /recent/relay-descriptors/votes/ on ----- ----- Time: 2024-11-04 13:35:27.544 +0000 UTC Summary: Collector index stale for more than 10 minutes Description: The collector index freshness metric has been in a critical state for more than 10 minutes for /archive/relay-descriptors/microdescs/ on ----- ##Resolved Alerts

1 0

Cron <exonerator@materculae> cd /srv/exonerator.torproject.org/exonerator/ && ~exonerator/bin/run-exonerator
by root＠materculae.torproject.org 04 Nov '24

04 Nov '24

2024-11-04 13:22:01,607 ERROR o.t.m.e.ExoneraTorDatabaseImporter:167 Lock file /srv/exonerator.torproject.org/exonerator/exonerator-lock is less than 6 hours old. Either make sure that there are no other ExoneraTor database importers running and manually delete that file, or wait until the file is 6 hours old when it will be overwritten automatically. Exiting.

1 0

Cron <exonerator@materculae> cd /srv/exonerator.torproject.org/exonerator/ && ~exonerator/bin/run-exonerator
by root＠materculae.torproject.org 04 Nov '24

04 Nov '24

2024-11-04 12:22:02,445 ERROR o.t.m.e.ExoneraTorDatabaseImporter:167 Lock file /srv/exonerator.torproject.org/exonerator/exonerator-lock is less than 6 hours old. Either make sure that there are no other ExoneraTor database importers running and manually delete that file, or wait until the file is 6 hours old when it will be overwritten automatically. Exiting.

1 0

Cron <exonerator@materculae> cd /srv/exonerator.torproject.org/exonerator/ && ~exonerator/bin/run-exonerator
by root＠materculae.torproject.org 04 Nov '24

04 Nov '24

2024-11-04 11:22:01,577 ERROR o.t.m.e.ExoneraTorDatabaseImporter:167 Lock file /srv/exonerator.torproject.org/exonerator/exonerator-lock is less than 6 hours old. Either make sure that there are no other ExoneraTor database importers running and manually delete that file, or wait until the file is 6 hours old when it will be overwritten automatically. Exiting.

1 0

Cron <exonerator@materculae> cd /srv/exonerator.torproject.org/exonerator/ && ~exonerator/bin/run-exonerator
by root＠materculae.torproject.org 04 Nov '24

04 Nov '24

2024-11-04 10:22:01,630 ERROR o.t.m.e.ExoneraTorDatabaseImporter:167 Lock file /srv/exonerator.torproject.org/exonerator/exonerator-lock is less than 6 hours old. Either make sure that there are no other ExoneraTor database importers running and manually delete that file, or wait until the file is 6 hours old when it will be overwritten automatically. Exiting.

1 0

Cron <exonerator@materculae> cd /srv/exonerator.torproject.org/exonerator/ && ~exonerator/bin/run-exonerator
by root＠materculae.torproject.org 04 Nov '24

04 Nov '24

2024-11-04 09:22:01,618 ERROR o.t.m.e.ExoneraTorDatabaseImporter:167 Lock file /srv/exonerator.torproject.org/exonerator/exonerator-lock is less than 6 hours old. Either make sure that there are no other ExoneraTor database importers running and manually delete that file, or wait until the file is 6 hours old when it will be overwritten automatically. Exiting.

1 0

Cron <exonerator@materculae> cd /srv/exonerator.torproject.org/exonerator/ && ~exonerator/bin/run-exonerator
by root＠materculae.torproject.org 04 Nov '24

04 Nov '24

2024-11-04 07:22:02,139 ERROR o.t.m.e.ExoneraTorDatabaseImporter:167 Lock file /srv/exonerator.torproject.org/exonerator/exonerator-lock is less than 6 hours old. Either make sure that there are no other ExoneraTor database importers running and manually delete that file, or wait until the file is 6 hours old when it will be overwritten automatically. Exiting.

1 0

Cron <exonerator@materculae> cd /srv/exonerator.torproject.org/exonerator/ && ~exonerator/bin/run-exonerator
by root＠materculae.torproject.org 04 Nov '24

04 Nov '24

2024-11-04 06:22:02,322 ERROR o.t.m.e.ExoneraTorDatabaseImporter:167 Lock file /srv/exonerator.torproject.org/exonerator/exonerator-lock is less than 6 hours old. Either make sure that there are no other ExoneraTor database importers running and manually delete that file, or wait until the file is 6 hours old when it will be overwritten automatically. Exiting.

1 0

[FIRING:8] Collector Index Stale -- collector
by alertmanager＠hetzner-nbg1-02.torproject.org 04 Nov '24

04 Nov '24

Total firing alerts: 8 Total resolved alerts: 0 ## Firing Alerts ----- Time: 2024-10-18 08:06:12.544 +0000 UTC Summary: Collector index stale for more than 10 minutes Description: The collector index freshness metric has been in a critical state for more than 10 minutes for /recent/relay-descriptors/bandwidths/ on ----- ----- Time: 2024-10-18 08:06:12.544 +0000 UTC Summary: Collector index stale for more than 10 minutes Description: The collector index freshness metric has been in a critical state for more than 10 minutes for /recent/relay-descriptors/consensuses/ on ----- ----- Time: 2024-10-18 08:06:12.544 +0000 UTC Summary: Collector index stale for more than 10 minutes Description: The collector index freshness metric has been in a critical state for more than 10 minutes for /recent/relay-descriptors/extra-infos/ on ----- ----- Time: 2024-10-18 08:06:12.544 +0000 UTC Summary: Collector index stale for more than 10 minutes Description: The collector index freshness metric has been in a critical state for more than 10 minutes for /recent/relay-descriptors/microdescs/consensus-microdesc/ on ----- ----- Time: 2024-10-18 08:06:12.544 +0000 UTC Summary: Collector index stale for more than 10 minutes Description: The collector index freshness metric has been in a critical state for more than 10 minutes for /recent/relay-descriptors/microdescs/micro/ on ----- ----- Time: 2024-10-18 08:06:12.544 +0000 UTC Summary: Collector index stale for more than 10 minutes Description: The collector index freshness metric has been in a critical state for more than 10 minutes for /recent/relay-descriptors/server-descriptors/ on ----- ----- Time: 2024-10-18 08:06:12.544 +0000 UTC Summary: Collector index stale for more than 10 minutes Description: The collector index freshness metric has been in a critical state for more than 10 minutes for /recent/relay-descriptors/votes/ on ----- ----- Time: 2024-11-04 06:00:27.544 +0000 UTC Summary: Collector index stale for more than 10 minutes Description: The collector index freshness metric has been in a critical state for more than 10 minutes for /archive/relay-descriptors/extra-infos/ on ----- ##Resolved Alerts

1 0