June 2023 - metrics-alerts - lists.torproject.org

Cron <exonerator@materculae> cd /srv/exonerator.torproject.org/exonerator/ && ~exonerator/bin/run-exonerator
by root＠materculae.torproject.org 01 Jun '23

01 Jun '23

2023-06-01 10:22:02,113 ERROR o.t.m.e.ExoneraTorDatabaseImporter:167 Lock file /srv/exonerator.torproject.org/exonerator/exonerator-lock is less than 6 hours old. Either make sure that there are no other ExoneraTor database importers running and manually delete that file, or wait until the file is 6 hours old when it will be overwritten automatically. Exiting.

1 0

Cron <exonerator@materculae> cd /srv/exonerator.torproject.org/exonerator/ && ~exonerator/bin/run-exonerator
by root＠materculae.torproject.org 01 Jun '23

01 Jun '23

2023-06-01 09:22:01,595 ERROR o.t.m.e.ExoneraTorDatabaseImporter:167 Lock file /srv/exonerator.torproject.org/exonerator/exonerator-lock is less than 6 hours old. Either make sure that there are no other ExoneraTor database importers running and manually delete that file, or wait until the file is 6 hours old when it will be overwritten automatically. Exiting.

1 0

** PROBLEM Service Alert: global/collector2 is WARNING **
by nagios＠hetzner-hel1-01.torproject.org 01 Jun '23

01 Jun '23

***** Icinga ***** Notification Type: PROBLEM Service: collector2 Host: global Address: global State: WARNING Date/Time: Thu Jun 1 09:05:36 UTC 2023 Additional Info: WARNING: Got a valid response from https://collector2.torproject.org/index/index.json with at least one timestamp being outdated: /recent/relay-descriptors/bandwidths/ = 2023-06-01 05:23 (3 = 3 hours ago), /recent/relay-descriptors/consensuses/ = 2023-06-01 05:20 (3 = 3 hours ago), /recent/relay-descriptors/extra-infos/ = 2023-06-01 05:21 (3 = 3 hours ago), /recent/relay-descriptors/microdescs/consensus-microdesc/ = 2023-06-01 05:20 (3 = 3 hours ago), /recent/relay-descriptors/microdescs/micro/ = 2023-06-01 05:21 (3 = 3 hours ago), /recent/relay-descriptors/server-descriptors/ = 2023-06-01 05:21 (3 = 3 hours ago), /recent/relay-descriptors/votes/ = 2023-06-01 05:20 (3 = 3 hours ago)

1 0

Cron <exonerator@materculae> cd /srv/exonerator.torproject.org/exonerator/ && ~exonerator/bin/run-exonerator
by root＠materculae.torproject.org 01 Jun '23

01 Jun '23

2023-06-01 08:22:02,048 ERROR o.t.m.e.ExoneraTorDatabaseImporter:167 Lock file /srv/exonerator.torproject.org/exonerator/exonerator-lock is less than 6 hours old. Either make sure that there are no other ExoneraTor database importers running and manually delete that file, or wait until the file is 6 hours old when it will be overwritten automatically. Exiting.

1 0

Cron <exonerator@materculae> cd /srv/exonerator.torproject.org/exonerator/ && ~exonerator/bin/run-exonerator
by root＠materculae.torproject.org 01 Jun '23

01 Jun '23

2023-06-01 07:22:02,531 ERROR o.t.m.e.ExoneraTorDatabaseImporter:167 Lock file /srv/exonerator.torproject.org/exonerator/exonerator-lock is less than 6 hours old. Either make sure that there are no other ExoneraTor database importers running and manually delete that file, or wait until the file is 6 hours old when it will be overwritten automatically. Exiting.

1 0

** PROBLEM Service Alert: global/collector2 is CRITICAL **
by nagios＠hetzner-hel1-01.torproject.org 01 Jun '23

01 Jun '23

***** Icinga ***** Notification Type: PROBLEM Service: collector2 Host: global Address: global State: CRITICAL Date/Time: Thu Jun 1 06:35:36 UTC 2023 Additional Info: CRITICAL: Got a valid response from https://collector2.torproject.org/index/index.json with at least one timestamp being critically outdated: /recent/relay-descriptors/extra-infos/ = 2023-06-01 00:26 (6 = 3 + 3 hours ago), /recent/relay-descriptors/microdescs/micro/ = 2023-06-01 00:26 (6 = 3 + 3 hours ago), /recent/relay-descriptors/server-descriptors/ = 2023-06-01 00:26 (6 = 3 + 3 hours ago)

1 0

Cron <exonerator@materculae> cd /srv/exonerator.torproject.org/exonerator/ && ~exonerator/bin/run-exonerator
by root＠materculae.torproject.org 01 Jun '23

01 Jun '23

2023-06-01 06:31:09,229 ERROR o.t.m.e.ExoneraTorDatabaseImporter:413 Caught an SQL exception while importing status entry with base64-encoded fingerprint 0OGWKyRgulTYHhQmMA0Jcg5tQ+s and valid-after time 2023-06-01T06:00. Check the exception for details. Exiting. org.postgresql.util.PSQLException: FATAL: terminating connection due to administrator command at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2440) at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2183) at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:308) at org.postgresql.jdbc.PgStatement.executeInternal(PgStatement.java:441) at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:365) at org.postgresql.jdbc.PgPreparedStatement.executeWithFlags(PgPreparedStatement.java:143) at org.postgresql.jdbc.PgCallableStatement.executeWithFlags(PgCallableStatement.java:77) at org.postgresql.jdbc.PgPreparedStatement.execute(PgPreparedStatement.java:132) at org.torproject.metrics.exonerator.ExoneraTorDatabaseImporter.importStatusentry(ExoneraTorDatabaseImporter.java:410) at org.torproject.metrics.exonerator.ExoneraTorDatabaseImporter.parseConsensus(ExoneraTorDatabaseImporter.java:338) at org.torproject.metrics.exonerator.ExoneraTorDatabaseImporter.parseDescriptors(ExoneraTorDatabaseImporter.java:276) at org.torproject.metrics.exonerator.ExoneraTorDatabaseImporter.main(ExoneraTorDatabaseImporter.java:68)

1 0

** PROBLEM Service Alert: global/collector2 is WARNING **
by nagios＠hetzner-hel1-01.torproject.org 01 Jun '23

01 Jun '23

***** Icinga ***** Notification Type: PROBLEM Service: collector2 Host: global Address: global State: WARNING Date/Time: Thu Jun 1 02:35:36 UTC 2023 Additional Info: WARNING: Got a valid response from https://collector2.torproject.org/index/index.json with at least one timestamp being outdated: /recent/relay-descriptors/bandwidths/ = 2023-05-31 23:23 (3 = 3 hours ago), /recent/relay-descriptors/extra-infos/ = 2023-05-31 23:20 (3 = 3 hours ago), /recent/relay-descriptors/microdescs/micro/ = 2023-05-31 23:20 (3 = 3 hours ago), /recent/relay-descriptors/server-descriptors/ = 2023-05-31 23:20 (3 = 3 hours ago)

1 0

[FIRING:1] Relays per nickname prefix MEDIUM
by alertmanager＠hetzner-nbg1-02.torproject.org 01 Jun '23

01 Jun '23

Total firing alerts: 1 Total resolved alerts: 0 ## Firing Alerts ----- Time: 2023-06-01 02:16:27.544031668 +0000 UTC Summary: More than 10 relays have joined the network with the nickname prefix "prsv" Description: More than 10 relays have joined the network with the nickname prefix "prsv" in the last 2 hours. ----- ##Resolved Alerts

1 0

[FIRING:1] Relays per contact string MEDIUM
by alertmanager＠hetzner-nbg1-02.torproject.org 01 Jun '23

01 Jun '23

Total firing alerts: 1 Total resolved alerts: 0 ## Firing Alerts ----- Time: 2023-06-01 02:16:27.544031668 +0000 UTC Summary: More than 10 relays have joined the network with the contact string "email:admin[]prsv.ch url:https://prsv.ch/ proof:uri-rsa ciissversion:2" Description: More than 10 relays have joined the network with the contact string "email:admin[]prsv.ch url:https://prsv.ch/ proof:uri-rsa ciissversion:2" in the last 2 hours. ----- ##Resolved Alerts

1 0