December 2023 - metrics-alerts - lists.torproject.org

Cron <exonerator@materculae> cd /srv/exonerator.torproject.org/exonerator/ && ~exonerator/bin/run-exonerator
by root＠materculae.torproject.org 14 Dec '23

14 Dec '23

2023-12-14 21:22:02,408 WARN o.t.m.e.ExoneraTorDatabaseImporter:174 Lock file /srv/exonerator.torproject.org/exonerator/exonerator-lock is at least 6 hours old. Overwriting and continuing with the database import.

1 0

Cron <exonerator@materculae> cd /srv/exonerator.torproject.org/exonerator/ && ~exonerator/bin/run-exonerator
by root＠materculae.torproject.org 14 Dec '23

14 Dec '23

2023-12-14 20:22:01,914 ERROR o.t.m.e.ExoneraTorDatabaseImporter:167 Lock file /srv/exonerator.torproject.org/exonerator/exonerator-lock is less than 6 hours old. Either make sure that there are no other ExoneraTor database importers running and manually delete that file, or wait until the file is 6 hours old when it will be overwritten automatically. Exiting.

1 0

Cron <exonerator@materculae> cd /srv/exonerator.torproject.org/exonerator/ && ~exonerator/bin/run-exonerator
by root＠materculae.torproject.org 14 Dec '23

14 Dec '23

2023-12-14 19:22:01,996 ERROR o.t.m.e.ExoneraTorDatabaseImporter:167 Lock file /srv/exonerator.torproject.org/exonerator/exonerator-lock is less than 6 hours old. Either make sure that there are no other ExoneraTor database importers running and manually delete that file, or wait until the file is 6 hours old when it will be overwritten automatically. Exiting.

1 0

[FIRING:1] Relays per Flag MEDIUM
by alertmanager＠hetzner-nbg1-02.torproject.org 14 Dec '23

14 Dec '23

Total firing alerts: 1 Total resolved alerts: 0 ## Firing Alerts ----- Time: 2023-12-14 19:12:42.544 +0000 UTC Summary: More than 50 relays have joined the network with "HSDir" flag Description: More than 50 relays with "HSDir" flag have joined the network in the last 2 hours. ----- ##Resolved Alerts

1 0

Cron <exonerator@materculae> cd /srv/exonerator.torproject.org/exonerator/ && ~exonerator/bin/run-exonerator
by root＠materculae.torproject.org 14 Dec '23

14 Dec '23

2023-12-14 18:22:02,179 ERROR o.t.m.e.ExoneraTorDatabaseImporter:167 Lock file /srv/exonerator.torproject.org/exonerator/exonerator-lock is less than 6 hours old. Either make sure that there are no other ExoneraTor database importers running and manually delete that file, or wait until the file is 6 hours old when it will be overwritten automatically. Exiting.

1 0

Cron <exonerator@materculae> cd /srv/exonerator.torproject.org/exonerator/ && ~exonerator/bin/run-exonerator
by root＠materculae.torproject.org 14 Dec '23

14 Dec '23

2023-12-14 17:22:02,400 ERROR o.t.m.e.ExoneraTorDatabaseImporter:167 Lock file /srv/exonerator.torproject.org/exonerator/exonerator-lock is less than 6 hours old. Either make sure that there are no other ExoneraTor database importers running and manually delete that file, or wait until the file is 6 hours old when it will be overwritten automatically. Exiting.

1 0

Cron <exonerator@materculae> cd /srv/exonerator.torproject.org/exonerator/ && ~exonerator/bin/run-exonerator
by root＠materculae.torproject.org 14 Dec '23

14 Dec '23

2023-12-14 16:22:02,702 ERROR o.t.m.e.ExoneraTorDatabaseImporter:167 Lock file /srv/exonerator.torproject.org/exonerator/exonerator-lock is less than 6 hours old. Either make sure that there are no other ExoneraTor database importers running and manually delete that file, or wait until the file is 6 hours old when it will be overwritten automatically. Exiting.

1 0

[FIRING:2] Relays per Flag MEDIUM
by alertmanager＠hetzner-nbg1-02.torproject.org 14 Dec '23

14 Dec '23

Total firing alerts: 2 Total resolved alerts: 0 ## Firing Alerts ----- Time: 2023-12-14 16:10:42.544 +0000 UTC Summary: More than 50 relays have joined the network with "Guard" flag Description: More than 50 relays with "Guard" flag have joined the network in the last 2 hours. ----- ----- Time: 2023-12-14 16:10:42.544 +0000 UTC Summary: More than 50 relays have joined the network with "Stable" flag Description: More than 50 relays with "Stable" flag have joined the network in the last 2 hours. ----- ##Resolved Alerts

1 0

Cron <exonerator@materculae> cd /srv/exonerator.torproject.org/exonerator/ && ~exonerator/bin/run-exonerator
by root＠materculae.torproject.org 14 Dec '23

14 Dec '23

2023-12-14 15:24:15,748 ERROR o.t.m.e.ExoneraTorDatabaseImporter:413 Caught an SQL exception while importing status entry with base64-encoded fingerprint vkwsR5lAgzt/kip4AmIUlFEuVGM and valid-after time 2023-12-14T15:00. Check the exception for details. Exiting. org.postgresql.util.PSQLException: FATAL: terminating connection due to administrator command at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2440) at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2183) at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:308) at org.postgresql.jdbc.PgStatement.executeInternal(PgStatement.java:441) at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:365) at org.postgresql.jdbc.PgPreparedStatement.executeWithFlags(PgPreparedStatement.java:143) at org.postgresql.jdbc.PgCallableStatement.executeWithFlags(PgCallableStatement.java:77) at org.postgresql.jdbc.PgPreparedStatement.execute(PgPreparedStatement.java:132) at org.torproject.metrics.exonerator.ExoneraTorDatabaseImporter.importStatusentry(ExoneraTorDatabaseImporter.java:410) at org.torproject.metrics.exonerator.ExoneraTorDatabaseImporter.parseConsensus(ExoneraTorDatabaseImporter.java:338) at org.torproject.metrics.exonerator.ExoneraTorDatabaseImporter.parseDescriptors(ExoneraTorDatabaseImporter.java:276) at org.torproject.metrics.exonerator.ExoneraTorDatabaseImporter.main(ExoneraTorDatabaseImporter.java:68)

1 0

** RECOVERY Service Alert: onionoo-backend-01/network service - onionoo varnish is OK **
by nagios＠hetzner-hel1-01.torproject.org 14 Dec '23

14 Dec '23

***** Icinga ***** Notification Type: RECOVERY Service: network service - onionoo varnish Host: onionoo-backend-01 Address: 116.202.120.163 State: OK Date/Time: Thu Dec 14 14:36:27 UTC 2023 Additional Info: OK: Valid response with recent timestamps (2586 sec): relays_published=2023-12-14 14:00:00, bridges_published=2023-12-14 13:53:17

1 0