May 2022 - metrics-alerts - lists.torproject.org

Cron <exonerator@materculae> cd /srv/exonerator.torproject.org/exonerator/ && ~exonerator/bin/run-exonerator
by root＠materculae.torproject.org 19 May '22

19 May '22

2022-05-19 07:22:02,163 ERROR o.t.m.e.ExoneraTorDatabaseImporter:128 Caught an SQL exception while connecting to the database. Make sure that the database exists and that the configured JDBC string is correct. org.postgresql.util.PSQLException: Connection to localhost:5432 refused. Check that the hostname and port are correct and that the postmaster is accepting TCP/IP connections. at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:280) at org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:49) at org.postgresql.jdbc.PgConnection.<init>(PgConnection.java:195) at org.postgresql.Driver.makeConnection(Driver.java:454) at org.postgresql.Driver.connect(Driver.java:256) at java.sql/java.sql.DriverManager.getConnection(DriverManager.java:677) at java.sql/java.sql.DriverManager.getConnection(DriverManager.java:251) at org.torproject.metrics.exonerator.ExoneraTorDatabaseImporter.openDatabaseConnection(ExoneraTorDatabaseImporter.java:126) at org.torproject.metrics.exonerator.ExoneraTorDatabaseImporter.main(ExoneraTorDatabaseImporter.java:61) Caused by: java.net.ConnectException: Connection refused (Connection refused) at java.base/java.net.PlainSocketImpl.socketConnect(Native Method) at java.base/java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:412) at java.base/java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:255) at java.base/java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:237) at java.base/java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.base/java.net.Socket.connect(Socket.java:609) at org.postgresql.core.PGStream.<init>(PGStream.java:70) at org.postgresql.core.v3.ConnectionFactoryImpl.tryConnect(ConnectionFactoryImpl.java:91) at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:192) ... 8 common frames omitted

1 0

Cron <exonerator@materculae> cd /srv/exonerator.torproject.org/exonerator/ && ~exonerator/bin/run-exonerator
by root＠materculae.torproject.org 19 May '22

19 May '22

2022-05-19 06:26:17,566 ERROR o.t.m.e.ExoneraTorDatabaseImporter:413 Caught an SQL exception while importing status entry with base64-encoded fingerprint ClR2YlusBXOoL8tDKs7XDh7YPpM and valid-after time 2022-05-19T06:00. Check the exception for details. Exiting. org.postgresql.util.PSQLException: An I/O error occurred while sending to the backend. at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:335) at org.postgresql.jdbc.PgStatement.executeInternal(PgStatement.java:441) at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:365) at org.postgresql.jdbc.PgPreparedStatement.executeWithFlags(PgPreparedStatement.java:143) at org.postgresql.jdbc.PgCallableStatement.executeWithFlags(PgCallableStatement.java:77) at org.postgresql.jdbc.PgPreparedStatement.execute(PgPreparedStatement.java:132) at org.torproject.metrics.exonerator.ExoneraTorDatabaseImporter.importStatusentry(ExoneraTorDatabaseImporter.java:410) at org.torproject.metrics.exonerator.ExoneraTorDatabaseImporter.parseConsensus(ExoneraTorDatabaseImporter.java:338) at org.torproject.metrics.exonerator.ExoneraTorDatabaseImporter.parseDescriptors(ExoneraTorDatabaseImporter.java:276) at org.torproject.metrics.exonerator.ExoneraTorDatabaseImporter.main(ExoneraTorDatabaseImporter.java:68) Caused by: java.io.EOFException: null at org.postgresql.core.PGStream.receiveChar(PGStream.java:308) at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1952) at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:308) ... 9 common frames omitted

1 0

** PROBLEM Service Alert: global/collector is CRITICAL **
by nagios＠hetzner-hel1-01.torproject.org 19 May '22

19 May '22

***** Icinga ***** Notification Type: PROBLEM Service: collector Host: global Address: global State: CRITICAL Date/Time: Thu May 19 02:13:01 UTC 2022 Additional Info: CRITICAL: Got a valid response from https://collector.torproject.org/index/index.json with at least one timestamp being critically outdated: /recent/bridge-descriptors/extra-infos/ = 2022-05-18 20:09 (6 = 3 + 3 hours ago), /recent/bridge-descriptors/server-descriptors/ = 2022-05-18 20:09 (6 = 3 + 3 hours ago), /recent/bridge-descriptors/statuses/ = 2022-05-18 20:09 (6 = 3 + 3 hours ago)

1 0

** PROBLEM Service Alert: onionoo-frontend-02/network service - onionoo full service check is CRITICAL **
by nagios＠hetzner-hel1-01.torproject.org 19 May '22

19 May '22

***** Icinga ***** Notification Type: PROBLEM Service: network service - onionoo full service check Host: onionoo-frontend-02 Address: 49.12.57.137 State: CRITICAL Date/Time: Thu May 19 01:28:01 UTC 2022 Additional Info: CRITICAL: Either timestamp is too old (22388 sec): relays_published=2022-05-19 01:00:00, bridges_published=2022-05-18 19:14:43

1 0

** PROBLEM Service Alert: onionoo-backend-01/network service - onionoo varnish is CRITICAL **
by nagios＠hetzner-hel1-01.torproject.org 19 May '22

19 May '22

***** Icinga ***** Notification Type: PROBLEM Service: network service - onionoo varnish Host: onionoo-backend-01 Address: 116.202.120.163 State: CRITICAL Date/Time: Thu May 19 01:24:11 UTC 2022 Additional Info: CRITICAL: Either timestamp is too old (22160 sec): relays_published=2022-05-19 00:00:00, bridges_published=2022-05-18 19:14:43

1 0

** PROBLEM Service Alert: onionoo-backend-02/network service - onionoo varnish is CRITICAL **
by nagios＠hetzner-hel1-01.torproject.org 19 May '22

19 May '22

***** Icinga ***** Notification Type: PROBLEM Service: network service - onionoo varnish Host: onionoo-backend-02 Address: 116.202.120.178 State: CRITICAL Date/Time: Thu May 19 01:21:41 UTC 2022 Additional Info: CRITICAL: Either timestamp is too old (22013 sec): relays_published=2022-05-19 00:00:00, bridges_published=2022-05-18 19:14:43

1 0

** PROBLEM Service Alert: onionoo-backend-01/network service - onionoo backend is CRITICAL **
by nagios＠hetzner-hel1-01.torproject.org 19 May '22

19 May '22

***** Icinga ***** Notification Type: PROBLEM Service: network service - onionoo backend Host: onionoo-backend-01 Address: 116.202.120.163 State: CRITICAL Date/Time: Thu May 19 01:21:01 UTC 2022 Additional Info: CRITICAL: Either timestamp is too old (21974 sec): relays_published=2022-05-19 00:00:00, bridges_published=2022-05-18 19:14:43

1 0

** PROBLEM Service Alert: onionoo-frontend-02/network service - onionoo varnish is CRITICAL **
by nagios＠hetzner-hel1-01.torproject.org 19 May '22

19 May '22

***** Icinga ***** Notification Type: PROBLEM Service: network service - onionoo varnish Host: onionoo-frontend-02 Address: 49.12.57.137 State: CRITICAL Date/Time: Thu May 19 01:20:31 UTC 2022 Additional Info: CRITICAL: Either timestamp is too old (21947 sec): relays_published=2022-05-19 00:00:00, bridges_published=2022-05-18 19:14:43

1 0

** PROBLEM Service Alert: onionoo-frontend-01/network service - onionoo full service check is CRITICAL **
by nagios＠hetzner-hel1-01.torproject.org 19 May '22

19 May '22

***** Icinga ***** Notification Type: PROBLEM Service: network service - onionoo full service check Host: onionoo-frontend-01 Address: 116.202.120.171 State: CRITICAL Date/Time: Thu May 19 01:19:51 UTC 2022 Additional Info: CRITICAL: Either timestamp is too old (21898 sec): relays_published=2022-05-19 00:00:00, bridges_published=2022-05-18 19:14:43

1 0

** PROBLEM Service Alert: onionoo-backend-02/network service - onionoo backend is CRITICAL **
by nagios＠hetzner-hel1-01.torproject.org 19 May '22

19 May '22

***** Icinga ***** Notification Type: PROBLEM Service: network service - onionoo backend Host: onionoo-backend-02 Address: 116.202.120.178 State: CRITICAL Date/Time: Thu May 19 01:17:01 UTC 2022 Additional Info: CRITICAL: Either timestamp is too old (21729 sec): relays_published=2022-05-19 00:00:00, bridges_published=2022-05-18 19:14:43

1 0