December 2023 - metrics-alerts - lists.torproject.org

** RECOVERY Service Alert: onionoo-frontend-02/network service - onionoo full service check is OK **
by nagios＠hetzner-hel1-01.torproject.org 07 Dec '23

07 Dec '23

***** Icinga ***** Notification Type: RECOVERY Service: network service - onionoo full service check Host: onionoo-frontend-02 Address: 49.12.57.137 State: OK Date/Time: Thu Dec 7 14:18:49 UTC 2023 Additional Info: OK: Valid response with recent timestamps (5280 sec): relays_published=2023-12-07 13:00:00, bridges_published=2023-12-07 12:50:45

1 0

** RECOVERY Service Alert: onionoo-backend-02/network service - onionoo varnish is OK **
by nagios＠hetzner-hel1-01.torproject.org 07 Dec '23

07 Dec '23

***** Icinga ***** Notification Type: RECOVERY Service: network service - onionoo varnish Host: onionoo-backend-02 Address: 116.202.120.178 State: OK Date/Time: Thu Dec 7 14:18:29 UTC 2023 Additional Info: OK: Valid response with recent timestamps (5254 sec): relays_published=2023-12-07 13:00:00, bridges_published=2023-12-07 12:50:45

1 0

** RECOVERY Service Alert: onionoo-frontend-02/network service - onionoo varnish is OK **
by nagios＠hetzner-hel1-01.torproject.org 07 Dec '23

07 Dec '23

***** Icinga ***** Notification Type: RECOVERY Service: network service - onionoo varnish Host: onionoo-frontend-02 Address: 49.12.57.137 State: OK Date/Time: Thu Dec 7 14:18:29 UTC 2023 Additional Info: OK: Valid response with recent timestamps (5257 sec): relays_published=2023-12-07 13:00:00, bridges_published=2023-12-07 12:50:45

1 0

** RECOVERY Service Alert: onionoo-backend-02/network service - onionoo backend is OK **
by nagios＠hetzner-hel1-01.torproject.org 07 Dec '23

07 Dec '23

***** Icinga ***** Notification Type: RECOVERY Service: network service - onionoo backend Host: onionoo-backend-02 Address: 116.202.120.178 State: OK Date/Time: Thu Dec 7 14:16:49 UTC 2023 Additional Info: OK: Valid response with recent timestamps (5156 sec): relays_published=2023-12-07 13:00:00, bridges_published=2023-12-07 12:50:45

1 0

** RECOVERY Service Alert: global/collector is OK **
by nagios＠hetzner-hel1-01.torproject.org 07 Dec '23

07 Dec '23

***** Icinga ***** Notification Type: RECOVERY Service: collector Host: global Address: global State: OK Date/Time: Thu Dec 7 13:17:09 UTC 2023 Additional Info: OK: Got a valid response from https://collector.torproject.org/index/index.json with recent timestamps: /archive/bridgedb-metrics/ = 2023-12-06 09:23 (27 72 hours ago), /archive/bridge-descriptors/extra-infos/ = 2023-12-06 09:46 (27 72 hours ago), /archive/bridge-descriptors/server-descriptors/ = 2023-12-06 09:20 (27 72 hours ago), /archive/bridge-descriptors/statuses/ = 2023-12-06 08:22 (28 72 hours ago), /archive/bridge-pool-assignments/ = 2023-12-06 09:23 (27 72 hours ago), /archive/exit-lists/ = 2023-12-06 03:27 (33 72 hours ago), /archive/onionperf/ = 2023-12-06 03:27 (33 72 hours ago), /archive/relay-descriptors/ = 2023-12-06 03:27 (33 720 hours ago), /archive/relay-descriptors/bandwidths/ = 2023-12-06 06:51 (30 72 hours ago), /archive/relay-descriptors/consensuses/ = 2023-12-06 03:36 (33 72 hours ago), /archive/relay-descriptors/extra-infos/ = 2023-12-06 08:23 (28 72 hours ago), /archive/relay-descriptors/microdescs/ = 2023-12-06 03:35 (33 72 hours ago), /archive/r elay-descriptors/server-descriptors/ = 2023-12-06 08:14 (29 72 hours ago), /archive/relay-descriptors/votes/ = 2023-12-06 06:25 (30 72 hours ago), /archive/snowflakes/ = 2023-12-06 09:20 (27 72 hours ago), /archive/webstats/ = 2023-12-06 09:47 (27 72 hours ago), /recent/bridgedb-metrics/ = 2023-12-06 05:40 (31 60 hours ago), /recent/bridge-descriptors/extra-infos/ = 2023-12-07 13:10 (0 3 hours ago), /recent/bridge-descriptors/server-descriptors/ = 2023-12-07 13:10 (0 3 hours ago), /recent/bridge-descriptors/statuses/ = 2023-12-07 13:10 (0 3 hours ago), /recent/bridge-pool-assignments/ = 2023-12-07 13:09 (0 30 hours ago), /recent/exit-lists/ = 2023-12-07 13:02 (0 3 hours ago), /recent/onionperf/ = 2023-12-06 14:10 (23 30 hours ago), /recent/relay-descriptors/bandwidths/ = 2023-12-07 13:09 (0 3 hours ago), /recent/relay-descriptors/consensuses/ = 2023-12-07 13:05 (0 3 hours ago), /recent/relay-descriptors/extra-infos/ = 2023-12-07 13:09 (0 3 hours ago), /recent/relay-des criptors/microdescs/consensus-microdesc/ = 2023-12-07 13:09 (0 3 hours ago), /recent/relay-descriptors/microdescs/micro/ = 2023-12-07 13:09 (0 3 hours ago), /recent/relay-descriptors/server-descriptors/ = 2023-12-07 13:09 (0 3 hours ago), /recent/relay-descriptors/votes/ = 2023-12-07 13:09 (0 3 hours ago), /recent/snowflakes/ = 2023-12-06 17:40 (19 30 hours ago), /recent/webstats/ = 2023-12-06 13:41 (23 30 hours ago)

1 0

** PROBLEM Service Alert: global/collector is CRITICAL **
by nagios＠hetzner-hel1-01.torproject.org 07 Dec '23

07 Dec '23

***** Icinga ***** Notification Type: PROBLEM Service: collector Host: global Address: global State: CRITICAL Date/Time: Thu Dec 7 12:47:09 UTC 2023 Additional Info: CRITICAL: Got a valid response from https://collector.torproject.org/index/index.json with at least one timestamp being critically outdated: /recent/bridge-descriptors/extra-infos/ = 2023-12-06 22:09 (14 = 3 + 3 hours ago), /recent/bridge-descriptors/server-descriptors/ = 2023-12-06 22:09 (14 = 3 + 3 hours ago), /recent/bridge-descriptors/statuses/ = 2023-12-06 22:09 (14 = 3 + 3 hours ago), /recent/bridge-pool-assignments/ = 2023-12-06 00:09 (36 = 30 + 3 hours ago), /recent/exit-lists/ = 2023-12-06 22:02 (14 = 3 + 3 hours ago)

1 0

Cron <exonerator@materculae> cd /srv/exonerator.torproject.org/exonerator/ && ~exonerator/bin/run-exonerator
by root＠materculae.torproject.org 07 Dec '23

07 Dec '23

2023-12-07 12:22:04,877 WARN o.t.d.i.DescriptorIndexCollector:79 Cannot fetch index file https://collector.torproject.org/index/index.json and hence cannot determine which remote files to fetch. Aborting descriptor collection. java.net.NoRouteToHostException: No route to host (Host unreachable) at java.base/java.net.PlainSocketImpl.socketConnect(Native Method) at java.base/java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:412) at java.base/java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:255) at java.base/java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:237) at java.base/java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.base/java.net.Socket.connect(Socket.java:609) at java.base/sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:305) at java.base/sun.net.NetworkClient.doConnect(NetworkClient.java:177) at java.base/sun.net.www.http.HttpClient.openServer(HttpClient.java:509) at java.base/sun.net.www.http.HttpClient.openServer(HttpClient.java:604) at java.base/sun.net.www.protocol.https.HttpsClient.<init>(HttpsClient.java:266) at java.base/sun.net.www.protocol.https.HttpsClient.New(HttpsClient.java:373) at java.base/sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.getNewHttpClient(AbstractDelegateHttpsURLConnection.java:207) at java.base/sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1187) at java.base/sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1081) at java.base/sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:193) at java.base/sun.net.www.protocol.https.HttpsURLConnectionImpl.connect(HttpsURLConnectionImpl.java:168) at org.torproject.descriptor.index.IndexNode.fetchIndex(IndexNode.java:110) at org.torproject.descriptor.index.DescriptorIndexCollector.collectDescriptors(DescriptorIndexCollector.java:76) at org.torproject.metrics.exonerator.ExoneraTorDatabaseImporter.fetchDescriptors(ExoneraTorDatabaseImporter.java:201) at org.torproject.metrics.exonerator.ExoneraTorDatabaseImporter.main(ExoneraTorDatabaseImporter.java:65)

1 0

Cron <exonerator@materculae> cd /srv/exonerator.torproject.org/exonerator/ && ~exonerator/bin/run-exonerator
by root＠materculae.torproject.org 07 Dec '23

07 Dec '23

2023-12-07 11:22:04,872 WARN o.t.d.i.DescriptorIndexCollector:79 Cannot fetch index file https://collector.torproject.org/index/index.json and hence cannot determine which remote files to fetch. Aborting descriptor collection. java.net.NoRouteToHostException: No route to host (Host unreachable) at java.base/java.net.PlainSocketImpl.socketConnect(Native Method) at java.base/java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:412) at java.base/java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:255) at java.base/java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:237) at java.base/java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.base/java.net.Socket.connect(Socket.java:609) at java.base/sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:305) at java.base/sun.net.NetworkClient.doConnect(NetworkClient.java:177) at java.base/sun.net.www.http.HttpClient.openServer(HttpClient.java:509) at java.base/sun.net.www.http.HttpClient.openServer(HttpClient.java:604) at java.base/sun.net.www.protocol.https.HttpsClient.<init>(HttpsClient.java:266) at java.base/sun.net.www.protocol.https.HttpsClient.New(HttpsClient.java:373) at java.base/sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.getNewHttpClient(AbstractDelegateHttpsURLConnection.java:207) at java.base/sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1187) at java.base/sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1081) at java.base/sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:193) at java.base/sun.net.www.protocol.https.HttpsURLConnectionImpl.connect(HttpsURLConnectionImpl.java:168) at org.torproject.descriptor.index.IndexNode.fetchIndex(IndexNode.java:110) at org.torproject.descriptor.index.DescriptorIndexCollector.collectDescriptors(DescriptorIndexCollector.java:76) at org.torproject.metrics.exonerator.ExoneraTorDatabaseImporter.fetchDescriptors(ExoneraTorDatabaseImporter.java:201) at org.torproject.metrics.exonerator.ExoneraTorDatabaseImporter.main(ExoneraTorDatabaseImporter.java:65)

1 0

Cron <exonerator@materculae> cd /srv/exonerator.torproject.org/exonerator/ && ~exonerator/bin/run-exonerator
by root＠materculae.torproject.org 07 Dec '23

07 Dec '23

2023-12-07 10:22:05,672 WARN o.t.d.i.DescriptorIndexCollector:79 Cannot fetch index file https://collector.torproject.org/index/index.json and hence cannot determine which remote files to fetch. Aborting descriptor collection. java.net.NoRouteToHostException: No route to host (Host unreachable) at java.base/java.net.PlainSocketImpl.socketConnect(Native Method) at java.base/java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:412) at java.base/java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:255) at java.base/java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:237) at java.base/java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.base/java.net.Socket.connect(Socket.java:609) at java.base/sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:305) at java.base/sun.net.NetworkClient.doConnect(NetworkClient.java:177) at java.base/sun.net.www.http.HttpClient.openServer(HttpClient.java:509) at java.base/sun.net.www.http.HttpClient.openServer(HttpClient.java:604) at java.base/sun.net.www.protocol.https.HttpsClient.<init>(HttpsClient.java:266) at java.base/sun.net.www.protocol.https.HttpsClient.New(HttpsClient.java:373) at java.base/sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.getNewHttpClient(AbstractDelegateHttpsURLConnection.java:207) at java.base/sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1187) at java.base/sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1081) at java.base/sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:193) at java.base/sun.net.www.protocol.https.HttpsURLConnectionImpl.connect(HttpsURLConnectionImpl.java:168) at org.torproject.descriptor.index.IndexNode.fetchIndex(IndexNode.java:110) at org.torproject.descriptor.index.DescriptorIndexCollector.collectDescriptors(DescriptorIndexCollector.java:76) at org.torproject.metrics.exonerator.ExoneraTorDatabaseImporter.fetchDescriptors(ExoneraTorDatabaseImporter.java:201) at org.torproject.metrics.exonerator.ExoneraTorDatabaseImporter.main(ExoneraTorDatabaseImporter.java:65)

1 0

Cron <exonerator@materculae> cd /srv/exonerator.torproject.org/exonerator/ && ~exonerator/bin/run-exonerator
by root＠materculae.torproject.org 07 Dec '23

07 Dec '23

2023-12-07 09:22:05,736 WARN o.t.d.i.DescriptorIndexCollector:79 Cannot fetch index file https://collector.torproject.org/index/index.json and hence cannot determine which remote files to fetch. Aborting descriptor collection. java.net.NoRouteToHostException: No route to host (Host unreachable) at java.base/java.net.PlainSocketImpl.socketConnect(Native Method) at java.base/java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:412) at java.base/java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:255) at java.base/java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:237) at java.base/java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.base/java.net.Socket.connect(Socket.java:609) at java.base/sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:305) at java.base/sun.net.NetworkClient.doConnect(NetworkClient.java:177) at java.base/sun.net.www.http.HttpClient.openServer(HttpClient.java:509) at java.base/sun.net.www.http.HttpClient.openServer(HttpClient.java:604) at java.base/sun.net.www.protocol.https.HttpsClient.<init>(HttpsClient.java:266) at java.base/sun.net.www.protocol.https.HttpsClient.New(HttpsClient.java:373) at java.base/sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.getNewHttpClient(AbstractDelegateHttpsURLConnection.java:207) at java.base/sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1187) at java.base/sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1081) at java.base/sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:193) at java.base/sun.net.www.protocol.https.HttpsURLConnectionImpl.connect(HttpsURLConnectionImpl.java:168) at org.torproject.descriptor.index.IndexNode.fetchIndex(IndexNode.java:110) at org.torproject.descriptor.index.DescriptorIndexCollector.collectDescriptors(DescriptorIndexCollector.java:76) at org.torproject.metrics.exonerator.ExoneraTorDatabaseImporter.fetchDescriptors(ExoneraTorDatabaseImporter.java:201) at org.torproject.metrics.exonerator.ExoneraTorDatabaseImporter.main(ExoneraTorDatabaseImporter.java:65)

1 0