December 2023 - metrics-alerts - lists.torproject.org

** RECOVERY Service Alert: onionoo-frontend-02/network service - onionoo varnish is OK **
by nagios＠hetzner-hel1-01.torproject.org 06 Dec '23

06 Dec '23

***** Icinga ***** Notification Type: RECOVERY Service: network service - onionoo varnish Host: onionoo-frontend-02 Address: 49.12.57.137 State: OK Date/Time: Wed Dec 6 22:23:29 UTC 2023 Additional Info: OK: Valid response with recent timestamps (5558 sec): relays_published=2023-12-06 21:00:00, bridges_published=2023-12-06 20:50:44

1 0

Cron <exonerator@materculae> cd /srv/exonerator.torproject.org/exonerator/ && ~exonerator/bin/run-exonerator
by root＠materculae.torproject.org 06 Dec '23

06 Dec '23

2023-12-06 22:22:20,648 WARN o.t.d.i.DescriptorIndexCollector:79 Cannot fetch index file https://collector.torproject.org/index/index.json and hence cannot determine which remote files to fetch. Aborting descriptor collection. java.net.NoRouteToHostException: No route to host (Host unreachable) at java.base/java.net.PlainSocketImpl.socketConnect(Native Method) at java.base/java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:412) at java.base/java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:255) at java.base/java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:237) at java.base/java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.base/java.net.Socket.connect(Socket.java:609) at java.base/sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:305) at java.base/sun.net.NetworkClient.doConnect(NetworkClient.java:177) at java.base/sun.net.www.http.HttpClient.openServer(HttpClient.java:509) at java.base/sun.net.www.http.HttpClient.openServer(HttpClient.java:604) at java.base/sun.net.www.protocol.https.HttpsClient.<init>(HttpsClient.java:266) at java.base/sun.net.www.protocol.https.HttpsClient.New(HttpsClient.java:373) at java.base/sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.getNewHttpClient(AbstractDelegateHttpsURLConnection.java:207) at java.base/sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1187) at java.base/sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1081) at java.base/sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:193) at java.base/sun.net.www.protocol.https.HttpsURLConnectionImpl.connect(HttpsURLConnectionImpl.java:168) at org.torproject.descriptor.index.IndexNode.fetchIndex(IndexNode.java:110) at org.torproject.descriptor.index.DescriptorIndexCollector.collectDescriptors(DescriptorIndexCollector.java:76) at org.torproject.metrics.exonerator.ExoneraTorDatabaseImporter.fetchDescriptors(ExoneraTorDatabaseImporter.java:201) at org.torproject.metrics.exonerator.ExoneraTorDatabaseImporter.main(ExoneraTorDatabaseImporter.java:65)

1 0

** RECOVERY Service Alert: onionoo-backend-02/network service - onionoo backend is OK **
by nagios＠hetzner-hel1-01.torproject.org 06 Dec '23

06 Dec '23

***** Icinga ***** Notification Type: RECOVERY Service: network service - onionoo backend Host: onionoo-backend-02 Address: 116.202.120.178 State: OK Date/Time: Wed Dec 6 22:21:49 UTC 2023 Additional Info: OK: Valid response with recent timestamps (5457 sec): relays_published=2023-12-06 21:00:00, bridges_published=2023-12-06 20:50:44

1 0

** PROBLEM Service Alert: onionoo-frontend-01/network service - onionoo full service check is WARNING **
by nagios＠hetzner-hel1-01.torproject.org 06 Dec '23

06 Dec '23

***** Icinga ***** Notification Type: PROBLEM Service: network service - onionoo full service check Host: onionoo-frontend-01 Address: 116.202.120.171 State: WARNING Date/Time: Wed Dec 6 21:09:09 UTC 2023 Additional Info: WARNING: Either timestamp is not recent (11901 sec): relays_published=2023-12-06 18:00:00, bridges_published=2023-12-06 17:50:44

1 0

** PROBLEM Service Alert: onionoo-frontend-01/network service - onionoo varnish is WARNING **
by nagios＠hetzner-hel1-01.torproject.org 06 Dec '23

06 Dec '23

***** Icinga ***** Notification Type: PROBLEM Service: network service - onionoo varnish Host: onionoo-frontend-01 Address: 116.202.120.171 State: WARNING Date/Time: Wed Dec 6 21:08:49 UTC 2023 Additional Info: WARNING: Either timestamp is not recent (11879 sec): relays_published=2023-12-06 18:00:00, bridges_published=2023-12-06 17:50:44

1 0

** PROBLEM Service Alert: onionoo-frontend-02/network service - onionoo full service check is WARNING **
by nagios＠hetzner-hel1-01.torproject.org 06 Dec '23

06 Dec '23

***** Icinga ***** Notification Type: PROBLEM Service: network service - onionoo full service check Host: onionoo-frontend-02 Address: 49.12.57.137 State: WARNING Date/Time: Wed Dec 6 21:08:49 UTC 2023 Additional Info: WARNING: Either timestamp is not recent (11881 sec): relays_published=2023-12-06 18:00:00, bridges_published=2023-12-06 17:50:44

1 0

** PROBLEM Service Alert: onionoo-backend-02/network service - onionoo varnish is WARNING **
by nagios＠hetzner-hel1-01.torproject.org 06 Dec '23

06 Dec '23

***** Icinga ***** Notification Type: PROBLEM Service: network service - onionoo varnish Host: onionoo-backend-02 Address: 116.202.120.178 State: WARNING Date/Time: Wed Dec 6 21:08:29 UTC 2023 Additional Info: WARNING: Either timestamp is not recent (11855 sec): relays_published=2023-12-06 18:00:00, bridges_published=2023-12-06 17:50:44

1 0

** PROBLEM Service Alert: onionoo-frontend-02/network service - onionoo varnish is WARNING **
by nagios＠hetzner-hel1-01.torproject.org 06 Dec '23

06 Dec '23

***** Icinga ***** Notification Type: PROBLEM Service: network service - onionoo varnish Host: onionoo-frontend-02 Address: 49.12.57.137 State: WARNING Date/Time: Wed Dec 6 21:08:29 UTC 2023 Additional Info: WARNING: Either timestamp is not recent (11858 sec): relays_published=2023-12-06 18:00:00, bridges_published=2023-12-06 17:50:44

1 0

** PROBLEM Service Alert: onionoo-backend-02/network service - onionoo backend is WARNING **
by nagios＠hetzner-hel1-01.torproject.org 06 Dec '23

06 Dec '23

***** Icinga ***** Notification Type: PROBLEM Service: network service - onionoo backend Host: onionoo-backend-02 Address: 116.202.120.178 State: WARNING Date/Time: Wed Dec 6 21:06:49 UTC 2023 Additional Info: WARNING: Either timestamp is not recent (11757 sec): relays_published=2023-12-06 18:00:00, bridges_published=2023-12-06 17:50:44

1 0

** RECOVERY Service Alert: global/collector is OK **
by nagios＠hetzner-hel1-01.torproject.org 06 Dec '23

06 Dec '23

***** Icinga ***** Notification Type: RECOVERY Service: collector Host: global Address: global State: OK Date/Time: Wed Dec 6 21:05:09 UTC 2023 Additional Info: OK: Got a valid response from https://collector.torproject.org/index/index.json with recent timestamps: /archive/bridgedb-metrics/ = 2023-12-06 09:23 (11 72 hours ago), /archive/bridge-descriptors/extra-infos/ = 2023-12-06 09:46 (11 72 hours ago), /archive/bridge-descriptors/server-descriptors/ = 2023-12-06 09:20 (11 72 hours ago), /archive/bridge-descriptors/statuses/ = 2023-12-06 08:22 (12 72 hours ago), /archive/bridge-pool-assignments/ = 2023-12-06 09:23 (11 72 hours ago), /archive/exit-lists/ = 2023-12-06 03:27 (17 72 hours ago), /archive/onionperf/ = 2023-12-06 03:27 (17 72 hours ago), /archive/relay-descriptors/ = 2023-12-06 03:27 (17 720 hours ago), /archive/relay-descriptors/bandwidths/ = 2023-12-06 06:51 (14 72 hours ago), /archive/relay-descriptors/consensuses/ = 2023-12-06 03:36 (17 72 hours ago), /archive/relay-descriptors/extra-infos/ = 2023-12-06 08:23 (12 72 hours ago), /archive/relay-descriptors/microdescs/ = 2023-12-06 03:35 (17 72 hours ago), /archive/r elay-descriptors/server-descriptors/ = 2023-12-06 08:14 (12 72 hours ago), /archive/relay-descriptors/votes/ = 2023-12-06 06:25 (14 72 hours ago), /archive/snowflakes/ = 2023-12-06 09:20 (11 72 hours ago), /archive/webstats/ = 2023-12-06 09:47 (11 72 hours ago), /recent/bridgedb-metrics/ = 2023-12-06 05:40 (15 60 hours ago), /recent/bridge-descriptors/extra-infos/ = 2023-12-06 19:09 (1 3 hours ago), /recent/bridge-descriptors/server-descriptors/ = 2023-12-06 19:09 (1 3 hours ago), /recent/bridge-descriptors/statuses/ = 2023-12-06 19:09 (1 3 hours ago), /recent/bridge-pool-assignments/ = 2023-12-06 00:09 (20 30 hours ago), /recent/exit-lists/ = 2023-12-06 19:02 (2 3 hours ago), /recent/onionperf/ = 2023-12-06 14:10 (6 30 hours ago), /recent/relay-descriptors/bandwidths/ = 2023-12-06 19:37 (1 3 hours ago), /recent/relay-descriptors/consensuses/ = 2023-12-06 19:05 (2 3 hours ago), /recent/relay-descriptors/extra-infos/ = 2023-12-06 19:06 (1 3 hours ago), /recent/relay-des criptors/microdescs/consensus-microdesc/ = 2023-12-06 19:05 (2 3 hours ago), /recent/relay-descriptors/microdescs/micro/ = 2023-12-06 19:05 (2 3 hours ago), /recent/relay-descriptors/server-descriptors/ = 2023-12-06 19:05 (2 3 hours ago), /recent/relay-descriptors/votes/ = 2023-12-06 19:05 (2 3 hours ago), /recent/snowflakes/ = 2023-12-06 17:40 (3 30 hours ago), /recent/webstats/ = 2023-12-06 13:41 (7 30 hours ago)

1 0