December 2023 - metrics-alerts - lists.torproject.org

** PROBLEM Service Alert: onionoo-frontend-02/network service - onionoo full service check is WARNING **
by nagios＠hetzner-hel1-01.torproject.org 07 Dec '23

07 Dec '23

***** Icinga ***** Notification Type: PROBLEM Service: network service - onionoo full service check Host: onionoo-frontend-02 Address: 49.12.57.137 State: WARNING Date/Time: Thu Dec 7 00:03:49 UTC 2023 Additional Info: WARNING: Either timestamp is not recent (11581 sec): relays_published=2023-12-06 21:00:00, bridges_published=2023-12-06 20:50:44

1 0

** PROBLEM Service Alert: onionoo-backend-02/network service - onionoo varnish is WARNING **
by nagios＠hetzner-hel1-01.torproject.org 07 Dec '23

07 Dec '23

***** Icinga ***** Notification Type: PROBLEM Service: network service - onionoo varnish Host: onionoo-backend-02 Address: 116.202.120.178 State: WARNING Date/Time: Thu Dec 7 00:03:29 UTC 2023 Additional Info: WARNING: Either timestamp is not recent (11555 sec): relays_published=2023-12-06 21:00:00, bridges_published=2023-12-06 20:50:44

1 0

** PROBLEM Service Alert: onionoo-frontend-02/network service - onionoo varnish is WARNING **
by nagios＠hetzner-hel1-01.torproject.org 07 Dec '23

07 Dec '23

***** Icinga ***** Notification Type: PROBLEM Service: network service - onionoo varnish Host: onionoo-frontend-02 Address: 49.12.57.137 State: WARNING Date/Time: Thu Dec 7 00:03:29 UTC 2023 Additional Info: WARNING: Either timestamp is not recent (11558 sec): relays_published=2023-12-06 21:00:00, bridges_published=2023-12-06 20:50:44

1 0

** PROBLEM Service Alert: onionoo-backend-02/network service - onionoo backend is WARNING **
by nagios＠hetzner-hel1-01.torproject.org 07 Dec '23

07 Dec '23

***** Icinga ***** Notification Type: PROBLEM Service: network service - onionoo backend Host: onionoo-backend-02 Address: 116.202.120.178 State: WARNING Date/Time: Thu Dec 7 00:01:49 UTC 2023 Additional Info: WARNING: Either timestamp is not recent (11457 sec): relays_published=2023-12-06 21:00:00, bridges_published=2023-12-06 20:50:44

1 0

Cron <exonerator@materculae> cd /srv/exonerator.torproject.org/exonerator/ && ~exonerator/bin/run-exonerator
by root＠materculae.torproject.org 06 Dec '23

06 Dec '23

2023-12-06 23:22:04,812 WARN o.t.d.i.DescriptorIndexCollector:79 Cannot fetch index file https://collector.torproject.org/index/index.json and hence cannot determine which remote files to fetch. Aborting descriptor collection. java.net.NoRouteToHostException: No route to host (Host unreachable) at java.base/java.net.PlainSocketImpl.socketConnect(Native Method) at java.base/java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:412) at java.base/java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:255) at java.base/java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:237) at java.base/java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.base/java.net.Socket.connect(Socket.java:609) at java.base/sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:305) at java.base/sun.net.NetworkClient.doConnect(NetworkClient.java:177) at java.base/sun.net.www.http.HttpClient.openServer(HttpClient.java:509) at java.base/sun.net.www.http.HttpClient.openServer(HttpClient.java:604) at java.base/sun.net.www.protocol.https.HttpsClient.<init>(HttpsClient.java:266) at java.base/sun.net.www.protocol.https.HttpsClient.New(HttpsClient.java:373) at java.base/sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.getNewHttpClient(AbstractDelegateHttpsURLConnection.java:207) at java.base/sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1187) at java.base/sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1081) at java.base/sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:193) at java.base/sun.net.www.protocol.https.HttpsURLConnectionImpl.connect(HttpsURLConnectionImpl.java:168) at org.torproject.descriptor.index.IndexNode.fetchIndex(IndexNode.java:110) at org.torproject.descriptor.index.DescriptorIndexCollector.collectDescriptors(DescriptorIndexCollector.java:76) at org.torproject.metrics.exonerator.ExoneraTorDatabaseImporter.fetchDescriptors(ExoneraTorDatabaseImporter.java:201) at org.torproject.metrics.exonerator.ExoneraTorDatabaseImporter.main(ExoneraTorDatabaseImporter.java:65)

1 0

** PROBLEM Service Alert: global/collector is UNKNOWN **
by nagios＠hetzner-hel1-01.torproject.org 06 Dec '23

06 Dec '23

***** Icinga ***** Notification Type: PROBLEM Service: collector Host: global Address: global State: UNKNOWN Date/Time: Wed Dec 6 22:33:09 UTC 2023 Additional Info: (Service Check Timed Out)

1 0

** RECOVERY Service Alert: onionoo-frontend-01/network service - onionoo full service check is OK **
by nagios＠hetzner-hel1-01.torproject.org 06 Dec '23

06 Dec '23

***** Icinga ***** Notification Type: RECOVERY Service: network service - onionoo full service check Host: onionoo-frontend-01 Address: 116.202.120.171 State: OK Date/Time: Wed Dec 6 22:24:09 UTC 2023 Additional Info: OK: Valid response with recent timestamps (5601 sec): relays_published=2023-12-06 21:00:00, bridges_published=2023-12-06 20:50:44

1 0

** RECOVERY Service Alert: onionoo-frontend-01/network service - onionoo varnish is OK **
by nagios＠hetzner-hel1-01.torproject.org 06 Dec '23

06 Dec '23

***** Icinga ***** Notification Type: RECOVERY Service: network service - onionoo varnish Host: onionoo-frontend-01 Address: 116.202.120.171 State: OK Date/Time: Wed Dec 6 22:23:49 UTC 2023 Additional Info: OK: Valid response with recent timestamps (5579 sec): relays_published=2023-12-06 21:00:00, bridges_published=2023-12-06 20:50:44

1 0

** RECOVERY Service Alert: onionoo-frontend-02/network service - onionoo full service check is OK **
by nagios＠hetzner-hel1-01.torproject.org 06 Dec '23

06 Dec '23

***** Icinga ***** Notification Type: RECOVERY Service: network service - onionoo full service check Host: onionoo-frontend-02 Address: 49.12.57.137 State: OK Date/Time: Wed Dec 6 22:23:49 UTC 2023 Additional Info: OK: Valid response with recent timestamps (5581 sec): relays_published=2023-12-06 21:00:00, bridges_published=2023-12-06 20:50:44

1 0

** RECOVERY Service Alert: onionoo-backend-02/network service - onionoo varnish is OK **
by nagios＠hetzner-hel1-01.torproject.org 06 Dec '23

06 Dec '23

***** Icinga ***** Notification Type: RECOVERY Service: network service - onionoo varnish Host: onionoo-backend-02 Address: 116.202.120.178 State: OK Date/Time: Wed Dec 6 22:23:29 UTC 2023 Additional Info: OK: Valid response with recent timestamps (5555 sec): relays_published=2023-12-06 21:00:00, bridges_published=2023-12-06 20:50:44

1 0