December 2022 - metrics-alerts - lists.torproject.org

** PROBLEM Service Alert: global/collector is WARNING **
by nagios＠hetzner-hel1-01.torproject.org 18 Dec '22

18 Dec '22

***** Icinga ***** Notification Type: PROBLEM Service: collector Host: global Address: global State: WARNING Date/Time: Sun Dec 18 08:35:25 UTC 2022 Additional Info: WARNING: Got a valid response from https://collector.torproject.org/index/index.json with at least one timestamp being outdated: /recent/onionperf/ = 2022-12-17 02:15 (30 = 30 hours ago)

1 0

[FIRING:3] Relays per Flag MEDIUM
by alertmanager＠hetzner-nbg1-02.torproject.org 18 Dec '22

18 Dec '22

Total firing alerts: 3 Total resolved alerts: 0 ## Firing Alerts ----- Time: 2022-12-18 08:17:27.544031668 +0000 UTC Summary: More than 50 relays have joined the network with "Fast" flag Description: More than 50 relays with "Fast" flag have joined the network in the last 2 hours. ----- ----- Time: 2022-12-18 08:17:27.544031668 +0000 UTC Summary: More than 50 relays have joined the network with "Guard" flag Description: More than 50 relays with "Guard" flag have joined the network in the last 2 hours. ----- ----- Time: 2022-12-18 08:17:27.544031668 +0000 UTC Summary: More than 50 relays have joined the network with "HSDir" flag Description: More than 50 relays with "HSDir" flag have joined the network in the last 2 hours. ----- ##Resolved Alerts

1 0

Cron <metrics@meronense> cd /srv/metrics.torproject.org/metrics/bin/website && ./run-web.sh
by root＠meronense.torproject.org 18 Dec '22

18 Dec '22

2022-12-18 06:43:35,628 WARN o.t.m.s.main.Main:76 Caught an exception when invoking the main method of the org.torproject.metrics.stats.clients.Main module. Moving on to the next module, if available. java.lang.reflect.InvocationTargetException: null at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:566) at org.torproject.metrics.stats.main.Main.main(Main.java:72) Caused by: org.postgresql.util.PSQLException: FATAL: terminating connection due to administrator command Where: while updating tuple (1247021,13) in relation "merged" SQL statement "UPDATE merged SET stats_end = last_end, val = last_val WHERE id = last_id" PL/pgSQL function merge() line 224 at SQL statement at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2440) at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2183) at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:308) at org.postgresql.jdbc.PgStatement.executeInternal(PgStatement.java:441) at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:365) at org.postgresql.jdbc.PgStatement.executeWithFlags(PgStatement.java:307) at org.postgresql.jdbc.PgStatement.executeCachedSql(PgStatement.java:293) at org.postgresql.jdbc.PgStatement.executeWithFlags(PgStatement.java:270) at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:266) at org.torproject.metrics.stats.clients.Database.processImported(Database.java:80) at org.torproject.metrics.stats.clients.Main.main(Main.java:50) ... 5 common frames omitted 2022-12-18 06:43:35,777 ERROR o.t.m.s.servers.Main:115 Cannot recover from SQL exception while querying. Not writing output file. org.postgresql.util.PSQLException: FATAL: the database system is shutting down at org.postgresql.core.v3.ConnectionFactoryImpl.doAuthentication(ConnectionFactoryImpl.java:514) at org.postgresql.core.v3.ConnectionFactoryImpl.tryConnect(ConnectionFactoryImpl.java:141) at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:192) at org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:49) at org.postgresql.jdbc.PgConnection.<init>(PgConnection.java:195) at org.postgresql.Driver.makeConnection(Driver.java:454) at org.postgresql.Driver.connect(Driver.java:256) at java.sql/java.sql.DriverManager.getConnection(DriverManager.java:677) at java.sql/java.sql.DriverManager.getConnection(DriverManager.java:251) at org.torproject.metrics.stats.servers.Database.connect(Database.java:97) at org.torproject.metrics.stats.servers.Database.<init>(Database.java:91) at org.torproject.metrics.stats.servers.Main.main(Main.java:51) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:566) at org.torproject.metrics.stats.main.Main.main(Main.java:72) 2022-12-18 06:43:35,817 WARN o.t.m.s.main.Main:76 Caught an exception when invoking the main method of the org.torproject.metrics.stats.webstats.Main module. Moving on to the next module, if available. java.lang.reflect.InvocationTargetException: null at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:566) at org.torproject.metrics.stats.main.Main.main(Main.java:72) Caused by: org.postgresql.util.PSQLException: FATAL: the database system is shutting down at org.postgresql.core.v3.ConnectionFactoryImpl.doAuthentication(ConnectionFactoryImpl.java:514) at org.postgresql.core.v3.ConnectionFactoryImpl.tryConnect(ConnectionFactoryImpl.java:141) at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:192) at org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:49) at org.postgresql.jdbc.PgConnection.<init>(PgConnection.java:195) at org.postgresql.Driver.makeConnection(Driver.java:454) at org.postgresql.Driver.connect(Driver.java:256) at java.sql/java.sql.DriverManager.getConnection(DriverManager.java:677) at java.sql/java.sql.DriverManager.getConnection(DriverManager.java:251) at org.torproject.metrics.stats.webstats.Main.connectToDatabase(Main.java:94) at org.torproject.metrics.stats.webstats.Main.main(Main.java:76) ... 5 common frames omitted 2022-12-18 06:43:35,834 ERROR o.t.m.s.totalcw.Main:84 Cannot recover from SQL exception while querying. Not writing output file. org.postgresql.util.PSQLException: FATAL: the database system is shutting down at org.postgresql.core.v3.ConnectionFactoryImpl.doAuthentication(ConnectionFactoryImpl.java:514) at org.postgresql.core.v3.ConnectionFactoryImpl.tryConnect(ConnectionFactoryImpl.java:141) at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:192) at org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:49) at org.postgresql.jdbc.PgConnection.<init>(PgConnection.java:195) at org.postgresql.Driver.makeConnection(Driver.java:454) at org.postgresql.Driver.connect(Driver.java:256) at java.sql/java.sql.DriverManager.getConnection(DriverManager.java:677) at java.sql/java.sql.DriverManager.getConnection(DriverManager.java:251) at org.torproject.metrics.stats.totalcw.Database.connect(Database.java:55) at org.torproject.metrics.stats.totalcw.Database.<init>(Database.java:50) at org.torproject.metrics.stats.totalcw.Main.main(Main.java:46) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:566) at org.torproject.metrics.stats.main.Main.main(Main.java:72) 2022-12-18 06:43:37,131 WARN o.t.m.s.main.Main:120 Unable to copy module stats file /srv/metrics.torproject.org/metrics/work/modules/onionperf/stats/latencies.csv to stats output directory /srv/metrics.torproject.org/metrics/shared/stats. Skipping. java.nio.file.FileAlreadyExistsException: /srv/metrics.torproject.org/metrics/shared/stats/latencies.csv at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:94) at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111) at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:116) at java.base/sun.nio.fs.UnixCopyFile.copyFile(UnixCopyFile.java:248) at java.base/sun.nio.fs.UnixCopyFile.copy(UnixCopyFile.java:603) at java.base/sun.nio.fs.UnixFileSystemProvider.copy(UnixFileSystemProvider.java:258) at java.base/java.nio.file.Files.copy(Files.java:1295) at org.torproject.metrics.stats.main.Main.main(Main.java:115) 2022-12-18 06:43:39,638 WARN o.t.m.s.main.Main:120 Unable to copy module stats file /srv/metrics.torproject.org/metrics/work/modules/clients/stats/userstats-combined.csv to stats output directory /srv/metrics.torproject.org/metrics/shared/stats. Skipping. java.nio.file.FileAlreadyExistsException: /srv/metrics.torproject.org/metrics/shared/stats/userstats-combined.csv at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:94) at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111) at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:116) at java.base/sun.nio.fs.UnixCopyFile.copyFile(UnixCopyFile.java:248) at java.base/sun.nio.fs.UnixCopyFile.copy(UnixCopyFile.java:603) at java.base/sun.nio.fs.UnixFileSystemProvider.copy(UnixFileSystemProvider.java:258) at java.base/java.nio.file.Files.copy(Files.java:1295) at org.torproject.metrics.stats.main.Main.main(Main.java:115)

1 0

** RECOVERY Service Alert: onionoo-frontend-02/network service - onionoo full service check is OK **
by nagios＠hetzner-hel1-01.torproject.org 18 Dec '22

18 Dec '22

***** Icinga ***** Notification Type: RECOVERY Service: network service - onionoo full service check Host: onionoo-frontend-02 Address: 49.12.57.137 State: OK Date/Time: Sun Dec 18 04:24:35 UTC 2022 Additional Info: OK: Valid response with recent timestamps (5989 sec): relays_published=2022-12-18 03:00:00, bridges_published=2022-12-18 02:44:40

1 0

** RECOVERY Service Alert: onionoo-backend-02/network service - onionoo varnish is OK **
by nagios＠hetzner-hel1-01.torproject.org 18 Dec '22

18 Dec '22

***** Icinga ***** Notification Type: RECOVERY Service: network service - onionoo varnish Host: onionoo-backend-02 Address: 116.202.120.178 State: OK Date/Time: Sun Dec 18 04:22:05 UTC 2022 Additional Info: OK: Valid response with recent timestamps (5838 sec): relays_published=2022-12-18 03:00:00, bridges_published=2022-12-18 02:44:40

1 0

** RECOVERY Service Alert: onionoo-backend-02/network service - onionoo backend is OK **
by nagios＠hetzner-hel1-01.torproject.org 18 Dec '22

18 Dec '22

***** Icinga ***** Notification Type: RECOVERY Service: network service - onionoo backend Host: onionoo-backend-02 Address: 116.202.120.178 State: OK Date/Time: Sun Dec 18 04:21:35 UTC 2022 Additional Info: OK: Valid response with recent timestamps (5806 sec): relays_published=2022-12-18 03:00:00, bridges_published=2022-12-18 02:44:40

1 0

** RECOVERY Service Alert: onionoo-frontend-01/network service - onionoo varnish is OK **
by nagios＠hetzner-hel1-01.torproject.org 18 Dec '22

18 Dec '22

***** Icinga ***** Notification Type: RECOVERY Service: network service - onionoo varnish Host: onionoo-frontend-01 Address: 116.202.120.171 State: OK Date/Time: Sun Dec 18 04:21:05 UTC 2022 Additional Info: OK: Valid response with recent timestamps (5779 sec): relays_published=2022-12-18 03:00:00, bridges_published=2022-12-18 02:44:40

1 0

** RECOVERY Service Alert: onionoo-frontend-02/network service - onionoo varnish is OK **
by nagios＠hetzner-hel1-01.torproject.org 18 Dec '22

18 Dec '22

***** Icinga ***** Notification Type: RECOVERY Service: network service - onionoo varnish Host: onionoo-frontend-02 Address: 49.12.57.137 State: OK Date/Time: Sun Dec 18 04:14:05 UTC 2022 Additional Info: OK: Valid response with recent timestamps (5362 sec): relays_published=2022-12-18 03:00:00, bridges_published=2022-12-18 02:44:40

1 0

** RECOVERY Service Alert: onionoo-backend-01/network service - onionoo varnish is OK **
by nagios＠hetzner-hel1-01.torproject.org 18 Dec '22

18 Dec '22

***** Icinga ***** Notification Type: RECOVERY Service: network service - onionoo varnish Host: onionoo-backend-01 Address: 116.202.120.163 State: OK Date/Time: Sun Dec 18 04:13:05 UTC 2022 Additional Info: OK: Valid response with recent timestamps (5304 sec): relays_published=2022-12-18 03:00:00, bridges_published=2022-12-18 02:44:40

1 0

** RECOVERY Service Alert: onionoo-backend-01/network service - onionoo backend is OK **
by nagios＠hetzner-hel1-01.torproject.org 18 Dec '22

18 Dec '22

***** Icinga ***** Notification Type: RECOVERY Service: network service - onionoo backend Host: onionoo-backend-01 Address: 116.202.120.163 State: OK Date/Time: Sun Dec 18 04:12:35 UTC 2022 Additional Info: OK: Valid response with recent timestamps (5270 sec): relays_published=2022-12-18 03:00:00, bridges_published=2022-12-18 02:44:40

1 0