- metrics-alerts - lists.torproject.org

Cron <onionoo@onionoo-backend-02> /srv/onionoo.torproject.org/onionoo/bin/service/onionoo
by root＠onionoo-backend-02.torproject.org 05 Dec '21

05 Dec '21

Traceback (most recent call last): File "/srv/onionoo.torproject.org/onionoo/bin/service/onionoo", line 23, in <module> list_of_errors = Counter([(x.split(' ')[2], x.split(' ')[3].split(':')[0]) for x in str(out).split('\\n') if x != '"']) File "/srv/onionoo.torproject.org/onionoo/bin/service/onionoo", line 23, in <listcomp> list_of_errors = Counter([(x.split(' ')[2], x.split(' ')[3].split(':')[0]) for x in str(out).split('\\n') if x != '"']) IndexError: list index out of range During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/srv/onionoo.torproject.org/onionoo/bin/service/onionoo", line 29, in <module> g_warnings.labels(host, "Problem running exporter - {}".format(e)).set(1) File "/usr/lib/python3/dist-packages/prometheus_client/metrics.py", line 146, in labels raise ValueError('Incorrect label count') ValueError: Incorrect label count

1 0

Cron <onionoo@onionoo-backend-02> [ $(expr $(date +
by root＠onionoo-backend-02.torproject.org 05 Dec '21

05 Dec '21

/bin/sh: 1: Syntax error: end of file unexpected (expecting ")")

1 0

Cron <onionoo@onionoo-backend-01> [ $(expr $(date +
by root＠onionoo-backend-01.torproject.org 05 Dec '21

05 Dec '21

/bin/sh: 1: Syntax error: end of file unexpected (expecting ")")

1 0

Cron <collector@colchicifolium> cd /srv/collector.torproject.org/collector/logs/ && cat metrics.log | grep -E "WARN|ERROR|FATAL" | grep -v "Overwriting existing descriptor file"
by root＠colchicifolium.torproject.org 04 Dec '21

04 Dec '21

2021-12-04 00:07:49,433 WARN o.t.m.c.r.ReferenceChecker:314 Missing too many referenced descriptors (54.5593436457474). 2021-12-04 00:38:48,900 WARN o.t.m.c.r.ReferenceChecker:314 Missing too many referenced descriptors (41.748946518756135). 2021-12-04 01:07:54,150 WARN o.t.m.c.r.ReferenceChecker:314 Missing too many referenced descriptors (33.9596444962831). 2021-12-04 01:38:15,749 WARN o.t.m.c.r.ReferenceChecker:314 Missing too many referenced descriptors (31.42263205949164). 2021-12-04 02:15:18,792 WARN o.t.m.c.r.ReferenceChecker:314 Missing too many referenced descriptors (32.01320038375482). 2021-12-04 02:37:46,189 WARN o.t.m.c.r.ReferenceChecker:314 Missing too many referenced descriptors (28.115779817026688). 2021-12-04 03:08:26,858 WARN o.t.m.c.r.ReferenceChecker:314 Missing too many referenced descriptors (38.13732607999664). 2021-12-04 03:39:12,533 WARN o.t.m.c.r.ReferenceChecker:314 Missing too many referenced descriptors (17.1944829366498). 2021-12-04 04:09:18,560 WARN o.t.m.c.r.ReferenceChecker:314 Missing too many referenced descriptors (21.361454507295655). 2021-12-04 04:38:35,030 WARN o.t.m.c.r.ReferenceChecker:314 Missing too many referenced descriptors (14.588616064697042). 2021-12-04 05:10:15,486 WARN o.t.m.c.r.ReferenceChecker:314 Missing too many referenced descriptors (26.78675403699796). 2021-12-04 05:45:06,729 WARN o.t.m.c.r.ReferenceChecker:314 Missing too many referenced descriptors (22.558422241149756). 2021-12-04 06:09:48,928 WARN o.t.m.c.r.ReferenceChecker:314 Missing too many referenced descriptors (29.36210650101874). 2021-12-04 06:44:53,633 WARN o.t.m.c.r.ReferenceChecker:314 Missing too many referenced descriptors (23.38366689086582). 2021-12-04 07:11:09,522 WARN o.t.m.c.r.ReferenceChecker:314 Missing too many referenced descriptors (24.224633830277863). 2021-12-04 07:38:51,948 WARN o.t.m.c.r.ReferenceChecker:314 Missing too many referenced descriptors (20.008494157924286). 2021-12-04 08:09:53,034 WARN o.t.m.c.r.ReferenceChecker:314 Missing too many referenced descriptors (20.668660724163548). 2021-12-04 08:39:28,123 WARN o.t.m.c.r.ReferenceChecker:314 Missing too many referenced descriptors (19.567860241960314). 2021-12-04 09:13:18,335 WARN o.t.m.c.r.ReferenceChecker:314 Missing too many referenced descriptors (27.942539772244). 2021-12-04 09:39:16,209 WARN o.t.m.c.r.ReferenceChecker:314 Missing too many referenced descriptors (21.277471699646494). 2021-12-04 10:10:00,143 WARN o.t.m.c.r.ReferenceChecker:314 Missing too many referenced descriptors (25.6846553699571). 2021-12-04 10:42:56,241 WARN o.t.m.c.r.ReferenceChecker:314 Missing too many referenced descriptors (24.334648322666606). 2021-12-04 11:12:04,884 WARN o.t.m.c.r.ReferenceChecker:314 Missing too many referenced descriptors (33.28373303057834). 2021-12-04 11:37:27,835 WARN o.t.m.c.r.ReferenceChecker:314 Missing too many referenced descriptors (32.6181038184192). 2021-12-04 12:09:11,372 WARN o.t.m.c.r.ReferenceChecker:314 Missing too many referenced descriptors (39.94498698356726). 2021-12-04 12:43:42,780 WARN o.t.m.c.r.ReferenceChecker:314 Missing too many referenced descriptors (39.022560686295826). 2021-12-04 13:12:14,113 WARN o.t.m.c.r.ReferenceChecker:314 Missing too many referenced descriptors (44.58066899568952). 2021-12-04 13:37:55,540 WARN o.t.m.c.r.ReferenceChecker:314 Missing too many referenced descriptors (39.946607263891615). 2021-12-04 14:10:57,024 WARN o.t.m.c.r.ReferenceChecker:314 Missing too many referenced descriptors (48.716132052202425). 2021-12-04 14:38:12,774 WARN o.t.m.c.r.ReferenceChecker:314 Missing too many referenced descriptors (45.11342573609015). 2021-12-04 15:07:33,231 WARN o.t.m.c.r.ReferenceChecker:314 Missing too many referenced descriptors (49.09153993271919). 2021-12-04 15:41:34,945 WARN o.t.m.c.r.ReferenceChecker:314 Missing too many referenced descriptors (44.824684008181315). 2021-12-04 16:09:14,482 WARN o.t.m.c.r.ReferenceChecker:314 Missing too many referenced descriptors (40.59211771951556). 2021-12-04 16:41:58,120 WARN o.t.m.c.r.ReferenceChecker:314 Missing too many referenced descriptors (36.58668575634407). 2021-12-04 17:08:36,888 WARN o.t.m.c.r.ReferenceChecker:314 Missing too many referenced descriptors (47.1716746203715). 2021-12-04 17:38:45,682 WARN o.t.m.c.r.ReferenceChecker:314 Missing too many referenced descriptors (41.84449476852599). 2021-12-04 18:08:52,774 WARN o.t.m.c.r.ReferenceChecker:314 Missing too many referenced descriptors (49.190846850860595).

1 0

Cron <metrics@meronense> cd /srv/metrics.torproject.org/metrics/work/modules/logs && cat metrics.log | grep -v "Could not insert bandwidth "
by root＠meronense.torproject.org 04 Dec '21

04 Dec '21

2021-12-04 08:40:02,189 INFO o.t.m.s.main.Main:32 Starting metrics update run. 2021-12-04 08:40:02,252 INFO o.t.m.s.main.Main:70 Starting org.torproject.metrics.stats.collectdescs.Main module. 2021-12-04 08:40:02,254 INFO o.t.d.i.DescriptorIndexCollector:47 Starting descriptor collection. 2021-12-04 08:40:02,255 INFO o.t.d.i.DescriptorIndexCollector:63 Indexing local directory /srv/metrics.torproject.org/metrics/work/shared/in. 2021-12-04 08:40:03,852 INFO o.t.d.i.DescriptorIndexCollector:75 Fetching remote index file https://collector.torproject.org/index/index.json. 2021-12-04 08:40:05,428 INFO o.t.d.i.DescriptorIndexCollector:84 Fetching remote files from https://collector.torproject.org. 2021-12-04 08:40:53,358 INFO o.t.d.i.DescriptorIndexCollector:90 Deleting extraneous files from local directory /srv/metrics.torproject.org/metrics/work/shared/in. 2021-12-04 08:40:53,864 INFO o.t.d.i.DescriptorIndexCollector:95 Finished descriptor collection. 2021-12-04 08:40:53,864 INFO o.t.m.s.main.Main:73 Completed org.torproject.metrics.stats.collectdescs.Main module. 2021-12-04 08:40:53,864 INFO o.t.m.s.main.Main:70 Starting org.torproject.metrics.stats.connbidirect.Main module. 2021-12-04 08:41:08,669 INFO o.t.m.s.main.Main:73 Completed org.torproject.metrics.stats.connbidirect.Main module. 2021-12-04 08:41:08,669 INFO o.t.m.s.main.Main:70 Starting org.torproject.metrics.stats.onionperf.Main module. 2021-12-04 08:41:08,674 INFO o.t.m.s.o.Main:46 Starting onionperf module. 2021-12-04 08:41:08,674 INFO o.t.m.s.o.Main:67 Connecting to database. 2021-12-04 08:41:09,585 INFO o.t.m.s.o.Main:70 Successfully connected to database. 2021-12-04 08:53:44,068 INFO o.t.m.s.o.Main:261 Querying statistics including partials from database. 2021-12-04 08:54:29,740 INFO o.t.m.s.o.Main:402 Writing 59452 lines to /srv/metrics.torproject.org/metrics/work/modules/onionperf/stats/onionperf-including-partials.csv. 2021-12-04 08:54:29,858 INFO o.t.m.s.o.Main:286 Querying timeout and failure statistics from database. 2021-12-04 08:54:33,062 INFO o.t.m.s.o.Main:402 Writing 19821 lines to /srv/metrics.torproject.org/metrics/work/modules/onionperf/stats/onionperf-failures.csv. 2021-12-04 08:54:33,119 INFO o.t.m.s.o.Main:309 Querying buildtime statistics from database. 2021-12-04 08:55:14,531 INFO o.t.m.s.o.Main:402 Writing 47890 lines to /srv/metrics.torproject.org/metrics/work/modules/onionperf/stats/buildtimes.csv. 2021-12-04 08:55:14,557 INFO o.t.m.s.o.Main:332 Querying latency statistics from database. 2021-12-04 08:55:27,458 INFO o.t.m.s.o.Main:402 Writing 18854 lines to /srv/metrics.torproject.org/metrics/work/modules/onionperf/stats/latencies.csv. 2021-12-04 08:55:27,462 INFO o.t.m.s.o.Main:357 Querying throughput statistics from database. 2021-12-04 08:55:39,006 INFO o.t.m.s.o.Main:402 Writing 19821 lines to /srv/metrics.torproject.org/metrics/work/modules/onionperf/stats/onionperf-throughput.csv. 2021-12-04 08:55:39,009 INFO o.t.m.s.o.Main:409 Disconnecting from database. 2021-12-04 08:55:39,010 INFO o.t.m.s.o.Main:62 Terminated onionperf module. 2021-12-04 08:55:39,010 INFO o.t.m.s.main.Main:73 Completed org.torproject.metrics.stats.onionperf.Main module. 2021-12-04 08:55:39,010 INFO o.t.m.s.main.Main:70 Starting org.torproject.metrics.stats.bwhist.Main module. 2021-12-04 08:55:39,010 INFO o.t.m.s.bwhist.Main:37 Starting bwhist module. 2021-12-04 08:55:39,010 INFO o.t.m.s.bwhist.Main:39 Reading descriptors and inserting relevant parts into the database. 2021-12-04 09:08:27,645 INFO o.t.m.s.b.RelayDescriptorDatabaseImporter:543 Finished importing relay descriptors: 147907 network status entries and 136769 bandwidth history elements 2021-12-04 09:08:30,597 INFO o.t.m.s.bwhist.Main:50 Aggregating database entries. 2021-12-04 09:09:27,128 INFO o.t.m.s.b.RelayDescriptorDatabaseImporter:543 Finished importing relay descriptors: 147907 network status entries and 136769 bandwidth history elements 2021-12-04 09:09:28,096 INFO o.t.m.s.bwhist.Main:53 Querying aggregated statistics from the database. 2021-12-04 09:09:28,618 INFO o.t.m.s.bwhist.Main:57 Closing database connection. 2021-12-04 09:09:28,621 INFO o.t.m.s.bwhist.Main:60 Terminating bwhist module. 2021-12-04 09:09:28,622 INFO o.t.m.s.main.Main:73 Completed org.torproject.metrics.stats.bwhist.Main module. 2021-12-04 09:09:28,622 INFO o.t.m.s.main.Main:70 Starting org.torproject.metrics.stats.advbwdist.Main module. 2021-12-04 09:10:25,116 INFO o.t.m.s.main.Main:73 Completed org.torproject.metrics.stats.advbwdist.Main module. 2021-12-04 09:10:25,265 INFO o.t.m.s.main.Main:70 Starting org.torproject.metrics.stats.hidserv.Main module. 2021-12-04 09:10:25,265 INFO o.t.m.s.hidserv.Main:37 Initializing parser and reading parse history... 2021-12-04 09:10:25,301 INFO o.t.m.s.hidserv.Main:52 Parsing descriptors... 2021-12-04 09:12:06,593 INFO o.t.m.s.hidserv.Main:59 Writing parse history... 2021-12-04 09:12:06,905 INFO o.t.m.s.hidserv.Main:66 Extrapolating statistics... 2021-12-04 09:17:29,298 INFO o.t.m.s.hidserv.Main:82 Aggregating v2 statistics... 2021-12-04 09:19:21,935 INFO o.t.m.s.hidserv.Main:98 Aggregating v3 statistics... 2021-12-04 09:19:22,577 INFO o.t.m.s.hidserv.Main:108 Terminating. 2021-12-04 09:19:22,586 INFO o.t.m.s.main.Main:73 Completed org.torproject.metrics.stats.hidserv.Main module. 2021-12-04 09:19:22,587 INFO o.t.m.s.main.Main:70 Starting org.torproject.metrics.stats.clients.Main module. 2021-12-04 09:19:22,606 INFO o.t.m.s.clients.Main:36 Starting clients module. 2021-12-04 09:19:22,606 INFO o.t.m.s.clients.Main:38 Connecting to database. 2021-12-04 09:19:22,876 INFO o.t.m.s.clients.Main:41 Reading relay descriptors and importing relevant parts into the database. 2021-12-04 09:21:06,721 INFO o.t.m.s.clients.Main:45 Reading bridge descriptors and importing relevant parts into the database. 2021-12-04 09:22:08,459 INFO o.t.m.s.clients.Main:49 Processing newly imported data. 2021-12-04 10:29:41,776 INFO o.t.m.s.clients.Main:53 Querying aggregated statistics from the database. 2021-12-04 10:29:58,697 INFO o.t.m.s.clients.Main:59 Disconnecting from database. 2021-12-04 10:29:58,698 INFO o.t.m.s.clients.Main:62 Running detector. 2021-12-04 10:30:36,839 INFO o.t.m.s.clients.Main:65 Terminating clients module. 2021-12-04 10:30:36,984 INFO o.t.m.s.main.Main:73 Completed org.torproject.metrics.stats.clients.Main module. 2021-12-04 10:30:36,984 INFO o.t.m.s.main.Main:70 Starting org.torproject.metrics.stats.servers.Main module. 2021-12-04 10:30:36,985 INFO o.t.m.s.servers.Main:43 Starting servers module. 2021-12-04 10:30:36,985 INFO o.t.m.s.servers.Main:45 Reading descriptors and inserting relevant parts into the database. 2021-12-04 10:45:04,129 INFO o.t.m.s.servers.Main:77 Aggregating database entries. 2021-12-04 11:23:29,272 INFO o.t.m.s.servers.Main:80 Committing all updated parts in the database. 2021-12-04 11:23:29,466 INFO o.t.m.s.servers.Main:90 Querying aggregated statistics from the database. 2021-12-04 11:40:05,341 INFO o.t.m.s.servers.Main:105 Terminating servers module. 2021-12-04 11:40:05,411 INFO o.t.m.s.main.Main:73 Completed org.torproject.metrics.stats.servers.Main module. 2021-12-04 11:40:05,411 INFO o.t.m.s.main.Main:70 Starting org.torproject.metrics.stats.webstats.Main module. 2021-12-04 11:40:05,497 INFO o.t.m.s.w.Main:75 Starting webstats module. 2021-12-04 11:40:05,497 INFO o.t.m.s.w.Main:92 Connecting to database. 2021-12-04 11:40:05,616 INFO o.t.m.s.w.Main:95 Successfully connected to database. 2021-12-04 11:40:05,694 INFO o.t.m.s.w.Main:101 Querying previously imported log files. 2021-12-04 11:40:08,108 INFO o.t.m.s.w.Main:113 Found 286854 previously imported log files. 2021-12-04 11:48:57,340 INFO o.t.m.s.w.Main:268 Querying statistics from database. 2021-12-04 12:05:17,731 INFO o.t.m.s.w.Main:298 Writing 1427864 lines to /srv/metrics.torproject.org/metrics/work/modules/webstats/stats/webstats.csv. 2021-12-04 12:05:19,524 INFO o.t.m.s.w.Main:305 Disconnecting from database. 2021-12-04 12:05:19,540 INFO o.t.m.s.w.Main:87 Terminated webstats module. 2021-12-04 12:05:19,540 INFO o.t.m.s.main.Main:73 Completed org.torproject.metrics.stats.webstats.Main module. 2021-12-04 12:05:19,540 INFO o.t.m.s.main.Main:70 Starting org.torproject.metrics.stats.totalcw.Main module. 2021-12-04 12:05:19,543 INFO o.t.m.s.totalcw.Main:38 Starting totalcw module. 2021-12-04 12:05:19,543 INFO o.t.m.s.totalcw.Main:40 Reading consensuses and votes and inserting relevant parts into the database. 2021-12-04 12:07:07,827 INFO o.t.m.s.totalcw.Main:64 Committing all updated parts in the database. 2021-12-04 12:07:07,997 INFO o.t.m.s.totalcw.Main:74 Querying aggregated statistics from the database. 2021-12-04 12:07:11,352 INFO o.t.m.s.totalcw.Main:77 Writing aggregated statistics to /srv/metrics.torproject.org/metrics/work/modules/totalcw/stats/totalcw.csv. 2021-12-04 12:07:11,544 INFO o.t.m.s.totalcw.Main:82 Terminating totalcw module. 2021-12-04 12:07:11,544 INFO o.t.m.s.main.Main:73 Completed org.torproject.metrics.stats.totalcw.Main module. 2021-12-04 12:07:11,545 INFO o.t.m.s.main.Main:70 Starting org.torproject.metrics.stats.bridgedb.Main module. 2021-12-04 12:07:11,963 INFO o.t.m.s.main.Main:73 Completed org.torproject.metrics.stats.bridgedb.Main module. 2021-12-04 12:07:11,963 INFO o.t.m.s.main.Main:82 Making module data available. 2021-12-04 12:07:15,299 INFO o.t.m.s.main.Main:126 Successfully copied 22 files to stats output directory: [connbidirect2.csv, onionperf-failures.csv, buildtimes.csv, latencies.csv, torperf-1.1.csv, onionperf-throughput.csv, onionperf-including-partials.csv, bandwidth.csv, advbwdist.csv, v3hidserv.csv, hidserv.csv, clients.csv, userstats-combined.csv, platforms.csv, relayflags.csv, versions.csv, advbw.csv, networksize.csv, ipv6servers.csv, webstats.csv, totalcw.csv, bridgedb-stats.csv] 2021-12-04 12:07:15,299 INFO o.t.m.s.main.Main:130 Completed metrics update run.

1 0

** RECOVERY Service Alert: onionoo-frontend-01/network service - onionoo full service check is OK **
by nagios＠hetzner-hel1-01.torproject.org 04 Dec '21

04 Dec '21

***** Icinga ***** Notification Type: RECOVERY Service: network service - onionoo full service check Host: onionoo-frontend-01 Address: 116.202.120.171 State: OK Date/Time: Sat Dec 4 17:39:01 UTC 2021 Additional Info: OK: Valid response with recent timestamps (3121 sec): relays_published=2021-12-04 17:00:00, bridges_published=2021-12-04 16:46:52

1 0

** RECOVERY Service Alert: onionoo-backend-02/network service - onionoo backend is OK **
by nagios＠hetzner-hel1-01.torproject.org 04 Dec '21

04 Dec '21

***** Icinga ***** Notification Type: RECOVERY Service: network service - onionoo backend Host: onionoo-backend-02 Address: 116.202.120.178 State: OK Date/Time: Sat Dec 4 17:38:31 UTC 2021 Additional Info: OK: Valid response with recent timestamps (3096 sec): relays_published=2021-12-04 17:00:00, bridges_published=2021-12-04 16:46:52

1 0

** RECOVERY Service Alert: onionoo-backend-02/network service - onionoo varnish is OK **
by nagios＠hetzner-hel1-01.torproject.org 04 Dec '21

04 Dec '21

***** Icinga ***** Notification Type: RECOVERY Service: network service - onionoo varnish Host: onionoo-backend-02 Address: 116.202.120.178 State: OK Date/Time: Sat Dec 4 17:35:41 UTC 2021 Additional Info: OK: Valid response with recent timestamps (2927 sec): relays_published=2021-12-04 17:00:00, bridges_published=2021-12-04 16:46:52

1 0

** RECOVERY Service Alert: onionoo-frontend-02/network service - onionoo varnish is OK **
by nagios＠hetzner-hel1-01.torproject.org 04 Dec '21

04 Dec '21

***** Icinga ***** Notification Type: RECOVERY Service: network service - onionoo varnish Host: onionoo-frontend-02 Address: 49.12.57.137 State: OK Date/Time: Sat Dec 4 17:31:01 UTC 2021 Additional Info: OK: Valid response with recent timestamps (2644 sec): relays_published=2021-12-04 17:00:00, bridges_published=2021-12-04 16:46:52

1 0

** RECOVERY Service Alert: onionoo-frontend-02/network service - onionoo full service check is OK **
by nagios＠hetzner-hel1-01.torproject.org 04 Dec '21

04 Dec '21

***** Icinga ***** Notification Type: RECOVERY Service: network service - onionoo full service check Host: onionoo-frontend-02 Address: 49.12.57.137 State: OK Date/Time: Sat Dec 4 17:30:31 UTC 2021 Additional Info: OK: Valid response with recent timestamps (2611 sec): relays_published=2021-12-04 17:00:00, bridges_published=2021-12-04 16:46:52

1 0

2024

2023

2022

2021

2020

2019

2018

metrics-alerts