- metrics-alerts - lists.torproject.org

** PROBLEM Service Alert: global/collector is WARNING **
by nagios＠hetzner-hel1-01.torproject.org 08 Dec '21

08 Dec '21

***** Icinga ***** Notification Type: PROBLEM Service: collector Host: global Address: global State: WARNING Date/Time: Wed Dec 8 10:38:42 UTC 2021 Additional Info: WARNING: Got a valid response from https://collector.torproject.org/index/index.json with at least one timestamp being outdated: /archive/relay-descriptors/bandwidths/ = 2021-12-05 10:24 (72 = 72 hours ago)

1 0

Cron <metrics@meronense> cd /srv/metrics.torproject.org/metrics/work/modules/logs && cat metrics.log | grep -v "Could not insert bandwidth "
by root＠meronense.torproject.org 08 Dec '21

08 Dec '21

2021-12-08 02:26:42,643 INFO o.t.m.s.w.Main:298 Writing 1430445 lines to /srv/metrics.torproject.org/metrics/work/modules/webstats/stats/webstats.csv. 2021-12-08 02:26:47,914 INFO o.t.m.s.w.Main:305 Disconnecting from database. 2021-12-08 02:26:47,919 INFO o.t.m.s.w.Main:87 Terminated webstats module. 2021-12-08 02:26:47,919 INFO o.t.m.s.main.Main:73 Completed org.torproject.metrics.stats.webstats.Main module. 2021-12-08 02:26:47,919 INFO o.t.m.s.main.Main:70 Starting org.torproject.metrics.stats.totalcw.Main module. 2021-12-08 02:26:47,921 INFO o.t.m.s.totalcw.Main:38 Starting totalcw module. 2021-12-08 02:26:47,921 INFO o.t.m.s.totalcw.Main:40 Reading consensuses and votes and inserting relevant parts into the database. 2021-12-08 02:31:05,608 INFO o.t.m.s.totalcw.Main:64 Committing all updated parts in the database. 2021-12-08 02:31:05,813 INFO o.t.m.s.totalcw.Main:74 Querying aggregated statistics from the database. 2021-12-08 02:32:31,927 INFO o.t.m.s.totalcw.Main:77 Writing aggregated statistics to /srv/metrics.torproject.org/metrics/work/modules/totalcw/stats/totalcw.csv. 2021-12-08 02:32:32,110 INFO o.t.m.s.totalcw.Main:82 Terminating totalcw module. 2021-12-08 02:32:32,111 INFO o.t.m.s.main.Main:73 Completed org.torproject.metrics.stats.totalcw.Main module. 2021-12-08 02:32:32,111 INFO o.t.m.s.main.Main:70 Starting org.torproject.metrics.stats.bridgedb.Main module. 2021-12-08 02:32:34,752 INFO o.t.m.s.main.Main:73 Completed org.torproject.metrics.stats.bridgedb.Main module. 2021-12-08 02:32:34,752 INFO o.t.m.s.main.Main:82 Making module data available. 2021-12-08 02:33:08,235 INFO o.t.m.s.main.Main:126 Successfully copied 22 files to stats output directory: [connbidirect2.csv, onionperf-failures.csv, buildtimes.csv, latencies.csv, torperf-1.1.csv, onionperf-throughput.csv, onionperf-including-partials.csv, bandwidth.csv, advbwdist.csv, v3hidserv.csv, hidserv.csv, clients.csv, userstats-combined.csv, platforms.csv, relayflags.csv, versions.csv, advbw.csv, networksize.csv, ipv6servers.csv, webstats.csv, totalcw.csv, bridgedb-stats.csv] 2021-12-08 02:33:09,279 INFO o.t.m.s.main.Main:130 Completed metrics update run.

1 0

Cron <collector@colchicifolium> cd /srv/collector.torproject.org/collector/logs/ && cat metrics.log | grep -E "WARN|ERROR|FATAL" | grep -v "Overwriting existing descriptor file"
by root＠colchicifolium.torproject.org 08 Dec '21

08 Dec '21

2021-12-08 00:19:45,974 WARN o.t.m.c.r.ReferenceChecker:314 Missing too many referenced descriptors (40.34296545640288). 2021-12-08 00:40:06,322 WARN o.t.m.c.r.ReferenceChecker:314 Missing too many referenced descriptors (43.61316247703102). 2021-12-08 00:51:48,967 WARN o.t.m.c.r.ReferenceChecker:314 Missing too many referenced descriptors (38.06522641812296). 2021-12-08 01:10:39,284 WARN o.t.m.c.r.ReferenceChecker:314 Missing too many referenced descriptors (35.79460677669415). 2021-12-08 02:10:58,831 WARN o.t.m.c.r.ReferenceChecker:314 Missing too many referenced descriptors (32.01640473817632). 2021-12-08 02:50:33,185 WARN o.t.m.c.r.ReferenceChecker:314 Missing too many referenced descriptors (37.95780153742516). 2021-12-08 02:56:55,021 WARN o.t.m.c.r.ReferenceChecker:314 Missing too many referenced descriptors (33.93426964278531). 2021-12-08 03:22:17,973 WARN o.t.m.c.r.ReferenceChecker:314 Missing too many referenced descriptors (40.65785080234353). 2021-12-08 03:44:40,466 WARN o.t.m.c.r.ReferenceChecker:314 Missing too many referenced descriptors (29.597393255535156). 2021-12-08 04:16:19,614 WARN o.t.m.c.r.ReferenceChecker:314 Missing too many referenced descriptors (33.18006154834784). 2021-12-08 04:49:08,655 WARN o.t.m.c.r.ReferenceChecker:314 Missing too many referenced descriptors (18.539068939581195). 2021-12-08 05:18:14,794 WARN o.t.m.c.r.ReferenceChecker:314 Missing too many referenced descriptors (26.678840673878643). 2021-12-08 05:44:46,967 WARN o.t.m.c.r.ReferenceChecker:314 Missing too many referenced descriptors (19.51282419112462). 2021-12-08 06:11:34,932 WARN o.t.m.c.r.ReferenceChecker:314 Missing too many referenced descriptors (17.782854845092103).

1 0

Cron <exonerator@materculae> cd /srv/exonerator.torproject.org/exonerator/ && ~exonerator/bin/run-exonerator
by root＠materculae.torproject.org 08 Dec '21

08 Dec '21

2021-12-08 05:22:01,664 ERROR o.t.m.e.ExoneraTorDatabaseImporter:167 Lock file /srv/exonerator.torproject.org/exonerator/exonerator-lock is less than 6 hours old. Either make sure that there are no other ExoneraTor database importers running and manually delete that file, or wait until the file is 6 hours old when it will be overwritten automatically. Exiting.

1 0

Cron <exonerator@materculae> cd /srv/exonerator.torproject.org/exonerator/ && ~exonerator/bin/run-exonerator
by root＠materculae.torproject.org 08 Dec '21

08 Dec '21

2021-12-08 03:22:02,052 ERROR o.t.m.e.ExoneraTorDatabaseImporter:167 Lock file /srv/exonerator.torproject.org/exonerator/exonerator-lock is less than 6 hours old. Either make sure that there are no other ExoneraTor database importers running and manually delete that file, or wait until the file is 6 hours old when it will be overwritten automatically. Exiting.

1 0

Cron <exonerator@materculae> cd /srv/exonerator.torproject.org/exonerator/ && ~exonerator/bin/run-exonerator
by root＠materculae.torproject.org 08 Dec '21

08 Dec '21

2021-12-08 02:22:01,971 ERROR o.t.m.e.ExoneraTorDatabaseImporter:167 Lock file /srv/exonerator.torproject.org/exonerator/exonerator-lock is less than 6 hours old. Either make sure that there are no other ExoneraTor database importers running and manually delete that file, or wait until the file is 6 hours old when it will be overwritten automatically. Exiting.

1 0

Cron <onionoo@onionoo-backend-02> /srv/onionoo.torproject.org/onionoo/bin/service/onionoo
by root＠onionoo-backend-02.torproject.org 08 Dec '21

08 Dec '21

Traceback (most recent call last): File "/srv/onionoo.torproject.org/onionoo/bin/service/onionoo", line 23, in <module> list_of_errors = Counter([(x.split(' ')[2], x.split(' ')[3].split(':')[0]) for x in str(out).split('\\n') if x != '"']) File "/srv/onionoo.torproject.org/onionoo/bin/service/onionoo", line 23, in <listcomp> list_of_errors = Counter([(x.split(' ')[2], x.split(' ')[3].split(':')[0]) for x in str(out).split('\\n') if x != '"']) IndexError: list index out of range During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/srv/onionoo.torproject.org/onionoo/bin/service/onionoo", line 29, in <module> g_warnings.labels(host, "Problem running exporter - {}".format(e)).set(1) File "/usr/lib/python3/dist-packages/prometheus_client/metrics.py", line 146, in labels raise ValueError('Incorrect label count') ValueError: Incorrect label count

1 0

Cron <onionoo@onionoo-backend-02> /srv/onionoo.torproject.org/onionoo/bin/service/onionoo
by root＠onionoo-backend-02.torproject.org 08 Dec '21

08 Dec '21

Traceback (most recent call last): File "/srv/onionoo.torproject.org/onionoo/bin/service/onionoo", line 23, in <module> list_of_errors = Counter([(x.split(' ')[2], x.split(' ')[3].split(':')[0]) for x in str(out).split('\\n') if x != '"']) File "/srv/onionoo.torproject.org/onionoo/bin/service/onionoo", line 23, in <listcomp> list_of_errors = Counter([(x.split(' ')[2], x.split(' ')[3].split(':')[0]) for x in str(out).split('\\n') if x != '"']) IndexError: list index out of range During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/srv/onionoo.torproject.org/onionoo/bin/service/onionoo", line 29, in <module> g_warnings.labels(host, "Problem running exporter - {}".format(e)).set(1) File "/usr/lib/python3/dist-packages/prometheus_client/metrics.py", line 146, in labels raise ValueError('Incorrect label count') ValueError: Incorrect label count

1 0

Cron <onionoo@onionoo-backend-02> /srv/onionoo.torproject.org/onionoo/bin/service/onionoo
by root＠onionoo-backend-02.torproject.org 08 Dec '21

08 Dec '21

Traceback (most recent call last): File "/srv/onionoo.torproject.org/onionoo/bin/service/onionoo", line 23, in <module> list_of_errors = Counter([(x.split(' ')[2], x.split(' ')[3].split(':')[0]) for x in str(out).split('\\n') if x != '"']) File "/srv/onionoo.torproject.org/onionoo/bin/service/onionoo", line 23, in <listcomp> list_of_errors = Counter([(x.split(' ')[2], x.split(' ')[3].split(':')[0]) for x in str(out).split('\\n') if x != '"']) IndexError: list index out of range During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/srv/onionoo.torproject.org/onionoo/bin/service/onionoo", line 29, in <module> g_warnings.labels(host, "Problem running exporter - {}".format(e)).set(1) File "/usr/lib/python3/dist-packages/prometheus_client/metrics.py", line 146, in labels raise ValueError('Incorrect label count') ValueError: Incorrect label count

1 0

Cron <onionoo@onionoo-backend-02> /srv/onionoo.torproject.org/onionoo/bin/service/onionoo
by root＠onionoo-backend-02.torproject.org 08 Dec '21

08 Dec '21

Traceback (most recent call last): File "/srv/onionoo.torproject.org/onionoo/bin/service/onionoo", line 23, in <module> list_of_errors = Counter([(x.split(' ')[2], x.split(' ')[3].split(':')[0]) for x in str(out).split('\\n') if x != '"']) File "/srv/onionoo.torproject.org/onionoo/bin/service/onionoo", line 23, in <listcomp> list_of_errors = Counter([(x.split(' ')[2], x.split(' ')[3].split(':')[0]) for x in str(out).split('\\n') if x != '"']) IndexError: list index out of range During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/srv/onionoo.torproject.org/onionoo/bin/service/onionoo", line 29, in <module> g_warnings.labels(host, "Problem running exporter - {}".format(e)).set(1) File "/usr/lib/python3/dist-packages/prometheus_client/metrics.py", line 146, in labels raise ValueError('Incorrect label count') ValueError: Incorrect label count

1 0

2025

2024

2023

2022

2021

2020

2019

2018

metrics-alerts