December 2021 - metrics-alerts - lists.torproject.org

Cron <onionoo@onionoo-backend-02> /srv/onionoo.torproject.org/onionoo/bin/service/onionoo
by root＠onionoo-backend-02.torproject.org 22 Dec '21

22 Dec '21

Traceback (most recent call last): File "/srv/onionoo.torproject.org/onionoo/bin/service/onionoo", line 23, in <module> list_of_errors = Counter([(x.split(' ')[2], x.split(' ')[3].split(':')[0]) for x in str(out).split('\\n') if x != '"']) File "/srv/onionoo.torproject.org/onionoo/bin/service/onionoo", line 23, in <listcomp> list_of_errors = Counter([(x.split(' ')[2], x.split(' ')[3].split(':')[0]) for x in str(out).split('\\n') if x != '"']) IndexError: list index out of range During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/srv/onionoo.torproject.org/onionoo/bin/service/onionoo", line 29, in <module> g_warnings.labels(host, "Problem running exporter - {}".format(e)).set(1) File "/usr/lib/python3/dist-packages/prometheus_client/metrics.py", line 146, in labels raise ValueError('Incorrect label count') ValueError: Incorrect label count

1 0

Cron <onionoo@onionoo-backend-02> /srv/onionoo.torproject.org/onionoo/bin/service/onionoo
by root＠onionoo-backend-02.torproject.org 22 Dec '21

22 Dec '21

Traceback (most recent call last): File "/srv/onionoo.torproject.org/onionoo/bin/service/onionoo", line 23, in <module> list_of_errors = Counter([(x.split(' ')[2], x.split(' ')[3].split(':')[0]) for x in str(out).split('\\n') if x != '"']) File "/srv/onionoo.torproject.org/onionoo/bin/service/onionoo", line 23, in <listcomp> list_of_errors = Counter([(x.split(' ')[2], x.split(' ')[3].split(':')[0]) for x in str(out).split('\\n') if x != '"']) IndexError: list index out of range During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/srv/onionoo.torproject.org/onionoo/bin/service/onionoo", line 29, in <module> g_warnings.labels(host, "Problem running exporter - {}".format(e)).set(1) File "/usr/lib/python3/dist-packages/prometheus_client/metrics.py", line 146, in labels raise ValueError('Incorrect label count') ValueError: Incorrect label count

1 0

Cron <metrics@meronense> cd /srv/metrics.torproject.org/metrics/work/modules/logs && cat metrics.log | grep -v "Could not insert bandwidth "
by root＠meronense.torproject.org 22 Dec '21

22 Dec '21

2021-12-21 08:40:02,284 INFO o.t.m.s.main.Main:32 Starting metrics update run. 2021-12-21 08:40:02,324 INFO o.t.m.s.main.Main:70 Starting org.torproject.metrics.stats.collectdescs.Main module. 2021-12-21 08:40:02,327 INFO o.t.d.i.DescriptorIndexCollector:47 Starting descriptor collection. 2021-12-21 08:40:02,327 INFO o.t.d.i.DescriptorIndexCollector:63 Indexing local directory /srv/metrics.torproject.org/metrics/work/shared/in. 2021-12-21 08:40:02,366 INFO o.t.d.i.DescriptorIndexCollector:75 Fetching remote index file https://collector.torproject.org/index/index.json. 2021-12-21 08:40:03,118 INFO o.t.d.i.DescriptorIndexCollector:84 Fetching remote files from https://collector.torproject.org. 2021-12-21 08:40:26,141 INFO o.t.d.i.DescriptorIndexCollector:90 Deleting extraneous files from local directory /srv/metrics.torproject.org/metrics/work/shared/in. 2021-12-21 08:40:26,371 INFO o.t.d.i.DescriptorIndexCollector:95 Finished descriptor collection. 2021-12-21 08:40:26,371 INFO o.t.m.s.main.Main:73 Completed org.torproject.metrics.stats.collectdescs.Main module. 2021-12-21 08:40:26,371 INFO o.t.m.s.main.Main:70 Starting org.torproject.metrics.stats.connbidirect.Main module. 2021-12-21 08:40:33,429 INFO o.t.m.s.main.Main:73 Completed org.torproject.metrics.stats.connbidirect.Main module. 2021-12-21 08:40:33,429 INFO o.t.m.s.main.Main:70 Starting org.torproject.metrics.stats.onionperf.Main module. 2021-12-21 08:40:33,430 INFO o.t.m.s.o.Main:46 Starting onionperf module. 2021-12-21 08:40:33,431 INFO o.t.m.s.o.Main:67 Connecting to database. 2021-12-21 08:40:33,745 INFO o.t.m.s.o.Main:70 Successfully connected to database. 2021-12-21 08:47:44,857 INFO o.t.m.s.o.Main:261 Querying statistics including partials from database. 2021-12-21 08:48:13,599 INFO o.t.m.s.o.Main:402 Writing 59959 lines to /srv/metrics.torproject.org/metrics/work/modules/onionperf/stats/onionperf-including-partials.csv. 2021-12-21 08:48:14,493 INFO o.t.m.s.o.Main:286 Querying timeout and failure statistics from database. 2021-12-21 08:48:18,073 INFO o.t.m.s.o.Main:402 Writing 19991 lines to /srv/metrics.torproject.org/metrics/work/modules/onionperf/stats/onionperf-failures.csv. 2021-12-21 08:48:18,076 INFO o.t.m.s.o.Main:309 Querying buildtime statistics from database. 2021-12-21 08:49:05,533 INFO o.t.m.s.o.Main:402 Writing 48196 lines to /srv/metrics.torproject.org/metrics/work/modules/onionperf/stats/buildtimes.csv. 2021-12-21 08:49:05,540 INFO o.t.m.s.o.Main:332 Querying latency statistics from database. 2021-12-21 08:49:17,125 INFO o.t.m.s.o.Main:402 Writing 19024 lines to /srv/metrics.torproject.org/metrics/work/modules/onionperf/stats/latencies.csv. 2021-12-21 08:49:17,129 INFO o.t.m.s.o.Main:357 Querying throughput statistics from database. 2021-12-21 08:49:28,487 INFO o.t.m.s.o.Main:402 Writing 19991 lines to /srv/metrics.torproject.org/metrics/work/modules/onionperf/stats/onionperf-throughput.csv. 2021-12-21 08:49:28,490 INFO o.t.m.s.o.Main:409 Disconnecting from database. 2021-12-21 08:49:28,491 INFO o.t.m.s.o.Main:62 Terminated onionperf module. 2021-12-21 08:49:28,491 INFO o.t.m.s.main.Main:73 Completed org.torproject.metrics.stats.onionperf.Main module. 2021-12-21 08:49:28,491 INFO o.t.m.s.main.Main:70 Starting org.torproject.metrics.stats.bwhist.Main module. 2021-12-21 08:49:28,491 INFO o.t.m.s.bwhist.Main:37 Starting bwhist module. 2021-12-21 08:49:28,491 INFO o.t.m.s.bwhist.Main:39 Reading descriptors and inserting relevant parts into the database. 2021-12-21 09:01:52,339 INFO o.t.m.s.b.RelayDescriptorDatabaseImporter:543 Finished importing relay descriptors: 158227 network status entries and 140887 bandwidth history elements 2021-12-21 09:01:55,350 INFO o.t.m.s.bwhist.Main:50 Aggregating database entries. 2021-12-21 09:03:28,978 INFO o.t.m.s.b.RelayDescriptorDatabaseImporter:543 Finished importing relay descriptors: 158227 network status entries and 140887 bandwidth history elements 2021-12-21 09:03:31,114 INFO o.t.m.s.bwhist.Main:53 Querying aggregated statistics from the database. 2021-12-21 09:03:31,518 INFO o.t.m.s.bwhist.Main:57 Closing database connection. 2021-12-21 09:03:31,520 INFO o.t.m.s.bwhist.Main:60 Terminating bwhist module. 2021-12-21 09:03:31,520 INFO o.t.m.s.main.Main:73 Completed org.torproject.metrics.stats.bwhist.Main module. 2021-12-21 09:03:31,520 INFO o.t.m.s.main.Main:70 Starting org.torproject.metrics.stats.advbwdist.Main module. 2021-12-21 09:04:31,437 INFO o.t.m.s.main.Main:73 Completed org.torproject.metrics.stats.advbwdist.Main module. 2021-12-21 09:04:31,438 INFO o.t.m.s.main.Main:70 Starting org.torproject.metrics.stats.hidserv.Main module. 2021-12-21 09:04:31,438 INFO o.t.m.s.hidserv.Main:37 Initializing parser and reading parse history... 2021-12-21 09:04:31,521 INFO o.t.m.s.hidserv.Main:52 Parsing descriptors... 2021-12-21 09:06:08,290 INFO o.t.m.s.hidserv.Main:59 Writing parse history... 2021-12-21 09:06:08,295 INFO o.t.m.s.hidserv.Main:66 Extrapolating statistics... 2021-12-21 09:11:14,809 INFO o.t.m.s.hidserv.Main:82 Aggregating v2 statistics... 2021-12-21 09:12:56,336 INFO o.t.m.s.hidserv.Main:98 Aggregating v3 statistics... 2021-12-21 09:12:56,844 INFO o.t.m.s.hidserv.Main:108 Terminating. 2021-12-21 09:12:56,845 INFO o.t.m.s.main.Main:73 Completed org.torproject.metrics.stats.hidserv.Main module. 2021-12-21 09:12:56,845 INFO o.t.m.s.main.Main:70 Starting org.torproject.metrics.stats.clients.Main module. 2021-12-21 09:12:56,865 INFO o.t.m.s.clients.Main:36 Starting clients module. 2021-12-21 09:12:56,865 INFO o.t.m.s.clients.Main:38 Connecting to database. 2021-12-21 09:12:56,960 INFO o.t.m.s.clients.Main:41 Reading relay descriptors and importing relevant parts into the database. 2021-12-21 09:14:36,026 INFO o.t.m.s.clients.Main:45 Reading bridge descriptors and importing relevant parts into the database. 2021-12-21 09:15:21,811 INFO o.t.m.s.clients.Main:49 Processing newly imported data. 2021-12-21 10:25:09,189 INFO o.t.m.s.clients.Main:53 Querying aggregated statistics from the database. 2021-12-21 10:25:27,344 INFO o.t.m.s.clients.Main:59 Disconnecting from database. 2021-12-21 10:25:27,345 INFO o.t.m.s.clients.Main:62 Running detector. 2021-12-21 10:26:01,318 INFO o.t.m.s.clients.Main:65 Terminating clients module. 2021-12-21 10:26:01,381 INFO o.t.m.s.main.Main:73 Completed org.torproject.metrics.stats.clients.Main module. 2021-12-21 10:26:01,381 INFO o.t.m.s.main.Main:70 Starting org.torproject.metrics.stats.servers.Main module. 2021-12-21 10:26:01,381 INFO o.t.m.s.servers.Main:43 Starting servers module. 2021-12-21 10:26:01,381 INFO o.t.m.s.servers.Main:45 Reading descriptors and inserting relevant parts into the database. 2021-12-21 10:40:55,132 INFO o.t.m.s.servers.Main:77 Aggregating database entries. 2021-12-21 10:45:04,522 INFO o.t.m.s.servers.Main:80 Committing all updated parts in the database. 2021-12-21 10:45:04,713 INFO o.t.m.s.servers.Main:90 Querying aggregated statistics from the database. 2021-12-21 10:59:08,440 INFO o.t.m.s.servers.Main:105 Terminating servers module. 2021-12-21 10:59:08,490 INFO o.t.m.s.main.Main:73 Completed org.torproject.metrics.stats.servers.Main module. 2021-12-21 10:59:08,491 INFO o.t.m.s.main.Main:70 Starting org.torproject.metrics.stats.webstats.Main module. 2021-12-21 10:59:08,594 INFO o.t.m.s.w.Main:75 Starting webstats module. 2021-12-21 10:59:08,594 INFO o.t.m.s.w.Main:92 Connecting to database. 2021-12-21 10:59:09,047 INFO o.t.m.s.w.Main:95 Successfully connected to database. 2021-12-21 10:59:09,048 INFO o.t.m.s.w.Main:101 Querying previously imported log files. 2021-12-21 10:59:11,096 INFO o.t.m.s.w.Main:113 Found 289742 previously imported log files. 2021-12-21 11:07:34,551 INFO o.t.m.s.w.Main:268 Querying statistics from database. 2021-12-21 11:25:06,884 INFO o.t.m.s.w.Main:298 Writing 1442773 lines to /srv/metrics.torproject.org/metrics/work/modules/webstats/stats/webstats.csv. 2021-12-21 11:25:08,658 INFO o.t.m.s.w.Main:305 Disconnecting from database. 2021-12-21 11:25:08,659 INFO o.t.m.s.w.Main:87 Terminated webstats module. 2021-12-21 11:25:08,659 INFO o.t.m.s.main.Main:73 Completed org.torproject.metrics.stats.webstats.Main module. 2021-12-21 11:25:08,659 INFO o.t.m.s.main.Main:70 Starting org.torproject.metrics.stats.totalcw.Main module. 2021-12-21 11:25:08,661 INFO o.t.m.s.totalcw.Main:38 Starting totalcw module. 2021-12-21 11:25:08,661 INFO o.t.m.s.totalcw.Main:40 Reading consensuses and votes and inserting relevant parts into the database. 2021-12-21 11:27:43,227 INFO o.t.m.s.totalcw.Main:64 Committing all updated parts in the database. 2021-12-21 11:27:43,343 INFO o.t.m.s.totalcw.Main:74 Querying aggregated statistics from the database. 2021-12-21 11:27:48,953 INFO o.t.m.s.totalcw.Main:77 Writing aggregated statistics to /srv/metrics.torproject.org/metrics/work/modules/totalcw/stats/totalcw.csv. 2021-12-21 11:27:49,225 INFO o.t.m.s.totalcw.Main:82 Terminating totalcw module. 2021-12-21 11:27:49,226 INFO o.t.m.s.main.Main:73 Completed org.torproject.metrics.stats.totalcw.Main module. 2021-12-21 11:27:49,226 INFO o.t.m.s.main.Main:70 Starting org.torproject.metrics.stats.bridgedb.Main module. 2021-12-21 11:27:49,626 INFO o.t.m.s.main.Main:73 Completed org.torproject.metrics.stats.bridgedb.Main module. 2021-12-21 11:27:49,626 INFO o.t.m.s.main.Main:82 Making module data available. 2021-12-21 11:28:01,072 INFO o.t.m.s.main.Main:126 Successfully copied 22 files to stats output directory: [connbidirect2.csv, onionperf-failures.csv, buildtimes.csv, latencies.csv, torperf-1.1.csv, onionperf-throughput.csv, onionperf-including-partials.csv, bandwidth.csv, advbwdist.csv, v3hidserv.csv, hidserv.csv, clients.csv, userstats-combined.csv, platforms.csv, relayflags.csv, versions.csv, advbw.csv, networksize.csv, ipv6servers.csv, webstats.csv, totalcw.csv, bridgedb-stats.csv] 2021-12-21 11:28:01,072 INFO o.t.m.s.main.Main:130 Completed metrics update run. 2021-12-21 21:33:59,046 INFO o.t.m.s.main.Main:32 Starting metrics update run. 2021-12-21 21:33:59,111 INFO o.t.m.s.main.Main:70 Starting org.torproject.metrics.stats.collectdescs.Main module. 2021-12-21 21:33:59,136 INFO o.t.d.i.DescriptorIndexCollector:47 Starting descriptor collection. 2021-12-21 21:33:59,137 INFO o.t.d.i.DescriptorIndexCollector:63 Indexing local directory /srv/metrics.torproject.org/metrics/work/shared/in. 2021-12-21 21:33:59,177 INFO o.t.d.i.DescriptorIndexCollector:75 Fetching remote index file https://collector.torproject.org/index/index.json. 2021-12-21 21:33:59,998 INFO o.t.d.i.DescriptorIndexCollector:84 Fetching remote files from https://collector.torproject.org. 2021-12-21 21:34:10,118 INFO o.t.d.i.DescriptorIndexCollector:90 Deleting extraneous files from local directory /srv/metrics.torproject.org/metrics/work/shared/in. 2021-12-21 21:34:10,364 INFO o.t.d.i.DescriptorIndexCollector:95 Finished descriptor collection. 2021-12-21 21:34:10,364 INFO o.t.m.s.main.Main:73 Completed org.torproject.metrics.stats.collectdescs.Main module. 2021-12-21 21:34:10,364 INFO o.t.m.s.main.Main:70 Starting org.torproject.metrics.stats.connbidirect.Main module. 2021-12-21 21:34:14,607 INFO o.t.m.s.main.Main:73 Completed org.torproject.metrics.stats.connbidirect.Main module. 2021-12-21 21:34:14,607 INFO o.t.m.s.main.Main:70 Starting org.torproject.metrics.stats.onionperf.Main module. 2021-12-21 21:34:14,609 INFO o.t.m.s.o.Main:46 Starting onionperf module. 2021-12-21 21:34:14,609 INFO o.t.m.s.o.Main:67 Connecting to database. 2021-12-21 21:34:14,932 INFO o.t.m.s.o.Main:70 Successfully connected to database. 2021-12-21 21:34:25,918 INFO o.t.m.s.o.Main:261 Querying statistics including partials from database. 2021-12-21 21:35:02,186 INFO o.t.m.s.o.Main:402 Writing 59959 lines to /srv/metrics.torproject.org/metrics/work/modules/onionperf/stats/onionperf-including-partials.csv. 2021-12-21 21:35:02,663 INFO o.t.m.s.o.Main:286 Querying timeout and failure statistics from database. 2021-12-21 21:35:06,271 INFO o.t.m.s.o.Main:402 Writing 19991 lines to /srv/metrics.torproject.org/metrics/work/modules/onionperf/stats/onionperf-failures.csv. 2021-12-21 21:35:06,282 INFO o.t.m.s.o.Main:309 Querying buildtime statistics from database. 2021-12-21 21:35:56,804 INFO o.t.m.s.o.Main:402 Writing 48196 lines to /srv/metrics.torproject.org/metrics/work/modules/onionperf/stats/buildtimes.csv. 2021-12-21 21:35:57,116 INFO o.t.m.s.o.Main:332 Querying latency statistics from database. 2021-12-21 21:36:10,900 INFO o.t.m.s.o.Main:402 Writing 19024 lines to /srv/metrics.torproject.org/metrics/work/modules/onionperf/stats/latencies.csv. 2021-12-21 21:36:10,904 INFO o.t.m.s.o.Main:357 Querying throughput statistics from database. 2021-12-21 21:36:23,735 INFO o.t.m.s.o.Main:402 Writing 19991 lines to /srv/metrics.torproject.org/metrics/work/modules/onionperf/stats/onionperf-throughput.csv. 2021-12-21 21:36:23,738 INFO o.t.m.s.o.Main:409 Disconnecting from database. 2021-12-21 21:36:23,738 INFO o.t.m.s.o.Main:62 Terminated onionperf module. 2021-12-21 21:36:23,739 INFO o.t.m.s.main.Main:73 Completed org.torproject.metrics.stats.onionperf.Main module. 2021-12-21 21:36:23,739 INFO o.t.m.s.main.Main:70 Starting org.torproject.metrics.stats.bwhist.Main module. 2021-12-21 21:36:23,739 INFO o.t.m.s.bwhist.Main:37 Starting bwhist module. 2021-12-21 21:36:23,739 INFO o.t.m.s.bwhist.Main:39 Reading descriptors and inserting relevant parts into the database. 2021-12-21 21:47:33,929 INFO o.t.m.s.b.RelayDescriptorDatabaseImporter:543 Finished importing relay descriptors: 85734 network status entries and 59705 bandwidth history elements 2021-12-21 21:47:36,756 INFO o.t.m.s.bwhist.Main:50 Aggregating database entries. 2021-12-21 21:48:13,962 INFO o.t.m.s.b.RelayDescriptorDatabaseImporter:543 Finished importing relay descriptors: 85734 network status entries and 59705 bandwidth history elements 2021-12-21 21:48:15,123 INFO o.t.m.s.bwhist.Main:53 Querying aggregated statistics from the database. 2021-12-21 21:48:15,708 INFO o.t.m.s.bwhist.Main:57 Closing database connection. 2021-12-21 21:48:15,708 INFO o.t.m.s.bwhist.Main:60 Terminating bwhist module. 2021-12-21 21:48:15,708 INFO o.t.m.s.main.Main:73 Completed org.torproject.metrics.stats.bwhist.Main module. 2021-12-21 21:48:15,709 INFO o.t.m.s.main.Main:70 Starting org.torproject.metrics.stats.advbwdist.Main module. 2021-12-21 21:49:10,026 INFO o.t.m.s.main.Main:73 Completed org.torproject.metrics.stats.advbwdist.Main module. 2021-12-21 21:49:10,026 INFO o.t.m.s.main.Main:70 Starting org.torproject.metrics.stats.hidserv.Main module. 2021-12-21 21:49:10,027 INFO o.t.m.s.hidserv.Main:37 Initializing parser and reading parse history... 2021-12-21 21:49:10,047 INFO o.t.m.s.hidserv.Main:52 Parsing descriptors... 2021-12-21 21:50:44,864 INFO o.t.m.s.hidserv.Main:59 Writing parse history... 2021-12-21 21:50:44,865 INFO o.t.m.s.hidserv.Main:66 Extrapolating statistics... 2021-12-21 21:55:15,657 INFO o.t.m.s.hidserv.Main:82 Aggregating v2 statistics... 2021-12-21 21:56:25,250 INFO o.t.m.s.hidserv.Main:98 Aggregating v3 statistics... 2021-12-21 21:56:25,674 INFO o.t.m.s.hidserv.Main:108 Terminating. 2021-12-21 21:56:25,674 INFO o.t.m.s.main.Main:73 Completed org.torproject.metrics.stats.hidserv.Main module. 2021-12-21 21:56:25,675 INFO o.t.m.s.main.Main:70 Starting org.torproject.metrics.stats.clients.Main module. 2021-12-21 21:56:25,679 INFO o.t.m.s.clients.Main:36 Starting clients module. 2021-12-21 21:56:25,680 INFO o.t.m.s.clients.Main:38 Connecting to database. 2021-12-21 21:56:25,798 INFO o.t.m.s.clients.Main:41 Reading relay descriptors and importing relevant parts into the database. 2021-12-21 21:57:45,627 INFO o.t.m.s.clients.Main:45 Reading bridge descriptors and importing relevant parts into the database. 2021-12-21 21:58:09,639 INFO o.t.m.s.clients.Main:49 Processing newly imported data. 2021-12-21 23:02:56,067 INFO o.t.m.s.clients.Main:53 Querying aggregated statistics from the database. 2021-12-21 23:03:16,709 INFO o.t.m.s.clients.Main:59 Disconnecting from database. 2021-12-21 23:03:16,711 INFO o.t.m.s.clients.Main:62 Running detector. 2021-12-21 23:03:51,766 INFO o.t.m.s.clients.Main:65 Terminating clients module. 2021-12-21 23:03:51,816 INFO o.t.m.s.main.Main:73 Completed org.torproject.metrics.stats.clients.Main module. 2021-12-21 23:03:51,816 INFO o.t.m.s.main.Main:70 Starting org.torproject.metrics.stats.servers.Main module. 2021-12-21 23:03:51,817 INFO o.t.m.s.servers.Main:43 Starting servers module. 2021-12-21 23:03:51,817 INFO o.t.m.s.servers.Main:45 Reading descriptors and inserting relevant parts into the database. 2021-12-21 23:14:21,386 INFO o.t.m.s.servers.Main:77 Aggregating database entries. 2021-12-21 23:22:26,401 INFO o.t.m.s.servers.Main:80 Committing all updated parts in the database. 2021-12-21 23:22:26,837 INFO o.t.m.s.servers.Main:90 Querying aggregated statistics from the database. 2021-12-21 23:37:40,995 INFO o.t.m.s.servers.Main:105 Terminating servers module. 2021-12-21 23:37:41,082 INFO o.t.m.s.main.Main:73 Completed org.torproject.metrics.stats.servers.Main module. 2021-12-21 23:37:41,082 INFO o.t.m.s.main.Main:70 Starting org.torproject.metrics.stats.webstats.Main module. 2021-12-21 23:37:41,102 INFO o.t.m.s.w.Main:75 Starting webstats module. 2021-12-21 23:37:41,102 INFO o.t.m.s.w.Main:92 Connecting to database. 2021-12-21 23:37:41,165 INFO o.t.m.s.w.Main:95 Successfully connected to database. 2021-12-21 23:37:41,166 INFO o.t.m.s.w.Main:101 Querying previously imported log files. 2021-12-21 23:37:43,036 INFO o.t.m.s.w.Main:113 Found 289918 previously imported log files. 2021-12-21 23:38:38,954 INFO o.t.m.s.w.Main:268 Querying statistics from database. 2021-12-21 23:54:07,223 INFO o.t.m.s.w.Main:298 Writing 1442773 lines to /srv/metrics.torproject.org/metrics/work/modules/webstats/stats/webstats.csv. 2021-12-21 23:54:09,202 INFO o.t.m.s.w.Main:305 Disconnecting from database. 2021-12-21 23:54:09,203 INFO o.t.m.s.w.Main:87 Terminated webstats module. 2021-12-21 23:54:09,203 INFO o.t.m.s.main.Main:73 Completed org.torproject.metrics.stats.webstats.Main module. 2021-12-21 23:54:09,203 INFO o.t.m.s.main.Main:70 Starting org.torproject.metrics.stats.totalcw.Main module. 2021-12-21 23:54:09,205 INFO o.t.m.s.totalcw.Main:38 Starting totalcw module. 2021-12-21 23:54:09,205 INFO o.t.m.s.totalcw.Main:40 Reading consensuses and votes and inserting relevant parts into the database. 2021-12-21 23:55:15,775 INFO o.t.m.s.totalcw.Main:64 Committing all updated parts in the database. 2021-12-21 23:55:15,939 INFO o.t.m.s.totalcw.Main:74 Querying aggregated statistics from the database. 2021-12-21 23:55:18,996 INFO o.t.m.s.totalcw.Main:77 Writing aggregated statistics to /srv/metrics.torproject.org/metrics/work/modules/totalcw/stats/totalcw.csv. 2021-12-21 23:55:19,203 INFO o.t.m.s.totalcw.Main:82 Terminating totalcw module. 2021-12-21 23:55:19,203 INFO o.t.m.s.main.Main:73 Completed org.torproject.metrics.stats.totalcw.Main module. 2021-12-21 23:55:19,204 INFO o.t.m.s.main.Main:70 Starting org.torproject.metrics.stats.bridgedb.Main module. 2021-12-21 23:55:19,419 INFO o.t.m.s.main.Main:73 Completed org.torproject.metrics.stats.bridgedb.Main module. 2021-12-21 23:55:19,419 INFO o.t.m.s.main.Main:82 Making module data available. 2021-12-21 23:55:21,440 INFO o.t.m.s.main.Main:126 Successfully copied 22 files to stats output directory: [connbidirect2.csv, onionperf-failures.csv, buildtimes.csv, latencies.csv, torperf-1.1.csv, onionperf-throughput.csv, onionperf-including-partials.csv, bandwidth.csv, advbwdist.csv, v3hidserv.csv, hidserv.csv, clients.csv, userstats-combined.csv, platforms.csv, relayflags.csv, versions.csv, advbw.csv, networksize.csv, ipv6servers.csv, webstats.csv, totalcw.csv, bridgedb-stats.csv] 2021-12-21 23:55:21,440 INFO o.t.m.s.main.Main:130 Completed metrics update run.

1 0

Cron <onionoo@onionoo-backend-02> /srv/onionoo.torproject.org/onionoo/bin/service/onionoo
by root＠onionoo-backend-02.torproject.org 22 Dec '21

22 Dec '21

Traceback (most recent call last): File "/srv/onionoo.torproject.org/onionoo/bin/service/onionoo", line 23, in <module> list_of_errors = Counter([(x.split(' ')[2], x.split(' ')[3].split(':')[0]) for x in str(out).split('\\n') if x != '"']) File "/srv/onionoo.torproject.org/onionoo/bin/service/onionoo", line 23, in <listcomp> list_of_errors = Counter([(x.split(' ')[2], x.split(' ')[3].split(':')[0]) for x in str(out).split('\\n') if x != '"']) IndexError: list index out of range During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/srv/onionoo.torproject.org/onionoo/bin/service/onionoo", line 29, in <module> g_warnings.labels(host, "Problem running exporter - {}".format(e)).set(1) File "/usr/lib/python3/dist-packages/prometheus_client/metrics.py", line 146, in labels raise ValueError('Incorrect label count') ValueError: Incorrect label count

1 0

Cron <metrics@meronense> cd /srv/metrics.torproject.org/metrics/work/modules/logs && cat metrics.log | grep -v "Could not insert bandwidth "
by root＠meronense.torproject.org 21 Dec '21

21 Dec '21

2021-12-21 08:40:02,284 INFO o.t.m.s.main.Main:32 Starting metrics update run. 2021-12-21 08:40:02,324 INFO o.t.m.s.main.Main:70 Starting org.torproject.metrics.stats.collectdescs.Main module. 2021-12-21 08:40:02,327 INFO o.t.d.i.DescriptorIndexCollector:47 Starting descriptor collection. 2021-12-21 08:40:02,327 INFO o.t.d.i.DescriptorIndexCollector:63 Indexing local directory /srv/metrics.torproject.org/metrics/work/shared/in. 2021-12-21 08:40:02,366 INFO o.t.d.i.DescriptorIndexCollector:75 Fetching remote index file https://collector.torproject.org/index/index.json. 2021-12-21 08:40:03,118 INFO o.t.d.i.DescriptorIndexCollector:84 Fetching remote files from https://collector.torproject.org. 2021-12-21 08:40:26,141 INFO o.t.d.i.DescriptorIndexCollector:90 Deleting extraneous files from local directory /srv/metrics.torproject.org/metrics/work/shared/in. 2021-12-21 08:40:26,371 INFO o.t.d.i.DescriptorIndexCollector:95 Finished descriptor collection. 2021-12-21 08:40:26,371 INFO o.t.m.s.main.Main:73 Completed org.torproject.metrics.stats.collectdescs.Main module. 2021-12-21 08:40:26,371 INFO o.t.m.s.main.Main:70 Starting org.torproject.metrics.stats.connbidirect.Main module. 2021-12-21 08:40:33,429 INFO o.t.m.s.main.Main:73 Completed org.torproject.metrics.stats.connbidirect.Main module. 2021-12-21 08:40:33,429 INFO o.t.m.s.main.Main:70 Starting org.torproject.metrics.stats.onionperf.Main module. 2021-12-21 08:40:33,430 INFO o.t.m.s.o.Main:46 Starting onionperf module. 2021-12-21 08:40:33,431 INFO o.t.m.s.o.Main:67 Connecting to database. 2021-12-21 08:40:33,745 INFO o.t.m.s.o.Main:70 Successfully connected to database. 2021-12-21 08:47:44,857 INFO o.t.m.s.o.Main:261 Querying statistics including partials from database. 2021-12-21 08:48:13,599 INFO o.t.m.s.o.Main:402 Writing 59959 lines to /srv/metrics.torproject.org/metrics/work/modules/onionperf/stats/onionperf-including-partials.csv. 2021-12-21 08:48:14,493 INFO o.t.m.s.o.Main:286 Querying timeout and failure statistics from database. 2021-12-21 08:48:18,073 INFO o.t.m.s.o.Main:402 Writing 19991 lines to /srv/metrics.torproject.org/metrics/work/modules/onionperf/stats/onionperf-failures.csv. 2021-12-21 08:48:18,076 INFO o.t.m.s.o.Main:309 Querying buildtime statistics from database. 2021-12-21 08:49:05,533 INFO o.t.m.s.o.Main:402 Writing 48196 lines to /srv/metrics.torproject.org/metrics/work/modules/onionperf/stats/buildtimes.csv. 2021-12-21 08:49:05,540 INFO o.t.m.s.o.Main:332 Querying latency statistics from database. 2021-12-21 08:49:17,125 INFO o.t.m.s.o.Main:402 Writing 19024 lines to /srv/metrics.torproject.org/metrics/work/modules/onionperf/stats/latencies.csv. 2021-12-21 08:49:17,129 INFO o.t.m.s.o.Main:357 Querying throughput statistics from database. 2021-12-21 08:49:28,487 INFO o.t.m.s.o.Main:402 Writing 19991 lines to /srv/metrics.torproject.org/metrics/work/modules/onionperf/stats/onionperf-throughput.csv. 2021-12-21 08:49:28,490 INFO o.t.m.s.o.Main:409 Disconnecting from database. 2021-12-21 08:49:28,491 INFO o.t.m.s.o.Main:62 Terminated onionperf module. 2021-12-21 08:49:28,491 INFO o.t.m.s.main.Main:73 Completed org.torproject.metrics.stats.onionperf.Main module. 2021-12-21 08:49:28,491 INFO o.t.m.s.main.Main:70 Starting org.torproject.metrics.stats.bwhist.Main module. 2021-12-21 08:49:28,491 INFO o.t.m.s.bwhist.Main:37 Starting bwhist module. 2021-12-21 08:49:28,491 INFO o.t.m.s.bwhist.Main:39 Reading descriptors and inserting relevant parts into the database. 2021-12-21 09:01:52,339 INFO o.t.m.s.b.RelayDescriptorDatabaseImporter:543 Finished importing relay descriptors: 158227 network status entries and 140887 bandwidth history elements 2021-12-21 09:01:55,350 INFO o.t.m.s.bwhist.Main:50 Aggregating database entries. 2021-12-21 09:03:28,978 INFO o.t.m.s.b.RelayDescriptorDatabaseImporter:543 Finished importing relay descriptors: 158227 network status entries and 140887 bandwidth history elements 2021-12-21 09:03:31,114 INFO o.t.m.s.bwhist.Main:53 Querying aggregated statistics from the database. 2021-12-21 09:03:31,518 INFO o.t.m.s.bwhist.Main:57 Closing database connection. 2021-12-21 09:03:31,520 INFO o.t.m.s.bwhist.Main:60 Terminating bwhist module. 2021-12-21 09:03:31,520 INFO o.t.m.s.main.Main:73 Completed org.torproject.metrics.stats.bwhist.Main module. 2021-12-21 09:03:31,520 INFO o.t.m.s.main.Main:70 Starting org.torproject.metrics.stats.advbwdist.Main module. 2021-12-21 09:04:31,437 INFO o.t.m.s.main.Main:73 Completed org.torproject.metrics.stats.advbwdist.Main module. 2021-12-21 09:04:31,438 INFO o.t.m.s.main.Main:70 Starting org.torproject.metrics.stats.hidserv.Main module. 2021-12-21 09:04:31,438 INFO o.t.m.s.hidserv.Main:37 Initializing parser and reading parse history... 2021-12-21 09:04:31,521 INFO o.t.m.s.hidserv.Main:52 Parsing descriptors... 2021-12-21 09:06:08,290 INFO o.t.m.s.hidserv.Main:59 Writing parse history... 2021-12-21 09:06:08,295 INFO o.t.m.s.hidserv.Main:66 Extrapolating statistics... 2021-12-21 09:11:14,809 INFO o.t.m.s.hidserv.Main:82 Aggregating v2 statistics... 2021-12-21 09:12:56,336 INFO o.t.m.s.hidserv.Main:98 Aggregating v3 statistics... 2021-12-21 09:12:56,844 INFO o.t.m.s.hidserv.Main:108 Terminating. 2021-12-21 09:12:56,845 INFO o.t.m.s.main.Main:73 Completed org.torproject.metrics.stats.hidserv.Main module. 2021-12-21 09:12:56,845 INFO o.t.m.s.main.Main:70 Starting org.torproject.metrics.stats.clients.Main module. 2021-12-21 09:12:56,865 INFO o.t.m.s.clients.Main:36 Starting clients module. 2021-12-21 09:12:56,865 INFO o.t.m.s.clients.Main:38 Connecting to database. 2021-12-21 09:12:56,960 INFO o.t.m.s.clients.Main:41 Reading relay descriptors and importing relevant parts into the database. 2021-12-21 09:14:36,026 INFO o.t.m.s.clients.Main:45 Reading bridge descriptors and importing relevant parts into the database. 2021-12-21 09:15:21,811 INFO o.t.m.s.clients.Main:49 Processing newly imported data. 2021-12-21 10:25:09,189 INFO o.t.m.s.clients.Main:53 Querying aggregated statistics from the database. 2021-12-21 10:25:27,344 INFO o.t.m.s.clients.Main:59 Disconnecting from database. 2021-12-21 10:25:27,345 INFO o.t.m.s.clients.Main:62 Running detector. 2021-12-21 10:26:01,318 INFO o.t.m.s.clients.Main:65 Terminating clients module. 2021-12-21 10:26:01,381 INFO o.t.m.s.main.Main:73 Completed org.torproject.metrics.stats.clients.Main module. 2021-12-21 10:26:01,381 INFO o.t.m.s.main.Main:70 Starting org.torproject.metrics.stats.servers.Main module. 2021-12-21 10:26:01,381 INFO o.t.m.s.servers.Main:43 Starting servers module. 2021-12-21 10:26:01,381 INFO o.t.m.s.servers.Main:45 Reading descriptors and inserting relevant parts into the database. 2021-12-21 10:40:55,132 INFO o.t.m.s.servers.Main:77 Aggregating database entries. 2021-12-21 10:45:04,522 INFO o.t.m.s.servers.Main:80 Committing all updated parts in the database. 2021-12-21 10:45:04,713 INFO o.t.m.s.servers.Main:90 Querying aggregated statistics from the database. 2021-12-21 10:59:08,440 INFO o.t.m.s.servers.Main:105 Terminating servers module. 2021-12-21 10:59:08,490 INFO o.t.m.s.main.Main:73 Completed org.torproject.metrics.stats.servers.Main module. 2021-12-21 10:59:08,491 INFO o.t.m.s.main.Main:70 Starting org.torproject.metrics.stats.webstats.Main module. 2021-12-21 10:59:08,594 INFO o.t.m.s.w.Main:75 Starting webstats module. 2021-12-21 10:59:08,594 INFO o.t.m.s.w.Main:92 Connecting to database. 2021-12-21 10:59:09,047 INFO o.t.m.s.w.Main:95 Successfully connected to database. 2021-12-21 10:59:09,048 INFO o.t.m.s.w.Main:101 Querying previously imported log files. 2021-12-21 10:59:11,096 INFO o.t.m.s.w.Main:113 Found 289742 previously imported log files. 2021-12-21 11:07:34,551 INFO o.t.m.s.w.Main:268 Querying statistics from database. 2021-12-21 11:25:06,884 INFO o.t.m.s.w.Main:298 Writing 1442773 lines to /srv/metrics.torproject.org/metrics/work/modules/webstats/stats/webstats.csv. 2021-12-21 11:25:08,658 INFO o.t.m.s.w.Main:305 Disconnecting from database. 2021-12-21 11:25:08,659 INFO o.t.m.s.w.Main:87 Terminated webstats module. 2021-12-21 11:25:08,659 INFO o.t.m.s.main.Main:73 Completed org.torproject.metrics.stats.webstats.Main module. 2021-12-21 11:25:08,659 INFO o.t.m.s.main.Main:70 Starting org.torproject.metrics.stats.totalcw.Main module. 2021-12-21 11:25:08,661 INFO o.t.m.s.totalcw.Main:38 Starting totalcw module. 2021-12-21 11:25:08,661 INFO o.t.m.s.totalcw.Main:40 Reading consensuses and votes and inserting relevant parts into the database. 2021-12-21 11:27:43,227 INFO o.t.m.s.totalcw.Main:64 Committing all updated parts in the database. 2021-12-21 11:27:43,343 INFO o.t.m.s.totalcw.Main:74 Querying aggregated statistics from the database. 2021-12-21 11:27:48,953 INFO o.t.m.s.totalcw.Main:77 Writing aggregated statistics to /srv/metrics.torproject.org/metrics/work/modules/totalcw/stats/totalcw.csv. 2021-12-21 11:27:49,225 INFO o.t.m.s.totalcw.Main:82 Terminating totalcw module. 2021-12-21 11:27:49,226 INFO o.t.m.s.main.Main:73 Completed org.torproject.metrics.stats.totalcw.Main module. 2021-12-21 11:27:49,226 INFO o.t.m.s.main.Main:70 Starting org.torproject.metrics.stats.bridgedb.Main module. 2021-12-21 11:27:49,626 INFO o.t.m.s.main.Main:73 Completed org.torproject.metrics.stats.bridgedb.Main module. 2021-12-21 11:27:49,626 INFO o.t.m.s.main.Main:82 Making module data available. 2021-12-21 11:28:01,072 INFO o.t.m.s.main.Main:126 Successfully copied 22 files to stats output directory: [connbidirect2.csv, onionperf-failures.csv, buildtimes.csv, latencies.csv, torperf-1.1.csv, onionperf-throughput.csv, onionperf-including-partials.csv, bandwidth.csv, advbwdist.csv, v3hidserv.csv, hidserv.csv, clients.csv, userstats-combined.csv, platforms.csv, relayflags.csv, versions.csv, advbw.csv, networksize.csv, ipv6servers.csv, webstats.csv, totalcw.csv, bridgedb-stats.csv] 2021-12-21 11:28:01,072 INFO o.t.m.s.main.Main:130 Completed metrics update run.

1 0

Retiring metrics alerts in its current format
by Silvia/Hiro 21 Dec '21

21 Dec '21

Hi everyone, Moving forward we are going to use this list as a service only list where admins will be receiving information about metrics services. We are talking about heartbeats and logs. Because we don't want to spam subscribers with services internals we are going to unsubscribe most people from the list. For discussions about metrics and activities from the Network health team you can subscribe to network-health(a)lists.torproject.org. Cheers, -hiro

1 0

Cron <metrics@meronense> cd /srv/metrics.torproject.org/metrics/work/modules/logs && cat metrics.log | grep -v "Could not insert bandwidth "
by root＠meronense.torproject.org 21 Dec '21

21 Dec '21

2021-12-21 08:40:02,284 INFO o.t.m.s.main.Main:32 Starting metrics update run. 2021-12-21 08:40:02,324 INFO o.t.m.s.main.Main:70 Starting org.torproject.metrics.stats.collectdescs.Main module. 2021-12-21 08:40:02,327 INFO o.t.d.i.DescriptorIndexCollector:47 Starting descriptor collection. 2021-12-21 08:40:02,327 INFO o.t.d.i.DescriptorIndexCollector:63 Indexing local directory /srv/metrics.torproject.org/metrics/work/shared/in. 2021-12-21 08:40:02,366 INFO o.t.d.i.DescriptorIndexCollector:75 Fetching remote index file https://collector.torproject.org/index/index.json. 2021-12-21 08:40:03,118 INFO o.t.d.i.DescriptorIndexCollector:84 Fetching remote files from https://collector.torproject.org. 2021-12-21 08:40:26,141 INFO o.t.d.i.DescriptorIndexCollector:90 Deleting extraneous files from local directory /srv/metrics.torproject.org/metrics/work/shared/in. 2021-12-21 08:40:26,371 INFO o.t.d.i.DescriptorIndexCollector:95 Finished descriptor collection. 2021-12-21 08:40:26,371 INFO o.t.m.s.main.Main:73 Completed org.torproject.metrics.stats.collectdescs.Main module. 2021-12-21 08:40:26,371 INFO o.t.m.s.main.Main:70 Starting org.torproject.metrics.stats.connbidirect.Main module. 2021-12-21 08:40:33,429 INFO o.t.m.s.main.Main:73 Completed org.torproject.metrics.stats.connbidirect.Main module. 2021-12-21 08:40:33,429 INFO o.t.m.s.main.Main:70 Starting org.torproject.metrics.stats.onionperf.Main module. 2021-12-21 08:40:33,430 INFO o.t.m.s.o.Main:46 Starting onionperf module. 2021-12-21 08:40:33,431 INFO o.t.m.s.o.Main:67 Connecting to database. 2021-12-21 08:40:33,745 INFO o.t.m.s.o.Main:70 Successfully connected to database. 2021-12-21 08:47:44,857 INFO o.t.m.s.o.Main:261 Querying statistics including partials from database. 2021-12-21 08:48:13,599 INFO o.t.m.s.o.Main:402 Writing 59959 lines to /srv/metrics.torproject.org/metrics/work/modules/onionperf/stats/onionperf-including-partials.csv. 2021-12-21 08:48:14,493 INFO o.t.m.s.o.Main:286 Querying timeout and failure statistics from database. 2021-12-21 08:48:18,073 INFO o.t.m.s.o.Main:402 Writing 19991 lines to /srv/metrics.torproject.org/metrics/work/modules/onionperf/stats/onionperf-failures.csv. 2021-12-21 08:48:18,076 INFO o.t.m.s.o.Main:309 Querying buildtime statistics from database. 2021-12-21 08:49:05,533 INFO o.t.m.s.o.Main:402 Writing 48196 lines to /srv/metrics.torproject.org/metrics/work/modules/onionperf/stats/buildtimes.csv. 2021-12-21 08:49:05,540 INFO o.t.m.s.o.Main:332 Querying latency statistics from database. 2021-12-21 08:49:17,125 INFO o.t.m.s.o.Main:402 Writing 19024 lines to /srv/metrics.torproject.org/metrics/work/modules/onionperf/stats/latencies.csv. 2021-12-21 08:49:17,129 INFO o.t.m.s.o.Main:357 Querying throughput statistics from database. 2021-12-21 08:49:28,487 INFO o.t.m.s.o.Main:402 Writing 19991 lines to /srv/metrics.torproject.org/metrics/work/modules/onionperf/stats/onionperf-throughput.csv. 2021-12-21 08:49:28,490 INFO o.t.m.s.o.Main:409 Disconnecting from database. 2021-12-21 08:49:28,491 INFO o.t.m.s.o.Main:62 Terminated onionperf module. 2021-12-21 08:49:28,491 INFO o.t.m.s.main.Main:73 Completed org.torproject.metrics.stats.onionperf.Main module. 2021-12-21 08:49:28,491 INFO o.t.m.s.main.Main:70 Starting org.torproject.metrics.stats.bwhist.Main module. 2021-12-21 08:49:28,491 INFO o.t.m.s.bwhist.Main:37 Starting bwhist module. 2021-12-21 08:49:28,491 INFO o.t.m.s.bwhist.Main:39 Reading descriptors and inserting relevant parts into the database. 2021-12-21 09:01:52,339 INFO o.t.m.s.b.RelayDescriptorDatabaseImporter:543 Finished importing relay descriptors: 158227 network status entries and 140887 bandwidth history elements 2021-12-21 09:01:55,350 INFO o.t.m.s.bwhist.Main:50 Aggregating database entries. 2021-12-21 09:03:28,978 INFO o.t.m.s.b.RelayDescriptorDatabaseImporter:543 Finished importing relay descriptors: 158227 network status entries and 140887 bandwidth history elements 2021-12-21 09:03:31,114 INFO o.t.m.s.bwhist.Main:53 Querying aggregated statistics from the database. 2021-12-21 09:03:31,518 INFO o.t.m.s.bwhist.Main:57 Closing database connection. 2021-12-21 09:03:31,520 INFO o.t.m.s.bwhist.Main:60 Terminating bwhist module. 2021-12-21 09:03:31,520 INFO o.t.m.s.main.Main:73 Completed org.torproject.metrics.stats.bwhist.Main module. 2021-12-21 09:03:31,520 INFO o.t.m.s.main.Main:70 Starting org.torproject.metrics.stats.advbwdist.Main module. 2021-12-21 09:04:31,437 INFO o.t.m.s.main.Main:73 Completed org.torproject.metrics.stats.advbwdist.Main module. 2021-12-21 09:04:31,438 INFO o.t.m.s.main.Main:70 Starting org.torproject.metrics.stats.hidserv.Main module. 2021-12-21 09:04:31,438 INFO o.t.m.s.hidserv.Main:37 Initializing parser and reading parse history... 2021-12-21 09:04:31,521 INFO o.t.m.s.hidserv.Main:52 Parsing descriptors... 2021-12-21 09:06:08,290 INFO o.t.m.s.hidserv.Main:59 Writing parse history... 2021-12-21 09:06:08,295 INFO o.t.m.s.hidserv.Main:66 Extrapolating statistics... 2021-12-21 09:11:14,809 INFO o.t.m.s.hidserv.Main:82 Aggregating v2 statistics... 2021-12-21 09:12:56,336 INFO o.t.m.s.hidserv.Main:98 Aggregating v3 statistics... 2021-12-21 09:12:56,844 INFO o.t.m.s.hidserv.Main:108 Terminating. 2021-12-21 09:12:56,845 INFO o.t.m.s.main.Main:73 Completed org.torproject.metrics.stats.hidserv.Main module. 2021-12-21 09:12:56,845 INFO o.t.m.s.main.Main:70 Starting org.torproject.metrics.stats.clients.Main module. 2021-12-21 09:12:56,865 INFO o.t.m.s.clients.Main:36 Starting clients module. 2021-12-21 09:12:56,865 INFO o.t.m.s.clients.Main:38 Connecting to database. 2021-12-21 09:12:56,960 INFO o.t.m.s.clients.Main:41 Reading relay descriptors and importing relevant parts into the database. 2021-12-21 09:14:36,026 INFO o.t.m.s.clients.Main:45 Reading bridge descriptors and importing relevant parts into the database. 2021-12-21 09:15:21,811 INFO o.t.m.s.clients.Main:49 Processing newly imported data. 2021-12-21 10:25:09,189 INFO o.t.m.s.clients.Main:53 Querying aggregated statistics from the database. 2021-12-21 10:25:27,344 INFO o.t.m.s.clients.Main:59 Disconnecting from database. 2021-12-21 10:25:27,345 INFO o.t.m.s.clients.Main:62 Running detector. 2021-12-21 10:26:01,318 INFO o.t.m.s.clients.Main:65 Terminating clients module. 2021-12-21 10:26:01,381 INFO o.t.m.s.main.Main:73 Completed org.torproject.metrics.stats.clients.Main module. 2021-12-21 10:26:01,381 INFO o.t.m.s.main.Main:70 Starting org.torproject.metrics.stats.servers.Main module. 2021-12-21 10:26:01,381 INFO o.t.m.s.servers.Main:43 Starting servers module. 2021-12-21 10:26:01,381 INFO o.t.m.s.servers.Main:45 Reading descriptors and inserting relevant parts into the database. 2021-12-21 10:40:55,132 INFO o.t.m.s.servers.Main:77 Aggregating database entries. 2021-12-21 10:45:04,522 INFO o.t.m.s.servers.Main:80 Committing all updated parts in the database. 2021-12-21 10:45:04,713 INFO o.t.m.s.servers.Main:90 Querying aggregated statistics from the database. 2021-12-21 10:59:08,440 INFO o.t.m.s.servers.Main:105 Terminating servers module. 2021-12-21 10:59:08,490 INFO o.t.m.s.main.Main:73 Completed org.torproject.metrics.stats.servers.Main module. 2021-12-21 10:59:08,491 INFO o.t.m.s.main.Main:70 Starting org.torproject.metrics.stats.webstats.Main module. 2021-12-21 10:59:08,594 INFO o.t.m.s.w.Main:75 Starting webstats module. 2021-12-21 10:59:08,594 INFO o.t.m.s.w.Main:92 Connecting to database. 2021-12-21 10:59:09,047 INFO o.t.m.s.w.Main:95 Successfully connected to database. 2021-12-21 10:59:09,048 INFO o.t.m.s.w.Main:101 Querying previously imported log files. 2021-12-21 10:59:11,096 INFO o.t.m.s.w.Main:113 Found 289742 previously imported log files. 2021-12-21 11:07:34,551 INFO o.t.m.s.w.Main:268 Querying statistics from database. 2021-12-21 11:25:06,884 INFO o.t.m.s.w.Main:298 Writing 1442773 lines to /srv/metrics.torproject.org/metrics/work/modules/webstats/stats/webstats.csv. 2021-12-21 11:25:08,658 INFO o.t.m.s.w.Main:305 Disconnecting from database. 2021-12-21 11:25:08,659 INFO o.t.m.s.w.Main:87 Terminated webstats module. 2021-12-21 11:25:08,659 INFO o.t.m.s.main.Main:73 Completed org.torproject.metrics.stats.webstats.Main module. 2021-12-21 11:25:08,659 INFO o.t.m.s.main.Main:70 Starting org.torproject.metrics.stats.totalcw.Main module. 2021-12-21 11:25:08,661 INFO o.t.m.s.totalcw.Main:38 Starting totalcw module. 2021-12-21 11:25:08,661 INFO o.t.m.s.totalcw.Main:40 Reading consensuses and votes and inserting relevant parts into the database. 2021-12-21 11:27:43,227 INFO o.t.m.s.totalcw.Main:64 Committing all updated parts in the database. 2021-12-21 11:27:43,343 INFO o.t.m.s.totalcw.Main:74 Querying aggregated statistics from the database. 2021-12-21 11:27:48,953 INFO o.t.m.s.totalcw.Main:77 Writing aggregated statistics to /srv/metrics.torproject.org/metrics/work/modules/totalcw/stats/totalcw.csv. 2021-12-21 11:27:49,225 INFO o.t.m.s.totalcw.Main:82 Terminating totalcw module. 2021-12-21 11:27:49,226 INFO o.t.m.s.main.Main:73 Completed org.torproject.metrics.stats.totalcw.Main module. 2021-12-21 11:27:49,226 INFO o.t.m.s.main.Main:70 Starting org.torproject.metrics.stats.bridgedb.Main module. 2021-12-21 11:27:49,626 INFO o.t.m.s.main.Main:73 Completed org.torproject.metrics.stats.bridgedb.Main module. 2021-12-21 11:27:49,626 INFO o.t.m.s.main.Main:82 Making module data available. 2021-12-21 11:28:01,072 INFO o.t.m.s.main.Main:126 Successfully copied 22 files to stats output directory: [connbidirect2.csv, onionperf-failures.csv, buildtimes.csv, latencies.csv, torperf-1.1.csv, onionperf-throughput.csv, onionperf-including-partials.csv, bandwidth.csv, advbwdist.csv, v3hidserv.csv, hidserv.csv, clients.csv, userstats-combined.csv, platforms.csv, relayflags.csv, versions.csv, advbw.csv, networksize.csv, ipv6servers.csv, webstats.csv, totalcw.csv, bridgedb-stats.csv] 2021-12-21 11:28:01,072 INFO o.t.m.s.main.Main:130 Completed metrics update run.

1 0

Cron <metrics@meronense> cd /srv/metrics.torproject.org/metrics/work/modules/logs && cat metrics.log | grep -v "Could not insert bandwidth "
by root＠meronense.torproject.org 21 Dec '21

21 Dec '21

2021-12-20 08:40:02,547 INFO o.t.m.s.main.Main:32 Starting metrics update run. 2021-12-20 08:40:02,587 INFO o.t.m.s.main.Main:70 Starting org.torproject.metrics.stats.collectdescs.Main module. 2021-12-20 08:40:02,590 INFO o.t.d.i.DescriptorIndexCollector:47 Starting descriptor collection. 2021-12-20 08:40:02,590 INFO o.t.d.i.DescriptorIndexCollector:63 Indexing local directory /srv/metrics.torproject.org/metrics/work/shared/in. 2021-12-20 08:40:02,629 INFO o.t.d.i.DescriptorIndexCollector:75 Fetching remote index file https://collector.torproject.org/index/index.json. 2021-12-20 08:40:03,487 INFO o.t.d.i.DescriptorIndexCollector:84 Fetching remote files from https://collector.torproject.org. 2021-12-20 08:40:27,338 INFO o.t.d.i.DescriptorIndexCollector:90 Deleting extraneous files from local directory /srv/metrics.torproject.org/metrics/work/shared/in. 2021-12-20 08:40:27,405 INFO o.t.d.i.DescriptorIndexCollector:95 Finished descriptor collection. 2021-12-20 08:40:27,405 INFO o.t.m.s.main.Main:73 Completed org.torproject.metrics.stats.collectdescs.Main module. 2021-12-20 08:40:27,405 INFO o.t.m.s.main.Main:70 Starting org.torproject.metrics.stats.connbidirect.Main module. 2021-12-20 08:40:33,795 INFO o.t.m.s.main.Main:73 Completed org.torproject.metrics.stats.connbidirect.Main module. 2021-12-20 08:40:33,795 INFO o.t.m.s.main.Main:70 Starting org.torproject.metrics.stats.onionperf.Main module. 2021-12-20 08:40:33,796 INFO o.t.m.s.o.Main:46 Starting onionperf module. 2021-12-20 08:40:33,796 INFO o.t.m.s.o.Main:67 Connecting to database. 2021-12-20 08:40:33,986 INFO o.t.m.s.o.Main:70 Successfully connected to database. 2021-12-20 08:45:34,467 INFO o.t.m.s.o.Main:261 Querying statistics including partials from database. 2021-12-20 08:46:04,971 INFO o.t.m.s.o.Main:402 Writing 59929 lines to /srv/metrics.torproject.org/metrics/work/modules/onionperf/stats/onionperf-including-partials.csv. 2021-12-20 08:46:05,402 INFO o.t.m.s.o.Main:286 Querying timeout and failure statistics from database. 2021-12-20 08:46:07,772 INFO o.t.m.s.o.Main:402 Writing 19981 lines to /srv/metrics.torproject.org/metrics/work/modules/onionperf/stats/onionperf-failures.csv. 2021-12-20 08:46:07,776 INFO o.t.m.s.o.Main:309 Querying buildtime statistics from database. 2021-12-20 08:46:49,667 INFO o.t.m.s.o.Main:402 Writing 48178 lines to /srv/metrics.torproject.org/metrics/work/modules/onionperf/stats/buildtimes.csv. 2021-12-20 08:46:49,683 INFO o.t.m.s.o.Main:332 Querying latency statistics from database. 2021-12-20 08:47:01,628 INFO o.t.m.s.o.Main:402 Writing 19014 lines to /srv/metrics.torproject.org/metrics/work/modules/onionperf/stats/latencies.csv. 2021-12-20 08:47:01,631 INFO o.t.m.s.o.Main:357 Querying throughput statistics from database. 2021-12-20 08:47:13,749 INFO o.t.m.s.o.Main:402 Writing 19981 lines to /srv/metrics.torproject.org/metrics/work/modules/onionperf/stats/onionperf-throughput.csv. 2021-12-20 08:47:13,752 INFO o.t.m.s.o.Main:409 Disconnecting from database. 2021-12-20 08:47:13,753 INFO o.t.m.s.o.Main:62 Terminated onionperf module. 2021-12-20 08:47:13,753 INFO o.t.m.s.main.Main:73 Completed org.torproject.metrics.stats.onionperf.Main module. 2021-12-20 08:47:13,753 INFO o.t.m.s.main.Main:70 Starting org.torproject.metrics.stats.bwhist.Main module. 2021-12-20 08:47:13,753 INFO o.t.m.s.bwhist.Main:37 Starting bwhist module. 2021-12-20 08:47:13,753 INFO o.t.m.s.bwhist.Main:39 Reading descriptors and inserting relevant parts into the database. 2021-12-20 08:57:53,157 INFO o.t.m.s.b.RelayDescriptorDatabaseImporter:543 Finished importing relay descriptors: 158149 network status entries and 161082 bandwidth history elements 2021-12-20 08:57:54,249 INFO o.t.m.s.bwhist.Main:50 Aggregating database entries. 2021-12-20 08:58:36,826 INFO o.t.m.s.b.RelayDescriptorDatabaseImporter:543 Finished importing relay descriptors: 158149 network status entries and 161082 bandwidth history elements 2021-12-20 08:58:39,522 INFO o.t.m.s.bwhist.Main:53 Querying aggregated statistics from the database. 2021-12-20 08:58:40,169 INFO o.t.m.s.bwhist.Main:57 Closing database connection. 2021-12-20 08:58:40,169 INFO o.t.m.s.bwhist.Main:60 Terminating bwhist module. 2021-12-20 08:58:40,169 INFO o.t.m.s.main.Main:73 Completed org.torproject.metrics.stats.bwhist.Main module. 2021-12-20 08:58:40,169 INFO o.t.m.s.main.Main:70 Starting org.torproject.metrics.stats.advbwdist.Main module. 2021-12-20 08:59:34,703 INFO o.t.m.s.main.Main:73 Completed org.torproject.metrics.stats.advbwdist.Main module. 2021-12-20 08:59:34,703 INFO o.t.m.s.main.Main:70 Starting org.torproject.metrics.stats.hidserv.Main module. 2021-12-20 08:59:34,703 INFO o.t.m.s.hidserv.Main:37 Initializing parser and reading parse history... 2021-12-20 08:59:34,802 INFO o.t.m.s.hidserv.Main:52 Parsing descriptors... 2021-12-20 09:01:08,526 INFO o.t.m.s.hidserv.Main:59 Writing parse history... 2021-12-20 09:01:08,532 INFO o.t.m.s.hidserv.Main:66 Extrapolating statistics... 2021-12-20 09:06:44,978 INFO o.t.m.s.hidserv.Main:82 Aggregating v2 statistics... 2021-12-20 09:08:23,654 INFO o.t.m.s.hidserv.Main:98 Aggregating v3 statistics... 2021-12-20 09:08:24,053 INFO o.t.m.s.hidserv.Main:108 Terminating. 2021-12-20 09:08:24,054 INFO o.t.m.s.main.Main:73 Completed org.torproject.metrics.stats.hidserv.Main module. 2021-12-20 09:08:24,054 INFO o.t.m.s.main.Main:70 Starting org.torproject.metrics.stats.clients.Main module. 2021-12-20 09:08:24,054 INFO o.t.m.s.clients.Main:36 Starting clients module. 2021-12-20 09:08:24,055 INFO o.t.m.s.clients.Main:38 Connecting to database. 2021-12-20 09:08:24,103 INFO o.t.m.s.clients.Main:41 Reading relay descriptors and importing relevant parts into the database. 2021-12-20 09:09:40,978 INFO o.t.m.s.clients.Main:45 Reading bridge descriptors and importing relevant parts into the database. 2021-12-20 09:10:25,465 INFO o.t.m.s.clients.Main:49 Processing newly imported data. 2021-12-20 10:17:47,320 INFO o.t.m.s.clients.Main:53 Querying aggregated statistics from the database. 2021-12-20 10:18:03,149 INFO o.t.m.s.clients.Main:59 Disconnecting from database. 2021-12-20 10:18:03,150 INFO o.t.m.s.clients.Main:62 Running detector. 2021-12-20 10:18:36,312 INFO o.t.m.s.clients.Main:65 Terminating clients module. 2021-12-20 10:18:36,321 INFO o.t.m.s.main.Main:73 Completed org.torproject.metrics.stats.clients.Main module. 2021-12-20 10:18:36,321 INFO o.t.m.s.main.Main:70 Starting org.torproject.metrics.stats.servers.Main module. 2021-12-20 10:18:36,321 INFO o.t.m.s.servers.Main:43 Starting servers module. 2021-12-20 10:18:36,321 INFO o.t.m.s.servers.Main:45 Reading descriptors and inserting relevant parts into the database. 2021-12-20 10:39:23,925 INFO o.t.m.s.servers.Main:77 Aggregating database entries. 2021-12-20 10:45:02,474 INFO o.t.m.s.servers.Main:80 Committing all updated parts in the database. 2021-12-20 10:45:02,515 INFO o.t.m.s.servers.Main:90 Querying aggregated statistics from the database. 2021-12-20 11:00:54,576 INFO o.t.m.s.servers.Main:105 Terminating servers module. 2021-12-20 11:00:54,595 INFO o.t.m.s.main.Main:73 Completed org.torproject.metrics.stats.servers.Main module. 2021-12-20 11:00:54,595 INFO o.t.m.s.main.Main:70 Starting org.torproject.metrics.stats.webstats.Main module. 2021-12-20 11:00:54,936 INFO o.t.m.s.w.Main:75 Starting webstats module. 2021-12-20 11:00:54,937 INFO o.t.m.s.w.Main:92 Connecting to database. 2021-12-20 11:00:55,030 INFO o.t.m.s.w.Main:95 Successfully connected to database. 2021-12-20 11:00:55,030 INFO o.t.m.s.w.Main:101 Querying previously imported log files. 2021-12-20 11:00:58,332 INFO o.t.m.s.w.Main:113 Found 289576 previously imported log files. 2021-12-20 11:09:56,980 INFO o.t.m.s.w.Main:268 Querying statistics from database. 2021-12-20 11:37:13,272 INFO o.t.m.s.w.Main:298 Writing 1441906 lines to /srv/metrics.torproject.org/metrics/work/modules/webstats/stats/webstats.csv. 2021-12-20 11:37:14,020 INFO o.t.m.s.w.Main:305 Disconnecting from database. 2021-12-20 11:37:14,022 INFO o.t.m.s.w.Main:87 Terminated webstats module. 2021-12-20 11:37:14,022 INFO o.t.m.s.main.Main:73 Completed org.torproject.metrics.stats.webstats.Main module. 2021-12-20 11:37:14,022 INFO o.t.m.s.main.Main:70 Starting org.torproject.metrics.stats.totalcw.Main module. 2021-12-20 11:37:14,024 INFO o.t.m.s.totalcw.Main:38 Starting totalcw module. 2021-12-20 11:37:14,024 INFO o.t.m.s.totalcw.Main:40 Reading consensuses and votes and inserting relevant parts into the database. 2021-12-20 11:39:08,418 INFO o.t.m.s.totalcw.Main:64 Committing all updated parts in the database. 2021-12-20 11:39:08,579 INFO o.t.m.s.totalcw.Main:74 Querying aggregated statistics from the database. 2021-12-20 11:39:12,457 INFO o.t.m.s.totalcw.Main:77 Writing aggregated statistics to /srv/metrics.torproject.org/metrics/work/modules/totalcw/stats/totalcw.csv. 2021-12-20 11:39:12,774 INFO o.t.m.s.totalcw.Main:82 Terminating totalcw module. 2021-12-20 11:39:12,775 INFO o.t.m.s.main.Main:73 Completed org.torproject.metrics.stats.totalcw.Main module. 2021-12-20 11:39:12,775 INFO o.t.m.s.main.Main:70 Starting org.torproject.metrics.stats.bridgedb.Main module. 2021-12-20 11:39:12,986 INFO o.t.m.s.main.Main:73 Completed org.torproject.metrics.stats.bridgedb.Main module. 2021-12-20 11:39:12,987 INFO o.t.m.s.main.Main:82 Making module data available. 2021-12-20 11:39:14,498 INFO o.t.m.s.main.Main:126 Successfully copied 22 files to stats output directory: [connbidirect2.csv, onionperf-failures.csv, buildtimes.csv, latencies.csv, torperf-1.1.csv, onionperf-throughput.csv, onionperf-including-partials.csv, bandwidth.csv, advbwdist.csv, v3hidserv.csv, hidserv.csv, clients.csv, userstats-combined.csv, platforms.csv, relayflags.csv, versions.csv, advbw.csv, networksize.csv, ipv6servers.csv, webstats.csv, totalcw.csv, bridgedb-stats.csv] 2021-12-20 11:39:14,498 INFO o.t.m.s.main.Main:130 Completed metrics update run.

1 0

Cron <onionoo@onionoo-backend-02> /srv/onionoo.torproject.org/onionoo/bin/service/onionoo
by root＠onionoo-backend-02.torproject.org 21 Dec '21

21 Dec '21

Traceback (most recent call last): File "/srv/onionoo.torproject.org/onionoo/bin/service/onionoo", line 23, in <module> list_of_errors = Counter([(x.split(' ')[2], x.split(' ')[3].split(':')[0]) for x in str(out).split('\\n') if x != '"']) File "/srv/onionoo.torproject.org/onionoo/bin/service/onionoo", line 23, in <listcomp> list_of_errors = Counter([(x.split(' ')[2], x.split(' ')[3].split(':')[0]) for x in str(out).split('\\n') if x != '"']) IndexError: list index out of range During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/srv/onionoo.torproject.org/onionoo/bin/service/onionoo", line 29, in <module> g_warnings.labels(host, "Problem running exporter - {}".format(e)).set(1) File "/usr/lib/python3/dist-packages/prometheus_client/metrics.py", line 146, in labels raise ValueError('Incorrect label count') ValueError: Incorrect label count

1 0

Cron <onionoo@onionoo-backend-02> /srv/onionoo.torproject.org/onionoo/bin/service/onionoo
by root＠onionoo-backend-02.torproject.org 21 Dec '21

21 Dec '21

Traceback (most recent call last): File "/srv/onionoo.torproject.org/onionoo/bin/service/onionoo", line 23, in <module> list_of_errors = Counter([(x.split(' ')[2], x.split(' ')[3].split(':')[0]) for x in str(out).split('\\n') if x != '"']) File "/srv/onionoo.torproject.org/onionoo/bin/service/onionoo", line 23, in <listcomp> list_of_errors = Counter([(x.split(' ')[2], x.split(' ')[3].split(':')[0]) for x in str(out).split('\\n') if x != '"']) IndexError: list index out of range During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/srv/onionoo.torproject.org/onionoo/bin/service/onionoo", line 29, in <module> g_warnings.labels(host, "Problem running exporter - {}".format(e)).set(1) File "/usr/lib/python3/dist-packages/prometheus_client/metrics.py", line 146, in labels raise ValueError('Incorrect label count') ValueError: Incorrect label count

1 0