November 2021 - metrics-alerts - lists.torproject.org

Cron <metrics@meronense> cd /srv/metrics.torproject.org/metrics/work/modules/logs && cat metrics.log | grep -v "Could not insert bandwidth "
by root＠meronense.torproject.org 23 Nov '21

23 Nov '21

2021-11-23 08:40:03,201 INFO o.t.m.s.main.Main:32 Starting metrics update run. 2021-11-23 08:40:03,669 INFO o.t.m.s.main.Main:70 Starting org.torproject.metrics.stats.collectdescs.Main module. 2021-11-23 08:40:03,921 INFO o.t.d.i.DescriptorIndexCollector:47 Starting descriptor collection. 2021-11-23 08:40:04,146 INFO o.t.d.i.DescriptorIndexCollector:63 Indexing local directory /srv/metrics.torproject.org/metrics/work/shared/in. 2021-11-23 08:40:08,889 INFO o.t.d.i.DescriptorIndexCollector:75 Fetching remote index file https://collector.torproject.org/index/index.json. 2021-11-23 08:40:12,934 INFO o.t.d.i.DescriptorIndexCollector:84 Fetching remote files from https://collector.torproject.org. 2021-11-23 08:40:25,247 WARN o.t.d.i.DescriptorIndexCollector:132 Fetched remote file 2021-11-21-04-00-00-vote-0232AF901C31A04EE9848595AF9BB7620D4C5B2E-9DAFAF1CE0ABC369CCD86FB1D7E38C33CCFAB1D6 from https://collector.torproject.org has a size of 4669122 bytes which is different from the expected 1273856 bytes. Not storing this file. 2021-11-23 08:40:54,891 INFO o.t.d.i.DescriptorIndexCollector:90 Deleting extraneous files from local directory /srv/metrics.torproject.org/metrics/work/shared/in. 2021-11-23 08:40:57,279 INFO o.t.d.i.DescriptorIndexCollector:95 Finished descriptor collection. 2021-11-23 08:40:57,279 INFO o.t.m.s.main.Main:73 Completed org.torproject.metrics.stats.collectdescs.Main module. 2021-11-23 08:40:57,279 INFO o.t.m.s.main.Main:70 Starting org.torproject.metrics.stats.connbidirect.Main module. 2021-11-23 08:41:10,677 INFO o.t.m.s.main.Main:73 Completed org.torproject.metrics.stats.connbidirect.Main module. 2021-11-23 08:41:10,678 INFO o.t.m.s.main.Main:70 Starting org.torproject.metrics.stats.onionperf.Main module. 2021-11-23 08:41:10,679 INFO o.t.m.s.o.Main:46 Starting onionperf module. 2021-11-23 08:41:10,680 INFO o.t.m.s.o.Main:67 Connecting to database. 2021-11-23 08:41:14,668 INFO o.t.m.s.o.Main:70 Successfully connected to database. 2021-11-23 08:52:04,203 INFO o.t.m.s.o.Main:261 Querying statistics including partials from database. 2021-11-23 08:52:53,028 INFO o.t.m.s.o.Main:402 Writing 59119 lines to /srv/metrics.torproject.org/metrics/work/modules/onionperf/stats/onionperf-including-partials.csv. 2021-11-23 08:52:53,117 INFO o.t.m.s.o.Main:286 Querying timeout and failure statistics from database. 2021-11-23 08:53:03,018 INFO o.t.m.s.o.Main:402 Writing 19711 lines to /srv/metrics.torproject.org/metrics/work/modules/onionperf/stats/onionperf-failures.csv. 2021-11-23 08:53:03,022 INFO o.t.m.s.o.Main:309 Querying buildtime statistics from database. 2021-11-23 08:53:43,329 INFO o.t.m.s.o.Main:402 Writing 47692 lines to /srv/metrics.torproject.org/metrics/work/modules/onionperf/stats/buildtimes.csv. 2021-11-23 08:53:43,558 INFO o.t.m.s.o.Main:332 Querying latency statistics from database. 2021-11-23 08:53:55,146 INFO o.t.m.s.o.Main:402 Writing 18744 lines to /srv/metrics.torproject.org/metrics/work/modules/onionperf/stats/latencies.csv. 2021-11-23 08:53:55,151 INFO o.t.m.s.o.Main:357 Querying throughput statistics from database. 2021-11-23 08:54:06,500 INFO o.t.m.s.o.Main:402 Writing 19711 lines to /srv/metrics.torproject.org/metrics/work/modules/onionperf/stats/onionperf-throughput.csv. 2021-11-23 08:54:06,582 INFO o.t.m.s.o.Main:409 Disconnecting from database. 2021-11-23 08:54:06,583 INFO o.t.m.s.o.Main:62 Terminated onionperf module. 2021-11-23 08:54:06,583 INFO o.t.m.s.main.Main:73 Completed org.torproject.metrics.stats.onionperf.Main module. 2021-11-23 08:54:06,583 INFO o.t.m.s.main.Main:70 Starting org.torproject.metrics.stats.bwhist.Main module. 2021-11-23 08:54:06,583 INFO o.t.m.s.bwhist.Main:37 Starting bwhist module. 2021-11-23 08:54:06,583 INFO o.t.m.s.bwhist.Main:39 Reading descriptors and inserting relevant parts into the database. 2021-11-23 09:09:03,588 INFO o.t.m.s.b.RelayDescriptorDatabaseImporter:543 Finished importing relay descriptors: 94448 network status entries and 83665 bandwidth history elements 2021-11-23 09:09:05,237 INFO o.t.m.s.bwhist.Main:50 Aggregating database entries. 2021-11-23 09:10:14,357 INFO o.t.m.s.b.RelayDescriptorDatabaseImporter:543 Finished importing relay descriptors: 94448 network status entries and 83665 bandwidth history elements 2021-11-23 09:10:16,740 INFO o.t.m.s.bwhist.Main:53 Querying aggregated statistics from the database. 2021-11-23 09:10:18,688 INFO o.t.m.s.bwhist.Main:57 Closing database connection. 2021-11-23 09:10:18,693 INFO o.t.m.s.bwhist.Main:60 Terminating bwhist module. 2021-11-23 09:10:18,694 INFO o.t.m.s.main.Main:73 Completed org.torproject.metrics.stats.bwhist.Main module. 2021-11-23 09:10:18,695 INFO o.t.m.s.main.Main:70 Starting org.torproject.metrics.stats.advbwdist.Main module. 2021-11-23 09:11:13,855 INFO o.t.m.s.main.Main:73 Completed org.torproject.metrics.stats.advbwdist.Main module. 2021-11-23 09:11:14,646 INFO o.t.m.s.main.Main:70 Starting org.torproject.metrics.stats.hidserv.Main module. 2021-11-23 09:11:14,647 INFO o.t.m.s.hidserv.Main:37 Initializing parser and reading parse history... 2021-11-23 09:11:14,748 INFO o.t.m.s.hidserv.Main:52 Parsing descriptors... 2021-11-23 09:12:56,964 INFO o.t.m.s.hidserv.Main:59 Writing parse history... 2021-11-23 09:12:57,087 INFO o.t.m.s.hidserv.Main:66 Extrapolating statistics... 2021-11-23 09:17:46,860 INFO o.t.m.s.hidserv.Main:82 Aggregating v2 statistics... 2021-11-23 09:19:05,011 INFO o.t.m.s.hidserv.Main:98 Aggregating v3 statistics... 2021-11-23 09:19:05,243 INFO o.t.m.s.hidserv.Main:108 Terminating. 2021-11-23 09:19:05,244 INFO o.t.m.s.main.Main:73 Completed org.torproject.metrics.stats.hidserv.Main module. 2021-11-23 09:19:05,244 INFO o.t.m.s.main.Main:70 Starting org.torproject.metrics.stats.clients.Main module. 2021-11-23 09:19:05,252 INFO o.t.m.s.clients.Main:36 Starting clients module. 2021-11-23 09:19:05,253 INFO o.t.m.s.clients.Main:38 Connecting to database. 2021-11-23 09:19:05,360 INFO o.t.m.s.clients.Main:41 Reading relay descriptors and importing relevant parts into the database. 2021-11-23 09:20:21,921 INFO o.t.m.s.clients.Main:45 Reading bridge descriptors and importing relevant parts into the database. 2021-11-23 09:20:58,281 INFO o.t.m.s.clients.Main:49 Processing newly imported data. 2021-11-23 10:35:39,883 INFO o.t.m.s.clients.Main:53 Querying aggregated statistics from the database. 2021-11-23 10:36:20,503 INFO o.t.m.s.clients.Main:59 Disconnecting from database. 2021-11-23 10:36:20,640 INFO o.t.m.s.clients.Main:62 Running detector. 2021-11-23 10:36:55,136 INFO o.t.m.s.clients.Main:65 Terminating clients module. 2021-11-23 10:36:55,137 INFO o.t.m.s.main.Main:73 Completed org.torproject.metrics.stats.clients.Main module. 2021-11-23 10:36:55,137 INFO o.t.m.s.main.Main:70 Starting org.torproject.metrics.stats.servers.Main module. 2021-11-23 10:36:55,138 INFO o.t.m.s.servers.Main:43 Starting servers module. 2021-11-23 10:36:55,138 INFO o.t.m.s.servers.Main:45 Reading descriptors and inserting relevant parts into the database. 2021-11-23 10:51:10,827 INFO o.t.m.s.servers.Main:77 Aggregating database entries. 2021-11-23 11:12:04,215 INFO o.t.m.s.servers.Main:80 Committing all updated parts in the database. 2021-11-23 11:12:04,568 INFO o.t.m.s.servers.Main:90 Querying aggregated statistics from the database. 2021-11-23 11:29:00,592 INFO o.t.m.s.servers.Main:105 Terminating servers module. 2021-11-23 11:29:00,724 INFO o.t.m.s.main.Main:73 Completed org.torproject.metrics.stats.servers.Main module. 2021-11-23 11:29:00,725 INFO o.t.m.s.main.Main:70 Starting org.torproject.metrics.stats.webstats.Main module. 2021-11-23 11:29:00,861 INFO o.t.m.s.w.Main:75 Starting webstats module. 2021-11-23 11:29:00,861 INFO o.t.m.s.w.Main:92 Connecting to database. 2021-11-23 11:29:01,233 INFO o.t.m.s.w.Main:95 Successfully connected to database. 2021-11-23 11:29:01,233 INFO o.t.m.s.w.Main:101 Querying previously imported log files. 2021-11-23 11:29:04,249 INFO o.t.m.s.w.Main:113 Found 285032 previously imported log files. 2021-11-23 11:38:33,425 INFO o.t.m.s.w.Main:268 Querying statistics from database. 2021-11-23 12:00:25,371 INFO o.t.m.s.w.Main:298 Writing 1418558 lines to /srv/metrics.torproject.org/metrics/work/modules/webstats/stats/webstats.csv. 2021-11-23 12:00:26,129 INFO o.t.m.s.w.Main:305 Disconnecting from database. 2021-11-23 12:00:26,129 INFO o.t.m.s.w.Main:87 Terminated webstats module. 2021-11-23 12:00:26,130 INFO o.t.m.s.main.Main:73 Completed org.torproject.metrics.stats.webstats.Main module. 2021-11-23 12:00:26,130 INFO o.t.m.s.main.Main:70 Starting org.torproject.metrics.stats.totalcw.Main module. 2021-11-23 12:00:26,131 INFO o.t.m.s.totalcw.Main:38 Starting totalcw module. 2021-11-23 12:00:26,131 INFO o.t.m.s.totalcw.Main:40 Reading consensuses and votes and inserting relevant parts into the database. 2021-11-23 12:02:41,478 INFO o.t.m.s.totalcw.Main:64 Committing all updated parts in the database. 2021-11-23 12:02:41,658 INFO o.t.m.s.totalcw.Main:74 Querying aggregated statistics from the database. 2021-11-23 12:02:46,276 INFO o.t.m.s.totalcw.Main:77 Writing aggregated statistics to /srv/metrics.torproject.org/metrics/work/modules/totalcw/stats/totalcw.csv. 2021-11-23 12:02:46,530 INFO o.t.m.s.totalcw.Main:82 Terminating totalcw module. 2021-11-23 12:02:46,530 INFO o.t.m.s.main.Main:73 Completed org.torproject.metrics.stats.totalcw.Main module. 2021-11-23 12:02:46,530 INFO o.t.m.s.main.Main:70 Starting org.torproject.metrics.stats.bridgedb.Main module. 2021-11-23 12:02:46,716 INFO o.t.m.s.main.Main:73 Completed org.torproject.metrics.stats.bridgedb.Main module. 2021-11-23 12:02:46,716 INFO o.t.m.s.main.Main:82 Making module data available. 2021-11-23 12:02:51,211 INFO o.t.m.s.main.Main:126 Successfully copied 22 files to stats output directory: [connbidirect2.csv, onionperf-failures.csv, buildtimes.csv, latencies.csv, torperf-1.1.csv, onionperf-throughput.csv, onionperf-including-partials.csv, bandwidth.csv, advbwdist.csv, v3hidserv.csv, hidserv.csv, clients.csv, userstats-combined.csv, platforms.csv, relayflags.csv, versions.csv, advbw.csv, networksize.csv, ipv6servers.csv, webstats.csv, totalcw.csv, bridgedb-stats.csv] 2021-11-23 12:02:51,211 INFO o.t.m.s.main.Main:130 Completed metrics update run.

1 0

Cron <collector@colchicifolium> cd /srv/collector.torproject.org/collector/logs/ && cat metrics.log | grep -E "WARN|ERROR|FATAL" | grep -v "Overwriting existing descriptor file"
by root＠colchicifolium.torproject.org 23 Nov '21

23 Nov '21

2021-11-23 00:07:34,702 WARN o.t.m.c.r.ReferenceChecker:314 Missing too many referenced descriptors (29.118167462542278). 2021-11-23 00:37:25,309 WARN o.t.m.c.r.ReferenceChecker:314 Missing too many referenced descriptors (20.657715239550473). 2021-11-23 01:07:07,539 WARN o.t.m.c.r.ReferenceChecker:314 Missing too many referenced descriptors (22.320650794880187). 2021-11-23 01:37:01,839 WARN o.t.m.c.r.ReferenceChecker:314 Missing too many referenced descriptors (21.40616260590386). 2021-11-23 02:07:37,315 WARN o.t.m.c.r.ReferenceChecker:314 Missing too many referenced descriptors (27.587079554426996). 2021-11-23 02:37:02,098 WARN o.t.m.c.r.ReferenceChecker:314 Missing too many referenced descriptors (12.44337105447559). 2021-11-23 03:07:18,918 WARN o.t.m.c.r.ReferenceChecker:314 Missing too many referenced descriptors (24.08873833936836). 2021-11-23 03:37:30,176 WARN o.t.m.c.r.ReferenceChecker:314 Missing too many referenced descriptors (20.59504687503056). 2021-11-23 04:13:05,557 WARN o.t.m.c.r.ReferenceChecker:314 Missing too many referenced descriptors (23.154612396904263). 2021-11-23 04:37:44,465 WARN o.t.m.c.r.ReferenceChecker:314 Missing too many referenced descriptors (22.253456313968897). 2021-11-23 05:08:20,817 WARN o.t.m.c.r.ReferenceChecker:314 Missing too many referenced descriptors (28.92386645362239). 2021-11-23 05:38:37,624 WARN o.t.m.c.r.ReferenceChecker:314 Missing too many referenced descriptors (18.147977277159164). 2021-11-23 06:07:44,322 WARN o.t.m.c.r.ReferenceChecker:314 Missing too many referenced descriptors (26.684766755741634). 2021-11-23 06:38:59,773 WARN o.t.m.c.r.ReferenceChecker:314 Missing too many referenced descriptors (20.206728787382094). 2021-11-23 07:09:07,790 WARN o.t.m.c.r.ReferenceChecker:314 Missing too many referenced descriptors (19.646452442151844). 2021-11-23 07:38:35,737 WARN o.t.m.c.r.ReferenceChecker:314 Missing too many referenced descriptors (18.620587730464976). 2021-11-23 08:09:04,410 WARN o.t.m.c.r.ReferenceChecker:314 Missing too many referenced descriptors (27.25345350158418). 2021-11-23 08:38:10,617 WARN o.t.m.c.r.ReferenceChecker:314 Missing too many referenced descriptors (26.692008975943416). 2021-11-23 09:08:57,687 WARN o.t.m.c.r.ReferenceChecker:314 Missing too many referenced descriptors (35.11732319837098). 2021-11-23 09:38:15,685 WARN o.t.m.c.r.ReferenceChecker:314 Missing too many referenced descriptors (14.886569191297715). 2021-11-23 10:07:43,682 WARN o.t.m.c.r.ReferenceChecker:314 Missing too many referenced descriptors (19.610205365267184). 2021-11-23 10:38:37,718 WARN o.t.m.c.r.ReferenceChecker:314 Missing too many referenced descriptors (11.071846807373833). 2021-11-23 11:10:04,754 WARN o.t.m.c.r.ReferenceChecker:314 Missing too many referenced descriptors (14.753283926698648). 2021-11-23 11:37:39,419 WARN o.t.m.c.r.ReferenceChecker:314 Missing too many referenced descriptors (7.907888613400152). 2021-11-23 12:08:10,186 WARN o.t.m.c.r.ReferenceChecker:314 Missing too many referenced descriptors (19.120123285725132). 2021-11-23 12:37:12,818 WARN o.t.m.c.r.ReferenceChecker:314 Missing too many referenced descriptors (12.315675431016105). 2021-11-23 13:07:24,822 WARN o.t.m.c.r.ReferenceChecker:314 Missing too many referenced descriptors (19.21286299926302). 2021-11-23 13:37:13,351 WARN o.t.m.c.r.ReferenceChecker:314 Missing too many referenced descriptors (18.75286521386893). 2021-11-23 14:07:21,893 WARN o.t.m.c.r.ReferenceChecker:314 Missing too many referenced descriptors (29.453642588376912). 2021-11-23 14:37:07,327 WARN o.t.m.c.r.ReferenceChecker:314 Missing too many referenced descriptors (29.123839390344923). 2021-11-23 15:07:16,887 WARN o.t.m.c.r.ReferenceChecker:314 Missing too many referenced descriptors (31.42862939181393). 2021-11-23 15:37:14,098 WARN o.t.m.c.r.ReferenceChecker:314 Missing too many referenced descriptors (29.957423979759856). 2021-11-23 16:07:07,620 WARN o.t.m.c.r.ReferenceChecker:314 Missing too many referenced descriptors (38.894536864619575). 2021-11-23 16:37:03,697 WARN o.t.m.c.r.ReferenceChecker:314 Missing too many referenced descriptors (38.41701717958027). 2021-11-23 17:07:25,341 WARN o.t.m.c.r.ReferenceChecker:314 Missing too many referenced descriptors (30.580734772350123). 2021-11-23 17:36:57,568 WARN o.t.m.c.r.ReferenceChecker:314 Missing too many referenced descriptors (29.8645262383488). 2021-11-23 18:07:22,896 WARN o.t.m.c.r.ReferenceChecker:314 Missing too many referenced descriptors (27.177385019931776).

1 0

Cron <collector@colchicifolium> cd /srv/collector.torproject.org/collector/logs/ && cat metrics.log | grep -E "WARN|ERROR|FATAL" | grep -v "Overwriting existing descriptor file"
by root＠colchicifolium.torproject.org 23 Nov '21

23 Nov '21

2021-11-23 00:07:34,702 WARN o.t.m.c.r.ReferenceChecker:314 Missing too many referenced descriptors (29.118167462542278). 2021-11-23 00:37:25,309 WARN o.t.m.c.r.ReferenceChecker:314 Missing too many referenced descriptors (20.657715239550473). 2021-11-23 01:07:07,539 WARN o.t.m.c.r.ReferenceChecker:314 Missing too many referenced descriptors (22.320650794880187). 2021-11-23 01:37:01,839 WARN o.t.m.c.r.ReferenceChecker:314 Missing too many referenced descriptors (21.40616260590386). 2021-11-23 02:07:37,315 WARN o.t.m.c.r.ReferenceChecker:314 Missing too many referenced descriptors (27.587079554426996). 2021-11-23 02:37:02,098 WARN o.t.m.c.r.ReferenceChecker:314 Missing too many referenced descriptors (12.44337105447559). 2021-11-23 03:07:18,918 WARN o.t.m.c.r.ReferenceChecker:314 Missing too many referenced descriptors (24.08873833936836). 2021-11-23 03:37:30,176 WARN o.t.m.c.r.ReferenceChecker:314 Missing too many referenced descriptors (20.59504687503056). 2021-11-23 04:13:05,557 WARN o.t.m.c.r.ReferenceChecker:314 Missing too many referenced descriptors (23.154612396904263). 2021-11-23 04:37:44,465 WARN o.t.m.c.r.ReferenceChecker:314 Missing too many referenced descriptors (22.253456313968897). 2021-11-23 05:08:20,817 WARN o.t.m.c.r.ReferenceChecker:314 Missing too many referenced descriptors (28.92386645362239). 2021-11-23 05:38:37,624 WARN o.t.m.c.r.ReferenceChecker:314 Missing too many referenced descriptors (18.147977277159164). 2021-11-23 06:07:44,322 WARN o.t.m.c.r.ReferenceChecker:314 Missing too many referenced descriptors (26.684766755741634). 2021-11-23 06:38:59,773 WARN o.t.m.c.r.ReferenceChecker:314 Missing too many referenced descriptors (20.206728787382094). 2021-11-23 07:09:07,790 WARN o.t.m.c.r.ReferenceChecker:314 Missing too many referenced descriptors (19.646452442151844). 2021-11-23 07:38:35,737 WARN o.t.m.c.r.ReferenceChecker:314 Missing too many referenced descriptors (18.620587730464976). 2021-11-23 08:09:04,410 WARN o.t.m.c.r.ReferenceChecker:314 Missing too many referenced descriptors (27.25345350158418). 2021-11-23 08:38:10,617 WARN o.t.m.c.r.ReferenceChecker:314 Missing too many referenced descriptors (26.692008975943416). 2021-11-23 09:08:57,687 WARN o.t.m.c.r.ReferenceChecker:314 Missing too many referenced descriptors (35.11732319837098). 2021-11-23 09:38:15,685 WARN o.t.m.c.r.ReferenceChecker:314 Missing too many referenced descriptors (14.886569191297715). 2021-11-23 10:07:43,682 WARN o.t.m.c.r.ReferenceChecker:314 Missing too many referenced descriptors (19.610205365267184). 2021-11-23 10:38:37,718 WARN o.t.m.c.r.ReferenceChecker:314 Missing too many referenced descriptors (11.071846807373833). 2021-11-23 11:10:04,754 WARN o.t.m.c.r.ReferenceChecker:314 Missing too many referenced descriptors (14.753283926698648). 2021-11-23 11:37:39,419 WARN o.t.m.c.r.ReferenceChecker:314 Missing too many referenced descriptors (7.907888613400152). 2021-11-23 12:08:10,186 WARN o.t.m.c.r.ReferenceChecker:314 Missing too many referenced descriptors (19.120123285725132).

1 0

Cron <metrics@meronense> cd /srv/metrics.torproject.org/metrics/work/modules/logs && cat metrics.log | grep -v "Could not insert bandwidth "
by root＠meronense.torproject.org 23 Nov '21

23 Nov '21

2021-11-23 08:40:03,201 INFO o.t.m.s.main.Main:32 Starting metrics update run. 2021-11-23 08:40:03,669 INFO o.t.m.s.main.Main:70 Starting org.torproject.metrics.stats.collectdescs.Main module. 2021-11-23 08:40:03,921 INFO o.t.d.i.DescriptorIndexCollector:47 Starting descriptor collection. 2021-11-23 08:40:04,146 INFO o.t.d.i.DescriptorIndexCollector:63 Indexing local directory /srv/metrics.torproject.org/metrics/work/shared/in. 2021-11-23 08:40:08,889 INFO o.t.d.i.DescriptorIndexCollector:75 Fetching remote index file https://collector.torproject.org/index/index.json. 2021-11-23 08:40:12,934 INFO o.t.d.i.DescriptorIndexCollector:84 Fetching remote files from https://collector.torproject.org. 2021-11-23 08:40:25,247 WARN o.t.d.i.DescriptorIndexCollector:132 Fetched remote file 2021-11-21-04-00-00-vote-0232AF901C31A04EE9848595AF9BB7620D4C5B2E-9DAFAF1CE0ABC369CCD86FB1D7E38C33CCFAB1D6 from https://collector.torproject.org has a size of 4669122 bytes which is different from the expected 1273856 bytes. Not storing this file. 2021-11-23 08:40:54,891 INFO o.t.d.i.DescriptorIndexCollector:90 Deleting extraneous files from local directory /srv/metrics.torproject.org/metrics/work/shared/in. 2021-11-23 08:40:57,279 INFO o.t.d.i.DescriptorIndexCollector:95 Finished descriptor collection. 2021-11-23 08:40:57,279 INFO o.t.m.s.main.Main:73 Completed org.torproject.metrics.stats.collectdescs.Main module. 2021-11-23 08:40:57,279 INFO o.t.m.s.main.Main:70 Starting org.torproject.metrics.stats.connbidirect.Main module. 2021-11-23 08:41:10,677 INFO o.t.m.s.main.Main:73 Completed org.torproject.metrics.stats.connbidirect.Main module. 2021-11-23 08:41:10,678 INFO o.t.m.s.main.Main:70 Starting org.torproject.metrics.stats.onionperf.Main module. 2021-11-23 08:41:10,679 INFO o.t.m.s.o.Main:46 Starting onionperf module. 2021-11-23 08:41:10,680 INFO o.t.m.s.o.Main:67 Connecting to database. 2021-11-23 08:41:14,668 INFO o.t.m.s.o.Main:70 Successfully connected to database. 2021-11-23 08:52:04,203 INFO o.t.m.s.o.Main:261 Querying statistics including partials from database. 2021-11-23 08:52:53,028 INFO o.t.m.s.o.Main:402 Writing 59119 lines to /srv/metrics.torproject.org/metrics/work/modules/onionperf/stats/onionperf-including-partials.csv. 2021-11-23 08:52:53,117 INFO o.t.m.s.o.Main:286 Querying timeout and failure statistics from database. 2021-11-23 08:53:03,018 INFO o.t.m.s.o.Main:402 Writing 19711 lines to /srv/metrics.torproject.org/metrics/work/modules/onionperf/stats/onionperf-failures.csv. 2021-11-23 08:53:03,022 INFO o.t.m.s.o.Main:309 Querying buildtime statistics from database. 2021-11-23 08:53:43,329 INFO o.t.m.s.o.Main:402 Writing 47692 lines to /srv/metrics.torproject.org/metrics/work/modules/onionperf/stats/buildtimes.csv. 2021-11-23 08:53:43,558 INFO o.t.m.s.o.Main:332 Querying latency statistics from database. 2021-11-23 08:53:55,146 INFO o.t.m.s.o.Main:402 Writing 18744 lines to /srv/metrics.torproject.org/metrics/work/modules/onionperf/stats/latencies.csv. 2021-11-23 08:53:55,151 INFO o.t.m.s.o.Main:357 Querying throughput statistics from database. 2021-11-23 08:54:06,500 INFO o.t.m.s.o.Main:402 Writing 19711 lines to /srv/metrics.torproject.org/metrics/work/modules/onionperf/stats/onionperf-throughput.csv. 2021-11-23 08:54:06,582 INFO o.t.m.s.o.Main:409 Disconnecting from database. 2021-11-23 08:54:06,583 INFO o.t.m.s.o.Main:62 Terminated onionperf module. 2021-11-23 08:54:06,583 INFO o.t.m.s.main.Main:73 Completed org.torproject.metrics.stats.onionperf.Main module. 2021-11-23 08:54:06,583 INFO o.t.m.s.main.Main:70 Starting org.torproject.metrics.stats.bwhist.Main module. 2021-11-23 08:54:06,583 INFO o.t.m.s.bwhist.Main:37 Starting bwhist module. 2021-11-23 08:54:06,583 INFO o.t.m.s.bwhist.Main:39 Reading descriptors and inserting relevant parts into the database. 2021-11-23 09:09:03,588 INFO o.t.m.s.b.RelayDescriptorDatabaseImporter:543 Finished importing relay descriptors: 94448 network status entries and 83665 bandwidth history elements 2021-11-23 09:09:05,237 INFO o.t.m.s.bwhist.Main:50 Aggregating database entries. 2021-11-23 09:10:14,357 INFO o.t.m.s.b.RelayDescriptorDatabaseImporter:543 Finished importing relay descriptors: 94448 network status entries and 83665 bandwidth history elements 2021-11-23 09:10:16,740 INFO o.t.m.s.bwhist.Main:53 Querying aggregated statistics from the database. 2021-11-23 09:10:18,688 INFO o.t.m.s.bwhist.Main:57 Closing database connection. 2021-11-23 09:10:18,693 INFO o.t.m.s.bwhist.Main:60 Terminating bwhist module. 2021-11-23 09:10:18,694 INFO o.t.m.s.main.Main:73 Completed org.torproject.metrics.stats.bwhist.Main module. 2021-11-23 09:10:18,695 INFO o.t.m.s.main.Main:70 Starting org.torproject.metrics.stats.advbwdist.Main module. 2021-11-23 09:11:13,855 INFO o.t.m.s.main.Main:73 Completed org.torproject.metrics.stats.advbwdist.Main module. 2021-11-23 09:11:14,646 INFO o.t.m.s.main.Main:70 Starting org.torproject.metrics.stats.hidserv.Main module. 2021-11-23 09:11:14,647 INFO o.t.m.s.hidserv.Main:37 Initializing parser and reading parse history... 2021-11-23 09:11:14,748 INFO o.t.m.s.hidserv.Main:52 Parsing descriptors... 2021-11-23 09:12:56,964 INFO o.t.m.s.hidserv.Main:59 Writing parse history... 2021-11-23 09:12:57,087 INFO o.t.m.s.hidserv.Main:66 Extrapolating statistics... 2021-11-23 09:17:46,860 INFO o.t.m.s.hidserv.Main:82 Aggregating v2 statistics... 2021-11-23 09:19:05,011 INFO o.t.m.s.hidserv.Main:98 Aggregating v3 statistics... 2021-11-23 09:19:05,243 INFO o.t.m.s.hidserv.Main:108 Terminating. 2021-11-23 09:19:05,244 INFO o.t.m.s.main.Main:73 Completed org.torproject.metrics.stats.hidserv.Main module. 2021-11-23 09:19:05,244 INFO o.t.m.s.main.Main:70 Starting org.torproject.metrics.stats.clients.Main module. 2021-11-23 09:19:05,252 INFO o.t.m.s.clients.Main:36 Starting clients module. 2021-11-23 09:19:05,253 INFO o.t.m.s.clients.Main:38 Connecting to database. 2021-11-23 09:19:05,360 INFO o.t.m.s.clients.Main:41 Reading relay descriptors and importing relevant parts into the database. 2021-11-23 09:20:21,921 INFO o.t.m.s.clients.Main:45 Reading bridge descriptors and importing relevant parts into the database. 2021-11-23 09:20:58,281 INFO o.t.m.s.clients.Main:49 Processing newly imported data. 2021-11-23 10:35:39,883 INFO o.t.m.s.clients.Main:53 Querying aggregated statistics from the database. 2021-11-23 10:36:20,503 INFO o.t.m.s.clients.Main:59 Disconnecting from database. 2021-11-23 10:36:20,640 INFO o.t.m.s.clients.Main:62 Running detector. 2021-11-23 10:36:55,136 INFO o.t.m.s.clients.Main:65 Terminating clients module. 2021-11-23 10:36:55,137 INFO o.t.m.s.main.Main:73 Completed org.torproject.metrics.stats.clients.Main module. 2021-11-23 10:36:55,137 INFO o.t.m.s.main.Main:70 Starting org.torproject.metrics.stats.servers.Main module. 2021-11-23 10:36:55,138 INFO o.t.m.s.servers.Main:43 Starting servers module. 2021-11-23 10:36:55,138 INFO o.t.m.s.servers.Main:45 Reading descriptors and inserting relevant parts into the database. 2021-11-23 10:51:10,827 INFO o.t.m.s.servers.Main:77 Aggregating database entries. 2021-11-23 11:12:04,215 INFO o.t.m.s.servers.Main:80 Committing all updated parts in the database. 2021-11-23 11:12:04,568 INFO o.t.m.s.servers.Main:90 Querying aggregated statistics from the database. 2021-11-23 11:29:00,592 INFO o.t.m.s.servers.Main:105 Terminating servers module. 2021-11-23 11:29:00,724 INFO o.t.m.s.main.Main:73 Completed org.torproject.metrics.stats.servers.Main module. 2021-11-23 11:29:00,725 INFO o.t.m.s.main.Main:70 Starting org.torproject.metrics.stats.webstats.Main module. 2021-11-23 11:29:00,861 INFO o.t.m.s.w.Main:75 Starting webstats module. 2021-11-23 11:29:00,861 INFO o.t.m.s.w.Main:92 Connecting to database. 2021-11-23 11:29:01,233 INFO o.t.m.s.w.Main:95 Successfully connected to database. 2021-11-23 11:29:01,233 INFO o.t.m.s.w.Main:101 Querying previously imported log files. 2021-11-23 11:29:04,249 INFO o.t.m.s.w.Main:113 Found 285032 previously imported log files. 2021-11-23 11:38:33,425 INFO o.t.m.s.w.Main:268 Querying statistics from database. 2021-11-23 12:00:25,371 INFO o.t.m.s.w.Main:298 Writing 1418558 lines to /srv/metrics.torproject.org/metrics/work/modules/webstats/stats/webstats.csv. 2021-11-23 12:00:26,129 INFO o.t.m.s.w.Main:305 Disconnecting from database. 2021-11-23 12:00:26,129 INFO o.t.m.s.w.Main:87 Terminated webstats module. 2021-11-23 12:00:26,130 INFO o.t.m.s.main.Main:73 Completed org.torproject.metrics.stats.webstats.Main module. 2021-11-23 12:00:26,130 INFO o.t.m.s.main.Main:70 Starting org.torproject.metrics.stats.totalcw.Main module. 2021-11-23 12:00:26,131 INFO o.t.m.s.totalcw.Main:38 Starting totalcw module. 2021-11-23 12:00:26,131 INFO o.t.m.s.totalcw.Main:40 Reading consensuses and votes and inserting relevant parts into the database. 2021-11-23 12:02:41,478 INFO o.t.m.s.totalcw.Main:64 Committing all updated parts in the database. 2021-11-23 12:02:41,658 INFO o.t.m.s.totalcw.Main:74 Querying aggregated statistics from the database. 2021-11-23 12:02:46,276 INFO o.t.m.s.totalcw.Main:77 Writing aggregated statistics to /srv/metrics.torproject.org/metrics/work/modules/totalcw/stats/totalcw.csv. 2021-11-23 12:02:46,530 INFO o.t.m.s.totalcw.Main:82 Terminating totalcw module. 2021-11-23 12:02:46,530 INFO o.t.m.s.main.Main:73 Completed org.torproject.metrics.stats.totalcw.Main module. 2021-11-23 12:02:46,530 INFO o.t.m.s.main.Main:70 Starting org.torproject.metrics.stats.bridgedb.Main module. 2021-11-23 12:02:46,716 INFO o.t.m.s.main.Main:73 Completed org.torproject.metrics.stats.bridgedb.Main module. 2021-11-23 12:02:46,716 INFO o.t.m.s.main.Main:82 Making module data available. 2021-11-23 12:02:51,211 INFO o.t.m.s.main.Main:126 Successfully copied 22 files to stats output directory: [connbidirect2.csv, onionperf-failures.csv, buildtimes.csv, latencies.csv, torperf-1.1.csv, onionperf-throughput.csv, onionperf-including-partials.csv, bandwidth.csv, advbwdist.csv, v3hidserv.csv, hidserv.csv, clients.csv, userstats-combined.csv, platforms.csv, relayflags.csv, versions.csv, advbw.csv, networksize.csv, ipv6servers.csv, webstats.csv, totalcw.csv, bridgedb-stats.csv] 2021-11-23 12:02:51,211 INFO o.t.m.s.main.Main:130 Completed metrics update run.

1 0

Cron <metrics@meronense> cd /srv/metrics.torproject.org/metrics/ && ./run-web.sh
by root＠meronense.torproject.org 23 Nov '21

23 Nov '21

2021-11-23 08:40:25,247 WARN o.t.d.i.DescriptorIndexCollector:132 Fetched remote file 2021-11-21-04-00-00-vote-0232AF901C31A04EE9848595AF9BB7620D4C5B2E-9DAFAF1CE0ABC369CCD86FB1D7E38C33CCFAB1D6 from https://collector.torproject.org has a size of 4669122 bytes which is different from the expected 1273856 bytes. Not storing this file.

1 0

Cron <collector@colchicifolium> cd /srv/collector.torproject.org/collector/ && src/main/resources/create-tarballs.sh
by root＠colchicifolium.torproject.org 23 Nov '21

23 Nov '21

Tue Nov 23 03:25:01 UTC 2021 Starting Tue Nov 23 03:25:01 UTC 2021 Touching symlink and directories for exit-list-2021-11 Tue Nov 23 03:25:02 UTC 2021 Creating symlink for torperf-2021-11 ln: failed to create symbolic link 'torperf-2021-11': File exists Tue Nov 23 03:25:02 UTC 2021 Touching symlink and directories for onionperf-2021-11 Tue Nov 23 03:25:02 UTC 2021 Touching symlink and directories for certs Tue Nov 23 03:25:02 UTC 2021 Touching symlink and directories for microdescs-2021-11 Tue Nov 23 03:25:02 UTC 2021 Touching symlink and directories for consensuses-2021-11 Tue Nov 23 03:25:02 UTC 2021 Touching symlink and directories for votes-2021-11 Tue Nov 23 03:25:02 UTC 2021 Touching symlink and directories for bandwidths-2021-11 Tue Nov 23 03:25:02 UTC 2021 Touching symlink and directories for server-descriptors-2021-11 Tue Nov 23 03:25:04 UTC 2021 Touching symlink and directories for extra-infos-2021-11 Tue Nov 23 03:25:06 UTC 2021 Touching symlink and directories for bridge-statuses-2021-11 Tue Nov 23 03:25:06 UTC 2021 Touching symlink and directories for bridge-server-descriptors-2021-11 Tue Nov 23 03:25:06 UTC 2021 Touching symlink and directories for bridge-extra-infos-2021-11 Tue Nov 23 03:25:07 UTC 2021 Touching symlink and directories for snowflakes-2021-11 Tue Nov 23 03:25:07 UTC 2021 Touching symlink and directories for bridge-pool-assignments-2021-11 Tue Nov 23 03:25:07 UTC 2021 Touching symlink and directories for bridgedb-metrics-2021-11 Tue Nov 23 03:25:07 UTC 2021 Touching symlink and directories for bridgestrap-2021-11 Tue Nov 23 03:25:07 UTC 2021 Creating exit-list-2021-11.tar Tue Nov 23 03:25:16 UTC 2021 Compressing exit-list-2021-11.tar Tue Nov 23 03:26:26 UTC 2021 Creating torperf-2021-11.tar tar: torperf-2021-11: Cannot stat: No such file or directory tar: Exiting with failure status due to previous errors Tue Nov 23 03:26:26 UTC 2021 Compressing torperf-2021-11.tar Tue Nov 23 03:26:26 UTC 2021 Creating onionperf-2021-11.tar Tue Nov 23 03:26:28 UTC 2021 Compressing onionperf-2021-11.tar Tue Nov 23 03:26:54 UTC 2021 Creating certs.tar Tue Nov 23 03:26:55 UTC 2021 Compressing certs.tar Tue Nov 23 03:26:55 UTC 2021 Creating microdescs-2021-11.tar Tue Nov 23 03:30:07 UTC 2021 Compressing microdescs-2021-11.tar Tue Nov 23 03:35:32 UTC 2021 Creating consensuses-2021-11.tar Tue Nov 23 03:36:38 UTC 2021 Compressing consensuses-2021-11.tar Tue Nov 23 03:42:41 UTC 2021 Creating votes-2021-11.tar Tue Nov 23 04:34:46 UTC 2021 Compressing votes-2021-11.tar Tue Nov 23 07:01:50 UTC 2021 Creating bandwidths-2021-11.tar Tue Nov 23 07:26:49 UTC 2021 Compressing bandwidths-2021-11.tar Tue Nov 23 08:51:17 UTC 2021 Creating server-descriptors-2021-11.tar tar: server-descriptors-2021-11/5/f: file changed as we read it Tue Nov 23 09:26:38 UTC 2021 Compressing server-descriptors-2021-11.tar Tue Nov 23 09:39:30 UTC 2021 Creating extra-infos-2021-11.tar Tue Nov 23 10:02:02 UTC 2021 Compressing extra-infos-2021-11.tar Tue Nov 23 10:16:40 UTC 2021 Creating bridge-statuses-2021-11.tar Tue Nov 23 10:16:56 UTC 2021 Compressing bridge-statuses-2021-11.tar Tue Nov 23 10:18:09 UTC 2021 Creating bridge-server-descriptors-2021-11.tar Tue Nov 23 10:29:16 UTC 2021 Compressing bridge-server-descriptors-2021-11.tar Tue Nov 23 10:33:28 UTC 2021 Creating bridge-extra-infos-2021-11.tar Tue Nov 23 10:47:41 UTC 2021 Compressing bridge-extra-infos-2021-11.tar Tue Nov 23 10:55:23 UTC 2021 Creating snowflakes-2021-11.tar Tue Nov 23 10:55:23 UTC 2021 Compressing snowflakes-2021-11.tar Tue Nov 23 10:55:24 UTC 2021 Creating bridge-pool-assignments-2021-11.tar Tue Nov 23 10:55:39 UTC 2021 Compressing bridge-pool-assignments-2021-11.tar Tue Nov 23 10:56:08 UTC 2021 Creating bridgedb-metrics-2021-11.tar Tue Nov 23 10:56:08 UTC 2021 Compressing bridgedb-metrics-2021-11.tar Tue Nov 23 10:56:08 UTC 2021 Creating bridgestrap-2021-11.tar Tue Nov 23 10:56:09 UTC 2021 Compressing bridgestrap-2021-11.tar Tue Nov 23 10:56:11 UTC 2021 Creating webstats-2021-11.tar Tue Nov 23 10:56:21 UTC 2021 Creating webstats-2021-11.tar Tue Nov 23 10:56:25 UTC 2021 Moving tarballs into place Tue Nov 23 10:56:25 UTC 2021 Finished tarball creation. Starting symlink-update ... Tue Nov 23 10:56:25 UTC 2021 Finished.

1 0

Cron <metrics@meronense> cd /srv/metrics.torproject.org/metrics/work/modules/logs && cat metrics.log | grep -v "Could not insert bandwidth "
by root＠meronense.torproject.org 23 Nov '21

23 Nov '21

2021-11-22 08:40:02,420 INFO o.t.m.s.main.Main:32 Starting metrics update run. 2021-11-22 08:40:02,473 INFO o.t.m.s.main.Main:70 Starting org.torproject.metrics.stats.collectdescs.Main module. 2021-11-22 08:40:02,476 INFO o.t.d.i.DescriptorIndexCollector:47 Starting descriptor collection. 2021-11-22 08:40:02,476 INFO o.t.d.i.DescriptorIndexCollector:63 Indexing local directory /srv/metrics.torproject.org/metrics/work/shared/in. 2021-11-22 08:40:06,386 INFO o.t.d.i.DescriptorIndexCollector:75 Fetching remote index file https://collector.torproject.org/index/index.json. 2021-11-22 08:40:07,128 INFO o.t.d.i.DescriptorIndexCollector:84 Fetching remote files from https://collector.torproject.org. 2021-11-22 08:40:10,899 WARN o.t.d.i.DescriptorIndexCollector:132 Fetched remote file 2021-11-21-04-00-00-vote-0232AF901C31A04EE9848595AF9BB7620D4C5B2E-9DAFAF1CE0ABC369CCD86FB1D7E38C33CCFAB1D6 from https://collector.torproject.org has a size of 4669122 bytes which is different from the expected 1273856 bytes. Not storing this file. 2021-11-22 08:40:28,727 INFO o.t.d.i.DescriptorIndexCollector:90 Deleting extraneous files from local directory /srv/metrics.torproject.org/metrics/work/shared/in. 2021-11-22 08:40:29,959 INFO o.t.d.i.DescriptorIndexCollector:95 Finished descriptor collection. 2021-11-22 08:40:29,959 INFO o.t.m.s.main.Main:73 Completed org.torproject.metrics.stats.collectdescs.Main module. 2021-11-22 08:40:29,959 INFO o.t.m.s.main.Main:70 Starting org.torproject.metrics.stats.connbidirect.Main module. 2021-11-22 08:40:37,280 INFO o.t.m.s.main.Main:73 Completed org.torproject.metrics.stats.connbidirect.Main module. 2021-11-22 08:40:37,280 INFO o.t.m.s.main.Main:70 Starting org.torproject.metrics.stats.onionperf.Main module. 2021-11-22 08:40:37,282 INFO o.t.m.s.o.Main:46 Starting onionperf module. 2021-11-22 08:40:37,282 INFO o.t.m.s.o.Main:67 Connecting to database. 2021-11-22 08:40:37,746 INFO o.t.m.s.o.Main:70 Successfully connected to database. 2021-11-22 08:45:26,703 INFO o.t.m.s.o.Main:261 Querying statistics including partials from database. 2021-11-22 08:46:12,639 INFO o.t.m.s.o.Main:402 Writing 59089 lines to /srv/metrics.torproject.org/metrics/work/modules/onionperf/stats/onionperf-including-partials.csv. 2021-11-22 08:46:13,521 INFO o.t.m.s.o.Main:286 Querying timeout and failure statistics from database. 2021-11-22 08:46:16,429 INFO o.t.m.s.o.Main:402 Writing 19701 lines to /srv/metrics.torproject.org/metrics/work/modules/onionperf/stats/onionperf-failures.csv. 2021-11-22 08:46:16,454 INFO o.t.m.s.o.Main:309 Querying buildtime statistics from database. 2021-11-22 08:46:58,997 INFO o.t.m.s.o.Main:402 Writing 47674 lines to /srv/metrics.torproject.org/metrics/work/modules/onionperf/stats/buildtimes.csv. 2021-11-22 08:46:59,798 INFO o.t.m.s.o.Main:332 Querying latency statistics from database. 2021-11-22 08:47:12,732 INFO o.t.m.s.o.Main:402 Writing 18734 lines to /srv/metrics.torproject.org/metrics/work/modules/onionperf/stats/latencies.csv. 2021-11-22 08:47:12,794 INFO o.t.m.s.o.Main:357 Querying throughput statistics from database. 2021-11-22 08:47:24,178 INFO o.t.m.s.o.Main:402 Writing 19701 lines to /srv/metrics.torproject.org/metrics/work/modules/onionperf/stats/onionperf-throughput.csv. 2021-11-22 08:47:24,180 INFO o.t.m.s.o.Main:409 Disconnecting from database. 2021-11-22 08:47:24,180 INFO o.t.m.s.o.Main:62 Terminated onionperf module. 2021-11-22 08:47:24,181 INFO o.t.m.s.main.Main:73 Completed org.torproject.metrics.stats.onionperf.Main module. 2021-11-22 08:47:24,181 INFO o.t.m.s.main.Main:70 Starting org.torproject.metrics.stats.bwhist.Main module. 2021-11-22 08:47:24,181 INFO o.t.m.s.bwhist.Main:37 Starting bwhist module. 2021-11-22 08:47:24,181 INFO o.t.m.s.bwhist.Main:39 Reading descriptors and inserting relevant parts into the database. 2021-11-22 08:57:09,946 INFO o.t.m.s.b.RelayDescriptorDatabaseImporter:543 Finished importing relay descriptors: 150891 network status entries and 137335 bandwidth history elements 2021-11-22 08:57:12,269 INFO o.t.m.s.bwhist.Main:50 Aggregating database entries. 2021-11-22 08:58:06,669 INFO o.t.m.s.b.RelayDescriptorDatabaseImporter:543 Finished importing relay descriptors: 150891 network status entries and 137335 bandwidth history elements 2021-11-22 08:58:08,270 INFO o.t.m.s.bwhist.Main:53 Querying aggregated statistics from the database. 2021-11-22 08:58:08,690 INFO o.t.m.s.bwhist.Main:57 Closing database connection. 2021-11-22 08:58:08,691 INFO o.t.m.s.bwhist.Main:60 Terminating bwhist module. 2021-11-22 08:58:08,691 INFO o.t.m.s.main.Main:73 Completed org.torproject.metrics.stats.bwhist.Main module. 2021-11-22 08:58:08,691 INFO o.t.m.s.main.Main:70 Starting org.torproject.metrics.stats.advbwdist.Main module. 2021-11-22 08:59:02,870 INFO o.t.m.s.main.Main:73 Completed org.torproject.metrics.stats.advbwdist.Main module. 2021-11-22 08:59:02,915 INFO o.t.m.s.main.Main:70 Starting org.torproject.metrics.stats.hidserv.Main module. 2021-11-22 08:59:02,916 INFO o.t.m.s.hidserv.Main:37 Initializing parser and reading parse history... 2021-11-22 08:59:03,068 INFO o.t.m.s.hidserv.Main:52 Parsing descriptors... 2021-11-22 09:00:47,073 INFO o.t.m.s.hidserv.Main:59 Writing parse history... 2021-11-22 09:00:47,122 INFO o.t.m.s.hidserv.Main:66 Extrapolating statistics... 2021-11-22 09:06:23,694 INFO o.t.m.s.hidserv.Main:82 Aggregating v2 statistics... 2021-11-22 09:08:02,717 INFO o.t.m.s.hidserv.Main:98 Aggregating v3 statistics... 2021-11-22 09:08:02,951 INFO o.t.m.s.hidserv.Main:108 Terminating. 2021-11-22 09:08:02,952 INFO o.t.m.s.main.Main:73 Completed org.torproject.metrics.stats.hidserv.Main module. 2021-11-22 09:08:02,954 INFO o.t.m.s.main.Main:70 Starting org.torproject.metrics.stats.clients.Main module. 2021-11-22 09:08:02,971 INFO o.t.m.s.clients.Main:36 Starting clients module. 2021-11-22 09:08:02,972 INFO o.t.m.s.clients.Main:38 Connecting to database. 2021-11-22 09:08:03,222 INFO o.t.m.s.clients.Main:41 Reading relay descriptors and importing relevant parts into the database. 2021-11-22 09:09:30,378 INFO o.t.m.s.clients.Main:45 Reading bridge descriptors and importing relevant parts into the database. 2021-11-22 09:10:07,372 INFO o.t.m.s.clients.Main:49 Processing newly imported data. 2021-11-22 10:19:27,740 INFO o.t.m.s.clients.Main:53 Querying aggregated statistics from the database. 2021-11-22 10:19:49,856 INFO o.t.m.s.clients.Main:59 Disconnecting from database. 2021-11-22 10:19:49,889 INFO o.t.m.s.clients.Main:62 Running detector. 2021-11-22 10:20:25,828 INFO o.t.m.s.clients.Main:65 Terminating clients module. 2021-11-22 10:20:25,846 INFO o.t.m.s.main.Main:73 Completed org.torproject.metrics.stats.clients.Main module. 2021-11-22 10:20:25,847 INFO o.t.m.s.main.Main:70 Starting org.torproject.metrics.stats.servers.Main module. 2021-11-22 10:20:25,865 INFO o.t.m.s.servers.Main:43 Starting servers module. 2021-11-22 10:20:25,865 INFO o.t.m.s.servers.Main:45 Reading descriptors and inserting relevant parts into the database. 2021-11-22 10:46:15,034 INFO o.t.m.s.servers.Main:77 Aggregating database entries. 2021-11-22 11:48:39,634 INFO o.t.m.s.servers.Main:80 Committing all updated parts in the database. 2021-11-22 11:48:40,099 INFO o.t.m.s.servers.Main:90 Querying aggregated statistics from the database. 2021-11-22 12:05:12,459 INFO o.t.m.s.servers.Main:105 Terminating servers module. 2021-11-22 12:05:12,666 INFO o.t.m.s.main.Main:73 Completed org.torproject.metrics.stats.servers.Main module. 2021-11-22 12:05:12,667 INFO o.t.m.s.main.Main:70 Starting org.torproject.metrics.stats.webstats.Main module. 2021-11-22 12:05:12,708 INFO o.t.m.s.w.Main:75 Starting webstats module. 2021-11-22 12:05:12,708 INFO o.t.m.s.w.Main:92 Connecting to database. 2021-11-22 12:05:13,082 INFO o.t.m.s.w.Main:95 Successfully connected to database. 2021-11-22 12:05:13,083 INFO o.t.m.s.w.Main:101 Querying previously imported log files. 2021-11-22 12:05:15,131 INFO o.t.m.s.w.Main:113 Found 284694 previously imported log files. 2021-11-22 12:25:52,295 INFO o.t.m.s.w.Main:268 Querying statistics from database. 2021-11-22 12:47:57,595 INFO o.t.m.s.w.Main:298 Writing 1417810 lines to /srv/metrics.torproject.org/metrics/work/modules/webstats/stats/webstats.csv. 2021-11-22 12:47:58,408 INFO o.t.m.s.w.Main:305 Disconnecting from database. 2021-11-22 12:47:58,410 INFO o.t.m.s.w.Main:87 Terminated webstats module. 2021-11-22 12:47:58,410 INFO o.t.m.s.main.Main:73 Completed org.torproject.metrics.stats.webstats.Main module. 2021-11-22 12:47:58,410 INFO o.t.m.s.main.Main:70 Starting org.torproject.metrics.stats.totalcw.Main module. 2021-11-22 12:47:58,412 INFO o.t.m.s.totalcw.Main:38 Starting totalcw module. 2021-11-22 12:47:58,412 INFO o.t.m.s.totalcw.Main:40 Reading consensuses and votes and inserting relevant parts into the database. 2021-11-22 12:51:28,183 INFO o.t.m.s.totalcw.Main:64 Committing all updated parts in the database. 2021-11-22 12:51:28,246 INFO o.t.m.s.totalcw.Main:74 Querying aggregated statistics from the database. 2021-11-22 12:51:38,353 INFO o.t.m.s.totalcw.Main:77 Writing aggregated statistics to /srv/metrics.torproject.org/metrics/work/modules/totalcw/stats/totalcw.csv. 2021-11-22 12:51:38,553 INFO o.t.m.s.totalcw.Main:82 Terminating totalcw module. 2021-11-22 12:51:38,554 INFO o.t.m.s.main.Main:73 Completed org.torproject.metrics.stats.totalcw.Main module. 2021-11-22 12:51:38,554 INFO o.t.m.s.main.Main:70 Starting org.torproject.metrics.stats.bridgedb.Main module. 2021-11-22 12:51:38,977 INFO o.t.m.s.main.Main:73 Completed org.torproject.metrics.stats.bridgedb.Main module. 2021-11-22 12:51:38,977 INFO o.t.m.s.main.Main:82 Making module data available. 2021-11-22 12:51:47,361 INFO o.t.m.s.main.Main:126 Successfully copied 22 files to stats output directory: [connbidirect2.csv, onionperf-failures.csv, buildtimes.csv, latencies.csv, torperf-1.1.csv, onionperf-throughput.csv, onionperf-including-partials.csv, bandwidth.csv, advbwdist.csv, v3hidserv.csv, hidserv.csv, clients.csv, userstats-combined.csv, platforms.csv, relayflags.csv, versions.csv, advbw.csv, networksize.csv, ipv6servers.csv, webstats.csv, totalcw.csv, bridgedb-stats.csv] 2021-11-22 12:51:47,366 INFO o.t.m.s.main.Main:130 Completed metrics update run. 2021-11-22 15:21:22,435 INFO o.t.m.s.main.Main:32 Starting metrics update run. 2021-11-22 15:21:22,458 INFO o.t.m.s.main.Main:70 Starting org.torproject.metrics.stats.collectdescs.Main module. 2021-11-22 15:21:22,460 INFO o.t.d.i.DescriptorIndexCollector:47 Starting descriptor collection. 2021-11-22 15:21:22,460 INFO o.t.d.i.DescriptorIndexCollector:63 Indexing local directory /srv/metrics.torproject.org/metrics/work/shared/in. 2021-11-22 15:21:22,501 INFO o.t.d.i.DescriptorIndexCollector:75 Fetching remote index file https://collector.torproject.org/index/index.json. 2021-11-22 15:21:23,108 INFO o.t.d.i.DescriptorIndexCollector:84 Fetching remote files from https://collector.torproject.org. 2021-11-22 15:21:24,538 WARN o.t.d.i.DescriptorIndexCollector:132 Fetched remote file 2021-11-21-04-00-00-vote-0232AF901C31A04EE9848595AF9BB7620D4C5B2E-9DAFAF1CE0ABC369CCD86FB1D7E38C33CCFAB1D6 from https://collector.torproject.org has a size of 4669122 bytes which is different from the expected 1273856 bytes. Not storing this file. 2021-11-22 15:21:29,553 INFO o.t.d.i.DescriptorIndexCollector:90 Deleting extraneous files from local directory /srv/metrics.torproject.org/metrics/work/shared/in. 2021-11-22 15:21:29,644 INFO o.t.d.i.DescriptorIndexCollector:95 Finished descriptor collection. 2021-11-22 15:21:29,644 INFO o.t.m.s.main.Main:73 Completed org.torproject.metrics.stats.collectdescs.Main module. 2021-11-22 15:21:29,644 INFO o.t.m.s.main.Main:70 Starting org.torproject.metrics.stats.connbidirect.Main module. 2021-11-22 15:21:33,390 INFO o.t.m.s.main.Main:73 Completed org.torproject.metrics.stats.connbidirect.Main module. 2021-11-22 15:21:33,390 INFO o.t.m.s.main.Main:70 Starting org.torproject.metrics.stats.onionperf.Main module. 2021-11-22 15:21:33,391 INFO o.t.m.s.o.Main:46 Starting onionperf module. 2021-11-22 15:21:33,690 INFO o.t.m.s.o.Main:67 Connecting to database. 2021-11-22 15:21:33,941 INFO o.t.m.s.o.Main:70 Successfully connected to database. 2021-11-22 15:21:44,197 INFO o.t.m.s.o.Main:261 Querying statistics including partials from database. 2021-11-22 15:22:32,742 INFO o.t.m.s.o.Main:402 Writing 59089 lines to /srv/metrics.torproject.org/metrics/work/modules/onionperf/stats/onionperf-including-partials.csv. 2021-11-22 15:22:33,356 INFO o.t.m.s.o.Main:286 Querying timeout and failure statistics from database. 2021-11-22 15:22:37,612 INFO o.t.m.s.o.Main:402 Writing 19701 lines to /srv/metrics.torproject.org/metrics/work/modules/onionperf/stats/onionperf-failures.csv. 2021-11-22 15:22:37,615 INFO o.t.m.s.o.Main:309 Querying buildtime statistics from database. 2021-11-22 15:23:59,456 INFO o.t.m.s.o.Main:402 Writing 47674 lines to /srv/metrics.torproject.org/metrics/work/modules/onionperf/stats/buildtimes.csv. 2021-11-22 15:23:59,463 INFO o.t.m.s.o.Main:332 Querying latency statistics from database. 2021-11-22 15:24:11,591 INFO o.t.m.s.o.Main:402 Writing 18734 lines to /srv/metrics.torproject.org/metrics/work/modules/onionperf/stats/latencies.csv. 2021-11-22 15:24:11,665 INFO o.t.m.s.o.Main:357 Querying throughput statistics from database. 2021-11-22 15:24:23,114 INFO o.t.m.s.o.Main:402 Writing 19701 lines to /srv/metrics.torproject.org/metrics/work/modules/onionperf/stats/onionperf-throughput.csv. 2021-11-22 15:24:23,118 INFO o.t.m.s.o.Main:409 Disconnecting from database. 2021-11-22 15:24:23,118 INFO o.t.m.s.o.Main:62 Terminated onionperf module. 2021-11-22 15:24:23,118 INFO o.t.m.s.main.Main:73 Completed org.torproject.metrics.stats.onionperf.Main module. 2021-11-22 15:24:23,118 INFO o.t.m.s.main.Main:70 Starting org.torproject.metrics.stats.bwhist.Main module. 2021-11-22 15:24:23,119 INFO o.t.m.s.bwhist.Main:37 Starting bwhist module. 2021-11-22 15:24:23,119 INFO o.t.m.s.bwhist.Main:39 Reading descriptors and inserting relevant parts into the database. 2021-11-22 15:29:20,611 INFO o.t.m.s.b.RelayDescriptorDatabaseImporter:543 Finished importing relay descriptors: 44045 network status entries and 30850 bandwidth history elements 2021-11-22 15:29:25,224 INFO o.t.m.s.bwhist.Main:50 Aggregating database entries. 2021-11-22 15:30:25,037 INFO o.t.m.s.b.RelayDescriptorDatabaseImporter:543 Finished importing relay descriptors: 44045 network status entries and 30850 bandwidth history elements 2021-11-22 15:30:25,916 INFO o.t.m.s.bwhist.Main:53 Querying aggregated statistics from the database. 2021-11-22 15:30:26,293 INFO o.t.m.s.bwhist.Main:57 Closing database connection. 2021-11-22 15:30:26,293 INFO o.t.m.s.bwhist.Main:60 Terminating bwhist module. 2021-11-22 15:30:26,293 INFO o.t.m.s.main.Main:73 Completed org.torproject.metrics.stats.bwhist.Main module. 2021-11-22 15:30:26,294 INFO o.t.m.s.main.Main:70 Starting org.torproject.metrics.stats.advbwdist.Main module. 2021-11-22 15:31:41,138 INFO o.t.m.s.main.Main:73 Completed org.torproject.metrics.stats.advbwdist.Main module. 2021-11-22 15:31:41,139 INFO o.t.m.s.main.Main:70 Starting org.torproject.metrics.stats.hidserv.Main module. 2021-11-22 15:31:41,139 INFO o.t.m.s.hidserv.Main:37 Initializing parser and reading parse history... 2021-11-22 15:31:41,240 INFO o.t.m.s.hidserv.Main:52 Parsing descriptors... 2021-11-22 15:33:17,514 INFO o.t.m.s.hidserv.Main:59 Writing parse history... 2021-11-22 15:33:17,642 INFO o.t.m.s.hidserv.Main:66 Extrapolating statistics... 2021-11-22 15:37:30,549 INFO o.t.m.s.main.Main:32 Starting metrics update run. 2021-11-22 15:37:30,574 INFO o.t.m.s.main.Main:70 Starting org.torproject.metrics.stats.collectdescs.Main module. 2021-11-22 15:37:30,577 INFO o.t.d.i.DescriptorIndexCollector:47 Starting descriptor collection. 2021-11-22 15:37:30,578 INFO o.t.d.i.DescriptorIndexCollector:63 Indexing local directory /srv/metrics.torproject.org/metrics/work/shared/in. 2021-11-22 15:37:32,053 INFO o.t.d.i.DescriptorIndexCollector:75 Fetching remote index file https://collector.torproject.org/index/index.json. 2021-11-22 15:37:32,907 INFO o.t.d.i.DescriptorIndexCollector:84 Fetching remote files from https://collector.torproject.org. 2021-11-22 15:37:33,058 WARN o.t.d.i.DescriptorIndexCollector:132 Fetched remote file 2021-11-21-04-00-00-vote-0232AF901C31A04EE9848595AF9BB7620D4C5B2E-9DAFAF1CE0ABC369CCD86FB1D7E38C33CCFAB1D6 from https://collector.torproject.org has a size of 4669122 bytes which is different from the expected 1273856 bytes. Not storing this file. 2021-11-22 15:37:33,060 INFO o.t.d.i.DescriptorIndexCollector:90 Deleting extraneous files from local directory /srv/metrics.torproject.org/metrics/work/shared/in. 2021-11-22 15:37:33,064 INFO o.t.d.i.DescriptorIndexCollector:95 Finished descriptor collection. 2021-11-22 15:37:33,064 INFO o.t.m.s.main.Main:73 Completed org.torproject.metrics.stats.collectdescs.Main module. 2021-11-22 15:37:33,064 INFO o.t.m.s.main.Main:70 Starting org.torproject.metrics.stats.connbidirect.Main module. 2021-11-22 15:37:34,932 INFO o.t.m.s.main.Main:73 Completed org.torproject.metrics.stats.connbidirect.Main module. 2021-11-22 15:37:34,932 INFO o.t.m.s.main.Main:70 Starting org.torproject.metrics.stats.onionperf.Main module. 2021-11-22 15:37:34,934 INFO o.t.m.s.o.Main:46 Starting onionperf module. 2021-11-22 15:37:34,934 INFO o.t.m.s.o.Main:67 Connecting to database. 2021-11-22 15:37:35,879 INFO o.t.m.s.o.Main:70 Successfully connected to database. 2021-11-22 15:37:48,888 INFO o.t.m.s.o.Main:261 Querying statistics including partials from database. 2021-11-22 15:38:32,458 INFO o.t.m.s.o.Main:402 Writing 59089 lines to /srv/metrics.torproject.org/metrics/work/modules/onionperf/stats/onionperf-including-partials.csv. 2021-11-22 15:38:32,560 INFO o.t.m.s.o.Main:286 Querying timeout and failure statistics from database. 2021-11-22 15:38:36,776 INFO o.t.m.s.o.Main:402 Writing 19701 lines to /srv/metrics.torproject.org/metrics/work/modules/onionperf/stats/onionperf-failures.csv. 2021-11-22 15:38:36,833 INFO o.t.m.s.o.Main:309 Querying buildtime statistics from database. 2021-11-22 15:39:31,250 INFO o.t.m.s.o.Main:402 Writing 47674 lines to /srv/metrics.torproject.org/metrics/work/modules/onionperf/stats/buildtimes.csv. 2021-11-22 15:39:31,871 INFO o.t.m.s.o.Main:332 Querying latency statistics from database. 2021-11-22 15:39:43,212 INFO o.t.m.s.o.Main:402 Writing 18734 lines to /srv/metrics.torproject.org/metrics/work/modules/onionperf/stats/latencies.csv. 2021-11-22 15:39:43,214 INFO o.t.m.s.o.Main:357 Querying throughput statistics from database. 2021-11-22 15:39:54,502 INFO o.t.m.s.o.Main:402 Writing 19701 lines to /srv/metrics.torproject.org/metrics/work/modules/onionperf/stats/onionperf-throughput.csv. 2021-11-22 15:39:54,505 INFO o.t.m.s.o.Main:409 Disconnecting from database. 2021-11-22 15:39:54,506 INFO o.t.m.s.o.Main:62 Terminated onionperf module. 2021-11-22 15:39:54,506 INFO o.t.m.s.main.Main:73 Completed org.torproject.metrics.stats.onionperf.Main module. 2021-11-22 15:39:54,506 INFO o.t.m.s.main.Main:70 Starting org.torproject.metrics.stats.bwhist.Main module. 2021-11-22 15:39:54,506 INFO o.t.m.s.bwhist.Main:37 Starting bwhist module. 2021-11-22 15:39:54,506 INFO o.t.m.s.bwhist.Main:39 Reading descriptors and inserting relevant parts into the database. 2021-11-22 15:39:54,674 INFO o.t.m.s.b.RelayDescriptorDatabaseImporter:543 Finished importing relay descriptors: 0 network status entries and 0 bandwidth history elements 2021-11-22 15:39:54,675 INFO o.t.m.s.bwhist.Main:50 Aggregating database entries. 2021-11-22 15:40:22,541 INFO o.t.m.s.b.RelayDescriptorDatabaseImporter:543 Finished importing relay descriptors: 0 network status entries and 0 bandwidth history elements 2021-11-22 15:40:22,589 INFO o.t.m.s.bwhist.Main:53 Querying aggregated statistics from the database. 2021-11-22 15:40:22,782 INFO o.t.m.s.bwhist.Main:57 Closing database connection. 2021-11-22 15:40:22,782 INFO o.t.m.s.bwhist.Main:60 Terminating bwhist module. 2021-11-22 15:40:22,782 INFO o.t.m.s.main.Main:73 Completed org.torproject.metrics.stats.bwhist.Main module. 2021-11-22 15:40:22,782 INFO o.t.m.s.main.Main:70 Starting org.torproject.metrics.stats.advbwdist.Main module. 2021-11-22 15:41:07,158 INFO o.t.m.s.main.Main:73 Completed org.torproject.metrics.stats.advbwdist.Main module. 2021-11-22 15:41:07,158 INFO o.t.m.s.main.Main:70 Starting org.torproject.metrics.stats.hidserv.Main module. 2021-11-22 15:41:07,158 INFO o.t.m.s.hidserv.Main:37 Initializing parser and reading parse history... 2021-11-22 15:41:07,264 INFO o.t.m.s.hidserv.Main:52 Parsing descriptors... 2021-11-22 15:42:28,580 INFO o.t.m.s.hidserv.Main:59 Writing parse history... 2021-11-22 15:42:28,815 INFO o.t.m.s.hidserv.Main:66 Extrapolating statistics... 2021-11-22 15:47:20,105 INFO o.t.m.s.hidserv.Main:82 Aggregating v2 statistics... 2021-11-22 15:48:09,834 INFO o.t.m.s.hidserv.Main:98 Aggregating v3 statistics... 2021-11-22 15:48:10,043 INFO o.t.m.s.hidserv.Main:108 Terminating. 2021-11-22 15:48:10,045 INFO o.t.m.s.main.Main:73 Completed org.torproject.metrics.stats.hidserv.Main module. 2021-11-22 15:48:10,045 INFO o.t.m.s.main.Main:70 Starting org.torproject.metrics.stats.clients.Main module. 2021-11-22 15:48:10,072 INFO o.t.m.s.clients.Main:36 Starting clients module. 2021-11-22 15:48:10,073 INFO o.t.m.s.clients.Main:38 Connecting to database. 2021-11-22 15:48:10,329 INFO o.t.m.s.clients.Main:41 Reading relay descriptors and importing relevant parts into the database. 2021-11-22 15:48:34,936 INFO o.t.m.s.clients.Main:45 Reading bridge descriptors and importing relevant parts into the database. 2021-11-22 15:48:43,601 INFO o.t.m.s.clients.Main:49 Processing newly imported data. 2021-11-22 16:05:35,513 INFO o.t.m.s.main.Main:32 Starting metrics update run. 2021-11-22 16:05:35,571 INFO o.t.m.s.main.Main:70 Starting org.torproject.metrics.stats.collectdescs.Main module. 2021-11-22 16:05:35,574 INFO o.t.d.i.DescriptorIndexCollector:47 Starting descriptor collection. 2021-11-22 16:05:35,614 INFO o.t.d.i.DescriptorIndexCollector:63 Indexing local directory /srv/metrics.torproject.org/metrics/work/shared/in. 2021-11-22 16:05:38,664 INFO o.t.d.i.DescriptorIndexCollector:75 Fetching remote index file https://collector.torproject.org/index/index.json. 2021-11-22 16:05:39,942 INFO o.t.d.i.DescriptorIndexCollector:84 Fetching remote files from https://collector.torproject.org. 2021-11-22 16:05:40,148 WARN o.t.d.i.DescriptorIndexCollector:132 Fetched remote file 2021-11-21-04-00-00-vote-0232AF901C31A04EE9848595AF9BB7620D4C5B2E-9DAFAF1CE0ABC369CCD86FB1D7E38C33CCFAB1D6 from https://collector.torproject.org has a size of 4669122 bytes which is different from the expected 1273856 bytes. Not storing this file. 2021-11-22 16:05:40,150 INFO o.t.d.i.DescriptorIndexCollector:90 Deleting extraneous files from local directory /srv/metrics.torproject.org/metrics/work/shared/in. 2021-11-22 16:05:40,181 INFO o.t.d.i.DescriptorIndexCollector:95 Finished descriptor collection. 2021-11-22 16:05:40,181 INFO o.t.m.s.main.Main:73 Completed org.torproject.metrics.stats.collectdescs.Main module. 2021-11-22 16:05:40,181 INFO o.t.m.s.main.Main:70 Starting org.torproject.metrics.stats.connbidirect.Main module. 2021-11-22 16:05:44,606 INFO o.t.m.s.main.Main:73 Completed org.torproject.metrics.stats.connbidirect.Main module. 2021-11-22 16:05:44,607 INFO o.t.m.s.main.Main:70 Starting org.torproject.metrics.stats.onionperf.Main module. 2021-11-22 16:05:44,608 INFO o.t.m.s.o.Main:46 Starting onionperf module. 2021-11-22 16:05:44,608 INFO o.t.m.s.o.Main:67 Connecting to database. 2021-11-22 16:05:45,049 INFO o.t.m.s.o.Main:70 Successfully connected to database. 2021-11-22 16:05:55,789 INFO o.t.m.s.o.Main:261 Querying statistics including partials from database. 2021-11-22 16:06:31,639 INFO o.t.m.s.o.Main:402 Writing 59089 lines to /srv/metrics.torproject.org/metrics/work/modules/onionperf/stats/onionperf-including-partials.csv. 2021-11-22 16:06:31,880 INFO o.t.m.s.o.Main:286 Querying timeout and failure statistics from database. 2021-11-22 16:06:49,147 INFO o.t.m.s.o.Main:402 Writing 19701 lines to /srv/metrics.torproject.org/metrics/work/modules/onionperf/stats/onionperf-failures.csv. 2021-11-22 16:06:49,150 INFO o.t.m.s.o.Main:309 Querying buildtime statistics from database. 2021-11-22 16:07:48,883 INFO o.t.m.s.o.Main:402 Writing 47674 lines to /srv/metrics.torproject.org/metrics/work/modules/onionperf/stats/buildtimes.csv. 2021-11-22 16:07:48,904 INFO o.t.m.s.o.Main:332 Querying latency statistics from database. 2021-11-22 16:08:00,750 INFO o.t.m.s.o.Main:402 Writing 18734 lines to /srv/metrics.torproject.org/metrics/work/modules/onionperf/stats/latencies.csv. 2021-11-22 16:08:00,753 INFO o.t.m.s.o.Main:357 Querying throughput statistics from database. 2021-11-22 16:08:12,548 INFO o.t.m.s.o.Main:402 Writing 19701 lines to /srv/metrics.torproject.org/metrics/work/modules/onionperf/stats/onionperf-throughput.csv. 2021-11-22 16:08:12,551 INFO o.t.m.s.o.Main:409 Disconnecting from database. 2021-11-22 16:08:12,551 INFO o.t.m.s.o.Main:62 Terminated onionperf module. 2021-11-22 16:08:12,551 INFO o.t.m.s.main.Main:73 Completed org.torproject.metrics.stats.onionperf.Main module. 2021-11-22 16:08:12,551 INFO o.t.m.s.main.Main:70 Starting org.torproject.metrics.stats.bwhist.Main module. 2021-11-22 16:08:12,551 INFO o.t.m.s.bwhist.Main:37 Starting bwhist module. 2021-11-22 16:08:12,552 INFO o.t.m.s.bwhist.Main:39 Reading descriptors and inserting relevant parts into the database. 2021-11-22 16:08:25,795 INFO o.t.m.s.b.RelayDescriptorDatabaseImporter:543 Finished importing relay descriptors: 0 network status entries and 1964 bandwidth history elements 2021-11-22 16:08:30,634 INFO o.t.m.s.bwhist.Main:50 Aggregating database entries. 2021-11-22 16:08:43,202 INFO o.t.m.s.b.RelayDescriptorDatabaseImporter:543 Finished importing relay descriptors: 0 network status entries and 1964 bandwidth history elements 2021-11-22 16:08:43,231 INFO o.t.m.s.bwhist.Main:53 Querying aggregated statistics from the database. 2021-11-22 16:08:43,416 INFO o.t.m.s.bwhist.Main:57 Closing database connection. 2021-11-22 16:08:43,416 INFO o.t.m.s.bwhist.Main:60 Terminating bwhist module. 2021-11-22 16:08:43,416 INFO o.t.m.s.main.Main:73 Completed org.torproject.metrics.stats.bwhist.Main module. 2021-11-22 16:08:43,416 INFO o.t.m.s.main.Main:70 Starting org.torproject.metrics.stats.advbwdist.Main module. 2021-11-22 16:09:27,670 INFO o.t.m.s.main.Main:73 Completed org.torproject.metrics.stats.advbwdist.Main module. 2021-11-22 16:09:27,705 INFO o.t.m.s.main.Main:70 Starting org.torproject.metrics.stats.hidserv.Main module. 2021-11-22 16:09:27,705 INFO o.t.m.s.hidserv.Main:37 Initializing parser and reading parse history... 2021-11-22 16:09:27,763 INFO o.t.m.s.hidserv.Main:52 Parsing descriptors... 2021-11-22 16:10:56,887 INFO o.t.m.s.hidserv.Main:59 Writing parse history... 2021-11-22 16:10:56,953 INFO o.t.m.s.hidserv.Main:66 Extrapolating statistics... 2021-11-22 16:14:47,208 INFO o.t.m.s.hidserv.Main:82 Aggregating v2 statistics... 2021-11-22 16:15:43,512 INFO o.t.m.s.hidserv.Main:98 Aggregating v3 statistics... 2021-11-22 16:15:43,897 INFO o.t.m.s.hidserv.Main:108 Terminating. 2021-11-22 16:15:43,897 INFO o.t.m.s.main.Main:73 Completed org.torproject.metrics.stats.hidserv.Main module. 2021-11-22 16:15:43,898 INFO o.t.m.s.main.Main:70 Starting org.torproject.metrics.stats.clients.Main module. 2021-11-22 16:15:43,910 INFO o.t.m.s.clients.Main:36 Starting clients module. 2021-11-22 16:15:43,910 INFO o.t.m.s.clients.Main:38 Connecting to database. 2021-11-22 16:15:44,273 INFO o.t.m.s.clients.Main:41 Reading relay descriptors and importing relevant parts into the database. 2021-11-22 16:15:44,968 INFO o.t.m.s.clients.Main:45 Reading bridge descriptors and importing relevant parts into the database. 2021-11-22 16:15:45,159 INFO o.t.m.s.clients.Main:49 Processing newly imported data. 2021-11-22 17:22:20,708 INFO o.t.m.s.main.Main:32 Starting metrics update run. 2021-11-22 17:22:21,031 INFO o.t.m.s.main.Main:70 Starting org.torproject.metrics.stats.collectdescs.Main module. 2021-11-22 17:22:21,034 INFO o.t.d.i.DescriptorIndexCollector:47 Starting descriptor collection. 2021-11-22 17:22:21,034 INFO o.t.d.i.DescriptorIndexCollector:63 Indexing local directory /srv/metrics.torproject.org/metrics/work/shared/in. 2021-11-22 17:22:22,451 INFO o.t.d.i.DescriptorIndexCollector:75 Fetching remote index file https://collector.torproject.org/index/index.json. 2021-11-22 17:22:24,521 INFO o.t.d.i.DescriptorIndexCollector:84 Fetching remote files from https://collector.torproject.org. 2021-11-22 17:22:25,583 WARN o.t.d.i.DescriptorIndexCollector:132 Fetched remote file 2021-11-21-04-00-00-vote-0232AF901C31A04EE9848595AF9BB7620D4C5B2E-9DAFAF1CE0ABC369CCD86FB1D7E38C33CCFAB1D6 from https://collector.torproject.org has a size of 4669122 bytes which is different from the expected 1273856 bytes. Not storing this file. 2021-11-22 17:22:26,447 INFO o.t.d.i.DescriptorIndexCollector:90 Deleting extraneous files from local directory /srv/metrics.torproject.org/metrics/work/shared/in. 2021-11-22 17:22:27,267 INFO o.t.d.i.DescriptorIndexCollector:95 Finished descriptor collection. 2021-11-22 17:22:27,267 INFO o.t.m.s.main.Main:73 Completed org.torproject.metrics.stats.collectdescs.Main module. 2021-11-22 17:22:27,267 INFO o.t.m.s.main.Main:70 Starting org.torproject.metrics.stats.connbidirect.Main module. 2021-11-22 17:22:33,432 INFO o.t.m.s.main.Main:73 Completed org.torproject.metrics.stats.connbidirect.Main module. 2021-11-22 17:22:33,444 INFO o.t.m.s.main.Main:70 Starting org.torproject.metrics.stats.onionperf.Main module. 2021-11-22 17:22:33,446 INFO o.t.m.s.o.Main:46 Starting onionperf module. 2021-11-22 17:22:33,446 INFO o.t.m.s.o.Main:67 Connecting to database. 2021-11-22 17:22:33,653 INFO o.t.m.s.o.Main:70 Successfully connected to database. 2021-11-22 17:22:45,602 INFO o.t.m.s.o.Main:261 Querying statistics including partials from database. 2021-11-22 17:26:37,438 INFO o.t.m.s.o.Main:402 Writing 59089 lines to /srv/metrics.torproject.org/metrics/work/modules/onionperf/stats/onionperf-including-partials.csv. 2021-11-22 17:26:37,977 INFO o.t.m.s.o.Main:286 Querying timeout and failure statistics from database. 2021-11-22 17:27:32,551 INFO o.t.m.s.o.Main:402 Writing 19701 lines to /srv/metrics.torproject.org/metrics/work/modules/onionperf/stats/onionperf-failures.csv. 2021-11-22 17:27:32,746 INFO o.t.m.s.o.Main:309 Querying buildtime statistics from database. 2021-11-22 17:35:16,854 INFO o.t.m.s.o.Main:402 Writing 47674 lines to /srv/metrics.torproject.org/metrics/work/modules/onionperf/stats/buildtimes.csv. 2021-11-22 17:35:17,000 INFO o.t.m.s.o.Main:332 Querying latency statistics from database. 2021-11-22 17:36:22,624 INFO o.t.m.s.o.Main:402 Writing 18734 lines to /srv/metrics.torproject.org/metrics/work/modules/onionperf/stats/latencies.csv. 2021-11-22 17:36:22,766 INFO o.t.m.s.o.Main:357 Querying throughput statistics from database. 2021-11-22 17:37:08,585 INFO o.t.m.s.o.Main:402 Writing 19701 lines to /srv/metrics.torproject.org/metrics/work/modules/onionperf/stats/onionperf-throughput.csv. 2021-11-22 17:37:08,599 INFO o.t.m.s.o.Main:409 Disconnecting from database. 2021-11-22 17:37:08,604 INFO o.t.m.s.o.Main:62 Terminated onionperf module. 2021-11-22 17:37:08,605 INFO o.t.m.s.main.Main:73 Completed org.torproject.metrics.stats.onionperf.Main module. 2021-11-22 17:37:08,605 INFO o.t.m.s.main.Main:70 Starting org.torproject.metrics.stats.bwhist.Main module. 2021-11-22 17:37:08,605 INFO o.t.m.s.bwhist.Main:37 Starting bwhist module. 2021-11-22 17:37:08,605 INFO o.t.m.s.bwhist.Main:39 Reading descriptors and inserting relevant parts into the database. 2021-11-22 17:37:56,597 INFO o.t.m.s.b.RelayDescriptorDatabaseImporter:543 Finished importing relay descriptors: 12532 network status entries and 5749 bandwidth history elements 2021-11-22 17:37:59,799 INFO o.t.m.s.bwhist.Main:50 Aggregating database entries. 2021-11-22 17:38:13,258 INFO o.t.m.s.b.RelayDescriptorDatabaseImporter:543 Finished importing relay descriptors: 12532 network status entries and 5749 bandwidth history elements 2021-11-22 17:38:13,609 INFO o.t.m.s.bwhist.Main:53 Querying aggregated statistics from the database. 2021-11-22 17:38:14,067 INFO o.t.m.s.bwhist.Main:57 Closing database connection. 2021-11-22 17:38:14,067 INFO o.t.m.s.bwhist.Main:60 Terminating bwhist module. 2021-11-22 17:38:14,067 INFO o.t.m.s.main.Main:73 Completed org.torproject.metrics.stats.bwhist.Main module. 2021-11-22 17:38:14,068 INFO o.t.m.s.main.Main:70 Starting org.torproject.metrics.stats.advbwdist.Main module. 2021-11-22 17:39:03,556 INFO o.t.m.s.main.Main:73 Completed org.torproject.metrics.stats.advbwdist.Main module. 2021-11-22 17:39:03,557 INFO o.t.m.s.main.Main:70 Starting org.torproject.metrics.stats.hidserv.Main module. 2021-11-22 17:39:03,558 INFO o.t.m.s.hidserv.Main:37 Initializing parser and reading parse history... 2021-11-22 17:39:03,586 INFO o.t.m.s.hidserv.Main:52 Parsing descriptors... 2021-11-22 17:44:00,218 INFO o.t.m.s.clients.Main:53 Querying aggregated statistics from the database.

1 0

Cron <collector@colchicifolium> cd /srv/collector.torproject.org/collector/logs/ && cat metrics.log | grep -E "WARN|ERROR|FATAL" | grep -v "Overwriting existing descriptor file"
by root＠colchicifolium.torproject.org 23 Nov '21

23 Nov '21

2021-11-23 00:07:34,702 WARN o.t.m.c.r.ReferenceChecker:314 Missing too many referenced descriptors (29.118167462542278). 2021-11-23 00:37:25,309 WARN o.t.m.c.r.ReferenceChecker:314 Missing too many referenced descriptors (20.657715239550473). 2021-11-23 01:07:07,539 WARN o.t.m.c.r.ReferenceChecker:314 Missing too many referenced descriptors (22.320650794880187). 2021-11-23 01:37:01,839 WARN o.t.m.c.r.ReferenceChecker:314 Missing too many referenced descriptors (21.40616260590386). 2021-11-23 02:07:37,315 WARN o.t.m.c.r.ReferenceChecker:314 Missing too many referenced descriptors (27.587079554426996). 2021-11-23 02:37:02,098 WARN o.t.m.c.r.ReferenceChecker:314 Missing too many referenced descriptors (12.44337105447559). 2021-11-23 03:07:18,918 WARN o.t.m.c.r.ReferenceChecker:314 Missing too many referenced descriptors (24.08873833936836). 2021-11-23 03:37:30,176 WARN o.t.m.c.r.ReferenceChecker:314 Missing too many referenced descriptors (20.59504687503056). 2021-11-23 04:13:05,557 WARN o.t.m.c.r.ReferenceChecker:314 Missing too many referenced descriptors (23.154612396904263). 2021-11-23 04:37:44,465 WARN o.t.m.c.r.ReferenceChecker:314 Missing too many referenced descriptors (22.253456313968897). 2021-11-23 05:08:20,817 WARN o.t.m.c.r.ReferenceChecker:314 Missing too many referenced descriptors (28.92386645362239). 2021-11-23 05:38:37,624 WARN o.t.m.c.r.ReferenceChecker:314 Missing too many referenced descriptors (18.147977277159164). 2021-11-23 06:07:44,322 WARN o.t.m.c.r.ReferenceChecker:314 Missing too many referenced descriptors (26.684766755741634).

1 0

Cron <onionoo@onionoo-backend-01> /srv/onionoo.torproject.org/onionoo/bin/service/onionoo
by root＠onionoo-backend-01.torproject.org 23 Nov '21

23 Nov '21

Traceback (most recent call last): File "/srv/onionoo.torproject.org/onionoo/bin/service/onionoo", line 23, in <module> list_of_errors = Counter([(x.split(' ')[2], x.split(' ')[3].split(':')[0]) for x in str(out).split('\\n') if x != '"']) File "/srv/onionoo.torproject.org/onionoo/bin/service/onionoo", line 23, in <listcomp> list_of_errors = Counter([(x.split(' ')[2], x.split(' ')[3].split(':')[0]) for x in str(out).split('\\n') if x != '"']) IndexError: list index out of range During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/srv/onionoo.torproject.org/onionoo/bin/service/onionoo", line 29, in <module> g_warnings.labels(host, "Problem running exporter - {}".format(e)).set(1) File "/usr/lib/python3/dist-packages/prometheus_client/metrics.py", line 146, in labels raise ValueError('Incorrect label count') ValueError: Incorrect label count

1 0

Cron <onionoo@onionoo-backend-01> /srv/onionoo.torproject.org/onionoo/bin/service/onionoo
by root＠onionoo-backend-01.torproject.org 23 Nov '21

23 Nov '21

Traceback (most recent call last): File "/srv/onionoo.torproject.org/onionoo/bin/service/onionoo", line 23, in <module> list_of_errors = Counter([(x.split(' ')[2], x.split(' ')[3].split(':')[0]) for x in str(out).split('\\n') if x != '"']) File "/srv/onionoo.torproject.org/onionoo/bin/service/onionoo", line 23, in <listcomp> list_of_errors = Counter([(x.split(' ')[2], x.split(' ')[3].split(':')[0]) for x in str(out).split('\\n') if x != '"']) IndexError: list index out of range During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/srv/onionoo.torproject.org/onionoo/bin/service/onionoo", line 29, in <module> g_warnings.labels(host, "Problem running exporter - {}".format(e)).set(1) File "/usr/lib/python3/dist-packages/prometheus_client/metrics.py", line 146, in labels raise ValueError('Incorrect label count') ValueError: Incorrect label count

1 0