November 2021 - metrics-alerts - lists.torproject.org

Cron <metrics@meronense> cd /srv/metrics.torproject.org/metrics/work/modules/logs && cat metrics.log | grep -v "Could not insert bandwidth "
by root＠meronense.torproject.org 16 Nov '21

16 Nov '21

2021-11-15 08:40:02,177 INFO o.t.m.s.main.Main:32 Starting metrics update run. 2021-11-15 08:40:02,243 INFO o.t.m.s.main.Main:70 Starting org.torproject.metrics.stats.collectdescs.Main module. 2021-11-15 08:40:02,245 INFO o.t.d.i.DescriptorIndexCollector:47 Starting descriptor collection. 2021-11-15 08:40:02,257 INFO o.t.d.i.DescriptorIndexCollector:63 Indexing local directory /srv/metrics.torproject.org/metrics/work/shared/in. 2021-11-15 08:40:06,951 INFO o.t.d.i.DescriptorIndexCollector:75 Fetching remote index file https://collector.torproject.org/index/index.json. 2021-11-15 08:40:07,768 INFO o.t.d.i.DescriptorIndexCollector:84 Fetching remote files from https://collector.torproject.org. 2021-11-15 08:40:57,832 INFO o.t.d.i.DescriptorIndexCollector:90 Deleting extraneous files from local directory /srv/metrics.torproject.org/metrics/work/shared/in. 2021-11-15 08:40:58,167 INFO o.t.d.i.DescriptorIndexCollector:95 Finished descriptor collection. 2021-11-15 08:40:58,167 INFO o.t.m.s.main.Main:73 Completed org.torproject.metrics.stats.collectdescs.Main module. 2021-11-15 08:40:58,167 INFO o.t.m.s.main.Main:70 Starting org.torproject.metrics.stats.connbidirect.Main module. 2021-11-15 08:41:07,449 INFO o.t.m.s.main.Main:73 Completed org.torproject.metrics.stats.connbidirect.Main module. 2021-11-15 08:41:07,449 INFO o.t.m.s.main.Main:70 Starting org.torproject.metrics.stats.onionperf.Main module. 2021-11-15 08:41:07,452 INFO o.t.m.s.o.Main:46 Starting onionperf module. 2021-11-15 08:41:07,453 INFO o.t.m.s.o.Main:67 Connecting to database. 2021-11-15 08:41:07,880 INFO o.t.m.s.o.Main:70 Successfully connected to database. 2021-11-15 08:51:01,150 INFO o.t.m.s.o.Main:261 Querying statistics including partials from database. 2021-11-15 08:51:41,624 INFO o.t.m.s.o.Main:402 Writing 58879 lines to /srv/metrics.torproject.org/metrics/work/modules/onionperf/stats/onionperf-including-partials.csv. 2021-11-15 08:51:42,356 INFO o.t.m.s.o.Main:286 Querying timeout and failure statistics from database. 2021-11-15 08:51:44,973 INFO o.t.m.s.o.Main:402 Writing 19631 lines to /srv/metrics.torproject.org/metrics/work/modules/onionperf/stats/onionperf-failures.csv. 2021-11-15 08:51:44,989 INFO o.t.m.s.o.Main:309 Querying buildtime statistics from database. 2021-11-15 08:52:25,281 INFO o.t.m.s.o.Main:402 Writing 47548 lines to /srv/metrics.torproject.org/metrics/work/modules/onionperf/stats/buildtimes.csv. 2021-11-15 08:52:25,608 INFO o.t.m.s.o.Main:332 Querying latency statistics from database. 2021-11-15 08:52:37,028 INFO o.t.m.s.o.Main:402 Writing 18664 lines to /srv/metrics.torproject.org/metrics/work/modules/onionperf/stats/latencies.csv. 2021-11-15 08:52:37,031 INFO o.t.m.s.o.Main:357 Querying throughput statistics from database. 2021-11-15 08:52:48,334 INFO o.t.m.s.o.Main:402 Writing 19631 lines to /srv/metrics.torproject.org/metrics/work/modules/onionperf/stats/onionperf-throughput.csv. 2021-11-15 08:52:48,337 INFO o.t.m.s.o.Main:409 Disconnecting from database. 2021-11-15 08:52:48,337 INFO o.t.m.s.o.Main:62 Terminated onionperf module. 2021-11-15 08:52:48,337 INFO o.t.m.s.main.Main:73 Completed org.torproject.metrics.stats.onionperf.Main module. 2021-11-15 08:52:48,337 INFO o.t.m.s.main.Main:70 Starting org.torproject.metrics.stats.bwhist.Main module. 2021-11-15 08:52:48,337 INFO o.t.m.s.bwhist.Main:37 Starting bwhist module. 2021-11-15 08:52:48,337 INFO o.t.m.s.bwhist.Main:39 Reading descriptors and inserting relevant parts into the database. 2021-11-15 09:06:24,553 INFO o.t.m.s.b.RelayDescriptorDatabaseImporter:543 Finished importing relay descriptors: 147617 network status entries and 125734 bandwidth history elements 2021-11-15 09:06:27,783 INFO o.t.m.s.bwhist.Main:50 Aggregating database entries. 2021-11-15 09:07:48,037 INFO o.t.m.s.b.RelayDescriptorDatabaseImporter:543 Finished importing relay descriptors: 147617 network status entries and 125734 bandwidth history elements 2021-11-15 09:07:51,362 INFO o.t.m.s.bwhist.Main:53 Querying aggregated statistics from the database. 2021-11-15 09:07:51,785 INFO o.t.m.s.bwhist.Main:57 Closing database connection. 2021-11-15 09:07:51,785 INFO o.t.m.s.bwhist.Main:60 Terminating bwhist module. 2021-11-15 09:07:51,785 INFO o.t.m.s.main.Main:73 Completed org.torproject.metrics.stats.bwhist.Main module. 2021-11-15 09:07:51,785 INFO o.t.m.s.main.Main:70 Starting org.torproject.metrics.stats.advbwdist.Main module. 2021-11-15 09:08:54,841 INFO o.t.m.s.main.Main:73 Completed org.torproject.metrics.stats.advbwdist.Main module. 2021-11-15 09:08:55,210 INFO o.t.m.s.main.Main:70 Starting org.torproject.metrics.stats.hidserv.Main module. 2021-11-15 09:08:55,211 INFO o.t.m.s.hidserv.Main:37 Initializing parser and reading parse history... 2021-11-15 09:08:55,548 INFO o.t.m.s.hidserv.Main:52 Parsing descriptors... 2021-11-15 09:10:39,287 INFO o.t.m.s.hidserv.Main:59 Writing parse history... 2021-11-15 09:10:39,811 INFO o.t.m.s.hidserv.Main:66 Extrapolating statistics... 2021-11-15 09:16:41,815 INFO o.t.m.s.hidserv.Main:82 Aggregating v2 statistics... 2021-11-15 09:19:09,755 INFO o.t.m.s.hidserv.Main:98 Aggregating v3 statistics... 2021-11-15 09:19:10,022 INFO o.t.m.s.hidserv.Main:108 Terminating. 2021-11-15 09:19:10,022 INFO o.t.m.s.main.Main:73 Completed org.torproject.metrics.stats.hidserv.Main module. 2021-11-15 09:19:10,023 INFO o.t.m.s.main.Main:70 Starting org.torproject.metrics.stats.clients.Main module. 2021-11-15 09:19:10,049 INFO o.t.m.s.clients.Main:36 Starting clients module. 2021-11-15 09:19:10,049 INFO o.t.m.s.clients.Main:38 Connecting to database. 2021-11-15 09:19:10,375 INFO o.t.m.s.clients.Main:41 Reading relay descriptors and importing relevant parts into the database. 2021-11-15 09:22:58,099 INFO o.t.m.s.clients.Main:45 Reading bridge descriptors and importing relevant parts into the database. 2021-11-15 09:24:22,016 INFO o.t.m.s.clients.Main:49 Processing newly imported data. 2021-11-15 10:41:16,643 INFO o.t.m.s.clients.Main:53 Querying aggregated statistics from the database. 2021-11-15 10:41:35,981 INFO o.t.m.s.clients.Main:59 Disconnecting from database. 2021-11-15 10:41:36,078 INFO o.t.m.s.clients.Main:62 Running detector. 2021-11-15 10:42:18,348 INFO o.t.m.s.clients.Main:65 Terminating clients module. 2021-11-15 10:42:18,388 INFO o.t.m.s.main.Main:73 Completed org.torproject.metrics.stats.clients.Main module. 2021-11-15 10:42:18,389 INFO o.t.m.s.main.Main:70 Starting org.torproject.metrics.stats.servers.Main module. 2021-11-15 10:42:18,399 INFO o.t.m.s.servers.Main:43 Starting servers module. 2021-11-15 10:42:18,399 INFO o.t.m.s.servers.Main:45 Reading descriptors and inserting relevant parts into the database. 2021-11-15 10:58:17,973 INFO o.t.m.s.servers.Main:77 Aggregating database entries. 2021-11-15 11:34:45,866 INFO o.t.m.s.servers.Main:80 Committing all updated parts in the database. 2021-11-15 11:34:46,822 INFO o.t.m.s.servers.Main:90 Querying aggregated statistics from the database. 2021-11-15 11:54:58,189 INFO o.t.m.s.servers.Main:105 Terminating servers module. 2021-11-15 11:54:59,140 INFO o.t.m.s.main.Main:73 Completed org.torproject.metrics.stats.servers.Main module. 2021-11-15 11:54:59,140 INFO o.t.m.s.main.Main:70 Starting org.torproject.metrics.stats.webstats.Main module. 2021-11-15 11:54:59,213 INFO o.t.m.s.w.Main:75 Starting webstats module. 2021-11-15 11:54:59,213 INFO o.t.m.s.w.Main:92 Connecting to database. 2021-11-15 11:54:59,349 INFO o.t.m.s.w.Main:95 Successfully connected to database. 2021-11-15 11:54:59,349 INFO o.t.m.s.w.Main:101 Querying previously imported log files. 2021-11-15 11:55:09,154 INFO o.t.m.s.w.Main:113 Found 283713 previously imported log files. 2021-11-15 12:08:01,500 INFO o.t.m.s.w.Main:268 Querying statistics from database. 2021-11-15 12:30:15,940 INFO o.t.m.s.w.Main:298 Writing 1411956 lines to /srv/metrics.torproject.org/metrics/work/modules/webstats/stats/webstats.csv. 2021-11-15 12:30:16,906 INFO o.t.m.s.w.Main:305 Disconnecting from database. 2021-11-15 12:30:16,907 INFO o.t.m.s.w.Main:87 Terminated webstats module. 2021-11-15 12:30:16,907 INFO o.t.m.s.main.Main:73 Completed org.torproject.metrics.stats.webstats.Main module. 2021-11-15 12:30:16,907 INFO o.t.m.s.main.Main:70 Starting org.torproject.metrics.stats.totalcw.Main module. 2021-11-15 12:30:16,908 INFO o.t.m.s.totalcw.Main:38 Starting totalcw module. 2021-11-15 12:30:16,908 INFO o.t.m.s.totalcw.Main:40 Reading consensuses and votes and inserting relevant parts into the database. 2021-11-15 12:32:32,871 INFO o.t.m.s.totalcw.Main:64 Committing all updated parts in the database. 2021-11-15 12:32:32,959 INFO o.t.m.s.totalcw.Main:74 Querying aggregated statistics from the database. 2021-11-15 12:32:36,816 INFO o.t.m.s.totalcw.Main:77 Writing aggregated statistics to /srv/metrics.torproject.org/metrics/work/modules/totalcw/stats/totalcw.csv. 2021-11-15 12:32:37,408 INFO o.t.m.s.totalcw.Main:82 Terminating totalcw module. 2021-11-15 12:32:37,413 INFO o.t.m.s.main.Main:73 Completed org.torproject.metrics.stats.totalcw.Main module. 2021-11-15 12:32:37,413 INFO o.t.m.s.main.Main:70 Starting org.torproject.metrics.stats.bridgedb.Main module. 2021-11-15 12:32:37,818 INFO o.t.m.s.main.Main:73 Completed org.torproject.metrics.stats.bridgedb.Main module. 2021-11-15 12:32:37,818 INFO o.t.m.s.main.Main:82 Making module data available. 2021-11-15 12:32:48,569 INFO o.t.m.s.main.Main:126 Successfully copied 22 files to stats output directory: [connbidirect2.csv, onionperf-failures.csv, buildtimes.csv, latencies.csv, torperf-1.1.csv, onionperf-throughput.csv, onionperf-including-partials.csv, bandwidth.csv, advbwdist.csv, v3hidserv.csv, hidserv.csv, clients.csv, userstats-combined.csv, platforms.csv, relayflags.csv, versions.csv, advbw.csv, networksize.csv, ipv6servers.csv, webstats.csv, totalcw.csv, bridgedb-stats.csv] 2021-11-15 12:32:48,569 INFO o.t.m.s.main.Main:130 Completed metrics update run.

1 0

Cron <collector@colchicifolium> cd /srv/collector.torproject.org/collector/logs/ && cat metrics.log | grep -E "WARN|ERROR|FATAL" | grep -v "Overwriting existing descriptor file"
by root＠colchicifolium.torproject.org 16 Nov '21

16 Nov '21

2021-11-16 00:07:46,655 WARN o.t.m.c.r.ReferenceChecker:314 Missing too many referenced descriptors (35.66600268841573). 2021-11-16 00:37:59,084 WARN o.t.m.c.r.ReferenceChecker:314 Missing too many referenced descriptors (33.59906050855842). 2021-11-16 01:07:26,729 WARN o.t.m.c.r.ReferenceChecker:314 Missing too many referenced descriptors (45.247214754274786). 2021-11-16 01:37:48,140 WARN o.t.m.c.r.ReferenceChecker:314 Missing too many referenced descriptors (28.00423062516502). 2021-11-16 02:07:48,981 WARN o.t.m.c.r.ReferenceChecker:314 Missing too many referenced descriptors (25.529720305187507). 2021-11-16 02:37:13,123 WARN o.t.m.c.r.ReferenceChecker:314 Missing too many referenced descriptors (8.163905768846943). 2021-11-16 03:07:10,768 WARN o.t.m.c.r.ReferenceChecker:314 Missing too many referenced descriptors (19.56182842418565). 2021-11-16 03:37:31,828 WARN o.t.m.c.r.ReferenceChecker:314 Missing too many referenced descriptors (10.710245013996218). 2021-11-16 04:07:49,363 WARN o.t.m.c.r.ReferenceChecker:314 Missing too many referenced descriptors (17.878988098306095). 2021-11-16 04:37:46,783 WARN o.t.m.c.r.ReferenceChecker:314 Missing too many referenced descriptors (7.678426709011057). 2021-11-16 05:08:33,858 WARN o.t.m.c.r.ReferenceChecker:314 Missing too many referenced descriptors (11.66398701095097). 2021-11-16 05:37:58,258 WARN o.t.m.c.r.ReferenceChecker:314 Missing too many referenced descriptors (7.898620790843306). 2021-11-16 06:08:10,407 WARN o.t.m.c.r.ReferenceChecker:314 Missing too many referenced descriptors (16.112215613555065).

1 0

Cron <collector@colchicifolium> cd /srv/collector.torproject.org/collector/logs/ && cat metrics.log | grep -E "WARN|ERROR|FATAL" | grep -v "Overwriting existing descriptor file"
by root＠colchicifolium.torproject.org 16 Nov '21

16 Nov '21

2021-11-16 00:07:46,655 WARN o.t.m.c.r.ReferenceChecker:314 Missing too many referenced descriptors (35.66600268841573).

1 0

Cron <metrics@meronense> cd /srv/metrics.torproject.org/metrics/work/modules/logs && cat metrics.log | grep -v "Could not insert bandwidth "
by root＠meronense.torproject.org 16 Nov '21

16 Nov '21

2021-11-15 08:40:02,177 INFO o.t.m.s.main.Main:32 Starting metrics update run. 2021-11-15 08:40:02,243 INFO o.t.m.s.main.Main:70 Starting org.torproject.metrics.stats.collectdescs.Main module. 2021-11-15 08:40:02,245 INFO o.t.d.i.DescriptorIndexCollector:47 Starting descriptor collection. 2021-11-15 08:40:02,257 INFO o.t.d.i.DescriptorIndexCollector:63 Indexing local directory /srv/metrics.torproject.org/metrics/work/shared/in. 2021-11-15 08:40:06,951 INFO o.t.d.i.DescriptorIndexCollector:75 Fetching remote index file https://collector.torproject.org/index/index.json. 2021-11-15 08:40:07,768 INFO o.t.d.i.DescriptorIndexCollector:84 Fetching remote files from https://collector.torproject.org. 2021-11-15 08:40:57,832 INFO o.t.d.i.DescriptorIndexCollector:90 Deleting extraneous files from local directory /srv/metrics.torproject.org/metrics/work/shared/in. 2021-11-15 08:40:58,167 INFO o.t.d.i.DescriptorIndexCollector:95 Finished descriptor collection. 2021-11-15 08:40:58,167 INFO o.t.m.s.main.Main:73 Completed org.torproject.metrics.stats.collectdescs.Main module. 2021-11-15 08:40:58,167 INFO o.t.m.s.main.Main:70 Starting org.torproject.metrics.stats.connbidirect.Main module. 2021-11-15 08:41:07,449 INFO o.t.m.s.main.Main:73 Completed org.torproject.metrics.stats.connbidirect.Main module. 2021-11-15 08:41:07,449 INFO o.t.m.s.main.Main:70 Starting org.torproject.metrics.stats.onionperf.Main module. 2021-11-15 08:41:07,452 INFO o.t.m.s.o.Main:46 Starting onionperf module. 2021-11-15 08:41:07,453 INFO o.t.m.s.o.Main:67 Connecting to database. 2021-11-15 08:41:07,880 INFO o.t.m.s.o.Main:70 Successfully connected to database. 2021-11-15 08:51:01,150 INFO o.t.m.s.o.Main:261 Querying statistics including partials from database. 2021-11-15 08:51:41,624 INFO o.t.m.s.o.Main:402 Writing 58879 lines to /srv/metrics.torproject.org/metrics/work/modules/onionperf/stats/onionperf-including-partials.csv. 2021-11-15 08:51:42,356 INFO o.t.m.s.o.Main:286 Querying timeout and failure statistics from database. 2021-11-15 08:51:44,973 INFO o.t.m.s.o.Main:402 Writing 19631 lines to /srv/metrics.torproject.org/metrics/work/modules/onionperf/stats/onionperf-failures.csv. 2021-11-15 08:51:44,989 INFO o.t.m.s.o.Main:309 Querying buildtime statistics from database. 2021-11-15 08:52:25,281 INFO o.t.m.s.o.Main:402 Writing 47548 lines to /srv/metrics.torproject.org/metrics/work/modules/onionperf/stats/buildtimes.csv. 2021-11-15 08:52:25,608 INFO o.t.m.s.o.Main:332 Querying latency statistics from database. 2021-11-15 08:52:37,028 INFO o.t.m.s.o.Main:402 Writing 18664 lines to /srv/metrics.torproject.org/metrics/work/modules/onionperf/stats/latencies.csv. 2021-11-15 08:52:37,031 INFO o.t.m.s.o.Main:357 Querying throughput statistics from database. 2021-11-15 08:52:48,334 INFO o.t.m.s.o.Main:402 Writing 19631 lines to /srv/metrics.torproject.org/metrics/work/modules/onionperf/stats/onionperf-throughput.csv. 2021-11-15 08:52:48,337 INFO o.t.m.s.o.Main:409 Disconnecting from database. 2021-11-15 08:52:48,337 INFO o.t.m.s.o.Main:62 Terminated onionperf module. 2021-11-15 08:52:48,337 INFO o.t.m.s.main.Main:73 Completed org.torproject.metrics.stats.onionperf.Main module. 2021-11-15 08:52:48,337 INFO o.t.m.s.main.Main:70 Starting org.torproject.metrics.stats.bwhist.Main module. 2021-11-15 08:52:48,337 INFO o.t.m.s.bwhist.Main:37 Starting bwhist module. 2021-11-15 08:52:48,337 INFO o.t.m.s.bwhist.Main:39 Reading descriptors and inserting relevant parts into the database. 2021-11-15 09:06:24,553 INFO o.t.m.s.b.RelayDescriptorDatabaseImporter:543 Finished importing relay descriptors: 147617 network status entries and 125734 bandwidth history elements 2021-11-15 09:06:27,783 INFO o.t.m.s.bwhist.Main:50 Aggregating database entries. 2021-11-15 09:07:48,037 INFO o.t.m.s.b.RelayDescriptorDatabaseImporter:543 Finished importing relay descriptors: 147617 network status entries and 125734 bandwidth history elements 2021-11-15 09:07:51,362 INFO o.t.m.s.bwhist.Main:53 Querying aggregated statistics from the database. 2021-11-15 09:07:51,785 INFO o.t.m.s.bwhist.Main:57 Closing database connection. 2021-11-15 09:07:51,785 INFO o.t.m.s.bwhist.Main:60 Terminating bwhist module. 2021-11-15 09:07:51,785 INFO o.t.m.s.main.Main:73 Completed org.torproject.metrics.stats.bwhist.Main module. 2021-11-15 09:07:51,785 INFO o.t.m.s.main.Main:70 Starting org.torproject.metrics.stats.advbwdist.Main module. 2021-11-15 09:08:54,841 INFO o.t.m.s.main.Main:73 Completed org.torproject.metrics.stats.advbwdist.Main module. 2021-11-15 09:08:55,210 INFO o.t.m.s.main.Main:70 Starting org.torproject.metrics.stats.hidserv.Main module. 2021-11-15 09:08:55,211 INFO o.t.m.s.hidserv.Main:37 Initializing parser and reading parse history... 2021-11-15 09:08:55,548 INFO o.t.m.s.hidserv.Main:52 Parsing descriptors... 2021-11-15 09:10:39,287 INFO o.t.m.s.hidserv.Main:59 Writing parse history... 2021-11-15 09:10:39,811 INFO o.t.m.s.hidserv.Main:66 Extrapolating statistics... 2021-11-15 09:16:41,815 INFO o.t.m.s.hidserv.Main:82 Aggregating v2 statistics... 2021-11-15 09:19:09,755 INFO o.t.m.s.hidserv.Main:98 Aggregating v3 statistics... 2021-11-15 09:19:10,022 INFO o.t.m.s.hidserv.Main:108 Terminating. 2021-11-15 09:19:10,022 INFO o.t.m.s.main.Main:73 Completed org.torproject.metrics.stats.hidserv.Main module. 2021-11-15 09:19:10,023 INFO o.t.m.s.main.Main:70 Starting org.torproject.metrics.stats.clients.Main module. 2021-11-15 09:19:10,049 INFO o.t.m.s.clients.Main:36 Starting clients module. 2021-11-15 09:19:10,049 INFO o.t.m.s.clients.Main:38 Connecting to database. 2021-11-15 09:19:10,375 INFO o.t.m.s.clients.Main:41 Reading relay descriptors and importing relevant parts into the database. 2021-11-15 09:22:58,099 INFO o.t.m.s.clients.Main:45 Reading bridge descriptors and importing relevant parts into the database. 2021-11-15 09:24:22,016 INFO o.t.m.s.clients.Main:49 Processing newly imported data. 2021-11-15 10:41:16,643 INFO o.t.m.s.clients.Main:53 Querying aggregated statistics from the database. 2021-11-15 10:41:35,981 INFO o.t.m.s.clients.Main:59 Disconnecting from database. 2021-11-15 10:41:36,078 INFO o.t.m.s.clients.Main:62 Running detector. 2021-11-15 10:42:18,348 INFO o.t.m.s.clients.Main:65 Terminating clients module. 2021-11-15 10:42:18,388 INFO o.t.m.s.main.Main:73 Completed org.torproject.metrics.stats.clients.Main module. 2021-11-15 10:42:18,389 INFO o.t.m.s.main.Main:70 Starting org.torproject.metrics.stats.servers.Main module. 2021-11-15 10:42:18,399 INFO o.t.m.s.servers.Main:43 Starting servers module. 2021-11-15 10:42:18,399 INFO o.t.m.s.servers.Main:45 Reading descriptors and inserting relevant parts into the database. 2021-11-15 10:58:17,973 INFO o.t.m.s.servers.Main:77 Aggregating database entries. 2021-11-15 11:34:45,866 INFO o.t.m.s.servers.Main:80 Committing all updated parts in the database. 2021-11-15 11:34:46,822 INFO o.t.m.s.servers.Main:90 Querying aggregated statistics from the database. 2021-11-15 11:54:58,189 INFO o.t.m.s.servers.Main:105 Terminating servers module. 2021-11-15 11:54:59,140 INFO o.t.m.s.main.Main:73 Completed org.torproject.metrics.stats.servers.Main module. 2021-11-15 11:54:59,140 INFO o.t.m.s.main.Main:70 Starting org.torproject.metrics.stats.webstats.Main module. 2021-11-15 11:54:59,213 INFO o.t.m.s.w.Main:75 Starting webstats module. 2021-11-15 11:54:59,213 INFO o.t.m.s.w.Main:92 Connecting to database. 2021-11-15 11:54:59,349 INFO o.t.m.s.w.Main:95 Successfully connected to database. 2021-11-15 11:54:59,349 INFO o.t.m.s.w.Main:101 Querying previously imported log files. 2021-11-15 11:55:09,154 INFO o.t.m.s.w.Main:113 Found 283713 previously imported log files. 2021-11-15 12:08:01,500 INFO o.t.m.s.w.Main:268 Querying statistics from database. 2021-11-15 12:30:15,940 INFO o.t.m.s.w.Main:298 Writing 1411956 lines to /srv/metrics.torproject.org/metrics/work/modules/webstats/stats/webstats.csv. 2021-11-15 12:30:16,906 INFO o.t.m.s.w.Main:305 Disconnecting from database. 2021-11-15 12:30:16,907 INFO o.t.m.s.w.Main:87 Terminated webstats module. 2021-11-15 12:30:16,907 INFO o.t.m.s.main.Main:73 Completed org.torproject.metrics.stats.webstats.Main module. 2021-11-15 12:30:16,907 INFO o.t.m.s.main.Main:70 Starting org.torproject.metrics.stats.totalcw.Main module. 2021-11-15 12:30:16,908 INFO o.t.m.s.totalcw.Main:38 Starting totalcw module. 2021-11-15 12:30:16,908 INFO o.t.m.s.totalcw.Main:40 Reading consensuses and votes and inserting relevant parts into the database. 2021-11-15 12:32:32,871 INFO o.t.m.s.totalcw.Main:64 Committing all updated parts in the database. 2021-11-15 12:32:32,959 INFO o.t.m.s.totalcw.Main:74 Querying aggregated statistics from the database. 2021-11-15 12:32:36,816 INFO o.t.m.s.totalcw.Main:77 Writing aggregated statistics to /srv/metrics.torproject.org/metrics/work/modules/totalcw/stats/totalcw.csv. 2021-11-15 12:32:37,408 INFO o.t.m.s.totalcw.Main:82 Terminating totalcw module. 2021-11-15 12:32:37,413 INFO o.t.m.s.main.Main:73 Completed org.torproject.metrics.stats.totalcw.Main module. 2021-11-15 12:32:37,413 INFO o.t.m.s.main.Main:70 Starting org.torproject.metrics.stats.bridgedb.Main module. 2021-11-15 12:32:37,818 INFO o.t.m.s.main.Main:73 Completed org.torproject.metrics.stats.bridgedb.Main module. 2021-11-15 12:32:37,818 INFO o.t.m.s.main.Main:82 Making module data available. 2021-11-15 12:32:48,569 INFO o.t.m.s.main.Main:126 Successfully copied 22 files to stats output directory: [connbidirect2.csv, onionperf-failures.csv, buildtimes.csv, latencies.csv, torperf-1.1.csv, onionperf-throughput.csv, onionperf-including-partials.csv, bandwidth.csv, advbwdist.csv, v3hidserv.csv, hidserv.csv, clients.csv, userstats-combined.csv, platforms.csv, relayflags.csv, versions.csv, advbw.csv, networksize.csv, ipv6servers.csv, webstats.csv, totalcw.csv, bridgedb-stats.csv] 2021-11-15 12:32:48,569 INFO o.t.m.s.main.Main:130 Completed metrics update run.

1 0

Cron <metrics@meronense> cd /srv/metrics.torproject.org/metrics/work/modules/logs && cat metrics.log | grep -v "Could not insert bandwidth "
by root＠meronense.torproject.org 15 Nov '21

15 Nov '21

2021-11-15 08:40:02,177 INFO o.t.m.s.main.Main:32 Starting metrics update run. 2021-11-15 08:40:02,243 INFO o.t.m.s.main.Main:70 Starting org.torproject.metrics.stats.collectdescs.Main module. 2021-11-15 08:40:02,245 INFO o.t.d.i.DescriptorIndexCollector:47 Starting descriptor collection. 2021-11-15 08:40:02,257 INFO o.t.d.i.DescriptorIndexCollector:63 Indexing local directory /srv/metrics.torproject.org/metrics/work/shared/in. 2021-11-15 08:40:06,951 INFO o.t.d.i.DescriptorIndexCollector:75 Fetching remote index file https://collector.torproject.org/index/index.json. 2021-11-15 08:40:07,768 INFO o.t.d.i.DescriptorIndexCollector:84 Fetching remote files from https://collector.torproject.org. 2021-11-15 08:40:57,832 INFO o.t.d.i.DescriptorIndexCollector:90 Deleting extraneous files from local directory /srv/metrics.torproject.org/metrics/work/shared/in. 2021-11-15 08:40:58,167 INFO o.t.d.i.DescriptorIndexCollector:95 Finished descriptor collection. 2021-11-15 08:40:58,167 INFO o.t.m.s.main.Main:73 Completed org.torproject.metrics.stats.collectdescs.Main module. 2021-11-15 08:40:58,167 INFO o.t.m.s.main.Main:70 Starting org.torproject.metrics.stats.connbidirect.Main module. 2021-11-15 08:41:07,449 INFO o.t.m.s.main.Main:73 Completed org.torproject.metrics.stats.connbidirect.Main module. 2021-11-15 08:41:07,449 INFO o.t.m.s.main.Main:70 Starting org.torproject.metrics.stats.onionperf.Main module. 2021-11-15 08:41:07,452 INFO o.t.m.s.o.Main:46 Starting onionperf module. 2021-11-15 08:41:07,453 INFO o.t.m.s.o.Main:67 Connecting to database. 2021-11-15 08:41:07,880 INFO o.t.m.s.o.Main:70 Successfully connected to database. 2021-11-15 08:51:01,150 INFO o.t.m.s.o.Main:261 Querying statistics including partials from database. 2021-11-15 08:51:41,624 INFO o.t.m.s.o.Main:402 Writing 58879 lines to /srv/metrics.torproject.org/metrics/work/modules/onionperf/stats/onionperf-including-partials.csv. 2021-11-15 08:51:42,356 INFO o.t.m.s.o.Main:286 Querying timeout and failure statistics from database. 2021-11-15 08:51:44,973 INFO o.t.m.s.o.Main:402 Writing 19631 lines to /srv/metrics.torproject.org/metrics/work/modules/onionperf/stats/onionperf-failures.csv. 2021-11-15 08:51:44,989 INFO o.t.m.s.o.Main:309 Querying buildtime statistics from database. 2021-11-15 08:52:25,281 INFO o.t.m.s.o.Main:402 Writing 47548 lines to /srv/metrics.torproject.org/metrics/work/modules/onionperf/stats/buildtimes.csv. 2021-11-15 08:52:25,608 INFO o.t.m.s.o.Main:332 Querying latency statistics from database. 2021-11-15 08:52:37,028 INFO o.t.m.s.o.Main:402 Writing 18664 lines to /srv/metrics.torproject.org/metrics/work/modules/onionperf/stats/latencies.csv. 2021-11-15 08:52:37,031 INFO o.t.m.s.o.Main:357 Querying throughput statistics from database. 2021-11-15 08:52:48,334 INFO o.t.m.s.o.Main:402 Writing 19631 lines to /srv/metrics.torproject.org/metrics/work/modules/onionperf/stats/onionperf-throughput.csv. 2021-11-15 08:52:48,337 INFO o.t.m.s.o.Main:409 Disconnecting from database. 2021-11-15 08:52:48,337 INFO o.t.m.s.o.Main:62 Terminated onionperf module. 2021-11-15 08:52:48,337 INFO o.t.m.s.main.Main:73 Completed org.torproject.metrics.stats.onionperf.Main module. 2021-11-15 08:52:48,337 INFO o.t.m.s.main.Main:70 Starting org.torproject.metrics.stats.bwhist.Main module. 2021-11-15 08:52:48,337 INFO o.t.m.s.bwhist.Main:37 Starting bwhist module. 2021-11-15 08:52:48,337 INFO o.t.m.s.bwhist.Main:39 Reading descriptors and inserting relevant parts into the database. 2021-11-15 09:06:24,553 INFO o.t.m.s.b.RelayDescriptorDatabaseImporter:543 Finished importing relay descriptors: 147617 network status entries and 125734 bandwidth history elements 2021-11-15 09:06:27,783 INFO o.t.m.s.bwhist.Main:50 Aggregating database entries. 2021-11-15 09:07:48,037 INFO o.t.m.s.b.RelayDescriptorDatabaseImporter:543 Finished importing relay descriptors: 147617 network status entries and 125734 bandwidth history elements 2021-11-15 09:07:51,362 INFO o.t.m.s.bwhist.Main:53 Querying aggregated statistics from the database. 2021-11-15 09:07:51,785 INFO o.t.m.s.bwhist.Main:57 Closing database connection. 2021-11-15 09:07:51,785 INFO o.t.m.s.bwhist.Main:60 Terminating bwhist module. 2021-11-15 09:07:51,785 INFO o.t.m.s.main.Main:73 Completed org.torproject.metrics.stats.bwhist.Main module. 2021-11-15 09:07:51,785 INFO o.t.m.s.main.Main:70 Starting org.torproject.metrics.stats.advbwdist.Main module. 2021-11-15 09:08:54,841 INFO o.t.m.s.main.Main:73 Completed org.torproject.metrics.stats.advbwdist.Main module. 2021-11-15 09:08:55,210 INFO o.t.m.s.main.Main:70 Starting org.torproject.metrics.stats.hidserv.Main module. 2021-11-15 09:08:55,211 INFO o.t.m.s.hidserv.Main:37 Initializing parser and reading parse history... 2021-11-15 09:08:55,548 INFO o.t.m.s.hidserv.Main:52 Parsing descriptors... 2021-11-15 09:10:39,287 INFO o.t.m.s.hidserv.Main:59 Writing parse history... 2021-11-15 09:10:39,811 INFO o.t.m.s.hidserv.Main:66 Extrapolating statistics... 2021-11-15 09:16:41,815 INFO o.t.m.s.hidserv.Main:82 Aggregating v2 statistics... 2021-11-15 09:19:09,755 INFO o.t.m.s.hidserv.Main:98 Aggregating v3 statistics... 2021-11-15 09:19:10,022 INFO o.t.m.s.hidserv.Main:108 Terminating. 2021-11-15 09:19:10,022 INFO o.t.m.s.main.Main:73 Completed org.torproject.metrics.stats.hidserv.Main module. 2021-11-15 09:19:10,023 INFO o.t.m.s.main.Main:70 Starting org.torproject.metrics.stats.clients.Main module. 2021-11-15 09:19:10,049 INFO o.t.m.s.clients.Main:36 Starting clients module. 2021-11-15 09:19:10,049 INFO o.t.m.s.clients.Main:38 Connecting to database. 2021-11-15 09:19:10,375 INFO o.t.m.s.clients.Main:41 Reading relay descriptors and importing relevant parts into the database. 2021-11-15 09:22:58,099 INFO o.t.m.s.clients.Main:45 Reading bridge descriptors and importing relevant parts into the database. 2021-11-15 09:24:22,016 INFO o.t.m.s.clients.Main:49 Processing newly imported data. 2021-11-15 10:41:16,643 INFO o.t.m.s.clients.Main:53 Querying aggregated statistics from the database. 2021-11-15 10:41:35,981 INFO o.t.m.s.clients.Main:59 Disconnecting from database. 2021-11-15 10:41:36,078 INFO o.t.m.s.clients.Main:62 Running detector. 2021-11-15 10:42:18,348 INFO o.t.m.s.clients.Main:65 Terminating clients module. 2021-11-15 10:42:18,388 INFO o.t.m.s.main.Main:73 Completed org.torproject.metrics.stats.clients.Main module. 2021-11-15 10:42:18,389 INFO o.t.m.s.main.Main:70 Starting org.torproject.metrics.stats.servers.Main module. 2021-11-15 10:42:18,399 INFO o.t.m.s.servers.Main:43 Starting servers module. 2021-11-15 10:42:18,399 INFO o.t.m.s.servers.Main:45 Reading descriptors and inserting relevant parts into the database. 2021-11-15 10:58:17,973 INFO o.t.m.s.servers.Main:77 Aggregating database entries. 2021-11-15 11:34:45,866 INFO o.t.m.s.servers.Main:80 Committing all updated parts in the database. 2021-11-15 11:34:46,822 INFO o.t.m.s.servers.Main:90 Querying aggregated statistics from the database. 2021-11-15 11:54:58,189 INFO o.t.m.s.servers.Main:105 Terminating servers module. 2021-11-15 11:54:59,140 INFO o.t.m.s.main.Main:73 Completed org.torproject.metrics.stats.servers.Main module. 2021-11-15 11:54:59,140 INFO o.t.m.s.main.Main:70 Starting org.torproject.metrics.stats.webstats.Main module. 2021-11-15 11:54:59,213 INFO o.t.m.s.w.Main:75 Starting webstats module. 2021-11-15 11:54:59,213 INFO o.t.m.s.w.Main:92 Connecting to database. 2021-11-15 11:54:59,349 INFO o.t.m.s.w.Main:95 Successfully connected to database. 2021-11-15 11:54:59,349 INFO o.t.m.s.w.Main:101 Querying previously imported log files. 2021-11-15 11:55:09,154 INFO o.t.m.s.w.Main:113 Found 283713 previously imported log files. 2021-11-15 12:08:01,500 INFO o.t.m.s.w.Main:268 Querying statistics from database. 2021-11-15 12:30:15,940 INFO o.t.m.s.w.Main:298 Writing 1411956 lines to /srv/metrics.torproject.org/metrics/work/modules/webstats/stats/webstats.csv. 2021-11-15 12:30:16,906 INFO o.t.m.s.w.Main:305 Disconnecting from database. 2021-11-15 12:30:16,907 INFO o.t.m.s.w.Main:87 Terminated webstats module. 2021-11-15 12:30:16,907 INFO o.t.m.s.main.Main:73 Completed org.torproject.metrics.stats.webstats.Main module. 2021-11-15 12:30:16,907 INFO o.t.m.s.main.Main:70 Starting org.torproject.metrics.stats.totalcw.Main module. 2021-11-15 12:30:16,908 INFO o.t.m.s.totalcw.Main:38 Starting totalcw module. 2021-11-15 12:30:16,908 INFO o.t.m.s.totalcw.Main:40 Reading consensuses and votes and inserting relevant parts into the database. 2021-11-15 12:32:32,871 INFO o.t.m.s.totalcw.Main:64 Committing all updated parts in the database. 2021-11-15 12:32:32,959 INFO o.t.m.s.totalcw.Main:74 Querying aggregated statistics from the database. 2021-11-15 12:32:36,816 INFO o.t.m.s.totalcw.Main:77 Writing aggregated statistics to /srv/metrics.torproject.org/metrics/work/modules/totalcw/stats/totalcw.csv. 2021-11-15 12:32:37,408 INFO o.t.m.s.totalcw.Main:82 Terminating totalcw module. 2021-11-15 12:32:37,413 INFO o.t.m.s.main.Main:73 Completed org.torproject.metrics.stats.totalcw.Main module. 2021-11-15 12:32:37,413 INFO o.t.m.s.main.Main:70 Starting org.torproject.metrics.stats.bridgedb.Main module. 2021-11-15 12:32:37,818 INFO o.t.m.s.main.Main:73 Completed org.torproject.metrics.stats.bridgedb.Main module. 2021-11-15 12:32:37,818 INFO o.t.m.s.main.Main:82 Making module data available. 2021-11-15 12:32:48,569 INFO o.t.m.s.main.Main:126 Successfully copied 22 files to stats output directory: [connbidirect2.csv, onionperf-failures.csv, buildtimes.csv, latencies.csv, torperf-1.1.csv, onionperf-throughput.csv, onionperf-including-partials.csv, bandwidth.csv, advbwdist.csv, v3hidserv.csv, hidserv.csv, clients.csv, userstats-combined.csv, platforms.csv, relayflags.csv, versions.csv, advbw.csv, networksize.csv, ipv6servers.csv, webstats.csv, totalcw.csv, bridgedb-stats.csv] 2021-11-15 12:32:48,569 INFO o.t.m.s.main.Main:130 Completed metrics update run.

1 0

Cron <collector@colchicifolium> cd /srv/collector.torproject.org/collector/logs/ && cat metrics.log | grep -E "WARN|ERROR|FATAL" | grep -v "Overwriting existing descriptor file"
by root＠colchicifolium.torproject.org 15 Nov '21

15 Nov '21

2021-11-15 00:08:00,501 WARN o.t.m.c.r.ReferenceChecker:314 Missing too many referenced descriptors (32.56627646670921). 2021-11-15 00:37:51,083 WARN o.t.m.c.r.ReferenceChecker:314 Missing too many referenced descriptors (28.814083586150318). 2021-11-15 01:07:41,669 WARN o.t.m.c.r.ReferenceChecker:314 Missing too many referenced descriptors (32.3902185640746). 2021-11-15 01:37:24,578 WARN o.t.m.c.r.ReferenceChecker:314 Missing too many referenced descriptors (31.339010148046622). 2021-11-15 02:07:43,565 WARN o.t.m.c.r.ReferenceChecker:314 Missing too many referenced descriptors (23.15378612444714). 2021-11-15 02:37:11,402 WARN o.t.m.c.r.ReferenceChecker:314 Missing too many referenced descriptors (8.112078868610054). 2021-11-15 03:07:26,191 WARN o.t.m.c.r.ReferenceChecker:314 Missing too many referenced descriptors (8.221952121488712). 2021-11-15 03:37:47,620 WARN o.t.m.c.r.ReferenceChecker:314 Missing too many referenced descriptors (6.5908252548806034). 2021-11-15 04:10:15,844 WARN o.t.m.c.r.ReferenceChecker:314 Missing too many referenced descriptors (12.142637804313617). 2021-11-15 04:37:48,399 WARN o.t.m.c.r.ReferenceChecker:314 Missing too many referenced descriptors (11.026459123407195). 2021-11-15 05:08:09,974 WARN o.t.m.c.r.ReferenceChecker:314 Missing too many referenced descriptors (23.032198541378662). 2021-11-15 05:37:53,401 WARN o.t.m.c.r.ReferenceChecker:314 Missing too many referenced descriptors (22.49415712782218). 2021-11-15 06:08:24,465 WARN o.t.m.c.r.ReferenceChecker:314 Missing too many referenced descriptors (28.109328653940988). 2021-11-15 06:38:27,462 WARN o.t.m.c.r.ReferenceChecker:314 Missing too many referenced descriptors (21.96334601987973). 2021-11-15 07:08:52,456 WARN o.t.m.c.r.ReferenceChecker:314 Missing too many referenced descriptors (31.372034705001074). 2021-11-15 07:38:49,971 WARN o.t.m.c.r.ReferenceChecker:314 Missing too many referenced descriptors (28.637196040266748). 2021-11-15 08:09:45,938 WARN o.t.m.c.r.ReferenceChecker:314 Missing too many referenced descriptors (40.19111006343131). 2021-11-15 08:37:49,942 WARN o.t.m.c.r.ReferenceChecker:314 Missing too many referenced descriptors (39.79814539235661). 2021-11-15 09:09:08,262 WARN o.t.m.c.r.ReferenceChecker:314 Missing too many referenced descriptors (44.03425458894832). 2021-11-15 09:37:26,389 WARN o.t.m.c.r.ReferenceChecker:314 Missing too many referenced descriptors (39.58035018266842). 2021-11-15 10:08:06,176 WARN o.t.m.c.r.ReferenceChecker:314 Missing too many referenced descriptors (45.042925202795615). 2021-11-15 10:37:27,732 WARN o.t.m.c.r.ReferenceChecker:314 Missing too many referenced descriptors (23.231536840907467). 2021-11-15 11:07:54,325 WARN o.t.m.c.r.ReferenceChecker:314 Missing too many referenced descriptors (31.11412395077658). 2021-11-15 11:37:27,044 WARN o.t.m.c.r.ReferenceChecker:314 Missing too many referenced descriptors (29.71619174417227). 2021-11-15 12:07:47,830 WARN o.t.m.c.r.ReferenceChecker:314 Missing too many referenced descriptors (37.69301410919504). 2021-11-15 12:37:40,437 WARN o.t.m.c.r.ReferenceChecker:314 Missing too many referenced descriptors (28.47607354530737). 2021-11-15 13:07:45,006 WARN o.t.m.c.r.ReferenceChecker:314 Missing too many referenced descriptors (38.815301175824935). 2021-11-15 13:37:52,666 WARN o.t.m.c.r.ReferenceChecker:314 Missing too many referenced descriptors (19.8213985729944). 2021-11-15 14:07:33,771 WARN o.t.m.c.r.ReferenceChecker:314 Missing too many referenced descriptors (26.194476859489157). 2021-11-15 14:37:21,518 WARN o.t.m.c.r.ReferenceChecker:314 Missing too many referenced descriptors (25.256459265884107). 2021-11-15 15:07:48,413 WARN o.t.m.c.r.ReferenceChecker:314 Missing too many referenced descriptors (32.59525134599692). 2021-11-15 15:37:47,506 WARN o.t.m.c.r.ReferenceChecker:314 Missing too many referenced descriptors (15.409581355265557). 2021-11-15 16:07:19,260 WARN o.t.m.c.r.ReferenceChecker:314 Missing too many referenced descriptors (28.148057172785037). 2021-11-15 16:37:38,319 WARN o.t.m.c.r.ReferenceChecker:314 Missing too many referenced descriptors (27.67230566977899). 2021-11-15 17:07:28,152 WARN o.t.m.c.r.ReferenceChecker:314 Missing too many referenced descriptors (38.05483940204183). 2021-11-15 17:37:30,886 WARN o.t.m.c.r.ReferenceChecker:314 Missing too many referenced descriptors (25.66781475313374). 2021-11-15 18:07:39,063 WARN o.t.m.c.r.ReferenceChecker:314 Missing too many referenced descriptors (34.286257101006946).

1 0

** RECOVERY Service Alert: global/collector is OK **
by nagios＠hetzner-hel1-01.torproject.org 02 Nov '21

02 Nov '21

***** Icinga ***** Notification Type: RECOVERY Service: collector Host: global Address: global State: OK Date/Time: Tue Nov 2 01:57:21 UTC 2021 Additional Info: OK: Got a valid response from https://collector.torproject.org/index/index.json with recent timestamps: /archive/bridgedb-metrics/ = 2021-11-01 12:14 (13 72 hours ago), /archive/bridge-descriptors/extra-infos/ = 2021-11-01 12:02 (13 72 hours ago), /archive/bridge-descriptors/server-descriptors/ = 2021-11-01 11:45 (14 72 hours ago), /archive/bridge-descriptors/statuses/ = 2021-11-01 11:32 (14 72 hours ago), /archive/bridge-pool-assignments/ = 2021-11-01 12:13 (13 72 hours ago), /archive/exit-lists/ = 2021-11-01 03:26 (22 72 hours ago), /archive/onionperf/ = 2021-11-01 03:27 (22 72 hours ago), /archive/relay-descriptors/ = 2021-11-01 03:28 (22 720 hours ago), /archive/relay-descriptors/bandwidths/ = 2021-11-01 08:24 (17 72 hours ago), /archive/relay-descriptors/consensuses/ = 2021-11-01 03:48 (22 72 hours ago), /archive/relay-descriptors/extra-infos/ = 2021-11-01 11:14 (14 72 hours ago), /archive/relay-descriptors/microdescs/ = 2021-11-01 03:36 (22 72 hours ago), /archive/r elay-descriptors/server-descriptors/ = 2021-11-01 10:39 (15 72 hours ago), /archive/relay-descriptors/votes/ = 2021-11-01 05:06 (20 72 hours ago), /archive/snowflakes/ = 2021-11-01 12:13 (13 72 hours ago), /archive/webstats/ = 2021-11-01 12:14 (13 72 hours ago), /recent/bridgedb-metrics/ = 2021-11-01 21:40 (4 60 hours ago), /recent/bridge-descriptors/extra-infos/ = 2021-11-02 01:09 (0 3 hours ago), /recent/bridge-descriptors/server-descriptors/ = 2021-11-02 01:09 (0 3 hours ago), /recent/bridge-descriptors/statuses/ = 2021-11-02 01:09 (0 3 hours ago), /recent/bridge-pool-assignments/ = 2021-11-02 00:09 (1 30 hours ago), /recent/exit-lists/ = 2021-11-02 01:02 (0 3 hours ago), /recent/onionperf/ = 2021-11-01 02:10 (23 30 hours ago), /recent/relay-descriptors/bandwidths/ = 2021-11-02 01:36 (0 3 hours ago), /recent/relay-descriptors/consensuses/ = 2021-11-02 01:05 (0 3 hours ago), /recent/relay-descriptors/extra-infos/ = 2021-11-02 01:35 (0 3 hours ago), /recent/relay-desc riptors/microdescs/consensus-microdesc/ = 2021-11-02 01:05 (0 3 hours ago), /recent/relay-descriptors/microdescs/micro/ = 2021-11-02 01:05 (0 3 hours ago), /recent/relay-descriptors/server-descriptors/ = 2021-11-02 01:35 (0 3 hours ago), /recent/relay-descriptors/votes/ = 2021-11-02 01:05 (0 3 hours ago), /recent/snowflakes/ = 2021-11-02 01:40 (0 30 hours ago), /recent/webstats/ = 2021-11-02 01:47 (0 30 hours ago)

1 0

** PROBLEM Service Alert: global/collector is CRITICAL **
by nagios＠hetzner-hel1-01.torproject.org 01 Nov '21

01 Nov '21

***** Icinga ***** Notification Type: PROBLEM Service: collector Host: global Address: global State: CRITICAL Date/Time: Mon Nov 1 10:42:19 UTC 2021 Additional Info: CRITICAL: Got a valid response from https://collector.torproject.org/index/index.json with at least one timestamp being critically outdated: /recent/snowflakes/ = 2021-10-31 01:40 (33 = 30 + 3 hours ago)

1 0

** PROBLEM Service Alert: global/collector is WARNING **
by nagios＠hetzner-hel1-01.torproject.org 01 Nov '21

01 Nov '21

***** Icinga ***** Notification Type: PROBLEM Service: collector Host: global Address: global State: WARNING Date/Time: Mon Nov 1 07:57:19 UTC 2021 Additional Info: WARNING: Got a valid response from https://collector.torproject.org/index/index.json with at least one timestamp being outdated: /recent/snowflakes/ = 2021-10-31 01:40 (30 = 30 hours ago)

1 0

** RECOVERY Service Alert: global/collector is OK **
by nagios＠hetzner-hel1-01.torproject.org 01 Nov '21

01 Nov '21

***** Icinga ***** Notification Type: RECOVERY Service: collector Host: global Address: global State: OK Date/Time: Mon Nov 1 03:47:19 UTC 2021 Additional Info: OK: Got a valid response from https://collector.torproject.org/index/index.json with recent timestamps: /archive/bridgedb-metrics/ = 2021-10-31 13:49 (13 72 hours ago), /archive/bridge-descriptors/extra-infos/ = 2021-10-31 13:36 (14 72 hours ago), /archive/bridge-descriptors/server-descriptors/ = 2021-10-31 13:03 (14 72 hours ago), /archive/bridge-descriptors/statuses/ = 2021-10-31 12:34 (15 72 hours ago), /archive/bridge-pool-assignments/ = 2021-10-31 13:49 (13 72 hours ago), /archive/exit-lists/ = 2021-10-31 03:25 (24 72 hours ago), /archive/onionperf/ = 2021-10-31 03:26 (24 72 hours ago), /archive/relay-descriptors/ = 2021-10-31 03:27 (24 720 hours ago), /archive/relay-descriptors/bandwidths/ = 2021-10-31 08:18 (19 72 hours ago), /archive/relay-descriptors/consensuses/ = 2021-10-31 03:39 (24 72 hours ago), /archive/relay-descriptors/extra-infos/ = 2021-10-31 12:14 (15 72 hours ago), /archive/relay-descriptors/microdescs/ = 2021-10-31 03:30 (24 72 hours ago), /archive/r elay-descriptors/server-descriptors/ = 2021-10-31 11:08 (16 72 hours ago), /archive/relay-descriptors/votes/ = 2021-10-31 04:48 (22 72 hours ago), /archive/snowflakes/ = 2021-10-31 13:48 (13 72 hours ago), /archive/webstats/ = 2021-10-31 13:50 (13 72 hours ago), /recent/bridgedb-metrics/ = 2021-10-31 21:40 (6 60 hours ago), /recent/bridge-descriptors/extra-infos/ = 2021-11-01 03:09 (0 3 hours ago), /recent/bridge-descriptors/server-descriptors/ = 2021-11-01 03:09 (0 3 hours ago), /recent/bridge-descriptors/statuses/ = 2021-11-01 03:09 (0 3 hours ago), /recent/bridge-pool-assignments/ = 2021-11-01 00:09 (3 30 hours ago), /recent/exit-lists/ = 2021-11-01 03:02 (0 3 hours ago), /recent/onionperf/ = 2021-11-01 02:10 (1 30 hours ago), /recent/relay-descriptors/bandwidths/ = 2021-11-01 03:43 (0 3 hours ago), /recent/relay-descriptors/consensuses/ = 2021-11-01 03:38 (0 3 hours ago), /recent/relay-descriptors/extra-infos/ = 2021-11-01 03:40 (0 3 hours ago), /recent/relay-descr iptors/microdescs/consensus-microdesc/ = 2021-11-01 03:38 (0 3 hours ago), /recent/relay-descriptors/microdescs/micro/ = 2021-11-01 03:39 (0 3 hours ago), /recent/relay-descriptors/server-descriptors/ = 2021-11-01 03:43 (0 3 hours ago), /recent/relay-descriptors/votes/ = 2021-11-01 03:38 (0 3 hours ago), /recent/snowflakes/ = 2021-10-31 01:40 (26 30 hours ago), /recent/webstats/ = 2021-11-01 01:46 (2 30 hours ago)

1 0