- metrics-alerts - lists.torproject.org

Cron <collector@colchicifolium> cd /srv/collector.torproject.org/collector/logs/ && cat metrics.log | grep -E "WARN|ERROR|FATAL" | grep -v "Overwriting existing descriptor file"
by root＠colchicifolium.torproject.org 04 Dec '21

04 Dec '21

2021-12-04 00:07:49,433 WARN o.t.m.c.r.ReferenceChecker:314 Missing too many referenced descriptors (54.5593436457474). 2021-12-04 00:38:48,900 WARN o.t.m.c.r.ReferenceChecker:314 Missing too many referenced descriptors (41.748946518756135). 2021-12-04 01:07:54,150 WARN o.t.m.c.r.ReferenceChecker:314 Missing too many referenced descriptors (33.9596444962831). 2021-12-04 01:38:15,749 WARN o.t.m.c.r.ReferenceChecker:314 Missing too many referenced descriptors (31.42263205949164). 2021-12-04 02:15:18,792 WARN o.t.m.c.r.ReferenceChecker:314 Missing too many referenced descriptors (32.01320038375482). 2021-12-04 02:37:46,189 WARN o.t.m.c.r.ReferenceChecker:314 Missing too many referenced descriptors (28.115779817026688). 2021-12-04 03:08:26,858 WARN o.t.m.c.r.ReferenceChecker:314 Missing too many referenced descriptors (38.13732607999664). 2021-12-04 03:39:12,533 WARN o.t.m.c.r.ReferenceChecker:314 Missing too many referenced descriptors (17.1944829366498). 2021-12-04 04:09:18,560 WARN o.t.m.c.r.ReferenceChecker:314 Missing too many referenced descriptors (21.361454507295655). 2021-12-04 04:38:35,030 WARN o.t.m.c.r.ReferenceChecker:314 Missing too many referenced descriptors (14.588616064697042). 2021-12-04 05:10:15,486 WARN o.t.m.c.r.ReferenceChecker:314 Missing too many referenced descriptors (26.78675403699796). 2021-12-04 05:45:06,729 WARN o.t.m.c.r.ReferenceChecker:314 Missing too many referenced descriptors (22.558422241149756). 2021-12-04 06:09:48,928 WARN o.t.m.c.r.ReferenceChecker:314 Missing too many referenced descriptors (29.36210650101874). 2021-12-04 06:44:53,633 WARN o.t.m.c.r.ReferenceChecker:314 Missing too many referenced descriptors (23.38366689086582). 2021-12-04 07:11:09,522 WARN o.t.m.c.r.ReferenceChecker:314 Missing too many referenced descriptors (24.224633830277863). 2021-12-04 07:38:51,948 WARN o.t.m.c.r.ReferenceChecker:314 Missing too many referenced descriptors (20.008494157924286). 2021-12-04 08:09:53,034 WARN o.t.m.c.r.ReferenceChecker:314 Missing too many referenced descriptors (20.668660724163548). 2021-12-04 08:39:28,123 WARN o.t.m.c.r.ReferenceChecker:314 Missing too many referenced descriptors (19.567860241960314). 2021-12-04 09:13:18,335 WARN o.t.m.c.r.ReferenceChecker:314 Missing too many referenced descriptors (27.942539772244). 2021-12-04 09:39:16,209 WARN o.t.m.c.r.ReferenceChecker:314 Missing too many referenced descriptors (21.277471699646494). 2021-12-04 10:10:00,143 WARN o.t.m.c.r.ReferenceChecker:314 Missing too many referenced descriptors (25.6846553699571). 2021-12-04 10:42:56,241 WARN o.t.m.c.r.ReferenceChecker:314 Missing too many referenced descriptors (24.334648322666606). 2021-12-04 11:12:04,884 WARN o.t.m.c.r.ReferenceChecker:314 Missing too many referenced descriptors (33.28373303057834). 2021-12-04 11:37:27,835 WARN o.t.m.c.r.ReferenceChecker:314 Missing too many referenced descriptors (32.6181038184192). 2021-12-04 12:09:11,372 WARN o.t.m.c.r.ReferenceChecker:314 Missing too many referenced descriptors (39.94498698356726). 2021-12-04 12:43:42,780 WARN o.t.m.c.r.ReferenceChecker:314 Missing too many referenced descriptors (39.022560686295826). 2021-12-04 13:12:14,113 WARN o.t.m.c.r.ReferenceChecker:314 Missing too many referenced descriptors (44.58066899568952). 2021-12-04 13:37:55,540 WARN o.t.m.c.r.ReferenceChecker:314 Missing too many referenced descriptors (39.946607263891615). 2021-12-04 14:10:57,024 WARN o.t.m.c.r.ReferenceChecker:314 Missing too many referenced descriptors (48.716132052202425). 2021-12-04 14:38:12,774 WARN o.t.m.c.r.ReferenceChecker:314 Missing too many referenced descriptors (45.11342573609015). 2021-12-04 15:07:33,231 WARN o.t.m.c.r.ReferenceChecker:314 Missing too many referenced descriptors (49.09153993271919). 2021-12-04 15:41:34,945 WARN o.t.m.c.r.ReferenceChecker:314 Missing too many referenced descriptors (44.824684008181315). 2021-12-04 16:09:14,482 WARN o.t.m.c.r.ReferenceChecker:314 Missing too many referenced descriptors (40.59211771951556). 2021-12-04 16:41:58,120 WARN o.t.m.c.r.ReferenceChecker:314 Missing too many referenced descriptors (36.58668575634407). 2021-12-04 17:08:36,888 WARN o.t.m.c.r.ReferenceChecker:314 Missing too many referenced descriptors (47.1716746203715). 2021-12-04 17:38:45,682 WARN o.t.m.c.r.ReferenceChecker:314 Missing too many referenced descriptors (41.84449476852599). 2021-12-04 18:08:52,774 WARN o.t.m.c.r.ReferenceChecker:314 Missing too many referenced descriptors (49.190846850860595).

1 0

** RECOVERY Service Alert: onionoo-frontend-01/network service - onionoo full service check is OK **
by nagios＠hetzner-hel1-01.torproject.org 04 Dec '21

04 Dec '21

***** Icinga ***** Notification Type: RECOVERY Service: network service - onionoo full service check Host: onionoo-frontend-01 Address: 116.202.120.171 State: OK Date/Time: Sat Dec 4 17:39:01 UTC 2021 Additional Info: OK: Valid response with recent timestamps (3121 sec): relays_published=2021-12-04 17:00:00, bridges_published=2021-12-04 16:46:52

1 0

** RECOVERY Service Alert: onionoo-backend-02/network service - onionoo backend is OK **
by nagios＠hetzner-hel1-01.torproject.org 04 Dec '21

04 Dec '21

***** Icinga ***** Notification Type: RECOVERY Service: network service - onionoo backend Host: onionoo-backend-02 Address: 116.202.120.178 State: OK Date/Time: Sat Dec 4 17:38:31 UTC 2021 Additional Info: OK: Valid response with recent timestamps (3096 sec): relays_published=2021-12-04 17:00:00, bridges_published=2021-12-04 16:46:52

1 0

** RECOVERY Service Alert: onionoo-backend-02/network service - onionoo varnish is OK **
by nagios＠hetzner-hel1-01.torproject.org 04 Dec '21

04 Dec '21

***** Icinga ***** Notification Type: RECOVERY Service: network service - onionoo varnish Host: onionoo-backend-02 Address: 116.202.120.178 State: OK Date/Time: Sat Dec 4 17:35:41 UTC 2021 Additional Info: OK: Valid response with recent timestamps (2927 sec): relays_published=2021-12-04 17:00:00, bridges_published=2021-12-04 16:46:52

1 0

** RECOVERY Service Alert: onionoo-frontend-02/network service - onionoo varnish is OK **
by nagios＠hetzner-hel1-01.torproject.org 04 Dec '21

04 Dec '21

***** Icinga ***** Notification Type: RECOVERY Service: network service - onionoo varnish Host: onionoo-frontend-02 Address: 49.12.57.137 State: OK Date/Time: Sat Dec 4 17:31:01 UTC 2021 Additional Info: OK: Valid response with recent timestamps (2644 sec): relays_published=2021-12-04 17:00:00, bridges_published=2021-12-04 16:46:52

1 0

** RECOVERY Service Alert: onionoo-frontend-02/network service - onionoo full service check is OK **
by nagios＠hetzner-hel1-01.torproject.org 04 Dec '21

04 Dec '21

***** Icinga ***** Notification Type: RECOVERY Service: network service - onionoo full service check Host: onionoo-frontend-02 Address: 49.12.57.137 State: OK Date/Time: Sat Dec 4 17:30:31 UTC 2021 Additional Info: OK: Valid response with recent timestamps (2611 sec): relays_published=2021-12-04 17:00:00, bridges_published=2021-12-04 16:46:52

1 0

** RECOVERY Service Alert: onionoo-backend-01/network service - onionoo varnish is OK **
by nagios＠hetzner-hel1-01.torproject.org 04 Dec '21

04 Dec '21

***** Icinga ***** Notification Type: RECOVERY Service: network service - onionoo varnish Host: onionoo-backend-01 Address: 116.202.120.163 State: OK Date/Time: Sat Dec 4 17:29:41 UTC 2021 Additional Info: OK: Valid response with recent timestamps (2562 sec): relays_published=2021-12-04 17:00:00, bridges_published=2021-12-04 16:46:52

1 0

** RECOVERY Service Alert: onionoo-frontend-01/network service - onionoo varnish is OK **
by nagios＠hetzner-hel1-01.torproject.org 04 Dec '21

04 Dec '21

***** Icinga ***** Notification Type: RECOVERY Service: network service - onionoo varnish Host: onionoo-frontend-01 Address: 116.202.120.171 State: OK Date/Time: Sat Dec 4 17:26:31 UTC 2021 Additional Info: OK: Valid response with recent timestamps (2370 sec): relays_published=2021-12-04 17:00:00, bridges_published=2021-12-04 16:46:52

1 0

** RECOVERY Service Alert: onionoo-backend-01/network service - onionoo backend is OK **
by nagios＠hetzner-hel1-01.torproject.org 04 Dec '21

04 Dec '21

***** Icinga ***** Notification Type: RECOVERY Service: network service - onionoo backend Host: onionoo-backend-01 Address: 116.202.120.163 State: OK Date/Time: Sat Dec 4 17:25:51 UTC 2021 Additional Info: OK: Valid response with recent timestamps (2335 sec): relays_published=2021-12-04 17:00:00, bridges_published=2021-12-04 16:46:52

1 0

** PROBLEM Service Alert: onionoo-backend-02/network service - onionoo varnish is WARNING **
by nagios＠hetzner-hel1-01.torproject.org 04 Dec '21

04 Dec '21

***** Icinga ***** Notification Type: PROBLEM Service: network service - onionoo varnish Host: onionoo-backend-02 Address: 116.202.120.178 State: WARNING Date/Time: Sat Dec 4 17:05:41 UTC 2021 Additional Info: WARNING: Either timestamp is not recent (12193 sec): relays_published=2021-12-04 16:00:00, bridges_published=2021-12-04 13:42:26

1 0