Logs full of "eventdns: All nameservers have failed"
Hi, my logs are full of these messages: 05:54:07 [NOTICE] eventdns: Nameserver 127.0.0.1 is back up 05:54:07 [WARN] eventdns: All nameservers have failed At first I thought that the DNS of my ISP sucks, so I changed to Google Public DNS. But the warnings are still there. Google shows some older reports of this warning, where is was supposed to be an error in tor. Are there any news on this? I am wondering if the high bandwidth nodes from torservers and noisebridge also show this kind of messages and how they configured their nodes to get rid of it. For my node they come up every couple of minutes. Between fail and recover is always less than a second. Best regards, Klaus
Yes, I see them, and grown accustomed to them. Not that often though. On 03.12.2011 07:38, Klaus Layer wrote:
Hi,
my logs are full of these messages:
05:54:07 [NOTICE] eventdns: Nameserver 127.0.0.1 is back up 05:54:07 [WARN] eventdns: All nameservers have failed
At first I thought that the DNS of my ISP sucks, so I changed to Google Public DNS. But the warnings are still there.
-- Moritz Bartl https://www.torservers.net/
On Sat, Dec 3, 2011 at 4:09 AM, Moritz Bartl <moritz@torservers.net> wrote:
Yes, I see them, and grown accustomed to them. Not that often though.
On 03.12.2011 07:38, Klaus Layer wrote:
Hi,
my logs are full of these messages:
05:54:07 [NOTICE] eventdns: Nameserver 127.0.0.1 is back up 05:54:07 [WARN] eventdns: All nameservers have failed
At first I thought that the DNS of my ISP sucks, so I changed to Google Public DNS. But the warnings are still there.
-- Moritz Bartl https://www.torservers.net/
_______________________________________________
I also get them, tho not as frequent from the latest stable than I did on previous ones. However, I do not pay any attention to them anymore also as it does not appear to make any difference on the way tor works either for server or for regular usage. I might be more concerned if the time lapse was several minutes or more than 100th's of a second Jon
I see the same thing, and have a working theory: the network connection is saturated, which delays name resolution, causing the log entries. I run 2 relays on residential ISPs, limiting Tor to about 1/3 of my upload bandwidth. No problems (nameserver log entries) seen on these relays. I also run 2 relays on dedicated servers, attempting to fully-utilize a 10Mbps connection on each server. With the bandwidth configured for 900KB/sec I get a lot of these log entries. Both of these are running Unbound as a caching DNS server, and neither server is CPU- or memory-constrained. A couple days ago I changed to the BIND name server as an experiment. It seems to have helped, but is too early to say for sure. I've been resisting he reduction of the configured Tor bandwidth, but that will be the next attempt to eliminate the log entries. ------------------- # bin/tor-dns-fails.sh Oct 23 DNS failures: 30 Oct 24 DNS failures: 102 Oct 25 DNS failures: 68 Oct 26 DNS failures: 52 Oct 27 DNS failures: 77 Oct 28 DNS failures: 88 Oct 29 DNS failures: 59 Oct 30 DNS failures: 72 Oct 31 DNS failures: 119 Nov 01 DNS failures: 52 Nov 02 DNS failures: 38 Nov 03 DNS failures: 45 Nov 04 DNS failures: 45 Nov 05 DNS failures: 73 Nov 06 DNS failures: 21 Nov 07 DNS failures: 81 Nov 08 DNS failures: 57 Nov 09 DNS failures: 88 Nov 10 DNS failures: 43 Nov 11 DNS failures: 84 Nov 12 DNS failures: 45 Nov 13 DNS failures: 75 Nov 14 DNS failures: 80 Nov 15 DNS failures: 70 Nov 16 DNS failures: 47 Nov 17 DNS failures: 46 Nov 18 DNS failures: 91 Nov 19 DNS failures: 121 Nov 20 DNS failures: 85 Nov 21 DNS failures: 106 Nov 22 DNS failures: 95 Nov 23 DNS failures: 86 Nov 24 DNS failures: 113 Nov 25 DNS failures: 66 Nov 26 DNS failures: 47 Nov 27 DNS failures: 97 Nov 28 DNS failures: 40 Nov 29 DNS failures: 35 Nov 30 DNS failures: 91 Dec 01 DNS failures: 61 Dec 02 DNS failures: 9 Dec 03 DNS failures: 5 -----Original Message----- From: "Klaus Layer" <klaus.layer@gmx.de> Sent: Saturday, December 3, 2011 1:38am To: tor-relays@lists.torproject.org Subject: [tor-relays] Logs full of "eventdns: All nameservers have failed" _______________________________________________ tor-relays mailing list tor-relays@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays Hi, my logs are full of these messages: 05:54:07 [NOTICE] eventdns: Nameserver 127.0.0.1 is back up 05:54:07 [WARN] eventdns: All nameservers have failed At first I thought that the DNS of my ISP sucks, so I changed to Google Public DNS. But the warnings are still there. Google shows some older reports of this warning, where is was supposed to be an error in tor. Are there any news on this? I am wondering if the high bandwidth nodes from torservers and noisebridge also show this kind of messages and how they configured their nodes to get rid of it. For my node they come up every couple of minutes. Between fail and recover is always less than a second. Best regards, Klaus
On 03.12.2011 20:58, Steve Snyder wrote:
I see the same thing, and have a working theory: the network connection is saturated, which delays name resolution, causing the log entries.
We have a server on Gbit that only uses about 400 Mbit/s, and it still occasionally spits out these messages. I am using unbound, too. # cat /etc/resolv.conf nameserver 127.0.0.1 nameserver 8.8.8.8 nameserver 8.8.4.4 nameserver 4.2.2.6 # grep "ameserver" /var/log/tor/notices* notices0.log.1:Dec 02 13:22:08.000 [warn] eventdns: All nameservers have failed notices0.log.1:Dec 02 13:22:08.000 [notice] eventdns: Nameserver 8.8.4.4 is back up notices1.log.1:Dec 03 03:23:10.000 [warn] eventdns: All nameservers have failed notices1.log.1:Dec 03 03:23:10.000 [notice] eventdns: Nameserver 8.8.8.8 is back up notices2.log:Dec 03 07:17:46.000 [warn] eventdns: All nameservers have failed notices2.log:Dec 03 07:17:46.000 [notice] eventdns: Nameserver 4.2.2.6 is back up notices2.log.1:Dec 02 07:01:32.000 [warn] eventdns: All nameservers have failed notices2.log.1:Dec 02 07:01:32.000 [notice] eventdns: Nameserver 4.2.2.6 is back up notices2.log.1:Dec 02 09:08:55.000 [warn] eventdns: All nameservers have failed notices2.log.1:Dec 02 09:08:55.000 [notice] eventdns: Nameserver 4.2.2.6 is back up notices2.log.1:Dec 02 09:08:58.000 [warn] eventdns: All nameservers have failed notices2.log.1:Dec 02 09:08:58.000 [notice] eventdns: Nameserver 4.2.2.6 is back up notices2.log.1:Dec 02 11:26:21.000 [warn] eventdns: All nameservers have failed notices2.log.1:Dec 02 11:26:21.000 [notice] eventdns: Nameserver 4.2.2.6 is back up notices2.log.1:Dec 02 17:15:44.000 [warn] eventdns: All nameservers have failed notices2.log.1:Dec 02 17:15:45.000 [notice] eventdns: Nameserver 8.8.4.4 is back up notices2.log.1:Dec 02 20:04:43.000 [warn] eventdns: All nameservers have failed notices2.log.1:Dec 02 20:04:43.000 [notice] eventdns: Nameserver 8.8.4.4 is back up notices2.log.1:Dec 02 21:31:19.000 [warn] eventdns: All nameservers have failed notices2.log.1:Dec 02 21:31:19.000 [notice] eventdns: Nameserver 8.8.4.4 is back up notices3.log:Dec 03 09:39:41.000 [warn] eventdns: All nameservers have failed notices3.log:Dec 03 09:39:41.000 [notice] eventdns: Nameserver 8.8.8.8 is back up notices3.log.1:Dec 02 07:01:32.000 [warn] eventdns: All nameservers have failed notices3.log.1:Dec 02 07:01:32.000 [notice] eventdns: Nameserver 8.8.8.8 is back up notices3.log.1:Dec 02 09:03:11.000 [warn] eventdns: All nameservers have failed notices3.log.1:Dec 02 09:03:11.000 [notice] eventdns: Nameserver 4.2.2.6 is back up notices3.log.1:Dec 02 09:03:17.000 [warn] eventdns: All nameservers have failed notices3.log.1:Dec 02 09:03:17.000 [notice] eventdns: Nameserver 4.2.2.6 is back up notices3.log.1:Dec 02 10:44:30.000 [warn] eventdns: All nameservers have failed notices3.log.1:Dec 02 10:44:30.000 [notice] eventdns: Nameserver 8.8.8.8 is back up notices3.log.1:Dec 02 10:44:32.000 [warn] eventdns: All nameservers have failed notices3.log.1:Dec 02 10:44:32.000 [notice] eventdns: Nameserver 8.8.8.8 is back up notices3.log.1:Dec 02 12:20:12.000 [warn] eventdns: All nameservers have failed notices3.log.1:Dec 02 12:20:12.000 [notice] eventdns: Nameserver 8.8.4.4 is back up notices3.log.1:Dec 02 13:03:09.000 [warn] eventdns: All nameservers have failed notices3.log.1:Dec 02 13:03:10.000 [notice] eventdns: Nameserver 8.8.4.4 is back up notices3.log.1:Dec 02 13:50:40.000 [warn] eventdns: All nameservers have failed notices3.log.1:Dec 02 13:50:40.000 [notice] eventdns: Nameserver 8.8.8.8 is back up notices3.log.1:Dec 02 14:08:51.000 [warn] eventdns: All nameservers have failed notices3.log.1:Dec 02 14:08:51.000 [notice] eventdns: Nameserver 8.8.4.4 is back up notices3.log.1:Dec 02 14:43:09.000 [warn] eventdns: All nameservers have failed notices3.log.1:Dec 02 14:43:10.000 [notice] eventdns: Nameserver 8.8.4.4 is back up notices3.log.1:Dec 02 14:43:20.000 [warn] eventdns: All nameservers have failed notices3.log.1:Dec 02 14:43:20.000 [notice] eventdns: Nameserver 4.2.2.6 is back up notices3.log.1:Dec 02 16:16:38.000 [warn] eventdns: All nameservers have failed notices3.log.1:Dec 02 16:16:38.000 [notice] eventdns: Nameserver 4.2.2.6 is back up notices3.log.1:Dec 02 16:43:05.000 [warn] eventdns: All nameservers have failed notices3.log.1:Dec 02 16:43:05.000 [notice] eventdns: Nameserver 4.2.2.6 is back up notices3.log.1:Dec 02 17:03:33.000 [warn] eventdns: All nameservers have failed notices3.log.1:Dec 02 17:03:34.000 [notice] eventdns: Nameserver 8.8.4.4 is back up notices3.log.1:Dec 02 17:26:50.000 [warn] eventdns: All nameservers have failed notices3.log.1:Dec 02 17:26:50.000 [notice] eventdns: Nameserver 4.2.2.6 is back up notices3.log.1:Dec 02 17:34:26.000 [warn] eventdns: All nameservers have failed notices3.log.1:Dec 02 17:34:26.000 [notice] eventdns: Nameserver 8.8.4.4 is back up notices3.log.1:Dec 02 18:45:57.000 [warn] eventdns: All nameservers have failed notices3.log.1:Dec 02 18:45:57.000 [notice] eventdns: Nameserver 8.8.4.4 is back up notices3.log.1:Dec 02 22:32:45.000 [warn] eventdns: All nameservers have failed notices3.log.1:Dec 02 22:32:45.000 [notice] eventdns: Nameserver 8.8.4.4 is back up notices3.log.1:Dec 03 03:58:02.000 [warn] eventdns: All nameservers have failed notices3.log.1:Dec 03 03:58:02.000 [notice] eventdns: Nameserver 8.8.4.4 is back up -- Moritz Bartl https://www.torservers.net/
On Sat, Dec 03, 2011 at 07:38:05AM +0100, Klaus Layer wrote:
my logs are full of these messages: 05:54:07 [NOTICE] eventdns: Nameserver 127.0.0.1 is back up 05:54:07 [WARN] eventdns: All nameservers have failed [snip] I am wondering if the high bandwidth nodes from torservers and noisebridge also show this kind of messages and how they configured their nodes to get rid of it. For my node they come up every couple of minutes. Between fail and recover is always less than a second.
Yes, we do see that occasionally. Not very frequently though, and generally in spurts. Looking at the logs right now, I see a few dozen occurrences in a span of about 10 minutes on Dec 1, and a few scattered instances earlier in the logs -- a total of 56 "All nameservers have failed" messages from Nov 27 - Dec 3. Since DNS is the most frequent UDP traffic you'll see on a Tor node, perhaps this is simply a symptom of high packet loss on your NIC. We have 4 "nameserver" lines in our /etc/resolv.conf provided by our ISP. You could consider running a caching nameserver on localhost. That could have negative side effects, though; you're increasing memory and CPU load by doing so, and potentially increasing attack surface depending on your exact configuration. -andy
Andy Isaacson <adi@hexapodia.org> wrote on 03.12.2011:
Since DNS is the most frequent UDP traffic you'll see on a Tor node, perhaps this is simply a symptom of high packet loss on your NIC. It's a gigabit link, with at the moment only 30% load. I don't expect significant packet loss.
You could consider running a caching nameserver on localhost. That could have negative side effects, though; you're increasing memory and CPU load by doing so, and potentially increasing attack surface depending on your exact configuration. I am already running caching DNS. CPU is not an issue, but its eating up some memory.
Well, as these messages seems to be quite common I will just ignore them. Thanks, Klaus
On Mon, Dec 05, 2011 at 10:35:03PM +0100, Klaus Layer wrote:
Andy Isaacson <adi@hexapodia.org> wrote on 03.12.2011:
Since DNS is the most frequent UDP traffic you'll see on a Tor node, perhaps this is simply a symptom of high packet loss on your NIC.
It's a gigabit link, with at the moment only 30% load. I don't expect significant packet loss.
If it's a decent NIC (Intel or Broadcom) then I'd agree with you. If it's a RTL or other sub-par vendor / driver, then you're overly optimistic.
You could consider running a caching nameserver on localhost. That could have negative side effects, though; you're increasing memory and CPU load by doing so, and potentially increasing attack surface depending on your exact configuration.
I am already running caching DNS. CPU is not an issue, but its eating up some memory.
Well, as these messages seems to be quite common I will just ignore them.
Wait, you're seeing these DNS failures with "nameserver 127.0.0.1" in /etc/hosts? That's more interesting, since then DNS UDP lossage on GigE pacet loss is unlikely to be the cause. Could you clarify the configuration? Is Tor doing DNS over the GigE or to localhost? -andy
Andy Isaacson <adi@hexapodia.org> wrote on 10.12.2011:
If it's a decent NIC (Intel or Broadcom) then I'd agree with you. If it's a RTL or other sub-par vendor / driver, then you're overly optimistic. lspci shows: 01:00.0 Ethernet controller: Intel Corporation 82574L Gigabit Network Connection 02:00.0 Ethernet controller: Intel Corporation 82574L Gigabit Network Connection
I assume this should be a decent NIC.
Wait, you're seeing these DNS failures with "nameserver 127.0.0.1" in /etc/hosts? That's more interesting, since then DNS UDP lossage on GigE pacet loss is unlikely to be the cause.
Could you clarify the configuration? Is Tor doing DNS over the GigE or to localhost?
-andy Yes I have configure a local caching DNS server. cat /etc/resolve.conf shows nameserver 127.0.0.1. So I assume TOR resolves via the local configured DNS. I verified with nslookup. DNS requests indeed go through the 127.0.0.1:53.
Thanks, Klaus
On Saturday, December 10, 2011 5:58am, "Klaus Layer" <klaus.layer@gmx.de> said:
Could you clarify the configuration? Is Tor doing DNS over the GigE or to localhost?
-andy Yes I have configure a local caching DNS server. cat /etc/resolve.conf shows nameserver 127.0.0.1. So I assume TOR resolves via the local configured DNS. I verified with nslookup. DNS requests indeed go through the 127.0.0.1:53.
Yes, I also have a local caching nameserver, pointed to by the "nameserver 127.0.0.1" entry in /etc/resolve.conf. Also, it seems that Tor/libevent is smart enough to filter out duplicate DNS servers as 3 copies of "nameserver 127.0.0.1" doesn't give you more retries on the same server. On a semi-related topic, I can now say the using BIND v9.7 greatly reduces the number of these Tor entries relative to using Unbound v1.4.13. This is true with Tor 0.2.2.34 on the low (Intel Atom) and high (Xeon 56xx) ends of the CPU spectrum.
"Steve Snyder" <swsnyder@snydernet.net> wrote on 10.12.2011:
Yes, I also have a local caching nameserver, pointed to by the "nameserver 127.0.0.1" entry in /etc/resolve.conf. Also, it seems that Tor/libevent is smart enough to filter out duplicate DNS servers as 3 copies of "nameserver 127.0.0.1" doesn't give you more retries on the same server.
On a semi-related topic, I can now say the using BIND v9.7 greatly reduces the number of these Tor entries relative to using Unbound v1.4.13. This is true with Tor 0.2.2.34 on the low (Intel Atom) and high (Xeon 56xx) ends of the CPU spectrum.
I am running ubound. I increased the num-threads parameter in ubound.conf but that did not to help. Than I stopped unbound and installed BIND 9.7.3 but I don't see any difference. Regards, Klaus
participants (5)
-
Andy Isaacson -
Jon -
Klaus Layer -
Moritz Bartl -
Steve Snyder