As with the earlier incident, problem came back within hours of restarting the daemons.
Was able to figure out what's happening Operators running 'unbound' take note!
Problem appears to be the result of someone attempting to DDOS a DNS service, in this case GoDaddy.
Ran
lsof -Pn -p <unbnd_pid>
a few times and observed numerous SYN_SENT TCP connections, of of them to 208.109.255.0/24, where GoDaddy DNS servers are found. Appears GoDaddy is rate-limiting or blocking requests from the 'unbound' instance on the relay IP.
Ran
unbound-control dump_requestlist
and see a large queue of requests to GoDaddy. Finally ran
unbound-control dump_infra >infralst
and see 14000 lines similar to
208.109.255.26 cycsErvicioSsAS.coM. expired rto 120000
indicating a huge number of requests have been made to GoDaddy and have expired after 120 seconds.
Presently the quantity of requests has fallen off and the exit is operating fine. Have alarmed the tell-tale log message. When it recurs I expect
unbound-control purge_requestlist
will mitigate the problem. Presently looking into configuring 'ratelimit' feature of 'unbound'. If anyone has already done this successfully please post to this thread.