Bind suddenly starts responding clients with servfail

[Søren Andersen]

Hello List,

I'm running a few BIND servers, but lately one of my servers suddenly starts responding to clients with servfail for every request from the clients, and BIND doesn't respond to the rndc or statistics interface anymore.

My logs for client-channel show me this:
25-Apr-2020 21:52:04.501 client @XX XX.37#2921 (google.dk<http://google.dk/>): no more recursive clients (1000/900/1000): quota reached

I've removed all the dns traffic from the server, and the quota is still reached after 6+ hours?

Do you guys have some clue what all this is about? - Or any suggestions where to look for any further information?

I'm running BIND 9.16.1 on CentOS 7:

named -V
BIND 9.16.1 (Stable Release) <id:d497c32>
running on Linux x86_64 3.10.0-1062.el7.x86_64 #1 SMP Wed Aug 7 18:08:02 UTC 2019
built by make with '--build=x86_64-redhat-linux-gnu' '--host=x86_64-redhat-linux-gnu' '--program-prefix=' '--disable-dependency-tracking' '--prefix=/opt/isc/isc-bind/root/usr' '--exec-prefix=/opt/isc/isc-bind/root/usr' '--bindir=/opt/isc/isc-bind/root/usr/bin' '--sbindir=/opt/isc/isc-bind/root/usr/sbin' '--sysconfdir=/etc/opt/isc/isc-bind' '--datadir=/opt/isc/isc-bind/root/usr/share' '--includedir=/opt/isc/isc-bind/root/usr/include' '--libdir=/opt/isc/isc-bind/root/usr/lib64' '--libexecdir=/opt/isc/isc-bind/root/usr/libexec' '--localstatedir=/var/opt/isc/isc-bind' '--sharedstatedir=/var/opt/isc/isc-bind/lib' '--mandir=/opt/isc/isc-bind/root/usr/share/man' '--infodir=/opt/isc/isc-bind/root/usr/share/info' '--disable-static' '--enable-dnstap' '--with-pic' '--with-gssapi' '--with-json-c' '--with-libtool' '--with-libxml2' '--without-lmdb' '--with-docbook-xsl=/usr/share/sgml/docbook/xsl-stylesheets' '--with-python' 'build_alias=x86_64-redhat-linux-gnu' 'host_alias=x86_64-redhat-linux-gnu' 'CFLAGS=-O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic' 'LDFLAGS= -L/opt/isc/isc-bind/root/usr/lib64' 'PKG_CONFIG_PATH=:/opt/isc/isc-bind/root/usr/lib64/pkgconfig:/opt/isc/isc-bind/root/usr/share/pkgconfig'
compiled by GCC 4.8.5 20150623 (Red Hat 4.8.5-39)
compiled with OpenSSL version: OpenSSL 1.0.2k-fips 26 Jan 2017
linked to OpenSSL version: OpenSSL 1.0.2k-fips 26 Jan 2017
compiled with libxml2 version: 2.9.1
linked to libxml2 version: 20901
compiled with json-c version: 0.11
linked to json-c version: 0.11
compiled with zlib version: 1.2.7
linked to zlib version: 1.2.7
compiled with protobuf-c version: 1.3.2
linked to protobuf-c version: 1.3.2
threads support is enabled

/Søren

[Greg Rivers]

On Monday, 27 April 2020 03:59:39 CDT Søren Andersen wrote:
> I'm running a few BIND servers, but lately one of my servers suddenly starts
> responding to clients with servfail for every request from the clients, and
> BIND doesn't respond to the rndc or statistics interface anymore.
>
> My logs for client-channel show me this:
> 25-Apr-2020 21:52:04.501 client @XX XX.37#2921 (google.dk): no more
> recursive clients (1000/900/1000): quota reached
>
> I've removed all the dns traffic from the server, and the quota is still
> reached after 6+ hours?
>
> Do you guys have some clue what all this is about? - Or any suggestions
> where to look for any further information?
>
> I'm running BIND 9.16.1 on CentOS 7:
>
I've had the very same thing happen twice in the past two weeks on different production recursive servers running BIND 9.16.2 on FreeBSD. I've opened a ticket with ISC, and they are looking into it. Can you share any additional information that might aid troubleshooting?

If anyone else experiences this, please report it.

--
Greg

[Søren Andersen]

Hi Greg,

I'm glad what I'm not the only one having this issue. Currently i've not more information that are not already mention in this mail thread.

But do you have a link to the ticket you have created?

/Søren
________________________________
From: Greg Rivers <gcr+bind-***@tharned.org>
Sent: Friday, May 8, 2020 04:03
To: Søren Andersen <***@stofa.dk>
Cc: bind-***@lists.isc.org <bind-***@lists.isc.org>
Subject: Re: Bind suddenly starts responding clients with servfail

[EXTERNAL MAIL]


On Monday, 27 April 2020 03:59:39 CDT Søren Andersen wrote:
> I'm running a few BIND servers, but lately one of my servers suddenly starts
> responding to clients with servfail for every request from the clients, and
> BIND doesn't respond to the rndc or statistics interface anymore.
>
> My logs for client-channel show me this:
> 25-Apr-2020 21:52:04.501 client @XX XX.37#2921 (google.dk): no more
> recursive clients (1000/900/1000): quota reached
>
> I've removed all the dns traffic from the server, and the quota is still
> reached after 6+ hours?
>
> Do you guys have some clue what all this is about? - Or any suggestions
> where to look for any further information?
>
> I'm running BIND 9.16.1 on CentOS 7:
>
I've had the very same thing happen twice in the past two weeks on different production recursive servers running BIND 9.16.2 on FreeBSD. I've opened a ticket with ISC, and they are looking into it. Can you share any additional information that might aid troubleshooting?

If anyone else experiences this, please report it.

--
Greg

[Greg Rivers]

On Friday, 8 May 2020 16:27:35 CDT Søren Andersen wrote:
> I'm glad what I'm not the only one having this issue. Currently i've not
> more information that are not already mention in this mail thread.
>
> But do you have a link to the ticket you have created?
>
<https://gitlab.isc.org/isc-projects/bind9/-/issues/1859>

--
Greg