Discussion:
named tcp dos?
(too old to reply)
Randy Bush
2018-07-29 14:13:37 UTC
Permalink
[ pulls head out of sand ]

so, i guess there is a named tcp dos going around. using bind9, is
there an amelioration? or am i misconfigured in some way?

randy


Jul 29 14:07:26 rip named[4146]: 29-Jul-2018 14:07:26.428 client: warning: client 67.205.183.100#60084: no more TCP clients: quota reached
Jul 29 14:07:26 rip named[4146]: 29-Jul-2018 14:07:26.431 client: warning: client 162.243.65.143#56404: no more TCP clients: quota reached
Jul 29 14:07:26 rip named[4146]: 29-Jul-2018 14:07:26.433 client: warning: client 138.68.43.154#58193: no more TCP clients: quota reached
Jul 29 14:07:26 rip named[4146]: 29-Jul-2018 14:07:26.434 client: warning: client 138.68.180.5#55238: no more TCP clients: quota reached
Jul 29 14:07:26 rip named[4146]: 29-Jul-2018 14:07:26.440 client: warning: client 45.55.10.126#34778: no more TCP clients: quota reached
Jul 29 14:07:26 rip named[4146]: 29-Jul-2018 14:07:26.441 client: warning: client 178.62.218.234#35458: no more TCP clients: quota reached
Jul 29 14:07:26 rip named[4146]: 29-Jul-2018 14:07:26.478 client: warning: client 192.241.229.202#33386: no more TCP clients: quota reached
Jul 29 14:07:26 rip named[4146]: 29-Jul-2018 14:07:26.480 client: warning: client 69.171.225.20#44623: no more TCP clients: quota reached
Jul 29 14:07:26 rip named[4146]: 29-Jul-2018 14:07:26.480 client: warning: client 178.62.218.234#34884: no more TCP clients: quota reached
Jul 29 14:07:26 rip named[4146]: 29-Jul-2018 14:07:26.487 client: warning: client 162.243.65.143#40725: no more TCP clients: quota reached
Jul 29 14:07:26 rip named[4146]: 29-Jul-2018 14:07:26.489 client: warning: client 138.68.43.154#57905: no more TCP clients: quota reached
Jul 29 14:07:26 rip named[4146]: 29-Jul-2018 14:07:26.496 client: warning: client 68.29.8.228#42665: no more TCP clients: quota reached
Jul 29 14:07:26 rip named[4146]: 29-Jul-2018 14:07:26.500 client: warning: client 159.203.79.41#55609: no more TCP clients: quota reached
Jul 29 14:07:26 rip named[4146]: 29-Jul-2018 14:07:26.501 client: warning: client 192.241.229.202#35258: no more TCP clients: quota reached
Jul 29 14:07:26 rip named[4146]: 29-Jul-2018 14:07:26.504 client: warning: client 67.205.183.100#35372: no more TCP clients: quota reached
Jul 29 14:07:26 rip named[4146]: 29-Jul-2018 14:07:26.508 client: warning: client 185.14.185.86#54087: no more TCP clients: quota reached
Jul 29 14:07:26 rip named[4146]: 29-Jul-2018 14:07:26.509 client: warning: client 138.68.43.154#35134: no more TCP clients: quota reached
Jul 29 14:07:26 rip named[4146]: 29-Jul-2018 14:07:26.510 client: warning: client 138.68.180.5#46315: no more TCP clients: quota reached
Jul 29 14:07:26 rip named[4146]: 29-Jul-2018 14:07:26.510 client: warning: client 45.55.131.229#46196: no more TCP clients: quota reached
Jul 29 14:07:26 rip named[4146]: 29-Jul-2018 14:07:26.515 client: warning: client 139.59.16.60#55659: no more TCP clients: quota reached
Jul 29 14:07:26 rip named[4146]: 29-Jul-2018 14:07:26.516 client: warning: client 162.243.85.144#49334: no more TCP clients: quota reached
Jul 29 14:07:26 rip named[4146]: 29-Jul-2018 14:07:26.517 client: warning: client 46.101.12.191#60032: no more TCP clients: quota reached
Jul 29 14:07:26 rip named[4146]: 29-Jul-2018 14:07:26.523 client: warning: client 198.211.108.251#54753: no more TCP clients: quota reached
Jul 29 14:07:26 rip named[4146]: 29-Jul-2018 14:07:26.530 client: warning: client 45.55.131.229#54741: no more TCP clients: quota reached
Jul 29 14:07:26 rip named[4146]: 29-Jul-2018 14:07:26.536 client: warning: client 198.211.108.251#34951: no more TCP clients: quota reached
Jul 29 14:07:26 rip named[4146]: 29-Jul-2018 14:07:26.536 client: warning: client 141.8.183.209#62028: no more TCP clients: quota reached
Jul 29 14:07:26 rip named[4146]: 29-Jul-2018 14:07:26.542 client: warning: client 141.8.183.209#55135: no more TCP clients: quota reached
Jul 29 14:07:26 rip named[4146]: 29-Jul-2018 14:07:26.546 client: warning: client 141.8.183.209#58573: no more TCP clients: quota reached
Jul 29 14:07:26 rip named[4146]: 29-Jul-2018 14:07:26.547 client: warning: client 139.59.61.71#38523: no more TCP clients: quota reached
Jul 29 14:07:26 rip named[4146]: 29-Jul-2018 14:07:26.549 client: warning: client 138.68.93.203#48814: no more TCP clients: quota reached
Jul 29 14:07:26 rip named[4146]: 29-Jul-2018 14:07:26.551 client: warning: client 162.243.65.143#37260: no more TCP clients: quota reached
Jul 29 14:07:26 rip named[4146]: 29-Jul-2018 14:07:26.558 client: warning: client 67.205.183.100#59649: no more TCP clients: quota reached
Jul 29 14:07:26 rip named[4146]: 29-Jul-2018 14:07:26.561 client: warning: client 192.241.229.202#40284: no more TCP clients: quota reached
Jul 29 14:07:26 rip named[4146]: 29-Jul-2018 14:07:26.562 client: warning: client 67.205.183.100#57392: no more TCP clients: quota reached
Jul 29 14:07:26 rip named[4146]: 29-Jul-2018 14:07:26.566 client: warning: client 138.68.43.154#56098: no more TCP clients: quota reached
Jul 29 14:07:26 rip named[4146]: 29-Jul-2018 14:07:26.584 client: warning: client 138.68.180.5#49567: no more TCP clients: quota reached
Jul 29 14:07:26 rip named[4146]: 29-Jul-2018 14:07:26.591 client: warning: client 138.68.43.154#33610: no more TCP clients: quota reached
Jul 29 14:07:26 rip named[4146]: 29-Jul-2018 14:07:26.603 client: warning: client 87.250.233.76#62599: no more TCP clients: quota reached
Jul 29 14:07:26 rip named[4146]: 29-Jul-2018 14:07:26.604 client: warning: client 138.68.43.154#52112: no more TCP clients: quota reached
Daniel Stirnimann
2018-08-02 06:28:49 UTC
Permalink
Hello Randy,
Post by Randy Bush
so, i guess there is a named tcp dos going around. using bind9, is
there an amelioration? or am i misconfigured in some way?
It looks to me that this is a side effect of a very permissive RRL
configuration. My tests with the following command indicate that you
have set responses-per-second to 5.

mdig @147.28.0.39 -f queries.txt

queries.txt contains 40x
switch.ch A

I would suggest something like this:

rate-limit {
// start rate-limiting if more then X identical
// responses per second, default 0 i.e. unlimited
responses-per-second 25;
nxdomains-per-second 25;
errors-per-second 25;
// credit/penalty WINDOW, default 15
window 10;
// send TC for every X-th rate-limited response, default 2
slip 1;
};

Depending on your "max-udp-size" value (default 4096) you may also want
to increase "tcp-clients" setting (default 150).

Daniel
Randy Bush
2018-08-02 19:58:32 UTC
Permalink
Post by Daniel Stirnimann
queries.txt contains 40x
switch.ch A
rate-limit {
// start rate-limiting if more then X identical
// responses per second, default 0 i.e. unlimited
responses-per-second 25;
nxdomains-per-second 25;
errors-per-second 25;
// credit/penalty WINDOW, default 15
window 10;
// send TC for every X-th rate-limited response, default 2
slip 1;
};
ok. done. thank you.
Post by Daniel Stirnimann
Depending on your "max-udp-size" value (default 4096) you may also want
to increase "tcp-clients" setting (default 150).
both are default. suggestions? are there that many folk doing tcp out
there?

randy
Greg Rivers
2018-08-02 20:05:02 UTC
Permalink
... are there that many folk doing tcp out there?
All name servers fall back to TCP when they receive truncated replies.
--
Greg Rivers
Randy Bush
2018-08-02 20:07:54 UTC
Permalink
Post by Greg Rivers
... are there that many folk doing tcp out there?
All name servers fall back to TCP when they receive truncated replies.
we know the protocol. [ and we know folk have idiot middleboxen ]

what i was asking was the distribution of this in the wild.

randy
Randy Bush
2018-08-02 20:16:13 UTC
Permalink
Post by Randy Bush
Post by Greg Rivers
... are there that many folk doing tcp out there?
All name servers fall back to TCP when they receive truncated replies.
we know the protocol. [ and we know folk have idiot middleboxen ]
what i was asking was the distribution of this in the wild
one word: DNSSEC
i.e. it is in a contest with ipv6 for non-deployment :(

let me try and phrase my question to narrow the result to a number as
opposed to a religion.

estimate or measure the distribution of the ratio of udp to tcp queries
on say 100 cctld servers.

randy
Dennis Clarke
2018-08-02 20:20:36 UTC
Permalink
Post by Randy Bush
it is in a contest with ipv6 for non-deployment
I read this mail list ALL the time and finally something appears that
quite literally made me call over a few guys to point at my screen.
Well done. Let's make up a tee-shirt with that on it :


DNSSEC? IPv6?

Which will deploy last?


Something similar .. maybe a cartoon is needed.

Dennis
Randy Bush
2018-08-02 20:27:16 UTC
Permalink
Post by Randy Bush
estimate or measure the distribution of the ratio of udp to tcp
queries on say 100 cctld servers
bla - 512 bytes are easily exceeded
more than 10 years ago i also thought i am smart and TCP 53 is only
needed for zone-transfers until i realized that random e-mail errors
where the result of large TXT records including way too verbose SPF
open TCP 53 for the world and all problems where gone forever
in my case, tcp was open and that is where the problem arose :)

i am trying to tune per daniel's
Depending on your "max-udp-size" value (default 4096) you may also
want to increase "tcp-clients" setting (default 150).
and am trying to understand the space

randy
Greg Rivers
2018-08-02 20:33:58 UTC
Permalink
Post by Randy Bush
Post by Greg Rivers
... are there that many folk doing tcp out there?
All name servers fall back to TCP when they receive truncated replies.
we know the protocol. [ and we know folk have idiot middleboxen ]
what i was asking was the distribution of this in the wild
one word: DNSSEC
Indeed, DNSSEC is a prime example. My point was that TCP queries to your servers are determined largely by the size of the RRSETs you serve. If your answers don't fit in 512 bytes (without EDNS) or ~4096 bytes (with EDNS), you're going to be serving over TCP. Obviously you're way more likely to see TCP queries from systems that don't support EDNS. Perhaps you have many such systems (and or idiot middleboxen) querying you?
--
Greg
Randy Bush
2018-08-02 20:47:21 UTC
Permalink
Post by Greg Rivers
Post by Randy Bush
Post by Greg Rivers
... are there that many folk doing tcp out there?
All name servers fall back to TCP when they receive truncated
replies.
we know the protocol. [ and we know folk have idiot middleboxen ]
what i was asking was the distribution of this in the wild
one word: DNSSEC
Indeed, DNSSEC is a prime example. My point was that TCP queries to
your servers are determined largely by the size of the RRSETs you
serve. If your answers don't fit in 512 bytes (without EDNS) or ~4096
bytes (with EDNS), you're going to be serving over TCP.
as i said, let's assume we know the protocol.
Post by Greg Rivers
Obviously you're way more likely to see TCP queries from systems that
don't support EDNS. Perhaps you have many such systems (and or idiot
middleboxen) querying you?
two $dayjobs are interfering with my trying to schedule the time to
actually measure what i am seeing on my servers. :) there are a fair
number of zones here, including a large cctld with a lot of signage.
so my guess (i.e. no real measurements [0]) is that at least that server
sees a higher tcp ratio than the average bear.,

but if i get those data, are they 'normal?' are they similar to what
others see?

randy

[0] - i confess to being a measurement researcher in one of my real
lives. so i take measurement a bit seriously. but i have not
been measuring dns for a couple of decades.
Browne, Stuart
2018-08-02 23:29:05 UTC
Permalink
-----Original Message-----
Randy Bush
Sent: Friday, 3 August 2018 6:08 AM
Post by Greg Rivers
... are there that many folk doing tcp out there?
All name servers fall back to TCP when they receive truncated replies.
we know the protocol. [ and we know folk have idiot middleboxen ]
what i was asking was the distribution of this in the wild.
We run about 300 TLD's on our DNS platform and get roughly 5-10% TCP queries. In comparison, we get about 25-30% IPv6 queries.

Those rates are fairly consistent.
randy
Stuart
Randy Bush
2018-08-03 01:13:21 UTC
Permalink
Post by Browne, Stuart
We run about 300 TLD's on our DNS platform and get roughly 5-10% TCP
queries.
that is quite a variance
Post by Browne, Stuart
In comparison, we get about 25-30% IPv6 queries.
wonder how that compares to others

thanks for actual data

randy
Daniel Stirnimann
2018-08-03 06:35:30 UTC
Permalink
Post by Randy Bush
Post by Browne, Stuart
We run about 300 TLD's on our DNS platform and get roughly 5-10% TCP
queries.
that is quite a variance
Post by Browne, Stuart
In comparison, we get about 25-30% IPv6 queries.
wonder how that compares to others
We have slightly less then 25% for IPv6 queries.
And about 4-5% TCP queries.

In our case, the default for "tcp-clients" setting is still good enough.

In BIND 9.9/9.10 it is 100 by default
In BIND 9.11/9.12 it is 150 by default.

If you want the future, you can set it to 200 ;-)

Daniel
Tony Finch
2018-08-03 11:10:04 UTC
Permalink
Post by Randy Bush
estimate or measure the distribution of the ratio of udp to tcp queries
on say 100 cctld servers.
On a recently rebooted auth server, which hosts zones for a handful of
universities with and without DNSSEC, slightly less than 1% of queries are
over TCP.

$ curl -Ssf http://authdns1.csx.cam.ac.uk:8053/json/v1 |
jq '[ .nsstats.QryUDP, .nsstats.QryTCP ]'
[
6994195,
61575
]

I have a few config options which can affect TCP usage. These two should
reduce it:

minimal-responses yes;
minimal-any yes;

These ones can increase it:

rate-limit {
responses-per-second 10;
ipv4-prefix-length 32;
exempt-clients { cudn; };
};

max-udp-size 1420;

(The latter is to avoid UDP fragmentation.)

This is not a very beefy server so I haven't increased the TCP concurrency
very much:

tcp-clients 256;

Tony.
--
f.anthony.n.finch <***@dotat.at> http://dotat.at/
Fisher, German Bight: South, veering west 3 or 4, occasionally 5 later. Smooth
or slight. Showers. Moderate or good.
Tony Finch
2018-08-03 11:19:34 UTC
Permalink
Post by Daniel Stirnimann
Post by Browne, Stuart
In comparison, we get about 25-30% IPv6 queries.
We have slightly less then 25% for IPv6 queries.
Hmm, I have 20% on one server and 22% on another.

Tony.
--
f.anthony.n.finch <***@dotat.at> http://dotat.at/
a fair voting system for all elections
Matus UHLAR - fantomas
2018-08-03 12:17:41 UTC
Permalink
Post by Tony Finch
Post by Randy Bush
estimate or measure the distribution of the ratio of udp to tcp queries
on say 100 cctld servers.
On a recently rebooted auth server, which hosts zones for a handful of
universities with and without DNSSEC, slightly less than 1% of queries are
over TCP.
$ curl -Ssf http://authdns1.csx.cam.ac.uk:8053/json/v1 |
jq '[ .nsstats.QryUDP, .nsstats.QryTCP ]'
[
6994195,
61575
]
I have a few config options which can affect TCP usage. These two should
minimal-responses yes;
minimal-any yes;
I don't think so. minimal-responses only skip unnecessary info, so they
should have no effect on TCP retries.
--
Matus UHLAR - fantomas, ***@fantomas.sk ; http://www.fantomas.sk/
Warning: I wish NOT to receive e-mail advertising to this address.
Varovanie: na tuto adresu chcem NEDOSTAVAT akukolvek reklamnu postu.
Windows 2000: 640 MB ought to be enough for anybody
Matus UHLAR - fantomas
2018-08-03 12:19:20 UTC
Permalink
Post by Tony Finch
Post by Daniel Stirnimann
Post by Browne, Stuart
In comparison, we get about 25-30% IPv6 queries.
We have slightly less then 25% for IPv6 queries.
Hmm, I have 20% on one server and 22% on another.
it's the max-udp-size 1420 apparently.

I set it to similar value because of problematic L3 switch in front of our
DNS servers long ago.

Should not be needed now.
--
Matus UHLAR - fantomas, ***@fantomas.sk ; http://www.fantomas.sk/
Warning: I wish NOT to receive e-mail advertising to this address.
Varovanie: na tuto adresu chcem NEDOSTAVAT akukolvek reklamnu postu.
Save the whales. Collect the whole set.
Tony Finch
2018-08-03 14:09:26 UTC
Permalink
Post by Matus UHLAR - fantomas
Post by Tony Finch
I have a few config options which can affect TCP usage. These two should
minimal-responses yes;
minimal-any yes;
I don't think so. minimal-responses only skip unnecessary info, so they
should have no effect on TCP retries.
minimal-any definitely reduces truncated responses - that's why I
implemented it :-)

Tony.
--
f.anthony.n.finch <***@dotat.at> http://dotat.at/
Cromarty, Forth: Variable, mainly west, 3 or 4, occasionally 5 for a time.
Smooth or slight. Thundery showers, fog patches. Moderate or good,
occasionally very poor.
Tony Finch
2018-08-03 14:12:23 UTC
Permalink
Post by Matus UHLAR - fantomas
it's the max-udp-size 1420 apparently.
I set it to similar value because of problematic L3 switch in front of our
DNS servers long ago.
Should not be needed now.
I don't have that because of my network (which works OK), but because of
other people's broken networks that screw up fragmented responses and so
have trouble resolving our domain names.

Tony.
--
f.anthony.n.finch <***@dotat.at> http://dotat.at/
Cromarty, Forth: Variable, mainly west, 3 or 4, occasionally 5 for a time.
Smooth or slight. Thundery showers, fog patches. Moderate or good,
occasionally very poor.
Matus UHLAR - fantomas
2018-08-03 14:47:43 UTC
Permalink
Post by Tony Finch
Post by Matus UHLAR - fantomas
Post by Tony Finch
I have a few config options which can affect TCP usage. These two should
minimal-responses yes;
minimal-any yes;
I don't think so. minimal-responses only skip unnecessary info, so they
should have no effect on TCP retries.
minimal-any definitely reduces truncated responses - that's why I
implemented it :-)
- are they so common that it does matter?

- if anyone wants to get full reply now, they do need to use TCP now, even
if the response would fit to 1420 chars...
--
Matus UHLAR - fantomas, ***@fantomas.sk ; http://www.fantomas.sk/
Warning: I wish NOT to receive e-mail advertising to this address.
Varovanie: na tuto adresu chcem NEDOSTAVAT akukolvek reklamnu postu.
A day without sunshine is like, night.
Tony Finch
2018-08-03 15:09:53 UTC
Permalink
Post by Matus UHLAR - fantomas
Post by Tony Finch
minimal-any definitely reduces truncated responses - that's why I
implemented it :-)
- are they so common that it does matter?
Well, they used to be, but Chris Thompson and I have done a lot to make
ANY queries on our servers unattractive to attackers.

Tony.
--
f.anthony.n.finch <***@dotat.at> http://dotat.at/
oppose all forms of entrenched privilege and inequality
Randy Bush
2018-08-04 17:57:49 UTC
Permalink
Post by Daniel Stirnimann
We have slightly less then 25% for IPv6 queries.
And about 4-5% TCP queries.
considering we share the load of the same non-trivial signed cctld, i
should be seeing similarly. though i am sure both of us serve a few
more <g>. and tony and hugo (the latter privately) are seeing similar,
though maybe slightly less v6. or they admit to more variance :)
Post by Daniel Stirnimann
In our case, the default for "tcp-clients" setting is still good enough.
In BIND 9.9/9.10 it is 100 by default
In BIND 9.11/9.12 it is 150 by default.
i am currently running default on 9.10
Post by Daniel Stirnimann
If you want the future, you can set it to 200 ;-)
tony's reply/advice on this is interesting. i am considering his
Post by Daniel Stirnimann
minimal-responses yes;
minimal-any yes;
but maybe i should just suck it up; tcp and tls are the wave of the dns
future.

an aside: folk seem to be in the 20% range for ipv6, while overall
backbone traffic stats are about half that. are dns caches more likely
to be v6 enabled than the average bear? yet another measurement project
for which we have no time. hi duane :)

and thanks for the real numbers. much better than, though not as
amusing as, the email i received from two frat boys who probably should
not drink and type. reminiscences of the usenet!

randy
Tony Finch
2018-08-06 11:49:48 UTC
Permalink
Post by Randy Bush
an aside: folk seem to be in the 20% range for ipv6, while overall
backbone traffic stats are about half that. are dns caches more likely
to be v6 enabled than the average bear?
I get the impression from various discussions that yes, they are. Actual
citation:

http://www.potaroo.net/ispcol/2016-10/dnsipv6.html

Tony.
--
f.anthony.n.finch <***@dotat.at> http://dotat.at/
Bailey: Southwest, becoming cyclonic 5 to 7, decreasing 4 at times. Moderate
or rough. Showers. Good, occasionally poor.
Greg Rivers
2018-08-06 22:21:35 UTC
Permalink
Post by Randy Bush
Post by Browne, Stuart
We run about 300 TLD's on our DNS platform and get roughly 5-10% TCP
queries.
that is quite a variance
Post by Browne, Stuart
In comparison, we get about 25-30% IPv6 queries.
wonder how that compares to others
On the secondaries for a Fortune 50 company with a sizeable ecommerce presence, we see ~17% of queries come in over IPv6, and ~2.5% are TCP queries. With respect to the Internet, the v6 percentage is probably low, as the servers I checked answer quite a lot of queries from internal IPv4 networks.

For grins, I turned on query logging on one server (BIND 9.11.4) for a short time and produced a histogram of the unique query attribute combinations:

$ awk '"query:"==$10 {print $(NF-1)}' /var/log/daemon.2 | sort | uniq -c | sort -rn | tee >(awk '{s+=$1}END{print s}')
38111265 -E(0)DC
4963452 -E(0)D
4784394 -
3268810 -E(0)
896136 +E(0)DC
551934 -E(0)TDC
406856 -E(0)DCV
318068 -E(0)DV
282536 -E(0)DCK
173078 -T
149780 -E(0)TD
132303 -E(0)DK
107240 -C
105752 -E(0)T
32748 -E(0)TDV
24677 +
21722 -E(0)TDCV
10958 -E(0)C
10907 +T
337 -E(0)TDCK
174 +E(0)
135 -TC
131 -E(0)TDK
98 +E(0)TDC
19 +E(0)D
18 +E(0)K
8 -E(0)TC
3 +E(0)T
54353539

FWIW, this indicates that most TCP queries come from clients that claim to support EDNS0.
--
Greg Rivers
Continue reading on narkive:
Loading...