Discussion:
NSEC3 salt change - temporary performance decline
(too old to reply)
Niels Haarbo
2020-01-21 14:43:44 UTC
Permalink
Hello BIND users

Our DNSSEC signer changes NSEC3 salt every 30 days. The signer resigns all the relevant records and the zone is transferred using IXFR to the authoritative servers (6 nodes).

Two of the 6 authoritative servers (BIND 9.11.13 and 9.11.14) are affected by a performance decline shortly after the change of salt. This has happened after the last 3 changes of salt and the period of performance decline is within 30 - 90 minutes. Most queries are dropped by the affected nodes during the period. The normal rate is between 1.000 and 1.500 queries/second.

Other nodes running NSD and Knot are not affected.

What could be the reason for the performance decline?

Best regards

Niels Haarbo
DK Hostmaster A/S
Ondřej Surý
2020-01-21 15:40:49 UTC
Permalink
Hi Niels,
Post by Niels Haarbo
Hello BIND users
Our DNSSEC signer changes NSEC3 salt every 30 days. The signer resigns all the relevant records and the zone is transferred using IXFR to the authoritative servers (6 nodes).
Just don’t do that, there’s no sensible reason to change salt that often (or ever). I don’t know where the advice to change salt often comes from, but the advice has been wrong for so many years.
Post by Niels Haarbo
Two of the 6 authoritative servers (BIND 9.11.13 and 9.11.14) are affected by a performance decline shortly after the change of salt. This has happened after the last 3 changes of salt and the period of performance decline is within 30 – 90 minutes. Most queries are dropped by the affected nodes during the period. The normal rate is between 1.000 and 1.500 queries/second.
Other nodes running NSD and Knot are not affected.
What could be the reason for the performance decline?
We are currently investigating performance degradation related to big IXFRs. Do you use ixfr-from-differences in your BIND configuration? You could try enforcing AFRX on salt change.

This is currently tracked as https://gitlab.isc.org/isc-projects/bind9/issues/1447

and associated feature request: https://gitlab.isc.org/isc-projects/bind9/issues/1515

Ondrej
--
Ondřej Surý
***@isc.org
Daniel Stirnimann
2020-01-21 15:59:39 UTC
Permalink
Post by Ondřej Surý
Just don’t do that, there’s no sensible reason to change salt that often (or ever). I don’t know where the advice to change salt often comes from, but the advice has been wrong for so many years.
I agree that re-salting is kind of pointless (we still do it for .ch
though because so far I've been to lazy to change the code) but here is
one reference where it is recommended.

The salt SHOULD be changed periodically to prevent pre-computation
using a single salt. It is RECOMMENDED that the salt be changed for
every re-signing.

https://tools.ietf.org/html/rfc5155#appendix-C.1
Post by Ondřej Surý
Post by Niels Haarbo
What could be the reason for the performance decline?
We are currently investigating performance degradation related to big IXFRs. Do you use ixfr-from-differences in your BIND configuration? You could try enforcing AFRX on salt change.
I use "max-journal-size" to force AXFR on big changes. A good value
depends on your zone size.

Daniel
Jim Reid
2020-01-21 16:05:40 UTC
Permalink
Post by Daniel Stirnimann
I agree that re-salting is kind of pointless
So, just like NSEC3 then? :-)
Ondřej Surý
2020-01-21 17:15:48 UTC
Permalink
NSEC3 is like a toilet window. You want it translucent, not transparent. For that purpose, it serves well.

--
Ondřej Surý — ISC
Post by Jim Reid
Post by Daniel Stirnimann
I agree that re-salting is kind of pointless
So, just like NSEC3 then? :-)
_______________________________________________
Please visit https://lists.isc.org/mailman/listinfo/bind-users to unsubscribe from this list
bind-users mailing list
https://lists.isc.org/mailman/listinfo/bind-users
Niels Haarbo
2020-01-23 08:16:18 UTC
Permalink
Thank you all for the answers.

We do not use ixfr-from-differences on the actual zone, but on several others on the same server. Not sure how a BIND handles that scenario.

I will try to solve the problem by changing the max-journal-size. According to the docs https://kb.isc.org/docs/aa-01641 it cannot 'hurt' integrity to set a low value - but a value too low will affect performance.

If I can't find a solution by lowering the max-journal-size, I will disable NSEC3 salt changes.

Best regards

Niels Haarbo
DK Hostmaster A/S


-----Original Message-----
From: Ondřej Surý <***@isc.org>
Sent: Tuesday, January 21, 2020 4:41 PM
To: Niels Haarbo <***@dk-hostmaster.dk>
Cc: bind-***@lists.isc.org
Subject: Re: NSEC3 salt change - temporary performance decline

Hi Niels,
Post by Niels Haarbo
Hello BIND users
Our DNSSEC signer changes NSEC3 salt every 30 days. The signer resigns all the relevant records and the zone is transferred using IXFR to the authoritative servers (6 nodes).
Just don’t do that, there’s no sensible reason to change salt that often (or ever). I don’t know where the advice to change salt often comes from, but the advice has been wrong for so many years.
Post by Niels Haarbo
Two of the 6 authoritative servers (BIND 9.11.13 and 9.11.14) are affected by a performance decline shortly after the change of salt. This has happened after the last 3 changes of salt and the period of performance decline is within 30 – 90 minutes. Most queries are dropped by the affected nodes during the period. The normal rate is between 1.000 and 1.500 queries/second.
Other nodes running NSD and Knot are not affected.
What could be the reason for the performance decline?
We are currently investigating performance degradation related to big IXFRs. Do you use ixfr-from-differences in your BIND configuration? You could try enforcing AFRX on salt change.

This is currently tracked as https://gitlab.isc.org/isc-projects/bind9/issues/1447

and associated feature request: https://gitlab.isc.org/isc-projects/bind9/issues/1515

Ondrej
--
Ondřej Surý
***@isc.org
Klaus Darilion
2020-01-29 11:50:13 UTC
Permalink
Hello Niels!

Thanks for bringing this to attention. I have reported it before [1][2]
without response.

We see this regulary. AFAIS it happens actually always, but if the IXFR
is small, the performance decline is so short that you usually won't
notice it.

The bigger the zonechange ie NSEC3 change, full resigning ....* the
longer is the performance decline and you will notice it more often.

*we don't resalt or resign completele - but this is what several of our
TLD customers do.

I hope it will be fixed soon, we already test other software.

regards
Klaus


[1] https://lists.isc.org/pipermail/bind-users/2018-March/099814.html
[2] https://lists.isc.org/pipermail/bind-users/2019-March/101579.html
Post by Niels Haarbo
Hello BIND users
Our DNSSEC signer changes NSEC3 salt every 30 days. The signer resigns all the relevant records and the zone is transferred using IXFR to the authoritative servers (6 nodes).
Two of the 6 authoritative servers (BIND 9.11.13 and 9.11.14) are affected by a performance decline shortly after the change of salt. This has happened after the last 3 changes of salt and the period of performance decline is within 30 - 90 minutes. Most queries are dropped by the affected nodes during the period. The normal rate is between 1.000 and 1.500 queries/second.
Other nodes running NSD and Knot are not affected.
What could be the reason for the performance decline?
Best regards
Niels Haarbo
DK Hostmaster A/S
_______________________________________________
Please visit https://lists.isc.org/mailman/listinfo/bind-users to unsubscribe from this list
bind-users mailing list
https://lists.isc.org/mailman/listinfo/bind-users
Klaus Darilion
2020-01-29 11:52:07 UTC
Permalink
Post by Ondřej Surý
We are currently investigating performance degradation related to big IXFRs. Do you use ixfr-from-differences in your BIND configuration? You could try enforcing AFRX on salt change.
This is currently tracked as https://gitlab.isc.org/isc-projects/bind9/issues/1447
and associated feature request: https://gitlab.isc.org/isc-projects/bind9/issues/1515
thanks for giving priority to this issue.

Regards
Klaus
Cathy Almond
2020-06-09 12:29:30 UTC
Permalink
Post by Klaus Darilion
Hello Niels!
Thanks for bringing this to attention. I have reported it before [1][2]
without response.
We see this regulary. AFAIS it happens actually always, but if the IXFR
is small, the performance decline is so short that you usually won't
notice it.
The bigger the zonechange ie NSEC3 change, full resigning ....* the
longer is the performance decline and you will notice it more often.
*we don't resalt or resign completele - but this is what several of our
TLD customers do.
I hope it will be fixed soon, we already test other software.
regards
Klaus
[1] https://lists.isc.org/pipermail/bind-users/2018-March/099814.html
[2] https://lists.isc.org/pipermail/bind-users/2019-March/101579.html
FYI this will be fixed in the June 2020 BIND releases (in 9.11.20,
9.16.4 and 9.17.2):

https://gitlab.isc.org/isc-projects/bind9/-/issues/1834

Cathy
Klaus Darilion
2020-06-09 15:31:50 UTC
Permalink
-----Ursprüngliche Nachricht-----
Almond
Gesendet: Dienstag, 9. Juni 2020 14:30
Betreff: Re: NSEC3 salt change - temporary performance decline
...
FYI this will be fixed in the June 2020 BIND releases (in 9.11.20,
https://gitlab.isc.org/isc-projects/bind9/-/issues/1834
Great news. Thank's for the work.
Klaus

Loading...