fix: handle DNSSEC validation for large RSA exponents and improve DNSKEY matching

[0xERR0R]

closes #1934

[codecov]

Codecov Report

❌ Patch coverage is 41.36364% with 129 lines in your changes missing coverage. Please review.
✅ Project coverage is 90.19%. Comparing base (b776ae5) to head (14e7b7c).
⚠️ Report is 2 commits behind head on main.

Files with missing lines	Patch %	Lines
resolver/dnssec/validator.go	48.31%	39 Missing and 7 partials ⚠️
resolver/dnssec/rrset.go	45.45%	39 Missing and 3 partials ⚠️
resolver/dnssec/chain.go	12.76%	41 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1935      +/-   ##
==========================================
- Coverage   91.70%   90.19%   -1.52%     
==========================================
  Files          90       90              
  Lines        6259     6434     +175     
==========================================
+ Hits         5740     5803      +63     
- Misses        387      492     +105     
- Partials      132      139       +7     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[Copilot]

Pull Request Overview

This PR fixes DNSSEC validation issues related to large RSA exponents and improves DNSKEY matching by requiring both key tag and algorithm to match. Additionally, it relaxes wildcard validation requirements for positive responses, accepting wildcard expansions without NSEC/NSEC3 proof when cryptographic signatures are present.

Key changes:

• Enhanced DNSKEY matching to check both algorithm and key tag (preventing potential algorithm confusion attacks)
• Added detection and handling for RSA keys with exponents exceeding Go crypto's 2^31-1 limit
• Modified wildcard validation to accept positive responses without NSEC/NSEC3 proof (aligning with major validators like BIND and Unbound)

Reviewed Changes

Copilot reviewed 7 out of 7 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
resolver/dnssec/wildcard.go	Relaxes wildcard validation to accept positive responses without NSEC/NSEC3 proof, relying on cryptographic signature validation
resolver/dnssec/wildcard_test.go	Updates tests to reflect new wildcard validation behavior for positive responses
resolver/dnssec/validator.go	Refactors RRset validation to group by name+type instead of type only, adds handling for unsupported RSA exponents, improves RRSIG matching
resolver/dnssec/validator_test.go	Updates tests for new grouping logic and wildcard validation behavior
resolver/dnssec/rrset.go	Adds RSA exponent validation, enhances DNSKEY matching with algorithm check, implements RRSIG sorting by strength, adds new findMatchingRRSIGs function
resolver/dnssec/rrset_test.go	Updates tests to pass algorithm parameter to findMatchingDNSKEY and queryAndMatchDNSKEY functions
resolver/dnssec/chain.go	Adds DNSKEY RRset verification using validated KSK, implements findAndValidateKSK helper function

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull Request Overview

Copilot reviewed 7 out of 7 changed files in this pull request and generated 3 comments.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull Request Overview

Copilot reviewed 9 out of 9 changed files in this pull request and generated 5 comments.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull Request Overview

Copilot reviewed 9 out of 9 changed files in this pull request and generated 2 comments.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.