LUNAROPS · OPERATIONAL UPLINK 100% UPTIME 1,247d POSTS 908 JEFF.MOON@LUNAROPS.DEV UTC --:--:--

Debugging DNS: dig, delv, and the Resolution Path

dnstroubleshootingdigdnssecednsnetworking

Almost every DNS debugging session starts with dig name.example.com, and almost every one of those starts in the wrong place. That command asks one specific recursive resolver, gets one answer, and tells you nothing about where the answer came from or whether your application would have received the same thing.

DNS failures are nearly always a question of which component is lying: a cache holding stale data, an authoritative server out of sync with its siblings, a delegation pointing somewhere obsolete, a validating resolver rejecting a signature, a firewall dropping large responses, or — surprisingly often — the application never asking DNS what you think it asked. Each of those has a specific diagnostic, and running them in the right order turns a vague outage into a located fault in about two minutes.

The Model, Before the Commands

There are two different resolution processes, and conflating them wastes most people’s first ten minutes.

Your stub resolver does something trivial: it reads /etc/resolv.conf, sends the query to a configured recursor with the rd (recursion desired) bit set, and takes whatever comes back.

The recursor does the real work: it asks a root server, follows the referral to the TLD, follows that referral to your authoritative servers, and caches everything it learns along the way.

dig by default behaves like a stub — it asks your configured recursor and prints the result. It does not show you the recursive path, and it does not tell you whether the answer came from a cache or from the source. Internalizing that single fact prevents most dead ends.

A given answer can come from three places: a cache (yours, your recursor’s, or a forwarder’s), an authoritative server, or a policy layer rewriting it (split-horizon views, Pi-hole, a corporate filter). Debugging is the process of determining which.

Reading dig Output

; <<>> DiG 9.18.24 <<>> example.com A
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 42183
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags: do; udp: 1232
;; QUESTION SECTION:
;example.com.			IN	A

;; ANSWER SECTION:
example.com.		278	IN	A	203.0.113.10

;; Query time: 4 msec
;; SERVER: 127.0.0.53#53(127.0.0.53) (UDP)
;; WHEN: Tue Oct 07 12:14:03 UTC 2026
;; MSG SIZE  rcvd: 60

The flags line carries most of the diagnostic value:

  • aa — authoritative answer. Present means you reached a server that owns the zone. Its absence when you queried an authoritative server directly is a red flag: that server does not think it is authoritative, which is a lame delegation.
  • ra — recursion available. A REFUSED with no ra means you asked a server that will not recurse for you. That’s an authoritative-only server or an access-control rule, not a broken name.
  • ad — authenticated data. The resolver validated DNSSEC. It is not proof your query path was secure, only that the resolver claims to have validated.
  • cd — checking disabled. You can set it (+cd) to ask the resolver to skip validation, which is the single best quick test for “is this a DNSSEC problem.”
  • tc — truncated. The answer didn’t fit; dig normally retries over TCP automatically.

Two status values get confused constantly. NXDOMAIN means the name does not exist. NOERROR with an empty ANSWER section — often called NODATA — means the name exists but has no records of the type you asked for. dig AAAA against an IPv4-only host returns NODATA, not NXDOMAIN, and a monitoring check that treats “no answer” as “name missing” will mislead you.

Also note the SERVER: line. 127.0.0.53 means you went through systemd-resolved’s stub, which has its own cache and its own behavior — a layer you may not have known was in the path.

Flags worth keeping in muscle memory:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
dig +short example.com A              # value only
dig +noall +answer example.com A      # answer section only, with TTL
dig +norecurse @resolver example.com  # "do you have this cached?"
dig +trace example.com                # iterate from the root yourself
dig +dnssec example.com               # include RRSIG records
dig +cd example.com                   # skip DNSSEC validation
dig +tcp example.com                  # force TCP
dig +bufsize=1232 example.com         # set the EDNS buffer size
dig -x 203.0.113.10                   # reverse lookup
dig +yaml example.com                 # structured output for scripts

What +trace Actually Does

+trace makes dig perform the iteration: it queries a root server, prints the referral, queries a TLD server, prints that referral, and continues to the authoritative answer. Each step is sent with recursion disabled.

The critical property is what it bypasses. +trace does not use your resolver’s cache, does not use your resolver’s forwarder, and may well take a different path than your resolver would. That makes the comparison diagnostic:

+trace succeeds but a normal dig fails → the authoritative data is fine and your resolver is the problem. Stale cache, a broken forwarder, DNSSEC validation failure, or policy-based blocking.

+trace fails too → the problem is in the delegation chain or at the authoritative servers, and the trace output shows you which hop.

When reading trace output, each step’s authority records are the delegation. Watch for three things: NS names with no accompanying glue addresses, NS names that themselves fail to resolve, and servers that answer without the aa flag (lame delegation — listed as authoritative, doesn’t believe it is).

+norecurse deserves its own mention as a cache inspector:

1
2
# ask a resolver what it already knows, without making it go fetch anything
dig +norecurse @10.0.0.53 example.com A

An answer means it’s cached. No answer means it isn’t. And querying twice in a row reveals caching directly — if the TTL counts down between queries, you are reading a cache and the number tells you how long until it expires.

The Decision Tree

Run these in order. Each one eliminates a layer.

1. Is it just you?

1
2
3
dig example.com A              # your configured resolver
dig @1.1.1.1 example.com A     # a public resolver
dig @8.8.8.8 example.com A     # a second public resolver

Different answers mean the problem is local: your resolver, its cache, your network, or a policy layer. Identical failures everywhere mean it’s upstream.

2. Do the authoritative servers agree?

1
2
3
4
for ns in $(dig +short NS example.com); do
  printf '%-32s ' "$ns"
  dig +short A www.example.com "@$ns"
done

Inconsistent answers mean zone transfer failure or drift — one server never got the update. This is the single highest-value loop in DNS debugging, and it catches the “works for half of users” class of report immediately.

3. Does the parent agree with the child?

1
2
3
4
# what the TLD says your nameservers are
dig +norecurse NS example.com @$(dig +short NS com. | head -1)
# what your zone says they are
dig +short NS example.com @ns1.example.com

A mismatch is the classic aftermath of a provider migration: the registrar points at the new provider, the old provider is still answering with stale data, and resolvers pick one at random. The symptom is intermittent wrong answers that no single query reproduces reliably.

4. Is it DNSSEC?

1
2
dig example.com A            # SERVFAIL
dig +cd example.com A        # NOERROR → it's a validation failure

If disabling validation fixes it, you have a DNSSEC problem and the next section is for you.

5. Is it size or transport?

1
2
3
dig +notcp +bufsize=512  TXT example.com    # small, UDP only
dig +notcp +bufsize=4096 TXT example.com    # large, UDP only — hangs?
dig +tcp TXT example.com                    # works over TCP?

Small works, large hangs, TCP works: something in the path drops fragmented UDP or large responses.

6. Is it DNS at all?

1
2
3
4
dig +short app.internal          # works
getent hosts app.internal        # fails?
resolvectl query app.internal    # what the system resolver actually does
cat /etc/resolv.conf

This step is skipped constantly and resolves a surprising fraction of “DNS is broken” reports. dig talks to a nameserver directly; applications go through the system resolver, which also consults /etc/hosts, nsswitch rules, mDNS, and search domains. If dig works and the application doesn’t, stop debugging DNS and start debugging resolver configuration.

delv and DNSSEC Failures

delv is dig’s validating sibling. It performs DNSSEC validation locally and states the verdict plainly:

1
2
3
4
5
6
7
delv example.com A
# ; fully validated
# example.com.  300  IN  A      203.0.113.10
# example.com.  300  IN  RRSIG  A 13 2 300 20261101000000 ...

delv +rtrace example.com A    # show the chain-of-trust fetches
delv +vtrace example.com A    # show each validation step

The failure modes, roughly in order of how often they cause outages:

Expired RRSIGs. Signatures have an expiry, typically days to a few weeks, and are refreshed by a re-signing job. When that job dies, nothing breaks immediately — then the signatures expire and every validating resolver on earth returns SERVFAIL while non-validating resolvers keep working perfectly. The symptom is “the site works for some people,” and it’s a silent time bomb because almost nobody monitors signature expiry the way they monitor certificate expiry. They should: it is exactly the same class of problem, and the same external-check discipline applies.

DS and DNSKEY disagreement. After a key rollover, the DS record at the parent must match a DNSKEY in the child. Publish the new key without updating the DS — or update the DS before the new key has propagated — and validation fails hard.

A missing or orphaned DS. DS present at the parent for a zone that is no longer signed makes the zone unresolvable for validators.

Quick check of the chain:

1
2
dig +short DS example.com @$(dig +short NS com. | head -1)   # what the parent publishes
dig +short DNSKEY example.com | head                          # what the zone publishes

The operational lesson generalizes beyond DNSSEC: dig +cd is your five-second discriminator, and delv tells you which link broke. Further detail on key management is in the DNSSEC post.

EDNS0, Buffers, and Fragmentation

Original DNS capped UDP responses at 512 bytes. DNSSEC signatures, long TXT records, and large NS sets blow through that routinely, so EDNS0 (RFC 6891) lets a client advertise a bigger buffer.

The failure this creates is distinctive: small answers work, large answers hang. A response that exceeds the path MTU gets fragmented, and plenty of firewalls and middleboxes drop UDP fragments. The query doesn’t fail fast — it times out, retries, maybe falls back to TCP, and if TCP port 53 is also blocked, it fails entirely. Symptomatically this reads as “DNS is broken for exactly one domain,” usually a DNSSEC-signed one.

DNS Flag Day 2020 addressed it by lowering the recommended advertised buffer to 1232 bytes, which stays under common MTUs and avoids fragmentation. Modern resolvers default there.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
# bisect the size at which the path breaks
dig +notcp +bufsize=512  DNSKEY example.com
dig +notcp +bufsize=1232 DNSKEY example.com
dig +notcp +bufsize=4096 DNSKEY example.com

# is TCP available at all? DNSSEC and AXFR depend on it
dig +tcp DNSKEY example.com

# which anycast node answered — useful when only some fail
dig +nsid example.com @ns1.example.com

Two configuration notes. On the server side, BIND’s max-udp-size and edns-udp-size control this, and setting them too high reintroduces fragmentation. On the network side, a firewall rule allowing only UDP/53 and not TCP/53 is a misconfiguration — TCP is not a legacy fallback, it is required for large answers and transfers. ISC’s ednscomp service tests a zone’s EDNS compliance from outside, which is worth running once per zone.

Negative Caching and TTL Traps

A nonexistent name gets cached too. RFC 2308 says the duration comes from the SOA minimum field of the zone that denied it — not from any record’s TTL, because there is no record.

This produces the trap that recurs through every DNS-adjacent workflow: you create a record, and anything that queried the name before it existed continues to get NXDOMAIN for the SOA minimum. A zone with a one-hour minimum means a brand-new record can appear absent for an hour to resolvers that asked early. It is the leading cause of ACME DNS-01 validation failures, and the reason clients should check against authoritative servers rather than through a recursive resolver, as covered in the DNS-01 post.

Resolvers also impose their own caps, so your intended TTL is advisory in both directions: BIND’s max-cache-ttl and max-ncache-ttl, Unbound’s cache-max-ttl and cache-max-negative-ttl. A resolver may hold data far longer or shorter than you specified.

Then there is serve-stale (RFC 8767): when authoritative servers are unreachable, a resolver may keep serving expired data rather than failing. This is good for resilience and confusing for debugging — “it still resolves” does not mean your nameservers are up. Check authoritative servers directly before concluding they’re healthy.

Flushing, by platform:

1
2
3
4
5
rndc flushname example.com                  # BIND, one name
unbound-control flush example.com           # Unbound
resolvectl flush-caches                     # systemd-resolved
dscacheutil -flushcache && sudo killall -HUP mDNSResponder   # macOS
ipconfig /flushdns                          # Windows

Remember that public resolvers have caches you cannot flush. Cloudflare and Google both offer purge tools, but the honest answer during a migration is to lower TTLs before the change, not to flush after it.

CNAME Chains

Three rules cause most CNAME trouble.

A CNAME cannot coexist with other records at the same name. If www is a CNAME, it cannot also have a TXT record. Providers that let you create both are storing data that resolvers will handle inconsistently.

A CNAME is not allowed at the zone apex, because the apex must carry SOA and NS records. Every provider’s ALIAS, ANAME, or “CNAME flattening” feature is a non-standard workaround — useful, but it means your apex configuration is not portable between providers.

MX and NS records must never point at a CNAME (RFC 2181). Mail servers and resolvers are entitled to refuse, and the resulting breakage is intermittent and provider-dependent.

Chains also cost latency. Each link may be a separate round trip, and chains that cross providers — a CNAME to a CDN that CNAMEs to a regional endpoint that CNAMEs again — add up in a way that shows on first-byte timing. Resolvers cap chain length (commonly 8 to 16 links) and return SERVFAIL beyond it.

Walk a chain explicitly when it misbehaves:

1
2
3
dig +noall +answer www.example.com CNAME
dig +noall +answer cdn.example.net CNAME
dig +noall +answer edge-eu.cdnprovider.com A

And watch for the dangling case: a chain ending in NXDOMAIN at a cloud provider’s namespace is not just broken, it’s a subdomain takeover waiting for someone to register the other end and serve content under your name. Keeping that visible is one of the stronger arguments for managing zones as code.

Four Outages and Their Signatures

“It works for some people.” Validating resolvers SERVFAIL, non-validating ones succeed. Almost always expired RRSIGs. Discriminator: dig +cd works.

“Intermittently wrong answers.” Two sets of nameservers are live — the new provider and the old one — and resolvers pick between them. Discriminator: the parent’s NS set differs from the zone’s, and querying each set gives different data.

“The record exists but validation fails.” Negative cache from a query that preceded the record’s creation. Discriminator: the authoritative server has it, the recursor returns NXDOMAIN, and the problem self-heals after the SOA minimum.

“One domain times out from one office.” Large DNSSEC-signed response, fragmented UDP dropped by a middlebox on that path. Discriminator: +bufsize=512 works, +bufsize=4096 +notcp hangs, +tcp works.

And a fifth that isn’t DNS at all: inside Kubernetes, /etc/resolv.conf typically carries options ndots:5, meaning any name with fewer than five dots is tried against every search domain first. A lookup for api.github.com therefore generates several failing queries before the correct one, which shows up as slow or flaky external resolution under load. Discriminator: dig is fast, the application is slow, and cat /etc/resolv.conf explains why. The fix is a fully-qualified name with a trailing dot, or a tuned dnsConfig.

Verdict

Start by establishing which component is wrong, not by staring harder at one answer. The ordered sequence is: compare your resolver against a public one, then query every authoritative server directly, then compare the parent’s delegation against the zone’s, then rule DNSSEC in or out with +cd, then bisect response size, and finally check whether the application is even asking what you think.

Three habits do the heavy lifting. The loop over every authoritative nameserver catches drift and partial propagation, which single queries through a recursor will hide indefinitely. dig +cd separates DNSSEC failures from everything else in five seconds. And getent hosts versus dig separates resolver configuration from DNS, which is where a frustrating share of “DNS outages” actually live.

Monitor RRSIG expiry like you monitor certificate expiry, keep TCP/53 open, and know that your zone’s SOA minimum — not your record TTLs — decides how long a name stays missing after you create it.

Sources

Comments