Skip to main content

Service Discovery & CoreDNS

Key Takeaways for AI & Readers
  • Naming over IPs: Kubernetes uses Service names for communication. Pod IPs are ephemeral and change on restarts -- Service names provide a stable network identity backed by automatic DNS records.
  • CoreDNS Automation: The cluster's built-in DNS server (CoreDNS) watches the Kubernetes API and automatically creates DNS records for every Service and pod. No manual DNS configuration is needed.
  • Service FQDN: The full address of a service follows the pattern <service>.<namespace>.svc.cluster.local. Within the same namespace, you can use just the service name (e.g., auth-service).
  • Kubelet Integration: The kubelet configures every container's /etc/resolv.conf to point to CoreDNS, with search domains that enable short-name resolution.
  • ndots Gotcha: The default ndots:5 setting means any name with fewer than 5 dots is appended with search domains before being queried as-is. This can cause up to 10 extra DNS lookups for external domains and is a common source of latency.
  • Headless Services: Setting clusterIP: None creates a headless Service that returns individual pod IPs instead of a virtual IP, essential for stateful workloads like databases.

In Kubernetes, you never hardcode IP addresses. Instead, you use the Service Name. But how does a container know that auth-service actually means 10.96.0.15?

1. Internal DNS Trace​

Visualize how a request is resolved inside the cluster.

📦
Client Pod
📋
CoreDNS
Every Pod's /etc/resolv.conf points to the CoreDNS Service IP. CoreDNS watches the K8s API and automatically maps service names to their ClusterIPs.

2. CoreDNS Architecture​

Every Kubernetes cluster runs a DNS server called CoreDNS as a Deployment in the kube-system namespace, exposed via a ClusterIP Service (typically at 10.96.0.10).

How CoreDNS Works​

  1. Watch: CoreDNS uses the kubernetes plugin to watch the Kubernetes API server for Service and Endpoint changes.
  2. Record creation: When a new Service is created, CoreDNS immediately creates the corresponding DNS records -- no delay, no propagation lag (within the cluster).
  3. Serve: When a pod makes a DNS query, the request goes to CoreDNS, which resolves it from its in-memory database.
  4. Forward: For queries that do not match any cluster Service (e.g., google.com), CoreDNS forwards the request to upstream DNS servers (configured in the CoreDNS ConfigMap, typically the node's DNS servers).

CoreDNS Configuration​

CoreDNS is configured via a ConfigMap in kube-system:

kubectl get configmap coredns -n kube-system -o yaml

A typical CoreDNS Corefile looks like:

.:53 {
    errors                          # Log errors
    health {                        # Health check endpoint on :8080/health
        lameduck 5s
    }
    ready                           # Readiness endpoint on :8181/ready
    kubernetes cluster.local in-addr.arpa ip6.arpa {  # Serve cluster DNS
        pods insecure
        fallthrough in-addr.arpa ip6.arpa
        ttl 30                      # Cache TTL for cluster records
    }
    prometheus :9153                # Expose Prometheus metrics
    forward . /etc/resolv.conf {   # Forward external queries upstream
        max_concurrent 1000
    }
    cache 30                        # Cache external DNS for 30 seconds
    loop                            # Detect and break DNS loops
    reload                          # Auto-reload on ConfigMap changes
    loadbalance                     # Round-robin A records
}

3. DNS Record Types​

CoreDNS creates several types of DNS records for Kubernetes resources.

A/AAAA Records for Services​

Every ClusterIP Service gets an A record (IPv4) or AAAA record (IPv6):

# Format: <service>.<namespace>.svc.cluster.local
auth-service.default.svc.cluster.local  ->  10.96.0.15

The record resolves to the Service's ClusterIP, which is a virtual IP that kube-proxy routes to healthy backend pods.

SRV Records for Ports​

SRV records are created for named ports in a Service. They contain the port number and the target hostname:

# Format: _<port-name>._<protocol>.<service>.<namespace>.svc.cluster.local
_http._tcp.auth-service.default.svc.cluster.local  ->  SRV 0 100 8080 auth-service.default.svc.cluster.local

SRV records are useful for service discovery in frameworks that support them, allowing clients to discover both the hostname and port dynamically.

Pod DNS Records​

Pods also get DNS records, though they are less commonly used directly:

# Format: <pod-ip-dashed>.<namespace>.pod.cluster.local
10-244-0-5.default.pod.cluster.local  ->  10.244.0.5

4. Service FQDN and Search Domains​

The FQDN Format​

Every Service has a Fully Qualified Domain Name:

<service-name>.<namespace>.svc.<cluster-domain>

The default cluster domain is cluster.local, so a Service named auth-service in the default namespace has the FQDN auth-service.default.svc.cluster.local.

How Short Names Work​

You do not need to use the full FQDN. When the kubelet starts a container, it injects /etc/resolv.conf:

nameserver 10.96.0.10
search default.svc.cluster.local svc.cluster.local cluster.local
options ndots:5

The search domains allow short-name resolution:

What you typeWhat gets resolved (from default namespace)
auth-serviceauth-service.default.svc.cluster.local
auth-service.backendauth-service.backend.svc.cluster.local
auth-service.backend.svcauth-service.backend.svc.cluster.local
auth-service.backend.svc.cluster.localDirect FQDN lookup (no search)

Cross-namespace resolution: If a pod in the frontend namespace needs to reach a service in the backend namespace, use auth-service.backend. The search domain appends .svc.cluster.local automatically.

The ndots Problem​

The ndots:5 setting means: "If a name has fewer than 5 dots, try appending each search domain before querying the name as-is."

When a pod queries api.example.com (2 dots, fewer than 5):

  1. Try api.example.com.default.svc.cluster.local -- NXDOMAIN
  2. Try api.example.com.svc.cluster.local -- NXDOMAIN
  3. Try api.example.com.cluster.local -- NXDOMAIN
  4. Try api.example.com -- SUCCESS

That is 4 DNS queries instead of 1. For applications that make many external DNS lookups, this causes measurable latency and load on CoreDNS.

Mitigations:

  • Trailing dot: Use api.example.com. (note the trailing dot). This tells the resolver the name is already fully qualified and skips search domain expansion.
  • Lower ndots: Set ndots:2 in the pod's DNS config (see Pod DNS Policies below). This reduces unnecessary lookups but may break short-name resolution for services in other namespaces.
  • NodeLocal DNS Cache: Deploy NodeLocal DNSCache as a DaemonSet that caches DNS responses on each node, reducing latency and load on CoreDNS.

5. Headless Services​

A normal ClusterIP Service provides a single virtual IP. A headless Service (clusterIP: None) skips the virtual IP and returns the individual pod IPs directly in the DNS response.

apiVersion: v1
kind: Service
metadata:
  name: postgres
  namespace: database
spec:
  clusterIP: None                     # This makes it headless
  selector:
    app: postgres
  ports:
  - port: 5432
    targetPort: 5432

DNS Behavior​

A DNS lookup for postgres.database.svc.cluster.local returns an A record for each ready pod:

postgres.database.svc.cluster.local  ->  10.244.0.5, 10.244.0.6, 10.244.0.7

When used with a StatefulSet, each pod also gets a predictable DNS name:

postgres-0.postgres.database.svc.cluster.local  ->  10.244.0.5
postgres-1.postgres.database.svc.cluster.local  ->  10.244.0.6
postgres-2.postgres.database.svc.cluster.local  ->  10.244.0.7

This is essential for stateful workloads like databases, where clients need to connect to a specific replica (e.g., the primary) or where the application itself needs peer discovery.

6. Pod DNS Policies​

You can control how a pod resolves DNS using the dnsPolicy field:

PolicyBehavior
ClusterFirst (default)Use CoreDNS for cluster names. Forward external queries upstream via CoreDNS.
DefaultInherit the DNS config from the node the pod runs on. Does not use CoreDNS.
NoneDo not auto-configure DNS. You must provide custom config via dnsConfig.
ClusterFirstWithHostNetSame as ClusterFirst, but for pods using hostNetwork: true.

Custom DNS Configuration​

When you need fine-grained control, use dnsPolicy: None with dnsConfig:

apiVersion: v1
kind: Pod
metadata:
  name: custom-dns-pod
spec:
  dnsPolicy: "None"
  dnsConfig:
    nameservers:
    - 10.96.0.10                   # CoreDNS
    - 8.8.8.8                      # Google DNS as fallback
    searches:
    - default.svc.cluster.local
    - svc.cluster.local
    options:
    - name: ndots
      value: "2"                   # Reduce from default 5 to cut external lookups
    - name: timeout
      value: "3"
    - name: attempts
      value: "2"
  containers:
  - name: app
    image: myapp:v1

More commonly, you keep dnsPolicy: ClusterFirst and use dnsConfig only to override ndots:

spec:
  dnsPolicy: ClusterFirst
  dnsConfig:
    options:
    - name: ndots
      value: "2"

7. ExternalName Services​

An ExternalName Service maps a Kubernetes service name to an external DNS name. It creates a CNAME record instead of an A record.

apiVersion: v1
kind: Service
metadata:
  name: external-db
  namespace: production
spec:
  type: ExternalName
  externalName: mydb.us-east-1.rds.amazonaws.com

Now, any pod that resolves external-db.production.svc.cluster.local gets a CNAME pointing to mydb.us-east-1.rds.amazonaws.com. This provides a layer of indirection -- if the external hostname changes, you update the Service, not every application configuration.

Limitations:

  • ExternalName Services do not proxy traffic. There is no ClusterIP and no kube-proxy rules.
  • HTTPS connections may fail if the certificate does not match the service name. The TLS handshake uses the original hostname, not the CNAME target.
  • ExternalName does not support ports. The client must know the correct port.

8. DNS Debugging​

When DNS resolution fails, here is a systematic debugging approach.

Step 1: Start a Debug Pod​

kubectl run dns-debug --image=busybox:1.36 --restart=Never --rm -it -- sh

Or use a more complete toolkit:

kubectl run dns-debug --image=nicolaka/netshoot --restart=Never --rm -it -- bash

Step 2: Test Internal Resolution​

# Inside the debug pod
nslookup auth-service.default.svc.cluster.local
# Expected: returns the ClusterIP

nslookup kubernetes.default.svc.cluster.local
# Expected: returns the API server ClusterIP (always exists)

Step 3: Test External Resolution​

nslookup google.com
# If this fails, CoreDNS cannot reach upstream DNS

Step 4: Check CoreDNS Health​

# From your workstation
kubectl get pods -n kube-system -l k8s-app=kube-dns
kubectl logs -n kube-system -l k8s-app=kube-dns

Step 5: Verify Endpoints​

# Ensure the Service has endpoints (backing pods)
kubectl get endpoints auth-service
# If empty, the Service selector does not match any running pods

Common Pitfalls​

1. ndots:5 causing excessive DNS queries. Every external domain lookup generates up to 4 extra queries. For applications making hundreds of external calls per second, this adds significant latency. Lower ndots or use FQDNs with trailing dots.

2. DNS caching surprises. CoreDNS caches records for 30 seconds by default. If a Service is deleted and recreated, clients may resolve stale IPs for up to 30 seconds. Some language runtimes (Java, notably) cache DNS at the application level indefinitely -- configure your runtime's DNS TTL.

3. Service exists but DNS returns NXDOMAIN. Check that the Service is in the correct namespace and that you are using the right namespace in the FQDN. Also verify CoreDNS pods are running.

4. Headless Service returning stale pod IPs. If pods are terminated but CoreDNS has not yet updated, DNS may return IPs of terminated pods. Ensure readiness probes are configured so unhealthy pods are removed from endpoints.

5. ExternalName and HTTPS. Browsers and HTTP clients validate TLS certificates against the requested hostname. If your code connects to external-db but the certificate is for mydb.us-east-1.rds.amazonaws.com, the TLS handshake may fail depending on how your client handles CNAMEs.

Best Practices​

  1. Use short service names for same-namespace communication. Just auth-service is sufficient and most readable. Use auth-service.other-namespace for cross-namespace calls.

  2. Always use the trailing dot for external FQDNs in configuration files (e.g., api.external.com.). This prevents unnecessary search domain expansion.

  3. Monitor CoreDNS performance. CoreDNS exposes Prometheus metrics on port 9153. Track coredns_dns_requests_total, coredns_dns_responses_total, and coredns_dns_request_duration_seconds to detect overload.

  4. Deploy NodeLocal DNSCache in production clusters. It runs a DNS cache on every node, reducing latency and eliminating CoreDNS as a single point of failure for DNS resolution.

  5. Configure readiness probes on all pods behind Services. DNS returns IPs of pods in the Service's endpoints list. Without readiness probes, pods receive traffic (and DNS records) before they are ready to serve.

  6. Lower ndots for workloads that primarily call external services. Set ndots: 2 in the pod's dnsConfig to reduce external lookup latency while still supporting internal short names.

What's Next?​

  • Ingress (Routing) -- Once DNS resolves your internal Services, learn how Ingress routes external HTTP traffic to those Services.
  • Services -- Review the Service resource types (ClusterIP, NodePort, LoadBalancer) that DNS records point to.
  • StatefulSets -- Understand how headless Services provide stable network identities for stateful pods.
  • Observability -- Monitor CoreDNS metrics and detect DNS resolution failures in your monitoring stack.
  • Probes (Health Checks) -- Readiness probes determine which pod IPs appear in DNS for headless Services.