CoreDNS in AKS: service discovery, upstream DNS, and failover

A pod can successfully resolve a Kubernetes Service but struggle to resolve a database name located outside the cluster. Even if CoreDNS appears to be functioning properly, this issue can still arise.

Understanding this distinction is vital, as multiple issues can mimic a DNS outage:

• A cluster Service name fails to resolve. This could be due to an incorrect namespace, an erroneous Service record, or an incorrect path from the pod to the cluster DNS.
• Resolution fails for an external name. Although CoreDNS may be accessible, the upstream server or its network path might be down.
• A DNS lookup eventually succeeds, but the application fails. It’s possible that the application has given up waiting for a DNS response before CoreDNS has returned the answer.

CoreDNS serves as the default cluster DNS service in AKS, running as pods within the kube-system namespace, rather than as embedded code in your application or as a standalone server in the AKS control plane.

It performs two crucial functions:

1. Kubernetes Service discovery: It provides responses for names associated with Kubernetes resources, eliminating the need for workloads to keep track of changing addresses.
2. Upstream forwarding: CoreDNS forwards queries that need another DNS server to the pre-configured upstream resolvers.

To delve deeper into this, check out Microsoft’s AKS DNS concepts guide. The key takeaway here is straightforward: CoreDNS is part of the resolution pathway, but it is not responsible for every DNS answer.

Who should read this: This guide is intended for platform engineers, application developers, and operations teams aiming to gain a practical insight into CoreDNS within AKS.

For a standard pod with dnsPolicy: ClusterFirst, and without a node-local DNS layer, the DNS resolution path typically looks something like this:

Application in a pod
    -> Cluster DNS Service: kube-dns
    -> CoreDNS pod
         -> Kubernetes Service information: provide an answer for a cluster Service name
         -> Upstream DNS server: forward a name that is handled externally
    -> DNS response is sent back to the application

It’s worth noting that the Service remains referred to as kube-dns even though CoreDNS takes care of its traffic. This doesn’t indicate that the cluster is still using the older kube-dns version.

Kubernetes automatically configures the resolver settings for the pod, which include the nameserver and search domains. For those using Linux containers, this information can be found in /etc/resolv.conf.

Let’s consider a standard ClusterIP Service named orders that resides in the checkout namespace. Given the cluster domain cluster.local, its complete name would be:

orders.checkout.svc.cluster.local

The kubernetes plugin of CoreDNS utilises Kubernetes resource data to resolve that name to the Service’s cluster IP. It does not reach out to an external corporate or public DNS server for that Service record.

It’s also essential to note the significance of the namespace. A pod that exists in the checkout namespace can use the shorthand orders; however, a pod in a different namespace typically needs to refer to orders.checkout or use the full name. For naming rules, refer to Kubernetes DNS for Services and Pods.

For names outside of its configured cluster zones, CoreDNS can employ the forward plugin to contact an alternative DNS server. Depending on the cluster’s particular DNS configurations, this might refer to an Azure-provided resolver or a custom resolver.

Be careful not to assume that every AKS cluster will forward DNS requests to the same address, or that two configured addresses always imply a primary and a backup. Always check the actual CoreDNS configuration and the upstream resolver settings it uses.

Caching can also alter the DNS resolution pathway. A valid cached answer can completely bypass an upstream query.

If AKS LocalDNS is enabled, a pod will first connect to a DNS proxy and cache on its node. Cluster domain queries will be routed to CoreDNS, while other queries may either go through CoreDNS or head directly to an upstream resolver, based on the DNS policy and LocalDNS configuration.

Therefore, the diagram above serves as a starting reference rather than a definitive packet trace. Before determining which resolver to inspect, check if LocalDNS is active. The AKS DNS concepts guide elaborates on these alternative pathways.

AKS manages the deployment of CoreDNS and its fundamental configurations. The Corefile, which is stored in the coredns ConfigMap, informs CoreDNS of the plugins and forwarding rules it should utilize.

Here are three key plugin responsibilities to recognise:

If an upstream DNS failure occurs, it will affect queries needing that upstream service. However, this doesn’t imply that standard Kubernetes Service records are also unreachable.

For customization support, AKS provides the coredns-custom ConfigMap. Microsoft has documented custom entries that end in .server or .override, inclusive of domain-specific forwarding examples.

It is crucial not to directly modify the managed master Corefile as a shortcut. Just because a CoreDNS option appears valid doesn’t mean that every method of applying it is supported in AKS.

For instance, forwarding a private domain to specific resolvers is different from altering the default root forwarding behaviour for the entire cluster. For any intended changes, refer to the AKS CoreDNS customization guide.

It’s also good practice to verify the deployed image version. The tests mentioned here were conducted using CoreDNS version 1.13.1. Therefore, a setting found in documentation for a more recent version may not exist in the image you’re working with.

Once a query engages the forward plugin, there are three distinct decisions to be made: selecting which server to query first, determining when to try another server, and deciding whether to skip over a server in subsequent queries.

This illustrates the behaviour for CoreDNS 1.13.1:

• random selects among available upstream services and is the default option if no policy is specified.
• round_robin rotates through the options on each selection.
• sequential goes through upstreams in their predefined order, prioritising the first one that is not marked as unhealthy.

Here’s an example of an isolated lab configuration, not a directive to implement in AKS-managed CoreDNS. The addresses indicated are for illustration only:

forward .   {
    policy sequential
    failover SERVFAIL
}

policy sequential dictates the order of upstream queries, while failover SERVFAIL allows another attempt with an upstream service if the server signals it could not complete the lookup. These are distinct settings for different decision points.

If packets are silently discarded, CoreDNS doesn’t receive a response and must wait for the timeout period to elapse. If the server issues a SERVFAIL response, CoreDNS has received a DNS error notification.

By default, the forward plugin returns SERVFAIL to the caller instead of trying to reach out to another server. An explicit failover rule can modify how certain response codes are handled.

Protocol type is also important. User Datagram Protocol (UDP) does not establish a connection prior to sending a query, whereas Transmission Control Protocol (TCP) does make a connection first.

In this version, the read timeout is set to two seconds. Meanwhile, the TCP connection initiation has a different timeout starting at 30 seconds and may decrease to one second based on previous connection durations. There is no single timeout that covers every possible failure scenario.

For further details, these behaviours are documented in the CoreDNS 1.13.1 forward plugin reference.

The forward plugin initiates health checks for upstream servers only after a network error occurs. It does not continuously check the health of every untouched functioning upstream until a failure is detected.

The documented default check interval is 0.5 seconds. This is not a guarantee that failover will complete within half a second; the triggering query may already be waiting for a timeout at this point.

max_fails determines how many failed health checks will mark an upstream as unhealthy, with a default setting of 2. In the application-impact tests, this was set to 1 for easier observation of changes.

Just because a DNS error response is received doesn’t always indicate a loss of network connectivity. Therefore, health checking doesn’t necessarily verify that an upstream can resolve every name required by your application successfully.

Each CoreDNS process maintains its own health state for upstream servers. Testing one replica does not guarantee the health status of every replica, and restarting a resolver resets this state. The health-check guide offers a distinction between performance-backed behaviour and noted, partial, as well as inconclusive scenarios.

The tests were carried out using Kubernetes v1.35.7 and image mcr.microsoft.com/oss/v2/kubernetes/coredns:v1.13.1-20.

Each test was performed against separate CoreDNS resolvers and controlled upstream servers within an isolated namespace. No faults were injected into the AKS-managed CoreDNS, nor were all production DNS pathways validated, including LocalDNS.

The test resolver setup employed sequential selection with max_fails 1. Query response times are measured in milliseconds (ms).

Five additional independent UDP failure cycles reached the backup in 2,000-2,004 ms on the initial query. Subsequent queries took 0-4 ms.

For a summary of the results, please refer to result summary and raw query output for detailed insights.

These observations were made using one specific image and controlled failure conditions, hence they do not guarantee timings for AKS in general. A display of 0 ms indicates the tool’s timing precision rather than an absence of latency.

The takeaway here is: an available CoreDNS pod, a reachable backup server, and a successful application request are three distinctly different scenarios.

Begin by differentiating between cluster Service resolution and upstream resolution, then examine the configuration before making any changes.

Prerequisites: You’ll need PowerShell 7+, an installed and authenticated kubectl, and permission to read CoreDNS Deployments, pods, Services, and ConfigMaps within your desired cluster. Always verify the cluster context beforehand and cease if it’s incorrect. The following commands are solely for reading purposes.

To check configurations, run the following commands:

kubectl config current-context
kubectl get deployment coredns -n kube-system -o jsonpath="{.spec.template.spec.containers[0].image}"
kubectl get pods -n kube-system -l k8s-app=kube-dns
kubectl get service kube-dns -n kube-system
kubectl get configmap coredns -n kube-system -o yaml

Utilise your findings to refine your investigation:

• If only a cluster Service name is failing: Confirm the Service name, namespace, and record type. Ensure that the Service is indeed in place and that the pod is accessing the correct cluster DNS path.
• If only names requiring an upstream fail: Check the pursuant forwarding rule, assess the upstream’s reachability, and verify that the upstream can resolve the required name.
• If only some pods are encountering failures: Compare their DNS policies, resolver settings, node-local DNS paths, and network access. While a successful query from an alternate pod is informative, it doesn’t provide all the context.
• If the first query is slow but subsequent queries are fast: Look into upstream health states and caching. Don’t assume the initial result was simply a random glitch.
• If CoreDNS returns SERVFAIL: Ascertain whether it originated from an upstream response or another sort of failure. Avoid making assumptions about lost packets based on this response.

To assess DNS latency, you can use the dig command’s Query time feature. Note that the elapsed time of kubectl exec also factor in Kubernetes API communication and process startup times.

Lastly, take a look at the application. Reusing a connection might bypass DNS resolution, a cached answer can mask an outage, and retries might add to the load. Even if CoreDNS eventually resolves a name, clients might time out beforehand.

The most recent suite comprised 13 successful DNS and environment checks alongside five blocked application-specific cases. No representative application was supplied, so the effects of caching, connection reuse, retries, application monitoring, and service-level objectives (SLO) remain unaddressed. A DNS test alone cannot guarantee the user’s experience.

The CoreDNS repository folder categorises these queries into five comprehensive guides:

Each guide maps specific questions to test cases, expected outcomes, criteria for passing or failing, and an Established evidence section that clarifies what occurred, when, and what cannot be conclusively proved.

Prerequisites for fault testing: You’ll need access to an authorized test cluster, PowerShell 7+, authenticated kubectl, the rights to create and delete namespace-local test resources, and a network setup that enforces the test Network Policies. Before executing tests, review the complete setup and cleanup instructions outlined in each guide.

Always use an isolated test resolver. Do not modify managed kube-system resources when trying to reproduce an upstream error. If another test namespace is already present and its ownership is unclear, opt for a unique namespace rather than deleting the existing one.

The published examples come with the original lab context and paths starting with 07-CoreDNS. A copy from GitHub will label the directory as coredns. Tailor these values to suit your environment before running any test suite.

Cleanup is part of the testing process. In the last run, the unique test namespace was deleted, all six base-lab Deployments remained available, both managed CoreDNS replicas stayed accessible, and the version of the managed Deployment resource remained unchanged.

CoreDNS links AKS workloads to both Kubernetes service discovery and the broader DNS ecosystem. Grasping the query’s resolution path is the first step towards understanding any resulting failures.

Begin with the DNS path from the pod. Look into the CoreDNS version and its configurations. Separate Service records from upstream queries. Finally, assess selection, timeouts, health checks, and application behavior without altering the managed cluster DNS.

Understand the current configuration before changing policy settings. Validate expected behaviours before asserting results.

Prerequisites: You should have Git, PowerShell installed, access to the GitHub network, and a working directory without a pre-existing aks-stuff folder. No Azure permissions are required to download the guides. The commands provided do not deploy resources or run fault tests.

git clone https://github.com/jvargh/aks-stuff.git
Set-Location .\aks-stuff
Get-Content .\coredns\README.md

Disclose any reproducible findings through the GitHub repository. Ensure you include the CoreDNS version, DNS path, test case details, failure conditions, client timeout settings, recorded results, and cleanup outcomes. Remember to omit any credentials, private addresses, and customer identifiers prior to sharing logs.

Get started here: github.com/jvargh/aks-stuff/tree/main/coredns