Powering multi-cluster workloads with seamless cross‑cluster networking for Azure Kubernetes Fleet Manager

We’re excited to present a superior managed network solution based on Cilium, designed to enhance cross-cluster connectivity across your entire fleet.

As businesses work to modernise their application setups, we’re observing a major transformation in the architecture of cloud-native infrastructures. Instead of asking, “How do we scale a cluster?”, the focus has shifted to “How can we scale across multiple clusters, regions, and clouds while keeping things simple?”

Today, we’re pleased to unveil the public preview of cross-cluster networking for Azure Kubernetes Fleet Manager. This development represents significant progress in multi-cluster management, introducing seamless east-west networking across clusters, all supported by Advanced Container Networking Services.

The Challenge of Multi-Cluster Networking

Businesses of all sizes often manage multiple Azure Kubernetes Service (AKS) clusters due to regulatory compliance, disaster recovery needs, or the desire to create isolated environments. However, overseeing these clusters typically incurs a “networking tax.” Traditional solutions typically involve complicated VPNs, gateways, and manual service discovery, which can lead to increased latency and operational burdens.

Even when managing only a few clusters, especially large-scale fleets, teams require consistent and reliable connectivity across clusters. This is essential for scenarios like failover, shared services architectures, and moving workloads across regions to optimise capacity and latency. Simultaneously, platform teams aim to abstract infrastructure complexities from developers, allowing seamless cluster-level modifications without disrupting applications.

Our Vision: Simplified Multi-Cluster Management

To tackle these challenges, we created Azure Kubernetes Fleet Manager. Fleet Manager is aimed at simplifying Kubernetes management across multiple clusters. While it has made workload deployment and update coordination easier, networking has remained a challenging area.

By introducing Cilium-based cross-cluster networking in Azure Kubernetes Fleet Manager, we provide a managed, high-performance network that spans your entire fleet.

This functionality extends the Kubernetes networking model across clusters, allowing services and workloads to communicate as if they were local, while still maintaining cluster-level isolation and governance.

Built on a foundation of open-source technology, we utilise Cilium for the dataplane and Kubefleet for fleet orchestration, both of which are part of active Cloud Native Computing Foundation (CNCF) projects. This promotes transparency, portability, and ensures alignment with the wider Kubernetes ecosystem, taking advantage of ongoing innovation within the open-source community.

The following illustration demonstrates how clusters within a fleet are connected through a unified, managed network that facilitates smooth communication and policy enforcement.

Key features include:

• Smooth east-west connectivity: Thanks to eBPF-based routing with the power of Azure CNI, pods can communicate across clusters with native performance, eliminating the need for proxies or gateways.
• Global service discovery: By simply adding an annotation (service.cilium.io/global=true), a standard Kubernetes Service is transformed into a “global” one. Cross-cluster networking automatically detects endpoints across member clusters, ensuring transparent load balancing and failover.
• Comprehensive multi-cluster observability: Get a holistic view of network health across clusters with combined metrics, logs, and flow visibility. Advanced Container Networking Services integrates Cilium telemetry for consistent insights, quicker troubleshooting, and complete visibility across your fleet.
• Unified security and governance: Security policies are now not confined by cluster boundaries. With Advanced Container Networking Services, enforce network policies that are enterprise-grade, providing visibility across your entire global footprint while ensuring identity-based security follows workloads wherever they go.
• Zero-touch management: Fleet Manager manages the complex lifecycle, including certificates and network configurations, so you don’t have to worry about it.

These features leverage eBPF to provide efficient routing, policy enforcement, and observability for high-performance networking.

Strategic Resilience with Cross-Cluster Networking

In today’s digital-first economy, resilience can give you a competitive edge. Cross-cluster networking allows customers to create infrastructures that are naturally resilient to failures of single clusters or regions.

Whether you’re managing “Shared Services” clusters for multiple tenants or developing “Global Services” that direct traffic to the healthiest available endpoints, cross-cluster networking with Azure Kubernetes Fleet Manager ensures your infrastructure is as flexible as your business demands.

We’re dedicated to providing a secure, robust, and high-performance platform for multi-cluster environments. Cross-cluster networking is a significant advancement towards a future where the physical limits of a cluster do not stifle innovation.

Getting Started with Cross-Cluster Networking

Cross-cluster networking for Azure Kubernetes Fleet Manager is crafted to minimise operational complexity:

Clusters must meet these prerequisites:

• Azure CNI powered by Cilium as the networking dataplane.
• Advanced Container Networking Services enabled.

Here’s how to set up cross-cluster networking:

1. Join clusters to a Fleet.
2. Link the members with a cross-cluster network profile.
3. Deploy services with global annotations to enable cross-cluster communication.

Once configured, Fleet Manager will automatically deploy and manage the necessary components, facilitating direct pod-to-pod communication across clusters without the need for extra gateways or overlays.

This managed approach alleviates the burden of manually setting up and maintaining Cilium multi-cluster components, allowing your teams to concentrate on delivering applications rather than managing infrastructure.

See it in action: Check out the Cross-Cluster Networking for Azure Kubernetes Fleet Manager Video Guide for more details and a quick demo!

Documentation and Resources

If you have any feedback or want to learn more, please connect with your Microsoft account team or share your thoughts through the Azure Kubernetes Service community channels. We’re eager to hear from you!