From Silicon to Systems: Bending the Cost and Complexity Curves of AI

The future of AI infrastructure isn’t just about a single groundbreaking moment; rather, it’s about our ability to collaboratively design every element — from models and software, right through to silicon, servers, racks, and data centres. At the Open Compute Project Asia Pacific Summit, we’re excited to present the latest innovations from our community, all aimed at addressing the diverse infrastructure needs driven by the varied workloads that support agentic AI.

This holistic approach is at the heart of how Microsoft creates specialised AI infrastructure designed for cloud scalability. We focus on optimising silicon, software, and systems to boost efficiency, performance, and security. Our Maia 200 accelerator embodies this philosophy by enhancing Microsoft’s production inference workloads. We carefully co-developed the chip, model, and system to achieve better performance for every dollar spent and improved energy efficiency. The Azure Cobalt 200 CPU expands this concept to cloud-native and agentic computing, featuring an architecture optimised for concurrency, orchestration, and faster calls from agents. Furthermore, the Azure Boost DPU enhances performance for storage and networking by relieving host CPUs of infrastructure tasks, thus promoting greater consistency, efficiency, and cost-effectiveness while lowering energy consumption and heat generation. Together, these solutions exemplify the benefits of integrating hardware and software into a unified system — setting the stage for innovations that go beyond just chips to include entire racks and data centres.

Across the Azure platform, we’re implementing rack-level architectures, liquid cooling solutions, and highly efficient compute and storage services, all alongside cutting-edge silicon. In our Fairwater data centre, the infrastructure is specifically designed for the density and operational requirements of today’s AI systems. On a component level, next-gen heat exchangers efficiently transfer heat away from high-density racks while microfluidics technologies provide even more precise cooling directly at the chip level. Regarding power distribution, Mt. Diablo decouples rack-scale power from computation, simple solid-state transformers ease energy conversion, high-temperature superconductors allow the transmission of greater power with reduced losses, and high-temperature fuel cells present a means to generate energy compatible with future 800 VDC data centre designs. Each of these efforts addresses unique bottlenecks, collectively allowing for denser, more reliable, and more sustainable AI infrastructures.

The power architecture exemplifies the importance of open, system-level collaboration. Microsoft, NVIDIA, and Meta are actively contributing to the industry’s shift towards 800 VDC through Open Compute Project (OCP) initiatives that integrate facility power, conversion, distribution, and rack design into a workable strategy. Transitioning to higher-voltage direct current means we can minimise conversion stages between the grid and the accelerator, ensuring more accessible power reaches our compute resources while accommodating increasing rack densities. The broader design supports various adoption pathways; hybrid power racks can retrofit 800 VDC into existing AC setups, row-level systems can efficiently distribute power across multiple racks, and future-built designs can convert medium-voltage power directly to 800 VDC. Coupled with the evolving Diablo specification and the solid state transformer specification for medium voltage to 800VDC power conversion, these collaborations offer equipment manufacturers common frameworks for design, testing, and production. This openness is crucial for establishing a resilient supply chain and providing operators a realistic path forward, from today’s facilities to the AI factories of the next decade.

Innovations in cooling are just as essential, especially as the traditional trade-offs between system density and thermal management have reached their limits. Instead of considering cooling merely an afterthought, we are designing it in tandem with chips, modules, servers, and racks. Closed-loop liquid systems support sustained performance while minimising water consumption. Heat exchangers enhance the efficiency and serviceability of rack-level cooling. Microfluidic technologies allow for better placement of coolant to where heat is generated, potentially reducing peak temperatures and enabling higher density. Our aim is not merely a collection of disparate technologies, but a comprehensive thermal architecture that can evolve alongside future generations of compute.

Networking also needs to evolve with this comprehensive approach in mind. Within scaling domains, AI systems require high bandwidth, low latency, low power usage, and predictable reliability across closely-knit accelerators. Our commitment to advancing Ethernet Scale-Up Networking (ESUN) promotes open Ethernet as a standard fabric for scale-up, drawing from diverse ecosystem contributions to ensure we can scale without imposing a closed approach on the industry. On a larger scale, scale-out networks must interconnect vast clusters while maintaining real-time insights, resilient pathways, and effective utilisation. To meet these demands, Multipath Reliable Connection (MRC) enables traffic to utilise multiple pathways, adapt in real-time, and support consistent, dependable training across systems with over 100,000 GPUs. Optical technology plays a significant role in this narrative by enhancing scale in an efficient manner. Initiatives like the OCI-MSA focus on better integration between optical connectivity, compute, and networking silicon to increase bandwidth density and scalability across systems. Collectively, MRC, Ethernet Scale-Up Networking, and scale-out optics create a continuous flow of connectivity — from accelerators to pods, pods to data centres, and from data centres to extensive distributed AI supercomputers.

This comprehensive approach amplifies the value at every stage. Improved silicon leads to greater system efficiency. Better system designs enhance infrastructure utilisation. Enhanced power, cooling, and networking solutions reduce the costs and complexities associated with training and deploying AI models. Every advancement across these layers results in more capable, dependable, and accessible AI services for users. This is our path forward, enabling the industry to progress beyond mere incremental enhancements and begin reshaping cost, power, and complexity in a unified effort.

However, no entity can tackle these challenges in isolation. Progress relies on early collaboration among silicon producers, packaging partners, manufacturers, network providers, cloud services, data centre operators, researchers, and standards organisations. We are proud to be part of the larger Open Compute Project Global Foundation community, working collaboratively in an open environment to turn individual breakthroughs into collective advancements. Microsoft looks forward to deeper partnerships across power, cooling, networking, security, sustainability, and system design — especially within the APAC ecosystem, where so much of the world’s leading computing infrastructure is developed.