Your H100s on Azure won't do distributed training. Here's probably why.

I recently needed to investigate why a customer’s H100 virtual machines (VMs) were unable to perform distributed training on Azure. While they had the ND-series capacity set up and the necessary hardware, the VMs weren’t connected via InfiniBand. The underlying issue was traced back to a problem in the deployment configuration rather than the hardware itself. It wasn’t easy to find a straightforward explanation of what went wrong and how to resolve it.

The root cause boils down to one of two possibilities: either the VMs are running the incorrect H100 SKU, or they aren’t situated in the same placement group. Thankfully, both issues can be addressed, but first, it’s critical to identify what the problem is.

Azure offers two families of H100 VMs. The names are similar enough that it’s easy for users to mistakenly select the wrong option without noticing.

The NCads H100 v5 family comprises two sizes: NC40ads (featuring 1 GPU and 40 Gbps) and NC80adis (with 2 GPUs and 80 Gbps). Neither of these options supports InfiniBand. If you deploy ten of these and attempt a multi-node training job, the GPUs will communicate using standard Ethernet. For the all-reduce operations crucial for distributed training, you’ll experience around 40 Gbps per VM compared to a whopping 3.2 Tbps on a single ND node, which translates to roughly 80 times less bandwidth. In practice, it can be even worse because Ethernet doesn’t support GPUDirect RDMA, causing every transfer to go through main memory and the CPU.

The ND VMs are specifically designed for training jobs. They come with eight H100 SXM GPUs per VM, linked internally via NVLink, and each GPU has its own dedicated 400 Gb/s InfiniBand connection for communicating with GPUs across different VMs. With GPUDirect RDMA, NCCL can send data directly across nodes without involving the CPU.

If your training task requires more GPUs than one VM can provide, you must use ND. There’s simply no workaround for this.

Not every GPU workload necessitates InfiniBand. For instance, single-GPU inference or development work is perfectly fine on NC, as there’s no need for inter-node communication.

LoRA and QLoRA fine-tuning for models up to around 70B usually fits within a single ND VM (utilising 8 GPUs with model parallelism). Since there’s no crossing of VM boundaries, InfiniBand isn’t a concern here.

When it comes to full-parameter SFT or RLHF on models with 8B+, things get more complex. An 8B model in bf16 takes up about 16 GB just for parameters. Full-parameter Adam training typically demands 12 to 16 bytes per parameter when including fp32 master weights, gradients, and optimizer first/second moments. That places an unsharded 8B run at 96 to 128 GB before even considering activations. Using FSDP or ZeRO-3 sharding across the 8 GPUs within a single ND VM allows it to fit nicely without crossing node boundaries. Attempting to accomplish this with standalone VMs without a sharding strategy means you’ll need multi-node setups, which in turn require a fast interconnect.

For large-scale pre-training of models with 70B and above, the workload spans dozens or even hundreds of GPUs. At this scale, InfiniBand becomes essential. Gradient synchronisation over Ethernet would render the job impractical.

A useful rule of thumb: if your training job requires more GPUs than what fits in one VM, you should consider InfiniBand crucial. If you’re working with more than two VMs, it becomes significantly more important.

Having ND VMs within the same region isn’t sufficient. They must operate on the same InfiniBand fabric to communicate at their full potential.

Azure manages this through placement groups. A placement group is essentially a collection of VMs that are deployed on the same network fabric segment. VMs in the same group enjoy non-blocking, full-bisection IB connectivity. Conversely, VMs in different groups, or standalone VMs not part of any VMSS or availability set, won’t establish InfiniBand communication with one another.

The consequences of lacking IB depend on your NCCL configuration. If NCCL_IB_DISABLE=1 is enabled, or if NCCL can’t detect IB devices, it will default to TCP over Ethernet (which is slower but still functional). In cases where IB interfaces exist but aren’t connected to peers (as is the case with standalone ND), NCCL might indefinitely hang while it waits to establish the IB connection. To investigate which transport NCCL is using and to pinpoint where it’s getting stuck, you can set NCCL_DEBUG=INFO in your environment.

The following diagram illustrates this specific scenario: the same ND hardware deployed differently leading to very different results.

To place your NDs into a single placement group, you should deploy them in a Virtual Machine Scale Set (VMSS) with singlePlacementGroup set to true:

{ "type": "Microsoft.Compute/virtualMachineScaleSets", "properties": { "singlePlacementGroup": true } }

This works for both Flexible and Uniform orchestration modes (the InfiniBand setup documentation confirms support for both). The key aspect is the placement group, not the orchestration mode. When set to singlePlacementGroup=true, the scale set defaults to a limit of 100 VMs (this can expand to 300 with a support request). For most initial training jobs, even this default cap provides you with 800 H100 GPUs, which is ample for post-training and medium-scale pre-training tasks. Should you need over 2,400 GPUs on a single IB fabric, that will require deeper discussions involving Azure’s HPC team.

The vital point to remember is that setting singlePlacementGroup=false (which allows larger scale sets beyond 100 VMs) will completely disrupt InfiniBand networking. According to the placement groups documentation, “Large scale (SPG=false) does not support InfiniBand networking.” VMs will end up in different placement groups, and communication via IB won’t function across those groups. If you require more than 100 nodes for IB training, it’s best to submit a support request to increase the single placement group limit rather than switching to a multi-placement-group setup.

Once your NDs are grouped in a single placement group, InfiniBand connectivity between VMs is automatically set up. No manual configuration is necessary. The ND H100 v5 documentation confirms: “These connections are automatically configured between VMs in the same virtual machine scale set and support GPU Direct RDMA.”

Contiguous capacity must be available in the region at the time of deployment. If the region is fragmented, the deployment might fail even though individual VMs would work. This concern is about capacity planning, not a technical issue.

You can also achieve similar results using an availability set. VMs located in the same availability set will receive IB connectivity. However, using VMSS is generally more advantageous because it provides scaling and lifecycle management, although availability sets can be efficient when managing VMs individually.

Documentation: Set up InfiniBand on HPC VMs | Placement groups

A common follow-up question is: “Can we rearrange our currently running standalone ND VMs into a contiguous IB cluster without any downtime?”

The unfortunate answer is no. Azure does not provide a way for customers to move VMs onto contiguous physical hardware. These H100 VM families don’t support live migration, resulting in no method (either customer-initiated or platform-initiated) to move a running VM to different physical hardware for networking purposes.

The correct path forward is to deallocate your current ND VMs, create a VMSS with singlePlacementGroup=true, and then redeploy into this new scale set.

This process does involve downtime and requires that contiguous capacity is available in the region at that moment. If you’re in the early phase of deployment (and many startup operations find themselves there), ensure this is set up properly from the outset. Retrofitting later can be done but is often cumbersome, especially if your region faces capacity constraints.

The worst-case scenario is genuine: you deallocate your VMs, attempt to redeploy them into a placement group, and find the region lacks enough contiguous rack space. You could be left without a functioning training cluster and waiting for sufficient capacity to become available again. Should you be in a crowded region, coordinate with your Microsoft account team to check for contiguous availability prior to planning any maintenance windows.

Azure offers HPC marketplace images (like Ubuntu-HPC, AlmaLinux-HPC) preloaded with NVIDIA GPU drivers, CUDA, NCCL, and Mellanox OFED. Using these images can save you significant time troubleshooting driver issues.

If you require a custom image or container, make sure to include NVIDIA GPU drivers that align with your CUDA version, Mellanox OFED for InfiniBand, and NCCL built to support IB and GPUDirect RDMA.

After deployment, validate your setup with the following checks:

• ibstat should indicate that IB devices are active on every VM. Here’s what a successful outcome looks like:

  CA 'mlx5_0' CA type: MT4129 Port 1: State: Active Physical state: LinkUp Rate: 400

  If you encounter State: Down or Physical state: Polling, it suggests the IB connection isn’t established. Verify that your VMs are indeed part of the same placement group.

• lspci | grep Mellanox confirms that the hardware is recognised by the OS. You should see 8 ConnectX-7 devices on an ND96isr instance.
• Execute NCCL all-reduce tests across nodes with NCCL_DEBUG=INFO set:

  export NCCL_DEBUG=INFO export NCCL_IB_DISABLE=0 mpirun -np 16 --hostfile hosts nccl-tests/build/all_reduce_perf -b 8 -e 8G -f 2 -g 1

When GPUDirect RDMA is functioning correctly, you should observe bus bandwidth hitting around 380+ Gb/s for large messages (8 GB+). If your figures are significantly lower than this, or if NCCL logs indicate NET/Socket rather than NET/IB, check that OFED is loaded (ofed_info), verify that the IB interface is up (ibstat), and ensure your VMs are within the same placement group.

Documentation: Configure HPC VMs | ND H100 v5 specs

Before you kick off your first distributed job, here’s a handy checklist to ensure everything’s in order:

• Confirm you’re using ND96isr H100 v5, not NC40ads H100 v5.
• Ensure your VMs are deployed within a VMSS with singlePlacementGroup=true (or within the same availability set). Standalone VMs will not establish IB connectivity with each other.
• Use either the Ubuntu-HPC or AlmaLinux-HPC image (or a custom image that includes OFED and NCCL).
• Check that ibstat displays IB as Active on all nodes (which should reflect all 8 ports per VM).
• Run an NCCL all-reduce test to ensure bandwidth is as expected for large messages.
• Verify that NCCL logs indicate NET/IB transport rather than NET/Socket.
• Make sure your training framework is set up with backend=’nccl’.

1. ND H100 v5 specifications
2. NC H100 v5 specifications
3. Set up InfiniBand on HPC VMs
4. VMSS placement groups
5. Distributed GPU training guide (Azure ML)