Skip to content

Research

Microsoft AI’s MAI-Base-2 trained on 16K GPUs with Muon

6 min read

Written by an AI model from the linked sources and checked against them by a second model. How we work

Image: threads.com

Microsoft AI has finished pre-training MAI-Base-2. In an October 15 Microsoft AI post, the lab says the model is 4-12x more pre-training FLOPs-efficient than DeepSeek-V4-Pro-Base, which it calls the strongest open base model. It also says that saving went into training a larger model.

The headline claim, in the company’s words: “MAI-Base-2 outperforms every open-weight model we tested on our held-out evaluations.” The comparison chart covers only models with at least one trillion total parameters, across a range of pre-training evaluations where lower is better. Note who ran the scoring: these are Microsoft’s own held-out evaluations, against open-weight models it chose to test.

The base model sits under a product line that already ships. At Build on June 2, Microsoft AI announced seven in-house MAI models, among them MAI-Thinking-1, a mixture-of-experts model with 1T total and 35B active parameters trained on about 33T tokens. Microsoft said all of them were trained from scratch, with no distillation from other labs. Narrower products are lined up behind the base models too: Microsoft plans to rebase its Decision-1 scoring model, now a post-trained Qwen3.5-9B, onto MAI and OpenAI models.

Most of the post is about the engineering behind the efficiency figure on a 16K-GPU cluster: how sparse expert layers are scheduled, what the Muon optimizer costs per step, and a change to gradient clipping.

Experts with a fixed token budget

In a mixture-of-experts layer, a router sends each token to a few experts, and some experts always get more tokens than others. That imbalance usually shows up as unpredictable memory use. Microsoft groups the experts on each expert-parallel rank into groups of configurable size. Each group runs one or more dispatch → compute → collect rounds, and every expert has a fixed token capacity per round. Tokens past an expert’s capacity are handled in extra rounds. To keep the number of extra rounds down, an overloaded expert can use spare capacity from other experts in its group.

The backward pass recomputes per expert group on a similar multi-round schedule. Because of that, memory use stays static however unevenly tokens are routed. Capacity is the tuning knob: more capacity costs memory, while less capacity can mean more rounds and extra compute. In practice Microsoft sets it so that steady-state training runs a single round.

Dispatch, compute and collect are pipelined across all rounds of all expert groups, so only the first dispatch and the last collect are left outside expert compute. In the forward pass, weight quantization and shared-expert compute fill that exposed communication time. In the backward pass all expert communication is hidden: the exposed input dispatch overlaps with recomputing the same layer, and the exposed gradient collect overlaps with weight-gradient computation.

On top of the scheduling, Microsoft cuts per-expert overhead in four ways:

  • grouping both computation and communication per expert group, which improves GEMM and NVLink efficiency and reduces kernel-launch overhead and tail effects;
  • keeping token metadata on the device and starting all-to-all communication directly from the GPU, which cuts host-device synchronization;
  • using symmetric memory so tokens move directly over NVLink without staging copies, which raises communication throughput and reduces HBM contention while GEMMs run alongside;
  • batching the load-balancing loss across layers, so one batched reduction replaces an all-reduce per layer.

The payoff, the post says, is the ability “to train sparser models without significantly degrading MFU”, the share of the hardware’s peak arithmetic a training run actually uses.

What Muon costs per step

The optimizer changed too. When scaling up to MAI-Base-2, Microsoft found that Muon gave more stable training dynamics and large efficiency gains over AdamW. The post gives no number for that gain. For an outside reference, Moonshot AI’s paper “Muon is Scalable for LLM Training” reported that in its scaling-law experiments Muon reached about 2x the computational efficiency of AdamW under compute-optimal training, and that scaling it up needs weight decay and careful adjustment of the per-parameter update scale.

Muon orthogonalizes the momentum of each weight matrix with Newton-Schulz, an iterative algorithm made of several matrix multiplications. It needs the full matrix. ZeRO, which splits optimizer state across the data-parallel GPUs to save memory, keeps only a shard of that momentum on each one. Following prior work, Microsoft reshards the momentum during the optimizer step:

  • each GPU updates its local FP32 momentum shard and casts it to BF16;
  • an all-to-all exchange, batched over parameters of the same shape, hands each GPU a subset of complete momentum matrices;
  • each GPU orthogonalizes those matrices locally with a symmetry-aware Newton-Schulz kernel;
  • a second all-to-all sends the updates back to their original shards, where they are cast back to FP32 and applied to the weights.

At 16K GPUs, three further tweaks matter. Communication and computation overlap by prefetching multiple batches. The resharding metadata is cached and reused instead of recomputing the indexing each step, which helps most in sparse MoE models with thousands of identically shaped parameters. And some weights, such as the MoE router weights, are replicated on every GPU rather than sharded: for them the first exchange is skipped, and an all-gather after Newton-Schulz collects the full set of replicated updates.

After that work, Muon took 2.1% (165 ms) of MAI-Base-2’s average training iteration, which puts a full iteration at about 7.9 seconds. Microsoft estimates the theoretical peak for Muon in MAI-Base-2’s final training setup at 55 ms, limited by communication, so the measured cost still sits well above the lab’s own floor. The team blames the gap on stragglers, kernel dispatch overheads, load imbalance, and memory copies and allocations during resharding. It tracked the implementation with both large-scale and small-scale test setups, comparing measurements against that estimated peak and choosing changes from profiles.

Clipping where the outlier is

Standard gradient clipping works on the global norm of the weight gradient for a whole batch. Outlier gradients usually come from a small part of the data in that batch, so one outlier can scale down the entire batch’s gradient, including the well-behaved examples. Microsoft instead clips the gradients of activations at selected places in the network. Combined with Muon, the post says, this keeps MAI-Base-2’s gradient norm in a much tighter range than MAI-Base-1’s.

MAI-Base-3 is next. Testing Catalog says on Threads that its training cycle is already being prepared.

No comments yet. Be the first.

Join the conversation

Sign in to leave a comment. It appears after moderation.