Maia 300 promises Microsoft 40% more tokens per megawatt

Microsoft said on October 12 that it will scale its in-house Azure Maia accelerators to multi-gigawatt deployments starting in 2027, making Maia the primary inference system for its products and services. The second-generation Maia 200 already serves Microsoft Copilot in production. The third generation, Maia 300, starts production deployments next year.
Microsoft’s announcement, signed by Azure chief Scott Guthrie, makes two headline claims for Maia 300. It should deliver 40% more tokens per megawatt than “leading GPU-based systems.” In a full 96-accelerator configuration, it reaches up to 7x the rack-scale token throughput of Maia 200. Microsoft does not name the GPU systems it measured against. Ignite takes place next month in San Francisco, and Microsoft says it will share more about its strategy there.
Maia 200 in production
Maia 200 is deploying across the United States, Europe and Australia. Microsoft says it delivers up to 1.7x more tokens per watt than “leading GPU infrastructure” on the AI features built into Teams, Copilot and Cowork. Besides Copilot, it supports the MAI team as it develops next-generation models. GitHub Copilot and Microsoft Foundry are due to move onto it by the end of the year.
Maia 200 runs more than Microsoft’s own models. Across these workloads it serves OpenAI models, Microsoft’s MAI models and “leading open-source model families.” Microsoft lists the next jobs for the two generations together: multi-step agentic work for GitHub, cloud-scale model serving for Foundry, inference and data generation for MAI, and latency-sensitive interactions that should help Dragon Copilot scale workflows and reduce clinicians’ administrative burden.
Microsoft introduced Maia 200 on January 26 as an inference accelerator, and with it Maia became an inference-focused program. The chip is built on TSMC’s 3-nanometer process. It is a 750-watt part with 216GB of HBM3e at 7TB/s, 272MB of on-chip SRAM and more than 10 petaFLOPS at FP4. Maia 100, announced in November 2023, used TSMC N5 and carried 64 GB of HBM2E at 1.8 TB/s.
What changes in Maia 300
Microsoft designed Maia 300 around memory traffic rather than arithmetic. “Moving data efficiently, rather than raw compute power, is increasingly the bottleneck in large inference systems,” the announcement says. The chip targets long-context, memory-intensive and decode-heavy workloads. Decoding is the phase where a model generates its answer one token at a time, which is why memory bandwidth tends to set the pace.
Eight HBM4 stacks give 18.0 TB/s of peak bandwidth within a 950-watt TDP. Compared with Maia 200, that is roughly 2.6 times the bandwidth for 200 more watts. Specialized kernels also use on-die SRAM as a software-managed scratchpad to reduce pressure on HBM.
The other changes:
- Chiplets: compute, memory-controller and I/O functions now sit on separate dies. The compute chiplets are made on TSMC’s N2P node.
- Low precision: support for formats including an enhanced MXFP4 cuts memory and bandwidth needs for computation and for KV-cache storage, the attention state a model keeps for its context. Microsoft says the result is larger context windows and higher throughput within the same power envelope.
- Software: a “software-defined dataflow architecture” paired with AI-assisted kernel optimization, meant to keep data local and reduce movement across compute, memory and network.
Microsoft has published a separate technical paper on Maia 300’s architecture.
The rack
On networking, Maia 200 uses standard Ethernet with a custom transport layer. It provides 2.8TB/s of bidirectional scale-up bandwidth per accelerator and supports collective operations across clusters of up to 6,144 devices. For Maia 300, Microsoft describes high-bandwidth links between accelerators optimized for tensor parallelism. A custom AI transport runs over Ethernet-based infrastructure, so one topology scales from a single rack to a full datacenter.
A Maia 300 rack holds 96 accelerators in a 167-kilowatt power envelope, with 27.6 TB of HBM capacity and 1.7 PB/s of aggregate HBM bandwidth across one scale-up domain. That comes to about 287 GB of HBM per accelerator, against 216GB on a single Maia 200. Microsoft says this density lets it serve more simultaneous AI interactions and larger workloads from each rack.
Up next
Opus 5.5 traffic may be hitting a new Claude checkpoint
Per TestingCatalog, X users caught Opus 5.5 routing some requests to a new Claude checkpoint. One said it spent 2 hours in Blender MCP building a 3D scene from one sentence. Have you noticed your Opus 5.5 answers jump in quality lately?
Related stories
- Google adds TPU 8i Max for big contexts, 8tx for secure data
- Copilot on Windows will hand some tasks to local models
- Microsoft AI’s MAI-Base-2 trained on 16K GPUs with Muon
- Microsoft brings streaming MAI-Transcribe-2 to its APIs
- Mythos put Claude on AMD in about 3 days, not 6-12 months
- Jensen Huang quits NVIDIA, CFO Colette Kress steps in
Comments
No comments yet. Be the first.
Join the conversation
Sign in to leave a comment. It appears after moderation.

