Skip to content

OpenAI

First results land for OpenAI’s Jalapeño inference chip

8 min read

Written by an AI model from the linked sources and checked against them by a second model. How we work

All about: OpenAI models and products →
Image: openai.com

OpenAI on August 25 published first results for Jalapeño, the inference chip it built with Broadcom, and the standout figure comes from SemiAnalysis: more than 700 tokens per second per user on DeepSeek R1 at a concurrency of 1, measured with its InferenceX benchmark in OpenAI’s lab. That run used single-token prediction with no speculative decoding, a technique where a small draft model proposes several tokens and the main model checks them in one pass.

In its post on the first results, OpenAI describes faster, more power-efficient inference, with higher throughput and lower latency on modern models. The chip was presented at Hot Chips. OpenAI invited SemiAnalysis to its labs to check the hardware and benchmark it, and SemiAnalysis says Jalapeño beat every Nvidia, AMD and Google chip it has been able to test, across several leading open-source models. It adds that first-generation chips are generally not competitive.

One pool of chips

Much of the coverage since the June unveiling followed remarks from OpenAI that the chip would be optimized for its own models. SemiAnalysis disputes that reading. It calls Jalapeño a general inference chip that runs all sorts of models and workloads, including the third-party InferenceX suite. As a joke, OpenAI showed the analysts Doom running on the chip, ported with nothing but Codex prompts.

The main architectural choice concerns where the two stages of inference run. Prefill reads the prompt, and decode generates the answer one token at a time. Many large deployments split these across separate pools of hardware. OpenAI keeps them together: the draft model and the main model share the same chips and the same fabric, with no prefill-decode split. SemiAnalysis says OpenAI’s reasoning is that the workload mix keeps shifting. The ratio of input tokens to cache writes, cache reads and output tokens has changed considerably as models moved from knowledge to reasoning to agentic use. Commit a fixed share of silicon to prefill and another to decode up front, and that split can turn inefficient over time. So OpenAI built one homogeneous pool and tried to make the chip perform well at everything, trading some theoretical efficiency for simpler operations.

That fits what Richard Ho, who leads OpenAI’s hardware program, said at the June unveiling. The chip was architected from a blank slate around the kernels, memory movement, networking and serving patterns of frontier models, for inference only and not training. Ho listed less data movement between compute and memory, a better balance of compute, memory and networking, and utilization “much closer to theoretical peak performance” than existing solutions.

On the silicon, SemiAnalysis gives these details:

  • A single reticle-sized compute die on TSMC’s N3P. In the B0 stepping it delivers 13.4 PFLOPs of MXFP4. A Rubin compute die of similar size on the same node delivers 17.5 PFLOPs of dense NVFP4.
  • A TDP of 700W, against 900-1,150W per Rubin compute die.
  • HBM4 memory. SemiAnalysis says this puts OpenAI among the early adopters after Nvidia and AMD, ahead of the TPU and Trainium programs.
  • An N3E I/O chiplet with 32 lanes of 800G SerDes. 24 lanes (600GB/s) handle scale-up within the rack, and 8 lanes (200GB/s) connect to a multi-rack domain of 2,048 chips.
  • PCIe Gen 5 to an x86 host CPU.

Set against other accelerators, SemiAnalysis says Jalapeño has the highest HBM bandwidth per watt and the highest FLOPs per watt, a level comparable to the 1,800W Rubin Max-Q configuration. Because the chip is aimed at inference rather than training, OpenAI does not need to push TDP higher to maximize FLOPs, the analysts say.

OpenAI designed the chip. Broadcom handled the silicon implementation and supplies Tomahawk networking silicon, and Celestica builds the boards and racks and does system integration.

The numbers on 8k1k

Every test ran on 8k1k, a single-turn workload with roughly 8,000 input tokens and 1,000 output tokens per request. SemiAnalysis’s headline metric is output tokens per second per megawatt of all-in utility power. On that chart, it says, “Jalapeño smokes every other chip”. Jalapeño ran without multi-token prediction, while the other chips on the chart appear in the best-performing configuration of each SKU, all with multi-token prediction. The analysts add that Jalapeño leads Blackwell on performance per watt in almost every scenario, at both low-latency and high-throughput points, without being tuned for any one point on the curve.

The per-model results, all with single-token prediction, no speculative decoding and no prefill-decode split:

  • DeepSeek R1: more than 700 tokens per second per user at concurrency 1.
  • Kimi K2.5, the base of Cursor Composer 2.5: nearly 700 tokens per second per user, and, in SemiAnalysis’s words, “more than 9x the next best performing chip at 100tok/s/user”.
  • GPT-OSS: about 1,400 tokens per second per user. Its throughput per megawatt at matched interactivity is nearly double GB200’s highest throughput point and more than 50x GB200’s concurrency-1 point. The higher-concurrency Jalapeño points use an EP8 configuration.

SemiAnalysis checked GSM8k scores for every model and found them on par with Nvidia chips.

All of this was measured on the A0 stepping, nine months into the program. The B0 stepping is already in the fab, and SemiAnalysis says it brings roughly 25% better performance per watt than A0.

Blackwell or Rubin?

SemiAnalysis lists its own caveats next to the results. All numbers were provided by OpenAI. The analysts verified the InferenceX runs in person, but they did not run the full suite and have not seen results on AgentX, their preferred benchmark. AgentX uses long-context, multi-turn datasets that stress routers, prefix caching, cache management and offload, and 8k1k tests none of those. The tested models are also not on the open frontier: Nvidia and AMD have already published AgentX results on larger models, DeepSeek V4 Pro and Kimi K3.

The analysts also call the Blackwell comparison somewhat incomplete and unfair. Jalapeño uses HBM4, so its real rival is Rubin. In July, Nvidia and CoreWeave published Vera Rubin figures that SemiAnalysis says show Vera Rubin NVL72 at 5.4x the performance per megawatt of GB200 NVL72. Even against that, Jalapeño’s single-token output throughput per megawatt beats Rubin’s multi-token results. SemiAnalysis compares July 2026 Rubin numbers with 2025 GB200 numbers and today’s Jalapeño numbers, reasoning that all three come from a similar early stage of bring-up.

On performance per total cost of ownership, the two come out level, with nearly the same number of output tokens per dollar. But Rubin’s results use speculative decoding and Jalapeño’s do not. SemiAnalysis puts the cost-per-token gain from speculative decoding at 3x or more, which is why the analysts expect much cheaper Jalapeño tokens once it is implemented. They attribute part of the cost edge to swapping Nvidia’s margins for Broadcom’s, which are lower but still high. In June, Broadcom CEO Hock Tan put the savings at roughly 50% against current AI GPUs, per Bloomberg.

Timing still separates the two designs. Vera Rubin systems are starting to ship to customers now, while, SemiAnalysis says, it will be some time before OpenAI has anything beyond engineering samples of Jalapeño.

Why tokens per megawatt

OpenAI optimizes for performance per watt because, SemiAnalysis says, it is currently constrained by datacenter power, not by budget or floor space. Hardware can be added much faster than grid capacity. Utility interconnects, cooling and backup generation all set limits, and grid delays have pushed operators toward gas turbines and on-site generators. Tokens per second per megawatt works out to tokens per joule, which makes it a direct measure of how efficiently a system turns energy into output.

Nvidia uses the same arithmetic in its pitch. At Computex 2026, Jensen Huang said: “If you have 1 gigawatt of power, then throughput per watt is revenue.” At the Vera talk at Hot Chips 2026, Nvidia put it more briefly: “The data center is power limited today.”

Other labs’ silicon

Anthropic told Business Insider it is staffing an internal custom silicon team for Claude while keeping AWS, Google, Nvidia and AMD hardware. The Information reported that Alphabet is developing “Frozen v2”, a chip that etches parts of Gemini into the silicon. SemiAnalysis points to Meta’s and Microsoft’s AI ASIC programs, which have not gotten off the ground despite running much longer, as a sign that cost is only one part of the equation.

Nine months to tape-out

When OpenAI and Broadcom unveiled the chip on June 24, they gave nine months from initial design to tape-out and called it possibly the fastest ASIC development cycle ever achieved in high-performance advanced semiconductors. SemiAnalysis puts the start of design work in mid-2024 and says the path from initial team hiring to tape-out took about 16 months. Greg Brockman credited close hardware-software co-development, Broadcom’s implementation work and OpenAI’s own models, which sped up parts of the design and optimization. “The degree to which our models have been able to accelerate it was very surprising to us,” he said, per CNBC.

No comments yet. Be the first.

Join the conversation

Sign in to leave a comment. It appears after moderation.