OpenAI built idle silicon into Jalapeño on purpose

By OpenAI's own arithmetic, shown at Hot Chips 2026, 128 chips with a petabyte per second of aggregate HBM4 bandwidth should be able to read a roughly 1-trillion-parameter FP4 model about 2,000 times a second. Real single-user token rates sit at 20–200 per second, and closing that gap is what shaped Jalapeño, OpenAI's inference accelerator, as Siliconcodesign reconstructs from the presentation deck.

At a glance

• OpenAI put prefill, speculate and decode on one balanced die with the KV cache sitting next to compute and memory, instead of chaining specialized accelerators that go idle between phases.
• The chip is TSMC 3nm, draws 700W, pairs a compute die with six HBM4 stacks for 15.4 TB/s and 216 GiB, and does 3–13 PFLOPS/s on fp4 and fp8 matrix multiplication.
• Multi-token prediction is wired into the microarchitecture, but OpenAI is running single-token prediction workloads first to prove out the silicon, and Jalapeño handles inference only.

If you missed the earlier beats: ServeTheHome says OpenAI's project timeline slide puts the architecture concept phase in late 2024, and Futurum reports the chip went from initial design to tape-out in nine months, a cycle OpenAI believes is the fastest in high-performance advanced semiconductors. Along the way the team used Google XLS, an open-source high-level synthesis tool, for block-level PPA optimization and formal verification with OpenAI's own models.

The memory roofline says 2,000 model reads a second; users see 20–200 tokens

The roofline assumes those 128 chips, 1 PB/s of HBM4 bandwidth and a roughly 1-trillion-parameter model at FP4. Read the full weight set about 2,000 times a second, add speculative decoding, and the theoretical ceiling rises to 5,000–10,000 tokens per second for one user.

Reality is 20–200. The losses come from inter-chip network latency, synchronization delays and contention in on-chip routing, which leaves HBM idle between tokens. Operators raise batch size to saturate that bandwidth, which lifts server throughput and does nothing for your own prompt.

So OpenAI worked backwards from the user: time to first token during prefill, time per output token during decode, time to last token for the whole request. Each service tier gets an end-to-end latency SLA, and only once that holds is compute efficiency, requests per second per watt, optimized.

Ravi Narayanaswami would rather ship idle blocks than missing features

Prefill pushes all prompt tokens through the model in parallel and fills the KV cache; it is compute heavy and stresses power delivery, thermals, local SRAM and interconnect. Speculate runs a lightweight draft model and leans on SerDes and optics, where packet latency decides the payoff. Decode is memory bound, streaming weights and the accumulated KV cache for every token, with all-to-all traffic in MoE models.

Those ratios swing from request to request. You could build a specialized chip per phase and chain them, but then accelerators sit idle; Narayanaswami's line was that "opportunity cost is higher than marginal cost because it has a greater regret factor". OpenAI's conclusion is that dark silicon on one balanced die is the lesser evil.

Large context files push 10s to 100s of GB per request into the KV cache, and on a split design all of it crosses the external network. Jalapeño keeps the KV cache close to compute, memory and the network so data movement balances across all three.

One Tomahawk 6 hangs 128 Jalapeños off 800GBps links

Global reductions, all-reduce and reduce-scatter, keep parallel workers in step, so one late packet stalls the rest: the straggler effect. The budget is tight. An HBM read costs 30–50 ns by itself, every switch on the path adds more, and time of flight is small next to that hardware overhead.

The fabric is a half-flattened Clos with a hard two-hop bound. In the local domain a single Broadcom Tomahawk 6, a 102.4 Tbps Ethernet switch die with up to 512 ports at 200GbE, connects 128 Jalapeños at 800GBps each, which covers tensor parallelism. Eight of them tie 2,048 chips together for MoE parallelism. Celestica handled rack integration.

A 2-tier fat-tree was passed over because three hops add tail-latency jitter and power, a 3D/4D torus because high average hop counts make link latency unpredictable, dragonfly because non-uniform global paths need adaptive routing that hurts predictability again. The pod is 2,048 chips across roughly 28–32 racks: 27 EFLOPS/s, 32 PB/s, 432 TiB.

3nm, 700 watts, 15.4 TB/s and 216 GiB per chip

Jalapeño is a TSMC 3nm compute die with six HBM4 stacks, three on each side, for 15.4 TB/s and 216 GiB. It draws 700W and delivers 3–13 PFLOPS/s on fp4 and fp8 matrix multiplication. For scale, the breakdown puts Nvidia's B200 at 9 PFLOPS/s and B300 at 15 PFLOPS/s in fp4 on 1000W-plus parts, with the caveat that Jalapeño is inference only.

Inside, the chip is spatial rather than the temporal SIMT of a GPU. Independent core slices are tiled with their own HBM interfaces, their own L1 view and local tensor, SIMD and scalar units, streaming core to core without a central warp scheduler dispatching work. Gluon, OpenAI's kernel language built on Triton, keeps the familiar thread-block abstraction over that layout.

According to ServeTheHome, OpenAI benchmarked on InferenceX, normalized to package TDP (Jalapeño at 700 watts, GB200 at 1.2 kW, GB300 and MI355X at 1.4 kW), and reported about 1.9x higher peak mixed tokens per second per kilowatt and roughly 1.7x lower end-to-end latency on GPT-OSS 120B at matched operating points. The same report says GPT-OSS, DeepSeek R1 and the 1-trillion-parameter Kimi K2.5 all run, none co-designed for the chip.

Speculative decoding is guessing how a friend will finish a sentence and waiting for the nod. A small draft model proposes the next several tokens, the large model verifies them in one pass, and only accepted tokens survive, so the text matches ordinary decoding. Single-token prediction spends eight forward passes on eight tokens; the multi-token variant, per ServeTheHome, uses seven tiny draft turns plus one batched trunk pass, cutting expensive large-model passes by up to 8x.

Jalapeño is an inference part, and the comparison figures ServeTheHome reports are power-normalized at matched operating points on one open model, so they say little about other shapes of workload. The switch pick is the loose end: in our view, 600–700 ns of Tomahawk 6 latency is a lot to carry in a design built around tail latency, when Tomahawk Ultra fixes latency at 250 ns for half the throughput at 51.2 Tbps.

Where the gigawatt rollout lands According to Futurum, gigawatt-scale deployment alongside Microsoft and other partners is set to begin at the end of 2026, and engineering samples are already running ML workloads in the lab at production target frequency and power, including GPT-5.3-Codex-Spark. What the deck does not give is a date for switching production traffic from single-token prediction to the multi-token prediction the microarchitecture was drawn around.