OpenAI built idle silicon into Jalapeño on purpose

OpenAI designed its Jalapeño inference chip around one gap. By the team’s own arithmetic, 128 chips sharing 1 PB/s of HBM4 bandwidth could read the full weights of a ~1T-parameter FP4 model about 2,000 times a second, yet single users on today’s systems get 20 to 200 tokens per second. The figure comes from OpenAI’s Hot Chips 2026 talk, “You Can Just Build Things … Chips,” given by R. Ho, Ravi Narayanaswami and C. Leary. A case study by Siliconcodesign, published September 12, works through the deck and related material to show what OpenAI built in response.

OpenAI calls Jalapeño a general-purpose AI inference accelerator. It is designed mainly for OpenAI’s own workloads but meant to carry over to other frontier models. OpenAI had already published first results claiming industry-leading speed and efficiency. The Hot Chips deck explains why the chip looks the way it does, and the case study’s author says up front that it is no “magic black box that replaces GPUs overnight.” The deck instead lays out a series of system-level tradeoffs.

Where the 2,000 comes from

The ceiling is theoretical. It treats HBM throughput as the only physical constraint and assumes 1 PB/s of aggregate HBM4 bandwidth across 128 chips and a ~1T-parameter model stored in FP4. OpenAI’s deck says speculative decoding can raise it to 5,000 to 10,000 tokens per second per user.

Observed rates sit one to two orders of magnitude below that. The deck blames the overheads of parallelism: latency between chips, synchronization delays and contention in the network-on-chip, the routing fabric inside a die. Because of them, HBM sits idle between tokens. Operators make up for it by batching many requests until the bandwidth is saturated. That helps total server throughput but does nothing for any one user’s prompt.

OpenAI therefore started from what users feel. The talk breaks latency into time to first token (prompt processing in prefill), time per output token (generation speed in decode) and time to last token (the whole request). Working back from ChatGPT, real-time API clients and internal agentic workflows, OpenAI sets an end-to-end latency target for each service tier. Only once that target is met does it optimize compute efficiency, measured in requests per second per watt. Of the many tradeoffs a chip architect could chase, the deck narrows to two: end-to-end request latency against mixed throughput per kilowatt, and decode speed against the same throughput metric. The author contrasts this with traditional vendors, who tend to optimize peak theoretical FLOPs and unconstrained batch throughput.

One chip, some of it dark

LLM inference has three phases that load the hardware differently. Prefill pushes the whole prompt through the model in parallel and fills the KV cache, the stored attention state for the conversation so far. It is compute-heavy and strains power delivery, thermals, local SRAM and interconnect bandwidth. Speculation is optional. A cheap draft model proposes the next several tokens, and one verification pass checks all of them at once. This phase strains the optical and SerDes links, because latency matters as data moves between stages. Decode produces one token at a time, reading the weights and the accumulated KV cache on every pass. It is memory-bound and, in mixture-of-experts models, bursty.

Request mixes vary widely, so no fixed split of silicon fits every request. One option is a dedicated chip for each phase, chained together, with some of those chips sitting idle. OpenAI went the other way and put all three phases on one balanced chip, with the KV cache kept close to compute, memory and the network. The company considers dark silicon, blocks that go unused for long stretches, a lesser evil than idle specialized accelerators. The deck also notes that with separate chips the whole KV cache would still have to cross the external network, with users passing in files that add up to tens or hundreds of gigabytes per request.

Narayanaswami’s argument was that missing a feature later does more damage than paying for an unused one now. “Opportunity cost is higher than marginal cost because it has a greater regret factor,” he said. The author compares this to how CPUs are kept general enough for workloads nobody can specify yet.

Two hops to any chip

Tail latency is the network problem OpenAI focuses on. Inference relies on collective operations such as all-reduce and reduce-scatter to keep parallel workers in sync, so a single late packet can hold up a large computation even when every other packet arrives on time. In a processing-element array that runs in lockstep, an operand that arrives late leaves compute units waiting. The deck lists the sources. One is network latency from time of flight and hardware complexity. Another is the memory system, where HBM reads and writes take 30 to 50 nanoseconds and shared memory paths contend at NoC routes, crossbars and banks. The case study adds ballpark figures from an ISSCC 2026 forum by Nikola Nedovic: forward error correction adds tens to hundreds of nanoseconds of fixed delay, while time of flight is small next to the other sources.

The answer is what OpenAI calls a “half flattened” two-hop Clos network built on Broadcom’s Tomahawk 6, a 102.4 Tb/s Ethernet switch chip with up to 512 ports at 200GbE. In the local scale-up domain, one TH6 connects 128 Jalapeños at 800 GBps each, enough for tensor parallelism. In the global domain, eight TH6s connect 2,048 chips, sized for mixture-of-experts traffic. Any chip can reach any other in at most two hops. Celestica handles the rack-scale integration.

The case study lays out the topologies OpenAI passed on and why:

• 2-tier fat-tree, used in Nvidia’s GB200 InfiniBand SuperPOD reference architectures: three hops between racks add tail-latency jitter and power.
• 3D/4D torus or mesh, used in Google’s TPU 8t: cost-effective for regular tensor-parallel traffic, but high average hop counts make link latency unpredictable.
• Dragonfly, used in Google’s TPU 8i: it saves on optical transceivers, but its global paths are non-uniform, and relieving congestion takes adaptive routing that makes packet timing less predictable.

The author also questions part of OpenAI’s choice. Tomahawk 6 is built to cut hops in scale-out networks and has a latency of 600 to 700 ns, while the Tomahawk Ultra has a fixed 250 ns at half the throughput, 51.2 Tb/s. In the author’s view, the Ultra would have been the better fit for the scale-up domain. The author also thinks OpenAI skipped co-packaged optics, the low-latency links drawing much of the industry’s investment, because they are not ready for real-time deployment.

Broadcom’s role goes beyond the switch. On October 13, 2025, OpenAI and Broadcom announced a collaboration on 10 gigawatts of custom AI accelerators: OpenAI designs the chips and systems, and the two companies develop and deploy them together. Rack deployment was set to begin in the second half of 2026 and finish by the end of 2029.

Spatial cores, GPU-style code

GPUs run thousands of threads in lockstep under warp schedulers. Jalapeño uses a spatial design instead: independent core slices are tiled with dedicated HBM interfaces, and specialized collectives let cores stream data directly to each other. Each core has its own L1 memory view and runs its tensor, SIMD and scalar units without waiting for a central controller to dispatch instructions.

Spatial architectures are hard to program, and OpenAI’s fix is Gluon, its own kernel language. It is built on the open-source Triton compiler but gives finer hardware control. Gluon lets engineers write kernels with familiar GPU abstractions such as thread blocks, without hand-building spatial dataflow graphs.

The decode path was designed around speculation. The deck compares single-token prediction, where eight forward passes produce eight tokens, with multi-token prediction, where seven passes run on a cheap graph model and one runs on the large model. OpenAI chose MTP for user experience, energy efficiency and higher token throughput, and engineered the microarchitecture around it from the ground up. For now, though, OpenAI is running single-token prediction workloads to prove out the silicon, so the mode the chip was built around is still waiting its turn.

The spec sheet

Jalapeño is made on TSMC 3nm. One compute die sits between six HBM4 stacks, three on each side, for 15.4 TB/s and 216 GiB of memory. The chip draws 700W and delivers 3 to 13 PFLOPS on fp4 and fp8 matrix multiplication. For comparison, the case study gives Nvidia’s B200 at 9 PFLOPS and B300 at 15 PFLOPS of fp4 on chips drawing over 1000W, and notes that Jalapeño only handles inference. In the author’s view, most of the gains can be attributed to HBM4.

A full 2,048-chip cluster takes up a pod or datacenter row of roughly 28 to 32 racks. It delivers 27 EFLOPS, 32 PB/s and 432 TiB in total.

The later sections of the case study cover how the chip was designed. OpenAI used Google XLS, an open-source high-level synthesis tool, with its frontier models to optimize blocks for power, performance and area and to run formal verification. The author credits XLS with shortening the schedule but argues that the iterative, cross-domain way the team worked mattered more. Greg Brockman described the same effort when the Broadcom deal was announced, saying, per CNBC, that OpenAI took components humans had already optimized and poured compute into them, after which “the model comes out with its own optimizations.”