Skip to content

OpenAI

Stock OpenAI models saved Jalapeño over 13% of die area

7 min read

Written by an AI model from the linked sources and checked against them by a second model. How we work

All about: OpenAI models and products →
Image: pexels.com

OpenAI’s internal models, used without any fine-tuning, saved more than 13% of die area in one example on the Jalapeño inference chip, Richard Ho, who leads OpenAI’s hardware program, said in an interview with More Than Moore on Substack published September 30. The models’ work still has to clear standard EDA sign-off before tape-out, and the edge over Nvidia’s GB200/GB300 that OpenAI showed at Hot Chips 2026 comes from a short sprint on a third-party benchmark: three open-source models, with no speculative decoding on OpenAI’s side.

Solve for X

OpenAI had promised Jalapeño’s performance targets based on simulation and on estimates of how much logic would fit in the available area. Those estimates turned out to be a little off, which left the team looking at a performance hit. It turned to the models instead: “Solve for X,” as Ho puts it. They were raw models, mostly internal ones he calls “a little bit ahead of the public models” but “not that much more capable”. Good hardware data was scarce enough that the team was scrambling to find more hardware repositories to feed them.

Ho’s explanation for why it worked is breadth of view. The models are “very systematic”, he says, and hold enough of the design in their context window to find soft spots across the whole chip, while a human struggles to correlate one part of a design with another. He also rates OpenAI’s own model as “a lot stronger” at this work than the frameworks EDA vendors are building around models.

Much of the optimized code was written in XLS, a high-level synthesis tool with Rust-like syntax and semantics that team member Chris Leary wrote and open-sourced at Google. That helped the models, Ho says, because at the time they did not fully understand SystemVerilog but could reason about software. Models also tuned kernels once the hardware existed: one attention implementation went from under 1% of its memory/compute roofline, the ceiling set by the chip’s bandwidth and arithmetic, to nearly 90% in roughly 40 hours.

Vendors pitching AI design tools to OpenAI get a standing test from Ho. They have to use AI to design a PCIe controller that passes OpenAI’s compliance suite, do it faster and better than his team, and deliver a smaller area with the power and performance to match. “If you can do that for me, then send me an e-mail,” he says.

Where the models stop

Ho still doesn’t “100% trust” the models and doesn’t expect them to replace EDA tools in the near term. On a design of around 300 million gates, “if the model is 99.99% correct, you can’t tape that out,” so sign-off runs through standard flows. At first the team didn’t understand some of the source-code optimizations the models made, and it was moving too fast to stop and work them out. The full validation flow that followed was there to make sure nothing else broke. “It does worry me,” Ho says, though he calls it a good risk as long as it is taken deliberately.

Simulation and FPGA models correlated with the silicon, by his account. Analog parts such as signal integrity were the one area where the team wasn’t sure things would work. He also hesitates to use the models as a reason to shrink the team: “do more, do it better, do it faster with the models.”

What the Hot Chips numbers measure

At Hot Chips 2026, Ho, Ravi Narayanaswami and Chris Leary presented the first performance measured on working Jalapeño silicon. Against Nvidia’s GB200/GB300, OpenAI claimed 1.5x to 1.9x better performance per watt at peak throughput, 1.7x to 3.6x better latency, and up to 104x better at operating points Nvidia struggles to reach.

The chip came back around mid-May, and OpenAI confirmed its Hot Chips slot late. For the sprint that followed, the team picked InferenceX, a third-party end-to-end benchmark with defined metrics, so that the numbers would be “unassailable” on methodology. It also invited SemiAnalysis in to evaluate the chip. The catch is that InferenceX uses open-source models that won’t run on Jalapeño in production. OpenAI picked three of them to cover a range of sizes.

Speculative decoding needs a draft model, and OpenAI had none for those open-source models. So its runs used single-token prediction, while production runs multi-token prediction “all the way down”, Ho says. The goal was for OpenAI’s single-token numbers to beat the published multi-token ones, and he says they did. As he describes it, that setup handicaps Jalapeño rather than flatters it.

OpenAI may not publish much more of this. The team has moved on to production work, and it may run AgentX when it gets a chance but probably won’t spend much time optimizing results, according to Ho.

One chip for the whole pipeline

Jalapeño pairs 216 GiB of HBM4 at 15.4 TB/s with a compute die and an I/O chiplet. It is rated at 700 W peak, with a measured sustained draw closer to 550 W. A local domain is 128 accelerators, and a full system of 2,048 accelerators reaches 27 EFLOP/s at four-bit precision.

Other hardware vendors are splitting inference into prefill, speculative decode and full decode, with specialised hardware for each stage. OpenAI built a single part for all of it. Ho’s reasoning is about the whole fleet. A split forces you to commit a fixed ratio of hardware, capex and power to each stage, and those ratios “flip around a lot”, so a part that can move between workloads keeps regret to a minimum. On whether that generality costs efficiency in any one stage, he says a tax “may be identifiable when we get into a real deployment, but it’s not identifiable right now.”

The memory design ties HBM banks to specific cores, so work stays inside one bank and its core. Nvidia’s SMs work the other way, sharing one L2 that any of them can reach. OpenAI has not yet found an open-source model it could not map onto the chip. The limit Ho names is very large context lengths, where “you may run into some trouble”; the team hasn’t hit it or analysed it yet. He expects others, GPU makers included, to follow the design, and says OpenAI made it public because “we still need GPUs, and we want GPUs to get better.”

From A0 to volume

Jalapeño is technically OpenAI’s first chip, but its I/O chiplet arrived first and was brought up early. The A0 and B0 steppings were planned from the start, and A0 carried every performance feature needed to bring up the software while the team finished the design. Broadcom’s release at the June 24 unveiling put the span “from design to production” at nine months. A September account counted under 20 months from first architecture concept to first silicon, with a team that averaged fewer than 100 people. The two figures start the clock at different points. Ho says the TPU generations he worked on at Google were built by teams of around 300.

Broadcom handles silicon implementation and networking, and Celestica handles board, rack and system integration. Ho chose Broadcom for its SerDes and controller IP, for a working history that goes back a decade for some of his engineers, and for volume: “if you’re a startup team in a software company, are you going to get those wafers at TSMC? Not off the bat.” He says the part OpenAI sped up was design and bring-up, which is independent of the wafer queue. Of the volume ramp now under way, he allows that “there may be some things going on there.”

The name comes from the team’s nickname, the Hot Peppers, which it took from a spice rack in the cafeteria of its old Mission district building. Ho didn’t rule out Scotch Bonnet for a successor.

B0 is in the lab now, going through qualification.

No comments yet. Be the first.

Join the conversation

Sign in to leave a comment. It appears after moderation.