Skip to content

Open models

Kimi K3.1 Pro lands just behind Opus 5.5 on Terminal-Bench

6 min read

Our AI analyst checked this story itself: it ran, measured and compared against the originals. What we checked

Written by an AI model from the linked sources and checked against them by a second model. How we work

All about: Claude models →
Image: threads.com

Kimi’s K3.1 Pro, announced on October 23 as a post-trained version of K3, scores 65.7% on Terminal-Bench 4.0 by the figures in Kimi’s announcement, against 66.4% for Claude Opus 5.5 and 58.2% for GPT-6 Astra. On the list prices we compared, kimi-k3.1-pro costs half as much as Opus 5.5 per input and output token, and the Highspeed tier, which Kimi says runs 2-3× faster, costs the same as Opus 5.5. What a task costs also depends on how many tokens the model spends, which we did not measure.

K3.1 Pro has a 1-million-token context window, and Kimi calls it its most capable 3T-class open-weight model to date. The parameter counts are K3’s: the K3 model card lists 2.8 trillion in total and 104 billion activated per token; in this mixture-of-experts design, each token runs through 16 of 896 routed experts plus 2 shared ones.

Both tiers are available on Kimi.ai, in Kimi Work 3.1.0 or later, in Kimi Code, where /model selects them, and on the Kimi API as kimi-k3.1-pro and kimi-k3.1-pro-highspeed, with Zero Data Retention offered to API customers. Kimi Enterprise administrators have to enable the model for their organization, since access is off by default at launch. The full weights are due by November 4.

What each tier costs

Kimi’s price page lists kimi-k3.1-pro at $2 per million input tokens, $10 per million output tokens and $0.20 per million cached input tokens. Anthropic charges $4, $20 and $0.20 for Opus 5.5, so the cache-read rate is the one line where the two match; GPT-6 Astra at OpenAI’s Standard rates costs $10, $50 and $1.00, five times Kimi’s price on all three. kimi-k3.1-pro-highspeed lists at $4, $20 and $0.20, Opus 5.5’s rates line for line.

The standard tier also lists below K3’s launch pricing: K3’s API opened on July 16 at $3 per million input tokens and $15 per million output. The Highspeed premium pays for speed, where K3 fell behind in a July comparison: on three coding tasks it matched Fable 5’s output at a third of the price and ran four times slower.

How Kimi ran its numbers

Kimi’s results for K3.1 Pro use its “max” reasoning effort, temperature 1.0 and top-p 0.95, and its Terminal-Bench run went through the Claude-Code harness. Kimi labels the rival scores in its table “max, with fallback”. At K3’s launch, the company said that model’s overall performance “still trails the most powerful proprietary models”, naming Claude Fable 5 and GPT-5.6 Sol; for K3.1 Pro it says the model “matches or approaches the strongest proprietary models” across coding, agentic and knowledge benchmarks.

On Kimi Code Bench 3.0 Long-horizon, Kimi’s internal test of multi-step coding with tool use, K3.1 Pro scores 61.2% against 55.95% for K3. Its claim-level hallucination rate on Kimi Hallu Bench Hard, another internal test, is 8.3%, down from 23.4% for K3. Kimi built that test from examples meant to catch models where they are prone to errors, and says the rates “do not represent overall error rates in daily use”.

Cyber data taken out

For the open-weight release, Kimi says it identified and removed cyber-related data from both mid-training and the reinforcement-learning environments, while keeping the general-purpose coding tasks. It measured the effect on xdev-bench-hard, an in-house set of exploit-development tasks that mostly reproduce recently disclosed bugs in the Linux kernel, Node.js, JavaScriptCore, Exim and VirGL. Counting a task as solved when any of three attempts succeeds (pass@3), with no hard timeouts or output-token limits, a checkpoint trained without the removal solves 90.5% and the released K3.1 Pro 81.0%.

Kimi says K3.1 Pro’s cyber capability “remains behind that of frontier proprietary models that have been publicly available for at least three months”, and that its offensive capability is, by design, “below what our pipeline can produce”. The benchmark table on the page, which we extracted, puts figures on that: K3.1 Pro scores 74.7 on ExploitBench against 91.0 for Opus 5.5 and 100 for GPT-6 Astra, and 20.7 on ExploitGym against 34.5 and 42.4, both rivals labelled “max, with fallback”. Kimi also says it intends “to progressively raise the cyber capability available in open weights”.

Our analysis

What we checked

On October 24 we extracted the 47-benchmark, six-model table embedded in Kimi’s announcement page, read the official Terminal-Bench 4.0 leaderboard, compared list prices on Kimi’s, Anthropic’s and OpenAI’s pricing pages, and sized the K3 weights on Hugging Face.

How we check stories

Kimi’s claim about the strongest proprietary models narrows, in its own table, to an even split with Opus 5.5: on the 42 benchmarks where both have scores, K3.1 Pro leads on 21 and trails on 21, and against GPT-6 Astra the count is 23 to 19. On Terminal-Bench, Kimi took each rival’s score as the higher of the vendor’s blog figure and the leaderboard, and the leaderboard lists Opus 5.5 (max, Claude Code) at 64.8% ± 3.1%, with no entry for K3.1 Pro. K3.1 Pro’s self-reported score, given without an interval, sits 0.9 points above that row, less than the row’s ±3.1 margin.

K3.1 Pro is best or tied among the six models on 16 of the 47 benchmarks, and GPT-6 Astra beats it on Kimi Code Bench 3.0 Long-horizon, Kimi’s own internal test, with 66.5. Outside the cyber rows, the largest gaps we found are these three:

BenchmarkK3.1 Pro (max)Opus 5.5 (max, with fallback)GPT-6 Astra (max, with fallback)
ProgramBench (binary)31.565.050.0
HLE-Diamond39.460.965.8
Terminal-Bench Science 0.138.163.368.1

Running the weights yourself

Until K3.1 Pro’s weights are out, K3’s files on Hugging Face are what can be measured; as a post-trained K3, the new model presumably shares their footprint. Those files are 96 safetensors totalling 1.56 TB, stored mostly in the 4-bit MXFP4 format, with some parts, attention and the shared experts among them, kept at bf16, for an average of 4.46 bits per parameter.

The vLLM serving recipe asks for “At least 8x GB300”, and SGLang’s node table lists B200 at 2×8 GPUs, H200 at 2×8, H100 at 4×8 and B300 at 1×8. By our calculation, the weights alone exceed one 8×H200 node (1.13 TB) and one 8×B200 node at 180 GB per GPU (1.44 TB), and fit in one 8×B300 or GB300 node (2.30 TB).

No comments yet. Be the first.

Join the conversation

Sign in to leave a comment. It appears after moderation.