Jev runs 1,000 code reviews for $0.04, scoring 98%

A benchmark run on September 17 tested TypeSafe AI’s Jev against Gemini Flash and Claude Fable on one narrow code-review job. Jev cost 45 times less than Flash and 274 times less than Fable, and it scored 98% correctness where both rivals scored 100%. The repository, published on GitHub, includes the method, the raw data and a runner for repeating the study.

The job: give each model a piece of Python and four rules, and ask whether the code follows each rule. The models tested were jev-1.13.0, google/gemini-3.8-flash and anthropic/claude-fable-5.1. Both comparison models ran at medium reasoning effort. The team tried a cheaper reasoning setting for Gemini, but the API rejected it, so the report says the study does not show an advantage over every model or setting.

Jev went into limited early access on September 15, two days before the run, alongside a $40 million seed round led by DCVC. It is not a text generator. It takes an input plus typed questions and returns answers from options defined in advance, with probabilities and confidence scores attached, TypeSafe says in its launch post. All outputs are produced in one pass, where an LLM generates one token at a time. Input costs $0.042 per million tokens, and output is free.

What 360 reviews cost

Each model handled 360 reviews in the main study. The actual charges were $0.01545 for Jev, $0.69965 for Flash and $4.24097 for Fable. Scaled linearly to 1,000 reviews, that comes to $0.043, $1.943 and $11.780. The report notes that the per-1,000 figure is a linear estimate for the same workload. It is not an actual charge, not a separate volume test and not a price guarantee.

Median response times were 0.75 seconds for Jev, 3.59 for Flash and 4.31 for Fable. TypeSafe itself claims end-to-end responses of 70 to 500 milliseconds, so the benchmark’s median for Jev came in above the top of that range.

Accuracy goes the other way. Jev’s correctness score was 98.0%, with a 95% interval of 94.7-100%. Its code-quality rule score was 99.5%, with an interval of 98.8-100%. Flash and Fable scored 100% on both. Over three rounds, 0.83% of Jev’s decisions changed, against 0% for each of the other two models. The report says the study does not establish equal accuracy, better accuracy or deterministic decisions for Jev.

How the suite was built

The suite has 24 small program families with five code versions each, run over three rounds: 1,080 main review calls across the three models. A family is one program and its related versions, so the 120 code versions are not 120 independent examples. Hidden behaviour checks and source checks supplied the expected answers. The score averages two rates, correct rejections and correct approvals, and gives each family equal weight. The confidence intervals describe variation within this constructed suite only.

The report sets its own limits. The examples are small and the rules are stated explicitly. The study does not measure production code review, human readability, open-ended bug finding or other kinds of task. Whether Jev is also cheaper on other work remains a hypothesis.

Running it again

The offline checks need Python 3.12, uv and no API keys: uv sync --locked, uv run determinest check and uv run pytest. To run new reviews, copy .env.example to .env and add your keys. Then run uv run determinest discover, followed by uv run determinest run --profile smoke (capped at $2) and uv run determinest run --profile study (capped at $25). The runner records attempts, costs, settings and failures. A run that stops at its budget is left marked incomplete. Current prices and service behaviour may differ from the published run.

That published run ended with 1,620 valid reviews and 30 failed secondary reviews. Known charges came to $7.98. Another 38 failed attempts had unknown charges, and with allowances for those, the budget record reached $8.37. None of the main comparison’s charges are unknown. The code and study materials are MIT-licensed, and the licence gives no access to the model services.

Jev first, Gemini on referral

The report also costs out a setup it did not test: Jev reviews everything first, and a share of the cases also goes to Gemini. Per review, that costs Jev’s mean request price of about $0.0000429 plus the referred share times Gemini’s mean of about $0.00194. Here is how the estimated cost per 1,000 reviews changes with the share referred:

• 0% referred: $0.043, 45.3 times cheaper than Gemini alone
• 1%: $0.062, 31.2 times cheaper
• 5%: $0.140, 13.9 times cheaper
• 10%: $0.237, 8.2 times cheaper
• 100%: $1.986, 0.98 times Gemini’s cost

These figures cover cost, not accuracy. They assume referred cases cost the same on average as the rest, and they leave out routing, storage and other system costs. Harder cases may cost more. The referral rate is a design choice to test, and the report warns that a 2% gap in scores does not mean a router could find and forward only the 2% of cases Jev gets wrong.

TypeSafe’s case for Jev rests on this point. In the launch post, chief executive Diogo Almeida argues that a model that does a task 95% of the time can’t automate it if it doesn’t say when it is in the other 5%. The company says Jev provides calibrated probabilities on every answer, using a training method it calls Reinforcement Learning for Calibrated Decisions. The benchmark’s authors say a routing rule has to be tested on new examples to show which errors it catches and which it misses, and that Jev’s confidence values alone do not guarantee that.