Skip to content

Benchmarks

$200 fine-tune beats OpenAI’s Decisions API on email attacks

7 min read

Written by an AI model from the linked sources and checked against them by a second model. How we work

All about: OpenAI models and products →
Image: builders.abnormal.ai

Abnormal classifies billions of emails a week. It ran 40 models over a set of its hardest cases and ranked them mainly on one figure: how many attacks a model catches while 99% of its attack calls are correct. The results, in an October 8 post on Abnormal’s engineering blog, have OpenAI’s Decisions API leading on that figure for its price and TypeSafe’s Jev leading on calibration. A $200 fine-tune of a small open model beat both, with caveats the authors list themselves.

Abnormal’s production models are behavioral. They learn what normal looks like for each sender, recipient and organization, using signals like sender history, authentication and the org directory. What a feature-based model can’t do is read the request itself. The authors’ example is judging “whether a vendor would really change bank details in the middle of an invoice thread.” Decision models bring that LLM-style reading at a different price. They take in the email and every question in one forward pass and return a probability for each option without generating a token, so an answer costs about as much as reading the input. And because each question is written in plain language, a new detector can start zero-shot as a written question, with no feature to engineer and no model to retrain.

The field covered Jev, the OpenAI Decisions API (in public beta since October 6), 34 open decision models Abnormal hosted itself, and four frontier LLM configurations: sonnet-5.5 and luna-6, each at low and max reasoning effort. Each open model ran on one L40S, except the three largest, which got eight A100s.

The dataset is 1,985 fully synthetic emails with no customer data. They are modeled on the attacks Abnormal sees in production and on the false positives that look like them. Each email was generated to match the class, attack type and detection signals of a real hard case without reusing its identifiers. Attacks make up 40% of the set, far more than in any real inbox, so high precision is easier to hit here than in production. Every model got the same record, including signals a raw email doesn’t carry, such as sender history, directory matches and domain age. Each had to answer three questions in one request: is this an attack; is it an attack, spam, graymail or safe; and for attacks, is it phishing, BEC, scam, malware or recon. The LLMs got the record as JSON and had to return a probability for every option under a strict schema.

On a composite email score, the mean of three sub-scores (four-way verdict accuracy, attack-type accuracy and attack recall at 95% precision), bigger models generally scored higher and the frontier LLMs led. Each step up the curve cost far more than the one before. The top configuration, sonnet-5.5-max, scored 88.3 at $52 per thousand emails. Jev and the Decisions API both sat on the Pareto frontier, the set of models no other model beats on both price and score.

Abnormal says the composite isn’t what decides whether a model is usable. When a model calls an email an attack, Abnormal pulls it from the inbox, so every false positive is a legitimate and often important email a customer never sees. “Precision is the constraint and recall is the prize,” the authors write. That is why they read each model’s recall at 99% precision.

Measured that way, the frontier shrinks to a handful of models. Most of the field catches only a small fraction of attacks, including every open model Abnormal hosted. Jev stays on the frontier on price alone. The Decisions API catches most of what luna-6-max does at a fraction of the cost and latency.

The authors single out luna-6-low, the same luna-6 model prompted as a regular LLM. It matches the Decisions API on average precision, a summary of the whole precision-recall curve, but falls far behind at 99% precision, “because threshold-free metrics hide the part of the curve you actually operate in.”

Calibration matters because security systems act on the probability itself. Above one threshold an email is removed automatically, a middle band goes to review, and everything below is delivered. Those thresholds have to mean the same thing across tenants and model upgrades, which only works if a stated 0.9 means about 90% of those emails really are attacks. Probabilities were scored exactly as returned, with no per-deployment tuning. Jev was the best calibrated of all 40 models, and the two decision APIs were among the best overall. The frontier LLMs landed in the middle, even when the prompt asked them directly for honest, calibrated probabilities. The small open decision models did worst, base kev-4b among them, which leads the authors to conclude that calibration depends on the model, not the interface.

Abnormal then fine-tuned that same kev-4b. It was a single-epoch LoRA run (LoRA trains a small set of added weights and leaves the base model frozen) over 60K examples of email and non-email classification tasks, 7,500 steps on L40S GPUs for about $200. None of the training emails or task formats overlap with the benchmark. Recall at 99% precision went from 7% to 66%, higher than any model tested at any price, and calibration went from one of the worst in the field to among the best.

The authors attach two caveats to that result. The 95% confidence intervals overlap: the fine-tune’s recall is 65.5% [49.8, 80.2] and sonnet-5.5-max’s is 49.9% [45.5, 83.2], bootstrapped over 637 test attacks, so more data is needed to call the gap with confidence. When the threshold is set on a held-out 20% split instead of the test set, the fine-tune still holds 98.6% precision at 67.7% recall. Precision is also computed as if half of all mail were attacks, so the authors say to read these numbers as relative.

The run also had a cost. Attack-type macro-F1 fell from 68 to 37, and the model now labels most BEC and recon emails as scams. The training mix covered the four-way verdict but never attack types, so the authors suspect the fine-tune sharpened the question it was trained on and degraded one it wasn’t.

The benchmark doesn’t answer whether Abnormal can serve such a model at its scale. EU mail has to be processed in the EU, so an API has to serve there, and a self-hosted fleet has to exist in every region Abnormal operates in. The cost charts assume every GPU is busy all the time. Priced from kev-4b’s measured throughput at on-demand GPU rates, serving it costs more than the charts show even with no idle time, and mail follows the workday, so real utilization is lower still. Running a 4B model on every email at Abnormal’s volume also needs a GPU fleet that the authors call hard to get even with the budget for it.

For comparison, the equivalent detection in Abnormal’s live stack costs about a cent per thousand emails, running on a mix of CPUs and GPU accelerators. That figure leaves out ingestion, parsing, storage and offline training, which an LLM-based system would need too. By the authors’ account, decision models cut LLM-class judgment from dollars per thousand emails to cents and from seconds to milliseconds. That is cheap enough for the slice of traffic a prefilter sends them, but not for every email.

The architecture Abnormal expects to build has cheap models on all mail, decision models behind prefilters, frontier LLMs on the few emails that need real reasoning, and feedback loops that let the system improve itself.

No comments yet. Be the first.

Join the conversation

Sign in to leave a comment. It appears after moderation.