Detection climbs from 13.8 to 46.2 mAP at Roboflow

Roboflow tested OpenAI’s three GPT-5.6 models on a VLM benchmark it has yet to release, and its August 17 post puts Sol at 46.2 mAP@50 on object detection against 13.8 for GPT-5.5, while asking these models for boxes in the wrong coordinate format cost them around 15 mAP points. They scored highest with absolute XYXY coordinates in image pixels; Gemini 3.5 Flash did best with YXYX normalized to a 0-1000 range, so a detection prompt carried over from a Gemini pipeline needs rewriting before it goes to GPT-5.6.
The two formats differ in axis order and in scale, so the code that reads the boxes has to change along with the prompt. Simon Willison reported in August 2024 that Google’s documentation described Gemini as trained to give box coordinates for a 1000x1000 version of the image, which then have to be converted back to its real dimensions.
Sol, Terra or Luna
For Roboflow, Sol is the best vision model OpenAI has released so far, and detection went from a major weakness to a practical capability, with Terra at 44.7 and Luna at 43.3 close behind. In May, the same team framed GPT-5.5 as an analysis tool rather than a source of precise boxes, and recommended leaving runtime inference to smaller specialized models such as RF-DETR or YOLO.
At 0.8 cents per image, Gemini 3.5 Flash leads Roboflow’s detection and counting results, and the post names it the better practical choice for high-volume detection and counting. Sol takes close to 10 seconds and roughly 2.5 cents per image, which makes it the second most expensive model in the benchmark after Claude Fable 5, and the whole GPT-5.6 lineup pays for its vision gains with higher token usage, a cost that grows with volume.
Luna, the cheapest model in the lineup, lands only 2.9 points behind Sol on detection, costs less than 0.5 cents per image and finishes in slightly over 5 seconds; Terra costs about 1 cent and takes around 6 seconds. Roboflow rates Luna the strongest latency-quality balance of the three, with speed close to Gemini 3.5 Flash while still beating GPT-5.5 on detection and counting.
What about counting and OCR?
Counting improved across the family: in Roboflow’s counting tests Sol scored 73.0%, Terra 67.6% and Luna 66.2%, all above GPT-5.5’s 64.9%. Sol handled heavily overlapping metal brackets and counted bullet holes only inside the selected scoring zones, but blister packs proved much harder, whether the task was the empty slots or the pills still sealed inside.
On OCR, where the model transcribes all visible text, Sol’s mean similarity score of 90.7% sits 0.5 points under GPT-5.5’s 91.2%, with Terra at 88.8% and Luna at 88.4%, and on targeted text extraction GPT-5.5 scored 87.6% against 82.5% for Sol, 81.4% for Luna and 79.4% for Terra. A pipeline that only reads text would score lower after the upgrade in Roboflow’s benchmark: the older model came out above all three new ones on both tasks. Sol read handwritten notes and a tire size printed along a dirty, worn tire, yet missed the expiration date on a blister pack, printed small, vertical and low in contrast.
Where Sol misplaces boxes
VLMs write every class label and set of coordinates out as text, so the more objects a scene holds, the longer the answer and the higher the risk of missed objects, duplicates or coordinate errors. Even so, Sol found most of the objects in two dense scenes of pills and eggs in Roboflow’s tests. In a few cases, though, it placed boxes in seemingly random parts of the image, many with little or no overlap with the ground truth and often lined up in straight rows or evenly spaced groups.
OpenAI’s team, shown those examples by Roboflow, confirmed that Sol gets less stable on images of around 2,000 by 2,000 pixels or larger, especially at lower reasoning effort. Higher effort improves stability but raises token use, latency and cost. The workaround the post calls the most practical is to resize or crop large images before sending them to the OpenAI API.
Up next
Priciest tokens still pull the most gateway spend
“In July, DeepSeek surpassed Google to take that place,” Vercel’s AI Gateway report says of second place by volume. Anthropic still took 65.1% of gateway spend at 4.4 times the average token price.
Related stories
- Same model, better scaffolding: 13.3% to 38.3% on ARC-AGI-3
- Claude Haiku 5.5 spotted a data leak GPT-6.1 Sol missed
- Cheaper per token, Fable 5.2 cost $13.39 to Astra’s $3.99
- Opus 5.5 beats GPT-6 Astra on data-analysis repeatability
- GPT-6 Astra hits 66.7% on wet-lab tasks, GPT-5.6 Sol 43.8%
- GIFs and 50% faster generation land in ChatGPT Images 2.6
Comments
No comments yet. Be the first.
Join the conversation
Sign in to leave a comment. It appears after moderation.

