Three outside labs ran their own studies on Claude data

Three outside research groups wrote their own questions about roughly 250,000 real Claude conversations. Anthropic ran those questions through its internal analysis tool and gave the researchers back only aggregated categories, the company said in an August 26 post.
In Anthropic’s writeup of the pilot, the partners are Stanford’s Social and Language Technologies (SALT) Lab, the Human Information Processing Lab at Oxford and METR, a non-profit that evaluates frontier models. Each group studied Claude.ai or Claude Code conversations from April-May 2026. Anthropic’s contractual review rights covered four areas: user privacy, information that could help people violate its usage policies, its own confidential information, and research accuracy. Beyond that, the agreements say the partners can publish even when the results are inconvenient for Anthropic. The company calls this the first time external researchers have run public independent studies on an AI company’s own usage data. The aggregate data from each project is now public, and Imperial College London ran a third-party privacy audit on everything that was shared.
Anthropic gives its reason plainly. Outsiders have had two options so far. One is analyses the labs publish, which reflect real usage but tend to answer the lab’s own questions. The other is public datasets, which skew toward casual use.
How a question becomes a percentage
The tool is Anthropic Insights. Until recently it was called Clio, and Anthropic’s Clio page noted the rename on August 24. A researcher writes a question such as “What type of guidance is this person asking for?” and Claude answers it for every conversation in the study. The answers are then grouped into categories. Researchers see only the final categories and the share of conversations in each one. The Clio design adds a floor: a category has to represent a minimum number of unique users or conversations before anyone sees it, and Claude checks each summary for identifying details before it is displayed. The partners never had access to raw conversations, and their outputs went through the same legal and privacy review as Anthropic’s internal work.
The weak point is the wording. Claude does the labelling, so a badly phrased question can sort conversations into categories that misrepresent them. And because nobody can read the underlying text, those errors are hard to catch.
Anthropic’s own teams deal with this by revising their questions many times over several weeks. External partners couldn’t, because a fresh privacy review before each dataset was shared would have made the studies infeasible. So they tested their questions on WildChat, a public dataset of human-AI conversations where they could check answers against the source text. WildChat skews toward casual and creative use, unlike Claude traffic, and some questions that worked well there produced misleading categories once they ran on real Claude conversations. Anthropic’s response was written guidance on how to interpret the outputs, published in the appendix. It says it is still exploring how outside researchers can develop their questions and categories ahead of time.
What the three groups found
SALT looked at how people split work with AI. Prior research suggested people keep consequential tasks for themselves, meaning work that affects others or is hard to undo. Yet in over half of Claude conversations, people were delegating exactly that kind of work. They were most likely to do so when seeking professional guidance, particularly on legal or financial questions. In nearly three-quarters of conversations, people set the direction while Claude assisted, and they usually adapted the output instead of using it verbatim. The lab found that friction was common but often productive: working out where a request was unclear and iterating on it led to better results.
The Oxford lab paired how people felt with how Claude behaved. When Claude was warm, people tended to be more positive. When Claude refused or disagreed, people pushed back. When Claude was eccentric, people got more intellectually engaged, and when it simply helped, people seemed satisfied. The way states like absorption, frustration and enjoyment co-occurred also closely resembled a separate study of everyday web browsing. Oxford’s full writeup is not out yet.
METR is measuring real-world productivity gains from coding agents. It compared Claude’s guesses of how long tasks would have taken without AI with how long they actually took using different Claude models. The preliminary finding is that newer models deliver significant speedup over older ones, so the headline result rests on Claude’s own estimate of the counterfactual. METR checked that estimate against known completion times from a prior developer study and found Claude’s guesses correlated with the time developers actually took. Its analysis is still underway. The next step is to estimate how much AI speeds up researchers’ work, which METR says could matter more as AI takes on more of its own development. The question overlaps with Anthropic’s own economics paper on “Agentic coding and persistent returns to expertise”. Anthropic says it will follow whether the two sets of conclusions line up, and it connected METR with its Economics team.
What got held back?
Some categories in the partners’ outputs surfaced violations of Anthropic’s Acceptable Use Policy or Terms of Service, such as a category of people seeking guidance on a prohibited activity. Anthropic shared most of these. The exceptions were categories that described how users got around safeguards, as opposed to what they attempted. Fewer than 5% of categories and conversations were affected in each study, and researchers were told which clusters had been altered or removed and why. As standard practice, aggregated data on such violations also goes to Anthropic’s Safeguards team for review.
Anthropic describes the pilot as slow by an AI lab’s normal research pace and resource intensive to run, and it names both as obstacles to scaling. It says it is starting slowly for the sake of privacy, safety and research quality. Researchers who could do work they can’t do today with access to Anthropic Insights can fill out the expression of interest form attached to the post.
Up next
Codex runs OpenAI workflows through Runme
OpenAI uses Runme notebooks to give Codex a reviewed workflow for model evaluations. The agent records commands and decisions, so later runs can reuse the context.
Related stories
- Scientists use Claude least where they say it matters most
- Anthropic’s $150M buys Claude time for federal scientists
- Suleyman wants consciousness talk out of Claude’s training
- Anthropic starts counting how much of its R&D AI does
- Anthropic’s alignment lead puts extinction risk above 10%
- How Claude’s watermark works: it only changes the source of randomness
Comments
No comments yet. Be the first.
Join the conversation
Sign in to leave a comment. It appears after moderation.

