AWS Emulator mocks 438 AWS APIs, written by agents

AWS Emulator, a local stand-in for more than 200 AWS services that AWS made public on November 11, was written entirely by AI agents, and the agents will keep maintaining it, AWS’s Tom ‘Spot’ Callaway told The New Stack. On launch day it passed 24,243 of 42,937 test units in AWS’s own published fidelity scores, 56.5% by our count, and calls that create and configure resources pass more often than calls that move data, so a passing test says most about the control plane, which AWS’s documentation also names as the emulator’s closest match to AWS.
It covers 438 public AWS API models, is written in Rust and needs neither an AWS account nor a network connection, although we found telemetry to AWS on by default. “There’s no AI inside of AWS Emulator,” Callaway said. In our run a Bedrock converse call got back a canned sentence saying no model runs, and operations the emulator doesn’t implement answer with HTTP 501, which 1,264 failing test units in the launch-day scores hit. The code is under Creative Commons Zero, which puts it in the public domain, because AWS wasn’t sure it could copyright work written by agents and didn’t want to assert rights that may not exist; the vendor/ directory is Apache-2.0.
A weekend with Kiro
The project began with Matt Goldberg, a senior principal engineer on the AWS Developer Tools team. In his post on the build, he describes a three-step prompt to Kiro, AWS’s agentic coding tool: write tests based on the AWS CLI, iterate until they pass against real AWS, then build an in-memory implementation that passes the same tests. A passable SQS came back in 15 minutes, Kiro built “the rest of AWS over the weekend,” and his team was testing its own software against it that Monday. The trigger was Anthropic’s agent-built C compiler; in Goldberg’s words, “the AWS API Models are our spec and the actual AWS cloud is our reference implementation.”
The system that replaced the weekend version splits agents into three teams. The first writes scenarios from AWS’s public Smithy models, the API definitions AWS also generates its SDKs from, and checks each one against live AWS; Goldberg’s example runs “creating a queue, sending a message, receiving it, acknowledging it, and deleting the queue.” The second runs those scenarios against the emulator, groups failures that appear to share a cause and files issues. The third changes the code, and a merge needs passing builds and tests plus reviews by other agents for security, coding standards and regressions. Goldberg’s post puts the scale at “hundreds of agents” working on the codebase concurrently.
Engineers tune the agents and leave the code alone. “We don’t edit it. We don’t modify it,” Callaway said; a bug gets traced back to the agent that produced it. An earlier version of the agents decided to write its own Python interpreter so Lambda functions could run, and the team has since told them to reuse existing libraries. The build we ran fits that account: Python runs on RustPython 0.6.0 and JavaScript on QuickJS, both bundled into the binary, and a function declared as python3.12 reported itself as Python 3.14.0 alpha. import boto3 inside a function succeeds, but judging by a missing version attribute it is not the real SDK.
Our analysis
What we checked
On November 11 we summed the per-service score files AWS attached to releases v1.0.290 and v1.0.322, matched the latter to the public test manifest, and ran one build of the emulator through boto3.
AWS reports fidelity per service and gives no total; summed, the emulator passes just over half of its own tests, and CloudFormation only 18.9%. Callaway’s “There’s no database” is narrower in practice: SQL ran and returned correct results, but the engine behind an Aurora PostgreSQL cluster is not PostgreSQL. And where The New Stack lists Linux, macOS and Windows, AWS’s documentation says the emulator runs natively only on Linux; macOS and Windows need a Docker image you build from AWS’s Dockerfile.
| Scope (v1.0.322) | Test units passed | Share |
|---|---|---|
| All services | 24,243 of 42,937 | 56.5% |
| DynamoDB | 1,153 of 1,694 | 68.1% |
| SQS | 1,032 of 1,523 | 67.8% |
| S3 | 1,763 of 2,814 | 62.7% |
| Lambda | 1,561 of 3,006 | 51.9% |
| EC2 | 958 of 2,470 | 38.8% |
| CloudFormation | 795 of 4,206 | 18.9% |
| Control plane | 12,973 of 19,300 | 67.2% |
| Data plane | 5,576 of 10,437 | 53.4% |
- The plane rows cover the 29,255 behavioral units we could match to the manifest, which is about 17 hours newer than the scores; tests of native data-plane protocols passed 0 of 156. A unit naming several services counts toward each.
- Through the RDS Data API, SQL created a table, inserted 4 rows and returned correct GROUP BY totals, and Athena answered
SELECT 1+1. On the cluster reporting Aurora PostgreSQL 17.9,SELECT version()failed with a SQLite-style error andgenerate_seriesfailed too. That the engine is SQLite is our inference from that error and the rusqlite library in the package’s licence list.
Issues, no pull requests
Part of the pitch is letting coding agents test against AWS without handing them AWS credentials. No account keys are needed, but by default the emulator requires keys it generates itself and rejects dummy ones, so a test setup with placeholder credentials gets turned away.
Outside code isn’t accepted. “We’re not going to take pull requests on the project at all,” Callaway said. “The agents will be doing that. We will take issues from customers.” He expects those to be mostly scenarios the emulator doesn’t handle well yet, and an API the agents keep getting wrong counts as “a bug in the agent.”
Callaway expects new AWS services to reach the emulator in “days, not weeks, not months,” while noting the team hasn’t seen how that plays out in public. Between the two releases we compared, scored about 17 hours apart, the pass share rose from 55.8% to 56.5%, with 651 test units moving from fail to pass and 204 the other way; AWS’s docs say passing tests don’t regress unless a test is rebuilt, and we can’t tell which of the 204 were. AWS plans to say more at re:Invent, which starts November 30.
AWS also publishes the benchmark it scores the emulator with, and the releases we checked carry their per-service score files, so a team can look up the services its own tests touch before trusting a green run against the emulator.
Up next
ByteDance plans a Muse-like AI agent in the coming weeks
Per Bloomberg, Zhang Yiming told ByteDance’s AI team it plans a Muse-like agent in the coming weeks, but catching Anthropic and OpenAI will take until at least 2029 or 2030. Right now, is your everyday assistant from Anthropic or OpenAI, or neither?
Related stories
- AWS says its open agent runs 45% cheaper than Claude Code
- Email-style inbox for agents comes out of Amazon
- Cursor hosts and deploys apps its cloud agents build
- OpenAI’s Pando cut monorepo push latency nearly 5x
- Claude API enforces thinking blocks and moves to Rust
- GitHub Copilot gets five new features at Universe’26
Comments
No comments yet. Be the first.
Join the conversation
Sign in to leave a comment. It appears after moderation.

