Perplexity’s agents built a database, then got locked out

Perplexity has moved part of its production search traffic off DynamoDB and onto CobbleDB, a Rust key-value store of roughly 40,000 lines. Two engineers built it in two months, working with hundreds of persistent coding agents. After the cutover, median batch-read latency fell from 31.4 ms to 5.6 ms, and p99 fell from 123 ms to 24.2 ms.

The figures come from The New Stack‘s September 16 account of the project. The agents helped write the database. The two engineers kept control of the production system.

What one search asks of storage

A single Search API call fetches 100 to 120 page keys, and the retrieval service splits them into parallel batches of 10 to 20. The stored values are large because a language model needs more than a link. Each record holds the page’s pre-chunked passages and their vector embeddings, about 50 KB on average.

On DynamoDB that workload ran into two problems. The first was cost. AWS meters every byte transferred, and search, crawling and reprocessing all produce a steady flow of large reads and writes. At more than 200,000 requests per second, the pricing became financially unsustainable, and cloud costs got harder to justify as traffic and the corpus grew. The second was control. DynamoDB doesn’t show where partitions sit, what it caches in memory or how it routes to replicas, so engineers had no way to stop the tail-latency spikes caused by uncached reads, hops across availability zones or a lagging replica. One slow replica could hold up an entire batch.

Writes made it worse. Whenever Perplexity changed its chunking method or moved to a newer embedding model, the reprocessing jobs sent large volumes of writes straight into DynamoDB, and they competed with live user requests for the same capacity.

Three tiers instead of one table

The fix was to separate long-term document storage from the database that answers live searches. The storage stack now has three parts:

• Pillar holds durable document state, including versioned metadata, chunks and embeddings, in YTsaurus running on high-capacity hard drives.
• Lorry is a stateless queue consumer. It groups Pillar exports into batch files aligned with CobbleDB’s partitions, writes the payloads to Amazon S3 and sends CobbleDB a metadata notice.
• CobbleDB serves reads at low latency.

CobbleDB worker nodes pull and ingest the S3 batches independently, which keeps the write-heavy crawl pipeline away from the nodes serving queries. Each replica applies updates on its own, so one can fall behind and catch up without blocking the others.

Inside CobbleDB

CobbleDB handles one access pattern only: batched lookups. Keys are hashed page URLs, and values are the passages plus their embeddings. Each node runs a daemon that uses RocksDB as its embedded storage engine. Frequently read data stays in a memory-mapped cache and the rest sits on local NVMe SSDs. Within a node, a batch is served through RocksDB’s MultiGet interface, which fetches many keys in one call.

Every partition has three replicas. A stateless query router maps each hashed key to its partition and sends the request to a replica in the same availability zone when it can. If that replica is responding slowly, the router hedges: it sends a concurrent read to another replica on a different node so the batch isn’t held up. Replication between replicas is asynchronous, so the store is eventually consistent.

Before and after the cutover

Perplexity took its CobbleDB measurements at about 200,000 requests per second. Median batch-read latency was 5.6 ms, against 31.4 ms recorded on DynamoDB. At p90, latency dropped from 56.7 ms to 9.77 ms, and at p99 from 123 ms to 24.2 ms, according to InfoQ‘s write-up of the results. In later load testing, CobbleDB handled 500,000 requests per second before performance began to drop.

The two systems never served identical traffic side by side. The DynamoDB numbers were recorded before the cutover and the CobbleDB numbers after it. Perplexity also ran separate synthetic benchmarks using batches of 10 to 15 keys and values from 100 bytes to 100 KiB.

Perplexity’s cost model puts CobbleDB at least 20% below DynamoDB across the commitment options evaluated. That estimate leaves out the engineering cost of supporting the database: the people who maintain it and respond when something breaks. Perplexity plans to open-source CobbleDB at some point.

Where the agents stopped

CMU professor Andy Pavlo argued at Percona Live earlier this year that databases are the hardest and most important challenge for AI agents, partly because mistakes involving production data can be difficult or impossible to reverse. Perplexity used hundreds of agents to build one anyway, but limited what they could touch.

The agents kept context across sessions and worked on fixes and tests. Along the way they caught problems with restore assumptions and with runtime configuration. They did not run the database. The two engineers kept control of the architecture and of the production system that now serves part of Perplexity’s search traffic.