FutureOS kept 147 of 178 answers, Codex kept 68

FutureOS tested its default context compaction against reimplementations of Codex and OpenCode. All three ran on the same model, with the same 178 memory questions and the same call shape. FutureOS’s strategy still answered 147 of them (83%), OpenCode 83 (47%) and Codex 68 (38%). The write-up on the FutureOS engineering blog on GitHub, which its tag page dates August 4, traces most of the gap to one decision: whether the agent’s own prose survives a compaction.

The exam works like this. Grow a session until the context window is full, force a compaction, then ask for things only memory can supply: line three of a script’s output, an ID a tool returned, a constraint the user set hours earlier. Eight questions were decoys with no answer anywhere, and no system fell for one. The rival columns are pinned to Codex b13164d8 and OpenCode e03db9bc (package 1.18.31), and any later upstream commit invalidates them. Codex’s own retrieval tools need its hosted backend, so its column measures the local fallback.

What fills the window

FutureOS counted five frozen chains from real sessions: 9,665 records and 6,889,611 characters. Tool output made up 96.1% of the characters, assistant text 3.7% and user text 0.2%. Chain by chain, tool output’s share ran from 93.2% to 99.4%. The follow-up questions point the other way. Of the “what happened earlier” questions, 80% refer to something the assistant said, 10% to user turns and 10% to tool output.

So the conversation takes up 3.9% of the window and draws 90% of the questions. Compressing it frees almost no room. Tool output can be shrunk to signposts and recall barely drops. Codex and OpenCode compress the two together.

Three meanings of “compact”

Codex keeps every user message, capped at 20,000 tokens, plus a summary of the whole history. It drops assistant prose and tool output outright, which by the tally above gives up 80% of the questions before the exam starts. OpenCode keeps a summary plus a recent tail capped at 15,000 tokens, and relies on detail surviving inside the summary. FutureOS puts protected user and assistant originals first. The summary and a tool-evidence index fill in for whatever doesn’t fit.

In FutureOS, compaction produces a projection. Nothing in the journal on disk is deleted or rewritten; only what the next request sees changes. The trigger fires at 80% of the window, 800,000 tokens on a 1M model. It fires earlier if the window can’t also hold the model’s declared output ceiling plus a margin of up to 2,048 tokens. Inside the projection, user text always stays and assistant originals get the remaining room. The recent tail is about 8K tokens, and the history target is about 32K, going as high as 128K. Tool output becomes a 2,048-token index built without a model call. Errors come first, then results grouped by tool and target, with config, schema, validation and test targets ranked highest. Each entry is one JSON row that records its source order, so an old error isn’t mistaken for a current one. An assistant output that gets pushed out is labelled “omitted / not summarised” and stays queryable.

The reason FutureOS gives for favouring user text is recoverability. A user’s constraints and goals can’t be regenerated, while tool output and assistant prose usually can be rebuilt from the originals.

There are two tiers. summarized-evidence-v1, the default, adds a handoff summary in one model call. deterministic-evidence-v1 makes no call at all. It is also the fallback when no provider is reachable or the summary fails, and on its own it scored 127/178 (71%).

What recall costs

Recall goes up with projection size. Median projections came to 12,113 tokens for FutureOS summarized (5.7% of the uncompacted 232,777), 9,945 for the deterministic tier, 5,152 for OpenCode and 1,706 for Codex. FutureOS’s projection is about seven times the size of Codex’s, and so is its per-turn cost, the price of re-sending the projection on every later turn: ¥0.000480 against ¥0.000068 (all prices in CNY). With no compaction at all, a turn costs ¥0.009219.

The price of the compaction call itself depends on the provider’s prefix cache, which matches a request from token 0. FutureOS’s summary request reuses the session’s own system prompt and tool definitions to stay inside that cache. On a primed prefix, an identical request shape got a 93.7% cache hit. Swapping the system prompt, dropping the tool definitions or adding a single line each cut it to 0%. In production, a session that had grown to 212,911 tokens compacted with 212,548 of them read from cache, or 99.8%. Thanks to the cache, a FutureOS compaction costs ¥0.53 instead of ¥7.98 cold. Codex also shares the prefix and drops from ¥7.64 to ¥0.42. OpenCode uses a dedicated compaction system prompt and pays a flat ¥0.99 either way. The cached prices are modelled from that one production measurement, because the cache counters in the experiment runs were skewed by the order in which arms and runs executed.

Over one compaction plus 100 turns, a point of recall cost 0.0070 with FutureOS, 0.0112 with Codex and 0.0217 with OpenCode. Compaction pays for itself after 46 turns for Codex, 61 for FutureOS summarized and 110 for OpenCode. In a shorter session it costs more than it saves, and what it buys is recall and headroom.

Why not let the model search?

The originals stay on disk, so FutureOS also ran an open-book version: the summarized arm, 18 cases, 178 values. The rounds tried the production prompt and tools, stronger recall guidance, and wording that stopped calling the projection “the record”. Scores stayed between 145 and 149, and the model made zero tool calls in every round. The same 27 values were sitting in the archive each time and were never looked up. Recall reached 166/178 (93.3%) only when an explicit verification instruction forced the first call. That round’s 100 tool calls recovered 20 of the 28 missing values. FutureOS counts it as an upper bound, since no such instruction exists in production.

FutureOS then removed the recall guidance from the runtime and kept the retrieval CLI, future session history search / get, after three variants and 54 cases produced no change in behaviour. One side effect: a session’s system prompt no longer changes when a checkpoint is committed.

The authors list their own limits. The exam tests recognition of exact values, which favours verbatim retention over summaries, so a test built around continuing a task could come out differently. Each cell was run once, so differences of a few points can’t be told apart. Their conclusion from the open-book run: “retrieval can’t close a retention-scope gap.”