The Rollout
Session-Bench: 20 pass/fail gates for coding-agent session formats
A coding agent’s session file is the only durable record of the conversation behind your code: what was asked, what ran, what it cost, what broke. We parse ten of these formats in production and fingerprint them for drift weekly, and the standard of care is low. Formats drift without notice, three of ten carry no version marker, and Claude Code, Codex, Cursor, and Antigravity publish nothing about their persisted formats.
Opinions about that are cheap, so we turned ours into a benchmark — of the layer existing benchmarks skip. SWE-bench measures whether the agent completed the work; Session-Bench measures what the harness preserved after the work was done. Session-Bench scores the session format of every harness we support on twenty scored pass/fail gates across five areas: signal density, completeness, stability, openness, and tooling cost. No partial credit, no vibes; a gate is pass, fail, or explicitly not run. Scores are mechanically generated from published measurements and checklist verdicts — to dispute a grade, dispute an input and re-run the evaluator — though some underlying raw artifacts (probe session files, corpus query transcripts) are not yet public; archiving them is the 1.0 milestone. The bench page is permanent, and grades will change when formats do. One caveat on scope, stated plainly: the bench measures each harness’s CLI session store; desktop and IDE surfaces can differ and get their own rows in a future version.
The v0.3 leaderboard
| # | Harness | Format | Gates cleared |
|---|---|---|---|
| 1 | Pi | flat parent-linked JSONL | 18 / 19 |
| 2† | OpenClaw | flat parent-linked JSONL | 17 / 18 |
| 3 | Claude Code | nested-envelope JSONL | 12 / 18 |
| 3 | Codex | nested-envelope JSONL | 12 / 18 |
| 5† | OpenCode | SQLite rows | 11 / 17 |
| 6† | Kimi Code | wire-op journal | 11 / 18 |
| 7† | Hermes | SQLite ledger + FTS | 10 / 17 |
| 8 | Copilot CLI | event-sourced JSONL | 10 / 18 |
| 9 | Antigravity | step-typed JSONL | 9 / 18 |
| 10 | Cursor Agent | flat JSONL + binary metadata DB | 7 / 18 |
† One measurement could not be taken — a broken headless runtime, an auth failure, no store-size figure comparable across formats, or an observation window too short to judge stability (Kimi joined monitoring in late July). That gate is not run, drops out of the denominator, and the rank is provisional within a stated best/worst range on the bench page. An authentication error is not evidence about a format. Ranking is by fraction of scored gates cleared; ties share a rank.
The signal gates come from a controlled probe: the identical one-line prompt (“List the files in the current directory, then say hello in one sentence.”), attempted headless through every harness, with each resulting session artifact measured byte for byte and each failed run recorded as exactly that.
The full gate-by-gate matrix, with the evidence behind every cell, is on the bench page. The short version of how each grade was earned:
Pi, 18/19. The smallest harness in the set keeps the cleanest record: per-event timestamps, per-message model, token counts, dollar cost, readable thinking, first-class compaction events, a 1.5 KB log for our standardized one-line task — and it is the only harness that publishes a proper session-format specification, in its docs since April. It drops a single point, for leaving abandoned branches in the file for naive readers to trip over.
OpenClaw, 17/18. The other small harness. Full usage accounting including dollars, tool calls with outputs, stable schema across its ~4-month observation window, and official documentation of its session store schema and transcript event structure. Its size probe is marked not-run rather than scored — the headless path refused to authenticate, which is a fact about automation readiness, not about the format — and the bench says so instead of guessing. Where it writes those files: OpenClaw local agent history.
Claude Code, 12/18. Genuinely good bones — per-line writer-version stamps (an app version, not a schema version — that now costs a gate), model and usage on every assistant event, subagent trees, fifteen-plus additive changes across the full four-month window without one breakage. It bleeds points on what it writes: 40 KB for a one-line exchange (two-thirds of it attachment records), a content-key share of 23.9% of corpus bytes, no dollar figure, and about 97.5% of recent thinking blocks are cryptographic signature with the text stripped. The reasoning is certified to have happened. You just can’t read it.
Codex, 12/18. The most complete wire-replay record we measured, and
one of the most wasteful. Every turn is logged twice by design (a UI event plus the exact
wire item), and its world_state context snapshots reach 41.6 KB per line
in real sessions. Two of its v0.1 fails reversed on re-verification: raw
reasoning is encrypted, but about a third of reasoning items carry a
plaintext summary, and real rollouts do record thread and subagent links —
both corrections credited to external review. It is also the only format here whose log
occasionally answers the prompt: in our controlled run, the model read the
file listing out of its own session file’s world snapshot instead of
running the tool.
OpenCode, 11/18. A transparently queryable database design: every text chunk an
addressable row, plaintext reasoning, dollar cost recorded, and its biggest
rows turn out to be actual work product (tool outputs, diff summaries), not
bookkeeping dumps. Its lean-session gate is not-run, stated plainly: the
probe wrote about 5 KB of logical rows, but a physical before-and-after
store measurement comparable to the JSONL harnesses (pages, indexes, WAL)
hasn’t been taken yet, and we won’t score a number we didn’t measure. It
still pays for the architecture in bench terms: nothing to tail -f, and a
JSON-to-SQLite migration inside our observation window.
Copilot CLI, 10/18. The richest telemetry in the set (router scores, token checkpoints, a full shutdown accounting event) wrapped in the worst fixed cost: it writes its complete system prompt into every session, a 52 KB line in a 67 KB log. It also moved its entire storage layout at v1.0, and its session title lives in a sidecar YAML file.
Kimi Code, 11/18. The strangest format we track: a wire-operations
journal where retries (attempt: "2/10"), steering, and cancels are all in
the record — real forensic value. Its probe log is 0.9% conversation by
bytes (7% across the whole corpus),
snapshots a 70 KB tool catalog every session, splits user and model content
into two disjoint op families that a naive reader will half-miss, and is
the only format whose artifact declares a wire-protocol version field.
We initially scored that marker as dishonest — the installed bundle ships a
1.5 constant while sessions say 1.4 — then traced the actual writer and
reversed ourselves: the binary carries two protocol generations side by
side, and the one that wrote our session stamps 1.4 truthfully. The bench
keeps the receipts either way.
Hermes, 10/17. The only format that can search itself — a trigram full-text index inside the database — plus real dollar accounting. But model attribution is session-level, there is nothing to tail, it migrated storage backends inside the window, and querying the live database turned up a 36 KB system prompt stored in nearly every session row (116 of 118) — the same fixed-cost habit as Copilot, just hidden in a column instead of a log line. Its size probe is not-run: the headless one-shot mode has returned no output since late June.
Antigravity, 9/18. Tidy 6.8 KB logs with per-event timestamps, and then the completeness floor gives out: no model field anywhere (our parser regexes model changes out of user-input text), no token counts, user content wrapped in XML tags inside JSON strings, and the largest storage migration we observed — markdown artifacts to JSONL, mid-window. Both sides of that migration are mapped in the Antigravity CLI history guide.
Cursor Agent, 7/18. Last place, and the leanest file in the bench, which is the point: 1.9 KB per session because the transcript records no timestamps, no model, no token counts, no cost, and no tool outputs. (Tool calls appeared sometime since July — an undocumented change we caught by re-measuring.) Its transcript is readable JSONL — but complete reconstruction, including model, timestamps, and metadata, requires an opaque sidecar database of protobuf blobs and hex-encoded JSON that standard tools cannot read.
The gate we scored wrong, and what it actually shows
The first version of this post claimed O3 — documented schema — failed all ten harnesses. That was an error, caught by external review five days after publication, and the bench works the way we said it does: the verdicts changed, the corrections log says exactly what and why. Pi has published a genuine session-format specification since April. OpenClaw documents its session store schema and transcript event structure. Hermes documents its persistence layers and stored fields; Copilot documents its storage layout and persisted event types. Four cells flipped.
The corrected finding is sharper than the wrong one: the two leaders on the board document their formats. Claude Code, Codex, Cursor, and Antigravity do not document their persisted session formats at all, OpenCode’s tables live only in source code, and Kimi documents where sessions live but not what the records mean. Every parser of the undocumented formats — ours included — is reverse-engineered and breaks a little when a vendor renames a field. These files hold months of paid working history.
The same review cycle then found the reverse error on Codex — we had underscored it. Real rollouts carry plaintext reasoning summaries and subagent-thread records our evidence missed, so two fails flipped to passes and Codex rose to joint third. The rubric also moved twice in the same cycle: v0.2 made the honest-version gate conditional on a version existing at all and gave each stability verdict its actual observation window; v0.3 tightened the version gate to format versions only — an application version identifies the writer, not the schema, so Claude Code, Codex, Copilot, and OpenCode lost that point while Pi, OpenClaw, and Kimi, who stamp true format versions, kept it. A bench that grades others on their records has to keep its own this way.
The drift data says this is not hypothetical caution. Across up to four months of weekly checks (observation windows vary per harness and each stability verdict states its own), four of ten formats had a structural break: Copilot moved flat files to per-session directories at v1.0, OpenCode and Hermes each replaced JSON storage with SQLite, and Antigravity swapped markdown artifacts for JSONL wholesale. Three of ten still carry no version marker of any kind, so a parser learns about changes by failing.
Using this to choose a harness
The composite score rewards all-rounders; most developers care about one property more than the rest. From the gate data: if you want local cost accounting in dollars, only Pi, OpenClaw, OpenCode, and Hermes write it. If you want history you can grep and tail, every JSONL harness qualifies — Pi, OpenClaw, Claude Code, Codex, Copilot, Kimi, and Antigravity — while the database camp and Cursor’s split store do not. If you want to search your history without installing anything, Hermes is alone. If you want reasoning (or at least a stored summary of it) you can still read next quarter, that rules out Claude Code, OpenClaw, and Cursor today. And if being able to read your own record back matters at all, Cursor’s binary metadata store is the one to avoid. The bench page keeps a per-area breakdown for exactly this kind of narrowing.
What passing would look like
The bench encodes it: write append-only lines you can tail, keep conversation above a quarter of the bytes, stamp model and usage on every message, record cost in currency, keep a readable rationale in the record (sealed chain-of-thought may be legitimate policy, but then store a summary), declare a version honestly, don’t move the store, and publish the schema. No harness clears every scored gate today. Pi is one away. The gates are not exotic; they describe a format that respects the person whose disk it is written to.
Grades will move — that is the point of a standing bench. The monitoring ledger and methodology and the drift log are public, re-runs happen with our weekly monitoring, and disputes are welcome as issues. Agent Sessions itself is a free, local-only macOS browser for all ten of these stores; reading them every day is how the bench got built.