Agent Sessions

Session-Bench

Session-Bench measures whether coding-agent harnesses preserve a useful, inspectable, and portable record of their work — the session files they write to disk. SWE-bench measures whether the agent completed the work; Session-Bench measures what the harness preserved after the work was done. Nineteen scored pass/fail gates across five areas, each backed by a measurement, a fixture, or a monitoring-ledger entry. No partial credit: a gate is pass, fail, or explicitly not run — never averaged.

Bench v0.3 · probe 2026-08-04 · store queries 2026-08-05 · docs review 2026-08-12 · last corrected 2026-08-12 · surface: CLI, headless/one-shot mode · github.com/jazzyalex/session-bench — methodology, data, evaluator & corrections

Leaderboard

Ranking is by fraction of scored gates cleared; ties share a rank. A gate whose measurement could not be taken for a harness is marked not run and drops out of that harness's denominator — an authentication failure is not evidence about a format, and an observation window too short to judge stability (Kimi, onboarded 2026-07-25) is not evidence of stability. Ranks marked † are provisional: hover for the best/worst range pending the missing measurement. Every gate carries equal weight within the composite; the per-area columns are there so you can re-weight by eye. One gate (crash tolerance) is defined but unscored pending a real cross-harness truncation experiment.

View or download the Session-Bench v0.3 poster

The vendor report card

The bench’s primary output is not the ranking — it is what each harness would have to change to pass. Generated from the matrix:

Hover an item for the evidence behind the fail. A vendor that fixes a gate flips it on the next re-score, and the change is recorded in the corrections log — the first harness to clear every scored gate gets named here.

Each area answers one question a developer actually has:

• Signal — is the log your work, or the harness’s paperwork?
• Completeness — can you audit what happened and what it cost?
• Stability — will your archive still parse after the next update?
• Openness — can you read your own history with tools you already have?
• Tooling — will scripts and history browsers survive its quirks?

The gate matrix

✓ pass · ✗ fail · — not run · ◦ unscored · · not applicable. Hover a cell for the evidence behind it.

If you’re building on session files

For tool builders, auditors, and teams with retention or cost-analysis requirements — the properties that matter to a consumer of these records, These rows are generated from the matrix by the evaluator, so they can never disagree with it:

Two editorial observations from the measurement work, decision-relevant but carrying no points: Hermes is the only store with built-in full-text search, and Codex keeps the most complete wire-replay record. And one caution from the gates: Cursor’s transcript is readable JSONL, but full reconstruction — model, timestamps, metadata — requires its opaque binary sidecar (gates O1, O2, P3).

The gates

• S1 — Lean session (Signal): The identical one-line probe task adds ≤ 10 KB to the store (incremental bytes; database cold-start allocation is reported separately, not scored).

• S2 — Content-key share (Signal): Strings under a published list of content-like keys (text, content, message, thinking, output, result, …) are ≥ 25% of stored bytes (corpus measurement). This heuristic captures real work product — messages, reasoning, tool output — but also counts bookkeeping stored under those keys, noted per-agent where known (Codex world_state, Copilot system prompt). Format-specific per-event classifiers are a 1.0 milestone.

• S3 — No fixed-cost dumps (Signal): No single per-session bookkeeping record over 25 KB (system-prompt dumps, tool-catalog snapshots, context snapshots) in recent real sessions.

• C1 — Per-event timestamps (Completeness): Every event carries a timestamp.

• C2 — Model attributable (Completeness): The model behind every assistant message is recoverable from the record.

• C3 — Token usage recorded (Completeness): Token counts are written locally.

• C4 — Dollar cost recorded (Completeness): A dollar cost figure is written locally (premium-request counters do not pass).

• C5 — Tool calls and outputs (Completeness): Both the tool invocation and its output are in the record.

• C6 — Readable rationale (Completeness): A readable account of the model’s reasoning survives in the record — raw thinking or a stored summary beyond the final answer. Sealing may be deliberate vendor policy; the gate scores what the record contains, not the policy’s merits.

• C7 — Thread structure (Completeness): Threads, branches, or subagent runs are recoverable via links or artifacts.

• T1 — No schema break (Stability): No breaking schema change while under observation. Observation windows differ per harness (stated in each cell); a window too short to establish stability scores not_run.

• T2 — Declares a format version (Stability): The artifact declares the version of its session FORMAT — a schema/protocol version an external reader can dispatch on. An application/CLI version stamped in the artifact does not pass: it identifies the writer, not the schema (tightened in v0.3; v0.2 wrongly accepted writer versions).

• T3 — Honest version signal (Stability): The declared version matches the shipped reality. Scored only when T2 passes: with no version signal there is nothing to judge, and absence is already penalized by T2 (changed in v0.2 — v0.1 wrongly awarded this point to harnesses with no version at all).

• O1 — Standard-tools readable (Openness): Readable at rest with a text editor or sqlite3; protobuf or hex-encoded payloads fail.

• O2 — Live-tailable (Openness): One artifact you can tail -f holds the full running record.

• O3 — Documented schema (Openness): The vendor publishes documentation for the session format.

• O4 — Stable location (Openness): No wholesale storage-layout migration in the observation window.

• P1 — Naive-reader safe (Tooling): A reader following the obvious structure gets every message exactly once.

• P2 — Crash tolerant (Tooling): A truncated final write costs at most one record. UNSCORED in v0.2: asserted by format family only; needs a cross-harness truncation experiment (a 1.0 milestone).

• P3 — Self-contained (Tooling): No sidecar file join is needed for title, cwd, or model.

Corrections

• 2026-08-12 — O3 (documented schema) was originally scored fail for all ten harnesses. That was wrong. An external re-check found real vendor documentation that our search missed: Pi publishes a full session-format spec (in its tree since April 2026), OpenClaw documents its session store schema and transcript event structure, Hermes documents its persistence layers and stored fields, and Copilot documents its storage layout and persisted event types. Four cells flipped: Pi 17→18/19, OpenClaw 16→17/18, Copilot 11→12/19, Hermes 10→11/18. Kimi documents its session directory layout but not the wire-record schema, so its fail stands, as do Claude Code, Codex, Antigravity, and Cursor (nothing published) and OpenCode (table definitions in source, not documentation).

• 2026-08-12 (second correction, same review cycle) — v0.1 → v0.2. Codex was underscored by two gates: real rollouts carry plaintext summary_text on ~35% of reasoning items (C6 passes — the gate accepts a stored summary) and parent_thread_id + sub_agent_activity records (C7 passes); Codex moves 12/19 → 14/19, tying Claude Code at rank 3. Antigravity’s P1 flips to pass — XML-wrapped content is awkward, not message loss. Two rubric fixes make this v0.2: T3 is now scored only when T2 passes (v0.1 wrongly awarded “honest version” points to harnesses with no version at all), and T1 states each harness’s actual observation window — Kimi’s ~10 days is too short to establish stability and is now not-run. Denominators shifted accordingly.

• 2026-08-12 (third correction) — v0.2 → v0.3: T2 tightened to format versions only. v0.2 still awarded “declares a version” to harnesses that stamp an application/CLI version — the original defect restated. T2 now requires a schema/protocol version an external reader can dispatch on: Pi (version 1-3, documented), OpenClaw (version 3), and Kimi (protocol_version) keep the point; Claude Code, Codex, Copilot, and OpenCode lose it (their stamps identify the writer, not the format), and their T3 becomes not-applicable. Claude and Codex 14→12/18, OpenCode 13→11/17, Copilot 12→10/18. The Codex C6/C7 evidence now cites a pinned artifact set with hashes (receipt).

Method

What this is, stated precisely: scores are mechanically generated from published measurements and checklist verdicts; some underlying raw artifacts (probe session files, corpus query transcripts) are not yet public. A measurements manifest plus an evidence checklist feed evaluate.py, which computes the Signal gates from fixed thresholds, merges the checklist verdicts, and emits the data file this page renders; measure.py recomputes per-artifact numbers from a session file. Re-running the evaluator reproduces the scoring from the versioned inputs; reproducing the observations end-to-end — harness execution to archived raw artifacts to extraction — is the 1.0 milestone, and until it lands this page does not claim to be a push-button-reproducible benchmark. The benchmark’s canonical home is github.com/jazzyalex/session-bench — methodology, data, evaluator, tests, and the public corrections changelog. To dispute a score, open an issue there with the measurement or evidence you contest, and re-run the evaluator.

The probe is one identical prompt — “List the files in the current directory, then say hello in one sentence.” — attempted through each harness’s CLI, headless/one-shot mode; where it produced a session, the artifact was measured byte-for-byte, and where it could not run, the affected gate says so. Desktop apps and IDE surfaces may use different stores; this bench scores the CLI session store only, and says so. Completeness, Openness, and Tooling verdicts are checklist items evaluated against real session files, the sanitized fixtures in the Agent Sessions repo, and the parsers that read all ten formats in production. Stability comes from a ledger that has fingerprinted each format’s schema roughly weekly since it entered monitoring — 2026-03-31 for the longest-observed harnesses, later for others (Cursor and OpenClaw from April, Pi from May, Kimi from late July); every stability verdict states its own window.

Grades can change, and the version scheme separates the two ways they can: the bench version (currently v0.3) is the rubric — gates, thresholds, scoring rules — and bumps only when a change breaks score comparability; the data date identifies a re-run under the same rubric. Formats drift — four of ten had a structural break in the last four months — and the bench re-scores when they do. v1.0 is reserved for end-to-end reproducibility: harness execution to archived artifacts to extraction, a tested crash gate, and per-event classifiers. Disputes and corrections are welcome as issues.

← The Rollout