Agent Sessions

Session Bench

A benchmark of coding-agent session formats — the durable work record each harness writes to disk. Nineteen scored pass/fail gates across five areas, each backed by a measurement, a fixture, or a monitoring-ledger entry. No partial credit: a gate is pass, fail, or explicitly not run — never averaged.

Bench v0.1 · data measured 2026-08-04 · surface: CLI, headless/one-shot mode · measurements, checklist & evaluator

Leaderboard

Ranking is by fraction of scored gates cleared; ties share a rank. A gate whose measurement could not be taken for a harness is marked not run and drops out of that harness's denominator — an authentication failure is not evidence about a format. Ranks marked † are provisional: hover for the best/worst range pending the missing measurement. Every gate carries equal weight within the composite; the per-area columns are there so you can re-weight by eye. One gate (crash tolerance) is defined but unscored in v0.1 pending a real cross-harness truncation experiment.

Each area answers one question a developer actually has:

• Signal — is the log your work, or the harness’s paperwork?
• Completeness — can you audit what happened and what it cost?
• Stability — will your archive still parse after the next update?
• Openness — can you read your own history with tools you already have?
• Tooling — will scripts and history browsers survive its quirks?

If you’re choosing a harness

The composite rewards all-round formats. If one property matters more to you, the gate data narrows it fast:

* Editorial observations from the measurement work, noted because they're decision-relevant; they carry no points.

The gate matrix

✓ pass · ✗ fail · — not run · ◦ unscored. Hover a cell for the evidence behind it.

The gates

• S1 — Lean session (Signal): The identical one-line probe task adds ≤ 10 KB to the store (incremental bytes; database cold-start allocation is reported separately, not scored).

• S2 — Content-key share (Signal): Strings under a published list of content-like keys (text, content, message, thinking, output, result, …) are ≥ 25% of stored bytes (corpus measurement). This heuristic captures real work product — messages, reasoning, tool output — but also counts bookkeeping stored under those keys, noted per-agent where known (Codex world_state, Copilot system prompt). Format-specific per-event classifiers are a 1.0 milestone.

• S3 — No fixed-cost dumps (Signal): No single per-session bookkeeping record over 25 KB (system-prompt dumps, tool-catalog snapshots, context snapshots) in recent real sessions.

• C1 — Per-event timestamps (Completeness): Every event carries a timestamp.

• C2 — Model attributable (Completeness): The model behind every assistant message is recoverable from the record.

• C3 — Token usage recorded (Completeness): Token counts are written locally.

• C4 — Dollar cost recorded (Completeness): A dollar cost figure is written locally (premium-request counters do not pass).

• C5 — Tool calls and outputs (Completeness): Both the tool invocation and its output are in the record.

• C6 — Readable rationale (Completeness): A readable account of the model’s reasoning survives in the record — raw thinking or a stored summary beyond the final answer. Sealing may be deliberate vendor policy; the gate scores what the record contains, not the policy’s merits.

• C7 — Thread structure (Completeness): Threads, branches, or subagent runs are recoverable via links or artifacts.

• T1 — No schema break (Stability): No breaking schema change in the observation window (2026-03-31 → 2026-08-04).

• T2 — Declares a version (Stability): The artifact itself declares a format/schema version.

• T3 — Honest version signal (Stability): Any declared version matches the shipped reality.

• O1 — Standard-tools readable (Openness): Readable at rest with a text editor or sqlite3; protobuf or hex-encoded payloads fail.

• O2 — Live-tailable (Openness): One artifact you can tail -f holds the full running record.

• O3 — Documented schema (Openness): The vendor publishes documentation for the session format.

• O4 — Stable location (Openness): No wholesale storage-layout migration in the observation window.

• P1 — Naive-reader safe (Tooling): A reader following the obvious structure gets every message exactly once.

• P2 — Crash tolerant (Tooling): A truncated final write costs at most one record. UNSCORED in v0.1: asserted by format family only; needs a cross-harness truncation experiment (a 1.0 milestone).

• P3 — Self-contained (Tooling): No sidecar file join is needed for title, cwd, or model.

Method

What this is, stated precisely: a versioned, evidence-backed comparison with mechanically generated scores. A measurements manifest (probe results and corpus statistics) plus an evidence checklist feed evaluate.py, which computes the Signal gates from fixed thresholds, merges the checklist verdicts, and emits the data file this page renders; measure.py recomputes per-artifact numbers from a session file. Re-running the evaluator reproduces the scoring from the versioned inputs; reproducing the observations end-to-end — harness execution to raw artifacts to extraction — is the 1.0 milestone, and until it lands this page does not claim to be a push-button-reproducible benchmark. All inputs live in the repo; to dispute a score, dispute a measurement or an evidence line and re-run the evaluator.

The probe is one identical prompt — “List the files in the current directory, then say hello in one sentence.” — attempted through each harness’s CLI, headless/one-shot mode; where it produced a session, the artifact was measured byte-for-byte, and where it could not run, the affected gate says so. Desktop apps and IDE surfaces may use different stores; v0.1 scores the CLI session store only, and says so. Completeness, Openness, and Tooling verdicts are checklist items evaluated against real session files, the sanitized fixtures in the Agent Sessions repo, and the parsers that read all ten formats in production. Stability comes from a ledger that has fingerprinted every format’s schema roughly weekly since 2026-03-31.

Grades can change, and the version scheme separates the two ways they can: the bench version (v0.1) is the rubric — gates, thresholds, scoring rules — and bumps only when a change breaks score comparability; the data date identifies a re-run under the same rubric. Formats drift — four of ten had a structural break in the last four months — and the bench re-scores when they do. v1.0 is reserved for end-to-end reproducibility: harness execution to archived artifacts to extraction, a tested crash gate, and per-event classifiers. Disputes and corrections are welcome as issues.

← The Rollout