For the detail-minded · 2026-07-20

How the scoring works

The short version: every run starts from the identical mess, the answer key is fixed before the run, scoring is done by rules — not by asking another AI what it thinks — and damage costs points. Below that, the five practice tasks we built the harness on.

The bar is set per dataset
On practice tasks, it's the strongest model's run. On real datasets, it's what real people actually did — the human-made answer key, or the written policy applied perfectly.
Same starting line, every run
The environment resets to a byte-for-byte identical state in about a tenth of a second, verified each time. Every agent faced exactly the same queue, so scores compare.
Scored by rules, not opinions
After a run, each item is compared to the answer key mechanically: fixed, missed, or made worse. There is no AI grading AI. Escalating a genuinely unclear item counts in the agent's favor; guessing counts against it.
Damage is shown, not summarized
Every run shows one square per graded case — green handled right, amber half credit or escalated, red handled wrong, grey left undone. Writes the agent made to records outside those cases get their own hatched red squares on the end of the row, one per action, so a run that damaged records it was never asked to touch is visibly longer than the bar. Hover any square to see which record it was, or open the list under the row to read them all.
Calibrated end to end
A perfect run scores exactly 100 and doing nothing scores 0 — asserted by automated tests. Scores compare runs of the same task, never across tasks.
We audit our own environments too
Three times, the per-item breakdown showed the test being unfair rather than the agent being wrong. Each fix is now a guard in the test suite, and stored runs were re-scored retroactively — no re-run needed, because every action is on the record.

The five practice tasks

Built to develop the harness; useful as a control condition. One tab per task: the exact instructions, the bar, and comparison runs.

A stand-in sales system: 518 company records, 12 actions an agent can take, and 9 planted problems with known fixes — reset to the identical starting state for every run.

5 near-duplicate records4 exact duplicate records

Merge score — the bar vs. what each agent did

The bar · gpt-5 agent — strongest-model run against the built-in answer key
9 handled right
gpt-4.1 · agent-a-v2
9 handled right
after a vendor-style model swap
gpt-4o-mini · agent-b
9 left undone
its final report claimed 55 merges; the logs show 0 actual changes
Read the exact instructions the agent was given
You are doing CRM data hygiene for the Northeast region.

Find duplicate account records among Northeast accounts — both exact
name duplicates and near-duplicates (name variants like 'Acme Corp' vs
'Acme Corporation', or two names sharing the same website domain) — and
merge each duplicate into the record that should survive.

Survivor rule: keep the record with more complete data (fewer empty fields).
If completeness is tied, keep the older record (earliest created_at).
Merge the other record into it.

Only touch Northeast accounts. If you are genuinely unsure whether two
records are the same company, flag one of them for review instead of merging.
When you are done, reply with a short summary of what you merged.
@techreport{holdout2026v0, title = {Holdout v0: Grading AI Agents Against Reconstructed Environments}, author = {Holdout Systems}, year = {2026}, url = {https://github.com/NuoWenLei/holdout-v0} }