The problem
If you work in operations, you've lived this: an AI tool arrives with a great demo, gets pointed at your queue, and a month later you're untangling what it did. Agents are bought on demos and updated silently. The demo shows the examples that worked. The update changes behavior with no release note. And nobody re-tests after go-live.
Both failure modes showed up in our own runs. One agent submitted a confident, itemized report of 55 record merges; its own activity log shows 0 actual changes. A model update dropped one duty from 94.4 to 21.4 while everything else stayed perfect — exactly the kind of quiet break nobody notices until the queue backs up.
What Holdout does
Build a copy, run the agent on the copy, audit everything it did:
- Env inference — rebuild the system from logs and exports you already have — no new instrumentation. Proven by replaying real traffic the copy never saw (98.5% of answers matched on our practice system, 100% on a real help desk's year of history).
- Set the bar before the run. Real records with a known right answer — what real people actually did, or the written policy applied perfectly.
- Run the agent through its normal tools. It can't tell it's in a copy. Every action lands on the record.
- Audit the end state item by item: fixed, missed, made worse. Damage subtracts points. No AI grading AI.
- Re-run nightly. Reset takes a tenth of a second. A score drop names the exact records that broke.
What we found
- Real mess is harder than practice mess — sometimes. An agent that scores 100 deduplicating our practice CRM scores 24.4 on real records with a human answer key. On invoice review, real data barely moved scores (the bar 100, comparison 90.5) — there, the difficulty is the policy, not the records. You only learn which kind of task you have by running it.
- “Task complete” can still mean a cleanup job. On a real help-desk queue, the strongest model handled every checklist item — and closed 11 healthy tickets on the way out, reporting dates that weren't what the records said. With damage subtracted it scores 12/100. The summary said done; the audit said otherwise.
- The report and the activity log are different documents. A confident summary of work never performed reads identically to a real one. Only the log, checked against an answer key, tells them apart.
- Your existing data is enough to start. Full history for a quarter of the records plus one export replayed the whole population (99.7% covered, 100% of answers matched).
What's real, what isn't
Practice environments are built and messed up by us, on purpose. Real-data results use public datasets where the right answer is known — from what people actually did, or from written policy. We disclose per dataset which parts are real and which are constructed. Every number on this site is exported from run records; none are typed in.
Env inference results →
Agent testing results →
Code and run records →