Feature 1 · 2026-07-20

Env inference

Environment inference: a working copy of your system, built from its logs. To find out what an AI agent would really do to your systems, you need somewhere it can do the job without touching production. Building that practice copy by hand is the expensive part — so Holdout builds it automatically, from logs and exports your systems already produce. The proof it worked: we replay real traffic the copy has never seen and check that it responds the way the real system did.

The dataset

The complete change history of a corporate IT help desk running ServiceNow: 24,918 real support tickets and every update made to them over a full year, published (anonymized) by the company for research. This is exactly the kind of log most ops teams already have and never think of as an asset.

Source: UCI Machine Learning Repository, dataset #498 (CC-BY)

A year of real help-desk history100% answers matched · 100% of traffic covered
Built from 11,466 recorded updates across 2,000 real tickets; then asked to answer 4,769 requests it had never seen — 1,468 of them answerable only by remembering earlier changes. We made it hard on ourselves: confirmations carried no data back, so the copy couldn't echo answers — it had to keep its own books on every ticket.
Partial history + one export, combined100% answers matched · 99.7% of traffic covered
Built from complete history for just 500 of 2,000 tickets, plus a one-time export of all of them; then asked to handle traffic across the full 2,000. Merging the export lifted coverage from 76.7% to 99.7% — what it learned from 500 tickets carried over to the rest.
How to read the numbers. We build the copy from one part of the history, then replay requests from a later part it has never seen. “Answers matched” is how often the copy responded exactly like the real system did; “traffic covered” is how much of the replay touched tickets the copy had learned about. The request format is authored by our adapter — everything learned comes from the real data inside it.