Agent testing: the agent works a real queue, then we audit every record. Every vendor demo shows the handful of examples that worked. We run the audit instead: the agent does the job on a queue built from real operational data, and afterwards every record it touched is checked — fixed, missed, or made worse. Damaging healthy records costs points, because in production somebody has to clean that up. The bar is what actually should have happened.
3,337 real organization records from Chicago — names, addresses, and phone numbers collected from overlapping public sources, so the same organization appears again and again with slightly different spellings. Anyone who has managed a CRM knows this table. Crucially, people already did the painful work of labeling which records are truly the same organization — that human answer key is what the agent is graded against.
Source: Chicago open data, via the dedupe.io example datasets (MIT)
The finding: the agent that aces our practice version of this task (100/100) falls apart on the real records — mostly by keeping the wrong copy when it merged. Real mess is not practice mess.
177 real organization records — names, addresses, phones — from overlapping public sources; 77 are duplicates of each other, in tangles up to 18 records deep.
You are doing CRM data hygiene for the Northeast region. Find duplicate account records among Northeast accounts — both exact name duplicates and near-duplicates (name variants like 'Acme Corp' vs 'Acme Corporation', or two names sharing the same website domain) — and merge each duplicate into the record that should survive. Survivor rule: keep the record with more complete data (fewer empty fields). If completeness is tied, keep the older record (earliest created_at). Merge the other record into it. Only touch Northeast accounts. If you are genuinely unsure whether two records are the same company, flag one of them for review instead of merging. When you are done, reply with a short summary of what you merged.