How we measure agents on enterprise work

Arcadia is a simulated robotics manufacturer of roughly eight hundred people, with an org chart, departments, payroll, procurement, hiring pipelines, calendars, and the mail that stitches them together. It runs across seven real business applications (an ATS, two HR systems, procurement, mail, calendar, and spreadsheets), each a faithful server the agent drives through the same kind of API surface it would meet in production.

Tasks aren't written one by one. The world already contains situations, such as a new-hire cohort mid-onboarding, a spend anomaly in procurement, two systems that quietly disagree about the same employee, and a generator walks a typed graph of the available tools outward from those situations, producing a task, a reference solution, and the checks that define success, all at once. Every generated task is validated before it ships: the reference solution must replay through the production grader at exactly 1.0, and a do-nothing attempt must score exactly 0.

The second environment is a staffing agency generated from population-level parameters. Recruiters, clients, requisitions, and candidate flow are sampled so that each world instance is different while the business stays statistically the same. What makes it hard isn't the volume; it's the handbook. The agency's operating handbook is binding: submission rules, disclosure rules, rate policies. An agent that completes the work while breaking the handbook has not completed the work.

• Deterministic checks carry 100% of the score. Success is defined by the state of the world after the agent worked: records created, fields set, and approvals routed, all verified mechanically. No model grades its own homework into the headline number.
• Guards gate multiplicatively. Policy and safety guards multiply the score rather than average into it, so a do-nothing agent scores exactly 0 and a rule-breaking one can't buy back points with volume.
• Gold solutions replay at exactly 1.0. Every task's reference solution is replayed through the same production grader that scores agents. If it doesn't come back at 1.0, the task is broken and rejected rather than the grader bent to fit.
• Tasks must require the tools. If a model with no tool access can answer from the instruction alone, the task measures reading comprehension rather than agency, so it is rejected.
• pass^k with confidence intervals. We report the unbiased pass^k estimator, the probability that all k of k sampled attempts succeed, averaged across tasks, with intervals computed across task-level means rather than pooled attempts, which would understate the uncertainty.
• Environment errors are attributed, not scored. Every rollout is classified as completed, partial, agent error, or environment error. Environment errors are excluded from every success denominator: a broken environment never reads as a dumb agent.

This section is a placeholder. We publish numbers only from completed measurement runs, and the first one is still pending. When it lands, the results will appear here and on The Trials as a committed snapshot of the same leaderboard we read internally.