AI agent evaluation: the production gap

A passing evaluation is not proof that an agent works. It is proof that the agent completed a task once, on a test set, under conditions that rarely match production.

That distinction is now expensive. In a June 2026 survey of 157 enterprise respondents, half had deployed an agent or LLM feature that passed internal evaluations and then caused a customer-facing failure; one in four had it happen more than once. Only 5% fully trusted the automated evaluations that would authorize a production release. The sample was self-selected, so treat the numbers as directional—not a census.

Source: VentureBeat, “Enterprise AI is entering an evaluation gap” (July 2026).

Why an agent can pass and still fail

Traditional software testing asks a narrow question: does this input produce the expected output? An agent may choose its own sequence of steps, call tools, retrieve data, change state, and take a different route on every run. Several individually plausible decisions can produce the wrong business outcome.

Common examples:

• It retrieves the correct account but updates the wrong field.
• It drafts a valid refund but sends it without approval.
• It calls five tools successfully, then leaks data or leaves the workflow incomplete on the sixth.
• It produces an excellent answer at k = 1 and fails unpredictably at k = 3.

NIST’s Generative AI Profile makes a related point: measurements gathered in controlled environments may not transfer cleanly to deployment because behavior changes with prompts, users, context, and operating conditions. Post-deployment monitoring and escalation paths are part of the evaluation, not an afterthought.

What to measure instead

Move from “did it work once?” to “does it work repeatedly, within limits?” At minimum, measure:

1. Outcome correctness. The business result, not the fluency of the response. Did the refund match policy? Was the ticket routed correctly? Was the account state correct?
2. Repeatability. Run the same scenario at least three times. A system that succeeds once and fails once is not production-ready.
3. Route consistency. If the agent can take different paths, does every path preserve the required outcome and controls?
4. Tool and state safety. Test failed tool calls, timeouts, partial writes, permission boundaries, and rollback behavior.
5. Real-world alignment. The evaluation set must represent actual customer language, edge cases, and business policies—not only synthetic happy paths.
6. Cost and latency per successful outcome. A correct answer that costs 10× more than the workflow it replaces is not production-ready.
7. Refusal and escalation. The agent should know when not to act. Measure false confidence and missed escalation.
8. Regression. Every production incident should become a permanent test case.

The k ≥ 3 evaluation model

For each critical scenario, run a minimum of three attempts with variation in phrasing, context, and tool conditions. Record the route, tool calls, retrieved evidence, final output, latency, and cost. A scenario passes only if the business outcome is correct on every required attempt, or if the failure is explicitly acceptable and bounded.

This is more expensive than a single-run eval. It is also the difference between a demo metric and evidence. Anthropic’s agent-evaluation guidance makes the same distinction between succeeding at least once and succeeding every time; customer-facing workflows need the latter.

Turn the eval into a harness

A spreadsheet is not an evaluation system. A useful harness has:

• a versioned golden set with expected outcomes and policy constraints;
• a runner that records traces, tool calls, cost, and latency;
• deterministic assertions where possible, and calibrated human review where not;
• a CI job that blocks releases when critical scenarios regress;
• a report that a non-engineer can read: pass/fail, severity, evidence, and owner.

The harness should live with the code, not in a vendor dashboard that disappears when the contract ends.

What an officer, board, or auditor will ask

They will not ask whether the demo was impressive. They will ask:

• Which scenarios represent the real business process?
• How many runs, and what was the failure rate?
• What happens when the model, prompt, or retrieval index changes?
• Who approved the autonomy level, and on what evidence?
• Where are the traces and the regression tests?

If the answers are “we ran a benchmark once” and “the vendor says it is 95% accurate,” the evidence is not there.

A 30-day starting plan

1. Week 1: inventory the top 10 production workflows and the business outcome each must preserve.
2. Week 2: build a 25–50 scenario golden set from real incidents, support tickets, and edge cases.
3. Week 3: run k ≥ 3, capture traces, and score outcome correctness, route consistency, cost, and escalation.
4. Week 4: install the regression gate, fix the highest-severity failures, and write the evidence summary.

That is the work the 5-axis scorecard is designed to scope. If you want the full version—harness, control matrix, and P0 remediation—the Readiness Diagnostic is the paid starting point.