Testing AI Agents with Repeatable Evaluations

Define what success means

Start from the workflow contract. A support task may require correct intent, approved policy evidence, authenticated account access, valid tool arguments, confirmed action, and accurate final explanation. List prohibited outcomes too, such as cross-account disclosure, invented policy, duplicate writes, or failure to escalate. A single overall score hides which guarantee failed.

Choose metrics that reflect user and operational consequences. Useful measures include task completion, evidence support, correct tool selection, argument validity, unnecessary calls, policy violations, reviewer correction, and recovery from tool errors. Set thresholds by risk and class. Rare consequential failures may require zero observed violations in the release set rather than a strong average.

Build a representative case set

Create cases from real workflows, reviewed incidents, domain-owner examples, and known boundaries. Include ordinary requests, ambiguous input, missing evidence, unsupported goals, permission differences, tool failures, conflicting sources, and adversarial content. Hold back some cases so prompt authors do not tune every visible example while leaving the underlying behavior weak.

Each case needs initial state, user request, available tools, expected or acceptable decisions, required evidence, forbidden actions, and completion criteria. Use a sandbox or simulated tools with controlled responses for reproducibility. For side effects, verify the resulting state and call count. A final message that claims success must fail when the mock system shows no change.

Choose reliable graders

Prefer deterministic assertions for schema validity, tool names, arguments, permissions, citations, arithmetic, state transitions, and prohibited strings. Use domain experts for nuanced policy or communication judgments with a concrete rubric and examples. A model grader can assist at scale, but validate it against human labels, constrain its inputs, and monitor disagreement rather than treating it as unquestionable truth.

Grade intermediate behavior when it matters. Check whether the expected source was retrieved, whether an approval preceded the write, and whether a retry reused an idempotency key. Allow multiple valid paths where the workflow permits them. Do not require one exact sentence when the real contract is factual support, safe action, and understandable communication.

Run comparisons and preserve evidence

Pin the model configuration, prompts, tool schemas, data fixture, and grader version for each run. Repeat important generated cases enough times to estimate failure frequency. Compare a proposed change with the current baseline on the same cases and inspect regressions by category. Record traces and grader reasons under appropriate privacy limits.

Promote every meaningful production failure into a minimized regression case. Keep release gates focused on user harm and contractual behavior, not cosmetic variation. Monitor evaluation-set drift as workflows and policies change. Provider behavior evolves, so consult current official evaluation guidance quarterly while preserving your own versioned test inputs and expected outcomes.

Evaluate a refund-preparation agent

A team has an agent that reads an order, checks policy, drafts a recommendation, and requests employee approval before refunding.

1. Create fixtures for eligible, ineligible, ambiguous, already-refunded, wrong-account, missing-policy, and tool-timeout cases.
2. Assert account authorization, expected policy retrieval, valid amount calculation, and absence of a refund call before approval.
3. Simulate approve, reject, expired approval, changed order state, and duplicate execution, then inspect actual ledger state and call count.
4. Use a support expert rubric for explanation quality while deterministic checks grade money, identity, sequence, and policy citations.
5. Repeat variable cases, compare failure rates with the baseline, and add any discovered unsafe trajectory to the permanent suite.

Agent evaluation case template

Define each test so another reviewer can reproduce the setup, path checks, and verdict.

• Setup: user role, external state, available tools, source data, and dependency versions.
• Request: exact input, relevant history, ambiguity, and expected handling class.
• Required behavior: evidence, decisions, calls, approvals, state transitions, and final outcome.
• Forbidden behavior: disclosure, unsupported claim, unauthorized call, duplicate effect, and false success.
• Grading: deterministic assertions, expert rubric, repetitions, threshold, and retained trace fields.

Common mistakes

• Judging an agent from polished final answers while ignoring incorrect tool calls and unchanged external state.
• Using only happy-path examples written by the same person who wrote the prompt and workflow.
• Changing model, prompt, tools, fixtures, and graders together so a score difference cannot be diagnosed.

Try one

Sources

• OpenAI evaluation best practicesOfficial methods for defining evaluation criteria and testing model behavior.
• OpenAI agents guideOfficial guidance for building agent workflows with current OpenAI platform capabilities.