AI Agent Evaluation Explained
Agent evaluation asks whether a goal-directed system succeeded, behaved acceptably, and did so reliably enough to operate.
An AI agent can write an excellent final message and still fail its job. It may have queried the wrong account, ignored an instruction, used an unauthorized tool, repeated an expensive call, or confidently described a change it never made.
That is why agent evaluation is broader than scoring text. It measures the outcome, the observable path to that outcome, and the system behavior surrounding the run.
For a beginner, the key idea is simple: evaluate the job the agent was supposed to complete, not just the words it produced.
What counts as an agent
An agent is a system that works toward a goal through one or more steps. It may use a model to choose actions, call tools, retrieve information, maintain state, ask a person for approval, hand work to another agent, and stop when a condition is met.
That means the evaluated object includes more than an LLM:
- instructions and prompts;
- models and parameters;
- tools and their schemas;
- retrieval and memory;
- orchestration logic;
- state transitions;
- permissions and approvals;
- external environments;
- final outputs.
A single model response can be one component of an agent run. Agent evaluation judges how the components work together.
The agent evaluation stack
Start from the outside and work inward.
Goal and outcome
Did the task actually succeed?
For a support agent, success might mean identifying the correct customer, diagnosing the issue, making only permitted changes, and resolving the ticket. A friendly response is valuable, but it does not compensate for changing the wrong subscription.
Outcome checks may inspect a final answer, database state, generated artifact, task status, user rating, or business result.
Trajectory
What observable sequence of actions led to the result?
A trajectory can include model steps, tool calls, observations, state updates, handoffs, approvals, retries, and stopping. Two runs can reach the same answer through very different paths. One may be direct and safe; the other may leak data, call unnecessary tools, and succeed by accident.
Tool behavior
Did the agent choose the right tool, prepare valid arguments, interpret the result correctly, and avoid unsupported actions?
Tool evaluation can check name selection, argument values, call order, permissions, result use, and side effects.
Constraints and safety
Did the agent follow business rules, security boundaries, user instructions, and human-approval requirements?
Constraint adherence is not an optional quality dimension. A high-scoring answer produced through a prohibited action is a failed run.
System quality
Was the run reliable, timely, and affordable? Production suitability depends on latency, failure rate, retry behavior, token use, tool cost, and variance across repeated trials.
The dimensions of a useful evaluation
Most agents need several dimensions:
Task success. Did the requested outcome occur?
Correctness. Are claims, calculations, classifications, and actions accurate?
Completeness. Were all required parts handled?
Trajectory quality. Was the action sequence reasonable and efficient?
Tool accuracy. Were tools and arguments appropriate?
Constraint adherence. Were policy, format, scope, and approval rules followed?
Safety and security. Did the run avoid harmful, unauthorized, or privacy-violating behavior?
Reliability. Does it succeed repeatedly across representative variation?
Latency. Does it finish within the expected service level?
Cost. Are model, tool, infrastructure, and human-review costs acceptable?
Not every dimension needs equal weight. A medical scheduling agent may prioritize correct identity and authorization over elegance. A research assistant may prioritize evidence quality and source coverage. A high-volume router may place strict limits on latency and cost.
Evaluation is not the same as testing
Traditional tests usually assert deterministic behavior: given fixed input, a function returns an exact output.
Agent evaluation often handles valid variation. Multiple trajectories and final responses can be acceptable. Evaluators may produce scores or labels rather than a binary assertion.
The methods overlap. Deterministic checks are excellent when the rule is exact:
- a required field exists;
- the total matches the database;
- a forbidden tool was not called;
- the task reached
completed; - latency stayed below a threshold.
Use flexible graders only where variation requires judgment.
Evaluation is not observability
Observability captures evidence about a running system: traces, logs, metrics, events, and state changes.
Evaluation applies criteria to that evidence. A trace can show that the agent called refund_customer twice. An evaluator decides whether that violated the task and whether the environment experienced a duplicate refund.
Observability makes evaluation possible at scale, while evaluation gives telemetry meaning.
Evaluation is not production monitoring
Offline evaluation runs a controlled dataset before or during development. Production monitoring watches live health and behavior.
Both matter:
- offline evals detect regressions before deployment;
- online evals sample real runs against defined criteria;
- monitoring alerts on error, latency, cost, or safety signals;
- production failures become new offline cases.
A dashboard showing 99.9% API availability does not prove agent task success. A benchmark showing high success does not prove the deployed system remains healthy.
The evaluation unit
Define what one case represents. It might be:
- one user request and response;
- one complete agent run;
- one tool decision;
- one handoff;
- one multi-turn task;
- one business workflow over several systems.
Choose the unit from the user promise. If the product promises to “resolve a support issue,” evaluating only the first answer is too narrow.
Each case should specify:
- task and relevant context;
- initial environment state;
- constraints and permissions;
- allowed or expected tools;
- success evidence;
- acceptable variation;
- failure conditions;
- evaluation rules.
The reference is not always one perfect sentence. For an agent, the reference may be a target state plus prohibited actions.
Types of evaluators
Deterministic checks
Code and database assertions are fast, repeatable, and explainable. Use them for exact properties.
Examples:
- customer ID equals the test account;
- no external-email tool was called;
- generated JSON matches the schema;
- ticket status changed to
resolved; - total tool calls did not exceed five.
Human review
People are valuable for nuanced quality, policy interpretation, and cases where the rubric is still evolving. Human labels are also useful for calibrating automated graders.
Reviewers need a clear rubric and examples. “Looks good” produces inconsistent data.
Model-based graders
A model can compare an outcome or trajectory against a detailed rubric. This scales judgment but introduces grader error, bias, variance, and possible sensitivity to irrelevant phrasing.
Prefer focused criteria, structured outputs, blinded comparisons where possible, and periodic agreement checks against qualified humans.
Environment-based evaluation
Inspect the system after the run. Did the order exist? Was the document saved? Was the correct record updated? Environment checks often provide the strongest evidence of actual task success.
User and business signals
Resolution rate, escalation, correction, adoption, or downstream conversion can reveal real value. These signals are delayed and confounded, so they should complement—not replace—controlled evaluation.
Build representative cases
An evaluation set should reflect the operating distribution, not a handful of happy-path demos.
Include:
- common tasks;
- difficult but legitimate tasks;
- ambiguous requests;
- missing information;
- tool and network failures;
- permission boundaries;
- long or conflicting context;
- adversarial or injected content;
- cases requiring approval;
- cases where the agent should refuse or escalate.
Segment results. A 90% average can hide 40% success for high-risk cancellations. Report by task type, customer segment, language, tool, risk class, and other meaningful slices.
Scores need thresholds and failure categories
A number is useful only when connected to a decision.
Define release gates such as:
- zero critical authorization violations;
- at least 95% task success on core flows;
- no more than 2% unnecessary tool calls;
- p95 latency below the product target;
- cost per successful task within budget.
Then classify failures:
- planning;
- tool selection;
- invalid arguments;
- retrieval;
- state handling;
- permission;
- model output;
- environment;
- timeout;
- stopping.
Failure categories make the result actionable. “Score dropped three points” does not tell an engineer what to change.
Repeated trials matter
Agent behavior can vary with sampling, model updates, retrieval order, and external services. Run important cases multiple times and report pass rate, variance, latency distribution, and cost distribution.
One successful attempt proves possibility, not reliability. For a consequential workflow, the question is how often it succeeds and how failures behave.
Common evaluation mistakes
- Scoring only the final answer.
- Using one ideal response as the whole reference.
- Evaluating tool choice without checking side effects.
- Averaging away critical safety failures.
- Testing only happy paths.
- Treating a model grader as ground truth.
- Changing prompts and evaluation rules at the same time.
- Ignoring latency, cost, and repeated-run variance.
- Collecting traces without a task-specific rubric.
- Using production user ratings as the only measure.
A small example
For a subscription-support agent, one evaluation case might say:
Task: Cancel auto-renewal for account A-1042 but preserve service through the paid period.
Constraints: Verify identity; never issue a refund; require approval before the account change.
Success evidence: Auto-renewal is off, end date is unchanged, one approval occurred, and the final response states the effective date.
Failure conditions: Wrong account, immediate cancellation, refund, missing approval, duplicate write, or unsupported claim.
This case evaluates the user outcome and the execution path. A beautiful response without the correct account state fails.
My Take
Agent evaluation becomes much easier when the team stops asking, “Was the answer good?” and starts asking, “What promise did the system make, what evidence proves it kept that promise, and what behavior is unacceptable even if it succeeded?”
That framing produces better datasets, better traces, and clearer release gates. It also reveals where deterministic checks outperform sophisticated graders. Use judgment for genuine judgment calls; use system state for facts the system can prove.
