LLM Evaluation vs Agent Evaluation
LLM evaluation judges model behavior; agent evaluation judges a goal-directed system that may use models, tools, state, and external environments.
An LLM can produce a correct, clear answer while an agent fails its task. The agent may query the wrong record, skip a required approval, call a tool twice, or never commit the requested change. The reverse can happen too: a workflow may complete the right business action but communicate it poorly.
These are related quality problems with different evaluation scopes.
LLM evaluation focuses on a model’s response or model-level behavior under defined inputs.
Agent evaluation focuses on the outcome and observable execution of the entire system pursuing a goal.
Agent evaluation includes relevant model quality, but it cannot be reduced to it.
Side-by-side comparison
| Dimension | LLM evaluation | Agent evaluation |
|---|---|---|
| Primary object | A model response or model behavior | A complete goal-directed system and run |
| Typical input | Prompt, messages, context | Task, initial environment, state, permissions, tools, and context |
| Typical output | Text, classification, structured response, embedding, or model decision | Outcome, artifacts, state changes, final response, and execution record |
| Number of steps | Often one model call or bounded exchange | Frequently multi-step, iterative, asynchronous, or multi-agent |
| Tools | May evaluate a proposed tool call | Evaluates tool selection, arguments, order, results, permissions, and side effects |
| State | Usually limited to supplied context | Persistent working, workflow, and environment state may be central |
| Environment | Often static evaluation input | Databases, APIs, files, queues, users, and remote agents can change |
| Success target | Output correctness, relevance, style, safety, or preference | Task success plus trajectory, constraints, safety, reliability, latency, and cost |
| Reference | Expected answer, label, rubric, or preference | Target outcome, acceptable trajectories, invariants, and prohibited actions |
| Failure modes | Hallucination, poor reasoning result, format error, bias, refusal error | Planning, tool, state, permission, environment, loop, handoff, timeout, stopping, plus model failures |
| Repetition | Measures model variance | Measures compounded variance across model and system dependencies |
| Evidence | Prompt, response, label, grader result | Trace, tool records, state, artifacts, final output, and environment checks |
| Reliability | Model success across a distribution | End-to-end success across tasks, dependencies, retries, and operational conditions |
| Latency | Time for model response | Total time including planning, tools, queues, approval, and remote work |
| Cost | Usually token or inference cost | Model, tool, retrieval, infrastructure, remote-agent, and human-review cost |
| Security | Output safety and model behavior | Identity, authorization, approval, data movement, and side-effect safety |
The table does not imply that every LLM eval is single-turn or every agent eval is long-running. The distinction is the evaluated boundary.
Evaluation scope
Suppose a subscription agent receives:
Cancel renewal but keep my service active through the paid period.
At the model level, you might evaluate whether the response is polite, mentions the correct effective date, and does not invent a refund.
At the agent level, you also evaluate whether it:
- identified the authenticated account;
- read the current cancellation policy;
- obtained approval before changing the account;
- called the correct tool with the correct identifier;
- changed only auto-renewal;
- avoided a duplicate write;
- verified the final state;
- answered consistently with that state;
- completed within latency and cost limits.
The final response is one piece of evidence, not the entire outcome.
What LLM evaluation is good at
LLM evaluation is appropriate when the unit under study is the model or a bounded model interaction.
Common dimensions include:
- factual correctness against a reference;
- relevance;
- instruction following;
- classification accuracy;
- structured-output validity;
- summarization fidelity;
- tone and style;
- safety behavior;
- preference between responses;
- calibration or uncertainty expression.
Methods include exact match, string or schema checks, task-specific metrics, human review, pairwise preference, and model-based graders.
Model evals are valuable for choosing a model, tuning instructions, comparing prompt templates, validating structured output, and diagnosing response quality. They isolate the model layer so teams can improve it without confounding every external dependency.
What agent evaluation adds
An agent makes or participates in decisions over time. Evaluation therefore adds:
Outcome state
Did the external world reach the intended state? Inspect the database, artifact, task record, or system of record.
Trajectory
Was the observable action sequence valid, efficient, and safe? Check required steps, prohibited actions, ordering, retries, and stopping.
Tool semantics
Did the agent choose the correct capability and prepare arguments that match the user and resource? A schema-valid call can still target the wrong account.
Permission and approval
Was the caller authorized? Did the user approve the actual consequential action after its details were known?
Environment behavior
How did the agent respond to timeouts, stale results, partial failures, concurrency, and changing data?
System performance
What were end-to-end latency, success rate, variance, and cost per successful task?
These properties belong to the application, orchestration, tools, and deployment as much as to the model.
A model pass with an agent failure
Imagine the model produces:
Auto-renewal is now disabled. Your Pro service remains active through August 31.
The response passes tone, clarity, and factual-format checks. But the trace shows the write tool returned a timeout and the agent never reconciled the account state. Auto-renewal remains on.
The LLM response may be linguistically strong. The agent outcome is false, so the run fails.
Another example: the model selects setautorenewal with valid JSON, but uses an account ID copied from untrusted document content instead of the authenticated session. Tool-call syntax passes; authorization and task correctness fail.
An agent pass with a model-quality problem
Suppose the correct account is updated, approval is captured, and the final state is verified. The final response says:
Done.
The core task succeeded, but communication quality is poor. It omits the paid-through date and leaves the user uncertain.
A useful scorecard does not force this into one binary result. It can mark:
- outcome: pass;
- authorization: pass;
- trajectory: pass;
- response completeness: fail;
- overall release decision: depends on policy.
This separation helps the team fix the correct layer.
Reference answers versus outcome specifications
LLM evals often use a target answer, label, or rubric. Agent tasks need richer references:
- initial state;
- expected final state;
- constraints;
- allowed capabilities;
- required approvals;
- acceptable action orderings;
- prohibited side effects;
- final-response requirements.
For a research agent, there may be many valid reports. The reference can specify required sources, claim support, date range, citation validity, and coverage rather than one golden paragraph.
For a transaction agent, environment state may be the primary reference.
Exact trajectory versus invariants
It is tempting to define one ideal sequence and penalize every deviation. That creates brittle evals.
Prefer invariants:
- identity verification occurs before protected access;
- approval occurs before the write;
- only one write occurs;
- final state is checked before claiming success;
- the agent stops after completion.
Allow harmless variations, such as reading policy before or after account state, when both satisfy the constraints.
Exact sequence checks remain appropriate for regulated or safety-critical workflows where only one order is acceptable.
Different failure taxonomies
LLM-level failure categories can include:
- incorrect answer;
- unsupported claim;
- irrelevant response;
- malformed structure;
- instruction failure;
- unsafe content;
- excessive refusal;
- poor calibration.
Agent-level categories add:
- task decomposition;
- wrong tool or remote agent;
- invalid tool arguments;
- retrieval failure;
- state loss or conflict;
- permission violation;
- missing approval;
- duplicated side effect;
- dependency timeout;
- retry loop;
- handoff failure;
- incorrect stopping;
- outcome-response mismatch.
Tagging failures at the right level prevents prompt tuning from becoming the default answer to every problem.
Reliability compounds
An agent’s reliability depends on several probabilistic and operational components.
If a workflow needs a model decision, retrieval, two tool calls, and a final synthesis, the end-to-end pass rate can be lower than the pass rate of any one component. Dependencies may also be correlated, so multiplying component averages is only a rough intuition—not a substitute for end-to-end measurement.
Run full tasks repeatedly under representative conditions. Measure pass rate, failure distribution, and variance. Then use component evals to locate the source.
An LLM benchmark can say a model is capable of a task. An agent eval says whether this configured system accomplishes it reliably.
Latency and cost boundaries
For an LLM, latency usually covers request to model response, and cost is dominated by inference tokens.
For an agent, latency may include:
- several model calls;
- retrieval;
- tool execution;
- queues;
- remote-agent work;
- retries;
- human approval.
Agent cost may include model tokens, searches, databases, paid APIs, remote services, infrastructure, and human review.
Measure cost per successful task. A cheap run that fails and escalates can be more expensive than a higher-token run that resolves the issue.
Safety and security
Model safety evals can test whether outputs follow content and behavior policies. Agent safety additionally tests what the system can do:
- access protected data;
- send messages;
- move money;
- modify accounts;
- publish content;
- delegate data to another agent.
A safe-sounding response does not prove safe execution. Agent evals need adversarial cases, permission boundaries, prompt-injected content, destination checks, and environment assertions.
Critical violations should usually be zero-tolerance gates rather than one dimension averaged with style.
Graders for each level
Use the simplest reliable method for each criterion.
For LLM outputs:
- exact or normalized match;
- schema validation;
- source-grounded factual checks;
- human rubric;
- calibrated model grader;
- pairwise comparison.
For agent systems:
- all of the above;
- trace assertions;
- tool and argument checks;
- state-transition checks;
- environment validation;
- permission and approval checks;
- latency and cost measurement;
- repeated-run statistics.
A model grader can judge whether a summary is clear. It should not be asked to infer whether a database was actually updated when the environment can answer directly.
How to combine the two
Use nested evaluation:
- define the end-to-end agent promise;
- evaluate task outcomes and critical constraints;
- evaluate trajectory and operational behavior;
- evaluate model outputs at important steps;
- attribute each failure to the owning layer;
- improve the component;
- rerun both focused and end-to-end suites.
This avoids two mistakes: relying only on broad agent scores, or optimizing isolated model metrics that do not improve the product.
For example, a model-level tool-selection eval can quickly compare prompts. The full agent suite then verifies that the selected call works with authentication, actual tool schemas, state, retries, and outcome.
Release decisions
A scorecard might use:
- critical agent constraints: pass/fail gates;
- end-to-end task success: primary outcome;
- trajectory quality: diagnostic and efficiency;
- response quality: user experience;
- model-specific metrics: component diagnosis;
- latency and cost: operational acceptance.
Do not add every metric into one opaque weighted average. A permission violation must not disappear behind excellent writing scores.
Compare candidate and baseline on the same cases and environment. Preserve a held-out set, record versions, and inspect newly introduced failure types.
When you need only LLM evaluation
LLM evals may be sufficient for:
- a bounded classifier;
- a rewrite feature;
- a summarizer with no actions;
- extraction into a schema;
- choosing between models for a single generation step.
Even then, evaluate the application contract around the model when parsing, retrieval, or user-facing integration can fail.
When agent evaluation is mandatory
Use end-to-end agent evaluation when the system:
- calls tools;
- changes external state;
- operates over several steps;
- uses persistent state or memory;
- requires authorization or approval;
- hands work to other agents;
- retries or runs asynchronously;
- promises a business outcome.
The greater the consequence and autonomy, the more important trajectory and environment evidence become.
Common mistakes
- Calling a final-response rubric an agent evaluation.
- Treating valid tool-call JSON as correct tool use.
- Ignoring external state.
- Requiring one golden action sequence.
- Averaging critical violations into a quality score.
- Using one model judge for every layer.
- Optimizing model accuracy while task success stays flat.
- Measuring inference cost but ignoring tools and human review.
- Comparing configurations on different environments.
- Diagnosing every end-to-end failure as a prompt problem.
My Take
LLM evaluation and agent evaluation should form a hierarchy, not a rivalry. Model evals isolate capabilities and make iteration fast. Agent evals protect the user promise by testing how models, tools, state, policy, and infrastructure behave together.
Start with the end-to-end outcome, then use model-level measures to explain and improve it. That ordering keeps the system honest: excellent language is valuable, but only after the agent has done the right work for the right user under the right constraints.
