Inference

What is inference?

Inference is the runtime process of giving a trained machine-learning model an input and receiving an output. For a [large language model](/glossary/large-language-model/), the input is represented as tokens, and the output may be generated text, structured data, or a request to call a tool.

Training creates or adjusts the model’s parameters. Inference uses those learned parameters without retraining the model for every request.

How language-model inference works

An application assembles a prompt and other context, converts the text into tokens, and sends them through the model. The model calculates probabilities for possible next tokens, selects one according to its decoding settings, and repeats this process until it reaches a stopping condition.

The generated output depends on the model, current context, available token capacity, and generation settings. The same request can sometimes produce different outputs because generation may include controlled randomness.

Inference inside an AI agent

An agent may perform several model inferences during one task. One inference might select a search tool. Application code executes the tool and returns a result. A later inference interprets that result and decides whether to search again, ask the user, or provide a final answer.

This means an agent run is not one continuous model thought. It is often a sequence of separate inferences connected by state, tool results, and the agent loop.

Inference versus reasoning

Inference is the technical act of running the model. Reasoning describes the interpretation and decision-making behavior expressed through the model’s output and the wider agent system. Every model response requires inference, but not every inference involves a complex reasoning task.

Inference also differs from retrieval. Retrieval finds information in an external source. The retrieved information may then be placed into context for a new inference.

Performance and limits

Inference consumes computing resources and takes time. Larger prompts and longer outputs generally require more processing. Agent systems can multiply this work through repeated model calls, retries, planning, and reflection.

An inference can also produce incorrect or invalid output. It does not verify facts or execute permissions by itself. Applications should validate structured outputs, enforce tool boundaries, and monitor failure and cost across the complete run.

Why it matters

Understanding inference makes agent architecture easier to reason about. The model is invoked at specific decision points; software connects those invocations to tools, memory, and state. Builders can improve systems by choosing when a model call is necessary, limiting irrelevant context, selecting an appropriate model, and avoiding loops that repeat inference without meaningful progress. See [Anatomy of an AI Agent](/anatomy-of-an-ai-agent/) for the full component model.