Latency

What is latency?

Latency is the elapsed time between starting an operation and receiving the response or completion that matters to the user or system. In AI agents, total latency can include model inference, retrieval, tool calls, planning steps, safety checks, retries, and coordination with other agents.

The first visible token and the final completed task can have very different latency, so teams should measure both when they matter.

Sources of agent latency

An agent request often crosses several components:

• model latency for input processing and output generation;
• retrieval latency for searching, filtering, and reranking context;
• tool latency for APIs, databases, browsers, or code execution;
• multi-step latency as the agent repeats reason–act–observe cycles;
• multi-agent latency from routing, handoffs, messaging, and waiting for parallel or sequential work.

Parallel execution can reduce elapsed time when operations are independent. It can also add coordination overhead or increase load.

Simple example

A research agent spends 1.5 seconds retrieving sources, 4 seconds on the first model call, 8 seconds across three web tools, and 5 seconds synthesizing the answer. The user experiences roughly 18.5 seconds if the stages are sequential, even though no single model call took that long.

Measuring only model speed would miss the main bottleneck.

Latency versus throughput

Latency measures how long one request or operation takes. Throughput measures how much work a system can complete per unit of time.

A service can have low latency for individual requests but low total throughput because it handles few requests concurrently. Another can process many requests per second while some users still wait a long time. Capacity planning needs both measures.

More steps versus better outcomes

Additional retrieval, reflection, or agent calls may improve quality, but each step adds delay and failure opportunities. More steps are useful only when their expected value exceeds their time and cost.

Why it matters

Latency changes user trust and product behavior. Interactive assistants need quick feedback, while background workflows may prioritize reliability over immediate completion.

Builders should trace end-to-end time, set budgets by stage, stream useful progress, cache safe results, parallelize independent work, and remove low-value loops. See [How AI Agents Work](/how-ai-agents-work/) for the execution cycle that creates these delays.