Token

What is a token?

A token is a unit of text processed or generated by a language model. A token may be a complete word, part of a word, punctuation, whitespace pattern, or another text fragment. Models work with token identifiers rather than directly reading text as people do.

The exact token boundaries depend on the model’s tokenizer. The same sentence can therefore produce different token counts for different models.

How tokenization works

Before inference, a tokenizer converts text into a sequence of token IDs. The model processes those IDs and predicts the next token. Generated token IDs are then converted back into readable text.

Common words may fit into one token, while uncommon names, code, numbers, or some languages may be split into several. A token is not reliably equal to one word or one character, so word count is only a rough estimate of model usage.

Input and output tokens

Input tokens include the model’s current [context](/glossary/context/): instructions, messages, retrieved documents, tool definitions, and other supplied information. Output tokens are generated by the model.

Both matter because a [context window](/glossary/context-window/) has a finite token capacity. A very large input can leave less room for the response. Token usage can also affect inference time and provider cost, although pricing rules vary by model and service.

Simple example

Suppose an agent receives a long policy document, the full conversation history, ten tool definitions, and the current task. All of that consumes input tokens. If only one policy section is relevant, retrieving that section instead of including the entire document reduces noise and preserves capacity for reasoning and output.

Tool calls and structured data also consume tokens when their definitions, arguments, or results are placed in context.

Common confusion

Tokens are not pieces of knowledge. A higher token count does not mean the model understands more or gives a better answer. Tokens are the representation units used for processing text.

Token limits are also not memory. An agent may store information externally for months, but the model can only use the part retrieved into its current context.

Why tokens matter

Token-aware design helps agents avoid truncated outputs, overloaded context, slow responses, and unnecessary cost. Builders can summarize older messages, retrieve only relevant documents, keep tool schemas concise, and store full records outside the prompt.

The goal is not always to minimize tokens. It is to spend them on the instructions and evidence that materially improve the next decision. See [Memory in AI Agents](/memory-in-ai-agents/) for how agents choose what to keep, retrieve, and forget.