Token Cost

What is token cost?

Token cost is the monetary or compute expense associated with processing input tokens and generating output tokens during model use. In hosted model services, pricing may differ by model and by token type. In self-hosted systems, the expense appears through compute capacity, energy, infrastructure, and operational overhead.

For agents, the useful business measure is usually cost per completed task, not cost per individual model call.

What drives token cost?

Agent workflows can accumulate tokens through:

• system and agent instructions repeated on each call;
• conversation history and retrieved context;
• model-generated reasoning or intermediate outputs;
• tool definitions and tool results;
• retries, reflection, and replanning;
• multiple agents exchanging messages;
• long final responses.

The [context window](/glossary/context-window/) sets how much tokenized information a model can consider in one inference, but it does not mean every available token should be filled.

Simple example

An agent uses 8,000 input tokens and 1,000 output tokens for its first call. It then performs three more calls, each including most of the same context plus new tool results. The task’s total processed tokens are much larger than the size of the final answer.

A cost review might compress old history, retrieve only relevant passages, shorten tool descriptions, or use a smaller model for routing.

Token cost versus token count

[Token count](/glossary/token/) measures the number of tokens processed or generated. Token cost converts that usage into money or compute expense under a particular model and deployment.

The same token count can have different costs across models, providers, caching policies, or hardware. Cost calculations should therefore retain both usage and applicable pricing or compute assumptions.

Long context versus low cost

A model supporting a long context does not guarantee inexpensive use. Supplying more context may increase processing cost and latency, and irrelevant text can reduce answer quality. Selective retrieval and summarization can be better than repeatedly sending everything.

Why it matters

Agents may call models many times, so small per-call inefficiencies compound. More steps do not automatically create better outcomes.

Teams should track cost per successful task, separate input and output usage, include failed attempts, and evaluate quality alongside savings. See [How AI Agents Work](/how-ai-agents-work/) for the loop that causes repeated model and tool activity.