Context and Token Cost Optimization

Build focused model context that preserves decision-relevant information while removing repeated and irrelevant tokens.

TL;DR

• A context window is capacity, not a target. Good context engineering selects the smallest set of instructions, state, history, knowledge, tool definitions, and evidence that lets the model make the current decision reliably. Token reduction is useful only when relevance and task quality survive.
• Start by making system instructions and examples and conversation history visible in the complete task path.
• Avoid the shortcut of pasting everything remembered; test the failure state before release.
• Judge the result with tokens by context component and task success after compaction, not an isolated infrastructure number.

The production mental model

A context window is capacity, not a target. Good context engineering selects the smallest set of instructions, state, history, knowledge, tool definitions, and evidence that lets the model make the current decision reliably. Token reduction is useful only when relevance and task quality survive.

Keep five layers separate. Model optimization changes inference; agent optimization changes decisions and tool use; workflow optimization changes sequencing and recovery; infrastructure optimization changes capacity and execution; business optimization changes whether the result is worthwhile. For Context and Token Cost Optimization, the most revealing evidence is usually tokens by context component together with task success after compaction. That pairing prevents a local improvement from disguising a worse task outcome.

More tokens can add cost and distraction without adding evidence.

A concrete example

A long-running support agent repeatedly sends system instructions, the entire conversation, every tool schema, old tool results, retrieved documents, memory, and the current task. Much of that material is duplicated or no longer relevant. A focused builder keeps durable facts in structured state, summarizes older dialogue, retrieves only useful memory, reranks knowledge, and includes tools relevant to the current step.

The example is deliberately vendor-neutral. Exact provider limits, prices, model features, and service guarantees change; the durable design is to discover those values from the selected service, keep them in versioned configuration, and test the behavior that occurs when an assumption stops being true.

How the system works

• System instructions and examples. Trace it as part of the complete task so local improvements do not hide downstream cost or failure.
• Conversation history. Give it an explicit owner, boundary, and failure result; implicit behavior becomes difficult to test.
• Structured agent state. Measure it by workload class because read-only, interactive, background, and side-effecting tasks have different constraints.
• Retrieved memory. Keep the mechanism deterministic where it enforces identity, authority, durability, or resource limits.
• Rag documents and chunks. Trace it as part of the complete task so local improvements do not hide downstream cost or failure.
• Tool definitions and results. Give it an explicit owner, boundary, and failure result; implicit behavior becomes difficult to test.
• Multi-agent messages. Measure it by workload class because read-only, interactive, background, and side-effecting tasks have different constraints.
• Input and generated output. Keep the mechanism deterministic where it enforces identity, authority, durability, or resource limits.
• Deduplication and provenance. Trace it as part of the complete task so local improvements do not hide downstream cost or failure.
• Information loss risk. Give it an explicit owner, boundary, and failure result; implicit behavior becomes difficult to test.

Relevance decisions happen before inference.

A practical implementation path

1. Inventory every source entering the prompt

Write down the current baseline and the evidence that would disprove the change. For this step, inspect structured agent state, watch tokens by context component, and explicitly test pasting everything remembered. Record the assumption, owner, expected result, and safe terminal state so the team can distinguish a genuine improvement from work shifted elsewhere in the system.

2. Separate durable state from conversational prose

Place this responsibility in the component that can enforce it consistently. For this step, inspect retrieved memory, watch task success after compaction, and explicitly test retrieving as much text as possible. Record the assumption, owner, expected result, and safe terminal state so the team can distinguish a genuine improvement from work shifted elsewhere in the system.

3. Select only history relevant to the current decision

Version the decision so a run can be reconstructed after dependencies change. For this step, inspect RAG documents and chunks, watch retrieval relevance, and explicitly test dropping critical constraints to save tokens. Record the assumption, owner, expected result, and safe terminal state so the team can distinguish a genuine improvement from work shifted elsewhere in the system.

4. Summarize older interactions with preserved facts and provenance

Exercise the denied, unavailable, stale, duplicate, and unknown-result paths as applicable. For this step, inspect tool definitions and results, watch duplicate chunk rate, and explicitly test summaries with no provenance. Record the assumption, owner, expected result, and safe terminal state so the team can distinguish a genuine improvement from work shifted elsewhere in the system.

5. Retrieve memory and knowledge on demand

Use a production-representative trajectory; an empty endpoint test misses accumulated agent behavior. For this step, inspect multi-agent messages, watch latency and cost per success, and explicitly test duplicating the same document in history and RAG. Record the assumption, owner, expected result, and safe terminal state so the team can distinguish a genuine improvement from work shifted elsewhere in the system.

6. Rerank and deduplicate before context construction

Keep identity, permission, and resource limits outside model discretion. For this step, inspect input and generated output, watch fact omission and contradiction rate, and explicitly test assuming a larger context is automatically better. Record the assumption, owner, expected result, and safe terminal state so the team can distinguish a genuine improvement from work shifted elsewhere in the system.

7. Trim large tool results into verified structured fields

Bound the work and define what happens when the bound is exhausted. For this step, inspect deduplication and provenance, watch tokens by context component, and explicitly test pasting everything remembered. Record the assumption, owner, expected result, and safe terminal state so the team can distinguish a genuine improvement from work shifted elsewhere in the system.

8. Load tool definitions selectively where supported

Attach telemetry to the task and compare the result with the previous version. For this step, inspect information loss risk, watch task success after compaction, and explicitly test retrieving as much text as possible. Record the assumption, owner, expected result, and safe terminal state so the team can distinguish a genuine improvement from work shifted elsewhere in the system.

9. Set output budgets appropriate to the task

Roll out gradually when the decision can affect live users or external systems. For this step, inspect system instructions and examples, watch retrieval relevance, and explicitly test dropping critical constraints to save tokens. Record the assumption, owner, expected result, and safe terminal state so the team can distinguish a genuine improvement from work shifted elsewhere in the system.

10. Evaluate whether compression changes decisions or omissions

Remove the mechanism if it adds burden without improving the intended outcome. For this step, inspect conversation history, watch duplicate chunk rate, and explicitly test summaries with no provenance. Record the assumption, owner, expected result, and safe terminal state so the team can distinguish a genuine improvement from work shifted elsewhere in the system.

Failure modes and anti-patterns

These mistakes are attractive because they appear to simplify the happy path. They move complexity into incidents, duplicate work, wrong outcomes, or unexplained bills:

• pasting everything remembered
• retrieving as much text as possible
• dropping critical constraints to save tokens
• summaries with no provenance
• duplicating the same document in history and RAG
• assuming a larger context is automatically better

Test each failure with the same seriousness as the successful path. Capture whether the system confirmed success, confirmed failure, returned a partial result, entered a degraded mode, or ended with an unknown external outcome. An unknown result must never be silently rewritten as success or treated as definitely safe to repeat.

Commonly confused concepts

Concept	Meaning	Compared with	Meaning
Context window	maximum inference capacity	Context	information actually supplied
Memory	stored information across time	Context	selected information used now
State	durable workflow facts	Conversation	language exchanged with a user or agent
Token count	quantity of tokens	Token cost	price associated with processing them
Compression	smaller representation	Information loss	meaning needed for the task is removed

The distinctions matter because each mechanism promises something different. A queue does not create capacity; a timeout does not prove an operation failed; a larger context does not guarantee relevance; and an alternate path does not become safe merely because the primary path is unavailable.

What to measure

• tokens by context component
• task success after compaction
• retrieval relevance
• duplicate chunk rate
• latency and cost per success
• fact omission and contradiction rate

Segment these signals by task type, deployment version, route, dependency, and tenant where appropriate. Averages alone can hide slow or failing task classes. Prefer cost and latency attached to successful, quality-checked outcomes. Use traces for causal investigation, metrics for trends, logs for discrete evidence, and production evaluations for behavioral quality. Do not invent a universal threshold: derive action levels from the service objective, baseline, consequence, and time available to respond.

When the complexity is unnecessary

A local, read-only experiment with public data and no durable workflow may not need the full machinery described here. For Context and Token Cost Optimization, begin with the simplest observable path that can measure tokens by context component. Add complexity when system instructions and examples, conversation history, or the consequences of pasting everything remembered create a concrete need. Simplicity is valuable only while it preserves the required outcome and makes failure visible.

Builder checklist

• Define the task, success evidence, and terminal failure states.
• Map system instructions and examples, conversation history, and structured agent state onto owned components.
• Rehearse pasting everything remembered and retrieving as much text as possible before increasing exposure.
• Bound calls, tokens, retries, elapsed time, queueing, and external side effects.
• Verify important outcomes before reporting completion.
• Compare changes using production-representative evaluations.
• Monitor tokens by context component beside task success after compaction.
• Maintain a clear fallback, escalation, or safe-stop path.

My Take

The strongest approach to Context and Token Cost Optimization is to make its boundary measurable before making it clever. My default is to expose system instructions and examples, attach it to tokens by context component, and preserve a clear response to pasting everything remembered. That creates a system a team can improve deliberately instead of one that looks efficient only on its happy path.

Continue learning

• [Context](/glossary/context/)
• [Context Window](/glossary/context-window/)
• [Token](/glossary/token/)
• [Memory In Ai Agents](/memory-in-ai-agents/)
• [Reranking In Rag](/reranking-in-rag/)
• [Ai Agent Cost Optimization](/ai-agent-cost-optimization/)