Prompt Injection in AI Agents
Prompt Injection in AI Agents is about making the surrounding system explicit enough to control consequences.
TL;DR
- Prompt injection in tool-using agents belongs to the complete agent system, not only the model.
- Instructions guide behavior; deterministic controls enforce important boundaries.
- Unknown outcomes require verification before side effects are repeated.
- Controls should match impact, reversibility, exposure, and sensitivity.
Origin and authority are different properties.
What prompt injection in tool-using agents means
Prompt injection in tool-using agents is a property of the complete system: model, agent loop, workflow, tools, state, infrastructure, and policy. The model interprets goals and proposes actions. Application code validates inputs, binds identity, checks permissions, manages state, invokes tools, and evaluates results. External services finally determine what happened. Generated language is not an enforced fact; a statement that an action is safe, permitted, complete, or reversible remains a proposal until the responsible system checks it. Prompt injection in tool-using agents is a property of the complete system: model, agent loop, workflow, tools, state, infrastructure, and policy. The model interprets goals and proposes actions. Application code validates inputs, binds identity, checks permissions, manages state, invokes tools, and evaluates results. External services finally determine what happened. Generated language is not an enforced fact; a statement that an action is safe, permitted, complete, or reversible remains a proposal until the responsible system checks it.
Why it matters for agents
A text generator can be wrong without changing the world. An agent may combine private context, credentials, tools, memory, and multiple steps. An error at one step becomes input to the next, so authority and state accumulate. An invoice agent is asked to summarize documents. One invoice includes embedded text telling the reader to ignore the task and transmit account information elsewhere. The content enters model context and competes with trusted instructions. The model may propose an unsafe send action. The decisive controls are outside the document and model: destination restrictions, scoped permissions, data policy, backend authorization, and human approval for a novel external side effect. The useful question is what happens on the worst plausible interpretation, stale read, duplicate event, denied request, or ambiguous response. A text generator can be wrong without changing the world. An agent may combine private context, credentials, tools, memory, and multiple steps. An error at one step becomes input to the next, so authority and state accumulate. An invoice agent is asked to summarize documents. One invoice includes embedded text telling the reader to ignore the task and transmit account information elsewhere. The content enters model context and competes with trusted instructions. The model may propose an unsafe send action. The decisive controls are outside the document and model: destination restrictions, scoped permissions, data policy, backend authorization, and human approval for a novel external side effect. The useful question is what happens on the worst plausible interpretation, stale read, duplicate event, denied request, or ambiguous response.
A production control flow
Begin with an authenticated request and a narrow goal. Validate identifiers, types, destinations, and current resource state before a side effect. The agent may select tools, but policy checks whether the principal, operation, resource, and arguments are in scope. Create durable state before crossing an external boundary. Execute through a typed adapter that returns status, data, error class, retry guidance, and metadata. This is an example pattern, not a universal schema. Validate the result, verify external state, persist the outcome, and only then report completion. Begin with an authenticated request and a narrow goal. Validate identifiers, types, destinations, and current resource state before a side effect. The agent may select tools, but policy checks whether the principal, operation, resource, and arguments are in scope. Create durable state before crossing an external boundary. Execute through a typed adapter that returns status, data, error class, retry guidance, and metadata. This is an example pattern, not a universal schema. Validate the result, verify external state, persist the outcome, and only then report completion.
No single prompt or filter is a complete defense.
Separate responsibilities
The model interprets language and chooses among permitted options. The agent loop coordinates reasoning and tool selection. The workflow owns sequencing, checkpoints, branches, compensation, and terminal states. Infrastructure controls processes, networks, files, secrets, resource limits, and durability. Policy binds identity to operations and resources. Improve instructions when the model chooses the wrong allowed tool; fix enforcement when an unauthorized resource is accessed. Prompt tuning cannot repair an infrastructure guarantee. The model interprets language and chooses among permitted options. The agent loop coordinates reasoning and tool selection. The workflow owns sequencing, checkpoints, branches, compensation, and terminal states. Infrastructure controls processes, networks, files, secrets, resource limits, and durability. Policy binds identity to operations and resources. Improve instructions when the model chooses the wrong allowed tool; fix enforcement when an unauthorized resource is accessed. Prompt tuning cannot repair an infrastructure guarantee.
Classify failures before acting
Hard outcomes include a timeout after a provider committed the action, a partial response, a valid but stale result, or a crash after execution but before persistence. Classify outcomes as confirmed success, confirmed failure, transient failure, repairable input, denied action, business rejection, partial result, stale result, or unknown. Only some failures are retryable, and retryable does not always mean retry-safe. An unknown payment, refund, booking, message, or write requires lookup or reuse of the same operation identity before another attempt. Hard outcomes include a timeout after a provider committed the action, a partial response, a valid but stale result, or a crash after execution but before persistence. Classify outcomes as confirmed success, confirmed failure, transient failure, repairable input, denied action, business rejection, partial result, stale result, or unknown. Only some failures are retryable, and retryable does not always mean retry-safe. An unknown payment, refund, booking, message, or write requires lookup or reuse of the same operation identity before another attempt.
Layered implementation choices
Use structured schemas for types and required fields, then semantic validation for tenant, ownership, amount, destination, and current state. Scope credentials to the smallest capability and lifetime. Preserve provenance for user input, retrieved content, memory, and tool results. Insert approval before high-impact actions. Bound waits and loops with time, step, tool-call, and cost limits. Retry transient operations with a maximum count, backoff, and jitter. Store checkpoints around external actions. Capture traces, denials, redacted arguments, and verification evidence. Use structured schemas for types and required fields, then semantic validation for tenant, ownership, amount, destination, and current state. Scope credentials to the smallest capability and lifetime. Preserve provenance for user input, retrieved content, memory, and tool results. Insert approval before high-impact actions. Bound waits and loops with time, step, tool-call, and cost limits. Retry transient operations with a maximum count, backoff, and jitter. Store checkpoints around external actions. Capture traces, denials, redacted arguments, and verification evidence.
Trade-offs and proportionality
More controls add latency, infrastructure, maintenance, and review. A read-only assistant over public documents may need only input limits, output review, timeouts, and a step cap. A system that writes production records, handles secrets, spends money, executes code, or communicates externally warrants stronger authorization, isolation, idempotency, verification, and audit. Approval on every harmless action creates fatigue. Choose controls according to consequence and reversibility, then measure prevention and user friction. More controls add latency, infrastructure, maintenance, and review. A read-only assistant over public documents may need only input limits, output review, timeouts, and a step cap. A system that writes production records, handles secrets, spends money, executes code, or communicates externally warrants stronger authorization, isolation, idempotency, verification, and audit. Approval on every harmless action creates fatigue. Choose controls according to consequence and reversibility, then measure prevention and user friction.
Adjacent concepts
These pairs are related but not interchangeable: prompt injection vs jailbreak; direct vs indirect injection; bad input vs adversarial instruction; injection vs data poisoning; prompt guardrail vs authorization. A guardrail may flag content but does not grant backend access. A timeout bounds waiting but does not prove failure. A retry repeats an operation but is not recovery by itself. A sandbox limits impact but does not authorize an action. Human approval records judgment at a boundary; it does not replace evaluation. Naming boundaries correctly prevents one mechanism from being asked for a guarantee it cannot provide. These pairs are related but not interchangeable: prompt injection vs jailbreak; direct vs indirect injection; bad input vs adversarial instruction; injection vs data poisoning; prompt guardrail vs authorization. A guardrail may flag content but does not grant backend access. A timeout bounds waiting but does not prove failure. A retry repeats an operation but is not recovery by itself. A sandbox limits impact but does not authorize an action. Human approval records judgment at a boundary; it does not replace evaluation. Naming boundaries correctly prevents one mechanism from being asked for a guarantee it cannot provide.
Common mistakes
Common mistakes include relying on a system prompt as a security boundary, giving one credential broad authority, retrying every failure, discarding workflow state, trusting a success-looking message without verification, and letting a model certify its own proposal. Another is silently switching tools or destinations after failure. Test malformed arguments, stale reads, denied permission, response loss after commit, duplicate events, partial results, cancellation, unavailable dependencies, contaminated memory, and exhausted budgets. Common mistakes include relying on a system prompt as a security boundary, giving one credential broad authority, retrying every failure, discarding workflow state, trusting a success-looking message without verification, and letting a model certify its own proposal. Another is silently switching tools or destinations after failure. Test malformed arguments, stale reads, denied permission, response loss after commit, duplicate events, partial results, cancellation, unavailable dependencies, contaminated memory, and exhausted budgets.
When it may be unnecessary
The full pattern is unnecessary for a local, stateless, read-only experiment with public data and no external effects. Even there, validation and stopping limits prevent runaway execution. Add stronger mechanisms when the system crosses a data, financial, communication, permission, production, or availability boundary. The aim is not maximal ceremony. A reversible draft and a money transfer should not share the same approval, credential, retry, or verification policy. The full pattern is unnecessary for a local, stateless, read-only experiment with public data and no external effects. Even there, validation and stopping limits prevent runaway execution. Add stronger mechanisms when the system crosses a data, financial, communication, permission, production, or availability boundary. The aim is not maximal ceremony. A reversible draft and a money transfer should not share the same approval, credential, retry, or verification policy.
Builder checklist
Define success evidence and terminal failure states. Bind work to an authenticated identity. Limit capabilities, resources, scopes, destinations, and credential duration. Validate inputs and tool arguments outside the model. Persist checkpoints around side effects. Make retries bounded and idempotent where needed. Verify final state. Expose pause, escalation, cancellation, and unresolved outcomes. Record enough telemetry to reconstruct the run without leaking secrets. Assign an owner to each boundary; an unowned rule is only an aspiration. Define success evidence and terminal failure states. Bind work to an authenticated identity. Limit capabilities, resources, scopes, destinations, and credential duration. Validate inputs and tool arguments outside the model. Persist checkpoints around side effects. Make retries bounded and idempotent where needed. Verify final state. Expose pause, escalation, cancellation, and unresolved outcomes. Record enough telemetry to reconstruct the run without leaking secrets. Assign an owner to each boundary; an unowned rule is only an aspiration.
My Take
Let the model interpret and propose, then make ordinary software enforce authority, state transitions, limits, and evidence. The model remains valuable because it handles ambiguity. The system becomes dependable because ambiguity cannot redefine permission or truth.
Continue learning
- [Prompt Injection](/glossary/prompt-injection/)
- [Tool Permissions And Least Privilege](/tool-permissions-and-least-privilege/)
- [Sandboxing Ai Agents](/sandboxing-ai-agents/)
- [Human In The Loop For Ai Agents](/human-in-the-loop-for-ai-agents/)
- [Observability For Ai Agents](/observability-for-ai-agents/)
