Defending Agents Against Prompt Injection

Recognize the instruction conflict

Direct injection arrives in a user request, while indirect injection is embedded in material the agent reads, such as a webpage, email, document, tool result, image text, or retrieved note. The content may claim to be a system message, ask for hidden instructions, redirect the task, or request a tool call. Formatting and polite wording do not establish authority.

A model processes instructions and data through the same language channel, so delimiters and reminders can help clarify intent but cannot provide a hard security boundary. Plan for occasional instruction-following failures. Start threat modeling from possible outcomes: disclosure, unauthorized actions, corrupted decisions, cross-user access, persistence in memory, and communication with an attacker-controlled destination.

Reduce available authority

Give each workflow only the tools it needs for the current state. Scope credentials by user, tenant, resource, operation, and environment. Separate read from write and use narrow operations instead of a general command executor. Keep secrets out of prompts and tool results. A compromised model decision should encounter the same authorization and business-rule checks as any other untrusted request.

Use destination allowlists for navigation and communication. Restrict uploads, downloads, and network access. For consequential writes, create a protected proposal that shows exact recipients, payload, and effect to an authorized reviewer. Revalidate the proposal during execution. Prompt injection should not be able to turn approval for one action into consent for a changed target or additional operation.

Contain untrusted content

Mark the origin and trust level of every context item. Retrieve the smallest fields needed and avoid mixing one user's private data with another user's content. Sanitize active formats where appropriate, but do not assume removing obvious phrases removes the threat. Preserve source text for evidence while preventing it from selecting tools, changing policy, or setting persistent memory.

Separate stages when possible. One constrained component can extract facts into a schema, application code can validate them, and another component can reason over approved fields. This does not eliminate attacks, because extracted text can remain malicious, but it narrows exposure and creates checkpoints. Treat model-generated summaries of hostile material as untrusted until validated for their next use.

Detect and rehearse failures

Build adversarial cases from each real input channel. Include hidden or quoted instructions, requests for credentials, data-exfiltration links, fake approval claims, encoded text, poisoned memory candidates, and tool results that urge another action. Test whether protected data appears in model output as well as whether a prohibited call executes. Vary wording and placement across repeated runs.

Log denied tools, unexpected destinations, approval changes, unusual call sequences, and sensitive-output filters with enough context for investigation. Do not expose hidden prompts or secrets in security logs. Provide a kill switch and credential-revocation procedure. Review current OWASP and platform guidance quarterly because attack techniques and available controls continue to change.

Protect a document-review assistant

An assistant reads supplier proposals and prepares a comparison, and one uploaded file contains instructions to email confidential bids externally.

1. Classify every uploaded proposal as untrusted and extract requested commercial fields with source locations into a restricted schema.
2. Run parsing with no email tool, no unrelated repository access, and credentials limited to the review workspace.
3. Validate supplier identifiers and numeric fields, then compare only approved extracted records while retaining links to original evidence.
4. Keep external communication unavailable; any later internal report delivery uses an allowlisted destination and exact human-approved preview.
5. Add the malicious proposal to repeated security evaluations and alert on attempted destination changes or requests for hidden data.

Prompt injection threat worksheet

Use one worksheet per workflow to connect attack surfaces to enforceable controls.

• Inputs: users, pages, files, messages, images, retrieval results, memory, and tool responses.
• Assets: secrets, private context, credentials, writable records, communications, and trusted outputs.
• Attack outcomes: disclosure, action, redirect, persistence, impersonation, and decision corruption.
• Controls: isolation, field limits, permissions, allowlists, validation, approval, and execution binding.
• Evidence: adversarial cases, denied events, residual risk, incident owner, and shutdown procedure.

Common mistakes

• Relying on a stronger warning in the system prompt as the only defense against hostile documents or webpages.
• Giving a research agent communication and write tools that its evidence-gathering stage does not need.
• Checking only for suspicious phrases while ignoring encoded, indirect, multilingual, or action-focused attacks.

Try one

Sources

• OWASP Top 10 for Large Language Model ApplicationsOfficial OWASP project covering major security risks in applications that use language models.
• OpenAI safety best practicesOfficial recommendations for testing, constraints, and human oversight.