Attack Chain

This brief describes new techniques for prompt injection and does not detail a specific end-to-end attack campaign from initial access to impact. The following outlines how these techniques are leveraged once an attacker introduces a malicious prompt or data to an AI system.

1. Initial Access (via Social Engineering): An adversary employs social engineering or deceptive tactics to trick an authorized user into submitting a crafted prompt, effectively turning the user into an "Unwitting User Delivery" (IM0005) vector. This can involve copying/pasting hidden commands or using compromised browser extensions.
2. Prompt Manipulation (Algorithmic Payload Decomposition): The attacker fragments a malicious instruction into multiple benign-looking steps, variables, or characters. This "Algorithmic Payload Decomposition" (PT0200) technique evades immediate detection by filters.
3. Bypass Defenses (Special Token Injection): The fragmented payload or a new prompt includes "Special Token Injection" (PT0198), mimicking internal structural cues (e.g., <tool_call>) used by the AI system to differentiate system commands from user input, causing boundary confusion and elevating untrusted content.
4. Evasion (Cognitive Token Suppression): The malicious prompt attempts "Cognitive Token Suppression" (PT0197) by instructing the AI model to avoid using specific safety-related terms or refusal vocabulary, hindering its ability to generate secure responses or block the attack.
5. Delayed Execution (Trigger-Activated Rule Addition): The attacker embeds a "Trigger-Activated Rule Addition" (PT0201) instruction into the AI's context. This instruction remains dormant until a specific trigger phrase, event, or condition occurs, at which point the AI agent begins to follow the new, malicious rule.
6. Action on Objectives (Execution): The compromised AI agent, following the injected rules, executes unintended or malicious commands, such as an execute_sql_query to access sensitive data, or shell commands to interact with underlying systems.
7. Impact (Data Exfiltration): The AI agent, now under adversary control, proceeds to exfiltrate data, for example, by duplicating and forwarding sensitive emails to an attacker-controlled address like anon@evilcorp.corp, or by accessing and leaking internal files.

Impact

The observed impact of successful prompt injection ranges from subtle manipulation of AI behavior to significant security breaches. Adversaries can hijack agent capabilities, leading to unintended actions like executing arbitrary shell commands, performing SQL injection-like queries, or accessing and exfiltrating sensitive data. This can bypass existing security rules, steer agents into unsafe actions, and result in data loss or system compromise. While specific victim counts are not provided, the techniques target general AI systems and agents, affecting any organization deploying such technologies, especially those relying on AI for critical operations or data handling.

Recommendation

• Implement robust AI threat modeling that accounts for all possible origins of model context, including prompts, files, RAG pipelines, agent memory, APIs, tool outputs, browser content, emails, and SaaS data, to understand potential injection points.
• Conduct targeted AI red teaming exercises that go beyond simple "ignore previous instructions" and incorporate boundary mimicry, indirect injection, and delayed activation scenarios, as described by techniques like PT0198 and PT0201.
• Monitor for and block outbound network connections from AI systems or agents to suspicious domains, such as evilcorp.corp, which was identified as an exfiltration target in an example for the IM0005 technique.
• Educate users on the risks of "Unwitting User Delivery" (IM0005) by training them to recognize and avoid deceptive tactics that could lead to submitting malicious prompts.
• Deploy specialized AI security solutions capable of analyzing and detecting fragmented (PT0200), suppressed (PT0197), or specially tokenized (PT0198) instructions within prompts before they are processed by AI models.