Attack Chain

1. Attacker Crafts Malicious Prompt: Adversary designs a prompt containing hidden or fragmented malicious instructions, using techniques like "Trigger-Activated Rule Addition," "Cognitive Token Suppression," "Algorithmic Payload Decomposition," or "Special Token Injection."
2. Delivery via Unwitting User: The malicious prompt is delivered to an AI agent, often through social engineering, where an authorized user is enticed to input the prompt into the AI system without realizing its true intent, such as copying from a compromised website or social media post.
3. AI Agent Processes Input: The AI agent, designed to follow user instructions, processes the maliciously crafted prompt, which includes the hidden or fragmented commands.
4. Injection Technique Activation: The embedded prompt injection technique successfully manipulates the AI's internal logic, causing it to misinterpret or prioritize the attacker's directives over its safety guidelines or original instructions. For example, a hidden rule is activated, or safety-related tokens are suppressed.
5. Unauthorized Tool Call/Action: The compromised AI agent initiates an unauthorized action based on the injected instructions, such as making tool calls to execute_sql_query, generating unexpected responses, or attempting to write shell commands.
6. Data Exfiltration or Impact: The AI agent, under the attacker's control, exfiltrates sensitive data (e.g., forwarding emails to anon@evilcorp.corp), modifies system configurations, or performs other actions impacting confidentiality, integrity, or availability.

Impact

The described prompt injection techniques enable adversaries to achieve significant impact on organizations leveraging AI agents. If successful, these attacks can lead to unauthorized data exfiltration, as demonstrated by the example of emails being duplicated and forwarded to attacker-controlled addresses. AI agents could be coerced into executing arbitrary shell commands, granting attackers remote code execution capabilities on underlying infrastructure. Furthermore, the manipulation of AI agents could result in the bypass of security controls, altered system behavior, and the generation of misleading or harmful content, leading to reputational damage, financial loss, and compromise of critical systems. These attacks target any organization integrating AI agents with access to internal systems or sensitive data.

Recommendation

• Prioritize comprehensive AI threat modeling that accounts for all potential sources of model context, including prompts, files, RAG pipelines, and external APIs.
• Enhance AI red teaming exercises to include advanced prompt injection techniques such as boundary mimicry, indirect injection, and delayed activation, beyond simple jailbreaking attempts.
• Configure AI agent logging to capture tool calls and system interactions, and deploy the provided Sigma rule to detect suspicious outbound network connections to domains like evilcorp.corp.
• Educate users about the risks of "Unwitting User Delivery" and social engineering tactics that entice them to input malicious prompts into AI systems.

Attack Chain

This brief describes new techniques for prompt injection and does not detail a specific end-to-end attack campaign from initial access to impact. The following outlines how these techniques are leveraged once an attacker introduces a malicious prompt or data to an AI system.

1. Initial Access (via Social Engineering): An adversary employs social engineering or deceptive tactics to trick an authorized user into submitting a crafted prompt, effectively turning the user into an "Unwitting User Delivery" (IM0005) vector. This can involve copying/pasting hidden commands or using compromised browser extensions.
2. Prompt Manipulation (Algorithmic Payload Decomposition): The attacker fragments a malicious instruction into multiple benign-looking steps, variables, or characters. This "Algorithmic Payload Decomposition" (PT0200) technique evades immediate detection by filters.
3. Bypass Defenses (Special Token Injection): The fragmented payload or a new prompt includes "Special Token Injection" (PT0198), mimicking internal structural cues (e.g., <tool_call>) used by the AI system to differentiate system commands from user input, causing boundary confusion and elevating untrusted content.
4. Evasion (Cognitive Token Suppression): The malicious prompt attempts "Cognitive Token Suppression" (PT0197) by instructing the AI model to avoid using specific safety-related terms or refusal vocabulary, hindering its ability to generate secure responses or block the attack.
5. Delayed Execution (Trigger-Activated Rule Addition): The attacker embeds a "Trigger-Activated Rule Addition" (PT0201) instruction into the AI's context. This instruction remains dormant until a specific trigger phrase, event, or condition occurs, at which point the AI agent begins to follow the new, malicious rule.
6. Action on Objectives (Execution): The compromised AI agent, following the injected rules, executes unintended or malicious commands, such as an execute_sql_query to access sensitive data, or shell commands to interact with underlying systems.
7. Impact (Data Exfiltration): The AI agent, now under adversary control, proceeds to exfiltrate data, for example, by duplicating and forwarding sensitive emails to an attacker-controlled address like anon@evilcorp.corp, or by accessing and leaking internal files.

Impact

The observed impact of successful prompt injection ranges from subtle manipulation of AI behavior to significant security breaches. Adversaries can hijack agent capabilities, leading to unintended actions like executing arbitrary shell commands, performing SQL injection-like queries, or accessing and exfiltrating sensitive data. This can bypass existing security rules, steer agents into unsafe actions, and result in data loss or system compromise. While specific victim counts are not provided, the techniques target general AI systems and agents, affecting any organization deploying such technologies, especially those relying on AI for critical operations or data handling.

Recommendation

• Implement robust AI threat modeling that accounts for all possible origins of model context, including prompts, files, RAG pipelines, agent memory, APIs, tool outputs, browser content, emails, and SaaS data, to understand potential injection points.
• Conduct targeted AI red teaming exercises that go beyond simple "ignore previous instructions" and incorporate boundary mimicry, indirect injection, and delayed activation scenarios, as described by techniques like PT0198 and PT0201.
• Monitor for and block outbound network connections from AI systems or agents to suspicious domains, such as evilcorp.corp, which was identified as an exfiltration target in an example for the IM0005 technique.
• Educate users on the risks of "Unwitting User Delivery" (IM0005) by training them to recognize and avoid deceptive tactics that could lead to submitting malicious prompts.
• Deploy specialized AI security solutions capable of analyzing and detecting fragmented (PT0200), suppressed (PT0197), or specially tokenized (PT0198) instructions within prompts before they are processed by AI models.