Attack Chain

1. Initial Access: An attacker attempts to interact with an AI agent through a chat interface or API endpoint.
2. Prompt Injection: The attacker crafts a malicious prompt designed to manipulate the agent’s behavior or extract sensitive information. This leverages the agent’s reliance on LLMs to carry out commands.
3. Bypass Guardrails (Attempted): The prompt is sent to the AI agent, which then passes it through NVIDIA NeMo Guardrails managed by Falcon AIDR.
4. Detection and Redaction: Falcon AIDR detects the prompt injection attempt using its built-in classification rules and custom policies. Sensitive data like PII or internal repository references are redacted.
5. Content Defanging: Malicious content, such as adversarial domains embedded in the prompt, is identified and defanged to prevent the agent from accessing or executing compromised workflows.
6. Policy Enforcement: The agent’s response is moderated to ensure it stays within compliance boundaries, preventing the disclosure of unauthorized information or the execution of unauthorized actions.
7. Action Blocking: The agent is blocked from executing any action triggered by the malicious prompt, preventing unauthorized transactions or access to sensitive data.
8. Safe Response Generation: The agent generates a safe and compliant response based on the filtered and sanitized input, maintaining a natural conversation flow without compromising security.

Impact

Compromised AI agents can lead to significant data breaches, unauthorized transactions, and compliance violations, affecting potentially thousands of interactions. The integration of Falcon AIDR and NVIDIA NeMo Guardrails aims to prevent financial losses, reputational damage, and legal repercussions associated with these breaches. The number of affected organizations is expected to rise as AI agents become more integrated into sensitive business processes across various sectors, including financial services, healthcare, customer service, and software development. Success in these attacks could lead to exposure of sensitive patient data, financial records, or intellectual property.

Recommendation

• Deploy the provided Sigma rule to detect prompt injection attempts targeting AI agents by monitoring for specific keywords and patterns in user inputs (Sigma rule: “Detect Prompt Injection Attempts”).
• Enable Falcon AIDR with NVIDIA NeMo Guardrails v0.20.0 to leverage its built-in classification rules and custom policies for real-time detection and prevention of AI agent attacks.
• Configure custom data classification rules within Falcon AIDR to identify and redact sensitive information specific to your organization, such as account numbers, SSNs, or PHI.
• Monitor network traffic for attempts to access adversarial domains or other malicious content blocked by Falcon AIDR’s content defanging capabilities.
• Review and update Falcon AIDR policies regularly to ensure they align with evolving threat landscapes and compliance requirements.

Attack Chain

1. Initial Access/Prompt Injection: An attacker crafts a malicious prompt to inject into the AI agent’s input, aiming to manipulate its behavior (T1566.001).
2. Bypass Input Sanitization: The malicious prompt attempts to bypass initial input sanitization mechanisms, exploiting vulnerabilities in the agent’s prompt parsing logic.
3. Agent Logic Manipulation: Successful prompt injection allows the attacker to manipulate the AI agent’s decision-making process, redirecting it towards unauthorized actions.
4. Data Exfiltration: The compromised AI agent is coerced into exfiltrating sensitive data, such as customer PII or internal business information, through its normal operational channels.
5. Unauthorized Transactions: The manipulated agent initiates unauthorized transactions, such as fund transfers or policy changes, leveraging its access to backend systems.
6. Compliance Violation: The agent performs actions that violate compliance regulations, such as disclosing protected health information (PHI) without proper authorization.
7. Workflow Compromise: The attacker uses the compromised agent to execute malicious workflows that damage business operations.
8. Impact: The successful exploitation leads to data breaches, financial losses, reputational damage, and legal repercussions for the organization.

Impact

A successful compromise of AI agents could lead to significant damage across various sectors. In financial services, attackers could manipulate transaction logic and exfiltrate sensitive account data. Healthcare organizations face the risk of exposing protected health information (PHI) and compromising medical advice accuracy. Customer service operations could suffer data leaks and policy manipulation, while software development teams could have hardcoded secrets exposed and code injected into their repositories. The number of potential victims depends on the scope and scale of the AI agent deployments, with the potential to affect thousands of customers or internal systems.

Recommendation

• Deploy Falcon AIDR with NVIDIA NeMo Guardrails (v0.20.0) to protect AI agents against runtime attacks.
• Utilize the built-in classification rules and custom data classification capabilities in Falcon AIDR to define specific security policies.
• Implement the provided Sigma rule to detect prompt injection attempts targeting AI agents through user inputs.
• Use the provided Sigma rule to detect data exfiltration attempts by AI agents.
• Monitor AI agent activity logs to identify suspicious behavior, particularly around data access and transaction initiation.

Attack Chain

1. Initial Access: An attacker crafts a malicious prompt designed to exploit vulnerabilities in the AI agent’s input processing.
2. Prompt Injection: The attacker injects the malicious prompt into the AI agent’s input stream, bypassing initial input validation checks.
3. Agent Manipulation: The injected prompt manipulates the agent’s behavior, causing it to deviate from its intended functionality.
4. Data Access: The compromised agent, under the attacker’s control, accesses sensitive data, such as customer PII, financial records, or internal code repositories.
5. Unauthorized Actions: The agent executes unauthorized actions, such as initiating fraudulent transactions, modifying system configurations, or disclosing confidential information.
6. Lateral Movement: The attacker uses the compromised agent to access other systems or data sources within the organization.
7. Data Exfiltration: The attacker extracts sensitive data from the compromised systems and exfiltrates it to an external location.
8. Impact: The organization suffers financial losses, reputational damage, and legal repercussions due to the data breach and unauthorized actions.

Impact

A successful attack on an AI agent can lead to significant consequences. This includes exposure of customer data, unauthorized transactions, and violations of compliance requirements. The number of potential victims scales with the agent’s deployment size. Organizations in financial services, healthcare, customer service, and software development are particularly vulnerable. The damage can range from financial losses and reputational damage to legal repercussions and loss of customer trust. The risk grows as more organizations adopt AI and the number of vulnerable AI agents increases.

Recommendation

• Deploy CrowdStrike Falcon AIDR with NVIDIA NeMo Guardrails (v0.20.0) to protect AI agents from runtime attacks and reduce the agentic blast radius.
• Create named detection policies tailored to specific security requirements using the Falcon AIDR API.
• Enable detectors to detect, block, redact, encrypt, or transform content at critical points in AI agent workflows as mentioned in the overview.
• Implement the Sigma rule “Detect Suspicious Prompt Injection Attempts” to identify and block malicious prompts attempting to manipulate AI agent behavior.
• Monitor AI agent activity logs for suspicious patterns and anomalies, leveraging the insights from CrowdStrike Falcon AIDR.
• Deploy the Sigma rule “Detect Sensitive Data Exposure by AI Agents” to identify and prevent the exfiltration of sensitive information by compromised agents.

Attack Chain

1. Initial Access (via AI Agent): An attacker gains initial access by compromising an AI agent running on an endpoint, potentially through prompt injection or other vulnerabilities in the agent’s design.
2. Privilege Escalation: The attacker leverages the compromised AI agent’s existing system permissions, which may be elevated, to gain further access to the system. AI agents often have high privileges to execute terminal commands, browse the web, and interact with files.
3. Living off the AI Land (LOTAIL): The attacker uses the compromised AI agent to perform malicious actions that appear as legitimate user behavior, such as executing terminal commands, browsing websites, or interacting with files.
4. Lateral Movement: The attacker utilizes the AI agent’s network connectivity to discover and access other systems within the network, including LLM runtimes, MCP servers, and IDE extensions.
5. Data Exfiltration: The attacker uses the AI agent to exfiltrate sensitive data from the compromised systems, such as source code, credentials, or other confidential information.
6. Supply Chain Compromise: The attacker uses access to development environments via compromised AI tools to introduce malicious code into the software supply chain.
7. Policy Violation: The attacker manipulates the AI agent to violate content policies or access control rules, potentially leading to unauthorized access to sensitive data or systems.

Impact

Successful attacks targeting AI agents and shadow AI can lead to significant data breaches, intellectual property theft, and supply chain compromises. The lack of visibility and governance over AI deployments creates a growing attack surface that traditional security controls are ill-equipped to handle. Compromised AI agents can be used to perform a wide range of malicious activities, including data exfiltration, lateral movement, and the introduction of malicious code into the software supply chain. The impact can range from financial losses and reputational damage to the compromise of critical infrastructure and sensitive government systems.

Recommendation

• Deploy the Sigma rule “AI Desktop Application Usage Detected” to identify and monitor the use of AI desktop applications such as ChatGPT, Gemini, and others within your environment. This rule uses process_creation logs to detect the execution of these applications (see rule below).
• Enable and configure AI Discovery in CrowdStrike Falcon Exposure Management to gain visibility into AI-related components running across endpoints, including AI apps, LLM runtimes, MCP servers, and IDE extensions. This leverages Falcon for IT telemetry as described in the overview.
• Implement Falcon AIDR policies to monitor and protect agents built in Microsoft Copilot Studio against prompt injection attacks, data leaks, and policy violations.
• Review and update access control policies for AI agents to minimize the potential impact of a compromise, focusing on the principle of least privilege.

Attack Chain

This brief focuses on the capabilities of Charlotte AI AgentWorks as a defensive tool. Therefore, the attack chain describes hypothetical scenarios where such a tool could be deployed to counter an attack.

1. Initial Access: An attacker gains initial access via a phishing email containing a malicious attachment (e.g., a weaponized document).
2. Execution: The user opens the malicious attachment, which executes a PowerShell script designed to download a second-stage payload.
3. Persistence: The PowerShell script creates a scheduled task to ensure the payload executes regularly, even after a system reboot.
4. Defense Evasion: The attacker attempts to disable or bypass security controls (e.g., disabling Windows Defender) to avoid detection.
5. Command and Control: The downloaded payload establishes a connection to a command-and-control (C2) server, allowing the attacker to issue commands and exfiltrate data.
6. Lateral Movement: The attacker uses compromised credentials or exploits vulnerabilities to move laterally within the network, targeting critical systems and data.
7. Data Exfiltration: The attacker exfiltrates sensitive data from the compromised systems to an external server under their control.
8. Impact: The attacker encrypts critical data, demanding a ransom for its decryption.

Impact

If an attack succeeds, organizations may experience significant data breaches, financial losses, and reputational damage. The rise of AI-powered adversaries is accelerating the speed of attacks, with breakout times collapsing to as fast as 27 seconds. Successful attacks may lead to ransomware deployment, intellectual property theft, and disruption of critical services. Organizations are looking to AI-driven security solutions, such as Charlotte AI AgentWorks, to enhance their defenses and mitigate these risks.

Recommendation

• Deploy and configure CrowdStrike Falcon to collect relevant telemetry data for the rules below, enabling detection of suspicious activities indicative of attack chains.
• Deploy the provided Sigma rules to detect potentially malicious PowerShell execution and scheduled task creation.
• Utilize Charlotte AI AgentWorks’s pre-built agents for malware analysis and triage to accelerate incident response.
• Experiment with Charlotte AI using the free AI credits to convert natural language into governed automation, improving security workflows.

Attack Chain

1. Initial Access: An attacker crafts a malicious prompt designed to bypass initial input sanitization.
2. Prompt Injection: The malicious prompt injects unauthorized commands into the AI agent’s workflow.
3. Data Exfiltration: The injected commands instruct the AI agent to access and extract sensitive data, such as customer PII or financial records.
4. Privilege Escalation: The attacker leverages the compromised AI agent to access internal tools or systems beyond the agent’s intended scope.
5. Unauthorized Transactions: The AI agent, under the attacker’s control, executes unauthorized financial transactions or modifies critical business processes.
6. Lateral Movement: The attacker utilizes the compromised AI agent to gain access to other AI agents or systems within the organization.
7. Compliance Violation: The attacker manipulates the AI agent to violate regulatory compliance policies, leading to potential legal and financial repercussions.
8. Impact: Sensitive data is exposed, unauthorized actions are executed, and the organization faces potential legal and financial damage due to compliance violations.

Impact

A successful attack on AI agents can lead to significant damage. Exposed customer data, unauthorized transactions, and compliance violations can result in financial losses and reputational damage. The number of victims and the sectors targeted depend on the scope of the AI agent’s access and the nature of the compromised data. The integration of Falcon AIDR with NVIDIA NeMo Guardrails aims to mitigate these risks and protect organizations from the potential consequences of compromised AI agents.

Recommendation

• Enable Falcon AIDR with NVIDIA NeMo Guardrails (v0.20.0) to protect AI agents from prompt injection and other runtime attacks (refer to the Overview).
• Implement custom data classification rules within Falcon AIDR to identify and redact sensitive information (refer to the Overview).
• Utilize the Falcon AIDR API to create named detection policies tailored to specific security requirements (refer to the Configuring Falcon AIDR Policies section).
• Deploy the Sigma rule to detect suspicious AI agent command line activity.

Attack Chain

1. An attacker gains initial access to an endpoint, potentially a developer machine, through social engineering or exploiting a software vulnerability.
2. The attacker leverages a compromised AI agent, such as OpenClaw, or an AI-powered application installed on the endpoint.
3. The compromised AI agent executes commands on the endpoint, leveraging the agent’s high system permissions, to enumerate sensitive files and network resources.
4. The attacker performs an indirect prompt injection attack against an AI application, modifying the application’s behavior to leak sensitive data.
5. The compromised agent initiates a connection to a command-and-control (C2) server to exfiltrate stolen data.
6. The attacker exploits a misconfigured Model Context Protocol (MCP) server within the development environment to access sensitive AI models and training data.
7. The attacker leverages a Copilot Studio agent with insufficient security guardrails to access and exfiltrate sensitive data from a SaaS application.
8. The attacker successfully exfiltrates sensitive data and potentially gains persistent access to the environment, impacting data confidentiality and integrity.

Impact

A successful attack targeting AI agents and shadow AI can lead to significant data breaches, intellectual property theft, and reputational damage. Organizations may experience compliance violations due to the leakage of sensitive data. The lack of visibility and governance over AI deployments can result in widespread vulnerabilities and increased attack surfaces, potentially affecting thousands of endpoints and cloud environments. The compromise of AI models and training data can lead to the manipulation of AI systems, causing them to make incorrect decisions or provide malicious outputs.

Recommendation

• Deploy the Sigma rule Detect AI Application Usage to identify the use of desktop AI applications like ChatGPT, Gemini, and Copilot on endpoints to gain visibility into shadow AI (logsource: process_creation).
• Utilize Falcon Exposure Management’s AI Discovery capabilities to identify AI-related components running on endpoints, including LLMs, MCP servers, and IDE extensions, to manage AI-related risks.
• Monitor network connections from processes associated with AI tools for suspicious outbound traffic to detect potential data exfiltration attempts (logsource: network_connection).

Attack Chain

1. Initial Access (Prompt Injection): An attacker crafts a malicious prompt designed to inject commands or bypass intended agent behavior via a user input field or API call.
2. Bypass Guardrails: The prompt injection attempt exploits vulnerabilities in the AI agent’s input validation or content filtering mechanisms to circumvent existing security measures.
3. Unauthorized Data Access: The injected commands enable the attacker to access sensitive data, such as customer PII, financial records, or internal system configurations, that the agent has access to.
4. Privilege Escalation: The attacker leverages the compromised agent’s privileges to escalate access to other systems or resources within the organization’s network.
5. Lateral Movement: Using the compromised agent as a foothold, the attacker moves laterally to other systems, potentially targeting critical infrastructure or high-value assets.
6. Data Exfiltration: The attacker exfiltrates sensitive data to an external location, potentially causing significant financial and reputational damage.
7. Malicious Code Execution: The attacker injects and executes malicious code through the agent, allowing for further compromise of the environment.

Impact

Compromised AI agents can lead to significant financial and reputational damage. Unauthorized access to sensitive data, such as customer PII or financial records, can result in regulatory fines and loss of customer trust. In financial services, compromised agents could manipulate transaction logic, leading to unauthorized transactions. In healthcare, compromised agents could provide inaccurate medical advice. The impact can range from data breaches and financial losses to compromised business processes and compliance violations.

Recommendation

• Deploy the provided Sigma rules to your SIEM to detect prompt injection attempts and unauthorized actions (see the “rules” section).
• Enable and configure CrowdStrike Falcon AIDR with NVIDIA NeMo Guardrails v0.20.0 to leverage its built-in classification rules and custom data classification capabilities.
• Implement strict input validation and content filtering mechanisms to prevent prompt injection attacks.
• Regularly monitor AI agent activity for suspicious behavior, such as unauthorized data access or privilege escalation.
• Use Falcon AIDR’s monitoring mode to understand your threat landscape and progressively enforce blocks and redactions as agents move from development to production.
• Configure Falcon AIDR policies tailored to your specific security requirements using the Falcon AIDR API, applying policies at critical points in AI agent and application workflows.