Jailbreak - CraftedSignal Threat Feed

M365 Copilot Impersonation Jailbreak Attempt Detection

hello@craftedsignal.io — Sat, 27 Jan 2024 14:30:00 +0000

This detection identifies attempts to jailbreak Microsoft 365 Copilot by manipulating the AI's persona. Users attempt to circumvent safety controls and acceptable use policies by prompting Copilot to adopt alternate identities or unrestricted behaviors. The activity is detected by analyzing exported eDiscovery prompt logs for specific keywords and phrases indicative of roleplay or impersonation requests. This includes prompts instructing the AI to "pretend you are," "act as," or adopt amoral stances. The detection categorizes these prompts into types such as "AI_Impersonation," "Malicious_AI_Persona," and "Unrestricted_AI_Persona" to pinpoint efforts to override the AI's guardrails through persona injection attacks. This matters because successful jailbreaks can lead to Copilot generating harmful or inappropriate content, revealing sensitive information, or performing unauthorized actions, potentially impacting organizational security and compliance. The detection utilizes prompt logs exported from M365 eDiscovery.

Attack Chain

User crafts a prompt containing roleplay keywords (e.g., "pretend you are," "act as," "amoral").
The prompt is submitted to M365 Copilot through a supported application (e.g., Teams, Word).
Copilot processes the prompt and generates a response based on the instruction.
The prompt and response are logged within the M365 environment.
An administrator or security analyst exports eDiscovery prompt logs from the Microsoft Purview compliance portal.
The exported logs, containing fields like Subject_Title (prompt text) and Sender, are ingested into a SIEM or security analytics platform.
The detection rule identifies prompts matching the defined impersonation patterns and categorizes the impersonation type.
If successful, the attacker gains the ability to influence Copilot's behavior beyond intended boundaries, potentially extracting sensitive data or generating malicious content.

Impact

A successful M365 Copilot jailbreak can lead to several negative consequences. The AI might generate responses that violate compliance regulations, reveal confidential company information, or be used to craft phishing emails or social engineering attacks. While specific victim counts are unavailable, organizations that heavily rely on M365 Copilot for sensitive tasks are at heightened risk. If the attack succeeds, the AI's output is no longer trustworthy, potentially damaging brand reputation, incurring legal penalties, and compromising internal security.

Recommendation

Configure Microsoft Purview eDiscovery to export M365 Copilot prompt logs, capturing user interactions and prompts (as described in how_to_implement).
Deploy the Sigma rule M365 Copilot Impersonation Jailbreak Attack to your SIEM and tune the m365_copilot_impersonation_jailbreak_attack_filter macro to reduce false positives based on your environment.
Investigate alerts generated by the Sigma rule, focusing on users with high-risk scores as identified by the RBA message User $user$ attempted M365 Copilot impersonation jailbreak with impersonation type $impersonation_type$.

Ollama API Prompt Injection and Jailbreak Attempts

hello@craftedsignal.io — Wed, 03 Jan 2024 12:00:00 +0000

This threat brief focuses on detecting prompt injection and jailbreak attempts targeting Ollama, an open-source framework for running large language models (LLMs) locally. Attackers are increasingly targeting LLMs with crafted prompts designed to bypass safety controls, extract sensitive information, or manipulate model behavior. This is achieved by injecting malicious instructions into user queries, leading to unintended or harmful outputs. The detection identifies suspicious activity against Ollama API endpoints (/api/generate and /v1/chat/completions) by monitoring response times. Requests exceeding 30 seconds, coupled with high request frequency, may indicate a sophisticated jailbreak attempt, multi-stage prompt injection, or extraction of sensitive data from the model. Defenders should be aware of this emerging threat to maintain the integrity and security of their LLM deployments. The provided Sigma rules and recommendations enable proactive monitoring and alerting for these types of attacks.

Attack Chain

Attacker identifies an Ollama instance exposed via API endpoints such as /api/generate or /v1/chat/completions.
The attacker crafts a malicious prompt designed to bypass safety filters or extract sensitive information (prompt injection).
The attacker sends the crafted prompt to the Ollama API endpoint via an HTTP POST request.
The Ollama server processes the complex prompt, leading to extended processing times.
The server logs the request details, including the URI path, source IP, HTTP method, response code, and response time.
A successful jailbreak or prompt injection may allow the attacker to extract internal data, manipulate model behavior, or bypass security controls.
The attacker repeats the process, refining the prompt based on previous responses.
The attacker potentially leverages the compromised Ollama instance for further malicious activities.

Impact

Successful prompt injection attacks against Ollama instances can lead to several critical impacts. Attackers may be able to extract sensitive data, manipulate the model to generate harmful or biased content, or bypass security controls designed to prevent misuse. While the specific number of victims is unknown, the increasing adoption of LLMs makes this a significant concern for organizations across various sectors. If successful, these attacks can compromise data confidentiality, integrity, and availability, leading to reputational damage and financial losses.

Recommendation

Deploy the Sigma rule Ollama Suspicious Long Request to detect abnormally long response times indicative of prompt injection attempts. Enable Ollama server logging to capture the required data (response times, URI paths, source IPs) and configure Splunk TA-ollama to ingest the logs (sourcetype: ollama:server).
Investigate alerts triggered by the Ollama Suspicious Long Request rule, focusing on the source IP address (src) and the requested URI (uri_path) to understand the nature of the interaction with the Ollama API.
Implement rate limiting and input validation on Ollama API endpoints to mitigate the risk of prompt injection attacks.
Regularly review and update Ollama's safety filters and security configurations to address emerging prompt injection techniques.
Monitor status_code values in the logs for unusual HTTP response codes that might indicate errors or vulnerabilities being exploited during prompt processing.

Microsoft 365 Copilot Jailbreak Attempts via Prompt Injection

hello@craftedsignal.io — Wed, 03 Jan 2024 10:00:00 +0000

This threat brief addresses the risk of prompt injection attacks targeting Microsoft 365 Copilot. Attackers attempt to bypass safety controls and manipulate the AI's behavior through crafted prompts. The attacks leverage techniques such as rule manipulation, system bypass commands, and AI impersonation requests. Defenders need to monitor for these jailbreak attempts because successful prompt injections could lead to data exfiltration, policy violations, or other unauthorized activities within the M365 environment. The detection focuses on analyzing exported eDiscovery prompt logs, specifically the Subject_Title field, to identify prompts containing jailbreak keywords.

Attack Chain

Attacker crafts a malicious prompt containing jailbreak keywords (e.g., "pretend you are", "act as", "rules=", "ignore", "bypass", "override").
The attacker submits the crafted prompt to Microsoft 365 Copilot.
The prompt is processed by Copilot, and an entry is logged in the M365 eDiscovery prompt logs, including the prompt text in the Subject_Title field.
An analyst exports the M365 eDiscovery prompt logs from the Microsoft Purview compliance portal.
The exported logs are ingested into a security information and event management (SIEM) system for analysis.
The SIEM searches the Subject_Title field for jailbreak keywords.
A risk score is assigned to the prompt based on the type of manipulation attempted.
If the jailbreak score meets or exceeds a predefined threshold, an alert is triggered.

Impact

Successful prompt injection attacks on Microsoft 365 Copilot can lead to various negative consequences, including data exfiltration, unauthorized access to sensitive information, violation of acceptable use policies, and the potential for the AI to generate harmful or misleading content. The number of victims depends on the scope of the compromised Copilot instance and the attacker's objectives. The sectors targeted would be any utilizing M365 Copilot.

Recommendation

Configure Microsoft Purview eDiscovery to export M365 Copilot prompt logs to a SIEM for analysis as described in the "how_to_implement" section of the source detection.
Deploy the provided Sigma rules to your SIEM to detect jailbreak attempts based on prompt injection techniques, tuning the threshold based on your environment.
Investigate triggered alerts, focusing on prompts with high jailbreak scores, to assess the potential impact and implement appropriate remediation measures.
Review and update M365 Copilot acceptable use policies to explicitly address the risks associated with prompt injection attacks.

Detection of M365 Copilot Jailbreak Attempts via Prompt Injection

hello@craftedsignal.io — Tue, 02 Jan 2024 12:00:00 +0000

This brief focuses on the detection of jailbreak attempts targeting Microsoft 365 Copilot. Attackers use prompt injection techniques to circumvent built-in safety controls. These attempts involve manipulating the AI's behavior through direct instruction, such as rule manipulation, system bypass commands, and unauthorized persona adoption. The attacks are identified by analyzing eDiscovery prompt logs for specific keywords associated with jailbreaking, including "pretend you are," "act as," "rules=," "ignore," "bypass," and "override." Successful jailbreak attempts can enable unauthorized access, data exfiltration, and policy violations, potentially leading to significant security breaches. This activity is detected by Splunk ES using the "M365 Copilot Jailbreak Attempts" detection rule.

Attack Chain

Attacker crafts a malicious prompt containing jailbreaking keywords (e.g., "ignore all previous rules").
The crafted prompt is submitted to M365 Copilot through a supported application.
M365 Copilot processes the prompt and attempts to fulfill the request, potentially bypassing safety controls.
The eDiscovery logs capture the prompt text in the Subject_Title field, along with user and timestamp information.
A security monitoring system, such as Splunk, ingests the eDiscovery logs.
The detection rule identifies prompts containing jailbreak keywords and assigns a severity score.
If the jailbreak score meets or exceeds a predefined threshold, an alert is triggered.
The attacker gains unauthorized control over Copilot's behavior, potentially leading to data exfiltration or other malicious activities.

Impact

A successful jailbreak of M365 Copilot can lead to several negative outcomes. It allows attackers to bypass the intended safety measures and manipulate Copilot for unauthorized purposes. This can result in the exposure of sensitive information, violation of data governance policies, and potentially the execution of malicious commands within the M365 environment. The exact scope of impact depends on the level of access granted to Copilot and the attacker's objectives.

Recommendation

Enable and configure M365 eDiscovery to capture Copilot prompt logs, ensuring the Subject_Title, Sender, timestamps, and workload metadata are included.
Deploy the provided Sigma rule M365 Copilot Jailbreak Attempts to your SIEM and tune the jailbreak_score threshold based on your environment.
Investigate any alerts generated by the rule, focusing on the user, Subject_Title, and jailbreak_score fields to understand the nature of the attempted jailbreak.
Implement user awareness training to educate employees about the risks of prompt injection and social engineering attacks targeting AI assistants.