Prompt Leak, Prompt Injection, and Jailbreak: What Are the Differences?
System prompt leakage is technically related to prompt injection and jailbreaking—which were already discussed in previous articles—but serves a different purpose. Whereas prompt injection hijacks a system’s behavior and jailbreaking bypasses its content safeguards, system prompt leakage specifically aims to make the system reveal the instructions that define its own operation: business logic, authorization rules, integration details, and safeguard configurations. Once these instructions are extracted, an attacker has an exact blueprint of how the system works, which then allows them to craft much more targeted attacks to bypass the defenses they have just uncovered.
Prompt Extraction Techniques: How Attackers Trick Large Language Models
Extraction methods range from the most trivial direct query—such as asking the model to repeat previous sentences or instructions—to obfuscation techniques designed to bypass text filters, such as Base64 or hexadecimal encoding. OWASP classifies encoding as one of the obfuscation techniques used in prompt injection attacks and recommends not limiting defenses to a simple keyword search.
Academic research has also shown that it is possible to automate the generation of extraction queries. The PLeak framework gradually optimizes an adversarial query to reconstruct the system’s prompt, while targeting an application accessible only through its query-response interface. The attacker therefore does not need direct access to the target system’s internal parameters. This approach was evaluated on real-world LLM applications hosted on Poe.
In January 2026, the researchers at Praetorian demonstrated that an LLM assistant could disclose its system prompt through editable fields in the interface, even though its chat responses were limited to predefined messages. In their scenario, the model entered fragments of the prompt into form fields and encoded them in Base64 to make them usable and bypass any filters. This example shows that a restriction limited solely to the text displayed in the chat is not enough: every field, action, log, or output controlled by the model can become a channel for data exfiltration.
Examples and Real-World Cases of System Prompt Leaks (Bing Chat, ChatGPT)
One of the first widely publicized cases was the leak of Bing Chat’s system prompt in February 2023. This Microsoft conversational assistant, integrated into the Bing search engine and based on an OpenAI language model, quickly became the target of attempts to uncover its internal instructions. Through simple injection queries, users obtained initial instructions revealing, among other things, the internal codename “Sydney,” behavioral constraints, and operational rules.
The episode demonstrated that a prompt hidden within the model’s context does not, on its own, constitute a reliable privacy boundary.
Since 2025, several public repositories have archived system prompts assigned to major chatbots such as ChatGPT, Claude, Gemini, Grok, and Perplexity. These collections document the existence of extraction techniques and provide access to numerous examples obtained through injection or reverse engineering.
OWASP illustrates the risk using the example of a banking application whose system prompt reveals a specific operational rule: a transaction limit set at $5,000 per day per user, as well as an overall loan limit. Disclosing this information could help an attacker bypass the associated controls, for example, by tailoring their transactions or credit requests to the known thresholds. This is a direct example of how a simple instruction leak can lead to exploitable financial fraud, without any technical vulnerabilities being required elsewhere in the system.
Best practices for system prompt security.
- Design with the assumption that the prompt will be disclosed. A system prompt can guide the model’s behavior, but it should not be considered either a secret or a security boundary. API keys, authentication tokens, and other sensitive information must be kept outside the prompt, in dedicated components.
- Enforce the rules on the backend. The model can interpret a request or propose an action, but authorizations, limits, privilege separations, and critical business controls must be verified and enforced in a deterministic manner by the surrounding system.
- Test the extraction before going live. Security campaigns must cover direct requests, multi-round attacks, obfuscation, structured outputs, interface fields, and tool calls. The results must be documented and periodically reevaluated.
- Minimize the impact of a leak. The system must be designed according to the principle of least privilege so that even if the prompt is fully disclosed, no further harmful actions are possible. Security must be based on controls that are independent of the model: server-side validation, authorization, quotas, logging, monitoring, and human intervention for sensitive operations.
→ See our guide to AI architectures and risk vectors to understand where this risk vector fits among the other risks targeting LLMs.
Frequently Asked Questions About the Prompt Leak
What scope should be analyzed for an AI system?
What deliverables can you expect from an AI risk analysis?
When should the risks of an AI system be reassessed?
Check how well your systems handle prompt extraction
Our offensive security experts test how well your models hold up against attacks involving data extraction, injection, and bypassing safeguards.
Would you like to discuss your project or assess your needs? Contact our experts.