Model Poisoning & AI Security: What Are the Risks for Your LLMs?

A long-held assumption used to assess the risk of poisoning was that the amount of corrupted data required increases with the total volume of training data. Recent research, however, calls this relationship into question, at least for certain backdoor attacks. In fact, Anthropic, in collaboration with the UK AI Security Institute and the Alan Turing Institute, in a study published in October 2025, directly contradicted this assumption. By training 72 models of varying sizes—ranging from 600 million to 13 billion parameters—the researchers demonstrated that a nearly constant number of approximately 250 poisoned documents was sufficient to implant a backdoor, regardless of the model’s size or the total volume of its training data. A model with 13 billion parameters—trained on more than twenty times as much data as a model with 600 million parameters—was found to be compromised by exactly the same number of corrupted documents.

Model Poisoning vs. RAG Poisoning: What Are the Differences for AI Security?

Model poisoning targets the training or fine-tuning process itself by corrupting the data from which the model learns, in order to introduce hidden behavior that becomes an integral part of its weights. This is what structurally distinguishes it from RAG poisoning: the latter corrupts a knowledge base consulted at the time of the query—a component external to the model that can be cleaned or reconstructed without modifying the model itself. A model that has been poisoned carries the corruption within its own parameters; removing it requires retraining or correcting the model—an operation that is significantly more costly than simply cleaning the document corpus.

AI Backdoor: How Does Backdoor Poisoning Work?

A backdoor installed in this way remains invisible during normal use: the model behaves identically to a healthy model for the vast majority of queries and reveals its corrupted behavior only when a specific trigger phrase known to the attacker is present. In the Anthropic study, the trigger tested produced incoherent text—a deliberately low-stakes choice to make the experiment measurable and reproducible—which the researchers themselvesthemselves describe as not very representative of the most dangerous backdoors that a motivated attacker might target in practice, such as generating code containing a vulnerability or disclosing sensitive information upon trigger activation.

A second research finding—distinct from the Anthropic study but consistent with it—illustrates the potential scale of the problem in a real-world context: A study published in *Nature Medicine* showed that replacing just 0.001% of the training tokens with medical misinformation was enough to produce models significantly more likely to spread clinical errors, while achieving scores identical to those of healthy models on all standard benchmarks, rendering detection via conventional evaluation completely ineffective.

AI Security Alignment: Why Do Backdoors Resist Fixes?

The most concerning finding documented by recent research pertains to the resistance of these backdoors to standard mitigation measures: several studies, including thestudy by Hubinger et al. (2024) have shown that a backdoor installed during training can survive subsequent security alignment phases, which are designed precisely to eliminate undesirable behaviors in a model. A model can thus successfully pass all standard security evaluations while retaining hidden behavior that can be activated by a specific trigger, making reliance on a post-training alignment process alone insufficient as a guarantee against compromise.

AI Risks in Business: Fine-Tuning, Datasets, and the Supply Chain

Few organizations today train their foundation models entirely from scratch; the bulk of the risk exposure in the AI supply chain stems from two practices that are already widely deployed. On the one hand, there is fine-tuning using proprietary data collected through external service providers or unverified contributors—a known vector for data poisoning, where a malicious actor can inject corrupted examples into otherwise legitimate datasets. Second, the use of pre-trained models and open datasets downloaded from community platforms, without any guarantee that their provenance and compilation history have been audited, exposes systems to risks of model poisoning and compromise of the data supply chain, similar to “slopsquatting” attacks on hallucinating software packages.

AI Model Safety: 4 Best Practices to Prevent Model Poisoning

1. Document the source of the training and fine-tuning data

Track and document the origin, licenses, limitations, and preprocessing history of each dataset used, via “dataset sheets” and a source registry with quality controls. Require the same traceability from any vendor providing fine-tuning data (certification of origin, collection methods, and quality controls). This documentation is a prerequisite for audits and compliance (AI Act, DMP, etc.).

2. Don’t rely solely on standard benchmarks

Poisoned models may perform normally on standard evaluations while exhibiting malicious behavior under certain conditions. Recent research shows that “sleeper cell” backdoors in LLMs can achieve a very high attack success rate while remaining undetectable on standard benchmarks. In the medical field, even a minuscule substitution of training tokens can lead to a significant increase in harmful completions without being detected by traditional evaluations. Success on benchmarks should therefore not be considered proof of safety.

3. Monitor behavior in response to suspicious triggers

In addition to performance tests, conduct specific assessments to detect abnormal behavior triggered by unusual phrases, patterns, or conditions. This includes:

• “behavioral probing” (systematic testing using a variety of inputs),
• the analysis of internal activations (tokens or patterns that produce unusual representations),
• Identifying triggers through reverse engineering or semantic drift analysis, followed by targeted testing.

To date, these assessments are the only reliable method for detecting a dormant backdoor.

4. Limit reliance on models fine-tuned on unaudited third-party data

For high-stakes applications, prioritize data sources with a documented and verifiable chain of custody over open datasets with no guarantee of origin.

Key Measures:

• map the AI supply chain and classify each dependency by risk level,
• require standardized certificates from suppliers regarding the origin and quality controls,
• Prioritize “trusted repositories” and maintain private model registries with immutable snapshots and internal signatures.
  If a provider cannot identify the sources and processing steps of the data, this must be treated as a risk of unknown provenance.

FAQ: Everything You Need to Know About Model Poisoning and LLM Security

Audit and Penetration Testing: Assessing Your Models’ Vulnerability to Model Poisoning

Our experts evaluate the data lineage of your training and fine-tuning data as part of ourAI audit service.

Would you like to discuss your project or assess your needs? Contact our experts.