The Tool Desk
Outbyte PC Repair FREERepair Windows errors before they cause bigger problemsFix Now →Outbyte Driver Updater FREEScan for outdated or missing drivers - takes under a minuteDriver Scan →An AI attack is an attempt to exploit a model or the system around it—by manipulating inputs, compromising data, extracting information, bypassing safeguards, or abusing connected tools. Not every attack changes model weights. The right defenses depend on what an attacker can control, what the system can access, and what outcome the attacker wants.
What is an adversarial attack on AI?
Adversarial machine learning (AML) covers attacks that exploit how an AI system is trained, queried, connected, or deployed. The target may be a model’s behavior, its training data, its confidentiality or availability, or an application that sends information to the model and acts on its output.
As an Amazon Associate I earn from qualifying purchases.
That distinction matters: a deployed system can be attacked without anyone modifying its weights. A carefully chosen input may cause a prediction error; a malicious document may steer a connected assistant; repeated queries may reveal information about a model; or an attacker may try to make a service unavailable.
NIST’s Adversarial Machine Learning: A Taxonomy and Terminology of Attacks and Mitigations (AI 100-2e2025), published in March 2025, organizes attacks by type, learning context, attacker goal, capabilities, and knowledge. NIST says it intends to update the report as the field changes. Its taxonomy covers both predictive and generative AI, but it is not a measure of how often attacks occur in the real world.
#1 Best Overall
How predictive AI attacks differ from generative AI attacks
Predictive models classify, score, detect, or forecast from input data. Generative models produce or transform content in response to prompts and may be embedded in chatbots, retrieval-augmented generation (RAG) applications, or agents that use tools. The categories overlap—both can face integrity, availability, and privacy threats—but the interaction model changes which attack paths matter most.
| System context | Typical attacker objective | Examples of relevant attacks |
|---|---|---|
| Predictive AI | Cause incorrect or attacker-favored predictions; disrupt operation; learn about the model or data | Evasion, poisoning and backdoors, availability or integrity attacks, model extraction, and inference about training data |
| Generative AI | Cause harmful or unintended output; bypass restrictions; expose information; disrupt service | Direct prompting attacks, indirect prompt injection, jailbreaking, poisoning, prompt extraction, training-data extraction, and leakage of user-interaction data |
| RAG application, chatbot, or agent | Influence the model through retrieved content or exploit what the application can do | Indirect prompt injection, misuse of connected tools, and exposure of data available to the application |
These are not mutually exclusive categories. A RAG chatbot, for example, contains a generative model but also has an application layer, retrieved documents, permissions, and possibly tools. An attack can exploit more than one of those components.
How can someone attack an AI model?
Start with four questions: what can the attacker control, what access does the system expose, what result does the attacker want, and what data or tools can the system reach? Those questions are more useful than treating “AI attack” as one technique.
Evasion: manipulate an input at use time
In an evasion attack, someone changes an input or how it is presented so a deployed model returns an incorrect or otherwise attacker-favored result. The details depend on the task and modality: an image classifier, a fraud detector, and a speech system do not have the same input surface. Evasion targets behavior at inference time; it does not, by itself, mean the model was retrained or its weights were changed.
Poisoning and backdoors: compromise development inputs
Poisoning occurs when an attacker influences training or other model-development inputs to compromise behavior, integrity, or another goal. A backdoor is a possible outcome in which the model behaves normally in ordinary cases but responds differently when a particular trigger or condition appears. Poisoning therefore concerns the data or process used to build or adapt a model, rather than simply an unusual user prompt at deployment.
Availability and integrity attacks: disrupt service or results
An availability attack aims to prevent a system from operating as intended; an integrity attack aims to make its behavior or outputs unreliable or attacker-controlled. These goals also apply to conventional software and data services. In AI deployments, risks may involve the model, the surrounding infrastructure, the data pipeline, or the application that consumes outputs.
Privacy attacks and model extraction: learn from access
Privacy attacks seek information about training data, a model, or user data handled by a system. They do not necessarily recover complete records. Model extraction refers to attempts to learn or reproduce information about a model through access to its outputs; it is not the same as stealing the model’s weights.
Recommended Free Tools
Prompt injection: introduce instructions the system should not trust
A prompt injection presents malicious instructions to a generative system. A direct injection arrives in the user’s prompt. An indirect injection is embedded in content the model processes, such as a retrieved document, webpage, or other external source. Indirect attacks become especially important when an application combines user requests with untrusted content or gives a model access to tools.
Rank #3
Jailbreaking and misuse: try to elicit disallowed behavior
Jailbreaking attempts to bypass a model’s restrictions; misuse describes harmful or unauthorized objectives pursued with a generative system. These are not synonyms for poisoning, and neither is the same as an ordinary hallucination. A hallucination is an inaccurate output; an attack involves an adversary’s actions or goals, though an attacker may exploit a system’s tendency to produce unreliable content.
What is prompt injection, and how is it different from a jailbreak?
Prompt injection is about where malicious instructions enter and how they influence a model’s task. A jailbreak is an attempt to bypass restrictions or safeguards. They can overlap, but they describe different aspects of an attack: an injection may arrive through a user prompt or external content, while a jailbreak describes the intended bypass. An application can also be exposed to indirect injection without the user deliberately trying to jailbreak the model—for example, if the system retrieves a document containing instructions that conflict with the user’s request.
The risk grows when a model is treated as if all text it receives has equal authority. Retrieved material may be useful evidence, but it should not automatically be allowed to override system instructions, disclose private context, or direct an agent to take actions. The application’s permissions and trust boundaries determine how consequential a successful manipulation could be.
Quick wins for a faster PC:
Scan for outdated or missing drivers - takes under a minuteDriver Scan →Clear out junk files and repair common Windows errorsFree Scan →Fix the driver behind crashes, sound loss and screen glitchesFind Drivers →Can an AI agent be tricked by instructions hidden in a document?
Yes. If an agent reads untrusted content and can act on tools or data, a malicious instruction in that content may try to steer its behavior. Whether that becomes a meaningful compromise depends on how the application handles the content, what permissions the agent has, and whether actions require checks outside the model. A model that can only summarize a public document has a different exposure from one that can send messages, access private records, or modify systems.
Design the application so untrusted content cannot grant itself authority. Keep tool permissions narrow, separate access to sensitive data from ordinary retrieval, and place consequential actions behind well-defined interfaces and appropriate approval or validation. Do not rely on the model alone to decide whether an instruction found in external content is trustworthy.
Can poisoned training data change what an AI model does?
Yes. If an attacker can influence training or development inputs, poisoned data can alter behavior, compromise integrity, or contribute to a backdoor that responds to a trigger. The possibility does not mean that every bad output proves poisoning: mistakes can also arise from ordinary model limitations, flawed data, application bugs, or other causes. Establishing poisoning requires evidence about the data or development process, not just an unexpected answer.
For development pipelines, protect data sources and access, track provenance and changes, review unexpected dataset shifts, and include security checks in training and fine-tuning workflows. Treat third-party data and model components as dependencies to assess, not as inherently trustworthy.
How do you defend an AI model against attacks?
There is no universal control that makes an AI system immune. NIST’s guidance treats mitigation as a set of layers across development and deployment. For prompt injection, it discusses task-specific training, detection, input processing, and designs that distinguish trusted from untrusted content. It also recommends assuming that injection remains possible when systems process untrusted inputs, then limiting access to those sources through separate permissions or well-defined interfaces. NIST cautions that current mitigations do not fully protect against every technique.
Best Value
Reduce exposure during design and integration
- Map where prompts, training data, retrieved content, model outputs, and user data enter or leave the system.
- Mark external content as untrusted and keep it distinct from instructions that govern the application.
- Give models and agents only the data and tools required for their task; separate sensitive resources and high-impact actions.
- Validate and constrain outputs before downstream software treats them as commands, decisions, or trusted data.
Protect development and operation
- Control access to training, fine-tuning, deployment, and evaluation pipelines; record important data and model changes.
- Use conventional security controls for software, infrastructure, accounts, and data. AI systems still face confidentiality, integrity, and availability risks.
- Test for relevant attack classes before deployment and after material changes to models, prompts, data sources, permissions, or connected tools.
- Monitor for abnormal inputs, output patterns, access, and resource use, with a response plan for incidents and suspected data exposure.
Test the system, not only the base model
A base-model evaluation cannot by itself establish how a complete application will behave. Test the actual configuration: prompts, retrieved sources, filters, permissions, tool calls, and downstream handling. Include both expected use and plausible misuse, and record what the attacker can access and what impact a failure would have.
NIST describes Dioptra as a research testbed for assessing model vulnerabilities and the effectiveness of defenses. It is a research resource, not a guarantee of protection or an endorsement of a particular product. NIST also notes that conventional security frameworks do not comprehensively address several AI-specific attacks or the complexity of AI attack surfaces, so AI-focused assessment should complement—not replace—standard security practice.
What the evidence does—and does not—say about AI attacks
NIST’s 2025 report says its review considered more than 11,354 references on arXiv.org since 2021, as of July 2024. That figure describes the size of the literature cited; it is not a count of real-world incidents, a measure of attack success, or proof that attacks are increasing. The report is also not an exhaustive survey of all adversarial-ML literature.
Free tools Windows power users keep installed
One-click scans. No signup required.
The practical conclusion is to treat AI security as an ongoing lifecycle and system-security responsibility. Identify attacker access and goals, limit what a model-connected application can reach, and evaluate the protections around the complete deployment. Those measures reduce risk, but no single model setting or filter should be presented as a foolproof defense.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
