A threat model for an AI application has to cover the software around the model, not only the model’s behavior. Model output becomes dangerous when application code, a tool, a database, or a downstream system acts on it with some authority. The review therefore covers user interfaces, orchestration, retrieval and memory, tools, permissions, identities, output consumers, hosting, and suppliers, and it traces how a bad input or a bad output could travel across those parts.
The method has four steps, repeated whenever the system changes: describe what you are building; walk the data flows and ask what could go wrong; decide what you will do about each credible scenario, based on its likelihood and impact; and check whether the model still matches the running system. No list of risk categories is a complete threat model. Which risks matter, and how much, depends on your architecture, your data, and what a failure would cost.
As an Amazon Associate I earn from qualifying purchases.
Start with the system boundary, not the model
Draw the system before you list threats. A useful diagram shows at least the following components:
What’s actually slowing this PC down?
Pick the symptom - the matching free tool is one click away.
- Users and the interfaces they use, including API clients and administrative consoles.
- The orchestration layer that assembles prompts, calls the model, and decides what happens next.
- The model itself, whether a hosted API or a model you run locally.
- Retrieval and memory: the vector store or search index, the embedding pipeline, conversation memory, and the documents that feed them.
- Input sources you do not control, such as uploaded files, web pages, email, or support tickets. Indirect prompt injection can arrive through any of them.
- Tools, the credentials they use, and every downstream service they reach.
- Output consumers: a page that renders the answer, a parser, a database writer, a ticketing system, or another service.
- Logging, monitoring, the hosting environment, and every external supplier.
Then mark four things on the diagram:
- Trust changes. Every arrow that carries user-controlled, third-party, or internet content into trusted code, and every point where a user’s identity should limit what the system is allowed to do.
- Sensitive data crossings. Where personal, financial, credential, or confidential data enters a prompt, a log, a retrieval result, a tool argument, or an output.
- Identities. Which account, service principal, or token makes each call to the model, to the retrieval store, and to each tool.
- Read and write paths. Which components can only read, and which can change state.
Finally, separate model-generated text from actions. A string the model produces is a draft or a claim until application code or a tool acts on it. The same bad answer has very different consequences if it stays on a screen or if it becomes the argument to a refund call. Your diagram should show exactly where that handoff happens.
#1 Best Overall
The four-part method
Step 1: What are we building?
Record the system’s purpose, its users, the classes of data it touches, its components, its external dependencies, and its trust boundaries. Give every component a named owner and state which identity it runs as. If you cannot say who owns a component or what it is allowed to access, the model is not ready to use yet.
Step 2: What can go wrong?
Walk each data flow and each attacker path. For every untrusted input, ask how it could change five things: what the retrieval step returns, how the model behaves, what tools are asked to do, what downstream consumers do with the output, and whether the service stays available and affordable. Include compromised dependencies in this walk. A package, a model file, an embedding set, or a container image can change upstream without any change to your prompts.
Step 3: What will we do about it?
For each credible scenario, choose a mitigation, assign an owner, and set a target. Choose controls against the likelihood and impact you recorded, not against the category name. Prefer controls that do not depend on the model deciding correctly. The model can be persuaded; an authorization check in tool code will refuse a request whether or not the model was persuaded.
Quick wins for a faster PC:
Clear out junk files and repair common Windows errorsFree Scan →Fix the driver behind crashes, sound loss and screen glitchesFind Drivers →Repair Windows errors before they cause bigger problemsFix Now →Step 4: Did we do a good job?
Check the model against the implementation. Confirm that the diagram matches the deployed configuration, including the actual token scopes on each tool and the actual filters on retrieval. Test the boundaries you care about: plant an instruction inside a document in a staging retrieval index and see whether it produces a tool call. Review logs for tool invocations and state changes. Feed incidents and design changes back into the model. Reassess when you change models, add a tool or data source, or when a supplier changes a component you depend on.
Use the OWASP risk categories as prompts, not as a scorecard
OWASP’s Top 10 for LLM and GenAI Applications, 2025 edition, names ten categories. Each one is a question to ask about your system. The table below pairs each category with the question that tests it. NIST’s adversarial machine learning taxonomy (NIST AI 100-2 E2025) offers complementary AI-specific categories if you want a second vocabulary.
| Category | Question to ask of your system |
|---|---|
| LLM01 Prompt Injection | Can a user, or a document the system retrieves, change the instructions the model follows? |
| LLM02 Sensitive Information Disclosure | Can the system reveal data the requester is not authorized to see, through output, logs, or tool results? |
| LLM03 Supply Chain | Can you verify the origin and integrity of models, packages, datasets, plugins, and hosting components, and do you control when they change? |
| LLM04 Data and Model Poisoning | Can tampered training, fine-tuning, or retrieval data alter behavior? |
| LLM05 Improper Output Handling | Is model output validated before a browser, interpreter, database, shell, URL fetcher, or tool consumes it? |
| LLM06 Excessive Agency | Can a tool act beyond the requester’s permissions, or make an irreversible change without approval? |
| LLM07 System Prompt Leakage | Does the system prompt contain secrets, or does the design depend on the prompt staying hidden? |
| LLM08 Vector and Embedding Weaknesses | Does retrieval enforce the requester’s permissions, and can attacker-supplied content get into the index? |
| LLM09 Misinformation | What happens when a plausible but wrong answer is shown to a customer or acted on by a system? |
| LLM10 Unbounded Consumption | Can repeated or oversized requests create unacceptable cost or degrade service for others? |
Not every category applies with equal force. A read-only help bot over public documents has a very different profile from an agent that can move money. Let the architecture decide which questions get deep attention.
Make agent capabilities explicit
Treating “agent” as a single risk category hides most of the differences that matter. NIST’s August 5, 2025 summary of its tool-use workshop states the framing directly: “AI agents can perceive and take actions in environments; the leading AI agent paradigm today embeds general-purpose AI models into systems with software scaffolding that enable a model to manipulate tools to take actions beyond simple text output.”
Windows Errors? Fix Them Before They Spread
Repair common Windows errors and clear accumulated junk for a smoother, more stable PC - no reinstall needed.Free scan · no reinstallCrashes, No Sound, or Screen Glitches?
Random freezes, missing sound and display glitches usually trace back to one bad driver. Find and replace yours safely.Free scan · under a minuteThe same summary describes several dimensions for characterizing a tool: the action it enables; whether it reaches external resources or has write permission; the severity, statefulness, and reversibility of the harm it could cause; the reliability of the tool and the model; the modality; how observable its behavior is; and how autonomously it acts. Its examples run from read-only retrieval-augmented generation in trusted environments, through constrained-write GUI or API use, to write-enabled coding or computer use. Each of those sits at a very different point on the risk scale.
Rank #3
Record each tool in a capability table. The rows below are illustrative, for a hypothetical support assistant; your values will differ.
| Tool | What it enables | Access and write scope | Reversibility and severity | Autonomy | Environment | Owner |
|---|---|---|---|---|---|---|
| search_help_center | Search public help articles | Read-only; index holds public documents only | No state change; low harm | Automatic | Trusted internal service | Support operations |
| lookup_invoice | Show invoice status | Read-only; must be scoped to the logged-in customer’s account ID | No state change; exposes billing data if scoping fails | Automatic | Internal billing API | Billing engineering |
| issue_refund | Create a refund up to a limit | Write; payment system | Partly reversible; money leaves the company | Approval required above threshold | Payment provider API | Finance operations |
| send_customer_email | Send a templated email | Write; external recipient | Not reversible once sent | Approval required | Outbound mail service | Support lead |
| run_analysis_code | Execute Python for data analysis | Write inside a sandbox only | Depends on sandbox isolation | Automatic | Isolated container with no production credentials | Platform team |
For every row, ask what happens if the model requests the worst plausible arguments. If the honest answer is “the tool executes them,” the control belongs in the tool layer.
Worked example: where model output meets authority
Consider the hypothetical support assistant above. It answers questions from the help center, looks up invoices for the logged-in customer, reads the customer’s ticket history for context, and can issue refunds up to a limit. The two scenarios below show how the method turns a model-level concern into engineering controls.
Free tools Windows power users keep installed
One-click scans. No signup required.
Scenario A: a planted instruction in a ticket
Someone submits a support ticket containing text written to look like instructions. It tells the assistant to refund a large amount to a different account and to leave that out of its reply. Later, another customer’s session retrieves that ticket as context, and the model proposes the refund call.
- Boundary crossed: third-party ticket content enters the prompt, where the model may treat it as instructions.
- Attacker prerequisite: the ability to submit a ticket that is indexed and later retrieved.
- Existing control: a line in the system prompt saying not to refund other accounts. This is a request to the model, not an enforced check, so it cannot serve as the control.
- Required controls: the refund tool checks on the server that the destination account matches the authenticated customer; refunds above the threshold enter a human approval queue; retrieved tickets are passed to the model labeled as customer data. Labeling helps, but it is not a guarantee. The enforcing control is the server-side check. Every tool call is also logged together with the retrieved items that were in the prompt.
- Verification: in a staging index, plant a ticket with the same instruction. Confirm that the refund is rejected or queued, and that the log records the attempt with the ticket identifier.
Scenario B: retrieval that ignores the requester
If the retrieval step searches all tickets and returns records from every customer, then the model is left to avoid mentioning records that belong to someone else. One customer’s invoice or ticket text can then appear in another customer’s answer. The fix belongs in the retrieval query: filter by the authenticated customer identifier before any result reaches the prompt. Verify it with two test accounts, confirming that a query made from one account returns no documents belonging to the other.
Record and rank scenarios
For every scenario, record eight fields: the asset or user affected; the attacker’s prerequisite; the trust boundary crossed; the plausible consequence; the control that exists today; a likelihood rating; an impact rating; and the accountable owner. Use simple levels such as low, medium, and high, and write one sentence of reasoning for each rating. Ratings with no reasoning attached cannot be reviewed later.
Do not adopt a universal ranking. The NIST Cybersecurity Framework examples call for recording likelihood and impact and for accounting for cascading failures, and both depend on the deployment. A refund call that is reversible and capped is a different priority from one that wires funds to external accounts without approval. Priority depends on data sensitivity, exposure to untrusted input, tool permissions, the degree of autonomy, reversibility, and the business consequence of a failure. A failure in one component can also cascade: an unauthorized refund can trigger a fraud review that blocks other customers’ payouts, and that second-order effect belongs in the record.
When you compare two designs, use the same axes for both. NIST notes that risk depends on the deployment conditions, and that access, autonomy, and monitoring differ between implementations.
Best Value
| Axis | Lower-exposure setting | Higher-exposure setting |
|---|---|---|
| Access | Read-only retrieval | Write-enabled tool or computer use |
| Input trust | Curated internal documents | Public web pages, uploaded files, inbound email |
| Identity scope | Per-user token limited to one account | Shared service account with broad rights |
| Autonomy | Model proposes, a person approves | Model acts without review |
| Reversibility | Drafts and reversible changes | Payments, deletions, external messages |
| Environment | Isolated, no production credentials | Production systems and secrets reachable |
| Monitoring | Every tool call logged with arguments and retrieved sources | Only infrastructure-level request logs |
| Data sensitivity | Public or non-personal data | Personal, financial, health, or credential data |
| Cost and availability | Per-user budgets and rate limits | Unmetered calls to paid model APIs |
Connect each threat to a control and a check
Each scenario needs one or more control options, and each control needs a way to verify it. The options below are candidates to evaluate against your scenarios. None of them guarantees security on its own, and each has costs in latency, complexity, or user experience.
| Threat | Control options to evaluate | How to verify |
|---|---|---|
| Injected instructions change tool use | Server-side authorization of every tool call against the authenticated user; approval gates for high-impact actions; retrieved content passed as labeled data | Plant instructions in a staging document and confirm the call is rejected or queued, and logged |
| Cross-account disclosure through retrieval | Filter at query time by requester identity; separate indexes per tenant where practical | Use two test accounts and confirm that one account’s queries return none of the other’s documents |
| Unsafe output handling | Treat model output as untrusted text; encode before rendering as HTML; parameterize any SQL; allowlist URLs and commands | Send output containing markup, SQL fragments, and internal URLs through each consumer and confirm nothing executes |
| Secrets or personal data in prompts and logs | Pass only the fields a task needs; redact before logging; keep credentials out of prompts entirely | Plant test identifiers in inputs and search a sample of logs and traces for them |
| Unbounded consumption | Per-user rate and token budgets; maximum input and output sizes; spend alerts | Load test in staging and confirm limits trigger and alerts fire |
| Tampered models, data, or packages | Pin versions; verify hashes or signatures where the supplier offers them; restrict who can publish model or index updates | Compare the deployed artifact’s hash with the recorded value |
| Unsafe agent action | Isolated execution; scoped, short-lived tokens; reversible-first designs that queue rather than execute | Attempt an out-of-scope call in staging and confirm it fails and is logged |
Beyond the model: suppliers and operations
Attackers do not always need to talk to the model. NIST’s summary of a 2025 MITRE ATLAS presentation describes demonstrated attacks against AI workloads and the wider GenAI ecosystem that could be deployed without user interaction. That supports including the surrounding service and its supply chain in the model, not only the prompt interface.
Inventory each of these components and record its provenance, its update path, who can change it, and its owner:
- Hosted model APIs, and model weights you host yourself.
- Training or fine-tuning data, where your organization controls it.
- Retrieval corpora, the vector database, and the embedding pipeline.
- Orchestration frameworks, packages, and container images.
- Cloud services, tool providers, and monitoring or logging systems.
- Configuration settings, including system prompts, retrieval filters, and tool scopes.
NIST’s control-overlay project for securing AI systems (COSAiS) explicitly includes training and test data, model weights, and configuration settings in its scope, which is a useful checklist for this inventory. For each supplier, write down the incident path: how you would learn of a compromise or an unannounced model change, how you would roll back to a pinned version, and which tokens and tools you would disable first.
What the sources establish, and what they do not
- Risk categories are not frequencies. The OWASP 2025 list identifies classes of risk. It does not say how often each occurs, and neither do the NIST documents cited here. Do not rank by the number of categories or by assumed incident rates. Rank by your own exposure.
- NIST AI 100-2 E2025 is voluntary guidance on adversarial machine learning. NIST has said it plans annual updates, so check for a newer edition before relying on a specific section number.
- NIST’s COSAiS project is active. Its project page included a discussion draft dated January 2026 at the time of the sources reviewed; later drafts may have been published.
- The MITRE ATLAS material is a presentation summary as reported by NIST, not a complete catalog of incidents.
- The NIST workshop figure of approximately 140 experts describes attendance at a January session on agent tool use. It is not a measure of attack prevalence or of how well any control works.
The practical consequence is that your threat model must rest on your own architecture: what data the system holds, what its tools can change, who can reach it, and what a failure would cost.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
