Evaluate an AI vendor against the risks of your particular use—not against a general promise that its product is “safe” or “private.” Define the task and potential harms, trace the data through the service, ask for evidence that matches the claims, and make key protections enforceable in the contract. Then record unresolved questions and decide whether they are acceptable for this deployment.
1. Define what the AI will do and what could go wrong
Before reviewing a vendor’s materials, write down the proposed deployment. A tool used to draft internal notes has a different risk profile from one that influences a decision about a person. Suitability depends on the task, the data, who is affected, and the consequences of an error or disclosure.
As an Amazon Associate I earn from qualifying purchases.
- What task will the service perform, and who will use its output?
- Who could be affected by the output, including people who are not your users?
- Will a person review the result before it is acted on? What can that reviewer realistically check?
- What data will be submitted, how sensitive is it, and at what scale?
- What foreseeable misuse, incorrect output, service failure, or data exposure could cause harm?
- What sector or jurisdictional obligations may apply? Get qualified legal advice where needed.
NIST’s voluntary AI Risk Management Framework (AI RMF) treats trustworthiness as context-dependent: characteristics such as validity and reliability, safety, security, privacy, explainability, and fairness can involve trade-offs that should be considered for the intended use. The framework page says the AI RMF is being revised, so record which version a vendor used if it maps its processes to the framework. The AI RMF is guidance, not a certification that a vendor or model has passed an independent safety test. NIST AI Risk Management Framework; NIST AI risks and trustworthiness.
2. Find out where your data goes and why
Ask for a data-flow diagram or an equally concrete written account for the exact product plan and configuration you are considering. A short statement such as “we protect your data” does not explain which information is collected, who can access it, or what happens after you stop using the service.
#1 Best Overall
Trace each data category through the service
Ask the vendor to address prompts, uploaded files, outputs, feedback, telemetry, logs, human review, support access, storage regions, subprocessors, and any upstream model providers. For each category, ask whether it is used for service delivery, abuse monitoring, evaluation, fine-tuning, or general model training. Distinguish those purposes rather than accepting an unqualified answer to “Do you train on my data?”
Get specific about retention and deletion
Ask how long each category is retained, what event starts that period, how deletion is requested and confirmed, whether copies remain in backups, and what happens to data after contract termination or service decommissioning. Clarify whether deletion covers information already sent to subprocessors or used in evaluation systems. NIST’s Generative AI Profile identifies retention, data security, third-party access, and possible leakage after decommissioning as governance concerns. NIST Generative AI Profile.
Read the documents together
Check the data processing agreement, privacy policy, product terms, and order form for the same product tier and geography. Look at definitions, exceptions, and opt-in or opt-out conditions; a marketing statement may not describe every plan or configuration. The FTC Office of Technology’s January 2024 staff post says companies should honor privacy and confidentiality commitments made in promotional materials, website terms, or marketplaces—including commitments about model training—and notes that omitting material information about data collection or use may matter. It is agency staff commentary, not a complete legal opinion for every jurisdiction. FTC Office of Technology: AI Companies—Uphold Your Privacy and Confidentiality Commitments.
3. Match each safety, security, and privacy claim to evidence
Ask for evidence that concerns the model, product configuration, and operating conditions you would actually use. For each claim, record what was evaluated, who performed the work, when it was done, what the result covers, and what remains outside its scope. Separate independently verified evidence from vendor-provided material, contractual promises, and unanswered questions.
For safety and performance
Ask for the test scope and methods, scenarios, model and version, configuration, operating conditions, date, results summary, known limitations, relevant incidents, remediation, and update history. Find out whether the tests reflect your users, workflow, and foreseeable misuse. Ask how you will be notified when a model or configuration changes in a way that could affect the assessment.
For security
Ask what systems and services an audit or certification covers, its scope and dates, and whether it includes the AI service and relevant upstream providers. Request details about access controls, encryption, isolation, vulnerability handling, security testing, and incident response. A badge or general security certification does not by itself show that the AI system behaves safely. NIST identifies AI-related security concerns including adversarial examples, data poisoning, and attempts to extract models, training data, or intellectual property through service endpoints. NIST AI risks and trustworthiness.
For privacy
Ask how the vendor assesses privacy risks in your workflow, including purpose limitation, data minimization, access, retention, deletion, and any applicable handling of data-subject requests. Consider risks created by inference or by information revealed in outputs, not only whether submitted files are stored. NIST SP 800-63-4 includes AI/ML provider information-sharing and privacy-assessment provisions in the specific context of digital identity systems; do not treat those provisions as a universal procurement rule for every AI vendor. NIST SP 800-63-4: Digital Identity Guidelines.
Do these 3 things before closing this tab:
1Fix the driver behind crashes, sound loss and screen glitches2Clear out junk files and repair common Windows errors3Scan for outdated or missing drivers - takes under a minute4. Examine the supply chain and put important promises in writing
Ask the vendor to identify relevant upstream models, embedded AI, APIs, fine-tunes, tools, data providers, and other third parties that can access your content. Find out which parties receive which data, what protections apply to them, and what changes—such as a new model provider or subprocessor—will trigger notice and a chance to reassess.
Translate critical assurances into contract, data processing agreement, or service-level terms that apply to your specific plan. Depending on the deployment, address:
- Permitted data uses, including any limits on training, fine-tuning, evaluation, or other secondary use.
- Ownership and usage rights for submitted material and generated outputs.
- Security requirements, incident notification, and cooperation during response.
- Retention, deletion, and handling of data held by subprocessors.
- Evaluation or audit access sufficient to check relevant processes and standards.
- Service availability, support response, responsibility for failures, and liability allocation.
- Transition assistance, fallback arrangements, and what happens when service ends.
Do not rely on a sales assurance where the written terms leave the point open or provide a broad exception. NIST’s Generative AI Profile recommends use-case-based supplier assessment and contract clauses that let organizations evaluate third-party generative AI processes and standards. NIST Generative AI Profile.
Independent reader supportYour contribution helps us test, update, and keep practical guides available for everyone.5. Compare vendors on the same evidence standard
Give each vendor the same questions and assess evidence against the deployment you defined—not against the most polished presentation. The table is a comparison worksheet: fill it with evidence and terms for each candidate, and mark missing information as “unknown” rather than assuming the answer is favorable.
| Comparison area | What to compare |
|---|---|
| Data use and retention | Training and other secondary uses; retention, deletion, regional handling, and subprocessor access. |
| Safety evidence | Relevance of tests to your use, the model and configuration assessed, limitations, and change transparency. |
| Security and incidents | Control coverage, independent assessment scope, vulnerability handling, incident response, and cooperation. |
| People and recourse | Human oversight in your workflow and how affected people can challenge or correct consequential outputs. |
| Contract and continuity | Evaluation rights, remedies, responsibilities, fallback options, and dependence on a single vendor. |
Weight these areas according to likely harm and operational need. NIST cautions that trustworthiness characteristics may need to be balanced in context, with trade-offs transparently justified; a single aggregate score can hide a critical weakness. NIST AI risks and trustworthiness.
Best Value
6. Reassess after approval
Approval is for an assessed service, configuration, data flow, and use case—not a permanent endorsement of everything the vendor may later offer. Keep an inventory of AI vendors and affected data, record the model or service version and configuration assessed, and assign an owner to track changes.
- Reassess after material changes to the model, product, data use, subprocessors, or contract.
- Review after an incident, a new use case, or a change in data sensitivity or potential impact.
- Set escalation and suspension criteria, identify a fallback, and agree who communicates with affected stakeholders before deployment.
NIST’s Generative AI Profile recommends ongoing monitoring and contingency processes for high-risk third-party systems. NIST Generative AI Profile.
When to pause or narrow a deployment
Do not treat missing evidence as proof of a breach, but do not treat it as proof of protection either. Pause, reduce the sensitivity or scope of the data, add human review, or choose another service if a gap prevents you from judging a material risk—especially when the vendor cannot explain data use, deletion, third-party access, or the limits of safety testing in terms that match your workflow.
Recommended Free Tools
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
