An AI audit should examine both the system and the organization around it: what the AI is used for, who may be affected, whether evidence supports its performance and safety, how people can intervene or seek recourse, and how risks are monitored and addressed after deployment. Start by defining the audit’s scope and risk criteria; then gather evidence across governance, data and system behavior, human oversight, operations, and reporting. This checklist is a starting point, not a substitute for requirements tailored to your jurisdiction, sector, and use case.
What should an AI audit check?
Check whether the organization can explain and oversee the AI system throughout its lifecycle, and whether its claims about the system are supported by evidence relevant to the way it is actually used. An audit may cover a model, an AI-enabled product or service, or a broader process in which AI influences decisions. Purchased products, third-party services, and vendor updates belong in scope when they affect the system being examined.
As an Amazon Associate I earn from qualifying purchases.
Use a risk-based approach: the depth of review should reflect the system’s context, scale, autonomy, reversibility, and potential consequences for people, organizations, society, or the environment. NIST describes its AI Risk Management Framework (AI RMF) as a voluntary, use-case-agnostic resource intended to help developers, users, and evaluators manage such risks. It organizes outcomes into four functions—Govern, Map, Measure, and Manage—but those functions are not a prescribed sequence or a ready-made audit checklist.
Recommended Free Tools
1. Define the scope and evaluation criteria
Set the boundaries before testing. Without a defined system, intended use, affected context, and evaluation criteria, audit findings can be difficult to interpret or apply.
#1 Best Overall
- Identify the system. Record its name, purpose, components, model or product versions, owner, and dependencies. Include relevant vendors and third-party services.
- Describe its use. Document intended uses, actual uses, the outputs it produces, and the decisions or actions those outputs influence. Note known uses outside the approved purpose.
- Set boundaries. Record what is in and out of scope, the lifecycle stage, deployment environment, scale, degree of autonomy, and whether decisions can be reversed or corrected.
- Identify affected people and groups. Consider direct users as well as employees, customers, communities, and people subject to decisions made with the system’s assistance.
- Assign ownership. Identify who is accountable for the business outcome, system operation, data, model, vendor relationship, risk acceptance, and audit follow-up.
- Define acceptable risk. State the assumptions, limitations, exclusions, and organizational risk tolerance against which evidence will be assessed.
- Map applicable obligations. Determine which laws, regulations, contracts, and internal policies apply to this particular deployment, and record who validated that mapping. Requirements vary with jurisdiction, sector, system, use, and affected population.
2. Examine governance and accountability
Review whether the organization has the authority, processes, and records needed to manage AI risks—not just whether an individual model performs as expected.
- Is there a complete AI system inventory, prioritized according to organizational risk?
- Are decision rights, responsibilities, escalation paths, and communication channels documented?
- Are development, deployment, risk oversight, and audit responsibilities sufficiently distinct to enable independent challenge?
- Are AI policies connected to existing enterprise risk, privacy, cybersecurity, safety, procurement, and internal audit processes?
- Have staff and decision makers been trained on intended use, system limitations, and incident procedures?
- Are third-party models, data, software, hardware, and services documented, with contract responsibilities and change notification or review arrangements understood?
- Where warranted, are impact assessments performed, and do their findings inform controls and approvals?
- Does each audit finding have an accountable owner, due date, and defined evidence required for closure?
3. Test data, model, and system evidence
Examine the evidence behind the system’s design and claimed performance. Tests should reflect the intended deployment context, and the audit record should make clear what was tested, what was not, and how much uncertainty remains.
Data and documentation
- Can the organization trace data sources, collection, rights, applicable consent or other legal basis, transformations, labeling, retention, access, and deletion?
- Are training, validation, and evaluation data suitable and sufficiently representative of the intended context and populations?
- Are known data gaps, historical biases, measurement errors, and other limitations recorded?
- Can reviewers inspect system and model documentation, versions, configurations, dependencies, and relevant prompts or rules?
- Are material vendor changes documented and assessed?
Evaluation design and results
- Are test sets, metrics, evaluation tools, experimental design, and validation procedures documented?
- Do tests cover realistic use, edge cases, foreseeable misuse, and conditions similar to deployment?
- Are validity and reliability demonstrated for the intended task? Are generalization limits, confidence limits, and failure modes clear to those who use or rely on outputs?
- Where relevant to the context, have safety, security, resilience, privacy, fairness, bias, transparency, explainability, and environmental impacts been evaluated?
- Are test results accompanied by plain-language limitations and residual risks, rather than presented as proof of performance outside the tested conditions?
Record the methods and conditions behind each result. A test conducted on a particular dataset or under controlled conditions does not, by itself, establish performance for every population or real-world deployment.
Quick wins for a faster PC:
Clear out junk files and repair common Windows errorsFree Scan →Fix the driver behind crashes, sound loss and screen glitchesFind Drivers →Repair Windows errors before they cause bigger problemsFix Now →4. Assess human oversight, affected people, and recourse
Determine whether people can understand, challenge, and appropriately act on AI outputs. A nominal human review is not meaningful oversight if the reviewer lacks authority, time, training, or information to question the system.
- Is a person accountable for consequential decisions, with the authority, time, training, and information needed to challenge an output?
- Are users told when AI is involved, what the system is intended to do, and where it may be unreliable?
- Can operators override or pause the system, or use a safe fallback when necessary?
- Can affected people contest an outcome, reach a responsible human, or report a problem?
- Are complaints, appeals, and feedback recorded and reviewed, and do they inform system evaluation and risk tracking?
- Where appropriate, did domain experts and affected groups help define relevant measures and interpret findings?
5. Review monitoring, incidents, and remediation
Deployment is not the end of the audit’s concern. Review how the organization detects changes in performance, risk, and actual use, and whether it can respond in time.
- Which production metrics and qualitative signals can reveal drift, errors, harmful bias, security problems, or changes in use?
- Who reviews those signals, how often, and against which thresholds or escalation criteria?
- Are there procedures for incidents, including containment, correction, rollback, reporting, and user notification where applicable?
- Do changes in data, model version, vendor, use, affected population, or operating environment trigger reassessment?
- Are risks tracked over time, including emerging risks that existing metrics may not capture?
- Is remediation assigned to named owners with evidence requirements, and does leadership explicitly accept or mitigate residual risk?
- Can the system be suspended, replaced, or decommissioned safely without creating new risks?
6. Write an audit report that distinguishes evidence from claims
A useful report enables decision makers to understand what was examined, what the findings mean in context, and what remains unresolved. Include:
- Scope, boundaries, and limitations of the audit.
- Criteria used to judge the evidence and the obligations considered.
- Evidence examined and tests performed, with methods and relevant conditions.
- Results and affected contexts, including uncertainty, control gaps, and residual risks.
- The rationale for the severity assigned to each finding.
- Management’s response, remediation owners and deadlines, and the follow-up method.
Separate verified evidence from management assertions. If an assessor did not independently test a claim, report it as an assertion rather than a tested result. NIST Special Publication 800-53A Revision 5 offers adaptable procedures for assessing security and privacy controls; it can inform planning and evidence analysis where applicable, but it is not, by itself, a complete AI audit framework.
How to compare AI audit approaches
An internal review, independent assessment, certification-related audit, and technical evaluation may address different questions. Compare the work on its actual scope and evidence rather than relying on its label.
- Does the scope match the system’s risk and intended use?
- Are the assessor’s competence and independence appropriate to the work?
- Did the assessor have access to relevant evidence and the system versions actually in use?
- Do data and population coverage, test design, and deployment conditions support the conclusions?
- Were relevant stakeholders involved, and are there workable routes for appeal?
- Does the work address post-deployment monitoring, remediation, and follow-up?
- Does it examine organizational controls as well as model performance?
NIST’s AI RMF emphasizes independent expertise, context-relevant measurement, documented test methods, and tracking risks over time. Those considerations help reveal what an assessment does and does not establish.
Independent reader supportYour contribution helps us test, update, and keep practical guides available for everyone.Which frameworks can help—and what they do not establish
NIST AI Risk Management Framework 1.0
The NIST AI RMF is a voluntary, non-sector-specific, use-case-agnostic reference organized around Govern, Map, Measure, and Manage. NIST’s current framework page states that version 1.0 is being revised; organizations should check the status of the framework when planning or updating an audit.
NIST AI RMF Playbook
The Playbook offers suggestions for achieving outcomes in the AI RMF Core. Its FAQ clarifies that it is not a checklist or an ordered implementation list: organizations can select actions suited to their context. Treat it as a source of ideas, not a mandatory audit sequence.
Free tools Windows power users keep installed
One-click scans. No signup required.
NIST SP 800-53A Revision 5
This publication provides adaptable procedures for assessing security and privacy controls. Use it where applicable to plan assessments and analyze evidence, while recognizing that security and privacy control assessment alone does not cover every part of an AI audit.
IIA AI Auditing Framework
The Institute of Internal Auditors’ AI Auditing Framework includes a practitioner guide and quick-start checklist for internal audit. The IIA recommends customizing the checklist to organizational considerations; it is a starting point for assessing how an organization approaches, uses, manages, and reports on AI.
NIST says more than 240 organizations from private industry, academia, civil society, and government contributed to the AI RMF’s open, multidisciplinary development. That breadth describes its development; it does not make the framework a certification or prove that an organization using it complies with a particular law.
Keep legal and compliance conclusions specific
This checklist does not establish that an organization complies with any particular statute or regulation. Legal and regulatory obligations depend on the jurisdiction, sector, system, use, and affected population. Have qualified legal or compliance owners map the obligations that apply to the specific deployment, then make that mapping and its validation part of the audit evidence.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
