The Tool Desk
Outbyte Driver Updater FREEScan for outdated or missing drivers - takes under a minuteDriver Scan →Outbyte PC Repair FREEClear out junk files and repair common Windows errorsFree Scan →AI providers should not be the sole judges of whether their own systems are trustworthy or safe, especially when the consequences of a wrong answer are serious. Their internal evaluations remain essential, and in practice they are currently more mature than independent testing. But an outside check can catch bias and conflicts of interest that an internal team may not see, or may not be able to disclose. The case is for independent scrutiny when the stakes, risks, or conflicts call for it, not for banning provider testing altogether.
In this article, “grading their own bills” means judging the safety and trustworthiness claims a provider makes about its own AI systems. The question is who checks those claims, and when.
As an Amazon Associate I earn from qualifying purchases.
What “self-grading” actually means in AI
When an AI company releases a model, it typically tests that model before and after release for accuracy, misuse potential, bias, and failure modes. The company then describes the results in documentation, a system card, or marketing material. Self-grading is the situation where the same organization that built the system designs the tests, runs them, interprets the results, and decides what to publish.
That arrangement is not unusual, and it is not inherently dishonest. The problem is structural. The party that gains from a favorable result also controls the process that produces it. Policy frameworks addressing AI accountability have started to recognize both the value of provider testing and the need for a second line of scrutiny.
#1 Best Overall
Why internal evaluation still matters
A provider knows things an outside tester usually does not. It has access to training choices, development data, architecture decisions, and the internal logs of earlier test runs. That context lets it design tests aimed at the risks that matter most for its particular system.
The National Telecommunications and Information Administration (NTIA) made this point directly in its Artificial Intelligence Accountability Policy Report (March 2024). It states that internal evaluations benefit from access to relevant material and are currently more mature and robust than independent evaluations. An argument that providers should be excluded from testing ignores that advantage. Any credible arrangement has to use it.
The conflict-of-interest problem
NIST’s AI Risk Management Framework 1.0 (2023) addresses the other side of the question. It states:
“Processes for independent review can improve the effectiveness of testing and can mitigate internal biases and potential conflicts of interest.”
Notice what the sentence claims and what it does not. It says independent review can improve testing, and it says independent review can mitigate bias and conflicts. It does not say that internal testing is worthless or that every system needs an outside auditor. The conflict is real even when the people running the tests are honest. A team that has spent a year building a product, reporting to leaders who need a launch date, may read ambiguous results more generously than an outsider would.
NTIA reached a compatible conclusion. It reports calls for independent evaluations where warranted, as a check on false claims and on risky AI, and it presents the two approaches as potentially complementary rather than as substitutes.
Internal and independent evaluation compared
The table below uses comparison axes drawn from the sources’ distinction between internal access and maturity on one hand and independent review’s role in mitigating conflicts on the other. It is an editorial framing, not a published scorecard. Where the sources do not quantify a factor, the table says so.
| Factor | Internal (provider) evaluation | Independent evaluation |
|---|---|---|
| Access to development data and system context | Strong. NTIA (March 2024) says internal evaluations benefit from access to relevant material. | Limited to what the provider shares. The sources reviewed do not describe typical access levels. |
| Independence from commercial incentives | Structurally limited. NIST (2023) identifies internal biases and potential conflicts of interest as the risks independent review can mitigate. | Designed to address this gap, though independence depends on the reviewer’s terms and funding. |
| Expertise and maturity of testing | NTIA (March 2024) describes internal evaluations as currently more mature and robust than independent ones. | NTIA describes independent evaluation as currently less mature in practice. NIST (2023) says independent review can improve testing effectiveness. |
| Transparency and ability to validate claims | Depends on what the provider discloses. The sources reviewed do not establish a standard level of disclosure. | An outside party can test claims without relying only on the provider’s account, but only to the extent it is given access. |
| Cost, timeliness, and scope | Not quantified in the sources reviewed. | Not quantified in the sources reviewed. Scope is set by the engagement terms. |
The pattern is clear enough to act on. Internal evaluation wins on context and maturity. Independent review wins on separation from the incentive to report a favorable result. Neither wins on every axis, which is why the policy question is about combining them.
Where the rules currently stand
Readers often assume that an independent check is already mandatory. For most AI systems in most places, that is not the case. The position depends on which framework applies and what role the company plays.
NIST AI Risk Management Framework: voluntary
NIST’s AI Risk Management Framework is voluntary. It is guidance for incorporating trustworthiness considerations into the design, development, use, and evaluation of AI. It is not a general legal mandate for third-party auditing. Organizations that adopt it decide how far to take its recommendations. NIST also operates an AI Resource Center with operational resources for testing, evaluation, verification, and validation.
EU AI Act Article 55: duties for systemic-risk models
Article 55 of the EU AI Act sets specific duties for providers of general-purpose AI models that present systemic risk. According to the European Commission’s AI Act Service Desk, which shows text based on the consolidated Act as of July 27, 2026, these duties include:
Free tools Windows power users keep installed
One-click scans. No signup required.
- Evaluating the model using state-of-the-art standardized protocols and tools.
- Conducting and documenting adversarial testing.
- Assessing and mitigating systemic risks.
- Reporting serious incidents.
- Maintaining cybersecurity protections.
These obligations are about evaluation and risk control. They do not automatically name an outside evaluator.
Recital 114: internal or external testing
Recital 114 of the same Act says that the necessary model evaluations may use internal or independent external testing. A reader should not take this to mean that independent external testing is required in every case. The Recital leaves room for provider testing, and it is the surrounding obligations and circumstances that determine how much outside scrutiny is appropriate. Legal requirements vary by provider role, model classification, and jurisdiction, so anyone relying on this text for compliance should check the current consolidated version.
When an independent check is warranted
Proportionality is the test. The following conditions are the ones where an outside review adds the most value. This is a reader-facing framework built from the reasoning in the sources, not a checklist any regulator has published.
- The consequences of error are severe. Systems that could cause physical harm, financial losses for many people, or security failures warrant more scrutiny than a writing assistant.
- The system presents systemic risk. Where Article 55 applies, the expectations for evaluation and risk controls are explicit.
- The provider has a strong stake in the result. A launch decision, a sales claim, or a regulatory filing that depends on passing a test creates the conflict NIST describes.
- The claims are hard for outsiders to verify. Vague or unverifiable assurances, such as “robust against misuse,” need an outside test before anyone relies on them.
- The system changes often. Frequent updates can outpace a single pre-release review.
Internal evaluation alone is more defensible when errors are low-stakes and easily reversed, when the system is used only internally, and when the provider publishes enough detail for others to check its methods.
Recommended Free Tools
How provider knowledge and outside scrutiny can work together
A practical sequence that reflects NTIA’s complementarity point and NIST’s independence rationale looks like this:
Best Value
- The provider runs its own evaluations with full development access and documents the methods, the scope, and the known limitations.
- It gives an independent reviewer the defined scope, the claims being made, and enough access to test them.
- The reviewer tests the claims and reports which ones hold, which do not, and which it could not verify.
- The provider publishes the results, including disagreements, and explains any changes it makes in response.
- The process repeats when the system changes materially, not only at launch.
The main risk in this arrangement is that the independent reviewer is given too little access to check anything meaningful. A review that only confirms the provider’s summary does little to address the conflict. The value of independent scrutiny depends on the scope and access written into the engagement.
Limits of the argument
The sources supporting this argument are principally policy documents and frameworks. They do not supply a measured effect size showing how much independent review reduces errors or bias, and this article does not claim one. The strength of the case rests on the structural conflict that NIST names, and on NTIA’s view that independent evaluation is currently less mature than internal testing. Both points cut in favor of combining the two approaches rather than choosing one.
The independent side also has its own risks, including reviewers whose own funding depends on the provider and tests that are less rigorous than the provider’s. Credibility comes from disclosed methods, defined access, and a willingness to publish findings that the provider would prefer to leave out.
The question is not whether AI companies can be trusted to test their own models. They can contribute important evidence, often better evidence than anyone else can produce. The question is whether their claims should be accepted without outside check when the stakes are high, and the answer is generally no.
If you are trying to judge a particular AI company’s claims, start with what the company has disclosed about its testing, who performed it, and whether any outside party was given access to verify the results.
Quick Recap
The Bottom Line
“”
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
