A completeness gate can only check an answer against the requirements it is given. If the model being evaluated also invents the entire required set, it may omit a criterion from both its answer and the checklist—leaving the gate unable to detect the gap. Keep the required set explicit, scoped, versioned, and independently reviewed; use model-generated criteria as drafts, not as the final authority.
What a completeness score tells you—and what it cannot
A completeness score answers a conditional question: given a specified set of required questions or criteria, how many did the response satisfy? It does not establish that the set covers the task. A response can therefore score well against an incomplete checklist and still leave out something important.
As an Amazon Associate I earn from qualifying purchases.
Amazon Bedrock describes its judge-based metric this way: “Completeness (Builtin.Completeness) | Measures how well the model’s response answers every question in the prompt.” The documentation also notes that an evaluator considers a supplied reference response when one is included. The measure is about coverage relative to the prompt and available reference, not proof that every necessary requirement was identified. Amazon Bedrock evaluation metrics
Free tools Windows power users keep installed
One-click scans. No signup required.
Future AGI similarly frames completeness as checking whether a response covers what the input asked for and flagging gaps. That is a vendor description of a product metric, not independent evidence that a completeness score validates its own criteria. Future AGI’s completeness metric
#1 Best Overall
Who should decide what the answer must cover?
For a consequential evaluation, the required set should be an explicit input owned outside the model output being scored. Define the task scope first, then record the criteria, their source, and the version used for scoring. This makes it possible to distinguish two different failures: the answer missed an established requirement, or the requirement set itself failed to include something relevant.
- Declare the scope: specify the task, intended use, and which cases the gate covers.
- Maintain criteria independently: preserve who established each item and why it applies.
- Version the set: keep the criteria version fixed during a scoring run and record changes between runs.
- Apply relevant items: distinguish broadly applicable criteria from those that matter only to a particular task or study design.
Human-authored criteria are not automatically complete. They still need a declared scope and review. The important safeguard is independence from the answer being graded, together with traceability and a process for finding omissions.
Where model-generated checklists fit
A model can help draft or expand candidate criteria, but its list should not silently become the authority that defines completeness. Compare it with an independently established reference—such as human-authored criteria—or have qualified reviewers examine omissions and disagreements before the checklist controls a score or decision.
What’s actually slowing this PC down?
Pick the symptom - the matching free tool is one click away.
The study TICKing All the Boxes: Generated Checklists Improve LLM Evaluation and Generation describes generating instruction-specific checklists and comparing generated lists with human-written ones. Its ResearchGate preprint listing reports experiments involving multiple language models and trained annotators, with question-level accuracy judged against annotator majority vote. This supports comparison with human criteria as a validation approach; it does not establish a universal guarantee that a generated checklist is complete. Study listing for TICKing All the Boxes
In practice, retain disagreements rather than smoothing them away: a reviewer should decide whether a proposed item belongs in scope, whether an existing item is missing, or whether the model-generated item is redundant or inapplicable. The approved set—not the model’s unreviewed draft—then becomes the evaluation input.
Make the checklist fit the task
There is no universal checklist that fits every AI evaluation. Reporting frameworks illustrate why scope matters. TRIPOD-LLM, a reporting guideline for studies using large language models, comprises 19 main items and 50 subitems. Its authors distinguish items that apply broadly from those specific to research designs or task categories; these counts describe the reporting guideline, not the effectiveness of a completeness gate. TRIPOD-LLM reporting guideline
The Open Source Initiative’s checklist likewise specifies components relevant to openness and reproducibility, with required and optional components. It is scoped to a declared framework, not presented as a universal inventory for every AI system. Open Source Initiative checklist to evaluate machine learning systems
These examples point to a useful design choice: separate common requirements from conditional ones, and record when a task-specific criterion applies. Otherwise, a gate may either overlook a necessary item or penalize an answer for failing to address something irrelevant to its scope.
Best Value
Record enough setup to interpret the score
A score is difficult to interpret if the evaluation setup is not recorded. Preserve the criteria and reference answers alongside the tested system, prompt or harness, elicitation method, and validity checks. OpenAI’s guidance on third-party evaluations emphasizes standardized setup so that score differences can be attributed to systems rather than changes in measurement. It also recommends reporting evaluation content, the tested system, budget, elicitation methods, and validity checks such as checks for reward hacking or contamination. OpenAI’s playbook for trustworthy third-party evaluations
At minimum, an evaluation record should let another reviewer determine which set of criteria was applied, what system and harness produced the output, what reference material the judge could use, and what checks were made for misleading results. If any of those change, the score may no longer be directly comparable with earlier runs.
Quick Recap
A practical completeness-gate workflow
- Write down the task scope. Specify what the answer is expected to do and which contexts or task variants are included.
- Establish candidate requirements outside the scored output. Use a maintained human-authored set, a relevant reporting framework, or another independent reference appropriate to the task.
- Use a model only to propose additions or clarifications. Keep generated suggestions identifiable as proposals until reviewed.
- Resolve omissions and disagreements. Compare candidates against the independent reference, investigate differences, and approve or reject them with a record of the decision.
- Freeze and version the approved set for scoring. Keep criteria stable within the run; revise them through a documented update rather than silently changing the gate.
- Log the evaluation setup and inspect validity. Record the system, prompt or harness, references, and relevant validity checks so that the result can be interpreted and reproduced.
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
