A reliable AI evaluation build needs two separate controls: an identity for the inputs you evaluated, and a scheduler that keeps parallel tests within declared resource capacity. A prompt hash can help identify an evaluation, but it does not make model output deterministic. CTest can schedule declared resource slots, but those slots are not a universal limit on process peak memory (RSS).
Why an AI evaluation build needs two separate controls
A passing CI result is useful only if you can tell what was evaluated and whether the runner had enough capacity to run the tests as intended. Those are related operational concerns, but they require different controls.
As an Amazon Associate I earn from qualifying purchases.
| Control | What it identifies or limits | What it does not guarantee |
|---|---|---|
| Evaluation manifest and hash | The exact, documented set of evaluation inputs represented by the manifest. | Identical model output, or a complete record if relevant inputs were omitted. |
| CTest resource allocation | Declared test resource slots, when a resource specification and test requirements are supplied. | A universal per-process or aggregate RSS ceiling. |
| Memory checker | Runs tests through a configured memory-checking tool. | By itself, a portable peak-RSS enforcement policy. |
Keeping these controls separate prevents two common mistakes: treating a prompt hash as a reproducibility guarantee and treating test scheduling as memory enforcement.
Recommended Free Tools
What should an evaluation hash identify?
Hash a defined evaluation identity, not just a prompt string
Evaluation systems contain more than prompt text. OpenAI’s Evals documentation describes evaluation criteria, data-source configuration, templated messages, graders, and runs; its documentation also gives prompt-version as an example metadata value. For an AI/C++ evaluation, record the inputs that can change the result, including:
- The prompt or message template, with a clear policy on whether the identity covers the source template or fully rendered messages.
- Variable values and any system or developer instructions that affect the tested input.
- Tool schemas or other context supplied to the model, when applicable.
- The evaluation dataset and its version.
- The grader, rubric, or scoring configuration and its version.
- The model snapshot and relevant parameters.
- The evaluation harness revision and build configuration.
These are project-level provenance recommendations based on the fields exposed by evaluation systems; OpenAI does not prescribe a canonical prompt-hashing protocol.
Define canonicalization before computing the hash
A hash is meaningful only if you can explain exactly which bytes it covers. Serialize a versioned manifest using a documented, stable representation—for example, with stable field ordering and explicit encodings—and hash those exact bytes. Include a manifest-schema or canonicalization version so a future representation change cannot silently be confused with an older one. Keep the readable manifest beside the hash so a reviewer can inspect what it represents.
Choose and document whether the hash represents a source template or the fully rendered messages, including variable values. If the evaluation also depends on data, graders, model settings, tool definitions, or harness code, preserve those identities separately in the manifest even if the project chooses not to include every field in one hash. The reviewed vendor documentation does not establish a standard set of fields or a canonicalization algorithm.
Do these 3 things before closing this tab:
1Scan for outdated or missing drivers - takes under a minute2Repair Windows errors before they cause bigger problems3Fix the driver behind crashes, sound loss and screen glitchesKeep prompt identity separate from model identity
A matching prompt hash does not mean that two runs used the same model or will produce the same answer. Store the model snapshot and parameters as distinct run metadata, along with dataset, grader, and harness versions. OpenAI notes that prompting behavior can vary between model snapshots, recommends pinned model versions where available, and recommends application evaluations for consistency. When a model version or evaluation input changes, rerun the application evaluation rather than assuming the old result still applies.
How CTest schedules declared resources
Describe capacity and test demand
CTest’s resource allocation feature is a cooperative scheduler. A resource specification file describes capacity on the runner, while each test’s RESOURCE_GROUPS property declares the slots it needs. When the feature is used, the CMake CTest documentation says CTest will not oversubscribe the configured resources.
The resource specification must reflect the machine or runner where the tests execute. CTest does not discover or manage GPU capacity for the project: the project or a generated resource specification must provide the capacity information. Tests should use the allocated-resource environment information to decide which resources to consume.
Activate allocation in the test invocation
Declaring a test’s resource requirements is not enough on its own. Supply the resource specification when invoking CTest, and ensure the test harness can tell whether allocation information is available. If the invocation omits the resource file, the harness must not assume that CTest has reserved resources for the test.
CTest reports a test that requests more slots than are available as not run. That is a scheduling outcome, not proof that the test passed or that its resource request was satisfied. Inspect the test result and invocation configuration in CI rather than treating every completed CTest run as evidence that allocation was active.
Fit declarations to actual test use
- Define the resource types and available slots for the runner in its resource specification.
- Declare each test’s resource-group needs with
RESOURCE_GROUPS. - Pass the matching resource specification to the CTest test step.
- Have tests consume the allocation information CTest provides instead of independently assuming a device or slot is free.
- Keep the specification, build configuration, test output, and evaluation manifest with the CI run record.
CTest also supports configure, build, and test steps in dashboard workflows, which can make those results part of a reported workflow. That reporting support does not define how a particular project retains CI artifacts; retention is a project or CI-platform decision.
Independent reader supportYour contribution helps us test, update, and keep practical guides available for everyone.Does CTest enforce an RSS limit?
No. CTest resource slots describe declared, schedulable resources; they are not documented as a universal per-process peak-RSS limit or an aggregate memory ceiling for a job or container. A test could request an abstract resource slot while still using more memory than expected, and the slot declaration alone does not measure or stop that use.
CTest’s memory-check step is a separate feature that runs tests through a memory checker. It should not be described as a built-in, cross-platform peak-RSS limiter. The official CTest material does not provide a universal recipe for sampling and enforcing RSS across platforms.
Choose an enforcement mechanism for the actual memory target
If CI must enforce a hard memory limit, select and verify a mechanism provided by the target runner, operating system, or container environment. First decide whether the limit applies to one process or the aggregate job/container. Then verify what the mechanism measures and how container accounting affects the reported reading. Document the runner-specific behavior: the available evidence does not establish one portable enforcement method.
Quick Recap
A practical CI workflow
- Define the evaluation manifest. Record prompt/template identity and rendering policy, dataset, grader or rubric, model snapshot and parameters, and harness revision. Version the manifest schema.
- Canonicalize and hash it. Specify the representation and exact bytes being hashed, retain the readable manifest, and store the hash with run metadata.
- Pin model versions where available. Rerun the application evaluation when the model version or evaluation inputs change; a stored hash identifies inputs, not output determinism.
- Declare test resource demand. Create a resource specification that matches the runner and add each test’s
RESOURCE_GROUPSrequirements. - Enable allocation at test time. Confirm the CTest invocation passes the resource specification and that tests use allocation information only when it is active.
- Enforce memory separately if required. Configure and verify the runner’s actual process or aggregate-memory mechanism; do not substitute resource slots or memory-check integration for that enforcement.
- Preserve the run record. Keep the evaluation identity, model and harness versions, resource specification, build configuration, and test output together in the CI record or artifacts.
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
