When evaluation is limited to a fixed number of records, building a fair test set across several attributes is a joint optimization problem—not a matter of balancing one column at a time. A subset can be optimized to match explicit group targets, but that makes it optimal only for the chosen targets and objective; it does not automatically make the set representative, intersectionally balanced, or suitable for every statistical inference.
Why choosing an evaluation subset is a joint problem
Suppose a pool contains many records, but the evaluation budget allows only a fixed number to be scored. Each record may belong to a category for several attributes: for example, sex, race, income class, and age band. Selecting one record changes all of those attribute counts at once.
As an Amazon Associate I earn from qualifying purchases.
Balancing sex first, then race, then age can undo earlier work. Treating every combination as its own stratum avoids that sequential conflict in principle, but the number of combinations grows quickly and many may have few or no records. In the Adult dataset example described by Vasileios Vonikakis in an article dated September 29, 2026, two sex categories, five race categories, two income classes, and ten age bins create 200 joint strata.
This is why the task is combinatorial: the curator is choosing a fixed-size subset from many possible subsets while trying to satisfy several targets at once.
#1 Best Overall
How target-based subset optimization works
Represent each candidate record with a binary inclusion variable: xᵢ = 1 if record i is selected, and xᵢ = 0 otherwise. For a fixed evaluation budget B, require that the selected indicators sum to the budget:
Σᵢ xᵢ = B
For each attribute category or bin, compare the selected count with its target count. Slack variables can represent the amount by which a selected count falls short of or exceeds a target. The optimizer then minimizes an aggregate measure of those deviations. The article also describes an optional objective term intended to reduce correlations between attributes.
The formulation matters. A different deviation measure, different target counts, or different weighting of attributes can produce a different subset. “Optimal” therefore means optimal for the specified formulation—not universally fair. A solver may prove that it found the optimum for that formulation, or it may return its best feasible subset when a time limit is reached without proving optimality.
Do these 3 things before closing this tab:
1Clear out junk files and repair common Windows errors2Fix the driver behind crashes, sound loss and screen glitches3Repair Windows errors before they cause bigger problemsWhat the targets might look like
For illustration, Vonikakis describes a fixed budget of 1,000 evaluations with targets of a 50/50 split by sex, equal representation across five race categories, a 50/50 split by income class, and a flat allocation across ten age bins. Those targets imply 500 records in each sex category, 200 in each race category, 500 in each income class, and 100 in each age bin. They are marginal targets: they do not specify how every race-and-age or sex-and-income combination should be represented.
In a real pool, all targets may not be jointly attainable. If counts conflict or the pool lacks enough records in a category, the optimizer must trade off deviations according to its objective; it cannot make unavailable records appear. Record which targets were met, which were missed, and why.
Decide what the evaluation is meant to estimate
There is no single composition that is best for every evaluation. A more evenly group-balanced set can make group comparisons easier to interpret because the groups have more comparable sample sizes. A set that follows the expected deployment population is better aligned with an aggregate-performance question about that population. These are different estimands, so a single test set should not be expected to answer both equally well.
For example, Vonikakis gives an illustrative arithmetic case in which group A has 95% accuracy, group B has 60%, and the test set is composed of 90% A and 10% B. The weighted overall accuracy is 91.5%. That number reflects both group performance and the test-set composition. It is an example, not an empirical study, and it shows why an aggregate score can conceal how results differ across groups.
The Tool Desk
Outbyte PC Repair FREEClear out junk files and repair common Windows errorsFree Scan →Outbyte Driver Updater FREEFix the driver behind crashes, sound loss and screen glitchesFind Drivers →A careful evaluation program can preserve a deployment-mix set for the aggregate question and report disaggregated results, while also using a more balanced set when group comparisons are a priority. State which question each reported result answers rather than treating one composition as neutral.
What marginal balance does—and does not—guarantee
Matching separate attribute histograms does not ensure that their intersections are balanced. A set can meet its target counts for sex, race, income, and age while still having very different counts in particular combinations, such as a specific race category within an age band. Inspect cross-tabs for combinations that matter to the evaluation. If those intersections are important and the pool supports them, encode their targets explicitly rather than assuming marginal balance will provide them.
- Check coverage before optimizing. Identify categories and intersections that are absent or sparse in the source pool. A selection procedure cannot repair a coverage hole.
- Review selected records within groups. Matching category counts does not ensure the selected records are typical of all records in those categories.
- Document the bin definitions. Changing age bands or other category boundaries changes the optimization problem and potentially the selected records.
- Report unmet targets. Distinguish an infeasible target from a target the solver did not meet because of its objective or time limit.
Evaluation selection is not the same as training on balanced data
A balanced training subset changes the data used to fit a model. A balanced evaluation subset changes the evidence used to assess it. Those choices answer different questions: training composition can affect what the model learns, while evaluation composition affects which cases and group proportions contribute to the reported results. The evaluation-set method described here is about selecting records for a fixed-budget assessment; it should not be treated as a claim that training on the same subset would improve model quality.
Likewise, macro-averaging changes how group-level results are weighted in a reported metric on an already labeled set. It can make a summary less dominated by a large group, but it does not create additional observations for an underrepresented group when the number of cases available for evaluation is limited.
How this differs from other approaches
| Approach | What it does | Key limitation or distinction |
|---|---|---|
| Joint target optimization (including the datacarve method described in the article) | Selects a fixed-size subset of real records while minimizing deviations from explicit targets across multiple attributes. | Marginal targets do not automatically balance every intersection. Results depend on the chosen targets and objective; optimality is only with respect to that formulation. |
| Cube probability sampling | Uses probability sampling with known inclusion probabilities, described as an alternative when design-based inference is central. | Balance is approximate when constraints cannot all be met exactly. Its probability-sampling properties differ from deterministic target-shaped selection. |
| Macro-averaging | Changes the weights given to group results in a reported metric. | It does not add observations to sparse groups or select a larger evaluation sample. |
| One-way stratification | Balances a single attribute by selecting from its categories. | Balancing several attributes separately can conflict; making every combination a separate stratum can create many sparse strata. |
Choose among these based on the quantity you need to estimate, whether known inclusion probabilities are necessary, whether you need a fixed-size set of real records, how important simultaneous targets are, and how sparse the relevant intersections are. A deterministic subset shaped to targets does not automatically have the inclusion-probability properties needed for design-based inference.
Best Value
How many records does each group need?
Balanced counts make group comparisons less dominated by unequal sample sizes, but balance alone does not guarantee that a study can detect a meaningful difference. Required group counts depend on the smallest gap worth detecting and on the study’s statistical assumptions. Calculate power for the specific evaluation rather than treating a general rule of thumb as a substitute.
Vonikakis gives an approximate two-group detectability rule of about a 6-percentage-point gap with 200 records per group at around 90% accuracy, and says that quadrupling group size roughly halves the gap. These are the author’s approximations, not a study-specific power analysis; use them only as an indication that smaller differences generally require substantially more observations.
A practical workflow for a fixed evaluation budget
- Define the evaluation question. Decide whether the primary result should reflect an expected deployment mix, support comparisons across groups, or serve both purposes through separate sets or reporting.
- Specify the budget and categories. Set the total number of evaluations and define the attribute bins. Write down target counts for each bin and any intersections that are important.
- Check the pool’s support. Count available records in every targeted category and relevant intersection. Revise infeasible targets or collect more data rather than implying that selection can fill missing groups.
- Choose and document the objective. State how deviations are measured and combined, whether any correlation-reduction term is used, and which targets take priority if they conflict.
- Run the selection and record its status. Note the selected counts, deviations from targets, and whether the solver proved optimality or stopped with a feasible result at a time limit.
- Audit the selected set. Review marginal counts, cross-tabs, and within-group coverage. If selection is randomized, preserve enough information to describe that procedure.
- Plan the analysis. Report group-specific results and uncertainty, and calculate whether the sample can detect the smallest gap that matters for the intended decision.
What the reported runtime does—and does not—show
For the 48,842-row Adult example, Vonikakis reports that the run took about three seconds on the author’s laptop, with 48,842 binary inclusion decisions. This is an author-reported example, not an independently tested performance guarantee. Runtime depends on the formulation, solver, hardware, and data; the figure should not be generalized to other datasets or target structures.
Free tools Windows power users keep installed
One-click scans. No signup required.
The article describes datacarve as an open-source Python library and links to its repository, PyPI, and example notebooks, but those links and current package details are not available here. Treat the runtime as context for the reported example, not as a basis for choosing a package or predicting performance on a different workload.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
