Choose metrics by starting with the decisions the data must support and the threats it must withstand—not by chasing one privacy or fidelity score. Evaluate task-specific utility and threat-informed disclosure risk together, set acceptance criteria before release, and document what the evaluation cannot establish.
Start with the intended use and release context
Before comparing synthetic data with its source, write down who will use it, what analyses or decisions it must support, and how users will access it. A public download, a query interface, and a protected enclave expose data differently; they may not require the same release approach or evidence.
As an Amazon Associate I earn from qualifying purchases.
Translate the intended use into concrete evaluation questions. For example: should a synthetic dataset support a particular regression, preserve a policy estimate, or let analysts test a predictive workflow? Which populations or outcomes matter most? The answer determines which utility metrics are relevant and which errors could change a decision.
Free tools Windows power users keep installed
One-click scans. No signup required.
There is no finite metric suite that proves validity for every possible future analysis. Claire McKay Bowen’s NIST article, “Utility Metrics for Differential Privacy: No One-Size-Fits-All” (November 29, 2021), notes that it is impossible to anticipate every analysis users may run. Its practical advice is to ask who the data practitioners are and what they will use the data and statistics for.
#1 Best Overall
Keep fidelity and task utility distinct
Fidelity describes how closely selected properties of synthetic data resemble properties of the source data. Task utility asks whether the synthetic data supports a particular analysis or decision adequately. These are related, but not interchangeable: matching averages or category frequencies does not establish that a regression, subgroup comparison, or policy estimate will remain useful.
Use a layered utility suite
- Univariate summaries: compare counts, means, rates, quantiles, missingness, and category frequencies that matter to users. Bias and root mean squared error can summarize differences for selected quantities.
- Relationships and distributions: compare correlations or joint distributions. NIST’s utility guidance gives chi-square tests for categorical variables and Kolmogorov–Smirnov tests for continuous variables as examples. Treat these as diagnostics, not universal pass/fail standards.
- Expected analyses: rerun representative stakeholder workflows—such as regressions, policy estimates, or predictive tasks—and assess whether results or decisions change materially.
- Discriminant metrics: train a classifier to distinguish real from synthetic rows. Weak discrimination suggests similarity on features that classifier can detect; the result depends on the model and does not establish either general fidelity or privacy.
- Subgroup and uncertainty checks: examine important populations separately and account for uncertainty introduced by synthesis. NIST SP 800-226, Guidelines for Evaluating Differential Privacy Guarantees (March 2025), identifies additional uncertainty and potential accuracy loss for subpopulations as issues to consider.
NIST’s November 2021 article discusses these measures from Census 2020 as examples, not as a standard bundle for enterprise data: mean absolute error, mean numeric error, root mean squared error, mean absolute percent error, coefficient of variation, total absolute error of shares, and counts of percent differences above selected thresholds. Select measures according to the variables and analyses that matter; do not treat a long metric list as a substitute for task-specific checks.
Rank #2
Define the privacy threat before choosing tests
Privacy evaluation should reflect the release context, plausible attackers, available auxiliary information, quasi-identifiers, and sensitive attributes. State what an attacker could know and what disclosure would count as harmful. A test only says something about the attack and assumptions it actually covers.
NIST SP 800-188, De-Identifying Government Datasets: Techniques and Governance (final, September 14, 2023), recommends setting measurable performance levels and using re-identification studies as part of governance. NIST SP 800-226 warns that non-differentially-private synthetic data generally has informal guarantees and is not robust against all privacy attacks. Passing selected empirical tests therefore cannot demonstrate zero disclosure risk.
Match disclosure tests to the threat model
| Test | What it examines | What to report |
|---|---|---|
| Replicated unique records or percentage replicated uniques | Whether synthetic rows match original unique records on selected quasi-identifiers. | The quasi-identifiers used, matching rule, and number or share of matches. |
| Apparent Match Distribution | Synthetic rows that exactly match unique real rows on quasi-identifiers, followed by comparison of sensitive or confidential attributes for those apparent matches. | The matching assumptions and the sensitive-attribute comparisons. |
| Count disclosure and percent disclosure | Replicated unique records considered “too close” on confidential variables under a chosen tolerance. | The tolerance and how it was selected; the tolerance is a design assumption, not an objective universal boundary. |
| Direct re-identification and partial-match exercises | Full and partial linkage, pairwise intersections, and other realistic attacks where appropriate. | Attacker knowledge, target selection, linkage information, and whether targeted attributes could be reconstructed. |
The NIST-hosted HLG-MOS synthetic-data materials describe these disclosure tests, including replicated uniques, apparent match distribution, and count or percent disclosure. They are useful evaluation patterns, not a guarantee that a dataset is safe in a different release setting.
Understand what differential privacy does—and does not—answer
Differential privacy (DP) provides a mathematical framework for quantifying privacy loss. If a generation method claims DP, report the guarantee along with the assumptions and implementation context needed to interpret it. Do not replace its formal privacy analysis with a similarity score or empirical attack result.
DP does not tell an organization whether a synthetic dataset preserves a desired analysis, whether the implementation is correct, or whether a particular utility trade-off is acceptable. Utility evaluation remains necessary, including checks for uncertainty and subgroup performance. Conversely, a synthetic dataset that looks statistically similar or passes selected attacks does not thereby acquire a formal DP guarantee. The HLG-MOS / NIST materials explicitly caution that training a synthesizer on data without formal DP does not make the resulting data formally differentially private.
What’s actually slowing this PC down?
Pick the symptom - the matching free tool is one click away.
Compare candidate methods across the same decision factors
Use a decision table to compare candidate generators or release approaches. The following factors synthesize NIST’s use-case-specific utility guidance, de-identification governance guidance, and synthetic-data evaluation resources; they do not imply a universal weighting scheme.
| Factor | Question for the evaluation |
|---|---|
| Intended analysis | Does the approach preserve the outcomes, estimates, or decisions data users need? |
| Privacy model | Is there a formal DP guarantee, or only empirical or informal evidence? What attacker and auxiliary information are assumed? |
| Sensitive subgroups | Are utility, error, or disclosure risks materially worse for small or vulnerable populations? |
| Uncertainty | How does synthesis affect variance and downstream inference? |
| Release model | Is public release necessary, or could a query interface or protected enclave meet the need? |
| Operational fit | Can the organization calculate, reproduce, govern, and explain the chosen metrics? |
NIST SP 800-188 describes synthetic-data publication alongside other sharing models, including protected enclaves and query interfaces. A release model that limits access may be a better fit than public distribution when the use case can be met without publishing a reusable dataset.
Set acceptance criteria through governance
Choose thresholds before evaluating or releasing a candidate, and tie each threshold to the use case, threat model, and risk tolerance. NIST’s guidance supports measurable performance levels and re-identification studies, but the cited sources do not establish a universal acceptable enterprise threshold or one metric bundle for every dataset.
- Record the use case: name user groups, intended analyses, important outcomes, and populations whose results must be assessed.
- Specify the release and threat model: state whether access is public, through a query interface, or within a protected environment; document plausible attacker knowledge and sensitive attributes.
- Select metrics and tolerances: choose distributional and task-utility measures, privacy tests, subgroup checks, and uncertainty analyses that directly address those conditions.
- Define decision rules: state what level of change in an analysis is acceptable, what disclosure findings trigger remediation, and who approves exceptions.
- Document limitations: report which attacks and analyses were tested, the assumptions behind them, and what was not evaluated.
Use tools as evaluation aids, not certification
NIST’s undated Collaborative Research Cycle page describes the SDNist Deidentified Data Report Generator as producing more than ten measures, including univariate and multivariate statistics, database distances, PCA, propensity, and basic privacy evaluation. The CRC also provides benchmark data. Those capabilities can help structure comparisons, but a generated report is not an automatic certification that data is safe for a specific release.
NIST SP 800-188 also cautions that tools that merely mask personal information may not provide sufficient functionality for de-identification, and that its tool list is not an endorsement. A tool’s output still needs interpretation against the organization’s intended use, threat assumptions, and acceptance criteria.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
