Free tools Windows power users keep installed
One-click scans. No signup required.
You cannot know a model’s actual accuracy on current, unlabeled data from its predictions alone. You can estimate performance by combining unlabeled predictions with labeled source or validation data and an explicit assumption about how the target data has changed. Treat the result as an assumption-dependent estimate—not observed accuracy—and use a representative labeled audit when the decision matters.
What “accuracy right now” means
Accuracy is the share of examples in a defined population for which the model’s prediction matches the true label. To calculate it, you need both predictions and ground-truth labels for that population. On an unlabeled deployment stream, the predictions are available but correctness is not, so the current accuracy cannot be directly calculated.
As an Amazon Associate I earn from qualifying purchases.
First define the metric and population you care about. Accuracy is not interchangeable with precision, recall, F1, calibration error, or business utility. A model can have well-calibrated confidence scores but low accuracy, or high accuracy while performing poorly on a consequential minority class. Also specify the time window, user or data population, and prediction task: an estimate for one slice of traffic may not describe another.
Do these 3 things before closing this tab:
1Clear out junk files and repair common Windows errors2Fix the driver behind crashes, sound loss and screen glitches3Repair Windows errors before they cause bigger problemsWhat unlabeled data can—and cannot—tell you
Unlabeled target examples can reveal prediction patterns, confidence scores, and changes in the inputs or outputs. They do not reveal which individual predictions are correct. To turn those signals into a performance estimate, a method needs additional structure, such as labeled source data, assumptions about the kind of distribution shift, or outputs from multiple predictors.
#1 Best Overall
Drift and accuracy are different quantities. A detected change in inputs is a reason to investigate, not a measurement of how many predictions are wrong. Conversely, a model may lose accuracy even when simple input-drift checks show little change—for example, if the relationship between inputs and labels changes. No unlabeled estimate can remove the need to check whether its assumptions fit the deployment.
Choose a method that matches the likely shift
Before selecting a method, identify what labeled data remains available and what kind of change is plausible. In label shift, the proportions of classes change while the distribution of inputs within each class stays fixed. In covariate shift, the input distribution changes while the relationship between inputs and labels is assumed to remain stable. A shift outside those assumptions, such as a changed meaning of the task or a new population, can make an estimate unreliable.
Rank #2
- Use scikit-learn to track an example ML project end to end
- Explore several models, including support vector machines, decision trees, random forests, and ensemble methods
- Exploit unsupervised learning techniques such as dimensionality reduction, clustering, and anomaly detection
- Dive into neural net architectures, including convolutional nets, recurrent nets, generative adversarial networks, autoencoders, diffusion models, and transformers
- Use TensorFlow and Keras to build and train neural nets for computer vision, natural language processing, generative models, and deep reinforcement learning
| Method | Assumption or scope | Data needed beyond unlabeled target examples | Estimated quantity | Main caveat |
|---|---|---|---|---|
| Bayesian or multi-predictor estimation | Uses a graphical model of classifier outputs; can combine multiple classifiers. | Outputs from predictors; method-specific structure and assumptions. | Classifier accuracy. | Reported results are specific to the tested datasets and setup. |
| Black Box Shift Estimation (BBSE) | Label shift: the label marginal changes while the class-conditional input distribution remains fixed. | A predictor’s confusion matrix estimated using labeled source data, plus its predictions on target examples. | Target label proportions directly; performance can be derived when combined with the relevant source performance information. | Its stated guarantees depend on an invertible confusion matrix and the label-shift assumption. |
| Average Thresholded Confidence (ATC) | Transfers a confidence threshold learned on source data to the target data. | Labeled source data to fit the scalar threshold, plus target confidence scores. | Target performance estimate from the share of target examples above the threshold. | How well the source-fitted threshold transfers depends on the deployment and shift. |
| LaSCal | Label shift. | Source information and unlabeled target data, as required by the method. | Calibration error, not accuracy itself; the estimate can inform post-hoc calibration on the target distribution. | Calibration error measures confidence alignment with accuracy and answers a different question from accuracy. |
| PAPE | Covariate shift for binary classification on unlabeled tabular data. | Unlabeled tabular target data and the method’s required source information. | Performance metrics defined from confusion-matrix elements. | The paper’s scope is specific to binary tabular classification under covariate shift. |
| Prediction intervals for performance estimates | Transfer-learning approach evaluated across drift conditions. | Method-specific source and target information. | An uncertainty interval around a performance estimate. | An interval does not fix a mismatch between the method’s assumptions and the deployment. |
Label shift: BBSE
Lipton, Wang, and Smola’s 2018 BBSE method uses a predictor’s source confusion matrix to estimate target class proportions. With those proportions and suitable source performance information, it can support a target accuracy estimate. BBSE does not require the predictor to be accurate, calibrated, or unbiased, but the confusion matrix must be invertible for the stated guarantees, and the label-shift assumption must be credible. A weak predictor can make the estimate less useful; the paper notes that better predictors can tighten it.
Confidence-based transfer: ATC
Garg and coauthors’ Average Thresholded Confidence (ATC) method learns a scalar confidence threshold using labeled source data, then estimates target performance by counting the share of unlabeled target examples above that threshold. Confidence is not correctness: ATC turns confidence into an estimate through its source-fitted threshold and transfer assumption, not by treating every confident prediction as true.
Rank #3
The ICLR 2022 paper reports ATC estimated performance up to 2–3 times more accurately than recently proposed methods across the architectures, shifts, and datasets it tested. That is a paper-specific comparison, not a general guarantee for a new model or deployment.
Covariate shift: PAPE
The NeurIPS 2025 PAPE paper addresses performance estimation for binary classification on unlabeled tabular data under covariate shift. It describes applicability to metrics defined from confusion-matrix elements and reports an evaluation over more than 900 dataset-model combinations using US census data. Those results describe the paper’s stated task and evaluation; they do not establish that PAPE applies to other data types, multiclass tasks, or arbitrary kinds of drift.
Rank #4
Multiple predictors and earlier frameworks
Platanios, Dubey, and Mitchell’s 2016 work uses a graphical model and nonparametric extensions to estimate classifier accuracy from unlabeled data, including by combining outputs from several classifiers. Its experiments on two real-world datasets reported estimates within a few percent of true accuracy. That finding belongs to those datasets and experimental conditions, not to every model ensemble or production setting.
Donmez, Lebanon, and Balasubramanian proposed an earlier general framework for estimating classification and regression errors from unlabeled data under mild assumptions, with consistency results. “General” does not mean assumption-free: the usefulness of any such estimate still depends on whether its conditions are appropriate for the data at hand.
Best Value
Calibration is related, but not accuracy
LaSCal estimates calibration error without target labels under label shift and uses that estimate with post-hoc calibration on the target distribution. Calibration concerns whether confidence scores align with observed correctness rates. It can matter when a system relies on probabilities, but a calibration-error estimate should not be reported as the model’s accuracy.
Independent reader supportYour contribution helps us test, update, and keep practical guides available for everyone.A practical workflow for estimating performance
- Define the question. Write down the metric, prediction task, target population, and time window. If the decision depends on a class-specific outcome, do not substitute overall accuracy for that measure.
- Inventory the labels and outputs you have. Identify labeled source or validation examples, whether their population resembles deployment, which model predictions or confidence scores are available, and whether outputs from multiple predictors can be compared.
- State the shift hypothesis. Decide whether label shift or covariate shift is plausible, or whether you have no defensible shift model. Use a method only within its stated scope; for example, PAPE’s described scope is binary tabular classification under covariate shift.
- Choose the corresponding estimator. For label-shift prevalence estimation, consider BBSE; for a source-fitted confidence threshold, consider ATC; for multiple predictors, consider a method designed to use their outputs. Do not choose solely because a paper reported a favorable result on different data.
- Report the estimate with its conditions. State the method, labeled source data used, target population and period, assumed shift, and uncertainty. If you cannot justify the assumption, describe the output as a diagnostic signal rather than a reliable accuracy estimate.
- Check against labels when feasible. Label a representative sample from the target population and compare predictions with ground truth. Use the result to verify or recalibrate the estimate; if decisions have serious consequences, do not rely on an unlabeled estimate alone.
Keep uncertainty visible
A point estimate can look more certain than the evidence warrants. Performance-estimation intervals have been studied, including a transfer-learning approach by Elder and coauthors evaluated across drift conditions. Where the chosen method supports an interval, report it alongside the estimate and explain what it covers. An interval communicates estimation uncertainty under the method; it does not guarantee protection against a wrong shift assumption, an unrepresentative source set, or a changed task.
Published headline results should be read with the same care. Platanios and coauthors’ “within a few percent” finding applies to experiments on two real-world datasets. ATC’s “2–3× more accurately” comparison is from its tested settings. PAPE’s evaluation covered more than 900 dataset-model combinations using US census data. None of these figures promises the same result for a different deployment.
The Tool Desk
Outbyte PC Repair FREEClear out junk files and repair common Windows errorsFree Scan →Outbyte Driver Updater FREEScan for outdated or missing drivers - takes under a minuteDriver Scan →When a labeled audit is the right next step
If the likely shift is unknown, the model serves a new population, the label meaning may have changed, or the estimate would support a high-consequence decision, treat unlabeled methods as triage rather than proof. Collect a carefully selected sample from the current target population, obtain reliable labels, and calculate the metric directly. Sampling should reflect the population and time period for which you intend to make a claim; a convenient but unrepresentative batch can create a misleading check.
Where labels arrive with a delay, a recurring target sample can provide a ground-truth check once those labels become available. In the meantime, track unlabeled estimates as estimates, preserve their assumptions and uncertainty, and investigate meaningful changes rather than presenting drift as a measured accuracy loss.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
