Free tools Windows power users keep installed
One-click scans. No signup required.
Evaluating Machine Learning Models by Alice Zheng is a concise, concept-first guide to deciding whether a machine-learning system is working. Its central lesson is to define project-specific success before choosing a metric, validation scheme, or tuning strategy. First published by O’Reilly Media on September 1, 2015, it remains useful as a foundation, but readers should supplement it with current software documentation and newer evaluation guidance.
Book at a glance
| Detail | Information |
|---|---|
| Author | Alice Zheng |
| Publisher | O’Reilly Media |
| First release | September 1, 2015 |
| ISBN | 9781492048756 |
| Level | Intermediate to advanced; also presented as an introduction for readers new to data science and applied machine learning |
| Catalog length | 20 pages |
| Catalog reading time | 1 hour 20 minutes |
The page count and reading time are O’Reilly catalog figures, not independent measurements. The book grew from six technical posts on the Dato Machine Learning Blog, according to Zheng’s preface.
As an Amazon Associate I earn from qualifying purchases.
The question that should come before every metric
Zheng attributes two questions to advice from her machine-learning mentors: “How can I measure success for this project?” and “How would I know when I’ve succeeded?” Those questions prevent a common mistake: treating a familiar score as proof that a model is useful.
Start by specifying the decision the model supports, the errors that matter most, the population and time period represented by the data, and an operational threshold for success. A fraud model, a search-ranking system, and a demand forecast can all be evaluated on the same dataset yet require different definitions of acceptable performance.
#1 Best Overall
What the book covers
Classification
The classification material includes accuracy, confusion matrices, per-class accuracy, log-loss, and area under the ROC curve (AUC). It also flags imbalanced classes. In an imbalanced problem, overall accuracy can conceal poor performance on the rare class, so class-level results and the cost of false positives versus false negatives need explicit attention.
Ranking
For ranking systems, the outline covers precision-recall measures, F1, and normalized discounted cumulative gain (NDCG). These metrics address different questions: how many retrieved items are relevant, how completely relevant items are found, how precision and recall balance at a chosen point, and whether highly relevant results appear near the top.
Regression
Regression chapters discuss root mean squared error (RMSE) and error quantiles. RMSE gives larger errors extra influence because errors are squared; quantiles show how error is distributed, which is often more informative when a typical case and a worst-case case have different business consequences.
What’s actually slowing this PC down?
Pick the symptom - the matching free tool is one click away.
Difficult data conditions
The contents specifically call out outliers and rare data. These conditions can change both the metric and the validation result, making it important to inspect the data-generating process rather than relying on a single aggregate number.
Validation, testing, and tuning are different jobs
A key distinction in Zheng’s preface is that cross-validation and hold-out validation estimate performance on unseen data, while hyperparameter tuning is a model-selection process. They are related, but asking one to perform the other’s job can produce an overoptimistic estimate.
| Activity | Primary question | Methods named in the book |
|---|---|---|
| Validation | How might this fitted model perform on unseen data? | Hold-out validation, cross-validation |
| Uncertainty estimation | How variable is the estimate or statistic? | Bootstrapping, jackknife |
| Testing | What is the final, protected estimate after choices are made? | A separate test set, in the book’s validation/testing discussion |
| Model selection | Which parameters or hyperparameters should be chosen? | Grid search, random search, other tuning approaches, nested cross-validation |
For example, repeatedly selecting the best hyperparameters against one validation fold gradually adapts decisions to that fold. Nested cross-validation addresses this by separating the loop used for selection from the loop used for estimating generalization.
Rank #3
Offline evaluation methods in context
Hold-out validation
Reserve part of the data for evaluation and train on the remainder. It is simple and computationally light, but the estimate can depend heavily on one split, especially with small or rare datasets.
Cross-validation
Rotate validation folds so that multiple partitions contribute to the estimate. This uses data more efficiently than a single split, but it still requires a split strategy that respects grouping, time order, and leakage risks in the project.
Bootstrapping and jackknife
These resampling approaches help study the variability of an estimate. They do not automatically repair a flawed target, contaminated split, or unrepresentative dataset.
Independent reader supportYour contribution helps us test, update, and keep practical guides available for everyone.Online tests add different risks
The book’s online-evaluation section covers A/B-testing pitfalls: choosing the metric, obtaining enough observations, false positives, repeated hypotheses, test duration, and distribution drift. A statistically detectable change is not automatically a valuable product change, and repeatedly checking or trying many outcomes increases the chance of a misleading positive result.
Multi-armed bandits are discussed as an alternative when an experiment must balance learning with allocating traffic. The choice depends on whether the primary need is a clean, fixed comparison or adaptive allocation during deployment.
Quick wins for a faster PC:
Clear out junk files and repair common Windows errorsFree Scan →Scan for outdated or missing drivers - takes under a minuteDriver Scan →A practical reading and application sequence
- Define success. Write the decision, target population, error costs, and a pass/fail threshold.
- Identify the task. Choose the classification, ranking, or regression metric family that matches the decision.
- Design the split. Select hold-out or cross-validation, while accounting for time, groups, rare cases, and leakage.
- Separate selection from estimation. Tune hyperparameters without treating the tuning result as an untouched final test.
- Inspect more than one number. Review confusion matrices, per-class behavior, error quantiles, ranking depth, and uncertainty where relevant.
- Plan online measurement separately. For an A/B test or bandit, predefine metrics, duration, sample-size logic, and how repeated looks at the data will be handled.
Who should read it today?
The book suits a learner who needs a compact map of evaluation concepts or a practitioner who wants a checklist for connecting metrics to project goals. It is not a current survey of machine-learning libraries, production monitoring, fairness evaluation, or newer experimental practice. Because it is a 2015 first edition, verify contemporary procedures and tool-specific behavior against current documentation before applying its examples.
For further reading, search for Evaluating Machine Learning Models Alice Zheng book. O’Reilly identifies the title and ISBN above; current retailer stock, formats, prices, and affiliate availability are not established here.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
