A Comprehensive Guide to Hypothesis Testing: Methods, Examples, and Interpretation

Some links on this page are affiliate links: if you buy through them we may earn a commission, at no extra cost to you.

Hypothesis testing uses sample data to assess whether results are sufficiently inconsistent with a specified null model. It can help distinguish signal from sampling variability, but it does not prove a claim, calculate the probability that a hypothesis is true, or replace a careful study design. A sound conclusion pairs the test result with an effect estimate, confidence interval, assumptions, and the practical stakes of the question.

What hypothesis testing does—and does not do

A researcher usually wants to learn about a population or data-generating process but can observe only a sample. Hypothesis testing provides a formal way to compare the observed data with a model representing a null claim. The test asks how unusual the observed result, or a more extreme result, would be if that null model and the test’s assumptions were correct. See Penn State’s explanation of hypothesis testing and p-values.

#	Preview	Product	Price
1		Introduction to Robust Estimation and Hypothesis Testing (Statistical Modeling and Decision Science)	$59.90	Buy on Amazon
2		Testing Statistical Hypotheses (Springer Texts in Statistics)	$79.99	Buy on Amazon
3		Introduction to Robust Estimation and Hypothesis Testing (Statistical Modeling and Decision Science)	$77.91	Buy on Amazon
4		Randomization Tests (Statistics: A Series of Textbooks and Monographs)	$140.00	Buy on Amazon
5		The Phylogenetic Handbook: A Practical Approach to Phylogenetic Analysis and Hypothesis Testing	$60.12	Buy on Amazon

A test is not a substitute for estimating how large an effect is, checking how the data were collected, or deciding whether the effect matters. It cannot fix biased sampling, confounding, measurement error, data leakage, dependence that the model ignores, or an analysis chosen after looking for favorable results.

• Estimation asks how large a parameter or effect may be.
• A confidence interval shows values compatible with the data under a particular procedure and its assumptions.
• Prediction concerns likely future observations.
• Decision analysis asks whether an effect justifies an action, given benefits, risks, and costs.
• Bayesian inference combines a probability model, prior information, and data to produce a posterior distribution. It answers a different question from a classical p-value.

Essential terms

• Population: the full group or process the research question concerns.
• Sample: the observations actually collected.
• Parameter: a population quantity, such as a mean, proportion, or correlation.
• Statistic: a quantity calculated from the sample, such as a sample mean.
• Null hypothesis (H₀): the specified reference claim, often no difference or no association.
• Alternative hypothesis (H_a or H₁): the effect or pattern the analysis is designed to detect.
• Test statistic: a number summarizing how far the observed result is from what the null predicts, usually measured relative to a standard error.
• Standard error: an estimate of the sampling variability of a statistic or estimate.
• Significance level (α): the prespecified Type I error rate for a test or testing procedure.
• P-value: under the null model and assumptions, the probability of a test statistic at least as extreme as the one observed, in the direction specified by the alternative.
• Critical region: the set of test-statistic values that leads to rejecting the null under the chosen rule.
• Type I error: rejecting a true null hypothesis.
• Type II error: failing to reject the null for a specified alternative that is true.
• Power: the probability of rejecting the null for a particular specified alternative; it is 1 − β, where β is the Type II error probability.
• Effect size: a measure of the magnitude of an effect, expressed in the outcome’s units or on a standardized scale.
• Degrees of freedom: a quantity used to determine a test’s reference distribution, reflecting the information available after estimating parameters or constraints.
• One-sided test: evaluates an alternative in one prespecified direction.
• Two-sided test: evaluates departures in either direction.

NIST describes α as the risk of rejecting a true null and power as the probability of rejecting the null under a specified alternative. Both depend on the design and analysis, not just on a software command. See the NIST overview of hypothesis testing, errors, and power.

#1 Best Overall

Sale

Introduction to Robust Estimation and Hypothesis Testing (Statistical Modeling and Decision Science)

$59.90

Buy on Amazon

State the hypotheses before examining the result

For a comparison of average productivity under a new and existing training program, define the outcome, population, and unit of analysis first. A two-sided question—whether the programs differ—can be written:

H₀: μ_new − μ_old = 0
H_a: μ_new − μ_old ≠ 0

If the genuine research question is whether the new program increases the mean, the alternative might instead be μ_new − μ_old > 0. Choose a one-sided alternative before seeing the data. Switching from two-sided to one-sided after observing which direction looks favorable changes the error properties and exaggerates the evidence.

The null is not always “nothing exists.” For example, a null for a correlation test may specify ρ = 0, while a noninferiority study tests whether a new treatment is no worse than a comparator by more than a prespecified margin. The hypothesis must match the scientific or operational question.

A practical workflow

1. Define the question and estimand. Identify the population, outcome, comparison, unit of analysis, and effect quantity of interest. Decide what effect would matter in practice, not merely what effect a test might detect.
2. Write H₀ and H_a. Specify the null value and whether the alternative is one- or two-sided. Set direction in advance.
3. Choose α before analysis. Values such as 0.10, 0.05, and 0.01 are common, but 0.05 is a convention, not a universal rule. Consider the consequences of false positives and false negatives, regulatory or disciplinary requirements, multiplicity, and whether the work is confirmatory or exploratory. NIST notes that the choice is partly arbitrary and that these values are frequently used (NIST).
4. Select a test that matches the design and data. Consider outcome type, number of groups, pairing or clustering, repeated measurements, covariates, sample size, and plausible assumptions. The design—not a list of test names—should lead the choice.
5. Check the relevant assumptions and data quality. Examine independence, sampling or assignment, the correct analysis unit, residual behavior, variance structure, cell counts, influential observations, missingness, and model form as applicable.
6. Calculate the estimate, test statistic, and p-value. A common form is (estimate − null value) / standard error. For a one-sample t-test, t = (x̄ − μ₀)/(s/√n), with n − 1 degrees of freedom. NIST provides the one-sample t-test formula.
7. Apply the prespecified decision rule. If p ≤ α, reject the null under that test and framework. If p > α, fail to reject it. Do not describe the latter as accepting or proving the null.
8. Interpret the result in context. Report the estimate, interval, p-value, sample size, effect magnitude, practical or clinical threshold, assumptions, and limitations. A statistical decision alone is not the substantive conclusion.

How to interpret a p-value

A p-value is conditional on the null hypothesis and the model. If the null and assumptions were true, it measures how often the test would produce a result at least as extreme as the observed result. A small value indicates that the data are relatively unusual under that null model; it does not measure the probability that the null is true. Penn State provides a fuller explanation of p-values and decisions. P-values can also be misleading when the data-generating process, analysis plan, or model is not credible; see Greenland and colleagues on common statistical misinterpretations.

A p-value does not tell you:

• the probability that H₀ is true;
• the probability that the finding happened “by chance”;
• the probability that the result will replicate;
• the size or practical importance of the effect;
• that a treatment works because p < 0.05; or
• that no effect exists because p > 0.05.

For example, if a prespecified test gives p = 0.03 at α = 0.05, reject the null under that test. Do not translate this into “there is a 97% chance the alternative is true.” If p = 0.18, fail to reject the null; that result could reflect a small effect, noisy data, an underpowered study, or no effect. The estimate and interval help distinguish what remains plausible.

Report the exact p-value to a sensible precision rather than rounding it to zero. Avoid treating a threshold as a cliff: p = 0.049 and p = 0.051 do not represent radically different evidence. “Statistically significant” should always be understood relative to a stated test, significance level, model, and multiplicity plan.

Confidence intervals and effect sizes belong beside the test

A confidence interval is produced by a procedure with a stated long-run coverage property when its assumptions hold. In the usual frequentist interpretation, a 95% interval is not a claim that there is a 95% probability that the fixed parameter is inside this particular interval. It conveys the range of values compatible with the observed data under the model and procedure.

Rank #2

Testing Statistical Hypotheses (Springer Texts in Statistics)

$79.99

Buy on Amazon

For compatible methods, a two-sided test at the 5% level rejects a hypothesized value when that value falls outside the corresponding 95% confidence interval. This relationship depends on the test and interval using compatible assumptions and methods; NIST describes the connection in its guidance on confidence intervals and tests.

Report an effect in a meaningful scale: a mean difference in original units, absolute risk difference, relative risk, odds ratio, correlation, regression coefficient, rate ratio, or hazard ratio, as relevant. A standardized effect size can aid comparisons across scales, but generic labels such as “small,” “medium,” and “large” are context-dependent and should not replace a domain-specific threshold.

Statistical significance and practical importance answer different questions. A tiny difference can yield a small p-value in a very large sample; an important difference can have a large p-value in a small, noisy study. Ask whether the estimate and interval include effects large enough to matter.

Errors, power, and planning sample size

Power is 1 − β, the chance that a test will reject the null for a particular alternative. It depends on sample size, effect size, variability, α, one- versus two-sided testing, analysis method, missing data, and multiplicity adjustments. “The study had 80% power” is incomplete unless the assumed effect, variance, design, α, and test are stated.

Before data collection, a power or sample-size analysis can help plan for a justified minimum important effect, a chosen α, and a target power such as 80% or 90%. Ground the target effect in prior evidence, subject-matter judgment, a minimum important difference, or a decision threshold—not in a convenient sample size. Power tools calculate under assumptions; they cannot decide whether those assumptions are credible. SciPy documents simulation-based power estimation in its statistics reference.

For a nonsignificant result, do not rely on “post hoc power” calculated from the observed effect as a substitute for interpretation. Examine the interval: does it rule out effects that would matter, or does substantial uncertainty remain?

Choosing a test: start with outcome and design

Test names are secondary to the data structure. Measurements from the same person, household, clinic, school, or company are not automatically independent. Repeated, paired, clustered, matched, and time-series data require methods that represent those dependencies.

Rank #3

Sale

Introduction to Robust Estimation and Hypothesis Testing (Statistical Modeling and Decision Science)

$77.91

Buy on Amazon

Common tests in brief

One-sample t-test

Compares a sample mean with a fixed value when the population standard deviation is unknown. The statistic is t = (x̄ − μ₀)/(s/√n), with n − 1 degrees of freedom. Its interpretation depends on the design and the behavior of the data or mean estimator.

Independent-samples t-test

Compares means from two independent groups. Welch’s form uses the standard error √(s₁²/n₁ + s₂²/n₂) and a Welch–Satterthwaite degrees-of-freedom approximation. It does not require equal variances. It does require independent observations across the groups.

Paired t-test

Reduces paired observations to differences d_i = after_i − before_i (or another prespecified order), then tests whether the mean difference is zero. Treating before and after values as independent discards the pairing and gives the wrong uncertainty calculation.

ANOVA and regression

ANOVA tests an overall contrast among group means under its model. A significant omnibus result indicates that not all modeled means are equal; it does not establish that every pair differs. Regression generalizes comparisons to include predictors or covariates, but its coefficients and uncertainty are meaningful only under a suitable model.

Proportion and chi-square tests

These analyze categorical outcomes or counts. For a landing-page conversion comparison, report the number of visitors and conversions in each group as well as rates. “10% versus 8%” lacks useful precision without denominators and intervals. Use exact or suitable model-based approaches when counts are sparse.

Correlation

A correlation test can assess whether a population linear association differs from zero, but a scatterplot should accompany it. Correlation is not causation, can miss nonlinear patterns, and is sensitive to influential points. Correlation also does not measure agreement between two methods.

Nonparametric and permutation tests

Rank-based tests can suit ordinal outcomes or some departures from parametric assumptions, but they are not assumption-free and do not automatically test a difference in means. Permutation tests rely on a valid exchangeability or randomization scheme. Paired, clustered, or otherwise dependent designs require restricted permutations that preserve the design.

Rank #4

Sale

Randomization Tests (Statistics: A Series of Textbooks and Monographs)

• Used Book in Good Condition

$140.00

Buy on Amazon

Worked examples

1. Is average battery life different from 10 hours?

Suppose a sample of batteries is tested and the question is whether the population mean differs from 10 hours. Set H₀: μ = 10 and H_a: μ ≠ 10. If the population standard deviation is unknown, a two-sided one-sample t-test is a usual choice, subject to an appropriate sampling process and reasonable behavior of the observations or mean estimator.

Calculate x̄, s, and n; then compute t = (x̄ − 10)/(s/√n) and use n − 1 degrees of freedom to obtain the p-value. Report the estimated mean and a 95% interval as well. Because no sample values are specified here, no numerical statistic or result can be responsibly supplied. A useful conclusion form is: “The estimated mean battery life was X hours (95% CI L to U). The two-sided test against 10 hours gave p = P. This provides [evidence / insufficient evidence] that the population mean differs from 10 hours under the stated model.”

2. Did a treatment change mean blood pressure?

For independent treatment and control groups, compare group means using Welch’s t-test unless a different model is justified. State the null as a zero mean difference and select a two-sided or directional alternative before examining results. Report each group’s mean and sample size, the estimated mean difference, its confidence interval, the test statistic with degrees of freedom, and the p-value. A standardized effect size may help comparisons across measures, but retain the difference in blood-pressure units for clinical interpretation.

Then compare the estimate and interval with a clinically meaningful threshold. A tiny reduction can be statistically detectable in a large trial but not clinically important. Conversely, a potentially meaningful reduction with a wide interval in a small trial is uncertain, not proof of no effect.

3. Did participants’ scores change after an intervention?

For each participant calculate d_i = after_i − before_i, then test H₀: μ_d = 0. The paired t-test uses the variation among within-person differences, not the two measurements as though they came from unrelated people. Report the mean change, its interval, and the test result. If participants are missing at one time point or measurements are clustered, the simple paired test may not answer the intended question; consider an appropriate repeated-measures or mixed model.

4. Did one landing page convert better?

For two independently assigned page groups, start with conversions and visitor counts, then report each conversion rate and the absolute percentage-point difference with an interval. Depending on the estimand, also report a relative risk or odds ratio. Use a two-proportion, chi-square, exact, or logistic-regression approach suited to the allocation and cell counts. A result without denominators is incomplete: 10 conversions among 100 visitors differs in precision from 1,000 among 10,000 even though both yield 10%.

5. Is study time associated with exam score?

Plot the paired observations and inspect for curvature and influential points before testing H₀: ρ = 0. Pearson correlation addresses linear association, not causality or whether study time predicts scores well enough for a decision. A statistically detectable association does not show that increasing study time caused a score change; that requires a causal design or defensible causal model.

Best Value

Sale

The Phylogenetic Handbook: A Practical Approach to Phylogenetic Analysis and Hypothesis Testing

• Used Book in Good Condition

$60.12

Buy on Amazon

6. Which of 20 outcomes changed?

If 20 independent null hypotheses were all tested at α = 0.05 without adjustment, the probability of at least one false positive under the global null would be 1 − 0.95²⁰, about 64%. Real tests may be correlated, so this exact calculation may not apply, but the general multiplicity risk remains. Prespecify primary outcomes and use a procedure matched to the aim: Bonferroni or Holm methods control family-wise error, while Benjamini–Hochberg controls the false discovery rate under its conditions. Label exploratory findings and report the full set of planned outcomes, not only favorable results.

Applications and interpretation

• Healthcare and clinical research: distinguish statistical from clinical importance, account for randomization, baseline measures, missingness, multiple outcomes, and safety signals. Superiority, equivalence, and noninferiority are distinct questions.
• Business experimentation: define the conversion or revenue estimand, randomize where possible, choose a stopping rule in advance, and account for repeated looks or multiple metrics. A p-value alone does not tell whether a change is profitable or operationally worthwhile.
• Manufacturing and quality control: a test may compare a process mean with a target, but process monitoring, dependence over time, and the cost of defects matter. Repeated observations from a production line are not necessarily independent.
• Social science and education: account for clustered students, classrooms, schools, repeated measures, and confounding. An association in observational data does not by itself establish a causal effect.
• Data science and machine learning: define the evaluation metric and comparison, avoid using test data to tune a model, and respect dependence among observations. A statistically significant metric difference may still be too small to matter in deployment.

Multiple testing, optional stopping, and selective reporting

When many hypotheses are tested, some small p-values can arise even when none of the null claims is false. Family-wise error procedures aim to control the probability of any false positive in a defined family; false-discovery-rate procedures aim to control the expected proportion of false discoveries among the rejected hypotheses. Pick the family and method based on the question, and disclose it.

Repeatedly checking results and stopping once p < 0.05, trying many outcomes or model specifications and reporting only the favorable one, and changing outcomes after seeing data can make nominal p-values misleading. Preregister confirmatory hypotheses and analysis plans when appropriate, define stopping rules, distinguish planned from exploratory work, and disclose deviations and all tested outcomes. These practices do not make a study infallible, but they make the evidential process more transparent.

What to do when assumptions fail

• Dependence: use paired or repeated-measures methods, mixed-effects models, generalized estimating equations, cluster-robust standard errors, or a time-series model as design warrants. Cluster-level analysis may suit some designs. Do not treat repeated observations as independent.
• Non-normality: t-tests do not require every raw observation to be perfectly normal. Consider sample size, skew, outliers, and the sampling distribution of the estimate; for regression and ANOVA inspect residuals. A normality test alone is not a complete diagnostic.
• Unequal variances: Welch’s t-test avoids the equal-variance assumption for two independent means and is often preferable to the pooled t-test.
• Outliers: investigate whether an extreme value is an error, measurement failure, legitimate observation, or evidence of a poor model. Do not delete it solely because it weakens significance; report a justified sensitivity analysis where useful.
• Sparse counts or very small samples: normal approximations and standard errors may be unreliable; logistic regression can encounter separation. Consider exact, permutation, bootstrap, robust, or Bayesian approaches only when their assumptions suit the design. Small samples also mean wide intervals and low power.
• Ordinal or otherwise non-normal outcomes: a rank-based alternative may be useful, but state what it tests. A Mann–Whitney result is not automatically evidence about a mean or median difference.
• Complex estimates: bootstrap intervals can help with some estimators but cannot correct biased or unrepresentative data. Resample at the correct unit, preserving clusters or pairs, and be cautious with tiny samples, dependence, and extreme sparsity.

Classical parametric tests can be efficient and interpretable when assumptions are reasonable, but can mislead under misspecification or ignored design. Nonparametric methods can be robust in some settings but may target ranks or distributional shifts rather than means. Permutation tests depend on a valid permutation scheme. Bayesian methods can represent uncertainty and prior information directly, but require explicit modeling and prior choices; they are not merely p-values reworded.

How to report a result

Prefer a result statement that makes the quantity, uncertainty, and decision context visible:

The estimated difference between groups was D units (95% CI L to U). The prespecified [test name] produced p = P. These results provide [evidence / insufficient evidence] against the null hypothesis of [state null value]. Relative to [domain threshold], the estimated effect is [meaningful / uncertain / likely trivial].

For a nonsignificant result:

The result was not statistically significant at the prespecified α level. This does not demonstrate that the groups are identical. The confidence interval, L to U, indicates that effects across this range remain compatible with the data under the model.

×

What’s actually slowing this PC down?

Pick the symptom - the matching free tool is one click away.

Crashes & glitchesFree fix →System errors & instabilityFree fix →Outdated driversFree fix →Junk files & broken settingsFree fix →

Special offer. See more information about Outbyte and uninstall instructions. Please review EULA and Privacy policy.

For equivalence, report the prespecified bounds and demonstrate that the appropriate interval lies entirely within them; do not infer equivalence from an ordinary nonsignificant superiority test. In every case include sample sizes, effect estimates, intervals, test and assumptions, and any multiplicity adjustment. Avoid “proved,” “happened by chance,” “no effect,” and “accepted the null” unless the design and inferential method truly support the stronger claim.

Software: useful calculators, not statistical judgment

R with RStudio is a free, flexible route for reproducible analysis; see Posit’s RStudio page. GraphPad Prism offers a point-and-click workflow commonly suited to scientific graphing and analysis; see its buying information. JMP provides interactive visual analysis and broader statistical workflows; see JMP’s licensing page. Availability, license terms, and prices can change, so check the vendors’ current terms if choosing software.

No package automatically knows the right estimand, whether observations are paired or clustered, which outcomes were prespecified, or what effect matters. Define the question and design, verify the method’s assumptions, and interpret the result yourself. For reproducible work, preserve the data-cleaning steps, analysis code or settings, and reporting decisions.

Quick Recap

SaleBestseller No. 1

Introduction to Robust Estimation and Hypothesis Testing (Statistical Modeling and Decision Science)

$59.90

Bestseller No. 2

Testing Statistical Hypotheses (Springer Texts in Statistics)

$79.99

SaleBestseller No. 3

Introduction to Robust Estimation and Hypothesis Testing (Statistical Modeling and Decision Science)

$77.91

SaleBestseller No. 4

Randomization Tests (Statistics: A Series of Textbooks and Monographs)

Used Book in Good Condition

$140.00

SaleBestseller No. 5

The Phylogenetic Handbook: A Practical Approach to Phylogenetic Analysis and Hypothesis Testing

Used Book in Good Condition

$60.12

Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.