A statistically significant A/B test result can still fail to produce a real-world lift. A 5% significance level limits the chance of a false positive for one properly conducted test under its assumptions; it does not mean a reported winner has a 95% chance of being real. And it does not guarantee that exactly one in 20 declared winners will be false. That second quantity—the false discovery rate—depends on how many tested changes truly work, how tests are run, and how results are selected.
What is a false positive in an A/B test?
A false positive, also called a Type I error, occurs when a test reports a difference even though there is no true difference between the variants under the null hypothesis. In conversion testing, that can make an ineffective change look like a winner.
As an Amazon Associate I earn from qualifying purchases.
The significance level, often written as alpha, sets the per-test probability of observing a result at least as extreme as the one measured if the null hypothesis is true—assuming the statistical method and test conditions are appropriate. It is not the probability that the particular result you observed is false. Adobe’s guidance describes the choice as a trade-off: a stricter threshold can reduce Type I errors, but may increase the chance of missing a real effect (a Type II error, or false negative). Adobe Target: How Do I Avoid Common A/B Testing Mistakes?
Do these 3 things before closing this tab:
1Repair Windows errors before they cause bigger problems2Fix the driver behind crashes, sound loss and screen glitches3Clear out junk files and repair common Windows errorsWhy can false discoveries outnumber the significance threshold?
The false discovery rate is the share of declared winners that are actually ineffective. It is not the same as the per-test Type I error rate. If most ideas being tested have no meaningful effect, then even a test with a 5% significance threshold can produce a substantial proportion of false discoveries among the results that pass that threshold.
#1 Best Overall
A 2022 study by Ron Berman and Christophe Van den Bulte analyzed 4,964 effects from 2,766 commercial website experiments. The authors estimated that about 70% of effects were true nulls, and that the false discovery rate among statistically significant results was 18–25% at a 5% significance level. At a 10% significance level, their estimate was 28–37%. They summarized the implication as about one in five interventions significant at 5% being ineffective when deployed in the field. These are estimates from the study’s commercial experimentation context, not a universal rate for every organization or testing program. Berman and Van den Bulte, “False Discovery in A/B Testing,” Management Science (2022).
How multiple comparisons make apparent winners more likely
Each additional variant, audience segment, metric, or offer examined creates another opportunity to find a nominally significant result by chance. For example, a test may show no overall lift, but one of many post-test segments may appear to improve. If you highlight that segment only after seeing the results, the original per-test significance threshold no longer describes the chance of a false positive across all the comparisons you explored.
Adobe specifically cautions against post-test segmentation and testing multiple offers without accounting for the extra comparisons. Decide which comparisons matter before launch. If you explore additional segments or metrics after the test, label those findings as exploratory and treat them as leads to validate rather than confirmed wins. Adobe Target: How Do I Avoid Common A/B Testing Mistakes?
How long should you run an A/B test?
There is no reliable universal number of days. Duration depends on the sample size required for the effect you care about, traffic, the significance level, and the test’s statistical power. Before launch, estimate the required sample size and expected duration, then choose a stopping rule that matches the method you will use. Adobe’s guidance covers the relationship between significance, sample size, power, and duration: How Long Should I Run an A/B Test?
Repeatedly checking results and stopping as soon as they become significant can inflate false-positive risk if the analysis assumes a fixed end point. A fixed-horizon test should follow its preselected sample size or duration. If you need to monitor results and stop early, use a sequential method designed to account for repeated looks; do not simply apply a fixed-horizon significance threshold at every check. The 2024 paper by Ron Kohavi and Nanyu Chen discusses sequential group testing as an approach to reducing false positives: “False Positives in A/B Tests,” KDD ’24.
A workflow for making conversion-test results more dependable
- Write the hypothesis and primary metric before launch. Specify what change you are testing, the outcome that will determine success, and the population included in the primary analysis.
- Set a minimum effect worth acting on. A tiny lift may be statistically detectable but not valuable enough to justify implementation. Choose the smallest effect that would change your decision.
- Plan significance, sample size, power, and duration. Calculate the sample needed to detect the chosen effect, and choose a fixed-horizon or valid sequential procedure before collecting results.
- Predefine the decision rule and planned comparisons. Record which variants, metrics, and segments are confirmatory. Account for multiple planned comparisons where appropriate; keep unplanned cuts clearly exploratory.
- Follow the stopping rule. Do not change the end point because the result looks promising or disappointing. If ongoing monitoring is necessary, use a method designed for sequential analysis.
- Separate exploration from confirmation. Use exploratory results to generate hypotheses. For a consequential change, validate the apparent winner in a follow-up test or staged rollout before broad deployment.
These practices reduce avoidable ambiguity; they cannot guarantee that every result is correct. The appropriate threshold and design depend on the cost of a false positive—shipping an ineffective change—as well as the cost of a false negative—discarding a change that would have helped.
Independent reader supportYour contribution helps us test, update, and keep practical guides available for everyone.Choosing an approach when comparing several options
Before testing multiple versions or making decisions from many possible results, check four things:
What’s actually slowing this PC down?
Pick the symptom - the matching free tool is one click away.
Quick Recap
- How many variants and comparisons are planned? More comparisons create more chances for a chance finding, so define them in advance and account for multiplicity where needed.
- How will the test stop? A fixed-horizon design and a sequential design require different analysis rules. Match the significance calculation to the stopping plan.
- Is the test powered for the effect that matters? A design aimed at an implausibly small effect may require excessive traffic, while an underpowered test can miss meaningful improvements.
- What is the cost of each kind of error? A high-impact rollout may warrant stronger confirmation than a low-risk, reversible change.
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
