A non-significant replication is not automatically a successful replication of an original null finding, and it is not proof that a previously reported effect is absent. Before you count it either way, you need three things: the claim the study was built to test, the smallest effect the sample could reliably detect, and an inferential method that actually speaks to absence rather than only to a point-null test.
What a non-significant p-value actually tells you
A p-value above the chosen threshold means the test did not cross that threshold. It does not show that the effect is zero. The same non-significant result can arise from a truly absent effect, from a small effect that the study was too noisy to detect, or from a large effect estimated imprecisely. The label alone cannot separate these readings, so the interpretation has to come from the estimate and its uncertainty.
As an Amazon Associate I earn from qualifying purchases.
Why “both studies were non-significant” can count as success by default
The authors of an eLife article from 2024, “Replication of null results: Absence of evidence or evidence of absence?”, make the key point directly: non-significance in both studies does not ensure that the studies provide evidence for the absence of an effect, and “replication success” can virtually always be achieved if the sample sizes are small enough. The logic is simple. If both the original and the replication are underpowered, both will often be non-significant regardless of whether the effect exists. A rule that counts “both null” as success will then label inconclusive studies as successes, and it does not control the error rates that a replication framework is supposed to manage.
Do these 3 things before closing this tab:
1Repair Windows errors before they cause bigger problems2Scan for outdated or missing drivers - takes under a minute3Clear out junk files and repair common Windows errorsThis is why a replication outcome should be judged against the design that produced it, not against the significance labels of the two studies.
#1 Best Overall
Plant the claim in the replication’s own data
The most useful habit is to restate the claim as an effect size with a precision, then ask whether the replication could have registered that effect at all. Think of it as asking what the replication would have shown if the original effect had been real at the size reported. This is a sensitivity question, and it is answered before you look at the replication’s p-value.
- Write the original effect estimate and its 95% confidence interval in the same units the replication uses. If the units differ, convert them first.
- Find the replication’s sample size, the number of observations per condition, and the standard error of its estimate.
- Ask whether, at that sample size, a true effect of the original size would usually produce an interval that excludes zero. If it would not, the replication was not positioned to detect the claim, and a null tells you little about it.
- Check whether the power calculation in the replication report used the original estimate or a smaller, more conservative effect. Replications often plan for the original effect; many original estimates are inflated because they were selected for significance.
Two cautions apply. A worked example makes the stakes concrete: a two-group comparison with 20 participants per group has low power for a standardized effect of 0.3, and a null from that design says little about whether an effect of that size exists. Also, do not compute “observed power” from the replication’s own estimate. Observed power is a transformation of the p-value and adds no independent information about the claim.
Compare estimates and intervals, not only significance labels
The same eLife article assessed four replication criteria across 15 replications of original null results. The criteria ask different questions, and they produced different figures. These are results for particular criteria in that examined set, not universal replication rates, and they should not be combined into one success statistic.
Rank #2
- This guide is a perfect overview for the topics covered in introductory statistics courses.
| Criterion | Question it asks | Result in the examined set (eLife, 2024) |
|---|---|---|
| Original estimate within the replication’s 95% confidence interval | Are the data from the replication compatible with the original effect size? | 11/15 (73%) |
| Replication estimate within the original’s 95% confidence interval | Is the replication’s estimate compatible with the original’s uncertainty range? | 12/15 (80%) |
| Replication estimate within the 95% prediction interval based on the original | Does the replication fall within the range a future study might plausibly produce, given the original? | 12/15 (80%) |
| Combined original-and-replication meta-analysis non-significant | Does the pooled estimate cross the significance threshold? | 10/15 (67%) |
Confidence intervals describe uncertainty around an estimated effect under the model. Prediction intervals describe the range in which a future study’s estimate may fall under the specified model, and they are wider because they include sampling variation in that future study. They are distinct criteria, not interchangeable ones. When the criteria disagree, that disagreement is itself informative: it usually signals a noisy estimate, a heterogeneous set of conditions, or an original estimate that was larger than the replication supports.
Methods that address absence directly
A null p-value tests a point hypothesis of exactly zero. To make a claim about absence, you need a method whose question is about a range of effects. Two approaches do this, and each has conditions.
Equivalence testing
Equivalence testing asks whether the effect lies within bounds that you define as practically negligible. The eLife article describes this approach as one of the methods more aligned with evidence for absence. The procedure works like this:
Rank #3
- Before analyzing the data, define a smallest effect size of interest, or equivalence bounds, in the outcome’s units. Record the justification.
- Compute a 90% confidence interval for the effect. In the usual two one-sided tests setup, this interval corresponds to a 5% significance level for each one-sided test.
- Conclude practical equivalence only if the entire interval lies inside the bounds.
- Report the bounds and the reasoning for them alongside the interval, so readers can judge whether a different bound would change the conclusion.
The bounds carry the whole argument. A wide bound makes equivalence easy to claim and says little; a narrow bound may be unattainable for a small study. Equivalence is established only when the uncertainty is contained within bounds that have substantive justification.
PC Slower Than It Used to Be?
A free scan shows the junk files, broken settings and background clutter dragging Windows down - then fixes them in one click.Free scan · Windows 10 & 11Outdated Drivers Are Slowing You Down
One free scan finds every outdated or missing driver and matches the right update for your exact hardware.Free scan · exact hardware matchBayes factors
A Bayes factor compares how well the data support one specified hypothesis relative to another. It can express evidence for a null hypothesis in a way a p-value cannot. Its conclusion depends on the hypotheses being compared and on the prior distributions or models chosen for the effect. Conventional thresholds for calling evidence moderate or strong vary across fields, and results can be sensitive to how wide the prior for the effect is. A Bayes factor is therefore not assumption-free proof that an effect is exactly zero. Report the priors and show how the conclusion changes across reasonable alternatives.
Combined meta-analysis
Pooling the original and replication estimates produces an aggregate estimate, which can be informative. A non-significant pooled p-value, however, does not by itself quantify evidence for absence. It has the same limitation as any point-null test applied to a combined sample, and the pooled estimate can hide a real disagreement between the two studies.
Check whether the replication tests the same claim
An interpretable replication has to address the same claim, which means the same population, the same treatment or exposure at a comparable dose, and an outcome measured in a way that reflects the original construct. A PLOS Biology article, “What is replication?”, frames fidelity to the original design and the relevance of the claim as central to interpreting any outcome. Use these questions when you read a replication report:
- Did the replication use the same outcome measure, or a different operationalization that captures a different construct?
- Were the population, setting, and treatment comparable, and were any changes made deliberately or by necessity?
- Does the replication estimate reflect a smaller effect, a boundary condition, or a failure to implement the original protocol?
- Would a meaningful effect, in practical terms, be detectable with this measure, given its variability?
Differences between studies are normal. Accumulated evidence may show that a finding holds under a narrower set of conditions than the original report implied. Treat a null as a statement about the conditions the replication actually created.
Free tools Windows power users keep installed
One-click scans. No signup required.
Reporting and analysis choices
The Office of Research Integrity lists selective reporting of results as a research-integrity concern, including the non-publication of null or inconclusive findings and the selection of analyses that best fit a hypothesized result. Its guidance makes these risks clear without establishing how common they are in replication studies. When you read a replication, ask whether the analysis plan was specified before data collection, how many analyses were run, and whether the null results from related outcomes or subgroups were reported. A visible literature with few nulls may reflect what was published rather than what was found.
Best Value
Reading a non-significant replication: decision patterns
The table below summarizes how the evidence should be read under common patterns. It assumes the design was faithful to the original claim and that the analysis was pre-specified.
| Pattern in the replication | What you can reasonably conclude | What you cannot conclude |
|---|---|---|
| Wide interval that includes the original estimate and zero | The study is inconclusive for this claim. | That the original effect is absent, or that the replication succeeded. |
| Narrow interval that excludes the original estimate | The data are not compatible with the original effect size as reported under this design. | That the effect is exactly zero, or that the original null (if the original was a null) is confirmed. |
| Interval lies entirely within pre-justified equivalence bounds | Practical equivalence within those bounds is supported, given the stated justification. | That the effect is zero, or that the bounds would be accepted by others without argument. |
| Non-significant result from a design that could not detect the claimed size | Almost nothing about the claim. | Any inference about absence. |
These patterns are guides to reading a report, not verdicts. Checking the specific data, the estimates, and the study context is what turns a pattern into a judgment.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
