Data dredging is the practice of searching through analyses for favorable results and emphasizing selected findings while leaving the selection process unclear. It overlaps with p-hacking and selective inference. The concern is not that researchers explore data; it is that a result chosen after looking at the data can be presented as though it came from a planned, independent test. That can make evidence look stronger than it is.
What data dredging means
The American Statistical Association (ASA) groups data dredging with cherry-picking, significance chasing, selective inference and p-hacking. These terms describe analysis or reporting choices that favor promising results. As the ASA puts it, “Proper inference requires full reporting and transparency.” Its 2016 statement also warns that cherry-picking can produce “a spurious excess of statistically significant results in the published literature” and should be avoided: ASA Statement on Statistical Significance and P-Values.
As an Amazon Associate I earn from qualifying purchases.
The defining issue is selective choice and incomplete disclosure. A researcher may examine several outcomes, models, subgroups or time windows, then report only the analysis that supports a desired conclusion. The same problem can arise without a formal battery of tests if the researcher chooses what to present based on results but does not explain that choice.
Recommended Free Tools
How data dredging can produce false confidence
A p-value is interpreted in relation to a specified analysis and its assumptions. If readers do not know how many analyses were considered, which choices were made after inspecting results, or which findings were omitted, they cannot properly assess the reported p-value. Selective reporting can make statistically significant findings appear more common or convincing than the underlying evidence warrants.
#1 Best Overall
A p-value is not the probability that a hypothesis is true, a measure of the effect’s size, or evidence by itself that the effect matters in practice. A threshold such as 0.05 is not a verdict. The ASA explains what p-values can and cannot establish in its statement on statistical significance and p-values.
An example: ten possible tests
The ASA’s explainer offers an illustrative medical example: researchers could define a vomiting outcome in different ways and use different time windows, creating ten possible tests. If all ten are run but only the tests with p < 0.05 are reported, readers need to know about the full set and how the reported result was selected to interpret it. The ten tests are an illustration, not a measured estimate of how often data dredging occurs or of a false-positive rate: ASA explainer on p-values.
Rank #2
Exploration is not automatically misconduct
Exploratory analysis can be useful for finding patterns and generating hypotheses. The important distinction is whether the analysis is described honestly. Findings identified after looking at data should be labeled and interpreted as exploratory, rather than presented as if they were a clean confirmatory test based on a plan fixed in advance.
Do these 3 things before closing this tab:
1Fix the driver behind crashes, sound loss and screen glitches2Repair Windows errors before they cause bigger problems3Scan for outdated or missing drivers - takes under a minutePrespecification helps readers distinguish planned tests from post hoc choices, but transparency remains essential either way. A report should make clear what was planned, what changed after the data were examined, and how those choices affect interpretation. Guidance from the ASA and reporting recommendations for animal research both emphasize clear, complete disclosure; ARRIVE’s recommendations apply specifically to animal studies, not as a universal regulation: ARRIVE guidelines.
Rank #3
What researchers should disclose
A reader needs enough information to understand the path from research question to reported result. Relevant details include:
- Which hypotheses and analyses were planned in advance, and which were chosen or changed after results were seen.
- How outcomes and predictors were defined, including any alternative definitions or time windows considered.
- Which models, covariates, exclusions and subgroup analyses were used, and the rationale for those choices.
- How missing data were handled and whether adjustments for multiple comparisons were made.
- Which analyses were relevant to the claim, including null or negative results, rather than only favorable findings.
- Effect sizes and uncertainty, interpreted in context rather than reduced to whether a p-value crossed a threshold.
- Software and version information when relevant to understanding or reproducing the analysis.
ARRIVE provides a concrete statistical-reporting example for animal research, while NOAA’s research-integrity guidance advises against selective reporting and stopping after significance and calls for reporting relevant null or negative findings: ARRIVE guidelines; NOAA Science Council guidance.
Rank #4
- Used Book in Good Condition
How to assess a study or compare findings
When evaluating a result, focus on the analysis process as well as the headline finding. These questions help identify whether the reported evidence is interpretable:
- Was the plan clear? Can you tell which hypotheses and methods were set before results were examined?
- Is reporting complete? Are the analyses relevant to the claim described, or does the report show only a selected result?
- Was multiplicity addressed? Does the study explain the number or range of comparisons and how that affects inference?
- Are null findings visible? Does the report discuss relevant results that did not support the hypothesis?
- Is the effect meaningful? Are magnitude and uncertainty explained, rather than relying only on statistical significance?
When comparing two studies, apply the same questions to each. A result from a prespecified analysis is not automatically reliable, and an exploratory result is not automatically useless; the distinction helps establish what each finding can support.
Is data dredging common?
The sources cited here do not establish a directly applicable prevalence statistic for data dredging. The ASA’s ten-test example explains how selective analysis can affect interpretation; it is not an estimate of how often researchers engage in the practice.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
