Free tools Windows power users keep installed
One-click scans. No signup required.
Citation counts and chemists’ judgments of a paper’s significance are related, but they are not the same thing. A 2018 study of 63 papers in one issue of the Journal of the American Chemical Society found that respondents were better at estimating which papers had accumulated citations than at identifying the papers they considered significant. The result is a useful warning against treating citation totals—or an author’s h-index—as a complete measure of research value.
What the study compared
Borchardt and co-authors surveyed chemists about 63 papers published in a single 2003 issue of the Journal of the American Chemical Society. Respondents were asked which papers they considered most significant, which they thought were most cited, which they would share with other chemists, and which they would share with a broader audience.
As an Amazon Associate I earn from qualifying purchases.
The researchers compared those answers with article citation counts recorded in Web of Science in November 2013 and November 2016, corresponding authors’ h-indexes recorded in January 2016, and page views and Mendeley metrics recorded in November 2016. The paper was published in PLOS One on March 28, 2018. Read the study.
Quick wins for a faster PC:
Scan for outdated or missing drivers - takes under a minuteDriver Scan →Clear out junk files and repair common Windows errorsFree Scan →What chemists’ answers revealed
Of 408 survey responses, 363 were retained for analysis after 45 were excluded. Within that analyzed group, respondents’ guesses about citation counts correlated with actual citation counts more strongly than their significance judgments or sharing preferences did. The corresponding author’s h-index had a stronger correlation with actual article citation counts than any of the four survey-response categories.
#1 Best Overall
That comparison concerns how well these measures tracked citation counts in this particular sample. It does not show that h-index measures a paper’s scientific significance.
Subdiscipline shaped what respondents noticed
Papers respondents thought were highly cited, or said they would share with other chemists, came from the respondent’s own subdiscipline 70% of the time. Papers respondents judged significant came from their own subdiscipline 63% of the time. These percentages describe the study’s respondents and paper set; they are not estimates for chemists generally.
Rank #2
Why citations and significance can diverge
A citation count records citation activity: how often a paper has been cited within a particular index and by the time it is counted. It does not directly record why another author cited it, whether the citing work builds on it, or whether researchers regard it as important. A highly cited paper may attract attention for reasons other than perceived significance; a valuable or useful paper may matter to a narrower community without accumulating as many citations.
The survey itself separated several judgments that are easy to collapse into one: expected citation frequency, perceived significance, usefulness to colleagues, and appeal to a broader audience. A chemist might consider a specialized paper important to a field while not expecting it to attract wide citations or public interest. The study shows that these judgments do not line up perfectly; it does not establish peer opinion as an objective ground truth or provide a universal formula for research assessment.
What citation counts and h-index can—and cannot—tell you
| Measure | What it describes in this study | Important limit |
|---|---|---|
| Article citation count | Citations accumulated by an individual paper, recorded in Web of Science at the stated collection dates. | It measures citation activity, not significance, quality, usefulness, or societal value by itself. |
| Corresponding-author h-index | An author-level bibliometric measure compared with the citation count of each paper. | It tracked article citations more closely than survey categories in this study, but that does not make it a measure of the paper’s significance. |
| Survey judgments | Respondents’ views about significance, likely citations, and sharing with specialist or broader audiences. | They reflect the views of a self-selected sample and can vary with field expertise and question framing. |
| Page views and Mendeley metrics | Web attention and reader-library activity collected in November 2016. | The study included these indicators, but neither should be treated as a complete measure of research value. |
Each measure has a different unit and target: an article, an author’s publication record, or some form of readership or attention. Comparing them without naming what they count can make a metric appear to answer a question it was not designed to answer.
Independent reader supportYour contribution helps us test, update, and keep practical guides available for everyone.How far the findings apply
The study is preliminary, not a census of chemists. Its respondents were recruited through online chemistry communication channels, making the sample self-selected rather than a probability sample. The paper set was limited to one journal issue from 2003, and the authors note that their subdiscipline classifications were author-led. The four survey questions were not randomized, so earlier questions may have influenced later answers. Chemistry World’s 2018 report quotes co-author Rachel Borchardt describing the result as a disconnect between citation and “actual humanly perceived significance.”
Rank #4
Because the papers, survey, and metric snapshots are historical, the findings do not establish how chemists currently judge papers, how other disciplines behave, or what citation patterns look like across publishing today. The authors’ broader conclusion is that peer judgments and metrics-based measurements differ, and that better assessment should capture more nuance than a single number.
Do these 3 things before closing this tab:
1Repair Windows errors before they cause bigger problems2Scan for outdated or missing drivers - takes under a minute3Clear out junk files and repair common Windows errorsQuick Recap
Best Value
How to read a paper’s citation record responsibly
- Ask what the number counts: citations to one paper, an author-level index, or another kind of attention.
- Check the database and date, since citation totals accumulate over time and depend on the source being counted.
- Consider the paper’s field and audience; a specialist contribution may be significant without broad reach.
- Use metrics alongside informed reading and relevant expert judgment, rather than treating any one measure as a verdict on quality or value.
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
