Citation metrics covary with researchers’ assessments of the quality of their works

Abstract

For a long time, citation counts have been used to measure scientific impact or quality. Do such measures align with researchers’ assessments of the quality of their work? In this study, we address this issue by decomposing the research quality concept into constituent parts and analyzing their correspondence with citation measures. The focus is on individual publications, their citation counts and how the publications are rated by the authors themselves along quality dimensions. Overall, the study shows a statistically significant relationship for all dimensions analyzed: solidity, novelty/originality, scientific importance and societal impact. The highest correlation is found for scientific importance. However, it is not very strong, but we find distinct gradients when publications are grouped by quality scores. This means that the higher the researchers rate their work, the more they are cited. The results suggest that citation metrics have low reliability as indicators at the level of individual articles, but at aggregated levels, the validity is higher, at least according to how authors perceive quality.