MURDOCH RESEARCH REPOSITORY This is the author s final version of the work, as accepted for publication following peer review but without the publisher s layout or pagination. The definitive version is available at http://dx.doi.org/10.1007/s11192-008-2229-y Calver, M.C. and Bradley, J.S. (2009) Should we use the mean citations per paper to summarise a journal's impact or to rank journals in the same field? Scientometrics, 81 (3). pp. 611-615. http://researchrepository.murdoch.edu.au/1020/ Copyright: 2009 Akadémiai Kiadó, Budapest It is posted here for your personal use. No further distribution is permitted.
Should we use the mean citations per paper to summarise a journal s impact or to rank journals in the same field? Michael C. Calver, J. Stuart Bradley School of Biological Sciences and Biotechnology, Murdoch University, Murdoch, Western Australia 6150, Australia Abstract The mean citations per paper is used increasingly as a simple metric for indicating the impact of a journal or comparing journal rankings. While convenient, we suggest that it has limitations given the highly skewed distributions of citations per paper in a wide range of journals. Introduction As criticism of the Journal Impact Factor (JIF) as a standard for ranking journals increases ([YANG & MEHO 2007; BROWMAN & STERGIOU 2008] and included references), other measures including the mean citations per paper have been proposed or used [TRIMBLE & AL., 2004; ANDREIS & JOKIC, 2006, HERMES-LIMA & AL., 2007, KADEMANI & AL., 2007; HARZING & VAN DER WAL, 2008; YI & AL., 2008]. Mean citations per paper can be calculated easily from data in many data bases, removing dependence from the limited list of journals covered in Thomson Reuters ISI Web of Science and Journal Citation Reports. This statistic, renamed the mean citations per document, is now listed at the bibliometric data site SCImago (http://www.scimagojr.com/) while Journal-Ranking.com (http://www.journalranking. com/ranking/web/index.html) provides mean citations per paper for periods of two, four and six years (called B2, B4 and B6 respectively).
Statistically, the mean is attractive because it typically has a lower standard error than the other measures of central tendency (the median and the mode) [SOKAL & ROHLF, 1995] and the means of samples from the same population tend to be distributed normally even if the population is not (the Central Limit Theorem see [TABACHNICK & FIDELL, 2001]). However, the mean is more sensitive than either the mode or the median to outlying values and therefore is not the best measure of central tendency for markedly non-normal distributions because it is biased by a few extreme values. The median is preferred in these cases because it indicates the 50th percentile, with exactly half the results above and half below it [SOKAL & ROHLF, 1995]. While the mode (the most frequently occurring value in the distribution) is also less sensitive to outliers, it is used less frequently because it does not offer the convenience of indicating a specific percentile. Here, we present data from 48 journals within four different categories of science indicating that citations are not normally distributed. We conclude that although the means and the medians correlate closely and therefore give similar rankings of journals, the median, not the mean, is a better measure of central tendency when discussing citations per paper because the distributions deviate so strongly from normality. Methods We chose four journal categories for analysis, based on the classifications at Journal Ranking.com (http://www.journal-ranking.com/ranking/web/index.html): Biodiversity Conservation (26 journals), Engineering Marine (7 journals), Microscopy (9 journals) and Medical Ethics (6 journals). These categories covered examples from the life and physical sciences. Using the Scopus database, we then determined for each journal the total citations for each paper published in the period 2000 2006, deeming this long enough to allow citations to accumulate in slow-paced disciplines while sufficiently
recent to be relevant to the current situation. Data were collected on April 25th or 26 th 2008 (Biodiversity Conservation journals) or September 27th or 28th 2008 (all other journals). From the distribution of citations for each journal we then calculated mean, median and mode as measures of central tendency and standard error of the mean, range, skewness (the sizes of the left and right tails of a distribution relative to the mean) and kurtosis (a measure of the height of the distribution around the mean compared to the tails) as indicators of the shape of the distributions. Results With one exception, all journals showed a consistent pattern where the mean exceeded the median and the median exceeded the mode, indicative of non-normal distributions with small numbers of very high values. This is confirmed by the large positive skew (indicating a distribution with the bulk of the values to the left and a long right-hand tail) and large positive kurtosis (indicating a strongly peaked distribution with a higher probability of extreme values than a normal distribution) observed (Table 1). The sole exception was the Mer-Marine Engineers Review, which had no citations at all. While the mean and median citations per paper for all journals considered together correlated significantly (r 46 =0.98, p<0.001), the slope for the plot of means against medians was less than 1 (Figure 1). Thus mean and median citations per paper both rank journals in a very similar order, but the gap between means and medians was greatest in those journals that attracted the most citations. The mode and mean citations per paper also correlated significantly (r 46 =0.86, p<0.001). Discussion Journal citations conform to the textbook situation in which the median and the mode are better measures of central tendency than the mean because of the excessive influence of a small number of
extreme values on the mean [SOKAL & ROHLF, 1995]. Thus the means may give a misleading impression of the citation performance of the bulk of each journal s papers because only a small minority of papers will reach or exceed the mean value [JEMEC & NYBAEK, 2006]. Furthermore, any statistical comparisons between journals where the dependent variable is the number of citations per paper will probably require either a data transformation [TABACHNICK & FIDELL, 2001] or a nonparametric approach (e.g. [MEDDIS, 1984]) because of the markedly nonnormal distributions. In contrast, the median indicates the 50th percentile and therefore shows more clearly the performance of most of the papers. It may be a useful statistic for journal ranking data bases to present. While the mode is also robust against outliers, it does not offer the convenience and ease of interpretation of corresponding to a particular percentile. HARZING & VAN DER WAL [2008] concluded that because of the skewed distribution of citations the h-index was a better measure than the mean citations per paper for ranking journals. However, CALVER & BRYANT [2008] found strong significant correlations between mean citations per paper and the h-index, the g-index and the median citations per paper for conservation biology journals, while in our study the mean and median citations per paper correlated strongly. Thus the mean citations per paper appear to give a robust ranking of journals despite the skewed distributions of citations, but the absolute magnitude of the differences in mean citations between journals is probably biased and of little value. Of course, the above discussion is not applicable if one is interested in which journals publish the few papers with very high citations. However, that is a very different question to the ones we have addressed; namely, using a measure of central tendency to summarise the citation frequency of a journal s full range of papers and to rank journals within a field.
References ANDREIS, M., JOKIC, M. (2006), An impact of Croatian journals measured by analysis from SCIexpanded database in time span 1975 2001. Scientometrics, 75 : 263 288. BROWMAN, H. I., STERGIOU, K. I. (2008), Factors and indices are one thing, deciding who is scholarly, why they are scholarly, and the relative value of their scholarship is something else entirely. Ethics in Science and Environmental Politics, 8 : 1 3. CALVER, M. C., BRYANT, K. A. (2008), Pacific Conservation Biology: An authorship and citation analysis. Pacific Conservation Biology, 14 : 285 303. HARZING, A. K., VAN DER WAL, R. (2008), Google Scholar as a new source for citation analysis. Ethics in Science and Environmental Politics, 8 : 61 73. HERMES-LIMA, M., ALENCASTRO, A. C. R., SANTOS, N. C. F., NAVAS, C. A., BELEBONI, R. O. (2007), The relevance and recognition of Latin American science. Introduction to the fourth issue of CBP-Latin America. Comparative Biochemistry and Physiology, Part C, 146 : 1 9. JEMEC, G. B. E., NYBAEK, H. (2006), A bibliometric study of dermatology in central Europe 1991 2002. International Journal of Dermatology, 45 : 922 926. KADEMANI, B. S., KUMAR, V., SURWASE, G., SAGAR, A., MOHAN, L., KUMAR, A., GADERAO, C. R. (2007), Research and citation impact of publications by the chemistry division at Bhabha Atomic Research Centre. Scientometrics, 71 : 25 57. MEDDIS, R. (1984) Statistics Using Ranks: A Unified Approach, Basil Blackwell, Oxford. SOKAL, R. R., ROHLF, F. J. (1995). Biometry, W. H. Freeman and Co., New York. TABACHNICK, B. G., FIDELL, L. S. (2001) Using Multivariate Statistics, Harper Collins Publishers, New York.
TRIMBLE, V., ZACH, P., BOSLER, T. (2004), Productivity and impact of optical telescopes. Publications of the Astronomical Society of the Pacific, 117 : 111 118. YANG, K., MEHO, L. (2007), CiteSearch: next-generation citation analysis. Proceedings of the ACM International Conference on Digital Libraries. Pp.101 102 in Proceedings of the 7th ACM/IEEE Joint Conference on Digital Libraries ed by R. LARSON, E. RASMUSSEN, S. SUGIMOTO, E. TOMS. Available on-line at: http://portal.acm.org/citation.cfm?id=1255194&dl=&coll= Accessed: August 25th, 2008. YI, H., AO, X., HO, Y.-S. (2008), Use of citation per publication as an indicator to evaluate pentachlorophenol research. Scientometrics, 75 : 67 80.
Table 1. The distribution of citations for all publications in 48 journals from four subject categories over the period 2000 2006. Data are drawn from Scopus. Data from the the Biodiversity onservation category were retrieved between April 25th 2008 and April 28th 2008, while all other data were retrieved on September 27th or 28th 2008.
Figure 1. Mean and median citations per paper for 48 science journals from four categories. The line indicates a slope of 1