Appropriate and Inappropriate Uses of Bibliometric Indicators (in Faculty Evaluation) Gianluca Setti

Appropriate and Inappropriate Uses of Bibliometric Indicators (in Faculty Evaluation) Gianluca Setti Department of Engineering, University of Ferrara 2013-2014 IEEE Vice President, Publication Services and Products gianluca.setti@unife.it 2015 ECEDHA Annual Conference and ECExpo Hilton Head, SC March 13-17, 2015

Outline 1. Overview on journal bibliometric indicators 2. Show that the "quality" of a journal as measured by journal bibliometric indicators is a multidimensional concept which cannot be captured by any single indicator 3. Show that the bibliometric indicators should not be misused by giving them "more significance than they have": a) the impact of an individual paper cannot be measured by the impact of the journal in which it has appeared b) there is no strong correlation between the Impact Factor of a journal and its selectivity (rejection rate) c) the Impact Factor of a journal is not a good proxy for the probability that an individual paper will be highly cited 4. Highlight that the misuse of journal bibliometric indicators has undesired consequences and that the same is happening also for citation based analysis (and its derivative like the h-index) 5. Give some Do's and Don ts about using Bibliometrics 2

Bibliometrics Definition: Bibliometrics is a set of methods to quantitatively analyze scientific and technological literature (it is part of Informetrics, which does the same for all information) Starts to be (mis)used to evaluate impact of scientists "Quality" Aggregation Bibliometric Indicator Proxy of Quality (citations, 3 downloads, tweets, )

Journal Bibliometric Indicators, i.e. numbers, numbers, numbers Many bibliometric indicators exist, each aiming to measure "journal quality"; they should: 1. Give a result which corresponds to the technical quality of the papers published in that journal: Nature, Science or Proceedings of the IEEE and the Journal of Obscurity should have a very different value of the indicator 2. Be "fair" if applied to different areas: different areas/communities may have different citation practices (e.g., long/short citation list) 3. Be immune to external manipulation: it should be very difficult to artificially manipulate its value 5

Impact Factor and its criticisms - I Introduce by Eugene Garfield in 1972 to help librarians understand how much a journal was being used (useful in renewal process) It is an average measure of usage across an entire journal It contains no information on the impact of an individual paper For a journal in a year Pros: simple, easy to compute, known and disseminated 6

Impact Factor and its criticisms - II Cons/criticisms: 1. Only 2 years of data to account for citations may not be enough in some areas to reach the citation peak IF varies very significantly among (sub)areas Ex: In SC Eng. E&E, = 1.32; max =7 In SC Biology, = 2.10; max =11.45 In SC Bioch and Molec. Bio = 3.78; max =34.31 2. Citations are counted in the same way independently of the source (i.e. a citation obtained from Science is the same as the Journal of Obscurity ) 3. IF has an "non-consistent" definition: elements considered at the numerator are different than the denominator 4. IF is liable to active manipulation 7

Impact Factor: manipulation (1/3) How has IF been manipulated? 1. Inconsistent definition: citations to notes/"letters to the editor"/editorials count in the numerator but the same items are not counted in the denominator. They can be cited and, even more importantly, their citations count normally. Its bibliography contains 25 citations to the same journal, 24 of which count toward the 2012 IF 8

Impact Factor: manipulation (2/3) 2. Coerce self-citations: EiCs "force" authors to add citations to their journal (not necessarily to the authors) to increase IF EICs of 175/832 journals in the area of economics, sociology, psychology, and multiple business disciplines were found to "coerce" self-cites Coercing was more frequent with young authors than experienced ones Relation to area: if one journal coerces its authors other journals will most likely follow 9

Impact Factor: manipulation (3/3) 3. Citation Cartel/Stacking: EiCs or other members of editorial board of and : publish in a paper with (several) tens of citation to publish in another journal as authors to do the same Four Brazilian journals (Rev Assoc. Medic B, Clinics, J. Bras. Pneum, Acta Ortop Bras.) were found to establish a citation cartel Three Italian journals in the area of medicine (with the 10 same EiC!)

Is the phenomenon widespread? No systematic study yet: one must use JCR data: For citation cartels the systematic analysis is very difficult, but one can rely on self-citation trends: 90% 80% 70% 60% 50% 40% 30% 20% 10% 00% % EE JCR Suspended 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 90% % Laser and Particles Beams (Phy Applied), Cortex (Neuroscience), Int. Journal of Hydrogen Energy (Energy and Fuels) show an increasing self-citation trend (and similar examples exist in many more areas) Our Area: Int J. Circuit Theory and Applications and Asian Journal of 11 Control shows that we are not immune. 80% 70% 60% 50% 40% 30% 20% 10% 0% EE JCR Suspended

EE Citation Stacking highlighted in Nature The EiC of the International Journal of Sensor Networks (IJSN) used his own conference papers to perform citation stacking (in US CS department) In two of his conference articles there were 74 and 82 citations to IJSN IJSN was banned from JCR since the above citations accounted for 82% of the total citations of IJSN 12

Why this is happening? The IF was historically created to give librarians tools for deciding renewals, yet It is currently more and more used as the gold standard to evaluate the impact of an individual's research activity (for hiring, tenure, promotion, salary increase ) This use is commonly based on 2 main "assumptions". Assume that has of, then 1. Any paper published in has more impact (has received more citations) than any paper published in 2. The review process of is more stringent than the one of Are these assumptions supported by data? NO 13

Some data - I 1. Evaluation of the impact of a single paper in a journal Probability 0.15 2012-IF 3.063 0.10 0.05 Probability 0.25 0.20 0.15 0.10 0.05 2012-IF 2.621 Probability 0.20 0.15 0.10 10 20 30 40 citations 2012-IF 2.240 Probability 20 40 60 80 citations 0.20 2012-IF 1.672 0.15 0.10 0.05 0.05 10 20 30 40 citations 0 5 10 15 20 25 citations JSSC, TIT, TCAS-I, and TIA distributions of citations for 2012 to papers of 2011 and 2010 show the same shape: most papers are cited only a few times or never cited and only very few have high impact 14

Some data - II Important: regardless of IF most papers in each journal are cited only a few times (if ever) and few papers are cited many times Assuming that a randomly chosen paper in JSSC (IF=3.063) is better (has more citations) than one of TCAS-I (IF=2.240) is wrong >36% of the time Assuming that a randomly chosen paper in TIT (IF=2.612) is better than one of TIA (IF=1.672) is wrong >43% of the time Is a better proxy of impact of papers published in 2012? Consider the distribution of citations for the paper published in 2012: the trend is similar and the results of this experiment look the same. The hypothesis that that a paper published in a journal with higher IF has also received more citations is wrong about > 40% of the time journal indicators are average quantities and give therefore no indication of the quality of any single published paper

Some data - III Indication of the selectivity of a journal: if the IF of a journal is large, is the review process "very strict"? This is not supported by data (at least if one assumes valid the equation "strict review process = high rejection rate"): the correlation coefficient is on the order of 0.2 2010 Impact Factor 6 5 4 3 2 y = 2.4109x + 0.5697 R² = 0.1737 1 0 0 0.2 0.4 0.6 0.8 1 2008 Rejection Rate A. Kurmin, T. Krimis, "Exploring the Relationship Between Impact Factor and Manuscript Rejection Rates in Radiologic Journals, Acad Radiol 2006; 13:77 83 16 43 IEEE titles, Rejection Rate obtained by internal reports

Some data IV Assumption: the IF of a journal is large, papers published there are highly cited, if I publish there my paper has an higher probability to be highly cited This is not supported by data (neither in terms of correlation nor of probability) [G. A. Lozano et al., "The Weakening Relationship Between the Impact Factor and Papers Citations in the Digital Age", J. American Society for Information Science and Technology, 63(11):2140 2145, 2012] "Correlation coefficient" between IF in year of publication and citation rate in the following 2 years 17 Probability that a paper in the top 5% cited in a given year was NOT published in a journal in the top 5% of the IF ranking

Why this is happening? While the IF was historically created to help librarians, it is misused to evaluate individual's research activity (for hiring, tenure, promotion ) The unintended use of the IF made it the target and not the measure and created incentive for its manipulation According to the 2013 Nature article of Richard Van Noorden the EiCs of the 4 journals involved in a citation cartel created it because "In Brazil, an agency in the education ministry, called CAPES, evaluates graduate programmes in part by the impact factors of the journals in which students publish research" 18

Is this happening only for journals? (1/2) The use of citations to measure impact is based on the assumption that this is a form of "scientific retribution", but this is far from perfect Citations can be related to wrong or weak results (to correct or improve them), i.e. the cited result could not be significant Citations could be "rhetorical" only. Especially in the introduction to of a manuscript citations are present to "generic" reference which are related to the argument being discusses Self-citations are present also in this case and are difficult to eliminated due to authors disambiguation problems Despite the above issues, citation analysis (and derived measures such as h-index) is used as an automatic (and unique) measure to evaluate scientist performance and incentivizes misuse 19

Is this happening only for journals? (2/2) http://retractionwatch.com/2014/07/08/sagepublications-busts-peer-review-and-citationring-60-papers-retracted/ A scientist created 130 fake identities to create a peer review and citation ring. He suggested his own fake identities as reviewers of his own papers as well as those of possible fake authors. Final Goal: to increase both his publication record and his own citation index The unintended use of the citation index made it the target and not the measure and incentivized its manipulation 20

Other To solve measures IF technical to solve issues IF issues for Journal evaluation Several "successful" new indicators: 5 in either WoS or Scopus Five Year Impact Factor (5YIF) Article Influence (AI) Eigenfactor (EF) Source Normalized Impact per Paper (SNIP) Scimago Journal Ranking (SJR) Increase the citation window : 3 or 5 years Introduce subject field normalization: explicit (SNIP) or implicit (EF, AI, SJR) Exclude all (or most) self-cites: eliminate the inflation issue (EF, AI, SJR) 21 Only count equivalent scientific documents both at numerator and denominator: eliminate another cause of inflation (EF, AI, SJR, SNIP)

Popularity vs Prestige An important distinction is between indicators measuring popularity or prestige 1. Popularity indicators: are based on an algebraic formula and count citations directly independently of their source (IF, 5YIF, SNIP) 2. Prestige indicators: are based on an recursive formula and weight the influence of citations depending on their source (EF, AI, SJR) They evaluate different aspects of Journal Impact At the very minimum, one needs to use both popularity (ex. IF) and prestige (ex. AI) indicators

Addressing the issues: IEEE Initiatives- I 1. Make clear that manipulation of any bibliometric indicators is unethical 2. Promote the adoption of multiple bibliometric indicators to evaluate the impact of scientific publications and of individual papers. Actions: a) Xplore journal home pages now display Eigenfactor and Article Influence as well as IF (since version 4.6) 23

Addressing the issues: IEEE Initiatives- I 2. Promote the adoption of multiple bibliometric indicators to evaluate the impact of scientific publications and of individual papers. Actions: a) visualization of the Eigenfactor and the Article Influence, in addition to the IF, in the home page of each journal in Xplore (since version 4.6) b) visualization of the number of citations from different sources (WoS, Scopus and CrossRef) and of the number of downloads for individual papers 24

Addressing the issues: IEEE Initiatives- II 3. Educate the community on the significance of all bibliometric indicator and their proper use. Actions: a) panel discussion at the 2013, 2014 and 2015 IEEE EiCs meeting b) presentation on this subject and major IEEE conferences and other venues (NSF, ECEDHA, ) c) preparation of white papers and articles on this subject to be distributed to the IEEE community 25 DOI: 10.1109/ACCESS.2013.2261115

Addressing the issues: IEEE Initiatives- III IEEE position statement on correct use of bibliometrics (approved by BoD in 09/2013) IEEE joins several professional and scientific institutions (but none in the area of Engineering) to stress that bibliometrics cannot be used (alone) to obtain an automatic evaluation of single researcher "scientific quality" 26

Addressing the issues: IEEE Initiatives- III IEEE position statement on correct use of bibliometrics (approved by BoD in 09/2013) 1. The use of multiple complementary bibliometric indicators is IEEE joins several professional and scientific institutions (but none in the fundamentally area of Engineering) important to to stress offer an that appropriate, bibliometrics comprehensive cannot and be used balanced view of each journal in the space of scholarly publications. (alone) to obtain an automatic evaluation of single researcher 2. "scientific Any journal-based quality" metric is not designed to capture qualities of individual papers and must therefore not be used as a proxy for single-article quality or to evaluate individual scientists. 3. While bibliometrics may be employed as a source of additional information for quality assessment within a specific area of research, the primary manner for assessment of either the scientific quality of a research project or of an individual scientist should be peer review. 4. The IEEE explicitly and firmly condemns any practice aimed at influencing the number of citations to a specific journal with the sole purpose of artificially influencing the corresponding indices. A web page was created to make the statement available to the IEEE community. 27 http://www.ieee.org/publications_standards/publications/rights/bibliometrics_statement.html

Addressing the issues: IEEE Initiatives- III A web page was created to make the statement available to the IEEE community. 28 http://www.ieee.org/publications_standards/publications/rights/bibliometrics_statement.html

Addressing the issues: the rest of the landscape In approving the statement IEEE joins several other research agencies and professional organizations in the area of Physics, Medical Sciences, Biology,. 29

Some Don'ts (1/3) 1. Journal Bibliometrics indicators have been designed to evaluate journal impact but cannot be employed as a measure of the quality of single papers or to evaluate the quality of a scientists. This is particularly problematic for the IF but applies to all journal indicators Examples: a. Do not rank faculty candidates using the IF of the journal they publish in 2. The application of aggregation or filter operations to Journal or Individual Bibliometric indicators makes their use to rank scientists even a worse abuse 30 Examples: a. Do not use the sum of publication IFs or use the average of publication IFs to rank candidates b. Do not apply a threshold to IF to make a particular publication count for raises (say first quarter in a specific subject category of JCR)

Some Don'ts (2/3) 3. Avoid to apply filters also to citation indicators if the goal is an automatic evaluation of the candidate. If filter are set they should assist human judgment and not replace it! (citations are easy to manipulate) Examples: a. Do not require a minimum values for h to allow a scientist to apply for a grant b. Do not require a minimum number of citation to get promotion 4. Avoid ranking faculty/scientists according to "one/multiple bibliometric parameter(s)": It is legitimate to rank candidates who have been short-listed, e.g., for a job position, according to relevant criteria, but ranking should not be merely based on bibliometrics 31

Some Do's 1. Journal Bibliometric indicators exist, each aiming to measure the journal scientific impact and they measure it in different ways One cannot use a single indicator (neither IF, nor any other) to measure journal impact. At the very least, one needs to use a. One popularity indicator (e.g. the IF, or the 5YIF) b. One prestige indicator (e.g. the AI) Use of multiple indicators provides a much more accurate evaluation of a journal s impact and can also make evident existing anomalies 2. Individual Bibliometric indicators are statistical quantities and if the faculties/candidates have a sufficiently large publication output, citation analysis can be used (with caution!!) as an additional source of information for evaluation 32 Examples: a. Different career progression dynamics may (will) exist b. Benchmarking is fundamental especially for multidisciplinary research c. Read the contribution and apply your own judgment!!

Not everything that can be counted counts and not everything that counts can be counted (A. Einstein) e.g. do not use only numbers and read the papers!! 33

Backup-Slides 34

Removing journals from JCR - I For the first time in 2014 Thomson made (partially) public the criteria for exclusion in the JCR

Removing journals from JCR - II 36

Evaluation individuals: h-index Pros h-index: Combine both the effect of the number of paper published and the rate of citations (reduce over-performance of small journals typical of IF) Cons h-index: 1. Always grows with age of the scientist Proposed by Hirsh in 2005 as a synthetic indicator to evaluate the impact of a scientist A scientist has h-index of if of the papers it has published have citations each and the other have fewer than citations each. 2. Cannot be used for cross-disciplinary comparison (and even cross subject) 3. Non-consistent indicator. If A and B are 2 scientists, A has 4 papers with 4 citations 37 each and B 3 papers with 5 citations. If both receive a paper with 4 citations the relative performance changes ( = = 4) which is counterintuitive

A misuse of the h-factor The h-factor has been proposed (with the #of papers, #years since first publication) to predict the "success" of a scientist career "Our formula is particularly useful for funding agencies, peer reviewers and hiring committees who have to deal with vast numbers of applications and can give each only a cursory examination." 38

How to analyze self-citation trends No systematic study yet: one must use JCR data For citation cartels the systematic analysis is very difficult For self-citation trends, there are 2 ways to analyze the data: 1. Find the % of Self-Citations used in computing the Impact Factor 2. Find the Self-Citation rate per published paper % has the disadvantage of depending on "external citations", is a more appropriate measure the journal self-citations 40 behavior

Recursive (Pagerank) Prestige Measures - I Developed by Carl Bergstrom in 2007. The EigenFactor is computed by Thomson using the same algorithm used by Google to rank web pages Consider a collection of journals. Each of them is represented by a node in a network Journal gives in total 10 citations, 3 to, 5 to, and 2 to. Of course 3/10 represents the conditioned probability that a reader reads and article in, assuming that he/she started reading Markov Chain with transition matrix computed as the fraction of total citations given by 2 different journals 41 The stationary distribution of the chain gives information on the importance of each journal (probability of reading it) 3 10 Note: 2 self-feedback in the node of 10 the chain 5 is missing, i.e. selfcitations 10are not considered

Recursive (Pagerank) Prestige Measures - II In formulas: how EF i (for journal ) is roughly defined?: "self-citations" are not included (no self-feedback in the Markov Chain) It may happen that the reader at some point stop reading, than starts reading again picking a journal at random. He reads again exactly journal with a probability given by the fraction of papers published in journal with respect to the entire collection. (1- ) is the probability that the reader stop reading. Problems of the "dangling nodes", which represents journals that are only cited, but do not 42 give any citations. Not considered in this version of the formula

Recursive (Pagerank) Prestige Measures - III One needs to compute Remarks: 1. The more "important" is the journal ( is large) the more a citation from it to journal increases 2. Normalization by all citations given by journal j (citation potential) 3. The represents the probability that a random reader picking journals at random and following citation will eventually read journal 4. It is a measure per-journal and not per-paper and therefore tends to be larger for journals publishing more papers (not necessarely a problem) 43

Recursive (Pagerank) Prestige Measures - IV The Article Influence is roughly the EF normalized to the number of papers published by each journal: Per-paper measure (similar physical meaning w.r.t. the IF) Further normalization to have AI=1 for the median journal Pros (EF/AI): 1. Citations are now weighted depending on the source (a citation from Science is valued more than one from the Journal of Obscurity ) 2. Time window for computing citations ( ) is 5 years. This index is expected to exhibit fewer fluctuations over time 3. Journal self-citations are not considered. The index is less prone to external influence Cons (EF/AI): 1. More difficult to understand and compute 44 2. Not necessarily correct to eliminate all self-cites.

SJR and SNIP - I SNIP Introduced by Moed in 2010. Contained in Scopus Relative Database Citation Potential = average number of citations contained in any paper citing in period normalized in such a way that the median journal in the database has RDCP = 1 45 Raw Impact per paper = average number of citations per paper published in in period =,, by papers published in all journals present in the data base in period = Same definition as the IF. The only difference is the rolling window of 3 years (instead of 2) to collect citations

SJR and SNIP - II Assume that articles in journal are cited by articles in journals,, and Consider each paper in each citing journal Without considering the Relative Database Citation Potential for is the average number of citations contained in every journal citing it, which are in and which refers to item contained in the database. 46

SJR and SNIP - III Pros (SNIP): 1. Time window for computing citations ( ) is 3 years. This index is expected to exhibit fewer fluctuations over time 2. An explicit normalization to the citation potential for each journal is considered which should make indicators for journals of different areas more comparable 3. It is freely available from the Scopus homepage, without subscription to the database Cons (SNIP): 1. Citations are not weighted depending on the source 2. More difficult to understand and compute (even if the definition is non-recursive) 3. Self-cites are still considered Title IF SNIP SC IEEE Proceedings 6.81 5.97 Eng. E&E PLOS Biology 11.45 1.94 Biology Annual Review Biochemistry 34.32 8.27 Bioch&Mol. Bio.

SJR and SNIP - IV SJR Introduced by González-Pereira, Guerrero-Boteb, Moya-Anegónc in 2010. Contained in Scopus It has a definition similar to the Article Influence (i.e. it is a measure per-paper), bu consider self-citations up to 30% Pros (SJR): 48 1. Citations are now weighted depending on the source (a citation from Science is valued more than one from the Journal of Obscurity ) 2. Time window for computing citations ( ) is 3 years. This index is expected to exhibit fewer fluctuations over time 3. Journal self-citations are considered only partially. The index is less prone to external influence 4. Freely available from Scopus homepage Cons (SJR): 1. More difficult to understand and compute

PCA Analysis of bibliometric indicators Prestige measures Total Cites IF H Popularity measures Usage Measures 39 bibliometric indicators (popularity, prestige, cites, and usage) computed using Scimago, 2007 JCR and MESUR project for usage (SNIP not included) J. Bollen, H. Van de Sompel, A. Hagberg, R. Chute, "A Principal Component Analysis of 39 Scientific Impact Measures," PlosOne, June 2009 Compute the "principal components": 1. The problem is roughly 2-dimensional (83.4% cumulative variance) 2. Different clusters are present: prestige, popularity and total cites measure different aspects of quality 49 3. One cannot use only one indicator to "measure journal quality"

Making decisions based on multiple indicators (1/2) EF, AI and IF measure journal quality, but IF uses self-cites while EF and AI do not use them If ranking wrt IF is much greater than wrt to EF and AI there may be a problem with self-cites Go back to the LPB vs Cortex issue. With respect to "SC per paper" Cortex in 2010 is worse than Laser and Particles Beams in 2008. Why was LPB removed from JCR and Cortex was not? Laser and Particles Beams (Physics, Applied) 2007 2008 2009 2010 2011 Rk-IF 6 8 SUS 17 49 Rk-EF 32 54 SUS 45 57 Rk-AI 50 67 SUS 70 69 Cortex (Behavioral Science) 2007 2008 2009 2010 2011 8 19 7 4 4 9 15 12 15 10 15 21 17 16 14 50 Difference in RK for LPB in 2008 is much larger than for CO in 2010

Making decisions based on multiple indicators (2/2) Similar information can be extracted using a linear predictor* = (, ). A large relative difference may indicate problems 3 2.5 2 1.5 1 0.5 0-0.5-1 -1.5 2007 2008 2009 2010 2011 IJHE LPB CO Mean ± -2 51 Using more than one indicator helps to obtain a better evaluation of the journal status *Linear predictor on all Journals in 2007 to 2010 JCR which have all three IF, EF and AI

Is the phenomenon widespread? No systematic study yet: one must use JCR data For citation cartels the systematic analysis is very difficult For self-citation trends, there are 2 ways to analyze the data: 90% 80% 70% 60% 50% 40% 30% 20% 10% 00% % 9.00 8.00 7.00 6.00 5.00 4.00 3.00 2.00 1.00 0.00 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 Laser and Particles Beams (Phy Applied), Cortex (Neuroscience), Int. Journal of Hydrogen Energy (Energy and Fuels) show an increasing self-citation trend (and similar examples exist in many more areas) LPB was suspended from JCR in 2009 due to excessive SC. 54

E&E Engineering (1/2) 1. LPB 2. Int J. Circuit Theory and Applications 3. Asian Journal of Control 90% 80% % 8 7 70% 6 60% 50% 40% 30% 5 4 3 20% 2 10% 1 0% 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 0 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 Comparing with LPB and AJC: our area is "not immune" to this phenomenon. AJC was also eliminated from JCR in 2013 (and there are 55 other examples)