The problems of field-normalization of bibliometric data and comparison among research institutions: Recent Developments

Size: px

Start display at page:

Download "The problems of field-normalization of bibliometric data and comparison among research institutions: Recent Developments"

Candace Chandler
6 years ago
Views:

1 The problems of field-normalization of bibliometric data and comparison among research institutions: Recent Developments Domenico MAISANO

journal papers, conference proceedings, book chapters,

patents, university spin-offs, consulting services etc )

2 Evaluating research output 1. scientific publications (e.g. journal papers, conference proceedings, book chapters, monographs, etc ) addressed to the scientific community; 2. technology transfer applications (e.g. patents, university spin-offs, consulting services etc ) addressed to the industry and the whole socio-economic system; 3. specialized courses and research seminars, popularization of science; D. Maisano The problems of field-normalization of bibliometric data, 12 th March

Several objects of evaluation Groups of scientific publications concerning: single scientists (competitive examinations for research position/promotion or funding at universities); groups of

3 Several objects of evaluation Groups of scientific publications concerning: single scientists (competitive examinations for research position/promotion or funding at universities); groups of scientists (e.g. of the same dept., school/faculty, university or research institution); scientific journals. D. Maisano The problems of field-normalization of bibliometric data, 12 th March

4 Why journals? For (at least) two fundamental reasons: supporting subscription and selection decisions of librarians; helping an author to see more clearly the characteristics of the showcases (journals) in which the products (papers) of his/her work have been (or will be) exposed. D. Maisano The problems of field-normalization of bibliometric data, 12 th March

5 Typical approaches The ideal solution is probably given by Peer Review committees. - However, it is not always objective and transparent; - Also, it is not practicable in case of large-scale evaluations (i.e., concerning hundreds or even thousands of articles). Citation Analysis is now used in almost (all) nations around the globe with a sizeable science enterprise. - It has been known for more than one century; - A citation index for science was first described in 1955 by Eugene Garfield; - Important and continuous growth in the last 25 years (evolution of database systems, Internet, etc.) D. Maisano The problems of field-normalization of bibliometric data, 12 th March

6 (a) What exactly are citations? (b) P 2 (c 2, r 2 ) P 5 (c 5, r 5 ) P 9 (c 9, r 9 ) P 1 (c 1, r 1 ) P 4 (c 4, r 4 ) P 8 (c 8, r 8 ) P 11 (c 11,r 11 ) P 7 (c 7, r 7 ) c i i-th paper r i P i (c i, r i ) P 3 (c 3, r 3 ) P 6 (c 6, r 6 ) P 10 (c 10,r 10 ) (P i ) (c) Fig. 1. (a) Representation of the scientific literature by a graph. (b) Nodes represent scientific papers while arcs represent mutual citations. (c) Regarding the i-th paper (P i ), c i denotes the total citations received (incoming arrows), while r i denotes the total citations made or references (outgoing arrows). P C = c = R = r i i = 1 i = 1 D. Maisano The problems of field-normalization of bibliometric data, 12 th March P i P P 1 1 c = c = r = r P i i = 1 P i = 1 i

7 Bibliometric Indicators They generally concern: - scientific journals; - single researchers; - groups of researchers (depts., scientific disciplinary sectors, universities, etc.). D. Maisano The problems of field-normalization of bibliometric data, 12 th March

8 Typical analysis dimensions Productivity; Impact/Diffusion; Regularity; Affinities between documents; Co-authors networks; D. Maisano The problems of field-normalization of bibliometric data, 12 th March

9 Brief excursus of most popular Indicators 1) P total number of publications indicator of productivity 2) C total number of citations received indicator of overall impact/diffusion 3) CPP (=C /P ) citations per paper indicator of average impact/diffusion 3.1) Impact Factor 3.2) Immediacy Index D. Maisano The problems of field-normalization of bibliometric data, 12 th March

CPP: variations on a theme (1) 3.1) (Journal) Impact Factor (JIF ) Ref. year = 2003 A specific time-window B=70 articles published during the two preceding years (i.e., 2001 and 2002).

10 CPP: variations on a theme (1) 3.1) (Journal) Impact Factor (JIF ) Ref. year = 2003 A specific time-window B=70 articles published during the two preceding years (i.e., 2001 and 2002). A=81 number of times these articles were cited by indexed journals during JIF = A/B = 81/70 = 1.16 Garfield (1972) showed that this 2-year citation time-window is wide enough to include a significant number of citations and dynamic enough to measure the evolution of scientific journals D. Maisano The problems of field-normalization of bibliometric data, 12 th March

11 CPP: variations on a theme (2) 3.2) Immediacy Index (II ) In a given year, II of a journal is the average number of citations to those papers that were published during the same year. The journal Immediacy Index is a measure of how quickly the average article in a journal is cited. Example: Ref. year = 2003 B=30 articles published during A=11 number of times these articles were cited by indexed journals during II = A/B = 11/30 = 0.37 D. Maisano The problems of field-normalization of bibliometric data, 12 th March

12 4) Hirsch (h) index h is defined as the number such that, for a general group of papers, h papers received at least h citations while the other papers received no more than h citations (Hirsch, 2005). citations for each rank publication h-core 2 dimensions: publications productivity citations diffusion/impact D. Maisano The problems of field-normalization of bibliometric data, 12 th March

13 Something on the h-index scale of measurement h is a Ν number defined over an ordinal scale with only equivalence and ordering properties. The distance, in terms of average effort between two consecutive classes is gradually increasing: h N D. Maisano The problems of field-normalization of bibliometric data, 12 th March

14 The h-index does not allow compositions Author (a) Author (b) Author (a) + (b) paper id c i rank h-core paper id c i rank h-core paper id c i rank h-core A 7 1 E 18 1 E 18 1 B 5 2 F 16 2 F 16 2 C 5 3 G 8 8 G 8 3 D 3 4 H 6 9 A 7 4 I 1 10 H 6 5 h=3 B 5 6 h=4 C 5 7 D 3 8 I 1 9 h=5 D. Maisano The problems of field-normalization of bibliometric data, 12 th March

15 5) Indicators considering the reputation of citations According to the indicators seen before, a citation has exactly the same prestige of another. Other more complex indicators weigh citations depending on the impact of the citing papers: a paper is important if it is endorsed by important papers. PageRank-inspired indicators websites links scientific papers citations D. Maisano The problems of field-normalization of bibliometric data, 12 th March

16 Simplifying hypotheses for the previous indicators Role of citation. Self citations. Co-authorship. Citation importance. Article types. Same scientific field. Database reliability. D. Maisano The problems of field-normalization of bibliometric data, 12 th March

17 The need for normalization Journals with different productivity (P ); Scientists with different seniority (Y ); Research institutions with different staff No. (N ); Papers with a different number of (co-)authors (A ); Articles from (sub-)fields with different citation propensity. D. Maisano The problems of field-normalization of bibliometric data, 12 th March

18 The problem of field-normalization All the previous indicators must regard a set of papers within the same field/discipline. Different scientific fields may have: 1. different propensities to the practice of citation (citation rates); 2. differences of maturing of citation impact (citation accumulation). D. Maisano The problems of field-normalization of bibliometric data, 12 th March

19 Average citations of papers issued in 2000 c 49 c 13 7 c c 19 Warning: do not compare apples with oranges! D. Maisano The problems of field-normalization of bibliometric data, 12 th March

20 Field-based groups of papers discipline 1 1, 1 discipline 2 c r c 2, r2 in general c d r d On the average: more references discipline 3 more citations received (and vice-versa) c, r 3 3 D. Maisano The problems of field-normalization of bibliometric data, 12 th March

A common feature of field-normalized indicators A common feature of field-normalized indicators is that they are based on the comparison between: (1) the amount of citations received by a group of

21 A common feature of field-normalized indicators A common feature of field-normalized indicators is that they are based on the comparison between: (1) the amount of citations received by a group of publications examined, (2) a comparison term (CT ) given by the expected number of the citations received/made by analogous publications in the specific discipline(s) of interest. CT should represent the citation propensity of the publications of interest. D. Maisano The problems of field-normalization of bibliometric data, 12 th March

22 How to determine CT? 1 st issue: selection of a reference sample of publications with similar citation propensity. e.g., same journal, superimposed classifications, neighborhood 2 nd issue: use an indicator based on references or citations received. 3 rd issue: choice of the central tendency indicator. e.g., mean, median, harmonic mean, etc D. Maisano The problems of field-normalization of bibliometric data, 12 th March

23 Sample selection: an unstable balance point! goodness of estimation Low representativeness confusion among different (sub-)fields sample size D. Maisano The problems of field-normalization of bibliometric data, 12 th March

24 CTs for some popular field-normalized indicators 1 st issue: 2 nd issue: 3 rd issue: Sample selection technique Refs or Cites? Central Tendency Indicator MNCS (Waltman et al., 2010) Classification dependent Cites Mean value Fractional Citation Counting (Leydesdorff & Opthof, 2010) Neighborhood (citing papers) Refs Harmonic Mean (of the references) Audience Factor (Zitt & Small, 2008) Neighborhood (journals of citing papers) Refs Harmonic mean (of the average references) SNIP (Moed, 2009) Neighborhood Refs Mean value Secondary aspects: (1) size of the time-window for counting the citations received (or made) by the publications examined, (2) the way of determining the neighbourhood of a publication, etc D. Maisano The problems of field-normalization of bibliometric data, 12 th March

25 Some popular field-normalized indicators Mean Normalized Citation Score (MNCS alias New Crown Indicator) MNCS 1 P c = i P e i= 1 i being: P the publications of interest, denoted by i =1,, P ; c i the number of citations of publication i ; e i the expected number of citations of publication i given the field in which publication i has been published. D. Maisano The problems of field-normalization of bibliometric data, 12 th March

26 MNCS (New Crown Indicator) Mech. Eng. e 1 = CPP Mech.Eng. c 1 =2 J1 J2 J3 =1.2 Ind. Eng. MNCS 1 = P ci P e i= 1 i c 1 =1 J4 J5 e 1 = CPP Ind. Eng. = = = 1.17 D. Maisano The problems of field-normalization of bibliometric data, 12 th March

27 Some popular field-normalized indicators Overlappings are very frequent! D. Maisano The problems of field-normalization of bibliometric data, 12 th March

28 D. Maisano The problems of field-normalization of bibliometric data, 12 th March

29 Some popular field-normalized indicators Indicators based on Fractional Citation Counting (FCC) FCC being: FCC i field-normalized indicator relating to a (i-th) publication of interest (i =1,, P ); j order number of a citation (j =1,, m ); r i,j HM i m 1 m = = = = 1, m j ri j 1 m / j= 1 ri, j m HM citations made (i.e., references) by the j-th citing document; harmonic mean of the r i,j values relating the i-th publication. D. Maisano The problems of field-normalization of bibliometric data, 12 th March

30 FCC (Indicator based on Fractional Citation Counting) publications of interest (and corresponding cites): c 1 = 3 citing publications (and corresponding refs): 1-st citing paper 2-nd citing paper r 1 = 5 r 2 = 7 Fractional contribution: 1/5 1/7 1/6 3-rd citing paper r 3 = 6 c 2 = 4 1-st citing paper r 4 = 3 2-nd citing paper r 4 = 3 3-rd citing paper r 5 = 4 1/3 1/3 1/4 1/8 4-th citing paper r 6 = 8 D. Maisano The problems of field-normalization of bibliometric data, 12 th March

This technique is adaptive since the CT is calculated considering the neighborhood of the publication(s) of interest.

31 Some popular field-normalized indicators Indicators based on Fractional Citation Counting Normalization of citations received is performed before joining them by an aggregated indicator (a priori normalization). This technique is adaptive since the CT is calculated considering the neighborhood of the publication(s) of interest. (typically consisting of the set of publications citing or being cited by them). D. Maisano The problems of field-normalization of bibliometric data, 12 th March

32 Some popular field-normalized indicators SNIP (Source Normalized Impact per Paper) publications of interest (and corresponding cites): c 1 = 3 citing publications (and corresponding refs): 1-st citing paper 2-nd citing paper r 1 = 5 r 2 = 7 3-rd citing paper r 3 = 6 c 2 = 4 4-th citing paper r 4 = 3 5-th citing paper r 5 = 4 6-th citing paper r 6 = 8 c 3 = 2 7-th citing paper r 7 = 5 - Applicable to Journals - Implemented by SCOPUS RIP = Avg ( c i ) = 3 8-th citing paper r 8 = 7 RIP 3 SNIP = K = K = 0.53 K DCP 5.6 DCP = Avg( r j ) = 5.6 D. Maisano The problems of field-normalization of bibliometric data, 12 th March

33 A common feature At the risk of oversimplifying, most of the previous field-normalized indicator can be expressed as: CPP /CT being: CPP citations per paper relating to a set of publications of interest CT a comparison term estimating the citation propensity of citing papers. Most of the previous indicators apply to homogeneous groups of articles i.e., from the same (sub-)field. Sometimes articles to be analyzed are heterogeneous! D. Maisano The problems of field-normalization of bibliometric data, 12 th March

34 Success-index Requisites: Simplicity and immediacy of meaning (likewise h-index); Applies to groups of heterogeneous papers (in terms of citation propensity). D. Maisano The problems of field-normalization of bibliometric data, 12 th March

35 Basic principle of Success-index The growth of impact/diffusion of a paper (tree) soil (literature: refs) crop (citations) D. Maisano The problems of field-normalization of bibliometric data, 12 th March

36 The expected crop (CT ) will depend on the type of tree (discipline). CT(Oranges) CT(Bananas) CT(Lemons) D. Maisano The problems of field-normalization of bibliometric data, 12 th March

37 Trees: A successful tree (paper) will produce more fruit than expected (CT ). Tree 1 (O) Tree 4 (B) Tree 2 (O) Tree 5 (B) Tree 3 (L) Tree 6 (O) CTs: CT(Oranges): CT(Bananas): CT(Lemons): D. Maisano The problems of field-normalization of bibliometric data, 12 th March

38 Example paper id c i CT i success-core A 7 4 B 7 5 C 6 6 D 4 3 E 3 5 F 2 5 G 1 4 success-index=3 D. Maisano The problems of field-normalization of bibliometric data, 12 th March

39 Philosophy of Success-index Developed by an intuition of Kosmulski (2011), Success-index makes it possible to isolate a subset of publications, defined as successful papers, among a group of publications examined e.g., those associated to a scientist or a journal. The (non) success-state of a paper is relative to the citation propensity of homologous papers. D. Maisano The problems of field-normalization of bibliometric data, 12 th March

40 How to determine CT i terms? CT i is an estimate of the number of citations that a i th publication, in a certain scientific context and period of time, should potentially achieve. (I) (II) It is given by the mean/median number of citations made/received by articles close to the publication of interest. (III) D. Maisano The problems of field-normalization of bibliometric data, 12 th March

41 How to determine CT i terms? goodness of estimation Low representativeneness confusion among different (sub-)fields sample size D. Maisano The problems of field-normalization of bibliometric data, 12 th March

42 1 st Example: 2 scientists from different disciplines A Chemist VS a Computer Scientist For simplicity and practicality, CT will be calculated as the median number of references made by the articles published by the same journal and year of the i-th publication concerned. D. Maisano The problems of field-normalization of bibliometric data, 12 th March

43 1 st Example: 2 scientists from different disciplines S1 (anonymous Chemist) S2 (anonymous Computer Scientist) Paper No. Year c i CT i success-core Paper No. Year c i CT i success-core P= Total - C=833 - h-index= Mean Median success-index= St.dev P= Total - C=416 - h-index=13 Mean Median success-index=7 St.dev D. Maisano The problems of field-normalization of bibliometric data, 12 th March

44 Important properties of Success-index Scale type h N effort Success N D. Maisano The problems of field-normalization of bibliometric data, 12 th March

45 Important properties of Success-index Composition of publication portfolios The success-state of a paper originates from its (1) received citations and (2) specific CT. it does not depend on the other publications statistics. Thanks to the ratio property of the scale, it is possible to aggregate blocks of publications (even from several sub-fields). E.g., papers from different journals, years, scientists, departments, etc.. D. Maisano The problems of field-normalization of bibliometric data, 12 th March

46 Important properties of Success-index Composition of publication portfolios Author (a) Author (b) Author (a) + (b) paper id ci rank h-core = ( ɶ ) CTi r successcore i i JY CT i i r successcore i core JY CT r success- JY A E E B F F C G G D H A I H h=3 Success-index=2 B h=4 Success-index=5 C D I paper id ci rank h-core = ( ɶ ) paper id ci rank h-core = ( ɶ ) h=5 Success-index=7 D. Maisano The problems of field-normalization of bibliometric data, 12 th March

47 Practical implications Considering the same capacity of producing successful papers, it is expected that the Success-index should increase proportionally with the (different types of) resources used. Very versatile for different types of normalizations: Resources N (staff No.) Y (years of activity) P (No. of papers of a journal) I (unit of investment) Normalization Success/N Success/Y Success/P Success/I D. Maisano The problems of field-normalization of bibliometric data, 12 th March

48 A (potential) drawback of the Success-index Excess citations are not taken into account. Condition 1 paper id c i CT i success-core A 7 4 B 7 5 C 6 6 D 4 3 E 3 5 F 2 5 G 1 4 success-index=3 Condition 2 paper id c i CT i success-core A B C 6 6 D 4 3 E 3 5 F 2 5 G 1 4 success-index=3 However, a partial patch to fix this issue is to use variable CTs (sensitivity analysis). D. Maisano The problems of field-normalization of bibliometric data, 12 th March

49 Example of sensitivity analysis Goal: comparison between two publication portfolios (A and B) CT calculation approach: CT i terms relating to individual papers are determined considering the distribution of the references made by the articles published by the same journal and year of the i-th publication concerned. Sensitivity analysis: apart from the median (5 th decile), additional CTs are introduced with reference to other points of the distribution (e.g., 4 th and 6 th decile). 0 (4 th decile) CT L ( ) 2 0 median (5 th decile) CT( ) = rɶ ( ) M JY i (6 th decile) CT H ( ) D. Maisano The problems of field-normalization of bibliometric data, 12 th March

50 How to deal with multiple CTs? Success index Portfolio A Portfolio B CT (L) CT (M) CT (H) (threshold) severity D. Maisano The problems of field-normalization of bibliometric data, 12 th March

51 2 nd Example: groups of scientists in the same field Comparison among four groups of scientists from four different Italian Universities, within the same disciplinary sector (SSD). Analysis of their overall scientific production over the period from 2006 to Let s start with 2006: 2006 Univ. N P C CPP h A B C D Citation statistics were retrieved from SCOPUS D. Maisano The problems of field-normalization of bibliometric data, 12 th March

52 2 nd Example: groups of scientists in the same field For each article, a CT is calculated following an approach inspired by Moed s DCP. Next, Success-indices concerning each group are calculated: 2006 Univ. N P C CPP h A B C D Success-index D. Maisano The problems of field-normalization of bibliometric data, 12 th March

53 Results concerning 2006 Success (in 2006) Success / N (in 2006) (Success-index per capita) A B C D A B C D D. Maisano The problems of field-normalization of bibliometric data, 12 th March

54 Analysis in the whole 2006-to-2008 period Results relating to individual years can be aggregated together (property of composition) in order to calculate the Success-indices with respect to the whole 2006-to-2008 period. Success-indices in 2006 Success-indices in 2007 Success-indices in the whole 2006-to-2008 period Success-indices in 2008 D. Maisano The problems of field-normalization of bibliometric data, 12 th March

55 Composition Success Success / N (Success-index per capita) A B C D 0.0 A B C D 2006 to D. Maisano The problems of field-normalization of bibliometric data, 12 th March

56 Analysis in the whole 2006-to-2008 period Comparison among four groups of researchers within the same disciplinary sector (SSD) from different Italian universities over the whole period from 2006 to 2008: 2006 to 2008 Univ. N P C CPP h A B C D Success-index D. Maisano The problems of field-normalization of bibliometric data, 12 th March

57 Results concerning the 2006-to-2008 period Success (2006 to 2008) Success / N (2006 to 2008) (Success-index per capita) A B C D A B C D D. Maisano The problems of field-normalization of bibliometric data, 12 th March

58 What about CT distributions? Statistics relating to the CT distributions in the whole 2006-to period: 2006 to 2008 Univ. Mean Median St.Dev. IQR A B C D D. Maisano The problems of field-normalization of bibliometric data, 12 th March

59 3 rd Example: university depts. within different fields Comparison among two university departments of PoliTo, within different disciplines. Analysis of their overall scientific production over the period from 2006 to Let s start with 2007: 2007 Dept. N P C CPP h D D Citation statistics were retrieved from SCOPUS D. Maisano The problems of field-normalization of bibliometric data, 12 th March

60 3 rd Example: university depts. within different fields For each article, a CT is calculated following an approach inspired by Moed s DCP. Next, Success-indices concerning each department are calculated: 2007 Dept. N P C CPP h D D Success-index D. Maisano The problems of field-normalization of bibliometric data, 12 th March

61 What about CT distributions? Statistics relating to the CT distributions in 2007: 2007 Dept. Mean Median St.Dev. IQR D D % CT distribution in % 20% D1 D2 10% 0% D. Maisano The problems of field-normalization of bibliometric data, 12 th March

62 Results concerning 2007 Success (in 2007) Success / N (in 2007) (Success-index per capita) D1 D2 0 D1 D2 D. Maisano The problems of field-normalization of bibliometric data, 12 th March

63 Results concerning the 2006-to-2008 period Success (2006 to 2008) Success / N (2006 to 2008) (Success-index per capita) D1 D2 0 D1 D2 D. Maisano The problems of field-normalization of bibliometric data, 12 th March

64 4 th Example: academic institutions Comparison among three Italian Universities. Analysis of their overall scientific production over the period from 2006 to Let s start with 2006: 2006 Univ. N P C CPP h U U U Citation statistics were retrieved from SCOPUS D. Maisano The problems of field-normalization of bibliometric data, 12 th March

65 4 rd Example: academic institutions For each article, a CT are calculated following an approach inspired by Moed s DCP. Next, Success-indices concerning each group are calculated: 2006 Univ. N P C CPP h U U U Success-index D. Maisano The problems of field-normalization of bibliometric data, 12 th March

66 Results concerning Success (in 2006) Success / N (in 2006) (Success-index per capita) U1 U2 U3 0.0 U1 U2 U3 D. Maisano The problems of field-normalization of bibliometric data, 12 th March

67 Results concerning the 2006-to-2008 period Success (2006 to 2008) Success / N (2006 to 2008) (Success-index per capita) U1 U2 U U1 U2 U3 D. Maisano The problems of field-normalization of bibliometric data, 12 th March

68 What about CT distributions? Statistics relating to the CT distributions in the whole 2006-to period: 2006 to 2008 Univ. Mean Median St.Dev. IQR U U U % 25% 20% 15% U1 U2 U3 10% 5% 0% D. Maisano The problems of field-normalization of bibliometric data, 12 th March

69 The rush to publish implicates more successful papers (?) 900 U P Success Year D. Maisano The problems of field-normalization of bibliometric data, 12 th March

70 Concluding remarks (1) Field normalization is often neglected. The problem of sub-field normalization when comparing scientific publications within the same field is even subtler and trickier. Field normalization is a sine-qua-non condition when evaluating multidisciplinary research institutions. In the Literature, most of the field-normalized indicators concern journals. D. Maisano The problems of field-normalization of bibliometric data, 12 th March

71 Concluding remarks (2) Success-index is an indicator of immediate meaning, able to deal with multidisciplinary groups of papers. applicable to the publication output of entire research institutions. Also, it is very flexible so as to perform other kinds of normalization (e.g., by years of activity, staff No., etc.). D. Maisano The problems of field-normalization of bibliometric data, 12 th March

72 Open Issues A crucial point is the determination of CT i.e., estimator of the citing propensity within a specific (sub-)field. Probably, the best approaches are those based on the citation propensity of citing articles of a journal. An intense empirical study should be performed to evaluate how robust the possible estimation approaches are. D. Maisano The problems of field-normalization of bibliometric data, 12 th March

73 Thank you for your attention website: D. Maisano The problems of field-normalization of bibliometric data, 12 th March

INTRODUCTION TO SCIENTOMETRICS. Farzaneh Aminpour, PhD. Ministry of Health and Medical Education

INTRODUCTION TO SCIENTOMETRICS Farzaneh Aminpour, PhD. aminpour@behdasht.gov.ir Ministry of Health and Medical Education Workshop Objectives Scientometrics: Basics Citation Databases Scientometrics Indices