Extended technical annex to chapter 5 of the Third European Report on S&T Indicators Bibliometric Analyses of World Science Robert J.W. Tijssen and Thed N. van Leeuwen Centre for Science and Technology Studies (CWTS) Leiden University The Netherlands
1 Introduction 1.1 Measuring scientific performance Scientific knowledge itself might be largely intangible, but the written information ( codified knowledge ) resulting from scientific research has an objective existence and can be studied and referred to. Some important measures of knowledge-production, transfer and utilization can thus be derived from the publications in which scientific knowledge is embodied. These publications represent an intellectual contribution to a research topic or sub-field. The majority of the scientific researchers, especially academics working in the advanced countries, publish their major research findings in the open international scientific and technical journals. In the process researchers usually acknowledge work that has influenced them by citing the relevant source documents (research articles, book, reports etc.) in the reference lists or footnotes of those publications. In doing so, they leave a paper trail of their scientific activities, research topics, collaborative linkages, and utilization of external scientific sources. The many thousands of articles, notes, letters and reviews that are published daily in those journals and millions of citations between these journal papers - provide a way of tapping into knowledge communication and dissemination processes in science and indirectly providing empirical data on research capacities and scientific activity of knowledgeproducing entities ( units ). Bibliometrics is usually defined as the application of mathematical and statistical methods to the entire scientific literature, books and documents included (Pritchard, 1969). It has become a generic term for a range of approaches directed at quantifying output levels, collaboration patterns and impact characteristics of scientific research (OECD, 1997). The advantage of bibliometric data on research documents is that they have great informative value. When authors publish they tell what they are doing, with whom they did it, when and where it was done. These literature-based measures enable systemic comparisons of scientific performance of institutions, countries and regions across a range of scientific fields (e.g. May, 1997). Most of these large scale macro-level bibliometric studies are based on the analysis of research papers and citation links between those papers published in international scientific and technical journals that are processed by the Institute for Scientific Information (ISI) - based in Philadelphia NJ, USA - for its series of international, multidisciplinary bibliographic databases. These citation indexes contain selected information from all the research papers (title and abstract, author names, author affiliations, reference list, etc.) published in about 12,000 sources. Some 8,000 of these sources are fully covered peer-reviewed international scientific and technical journals, the remainder being conference proceedings. Although the ISI-databases are quite large in terms of the number of indexed journals, the collective content of these ISI-covered journals are not necessarily always a good reflection of all worldwide scientific publication output and research activity, especially in the social and behavioural sciences, law, and the humanities. The databases are biased in favour of English-language journals. Non-English language journals are not as comprehensively indexed, and it contains more minor US journals than minor European journals. Research publications from English speaking nations (the US in particular) therefore dominate the databases. Nonetheless, one may assume that the international journal publications in these databases provide a satisfactorily representation of internationally accepted ( main stream ) research, especially high-quality laboratory-based basic research in the natural science, medical sciences and life sciences which is conducted in the advanced industrialized nations.
1.2 Bibliometric indicators The resulting quantitative data, statistics, and aggregate measures ( indicators ) enable an output-oriented systems analysis of research capabilities, activity and related scientific interactions. These bibliometric indicators, acting as impartial observers of science, provide objective data on several key features of research capabilities, collaboration, outputs and impacts. This information source therefore offers the attractive dual benefit of an external description as well as enabling easy transfer of comparative numerical data. An increasingly large number of government Science and Technology Indicators Reports contain bibliometric data describing, monitoring and comparing the scientific performance of research institutions, institutional sectors, countries and regions (e.g., NOWT, 2000: NSF, 2000; OST, 2000). The most commonly used journal-based bibliometric indicators are publication counts and citation counts. The most basic bibliometric measure is simply to count the number of publications produced. The number of scientific papers published in those international journals provides estimates of the volume of research activity and related knowledge production. Note that counts of the publications contained in the ISI databases do not necessarily equate to levels of research activity, but rather to the production of publicly available, research-based, codified knowledge for the international research community. Where publication counts measure output, counting the citations received by those papers provide a quantification of the transfer and utilization of knowledge. There are many reasons for one research article citing the other, not all of which are directly related to the scientific quality of the cited work or the contributing researchers and institutions (e.g. Weinstock, 1971). The basic premise is that a frequently cited paper has had a greater influence on subsequent research activities than a paper with no citations or only a few. The (relative) number of citations is also often considered a proxy measure of visibility in the scientific community. References in the scientific literature to research papers are also used to measure the impact of research and scientific quality. In is important to keep in mind that both publication propensities and citation rates may vary considerably between fields, disciplines and within institutions. Scientific research is usually performed within an institutional context, in most cases a university or research institute, and thus the efforts and recognition of the individual researchers affiliated with that research institution is reflected in the scientific output and prestige of the institution as a whole. By aggregating the publication output and citations at this institutional level one can measure and compare the institutional output and scientific reputation attributable to those researchers. Clearly, bibliometric indicators are partial indicators of knowledge creation and knowledge diffusion. As such they should be used in combination with information derived from other sources such as expert panels. Specific limitations of selected bibliographic data sources and pitfalls of bibliometric indicators are widely published in the scientific literature and are also outlined in the methodological appendix of the previous European Report on Science and Technology Indicators (EC, 1997). 2 CWTS bibliometric methodology and indicators 2.1 Sources and selection of papers The basic bibliographic data that used in this report by the Centre for Science and Technology Studies (CWTS) of the Leiden University in the Netherlands are retrieved from the following interrelated set of the ISI databases: the Science Citation Index, Social Science Citation
Index, and six Specialty Citation Indexes - Computer science and mathematics, Biochemistry, Biotechnology, Chemistry, Neurosciences, and Materials Sciences. The publication records were collected from the CD-ROM editions of the ISI databases (ISI- CDE). The 2000 editions cover more than 1 million papers and some 18 million citations. The coverage of these CDs is slightly less than ISI s Web of Science. Each CD-ROM covers one database year. A database year will usually include a small part of the publications from previous publication years (mainly the preceding year) due to delays in data processing. The analyses of the bibliographic data retrieved from those CD-ROMs are collected and analysed by database year (that is, the year in which the publication was issued in the citation index). All findings in the data tables refer to database years. The contents of these ISI databases are incorporated in the CWTS information system and further processed and tailored for large-scale comparative bibliometric studies. In this study CWTS uses only the research articles, reviews, notes and letters published in those journals, because they are most likely to report substantial research results and be peer reviewed. The CWTS analyses cover all journals in the ISI databases that meet three key criteria: (1) they are (almost) fully covered by ISI over an extended period of time, (2) assigned to a scientific discipline by ISI, and (3) the papers in those sources contain author address information and reference lists. 2.2 Definition of publishing units Papers are attributed to the contributing authors, institutions and countries according to the information available in the affiliate address(es) listed in the heading of the paper which is supplied by the author(s). All addresses are considered, not only the first-listed author address. In most cases, these working addresses refer to research-performing institutions (universities, research institutes, laboratories or science-based companies). Virtually all bibliographic records list the name or acronym of the related main organisation(s) and related country(ies). Institutions may appear with different names and abbreviations. Authors tend to use a wide range of variants and acronyms denoting the main organization and if included their division or unit within that organization. These affiliations are cleaned and unified both by ISI and CWTS into one standardized institutional name. Names of the countries listed in the address heading are also and unified cleaned by as part of data-entry procedures. However, the data cleaning and unification procedures do not always succeed in assigning all papers to the corresponding institutions. Specific information on the main institution is sometimes missing in the ISI records, or not in a format that enables (easy) identification of the corresponding main institution or subunit. Data cleaning and unification has been done at CWTS to the best of our ability using the information available to us. As a rule, CWTS assigns each paper only to those (main) institutions where the address information refers unmistakably to the respective (main) organization(s). CWTS cannot therefore guarantee complete coverage of all papers within the ISI databases that originated from a specific institution or author. Institution counts are based on these cleaned and unified author affiliations. A paper is attributed to an institution if the paper carries at least one author address referring of that institution. In most cases bibliometric analyses focus on legal physical entities at the main organizational level, that is an entire research institute, university, a research laboratory or firm. Departments, research groups, academic hospitals, etc. are incorporated in the respective
main organization. Network organizations, other intra-organizational entities, and virtual organisations are treated as separate units if indicated as such in the address headings. 2.3 Publication and citation counting schemes There is no fair method to determine how much money, effort, equipment and expertise each researcher, institute or country contribute to a paper and the underlying research effort. Dividing up a paper between the participating units is therefore to some extent arbitrary. Our basic assumption is that each author, main institution and country listed in the affiliated addresses made a non-negligible contribution. Each paper is therefore assigned in the full to all unique authors, institutions and countries listed in the address heading. When this whole counting scheme is used the data should be read as the number of papers in which a certain unit occurs. Hence, whole counts introduce multiple counting of all papers, for instance two or more countries in the case of international scientific collaboration. Double counting of units within the same aggregate level is avoided (e.g. countries within the same geographic region). CWTS counts the number of citations a given paper has received from all other papers in the database. No restrictions are made on the citing items in compiling the citation counts, other than that they are recorded from ISI-indexed journals covered by the CWTS information system. If the cited paper has been produced by several units the citations are also counted as whole counts and assigned in the full to each of the units. However, the absolute number of citations received is determined by many factors, especially field-dependent citation practices. Hence, citation frequencies are not very informative without international reference values. CWTS computes a normalized field-dependent citation impact value - the citation impact score - indicating whether the citation impact is above or below international average in the corresponding field or subfield. The average citation rates are differentiated for each of the four major document types (see above). CWTS adopts two reference levels as a world baseline: (1) journal(s) used by the publishing unit (author, university, country etc.) the journal citation score JCS; (2) the entire set of journals of the (sub)field the field citation score FCS. The CWTS citation analysis includes all papers and their citations received within a variable citation window. This system has the great advantage of being able to adapt to fielddependent citation speeds and can incorporate data for recent publication years. Such a window, say 1993-1999, contains all journal papers published in that time-interval, where the citations to those papers accumulate at a varying rate using a variable time span: the 1993 papers define a seven-year time interval (1993-99), those published in 1994 a six-year window (1994-99), while the citations to most recent papers published in 1999 are limited to (part of) 1999. Trend-analyses of different measurements comprises of a system of moving variable citation windows, in which each successive publication year is covered in full by a series of variable citation window lengths as time progresses. For example, an overlapping series of three-year windows: 1993-1995, 1994-1996, et cetera. Note that many research fields require citation windows of at least three to four years to produce reliable citation impact data on scientific visibility and influence.
The CWTS counting procedures also deal with the fact that many ISI-covered journals are assigned to more than one (sub)field. The absolute numbers of papers and citations published in these journals are fractionated accordingly. 2.4 CWTS classification system of research areas Scientific areas are defined by unique groupings of ISI-covered journals. This classification system follows the traditional academic division of research fields. It is not country-specific but is designed to provide an internationally acceptable division in areas and sub-areas. Each sub-field is assigned to only one discipline, which in turn is designated to one of the broad fields. The hierarchical classification system consists of three layers: 11 broad scientific fields, 26 disciplines and 178 sub-fields. The subfields and disciplines are identical to those used by CWTS for various macro-level bibliometric studies, which was especially designed for the Science and Technology Indicators Reports issued by the Netherlands Observatory of Science and Technology (NOWT, 2000). The top-layer of broad fields was designed specifically for this report in close collaboration with EC Services (DG Research). Each sub-field consists of one or more of the ISI-defined sets of scientific or technical journals, the Journal Categories. Each journal category contains a collection of journals covering the same, or close related, research topics or areas. Quite a few of those scientific journals are assigned to multiple journal categories. Wide-scope journals are therefore sometimes assigned to more than one discipline and even multiple broad fields. The prestigious general journals with broad multidisciplinary scopes, such as Nature and Science, are assigned to the broad journal category of their own, denoted by ISI as Multidisciplinary, which is listed in Table 1 as the 12 th broad field. However, like most international classification systems of research areas, this system contains a trade-off of various conceptual issues and technical considerations. Furthermore, this revised system differs significantly from the three-layer classification system used in the previous two editions of the European Science and Technology Indicators Reports which were technology-oriented rather than science-oriented. One of the most marked alterations is the shift of the large sub-field Biochemistry and molecular biology from the discipline biotechnology and its corresponding broad field Engineering to the new discipline and broad field of Basic life sciences. It is very important to stress that these journal-based sub-fields and disciplines, do not necessarily match with the organisational or administrative divisions of research-performing organisations at the main institutional level (universities, laboratories, research institutes, etc.), or primary research mission and scientific activities of research departments of groups. It is a well know fact that research units may publish papers within a wider range of journals and sub-fields, including Multidisciplinary category, than their name or scientific specialization would seem to suggest.
Table 1. CWTS hierarchical field classification system: broad fields, disciplines and sub-fields ---------------------------------------------------------------------------------------------------------------------------- 1 ENGINEERING SCIENCES Electrical Engineering Engineering - Electrical and electronic Telecommunications Materials Science Materials Science - General Materials Science - Biomaterials Materials Science - Ceramics Materials Science - Characterization and testing Materials Science - Coatings and films Materials Science - Composites Materials Science - Paper and wood Materials Science - Textiles Metallurgy and metallurgical engineering Metallurgy and mining Civil Engineering Construction and building technology 2 PHYSICS AND ASTRONOMY Physics Acoustics Crystallography Physics - General Physics - Applied Physics Atomic, Molecular and Chemical Physics - Condensed matter Physics - Fluids and plasmas Physics - Mathematical Optics Thermodynamics Physics - Miscellaneous Physics - Nuclear Physics - Particles and fields Spectroscopy Astronomy Astronomy and astrophysics Engineering - Civil Engineering - Environmental 3 CHEMISTRY Engineering - Marine Chemistry Transportation Chemistry - General Mechanical Engineering Chemistry - Analytical Engineering - Mechanical Mechanics Welding technology Engineering - Industrial Engineering - Manufacturing Robotics and automatic control Instruments and Instrumentation Instruments and Instrumentation Chemistry - Applied Chemistry - Inorganic and nuclear Chemistry - Miscellaneous Chemistry - Medicinal Chemistry - Organic Chemistry - Physical Electrochemistry Polymer Science Microscopy Photographic technology 4 MATHEMATICS AND STATISTICS Fuels and Energy Mathematics Energy and fuels Mathematics - General Engineering - Petroleum Mathematics - Applied Nuclear science and technology Mathematics - Miscellaneous Geological Engineering Statistics Mining and mineral processing Statistics and probability Geological engineering Social sciences - Mathematical methods Chemical Engineering Engineering - Chemical 5 COMPUTER SCIENCES Aerospace Engineering Computer Sciences Aerospace engineering and technology Computer applications and cybernetics Other Engineering Sciences Computer Science - Artificial intelligence Engineering - General Computer Science - Cybernetics Ergonomics Computer Science - Hardware and Operations research and management architecture science Computer Science - Information systems
Computer Science - Interdisciplinary applications Computer Science Software, Graphics, Programming Computer Science - Theory and methods 6 EARTH AND ENVIRONMENTAL SCIENCES Earth Sciences Geochemistry and geophysics Geography Geology Geosciences - General Geosciences - Interdisciplinary Remote sensing Meteorology and atmospheric sciences Mineralogy Oceanography Paleontology Environmental sciences Ecology Environmental Sciences Environmental sciences Limnology Water resources 7 BIOLOGICIAL SCIENCES Biology Biology - General Biology - Miscellaneous Botany Entomology Marine and freshwater biology Mycology Ornithology Plant sciences Zoology 8 AGRICULTURE AND FOOD SCIENCES Agriculture and food sciences Agricultural experiment station reports Agriculture - General Agriculture - Dairy and animal science Agriculture - Soil science Fisheries Food science and technology Forestry Horticulture Nutrition and dietetics Veterinary medicine Veterinary sciences 9 BASIC LIFE SCIENCES Basic Life Sciences Biochemical research methods Biochemistry and molecular biology Biomethods Biophysics Biotechnology and applied microbiology Cell biology Developmental biology Genetics and heredity Microbiology Reproductive biology Reproductive systems 10 BIOMEDICAL SCIENCES Biomedical Sciences Anatomy and morphology Andrology Cytology and histology Embryology Immunology Infectious diseases Engineering - Biomedical Medicine - Research and experimental Neurosciences Parasitology Pathology Radiology and nuclear medicine Physiology Virology Pharmacology Pharmacology and pharmacy Toxicology 11 CLINICAL MEDICINE Clinical Medicine Allergy Anesthesiology Cardiac and cardiovascular system Cardiovascular system Chemistry - Clinical and medicinal Clinical neurology Critical care Dermatology and venereal diseases Drugs and addiction Emergency medicine and critical care Endocrinology and metabolism Gastroenterology and hepatology Geriatrics and gerontology Hematology Medical informatics Medical laboratory technology Medicine - General and internal Medicine - Miscellaneous Obstetrics and gynecology Oncology Ophthalmology Orthopedics Otorhinolaryngology Pediatrics Peripheral vascular disease Psychiatry Respiratory system Rheumatology Sports science Surgery
Transplantation Tropical medicine Urology and nephrology Vascular diseases Health Sciences Drugs and addiction Hygiene and public health Nursing Public - Environmental and occupational health Rehabilitation Substance abuse Dentistry Dentistry and odontology Oral surgery and medicine 12 MULTIDISCIPLINARY Comprises of broad, general journals with a multidisciplinary collection of papers, notably the highly prestigious and highly influential journals Nature, Science, and Proceedings of the National Academy of Science
2.5 CWTS bibliometric indicators CWTS s standard research performance profile of publishing units comprises of the following list of bibliometric indicators (e.g., Moed et al., 1995; Van Raan, 1996). P presents the number of papers published of a publishing unit either an individual researcher, group, institution, country or region in individual journals, or journal sets defining sub-fields, disciplines or broad fields. See section 2.4 for an overview of those sub-fields, disciplines or broad fields. C is the number of citations received by those papers in a specified citation window. The average number of citations per publication CPP is calculated by dividing the total publication output during the entire time period, by total number of citations received during the citation window (which is not necessarily the same interval). CPPx presents the citation rate excluding the author self-citations. The average citation rate of all papers (world-wide) in the journals in which the unit has published is indicated by the mean Journal Citation Score - JCSm (see section 2.3). This index is calculated in the same way as the average citation rate CPP, but now for all publications in a set of journals instead of all publications of an unit. Comparison of an unit s CPP score with the average citation rate of its journal set (JCSm) introduces a specific problem related to journal status. For instance, if an unit publishes in prestigious (high impact) journals, and another unit in rather mediocre journals, the citation rate of articles published by both units may be equal relative to the average citation rate of their respective journal sets. But the first unit evidently performs better than the second. Therefore, a second international reference level, the mean Field Citation Score FCSm is used. It indicates a subfield-based world average which is based on the citation rate of all papers (world-wide) published in all journals of the field(s) either subfield(s) or broad field(s) - in which the unit is active, and not only the journals in which the researchers publish their papers. The measured average impact of an unit CPP the observed citation rate - is normalized to international reference values the expected citation rates, either by CPP/JCSm or CPP/FCSm. Since the citation rate of papers is now gauged against a worldwide standard incorporating the average citation rate of the corresponding journal(s) or field(s), this normalized measure is insensitive to differences in citation practices. Moreover, CWTS computes these scores separately for each of the four different document types, thus removing these (possible) distortions as well. By extension, author self-citations can be excluded entirely from the analysis, resulting in the adjusted (x) scores CPPx/JCSmx or CPPx/FCSmx. These field-normalized scores can be computed for individual papers as well as researchers and aggregate units thereof at any level through a process of summation. CWTS used the CPPx/FCSmx scores in this report. The values of this index should be interpreted as follows. If the ratio is equal to 1.0, the citation impact of the unit s set of papers equates to the world average impact score in the set of journals defining the field. Generally, a score within the range of 0.90 and 1.10 is considered average. Scores above 1.20 indicate citation impact levels significantly above world average and scores lower than 0.80 significantly below that global average. Typically, differences between scores of less than 0.20 should be interpreted with appropriate caution when ranking or comparing units.
Note that these averages are defined largely by the large scientific nations, where the EU-15 and USA each account for some 30% of the world total. When interpreting these data at the institutional level, one should keep in mind the following important features of citation data: high citation scores can be caused by only a few extremely highly cited papers; (2) as the size of the publication output of an institution increases in a discipline, it becomes more difficult to achieve a very high average citation score in a field due to the natural tendency of scores to converge to the worldwide average (=1) as the sample of papers grows larger. Moreover, the citation impact scores depend in part on (3) the presence of English-language papers, methodological papers or internationally coauthored papers, each of which are usually more highly cited than other types of research papers in scientific journals. 3. Identification of main research institutions in the EU-15 The selection comprises of a top 20 main research-performing institutions i.e. physical and legal entities at the high aggregate level - in each of the largest EU-15 countries (France, Germany, Italy, Sweden, Netherlands, Spain and United Kingdom) and the top 10 in the smaller EU-15 countries. The following step-wise procedure was adopted to generate a representative set of the largest institutions in each country while taking into account the relative size of disciplines: Step 1 Select the main institutions contributing the largest number of publications in ISIcovered journals for each separate discipline. The disciplines are defined by journal sets according to the CWTS field classification system. The procedure is restricted to the disciplines belonging to the natural sciences, medical sciences, life sciences, mathematics and statistics, and engineering (see Table 1 of technical annex). The publication counts are based on the number of papers published in the four-year timeinterval 1996-1999 according to a full counting scheme wherein each paper is attributed in the full to each of the main institutions listed in the author address(es). International research institutions are excluded. A lower threshold of 60 papers is set per discipline (an average of 15 per year) in order to be included in the selection. Luxembourg-based institutions were discarded due to insufficient numbers of papers. Step 2 If step 1 does not provide the required number of institutions for a country, this procedure is then repeated for the 2 nd in the ranking of each discipline. This process is continued with the 3 rd in the ranking, and so forth, until the required number of institutions is reached. In case of ties or an excess of additional entries in the last stage of the selection, the remaining positions go to those institutions with the largest numbers of papers in the corresponding discipline. Hence, the absence of an institution s name in the final selection indicates that the institution was not amongst the most actively publishing in any discipline, or it did not meet the lower output threshold. This discipline-dependent selection criterion ensures that the larger institutions active in the less prolific disciplines (e.g. the engineering sciences) are also included. Note that the selection of these main research institutions was based solely on their number (co-)authored research papers published in scientific and technical journals indexed
by ISI, irrespective of the citation impact of those papers or the productivity of the research personnel producing these papers. Hence, the institutions on this list do not necessarily have higher impact scores and/or higher productivity rates than those excluded from this selection.
References EC European Commission, European Report on Science and Technology Indicators, Brussels/Luxembourg, Report EUR 17639, 1997. NOWT-Nederlands Observatorium van Wetenschap en Technologie, Wetenschaps- en Technologie- Indicatoren 2000, Leiden/Maastricht: CWTS, MERIT, 2000. NSF National Science Foundation, Science and Engineering Indicators 2000, Washington: National Science Board (NSF), 2000. May, R., The scientific wealth of nations, Science, 275, 7 February, 793-796, 1997. Moed, H., De Bruin, R., and Th.N. van Leeuwen, New bibliometric tools for the assessment of national research performance: database description, overview of indicators and first applications, Scientometrics, 381-422, 1995. OECD, Bibliometric indicators and analysis of research systems: methods and examples, Paris: OECD-STI working papers 1997/1, 1997. OST Observatoire des Sciences et des Techniques, Science & Technologie Indicateurs 2000, Paris: Economica, 2000. Pritchard, A., Statistical bibliography or bibliometrics, Journal of Documentation, 24, 348-349, 1969. Van Raan, A.F.J., Advanced bibliometric methods as quantitative core of peer review based evaluation and foresight exercises. Scientometrics 36, 397-420, 1996. Weinstock, N. Citation indexes, in Kent, A. (Ed.). Encyclopedia of Library and Information Science, New York: Marcel Dekker, Vol. 5, 16-41, 1971