Towards a Bibliometric Database for the Social Sciences and Humanities

Transcription

1 Georgia Institute of Technology From the SelectedWorks of Diana Hicks April, 2009 Towards a Bibliometric Database for the Social Sciences and Humanities Diana Hicks, Georgia Institute of Technology - Main Campus Jian Wang, Georgia Institute of Technology - Main Campus Available at:

2 Towards a Bibliometric Database for the Social Sciences and Humanities Diana Hicks & Jian Wang School of Public Policy Georgia Institute of Technology April 2009 Executive Summary...2 Introduction...3 Journal lists...4 Criteria for inclusion on lists...6 A note on problems in the journal lists...8 Scholarliness analysis...9 Coverage Analysis...12 National evaluation systems...15 Recommendations...18 National Research Documentation Systems...18 Other possible approaches...20 References...21 Appendix 1 Description of journal level classifications...23 Appendix 2 Comparison of field classifications...24 This analysis was supported by Science and Technology Policy Research (SPRU), the University of Sussex, on behalf of, the ESRC/AHRC (United Kingdom), ANR (France), DFG (Germany), NWO (the Netherlands) and The European Science Foundation (ESF) 1

3 Executive Summary In the social sciences, humanities or arts it is largely impossible to substantiate statements on research excellence with reliable indicators for international benchmarking of fields and institutions. To help overcome this limitation, this report examined bibliometric systems in the social science and humanities from the perspective of assessing their potential for institutional research evaluation nationally or internationally. To assess the feasibility of an adequate bibliometric system in SSH, we must ask: how large is the SSH literature and how much of it should be counted in an evaluation? Working with limited time and resources, our efforts focused on assessing international and national journal literature using multi-disciplinary resources often used in evaluation and also ERIH. A comparison was made between six journal lists: Ulrich s, ERIH, the Norwegian reference list, the Australian ERA Humanities and Creative Arts list, WoS and Scopus. The analysis uncovered a set of issues that would arise in any attempt to establish a comprehensive database of European SSH scholarship. The size of the SSH literature cannot be estimated unless agreement is reached on the definition of literature. Although all the lists examined here are seen as lists of journal literature, the stringency of their criteria for inclusion vary and seem to determine their size. In increasing order of stringency/decreasing size we have: Ulrich s, Norwegian list, Scopus, WoS. ERIH and ERA HCA cover fewer fields and so are not comparable. Given this variability, we compared lists using a single definition of scholarliness. Restricting a journal list to scholarly, refereed material is a value held in high esteem by all parties to evaluation. However, our analysis demonstrated that the definition of scholarliness is contested with the distinction between international and national literatures pivotal. There is much more agreement for internationally oriented journals. Identifying the scholarly part of national literatures seems to be far more difficult. It is likely very difficult to devise and consistently apply criteria of scholarly quality across a range of languages. Given the importance of national language publishing in SSH, solving the problem of consistent, evidence-based criteria for journal scholarly quality that can be applied impartially and without favouritism across the range of European languages will be crucial to building a respected bibliometric infrastructure for SSH. A broadly consultative process will be required to devise an acceptable, transparent solution. Our analysis of coverage illustrates the challenges that any bibliometric infrastructure in European social sciences and humanities will face in achieving coverage of national literatures. Both the Norwegian list and ERIH aim to overcome English language bias of the big databases, and they do list more non-english language journals. Yet, there are far more academic journals in European languages than both lists cover and their coverage of English language journals is much more comprehensive than their coverage of European language journals. A brief overview of national evaluation systems suggests that the way forward is national research documentation systems in which universities submit bibliographic records of their publications and are responsible for data quality. Agencies then validate and standardize the data. Publications are differentiated according to a 2-4 level classification of the quality of the publication venue. Weighted publication counts or publication distributions across the levels are then produced. The first step in designing a research documentation system is a consultative design process to define fields, specify a journal list and define journal level categories. Each area involves difficult, subjective judgements and different processes come to different conclusions. Obtaining international agreement multiplies the difficulties. We also suggest an alternative, creating an electronic, full text infrastructure for European SSH literature. 2

4 Introduction In the social sciences, humanities or arts it is largely impossible to substantiate statements on research excellence with reliable indicators for international benchmarking of fields and institutions. To help overcome this limitation, this report will examine bibliometric systems in the social science and humanities from the perspective of assessing their potential for institutional research evaluation nationally or internationally. We will examine the criteria used to assemble journal lists in social science and humanities and then review existing evidence of the coverage of bibliometric databases. We will briefly report on institutional evaluation methods used in selected countries, placing the focus on state-of-the-art, metric oriented methods. We will suggest ways forward to build infrastructures that cover journal articles, monograph material, non textual output etc. Any successful infrastructure will need to productively engage with the scholarly community. And although this has happened in Norway and Australia, engagement never comes easily because the very idea of metrics is often antithetical to the values held by many scholars most especially in the humanities and arts. Therefore it seems useful to make explicit the values that will be embodied in any bibliometric system. While the humanities and arts place high value on the individual human experience of a single piece of work, bibliometrics is an attempt to comment on community use of a body of scholarship. Impact is the term used to describe what is measured; no claim should be made to measure quality a property inherent in an individual piece of work separate from its reception by the scholarly community. In contrast to the world of elite expert judgement, bibliometrics captures the judgements of the broad community and so tends to democratic rather than aristocratic values. Nevertheless, bibliometric impact measures always identify a small cadre of outstanding performers who compare to the bulk of scholars with much lower impact. This is the nature of the distribution of scholarly impact, which is elitist and uneven across the community. Bibliometric impact does not require consensus as a broad dispute can also create bibliometric traces. But attention is required; to be ignored is to have low impact in bibliometric measurements. Bibliometrics does not represent a substitute for scholarly judgment, rather it represents a tool to use in situations where amassing scholarly judgments would take so much time that scholars would be completely consumed and diverted from scholarly work. This is primarily an issue of scale. While assessments of individuals and their oeuvre require peer judgement, national or European scale institutional level assessments relying solely on peer judgement would create a crushing workload. It is also an issue of bias, bibliometric data can be useful also in small countries where impartiality in peer judgement is difficult to achieve. Those who employ bibliometrics place high value on scholars contributing to the public body of knowledge through publication whether it be journal articles, monograph material, or the popular press. 1 Since the publishing world is vast and quality varies, bibliometrics is interested in applying quality filters to what is allowed to be counted, as well as assessing impact once published. To employ bibliometrics is to accept that not everybody contributes equally, judgements will be made; there will be winners and losers. And judgments that traditionally were reserved for the community of scholars will be made in part by outsiders. Bibliometrics in the social science and humanities is challenging because the bibliometric infrastructure of comprehensive citation databases have largely indexed one type of literature international journal articles. In social science and humanities there are four distinct literatures: international journals, national journals, books, and enlightenment publications (Hicks, 2004). International journal articles are mostly English language, and most comprehensively indexed in databases such as Web of Science and Scopus. These are the currency of evaluation around the 1 In addition, there is great interest in extending methods to public exhibition and performance. 3

5 world. This is not wrong; using journal articles to communicate research results to an international audience is important in scholarly work. However, there is more to scholarly work in social science and humanities than the indexed international literature. Often books are written and have a very high impact (Clemens et al. 1995; Webster, 1998). National literature, not in English and published outside the US, UK or Netherlands, represents knowledge developed in and for a local context. Enlightenment literature represents knowledge reaching out to application and is found in periodicals whose goal is knowledge transfer or enlightenment of non-specialists. For example, in the US the economist Paul Krugman exerts influence through his New York Times column. Burnhill and Tubby-Hille (1994) found that in the UK projects in education [were] reaching practitioners through the Times Education Supplement, with researchers in sociology, social administration, and socio-legal studies publishing in such periodicals as New Society and Nursing Times. Kyvik (2003) found that in Norway one-half of social scientists published contributions to public debate. To add to the problems, each literature is more trans-disciplinary than comparable scientific literature. Social science and humanities bibliometric evaluation must make the best of the low citation rates associated with trans-disciplinary citation scatter and citation accumulation times which are too long for policy makers purposes (Hicks, 2004). The authors and topics associated with the four literatures overlap, but not completely, so the results of partial bibliometrics studies will not be the same as the results of an evaluation which included all four literatures. The ESF is interested in enabling full evaluation in the social sciences and humanities (SSH). This requires including all four literatures: international journals, national journals, books, and enlightenment publications as well as non-textual output in the fine arts. This report contributes to this aim. Journal lists The first issue to be addressed in assessing the feasibility of an adequate bibliometric system in SSH is how large is the SSH literature and how much of it should be counted in an evaluation? Ideally we need to know how big each of the four literatures is and how much of it is accessible using current evaluation tools in order to target resources for improvement. Working with limited time and resources, our efforts focused on assessing international and national journal literature in multi-disciplinary resources often used in evaluation and also ERIH. Our efforts were focused here because there is much less to say about the size of monograph and enlightenment literature since infrastructure in this area is embryonic or non-existent. A comparison was made between six journal lists: Ulrich s, ERIH, the Norwegian reference list, the Australian ERA Humanities and Creative Arts list (ERA HCA), WoS and Scopus. The first four are not databases of journal articles; rather they are lists of journals. WoS and Scopus are databases of articles that cover a specified list of journals, and we analyze their lists. All except ERIH and ERA HCA are comprehensive across scholarly fields. We only analyze the SSH journals in them. The analysis uncovered a set of issues that would arise in any attempt to establish a comprehensive database of European SSH scholarship. Table 1 compares these lists and a few others on several key dimensions. First note that the lists are built using two different processes. Commercial products use an editorial process; government sponsored lists such as ERIH, the Norwegian and Australian lists use peer committee based processes. The answer to the question: How big is the SSH journal literature? proves elusive as the number of journals in the lists varies quite bit. Several of the lists classify journals into different types, recognizing that broadly distinguishing levels of scholarly quality is a necessity because the literature is vast and variable. The table further notes whether the list provides the basis for a bibliographic database or a full text database with or without citations/references. The final column notes who uses the list for evaluative purposes. 4

6 Journal list Process to choose journals Table 1 Journal lists Estimated size of SSH Journal Journal list classification Database of articles Ulrich's editorial 17,900 refereed & academic ERIH peer 5,200 (3,900 verified in 3 levels Ulrich's) Norwegian peer 8,200 (6,100 unique verified in Ulrich's) 2 levels For institutional ERA HCA Australian Humanities and Creative Arts list peer 6,748 (5,538 verified in Ulrich s) submission 4 levels For institutional submission Full text Includes references/ citations Scopus Evaluative use of database in house in house WoS editorial 2,600 no, considered to diverse analysts be selective Scopus editorial 4,900 No diverse analysts GS unknown/convenience? unknown No attempted, accurate analysis extremely difficult Proposed infrastructure peer 1,000-5,000 depending on where WoS and Scopus enhancements stop No analysts would use WoS or Scopus 5

7 Criteria for inclusion on lists Ulrich s is the authoritative source of bibliographic and publisher information on more than 300,000 periodicals of all types from around the world. It includes: academic and scholarly journals, open access publications, peer-reviewed titles, popular magazines, newspapers, newsletters, and more. Ulrich s has been used in bibliometric studies as the benchmark against which WoS and Scopus coverage is measured (Archambault et al., 2006; De Moya-Anegon et al., 2007). About its inclusion criteria, Ulrich s says the following: While aiming for maximum title coverage, Ulrich's has established certain criteria for inclusion. Ulrich's covers publications that meet the definition of a serial except administrative publications of governmental agencies below state level that can be easily found elsewhere. A limited selection of membership directories, comic books, and puzzle and game books is also included. 2 Listing the entire world s periodicals, irrespective of language or country of publication is truly ambitious. In large measure Ulrich s succeeds. Studies have found only very small numbers of journals that are not yet indexed in Ulrich s. We found journals, all newer, that were not yet indexed. We told Ulrich s about these journals and they have been incorporated in the database. We bought 74k records covering active, regularly appearing periodicals in SSH fields. The Norwegian list is the reference list of journals whose papers are acceptable submissions to the Norwegian evaluation system. The list covers all fields of science, social science and humanities. The list covers scholarly publications which are defined as: presenting new insights in a form that allows the research findings to be verified and/or used in new research activity in a language and with a distribution that makes the publication accessible for a relevant audience in a publication channel with peer review. Publications in local publication channels are not counted. The level of a publication channel is defined by its mix of authors; local and so excluded journals are those with more than 2/3 of their authors from the same institution (Sivertsen, 2008). G. Sivertson kindly shared with us the SSH list containing 8,165 journals. 6,103 could be matched to Ulrich s records, and we analyze those. The European Reference Index for the Humanities, or ERIH, aimed initially to identify, and gain more visibility for top-quality European Humanities research published in academic journals in, potentially, all European languages. It is a fully peer-reviewed, Europe-wide process, in which 15 expert panels sift and aggregate input received from funding agencies, subject associations and specialist research centres across the continent. 3 ERIH includes good, peer-reviewed research journals in 15 broad disciplines of the Humanities. 4 The 15 fields are: Anthropology (Evolutionary); Anthropology (Social); Archaeology; Art, Architectural and Design History; Classical Studies; Gender Studies; History and Philosophy of Science; History; Linguistics; Literature; Music and Musicology; Pedagogical and Educational Research; Philosophy; Psychology; Religious Studies and Theology. After cleaning, we believe there are 5,197 journals in ERIH; 3,942 could be matched to Ulrich s records, and we analyze those

8 The ERA HCA list was developed as part of a larger process: 5 The Australian ERA initiative will use a range of indicators and other proxies to support the evaluation of research excellence. One of these indicators is discipline-specific tiered outlet rankings. The Australian Research Council (ARC) has consulted with the sector to assist with the development of research journal rankings, a subset of tiered outlet rankings. In late 2007 the four Learned Academies and 27 disciplinary bodies undertook a journal ranking exercise to develop draft journal rankings for their relevant disciplines. The lists have been reviewed by the ARC, in consultation with the Academies and the other list providers, to remove duplication and inconsistencies. 19,500 unique peer reviewed journals have been identified to form a draft list of ranked journals. Each journal has a single quality rating and is assigned to one or more disciplines... The consultation to develop outlet journal rankings occurred in The ERA-Humanities and Creative Arts (HCA) journal list was reviewed by discipline-specific experts to strengthen sector confidence in the accuracy of the journal rankings. The ARC will consult about discipline-specific ranked conferences, publishers' lists and other outlets with the relevant disciplines at a later time. Thomson-Reuters Web of Science (WoS) incorporates the Science Citation Index (SCI), Social Science Citation Index (SSCI) and Arts and Humanities Citation Index (A&HCI). WoS is often criticized for Anglo-Saxon bias and limited coverage. However, it is also recognized in many evaluation systems that articles published in WoS indexed journals have reached an internationally recognized standard. Journal editors feel it an honour to meet the criteria for inclusion in WoS. For these reasons, WoS s editorial standards for journal inclusion are described in some detail here: 6 The evaluation process consists of evaluation of many criteria such as, Basic Journal Publishing Standards (including Timeliness of publication, adherence to International Editorial Conventions, English Language Bibliographic Information (including English article titles, keywords, author abstracts, and cited references in the roman alphabet). Thomson Reuters also examines the journal's Editorial Content, the International Diversity of it authors and editors. Citation Analysis using Thomson Reuters data is applied to determine the journal's citation history and/or the citation history of its authors and editors. Basic Journal Standards: Timeliness of publication is a basic criterion in the evaluation process. It is of primary importance. A journal must be publishing according to its stated frequency to be considered for initial inclusion in the Thomson Scientific database. The ability to publish on time implies a healthy backlog of manuscripts essential for ongoing viability. It is not acceptable for a journal to appear chronically late, weeks or months after its cover date....thomson Scientific also notes whether or not the journal follows international editorial conventions,... informative journal titles, fully descriptive article titles and abstracts, complete bibliographic information for all cited references, and full address information for every author... Application of the peer review process is another indication of journal standards and indicates overall quality of the research presented and the completeness of cited references. Editorial Content:... Thomson Scientific editors determine if the content of a journal under evaluation will enrich the database or if the topic is already adequately addressed in existing coverage. International Diversity: Thomson Scientific editors look for International Diversity among the contributing authors and the journal s editors and Editorial Advisory Board members..... All regional journals selected must be publishing on time, have English-language bibliographic information (title, abstract, keywords), and be peer reviewed. Cited references must be in the Roman alphabet. Scopus is an Elsevier product and its inclusion policy is: 7 Scopus aims to be the most complete and comprehensive resource for all research literature in Science, Technology and Medicine and Social Science. Additional titles are selected annually for inclusion in Scopus by the external, independent CSAB based on its collective professional expertise and background. Criteria for inclusion in Scopus include, but are not limited to, the following: 1. A title must have an English-language title and publish English-language abstracts of all research articles. However, full-text articles can be in any language. 2. Timely publication of issues, with a minimum of one issue per year, is required Modified from: and

9 3. Overall quality must be high. 3.1 Assessment of a journal s quality may include, but is not limited to, the following: Authority: including the reputation of a commercial or society publisher, the diversity in affiliations of authors or if there is an editorial board the international recognition of the leading editors. Popularity & Availability: including the number of references the title has received in Scopus; the number of institutions that have subscribed to the title; and the number of times the title has been requested for inclusion. 3.2 A title must demonstrate some form of quality control (e.g. peer review). Google Scholar is a Google product. Google Scholar states that it includes: peer-reviewed papers, theses, books and abstracts and articles from academic publishers, professional societies, preprint repositories, universities and other scholarly organizations. Meho & Yang (2007) find not just the above, but also: working papers & conference papers posted on internet by authors (that is vanity publishing), bachelor s theses, presentations, grant and research proposals, doctoral qualifying examinations, submitted manuscripts, syllabi, term papers, web documents, preprints, and student portfolios. Because Google Scholar coverage is never explicitly stated, we exclude Google Scholar from this comparison of journal lists. Google Scholar is pre-eminent in providing findability. Full text indexing makes a dramatic difference to scholars searching for obscure material. For example, White (2006) searched for material on Gabriel Plattes a 17th century utopian and scientific author. In Google Scholar and JSTOR (also full text) he found articles. In WoS, which is bibliographic rather than full text, he found less than 5. Google Scholar succeeds in making information far more accessible than any other resource. But to be a basis for transparent and reproducible evaluation, the universe of included material must be specified, and Google Scholar therefore does not qualify as an evaluation infrastructure. The size of the SSH literature cannot be estimated unless agreement is reached on the definition of literature. Although all the lists examined here are seen as lists of journal literature, the stringency of their criteria for inclusion vary and it is their relative laxness that seems to determine their size. In increasing order of stringency/decreasing size we have: Ulrich s, Norwegian list, Scopus, WoS. Google Scholar cannot be included as its size is unknown, through criteria seem the most lax. ERIH and ERA HCA cover fewer fields and so are not comparable. Given this variability, we need to try to compare lists using a single definition of scholarliness. We do this below by taking Ulrich s as the comprehensive list and comparing the others with it. However, we must first point out some problems with the lists themselves. A note on problems in the journal lists Our work preparing the lists for analysis revealed that there would be problems constructing a database from journal lists established through peer consultation. These issues fall into the categories of: errors, journal status and inclusion of scientific journals. Although all lists and databases in this area are found to contain errors upon close examination, the peer lists suffer from a rather high rate of error. The ERIH list we obtained in January 2009 had not been cleaned or checked for errors. It contained duplicate records with slight differences in title or typos in ISSN in different fields, as well as erroneous ISSN numbers and titles. It contained material not identified with an ISSN (and every scholarly journal has an ISSN). Both ERIH and the Norwegian list contained old ISSNs. Journal publishing is dynamic and journals merge and change names and evolve. Tracking this accurately requires resources. We recommend that a librarian be employed to clean and correct the raw ERIH lists. The librarian could also flag non-scholarly material (see below). We recommend that an evaluation infrastructure only include 8

10 current, scholarly journals. Over time, the database would evolve with journals and managing these changes would be one complexity in building any infrastructure. ERIH and the Norwegian list contain journals that have ceased publication, are suspended, are published irregularly, and journals whose status is unknown. WoS and Scopus exclude such journals. This issue has not been noted in previous studies of WoS and Scopus coverage. Therefore, it is likely that all existing studies of WoS and Scopus coverage are unfair to the databases in that they did not narrow down the field of publications to the material the databases claim to cover. We would argue that an evaluation infrastructure should aim, like the databases, to cover active, regularly appearing journals. This is because the world of publishing is vast and many vehicles of dubious status come and go. It is not unfair to ask SSH researchers to focus on, and support, outlets with quality standards and some ongoing existence. There is in addition the problem that it is impossible to guarantee consistent coverage of a set of transient material unless resources would be infinite. ERIH contains a number of scientific journals, particularly in psychology. This is a choice ERIH may wish to make. However, if an investment were to be made in an infrastructure for evaluation of SSH work, it would be a waste of money to work with these journals, as they are already well covered in WoS and Scopus. In addition, we did not obtain science journals from Ulrich s because assessing ERIH s coverage of science fields would not be meaningful. Google Scholar presents problems of a different type; it is not in a form usable for structured analysis. Basically this is because Google Scholar is not built from structured records, that is from metadata fields. Rather than using the author, affiliation, reference etc. data provided by publishers, Google Scholar parses full text to obtain its best guess for these items. This is an imperfect process. Therefore, at one point the most published author in Google Scholar was I. Introduction. An author search in Google Scholar would not find any paper under the author s name if it had instead been tagged with Prof. Introduction as the author. Meho and Yang (2007) undertook a bibliometric study using WoS, Scopus and Google Scholar and counted the hours needed to collect, clean and standardize the data. WoS was the easiest to use at 100 hours, Scopus required 200 hours and Google Scholar 3,000 hours for the same job. They also determined the citations missed by each database due to database error. WoS missed 0.2%, Scopus 2.4% and Google Scholar 12%. WoS & Scopus failures were traced to incomplete cataloguing of reference lists. Google Scholar failures were traced to inability to match searched words and ignoring reference lists in documents if the keywords: Bibliography or References were absent. Scholarliness analysis Given the variability in accession criteria between the lists, it is useful to apply a single criterion to all lists to assess the overall scholarliness of their content. Both ERIH and the Norwegian list claim to be restricted to scholarly material. This claim is particularly strong for ERIH which claims to cover good, peer reviewed research journals. Both the ERIH and Norwegian list contain material assessed as non-scholarly by Ulrich s, for example consumer/magazines or trade journals. For example, the ERIH category history includes coin collecting magazines. We would argue that the stated intent of ERIH to cover quality, peer reviewed journals is correct; publishing in non-scholarly journals is important for reaching the general public, but should be dealt with separately as enlightenment rather than scholarly literature. If the first priority is advancing evaluation of scholarly publishing; enlightenment literature should be clearly differentiated. 9

11 We analyzed the overall scholarliness of the lists by calculating the share of non-academic material in them. Table 2 reports the share of non-scholarly material in each list judged in two ways. The first uses Ulrich s identification of a journal as refereed (which may be incomplete particularly for non-english language journals): In Ulrich s, the term refereed is applied to a journal that has been peer-reviewed. Refereed serials include articles that have been reviewed by experts and respected researchers in specific fields of study including the sciences, technology, the social sciences, and arts and humanities. The Ulrich's editorial team assigns the "refereed" status to a journal that is designated by its publisher as a refereed or peer-reviewed journal. Often, this designation comes to us in electronic data feeds from publishers. In other cases Ulrich's editors phone publishers directly for this information, or research the journal's information posted on the publisher's website. 8 The second is Ulrich s classification of a journal as academic/scholarly (which may be too broad). We can see that WoS has the most credible claim to being a purely scholarly database. Next are the Norwegian list and Scopus and finally ERIH and ERA HCA. The table also includes a breakdown by language of the journal. Combining the two methods of assessing scholarliness with the two categories of language gives a complex picture which we can simplify as follows. WoS contains the lowest share of material likely to be non-academic. The other lists will lead in some categories but be similar to their counterparts in others. ERIH is notable for the highest percentage of non-refereed material in European languages

12 Table 2 - Share of Non-academic Journals List (est. SSH size) Non-Refereed Non-Academic/Scholarly ERIH (3,900) 43% 10% English 24% 5% Non-English 79% 20% European 79% 20% Other 73% 12% ERA HCA 9 (3,817) 40% 9% English 26% 6% Non-English 70% 16% European 70% 16% Other 65% 13% Scopus (5,800) 32% 12% English 26% 11% Non-English 67% 22% European 65% 23% Other 74% 17% Norwegian (6,100) 30% 6% English 23% 5% Non-English 66% 11% European 67% 11% Other 45% 15% WoS (2,900) 16% 4% English 11% 3% Non-English 58% 10% European 60% 10% Other 20% 0 % This analysis is interesting because all the lists claim to include only scholarly, refereed material. This is a value held in high esteem by all parties to evaluation. However, the definition of scholarliness is clearly contested with the distinction between international and national literatures pivotal. Taking English language as defining international literature (which is handy but not entirely true), there is much more agreement between the lists and Ulrich s definitions of scholarly for internationally oriented journals. Identifying the scholarly part of national literatures seems to be far more difficult because the share of non-scholarly material is much higher in the non-english portion of the lists. It is unclear whether the peer or editorial processes are misguided in this, but most likely is that it is very difficult to devise and consistently apply criteria of scholarly quality across a range of languages. Indeed, WoS has only recently taken on this challenge with its campaign to extend coverage to regional journals. Given the importance of national language publishing in SSH, solving the problem of consistent, evidence based criteria for journal scholarly quality that can be applied impartially and without favouritism across the range of European languages will be crucial to 9 Excludes law for comparability with ERIH 11

13 building a respected bibliometric infrastructure for SSH. A broadly consultative process will be required to devise an acceptable, transparent solution. Coverage Analysis In tension with the value of scholarliness is the value of inclusiveness. An infrastructure adequate to representing European social science and humanities research would ideally incorporate all active, scholarly European social science and humanities journals accurately identified. How close are we to that goal? To analyze list coverage we did the following: 1. The count is at the level of journals not articles. Therefore, a journal that publishes few papers and a journal publishing many papers count equally. A different picture would be found at the article level, which would give more weight to larger journals. (See Norris & Oppenheim, 2007 for detailed analysis of this issue.) 2. The journals counted are active and regularly appearing. Irregular or defunct journals are not included. 3. The journals counted are those published in a European country or in the United States. 4. All social sciences and humanities fields were included in the Norwegian list analysis. This includes law and management. Only journals whose major subject as assigned by Ulrich s was one of the 15 ERIH fields were counted in the ERIH analysis. 5. The definition of scholarly used here was somewhat more sophisticated than that used above. All periodicals classified as academic/scholarly by Ulrich s were included except newspapers, newsletters, bulletins and magazines which were only included if they were also on one of the other lists. In addition, any periodical on any of the other lists was included if Ulrich s had not classified the periodical s type or if Ulrich s had classified the periodical as trade (as some journals, for example Energy Economics, were found to be classified as trade rather than scholarly journals). The results of the analysis are shown in a series of Venn diagrams in Figure 1. First note that the circles are larger in the Norwegian list comparison because more fields are included. Not surprisingly, we see that the lists of journals, Ulrich s, ERIH and the Norwegian list are larger than the databases of articles Scopus and WoS. The lists and databases overlap a great deal, but each contains journals not indexed by anybody else except Ulrich s. WoS is most completely incorporated in the other lists, perhaps because it is the de facto standard that others are working to improve % is the highest coverage obtained, for English language journals by ERIH, Norwegian list and Scopus. Coverage of non-english language journals is lower in every list with the Norwegian list achieving 16% and ERIH 26%. Also, there is less consensus about which non- English journals should be covered, indicated by less overlap between the lists. Journals published by large publishers, that appear to be scholarly but are not included in any list except Ulrich s include: Buddhist Studies Review (Equinox Publishing), Journal of Religion in Europe (Brill), International Journal of Contemporary Iraqi Studies (Intellect), Sikh Formations (Routledge), Wege zum Menschen (Vandenhoeck und Ruprecht), Per la Filosofia (Fabrizio\Serra Editore) and so on. These results anticipate the challenges that any bibliometric infrastructure in European social sciences and humanities will face in achieving coverage that can be defended as comprehensive enough, especially in non-english language literature. Both the Norwegian list and ERIH aim to 12

14 overcome English language bias of the big databases, and they do list more non-english language journals. Yet, there are far more academic journals in European languages than both lists cover and their coverage of English language journals is much better than their coverage of European language journals. 13

15 Figure 1 Analysis of European social science and humanities journal coverage Norwegian list coverage Norwegian list 4,494 36% Scopus 4,331 35% Norwegian list % Scopus % WoS 2,366 19% WoS, 258, 5% Ulrich's 12, % Ulrich's 5, % English language European language, not English ERIH coverage ERIH, 1,980 33% Scopus 1,534 26% ERIH 1,122 26% Scopus 250 5% WoS 199, 6% WoS 1,166 20% Ulrich's 3, % Ulrich's 5, % English language European language, not English Venn diagrams plotted using: Littlefield & Monroe, Venn Diagram Plotter, US Department of Energy, PNNL, Richland, WA,

16 A caveat must be added to this discussion. The situation is dynamic. Coverage has become a point of competition between WoS and Scopus, and they have responded in particular to ERIH. Both WoS and Scopus are adding several thousand journals to their lists. This analysis does not include these recent additions. In addition, the ERIH list is under revision, and the version used here will soon be out of date. National evaluation systems We undertook a scoping exercise to gain an initial understanding of how broadly national level research evaluation is being conducted. We drew on previous reviews of national evaluation systems in the HERA report and Geuna and Martin (Dolan, 2007; Guena & Martin, 2003). We also searched Google using the country name and research evaluation, university evaluation or higher education evaluation. These searches identified academic papers, reports and web pages from which we collected information. Also the searches identified organizations conducting evaluations, and we visited their websites as well as the website of the Ministry of Education in each country. The searches were conduced in English, except for China. For most of the countries not reviewed in the HERA report or GEUNA and MARTIN paper, the evaluation systems identified seem to be focused on education accreditation and evaluation, rather than research evaluation. Table 3 identifies the countries in which we found evaluations systems, whether the system is undergoing redesign, which agency conducts the evaluation, the type of unit evaluated and the databases used. We believe that there are some common elements in these evaluations. All of them seem to use lists of publications, and it doesn t seem that any of them except Australia use different metrics in SSH fields, though in systems based on peer evaluation such as the South African and the UK RAE, peer rating groups apply field-specific criteria. The Australian system allows for different metrics in different fields. Several systems such as Australia, UK, US and South Africa are more or less voluntary in that units are able to decide whether or not to be evaluated. It would seem that systems differ on whether funding depends on the results of the evaluation with about half of the countries allocating some funding based on the results. Table 4 provides short summaries of the evaluation systems. 15

17 Table 3 Country evaluation exercises identified Country Evaluator Level Databases used Australia* ARC disciplines within institutions data submitted & Scopus China CDGDC Discipline 10 WoS, EI, MEDLINE, CSCD, CSSCI Denmark* EVA University Finland MOE University KOTA Finland FINHEEC University Finland KAK program /project group Flanders* SOO University WoS France AERES University + Program Germany DFG University CEST Hong Kong UGC Cost centre Hungary HAS Institutions within HAS Hungary HAC University Japan NIAD-UE University Mexico Individual Netherlands VSNU Department New Zealand TEC Individual with aggregation to university Norway Government Universities, fields Data submitted, WoS & Bibsys, Norart used to verify Poland CSR University Slovenia ARRS University + Department WoS + Slovenia MOE University + Department SCI South Africa NRF Individual with aggregation to university Spain ANECA Sweden NAHE Subject areas and study program UK* RAE Department data submitted US NRC Department WoS * Countries known to be redesigning their evaluation systems 10 The evaluation unit in China is discipline, which does not correspond to department, because one department might have several different disciplines, and one discipline in one university may be located in several departments. 16

18 Table 4 Summaries of country evaluation mechanisms Country Short summary Australian Research Council (ARC), Excellence in Research for Australia (ERA) Initiative. Three categories of indicators are seen as appropriate for each discipline. Research publications and bibliometrics in focus for research quality, including publications and citations. Publications include Australia* book, book chapters, journal articles, and refereed conference publication, and journals and conferences are ranked. Publication reference period is a six years period ending on 31 Dec two year prior to the evaluation year. Institutions invited to submit data for evaluation. (Consultation Paper for ERA) China Academic Degrees & Graduate Education Development Center (CDGDC). Data collected from government agencies and universities submission. Quantitative indicators and peer review. Publications China data from SCI, SSCI, AHCI, EI, MEDLINE, and Chinese database CSCD (Chinese Sciences Citation Database) and CSSCI (Chinese Social Science Citation Information). (CDGDC website) Since 1995, funding has depended upon the volume of teaching and external research income. No other performance measures are used. (GEUNA and MARTIN) The Danish Evaluation Institute is an independent institution established in the summer of The Danish Centre for Quality Assurance and Evaluation of Higher Education (Evalueringscenteret) was established in Meta Evaluation was conducted which is mandated and not connected with funding allocation. First, questionnaire based surveys among heads of departments and heads of Denmark* faculties. Second, in-depth interviews with vice-chancellors. Finally, case studies among six educational fields covering different types of faculties (there was an evaluation from 1993 to 1997). The evaluation of the Centre was later redefined to concentrate on the lessons learned and to discuss methodological considerations for the future. The Centre was integrated into the Danish Evaluation Institute. The Danish are now implementing the Norwegian system. Sources: online paper Meta Evaluation of the Evaluations of Higher Education in Denmark, and website of EVA Universities negotiate their block grant with the Ministry of Education and a small proportion of this (3%) is performance related. Measurement uses data from the national database (KOTA), updated by universities. Data includes publication information. (HERA) Finland Finland Higher Education Evaluation Council (FINHEEC) formative institutional evaluation: peer review of a university self-evaluation. (HERA) Academy of Finland (AKA), self-evaluation by questionnaire, peer review of the questionnaires and site visits. (HERA) Steunpunt O&O Statistieken (SOO), bibliometric analysis. Due to limitations of SSCI and AHCI, Flanders* bibliometrics are not used for the allocation of funds to these agencies. (HERA) French National Agency for the Evaluation of Research and Higher Education (AERES) has evaluation France similar to its counterparts in other countries. Source: Pierre Batteau. Aspects of evaluation and accreditation in higher education in France. German Research Foundation (DFG) Funding Ranking : data from outside of universities, from Germany multiple organizations, bibliometric data: publications in international journals gleaned from the Centre for Scientific and Technology Studies (CEST) in Switzerland. (HERA) Hungarian Academy of Sciences conducted a comprehensive review of its institutes, using peer review and quantitative indicators. The idea was to support a more selective distribution. This led to a number of recommendations concerning the Academy s network, its management of resources, and the need for organizational change. Hungary Source: GEUNA and MARTIN Hungarian Accreditation Committee also has higher education evaluation similar with Japan, and Denmark. Source: HAC website 3 evaluation systems in Japan: Self-Assessment, mandatory; Certified Evaluation and Accreditation, several agencies are certified to conduct evaluation. The first one is the Japan University Accreditation Association (JUAA). National University Corporation Evaluation: performance-based evaluation of national university corporations and inter-university research institute corporations as to their Japan performances against their annual plans and the attainment of each mid-term goal. Evaluation is based on analysis of documents and site visits. These evaluations seem to be more like getting a certification of quality rather than ranking the universities. It is unclear whether biblometrics are used. 17

19 Netherlands Poland Slovenia South Africa Spain Sweden UK* US Source : NIAD-UE website Association of Netherlands Universities (VSNU) Quality Assessment of Research, peer review similar with RAE, but 4 dimensions: Scientific quality; Scientific productivity; Scientific relevance; Long-term viability. Biblometrics will be extended to AH/SS disciplines. (HERA) Committee for Scientific Research (CSR) schemes for funding allocation. (GEUNA and MARTIN) Quantitative: sum of the points received for performance R(P) and for so-called general results R(G) divided by the number of staff, giving an indicator of effectiveness (E). R(P) consists 6 indicators, including # of publications in refereed journals; and publication of books (monographs). R(G) includes numbers of citations. (HERA) Slovenian Research Agency (ARRS), qualitative methods and quantitative indicators, including publication in ISI journals, other DB journals, national journals, and books. (HERA) The Accreditation Committee was funded to evaluate academy institutions and departments, publications during the previous five years, classified by type, with ten representative publications; citation in SCI during previous 5 years, and other indicators are used. (GEUNA and MARTIN) National Research Foundation (NRF) evaluation system. Researchers apply for evaluation and choose one from among 22 panels (fields) to be evaluated in. Researchers are ranked into 6 categories, and researcher evaluation results by universities (research institutions, and other organizations) are also reported each year. Each panel has its own criterion for what are eligible as research outputs and weight for different types of outputs. Typically, they included peer review journal articles, books, conference invitations, textbooks, and so on, citation rate is also cautiously used. The National Agency for Quality Assessment and Accreditation of Higher Education of Spain (ANECA) has an education evaluation, accreditation, and certification systems. (ANECA website, and online paper: The Spanish University System ) The evaluation is of quality assurance nature, focuses on education quality, rather than research performance (e.g. publication). It also includes identifying and nominating centers of excellence, similar to Finland s KAK evaluation. Sources: Högskoleverket (Swedish National Agency for Higher Education) website, and the Swedish Universities & University Colleges Short Version of Annual Report 2008 Panel review, information submitted by universities. (HERA) University departments fill out questionnaire for National Research Council. Departmental bibliographies obtained from WoS. Opinion survey of departmental quality conducted. Final rankings based on formula devised from questionnaire and bibliometric results correlated with opinion survey. Recommendations National Research Documentation Systems The way forward for national or international level metrics-based evaluation of current research output in the social sciences and humanities is hinted at in two current metrics-based systems, the Norwegian and Australian. Both rely on national research documentation systems. In national research documentation systems universities submit bibliographic records of their publications and are responsible for data quality. Agencies then validate and standardize the data. Publications are differentiated according to a 2-4 level classification of the quality of the publication venue. Weighted publication counts or publication distributions across the levels are then produced. Such systems were seen as a promising way forward in the recent HERA report (Dolan, 2007). The first step in designing a research documentation system is a consultative design process in which the following are specified: 1. Fields 2. Journal list 3. Journal level definition Each involves difficult, subjective judgements and different processes come to different conclusions. Issues associated with the journal list have been discussed extensively above. Fields 18