One size doesn t fit all: On the co-evolution of national evaluation systems and social science publishing

Confero Vol. 1 no. 1 2013 pp. 67 90 doi:10.3384/confero13v1130117 One size doesn t fit all: On the co-evolution of national evaluation systems and social science publishing Diana Hicks I n recent decades governments have sought greater accountability from those who receive public money. In this environment, universities have faced changing funding regimes with the introduction of national systems of funding conditional on evaluation of research output, or performance based research funding systems. 1 Universities in many countries now face periodic measurement and comparison of their research output. They participate in a single national system used to evaluate research across all types of universities and all fields. Such systems are designed to best suit the most expensive and most powerful universities and fields. Others will need to adapt to better fit the evaluation protocol. In OECD countries, the natural sciences and engineering account for 70-80% of government research spending on higher education. 2 1 Hicks, 2012 2 OECD statistics on HERD, 2009

One size doesn t fit all These are the most expensive and powerful fields, thus evaluation assumptions and protocols are designed for them. Social sciences must adapt. Since research evaluation rests largely on consideration of publication output both quantity and impact it is the form of social science scholarly publication that is evolving in response to the imposition of national research evaluation. At the same time, governments have accepted the argument that a one size fits all research evaluation system is unfair, and research evaluation protocols have been revised to better suit social science and humanities scholarship. Research evaluation and publishing in the social sciences and humanities are co-evolving. To understand how the imposition of evaluation models favoring the sciences changes social science scholarship, we must understand how social science publishing traditionally differed from the science publishing around which evaluation systems tend to be structured when first introduced. Scientists work with two genres English language journal articles and patents. Scientists work within disciplinary frameworks and can expect to reach consensus. Related to this, scientists recognize a set of core journals that are high quality and high impact, so a database can offer good coverage of a field by indexing those core journals. Scientists are oriented to the frontier and the latest results, so they reference mostly recent papers. Therefore, papers accumulate citations over a few years at most, so citation analyses can provide a fairly current measure of impact. Publishing English language journal articles in a set of core journals and building quickly on important discoveries - these habits of scientists lend themselves to effective indexing of research output and citations in databases and it is these databases that are used in evaluation. 68

Diana Hicks Traditionally, the form of social science scholarship has differed. Although there is some swift referencing, archival referencing of much older foundational papers is common. 3 The notion of clear disciplinary boundaries and a core set of journals can be problematic. And, as will be shown in this article, social scientists work with a repertoire of four genres: English language journal articles, books, national journal articles and enlightenment literature. For these reasons social science scholarship has not been well represented in databases. Yet visibility in databases such as Web of Science (WoS) and Scopus is central to being judged productive and worthy of government support in many national performance based research funding systems. Over time, social scientists became aware of being disadvantaged in their evaluation systems and have lobbied for changes. Therefore some degree of mutual adjustment has taken place. This paper explores the repertoire of four genres that comprise traditional social science as well as evidence that the structure of social science scholarship differs from that of science. This is done in order to better understand the emerging mutual adjustments being made by evaluation systems, databases, publishers and the forms of publication in social science. International Journals The first literature of social science that will be discussed is internationally oriented, largely English language 4 peer reviewed 3 Hargens, 2000 4 Of course, not all English language journals are international. Not even all journals indexed in the Web of Science are international as 69

One size doesn t fit all journal articles, similar to science. But in social science, these journals comprise a smaller fraction of the literature than in science. Because international journals are highly likely to be indexed in WoS, assessing the coverage of WoS indicates the share of international journals in a nation s output. Butler and Visser examined bibliographies from nine Australian universities in 1997 and 1999. 5 While 90% of chemistry output was covered in the Web of Science database, the database covered only 25% of the output of economics and 17% of the output of policy & politics. Data on this point are also available for Flanders and Norway because both have collected complete bibliographies for their Social Scientists and Humanists (SSH). Ossenbock and colleagues found that in both places, about onethird of SSH publishing is indexed in the Web of Science. 6 Fields that behave more like natural sciences have much higher rates of coverage. More than half of psychology and economics papers are indexed in Web of Science. 7 In contrast, less than 5% of law papers and 19% of theology/religion papers are indexed. Less than one quarter of Flemish history, media studies and sociology papers are indexed. Less than a quarter of Norwegian comparative literature, education, media studies and philosophy papers are indexed. The deficient coverage of SSH literature by the Web of Science makes it a poor basis for evaluation of SSH scholarship. Evaluation systems based on WoS indexed journals minor US journals are more likely to be indexed than are minor journals from other countries. 5 Butler and Visser, 2006 6 Ossenbock, Engels and Sivertsen, 2012 7 Eighty-three percent of Flemish and 66% of Norwegian psychology papers are indexed in Web of Science. Fifty-eight percent of Flemish and 72% of Norwegian economics papers are indexed in the Web of Science. 70

Diana Hicks will be based on a smaller fraction of research output in the social sciences than in the natural sciences. Books One of the reasons that that databases index a small share of SSH output is that they do not include books, and books are integral to SSH scholarship. Books have always been important in SSH and insignificant in the scientific literature. 8 So although books are ignored when evaluating science, a social science evaluation that ignored books would miss the large number of citations received by books. Studies have found that within the same area, books are more highly cited than journal articles by ratios ranging from 3:1 up to 6:1. 9 Perhaps the results of journal-only evaluation correlate with the results of a journal and book based evaluation. Then the lessthan-ideal journal based evaluation would be good enough. Unfortunately not, books are not just large, highly-cited journal articles. Four studies investigated the correlation between cites to books and journal articles and showed that such correlations traditionally have been low. Nederhof and colleagues listed the citations per book and journal article for 19 departments; the correlation between the two was 0.32. 10 Hicks and Potter collected a bibliography of 17 authors output in the field of sociology of scientific knowledge; the correlation between citations per book and journal article was 0.35. 11 Bourke and colleagues compared the rankings of departments using total 8 Small and Crane, 1979 9 Clemens, Powell, McIlwaine and Okamoto, 1995, Webster, 1998 10 Nederhof, Zwaan, DeBruin and Dekker, 1989 11 Hicks and Potter, 1991 71

One size doesn t fit all and journal only citation counts. 12 They concluded: In the social sciences and humanities, the use of journal citation rates as a surrogate for total publication citation rates is more likely to be misleading than in the sciences. 13 Finally, Cronin and colleagues constructed a database comprising 30,000 references from 90 books reviewed in top sociology journals and published between 1985 and 1993. 14 Cronin and colleagues compared lists of the 26 authors most cited in the monographs and in the top 24 sociology journals. They found that nine authors featured on both lists. The five authors ranked 22 to 26 on the book list did not even appear among the top 532 authors most cited in the journals. Low correlations in citation counts combined with differing highly cited author sets suggests that the journal and book literature have developed as different genres. That these genres may overlap but retain a distinct identity was supported by Line. 15 Line constructed a set of 59,000 references: 11,041 from monographs and 47,925 from journals. Line found that about half the time journal articles referenced journal articles and books referenced books. The rest of the references were spread across many different publication types. This suggests that the journal and book literatures have been somewhat selfcontained, although obviously interdependent and overlapping. Why did social science literature develop in two genres? Perhaps because they carry two types of scholarship; journal articles may reflect a more scientific, and books a more humanities 12 Bourke, Butler and Biglia, 1996 13 Bourke et al., 1996, pp. 54 14 Cronin, Snyder and Atkins, 1997 15 Line, 1979 72

Diana Hicks approach to scholarship. Clemens and colleagues study of US sociology helps us understand this. 16 Clemens and colleagues compared book and journal publishing within the context of a long standing debate in sociology. Is sociology professional, technical, cumulative, and convergent as one would gather from its journal literature or is it a diversified, intellectually open endeavor as found in the books? Examining the two types of publishing sheds light on the themes of scientific integrity versus intellectual vitality that underpin the debate. Clemens and colleagues evidence supported the notion that book and journal publishing form different genres. 17 They argued that entry into article publishing is competitive and so more egalitarian than entry into book publishing, which relies more heavily on patronage, recommendations and reputation. They found that book authors were more likely to be trained and located at elite private universities than were journal article authors. Article authors were more junior than book authors. Articles were more likely to be based upon quantitative evidence and books on qualitative evidence (although books based on quantitative evidence were the most cited of all). Clemens concluded: books and articles play different roles. Books are high-stakes endeavors that, when successful, are effective in enrolling allies from neighboring fields. In contrast, articles discipline the troops, generating a common currency of evaluation, be it in comprehensive exams or tenure decisions. To the extent that we care about scholarly reputation, 16 Clemens et al., 1995 17 Clemens et al., 1995 73

One size doesn t fit all both our discipline s and our own, neither genre should be ignored. 18 Clemens and colleagues analysis painted a picture of a heterogeneous field of scholarship with distinct journal and book traditions. 19 Journals represent a more scientific type of research and books a more humanities type of scholarship. Because books are more transdisciplinary, very highly cited and often produced by different people than journal articles, journal article evaluation will differ from studies that are more inclusive. Each genre contributes differently to the efforts of social science scholarship to develop a full understanding of society. There is no reason to discourage book publishing and the type of scholarship it represents. National Scholarly Journals The third genre of social science is national. Scientific research transcends national borders, but social sciences are more embedded in their social context because society is their concern. Social science research agendas are influenced by national trends and by policy concerns of national governments. Theoretical concepts are subtle and expressed in national languages. They can often be fully appreciated only in the original language. Some disagree; Moed and colleagues have argued that: genuine scholarly research in any area leads to results relevant outside the home country. [Though] this may be less true for more applied or practical research. Therefore [at least some] 18 Clemens et al., 1995, pp. 484 19 Clemens et al., 1995 74

Diana Hicks outcomes of genuine scholarly research, even those primarily related to national aspects, deserve to be communicated in an appropriate form to scholars in other countries as well. 20 Optimists studying social science literature in the late 1980 s found that in the international literature indexed in the SSCI: With the exception of a minority of topics related to political science, to social issues, and to a lesser extent physical health and geographical location, the large majority of the topics seem to reflect a transnational substantive interest. In addition, the [US and European countries] studied here share many social and political issues. Of course, this may not be true for other countries, and in particular non-western countries. The present data suggest that the research front on many topics in the social and behavioral sciences is international in the late 1980s... Of course, this does not preclude that publications on national issues or national aspects of issues appear in journals or books that address primarily a national audience. 21 It is the final point, publications addressing a national audience, to which I now turn. To examine the existence and nature of national scholarly literatures, I will compare national and international journals. By national journals I mean those that primarily publish articles in a language other than English, and whose authors and 20 Moed, Nederhof and Luwel, 2002, pp. 513 21 Nederhof and Van Wijk, 1997, pp. 271 75

One size doesn t fit all readers largely work in one country. International journals are largely English language journals either those that were originally American or British but are now targeted by authors from many countries or more recently founded English language European journals. Bibliometric evidence suggested that traditionally both producers and consumers of social science were nationally oriented. Gläser established the continuing existence of differentiated national communities in social sciences, even in an English speaking country, Australia. 22 Kyvik, studying the writing habits of Norwegian scientists and social scientists in the early 1980 s, found that compared to the scientists fewer social scientists published in a foreign language and more published in Norwegian. 23 Taking authors citation patterns as an indication of their reading habits, Yitzhaki found that authors over-cite material in their own language. 24 American and British authors cited English language material 99% of the time, although English language sociology probably accounted for 70% of the world literature. German and French authors cited material in their own language more than 60% of the time although such material accounted for less than 10% of literature in the field. In a sense then, each national literature is a genre. In addition, national literature overlaps to a limited extent with literature indexed in the databases. This was strikingly illustrated by an analysis comparing a unique resource, a Polish sociological citation index (PSCI) with the Social Science 22 Gläser, 2004 23 Kyvik, 1988, pp. 165 24 Yitzhaki, 1998 76

Diana Hicks Citation Index (SSCI) coverage of Polish sociology. Using a list of Polish sociologists and counting their citations in the Polish index and the SSCI, Webster found that of the top 10 most cited journals in the Polish index only the three foreign ones were indexed in the SSCI. 25 The top 20 most cited documents by Polish sociologists in each index contained none in common. All but one of the SSCI cited documents were in English; all the PSCI cited documents were in Polish. The most cited sociologist on the Polish list (with 253 citations) was ranked 41st in the SSCI (with 19 citations). The most cited sociologist on the SSCI list (with 254 citations) was ranked 20th on the PSCI list (with 41 citations). Two studies were done using the Polish sociology citation index. The first covered pre-transition Polish sociology, 1980 to 1988, the second covered pre and post transition sociology. Pre-transition, the SSCI missed 90% of Polish sociologists; post transition, it missed 30%. 26 Webster s analysis illustrated the bibliometric consequences of the limited overlap between national and Web of Science literatures. Bibliometric indicators based on foreign literature painted one picture of Polish sociology, and the Polish sociology index another. Webster summarizes this point well, concluding that the Social Science Citation Index (SSCI) indicates the presence and the impact of Polish sociology on the international arena, focusing on areas of research done in Poland which are of interest to the international community and the best Polish sociologists and Polish sociological works; but the Social 25 Webster, 1998 26 Winclawska, 1996 77

One size doesn t fit all Science Citation Index does not allow for an in-depth analysis of the local dimensions of the discipline. 27 The Polish work suggested that the ascendancy of an international social science placed small-country social scientists in the position of applying others frameworks to their societies. Polish sociologists were recognized internationally mostly when their society presented picturesque episodes that become fashionable topics in big countries. National communities could develop method and theory, but big-country social scientists remain impervious. Polish sociologists highly cited handbooks in general sociology by Polish authors, works on the social structure of Polish society, and works on interesting theoretical or methodological issues. Works highly cited in the SSCI included six that dealt with theoretical issues, each was at least 20 years old, others dealt with social unrest in Poland in the early 1980s and the fall of Communism in Eastern Europe. Webster concluded that: the international sociological community does not notice Polish attempts to tackle universal issues in sociology; it is primarily interested in fashionable topics and fads associated with the velvet revolution and systemic transformation. 28 My own work in progress pursues this line of investigation, taking advantage of a Spanish citation index (INRECS) to compare Spanish authored sociology papers highly cited in WoS with those highly cited in the Spanish index. As in the Polish study, we find that the most cited authors differ. We find that the top 25 most cited authors in WoS are not among the most cited in INRECS. Of the top 25 most cited authors in INRECS, 27 Webster, 1998, pp. 31 28 Webster, 1998, pp. 23-24 78

Diana Hicks three are among the most cited in WoS; each ranked lower than 155. The most cited topics also differ. Five of the top ten most cited Spanish sociology papers in WoS are about tourism because the journal Annals of Tourism Research is classified in sociology (as well as in hospitality and tourism). There are no tourism studies among the most cited 100 papers in any social science field in INRECS. Apart from tourism, the topics of the most cited papers in WoS are: social indicators, language and society, religion, health and community research. The topics most cited in INRECS are poverty, social welfare and social policy; family research, organization and political sociology. Again we see the pattern that foreigners are interested in Spanish research for particular, colorful reasons, especially the tourism industry, whereas the topics of interest to the domestic audience are closer to the core of sociology. Small country social scientists can be internationally recognized, but perhaps have fewer possible strategies for doing so than US or UK social scientists. Imposing an evaluation system that privileges international citations will force scholars to choose topics that interest foreign academics. Over time this poses the danger of forcing non-english language scholars out of the disciplinary core and into a fringe of colorful topics in the hopes of attracting the international attention so valued by their governments. Enlightenment Literature The fourth genre in the repertoire of social scientists is intellectual or enlightenment writing. This is found in periodicals whose goal is knowledge transfer or enlightenment of non-specialists. For example, the Nobel prize winning Princeton economist Paul Krugman exerts 79

One size doesn t fit all influence through his New York Times column. Burnhill and Tubby-Hille found that in the UK projects in education [were] reaching practitioners through the Times Education Supplement, with researchers in sociology, social administration, and socio-legal studies publishing in such periodicals as New Society and Nursing Times. 29 Kyvik found that in Norway one-half of social scientists published contributions to public debate. 30 In contrast, one-quarter to one-third of scientists contributed to public debate. Burnhill and Tubby-Hille investigated this issue in some depth. 31 They constructed a publications database from grant holders reports to a granting agency, supplemented by a survey. They classified journals as peer-reviewed using two directories that identify peer-reviewed serials, or the judgment of at least two authors. Assigning non peer reviewed journals to the enlightenment category suggests that psychologists, statisticians and geographers did not publish much in non-scholarly literature. Other fields did. Even economics, normally quite scientific in its publication patterns, exhibited a healthy percentage of articles in non-scholarly venues. Linguistics, education and sociology led in share of enlightenment publications. Nederhof and Zwaan have also looked quite closely at this issue. 32 They surveyed Dutch and foreign scholars asking them about the scholarliness of a number of journals in which Dutch social scientists published. They found that journals considered 29 Burnhill and Tubby-Hille, 1994, pp. 142 30 Kyvik, 2003 31 Burnhill and Tubby-Hille, 1994 32 Nederhof and Zwaan, 1991 80

Diana Hicks scholarly in university annual reports were not always considered so by experts. The share of non-scholarly journals ranged from 11% in experimental psychology to 25% in public administration. If departmental output were recounted, including only articles in journals judged scholarly, in the best case one experimental psychology department would have lost only 1% of its output, and in the worst case one public administration department would have lost 61% of its output. The Nederhof and Zwaan study opened up the issue of distinguishing enlightenment from scholarly literature. 33 That enlightenment and national scholarly literatures are not usually distinguished may contribute to the devaluation of the later. The classic problem with the national literature is the lower level of critique and peer review applied, leading to a reputation for lower quality. If enlightenment literature was acknowledged as such and reported in a separate category from national scholarly literature, we might find that the national scholarly literature is not as problematic as its reputation suggests. True scholarly journals need to be distinguished from enlightenment literature so that the quality of the former and the outreach function of the later can both be appreciated and valued. Studies have found that separation of enlightenment literature from scholarly literature is laborious because people disagree on where the boundary lies. However, when scholarly and enlightenment literature are carefully distinguished, database coverage rates for scholarly literature rise substantially. 34 Enlightenment literature moves knowledge into application, performing a function for social scientists analogous to 33 Nederhof annd Zwaan, 1991 34 Burnhill and Tubby-Hille, 1994, Schoepflin, 1990 81

One size doesn t fit all patenting for scientists. But patent systems are indexed, contain citation structures enabling evaluation, and have gained respect as a valued output worthy of evaluation. In contrast, enlightenment literature being also national literature, is less well indexed, tends not to be cited and is often viewed as low quality scholarship. The result is that enlightenment literature is not valued as an output of scholarly work interacting with application. Discussion In social science there are four distinct genres: international journal articles, books, national scholarly journal articles and enlightenment publications. International journal articles are indexed in databases and have been the currency of evaluation around the world. This is not wrong; using journal articles to communicate research results to an international audience is important. However, there is more to scholarly work in social science. Books can have a very high impact. National scholarly literature represents a body of knowledge specific to a society, developed in a local context and of particular relevance to people who share that context. Enlightenment literature represents knowledge reaching out to application. The authors and topics associated with the four genres overlap somewhat, but not completely. So the results of international journal bibliometrics will not be the same as the results of an evaluation which included all four genres. National research output evaluation systems privilege the international journal literature. An early system, the Australian Composite Index simply counted papers indexed in the Web of Science (WoS). The Flemish government introduced performance-based funding in 2003 based on counts of WoS 82

Diana Hicks indexed papers. Such international journal-based evaluation models will work for scientific fields but will be partial and misleading when applied to social sciences. Social scientists and humanists, well aware of the limitations, have objected to WoS only systems. As a result, we are now seeing adaptation by all parties: the databases, the evaluation models, publishers, and the social scientists themselves. Because Thomson Reuters and Elsevier are in competition for government contracts to supply data to national evaluation systems, they are sensitive to the SSH coverage problem. In 2009 Web of Science and Scopus added a large number of social science and humanities journals, increasing the size of the social science list in WoS by 22% and in Scopus by 39%. Evaluation systems have adapted as well. Today systems go beyond the Web of Science to count a wider range of journal literature. For example, in 2008 construction of a comprehensive database of Flemish university social science output began. Australia has expanded beyond WoS as well. However, expansion does not completely solve the problem because national journal articles are positioned within these expanded literature counts as an inferior version of international journal articles. This is done by assigning them less weight in the count. It is this methodological detail that threatens to create a strong incentive to move away from the core of a discipline into colorful topics favored by foreigners, a danger revealed above through the Polish and Spanish sociology studies. The four genres perspective would suggest national literature should be seen as a different dimension of publishing, and not necessarily as an inferior version of English language publishing. 83

One size doesn t fit all Social scientists have adjusted as well. 35 Between 2000 and 2009, publication by Flemish social scientists in journals indexed in WoS almost tripled. Growth came both from increased publication in journals indexed throughout the period i.e. social scientists changing their publication habits and from more journals being indexed in WoS i.e. journal publishers seeking to meet the criteria for inclusion in WoS and WoS expanding its coverage. 36 Trends in coverage and publishing in indexed journals suggest a mutual adjustment between social science scholarship and systems implicated in evaluation. Book publishing seems not to decline even after the introduction of a national research evaluation system. 37 Seemingly books will not be discounted, and evaluation systems adjust to their presence. Significant in this regard is that Thomson-Reuters, publisher of the Web of Science, has introduced a book index. This product promises to make citation counts of books available in evaluation processes. Although this seems to suggest that no adjustment on the part of scholars will be necessary, the construction of the index mirrors that of the journal index in that large, English language publishers will be better represented than small, non-english language publishers. This happened even in the Flemish system. The first version of the Flemish social science and humanities database included only 17% of the submitted records with an ISBN. The excluded 83% were books produced by unapproved publishing houses. 35 Kyvik, 2003 36 The number of indexed journals that included Flemish authored publications grew from 133 to 858 over the period; see Engels Ossenblok and Spruyt, 2012. 37 Engels et al., 2012 84

Diana Hicks The top 11 of these unapproved publishers by frequency of records were local publishers accounting for 45% of the excluded book material. In this case, publishers are adjusting. Faced with the prospect of their academic book lists not being considered scholarly enough to be included in the Flemish university evaluation system, Flemish publishers have launched the Guaranteed Peer Reviewed Content label. By making peer review explicit and traceable, the publishers aim to make their content eligible for the evaluation system under the governing regulation that defines scholarly outputs as having been subject to peer review. 38 Even if more book publishers are included in the Flemish system, the scholarly value of their books could still be devalued. This would happen if differential weights were applied to locally and internationally published books. The weights used in the Flemish system are being renegotiated, which leaves open the possibility of higher valuations for books written in English and published with an international publisher than for those written in Dutch and published locally. There is no evidence regarding the fate of enlightenment publishing in systems with national research evaluation. The enlightenment literature only enters into evaluation systems, or studies of evaluation systems, as contamination to be eliminated before a sound analysis can begin. Therefore, adaptations affecting enlightenment publishing can only be speculated upon. We do know that performance-based university research funding systems neglect application of research, although 38 There are three other criteria in the regulation: be publicly accessible, have an ISBN or ISSN, contribute to the development or application of new insights. Engels et al., 2012, Verleysen and Engels, 2012 85

One size doesn t fit all research application is a long-standing concern of governments. 39 For social scientists, application is associated with being involved in the public debate and publishing enlightenment literature. It seems safe to guess that like the national literature, enlightenment publishing is in decline in countries with performance-based funding systems. Over the long run, this may serve to reduce the impact of social science research on society and the dissemination of new knowledge to decision makers. Conclusion Law and Urry argue that the social sciences have always been embedded in, produced by, and productive of the social. 40 Reflecting on their insight in relation to the shifts in the four genres brought about by evaluation systems suggests some disturbing possibilities. First, enlightenment literature is central to social scientists engaging in relationships with their societies, but it looks set to wither, potentially cutting off scholars from application of their ideas. The same applies to discussion among scholars of issues particular to their societies whether in books or national language journal articles. Governments explicitly devalue such discussion in their evaluation systems, discouraging scholars from engaging local issues. Finally, the push into international, i.e. English language literature, risks forcing scholars to adopt the perspective of American academics, who dominate such literature. In this case, those thinking about the future of a society will be thinking in American terms. One partial escape from this fate is offered by the launch of English language journals produced by European 39 Hicks, 2012 40 Law and Urry, 2004, pp. 392 86

Diana Hicks scholars. This softens the impact of the international push, while perhaps leading to a desirable European scale convergence in thinking about the future of society. To avoid social scientists retreating to an internationally approved ivory tower of scholarship, performance based evaluation systems need to be designed to value each of the four literatures of social science. If this does not happen, unintended consequences seem likely to damage societies over the long term. References Bourke, Paul, Butler, Linda and Biglia, Beverley. Monitoring Research in the Periphery: Australia and the ISI Indices. Research Evaluation and Policy Project, Monograph Series No. 3, Canberra, Australian National University, 1996. Burnhill, Peter M. and Tubby-Hille Margarete E. On measuring the relation between social science research activity and research publication. Research Evaluation, Vol. 4, no. 3, 1994, pp. 130-152. Butler, Linda and Visser, Martijn S. Extending citation analysis to non-source items. Scientometrics, Vol. 66, no. 2, 2006, pp. 327 343. Clemens, Elisabeth S., Powell, Walter W., McIlwaine, Kris and Okamoto, Diana. Careers in print: Books, journals, and scholarly reputations. The American Journal of Sociology, Vol. 101, no. 2, 1995, pp. 433-494. Cronin, Blaise, Snyder, Herbert and Atkins, Helen. Comparative citation rankings of authors in monographic and journal literature: A study of sociology. Journal of Documentation, Vol. 53, no. 3, 1997, pp. 263-273. Engels, Tim C.E., Ossenblok, Truyken L.B., and Spruyt, Eric H.J. Changing publication patterns in the social sciences 87

One size doesn t fit all and humanities, 2000 2009. Scientometrics, Vol. 93, no. 2, 2012, pp. 373-390. Gläser, Jochen. Why are the most influential books in Australian sociology not necessarily the most highly cited ones? Journal of Sociology, Vol. 40, no. 3, 2004, pp. 261-282. Hargens, Lowell. Using the literature: Reference networks, reference contexts, and the social structure of scholarship. American Sociological Review, Vol. 65, no. 6, 2000, pp. 846-865. Hicks, Diana. Performance-based university research funding systems. Research Policy, Vol. 41, no. 2, 2012, pp. 251-261. Hicks, Diana and Potter, Jonathan. Sociology of scientific knowledge: A reflexive citation analysis or science disciplines and disciplining science. Social Studies of Science, Vol. 21, no 3, 1991, pp. 459-501. Kyvik, Svein. Internationality of the social sciences: The Norwegian case. International Social Science Journal, Vol. 115, 1988, pp. 163-172. Kyvik, Svein. Changing trends in publishing behaviour among university faculty, 1980-2000. Scientometrics, Vol. 58, no. 1, 2003, pp. 35-48. Law, John and John Urry. Enacting the Social. Economy and Society, Vol. 33, no. 3, 2004, pp. 390-410. Line, Maurice B. The Influence of the Type of Sources Used on the Results of Citation Analyses. The Journal of Documentation, Vol. 35, no. 4, 1979, pp. 265-284. Moed, Henk, Nederhof, Anton and Marc Luwel. Towards performance in the humanities. Library Trends (Special 88

Diana Hicks Issue on Current Theory in Library and Information Science), Vol. 50, no. 3, 2002, pp. 498-520. Nederhof, A.J., Zwaan, R.A. DeBruin, R.E. and Dekker, P.J. Assessing the Usefulness of Bibliometric Indicators for the Humanities and the Social and Behavioural Sciences: A Comparative Study. Scientometrics, Vol. 15, no. 5-6, 1989, pp. 423-435. Nederhof, A.J. and Zwaan, R.A. Quality Judgments of Journals as Indicators of Research Performance in the Humanities and the Social and Behavioral Sciences. Journal of the American Society for Information Science, Vol. 42, no. 5, 1991, pp. 332-340. Nederhof, A. J. and Van Wijk, E. Mapping the Social and Behavioral Sciences World-Wide: Use of Maps in Portfolio Analysis of National Research Efforts. Scientometrics, Vol. 40, no. 2, 1997, pp. 237-276. OECD Main science and technology indicators. OECD Science, technology and R&D statistics (database). 2012. DOI: 10.1787/data-00182-en. Ossenbock, Truyken, Engels, Tim C.E. and Sivertsen, Gunnar. The representation of the social sciences and humanities in the Web of Science: A comparison of publication patterns and incentive structures in Flanders and Norway (2005 9). Research Evaluation, 2012, Vol. 21, no. 4. pp. 280-290. DOI:10.1093/reseval/rvs019. Schoepflin, Urs Problems of representativity in the Social Sciences Citation Index. In Representations of Science and Technology, Peter Weingart, Roswitha Sehringer and Matthias Winterhager (Eds.), Proceedings of the International Conference on Science and Technology Indicators, Bielefeld, Germany, 1990, pp. 177-188. 89

One size doesn t fit all Small, Henry G., and Crane, Diana. "Specialties and disciplines in science and social science: An examination of their structure using citation indexes." Scientometrics Vol. 1. No. 5-6, 1979, pp. 445-461. Verleysen, Frederick T. and Engels Tim C.E. A label for peer reviewed books. Forthcoming JASIST. 2012. Webster, Berenika M. Polish Sociology Citation Index as an Example of Usage of National Citation Indexes in Scientometric Analysis of Social Science. Journal of Information Science, Vol. 24, no. 1, 1998, pp. 19-32. Winclawska, Berenika M. Polish Sociology Citation Index Scientometrics, Vol. 35, no. 3, 1996, pp. 387-391. Yitzhaki, M. The Language Preference in Sociology. Scientometrics, Vol. 41, no. 1-2, 1998, pp. 243-254. Diana Hicks is Professor and Chair of the School of Public Policy, Georgia Institute of Technology, Atlanta, USA. 90