Using Bibliometric Analyses for Evaluating Leading Journals and Top Researchers in SoTL

Size: px

Start display at page:

Download "Using Bibliometric Analyses for Evaluating Leading Journals and Top Researchers in SoTL"

Howard Gilbert
5 years ago
Views:

Georgia Southern University Digital Commons@Georgia Southern SoTL Commons Conference SoTL Commons Conference Mar 26th, 2:00 PM - 2:45 PM Using Bibliometric Analyses for Evaluating Leading Journals

1 Georgia Southern University Digital Southern SoTL Commons Conference SoTL Commons Conference Mar 26th, 2:00 PM - 2:45 PM Using Bibliometric Analyses for Evaluating Leading Journals and Top Researchers in SoTL Josephine M. Csete The Hong Kong Polytechnic University, etjcsete@polyu.edu.hk Mei Li The Hong Kong Polytechnic University, lbmeili@polyu.edu.hk Follow this and additional works at: Part of the Curriculum and Instruction Commons, Educational Assessment, Evaluation, and Research Commons, Educational Methods Commons, Higher Education Commons, and the Social and Philosophical Foundations of Education Commons Recommended Citation Csete, Josephine M. and Li, Mei, "Using Bibliometric Analyses for Evaluating Leading Journals and Top Researchers in SoTL" (2015). SoTL Commons Conference This presentation (open access) is brought to you for free and open access by the Conferences & Events at Digital Commons@Georgia Southern. It has been accepted for inclusion in SoTL Commons Conference by an authorized administrator of Digital Commons@Georgia Southern. For more information, please contact digitalcommons@georgiasouthern.edu.

2 Using bibliometric analyses for evaluating leading journals and top researchers in SoTL SoTL: Commons Conference Savannah, Georgia March 2015 Josephine M. Csete & Mei Li The Hong Kong Polytechnic University

3 First lets take a little quiz. (see accompanying file)

4 Your Questions & Comments What are your goals for coming to this session? Which of the three questions most interest you?

5 Go to No Longer Operational Type your name and organization in blue box and click Join Then in Message box write any question/comment you have & click Say 4

6 Agenda Your goals How is scholarship currently measured? How does SoTL measure up as a discipline? How might I proceed as a scholar?

7 How is scholarship currently measured?

8 Quiz Part I What was your score?

9 How is scholarship currently measured? Three resources Web of Science Scopus Google Scholar Two measures Impact factor H-index There are other resources and measures, but these are the most common

Bibliometrics There are many definitions of the term bibliometrics, for example: Bibliometrics is referred to the quantitative measures used to assess research output, such

10 Bibliometrics There are many definitions of the term bibliometrics, for example: Bibliometrics is referred to the quantitative measures used to assess research output, such as publication and citation data analysis. Citations are generally considered to be a useful indication of a paper s impact and value to the scientific research community. 9

11 Why Measure Research Impact? Accountability Funding allocation University/Subject ranking QS World University Ranking: 20-30% for paper citations Times Higher Education (THE): 37% for citation impact Academic Ranking of World Universities (ARWU) by Shanghai JT Univ: 60% for publications Evaluate staff: appointment/appraisal/promotion Research trends 10

12 Comparing Citation Analysis Sources from University of Michigan Library Components Web of Science Scopus Google Scholar 3 citation indexes: Science Citation Index Expanded back to 1900 Social Sciences Citation Index back to 1956 Arts & Humanities Citation Index back to 1975 Conference Proceedings - - back to 1990 Life Sciences, 4,873 titles Health Sciences, 10,196 titles (including 100% coverage of Medline titles) Physical Sciences > 9,800 titles Social Sciences > 5,300 titles Arts & Humanities, 1,439 titles; the majority of these titles go back to 2002 Selections from PubMed, IEEE, American Institute of Physics, proceedings of the National Academy of Sciences, Nature.com, American Medical Association and other medicine journals, Ingenta, SpringerLink,Wiley Interscience, Cambridge journals, Taylor and Francis, Sage Publications, Blackwell-Synergy, OCLC First Search and others Open access journals and preprints Online dissertations and theses Coverage Over 12,000 journals 19,500 active journals Unknown Time Span Some journal files going back to 1900; 44.4 million records, of which: 23 million records include references going back to 1996 (78% include references) 21 million pre-1996 records go back as far as 1823 Theoretically, whatever is available on the Web 11

13 Web of Science Scopus Google Scholar Update Weekly 1-2 Times a week Monthly on average Strengths Weaknesses Deeper back-files especially for Science Journals While controversial, its journal citation reports, impact factors, and h-index are most widely used. More focused on U.S. research Offers citation mapping for visual presentation Limited no. of journals indexed 3-5 years for new journals to be indexed Back-files are expensive User friendly search interface Broader coverage of journals (16, 500 versus 10,000 in WOS) Downloadable reference list More internationally focused than WoS Includes more than 1,200 Open Access titles Citation tracking is limited to the relatively narrow time span of Citation errors due to name variations Not very strong in Social Science and Arts & Humanities coverage Provides a more comprehensive picture of scholarly impact as it indexes nontraditional sources not covered by WOS and Scopus. Includes peer-reviewed papers, theses, books, abstracts, and articles from academic publishers, professional societies, preprint repositories, universities, and other scholarly organizations Better coverage of newer materials than both WOS and Scopus International and multi-lingual coverage Limited search features Not providing non self-citations Inflated citation counts due to inclusion of non-scholarly sources such as promotional pages, table of contents pages, course readings lists and newspaper etc. Weeding irrelevant hits is time consuming Difficult to export citations No way to determine what sources, 12 and time spans are covered.

14 Scopus Coverage Image source:

15 14

16 What are the implications for SoTL?

17 Two Measures: Impact factor What is it? What questions does it attempt to answer? What is a high impact factor? What is a low impact factor?

18 Journal Impact Factors (JIF) The JIF is the average number of times articles from the journal published in the past two years have been cited in the JCR year. The Impact Factor is calculated by dividing the number of citations in the JCR year by the total number of articles published in the two previous years. An Impact Factor of 1.0 means that, on average, the articles published one or two years ago have been cited one time. An Impact Factor of 2.5 means that, on average, the articles published one or two years ago have been cited two and a half times. JIF is produced using data from the Web of Science. The JIF is updated every year and published in the Journal Citation Reports. Source: Journal Citation Reports (

19 How is a Journal Impact Factor calculated? For example, to find the JIF for Journal X in the year 2013 All citations received by Journal X in 2013 to any content published in Impact Factor 2013 = Articles & Reviews published in Journal X between This method of calculation will therefore result in a different JIF each year Note: Only references in articles within the 10,000+ journals indexed in Web of Science / JCR are counted in the JIF calculation.

20 Try ranking these journals in order of impact factor: American Historical Review Geology Journal of Organizational Behavior Medical Education Nature

21 Try ranking these journals in order of impact factor: American Historical Review Geology Journal of Organizational Behavior Medical Education Nature Lesson: Journals cannot be compared across disciplines

22 Two Measures: H-index What is it? (definition) What question does it answer? What is a high H-index? What is a low H-index?

23 H-index A person s H-index is the number of papers (N) that have N or more citations. The H-index gives an estimate of the importance, significance, and broad impact of a scientist's cumulative research contributions (Hirsch, 2005). H-index found to correlate well with peer judgment (Van Raan, 2006). Limitations of H-index Varies with subjects Increase with researcher s age Capture past accomplishments, not reflect to current productivity, or future achievements. Hirsch, J. E. (2005). An index to quantify an individual's scientific research output. Proceedings of the National academy of Sciences of the United States of America, 102(46), Van Raan, A. F. (2006). Comparison of the Hirsch-index with standard bibliometric indicators and with peer judgment for 147 chemistry research groups. scientometrics, 67(3),

H-index Related Parameters H-index Average Citation per Item A = Percentage of Non Self-citations over Total citations B = Percentage of No. of Cited Articles over Total no.

24 H-index Related Parameters H-index Average Citation per Item A = Percentage of Non Self-citations over Total citations B = Percentage of No. of Cited Articles over Total no. of Articles C = Percentage of Citing Articles with Non Self-citations over Total citations H-index : 36 Avg. citation per paper: B = 114/124 = 92% A = 4308/4461 = 97% C = 3518/4461 = 79% 23

25 Rank by H-index After 20 years (physics): > 20 successful scientist > 40 outstanding scientist > 60 truly unique individual Hirsch, J. E. (2005). An index to quantify an individual's scientific research output. Proceedings of the National academy of Sciences of the United States of America, 102(46),

An index to quantify an individual's scientific research output.

26 Rank by H-index After 20 years (physics): > 20 successful scientist 96 > 40 outstanding scientist > 60 truly unique individual 63 Hirsch, J. E. (2005). An index to quantify an individual's scientific research output. Proceedings of the National academy of Sciences of the United States of America, 102(46),

27 What are the implications for SoTL? How can we improve the likelihood of increased citations?

28 New Alternative Metrics: Altmetrics Altmetrics is a new and modern way to assess research impact that takes into account ways apart from citation counts. It is based on the Social Web for analyzing, and informing scholarship. 27

29 Altmetrics 28

30 Go to No Longer Operational Type your name and organization in blue box and click Join Then in Message box write any question you have & click Say 29

31 How does SoTL measure up as a discipline?

32 Quiz Part II What s your score? What does completing these questions suggest to you? Who are your SoTL scholars?

33 Which journal is the best? Where does Journal Title Impact Factor Subject NEW ENGLAND JOURNAL OF MEDICINE MEDICINE NATURE REVIEWS MICROBIOLOGY MICROBIOLOGY ACADEMY OF MANAGEMENT REVIEW Education go? BUSINESS GEOLOGY GEOLOGY BIRTH-ISSUES IN PERINATAL CARE NURSING Cliometrica HISTORY Citation practices can vary significantly from one field to another. For example, articles in some subject area (e.g. biochemistry) often contain over 50 cited references, while a typical mathematical paper has perhaps only 10. This difference explains why biochemical papers are cited so much more often than mathematical ones. The traditional impact factor cannot therefore be used to compare journals across different subject area. SNIP is an advanced citation indicator that corrects for such differences.

34 Disseminating SoTL What kind of journals? SoTL? Discipline based? Education? Conferences? Other sources?

35 Journals that publish SoTL ranked by JCR 2013 B: SoTL Journals Review of Educational Research 5 B: SoTL Journals Learning and Instruction B: SoTL Journals Educational Researcher B: SoTL Journals American Educational Research Journal D: Journals about higher education in general Computers in Human Behavior B: SoTL Journals American journal of education 1.59 B: SoTL Journals Studies in higher education B: SoTL Journals Research in higher education D: Journals about higher education in general D: Journals about higher education in general Journal of Creative Behavior Higher Education

36 Journals that publish SoTL ranked by SCImago 2013 B: SoTL Journals Review of Educational Research 4.36 B: SoTL Journals American Educational Research Journal 3.88 B: SoTL Journals Educational Researcher 3.58 B: SoTL Journals Learning and Instruction 3.36 B: SoTL Journals Review of Higher Education 2.34 B: SoTL Journals American Journal of Education 2.24 B: SoTL Journals Studies in Higher Education 2.01 B: SoTL Journals Journal of Higher Education 1.8 D: Journals about higher education in general D: Journals about higher education in general Computers in Human Behavior Educational Technology Research and Development

37 SoTL Journals International Journal for the Scholarship of Teaching and Learning Journal of the Scholarship of Teaching and Learning Teaching and Learning Inquiry (ISSOTL started 2013) New Directions in Teaching and Learning The Teaching Professor To Improve the Academy Mountain Rise (not active) Here is a sample list And another

38 What are the implications for SoTL?

39 The main points to remember when conducting bibliometric analyses are: 1. Always compare like with like for example: groups and individuals in the same or similar disciplines groups and individuals at the same stage in their academic careers journals in the same discipline or category similar size institutions 2. Don t rely on one single tool remember results can vary depending on the tool used. 3. Be aware that some disciplines rely less on publishing in journals than others 4. Use a combination of metrics and other qualitative information where appropriate. 5. Journal Quality Article Quality 6. Citation a work agreement with findings 7. Self citations

40 How might I proceed as a scholar?

41 Quiz Part III What s your score? What does completing these questions suggest to you?

42 What about looking up H-index? Where is it? How can you grow it? What are the caveats?

43 Eric Mazur

44 Eric Mazur But When refined to Educational Research results drop to 13 items with an H-index of 6

45 In a pilot study in our University For 2011 any entry in Scopus or Web of Science (both Science and Social Sciences) indicating an author from our University was compiled Three individuals read every title, and if it looked like it could be educational research (ER) the abstracts were then consulted List of all possibilities (indicated by any one reviewer) were then discussed by all three to arrive at consensus Is this ER? Is this also SoTL?

46 In a pilot study in our University ~1100 faculty members at our university generated 3350 items in 2011 How many qualified as Educational Research? How many qualified as SoTL?

47 In a pilot study in our University ~1100 faculty members at our university generated 3350 items in 2011 How many qualified as Educational Research? 67 2% How many qualified as SoTL? %

48 What can YOU do to increase your SoTL impact?

49 Advice from thoughtful librarians Get an ORCID number Pick your title wisely (keywords too) Spend time on the abstract Reference papers with high citations Co-author with high h-index authors Check whether journal is in Scopus/Web of Science before you submit Try to publish in a less specific journal (not niche)

What other metrics are available for my publications?

blogs, media - Articles mentions on Facebook, Twitter -

known as Alternative Metrics/Alt- Metrics/Article-Level Metrics.

50 What other metrics are available for my publications? - Article downloads (from publisher sites) - Article mentions in blogs, media - Articles mentions on Facebook, Twitter - Bookmarks/recommendations made to an article (using social booking marking tools, reference management tools e.g. CiteULike, Connotea, and Mendeley ) These new metrics are commonly known as Alternative Metrics/Alt- Metrics/Article-Level Metrics. Learn more about Alt-Metrics at Institutional Repositories (IRs) can also provide really useful data about your papers. Papers housed in IRs are freely available via the web and easily retrieved via a simple google search.

51 Of this session...what is your keeper?

52 My Keeper...do we play the game or create our own?

53 My Keeper (2).

54 Go to No Longer Operational Type your name and organization in blue box and click Join Then in Message box write any question you have & click Say

55 All slides with this symbol: Have been contributed by the helpful librarians at The Hong Kong Polytechnic University. These slides were part of their presentations in a workshop series on Research Skills held in March Thank you!

56 Thank You Very Much!

INTRODUCTION TO SCIENTOMETRICS. Farzaneh Aminpour, PhD. Ministry of Health and Medical Education

INTRODUCTION TO SCIENTOMETRICS Farzaneh Aminpour, PhD. aminpour@behdasht.gov.ir Ministry of Health and Medical Education Workshop Objectives Definitions & Concepts Importance & Applications Citation Databases