Absolute Relevance? Ranking in the Scholarly Domain. Tamar Sadeh, PhD CNI, Baltimore, MD April 2012

Transcription

1 Absolute Relevance? Ranking in the Scholarly Domain Tamar Sadeh, PhD CNI, Baltimore, MD April 2012

2 Copyright Statement All of the information and material inclusive of text, images, logos, product names is either the property of, or used with permission by Ex Libris Ltd. The information may not be distributed, modified, displayed, reproduced in whole or in part without the prior written permission of Ex Libris Ltd. TRADEMARKS Ex Libris, the Ex Libris logo, Aleph, SFX, SFXIT, MetaLib, DigiTool, Verde, Primo, Voyager, MetaSearch, MetaIndex and other Ex Libris products and services referenced herein are trademarks of Ex Libris, and may be registered in certain jurisdictions. All other product names, company names, marks and logos referenced may be trademarks of their respective owners. DISCLAIMER The information contained in this document is compiled from various sources and provided on an "AS IS" basis for general information purposes only without any representations, conditions or warranties whether express or implied, including any implied warranties of satisfactory quality, completeness, accuracy or fitness for a particular purpose. Ex Libris, its subsidiaries and related corporations ("Ex Libris Group") disclaim any and all liability for all use of this information, including losses, damages, claims or expenses any person may incur as a result of the use of this information, even if advised of the possibility of such loss or damage. Ex Libris Ltd., 2012

3

4 The top three keys to success

5 Content

6 Speed

7 Relevance Ranking

8 This talk is about relevance ranking

9 What is relevance? What is relevance in the scholarly domain? How do we measure relevance? What can be done and how?

10 relevance is. Information technology is tangible;; relevance is intangible. Information technology is relatively well understood formally;; relevance is understood intuitively. Information technology has to be learned;; relevance is tacit. Information technology has to be explained to Saracevic, 2007

11 Relevance is the measure of correspondence between a document and a query as determined by a user Based on Saracevic, 1975

12 ?

13 System or algorithmic relevance System or algorithmic relevance Topical or subject relevance Topical or subject relevance Cognitive relevance or pertinence Cognitive relevance or pertinence Situational relevance or utility Situational relevance or utility Affective relevance Affective relevance

14 There is no absolute relevance

15

16

17

18

19

20

21

22

23

24

25 Aboutness Bibliographic descriptions Classifications Ontologies Relevance Librarianship Information Retrieval (IR)

26 Aboutness and relevance Discovery Systems

27 Effectiveness of IR is measured by the probability of agreement between what the system retrieved or constructed as relevant and what a user assessed or derived as relevant

28 The ScholarRank Project

29 The Goal Enhance the Primo relevance ranking algorithm

30 Relevance ranking was not new to us.

31

32

33

34

35

36

37 Methodology

38 Setting up a team Building test environment, tools, and procedures Defining metrics to evaluate our current success and the improvements we make Defining measurements to assess the success of the changes, once implemented

39 Lab Mode

40 Precision and Recall Precision= Number of relevant documents retrieved Total number of documents retrieved Recall= Number of relevant documents retrieved Total number of existing relevant documents

41 Is this item relevant enough to be on the first result page? That is the question.

42 Evaluation Metrics Mean Average Precision (MAP) Mean Reciprocal Rank (MRR)

43 Infrastructure Data Tools Automated process to run queries Calculation of MAP and MRR metrics Ad hoc changes of parameters in test environment

44 Evaluators Academic researchers in various disciplines (physics, medicine, philosophy, anthropology, agriculture, biology)

45 Evaluation Evaluators sent their queries of four types broad-topic, narrow-topic, known-item, other Two sets of the first 20 results were sent back and evaluated One set returned by Primo One set returned by Google Scholar Evaluators were not aware of the origin of the sets Evaluators marked Yes/No If No, why not

46

47

48 * * Initial results (March 2011);; only 15% of the Primo Central content * MAP maximal value: 1.00

49

50 More Queries Many more evaluators Full scope of search UI changed to support evaluation

51

52

53 45% 40% 35% 30% 25% 20% 15% 10% 5% 0% MAP % 70% 60% 50% 40% 30% 20% 10% 0% MRR

54 Real-Life Mode

55 Monitoring after implementation Number of times that users moved to the next page of results Number of sessions that culminated in the selection of an item Average number of items that were selected per session Location of the selected items on the result list

56 So?

57 Phase 1 Maximized existing algorithm

58 Material type = journal article MAP MRR Boost value

59 Phase 2 Added factors

60

61

62 ?

63 abstract author date full text journal language type publisher subject title citations downloads journal impact factor eigenfactor pagerank

64 academic degree discipline(s) language location previous selections search history

65 ? broad-topic search currency exact-item search material type narrow-topic search

66 Broad-topic query

67 Narrow-topic query

68 Author-related query

69 Known-item query

70 ?

71 The match: traditional IR methods, adapted for the scholarly environment

72 1 4 39

73 ? no. of citations;; no. of selections;; recency;; type;; peer review

74

75 Academic degree, discipline?

76

77

78

79

80 Author-related query, known-item query, broad-?

81 Before

82 Tachycardia Wikipedia the free encyclopedia Tachycardia comes from the Greek words tachys (rapid or accelerated) and kardia (of the heart). Tachycardia typically refers to a heart rate that exceeds the normal range for a resting... After

83 ?

84

85 Bibliography Saracevic, T. (1975). Relevance: A review of and a framework for the thinking on the notion of information science. Journal of American Society for Information Science, 26(6), Saracevic, T. (2007) Relevance: A Review of the Literature and a Framework for Thinking on the Notion in Information Science. Part II: Nature and Manifestations of Relevance. Journal of the American Society for Information Science and Technology, 58(13): hm/all/1

86 Thank You! Tamar Sadeh, PhD