Absolute Relevance? Ranking in the Scholarly Domain Tamar Sadeh, PhD CNI, Baltimore, MD April 2012
Copyright Statement All of the information and material inclusive of text, images, logos, product names is either the property of, or used with permission by Ex Libris Ltd. The information may not be distributed, modified, displayed, reproduced in whole or in part without the prior written permission of Ex Libris Ltd. TRADEMARKS Ex Libris, the Ex Libris logo, Aleph, SFX, SFXIT, MetaLib, DigiTool, Verde, Primo, Voyager, MetaSearch, MetaIndex and other Ex Libris products and services referenced herein are trademarks of Ex Libris, and may be registered in certain jurisdictions. All other product names, company names, marks and logos referenced may be trademarks of their respective owners. DISCLAIMER The information contained in this document is compiled from various sources and provided on an "AS IS" basis for general information purposes only without any representations, conditions or warranties whether express or implied, including any implied warranties of satisfactory quality, completeness, accuracy or fitness for a particular purpose. Ex Libris, its subsidiaries and related corporations ("Ex Libris Group") disclaim any and all liability for all use of this information, including losses, damages, claims or expenses any person may incur as a result of the use of this information, even if advised of the possibility of such loss or damage. Ex Libris Ltd., 2012
The top three keys to success
Content
Speed
Relevance Ranking
This talk is about relevance ranking
What is relevance? What is relevance in the scholarly domain? How do we measure relevance? What can be done and how?
relevance is. Information technology is tangible;; relevance is intangible. Information technology is relatively well understood formally;; relevance is understood intuitively. Information technology has to be learned;; relevance is tacit. Information technology has to be explained to Saracevic, 2007
Relevance is the measure of correspondence between a document and a query as determined by a user Based on Saracevic, 1975
?
System or algorithmic relevance System or algorithmic relevance Topical or subject relevance Topical or subject relevance Cognitive relevance or pertinence Cognitive relevance or pertinence Situational relevance or utility Situational relevance or utility Affective relevance Affective relevance
There is no absolute relevance
Aboutness Bibliographic descriptions Classifications Ontologies Relevance Librarianship Information Retrieval (IR)
Aboutness and relevance Discovery Systems
Effectiveness of IR is measured by the probability of agreement between what the system retrieved or constructed as relevant and what a user assessed or derived as relevant
The ScholarRank Project
The Goal Enhance the Primo relevance ranking algorithm
Relevance ranking was not new to us.
Methodology
Setting up a team Building test environment, tools, and procedures Defining metrics to evaluate our current success and the improvements we make Defining measurements to assess the success of the changes, once implemented
Lab Mode
Precision and Recall Precision= Number of relevant documents retrieved Total number of documents retrieved Recall= Number of relevant documents retrieved Total number of existing relevant documents
Is this item relevant enough to be on the first result page? That is the question.
Evaluation Metrics Mean Average Precision (MAP) http://en.wikipedia.org/wiki/mean_average_precision Mean Reciprocal Rank (MRR) http://en.wikipedia.org/wiki/mean_reciprocal_rank
Infrastructure Data Tools Automated process to run queries Calculation of MAP and MRR metrics Ad hoc changes of parameters in test environment
Evaluators Academic researchers in various disciplines (physics, medicine, philosophy, anthropology, agriculture, biology)
Evaluation Evaluators sent their queries of four types broad-topic, narrow-topic, known-item, other Two sets of the first 20 results were sent back and evaluated One set returned by Primo One set returned by Google Scholar Evaluators were not aware of the origin of the sets Evaluators marked Yes/No If No, why not
* * Initial results (March 2011);; only 15% of the Primo Central content * MAP maximal value: 1.00
More Queries Many more evaluators Full scope of search UI changed to support evaluation
45% 40% 35% 30% 25% 20% 15% 10% 5% 0% MAP 0-0.19 0.2-0.39 0.4-0.59 0.6-0.79 0.8-1 80% 70% 60% 50% 40% 30% 20% 10% 0% 0-0.09 0.1-0.19 0.2-0.29 0.3-0.39 0.5 1 MRR
Real-Life Mode
Monitoring after implementation Number of times that users moved to the next page of results Number of sessions that culminated in the selection of an item Average number of items that were selected per session Location of the selected items on the result list
So?
Phase 1 Maximized existing algorithm
0.355 0.35 0.345 0.34 0.335 0.33 0.325 0.32 0.315 0.31 0.305 Material type = journal article 0 5 10 15 20 25 30 35 MAP MRR Boost value
Phase 2 Added factors
?
abstract author date full text journal language type publisher subject title citations downloads journal impact factor eigenfactor pagerank
academic degree discipline(s) language location previous selections search history
? broad-topic search currency exact-item search material type narrow-topic search
Broad-topic query
Narrow-topic query
Author-related query
Known-item query
?
The match: traditional IR methods, adapted for the scholarly environment
1 4 39
? no. of citations;; no. of selections;; recency;; type;; peer review
1 27 231
Academic degree, discipline?
Author-related query, known-item query, broad-?
Before
Tachycardia Wikipedia the free encyclopedia Tachycardia comes from the Greek words tachys (rapid or accelerated) and kardia (of the heart). Tachycardia typically refers to a heart rate that exceeds the normal range for a resting... After
?
Bibliography Saracevic, T. (1975). Relevance: A review of and a framework for the thinking on the notion of information science. Journal of American Society for Information Science, 26(6), 321 343. Saracevic, T. (2007) Relevance: A Review of the Literature and a Framework for Thinking on the Notion in Information Science. Part II: Nature and Manifestations of Relevance. Journal of the American Society for Information Science and Technology, 58(13):1915 1933 http://www.wired.com/magazine/2010/02/ff_google_algorit hm/all/1
Thank You! Tamar Sadeh, PhD tamar.sadeh@exlibrisgroup.com