COSC282 BIG DATA ANALYTICS FALL 2015 LECTURE 11 - OCT 21

Size: px

Start display at page:

Download "COSC282 BIG DATA ANALYTICS FALL 2015 LECTURE 11 - OCT 21"

Claude Goodwin
6 years ago
Views:

1 COSC282 BIG DATA ANALYTICS FALL 2015 LECTURE 11 - OCT 21 1

2 Topics for Today Assignment 6 Vector Space Model Term Weighting Term Frequency Inverse Document Frequency

3 Something about Assignment 6

6 Search Engine

7 W H AT I S O U R M I S S I O N H E R E? 7

8 Query 8 Search Results: a ranked list of documents

9 Document Retrieval It is the most important information retrieval task: document retrieval". it is to find relevant documents for a query not to find answers - that is question answering Sometimes also known as ad-hoc retrieval

10 Document Retrieval Process Information Need Document Representation Corpus Query Representation Indexing Retrieval Algorithms Index Retrieval Results Evaluation/ Feedback

11 How do Retrieval Algorithms Work?

12 How to find the relevant documents for a query? By keyword matching boolean models By similarity vector space model By imaging how to write out a query how likely a query is written with this document in mind generate with some randomness query generation language model By trusting how other people think about the documents /web pages link-based methods, pagerank, hits

13 Vector Space Model

14 Ch. 6 Formal Definition of Document Retrieval Task: rank-order the documents in the collection with respect to a query Input: a query and all documents in your collection Output: a ranked list of documents (of all documents in the collection, but you can stop early) 14

15 Basic Procedure Assign a score (say in [0, 1]) to each document in the collection This score measures how well the document and the given query match. Then, sort the scores usually in descending order of the score from the most relevant to the least relevant

16 Sec. 6.3 Vector Space Model Treat query as a tiny document Represent both query and documents as word vectors in a word space Rank documents according to their proximity to the query in the space of words

17 Represent Documents in a Space of Word Vectors Sec. 6.3 Suppose the corpus only has two words: Jealous and Gossip They form a space of Jealous and Gossip d1: gossip gossip jealous gossip gossip gossip gossip gossip gossip gossip gossip d2: gossip gossip jealous gossip gossip gossip gossip gossip gossip gossip jealous jealous jealous jealous jealous jealous jealous gossip jealous d3: jealous gossip jealous jealous jealous jealous jealous jealous jealous jealous jealous q: gossip gossip jealous gossip gossip gossip gossip gossip jealous jealous jealous jealous 17

18 Calculate the Query- Document Similarity

19 Sec. 6.3 Formalizing vector space proximity First cut: distance between the end points of the two vectors? How to do it?

20 Euclidean Distance In mathematics, the Euclidean distance or Euclidean metric is the "ordinary" (i.e. straight-line) distance between two points in Euclidean space. If if p = (p1, p2,..., pn) and q = (q1, q2,..., qn) are two points in the Euclidean space, their Euclidean distance is

21 Sec. 6.3 In a space of Jealous and Gossip Here, if you look at the content (or we say the word distributions) of each document, d2 is actually the most similar document to q However, d2 produces a bigger distance score to q d1: gossip gossip jealous gossip gossip gossip gossip gossip gossip gossip gossip d2: gossip gossip jealous gossip gossip gossip gossip gossip gossip gossip jealous jealous jealous jealous jealous jealous jealous gossip jealous d3: jealous gossip jealous jealous jealous jealous jealous jealous jealous jealous jealous q: gossip gossip jealous gossip gossip gossip gossip gossip jealous jealous jealous jealous 21

22 Sec. 6.3 In a space of Jealous and Gossip The Euclidean distance between q and d 2 is large even though the distribution of terms in the query q and the distribution of terms in the document d 2 are very similar.

23 Why Euclidean Distance is A Bad Idea for Query-Document Similarity Because Euclidean distance is large for vectors with different lengths. short query and long documents will be always have big Euclidean distance we cannot rank them fairly, as compared with others not possible to get a universal ranking

24 How can we do better?

26 What matters is the content similarity Here the angles between the vectors captures this similarity better, not the distance metrics.

27 Sec. 6.3 Use angle instead of distance Key idea: Rank documents according to angle with query The angle between similar vectors is small, between dissimilar vectors is large. This is exactly what we need to score a querydocument pair. This is equivalent to perform a document length normalization

28 Sec. 6.3 Document Length Normalization A vector can be (length-) normalized by dividing each of its components by its length: Dividing a vector this way makes it a unit (length) vector (on surface of unit hypersphere) Long and short documents now have comparable weights

29 Cosine similarity illustrated 12

30 Sec. 6.3 Cosine Similarity q i is the tf-idf weight of term i in the query d i is the tf-idf weight of term i in the document cos(q,d) is the cosine similarity of q and d or, equivalently, the cosine of the angle between q and d.

31 Exercise - Please go to Piazza Consider two documents D 1, D 2 and a query Q D 1 = (0.5, 0.8, 0.3), D 2 = (0.9, 0.4, 0.2), Q = (1.5, 1.0, 0)

32 Results Consider two documents D 1, D 2 and a query Q D 1 = (0.5, 0.8, 0.3), D 2 = (0.9, 0.4, 0.2), Q = (1.5, 1.0, 0)

33 What are the numbers in a vector? They are term weights to indicate the importance of a term in a document

34 Term Weighting

35 Sec. 1.1 Recall: Term-Document Matrix Antony and Cleopatra Julius Caesar The Tempest Hamlet Othello Macbeth Antony Brutus Caesar Calpurnia Cleopatra mercy worser if document contains term, 0 otherwise

36 the numbers in the matrix could be more meaningful to indicate the importance of each term in a document

37 Term Frequency How many times a term appears in a document

38 Sec. 6.2 Term-Document Matrix with Term Frequency Consider the number of occurrences of a term in a document: Each document is a word count vector in N v : a column below Antony and Cleopatra Julius Caesar The Tempest Hamlet Othello Macbeth Antony Brutus Caesar Calpurnia Cleopatra mercy worser

39 but,

41 Some terms are common, well less common than the stop words but still common e.g. Georgetown is uniquely important in NBA.com e.g. Georgetown appears at too many pages in our university web site, so it is not a very important term in those pages. How to discount them?

42 Inverse document frequency Document Frequency: How many documents in which a term appears Inverse Document Frequency: is the inverse of the above our way of discount the common terms

43 Sec Inverse Document Frequency (idf) df t is the document frequency of t = the number of documents that contain t df t is an inverse measure of the informativeness of t df t N N is the total number of documents We define the idf (inverse document frequency) of t by idf = log ( N/df t 10 t ) We use log (N/df t ) instead of N/df t to dampen the effect of idf. 43

44 Sec Exercise: Calculate IDF (Suppose N = 1 million) term df t idf t calpurnia 1 animal 100 sunday 1,000 fly 10,000 under 100,000 the 1,000,000 idf = log ( N/df t 10 t ) There is one idf value for each term t in a collection. 44

45 TF-IDF Term Weighting

46 Sec tf-idf weighting The tf-idf weight of a term is the product of its tf weight and its idf weight. w = log(1 + tf ) log t d t, d 10 ( N / df, t ) Best known weighting scheme in information retrieval Increases with the number of occurrences within a document Increases with the rarity of the term in the collection

47 Binary count weight matrix Antony and Cleopatra Julius Caesar The Tempest Hamlet Othello Macbeth Antony Brutus Caesar Calpurnia Cleopatra mercy worser Each document is now represented by a real-valued vector of tf-idf weights R V

48 Sec. 6.4 tf-idf weighting has many variants

49 Sec. 6.4 Weighting may differ in queries vs documents Many search engines allow for different weightings for queries vs. documents A very standard weighting scheme is: lnc.ltc Document: logarithmic tf (l as first character), no idf and cosine normalization Query: logarithmic tf (l in leftmost column), idf (t in second column), no normalization

50 Vector Space Model Advantages Simple computational framework for ranking documents given a query Any similarity measure or term weighting scheme could be used Disadvantages Assumption of term independence

51 Summary of Today Vector Space Weighting TF-IDF Term Weighting Assignment 6 due today Midterm next Wed

Inverted Index Construction

Inverted Index Construction Adapted from Lectures by Prabhakar Raghavan (Yahoo and Stanford) and Christopher Manning (Stanford) Prasad L3InvertedIndex 1 Unstructured data in 1650 Which plays of Shakespeare