Natural Language Processing

Transcription

1 atural Language Processg Info 159/259 Lecture 19: Semantic parsg (Oct. 31, 2017) David Bamman, UC Berkeley

2 Announcements 259 fal project presentations: 3:30-5pm Tuesday, Dec. 5 (RRR week), 202 South Hall

3 Semantics The meang of a sentence is the set of possible worlds it picks out Collections of possible worlds can be compactly represented with logical forms Pieces of logical forms correspond to pieces of language Types logic correspond to grammatical categories language From last time (Andreas)

4 Why is syntax important? Foundation for semantic analysis (on many levels of representation: semantic roles, compositional semantics, frame semantics) From 10/5

5 Why is syntax sufficient? Syntax encodes the structure of language but doesn t directly address meang. Even if we have a reference model for each word a sentence, syntax doesn t tell us how those referents changes as a function of their compositionally.

6 Representation of meang Constants name dividual entities the world Relations are sets of entities Variables refer to entities that have not yet been specified Quantifiers bd variables. (existential quantifier) (universal quantifier)

7 Representation of meang Constants: Pat, Sal Relations: likes(x,y) Pat likes Sal The denotation likes = the ordered set of entities for whom the relation is true likes(pat, Sal) = true likes = {(Pat, Sal), (, )}

8 Representation of meang Quantifiers bd variables. (existential quantifier) (universal quantifier) Order matters! x y speaks(x,y) y x speaks(x,y)

9 Representation of meang Relations: likes(x,y) is scoped over two variables We can represent the partial representation of meang with lambda expressions: λx.likes(x,sal) Expect one other argument to complete the meang of this relation

10 Representation of meang λx.likes(x,sal) Lambda expressions let us tie semantics explicitly to phrases (subtrees syntax) x S V VP likes Sal

11 Representation of meang λy.λx.likes(x,y) Lambda expressions let us tie semantics explicitly to phrases (subtrees syntax) x S V VP likes y

12 Compositional semantics is driven by syntax Prciple of compositionality

13 Syntax We could represent the relationship between syntax and semantics a CFG. But what we want is fe-graed control over the mappg between words and semantic primitives.

14 CCG Infitely large set of structured categories (types). Primitives: S,, PP, Complex types: S/ (S, except missg to right) S\ (S, except missg to left

15 CCG CFG has a large set of productions (e.g., S VP) CCG has a very small set of combators that us how to put the types together. Smith 2017

16 CCG Combators Forward application combator (X/Y X) / / yellow dog Smith 2017

17 CCG Combators Forward application combator (X/Y X) / / / the / yellow dog Smith 2017

18 CCG Combators Backward application combator (X\Y X) S\ S S S\ I (S\)/ saw / the / yellow dog Smith 2017

19 CCG Combators Conjunction combator (X and X X) and dogs and cats Smith 2017

20 CCG Combators Forward composition (X/Y Y/Z X/Z) and backward composition (Y\Z X\Y X\Z) I S (S\)/ S\ (S\)/(S\) would (S\)/ prefer olives Smith 2017

21 24085 / adjective noun 2583 (S[dcl]\)/ transitive verb (declarative) 2107 S[adj]\ predicative adjectives (man is old) 1679 (S[b]\)/ transitive verb (bare fitive) 1628 (/)/(/) adjective adjective pairs 1431 S[pss]\ transitive verb (past participles) 1385 (S[ng]\)/ transitive verb (present participle) 1308 [num] numerals 1227 S[dcl]\ transitive verb (present participle) 1112 (S\)\(S\) adverbs Most frequent types CCGBank lexicon

22 S S\ I S\ (S\)\(S\) (S\)/ (S\)\(S\)/ shot / / an elephant my pajamas S S\ S\ (S\)\(S\) (S\)/ / (S\)\(S\)/ / I shot an elephant my pajamas

23 S S\ \ (S\)/ / (\)/ / I shot an elephant my pajamas

24 S S\ I (S\)/ shot \ S / an elephant (\)/ / S\ my pajamas I S\ (S\)\(S\) (S\)/ (S\)\(S\)/ shot / / an elephant my pajamas

25 I shot an elephant my pajamas [0,1] (S\)/ [1,2] / [2,3] [3,4] (\)/ (S\)\(S\)/ / [5,6] Hockenmaier, 2003 [6,7]

26 I shot an elephant my pajamas [0,1] S/ (S\)/ [1,2] / [2,3] [3,4] (\)/ (S\)\(S\)/ / [5,6] Hockenmaier, 2003 [6,7]

27 I shot an elephant my pajamas [0,1] S/ (S\)/ [1,2] / [2,3] [3,4] (\)/ (S\)\(S\)/ (Leavg out forward composition for clarity) / [5,6] Hockenmaier, 2003 [6,7]

28 I shot an elephant my pajamas [0,1] S/ (S\)/ [1,2] / [2,3] [3,4] (\)/ (S\)\(S\)/ / [5,6] Hockenmaier, 2003 [6,7]

29 I shot an elephant my pajamas [0,1] S/ (S\)/ [1,2] / [2,3] S\ [3,4] (\)/ (S\)\(S\)/ / [5,6] Hockenmaier, 2003 [6,7]

30 I shot an elephant my pajamas [0,1] S/ S (S\)/ [1,2] / [2,3] S\ [3,4] (\)/ (S\)\(S\)/ / [5,6] Hockenmaier, 2003 [6,7]

31 I shot an elephant my pajamas [0,1] S/ S (S\)/ [1,2] S\ / [2,3] [3,4] (\)/ (S\)\(S\)/ / [5,6] Hockenmaier, 2003 [6,7]

32 I shot an elephant my pajamas [0,1] S/ S (S\)/ [1,2] S\ / [2,3] [3,4] (\)/ (S\)\(S\)/ / [5,6] Hockenmaier, 2003 [6,7]

33 I shot an elephant my pajamas [0,1] S/ S (S\)/ [1,2] S\ / [2,3] [3,4] (\)/ (S\)\(S\)/ / [5,6] \ (S\)\(S\) Hockenmaier, 2003 [6,7]

34 I shot an elephant my pajamas [0,1] S/ S (S\)/ [1,2] S\ / [2,3] [3,4] (\)/ (S\)\(S\)/ / [5,6] \ (S\)\(S\) Hockenmaier, 2003 [6,7]

35 I shot an elephant my pajamas [0,1] S/ S (S\)/ [1,2] S\ / [2,3] [3,4] (\)/ (S\)\(S\)/ / [5,6] \ (S\)\(S\) Hockenmaier, 2003 [6,7]

36 I shot an elephant my pajamas [0,1] S/ S (S\)/ [1,2] S\ S\ S\ / [2,3] [3,4] (\)/ (S\)\(S\)/ / [5,6] \ (S\)\(S\) Hockenmaier, 2003 [6,7]

37 I shot an elephant my pajamas [0,1] S/ S/ S S/ (S\)/ [1,2] (S\)/ S\ (S\)/ S\ S\ / [2,3] / [3,4] (\)/ (S\)\(S\)/ (\)/ (S\)\ (S\) / [5,6] \ (S\)\(S\) Hockenmaier, 2003 [6,7]

38 I shot an elephant my pajamas [0,1] S/ S/ S S/ S S (S\)/ [1,2] (S\)/ S\ (S\)/ S\ S\ / [2,3] / [3,4] (\)/ (S\)\(S\)/ (\)/ (S\)\ (S\) / [5,6] \ (S\)\(S\) Hockenmaier, 2003 [6,7]

39 S S\ I (S\)/ shot \ S / an elephant (\)/ / S\ my pajamas I S\ (S\)\(S\) (S\)/ (S\)\(S\)/ shot / / an elephant my pajamas

40 CCG Lexicon Ambiguity CCG comes from the lexicon, not grammar

41 CCG Lexicon I 1438 I 22 I / 4 I \ 3 shot 25 shot (S[dcl]\)/ 8 shot S[dcl]\ 7 shot S[pss]\ 5 shot (S[pss]\)/ 1 elephant 5 elephant / 1 an [nb]/ 3685 an (\)/ 76 an ((S\)\(S\))/ 16 an (((S\)\(S\))\((S\)\(S\)))/ 8 an / 3 an / 3 an ((S\)\(S\))/((S\)\(S\)) 2 an (/)/(/) 2 an (S[qem]/(S[dcl]/))/ 2 an ((S/S)\(S/S))/ 1 an (S\S)/ 1 an, 1 an 1

42 CCG Lexicon (\)/ 8013 ((S\)\(S\))/ 7035 PP/ 1644 (S/S)/ 374 (S\)\(S\) 279 ((S\)/(S\))/ 241 ((S\)\(S\))/(S[ng]\) 155 (((S\)\(S\))\((S\)\(S\)))/ 125 (\)/(S[ng]\) 121 ((S[adj]\)\(S[adj]\))/ 110 PP/(S[ng]\) 106

43 (\)/ ((S\)\(S\))/ PP/ (S/S)/ (S\)\(S\) ((S\)/(S\))/ ((S\)\(S\))/(S[ng]\) (((S\)\(S\))\((S\)\ (S\)))/ (\)/(S[ng]\) ((S[adj]\)\(S[adj]\))/ PP/(S[ng]\) (S\S)/ (PP\PP)/ (((S\)\(S\))/(S[to]\))/ S[adj]\ ((\)\(\))/ (\)/(S[adj]\) (\)/ (((S\)\(S\))/((S\)\ (S\)))/ (/)/ ((S\)\(S\))/(S[adj]\) (PP/(S[ng]\))/ ((S\)/(S\))/(S[ng]\) ((S\)/(S\))/(S[adj]\) (S/S)/(S[ng]\) (S\)/ (((S\)/(S\))\((S\)/(S\)))/ (((S\)\(S\))/(S[to]\))/ (((S\)\(S\))/S[dcl])/ ((S[adj]\)\(S[adj]\))/(S[ng]\) ((S\)\(S\))/PP (\)/PP ((/)\(/))/(/) ((\)/(\))/ ((S\)\(S\))/S[em] (\)/S[qem] \ PP/(S[adj]\) PP/PP (((S\)\(S\))\)/ ((S\)\(S\))/((S\)\(S\)) (\)/ (S/(S[to]\))/ (S\S)/(S[ng]\) \ ((((S\)\(S\))/((S\)\(S\)))\ (((S\)\(S\))/((S\)\(S\))))/ ((/)\(/))/ (PP/PP)/ (S[adj]\)\(S[adj]\) (((S\)\(S\))/)/ ((PP/PP)\(PP/PP))/ ((S[adj]\)/(S[adj]\))/ ((S\)\(S\))/ ((S\)\(S\))/S[qem] ((S\S)\(S\S))/ (PP\)/ / PP PP/S[qem] ((((S\)\(S\))\((S\)\(S\)))\ (((S\)\(S\))\((S\)\(S\))))/ ((((S\)\(S\))\((S\)\(S\)))\ (((S\)\(S\))\((S\)\(S\))))/ (((S/S)\(S/S))\((S/S)\(S/S)))/ (((S\)/(S\))/(S[to]\))/ (((S\)\(S\))/(S[ng]\))/ ((/)/(/))/ ((\)/(S[ng]\))/ ((\)/)/ ((S/S)/(S/S))/ ((S[dcl]\)/PP)/ ((S\)\(S\))\((S\)\(S\)) ((S\S)/(S[to]\))/ ((S\S)/)/ (/)/(/) (/)/ (\)/(\)

44 Supertaggg The CCG lexicon + very small set of combators dictates the overall parse for a sentence. The base categories CKY for syntactic parsg are POS tags (~45 PTB) The base categories CKY for CCG parsg are the lexical rules (~1,363 CCGBank) This blows up the complexity of parsg; supertaggg reduces the lexical categories to a much small reset by predictg the likeliest tags context.

45 I shot an elephant my pajamas / \ (S[dcl]\)/ S[dcl]\ S[pss]\ (S[pss]\)/ / [nb]/ (\)/ ((S\)\(S\))/ (((S\)\(S\))\((S\)\(S\)))/ / / ((S\)\(S\))/((S\)\(S\)) (/)/(/) (S[qem]/(S[dcl]/))/ ((S/S)\(S/S))/ (S\S)/, (\)/ ((S\)\(S\))/ PP/ (S/S)/ (S\)\(S\) ((S\)/(S\))/ ((S\)\(S\))/ (S[ng]\) (((S\)\(S\))\ ((S\)\(S\)))/ (\)/(S[ng] \) ((S[adj]\)\ (S[adj]\))/ PP/(S[ng]\) (S\S)/ (PP\PP)/ (((S\)\(S\))/ (S[to]\))/ S[adj]\ ((\)\ (\))/ (\)/(S[adj] \) (\)/ (S/S)\(S/S) / [nb]/ \

46 I shot an elephant my pajamas (S[dcl]\)/ (\)/ ((S\)\(S\))/ PP/ (\)/ [nb]/ (\)/

47 MEMM General maxent form arg max y P (y x, ) n Maxent with first-order Markov assumption: Maximum Entropy Markov Model arg max y i=1 P (y i y i 1,x)

48 MEMM y1 y2 y3 y4 y5 y6 y7 x1 x2 x3 x4 x5 x6 x7

49 Features : (\)/ ((S\)\(S\))/ feature example xi = man 1 f(t i,t i 1 ; x 1,...,x n ) ti-1 = JJ 1 Features are scoped over the previous predicted tag and the entire observed put i=n (last word of sentence) xi ends -ly 0 1

50 Viterbi decodg Viterbi for MEMM: max conditional probability P (y x) v t (y) = max u Y [v t 1(u) P (y t = y y t 1 = u, x, )]

51 Supertaggg The sgle best sequence is often still too errorful to be the put for CCG parsg. Rather than predictg the sgle best sequence, we can identify the top k tags for each word that have the highest probability. ote this is not P(yi yi-1, x, β) but rather P(yi x, β) we can calculate usg the forward-backward algorithm.

52 Semantics Semantic parsg with CCG is simply syntactic parsg, assumg mappg from syntactic primitives to logical forms.

53 CCG Lexicon Utah utah Idaho idaho borders (S\)/ λx.λy(borders(y,x) adjos (S\)/ λx.λy(adjos(y,x) abuts (S\)/ λx.λy(abuts(y,x)

54 CCG Combators Each combator tells us what to do with the correspondg semantics Forward application: X/Y : f Y : g X f(g) (S\)/ : λx.λy(borders(y,x) : idaho S\ : λx.λy(borders(y,x)(idaho) S\ : λy(borders(y,idaho) Smith 2017

55 Semantics S borders(utah,idaho) S\ λy(borders(y,idaho) utah (S\)/ λx.λy(borders(y,x) idaho Utah borders Idaho

56 Semantics Utah utah Idaho idaho borders (S\)/ λx.λy(borders(y,x) adjos (S\)/ λx.λy(adjos(y,x) abuts (S\)/ λx.λy(abuts(y,x)

57 Semantics Utah utah Idaho idaho borders (S\)/ λx.λy(borders(y,x) adjos (S\)/ λx.λy(borders(y,x) abuts (S\)/ λx.λy(borders(y,x)

58 Semantics Utah utah Idaho idaho borders (S\)/ λx.λy(borders(y,x) adjos (S\)/ λx.λy(borders(y,x) abuts (S\)/ λx.λy(borders(y,x) Does Utah border California?

59 Semantics Semantic parsg with CCG is simply syntactic parsg, assumg mappg from syntactic primitives to logical forms. But this encounters two problems: We don t have those manual mappgs (taskspecific). We can t parse anythg not our lexicon.

60 Learng from logical forms We can tra a semantic parser a number of ways: Full derivational trees (CCGBank) Logical forms (Zettlemoyer and Colls 2005) Denotations (Berant et al. 2013)

61 Learng from trees S λx.state(x) ^ (borders(x,texas) (S/S\) λg.λx.state(x) ^ g(x) (S\) λy(borders(y,texas) (S/S\)/ λf.λg.λx.f(x) ^ g(x) λx.state(x) (S\)/ λx.λy(borders(y,x) texas what states border texas

62 Learng from trees S (S/S\) (S\) (S/S\)/ (S\)/ what states border texas

63 Learng from logical forms sentence what states border texas logical form λx.state(x) ^ borders(x, texas) Two core ideas: We ll learn the lexicon (cludg the lambda expressions) We ll learn CCG parser from that lexicon, and treat the true tree as a latent variable

64 Learng from trees S (S/S\) (S\) (S/S\)/ (S\)/ what states border texas We ll treat the tree (derivation) as a latent variable

65 Learng the lexicon sentence logical form Utah borders Idaho borders(utah,idaho) For a given sentence and logical form, return the set of lexicon entries that could have generated the logical form. Utah utah Idaho idaho borders (S\)/ λx.λy(borders(y,x)

66 Learng the lexicon sentence logical form Utah borders Idaho borders(utah,idaho) For a given sentence and logical form, return the set of lexicon entries that could have generated the logical form. All subsequences of x GELEX(S, L) ={x := y x W (S),y C(L)} All categories found logical form

67 C(L)

68 logical form utah idaho borders borders borders(utah,idaho) : utah : idaho (S\)/ : λx.λy.borders(y,x) (S\)/ : λx.λy.borders(x,y)

69 Learng the lexicon All subsequences of x All categories found logical form GELEX(S, L) ={x := y x W (S),y C(L)} Utah utah Idaho idaho borders idaho borders utah (S\)/ λx.λy(borders(y,x) borders (S\)/ λx.λy(borders(y,x)

70 Learng from logical forms If we create a lexicon λi = itial lexicon Λ0 + lexicon entries identified by GELEX, we can fd many parses for the sentence. sentence Utah borders Idaho logical form utah idaho idaho utah borders borders borders(utah,idaho) : utah : idaho : utah : idaho (S\)/ : λx.λy.borders(y,x) (S\)/ : λx.λy.borders(x,y)

71 Learng from logical forms Calculate the jot probability of a logical form L and derivation T for sentence S as: feature P (L, T S; )= exp(f(l, T, S) ) exp(f(l, T, S) ) L,T Utah := : utah Utah := : idaho sums over all valid trees/logical forms for the sentence borders := (S\)/ : λx.λy.borders(y,x) borders := (S\)/ : λx.λy.borders(x,y) f(l,t,s)

72 Learng from logical forms For all <sentence, logical form> pairs trag data, maximize the probability of the logical form by margalizg over the jot probability: Where P (L S; )= T P (L, T S; ) Start with random values for θ; update with SGD P (L, T S; )= exp(f(l, T, S) ) L,T exp(f(l, T, S) )

73 Learng from logical forms Learng from logical forms is means we don t need trag data the form of full CCG derivations + semantically enriched lexicon. But we do still need trag data the form of logical forms. Utah borders Idaho borders(utah,idaho) number of dramas starrg tom cruise???

74 Learng from denotations sentence logical form denotation what states border texas λx.state(x) ^ borders(x, texas) new_mexico, oklahoma, arkansas, louisiana sentence logical form number of dramas starrg tom cruise count(λx.genre(x,drama) ^ y.performance(x,y) ^ actor(y,tom_cruise)) denotation 28

75 Learng from denotations sentence logical form denotation what states border texas λx.state(x) ^ borders(x, texas) new_mexico, oklahoma, arkansas, louisiana sentence logical form number of dramas starrg tom cruise count(λx.genre(x,drama) ^ y.performance(x,y) ^ actor(y,tom_cruise)) denotation 28

76 Learng from denotations How could we use the prciples of learng from logical forms to learn from denotations? The meang of a sentence is the set of possible worlds consistent with that statement. Utah borders Idaho TRUE number of dramas starrg tom cruise 28

77 Learng from denotations Basic idea: maximize the probability of the tree T/ logical form z that, when executed agast a knowledge base K, yield the correct denotation y i=1 log T : T.z K =y i P (T S i, ) objective function

78 Why do we need CCG (or a syntactic representation) at all?