Scalable Semantic Parsing with Partial Ontologies ACL 2015

Transcription

1 Scalable Semantic Parsing with Partial Ontologies Eunsol Choi Tom Kwiatkowski Luke Zettlemoyer ACL

2 Semantic Parsing: Long-term Goal Build meaning representations for open-domain texts How many people live in Seattle? Semantic Parser SELECT Population FROM CityData where City=="Seattle ; Executor 620,778 (Kwiatkowski et.al, 13, Liang et.al. 11, Cai & Yates 2013, Berant et.al. 13,14, Reddy et. a. 14) 2

3 Semantic Parsing: Large Domain is a large, community authored knowledge base with: 40 Million Entities 2 Billion Facts 20,000 Relations 10,000 Types 100 Domains 3

4 Current semantic parsers can parse How many people live in Seattle? Which college did Obama go to? What party did Clay establish? 4

5 Current semantic parsers can parse cannot parse How many people live in Seattle? Which college did Obama go to? How many people live in Anyang? Which college did Eunsol go to? What party did Clay establish? Who are Russian short story writers in 19th century? What is a popular seaside resort city in Italy? 5

6 Remaining Challenges Fact Incompleteness How many people live in Anyang? Which college did Eunsol go to? Schema Incompleteness Who are Russian short story writers in 19th century? What is a popular seaside resort city in Italy? 6

7 Remaining Challenges Fact Incompleteness How many people live in Anyang? Which college did Eunsol go to? Schema Incompleteness Who are Russian short story writers in 19th century? What is a popular seaside resort city in Italy? 7

8 Remaining Challenges: Fact Incompleteness 620,778 Seattle Population State How many people live in Anyang? Which college did Eunsol go to? Washington Anyang Unable to handle sentences Population with facts not in Freebase? 8

9 Remaining Challenges: Fact Incompleteness 620,778 Seattle Population State How many people live in Anyang? Which college did Eunsol go to? Washington Anyang 70% of people in FB have no birth place (West 14) Population 66% of facts missing in pilot study for our dataset? 9

10 Remaining Challenges Fact Incompleteness How many people live in Anyang Which college did Eunsol Schema Incompleteness Who are Russian short story writers in 19th century? What is a popular seaside resort city in Italy? 10

11 Remaining Challenges: Schema Incompleteness Who are Russian short story writers in 19th century? What is popular seaside resort city in Italy? Unable to handle concepts outside existing schema 11

12 Remaining Challenges: Schema Incompleteness Who are Russian short story writers in 19th century? What is popular seaside resort city in Italy? In a pilot study on our dataset: Unable to handle concepts 27.2% of sentences describe outside existing schema concepts not in Freebase 12

13 Previous Approach? Existing data is filtered to ensure completeness: FB917 dataset is created from Freebase (Cai and Yates 13) 93% of originally gathered questions cannot be answered with FB (WebQuestions, Berant 13) 13

14 Remaining Challenges Fact Incompleteness How New many learning people approach live in Anyang? with broad coverage lexical statistics Which college did Eunsol go to? Schema Incompleteness Who are Russian short story writers in 19th century? Semantic parser with partial groundings What is a popular seaside resort city in Italy? 14

15 This Work Build meaning representations with both Freebase concepts and open concepts British playwright, novelist and short story writer Semantic Parser 15

16 Parsing with Incompleteness 1. Open Information Extraction (Banko et al., 07; Fader et al., 11) 2. Matrix Factorization (Riedel, 13;Krishnamurthy, 15) 3. Web Search Queries (Joshi, 14)

17 Outline 1. Task and Applications 2. Data 3. Semantic Parser with Partial Ontology 4. Learning 5. Evaluation

18 Task Build a meaning representation with Freebase concepts and concepts outside Freebase British playwright, novelist and short story writer Semantic Parser

19 Focus: Noun Phrases Interesting noun-noun modifier, implicit relations. Itself is a referring expression, resembling queries. 19

20 Focus: Noun Phrases Interesting noun-noun modifier, implicit relations. Useful for information extraction, when paired with an entity. 20

21 Applications Input: Referring Expression Resolution (QA) Noun Phrase Entity Attribute Extraction (IE) (Entity, Noun Phrase) British playwright, novelist and short story writer Somerset Maugham, British playwright, novelist and short story writer Output: Sommerset Maugham (S. Maugham, Nationality, U.K) (S. Maugham, Profession, Novelist) (S. Maugham, Profession, Playwright)

22 Overview: Approach British playwright, novelist and short story writer Semantic Parser 22

23 Referring Expression Resolution (QA) British playwright, novelist and short story writer Semantic Parser Sommerset Maugham 23

24 Referring Expression Resolution (QA) Entity Attribute Extraction (IE) British playwright, novelist and short story writer Semantic Parser Sommerset Maugham (x, Nationality, U.K) (x, Profession, Novelist) (x, Profession, Playwright) 24

25 Referring Expression Resolution (QA) Entity Attribute Extraction (IE) Somerset Maugham, British playwright, novelist and short story writer Semantic Parser Sommerset Maugham (S.Maugham, Nationality, U.K) (S.Maugham, Profession, Novelist) (S.Maugham, Profession, Playwright) 25

26 Outline 1. Task and Applications 2. Data 3. Semantic Parser with Partial Ontology 4. Learning 5. Evaluation

27 Wikipedia Category 27

28 Wikipedia Category 28

29 Wikipedia Category Film directors from New York 29

30 Wikipedia Category Film directors from New York 30

31 Wikipedia Category On average, 15% entity overlap with Freebase Exciting opportunity for information extraction Challenge for existing learning techniques Film directors from New York 31

32 Wikipedia Category: Data statistics Entire Set Number of Category 365 K Number of words per category 4.1 Number of entity-category pair 7 million 32

33 Appositives Relation between a named entity and a nominal. Laurie Hays, a executive editor at Bloomberg News, is leaving the company. Laurie Hays, a executive editor at Bloomberg News 33

34 Appositives Extracted from open texts such as news articles Malta, an EU outpost in the Mediterranean, decided today Richard Nixon, a former president of the United States Maputo, the relaxed seaside capital of Mozambique, 34

35 Appositives Extracted from open texts such as news articles Malta, an EU outpost in the Mediterranean, decided today Richard Nixon, a former president of the United States Maputo, the relaxed seaside capital of Mozambique, 35

36 Appositives: Data statistics Entire Set Number of apposition 67 K vocab 25 K Number of words per apposition

37 Outline 1. Task and Applications 2. Data 3. Semantic Parser with Partial Ontology 4. Learning 5. Evaluation

38 Two Stage Semantic Parsing (EMNLP 13) British playwright, novelist and short story writer Domain Independent Parse Ontology Match 38

39 Two Stage Semantic Parsing (EMNLP 13) British playwright, novelist and short story writer Domain Independent Parse Ontology Match 39

40 Two Stage Semantic Parsing (EMNLP 13) British playwright, novelist and short story writer Domain Independent Parse Ontology Match 40

41 Two Stage Semantic Parsing with Partial Grounding British playwright, novelist and short story writer Domain Independent Parse Ontology Match 41

42 Partial Grounding: Open Schema Explicitly model concepts not in Knowledge base as OpenRel and OpenType Plants described in 1891 Lower_classification(plant, x) OpenRel_described_in(x, 1891) Former municipalities in Brandenburg OpenType_Former(x) OpenRel(x, Municipality) Located_In(x, Brandenburg) 42

43 Partial Grounding: Open Schema Explicitly model concepts not in Knowledge base Benefits of as open OpenRel schema: and OpenType Plants described in 1891 Help learn String-Freebase concepts (plant, lower_classification, x) Allow partial execution (x, OpenRel(described_in), 1891) Capture useful information, although not Former municipalities in Brandenburg grounded (x, Type, OpenType_Former) (x, OpenRel, Municipality) (x, Located_In, Brandenburg) 43

44 Two Stage Semantic Parsing Domain Independent Parse with Partial Grounding Ontology Match Structure Match Constant Matches for. OPEN Constant Matches OpenType OpenRel Municipality Location.ContainedBy Brandenburg 44

45 Outline 1. Task and Applications 2. Data 3. Semantic Parser with Partial Ontology 4. Learning 5. Evaluation

46 Previous Work: Direct Supervision How many people live in Seattle? How many people live in Seattle? Semantic Parser SELECT Population FROM CityData where City=="Seattle ; Latent Executor 620, ,778 46

47 Supervision from Unfiltered Data 47

48 Supervision from Unfiltered Data Social democratic parties in Greece Semantic Parser Executor {Agreement for the New Greece, Agricultural and Labour Party, Free Citizens, Democratic Social Movement } 4 entities 48

49 Supervision from Unfiltered Data Social democratic parties in Greece Semantic Parser Missing X Political_Party.Ideology facts make direct Social supervision Democratic difficult. X Political_Party.Country Greece Executor {Agreement for the New Greece, Agricultural and Labour Party, Free Citizens, Democratic Social Movement } 4 entities 49

50 Supervision from Unfiltered Data Social democratic parties in Greece Semantic Parser 50

51 Supervision from Unfiltered Data Social democratic parties in Greece Semantic Parser Executor Gold mapping is expensive to gather large scale. {Agreement for the New Greece, Agricultural and Labour Party, Free Citizens, Democratic Social Movement } 4 entities 51

52 Learning with Fact Incompleteness 1. Two-Stage Learning 2. Two kinds of data i. Small Annotated Dataset ii. Broad Coverage Lexical Statistics 52

53 Two-Stage Learning Social democratic parties in Greece CCG Domain Independent Parse Ontology Matcher ONT 53

54 Two-Stage Learning Social democratic parties in Greece CCG Domain Independent Parse Small training set with logical form British playwright, novelist and short story writer, ( ) x

55 Two-Stage Learning Social democratic parties in Greece CCG Domain Independent Parse Derivations are scored using a linear model Highest scoring logical form( second stage ) is passed to the 55

56 Two-Stage Learning ONT Ontology Matcher is a large, community authored knowledge base with: 20,000 Relations 10,000 Types 100 Domains 56

57 Broad Coverage Lexical Statistics : Wikipedia Category dataset 85 K vocabulary 365 K category 2.5 M entity 7 M category-entity pair 57

58 Mapping words to Freebase Attribute Pixar Feature Films Animation Films from Pixar Pixar songs 58

59 Mapping words to Freebase Attribute Pixar Feature Films Animation Films from Pixar Pixar songs Ratatouille Wall-E Finding Nemo Toy Story Monster Inc. Just keep swimming 59

60 Mapping words to Freebase Attribute Pixar Feature Films Animation Films from Pixar Pixar songs Ratatouille Wall-E Finding Nemo Toy Story Monster Inc. Just keep swimming (film.production_companies, Pixar) (film.film.produced_by, John Lasseter) (film.film.directed_by, John Lasseter) (film.film.film_festivals, 2011 Anima Mundi ) (film.film.starring.actor, Bob Peterson) 60

61 Mapping words to Freebase Attribute Pixar Feature Films Animation Films from Pixar Pixar songs Ratatouille Wall-E Finding Nemo Toy Story Monster Inc. Just keep swimming Large amount of information (film.production_companies, Pixar) for String - Entity - Freebase attribute alignment (film.film.produced_by, John Lasseter) (film.film.directed_by, John Lasseter) (film.film.film_festivals, 2011 Anima Mundi ) (film.film.starring.actor, Bob Peterson) 61

62 Mapping words to Freebase Attribute Pixar Feature Films Animation Films from Pixar Pixar songs Pointwise Mutual Information(PMI)= Ratatouille Wall-E Finding Nemo Toy Story Monster Inc. Just keep swimming P(String, Freebase Attribute) log( ) (film.production_companies, /m/0kk9v) (film.film.produced_by, John Lasseter) (film.film.directed_by, P(Freebase John Attribute) Lasseter) (film.film.film_festivals, m.0h15pp1 ) P(String) as a feature 62

63 Features Domain Independent Parse Parse Features: CCG Lexicon, Capitalization String -> Freebase features Wikipedia Lexical Statistics Ontology Match Surface Lexical Features String Match, Stem Match KnowledgeBase Features 63

64 Outline 1. Task and Applications 2. Data 3. Semantic Parser with Partial Ontology 4. Learning 5. Evaluation

65 Experimental Setup Training Set: 500 annotated Wikipedia Category Test Set (Manual Evaluation): 500 unseen Wikipedia Category 300 appositives Baseline: SVM Classifier trained with annotated logical forms 65

66 Applications Input: Referring Expression Resolution (QA) Noun Phrase Entity Attribute Extraction (IE) (Entity, Noun Phrase) British playwright, novelist and short story writer Somerset Maugham, British playwright, novelist and short story writer Output: Entity attributes for Freebase S. Maugham Nationality U.K S. Maugham Profession Novelist Sommerset Maugham S. Maugham Profession Playwright

67 Evaluation Metric: Referring Expression Resolution Alternative Rock Groups from Nevada Gold music group(x) music.artist.origin(x, NEVADA) music.genre.artist(alternative rock, x) X X Output music.artist.origin(x, NEVADA) music.genre.artist(hard rock, x) X Precision: 0.5 Recall: 0.3 F1:0.375 Exact Match: False 67

68 Referring Expression Resolution: (5 fold cross validation on Training Set) 40% 32% % % % 6.8 0% Exact Match F1 68

69 Referring Expression Resolution: (5 fold cross validation on Training Set) 40% 32% 24% % 8% % Exact Match F1 69

70 Referring Expression Resolution: (5 fold cross validation on Training Set) 40% 32% 24% % 8% % Exact Match F1 70

71 Referring Expression Resolution: (5 fold cross validation on Training Set) 40% Baseline Our System Without OpenSchema Without Lexical Statistics KCAZ13 32% 24% % 8% 0% Exact Match 1.4 F

72 Applications Input: Referring Expression Resolution (QA) Noun Phrase Entity Attribute Extraction (IE) (Entity, Noun Phrase) British playwright, novelist and short story writer Somerset Maugham, British playwright, novelist and short story writer Output: Entity attributes for Freebase (S. Maugham, Nationality, U.K) (S. Maugham, Profession, Novelist) (S. Maugham, Profession, Playwright) Sommerset Maugham

73 Entity Attribute Extraction: (5 fold cross validation on Training Set) Baseline Our System Precision Recall F1 73

74 Entity Attribute Extraction: Test Result Baseline Our System Wikipedia Appositive 74

75 Error analysis 10% : named entity retrieving failure 10% : spurious lexical match 10% : looking at different domain e.g: stage actor to film.actor 15% : wrong underspecified logical form 30% : mapping to superset or subset e.g: novel to book 75

76 Entity Attribute Extraction: Test Result % % 61.9% % 0 Baseline Our System Baseline Our System Wikipedia Appositive 76

77 Entity Attribute Extraction: Test Result % % 61.9% 7 million category-entity pairs, Given 66% missing facts, 12 million new facts! % 0 Baseline Our System Baseline Our System Wikipedia Appositive 77

78 Contributions Introduce large-scale semantic parsing datasets Partial grounding to large knowledge base Learn from two kinds of supervision: large-scale co-occurence statistics and small labeled tuning set 78

79 Future work Barack Hussein Obama is the 44th and current President of the United States, the first African American to hold the office. More compositional and complicated structures: orders, comparison, min, max, range Extending to declarative sentences 79

80 Questions? 80

81 The Number of Extracted Facts Baseline Our System Wikipedia (Dev) Wikipedia (Test) Appositive 81

82 Referring Expression Resolution: Test Set IE Baseline Our System 50 Exact Match (%) Wikipedia 0 Appositive