Storytelling Dialogue Systems. Sophia Mallaro and Christine Chao

Size: px

Start display at page:

Download "Storytelling Dialogue Systems. Sophia Mallaro and Christine Chao"

Kimberly Woods
5 years ago
Views:

1 Storytelling Dialogue Systems Sophia Mallaro and Christine Chao

2 Agenda Introductory Discussion on Storytelling Small Talk and Conversational Storytelling Structuring Content in the Facade Interactive Drama Architecture Say Anything: A Massively Collaborative Open Domain Story Writing Companion Towards a MultiDimensional Taxonomy of Stories in Dialogue What Kind of Stories Should a Virtual Human Swap? Concluding Discussion

3 Introduction

4 What is storytelling and why does it matter?

5 What is storytelling and why does it matter? Stories are prevalent in many forms of dialogue Stories build rapport among people Stories are a form of entertainment

6 Why do you think it is important for virtual agents to tell stories?

7 Small Talk and Conversational Storytelling Bickmore, T. & Cassell, J.

8 Real Estate Agent Goals Goals determine client desires establish rapport (interpersonal) establish credibility (interpersonal) Met through small talk and task oriented dialogue Focus on adding small talk to their system

REA Primary purpose of rapport building determine that the agent is like us> human like appearance important Gaze, hand gestures, facial expressions, body posture

9 REA Primary purpose of rapport building determine that the agent is like us> human like appearance important Gaze, hand gestures, facial expressions, body posture Modelled on discourse functions (inputs and outputs modelled based on desired function) Respond to and generate speech, shifts in gaze, gesture, and nonspeech audio

10 Small Talk Purpose of small talk Establish rapport and trust Establish credibility Develop interaction style Conduct small talk by adding requisite level of interpersonal distance as preconditions for asking specific questions Example: establish rapport before asking salary

11 Conversational Storytelling "specific, affirmative, past time narratives which tell about a series of events which did take place at specific unique moments in a unique past time world" Why do humans use stories? Why do chatterbots use stories? Seem more intelligent Give listeners more material with which to establish relevance relationships among conversational contributions Listeners will tolerate irrelevant stories if entertainment or information content is high enough

12 Storytelling Goals for Real Estate Agents Phatic Keep the conversation going Establishing expertise establish the agent s linguistic expertise and intelligence. Encouraging selfdisclosure people more likely to disclose personal information when the agent does Persuasion/Problem solving tell story in which similar problem to clients is solved Providing requested information answering question with story

13 Requirements for stories Locally Occasioned Close to user in space, time or relationship Recipient Designed Tailored to specific audience

14 Implementation Requirements Manage and pursue multiple conversational goals Plan for degrees of satisfaction instead of allornothing models typical in AI planning Detect topic shifts, feedback, and repair moves by the user Extremely reactive Short response time Incremental generation Under 1.2 seconds Responsiveness and reactiveness more important than soundness and completeness

15 Structuring Content in the Facade Interactive Drama Architecture Mateas, Michael and Andrew Stern

16 Background Many types of enjoyable theatre experiences Challenges with interactivity Immersion Agency Transformation Façade architecture designed to build high agency experiences

17 The Story

18 Progression Beginning Middle Zerosum affinity game determines whose side Grace and Trip think the user is on Hotbutton game Player can trigger more intense topics like divorce and sex Therapy game Player can increases characters degree of selfrealization about their own problems Continuous Tension level

19 Architecture and Content Structure reactiveplanning language ABL Drama manager sequences dramatic beats Forwardchaining rule system for understand NLU and gestures Animation engine for rendering of dialog, music and sound

20 Beats Consists of joint dialogue behaviors Beats are narrative sequencers responsible for sequencing a subset of jdb s in response to a player interaction JDB atomic unit of dramatic action Tightly coordinated exchange of 15 lines of dialogue between Grace and Trip Organized around common narrative goal Ie, revealing a secret like Trip s attempt to force Grace to enjoy their second honeymoon in Italy

21 Beats consist of Beat Goals transitionin goal several body goals that accomplish the beat body goals establish topicspecific conflicts that force players to choose sides a wait goal Provide transition into the beat Grace and Trip wait for the player to respond default transitionout Transition out of beat Reveal information about how beat has changed dynamic

22 More on Beats Beat mix ins jdbs are specific reactions to respond to player actions Logic handler responsible for adding removing and reordering beat goals and interjecting beat mixins Global mix ins caused by player interaction, refer to specific topic Deflective mix ins also used 27 distinct beats, ~15 used per game Beat sequencer selects the next beat Characters perform low level actions in coordination with the beats such as fixing drinks

23 Coherent Intermixing Goal: break away from traditional branching narrative Multiple fronts of progression often causally independent, only occasionally interdependent Variety of narrative sequencers sequence these multiple narrative progressions operate in parallel coherently intermix their performances with one another

24 Coherent Intermixing global mixin progressions are written to be causally independent of any beats narrative flow. If a global mixin is triggered during a beat goal, the current goal freezes and the mixin performs at the level to which the issue has progressed. Issues are causally independent Ie, during affinity game beats, chosen by the drama manager, it is safe to mix hotbutton game mixins, triggered by a player s reference to their topics They may fight about their issues in any order JDB s contain various tones and are played at the appropriate tones

25 Evaluating Agency Local Agency user behavior causes immediate action 25% of time the amount of control is fully satisfying 40% of the time it is partially satisfying 25% of the time there is shallow reactivity 10% of the time there is no reactivity Global Agency Ending of story follows from user behaviors Beat sequencing allows meaningfully different experiences Variation of mix ins changes Grace and Trip either reveal all their secrets and break up, do nothing, or learn enough about themselves to stay together

26 Successes and Failures Chose local agency over global agency Facade cannot generate sequences Does not clearly communicate the state of the game Significant NLU failures Exacerbated by large domain Having two characters allows the game to progress even if the user stops interactions

27 Say Anything: A Massively Collaborative Open Domain Story Writing Companion Swanson, R. and Goran, A.S

28 What is collaborative story writing?

29 Say Anything: Collaborative Story Writing Application Functionality Corpus 3.7 million web blog entries Focused on stories (17% of web content) Retrieval: Apache Lucene Generates the next sentence by finding the most similar sentence and returning the next sentence in that story combines standard Boolean indexing with term frequencyinverse document frequency (TFIDF) scoring functionality Interface Web based Panel for viewing and panel for writing

30 Evaluation

31 Challenges People who write on the internet are not all good writers Implied language Assumed drinking related to alcohol Rare words (assumes they re important) Negation Global coherence

Question from K.R.: Since "Say Anything" was published in 2008, there has been significant progress in modelling language using continuous embeddings. {.

32 Question from K.R.: Since "Say Anything" was published in 2008, there has been significant progress in modelling language using continuous embeddings. {...} It seems like these techniques would be very likely to provide a significant improvement compared to the system described in "Say Anything". Has this been done?

33 Towards A MultiDimensional Taxonomy of Stories in Dialogue Kathryn J Collins and David Traum

34 Objectives This paper suggests a taxonomy that can be used for classifying and annotating how stories are told in dialogue. This taxonomy can be used to: Automatically recognize aspects of stories Retrieve and retell stories in appropriate places in dialogue

35 Identifying Stories Labov s narrative structure categorizes dialogue acts as structural components Orientation expository or setting information 2. Action temporally ordered narrative clauses Evaluation embedded speech, evaluative commentary, and other nonnarrative related discussion This structure can act as a reference to distinguish story elements from nonstory elements in dialogues.

36 Possible Categorizations of Stories Source Official Invented/adapted Firsthand experiential Secondhand Culturally common Intention Me goals You goals Conversational goals

37 Proposed Scheme Narrative Point of View a. Number b. Person Story Representation a. Specificity b. Truth Value Orientation in Dialogue a. Source b. Function in Dialogue Identity a. Explicit Identity b. Identity c. Identity Testimony Affect

38 Proposed Scheme Narrative Point of View a. Number b. Person Story Representation a. Specificity b. Truth Value Orientation in Dialogue a. Source b. Function in Dialogue Identity a. Explicit Identity b. Identity c. Identity Testimony Affect Describes the relationship of the narrator to the protagonists of the story. The number of characters POVs are shown in a story. Singular, Plural or Mixed The relationship of the narrator to the protagonist. First person, Second person/generic, or Mixed

39 Proposed Scheme Narrative Point of View a. Number b. Person Story Representation a. Specificity b. Truth Value Orientation in Dialogue a. Source b. Function in Dialogue Identity a. Explicit Identity b. Identity c. Identity Testimony Affect How the story is to be understood as relating to actual events. Type of incident described in the story. Specific incident, Hypothetical incident, Habitual incident, Vague Categorizes real and fictitious events True story, Adapted story, Invented story, Vague

40 Proposed Scheme Narrative Point of View a. Number b. Person Story Representation a. Specificity b. Truth Value Orientation in Dialogue a. Source b. Function in Dialogue Identity a. Explicit Identity b. Identity c. Identity Testimony Affect The story s relation to the dialogue. The source of the story. Firsthand, Secondhand, Culturally common, Official, Vague Why the story has been told in the dialogue. Answer a question, Mirror story response, Support claim/statement, Refute/repair claim/statement, Transition story relevant to overall dialogue, Elaborate/continue previous story/statement, Vague

41 Proposed Scheme Narrative Point of View a. Number b. Person Story Representation a. Specificity b. Truth Value Orientation in Dialogue a. Source b. Function in Dialogue Identity a. Explicit Identity b. Identity c. Identity Testimony Affect The relationship of the story to the constructed identity of the narrator. Checks to see if the identity of the narrator is explicitly stated or relied upon in the story. Explicit/Relevant identity, Unspecified/Irrelevant identity If there is an explicit and relevant identity, the narrator s identity is annotated. Describes how the story relates to the identity. Story supports identity, Identity enhances Story, Neutral/Mixed.

42 Proposed Scheme Narrative Point of View a. Number b. Person Story Representation a. Specificity b. Truth Value Orientation in Dialogue a. Source b. Function in Dialogue Identity a. Explicit Identity b. Identity c. Identity Testimony Affect How the story affects the audiences emotions. Emotionally Relevant, Emotionally Irrelevant, Vague

43 Annotated Dialogue Data Corpora Switchboard Distress Analysis Interview Corpus Annotation Tool TAMSAnalyzer

44 Results 82 stories were identified from the 20 dialogue transcripts from switchboard, annotated using the taxonomy

45 Conclusions and Future Work Conclusions: Presented a multidimensional taxonomy of stories told in dialogue. Annotated switchboard corpus and examined the relative frequencies of the categories. Future Work: Test/Training Set for recognizing the features in the taxonomy, for example: number, person and function. Use distributional information to inform storytelling virtual agents in conversation.

46 Question from Piazza In section Story Presentation of paper "Towards A MultiDimensional Taxonomy Of Stories In Dialogue" it says longer narratives including sections with several types of specificities can be annotated as mixed specificity. Can we try to preserve which types it includes and generate more specific story dialogue of it rather than just a "mixed type"?

47 What Kind of Stories Should a Virtual Human Swap? Setareh Nasihati Gilani, Kraig Sheetz, Gale Lucas and David Traum

48 Storyswapping Stories are often swapped when one person replies to a story with another story. ¼ of stories in casual conversation were presented in response to stories told by the other participants. Story swapping can be a way for humans to stay engaged and build rapport with a virtual storytelling agent.

49 Principles for a Storytelling Virtual Human Be human Talk about yourself Be real Be interesting Don t gossip

50 Storyswapping Agents Virtual Human Human First Person/Third Person First Person/Third Person VHVH/VHHuman 1. Character/Perspective 2. Identity 3. Embodiment Presentation

51 VHVH/VHHuman Example VHVH response: VHHuman response: I have talked to some celebrities, but unfortunately I can t really get an autograph or a picture to show my friends. I do have all of the conversation logs though, even from my first time ever talking to a celebrity, when I talked to Hines Ward,... I m not huge into celebrities, but one time in college I saw a flyer that said that one of my childhood sports heroes, Hines Ward, was coming to do a signing on campus. I went and bought a football from the local sports store and headed over to the signing

52 Experimental Design 60 participants 10 question conversation with the agents Focused on perspective Dependent Variables: 9item rapport scale 6item ancillary rapport scale Two items on subjective sharing of personal information A set of 30 personality characteristics Length of participant responses Number of participant responses containing stories Identity Presentation Perspective Order VHVH First/Third VHVH Third/First VHHuman First/Third VHHuman Third/First HumanHuman First/Third HumanHuman Third/First

53 Results

54 Results 6a. 3rd person agent is seen as more threatening when users interact with the 1st person agent beforehand. 6b. 1st person agent is perceived as cheerful only if interacted with first. 7a. Unsympathetic effect on order for VHHuman agent 7b. Participants talked longer with whatever agent they spoke to second.

55 Summary Prefer first person over third person stories Users said they had greater rapport, shared more information, and saw the agent as less rude, less aloof, less threatening, more cheerful, and more trustworthy First person, humanlike stories should be told by virtual humans

56 Future Work Vary the withinsubjects variables Gender effects Other subject matter for stories and main task Mix of first and third person stories for an agent

57 Discussion

58 What attributes do you think are important for a storytelling agent to have?

59 How could storytelling agents be used in the real world?

60 Thank You! Questions?

Towards A Multi-Dimensional Taxonomy Of Stories In Dialogue

Towards A Multi-Dimensional Taxonomy Of Stories In Dialogue Kathryn J. Collins(1), David Traum(2) (1) Swarthmore College (2 )Institute for Creative Technologies, University of Southern California kcollin3@swarthmore.edu,