An ECA Expressing Appreciations

Transcription

1 2015 International Conference on Affective Computing and Intelligent Interaction (ACII) An ECA Expressing Appreciations Sabrina Campano, Caroline Langlet, Nadine Glas, Chloe Clavel Catherine Pelachaud Institut Mines-Te le com, Te le com-paristech, CNRS-LTCI 46 Rue Barrault, Paris, France Telephone: +33(0) CNRS-LTCI, Te le com-paristech 46 Rue Barrault, Paris, France Telephone: +33(0) Abstract In this paper, we propose a computational model that provides an Embodied Conversational Agent (ECA) with the ability to generate verbal other-repetition (repetitions of some of the words uttered in the previous user speaker turn) when interacting with a user in a museum setting. We focus on the generation of other-repetitions expressing emotional stances in appreciation sentences. Emotional stances and their semantic features are selected according to the user s verbal input, and ECA s utterance is generated according to these features. We present an evaluation of this model through users subjective reports. Results indicate that the expression of emotional stances by the ECA has a positive effect on user engagement, and that ECA s behaviours are rated as more believable by users when the ECA utters other-repetitions. Keywords other-repetition; engagement; alignment; emotional stance; embodied conversational agent I. I NTRODUCTION Embodied Conversational Agents (ECAs) are computergenerated characters that are able to produce and respond to verbal and nonverbal communication. Fostering user s engagement [1] during interactions with ECAs or robots is an important process in human-agent interaction. A disengaged user may leave the interaction too early, and prevents the agent from completing its task. A method that may contribute to engaging a user with an ECA is to simulate alignment processes. Various terminology are used to design alignment processes or similar processes. These processes differ on the way they integrate the temporal and dynamic aspects. For example, mimicry is defined as the direct imitation of what the other participant produces [2], while synchrony is defined as the dynamic and reciprocal adaptation of temporal structures of behaviors between interactive partners [3]. Implementations of alignment strategies in human-computer dialogues concerned mainly alignment on lexical and syntactic choices [4], while the community of human-agent face-to-face interaction furthers implementations of non verbal alignment using terminologies that are slightly different from the one used in corpus studies: mimicry [5], synchrony [3], social/emotional resonance [6], [7], emotional mirroring [8], dynamical coupling [9]. The work presented in this paper is conducted in the framework of the national french project A1:11, in which a human-size Embodied Conversational Agent (ECA) is being developed and is dedicated to sustain face to face interaction with museum visitors. The scientific focus of the project is to trigger and maintain user s engagement during his/her interaction. The ECA has knowledge about the museum s In this context, the present study focuses on the development of verbal-centered alignment strategies acting at two levels: the appreciation level (sharing appreciation) and at the lexical level by using other-repetition (OR) to express appreciations. OR is the intentional repetition by the hearer of part of what the speaker has just said, in order to convey a communicative function that was not present in the first instance [10] [2]. To our knowledge, no computational model of OR has been proposed so far in human-agent interaction. To take a step in this direction, we focus on one of the communicative functions of OR: OR expressing emotional stances [10]2. In this process, the hearer repeats a part of speaker s last sentence in order to convey surprise, or a negative / positive evaluation about it. Emotional stances about museum topics can be expressed through the verbal form of appreciations which is a basic activity for visitors in a museum [11]. Such type of OR has been identified in our previous work grounded on corpora analysis, including the human-agent interaction corpus SEMAINE [12]. In this paper, we describe and evaluate a computational model of OR, that allows an ECA to convey emotional stances through language by using sentences expressing an appreciation in response to the user s previous utterance. The model is able to select a given emotional stance, and to generate the corresponding appreciation sentence to be said by the ECA (Section II). The evaluation methods of the model a user study including questionnaires addressed to 33 participants and the arising results are presented in Section III. II. S ELECTION AND G ENERATION OF A PPRECIATION S ENTENCES The Detection of User s Appreciations Module (DetectAppr Module) transmits a set of semantic features corresponding to the user s last utterance as inputs to the Other-Repetition with Emotional Stances Module (OR-Emo Module). Inputs of the OR-Emo Module also include Agent s Preferences. The OR-Emo Module proceeds in two steps detailed in Figure 1. First (blue boxes of Figure 1), the module selects an emotional stance according to the inputs (semantic features of user s sentence and agent s preferences), and second (red boxes of Figure1), it generates a sentence corresponding to the emotional stance which was selected. This output sentence to be uttered by the agent contains an appreciation including an 2 Other communicative functions are expressing understanding, formulating a request for clarification, promoting topical talk /15/$ IEEE objects and artworks, and its role is to discuss about them with visitors. 962

2 Given a set T opics of different conversation topics, agent s preferences on T opics represent agent s liking/disliking for each topic (e.g. a specific painting or artist). Let agt be the agent conversing with the user usr. topic T opics, P ref agt (topic) [ 1, 0[ ]0, 1] is the preference value of topic from the point of view of agent agt (if P ref agt (topic) < 0, agt dislikes the topic). In this work, pol(p ref agt (topic)) refers to the polarity of agent s preference about topic. Fig. 1: Decision tree of the OR-Emo Module. The process starts once the user has produced a sentence. other-repetition. The OR-Emo module has been built both on theoretical studies [10] for the selection of emotional stances with OR and [13] for the generation of appreciation sentences and on the study of the SEMAINE corpus [12]. The module has been integrated in a Human-Agent system. The GRETA system [14] is used for the agent model. The dialogue model used in the GRETA platform is DISCO [15], a hierarchical task network. A. Inputs: User Appreciations and Agent Preferences A list of semantic features is associated to each user s sentence expressing an appreciation. In this study, the detection of user s appreciations is carried out manually by a human expert (acting as a Wizard Of Oz) during the interaction between a human participant and an agent. We built a database of appreciation sentences, and we associated each sentence with semantic features. During the interaction, when the human expert detects an appreciation, he/she selects the appreciation sentence that best matches the appreciation sentence uttered by the participant. The human expert is trained to this task and provides a quick answer that does not damage the quality of the interaction. When the expert selected the appreciation sentence, its semantic features were transmitted to the OR-Emo Module 3. Given a user usr, each appreciation App usr is associated to a semantic feature set following Martin and White s model [13] as described in [16]. The feature set includes the polarity of the appreciation pol Appusr = {positive, negative}, the source 4 of the appreciation src Appusr, the target of the appreciation (target Appusr T ), the lemmatized form of the appreciation lexia lexialem Appusr belonging to a list L of appreciation lexia, the part of speech of the lexical unit (lexiap ost ag Appusr {ADJ, V ERB}) expressing the appreciation. For example, the semantic features corresponding to the sentence I don t like the baroque style uttered by usr are: pol Appusr = negative; src Appusr = usr; target Appusr = baroque style; lexialem Appusr = like; lexiap ost ag Appusr = V ERB. 3 We are currently developing an automatic appreciation detection system [16] that is devoted to replace the Wizard of Oz. 4 In our interaction model, only the appreciations for which the user is the source in the user utterance have been considered, which is not necessarily the case: in the sentence My wife loves Klimt, where the source is the user s wife. B. Selecting Emotional Stance and Appreciation Sentences with OR The selection process is rule-based: we use a binary decision tree (Figure 1), which allows for a representation of conditional rule-based decision processes. This decision process is grounded on theoretical work issued from linguistic studies. Relying on the concepts of appreciation [13] and other-repetition [10], two levels of alignment between the ECA and a user are represented in our model. The first one concerns alignment at the level of the polarity (positive / negative) between a user s appreciation and an ECA s appreciation on the same target. In this case, the ECA shares its appreciation with the user. The second one concerns alignment at the lexical level with other-repetition [10]. In this case, the ECA expresses an appreciation in using intentionally word(s) previously uttered by the user. Thus, an emotional stance is selected according to: (i) the presence or absence of an appreciation in user s last sentence, (ii) the polarity of user s appreciation in user s last sentence, that can be the same or divergent from agent s preference for the target of user s appreciation. Two main decision parameters are set according to the inputs described in Section II-A. The first one is the appreciation {T rue, F alse} variable. It equals to T rue when the Detect-Appr module detected at least one appreciation in the user s last speaking turn and F alse otherwise. The second one is the divergence(app usr (target : name(topic), P ref agt (topic)) {T rue, F alse} variable. It equals to T rue when the polarity of the user s appreciation for the target name(topic) is the same as the polarity of the agent s preference for topic, and F alse otherwise. Formally, we denote App usr as an appreciation originating from a user usr, with target Appusr = name(topic). The decision process is carried out through a decision tree visible in Figure 1. It shows how an OR-Emo is selected. The ECA has the possibility to repeat an appreciation word uttered in the user s last sentence, denoted as lexialem Appusr, or a topic name uttered in last user s sentence, denoted as name(topic) T. The selection of the repeated word and the generation of the sentence is detailed in Section II-C. In case where multiple appreciations are detected in the user s speaking turn, the OR- Emo model takes into account the last appreciation formulated by the user, denoted as App usr. When a user s appreciation App usr is detected (left branch of the decision tree), the ECA has the ability to repeat the appreciation word lexialem Appusr. This could lead to the expression of two OR-Emo types. If there is no divergence (divergence(app usr (target : name(topic), P ref agt (topic)) = F alse), the ECA will display an Aligned OR-Emo. That means the polarity of ECA s appreciation will be the same as the polarity of user s appreciation App usr. When /15/$ IEEE 963

3 there is a divergence (divergence(app usr (target : name(topic), P ref agt (topic)) = T rue), then the ECA expresses a Surprise OR-Emo. Surprise is here modelled as a non valenced emotional stance. When an other-repetition formulated by the hearer expresses surprise, it is a request for elaboration that asks the speaker to tell more [10]. Using surprise is thus an interesting strategy to foster user s engagement because it allows us to (i) avoid disalignment, (ii) make the user tell more about a given topic 5. When no user s appreciation is detected (right branch of the decision tree), there are two possibilities: if the user uttered a topic name name(topic) T, the ECA still has the possibility to repeat this name(topic) within an appreciation (ex: I like name(topic) ). In this case, a Basic OR-Emo is selected. It represents a positive or negative emotional stance, which is unaligned (different from oppositely aligned); if no topic name was uttered, then the default sentence is selected. It does not represent any emotional stance. This sentence should be consistent with the scenario, and the topic which is currently discussed. It has to be pre-defined, and it is passed as parameter to the OR-Emo Module during the interaction. C. Generation of the Agent s OR-Emo Sentences For each kind of OR-Emo in the decision tree, one specific pattern for ECA s sentence is defined. First, the model provides rules in order to define relevant semantic features of the agent s utterance: the sentence form assertive or interrogative (sentencef orm : [interrogative assertive]) and the feature set of AppBase. The Surprise OR-Emo pattern has an interrogative form starting with a NewsMark 6 (e.g. Really ), the source of the appreciation is the user (agent repeats user s appreciation), and the polarity of AppBase corresponds to the polarity of the user s appreciation pol Appuser (ex: Ah bon, vous n aimez pas Picasso? in french, transl. : Really, you don t like Picasso ). The Basic OR-Emo and Aligned OR-Emo patterns have an assertive form, the source of the appreciation is the agent (it expresses its own appreciation), and the polarity of AppBase corresponds to the agent s agt preference pol(p ref agt (t)) (ex: Moi non plus, je n aime pas Picasso in french, transl. : Me neither, I don t like Picasso ). In Aligned OR-Emo, a syntactic variable AlignmentExpr is added at the beginning of the sentence. Two simple phrases express that the agent agrees with the user s appreciation: (i) AlignmentExpr(negation : f alse) Moi aussi ( Me too in english) (ii) AlignmentExpr(negation : true) Moi non plus ( Me neither in english). Once the AppBase features have been defined, its syntactic form is produced by using a predefined appreciation pattern. We defined two kinds of pattern: an adjectival one, (AppBase(termCat : adj)), which is used when the appreciation word is an adjective (AppAdj), and a verbal one (AppBase(termCat : verb), which is used when the appreciation word is a verb (AppV b). The adjectival pattern has for main verbal form (OpinionV b) trouver que ( to think, to consider in english), followed by an adjective. An example in english would be I consider this painting as beautiful. Two 5 Other work models surprise as a general emotional reaction to beliefdisconfirmation [17]. 6 which implies information that is treated as both new and as surprising or interesting [10]. forms are defined for this adjectival pattern: a negative one and an affirmative one. These two forms can be used either for an agent s appreciation (AppBase(src : agent)), or for a user s one (AppBase(src : user)). When AppBase(src : user), the pronoun has second person plural form ( vous ) and when AppBase(src : agent), it has a first person singular form ( je ). In the verbal pattern, the appreciation on the target is conveyed by the verb. An example in english would be I love this painting. The verbal pattern also has a negative and a positive form. Again, these two forms of the verbal pattern are used either for an agent s appreciation (AppBase(src : user)), or for a user s one (AppBase(src : agent)). As in the adjectival pattern, the verb is conjugated to the right form according to the pronoun and the nature of the source (user or agent). The ECA s sentences are accompanied by non verbal behaviors, corresponding to performative acts such as argue, inform, or to emotional stances (negative, positive, or surprise). The agents communicative intentions are described in the FML-APML standardised format [18]. III. A. Experimental Conditions EVALUATION AND RESULTS We rely on subjective measures of user engagement, assessed through a questionnaire. We also aim at the evaluation of the ECA s believability, as perceived by the user. We used 3 conditions for our experiment: 1) When the dialogue system is set in EMO-WITH-OR condition, the OR-Emo model is activated, and the ECA expresses emotional stances with ORs, following the decision rules of Figure 1. 2) When the dialogue system is set in EMO-WITHOUT-OR condition, the ECA expresses emotional stances without ORs. To do so, a list of generic appreciation words with negative or positive polarity has been defined. The word in the generated sentence is randomly chosen from this list (excluded the user s appreciation word), according to the polarity that must be expressed. 3) When the dialogue system is set in NO-EMO condition, the ECA does not express emotional stances. Instead, it utters a pre-defined sentence, that has approximately the same length as a verbal emotional stance produced by the OR-Emo model. These pre-defined sentences are written in the scenario script, such as when this condition is applied, the default sentence is returned instead of the OR-Emo model output. We asked human participants to visit an improvised museum, then talk to the virtual agent called Leonard that takes the role of another visitor, and ultimately fill in a questionnaire. We hung 4 pictures of existing artworks in the corridor. The objects were chosen as to vary in style and type of affect they might evoke: a foto of the exhibition of Balloon Dog by Jeff Koons, and printed images of the paintings The Kiss by Gustav Klimt, Composition A by Piet Mondrian, and The Anatomy Lesson of Dr. Frederick Ruysch by Jan Van Neck. We placed another artwork between the screen of the virtual agent and the user that serves as a first conversation topic: the picture of a statue named Soldier drawing his Bow, by Jacques Bousseau. Each participant went through one condition only, which was kept all along the interaction. Leonard was displayed on a 75-inch vertically placed screen and has the appearance of a /15/$ IEEE 964

4 cartoon-like version of a man of about 70 years old. Pictures of the experiment can be found in [19]. B. Questionnaire The questionnaire used in the experiment is composed of 5 separate sections, devoted to the evaluation of the OR-Emo model. The questions for sections 2 to 5 originally written in French are rated in a 7-point scale: from Not at all to Very much. Section 1 collects the user profile (5 items): birth date, gender, residence country for the 5 last years, education level, industry. Section 2 concerns engagement in the interaction grounded on the definition by [20]. Thus, we create 4 items 7 to assess both the user s own engagement and the perception of agent s engagement by the user: During the interaction, to what extent did you: 2.1 want to stay together with Leonard? 2.2 think Leonard would like to stay together with you? 2.3 want to continue the conversation? 2.4 think Leonard would like to continue the conversation? Section 3 concerns engagement in the interaction grounded on the Temple Presence Inventory (TPI) questionnaire [21]. This questionnaire already used in [1] aims at measuring a person s immersive tendency, or presence, in a virtual environment. As recommended by the authors of the TPI, we selected individual items that were useful and appropriate for our study, and we adapted the selected items to a human-agent interaction context. We added a supplementary item on the general liking of the interaction. The corresponding questions are presented to the user as follows: To what extent: 3.1 did you feel involved in the interaction? 3.2 this experience was boring or lively? 3.3 the information delivered by Leonard was interesting? 3.4 did you like this interaction? Section 4 is dedicated to the user s perception of the agent s emotional stances and appreciations (5 items). It aims at assessing whether the user (i) perceived that the agent reacted to user s appreciations, (ii) perceived these reactions as appropriate (iii) perceived that the agent has its own preferences (iv) felt that the agent and he/she share the same preferences. This makes it possible to assess whether the emotional stances and the appreciations formulated by the agent are effectively perceived by the user, and whether the difference between the agent s preferences and the user s appreciations could play a role in the user s engagement. For the questions included in this section, we used the word opinion instead of appreciation, in order to clarify the terminology for the user. The user is given the following instructions: Please indicate your level of agreement with the following statements. 4.1 In general, Leonard took into account what I said during the conversation. 4.2 Leonard reacted to the opinions I expressed. 4.3 When Leonard reacted to my opinions, it did it in an appropriate manner. I had the feeling that: 4.4 Leonard had its own opinions. 4.5 Leonard and me share the same opinions. Section 5 collects user s perception of agent s believability (6 items). The items of this section are inspired from the definition of believable agents by Bates (1994) [22], the TPI questionnaire [21], and the evaluation protocol used by [23] for assessing agent s believability. The user is given the following instructions: Please indicate your level of agreement with the 7 the same items have been used in [19] for another study following statements: I had the feeling that: 5.1 Leonard s utterances and behaviours are consistent. 5.2 Leonard s utterances and behaviours are lively. 5.3 Leonard is able to think by itself. 5.4 Leonard has feelings. 5.5 Leonard s utterances and behaviours are common to occur in human behaviour. 5.6 Leonard s utterances and behaviours could occur in human behaviour. C. Results and Discussion 33 participants (13 females and 20 men) took part in the study. They were recruited in the offices of Telecom-ParisTech (Paris, France), and were external to the virtual agents team. They worked in research laboratories or companies, and included administrative staff as well as researchers and engineers with a good level in French. 33 interaction sessions of approximately 6-10 minutes length were thus recorded. Before analysing the results, we applied the ShapiroWilk test [24] to each individual sample by group to check whether the samples are likely to come from a normally distributed population. For a quite large amount of samples ( 39%), the results indicate that the null hypothesis can be rejected (p < 0.05). This means that these samples are likely to come from a non normally distributed population. Hence, for all of the following results, we used statistical tests for non parametric data. As each participant has passed the experiment with only one experimental condition, the statistical tests to be used are then for non repeated measures. When only two samples were compared, we used the Wilcoxon rank sum tests [25] with continuity correction (W, p). When more than two samples were tested, we used the Kruskall Wallis test (χ 2 (2), p) [26], and then a post-hoc test which is Mann-Whitney with Bonferroni correction. The post-hoc test is used to do pairwise comparisons and obtain supplementary information. In order to test whether several items in the questionnaire can be combined into a single Likert s scale, we used the Cronbach s α (alpha) that verifies whether several items measure the same construct. [27]. Perception of the agent s emotional stances with appreciations. We analysed the results corresponding to the section 4 in the questionnaire, focusing on the answers of questions 4.3 and 4.5. Regarding 4.3 answers, when the ECA reacted to the user s appreciations, participants found that it reacted in an appropriate manner: EMO-WITHOUT-OR µ = 5.45, σ = 1.13, EMO-WITH-OR µ = 5.45 σ = 0.69 and NO-EMO µ = 4.27 σ = Overall, users had the feeling that they share the same appreciations as the ECA when it expressed emotional stances. This is shown by the ratings obtained for question 4.5. The statistics of this item for the groups are: EMO-WITHOUT-OR µ = 4.64, σ = 1.69, EMO-WITH-OR µ = 4.73 σ = 1.01 and NO-EMO µ = 3.64 σ = Regarding the impact of ORs, the comparison between groups EMO-WITH-OR and NO-EMO gave a significant result (W = 90.5, p < 0.05), whereas the comparison between groups EMO-WITHOUT-OR and NO-EMO was not significant (W = 86, p = ). This means that when the ECA expresses appreciations with ORs, it significantly reinforces the users feeling that they share the same appreciations as the ECA, in comparison with when the ECA did not express emotional stances. When the ECA expresses appreciations without ORs, the results suggest that /15/$ IEEE 965

5 there is no difference in comparison with when the ECA did not express emotional stances. This result concerning the ORs is encouraging as the sharing of appreciations is important to build rapport and affiliation between two speakers, which contributes to their engagement [28] Engagement. We tested whether the items of the two sections (2 and 3) can be aggregated into a single Likert scale. The Cronbach s α measure of scale reliability for the Likert items 8 is α = 0.903, which indicates an excellent reliability. We combined these scales into a single Likert scale, representing the user s engagement. A Kruskall-Wallis test on the user s engagement Likert scale for the 3 groups gave no significant results (χ 2 (2) = , p = ), which means that the 3 groups cannot be distinguished from each other. The mean and standard deviation by group are shown in Table I. The highest engagement score was obtained for the EMO-WITHOUT-OR group, closely followed by the EMO- WITH-OR condition. The results suggest that when the agent formulates emotional stances with ORs, user s engagement is not improved compared to when it expresses emotional stances without ORs. This result can be compared to the one obtained by Ivaldi et al. [29], who found that their gaze mechanism allows for improving the pace of interaction, but does not seem to increase perceived engagement. Subjective measures may not fully reflect the user s engagement; subjective measures should be completed by objective ones, such as the analysis of user s facial expressions, mutual gaze, speech or verbal content [30]. In order to evaluate the impact of ECA s expressions of emotional stances on user s engagement, two Wilcoxon rank sum tests with continuity correction were performed. The comparison between groups EMO-WITH-OR and NO- EMO gave no significant results (W = 84, p = ), as the comparison between the groups EMO-WITHOUT-OR and NO-EMO (W = 84, p = ). However, the null hypothesis can be rejected with a 87% confidence interval. This confidence interval suggests that it would be interesting to re-conduct the experiment, in order to test again the hypothesis that an ECA expressing emotional stances could enhance user engagement. During our experiment, we noticed that when the agent expressed an emotional stance about a topic, the user often responded in doing so and the expression of emotional stances from the agent seems thus to have a positive impact on user engagement. Although this has to be confirmed in a further study, this result shows that the expression of emotional stances containing verbal appreciations is an interesting feature to include in the capabilities of an ECA. As a supplementary result, we evaluated the impact of the 3 conditions on the agent s engagement as perceived by the users. The questions 2.2 and 2.4 were aggregated into a single Likert scale (average). The result is not significant (χ 2 (2) = , p = 0.154), but the confidence interval (85%) suggests that the experiment could be re-conducted. The statistics for the groups EMO-WITHOUT-OR, EMO-WITH- OR and NO-EMO are respectively µ = 5.82, σ = 1.19, µ = 5.18 σ = 1.27 and µ = 4.68 σ = Hence, agent engagement as perceived by users obtained a lower rating when emotional stances contained ORs, compared to when they did 8 As items 2.2 and 2.4 are related to the agent s engagement, they were not considered for the aggregation. TABLE I: MAIN STATISTICS User s Engagement Believability group mean sd mean sd EMO-WITH-OR EMO-WITHOUT-OR NO-EMO not. It would be interesting to re-conduct an experiment, in order to assess whether the user is less engaged when an agent uses the same vocabulary (appreciation words, and topic names). However, in the next paragraph we will see that when the agent uses the same words as the user, its behaviours are rated as more believable. These results offer an interesting research perspective, showing that the link between ORs and believability / engagement should be further explored. Believability. Cronbach s α measure is α = for the items of Section 5, which indicates a good reliability. The Likert items corresponding to the evaluation of the agent s believability were then averaged into a single Likert scale. A Kruskall-Wallis test was performed on this Likert scale for the 3 groups, which gave results very close to a significant threshold (χ 2 (2) = , p = ). This suggests that there is a difference between the 3 groups regarding agent s believability perceived by the user. Table I shows that the highest mean is obtained with the EMO-WITH-OR group and the lowest with the NO-EMO group A post-hoc test using Mann-Whitney tests with Bonferroni correction gave non significant results (p = for groups NO-EMO and EMO- WITHOUT-OR). This suggests that there is no difference between the two groups NO-EMO and EMO-WITHOUT-OR. A Kruskall-Wallis test was performed for each item of Section 5 in the questionnaire, in order to identify the most discriminant items. Regarding the answers to the question 5.4, the test gave a significant result (χ 2 (2) = , p < 0.01). The post-hoc test using Mann-Whitney tests showed significant differences between the groups EMO-WITH-OR and NO- EMO (p < 0.01), and between the groups EMO-WITHOUT- OR and NO-EMO (p < 0.05). The difference between the means for the group NO-APPR and the other groups is quite important (Table II). This means that users tend to think that the agent has feelings when it expresses emotional stances with appreciations, whereas in the other case they tend to think it has no feelings. Regarding the answers to the question 5.3, a Kruskall- Wallis test revealed a non significant effect of group, but the p value is close to the threshold of significance (χ 2 (2) = , p = ). The post-hoc test using Mann-Whitney test showed no significant differences, despite a good confidence interval for the groups EMO-WITH-OR and NO-EMO (p = 0.09, hence 91% confidence). The means for the 3 groups are above or equal to the medium of the scale (Table II). This experiment has to be re-conducted to test this hypothesis again, but these results could indicate that users tend to think that the agent has its own feelings rather when it expresses emotional stances with appreciations than when it does not. Regarding the answers to the question 5.6, the Kruskall- Wallis test gave a non significant result, despite a p value close to a significant threshold (χ 2 (2) = , p = ). The means for the 3 groups are shown in Table II. It can be /15/$ IEEE 966

6 TABLE II: STATISTICS FOR 3 ITEMS IN THE BELIEVABILITY SECTION Feelings (5.4) Thoughts (5.3) Possible behaviours (5.6) group mean sd mean sd mean sd EMO-WITH-OR EMO-WITHOUT-OR NO-EMO noticed that for emotional stances containing ORs, the agent s behaviours were rated as more believable (especially, more common to occur) than for emotional stances without ORs. IV. CONCLUSION This study presented a computational model of otherrepetitions (ORs) conveying emotional stances, that can be used by an Embodied Conversational Agent (ECA) in response to the user s utterances. The evaluation of the model showed that the expression of verbal emotional stances by an ECA tends to improve the user s perception of his/her own engagement. The OR model also seems to have an impact on some aspects of ECA s believability as perceived by the user. The results suggest that the experiment should be re-conducted to confirm/disconfirm this hypothesis. On the other hand, OR does not seem to affect the user s perception of his/her own engagement. This indicates that the study of ORs should be completed by objective measures for a comprehensive exploration of their contribution. However, the presence of ORs in the ECA s appreciations had a positive effect on the participants feeling that they shared the same appreciations as the ECA. We are currently integrating in our model other inputs such as the user s level of engagement (based on the level of the user s talkativeness), in order to juggle with timing aspects for the triggering of appreciations [31]. We further plan to replicate the evaluation protocol used in the present study. For future work, we would also like to add objective measures of engagement, such as gaze cues or speech rate. ACKNOWLEDGMENT The authors thank the members of Laboratoire Parole et Langage, France for valuable insights and suggestions, as well as A1:1 partners and Greta team of Telecom-ParisTech for their help in the experimental setup. REFERENCES [1] C. L. Sidner, C. Lee, C. D. Kidd, N. Lesh, and C. Rich, Explorations in engagement for humans and robots, Artificial Intelligence, vol. 166, no. 1, pp , [2] R. Bertrand, G. Ferré, M. Guardiola et al., French face-to-face interaction: repetition as a multimodal resource, Coverbal Synchrony in Human-Machine Interaction, p. 141, [3] E. Delaherche, M. Chetouani, A. Mahdhaoui, C. Saint-Georges, S. Viaux, and D. Cohen, Interpersonal synchrony: A survey of evaluation methods across disciplines, T. on Affective Computing, vol. 3, no. 3, pp , [4] H. Buschmeier, K. Bergmann, and S. Kopp, An alignment-capable microplanner for natural language generation, in Workshop on Natural Language Generation. ACL, 2009, pp [5] U. Hess, P. Philippot, and S. Blairy, 8. mimicry, The Social Context of Nonverbal Behavior, p. 213, [6] J. Gratch, S.-H. Kang, and N. Wang, Using social agents to explore theories of rapport and emotional resonance, Social Emotions in Nature and Artifact, p. 181, [7] S. Kopp, Social resonance and embodied coordination in face-toface conversation with artificial interlocutors, Speech Communication, vol. 52, no. 6, pp , [8] J. C. Acosta and N. G. Ward, Achieving rapport with turn-by-turn, user-responsive emotional coloring, Speech Communication, vol. 53, no. 910, pp , [9] K. Prepin, M. Ochs, and C. Pelachaud, Beyond backchannels: coconstruction of dyadic stance by reciprocal reinforcement of smiles between virtual agents, in CogSci, [10] J. Svennevig, Other-repetition as display of hearing, understanding and emotional stance, Discourse studies, vol. 6, no. 4, pp , [11] L. H. Silverman, Meaning making matters: Communication, consequences, and exhibit design, Exhibitionist, [12] S. Campano, J. Durand, and C. Clavel, Comparative analysis of verbal alignment in human-human and human-agent interactions. LREC, [13] J. R. Martin and P. R. White, The language of evaluation. Palgrave Macmillan Basingstoke and New York, [14] E. de Sevin, R. Niewiadomski, E. Bevacqua, A.-M. Pez, M. Mancini, and C. Pelachaud, Greta, une plateforme d agent conversationnel expressif et interactif, Technique et science informatiques, vol. 29, no. 7, p. 751, [15] C. Rich and C. L. Sidner, Using collaborative discourse theory to partially automate dialogue tree authoring, in Intelligent Virtual Agents. Springer, 2012, pp [16] C. Langlet and C. Clavel, Improving social relationships in face-to-face human-agent interactions: when dislikes, in ACL, 2015, to appear. [17] R. Reisenzein, E. Hudlicka, M. Dastani, J. Gratch, K. V. Hindriks, E. Lorini, and J.-J. C. Meyer, Computational modeling of emotion: Toward improving the inter- and intradisciplinary exchange, T. Affective Computing, vol. 4, no. 3, pp , [18] M. Mancini and C. Pelachaud, The fml-apml language, in Proc. of the Workshop on FML at AAMAS, vol. 8, [19] N. Glas and C. Pelachaud, User engagement and preferences in information-giving chat with virtual agents, in Workshop on Engagement in Social Intelligent Virtual Agents, 2015, p. Forthcoming. [20] I. Poggi, Mind, hands, face and body: a goal and belief view of multimodal communication. Weidler, [21] M. Lombard, L. Weinstein, and T. Ditton, Measuring telepresence: The validity of the temple presence inventory (tpi) in a gaming context, in ISPR, [22] J. Bates et al., The role of emotion in believable agents, Communications of the ACM, vol. 37, no. 7, pp , [23] S. Campano, N. Sabouret, E. de Sevin, and V. Corruble, An evaluation of the cor-e computational model for affective behaviors, in AAMAS. AAMAS, 2013, pp [24] S. S. Shapiro and M. B. Wilk, An analysis of variance test for normality (complete samples), Biometrika, pp , [25] H. B. Mann and D. R. Whitney, On a test of whether one of two random variables is stochastically larger than the other, The annals of mathematical statistics, pp , [26] W. H. Kruskal and W. A. Wallis, Use of ranks in one-criterion variance analysis, Journal of the American statistical Association, vol. 47, no. 260, pp , [27] J. C. Nunnally and I. Bernstein, The assessment of reliability, Psychometric theory, vol. 3, pp , [28] R. Zhao, A. Papangelis, and J. Cassell, Towards a dyadic computational model of rapport management for human-virtual agent interaction, in Intelligent Virtual Agents. Springer, 2014, pp [29] S. Ivaldi, S. M. Anzalone, W. Rousseau, O. Sigaud, and M. Chetouani, Robot initiative in a team learning task increases the rhythm of interaction but not the perceived engagement, Frontiers in Neurorobotics, vol. 8, p. 5, [30] K. Höök, P. Persson, and M. Sjölinder, Evaluating users experience of a character-enhanced information space, AI Communications, vol. 13, no. 3, pp , [31] S. Campano, C. Clavel, and C. Pelachaud, I like this painting too: When an eca shares appreciations to engage users, in AAMAS, /15/$ IEEE 967