Affect and Personality in Interaction with Ubiquitous Systems Professor Ruth Aylett Vision Interactive Systems & Graphical Environments MACS, Heriot-Watt University www.macs.hw.ac.uk/~ruth Summary of programme Introduction and overview (today) Affective outputs speech, language, gesture, facial expressions, music, colour Affective/Personality models and action-selection approaches Affective inputs Applications Embodied Conversational Characters, Intelligent Virtual, Agents, human-robot interaction Evaluation approaches Today s topics Displaying emotion Describing emotion Music Colour Shape and form Thanks to Catherine Pelachaud! Emotions can be shown via Acoustic and visual behaviors: facial expression, voice, gesture, posture Behavior expressivity: voice and body movement quality Music Colour Reasons to display emotional state: Create affective awareness Create social relationship Engage user in communication But how do we know what to output? Some systematic description of emotion? 1
Defining types of affective states Scherer et al.,univ. Geneva Types of Affect Design Features Emotions: angry, sad, joyful, fearful, ashamed, proud, elated, desperate Moods: cheerful, gloomy, irritable, listless, depressed, buoyant Interpersonal stances: distant, cold, warm, supportive, contemptuous Intensity Duration Synchronization Event focus Appraisal elicitation Rapidity of change Behavior impact Circumplex Model of Affect Russell 1980 two components (1) pleasuredispleasure VALANCE (2) arousal-sleep AROUSAL Russell s system Preferences/Attitudes: liking, loving, hating, valuing, desiring Affect dispositions: nervous, anxious, reckless, morose, hostile Sherer s descriptive framework EXCITED Hi Power/Control Positive adventurous triumphant AROUSED ASTONISHElusting D ambitious conceited courageous feeling superior convinced selfconfident DELIGHTEenthusiasti elated c lighthearted D determined amused excited HAPPY joyous passionate Conducive interested expectant bellicose hostile TENSE ALARMED hateful envious ANGRY AFRAID jealous impatient enraged defiant ANNOYED contemptuo angry Angryus DISTRESS disgusted indignant ED loathing FRUSTRATED bored suspicious distrustful discontente bitter d insulted startled feel well PLEASED impressed disappointe amourous astonished MISERABL d apathetic GLAD dissatisfied E confident taken aback content hopeful worried relaxed solemn attentive SERENE Active longing uncomforta SAD feel guilt DEPRESS ble despondent GLOOMY languid ashamed ED desperate Sad CONTENT AT EASEfriendly pensive SATISFIED contemplati polite serious embarrass RELAXED CALM ve ed melancholi wavering lonely hesitant c peaceful BORED anxious conscientio sad dejected insecure us empathic DROOPY reverent doubtful SLEEPY TIRED Passive Obstructive Negative Scherer et al. Univ. Geneva Lo Power/Control An empirical subset suitable for describing emotions in human-machine interaction Preliminary list of 55 terms, from HUMAINE summer school 2004, Belfast: stress, annoyance, boredom, panic, impatience, disapproval, hot anger, anxiety, disappointment, fear, satisfaction, sadness, surprise, shock, amusement, worry, excitement, pleasure, cold anger, interest, effervescent happiness, nervousness, approval, embarrassment, distraction, disagreeableness, disgust, despair, indifference, neutrality, hurt, friendliness, weariness, relief, confidence, contentment, shame, contempt, affection, sympathy, relaxation, mockery, pride, resentment, calm, guilt, jealousy, determination, serenity, coldness, cruelty, hopeful, wariness, greed, admiration 2
Affective Music Bresin, KTH Sweden Simulation of emotions in music performance Mapping between expressive acoustic cues and emotions Visualization of musical expression Colours Facial expressions Expressive cues Positive Valence From Juslin (2001) TENDERNESS HAPPINESS fast mean tempo (Ga95) slow mean tempo (Ga96) small tempo variability (Ju99) slow tone attacks (Ga96) staccato articulation (Ju99) low sound level (Ga96) large articulation variability (Ju99) small sound level variability high sound level (Ju00) (Ga96) little sound level variability (Ju99) legato articulation (Ga96) bright timbre (Ga96) soft timbre (Ga96) fast tone attacks (Ko76) large timing variations (Ga96) small timing variations (Ju/La00) accents on stable notes (Li99) sharp duration contrasts (Ga96) soft duration contrasts (Ga96) rising micro-intonation (Ra96) final ritardando (Ga96) Low Activity High Activity ANGER high sound level (Ju00) sharp timbre (Ju00) SADNESS spectral noise (Ga96) slow mean tempo (Ga95) FEAR fast mean tempo (Ju97a) legato articulation (Ju97a) staccato articulation (Ju97a) small tempo variability (Ju99) small articulation variability (Ju99) very low sound level (Ju00) staccato articulation (Ju99) low sound level (Ju00) large sound level variability abrupt tone attacks (Ko76) dull timbre (Ju00) (Ju99) sharp duration contrasts (Ga96) large timing variations (Ga96) fast mean tempo (Ju99) accents on unstable notes (Li99) soft duration contrasts (Ga96) large tempo variability (Ju99) large vibrato extent (Oh96b) slow tone attacks (Ko76) large timing variations (Ga96) no ritardando (Ga96) flat micro-intonation (Ba97) soft spectrum (Ju00) slow vibrato (Ko00) sharp micro-intonation (Oh96b) final ritardando (Ga96) fast, shallow, irregular vibrato (Ko00) Negative Valence Lens model: quantifies the expressive communication between performer and listener The Performer Encoding The Performance Decoding The Listener intention expressive cues judgment Anger.26.47.63 -.26 Accuracy Tempo Loudness Timbre Articula. others Cue Utilization Cue Utilization rperformer.87.22.55.61 -.39 rlistener Anger Expressive Cue Analysis Example: SADNESS Synthesis (Director Musices) Tempo Slow Tone IOI is lengthened by 30% Sound level Moderate or low Sound level is decreased by 6 db Articulation Legato Tone duration = Tone IOI Time deviations Moderate Duration Contrast Rule (k = -2) Final ritardando Yes Phrase Arch Rule applied on phrase level (k = 1.5) Phrase Arch Rule applied on sub - phrase level (k = 1.5) Obtained from the Phrase Arch Rule Matching.92 3
Better Polyphonic Ringtones MOODIES Bresin, KTH Coldplay Talk La Linea Hayfa Colour, Movement, Shape Original Classy Romantic Aggressive Original Classy Romantic Aggressive Original Classy Romantic Aggressive www.notesenses.com Color and Emotion Bresin et al, KTH Perceptual study: Link musical performances to colours Performances with different emotional intentions Set of colour nuances in hue, brightness, saturation Result of perceptual study: HUE Happiness Yellow Fear Blue Sadness Violet & Blue Anger Red Love Blue & Violet BRIGHTNESS Observed tendency: Minor tonality Low brightness (Dark colours) Major tonality High brightness (Light colours) Visualization of Musical Expression Tool for real-time visual feedback to expressive performance Mapping between acoustic cues and emotions ExpressiveBall: Mapping of emotions and colors GretaMusic: Mapping of emotions and facial expressions music emotion facial expression music volume spatial and power music tempo temporal and overall activation music articulation fluidity From Bresin 4
The ExpressiBall GretaMusic Bresin, Juslin, KTH Bresin, KTH Mancini, Pelachaud U Paris8 Slow Fast Legato Sad Slow attack Low energy Soft Soft Loud Staccato Angry Fast attack High energy Color Emotion Shape Articulation Loud X Tempo Y Sound level Z Attack velocity & Spectrum energy Slow Fast From Bresin Mutual Interaction Interactive virtual dancer: dance together with the user to the beat of the music adapt its performance to whatever the human user is doing - beat detection to align dance with music tempo - agent s movement chosen from database of capture movements Affective Diary Höök et al, SICS Diary: express inner thoughts and record experiences of past events Affective diary: capture emotional experience over time via mobile phone replay of the experience reflect on the experience From Höök 5
emoto - Expressing emotions in a digital world Affective Diary Höök et al, SICS [pointing at the first slightly red character] And then I become like this, here I am kind of, I am kind of both happy and sad in some way and something like that. I like him and then it is so sad that we see each other so little. And then I cannot really show it. Sundström, Ståhl, Höök, SICS-KTH From Höök emoto - Expressing emotions in a digital world Sundström, Ståhl, Höök, SICS-KTH emoto: mobile messaging service for sending and receiving affective messages Use affective gestures of users to convey the emotional content of their messages emoto Example Sundström, Ståhl, Höök, SICS-KTH Input: movement detection through pen Output: colours, shapes and animations on mobile Bored Excited Happiness 6
Help children to build the virtual and real world of Interactive Shadows Create a learning environment where children will be able to build logical narratives on-the-fly I-Shadows Paiva et al, Gaips AINI (Anticipatory Believability) Martinho, Paiva Gaips Agent with autonomous anticipatory mechanism Study of the relation between anticipation and emotion Prediction of the next sensor value of agent interpretation of the mismatch between sensation and anticipation to direct both the focus of attention and the expression of emotions. QUESTIONS? 7