MOTIVATION Thank you YouTube! Why do composers spend tremendous effort for the right combination of musical instruments? CHARACTERIZING THE EMOTION OF INDIVIDUAL PIANO AND OTHER MUSICAL INSTRUMENT SOUNDS HKUST PhD Thesis Defence CHAU Chuck Jee 27 July 2017 Inspiring from Emotional predisposition exists among different music instruments compared with each other Wu et al., 2013 2 AGENDA Music, Emotion, and Timbre A study of non-sustaining instrument sounds The piano The correlation of emotion and the sounds of Bowed string instruments Pitched percussion instruments MUSIC, EMOTION, AND TIMBRE The basic ideas 5 6
MUSIC AND EMOTION TIMBRE AND EMOTION Conveyed emotion the message of feelings in the sound Elicited emotion the feeling aroused when listening Aspects of music Harmony Liebetrau et al., 2012 Rhythm & tempo Plewa & Kostek, 2012 Lyrics Hu et al., 2009 Localization cues Ekman & Kajastila, 2009 Chords Lahdelma & Eerola, 2014 Types of music Environmental sounds Ellermeier et al., 2004 One-second sounds Bigand et al., 2009 Very short excerpts Krumhansl, 2010 Timbre the tone colour Difference between musical instruments Not a single measurement Early example of correlating timbre and emotion Scherer & Oshinsky, 1977 Rapid recognition of timbre Gjerdingen & Perrott, 2008 Krumhansl, 2010 Filipic et al., 2010 Acoustic features for mood estimation Baume, 2013 7 8 INSPIRING WORKS MEASURING THE TIMBRE SPACE Eerola et al., 2012 110 isolated, monophonic instrument sounds 3-dimensional model Valence Energy Arousal Tension Arousal Ratings, affective similarity comparison Results: Important timbral features Valence: Attack slope & envelope centroid Energy: Ratio of HF-LF energy, spectral flux Wu et al., 2013 Isolated tones of 8 sustaining instruments 8 emotional categories Happy, Sad, Heroic, Scary, Comic, Shy, Joyful, and Depressed Comparison of original tones, and further experiments on spectrally modified tones Results: Spectral centroid correlates strongly Even/odd harmonic ratio is a salient feature Measurements of the spectrum Spectral centroid and centroid deviation Spectral incoherence, irregularity, and variation Even/odd harmonic ratio Tristimulus 9 10
DENSITY OF SIGNIFICANT HARMONICS A novel measurement How tightly the important harmonics are packed together? Proven significant in our preliminary study Violin # sig. har. = 6 Bandwidth = 6 Density of Sig. Har. = 1 1 2 3 4 5 6 7 8 9 1011121314151617181920 Density = # significant harmonics / bandwidth Marimba # sig. har. = 2 Bandwidth = 4 Density of Sig. Har. = 0.5 1 2 3 4 5 6 7 8 9 1011121314151617181920 HOW DO DIFFERENT NON-SUSTAINING INSTRUMENTS EVOKE EMOTION? The research questions 11 12 PHASES OF A SOUND Amplitude Attack Marimba: Slow attack Fast decay Guitar Decay Marimba Guitar: Fast attack Slow decay No sustain here 0 0.25 0.5 0.75 1 Time (s) Release NON-SUSTAINING MUSICAL INSTRUMENTS A study to investigate emotional characteristics of non-sustaining instrument sounds Guitar, harp, plucked violin Marimba, vibraphone, xylophone Harpsichord, piano Equalized pitch, duration, loudness ICMC 2014, 2015 JAES 2015 13 14
0.25s tones HOW DO PITCH & DYNAMICS AFFECT EMOTION OF PIANO SOUNDS? 1.0s tones 15 The research questions EMOTION ON THE PIANO EXPERIMENT STIMULI A study on the emotional characteristics of piano sounds Short isolated sounds of duration 1 second 16 Avoid effects of melody, rhythm, etc. Effects of pitch Effects of dynamic levels All C pitches (C1 C8) Avoid effects of interval & harmony ICMC 2015 C1 JAES 2016 C2 C3 C4 C5 C6 C7 C8 Loud forte, medium mezzo, soft piano 24 sounds in total, obtained from RWC Music Database 17 18
RESULTS RESULTS forte mezzo piano 19 forte mezzo piano 20 RESULTS Many emotional categories show an arch-shape with pitch Peak often at around C6 Significant differences between dynamic levels Loud: strong for active emotions Soft: strong for inactive emotions Loud and low à Angry Soft and high à Shy Soft and low à Sad Loud and extreme register à Scary DOES THE DIFFERENCE APPLY TO OTHER MUSICAL INSTRUMENTS? The research questions 21 22
COMPARISON OF THE PIANO SOUNDS TO: EXPERIMENT STIMULI Bowed strings Pitched percussion Bowed strings Violin, Viola, Cello, Double Bass Pitched percussion Glockenspiel, Xylophone, Vibraphone, Marimba A family of homogenous instruments Equalizing temporal features ICMC 2016 JAES 2017 Further varieties of nonsustaining instruments Variation of playing techniques ICMC 2016 JAES to appear Vn C4 C5 C6 C7 Va C3 C4 C5 Vc C2 C3 C4 C5 Cb C1 C2 C3 Sounds from Prosonus library Forte and piano Equalized attack/decay Gl C6 C7 C8 Xy C5 C6 C7 C8 Vb C4 C5 C6 Ma C3 C4 C5 C6 C7 Sounds from RWC and Prosonus libraries With hard and soft mallets Equalized loudness 23 24 EMOTIONAL CATEGORIES LISTENING TEST PROCEDURES Category Happy Heroic Romantic Comic Calm Mysterious Shy Angry Scary Example of Italian musical terms giocoso grandioso appassionato capriccio tranquillo misterioso timido furioso terribile Why emotional categories instead of ratings (e.g. Valence Arousal)? For non-specialists to understand Judged more easily than numerical values Around 30 UST undergraduate students as subjects in each experiment Listening in a quiet room, using professional studio earphones Dictionary definitions of emotion categories provided Pairwise comparisons ALL combinations of sounds within each category Sad lacrimoso 25 26
PAIRWISE COMPARISON Fast decision-making from experiment subjects Simple decision than absolute rating Removing spammers: keystroke pattern Strings: 28C2 combinations 10 categories = 3780 trials Percussion: 30C2 combinations 10 categories = 4350 trials Further process: BTL scale values Probability p=[0,1] of a sound to be chosen among tested sounds 27 28 AN EXAMPLE OF BTL VALUES Piano RESULTS Bowed strings Pitched percussion 0.1 0.09 Romantic 0.08 0.07 0.07 0.06 Romantic 0.07 0.06 Romantic 0.06 0.05 0.05 0.05 0.04 0.04 0.04 0.03 0.03 0.03 0.02 0.01 0.02 0.02 0 C1 C2 C3 C4 C5 C6 C7 C8 forte mezzo piano 0.01 0 C1 C2 C3 C4 C5 C6 C7 0.01 0 C3 C4 C5 C6 C7 C8 29 30
ACTIVE EMOTIONS Piano Strings Percussion Happy Heroic Comic Angry Scary INACTIVE EMOTIONS Piano Strings Percussion Romantic Calm Mysterious Shy Sad Percussion Percussion Strings Strings Piano Piano 31 32 90% 80% 70% 60% INSTRUMENT-DEPENDENT CHARACTERISTICS * Cases of an instrument being more significant than others at the same conditions Double Bass Cello Viola Violin 40% 35% 30% 25% 20% 15% 10% 5% 0% Happy Heroic Romantic Comic Calm Mysterious Shy Angry Scary Sad Marimba 50% Xylophone 40% 30% Vibraphone 20% Glockenspiel 10% 0% Happy Heroic Romantic Comic Calm Mysterious Shy Angry Scary Sad 33 INSTRUMENT-DEPENDENT CHARACTERISTICS * Cases of an instrument being more significant than others at the same conditions 18 16 14 12 10 8 6 4 2 0 Cello, 1 Double Bass, 2 Viola, 1 Violin, 5 Violin, 6 Viola, 6 Cello, 4 C2 C3 C4 C5 Double Bass Cello Viola Violin 60 50 40 30 20 10 Vibraphone, 16 Xylophone, 7 Vibraphone, 4 Marimba, 5 0 Marimba, 1 Glockenspiel, 7 Vibraphone, 23 Xylophone, 13 Marimba, 9 Glockenspiel, 12 Xylophone, 9 Glockenspiel, 7 Marimba, 6 Xylophone, 5 C4 C5 C6 C7 C8 Marimba Xylophone Vibraphone Glockenspiel 34
CORRELATING EMOTIONS AND ACOUSTIC FEATURES Pearson correlation was used to study the relationship A linear relationship can be visualized easily BTL scale values vs. acoustic feature DO ACOUSTIC FEATURES AFFECT EMOTIONAL CHARACTERISTICS? The research questions The varying features are also considered: Pitch Log of fundamental frequency Dynamics Peak RMS amplitude Mallet hardness Attack/decay slope 35 36 THE PIANO SOUNDS THE BOWED STRINGS SOUNDS Only entries with p<0.05 is shown, +/ indicates sign of correlation Only entries with p<0.05 is shown, +/ indicates sign of correlation Happy Heroic Romantic Comic Calm Mysterious Shy Angry Scary Sad Happy Heroic Romantic Comic Calm Mysterious Shy Angry Scary Sad Log of Fundamental Frequency + + + + + Peak RMS Amplitude (db) + + + Attack time (ms) + + + Attack slope (per ms) Pitch and dynamics Log of Fundamental Frequency + + Peak RMS Amplitude (db) + + + Spectral Centroid + + + + Spectral Centroid Deviation + + + + Brightness Decay ratio + + + Decay slope (per ms) + + + + + Density of Significant Harmonics + + + + + Decay shape Spectral Incoherence + + Spectral Irregularity Density of Significant Harmonics + + Spectral Centroid + + + + Spectral Centroid Deviation + + + + Brightness Tristimulus T1 (harmonic 1) Tristimulus T2 (harmonics 2-4) + + + Spectral Irregularity + + + + + Tristimulus T3 (harmonics 5+) + + + Even/Odd Harmonic Ratio + + Tristimulus T1 (harmonic 1) + + + + + Tristimulus T2 (harmonics 2-4) + + + Brightness Tristimulus T3 (harmonics 5+) + + + + 37 38
THE PITCHED PERCUSSION SOUNDS REMOVING EFFECTS OF TESTING VARIABLES Only entries with p<0.05 is shown, +/ indicates sign of correlation Happy Heroic Romantic Comic Calm Mysterious Shy Angry Scary Sad Log of Fundamental Frequency + + Peak RMS Amplitude (db) + + + Attack time (ms) Attack slope (per ms) + + + Decay ratio + + + + + Decay slope (per ms) + + + + Density of Significant Harmonics + Spectral Centroid + + Spectral Centroid Deviation + + Spectral Incoherence + + Spectral Irregularity + + Tristimulus T1 (harmonic 1) Tristimulus T2 (harmonics 2-4) Tristimulus T3 (harmonics 5+) + 39 Attack and decay Partial Pearson correlation was used to eliminate variables Decrease in correlated features à more affected by pitch, and dynamics or mallet hardness Increase in correlated features à more affected by temporal or spectral features Prominent features Piano Bowed strings Pitched percussion Heroic and Shy Happy and Mysterious Comic Scary and Comic Shy Calm Density of Significant Harmonics Spectral Centroid Spectral Irregularity 40 GENERAL OBSERVATIONS A SUMMARY In a very broad sense High-Valence emotions intensify with increasing pitch Valence: how pleasant a sound is High-Arousal emotions intensify with increasing dynamics Arousal: how energetic a sound is With a lot of instrument-dependent variations, e.g. Heroic decreases with pitch on piano Shy is not affected by pitch on bowed strings Calm decreases with pitch on pitched percussion 41 42
CONCLUDING IDEAS We confirmed that there are statistically significant relationships between emotion and timbre for various musical instruments We developed a methodology to characterize Emotional characteristics of sounds with different variables Prominent timbral and acoustic features for emotional effects FURTHER POSSIBILITIES Only a subset of musical features and playing techniques can be explored in each experiment Possible ways of exploration: Playing techniques of instruments, e.g. piano pedal Repeated notes of the same instrument Blending or morphing of multiple instruments 43 44 HOW ARE ALL THESE USEFUL? The contributions AN EMOTION GUIDELINE FOR MUSICIANS Just like the loudness curve, the effects of sound perception can only be studied with careful experiments The emotion maps will benefit musicians in different fields: Composers and arrangers Live performers and conductors Audio or recording engineers Sound designers or electroacoustic musicians 45 46
AUTOMATED MUSIC EMOTION TRIGGER Algorithms to automatically adjust sound spectrum for emotional effect E.g., triggers on musical instruments similar to EQ adjustment Algorithms for recommending mix of musical sounds basing on timbre emotion Music generation algorithms to create an emotional background music with emotion messages E.g., film music and game music EXPLORATION OF THE TIMBRE SPACE Timbre space yet to be fully discovered How to measure it? How to represent it? Exploration of the timbre emotion space Decomposing timbre with the help of human perception 47 48 RELATED PUBLICATIONS C.-j. Chau, B. Wu, and A. Horner, Timbre features and music emotion in plucked string, mallet percussion, and keyboard tones. in Proc. 40th Int. Comp. Music Conf. (ICMC), 2014, pp. 982 989. C.-j. Chau and A. Horner, The emotional characteristics of mallet percussion instruments with different pitches and mallet hardness, in Proc. 42th Int. Comp. Music Conf. (ICMC), 2016, pp. 401 404. C.-j. Chau, B. Wu, and A. Horner, The effects of earlyrelease on emotion characteristics and timbre in nonsustaining musical instrument tones, in Proc. 41st Int. Comp. Music Conf. (ICMC), 2015, pp. 138 141. C.-j. Chau and A. Horner, The effects of pitch and dynamics on the emotional characteristics of piano sounds. in Proc. 41st Int. Comp. Music Conf. (ICMC), 2015, pp. 372 375. C.-j. Chau, B. Wu, and A. Horner, The emotional characteristics and timbre of nonsustaining instrument sounds, J. Audio Eng. Soc., vol. 63, no. 4, pp. 228 244, 2015. S. J. M. Gilburt, C.-j. Chau, and A. Horner, The effects of pitch and dynamics on the emotional characteristics of bowed string instruments, in Proc. 42th Int. Comp. Music Conf. (ICMC), 2016, pp. 405 410. C.-j. Chau, R. Mo, and A. Horner, The emotional characteristics of piano sounds with different pitch and dynamics, J. Audio Eng. Soc., vol. 64, no. 11, pp. 918 932, 2016. Y. Hong, C.-j. Chau, and A. Horner, An analysis of lowarousal piano music ratings to uncover what makes calm and sad music so difficult to distinguish in music emotion recognition, J. Audio Eng. Soc., vol. 65, no. 4, pp. 304 320, 2017. C.-j. Chau, S. J. M. Gilburt, R. Mo, and A. Horner, The emotional characteristics of bowed string instruments with different pitch and dynamics, J. Audio Eng. Soc., vol. 65, no. 7/8, 2017. QUESTIONS? ANSWER! End of the presentation. Thank you for LISTENING! 49