Using machine learning to decode the emotions expressed in music

Size: px

Start display at page:

Download "Using machine learning to decode the emotions expressed in music"

Rosalyn Harmon
5 years ago
Views:

Using machine learning to decode the emotions expressed in music Jens Madsen Postdoc in sound project Section for Cognitive Systems (CogSys) Department of Applied Mathematics and

1 Using machine learning to decode the emotions expressed in music Jens Madsen Postdoc in sound project Section for Cognitive Systems (CogSys) Department of Applied Mathematics and Computer Science (DTU Compute) Technical University of Denmark Website: Cognitive Systems, Technical University of Denmark

2 Ubiquity of music 2 Cognitive Systems, Technical University of Denmark

Music is with us everywhere Spotify (2006) 36+ million songs, (100+ million users) Apple music (2015) 37 million songs (17 million users) Deezer (2007), 40 million songs (16+ million users) TiDAL

3 Music is with us everywhere Spotify (2006) 36+ million songs, (100+ million users) Apple music (2015) 37 million songs (17 million users) Deezer (2007), 40 million songs (16+ million users) TiDAL (2010), 30+ mio tracks (4+ million users) YouTube (2005), 30+ mio tracks (1+ billion users) Globally, digital revenues of music account for 45% of total sales in 2016 Denmark, digital revenues of music account for 74.3% in Cognitive Systems, Technical University of Denmark

4 Streaming services Major selling points have been Catalogue Quality User Interface Functionality 4 Cognitive Systems, Technical University of Denmark

5 Design of a music system User Interface System 30+ mio tracks 5 Cognitive Systems, Technical University of Denmark

6 Music system interfacing Exploration Search (artist, title, genre, emotion, ) Interfaces (visual, auditory, tactile representations, ) Recommendation This is what other people are listening to Demographic, Collaborative, Content-based, Context-based filtering Hybrid Similar artist, tracks, Editorial Playlists, events, etc. Celma, Oscar. Music recommendation and discovery: The long tail, long fail, and long play in the digital music space. Springer Science & Business Media, Cognitive Systems, Technical University of Denmark

7 Design of a music system Does recommendation give us what we actually want? Do people want to explore large music collections? 7 Cognitive Systems, Technical University of Denmark

8 Why do people listen to music? 8 Cognitive Systems, Technical University of Denmark

9 Scientific answer to why people listen to music 1. Regulate their emotional state 2. Self-awareness and finding identity 3. Social bonding/relatedness Thomas Schäfer, Peter Sedlmeier, Christine Städler and David Huron The psychological functions of music listening in frontiers in Psychology 13 august 2013 Juslin, Patrik N., and Petri Laukka. "Expression, perception, and induction of musical emotions: A review and a questionnaire study of everyday listening." Journal of New Music Research 33.3 (2004): Rentfrow, Peter J;,The role of music in everyday life: Current directions in the social psychology of music,social and Personality Psychology Compass,6,5, ,2012,Blackwell Publishing Ltd 9 Cognitive Systems, Technical University of Denmark

Mechanisms for emotion regulation Audience Performance / expression Induced emotion Episodic memory Perceived emotion Emotional contagion

10 Mechanisms for emotion regulation Audience Performance / expression Induced emotion Episodic memory Perceived emotion Emotional contagion Visual imagery Musical expectancy Expressed emotion Evaluative conditioning Brain stem reflexes 10 Cognitive Systems, Technical University of Denmark

11 Building a music system regulating emotions Audio signal Annotations (tags, genre, etc.) User metadata (age, country, etc.) Lyrics Primary mechanisms of induced emotions Visual imagery Brain stem reflexes Episodic memory Emotional contagion Musical expectancy Evaluative conditioning 11 Cognitive Systems, Technical University of Denmark

12 Using emotional contagion for emotion regulation (Mis)matching the emotions expressed in music with the emotional state of the listener Performance Expressed emotion Audience Emotional state 12 Cognitive Systems, Technical University of Denmark

13 Technical challenge We need to know what emotions are expressed in 30+ mio tracks! Too much data to annotate! 13 Cognitive Systems, Technical University of Denmark

14 Overall goal of research Create mathematical models that can predict the emotions expressed in music 14 Cognitive Systems, Technical University of Denmark

15 Audio representation Modeling framework Elicitation of emotions Predictive model of emotions expressed in music User Internal representation of emotion Decision Elicitation of emotions User interface Model Predictions Modelling framework Feature representation Audio representation Feature extraction Music database 15 Cognitive Systems, Technical University of Denmark

16 Audio representation Modeling framework Elicitation of emotions Audio representation User Internal representation of emotion Decision User interface Model Predictions Feature representation Feature extraction Music database 30+ mio tracks 16 Cognitive Systems, Technical University of Denmark

17 Audio representation Feature extraction Beat Rhythm Pitch Chords Feature representation Melody Timbre Harmony Feature extraction Music database 17 Cognitive Systems, Technical University of Denmark

18 Audio representation Features describing music Dimensions No. time windows Pitch/Chroma (100 ms) Mel-frequency cepstral coefficients (23 ms) Loudness (23 ms) Beat/tempo (10s) 1 15 Feature representation Feature extraction Music database 18 Cognitive Systems, Technical University of Denmark

19 Audio representation Modeling framework Elicitation of emotions Predictive model of emotions expressed in music User Internal representation Core affect of emotion Decision Elicitation of emotions User interface Model Predictions Modelling framework Feature representation Feature extraction Audio representation Music database 19 Cognitive Systems, Technical University of Denmark

20 Elicitation of emotions Elicitation of emotions expressed in music User Internal representation of emotion Decision User interface Internal representation of emotions Define the question to the user Define users response format (decision) Scale? Rank? 20 Cognitive Systems, Technical University of Denmark

21 Elicitation of emotions Elicitation of emotions expressed in music User Internal representation of emotion Decision User interface 21 Cognitive Systems, Technical University of Denmark

22 Internal representation of emotion Valence and Arousal / Core affect Active Arousal Negative Valence Positive Passive 22 Cognitive Systems, Technical University of Denmark

23 Valence and arousal / Core affect Active Arousal Negative Valence Positive Passive 23 Cognitive Systems, Technical University of Denmark

24 Elicitation of emotions Predictive model of emotions expressed in music Arousal User Valence Decision User interface 24 Cognitive Systems, Technical University of Denmark

25 Response format for quantifying valence and arousal model Absolute Relative Excerpt A Excerpt A Excerpt B Excerpt K 25 Cognitive Systems, Technical University of Denmark

Audio representation Modeling framework Elicitation of emotions Predictive model of emotions expressed in music Arousal User Valence Decision Elicitation of emotions User

26 Audio representation Modeling framework Elicitation of emotions Predictive model of emotions expressed in music Arousal User Valence Decision Elicitation of emotions User interface Model Predictions Modelling framework Feature representation Feature extraction Audio representation Music database 26 Cognitive Systems, Technical University of Denmark

Audio representation Modelling framework User Observations u v Arousal Valence Decision X = p(x θ n ) n = 1: N X R DxT Y = y m m = 1: M y 1,1 User Interface Experimental Design Likelihood Decision

27 Audio representation Modelling framework User Observations u v Arousal Valence Decision X = p(x θ n ) n = 1: N X R DxT Y = y m m = 1: M y 1,1 User Interface Experimental Design Likelihood Decision Making Latent Function Model of Core affect Model Discrete choice model θ κ ~halfstudent t(ν, η) k p(x θ), p(x θ ) = න p(x θ) ρ p(x θ ) ρ dx Feature representation f X, θ κ ~GP 0, k p(x θ), θκ Feature Extraction Music Database π m f, σ = Φ y m f um f vm σ 2 y m π m ~Bernoulli 1 π m m = 1: M m = 1: M 27 Cognitive Systems, Technical University of Denmark

28 Visualizing the latent function (A-V space) No. Song name T and p combo 2 A-Ha - Living a boys adventure 3 Abba That s me 4 ACDC - What do you do for money honey 5 Aaliyah - The one I gave my heart to 6 Aerosmith - Mother popcorn 7 Alanis Morissette - These r the thoughts 8 Alice Cooper I m your gun 9 Alice in Chains - Killer is me 10 Aretha Franklin - A change 11 Moby Everloving 12 Rammstein - Feuerfrei 13 Santana - Maria caracoles 14 Stevie Wonder - Another star 15 Tool - Hooker with a pen.. 16 Toto - We made it 17 Tricky - Your name 18 U2 - Babyface 19 UB40 - Version girl 20 ZZ top - Hot blue and righteous J. Madsen, J. B. Nielsen, B. S. Jensen, and J. Larsen, Modeling expressed emotions in music using pairwise comparisons. in 9th International Symposium on Computer Music Modeling and Retrieval (CMMR) Music and Emotions, Cognitive Systems, Technical University of Denmark

29 Conclusion Defined a psychological based approach of designing of music system Investigated elicitation methods of emotions expressed in music Designed a predictive model of emotions expressed emotion 29 Cognitive Systems, Technical University of Denmark

30 Thank you! Website: Cognitive Systems, Technical University of Denmark

The relationship between properties of music and elicited emotions

The relationship between properties of music and elicited emotions Agnieszka Mensfelt Institute of Computing Science Poznan University of Technology, Poland December 5, 2017 1 / 19 Outline 1 Music and