Data Driven Music Understanding Dan Ellis Laboratory for Recognition and Organization of Speech and Audio Dept. Electrical Engineering, Columbia University, NY USA http://labrosa.ee.columbia.edu/ 1. Motivation: What is Music? 2. Eigenrhythms 3. Melodic-Harmonic Fragments 4. Example Applications Data-driven music understanding - Ellis 2008-10-03 p. 1 /24
LabROSA Overview Information Extraction Music Machine Learning Recognition Separation Retrieval Speech Environment Signal Processing Data-driven music understanding - Ellis 2008-10-03 p. 2 /24
1. Motivation: What is music? What does music evoke in a listener s mind?? Which are the things that we call music? Data-driven music understanding - Ellis 2008-10-03 p. 3 /24
Oodles of Music What can you do with a million tracks? Data-driven music understanding - Ellis 2008-10-03 p. 4 /24
Re-use in Music 60 Scatter of PCA(3:6) of 12x16 beatchroma What are the most popular chord progressions in pop music? 50 40 30 20 10 60 50 40 30 20 10 10 20 30 40 50 60 10 20 30 40 50 60 Data-driven music understanding - Ellis 2008-10-03 p. 5 /24
Potential Applications Compression Classification Manipulation Data-driven music understanding - Ellis 2008-10-03 p. 6 /24
2. Eigenrhythms: Drum Track Structure To first order, all pop music has the same beat: Ellis & Arroyo ISMIR 04 Can we learn this from examples? Data-driven music understanding - Ellis 2008-10-03 p. 7 /24
Basis Sets Combine a few basic patterns to make a larger dataset data weights X = W H patterns 1 = 0-1 Data-driven music understanding - Ellis 2008-10-03 p. 8/24
Drum Pattern Data Tempo normalization + downbeat alignment Data-driven music understanding - Ellis 2008-10-03 p. 9 /24
NMF Eigenrhythms Posirhythm 1 Posirhythm 2 HH HH SN SN BD BD Posirhythm 3 Posirhythm 4 HH HH SN SN BD BD Posirhythm 5 Posirhythm 6 HH HH SN SN BD BD 0 50 100 150 200 250 300 350 400 0 50 100 150 200 250 300 350 samples (@ 2 1 2 3 4 1 2 3 4 beats (@ 120 Nonnegative: only add beat-weight 0.1 0-0.1 Data-driven music understanding - Ellis 2008-10-03 p. 10/24
Eigenrhythm BeatBox Resynthesize rhythms from eigen-space Data-driven music understanding - Ellis 2008-10-03 p. 11 /24
3. Melodic-Harmonic Fragments How similar are two pieces? Can we find all the pop-music clichés? Data-driven music understanding - Ellis 2008-10-03 p. 12/24
MFCC Features Used in speech recognition Let It Be (LIB-1) - log-freq specgram freq / Hz 5915 1396 329 MFCCs Coefficient 12 10 8 6 4 2 Noise excited MFCC resynthesis (LIB-2) freq / Hz 5915 1396 329 0 5 10 15 20 25 time / sec Data-driven music understanding - Ellis 2008-10-03 p. 13/24
Chroma Features To capture musical content Let It Be - log-freq specgram (LIB-1) freq / Hz 6000 1400 300 chroma bin B A G E D C Chroma features Shepard tone resynthesis of chroma (LIB-3) freq / Hz 6000 1400 300 MFCC-filtered shepard tones (LIB-4) freq / Hz 6000 1400 300 0 5 10 15 20 25 time / sec Data-driven music understanding - Ellis 2008-10-03 p. 14/24
Beat-Synchronous Chroma Compact representation of harmonies Let It Be - log-freq specgram (LIB-1) freq / Hz 6000 1400 300 Onset envelope + beat times chroma bin B A G E D C Beat-synchronous chroma Beat-synchronous chroma + Shepard resynthesis (LIB-6) freq / Hz 6000 1400 300 0 5 10 15 20 25 time / sec Data-driven music understanding - Ellis 2008-10-03 p. 15/24
Finding Cover Songs Ellis & Poliner 07 freq / khz Little similarity in surface audio... Let It Be - The Beatles Let It Be / Beatles / verse 1 4 3 2 freq / khz Let It Be - Nick Cave Let It Be / Nick Cave / verse 1 4 3 2 1 1 0 2 4 6 8 10 time / sec 0 2 4 6 8 10 time / se chroma.. but appears in beat-chroma G F D C Beat-sync chroma features chroma G F D C Beat-sync chroma features A 5 10 15 20 25 beats A 5 10 15 20 25 beat Data-driven music understanding - Ellis 2008-10-03 p. 16/24
Finding Common Fragments Cluster beat-synchronous chroma patches chroma bins G F D C A G F D C A G F D C A G F D C A #32273-13 instances #51917-13 instances #65512-10 instances #9667-9 instances #10929-7 instances #61202-6 instances #55881-5 instances 5 10 15 20 a20-top10x5-cp4-4p0 #68445-5 instances 5 10 15 time / beats Data-driven music understanding - Ellis 2008-10-03 p. 17/24
Clustered Fragments chroma bins G F D C A depeche mode 13-Ice Machine 199.5-204.6s roxette 03-Fireworks 107.9-114.8s G F D C roxette 04-Waiting For The Rain 80.1-93.5s tori amos 11-Playboy Mommy 157.8-171.3s A 5 10 15 20 5 10 15 time / beats... for a dictionary of common themes? Data-driven music understanding - Ellis 2008-10-03 p. 18/24
4. Example Applications: Music Discovery Berenzweig & Ellis 03 Connecting listeners to musicians Data-driven music understanding - Ellis 2008-10-03 p. 19/24
Playlist Generation Mandel, Poliner, Ellis 06 Incremental learning of listeners preferences Data-driven music understanding - Ellis 2008-10-03 p. 20/24
MajorMiner: Music Tagging Describe music using words Mandel & Ellis 07, 08 Data-driven music understanding - Ellis 2008-10-03 p. 21/24
Music Transcription Training data and features: MIDI, multi-track recordings, playback piano, & resampled audio (less than 28 mins of train audio). Normalized magnitude STFT. Classification: N-binary SVMs (one for ea. note). Independent frame-level classification on 10 ms grid. Dist. to class bndy as posterior. Temporal Smoothing: Two state (on/off) independent HMM for ea. note. Parameters learned from training data. Find Viterbi sequence for ea. note. feature representation classification posteriors hmm smoothing Poliner & Ellis 05, 06, 07 feature vector Data-driven music understanding - Ellis 2008-10-03 p. 22/24
MEAPsoft Music Engineering Art Projects collaboration between EE and Computer Music Center with Douglas Repetto, Ron Weiss, and the rest of the MEAP team Data-driven music understanding - Ellis 2008-10-03 p. 23/24
Conclusions Low-level features Classification and Similarity browsing discovery production Music audio Melody and notes Key and chords Tempo and beat Music Structure Discovery modeling generation curiosity Lots of data + noisy transcription + weak clustering musical insights? Data-driven music understanding - Ellis 2008-10-03 p. 24/24