Data Driven Music Understanding

Transcription

1 Data Driven Music Understanding Dan Ellis Laboratory for Recognition and Organization of Speech and Audio Dept. Electrical Engineering, Columbia University, NY USA 1. Motivation: What is Music? 2. Eigenrhythms 3. Melodic-Harmonic Fragments 4. Example Applications Data-driven music understanding - Ellis p. 1 /24

2 LabROSA Overview Information Extraction Music Machine Learning Recognition Separation Retrieval Speech Environment Signal Processing Data-driven music understanding - Ellis p. 2 /24

3 1. Motivation: What is music? What does music evoke in a listener s mind?? Which are the things that we call music? Data-driven music understanding - Ellis p. 3 /24

4 Oodles of Music What can you do with a million tracks? Data-driven music understanding - Ellis p. 4 /24

5 Re-use in Music 60 Scatter of PCA(3:6) of 12x16 beatchroma What are the most popular chord progressions in pop music? Data-driven music understanding - Ellis p. 5 /24

6 Potential Applications Compression Classification Manipulation Data-driven music understanding - Ellis p. 6 /24

7 2. Eigenrhythms: Drum Track Structure To first order, all pop music has the same beat: Ellis & Arroyo ISMIR 04 Can we learn this from examples? Data-driven music understanding - Ellis p. 7 /24

8 Basis Sets Combine a few basic patterns to make a larger dataset data weights X = W H patterns 1 = 0-1 Data-driven music understanding - Ellis p. 8/24

9 Drum Pattern Data Tempo normalization + downbeat alignment Data-driven music understanding - Ellis p. 9 /24

10 NMF Eigenrhythms Posirhythm 1 Posirhythm 2 HH HH SN SN BD BD Posirhythm 3 Posirhythm 4 HH HH SN SN BD BD Posirhythm 5 Posirhythm 6 HH HH SN SN BD BD samples (@ beats (@ 120 Nonnegative: only add beat-weight Data-driven music understanding - Ellis p. 10/24

11 Eigenrhythm BeatBox Resynthesize rhythms from eigen-space Data-driven music understanding - Ellis p. 11 /24

12 3. Melodic-Harmonic Fragments How similar are two pieces? Can we find all the pop-music clichés? Data-driven music understanding - Ellis p. 12/24

13 MFCC Features Used in speech recognition Let It Be (LIB-1) - log-freq specgram freq / Hz MFCCs Coefficient Noise excited MFCC resynthesis (LIB-2) freq / Hz time / sec Data-driven music understanding - Ellis p. 13/24

14 Chroma Features To capture musical content Let It Be - log-freq specgram (LIB-1) freq / Hz chroma bin B A G E D C Chroma features Shepard tone resynthesis of chroma (LIB-3) freq / Hz MFCC-filtered shepard tones (LIB-4) freq / Hz time / sec Data-driven music understanding - Ellis p. 14/24

15 Beat-Synchronous Chroma Compact representation of harmonies Let It Be - log-freq specgram (LIB-1) freq / Hz Onset envelope + beat times chroma bin B A G E D C Beat-synchronous chroma Beat-synchronous chroma + Shepard resynthesis (LIB-6) freq / Hz time / sec Data-driven music understanding - Ellis p. 15/24

16 Finding Cover Songs Ellis & Poliner 07 freq / khz Little similarity in surface audio... Let It Be - The Beatles Let It Be / Beatles / verse freq / khz Let It Be - Nick Cave Let It Be / Nick Cave / verse time / sec time / se chroma.. but appears in beat-chroma G F D C Beat-sync chroma features chroma G F D C Beat-sync chroma features A beats A beat Data-driven music understanding - Ellis p. 16/24

17 Finding Common Fragments Cluster beat-synchronous chroma patches chroma bins G F D C A G F D C A G F D C A G F D C A # instances # instances # instances # instances # instances # instances # instances a20-top10x5-cp4-4p0 # instances time / beats Data-driven music understanding - Ellis p. 17/24

18 Clustered Fragments chroma bins G F D C A depeche mode 13-Ice Machine s roxette 03-Fireworks s G F D C roxette 04-Waiting For The Rain s tori amos 11-Playboy Mommy s A time / beats... for a dictionary of common themes? Data-driven music understanding - Ellis p. 18/24

19 4. Example Applications: Music Discovery Berenzweig & Ellis 03 Connecting listeners to musicians Data-driven music understanding - Ellis p. 19/24

20 Playlist Generation Mandel, Poliner, Ellis 06 Incremental learning of listeners preferences Data-driven music understanding - Ellis p. 20/24

21 MajorMiner: Music Tagging Describe music using words Mandel & Ellis 07, 08 Data-driven music understanding - Ellis p. 21/24

22 Music Transcription Training data and features: MIDI, multi-track recordings, playback piano, & resampled audio (less than 28 mins of train audio). Normalized magnitude STFT. Classification: N-binary SVMs (one for ea. note). Independent frame-level classification on 10 ms grid. Dist. to class bndy as posterior. Temporal Smoothing: Two state (on/off) independent HMM for ea. note. Parameters learned from training data. Find Viterbi sequence for ea. note. feature representation classification posteriors hmm smoothing Poliner & Ellis 05, 06, 07 feature vector Data-driven music understanding - Ellis p. 22/24

23 MEAPsoft Music Engineering Art Projects collaboration between EE and Computer Music Center with Douglas Repetto, Ron Weiss, and the rest of the MEAP team Data-driven music understanding - Ellis p. 23/24

24 Conclusions Low-level features Classification and Similarity browsing discovery production Music audio Melody and notes Key and chords Tempo and beat Music Structure Discovery modeling generation curiosity Lots of data + noisy transcription + weak clustering musical insights? Data-driven music understanding - Ellis p. 24/24