Meinard Müller Beethoven, Bach, and Billions of Bytes When Music meets Computer Science Meinard Müller International Laboratories Erlangen meinard.mueller@audiolabs-erlangen.de School of Mathematics University of Edinburgh 24.11.2017 Mathematics (Diplom/Master) Computer Science (PhD) Information Retrieval (Habilitation) Bonn University Combinatorics (Postdoc) Keio University, Japan Senior Researcher Max-Planck Institute, Saarland Professor: Semantic Processing Erlangen-Nürnberg University Group Members Stefan Balke Christian Dittmar Patricio López-Serrano Christof Weiß Book: Fundamentals of Music Processing Meinard Müller Fundamentals of Music Processing, Analysis, Algorithms, Applications 483 p., 249 illus., 30 illus. in color, hardcover ISBN: 978-3-319-21944-8 Springer, 2015 Accompanying website: www.music-processing.de Frank Zalkow Sebastian Rosenzweig International Laboratories Erlangen International Laboratories Erlangen
International Laboratories Erlangen Coding 3D Music Psychoacoustics Music Processing Music Information Retrieval (MIR) Music Information Retrieval (MIR) Sheet Music (Image) CD / MP3 () MusicXML (Text) Dance / Motion (Mocap) MIDI Music Singing / Voice () Music Film (Video) Music Literature (Text) Signal Processing Musicology Machine Learning Music User Interfaces Information Retrieval Library Sciences Piano Roll Representation Player Piano (1900)
Piano Roll Representation (MIDI) J.S. Bach, C-Major Fuge (Well Tempered Piano, BWV 846) Piano Roll Representation (MIDI) Query: Goal: Find all occurrences of the query Pitch Piano Roll Representation (MIDI) Query: Music Retrieval Query Database Goal: Find all occurrences of the query Hit Matches: -ID Version-ID Category-ID Bernstein (1962) Beethoven, Symphony No. 5 Beethoven, Symphony No. 5: Bernstein (1962) Karajan (1982) Gould (1992) Beethoven, Symphony No. 9 Beethoven, Symphony No. 3 Haydn Symphony No. 94 Music Synchronization: - Music Synchronization: - Beethoven s Fifth Beethoven s Fifth Orchester (Karajan) Piano (Scherbakov)
Music Synchronization: - Application: Interpretation Switcher Beethoven s Fifth Orchester (Karajan) Piano (Scherbakov) Music Synchronization: Image- Music Synchronization: Image- Image Image How to make the data comparable? How to make the data comparable? Image Processing: Optical Music Recognition Image Image
How to make the data comparable? Image Processing: Optical Music Recognition How to make the data comparable? Image Processing: Optical Music Recognition Image Image Processing: Fourier Analysis Processing: Fourier Analysis Application: Score Viewer Music Processing Coarse Level What do different versions have in common? Fine Level What are the characteristics of a specific version? Music Processing Music Processing Coarse Level Fine Level Coarse Level Fine Level What do different versions have in common? What are the characteristics of a specific version? What do different versions have in common? What are the characteristics of a specific version? What makes up a piece of music? What makes music come alive? What makes up a piece of music? What makes music come alive? Identify despite of differences Identify the differences
Music Processing Coarse Level What do different versions have in common? What makes up a piece of music? Identify despite of differences Example tasks: Matching Cover Song Identification Fine Level What are the characteristics of a specific version? What makes music come alive? Identify the differences Example tasks: Tempo Estimation Performance: Score (reference): Score (reference): Performance: Strategy: Compute score-audio synchronization and derive tempo curve Performance: Score (reference): Score (reference): Tempo Curve: Tempo Curves: Musical tempo (BPM) Musical tempo (BPM) Musical time (measures) Musical time (measures)
Score (reference): Score (reference): Tempo Curves: Musical tempo (BPM) Musical time (measures) Tempo Curves: Musical tempo (BPM) Musical time (measures)? What can be done if no reference is available? Music Processing Relative Given: Several versions Absolute Given: One version Tempo Curves: Musical tempo (BPM) Musical time (measures) Music Processing Music Processing Relative Absolute Relative Absolute Given: Several versions Given: One version Given: Several versions Given: One version Comparison of extracted parameters Direct interpretation of extracted parameters Comparison of extracted parameters Direct interpretation of extracted parameters Extraction errors have often no consequence on final result Extraction errors immediately become evident
Music Processing Relative Given: Several versions Comparison of extracted parameters Extraction errors have often no consequence on final result Example tasks: Music Synchronization Genre Classification Absolute Given: One version Direct interpretation of extracted parameters Extraction errors immediately become evident Example tasks: Music Transcription Tempo Estimation Basic task: Tapping the foot when listening to music Basic task: Tapping the foot when listening to music Basic task: Tapping the foot when listening to music Queen Another One Bites The Dust Queen Another One Bites The Dust Happy Birthday to you Happy Birthday to you Pulse level: Measure Pulse level: Tactus (beat)
Happy Birthday to you Chopin Mazurka Op. 68-3 Pulse level: Tatum (temporal atom) Pulse level: Quarter note Tempo:??? Chopin Mazurka Op. 68-3 Pulse level: Quarter note Tempo: 50-200 BPM Tempo curve Which temporal level? Local tempo deviations Sparse information (e.g., only note onsets available) Tempo (BPM) 200 50 Vague information (e.g., extracted note onsets corrupt) (beats) Why is Music Processing Challenging? Why is Music Processing Challenging? Chopin, Mazurka Op. 63 No. 3 Chopin, Mazurka Op. 63 No. 3 Waveform Amplitude
Why is Music Processing Challenging? Why is Music Processing Challenging? Chopin, Mazurka Op. 63 No. 3 Chopin, Mazurka Op. 63 No. 3 Waveform / Waveform / (Hz) Performance Tempo Dynamics Note deviations Sustain pedal Why is Music Processing Challenging? Source Separation Chopin, Mazurka Op. 63 No. 3 Decomposition of audio stream into different sound sources Waveform / Central task in digital signal processing Performance Tempo Dynamics Note deviations Sustain pedal Polyphony Main Melody Additional melody line Accompaniment Cocktail party effect Sources are often assumed to be statistically independent This is often not the case in music Strategy: Exploit additional information (e.g. musical score) to support the seperation process Score-Informed Source Separation Parametric Model Approach Rebuild spectrogram information Pitch Pitch Estimate Parameters Render Pitch (Hz) (Hz)
NMF (Nonnegative Matrix Factorization) NMF (Nonnegative Matrix Factorization) M K M K N 0 0 0 M K N M K Magnitude Templates Activations Templates: Pitch + Timbre Activations: Onset time + Duration How does it sound When does it sound NMF-Decomposition NMF-Decomposition Initialized template Initialized activations Initialized template Initialized activations Learnt templates Learnt activations Random initialization Random initialization No semantic meaning NMF-Decomposition NMF-Decomposition Initialized template Initialized activations Initialized template Initialized activations Template constraint for p=55 Activation constraints for p=55 Constrained initialization Constrained initialization
NMF-Decomposition Initialized template Initialized activations Score-Informed Decomposition Application: editing 1600 1600 Learnt templates Learnt activations 1200 800 1200 800 Constrained initialization NMF as refinement Org Model (Hertz) 400 580 523 500 0 0.5 1 6 7 8 9 (Hertz) 400 580 554 500 0 0.5 1 6 7 8 9 Informed Drum-Sound Decomposition Mosaicing Target signal: Beatles Let it be Source signal: Bees Remix: Mosaic signal: Let it Bee Literature: [Dittmar/Müller, IEEE/ACM-TASLP 2016] Demo: https://www.audiolabs-erlangen.de/resources/mir/2016-ieee-taslp-drumseparation Literature: [Driedger/Müller, ISMIR 2015] Demo: https://www.audiolabs-erlangen.de/resources/mir/2015-ismir-letitbee NMF-Inspired Mosaicing NMF-Inspired Mosaicing Non-negative matrix factorization (NMF) Non-negative matrix Components Activations. = fixed learned learned target source Activation matrix. = mosaic Proposed audio mosaicing approach Target s spectrogram Source s spectrogram Activations Mosaic s spectrogram. = fixed fixed learned
This image cannot currently be displayed. NMF-Inspired Mosaicing NMF-Inspired Mosaicing target source Iterative updates Activation matrix mosaic target source Activation matrix mosaic. =. = Preserve temporal context Core idea: support the development of sparse diagonal activation structures NMF-Inspired Mosaicing Mosaicing Target signal: Chic Good times Source signal: Whales target source Activation matrix mosaic. = Mosaic signal Mosaicing Motivic Similarity Target signal: Adele Rolling in the Deep Source signal: Race car Beethoven s Fifth (1st Mov.) Beethoven s Fifth (3rd Mov.) Mosaic signal Beethoven s Appassionata
Motivic Similarity Motivic Similarity B A C H Book: Fundamentals of Music Processing Book: Fundamentals of Music Processing Meinard Müller Fundamentals of Music Processing, Analysis, Algorithms, Applications 483 p., 249 illus., hardcover ISBN: 978-3-319-21944-8 Springer, 2015 Accompanying website: www.music-processing.de Meinard Müller Fundamentals of Music Processing, Analysis, Algorithms, Applications 483 p., 249 illus., hardcover ISBN: 978-3-319-21944-8 Springer, 2015 Accompanying website: www.music-processing.de