Beethoven, Bach und Billionen Bytes

Transcription

1 Meinard Müller Beethoven, Bach und Billionen Bytes Automatisierte Analyse von Musik und Klängen Meinard Müller Lehrerfortbildung in Informatik Dagstuhl, Dezember PhD, Bonn University 2002/2003 Postdoc, Keio University, Japan 2007 Habilitation, Bonn University Information Retrieval for Music and Motion Senior Researcher Max-Planck Institut für Informatik, Saarland 2012: Professor Semantic Processing Universität Erlangen-Nürnberg International Laboratories Erlangen International Laboratories Erlangen Fraunhofer IIS & Multimedia Realtime Systems Technical Faculty Electrical Engineering Dept. Uni / FAU International Laboratories Erlangen Labs-IIS Staff Labs-FAU 6 Professors International Laboratories Erlangen International Laboratories Erlangen Fraunhofer IIS & Multimedia Realtime Systems Technical Faculty Electrical Engineering Dept. Uni / FAU International Laboratories Erlangen Labs-IIS Staff Labs-FAU 6 Professors

2 International Laboratories Erlangen International Laboratories Erlangen Coding 3D Psychoacoustics Sheet Music (Image) CD / MP3 () MusicXML (Text) Music Dance / Motion (Mocap) Music MIDI Singing / Voice () Music Film (Video) Music Literature (Text) Research Goals Piano Roll Representation Music Information Retrieval (MIR) ISMIR Analysis of music signals (harmonic, melodic, rhythmic, motivic aspects) Design of musically relevant audio features Tools for multimodal search and interaction

3 Player Piano (1900) Piano Roll Representation (MIDI) J.S. Bach, C-Major Fuge (Well Tempered Piano, BWV 846) Time Pitch Piano Roll Representation (MIDI) Query: Goal: Find all occurrences of the query Piano Roll Representation (MIDI) Query: Goal: Find all occurrences of the query Matches: Data Data (Memory Requirements) Various interpretations Beethoven s Fifth Bernstein Karajan Scherbakov (piano) 1 Bit = 1: on 0: off 1 Byte = 8 Bits 1 Kilobyte (KB) = 1 Thousand Bytes 1 Megabyte (MB) = 1 Million Bytes 1 Gigabyte (GB) = 1 Billion Bytes 1 Terabyte (TB) = 1000 Billion Bytes MIDI (piano)

4 Data (Memory Requirements) Music Synchronization: MIDI files < 350 MB Beethoven s Fifth One audio CD 650 MB Two audio CDs > 1 Billion Bytes 1000 audio CDs Billions of Bytes Music Synchronization: - Music Synchronization: - Beethoven s Fifth Beethoven s Fifth Orchester (Karajan) Orchester (Karajan) Piano (Scherbakov) Piano (Scherbakov) Application: Interpretation Switcher Music Synchronization: Image- Image

5 Music Synchronization: Image- How to make the data comparable? Image Image How to make the data comparable? Image Processing: Optical Music Recognition How to make the data comparable? Image Processing: Optical Music Recognition Image Image Processing: Fourier Analyse How to make the data comparable? Application: Score Viewer Image Processing: Optical Music Recognition Image Processing: Fourier Analyse

6 Coarse Level Fine Level Coarse Level Fine Level What do different versions have in common? What are the characteristics of a specific version? What do different versions have in common? What are the characteristics of a specific version? What makes up a piece of music? What makes music come alive? Coarse Level Fine Level Coarse Level Fine Level What do different versions have in common? What are the characteristics of a specific version? What do different versions have in common? What are the characteristics of a specific version? What makes up a piece of music? What makes music come alive? What makes up a piece of music? What makes music come alive? Identify despite of differences Identify the differences Identify despite of differences Identify the differences Example tasks: Matching Cover Song Identification Example tasks: Tempo Estimation Score (reference): Performance: Performance:

7 Score (reference): Score (reference): Strategy: Compute score-audio synchronization and derive tempo curve Performance: Tempo Curve: Musical tempo (BPM) Musical time (measures) Score (reference): Score (reference): Tempo Curves: Tempo Curves: Musical tempo (BPM) Musical tempo (BPM) Musical time (measures) Musical time (measures) Score (reference): What can be done if no reference is available? Tempo Curves: Musical tempo (BPM)? Tempo Curves: Musical tempo (BPM) Musical time (measures) Musical time (measures)

8 Relative Absolute Relative Absolute Given: Several versions Given: One version Given: Several versions Given: One version Comparison of extracted parameters Direct interpretation of extracted parameters Relative Absolute Relative Absolute Given: Several versions Given: One version Given: Several versions Given: One version Comparison of extracted parameters Direct interpretation of extracted parameters Comparison of extracted parameters Direct interpretation of extracted parameters Extraction errors have often no consequence on final result Extraction errors immediately become evident Extraction errors have often no consequence on final result Extraction errors immediately become evident Example tasks: Music Synchronization Genre Classification Example tasks: Music Transcription Tempo Estimation Basic task: Tapping the foot when listening to music Basic task: Tapping the foot when listening to music Example: Queen Another One Bites The Dust

9 Basic task: Tapping the foot when listening to music Example: Happy Birthday to you Pulse level: Measure Example: Queen Another One Bites The Dust Example: Happy Birthday to you Example: Happy Birthday to you Pulse level: Tactus (beat) Pulse level: Tatum (temporal atom) Example: Chopin Mazurka Op Example: Chopin Mazurka Op Pulse level: Quarter note Pulse level: Quarter note Tempo:??? Tempo: BPM Tempo curve Tempo (BPM) Time (beats)

10 Which temporal level? Local tempo deviations Spectrogram Steps: 1. Spectrogram Sparse information (e.g., only note onsets available) Vague information (e.g., extracted note onsets corrupt) Frequency (Hz) Compressed Spectrogram Steps: Difference Spectrogram Steps: 1. Spectrogram 2. Log Compression 1. Spectrogram 2. Log Compression 3. Differentiation Frequency (Hz) Frequency (Hz) Steps: 1. Spectrogram 2. Log Compression 3. Differentiation 4. Accumulation Steps: 1. Spectrogram 2. Log Compression 3. Differentiation 4. Accumulation Novelty Curve Novelty Curve Local Average

11 Steps: 1. Spectrogram 2. Log Compression 3. Differentiation 4. Accumulation 5. Normalization Tempo (BPM) Intensity Novelty Curve Tempo (BPM) Intensity Tempo (BPM) Intensity Tempo (BPM) Intensity Tempo (BPM) Intensity

12 Light effects Novelty Curve Music recommendation DJ editing Predominant Local Pulse (PLP) Motivic Similarity Motivic Similarity Beethoven s Fifth (1st Mov.) Beethoven s Fifth (1st Mov.) Beethoven s Fifth (3rd Mov.) Motivic Similarity Motivic Similarity Beethoven s Fifth (1st Mov.) Beethoven s Fifth (3rd Mov.) Beethoven s Appassionata

13 Motivic Similarity Thanks Thomas Prätzlich (Labs Erlangen) B A C H Peter Grosche (Saarland University) Sebastian Ewert (Queen Mary University of London) Michael Clausen (Bonn University) Verena Konz (Saarland University) Joachim Veit (Hochschule für Musik Detmold) Rainer Kleinertz (Saarland University) Book Project A First Course on Textbook (approx. 500 pages) Project & Cooperations DFG-Project Harmonic Analysis Wagner Computergestützte Analyse harmonischer Strukturen Cooperation: Rainer Kleinertz Music Representations 2. Fourier Analysis of Signals 3. Music Synchronization 4. Music Structure Analysis 5. Chord Recogntion 6. Tempo and Beat Tracking 7. Content-based Retrieval 8. Informed Decomposition To appear (plan): End of 2015 DFG-Project: SIAMUS: Source Separation Notentext-Informierte parametrisierung von Musiksignalen BMBF-Project: Freischütz Digital Freischütz Digital Paradigmatische Umsetzung eines genuin digitalen Editionskonzepts Cooperation : Joachim Veit, Thomas Betzwieser, Gerd Szwillus DFG-Project: METRUM: Structure Analysis Mehrschichtige Analyse und Strukturierung von Musiksignalen Cooperation: Michael Clausen Laufzeit: