Music Information Retrieval

Transcription

1 Music Information Retrieval When Music Meets Computer Science Meinard Müller International Audio Laboratories Erlangen Berlin MIR Meetup

2 Meinard Müller 2001 PhD, Bonn University 2002/2003 Postdoc, Keio University, Japan 2007 Habilitation, Bonn University Information Retrieval for Music and Motion Senior Researcher Max-Planck Institut für Informatik, Saarland 2012: Professor Semantic Audio Processing Universität Erlangen-Nürnberg

3 Group Members Stefan Balke Christian Dittmar Patricio López-Serrano Christof Weiß Frank Zalkow Thomas Prätzlich

4 Group Members Stefan Balke Christian Dittmar Patricio López-Serrano Christof Weiß Frank Zalkow Thomas Prätzlich

5 Book: Fundamentals of Music Processing Meinard Müller Fundamentals of Music Processing Audio, Analysis, Algorithms, Applications 483 p., 249 illus., 30 illus. in color, hardcover ISBN: Springer, 2015 Accompanying website:

6 International Audio Laboratories Erlangen

7 International Audio Laboratories Erlangen Audio

8 International Audio Laboratories Erlangen Audio Coding 3D Audio Audio Psychoacoustics Music Processing

9 Music

10 Music Information Retrieval Sheet Music (Image) CD / MP3 (Audio) MusicXML (Text) Dance / Motion (Mocap) Music MIDI Singing / Voice (Audio) Music Film (Video) Music Literature (Text)

11 Music Information Retrieval Signal Processing Musicology Music User Interfaces Machine Learning Information Retrieval Library Sciences

12 Piano Roll Representation

13 Player Piano (1900)

14 Piano Roll Representation (MIDI) J.S. Bach, C-Major Fuge (Well Tempered Piano, BWV 846) Time Pitch

15 Piano Roll Representation (MIDI) Query: Goal: Find all occurrences of the query

16 Piano Roll Representation (MIDI) Query: Goal: Find all occurrences of the query Matches:

17 Music Retrieval Database Query Hit Audio-ID Version-ID Category-ID Bernstein (1962) Beethoven, Symphony No. 5 Beethoven, Symphony No. 5: Bernstein (1962) Karajan (1982) Gould (1992) Beethoven, Symphony No. 9 Beethoven, Symphony No. 3 Haydn Symphony No. 94

18 Music Synchronization: Audio-Audio Beethoven s Fifth

19 Music Synchronization: Audio-Audio Beethoven s Fifth Orchester (Karajan) Piano (Scherbakov) Time (seconds)

20 Music Synchronization: Audio-Audio Beethoven s Fifth Orchester (Karajan) Piano (Scherbakov) Time (seconds)

21 Application: Interpretation Switcher

22 Music Synchronization: Image-Audio Audio Image

23 Music Synchronization: Image-Audio Audio Image

24 How to make the data comparable? Audio Image

25 How to make the data comparable? Image Processing: Optical Music Recognition Audio Image

26 How to make the data comparable? Image Processing: Optical Music Recognition Audio Image Audio Processing: Fourier Analysis

27 How to make the data comparable? Image Processing: Optical Music Recognition Audio Image Audio Processing: Fourier Analysis

28 Application: Score Viewer

29 Music Processing Coarse Level What do different versions have in common? Fine Level What are the characteristics of a specific version?

30 Music Processing Coarse Level What do different versions have in common? What makes up a piece of music? Fine Level What are the characteristics of a specific version? What makes music come alive?

31 Music Processing Coarse Level What do different versions have in common? What makes up a piece of music? Identify despite of differences Fine Level What are the characteristics of a specific version? What makes music come alive? Identify the differences

32 Music Processing Coarse Level What do different versions have in common? What makes up a piece of music? Identify despite of differences Example tasks: Audio Matching Cover Song Identification Fine Level What are the characteristics of a specific version? What makes music come alive? Identify the differences Example tasks: Tempo Estimation Performance Analysis

33 Performance Analysis Schumann: Träumerei Performance: Time (seconds)

34 Performance Analysis Schumann: Träumerei Score (reference): Performance: Time (seconds)

35 Performance Analysis Schumann: Träumerei Score (reference): Strategy: Compute score-audio synchronization and derive tempo curve Performance: Time (seconds)

36 Performance Analysis Schumann: Träumerei Score (reference): Tempo Curve: Musical tempo (BPM) Musical time (measures)

37 Performance Analysis Schumann: Träumerei Score (reference): Tempo Curves: Musical tempo (BPM) Musical time (measures)

38 Performance Analysis Schumann: Träumerei Score (reference): Tempo Curves: Musical tempo (BPM) Musical time (measures)

39 Performance Analysis Schumann: Träumerei Score (reference): Tempo Curves: Musical tempo (BPM)? Musical time (measures)

40 Performance Analysis Schumann: Träumerei What can be done if no reference is available? Tempo Curves: Musical tempo (BPM) Musical time (measures)

41 Music Processing Relative Given: Several versions Absolute Given: One version

42 Music Processing Relative Given: Several versions Comparison of extracted parameters Absolute Given: One version Direct interpretation of extracted parameters

43 Music Processing Relative Given: Several versions Comparison of extracted parameters Extraction errors have often no consequence on final result Absolute Given: One version Direct interpretation of extracted parameters Extraction errors immediately become evident

44 Music Processing Relative Given: Several versions Comparison of extracted parameters Extraction errors have often no consequence on final result Example tasks: Music Synchronization Genre Classification Absolute Given: One version Direct interpretation of extracted parameters Extraction errors immediately become evident Example tasks: Music Transcription Tempo Estimation

45 Tempo Estimation and Beat Tracking Basic task: Tapping the foot when listening to music

46 Tempo Estimation and Beat Tracking Basic task: Tapping the foot when listening to music Example: Queen Another One Bites The Dust Time (seconds)

47 Tempo Estimation and Beat Tracking Basic task: Tapping the foot when listening to music Example: Queen Another One Bites The Dust Time (seconds)

48 Tempo Estimation and Beat Tracking Example: Happy Birthday to you Pulse level: Measure

49 Tempo Estimation and Beat Tracking Example: Happy Birthday to you Pulse level: Tactus (beat)

50 Tempo Estimation and Beat Tracking Example: Happy Birthday to you Pulse level: Tatum (temporal atom)

51 Tempo Estimation and Beat Tracking Example: Chopin Mazurka Op Pulse level: Quarter note Tempo:???

52 Tempo Estimation and Beat Tracking Example: Chopin Mazurka Op Pulse level: Quarter note Tempo: BPM Tempo curve Tempo (BPM) Time (beats)

53 Tempo Estimation and Beat Tracking Which temporal level? Local tempo deviations Sparse information (e.g., only note onsets available) Vague information (e.g., extracted note onsets corrupt)

54 Tempo Estimation and Beat Tracking Spectrogram Steps: 1. Spectrogram Frequency (Hz) Time (seconds)

55 Tempo Estimation and Beat Tracking Compressed Spectrogram Steps: 1. Spectrogram 2. Log Compression Frequency (Hz) Time (seconds)

56 Tempo Estimation and Beat Tracking Difference Spectrogram Steps: 1. Spectrogram 2. Log Compression 3. Differentiation Frequency (Hz) Time (seconds)

57 Tempo Estimation and Beat Tracking Steps: 1. Spectrogram 2. Log Compression 3. Differentiation 4. Accumulation Novelty Curve Time (seconds)

58 Tempo Estimation and Beat Tracking Steps: 1. Spectrogram 2. Log Compression 3. Differentiation 4. Accumulation Novelty Curve Local Average Time (seconds)

59 Tempo Estimation and Beat Tracking Steps: 1. Spectrogram 2. Log Compression 3. Differentiation 4. Accumulation 5. Normalization Novelty Curve Time (seconds)

60 Tempo Estimation and Beat Tracking Tempo (BPM) Intensity

61 Tempo Estimation and Beat Tracking Tempo (BPM) Intensity

62 Tempo Estimation and Beat Tracking Tempo (BPM) Intensity

63 Tempo Estimation and Beat Tracking Tempo (BPM) Intensity

64 Tempo Estimation and Beat Tracking Tempo (BPM) Intensity Time (seconds)

65 Tempo Estimation and Beat Tracking Novelty Curve Predominant Local Pulse (PLP) Time (seconds)

66 Tempo Estimation and Beat Tracking Light effects Music recommendation DJ Audio editing

67 Why is Music Processing Challenging? Example: Chopin, Mazurka Op. 63 No. 3

68 Why is Music Processing Challenging? Example: Chopin, Mazurka Op. 63 No. 3 Waveform Amplitude Time (seconds)

69 Why is Music Processing Challenging? Example: Chopin, Mazurka Op. 63 No. 3 Waveform / Spectrogram Frequency (Hz) Time (seconds)

70 Why is Music Processing Challenging? Example: Chopin, Mazurka Op. 63 No. 3 Waveform / Spectrogram Performance Tempo Dynamics Note deviations Sustain pedal

71 Why is Music Processing Challenging? Example: Chopin, Mazurka Op. 63 No. 3 Waveform / Spectrogram Performance Tempo Dynamics Note deviations Sustain pedal Polyphony Main Melody Additional melody line Accompaniment

72 Source Separation Decomposition of audio stream into different sound sources Central task in digital signal processing Cocktail party effect

73 Source Separation Decomposition of audio stream into different sound sources Central task in digital signal processing Cocktail party effect Several input signals Sources are assumed to be statistically independent

74 Source Separation (Music) Main melody, accompaniment, drum track Instrumental voices Individual note events Only mono or stereo Time Sources are often highly dependent Time

75 Harmonic-Percussive Decomposition Mixture:

76 Harmonic-Percussive Decomposition Mixture: Clearly harmonic sounds Clearly percussive sounds Harmonic component Percussive component

77 Harmonic-Percussive Decomposition Mixture: Clearly harmonic sounds Clearly percussive sounds Harmonic component Residual component Percussive component

78 Harmonic-Percussive Decomposition Mixture: Clearly harmonic sounds of singing voice and accompaniment Noise-like sounds Vibrato/glissando sounds Drum hits Fricatives & plosives in singing voice Harmonic component Residual component Percussive component Literature: [Driedger/Müller/Disch, ISMIR 2014] Demo:

79 Singing Voice Extraction Original Recording Singing voice Accompaniment

80 Singing Voice Extraction Frequency Time Original recording HPR F0 annotation Harmonic component Percussive component Residual component MR TR SL Harmonic portion singing voice Harmonic portion accompaniment Fricatives singing voice Instrument onsets accompaniment + + Vibrato & formants singing voice Diffuse instruments sounds accompaniment Estimate singing voice Estimate accompaniment

81 Score-Informed Source Separation Exploit musical score to support separation process Pitch Pitch Pitch Time Time Time

82 Parametric Model Approach Rebuild spectrogram information Estimate Parameters Render Frequency (Hz) Frequency (Hz) Time (seconds) Time (seconds)

83 NMF (Nonnegative Matrix Factorization) M K N M K

84 NMF (Nonnegative Matrix Factorization) M K M N K Magnitude Spectrogram Templates Activations Templates: Pitch + Timbre Activations: Onset time + Duration How does it sound When does it sound

85 NMF-Decomposition Initialized template Initialized activations Frequency Note number Note number Time Random initialization

86 NMF-Decomposition Initialized template Initialized activations Frequency Note number Frequency Note number Learnt templates Learnt activations Note number Time Random initialization No semantic meaning

87 NMF-Decomposition Initialized template Initialized activations Frequency Note number Note number Time Constrained initialization

88 NMF-Decomposition Initialized template Initialized activations Frequency Note number Note number Template constraint for p=55 Time Activation constraints for p=55 Constrained initialization

89 NMF-Decomposition Initialized template Initialized activations Frequency Note number Learnt templates Learnt activations Frequency Note number Org Model Note number Time Constrained initialization NMF as refinement

90 Score-Informed Audio Decomposition Application: Audio editing Frequency (Hertz) Time (seconds) Frequency (Hertz) Time (seconds)

91 Informed Drum-Sound Decomposition Remix: Literature: [Dittmar/Müller, IEEE/ACM-TASLP 2016] Demo:

92 Loop Decomposition of EDM Decomposition Patterns Activations Literature: [López-Serrano/Dittmar/Müller, ISMIR 2016] Demo:

93 Audio Mosaicing Target signal: Beatles Let it be Source signal: Bees Mosaic signal: Let it Bee Literature: [Driedger/Müller, ISMIR 2015] Demo:

94 NMF-Inspired Audio Mosaicing Non-negative matrix factorization (NMF) Non-negative matrix Components Activations. = fixed learned learned Proposed audio mosaicing approach Target s spectrogram Source s spectrogram Activations Mosaic s spectrogram Frequency. = Time source Frequency fixed Time source fixed Time target learned Time target

95 NMF-Inspired Audio Mosaicing Spectrogram target Spectrogram source Activation matrix Spectrogram mosaic Frequency Frequency Time source. = Frequency Time target Time source Time target Time target

96 This image cannot currently be displayed. NMF-Inspired Audio Mosaicing Spectrogram target Spectrogram source Iterative updates Activation matrix Spectrogram mosaic Frequency Frequency Time source. = Frequency Time target Time source Time target Time target Preserve temporal context Core idea: support the development of sparse diagonal activation structures

97 NMF-Inspired Audio Mosaicing Spectrogram target Spectrogram source Activation matrix Spectrogram mosaic Frequency Frequency Time source. = Frequency Time target Time source Time target Time target

98 NMF-Inspired Audio Mosaicing Spectrogram target Spectrogram source Activation matrix Spectrogram mosaic Frequency Frequency Time source. = Frequency Time target Time source Time target Time target

99 Audio Mosaicing Target signal: Chic Good times Source signal: Whales Mosaic signal

100 Audio Mosaicing Target signal: Adele Rolling in the Deep Source signal: Race car Mosaic signal

101 Motivic Similarity

102 Motivic Similarity B A C H

103 Summary Music information retrieval Audio decomposition techniques Machine learning Teaching Academic training of students Fundamental research Music applications & musicology Multimedia scenarios Web-based interfaces

104 Book: Fundamentals of Music Processing Meinard Müller Fundamentals of Music Processing Audio, Analysis, Algorithms, Applications 483 p., 249 illus., hardcover ISBN: Springer, 2015 Accompanying website:

105 Book: Fundamentals of Music Processing Meinard Müller Fundamentals of Music Processing Audio, Analysis, Algorithms, Applications 483 p., 249 illus., hardcover ISBN: Springer, 2015 Accompanying website:

106 MIR-Related Events in Germany AES Conference on Semantic Audio June 2017 Erlangen GI Jahrestagung September 2017 Chemnitz Workshop: Musik trifft Informatik 26 September 2017 Tutorial: Musikverarbeitung 25 September 2017