Further Topics in MIR

Transcription

1 Tutorial Automatisierte Methoden der Musikverarbeitung 47. Jahrestagung der Gesellschaft für Informatik Further Topics in MIR Meinard Müller, Christof Weiss, Stefan Balke International Audio Laboratories Erlangen {meinard.mueller, christof.weiss,

2 Why is Music Processing Challenging? Example: Chopin, Mazurka Op. 63 No. 3

3 Why is Music Processing Challenging? Example: Chopin, Mazurka Op. 63 No. 3 Waveform Amplitude Time (seconds)

4 Why is Music Processing Challenging? Example: Chopin, Mazurka Op. 63 No. 3 Waveform / Spectrogram (Hz) Time (seconds)

5 Why is Music Processing Challenging? Example: Chopin, Mazurka Op. 63 No. 3 Waveform / Spectrogram Performance Tempo Dynamics Note deviations Sustain pedal

6 Why is Music Processing Challenging? Example: Chopin, Mazurka Op. 63 No. 3 Waveform / Spectrogram Performance Tempo Dynamics Note deviations Sustain pedal Polyphony Main Melody Additional melody line Accompaniment

7 Source Separation Decomposition of audio stream into different sound sources Central task in digital signal processing Cocktail party effect

8 Source Separation Decomposition of audio stream into different sound sources Central task in digital signal processing Cocktail party effect Several input signals Sources are assumed to be statistically independent

9 Source Separation (Music) Main melody, accompaniment, drum track Instrumental voices Individual note events Only mono or stereo Time Sources are often highly dependent Time

10 Harmonic-Percussive Decomposition Mixture:

11 Harmonic-Percussive Decomposition Mixture: Clearly harmonic sounds Clearly percussive sounds Harmonic component Percussive component

12 Harmonic-Percussive Decomposition Mixture: Clearly harmonic sounds Clearly percussive sounds Harmonic component Residual component Percussive component

13 Harmonic-Percussive Decomposition Mixture: Clearly harmonic sounds of singing voice and accompaniment Noise-like sounds Vibrato/glissando sounds Drum hits Fricatives & plosives in singing voice Harmonic component Residual component Percussive component Literature: [Driedger/Müller/Disch, ISMIR 2014] Demo:

14 Singing Voice Extraction Original Recording Singing voice Accompaniment

15 Singing Voice Extraction Time Original recording HPR F0 annotation Harmonic component Percussive component Residual component MR TR SL Harmonic portion singing voice Harmonic portion accompaniment Fricatives singing voice Instrument onsets accompaniment + + Vibrato & formants singing voice Diffuse instruments sounds accompaniment Estimate singing voice Estimate accompaniment

16 Score-Informed Source Separation Exploit musical score to support separation process Pitch Pitch Pitch Time Time Time

17 Parametric Model Approach Rebuild spectrogram information Estimate Parameters Render (Hz) (Hz) Time (seconds) Time (seconds)

18 NMF (Nonnegative Matrix Factorization) M K N M K

19 NMF (Nonnegative Matrix Factorization) M K M N K Magnitude Spectrogram Templates Activations Templates: Pitch + Timbre Activations: Onset time + Duration How does it sound When does it sound

20 NMF-Decomposition Initialized template Initialized activations Note number Note number Time Random initialization

21 NMF-Decomposition Initialized template Initialized activations Note number Note number Learnt templates Learnt activations Note number Time Random initialization No semantic meaning

22 NMF-Decomposition Initialized template Initialized activations Note number Note number Time Constrained initialization

23 NMF-Decomposition Initialized template Initialized activations Note number Note number Template constraint for p=55 Time Activation constraints for p=55 Constrained initialization

24 NMF-Decomposition Initialized template Initialized activations Note number Learnt templates Learnt activations Note number Org Model Note number Time Constrained initialization NMF as refinement

25 Score-Informed Audio Decomposition Application: Audio editing (Hertz) Time (seconds) (Hertz) Time (seconds)

26 Informed Drum-Sound Decomposition Remix: Literature: [Dittmar/Müller, IEEE/ACM-TASLP 2016] Demo:

27 Loop Decomposition of EDM Decomposition Patterns Activations Literature: [López-Serrano/Dittmar/Müller, ISMIR 2016] Demo:

28 Audio Mosaicing Target signal: Beatles Let it be Source signal: Bees Mosaic signal: Let it Bee Literature: [Driedger/Müller, ISMIR 2015] Demo:

29 NMF-Inspired Audio Mosaicing Non-negative matrix factorization (NMF) Non-negative matrix Components Activations. = fixed learned learned Proposed audio mosaicing approach Target s spectrogram Source s spectrogram Activations Mosaic s spectrogram. = Time source fixed Time source fixed Time target learned Time target

30 NMF-Inspired Audio Mosaicing Spectrogram target Spectrogram source Activation matrix Spectrogram mosaic Time source. = Time target Time source Time target Time target

31 This image cannot currently be displayed. NMF-Inspired Audio Mosaicing Spectrogram target Spectrogram source Iterative updates Activation matrix Spectrogram mosaic Time source. = Time target Time source Time target Time target Preserve temporal context Core idea: support the development of sparse diagonal activation structures

32 NMF-Inspired Audio Mosaicing Spectrogram target Spectrogram source Activation matrix Spectrogram mosaic Time source. = Time target Time source Time target Time target

33 NMF-Inspired Audio Mosaicing Spectrogram target Spectrogram source Activation matrix Spectrogram mosaic Time source. = Time target Time source Time target Time target

34 Audio Mosaicing Target signal: Chic Good times Source signal: Whales Mosaic signal

35 Audio Mosaicing Target signal: Adele Rolling in the Deep Source signal: Race car Mosaic signal

36 Motivic Similarity

37 Motivic Similarity B A C H

38 Summary Music information retrieval Audio decomposition techniques Machine learning Teaching Academic training of students Fundamental research Music applications & musicology Multimedia scenarios Web-based interfaces

39 Book: Fundamentals of Music Processing Meinard Müller Fundamentals of Music Processing Audio, Analysis, Algorithms, Applications 483 p., 249 illus., hardcover ISBN: Springer, 2015 Accompanying website:

40 Book: Fundamentals of Music Processing Meinard Müller Fundamentals of Music Processing Audio, Analysis, Algorithms, Applications 483 p., 249 illus., hardcover ISBN: Springer, 2015 Accompanying website: