Aspects of Music Lecture Music Processing Piece of music hord Recognition Meinard Müller International Audio Laboratories rlangen meinard.mueller@audiolabs-erlangen.de Melody Rhythm Harmony Harmony: The Basis of Music Pachelbel s anon Musical hords ombination of three or more tones which sound simultaneously hord classes Triads including major, minor, diminished, augmented chords Many other more complex chords such as seventh chords Here: focus on major and minor triads oversong Die ine (Die irma) Musical hords The major chord Musical hords The minor chord Derived from the major scale Derived from the minor scale ---- the root ---- the (major) third ---- the fifth ---- the root b ---- the (minor) third ---- the fifth
Musical hords Structure of the 24 major/minor chords hord Recognition Development of automatic methods for the harmonic analysis of audio data 0 1 2 3 4 5 6 7 8 9 10 11 Applications in the field of music information retrieval: major minor # D D# # # A A# B music segmentation cover song identification audio matching music structure analysis hord Recognition hord Recognition Signal Result Signal eature xtraction lassification Result Segmentation hord label assignment hroma features Type Resolution ompression Smoothing Pattern matching Template aussian raphical Models hord Recognition Baseline Method for hord Recognition iven: Audio file hord templates 24 major/minor chords Output: Segmentation and chord labeling major # major D major D# major minor # minor B 0 0 0 0 0 0 A# 0 0 0 1 0 0 A 0 0 1 0 0 0 # 0 1 0 0 0 1 1 0 0 1 1 0 A:min :min A:min :min A:min :min # 0 0 1 0 0 0 0 1 0 0 0 0 1 0 0 0 0 1 D# 0 0 0 1 1 0 D 0 0 1 0 0 0 # 0 1 0 0 0 1 1 0 0 0 1 0
Baseline Method for hord Recognition hord templates 24 major/minor chords Baseline Method for hord Recognition 24 chord templates (12 major, 12 minor) hroma feature extraction (framewise) hroma hord labels Baseline Method for hord Recognition Baseline Method for hord Recognition 24 chord templates (12 major, 12 minor) hroma feature extraction (framewise) 24 chord templates (12 major, 12 minor) hroma feature extraction (framewise) ompute for each frame the distance of the feature vector to the 24 templates ompute for each frame the distance of the feature vector to the 24 templates Selected chord according to template with minimal distance to respective feature vector Problems in hord Recognition Problem: Transitions between subsequent chord xample: hopin Mazurka Op. 68 No.3 Problems in hord Recognition Problem: Monphonic musical passages xample: xcerpt of Wagner s Meistersinger orrect alse positive alse negative hromagram B A# A # # D# 1 0.6 0.4 D # 0 10 20 30 40 50 60 0
Problems in hord Recognition Problem: rame-wise chord analysis may not be meaningful xample: Bach: Prelude major, BWV 846 Problems in hord Recognition Problem: Ambiguity of chords A minor major minor Problem: Broken chords A B Measure-wise chord analysis necessary Problems in hord Recognition Problem: Reduction to the 24 major/minor chords makes the recognition of more complex chords difficult/impossible! xample: Prelude major, BWV 846, mm.19-25 Problems in hord Recognition Problem: Tuning problems xample: xcerpt of Wagner s Meistersinger hromagram (from MIDI) B A# A 1 # # D# D 0.6 0.4 # 0 10 20 30 40 50 60 0 Problems in hord Recognition Problem: Tuning problems xample: xcerpt of Wagner s Meistersinger Problems in hord Recognition Problem: Tuning problems xample: xcerpt of Wagner s Meistersinger hromagram (from MIDI) B A# 1 Problem: Audio is tuned more than half a semi-tone upwards hromagram (from MIDI) B A# 1 Problem: Audio is tuned more than half a semi-tone upwards A A # # D# 0.6 0.4 # # D# 0.6 0.4 Solution: Adjust frequency binning when computing pitch features. D D # # 0 10 20 30 40 50 60 0 0 10 20 30 40 50 60 0
Problems in hord Recognition Problem: Tuning problems Key Relations: ircle of ifths xample: The Beatles Lovely Rita Without tuning orrect alse positive alse negative With tuning rom http://en.wikipedia.org/wiki/ircle_of_fifths Key Relations: ircle of ifths Observation: or tonality reasons, some chord progressions are more likely than others. Idea: Usage of Hidden Markov Models (HMMs) to model chord dependencies Markov Models Description of certain stochastic processes Andrei Markov (Wikipedia) rom http://en.wikipedia.org/wiki/ircle_of_fifths Markov Models Description of certain stochastic processes 5 sleep 5 = (S,,P) S: States Processes over discrete time Sequence of random variables X1, X2, Process has to follow Markov property: no memory, only current state known future depends only on present, not on past P(Xn+1 = x Xn = y) = P(Xn+1 = x Xn = y, Xn-1 = y2, ) Andrei Markov (Wikipedia) social activity making music 5 5 coding eating : Transitions P: Transition Note: or each state, the sum of outgoing transition is equal to one. [Radu urticapean]
Smell = (S,P,V,B) = (S,P,V,B) S: States S: States P: Transition A P: Transition V: Observations V: Observations 0.4 B: mission D 0.4 B: mission Noise Light [Radu urticapean] 24 major/minor chords = (S,P,V,B) S: States Two computational problems 1. Training: learn model parameters (Baum-Welch Algorithm) 2. valuation: find optimal state sequence (Viterbi Algorithm) Probabilities for having a transition from one chord to another chord P: Transition Training data (chroma + labels) Musical knowledge hroma vectors V: Observations Probability for a chord model to produce a chorma distribution B: mission Training HMM chord models (transitions & emissions) Test data (unseen chroma) valuation Recognized chords (labels of state sequence) Training Training 0,2 0.4 0.6 Input: Sequence of features (observations) orresponding ground truth chord labels Input: Sequence of features (observations) orresponding ground truth chord labels Output: mission Transition Output: mission Transition
valuation valuation x 1 x 2 x 3 x 4... x 1 x 2 x 3 x 4... 0,2 0,2 0.4 0.6 0.4 0.6 Input: Sequence of features mission Transition Input: Sequence of features mission Transition Output: Optimal state sequence (estimated chord progression) Output: Optimal state sequence (estimated chord progression) Importance of hroma eature Variant Importance of hroma eature Variant P Normalized chromagram LP Logarithmic compression NS[1] Quantized chromagram NS[11] Temporal smoothing Importance of hroma eature Variant Importance of hroma eature Variant P RP[1] RP Boosting timbre invariance LP NS[1] RP[11] ISP ISP Instantaneous frequency NS[11] And many more chroma variants!
Importance of hroma eature Variant xperiment Importance of hroma eature Variant Dependency on feature type Beatles dataset Three-fold cross validation Measurement: -measure ramewise evaluation, each frame = 100 ms 12 major and 12 minor triads -measure eature type Importance of hroma eature Variant Dependency on logarithmic compression Importance of hroma eature Variant Dependency on smoothing (using RP features) -measure -measure ompression factor Smoothing length hroma Toolbox ross-version Analysis eneral Procedure reely available Matlab toolbox eature types: Pitch, hroma, NS, RP http://www.mpi-inf.mpg.de/resources/mir/chromatoolbox/ onduct analysis for multiple versions of the same object Relate the versions (using a reference) ompare analysis results accross different versions Look for consistencies and inconsistencies Harmonic analysis Different music recordings Same piece of music Music synchronization Musical score
Barwise Synchronization 1 2 3 Barwise Synchronization MIDI representation with bar information Time (bars) Barwise Synchronization MIDI representation with bar information Barwise Synchronization Music synchronization Time (bars) Time (seconds) Time (seconds) Barwise Synchronization Transfer bar information to audio domain ross-version Harmonic Analysis hord recognition result Barwise presentation of analysis results is of great benefit! Time (seconds) Time (seconds) Time (bars)
ross-version Harmonic Analysis hord recognition result Barwise overlay across different music recordings ross-version Harmonic Analysis ross-version chord recognition result hord labels onsistency Time (seconds) Time (bars) Time (bars) ross-version Harmonic Analysis ross-version chord recognition result ross-version Visualization xample: Bach s Prelude BWV 846 in major (bars 11-15) Highly consistent: -minor Inconsistent: -minor, -major, b -major Time (bars) ross-version Visualization xample: Bach s Prelude BWV 846 in major (bars 11-15) ross-version Visualization xample: Bach s Prelude BWV 846 in major (bars 11-15) Highly consistent: -major Inconsistent!
ross-version Visualization xample: Bach s Prelude BWV 846 in major (bars 11-15) ross-version Visualization xample: Bach s Prelude BWV 846 in major (bars 11-15) Highly consistent: D-minor round-truth visualization onvenient tool for manual error analysis and evaluation Quantitative valuation Quantitative valuation xample: Bach s Prelude BWV 846 xample: Beethoven s ifth P/R/ measure -measures for individual recordings: Min: 0.44 Max: 7 Mean: 0 P/R/ measure -measures for individual recordings: Min: 3 Max: 3 Mean: 0.60 Degree of consistency Degree of consistency onsistent regions tend to be classified correctly Precision high Recall not too bad Indication of harmonically stable, well-defined tonal centers onsistent regions tend to be classified correctly Precision high Recall not too bad Indication of harmonically stable, well-defined tonal centers Application: xploring Harmonic Structures xample: Beethoven s Piano Sonata Op. 49 No. 2 Application: xploring Harmonic Structures xample: Beethoven s Piano Sonata Op. 49 No. 2
Interface: Interpretation Switcher Absolute mode Reference mode hord annotations for four versions Simultaneous comparison of different version-dependent analysis results (here: chord labels) onclusions & uture Work Importance of feature design step ross-version framework Harmonic analysis Tempo analysis Structure analysis Musically meaningful timeline in bars very convenient! Stabilization of analysis results onsistencies seem to have musical meaning Which meaning Tonal centers Towards interdisciplinary research (MIR + musicology) Visualization as meanigful tool in musicology Helpful for analyis of harmonic relations across entire music corpora Literature Taemin ho, Ron J. Weiss, and Juan Pablo Bello. xploring common variations in state of the art chord recognition systems. Proc. Sound and Music omputing onference (SM), pages 1 8, Barcelona, Spain, 2010. Takuya ujishima. Realtime chord recognition of musical sound: A system using common lisp music. Proc. International omputer Music onference (IM), pages 464 467, Beijing, 1999. Nanzhu Jiang, Peter rosche, Verena Konz, and Meinard Müller. Analyzing chroma feature types for automated chord recognition. Proc. 42nd AS onference, 2011. Verena Konz, Meinard Müller, and Sebastian wert. A multi-perspective evaluation framework for chord recognition. Proc. ISMIR, pages 9 14, Utrecht, The Netherlands, 2010. Matthias Mauch and Simon Dixon. Simultaneous estimation of chords and musical context from audio. I Transactions on Audio, Speech, and Language Processing, 18(6):1280 1289, 2010. Meinard Müller and Sebastian wert. hroma Toolbox: MATLAB implementations for extracting variants of chroma-based audio features. Proc. ISMIR, pages 215 220, Miami, USA, 2011. Hélène Papadopoulos and eoffroy Peeters. Joint estimation of chords and downbeats from an audio signal. I Transactions on Audio, Speech, and Language Processing, 19(1):138 152, 2011. Alexander Sheh and Daniel P. W. llis. hord segmentation and recognition using M-trained hidden Markov models. Proc. ISMIR, pages 185 191, Baltimore, USA, 2003. Yushi Ueda, Yuuki Uchiyama, Takuya Nishimoto, Nobutaka Ono, and Shigeki Sagayama. HMMbased approach for automatic chord detection using refined acoustic features. Proc. IASSP, pages 5518 5521, Dallas, USA, 2010.