Lecture 15: Research at LabROSA

Size: px

Start display at page:

Download "Lecture 15: Research at LabROSA"

Thomasina Fisher
5 years ago
Views:

1 ELEN E4896 MUSIC SIGNAL PROCESSING Lecture 15: Research at LabROSA 1. Sources, Mixtures, & Perception 2. Spatial Filtering 3. Time-Frequency Masking 4. Model-Based Separation Dan Ellis Dept. Electrical Engineering, Columbia University E4896 Music Signal Processing (Dan Ellis) /19

2 Sparse + Low-Rank + NMF Optimization to decompose spectogram: minimize s.t. Zhuo Chen S 1 + L + D KL (Y S L H W) Y = S + L + H W Y H W L S E4896 Music Signal Processing (Dan Ellis) /19

3 Beta Process NMF Automatically choose how many components to use Liang, Hoffman X = D(S Z)+E E4896 Music Signal Processing (Dan Ellis) /19

4 Music Complexity Colin Raffel How can we capture musical patterns in the Million Song Dataset? Network analysis of quantized simultaneities after Serrà et al. 212 from Serrà, Corral, Boguña, Haro, & Arcos, 212 E4896 Music Signal Processing (Dan Ellis) /19

5 Large-Scale Cover Recognition 1 Thierry Bertin-Mahieux How can we find covers in 1M 1 sec / comparison, one search = 11.5 CPU-days full N 2 mining = 16, CPU-years Need a hashing technique landmark-based description of chroma patches Euclidean space projection? E4896 Music Signal Processing (Dan Ellis) /19

Large-Scale Cover Recognition 2 2D Fourier Transform Magnitude (2DFTM) fixed-size feature to capture essence of chromagram: First results on finding covers in

6 Large-Scale Cover Recognition 2 2D Fourier Transform Magnitude (2DFTM) fixed-size feature to capture essence of chromagram: First results on finding covers in 1M songs Thierry Bertin-Mahieux Average rank meanap random 5,. jumpcodes 2 38, DFTM (5 PC) 137,117.2 E4896 Music Signal Processing (Dan Ellis) /19

7 Jazz Discography Project How can MIR help organize jazz collections? our tools are quite genre-specific e.g. beat tracker is fine for pop, useless for Jazz E4896 Music Signal Processing (Dan Ellis) /19

Local Tagging MFCC-statistics classifiers on 5 sec windows trained from MajorMiner data 1 Soul

silence quiet noise solo strings indie house alternative r_b funk soft ambient british distortion

hip_hop jazz piano techno dance female bass vocal pop electronic rock synth male guitar drum 5 1

8 Local Tagging MFCC-statistics classifiers on 5 sec windows trained from MajorMiner data 1 Soul Eyes freq / Hz _9s club trance end drum_bass singing horns punk samples silence quiet noise solo strings indie house alternative r_b funk soft ambient british distortion drum_machine country keyboard saxophone fast instrumental electronica 8s voice beat slow rap hip_hop jazz piano techno dance female bass vocal pop electronic rock synth male guitar drum time / s E4896 Music Signal Processing (Dan Ellis) /

9 Onset Correlation Ahead of or behind the beat? Brian McFee Tony Williams Elvin Jones E4896 Music Signal Processing (Dan Ellis) /19

10 Structural Similarity Diego Silva Helene Papadopoulos Self-similarity shows repeating structure in music Can we find similar pieces by finding similar structures? from Bello 211 E4896 Music Signal Processing (Dan Ellis) /19

Ordinal LDA Segmentation Low-rank decomposition of skewed selfsimilarity to identify repeats Learned weighting of multiple factors to segment Linear Discriminant Analysis between adjacent segments

11 Ordinal LDA Segmentation Low-rank decomposition of skewed selfsimilarity to identify repeats Learned weighting of multiple factors to segment Linear Discriminant Analysis between adjacent segments Beat Lag Beat McFee E4896 Music Signal Processing (Dan Ellis) / Self-similarity Filtered self-sim Beat Lag Skewed self-sim Beat Factor Latent repetition Beat

Lyric Recognition Speech Recognition for

Matt McVicar 4 Polyphonic Audio 4 Acapella

4 6 8 4 Natural Speech 4 Synthesized Speech

(seconds) 3 2 1 1 2 3 4 5 6 7 Time (seconds)

12 Lyric Recognition Speech Recognition for Songs lots of interference atypical speech Matt McVicar 4 Polyphonic Audio 4 Acapella Audio Frequency (khz) Natural Speech 4 Synthesized Speech Frequency (khz) Time (seconds) Time (seconds) E4896 Music Signal Processing (Dan Ellis) /19

13 Singing ASR Speech recognition adapted to singing needs aligned data Align scraped acapellas and full mix including jumps McVicar E4896 Music Signal Processing (Dan Ellis) /19

14 Remixavier" Optimal align-and-cancel of mix and acapella timing and channel may differ Raffel E4896 Music Signal Processing (Dan Ellis) /19

15 Million Song Dataset Many Facets Echo Nest audio features + metadata Echo Nest taste profile user-song-listen count Second Hand Song covers musixmatch lyric BoW last.fm tags Now with audio? resolving artist / album / track / duration against what.cd Bertin-Mahieux McFee E4896 Music Signal Processing (Dan Ellis) /19

16 MIDI-to-MSD Aligned MIDI to Audio is a nice transcription Raffel Shi E4896 Music Signal Processing (Dan Ellis) /19

De-DTMF Problem: Stationary tones confuse speech detector Adaptively filter sinusoids with steady amplitude Frequency Input audio tcp_d1_2_counting_cia_irdial Spectrum and LPC fit 3 6 2 1 55 56 57

17 De-DTMF Problem: Stationary tones confuse speech detector Adaptively filter sinusoids with steady amplitude Frequency Input audio tcp_d1_2_counting_cia_irdial Spectrum and LPC fit Time Framing Gain / db Freq / Hz Imaginary Part 1 LPC poles Real Part t Imaginary Par.7 LPC poles detail Real Part LPC fit Find roots Transform radii Mapped radius Ouput audio Overlapadd Filter audio frames Add poles Map to zeros Frequency Filtered signal Time Filter response & spectrum 6 Gain / db Freq / Hz Real Part E4896 Music Signal Processing (Dan Ellis) /19 Imaginary Part 1 Transformed filter 15 Imaginary Part Transformed filter detail Real Part

18 Pitch-based Filtering Resample to flatten pitch, then filter E4896 Music Signal Processing (Dan Ellis) /19

19 Summary Signal Separation NMF, RPCA, cancellation, filtering Music Information Beat tracking, segmentation Large datasets Indexing & retrieval Speech Lyric recognition Speech detection & enhancement E4896 Music Signal Processing (Dan Ellis) /19

20 References [Bello 211] J P Bello, Measuring structural similarity in music, IEEE Tr. Audio, Speech, & Lang., 19(7): , 211. [Serra et al. 212] J Serrà, A Corral, M Boguña, M. Haro, & J. Arcos, Measuring the evolution of contemporary western popular music, Scientific Reports, 2:521, 212. E4896 Music Signal Processing (Dan Ellis) /19

Data Driven Music Understanding

Data Driven Music Understanding ata riven Music Understanding an Ellis Laboratory for Recognition and Organization of Speech and udio ept. Electrical Engineering, olumbia University, NY US http://labrosa.ee.columbia.edu/ 1. Motivation: