MuseGAN: Multi-track Sequential Generative Adversarial Networks for Symbolic Music Generation and Accompaniment

Transcription

1 MuseGAN: Multi-track Sequential Generative Adversarial Networks for Symbolic Music Generation and Accompaniment Hao-Wen Dong*, Wen-Yi Hsiao*, Li-Chia Yang, Yi-Hsuan Yang Research Center of IT Innovation, Academia Sinica Demo Page *these authors contributed equally to this work

2 Outline Goals & Challenges Data Proposed Model Results & Evaluation Future Works Source Code Demo Page 2

3 Generate pop music of multiple tracks Goals [Source Code] salu133445/musegan [Demo Page] github.io/musegan/ in piano-roll format using GAN with CNNs 3

4 Challenge I Multitrack Interdependency vocal Multi-track GAN piano strings bass drums music & clip by phycause 4

5 Challenge II Music Texture Convolutional Neural Networks melody chord (harmony) 5

6 Challenge III Temporal Structure song paragraph 1 paragraph 2 paragraph 3 phrase 1 phrase 2 phrase 3 phrase 4 bar 1 bar 2 bar 3 bar 4 4/4 time beat 1 beat 2 beat 3 beat 4 step 1 step 2 step 24 6

7 Challenge III Temporal Structure Convolutional Neural Networks Fixed Structure bar 1 bar 2 bar 3 bar 4 4/4 time beat 1 beat 2 beat 3 beat 4 step 1 step 2 step 24 7

8 Data Representation Piano-roll (with symbolic timing) time step polyphonic multi-track Bar 1 Bar 2 Bar 3 Bar 4 A3 pitch time t 0 t 1 8

9 Data Representation Multi-track Piano-roll (with symbolic timing) polyphonic multi-track pitch tracks time 9

10 Data Representation Bass Piano Drums Strings Guitar 4 bars 84 pitches 5 tracks 96 time steps a tensor 10

11 Data [Dataset] ub.io/musegan/dataset [Pypianoroll] github.io/pypianoroll/ LPD (Lakh Pianoroll Dataset) >170,000 multi-track piano-rolls Derived from Lakh MIDI Dataset Mainly pop songs Pypianoroll (Python package) Manipulation & Visualiation Efficient Save/Load Parse/Write MIDI files On PYPI (pip installable) 11

12 Generative Adversarial Networks random noise ~p() G fake data G() D 1/0 X real data 12

13 Generative Adversarial Networks random noise fake data Goal of G Make G() undistinguishable from real data for D ~p() G G() log(1-d(g())) D 1/0 X real data log(1-d(x)) + log(d(g())) Goal of D Distinguish G() being fake from X being real 13

14 Generative Adversarial Networks random noise ~p() Generator G fake data G() Discriminator critic (wgan-gp) D real/fake X real data 4-bar phrases of 5 tracks 14

15 MuseGAN An Overview temporal generator bar generator G temp G bar 1 random noise 4 latent variables 4 piano-roll matrices 15

16 Generator Bar Generator G 16

17 Generator Coordination track-independent No Coordination track-dependent Bar Generator G 17

18 Generator G Bar Generator G G 18

19 Generator G Bar Generator G G 19

20 Generator Time Dependent Independent Dependent Melody Groove Track Independent Chords Style Chords G Bar Generator Style Melody G G Groove 20

21 MuseGAN 21

22 Bass Line Results Sample 1 Sample 2 Drum pattern Chords More Samples on Demo Page Bass Drums Guitar Strings Piano Step 0 Step 700 Step 2500 Step 6000 Step

23 Monitor the Training Objective Metrics Negative Critic Loss UPC step step QN UPC QN number of used pitch classes per bar ratio of qualified notes step 23

24 User Study composer H: harmonious R: rhythmic MS: musically structured C: coherent OR: overall rating jamming hybrid 24

25 Accompaniment System Conditional GAN Generation from Scratch Accompaniment System nothing 5-track single-track 5-track 25

26 Summary MuseGAN a novel GAN for multi-track sequence generation multi-track, polyphonic music human-ai cooperative scenario Lakh Pianoroll Dataset (LPD) (new dataset!!) Pypianoroll (new package!!) 26

27 Full Song Generation Future Works song paragraph 1 paragraph 2 paragraph 3 phrase 1 phrase 2 phrase 3 phrase 4 bar 1 bar 2 bar 3 bar 4 beat 1 beat 2 beat 3 beat 4 step 1 step 2 step 24 Hierarchical Temporal Structure 27

28 Future Works Cross-modal Generation Music + Video Music + Lyrics Video + Text 28

29 MIR Music Information Research Analysis music features e.g. chord recognition, beat/downbeat detection, music transcription, source separation Retrieval query music e.g. query by humming, similarity search, music recommendation, playlist generation Generation X music e.g. generation, accompaniment, style transfer, mashup, remix 29

30 人聲分離 Music and Audio Computing Lab 分離音樂 分離人聲 MACLab Research Center for IT Innovation, Academia Sinica 音樂生成 MIDI 音樂格式 運用 machine learning 技術, 從歌曲中萃取出人聲以及音樂兩部分 音樂精彩段落擷取 音樂拼圖遊戲 ( 應用 : 音樂串燒生成 ) demo: [Lab Website] du.tw/ 創作系統 伴奏系統 多音軌 / 樂器模型 請搜尋 MuseGAN MidiNet Lab Director Dr. Yi-Hsuan Yang 30

31 AAAI

32 New Orleans 32

33 Mardi Gras 33

34 Source Code Demo Page Q&A MuseGAN: Multi-track Sequential Generative Adversarial Networks for Symbolic Music Generation and Accompaniment