LSTM Neural Style Transfer in Music Using Computational Musicology

Size: px
Start display at page:

Download "LSTM Neural Style Transfer in Music Using Computational Musicology"

Transcription

1 LSTM Neural Style Transfer in Music Using Computational Musicology Jett Oristaglio Dartmouth College, June Introduction In the 2016 paper A Neural Algorithm of Artistic Style, Gatys et al. discovered that by modifying CNNs trained on object recognition, they were capable of separating style from content in images. i This critical concept of content/style separation allowed them to merge the style of well-known artwork with the content of a regular photo, creating artistic variations on classic art that still kept the original content recognizable. This process is known as neural style transfer. Thus far, neural style transfer has been mostly limited in application to the blending of images by using CNN s. My project attempts to tackle the problem of neural style transfer in the domain of music. In music, the content of a song can be approximated by the melody line, and the style can be approximated by the harmonies around the melody. By first using a computational musicology library to separate melody from harmony in songs, I then used an LSTM to predict classical harmonies that accompany classical melody lines. ii The network is then fed jazz melody lines, in order to generate classical harmonies over jazz melodies, effectively resulting in neural style transfer in music using LSTMs. 2. Related Work This project is based on concepts introduced by Gatys et al. in the above-mentioned paper on neural style transfer. The idea of content/style separation is the basis of neural style transfer, although the domain that they applied it to is unrelated to music. In the field of music, there is work being done in pure music generation. DeepJazz is a neural network that uses a 2-layer LSTM to generate music based on a jazz song by training only on that song. iii It learns the general structure, chords, and melody of the song and creates similar (but not quite identical) samples. Project Magenta is an open-source project by Google to advance the state-of-the-art in music generation, and includes multiple different models that generate music using diverse methods. iv Both of these projects focus on pure music generation, and do not attempt neural style transfer. There is an attempt at neural style transfer in music, which seems to use raw audio converted to a 2D spectrogram image using Short Time Fourier Transformations. v They Gatys regular neural style transfer via CNNs on the 2D spectrogram. This leads to perceptually poor results when neural style transfer is attempted garbled songs that sound like they are played on top of each other. There does not appear to be a previous attempt to use LSTMs for neural style transfer in music.

2 3. Data and Methods Data Representation The first design choice involved choosing how to represent musical events (i.e. each note being played at a time step) as a vector for the network. To represent a sequence of musical events, I use a tensor: where and. On-notes are recorded as hot encodings, with each row representing a time step, and each columns representing a specific note on the scale. A single note in a single time step would be represented as a vector with a one-hotencoding: This vector only represents a single note among 12 possible pitches for example, middle C to middle B. A full harmony in this time step would be represented by a vector with multi-class encodings: And a full sequence with a length of N and 12 pitches would be represented by a matrix of dimensionality Nx12: Content/Style Separation Now that we have decided how to represent the song as a vector, the central problem of the project is to find a way to represent content separately from style. In order to do this, I approximate musical content as the melody line of the song; when one thinks of the song Ode to Joy, one doesn t imagine the low cello harmonies, but rather the soprano violin melodies. The style of the song can be approximated with the harmony that surrounds the melody; jazz chords are very distinctive from classical harmonies, and the exact same melody can result in the same song in two different styles depending on the surrounding harmony. In order to separate melody from harmony, I used the Computational Musicology Python library Music21. vi It can parse MIDI files, automatically translating them into Stream objects that contain each musical voice as a continuous stream of notes. I can then use this library to iterate through each song object, finding the high notes at each time step and adding them to a separate melody stream, and adding the accompanying lower notes to a separate harmony stream. I am then left with a stream that represents the content of the song in the form of melody, and another stream that represents the style of the song as the harmony. Data Preprocessing In the data preprocessing step, I first parse each of the MIDI files into Stream objects using the Music21 library. I transpose each song to the key of C (or A minor if it is in a minor scale, which uses the same notes as C). I restrict the number of octaves in order to limit the number of possible pitches. I then

3 breakdown the stream into its component melody and harmony streams, saving the streams separately as the content and style representations of the song. Finally, I translate these separated stream objects into vectorized inputs, where each song is represented by a tensor x as described above. I then splice n random samples from each song that each have a dimensionality of LxP, where L is a hyperparameter denoting a uniform sequence length, and P is a hyperparameter denoting the total number of possible pitches. This is built into a batch tensor of size [n, L, P] for each song. Finally, the samples from each song are joined together in a full batch of size [T, L, P], where T = n * (Total Number of Songs). Here is an example of this content and style separation in action: Combined song: Melody stream only: Neural Style Transfer Once the two tensors containing the full batch of sampled melodies and harmonies were built, the groundwork is set for neural style transfer. The full dataset is represented by the batch of sampled melody tensors as the network input, and the batch of harmony tensors as the network ground truth: An RNN (or more specifically, an LSTM) is given sequences of melody vectors as input, and is trained to output predictions for the sequence of surrounding harmonies. The model learns to act as a function that can predict full harmonies from a simple melody: If it is trained on a corpus from one genre, and then is fed the melody lines from songs of a different genre, I hypothesize that it will generate harmonies from the original genre around the new melodies, effectively implementing neural style transfer for music taking the content of one song, and changing its style. Chord stream only: Dataset I chose to train the network on classical music, and attempt neural style transfer on jazz melodies. I used the Classical Midi Dataset at which is a small corpus of 89 classical songs translated into full MIDI files. vii There are other classical datasets, but they are often poorly

4 formatted and bad translations of the original songs I found the best results by taking many subsamples of this well-formed dataset. For the actual neural style transfer using jazz melodies, I use the Weimar Jazz Solo Database at jazzomat.hfmweimar.de/dbformat/dbcontent.html. viii It is a corpus of 456 jazz songs with just their melody lines translated into MIDI files. These are small datasets, which was a challenge for the project. Unfortunately, there are very few high-quality MIDI datasets for jazz or classical music many of them are poorly translated, or have other problems that cause complications for a network (for instance, many songs encode the percussion as being played by a piano, which introduces noise and poor formatting into the note stream). Loss Function and Training The loss function used in the network was simple binary cross-entropy, which counts false positives as well as false negatives: 4. Model Design and Implementation Model Architecture I used many different design choices in order to try to produce perceptually compelling generated music with this method of neural style transfer. For the broader network architecture, I used a 2- layer LSTM, with 256 nodes each hidden layer, and 0.5 dropout. The number of possible pitches was restricted to 60, so the output layer was a densely connected layer with 60 nodes and a sigmoid activation. The input is a batch of melody vectors, and the output is a predicted batch of harmony vectors. Unfortunately, this loss function caused a problem: because there are so many more off-notes than on-notes in any given time step, the network was pushed to extremely low prediction values. I hypothesize that with a reweighted loss function, the network would achieve significantly better results. However, despite best efforts I was unable to find a way to successfully implement reweighting without porting the network to PyTorch and using dynamic graph computations; I instead used simple unweighted cross-entropy. Training and Hyperparameters For training, I used Adam gradient descent optimization, ix with weight decay of and a variable learning rate over 50 epochs:

5 0.01 for the first 10 epochs, then for the next 25, and until the end of training. The network trained for 50 epochs, making predictions on a jazz vector every 5 epochs throughout training. It trained on batch sizes of 32, and used 20 samples per song, each with uniform sequence length of 256. I will explain in the next section what each step in the sequence represented 5. Experiments and Results Format of Results Because the goal of the network is to generate perceptually compelling music through neural style transfer, the actual statistics of training on classical music such as accuracy and loss function graphs are not highly informative as to the performance of the network at test time on jazz melodies. Most of the networks achieved extremely high accuracy and low loss both on the training and validation sets unfortunately, this did not help the network generate good predictions, because the predictions mostly hovered close to 0 due to the weighting problem. Almost all of the training data looked like the following when graphed: By manually adjusting the way that predictions were translated to MIDIs I was capable of generating some level of predictions. I implemented both a manually adjustable cutoff for predicting notes, as well as a method for sampling the notes and predictions. However, for the most part it was an unsuccessful endeavor the cutoff points required heavy hand-tweaking, and generally predicted common classical notes across the whole song with little variation in time or note choice. The sampling, on the other hand, tended to produce scattered and entirely chaotic results. In order to try to improve the results, I also implemented two different ways to build the harmony vectors; in one, each step in the sequence represented a single uniform time step through the song, that was adjustable by a hyperparameter. In the second, each sequential step represented the jump to the next melody note, and the harmonies were forced to snap to the melody notes. This generated perceptually improved results, because it kept the timing of the original jazz melodies but the network did not learn the times on its own, and this therefore represents another form of hand-tweaking for perceptual quality.

6 Examples of Results As stated before, the results were extremely poor. Even with hand-selected cutoffs, it tended to find mostly degenerate solutions with the same notes playing repetitively, and most of my attempts at tweaking the network led to actually worse results. The network occasionally made predictions that sounded like there was some level of classical harmony over jazz melody, but they required too much hand-selection to be particularly compelling. The vast majority of predictions were highly perceptually uncompelling and repetitive. Attached are the embedded audio of the results of several experiments: Vectorized by: Time Time subdivision: 4 Cutoff level: Conclusion and Discussion Despite the poor results, I believe that neural style transfer is possible via this methodology and architecture. However, it would require successfully implementing a weighed loss function to counteract the heavy imbalance between off-notes and onnotes. The squeezing of predictions to nearzero made it difficult to get any perceptually compelling results. However, the data preprocessing that initially separated content from style was highly effective. With more time, future work could involve modifying the loss function to weigh on-notes equally as off-notes. This may lead to perceptually compelling neural style transfer for music using computation musicology preprocessing and LSTM inference. Vectorized by: Note Cutoff level: 0.15 Unfortunately, almost all of the results sound very similar to this, which makes sharing a broad range of results somewhat redundant.

7 References i Gatys, L., Ecker, A., & Bethge, M. (2016). A Neural Algorithm of Artistic Style. Journal Of Vision, 16(12), ii Hochreiter, Sepp, & Schmidhuber, Jürgen Long short-term memory. Neural Comput. 9, 9 (November 1997), iii Kim, Ji-Sung. (April 2016). DeepJazz. Github Repository. v Ulyanov, D., & Lebedev, D. (2017). Audio texture synthesis and style transfer. Dmitry Ulyanov. iv Google. Project Magenta (2017). vi Cuthbert, Michael, et al. (2017). MIT. Music21 Python Library. vii Krueger, B. (2017). Classical Piano Midi Page. Piano-midi.de. Retrieved 17 April 2017, from viii Weimar Jazz Database. (2017). Weimar Jazz Database. Retrieved 17 April 2017, from jazzomat.hfmweimar.de/dbformat/dbcontent.html ix Kingma, D. & Ba, J. (2015). Adam: A Method for Stochastic Gradient Optimization. Published as a conference paper at ICLR arxiv:

arxiv: v1 [cs.lg] 15 Jun 2016

arxiv: v1 [cs.lg] 15 Jun 2016 Deep Learning for Music arxiv:1606.04930v1 [cs.lg] 15 Jun 2016 Allen Huang Department of Management Science and Engineering Stanford University allenh@cs.stanford.edu Abstract Raymond Wu Department of

More information

Music Composition with RNN

Music Composition with RNN Music Composition with RNN Jason Wang Department of Statistics Stanford University zwang01@stanford.edu Abstract Music composition is an interesting problem that tests the creativity capacities of artificial

More information

Melody Extraction from Generic Audio Clips Thaminda Edirisooriya, Hansohl Kim, Connie Zeng

Melody Extraction from Generic Audio Clips Thaminda Edirisooriya, Hansohl Kim, Connie Zeng Melody Extraction from Generic Audio Clips Thaminda Edirisooriya, Hansohl Kim, Connie Zeng Introduction In this project we were interested in extracting the melody from generic audio files. Due to the

More information

An AI Approach to Automatic Natural Music Transcription

An AI Approach to Automatic Natural Music Transcription An AI Approach to Automatic Natural Music Transcription Michael Bereket Stanford University Stanford, CA mbereket@stanford.edu Karey Shi Stanford Univeristy Stanford, CA kareyshi@stanford.edu Abstract

More information

Noise (Music) Composition Using Classification Algorithms Peter Wang (pwang01) December 15, 2017

Noise (Music) Composition Using Classification Algorithms Peter Wang (pwang01) December 15, 2017 Noise (Music) Composition Using Classification Algorithms Peter Wang (pwang01) December 15, 2017 Background Abstract I attempted a solution at using machine learning to compose music given a large corpus

More information

Neural Network for Music Instrument Identi cation

Neural Network for Music Instrument Identi cation Neural Network for Music Instrument Identi cation Zhiwen Zhang(MSE), Hanze Tu(CCRMA), Yuan Li(CCRMA) SUN ID: zhiwen, hanze, yuanli92 Abstract - In the context of music, instrument identi cation would contribute

More information

Generating Music with Recurrent Neural Networks

Generating Music with Recurrent Neural Networks Generating Music with Recurrent Neural Networks 27 October 2017 Ushini Attanayake Supervised by Christian Walder Co-supervised by Henry Gardner COMP3740 Project Work in Computing The Australian National

More information

Deep Jammer: A Music Generation Model

Deep Jammer: A Music Generation Model Deep Jammer: A Music Generation Model Justin Svegliato and Sam Witty College of Information and Computer Sciences University of Massachusetts Amherst, MA 01003, USA {jsvegliato,switty}@cs.umass.edu Abstract

More information

Image-to-Markup Generation with Coarse-to-Fine Attention

Image-to-Markup Generation with Coarse-to-Fine Attention Image-to-Markup Generation with Coarse-to-Fine Attention Presenter: Ceyer Wakilpoor Yuntian Deng 1 Anssi Kanervisto 2 Alexander M. Rush 1 Harvard University 3 University of Eastern Finland ICML, 2017 Yuntian

More information

A STUDY ON LSTM NETWORKS FOR POLYPHONIC MUSIC SEQUENCE MODELLING

A STUDY ON LSTM NETWORKS FOR POLYPHONIC MUSIC SEQUENCE MODELLING A STUDY ON LSTM NETWORKS FOR POLYPHONIC MUSIC SEQUENCE MODELLING Adrien Ycart and Emmanouil Benetos Centre for Digital Music, Queen Mary University of London, UK {a.ycart, emmanouil.benetos}@qmul.ac.uk

More information

RoboMozart: Generating music using LSTM networks trained per-tick on a MIDI collection with short music segments as input.

RoboMozart: Generating music using LSTM networks trained per-tick on a MIDI collection with short music segments as input. RoboMozart: Generating music using LSTM networks trained per-tick on a MIDI collection with short music segments as input. Joseph Weel 10321624 Bachelor thesis Credits: 18 EC Bachelor Opleiding Kunstmatige

More information

Audio: Generation & Extraction. Charu Jaiswal

Audio: Generation & Extraction. Charu Jaiswal Audio: Generation & Extraction Charu Jaiswal Music Composition which approach? Feed forward NN can t store information about past (or keep track of position in song) RNN as a single step predictor struggle

More information

Bach2Bach: Generating Music Using A Deep Reinforcement Learning Approach Nikhil Kotecha Columbia University

Bach2Bach: Generating Music Using A Deep Reinforcement Learning Approach Nikhil Kotecha Columbia University Bach2Bach: Generating Music Using A Deep Reinforcement Learning Approach Nikhil Kotecha Columbia University Abstract A model of music needs to have the ability to recall past details and have a clear,

More information

Blues Improviser. Greg Nelson Nam Nguyen

Blues Improviser. Greg Nelson Nam Nguyen Blues Improviser Greg Nelson (gregoryn@cs.utah.edu) Nam Nguyen (namphuon@cs.utah.edu) Department of Computer Science University of Utah Salt Lake City, UT 84112 Abstract Computer-generated music has long

More information

CHORD GENERATION FROM SYMBOLIC MELODY USING BLSTM NETWORKS

CHORD GENERATION FROM SYMBOLIC MELODY USING BLSTM NETWORKS CHORD GENERATION FROM SYMBOLIC MELODY USING BLSTM NETWORKS Hyungui Lim 1,2, Seungyeon Rhyu 1 and Kyogu Lee 1,2 3 Music and Audio Research Group, Graduate School of Convergence Science and Technology 4

More information

Bach-Prop: Modeling Bach s Harmonization Style with a Back- Propagation Network

Bach-Prop: Modeling Bach s Harmonization Style with a Back- Propagation Network Indiana Undergraduate Journal of Cognitive Science 1 (2006) 3-14 Copyright 2006 IUJCS. All rights reserved Bach-Prop: Modeling Bach s Harmonization Style with a Back- Propagation Network Rob Meyerson Cognitive

More information

Building a Better Bach with Markov Chains

Building a Better Bach with Markov Chains Building a Better Bach with Markov Chains CS701 Implementation Project, Timothy Crocker December 18, 2015 1 Abstract For my implementation project, I explored the field of algorithmic music composition

More information

Laughbot: Detecting Humor in Spoken Language with Language and Audio Cues

Laughbot: Detecting Humor in Spoken Language with Language and Audio Cues Laughbot: Detecting Humor in Spoken Language with Language and Audio Cues Kate Park, Annie Hu, Natalie Muenster Email: katepark@stanford.edu, anniehu@stanford.edu, ncm000@stanford.edu Abstract We propose

More information

LEARNING AUDIO SHEET MUSIC CORRESPONDENCES. Matthias Dorfer Department of Computational Perception

LEARNING AUDIO SHEET MUSIC CORRESPONDENCES. Matthias Dorfer Department of Computational Perception LEARNING AUDIO SHEET MUSIC CORRESPONDENCES Matthias Dorfer Department of Computational Perception Short Introduction... I am a PhD Candidate in the Department of Computational Perception at Johannes Kepler

More information

arxiv: v1 [cs.sd] 8 Jun 2016

arxiv: v1 [cs.sd] 8 Jun 2016 Symbolic Music Data Version 1. arxiv:1.5v1 [cs.sd] 8 Jun 1 Christian Walder CSIRO Data1 7 London Circuit, Canberra,, Australia. christian.walder@data1.csiro.au June 9, 1 Abstract In this document, we introduce

More information

Singer Traits Identification using Deep Neural Network

Singer Traits Identification using Deep Neural Network Singer Traits Identification using Deep Neural Network Zhengshan Shi Center for Computer Research in Music and Acoustics Stanford University kittyshi@stanford.edu Abstract The author investigates automatic

More information

Detecting Musical Key with Supervised Learning

Detecting Musical Key with Supervised Learning Detecting Musical Key with Supervised Learning Robert Mahieu Department of Electrical Engineering Stanford University rmahieu@stanford.edu Abstract This paper proposes and tests performance of two different

More information

Audio spectrogram representations for processing with Convolutional Neural Networks

Audio spectrogram representations for processing with Convolutional Neural Networks Audio spectrogram representations for processing with Convolutional Neural Networks Lonce Wyse 1 1 National University of Singapore arxiv:1706.09559v1 [cs.sd] 29 Jun 2017 One of the decisions that arise

More information

Chord Classification of an Audio Signal using Artificial Neural Network

Chord Classification of an Audio Signal using Artificial Neural Network Chord Classification of an Audio Signal using Artificial Neural Network Ronesh Shrestha Student, Department of Electrical and Electronic Engineering, Kathmandu University, Dhulikhel, Nepal ---------------------------------------------------------------------***---------------------------------------------------------------------

More information

OPTICAL MUSIC RECOGNITION WITH CONVOLUTIONAL SEQUENCE-TO-SEQUENCE MODELS

OPTICAL MUSIC RECOGNITION WITH CONVOLUTIONAL SEQUENCE-TO-SEQUENCE MODELS OPTICAL MUSIC RECOGNITION WITH CONVOLUTIONAL SEQUENCE-TO-SEQUENCE MODELS First Author Affiliation1 author1@ismir.edu Second Author Retain these fake authors in submission to preserve the formatting Third

More information

Finding Sarcasm in Reddit Postings: A Deep Learning Approach

Finding Sarcasm in Reddit Postings: A Deep Learning Approach Finding Sarcasm in Reddit Postings: A Deep Learning Approach Nick Guo, Ruchir Shah {nickguo, ruchirfs}@stanford.edu Abstract We use the recently published Self-Annotated Reddit Corpus (SARC) with a recurrent

More information

Learning Musical Structure Directly from Sequences of Music

Learning Musical Structure Directly from Sequences of Music Learning Musical Structure Directly from Sequences of Music Douglas Eck and Jasmin Lapalme Dept. IRO, Université de Montréal C.P. 6128, Montreal, Qc, H3C 3J7, Canada Technical Report 1300 Abstract This

More information

Jazz Melody Generation from Recurrent Network Learning of Several Human Melodies

Jazz Melody Generation from Recurrent Network Learning of Several Human Melodies Jazz Melody Generation from Recurrent Network Learning of Several Human Melodies Judy Franklin Computer Science Department Smith College Northampton, MA 01063 Abstract Recurrent (neural) networks have

More information

Rewind: A Music Transcription Method

Rewind: A Music Transcription Method University of Nevada, Reno Rewind: A Music Transcription Method A thesis submitted in partial fulfillment of the requirements for the degree of Master of Science in Computer Science and Engineering by

More information

Algorithmic Music Composition using Recurrent Neural Networking

Algorithmic Music Composition using Recurrent Neural Networking Algorithmic Music Composition using Recurrent Neural Networking Kai-Chieh Huang kaichieh@stanford.edu Dept. of Electrical Engineering Quinlan Jung quinlanj@stanford.edu Dept. of Computer Science Jennifer

More information

Laughbot: Detecting Humor in Spoken Language with Language and Audio Cues

Laughbot: Detecting Humor in Spoken Language with Language and Audio Cues Laughbot: Detecting Humor in Spoken Language with Language and Audio Cues Kate Park katepark@stanford.edu Annie Hu anniehu@stanford.edu Natalie Muenster ncm000@stanford.edu Abstract We propose detecting

More information

Music Genre Classification and Variance Comparison on Number of Genres

Music Genre Classification and Variance Comparison on Number of Genres Music Genre Classification and Variance Comparison on Number of Genres Miguel Francisco, miguelf@stanford.edu Dong Myung Kim, dmk8265@stanford.edu 1 Abstract In this project we apply machine learning techniques

More information

Evaluating Melodic Encodings for Use in Cover Song Identification

Evaluating Melodic Encodings for Use in Cover Song Identification Evaluating Melodic Encodings for Use in Cover Song Identification David D. Wickland wickland@uoguelph.ca David A. Calvert dcalvert@uoguelph.ca James Harley jharley@uoguelph.ca ABSTRACT Cover song identification

More information

Music Genre Classification

Music Genre Classification Music Genre Classification chunya25 Fall 2017 1 Introduction A genre is defined as a category of artistic composition, characterized by similarities in form, style, or subject matter. [1] Some researchers

More information

Predicting the immediate future with Recurrent Neural Networks: Pre-training and Applications

Predicting the immediate future with Recurrent Neural Networks: Pre-training and Applications Predicting the immediate future with Recurrent Neural Networks: Pre-training and Applications Introduction Brandon Richardson December 16, 2011 Research preformed from the last 5 years has shown that the

More information

BachBot: Automatic composition in the style of Bach chorales

BachBot: Automatic composition in the style of Bach chorales BachBot: Automatic composition in the style of Bach chorales Developing, analyzing, and evaluating a deep LSTM model for musical style Feynman Liang Department of Engineering University of Cambridge M.Phil

More information

CPU Bach: An Automatic Chorale Harmonization System

CPU Bach: An Automatic Chorale Harmonization System CPU Bach: An Automatic Chorale Harmonization System Matt Hanlon mhanlon@fas Tim Ledlie ledlie@fas January 15, 2002 Abstract We present an automated system for the harmonization of fourpart chorales in

More information

Chord Label Personalization through Deep Learning of Integrated Harmonic Interval-based Representations

Chord Label Personalization through Deep Learning of Integrated Harmonic Interval-based Representations Chord Label Personalization through Deep Learning of Integrated Harmonic Interval-based Representations Hendrik Vincent Koops 1, W. Bas de Haas 2, Jeroen Bransen 2, and Anja Volk 1 arxiv:1706.09552v1 [cs.sd]

More information

Hidden Markov Model based dance recognition

Hidden Markov Model based dance recognition Hidden Markov Model based dance recognition Dragutin Hrenek, Nenad Mikša, Robert Perica, Pavle Prentašić and Boris Trubić University of Zagreb, Faculty of Electrical Engineering and Computing Unska 3,

More information

Take a Break, Bach! Let Machine Learning Harmonize That Chorale For You. Chris Lewis Stanford University

Take a Break, Bach! Let Machine Learning Harmonize That Chorale For You. Chris Lewis Stanford University Take a Break, Bach! Let Machine Learning Harmonize That Chorale For You Chris Lewis Stanford University cmslewis@stanford.edu Abstract In this project, I explore the effectiveness of the Naive Bayes Classifier

More information

arxiv: v1 [cs.cv] 16 Jul 2017

arxiv: v1 [cs.cv] 16 Jul 2017 OPTICAL MUSIC RECOGNITION WITH CONVOLUTIONAL SEQUENCE-TO-SEQUENCE MODELS Eelco van der Wel University of Amsterdam eelcovdw@gmail.com Karen Ullrich University of Amsterdam karen.ullrich@uva.nl arxiv:1707.04877v1

More information

Music Information Retrieval with Temporal Features and Timbre

Music Information Retrieval with Temporal Features and Timbre Music Information Retrieval with Temporal Features and Timbre Angelina A. Tzacheva and Keith J. Bell University of South Carolina Upstate, Department of Informatics 800 University Way, Spartanburg, SC

More information

Experiments on musical instrument separation using multiplecause

Experiments on musical instrument separation using multiplecause Experiments on musical instrument separation using multiplecause models J Klingseisen and M D Plumbley* Department of Electronic Engineering King's College London * - Corresponding Author - mark.plumbley@kcl.ac.uk

More information

Automated sound generation based on image colour spectrum with using the recurrent neural network

Automated sound generation based on image colour spectrum with using the recurrent neural network Automated sound generation based on image colour spectrum with using the recurrent neural network N A Nikitin 1, V L Rozaliev 1, Yu A Orlova 1 and A V Alekseev 1 1 Volgograd State Technical University,

More information

arxiv: v3 [cs.sd] 14 Jul 2017

arxiv: v3 [cs.sd] 14 Jul 2017 Music Generation with Variational Recurrent Autoencoder Supported by History Alexey Tikhonov 1 and Ivan P. Yamshchikov 2 1 Yandex, Berlin altsoph@gmail.com 2 Max Planck Institute for Mathematics in the

More information

Outline. Why do we classify? Audio Classification

Outline. Why do we classify? Audio Classification Outline Introduction Music Information Retrieval Classification Process Steps Pitch Histograms Multiple Pitch Detection Algorithm Musical Genre Classification Implementation Future Work Why do we classify

More information

arxiv: v1 [cs.sd] 9 Dec 2017

arxiv: v1 [cs.sd] 9 Dec 2017 Music Generation by Deep Learning Challenges and Directions Jean-Pierre Briot François Pachet Sorbonne Universités, UPMC Univ Paris 06, CNRS, LIP6, Paris, France Jean-Pierre.Briot@lip6.fr Spotify Creator

More information

CREATING all forms of art [1], [2], [3], [4], including

CREATING all forms of art [1], [2], [3], [4], including Grammar Argumented LSTM Neural Networks with Note-Level Encoding for Music Composition Zheng Sun, Jiaqi Liu, Zewang Zhang, Jingwen Chen, Zhao Huo, Ching Hua Lee, and Xiao Zhang 1 arxiv:1611.05416v1 [cs.lg]

More information

JazzGAN: Improvising with Generative Adversarial Networks

JazzGAN: Improvising with Generative Adversarial Networks JazzGAN: Improvising with Generative Adversarial Networks Nicholas Trieu and Robert M. Keller Harvey Mudd College Claremont, California, USA ntrieu@hmc.edu, keller@cs.hmc.edu Abstract For the purpose of

More information

Towards End-to-End Raw Audio Music Synthesis

Towards End-to-End Raw Audio Music Synthesis To be published in: Proceedings of the 27th Conference on Artificial Neural Networks (ICANN), Rhodes, Greece, 2018. (Author s Preprint) Towards End-to-End Raw Audio Music Synthesis Manfred Eppe, Tayfun

More information

POST-PROCESSING FIDDLE : A REAL-TIME MULTI-PITCH TRACKING TECHNIQUE USING HARMONIC PARTIAL SUBTRACTION FOR USE WITHIN LIVE PERFORMANCE SYSTEMS

POST-PROCESSING FIDDLE : A REAL-TIME MULTI-PITCH TRACKING TECHNIQUE USING HARMONIC PARTIAL SUBTRACTION FOR USE WITHIN LIVE PERFORMANCE SYSTEMS POST-PROCESSING FIDDLE : A REAL-TIME MULTI-PITCH TRACKING TECHNIQUE USING HARMONIC PARTIAL SUBTRACTION FOR USE WITHIN LIVE PERFORMANCE SYSTEMS Andrew N. Robertson, Mark D. Plumbley Centre for Digital Music

More information

SentiMozart: Music Generation based on Emotions

SentiMozart: Music Generation based on Emotions SentiMozart: Music Generation based on Emotions Rishi Madhok 1,, Shivali Goel 2, and Shweta Garg 1, 1 Department of Computer Science and Engineering, Delhi Technological University, New Delhi, India 2

More information

Effects of acoustic degradations on cover song recognition

Effects of acoustic degradations on cover song recognition Signal Processing in Acoustics: Paper 68 Effects of acoustic degradations on cover song recognition Julien Osmalskyj (a), Jean-Jacques Embrechts (b) (a) University of Liège, Belgium, josmalsky@ulg.ac.be

More information

Music Generation from MIDI datasets

Music Generation from MIDI datasets Music Generation from MIDI datasets Moritz Hilscher, Novin Shahroudi 2 Institute of Computer Science, University of Tartu moritz.hilscher@student.hpi.de, 2 novin@ut.ee Abstract. Many approaches are being

More information

Feature-Based Analysis of Haydn String Quartets

Feature-Based Analysis of Haydn String Quartets Feature-Based Analysis of Haydn String Quartets Lawson Wong 5/5/2 Introduction When listening to multi-movement works, amateur listeners have almost certainly asked the following situation : Am I still

More information

Improvised Duet Interaction: Learning Improvisation Techniques for Automatic Accompaniment

Improvised Duet Interaction: Learning Improvisation Techniques for Automatic Accompaniment Improvised Duet Interaction: Learning Improvisation Techniques for Automatic Accompaniment Gus G. Xia Dartmouth College Neukom Institute Hanover, NH, USA gxia@dartmouth.edu Roger B. Dannenberg Carnegie

More information

A probabilistic approach to determining bass voice leading in melodic harmonisation

A probabilistic approach to determining bass voice leading in melodic harmonisation A probabilistic approach to determining bass voice leading in melodic harmonisation Dimos Makris a, Maximos Kaliakatsos-Papakostas b, and Emilios Cambouropoulos b a Department of Informatics, Ionian University,

More information

Using Deep Learning to Annotate Karaoke Songs

Using Deep Learning to Annotate Karaoke Songs Distributed Computing Using Deep Learning to Annotate Karaoke Songs Semester Thesis Juliette Faille faillej@student.ethz.ch Distributed Computing Group Computer Engineering and Networks Laboratory ETH

More information

Sequence generation and classification with VAEs and RNNs

Sequence generation and classification with VAEs and RNNs Jay Hennig 1 * Akash Umakantha 1 * Ryan Williamson 1 * 1. Introduction Variational autoencoders (VAEs) (Kingma & Welling, 2013) are a popular approach for performing unsupervised learning that can also

More information

Computational Modelling of Harmony

Computational Modelling of Harmony Computational Modelling of Harmony Simon Dixon Centre for Digital Music, Queen Mary University of London, Mile End Rd, London E1 4NS, UK simon.dixon@elec.qmul.ac.uk http://www.elec.qmul.ac.uk/people/simond

More information

Recurrent Neural Networks and Pitch Representations for Music Tasks

Recurrent Neural Networks and Pitch Representations for Music Tasks Recurrent Neural Networks and Pitch Representations for Music Tasks Judy A. Franklin Smith College Department of Computer Science Northampton, MA 01063 jfranklin@cs.smith.edu Abstract We present results

More information

A Transfer Learning Based Feature Extractor for Polyphonic Sound Event Detection Using Connectionist Temporal Classification

A Transfer Learning Based Feature Extractor for Polyphonic Sound Event Detection Using Connectionist Temporal Classification INTERSPEECH 17 August, 17, Stockholm, Sweden A Transfer Learning Based Feature Extractor for Polyphonic Sound Event Detection Using Connectionist Temporal Classification Yun Wang and Florian Metze Language

More information

Algorithmic Composition: The Music of Mathematics

Algorithmic Composition: The Music of Mathematics Algorithmic Composition: The Music of Mathematics Carlo J. Anselmo 18 and Marcus Pendergrass Department of Mathematics, Hampden-Sydney College, Hampden-Sydney, VA 23943 ABSTRACT We report on several techniques

More information

Modeling Temporal Tonal Relations in Polyphonic Music Through Deep Networks with a Novel Image-Based Representation

Modeling Temporal Tonal Relations in Polyphonic Music Through Deep Networks with a Novel Image-Based Representation INTRODUCTION Modeling Temporal Tonal Relations in Polyphonic Music Through Deep Networks with a Novel Image-Based Representation Ching-Hua Chuan 1, 2 1 University of North Florida 2 University of Miami

More information

Various Artificial Intelligence Techniques For Automated Melody Generation

Various Artificial Intelligence Techniques For Automated Melody Generation Various Artificial Intelligence Techniques For Automated Melody Generation Nikahat Kazi Computer Engineering Department, Thadomal Shahani Engineering College, Mumbai, India Shalini Bhatia Assistant Professor,

More information

Composing a melody with long-short term memory (LSTM) Recurrent Neural Networks. Konstantin Lackner

Composing a melody with long-short term memory (LSTM) Recurrent Neural Networks. Konstantin Lackner Composing a melody with long-short term memory (LSTM) Recurrent Neural Networks Konstantin Lackner Bachelor s thesis Composing a melody with long-short term memory (LSTM) Recurrent Neural Networks Konstantin

More information

The Sparsity of Simple Recurrent Networks in Musical Structure Learning

The Sparsity of Simple Recurrent Networks in Musical Structure Learning The Sparsity of Simple Recurrent Networks in Musical Structure Learning Kat R. Agres (kra9@cornell.edu) Department of Psychology, Cornell University, 211 Uris Hall Ithaca, NY 14853 USA Jordan E. DeLong

More information

Automatic Piano Music Transcription

Automatic Piano Music Transcription Automatic Piano Music Transcription Jianyu Fan Qiuhan Wang Xin Li Jianyu.Fan.Gr@dartmouth.edu Qiuhan.Wang.Gr@dartmouth.edu Xi.Li.Gr@dartmouth.edu 1. Introduction Writing down the score while listening

More information

Representations of Sound in Deep Learning of Audio Features from Music

Representations of Sound in Deep Learning of Audio Features from Music Representations of Sound in Deep Learning of Audio Features from Music Sergey Shuvaev, Hamza Giaffar, and Alexei A. Koulakov Cold Spring Harbor Laboratory, Cold Spring Harbor, NY Abstract The work of a

More information

AUTOMATIC ACCOMPANIMENT OF VOCAL MELODIES IN THE CONTEXT OF POPULAR MUSIC

AUTOMATIC ACCOMPANIMENT OF VOCAL MELODIES IN THE CONTEXT OF POPULAR MUSIC AUTOMATIC ACCOMPANIMENT OF VOCAL MELODIES IN THE CONTEXT OF POPULAR MUSIC A Thesis Presented to The Academic Faculty by Xiang Cao In Partial Fulfillment of the Requirements for the Degree Master of Science

More information

Music Composition Using Recurrent Neural Networks and Evolutionary Algorithms

Music Composition Using Recurrent Neural Networks and Evolutionary Algorithms Music Composition Using Recurrent Neural Networks and Evolutionary Algorithms Calvin Pelletier Department of Electrical and Computer Engineering University of Illinois at Urbana-Champaign May 2017 Abstract

More information

Analysis of local and global timing and pitch change in ordinary

Analysis of local and global timing and pitch change in ordinary Alma Mater Studiorum University of Bologna, August -6 6 Analysis of local and global timing and pitch change in ordinary melodies Roger Watt Dept. of Psychology, University of Stirling, Scotland r.j.watt@stirling.ac.uk

More information

SIMSSA DB: A Database for Computational Musicological Research

SIMSSA DB: A Database for Computational Musicological Research SIMSSA DB: A Database for Computational Musicological Research Cory McKay Marianopolis College 2018 International Association of Music Libraries, Archives and Documentation Centres International Congress,

More information

Deep Recurrent Music Writer: Memory-enhanced Variational Autoencoder-based Musical Score Composition and an Objective Measure

Deep Recurrent Music Writer: Memory-enhanced Variational Autoencoder-based Musical Score Composition and an Objective Measure Deep Recurrent Music Writer: Memory-enhanced Variational Autoencoder-based Musical Score Composition and an Objective Measure Romain Sabathé, Eduardo Coutinho, and Björn Schuller Department of Computing,

More information

Chord Representations for Probabilistic Models

Chord Representations for Probabilistic Models R E S E A R C H R E P O R T I D I A P Chord Representations for Probabilistic Models Jean-François Paiement a Douglas Eck b Samy Bengio a IDIAP RR 05-58 September 2005 soumis à publication a b IDIAP Research

More information

Real-valued parametric conditioning of an RNN for interactive sound synthesis

Real-valued parametric conditioning of an RNN for interactive sound synthesis Real-valued parametric conditioning of an RNN for interactive sound synthesis Lonce Wyse Communications and New Media Department National University of Singapore Singapore lonce.acad@zwhome.org Abstract

More information

arxiv: v1 [cs.sd] 17 Dec 2018

arxiv: v1 [cs.sd] 17 Dec 2018 Learning to Generate Music with BachProp Florian Colombo School of Computer Science and School of Life Sciences École Polytechnique Fédérale de Lausanne, Lausanne, Switzerland florian.colombo@epfl.ch arxiv:1812.06669v1

More information

arxiv: v1 [cs.lg] 16 Dec 2017

arxiv: v1 [cs.lg] 16 Dec 2017 AUTOMATIC MUSIC HIGHLIGHT EXTRACTION USING CONVOLUTIONAL RECURRENT ATTENTION NETWORKS Jung-Woo Ha 1, Adrian Kim 1,2, Chanju Kim 2, Jangyeon Park 2, and Sung Kim 1,3 1 Clova AI Research and 2 Clova Music,

More information

1) New Paths to New Machine Learning Science. 2) How an Unruly Mob Almost Stole. Jeff Howbert University of Washington

1) New Paths to New Machine Learning Science. 2) How an Unruly Mob Almost Stole. Jeff Howbert University of Washington 1) New Paths to New Machine Learning Science 2) How an Unruly Mob Almost Stole the Grand Prize at the Last Moment Jeff Howbert University of Washington February 4, 2014 Netflix Viewing Recommendations

More information

Deep learning for music data processing

Deep learning for music data processing Deep learning for music data processing A personal (re)view of the state-of-the-art Jordi Pons www.jordipons.me Music Technology Group, DTIC, Universitat Pompeu Fabra, Barcelona. 31st January 2017 Jordi

More information

EVALUATING LANGUAGE MODELS OF TONAL HARMONY

EVALUATING LANGUAGE MODELS OF TONAL HARMONY EVALUATING LANGUAGE MODELS OF TONAL HARMONY David R. W. Sears 1 Filip Korzeniowski 2 Gerhard Widmer 2 1 College of Visual & Performing Arts, Texas Tech University, Lubbock, USA 2 Institute of Computational

More information

COMPARING RNN PARAMETERS FOR MELODIC SIMILARITY

COMPARING RNN PARAMETERS FOR MELODIC SIMILARITY COMPARING RNN PARAMETERS FOR MELODIC SIMILARITY Tian Cheng, Satoru Fukayama, Masataka Goto National Institute of Advanced Industrial Science and Technology (AIST), Japan {tian.cheng, s.fukayama, m.goto}@aist.go.jp

More information

Music Similarity and Cover Song Identification: The Case of Jazz

Music Similarity and Cover Song Identification: The Case of Jazz Music Similarity and Cover Song Identification: The Case of Jazz Simon Dixon and Peter Foster s.e.dixon@qmul.ac.uk Centre for Digital Music School of Electronic Engineering and Computer Science Queen Mary

More information

Music Radar: A Web-based Query by Humming System

Music Radar: A Web-based Query by Humming System Music Radar: A Web-based Query by Humming System Lianjie Cao, Peng Hao, Chunmeng Zhou Computer Science Department, Purdue University, 305 N. University Street West Lafayette, IN 47907-2107 {cao62, pengh,

More information

Sudhanshu Gautam *1, Sarita Soni 2. M-Tech Computer Science, BBAU Central University, Lucknow, Uttar Pradesh, India

Sudhanshu Gautam *1, Sarita Soni 2. M-Tech Computer Science, BBAU Central University, Lucknow, Uttar Pradesh, India International Journal of Scientific Research in Computer Science, Engineering and Information Technology 2018 IJSRCSEIT Volume 3 Issue 3 ISSN : 2456-3307 Artificial Intelligence Techniques for Music Composition

More information

Singer Recognition and Modeling Singer Error

Singer Recognition and Modeling Singer Error Singer Recognition and Modeling Singer Error Johan Ismael Stanford University jismael@stanford.edu Nicholas McGee Stanford University ndmcgee@stanford.edu 1. Abstract We propose a system for recognizing

More information

A QUERY BY EXAMPLE MUSIC RETRIEVAL ALGORITHM

A QUERY BY EXAMPLE MUSIC RETRIEVAL ALGORITHM A QUER B EAMPLE MUSIC RETRIEVAL ALGORITHM H. HARB AND L. CHEN Maths-Info department, Ecole Centrale de Lyon. 36, av. Guy de Collongue, 69134, Ecully, France, EUROPE E-mail: {hadi.harb, liming.chen}@ec-lyon.fr

More information

MUSICAL INSTRUMENT RECOGNITION WITH WAVELET ENVELOPES

MUSICAL INSTRUMENT RECOGNITION WITH WAVELET ENVELOPES MUSICAL INSTRUMENT RECOGNITION WITH WAVELET ENVELOPES PACS: 43.60.Lq Hacihabiboglu, Huseyin 1,2 ; Canagarajah C. Nishan 2 1 Sonic Arts Research Centre (SARC) School of Computer Science Queen s University

More information

Algorithmic Music Composition

Algorithmic Music Composition Algorithmic Music Composition MUS-15 Jan Dreier July 6, 2015 1 Introduction The goal of algorithmic music composition is to automate the process of creating music. One wants to create pleasant music without

More information

Perceptual Evaluation of Automatically Extracted Musical Motives

Perceptual Evaluation of Automatically Extracted Musical Motives Perceptual Evaluation of Automatically Extracted Musical Motives Oriol Nieto 1, Morwaread M. Farbood 2 Dept. of Music and Performing Arts Professions, New York University, USA 1 oriol@nyu.edu, 2 mfarbood@nyu.edu

More information

Singing voice synthesis based on deep neural networks

Singing voice synthesis based on deep neural networks INTERSPEECH 2016 September 8 12, 2016, San Francisco, USA Singing voice synthesis based on deep neural networks Masanari Nishimura, Kei Hashimoto, Keiichiro Oura, Yoshihiko Nankaku, and Keiichi Tokuda

More information

A Discriminative Approach to Topic-based Citation Recommendation

A Discriminative Approach to Topic-based Citation Recommendation A Discriminative Approach to Topic-based Citation Recommendation Jie Tang and Jing Zhang Department of Computer Science and Technology, Tsinghua University, Beijing, 100084. China jietang@tsinghua.edu.cn,zhangjing@keg.cs.tsinghua.edu.cn

More information

DJ Darwin a genetic approach to creating beats

DJ Darwin a genetic approach to creating beats Assaf Nir DJ Darwin a genetic approach to creating beats Final project report, course 67842 'Introduction to Artificial Intelligence' Abstract In this document we present two applications that incorporate

More information

Topic 10. Multi-pitch Analysis

Topic 10. Multi-pitch Analysis Topic 10 Multi-pitch Analysis What is pitch? Common elements of music are pitch, rhythm, dynamics, and the sonic qualities of timbre and texture. An auditory perceptual attribute in terms of which sounds

More information

CONDITIONING DEEP GENERATIVE RAW AUDIO MODELS FOR STRUCTURED AUTOMATIC MUSIC

CONDITIONING DEEP GENERATIVE RAW AUDIO MODELS FOR STRUCTURED AUTOMATIC MUSIC CONDITIONING DEEP GENERATIVE RAW AUDIO MODELS FOR STRUCTURED AUTOMATIC MUSIC Rachel Manzelli Vijay Thakkar Ali Siahkamari Brian Kulis Equal contributions ECE Department, Boston University {manzelli, thakkarv,

More information

CS229 Project Report Polyphonic Piano Transcription

CS229 Project Report Polyphonic Piano Transcription CS229 Project Report Polyphonic Piano Transcription Mohammad Sadegh Ebrahimi Stanford University Jean-Baptiste Boin Stanford University sadegh@stanford.edu jbboin@stanford.edu 1. Introduction In this project

More information

Robert Alexandru Dobre, Cristian Negrescu

Robert Alexandru Dobre, Cristian Negrescu ECAI 2016 - International Conference 8th Edition Electronics, Computers and Artificial Intelligence 30 June -02 July, 2016, Ploiesti, ROMÂNIA Automatic Music Transcription Software Based on Constant Q

More information

MUSI-6201 Computational Music Analysis

MUSI-6201 Computational Music Analysis MUSI-6201 Computational Music Analysis Part 9.1: Genre Classification alexander lerch November 4, 2015 temporal analysis overview text book Chapter 8: Musical Genre, Similarity, and Mood (pp. 151 155)

More information

A FUNCTIONAL CLASSIFICATION OF ONE INSTRUMENT S TIMBRES

A FUNCTIONAL CLASSIFICATION OF ONE INSTRUMENT S TIMBRES A FUNCTIONAL CLASSIFICATION OF ONE INSTRUMENT S TIMBRES Panayiotis Kokoras School of Music Studies Aristotle University of Thessaloniki email@panayiotiskokoras.com Abstract. This article proposes a theoretical

More information

Jazz Melody Generation and Recognition

Jazz Melody Generation and Recognition Jazz Melody Generation and Recognition Joseph Victor December 14, 2012 Introduction In this project, we attempt to use machine learning methods to study jazz solos. The reason we study jazz in particular

More information