A QUERY BY HUMMING SYSTEM THAT LEARNS FROM EXPERIENCE

A QUERY BY HUMMING SYSTEM THAT LEARNS FROM EXPERIENCE David Little, David Raffenspege, Byan Pado EECS Depatment Nothwesten Univesity Evanston, IL 60201 d-little,d-affenspege,pado@nothwesten.edu ABSTRACT Quey-by-Humming (QBH) systems tanscibe a sung o hummed quey and seach fo elated musical themes in a database, etuning the most simila themes. Since it is not possible to pedict all individual singe pofiles befoe system deployment, a obust QBH system should be able to adapt to diffeent singes afte deployment. Cuently deployed systems do not have this capability. We descibe a new QBH system that leans fom use povided feedback on the seach esults, letting the system impove while deployed, afte only a few queies. This is made possible by a tainable note segmentation system, an easily paameteized singe eo model and a staight-fowad genetic algoithm. Results show significant impovement in pefomance given only ten example queies fom a paticula use. 1. INTRODUCTION Deployed seach engines used to find music documents, such as amazon.com, ely on metadata about the song title and pefome name as thei indexing mechanism. Often, a peson is able to sing a potion of the piece, but cannot specify the title, compose o pefome. Quey by humming (QBH) systems [1] solve this mismatch between database keys and use knowledge by matching a sung quey to musical themes in a database, etuning the most simila themes. One of the main difficulties in building an effective QBH system is dealing with the vaiation between sung queies and the melodies used as database seach keys. Singes may go out of tune, sing at a diffeent tempo than expected, o in a diffeent key [1, 7]. Futhe, singes diffe in thei eo pofiles. One may have poo pitch, while anothe has poo hythm. Since it is not possible to pedict all individual singe pofiles befoe deployment, a obust QBH system should be able to adapt to diffeent singes afte deployment. Cuent QBH systems do not have this capability. While thee has been significant pio wok that addesses (o is applicable to) singe eo modelling [7, 9, 11] fo QBH, eseaches have not focused on fully automated, ongoing QBH optimization afte deployment. Thus, these appoaches ae unsuited fo this task, equiing eithe hundeds of example queies to customize to an individual [7, 9], o taining examples whee the intenal stuctue of each quey is aligned by the taine to the stuctue of the taget [11]. Figue 1. System diagam We ae developing a QBH system (Figue 1) that pesonalizes a singe model based on use feedback, leaning the model on-line, afte deployment without intevention fom the system developes and afte only a few example queies. The use sings a quey (step 1 in the figue). The system etuns a list of songs fom the database, anked by similaity (step 2). The use listens to the songs etuned and selects the desied one (step 3). The moe a peson uses and coects the system, the bette the system pefoms. Ou system employs use feedback to build a database of paied queies and coect tagets (step 4). These paiings ae used to optimize the paametes of ou note segmentation and note inteval similaity paametes fo specific uses (step 5) o goups of uses. In this pape, we focus on how we automatically optimize backend QBH system pefomance, given a small set of example queies. We efe the eade to [10] fo a desciption of the use inteface and use inteaction. 2. QUERY REPRESENTATION In a typical QBH system, a quey is fist tanscibed into a time-fequency epesentation whee the fundamental fequency and amplitude of the audio is estimated at vey shot fixed intevals (on the ode of 10 milliseconds). We call this sequence of fixed-fame estimates of fundamental fequency a melodic contou epesentation. Figue 2 shows the melodic contou of a sung quey as a dotted line. 2007 Austian Compute Society (OCG).

Figue 2. Example note intevals as <PI,,LIR> pais. We segment the melodic contou into notes and then use a note inteval epesentation. The pitch of each note is the median value in its segment. Each note inteval is epesented by the pitch inteval (PI) between adjacent note segments (encoded as un-quantized musical halfsteps) and the log of the atio between the length of a note segment and the length of the following segment (LIR) [8]. Figue 2 shows seveal note intevals as PI, LIR pais. This epesentation has seveal advantages ove a melodic contou: it is both tansposition and tempo invaiant. It is also compact, only encoding salient points of change (note tansitions), athe than evey 10 millisecond fame. This esults in a speed-up of two odes of magnitude when matching queies to tagets. Wok in [1] has shown that the pecision and ecall of seach using quantized note intevals is slightly wose than when using melodic contou. We use unquantized note intevals. Use of unquantized PI and LIR values makes the epesentation insensitive to issues caused by a singe inadvetently singing in an unexpected tuning (A4 440), o slowly changing tuning and tempo ove the couse of a quey. This impoves seach pefomance elative to quantized note intevals. A pevious study showed that use of unquantized note intevals significantly impoved seach pefomance compaed to quantized note intevals [5]. 3. NOTE SEGMENTATION Ou system fist tanscibes the quey as a sequence of 10 millisecond fames. Each fame is a thee element vecto containing values fo pitch, amplitude and hamonicity (elative stength of hamonic components to non hamonic components) [13]. We assume significant changes in these thee featues occu at note boundaies. Thus, we wish to detemine what constitutes significant change. Fo example, a singe may use vibato at times and not at othe times. Thus, the amount of local pitch vaiation that constitutes a meaningful note bounday in one quey may be insufficient to qualify as a note bounday in anothe quey by the same singe. We wish to take local vaiance into account when detemining whethe o not a note bounday has occued. Note segmentation is elated to the poblem of visual edge detection [3]. Accounting fo local vaiation has been helped edge detection in cases whee potions of the image may be bluy and othe potions ae shap [3]. The Mahalanobis distance [6] diffes fom the Euclidean distance in that it nomalizes distances ove a covaiance matix M. Using the Mahalanobis lets one measue distance between fames elative to local vaiation. In a egion of lage vaiance, a sudden change will mean less than in a elatively stable egion. A pevious study showed that ou use of the Mahalanobis ove Euclidean distance significantly impoved seach pefomance [5] We find the distance between adjacent fames in the sequence using the Mahalanobis distance measue, shown in Equation 1. Given a fame f i, we assume a new note has begun wheeve the distance between two adjacent fames f i and f i+1, exceeds a theshold, T. (f i " f i+1 )M "1 (f i " f i+1 )# > T $ new note (1) The matix M is a covaiance matix, calculated fom the vaiance within a ectangula window aound the fame f i. Ou note segmente has fou tuneable paametes: the segmentation theshold (T), and the weights (w) fo each of the thee featues (pitch, hamonicity and amplitude). We addess tuning of these fou paametes in Section 6. Once we have estimated note segment boundaies, we build note intevals fom these note segments. 4. MODELING SINGER ERROR Once a quey is encoded as a sequence of note intevals, we compae it to the melodies in ou database. Each database melody is scoed fo similaity to the quey using a dynamic-pogamming appoach to pefoming sting alignment [9]. Rathe than use a fixed match ewad, the match ewad is based on a similaity function s fo note intevals. Ideally we would like inteval a i to be simila to inteval b j if a i likely to be sung when a singe intended to sing b j. That is, likely eos should be consideed simila to the coect inteval, and unlikely eos should be less simila. Such a function lets a sting-alignment algoithm coectly match eo-pone singing to the coect taget, as long as the singe is elatively consistent with the kinds of eos poduced. In pevious wok [9], we had paticipants listen to note intevals and attempt epoduce the intevals by singing. This study showed that the most common eos wee octave displacements of one o two octaves. The next most common eos wee half-step and whole step eos aound the expected note inteval, o aound peaks offset by an octave. This suppots the obsevations of Shepad [12], who poposes a pitch choma epesentation whee octaves ae elatively close to each othe in the choma space. This suggests that singing eos can be effectively modelled by a set of Gaussian distibutions centeed on the expected pitch inteval and on intevals offset by one o moe octaves. The nomal function, N(a,µ,σ) etuns the value fo a given by a Gaussian function, centeed

on µ, with a standad deviation σ. Equation 4 shows ou note-inteval similaity function, based on the nomal function. n i (, ) = (,,! ) + p " ( p, p + 12,! p ) i=# n s x y w N y x w $ N y x i (4) Let x and y be two note intevals. Hee, x p and y p ae the pitch intevals of x and y espectively, and x and y ae the hythmic atios (LIRs) of x and y. The values w p and w ae the weights assigned to pitch and hythm. The sum of w p and w is 1. The pitch similaity is modeled using 2n+1 Gaussians, each centeed at one o moe octaves above o below the expected pitch inteval. The height of each Gaussian is detemined by an octave decay paamete λ, in the ange fom than 1 to 0. This similaity function povides us with five paametes to tune: the pitch and hythm weight (w p and w ), the sensitivity to distances fo pitch and hythm (σ p and σ ), and the octave decay (λ). Figue 3 shows two octaves of the positive potion of the pitch dimension of this function, given two example paamete settings. Figue 3. The pitch dimension of the similaity function in Equation 5. 5. SYSTEM TRAINING We tain the system by tuning the paametes of ou note segmente (Equations 1 and 2) and note similaity ewad function (Equation 5). We measue impovement using the mean ecipocal ank (MRR) of a set of n queies. We define the ank of a quey as the ode of the coect song in the seach esults. MRR emphasizes the impotance of placing coect taget songs nea the top of the list while still ewading impoved ankings lowe down on the etuned list of songs [1]. Values fo MRR ange fom 1 to 0, with highe numbes indicating bette pefomance. A MRR of 0.25 indicates the coect answe was, on aveage, in the top fou songs etuned by the seach engine. We use a simple genetic algoithm [14] to tune system paametes. Each individual in the population is one set of paamete values fo Equations 1, 2 and 5. The fitness function is the MRR of the paamete settings ove a set of queies. The genetic algoithm epesents each paamete as a binay faction of 7 bits, scaled to a ange of 0 to 1. We allow cossove to occu between (not within) paametes. Duing each geneation, the fitness of an individual is found based on the MRR of the coect tagets fo a set of queies. Paamete settings (individuals) with high MRR values ae given highe pobability of epoduction (fitness popotional epoduction). 6. EMPIRICAL EVALUATION Ou empiical evaluation sought to evaluate the extent to which the system was able to impove seach pefomance in esponse to taining, both in the case of individualized taining to a paticula singe and also geneal taining ove a lage set of singes. Ou quey set was dawn fom the QBSH copus [4] used duing the 2006 MIREX compaison of quey-byhumming systems [2]. We used 10 singes, each singing the same 15 songs fom this dataset. Ou taget database was composed of the 15 tagets coesponding to these queies plus 986 distacte melodies dawn fom a selection of Beatles songs, folk songs and classical music, esulting in a database of 1001 melodies. Chance pefomance, on a database of this size would esult in an MRR 0.005, given a unifom distibution. Fo the genetic algoithm, we chose a population size of 60. Initial tests showed leaning on this task typically ceases by the 30 th geneation, thus esults shown hee epot values fom taining uns of 40 geneations. In pactice, we would like to utilize use-specific taining only when it impoves pefomance elative to an un-pesonalized system. One simple option is to only use use-specific paametes if the use-specific pefomance (MRR u ) is supeio to the pefomance using paametes leaned on a geneal set of queies by multiple uses (MRR g ). To test this idea, we fist tained the system on all queies fom nine of ten singes. We then tested on all the queies fom the missing singe. Coss validation acoss singes was pefomed, thus the expeiment was epeated ten times, testing with the queies fom a diffeent singe each time. To speed leaning, taining was done using a andom sample of 250 taget songs fom the database. Fo each tial, the set of paametes with the best taining pefomance was evaluated by finding the MRR of the testing queies, seaching ove all 1001 melodies in the database. This gave us paametes fo each singe that wee leaned on the queies by the othe nine singes. These ae the geneal paamete settings fo a singe. The mean MRR testing pefomance of the geneal paametes was 0.235 (Std. Dev.=0.063). We then pefomed a use-specific vesion of taining. We used 3-fold coss validation acoss 15 queies fo each of the same ten singes used fo taining the geneal paametes: we optimized paametes on the selected ten queies and tested on the emaining five. This povided us with 30 total tails fo the specific paametes: thee tials fo each of the ten singes. The

mean MRR testing pefomance fo the specific paametes was: 0.228 (Std Dev. = 0.14). Fo each tial we compaed MRR s (the taining pefomance of the leaned use-specific paametes) to MRR g (the taining pefomance of the geneal paametes leaned fom the othe nine singes). If MRR s > MRR g + ε on the taining set, we used the usespecific paametes. Else, we used the geneal paametes. Fo this expeiment, ε was an eo magin set to 0.04. Once the paametes (geneal o use-specific) wee selected, we tested them on the testing set fo that tial. We called this a combined tial. The combined tials had an aveage MRR of 0.289 (Std. Dev. = 0.086). A t-test indicated the impovement of the combined esults ove the geneal and the specific paamete settings is statistically significant (p 0.024). On 50% of the combined tials the specific paametes wee used and impoved pefomance compaed to geneal paametes. On 13% of the tials, specific paametes wee used, but had wose testing pefomance than the geneal paametes. On the emaining 36% of tials, the geneal paametes wee used. Figue 4 shows the aveage MRR pefomance fo untained, geneal and combined paametes. Figue 4. Aveage seach pefomance using untained, geneal and combined (both Use-Specific and Geneal) paametes. 7. CONCLUSIONS We have descibed a QBH system that automatically customizes paametes to individuals o goups afte deployment. Ou esults show that by coectly combing paametes tained to specific uses, and a set tained ove a geneal population, these combined paametes significantly impove mean seach pefomance, esulting in a mean MRR of 0.289 on a database of 1001 melodies. This oughly coesponds to consistently placing the coect taget in the top fou esults. This compaes to an MRR of 0.0151 (Std. Dev. = 0.0018) pio to taining. Ou esults also show unquantized note intevals and note segmentation that takes into account local pitch vaiation significantly impove pefomance. In futue wok we will exploe how pefomance vaies with espect to the numbe of taining examples and impove the infomation that can be used fo taining while maintaining use-specificity. We will also exploe moe sophisticated citeia to detemine when usespecific taining should be used. 8. REFERENCES [1] R. Dannenbeg, W. Bimingham, B. Pado, N. Hu, C. Meek and G. Tzanetakis, "A Compaative Evaluation of Seach Techniques fo Quey-by-Humming Using the MUSART Testbed", Jounal of the Ameican Society fo Infomation Science and Technology (2007), pp. in pess. [2] J. S. Downie, K. West, A. Ehmann and E. Vincent, "The 2005 Music Infomation etieval Evaluation Exchange (MIREX 2005): Peliminay Oveview", 6th Intenational Confeence on Music Infomation Retieval, Septembe 11-15, London, UK, 2005. [3] J. H. Elde and S. W. Zucke, "Local scale contol fo edge detection and blu estimation", Patten Analysis and Machine Intelligence, IEEE Tansactions on, 20 (1998), pp. 699-716. [4] Jyh-Shing and R. Jang, "QBSH: A coups fo Designing QBSH (Quey by Singing/Humming) Systems", 2006. [5] D. Little, D. Raffenspege and B. Pado, "Online Taining of a Music Seach Engine", Nothwesten Univesity, Evanston, IL, NWU- EECS-07-03, 2007 [6] P. C. Mahalanobis, "On the genealised distance in statistics, " Poceedings of the National Institute of Science of India 12 (1936), pp. 49-55. [7] C. Meek and W. Bimingham, "A Compehensive Tainable Eo model fo sung music queies", Jounal of Atificial Intelligence Reseach, 22 (2004), pp. 57-91. [8] B. Pado and W. Bimingham, "Encoding Timing Infomation fo Music Quey Matching", Intenational Confeence on Music Infomation Retieval, Pais, Fance, 2002. [9] B. Pado, W. P. Bimingham and J. Shifin, "Name that Tune: A Pilot Study in Finding a Melody fom a Sung Quey", Jounal of the Ameican Society fo Infomation Science and Technology, 55 (2004), pp. 283-300. [10] B. Pado and D. Shamma, "Teaching a Music Seach Engine though Play", CHI 2007, Compute/Human Inteaction San Jose, Califonia, 2007. [11] C. Pake, A. Fen and P. Tadepalli, "Gadient boosting fo sequence alignment", The Twenty- Fist National Confeence on Atificial Intelligence, Boston, MA, 2006. [12] R. N. Shepad, "Geometical Appoximations to the stuctue of musical pitch", Psychological Review, 89 (1982), pp. 305-309. [13] G. Tzanetakis and F. Cook, "A famewok fo audio analysis based on classification and

tempoal segmentation", EUROMICRO Confeence, Milan, 1999, pp. 61-67. [14] A. Wight, "Genetic algoithms fo eal paamete optimization", The Fist wokshop on the Foundations of Genetic Algoithms and Classie Systems, Bloomington, Indiana, 1990.