Evaluation of a Singing Voice Conversion Method Based on Many-to-Many Eigenvoice Conversion

Transcription

1 INTERSEECH 2013 Evaluaion of a Singing Voice Conversion Mehod Based on Many-o-Many Eigenvoice Conversion Hironori Doi 1, Tomoki Toda 1, Tomoyasu Nakano 2, Masaaka Goo 2, Saoshi Nakamura 1 1 Graduae School of Informaion Science, Nara Insiue of Science and Technology, Nara, Japan 2 Naional Insiue of Advanced Indusrial Science and Technology AIST, Ibaraki, Japan 1 {hironori-d, omoki, s-nakamura}@is.nais.jp, 2 {.nakano, m.goo}@ais.go.jp Absrac In his paper, we evaluae our proposed singing voice conversion mehod from various perspecives. To enable singers o freely conrol heir voice imbre of singing voice, we have proposed a singing voice conversion mehod based on many-omany eigenvoice conversion EVC ha enables o conver he voice imbre of an arbirary source singer ino ha of anoher arbirary arge singer using a probabilisic model. Furhermore, o easily develop raining daa consising of muliple parallel daa ses beween a single reference singer and many oher singers, a echnique for efficienly and effecively generaing he parallel daa ses from nonparallel singing voice daa ses of many singers using a singing-o-singing synhesis sysem have been proposed. However, we have never conduced sufficien invesigaions ino he effeciveness of hese proposed mehods. In his paper, we conduc boh objecive and subjecive evaluaions o carefully invesigae he effeciveness of proposed mehods. Moreover, he differences beween singing voice conversion and speaking voice conversion are also analyzed. Experimenal resuls show ha our proposed mehod succeeds in enabling people o conrol heir own voice imbre by using only an exremely small amoun of he arge singing voice. Index Terms: singing voice, voice conversion, eigenvoice conversion, singing-o-singing synhesis, performance evaluaion 1. Inroducion Range of singing voice imbre ha can be produced by individual singers is limied by physical consrains. To produce a singing voice beyond physical consrains, many approaches have been sudied. One of he mos popular approaches is he use of singing synhesis sysems, which generae a singing voice from several pieces of informaion such as lyrics and he musical score. Among hem, a ex-o-singing approach, which synhesizes a singing voice from noe-level score informaion of he melody wih is lyrics, such as Vocaloid2 [1] and Sinsy [2] is popular in Japan. Moreover, singing-o-singing synhesis, which auomaically synhesizes a more naurally sounding singing voice by esimaing he parameers of he exo-singing sysem from a arge singing voice, has been proposed [3]. VocaLisener [3], which is he sysem used for he esimaion par of singing-o-singing synhesis, esimaes parameers of pich and dynamics for he singing synhesis sysem so ha he synhesized singing voice becomes more similar o he arge singing voice. If a user s singing voice and he corresponding lyrics wihou any score informaion are available, VocaLisener can synchronize hem auomaically o deermine he musical noe corresponding o each phoneme of he lyrics. However, i is sill difficul o generae singing voices wih arbirary and desired voice imbre. To make i possible for people o direcly sing wih a differen specific voice imbre, and hus overcome physical consrains, singing voice conversion has been proposed [4]. Saisical voice conversion VC echniques [, 6, 7] are used o conver he singing voice imbre of a source singer ino ha of a arge singer. In his echnique, Gaussian mixure model GMM of he join probabiliy densiy of an acousic feaure beween he source singer s singing voice and he arge singer s singing voice is rained in advance using a special daa se, called a parallel daa se, ha consiss of pairs of songs of he wo singers. The rained model is capable of convering he acousic feaures of he source singer s singing voice ino hose of he arge singer s singing voice for any song while keeping he linguisic informaion of he lyrics unchanged. Moreover, real-ime singing voice conversion can also be achieved using he lowdelay conversion algorihm [8]. Towards realizing a more flexible singing voice conversion echnique, we have proposed a singing voice conversion mehod [9] based on many-o-many eigenvoice conversion EVC [10]. Many-o-many EVC is a echnique of convering from he voice of an arbirary source singer ino ha of an arbirary arge singer. An eigenvoice GMM EV-GMM [11] is rained in advance using muliple parallel daa ses ha consis of a single predefined singer, called a reference singer in his paper, and many presored arge singers. The EV-GMM is capable of easily adaping he source/arge voice imbre o ha of is given voice samples in a ex-independen lyricsindependen manner. Furhermore, we have proposed a echnique for efficienly and effecively generaing parallel daa ses using a singing-o-singing synhesis sysem o arificially generae singing voices of he reference singer. In his paper, we describe our proposed mehods [9] and evaluae heir effeciveness. A comparison beween VC and EVC based singing voice conversion is conduced from various perspecives. Moreover, o analyze he differences beween speaking voice and singing voice in voice conversion, we conduc comparison beween singing voice conversion using EV- GMM rained from speaking voice and from singing voice. 2. Singing voice conversion based on many-o-many EVC In his secion, we describe singing voice conversion mehod based on many-o-many EVC and raining daa generaion using singing-o-singing synhesis sysem Training daa generaion The developmen of parallel daa ses consising of singing voice pairs of he single reference singer and many presored Copyrigh 2013 ISCA Augus 2013, Lyon, France

2 arge singers is laborious work. To address his issue, we have arificially generaed singing voices of he reference singer by applying a singing-o-singing synhesis sysem o singing voices of many presored arge singers. In his approach, we need o prepare only singing voices of muliple presored arge singers who need no sing he same song; hese are available in exising daabases, such as he RWC Music Daabase [12]. For he singing voices of each presored arge singer, corresponding singing voices of he reference singer are arificially generaed by using he singing-o-singing synhesis sysem. Thus, his raining daa generaion approach can efficienly and effecively develop parallel daa ses wihou recording singing voices of he reference singer Training process As acousic feaures of he reference singer and he s h presored arge singer, we employ wo D-dimensional join feaures, X = [x, x ] and Y s = [y s s, y ], consising of D-dimensional saic and dynamic specral feaures a frame, respecively, where denoes he ransposiion of he vecor. The join probabiliy densiy of reference and arge feaures is modeled wih he EV-GMM as follows: X, Y s λ EV, w s µ s = m = [ M α mn µ X m A mw s + b m [X, Y s ] ] ; µ s m, ΣX,Y m [, Σ X,Y m Σ XX = Y X Σ m Σ XY m m Σ m Y Y, 1 ], 2 where w s = [w s 1,, w s J] is he arge-speakerdependen weigh parameer for conrolling arge voice imbre. λ EV is a canonical EV-GMM parameer se consising of he weigh α m, he mean vecor µ X m, he covariance marix Σ X,Y m, he bias vecor b m, and he basis vecors A m = [a m1,, a mj] for he m h mixure componen, where he number of basis vecors is J. Acousic feaures of an arbirary arge speaker are modeled by seing only w s o he speaker s specific values. To alleviae he degradaion of performance of EV-GMM caused by effecs of acousic variaion of he many presored arge singers, he EV-GMM is rained by speaker adapive raining SAT [13, 14] using muliple parallel daa ses consising of uerance pairs of a reference and many presored arge singers Adapaion and conversion process In he adapaion process, he EV-GMM is adaped o an arbirary source singer and an arbirary arge singer by independenly esimaing he singer-dependen weigh parameer using a few singing voice samples. The weigh parameer for source singer ŵ i is esimaed by maximum a poseriori MA [1, 16] as ŵ i = argmax w w λ w τ = argmax w w λ w τ ˆλ ω = argmax λ s S s=1 T =1 T =1 X, Y i λ EV, w dx, Y i λ EV, w, ˆω λ ω, 3 where λ ω is a model parameer se consising of he mean vecor µ w and he covariance marix Σ ω. This model parameer se is rained in advance using a se of weigh parameers esimaed for individual presored arge singer. Y i is he acousic feaures of he given source singer s voice a frame. The balance beween w λ w and T =1 Y i λ EV, w is conrolled by he hyperparameer τ. The weigh parameer for he arge singer ŵ o is esimaed in he same manner. On he oher hand, our proposed mehod allows user o freely conrol voice imbre of he convered singing voice by manipulaing he arge singer s weigh parameers. Then, he join probabiliy densiy of he acousic feaures beween he source singer s voice and he arge singer s voice is derived as = Y i, Y o ŵ i, ŵ o, λ M m λ EV EV Y i X, m, w i, λ EV Y o X, m, w o, λ EV X m, λ EV dx, M [y ] [ ] [ ] i µ i = α mn y o m Σ Y Y Y XY m Σ m ; µ o Y XY m Σ m Σ Y m Y, 4 where Y XY Y X Σ m = Σ m Σ XX 1 m Σ XY m. In he conversion process, he convered saic feaure sequence vecor is esimaed using he adaped EV-GMM. Maximum likelihood esimaion considering dynamic feaures and a global variance [6] is adoped. Noe ha real-ime singing voice conversion is also achieved by using he low-delay conversion algorihm [8]. 3. Experimenal evaluaions To demonsrae effeciveness of our proposed mehod and invesigae he differences beween singing voice conversion and speaking voice conversion, four ypes of conversion model were compared. VC convenional singing voice conversion based on VC [6] EVC-human proposed singing voice conversion based on many-o-many EVC wih convenional raining daa generaion using a human voice as he reference singer s voice EVC-synh proposed singing voice conversion based on many-o-many EVC wih raining daa generaion using singing-o-singing synhesis EVC-speaking convenional many-o-many EVC for a speaking voice 3.1. Experimenal condiions In his evaluaion, only he specral feaure is convered in all conversion mehods because he voice imbre srongly depends on he specral feaure. The 1 h o 24 h mel-cepsral coefficiens were used as a specral feaure. STRAIGHT analysis [17] was employed o exrac hese coefficiens from singing voices. F 0 and he aperiodic componens of he source singer are direcly used o synhesize he convered singing voice. The shif lengh was ms and he sampling frequency was Hz. We used he solo singing voices of 30 Japanese songs in he RWC Music Daabase [12] as he presored arge singing voices o rain EV-GMM. The phoneme balance was no considered in hese songs. For EVC-human, he solo singing voices of one male singer were used as he singing voices of he reference singer. For EVC-synh, singing voices synhesized using he singing-o-singing synhesis sysem VocaLisener wih a singer daabase called Hasune Miku [18] based on Vocaloid2 were 1068

3 Mel-cepsral disorion [db] Mel-cepsral disorion [db] used as he reference singer. The number of basis vecors of he EV-GMMs was se o 29 and he number of mixure componens of he EV-GMMs was se o 128. On he oher hand, in EVC-speaking, we used parallel daa ses of a single reference male speaker and 12 presored arge speakers o rain he EV-GMM. These speakers were from he Japanese Newspaper Aricle Senence JNAS daabase. Each presored arge speaker uered one of seven subses. Each subse consiss of 0 phoneically balanced senences. The EV-GMM for specral conversion was rained from 12 parallel daa ses consising of he recorded reference speaking voices and he presored arge speaking voices. The number of basis vecors of he EV-GMMs was se o 11 and he number of mixure componens of he EV-GMMs was se o 128. For he adapaion and esing of he EV-GMMs and for he raining and esing of he GMM, we seleced wo Japanese songs from he RWC Music Daabase RWC-MDB No.46 and No.76, which were no included in he above 30 songs. Then, singers four male singers and one female singer sang hese wo songs. Thus, as adapaion/raining daa and es daa, we prepared 10 songs consising of wo songs sung by each singer. As he raining daa for he VC-based mehod and he adapaion daa for he EVC-based mehods, 2, 4, 8, 16, 32, or 64% of he sung pars of songs sung by he source and arge singers was used, hen, he remaining 36% of daa was used for he es. The GMM and EV-GMMs were prepared for all combinaions of he source and arge singers. Thus, for each mehod, 20 conversion models 10 models 2 song were prepared. The weigh parameers of he source and arge singer were independenly esimaed using he specral feaures from he source and arge singing voice samples. The hyperparameer of MA adapaion shown in eq. 3 was preliminarily opimized in each mehod. In his evaluaion, i was se o 20, 1000, and 100 for EVC-human, EVC-synh, and EVC-speaking, respecively. For VC, we also rained a sandard GMM for specral conversion using a parallel daa se consising of he source and arge singing voices. The number of mixure componens of he GMM was preliminarily opimized so ha he specral conversion accuracy was maximized in he es daa Objecive evaluaion We evaluaed wo condiions of song seing: 1 he samesong condiion, where he same song is used in boh he raining/adapaion process and he es process, and 2 he differensong condiion, where differen songs are used in he raining/adapaion process and he es process. Figure 1 shows melcepsral disorion as a funcion of he amoun of he singing voice adapaion daa used in he EVC-based mehods or he amoun of parallel daa of he singing voice pairs used in he VC-based mehod under he same-song condiion. Figure 2 shows hose under he differen-song condiion. In fig. 1 and 2, horizonal axis represens percenage of daa ha is used for raining or adapaion from he sung pars of songs. Under he same-song condiion, when using a small amoun of raining/adapaion daa, EVC-speaking is he bes, EVChuman is he nex, EVC-synh is he nex, and VC is he wors in conversion accuracy. Alhough EVC-speaking exhibis he highes conversion accuracy, he differences from EVC-human are no so large even if he amoun of raining daa for EVCspeaking is significanly larger han ha for EVC-human. When using a large amoun of raining/adapaion daa, VC is he bes, EVC-speaking is he nex, EVC-human is he nex, and he EVC-synh is he wors in conversion accuracy. Noe ha he VC EVC_human EVC_synh EVC_speaking Amoun of raining or adapaion daa [%] Figure 1: Mel-cepsral disorion as a funcion of amoun of arge singing voice daa i.e., singing voice pairs in VC-based mehod or singing voice adapaion daa in EVC-based mehods under he same-song condiion VC EVC_human EVC_synh EVC_speaking Amoun of raining or adapaion daa [%] Figure 2: Mel-cepsral disorion as a funcion of amoun of arge singing voice daa i.e., singing voice pairs in VC-based mehod or singing voice adapaion daa in EVC-based mehods under he differen-song condiion. EVC-based mehods do no require he use of parallel daa in he adapaion, in conras o VC. Under he differen-song condiion, VC has much lower conversion accuracy han under he same-song condiion. This is because he voice imbre of he singing voice of a singer significanly changes depending on he song. On he oher hand, i is observed ha he EVC-based mehods reduce his degradaion. Since he EV-GMM is rained wih many singers voices, i is more robus agains variaions of he singing voice imbre Subjecive evaluaion We conduced an opinion es on he nauralness of he singing voice and a preference es on singer individualiy. The opinion was expressed using a five poin scale i.e., 1 very poor o excellen. In his es, 10 liseners heard 16 ypes of convered singing voice sample, hen hey judged he nauralness of each sample using he opinion score. In he preference es, liseners heard a arge singing voice sample and wo convered singing voice samples, hen hey chose he convered singing voice sample wih more similar singer individualiy o he arge singing voice sample. The preference es was performed under he differen-song condiion because of is greaer realism han same-song condiion. In his ess, 9 liseners evaluaed eigh ypes of he singing voice generaed under he differensong condiion for all combinaions of 2% or 64% of raining/adapaion daa and four ypes of conversion mehod. Figure 3 shows he resul of he opinion es on he nauralness of he singing voice. Under he same condiion, VC us- 1069

4 Mean opinion score MOS reference score [%] same diff same diff same diff same diff same diff same diff same diff same diff [%] VC EVC_human EVC_synh EVC_speaking Figure 3: Resul of opinion es on nauralness. 9% confidence inervals VC EVC_human EVC_synh EVC_speaking Figure 4: Resul of preference es on singer individualiy under he differen-song condiion. ing 2% raining daa shows similar nauralness o ha of EVChuman using 2% adapaion daa in conras o objecive evaluaion. On he oher hand, he nauralness of EVC-speaking is no higher han ha of oher mehods when using a small amoun of adapaion daa. This resul suggess ha i is difficul for EV-GMM rained wih speaking voice o generae convered singing voice having high nauralness even if a large amoun of speaking voice is available as raining daa. Oher resuls show similar endency o ha observed in he resul of he objecive evaluaion. Figure 4 shows he resul of he preference es on singer individualiy. The preference score was calculaed as he raio of he number of samples seleced as having beer singer individualiy o he number of samples presened o he liseners. When using a small amoun of raining/adapaion daa, EVChuman is he bes, EVC-synh is he nex, EVC-speaking is he nex, and VC is he wors in preference score of singer individualiy. On he oher hand, when using a large amoun of raining/adapaion daa, VC and EVC-speaking show higher preference score of singer individualiy han oher mehods. Noe ha VC requires he parallel daa se of he source and arge singers and he canonical EV-GMM of EVC-speaking is rained wih significanly larger amoun of raining daa han ha of EVChuman and EVC-synh Comparison of each EV-GMMs Figure shows he cumulaive disribuion of occupancies of he canonical EV-GMM of EVC based mehods. These individual mixure componen occupancies have been calculaed from all parallel daa se in raining process wih SAT. In his figure, we can see ha he occupancies of EVC-human and EVC-synh are more biased han ha of EVC-speaking. Alhough he EV- GMM needs o model wide varieies of acousic feaures of all presored arge speakers, his resul shows ha some mixure [%] Cumulaive occupancy probabiliy [%] EVC-human EVC-synh EVC-speaking Sored mixure componen index Figure : Cumulaive occupancy probabiliy for all parallel daa se using several models. componens of EVC-human and EVC-synh model only acousic feaures of a par of presored arge speakers. Consequenly, i is expeced ha phonemic informaion and speaker individualiy were no separaed well in hem. I is possible ha his issue causes degradaion of conversion performance. The above resuls sugges ha 1 he proposed EVC-human yields beer conversion performance han VC when a small amoun of singing voice daa of he source and arge singers is available, 2 he conversion performance of he proposed EVC-synh is slighly degraded han EVC-human, 3 since hese proposed mehods are robus agains variaions of he singing voice imbre ofen observed beween differen songs, hey work reasonably well even when differen songs are used in he adapaion and conversion processes, 4 he occupancies of individual mixure componen of EV-GMM in EVC-human and EVC-synh are more biased han hose in EVC-speaking, and hen, his causes degradaion of conversion accuracy for singer individualiy, he differences beween a speaking voice and singing voice srongly affecs o nauralness of convered singing voice. Based on hese resuls, o more correcly conrol he voice imbre, i is necessary o rain EV-GMM from larger parallel daa ses considering phoneme balance. And hen, i is expeced ha raining daa generaion using singing-o-singing synhesis is significanly effecive o consruc hem. 4. Conclusion In his paper, we evaluaed our proposed singing voice conversion mehods. Our proposed mehods are capable of convering he singing voice imbre of an arbirary source singer ino ha of an arbirary arge singer by adaping a small number of adapive parameers of a conversion model using an exremely small amoun of source and arge singing voice daa. Moreover, our proposed raining daa generaion mehod can alleviae he burden of having o record singing voices o develop parallel daa ses, by using a singing-o-singing synhesis sysem. The experimenal resul demonsraed ha he proposed mehods enable he effecive conversion of a singing voice beween an arbirary singer pair even when using only several seconds of heir singing voices as adapaion daa. We plan o consruc larger parallel daa ses considering phoneme balance and furher improve conversion performance.. Acknowledgemens This work was suppored in par by a JSS KAKENHI Gran Number and JST OngaCREST projec. The auhors are graeful o professor Hideki Kawahara of Wakayama Universiy, Japan, for permission o use he STRAIGHT analysissynhesis mehod. 1070

5 6. References [1] H. Kenmochi and H. Ohshia, VOCALOID Commercial singing synhesizer based on sample concaenaion, roc. IN- TERSEECH, pp , Aug [2] K. Oura, A. Mase, T. Yamada, S. Muo, Y. Nankaku, and K. Tokuda, Recen developmen of he HMM-based singing voice synhesis sysem - Sinsy, SSW7, pp , Sep [3] T. Nakano and M. Goo, VocaLisener: A singing-o-singing synhesis sysem based on ieraive parameer esimaion, roc. SMC 2009, pp , May [4] Y. Kawakami, H. Banno, and F. Iakura, GMM voice conversion of singing voice using vocal rac area funcion, IEICE echnical repor. Speech Japanese ediion, pp , Nov [] Y. Sylianou, O. Cappe, and E. Moulines, Coninuous probabilisic ransform for voice conversion, IEEE Trans. SA, vol. 6, no. 2, pp , Mar [6] T. Toda, A. W. Black, and K. Tokuda, Voice conversion based on maximum likelihood esimaion of specral parameer rajecory, IEEE Trans. ASL, vol. 1, no. 8, pp , Nov [7] A. Kain and M. W. Macon, Specral voice conversion for ex-ospeech synhesis, roc. ICASS, pp , May [8] T. Muramasu, Y. Ohani, T. Toda, H. Saruwaari, and K. Shikano, Low-delay voice conversion based on maximum likelihood esimaion of specral parameer rajecory, roc. INTERSEECH, pp , Sep [9] H. Doi, T. Toda, T. Nakano, M. Goo, and S. Nakamura, Singing voice conversion mehod based on many-o-many eigenvoice conversion and raining daa generaion using a singing-o-singing synhesis sysem, ASIA ASC 2012, Dec [10] Y. Ohani, T. Toda, H. Saruwaari, and K. Shikano, Manyo-many eigenvoice conversion wih reference voice, INTER- SEECH, pp , Sep [11] T. Toda, Y. Ohani, and K. Shikano, One-o-many and many-oone voice conversion based on eigenvoices, roc. ICASS, pp , Apr [12] M. Goo, T. Nishimura, H. Hashiguchi, and R. Oka, RWC Music Daabase: Music genre daabase and musical insrumen sound daabase, roc. ISMIR, pp , Oc [13] T. Anasasakos, J. McDonough, S. R., and J. Makhoul, A compac model for speaker-adapive raining, roc. ICSL, vol. 2, pp , [14] Y. Ohani, T. Toda, H. Saruwaari, and K. Shikano, Adapive raining for voice conversion based on eigenvoices, IEICE Trans. Inf. and Sys., vol. E93-D, no. 6, pp , June [1] G.-L. Gauvain and C.-H. Lee, Maximum a poseriori esimaion for mulivariae Gaussian mixure observaions of Markov chains, IEEE Trans. Speech and Audio rocessing, vol. 2, no. 2, pp , [16] D. Tani, T. Toda, Y. Ohani, H. Saruwaari, and K. Shikano, Maximum a poseriori adapaion for many-o-many eigenvoice conversion, roc. INTERSEECH, pp , Sep [17] H. Kawahara, I. Masuda-Kasuse, and A. Cheveigne, Resrucuring speech represenaions using a pich-adapive ime-frequency smoohing and an insananeous-frequency-based f 0 exracion: ossible role of a repeiive srucure in sounds, Speech Communicaion, vol. 27, no. 3-4, pp , Apr [18] Crypon Fuure Media, Wha is he HATSUNE MIKU movemen? [Online]. Available: hp:// eng 1071