LOW LEVEL DESCRIPTORS BASED DBLSTM BOTTLENECK FEATURE FOR SPEECH DRIVEN TALKING AVATAR

Transcription

1 LOW LEVEL DESCRIPTORS BASED DBLSTM BOTTLENECK FEATURE FOR SPEECH DRIVEN TALKING AVATAR Xinyu Lan 1,2, Xu Li 1,2, Yishuang Ning 1,2, Zhiyong Wu 1,2,3, Helen Meng 1,3, Jia Jia 1,2, Lianhong Cai 1,2 1 Tsinghua-CUHK Join Research Cener or Media Sciences, Technologies and Sysems, Shenzhen Key Laboraory o Inormaion Science and Technology, Graduae School a Shenzhen, Tsinghua Universiy, Shenzhen , China 2 Tsinghua Naional Laboraory or Inormaion Science and Technology (TNLis), Deparmen o Compuer Science and Technology, Tsinghua Universiy, Beijing , China 3 Deparmen o Sysems Engineering and Engineering Managemen, The Chinese Universiy o Hong Kong, Shain, N.T., Hong Kong SAR, China {syslxy1991, dongangyixi}@gmail.com, ningys13@mails.singhua.edu.cn {zywu, hmmeng}@se.cuhk.edu.hk, {jjia, clh-dcs}@singhua.edu.cn ABSTRACT Speech is bimodal in naure. There are close correlaions beween he acousic speech signals and he visual gesures such as lip movemens, acial expressions and head moions. For speech driven alking avaar, how o derive more represenaive acousic eaures rom which o predic more accurae and realisic visual gesures sill remains he research problem. Inspired by he promising perormance o low level descripors (LLD) in speech emoion recogniion, in his work, we invesigae he usage o LLD eaure or he ask o speech driven alking avaar. Furhermore, visual gesures also demonsrae correlaions wih no only conex inormaion o pas or uure acousic eaures (e.g. anicipaory coariculaion phenomena) bu also exual inormaion (e.g. exual hins or lip movemen). To incorporae such inormaion, we also propose o use deep bidirecional long shor-erm memory (DBLSTM) as he boleneck eaure exracor, which can combine LLD eaure wih conexual inormaion. Experimenal resuls indicae ha he proposed LLD based DBLSTM boleneck eaure ouperorms he convenional specrum relaed eaures or he ask o speech driven alking avaar, and more sophisicaed conexual inormaion can urher improve he perormance. Index Terms boleneck eaure, deep bidirecional long shor-erm memory (DBLSTM), low level descripors (LLD), alking avaar 1. INTRODUCTION Talking avaar has drawn exensive aenion or is wide use in human-compuer ineracion ields, e.g. voice agen, virual eacher or hos, inelligen compuer assisan, ec. Speech is bimodal in naure. The visual gesures such as lip movemens, acial expressions and head moions are closely correlaed wih he acousic speech. Hence los o research ineress have been devoed o ind and model such correlaions. To build a speech driven alking avaar, exracing appropriae acousic eaures is very imporan. Tradiional specrum relaed eaures such as mel-requency cepsral coeiciens (MFCC) were widely used in acial animaion and head moion generaion [1-3]. Recen works have mainly ocused on developing machine learning models or he audio-visual mapping problem, bu pu less emphasis on how o ind more represenaive acousic eaures or more realisic and expressive visual gesure generaion. Speech emoion recogniion provides us some hins. [4] and [5] chose pich (F0), roo mean square energy (RMSE) and ormans as prosodic eaures o perorm aec recogniion. Aerwards, low level descripors (LLD) araced los o aenions and achieved sae-o-he-ar perormance. [6] and [7] developed sysems or emoion classiicaion rom LLD eaures. Inspired by he promising perormance o LLD in speech emoion recogniion, we invesigae i LLD could ouperorm he radiional eaures in he speech driven alking avaar ask. Mos previous sudies on alking avaar have ocused on acial animaion (lip movemens and/or acial expressions) only. Some phoo-real mehods reconsruced ace images wih lip movemen rom principle componen analysis (PCA) based visual eaures [8]. [9] adoped acive appearance model (AAM) eaures involving boh shape and exure inormaion or more realisic lip animaion. [10] used moion uni parameers (MUP) or alking ace animaion wih expression. Recenly, los o work rendered animaion using he MPEG-4 acial animaion parameers (FAPs) [11][12], where FAPs oer a parameerizaion approach or he animaion o eyes, mouh, ongue, eeh, head moion, ec. However, mos works have omied he head moions ha oen occur simulaneously wih lip movemens and acial expressions or an expressive and realisic alking avaar. Alhough some sudies have ried o generae head moions [13][14], he acial animaion and head moion are sill modeled separaely regardless he close correlaions beween hem. In his work, we adop FAPs as he visual eaures and predic lip movemens, acial expressions and head moions simulaneously rom acousic speech wih a single shared regression model. Visual gesures a a paricular ime sep are also correlaed wih he conex inormaion o pas or uure acousic eaures (e.g. he anicipaory co-ariculaion phenomena). How o model such correlaion provides anoher challenge. In previous work, here are mainly wo sreams o approaches. The irs one is hidden Markov model (HMM) based mehod [2][8] moivaed by he ideas rom auomaic speech recogniion (ASR). In his mehod, he exual conex inormaion can be easily incorporaed by sae-ransiion probabiliies. The second one is he direc acousic o visual eaure mapping approach using long shor-erm memory (LSTM) [15][9], bidirecional LSTM (BLSTM), ec. These LSTM derived models have shown superior perormance over HMM wih capabiliies in capuring long-range conex inormaion [9]. Whereas, he exual /16/$ IEEE 5550 ICASSP 2016

2 inormaion such as phoneme labels ha are mos valuable or lip movemens and head moions are no well considered in hese models. On he oher hand, deep bidirecional LSTM (DBLSTM) as he probabilisic eaure exracor, can involve conex o boh acousic eaures and phoneme labels during eaure exracion [16][17]. In his paper, we propose a LLD based DBLSTM boleneck eaure ha akes ino accoun no only he conexual acousic eaure correlaions bu also he exual inormaion. The res o he paper is organized as ollows. Secion 2 gives an overview o our sysem archiecure. Deailed acousic visual eaures and heir exracion mehod are lised in Secion 3. Secion 4 describes he LLD based DBLSTM boleneck eaure and he raining mehod o he DBLSTM or boleneck eaure exracion. Secion 5 discusses he objecive and subjecive experimens comparing he perormance o dieren eaures and nework archiecures. Finally, our conclusions are drawn in Secion SYSTEM FRAMEWORK Fig. 1 shows he block diagram o he proposed sysem, which involves a raining sage or wo DBLSTMs and a predicion sage. The irs DBLSTM is or boleneck eaure exracion and he second DBLSTM is or boleneck o visual eaure mapping. In he raining sage, given he audio visual bimodal corpus, we exrac acousic eaures and visual eaures (FAPs). Meanwhile, orced alignmen beween acousic eaures and conexual labels (e.g. phoneme labels) is perormed wih a homegrown HMM based speech recognizer. The DBLSTM eaure exracor is hen rained wih cross enropy error; hus a discriminaive mapping beween he acousic eaure and laen conexual sequence is esablished. Thereaer, we can derive he DBLSTM boleneck eaure. Then he second DBLSTM is rained o learn he regression model beween he boleneck eaure and FAPs. In he predicion sage, given an inpu speech uerance, he DBLSTM eaure exracor irs exracs boleneck eaures rom he raw acousic eaures, and hen he DBLSTM mapping model predics he FAP sequences rom he boleneck eaures. Finally, he visual gesures including lip movemens, acial expressions and head moions are reconsruced on a 3 dimensional avaar using he echnologies o our previous work [12][13][21]. 3. ACOUSTIC AND VISUAL FEATURES 3.1. Low level descripors (LLD) Taking advanage o he aracing perormance o LLD in emoion recogniion ask, we chose a 384 dimensional acousic eaure vecors ha served as he eaure se o he INTERSPEECH 2009 Emoion Challenge [22], which conains 16 low level descripors and heir irs order dela regression coeiciens (32 dimensions in oal) and 12 uncionals. Table 2 liss he saisical uncionals ha are applied o he low level descripors shown in Table Facial animaion parameers (FAPs) The MPEG-4 speciicaion deines oally 68 FAPs, including 66 low-level FAPs and 2 high-level FAPs [21]. The low-level FAPs, based on he movemens o acial deiniion poins, represen a complee se o basic acial acions; while he 2 high-level FAPs represen visemes and expressions respecively. All low-level FAPs are sandard values and expressed in erms o he acial animaion parameer unis (FAPUs), which allow he inerpreaion Fig 1. Sysem ramework Table low level descripors used Feaure Group Feaures in Group No. RMSE Roo mean square signal rame energy 0 MFCC Mel-requency cepsral coeiciens 1~12 PCM zcr Zero-crossing rae o ime signal 13 Voice Probabiliy F0 Voicing probabiliy compued rom he auocorrelaion uncions (ACF) The undamenal requency compued rom he Cepsrum Table uncionals applied o LLD conours Funcionals No. The max/min value o he conour 1~2 Range (max min) 3 The absolue posiion o he max/min value (in rames) 4~5 The arihmeic mean o he conour 6 The slope o a linear approximaion o he conour 7 The ose o a linear approximaion o he conour 8 The quadraic error compued as he dierence o he linear approximaion and he acual conour 9 The sandard deviaion o he values in he conour 10 The skewness and he kurosis 11~12 o he FAPs o i wih any ace models. MPEG-4 sandard deines a neural ace and all he FAPs are expressed as displacemens rom he posiions deined in he neural ace. Our work ocuses on conrolling he acial deiniion poins direcly. The 2 high-level FAPs are no considered. We adop he open source oolki visagesdk [18] o exrac FAPs rom raw video daa. Some o he parameers relaed o ongue, eeh, nose and ears currenly canno be reliably esimaed and do no aec he animaion o alking avaar on our ollowing predicion seps. The values o hese FAPs are simply se o be zeros. 4. DBLSTM BOTTLENECK FEATURE 4.1. DBLSTM boleneck eaure exracor As a kind o recurren neural nework (RNN), long shor erm memory (LSTM) has been demonsraed o be one o he mos eecive archiecures o map he long-erm hisory o inpus o he curren oupu by solving he vanishing gradien problem ha radiional RNN aces. To rerieve boh pas and uure conexual inormaion, bidirecional LSTM (BLSTM) wih wo separae hidden layers scanning he inpu sequences in boh direcions has

3 been popular recenly [20]. Inspired by is promising perormance in sequence classiicaion and regression, deep BLSTM (DBLSTM) has been developed o exrac poenial eaures rom inpu eaures. This paper proposes a LLD based DBLSTM boleneck eaure exracor, which acceps LLD acousic eaure vecors and oupus poenial conexual sequences, o exrac he boleneck eaure. Fig. 2 illusraes he nework srucure in deail. The eaure exracor can be divided ino wo pars, namely raining sage and exracing sage. During he raining sage, we rain he DBLSTM as a speech-based recognizer on he raining daase. LLD acousic eaure serves as he inpu o DBLSTM nework, while he rame wise conexual label serves as he oupu. In his work, wo kinds o conexual labels (phoneme label and HMM sae label) are involved o generae wo conexual levels o boleneck eaures. The rame wise correspondence inormaion are generaed by he orced alignmen process as described in Secion 2. Three hidden LSTM layers are used or boh orward and backward direcions. Taking advanage o he resuls rom previous work, we adoped wo hidden layers and a boleneck layer wih saionary size o 40 [17]. In he exracing sage, he acivaions o he oupu layer o DBLSTM are ignored, as we ocus on he oupu o he boleneck layers. As shown in Fig. 2, he inal 80 dimensional boleneck eaure is generaed by combining he oupus o he wo direcional boleneck layers o DBLSTM Training mehod Training a DBLSTM eaure exracor can be regarded as learning a special kind o neural nework or speech recogniion rom he LLD acousic eaures o he conexual label sequences. I can be rained o minimize he cross-enropy error beween he prediced p conexual sequence Ck and he ground ruh C o k. The loss uncion o he kh sequence can be: o p o p H C, C c log c, ⑴ k k k nk nk n where c o nk and c p nk are he nh dimension o ground ruh conexual label vecor and prediced one separaely. In addiion, we adop one-ho represenaion (a vecor wih a speciic dimension se o 1 while all oher dimensions 0) or he conexual label (phoneme label and/or HMM sae label). We eed orward he DBLSTM boleneck eaure exracor like radiional DBLSTM and adop he back propagaion hrough ime (BPTT) algorihm o rain he nework [19]. By applying he chain rule, we could obain ha: H w T ij H wij a j, ⑵ w a w where ij 1 j ij wij denoes he weigh uni i and uni j, and a j indicaes he inpu acivaions o uni j a ime. Then we could eed back he nework error and rain he exracor as radiional DBLSTM Experimenal seup 5. EXPERIENMENTS This work adoped an emphaic audio visual daabase wih 700 English uerances spoken by a emale, including 350 emphaic and 350 neural uerances, or experimenaion. The video rame rae is 25ps and well ormed in AVI orma. We divided he whole corpus ino 3 pars randomly, 600 uerances as raining se, 60 as es se and ohers as validaion se. To evaluae he proposed approach, we ake he advanage o he well-known objecive evaluaions crierion roo mean squared error (RMSE) and correlaion coeicien (CORR) beween he prediced FAPs and he ground ruh. These merics are deined as: where RMSE CORR M 2 T Tk o p 1 1 k N ap T k 1 k Tk o p corr, 1 1 k T k 1 k, ⑶, ⑷ denoes he scale o he es se, T k is he rame number o he kh es se, Fig 2. DBLSTM boleneck eaure exracor o and p mean he rame wise ground ruh and he prediced resul separaely, corr(.) indicaes he correlaion coeicien beween he wo vecors o each rame. Dieren archiecure o he DBLSTM mapping model may aec he predicion accuracy rom acousic eaures o FAPs. To exensively explore he perormances o dieren acousic eaures, we conduced experimens on several nework opologies or DBLSTM mapping model wih dieren numbers o hidden layers and numbers o hidden unis per hidden layer. For boh he DBLSTM neworks, he learning rae is se o be 1e-4, and he momenum is 0.9. In addiion, we choose he seepes opimizer and consider he nework o be he bes when i mees he sopping crierion Objecive experimen on LLD and MFCC-RMSE To compare he perormance o he proposed LLD eaure and he radiional eaure, a 39 dimensional MFCC-RMSE eaure conaining 12 order MFCCs and RMSE ogeher wih heir irs and second order derivaives was used as he baseline. The MFCC- RMSE and LLD were exraced wih 25ms rame size and 10ms rame shi. The dimension o he LLD acousic eaure is 384. Experimenal resuls are shown in Table 3, where he second column means dieren nework srucures o DBLSTM mapping model wih one (100), wo ( ) or hree hidden layers ( ) respecively. The hidden uni numbers per hidden layer 5552

4 are all se o 100. As can be seen, MFCC-RMSE perorms beer han LLD on he condiion ha BLSTM srucure is simple, such as one or wo bidirecional hidden layers. While when he nework goes deeper wih hree hidden layers, LLD ouperorms MFCC- RMSE signiicanly indicaing ha high dimensional LLD carries more inormaion ha can be learned wih complex deep nework Objecive experimen on boleneck eaures Furhermore, o compare he perormance o dieren conexual levels o boleneck eaures, we rained wo kinds o DBLSTM boleneck eaure exracors or exracing phoneme level and HMM sae level boleneck eaures respecively. The phoneme level boleneck eaure was exraced by a well-rained DBLSTM wih he oupu layer o 41 dimensional phoneme labels, while he HMM sae level boleneck eaure was exraced wih he oupu layer o 123 (41 3) dimensional HMM sae label. As or he srucure o DBLSTM boleneck eaure exracor, we adoped a commonly used DBLSTM srucure. The nework conains hree bidirecional hidden layers, wih 100, 100 and 40 hidden unis or each hidden layer respecively [17]. The srucure o DBLSTM mapping model is he same as he above experimen. Experimenal resuls are shown in Table 4. The irs column indicaes dieren kinds o boleneck eaures. MFCC-phoneme (LLD-phoneme) is he MFCC-RMSE (LLD) based phoneme level boleneck eaure where he DBLSTM eaure exracor is rained wih MFCC-RMSE (LLD) and phoneme labels; while MFCC-sae (LLD-sae) means he DBLSTM is rained wih MFCC-RMSE (LLD) and HMM sae labels. From Table 3 and Table 4, we can ind ha, or he same archiecure o DBLSTM mapping model, boleneck eaure shows superior perormance or boh RMSE and CORR. Furhermore, we can see ha boleneck eaures a sae level perorm bes or he complex and deep nework srucure o DBLSTM mapping model Subjecive evaluaion We urher conduced a se o subjecive experimens o evaluae he nauralness o synheic animaion o he alking avaar driven by he above menioned our dieren acousic eaures. 10 speech uerances were randomly seleced rom he es se and used o generae synheic lip movemens, acial expressions as well as head moions on a 3D alking avaar. The synheic visual animaion ogeher wih he acousic speech inpu were saved as 10 video iles. 20 subjecs were asked o wach he video ile and hen assign a score a 5-poin scale or each ile based on nauralness and synchronizaion beween he visual animaions and acousic speech. Higher score means more naural and closer synchrony. The mean opinion score (MOS) over 10 speech uerances and 20 subjecs are compued and presened in Table 5. As can be seen, he MOS score o raw LLD eaure is a bi higher han MFCC- RMSE eaure, while LLD based boleneck eaures lead o urher perormance improvemen. Furhermore, LLD based boleneck eaure o sae labels achieves he bes MOS score. The resuls sugges ha our proposed LLD based boleneck eaure is able o capure inheren inormaion o lip movemens, acial expressions and head moions simulaneously rom acousic speech signals, and can achieve more naure speech driven alking avaar. 6. CONCLUSION In his paper, we invesigaed new acousic eaures and proposed LLD based DBLSTM boleneck eaure or speech driven alking Table 3. Resuls or LLD and MFCC-RMSE, where Map Model means nework srucure o DBLSTM mapping model Acousic Feaure Map Model RMSE CORR MFCC-RMSE MFCC-RMSE MFCC-RMSE LLD LLD LLD Table 4. Resuls or dieren boleneck eaures, where Map Model mains nework srucure o DBLSTM mapping model Boleneck Feaure Map Model RMSE CORR MFCC-phoneme MFCC-phoneme MFCC-phoneme LLD-phoneme LLD-phoneme LLD-phoneme MFCC-sae MFCC-sae MFCC-sae LLD-sae LLD-sae LLD-sae Table 5. Resuls or subjecive evaluaion Acousic eaure MOS score MFCC-RMSE 3.3 LLD 3.4 LLD-phoneme 3.6 LLD-sae 3.8 avaar. Our work shows ha LLD can enhance he perormance o regression model or audio visual speech mapping. In addiion, he boleneck eaure shows srong represenaion abiliy and ges urher perormance improvemen. Speciically, boleneck eaure involving HMM sae conexual label inormaion perorms beer han he boleneck eaure wih phoneme label inormaion. In he uure, we plan o invesigae in deail he perormance o dieren LLD eaure ses or he ask. 7. ACKNOWLEDGEMENT This work is suppored by he Naional Basic Research Program o China (2012CB316401) and he Naional High Technology Research and Developmen Program o China (2015AA016305). This work is also parially suppored by he Naional Naural Science Foundaion o China (NSFC) ( , and ), he join und o NSFC-RGC (Research Gran Council o Hong Kong) ( and N_CUHK404/15) and he Major Program or Naional Social Science Foundaion o China (13&ZD189). 5553

5 8. REFERENCES [1] E. Yamamoo, S. Nakamura, K. Shikano, Lip movemen synhesis rom speech based on hidden Markov models, Speech Communicaion, vol. 26, pp , [2] G. Wang, M. Yang, C. Chiang, W. Tai, A alking ace driven by voice using hidden Markov model, Journal o inormaion science and engineering, vol. 22, pp , [3] N. Nalini, A. Chakrabory. Speech emoion recogniion using MFCC and AANN, in Proc. Inernaional Conerence on Engineering and Technology, pp , [4] Z. Zeng, J. Tu, M. Liu, T. Huang, B Pianei, D. Roh, S. Levinson, Audio-visual aec recogniion, IEEE Transacions on Mulimedia, vol. 9, pp , [5] M. Song, J. Bu, C. Chen, N. Li, Audio-visual based emoion recogniion-a new approach, Compuer Vision and Paern Recogniion CVPR, vol. 2, pp , [6] B. Schuller, A. Baliner, D. Seppi, S. Seidl, T. Vog, J. Wagner, L. Devillers, L. Vidrascu, N. Amir, L. Kessous, V. Aharonson, The relevance o eaure ype or he auomaic classiicaion o emoional user saes: low level descripors and uncionals, in Proc. Annual Conerence o Inernaional Speech Communicaion Associaion (INTERSPEECH), pp , [7] M. Wöllmer, F. Eyben, S. Reier, B. Schuller, C. Cox, E. Cowie, R. Cowie, Abandoning emoion classes-owards coninuous emoion recogniion wih modelling o longrange dependencies, in Proc. Annual Conerence o Inernaional Speech Communicaion Associaion (INTERSPEECH), pp , [8] L. Wang, X. Qian, W. Han, F. Soong, Synhesizing phooreal alking head via rajecory-guided sample selecion, in Proc. Annual Conerence o Inernaional Speech Communicaion Associaion (INTERSPEECH), vol. 10, pp , [9] B. Fan, L. Wang, F. Soong, L. Xie, Phoo-real alking head wih deep bidirecional LSTM, in Proc. Inernaional Conerence on Acousics, Speech and Signal Processing (ICASSP), pp , [10] P. Hong, Z. Wen, T. Huang, Real-ime speech-driven ace animaion wih expressions using neural neworks, IEEE Transacions on Neural Neworks, vol. 13, pp , [11] J. Jia, S. Zhang, F. Meng, Y. Wang, L. Cai, Emoional audio-visual speech synhesis based on PAD, IEEE Transacions on Audio, Speech, and Language Processing, vol. 19, pp , [12] S. Zhang, Z. Wu, H. Meng, L. Cai, Facial expression synhesis using PAD emoional parameers or a Chinese expressive avaar, Aecive Compuing and Inelligen Ineracion, Springer, Berlin Heidelberg, pp , [13] J. Jia, Z. Wu, S. Zhang, H. Meng, L. Cai, Head and acial gesures synhesis using PAD model or an expressive alking avaar, Mulimedia Tools and Applicaions, vol. 73, pp , [14] C. Ding, L. Xie, P. Zhu, Head moion synhesis rom speech using deep neural neworks, Mulimedia Tools and Applicaions, pp. 1-18, [15] S. Hochreier, J. Schmidhuber, Long shor-erm memory, Neural Compuaion, vol. 9, pp , [16] F. Grézl, M. Karaiá, S. Konár, J. Cernocky, Probabilisic and boleneck eaures or LVCSR o meeings, in Proc. Inernaional Conerence on Acousics, Speech and Signal Processing (ICASSP), vol. 4, pp , [17] M. Wöllme, B. Schuller, G. Rigoll, A novel boleneck- BLSTM ron-end or eaure-level conex modeling in conversaional speech recogniion, IEEE Workshop on Auomaic Speech Recogniion and Undersanding (ASRU), pp , [18] Visage SDK - Face racking ools [OL]. [ ]. hp:// [19] R. Williams, D. Zipser, Gradien-based learning algorihms or recurren neworks and heir compuaional complexiy, Back-propagaion: Theory, archiecures and applicaions, pp , [20] A. Graves, Supervised Sequence Labelling wih Recurren Neural Neworks, Springer, Berlin, vol. 385, pp. 5-13, [21] Z. Wu, S. Zhang, L. Cai, H. Meng, Real-ime synhesis o Chinese visual speech and acial expressions using MPEG-4 FAP eaures in a hree-dimensional avaar, in Proc. Inernaional Conerence on Spoken Language Processing (ICSLP), pp , [22] Schuller B, Seidl S, Baliner A, The INTERSPEECH 2009 emoion challenge, in Proc. Annual Conerence o Inernaional Speech Communicaion Associaion (INTERSPEECH), pp ,