GESTURE RECOGNITION RESEARCH FOR HUMAN-MACHINE SYMBIOTIC ENVIRONMENT. T.Kirishima 1, K.Sato 2, and K.Chihara 3

Transcription

1 GESTURE RECOGNITION RESEARCH FOR HUMAN-MACHINE SYMBIOTIC ENVIRONMENT T.Kirishima 1, K.Sao 2, ad K.Chihara 3 1 Deparme of Elecrical Egieerig, Nara Naioal College of Techology, 22 Yaa-cho, Yamaokoriyama-shi, Nara, Japa 2 Graduae School of Egieerig Sciece, Osaka Uiversiy, 1-3 Machikaeyama-cho, Toyoaka-shi, Osaka, Japa 3 Graduae School of Iformaio Sciece, Nara Isiue of Sciece ad Techology, Takayama-cho, Ikoma-shi, Nara, Japa ABSTRACT: The drasic icrease i compuig performace of mobile sesig devices will soo rigger emergece ad permeaio of ubiquious compuig echologies. Breakhroughs i hardware echology are already demadig ovel applicaios for iellige robos, securiy, elemedicie, ad virual realiy. Gesure recogiio echology ha eables he recogiio of huma aural behaviors i boh real ad virual worlds ca play impora roles ad will become idispesable i our fuure lives. Gesure recogiio research is a field ha aemps o make compuers udersad huma ieios ad body cofiguraios hrough visual observaios of heir behaviors ad aciviies. I ca coribue as a huma ierface i boh real ad virual worlds. 1. INTRODUCTION Everyoe kows ha huma body movemes ca bear rich meaigs i huma-o-huma commuicaio. We call i a "gesure." Today, i bega o emerge as a meas for ieracio wih digial compuers, or iellige machies. Usig a gesure as a meas for huma-machie commuicaio is a ew paradigm ha leads o huma-machie symbioic eviromes i fuure ubiquious sociey. For he symbiosis of humas ad machies, machies are required o udersad huma gesures[1]. Thaks o he iovaios i compuer hardware echology, eve a iy mobile compuig devices ca ow process sreams of visual iformaio i real-ime. The res is how o impleme ad orgaize he machie ielligece ha will widely spread i he real world. I his paper, mulilaeral aspecs of huma gesures are briefly oed ad some of he boleecks are explaied ha researchers frequely ecouer whe cosrucig a gesure recogiio sysem. The, some ideas are preseed o how o desig ad how o evaluae a gesure recogiio sysem. 2. IS GESTURE RECOGNITION AN EASY-TO-SOLVE PROBLEM? Today, gesure recogiio sysems are beig developed worldwide ad oe ca easily fid highly successful sysems. Some may hik ha mos of gesure recogiio problems i erms of egieerig have already bee resolved ad hey belog o easy-o-solve problems. I my view, oly a par of problems ha are easy o solve has bee solved sice gesure-relaed research belogs o a ierdiscipliary domai ha is so wide ad so deep. Before dealig wih gesures i erms of egieerig, mulilaeral aspecs of gesures eed o be addressed. 2.1 Social / Psychological / Cogiive Aspecs Gesures are used by people of ay age, ay sexualiy, ad ay race. The use of body movemes is uiversal ad idispesable for acquirig ma's such basic skills as followig: (1) The abiliy o behave ad ierac wih objecs / people i he real world. (E.g., eaig, ouchig, graspig, walkig, collisio deecio ad avoidace) (2) The abiliy o acquire oe s ow body image. (E.g., ieracio wih mirrors i order o kow I am real ad a social exisece )

2 Moreover, he use of gesures is idispesable for developig ma's such social commuicaio skills as followig: (1) Udersadig basic emoios ad ieios of commuicaio parer from his/her behavior[2]. (E.g., eural, ager, sadess, happiess, fear, surprise, disgus, ad easig.) (2) Udersadig Body Laguage. Each ypical body moio ad is meaig are loosely defied i a arbirary maer. This requires oly a local agreeme. (3) Udersadig Sig Laguage. Special body cofiguraio, moio, ad is meaigs are sricly defied ad shared i deaf commuiy. This requires a log-erm educaio ad raiig. I a social siuaio, gesures ha are based o commuicaio proocols are iuiive, flexible, ad easy o udersad ad share. Wihou commuicaio proocols, gesures are ofe ambiguous, misleadig, ad difficul o udersad ad share. How could we make use of such gesures i he coex of egieerig? 2.2 Egieerig Aspecs Gesure recogiio echiques are paricularly useful i our daily domais or coexs: * Smar / Iellige Homes - Securiy use (iruder deecio / behavior moiorig / perso ideificaio by gai aalysis) - Appliace use (room-guidace / auomaic corol of lighig, air-codiioig ad veilaio) - Welfare use (ursig-care a home by parer robos ha ca udersad huma ieio) * Huma-Compuer Ieracio (HCI) - Eeraime (iuiive corol of ieracive games ad avigaio i virual worlds) - Repeiive sress ijuries (RSI) sydromes preveio (especially, avoidig eosyoviis) - Physical exercises assisace (bedridde-sae preveio, caregiver s disracio suppor) Especially, i HCI, gesures ca be a excelle meas for: - Operaig a compuer (subsiuio for keyboards ad mice) - Navigaio i VR / AR / MR eviromes (walkhroughs, issuig gesure commads) - Ieracio wih a virual age / objec (CG) (direc commuicaio / maipulaio) - Ieracio wih a real age / objec (robos) (direc commuicaio / maipulaio) Apparely, he usabiliy of gesures i above domais is promisig ad we have wiessed may excelle demosraios. Bu oly small porios of hese are pu io pracical use. Why? 3. ARE THERE ANY TECHNICAL BOTTLENECKS? 3.1 Hardware aspec Alhough hardware resources are limied (e.g., limied umber of processors, ad limied amou of memory), hardware-relaed limiaios are o a obsacle o gesure recogiio research sice hardware performace ad archiecures have bee dramaically improved. 3.2 Sesor aspec Takig a gesure is a spaio-emporal eve. If oe wishes o deal wih all kids of gesure, he highes spaio-emporal resoluio will be required. I realiy, mos of visio sesors ha are available oday ad ailored o fi huma visio properies sricly operae a fixed samplig rae ad a fixed resoluio (NTSC, PAL, SECAM, ec). Uforuaely, facial or had gesures could require much higher spaial or emporal resoluio. Validiy of paricular spaial or emporal resoluio cao be judged wihou cosiderig he aure of arge gesures. Poeially, here is a grea eed for a visio sesor ha ca chage is samplig rae ad spaial resoluio dyamically respodig o he demads from compuer visio algorihms. 3.3 Sofware/Algorihm aspec Samplig problem

3 Samplig heorem is basically applicable o he sigals of oe-dimesioaliy. I provides crieria for he miimum samplig frequecy o recover or approximae origial sigal. Bu, is i also applicable o he gesure image samplig problem? Lack of samplig heory ha akes recogiio rae io accou suggess ha he ideal emporal resoluio cao be deermied heoreically. The, how ca ideal samplig rae be deermied ha could appropriaely classify he arge acios? I is clear ha a se of similar gesures could require much higher samplig rae ha a se of disic gesures. The ideal samplig rae will deped o he combiaio ad he complexiy of he gesures o be recogized Focus of aeio / spaial-segmeaio problem To figure ou he meaig of give gesures, oe eeds o kow which par of body o see ad wha kid of moio feaures o see. This kowledge or proocol is formed hrough he face-o-face commuicaio i our daily lives. This proocol-formig aciviy requires he so-called focus of aeio capabiliy. Uforuaely, i is o always evide which par of he body o exrac from a image. From image recogiio perspecive, he body pars exracio is o a prerequisie process. Whe we ry o ake or ierpre gesural acios, we usually pay greaer aeio o he movig pars of body. Acually, we ucosciously observe boh regios of chage ad regios of lile or o chage sice he combiaio of hese regios could be very useful segmeaio cues ha reflec he siger's ieio. A gesure is ake ad is udersood accordig o his visual commuicaio proocols. Formig ad sharig hese proocols o gesures should grealy faciliae he commuicaio bewee huma ad machies[3] Temporal-segmeaio problem Commuicaio proocols sipulae whe a gesure begis, coiues, ad eds. Siged gesures are usually complex ad cocurrely use facial gesures, sipulaed had posures ad movemes. They usually accompay orgaized, sysemaic, ad srog rules, ad require laguage level udersadig. Oly well-raied people ca udersad sig laguage commuicaio sice i requires emporal-segmeaio i ligh of coexs. Sig laguage proocols ha are cosise ad precisely defied are ecessary. These are acquired hrough iesive raiig wih a eacher. O he oher had, udersadig of local gesures requires oly proocols ha are arbirary ad emporary. Usually, local gesures are simple ad repeiive movemes. This suggess ha oly agreemes o spaial segmeaio cues are eeded. I realiy, compuaio of image differece is usually eough for he emporal segmeaio of local gesures. From above examples, we ca see ha here ca be a wide variey of huma commuicaio proocols. Whe we cosider he aure of Huma Compuer / Robo Ieracio, gesure proocol learig[3] is very useful sice i does o require iesive raiig o he par of user. A he same ime, proocol-based gesure recogiio provides he basis for sig laguage udersadig. I ca geerae a sequece of symbols ha will help symbolic represeaio ad ierpreaio of siged gesures. Temporal-segmeaio problems should be reaed separaely a differe levels Trade-off problem bewee processig speed ad recogiio rae I gesure recogiio sysems, we eed o pay aeio o boh processig speed ad recogiio rae. Parallel processig hardware ca saisfy boh requiremes a he same ime. Bu, oce he hardware archiecure is deermied, he maximum hardware performace is also fixed. This meas ha we cao expec much faser processig speed wihou a meas o corol i. Acually, he icrease i he umber of recogiio modules ad layers ieviably causes he icrease i compuaio ime. Bu, severe degradaio i recogiio performace is o allowed for real-ime applicaios. Uforuaely, i is iherely difficul o deec he recogiio rae for which here are o physical sesors. A mehod ha ca auomaically deec recogiio rae o-lie ad i real-ime is srogly required i order o maiai robusess ad accuracy of he recogiio sysem.

4 4. HOW DO WE DESIGN A GESTURE RECOGNITION SYSTEM? 4.1 Too may differe goals Ulike face recogiio research, here are may differe goals i gesure recogiio research. As show i Figure 1, arge body par iself could be head, arms, hads, orso, kee, legs, foo, ec., or combiaio of hese pars. The arge problem could be saic posure aalysis or dyamic moio aalysis. The required fucioaliy could be body pars or jois deecio, rackig, or recogiio. Moreover, some researchers may ry o deal wih he problems o muliple perso, specific perso, or uspecified perso. For he reasos described above, goals ad fudameal desig of a gesure recogiio sysem ca vary grealy. Of course, o sigle research projec could uderake all hese problems. For he ime beig, researchers will have o deal wih idividual or differe problems uder differe approaches uil hey fid ad share fudameal problems i his field. Figure 1: A huma body is made up of so may pars! 4.2 Mahemaical ools ad recogiio frameworks There are popular mahemaical ools such as HMM (Hidde Markov Models), NN (Neural Neworks), DP (Dyamic Programmig), ad SVM (Suppor Vecor Machies) ha have ofe bee applied o gesure recogiio problems. Surely, gesure recogiio is oe of he applicaio fields ha mahemaical ools ca hadle. Bu, is i possible o adequaely choose mahemaical ools wihou kowig each ool's advaages ad disadvaages? Also, is i possible o impleme sae-of-he-ar of each mahemaical ool ad o evaluae wih perfec imparialiy? For his, oe will have o apply as may mahemaical ools as possible ad compare heir resuls. Wihou ay selecio crieria for mahemaical ools, i should be difficul o say which oe is he bes. O he oher had, a recogiio framework also ca be emporary / problem-specific / comprehesive. Bu ay recogiio framework eeds o reflec he aure of arge problems. Ieresigly, researchers ed o develop similar recogiio framework whe he goals or problem defiiios are similar. Here, he same quesios arise. Is i possible o impleme sae-of-he-ar of recogiio framework ad o evaluae wih perfec imparialiy? For his, oe will have o impleme as may frameworks as possible ad compare heir resuls. Wihou ay selecio crieria for recogiio frameworks, i should be difficul o say which oe is he bes. Geerally, researchers who are more ieresed i he usefuless of mahemaical ools, hey usually develop a sequeial processig flow as show i Figure 2, focusig o a paricular mahemaical ool. O he oher had, researchers who are more ieresed i he framework for gesure recogiio, hey usually develop a parallel processig flow as show i Figure 3, focusig o a paricular recogiio framework. Ieresigly, i eiher case, researchers fially oice ha boh he mahemaical ools ad he recogiio framework are impora ad ecessary. I summary, i he desig phase, i is recommedable o cosider followig opics before implemeig he sysem. (1) Problem formulaio ad defiiio (his should o be affeced by he red of he imes) (2) Desig of recogiio framework (his could be affeced by he hardware/os evirome) (3) Selecio of mahemaical ools (his will be affeced by he red of he imes)

5 Mahemaical Tool A, B, C, ec Image Sequece Segmeaio Feaure Exracio Learig / Evaluaio Resuls Preprocessig Figure 2: A sequeial processig flow for gesure recogiio Re-samplig Framework Image Sequece S e g m e a i o Feaure Exracio Feaure Exracio Feaure Exracio Learig / Mahemaical Tool A Mahemaical Tool B Learig / Learig / I e g r a i o Evaluaio Resuls Mahemaical Tool C Preprocessig Similariy-based Fusio Figure 3: A parallel processig flow for gesure recogiio 5. HOW DO WE EVALUATE A GESTURE RECOGNITION SYSTEM? Alhough a evaluaio mehod for gesure recogiio sysems deserves a research i iself, a more sophisicaed ad reliable evaluaio framework is eeded o succicly compare he resuls of gesure recogiio researches. Currely, followig wo issues are he major cocers. 5.1 Lack of sadardized gesure image daabase The problem of evaluaio is closely relaed o he problem of wha should be recogized. As has bee meioed i secio 4.1, i gesure recogiio research, here are oo may variaios i recogiio arges. This suggess ha each arge may require domai-specific image daabases. For his reaso, he use of sadardized gesure image daabase is o regarded as madaory amog mos of he researchers i his field. Bu, he eed for sadardized gesure image daabase arises oce commo goals o cerai opics are shared amog researchers. For example, we ca obai sadardized image daabases for perso ideificaio by gai. By sharig sadardized image daabases, he compariso of resuls amog differe approaches ca be doe very easily. Bu, sadard gesure image daabases will have o accommodae localiy-specific movemes ad heir meaigs for sig laguage recogiio problems. We kow some cases i which differe meaigs are allocaed o he same movemes i differe pars of he world. For localiy-specific problems, localiy-specific daabases will have o be developed. For he reasos described above, i is difficul o expec ay sadardized gesure image daabases ha ca saisfy all problem domais i his field. Probably, he old-fashioed approach of accumulaig daabase maerials a oe place, a oe ime, ad a oe orgaizaio cao provide beer sadard gesure image daabases. Disribued daabases ha ay researchers ca easily access ad coribue ayime over he Iere will be oe of he soluios o his problem.

6 5.2 Lack of sadardized evaluaio procedure By sharig a sadardized evaluaio procedure, i will be easier o compare improvemes amog differe approaches. Bu, is i really possible o sadardize a evaluaio procedure? Currely, recogiio raes are calculaed afer huma judgme o he give resuls. The formulas ad he mehods o obai recogiio raes are o always he same amog researchers. Moreover, we have o admi he fac ha researchers cao always follow he same evaluaio procedure. Bu we ca be opimisic by akig aoher approach. Esseially, wha is required is he compariso of differe approaches uder he same experimeal framework ad codiios. This will be possible by exploiig a recogiio framework as show i Figure 3. A recogiio framework ha ca accommodae differe algorihms ad iegrae heir resuls will provide he basis for he compariso. Especially, full-auomaic or semi-auomaic evaluaio procedure is impora. I will help guaraee he reproducibiliy of he evaluaio experimes. Moreover, by auomaig he evaluaio procedure, oe ca esablish a well-kow P (Pla) - D (Do) - C (Check) - A (Acio) cycle ha eables he parameer-uig of he recogiio algorihm, hece, leadig o a framework ha ca opimize he recogiio performace for paricular eeds (see Figure 4). P ( Pla ) D ( Do ) C ( Check ) Problem Defiiio ad Requireme Aalysis Sysem Desig Sysem Implemeaio Parameer Tuig / Opimizaio Algorihms Refieme / Replaceme Performace Evaluaio Raes, Processig Speed, ec A ( Acio ) Figure 4: P-D-C-A framework for he research o gesure recogiio. 6. SUMMARY Cosiderig he ierdiscipliary aure of gesure recogiio research, i is sill a he begiig sage. There are oo may opics o be sudied. Foruaely, various kids of mahemaical ools have bee acively examied ad some have prove o be effecive. Bu, apar from hese mahemaical ools, discoverig a irisic problem o gesure recogiio is paricularly impora sice i requires ovel approaches ha will lead o he rue advaceme i his field. O he oher had, iovaios i hardware echology have made i possible o impleme gesure recogiio algorihms o various kids of hardware plaforms. To obai beer recogiio framework ad algorihms, researchers should share ew fidigs ad problems while akig advaage of he sae-of-he-ar hardware ad evirome. I fuure ma-machie symbioic ubiquious sociey, gesure recogiio sysems will be playig a promie role sice huma body-mediaed commuicaio wih iellige machies is esseial. REFERENCES [1] T. Masuyama: Preface, Proc. Firs I l Workshop Ma-Machie Symbioic Sysems, pp. iii-vii, Nov [2] R. Nakasu: Noverbal Iformaio ad Is Applicaio o Commuicaios, Proc. Third IEEE I l Cof. Auomaic Face ad Gesure (FG'98), pp.2-7, Apr [3] T. Kirishima, K. Sao, K. Chihara: Real-Time Gesure by Learig ad Selecive Corol of Visual Ieres Pois, IEEE Tras. Paer Aalysis ad Machie Ielligece, Vol. 27, No.3, pp , Mar