Compuer ercepual Vision and Sensor II Summer 4 Augmened Compuing Compuer Vision II Lecure 8 Tracking wih Linear Dnamic Models 2.5.24 Basian Leibe RWTH Aachen hp://www.vision.rwh-aachen.de leibe@vision.rwh-aachen.de
Compuer ercepual Vision and Sensor II Summer 4 Augmened Compuing Course Ouline Single-Objec Tracking Background modeling Templae based racking Color based racking Conour based racking Tracking b online classificaion Tracking-b-deecion Baesian Filering Kalman filer aricle filer Muli-Objec Tracking Ariculaed Tracking 2 Image source: Helmu Grabner Disne/ixar
Compuer ercepual Vision and Sensor II Summer 4 Augmened Compuing Recap: Tracking-b-Deecion Main ideas Appl a generic objec deecor o find objecs of a cerain class Based on he deecions exrac objec appearance models Link deecions ino rajecories 3
Compuer ercepual Vision and Sensor II Summer 4 Augmened Compuing Recap: Sliding-Window Objec Deecion Fleshing ou his pipeline a bi more we need o:. Obain raining daa 2. Define feaures 3. Define classifier Training examples Feaure exracion Car/non-car Classifier Slide credi: Krisen Grauman 4
Compuer ercepual Vision and Sensor II Summer 4 Augmened Compuing Recap: Objec Deecor Design In pracice he classifier ofen deermines he design. Tpes of feaures Speedup sraegies We ll look a 3 sae-of-he-ar deecor designs Based on SVMs Las lecure Based on Boosing Las lecure Based on Random Foress osponed o a laer slo... 5
Compuer ercepual Vision and Sensor II Summer 4 Augmened Compuing Recap: Hisograms of Oriened Gradiens (HOG) Holisic objec represenaion Localized gradien orienaions [............] Objec/Non-objec Linear SVM Collec HOGs over deecion window Conras normalize over overlapping spaial cells Weighed voe in spaial & orienaion cells Compue gradiens Gamma compression Image Window Slide adaped from Navnee Dalal 6
Compuer ercepual Vision and Sensor II Summer 4 Augmened Compuing Recap: Deformable ar-based Model (DM) Score of filer: do produc of filer wih HOG feaures underneah i Score of objec hpohesis is sum of filer scores minus deformaion coss Muliscale model capures feaures a wo resoluions Slide credi: edro Felzenszwalb 7 [Felzenszwalb McAlliser Ramanan CVR 8]
Compuer ercepual Vision and Sensor II Summer 4 Augmened Compuing Recap: DM Hpohesis Score Slide credi: edro Felzenszwalb 8 [Felzenszwalb McAlliser Ramanan CVR 8]
Compuer ercepual Vision and Sensor II Summer 4 Augmened Compuing Recap: Inegral Channel Feaures Generalizaion of Haar Wavele idea from Viola-Jones Insead of onl considering inensiies also ake ino accoun oher feaure channels (gradien orienaions color exure). Sill efficienl represened as inegral images.. Dollar Z. Tu. erona S. Belongie. Inegral Channel Feaures BMVC 9. 9
Compuer ercepual Vision and Sensor II Summer 4 Augmened Compuing Recap: Inegral Channel Feaures Generalize also block compuaion s order feaures: Sum of pixels in recangular region. 2 nd -order feaures: Haar-like difference of sum-over-blocks Generalized Haar: More complex combinaions of weighed recangles Hisograms Compued b evaluaing local sums on quanized images.
Compuer ercepual Vision and Sensor II Summer 4 Augmened Compuing Recap: VerFas Deecor Idea : Inver he relaion model 5 image scales 5 models image scale R. Benenson M. Mahias R. Timofe L. Van Gool. edesrian Deecion a Frames per Second CVR 2. Slide credi: Rodrigo Benenson
Compuer ercepual Vision and Sensor II Summer 4 Augmened Compuing Recap: VerFas Deecor Idea 2: Reduce raining ime b feaure inerpolaion 5 models image scale 5 models image scale Shown o be possible for Inegral Channel feaures. Dollár S. Belongie erona. The Fases edesrian Deecor in he Wes BMVC 2. Slide adaped from Rodrigo Benenson 2
Compuer ercepual Vision and Sensor II Summer 4 Augmened Compuing Recap: VerFas Classifier Consrucion + - + - + - + - + - + - score = w h + w 2 h 2 + Ensemble of shor rees learned b AdaBoos Slide credi: Rodrigo Benenson +w N h N 3
Compuer ercepual Vision and Sensor II Summer 4 Augmened Compuing Elemens of Tracking Deecion Daa associaion redicion Deecion Where are candidae objecs? Las lecure Daa associaion Which deecion corresponds o which objec? redicion Toda s opic Where will he racked objec be in he nex ime sep? 4
Compuer ercepual Vision and Sensor II Summer 4 Augmened Compuing Toda: Tracking wih Linear Dnamic Models 5 Figure from Forsh & once
Compuer ercepual Vision and Sensor II Summer 4 Augmened Compuing Topics of This Lecure Tracking wih Dnamics Deecion vs. Tracking Tracking as probabilisic inference redicion and Correcion Linear Dnamic Models Zero veloci model Consan veloci model Consan acceleraion model The Kalman Filer Kalman filer for D sae General Kalman filer Limiaions 6
Compuer ercepual Vision and Sensor II Summer 4 Augmened Compuing Tracking wih Dnamics Ke idea Given a model of expeced moion predic where objecs will occur in nex frame even before seeing he image. Goals Resric search for he objec Improved esimaes since measuremen noise is reduced b rajecor smoohness. Assumpion: coninuous moion paerns Camera is no moving insanl o new viewpoin. Objecs do no disappear and reappear in differen places. Gradual change in pose beween camera and scene. Slide adaped from S. Lazebnik K. Grauman 7
Compuer ercepual Vision and Sensor II Summer 4 Augmened Compuing General Model for Tracking Represenaion The moving objec of ineres is characerized b an underling sae. Sae gives rise o measuremens or observaions Y. A each ime he sae changes o and we ge a new observaion Y. 2 Y Y 2 Y Slide credi: Svelana Lazebnik 8
Compuer ercepual Vision and Sensor II Summer 4 Augmened Compuing Sae vs. Observaion Hidden sae : parameers of ineres Measuremen: wha we ge o direcl observe Slide credi: Krisen Grauman 9
Compuer ercepual Vision and Sensor II Summer 4 Augmened Compuing Tracking as Inference Inference problem The hidden sae consiss of he rue parameers we care abou denoed. The measuremen is our nois observaion ha resuls from he underling sae denoed Y. A each ime sep sae changes (from - o ) and we ge a new observaion Y. Our goal: recover mos likel sae given All observaions seen so far. Knowledge abou dnamics of sae ransiions. Slide credi: Krisen Grauman 2
Compuer ercepual Vision and Sensor II Summer 4 Augmened Compuing Seps of Tracking redicion: Wha is he nex sae of he objec given pas measuremens? Y Y Correcion: Compue an updaed esimae of he sae from predicion and measuremens. Y Y Tracking can be seen as he process of propagaing he poserior disribuion of sae given measuremens across ime. Y 2
Compuer ercepual Vision and Sensor II Summer 4 Augmened Compuing Simplifing Assumpions Onl he immediae pas maers Dnamics model Measuremens depend onl on he curren sae Y Y Y Y Observaion model 2 Slide credi: Svelana Lazebnik Y Y 2 Y 22
Compuer ercepual Vision and Sensor II Summer 4 Augmened Compuing Tracking as Inducion Base case: Assume we have iniial prior ha predics sae in absence of an evidence: ( ) A he firs frame correc his given he value of Y = ( ) ( ) ( Y ) ( ) ( ) ( ) oserior prob. of sae given measuremen Likelihood of measuremen rior of he sae Slide credi: Svelana Lazebnik 23
Compuer ercepual Vision and Sensor II Summer 4 Augmened Compuing Tracking as Inducion Base case: Assume we have iniial prior ha predics sae in absence of an evidence: ( ) A he firs frame correc his given he value of Y = Given correced esimae for frame : redic for frame + Correc for frame + predic correc Slide credi: Svelana Lazebnik 24
ercepual and Sensor Augmened Compuing Compuer Vision II Summer 4 Inducion Sep: redicion redicion involves represening given 25 d d d Law of oal probabili Slide credi: Svelana Lazebnik A A B db
ercepual and Sensor Augmened Compuing Compuer Vision II Summer 4 Inducion Sep: redicion redicion involves represening given 26 d d d Slide credi: Svelana Lazebnik Condiioning on A B A B B
ercepual and Sensor Augmened Compuing Compuer Vision II Summer 4 Inducion Sep: redicion redicion involves represening given 27 d d d Slide credi: Svelana Lazebnik Independence assumpion
ercepual and Sensor Augmened Compuing Compuer Vision II Summer 4 Inducion Sep: Correcion Correcion involves compuing given prediced value 28 d Baes rule B A A A B B Slide credi: Svelana Lazebnik
ercepual and Sensor Augmened Compuing Compuer Vision II Summer 4 Inducion Sep: Correcion Correcion involves compuing given prediced value 29 d Independence assumpion (observaion depends onl on sae ) Slide credi: Svelana Lazebnik
ercepual and Sensor Augmened Compuing Compuer Vision II Summer 4 Inducion Sep: Correcion Correcion involves compuing given prediced value 3 d Slide credi: Svelana Lazebnik Condiioning on
Compuer ercepual Vision and Sensor II Summer 4 Augmened Compuing Summar: redicion and Correcion redicion: d Dnamics model Correced esimae from previous sep Slide credi: Svelana Lazebnik 3
Compuer ercepual Vision and Sensor II Summer 4 Augmened Compuing Summar: redicion and Correcion redicion: d Dnamics model Correced esimae from previous sep Correcion: Observaion model rediced esimae d Slide credi: Svelana Lazebnik 32
Compuer ercepual Vision and Sensor II Summer 4 Augmened Compuing Topics of This Lecure Tracking wih Dnamics Deecion vs. Tracking Tracking as probabilisic inference redicion and Correcion Linear Dnamic Models Zero veloci model Consan veloci model Consan acceleraion model The Kalman Filer Kalman filer for D sae General Kalman filer Limiaions 33
Compuer ercepual Vision and Sensor II Summer 4 Augmened Compuing Noaion Reminder x ~ N( μ Σ) Random variable wih Gaussian probabili disribuion ha has he mean vecor ¹ and covariance marix. x and ¹ are d-dimensional is d d. d=2 d= If x is D we jus have one parameer: he variance ¾ 2 Slide credi: Krisen Grauman 34
Compuer ercepual Vision and Sensor II Summer 4 Augmened Compuing Linear Dnamic Models Dnamics model Sae undergoes linear ranformaion D plus Gaussian noise x ~ N D x d n nn n Observaion model Measuremen is linearl ransformed sae plus Gaussian noise ~ N M x m m mn n Slide credi: S. Lazebnik K. Grauman 35
Compuer ercepual Vision and Sensor II Summer 4 Augmened Compuing Example: Randoml Drifing oins Consider a saionar objec wih sae as posiion. osiion is consan onl moion due o random noise erm. x p p p Sae evoluion is described b ideni marix D=I x D x noise Ip noise Slide credi: Krisen Grauman 36
Compuer ercepual Vision and Sensor II Summer 4 Augmened Compuing Example: Consan Veloci (D oins) Measuremens Saes Slide credi: Krisen Grauman ime 37 Figure from Forsh & once
ercepual and Sensor Augmened Compuing Compuer Vision II Summer 4 Example: Consan Veloci (D oins) Sae vecor: posiion p and veloci v Measuremen is posiion onl 38 ) ( v v v p p v p x noise v p noise D x x (greek leers denoe noise erms) noise v p noise Mx Slide credi: S. Lazebnik K. Grauman
Compuer ercepual Vision and Sensor II Summer 4 Augmened Compuing Example: Consan Acceleraion (D oins) Slide credi: Krisen Grauman 4 Figure from Forsh & once
ercepual and Sensor Augmened Compuing Compuer Vision II Summer 4 Example: Consan Acceleraion (D oins) Sae vecor: posiion p veloci v and acceleraion a. Measuremen is posiion onl 4 ) ( ) ( a a a v v v p p a v p x noise a v p noise D x x (greek leers denoe noise erms) noise a v p noise Mx Slide credi: S. Lazebnik K. Grauman
Compuer ercepual Vision and Sensor II Summer 4 Augmened Compuing Example: General Moion Models Assuming we have differenial equaions for he moion E.g. for (undampened) periodic moion of a pendulum Subsiue variables o ransform his ino linear ssem 2 dp d p p p p2 p3 2 d d Then we have x p p p 2 3 p p ( ) p 2 p p ( ) p p 2 2 3 p 2 d p d 2 3 p D 43
Compuer ercepual Vision and Sensor II Summer 4 Augmened Compuing Topics of This Lecure Tracking wih Dnamics Deecion vs. Tracking Tracking as probabilisic inference redicion and Correcion Linear Dnamic Models Zero veloci model Consan veloci model Consan acceleraion model The Kalman Filer Kalman filer for D sae General Kalman filer Limiaions 44
Compuer ercepual Vision and Sensor II Summer 4 Augmened Compuing The Kalman Filer Kalman filer Mehod for racking linear dnamical models in Gaussian noise The prediced/correced sae disribuions are Gaussian You onl need o mainain he mean and covariance. The calculaions are eas (all he inegrals can be done in closed form). Slide credi: Svelana Lazebnik 45
Compuer ercepual Vision and Sensor II Summer 4 Augmened Compuing The Kalman Filer Know correced sae from previous ime sep and all measuremens up o he curren one redic disribuion over nex sae. Receive measuremen Know predicion of sae and nex measuremen Updae disribuion over curren sae. Time updae ( redic ) Measuremen updae ( Correc ) Mean and sd. dev. of prediced sae: Time advances: ++ Mean and sd. dev. of correced sae: Slide credi: Krisen Grauman 46
Compuer ercepual Vision and Sensor II Summer 4 Augmened Compuing Kalman Filer for D Sae Wan o represen and updae 2 x N ( ) 2 x N ( ) 47
Compuer ercepual Vision and Sensor II Summer 4 Augmened Compuing Slide credi: Svelana Lazebnik ropagaion of Gaussian densiies Shifing he mean Baesian combinaion Increasing he variance
Compuer ercepual Vision and Sensor II Summer 4 Augmened Compuing D Kalman Filer: redicion Have linear dnamic model defining prediced sae evoluion wih noise Wan o esimae prediced disribuion for nex sae 2 N ( ~ 2 N dx d ) Updae he mean: Updae he variance: Slide credi: Krisen Grauman ( d ) 2 2 d ( d ) 2 for derivaions see F& Chaper 7.3 49
Compuer ercepual Vision and Sensor II Summer 4 Augmened Compuing D Kalman Filer: Correcion Have linear model defining he mapping of sae o measuremens: 2 Y ~ N m mx Wan o esimae correced disribuion given laes measuremen: Updae he mean: Updae he variance: 2 N ( ) 5 Slide credi: Krisen Grauman Derivaions: F& Chaper 7.3 2 m 2 m m 2 m ( ( ) 2 2 2 m ( ) 2 2 m m ( ( ) ) 2 2 ) 2
Compuer ercepual Vision and Sensor II Summer 4 Augmened Compuing redicion vs. Correcion m ( ) 2 2 2 2 m 2 m ( 2 2 2 ) 2 2 2 m m ( ) m m ( ) ( ) Wha if here is no predicion uncerain 2 ( ) The measuremen is ignored! ( )? Wha if here is no measuremen uncerain Slide credi: Krisen Grauman ( ) m 2 The predicion is ignored! ( m )? 5
Compuer ercepual Vision and Sensor II Summer 4 Augmened Compuing posiion Recall: Consan Veloci Example measuremens sae ime Sae is 2D: posiion + veloci Measuremen is D: posiion Slide credi: Krisen Grauman 52 Figure from Forsh & once
Compuer ercepual Vision and Sensor II Summer 4 Augmened Compuing Consan Veloci Model o sae x measuremen * prediced mean esimae + correced mean esimae bars: variance esimaes before and afer measuremens Slide credi: Krisen Grauman 53 Figure from Forsh & once
Compuer ercepual Vision and Sensor II Summer 4 Augmened Compuing Consan Veloci Model o sae x measuremen * prediced mean esimae + correced mean esimae bars: variance esimaes before and afer measuremens Slide credi: Krisen Grauman 54 Figure from Forsh & once
Compuer ercepual Vision and Sensor II Summer 4 Augmened Compuing Consan Veloci Model o sae x measuremen * prediced mean esimae + correced mean esimae bars: variance esimaes before and afer measuremens Slide credi: Krisen Grauman 55 Figure from Forsh & once
Compuer ercepual Vision and Sensor II Summer 4 Augmened Compuing Consan Veloci Model o sae x measuremen * prediced mean esimae + correced mean esimae bars: variance esimaes before and afer measuremens Slide credi: Krisen Grauman 56 Figure from Forsh & once
Compuer ercepual Vision and Sensor II Summer 4 Augmened Compuing Kalman Filer: General Case (>dim) Wha if sae vecors have more han one dimension? REDICT CORRECT x D x D K T M M T D x x K Mx d I KM M T m residual More weigh on residual when measuremen error covariance approaches. for derivaions see F& Chaper 7.3 Slide credi: Krisen Grauman Less weigh on residual as a priori esimae error covariance approaches. 57
Compuer ercepual Vision and Sensor II Summer 4 Augmened Compuing Summar: Kalman Filer ros: Gaussian densiies everwhere Simple updaes compac and efficien Ver esablished mehod ver well undersood Cons: Unimodal disribuion onl single hpohesis Resriced class of moions defined b linear model Slide adaped from Svelana Lazebnik 58
Compuer ercepual Vision and Sensor II Summer 4 Augmened Compuing Wh Is This A Resricion? Man ineresing cases don have linear dnamics E.g. pedesrians walking E.g. a ball bouncing 59
Compuer ercepual Vision and Sensor II Summer 4 Augmened Compuing Ball Example: Wha Goes Wrong Here? Assuming consan acceleraion model redicion redicion is oo far from rue posiion o compensae ossible soluion: Keep muliple models Keep muliple differen moion models in parallel I.e. would check for bouncing a each ime sep redicion 2 redicion 3 redicion 4 redicion 5 Correc predicion 6
Compuer ercepual Vision and Sensor II Summer 4 Augmened Compuing References and Furher Reading A ver good inroducion o racking wih linear dnamic models and Kalman filers can be found in Chaper 7 of D. Forsh J. once Compuer Vision A Modern Approach. renice Hall 23 6