CS 4495 Compter Vision A. Bobick Featres 1: Harris CS 4495 Compter Vision Featres 1 Harris Aaron Bobick School of nteractie Compting Corners in A Corners in B
CS 4495 Compter Vision A. Bobick Featres 1: Harris Administriia PS 3: Will be ot tonight Sept 6. Will be de Snda Oct 6 th 11:55pm t is application of the last fe lectres. Mostl straight forard Matlab bt if o re linear algebra is rst it can take a hile to figre ot. Yo hae been arned t is cool Yo hae been arned Toda: Start on featres. Forsth and Ponce: 5.3-5.4 Szeliski also coers this ell Section 4 4.1.1 These net 3 lectres ill proide detail for Project 4.
CS 4495 Compter Vision A. Bobick Featres 1: Harris The basic image point matching problem Sppose hae to images related b some transformation. Or hae to images of the same object in different positions. Ho to find the transformation of image 1 that old align it ith image?
CS 4495 Compter Vision A. Bobick Featres 1: Harris We ant Local1 Featres Goal: Find points in an image that can be: Fond in other images Fond precisel ell localized Fond reliabl ell matched Wh? Want to compte a fndamental matri to recoer geometr Robotics/Vision: See ho a bnch of points moe from one frame to another. Allos comptation of ho camera moed -> depth -> moing objects Bild a panorama
CS 4495 Compter Vision A. Bobick Featres 1: Harris Sppose o ant to bild a panorama M. Bron and D. G. Loe. Recognising Panoramas. CCV 003
CS 4495 Compter Vision A. Bobick Ho do e bild panorama? Featres 1: Harris We need to match align images
CS 4495 Compter Vision A. Bobick Matching ith Featres Featres 1: Harris Detect featres featre points in both images
CS 4495 Compter Vision A. Bobick Featres 1: Harris Matching ith Featres Detect featres featre points in both images Match featres - find corresponding pairs
CS 4495 Compter Vision A. Bobick Featres 1: Harris Matching ith Featres Detect featres featre points in both images Match featres - find corresponding pairs Use these pairs to align images
CS 4495 Compter Vision A. Bobick Featres 1: Harris Matching ith Featres Problem 1: Detect the same point independentl in both images no chance to match! We need a repeatable detector
CS 4495 Compter Vision A. Bobick Featres 1: Harris Matching ith Featres Problem : For each point correctl recognize the corresponding one? We need a reliable and distinctie descriptor
CS 4495 Compter Vision A. Bobick Featres 1: Harris More motiation Featre points are sed also for: mage alignment e.g. homograph or fndamental matri 3D reconstrction Motion tracking Object recognition ndeing and database retrieal Robot naigation other
CS 4495 Compter Vision A. Bobick Characteristics of good featres Featres 1: Harris Repeatabilit/Precision The same featre can be fond in seeral images despite geometric and photometric transformations Salienc/Matchabilit ach featre has a distinctie description Compactness and efficienc Man feer featres than image piels Localit A featre occpies a relatiel small area of the image; robst to cltter and occlsion
CS 4495 Compter Vision A. Bobick Featres 1: Harris Finding Corners Ke propert: in the region arond a corner image gradient has to or more dominant directions Corners are repeatable and distinctie C.Harris and M.Stephens. "A Combined Corner and dge Detector. Proceedings of the 4th Ale Vision Conference: pages 147 151 1988
CS 4495 Compter Vision A. Bobick Featres 1: Harris Finding Harris Corners Ke propert: in the region arond a corner image gradient has to or more dominant directions Corners are repeatable and distinctie C. Harris and M.Stephens. "A Combined Corner and dge Detector. Proceedings of the 4th Ale Vision Conference: pages 147 151 1988
CS 4495 Compter Vision A. Bobick Featres 1: Harris Corner Detection: Basic dea We shold easil recognize the point b looking throgh a small indo Shifting a indo in an direction shold gie a large change in intensit Sorce: A. fros flat region: no change in all directions edge : no change along the edge direction corner : significant change in all directions ith small shift
CS 4495 Compter Vision A. Bobick Corner Detection: Mathematics Featres 1: Harris Change in appearance for the shift []: [ ] = Windo fnction Shifted intensit ntensit Windo fnction = or 1 in indo 0 otside Gassian Sorce: R. Szeliski
CS 4495 Compter Vision A. Bobick Corner Detection: Mathematics Featres 1: Harris Change in appearance for the shift []: [ ] = 00 3
CS 4495 Compter Vision A. Bobick Corner Detection: Mathematics Featres 1: Harris Change in appearance for the shift []: [ ] = We ant to find ot ho this fnction behaes for small shifts near 00 Second-order Talor epansion of abot 00 local qadratic approimation for small : df F δ F0 δ δ d 0 1 d F0 00 1 00 00 00 [ ] [ ] 00 00 00 d
Featres 1: Harris CS 4495 Compter Vision A. Bobick Corner Detection: Mathematics 00 00 00 00 ] [ 1 00 00 ] [ 00 [ ] = Second-order Talor epansion of abot 00: [ ] [ ] [ ] = = =
Featres 1: Harris CS 4495 Compter Vision A. Bobick Corner Detection: Mathematics 00 00 00 00 ] [ 1 00 00 ] [ 00 [ ] = Second-order Talor epansion of abot 00: [ ] [ ] [ ] = = =
Featres 1: Harris CS 4495 Compter Vision A. Bobick Corner Detection: Mathematics 00 00 00 00 ] [ 1 00 00 ] [ 00 [ ] = Second-order Talor epansion of abot 00: [ ] [ ] [ ] = = =
Featres 1: Harris CS 4495 Compter Vision A. Bobick Corner Detection: Mathematics 00 00 00 00 ] [ 1 00 00 ] [ 00 Second-order Talor epansion of abot 00: [ ] [ ] [ ] = = = [ ] =
Featres 1: Harris CS 4495 Compter Vision A. Bobick Corner Detection: Mathematics 00 00 00 00 ] [ 1 00 00 ] [ 00 alate at = 00: [ ] [ ] [ ] = = = = 0 = 0 = 0 [ ] =
Featres 1: Harris CS 4495 Compter Vision A. Bobick Corner Detection: Mathematics [ ] = Second-order Talor epansion of abot 00: 00 00 00 0 00 0 00 0 00 = = = = = = 00 00 00 00 ] [ 1 00 00 ] [ 00
Featres 1: Harris CS 4495 Compter Vision A. Bobick Corner Detection: Mathematics [ ] = Second-order Talor epansion of abot 00: ] [ 00 00 00 0 00 0 00 0 00 = = = = = =
CS 4495 Compter Vision A. Bobick Corner Detection: Mathematics The qadratic approimation simplifies to Featres 1: Harris [ ] M here M is a second moment matri compted from image deriaties: Withot eight M M = ach prodct is a rank 1
Featres 1: Harris CS 4495 Compter Vision A. Bobick The srface is locall approimated b a qadratic form. nterpreting the second moment matri M ] [ = M
CS 4495 Compter Vision A. Bobick nterpreting the second moment matri Consider a constant slice of : = k This is the eqation of an ellipse. [ ] M = Featres 1: Harris const
Featres 1: Harris CS 4495 Compter Vision A. Bobick = = 1 0 0 λ λ M First consider the ais-aligned case here gradients are either horizontal or ertical f either λ is close to 0 then this is not a corner so look for locations here both are large. nterpreting the second moment matri
Featres 1: Harris CS 4495 Compter Vision A. Bobick = = 1 0 0 λ λ M First consider the ais-aligned case here gradients are either horizontal or ertical f either λ is close to 0 then this is not a corner so look for locations here both are large. nterpreting the second moment matri
CS 4495 Compter Vision A. Bobick nterpreting the second moment matri Consider a horizontal slice of : [ ] M = This is the eqation of an ellipse. Diagonalization of M: M λ = R 1 1 0 R 0 λ The ais lengths of the ellipse are determined b the eigenales and the orientation is determined b R direction of the fastest change direction of the sloest change Featres 1: Harris const λ ma -1/ λ min -1/
CS 4495 Compter Vision A. Bobick nterpreting the eigenales λ Corner λ 1 and λ are large λ 1 ~ λ ; increases in all directions Featres 1: Harris Classification of image points sing eigenales of M: dge λ >> λ 1 λ 1 and λ are small; is almost constant in all directions Flat region dge λ 1 >> λ λ 1
CS 4495 Compter Vision A. Bobick Harris corner response fnction R = det M M α: constant 0.04 to 0.06 R depends onl on eigenales of M bt don t compte them no sqrt so reall fast! R is large for a corner R is negatie ith large magnitde for an edge R is small for a flat region α trace = λ1λ α λ1 dge R < 0 Flat region R small Corner R > 0 Featres 1: Harris λ dge R < 0
CS 4495 Compter Vision A. Bobick Lo tetre region Featres 1: Harris gradients hae small magnitde small λ 1 small λ
CS 4495 Compter Vision A. Bobick dge Featres 1: Harris large gradients all the same large λ 1 small λ
CS 4495 Compter Vision A. Bobick High tetred region Featres 1: Harris gradients are different large magnitdes large λ 1 large λ
CS 4495 Compter Vision A. Bobick Featres 1: Harris Harris detector: Algorithm 1. Compte Gassian deriaties at each piel. Compte second moment matri M in a Gassian indo arond each piel 3. Compte corner response fnction R 4. Threshold R 5. Find local maima of response fnction nonmaimm sppression C.Harris and M.Stephens. "A Combined Corner and dge Detector. Proceedings of the 4th Ale Vision Conference: pages 147 151 1988.
CS 4495 Compter Vision A. Bobick Harris Detector: Workflo Featres 1: Harris
CS 4495 Compter Vision A. Bobick Harris Detector: Workflo Compte corner response R Featres 1: Harris
CS 4495 Compter Vision A. Bobick Harris Detector: Workflo Find points ith large corner response: R>threshold Featres 1: Harris
CS 4495 Compter Vision A. Bobick Harris Detector: Workflo Take onl the points of local maima of R Featres 1: Harris
CS 4495 Compter Vision A. Bobick Harris Detector: Workflo Featres 1: Harris
CS 4495 Compter Vision A. Bobick Other corners: Featres 1: Harris Shi-Tomasi 94: Cornerness = min λ 1 λ Find local maimms cgoodfeatrestotrack... Reportedl better for region ndergoing affine deformations Bron M. Szeliski R. and Winder S. 005: there are others det M tr M = λλ 0 1 λ λ 0 1
CS 4495 Compter Vision A. Bobick Harris Detector: Some Properties Featres 1: Harris Rotation inariance?
CS 4495 Compter Vision A. Bobick Harris Detector: Some Properties Featres 1: Harris Rotation inariance llipse rotates bt its shape i.e. eigenales remains the same Corner response R is inariant to image rotation
CS 4495 Compter Vision A. Bobick Rotation nariant Detection Featres 1: Harris Harris Corner Detector C.Schmid et.al. alation of nterest Point Detectors. JCV 000
CS 4495 Compter Vision A. Bobick Harris Detector: Some Properties Featres 1: Harris nariance to image intensit change?
CS 4495 Compter Vision A. Bobick Harris Detector: Some Properties Featres 1: Harris Partial inariance to additie and mltiplicatie intensit changes threshold isse for mltiplicatie Onl deriaties are sed => inariance to intensit shift b ntensit scale: a R threshold R image coordinate image coordinate
CS 4495 Compter Vision A. Bobick Harris Detector: Some Properties Featres 1: Harris nariant to image scale?
CS 4495 Compter Vision A. Bobick Harris Detector: Some Properties Featres 1: Harris Not inariant to image scale! All points ill be classified as edges Corner!
CS 4495 Compter Vision A. Bobick Featres 1: Harris Harris Detector: Some Properties Qalit of Harris detector for different scale changes Repeatabilit rate: # correspondences # possible correspondences C.Schmid et.al. alation of nterest Point Detectors. JCV 000
CS 4495 Compter Vision A. Bobick Featres 1: Harris alation plots are from this paper
CS 4495 Compter Vision A. Bobick *F* e ant scale inariance Featres 1: Harris
CS 4495 Compter Vision A. Bobick Featres 1: Harris Scale nariant Detection Consider regions e.g. circles of different sizes arond a point Regions of corresponding sizes ill look the same in both images
CS 4495 Compter Vision A. Bobick Scale nariant Detection Featres 1: Harris The problem: ho do e choose corresponding circles independentl in each image?
CS 4495 Compter Vision A. Bobick Featres 1: Harris Scale nariant Detection Soltion: Design a fnction on the region circle hich is scale inariant the same for corresponding regions een if the are at different scales ample: aerage intensit. For corresponding regions een of different sizes it ill be the same. For a point in one image e can consider it as a fnction of region size circle radis f mage 1 f mage scale = 1/ region size region size
CS 4495 Compter Vision A. Bobick Featres 1: Harris Scale nariant Detection Common approach: Take a local maimm of this fnction Obseration: region size for hich the maimm is achieed shold be inariant to image scale. mportant: this scale inariant region size is fond in each image independentl! f mage 1 f mage scale = 1/ s 1 region size s region size
CS 4495 Compter Vision A. Bobick Featres 1: Harris Scale nariant Detection A good fnction for scale detection: has one stable sharp peak f bad f bad f Good! region size region size region size For sal images: a good fnction old be a one hich responds to contrast sharp local intensit change
CS 4495 Compter Vision A. Bobick trema at different scales Featres 1: Harris
CS 4495 Compter Vision A. Bobick Featres 1: Harris Scale nariant Detectors Harris-Laplacian 1 Find local maimm of: Harris corner detector in space image coordinates Laplacian in scale scale Harris Laplacian SFT Loe Find local maimm of: Difference of Gassians in space and scale scale DoG DoG 1 K.Mikolajczk C.Schmid. ndeing Based on Scale nariant nterest Points. CCV 001 D.Loe. Distinctie mage Featres from Scale-nariant Kepoints. JCV 004
CS 4495 Compter Vision A. Bobick Remoe lo contrast edge bond Featres 1: Harris trema points Contrast > C Not on edge
CS 4495 Compter Vision A. Bobick Scale nariant Detectors Featres 1: Harris perimental ealation of detectors.r.t. scale change Repeatabilit rate: # correspondences # possible correspondences K.Mikolajczk C.Schmid. ndeing Based on Scale nariant nterest Points. CCV 001
CS 4495 Compter Vision A. Bobick Featres 1: Harris Scale nariant Detection: Smmar Gien: to images of the same scene ith a large scale difference beteen them Goal: find the same interest points independentl in each image Soltion: search for maima of sitable fnctions in scale and in space oer the image Methods: 1. Harris-Laplacian [Mikolajczk Schmid]: maimize Laplacian oer scale Harris measre of corner response oer the image. SFT [Loe]: maimize Difference of Gassians oer scale and space
CS 4495 Compter Vision A. Bobick Featres 1: Harris Point Descriptors We kno ho to detect points Net qestion: Ho to match them?? Point descriptor shold be: 1. nariant. Distinctie
CS 4495 Compter Vision A. Bobick Net time Featres 1: Harris SFT SURF SFOP oh m