CS 4495 Compter Vision A. Bobick Featres 1: Harris CS 4495 Compter Vision Featres 1 Harris Aaron Bobick School of nteractie Compting
CS 4495 Compter Vision A. Bobick Featres 1: Harris Administriia PS 4: Ot and not changed. De Snda Oct 1 th 11:55pm t is application of the last few lectres. Mostl straight forward Matlab bt if o re linear algebra is rst it can take a while to figre ot. Qestion takes some doing to nderstand. Yo hae been warned t is cool Yo hae been warned Toda: Start on featres. Forsth and Ponce: 5.3-5.4 Szeliski also coers this well Section 4 4.1.1 These net 3 lectres will proide detail for Project 4.
CS 4495 Compter Vision A. Bobick Featres 1: Harris The basic image point matching problem Sppose hae two images related b some transformation. Or hae two images of the same object in different positions. How to find the transformation of image 1 that wold align it with image?
CS 4495 Compter Vision A. Bobick Featres 1: Harris We want Local1 Featres Goal: Find points in an image that can be: Fond in other images Fond precisel well localized Fond reliabl well matched Wh? Want to compte a fndamental matri to recoer geometr Robotics/Vision: See how a bnch of points moe from one frame to another. Allows comptation of how camera moed -> depth -> moing objects Bild a panorama
CS 4495 Compter Vision A. Bobick Featres 1: Harris Sppose o want to bild a panorama M. Brown and D. G. Lowe. Recognising Panoramas. CCV 003
CS 4495 Compter Vision A. Bobick How do we bild panorama? Featres 1: Harris We need to match align images
CS 4495 Compter Vision A. Bobick Matching with Featres Featres 1: Harris Detect featres featre points in both images
CS 4495 Compter Vision A. Bobick Featres 1: Harris Matching with Featres Detect featres featre points in both images Match featres - find corresponding pairs
CS 4495 Compter Vision A. Bobick Featres 1: Harris Matching with Featres Detect featres featre points in both images Match featres - find corresponding pairs Use these pairs to align images
CS 4495 Compter Vision A. Bobick Featres 1: Harris Matching with Featres Problem 1: Detect the same point independentl in both images no chance to match! We need a repeatable detector
CS 4495 Compter Vision A. Bobick Featres 1: Harris Matching with Featres Problem : For each point correctl recognize the corresponding one? We need a reliable and distinctie descriptor
CS 4495 Compter Vision A. Bobick Featres 1: Harris More motiation Featre points are sed also for: mage alignment e.g. homograph or fndamental matri 3D reconstrction Motion tracking Object recognition ndeing and database retrieal Robot naigation other
CS 4495 Compter Vision A. Bobick Characteristics of good featres Featres 1: Harris Repeatabilit/Precision The same featre can be fond in seeral images despite geometric and photometric transformations Salienc/Matchabilit ach featre has a distinctie description Compactness and efficienc Man fewer featres than image piels Localit A featre occpies a relatiel small area of the image; robst to cltter and occlsion
CS 4495 Compter Vision A. Bobick Featres 1: Harris Corner Detection: Basic dea We shold easil recognize the point b looking throgh a small window Shifting a window in an direction shold gie a large change in intensit Sorce: A. fros flat region: no change in all directions edge : no change along the edge direction corner : significant change in all directions with small shift
CS 4495 Compter Vision A. Bobick Featres 1: Harris Finding Corners Ke propert: in the region arond a corner image gradient has two or more dominant directions Corners are repeatable and distinctie C.Harris and M.Stephens. "A Combined Corner and dge Detector. Proceedings of the 4th Ale Vision Conference: pages 147 151 1988
CS 4495 Compter Vision A. Bobick Featres 1: Harris Finding Harris Corners Ke propert: in the region arond a corner image gradient has two or more dominant directions Corners are repeatable and distinctie C. Harris and M.Stephens. "A Combined Corner and dge Detector. Proceedings of the 4th Ale Vision Conference: pages 147 151 1988
CS 4495 Compter Vision A. Bobick Corner Detection: Mathematics Featres 1: Harris Change in appearance for the shift []: [ ] = w Window fnction Shifted intensit ntensit Window fnction w = or 1 in window 0 otside Gassian Sorce: R. Szeliski
CS 4495 Compter Vision A. Bobick Corner Detection: Mathematics Featres 1: Harris Change in appearance for the shift []: [ ] = w 00 3
CS 4495 Compter Vision A. Bobick Corner Detection: Mathematics Featres 1: Harris Change in appearance for the shift []: [ ] = w We want to find ot how this fnction behaes for small shifts near 00
CS 4495 Compter Vision A. Bobick Corner Detection: Mathematics Featres 1: Harris Change in appearance for the shift []: [ ] = w Second-order Talor epansion of abot 00 local qadratic approimation for small : 0 1 1D: F δ F 0 δ δ d 0 d df d F 00 1 00 00 00 [ ] [ ] 00 00 00
Featres 1: Harris CS 4495 Compter Vision A. Bobick Corner Detection: Mathematics 00 00 00 00 ] [ 1 00 00 ] [ 00 [ ] w = Second-order Talor epansion of abot 00: [ ] [ ] [ ] w w w w w = = =
Featres 1: Harris CS 4495 Compter Vision A. Bobick Corner Detection: Mathematics 00 00 00 00 ] [ 1 00 00 ] [ 00 [ ] w = Second-order Talor epansion of abot 00: [ ] [ ] [ ] w w w w w = = =
Featres 1: Harris CS 4495 Compter Vision A. Bobick Corner Detection: Mathematics 00 00 00 00 ] [ 1 00 00 ] [ 00 [ ] w = Second-order Talor epansion of abot 00: [ ] [ ] [ ] w w w w w = = =
Featres 1: Harris CS 4495 Compter Vision A. Bobick Corner Detection: Mathematics 00 00 00 00 ] [ 1 00 00 ] [ 00 Second-order Talor epansion of abot 00: [ ] [ ] [ ] w w w w w = = = [ ] w =
Featres 1: Harris CS 4495 Compter Vision A. Bobick Corner Detection: Mathematics 00 00 00 00 ] [ 1 00 00 ] [ 00 alate at = 00: [ ] [ ] [ ] w w w w w = = = = 0 = 0 = 0 [ ] w =
CS 4495 Compter Vision A. Bobick Corner Detection: Mathematics Featres 1: Harris [ ] = w Second-order Talor epansion of abot 00: 00 [ ] 00 00 1 [ ] 00 00 00 00 00 = 0 00 = 0 00 = 0 00 = w 00 = w 00 = w
Featres 1: Harris CS 4495 Compter Vision A. Bobick Corner Detection: Mathematics [ ] w = Second-order Talor epansion of abot 00: w w w w ] [ 00 0 00 0 00 0 = = = 00 00 00 w w w = = =
CS 4495 Compter Vision A. Bobick Corner Detection: Mathematics The qadratic approimation simplifies to Featres 1: Harris [ ] M where M is a second moment matri compted from image deriaties: Withot weight M M = w ach prodct is a rank 1
Featres 1: Harris CS 4495 Compter Vision A. Bobick The srface is locall approimated b a qadratic form. nterpreting the second moment matri M ] [ = w M
CS 4495 Compter Vision A. Bobick nterpreting the second moment matri Consider a constant slice of : = k This is the eqation of an ellipse. [ ] M = Featres 1: Harris const
Featres 1: Harris CS 4495 Compter Vision A. Bobick = = 1 0 0 λ λ w M First consider the ais-aligned case where gradients are either horizontal or ertical f either λ is close to 0 then this is not a corner so look for locations where both are large. nterpreting the second moment matri
Featres 1: Harris CS 4495 Compter Vision A. Bobick = = 1 0 0 λ λ w M First consider the ais-aligned case where gradients are either horizontal or ertical f either λ is close to 0 then this is not a corner so look for locations where both are large. nterpreting the second moment matri
CS 4495 Compter Vision A. Bobick nterpreting the second moment matri direction of the fastest change direction of the slowest change Featres 1: Harris Consider a horizontal slice of : [ ] M = const This is the eqation of an ellipse. Diagonalization of M: M λ = R 1 1 0 R 0 λ The ais lengths of the ellipse are determined b the eigenales and the orientation is determined b R λ ma -1/ λ min -1/
CS 4495 Compter Vision A. Bobick nterpreting the eigenales λ Corner λ 1 and λ are large λ 1 ~ λ ; increases in all directions Featres 1: Harris Classification of image points sing eigenales of M: dge λ >> λ 1 λ 1 and λ are small; is almost constant in all directions Flat region dge λ 1 >> λ λ 1
CS 4495 Compter Vision A. Bobick Harris corner response fnction R = det M trace M α: constant 0.04 to 0.06 R depends onl on eigenales of M bt don t compte them no sqrt so reall fast! R is large for a corner R is negatie with large magnitde for an edge R is small for a flat region α = 1 1 dge R < 0 Flat region λ λ R small α λ Corner R > 0 Featres 1: Harris λ dge R < 0
CS 4495 Compter Vision A. Bobick Low tetre region Featres 1: Harris gradients hae small magnitde small λ 1 small λ
CS 4495 Compter Vision A. Bobick dge Featres 1: Harris large gradients all the same large λ 1 small λ
CS 4495 Compter Vision A. Bobick High tetred region Featres 1: Harris gradients are different large magnitdes large λ 1 large λ
CS 4495 Compter Vision A. Bobick Featres 1: Harris Harris detector: Algorithm 1. Compte Gassian deriaties at each piel. Compte second moment matri M in a Gassian window arond each piel 3. Compte corner response fnction R 4. Threshold R 5. Find local maima of response fnction nonmaimm sppression C.Harris and M.Stephens. "A Combined Corner and dge Detector. Proceedings of the 4th Ale Vision Conference: pages 147 151 1988.
CS 4495 Compter Vision A. Bobick Harris Detector: Workflow Featres 1: Harris
CS 4495 Compter Vision A. Bobick Harris Detector: Workflow Compte corner response R Featres 1: Harris
CS 4495 Compter Vision A. Bobick Harris Detector: Workflow Find points with large corner response: R>threshold Featres 1: Harris
CS 4495 Compter Vision A. Bobick Harris Detector: Workflow Take onl the points of local maima of R Featres 1: Harris
CS 4495 Compter Vision A. Bobick Harris Detector: Workflow Featres 1: Harris
CS 4495 Compter Vision A. Bobick Other corners: Featres 1: Harris Shi-Tomasi 94: Cornerness = min λ 1 λ Find local maimms cgoodfeatrestotrack... Reportedl better for region ndergoing affine deformations Brown M. Szeliski R. and Winder S. 005: there are others det M tr M = λλ 0 1 λ λ 0 1
CS 4495 Compter Vision A. Bobick Harris Detector: Some Properties Featres 1: Harris
CS 4495 Compter Vision A. Bobick Harris Detector: Some Properties Featres 1: Harris Rotation inariance?
CS 4495 Compter Vision A. Bobick Harris Detector: Some Properties Featres 1: Harris Rotation inariance llipse rotates bt its shape i.e. eigenales remains the same Corner response R is inariant to image rotation
CS 4495 Compter Vision A. Bobick Featres 1: Harris Rotation nariant Detection Harris Corner Detector Repeatabilit rate: # correspondences # possible correspondences C.Schmid et.al. alation of nterest Point Detectors. JCV 000
CS 4495 Compter Vision A. Bobick Harris Detector: Some Properties Featres 1: Harris nariance to image intensit change?
CS 4495 Compter Vision A. Bobick Harris Detector: Some Properties Featres 1: Harris Partial inariance to additie and mltiplicatie intensit changes threshold isse for mltiplicatie Onl deriaties are sed => inariance to intensit shift b ntensit scale: a R threshold R image coordinate image coordinate
CS 4495 Compter Vision A. Bobick Harris Detector: Some Properties Featres 1: Harris nariant to image scale?
CS 4495 Compter Vision A. Bobick Harris Detector: Some Properties Featres 1: Harris Not inariant to image scale! All points will be classified as edges Corner!
CS 4495 Compter Vision A. Bobick Harris Detector: Some Properties Featres 1: Harris Qalit of Harris detector for different scale changes
CS 4495 Compter Vision A. Bobick *F* we want scale inariance Featres 1: Harris
CS 4495 Compter Vision A. Bobick Featres 1: Harris Scale nariant Detection Consider regions e.g. circles of different sizes arond a point Regions of corresponding sizes will look the same in both images
CS 4495 Compter Vision A. Bobick Scale nariant Detection Featres 1: Harris The problem: how do we choose corresponding circles independentl in each image?
CS 4495 Compter Vision A. Bobick Featres 1: Harris Scale nariant Detection Soltion: Design a fnction on the region circle which is scale inariant the same for corresponding regions een if the are at different scales ample: aerage intensit. For corresponding regions een of different sizes it will be the same. For some gien point in one image we can consider it as a fnction of region size circle radis f mage 1 f mage scale = 1/ region size region size
CS 4495 Compter Vision A. Bobick Scale nariant Detection Featres 1: Harris Common approach: Take a local maimm of this fnction Obseration: region size for which the maimm is achieed shold be inariant to image scale. mportant: this scale inariant region size is fond in each image independentl! f mage 1 f mage scale = 1/ s 1 region size s region size
CS 4495 Compter Vision A. Bobick Featres 1: Harris Scale nariant Detection A good fnction for scale detection: has one stable sharp peak f bad f bad f Good! region size region size region size For sal images: a good fnction wold be a one which responds to contrast sharp local intensit change
CS 4495 Compter Vision A. Bobick Scale sensitie response Featres 1: Harris
CS 4495 Compter Vision A. Bobick Scale nariant Detection Featres 1: Harris Fnction is jst application of a kernel: Laplacian of Gassian - LoG f = Kernel mage L= G G σ σ σ Laplacian of Gassian 10-6 10 8 6 4 0 10-100 80 60 40 0 0 0 50 100 150
CS 4495 Compter Vision A. Bobick Scale nariant Detection Fnctions for determining scale Kernels: L= G G σ σ σ Laplacian DoG= G kσ G σ Difference of Gassians where Gassian f = Featres 1: Harris Kernel mage 1 πσ σ G σ = e Note: both kernels are inariant to scale and rotation
Resample Blr Sbtract CS 4495 Compter Vision A. Bobick Ke point localization Featres 1: Harris General idea: find robst etremm maimm or minimm both in space and in scale.
Resample Blr Sbtract CS 4495 Compter Vision A. Bobick Ke point localization Featres 1: Harris SFT: Scale nariant Featre Transform Specific sggestion: se DoG pramid to find maimm ales remember edge detection? then eliminate edges and pick onl corners.
Resample Blr Sbtract CS 4495 Compter Vision A. Bobick Ke point localization Featres 1: Harris ach point is compared to its 8 neighbors in the crrent image and 9 neighbors each in the scales aboe and below.
CS 4495 Compter Vision A. Bobick Featres 1: Harris Scale space processed one octae at a time
CS 4495 Compter Vision A. Bobick trema at different scales Featres 1: Harris
CS 4495 Compter Vision A. Bobick Remoe low contrast edge bond Featres 1: Harris trema points Contrast > C Not on edge
CS 4495 Compter Vision A. Bobick Featres 1: Harris Scale nariant Detectors SFT Lowe 1 Find local maimm of: Difference of Gassians in space and scale scale DoG DoG Harris-Laplacian Find local maimm of: Harris corner detector in space image coordinates Laplacian in scale scale Harris Laplacian 1 D.Lowe. Distinctie mage Featres from Scale-nariant Kepoints. JCV 004 K.Mikolajczk C.Schmid. ndeing Based on Scale nariant nterest Points. CCV 001
CS 4495 Compter Vision A. Bobick Scale nariant Detectors Featres 1: Harris perimental ealation of detectors w.r.t. scale change Repeatabilit rate: # correspondences # possible correspondences K.Mikolajczk C.Schmid. ndeing Based on Scale nariant nterest Points. CCV 001
CS 4495 Compter Vision A. Bobick Featres 1: Harris Scale nariant Detection: Smmar Gien: two images of the same scene with a large scale difference between them Goal: find the same interest points independentl in each image Soltion: search for maima of sitable fnctions in scale and in space oer the image Methods: 1. SFT [Lowe]: maimize Difference of Gassians oer scale and space. Harris-Laplacian [Mikolajczk Schmid]: maimize Laplacian oer scale Harris measre of corner response oer the image
CS 4495 Compter Vision A. Bobick Featres 1: Harris Point Descriptors We know how to detect points Net qestion: How to match them?? Point descriptor shold be: 1. nariant. Distinctie
CS 4495 Compter Vision A. Bobick Net time Featres 1: Harris SFT SURF SFOP oh m