1 / 67

Local Invariant Feature Descriptors

Local Invariant Feature Descriptors. Bin Fan National Laboratory of Pattern Recognition (NLPR) Institute of Automation, Chinese Academy of Sciences. 局部图像特征描述 —— 应用. Wide-Baseline Image Matching Structure from Motion, Image-based Localization, Image Stitch Object/Instance/Scene Recognition

zola
Download Presentation

Local Invariant Feature Descriptors

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. Local Invariant Feature Descriptors Bin Fan National Laboratory of Pattern Recognition (NLPR) Institute of Automation, Chinese Academy of Sciences

  2. 局部图像特征描述 —— 应用 • Wide-Baseline Image Matching • Structure from Motion, Image-based Localization, Image Stitch • Object/Instance/Scene Recognition • Object Detection • Image Retrieval

  3. Structure From Motion

  4. Object Recognition Database …

  5. Categories of Descriptors • Design method: • Handcrafted Descriptors • Data-driven Descriptors

  6. Developments: Handcrafted Descriptors • 1999, SIFT [Citation: 23819] • 2003, Shape Context • 2006, SURF [Citation: 4093] • 2008, SMD, DAISY • 2009, OSID, CS-LBP • 2010, BRIEF, HRI-CSLTP, BiCE • 2011, ORB, BRISK, LIOP, MROGH • 2012, FREAK, KAZE, SYM • 2013, Line Context

  7. Developments: Data-driven Descriptors • 2004, PCA-SIFT • 2007, LDE, Learning descriptor[Brown et al.] • 2009, Best DAISY • 2012, D-BRIEF, Learning descriptor by convex optimization[Simonyan et al.], BGM/LBGM, LDAHash • 2013, BinBoost, SQ-SIFT/DAISY

  8. Categories of Descriptors • Design method: • Handcrafted Descriptors • Data-driven Descriptors • Encode information: • Gradient-based Descriptors • Intensity-based Descriptors • Descriptor-based Descriptors

  9. Gradient-Based • SIFT、DAISY、BiCE、MROGH、BGM、LBGM、BinBoost、Learning Descriptor[Brown et al., Simonyan et al.] • Intensity-Based • CS-LBP、OSID、BRIEF、ORB、BRISK、FREAK、LDE、D-BREIF、LIOP • Descriptor-Based • LDAHash, LDP[Cai et al.,PAMI’11]

  10. Categories of Descriptors • Design method: • Handcrafted Descriptors • Data-driven Descriptors • Encode information: • Gradient-based Descriptors • Intensity-based Descriptors • Descriptor-based Descriptors • Data type: • Floating-point Descriptors • Binary Descriptors

  11. Floating-point Descriptors • SIFT、SURF、DAISY、CS-LBP、OSID、MROGH、LIOP、LBGM、LDE… • Binary Descriptors • BiCE、BRIEF、ORB、FREAK、BRISK、BGM、BinBoost、LDAHash、D-BRIEF…

  12. Handcrafted Descriptors - SIFT SIFT Descriptor [Lowe’99] • Binning of Spatial Coordinates and Gradient Orientations • Soft Assignment of Binning • 4x4 spatial grids, 8 gradient orientations, 128 dim SIFT • Normalization

  13. Handcrafted Descriptors - DAISY DAISY Descriptor [Tola et al.’08] • Log-polar grid arrangement • Gaussian pooling of histograms of gradient orientations • Efficient for dense computation, but not for sparse keypoints!

  14. Descriptor Learning – Data Driven Methods Brown et al.’s method [CVPR’07, ICCV’07, PAMI’ 12] Learning Normalized Patch Low-level feature extraction Smooth Spatial pooling Post process Projection Descriptor

  15. Descriptor Learning – Data Driven Methods Brown et al.’s method [CVPR’07, ICCV’07, PAMI’ 12] • Pre-defined low level features: gradient-based, filter bank based • Pre-defined spatial poolings: SIFT-like, DAISY-like, GLOH-like • Optimized combination of low level feature + spatial pooling • Projection: PCA, LDE … 1st: DAISY-like spatial pooling + filter bank [high Dim] 2nd: DAISY-like spatial pooling + gradient [moderate Dim] PCA is better than LDE for projecting descriptor

  16. Descriptor Learning – Data Driven Methods Simonyan et al.’s method [ECCV’12] Learning Normalized Patch Gradient map calculation Smooth Spatial pooling Projection Descriptor • Spatial pooling is constrained to rings • Using L1 regularization to select pooling rings from a large pool • Max-Margin based objective function [convex] • Best reported results in the Brown et al.’s dataset

  17. Handcrafted Binary Descriptors Pioneering work: LBP

  18. Handcrafted Binary Descriptors BRIEF [ECCV’10, PAMI’12] Construct descriptor by binary tests: Binary tests: Pre-defined positions for binary tests:

  19. Handcrafted Binary Descriptors - BRIEF Low memory, Fast to compute and match Limited performance

  20. Handcrafted Binary Descriptors FREAK [CVPR’12] Organizing sampling points analogous to retina structure

  21. Learning Binary Descriptors D-BRIEF [ECCV’12] • Linear representation of projection matrix by Box/Gaussian/Rect filters • Approximate projection by filter responses • Efficient computation of Box/Gaussian/Rect filter responses • Binarization after discriminative projection • Extremely compact [only 32bits = 4 bytes]

  22. Learning Binary Descriptors BGM [NIPS’12] (P1(1), P2(1),c(1)) … (P1(2), P2(2),c(2)) • Explore gradient orientation maps as weak learners • Each bit is construct by one weak learner • Select discriminative gradient orientation maps by boosting (P1(n), P2(n),c(n))

  23. Learning Binary Descriptors BinBoost [CVPR’13] • Each bit as a linear combination of many gradient orientation maps • Optimization based on boosting • Very compact [64 bits = 8 bytes]

  24. Dataset and Evaluation • Different contexts • Image Matching • Object/Instance Recognition • Image Retrieval

  25. Dataset and Evaluation: Matching Oxford dataset [2D scenes]: popular benchmark http://www.robots.ox.ac.uk/~vgg/research/affine/index.html K. Mikolajczyk, C. Schmid,  A performance evaluation of local descriptors. PAMI’05 …

  26. Dataset and Evaluation: Matching Oxford dataset [2D scenes]: popular benchmark Evaluation protocol: recall vs. 1-precision

  27. Dataset and Evaluation: Matching Brown et al.’s dataset [image patches]: widely used for evaluation of learning based descriptors http://www.cs.ubc.ca/~mbrown/patchdata/patchdata.html M. Brown, G. Hua and S. Winder,  Discriminant Learning of Local Image Descriptors. PAMI’12 Three different subsets, each of which has more than 400k patch pairs Liberty Yosemite Notre Dame

  28. Dataset and Evaluation: Matching Brown et al.’s dataset [image patches]: widely used for evaluation of learning based descriptors Evaluation protocol: False Positive Rate(FPR) vs. Recall

  29. Dataset and Evaluation: Recognition • Dataset: Ukbench, ZuBuD, … • Evaluation Protocol: Recognition rate, recall

  30. Dataset and Evaluation: Retrieval • Dataset: Oxford/Paris Building, Holidays • Evaluation Protocol: mAP, Precision vs. Recall AP(Average Precision): Precision across all recalls mAP: mean AP of all queries

  31. Resources • OpenCV: http://opencv.org/ • SIFT, SURF, BRISK, BRIEF, ORB, FREAK • VLFeat: http://www.vlfeat.org/ • SIFT, LIOP, Covariant Feature Detectors • Oxford VGG: http://www.robots.ox.ac.uk/~vgg/research/affine/index.html • Authors’ pages…

  32. Published Evaluations: Matching • K. Mikolajczyk and C. Schmid,  A Performance Evaluation of Local Descriptors. PAMI’05 • P. Moreels and P. Perona,   Evaluation of Features Detectors and Descriptors based on 3D objects. IJCV’07 • Anders Lindbjerg Dahl et al., Finding the Best Feature Detector-Descriptor Combination. 3DIMPVT’11 • O.Miksik and K. Mikolajczyk, Evaluation of Local Detectors and Descriptors for Fast Feature Matching, ICPR’12 • J. Heinly et al., Comparative Evaluation of Binary Features, ECCV’12

  33. Published Evaluations: Classification/Recognition • K. Mikolajczyk et al.,  Local Features for Object Class Recognition. ICCV’05 • E. Seemann et al.,   An Evaluation of Local Shape-Based Features for Pedestrian Detection. BMVC’05 • M. Stark and B. Schiele, How Good are Local Features for Classes of Geometric Objects. ICCV’07 • J. Zhang et al., Local Features and Kernels for Classification of Texture and Object Categories: A Comprehensive Study, IJCV’07 • K. E. A. Van de Sande et al., Evaluation of Color Descriptors for Object and Scene Recognition, PAMI’10

  34. Our Work Feature Description by Intensity Order Pooling Local Intensity Order Pattern Joint work with Zhenhua Wang

  35. Feature Description by Intensity Order Pooling

  36. Category of handcrafted descriptors With a reference orientation: SIFT, SURF, DAISY, CS-LBP …

  37. Category of handcrafted descriptors With a reference orientation: SIFT, SURF, DAISY, CS-LBP … +: encode spatial information, high discriminability -: sensitive to orientation estimation error

  38. Match vs. Orientation error 64% 36%

  39. Category of handcrafted descriptors With a reference orientation: SIFT, SURF, DAISY, CS-LBP … +: encode spatial information, high discriminability -: sensitive to orientation estimation error • Distinctiveness • Robustness

  40. Category of handcrafted descriptors 0 255/2π r 0 Without a reference orientation: RIFT, Spin image +: inherently rotation invariance, robust to orientation estimation error -: discard some spatial information, limited discriminability

  41. Category of handcrafted descriptors • Distinctiveness • Robustness Without a reference orientation: RIFT, Spin image +: inherently rotation invariance, robust to orientation estimation error -: discard some spatial information, limited discriminablity

  42. Category of handcrafted descriptors With a reference orientation: SIFT, SURF, DAISY, CS-LBP … +: encode spatial information, high discriminability -: sensitive to orientation estimation error • Distinctiveness • Robustness Without a reference orientation: RIFT, Spin image +: inherently rotation invariance, robust to orientation estimation error -: discard some spatial information, limited discriminablity

  43. Category of handcrafted descriptors With a reference orientation: SIFT, SURF, DAISY, CS-LBP … • Distinctiveness +: encode spatial information, high discriminability -: sensitive to orientation estimation error • Distinctiveness • Robustness • Robustness Without a reference orientation: RIFT, Spin image +: inherently rotation invariance, robust to orientation estimation error -: discard some spatial information, limited discriminablity

  44. Our Solution Construct a local coordinate for low-level feature computation Gradient orientation maps [SIFT] Center-symmetrical binary pattern [CS-LBP]

  45. Our Solution Pool low-level features by intensity orders … … …… …… …

  46. Our Solution Using multiple support regions

  47. Using multiple support regions

More Related