local invariant feature descriptors n.
Download
Skip this Video
Loading SlideShow in 5 Seconds..
Local Invariant Feature Descriptors PowerPoint Presentation
Download Presentation
Local Invariant Feature Descriptors

Loading in 2 Seconds...

play fullscreen
1 / 67

Local Invariant Feature Descriptors - PowerPoint PPT Presentation


  • 293 Views
  • Uploaded on

Local Invariant Feature Descriptors. Bin Fan National Laboratory of Pattern Recognition (NLPR) Institute of Automation, Chinese Academy of Sciences. 局部图像特征描述 —— 应用. Wide-Baseline Image Matching Structure from Motion, Image-based Localization, Image Stitch Object/Instance/Scene Recognition

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about 'Local Invariant Feature Descriptors' - zola


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
local invariant feature descriptors

Local Invariant Feature Descriptors

Bin Fan

National Laboratory of Pattern Recognition (NLPR)

Institute of Automation, Chinese Academy of Sciences

slide4
局部图像特征描述 —— 应用
  • Wide-Baseline Image Matching
    • Structure from Motion, Image-based Localization, Image Stitch
  • Object/Instance/Scene Recognition
  • Object Detection
  • Image Retrieval
slide7

Categories of Descriptors

  • Design method:
    • Handcrafted Descriptors
    • Data-driven Descriptors
slide8

Developments: Handcrafted Descriptors

  • 1999, SIFT [Citation: 23819]
  • 2003, Shape Context
  • 2006, SURF [Citation: 4093]
  • 2008, SMD, DAISY
  • 2009, OSID, CS-LBP
  • 2010, BRIEF, HRI-CSLTP, BiCE
  • 2011, ORB, BRISK, LIOP, MROGH
  • 2012, FREAK, KAZE, SYM
  • 2013, Line Context
slide9

Developments: Data-driven Descriptors

  • 2004, PCA-SIFT
  • 2007, LDE, Learning descriptor[Brown et al.]
  • 2009, Best DAISY
  • 2012, D-BRIEF, Learning descriptor by convex optimization[Simonyan et al.], BGM/LBGM, LDAHash
  • 2013, BinBoost, SQ-SIFT/DAISY
slide10

Categories of Descriptors

  • Design method:
    • Handcrafted Descriptors
    • Data-driven Descriptors
  • Encode information:
    • Gradient-based Descriptors
    • Intensity-based Descriptors
    • Descriptor-based Descriptors
slide11

Gradient-Based

    • SIFT、DAISY、BiCE、MROGH、BGM、LBGM、BinBoost、Learning Descriptor[Brown et al., Simonyan et al.]
  • Intensity-Based
    • CS-LBP、OSID、BRIEF、ORB、BRISK、FREAK、LDE、D-BREIF、LIOP
  • Descriptor-Based
    • LDAHash, LDP[Cai et al.,PAMI’11]
slide12

Categories of Descriptors

  • Design method:
    • Handcrafted Descriptors
    • Data-driven Descriptors
  • Encode information:
    • Gradient-based Descriptors
    • Intensity-based Descriptors
    • Descriptor-based Descriptors
  • Data type:
    • Floating-point Descriptors
    • Binary Descriptors
slide13

Floating-point Descriptors

    • SIFT、SURF、DAISY、CS-LBP、OSID、MROGH、LIOP、LBGM、LDE…
  • Binary Descriptors
    • BiCE、BRIEF、ORB、FREAK、BRISK、BGM、BinBoost、LDAHash、D-BRIEF…
handcrafted descriptors sift
Handcrafted Descriptors - SIFT

SIFT Descriptor [Lowe’99]

  • Binning of Spatial Coordinates and Gradient Orientations
  • Soft Assignment of Binning
  • 4x4 spatial grids, 8 gradient orientations, 128 dim SIFT
  • Normalization
handcrafted descriptors daisy
Handcrafted Descriptors - DAISY

DAISY Descriptor [Tola et al.’08]

  • Log-polar grid arrangement
  • Gaussian pooling of histograms of gradient orientations
  • Efficient for dense computation, but not for sparse keypoints!
descriptor learning data driven methods
Descriptor Learning – Data Driven Methods

Brown et al.’s method [CVPR’07, ICCV’07, PAMI’ 12]

Learning

Normalized Patch

Low-level feature extraction

Smooth

Spatial pooling

Post process

Projection

Descriptor

descriptor learning data driven methods1
Descriptor Learning – Data Driven Methods

Brown et al.’s method [CVPR’07, ICCV’07, PAMI’ 12]

  • Pre-defined low level features: gradient-based, filter bank based
  • Pre-defined spatial poolings: SIFT-like, DAISY-like, GLOH-like
  • Optimized combination of low level feature + spatial pooling
  • Projection: PCA, LDE …

1st: DAISY-like spatial pooling + filter bank [high Dim]

2nd: DAISY-like spatial pooling + gradient [moderate Dim]

PCA is better than LDE for projecting descriptor

descriptor learning data driven methods2
Descriptor Learning – Data Driven Methods

Simonyan et al.’s method [ECCV’12]

Learning

Normalized Patch

Gradient map calculation

Smooth

Spatial pooling

Projection

Descriptor

  • Spatial pooling is constrained to rings
  • Using L1 regularization to select pooling rings from a large pool
  • Max-Margin based objective function [convex]
  • Best reported results in the Brown et al.’s dataset
handcrafted binary descriptors
Handcrafted Binary Descriptors

Pioneering work: LBP

handcrafted binary descriptors1
Handcrafted Binary Descriptors

BRIEF [ECCV’10, PAMI’12]

Construct descriptor by binary tests:

Binary tests:

Pre-defined positions for binary tests:

handcrafted binary descriptors brief
Handcrafted Binary Descriptors - BRIEF

Low memory, Fast to compute and match

Limited performance

handcrafted binary descriptors2
Handcrafted Binary Descriptors

FREAK [CVPR’12]

Organizing sampling points analogous to retina structure

learning binary descriptors
Learning Binary Descriptors

D-BRIEF [ECCV’12]

  • Linear representation of projection matrix by Box/Gaussian/Rect filters
  • Approximate projection by filter responses
  • Efficient computation of Box/Gaussian/Rect filter responses
  • Binarization after discriminative projection
  • Extremely compact [only 32bits = 4 bytes]
learning binary descriptors1
Learning Binary Descriptors

BGM [NIPS’12]

(P1(1), P2(1),c(1))

(P1(2), P2(2),c(2))

  • Explore gradient orientation maps as weak learners
  • Each bit is construct by one weak learner
  • Select discriminative gradient orientation maps by boosting

(P1(n), P2(n),c(n))

learning binary descriptors2
Learning Binary Descriptors

BinBoost [CVPR’13]

  • Each bit as a linear combination of many gradient orientation maps
  • Optimization based on boosting
  • Very compact [64 bits = 8 bytes]
dataset and evaluation
Dataset and Evaluation
  • Different contexts
    • Image Matching
    • Object/Instance Recognition
    • Image Retrieval
dataset and evaluation matching
Dataset and Evaluation: Matching

Oxford dataset [2D scenes]: popular benchmark

http://www.robots.ox.ac.uk/~vgg/research/affine/index.html

K. Mikolajczyk, C. Schmid,  A performance evaluation of local descriptors. PAMI’05

dataset and evaluation matching1
Dataset and Evaluation: Matching

Oxford dataset [2D scenes]: popular benchmark

Evaluation protocol: recall vs. 1-precision

dataset and evaluation matching2
Dataset and Evaluation: Matching

Brown et al.’s dataset [image patches]: widely used for evaluation of learning based descriptors

http://www.cs.ubc.ca/~mbrown/patchdata/patchdata.html

M. Brown, G. Hua and S. Winder,  Discriminant Learning of Local Image Descriptors. PAMI’12

Three different subsets, each of which has more than 400k patch pairs

Liberty

Yosemite

Notre Dame

dataset and evaluation matching3
Dataset and Evaluation: Matching

Brown et al.’s dataset [image patches]: widely used for evaluation of learning based descriptors

Evaluation protocol: False Positive Rate(FPR) vs. Recall

dataset and evaluation recognition
Dataset and Evaluation: Recognition
  • Dataset: Ukbench, ZuBuD, …
  • Evaluation Protocol: Recognition rate, recall
dataset and evaluation retrieval
Dataset and Evaluation: Retrieval
  • Dataset: Oxford/Paris Building, Holidays
  • Evaluation Protocol: mAP, Precision vs. Recall

AP(Average Precision): Precision across all recalls

mAP: mean AP of all queries

resources
Resources
  • OpenCV: http://opencv.org/
    • SIFT, SURF, BRISK, BRIEF, ORB, FREAK
  • VLFeat: http://www.vlfeat.org/
    • SIFT, LIOP, Covariant Feature Detectors
  • Oxford VGG: http://www.robots.ox.ac.uk/~vgg/research/affine/index.html
  • Authors’ pages…
published evaluations matching
Published Evaluations: Matching
  • K. Mikolajczyk and C. Schmid,  A Performance Evaluation of Local Descriptors. PAMI’05
  • P. Moreels and P. Perona,   Evaluation of Features Detectors and Descriptors based on 3D objects. IJCV’07
  • Anders Lindbjerg Dahl et al., Finding the Best Feature Detector-Descriptor Combination. 3DIMPVT’11
  • O.Miksik and K. Mikolajczyk, Evaluation of Local Detectors and Descriptors for Fast Feature Matching, ICPR’12
  • J. Heinly et al., Comparative Evaluation of Binary Features, ECCV’12
published evaluations classification recognition
Published Evaluations: Classification/Recognition
  • K. Mikolajczyk et al.,  Local Features for Object Class Recognition. ICCV’05
  • E. Seemann et al.,   An Evaluation of Local Shape-Based Features for Pedestrian Detection. BMVC’05
  • M. Stark and B. Schiele, How Good are Local Features for Classes of Geometric Objects. ICCV’07
  • J. Zhang et al., Local Features and Kernels for Classification of Texture and Object Categories: A Comprehensive Study, IJCV’07
  • K. E. A. Van de Sande et al., Evaluation of Color Descriptors for Object and Scene Recognition, PAMI’10
slide37

Our Work

Feature Description by Intensity Order Pooling

Local Intensity Order Pattern

Joint work with Zhenhua Wang

slide39

Category of handcrafted descriptors

With a reference orientation:

SIFT, SURF, DAISY, CS-LBP …

slide40

Category of handcrafted descriptors

With a reference orientation:

SIFT, SURF, DAISY, CS-LBP …

+: encode spatial information, high discriminability

-: sensitive to orientation estimation error

slide42

Category of handcrafted descriptors

With a reference orientation:

SIFT, SURF, DAISY, CS-LBP …

+: encode spatial information, high discriminability

-: sensitive to orientation estimation error

  • Distinctiveness
  • Robustness
slide43

Category of handcrafted descriptors

0

255/2π

r

0

Without a reference orientation:

RIFT, Spin image

+: inherently rotation invariance, robust to orientation estimation error

-: discard some spatial information, limited discriminability

slide44

Category of handcrafted descriptors

  • Distinctiveness
  • Robustness

Without a reference orientation:

RIFT, Spin image

+: inherently rotation invariance, robust to orientation estimation error

-: discard some spatial information, limited discriminablity

slide45

Category of handcrafted descriptors

With a reference orientation:

SIFT, SURF, DAISY, CS-LBP …

+: encode spatial information, high discriminability

-: sensitive to orientation estimation error

  • Distinctiveness
  • Robustness

Without a reference orientation:

RIFT, Spin image

+: inherently rotation invariance, robust to orientation estimation error

-: discard some spatial information, limited discriminablity

slide46

Category of handcrafted descriptors

With a reference orientation:

SIFT, SURF, DAISY, CS-LBP …

  • Distinctiveness

+: encode spatial information, high discriminability

-: sensitive to orientation estimation error

  • Distinctiveness
  • Robustness
  • Robustness

Without a reference orientation:

RIFT, Spin image

+: inherently rotation invariance, robust to orientation estimation error

-: discard some spatial information, limited discriminablity

slide47

Our Solution

Construct a local coordinate for low-level feature computation

Gradient orientation maps [SIFT]

Center-symmetrical binary pattern [CS-LBP]

slide48

Our Solution

Pool low-level features by intensity orders

……

……

slide49

Our Solution

Using multiple support regions

slide51

Gradient orientation maps -> MROGH

Center-symmetrical binary pattern -> MRRID

Code: http://www.openpr.org.cn/index.php/89-MROGH-v1.1/View-details.html

slide52

Experiments

Multiple Support Regions vs. Single Support Region

MROGH

MRRID

SIFT

SR-i: Results of using the i-th support region

MR: Results of using multiple support region

Averaged results over 140 image pairs

slide53

Experiments

Image Matching – Oxford Dataset

Hessian-Affine, Viewpoint change

slide54

Experiments

Image Matching – Oxford Dataset

Harris-Affine, Image Blur

slide55

Experiments

Object Recognition:

Datasets: 53 Objects, ZuBuD, Ukbench

265 images of 53 objects

Each object has 5 images of different viewpoints

slide56

Experiments

Object Recognition:

Datasets: 53 Objects, ZuBuD, Ukbench

1005 images of 201 buildings in the Zurich city

Each building has 5 images of different viewpoints, across seasons

slide57

Experiments

Object Recognition:

Datasets: 53 Objects, ZuBuD, Ukbench

10200 images of 2550 objects [first 4000 images used here]

Each object has 4 images of different viewpoints

slide58

Experiments

Object Recognition:

slide59

Experiments

Recognition examples: 53 Objects

input images

slide60

Experiments

Recognition examples: ZuBuD

input images

slide61

Experiments

Recognition examples: Ukbench

input images

slide63

Local Intensity Order Pattern

  • Explore the relative intensity relationship among neighboring points
  • Rotationally invariant computation of neighboring points’ intensities
  • Intensity order based pooling

Code: http://vision.ia.ac.cn/Students/wzh/publication/liop/index.html

http://www.vlfeat.org/api/liop.html

slide64

Experiments

Image Matching: Oxford dataset

slide65

Experiments

Image Matching: Oxford dataset

slide66

Experiments

Image Matching: Complex Brightness Change

slide67

Questions?

Thank you