Learning a scale invariant model for curvilinear continuity
This presentation is the property of its rightful owner.
Sponsored Links
1 / 62

Learning a Scale-Invariant Model for Curvilinear Continuity PowerPoint PPT Presentation


  • 39 Views
  • Uploaded on
  • Presentation posted in: General

Learning a Scale-Invariant Model for Curvilinear Continuity. Xiaofeng Ren. The Quest of Boundary Detection. Widely used for mid/high-level vision tasks Huge literature on edge detection [Canny 86] Typically measuring local contrast Approaching human performance?

Download Presentation

Learning a Scale-Invariant Model for Curvilinear Continuity

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript


Learning a scale invariant model for curvilinear continuity

Learning a Scale-Invariant Model for Curvilinear Continuity

Xiaofeng Ren

1


Learning a scale invariant model for curvilinear continuity

The Quest of Boundary Detection

  • Widely used for mid/high-level vision tasks

  • Huge literature on edge detection

    [Canny 86]

  • Typically measuring local contrast

  • Approaching human performance?

    [Martin, Fowlkes & Malik 02]

    [Fowlkes, Martin & Malik 03]

2


Limit of local boundary detection

Limit of Local Boundary Detection

1

2

3

4

3


Learning a scale invariant model for curvilinear continuity

Curvilinear Continuity

  • Good Continuation

  • Visual Completion

  • Illusory Contours

4


Continuity in human vision

Continuity in Human Vision

  • [Wertheimer 23]

  • [Kanizsa 55]

  • [von der Heydt et al 84]

    • evidence in V2

  • [Kellman & Shipley 91]

    • geometric conditions of completion

  • [Field, Hayes & Hess 93]

    • quantitative analysis of factors

  • [Kapadia, Westheimer & Gilbert 00]

    • evidence in V1

  • [Geisler et al 01]

    • evidence from ecological statistics

      …… … …

5


Continuity in computer vision

Extensive literature on curvilinear continuity

[Shashua & Ullman 88], [Parent & Zucker 89], [Heitger & von der Heydt 93], [Mumford 94], [Williams & Jacobs 95], [Elder & Zucker 96], [Williams & Thornber 99], [Jermyn & Ishikawa 99], [Mahamud et al 03], …, …

Problems with most of the previous approaches

no support from any groundtruth data

usually demonstrated on a few simple/synthetic images

no quantitative evaluation

Continuity in Computer Vision

6


Outline

Outline

  • Ecological Statistics of Contours

  • A Scale-Invariant Representation

  • Learning Models of Curvilinear Continuity

  • Quantitative Evaluation

  • Discussion and Future Work

7


Outline1

Outline

  • Ecological Statistics of Contours

    • Groundtruth boundary contours

    • Power law in contours

    • A multi-scale Markov model

  • A Scale-Invariant Representation

  • Learning Models of Curvilinear Continuity

  • Quantitative Evaluation

  • Discussion and Future Work

8


Human segmented natural images

Human-Segmented Natural Images

[Martin et al, ICCV 2001]

1,000 images, >14,000 segmentations

9


Contour geometry

t(s+1)

s+1

t(s)

s

Contour Geometry

  • First-Order Markov Model

    [Mumford 94, Williams & Jacobs 95]

    • Curvature: white noise ( independent from position to position )

    • Tangent t(s): random walk

    • Markov assumption: the tangent at the next position, t(s+1), only depends on the current tangent t(s)

10


Contours are smooth

t(s+1)

s+1

t(s)

s

Contours are Smooth

P( t(s+1) | t(s) )

marginal distribution of tangent change

11


Testing the markov assumption

Testing the Markov Assumption

Segment the contours at high-curvature positions

12


Learning a scale invariant model for curvilinear continuity

Prediction: Exponential Distribution

If the first-order Markov assumption holds…

  • At every step, there is a constant probability p that a high curvature event will occur

  • High curvature events are independent from step to step

    Let L be the length of a segment between high-curvature points

  • P( L>=k ) = (1-p)k

  • P( L=k ) = p(1-p)k

    L has an exponential distribution

  • 13


    Learning a scale invariant model for curvilinear continuity

    Empirical Distribution: Power Law

    Probability

    Contour segment length L

    14


    Learning a scale invariant model for curvilinear continuity

    Power Laws in Nature

    • Power laws widely exist in nature

      • Brightness of stars

      • Magnitude of earthquakes

      • Population of cities

      • Word frequency in natural languages

      • Revenue of commercial corporations

      • Connectivity in Internet topology

        … …

    • Usually characterized by self-similarity and scale-invariant phenomena

    15


    Learning a scale invariant model for curvilinear continuity

    t(1)(s+1)

    s+1

    • Coarse-to-fine contour completion

    • [Ren & Malik 02]

    Multi-scale Markov Models

    t(s+1)

    • Assume knowledge of contour orientation at coarser scales

    s+1

    2nd Order Markov:

    P( t(s+1) | t(s) , t(1)(s+1) )

    Higher Order Models:

    P( t(s+1) | t(s) , t(1)(s+1), t(2)(s+1), … )

    t(s)

    s

    16


    Learning a scale invariant model for curvilinear continuity

    Contour Synthesis

    First-Order Markov:

    P( t(s+1) | t(s) )

    Multi-scale Markov:

    P( t(s+1) | t(s) , t(1)(s+1), t(2)(s+1), … )

    [Ren & Malik 02]

    17


    Outline2

    Outline

    • Ecological Statistics of Contours

    • A Scale-Invariant Representation

      • Piecewise linear approximation

      • Constrained Delaunay Triangulation

    • Learning Models of Curvilinear Continuity

    • Quantitative Evaluation

    • Discussion and Future Work

    18


    Learning a scale invariant model for curvilinear continuity

    Use Pb (probability of boundary) as input

    Combining local brightness, texture and color cues

    Trained from human-marked segmentation boundaries

    Outperform existing local boundary detectors including Canny

    Local “Probability of Boundary”

    • [Martin, Fowlkes & Malik 02]

    19


    Learning a scale invariant model for curvilinear continuity

    Threshold Pb and find connected boundary pixels

    Recursively split the boundaries until each piece is approximately straight

    b

    b

    a

    c

    a

    c

    Split at C

    Piecewise Linear Approximation

    minimize 

    20


    Learning a scale invariant model for curvilinear continuity

    Standard in computational geometry

    Dual of the Voronoi Diagram

    Unique triangulation that maximizes the minimum angle

    avoiding long skinny triangles

    Efficient and simple randomized algorithm

    Delaunay Triangulation

    21


    Learning a scale invariant model for curvilinear continuity

    A variant of the standard Delaunay Triangulation

    Keeps a given set of edges in the triangulation

    [Chew 87]

    [Shewchuk 96]

    • Still maximizes the minimum angle

    • Widely used in geometric modeling and finite elements

    Constrained Delaunay Triangulation

    22


    Learning a scale invariant model for curvilinear continuity

    A typical scenario of contour completion

    high contrast

    high contrast

    low contrast

    • CDT picks the “right” edge, completing the gap

    The “Gap-filling” Property of CDT

    23


    Learning a scale invariant model for curvilinear continuity

    Examples

    Image

    Pb

    CDT

    24


    Learning a scale invariant model for curvilinear continuity

    Black: gradient edges or G-edges

    Green: completed edges or C-edges

    Examples

    25


    Outline3

    Outline

    • Ecological Statistics of Contours

    • A Scale-Invariant Representation

    • Learning Models of Curvilinear Continuity

      • Transferring Groundtruth to CDT

      • A simple model of local continuity

      • A global model w/ Conditional Random Fields

    • Quantitative Evaluation

    • Discussion and Future Work

    26


    Learning a scale invariant model for curvilinear continuity

    Transferring Groundtruth to CDT

    • Human-marked boundaries are given on the pixel-grid

    • Label the CDT edges by bipartite matching

    d

    distance threshold d in matching

    CDT edges

    Phuman: percentage of pixels

    matched to groundtruth

    human-marked

    boundaries

    27


    Learning a scale invariant model for curvilinear continuity

    pb1, G1

    pb0, G0

    Model for Continuity

    • Goal: define a continuity-enhanced Pb on CDT edges

    • Consider a pair of adjacent edges in CDT:

      • Each edge has an associated set of features

        • average Pb over the pixels belonging to this edge

        • indicator G, gradient edge or completed edge?

      • Continuity: angle 

    “bi-gram”

    28


    Learning a scale invariant model for curvilinear continuity

    Binary Classification

    • Assuming contours are always closed: each vertex in the CDT graph is adjacent to either zero or two true boundary edges

    • A binary classification problem: (0,0) or (1,1)

    “bi-gram”

    29


    Learning a scale invariant model for curvilinear continuity

    pb1, G1

    pb0, G0

    Learning Local Continuity

    • Binary classification: (0,0) or (1,1)

    • Transferred Groundtruth labels on CDT edges

    • Features:

      • average Pb

      • (G0*G1): both are gradient edges?

      • angle 

    • Logistic regression

    30


    Learning a scale invariant model for curvilinear continuity

    PbL: Pb + Local Continuity

    Evidence of continuity comes from both ends

    pb1, G1

    pb2, G2

    1

    2

    pb0, G0

    take max. over all possible pairs

    L

    L

    =

    PbL

    31


    Learning a scale invariant model for curvilinear continuity

    Variants of the Local Model

    • More variants of the local model

      • alternative classifiers ( SVM, HME, … )

      • 4-way classification

      • additional features

      • learning a 3-edge (tri-gram) model

      • learning how to combine evidence from both ends

    • No significant improvement in performance

    32


    Learning a scale invariant model for curvilinear continuity

    Local inference

    Xi+1

    Xi

    A Global Model of Continuity?

    X={X1,X2,…,Xm}

    Global inference incorporating all

    local continuity information?

    33


    Learning a scale invariant model for curvilinear continuity

    For each edge i, define a set of features

    {g1,g2,…,gh}

    Potential function exp(i)at edge i

    For each junction j, define a set of features

    {f1,f2,…,fk}

    Potential function exp(j)at juncion j

    Conditional Random Fields

    X={X1,X2,…,Xm}

    [Pietra, Pietra & Lafferty 97]

    [Lafferty, McCallum & Pereira 01]

    34


    Learning a scale invariant model for curvilinear continuity

    Conditional Random Fields

    Potential function on edges {exp(i)}

    Potential function on junctions {exp(j)}

    This defines a probability distribution over X:

    X={X1,X2,…,Xm}

    where

    Estimate P(Xi|)

    35


    Learning a scale invariant model for curvilinear continuity

    Buliding a CRF Model

    • What are the features?

      • edge features are easy: Pb, G

      • junction features: type and continuity

    • How to make inference?

    • How to learn the parameters?

    X={X1,X2,…,Xm}

    Estimate P(Xi|)

    36


    Learning a scale invariant model for curvilinear continuity

    degg=0,degc=2

    degg=0,degc=2

    Junction Features in CRF

    • Junction types (degg,degc):

    degg=1,degc=0

    degg=0,degc=2

    degg=1,degc=2

    • Continuity term for degree-2 junctions

    37


    Learning a scale invariant model for curvilinear continuity

    Inference w/ Belief Propagation

    Fr

    • Belief Propagation

      • Xi: state of the node (edge) i

      • Fq: state of the factor (junction) q

      • potentials on Xi,Xj,Xk, Fq={Xi, Xj, Xk}

      • want to compute PbG=P(Xi)

      • mqi: “belief” about Xi from Fq

    Xj

    mjq

    mir

    mqi

    Xi

    Fq

    mkq

    Xk

    • The CDT graph has many loops in it

    38


    Learning a scale invariant model for curvilinear continuity

    Inference w/ Loopy Belief Propagation

    • Loopy Belief Propagation

      • just like belief propagation

      • iterates message passing until convergence

      • lack of theoretical foundations and known to have convergence issues

      • however becoming popular in practice

      • typically applied on pixel-grid

    • Works well on CDT graphs

      • converges fast

      • produces empirically sound results

    [Berrou 93], [Freeman 98], [Murphy 99], [Weiss 97,99,01]

    39


    Learning a scale invariant model for curvilinear continuity

    Learning the Parameters

    • Maximum-likelihood estimation in CRF

      Let denote the groundtruth labeling on the CDT graph

    • Many possible optimization techniques

      • gradient descent, iterative scaling, conjugate gradient, …

    • Gradient descent works well

    40


    Learning a scale invariant model for curvilinear continuity

    there are more non-boundary edges than boundary edges

    a continuation is better than a line-ending

    junctions are rare

    G-edges are better for continuation than C-edges

    Interpreting the Parameters

    • The junction parameters (degg,degc) on the horse dataset:

    (0,0)= 2.8318

    (1,0)= 1.1279

    (2,0)= 1.3774

    (3,0)= 0.0342

    (2,0)= 1.3774

    (1,1)= -0.6106

    (0,2)= -0.9773

    41


    Outline4

    Outline

    • Ecological Statistics of Contours

    • A Scale-Invariant Representation

    • Learning Models of Curvilinear Continuity

    • Quantitative Evaluation

      • The precision-recall framework

      • Experimental results on three datasets

    • Discussion and Future Work

    42


    Learning a scale invariant model for curvilinear continuity

    Datasets

    • Baseball player dataset [Mori et al 04]

      • 30 news photos of baseball players in various poses, 15 training and 15 testing

    • Horse dataset [Borenstein & Ullman 02]

      • 350 images of standing horses facing left, 175 training and 175 testing

    • Berkeley Segmentation Dataset [Martin et al 01]

      • 300 Corel images of various natural scenes and ~2500 segmentations, 200 training and 100 testing

    43


    Learning a scale invariant model for curvilinear continuity

    Evaluating Boundary Operators

    • Precision-Recall Curves [Martin, Fowlkes & Malik 02]

      • threshold the output boundary map

      • bipartite matching with the groundtruth

    m pixels on human-marked boundaries

    k matched pairs

    n detected pixels above a given threshold

    Precision = k/n, percentage of true positives

    Recall = k/m, percentage of groundtruth being detected

    • Project CDT edges back to the pixel-grid

    44


    Learning a scale invariant model for curvilinear continuity

    Use Phuman the soft groundtruth

    label defined on CDT graphs:

    precision close to 100%

    Pb averaged over CDT edges: no worse than the orignal Pb

    No Loss of Structure in CDT

    45


    Learning a scale invariant model for curvilinear continuity

    Continuity improves boundary detection in both low-recall and high-recall ranges

    Global inference helps; mostly in low-recall/high-precision

    Roughly speaking,

    CRF>Local>CDT only>Pb

    46


    Learning a scale invariant model for curvilinear continuity

    47


    Learning a scale invariant model for curvilinear continuity

    48


    Learning a scale invariant model for curvilinear continuity

    Image

    Pb

    Local

    Global

    49


    Learning a scale invariant model for curvilinear continuity

    Image

    Pb

    Local

    Global

    50


    Learning a scale invariant model for curvilinear continuity

    Image

    Pb

    Local

    Global

    51


    Learning a scale invariant model for curvilinear continuity

    Image

    Pb

    Local

    Global

    52


    Learning a scale invariant model for curvilinear continuity

    In Conclusion…

    • Constrained Delaunay Triangulation is a scale-invariant discretization of images with little loss of structure;

    • Boundary contours are scale-invariant in nature;

    • Moving from 100,000 pixels to <1000 edges, CDT achieves great statistical and computational efficiency;

    • Curvilinear Continuity improves boundary detection;

      • the local model of continuity is simple yet very effective

      • global inference of continuity further improves performance

      • Conditional Random Fields w/ loopy belief propagation works well on CDT graphs

    • Mid-level vision is useful.

    53


    Learning a scale invariant model for curvilinear continuity

    Future Work

    • To add more features into CRF

      • region-based features

      • avoiding spurious completions

      • tri-gram model

    • To train CRF w/ different criteria

      • e.g., area under the precision-recall curve

      • Max-margin Markov networks

    • To use CRF for feature selection

    • To apply CDT+CRF to other mid-level vision problems, e.g., figure/ground organization

    54


    Learning a scale invariant model for curvilinear continuity

    Figure/Ground Organization

    • A classical problem in Gestalt psychology

      [Rubin 1921]

    • “Perceptual organization after grouping”

    • Gestalt principles for figure/ground

      • surroundedness, size, convexity, parallelism, symmetry, lower-region, common fate, familiar configuration, …

    • Very few computational studies

      [Hinton 86], [von der Heydt 93]

    55


    Learning a scale invariant model for curvilinear continuity

    • Shape context [Belongie, Malik & Punicha 01]

    • Clustering shape context into prototypical shape configurations or “shapemes”

    • Local figure/ground discrimination with shapemes

    Using Shapemes for Figure/Ground

    • To capture mid-level information:

      “local” shape configuration

    56


    Learning a scale invariant model for curvilinear continuity

    Shapemes

    57


    Learning a scale invariant model for curvilinear continuity

    F

    G

    G

    G

    F

    F

    common

    F

    G

    F

    G

    G

    F

    uncommon

    Junction Types for Figure/Ground

    58


    Learning a scale invariant model for curvilinear continuity

    • One feature for each junction type

    F

    G

    G

    G

    F

    F

    • Add a continuity term

    CRF for Figure/Ground

    F={F1,F2,…,Fm}

    Fi{Left,Right}

    59


    Learning a scale invariant model for curvilinear continuity

    Preliminary Results on Figure/Ground

    • Chance error rate

    • Local operator w/ shapemes

    • Using human segmentations:

      • Averaging local cues on human-marked boundaries

      • CRF w/ junction type

      • CRF w/ junction type and continuity

    • To use CDT graphs

    50%

    39%

    29%

    28%

    21%

    60


    Learning a scale invariant model for curvilinear continuity

    Thank You

    61


    Learning a scale invariant model for curvilinear continuity

    62


  • Login