medial visual fragment image representation for perceptual organization and segmentation l.
Download
Skip this Video
Loading SlideShow in 5 Seconds..
Medial Visual Fragment Image Representation for Perceptual Organization and Segmentation PowerPoint Presentation
Download Presentation
Medial Visual Fragment Image Representation for Perceptual Organization and Segmentation

Loading in 2 Seconds...

play fullscreen
1 / 56

Medial Visual Fragment Image Representation for Perceptual Organization and Segmentation - PowerPoint PPT Presentation


  • 162 Views
  • Uploaded on

Medial Visual Fragment Image Representation for Perceptual Organization and Segmentation. Amir Tamrakar and Benjamin Kimia LEMS, Brown University POCV 2004, Washington DC. Main Theme.

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about 'Medial Visual Fragment Image Representation for Perceptual Organization and Segmentation' - ghalib


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
medial visual fragment image representation for perceptual organization and segmentation

Medial Visual Fragment Image Representation for Perceptual Organization and Segmentation

Amir Tamrakar and Benjamin Kimia

LEMS, Brown University

POCV 2004,

Washington DC

main theme
Main Theme
  • Developed a novel intermediate representation of images, the Medial Visual Fragment representation that
    • Encodes simultaneously both contour and region properties.
    • Can deal with open contours and semi-closed regions
    • Represents fragments of objects or surfaces for which there is evidence.
  • Developed a language in terms of the shock graph suitable for perceptual reasoning with these visual fragments.
  • This talk is about a suitable intermediaterepresentation and not about an algorithm for segmentation.
  • Basically, this representation
    • Encodes spatial relationships between contours and regions
    • Improves the notion of good continuation of object contours by including region properties with it
goal of segmentation
Goal of Segmentation
  • To partition the image into units corresponding to perceived objects or surfaces of objects
  • Traditionally, this has involved segmenting it into a non-overlapping set of fragments. This “jigsaw puzzle” like segmentation doesn’t suffice to explain our perception.
    • Hence the term “Perceptual Organization” is often used to describe grouping of these fragments into units that agree with our perception.
  • A layered and hierarchical representation seems to be more appropriate

=

+

gestalt laws
Gestalt Laws
  • For Perceptual Grouping, the Gestalt Laws of grouping are still the de facto theory to use.
  • Laws of similarity and good continuity are the most popular.
    • Continuity of Contours
    • Continuity of Surfaces
    • Continuity of volumes

From Peter Tse, 1999

boundaries vs regions
Boundaries vs. Regions
  • Thus, two kinds of cues are available from an image for the presence of objects
      • Boundaries
        • Locations of discontinuities of surface properties.
      • Interior Regions
        • Cohesive regions due to similarities in surface properties.
  • These cues are related and often consider duals.
    • A closed boundary encloses a region and the perimeter of a region is a boundary.
    • However, not all boundaries are closed and not all regions are bounded by contours.
  • In the past, people have often worked with them separately but people have realized the need to combine them
    • Humans segmentation makes use of both these information (Fowlkes, Martin and Malik, ’03)
use of boundaries in segmentation
Use of Boundaries in Segmentation
  • Local differential operators are used to locate them (edgels)
    • Hence, the information is inherently noisy.
  • These edgels are grouped into long smooth curve segments on the basis of good continuation.
    • Shashua and Ullman, Alter and Basri, Guy and Medioni, Harold and Horaud, Sarkar and Boyer, Williams and Thornber etc.
  • A closed curve is more salient that an open one
    • Kovacz and Julesz, Elder, Williams, etc.

From Shashua and Ullman, 1988

use of boundaries in segmentation7
Use of Boundaries in Segmentation
  • Often times boundary fragments are forced to join others for producing a segmentation with rich topology instead of the low-order connectivity that most people look for (Rothwell, Mundy Hoffman,1995).

Using the VanDuc edge detector and linker,

(Rothwell, Mundy, Hoffman, Nguyen, 95)

contour continuity is not enough
Contour Continuity is NOT Enough.

It’s object fragment continuity that one really cares about.

surface continuity
Surface Continuity
  • Surface continuity on at least one side of the contour seems desirable.

Surface Does not Continue

Surface Continues

fundamental problem for contours
Fundamental Problem for Contours
  • There is no sense of spatial relationship between contours.
  • One cannot determine how far out from the curve one should venture to collect the required surface information.
  • In other words, contours do not code the extent of the object/surface that they bound unless they are closed.
  • Contour based grouping would benefit from having this kind of spatial information.
another problem
Another Problem
  • Some very salient contours are not closed because they arise due to 3D structure (folds joining the body).
  • They necessarily terminate (cusps)
  • Traditionally, these open contours are discarded and a lot of structural information is lost with it.
  • A segmentation scheme would tremendously benefit by having a representation that allows for such open contours.

From Zucker et al

use of regions in segmentation
Use of Regions in Segmentation
  • Pixels are grouped into atomic patches on the basis of the similarity of their surface properties like intensity, color, texture, etc.
  • Adjacent patches are grouped to form larger regions e.g., Seeded Region Growing, etc.
  • As a graph partitioning problem e.g. Normalized Cuts (Shi and Malik), etc.
  • Multigrid-based Segmentation by Weighted Aggregation (SWA) (Sharon et al)

SWA segmentation, From Sharon et al, 2001

common problems
Common Problems
  • The boundaries of these regions may or may not be meaningful as contours of objects.
    • A large portion arises merely due to competition between adjacent regions as they try to grow.
  • They are especially problematic if there are gradual variations in intensity (shading) or texture
    • Integrating over the patch integrates this variation and exaggerates the difference between the patches
  • Leakage is a common problem.
    • The addition of contours and reasoning about their continuity allows for a splitting of the regions that have merged due to leakage.
conclusion
Conclusion
  • One needs to reason with both kinds of information at the same time.
  • One thus requires an intermediate representation that can provide both kinds of information simultaneously.
  • One should also be able to represent open contours and regional properties associated with them.
motivation
Motivation
  • We want to append on to the contour the regional information around it.
motivation18
Motivation
  • We want to append on to the contour the regional information around it.

But the question is how far off should one go from the contour?

motivation19
Motivation
  • We want to append on to the contour the regional information around it.

Presumably, there are other contours that are

trying to capture some region as well.

motivation20
Motivation
  • We want to append on to the contour the regional information around it.

The Best Answer

is

as far as the

Medial Axis.

Axis

In absence of any other information, the medial axis is the bisector of two regions.

medial visual fragment representation

C+

Medial Axis

Medial Fragment

C-

Medial Visual Fragment Representation
  • The Medial Axis “binds” together a pair of contours and the region between them.
medial visual fragment representation22
Medial Visual Fragment Representation

Definition:

  • In the grassfire analogy of Blum, the burnt region corresponding to the each medial axis segment is a Medial Visual Fragment.
  • i.e., it is the union of all pairs of rays (PP+, PP-) arising from all points along the segment.

C+

Medial Axis

Medial Fragment

C-

medial visual fragment representation23
Medial Visual Fragment Representation

Proposition 1:

  • An image with an associated contour map (a set of curve segments) is partitioned into a set of medial visual fragments.
medial visual fragment representation24
Medial Visual Fragment Representation

Proposition 1:

    • An image with an associated contour map (a set of curve segments) is partitioned into a set of medial visual fragments.
    • Every point P in the image, there exists a shock segment k described by a curve γk parameterized by arclength s with a local coordinate system of axis tangent/normal (T(s), N(s)) and velocity v(s) such that for some t Є [0, r(s)],
  • Proposition 2:
    • These fragments satisfy the segmentation criterion.
medial visual fragment representation25
Medial Visual Fragment Representation

Average intensity

computed at each

Medial Fragment

Original Image

with its contour fragments

The Medial Axis

computed from

these contour

fragments

The Medial

Fragments

medial visual fragment representation26
Medial Visual Fragment Representation
  • Medial Visual Fragments formed by various configurations of curve-pairs.
    • Between two open contours
    • Between one single open contour
    • Enclosed by a single closed contour
    • Between a pair of closed contours

D

A

B

C

strengths of our representation
Strengths of Our Representation
  • Allows for a combined region and contour description explicitly.

C+

Medial Axis

C-

Visual Fragment

strengths of our representation28
Strengths of Our Representation
  • Allows for open contours and semi-closed regions.
strengths of our representation29
Strengths of Our Representation
  • Adaptively partitions the region around an open or closed contour into “influence zones” for gathering regional information around the contour.
strengths of our representation30
Strengths of Our Representation
  • Allows for reasoning about fragment continuity in terms of “skeletal continuity” (Continuity of a pair of contours and the region between them).
perceptual organization using visual fragments32
Perceptual Organization using Visual Fragments
  • Philosophy:
    • This representation ties PO and Recognition
      • We have proposed before that Perceptual Organization is only one half of Recognition (POCV 01)
    • The process of PO is a process of Perceptual Reasoning.
    • We have developed a Language in which to perform this type of reasoning
    • The language is that of transformations of the shock graph.
perceptual organization

Image

Medial Visual

Fragments

Medial Axis Transforms

Perceptual Organization
  • The transformations on the representation transform the underlying image domain as well.

From POCV ‘01

perceptual organization using visual fragments34
Perceptual Organization using Visual Fragments
  • Perceptual Organization is accomplished by Transforming the Medial Axis and hence the Medial Visual Fragments.
  • The viability of the transformed image defines cost of the transformation.
  • The Canonical Transforms on the Medial Axis are:
    • Gap Transform
    • Loop Transform
  • All the operations required for segmentation/perceptual grouping can be described as compositions of the canonical Medial Axis Transforms.
  • The choice of the optimal sequence of transformations is the process of perceptual Organization.
medial visual fragment transforms gap transform
Medial Visual Fragment Transforms: Gap Transform
  • Gaps in the contours produce “degenerate” Medial Axis segments (i.e. arising from a pair of points)
  • The removal of such a segment (Gap Transform) closes the gap by linking the contour fragments.

Completion Curve

Post Gap Transform

Contours with a gap

Gap Axis segment

medial visual fragment transforms gap transform36
Medial Visual Fragment Transforms: Gap Transform
  • Ingredients for a viable gap transform:
  • Fragments A & B go together AND/OR fragments C & D go together
  • Reasonable curve completion is possible between C1 and C2.
medial visual fragment transforms gap transform37
Medial Visual Fragment Transforms: Gap Transform
  • Post gap transform:
  • The curves C1 and C2 have been connected by the completion curve
  • The Medial Visual Fragments have merged
medial visual fragment transforms gap transform38
Medial Visual Fragment Transforms: Gap Transform
  • Different varieties of Gap Transforms:
    • Completion assisted by another contour
    • Completion assisted by contours on either side
    • To Form Junctions
medial visual fragment transforms gap transform39
Medial Visual Fragment Transforms: Gap Transform

From Berkeley Segmentation Database

medial visual fragment transforms gap transform40
Medial Visual Fragment Transforms: Gap Transform

From Berkeley Segmentation Database

medial visual fragment transforms loop transform42
Medial Visual Fragment Transforms Loop Transform

Motivation:

An intervening curve fragment will claim its territory preventing

C1 and C2 from talking to one another.

medial visual fragment transforms loop transform43
Medial Visual Fragment Transforms Loop Transform

Motivation:

Internal Structure (e.g. fold)

Surface Markings

medial visual fragment transforms loop transform44
Medial Visual Fragment Transforms Loop Transform

Implementation:

The reverse process is the Loop Transform.

medial visual fragment transforms loop transform45
Medial Visual Fragment Transforms Loop Transform

New layer

Implementation:

+

Denotes

attachment

  • Lift element into a new layer attached to the current fragment
  • Fill in under it to reflect this removal.
medial visual fragment transforms loop transform46

+

Medial Visual Fragment Transforms Loop Transform

New layer

Motivation:

=

Texture mappedon to it

Main fragment

medial visual fragment transforms loop transform47
Medial Visual Fragment Transforms Loop Transform

From Berkeley Segmentation Database

medial visual fragment transforms loop transform48
Medial Visual Fragment Transforms Loop Transform

From Berkeley Segmentation Database

reasoning with occlusion

+

Reasoning with Occlusion
  • An occlusion presents itself as a loop in the shock graph.
  • The removal of this loop “lifts” the occluder onto a new layer.
  • The occluded fragments can then be completed.
perceptual grouping example
Perceptual Grouping Example
  • A torus being grouped behind the occluder
perceptual grouping example51
Perceptual Grouping Example
  • A torus being grouped behind the occluder
perceptual grouping example52
Perceptual Grouping Example
  • A torus being grouped behind the occluder

Mustard region is grouped and “lifted” on to another layer as an occluder.

perceptual grouping example53
Perceptual Grouping Example
  • A torus being grouped behind the occluder

Only the contours inside the “holes” are left behind.

perceptual grouping example54
Perceptual Grouping Example
  • A torus being grouped behind the occluder
perceptual grouping example55

+

Perceptual Grouping Example
  • A torus being grouped behind the occluder

=

Final Segmentation:

summary
Summary
  • There was a clear need for an intermediate representation
    • That carried both contour and region properties
    • That could work with partial information in either domain
  • The Medial Visual Fragment Representation meets those requirements.
  • It also defines spatial relationships between contours and regions.
  • We developed a language for reasoning with them in terms of the transformations of the shock graphs: The gap and loop transforms.
  • This presents a whole new view of segmentation as a sequence of transformations.
  • One of the key features is a layered representation as the output of segmentation.