slide1 l.
Download
Skip this Video
Loading SlideShow in 5 Seconds..
Gist PowerPoint Presentation
Download Presentation
Gist

Loading in 2 Seconds...

play fullscreen
1 / 71

Gist - PowerPoint PPT Presentation


  • 187 Views
  • Uploaded on

Gist. The of the Scene. Essence . Trayambaka Karra. KT. and Garold Fuks. The “Gist” of a scene. If this is a street this must be a pedestrian . Physiological Evidence. People are excellent in identifying pictures (Standing L., QL. Exp. Psychol. 1973).

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about 'Gist' - lihua


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
slide1

Gist

The of the Scene

Essence

Trayambaka Karra

KT

andGarold Fuks

the gist of a scene
The “Gist” of a scene

If this is a street this must be a pedestrian

physiological evidence
Physiological Evidence
  • People are excellent in identifying pictures(Standing L., QL. Exp. Psychol. 1973)
  • Change Blindness (seconds)

(Simons DJ,Levin DT,Trends Cog.Sci. 97)

  • Gist: abstract meaning of scene
  • Obtained within 150 ms(Biederman, 1981, Thorpe S. et.al 1996 )
  • Obtained without attention(Oliva & Schyns, 1997, Wolfe,J.M. 1998)
  • Possibly derived via statistics of low-level structures
  • (e.g. Swain & Ballard, 1991)
what is the gist
What is the “gist”
  • Inventory of the objects

(2-3 objects in 150 msec Luck & Vogel, Nature 390, 1997 )

  • Relation between objects (layout)

(J. Wolfe, Curr. Bio. 1998, 8 )

  • Presence of other objects
  • “Visual stuff” – impression of low level features
how does the gist works
How does the “Gist” works

Statistical

Properties

Object

Properties

R.A. Rensink, lecture notes

outline
Context Modeling

Previous Models

Scene based Context Model

Context Based Applications

Place Identification

Object Priming

Control of Focus of Attention

Scale Selection

Scene Classification

Joint Local and Global Features Applications

Object Detection and Localization

Summary

Outline
probabilistic framework
Probabilistic Framework

MAP Estimator

  • v – image measurements
  • O – object property
  • Category (o)
  • Location (x)
  • Scale (σ)
object centered object detection
Object-Centered Object Detection
  • The only image features relevant to object detection
  • are those belonging to the object and not the
  • background

B. Moghaddam, A. Petland IEEE, PAMI-19 1997

the gist of a scene9
The “Gist” of a scene

Context can provide prior

Local features can be ambiguous

scene based context model
Scene Based Context Model

Background provides a likelihood of finding an object

Prob(Car/image) = low

Prob(Person/image) = high

context modeling
Context Modeling
  • Previous Context Models

(Fu, Hammond and Swain, 1994,Haralick, 1983; Song et al, 2000)

    • Rule Based Context Model
    • Object Based Context Model
  • Scene centered context representation

(Oliva and Torralba, 2001,2002)

structural description

Rule Based Context Model

Structural Description

O1

Above

O2

O2

Touch

Above

O3

Left-of

Right-of

O4

O4

rule based context model
Rule Based Context Model

Fu, Hammond and Swain, 1994

object based context model
Object Based Context Model
  • Context is incorporated only through prior
  • probability of object combinations in the world

R. Harralick, IEEE, PAMI-5 1983

scene based context model15
Scene Based Context Model

What are the features representing scene - ?

  • Statistics of local low level features
  • Color histograms
  • Oriented band pass filters
context features vc
Context Features - Vc

g1(x)

v(x,1)

g2(x)

v(x,2)

gK(x)

v(x,K)

context features vc17
Context Features - Vc

Gabor filter

People, no car

Car , no people

context features summary
Context Features - Summary

I(x)

Bank

Of

Filters

Dimension

Reduction

PCA

probability from features
Probability from Features

How to obtain context based probability priors P(O/vc)

on object properties - ?

  • GMM - Gaussian Mixture Model
  • Logistic regression
  • Parzen window
probability from features gmm
Probability from Features GMM

P(Object Property/Context)

Need to study two probabilities:

P(vc/O) – likelihood of the features given the presence of an object

P(vc/¬O) – likelihood of the features given the absence of an object

Gaussian Mixture Model:

The unknown parameters are learnt by EM algorithm

probability from features22
Probability from Features

How to obtain context based probability priors P(O/vc)

on object properties - ?

  • GMM - Gaussian Mixture Model
  • Logistic regression
  • Parzen window
probability from features logistic regression24
Probability from Features Logistic Regression

Example

O = having back problems

vc = age

Training Stage

  • - The log odds for 20 year old person
  • The log odds ratio when comparing two persons
  • who differ 1 year in age

Working Stage

probability from features25
Probability from Features

How to obtain context based probability priors P(O/vc)

on object properties - ?

  • GMM - Gaussian Mixture Model
  • Logistic regression
  • Parzen window
what did we have so far
What did we have so far…
  • Context Modeling
  • Context Based Applications
    • Place Identification
    • Object Priming
    • Control of Focus of Attention
    • Scale Selection
    • Scene Classification
place identification
Place Identification

Goal: Recognize specific locations

place identification29
Place Identification

A.Torralba, K.Murphy, W. Freeman, M. Rubin ICCV 2003

place identification30
Place Identification

Decide only when

Precision vs. Recall rate:

A.Torralba, P. Sinha, MIT AIM 2001-015

object priming
Object Priming
  • How do we detect objects in an image?
    • Search the whole image for the object model.
    • What if I am searching in images where the object doesn’t exist at all?
      • Obviously, wasting “my precious” computational resources. --------- GOLUM.
  • Can we do better and if so, how?
    • Use the “great eye”, the contextual features of the image (vC), to predict the probability of finding our object of interest, o in the image i.e. P(o / vC).
object priming32
Object Priming …..
  • What to do?
    • Use my experience to learn

from a database of images with

  • How to do it?
    • Learn the PDF , by a mixture of Gaussians
    • Also, learn the PDF
control of focus of attention
Control of Focus of Attention
  • How do biological visual systems use to deal with the analysis of complex real-world scenes?
    • by focusing attention into image regions that require detailed analysis.
modeling the control of focus of attention
Modeling the Control of Focus of Attention

How to decide which regions are “more” important than others?

  • Local–type methods
      • Low level saliency maps – regions that have different properties than their neighborhood are considered salient.
      • Object centered methods.
  • Global-type methods
      • Contextual control of focus of attention
contextual control of focus of attention
Contextual Control of Focus of Attention
  • Contextual control is both
    • Task driven (looking for a particular object o) and
    • Context driven (given global context information: vC)
  • No use of object models (i.e. ignores object centered features)
contextual control of focus of attention39
Contextual Control of Focus of Attention …
  • Focus on spatial regions that have high probability of containing the target object o given context information (vC)
  • For each location x, lets calculate the probability of presence of the object o given the context vC.
  • Evaluate the PDF

based on the past experience of the system.

contextual control of focus of attention40
Contextual Control of Focus of Attention …

Learning Stage: Use the Swiss Army Knife, the EM algorithm, to estimate the parameters

scale selection
Scale Selection
  • Scale selection is
      • a fundamental problem in computer vision.
      • a key bottleneck for object-centered object detection algorithms.
  • Can we estimate scale in a pre-processing stage?
      • Yes, using saliency measures of low-level operators across spatial scales.
      • Other methods? Of course, …..
scene classification
Scene Classification
  • Strong correlation between the presence of many types of objects.
  • Do not model this correlation directly. Rather, use a “common” cause, which we shall call “scene”.
  • Train a Classifier to identify scenes.
  • Then all we need is to calculate
what did we have so far47
What did we have so far…
  • Context Modeling
  • Context Based Applications
  • Joint Local and Global Features Applications
    • Object Detection and Localization

Need new tools: Learning and Boosting

weak learners
Weak Learners
  • Given (x1,y1),…,(xm,ym) where
  • Can we extract “rules of thumb” for classification purposes?
  • Weak learner finds a weak hypothesis (rule of thumb)

h : X {spam, non-spam}

decision stumps
Decision Stumps
  • Consider the following simple family of component classifiers generating ±1 labels:

h(x;p) = a[xk > t] - b

where p = {a, b, k, t}. These are called decision stumps.

  • Sign (h) for classification and mag (h) for a confidence measure.
  • Each decision stump pays attention to only a single component of the input vector.
ponders his maker ponders his will
Ponders his maker, ponders his will
  • Can we combine weak classifiers to produce a single strong classifier in a simple manner:

hm(x) = h(x;p1) + …. + h(x;pm)

where the predicted label for x is the sign of hm(x).

  • Is it beneficial to allow some of the weak classifiers to have more “votes” than others:

hm(x) = α1h(x;p1) + …. + αmh(x;pm)

where the non-negative votes αi can be used to emphasize the components more reliable than others.

boosting
Boosting

What is boosting?

  • A general method for improving the accuracy of any given weak learning algorithm.
  • Introduced in the framework of PAC learning model.
  • But, works with any weak learner (in our case the decision stumps).
boosting52
Boosting …..
  • A boosting algorithm sequentially estimates and combines classifiers by re-weighting training examples (each time concentrating on the harder examples)
    • each component classifier is presented with a slightly different problem depending on the weights
  • Base Algorithms
    • a set of “weak” binary (±1) classifiers h(x;p) such as decision stumps
    • normalized weights D1(i) on the training examples, initially set to uniform (D1(i) = 1 / m)
adaboost
AdaBoost
  • At the tth iteration we find a weak classifier h(x;pt) for which the classification error is better than chance.
  • The new component classifier is assigned “votes” based on its performance
  • The weights on the training examples are updated according to

where Zt is a normalization factor.

gambling
Gambling

Gari

KT

Uri

object detection and localization
Object Detection and Localization
  • 3 Families of Approaches
    • Parts based
      • Object defined as spatial arrangement of small parts.
    • Region based
      • Use segmentation to extract a region of image from the background and deduce shape and texture info from its local features.
    • Patch based
      • Use local features to classify each rectangular image region as object or background.
      • Object detection is reduced to a binary classification problemi.e compute just P(OCi = 1 / vCi)

where OCi = 1 if patch i contains (part of) an object of class C

vCi = the feature vector for patch i computed for class C.

summary feature vector extraction
Summary: Feature Vector Extraction

12 * 30 *2 = 720 features

object detection
Object Detection …..
  • Do I need all the features for a given object class?
  • If so, what features should I extract for a given object class?
    • Use training to learn which features are more important than others.
classifier boosted features
Classifier: Boosted Features
  • What is available?
    • Training data is v = the features of the patch containing an object o.
  • Weak learners pay attention to single features:
    • ht(v) picks best feature and threshold:
  • Output is
    • ht(v) = output of weak classifier at round t
    • αt = weight assigned by boosting
  • ~100 rounds of boosting
using the gist for object localization
Using the Gist for Object Localization
  • Use gist to predict the possible location of the object.
  • Should I run my detectors only in that region?
    • No! Misses detection if the object is at any other location.
    • So, search everywhere but penalize those that are far from predicted locations.
  • But how?
using the gist for object localization67
Using the Gist for Object Localization ….
  • Construct a feature vector

which combines the output of the boosted classifier, and the difference .

  • Train another classifier to compute
summary
Summary
  • Context Modeling
    • Previous Models
    • Scene based Context Model
summary70
Summary
  • Context Modeling
  • Context Based Applications
    • Place Identification
    • Object Priming
    • Control of Focus of Attention
    • Scale Selection
    • Scene Classification
summary71
Summary
  • Context Modeling
  • Context Based Applications
  • Joint Local and Global Features Applications
    • Object Detection and Localization