Depth Matters: Influence of Depth Cues on
Sponsored Links
This presentation is the property of its rightful owner.
1 / 42

Depth Matters: Influence of Depth Cues on Visual Saliency PowerPoint PPT Presentation


  • 122 Views
  • Uploaded on
  • Presentation posted in: General

Depth Matters: Influence of Depth Cues on Visual Saliency. Congyan Lang , Tam V. Nguyen, Harish Katti , Karthik Yadati , Mohan Kankanhalli , and Shuicheng Yan Todays ’ Presenter : Daniel Segal Computer Vision – ECCV 2012

Download Presentation

Depth Matters: Influence of Depth Cues on Visual Saliency

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript


Depth Matters: Influence of Depth Cues on

Visual Saliency

Congyan Lang , Tam V. Nguyen, Harish Katti , KarthikYadati ,

Mohan Kankanhalli , and Shuicheng Yan

Todays’ Presenter :

Daniel Segal

Computer Vision – ECCV 2012

12th European Conference on Computer Vision, Florence, Italy, October 7-13, 2012, Proceedings, Part II


Authors

Congyan Lang Tam V. Nguye Harish Katti


Seminar outline

  • Problem motivation

  • Challenges

  • Related work

  • Proposed solution

  • Pros & Cons

  • Limitations

  • Future ideas


Problem motivation

  • Problem presentation

  • Does depth affects saliency?

  • If so How to incorporate depth data?


Problem motivation

  • Why saliency?

  • Surveillance

  • Search and Rescue

  • Medical application

  • Image automated cropping

  • Advertising

  • Target recognition

  • Video summarizations

  • Preprocess for other algorithms


Problem motivation

  • Why incorporate depth?

  • Are observers fixation different when viewing 3D images?

  • Human natural visual attention evolved in 3D environment

  • Absent of 3D fixation data


Challenges

  • Finding efficient saliency models

  • Difficult to model top-down processes effect

  • Integrating depth additional information

  • Absent datasets for 3D stimulus

  • Absent of images dataset with corresponding depth maps

  • Subjective

Which one is more

conspicuous?


Related work

Computing Visual Attention from Scene Depth (2000) in ICPR

Authors :NabilOuerhani and Heinz Hiigli

  • Intro

  • Based on Itti & Koch saliency algorithm

  • Easy to compute in parallel

  • Easy to incorporate depth features

  • Only visual evaluation method


Top-Down overview

Integration

Feature extraction

Feature Map 1

Center Surround

Conspicuity Map 1

Range Finder

Conspicuity Map i

Saliency

Map

Feature Map i

Center Surround

Video

Camera

Feature Map n+m

Center Surround

Conspicuity Map n+m


Integration step

  • Assign weights that promotes conspicuity

M

Wi

Σ

m

Wi

Conspicuity map i


  • Limitation

  • No statistical analysis to prove improvements

  • No comparison with other methods

  • No quantities evaluation method

  • Proposed solution advantages

  • A new 3D dataset was created for statitical analysis

  • Evaluation and comparison with different methods

  • Experiments to investigate 3D saliency


Related work

Pre-Attentive detection of depth saliency using stereo vision(2010) in AIPR

Authors :M. Zaheer Aziz and BarbelMertsching

  • Intro

  • Depth approximation using stereo images

  • Relates only to depth saliency


Top-Down overview

IR

Algorithm

IL

Depth saliency magnitude


Preprocessing

Clipping

Smoothing

Segmentation

IR

IR

1 1

1 1 1

1 1 1

1/9

IL

Depth saliency magnitude


Key insight

IR

IL

IR

IL


Main Algorithm explained/1

Segmentation

IR

Δ(x, y)

Remove extra stripes

-

IL


Main Algorithm explained/2

Remove occluded stripes

Assign region depth


  • Experiments and results

  • Human subjects marked depth saliency labels

  • Efficiency factor defined

  • Capability factor defined

Nf

Nf

Nf<Ns

Ns

Ns

Nf>Ns

Penalty

Nf =# of labels found

Ns =# of labels


  • Experiments and results


Related work

  • Limitation

  • Requires stereo images

  • Approximated depth calculation

  • Emphasis on runtime

  • Experiments somehow dubious

  • Self invented evaluation index

  • Hard to compare with other methods

  • Ignoring fusion with contrast saliency algo.


Related work

  • Proposed solution advantages

  • Messured depth data

  • Statistical analysis and comparison

  • Applicable on all 2D saliency algorithms

  • Using conventional evaluation index


Proposed solution

  • Top-Down overview

    • Dataset collection (3D/2D) and analysis

      • Extracting Stereoscopic image pair generation for 3D display

      • Perform experiments to gain fixation maps

      • Observations and Statistics

    • Incorporating depth priors

    • Experiments and results


Experiment setup


Dataset Examples

  • Rejected images from dataset

    • images overlapping content with other images

    • Images with significant artifacts after the smoothing process


  • Apply smoothing to depth maps

D==0 ?

D==0 ?

1 1 1

1 0 1

1 1 1

Laplacian

Smooth

Depth map

1/8∙

In the same

super pixel

Depth Map


  • Stereoscopic image pair generation for 3D display

Experiments conducted using

active shutter glasses on a

3D LCD display

Pixels translation:

xl = xp+ρ, xr = xp−ρ

ρ = parallax/2


Quantitative evaluation methods/1

  • Three quantitative methods for performance evaluation

    • Correlation coefficient(for all images in dataset)

      • Correlation between to given maps for different depth range bins

    • Saliency Ratio

      • Saliency mean energy in depth bin

    • AUC of the ROC

      • Use each one saliency map to predict another


Quantitative evaluation methods/2

  • Correlation coefficient

  • Saliency Ratio

  • AUC of the ROC


Observations and statistics

  • Observation 1

  • Depth cues modulate visual saliency at farther depth ranges

  • humans fixate preferentially at closer depth ranges


Observations and statistics

  • Observation 2

  • few objects account for majority of the fixations

    The average AUC for the entire 3D fixation dataset is 0.7399

    and 0.7046 for 2D fixation dataset

Using fixation maps as predictor for labeled object


Observations and statistics

  • Observation 3

  • The relationship between depth and saliency is non-linear and characteristic for low and high depth-of-field scenes


Observations and statistics

  • Observation 4:

  • Fixation distribution discrepancy when multiple salient stimuli present different depth planes

CC – Correlation coefficient


  • Incorporating depth priors/1

  • Proposed pdf

    • P(s,d/DOF) is GMM (Gaussian Mixture Model)

Training set

(80% of the

dataset)

Down

Sample

EM

Algo.

GMM model

Parameters

For each

DOF bin

P(s(x)|d(x), DOF)

200x200 pix

640x480 pix

S(x) = ψ(x)(⊕/⊗)P(s/d,DOF)


  • Incorporating depth priors/2

GMM

Pdf

From training

Selected

GMM

Pdf

Calc

DOF

Depth map

Improved

Saliency map

+/*

Saliency map

Saliency

Algo.

Input image


  • Experiments and results


  • Experiments and results


  • Experiments and results


Proposed solution

  • Depth saliency


Pros & Cons

  • Pros

    • Novel and provides a basis for further research

    • Simple approach

    • Gives an overall improvement for existing alg.

  • Cons

    • Few missing implementation details

    • No source code/other descriptive material

    • Requires a depth map

    • Still no basis for comparison with other methods

Future ideas

Integrate depth priors in various algo. instead of late fusion


Questions ?


Thanks


  • Login