Depth Matters: Influence of Depth Cues on
This presentation is the property of its rightful owner.
Sponsored Links
1 / 42

Depth Matters: Influence of Depth Cues on Visual Saliency PowerPoint PPT Presentation


  • 99 Views
  • Uploaded on
  • Presentation posted in: General

Depth Matters: Influence of Depth Cues on Visual Saliency. Congyan Lang , Tam V. Nguyen, Harish Katti , Karthik Yadati , Mohan Kankanhalli , and Shuicheng Yan Todays ’ Presenter : Daniel Segal Computer Vision – ECCV 2012

Download Presentation

Depth Matters: Influence of Depth Cues on Visual Saliency

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript


Depth matters influence of depth cues on visual saliency

Depth Matters: Influence of Depth Cues on

Visual Saliency

Congyan Lang , Tam V. Nguyen, Harish Katti , KarthikYadati ,

Mohan Kankanhalli , and Shuicheng Yan

Todays’ Presenter :

Daniel Segal

Computer Vision – ECCV 2012

12th European Conference on Computer Vision, Florence, Italy, October 7-13, 2012, Proceedings, Part II


Depth matters influence of depth cues on visual saliency

Authors

Congyan Lang Tam V. Nguye Harish Katti


Depth matters influence of depth cues on visual saliency

Seminar outline

  • Problem motivation

  • Challenges

  • Related work

  • Proposed solution

  • Pros & Cons

  • Limitations

  • Future ideas


Depth matters influence of depth cues on visual saliency

Problem motivation

  • Problem presentation

  • Does depth affects saliency?

  • If so How to incorporate depth data?


Depth matters influence of depth cues on visual saliency

Problem motivation

  • Why saliency?

  • Surveillance

  • Search and Rescue

  • Medical application

  • Image automated cropping

  • Advertising

  • Target recognition

  • Video summarizations

  • Preprocess for other algorithms


Depth matters influence of depth cues on visual saliency

Problem motivation

  • Why incorporate depth?

  • Are observers fixation different when viewing 3D images?

  • Human natural visual attention evolved in 3D environment

  • Absent of 3D fixation data


Depth matters influence of depth cues on visual saliency

Challenges

  • Finding efficient saliency models

  • Difficult to model top-down processes effect

  • Integrating depth additional information

  • Absent datasets for 3D stimulus

  • Absent of images dataset with corresponding depth maps

  • Subjective

Which one is more

conspicuous?


Depth matters influence of depth cues on visual saliency

Related work

Computing Visual Attention from Scene Depth (2000) in ICPR

Authors :NabilOuerhani and Heinz Hiigli

  • Intro

  • Based on Itti & Koch saliency algorithm

  • Easy to compute in parallel

  • Easy to incorporate depth features

  • Only visual evaluation method


Depth matters influence of depth cues on visual saliency

Top-Down overview

Integration

Feature extraction

Feature Map 1

Center Surround

Conspicuity Map 1

Range Finder

Conspicuity Map i

Saliency

Map

Feature Map i

Center Surround

Video

Camera

Feature Map n+m

Center Surround

Conspicuity Map n+m


Depth matters influence of depth cues on visual saliency

Integration step

  • Assign weights that promotes conspicuity

M

Wi

Σ

m

Wi

Conspicuity map i


Depth matters influence of depth cues on visual saliency

  • Limitation

  • No statistical analysis to prove improvements

  • No comparison with other methods

  • No quantities evaluation method

  • Proposed solution advantages

  • A new 3D dataset was created for statitical analysis

  • Evaluation and comparison with different methods

  • Experiments to investigate 3D saliency


Depth matters influence of depth cues on visual saliency

Related work

Pre-Attentive detection of depth saliency using stereo vision(2010) in AIPR

Authors :M. Zaheer Aziz and BarbelMertsching

  • Intro

  • Depth approximation using stereo images

  • Relates only to depth saliency


Depth matters influence of depth cues on visual saliency

Top-Down overview

IR

Algorithm

IL

Depth saliency magnitude


Depth matters influence of depth cues on visual saliency

Preprocessing

Clipping

Smoothing

Segmentation

IR

IR

1 1

1 1 1

1 1 1

1/9

IL

Depth saliency magnitude


Depth matters influence of depth cues on visual saliency

Key insight

IR

IL

IR

IL


Depth matters influence of depth cues on visual saliency

Main Algorithm explained/1

Segmentation

IR

Δ(x, y)

Remove extra stripes

-

IL


Depth matters influence of depth cues on visual saliency

Main Algorithm explained/2

Remove occluded stripes

Assign region depth


Depth matters influence of depth cues on visual saliency

  • Experiments and results

  • Human subjects marked depth saliency labels

  • Efficiency factor defined

  • Capability factor defined

Nf

Nf

Nf<Ns

Ns

Ns

Nf>Ns

Penalty

Nf =# of labels found

Ns =# of labels


Depth matters influence of depth cues on visual saliency

  • Experiments and results


Depth matters influence of depth cues on visual saliency

Related work

  • Limitation

  • Requires stereo images

  • Approximated depth calculation

  • Emphasis on runtime

  • Experiments somehow dubious

  • Self invented evaluation index

  • Hard to compare with other methods

  • Ignoring fusion with contrast saliency algo.


Depth matters influence of depth cues on visual saliency

Related work

  • Proposed solution advantages

  • Messured depth data

  • Statistical analysis and comparison

  • Applicable on all 2D saliency algorithms

  • Using conventional evaluation index


Depth matters influence of depth cues on visual saliency

Proposed solution

  • Top-Down overview

    • Dataset collection (3D/2D) and analysis

      • Extracting Stereoscopic image pair generation for 3D display

      • Perform experiments to gain fixation maps

      • Observations and Statistics

    • Incorporating depth priors

    • Experiments and results


Depth matters influence of depth cues on visual saliency

Experiment setup


Depth matters influence of depth cues on visual saliency

Dataset Examples

  • Rejected images from dataset

    • images overlapping content with other images

    • Images with significant artifacts after the smoothing process


Depth matters influence of depth cues on visual saliency

  • Apply smoothing to depth maps

D==0 ?

D==0 ?

1 1 1

1 0 1

1 1 1

Laplacian

Smooth

Depth map

1/8∙

In the same

super pixel

Depth Map


Depth matters influence of depth cues on visual saliency

  • Stereoscopic image pair generation for 3D display

Experiments conducted using

active shutter glasses on a

3D LCD display

Pixels translation:

xl = xp+ρ, xr = xp−ρ

ρ = parallax/2


Depth matters influence of depth cues on visual saliency

Quantitative evaluation methods/1

  • Three quantitative methods for performance evaluation

    • Correlation coefficient(for all images in dataset)

      • Correlation between to given maps for different depth range bins

    • Saliency Ratio

      • Saliency mean energy in depth bin

    • AUC of the ROC

      • Use each one saliency map to predict another


Depth matters influence of depth cues on visual saliency

Quantitative evaluation methods/2

  • Correlation coefficient

  • Saliency Ratio

  • AUC of the ROC


Depth matters influence of depth cues on visual saliency

Observations and statistics

  • Observation 1

  • Depth cues modulate visual saliency at farther depth ranges

  • humans fixate preferentially at closer depth ranges


Depth matters influence of depth cues on visual saliency

Observations and statistics

  • Observation 2

  • few objects account for majority of the fixations

    The average AUC for the entire 3D fixation dataset is 0.7399

    and 0.7046 for 2D fixation dataset

Using fixation maps as predictor for labeled object


Depth matters influence of depth cues on visual saliency

Observations and statistics

  • Observation 3

  • The relationship between depth and saliency is non-linear and characteristic for low and high depth-of-field scenes


Depth matters influence of depth cues on visual saliency

Observations and statistics

  • Observation 4:

  • Fixation distribution discrepancy when multiple salient stimuli present different depth planes

CC – Correlation coefficient


Depth matters influence of depth cues on visual saliency

  • Incorporating depth priors/1

  • Proposed pdf

    • P(s,d/DOF) is GMM (Gaussian Mixture Model)

Training set

(80% of the

dataset)

Down

Sample

EM

Algo.

GMM model

Parameters

For each

DOF bin

P(s(x)|d(x), DOF)

200x200 pix

640x480 pix

S(x) = ψ(x)(⊕/⊗)P(s/d,DOF)


Depth matters influence of depth cues on visual saliency

  • Incorporating depth priors/2

GMM

Pdf

From training

Selected

GMM

Pdf

Calc

DOF

Depth map

Improved

Saliency map

+/*

Saliency map

Saliency

Algo.

Input image


Depth matters influence of depth cues on visual saliency

  • Experiments and results


Depth matters influence of depth cues on visual saliency

  • Experiments and results


Depth matters influence of depth cues on visual saliency

  • Experiments and results


Depth matters influence of depth cues on visual saliency

Proposed solution

  • Depth saliency


Depth matters influence of depth cues on visual saliency

Pros & Cons

  • Pros

    • Novel and provides a basis for further research

    • Simple approach

    • Gives an overall improvement for existing alg.

  • Cons

    • Few missing implementation details

    • No source code/other descriptive material

    • Requires a depth map

    • Still no basis for comparison with other methods

Future ideas

Integrate depth priors in various algo. instead of late fusion


Depth matters influence of depth cues on visual saliency

Questions ?


Depth matters influence of depth cues on visual saliency

Thanks


  • Login