Adding domain specific knowledge
1 / 21

Adding Domain-Specific Knowledge - PowerPoint PPT Presentation

  • Uploaded on
  • Presentation posted in: General

Adding Domain-Specific Knowledge. Amit Singhal & Jiebo Luo Research Laboratories Eastman Kodak Company FUSION 2001, Montreal August 7-10, 2001. Outline of Talk. Problem Statement Background Relevant Prior Art Evidence Fusion Framework Automatic Main Subject Detection System

I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.

Download Presentationdownload

Adding Domain-Specific Knowledge

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript

Adding domain specific knowledge

Adding Domain-Specific Knowledge

Amit Singhal & Jiebo Luo

Research Laboratories

Eastman Kodak Company

FUSION 2001, Montreal

August 7-10, 2001

Outline of talk

Outline of Talk

  • Problem Statement

  • Background

    • Relevant Prior Art

    • Evidence Fusion Framework

    • Automatic Main Subject Detection System

  • Injecting Orientation Information

    • Feature detectors

  • Conclusions

  • Future Work

Main subject detection

Main Subject Detection

  • What Is the Main Subject in A Picture?

    • 1st-party truth (the photographer): in general not available due to the specific knowledge the photographer may have about the setting

    • 3rd-party truth: in general there is good agreement among 3rd-party observers if the photographer successfully used the picture to communicate his interest in the main subject to the viewers

Related prior art

Related Prior Art

  • Main subject (region-of-interest) detection

    • Milanese (1993) : Uses biologically motivated models for identifying regions of interest in simple pictures containing highly contrasting foreground and background.

    • Marichal (et al.) (1996), Zhao (et al.) (1996) : Use a subjective fuzzy modeling approach to describe semantic interest in video sequences (primarily video-conferencing).

    • Syeda-Mahmood (1998) : Uses a color-based approach to isolate regions in an image likely to belong to the same object. Main application is reduction of search space for object recognition

  • Evidence Fusion

    • Pearl (1988) :Provides a theory and evidence propagation scheme for Bayesian networks.

    • Rimey & Brown (1994) : Use Bayesian networks for control of selective perception in a structured spatial scene.

    • Buxton (et al.) (1998) : Use a set of Bayesian networks to integrate sensor information to infer behaviors in a traffic monitoring application.

The evidence fusion framework

The Evidence Fusion Framework

  • Region based representation scheme.

  • Virtual belief sensors map output of physical sensors and algorithmic feature detectors to probabilistic space.

  • Domain knowledge used to generate network structure.

  • Expert knowledge and ground truth-based training methodologies to generate the priors and the conditional probability matrices.

  • Bayesian network combines evidence generated by the sensors and feature detectors using a very fast message passing scheme.

Bayesian networks

Bayesian Networks

  • A directed acyclic graph

  • Each node represents an entity (random variable) in the domain

  • Each link represents a causality relationship and connects two nodes in the network

  • The direction of the link represents the direction of causality

  • Each link encodes the conditional probability between the parent and child nodes

  • Evaluation of the Bayes network is equivalent to knowing the joint probability distribution

Automatic main subject detection system

Automatic Main Subject Detection System

  • An Interesting Research Problem

    • Conventional wisdom (or how a human performs such a task)

      • Object Segmentation -> Object Recognition -> Main Subject Determination

      • Object recognition is an unconstrained problem in consumer photographs

    • Inherent Ambiguity

      • 3rd party probabilistic ground truth

      • Large number of camera sensors and feature detectors

    • Speed and performance scalability concerns

  • Of extreme industrial interest to digital photofinishing

    • Allows for automatic image enhancements to produce better photographic prints

    • Other applications such as

      • Image compression, storage, and transmission

      • Automatic image recompositing

      • Object-based image indexing and retrieval



  • Methodology

    • Produce a belief map of regions in the scene being part of the main subject

    • Utilize a region-based representation of the image derived from image segmentation and perceptual grouping

    • Utilize semantic features (human flesh and face, sky, grass) and general saliency features (color, texture, shape and geometric features)

    • Utilize a Bayes Net-based architecture for knowledge representation and evidence inference

  • Dealing with Intrinsic Ambiguity

    • Ground truth is “probabilistic” not “deterministic”

    • Limitations in our understanding of the problem

  • Dealing with “Weak” Vision Features

    • Reality of the state-of-the-art of computer vision

    • Limited accuracy of the current feature extraction algorithms

The multilevel bayesian network

The Multilevel Bayesian Network

Injecting metadata into the system

Injecting Metadata into the System

  • Sources of metadata

    • Camera : Flash fired, Subject distance, Orientation etc.

    • IU Algorithms : Indoor/Outdoor, Scene type, Orientation etc.

    • User annotation

  • The Bayesian network is very flexible and can be quickly adapted to take advantage of available metadata

  • Metadata enabled knowledge can be injected into the system using

    • Metadata-aware feature detectors

    • Metadata-enhanced Bayesian networks



  • Main difference between orientation-aware and orientation non-aware systems is in the location features

The centrality feature detector

The Centrality Feature Detector

Orientation unaware centrality feature detector

Orientation Unaware Centrality Feature Detector

Orientation aware centrality feature detector

Orientation Aware Centrality Feature Detector

Borderness feature

Borderness Feature

  • Orientation Unaware

    • a=b=c=d=e

  • Orientation Aware

    • a < b < c < d < e

Orientation aware bayesian network

Orientation Aware Bayesian Network

  • Use orientation aware centrality and borderness features

  • Other feature detectors affected by orientation but not retrained:

    • sky, grass

      • Not retrained if BN is used for main subject detection as the location features would account for the orientation information

      • Using orientation information to compute the sky and grass evidence would lead to better performance for a sky or grass detection system.

  • Retrain the links in the Bayesian network for each feature affected by orientation information

    • BorderA-Borderness

    • BorderD-Borderness

    • Borderness-Location

    • Centrality-Location

    • Location-MainSubject

Experimental results

Experimental Results

Experimental results1

Experimental Results

Experimental results2

Experimental Results

Dks plot of orientation aware msd

dKS Plot of Orientation-Aware MSD

Conclusions and future work

Conclusions and Future Work

  • Bayesian networks offer the flexibility of easily incorporating domain specific knowledge such as orientation information into the system

  • This knowledge can be added by :

    • modifying the feature detectors

    • using new feature detectors

    • changing the structure of the Bayesian network

    • retraining the conditional probability matrices associated with the Bayesian network

  • Directions for Future Work

    • Use of additional metadata such as indoor/outdoor, urban/rural, day/night

    • Single super BN versus a library of metadata-aware BNs?

  • Login