semi automatic image annotation n.
Skip this Video
Loading SlideShow in 5 Seconds..
Semi-Automatic Image Annotation PowerPoint Presentation
Download Presentation
Semi-Automatic Image Annotation

Loading in 2 Seconds...

play fullscreen
1 / 20

Semi-Automatic Image Annotation - PowerPoint PPT Presentation

  • Uploaded on

Semi-Automatic Image Annotation. Liu Wenyin, Susan Dumais, Yanfeng Sun, HongJiang Zhang, Mary Czerwinski and Brent Field Microsoft Research. Outline. Introduction: What, Why, and How Our Approach: Semi-Automatic Processes and Algorithms Automated Performance Evaluation

I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
Download Presentation

PowerPoint Slideshow about 'Semi-Automatic Image Annotation' - Jimmy

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
semi automatic image annotation

Semi-Automatic Image Annotation

Liu Wenyin, Susan Dumais, Yanfeng Sun,

HongJiang Zhang, Mary Czerwinski

and Brent Field

Microsoft Research

  • Introduction: What, Why, and How
  • Our Approach:
    • Semi-Automatic
    • Processes and Algorithms
  • Automated Performance Evaluation
  • Usability Studies
  • Concluding Remarks
what it is and why
What it is and Why
  • Image Annotation is a process of labeling images with keywords to describe semantic content
  • For image indexing and retrieval in image databases
  • Annotated images can be found more easily using keyword-based search
image annotation approaches
Image Annotation Approaches
  • Totally Manual Labeling (Gong et al., 1994)
    • Enter keywords when image is loaded/registered/browsed
    • Accurate but labor-intensive, tedious, and subjective
  • Direct Manipulation Annotation (Shneiderman and Kang 2000)
    • Drag and drop keywords (from a predefined list ) onto image
    • Still manual, also limited to predefined keywords (can’t be many)
  • Automatic Approaches: Efficient but less reliable and not always applicable compared to human annotation---how to grab this when no text context?
    • By Image Understanding/Recognition (Ono et al. 1996)
    • By Associating with environmental text (Shen et al. 2000; Srihari et al. 2000; Lieberman 2000)
our proposed approach
Our Proposed Approach
  • Semi-Automatic Approach
    • User provides initial query and relevance feed back.
    • Feedback used to “semi-automatically” annotate images
    • Trade-off between manual and automatic
    • Achieve both accuracy and efficiency
    • Increase productivity
  • Employ Content-Based Image Retrieval (CBIR), text matching, and Relevance Feedback (RF)
algorithms for matching
Algorithms for Matching
  • Visual Similarity Measurement
    • Features: color histogram/moments/coherence, Tamura coarseness, pyramid wavelet texture, etc
    • Distance model: Euclidean distance
  • Semantic (Keywords) Similarity Measurement
    • Features: keyword vectors, TF*IDF
    • Metrics: dot product and cosine normalization
  • Overall similarity: weighted average of the above two
algorithms to refine search
Algorithms to Refine Search
  • Image Relevance Feedback Algorithms
    • There are many algorithms can be used
    • Cox et al. (1996)
    • Rui and Huang (2000)
    • Vasconcelos and Lippman (1999)
  • Lu et al. 2000 is employed in MiAlbum for text and images
    • Modified Rocchio’s Formula
    • Uses both semantics (keywords) and image-based features during relevance feedback
semi automatic annotation during relevance feedback
Semi-Automatic Annotation During Relevance Feedback
  • In each keyword-query search cycle
  • When positive and negative examples provided,
    • Increase the weight of the keyword for all positive examples
    • Decrease the weight of the keyword for all negative examples
    • Relevance feedback algorithm refines and puts more relevant images in top ranks for further selection as positive examples
  • Repeat the feedback process
possible future automatic annotation
Possible Future Automatic Annotation
  • When a new image is added…
  • Find top N similar images using image metrics
  • Most frequent keywords among annotations of these top N similar images are potential annotations, and could be automatically added with low weight or presented to user as potential annotations
  • TBD--Need to be confirmed in further RF process
automated performance evaluation
Automated Performance Evaluation
  • Test Ground Truth Database
    • 12,200 images in 122 categories from Corel DB
    • Category name is ground truth annotation
  • Automatic Experimental Process
    • Use category name as query feature for image retrieval
    • Among first 100 retrieved images, those belonging to this category are used as positive feedback examples others as negative
  • Performance Metrics
    • Retrieval accuracy and annotation coverage
usability studies
Usability Studies
  • Objectives
    • 2 studies examined overall usability of MiAlbum
    • The usability of the semi-automatic annotation strategy
  • Tasks
    • Import pictures, annotate pictures, find pictures, and use relevance feedback
  • Questionnaires including but not limited to
    • Overall ease of entering annotations for images
    • Impact of annotation on ease of searching for images
    • Satisfaction of search refinement & relevance feedback
questionnaire results
Questionnaire Results
  • Overall ease of entering annotations: 5.6/7.0
  • Ease to search annotated photos: 6.3/7.0
  • Intuitiveness of refining search: 4.1/7.0
  • Other Comments
    • Positive on “semi-automatic”: (1) When using the up and down hands the software automatically annotated the photos chosen. (2) The ability to rate pictures on like/dislike and have the software go from there.
    • Negative: difficulties in understanding the feedback process and how the matching algorithm operated.
concluding remarks
Concluding Remarks
  • A Semi-automatic Annotation Strategy Employing
    • Available image retrieval algorithms and
    • Relevance feedback
  • Automatic Performance Evaluation
    • Efficient compared to manual annotation?
    • More accurate than automatic annotation
  • Usability Studies
    • Preliminary usability results are promising
    • Need to improve the discoverability of the feedback process and the underlying matching algorithm