300 likes | 397 Views
This study by Devi Parikh and Kristen Grauman explores traditional recognition of animals like dogs, chimpanzees, and tigers through attributes-based recognition. Attributes such as furry, white, black, big, stripped, yellow, help in applications like zero-shot learning. The research emphasizes the importance of discriminative and nameable attributes for effective communication between humans and machines. The study uses an interactive system to engage users in identifying candidate attributes and ensure proposals are both discriminative and nameable. Evaluation involves annotating candidates offline with thousands of responses, focusing on attributes like "black," "spotted," "green," and "smiling." Results show that a structured nameability space aids prediction, an active approach discovers more discriminative splits compared to baselines, and automatically generated descriptions enhance understanding. This interactive system aims to bridge the gap between humans and machines for attribute discovery.
E N D
Interactively Discovery of Attributes Vocabulary Devi Parikh and Kristen Grauman
Traditional Recognition Dog Chimpanzee Tiger ???
Attributes-based Recognition Furry White Black Big Stripped Yellow Stripped Black White Big Dog Chimpanzee Tiger
Applications Zero-shot learning Attributes provide a mode of communication between humans and machines! A Zebra is… White Black Stripped Zebra Image description Stripped Black White Big
Attributes Attributes are most useful if they are • Discriminative • Nameable
Attributes Attributes are most useful if they are • Discriminative • Nameable
Attributes Attributes are most useful if they are • Discriminative • Nameable
Attributes Attributes are most useful if they are • Discriminative • Nameable
Attributes Attributes are most useful if they are • Discriminative • Nameable
Interactive System 1. Name: Fluffy 2. Name: x 3. Name: Metal … How do we show the user a candidate-attribute? How do we ensure proposals are discriminative? How do we ensure proposals are nameable?
Ensure Discriminability Normalized cuts Max Margin Clustering
Ensure Nameability 1. Name: Fluffy 2. Name: x 3. Name: Metal …
Ensure Nameability 1. Name: Fluffy 2. Name: x 3. Name: Metal … Mixture of Probabilistic PCA
Evaluation • Outdoor Scenes • Animals with Attributes • Public Figures Face • Gist and Color features (LDA)
Evaluation • Annotate all candidates off-line “Black” … ~25000 responses
Evaluation • Annotate all candidates off-line … ~25000 responses “Spotted”
Evaluation • Annotate all candidates off-line … ~25000 responses Unnameable
Evaluation • Annotate all candidates off-line … ~25000 responses “Green”
Evaluation • Annotate all candidates off-line … ~25000 responses “Congested”
Evaluation • Annotate all candidates off-line … ~25000 responses “Smiling”
Results Structure exists in nameability space allowing for prediction Our active approach discovers more discriminative splits than baselines
Results Comparing to discriminative-only baseline
Results Comparing to descriptive-only baseline
Results Automatically generated descriptions
Summary • Machines need to understand us • Attributes need to be detectable & discriminative • We need to understand machines • Attributes need to be nameable • Interactive system for discovering attributes