sky finder attribute based sky image search
Download
Skip this Video
Download Presentation
Sky Finder: Attribute-based Sky Image Search

Loading in 2 Seconds...

play fullscreen
1 / 38

Sky Finder: - PowerPoint PPT Presentation


  • 1045 Views
  • Uploaded on

Sky Finder: Attribute-based Sky Image Search Litian Tao Lu Yuan Jian Sun SIGGRAPH 2009 Hong Kong University of Science and Technology Beihang University Microsoft Research Asia Difficult to capture a beautiful sky Dynamic rang of scene > dynamic rang of camera

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about 'Sky Finder: ' - Audrey


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
sky finder attribute based sky image search

Sky Finder: Attribute-based Sky Image Search

Litian Tao Lu Yuan Jian Sun

SIGGRAPH 2009

Hong Kong University of

Science and Technology

Beihang University

Microsoft Research Asia

difficult to capture a beautiful sky
Difficult to capture a beautiful sky
  • Dynamic rang of scene > dynamic rang of camera
  • Good timing is very important
need to search a sky image
Need to search a sky image
  • Sky replacement
  • Background images for composition
    • 2D design, film production, image editing
frustrating using search engines
Frustrating using search engines

Bing Image Search

Google Image Search

our system
Our System
  • Search desired sky images from a sky dataset

500,000 sky images from Flickr

related work image as query
Related Work – Image as Query
  • Scene Completion [Hays and Efros 2007]
  • Face Swapping [Bitouk et al. 2008]
  • Photo Clip Art [Lalonde et al. 2008]
related work text as query
Related Work – Text as Query
  • Semantic Photo Synthesis [Johnson et al. 2006]
challenges
Challenges
  • Image-query based approahces
    • Finding a good image example is also a search problem.
    • Difficult for interactive search in a large dataset
  • Text -query based (Semantic Photo Syn/Image Search E)
    • Only can find images including sky
    • Without further control
our solution attribute based search
Our solution - Attribute based search
  • Semantic attributes
    • Category
    • Layout
    • Horizon height
    • Sun existence/position
    • Richness
  • A large sky database
data collection
Data Collection
  • Images from Flickr. com
    • High quality photos
    • Human-labeled tags

sky

cloudy

sunset

sunrise

storm

colourskies

southfloridasky

beautyofsky

sky&clouds

5 keywords

95 user groups

1.3 million images

attributes category
Attributes – Category
  • 3 Categories
    • blue-sky
    • cloudy-sky
    • sunset
  • 2000 training images

blue-sky

cloudy-sky

sunset

uncertain

bag of word representation
Bag-of-Word Representation

Feature: SIFT + Color

Step1: Codebook Generation

Feature Space Partition

Feature Quantization

Randomized Forests

[Moosmann et al. 2006]

2:Representation

bag of codewords

attributes category15
Attributes – Category

cloudy-sky SVM

blue-sky SVM

sunset SVM

bag of blue-sky words

blue-sky score

bag of cloudy-sky words

cloudy score

bag of sunset words

sunset score

attributes
Attributes
  • Category
  • Layout
  • Horizon height
  • Sun existence/position
  • Richness

Defined on sky region segmentation

sky region segmentation
Sky Region Segmentation

Test

Training

Three Sky/non-sky pixel classifiers

blue-sky

cloudy-sky

Graph-cut segmentation

sunset

attributes layout horizon
Attributes – Layout & Horizon

full-sky

object-in-sky

landscape

normal-sky

others

A>95%

95%>A>70%

A<70%

: Bounding Box –cover 95% sky pixels

# sky pixels

A

: in the bounding box

# pixels

: Horizon height

attributes sun
Attributes – Sun

luminance channel

magenta channel in CMYK

sunset image

Bright Region

Extraction

Shape filter

sun mask

attributes richness
Attributes – Richness

Images

Sky region

Edge map

Richness

attribute accuracy
Attribute Accuracy
  • Test dataset
    • 6, 000 images
  • Performance

Precision =

Recall =

# trueclassifiedblue-sky images

# true classifiedblue-sky images

# classifiedblue-sky images

# total blue-sky images

experimental evaluation
Experimental Evaluation
  • Test dataset
    • 6, 000 images
  • Performance

Precision =

in blue-sky

Recall =

in blue-sky

# true detectedsky pixels

# truedetectedsky pixels

Both sky and non-sky region are gloomy

# detectedsky pixels

# total sky pixels

experimental evaluation23
Experimental Evaluation
  • Test dataset
    • 6, 000 images
  • Performance

Precision =

in sunset

Recall =

in sunset

# true detectedsuns

# truedetectedsuns

# detectedsuns

# total suns

Sun is largely occluded by clouds

color based re ranking
Color based re-ranking
  • Sky color representation
    • Color signature:
  • Similarity
    • Earth Mover’s Distance (EMD)

(a)

sunset + landscape + horizon + sun position

after color-based re-ranking

(b)

user interface
User Interface

Category SVM scores (3D)

user interface26
User Interface

Horizon and sun canvas

Category Triangle

blue-sky

Layout

Richness

landscape

cloudy-sky

sunset

normal-sky

full-sky

after PCA

object-in-sky

others

path search
Path Search
  • How to find such a sky image?

Our solution: sky graph + smoothed path

?

graph construction
Graph Construction
  • Building a graph is difficult
    • Pairwise distance computation is expensive
    • Semantic metric is required
  • Sparse graph using attributes
    • 1: use categoryand richness attributes -> 2000 candidates
    • 2: re-rank candidates by color ->Top 200 neighbours
    • 3: use color similarity (EMD) as edge weight
path search30
Path Search

max-transition-cost = 2

  • Finding a path
    • Shortest path
    • Our path

2

B

C

2

2

A

D

5

max-transition-cost = 5

sky replacement
Sky Replacement
  • Need to change the foreground color
    • Sometimes not visually plausible
  • Our method: category-specific color transfer

Apply learned transfer from(s0,s1) to foreground

original image

retrieved image

blue-sky

a

sky_o

sky

b

?

non-sky_o

non-sky

cloudy-sky

sky replacement32
Sky Replacement
  • Direct cut-and-paste?
    • Sometimes not visually plausible
    • Need to change the foreground color
  • How to compute the color transfer variables?

Apply b on foreground

original image

retrieved image

a

sunset

sky

sky

b

?

non-sky

non-sky

summary
Summary
  • Semantic level search
    • Semantic attributes
    • Intuitive user interface
    • Path finding
  • Very efficient and scalable
    • Image search -> attributes -> text search
on line demo
On-line Demo
  • http://jiansun5/SkyFinderEntry/
sun in blue sky category
Sun in blue-sky category
  • Ratio of images containing sun in three categories
    • blue-sky: 0.9%
    • cloudy-sky: 10.0%
    • sunset: 25.6%.
other attributes
Other attributes
  • Common attributes in our half a million dataset
    • Category, layout, horizon height, sun, richness
  • Other potential attributes:
    • Lightening
    • Moon
    • Polar light
  • We will add these attributes as the database size grows.
ad