Sky Finder: Attribute-based Sky Image Search - PowerPoint PPT Presentation

Sky finder attribute based sky image search l.jpg
Download
1 / 38

Sky Finder: Attribute-based Sky Image Search Litian Tao Lu Yuan Jian Sun SIGGRAPH 2009 Hong Kong University of Science and Technology Beihang University Microsoft Research Asia Difficult to capture a beautiful sky Dynamic rang of scene > dynamic rang of camera

Related searches for Sky Finder: Attribute-based Sky Image Search

I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.

Download Presentation

Sky Finder: Attribute-based Sky Image Search

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript


Sky finder attribute based sky image search l.jpg

Sky Finder: Attribute-based Sky Image Search

Litian Tao Lu Yuan Jian Sun

SIGGRAPH 2009

Hong Kong University of

Science and Technology

Beihang University

Microsoft Research Asia


Difficult to capture a beautiful sky l.jpg

Difficult to capture a beautiful sky

  • Dynamic rang of scene > dynamic rang of camera

  • Good timing is very important


Need to search a sky image l.jpg

Need to search a sky image

  • Sky replacement

  • Background images for composition

    • 2D design, film production, image editing


Frustrating using search engines l.jpg

Frustrating using search engines

Bing Image Search

Google Image Search


Our system l.jpg

Our System

  • Search desired sky images from a sky dataset

500,000 sky images from Flickr


Related work image as query l.jpg

Related Work – Image as Query

  • Scene Completion [Hays and Efros 2007]

  • Face Swapping [Bitouk et al. 2008]

  • Photo Clip Art [Lalonde et al. 2008]


Related work text as query l.jpg

Related Work – Text as Query

  • Semantic Photo Synthesis [Johnson et al. 2006]


Challenges l.jpg

Challenges

  • Image-query based approahces

    • Finding a good image example is also a search problem.

    • Difficult for interactive search in a large dataset

  • Text -query based (Semantic Photo Syn/Image Search E)

    • Only can find images including sky

    • Without further control


Our solution attribute based search l.jpg

Our solution - Attribute based search

  • Semantic attributes

    • Category

    • Layout

    • Horizon height

    • Sun existence/position

    • Richness

  • A large sky database


Data collection l.jpg

Data Collection

  • Images from Flickr. com

    • High quality photos

    • Human-labeled tags

sky

cloudy

sunset

sunrise

storm

colourskies

southfloridasky

beautyofsky

sky&clouds

5 keywords

95 user groups

1.3 million images


Attributes category l.jpg

Attributes – Category

  • 3 Categories

    • blue-sky

    • cloudy-sky

    • sunset

  • 2000 training images

blue-sky

cloudy-sky

sunset

uncertain


Bag of word representation l.jpg

Bag-of-Word Representation

Feature: SIFT + Color

Step1: Codebook Generation

Feature Space Partition

Feature Quantization

Randomized Forests

[Moosmann et al. 2006]

2:Representation

bag of codewords


Attributes category15 l.jpg

Attributes – Category

cloudy-sky SVM

blue-sky SVM

sunset SVM

bag of blue-sky words

blue-sky score

bag of cloudy-sky words

cloudy score

bag of sunset words

sunset score


Attributes l.jpg

Attributes

  • Category

  • Layout

  • Horizon height

  • Sun existence/position

  • Richness

Defined on sky region segmentation


Sky region segmentation l.jpg

Sky Region Segmentation

Test

Training

Three Sky/non-sky pixel classifiers

blue-sky

cloudy-sky

Graph-cut segmentation

sunset


Attributes layout horizon l.jpg

Attributes – Layout & Horizon

full-sky

object-in-sky

landscape

normal-sky

others

A>95%

95%>A>70%

A<70%

: Bounding Box –cover 95% sky pixels

# sky pixels

A

: in the bounding box

# pixels

: Horizon height


Attributes sun l.jpg

Attributes – Sun

luminance channel

magenta channel in CMYK

sunset image

Bright Region

Extraction

Shape filter

sun mask


Attributes richness l.jpg

Attributes – Richness

Images

Sky region

Edge map

Richness


Attribute accuracy l.jpg

Attribute Accuracy

  • Test dataset

    • 6, 000 images

  • Performance

Precision =

Recall =

# trueclassifiedblue-sky images

# true classifiedblue-sky images

# classifiedblue-sky images

# total blue-sky images


Experimental evaluation l.jpg

Experimental Evaluation

  • Test dataset

    • 6, 000 images

  • Performance

Precision =

in blue-sky

Recall =

in blue-sky

# true detectedsky pixels

# truedetectedsky pixels

Both sky and non-sky region are gloomy

# detectedsky pixels

# total sky pixels


Experimental evaluation23 l.jpg

Experimental Evaluation

  • Test dataset

    • 6, 000 images

  • Performance

Precision =

in sunset

Recall =

in sunset

# true detectedsuns

# truedetectedsuns

# detectedsuns

# total suns

Sun is largely occluded by clouds


Color based re ranking l.jpg

Color based re-ranking

  • Sky color representation

    • Color signature:

  • Similarity

    • Earth Mover’s Distance (EMD)

(a)

sunset + landscape + horizon + sun position

after color-based re-ranking

(b)


User interface l.jpg

User Interface

Category SVM scores (3D)


User interface26 l.jpg

User Interface

Horizon and sun canvas

Category Triangle

blue-sky

Layout

Richness

landscape

cloudy-sky

sunset

normal-sky

full-sky

after PCA

object-in-sky

others


Slide27 l.jpg

Demo


Path search l.jpg

Path Search

  • How to find such a sky image?

    Our solution: sky graph + smoothed path

?


Graph construction l.jpg

Graph Construction

  • Building a graph is difficult

    • Pairwise distance computation is expensive

    • Semantic metric is required

  • Sparse graph using attributes

    • 1: use categoryand richness attributes -> 2000 candidates

    • 2: re-rank candidates by color ->Top 200 neighbours

    • 3: use color similarity (EMD) as edge weight


Path search30 l.jpg

Path Search

max-transition-cost = 2

  • Finding a path

    • Shortest path

    • Our path

2

B

C

2

2

A

D

5

max-transition-cost = 5


Sky replacement l.jpg

Sky Replacement

  • Need to change the foreground color

    • Sometimes not visually plausible

  • Our method: category-specific color transfer

Apply learned transfer from(s0,s1) to foreground

original image

retrieved image

blue-sky

a

sky_o

sky

b

?

non-sky_o

non-sky

cloudy-sky


Sky replacement32 l.jpg

Sky Replacement

  • Direct cut-and-paste?

    • Sometimes not visually plausible

    • Need to change the foreground color

  • How to compute the color transfer variables?

Apply b on foreground

original image

retrieved image

a

sunset

sky

sky

b

?

non-sky

non-sky


Slide33 l.jpg

Demo


Summary l.jpg

Summary

  • Semantic level search

    • Semantic attributes

    • Intuitive user interface

    • Path finding

  • Very efficient and scalable

    • Image search -> attributes -> text search


On line demo l.jpg

On-line Demo

  • http://jiansun5/SkyFinderEntry/


Questions l.jpg

Questions?


Sun in blue sky category l.jpg

Sun in blue-sky category

  • Ratio of images containing sun in three categories

    • blue-sky: 0.9%

    • cloudy-sky: 10.0%

    • sunset: 25.6%.


Other attributes l.jpg

Other attributes

  • Common attributes in our half a million dataset

    • Category, layout, horizon height, sun, richness

  • Other potential attributes:

    • Lightening

    • Moon

    • Polar light

  • We will add these attributes as the database size grows.


  • Login