1 / 38

Photo-Quality Enhancement based on Visual Aesthetics

Photo-Quality Enhancement based on Visual Aesthetics. S. Bhattacharya*, R. Sukthankar **, M.Shah * *University of Central Florida, **Intel labs & CMU. Motivation. Outline. Related work Quality enhancement framework Visual aesthetics Aesthetic appeal assessment

bryant
Download Presentation

Photo-Quality Enhancement based on Visual Aesthetics

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. Photo-Quality Enhancement based on Visual Aesthetics S. Bhattacharya*, R. Sukthankar**, M.Shah* *University of Central Florida, **Intel labs & CMU

  2. Motivation

  3. Outline • Related work • Quality enhancement framework • Visual aesthetics • Aesthetic appeal assessment • Enhancement through recomposition • Experimental results • Conclusions • Future directions

  4. Related Work • Low-level (dehazing etc.) K. He, J.Sun X. Tang, “Single Image Haze Removal using Dark Channel Prior”, CVPR 09 • Domain-specific (face beautification etc.) T.Leyvand,et al. , “Data-Driven Enhancement of Facial Attractiveness”, ACM SIGGRAPH 08

  5. Overview Assessment Engine Aesthetic Features Appeal Prediction Image Semantics, Aesthetic Features Aesthetic Model Input Image Enhanced Image Recomposition Enhancement Engine

  6. Visual Aesthetics: Rule of Thirds Motivated by Renaissance Paintings… http://hoocher.com/Joseph_William_Turner/Joseph_William_Turner.htm Rule of thirds: Subject of interest is aligned to one of the stress points Professional photographs also abide this: http://howtophotography.org/wp-content/uploads/2010/06/rule-of-thirds-photo2.jpg

  7. Visual Aesthetics: Golden Ratio ~1.618k Sky Sky Land Sea k http://hoocher.com/Joseph_William_Turner/Joseph_William_Turner.htm Divine proportion: Horizon divides sky and sea/land according to golden ratio. An example professional photographic composition: http://www.dptips-central.com/rules-of-composition.html

  8. Modeling Aesthetics: Dataset Single subject Compositions (384) Landscapes/Seascapes (248) http://www.flickr.com

  9. Modeling Aesthetics: User study … Rank Assignment between 1-5 1 2 15 14 Single subjects Landscapes/Seascapes Ground Truth Appeal Factors http://www.flickr.com

  10. Modeling Aesthetics: User study 1.76 4.23 Best rated images Poorly rated images

  11. Modeling Aesthetics: User study Poor compositions Good Compositions User Agreements Appeal Factor Intervals

  12. Modeling Aesthetics: Features (a) Relative Foreground Location (Rule of Thirds) Visual Attention Center Stress Point

  13. Modeling Aesthetics: Features (b) Visual weight deviation from Golden Ratio (Divine Proportion) Yk Yg

  14. Experiments (Assessment) • Learn Support Vector Regression models • Prediction accuracy: • Single subject compositions ~ 87% • Landscapes/Seascapes ~ 91% Smooth mapping between Appeal factor and Aesthetic Features • Relative Foreground Location • Visual Weight Deviation

  15. Spatial Recomposition

  16. Why Cropping does not work? Optimal Crop

  17. Recomposition: Algorithm I Input Image Optimal Object Placement Spatial Recomposition In-painting Semantic Segmentation Single Subject? Labeled Elements

  18. Semantic Segmentation Input Image Segmented Foreground Sky Geometric Context Classifier* Post Processing Horizon Support *D. Hoiem, A.A. Efros, and M. Hebert, "Geometric Context from a Single Image", ICCV 2005

  19. Optimal Object Placement Find x that Maximizes Appeal Support Neighborhood Labeled Image s.t. neighbors stay “like neighbors” + Intensity Term Gradient Term

  20. Optimization (Example) X PAF = 3.22 Original Image Semantic constraint prevents this Optimal Solution PAF = 3.31 PAF = 4.53 PAF = 3.68

  21. Perspective Scaling Scaled Foreground Visual Attention Center Scaling Factor Optimal location Vanishing Point

  22. Inpainting Foreground Hole Inpaint Hole • Yunjun Zhang. Jiangjian Xiao. Mubarak Shah, “Region Completion in a Single Image”, EUROGRAPHICS 04

  23. Recomposition: Algorithm 2 Input Image Semantic Segmentation Land/Sea scape? Optimally Crop/Expand Labeled Elements Visual Weight Balancing

  24. Balancing Visual Weights • Ratio of Current extents Yk+h Yk • h= vertical extent of the balanced image Yg Yg • Solve for h (sign of h determines crop/expansion)

  25. Experimental Results • Single Subject Composition • Horse is moved to a more visually pleasing location • Scaled appropriately • Appeal increases by 64% • Before Recomposition • After Recomposition

  26. Results PAF = 4.29 PAF = 2.45 • Before • After PAF = 3.98 PAF = 4.46

  27. Results PAF = 4.02 PAF = 4.34 • Before • After PAF = 3.13 PAF = 4.19

  28. Results PAF = 3.77 PAF = 4.25 • Before • After PAF = 3.92 PAF = 4.11

  29. Results PAF = 2.71 PAF = 3.26 • Before • After PAF = 4.68 PAF = 4.06

  30. Results • Optimally cropped support region to increase weights for sky • Appeal factor increased by 51% • Visual weight balancing • Before Recomposition • After Recomposition

  31. Balancing Visual weights PAF = 4.02 PAF = 3.83 • Before • After PAF = 4.38 PAF = 3.92

  32. Balancing Visual weights PAF = 4.71 PAF = 4.02 • Before • After PAF = 4.49 PAF = 4.17

  33. Not Perfect • Algorithm says nice, humans: otherwise Fa = 2.41 (Ground Truth) PAF = 2.34 Fa = 2.54 (Ground Truth) PAF = 3.63 • Before • After

  34. Summary: Optimal Placement • Increased # of Highly rated Images • Decreased # of Poorly rated Images • Before • After

  35. Summary: Visual Weights • Increased # of Highly rated Images • Decreased # of Poorly rated Images • Before • After

  36. Conclusion • Intelligent photo recomposition • Can also be used for aesthetic filtering • Easy to use practical tool

  37. Future Work • Synthesizing ideal image from many photos of the same scene • Recompositionfor videos

  38. Questions?

More Related