besov bayes chomsky plato n.
Download
Skip this Video
Download Presentation
Besov Bayes Chomsky Plato

Loading in 2 Seconds...

play fullscreen
1 / 122

Besov Bayes Chomsky Plato - PowerPoint PPT Presentation


  • 53 Views
  • Uploaded on

Multiscale Geometric Image Processing. Besov Bayes Chomsky Plato. Richard Baraniuk Rice University dsp.rice.edu Joint work with Hyeokho Choi Justin Romberg Mike Wakin. Image Processing. Analyzing, modeling, processing images special class of 2-d functions ex: digital photos

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about 'Besov Bayes Chomsky Plato' - cambria-jovan


Download Now An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
besov bayes chomsky plato

Multiscale Geometric Image Processing

Besov BayesChomsky Plato

Richard Baraniuk

Rice University

dsp.rice.edu

Joint work with

Hyeokho ChoiJustin RombergMike Wakin

image processing
Image Processing
  • Analyzing, modeling, processingimages
    • special class of 2-d functions
    • ex: digital photos
  • Problems:
    • representation, approximation, compression, estimation, restoration, deconvolution, detection, classification, segmentation, …
low level image structures
Low-Level Image Structures
  • Prevalent image structures:
    • smooth regions - grayscale regularity
    • smooth edge contours - geometric regularity
  • Must exploit both for maximum performance
lossy image compression
Lossy Image Compression
  • Given image

approximate using bits

  • Error incurred = distortionExample: squared error
rate distortion analysis
Rate-Distortion Analysis

as

  • Asymptotic analysis; maximize rate

achievable region

transform coding
Transform Coding
  • Quantize coefficients of an image expansion

coefficients

basis, frame

Ex: Fourier sinusoids

DCT (JPEG)

wavelets curvelets …

transform coding1
Transform Coding
  • Quantize coefficients of an image expansion

coefficients

basis, frame

quantize to total bits

jpeg compressed dct
JPEG Compressed (DCT)

PSNR = 26.32dB @ 0.0103bpp

multiscale image analysis
Multiscale Image Analysis
  • Analyze an image a multiple scales
  • How? Zoom out and record information lost in wavelet coefficients

info 1

info 2

info 3

wavelet based image processing
Wavelet-based Image Processing
  • Standard 2-D tensor product wavelet transform
transform domain modeling
Transform-domain Modeling
  • Transform-domain modeling and processing

transform

coefficient

model

transform domain modeling1
Transform-domain Modeling
  • Transform-domain modeling and processing

transformvocabulary

coefficient

model

transform domain modeling2
Transform-domain Modeling
  • Transform-domain modeling and processing
  • Vocabulary + grammar capture image structure
  • Challenging co-design problem

transformvocabulary

coefficient grammar

model

nonlinear image modeling
Nonlinear Image Modeling
  • Natural images do not form a linear space!
  • Form of the “set of natural images”?

+

=

set of natural images
Set of Natural Images

Small: NxN sampled images comprise an extremely small subset of RN2

set of natural images1
Set of Natural Images

Small: NxN sampled images comprise an extremely small subset of RN2

set of natural images2
Set of Natural Images

Small: NxN sampled images comprise an extremely small subset of RN2

Complicated: “Manifold” structure

RN2

wedge manifold1
Wedge Manifold

3 Haar wavelets

wedge manifold2
Wedge Manifold

3 Haar wavelets

set of natural images3
Set of Natural Images

Small: NxN sampled images comprise an extremely small subset of RN2

Complicated: “Manifold” structure

RN2

stochastic projections
Stochastic Projections
  • Projection

= “shadow”

higher d stochastic projections
Higher-D Stochastic Projections
  • Model (partial) wavelet coefficient joint statistics
  • Shadow more faithful to manifold
wavelet based image processing1
Wavelet-based Image Processing
  • Standard 2-D tensor product wavelet transform
wavelet quadtree structure
Wavelet Quadtree Structure
  • Wavelet analysis is self-similar
  • Image parent square divides into 4 children squares at next finer scale
wavelet quadtree correlations
Wavelet Quadtree Correlations
  • Smooth regionsmallwc’s tumble down tree
  • Edge/ridge regionlarge wc’s tumble down tree
zero tree compression
Zero Tree Compression
  • Idea: Prune wavelet subtrees in smooth regions

Z

  • tree-structured thresholding
  • spend bits on (large) “significant” wc’s
  • spend no bits on (small) wc’s…
  • zerotree symbolZ = {wc and all decendants = 0}
zero tree compression1
Zero Tree Compression
  • Label pruned wavelet quadtree with 2 stateszero-tree- smooth region (prune)significant- edge/texture region (keep)

S

smooth

smooth

Z

Z

S

not

smooth

Z

smooth

S

S

S

S

zero tree compression2
Zero Tree Compression
  • Label pruned wavelet quadtree with 2 stateszero-tree- smooth region (prune)significant- edge/texture region (keep)
  • Can optimize placement of Z and S with dynamic programming (SFQ) [Xiong, Ramchandran, Orchard]

S

smooth

smooth

Z

Z

S

not

smooth

Z

smooth

S

S

S

S

jpeg compressed dct1
JPEG Compressed (DCT)

PSNR = 26.32dB @ 0.0103bpp

jpeg2000 compressed wavelets
JPEG2000 Compressed (Wavelets)

PSNR = 29.77dB @ 0.0103bpp

sfq compressed wavelets
SFQ Compressed (Wavelets)

PSNR = 29.77dB @ 0.0103bpp

sfq wc code map
SFQ WC Code Map

green = significantblue = zero tree

the jpeg2000 disappointment
The JPEG2000 Disappointment
  • JPEG2000: replace DCT with wavelet transform
  • Does not improve on decay rate of JPEG!
  • WHY?neither DCT norwavelets are theright transform

JPEG (DCT)

JPEG2000 (wavelets)

slide41

Wavelet Challenges

  • Geometrical info not explicit
  • Inefficient-large number of large wc’s cluster around edge contours, no matter how smooth
wavelets and cartoons
Wavelets and Cartoons

13

26

52

  • Even for a smooth C2 contour, which straightens at fine scales…
2 d wavelets poor approximation
2-D Wavelets: Poor Approximation

13

26

52

  • Even for a smooth C2 contour, which straightens at fine scales…
  • Too many wavelets required!

-term wavelet approximation

not

2 d wavelets poor approximation1
2-D Wavelets: Poor Approximation

13

26

52

  • Even for a smooth C2 contour, which straightens at fine scales…
  • Too many wavelets required!

-bit wavelet approximation

not

solution 1 upgrade the transform
Solution 1: Upgrade the Transform

13

26

52

  • Introduce anisotropic transform
    • curvelets, ridgelets, contourlets, …
  • Optimal error decay rates for some simple images
solution 2 upgrade the processing
Solution 2: Upgrade the Processing

13

26

52

  • Replace coefficient thresholding by a new wc modelthat captures anisotropicspatial correlations
our goal new image models
Our Goal: New Image Models
  • Wavelet coefficient models that capture both geometric and textural aspects of natural images
  • Optimal error decay rates for some image class
geometry model for cartoons
Geometry Model for Cartoons

C2 boundary

  • Toy model: flat regions separated by smooth contours
  • Goal: representation that is
    • sparse
    • simply modeled
    • efficiently computed
    • extensible to texture regions
2 d dyadic partition
2-D Dyadic Partition
  • Multiscale analysis
  • Partition; not a basis/frame
  • Zoom in by factor of 2 each scale
2 d dyadic partition quadtree
2-D Dyadic Partition = Quadtree
  • Multiscaleanalysis
  • Partition; not a basis/frame
  • Zoom in by factor of 2 each scale
  • Each parent node has 4 children at next finer scale
wedgelet representation donoho
Wedgelet Representation [Donoho]

r

  • Build a cartoon using wedgelets on dyadic squares

wedgelet = atomic geometric element

wedgelet representation
Wedgelet Representation

r

  • Build a cartoon using wedgelets on dyadic squares
  • Choose orientation (r,q) from finite dictionary (toroidal sampling)
  • Quad-tree structuredeeper in tree finer curve approximation
wedgelet representation1
Wedgelet Representation
  • Prune wedgelet quadtree to approximate local geometry (adaptive)
  • Decorate leaves with (r,q) parameters
wedgelet inference
Wedgelet Inference
  • Find representation / prune tree to balance a fidelity vs. complexity trade-off
  • For Comp(W) –need a model for the wedgelet representation(quadtree + (r,q))
  • Donoho: Comp(W) = #leaves
wedgelet inference1
Wedgelet Inference
  • Find representation / prune tree to balance a fidelity vs. complexity trade-off
  • O(N) dynamic programmingsolution (bottom-up recursion)
  • Optimal approximation
  • Near-optimal rate/distortion decay
leaves complexity penalty
#Leaves Complexity Penalty
  • Accounts for wedgelet partition size, but not wedgelet orientation

#leaves

#leaves

“smooth”

“simple”

“rough”

“complex”

multiscale geometry model mgm
Multiscale Geometry Model (MGM)
  • Decorate each tree node with orientation (r,q) and then model dependencies thru scale
  • Insight: Smooth curve Geometric innovations small at fine scales
  • Model: Favor small innovations over large innovations (statistically)
multiscale geometry model mgm1
Multiscale Geometry Model (MGM)
  • Wavelet-like geometry model: coarse-to-fine prediction
    • model parent-to-child transitions of orientations

small

innovations

large

slide64
MGM
  • Wavelet-like geometry model: coarse-to-fine prediction
    • model parent-to-child transitions of orientations
  • Markov-1 statistical model
    • state = (r,q) orientation of wedgelet
    • parent-to-child state transition matrix
slide65
MGM
  • Markov-1 statistical model
  • Joint wedgelet Markov probability model:
  • Complexity = Shannon codelength == number of bits to encode
mgm and edge smoothness
MGM and Edge Smoothness

“smooth”

“simple”

“rough”

“complex”

mgm inference
MGM Inference
  • Find representation / prune tree to balance the fidelity vs. complexity trade-off
  • Efficient O(N) solution via dynamic programming
mgm approximation
MGM Approximation
  • Find representation / prune tree to balance the fidelity vs. complexity trade-off
  • OptimalL2 error decay rate for cartoons

single scale

multiscale

wedgelet coding of cartoon images
Wedgelet Coding of Cartoon Images
  • Choosing wedgelets = rate-distortion optimization

Shannon code length

to encode

wedgelet coding of cartoon images1
Wedgelet Coding of Cartoon Images
  • Choosing wedgelets = rate-distortion optimization

~130 bits coded independently

~25 bits coded with MGM

predictive wedgelet coding is optimal
Predictive Wedgelet Coding is Optimal

C2contour (1-d)

Optimal rate-distortion performance

compared to

for Donoho’s leaf-only encoding

joint texture geometry compression
Joint Texture/Geometry Compression
  • Dictionary D = {wavelets} U {wedgelets}
  • Representation tradeoff: texture vs.geometry
recall zero tree compression
Recall – Zero Tree Compression
  • Idea: Prune wavelet subtrees in smooth regions

Z

  • tree-structured thresholding
  • spend bits on (large) “significant” wc’s
  • spend no bits on (small) wc’s…
  • zerotree symbolZ = {wc and all decendants = 0}
zero tree compression3
Zero Tree Compression
  • Label pruned wavelet quadtree with 2 stateszero-tree- smooth region (prune)significant- edge/texture region (keep)

S

smooth

smooth

Z

Z

S

not

smooth

Z

smooth

S

S

Z: all wc’s below=0

S

S

enter geometry
Enter Geometry
  • Label pruned wavelet quadtree with 2 stateszero-tree- smooth region (prune)significant- edge/texture region (keep)
  • Suboptimal NLA/R-D decay w/ edges

S

smooth

smooth

Z

Z

S

S

texture

geometry

S

S

S

S

Z

S

S

S

wedgelets for geometry
Wedgelets for Geometry
  • Label pruned wavelet quadtree with 2 stateszero-tree- smooth region (prune)significant- edge/texture region (keep)
  • Idea: use wedgelets in geometric edge squares

S

smooth

smooth

Z

Z

S

G

texture

geometry

G: wc’s below are wc’s of a wedgelet

S

S

S

S

wedgelets for geometry1
Wedgelets for Geometry
  • Label pruned wavelet quadtree with 3 stateszero-tree- smooth region (prune)geometry- edge region (prune)significant- texture region (keep)

S

smooth

smooth

Z

Z

S

G

texture

geometry

G: wc’s below are wc’s of a wedgelet

S

S

S

S

wedgelets for geometry2
Wedgelets for Geometry
  • Label pruned wavelet quadtree with 3 stateszero-tree- smooth region (prune)geometry- edge region (prune)significant- texture region (keep)
  • Optimize placement of Z, G, S by dyn. programming

S

smooth

smooth

Z

Z

S

G

texture

geometry

G: wc’s below are wc’s of a wedgelet

S

S

S

S

wedgelet trees for geometry
Wedgelet Trees for Geometry
  • Label pruned wavelet quadtree with 3 stateszero-tree- smooth region (prune)geometry- edge region (prune)significant- texture region (keep)
  • Optimize placement of Z, G, S by dyn. programming

S

smooth

smooth

Z

Z

S

G

texture

G: wc’s below are wc’s of a wedgelet tree

S

S

geometry

S

S

image approximation
Image Approximation

S

Z

Z

S

G

S

S

S

S

Z

Z

S

S

S: describe (code) wc’s

G

S

S

image approximation1
Image Approximation

S

Z

Z

S

G

S

S

S

S

Z: all wc’s below=0

ie: wc’s of

Z

Z

S

S

S: describe (code) wc’s

G

S

S

image approximation2
Image Approximation

S

Z

Z

S

G

S

S

S

S

Z: all wc’s below=0

ie: wc’s of

G: wc’s below are wc’s of a wedgelet ie: wc’s of

“wedgeprint”

Z

Z

S

S

S: describe (code) wc’s

G

S

S

optimality of wedgeprints
Optimality of Wedgeprints

C2texture (2-d)

C2contour (1-d)

  • TheoremFor C2 / C2 images, optimal asymptotic L2 error decay and near-optimal rate-distortion
practical image coder wsfq
Practical Image Coder – WSFQ
  • Wedgelet-SFQ (WSFQ) coder builds on SFQ coder [Xiong, Ramchandran, Orchard]
  • At low bit rates, often significant improvement in visual quality over SFQ and JPEG-2k (sharper edges)
  • Bonus: WSFQ representation contains explicit “cartoon” geometry information
jpeg compressed dct2
JPEG Compressed (DCT)

PSNR = 26.32dB @ 0.0103bpp

jpeg2000 compressed wavelets1
JPEG2000 Compressed (Wavelets)

PSNR = 29.77dB @ 0.0103bpp

sfq compressed wavelets1
SFQ Compressed (Wavelets)

PSNR = 29.77dB @ 0.0103bpp

wsfq compressed
WSFQ Compressed

PSNR = 30.19dB @ 0.0102bpp

wsfq wc code map
WSFQ WC Code Map

green=significantblue=zero tree red=contour tree

slide93

original

SFQ

JPEG2000

WSFQ

gain over jpeg2000 cameraman
Gain Over JPEG2000 – Cameraman

40% MSE improvement

WSFQ

SFQ

gain over jpeg2000 peppers
Gain Over JPEG2000 – Peppers

18% MSE improvement

WSFQ

SFQ

gain over jpeg2000 lena
Gain Over JPEG2000 – Lena

14% MSE improvement

WSFQ

SFQ

extensions
Extensions

S

Z

S

S

W

S

S

S

S

zerotree

wedgeprint

Z

D

S

S

W

S

S

extensions1
Extensions

barprint

DCTprint

S

Z

D

S

W

B

B

B

B

zerotree

wedgeprint

Z

D

“Coifman’s

Dream”

B

B

W

B

B

extensions2
Extensions

barprint

DCTprint

S

Z

D

S

W

V

B

B

B

zerotree

wedgeprint

Z

D

Gastprint

B

S

W

B

B

bars ridges
Bars / Ridges

16x16 image block

multiple wedgelet coding
Multiple Wedgelet Coding

80 bits to jointly encode 5 wedgelets

barlet coding
“Barlet” Coding

(fat edgelet/beamlet)

22 bits to encode 1 barlet

dctprint
DCTprint

DCT

5dB

PSNR

wavelets

# terms

32x32 block = 1024 pixels

dctprint1
DCTprint

DCT

PSNR

wavelets

# terms

32x32 block = 1024 pixels

4x fewer coefficients

conclusions
Conclusions
  • Capturing geometrical information in images requires new tools: vocabulary+grammar

wavelets

basis

conclusions1
Conclusions
  • Capturing geometrical information in images requires new tools: vocabulary+grammar

wavelets

wedgelets

basis

model

conclusions2
Conclusions
  • Capturing geometrical information in images requires new tools: vocabulary+grammar

wavelets

wedgelets

basis

model

wedgeprints

conclusions3
Conclusions
  • Capturing geometrical information in images requires new tools: vocabulary+grammar

wavelets

wedgelets

basis

model

  • wedgelet VQ
  • adaptive wedgeprint atoms

wedgeprints

conclusions4
Conclusions
  • Capturing geometrical information in images requires new tools: vocabulary+grammar

wavelets

wedgelets

basis

model

  • Optimal approximation of C2/C2 function class

wedgeprints

related work
Related Work
  • Platelets, Dyadic Decision Trees, Sensor Nets, …[Nowak, Willett, Castro, Scott, Mitra]
  • Multiresolution Fourier transform[Calway, Pearson, Wilson]
  • “Prune and join” quadtree approximation[Shukla, Dragotti, Do, Vetterli]
  • Wavelet footprints[Dragotti, Vetterli]
  • Bandelets[Le Pennec, Mallat]
future work
Future Work

dsp.rice.edu

  • Build “Coifman’s Dream” coder
  • Beyond uniform quadtree splits
  • Other transforms beyond wavelets
    • complex wavelets
    • curvelets, …
  • Extensions to EQ coder, other image coders
  • Detection, classification, …
conclusions5
Conclusions
  • Capturing geometrical information in images requires new tools

wavelets

wedgelets

basis

model

dsp.rice.edu

wedgeprints

transform coding2
Transform Coding
  • Quantize coefficients of an image expansion

coefficients

basis, frame

Ex: Fourier sinusoids

DCT (JPEG)

wavelets, …

transform domain modeling3
Transform-domain Modeling
  • Transform-domain modeling and processing

transform

coefficient

model

the need for compression
The Need for Compression
  • Modern digital camera megapixels
slide122

Multiscale Geometric Image Processing

Richard Baraniuk

Rice University

dsp.rice.edu

Joint work with

Hyeokho Choi, Justin Romberg, Mike Wakin