Besov bayes chomsky plato
Download
1 / 122

Besov Bayes Chomsky Plato - PowerPoint PPT Presentation


  • 53 Views
  • Uploaded on

Multiscale Geometric Image Processing. Besov Bayes Chomsky Plato. Richard Baraniuk Rice University dsp.rice.edu Joint work with Hyeokho Choi Justin Romberg Mike Wakin. Image Processing. Analyzing, modeling, processing images special class of 2-d functions ex: digital photos

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about ' Besov Bayes Chomsky Plato' - cambria-jovan


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
Besov bayes chomsky plato

Multiscale Geometric Image Processing

Besov BayesChomsky Plato

Richard Baraniuk

Rice University

dsp.rice.edu

Joint work with

Hyeokho ChoiJustin RombergMike Wakin


Image processing
Image Processing

  • Analyzing, modeling, processingimages

    • special class of 2-d functions

    • ex: digital photos

  • Problems:

    • representation, approximation, compression, estimation, restoration, deconvolution, detection, classification, segmentation, …


Low level image structures
Low-Level Image Structures

  • Prevalent image structures:

    • smooth regions - grayscale regularity

    • smooth edge contours - geometric regularity

  • Must exploit both for maximum performance


Lossy image compression
Lossy Image Compression

  • Given image

    approximate using bits

  • Error incurred = distortionExample: squared error


Rate distortion analysis
Rate-Distortion Analysis

as

  • Asymptotic analysis; maximize rate

achievable region


Transform coding
Transform Coding

  • Quantize coefficients of an image expansion

coefficients

basis, frame

Ex: Fourier sinusoids

DCT (JPEG)

wavelets curvelets …


Transform coding1
Transform Coding

  • Quantize coefficients of an image expansion

coefficients

basis, frame

quantize to total bits


Wet paint test image
Wet Paint Test Image


Jpeg compressed dct
JPEG Compressed (DCT)

PSNR = 26.32dB @ 0.0103bpp


Multiscale image processing
MultiscaleImage Processing


Multiscale image analysis
Multiscale Image Analysis

  • Analyze an image a multiple scales

  • How? Zoom out and record information lost in wavelet coefficients

info 1

info 2

info 3


Wavelet based image processing
Wavelet-based Image Processing

  • Standard 2-D tensor product wavelet transform


Transform domain modeling
Transform-domain Modeling

  • Transform-domain modeling and processing

transform

coefficient

model


Transform domain modeling1
Transform-domain Modeling

  • Transform-domain modeling and processing

transformvocabulary

coefficient

model


Transform domain modeling2
Transform-domain Modeling

  • Transform-domain modeling and processing

  • Vocabulary + grammar capture image structure

  • Challenging co-design problem

transformvocabulary

coefficient grammar

model


Nonlinear image modeling
Nonlinear Image Modeling

  • Natural images do not form a linear space!

  • Form of the “set of natural images”?

+

=


Set of natural images
Set of Natural Images

Small: NxN sampled images comprise an extremely small subset of RN2


Set of natural images1
Set of Natural Images

Small: NxN sampled images comprise an extremely small subset of RN2


Set of natural images2
Set of Natural Images

Small: NxN sampled images comprise an extremely small subset of RN2

Complicated: “Manifold” structure

RN2



Wedge manifold1
Wedge Manifold

3 Haar wavelets


Wedge manifold2
Wedge Manifold

3 Haar wavelets


Set of natural images3
Set of Natural Images

Small: NxN sampled images comprise an extremely small subset of RN2

Complicated: “Manifold” structure

RN2


Stochastic projections
Stochastic Projections

  • Projection

    = “shadow”


Higher d stochastic projections
Higher-D Stochastic Projections

  • Model (partial) wavelet coefficient joint statistics

  • Shadow more faithful to manifold



Wavelet based image processing1
Wavelet-based Image Processing

  • Standard 2-D tensor product wavelet transform


Wavelet coefficient sparsity
Wavelet Coefficient Sparsity

  • Most


Wavelet quadtree structure
Wavelet Quadtree Structure

  • Wavelet analysis is self-similar

  • Image parent square divides into 4 children squares at next finer scale


Wavelet quadtree correlations
Wavelet Quadtree Correlations

  • Smooth regionsmallwc’s tumble down tree

  • Edge/ridge regionlarge wc’s tumble down tree


Zero tree compression
Zero Tree Compression

  • Idea: Prune wavelet subtrees in smooth regions

Z

  • tree-structured thresholding

  • spend bits on (large) “significant” wc’s

  • spend no bits on (small) wc’s…

  • zerotree symbolZ = {wc and all decendants = 0}


Zero tree compression1
Zero Tree Compression

  • Label pruned wavelet quadtree with 2 stateszero-tree- smooth region (prune)significant- edge/texture region (keep)

S

smooth

smooth

Z

Z

S

not

smooth

Z

smooth

S

S

S

S


Zero tree compression2
Zero Tree Compression

  • Label pruned wavelet quadtree with 2 stateszero-tree- smooth region (prune)significant- edge/texture region (keep)

  • Can optimize placement of Z and S with dynamic programming (SFQ) [Xiong, Ramchandran, Orchard]

S

smooth

smooth

Z

Z

S

not

smooth

Z

smooth

S

S

S

S


Wet paint test image1
Wet Paint Test Image


Jpeg compressed dct1
JPEG Compressed (DCT)

PSNR = 26.32dB @ 0.0103bpp


Jpeg2000 compressed wavelets
JPEG2000 Compressed (Wavelets)

PSNR = 29.77dB @ 0.0103bpp


Sfq compressed wavelets
SFQ Compressed (Wavelets)

PSNR = 29.77dB @ 0.0103bpp


Sfq wc code map
SFQ WC Code Map

green = significantblue = zero tree



The jpeg2000 disappointment
The JPEG2000 Disappointment

  • JPEG2000: replace DCT with wavelet transform

  • Does not improve on decay rate of JPEG!

  • WHY?neither DCT norwavelets are theright transform

JPEG (DCT)

JPEG2000 (wavelets)


Wavelet Challenges

  • Geometrical info not explicit

  • Inefficient-large number of large wc’s cluster around edge contours, no matter how smooth


Wavelets and cartoons
Wavelets and Cartoons

13

26

52

  • Even for a smooth C2 contour, which straightens at fine scales…


2 d wavelets poor approximation
2-D Wavelets: Poor Approximation

13

26

52

  • Even for a smooth C2 contour, which straightens at fine scales…

  • Too many wavelets required!

-term wavelet approximation

not


2 d wavelets poor approximation1
2-D Wavelets: Poor Approximation

13

26

52

  • Even for a smooth C2 contour, which straightens at fine scales…

  • Too many wavelets required!

-bit wavelet approximation

not


Solution 1 upgrade the transform
Solution 1: Upgrade the Transform

13

26

52

  • Introduce anisotropic transform

    • curvelets, ridgelets, contourlets, …

  • Optimal error decay rates for some simple images


Solution 2 upgrade the processing
Solution 2: Upgrade the Processing

13

26

52

  • Replace coefficient thresholding by a new wc modelthat captures anisotropicspatial correlations


Our goal new image models
Our Goal: New Image Models

  • Wavelet coefficient models that capture both geometric and textural aspects of natural images

  • Optimal error decay rates for some image class



Geometry model for cartoons
Geometry Model for Cartoons

C2 boundary

  • Toy model: flat regions separated by smooth contours

  • Goal: representation that is

    • sparse

    • simply modeled

    • efficiently computed

    • extensible to texture regions


2 d dyadic partition
2-D Dyadic Partition

  • Multiscale analysis

  • Partition; not a basis/frame

  • Zoom in by factor of 2 each scale


2 d dyadic partition quadtree
2-D Dyadic Partition = Quadtree

  • Multiscaleanalysis

  • Partition; not a basis/frame

  • Zoom in by factor of 2 each scale

  • Each parent node has 4 children at next finer scale


Wedgelet representation donoho
Wedgelet Representation [Donoho]

r

  • Build a cartoon using wedgelets on dyadic squares

    wedgelet = atomic geometric element




Wedgelet representation
Wedgelet Representation

r

  • Build a cartoon using wedgelets on dyadic squares

  • Choose orientation (r,q) from finite dictionary (toroidal sampling)

  • Quad-tree structuredeeper in tree finer curve approximation


Wedgelet representation1
Wedgelet Representation

  • Prune wedgelet quadtree to approximate local geometry (adaptive)

  • Decorate leaves with (r,q) parameters


Wedgelet inference
Wedgelet Inference

  • Find representation / prune tree to balance a fidelity vs. complexity trade-off

  • For Comp(W) –need a model for the wedgelet representation(quadtree + (r,q))

  • Donoho: Comp(W) = #leaves


Wedgelet inference1
Wedgelet Inference

  • Find representation / prune tree to balance a fidelity vs. complexity trade-off

  • O(N) dynamic programmingsolution (bottom-up recursion)

  • Optimal approximation

  • Near-optimal rate/distortion decay


Leaves complexity penalty
#Leaves Complexity Penalty

  • Accounts for wedgelet partition size, but not wedgelet orientation

#leaves

#leaves

“smooth”

“simple”

“rough”

“complex”


Multiscale geometry model mgm
Multiscale Geometry Model (MGM)

  • Decorate each tree node with orientation (r,q) and then model dependencies thru scale

  • Insight: Smooth curve Geometric innovations small at fine scales

  • Model: Favor small innovations over large innovations (statistically)


Multiscale geometry model mgm1
Multiscale Geometry Model (MGM)

  • Wavelet-like geometry model: coarse-to-fine prediction

    • model parent-to-child transitions of orientations

small

innovations

large


MGM

  • Wavelet-like geometry model: coarse-to-fine prediction

    • model parent-to-child transitions of orientations

  • Markov-1 statistical model

    • state = (r,q) orientation of wedgelet

    • parent-to-child state transition matrix


MGM

  • Markov-1 statistical model

  • Joint wedgelet Markov probability model:

  • Complexity = Shannon codelength == number of bits to encode


Mgm and edge smoothness
MGM and Edge Smoothness

“smooth”

“simple”

“rough”

“complex”


Mgm inference
MGM Inference

  • Find representation / prune tree to balance the fidelity vs. complexity trade-off

  • Efficient O(N) solution via dynamic programming


Mgm approximation
MGM Approximation

  • Find representation / prune tree to balance the fidelity vs. complexity trade-off

  • OptimalL2 error decay rate for cartoons

single scale

multiscale


Wedgelet coding of cartoon images
Wedgelet Coding of Cartoon Images

  • Choosing wedgelets = rate-distortion optimization

Shannon code length

to encode


Wedgelet coding of cartoon images1
Wedgelet Coding of Cartoon Images

  • Choosing wedgelets = rate-distortion optimization

~130 bits coded independently

~25 bits coded with MGM


Predictive wedgelet coding is optimal
Predictive Wedgelet Coding is Optimal

C2contour (1-d)

Optimal rate-distortion performance

compared to

for Donoho’s leaf-only encoding



Joint texture geometry compression
Joint Texture/Geometry Compression

  • Dictionary D = {wavelets} U {wedgelets}

  • Representation tradeoff: texture vs.geometry


Recall zero tree compression
Recall – Zero Tree Compression

  • Idea: Prune wavelet subtrees in smooth regions

Z

  • tree-structured thresholding

  • spend bits on (large) “significant” wc’s

  • spend no bits on (small) wc’s…

  • zerotree symbolZ = {wc and all decendants = 0}


Zero tree compression3
Zero Tree Compression

  • Label pruned wavelet quadtree with 2 stateszero-tree- smooth region (prune)significant- edge/texture region (keep)

S

smooth

smooth

Z

Z

S

not

smooth

Z

smooth

S

S

Z: all wc’s below=0

S

S


Enter geometry
Enter Geometry

  • Label pruned wavelet quadtree with 2 stateszero-tree- smooth region (prune)significant- edge/texture region (keep)

  • Suboptimal NLA/R-D decay w/ edges

S

smooth

smooth

Z

Z

S

S

texture

geometry

S

S

S

S

Z

S

S

S


Wedgelets for geometry
Wedgelets for Geometry

  • Label pruned wavelet quadtree with 2 stateszero-tree- smooth region (prune)significant- edge/texture region (keep)

  • Idea: use wedgelets in geometric edge squares

S

smooth

smooth

Z

Z

S

G

texture

geometry

G: wc’s below are wc’s of a wedgelet

S

S

S

S


Wedgelets for geometry1
Wedgelets for Geometry

  • Label pruned wavelet quadtree with 3 stateszero-tree- smooth region (prune)geometry- edge region (prune)significant- texture region (keep)

S

smooth

smooth

Z

Z

S

G

texture

geometry

G: wc’s below are wc’s of a wedgelet

S

S

S

S


Wedgelets for geometry2
Wedgelets for Geometry

  • Label pruned wavelet quadtree with 3 stateszero-tree- smooth region (prune)geometry- edge region (prune)significant- texture region (keep)

  • Optimize placement of Z, G, S by dyn. programming

S

smooth

smooth

Z

Z

S

G

texture

geometry

G: wc’s below are wc’s of a wedgelet

S

S

S

S


Wedgelet trees for geometry
Wedgelet Trees for Geometry

  • Label pruned wavelet quadtree with 3 stateszero-tree- smooth region (prune)geometry- edge region (prune)significant- texture region (keep)

  • Optimize placement of Z, G, S by dyn. programming

S

smooth

smooth

Z

Z

S

G

texture

G: wc’s below are wc’s of a wedgelet tree

S

S

geometry

S

S


Image approximation
Image Approximation

S

Z

Z

S

G

S

S

S

S

Z

Z

S

S

S: describe (code) wc’s

G

S

S


Image approximation1
Image Approximation

S

Z

Z

S

G

S

S

S

S

Z: all wc’s below=0

ie: wc’s of

Z

Z

S

S

S: describe (code) wc’s

G

S

S


Image approximation2
Image Approximation

S

Z

Z

S

G

S

S

S

S

Z: all wc’s below=0

ie: wc’s of

G: wc’s below are wc’s of a wedgelet ie: wc’s of

“wedgeprint”

Z

Z

S

S

S: describe (code) wc’s

G

S

S


Optimality of wedgeprints
Optimality of Wedgeprints

C2texture (2-d)

C2contour (1-d)

  • TheoremFor C2 / C2 images, optimal asymptotic L2 error decay and near-optimal rate-distortion


Practical image coder wsfq
Practical Image Coder – WSFQ

  • Wedgelet-SFQ (WSFQ) coder builds on SFQ coder [Xiong, Ramchandran, Orchard]

  • At low bit rates, often significant improvement in visual quality over SFQ and JPEG-2k (sharper edges)

  • Bonus: WSFQ representation contains explicit “cartoon” geometry information


Wet paint test image2
Wet Paint Test Image


Jpeg compressed dct2
JPEG Compressed (DCT)

PSNR = 26.32dB @ 0.0103bpp


Jpeg2000 compressed wavelets1
JPEG2000 Compressed (Wavelets)

PSNR = 29.77dB @ 0.0103bpp


Sfq compressed wavelets1
SFQ Compressed (Wavelets)

PSNR = 29.77dB @ 0.0103bpp


Wsfq compressed
WSFQ Compressed

PSNR = 30.19dB @ 0.0102bpp


Wsfq wc code map
WSFQ WC Code Map

green=significantblue=zero tree red=contour tree



original

SFQ

JPEG2000

WSFQ


Gain over jpeg2000 cameraman
Gain Over JPEG2000 – Cameraman

40% MSE improvement

WSFQ

SFQ


Gain over jpeg2000 peppers
Gain Over JPEG2000 – Peppers

18% MSE improvement

WSFQ

SFQ


Gain over jpeg2000 lena
Gain Over JPEG2000 – Lena

14% MSE improvement

WSFQ

SFQ


Extensions
Extensions

S

Z

S

S

W

S

S

S

S

zerotree

wedgeprint

Z

D

S

S

W

S

S


Extensions1
Extensions

barprint

DCTprint

S

Z

D

S

W

B

B

B

B

zerotree

wedgeprint

Z

D

“Coifman’s

Dream”

B

B

W

B

B


Extensions2
Extensions

barprint

DCTprint

S

Z

D

S

W

V

B

B

B

zerotree

wedgeprint

Z

D

Gastprint

B

S

W

B

B


Bars ridges
Bars / Ridges

16x16 image block


Multiple wedgelet coding
Multiple Wedgelet Coding

80 bits to jointly encode 5 wedgelets


Barlet coding
“Barlet” Coding

(fat edgelet/beamlet)

22 bits to encode 1 barlet



Dctprint
DCTprint

DCT

5dB

PSNR

wavelets

# terms

32x32 block = 1024 pixels


Dctprint1
DCTprint

DCT

PSNR

wavelets

# terms

32x32 block = 1024 pixels

4x fewer coefficients


Conclusions
Conclusions

  • Capturing geometrical information in images requires new tools: vocabulary+grammar

wavelets

basis


Conclusions1
Conclusions

  • Capturing geometrical information in images requires new tools: vocabulary+grammar

wavelets

wedgelets

basis

model


Conclusions2
Conclusions

  • Capturing geometrical information in images requires new tools: vocabulary+grammar

wavelets

wedgelets

basis

model

wedgeprints


Conclusions3
Conclusions

  • Capturing geometrical information in images requires new tools: vocabulary+grammar

wavelets

wedgelets

basis

model

  • wedgelet VQ

  • adaptive wedgeprint atoms

wedgeprints


Conclusions4
Conclusions

  • Capturing geometrical information in images requires new tools: vocabulary+grammar

wavelets

wedgelets

basis

model

  • Optimal approximation of C2/C2 function class

wedgeprints


Related work
Related Work

  • Platelets, Dyadic Decision Trees, Sensor Nets, …[Nowak, Willett, Castro, Scott, Mitra]

  • Multiresolution Fourier transform[Calway, Pearson, Wilson]

  • “Prune and join” quadtree approximation[Shukla, Dragotti, Do, Vetterli]

  • Wavelet footprints[Dragotti, Vetterli]

  • Bandelets[Le Pennec, Mallat]


Future work
Future Work

dsp.rice.edu

  • Build “Coifman’s Dream” coder

  • Beyond uniform quadtree splits

  • Other transforms beyond wavelets

    • complex wavelets

    • curvelets, …

  • Extensions to EQ coder, other image coders

  • Detection, classification, …


Conclusions5
Conclusions

  • Capturing geometrical information in images requires new tools

wavelets

wedgelets

basis

model

dsp.rice.edu

wedgeprints


Transform coding2
Transform Coding

  • Quantize coefficients of an image expansion

coefficients

basis, frame

Ex: Fourier sinusoids

DCT (JPEG)

wavelets, …


Transform domain modeling3
Transform-domain Modeling

  • Transform-domain modeling and processing

transform

coefficient

model


The need for compression
The Need for Compression

  • Modern digital camera megapixels


Multiscale Geometric Image Processing

Richard Baraniuk

Rice University

dsp.rice.edu

Joint work with

Hyeokho Choi, Justin Romberg, Mike Wakin


ad