Loading in 2 Seconds...
Loading in 2 Seconds...
Markov Random FieldBased EdgeCentric Image/Video Processing. Min Li Advisor: Prof. Truong Nguyen 08/17/07. Outline. Part I: Motivations Contributions Part II: Concepts of MRF models Previous work Applications in image/video processing Used constraints Our work
Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.
Min Li
Advisor: Prof. Truong Nguyen
08/17/07
Requirements:
Methods:
Challenges:
[Li, 2001] X. Li and M. T. Orchard, ``New edgedirected interpolation”, IEEE Trans. on
Image Processing, vol.10,no.10, pp. 1521—1526,2001.
Natural edges
Explicit edge detectors:
Ideal step edges
works with ideal step edges.
has difficulties with natural edges.
(e.g. edge position and thickness,
corners and crossings)
MRF concepts &
applications
in image processing
Formulation of the
2D DAS constraint
U(ω)
MAP
The application in
deinterlacing &
spatial interpolation
Implementation
&
Simulation results
In MRF model, an image is regarded as a 2D random field on a 2D lattice.
Furthermore, in this random field, there are
[Stan Z. Li, 2001] S.Z. Li, Markov Random Field Modeling in Image Analysis,
SpringerVerlag,2001.
[Geman&Geman, 1984] S. Geman and D. Geman,``Stochastic Relaxation, Gibbs
distribution, and the Bayesian restoration of images”,vol.6,no.6,pp. 721—741,Nov. 1984.
[Malfait&Roose, 1997] M. Malfait, et al., ``Waveletbased image denoising using a Markov Random Field a priori model”, IEEE Trans. on Image Processing,vol.6,no.4,
pp.549565,Apr.,1997.
[Zhu,1998] S.C. Zhu, ``Filters, random fields and maximum entropy (FRAME): towards a
unified theory for texture modeling”, International Journal of Computer Vision,vol.27,no.2,
pp. 107—126, 1998.
[Xia, 2006]Y. Xia, et al.,``Adaptive segmentation of textured images by using the coupled Markov Random Field model”, IEEE Trans. on Image Proc., vol.15,no.11, pp. 3559—3566, Nov., 2006
Wavelet
decomposition
Coefficient
modification
Inverse
wavelet
transform
clean
image
noisy
image
Decision map
based on noise variance and
magnitude of
.
MRF helps with the decision map and coefficient updating rule.
where
, and
are the features (histograms) extracted by filter
is a vector representing the potential functions.
PRO: Being able to capture local structures that are large than three or four pixels.
Limitations:
Much more expensive than cliques;
Challenging to choose the number and the size of filters;
Limited to homogeneous texture modeling.
energy
where is the featurerelated energy and is the labelingrelated energy.
[Rue & Held, 2005] H. Rue and L. Held, ``Gaussian Markov Random Fields : Theory and Applications”, Chapman & Hall/CRC, Taylor & Francis Group, 2005.
[Stan Z. Li,1995] S. Z. Li,``On discontinuityadaptive smoothness priors in computer vision”,
IEEE Trans. on Pattern and Machine Intelligence vol.17,no.6, pp.576—586,June 1995.
The joint probability of multivariate Gaussian distribution is a Gibb’s distribution
Where B is the interaction matrix (reverse of the covariance matrix) and w is a vectorized configuration. The corresponding energy is
to be expressed in terms of potential functions,
,
where
and
If identical stationary assumption is made, the energy function is a pure
smoothness constraint.
A line process example
The local energy term is expressed as two terms, one is the energy of
the pixel process (F) and the other is the energy of the line process (L).
Adaptive Potential Functions
g(η) is designed through the
design of the Adaptive Interaction
Function h(η), where
g’(η)=2 η h(η)
In order to be adaptive to discontinuity,
Graphic demonstration of four DAS constraints
Choose
window W
Calculate
PIVs
Derive
weights
Flow diagram
Take one direction (k, q) as an example to show the calculation
1) Window W is of adaptive size
2) PIVs calculation
3) Weights:
,
or
magnitude
Weights of the central
Pixel is calculated
row
col.
Edge pixel
Weights in sixteen
discrete directions
MRF concepts &
applications
in image processing
Formulation of the
2D DAS constraint
U(ω)
MAP
The application in
deinterlacing &
spatial interpolation
Implementation
&
Simulation results
T can be constant (in Metropolis method [Metropolis, 1953]) or gradually decreases (in Simulated Annealing method [Geman&Geman]). One updating
equation of T can be
[Metropolis, 1953] N. Metropolis, et al., ``Equation of state calculations by fast
computing machines, J. Chem. Phys.,vol. 21, pp. 1087—1092, 1953.
B Candidate set propose
(based on pixels only available
in the low resolution image)
A Interpolation initialization
(bilinear, spline, etc.)
Pixel from low res. image
7x7 local window
(16 discrete directions)
Pixel to be interpolated
Example pixel
D Iteratively,
I. New candidate propose
II. Local energy minimization
C Weighted local energy calculation
I. Formulation of DA Smoothness spatial constraint;
II. Weights indicate continuity strength
in each discrete directions
p(ω1)=exp{U(ω1)/T}/Z and p(ω2)=exp{U(ω2)/T}/Z,
then,
p(ω1)/p(ω2)= exp {(U(ω1)U(ω2))/T} = exp{ ΔU/T}.
probability Pc=min(1, p(ω1)/p(ω2)).
To calculateΔU, ΔU=E1(i, j)E2(i, j)
[Freeman, 2002] W. T. Freeman, et al., ``Examplebased superresolution”, IEEE Transaction on Computer Graphics and Application, vol. 22, no. 2, pp. 56—65,
Apr., 2002.
Conclusions:
Future Work:
Gaussian process
Where p is defined as
minimal energy state Emin .
For an M x N image, if each pixel site has L possible values, the state space S is of size LMxN .
i.e., [best spatial interpolation result, motion adaptive result, various motion compensated candidates]
Input
Video
Spatial layer 2
4CIF
Decimation
CIF
Spatial layer 1
H0(Z)
Decimation
2
Spatial layer 0
QCIF
Spatial Scalability
Pred. frame
R0
Error frame
E0
Current frame
C0
Encoder
Layer 0
=
E0
H0(z)(↓2)
E
E1
R1
C1
+
+
=

H0(z)(↓2)
(↑2)F0(z)
H0(z)(↓2)
(↑2)F0(z)
E
+
+
E1
R1
(↑2)F0(z)
E1
c1
Layer 1

=
E0
+
+
+
R0
H0(z)(↓2)

(↑2)F0(z)
Condition
+
C1
X
X
H0(z)(↓2)
(↑2)F0(z)
LowBand Correction(1) Daubechies length9 filter is used as a prototype lowpass filter
(2) Two of the four zeros at of the prototype filter are
retained while the other two zeros are moved along the unit
circle towards /2 and /2
(3) Cost function:
=0 (Es{H0(j)})+1 (1/Hr {H0(z)})+(1 0 1)(1/Hr{ F0(z)}),
is minimized over , where 0< 0, 1<1 and 0+1<1
Es{}: stopband energy;
Hr{}: holder regularity measure
(4) The lowpass filter h0(n) and f0(n) are related via the halfband condition
(as), (bs), (cs),
(ds), (es) and (fs)
The order6 scaling
function impulse
response
Magnitude responses of (a) Daubechies (9,7) filter, (b) MPEG filter
(c), (d), (e), and (f) Newly designed filter pairs
Current frame
Motion vector field
Reference frame
Level 0: L0
L0
mv0
FSME
L1
L1
mv1
FSME
FSME
L2
L2
mv2
FSME
mv3
L3
L3
Large search ranges at level L1 and L0 results high calculation complexity
: Full Search Motion Estimation
Current frame
Motion vector field
Reference frame
Level 0: L0
L0
mv0
L1
L1
mv1
L2
L2
mv2
FSME
L3
L3
mv3
Calculation complexity low and transmit mv3 only
Coding of mode maps:
1 bit to indicate WS1, 2 bits to indicate repeat method and 3 bits to represent WS2 and smooth methods.
At layer 1
At layer 0
Video
FGS Coding
MCTF
prediction
Decimation
FGS Coding
MCTF
prediction
Decimation
FGS Coding
UMCTF
System Diagram of H.264based SVCScalable
Bit stream
MCTF:
Motion Compensated
Temporal Filtering
Block Coding
Multiplex
FGS:
Fine Granular
SNR Scalability
Block Coding
Block Coding
Base layer is H.264/AVC compatible
Flexible combined scalability
The energy function is defined as U(f,l)=U(fl) +U(l)
If there is edge element in between two pixels r and s, the potential function
that is contributed to term U(fl) is zero.
Features
Energy levels
Low
Energy levels
Pixel intensity
variations
High
Energy levels
Artifacts
Artifacts can then be
suppressed via energy
minimization
(a) High
(b) Low
Formulation of 2D DAS