Introduction to Estimation Theory: A Tutorial

1 / 40

# Introduction to Estimation Theory: A Tutorial - PowerPoint PPT Presentation

Introduction to Estimation Theory: A Tutorial. Volkan Cevher. Outline. Introduction Terminology and Preliminaries Bayesian (Random) Parameter Estimation Nonrandom Parameter Estimation Questions. Introduction. Classical detection problem:

I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.

## Introduction to Estimation Theory: A Tutorial

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript

### Introduction to Estimation Theory: A Tutorial

Volkan Cevher

Outline
• Introduction
• Terminology and Preliminaries
• Bayesian (Random) Parameter Estimation
• Nonrandom Parameter Estimation
• Questions

Georgia Institute of Technology Center for Signal and Image Processing

Introduction
• Classical detection problem:
• Design of optimum procedures for deciding between possible statistical situations given a random observation:
• The model has the following components:
• Parameter Space (for parametric detection problems)
• Probabilistic Mapping from Parameter Space to Observation Space
• Observation Space
• Detection Rule

Georgia Institute of Technology Center for Signal and Image Processing

Introduction
• Parameter Space:
• Completely characterizes the output given the mapping.
• Each hypothesis corresponds to a point in the parameter space. This mapping is one-to-one.
• Probabilistic Mapping from Parameter Space to Observation Space:
• The probability law that governs the effect of a parameter on the observation.

Example 1:

Probabilistic mapping

Parameter Space

Georgia Institute of Technology Center for Signal and Image Processing

Introduction
• Observation Space:
• Finite dimensional, i.e. Yn, where n is finite.
• Detection Rule
• Mapping of the observation space into its parameters in the parameter space is called a detection rule.

Georgia Institute of Technology Center for Signal and Image Processing

Introduction
• Classical estimation problem:
• Interested in not making a choice among several discrete situations, but rather making a choice among a continuum of possible states.
• Think of a family of distributions on the observation space, indexed by a set of parameters.
• Given the observation, determine as accurately as possible the actual value of the parameter.
• In this example, given the observations, parameter  is being estimated. Its value is not chosen among a set of discrete values, but rather is estimated as accurately as possible.

Example 2:

Georgia Institute of Technology Center for Signal and Image Processing

Introduction
• Estimation problem also has the same components as the detection problem.
• Parameter Space
• Probabilistic Mapping from Parameter Space to Observation Space
• Observation Space
• Estimation Rule
• Detection problem can be thought of as a special case of the estimation problem.
• There are a variety of estimation procedures differing basically in the amount of prior information about the parameter and in the performance criteria applied.
• Estimation theory is less structured than detection theory. “Detection is science, estimation is art.” Array Signal Processing by Johnson, Dudgeon.

Georgia Institute of Technology Center for Signal and Image Processing

Introduction
• Based on the a priori information about the parameter, there are two basic approaches to parameter estimation:
• Bayesian Parameter Estimation
• Nonrandom Parameter Estimation
• Bayesian Parameter Estimation:
• Parameter is assumed to be a random quantity related statistically to the observation.
• Nonrandom Parameter Estimation:
• Parameter is a constant without any probabilistic structure.

Georgia Institute of Technology Center for Signal and Image Processing

Terminology and Preliminaries
• Estimation theory relies on jargon to characterize the properties of estimators. In this presentation, the following definitions are used:
• The set of n observations are represented by the n-dimensional vector y (observation space).
• The values of the parameters are denoted by the vector  (parameter space).
• The estimate of this parameter vector is denoted by : .

Georgia Institute of Technology Center for Signal and Image Processing

Terminology and Preliminaries
• Definitions (continued):
• The estimation error (y)( in short) is defined by the difference between the estimate and the actual parameter:
• The function C[a,]: + is the cost of estimating a true value of  as a.
• Given such a cost function C, the Bayes risk (average risk) of the estimator is defined by the following:

Georgia Institute of Technology Center for Signal and Image Processing

Terminology and Preliminaries

Example 3:

• Suppose we would like to minimize the Bayes risk defined by

for a given cost function C.

By inspection, one can see that the Bayes estimate of  can be found (if it exists) by minimizing, for each y, the posterior cost given Y=y:

Georgia Institute of Technology Center for Signal and Image Processing

Terminology and Preliminaries
• Definitions (continued):
• An estimate is said to be unbiased if the expected value of the estimate equals the true value of the parameter

. Otherwise the estimate is said to be biased. The bias b() is usually considered to be additive, so that:

• An estimate is said to be asymptotically unbiased if the bias tends to zero as the number of observations tend to infinity.
• An estimate is said to be consistent if the mean-squared estimation error tends to zero as the number of observations becomes large.

Georgia Institute of Technology Center for Signal and Image Processing

Terminology and Preliminaries
• Definitions (continued):
• An efficient estimate has a mean-squared error that equals a particular lower bound: the Cramer-Rao bound. If an efficient estimate exists, it is optimum in the mean-squared sense: No other estimate has a smaller mean-squared error.
• Following shorthand notations will also be used for brevity:

Georgia Institute of Technology Center for Signal and Image Processing

Terminology and Preliminaries
• Following definitions and theorems will be useful later in the presentation:
• Definition: Sufficiency

Suppose that  is an arbitrary set. A function T:  is said to be a sufficient statistic for the parameter set  if the distribution of y conditioned on T(y) does not depend on  for .

If knowing T(y) removes any further dependence on  of the distribution of y, one can conclude that T(y) contains all the information in y that is useful for estimating . Hence, it is sufficient.

Georgia Institute of Technology Center for Signal and Image Processing

Terminology and Preliminaries
• Definition: Minimal Sufficiency

A function T on  is said to be minimal sufficient for the parameter set  if it is a function of every other sufficient statistic for .

A minimal sufficient statistic represents the furthest reduction in the observation without destroying information about .

Minimal sufficient statistic does not necessarily exist for every problem. Even if it exists, it is usually very difficult to identify it.

Georgia Institute of Technology Center for Signal and Image Processing

Terminology and Preliminaries
• The Factorization Theorem:

Suppose that the parameter set  has a corresponding families of densities p. A statistic T is sufficient for  iff there are functions g and h such that

for all y and .

Refer to the supplement for a proof.

Georgia Institute of Technology Center for Signal and Image Processing

Terminology and Preliminaries

Example 4:

(Poor) Consider the hypothesis-testing problem ={0,1} with densities p0 and p1. Noting that

the factorization is possible with

Thus the likelihood ratio L is a sufficient statistic for the binary hypothesis-testing problem.

Georgia Institute of Technology Center for Signal and Image Processing

Terminology and Preliminaries
• The Rao-Blackwell Theorem:

Suppose that ĝ(y) is an unbiased estimate of g() and that T is sufficient for . Define

Then is also an unbiased estimate of g(). Furthermore,

with equality iff

Refer to the supplement for a proof.

Georgia Institute of Technology Center for Signal and Image Processing

Terminology and Preliminaries
• Definition: Completeness

The parameter family  is said to be complete if the condition E{f(Y)}=0 for all  implies that P(f(Y)=0)=1 for all .

(Poor) Suppose that ={0,1,…,n}, ={0,1}, and

For any function f on , we have

The condition E{f(Y)}=0 for all  implies that

However, an nth order polynomial has at most n zeros unless all of its coefficients are zero. Hence,  is complete.

Example 5:

Georgia Institute of Technology Center for Signal and Image Processing

Terminology and Preliminaries
• Definition: Exponential Families
• A class of distributions with parameter set  is said to be an exponential family if there are real-valued functions C,Q1,…,Qm,T1,…,Tm, and h such that
• T(y)=[T1(y),…,Tm(y)]T is a complete sufficient statistic.

Georgia Institute of Technology Center for Signal and Image Processing

Bayesian Parameter Estimation
• For the random observation Y, indexed by a parameter m, our goal is to find a function such that is the best guess of the true value of  given Y=y.
• Bayesian estimators are the estimators that minimize the Bayesian risk function.
• The following estimators are commonly used in practice and can be distinguished by their cost functions.

Georgia Institute of Technology Center for Signal and Image Processing

Bayesian Parameter Estimation
• Minimum-Mean-Squared-Error (MMSE):
• Euclidian Cost function:
• The posterior cost given Y=y is given by
• Minimizing this cost function also minimizes the Bayes risk . Hence, on differentiating with respect to , one can obtain the Bayes estimate

Georgia Institute of Technology Center for Signal and Image Processing

Bayesian Parameter Estimation
• Minimum-Mean-Absolute-Error (MMAE):
• Absolute Error Cost function:
• The posterior cost given Y=y is given by
• Here we used the fact that with P(X0)=1, then

MMAE 1of3

Georgia Institute of Technology Center for Signal and Image Processing

Bayesian Parameter Estimation
• Further simplification is also possible:

MMAE 2of3

Georgia Institute of Technology Center for Signal and Image Processing

Bayesian Parameter Estimation
• Taking the derivative with respect to each , one can see that

This derivative is a nondecreasing function of

that approaches –1 as and +1 as . Thus achieves its minimum where its derivative changes sign:

MMAE 3of3

Georgia Institute of Technology Center for Signal and Image Processing

Bayesian Parameter Estimation
• Maximum A Posteriori Probability (MAP):
• Uniform Error Cost function:
• The posterior cost given Y=y is given by
• Within some smoothness conditions, the estimator that maximizes this cost function is given by

Georgia Institute of Technology Center for Signal and Image Processing

Bayesian Parameter Estimation
• Observations:
• MMSE Estimator:
• The MMSE estimate of  given Y=y is the conditional mean of given Y=y .
• MMAE Estimator:
• The MMAE estimate of  given Y=y is the conditional median of given Y=y .
• MAP Estimator:
• The MMAE estimate of  given Y=y is the conditional mode of given Y=y .

Georgia Institute of Technology Center for Signal and Image Processing

Bayesian Parameter Estimation

Example 6:

• (Poor) Given the following conditional probability density function

hence y has an exponential density with parameter . Suppose  is also exponential random variable with density

Then, the posterior distribution of  given Y=y is given by

for 0 and y0, and w(|y)=0 otherwise.

Georgia Institute of Technology Center for Signal and Image Processing

Bayesian Parameter Estimation

Example 7:

• (Continued.)
• The MMSE is the mean of this distribution:
• The MMAE is the median of this distribution:
• The MAP estimate is the mode of this distribution (where it is maximum):
• To decide which one to use, one must decide which three of the cost functions best suits the problem at hand.

Georgia Institute of Technology Center for Signal and Image Processing

Nonrandom Parameter Estimation
• Our goal is the same in Bayesian parameter estimation problem. Find .
• Assume that the parameter set  is real valued. In the nonrandom parameter estimation problem, we do not know anything about the true value of  other than the fact that it lies in . Hence, given the observation Y=y, what is the best estimate of  is the question we would like to answer.

Georgia Institute of Technology Center for Signal and Image Processing

Nonrandom Parameter Estimation
• The only average performance cost that can be done is with respect to the distribution of Y given , given a cost function C.
• A reasonable restriction to place on an estimate of  is that its expected value is equal to the true parameter value:
• For its tractability, the Euclidian norm squared cost function will be used.

Georgia Institute of Technology Center for Signal and Image Processing

Nonrandom Parameter Estimation
• When the squared-error cost is used, the risk function is the following:
• One can not generally expect to minimize this risk function uniformly for all . This is easily seen for the squared error cost since for any particular value of , say 0 the conditional mean-squared error can be made zero by choosing the estimate to be identically 0 for all observations y.
• However, if  is not close to 0, such an estimate would perform poorly.

Georgia Institute of Technology Center for Signal and Image Processing

Nonrandom Parameter Estimation
• With the unbiased-ness restriction, the conditional mean-squared error becomes the variance of the estimate. Hence, these estimators are termed minimum-variance unbiased estimators (MVUEs).
• The procedure for seeking MVUEs:
• Find a complete sufficient statistics T for .
• Find any unbiased estimator ĝ(y) of g().
• Then, is an MVUE of g().

Georgia Institute of Technology Center for Signal and Image Processing

Nonrandom Parameter Estimation

Example 8:

• (Poor) Consider the model

where N1,…,Nn are i.i.d. N(0,2) noise samples, and sk is a known signal for k=1,…,n. Our objective is to estimate  and 2.

1. The density of Y is given by

where =[ 1 2 ]T and

Georgia Institute of Technology Center for Signal and Image Processing

Nonrandom Parameter Estimation

Example 9:

• (Continued.) Note that T= [ T1 T2 ]T is a complete

sufficient statistic for .

2.We wish to estimate

Assuming that s10, the estimate ĝ1(y)=y1/s1 is an unbiased estimator of g1().

Moreover, note that

and that

Hence, is an unbiased estimate of g2().

Georgia Institute of Technology Center for Signal and Image Processing

Nonrandom Parameter Estimation

Example 10:

• (Continued.)

3. Since T1 and T2 are complete, the estimates

are MVUEs of . Note that ĝ1(y) and T1 (y) are both linear functions of Y and are jointly Gaussian. Hence, MVUEs are

Georgia Institute of Technology Center for Signal and Image Processing

Nonrandom Parameter Estimation
• Maximum-Likelihood (ML) Estimation:
• For many problems arising in practice, it is not usually feasible to find MVUEs.
• Another method for seeking good estimators are needed.
• ML is one of the most commonly used methods in signal processing literature.
• Consider MAP estimation for :
• In the absence of any prior information about the parameter , we can assume that it is uniformly distributed (w() becomes a uniform distribution) since this represents the worst case scenario.

Georgia Institute of Technology Center for Signal and Image Processing

Nonrandom Parameter Estimation
• ML Estimation: (Continued.)
• Hence, the MAP estimate for a given y is any value of  that maximizes p(y) over .
• p(y) is usually called the likelihood ratio.
• Hence, the ML estimate is
• Maximizing p(y) is the same as maximizing log p(y) (log-likelihood function). Therefore, a necessary condition for the maximum-likelihood estimate is
• The above condition is also known as the likelihood equation.

Georgia Institute of Technology Center for Signal and Image Processing

Nonrandom Parameter Estimation
• Cramer-Rao Bound:
• Let be some unbiased estimator of  Then the error covariance matrix is bounded by the Cramer-Rao bound (refer to the supplement).
• If the Cramer-Rao bound can be satisfied with equality, only the maximum likelihood estimate achieves it. Hence, if an efficient estimate exists, it is the maximum likelihood estimate.

refer to the attached paper: “The Stochastic CRB for Array Processing: A Textbook Derivation” by Stoica, Larsson, and Gershman.

Example 11:

Georgia Institute of Technology Center for Signal and Image Processing

Questions

Georgia Institute of Technology Center for Signal and Image Processing