1 / 30

Maximum-Likelihood estimation

Maximum-Likelihood estimation. Consider as usual a random sample x = x 1 , … , x n from a distribution with p.d.f. f ( x ;  ) (and c.d.f. F ( x ;  ) )

kelda
Download Presentation

Maximum-Likelihood estimation

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. Maximum-Likelihood estimation Consider as usual a random sample x = x1, … , xnfrom a distribution with p.d.f. f (x;  ) (and c.d.f. F(x;  ) ) The maximum likelihood point estimatorof  is the value of  that maximizes L( ; x) or equivalently maximizes l( ; x) Useful notation: With a k-dimensional parameter:

  2. Complete sample case: If all sample values are explicitly known, then Censored data case: If some ( say nc)of the sample values are censored , e.g. xi< k1 or xi> k2 , then where

  3. When the sample comes from a continuous distribution the censored data case can be written In the case the distribution is discrete the use of F is also possible: If k1 and k2 are values that can be attained by the random variables then we may write where

  4. Example:

  5. Example: Solution must be numerically found

  6. For the exponential family of distributions: Use the canonical form (natural parameterization): Let Then the maximum likelihood estimators (MLEs) of 1, … , k are found by solving the system of equations

  7. Example:

  8. Computational aspects • When the MLEs can be found by evaluating • numerical routines for solving the generic equation g( ) = 0 can be used. • Newton-Raphson method • Fisher’s method of scoring (makes use of the fact that under regularity conditions: • ) • This is the multidimensional analogue of Lemma 2.1 ( see page 17)

  9. When the MLEs cannot be found the above way other numerical routines must be used: • Simplex method • EM-algorithm • For description of the numerical routines see textbook. • Maximum Likelihood estimation comes into natural use not for handling the standard case, i.e. a complete random sample from a distribution within the exponential family , but for finding estimators in more non-standard and complex situations.

  10. Example:

  11. Properties of MLEs Invariance: Consistency: Under some weak regularity conditionsall MLEs are consistent Efficiency: Under the usual regularity conditions: (Asymptotically efficient and normally distributed)

  12. Sufficiency: Example:

  13. Invariance property 

  14. i.e. the two MLEs are asymptotically uncorrelated (and by the normal distribution independent)

  15. Modifications and extensions Ancillarity and conditional sufficiency:

  16. Profile likelihood: This concept has its main use in cases where  1 contains the parameters of “interest” and  2 contains nuisance parameters. The same ML point estimator for  1 is obtained by maximizing the profile likelihood as by maximizing the full likelihood function

  17. Marginal and conditional likelihood: Again, these concepts have their main use in cases where  1 contains the parameters of “interest” and  2 contains nuisance parameters.

  18. Penalized likelihood: MLEs can be derived subjected to some criteria od smoothness. In particulare this is applicable when the parameter is no longer a single value (one- or multidimensional), but a function such as an unknown density function or a regression curve. The penalized log-likelihood function is written

  19. Method of moments estimation (MM )

  20. The method of moments point estimator of  = ( 1, … ,  k ) is obtained by solving for  1, … ,  k the systems of equations

  21. Example:

  22. Method of Least Squares (LS) First principles: Assume a sample xwhere the random variable Xi can be written The least-squares estimator of  is the value of  that minimizes i.e.

  23. A more general approach: Assume the sample can be written (x, z ) where xirepresents the random variable of interest (endogenous variable) and zi represent either an auxiliary random variable (exogenous) or a given constant for sample point i The least squares estimator of  is then

  24. Special cases: The ordinary linear regression model: The heteroscedastic regression model:

  25. The first-order auto-regressive model: The conditional least-squares estimator of  (given ) is

More Related