1 / 40

Linear Models for Classification : Probabilistic Methods

Linear Models for Classification : Probabilistic Methods. Adopted from Seung-Joon Yi Biointelligence Laboratory, Seoul National University http://bi.snu.ac.kr/. Recall, Linear Methods for Classification.

jered
Download Presentation

Linear Models for Classification : Probabilistic Methods

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. Linear Models for Classification: Probabilistic Methods Adopted from Seung-Joon Yi Biointelligence Laboratory, Seoul National University http://bi.snu.ac.kr/

  2. Recall, Linear Methods for Classification Problem Definition: Given the training data {xn,tn}, find a linear model for each class yk(x) to partition the feature space into decision regions • Deterministic Models: • Discriminant Functions • Fisher Discriminant function • Perceptron

  3. Probabilistic Approaches for Classification • Generative Models: • Inference : Model p(x/Ck) and p(Ck) • Decision : Model p(Ck/x) • Discriminative Models • Model p(Ck/x) directly • Use the functional form of the generalized linear model explicitly • Determine the parameters directly using Maximum Likelihood

  4. simple[2] logistic function may be defined by the formula Logistic Sigmoid Function • Comes from population growth • Prob distribution function of Normal R.V. İs Logistic sigmoid • İf class conditional densities are Normal, posteriors become logistic sigmoid

  5. Posterior Probabilities can be formulated by • 2-Class: Logistic sigmoid acting on a linear function of x • K-Class: Softmax transformation of a linear function of x Then, • The parameters of the densities as well as the class priors can be determined using Maximum Likelihood

  6. Probabilistic Generative Models: 2-Class • Recall, given • Posterior can be expresses by Logistic Sigmoid • a is called logit function

  7. Probabilistic Generative Models K-Class • Posterior can be expresses by Softmax function or normalized exponential • Multi-class generalisation of logistic sigmoid:

  8. Probabilistic Generative ModelsGaussian Class Conditionals for 2-Class • Assume same covariance matrix ∑, • Note • The quadratic terms in x from the exponents are cancelled. • The resulting decision boundary is linear in input space. • The prior only shifts the decision boundary, i.e. parallel contour.

  9. Probabilistic Generative Models: Gaussian Class Conditionals for K-classes • When, covariance matrix is the same, decision boundaries are linear. • When, each class-condition density have its own covariance matrix, ak becomes quadratic functions of x, giving rise to a quadratic discriminant. (C) 2006, SNU Biointelligence Lab, http://bi.snu.ac.kr/

  10. Probabilistic Generative Models-Maximum Likelihood Solution- • Two classes • Given

  11. Q: Find P(C1) = π and P(C2) = 1- π and parameters of p(Ck/x): μ1, μ2 and 

  12. Probabilistic Generative Models-Maximum Likelihood SolutionLet P(C1) = π and P(C2) = 1- π

  13. Probabilistic Generative Models-Maximize log likelihood w r to. π ,μ1 μ2. ∑ .

  14. Probabilistic Generative Models-Discrete Features- • Discrete feature values • When we have D inputs, the table size grows exponentially with the number of featuresto a 2D size table. • . • Naïve Bayes assumption, conditioned on the class Ck • Linear with respect to the features as in the continuous features. (C) 2006, SNU Biointelligence Lab, http://bi.snu.ac.kr/

  15. Bayes Decision Boundaries: 2D-Pattern Classification, Duda et al. pp.42 (C) 2006, SNU Biointelligence Lab, http://bi.snu.ac.kr/

  16. Bayes Decision Boundaries: 3D-Pattern Classification, Duda et al. pp.43 (C) 2006, SNU Biointelligence Lab, http://bi.snu.ac.kr/

  17. For both Gaussian distributed and discrete inputs • The posterior class probabilities are given by • Generalized linear models with logistic sigmoid or • softmax activation functions.

  18. Probabilistic Generative Models-Exponential Family- • Recall, bernoulli, binomial, multinomial, Gaussian can be expressed in a general form

  19. Probabilistic Generative ModelsExponential Family- • 2- Classes: Logistic Function • The subclass for which u(x) = x. • K-Classes: Softmax function. Linear with respect to x again.

  20. Probabilistic Discriminative Models • Goal: Find p(Ck/x) directly • No inferrence step • Discriminative Training: Max likelihood p(Ck/x) • İmproves prediction performance when p(x/Ck) is poorly estimated

  21. Fixed basis functions: x • Assume fixed nonlinear transformation • Transform inputs using a vector of basis functions • The resulting decision boundaries will be linear in the feature space y(x)= WT Φ (C) 2006, SNU Biointelligence Lab, http://bi.snu.ac.kr/

  22. Posterior probability of a class for two-class problem: • The number of adjustable parameters (M-dimensional, 2-class) • 2 Gaussian class conditional densities (generative model) • 2M parameters for means • M(M+1)/2 parameters for (shared) covariance matrix • Grows quadratically with M • Logistic regression (discriminative model) • M parameters for • Grows linearly with M

  23. Determining the parameters using Likelihood function: • Take negative log likelihood: Cross-entropy error function • Recall,cross entropy between two probability distributions measures the average number of bits needed to identify an event from a set of possibilities, if a coding scheme is used based on a given probability distribution q, rather than the "true" distribution p. (C) 2006, SNU Biointelligence Lab, http://bi.snu.ac.kr/

  24. The gradient of the error function w.r.t. W • The same form as the linear regressionpredictiontarget value

  25. Iterative Reweighted Least Squares • Recall, Linear regression models in ch.3 • ML solution on the assumption of a Gaussian noise leads to a close-form solution, as a consequence of the quadratic dependence of the log likelihood on the parameter w. • Logistic regression model • No longer a closed-form solution • But the error function is concave and has a unique minimum • Efficient iterative technique can be used • The Newton-Raphson update to minimize a function E(w) • Where H is the Hessian matrix, the second derivatives of E(w) (C) 2006, SNU Biointelligence Lab, http://bi.snu.ac.kr/

  26. Iterative reweighted least squares (Cont’d) • CASE 1: SSE function: • Newton-Raphson update: • CASE 2: Cross-entropy error function: • Newton-Rhapson update: (iterative reweighted least squares) (C) 2006, SNU Biointelligence Lab, http://bi.snu.ac.kr/

  27. Multiclass logistic regerssion • Posterior probability for multiclass classification • We can use ML to determine the parameters directly. • Likelihood function using 1-of-K coding scheme • Cross-entropy error function for the multiclass classification

  28. Multiclass logistic regression (Cont’d) • The derivative of the error function • Same form, the product of error times the basis function. • The Hessian matrix • IRLS algorithm can also be used for a batch processing (C) 2006, SNU Biointelligence Lab, http://bi.snu.ac.kr/

  29. Generalized Linear Models • Recall, for a broad range of class-conditional distributions, described by the exponential family, the resulting posterior class probabilities are given by a logistic(or softmax) transformation acting on a linear function of the feature variables. • However this is not the case for all choices of class-conditional density • It might be worth exploring other types of discriminative probabilistic model (C) 2006, SNU Biointelligence Lab, http://bi.snu.ac.kr/

  30. Generalized Linear Model: 2 Classes For example: For each input, we evaluate an=wTΦn θ

  31. Noisy Threshold model • Corresponding activation function when θ is drawn from p(θ), mixture of Gaussian

  32. Probit Function • Sigmoidal shape • The generalized linear model based on a probit activation function is known as probit regression.

  33. Canonical link functions • Recall, if we take the derivative of the error function w.r.t the parameter w, it takes the form of the error times the feature vector. • Logistic regression model with sigmoid activation function • Logistic regression model with softmax activation function • This is a general result of assuming a conditional distribution for the target variable from the exponential family, along with a corresponding choice for the activation function known as the canonical link function.

  34. Canonical link functions (Cont’d) • Consider the exponential family, Conditional distributions of the target variable • Log likelihood: • The derivative of the log likelihood: where • The canonical link function: then (C) 2006, SNU Biointelligence Lab, http://bi.snu.ac.kr/

  35. The Laplace approximation • Goal:Find a Gaussian approximation to a non-Gaussian density, centered on the mode z0of the distribution. • Suppose: p(z)= (1 /Z)f(z) , non Gaussian • Taylor expansion,arround mode z0,of the logarithm of the target function: • Resulting approximated Gaussian distribution:

  36. Laplace approximation forp(z) ∝ exp(-z2/2)σ(20z +4) • Left: the normalized distribution p(z) in yellow, together with the Laplace approximation centred on the mode z0 of p(z) in red. • Right:The negative logarithms of the corresponding curves (C) 2006, SNU Biointelligence Lab, http://bi.snu.ac.kr/

  37. Model comparison and BIC • Laplace approximation to the normalization constant Z • This result can be used to obtain an approximation to the model evidence, which plays a central role in Bayesian model comparison. • Consider a set of models having parameters • The log of model evidence can be approximated as • Further approximation with some more assumption: Bayesian Information Criterion (BIC) (C) 2006, SNU Biointelligence Lab, http://bi.snu.ac.kr/

  38. Bayesian Logistic Regression • Exact Bayesian inference is intractable. • Gaussian prior: • Posterior: • Log of posterior: • Laplace approximation of posterior distribution (C) 2006, SNU Biointelligence Lab, http://bi.snu.ac.kr/

  39. Predictive distribution • Can be obtained by marginalizing w.r.t the posterior distribution p (w|t) which is approximated by a Gaussian q(w) where • a is a marginal distribution of a Gaussian which is also Gaussian (C) 2006, SNU Biointelligence Lab, http://bi.snu.ac.kr/

  40. Predictive distribution • Resulting variational approximation to the predictive distribution • To integrate over a, we make use of the close similarity between the logistic sigmoid function and the probit function Then where • Finally we get (C) 2006, SNU Biointelligence Lab, http://bi.snu.ac.kr/

More Related